Polaris Telemetry and Audit Trail

[Anand Kumar Sankaran via dev <de...@polaris.apache.org>]

Hi all

My first PR for adding AWS STS Session Tags support for credential vending was merged. https://github.com/apache/polaris/pull/3327

I have been working with Prashant on an end-to-end telemetry and audit trail tracking for Polaris.  It is documented here. https://docs.google.com/document/d/1Ehzvi5RNPs4hChkBFI6VD23myEqm-7sWW3d2kjmuYj8/edit?tab=t.0

Based on Prashant’s initial feedback (parity with Apache Gravitino for metrics reports), I have an initial PR here. https://github.com/apache/polaris/pull/3385

I will be reaching out to various folks for review. I am open to breaking down the PR into smaller logical groups of PR if it helps.

I request you to look at the Google doc and let me know what you think.

—
Anand

Re: Polaris Telemetry and Audit Trail

[Dmitri Bourlatchkov <dm...@dremio.com>]

Hi Anand,

I left some comments on the SPI PR [3616].

From your javadoc on ReportIdToken, I can see that you have given some
thought to time-series backends for this data. Do you have any concrete
implementations in mind?

What do you think might be a good fit for storage other than JDBC / RDBMS?

[3616] https://github.com/apache/polaris/pull/3616

Thanks,
Dmitri.

On Thu, Jan 29, 2026 at 1:49 PM Anand Kumar Sankaran <
anand.sankaran@workday.com> wrote:

> Hi Dmitri,
>
> Thank you for taking the time to review. I had pulled out the schema in
> its own PR (based on comments) here:
> https://github.com/apache/polaris/pull/3523
>
> I will update the document with the SPI and carve out another PR that
> focuses on the SPI.
>
> —
> Anand
>
> *From: *Dmitri Bourlatchkov <dm...@dremio.com>
> *Date: *Thursday, January 29, 2026 at 10:43 AM
> *To: *dev@polaris.apache.org <de...@polaris.apache.org>
> *Cc: *Anand Kumar Sankaran <an...@workday.com>
> *Subject: *Re: Polaris Telemetry and Audit Trail
>
> This Message Is From an External Sender
> This message came from outside your organization.
> Report Suspicious
> <https://us-phishalarm-ewt.proofpoint.com/EWT/v1/Iz9xO38YGHZK!YhNDZAGr2cumY1cAlUeoPrU2eNUPz9Wxq2PaG1t99vEN3v60gucxC3Qsk-xK09RndcW46WtNjMMjQZbaQ495h1D1g293F_uqLhPM8gDMlA6xRmo5b5uqrSh2v7QKHrhQ$>
>
> Hi Anand,
>
> Thanks for making a proposal doc and starting this thread. Apologies for
> late comments.
>
> As you know I already reviewed a couple of related PRs, which had a
> smaller impact on the codebase, but now, I believe, we're coming to the
> core of this feature and it might need a deeper discussion.
>
> For the sake of awareness of other project members, I'd like to
> highlight some aspects by email.
>
> * Storing scan metrics in RDBMS in general looks like a
> reasonable approach to me. However, Polaris Persistence is an extensible
> mechanism. Other backends are possible, including privately developed
> backends.
>
> From this POV, I believe it might be preferable to start not with an RDBMS
> schema, but with a java SPI outlining expected read/write operations and
> the data model. This will make it easier to assess impact on other
> Persistence implementations. Naturally, Polaris service code will have to
> be able to work seamlessly regardless of the backend impl. (including the
> do-nothing impl.).
>
> I see that some connection between the SQL schema and java code is made in
> PR [3385], but it does not appear to offer a clean SPI that could be
> implemented by different Persistence backends. Would it be ok from your
> perspective to make another PR with just SPI code?
>
> * Regarding ad-hoc SQL queries - it is certainly a valid use case for
> custom code to query the Polaris database directly. However, those ad-hoc
> queries may require certain indexes, which may not have a relevant use case
> in OSS code. In that case, I believe it would be preferable to manage the
> extra indexes in custom code (not in Polaris). Whether this is the case or
> not currently, I cannot say with certainty, so apologies if this is a false
> alarm... Having the java SPI available would be a huge aid to reasoning
> about the proposed SQL schema, I think.
>
> That said, I do not mean to complicate the development of this feature,
> just trying to structure it in a way that is hopefully easy to understand
> (including myself), maintain and expand later. If I missed something,
> please feel free to point it out.
>
> [3385] https://github.com/apache/polaris/pull/3385
> <https://urldefense.com/v3/__https://github.com/apache/polaris/pull/3385__;!!Iz9xO38YGHZK!_dbBfwEnZEuPp_8qIuBDoYfWRgCZKiPDMg9ouDltqsHhr9hqCHVlF9IedSkNVBDWupbneU7inC9YK-THL-lTjWkkyZhI0OE$>
>
> Thanks,
> Dmitri.
>
> On Wed, Jan 14, 2026 at 10:05 PM Anand Kumar Sankaran via dev <
> dev@polaris.apache.org> wrote:
>
> Hi Yufei,
>
> Gave commenting privileges to all.
>
> Thanks.
>
> —
> Anand
>
> From: Yufei Gu <fl...@gmail.com>
> Date: Wednesday, January 14, 2026 at 6:13 PM
> To: dev@polaris.apache.org <de...@polaris.apache.org>
> Cc: Anand Kumar Sankaran <an...@workday.com>
> Subject: Re: Polaris Telemetry and Audit Trail
>
> This Message Is From an External Sender
> This message came from outside your organization.
> Report Suspicious<
> https://us-phishalarm-ewt.proofpoint.com/EWT/v1/Iz9xO38YGHZK!YhNDZABkHi1B6hyPVYUK0T-K5j4Aoqdrlu07UjBpJcpjOsz7Ie7d_DhPl-ywi50dfQz4O6LL-AHLCOQTQ0X7Le8yD2cF0ogquGFB8Aw8uQLgB0RL6Ezpa-pLDPA4ziXA$
> >
>
> Hi Anand, thanks for the design doc and PR. Can you allow comments in the
> doc so that people can chime in? Thanks!
>
> Yufei
>
>
> On Sat, Jan 10, 2026 at 9:09 AM Anand Kumar Sankaran via dev <
> dev@polaris.apache.org<ma...@polaris.apache.org>> wrote:
> Hi all
>
> My first PR for adding AWS STS Session Tags support for credential vending
> was merged. https://github.com/apache/polaris/pull/3327
> <https://urldefense.com/v3/__https://github.com/apache/polaris/pull/3327__;!!Iz9xO38YGHZK!_dbBfwEnZEuPp_8qIuBDoYfWRgCZKiPDMg9ouDltqsHhr9hqCHVlF9IedSkNVBDWupbneU7inC9YK-THL-lTjWkklJ4Xf9Y$>
> <
> https://urldefense.com/v3/__https://github.com/apache/polaris/pull/3327__;!!Iz9xO38YGHZK!8vlQ2_OqjmZNIpbrTTluFFNKFCEeM87nr95MP5pMpMu5fFWVSoBZ3eb6-C2xbAGCG-ry5eaCO-pFLxl3IZAJGqY$
> >
>
> I have been working with Prashant on an end-to-end telemetry and audit
> trail tracking for Polaris.  It is documented here.
> https://docs.google.com/document/d/1Ehzvi5RNPs4hChkBFI6VD23myEqm-7sWW3d2kjmuYj8/edit?tab=t.0
> <https://urldefense.com/v3/__https://docs.google.com/document/d/1Ehzvi5RNPs4hChkBFI6VD23myEqm-7sWW3d2kjmuYj8/edit?tab=t.0__;!!Iz9xO38YGHZK!_dbBfwEnZEuPp_8qIuBDoYfWRgCZKiPDMg9ouDltqsHhr9hqCHVlF9IedSkNVBDWupbneU7inC9YK-THL-lTjWkk8MVJPlY$>
> <
> https://urldefense.com/v3/__https://docs.google.com/document/d/1Ehzvi5RNPs4hChkBFI6VD23myEqm-7sWW3d2kjmuYj8/edit?tab=t.0__;!!Iz9xO38YGHZK!8vlQ2_OqjmZNIpbrTTluFFNKFCEeM87nr95MP5pMpMu5fFWVSoBZ3eb6-C2xbAGCG-ry5eaCO-pFLxl3hYqTlcg$
> >
>
> Based on Prashant’s initial feedback (parity with Apache Gravitino for
> metrics reports), I have an initial PR here.
> https://github.com/apache/polaris/pull/3385
> <https://urldefense.com/v3/__https://github.com/apache/polaris/pull/3385__;!!Iz9xO38YGHZK!_dbBfwEnZEuPp_8qIuBDoYfWRgCZKiPDMg9ouDltqsHhr9hqCHVlF9IedSkNVBDWupbneU7inC9YK-THL-lTjWkkyZhI0OE$>
> <
> https://urldefense.com/v3/__https://github.com/apache/polaris/pull/3385__;!!Iz9xO38YGHZK!8vlQ2_OqjmZNIpbrTTluFFNKFCEeM87nr95MP5pMpMu5fFWVSoBZ3eb6-C2xbAGCG-ry5eaCO-pFLxl39opCznI$
> >
>
> I will be reaching out to various folks for review. I am open to breaking
> down the PR into smaller logical groups of PR if it helps.
>
> I request you to look at the Google doc and let me know what you think.
>
> —
> Anand
>
>
>
> --
> Dmitri Bourlatchkov
> Senior Staff Software Engineer, Dremio
> Dremio.com
> <https://urldefense.com/v3/__https://www.dremio.com/?utm_medium=email&utm_source=signature&utm_term=na&utm_content=email-signature&utm_campaign=email-signature__;!!Iz9xO38YGHZK!_dbBfwEnZEuPp_8qIuBDoYfWRgCZKiPDMg9ouDltqsHhr9hqCHVlF9IedSkNVBDWupbneU7inC9YK-THL-lTjWkkiOwjJc0$> /
> Follow Us on LinkedIn
> <https://urldefense.com/v3/__https://www.linkedin.com/company/dremio__;!!Iz9xO38YGHZK!_dbBfwEnZEuPp_8qIuBDoYfWRgCZKiPDMg9ouDltqsHhr9hqCHVlF9IedSkNVBDWupbneU7inC9YK-THL-lTjWkkCLjBQ2M$> /
> Get Started
> <https://urldefense.com/v3/__https://www.dremio.com/get-started/__;!!Iz9xO38YGHZK!_dbBfwEnZEuPp_8qIuBDoYfWRgCZKiPDMg9ouDltqsHhr9hqCHVlF9IedSkNVBDWupbneU7inC9YK-THL-lTjWkkoKQUMII$>
>
>
> The Agentic Lakehouse
> *The only lakehouse built for agents, managed by agents*
>
>
>

-- 
Dmitri Bourlatchkov
Senior Staff Software Engineer, Dremio
Dremio.com
<https://www.dremio.com/?utm_medium=email&utm_source=signature&utm_term=na&utm_content=email-signature&utm_campaign=email-signature>
/
Follow Us on LinkedIn <https://www.linkedin.com/company/dremio> / Get
Started <https://www.dremio.com/get-started/>


The Agentic Lakehouse
The only lakehouse built for agents, managed by agents

Re: Proposal for REST endpoints for table metrics and events

[Yufei Gu <fl...@gmail.com>]

Hi JB,

Thanks for raising this, I agree that we should make sure the process stays
welcoming and that contributors have a clear path forward.

That said, I don’t think contributors are currently blocked. The discussion
so far has mainly been about aligning with Iceberg and avoiding APIs that
we may need to significantly change later. From my perspective, this is
about ensuring long term consistency rather than slowing things down. Here
is a precedent example to ask for alignment in the Iceberg community to
move forward with Polaris.

One thing I’d like to call out is that some of the earlier design questions
haven’t been fully addressed yet. In particular:

Do we need to expose metrics via a Polaris REST endpoint at all? Users
could potentially access metrics directly from the backend storage. While
this may not provide RBAC, it offers flexibility. For example, some
deployments may choose alternative persistence models such as a KV store or
storing metrics as objects in S3, which can scale better than an RDBMS like
Postgres. More broadly, I don’t see Polaris as a full metrics system, but
rather as a mechanism for downstream systems to consume this data.
Clarifying this boundary would help guide whether a REST API is the right
abstraction here.

For the metrics API specifically, since there isn’t an active effort in the
Iceberg community yet, I think it makes sense to continue moving forward in
Polaris, while also starting a discussion upstream to explore alignment.

Overall, I’m supportive of continuing the work, while being mindful of
these design questions and compatibility with Iceberg where it matters.

Curious to hear others’ thoughts.

1. https://github.com/apache/polaris/pull/2048#issuecomment-3187155240

Yufei


On Fri, Apr 3, 2026 at 5:52 AM Dmitri Bourlatchkov <di...@apache.org> wrote:

> Hi All,
>
> +1 to JB's points.
>
> I also welcome Anand's contributions to Polaris and I believe they are
> beneficial to the project and the larger community.
>
> Supporting multiple APIs backed by the same data in storage is perfectly
> normal. Most of the maintenance work happens at the layer that handles
> actual data. I would not expect much overhead in adding Iceberg APIs (once
> they are released) to the body of code currently proposed in Anand's PRs.
>
> Cheers,
> Dmitri.
>
> On Fri, Apr 3, 2026 at 7:34 AM Jean-Baptiste Onofré <jb...@nanthrax.net>
> wrote:
>
> > Hi all,
> >
> > I am concerned by this discussion from a contribution standpoint.
> >
> > Regarding the point about Iceberg synchronization, it is discouraging
> for a
> > passionate and committed contributor to spend a month in discussion
> without
> > a clear path forward. We have precedents for starting implementations in
> > Polaris with the intention of syncing with the Iceberg community later.
> >
> > If we were certain that the Event API in Iceberg would be completed
> within
> > a week or so, I would agree with the current plan. However, I do not
> > believe that is the case. Projects evolve, and I see no issue with
> > revisiting or updating current implementations in the future.
> >
> > I am in favor of moving forward with Anand's PR and creating an issue to
> > track the Iceberg Event API update. I am sure Anand would be happy to
> > rework the implementation once there is a clearer picture. To do
> otherwise
> > is not welcoming to contributors.
> >
> > Regards,
> > JB
> >
> > On Thu, Apr 2, 2026 at 11:56 PM Anand Kumar Sankaran via dev <
> > dev@polaris.apache.org> wrote:
> >
> > > Hi all,
> > >
> > > I have an implementation of the metrics REST API available for review
> > > https://github.com/apache/polaris/pull/4115. Unlike the events API,
> > there
> > > is no proposal yet in Iceberg community yet. I need to a way to read
> the
> > > persisted table metrics and events as we are rolling out to production.
> > >
> > > Please look at this. If this is not acceptable and we prefer to wait
> for
> > > the Iceberg community, I would have to implement them as proprietary
> > > extensions to the REST API to move forward.
> > >
> > > —
> > > Anand
> > >
> > > From: Yufei Gu <fl...@gmail.com>
> > > Date: Monday, March 23, 2026 at 11:53 AM
> > > To: dev@polaris.apache.org <de...@polaris.apache.org>
> > > Subject: Re: Proposal for REST endpoints for table metrics and events
> > >
> > > This Message Is From an External Sender
> > > This message came from outside your organization.
> > > Report Suspicious<
> > >
> >
> https://us-phishalarm-ewt.proofpoint.com/EWT/v1/Iz9xO38YGHZK!YhNDZABkHi1B6_8r_oHOHjE1XzdKC66MNTZUq0rzz6aeuQJd2NnDmIlKwHgrUG7hWJidUrkh_sDqEZ3ZHa1HIzhjNw83YQe2WEa1Gn4LzkyEDLyXuTmvkoY9fnc8GtBQ$
> > > >
> > >
> > >
> > > I think the Events API in Iceberg is already quite close to
> finalization,
> > > so I don’t see much value in introducing a separate unstable version in
> > > Polaris ATM.
> > >
> > > For the metrics endpoint, I’m not aware of any ongoing discussion in
> the
> > > Iceberg community. I’d suggest starting there first. If there isn’t
> > > interest, Polaris can move forward and introduce a metrics endpoint
> > > directly, without necessarily making it experimental. If there is
> > interest,
> > > it would make more sense to align with Iceberg and contribute there. We
> > can
> > > then decide whether we still need a fallback option in Polaris, such as
> > an
> > > experimental endpoint.
> > >
> > > Overall, it seems worth checking with the Iceberg community before
> moving
> > > forward.
> > >
> > > Yufei
> > >
> > >
> > > On Mon, Mar 23, 2026 at 11:33 AM Dmitri Bourlatchkov <dimas@apache.org
> >
> > > wrote:
> > >
> > > > Hi Nandor,
> > > >
> > > > Good point! The new metrics and events APIs should definitely be
> marked
> > > as
> > > > "experimental" or "beta" initially.
> > > >
> > > > That's a common practice for new APIs in Polaris.
> > > >
> > > > Cheers,
> > > > Dmitri.
> > > >
> > > > On Thu, Mar 19, 2026 at 4:30 PM Nándor Kollár <nk...@apache.org>
> > > wrote:
> > > >
> > > > > Hi All,
> > > > >
> > > > > I don’t see a problem with having two REST APIs for catalog events.
> > We
> > > > can
> > > > > mark the local API as experimental (if there’s a way to do that in
> > > > > Polaris), document that it isn’t stable, and deprecate it once the
> > > > Iceberg
> > > > > API is released. Alternatively, we could keep and continue
> improving
> > it
> > > > > with features that are missing from the Iceberg REST spec but are
> > > > relevant
> > > > > for Polaris.
> > > > >
> > > > > Nandor
> > > > >
> > > > > Dmitri Bourlatchkov <di...@apache.org> ezt írta (időpont: 2026.
> > márc.
> > > > 19.,
> > > > > Cs, 17:44):
> > > > >
> > > > > > Hi All,
> > > > > >
> > > > > > Polaris can support both a local API and new endpoints in the IRC
> > API
> > > > > once
> > > > > > the Iceberg community adopts the latter.
> > > > > >
> > > > > > What is the concern with having different APIs to access the same
> > > data?
> > > > > >
> > > > > > Cheers,
> > > > > > Dmitri.
> > > > > >
> > > > > > On Tue, Mar 17, 2026 at 3:49 PM EJ Wang <
> > > > ej.wang.dev.interest@gmail.com>
> > > > > > wrote:
> > > > > >
> > > > > > > I share the same feeling with Yifei as reviewing the PRs. I
> want
> > to
> > > > > avoid
> > > > > > > creating discrepancies if the IRC side ended up supporting it.
> > > > > > >
> > > > > > > -ej
> > > > > > >
> > > > > > > On Wed, Mar 11, 2026 at 2:47 PM Yufei Gu <flyrain000@gmail.com
> >
> > > > wrote:
> > > > > > >
> > > > > > > > Thanks Anand for working on this.
> > > > > > > >
> > > > > > > > Given the IRC event endpoint is WIP, I think it'd be best to
> be
> > > > > > > consistent
> > > > > > > > with the IRC endpoint. For more context on the IRC event, the
> > > event
> > > > > > > > endpoint is not finalized yet. I'd recommend speeding up the
> > IRC
> > > > side
> > > > > > > work
> > > > > > > > to avoid any inconsistencies between Polaris and IRC spec.
> > > > > > > >
> > > > > > > > I have a few questions about the metrics endpoint:
> > > > > > > > 1. Do we need to expose them via Polaris REST endpoint? Can
> > users
> > > > > grab
> > > > > > > the
> > > > > > > > metrics from the backend directly? I understand the RBAC
> won't
> > be
> > > > > > there,
> > > > > > > > but it provides flexibility for users. For example, some
> users
> > > may
> > > > > > > choose a
> > > > > > > > different persistence model such as a KV store or storing the
> > > > metrics
> > > > > > as
> > > > > > > > objects in S3, which usually scales better than an RDBMS like
> > > > > Postgres.
> > > > > > > My
> > > > > > > > understanding is that Polaris is not intended to be a full
> > > metrics
> > > > > > system
> > > > > > > > anyway, but rather to provide a way for downstream systems to
> > > > consume
> > > > > > > these
> > > > > > > > data.
> > > > > > > > 2. Should we consider it as an IRC endpoint? Given the
> metrics
> > > > report
> > > > > > > > endpoint in IRC, the Iceberg community might also be
> interested
> > > in
> > > > > > > serving
> > > > > > > > back metrics, similar to the event endpoint. In that case,
> > there
> > > > is a
> > > > > > > risk
> > > > > > > > of fragmentation if we create a Polaris endpoint now. We
> should
> > > > avoid
> > > > > > > that
> > > > > > > > if possible. It might be worth checking with the Iceberg
> > > community
> > > > > > first.
> > > > > > > >
> > > > > > > > Happy to hear others’ thoughts on this.
> > > > > > > >
> > > > > > > > Yufei
> > > > > > > >
> > > > > > > >
> > > > > > > > On Wed, Mar 11, 2026 at 9:27 AM Nándor Kollár <
> > > nkollar@apache.org>
> > > > > > > wrote:
> > > > > > > >
> > > > > > > > > I agree with Dmitri. The direction outlined in the proposal
> > > looks
> > > > > > good
> > > > > > > to
> > > > > > > > > me, and the finer details can be worked out as
> implementation
> > > > gets
> > > > > > > > > underway. We can adjust the design doc accordingly, later
> on.
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > Thanks,
> > > > > > > > > Nandor
> > > > > > > > >
> > > > > > > > > Dmitri Bourlatchkov <di...@apache.org> ezt írta (időpont:
> > > 2026.
> > > > > > márc.
> > > > > > > > 10.,
> > > > > > > > > K, 20:43):
> > > > > > > > >
> > > > > > > > > > Hi Anand and All,
> > > > > > > > > >
> > > > > > > > > > The proposal LGTM in its current form.
> > > > > > > > > >
> > > > > > > > > > My personal approach is that a proposal does not have to
> be
> > > as
> > > > > > > > "polished"
> > > > > > > > > > as the final API spec. As long as we have consensus on
> the
> > > > > general
> > > > > > > > > > approach and the basic API principles, I think we can
> > proceed
> > > > to
> > > > > > > > > > implementation and iron out final wrinkles during actual
> > API
> > > > spec
> > > > > > and
> > > > > > > > > code
> > > > > > > > > > PRs.
> > > > > > > > > >
> > > > > > > > > > Would this approach work for everyone?
> > > > > > > > > >
> > > > > > > > > > Re-posting my comment from GH [1] here for visibility, in
> > > case
> > > > > > people
> > > > > > > > > have
> > > > > > > > > > different opinions and wish to discuss this in more
> depth:
> > > > > > > > > >
> > > > > > > > > > From my POV, Polaris is a platform, indeed. In this
> sense,
> > I
> > > > > think
> > > > > > it
> > > > > > > > is
> > > > > > > > > > > critical to enable users to control what features are
> at
> > > play
> > > > > in
> > > > > > > > > runtime,
> > > > > > > > > > > since different users have different use cases. This is
> > > why I
> > > > > > > > > originally
> > > > > > > > > > > advocated for isolating Metrics Persistence from
> > MetaStore
> > > > > > > > Persistence.
> > > > > > > > > > >
> > > > > > > > > > > If a user decides to leverage the "native" Polaris
> > > > > (scan/commit)
> > > > > > > > > Metrics
> > > > > > > > > > > Persistence, I do not see any disadvantage in also
> > exposing
> > > > an
> > > > > > > > > (optional)
> > > > > > > > > > > REST API for loading these metrics from Polaris
> > > Persistence.
> > > > > > > > > > >
> > > > > > > > > > > The degree of support and sophistication that goes into
> > > this
> > > > > > > > sub-system
> > > > > > > > > > is
> > > > > > > > > > > up to the community. If we have contributors (like
> > > @obelix74
> > > > )
> > > > > > who
> > > > > > > > are
> > > > > > > > > > > willing to evolve it, I see no harm in some
> functionality
> > > > > overlap
> > > > > > > > with
> > > > > > > > > > more
> > > > > > > > > > > focused metrics platforms. Again, the key point is for
> > all
> > > > > users
> > > > > > to
> > > > > > > > > have
> > > > > > > > > > > control and be able to opt in or out of this feature in
> > > their
> > > > > > > > specific
> > > > > > > > > > > deployments.
> > > > > > > > > > >
> > > > > > > > > > > Of course, offloading scan/commit metrics storage to a
> > > > > > specialized
> > > > > > > > > > > observability system is possible too (assuming someone
> > > > develops
> > > > > > > > > > integration
> > > > > > > > > > > code for that, which is very welcome).
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > [1]
> > > > > > >
> > >
> >
> https://urldefense.com/v3/__https://github.com/apache/polaris/pull/3924*discussion_r2913947744__;Iw!!Iz9xO38YGHZK!5rBOI-ESxR2K-8_HPN195cT-mCOoNghT74Di8M2TIB97esSZ7-VrE50guaHsg7gd_9_HBIqfdgZ1NAhyQCoxjlo$
> > > > > > > > > >
> > > > > > > > > > Thanks,
> > > > > > > > > > Dmitri.
> > > > > > > > > >
> > > > > > > > > > On Tue, Mar 10, 2026 at 12:37 PM Anand Kumar Sankaran via
> > > dev <
> > > > > > > > > > dev@polaris.apache.org> wrote:
> > > > > > > > > >
> > > > > > > > > > > Hi EJ Wang and Dmitri,
> > > > > > > > > > >
> > > > > > > > > > > I addressed all your concerns about the proposal, in
> > > > particular
> > > > > > > > > > >
> > > > > >
> > >
> >
> https://urldefense.com/v3/__https://github.com/apache/polaris/pull/3924*discussion_r2908317696__;Iw!!Iz9xO38YGHZK!5rBOI-ESxR2K-8_HPN195cT-mCOoNghT74Di8M2TIB97esSZ7-VrE50guaHsg7gd_9_HBIqfdgZ1NAhyoAh9fLA$
> > > > > > > .
> > > > > > > > > > >
> > > > > > > > > > > Does this address your concerns?
> > > > > > > > > > >
> > > > > > > > > > > -
> > > > > > > > > > > Anand
> > > > > > > > > > >
> > > > > > > > > > > From: Dmitri Bourlatchkov <di...@apache.org>
> > > > > > > > > > > Date: Monday, March 9, 2026 at 1:17 PM
> > > > > > > > > > > To: dev@polaris.apache.org <de...@polaris.apache.org>
> > > > > > > > > > > Subject: Re: Proposal for REST endpoints for table
> > metrics
> > > > and
> > > > > > > events
> > > > > > > > > > >
> > > > > > > > > > > This Message Is From an External Sender
> > > > > > > > > > > This message came from outside your organization.
> > > > > > > > > > > Report Suspicious<
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://us-phishalarm-ewt.proofpoint.com/EWT/v1/Iz9xO38YGHZK!YhNDZAGr2cumYdtJ_UTm0gR9PfI_-PwSpR_GtNr1uVQ_xo-s2AskvUmbkLZ-C5V8eOKN-omus47On4k4hfFo-0G7CHMLwVjEego-rZrPuepAybX7DP8Ua0VNSrsZ83C4$
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > Hi EJ,
> > > > > > > > > > >
> > > > > > > > > > > You make good points about the metrics API
> extensibility
> > > and
> > > > > > > > evolution.
> > > > > > > > > > >
> > > > > > > > > > > However, we need to consider practical aspects too.
> Anand
> > > > > appears
> > > > > > > to
> > > > > > > > > have
> > > > > > > > > > > some specific use cases in mind, and I assume his
> > proposal
> > > > > > > addresses
> > > > > > > > > > them.
> > > > > > > > > > >
> > > > > > > > > > > Starting with an API + implementation that works for
> some
> > > > real
> > > > > > > > > > > world applications will validate the feature's
> usability.
> > > > > > > > > > >
> > > > > > > > > > > We can revamp the API completely in its v2 after v1 is
> > > > merged.
> > > > > > New
> > > > > > > > > major
> > > > > > > > > > > API versions do not have to be backward-compatible with
> > > older
> > > > > > > > versions
> > > > > > > > > of
> > > > > > > > > > > the same API [1].
> > > > > > > > > > >
> > > > > > > > > > > In my personal experience, a v1 API can hardly be
> > expected
> > > to
> > > > > > cover
> > > > > > > > all
> > > > > > > > > > use
> > > > > > > > > > > cases and extensions well. We can certainly take a bit
> > more
> > > > > time
> > > > > > to
> > > > > > > > > > polish
> > > > > > > > > > > it, but from my POV it might be best to iterate in
> terms
> > of
> > > > API
> > > > > > > > > versions
> > > > > > > > > > > rather than on unmerged commits in the initial
> proposal.
> > > Just
> > > > > my
> > > > > > 2
> > > > > > > > > cents
> > > > > > > > > > :)
> > > > > > > > > > >
> > > > > > > > > > > That said, we should flag the new APIs in this proposal
> > as
> > > > > > > "beta"...
> > > > > > > > at
> > > > > > > > > > > least initially (which is the usual practice in
> Polaris).
> > > > > > > > > > >
> > > > > > > > > > > > I wonder if it would help to evaluate the Events API
> > and
> > > > > > Metrics
> > > > > > > > API
> > > > > > > > > a
> > > > > > > > > > > bit more independently.
> > > > > > > > > > >
> > > > > > > > > > > That makes sense to me. However, the current proposal
> > > > > progressed
> > > > > > a
> > > > > > > > lot
> > > > > > > > > > > since its initial submission and contained both APIs. I
> > > would
> > > > > not
> > > > > > > > want
> > > > > > > > > to
> > > > > > > > > > > lose this momentum.
> > > > > > > > > > >
> > > > > > > > > > > It might still be advisable to implement the events and
> > > > metrics
> > > > > > > APIs
> > > > > > > > > > > separately and gather additional feedback at that time.
> > > > > > > > > > >
> > > > > > > > > > > [1]
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://urldefense.com/v3/__https://polaris.apache.org/in-dev/unreleased/evolution/__;!!Iz9xO38YGHZK!8KJ0uv4jK3mxZP4nYFrL1hZ0fMkQvoVEAJa8t9LBCzVtm_PWVFGQfIcZp-ykn3_F9_ph6EYyu3dUZjPAcQ$
> > > >> > > > > > >
> > > > > > > > > > > Cheers,
> > > > > > > > > > > Dmitri.
> > > > > > > > > > >
> > > > > > > > > > > On Mon, Mar 9, 2026 at 3:48 PM EJ Wang <
> > > > > > > > ej.wang.dev.interest@gmail.com
> > > > > > > > > >
> > > > > > > > > > > wrote:
> > > > > > > > > > >
> > > > > > > > > > > > Hi Anand,
> > > > > > > > > > > >
> > > > > > > > > > > > I think the proposal is moving in a better direction,
> > > > > > especially
> > > > > > > on
> > > > > > > > > the
> > > > > > > > > > > > Events side, and I appreciate the iteration so far.
> > That
> > > > > said,
> > > > > > I
> > > > > > > > > still
> > > > > > > > > > > have
> > > > > > > > > > > > some concerns about the Metrics side, but they are
> less
> > > > about
> > > > > > > > > > individual
> > > > > > > > > > > > parameters or endpoint shape, and more about product
> > > > > boundary.
> > > > > > > > > > > >
> > > > > > > > > > > > 2 cents: I wonder if it would help to evaluate the
> > Events
> > > > API
> > > > > > and
> > > > > > > > > > Metrics
> > > > > > > > > > > > API a bit more independently.
> > > > > > > > > > > >
> > > > > > > > > > > > The Events side feels relatively close to Polaris'
> > > > > > > > catalog/change-log
> > > > > > > > > > > > scope. It is easier to justify as part of the
> > > > core/community
> > > > > > > > surface,
> > > > > > > > > > > > especially if the goal is to expose completed catalog
> > > > > mutations
> > > > > > > in
> > > > > > > > a
> > > > > > > > > > way
> > > > > > > > > > > > that aligns with Iceberg-style events.
> > > > > > > > > > > >
> > > > > > > > > > > > The Metrics side feels different to me. Once we start
> > > > adding
> > > > > > more
> > > > > > > > and
> > > > > > > > > > > more
> > > > > > > > > > > > type-specific filters, query semantics, and schema
> > shape
> > > > for
> > > > > > > > > individual
> > > > > > > > > > > > metric families, it seems easy for Polaris to drift
> > > toward
> > > > a
> > > > > > > > built-in
> > > > > > > > > > > > observability backend. My bias would be for Polaris
> to
> > > > > support
> > > > > > a
> > > > > > > > > > smaller
> > > > > > > > > > > > set of community-recognized built-in metrics well,
> > while
> > > > > > > providing
> > > > > > > > > good
> > > > > > > > > > > > extensibility points for deployments that want richer
> > > > > querying,
> > > > > > > > > > > > visualization, or use-case-specific metrics.
> > > > > > > > > > > >
> > > > > > > > > > > > Related to that, I am not yet convinced the current
> > > metrics
> > > > > > model
> > > > > > > > is
> > > > > > > > > > > > generic enough as a long-term direction. Even after
> > > > > > consolidating
> > > > > > > > to
> > > > > > > > > a
> > > > > > > > > > > > single endpoint, the design still feels fairly tied
> to
> > > the
> > > > > > > current
> > > > > > > > > > > > scan/commit shape. I worry that otherwise each new
> > metric
> > > > > > family
> > > > > > > > will
> > > > > > > > > > > keep
> > > > > > > > > > > > pulling us into more storage/schema/API reshaping
> > inside
> > > > > > Polaris
> > > > > > > > > core.
> > > > > > > > > > > > So the framing question I would suggest is something
> > > like:
> > > > > > > > > > > > > What is the minimal built-in metrics surface
> Polaris
> > > > should
> > > > > > own
> > > > > > > > in
> > > > > > > > > > > core,
> > > > > > > > > > > > and where should we instead rely on extensibility /
> > > > > > sink-export /
> > > > > > > > > > > > plugin-style integration?
> > > > > > > > > > > >
> > > > > > > > > > > > For me, getting that boundary right matters more than
> > > > > settling
> > > > > > > > every
> > > > > > > > > > > query
> > > > > > > > > > > > parameter detail first.
> > > > > > > > > > > >
> > > > > > > > > > > > -ej
> > > > > > > > > > > >
> > > > > > > > > > > > On Tue, Mar 3, 2026 at 12:29 PM Anand Kumar Sankaran
> > via
> > > > dev
> > > > > <
> > > > > > > > > > > > dev@polaris.apache.org> wrote:
> > > > > > > > > > > >
> > > > > > > > > > > > > Hi Yufei and Dmitri,
> > > > > > > > > > > > >
> > > > > > > > > > > > > Here is a proposal for the REST endpoints for
> metrics
> > > and
> > > > > > > events.
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://urldefense.com/v3/__https://github.com/apache/polaris/pull/3924/changes__;!!Iz9xO38YGHZK!8KJ0uv4jK3mxZP4nYFrL1hZ0fMkQvoVEAJa8t9LBCzVtm_PWVFGQfIcZp-ykn3_F9_ph6EYyu3fXDjRYWA$
> > > >> > > > > > > > >
> > > > > > > > > > > > > I did not see any precursors for raising a PR for
> > > > > proposals,
> > > > > > so
> > > > > > > > > > trying
> > > > > > > > > > > > > this.  Please let me know what you think.
> > > > > > > > > > > > >
> > > > > > > > > > > > > -
> > > > > > > > > > > > > Anand
> > > > > > > > > > > > >
> > > > > > > > > > > > > From: Anand Kumar Sankaran <
> > anand.sankaran@workday.com
> > > >
> > > > > > > > > > > > > Date: Monday, March 2, 2026 at 10:25 AM
> > > > > > > > > > > > > To: dev@polaris.apache.org <dev@polaris.apache.org
> >
> > > > > > > > > > > > > Subject: Re: Polaris Telemetry and Audit Trail
> > > > > > > > > > > > >
> > > > > > > > > > > > > About the REST API, based on my use cases:
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > >   1.
> > > > > > > > > > > > > I want to be able to query commit metrics to track
> > > files
> > > > > > added
> > > > > > > /
> > > > > > > > > > > removed
> > > > > > > > > > > > > per commit, along with record counts. The ingestion
> > > > > pipeline
> > > > > > > that
> > > > > > > > > > > writes
> > > > > > > > > > > > > this data is owned by us and we are guaranteed to
> > write
> > > > > this
> > > > > > > > > > > information
> > > > > > > > > > > > > for each write.
> > > > > > > > > > > > >   2.
> > > > > > > > > > > > > I want to be able to query scan metrics for read. I
> > > > > > understand
> > > > > > > > > > clients
> > > > > > > > > > > do
> > > > > > > > > > > > > not fulfill this requirement.
> > > > > > > > > > > > >   3.
> > > > > > > > > > > > > I want to be able to query the events table (events
> > are
> > > > > > > > persisted)
> > > > > > > > > -
> > > > > > > > > > > this
> > > > > > > > > > > > > may supersede #2, I am not sure yet.
> > > > > > > > > > > > >
> > > > > > > > > > > > > All this information is in the JDBC based
> persistence
> > > > model
> > > > > > and
> > > > > > > > is
> > > > > > > > > > > > > persisted in the metastore. I currently don’t have
> a
> > > need
> > > > > to
> > > > > > > > query
> > > > > > > > > > > > > prometheus or open telemetry. I do publish some
> > events
> > > to
> > > > > > > > > Prometheus
> > > > > > > > > > > and
> > > > > > > > > > > > > they are forwarded to our dashboards elsewhere.
> > > > > > > > > > > > >
> > > > > > > > > > > > > About the CLI utilities, I meant the admin user
> > > > utilities.
> > > > > In
> > > > > > > one
> > > > > > > > > of
> > > > > > > > > > > the
> > > > > > > > > > > > > earliest drafts of my proposal, Prashant mentioned
> > that
> > > > the
> > > > > > > > metrics
> > > > > > > > > > > > tables
> > > > > > > > > > > > > can grow indefinitely and that a similar problem
> > exists
> > > > > with
> > > > > > > the
> > > > > > > > > > events
> > > > > > > > > > > > > table as well. We discussed that cleaning up of old
> > > > records
> > > > > > > from
> > > > > > > > > both
> > > > > > > > > > > > > metrics tables and events tables can be done via a
> > CLI
> > > > > > utility.
> > > > > > > > > > > > >
> > > > > > > > > > > > > I see that Yufei has covered the discussion about
> > > > > > datasources.
> > > > > > > > > > > > >
> > > > > > > > > > > > > -
> > > > > > > > > > > > > Anand
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > > From: Yufei Gu <fl...@gmail.com>
> > > > > > > > > > > > > Date: Friday, February 27, 2026 at 9:54 PM
> > > > > > > > > > > > > To: dev@polaris.apache.org <dev@polaris.apache.org
> >
> > > > > > > > > > > > > Subject: Re: Polaris Telemetry and Audit Trail
> > > > > > > > > > > > >
> > > > > > > > > > > > > This Message Is From an External Sender
> > > > > > > > > > > > > This message came from outside your organization.
> > > > > > > > > > > > > Report Suspicious<
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://us-phishalarm-ewt.proofpoint.com/EWT/v1/Iz9xO38YGHZK!YhNDZABkHi1B699ote2uMwpOZw8i0QMCGO2Szc-HshuABGhGvwPJcymE6G2oUUxtS8xDkSrtGTPm_I3QnVDHoLMk50m9v8z_nZKTkd-bnVUbreF1u0WnfV_X5eYevZl_$
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > > As I mentioned in
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://urldefense.com/v3/__https://github.com/apache/polaris/issues/3890__;!!Iz9xO38YGHZK!5EuyFFkk3vhRWVIRvQAWBSQfpJkTMA9HxugzDwXmN0LPPqhEFxYkFRGVhtb8AqUwXtDh2OplcMnbMDHKOxrvDU0$
> > > >> > > > > > > >,
> > > > > > > > > > > > > supporting
> > > > > > > > > > > > > multiple data sources is not a trivial change. I
> > would
> > > > > > strongly
> > > > > > > > > > > recommend
> > > > > > > > > > > > > starting with a design document to carefully
> evaluate
> > > the
> > > > > > > > > > architectural
> > > > > > > > > > > > > implications and long term impact.
> > > > > > > > > > > > >
> > > > > > > > > > > > > A REST endpoint to query metrics seems reasonable
> > given
> > > > the
> > > > > > > > current
> > > > > > > > > > > JDBC
> > > > > > > > > > > > > based persistence model. That said, we may also
> > > consider
> > > > > > > > > alternative
> > > > > > > > > > > > > storage models. For example, if we later adopt a
> time
> > > > > series
> > > > > > > > system
> > > > > > > > > > > such
> > > > > > > > > > > > as
> > > > > > > > > > > > > Prometheus to store metrics, the query model and
> > access
> > > > > > > patterns
> > > > > > > > > > would
> > > > > > > > > > > be
> > > > > > > > > > > > > fundamentally different. Designing the REST API
> > without
> > > > > > > > considering
> > > > > > > > > > > these
> > > > > > > > > > > > > potential evolutions may limit flexibility. I'd
> > suggest
> > > > to
> > > > > > > start
> > > > > > > > > with
> > > > > > > > > > > the
> > > > > > > > > > > > > use case.
> > > > > > > > > > > > >
> > > > > > > > > > > > > Yufei
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > > On Fri, Feb 27, 2026 at 3:42 PM Dmitri
> Bourlatchkov <
> > > > > > > > > > dimas@apache.org>
> > > > > > > > > > > > > wrote:
> > > > > > > > > > > > >
> > > > > > > > > > > > > > Hi Anand,
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Sharing my view... subject to discussion:
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > 1. Adding non-IRC REST API to Polaris is
> perfectly
> > > > fine.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Figuring out specific endpoint URIs and payloads
> > > might
> > > > > > > require
> > > > > > > > a
> > > > > > > > > > few
> > > > > > > > > > > > > > roundtrips, so opening a separate thread for that
> > > might
> > > > > be
> > > > > > > > best.
> > > > > > > > > > > > > > Contributors commonly create Google Docs for new
> > API
> > > > > > > proposals
> > > > > > > > > too
> > > > > > > > > > > > (they
> > > > > > > > > > > > > > fairly easy to update as the email discussion
> > > > > progresses).
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > There was a suggestion to try Markdown (with PRs)
> > for
> > > > > > > proposals
> > > > > > > > > [1]
> > > > > > > > > > > ...
> > > > > > > > > > > > > > feel free to give it a try if you are comfortable
> > > with
> > > > > > that.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > 2. Could you clarify whether you mean end user
> > > > utilities
> > > > > or
> > > > > > > > admin
> > > > > > > > > > > user
> > > > > > > > > > > > > > utilities? In the latter case those might be more
> > > > > suitable
> > > > > > > for
> > > > > > > > > the
> > > > > > > > > > > > Admin
> > > > > > > > > > > > > > CLI (java) not the Python CLI, IMHO.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Why would these utilities be common with events?
> > > IMHO,
> > > > > > event
> > > > > > > > use
> > > > > > > > > > > cases
> > > > > > > > > > > > > are
> > > > > > > > > > > > > > distinct from scan/commit metrics.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > 3. I'd prefer separating metrics persistence from
> > > > > MetaStore
> > > > > > > > > > > persistence
> > > > > > > > > > > > > at
> > > > > > > > > > > > > > the code level, so that they could be mixed and
> > > matched
> > > > > > > > > > > independently.
> > > > > > > > > > > > > The
> > > > > > > > > > > > > > separate datasource question will become a
> > non-issue
> > > > with
> > > > > > > that
> > > > > > > > > > > > approach,
> > > > > > > > > > > > > I
> > > > > > > > > > > > > > guess.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > The rationale for separating scan metrics and
> > > metastore
> > > > > > > > > persistence
> > > > > > > > > > > is
> > > > > > > > > > > > > that
> > > > > > > > > > > > > > "cascading deletes" between them are hardly ever
> > > > > required.
> > > > > > > > > > > Furthermore,
> > > > > > > > > > > > > the
> > > > > > > > > > > > > > data and query patterns are very different so
> > > different
> > > > > > > > > > technologies
> > > > > > > > > > > > > might
> > > > > > > > > > > > > > be beneficial in each case.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > [1]
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://urldefense.com/v3/__https://lists.apache.org/thread/yto2wp982t43h1mqjwnslswhws5z47cy__;!!Iz9xO38YGHZK!5EuyFFkk3vhRWVIRvQAWBSQfpJkTMA9HxugzDwXmN0LPPqhEFxYkFRGVhtb8AqUwXtDh2OplcMnbMDHKxYDakNU$
> > > >> > > > > > > >> >
> > > > > > > > > > > > > > Cheers,
> > > > > > > > > > > > > > Dmitri.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > On Fri, Feb 27, 2026 at 6:19 PM Anand Kumar
> > Sankaran
> > > > via
> > > > > > dev
> > > > > > > <
> > > > > > > > > > > > > > dev@polaris.apache.org> wrote:
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Thanks all. This PR is merged now.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Here are the follow-up features / work needed.
> > > These
> > > > > > were
> > > > > > > > all
> > > > > > > > > > part
> > > > > > > > > > > > of
> > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > merged PR at some point in time and were
> removed
> > to
> > > > > > reduce
> > > > > > > > > scope.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Please let me know what you think.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >   1.  A REST API to paginate through table
> > metrics.
> > > > > This
> > > > > > > will
> > > > > > > > > be
> > > > > > > > > > > > > non-IRC
> > > > > > > > > > > > > > > standard addition.
> > > > > > > > > > > > > > >   2.  Utilities for managing old records,
> should
> > be
> > > > > > common
> > > > > > > > with
> > > > > > > > > > > > events.
> > > > > > > > > > > > > > > There was some discussion that it belongs to
> the
> > > CLI.
> > > > > > > > > > > > > > >   3.  Separate datasource (metrics, events,
> even
> > > > other
> > > > > > > > > tables?).
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Anything else?
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > -
> > > > > > > > > > > > > > > Anand
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> > >
> >
>

Re: Proposal for REST endpoints for table metrics and events

[Dmitri Bourlatchkov <di...@apache.org>]

Hi All,

+1 to JB's points.

I also welcome Anand's contributions to Polaris and I believe they are
beneficial to the project and the larger community.

Supporting multiple APIs backed by the same data in storage is perfectly
normal. Most of the maintenance work happens at the layer that handles
actual data. I would not expect much overhead in adding Iceberg APIs (once
they are released) to the body of code currently proposed in Anand's PRs.

Cheers,
Dmitri.

On Fri, Apr 3, 2026 at 7:34 AM Jean-Baptiste Onofré <jb...@nanthrax.net> wrote:

> Hi all,
>
> I am concerned by this discussion from a contribution standpoint.
>
> Regarding the point about Iceberg synchronization, it is discouraging for a
> passionate and committed contributor to spend a month in discussion without
> a clear path forward. We have precedents for starting implementations in
> Polaris with the intention of syncing with the Iceberg community later.
>
> If we were certain that the Event API in Iceberg would be completed within
> a week or so, I would agree with the current plan. However, I do not
> believe that is the case. Projects evolve, and I see no issue with
> revisiting or updating current implementations in the future.
>
> I am in favor of moving forward with Anand's PR and creating an issue to
> track the Iceberg Event API update. I am sure Anand would be happy to
> rework the implementation once there is a clearer picture. To do otherwise
> is not welcoming to contributors.
>
> Regards,
> JB
>
> On Thu, Apr 2, 2026 at 11:56 PM Anand Kumar Sankaran via dev <
> dev@polaris.apache.org> wrote:
>
> > Hi all,
> >
> > I have an implementation of the metrics REST API available for review
> > https://github.com/apache/polaris/pull/4115. Unlike the events API,
> there
> > is no proposal yet in Iceberg community yet. I need to a way to read the
> > persisted table metrics and events as we are rolling out to production.
> >
> > Please look at this. If this is not acceptable and we prefer to wait for
> > the Iceberg community, I would have to implement them as proprietary
> > extensions to the REST API to move forward.
> >
> > —
> > Anand
> >
> > From: Yufei Gu <fl...@gmail.com>
> > Date: Monday, March 23, 2026 at 11:53 AM
> > To: dev@polaris.apache.org <de...@polaris.apache.org>
> > Subject: Re: Proposal for REST endpoints for table metrics and events
> >
> > This Message Is From an External Sender
> > This message came from outside your organization.
> > Report Suspicious<
> >
> https://us-phishalarm-ewt.proofpoint.com/EWT/v1/Iz9xO38YGHZK!YhNDZABkHi1B6_8r_oHOHjE1XzdKC66MNTZUq0rzz6aeuQJd2NnDmIlKwHgrUG7hWJidUrkh_sDqEZ3ZHa1HIzhjNw83YQe2WEa1Gn4LzkyEDLyXuTmvkoY9fnc8GtBQ$
> > >
> >
> >
> > I think the Events API in Iceberg is already quite close to finalization,
> > so I don’t see much value in introducing a separate unstable version in
> > Polaris ATM.
> >
> > For the metrics endpoint, I’m not aware of any ongoing discussion in the
> > Iceberg community. I’d suggest starting there first. If there isn’t
> > interest, Polaris can move forward and introduce a metrics endpoint
> > directly, without necessarily making it experimental. If there is
> interest,
> > it would make more sense to align with Iceberg and contribute there. We
> can
> > then decide whether we still need a fallback option in Polaris, such as
> an
> > experimental endpoint.
> >
> > Overall, it seems worth checking with the Iceberg community before moving
> > forward.
> >
> > Yufei
> >
> >
> > On Mon, Mar 23, 2026 at 11:33 AM Dmitri Bourlatchkov <di...@apache.org>
> > wrote:
> >
> > > Hi Nandor,
> > >
> > > Good point! The new metrics and events APIs should definitely be marked
> > as
> > > "experimental" or "beta" initially.
> > >
> > > That's a common practice for new APIs in Polaris.
> > >
> > > Cheers,
> > > Dmitri.
> > >
> > > On Thu, Mar 19, 2026 at 4:30 PM Nándor Kollár <nk...@apache.org>
> > wrote:
> > >
> > > > Hi All,
> > > >
> > > > I don’t see a problem with having two REST APIs for catalog events.
> We
> > > can
> > > > mark the local API as experimental (if there’s a way to do that in
> > > > Polaris), document that it isn’t stable, and deprecate it once the
> > > Iceberg
> > > > API is released. Alternatively, we could keep and continue improving
> it
> > > > with features that are missing from the Iceberg REST spec but are
> > > relevant
> > > > for Polaris.
> > > >
> > > > Nandor
> > > >
> > > > Dmitri Bourlatchkov <di...@apache.org> ezt írta (időpont: 2026.
> márc.
> > > 19.,
> > > > Cs, 17:44):
> > > >
> > > > > Hi All,
> > > > >
> > > > > Polaris can support both a local API and new endpoints in the IRC
> API
> > > > once
> > > > > the Iceberg community adopts the latter.
> > > > >
> > > > > What is the concern with having different APIs to access the same
> > data?
> > > > >
> > > > > Cheers,
> > > > > Dmitri.
> > > > >
> > > > > On Tue, Mar 17, 2026 at 3:49 PM EJ Wang <
> > > ej.wang.dev.interest@gmail.com>
> > > > > wrote:
> > > > >
> > > > > > I share the same feeling with Yifei as reviewing the PRs. I want
> to
> > > > avoid
> > > > > > creating discrepancies if the IRC side ended up supporting it.
> > > > > >
> > > > > > -ej
> > > > > >
> > > > > > On Wed, Mar 11, 2026 at 2:47 PM Yufei Gu <fl...@gmail.com>
> > > wrote:
> > > > > >
> > > > > > > Thanks Anand for working on this.
> > > > > > >
> > > > > > > Given the IRC event endpoint is WIP, I think it'd be best to be
> > > > > > consistent
> > > > > > > with the IRC endpoint. For more context on the IRC event, the
> > event
> > > > > > > endpoint is not finalized yet. I'd recommend speeding up the
> IRC
> > > side
> > > > > > work
> > > > > > > to avoid any inconsistencies between Polaris and IRC spec.
> > > > > > >
> > > > > > > I have a few questions about the metrics endpoint:
> > > > > > > 1. Do we need to expose them via Polaris REST endpoint? Can
> users
> > > > grab
> > > > > > the
> > > > > > > metrics from the backend directly? I understand the RBAC won't
> be
> > > > > there,
> > > > > > > but it provides flexibility for users. For example, some users
> > may
> > > > > > choose a
> > > > > > > different persistence model such as a KV store or storing the
> > > metrics
> > > > > as
> > > > > > > objects in S3, which usually scales better than an RDBMS like
> > > > Postgres.
> > > > > > My
> > > > > > > understanding is that Polaris is not intended to be a full
> > metrics
> > > > > system
> > > > > > > anyway, but rather to provide a way for downstream systems to
> > > consume
> > > > > > these
> > > > > > > data.
> > > > > > > 2. Should we consider it as an IRC endpoint? Given the metrics
> > > report
> > > > > > > endpoint in IRC, the Iceberg community might also be interested
> > in
> > > > > > serving
> > > > > > > back metrics, similar to the event endpoint. In that case,
> there
> > > is a
> > > > > > risk
> > > > > > > of fragmentation if we create a Polaris endpoint now. We should
> > > avoid
> > > > > > that
> > > > > > > if possible. It might be worth checking with the Iceberg
> > community
> > > > > first.
> > > > > > >
> > > > > > > Happy to hear others’ thoughts on this.
> > > > > > >
> > > > > > > Yufei
> > > > > > >
> > > > > > >
> > > > > > > On Wed, Mar 11, 2026 at 9:27 AM Nándor Kollár <
> > nkollar@apache.org>
> > > > > > wrote:
> > > > > > >
> > > > > > > > I agree with Dmitri. The direction outlined in the proposal
> > looks
> > > > > good
> > > > > > to
> > > > > > > > me, and the finer details can be worked out as implementation
> > > gets
> > > > > > > > underway. We can adjust the design doc accordingly, later on.
> > > > > > > >
> > > > > > > >
> > > > > > > > Thanks,
> > > > > > > > Nandor
> > > > > > > >
> > > > > > > > Dmitri Bourlatchkov <di...@apache.org> ezt írta (időpont:
> > 2026.
> > > > > márc.
> > > > > > > 10.,
> > > > > > > > K, 20:43):
> > > > > > > >
> > > > > > > > > Hi Anand and All,
> > > > > > > > >
> > > > > > > > > The proposal LGTM in its current form.
> > > > > > > > >
> > > > > > > > > My personal approach is that a proposal does not have to be
> > as
> > > > > > > "polished"
> > > > > > > > > as the final API spec. As long as we have consensus on the
> > > > general
> > > > > > > > > approach and the basic API principles, I think we can
> proceed
> > > to
> > > > > > > > > implementation and iron out final wrinkles during actual
> API
> > > spec
> > > > > and
> > > > > > > > code
> > > > > > > > > PRs.
> > > > > > > > >
> > > > > > > > > Would this approach work for everyone?
> > > > > > > > >
> > > > > > > > > Re-posting my comment from GH [1] here for visibility, in
> > case
> > > > > people
> > > > > > > > have
> > > > > > > > > different opinions and wish to discuss this in more depth:
> > > > > > > > >
> > > > > > > > > From my POV, Polaris is a platform, indeed. In this sense,
> I
> > > > think
> > > > > it
> > > > > > > is
> > > > > > > > > > critical to enable users to control what features are at
> > play
> > > > in
> > > > > > > > runtime,
> > > > > > > > > > since different users have different use cases. This is
> > why I
> > > > > > > > originally
> > > > > > > > > > advocated for isolating Metrics Persistence from
> MetaStore
> > > > > > > Persistence.
> > > > > > > > > >
> > > > > > > > > > If a user decides to leverage the "native" Polaris
> > > > (scan/commit)
> > > > > > > > Metrics
> > > > > > > > > > Persistence, I do not see any disadvantage in also
> exposing
> > > an
> > > > > > > > (optional)
> > > > > > > > > > REST API for loading these metrics from Polaris
> > Persistence.
> > > > > > > > > >
> > > > > > > > > > The degree of support and sophistication that goes into
> > this
> > > > > > > sub-system
> > > > > > > > > is
> > > > > > > > > > up to the community. If we have contributors (like
> > @obelix74
> > > )
> > > > > who
> > > > > > > are
> > > > > > > > > > willing to evolve it, I see no harm in some functionality
> > > > overlap
> > > > > > > with
> > > > > > > > > more
> > > > > > > > > > focused metrics platforms. Again, the key point is for
> all
> > > > users
> > > > > to
> > > > > > > > have
> > > > > > > > > > control and be able to opt in or out of this feature in
> > their
> > > > > > > specific
> > > > > > > > > > deployments.
> > > > > > > > > >
> > > > > > > > > > Of course, offloading scan/commit metrics storage to a
> > > > > specialized
> > > > > > > > > > observability system is possible too (assuming someone
> > > develops
> > > > > > > > > integration
> > > > > > > > > > code for that, which is very welcome).
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > [1]
> > > > > >
> >
> https://urldefense.com/v3/__https://github.com/apache/polaris/pull/3924*discussion_r2913947744__;Iw!!Iz9xO38YGHZK!5rBOI-ESxR2K-8_HPN195cT-mCOoNghT74Di8M2TIB97esSZ7-VrE50guaHsg7gd_9_HBIqfdgZ1NAhyQCoxjlo$
> > > > > > > > >
> > > > > > > > > Thanks,
> > > > > > > > > Dmitri.
> > > > > > > > >
> > > > > > > > > On Tue, Mar 10, 2026 at 12:37 PM Anand Kumar Sankaran via
> > dev <
> > > > > > > > > dev@polaris.apache.org> wrote:
> > > > > > > > >
> > > > > > > > > > Hi EJ Wang and Dmitri,
> > > > > > > > > >
> > > > > > > > > > I addressed all your concerns about the proposal, in
> > > particular
> > > > > > > > > >
> > > > >
> >
> https://urldefense.com/v3/__https://github.com/apache/polaris/pull/3924*discussion_r2908317696__;Iw!!Iz9xO38YGHZK!5rBOI-ESxR2K-8_HPN195cT-mCOoNghT74Di8M2TIB97esSZ7-VrE50guaHsg7gd_9_HBIqfdgZ1NAhyoAh9fLA$
> > > > > > .
> > > > > > > > > >
> > > > > > > > > > Does this address your concerns?
> > > > > > > > > >
> > > > > > > > > > -
> > > > > > > > > > Anand
> > > > > > > > > >
> > > > > > > > > > From: Dmitri Bourlatchkov <di...@apache.org>
> > > > > > > > > > Date: Monday, March 9, 2026 at 1:17 PM
> > > > > > > > > > To: dev@polaris.apache.org <de...@polaris.apache.org>
> > > > > > > > > > Subject: Re: Proposal for REST endpoints for table
> metrics
> > > and
> > > > > > events
> > > > > > > > > >
> > > > > > > > > > This Message Is From an External Sender
> > > > > > > > > > This message came from outside your organization.
> > > > > > > > > > Report Suspicious<
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://us-phishalarm-ewt.proofpoint.com/EWT/v1/Iz9xO38YGHZK!YhNDZAGr2cumYdtJ_UTm0gR9PfI_-PwSpR_GtNr1uVQ_xo-s2AskvUmbkLZ-C5V8eOKN-omus47On4k4hfFo-0G7CHMLwVjEego-rZrPuepAybX7DP8Ua0VNSrsZ83C4$
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > Hi EJ,
> > > > > > > > > >
> > > > > > > > > > You make good points about the metrics API extensibility
> > and
> > > > > > > evolution.
> > > > > > > > > >
> > > > > > > > > > However, we need to consider practical aspects too. Anand
> > > > appears
> > > > > > to
> > > > > > > > have
> > > > > > > > > > some specific use cases in mind, and I assume his
> proposal
> > > > > > addresses
> > > > > > > > > them.
> > > > > > > > > >
> > > > > > > > > > Starting with an API + implementation that works for some
> > > real
> > > > > > > > > > world applications will validate the feature's usability.
> > > > > > > > > >
> > > > > > > > > > We can revamp the API completely in its v2 after v1 is
> > > merged.
> > > > > New
> > > > > > > > major
> > > > > > > > > > API versions do not have to be backward-compatible with
> > older
> > > > > > > versions
> > > > > > > > of
> > > > > > > > > > the same API [1].
> > > > > > > > > >
> > > > > > > > > > In my personal experience, a v1 API can hardly be
> expected
> > to
> > > > > cover
> > > > > > > all
> > > > > > > > > use
> > > > > > > > > > cases and extensions well. We can certainly take a bit
> more
> > > > time
> > > > > to
> > > > > > > > > polish
> > > > > > > > > > it, but from my POV it might be best to iterate in terms
> of
> > > API
> > > > > > > > versions
> > > > > > > > > > rather than on unmerged commits in the initial proposal.
> > Just
> > > > my
> > > > > 2
> > > > > > > > cents
> > > > > > > > > :)
> > > > > > > > > >
> > > > > > > > > > That said, we should flag the new APIs in this proposal
> as
> > > > > > "beta"...
> > > > > > > at
> > > > > > > > > > least initially (which is the usual practice in Polaris).
> > > > > > > > > >
> > > > > > > > > > > I wonder if it would help to evaluate the Events API
> and
> > > > > Metrics
> > > > > > > API
> > > > > > > > a
> > > > > > > > > > bit more independently.
> > > > > > > > > >
> > > > > > > > > > That makes sense to me. However, the current proposal
> > > > progressed
> > > > > a
> > > > > > > lot
> > > > > > > > > > since its initial submission and contained both APIs. I
> > would
> > > > not
> > > > > > > want
> > > > > > > > to
> > > > > > > > > > lose this momentum.
> > > > > > > > > >
> > > > > > > > > > It might still be advisable to implement the events and
> > > metrics
> > > > > > APIs
> > > > > > > > > > separately and gather additional feedback at that time.
> > > > > > > > > >
> > > > > > > > > > [1]
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://urldefense.com/v3/__https://polaris.apache.org/in-dev/unreleased/evolution/__;!!Iz9xO38YGHZK!8KJ0uv4jK3mxZP4nYFrL1hZ0fMkQvoVEAJa8t9LBCzVtm_PWVFGQfIcZp-ykn3_F9_ph6EYyu3dUZjPAcQ$
> > >> > > > > > >
> > > > > > > > > > Cheers,
> > > > > > > > > > Dmitri.
> > > > > > > > > >
> > > > > > > > > > On Mon, Mar 9, 2026 at 3:48 PM EJ Wang <
> > > > > > > ej.wang.dev.interest@gmail.com
> > > > > > > > >
> > > > > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > > Hi Anand,
> > > > > > > > > > >
> > > > > > > > > > > I think the proposal is moving in a better direction,
> > > > > especially
> > > > > > on
> > > > > > > > the
> > > > > > > > > > > Events side, and I appreciate the iteration so far.
> That
> > > > said,
> > > > > I
> > > > > > > > still
> > > > > > > > > > have
> > > > > > > > > > > some concerns about the Metrics side, but they are less
> > > about
> > > > > > > > > individual
> > > > > > > > > > > parameters or endpoint shape, and more about product
> > > > boundary.
> > > > > > > > > > >
> > > > > > > > > > > 2 cents: I wonder if it would help to evaluate the
> Events
> > > API
> > > > > and
> > > > > > > > > Metrics
> > > > > > > > > > > API a bit more independently.
> > > > > > > > > > >
> > > > > > > > > > > The Events side feels relatively close to Polaris'
> > > > > > > catalog/change-log
> > > > > > > > > > > scope. It is easier to justify as part of the
> > > core/community
> > > > > > > surface,
> > > > > > > > > > > especially if the goal is to expose completed catalog
> > > > mutations
> > > > > > in
> > > > > > > a
> > > > > > > > > way
> > > > > > > > > > > that aligns with Iceberg-style events.
> > > > > > > > > > >
> > > > > > > > > > > The Metrics side feels different to me. Once we start
> > > adding
> > > > > more
> > > > > > > and
> > > > > > > > > > more
> > > > > > > > > > > type-specific filters, query semantics, and schema
> shape
> > > for
> > > > > > > > individual
> > > > > > > > > > > metric families, it seems easy for Polaris to drift
> > toward
> > > a
> > > > > > > built-in
> > > > > > > > > > > observability backend. My bias would be for Polaris to
> > > > support
> > > > > a
> > > > > > > > > smaller
> > > > > > > > > > > set of community-recognized built-in metrics well,
> while
> > > > > > providing
> > > > > > > > good
> > > > > > > > > > > extensibility points for deployments that want richer
> > > > querying,
> > > > > > > > > > > visualization, or use-case-specific metrics.
> > > > > > > > > > >
> > > > > > > > > > > Related to that, I am not yet convinced the current
> > metrics
> > > > > model
> > > > > > > is
> > > > > > > > > > > generic enough as a long-term direction. Even after
> > > > > consolidating
> > > > > > > to
> > > > > > > > a
> > > > > > > > > > > single endpoint, the design still feels fairly tied to
> > the
> > > > > > current
> > > > > > > > > > > scan/commit shape. I worry that otherwise each new
> metric
> > > > > family
> > > > > > > will
> > > > > > > > > > keep
> > > > > > > > > > > pulling us into more storage/schema/API reshaping
> inside
> > > > > Polaris
> > > > > > > > core.
> > > > > > > > > > > So the framing question I would suggest is something
> > like:
> > > > > > > > > > > > What is the minimal built-in metrics surface Polaris
> > > should
> > > > > own
> > > > > > > in
> > > > > > > > > > core,
> > > > > > > > > > > and where should we instead rely on extensibility /
> > > > > sink-export /
> > > > > > > > > > > plugin-style integration?
> > > > > > > > > > >
> > > > > > > > > > > For me, getting that boundary right matters more than
> > > > settling
> > > > > > > every
> > > > > > > > > > query
> > > > > > > > > > > parameter detail first.
> > > > > > > > > > >
> > > > > > > > > > > -ej
> > > > > > > > > > >
> > > > > > > > > > > On Tue, Mar 3, 2026 at 12:29 PM Anand Kumar Sankaran
> via
> > > dev
> > > > <
> > > > > > > > > > > dev@polaris.apache.org> wrote:
> > > > > > > > > > >
> > > > > > > > > > > > Hi Yufei and Dmitri,
> > > > > > > > > > > >
> > > > > > > > > > > > Here is a proposal for the REST endpoints for metrics
> > and
> > > > > > events.
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://urldefense.com/v3/__https://github.com/apache/polaris/pull/3924/changes__;!!Iz9xO38YGHZK!8KJ0uv4jK3mxZP4nYFrL1hZ0fMkQvoVEAJa8t9LBCzVtm_PWVFGQfIcZp-ykn3_F9_ph6EYyu3fXDjRYWA$
> > >> > > > > > > > >
> > > > > > > > > > > > I did not see any precursors for raising a PR for
> > > > proposals,
> > > > > so
> > > > > > > > > trying
> > > > > > > > > > > > this.  Please let me know what you think.
> > > > > > > > > > > >
> > > > > > > > > > > > -
> > > > > > > > > > > > Anand
> > > > > > > > > > > >
> > > > > > > > > > > > From: Anand Kumar Sankaran <
> anand.sankaran@workday.com
> > >
> > > > > > > > > > > > Date: Monday, March 2, 2026 at 10:25 AM
> > > > > > > > > > > > To: dev@polaris.apache.org <de...@polaris.apache.org>
> > > > > > > > > > > > Subject: Re: Polaris Telemetry and Audit Trail
> > > > > > > > > > > >
> > > > > > > > > > > > About the REST API, based on my use cases:
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > >   1.
> > > > > > > > > > > > I want to be able to query commit metrics to track
> > files
> > > > > added
> > > > > > /
> > > > > > > > > > removed
> > > > > > > > > > > > per commit, along with record counts. The ingestion
> > > > pipeline
> > > > > > that
> > > > > > > > > > writes
> > > > > > > > > > > > this data is owned by us and we are guaranteed to
> write
> > > > this
> > > > > > > > > > information
> > > > > > > > > > > > for each write.
> > > > > > > > > > > >   2.
> > > > > > > > > > > > I want to be able to query scan metrics for read. I
> > > > > understand
> > > > > > > > > clients
> > > > > > > > > > do
> > > > > > > > > > > > not fulfill this requirement.
> > > > > > > > > > > >   3.
> > > > > > > > > > > > I want to be able to query the events table (events
> are
> > > > > > > persisted)
> > > > > > > > -
> > > > > > > > > > this
> > > > > > > > > > > > may supersede #2, I am not sure yet.
> > > > > > > > > > > >
> > > > > > > > > > > > All this information is in the JDBC based persistence
> > > model
> > > > > and
> > > > > > > is
> > > > > > > > > > > > persisted in the metastore. I currently don’t have a
> > need
> > > > to
> > > > > > > query
> > > > > > > > > > > > prometheus or open telemetry. I do publish some
> events
> > to
> > > > > > > > Prometheus
> > > > > > > > > > and
> > > > > > > > > > > > they are forwarded to our dashboards elsewhere.
> > > > > > > > > > > >
> > > > > > > > > > > > About the CLI utilities, I meant the admin user
> > > utilities.
> > > > In
> > > > > > one
> > > > > > > > of
> > > > > > > > > > the
> > > > > > > > > > > > earliest drafts of my proposal, Prashant mentioned
> that
> > > the
> > > > > > > metrics
> > > > > > > > > > > tables
> > > > > > > > > > > > can grow indefinitely and that a similar problem
> exists
> > > > with
> > > > > > the
> > > > > > > > > events
> > > > > > > > > > > > table as well. We discussed that cleaning up of old
> > > records
> > > > > > from
> > > > > > > > both
> > > > > > > > > > > > metrics tables and events tables can be done via a
> CLI
> > > > > utility.
> > > > > > > > > > > >
> > > > > > > > > > > > I see that Yufei has covered the discussion about
> > > > > datasources.
> > > > > > > > > > > >
> > > > > > > > > > > > -
> > > > > > > > > > > > Anand
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > From: Yufei Gu <fl...@gmail.com>
> > > > > > > > > > > > Date: Friday, February 27, 2026 at 9:54 PM
> > > > > > > > > > > > To: dev@polaris.apache.org <de...@polaris.apache.org>
> > > > > > > > > > > > Subject: Re: Polaris Telemetry and Audit Trail
> > > > > > > > > > > >
> > > > > > > > > > > > This Message Is From an External Sender
> > > > > > > > > > > > This message came from outside your organization.
> > > > > > > > > > > > Report Suspicious<
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://us-phishalarm-ewt.proofpoint.com/EWT/v1/Iz9xO38YGHZK!YhNDZABkHi1B699ote2uMwpOZw8i0QMCGO2Szc-HshuABGhGvwPJcymE6G2oUUxtS8xDkSrtGTPm_I3QnVDHoLMk50m9v8z_nZKTkd-bnVUbreF1u0WnfV_X5eYevZl_$
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > As I mentioned in
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://urldefense.com/v3/__https://github.com/apache/polaris/issues/3890__;!!Iz9xO38YGHZK!5EuyFFkk3vhRWVIRvQAWBSQfpJkTMA9HxugzDwXmN0LPPqhEFxYkFRGVhtb8AqUwXtDh2OplcMnbMDHKOxrvDU0$
> > >> > > > > > > >,
> > > > > > > > > > > > supporting
> > > > > > > > > > > > multiple data sources is not a trivial change. I
> would
> > > > > strongly
> > > > > > > > > > recommend
> > > > > > > > > > > > starting with a design document to carefully evaluate
> > the
> > > > > > > > > architectural
> > > > > > > > > > > > implications and long term impact.
> > > > > > > > > > > >
> > > > > > > > > > > > A REST endpoint to query metrics seems reasonable
> given
> > > the
> > > > > > > current
> > > > > > > > > > JDBC
> > > > > > > > > > > > based persistence model. That said, we may also
> > consider
> > > > > > > > alternative
> > > > > > > > > > > > storage models. For example, if we later adopt a time
> > > > series
> > > > > > > system
> > > > > > > > > > such
> > > > > > > > > > > as
> > > > > > > > > > > > Prometheus to store metrics, the query model and
> access
> > > > > > patterns
> > > > > > > > > would
> > > > > > > > > > be
> > > > > > > > > > > > fundamentally different. Designing the REST API
> without
> > > > > > > considering
> > > > > > > > > > these
> > > > > > > > > > > > potential evolutions may limit flexibility. I'd
> suggest
> > > to
> > > > > > start
> > > > > > > > with
> > > > > > > > > > the
> > > > > > > > > > > > use case.
> > > > > > > > > > > >
> > > > > > > > > > > > Yufei
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > On Fri, Feb 27, 2026 at 3:42 PM Dmitri Bourlatchkov <
> > > > > > > > > dimas@apache.org>
> > > > > > > > > > > > wrote:
> > > > > > > > > > > >
> > > > > > > > > > > > > Hi Anand,
> > > > > > > > > > > > >
> > > > > > > > > > > > > Sharing my view... subject to discussion:
> > > > > > > > > > > > >
> > > > > > > > > > > > > 1. Adding non-IRC REST API to Polaris is perfectly
> > > fine.
> > > > > > > > > > > > >
> > > > > > > > > > > > > Figuring out specific endpoint URIs and payloads
> > might
> > > > > > require
> > > > > > > a
> > > > > > > > > few
> > > > > > > > > > > > > roundtrips, so opening a separate thread for that
> > might
> > > > be
> > > > > > > best.
> > > > > > > > > > > > > Contributors commonly create Google Docs for new
> API
> > > > > > proposals
> > > > > > > > too
> > > > > > > > > > > (they
> > > > > > > > > > > > > fairly easy to update as the email discussion
> > > > progresses).
> > > > > > > > > > > > >
> > > > > > > > > > > > > There was a suggestion to try Markdown (with PRs)
> for
> > > > > > proposals
> > > > > > > > [1]
> > > > > > > > > > ...
> > > > > > > > > > > > > feel free to give it a try if you are comfortable
> > with
> > > > > that.
> > > > > > > > > > > > >
> > > > > > > > > > > > > 2. Could you clarify whether you mean end user
> > > utilities
> > > > or
> > > > > > > admin
> > > > > > > > > > user
> > > > > > > > > > > > > utilities? In the latter case those might be more
> > > > suitable
> > > > > > for
> > > > > > > > the
> > > > > > > > > > > Admin
> > > > > > > > > > > > > CLI (java) not the Python CLI, IMHO.
> > > > > > > > > > > > >
> > > > > > > > > > > > > Why would these utilities be common with events?
> > IMHO,
> > > > > event
> > > > > > > use
> > > > > > > > > > cases
> > > > > > > > > > > > are
> > > > > > > > > > > > > distinct from scan/commit metrics.
> > > > > > > > > > > > >
> > > > > > > > > > > > > 3. I'd prefer separating metrics persistence from
> > > > MetaStore
> > > > > > > > > > persistence
> > > > > > > > > > > > at
> > > > > > > > > > > > > the code level, so that they could be mixed and
> > matched
> > > > > > > > > > independently.
> > > > > > > > > > > > The
> > > > > > > > > > > > > separate datasource question will become a
> non-issue
> > > with
> > > > > > that
> > > > > > > > > > > approach,
> > > > > > > > > > > > I
> > > > > > > > > > > > > guess.
> > > > > > > > > > > > >
> > > > > > > > > > > > > The rationale for separating scan metrics and
> > metastore
> > > > > > > > persistence
> > > > > > > > > > is
> > > > > > > > > > > > that
> > > > > > > > > > > > > "cascading deletes" between them are hardly ever
> > > > required.
> > > > > > > > > > Furthermore,
> > > > > > > > > > > > the
> > > > > > > > > > > > > data and query patterns are very different so
> > different
> > > > > > > > > technologies
> > > > > > > > > > > > might
> > > > > > > > > > > > > be beneficial in each case.
> > > > > > > > > > > > >
> > > > > > > > > > > > > [1]
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://urldefense.com/v3/__https://lists.apache.org/thread/yto2wp982t43h1mqjwnslswhws5z47cy__;!!Iz9xO38YGHZK!5EuyFFkk3vhRWVIRvQAWBSQfpJkTMA9HxugzDwXmN0LPPqhEFxYkFRGVhtb8AqUwXtDh2OplcMnbMDHKxYDakNU$
> > >> > > > > > > >> >
> > > > > > > > > > > > > Cheers,
> > > > > > > > > > > > > Dmitri.
> > > > > > > > > > > > >
> > > > > > > > > > > > > On Fri, Feb 27, 2026 at 6:19 PM Anand Kumar
> Sankaran
> > > via
> > > > > dev
> > > > > > <
> > > > > > > > > > > > > dev@polaris.apache.org> wrote:
> > > > > > > > > > > > >
> > > > > > > > > > > > > > Thanks all. This PR is merged now.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Here are the follow-up features / work needed.
> > These
> > > > > were
> > > > > > > all
> > > > > > > > > part
> > > > > > > > > > > of
> > > > > > > > > > > > > the
> > > > > > > > > > > > > > merged PR at some point in time and were removed
> to
> > > > > reduce
> > > > > > > > scope.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Please let me know what you think.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >   1.  A REST API to paginate through table
> metrics.
> > > > This
> > > > > > will
> > > > > > > > be
> > > > > > > > > > > > non-IRC
> > > > > > > > > > > > > > standard addition.
> > > > > > > > > > > > > >   2.  Utilities for managing old records, should
> be
> > > > > common
> > > > > > > with
> > > > > > > > > > > events.
> > > > > > > > > > > > > > There was some discussion that it belongs to the
> > CLI.
> > > > > > > > > > > > > >   3.  Separate datasource (metrics, events, even
> > > other
> > > > > > > > tables?).
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Anything else?
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > -
> > > > > > > > > > > > > > Anand
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> >
>

Re: Proposal for REST endpoints for table metrics and events

[Jean-Baptiste Onofré <jb...@nanthrax.net>]

Hi all,

I am concerned by this discussion from a contribution standpoint.

Regarding the point about Iceberg synchronization, it is discouraging for a
passionate and committed contributor to spend a month in discussion without
a clear path forward. We have precedents for starting implementations in
Polaris with the intention of syncing with the Iceberg community later.

If we were certain that the Event API in Iceberg would be completed within
a week or so, I would agree with the current plan. However, I do not
believe that is the case. Projects evolve, and I see no issue with
revisiting or updating current implementations in the future.

I am in favor of moving forward with Anand's PR and creating an issue to
track the Iceberg Event API update. I am sure Anand would be happy to
rework the implementation once there is a clearer picture. To do otherwise
is not welcoming to contributors.

Regards,
JB

On Thu, Apr 2, 2026 at 11:56 PM Anand Kumar Sankaran via dev <
dev@polaris.apache.org> wrote:

> Hi all,
>
> I have an implementation of the metrics REST API available for review
> https://github.com/apache/polaris/pull/4115. Unlike the events API, there
> is no proposal yet in Iceberg community yet. I need to a way to read the
> persisted table metrics and events as we are rolling out to production.
>
> Please look at this. If this is not acceptable and we prefer to wait for
> the Iceberg community, I would have to implement them as proprietary
> extensions to the REST API to move forward.
>
> —
> Anand
>
> From: Yufei Gu <fl...@gmail.com>
> Date: Monday, March 23, 2026 at 11:53 AM
> To: dev@polaris.apache.org <de...@polaris.apache.org>
> Subject: Re: Proposal for REST endpoints for table metrics and events
>
> This Message Is From an External Sender
> This message came from outside your organization.
> Report Suspicious<
> https://us-phishalarm-ewt.proofpoint.com/EWT/v1/Iz9xO38YGHZK!YhNDZABkHi1B6_8r_oHOHjE1XzdKC66MNTZUq0rzz6aeuQJd2NnDmIlKwHgrUG7hWJidUrkh_sDqEZ3ZHa1HIzhjNw83YQe2WEa1Gn4LzkyEDLyXuTmvkoY9fnc8GtBQ$
> >
>
>
> I think the Events API in Iceberg is already quite close to finalization,
> so I don’t see much value in introducing a separate unstable version in
> Polaris ATM.
>
> For the metrics endpoint, I’m not aware of any ongoing discussion in the
> Iceberg community. I’d suggest starting there first. If there isn’t
> interest, Polaris can move forward and introduce a metrics endpoint
> directly, without necessarily making it experimental. If there is interest,
> it would make more sense to align with Iceberg and contribute there. We can
> then decide whether we still need a fallback option in Polaris, such as an
> experimental endpoint.
>
> Overall, it seems worth checking with the Iceberg community before moving
> forward.
>
> Yufei
>
>
> On Mon, Mar 23, 2026 at 11:33 AM Dmitri Bourlatchkov <di...@apache.org>
> wrote:
>
> > Hi Nandor,
> >
> > Good point! The new metrics and events APIs should definitely be marked
> as
> > "experimental" or "beta" initially.
> >
> > That's a common practice for new APIs in Polaris.
> >
> > Cheers,
> > Dmitri.
> >
> > On Thu, Mar 19, 2026 at 4:30 PM Nándor Kollár <nk...@apache.org>
> wrote:
> >
> > > Hi All,
> > >
> > > I don’t see a problem with having two REST APIs for catalog events. We
> > can
> > > mark the local API as experimental (if there’s a way to do that in
> > > Polaris), document that it isn’t stable, and deprecate it once the
> > Iceberg
> > > API is released. Alternatively, we could keep and continue improving it
> > > with features that are missing from the Iceberg REST spec but are
> > relevant
> > > for Polaris.
> > >
> > > Nandor
> > >
> > > Dmitri Bourlatchkov <di...@apache.org> ezt írta (időpont: 2026. márc.
> > 19.,
> > > Cs, 17:44):
> > >
> > > > Hi All,
> > > >
> > > > Polaris can support both a local API and new endpoints in the IRC API
> > > once
> > > > the Iceberg community adopts the latter.
> > > >
> > > > What is the concern with having different APIs to access the same
> data?
> > > >
> > > > Cheers,
> > > > Dmitri.
> > > >
> > > > On Tue, Mar 17, 2026 at 3:49 PM EJ Wang <
> > ej.wang.dev.interest@gmail.com>
> > > > wrote:
> > > >
> > > > > I share the same feeling with Yifei as reviewing the PRs. I want to
> > > avoid
> > > > > creating discrepancies if the IRC side ended up supporting it.
> > > > >
> > > > > -ej
> > > > >
> > > > > On Wed, Mar 11, 2026 at 2:47 PM Yufei Gu <fl...@gmail.com>
> > wrote:
> > > > >
> > > > > > Thanks Anand for working on this.
> > > > > >
> > > > > > Given the IRC event endpoint is WIP, I think it'd be best to be
> > > > > consistent
> > > > > > with the IRC endpoint. For more context on the IRC event, the
> event
> > > > > > endpoint is not finalized yet. I'd recommend speeding up the IRC
> > side
> > > > > work
> > > > > > to avoid any inconsistencies between Polaris and IRC spec.
> > > > > >
> > > > > > I have a few questions about the metrics endpoint:
> > > > > > 1. Do we need to expose them via Polaris REST endpoint? Can users
> > > grab
> > > > > the
> > > > > > metrics from the backend directly? I understand the RBAC won't be
> > > > there,
> > > > > > but it provides flexibility for users. For example, some users
> may
> > > > > choose a
> > > > > > different persistence model such as a KV store or storing the
> > metrics
> > > > as
> > > > > > objects in S3, which usually scales better than an RDBMS like
> > > Postgres.
> > > > > My
> > > > > > understanding is that Polaris is not intended to be a full
> metrics
> > > > system
> > > > > > anyway, but rather to provide a way for downstream systems to
> > consume
> > > > > these
> > > > > > data.
> > > > > > 2. Should we consider it as an IRC endpoint? Given the metrics
> > report
> > > > > > endpoint in IRC, the Iceberg community might also be interested
> in
> > > > > serving
> > > > > > back metrics, similar to the event endpoint. In that case, there
> > is a
> > > > > risk
> > > > > > of fragmentation if we create a Polaris endpoint now. We should
> > avoid
> > > > > that
> > > > > > if possible. It might be worth checking with the Iceberg
> community
> > > > first.
> > > > > >
> > > > > > Happy to hear others’ thoughts on this.
> > > > > >
> > > > > > Yufei
> > > > > >
> > > > > >
> > > > > > On Wed, Mar 11, 2026 at 9:27 AM Nándor Kollár <
> nkollar@apache.org>
> > > > > wrote:
> > > > > >
> > > > > > > I agree with Dmitri. The direction outlined in the proposal
> looks
> > > > good
> > > > > to
> > > > > > > me, and the finer details can be worked out as implementation
> > gets
> > > > > > > underway. We can adjust the design doc accordingly, later on.
> > > > > > >
> > > > > > >
> > > > > > > Thanks,
> > > > > > > Nandor
> > > > > > >
> > > > > > > Dmitri Bourlatchkov <di...@apache.org> ezt írta (időpont:
> 2026.
> > > > márc.
> > > > > > 10.,
> > > > > > > K, 20:43):
> > > > > > >
> > > > > > > > Hi Anand and All,
> > > > > > > >
> > > > > > > > The proposal LGTM in its current form.
> > > > > > > >
> > > > > > > > My personal approach is that a proposal does not have to be
> as
> > > > > > "polished"
> > > > > > > > as the final API spec. As long as we have consensus on the
> > > general
> > > > > > > > approach and the basic API principles, I think we can proceed
> > to
> > > > > > > > implementation and iron out final wrinkles during actual API
> > spec
> > > > and
> > > > > > > code
> > > > > > > > PRs.
> > > > > > > >
> > > > > > > > Would this approach work for everyone?
> > > > > > > >
> > > > > > > > Re-posting my comment from GH [1] here for visibility, in
> case
> > > > people
> > > > > > > have
> > > > > > > > different opinions and wish to discuss this in more depth:
> > > > > > > >
> > > > > > > > From my POV, Polaris is a platform, indeed. In this sense, I
> > > think
> > > > it
> > > > > > is
> > > > > > > > > critical to enable users to control what features are at
> play
> > > in
> > > > > > > runtime,
> > > > > > > > > since different users have different use cases. This is
> why I
> > > > > > > originally
> > > > > > > > > advocated for isolating Metrics Persistence from MetaStore
> > > > > > Persistence.
> > > > > > > > >
> > > > > > > > > If a user decides to leverage the "native" Polaris
> > > (scan/commit)
> > > > > > > Metrics
> > > > > > > > > Persistence, I do not see any disadvantage in also exposing
> > an
> > > > > > > (optional)
> > > > > > > > > REST API for loading these metrics from Polaris
> Persistence.
> > > > > > > > >
> > > > > > > > > The degree of support and sophistication that goes into
> this
> > > > > > sub-system
> > > > > > > > is
> > > > > > > > > up to the community. If we have contributors (like
> @obelix74
> > )
> > > > who
> > > > > > are
> > > > > > > > > willing to evolve it, I see no harm in some functionality
> > > overlap
> > > > > > with
> > > > > > > > more
> > > > > > > > > focused metrics platforms. Again, the key point is for all
> > > users
> > > > to
> > > > > > > have
> > > > > > > > > control and be able to opt in or out of this feature in
> their
> > > > > > specific
> > > > > > > > > deployments.
> > > > > > > > >
> > > > > > > > > Of course, offloading scan/commit metrics storage to a
> > > > specialized
> > > > > > > > > observability system is possible too (assuming someone
> > develops
> > > > > > > > integration
> > > > > > > > > code for that, which is very welcome).
> > > > > > > >
> > > > > > > >
> > > > > > > > [1]
> > > > >
> https://urldefense.com/v3/__https://github.com/apache/polaris/pull/3924*discussion_r2913947744__;Iw!!Iz9xO38YGHZK!5rBOI-ESxR2K-8_HPN195cT-mCOoNghT74Di8M2TIB97esSZ7-VrE50guaHsg7gd_9_HBIqfdgZ1NAhyQCoxjlo$
> > > > > > > >
> > > > > > > > Thanks,
> > > > > > > > Dmitri.
> > > > > > > >
> > > > > > > > On Tue, Mar 10, 2026 at 12:37 PM Anand Kumar Sankaran via
> dev <
> > > > > > > > dev@polaris.apache.org> wrote:
> > > > > > > >
> > > > > > > > > Hi EJ Wang and Dmitri,
> > > > > > > > >
> > > > > > > > > I addressed all your concerns about the proposal, in
> > particular
> > > > > > > > >
> > > >
> https://urldefense.com/v3/__https://github.com/apache/polaris/pull/3924*discussion_r2908317696__;Iw!!Iz9xO38YGHZK!5rBOI-ESxR2K-8_HPN195cT-mCOoNghT74Di8M2TIB97esSZ7-VrE50guaHsg7gd_9_HBIqfdgZ1NAhyoAh9fLA$
> > > > > .
> > > > > > > > >
> > > > > > > > > Does this address your concerns?
> > > > > > > > >
> > > > > > > > > -
> > > > > > > > > Anand
> > > > > > > > >
> > > > > > > > > From: Dmitri Bourlatchkov <di...@apache.org>
> > > > > > > > > Date: Monday, March 9, 2026 at 1:17 PM
> > > > > > > > > To: dev@polaris.apache.org <de...@polaris.apache.org>
> > > > > > > > > Subject: Re: Proposal for REST endpoints for table metrics
> > and
> > > > > events
> > > > > > > > >
> > > > > > > > > This Message Is From an External Sender
> > > > > > > > > This message came from outside your organization.
> > > > > > > > > Report Suspicious<
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://us-phishalarm-ewt.proofpoint.com/EWT/v1/Iz9xO38YGHZK!YhNDZAGr2cumYdtJ_UTm0gR9PfI_-PwSpR_GtNr1uVQ_xo-s2AskvUmbkLZ-C5V8eOKN-omus47On4k4hfFo-0G7CHMLwVjEego-rZrPuepAybX7DP8Ua0VNSrsZ83C4$
> > > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > Hi EJ,
> > > > > > > > >
> > > > > > > > > You make good points about the metrics API extensibility
> and
> > > > > > evolution.
> > > > > > > > >
> > > > > > > > > However, we need to consider practical aspects too. Anand
> > > appears
> > > > > to
> > > > > > > have
> > > > > > > > > some specific use cases in mind, and I assume his proposal
> > > > > addresses
> > > > > > > > them.
> > > > > > > > >
> > > > > > > > > Starting with an API + implementation that works for some
> > real
> > > > > > > > > world applications will validate the feature's usability.
> > > > > > > > >
> > > > > > > > > We can revamp the API completely in its v2 after v1 is
> > merged.
> > > > New
> > > > > > > major
> > > > > > > > > API versions do not have to be backward-compatible with
> older
> > > > > > versions
> > > > > > > of
> > > > > > > > > the same API [1].
> > > > > > > > >
> > > > > > > > > In my personal experience, a v1 API can hardly be expected
> to
> > > > cover
> > > > > > all
> > > > > > > > use
> > > > > > > > > cases and extensions well. We can certainly take a bit more
> > > time
> > > > to
> > > > > > > > polish
> > > > > > > > > it, but from my POV it might be best to iterate in terms of
> > API
> > > > > > > versions
> > > > > > > > > rather than on unmerged commits in the initial proposal.
> Just
> > > my
> > > > 2
> > > > > > > cents
> > > > > > > > :)
> > > > > > > > >
> > > > > > > > > That said, we should flag the new APIs in this proposal as
> > > > > "beta"...
> > > > > > at
> > > > > > > > > least initially (which is the usual practice in Polaris).
> > > > > > > > >
> > > > > > > > > > I wonder if it would help to evaluate the Events API and
> > > > Metrics
> > > > > > API
> > > > > > > a
> > > > > > > > > bit more independently.
> > > > > > > > >
> > > > > > > > > That makes sense to me. However, the current proposal
> > > progressed
> > > > a
> > > > > > lot
> > > > > > > > > since its initial submission and contained both APIs. I
> would
> > > not
> > > > > > want
> > > > > > > to
> > > > > > > > > lose this momentum.
> > > > > > > > >
> > > > > > > > > It might still be advisable to implement the events and
> > metrics
> > > > > APIs
> > > > > > > > > separately and gather additional feedback at that time.
> > > > > > > > >
> > > > > > > > > [1]
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://urldefense.com/v3/__https://polaris.apache.org/in-dev/unreleased/evolution/__;!!Iz9xO38YGHZK!8KJ0uv4jK3mxZP4nYFrL1hZ0fMkQvoVEAJa8t9LBCzVtm_PWVFGQfIcZp-ykn3_F9_ph6EYyu3dUZjPAcQ$
> >> > > > > > >
> > > > > > > > > Cheers,
> > > > > > > > > Dmitri.
> > > > > > > > >
> > > > > > > > > On Mon, Mar 9, 2026 at 3:48 PM EJ Wang <
> > > > > > ej.wang.dev.interest@gmail.com
> > > > > > > >
> > > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > Hi Anand,
> > > > > > > > > >
> > > > > > > > > > I think the proposal is moving in a better direction,
> > > > especially
> > > > > on
> > > > > > > the
> > > > > > > > > > Events side, and I appreciate the iteration so far. That
> > > said,
> > > > I
> > > > > > > still
> > > > > > > > > have
> > > > > > > > > > some concerns about the Metrics side, but they are less
> > about
> > > > > > > > individual
> > > > > > > > > > parameters or endpoint shape, and more about product
> > > boundary.
> > > > > > > > > >
> > > > > > > > > > 2 cents: I wonder if it would help to evaluate the Events
> > API
> > > > and
> > > > > > > > Metrics
> > > > > > > > > > API a bit more independently.
> > > > > > > > > >
> > > > > > > > > > The Events side feels relatively close to Polaris'
> > > > > > catalog/change-log
> > > > > > > > > > scope. It is easier to justify as part of the
> > core/community
> > > > > > surface,
> > > > > > > > > > especially if the goal is to expose completed catalog
> > > mutations
> > > > > in
> > > > > > a
> > > > > > > > way
> > > > > > > > > > that aligns with Iceberg-style events.
> > > > > > > > > >
> > > > > > > > > > The Metrics side feels different to me. Once we start
> > adding
> > > > more
> > > > > > and
> > > > > > > > > more
> > > > > > > > > > type-specific filters, query semantics, and schema shape
> > for
> > > > > > > individual
> > > > > > > > > > metric families, it seems easy for Polaris to drift
> toward
> > a
> > > > > > built-in
> > > > > > > > > > observability backend. My bias would be for Polaris to
> > > support
> > > > a
> > > > > > > > smaller
> > > > > > > > > > set of community-recognized built-in metrics well, while
> > > > > providing
> > > > > > > good
> > > > > > > > > > extensibility points for deployments that want richer
> > > querying,
> > > > > > > > > > visualization, or use-case-specific metrics.
> > > > > > > > > >
> > > > > > > > > > Related to that, I am not yet convinced the current
> metrics
> > > > model
> > > > > > is
> > > > > > > > > > generic enough as a long-term direction. Even after
> > > > consolidating
> > > > > > to
> > > > > > > a
> > > > > > > > > > single endpoint, the design still feels fairly tied to
> the
> > > > > current
> > > > > > > > > > scan/commit shape. I worry that otherwise each new metric
> > > > family
> > > > > > will
> > > > > > > > > keep
> > > > > > > > > > pulling us into more storage/schema/API reshaping inside
> > > > Polaris
> > > > > > > core.
> > > > > > > > > > So the framing question I would suggest is something
> like:
> > > > > > > > > > > What is the minimal built-in metrics surface Polaris
> > should
> > > > own
> > > > > > in
> > > > > > > > > core,
> > > > > > > > > > and where should we instead rely on extensibility /
> > > > sink-export /
> > > > > > > > > > plugin-style integration?
> > > > > > > > > >
> > > > > > > > > > For me, getting that boundary right matters more than
> > > settling
> > > > > > every
> > > > > > > > > query
> > > > > > > > > > parameter detail first.
> > > > > > > > > >
> > > > > > > > > > -ej
> > > > > > > > > >
> > > > > > > > > > On Tue, Mar 3, 2026 at 12:29 PM Anand Kumar Sankaran via
> > dev
> > > <
> > > > > > > > > > dev@polaris.apache.org> wrote:
> > > > > > > > > >
> > > > > > > > > > > Hi Yufei and Dmitri,
> > > > > > > > > > >
> > > > > > > > > > > Here is a proposal for the REST endpoints for metrics
> and
> > > > > events.
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://urldefense.com/v3/__https://github.com/apache/polaris/pull/3924/changes__;!!Iz9xO38YGHZK!8KJ0uv4jK3mxZP4nYFrL1hZ0fMkQvoVEAJa8t9LBCzVtm_PWVFGQfIcZp-ykn3_F9_ph6EYyu3fXDjRYWA$
> >> > > > > > > > >
> > > > > > > > > > > I did not see any precursors for raising a PR for
> > > proposals,
> > > > so
> > > > > > > > trying
> > > > > > > > > > > this.  Please let me know what you think.
> > > > > > > > > > >
> > > > > > > > > > > -
> > > > > > > > > > > Anand
> > > > > > > > > > >
> > > > > > > > > > > From: Anand Kumar Sankaran <anand.sankaran@workday.com
> >
> > > > > > > > > > > Date: Monday, March 2, 2026 at 10:25 AM
> > > > > > > > > > > To: dev@polaris.apache.org <de...@polaris.apache.org>
> > > > > > > > > > > Subject: Re: Polaris Telemetry and Audit Trail
> > > > > > > > > > >
> > > > > > > > > > > About the REST API, based on my use cases:
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > >   1.
> > > > > > > > > > > I want to be able to query commit metrics to track
> files
> > > > added
> > > > > /
> > > > > > > > > removed
> > > > > > > > > > > per commit, along with record counts. The ingestion
> > > pipeline
> > > > > that
> > > > > > > > > writes
> > > > > > > > > > > this data is owned by us and we are guaranteed to write
> > > this
> > > > > > > > > information
> > > > > > > > > > > for each write.
> > > > > > > > > > >   2.
> > > > > > > > > > > I want to be able to query scan metrics for read. I
> > > > understand
> > > > > > > > clients
> > > > > > > > > do
> > > > > > > > > > > not fulfill this requirement.
> > > > > > > > > > >   3.
> > > > > > > > > > > I want to be able to query the events table (events are
> > > > > > persisted)
> > > > > > > -
> > > > > > > > > this
> > > > > > > > > > > may supersede #2, I am not sure yet.
> > > > > > > > > > >
> > > > > > > > > > > All this information is in the JDBC based persistence
> > model
> > > > and
> > > > > > is
> > > > > > > > > > > persisted in the metastore. I currently don’t have a
> need
> > > to
> > > > > > query
> > > > > > > > > > > prometheus or open telemetry. I do publish some events
> to
> > > > > > > Prometheus
> > > > > > > > > and
> > > > > > > > > > > they are forwarded to our dashboards elsewhere.
> > > > > > > > > > >
> > > > > > > > > > > About the CLI utilities, I meant the admin user
> > utilities.
> > > In
> > > > > one
> > > > > > > of
> > > > > > > > > the
> > > > > > > > > > > earliest drafts of my proposal, Prashant mentioned that
> > the
> > > > > > metrics
> > > > > > > > > > tables
> > > > > > > > > > > can grow indefinitely and that a similar problem exists
> > > with
> > > > > the
> > > > > > > > events
> > > > > > > > > > > table as well. We discussed that cleaning up of old
> > records
> > > > > from
> > > > > > > both
> > > > > > > > > > > metrics tables and events tables can be done via a CLI
> > > > utility.
> > > > > > > > > > >
> > > > > > > > > > > I see that Yufei has covered the discussion about
> > > > datasources.
> > > > > > > > > > >
> > > > > > > > > > > -
> > > > > > > > > > > Anand
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > From: Yufei Gu <fl...@gmail.com>
> > > > > > > > > > > Date: Friday, February 27, 2026 at 9:54 PM
> > > > > > > > > > > To: dev@polaris.apache.org <de...@polaris.apache.org>
> > > > > > > > > > > Subject: Re: Polaris Telemetry and Audit Trail
> > > > > > > > > > >
> > > > > > > > > > > This Message Is From an External Sender
> > > > > > > > > > > This message came from outside your organization.
> > > > > > > > > > > Report Suspicious<
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://us-phishalarm-ewt.proofpoint.com/EWT/v1/Iz9xO38YGHZK!YhNDZABkHi1B699ote2uMwpOZw8i0QMCGO2Szc-HshuABGhGvwPJcymE6G2oUUxtS8xDkSrtGTPm_I3QnVDHoLMk50m9v8z_nZKTkd-bnVUbreF1u0WnfV_X5eYevZl_$
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > As I mentioned in
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://urldefense.com/v3/__https://github.com/apache/polaris/issues/3890__;!!Iz9xO38YGHZK!5EuyFFkk3vhRWVIRvQAWBSQfpJkTMA9HxugzDwXmN0LPPqhEFxYkFRGVhtb8AqUwXtDh2OplcMnbMDHKOxrvDU0$
> >> > > > > > > >,
> > > > > > > > > > > supporting
> > > > > > > > > > > multiple data sources is not a trivial change. I would
> > > > strongly
> > > > > > > > > recommend
> > > > > > > > > > > starting with a design document to carefully evaluate
> the
> > > > > > > > architectural
> > > > > > > > > > > implications and long term impact.
> > > > > > > > > > >
> > > > > > > > > > > A REST endpoint to query metrics seems reasonable given
> > the
> > > > > > current
> > > > > > > > > JDBC
> > > > > > > > > > > based persistence model. That said, we may also
> consider
> > > > > > > alternative
> > > > > > > > > > > storage models. For example, if we later adopt a time
> > > series
> > > > > > system
> > > > > > > > > such
> > > > > > > > > > as
> > > > > > > > > > > Prometheus to store metrics, the query model and access
> > > > > patterns
> > > > > > > > would
> > > > > > > > > be
> > > > > > > > > > > fundamentally different. Designing the REST API without
> > > > > > considering
> > > > > > > > > these
> > > > > > > > > > > potential evolutions may limit flexibility. I'd suggest
> > to
> > > > > start
> > > > > > > with
> > > > > > > > > the
> > > > > > > > > > > use case.
> > > > > > > > > > >
> > > > > > > > > > > Yufei
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > On Fri, Feb 27, 2026 at 3:42 PM Dmitri Bourlatchkov <
> > > > > > > > dimas@apache.org>
> > > > > > > > > > > wrote:
> > > > > > > > > > >
> > > > > > > > > > > > Hi Anand,
> > > > > > > > > > > >
> > > > > > > > > > > > Sharing my view... subject to discussion:
> > > > > > > > > > > >
> > > > > > > > > > > > 1. Adding non-IRC REST API to Polaris is perfectly
> > fine.
> > > > > > > > > > > >
> > > > > > > > > > > > Figuring out specific endpoint URIs and payloads
> might
> > > > > require
> > > > > > a
> > > > > > > > few
> > > > > > > > > > > > roundtrips, so opening a separate thread for that
> might
> > > be
> > > > > > best.
> > > > > > > > > > > > Contributors commonly create Google Docs for new API
> > > > > proposals
> > > > > > > too
> > > > > > > > > > (they
> > > > > > > > > > > > fairly easy to update as the email discussion
> > > progresses).
> > > > > > > > > > > >
> > > > > > > > > > > > There was a suggestion to try Markdown (with PRs) for
> > > > > proposals
> > > > > > > [1]
> > > > > > > > > ...
> > > > > > > > > > > > feel free to give it a try if you are comfortable
> with
> > > > that.
> > > > > > > > > > > >
> > > > > > > > > > > > 2. Could you clarify whether you mean end user
> > utilities
> > > or
> > > > > > admin
> > > > > > > > > user
> > > > > > > > > > > > utilities? In the latter case those might be more
> > > suitable
> > > > > for
> > > > > > > the
> > > > > > > > > > Admin
> > > > > > > > > > > > CLI (java) not the Python CLI, IMHO.
> > > > > > > > > > > >
> > > > > > > > > > > > Why would these utilities be common with events?
> IMHO,
> > > > event
> > > > > > use
> > > > > > > > > cases
> > > > > > > > > > > are
> > > > > > > > > > > > distinct from scan/commit metrics.
> > > > > > > > > > > >
> > > > > > > > > > > > 3. I'd prefer separating metrics persistence from
> > > MetaStore
> > > > > > > > > persistence
> > > > > > > > > > > at
> > > > > > > > > > > > the code level, so that they could be mixed and
> matched
> > > > > > > > > independently.
> > > > > > > > > > > The
> > > > > > > > > > > > separate datasource question will become a non-issue
> > with
> > > > > that
> > > > > > > > > > approach,
> > > > > > > > > > > I
> > > > > > > > > > > > guess.
> > > > > > > > > > > >
> > > > > > > > > > > > The rationale for separating scan metrics and
> metastore
> > > > > > > persistence
> > > > > > > > > is
> > > > > > > > > > > that
> > > > > > > > > > > > "cascading deletes" between them are hardly ever
> > > required.
> > > > > > > > > Furthermore,
> > > > > > > > > > > the
> > > > > > > > > > > > data and query patterns are very different so
> different
> > > > > > > > technologies
> > > > > > > > > > > might
> > > > > > > > > > > > be beneficial in each case.
> > > > > > > > > > > >
> > > > > > > > > > > > [1]
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://urldefense.com/v3/__https://lists.apache.org/thread/yto2wp982t43h1mqjwnslswhws5z47cy__;!!Iz9xO38YGHZK!5EuyFFkk3vhRWVIRvQAWBSQfpJkTMA9HxugzDwXmN0LPPqhEFxYkFRGVhtb8AqUwXtDh2OplcMnbMDHKxYDakNU$
> >> > > > > > > >> >
> > > > > > > > > > > > Cheers,
> > > > > > > > > > > > Dmitri.
> > > > > > > > > > > >
> > > > > > > > > > > > On Fri, Feb 27, 2026 at 6:19 PM Anand Kumar Sankaran
> > via
> > > > dev
> > > > > <
> > > > > > > > > > > > dev@polaris.apache.org> wrote:
> > > > > > > > > > > >
> > > > > > > > > > > > > Thanks all. This PR is merged now.
> > > > > > > > > > > > >
> > > > > > > > > > > > > Here are the follow-up features / work needed.
> These
> > > > were
> > > > > > all
> > > > > > > > part
> > > > > > > > > > of
> > > > > > > > > > > > the
> > > > > > > > > > > > > merged PR at some point in time and were removed to
> > > > reduce
> > > > > > > scope.
> > > > > > > > > > > > >
> > > > > > > > > > > > > Please let me know what you think.
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > >   1.  A REST API to paginate through table metrics.
> > > This
> > > > > will
> > > > > > > be
> > > > > > > > > > > non-IRC
> > > > > > > > > > > > > standard addition.
> > > > > > > > > > > > >   2.  Utilities for managing old records, should be
> > > > common
> > > > > > with
> > > > > > > > > > events.
> > > > > > > > > > > > > There was some discussion that it belongs to the
> CLI.
> > > > > > > > > > > > >   3.  Separate datasource (metrics, events, even
> > other
> > > > > > > tables?).
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > > Anything else?
> > > > > > > > > > > > >
> > > > > > > > > > > > > -
> > > > > > > > > > > > > Anand
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
>

Re: Proposal for REST endpoints for table metrics and events

[Anand Kumar Sankaran via dev <de...@polaris.apache.org>]

Hi all,

I have an implementation of the metrics REST API available for review https://github.com/apache/polaris/pull/4115. Unlike the events API, there is no proposal yet in Iceberg community yet. I need to a way to read the persisted table metrics and events as we are rolling out to production.

Please look at this. If this is not acceptable and we prefer to wait for the Iceberg community, I would have to implement them as proprietary extensions to the REST API to move forward.

—
Anand

From: Yufei Gu <fl...@gmail.com>
Date: Monday, March 23, 2026 at 11:53 AM
To: dev@polaris.apache.org <de...@polaris.apache.org>
Subject: Re: Proposal for REST endpoints for table metrics and events

This Message Is From an External Sender
This message came from outside your organization.
Report Suspicious<https://us-phishalarm-ewt.proofpoint.com/EWT/v1/Iz9xO38YGHZK!YhNDZABkHi1B6_8r_oHOHjE1XzdKC66MNTZUq0rzz6aeuQJd2NnDmIlKwHgrUG7hWJidUrkh_sDqEZ3ZHa1HIzhjNw83YQe2WEa1Gn4LzkyEDLyXuTmvkoY9fnc8GtBQ$>


I think the Events API in Iceberg is already quite close to finalization,
so I don’t see much value in introducing a separate unstable version in
Polaris ATM.

For the metrics endpoint, I’m not aware of any ongoing discussion in the
Iceberg community. I’d suggest starting there first. If there isn’t
interest, Polaris can move forward and introduce a metrics endpoint
directly, without necessarily making it experimental. If there is interest,
it would make more sense to align with Iceberg and contribute there. We can
then decide whether we still need a fallback option in Polaris, such as an
experimental endpoint.

Overall, it seems worth checking with the Iceberg community before moving
forward.

Yufei


On Mon, Mar 23, 2026 at 11:33 AM Dmitri Bourlatchkov <di...@apache.org>
wrote:

> Hi Nandor,
>
> Good point! The new metrics and events APIs should definitely be marked as
> "experimental" or "beta" initially.
>
> That's a common practice for new APIs in Polaris.
>
> Cheers,
> Dmitri.
>
> On Thu, Mar 19, 2026 at 4:30 PM Nándor Kollár <nk...@apache.org> wrote:
>
> > Hi All,
> >
> > I don’t see a problem with having two REST APIs for catalog events. We
> can
> > mark the local API as experimental (if there’s a way to do that in
> > Polaris), document that it isn’t stable, and deprecate it once the
> Iceberg
> > API is released. Alternatively, we could keep and continue improving it
> > with features that are missing from the Iceberg REST spec but are
> relevant
> > for Polaris.
> >
> > Nandor
> >
> > Dmitri Bourlatchkov <di...@apache.org> ezt írta (időpont: 2026. márc.
> 19.,
> > Cs, 17:44):
> >
> > > Hi All,
> > >
> > > Polaris can support both a local API and new endpoints in the IRC API
> > once
> > > the Iceberg community adopts the latter.
> > >
> > > What is the concern with having different APIs to access the same data?
> > >
> > > Cheers,
> > > Dmitri.
> > >
> > > On Tue, Mar 17, 2026 at 3:49 PM EJ Wang <
> ej.wang.dev.interest@gmail.com>
> > > wrote:
> > >
> > > > I share the same feeling with Yifei as reviewing the PRs. I want to
> > avoid
> > > > creating discrepancies if the IRC side ended up supporting it.
> > > >
> > > > -ej
> > > >
> > > > On Wed, Mar 11, 2026 at 2:47 PM Yufei Gu <fl...@gmail.com>
> wrote:
> > > >
> > > > > Thanks Anand for working on this.
> > > > >
> > > > > Given the IRC event endpoint is WIP, I think it'd be best to be
> > > > consistent
> > > > > with the IRC endpoint. For more context on the IRC event, the event
> > > > > endpoint is not finalized yet. I'd recommend speeding up the IRC
> side
> > > > work
> > > > > to avoid any inconsistencies between Polaris and IRC spec.
> > > > >
> > > > > I have a few questions about the metrics endpoint:
> > > > > 1. Do we need to expose them via Polaris REST endpoint? Can users
> > grab
> > > > the
> > > > > metrics from the backend directly? I understand the RBAC won't be
> > > there,
> > > > > but it provides flexibility for users. For example, some users may
> > > > choose a
> > > > > different persistence model such as a KV store or storing the
> metrics
> > > as
> > > > > objects in S3, which usually scales better than an RDBMS like
> > Postgres.
> > > > My
> > > > > understanding is that Polaris is not intended to be a full metrics
> > > system
> > > > > anyway, but rather to provide a way for downstream systems to
> consume
> > > > these
> > > > > data.
> > > > > 2. Should we consider it as an IRC endpoint? Given the metrics
> report
> > > > > endpoint in IRC, the Iceberg community might also be interested in
> > > > serving
> > > > > back metrics, similar to the event endpoint. In that case, there
> is a
> > > > risk
> > > > > of fragmentation if we create a Polaris endpoint now. We should
> avoid
> > > > that
> > > > > if possible. It might be worth checking with the Iceberg community
> > > first.
> > > > >
> > > > > Happy to hear others’ thoughts on this.
> > > > >
> > > > > Yufei
> > > > >
> > > > >
> > > > > On Wed, Mar 11, 2026 at 9:27 AM Nándor Kollár <nk...@apache.org>
> > > > wrote:
> > > > >
> > > > > > I agree with Dmitri. The direction outlined in the proposal looks
> > > good
> > > > to
> > > > > > me, and the finer details can be worked out as implementation
> gets
> > > > > > underway. We can adjust the design doc accordingly, later on.
> > > > > >
> > > > > >
> > > > > > Thanks,
> > > > > > Nandor
> > > > > >
> > > > > > Dmitri Bourlatchkov <di...@apache.org> ezt írta (időpont: 2026.
> > > márc.
> > > > > 10.,
> > > > > > K, 20:43):
> > > > > >
> > > > > > > Hi Anand and All,
> > > > > > >
> > > > > > > The proposal LGTM in its current form.
> > > > > > >
> > > > > > > My personal approach is that a proposal does not have to be as
> > > > > "polished"
> > > > > > > as the final API spec. As long as we have consensus on the
> > general
> > > > > > > approach and the basic API principles, I think we can proceed
> to
> > > > > > > implementation and iron out final wrinkles during actual API
> spec
> > > and
> > > > > > code
> > > > > > > PRs.
> > > > > > >
> > > > > > > Would this approach work for everyone?
> > > > > > >
> > > > > > > Re-posting my comment from GH [1] here for visibility, in case
> > > people
> > > > > > have
> > > > > > > different opinions and wish to discuss this in more depth:
> > > > > > >
> > > > > > > From my POV, Polaris is a platform, indeed. In this sense, I
> > think
> > > it
> > > > > is
> > > > > > > > critical to enable users to control what features are at play
> > in
> > > > > > runtime,
> > > > > > > > since different users have different use cases. This is why I
> > > > > > originally
> > > > > > > > advocated for isolating Metrics Persistence from MetaStore
> > > > > Persistence.
> > > > > > > >
> > > > > > > > If a user decides to leverage the "native" Polaris
> > (scan/commit)
> > > > > > Metrics
> > > > > > > > Persistence, I do not see any disadvantage in also exposing
> an
> > > > > > (optional)
> > > > > > > > REST API for loading these metrics from Polaris Persistence.
> > > > > > > >
> > > > > > > > The degree of support and sophistication that goes into this
> > > > > sub-system
> > > > > > > is
> > > > > > > > up to the community. If we have contributors (like @obelix74
> )
> > > who
> > > > > are
> > > > > > > > willing to evolve it, I see no harm in some functionality
> > overlap
> > > > > with
> > > > > > > more
> > > > > > > > focused metrics platforms. Again, the key point is for all
> > users
> > > to
> > > > > > have
> > > > > > > > control and be able to opt in or out of this feature in their
> > > > > specific
> > > > > > > > deployments.
> > > > > > > >
> > > > > > > > Of course, offloading scan/commit metrics storage to a
> > > specialized
> > > > > > > > observability system is possible too (assuming someone
> develops
> > > > > > > integration
> > > > > > > > code for that, which is very welcome).
> > > > > > >
> > > > > > >
> > > > > > > [1]
> > > > https://urldefense.com/v3/__https://github.com/apache/polaris/pull/3924*discussion_r2913947744__;Iw!!Iz9xO38YGHZK!5rBOI-ESxR2K-8_HPN195cT-mCOoNghT74Di8M2TIB97esSZ7-VrE50guaHsg7gd_9_HBIqfdgZ1NAhyQCoxjlo$
> > > > > > >
> > > > > > > Thanks,
> > > > > > > Dmitri.
> > > > > > >
> > > > > > > On Tue, Mar 10, 2026 at 12:37 PM Anand Kumar Sankaran via dev <
> > > > > > > dev@polaris.apache.org> wrote:
> > > > > > >
> > > > > > > > Hi EJ Wang and Dmitri,
> > > > > > > >
> > > > > > > > I addressed all your concerns about the proposal, in
> particular
> > > > > > > >
> > > https://urldefense.com/v3/__https://github.com/apache/polaris/pull/3924*discussion_r2908317696__;Iw!!Iz9xO38YGHZK!5rBOI-ESxR2K-8_HPN195cT-mCOoNghT74Di8M2TIB97esSZ7-VrE50guaHsg7gd_9_HBIqfdgZ1NAhyoAh9fLA$
> > > > .
> > > > > > > >
> > > > > > > > Does this address your concerns?
> > > > > > > >
> > > > > > > > -
> > > > > > > > Anand
> > > > > > > >
> > > > > > > > From: Dmitri Bourlatchkov <di...@apache.org>
> > > > > > > > Date: Monday, March 9, 2026 at 1:17 PM
> > > > > > > > To: dev@polaris.apache.org <de...@polaris.apache.org>
> > > > > > > > Subject: Re: Proposal for REST endpoints for table metrics
> and
> > > > events
> > > > > > > >
> > > > > > > > This Message Is From an External Sender
> > > > > > > > This message came from outside your organization.
> > > > > > > > Report Suspicious<
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://us-phishalarm-ewt.proofpoint.com/EWT/v1/Iz9xO38YGHZK!YhNDZAGr2cumYdtJ_UTm0gR9PfI_-PwSpR_GtNr1uVQ_xo-s2AskvUmbkLZ-C5V8eOKN-omus47On4k4hfFo-0G7CHMLwVjEego-rZrPuepAybX7DP8Ua0VNSrsZ83C4$
> > > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > Hi EJ,
> > > > > > > >
> > > > > > > > You make good points about the metrics API extensibility and
> > > > > evolution.
> > > > > > > >
> > > > > > > > However, we need to consider practical aspects too. Anand
> > appears
> > > > to
> > > > > > have
> > > > > > > > some specific use cases in mind, and I assume his proposal
> > > > addresses
> > > > > > > them.
> > > > > > > >
> > > > > > > > Starting with an API + implementation that works for some
> real
> > > > > > > > world applications will validate the feature's usability.
> > > > > > > >
> > > > > > > > We can revamp the API completely in its v2 after v1 is
> merged.
> > > New
> > > > > > major
> > > > > > > > API versions do not have to be backward-compatible with older
> > > > > versions
> > > > > > of
> > > > > > > > the same API [1].
> > > > > > > >
> > > > > > > > In my personal experience, a v1 API can hardly be expected to
> > > cover
> > > > > all
> > > > > > > use
> > > > > > > > cases and extensions well. We can certainly take a bit more
> > time
> > > to
> > > > > > > polish
> > > > > > > > it, but from my POV it might be best to iterate in terms of
> API
> > > > > > versions
> > > > > > > > rather than on unmerged commits in the initial proposal. Just
> > my
> > > 2
> > > > > > cents
> > > > > > > :)
> > > > > > > >
> > > > > > > > That said, we should flag the new APIs in this proposal as
> > > > "beta"...
> > > > > at
> > > > > > > > least initially (which is the usual practice in Polaris).
> > > > > > > >
> > > > > > > > > I wonder if it would help to evaluate the Events API and
> > > Metrics
> > > > > API
> > > > > > a
> > > > > > > > bit more independently.
> > > > > > > >
> > > > > > > > That makes sense to me. However, the current proposal
> > progressed
> > > a
> > > > > lot
> > > > > > > > since its initial submission and contained both APIs. I would
> > not
> > > > > want
> > > > > > to
> > > > > > > > lose this momentum.
> > > > > > > >
> > > > > > > > It might still be advisable to implement the events and
> metrics
> > > > APIs
> > > > > > > > separately and gather additional feedback at that time.
> > > > > > > >
> > > > > > > > [1]
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://urldefense.com/v3/__https://polaris.apache.org/in-dev/unreleased/evolution/__;!!Iz9xO38YGHZK!8KJ0uv4jK3mxZP4nYFrL1hZ0fMkQvoVEAJa8t9LBCzVtm_PWVFGQfIcZp-ykn3_F9_ph6EYyu3dUZjPAcQ$
>> > > > > > >
> > > > > > > > Cheers,
> > > > > > > > Dmitri.
> > > > > > > >
> > > > > > > > On Mon, Mar 9, 2026 at 3:48 PM EJ Wang <
> > > > > ej.wang.dev.interest@gmail.com
> > > > > > >
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > > > Hi Anand,
> > > > > > > > >
> > > > > > > > > I think the proposal is moving in a better direction,
> > > especially
> > > > on
> > > > > > the
> > > > > > > > > Events side, and I appreciate the iteration so far. That
> > said,
> > > I
> > > > > > still
> > > > > > > > have
> > > > > > > > > some concerns about the Metrics side, but they are less
> about
> > > > > > > individual
> > > > > > > > > parameters or endpoint shape, and more about product
> > boundary.
> > > > > > > > >
> > > > > > > > > 2 cents: I wonder if it would help to evaluate the Events
> API
> > > and
> > > > > > > Metrics
> > > > > > > > > API a bit more independently.
> > > > > > > > >
> > > > > > > > > The Events side feels relatively close to Polaris'
> > > > > catalog/change-log
> > > > > > > > > scope. It is easier to justify as part of the
> core/community
> > > > > surface,
> > > > > > > > > especially if the goal is to expose completed catalog
> > mutations
> > > > in
> > > > > a
> > > > > > > way
> > > > > > > > > that aligns with Iceberg-style events.
> > > > > > > > >
> > > > > > > > > The Metrics side feels different to me. Once we start
> adding
> > > more
> > > > > and
> > > > > > > > more
> > > > > > > > > type-specific filters, query semantics, and schema shape
> for
> > > > > > individual
> > > > > > > > > metric families, it seems easy for Polaris to drift toward
> a
> > > > > built-in
> > > > > > > > > observability backend. My bias would be for Polaris to
> > support
> > > a
> > > > > > > smaller
> > > > > > > > > set of community-recognized built-in metrics well, while
> > > > providing
> > > > > > good
> > > > > > > > > extensibility points for deployments that want richer
> > querying,
> > > > > > > > > visualization, or use-case-specific metrics.
> > > > > > > > >
> > > > > > > > > Related to that, I am not yet convinced the current metrics
> > > model
> > > > > is
> > > > > > > > > generic enough as a long-term direction. Even after
> > > consolidating
> > > > > to
> > > > > > a
> > > > > > > > > single endpoint, the design still feels fairly tied to the
> > > > current
> > > > > > > > > scan/commit shape. I worry that otherwise each new metric
> > > family
> > > > > will
> > > > > > > > keep
> > > > > > > > > pulling us into more storage/schema/API reshaping inside
> > > Polaris
> > > > > > core.
> > > > > > > > > So the framing question I would suggest is something like:
> > > > > > > > > > What is the minimal built-in metrics surface Polaris
> should
> > > own
> > > > > in
> > > > > > > > core,
> > > > > > > > > and where should we instead rely on extensibility /
> > > sink-export /
> > > > > > > > > plugin-style integration?
> > > > > > > > >
> > > > > > > > > For me, getting that boundary right matters more than
> > settling
> > > > > every
> > > > > > > > query
> > > > > > > > > parameter detail first.
> > > > > > > > >
> > > > > > > > > -ej
> > > > > > > > >
> > > > > > > > > On Tue, Mar 3, 2026 at 12:29 PM Anand Kumar Sankaran via
> dev
> > <
> > > > > > > > > dev@polaris.apache.org> wrote:
> > > > > > > > >
> > > > > > > > > > Hi Yufei and Dmitri,
> > > > > > > > > >
> > > > > > > > > > Here is a proposal for the REST endpoints for metrics and
> > > > events.
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://urldefense.com/v3/__https://github.com/apache/polaris/pull/3924/changes__;!!Iz9xO38YGHZK!8KJ0uv4jK3mxZP4nYFrL1hZ0fMkQvoVEAJa8t9LBCzVtm_PWVFGQfIcZp-ykn3_F9_ph6EYyu3fXDjRYWA$
>> > > > > > > > >
> > > > > > > > > > I did not see any precursors for raising a PR for
> > proposals,
> > > so
> > > > > > > trying
> > > > > > > > > > this.  Please let me know what you think.
> > > > > > > > > >
> > > > > > > > > > -
> > > > > > > > > > Anand
> > > > > > > > > >
> > > > > > > > > > From: Anand Kumar Sankaran <an...@workday.com>
> > > > > > > > > > Date: Monday, March 2, 2026 at 10:25 AM
> > > > > > > > > > To: dev@polaris.apache.org <de...@polaris.apache.org>
> > > > > > > > > > Subject: Re: Polaris Telemetry and Audit Trail
> > > > > > > > > >
> > > > > > > > > > About the REST API, based on my use cases:
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >   1.
> > > > > > > > > > I want to be able to query commit metrics to track files
> > > added
> > > > /
> > > > > > > > removed
> > > > > > > > > > per commit, along with record counts. The ingestion
> > pipeline
> > > > that
> > > > > > > > writes
> > > > > > > > > > this data is owned by us and we are guaranteed to write
> > this
> > > > > > > > information
> > > > > > > > > > for each write.
> > > > > > > > > >   2.
> > > > > > > > > > I want to be able to query scan metrics for read. I
> > > understand
> > > > > > > clients
> > > > > > > > do
> > > > > > > > > > not fulfill this requirement.
> > > > > > > > > >   3.
> > > > > > > > > > I want to be able to query the events table (events are
> > > > > persisted)
> > > > > > -
> > > > > > > > this
> > > > > > > > > > may supersede #2, I am not sure yet.
> > > > > > > > > >
> > > > > > > > > > All this information is in the JDBC based persistence
> model
> > > and
> > > > > is
> > > > > > > > > > persisted in the metastore. I currently don’t have a need
> > to
> > > > > query
> > > > > > > > > > prometheus or open telemetry. I do publish some events to
> > > > > > Prometheus
> > > > > > > > and
> > > > > > > > > > they are forwarded to our dashboards elsewhere.
> > > > > > > > > >
> > > > > > > > > > About the CLI utilities, I meant the admin user
> utilities.
> > In
> > > > one
> > > > > > of
> > > > > > > > the
> > > > > > > > > > earliest drafts of my proposal, Prashant mentioned that
> the
> > > > > metrics
> > > > > > > > > tables
> > > > > > > > > > can grow indefinitely and that a similar problem exists
> > with
> > > > the
> > > > > > > events
> > > > > > > > > > table as well. We discussed that cleaning up of old
> records
> > > > from
> > > > > > both
> > > > > > > > > > metrics tables and events tables can be done via a CLI
> > > utility.
> > > > > > > > > >
> > > > > > > > > > I see that Yufei has covered the discussion about
> > > datasources.
> > > > > > > > > >
> > > > > > > > > > -
> > > > > > > > > > Anand
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > From: Yufei Gu <fl...@gmail.com>
> > > > > > > > > > Date: Friday, February 27, 2026 at 9:54 PM
> > > > > > > > > > To: dev@polaris.apache.org <de...@polaris.apache.org>
> > > > > > > > > > Subject: Re: Polaris Telemetry and Audit Trail
> > > > > > > > > >
> > > > > > > > > > This Message Is From an External Sender
> > > > > > > > > > This message came from outside your organization.
> > > > > > > > > > Report Suspicious<
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://us-phishalarm-ewt.proofpoint.com/EWT/v1/Iz9xO38YGHZK!YhNDZABkHi1B699ote2uMwpOZw8i0QMCGO2Szc-HshuABGhGvwPJcymE6G2oUUxtS8xDkSrtGTPm_I3QnVDHoLMk50m9v8z_nZKTkd-bnVUbreF1u0WnfV_X5eYevZl_$
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > As I mentioned in
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://urldefense.com/v3/__https://github.com/apache/polaris/issues/3890__;!!Iz9xO38YGHZK!5EuyFFkk3vhRWVIRvQAWBSQfpJkTMA9HxugzDwXmN0LPPqhEFxYkFRGVhtb8AqUwXtDh2OplcMnbMDHKOxrvDU0$
>> > > > > > > >,
> > > > > > > > > > supporting
> > > > > > > > > > multiple data sources is not a trivial change. I would
> > > strongly
> > > > > > > > recommend
> > > > > > > > > > starting with a design document to carefully evaluate the
> > > > > > > architectural
> > > > > > > > > > implications and long term impact.
> > > > > > > > > >
> > > > > > > > > > A REST endpoint to query metrics seems reasonable given
> the
> > > > > current
> > > > > > > > JDBC
> > > > > > > > > > based persistence model. That said, we may also consider
> > > > > > alternative
> > > > > > > > > > storage models. For example, if we later adopt a time
> > series
> > > > > system
> > > > > > > > such
> > > > > > > > > as
> > > > > > > > > > Prometheus to store metrics, the query model and access
> > > > patterns
> > > > > > > would
> > > > > > > > be
> > > > > > > > > > fundamentally different. Designing the REST API without
> > > > > considering
> > > > > > > > these
> > > > > > > > > > potential evolutions may limit flexibility. I'd suggest
> to
> > > > start
> > > > > > with
> > > > > > > > the
> > > > > > > > > > use case.
> > > > > > > > > >
> > > > > > > > > > Yufei
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > On Fri, Feb 27, 2026 at 3:42 PM Dmitri Bourlatchkov <
> > > > > > > dimas@apache.org>
> > > > > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > > Hi Anand,
> > > > > > > > > > >
> > > > > > > > > > > Sharing my view... subject to discussion:
> > > > > > > > > > >
> > > > > > > > > > > 1. Adding non-IRC REST API to Polaris is perfectly
> fine.
> > > > > > > > > > >
> > > > > > > > > > > Figuring out specific endpoint URIs and payloads might
> > > > require
> > > > > a
> > > > > > > few
> > > > > > > > > > > roundtrips, so opening a separate thread for that might
> > be
> > > > > best.
> > > > > > > > > > > Contributors commonly create Google Docs for new API
> > > > proposals
> > > > > > too
> > > > > > > > > (they
> > > > > > > > > > > fairly easy to update as the email discussion
> > progresses).
> > > > > > > > > > >
> > > > > > > > > > > There was a suggestion to try Markdown (with PRs) for
> > > > proposals
> > > > > > [1]
> > > > > > > > ...
> > > > > > > > > > > feel free to give it a try if you are comfortable with
> > > that.
> > > > > > > > > > >
> > > > > > > > > > > 2. Could you clarify whether you mean end user
> utilities
> > or
> > > > > admin
> > > > > > > > user
> > > > > > > > > > > utilities? In the latter case those might be more
> > suitable
> > > > for
> > > > > > the
> > > > > > > > > Admin
> > > > > > > > > > > CLI (java) not the Python CLI, IMHO.
> > > > > > > > > > >
> > > > > > > > > > > Why would these utilities be common with events? IMHO,
> > > event
> > > > > use
> > > > > > > > cases
> > > > > > > > > > are
> > > > > > > > > > > distinct from scan/commit metrics.
> > > > > > > > > > >
> > > > > > > > > > > 3. I'd prefer separating metrics persistence from
> > MetaStore
> > > > > > > > persistence
> > > > > > > > > > at
> > > > > > > > > > > the code level, so that they could be mixed and matched
> > > > > > > > independently.
> > > > > > > > > > The
> > > > > > > > > > > separate datasource question will become a non-issue
> with
> > > > that
> > > > > > > > > approach,
> > > > > > > > > > I
> > > > > > > > > > > guess.
> > > > > > > > > > >
> > > > > > > > > > > The rationale for separating scan metrics and metastore
> > > > > > persistence
> > > > > > > > is
> > > > > > > > > > that
> > > > > > > > > > > "cascading deletes" between them are hardly ever
> > required.
> > > > > > > > Furthermore,
> > > > > > > > > > the
> > > > > > > > > > > data and query patterns are very different so different
> > > > > > > technologies
> > > > > > > > > > might
> > > > > > > > > > > be beneficial in each case.
> > > > > > > > > > >
> > > > > > > > > > > [1]
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://urldefense.com/v3/__https://lists.apache.org/thread/yto2wp982t43h1mqjwnslswhws5z47cy__;!!Iz9xO38YGHZK!5EuyFFkk3vhRWVIRvQAWBSQfpJkTMA9HxugzDwXmN0LPPqhEFxYkFRGVhtb8AqUwXtDh2OplcMnbMDHKxYDakNU$
>> > > > > > > >> >
> > > > > > > > > > > Cheers,
> > > > > > > > > > > Dmitri.
> > > > > > > > > > >
> > > > > > > > > > > On Fri, Feb 27, 2026 at 6:19 PM Anand Kumar Sankaran
> via
> > > dev
> > > > <
> > > > > > > > > > > dev@polaris.apache.org> wrote:
> > > > > > > > > > >
> > > > > > > > > > > > Thanks all. This PR is merged now.
> > > > > > > > > > > >
> > > > > > > > > > > > Here are the follow-up features / work needed.  These
> > > were
> > > > > all
> > > > > > > part
> > > > > > > > > of
> > > > > > > > > > > the
> > > > > > > > > > > > merged PR at some point in time and were removed to
> > > reduce
> > > > > > scope.
> > > > > > > > > > > >
> > > > > > > > > > > > Please let me know what you think.
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > >   1.  A REST API to paginate through table metrics.
> > This
> > > > will
> > > > > > be
> > > > > > > > > > non-IRC
> > > > > > > > > > > > standard addition.
> > > > > > > > > > > >   2.  Utilities for managing old records, should be
> > > common
> > > > > with
> > > > > > > > > events.
> > > > > > > > > > > > There was some discussion that it belongs to the CLI.
> > > > > > > > > > > >   3.  Separate datasource (metrics, events, even
> other
> > > > > > tables?).
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > Anything else?
> > > > > > > > > > > >
> > > > > > > > > > > > -
> > > > > > > > > > > > Anand
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>

Re: Proposal for REST endpoints for table metrics and events

[Yufei Gu <fl...@gmail.com>]

I think the Events API in Iceberg is already quite close to finalization,
so I don’t see much value in introducing a separate unstable version in
Polaris ATM.

For the metrics endpoint, I’m not aware of any ongoing discussion in the
Iceberg community. I’d suggest starting there first. If there isn’t
interest, Polaris can move forward and introduce a metrics endpoint
directly, without necessarily making it experimental. If there is interest,
it would make more sense to align with Iceberg and contribute there. We can
then decide whether we still need a fallback option in Polaris, such as an
experimental endpoint.

Overall, it seems worth checking with the Iceberg community before moving
forward.

Yufei


On Mon, Mar 23, 2026 at 11:33 AM Dmitri Bourlatchkov <di...@apache.org>
wrote:

> Hi Nandor,
>
> Good point! The new metrics and events APIs should definitely be marked as
> "experimental" or "beta" initially.
>
> That's a common practice for new APIs in Polaris.
>
> Cheers,
> Dmitri.
>
> On Thu, Mar 19, 2026 at 4:30 PM Nándor Kollár <nk...@apache.org> wrote:
>
> > Hi All,
> >
> > I don’t see a problem with having two REST APIs for catalog events. We
> can
> > mark the local API as experimental (if there’s a way to do that in
> > Polaris), document that it isn’t stable, and deprecate it once the
> Iceberg
> > API is released. Alternatively, we could keep and continue improving it
> > with features that are missing from the Iceberg REST spec but are
> relevant
> > for Polaris.
> >
> > Nandor
> >
> > Dmitri Bourlatchkov <di...@apache.org> ezt írta (időpont: 2026. márc.
> 19.,
> > Cs, 17:44):
> >
> > > Hi All,
> > >
> > > Polaris can support both a local API and new endpoints in the IRC API
> > once
> > > the Iceberg community adopts the latter.
> > >
> > > What is the concern with having different APIs to access the same data?
> > >
> > > Cheers,
> > > Dmitri.
> > >
> > > On Tue, Mar 17, 2026 at 3:49 PM EJ Wang <
> ej.wang.dev.interest@gmail.com>
> > > wrote:
> > >
> > > > I share the same feeling with Yifei as reviewing the PRs. I want to
> > avoid
> > > > creating discrepancies if the IRC side ended up supporting it.
> > > >
> > > > -ej
> > > >
> > > > On Wed, Mar 11, 2026 at 2:47 PM Yufei Gu <fl...@gmail.com>
> wrote:
> > > >
> > > > > Thanks Anand for working on this.
> > > > >
> > > > > Given the IRC event endpoint is WIP, I think it'd be best to be
> > > > consistent
> > > > > with the IRC endpoint. For more context on the IRC event, the event
> > > > > endpoint is not finalized yet. I'd recommend speeding up the IRC
> side
> > > > work
> > > > > to avoid any inconsistencies between Polaris and IRC spec.
> > > > >
> > > > > I have a few questions about the metrics endpoint:
> > > > > 1. Do we need to expose them via Polaris REST endpoint? Can users
> > grab
> > > > the
> > > > > metrics from the backend directly? I understand the RBAC won't be
> > > there,
> > > > > but it provides flexibility for users. For example, some users may
> > > > choose a
> > > > > different persistence model such as a KV store or storing the
> metrics
> > > as
> > > > > objects in S3, which usually scales better than an RDBMS like
> > Postgres.
> > > > My
> > > > > understanding is that Polaris is not intended to be a full metrics
> > > system
> > > > > anyway, but rather to provide a way for downstream systems to
> consume
> > > > these
> > > > > data.
> > > > > 2. Should we consider it as an IRC endpoint? Given the metrics
> report
> > > > > endpoint in IRC, the Iceberg community might also be interested in
> > > > serving
> > > > > back metrics, similar to the event endpoint. In that case, there
> is a
> > > > risk
> > > > > of fragmentation if we create a Polaris endpoint now. We should
> avoid
> > > > that
> > > > > if possible. It might be worth checking with the Iceberg community
> > > first.
> > > > >
> > > > > Happy to hear others’ thoughts on this.
> > > > >
> > > > > Yufei
> > > > >
> > > > >
> > > > > On Wed, Mar 11, 2026 at 9:27 AM Nándor Kollár <nk...@apache.org>
> > > > wrote:
> > > > >
> > > > > > I agree with Dmitri. The direction outlined in the proposal looks
> > > good
> > > > to
> > > > > > me, and the finer details can be worked out as implementation
> gets
> > > > > > underway. We can adjust the design doc accordingly, later on.
> > > > > >
> > > > > >
> > > > > > Thanks,
> > > > > > Nandor
> > > > > >
> > > > > > Dmitri Bourlatchkov <di...@apache.org> ezt írta (időpont: 2026.
> > > márc.
> > > > > 10.,
> > > > > > K, 20:43):
> > > > > >
> > > > > > > Hi Anand and All,
> > > > > > >
> > > > > > > The proposal LGTM in its current form.
> > > > > > >
> > > > > > > My personal approach is that a proposal does not have to be as
> > > > > "polished"
> > > > > > > as the final API spec. As long as we have consensus on the
> > general
> > > > > > > approach and the basic API principles, I think we can proceed
> to
> > > > > > > implementation and iron out final wrinkles during actual API
> spec
> > > and
> > > > > > code
> > > > > > > PRs.
> > > > > > >
> > > > > > > Would this approach work for everyone?
> > > > > > >
> > > > > > > Re-posting my comment from GH [1] here for visibility, in case
> > > people
> > > > > > have
> > > > > > > different opinions and wish to discuss this in more depth:
> > > > > > >
> > > > > > > From my POV, Polaris is a platform, indeed. In this sense, I
> > think
> > > it
> > > > > is
> > > > > > > > critical to enable users to control what features are at play
> > in
> > > > > > runtime,
> > > > > > > > since different users have different use cases. This is why I
> > > > > > originally
> > > > > > > > advocated for isolating Metrics Persistence from MetaStore
> > > > > Persistence.
> > > > > > > >
> > > > > > > > If a user decides to leverage the "native" Polaris
> > (scan/commit)
> > > > > > Metrics
> > > > > > > > Persistence, I do not see any disadvantage in also exposing
> an
> > > > > > (optional)
> > > > > > > > REST API for loading these metrics from Polaris Persistence.
> > > > > > > >
> > > > > > > > The degree of support and sophistication that goes into this
> > > > > sub-system
> > > > > > > is
> > > > > > > > up to the community. If we have contributors (like @obelix74
> )
> > > who
> > > > > are
> > > > > > > > willing to evolve it, I see no harm in some functionality
> > overlap
> > > > > with
> > > > > > > more
> > > > > > > > focused metrics platforms. Again, the key point is for all
> > users
> > > to
> > > > > > have
> > > > > > > > control and be able to opt in or out of this feature in their
> > > > > specific
> > > > > > > > deployments.
> > > > > > > >
> > > > > > > > Of course, offloading scan/commit metrics storage to a
> > > specialized
> > > > > > > > observability system is possible too (assuming someone
> develops
> > > > > > > integration
> > > > > > > > code for that, which is very welcome).
> > > > > > >
> > > > > > >
> > > > > > > [1]
> > > > https://github.com/apache/polaris/pull/3924#discussion_r2913947744
> > > > > > >
> > > > > > > Thanks,
> > > > > > > Dmitri.
> > > > > > >
> > > > > > > On Tue, Mar 10, 2026 at 12:37 PM Anand Kumar Sankaran via dev <
> > > > > > > dev@polaris.apache.org> wrote:
> > > > > > >
> > > > > > > > Hi EJ Wang and Dmitri,
> > > > > > > >
> > > > > > > > I addressed all your concerns about the proposal, in
> particular
> > > > > > > >
> > > https://github.com/apache/polaris/pull/3924#discussion_r2908317696
> > > > .
> > > > > > > >
> > > > > > > > Does this address your concerns?
> > > > > > > >
> > > > > > > > -
> > > > > > > > Anand
> > > > > > > >
> > > > > > > > From: Dmitri Bourlatchkov <di...@apache.org>
> > > > > > > > Date: Monday, March 9, 2026 at 1:17 PM
> > > > > > > > To: dev@polaris.apache.org <de...@polaris.apache.org>
> > > > > > > > Subject: Re: Proposal for REST endpoints for table metrics
> and
> > > > events
> > > > > > > >
> > > > > > > > This Message Is From an External Sender
> > > > > > > > This message came from outside your organization.
> > > > > > > > Report Suspicious<
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://us-phishalarm-ewt.proofpoint.com/EWT/v1/Iz9xO38YGHZK!YhNDZAGr2cumYdtJ_UTm0gR9PfI_-PwSpR_GtNr1uVQ_xo-s2AskvUmbkLZ-C5V8eOKN-omus47On4k4hfFo-0G7CHMLwVjEego-rZrPuepAybX7DP8Ua0VNSrsZ83C4$
> > > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > Hi EJ,
> > > > > > > >
> > > > > > > > You make good points about the metrics API extensibility and
> > > > > evolution.
> > > > > > > >
> > > > > > > > However, we need to consider practical aspects too. Anand
> > appears
> > > > to
> > > > > > have
> > > > > > > > some specific use cases in mind, and I assume his proposal
> > > > addresses
> > > > > > > them.
> > > > > > > >
> > > > > > > > Starting with an API + implementation that works for some
> real
> > > > > > > > world applications will validate the feature's usability.
> > > > > > > >
> > > > > > > > We can revamp the API completely in its v2 after v1 is
> merged.
> > > New
> > > > > > major
> > > > > > > > API versions do not have to be backward-compatible with older
> > > > > versions
> > > > > > of
> > > > > > > > the same API [1].
> > > > > > > >
> > > > > > > > In my personal experience, a v1 API can hardly be expected to
> > > cover
> > > > > all
> > > > > > > use
> > > > > > > > cases and extensions well. We can certainly take a bit more
> > time
> > > to
> > > > > > > polish
> > > > > > > > it, but from my POV it might be best to iterate in terms of
> API
> > > > > > versions
> > > > > > > > rather than on unmerged commits in the initial proposal. Just
> > my
> > > 2
> > > > > > cents
> > > > > > > :)
> > > > > > > >
> > > > > > > > That said, we should flag the new APIs in this proposal as
> > > > "beta"...
> > > > > at
> > > > > > > > least initially (which is the usual practice in Polaris).
> > > > > > > >
> > > > > > > > > I wonder if it would help to evaluate the Events API and
> > > Metrics
> > > > > API
> > > > > > a
> > > > > > > > bit more independently.
> > > > > > > >
> > > > > > > > That makes sense to me. However, the current proposal
> > progressed
> > > a
> > > > > lot
> > > > > > > > since its initial submission and contained both APIs. I would
> > not
> > > > > want
> > > > > > to
> > > > > > > > lose this momentum.
> > > > > > > >
> > > > > > > > It might still be advisable to implement the events and
> metrics
> > > > APIs
> > > > > > > > separately and gather additional feedback at that time.
> > > > > > > >
> > > > > > > > [1]
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://urldefense.com/v3/__https://polaris.apache.org/in-dev/unreleased/evolution/__;!!Iz9xO38YGHZK!8KJ0uv4jK3mxZP4nYFrL1hZ0fMkQvoVEAJa8t9LBCzVtm_PWVFGQfIcZp-ykn3_F9_ph6EYyu3dUZjPAcQ$
> > > > > > > >
> > > > > > > > Cheers,
> > > > > > > > Dmitri.
> > > > > > > >
> > > > > > > > On Mon, Mar 9, 2026 at 3:48 PM EJ Wang <
> > > > > ej.wang.dev.interest@gmail.com
> > > > > > >
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > > > Hi Anand,
> > > > > > > > >
> > > > > > > > > I think the proposal is moving in a better direction,
> > > especially
> > > > on
> > > > > > the
> > > > > > > > > Events side, and I appreciate the iteration so far. That
> > said,
> > > I
> > > > > > still
> > > > > > > > have
> > > > > > > > > some concerns about the Metrics side, but they are less
> about
> > > > > > > individual
> > > > > > > > > parameters or endpoint shape, and more about product
> > boundary.
> > > > > > > > >
> > > > > > > > > 2 cents: I wonder if it would help to evaluate the Events
> API
> > > and
> > > > > > > Metrics
> > > > > > > > > API a bit more independently.
> > > > > > > > >
> > > > > > > > > The Events side feels relatively close to Polaris'
> > > > > catalog/change-log
> > > > > > > > > scope. It is easier to justify as part of the
> core/community
> > > > > surface,
> > > > > > > > > especially if the goal is to expose completed catalog
> > mutations
> > > > in
> > > > > a
> > > > > > > way
> > > > > > > > > that aligns with Iceberg-style events.
> > > > > > > > >
> > > > > > > > > The Metrics side feels different to me. Once we start
> adding
> > > more
> > > > > and
> > > > > > > > more
> > > > > > > > > type-specific filters, query semantics, and schema shape
> for
> > > > > > individual
> > > > > > > > > metric families, it seems easy for Polaris to drift toward
> a
> > > > > built-in
> > > > > > > > > observability backend. My bias would be for Polaris to
> > support
> > > a
> > > > > > > smaller
> > > > > > > > > set of community-recognized built-in metrics well, while
> > > > providing
> > > > > > good
> > > > > > > > > extensibility points for deployments that want richer
> > querying,
> > > > > > > > > visualization, or use-case-specific metrics.
> > > > > > > > >
> > > > > > > > > Related to that, I am not yet convinced the current metrics
> > > model
> > > > > is
> > > > > > > > > generic enough as a long-term direction. Even after
> > > consolidating
> > > > > to
> > > > > > a
> > > > > > > > > single endpoint, the design still feels fairly tied to the
> > > > current
> > > > > > > > > scan/commit shape. I worry that otherwise each new metric
> > > family
> > > > > will
> > > > > > > > keep
> > > > > > > > > pulling us into more storage/schema/API reshaping inside
> > > Polaris
> > > > > > core.
> > > > > > > > > So the framing question I would suggest is something like:
> > > > > > > > > > What is the minimal built-in metrics surface Polaris
> should
> > > own
> > > > > in
> > > > > > > > core,
> > > > > > > > > and where should we instead rely on extensibility /
> > > sink-export /
> > > > > > > > > plugin-style integration?
> > > > > > > > >
> > > > > > > > > For me, getting that boundary right matters more than
> > settling
> > > > > every
> > > > > > > > query
> > > > > > > > > parameter detail first.
> > > > > > > > >
> > > > > > > > > -ej
> > > > > > > > >
> > > > > > > > > On Tue, Mar 3, 2026 at 12:29 PM Anand Kumar Sankaran via
> dev
> > <
> > > > > > > > > dev@polaris.apache.org> wrote:
> > > > > > > > >
> > > > > > > > > > Hi Yufei and Dmitri,
> > > > > > > > > >
> > > > > > > > > > Here is a proposal for the REST endpoints for metrics and
> > > > events.
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://urldefense.com/v3/__https://github.com/apache/polaris/pull/3924/changes__;!!Iz9xO38YGHZK!8KJ0uv4jK3mxZP4nYFrL1hZ0fMkQvoVEAJa8t9LBCzVtm_PWVFGQfIcZp-ykn3_F9_ph6EYyu3fXDjRYWA$
> > > > > > > > > >
> > > > > > > > > > I did not see any precursors for raising a PR for
> > proposals,
> > > so
> > > > > > > trying
> > > > > > > > > > this.  Please let me know what you think.
> > > > > > > > > >
> > > > > > > > > > -
> > > > > > > > > > Anand
> > > > > > > > > >
> > > > > > > > > > From: Anand Kumar Sankaran <an...@workday.com>
> > > > > > > > > > Date: Monday, March 2, 2026 at 10:25 AM
> > > > > > > > > > To: dev@polaris.apache.org <de...@polaris.apache.org>
> > > > > > > > > > Subject: Re: Polaris Telemetry and Audit Trail
> > > > > > > > > >
> > > > > > > > > > About the REST API, based on my use cases:
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >   1.
> > > > > > > > > > I want to be able to query commit metrics to track files
> > > added
> > > > /
> > > > > > > > removed
> > > > > > > > > > per commit, along with record counts. The ingestion
> > pipeline
> > > > that
> > > > > > > > writes
> > > > > > > > > > this data is owned by us and we are guaranteed to write
> > this
> > > > > > > > information
> > > > > > > > > > for each write.
> > > > > > > > > >   2.
> > > > > > > > > > I want to be able to query scan metrics for read. I
> > > understand
> > > > > > > clients
> > > > > > > > do
> > > > > > > > > > not fulfill this requirement.
> > > > > > > > > >   3.
> > > > > > > > > > I want to be able to query the events table (events are
> > > > > persisted)
> > > > > > -
> > > > > > > > this
> > > > > > > > > > may supersede #2, I am not sure yet.
> > > > > > > > > >
> > > > > > > > > > All this information is in the JDBC based persistence
> model
> > > and
> > > > > is
> > > > > > > > > > persisted in the metastore. I currently don’t have a need
> > to
> > > > > query
> > > > > > > > > > prometheus or open telemetry. I do publish some events to
> > > > > > Prometheus
> > > > > > > > and
> > > > > > > > > > they are forwarded to our dashboards elsewhere.
> > > > > > > > > >
> > > > > > > > > > About the CLI utilities, I meant the admin user
> utilities.
> > In
> > > > one
> > > > > > of
> > > > > > > > the
> > > > > > > > > > earliest drafts of my proposal, Prashant mentioned that
> the
> > > > > metrics
> > > > > > > > > tables
> > > > > > > > > > can grow indefinitely and that a similar problem exists
> > with
> > > > the
> > > > > > > events
> > > > > > > > > > table as well. We discussed that cleaning up of old
> records
> > > > from
> > > > > > both
> > > > > > > > > > metrics tables and events tables can be done via a CLI
> > > utility.
> > > > > > > > > >
> > > > > > > > > > I see that Yufei has covered the discussion about
> > > datasources.
> > > > > > > > > >
> > > > > > > > > > -
> > > > > > > > > > Anand
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > From: Yufei Gu <fl...@gmail.com>
> > > > > > > > > > Date: Friday, February 27, 2026 at 9:54 PM
> > > > > > > > > > To: dev@polaris.apache.org <de...@polaris.apache.org>
> > > > > > > > > > Subject: Re: Polaris Telemetry and Audit Trail
> > > > > > > > > >
> > > > > > > > > > This Message Is From an External Sender
> > > > > > > > > > This message came from outside your organization.
> > > > > > > > > > Report Suspicious<
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://us-phishalarm-ewt.proofpoint.com/EWT/v1/Iz9xO38YGHZK!YhNDZABkHi1B699ote2uMwpOZw8i0QMCGO2Szc-HshuABGhGvwPJcymE6G2oUUxtS8xDkSrtGTPm_I3QnVDHoLMk50m9v8z_nZKTkd-bnVUbreF1u0WnfV_X5eYevZl_$
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > As I mentioned in
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://urldefense.com/v3/__https://github.com/apache/polaris/issues/3890__;!!Iz9xO38YGHZK!5EuyFFkk3vhRWVIRvQAWBSQfpJkTMA9HxugzDwXmN0LPPqhEFxYkFRGVhtb8AqUwXtDh2OplcMnbMDHKOxrvDU0$
> > > > > > > > >,
> > > > > > > > > > supporting
> > > > > > > > > > multiple data sources is not a trivial change. I would
> > > strongly
> > > > > > > > recommend
> > > > > > > > > > starting with a design document to carefully evaluate the
> > > > > > > architectural
> > > > > > > > > > implications and long term impact.
> > > > > > > > > >
> > > > > > > > > > A REST endpoint to query metrics seems reasonable given
> the
> > > > > current
> > > > > > > > JDBC
> > > > > > > > > > based persistence model. That said, we may also consider
> > > > > > alternative
> > > > > > > > > > storage models. For example, if we later adopt a time
> > series
> > > > > system
> > > > > > > > such
> > > > > > > > > as
> > > > > > > > > > Prometheus to store metrics, the query model and access
> > > > patterns
> > > > > > > would
> > > > > > > > be
> > > > > > > > > > fundamentally different. Designing the REST API without
> > > > > considering
> > > > > > > > these
> > > > > > > > > > potential evolutions may limit flexibility. I'd suggest
> to
> > > > start
> > > > > > with
> > > > > > > > the
> > > > > > > > > > use case.
> > > > > > > > > >
> > > > > > > > > > Yufei
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > On Fri, Feb 27, 2026 at 3:42 PM Dmitri Bourlatchkov <
> > > > > > > dimas@apache.org>
> > > > > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > > Hi Anand,
> > > > > > > > > > >
> > > > > > > > > > > Sharing my view... subject to discussion:
> > > > > > > > > > >
> > > > > > > > > > > 1. Adding non-IRC REST API to Polaris is perfectly
> fine.
> > > > > > > > > > >
> > > > > > > > > > > Figuring out specific endpoint URIs and payloads might
> > > > require
> > > > > a
> > > > > > > few
> > > > > > > > > > > roundtrips, so opening a separate thread for that might
> > be
> > > > > best.
> > > > > > > > > > > Contributors commonly create Google Docs for new API
> > > > proposals
> > > > > > too
> > > > > > > > > (they
> > > > > > > > > > > fairly easy to update as the email discussion
> > progresses).
> > > > > > > > > > >
> > > > > > > > > > > There was a suggestion to try Markdown (with PRs) for
> > > > proposals
> > > > > > [1]
> > > > > > > > ...
> > > > > > > > > > > feel free to give it a try if you are comfortable with
> > > that.
> > > > > > > > > > >
> > > > > > > > > > > 2. Could you clarify whether you mean end user
> utilities
> > or
> > > > > admin
> > > > > > > > user
> > > > > > > > > > > utilities? In the latter case those might be more
> > suitable
> > > > for
> > > > > > the
> > > > > > > > > Admin
> > > > > > > > > > > CLI (java) not the Python CLI, IMHO.
> > > > > > > > > > >
> > > > > > > > > > > Why would these utilities be common with events? IMHO,
> > > event
> > > > > use
> > > > > > > > cases
> > > > > > > > > > are
> > > > > > > > > > > distinct from scan/commit metrics.
> > > > > > > > > > >
> > > > > > > > > > > 3. I'd prefer separating metrics persistence from
> > MetaStore
> > > > > > > > persistence
> > > > > > > > > > at
> > > > > > > > > > > the code level, so that they could be mixed and matched
> > > > > > > > independently.
> > > > > > > > > > The
> > > > > > > > > > > separate datasource question will become a non-issue
> with
> > > > that
> > > > > > > > > approach,
> > > > > > > > > > I
> > > > > > > > > > > guess.
> > > > > > > > > > >
> > > > > > > > > > > The rationale for separating scan metrics and metastore
> > > > > > persistence
> > > > > > > > is
> > > > > > > > > > that
> > > > > > > > > > > "cascading deletes" between them are hardly ever
> > required.
> > > > > > > > Furthermore,
> > > > > > > > > > the
> > > > > > > > > > > data and query patterns are very different so different
> > > > > > > technologies
> > > > > > > > > > might
> > > > > > > > > > > be beneficial in each case.
> > > > > > > > > > >
> > > > > > > > > > > [1]
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://urldefense.com/v3/__https://lists.apache.org/thread/yto2wp982t43h1mqjwnslswhws5z47cy__;!!Iz9xO38YGHZK!5EuyFFkk3vhRWVIRvQAWBSQfpJkTMA9HxugzDwXmN0LPPqhEFxYkFRGVhtb8AqUwXtDh2OplcMnbMDHKxYDakNU$
> > > > > > > > >> >
> > > > > > > > > > > Cheers,
> > > > > > > > > > > Dmitri.
> > > > > > > > > > >
> > > > > > > > > > > On Fri, Feb 27, 2026 at 6:19 PM Anand Kumar Sankaran
> via
> > > dev
> > > > <
> > > > > > > > > > > dev@polaris.apache.org> wrote:
> > > > > > > > > > >
> > > > > > > > > > > > Thanks all. This PR is merged now.
> > > > > > > > > > > >
> > > > > > > > > > > > Here are the follow-up features / work needed.  These
> > > were
> > > > > all
> > > > > > > part
> > > > > > > > > of
> > > > > > > > > > > the
> > > > > > > > > > > > merged PR at some point in time and were removed to
> > > reduce
> > > > > > scope.
> > > > > > > > > > > >
> > > > > > > > > > > > Please let me know what you think.
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > >   1.  A REST API to paginate through table metrics.
> > This
> > > > will
> > > > > > be
> > > > > > > > > > non-IRC
> > > > > > > > > > > > standard addition.
> > > > > > > > > > > >   2.  Utilities for managing old records, should be
> > > common
> > > > > with
> > > > > > > > > events.
> > > > > > > > > > > > There was some discussion that it belongs to the CLI.
> > > > > > > > > > > >   3.  Separate datasource (metrics, events, even
> other
> > > > > > tables?).
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > Anything else?
> > > > > > > > > > > >
> > > > > > > > > > > > -
> > > > > > > > > > > > Anand
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>

Re: Proposal for REST endpoints for table metrics and events

[Dmitri Bourlatchkov <di...@apache.org>]

Hi Nandor,

Good point! The new metrics and events APIs should definitely be marked as
"experimental" or "beta" initially.

That's a common practice for new APIs in Polaris.

Cheers,
Dmitri.

On Thu, Mar 19, 2026 at 4:30 PM Nándor Kollár <nk...@apache.org> wrote:

> Hi All,
>
> I don’t see a problem with having two REST APIs for catalog events. We can
> mark the local API as experimental (if there’s a way to do that in
> Polaris), document that it isn’t stable, and deprecate it once the Iceberg
> API is released. Alternatively, we could keep and continue improving it
> with features that are missing from the Iceberg REST spec but are relevant
> for Polaris.
>
> Nandor
>
> Dmitri Bourlatchkov <di...@apache.org> ezt írta (időpont: 2026. márc. 19.,
> Cs, 17:44):
>
> > Hi All,
> >
> > Polaris can support both a local API and new endpoints in the IRC API
> once
> > the Iceberg community adopts the latter.
> >
> > What is the concern with having different APIs to access the same data?
> >
> > Cheers,
> > Dmitri.
> >
> > On Tue, Mar 17, 2026 at 3:49 PM EJ Wang <ej...@gmail.com>
> > wrote:
> >
> > > I share the same feeling with Yifei as reviewing the PRs. I want to
> avoid
> > > creating discrepancies if the IRC side ended up supporting it.
> > >
> > > -ej
> > >
> > > On Wed, Mar 11, 2026 at 2:47 PM Yufei Gu <fl...@gmail.com> wrote:
> > >
> > > > Thanks Anand for working on this.
> > > >
> > > > Given the IRC event endpoint is WIP, I think it'd be best to be
> > > consistent
> > > > with the IRC endpoint. For more context on the IRC event, the event
> > > > endpoint is not finalized yet. I'd recommend speeding up the IRC side
> > > work
> > > > to avoid any inconsistencies between Polaris and IRC spec.
> > > >
> > > > I have a few questions about the metrics endpoint:
> > > > 1. Do we need to expose them via Polaris REST endpoint? Can users
> grab
> > > the
> > > > metrics from the backend directly? I understand the RBAC won't be
> > there,
> > > > but it provides flexibility for users. For example, some users may
> > > choose a
> > > > different persistence model such as a KV store or storing the metrics
> > as
> > > > objects in S3, which usually scales better than an RDBMS like
> Postgres.
> > > My
> > > > understanding is that Polaris is not intended to be a full metrics
> > system
> > > > anyway, but rather to provide a way for downstream systems to consume
> > > these
> > > > data.
> > > > 2. Should we consider it as an IRC endpoint? Given the metrics report
> > > > endpoint in IRC, the Iceberg community might also be interested in
> > > serving
> > > > back metrics, similar to the event endpoint. In that case, there is a
> > > risk
> > > > of fragmentation if we create a Polaris endpoint now. We should avoid
> > > that
> > > > if possible. It might be worth checking with the Iceberg community
> > first.
> > > >
> > > > Happy to hear others’ thoughts on this.
> > > >
> > > > Yufei
> > > >
> > > >
> > > > On Wed, Mar 11, 2026 at 9:27 AM Nándor Kollár <nk...@apache.org>
> > > wrote:
> > > >
> > > > > I agree with Dmitri. The direction outlined in the proposal looks
> > good
> > > to
> > > > > me, and the finer details can be worked out as implementation gets
> > > > > underway. We can adjust the design doc accordingly, later on.
> > > > >
> > > > >
> > > > > Thanks,
> > > > > Nandor
> > > > >
> > > > > Dmitri Bourlatchkov <di...@apache.org> ezt írta (időpont: 2026.
> > márc.
> > > > 10.,
> > > > > K, 20:43):
> > > > >
> > > > > > Hi Anand and All,
> > > > > >
> > > > > > The proposal LGTM in its current form.
> > > > > >
> > > > > > My personal approach is that a proposal does not have to be as
> > > > "polished"
> > > > > > as the final API spec. As long as we have consensus on the
> general
> > > > > > approach and the basic API principles, I think we can proceed to
> > > > > > implementation and iron out final wrinkles during actual API spec
> > and
> > > > > code
> > > > > > PRs.
> > > > > >
> > > > > > Would this approach work for everyone?
> > > > > >
> > > > > > Re-posting my comment from GH [1] here for visibility, in case
> > people
> > > > > have
> > > > > > different opinions and wish to discuss this in more depth:
> > > > > >
> > > > > > From my POV, Polaris is a platform, indeed. In this sense, I
> think
> > it
> > > > is
> > > > > > > critical to enable users to control what features are at play
> in
> > > > > runtime,
> > > > > > > since different users have different use cases. This is why I
> > > > > originally
> > > > > > > advocated for isolating Metrics Persistence from MetaStore
> > > > Persistence.
> > > > > > >
> > > > > > > If a user decides to leverage the "native" Polaris
> (scan/commit)
> > > > > Metrics
> > > > > > > Persistence, I do not see any disadvantage in also exposing an
> > > > > (optional)
> > > > > > > REST API for loading these metrics from Polaris Persistence.
> > > > > > >
> > > > > > > The degree of support and sophistication that goes into this
> > > > sub-system
> > > > > > is
> > > > > > > up to the community. If we have contributors (like @obelix74 )
> > who
> > > > are
> > > > > > > willing to evolve it, I see no harm in some functionality
> overlap
> > > > with
> > > > > > more
> > > > > > > focused metrics platforms. Again, the key point is for all
> users
> > to
> > > > > have
> > > > > > > control and be able to opt in or out of this feature in their
> > > > specific
> > > > > > > deployments.
> > > > > > >
> > > > > > > Of course, offloading scan/commit metrics storage to a
> > specialized
> > > > > > > observability system is possible too (assuming someone develops
> > > > > > integration
> > > > > > > code for that, which is very welcome).
> > > > > >
> > > > > >
> > > > > > [1]
> > > https://github.com/apache/polaris/pull/3924#discussion_r2913947744
> > > > > >
> > > > > > Thanks,
> > > > > > Dmitri.
> > > > > >
> > > > > > On Tue, Mar 10, 2026 at 12:37 PM Anand Kumar Sankaran via dev <
> > > > > > dev@polaris.apache.org> wrote:
> > > > > >
> > > > > > > Hi EJ Wang and Dmitri,
> > > > > > >
> > > > > > > I addressed all your concerns about the proposal, in particular
> > > > > > >
> > https://github.com/apache/polaris/pull/3924#discussion_r2908317696
> > > .
> > > > > > >
> > > > > > > Does this address your concerns?
> > > > > > >
> > > > > > > -
> > > > > > > Anand
> > > > > > >
> > > > > > > From: Dmitri Bourlatchkov <di...@apache.org>
> > > > > > > Date: Monday, March 9, 2026 at 1:17 PM
> > > > > > > To: dev@polaris.apache.org <de...@polaris.apache.org>
> > > > > > > Subject: Re: Proposal for REST endpoints for table metrics and
> > > events
> > > > > > >
> > > > > > > This Message Is From an External Sender
> > > > > > > This message came from outside your organization.
> > > > > > > Report Suspicious<
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://us-phishalarm-ewt.proofpoint.com/EWT/v1/Iz9xO38YGHZK!YhNDZAGr2cumYdtJ_UTm0gR9PfI_-PwSpR_GtNr1uVQ_xo-s2AskvUmbkLZ-C5V8eOKN-omus47On4k4hfFo-0G7CHMLwVjEego-rZrPuepAybX7DP8Ua0VNSrsZ83C4$
> > > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > Hi EJ,
> > > > > > >
> > > > > > > You make good points about the metrics API extensibility and
> > > > evolution.
> > > > > > >
> > > > > > > However, we need to consider practical aspects too. Anand
> appears
> > > to
> > > > > have
> > > > > > > some specific use cases in mind, and I assume his proposal
> > > addresses
> > > > > > them.
> > > > > > >
> > > > > > > Starting with an API + implementation that works for some real
> > > > > > > world applications will validate the feature's usability.
> > > > > > >
> > > > > > > We can revamp the API completely in its v2 after v1 is merged.
> > New
> > > > > major
> > > > > > > API versions do not have to be backward-compatible with older
> > > > versions
> > > > > of
> > > > > > > the same API [1].
> > > > > > >
> > > > > > > In my personal experience, a v1 API can hardly be expected to
> > cover
> > > > all
> > > > > > use
> > > > > > > cases and extensions well. We can certainly take a bit more
> time
> > to
> > > > > > polish
> > > > > > > it, but from my POV it might be best to iterate in terms of API
> > > > > versions
> > > > > > > rather than on unmerged commits in the initial proposal. Just
> my
> > 2
> > > > > cents
> > > > > > :)
> > > > > > >
> > > > > > > That said, we should flag the new APIs in this proposal as
> > > "beta"...
> > > > at
> > > > > > > least initially (which is the usual practice in Polaris).
> > > > > > >
> > > > > > > > I wonder if it would help to evaluate the Events API and
> > Metrics
> > > > API
> > > > > a
> > > > > > > bit more independently.
> > > > > > >
> > > > > > > That makes sense to me. However, the current proposal
> progressed
> > a
> > > > lot
> > > > > > > since its initial submission and contained both APIs. I would
> not
> > > > want
> > > > > to
> > > > > > > lose this momentum.
> > > > > > >
> > > > > > > It might still be advisable to implement the events and metrics
> > > APIs
> > > > > > > separately and gather additional feedback at that time.
> > > > > > >
> > > > > > > [1]
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://urldefense.com/v3/__https://polaris.apache.org/in-dev/unreleased/evolution/__;!!Iz9xO38YGHZK!8KJ0uv4jK3mxZP4nYFrL1hZ0fMkQvoVEAJa8t9LBCzVtm_PWVFGQfIcZp-ykn3_F9_ph6EYyu3dUZjPAcQ$
> > > > > > >
> > > > > > > Cheers,
> > > > > > > Dmitri.
> > > > > > >
> > > > > > > On Mon, Mar 9, 2026 at 3:48 PM EJ Wang <
> > > > ej.wang.dev.interest@gmail.com
> > > > > >
> > > > > > > wrote:
> > > > > > >
> > > > > > > > Hi Anand,
> > > > > > > >
> > > > > > > > I think the proposal is moving in a better direction,
> > especially
> > > on
> > > > > the
> > > > > > > > Events side, and I appreciate the iteration so far. That
> said,
> > I
> > > > > still
> > > > > > > have
> > > > > > > > some concerns about the Metrics side, but they are less about
> > > > > > individual
> > > > > > > > parameters or endpoint shape, and more about product
> boundary.
> > > > > > > >
> > > > > > > > 2 cents: I wonder if it would help to evaluate the Events API
> > and
> > > > > > Metrics
> > > > > > > > API a bit more independently.
> > > > > > > >
> > > > > > > > The Events side feels relatively close to Polaris'
> > > > catalog/change-log
> > > > > > > > scope. It is easier to justify as part of the core/community
> > > > surface,
> > > > > > > > especially if the goal is to expose completed catalog
> mutations
> > > in
> > > > a
> > > > > > way
> > > > > > > > that aligns with Iceberg-style events.
> > > > > > > >
> > > > > > > > The Metrics side feels different to me. Once we start adding
> > more
> > > > and
> > > > > > > more
> > > > > > > > type-specific filters, query semantics, and schema shape for
> > > > > individual
> > > > > > > > metric families, it seems easy for Polaris to drift toward a
> > > > built-in
> > > > > > > > observability backend. My bias would be for Polaris to
> support
> > a
> > > > > > smaller
> > > > > > > > set of community-recognized built-in metrics well, while
> > > providing
> > > > > good
> > > > > > > > extensibility points for deployments that want richer
> querying,
> > > > > > > > visualization, or use-case-specific metrics.
> > > > > > > >
> > > > > > > > Related to that, I am not yet convinced the current metrics
> > model
> > > > is
> > > > > > > > generic enough as a long-term direction. Even after
> > consolidating
> > > > to
> > > > > a
> > > > > > > > single endpoint, the design still feels fairly tied to the
> > > current
> > > > > > > > scan/commit shape. I worry that otherwise each new metric
> > family
> > > > will
> > > > > > > keep
> > > > > > > > pulling us into more storage/schema/API reshaping inside
> > Polaris
> > > > > core.
> > > > > > > > So the framing question I would suggest is something like:
> > > > > > > > > What is the minimal built-in metrics surface Polaris should
> > own
> > > > in
> > > > > > > core,
> > > > > > > > and where should we instead rely on extensibility /
> > sink-export /
> > > > > > > > plugin-style integration?
> > > > > > > >
> > > > > > > > For me, getting that boundary right matters more than
> settling
> > > > every
> > > > > > > query
> > > > > > > > parameter detail first.
> > > > > > > >
> > > > > > > > -ej
> > > > > > > >
> > > > > > > > On Tue, Mar 3, 2026 at 12:29 PM Anand Kumar Sankaran via dev
> <
> > > > > > > > dev@polaris.apache.org> wrote:
> > > > > > > >
> > > > > > > > > Hi Yufei and Dmitri,
> > > > > > > > >
> > > > > > > > > Here is a proposal for the REST endpoints for metrics and
> > > events.
> > > > > > > > >
> > > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://urldefense.com/v3/__https://github.com/apache/polaris/pull/3924/changes__;!!Iz9xO38YGHZK!8KJ0uv4jK3mxZP4nYFrL1hZ0fMkQvoVEAJa8t9LBCzVtm_PWVFGQfIcZp-ykn3_F9_ph6EYyu3fXDjRYWA$
> > > > > > > > >
> > > > > > > > > I did not see any precursors for raising a PR for
> proposals,
> > so
> > > > > > trying
> > > > > > > > > this.  Please let me know what you think.
> > > > > > > > >
> > > > > > > > > -
> > > > > > > > > Anand
> > > > > > > > >
> > > > > > > > > From: Anand Kumar Sankaran <an...@workday.com>
> > > > > > > > > Date: Monday, March 2, 2026 at 10:25 AM
> > > > > > > > > To: dev@polaris.apache.org <de...@polaris.apache.org>
> > > > > > > > > Subject: Re: Polaris Telemetry and Audit Trail
> > > > > > > > >
> > > > > > > > > About the REST API, based on my use cases:
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >   1.
> > > > > > > > > I want to be able to query commit metrics to track files
> > added
> > > /
> > > > > > > removed
> > > > > > > > > per commit, along with record counts. The ingestion
> pipeline
> > > that
> > > > > > > writes
> > > > > > > > > this data is owned by us and we are guaranteed to write
> this
> > > > > > > information
> > > > > > > > > for each write.
> > > > > > > > >   2.
> > > > > > > > > I want to be able to query scan metrics for read. I
> > understand
> > > > > > clients
> > > > > > > do
> > > > > > > > > not fulfill this requirement.
> > > > > > > > >   3.
> > > > > > > > > I want to be able to query the events table (events are
> > > > persisted)
> > > > > -
> > > > > > > this
> > > > > > > > > may supersede #2, I am not sure yet.
> > > > > > > > >
> > > > > > > > > All this information is in the JDBC based persistence model
> > and
> > > > is
> > > > > > > > > persisted in the metastore. I currently don’t have a need
> to
> > > > query
> > > > > > > > > prometheus or open telemetry. I do publish some events to
> > > > > Prometheus
> > > > > > > and
> > > > > > > > > they are forwarded to our dashboards elsewhere.
> > > > > > > > >
> > > > > > > > > About the CLI utilities, I meant the admin user utilities.
> In
> > > one
> > > > > of
> > > > > > > the
> > > > > > > > > earliest drafts of my proposal, Prashant mentioned that the
> > > > metrics
> > > > > > > > tables
> > > > > > > > > can grow indefinitely and that a similar problem exists
> with
> > > the
> > > > > > events
> > > > > > > > > table as well. We discussed that cleaning up of old records
> > > from
> > > > > both
> > > > > > > > > metrics tables and events tables can be done via a CLI
> > utility.
> > > > > > > > >
> > > > > > > > > I see that Yufei has covered the discussion about
> > datasources.
> > > > > > > > >
> > > > > > > > > -
> > > > > > > > > Anand
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > From: Yufei Gu <fl...@gmail.com>
> > > > > > > > > Date: Friday, February 27, 2026 at 9:54 PM
> > > > > > > > > To: dev@polaris.apache.org <de...@polaris.apache.org>
> > > > > > > > > Subject: Re: Polaris Telemetry and Audit Trail
> > > > > > > > >
> > > > > > > > > This Message Is From an External Sender
> > > > > > > > > This message came from outside your organization.
> > > > > > > > > Report Suspicious<
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://us-phishalarm-ewt.proofpoint.com/EWT/v1/Iz9xO38YGHZK!YhNDZABkHi1B699ote2uMwpOZw8i0QMCGO2Szc-HshuABGhGvwPJcymE6G2oUUxtS8xDkSrtGTPm_I3QnVDHoLMk50m9v8z_nZKTkd-bnVUbreF1u0WnfV_X5eYevZl_$
> > > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > As I mentioned in
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://urldefense.com/v3/__https://github.com/apache/polaris/issues/3890__;!!Iz9xO38YGHZK!5EuyFFkk3vhRWVIRvQAWBSQfpJkTMA9HxugzDwXmN0LPPqhEFxYkFRGVhtb8AqUwXtDh2OplcMnbMDHKOxrvDU0$
> > > > > > > >,
> > > > > > > > > supporting
> > > > > > > > > multiple data sources is not a trivial change. I would
> > strongly
> > > > > > > recommend
> > > > > > > > > starting with a design document to carefully evaluate the
> > > > > > architectural
> > > > > > > > > implications and long term impact.
> > > > > > > > >
> > > > > > > > > A REST endpoint to query metrics seems reasonable given the
> > > > current
> > > > > > > JDBC
> > > > > > > > > based persistence model. That said, we may also consider
> > > > > alternative
> > > > > > > > > storage models. For example, if we later adopt a time
> series
> > > > system
> > > > > > > such
> > > > > > > > as
> > > > > > > > > Prometheus to store metrics, the query model and access
> > > patterns
> > > > > > would
> > > > > > > be
> > > > > > > > > fundamentally different. Designing the REST API without
> > > > considering
> > > > > > > these
> > > > > > > > > potential evolutions may limit flexibility. I'd suggest to
> > > start
> > > > > with
> > > > > > > the
> > > > > > > > > use case.
> > > > > > > > >
> > > > > > > > > Yufei
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > On Fri, Feb 27, 2026 at 3:42 PM Dmitri Bourlatchkov <
> > > > > > dimas@apache.org>
> > > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > Hi Anand,
> > > > > > > > > >
> > > > > > > > > > Sharing my view... subject to discussion:
> > > > > > > > > >
> > > > > > > > > > 1. Adding non-IRC REST API to Polaris is perfectly fine.
> > > > > > > > > >
> > > > > > > > > > Figuring out specific endpoint URIs and payloads might
> > > require
> > > > a
> > > > > > few
> > > > > > > > > > roundtrips, so opening a separate thread for that might
> be
> > > > best.
> > > > > > > > > > Contributors commonly create Google Docs for new API
> > > proposals
> > > > > too
> > > > > > > > (they
> > > > > > > > > > fairly easy to update as the email discussion
> progresses).
> > > > > > > > > >
> > > > > > > > > > There was a suggestion to try Markdown (with PRs) for
> > > proposals
> > > > > [1]
> > > > > > > ...
> > > > > > > > > > feel free to give it a try if you are comfortable with
> > that.
> > > > > > > > > >
> > > > > > > > > > 2. Could you clarify whether you mean end user utilities
> or
> > > > admin
> > > > > > > user
> > > > > > > > > > utilities? In the latter case those might be more
> suitable
> > > for
> > > > > the
> > > > > > > > Admin
> > > > > > > > > > CLI (java) not the Python CLI, IMHO.
> > > > > > > > > >
> > > > > > > > > > Why would these utilities be common with events? IMHO,
> > event
> > > > use
> > > > > > > cases
> > > > > > > > > are
> > > > > > > > > > distinct from scan/commit metrics.
> > > > > > > > > >
> > > > > > > > > > 3. I'd prefer separating metrics persistence from
> MetaStore
> > > > > > > persistence
> > > > > > > > > at
> > > > > > > > > > the code level, so that they could be mixed and matched
> > > > > > > independently.
> > > > > > > > > The
> > > > > > > > > > separate datasource question will become a non-issue with
> > > that
> > > > > > > > approach,
> > > > > > > > > I
> > > > > > > > > > guess.
> > > > > > > > > >
> > > > > > > > > > The rationale for separating scan metrics and metastore
> > > > > persistence
> > > > > > > is
> > > > > > > > > that
> > > > > > > > > > "cascading deletes" between them are hardly ever
> required.
> > > > > > > Furthermore,
> > > > > > > > > the
> > > > > > > > > > data and query patterns are very different so different
> > > > > > technologies
> > > > > > > > > might
> > > > > > > > > > be beneficial in each case.
> > > > > > > > > >
> > > > > > > > > > [1]
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://urldefense.com/v3/__https://lists.apache.org/thread/yto2wp982t43h1mqjwnslswhws5z47cy__;!!Iz9xO38YGHZK!5EuyFFkk3vhRWVIRvQAWBSQfpJkTMA9HxugzDwXmN0LPPqhEFxYkFRGVhtb8AqUwXtDh2OplcMnbMDHKxYDakNU$
> > > > > > > >> >
> > > > > > > > > > Cheers,
> > > > > > > > > > Dmitri.
> > > > > > > > > >
> > > > > > > > > > On Fri, Feb 27, 2026 at 6:19 PM Anand Kumar Sankaran via
> > dev
> > > <
> > > > > > > > > > dev@polaris.apache.org> wrote:
> > > > > > > > > >
> > > > > > > > > > > Thanks all. This PR is merged now.
> > > > > > > > > > >
> > > > > > > > > > > Here are the follow-up features / work needed.  These
> > were
> > > > all
> > > > > > part
> > > > > > > > of
> > > > > > > > > > the
> > > > > > > > > > > merged PR at some point in time and were removed to
> > reduce
> > > > > scope.
> > > > > > > > > > >
> > > > > > > > > > > Please let me know what you think.
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > >   1.  A REST API to paginate through table metrics.
> This
> > > will
> > > > > be
> > > > > > > > > non-IRC
> > > > > > > > > > > standard addition.
> > > > > > > > > > >   2.  Utilities for managing old records, should be
> > common
> > > > with
> > > > > > > > events.
> > > > > > > > > > > There was some discussion that it belongs to the CLI.
> > > > > > > > > > >   3.  Separate datasource (metrics, events, even other
> > > > > tables?).
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > Anything else?
> > > > > > > > > > >
> > > > > > > > > > > -
> > > > > > > > > > > Anand
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>

Re: Proposal for REST endpoints for table metrics and events

[Nándor Kollár <nk...@apache.org>]

Hi All,

I don’t see a problem with having two REST APIs for catalog events. We can
mark the local API as experimental (if there’s a way to do that in
Polaris), document that it isn’t stable, and deprecate it once the Iceberg
API is released. Alternatively, we could keep and continue improving it
with features that are missing from the Iceberg REST spec but are relevant
for Polaris.

Nandor

Dmitri Bourlatchkov <di...@apache.org> ezt írta (időpont: 2026. márc. 19.,
Cs, 17:44):

> Hi All,
>
> Polaris can support both a local API and new endpoints in the IRC API once
> the Iceberg community adopts the latter.
>
> What is the concern with having different APIs to access the same data?
>
> Cheers,
> Dmitri.
>
> On Tue, Mar 17, 2026 at 3:49 PM EJ Wang <ej...@gmail.com>
> wrote:
>
> > I share the same feeling with Yifei as reviewing the PRs. I want to avoid
> > creating discrepancies if the IRC side ended up supporting it.
> >
> > -ej
> >
> > On Wed, Mar 11, 2026 at 2:47 PM Yufei Gu <fl...@gmail.com> wrote:
> >
> > > Thanks Anand for working on this.
> > >
> > > Given the IRC event endpoint is WIP, I think it'd be best to be
> > consistent
> > > with the IRC endpoint. For more context on the IRC event, the event
> > > endpoint is not finalized yet. I'd recommend speeding up the IRC side
> > work
> > > to avoid any inconsistencies between Polaris and IRC spec.
> > >
> > > I have a few questions about the metrics endpoint:
> > > 1. Do we need to expose them via Polaris REST endpoint? Can users grab
> > the
> > > metrics from the backend directly? I understand the RBAC won't be
> there,
> > > but it provides flexibility for users. For example, some users may
> > choose a
> > > different persistence model such as a KV store or storing the metrics
> as
> > > objects in S3, which usually scales better than an RDBMS like Postgres.
> > My
> > > understanding is that Polaris is not intended to be a full metrics
> system
> > > anyway, but rather to provide a way for downstream systems to consume
> > these
> > > data.
> > > 2. Should we consider it as an IRC endpoint? Given the metrics report
> > > endpoint in IRC, the Iceberg community might also be interested in
> > serving
> > > back metrics, similar to the event endpoint. In that case, there is a
> > risk
> > > of fragmentation if we create a Polaris endpoint now. We should avoid
> > that
> > > if possible. It might be worth checking with the Iceberg community
> first.
> > >
> > > Happy to hear others’ thoughts on this.
> > >
> > > Yufei
> > >
> > >
> > > On Wed, Mar 11, 2026 at 9:27 AM Nándor Kollár <nk...@apache.org>
> > wrote:
> > >
> > > > I agree with Dmitri. The direction outlined in the proposal looks
> good
> > to
> > > > me, and the finer details can be worked out as implementation gets
> > > > underway. We can adjust the design doc accordingly, later on.
> > > >
> > > >
> > > > Thanks,
> > > > Nandor
> > > >
> > > > Dmitri Bourlatchkov <di...@apache.org> ezt írta (időpont: 2026.
> márc.
> > > 10.,
> > > > K, 20:43):
> > > >
> > > > > Hi Anand and All,
> > > > >
> > > > > The proposal LGTM in its current form.
> > > > >
> > > > > My personal approach is that a proposal does not have to be as
> > > "polished"
> > > > > as the final API spec. As long as we have consensus on the general
> > > > > approach and the basic API principles, I think we can proceed to
> > > > > implementation and iron out final wrinkles during actual API spec
> and
> > > > code
> > > > > PRs.
> > > > >
> > > > > Would this approach work for everyone?
> > > > >
> > > > > Re-posting my comment from GH [1] here for visibility, in case
> people
> > > > have
> > > > > different opinions and wish to discuss this in more depth:
> > > > >
> > > > > From my POV, Polaris is a platform, indeed. In this sense, I think
> it
> > > is
> > > > > > critical to enable users to control what features are at play in
> > > > runtime,
> > > > > > since different users have different use cases. This is why I
> > > > originally
> > > > > > advocated for isolating Metrics Persistence from MetaStore
> > > Persistence.
> > > > > >
> > > > > > If a user decides to leverage the "native" Polaris (scan/commit)
> > > > Metrics
> > > > > > Persistence, I do not see any disadvantage in also exposing an
> > > > (optional)
> > > > > > REST API for loading these metrics from Polaris Persistence.
> > > > > >
> > > > > > The degree of support and sophistication that goes into this
> > > sub-system
> > > > > is
> > > > > > up to the community. If we have contributors (like @obelix74 )
> who
> > > are
> > > > > > willing to evolve it, I see no harm in some functionality overlap
> > > with
> > > > > more
> > > > > > focused metrics platforms. Again, the key point is for all users
> to
> > > > have
> > > > > > control and be able to opt in or out of this feature in their
> > > specific
> > > > > > deployments.
> > > > > >
> > > > > > Of course, offloading scan/commit metrics storage to a
> specialized
> > > > > > observability system is possible too (assuming someone develops
> > > > > integration
> > > > > > code for that, which is very welcome).
> > > > >
> > > > >
> > > > > [1]
> > https://github.com/apache/polaris/pull/3924#discussion_r2913947744
> > > > >
> > > > > Thanks,
> > > > > Dmitri.
> > > > >
> > > > > On Tue, Mar 10, 2026 at 12:37 PM Anand Kumar Sankaran via dev <
> > > > > dev@polaris.apache.org> wrote:
> > > > >
> > > > > > Hi EJ Wang and Dmitri,
> > > > > >
> > > > > > I addressed all your concerns about the proposal, in particular
> > > > > >
> https://github.com/apache/polaris/pull/3924#discussion_r2908317696
> > .
> > > > > >
> > > > > > Does this address your concerns?
> > > > > >
> > > > > > -
> > > > > > Anand
> > > > > >
> > > > > > From: Dmitri Bourlatchkov <di...@apache.org>
> > > > > > Date: Monday, March 9, 2026 at 1:17 PM
> > > > > > To: dev@polaris.apache.org <de...@polaris.apache.org>
> > > > > > Subject: Re: Proposal for REST endpoints for table metrics and
> > events
> > > > > >
> > > > > > This Message Is From an External Sender
> > > > > > This message came from outside your organization.
> > > > > > Report Suspicious<
> > > > > >
> > > > >
> > > >
> > >
> >
> https://us-phishalarm-ewt.proofpoint.com/EWT/v1/Iz9xO38YGHZK!YhNDZAGr2cumYdtJ_UTm0gR9PfI_-PwSpR_GtNr1uVQ_xo-s2AskvUmbkLZ-C5V8eOKN-omus47On4k4hfFo-0G7CHMLwVjEego-rZrPuepAybX7DP8Ua0VNSrsZ83C4$
> > > > > > >
> > > > > >
> > > > > >
> > > > > > Hi EJ,
> > > > > >
> > > > > > You make good points about the metrics API extensibility and
> > > evolution.
> > > > > >
> > > > > > However, we need to consider practical aspects too. Anand appears
> > to
> > > > have
> > > > > > some specific use cases in mind, and I assume his proposal
> > addresses
> > > > > them.
> > > > > >
> > > > > > Starting with an API + implementation that works for some real
> > > > > > world applications will validate the feature's usability.
> > > > > >
> > > > > > We can revamp the API completely in its v2 after v1 is merged.
> New
> > > > major
> > > > > > API versions do not have to be backward-compatible with older
> > > versions
> > > > of
> > > > > > the same API [1].
> > > > > >
> > > > > > In my personal experience, a v1 API can hardly be expected to
> cover
> > > all
> > > > > use
> > > > > > cases and extensions well. We can certainly take a bit more time
> to
> > > > > polish
> > > > > > it, but from my POV it might be best to iterate in terms of API
> > > > versions
> > > > > > rather than on unmerged commits in the initial proposal. Just my
> 2
> > > > cents
> > > > > :)
> > > > > >
> > > > > > That said, we should flag the new APIs in this proposal as
> > "beta"...
> > > at
> > > > > > least initially (which is the usual practice in Polaris).
> > > > > >
> > > > > > > I wonder if it would help to evaluate the Events API and
> Metrics
> > > API
> > > > a
> > > > > > bit more independently.
> > > > > >
> > > > > > That makes sense to me. However, the current proposal progressed
> a
> > > lot
> > > > > > since its initial submission and contained both APIs. I would not
> > > want
> > > > to
> > > > > > lose this momentum.
> > > > > >
> > > > > > It might still be advisable to implement the events and metrics
> > APIs
> > > > > > separately and gather additional feedback at that time.
> > > > > >
> > > > > > [1]
> > > > > >
> > > > >
> > > >
> > >
> >
> https://urldefense.com/v3/__https://polaris.apache.org/in-dev/unreleased/evolution/__;!!Iz9xO38YGHZK!8KJ0uv4jK3mxZP4nYFrL1hZ0fMkQvoVEAJa8t9LBCzVtm_PWVFGQfIcZp-ykn3_F9_ph6EYyu3dUZjPAcQ$
> > > > > >
> > > > > > Cheers,
> > > > > > Dmitri.
> > > > > >
> > > > > > On Mon, Mar 9, 2026 at 3:48 PM EJ Wang <
> > > ej.wang.dev.interest@gmail.com
> > > > >
> > > > > > wrote:
> > > > > >
> > > > > > > Hi Anand,
> > > > > > >
> > > > > > > I think the proposal is moving in a better direction,
> especially
> > on
> > > > the
> > > > > > > Events side, and I appreciate the iteration so far. That said,
> I
> > > > still
> > > > > > have
> > > > > > > some concerns about the Metrics side, but they are less about
> > > > > individual
> > > > > > > parameters or endpoint shape, and more about product boundary.
> > > > > > >
> > > > > > > 2 cents: I wonder if it would help to evaluate the Events API
> and
> > > > > Metrics
> > > > > > > API a bit more independently.
> > > > > > >
> > > > > > > The Events side feels relatively close to Polaris'
> > > catalog/change-log
> > > > > > > scope. It is easier to justify as part of the core/community
> > > surface,
> > > > > > > especially if the goal is to expose completed catalog mutations
> > in
> > > a
> > > > > way
> > > > > > > that aligns with Iceberg-style events.
> > > > > > >
> > > > > > > The Metrics side feels different to me. Once we start adding
> more
> > > and
> > > > > > more
> > > > > > > type-specific filters, query semantics, and schema shape for
> > > > individual
> > > > > > > metric families, it seems easy for Polaris to drift toward a
> > > built-in
> > > > > > > observability backend. My bias would be for Polaris to support
> a
> > > > > smaller
> > > > > > > set of community-recognized built-in metrics well, while
> > providing
> > > > good
> > > > > > > extensibility points for deployments that want richer querying,
> > > > > > > visualization, or use-case-specific metrics.
> > > > > > >
> > > > > > > Related to that, I am not yet convinced the current metrics
> model
> > > is
> > > > > > > generic enough as a long-term direction. Even after
> consolidating
> > > to
> > > > a
> > > > > > > single endpoint, the design still feels fairly tied to the
> > current
> > > > > > > scan/commit shape. I worry that otherwise each new metric
> family
> > > will
> > > > > > keep
> > > > > > > pulling us into more storage/schema/API reshaping inside
> Polaris
> > > > core.
> > > > > > > So the framing question I would suggest is something like:
> > > > > > > > What is the minimal built-in metrics surface Polaris should
> own
> > > in
> > > > > > core,
> > > > > > > and where should we instead rely on extensibility /
> sink-export /
> > > > > > > plugin-style integration?
> > > > > > >
> > > > > > > For me, getting that boundary right matters more than settling
> > > every
> > > > > > query
> > > > > > > parameter detail first.
> > > > > > >
> > > > > > > -ej
> > > > > > >
> > > > > > > On Tue, Mar 3, 2026 at 12:29 PM Anand Kumar Sankaran via dev <
> > > > > > > dev@polaris.apache.org> wrote:
> > > > > > >
> > > > > > > > Hi Yufei and Dmitri,
> > > > > > > >
> > > > > > > > Here is a proposal for the REST endpoints for metrics and
> > events.
> > > > > > > >
> > > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://urldefense.com/v3/__https://github.com/apache/polaris/pull/3924/changes__;!!Iz9xO38YGHZK!8KJ0uv4jK3mxZP4nYFrL1hZ0fMkQvoVEAJa8t9LBCzVtm_PWVFGQfIcZp-ykn3_F9_ph6EYyu3fXDjRYWA$
> > > > > > > >
> > > > > > > > I did not see any precursors for raising a PR for proposals,
> so
> > > > > trying
> > > > > > > > this.  Please let me know what you think.
> > > > > > > >
> > > > > > > > -
> > > > > > > > Anand
> > > > > > > >
> > > > > > > > From: Anand Kumar Sankaran <an...@workday.com>
> > > > > > > > Date: Monday, March 2, 2026 at 10:25 AM
> > > > > > > > To: dev@polaris.apache.org <de...@polaris.apache.org>
> > > > > > > > Subject: Re: Polaris Telemetry and Audit Trail
> > > > > > > >
> > > > > > > > About the REST API, based on my use cases:
> > > > > > > >
> > > > > > > >
> > > > > > > >   1.
> > > > > > > > I want to be able to query commit metrics to track files
> added
> > /
> > > > > > removed
> > > > > > > > per commit, along with record counts. The ingestion pipeline
> > that
> > > > > > writes
> > > > > > > > this data is owned by us and we are guaranteed to write this
> > > > > > information
> > > > > > > > for each write.
> > > > > > > >   2.
> > > > > > > > I want to be able to query scan metrics for read. I
> understand
> > > > > clients
> > > > > > do
> > > > > > > > not fulfill this requirement.
> > > > > > > >   3.
> > > > > > > > I want to be able to query the events table (events are
> > > persisted)
> > > > -
> > > > > > this
> > > > > > > > may supersede #2, I am not sure yet.
> > > > > > > >
> > > > > > > > All this information is in the JDBC based persistence model
> and
> > > is
> > > > > > > > persisted in the metastore. I currently don’t have a need to
> > > query
> > > > > > > > prometheus or open telemetry. I do publish some events to
> > > > Prometheus
> > > > > > and
> > > > > > > > they are forwarded to our dashboards elsewhere.
> > > > > > > >
> > > > > > > > About the CLI utilities, I meant the admin user utilities. In
> > one
> > > > of
> > > > > > the
> > > > > > > > earliest drafts of my proposal, Prashant mentioned that the
> > > metrics
> > > > > > > tables
> > > > > > > > can grow indefinitely and that a similar problem exists with
> > the
> > > > > events
> > > > > > > > table as well. We discussed that cleaning up of old records
> > from
> > > > both
> > > > > > > > metrics tables and events tables can be done via a CLI
> utility.
> > > > > > > >
> > > > > > > > I see that Yufei has covered the discussion about
> datasources.
> > > > > > > >
> > > > > > > > -
> > > > > > > > Anand
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > From: Yufei Gu <fl...@gmail.com>
> > > > > > > > Date: Friday, February 27, 2026 at 9:54 PM
> > > > > > > > To: dev@polaris.apache.org <de...@polaris.apache.org>
> > > > > > > > Subject: Re: Polaris Telemetry and Audit Trail
> > > > > > > >
> > > > > > > > This Message Is From an External Sender
> > > > > > > > This message came from outside your organization.
> > > > > > > > Report Suspicious<
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://us-phishalarm-ewt.proofpoint.com/EWT/v1/Iz9xO38YGHZK!YhNDZABkHi1B699ote2uMwpOZw8i0QMCGO2Szc-HshuABGhGvwPJcymE6G2oUUxtS8xDkSrtGTPm_I3QnVDHoLMk50m9v8z_nZKTkd-bnVUbreF1u0WnfV_X5eYevZl_$
> > > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > As I mentioned in
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://urldefense.com/v3/__https://github.com/apache/polaris/issues/3890__;!!Iz9xO38YGHZK!5EuyFFkk3vhRWVIRvQAWBSQfpJkTMA9HxugzDwXmN0LPPqhEFxYkFRGVhtb8AqUwXtDh2OplcMnbMDHKOxrvDU0$
> > > > > > >,
> > > > > > > > supporting
> > > > > > > > multiple data sources is not a trivial change. I would
> strongly
> > > > > > recommend
> > > > > > > > starting with a design document to carefully evaluate the
> > > > > architectural
> > > > > > > > implications and long term impact.
> > > > > > > >
> > > > > > > > A REST endpoint to query metrics seems reasonable given the
> > > current
> > > > > > JDBC
> > > > > > > > based persistence model. That said, we may also consider
> > > > alternative
> > > > > > > > storage models. For example, if we later adopt a time series
> > > system
> > > > > > such
> > > > > > > as
> > > > > > > > Prometheus to store metrics, the query model and access
> > patterns
> > > > > would
> > > > > > be
> > > > > > > > fundamentally different. Designing the REST API without
> > > considering
> > > > > > these
> > > > > > > > potential evolutions may limit flexibility. I'd suggest to
> > start
> > > > with
> > > > > > the
> > > > > > > > use case.
> > > > > > > >
> > > > > > > > Yufei
> > > > > > > >
> > > > > > > >
> > > > > > > > On Fri, Feb 27, 2026 at 3:42 PM Dmitri Bourlatchkov <
> > > > > dimas@apache.org>
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > > > Hi Anand,
> > > > > > > > >
> > > > > > > > > Sharing my view... subject to discussion:
> > > > > > > > >
> > > > > > > > > 1. Adding non-IRC REST API to Polaris is perfectly fine.
> > > > > > > > >
> > > > > > > > > Figuring out specific endpoint URIs and payloads might
> > require
> > > a
> > > > > few
> > > > > > > > > roundtrips, so opening a separate thread for that might be
> > > best.
> > > > > > > > > Contributors commonly create Google Docs for new API
> > proposals
> > > > too
> > > > > > > (they
> > > > > > > > > fairly easy to update as the email discussion progresses).
> > > > > > > > >
> > > > > > > > > There was a suggestion to try Markdown (with PRs) for
> > proposals
> > > > [1]
> > > > > > ...
> > > > > > > > > feel free to give it a try if you are comfortable with
> that.
> > > > > > > > >
> > > > > > > > > 2. Could you clarify whether you mean end user utilities or
> > > admin
> > > > > > user
> > > > > > > > > utilities? In the latter case those might be more suitable
> > for
> > > > the
> > > > > > > Admin
> > > > > > > > > CLI (java) not the Python CLI, IMHO.
> > > > > > > > >
> > > > > > > > > Why would these utilities be common with events? IMHO,
> event
> > > use
> > > > > > cases
> > > > > > > > are
> > > > > > > > > distinct from scan/commit metrics.
> > > > > > > > >
> > > > > > > > > 3. I'd prefer separating metrics persistence from MetaStore
> > > > > > persistence
> > > > > > > > at
> > > > > > > > > the code level, so that they could be mixed and matched
> > > > > > independently.
> > > > > > > > The
> > > > > > > > > separate datasource question will become a non-issue with
> > that
> > > > > > > approach,
> > > > > > > > I
> > > > > > > > > guess.
> > > > > > > > >
> > > > > > > > > The rationale for separating scan metrics and metastore
> > > > persistence
> > > > > > is
> > > > > > > > that
> > > > > > > > > "cascading deletes" between them are hardly ever required.
> > > > > > Furthermore,
> > > > > > > > the
> > > > > > > > > data and query patterns are very different so different
> > > > > technologies
> > > > > > > > might
> > > > > > > > > be beneficial in each case.
> > > > > > > > >
> > > > > > > > > [1]
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://urldefense.com/v3/__https://lists.apache.org/thread/yto2wp982t43h1mqjwnslswhws5z47cy__;!!Iz9xO38YGHZK!5EuyFFkk3vhRWVIRvQAWBSQfpJkTMA9HxugzDwXmN0LPPqhEFxYkFRGVhtb8AqUwXtDh2OplcMnbMDHKxYDakNU$
> > > > > > >> >
> > > > > > > > > Cheers,
> > > > > > > > > Dmitri.
> > > > > > > > >
> > > > > > > > > On Fri, Feb 27, 2026 at 6:19 PM Anand Kumar Sankaran via
> dev
> > <
> > > > > > > > > dev@polaris.apache.org> wrote:
> > > > > > > > >
> > > > > > > > > > Thanks all. This PR is merged now.
> > > > > > > > > >
> > > > > > > > > > Here are the follow-up features / work needed.  These
> were
> > > all
> > > > > part
> > > > > > > of
> > > > > > > > > the
> > > > > > > > > > merged PR at some point in time and were removed to
> reduce
> > > > scope.
> > > > > > > > > >
> > > > > > > > > > Please let me know what you think.
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >   1.  A REST API to paginate through table metrics. This
> > will
> > > > be
> > > > > > > > non-IRC
> > > > > > > > > > standard addition.
> > > > > > > > > >   2.  Utilities for managing old records, should be
> common
> > > with
> > > > > > > events.
> > > > > > > > > > There was some discussion that it belongs to the CLI.
> > > > > > > > > >   3.  Separate datasource (metrics, events, even other
> > > > tables?).
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > Anything else?
> > > > > > > > > >
> > > > > > > > > > -
> > > > > > > > > > Anand
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>

Re: Proposal for REST endpoints for table metrics and events

[Dmitri Bourlatchkov <di...@apache.org>]

Hi All,

Polaris can support both a local API and new endpoints in the IRC API once
the Iceberg community adopts the latter.

What is the concern with having different APIs to access the same data?

Cheers,
Dmitri.

On Tue, Mar 17, 2026 at 3:49 PM EJ Wang <ej...@gmail.com>
wrote:

> I share the same feeling with Yifei as reviewing the PRs. I want to avoid
> creating discrepancies if the IRC side ended up supporting it.
>
> -ej
>
> On Wed, Mar 11, 2026 at 2:47 PM Yufei Gu <fl...@gmail.com> wrote:
>
> > Thanks Anand for working on this.
> >
> > Given the IRC event endpoint is WIP, I think it'd be best to be
> consistent
> > with the IRC endpoint. For more context on the IRC event, the event
> > endpoint is not finalized yet. I'd recommend speeding up the IRC side
> work
> > to avoid any inconsistencies between Polaris and IRC spec.
> >
> > I have a few questions about the metrics endpoint:
> > 1. Do we need to expose them via Polaris REST endpoint? Can users grab
> the
> > metrics from the backend directly? I understand the RBAC won't be there,
> > but it provides flexibility for users. For example, some users may
> choose a
> > different persistence model such as a KV store or storing the metrics as
> > objects in S3, which usually scales better than an RDBMS like Postgres.
> My
> > understanding is that Polaris is not intended to be a full metrics system
> > anyway, but rather to provide a way for downstream systems to consume
> these
> > data.
> > 2. Should we consider it as an IRC endpoint? Given the metrics report
> > endpoint in IRC, the Iceberg community might also be interested in
> serving
> > back metrics, similar to the event endpoint. In that case, there is a
> risk
> > of fragmentation if we create a Polaris endpoint now. We should avoid
> that
> > if possible. It might be worth checking with the Iceberg community first.
> >
> > Happy to hear others’ thoughts on this.
> >
> > Yufei
> >
> >
> > On Wed, Mar 11, 2026 at 9:27 AM Nándor Kollár <nk...@apache.org>
> wrote:
> >
> > > I agree with Dmitri. The direction outlined in the proposal looks good
> to
> > > me, and the finer details can be worked out as implementation gets
> > > underway. We can adjust the design doc accordingly, later on.
> > >
> > >
> > > Thanks,
> > > Nandor
> > >
> > > Dmitri Bourlatchkov <di...@apache.org> ezt írta (időpont: 2026. márc.
> > 10.,
> > > K, 20:43):
> > >
> > > > Hi Anand and All,
> > > >
> > > > The proposal LGTM in its current form.
> > > >
> > > > My personal approach is that a proposal does not have to be as
> > "polished"
> > > > as the final API spec. As long as we have consensus on the general
> > > > approach and the basic API principles, I think we can proceed to
> > > > implementation and iron out final wrinkles during actual API spec and
> > > code
> > > > PRs.
> > > >
> > > > Would this approach work for everyone?
> > > >
> > > > Re-posting my comment from GH [1] here for visibility, in case people
> > > have
> > > > different opinions and wish to discuss this in more depth:
> > > >
> > > > From my POV, Polaris is a platform, indeed. In this sense, I think it
> > is
> > > > > critical to enable users to control what features are at play in
> > > runtime,
> > > > > since different users have different use cases. This is why I
> > > originally
> > > > > advocated for isolating Metrics Persistence from MetaStore
> > Persistence.
> > > > >
> > > > > If a user decides to leverage the "native" Polaris (scan/commit)
> > > Metrics
> > > > > Persistence, I do not see any disadvantage in also exposing an
> > > (optional)
> > > > > REST API for loading these metrics from Polaris Persistence.
> > > > >
> > > > > The degree of support and sophistication that goes into this
> > sub-system
> > > > is
> > > > > up to the community. If we have contributors (like @obelix74 ) who
> > are
> > > > > willing to evolve it, I see no harm in some functionality overlap
> > with
> > > > more
> > > > > focused metrics platforms. Again, the key point is for all users to
> > > have
> > > > > control and be able to opt in or out of this feature in their
> > specific
> > > > > deployments.
> > > > >
> > > > > Of course, offloading scan/commit metrics storage to a specialized
> > > > > observability system is possible too (assuming someone develops
> > > > integration
> > > > > code for that, which is very welcome).
> > > >
> > > >
> > > > [1]
> https://github.com/apache/polaris/pull/3924#discussion_r2913947744
> > > >
> > > > Thanks,
> > > > Dmitri.
> > > >
> > > > On Tue, Mar 10, 2026 at 12:37 PM Anand Kumar Sankaran via dev <
> > > > dev@polaris.apache.org> wrote:
> > > >
> > > > > Hi EJ Wang and Dmitri,
> > > > >
> > > > > I addressed all your concerns about the proposal, in particular
> > > > > https://github.com/apache/polaris/pull/3924#discussion_r2908317696
> .
> > > > >
> > > > > Does this address your concerns?
> > > > >
> > > > > -
> > > > > Anand
> > > > >
> > > > > From: Dmitri Bourlatchkov <di...@apache.org>
> > > > > Date: Monday, March 9, 2026 at 1:17 PM
> > > > > To: dev@polaris.apache.org <de...@polaris.apache.org>
> > > > > Subject: Re: Proposal for REST endpoints for table metrics and
> events
> > > > >
> > > > > This Message Is From an External Sender
> > > > > This message came from outside your organization.
> > > > > Report Suspicious<
> > > > >
> > > >
> > >
> >
> https://us-phishalarm-ewt.proofpoint.com/EWT/v1/Iz9xO38YGHZK!YhNDZAGr2cumYdtJ_UTm0gR9PfI_-PwSpR_GtNr1uVQ_xo-s2AskvUmbkLZ-C5V8eOKN-omus47On4k4hfFo-0G7CHMLwVjEego-rZrPuepAybX7DP8Ua0VNSrsZ83C4$
> > > > > >
> > > > >
> > > > >
> > > > > Hi EJ,
> > > > >
> > > > > You make good points about the metrics API extensibility and
> > evolution.
> > > > >
> > > > > However, we need to consider practical aspects too. Anand appears
> to
> > > have
> > > > > some specific use cases in mind, and I assume his proposal
> addresses
> > > > them.
> > > > >
> > > > > Starting with an API + implementation that works for some real
> > > > > world applications will validate the feature's usability.
> > > > >
> > > > > We can revamp the API completely in its v2 after v1 is merged. New
> > > major
> > > > > API versions do not have to be backward-compatible with older
> > versions
> > > of
> > > > > the same API [1].
> > > > >
> > > > > In my personal experience, a v1 API can hardly be expected to cover
> > all
> > > > use
> > > > > cases and extensions well. We can certainly take a bit more time to
> > > > polish
> > > > > it, but from my POV it might be best to iterate in terms of API
> > > versions
> > > > > rather than on unmerged commits in the initial proposal. Just my 2
> > > cents
> > > > :)
> > > > >
> > > > > That said, we should flag the new APIs in this proposal as
> "beta"...
> > at
> > > > > least initially (which is the usual practice in Polaris).
> > > > >
> > > > > > I wonder if it would help to evaluate the Events API and Metrics
> > API
> > > a
> > > > > bit more independently.
> > > > >
> > > > > That makes sense to me. However, the current proposal progressed a
> > lot
> > > > > since its initial submission and contained both APIs. I would not
> > want
> > > to
> > > > > lose this momentum.
> > > > >
> > > > > It might still be advisable to implement the events and metrics
> APIs
> > > > > separately and gather additional feedback at that time.
> > > > >
> > > > > [1]
> > > > >
> > > >
> > >
> >
> https://urldefense.com/v3/__https://polaris.apache.org/in-dev/unreleased/evolution/__;!!Iz9xO38YGHZK!8KJ0uv4jK3mxZP4nYFrL1hZ0fMkQvoVEAJa8t9LBCzVtm_PWVFGQfIcZp-ykn3_F9_ph6EYyu3dUZjPAcQ$
> > > > >
> > > > > Cheers,
> > > > > Dmitri.
> > > > >
> > > > > On Mon, Mar 9, 2026 at 3:48 PM EJ Wang <
> > ej.wang.dev.interest@gmail.com
> > > >
> > > > > wrote:
> > > > >
> > > > > > Hi Anand,
> > > > > >
> > > > > > I think the proposal is moving in a better direction, especially
> on
> > > the
> > > > > > Events side, and I appreciate the iteration so far. That said, I
> > > still
> > > > > have
> > > > > > some concerns about the Metrics side, but they are less about
> > > > individual
> > > > > > parameters or endpoint shape, and more about product boundary.
> > > > > >
> > > > > > 2 cents: I wonder if it would help to evaluate the Events API and
> > > > Metrics
> > > > > > API a bit more independently.
> > > > > >
> > > > > > The Events side feels relatively close to Polaris'
> > catalog/change-log
> > > > > > scope. It is easier to justify as part of the core/community
> > surface,
> > > > > > especially if the goal is to expose completed catalog mutations
> in
> > a
> > > > way
> > > > > > that aligns with Iceberg-style events.
> > > > > >
> > > > > > The Metrics side feels different to me. Once we start adding more
> > and
> > > > > more
> > > > > > type-specific filters, query semantics, and schema shape for
> > > individual
> > > > > > metric families, it seems easy for Polaris to drift toward a
> > built-in
> > > > > > observability backend. My bias would be for Polaris to support a
> > > > smaller
> > > > > > set of community-recognized built-in metrics well, while
> providing
> > > good
> > > > > > extensibility points for deployments that want richer querying,
> > > > > > visualization, or use-case-specific metrics.
> > > > > >
> > > > > > Related to that, I am not yet convinced the current metrics model
> > is
> > > > > > generic enough as a long-term direction. Even after consolidating
> > to
> > > a
> > > > > > single endpoint, the design still feels fairly tied to the
> current
> > > > > > scan/commit shape. I worry that otherwise each new metric family
> > will
> > > > > keep
> > > > > > pulling us into more storage/schema/API reshaping inside Polaris
> > > core.
> > > > > > So the framing question I would suggest is something like:
> > > > > > > What is the minimal built-in metrics surface Polaris should own
> > in
> > > > > core,
> > > > > > and where should we instead rely on extensibility / sink-export /
> > > > > > plugin-style integration?
> > > > > >
> > > > > > For me, getting that boundary right matters more than settling
> > every
> > > > > query
> > > > > > parameter detail first.
> > > > > >
> > > > > > -ej
> > > > > >
> > > > > > On Tue, Mar 3, 2026 at 12:29 PM Anand Kumar Sankaran via dev <
> > > > > > dev@polaris.apache.org> wrote:
> > > > > >
> > > > > > > Hi Yufei and Dmitri,
> > > > > > >
> > > > > > > Here is a proposal for the REST endpoints for metrics and
> events.
> > > > > > >
> > > > > > >
> > > > >
> > > >
> > >
> >
> https://urldefense.com/v3/__https://github.com/apache/polaris/pull/3924/changes__;!!Iz9xO38YGHZK!8KJ0uv4jK3mxZP4nYFrL1hZ0fMkQvoVEAJa8t9LBCzVtm_PWVFGQfIcZp-ykn3_F9_ph6EYyu3fXDjRYWA$
> > > > > > >
> > > > > > > I did not see any precursors for raising a PR for proposals, so
> > > > trying
> > > > > > > this.  Please let me know what you think.
> > > > > > >
> > > > > > > -
> > > > > > > Anand
> > > > > > >
> > > > > > > From: Anand Kumar Sankaran <an...@workday.com>
> > > > > > > Date: Monday, March 2, 2026 at 10:25 AM
> > > > > > > To: dev@polaris.apache.org <de...@polaris.apache.org>
> > > > > > > Subject: Re: Polaris Telemetry and Audit Trail
> > > > > > >
> > > > > > > About the REST API, based on my use cases:
> > > > > > >
> > > > > > >
> > > > > > >   1.
> > > > > > > I want to be able to query commit metrics to track files added
> /
> > > > > removed
> > > > > > > per commit, along with record counts. The ingestion pipeline
> that
> > > > > writes
> > > > > > > this data is owned by us and we are guaranteed to write this
> > > > > information
> > > > > > > for each write.
> > > > > > >   2.
> > > > > > > I want to be able to query scan metrics for read. I understand
> > > > clients
> > > > > do
> > > > > > > not fulfill this requirement.
> > > > > > >   3.
> > > > > > > I want to be able to query the events table (events are
> > persisted)
> > > -
> > > > > this
> > > > > > > may supersede #2, I am not sure yet.
> > > > > > >
> > > > > > > All this information is in the JDBC based persistence model and
> > is
> > > > > > > persisted in the metastore. I currently don’t have a need to
> > query
> > > > > > > prometheus or open telemetry. I do publish some events to
> > > Prometheus
> > > > > and
> > > > > > > they are forwarded to our dashboards elsewhere.
> > > > > > >
> > > > > > > About the CLI utilities, I meant the admin user utilities. In
> one
> > > of
> > > > > the
> > > > > > > earliest drafts of my proposal, Prashant mentioned that the
> > metrics
> > > > > > tables
> > > > > > > can grow indefinitely and that a similar problem exists with
> the
> > > > events
> > > > > > > table as well. We discussed that cleaning up of old records
> from
> > > both
> > > > > > > metrics tables and events tables can be done via a CLI utility.
> > > > > > >
> > > > > > > I see that Yufei has covered the discussion about datasources.
> > > > > > >
> > > > > > > -
> > > > > > > Anand
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > From: Yufei Gu <fl...@gmail.com>
> > > > > > > Date: Friday, February 27, 2026 at 9:54 PM
> > > > > > > To: dev@polaris.apache.org <de...@polaris.apache.org>
> > > > > > > Subject: Re: Polaris Telemetry and Audit Trail
> > > > > > >
> > > > > > > This Message Is From an External Sender
> > > > > > > This message came from outside your organization.
> > > > > > > Report Suspicious<
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://us-phishalarm-ewt.proofpoint.com/EWT/v1/Iz9xO38YGHZK!YhNDZABkHi1B699ote2uMwpOZw8i0QMCGO2Szc-HshuABGhGvwPJcymE6G2oUUxtS8xDkSrtGTPm_I3QnVDHoLMk50m9v8z_nZKTkd-bnVUbreF1u0WnfV_X5eYevZl_$
> > > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > As I mentioned in
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://urldefense.com/v3/__https://github.com/apache/polaris/issues/3890__;!!Iz9xO38YGHZK!5EuyFFkk3vhRWVIRvQAWBSQfpJkTMA9HxugzDwXmN0LPPqhEFxYkFRGVhtb8AqUwXtDh2OplcMnbMDHKOxrvDU0$
> > > > > >,
> > > > > > > supporting
> > > > > > > multiple data sources is not a trivial change. I would strongly
> > > > > recommend
> > > > > > > starting with a design document to carefully evaluate the
> > > > architectural
> > > > > > > implications and long term impact.
> > > > > > >
> > > > > > > A REST endpoint to query metrics seems reasonable given the
> > current
> > > > > JDBC
> > > > > > > based persistence model. That said, we may also consider
> > > alternative
> > > > > > > storage models. For example, if we later adopt a time series
> > system
> > > > > such
> > > > > > as
> > > > > > > Prometheus to store metrics, the query model and access
> patterns
> > > > would
> > > > > be
> > > > > > > fundamentally different. Designing the REST API without
> > considering
> > > > > these
> > > > > > > potential evolutions may limit flexibility. I'd suggest to
> start
> > > with
> > > > > the
> > > > > > > use case.
> > > > > > >
> > > > > > > Yufei
> > > > > > >
> > > > > > >
> > > > > > > On Fri, Feb 27, 2026 at 3:42 PM Dmitri Bourlatchkov <
> > > > dimas@apache.org>
> > > > > > > wrote:
> > > > > > >
> > > > > > > > Hi Anand,
> > > > > > > >
> > > > > > > > Sharing my view... subject to discussion:
> > > > > > > >
> > > > > > > > 1. Adding non-IRC REST API to Polaris is perfectly fine.
> > > > > > > >
> > > > > > > > Figuring out specific endpoint URIs and payloads might
> require
> > a
> > > > few
> > > > > > > > roundtrips, so opening a separate thread for that might be
> > best.
> > > > > > > > Contributors commonly create Google Docs for new API
> proposals
> > > too
> > > > > > (they
> > > > > > > > fairly easy to update as the email discussion progresses).
> > > > > > > >
> > > > > > > > There was a suggestion to try Markdown (with PRs) for
> proposals
> > > [1]
> > > > > ...
> > > > > > > > feel free to give it a try if you are comfortable with that.
> > > > > > > >
> > > > > > > > 2. Could you clarify whether you mean end user utilities or
> > admin
> > > > > user
> > > > > > > > utilities? In the latter case those might be more suitable
> for
> > > the
> > > > > > Admin
> > > > > > > > CLI (java) not the Python CLI, IMHO.
> > > > > > > >
> > > > > > > > Why would these utilities be common with events? IMHO, event
> > use
> > > > > cases
> > > > > > > are
> > > > > > > > distinct from scan/commit metrics.
> > > > > > > >
> > > > > > > > 3. I'd prefer separating metrics persistence from MetaStore
> > > > > persistence
> > > > > > > at
> > > > > > > > the code level, so that they could be mixed and matched
> > > > > independently.
> > > > > > > The
> > > > > > > > separate datasource question will become a non-issue with
> that
> > > > > > approach,
> > > > > > > I
> > > > > > > > guess.
> > > > > > > >
> > > > > > > > The rationale for separating scan metrics and metastore
> > > persistence
> > > > > is
> > > > > > > that
> > > > > > > > "cascading deletes" between them are hardly ever required.
> > > > > Furthermore,
> > > > > > > the
> > > > > > > > data and query patterns are very different so different
> > > > technologies
> > > > > > > might
> > > > > > > > be beneficial in each case.
> > > > > > > >
> > > > > > > > [1]
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://urldefense.com/v3/__https://lists.apache.org/thread/yto2wp982t43h1mqjwnslswhws5z47cy__;!!Iz9xO38YGHZK!5EuyFFkk3vhRWVIRvQAWBSQfpJkTMA9HxugzDwXmN0LPPqhEFxYkFRGVhtb8AqUwXtDh2OplcMnbMDHKxYDakNU$
> > > > > >> >
> > > > > > > > Cheers,
> > > > > > > > Dmitri.
> > > > > > > >
> > > > > > > > On Fri, Feb 27, 2026 at 6:19 PM Anand Kumar Sankaran via dev
> <
> > > > > > > > dev@polaris.apache.org> wrote:
> > > > > > > >
> > > > > > > > > Thanks all. This PR is merged now.
> > > > > > > > >
> > > > > > > > > Here are the follow-up features / work needed.  These were
> > all
> > > > part
> > > > > > of
> > > > > > > > the
> > > > > > > > > merged PR at some point in time and were removed to reduce
> > > scope.
> > > > > > > > >
> > > > > > > > > Please let me know what you think.
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >   1.  A REST API to paginate through table metrics. This
> will
> > > be
> > > > > > > non-IRC
> > > > > > > > > standard addition.
> > > > > > > > >   2.  Utilities for managing old records, should be common
> > with
> > > > > > events.
> > > > > > > > > There was some discussion that it belongs to the CLI.
> > > > > > > > >   3.  Separate datasource (metrics, events, even other
> > > tables?).
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > Anything else?
> > > > > > > > >
> > > > > > > > > -
> > > > > > > > > Anand
> > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > > >
> > > >
> > >
> >
>

Re: Proposal for REST endpoints for table metrics and events

[EJ Wang <ej...@gmail.com>]

I share the same feeling with Yifei as reviewing the PRs. I want to avoid
creating discrepancies if the IRC side ended up supporting it.

-ej

On Wed, Mar 11, 2026 at 2:47 PM Yufei Gu <fl...@gmail.com> wrote:

> Thanks Anand for working on this.
>
> Given the IRC event endpoint is WIP, I think it'd be best to be consistent
> with the IRC endpoint. For more context on the IRC event, the event
> endpoint is not finalized yet. I'd recommend speeding up the IRC side work
> to avoid any inconsistencies between Polaris and IRC spec.
>
> I have a few questions about the metrics endpoint:
> 1. Do we need to expose them via Polaris REST endpoint? Can users grab the
> metrics from the backend directly? I understand the RBAC won't be there,
> but it provides flexibility for users. For example, some users may choose a
> different persistence model such as a KV store or storing the metrics as
> objects in S3, which usually scales better than an RDBMS like Postgres. My
> understanding is that Polaris is not intended to be a full metrics system
> anyway, but rather to provide a way for downstream systems to consume these
> data.
> 2. Should we consider it as an IRC endpoint? Given the metrics report
> endpoint in IRC, the Iceberg community might also be interested in serving
> back metrics, similar to the event endpoint. In that case, there is a risk
> of fragmentation if we create a Polaris endpoint now. We should avoid that
> if possible. It might be worth checking with the Iceberg community first.
>
> Happy to hear others’ thoughts on this.
>
> Yufei
>
>
> On Wed, Mar 11, 2026 at 9:27 AM Nándor Kollár <nk...@apache.org> wrote:
>
> > I agree with Dmitri. The direction outlined in the proposal looks good to
> > me, and the finer details can be worked out as implementation gets
> > underway. We can adjust the design doc accordingly, later on.
> >
> >
> > Thanks,
> > Nandor
> >
> > Dmitri Bourlatchkov <di...@apache.org> ezt írta (időpont: 2026. márc.
> 10.,
> > K, 20:43):
> >
> > > Hi Anand and All,
> > >
> > > The proposal LGTM in its current form.
> > >
> > > My personal approach is that a proposal does not have to be as
> "polished"
> > > as the final API spec. As long as we have consensus on the general
> > > approach and the basic API principles, I think we can proceed to
> > > implementation and iron out final wrinkles during actual API spec and
> > code
> > > PRs.
> > >
> > > Would this approach work for everyone?
> > >
> > > Re-posting my comment from GH [1] here for visibility, in case people
> > have
> > > different opinions and wish to discuss this in more depth:
> > >
> > > From my POV, Polaris is a platform, indeed. In this sense, I think it
> is
> > > > critical to enable users to control what features are at play in
> > runtime,
> > > > since different users have different use cases. This is why I
> > originally
> > > > advocated for isolating Metrics Persistence from MetaStore
> Persistence.
> > > >
> > > > If a user decides to leverage the "native" Polaris (scan/commit)
> > Metrics
> > > > Persistence, I do not see any disadvantage in also exposing an
> > (optional)
> > > > REST API for loading these metrics from Polaris Persistence.
> > > >
> > > > The degree of support and sophistication that goes into this
> sub-system
> > > is
> > > > up to the community. If we have contributors (like @obelix74 ) who
> are
> > > > willing to evolve it, I see no harm in some functionality overlap
> with
> > > more
> > > > focused metrics platforms. Again, the key point is for all users to
> > have
> > > > control and be able to opt in or out of this feature in their
> specific
> > > > deployments.
> > > >
> > > > Of course, offloading scan/commit metrics storage to a specialized
> > > > observability system is possible too (assuming someone develops
> > > integration
> > > > code for that, which is very welcome).
> > >
> > >
> > > [1] https://github.com/apache/polaris/pull/3924#discussion_r2913947744
> > >
> > > Thanks,
> > > Dmitri.
> > >
> > > On Tue, Mar 10, 2026 at 12:37 PM Anand Kumar Sankaran via dev <
> > > dev@polaris.apache.org> wrote:
> > >
> > > > Hi EJ Wang and Dmitri,
> > > >
> > > > I addressed all your concerns about the proposal, in particular
> > > > https://github.com/apache/polaris/pull/3924#discussion_r2908317696.
> > > >
> > > > Does this address your concerns?
> > > >
> > > > -
> > > > Anand
> > > >
> > > > From: Dmitri Bourlatchkov <di...@apache.org>
> > > > Date: Monday, March 9, 2026 at 1:17 PM
> > > > To: dev@polaris.apache.org <de...@polaris.apache.org>
> > > > Subject: Re: Proposal for REST endpoints for table metrics and events
> > > >
> > > > This Message Is From an External Sender
> > > > This message came from outside your organization.
> > > > Report Suspicious<
> > > >
> > >
> >
> https://us-phishalarm-ewt.proofpoint.com/EWT/v1/Iz9xO38YGHZK!YhNDZAGr2cumYdtJ_UTm0gR9PfI_-PwSpR_GtNr1uVQ_xo-s2AskvUmbkLZ-C5V8eOKN-omus47On4k4hfFo-0G7CHMLwVjEego-rZrPuepAybX7DP8Ua0VNSrsZ83C4$
> > > > >
> > > >
> > > >
> > > > Hi EJ,
> > > >
> > > > You make good points about the metrics API extensibility and
> evolution.
> > > >
> > > > However, we need to consider practical aspects too. Anand appears to
> > have
> > > > some specific use cases in mind, and I assume his proposal addresses
> > > them.
> > > >
> > > > Starting with an API + implementation that works for some real
> > > > world applications will validate the feature's usability.
> > > >
> > > > We can revamp the API completely in its v2 after v1 is merged. New
> > major
> > > > API versions do not have to be backward-compatible with older
> versions
> > of
> > > > the same API [1].
> > > >
> > > > In my personal experience, a v1 API can hardly be expected to cover
> all
> > > use
> > > > cases and extensions well. We can certainly take a bit more time to
> > > polish
> > > > it, but from my POV it might be best to iterate in terms of API
> > versions
> > > > rather than on unmerged commits in the initial proposal. Just my 2
> > cents
> > > :)
> > > >
> > > > That said, we should flag the new APIs in this proposal as "beta"...
> at
> > > > least initially (which is the usual practice in Polaris).
> > > >
> > > > > I wonder if it would help to evaluate the Events API and Metrics
> API
> > a
> > > > bit more independently.
> > > >
> > > > That makes sense to me. However, the current proposal progressed a
> lot
> > > > since its initial submission and contained both APIs. I would not
> want
> > to
> > > > lose this momentum.
> > > >
> > > > It might still be advisable to implement the events and metrics APIs
> > > > separately and gather additional feedback at that time.
> > > >
> > > > [1]
> > > >
> > >
> >
> https://urldefense.com/v3/__https://polaris.apache.org/in-dev/unreleased/evolution/__;!!Iz9xO38YGHZK!8KJ0uv4jK3mxZP4nYFrL1hZ0fMkQvoVEAJa8t9LBCzVtm_PWVFGQfIcZp-ykn3_F9_ph6EYyu3dUZjPAcQ$
> > > >
> > > > Cheers,
> > > > Dmitri.
> > > >
> > > > On Mon, Mar 9, 2026 at 3:48 PM EJ Wang <
> ej.wang.dev.interest@gmail.com
> > >
> > > > wrote:
> > > >
> > > > > Hi Anand,
> > > > >
> > > > > I think the proposal is moving in a better direction, especially on
> > the
> > > > > Events side, and I appreciate the iteration so far. That said, I
> > still
> > > > have
> > > > > some concerns about the Metrics side, but they are less about
> > > individual
> > > > > parameters or endpoint shape, and more about product boundary.
> > > > >
> > > > > 2 cents: I wonder if it would help to evaluate the Events API and
> > > Metrics
> > > > > API a bit more independently.
> > > > >
> > > > > The Events side feels relatively close to Polaris'
> catalog/change-log
> > > > > scope. It is easier to justify as part of the core/community
> surface,
> > > > > especially if the goal is to expose completed catalog mutations in
> a
> > > way
> > > > > that aligns with Iceberg-style events.
> > > > >
> > > > > The Metrics side feels different to me. Once we start adding more
> and
> > > > more
> > > > > type-specific filters, query semantics, and schema shape for
> > individual
> > > > > metric families, it seems easy for Polaris to drift toward a
> built-in
> > > > > observability backend. My bias would be for Polaris to support a
> > > smaller
> > > > > set of community-recognized built-in metrics well, while providing
> > good
> > > > > extensibility points for deployments that want richer querying,
> > > > > visualization, or use-case-specific metrics.
> > > > >
> > > > > Related to that, I am not yet convinced the current metrics model
> is
> > > > > generic enough as a long-term direction. Even after consolidating
> to
> > a
> > > > > single endpoint, the design still feels fairly tied to the current
> > > > > scan/commit shape. I worry that otherwise each new metric family
> will
> > > > keep
> > > > > pulling us into more storage/schema/API reshaping inside Polaris
> > core.
> > > > > So the framing question I would suggest is something like:
> > > > > > What is the minimal built-in metrics surface Polaris should own
> in
> > > > core,
> > > > > and where should we instead rely on extensibility / sink-export /
> > > > > plugin-style integration?
> > > > >
> > > > > For me, getting that boundary right matters more than settling
> every
> > > > query
> > > > > parameter detail first.
> > > > >
> > > > > -ej
> > > > >
> > > > > On Tue, Mar 3, 2026 at 12:29 PM Anand Kumar Sankaran via dev <
> > > > > dev@polaris.apache.org> wrote:
> > > > >
> > > > > > Hi Yufei and Dmitri,
> > > > > >
> > > > > > Here is a proposal for the REST endpoints for metrics and events.
> > > > > >
> > > > > >
> > > >
> > >
> >
> https://urldefense.com/v3/__https://github.com/apache/polaris/pull/3924/changes__;!!Iz9xO38YGHZK!8KJ0uv4jK3mxZP4nYFrL1hZ0fMkQvoVEAJa8t9LBCzVtm_PWVFGQfIcZp-ykn3_F9_ph6EYyu3fXDjRYWA$
> > > > > >
> > > > > > I did not see any precursors for raising a PR for proposals, so
> > > trying
> > > > > > this.  Please let me know what you think.
> > > > > >
> > > > > > -
> > > > > > Anand
> > > > > >
> > > > > > From: Anand Kumar Sankaran <an...@workday.com>
> > > > > > Date: Monday, March 2, 2026 at 10:25 AM
> > > > > > To: dev@polaris.apache.org <de...@polaris.apache.org>
> > > > > > Subject: Re: Polaris Telemetry and Audit Trail
> > > > > >
> > > > > > About the REST API, based on my use cases:
> > > > > >
> > > > > >
> > > > > >   1.
> > > > > > I want to be able to query commit metrics to track files added /
> > > > removed
> > > > > > per commit, along with record counts. The ingestion pipeline that
> > > > writes
> > > > > > this data is owned by us and we are guaranteed to write this
> > > > information
> > > > > > for each write.
> > > > > >   2.
> > > > > > I want to be able to query scan metrics for read. I understand
> > > clients
> > > > do
> > > > > > not fulfill this requirement.
> > > > > >   3.
> > > > > > I want to be able to query the events table (events are
> persisted)
> > -
> > > > this
> > > > > > may supersede #2, I am not sure yet.
> > > > > >
> > > > > > All this information is in the JDBC based persistence model and
> is
> > > > > > persisted in the metastore. I currently don’t have a need to
> query
> > > > > > prometheus or open telemetry. I do publish some events to
> > Prometheus
> > > > and
> > > > > > they are forwarded to our dashboards elsewhere.
> > > > > >
> > > > > > About the CLI utilities, I meant the admin user utilities. In one
> > of
> > > > the
> > > > > > earliest drafts of my proposal, Prashant mentioned that the
> metrics
> > > > > tables
> > > > > > can grow indefinitely and that a similar problem exists with the
> > > events
> > > > > > table as well. We discussed that cleaning up of old records from
> > both
> > > > > > metrics tables and events tables can be done via a CLI utility.
> > > > > >
> > > > > > I see that Yufei has covered the discussion about datasources.
> > > > > >
> > > > > > -
> > > > > > Anand
> > > > > >
> > > > > >
> > > > > >
> > > > > > From: Yufei Gu <fl...@gmail.com>
> > > > > > Date: Friday, February 27, 2026 at 9:54 PM
> > > > > > To: dev@polaris.apache.org <de...@polaris.apache.org>
> > > > > > Subject: Re: Polaris Telemetry and Audit Trail
> > > > > >
> > > > > > This Message Is From an External Sender
> > > > > > This message came from outside your organization.
> > > > > > Report Suspicious<
> > > > > >
> > > > >
> > > >
> > >
> >
> https://us-phishalarm-ewt.proofpoint.com/EWT/v1/Iz9xO38YGHZK!YhNDZABkHi1B699ote2uMwpOZw8i0QMCGO2Szc-HshuABGhGvwPJcymE6G2oUUxtS8xDkSrtGTPm_I3QnVDHoLMk50m9v8z_nZKTkd-bnVUbreF1u0WnfV_X5eYevZl_$
> > > > > > >
> > > > > >
> > > > > >
> > > > > > As I mentioned in
> > > > > >
> > > > >
> > > >
> > >
> >
> https://urldefense.com/v3/__https://github.com/apache/polaris/issues/3890__;!!Iz9xO38YGHZK!5EuyFFkk3vhRWVIRvQAWBSQfpJkTMA9HxugzDwXmN0LPPqhEFxYkFRGVhtb8AqUwXtDh2OplcMnbMDHKOxrvDU0$
> > > > >,
> > > > > > supporting
> > > > > > multiple data sources is not a trivial change. I would strongly
> > > > recommend
> > > > > > starting with a design document to carefully evaluate the
> > > architectural
> > > > > > implications and long term impact.
> > > > > >
> > > > > > A REST endpoint to query metrics seems reasonable given the
> current
> > > > JDBC
> > > > > > based persistence model. That said, we may also consider
> > alternative
> > > > > > storage models. For example, if we later adopt a time series
> system
> > > > such
> > > > > as
> > > > > > Prometheus to store metrics, the query model and access patterns
> > > would
> > > > be
> > > > > > fundamentally different. Designing the REST API without
> considering
> > > > these
> > > > > > potential evolutions may limit flexibility. I'd suggest to start
> > with
> > > > the
> > > > > > use case.
> > > > > >
> > > > > > Yufei
> > > > > >
> > > > > >
> > > > > > On Fri, Feb 27, 2026 at 3:42 PM Dmitri Bourlatchkov <
> > > dimas@apache.org>
> > > > > > wrote:
> > > > > >
> > > > > > > Hi Anand,
> > > > > > >
> > > > > > > Sharing my view... subject to discussion:
> > > > > > >
> > > > > > > 1. Adding non-IRC REST API to Polaris is perfectly fine.
> > > > > > >
> > > > > > > Figuring out specific endpoint URIs and payloads might require
> a
> > > few
> > > > > > > roundtrips, so opening a separate thread for that might be
> best.
> > > > > > > Contributors commonly create Google Docs for new API proposals
> > too
> > > > > (they
> > > > > > > fairly easy to update as the email discussion progresses).
> > > > > > >
> > > > > > > There was a suggestion to try Markdown (with PRs) for proposals
> > [1]
> > > > ...
> > > > > > > feel free to give it a try if you are comfortable with that.
> > > > > > >
> > > > > > > 2. Could you clarify whether you mean end user utilities or
> admin
> > > > user
> > > > > > > utilities? In the latter case those might be more suitable for
> > the
> > > > > Admin
> > > > > > > CLI (java) not the Python CLI, IMHO.
> > > > > > >
> > > > > > > Why would these utilities be common with events? IMHO, event
> use
> > > > cases
> > > > > > are
> > > > > > > distinct from scan/commit metrics.
> > > > > > >
> > > > > > > 3. I'd prefer separating metrics persistence from MetaStore
> > > > persistence
> > > > > > at
> > > > > > > the code level, so that they could be mixed and matched
> > > > independently.
> > > > > > The
> > > > > > > separate datasource question will become a non-issue with that
> > > > > approach,
> > > > > > I
> > > > > > > guess.
> > > > > > >
> > > > > > > The rationale for separating scan metrics and metastore
> > persistence
> > > > is
> > > > > > that
> > > > > > > "cascading deletes" between them are hardly ever required.
> > > > Furthermore,
> > > > > > the
> > > > > > > data and query patterns are very different so different
> > > technologies
> > > > > > might
> > > > > > > be beneficial in each case.
> > > > > > >
> > > > > > > [1]
> > > > > >
> > > > >
> > > >
> > >
> >
> https://urldefense.com/v3/__https://lists.apache.org/thread/yto2wp982t43h1mqjwnslswhws5z47cy__;!!Iz9xO38YGHZK!5EuyFFkk3vhRWVIRvQAWBSQfpJkTMA9HxugzDwXmN0LPPqhEFxYkFRGVhtb8AqUwXtDh2OplcMnbMDHKxYDakNU$
> > > > >> >
> > > > > > > Cheers,
> > > > > > > Dmitri.
> > > > > > >
> > > > > > > On Fri, Feb 27, 2026 at 6:19 PM Anand Kumar Sankaran via dev <
> > > > > > > dev@polaris.apache.org> wrote:
> > > > > > >
> > > > > > > > Thanks all. This PR is merged now.
> > > > > > > >
> > > > > > > > Here are the follow-up features / work needed.  These were
> all
> > > part
> > > > > of
> > > > > > > the
> > > > > > > > merged PR at some point in time and were removed to reduce
> > scope.
> > > > > > > >
> > > > > > > > Please let me know what you think.
> > > > > > > >
> > > > > > > >
> > > > > > > >   1.  A REST API to paginate through table metrics. This will
> > be
> > > > > > non-IRC
> > > > > > > > standard addition.
> > > > > > > >   2.  Utilities for managing old records, should be common
> with
> > > > > events.
> > > > > > > > There was some discussion that it belongs to the CLI.
> > > > > > > >   3.  Separate datasource (metrics, events, even other
> > tables?).
> > > > > > > >
> > > > > > > >
> > > > > > > > Anything else?
> > > > > > > >
> > > > > > > > -
> > > > > > > > Anand
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > > >
> > > > >
> > > >
> > > >
> > >
> >
>

Re: Proposal for REST endpoints for table metrics and events

[Yufei Gu <fl...@gmail.com>]

Thanks Anand for working on this.

Given the IRC event endpoint is WIP, I think it'd be best to be consistent
with the IRC endpoint. For more context on the IRC event, the event
endpoint is not finalized yet. I'd recommend speeding up the IRC side work
to avoid any inconsistencies between Polaris and IRC spec.

I have a few questions about the metrics endpoint:
1. Do we need to expose them via Polaris REST endpoint? Can users grab the
metrics from the backend directly? I understand the RBAC won't be there,
but it provides flexibility for users. For example, some users may choose a
different persistence model such as a KV store or storing the metrics as
objects in S3, which usually scales better than an RDBMS like Postgres. My
understanding is that Polaris is not intended to be a full metrics system
anyway, but rather to provide a way for downstream systems to consume these
data.
2. Should we consider it as an IRC endpoint? Given the metrics report
endpoint in IRC, the Iceberg community might also be interested in serving
back metrics, similar to the event endpoint. In that case, there is a risk
of fragmentation if we create a Polaris endpoint now. We should avoid that
if possible. It might be worth checking with the Iceberg community first.

Happy to hear others’ thoughts on this.

Yufei


On Wed, Mar 11, 2026 at 9:27 AM Nándor Kollár <nk...@apache.org> wrote:

> I agree with Dmitri. The direction outlined in the proposal looks good to
> me, and the finer details can be worked out as implementation gets
> underway. We can adjust the design doc accordingly, later on.
>
>
> Thanks,
> Nandor
>
> Dmitri Bourlatchkov <di...@apache.org> ezt írta (időpont: 2026. márc. 10.,
> K, 20:43):
>
> > Hi Anand and All,
> >
> > The proposal LGTM in its current form.
> >
> > My personal approach is that a proposal does not have to be as "polished"
> > as the final API spec. As long as we have consensus on the general
> > approach and the basic API principles, I think we can proceed to
> > implementation and iron out final wrinkles during actual API spec and
> code
> > PRs.
> >
> > Would this approach work for everyone?
> >
> > Re-posting my comment from GH [1] here for visibility, in case people
> have
> > different opinions and wish to discuss this in more depth:
> >
> > From my POV, Polaris is a platform, indeed. In this sense, I think it is
> > > critical to enable users to control what features are at play in
> runtime,
> > > since different users have different use cases. This is why I
> originally
> > > advocated for isolating Metrics Persistence from MetaStore Persistence.
> > >
> > > If a user decides to leverage the "native" Polaris (scan/commit)
> Metrics
> > > Persistence, I do not see any disadvantage in also exposing an
> (optional)
> > > REST API for loading these metrics from Polaris Persistence.
> > >
> > > The degree of support and sophistication that goes into this sub-system
> > is
> > > up to the community. If we have contributors (like @obelix74 ) who are
> > > willing to evolve it, I see no harm in some functionality overlap with
> > more
> > > focused metrics platforms. Again, the key point is for all users to
> have
> > > control and be able to opt in or out of this feature in their specific
> > > deployments.
> > >
> > > Of course, offloading scan/commit metrics storage to a specialized
> > > observability system is possible too (assuming someone develops
> > integration
> > > code for that, which is very welcome).
> >
> >
> > [1] https://github.com/apache/polaris/pull/3924#discussion_r2913947744
> >
> > Thanks,
> > Dmitri.
> >
> > On Tue, Mar 10, 2026 at 12:37 PM Anand Kumar Sankaran via dev <
> > dev@polaris.apache.org> wrote:
> >
> > > Hi EJ Wang and Dmitri,
> > >
> > > I addressed all your concerns about the proposal, in particular
> > > https://github.com/apache/polaris/pull/3924#discussion_r2908317696.
> > >
> > > Does this address your concerns?
> > >
> > > -
> > > Anand
> > >
> > > From: Dmitri Bourlatchkov <di...@apache.org>
> > > Date: Monday, March 9, 2026 at 1:17 PM
> > > To: dev@polaris.apache.org <de...@polaris.apache.org>
> > > Subject: Re: Proposal for REST endpoints for table metrics and events
> > >
> > > This Message Is From an External Sender
> > > This message came from outside your organization.
> > > Report Suspicious<
> > >
> >
> https://us-phishalarm-ewt.proofpoint.com/EWT/v1/Iz9xO38YGHZK!YhNDZAGr2cumYdtJ_UTm0gR9PfI_-PwSpR_GtNr1uVQ_xo-s2AskvUmbkLZ-C5V8eOKN-omus47On4k4hfFo-0G7CHMLwVjEego-rZrPuepAybX7DP8Ua0VNSrsZ83C4$
> > > >
> > >
> > >
> > > Hi EJ,
> > >
> > > You make good points about the metrics API extensibility and evolution.
> > >
> > > However, we need to consider practical aspects too. Anand appears to
> have
> > > some specific use cases in mind, and I assume his proposal addresses
> > them.
> > >
> > > Starting with an API + implementation that works for some real
> > > world applications will validate the feature's usability.
> > >
> > > We can revamp the API completely in its v2 after v1 is merged. New
> major
> > > API versions do not have to be backward-compatible with older versions
> of
> > > the same API [1].
> > >
> > > In my personal experience, a v1 API can hardly be expected to cover all
> > use
> > > cases and extensions well. We can certainly take a bit more time to
> > polish
> > > it, but from my POV it might be best to iterate in terms of API
> versions
> > > rather than on unmerged commits in the initial proposal. Just my 2
> cents
> > :)
> > >
> > > That said, we should flag the new APIs in this proposal as "beta"... at
> > > least initially (which is the usual practice in Polaris).
> > >
> > > > I wonder if it would help to evaluate the Events API and Metrics API
> a
> > > bit more independently.
> > >
> > > That makes sense to me. However, the current proposal progressed a lot
> > > since its initial submission and contained both APIs. I would not want
> to
> > > lose this momentum.
> > >
> > > It might still be advisable to implement the events and metrics APIs
> > > separately and gather additional feedback at that time.
> > >
> > > [1]
> > >
> >
> https://urldefense.com/v3/__https://polaris.apache.org/in-dev/unreleased/evolution/__;!!Iz9xO38YGHZK!8KJ0uv4jK3mxZP4nYFrL1hZ0fMkQvoVEAJa8t9LBCzVtm_PWVFGQfIcZp-ykn3_F9_ph6EYyu3dUZjPAcQ$
> > >
> > > Cheers,
> > > Dmitri.
> > >
> > > On Mon, Mar 9, 2026 at 3:48 PM EJ Wang <ej.wang.dev.interest@gmail.com
> >
> > > wrote:
> > >
> > > > Hi Anand,
> > > >
> > > > I think the proposal is moving in a better direction, especially on
> the
> > > > Events side, and I appreciate the iteration so far. That said, I
> still
> > > have
> > > > some concerns about the Metrics side, but they are less about
> > individual
> > > > parameters or endpoint shape, and more about product boundary.
> > > >
> > > > 2 cents: I wonder if it would help to evaluate the Events API and
> > Metrics
> > > > API a bit more independently.
> > > >
> > > > The Events side feels relatively close to Polaris' catalog/change-log
> > > > scope. It is easier to justify as part of the core/community surface,
> > > > especially if the goal is to expose completed catalog mutations in a
> > way
> > > > that aligns with Iceberg-style events.
> > > >
> > > > The Metrics side feels different to me. Once we start adding more and
> > > more
> > > > type-specific filters, query semantics, and schema shape for
> individual
> > > > metric families, it seems easy for Polaris to drift toward a built-in
> > > > observability backend. My bias would be for Polaris to support a
> > smaller
> > > > set of community-recognized built-in metrics well, while providing
> good
> > > > extensibility points for deployments that want richer querying,
> > > > visualization, or use-case-specific metrics.
> > > >
> > > > Related to that, I am not yet convinced the current metrics model is
> > > > generic enough as a long-term direction. Even after consolidating to
> a
> > > > single endpoint, the design still feels fairly tied to the current
> > > > scan/commit shape. I worry that otherwise each new metric family will
> > > keep
> > > > pulling us into more storage/schema/API reshaping inside Polaris
> core.
> > > > So the framing question I would suggest is something like:
> > > > > What is the minimal built-in metrics surface Polaris should own in
> > > core,
> > > > and where should we instead rely on extensibility / sink-export /
> > > > plugin-style integration?
> > > >
> > > > For me, getting that boundary right matters more than settling every
> > > query
> > > > parameter detail first.
> > > >
> > > > -ej
> > > >
> > > > On Tue, Mar 3, 2026 at 12:29 PM Anand Kumar Sankaran via dev <
> > > > dev@polaris.apache.org> wrote:
> > > >
> > > > > Hi Yufei and Dmitri,
> > > > >
> > > > > Here is a proposal for the REST endpoints for metrics and events.
> > > > >
> > > > >
> > >
> >
> https://urldefense.com/v3/__https://github.com/apache/polaris/pull/3924/changes__;!!Iz9xO38YGHZK!8KJ0uv4jK3mxZP4nYFrL1hZ0fMkQvoVEAJa8t9LBCzVtm_PWVFGQfIcZp-ykn3_F9_ph6EYyu3fXDjRYWA$
> > > > >
> > > > > I did not see any precursors for raising a PR for proposals, so
> > trying
> > > > > this.  Please let me know what you think.
> > > > >
> > > > > -
> > > > > Anand
> > > > >
> > > > > From: Anand Kumar Sankaran <an...@workday.com>
> > > > > Date: Monday, March 2, 2026 at 10:25 AM
> > > > > To: dev@polaris.apache.org <de...@polaris.apache.org>
> > > > > Subject: Re: Polaris Telemetry and Audit Trail
> > > > >
> > > > > About the REST API, based on my use cases:
> > > > >
> > > > >
> > > > >   1.
> > > > > I want to be able to query commit metrics to track files added /
> > > removed
> > > > > per commit, along with record counts. The ingestion pipeline that
> > > writes
> > > > > this data is owned by us and we are guaranteed to write this
> > > information
> > > > > for each write.
> > > > >   2.
> > > > > I want to be able to query scan metrics for read. I understand
> > clients
> > > do
> > > > > not fulfill this requirement.
> > > > >   3.
> > > > > I want to be able to query the events table (events are persisted)
> -
> > > this
> > > > > may supersede #2, I am not sure yet.
> > > > >
> > > > > All this information is in the JDBC based persistence model and is
> > > > > persisted in the metastore. I currently don’t have a need to query
> > > > > prometheus or open telemetry. I do publish some events to
> Prometheus
> > > and
> > > > > they are forwarded to our dashboards elsewhere.
> > > > >
> > > > > About the CLI utilities, I meant the admin user utilities. In one
> of
> > > the
> > > > > earliest drafts of my proposal, Prashant mentioned that the metrics
> > > > tables
> > > > > can grow indefinitely and that a similar problem exists with the
> > events
> > > > > table as well. We discussed that cleaning up of old records from
> both
> > > > > metrics tables and events tables can be done via a CLI utility.
> > > > >
> > > > > I see that Yufei has covered the discussion about datasources.
> > > > >
> > > > > -
> > > > > Anand
> > > > >
> > > > >
> > > > >
> > > > > From: Yufei Gu <fl...@gmail.com>
> > > > > Date: Friday, February 27, 2026 at 9:54 PM
> > > > > To: dev@polaris.apache.org <de...@polaris.apache.org>
> > > > > Subject: Re: Polaris Telemetry and Audit Trail
> > > > >
> > > > > This Message Is From an External Sender
> > > > > This message came from outside your organization.
> > > > > Report Suspicious<
> > > > >
> > > >
> > >
> >
> https://us-phishalarm-ewt.proofpoint.com/EWT/v1/Iz9xO38YGHZK!YhNDZABkHi1B699ote2uMwpOZw8i0QMCGO2Szc-HshuABGhGvwPJcymE6G2oUUxtS8xDkSrtGTPm_I3QnVDHoLMk50m9v8z_nZKTkd-bnVUbreF1u0WnfV_X5eYevZl_$
> > > > > >
> > > > >
> > > > >
> > > > > As I mentioned in
> > > > >
> > > >
> > >
> >
> https://urldefense.com/v3/__https://github.com/apache/polaris/issues/3890__;!!Iz9xO38YGHZK!5EuyFFkk3vhRWVIRvQAWBSQfpJkTMA9HxugzDwXmN0LPPqhEFxYkFRGVhtb8AqUwXtDh2OplcMnbMDHKOxrvDU0$
> > > >,
> > > > > supporting
> > > > > multiple data sources is not a trivial change. I would strongly
> > > recommend
> > > > > starting with a design document to carefully evaluate the
> > architectural
> > > > > implications and long term impact.
> > > > >
> > > > > A REST endpoint to query metrics seems reasonable given the current
> > > JDBC
> > > > > based persistence model. That said, we may also consider
> alternative
> > > > > storage models. For example, if we later adopt a time series system
> > > such
> > > > as
> > > > > Prometheus to store metrics, the query model and access patterns
> > would
> > > be
> > > > > fundamentally different. Designing the REST API without considering
> > > these
> > > > > potential evolutions may limit flexibility. I'd suggest to start
> with
> > > the
> > > > > use case.
> > > > >
> > > > > Yufei
> > > > >
> > > > >
> > > > > On Fri, Feb 27, 2026 at 3:42 PM Dmitri Bourlatchkov <
> > dimas@apache.org>
> > > > > wrote:
> > > > >
> > > > > > Hi Anand,
> > > > > >
> > > > > > Sharing my view... subject to discussion:
> > > > > >
> > > > > > 1. Adding non-IRC REST API to Polaris is perfectly fine.
> > > > > >
> > > > > > Figuring out specific endpoint URIs and payloads might require a
> > few
> > > > > > roundtrips, so opening a separate thread for that might be best.
> > > > > > Contributors commonly create Google Docs for new API proposals
> too
> > > > (they
> > > > > > fairly easy to update as the email discussion progresses).
> > > > > >
> > > > > > There was a suggestion to try Markdown (with PRs) for proposals
> [1]
> > > ...
> > > > > > feel free to give it a try if you are comfortable with that.
> > > > > >
> > > > > > 2. Could you clarify whether you mean end user utilities or admin
> > > user
> > > > > > utilities? In the latter case those might be more suitable for
> the
> > > > Admin
> > > > > > CLI (java) not the Python CLI, IMHO.
> > > > > >
> > > > > > Why would these utilities be common with events? IMHO, event use
> > > cases
> > > > > are
> > > > > > distinct from scan/commit metrics.
> > > > > >
> > > > > > 3. I'd prefer separating metrics persistence from MetaStore
> > > persistence
> > > > > at
> > > > > > the code level, so that they could be mixed and matched
> > > independently.
> > > > > The
> > > > > > separate datasource question will become a non-issue with that
> > > > approach,
> > > > > I
> > > > > > guess.
> > > > > >
> > > > > > The rationale for separating scan metrics and metastore
> persistence
> > > is
> > > > > that
> > > > > > "cascading deletes" between them are hardly ever required.
> > > Furthermore,
> > > > > the
> > > > > > data and query patterns are very different so different
> > technologies
> > > > > might
> > > > > > be beneficial in each case.
> > > > > >
> > > > > > [1]
> > > > >
> > > >
> > >
> >
> https://urldefense.com/v3/__https://lists.apache.org/thread/yto2wp982t43h1mqjwnslswhws5z47cy__;!!Iz9xO38YGHZK!5EuyFFkk3vhRWVIRvQAWBSQfpJkTMA9HxugzDwXmN0LPPqhEFxYkFRGVhtb8AqUwXtDh2OplcMnbMDHKxYDakNU$
> > > >> >
> > > > > > Cheers,
> > > > > > Dmitri.
> > > > > >
> > > > > > On Fri, Feb 27, 2026 at 6:19 PM Anand Kumar Sankaran via dev <
> > > > > > dev@polaris.apache.org> wrote:
> > > > > >
> > > > > > > Thanks all. This PR is merged now.
> > > > > > >
> > > > > > > Here are the follow-up features / work needed.  These were all
> > part
> > > > of
> > > > > > the
> > > > > > > merged PR at some point in time and were removed to reduce
> scope.
> > > > > > >
> > > > > > > Please let me know what you think.
> > > > > > >
> > > > > > >
> > > > > > >   1.  A REST API to paginate through table metrics. This will
> be
> > > > > non-IRC
> > > > > > > standard addition.
> > > > > > >   2.  Utilities for managing old records, should be common with
> > > > events.
> > > > > > > There was some discussion that it belongs to the CLI.
> > > > > > >   3.  Separate datasource (metrics, events, even other
> tables?).
> > > > > > >
> > > > > > >
> > > > > > > Anything else?
> > > > > > >
> > > > > > > -
> > > > > > > Anand
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > > >
> > > >
> > >
> > >
> >
>

Re: Proposal for REST endpoints for table metrics and events

[Nándor Kollár <nk...@apache.org>]

I agree with Dmitri. The direction outlined in the proposal looks good to
me, and the finer details can be worked out as implementation gets
underway. We can adjust the design doc accordingly, later on.


Thanks,
Nandor

Dmitri Bourlatchkov <di...@apache.org> ezt írta (időpont: 2026. márc. 10.,
K, 20:43):

> Hi Anand and All,
>
> The proposal LGTM in its current form.
>
> My personal approach is that a proposal does not have to be as "polished"
> as the final API spec. As long as we have consensus on the general
> approach and the basic API principles, I think we can proceed to
> implementation and iron out final wrinkles during actual API spec and code
> PRs.
>
> Would this approach work for everyone?
>
> Re-posting my comment from GH [1] here for visibility, in case people have
> different opinions and wish to discuss this in more depth:
>
> From my POV, Polaris is a platform, indeed. In this sense, I think it is
> > critical to enable users to control what features are at play in runtime,
> > since different users have different use cases. This is why I originally
> > advocated for isolating Metrics Persistence from MetaStore Persistence.
> >
> > If a user decides to leverage the "native" Polaris (scan/commit) Metrics
> > Persistence, I do not see any disadvantage in also exposing an (optional)
> > REST API for loading these metrics from Polaris Persistence.
> >
> > The degree of support and sophistication that goes into this sub-system
> is
> > up to the community. If we have contributors (like @obelix74 ) who are
> > willing to evolve it, I see no harm in some functionality overlap with
> more
> > focused metrics platforms. Again, the key point is for all users to have
> > control and be able to opt in or out of this feature in their specific
> > deployments.
> >
> > Of course, offloading scan/commit metrics storage to a specialized
> > observability system is possible too (assuming someone develops
> integration
> > code for that, which is very welcome).
>
>
> [1] https://github.com/apache/polaris/pull/3924#discussion_r2913947744
>
> Thanks,
> Dmitri.
>
> On Tue, Mar 10, 2026 at 12:37 PM Anand Kumar Sankaran via dev <
> dev@polaris.apache.org> wrote:
>
> > Hi EJ Wang and Dmitri,
> >
> > I addressed all your concerns about the proposal, in particular
> > https://github.com/apache/polaris/pull/3924#discussion_r2908317696.
> >
> > Does this address your concerns?
> >
> > -
> > Anand
> >
> > From: Dmitri Bourlatchkov <di...@apache.org>
> > Date: Monday, March 9, 2026 at 1:17 PM
> > To: dev@polaris.apache.org <de...@polaris.apache.org>
> > Subject: Re: Proposal for REST endpoints for table metrics and events
> >
> > This Message Is From an External Sender
> > This message came from outside your organization.
> > Report Suspicious<
> >
> https://us-phishalarm-ewt.proofpoint.com/EWT/v1/Iz9xO38YGHZK!YhNDZAGr2cumYdtJ_UTm0gR9PfI_-PwSpR_GtNr1uVQ_xo-s2AskvUmbkLZ-C5V8eOKN-omus47On4k4hfFo-0G7CHMLwVjEego-rZrPuepAybX7DP8Ua0VNSrsZ83C4$
> > >
> >
> >
> > Hi EJ,
> >
> > You make good points about the metrics API extensibility and evolution.
> >
> > However, we need to consider practical aspects too. Anand appears to have
> > some specific use cases in mind, and I assume his proposal addresses
> them.
> >
> > Starting with an API + implementation that works for some real
> > world applications will validate the feature's usability.
> >
> > We can revamp the API completely in its v2 after v1 is merged. New major
> > API versions do not have to be backward-compatible with older versions of
> > the same API [1].
> >
> > In my personal experience, a v1 API can hardly be expected to cover all
> use
> > cases and extensions well. We can certainly take a bit more time to
> polish
> > it, but from my POV it might be best to iterate in terms of API versions
> > rather than on unmerged commits in the initial proposal. Just my 2 cents
> :)
> >
> > That said, we should flag the new APIs in this proposal as "beta"... at
> > least initially (which is the usual practice in Polaris).
> >
> > > I wonder if it would help to evaluate the Events API and Metrics API a
> > bit more independently.
> >
> > That makes sense to me. However, the current proposal progressed a lot
> > since its initial submission and contained both APIs. I would not want to
> > lose this momentum.
> >
> > It might still be advisable to implement the events and metrics APIs
> > separately and gather additional feedback at that time.
> >
> > [1]
> >
> https://urldefense.com/v3/__https://polaris.apache.org/in-dev/unreleased/evolution/__;!!Iz9xO38YGHZK!8KJ0uv4jK3mxZP4nYFrL1hZ0fMkQvoVEAJa8t9LBCzVtm_PWVFGQfIcZp-ykn3_F9_ph6EYyu3dUZjPAcQ$
> >
> > Cheers,
> > Dmitri.
> >
> > On Mon, Mar 9, 2026 at 3:48 PM EJ Wang <ej...@gmail.com>
> > wrote:
> >
> > > Hi Anand,
> > >
> > > I think the proposal is moving in a better direction, especially on the
> > > Events side, and I appreciate the iteration so far. That said, I still
> > have
> > > some concerns about the Metrics side, but they are less about
> individual
> > > parameters or endpoint shape, and more about product boundary.
> > >
> > > 2 cents: I wonder if it would help to evaluate the Events API and
> Metrics
> > > API a bit more independently.
> > >
> > > The Events side feels relatively close to Polaris' catalog/change-log
> > > scope. It is easier to justify as part of the core/community surface,
> > > especially if the goal is to expose completed catalog mutations in a
> way
> > > that aligns with Iceberg-style events.
> > >
> > > The Metrics side feels different to me. Once we start adding more and
> > more
> > > type-specific filters, query semantics, and schema shape for individual
> > > metric families, it seems easy for Polaris to drift toward a built-in
> > > observability backend. My bias would be for Polaris to support a
> smaller
> > > set of community-recognized built-in metrics well, while providing good
> > > extensibility points for deployments that want richer querying,
> > > visualization, or use-case-specific metrics.
> > >
> > > Related to that, I am not yet convinced the current metrics model is
> > > generic enough as a long-term direction. Even after consolidating to a
> > > single endpoint, the design still feels fairly tied to the current
> > > scan/commit shape. I worry that otherwise each new metric family will
> > keep
> > > pulling us into more storage/schema/API reshaping inside Polaris core.
> > > So the framing question I would suggest is something like:
> > > > What is the minimal built-in metrics surface Polaris should own in
> > core,
> > > and where should we instead rely on extensibility / sink-export /
> > > plugin-style integration?
> > >
> > > For me, getting that boundary right matters more than settling every
> > query
> > > parameter detail first.
> > >
> > > -ej
> > >
> > > On Tue, Mar 3, 2026 at 12:29 PM Anand Kumar Sankaran via dev <
> > > dev@polaris.apache.org> wrote:
> > >
> > > > Hi Yufei and Dmitri,
> > > >
> > > > Here is a proposal for the REST endpoints for metrics and events.
> > > >
> > > >
> >
> https://urldefense.com/v3/__https://github.com/apache/polaris/pull/3924/changes__;!!Iz9xO38YGHZK!8KJ0uv4jK3mxZP4nYFrL1hZ0fMkQvoVEAJa8t9LBCzVtm_PWVFGQfIcZp-ykn3_F9_ph6EYyu3fXDjRYWA$
> > > >
> > > > I did not see any precursors for raising a PR for proposals, so
> trying
> > > > this.  Please let me know what you think.
> > > >
> > > > -
> > > > Anand
> > > >
> > > > From: Anand Kumar Sankaran <an...@workday.com>
> > > > Date: Monday, March 2, 2026 at 10:25 AM
> > > > To: dev@polaris.apache.org <de...@polaris.apache.org>
> > > > Subject: Re: Polaris Telemetry and Audit Trail
> > > >
> > > > About the REST API, based on my use cases:
> > > >
> > > >
> > > >   1.
> > > > I want to be able to query commit metrics to track files added /
> > removed
> > > > per commit, along with record counts. The ingestion pipeline that
> > writes
> > > > this data is owned by us and we are guaranteed to write this
> > information
> > > > for each write.
> > > >   2.
> > > > I want to be able to query scan metrics for read. I understand
> clients
> > do
> > > > not fulfill this requirement.
> > > >   3.
> > > > I want to be able to query the events table (events are persisted) -
> > this
> > > > may supersede #2, I am not sure yet.
> > > >
> > > > All this information is in the JDBC based persistence model and is
> > > > persisted in the metastore. I currently don’t have a need to query
> > > > prometheus or open telemetry. I do publish some events to Prometheus
> > and
> > > > they are forwarded to our dashboards elsewhere.
> > > >
> > > > About the CLI utilities, I meant the admin user utilities. In one of
> > the
> > > > earliest drafts of my proposal, Prashant mentioned that the metrics
> > > tables
> > > > can grow indefinitely and that a similar problem exists with the
> events
> > > > table as well. We discussed that cleaning up of old records from both
> > > > metrics tables and events tables can be done via a CLI utility.
> > > >
> > > > I see that Yufei has covered the discussion about datasources.
> > > >
> > > > -
> > > > Anand
> > > >
> > > >
> > > >
> > > > From: Yufei Gu <fl...@gmail.com>
> > > > Date: Friday, February 27, 2026 at 9:54 PM
> > > > To: dev@polaris.apache.org <de...@polaris.apache.org>
> > > > Subject: Re: Polaris Telemetry and Audit Trail
> > > >
> > > > This Message Is From an External Sender
> > > > This message came from outside your organization.
> > > > Report Suspicious<
> > > >
> > >
> >
> https://us-phishalarm-ewt.proofpoint.com/EWT/v1/Iz9xO38YGHZK!YhNDZABkHi1B699ote2uMwpOZw8i0QMCGO2Szc-HshuABGhGvwPJcymE6G2oUUxtS8xDkSrtGTPm_I3QnVDHoLMk50m9v8z_nZKTkd-bnVUbreF1u0WnfV_X5eYevZl_$
> > > > >
> > > >
> > > >
> > > > As I mentioned in
> > > >
> > >
> >
> https://urldefense.com/v3/__https://github.com/apache/polaris/issues/3890__;!!Iz9xO38YGHZK!5EuyFFkk3vhRWVIRvQAWBSQfpJkTMA9HxugzDwXmN0LPPqhEFxYkFRGVhtb8AqUwXtDh2OplcMnbMDHKOxrvDU0$
> > >,
> > > > supporting
> > > > multiple data sources is not a trivial change. I would strongly
> > recommend
> > > > starting with a design document to carefully evaluate the
> architectural
> > > > implications and long term impact.
> > > >
> > > > A REST endpoint to query metrics seems reasonable given the current
> > JDBC
> > > > based persistence model. That said, we may also consider alternative
> > > > storage models. For example, if we later adopt a time series system
> > such
> > > as
> > > > Prometheus to store metrics, the query model and access patterns
> would
> > be
> > > > fundamentally different. Designing the REST API without considering
> > these
> > > > potential evolutions may limit flexibility. I'd suggest to start with
> > the
> > > > use case.
> > > >
> > > > Yufei
> > > >
> > > >
> > > > On Fri, Feb 27, 2026 at 3:42 PM Dmitri Bourlatchkov <
> dimas@apache.org>
> > > > wrote:
> > > >
> > > > > Hi Anand,
> > > > >
> > > > > Sharing my view... subject to discussion:
> > > > >
> > > > > 1. Adding non-IRC REST API to Polaris is perfectly fine.
> > > > >
> > > > > Figuring out specific endpoint URIs and payloads might require a
> few
> > > > > roundtrips, so opening a separate thread for that might be best.
> > > > > Contributors commonly create Google Docs for new API proposals too
> > > (they
> > > > > fairly easy to update as the email discussion progresses).
> > > > >
> > > > > There was a suggestion to try Markdown (with PRs) for proposals [1]
> > ...
> > > > > feel free to give it a try if you are comfortable with that.
> > > > >
> > > > > 2. Could you clarify whether you mean end user utilities or admin
> > user
> > > > > utilities? In the latter case those might be more suitable for the
> > > Admin
> > > > > CLI (java) not the Python CLI, IMHO.
> > > > >
> > > > > Why would these utilities be common with events? IMHO, event use
> > cases
> > > > are
> > > > > distinct from scan/commit metrics.
> > > > >
> > > > > 3. I'd prefer separating metrics persistence from MetaStore
> > persistence
> > > > at
> > > > > the code level, so that they could be mixed and matched
> > independently.
> > > > The
> > > > > separate datasource question will become a non-issue with that
> > > approach,
> > > > I
> > > > > guess.
> > > > >
> > > > > The rationale for separating scan metrics and metastore persistence
> > is
> > > > that
> > > > > "cascading deletes" between them are hardly ever required.
> > Furthermore,
> > > > the
> > > > > data and query patterns are very different so different
> technologies
> > > > might
> > > > > be beneficial in each case.
> > > > >
> > > > > [1]
> > > >
> > >
> >
> https://urldefense.com/v3/__https://lists.apache.org/thread/yto2wp982t43h1mqjwnslswhws5z47cy__;!!Iz9xO38YGHZK!5EuyFFkk3vhRWVIRvQAWBSQfpJkTMA9HxugzDwXmN0LPPqhEFxYkFRGVhtb8AqUwXtDh2OplcMnbMDHKxYDakNU$
> > >> >
> > > > > Cheers,
> > > > > Dmitri.
> > > > >
> > > > > On Fri, Feb 27, 2026 at 6:19 PM Anand Kumar Sankaran via dev <
> > > > > dev@polaris.apache.org> wrote:
> > > > >
> > > > > > Thanks all. This PR is merged now.
> > > > > >
> > > > > > Here are the follow-up features / work needed.  These were all
> part
> > > of
> > > > > the
> > > > > > merged PR at some point in time and were removed to reduce scope.
> > > > > >
> > > > > > Please let me know what you think.
> > > > > >
> > > > > >
> > > > > >   1.  A REST API to paginate through table metrics. This will be
> > > > non-IRC
> > > > > > standard addition.
> > > > > >   2.  Utilities for managing old records, should be common with
> > > events.
> > > > > > There was some discussion that it belongs to the CLI.
> > > > > >   3.  Separate datasource (metrics, events, even other tables?).
> > > > > >
> > > > > >
> > > > > > Anything else?
> > > > > >
> > > > > > -
> > > > > > Anand
> > > > > >
> > > > > >
> > > > >
> > > >
> > > >
> > >
> >
> >
>

Re: Proposal for REST endpoints for table metrics and events

[Dmitri Bourlatchkov <di...@apache.org>]

Hi Anand and All,

The proposal LGTM in its current form.

My personal approach is that a proposal does not have to be as "polished"
as the final API spec. As long as we have consensus on the general
approach and the basic API principles, I think we can proceed to
implementation and iron out final wrinkles during actual API spec and code
PRs.

Would this approach work for everyone?

Re-posting my comment from GH [1] here for visibility, in case people have
different opinions and wish to discuss this in more depth:

From my POV, Polaris is a platform, indeed. In this sense, I think it is
> critical to enable users to control what features are at play in runtime,
> since different users have different use cases. This is why I originally
> advocated for isolating Metrics Persistence from MetaStore Persistence.
>
> If a user decides to leverage the "native" Polaris (scan/commit) Metrics
> Persistence, I do not see any disadvantage in also exposing an (optional)
> REST API for loading these metrics from Polaris Persistence.
>
> The degree of support and sophistication that goes into this sub-system is
> up to the community. If we have contributors (like @obelix74 ) who are
> willing to evolve it, I see no harm in some functionality overlap with more
> focused metrics platforms. Again, the key point is for all users to have
> control and be able to opt in or out of this feature in their specific
> deployments.
>
> Of course, offloading scan/commit metrics storage to a specialized
> observability system is possible too (assuming someone develops integration
> code for that, which is very welcome).


[1] https://github.com/apache/polaris/pull/3924#discussion_r2913947744

Thanks,
Dmitri.

On Tue, Mar 10, 2026 at 12:37 PM Anand Kumar Sankaran via dev <
dev@polaris.apache.org> wrote:

> Hi EJ Wang and Dmitri,
>
> I addressed all your concerns about the proposal, in particular
> https://github.com/apache/polaris/pull/3924#discussion_r2908317696.
>
> Does this address your concerns?
>
> -
> Anand
>
> From: Dmitri Bourlatchkov <di...@apache.org>
> Date: Monday, March 9, 2026 at 1:17 PM
> To: dev@polaris.apache.org <de...@polaris.apache.org>
> Subject: Re: Proposal for REST endpoints for table metrics and events
>
> This Message Is From an External Sender
> This message came from outside your organization.
> Report Suspicious<
> https://us-phishalarm-ewt.proofpoint.com/EWT/v1/Iz9xO38YGHZK!YhNDZAGr2cumYdtJ_UTm0gR9PfI_-PwSpR_GtNr1uVQ_xo-s2AskvUmbkLZ-C5V8eOKN-omus47On4k4hfFo-0G7CHMLwVjEego-rZrPuepAybX7DP8Ua0VNSrsZ83C4$
> >
>
>
> Hi EJ,
>
> You make good points about the metrics API extensibility and evolution.
>
> However, we need to consider practical aspects too. Anand appears to have
> some specific use cases in mind, and I assume his proposal addresses them.
>
> Starting with an API + implementation that works for some real
> world applications will validate the feature's usability.
>
> We can revamp the API completely in its v2 after v1 is merged. New major
> API versions do not have to be backward-compatible with older versions of
> the same API [1].
>
> In my personal experience, a v1 API can hardly be expected to cover all use
> cases and extensions well. We can certainly take a bit more time to polish
> it, but from my POV it might be best to iterate in terms of API versions
> rather than on unmerged commits in the initial proposal. Just my 2 cents :)
>
> That said, we should flag the new APIs in this proposal as "beta"... at
> least initially (which is the usual practice in Polaris).
>
> > I wonder if it would help to evaluate the Events API and Metrics API a
> bit more independently.
>
> That makes sense to me. However, the current proposal progressed a lot
> since its initial submission and contained both APIs. I would not want to
> lose this momentum.
>
> It might still be advisable to implement the events and metrics APIs
> separately and gather additional feedback at that time.
>
> [1]
> https://urldefense.com/v3/__https://polaris.apache.org/in-dev/unreleased/evolution/__;!!Iz9xO38YGHZK!8KJ0uv4jK3mxZP4nYFrL1hZ0fMkQvoVEAJa8t9LBCzVtm_PWVFGQfIcZp-ykn3_F9_ph6EYyu3dUZjPAcQ$
>
> Cheers,
> Dmitri.
>
> On Mon, Mar 9, 2026 at 3:48 PM EJ Wang <ej...@gmail.com>
> wrote:
>
> > Hi Anand,
> >
> > I think the proposal is moving in a better direction, especially on the
> > Events side, and I appreciate the iteration so far. That said, I still
> have
> > some concerns about the Metrics side, but they are less about individual
> > parameters or endpoint shape, and more about product boundary.
> >
> > 2 cents: I wonder if it would help to evaluate the Events API and Metrics
> > API a bit more independently.
> >
> > The Events side feels relatively close to Polaris' catalog/change-log
> > scope. It is easier to justify as part of the core/community surface,
> > especially if the goal is to expose completed catalog mutations in a way
> > that aligns with Iceberg-style events.
> >
> > The Metrics side feels different to me. Once we start adding more and
> more
> > type-specific filters, query semantics, and schema shape for individual
> > metric families, it seems easy for Polaris to drift toward a built-in
> > observability backend. My bias would be for Polaris to support a smaller
> > set of community-recognized built-in metrics well, while providing good
> > extensibility points for deployments that want richer querying,
> > visualization, or use-case-specific metrics.
> >
> > Related to that, I am not yet convinced the current metrics model is
> > generic enough as a long-term direction. Even after consolidating to a
> > single endpoint, the design still feels fairly tied to the current
> > scan/commit shape. I worry that otherwise each new metric family will
> keep
> > pulling us into more storage/schema/API reshaping inside Polaris core.
> > So the framing question I would suggest is something like:
> > > What is the minimal built-in metrics surface Polaris should own in
> core,
> > and where should we instead rely on extensibility / sink-export /
> > plugin-style integration?
> >
> > For me, getting that boundary right matters more than settling every
> query
> > parameter detail first.
> >
> > -ej
> >
> > On Tue, Mar 3, 2026 at 12:29 PM Anand Kumar Sankaran via dev <
> > dev@polaris.apache.org> wrote:
> >
> > > Hi Yufei and Dmitri,
> > >
> > > Here is a proposal for the REST endpoints for metrics and events.
> > >
> > >
> https://urldefense.com/v3/__https://github.com/apache/polaris/pull/3924/changes__;!!Iz9xO38YGHZK!8KJ0uv4jK3mxZP4nYFrL1hZ0fMkQvoVEAJa8t9LBCzVtm_PWVFGQfIcZp-ykn3_F9_ph6EYyu3fXDjRYWA$
> > >
> > > I did not see any precursors for raising a PR for proposals, so trying
> > > this.  Please let me know what you think.
> > >
> > > -
> > > Anand
> > >
> > > From: Anand Kumar Sankaran <an...@workday.com>
> > > Date: Monday, March 2, 2026 at 10:25 AM
> > > To: dev@polaris.apache.org <de...@polaris.apache.org>
> > > Subject: Re: Polaris Telemetry and Audit Trail
> > >
> > > About the REST API, based on my use cases:
> > >
> > >
> > >   1.
> > > I want to be able to query commit metrics to track files added /
> removed
> > > per commit, along with record counts. The ingestion pipeline that
> writes
> > > this data is owned by us and we are guaranteed to write this
> information
> > > for each write.
> > >   2.
> > > I want to be able to query scan metrics for read. I understand clients
> do
> > > not fulfill this requirement.
> > >   3.
> > > I want to be able to query the events table (events are persisted) -
> this
> > > may supersede #2, I am not sure yet.
> > >
> > > All this information is in the JDBC based persistence model and is
> > > persisted in the metastore. I currently don’t have a need to query
> > > prometheus or open telemetry. I do publish some events to Prometheus
> and
> > > they are forwarded to our dashboards elsewhere.
> > >
> > > About the CLI utilities, I meant the admin user utilities. In one of
> the
> > > earliest drafts of my proposal, Prashant mentioned that the metrics
> > tables
> > > can grow indefinitely and that a similar problem exists with the events
> > > table as well. We discussed that cleaning up of old records from both
> > > metrics tables and events tables can be done via a CLI utility.
> > >
> > > I see that Yufei has covered the discussion about datasources.
> > >
> > > -
> > > Anand
> > >
> > >
> > >
> > > From: Yufei Gu <fl...@gmail.com>
> > > Date: Friday, February 27, 2026 at 9:54 PM
> > > To: dev@polaris.apache.org <de...@polaris.apache.org>
> > > Subject: Re: Polaris Telemetry and Audit Trail
> > >
> > > This Message Is From an External Sender
> > > This message came from outside your organization.
> > > Report Suspicious<
> > >
> >
> https://us-phishalarm-ewt.proofpoint.com/EWT/v1/Iz9xO38YGHZK!YhNDZABkHi1B699ote2uMwpOZw8i0QMCGO2Szc-HshuABGhGvwPJcymE6G2oUUxtS8xDkSrtGTPm_I3QnVDHoLMk50m9v8z_nZKTkd-bnVUbreF1u0WnfV_X5eYevZl_$
> > > >
> > >
> > >
> > > As I mentioned in
> > >
> >
> https://urldefense.com/v3/__https://github.com/apache/polaris/issues/3890__;!!Iz9xO38YGHZK!5EuyFFkk3vhRWVIRvQAWBSQfpJkTMA9HxugzDwXmN0LPPqhEFxYkFRGVhtb8AqUwXtDh2OplcMnbMDHKOxrvDU0$
> >,
> > > supporting
> > > multiple data sources is not a trivial change. I would strongly
> recommend
> > > starting with a design document to carefully evaluate the architectural
> > > implications and long term impact.
> > >
> > > A REST endpoint to query metrics seems reasonable given the current
> JDBC
> > > based persistence model. That said, we may also consider alternative
> > > storage models. For example, if we later adopt a time series system
> such
> > as
> > > Prometheus to store metrics, the query model and access patterns would
> be
> > > fundamentally different. Designing the REST API without considering
> these
> > > potential evolutions may limit flexibility. I'd suggest to start with
> the
> > > use case.
> > >
> > > Yufei
> > >
> > >
> > > On Fri, Feb 27, 2026 at 3:42 PM Dmitri Bourlatchkov <di...@apache.org>
> > > wrote:
> > >
> > > > Hi Anand,
> > > >
> > > > Sharing my view... subject to discussion:
> > > >
> > > > 1. Adding non-IRC REST API to Polaris is perfectly fine.
> > > >
> > > > Figuring out specific endpoint URIs and payloads might require a few
> > > > roundtrips, so opening a separate thread for that might be best.
> > > > Contributors commonly create Google Docs for new API proposals too
> > (they
> > > > fairly easy to update as the email discussion progresses).
> > > >
> > > > There was a suggestion to try Markdown (with PRs) for proposals [1]
> ...
> > > > feel free to give it a try if you are comfortable with that.
> > > >
> > > > 2. Could you clarify whether you mean end user utilities or admin
> user
> > > > utilities? In the latter case those might be more suitable for the
> > Admin
> > > > CLI (java) not the Python CLI, IMHO.
> > > >
> > > > Why would these utilities be common with events? IMHO, event use
> cases
> > > are
> > > > distinct from scan/commit metrics.
> > > >
> > > > 3. I'd prefer separating metrics persistence from MetaStore
> persistence
> > > at
> > > > the code level, so that they could be mixed and matched
> independently.
> > > The
> > > > separate datasource question will become a non-issue with that
> > approach,
> > > I
> > > > guess.
> > > >
> > > > The rationale for separating scan metrics and metastore persistence
> is
> > > that
> > > > "cascading deletes" between them are hardly ever required.
> Furthermore,
> > > the
> > > > data and query patterns are very different so different technologies
> > > might
> > > > be beneficial in each case.
> > > >
> > > > [1]
> > >
> >
> https://urldefense.com/v3/__https://lists.apache.org/thread/yto2wp982t43h1mqjwnslswhws5z47cy__;!!Iz9xO38YGHZK!5EuyFFkk3vhRWVIRvQAWBSQfpJkTMA9HxugzDwXmN0LPPqhEFxYkFRGVhtb8AqUwXtDh2OplcMnbMDHKxYDakNU$
> >> >
> > > > Cheers,
> > > > Dmitri.
> > > >
> > > > On Fri, Feb 27, 2026 at 6:19 PM Anand Kumar Sankaran via dev <
> > > > dev@polaris.apache.org> wrote:
> > > >
> > > > > Thanks all. This PR is merged now.
> > > > >
> > > > > Here are the follow-up features / work needed.  These were all part
> > of
> > > > the
> > > > > merged PR at some point in time and were removed to reduce scope.
> > > > >
> > > > > Please let me know what you think.
> > > > >
> > > > >
> > > > >   1.  A REST API to paginate through table metrics. This will be
> > > non-IRC
> > > > > standard addition.
> > > > >   2.  Utilities for managing old records, should be common with
> > events.
> > > > > There was some discussion that it belongs to the CLI.
> > > > >   3.  Separate datasource (metrics, events, even other tables?).
> > > > >
> > > > >
> > > > > Anything else?
> > > > >
> > > > > -
> > > > > Anand
> > > > >
> > > > >
> > > >
> > >
> > >
> >
>
>

Re: Proposal for REST endpoints for table metrics and events

[Anand Kumar Sankaran via dev <de...@polaris.apache.org>]

Hi EJ Wang and Dmitri,

I addressed all your concerns about the proposal, in particular https://github.com/apache/polaris/pull/3924#discussion_r2908317696.

Does this address your concerns?

-
Anand

From: Dmitri Bourlatchkov <di...@apache.org>
Date: Monday, March 9, 2026 at 1:17 PM
To: dev@polaris.apache.org <de...@polaris.apache.org>
Subject: Re: Proposal for REST endpoints for table metrics and events

This Message Is From an External Sender
This message came from outside your organization.
Report Suspicious<https://us-phishalarm-ewt.proofpoint.com/EWT/v1/Iz9xO38YGHZK!YhNDZAGr2cumYdtJ_UTm0gR9PfI_-PwSpR_GtNr1uVQ_xo-s2AskvUmbkLZ-C5V8eOKN-omus47On4k4hfFo-0G7CHMLwVjEego-rZrPuepAybX7DP8Ua0VNSrsZ83C4$>


Hi EJ,

You make good points about the metrics API extensibility and evolution.

However, we need to consider practical aspects too. Anand appears to have
some specific use cases in mind, and I assume his proposal addresses them.

Starting with an API + implementation that works for some real
world applications will validate the feature's usability.

We can revamp the API completely in its v2 after v1 is merged. New major
API versions do not have to be backward-compatible with older versions of
the same API [1].

In my personal experience, a v1 API can hardly be expected to cover all use
cases and extensions well. We can certainly take a bit more time to polish
it, but from my POV it might be best to iterate in terms of API versions
rather than on unmerged commits in the initial proposal. Just my 2 cents :)

That said, we should flag the new APIs in this proposal as "beta"... at
least initially (which is the usual practice in Polaris).

> I wonder if it would help to evaluate the Events API and Metrics API a
bit more independently.

That makes sense to me. However, the current proposal progressed a lot
since its initial submission and contained both APIs. I would not want to
lose this momentum.

It might still be advisable to implement the events and metrics APIs
separately and gather additional feedback at that time.

[1] https://urldefense.com/v3/__https://polaris.apache.org/in-dev/unreleased/evolution/__;!!Iz9xO38YGHZK!8KJ0uv4jK3mxZP4nYFrL1hZ0fMkQvoVEAJa8t9LBCzVtm_PWVFGQfIcZp-ykn3_F9_ph6EYyu3dUZjPAcQ$

Cheers,
Dmitri.

On Mon, Mar 9, 2026 at 3:48 PM EJ Wang <ej...@gmail.com>
wrote:

> Hi Anand,
>
> I think the proposal is moving in a better direction, especially on the
> Events side, and I appreciate the iteration so far. That said, I still have
> some concerns about the Metrics side, but they are less about individual
> parameters or endpoint shape, and more about product boundary.
>
> 2 cents: I wonder if it would help to evaluate the Events API and Metrics
> API a bit more independently.
>
> The Events side feels relatively close to Polaris' catalog/change-log
> scope. It is easier to justify as part of the core/community surface,
> especially if the goal is to expose completed catalog mutations in a way
> that aligns with Iceberg-style events.
>
> The Metrics side feels different to me. Once we start adding more and more
> type-specific filters, query semantics, and schema shape for individual
> metric families, it seems easy for Polaris to drift toward a built-in
> observability backend. My bias would be for Polaris to support a smaller
> set of community-recognized built-in metrics well, while providing good
> extensibility points for deployments that want richer querying,
> visualization, or use-case-specific metrics.
>
> Related to that, I am not yet convinced the current metrics model is
> generic enough as a long-term direction. Even after consolidating to a
> single endpoint, the design still feels fairly tied to the current
> scan/commit shape. I worry that otherwise each new metric family will keep
> pulling us into more storage/schema/API reshaping inside Polaris core.
> So the framing question I would suggest is something like:
> > What is the minimal built-in metrics surface Polaris should own in core,
> and where should we instead rely on extensibility / sink-export /
> plugin-style integration?
>
> For me, getting that boundary right matters more than settling every query
> parameter detail first.
>
> -ej
>
> On Tue, Mar 3, 2026 at 12:29 PM Anand Kumar Sankaran via dev <
> dev@polaris.apache.org> wrote:
>
> > Hi Yufei and Dmitri,
> >
> > Here is a proposal for the REST endpoints for metrics and events.
> >
> > https://urldefense.com/v3/__https://github.com/apache/polaris/pull/3924/changes__;!!Iz9xO38YGHZK!8KJ0uv4jK3mxZP4nYFrL1hZ0fMkQvoVEAJa8t9LBCzVtm_PWVFGQfIcZp-ykn3_F9_ph6EYyu3fXDjRYWA$
> >
> > I did not see any precursors for raising a PR for proposals, so trying
> > this.  Please let me know what you think.
> >
> > -
> > Anand
> >
> > From: Anand Kumar Sankaran <an...@workday.com>
> > Date: Monday, March 2, 2026 at 10:25 AM
> > To: dev@polaris.apache.org <de...@polaris.apache.org>
> > Subject: Re: Polaris Telemetry and Audit Trail
> >
> > About the REST API, based on my use cases:
> >
> >
> >   1.
> > I want to be able to query commit metrics to track files added / removed
> > per commit, along with record counts. The ingestion pipeline that writes
> > this data is owned by us and we are guaranteed to write this information
> > for each write.
> >   2.
> > I want to be able to query scan metrics for read. I understand clients do
> > not fulfill this requirement.
> >   3.
> > I want to be able to query the events table (events are persisted) - this
> > may supersede #2, I am not sure yet.
> >
> > All this information is in the JDBC based persistence model and is
> > persisted in the metastore. I currently don’t have a need to query
> > prometheus or open telemetry. I do publish some events to Prometheus and
> > they are forwarded to our dashboards elsewhere.
> >
> > About the CLI utilities, I meant the admin user utilities. In one of the
> > earliest drafts of my proposal, Prashant mentioned that the metrics
> tables
> > can grow indefinitely and that a similar problem exists with the events
> > table as well. We discussed that cleaning up of old records from both
> > metrics tables and events tables can be done via a CLI utility.
> >
> > I see that Yufei has covered the discussion about datasources.
> >
> > -
> > Anand
> >
> >
> >
> > From: Yufei Gu <fl...@gmail.com>
> > Date: Friday, February 27, 2026 at 9:54 PM
> > To: dev@polaris.apache.org <de...@polaris.apache.org>
> > Subject: Re: Polaris Telemetry and Audit Trail
> >
> > This Message Is From an External Sender
> > This message came from outside your organization.
> > Report Suspicious<
> >
> https://us-phishalarm-ewt.proofpoint.com/EWT/v1/Iz9xO38YGHZK!YhNDZABkHi1B699ote2uMwpOZw8i0QMCGO2Szc-HshuABGhGvwPJcymE6G2oUUxtS8xDkSrtGTPm_I3QnVDHoLMk50m9v8z_nZKTkd-bnVUbreF1u0WnfV_X5eYevZl_$
> > >
> >
> >
> > As I mentioned in
> >
> https://urldefense.com/v3/__https://github.com/apache/polaris/issues/3890__;!!Iz9xO38YGHZK!5EuyFFkk3vhRWVIRvQAWBSQfpJkTMA9HxugzDwXmN0LPPqhEFxYkFRGVhtb8AqUwXtDh2OplcMnbMDHKOxrvDU0$
>,
> > supporting
> > multiple data sources is not a trivial change. I would strongly recommend
> > starting with a design document to carefully evaluate the architectural
> > implications and long term impact.
> >
> > A REST endpoint to query metrics seems reasonable given the current JDBC
> > based persistence model. That said, we may also consider alternative
> > storage models. For example, if we later adopt a time series system such
> as
> > Prometheus to store metrics, the query model and access patterns would be
> > fundamentally different. Designing the REST API without considering these
> > potential evolutions may limit flexibility. I'd suggest to start with the
> > use case.
> >
> > Yufei
> >
> >
> > On Fri, Feb 27, 2026 at 3:42 PM Dmitri Bourlatchkov <di...@apache.org>
> > wrote:
> >
> > > Hi Anand,
> > >
> > > Sharing my view... subject to discussion:
> > >
> > > 1. Adding non-IRC REST API to Polaris is perfectly fine.
> > >
> > > Figuring out specific endpoint URIs and payloads might require a few
> > > roundtrips, so opening a separate thread for that might be best.
> > > Contributors commonly create Google Docs for new API proposals too
> (they
> > > fairly easy to update as the email discussion progresses).
> > >
> > > There was a suggestion to try Markdown (with PRs) for proposals [1] ...
> > > feel free to give it a try if you are comfortable with that.
> > >
> > > 2. Could you clarify whether you mean end user utilities or admin user
> > > utilities? In the latter case those might be more suitable for the
> Admin
> > > CLI (java) not the Python CLI, IMHO.
> > >
> > > Why would these utilities be common with events? IMHO, event use cases
> > are
> > > distinct from scan/commit metrics.
> > >
> > > 3. I'd prefer separating metrics persistence from MetaStore persistence
> > at
> > > the code level, so that they could be mixed and matched independently.
> > The
> > > separate datasource question will become a non-issue with that
> approach,
> > I
> > > guess.
> > >
> > > The rationale for separating scan metrics and metastore persistence is
> > that
> > > "cascading deletes" between them are hardly ever required. Furthermore,
> > the
> > > data and query patterns are very different so different technologies
> > might
> > > be beneficial in each case.
> > >
> > > [1]
> >
> https://urldefense.com/v3/__https://lists.apache.org/thread/yto2wp982t43h1mqjwnslswhws5z47cy__;!!Iz9xO38YGHZK!5EuyFFkk3vhRWVIRvQAWBSQfpJkTMA9HxugzDwXmN0LPPqhEFxYkFRGVhtb8AqUwXtDh2OplcMnbMDHKxYDakNU$
>> >
> > > Cheers,
> > > Dmitri.
> > >
> > > On Fri, Feb 27, 2026 at 6:19 PM Anand Kumar Sankaran via dev <
> > > dev@polaris.apache.org> wrote:
> > >
> > > > Thanks all. This PR is merged now.
> > > >
> > > > Here are the follow-up features / work needed.  These were all part
> of
> > > the
> > > > merged PR at some point in time and were removed to reduce scope.
> > > >
> > > > Please let me know what you think.
> > > >
> > > >
> > > >   1.  A REST API to paginate through table metrics. This will be
> > non-IRC
> > > > standard addition.
> > > >   2.  Utilities for managing old records, should be common with
> events.
> > > > There was some discussion that it belongs to the CLI.
> > > >   3.  Separate datasource (metrics, events, even other tables?).
> > > >
> > > >
> > > > Anything else?
> > > >
> > > > -
> > > > Anand
> > > >
> > > >
> > >
> >
> >
>

Re: Proposal for REST endpoints for table metrics and events

[Dmitri Bourlatchkov <di...@apache.org>]

Hi EJ,

You make good points about the metrics API extensibility and evolution.

However, we need to consider practical aspects too. Anand appears to have
some specific use cases in mind, and I assume his proposal addresses them.

Starting with an API + implementation that works for some real
world applications will validate the feature's usability.

We can revamp the API completely in its v2 after v1 is merged. New major
API versions do not have to be backward-compatible with older versions of
the same API [1].

In my personal experience, a v1 API can hardly be expected to cover all use
cases and extensions well. We can certainly take a bit more time to polish
it, but from my POV it might be best to iterate in terms of API versions
rather than on unmerged commits in the initial proposal. Just my 2 cents :)

That said, we should flag the new APIs in this proposal as "beta"... at
least initially (which is the usual practice in Polaris).

> I wonder if it would help to evaluate the Events API and Metrics API a
bit more independently.

That makes sense to me. However, the current proposal progressed a lot
since its initial submission and contained both APIs. I would not want to
lose this momentum.

It might still be advisable to implement the events and metrics APIs
separately and gather additional feedback at that time.

[1] https://polaris.apache.org/in-dev/unreleased/evolution/

Cheers,
Dmitri.

On Mon, Mar 9, 2026 at 3:48 PM EJ Wang <ej...@gmail.com>
wrote:

> Hi Anand,
>
> I think the proposal is moving in a better direction, especially on the
> Events side, and I appreciate the iteration so far. That said, I still have
> some concerns about the Metrics side, but they are less about individual
> parameters or endpoint shape, and more about product boundary.
>
> 2 cents: I wonder if it would help to evaluate the Events API and Metrics
> API a bit more independently.
>
> The Events side feels relatively close to Polaris' catalog/change-log
> scope. It is easier to justify as part of the core/community surface,
> especially if the goal is to expose completed catalog mutations in a way
> that aligns with Iceberg-style events.
>
> The Metrics side feels different to me. Once we start adding more and more
> type-specific filters, query semantics, and schema shape for individual
> metric families, it seems easy for Polaris to drift toward a built-in
> observability backend. My bias would be for Polaris to support a smaller
> set of community-recognized built-in metrics well, while providing good
> extensibility points for deployments that want richer querying,
> visualization, or use-case-specific metrics.
>
> Related to that, I am not yet convinced the current metrics model is
> generic enough as a long-term direction. Even after consolidating to a
> single endpoint, the design still feels fairly tied to the current
> scan/commit shape. I worry that otherwise each new metric family will keep
> pulling us into more storage/schema/API reshaping inside Polaris core.
> So the framing question I would suggest is something like:
> > What is the minimal built-in metrics surface Polaris should own in core,
> and where should we instead rely on extensibility / sink-export /
> plugin-style integration?
>
> For me, getting that boundary right matters more than settling every query
> parameter detail first.
>
> -ej
>
> On Tue, Mar 3, 2026 at 12:29 PM Anand Kumar Sankaran via dev <
> dev@polaris.apache.org> wrote:
>
> > Hi Yufei and Dmitri,
> >
> > Here is a proposal for the REST endpoints for metrics and events.
> >
> > https://github.com/apache/polaris/pull/3924/changes
> >
> > I did not see any precursors for raising a PR for proposals, so trying
> > this.  Please let me know what you think.
> >
> > -
> > Anand
> >
> > From: Anand Kumar Sankaran <an...@workday.com>
> > Date: Monday, March 2, 2026 at 10:25 AM
> > To: dev@polaris.apache.org <de...@polaris.apache.org>
> > Subject: Re: Polaris Telemetry and Audit Trail
> >
> > About the REST API, based on my use cases:
> >
> >
> >   1.
> > I want to be able to query commit metrics to track files added / removed
> > per commit, along with record counts. The ingestion pipeline that writes
> > this data is owned by us and we are guaranteed to write this information
> > for each write.
> >   2.
> > I want to be able to query scan metrics for read. I understand clients do
> > not fulfill this requirement.
> >   3.
> > I want to be able to query the events table (events are persisted) - this
> > may supersede #2, I am not sure yet.
> >
> > All this information is in the JDBC based persistence model and is
> > persisted in the metastore. I currently don’t have a need to query
> > prometheus or open telemetry. I do publish some events to Prometheus and
> > they are forwarded to our dashboards elsewhere.
> >
> > About the CLI utilities, I meant the admin user utilities. In one of the
> > earliest drafts of my proposal, Prashant mentioned that the metrics
> tables
> > can grow indefinitely and that a similar problem exists with the events
> > table as well. We discussed that cleaning up of old records from both
> > metrics tables and events tables can be done via a CLI utility.
> >
> > I see that Yufei has covered the discussion about datasources.
> >
> > -
> > Anand
> >
> >
> >
> > From: Yufei Gu <fl...@gmail.com>
> > Date: Friday, February 27, 2026 at 9:54 PM
> > To: dev@polaris.apache.org <de...@polaris.apache.org>
> > Subject: Re: Polaris Telemetry and Audit Trail
> >
> > This Message Is From an External Sender
> > This message came from outside your organization.
> > Report Suspicious<
> >
> https://us-phishalarm-ewt.proofpoint.com/EWT/v1/Iz9xO38YGHZK!YhNDZABkHi1B699ote2uMwpOZw8i0QMCGO2Szc-HshuABGhGvwPJcymE6G2oUUxtS8xDkSrtGTPm_I3QnVDHoLMk50m9v8z_nZKTkd-bnVUbreF1u0WnfV_X5eYevZl_$
> > >
> >
> >
> > As I mentioned in
> >
> https://urldefense.com/v3/__https://github.com/apache/polaris/issues/3890__;!!Iz9xO38YGHZK!5EuyFFkk3vhRWVIRvQAWBSQfpJkTMA9HxugzDwXmN0LPPqhEFxYkFRGVhtb8AqUwXtDh2OplcMnbMDHKOxrvDU0$
> ,
> > supporting
> > multiple data sources is not a trivial change. I would strongly recommend
> > starting with a design document to carefully evaluate the architectural
> > implications and long term impact.
> >
> > A REST endpoint to query metrics seems reasonable given the current JDBC
> > based persistence model. That said, we may also consider alternative
> > storage models. For example, if we later adopt a time series system such
> as
> > Prometheus to store metrics, the query model and access patterns would be
> > fundamentally different. Designing the REST API without considering these
> > potential evolutions may limit flexibility. I'd suggest to start with the
> > use case.
> >
> > Yufei
> >
> >
> > On Fri, Feb 27, 2026 at 3:42 PM Dmitri Bourlatchkov <di...@apache.org>
> > wrote:
> >
> > > Hi Anand,
> > >
> > > Sharing my view... subject to discussion:
> > >
> > > 1. Adding non-IRC REST API to Polaris is perfectly fine.
> > >
> > > Figuring out specific endpoint URIs and payloads might require a few
> > > roundtrips, so opening a separate thread for that might be best.
> > > Contributors commonly create Google Docs for new API proposals too
> (they
> > > fairly easy to update as the email discussion progresses).
> > >
> > > There was a suggestion to try Markdown (with PRs) for proposals [1] ...
> > > feel free to give it a try if you are comfortable with that.
> > >
> > > 2. Could you clarify whether you mean end user utilities or admin user
> > > utilities? In the latter case those might be more suitable for the
> Admin
> > > CLI (java) not the Python CLI, IMHO.
> > >
> > > Why would these utilities be common with events? IMHO, event use cases
> > are
> > > distinct from scan/commit metrics.
> > >
> > > 3. I'd prefer separating metrics persistence from MetaStore persistence
> > at
> > > the code level, so that they could be mixed and matched independently.
> > The
> > > separate datasource question will become a non-issue with that
> approach,
> > I
> > > guess.
> > >
> > > The rationale for separating scan metrics and metastore persistence is
> > that
> > > "cascading deletes" between them are hardly ever required. Furthermore,
> > the
> > > data and query patterns are very different so different technologies
> > might
> > > be beneficial in each case.
> > >
> > > [1]
> >
> https://urldefense.com/v3/__https://lists.apache.org/thread/yto2wp982t43h1mqjwnslswhws5z47cy__;!!Iz9xO38YGHZK!5EuyFFkk3vhRWVIRvQAWBSQfpJkTMA9HxugzDwXmN0LPPqhEFxYkFRGVhtb8AqUwXtDh2OplcMnbMDHKxYDakNU$
> > >
> > > Cheers,
> > > Dmitri.
> > >
> > > On Fri, Feb 27, 2026 at 6:19 PM Anand Kumar Sankaran via dev <
> > > dev@polaris.apache.org> wrote:
> > >
> > > > Thanks all. This PR is merged now.
> > > >
> > > > Here are the follow-up features / work needed.  These were all part
> of
> > > the
> > > > merged PR at some point in time and were removed to reduce scope.
> > > >
> > > > Please let me know what you think.
> > > >
> > > >
> > > >   1.  A REST API to paginate through table metrics. This will be
> > non-IRC
> > > > standard addition.
> > > >   2.  Utilities for managing old records, should be common with
> events.
> > > > There was some discussion that it belongs to the CLI.
> > > >   3.  Separate datasource (metrics, events, even other tables?).
> > > >
> > > >
> > > > Anything else?
> > > >
> > > > -
> > > > Anand
> > > >
> > > >
> > >
> >
> >
>

Re: Proposal for REST endpoints for table metrics and events

[EJ Wang <ej...@gmail.com>]

Hi Anand,

I think the proposal is moving in a better direction, especially on the
Events side, and I appreciate the iteration so far. That said, I still have
some concerns about the Metrics side, but they are less about individual
parameters or endpoint shape, and more about product boundary.

2 cents: I wonder if it would help to evaluate the Events API and Metrics
API a bit more independently.

The Events side feels relatively close to Polaris' catalog/change-log
scope. It is easier to justify as part of the core/community surface,
especially if the goal is to expose completed catalog mutations in a way
that aligns with Iceberg-style events.

The Metrics side feels different to me. Once we start adding more and more
type-specific filters, query semantics, and schema shape for individual
metric families, it seems easy for Polaris to drift toward a built-in
observability backend. My bias would be for Polaris to support a smaller
set of community-recognized built-in metrics well, while providing good
extensibility points for deployments that want richer querying,
visualization, or use-case-specific metrics.

Related to that, I am not yet convinced the current metrics model is
generic enough as a long-term direction. Even after consolidating to a
single endpoint, the design still feels fairly tied to the current
scan/commit shape. I worry that otherwise each new metric family will keep
pulling us into more storage/schema/API reshaping inside Polaris core.
So the framing question I would suggest is something like:
> What is the minimal built-in metrics surface Polaris should own in core,
and where should we instead rely on extensibility / sink-export /
plugin-style integration?

For me, getting that boundary right matters more than settling every query
parameter detail first.

-ej

On Tue, Mar 3, 2026 at 12:29 PM Anand Kumar Sankaran via dev <
dev@polaris.apache.org> wrote:

> Hi Yufei and Dmitri,
>
> Here is a proposal for the REST endpoints for metrics and events.
>
> https://github.com/apache/polaris/pull/3924/changes
>
> I did not see any precursors for raising a PR for proposals, so trying
> this.  Please let me know what you think.
>
> -
> Anand
>
> From: Anand Kumar Sankaran <an...@workday.com>
> Date: Monday, March 2, 2026 at 10:25 AM
> To: dev@polaris.apache.org <de...@polaris.apache.org>
> Subject: Re: Polaris Telemetry and Audit Trail
>
> About the REST API, based on my use cases:
>
>
>   1.
> I want to be able to query commit metrics to track files added / removed
> per commit, along with record counts. The ingestion pipeline that writes
> this data is owned by us and we are guaranteed to write this information
> for each write.
>   2.
> I want to be able to query scan metrics for read. I understand clients do
> not fulfill this requirement.
>   3.
> I want to be able to query the events table (events are persisted) - this
> may supersede #2, I am not sure yet.
>
> All this information is in the JDBC based persistence model and is
> persisted in the metastore. I currently don’t have a need to query
> prometheus or open telemetry. I do publish some events to Prometheus and
> they are forwarded to our dashboards elsewhere.
>
> About the CLI utilities, I meant the admin user utilities. In one of the
> earliest drafts of my proposal, Prashant mentioned that the metrics tables
> can grow indefinitely and that a similar problem exists with the events
> table as well. We discussed that cleaning up of old records from both
> metrics tables and events tables can be done via a CLI utility.
>
> I see that Yufei has covered the discussion about datasources.
>
> -
> Anand
>
>
>
> From: Yufei Gu <fl...@gmail.com>
> Date: Friday, February 27, 2026 at 9:54 PM
> To: dev@polaris.apache.org <de...@polaris.apache.org>
> Subject: Re: Polaris Telemetry and Audit Trail
>
> This Message Is From an External Sender
> This message came from outside your organization.
> Report Suspicious<
> https://us-phishalarm-ewt.proofpoint.com/EWT/v1/Iz9xO38YGHZK!YhNDZABkHi1B699ote2uMwpOZw8i0QMCGO2Szc-HshuABGhGvwPJcymE6G2oUUxtS8xDkSrtGTPm_I3QnVDHoLMk50m9v8z_nZKTkd-bnVUbreF1u0WnfV_X5eYevZl_$
> >
>
>
> As I mentioned in
> https://urldefense.com/v3/__https://github.com/apache/polaris/issues/3890__;!!Iz9xO38YGHZK!5EuyFFkk3vhRWVIRvQAWBSQfpJkTMA9HxugzDwXmN0LPPqhEFxYkFRGVhtb8AqUwXtDh2OplcMnbMDHKOxrvDU0$,
> supporting
> multiple data sources is not a trivial change. I would strongly recommend
> starting with a design document to carefully evaluate the architectural
> implications and long term impact.
>
> A REST endpoint to query metrics seems reasonable given the current JDBC
> based persistence model. That said, we may also consider alternative
> storage models. For example, if we later adopt a time series system such as
> Prometheus to store metrics, the query model and access patterns would be
> fundamentally different. Designing the REST API without considering these
> potential evolutions may limit flexibility. I'd suggest to start with the
> use case.
>
> Yufei
>
>
> On Fri, Feb 27, 2026 at 3:42 PM Dmitri Bourlatchkov <di...@apache.org>
> wrote:
>
> > Hi Anand,
> >
> > Sharing my view... subject to discussion:
> >
> > 1. Adding non-IRC REST API to Polaris is perfectly fine.
> >
> > Figuring out specific endpoint URIs and payloads might require a few
> > roundtrips, so opening a separate thread for that might be best.
> > Contributors commonly create Google Docs for new API proposals too (they
> > fairly easy to update as the email discussion progresses).
> >
> > There was a suggestion to try Markdown (with PRs) for proposals [1] ...
> > feel free to give it a try if you are comfortable with that.
> >
> > 2. Could you clarify whether you mean end user utilities or admin user
> > utilities? In the latter case those might be more suitable for the Admin
> > CLI (java) not the Python CLI, IMHO.
> >
> > Why would these utilities be common with events? IMHO, event use cases
> are
> > distinct from scan/commit metrics.
> >
> > 3. I'd prefer separating metrics persistence from MetaStore persistence
> at
> > the code level, so that they could be mixed and matched independently.
> The
> > separate datasource question will become a non-issue with that approach,
> I
> > guess.
> >
> > The rationale for separating scan metrics and metastore persistence is
> that
> > "cascading deletes" between them are hardly ever required. Furthermore,
> the
> > data and query patterns are very different so different technologies
> might
> > be beneficial in each case.
> >
> > [1]
> https://urldefense.com/v3/__https://lists.apache.org/thread/yto2wp982t43h1mqjwnslswhws5z47cy__;!!Iz9xO38YGHZK!5EuyFFkk3vhRWVIRvQAWBSQfpJkTMA9HxugzDwXmN0LPPqhEFxYkFRGVhtb8AqUwXtDh2OplcMnbMDHKxYDakNU$
> >
> > Cheers,
> > Dmitri.
> >
> > On Fri, Feb 27, 2026 at 6:19 PM Anand Kumar Sankaran via dev <
> > dev@polaris.apache.org> wrote:
> >
> > > Thanks all. This PR is merged now.
> > >
> > > Here are the follow-up features / work needed.  These were all part of
> > the
> > > merged PR at some point in time and were removed to reduce scope.
> > >
> > > Please let me know what you think.
> > >
> > >
> > >   1.  A REST API to paginate through table metrics. This will be
> non-IRC
> > > standard addition.
> > >   2.  Utilities for managing old records, should be common with events.
> > > There was some discussion that it belongs to the CLI.
> > >   3.  Separate datasource (metrics, events, even other tables?).
> > >
> > >
> > > Anything else?
> > >
> > > -
> > > Anand
> > >
> > >
> >
>
>

Re: Proposal for REST endpoints for table metrics and events

[Anand Kumar Sankaran via dev <de...@polaris.apache.org>]

Adnan, Nandor and Dmitry,

Thanks for your review. I addressed them.

-
Anand

From: Anand Kumar Sankaran via dev <de...@polaris.apache.org>
Date: Tuesday, March 3, 2026 at 12:30 PM
To: dev@polaris.apache.org <de...@polaris.apache.org>
Cc: Anand Kumar Sankaran <an...@workday.com>
Subject: Proposal for REST endpoints for table metrics and events

This Message Is From an External Sender
This message came from outside your organization.
Report Suspicious<https://us-phishalarm-ewt.proofpoint.com/EWT/v1/Iz9xO38YGHZK!YJNDRa5Ie2LtT1kglU0n36o7TOU2gxUZCM4nmw2zFpY25tphgwGc3c82amfAXNSuyxT3ITC8eJNarpRGZEedUnvPfC8z1X_upobYNrI_-ISdBKu-DSh27LnKUcwaXX2u0OwZsYGsSAvF_FLR$>


Hi Yufei and Dmitri,

Here is a proposal for the REST endpoints for metrics and events.

https://urldefense.com/v3/__https://github.com/apache/polaris/pull/3924/changes__;!!Iz9xO38YGHZK!5shRveArde7k-0FXp8zTTE1rQVo3SkIuP2jxTvGxJMar6yp-Lm0pvw9fxcgaDNxJigO6fOC1UNoZ858b6sMDPw$

I did not see any precursors for raising a PR for proposals, so trying this.  Please let me know what you think.

-
Anand

From: Anand Kumar Sankaran <an...@workday.com>
Date: Monday, March 2, 2026 at 10:25 AM
To: dev@polaris.apache.org <de...@polaris.apache.org>
Subject: Re: Polaris Telemetry and Audit Trail

About the REST API, based on my use cases:


  1.
I want to be able to query commit metrics to track files added / removed per commit, along with record counts. The ingestion pipeline that writes this data is owned by us and we are guaranteed to write this information for each write.
  2.
I want to be able to query scan metrics for read. I understand clients do not fulfill this requirement.
  3.
I want to be able to query the events table (events are persisted) - this may supersede #2, I am not sure yet.

All this information is in the JDBC based persistence model and is persisted in the metastore. I currently don’t have a need to query prometheus or open telemetry. I do publish some events to Prometheus and they are forwarded to our dashboards elsewhere.

About the CLI utilities, I meant the admin user utilities. In one of the earliest drafts of my proposal, Prashant mentioned that the metrics tables can grow indefinitely and that a similar problem exists with the events table as well. We discussed that cleaning up of old records from both metrics tables and events tables can be done via a CLI utility.

I see that Yufei has covered the discussion about datasources.

-
Anand



From: Yufei Gu <fl...@gmail.com>
Date: Friday, February 27, 2026 at 9:54 PM
To: dev@polaris.apache.org <de...@polaris.apache.org>
Subject: Re: Polaris Telemetry and Audit Trail

This Message Is From an External Sender
This message came from outside your organization.
Report Suspicious<https://us-phishalarm-ewt.proofpoint.com/EWT/v1/Iz9xO38YGHZK!YhNDZABkHi1B699ote2uMwpOZw8i0QMCGO2Szc-HshuABGhGvwPJcymE6G2oUUxtS8xDkSrtGTPm_I3QnVDHoLMk50m9v8z_nZKTkd-bnVUbreF1u0WnfV_X5eYevZl_$ >


As I mentioned in https://urldefense.com/v3/__https://github.com/apache/polaris/issues/3890__;!!Iz9xO38YGHZK!5EuyFFkk3vhRWVIRvQAWBSQfpJkTMA9HxugzDwXmN0LPPqhEFxYkFRGVhtb8AqUwXtDh2OplcMnbMDHKOxrvDU0$,supporting
multiple data sources is not a trivial change. I would strongly recommend
starting with a design document to carefully evaluate the architectural
implications and long term impact.

A REST endpoint to query metrics seems reasonable given the current JDBC
based persistence model. That said, we may also consider alternative
storage models. For example, if we later adopt a time series system such as
Prometheus to store metrics, the query model and access patterns would be
fundamentally different. Designing the REST API without considering these
potential evolutions may limit flexibility. I'd suggest to start with the
use case.

Yufei


On Fri, Feb 27, 2026 at 3:42 PM Dmitri Bourlatchkov <di...@apache.org>
wrote:

> Hi Anand,
>
> Sharing my view... subject to discussion:
>
> 1. Adding non-IRC REST API to Polaris is perfectly fine.
>
> Figuring out specific endpoint URIs and payloads might require a few
> roundtrips, so opening a separate thread for that might be best.
> Contributors commonly create Google Docs for new API proposals too (they
> fairly easy to update as the email discussion progresses).
>
> There was a suggestion to try Markdown (with PRs) for proposals [1] ...
> feel free to give it a try if you are comfortable with that.
>
> 2. Could you clarify whether you mean end user utilities or admin user
> utilities? In the latter case those might be more suitable for the Admin
> CLI (java) not the Python CLI, IMHO.
>
> Why would these utilities be common with events? IMHO, event use cases are
> distinct from scan/commit metrics.
>
> 3. I'd prefer separating metrics persistence from MetaStore persistence at
> the code level, so that they could be mixed and matched independently. The
> separate datasource question will become a non-issue with that approach, I
> guess.
>
> The rationale for separating scan metrics and metastore persistence is that
> "cascading deletes" between them are hardly ever required. Furthermore, the
> data and query patterns are very different so different technologies might
> be beneficial in each case.
>
> [1] https://urldefense.com/v3/__https://lists.apache.org/thread/yto2wp982t43h1mqjwnslswhws5z47cy__;!!Iz9xO38YGHZK!5EuyFFkk3vhRWVIRvQAWBSQfpJkTMA9HxugzDwXmN0LPPqhEFxYkFRGVhtb8AqUwXtDh2OplcMnbMDHKxYDakNU$
>
>Cheers,
> Dmitri.
>
> On Fri, Feb 27, 2026 at 6:19 PM Anand Kumar Sankaran via dev <
> dev@polaris.apache.org> wrote:
>
> > Thanks all. This PR is merged now.
> >
> > Here are the follow-up features / work needed.  These were all part of
> the
> > merged PR at some point in time and were removed to reduce scope.
> >
> > Please let me know what you think.
> >
> >
> >   1.  A REST API to paginate through table metrics. This will be non-IRC
> > standard addition.
> >   2.  Utilities for managing old records, should be common with events.
> > There was some discussion that it belongs to the CLI.
> >   3.  Separate datasource (metrics, events, even other tables?).
> >
> >
> > Anything else?
> >
> > -
> > Anand
> >
> >
>

[DISCUSS] Re: Proposal for REST endpoints for table metrics and events

[Anand Kumar Sankaran via dev <de...@polaris.apache.org>]

Hi Yufei,

Can you please take a look at the proposal?  A few folks have reviewed it already. Since there are comments on this already, I would like to continue the markdown approach. Once you are done reviewing, I will be happy to summarize it as a google doc if necessary.

As discussed in the other thread, even if I don’t rely on table metrics and just rely on load_table event - since the load_table event is persisted in the events table in the metastore (I wrote a custom event listener for persisting events), I need a REST API to get the persisted data.

-
Anand

From: Yufei Gu <fl...@gmail.com>
Date: Tuesday, March 3, 2026 at 4:10 PM
To: dev@polaris.apache.org <de...@polaris.apache.org>
Subject: Re: Proposal for REST endpoints for table metrics and events

This Message Is From an External Sender
This message came from outside your organization.
Report Suspicious<https://us-phishalarm-ewt.proofpoint.com/EWT/v1/Iz9xO38YGHZK!YhNDZABkHi1B6hvBdYYm838FDmImuiAUSE78VzyujyOT0ZyDxRKpnXdfCrIfG-isPgAO__U2Oeg5gxuWjVLpaiLfM99fLtXl_fey3sCb_5Woe4NzQK0ywvUVhgSIiPME$>


Thanks for the proposal, Anand. I will take a look. I generally prefer
Google Docs for design doc, esp. for the early design, which usually needs
fast iterations. But I'm OK with a markdown. BTW, we haven't concluded the
dev thread on the proposal, I think we will need a conclusion that, so that
people wont' be confused.

Yufei


On Tue, Mar 3, 2026 at 4:03 PM Dmitri Bourlatchkov <di...@apache.org> wrote:

> Hi Anand,
>
> Thanks for starting this proposal. I see that the linked GH PR gathered
> some review comments. Let's continue evolving it!
>
> From my personal POV the review on the .md doc PR is pretty easy and
> generally resembles regular code reviews in GH, which is a good thing as
> most people here are familiar with that.
>
> For those interested, the ASF mailing lists archive this review at [1].
> That trail is a bit verbose because it shows very long diffs, but it's
> still readable.
>
> In terms of process, I suppose once the PR is approved, we will merge it
> and then proceed with implementation PRs. WDYT?
>
> [1] https://urldefense.com/v3/__https://lists.apache.org/thread/1s28pv4qsmjc6m3lypzmfkgrdom82fjd__;!!Iz9xO38YGHZK!-kXZ-ywdNcPgtaWa6yDyeRKdwv4JKFhVgLAU9RvdKW8rpUYbc0vQj4wpgc3GIpIkNdDRujBzFn5vGBetjBwta-g$
>
> Cheers,
> Dmitri.
>
> On Tue, Mar 3, 2026 at 3:29 PM Anand Kumar Sankaran via dev <
> dev@polaris.apache.org> wrote:
>
> > Hi Yufei and Dmitri,
> >
> > Here is a proposal for the REST endpoints for metrics and events.
> >
> > https://urldefense.com/v3/__https://github.com/apache/polaris/pull/3924/changes__;!!Iz9xO38YGHZK!-kXZ-ywdNcPgtaWa6yDyeRKdwv4JKFhVgLAU9RvdKW8rpUYbc0vQj4wpgc3GIpIkNdDRujBzFn5vGBetaQ8TVgY$
> >
> > I did not see any precursors for raising a PR for proposals, so trying
> > this.  Please let me know what you think.
> >
> > -
> > Anand
> >
> > From: Anand Kumar Sankaran <an...@workday.com>
> > Date: Monday, March 2, 2026 at 10:25 AM
> > To: dev@polaris.apache.org <de...@polaris.apache.org>
> > Subject: Re: Polaris Telemetry and Audit Trail
> >
> > About the REST API, based on my use cases:
> >
> >
> >   1.
> > I want to be able to query commit metrics to track files added / removed
> > per commit, along with record counts. The ingestion pipeline that writes
> > this data is owned by us and we are guaranteed to write this information
> > for each write.
> >   2.
> > I want to be able to query scan metrics for read. I understand clients do
> > not fulfill this requirement.
> >   3.
> > I want to be able to query the events table (events are persisted) - this
> > may supersede #2, I am not sure yet.
> >
> > All this information is in the JDBC based persistence model and is
> > persisted in the metastore. I currently don’t have a need to query
> > prometheus or open telemetry. I do publish some events to Prometheus and
> > they are forwarded to our dashboards elsewhere.
> >
> > About the CLI utilities, I meant the admin user utilities. In one of the
> > earliest drafts of my proposal, Prashant mentioned that the metrics
> tables
> > can grow indefinitely and that a similar problem exists with the events
> > table as well. We discussed that cleaning up of old records from both
> > metrics tables and events tables can be done via a CLI utility.
> >
> > I see that Yufei has covered the discussion about datasources.
> >
> > -
> > Anand
> >
> >
> >
> > From: Yufei Gu <fl...@gmail.com>
> > Date: Friday, February 27, 2026 at 9:54 PM
> > To: dev@polaris.apache.org <de...@polaris.apache.org>
> > Subject: Re: Polaris Telemetry and Audit Trail
> >
> > This Message Is From an External Sender
> > This message came from outside your organization.
> > Report Suspicious<
> >
> https://us-phishalarm-ewt.proofpoint.com/EWT/v1/Iz9xO38YGHZK!YhNDZABkHi1B699ote2uMwpOZw8i0QMCGO2Szc-HshuABGhGvwPJcymE6G2oUUxtS8xDkSrtGTPm_I3QnVDHoLMk50m9v8z_nZKTkd-bnVUbreF1u0WnfV_X5eYevZl_$
> > >
> >
> >
> > As I mentioned in
> >
> https://urldefense.com/v3/__https://github.com/apache/polaris/issues/3890__;!!Iz9xO38YGHZK!5EuyFFkk3vhRWVIRvQAWBSQfpJkTMA9HxugzDwXmN0LPPqhEFxYkFRGVhtb8AqUwXtDh2OplcMnbMDHKOxrvDU0$
>,
> > supporting
> > multiple data sources is not a trivial change. I would strongly recommend
> > starting with a design document to carefully evaluate the architectural
> > implications and long term impact.
> >
> > A REST endpoint to query metrics seems reasonable given the current JDBC
> > based persistence model. That said, we may also consider alternative
> > storage models. For example, if we later adopt a time series system such
> as
> > Prometheus to store metrics, the query model and access patterns would be
> > fundamentally different. Designing the REST API without considering these
> > potential evolutions may limit flexibility. I'd suggest to start with the
> > use case.
> >
> > Yufei
> >
> >
> > On Fri, Feb 27, 2026 at 3:42 PM Dmitri Bourlatchkov <di...@apache.org>
> > wrote:
> >
> > > Hi Anand,
> > >
> > > Sharing my view... subject to discussion:
> > >
> > > 1. Adding non-IRC REST API to Polaris is perfectly fine.
> > >
> > > Figuring out specific endpoint URIs and payloads might require a few
> > > roundtrips, so opening a separate thread for that might be best.
> > > Contributors commonly create Google Docs for new API proposals too
> (they
> > > fairly easy to update as the email discussion progresses).
> > >
> > > There was a suggestion to try Markdown (with PRs) for proposals [1] ...
> > > feel free to give it a try if you are comfortable with that.
> > >
> > > 2. Could you clarify whether you mean end user utilities or admin user
> > > utilities? In the latter case those might be more suitable for the
> Admin
> > > CLI (java) not the Python CLI, IMHO.
> > >
> > > Why would these utilities be common with events? IMHO, event use cases
> > are
> > > distinct from scan/commit metrics.
> > >
> > > 3. I'd prefer separating metrics persistence from MetaStore persistence
> > at
> > > the code level, so that they could be mixed and matched independently.
> > The
> > > separate datasource question will become a non-issue with that
> approach,
> > I
> > > guess.
> > >
> > > The rationale for separating scan metrics and metastore persistence is
> > that
> > > "cascading deletes" between them are hardly ever required. Furthermore,
> > the
> > > data and query patterns are very different so different technologies
> > might
> > > be beneficial in each case.
> > >
> > > [1]
> >
> https://urldefense.com/v3/__https://lists.apache.org/thread/yto2wp982t43h1mqjwnslswhws5z47cy__;!!Iz9xO38YGHZK!5EuyFFkk3vhRWVIRvQAWBSQfpJkTMA9HxugzDwXmN0LPPqhEFxYkFRGVhtb8AqUwXtDh2OplcMnbMDHKxYDakNU$
>> >
> > > Cheers,
> > > Dmitri.
> > >
> > > On Fri, Feb 27, 2026 at 6:19 PM Anand Kumar Sankaran via dev <
> > > dev@polaris.apache.org> wrote:
> > >
> > > > Thanks all. This PR is merged now.
> > > >
> > > > Here are the follow-up features / work needed.  These were all part
> of
> > > the
> > > > merged PR at some point in time and were removed to reduce scope.
> > > >
> > > > Please let me know what you think.
> > > >
> > > >
> > > >   1.  A REST API to paginate through table metrics. This will be
> > non-IRC
> > > > standard addition.
> > > >   2.  Utilities for managing old records, should be common with
> events.
> > > > There was some discussion that it belongs to the CLI.
> > > >   3.  Separate datasource (metrics, events, even other tables?).
> > > >
> > > >
> > > > Anything else?
> > > >
> > > > -
> > > > Anand
> > > >
> > > >
> > >
> >
> >
>

Re: Proposal for REST endpoints for table metrics and events

[Yufei Gu <fl...@gmail.com>]

Thanks for the proposal, Anand. I will take a look. I generally prefer
Google Docs for design doc, esp. for the early design, which usually needs
fast iterations. But I'm OK with a markdown. BTW, we haven't concluded the
dev thread on the proposal, I think we will need a conclusion that, so that
people wont' be confused.

Yufei


On Tue, Mar 3, 2026 at 4:03 PM Dmitri Bourlatchkov <di...@apache.org> wrote:

> Hi Anand,
>
> Thanks for starting this proposal. I see that the linked GH PR gathered
> some review comments. Let's continue evolving it!
>
> From my personal POV the review on the .md doc PR is pretty easy and
> generally resembles regular code reviews in GH, which is a good thing as
> most people here are familiar with that.
>
> For those interested, the ASF mailing lists archive this review at [1].
> That trail is a bit verbose because it shows very long diffs, but it's
> still readable.
>
> In terms of process, I suppose once the PR is approved, we will merge it
> and then proceed with implementation PRs. WDYT?
>
> [1] https://lists.apache.org/thread/1s28pv4qsmjc6m3lypzmfkgrdom82fjd
>
> Cheers,
> Dmitri.
>
> On Tue, Mar 3, 2026 at 3:29 PM Anand Kumar Sankaran via dev <
> dev@polaris.apache.org> wrote:
>
> > Hi Yufei and Dmitri,
> >
> > Here is a proposal for the REST endpoints for metrics and events.
> >
> > https://github.com/apache/polaris/pull/3924/changes
> >
> > I did not see any precursors for raising a PR for proposals, so trying
> > this.  Please let me know what you think.
> >
> > -
> > Anand
> >
> > From: Anand Kumar Sankaran <an...@workday.com>
> > Date: Monday, March 2, 2026 at 10:25 AM
> > To: dev@polaris.apache.org <de...@polaris.apache.org>
> > Subject: Re: Polaris Telemetry and Audit Trail
> >
> > About the REST API, based on my use cases:
> >
> >
> >   1.
> > I want to be able to query commit metrics to track files added / removed
> > per commit, along with record counts. The ingestion pipeline that writes
> > this data is owned by us and we are guaranteed to write this information
> > for each write.
> >   2.
> > I want to be able to query scan metrics for read. I understand clients do
> > not fulfill this requirement.
> >   3.
> > I want to be able to query the events table (events are persisted) - this
> > may supersede #2, I am not sure yet.
> >
> > All this information is in the JDBC based persistence model and is
> > persisted in the metastore. I currently don’t have a need to query
> > prometheus or open telemetry. I do publish some events to Prometheus and
> > they are forwarded to our dashboards elsewhere.
> >
> > About the CLI utilities, I meant the admin user utilities. In one of the
> > earliest drafts of my proposal, Prashant mentioned that the metrics
> tables
> > can grow indefinitely and that a similar problem exists with the events
> > table as well. We discussed that cleaning up of old records from both
> > metrics tables and events tables can be done via a CLI utility.
> >
> > I see that Yufei has covered the discussion about datasources.
> >
> > -
> > Anand
> >
> >
> >
> > From: Yufei Gu <fl...@gmail.com>
> > Date: Friday, February 27, 2026 at 9:54 PM
> > To: dev@polaris.apache.org <de...@polaris.apache.org>
> > Subject: Re: Polaris Telemetry and Audit Trail
> >
> > This Message Is From an External Sender
> > This message came from outside your organization.
> > Report Suspicious<
> >
> https://us-phishalarm-ewt.proofpoint.com/EWT/v1/Iz9xO38YGHZK!YhNDZABkHi1B699ote2uMwpOZw8i0QMCGO2Szc-HshuABGhGvwPJcymE6G2oUUxtS8xDkSrtGTPm_I3QnVDHoLMk50m9v8z_nZKTkd-bnVUbreF1u0WnfV_X5eYevZl_$
> > >
> >
> >
> > As I mentioned in
> >
> https://urldefense.com/v3/__https://github.com/apache/polaris/issues/3890__;!!Iz9xO38YGHZK!5EuyFFkk3vhRWVIRvQAWBSQfpJkTMA9HxugzDwXmN0LPPqhEFxYkFRGVhtb8AqUwXtDh2OplcMnbMDHKOxrvDU0$
> ,
> > supporting
> > multiple data sources is not a trivial change. I would strongly recommend
> > starting with a design document to carefully evaluate the architectural
> > implications and long term impact.
> >
> > A REST endpoint to query metrics seems reasonable given the current JDBC
> > based persistence model. That said, we may also consider alternative
> > storage models. For example, if we later adopt a time series system such
> as
> > Prometheus to store metrics, the query model and access patterns would be
> > fundamentally different. Designing the REST API without considering these
> > potential evolutions may limit flexibility. I'd suggest to start with the
> > use case.
> >
> > Yufei
> >
> >
> > On Fri, Feb 27, 2026 at 3:42 PM Dmitri Bourlatchkov <di...@apache.org>
> > wrote:
> >
> > > Hi Anand,
> > >
> > > Sharing my view... subject to discussion:
> > >
> > > 1. Adding non-IRC REST API to Polaris is perfectly fine.
> > >
> > > Figuring out specific endpoint URIs and payloads might require a few
> > > roundtrips, so opening a separate thread for that might be best.
> > > Contributors commonly create Google Docs for new API proposals too
> (they
> > > fairly easy to update as the email discussion progresses).
> > >
> > > There was a suggestion to try Markdown (with PRs) for proposals [1] ...
> > > feel free to give it a try if you are comfortable with that.
> > >
> > > 2. Could you clarify whether you mean end user utilities or admin user
> > > utilities? In the latter case those might be more suitable for the
> Admin
> > > CLI (java) not the Python CLI, IMHO.
> > >
> > > Why would these utilities be common with events? IMHO, event use cases
> > are
> > > distinct from scan/commit metrics.
> > >
> > > 3. I'd prefer separating metrics persistence from MetaStore persistence
> > at
> > > the code level, so that they could be mixed and matched independently.
> > The
> > > separate datasource question will become a non-issue with that
> approach,
> > I
> > > guess.
> > >
> > > The rationale for separating scan metrics and metastore persistence is
> > that
> > > "cascading deletes" between them are hardly ever required. Furthermore,
> > the
> > > data and query patterns are very different so different technologies
> > might
> > > be beneficial in each case.
> > >
> > > [1]
> >
> https://urldefense.com/v3/__https://lists.apache.org/thread/yto2wp982t43h1mqjwnslswhws5z47cy__;!!Iz9xO38YGHZK!5EuyFFkk3vhRWVIRvQAWBSQfpJkTMA9HxugzDwXmN0LPPqhEFxYkFRGVhtb8AqUwXtDh2OplcMnbMDHKxYDakNU$
> > >
> > > Cheers,
> > > Dmitri.
> > >
> > > On Fri, Feb 27, 2026 at 6:19 PM Anand Kumar Sankaran via dev <
> > > dev@polaris.apache.org> wrote:
> > >
> > > > Thanks all. This PR is merged now.
> > > >
> > > > Here are the follow-up features / work needed.  These were all part
> of
> > > the
> > > > merged PR at some point in time and were removed to reduce scope.
> > > >
> > > > Please let me know what you think.
> > > >
> > > >
> > > >   1.  A REST API to paginate through table metrics. This will be
> > non-IRC
> > > > standard addition.
> > > >   2.  Utilities for managing old records, should be common with
> events.
> > > > There was some discussion that it belongs to the CLI.
> > > >   3.  Separate datasource (metrics, events, even other tables?).
> > > >
> > > >
> > > > Anything else?
> > > >
> > > > -
> > > > Anand
> > > >
> > > >
> > >
> >
> >
>

Re: Proposal for REST endpoints for table metrics and events

[Dmitri Bourlatchkov <di...@apache.org>]

Hi Anand,

Thanks for starting this proposal. I see that the linked GH PR gathered
some review comments. Let's continue evolving it!

From my personal POV the review on the .md doc PR is pretty easy and
generally resembles regular code reviews in GH, which is a good thing as
most people here are familiar with that.

For those interested, the ASF mailing lists archive this review at [1].
That trail is a bit verbose because it shows very long diffs, but it's
still readable.

In terms of process, I suppose once the PR is approved, we will merge it
and then proceed with implementation PRs. WDYT?

[1] https://lists.apache.org/thread/1s28pv4qsmjc6m3lypzmfkgrdom82fjd

Cheers,
Dmitri.

On Tue, Mar 3, 2026 at 3:29 PM Anand Kumar Sankaran via dev <
dev@polaris.apache.org> wrote:

> Hi Yufei and Dmitri,
>
> Here is a proposal for the REST endpoints for metrics and events.
>
> https://github.com/apache/polaris/pull/3924/changes
>
> I did not see any precursors for raising a PR for proposals, so trying
> this.  Please let me know what you think.
>
> -
> Anand
>
> From: Anand Kumar Sankaran <an...@workday.com>
> Date: Monday, March 2, 2026 at 10:25 AM
> To: dev@polaris.apache.org <de...@polaris.apache.org>
> Subject: Re: Polaris Telemetry and Audit Trail
>
> About the REST API, based on my use cases:
>
>
>   1.
> I want to be able to query commit metrics to track files added / removed
> per commit, along with record counts. The ingestion pipeline that writes
> this data is owned by us and we are guaranteed to write this information
> for each write.
>   2.
> I want to be able to query scan metrics for read. I understand clients do
> not fulfill this requirement.
>   3.
> I want to be able to query the events table (events are persisted) - this
> may supersede #2, I am not sure yet.
>
> All this information is in the JDBC based persistence model and is
> persisted in the metastore. I currently don’t have a need to query
> prometheus or open telemetry. I do publish some events to Prometheus and
> they are forwarded to our dashboards elsewhere.
>
> About the CLI utilities, I meant the admin user utilities. In one of the
> earliest drafts of my proposal, Prashant mentioned that the metrics tables
> can grow indefinitely and that a similar problem exists with the events
> table as well. We discussed that cleaning up of old records from both
> metrics tables and events tables can be done via a CLI utility.
>
> I see that Yufei has covered the discussion about datasources.
>
> -
> Anand
>
>
>
> From: Yufei Gu <fl...@gmail.com>
> Date: Friday, February 27, 2026 at 9:54 PM
> To: dev@polaris.apache.org <de...@polaris.apache.org>
> Subject: Re: Polaris Telemetry and Audit Trail
>
> This Message Is From an External Sender
> This message came from outside your organization.
> Report Suspicious<
> https://us-phishalarm-ewt.proofpoint.com/EWT/v1/Iz9xO38YGHZK!YhNDZABkHi1B699ote2uMwpOZw8i0QMCGO2Szc-HshuABGhGvwPJcymE6G2oUUxtS8xDkSrtGTPm_I3QnVDHoLMk50m9v8z_nZKTkd-bnVUbreF1u0WnfV_X5eYevZl_$
> >
>
>
> As I mentioned in
> https://urldefense.com/v3/__https://github.com/apache/polaris/issues/3890__;!!Iz9xO38YGHZK!5EuyFFkk3vhRWVIRvQAWBSQfpJkTMA9HxugzDwXmN0LPPqhEFxYkFRGVhtb8AqUwXtDh2OplcMnbMDHKOxrvDU0$,
> supporting
> multiple data sources is not a trivial change. I would strongly recommend
> starting with a design document to carefully evaluate the architectural
> implications and long term impact.
>
> A REST endpoint to query metrics seems reasonable given the current JDBC
> based persistence model. That said, we may also consider alternative
> storage models. For example, if we later adopt a time series system such as
> Prometheus to store metrics, the query model and access patterns would be
> fundamentally different. Designing the REST API without considering these
> potential evolutions may limit flexibility. I'd suggest to start with the
> use case.
>
> Yufei
>
>
> On Fri, Feb 27, 2026 at 3:42 PM Dmitri Bourlatchkov <di...@apache.org>
> wrote:
>
> > Hi Anand,
> >
> > Sharing my view... subject to discussion:
> >
> > 1. Adding non-IRC REST API to Polaris is perfectly fine.
> >
> > Figuring out specific endpoint URIs and payloads might require a few
> > roundtrips, so opening a separate thread for that might be best.
> > Contributors commonly create Google Docs for new API proposals too (they
> > fairly easy to update as the email discussion progresses).
> >
> > There was a suggestion to try Markdown (with PRs) for proposals [1] ...
> > feel free to give it a try if you are comfortable with that.
> >
> > 2. Could you clarify whether you mean end user utilities or admin user
> > utilities? In the latter case those might be more suitable for the Admin
> > CLI (java) not the Python CLI, IMHO.
> >
> > Why would these utilities be common with events? IMHO, event use cases
> are
> > distinct from scan/commit metrics.
> >
> > 3. I'd prefer separating metrics persistence from MetaStore persistence
> at
> > the code level, so that they could be mixed and matched independently.
> The
> > separate datasource question will become a non-issue with that approach,
> I
> > guess.
> >
> > The rationale for separating scan metrics and metastore persistence is
> that
> > "cascading deletes" between them are hardly ever required. Furthermore,
> the
> > data and query patterns are very different so different technologies
> might
> > be beneficial in each case.
> >
> > [1]
> https://urldefense.com/v3/__https://lists.apache.org/thread/yto2wp982t43h1mqjwnslswhws5z47cy__;!!Iz9xO38YGHZK!5EuyFFkk3vhRWVIRvQAWBSQfpJkTMA9HxugzDwXmN0LPPqhEFxYkFRGVhtb8AqUwXtDh2OplcMnbMDHKxYDakNU$
> >
> > Cheers,
> > Dmitri.
> >
> > On Fri, Feb 27, 2026 at 6:19 PM Anand Kumar Sankaran via dev <
> > dev@polaris.apache.org> wrote:
> >
> > > Thanks all. This PR is merged now.
> > >
> > > Here are the follow-up features / work needed.  These were all part of
> > the
> > > merged PR at some point in time and were removed to reduce scope.
> > >
> > > Please let me know what you think.
> > >
> > >
> > >   1.  A REST API to paginate through table metrics. This will be
> non-IRC
> > > standard addition.
> > >   2.  Utilities for managing old records, should be common with events.
> > > There was some discussion that it belongs to the CLI.
> > >   3.  Separate datasource (metrics, events, even other tables?).
> > >
> > >
> > > Anything else?
> > >
> > > -
> > > Anand
> > >
> > >
> >
>
>

Re: Proposal for REST endpoints for table metrics and events

[Dmitri Bourlatchkov <di...@apache.org>]

I guess I replied to an old message in the middle of the thread :) My
message still holds :)

Cheers,
Dmitri.

On Wed, Sep 23, 2026 at 7:53 PM Dmitri Bourlatchkov <di...@apache.org>
wrote:

> Hi EJ,
>
> Thanks for the meeting recap!
>
> All:
>
> Worth noting: PR [4115] also contains Management API changes that were not
> anticipated initially and did not get an API change review until now. These
> are small changes to add new grant types for metrics RBAC..
>
> I've reviewed and approved PR [4115] in GH.
>
> [4115] https://github.com/apache/polaris/pull/4115
>
> Cheers,
> Dmitri.
>
> On Wed, Aug 12, 2026 at 5:21 PM EJ Wang <ej...@gmail.com>
> wrote:
>
>> Hi folks,
>>
>> Notes from today's metrics architecture sync are in the usual doc:
>>
>> https://docs.google.com/document/d/100h7c4damrUzVuquYbBHM0EvA4LSWuW2IT2dN_7nYVA/edit?pli=1&tab=t.18iv1m7p6i6b
>>
>> Highlights:
>>
>> Module placement. core/ now holds the entity data model only. A new spi/
>> takes every other shared contract: feature SPIs, feature non-SPI
>> contracts,
>> substrate SPIs, substrate non-SPI contracts, and contract-required types.
>> That settles the question PR #5204 reopened, by placing into a new SPI
>> module. Note that it changes slightly towards the referenced WIP RFC [1]
>> recommendations: what proposed to place in core/, now should go to spi/
>> except for the entity data model.
>>
>> Metrics PRs. #5068 merged. #5204 needs reshaping onto the split. #4115 and
>> #4756 stay open with action on the author to follow up on review comments.
>>
>> [1]
>>
>> https://docs.google.com/document/d/1mj3vGpix9zTDTFFJKtWT5ZxI3PTGI1YNb5xFqyWrOkc/edit?tab=t.0
>>
>> On Wed, Jul 15, 2026 at 11:01 AM Anand Kumar Sankaran via dev <
>> dev@polaris.apache.org> wrote:
>>
>> > Thanks EJ.  My plan is to introduce PR0 in a new branch / PR and
>> repurpose
>> > #4115 and #4756 for PR1 and PR2 to maintain all the valuable discussions
>> > that happened there.
>> >
>> > From: EJ Wang <ej...@gmail.com>
>> > Date: Wednesday, July 15, 2026 at 10:41 AM
>> > To: dev@polaris.apache.org <de...@polaris.apache.org>
>> > Subject: Re: Proposal for REST endpoints for table metrics and events
>> >
>> > Hi folks, Thanks again for the discussion today. I updated the sync doc
>> > with notes from today's metrics sync: https: //urldefense.
>> com/v3/__https:
>> > //docs. google.
>> >
>> com/document/d/100h7c4damrUzVuquYbBHM0EvA4LSWuW2IT2dN_7nYVA/edit?pli=1&tab=t.
>> >
>> 2ahd7wze5f__;!!Iz9xO38YGHZK!4rnlVIqBQQQ6xpGe1PMV58edOS_cGrKoqdfhw4qSp_JQCavvEjaxXk0OwGPnxyEcO60AzKQK_XDtp7H3DdkFXcrXJwAz1g0$
>> >
>> >
>> > Hi folks,
>> >
>> > Thanks again for the discussion today. I updated the sync doc with notes
>> > from today's metrics sync:
>> >
>> >
>> https://urldefense.com/v3/__https://docs.google.com/document/d/100h7c4damrUzVuquYbBHM0EvA4LSWuW2IT2dN_7nYVA/edit?pli=1&tab=t.2ahd7wze5f__;!!Iz9xO38YGHZK!4rnlVIqBQQQ6xpGe1PMV58edOS_cGrKoqdfhw4qSp_JQCavvEjaxXk0OwGPnxyEcO60AzKQK_XDtp7H3DdkFXcrXJwAz1g0$
>> >
>> > See highlights below:
>> >
>> > In previous meetings, EJ proposed a two prs framing to push forward the
>> > metrics query work in polaris. Dmitri pointed out a subtle point that
>> pr2
>> > is currently also trying to do the persistence related code placement
>> > refactor. For the sake of straightforward review process, EJ, Dmitri and
>> > Anand (the author) agreed to separate the persistence refactoring into
>> pr0,
>> > and reframing the pr scopes as follow:
>> >
>> >    1.
>> >
>> >    Pr0: refactor irc metrics spi
>> >    1.
>> >
>> >       SPI should stay in core/ with metric type enveloped
>> >       2.
>> >
>> >       default spi impl, no-op, in extensions/ - default selection at
>> >       runtime/
>> >       3.
>> >
>> >       move metrics persistence and correspondings to extensions/ as an
>> >       alternative SPI impl
>> >       2.
>> >
>> >    Pr1: rest api for query
>> >
>> >
>> https://urldefense.com/v3/__https://github.com/apache/polaris/pull/4115/changes__;!!Iz9xO38YGHZK!4rnlVIqBQQQ6xpGe1PMV58edOS_cGrKoqdfhw4qSp_JQCavvEjaxXk0OwGPnxyEcO60AzKQK_XDtp7H3DdkFXcrXFCbEbRk$
>> >    1.
>> >
>> >       rest api spec in extensions/ as optional spec
>> >       2.
>> >
>> >       SPI in extensions/ as optional spec spi
>> >       3.
>> >
>> >       default spi impl in extensions/ no-op
>> >       4.
>> >
>> >       handler related in runtime/ - thin adaptor from http shape to spi
>> >       shape
>> >       3.
>> >
>> >    Pr2: jdbc impl for query
>> >
>> https://urldefense.com/v3/__https://github.com/apache/polaris/pull/4756__;!!Iz9xO38YGHZK!4rnlVIqBQQQ6xpGe1PMV58edOS_cGrKoqdfhw4qSp_JQCavvEjaxXk0OwGPnxyEcO60AzKQK_XDtp7H3DdkFXcrXRVMBzqE$
>> >    1.
>> >
>> >       alternative spi impl in extensions/ -> query with jdbc
>> >
>> >
>> > Thanks,
>> > -ej
>> >
>> > On Wed, Jul 1, 2026 at 8:01 AM Alexandre Dutra <ad...@apache.org>
>> wrote:
>> >
>> > > Hi all,
>> > >
>> > > > While the current implementation stores metrics in the metastore,
>> other
>> > > deployments
>> > > may choose a KV store, a dedicated metrics warehouse, or another
>> backend
>> > > entirely. In those cases, exposing a Polaris REST API over a specific
>> > > schema feels like coupling the API to an implementation detail.
>> > >
>> > > I wouldn't say to an implementation detail, but rather to a particular
>> > > setup / configuration. I think that's fine. If you don't store
>> > > metrics, you don't read metrics. Similar situations will certainly
>> > > happen in the future, e.g. if we ever introduce an API to read events,
>> > > that would only work if events are being persisted in the first place.
>> > > Again I think that's a fair trade-off.
>> > >
>> > > Thanks,
>> > > Alex
>> > >
>> > > On Wed, Jul 1, 2026 at 7:23 AM Dmitri Bourlatchkov <di...@apache.org>
>> > > wrote:
>> > > >
>> > > > Hi Yufei,
>> > > >
>> > > > From my POV, it is fine for Polaris to have multiple REST API
>> surfaces
>> > > with
>> > > > different features and different degrees of "readiness". Each API
>> can
>> > > > evolve fairly independently. With proper modularization (which Anand
>> > > did),
>> > > > changes to the main Polaris code and these extra modules should
>> > normally
>> > > be
>> > > > non-intrusive to each other.
>> > > >
>> > > > I assume Anand is going to be the first user of this API. However,
>> if
>> > we
>> > > do
>> > > > not merge it we'll never know whether other users are interested. I
>> do
>> > > not
>> > > > think many end users read the dev ML, but they certainly read the
>> docs,
>> > > > code and release notes.
>> > > >
>> > > > There is no risk for Polaris as a project in exposing this
>> experimental
>> > > > API. If usage dwindles over time, we can remove it. If it gains a
>> > > > considerable user / contributor base, we will promote it to a
>> > > "first-class"
>> > > > (non-beta) API.
>> > > >
>> > > > Cheers,
>> > > > Dmitri.
>> > > >
>> > > > On Tue, Jun 30, 2026 at 7:53 PM Yufei Gu <fl...@gmail.com>
>> wrote:
>> > > >
>> > > > > Hi all,
>> > > > >
>> > > > > Thanks for driving this work forward. Building this as a POC makes
>> > > perfect
>> > > > > sense if the goal is to validate the idea and gather feedback.
>> Should
>> > > we
>> > > > > merge a POC into the main branch before agreeing on the overall
>> > > direction?
>> > > > > To me, that feels like a separate discussion from whether the
>> > > > > implementation itself looks reasonable.
>> > > > >
>> > > > > One concern that I don't think has been fully addressed is
>> whether we
>> > > need
>> > > > > to expose metrics consumption through a Polaris REST endpoint at
>> all.
>> > > > >
>> > > > > My concern is less about the API itself and more about the
>> boundary
>> > > Polaris
>> > > > > should own. The proposed REST API is tied to a particular
>> persistence
>> > > > > schema, but that schema is just one possible implementation. While
>> > the
>> > > > > current implementation stores metrics in the metastore, other
>> > > deployments
>> > > > > may choose a KV store, a dedicated metrics warehouse, or another
>> > > backend
>> > > > > entirely. In those cases, exposing a Polaris REST API over a
>> specific
>> > > > > schema feels like coupling the API to an implementation detail.
>> > > > >
>> > > > > Polaris isn't intended to become a full metrics system. I'd
>> expect it
>> > > to
>> > > > > focus on producing metrics and let deployments choose the most
>> > > appropriate
>> > > > > storage and consumption model for their environment.
>> > > > >
>> > > > > I'm not opposed to exploring this direction, but we should first
>> > align
>> > > on
>> > > > > whether exposing a REST API through Polaris is the right long term
>> > > approach
>> > > > > before merging it into the main branch.
>> > > > >
>> > > > > Thanks,
>> > > > > Yufei
>> > > > >
>> > > > > Yufei
>> > > > >
>> > > > >
>> > > > > On Mon, Jun 29, 2026 at 9:27 AM Dmitri Bourlatchkov <
>> > dimas@apache.org>
>> > > > > wrote:
>> > > > >
>> > > > > > Hi All,
>> > > > > >
>> > > > > > It looks like Anand addressed all action items in [4115].
>> > > > > >
>> > > > > > Please review the latest state of the code. The PR LGTM and I
>> > > approved in
>> > > > > > GH from my side.
>> > > > > >
>> > > > > > [4115]
>> >
>> https://urldefense.com/v3/__https://github.com/apache/polaris/pull/4115__;!!Iz9xO38YGHZK!4rnlVIqBQQQ6xpGe1PMV58edOS_cGrKoqdfhw4qSp_JQCavvEjaxXk0OwGPnxyEcO60AzKQK_XDtp7H3DdkFXcrXtFRscKs$
>> > > > > >
>> > > > > > Thanks,
>> > > > > > Dmitri.
>> > > > > >
>> > > > > > On Thu, Jun 25, 2026 at 1:56 PM Dmitri Bourlatchkov <
>> > > dimas@apache.org>
>> > > > > > wrote:
>> > > > > >
>> > > > > > > Hi All,
>> > > > > > >
>> > > > > > > Recap of the discussion in today's Community Sync:
>> > > > > > >
>> > > > > > > * The modular approach for Metrics SPI(s) and implementation
>> > seems
>> > > to
>> > > > > > have
>> > > > > > > community consensus (please comment if you still have
>> concerns)
>> > > > > > >
>> > > > > > > * What modules / feature flags are enabled in Apache Polaris
>> > > images by
>> > > > > > > default will be discussed later (when the feature is close to
>> > > > > > completion).
>> > > > > > >
>> > > > > > > In any case, users will be able to turn the full feature
>> on/off
>> > at
>> > > > > their
>> > > > > > > discretion, regardless of the default flag values.
>> > > > > > >
>> > > > > > > * It is generally preferable to have smaller PRs and introduce
>> > new
>> > > > > > > features gradually to simplify reviews.
>> > > > > > >
>> > > > > > > What this means for the Metrics API, IMHO is:
>> > > > > > >
>> > > > > > > a) The scope of PR 4115 is probably ok as it stands right
>> now. If
>> > > > > someone
>> > > > > > > wants to propose splitting it further, please comment.
>> > > > > > >
>> > > > > > > b) After 4115 is merged, Metrics Persistence PRs will follow,
>> > with
>> > > > > > smaller
>> > > > > > > scope per PR as appropriate for the nature of the code
>> changes.
>> > > > > > >
>> > > > > > > * The major concern was about exposing the new Metrics REST
>> API
>> > to
>> > > end
>> > > > > > > users. The nature of this concern was about avoiding the
>> (false)
>> > > > > > impression
>> > > > > > > that the new API is a firm and adopted Polaris specification.
>> > > > > > >
>> > > > > > > I think this concern is reasonable.
>> > > > > > >
>> > > > > > > I propose that Anand update the documentation sections of PR
>> 4115
>> > > to
>> > > > > > > emphasise that the API is currently an experimental / POC API
>> at
>> > > this
>> > > > > > time
>> > > > > > > and is subject to change (including breaking changes). I
>> believe
>> > it
>> > > > > > already
>> > > > > > > has the "beta" label. So this request is mostly for doc
>> changes
>> > to
>> > > > > > clarify
>> > > > > > > user expectations, since the "beta" label alone may not be
>> > > interpreted
>> > > > > > the
>> > > > > > > same way by all users.
>> > > > > > >
>> > > > > > > I hope this update will resolve all remaining concerns with
>> 4115
>> > > and
>> > > > > > allow
>> > > > > > > for it to be merged so that the community will be able to
>> proceed
>> > > to
>> > > > > > > follow-up PRs (as noted above).
>> > > > > > >
>> > > > > > > Please add/correct me if I missed anything relevant.
>> > > > > > >
>> > > > > > > Thanks,
>> > > > > > > Dmitri.
>> > > > > > >
>> > > > > > > On Wed, Jun 24, 2026 at 8:11 PM Yufei Gu <
>> flyrain000@gmail.com>
>> > > wrote:
>> > > > > > >
>> > > > > > >> I do not think we have agreement on the query REST APIs yet.
>> > > > > > >>
>> > > > > > >> In particular, the current query API shape seems highly
>> coupled
>> > > to the
>> > > > > > >> *example[1]* metrics in the Iceberg REST spec. I do not think
>> > > that is
>> > > > > a
>> > > > > > >> good way to design a REST API contract. For example, fields
>> like
>> > > > > > >> resultDataFiles, resultDeleteFiles, totalFileSizeBytes,
>> > > > > > >> totalDataManifests,
>> > > > > > >> totalDeleteManifests, and scannedDataManifests are very
>> specific
>> > > to
>> > > > > one
>> > > > > > >> style of scan metric. That makes the API look more like a
>> direct
>> > > > > > >> projection
>> > > > > > >> of the current example payload than a general query model.
>> > > > > > >>
>> > > > > > >> My preference is that Polaris should not define or strongly
>> > opine
>> > > on
>> > > > > the
>> > > > > > >> Iceberg scan or commit metrics query REST API at this point.
>> > > Polaris
>> > > > > can
>> > > > > > >> resolve and authorize the table, accept the metrics report,
>> and
>> > > > > delegate
>> > > > > > >> it
>> > > > > > >> to the configured reporting path. Durable storage, filtering,
>> > > > > > dashboards,
>> > > > > > >> and query APIs should remain downstream impl. choices unless
>> and
>> > > until
>> > > > > > the
>> > > > > > >> Iceberg community standardizes a query contract, which I
>> doubt
>> > > will
>> > > > > > >> happen.
>> > > > > > >>
>> > > > > > >> So I do not think the intake path has to be blocked, but I
>> would
>> > > > > prefer
>> > > > > > >> not
>> > > > > > >> to merge a Polaris specific query API that may be hard to
>> evolve
>> > > > > later.
>> > > > > > >>
>> > > > > > >> 1.
>> > > > > > >>
>> > > > > > >>
>> > > > > >
>> > > > >
>> > >
>> >
>> https://urldefense.com/v3/__https://github.com/apache/iceberg/blob/2a6c556c0c883c04c6d3fbd68e6aeda36b91e0aa/open-api/rest-catalog-open-api.yaml*L4152__;Iw!!Iz9xO38YGHZK!4rnlVIqBQQQ6xpGe1PMV58edOS_cGrKoqdfhw4qSp_JQCavvEjaxXk0OwGPnxyEcO60AzKQK_XDtp7H3DdkFXcrXCDsqYGo$
>> > > > > > >>
>> > > > > > >> Yufei
>> > > > > > >>
>> > > > > > >>
>> > > > > > >> On Tue, Jun 23, 2026 at 9:30 AM Dmitri Bourlatchkov <
>> > > dimas@apache.org
>> > > > > >
>> > > > > > >> wrote:
>> > > > > > >>
>> > > > > > >> > Hi All,
>> > > > > > >> >
>> > > > > > >> > So, what are the blockers for merging [4115], if any?
>> > > > > > >> >
>> > > > > > >> > [4115]
>> >
>> https://urldefense.com/v3/__https://github.com/apache/polaris/pull/4115__;!!Iz9xO38YGHZK!4rnlVIqBQQQ6xpGe1PMV58edOS_cGrKoqdfhw4qSp_JQCavvEjaxXk0OwGPnxyEcO60AzKQK_XDtp7H3DdkFXcrXtFRscKs$
>> > > > > > >> >
>> > > > > > >> > Thanks,
>> > > > > > >> > Dmitri.
>> > > > > > >> >
>> > > > > > >> > On Wed, Jun 17, 2026 at 5:28 PM EJ Wang <
>> > > > > > ej.wang.dev.interest@gmail.com
>> > > > > > >> >
>> > > > > > >> > wrote:
>> > > > > > >> >
>> > > > > > >> > > Hi folks,
>> > > > > > >> > >
>> > > > > > >> > > Thanks again for the discussion today. I updated the sync
>> > doc
>> > > with
>> > > > > > >> notes
>> > > > > > >> > > from today's metrics sync:
>> > > > > > >> > >
>> > > > > > >> > >
>> > > > > > >> > >
>> > > > > > >> >
>> > > > > > >>
>> > > > > >
>> > > > >
>> > >
>> >
>> https://urldefense.com/v3/__https://docs.google.com/document/d/100h7c4damrUzVuquYbBHM0EvA4LSWuW2IT2dN_7nYVA/edit?tab=t.ezk0rgdx0c6m__;!!Iz9xO38YGHZK!4rnlVIqBQQQ6xpGe1PMV58edOS_cGrKoqdfhw4qSp_JQCavvEjaxXk0OwGPnxyEcO60AzKQK_XDtp7H3DdkFXcrX83IZkm0$
>> > > > > > >> > >
>> > > > > > >> > > A few highlights:
>> > > > > > >> > >
>> > > > > > >> > >    - No objection to keeping the default metrics behavior
>> > > small:
>> > > > > > >> > >    no-op/log-only is enough as the built-in default.
>> > > > > > >> > >    - Durable storage, event forwarding, external queues,
>> > > > > dashboards,
>> > > > > > >> and
>> > > > > > >> > >    custom filtering should be implementation choices
>> behind
>> > > the
>> > > > > > >> metrics
>> > > > > > >> > >    reporting path.
>> > > > > > >> > >    - For PR #4115, the REST layer should
>> > resolve/authz/accept
>> > > the
>> > > > > > >> report,
>> > > > > > >> > >    then delegate to the selected metrics implementation.
>> It
>> > > should
>> > > > > > not
>> > > > > > >> > own
>> > > > > > >> > >    durable storage semantics.
>> > > > > > >> > >    - The rough boundary we discussed is: spec owns the
>> wire
>> > > > > > contract,
>> > > > > > >> > >    runtime owns framework wiring, API/contract modules
>> own
>> > > > > > >> > provider-facing
>> > > > > > >> > >    contracts, and extensions own replaceable
>> > implementations.
>> > > > > > >> > >    - OpenLineage has a similar path-naming question. We
>> > should
>> > > > > avoid
>> > > > > > >> > >    occupying a generic lineage path too early if we want
>> > room
>> > > for
>> > > > > > >> other
>> > > > > > >> > >    lineage systems later.
>> > > > > > >> > >
>> > > > > > >> > > I believe the boundary rules previewed in sync meeting
>> are
>> > > useful
>> > > > > > for
>> > > > > > >> > > reviewing PR #4115, but it is not
>> > > > > > >> > > a final module-layout proposal.
>> > > > > > >> > >
>> > > > > > >> > > For metrics specifically, I think the target remains:
>> > > > > > >> > >    runtime
>> > > > > > >> > >      -> metrics reporting/emitting contract
>> > > > > > >> > >      -> default implementation
>> > > > > > >> > >           -> lower implementation details, if needed
>> > > > > > >> > >
>> > > > > > >> > > That keeps the default simple, while allowing durable
>> JDBC,
>> > > > > > >> event-backed,
>> > > > > > >> > > or external-queue implementations to be added separately.
>> > > > > > >> > >
>> > > > > > >> > > We also touched on the split between Iceberg metrics
>> > emitting
>> > > and
>> > > > > > >> > > Polaris-owned metrics querying. The querying side can
>> keep
>> > > > > evolving,
>> > > > > > >> > > especially for dashboard or generic-table use cases, but
>> I
>> > do
>> > > not
>> > > > > > >> think
>> > > > > > >> > > that needs to block the intake path in this PR.
>> > > > > > >> > >
>> > > > > > >> > > Thanks,
>> > > > > > >> > > -ej
>> > > > > > >> > >
>> > > > > > >> > > On Tue, Jun 16, 2026 at 2:17 PM Yufei Gu <
>> > > flyrain000@gmail.com>
>> > > > > > >> wrote:
>> > > > > > >> > >
>> > > > > > >> > > > Thanks for chiming in, EJ. Agreed that the default
>> battery
>> > > > > should
>> > > > > > >> stay
>> > > > > > >> > > > small. I'm leaning toward not including metrics
>> > persistence
>> > > in
>> > > > > the
>> > > > > > >> > > default
>> > > > > > >> > > > battery.
>> > > > > > >> > > >
>> > > > > > >> > > > Yufei
>> > > > > > >> > > >
>> > > > > > >> > > >
>> > > > > > >> > > > On Thu, Jun 11, 2026 at 8:30 PM EJ Wang <
>> > > > > > >> > ej.wang.dev.interest@gmail.com>
>> > > > > > >> > > > wrote:
>> > > > > > >> > > >
>> > > > > > >> > > > > Hi Yufei,
>> > > > > > >> > > > >
>> > > > > > >> > > > > Thanks for connecting this back to the REST endpoint
>> > > > > proposal. I
>> > > > > > >> > > replied
>> > > > > > >> > > > > with the fuller version on the event-forwarding
>> thread,
>> > > but
>> > > > > > >> wanted to
>> > > > > > >> > > add
>> > > > > > >> > > > > the shorter version here too.
>> > > > > > >> > > > >
>> > > > > > >> > > > > I agree that metrics reporting/emitting is the right
>> > > > > conceptual
>> > > > > > >> > > boundary,
>> > > > > > >> > > > > and I also think the event/listener path is a good
>> > > > > > implementation
>> > > > > > >> > > > direction
>> > > > > > >> > > > > to explore. The distinction I want to keep clear is
>> > > default
>> > > > > > >> battery
>> > > > > > >> > vs
>> > > > > > >> > > > > extension implementation.
>> > > > > > >> > > > >
>> > > > > > >> > > > > For the REST/API side, I would keep the semantics
>> > narrow:
>> > > > > > Polaris
>> > > > > > >> > > > resolves
>> > > > > > >> > > > > and authorizes the table, accepts the Iceberg
>> > scan/commit
>> > > > > > metrics
>> > > > > > >> > > report
>> > > > > > >> > > > > into the configured reporting/emitting path, and
>> returns
>> > > 204.
>> > > > > > That
>> > > > > > >> > 204
>> > > > > > >> > > > > should mean Polaris accepted the report into the
>> > ingestion
>> > > > > path;
>> > > > > > >> it
>> > > > > > >> > > > should
>> > > > > > >> > > > > not imply durable storage.
>> > > > > > >> > > > >
>> > > > > > >> > > > > Durable storage, event forwarding, external telemetry
>> > > routing,
>> > > > > > >> > > filtering,
>> > > > > > >> > > > > and retention should sit behind that
>> reporting/emitting
>> > > > > boundary
>> > > > > > >> as
>> > > > > > >> > > > > implementation-layer behavior. A durable JDBC path
>> can
>> > be
>> > > one
>> > > > > > >> named
>> > > > > > >> > > > > extension implementation. An event/listener
>> forwarding
>> > > path
>> > > > > can
>> > > > > > be
>> > > > > > >> > > > another
>> > > > > > >> > > > > named extension implementation.
>> > > > > > >> > > > >
>> > > > > > >> > > > > So I am +1 on the event-forwarding idea as a
>> non-default
>> > > > > > extension
>> > > > > > >> > > > > implementation. I just would not make it the default
>> > > battery
>> > > > > or
>> > > > > > >> the
>> > > > > > >> > > core
>> > > > > > >> > > > > API shape. The default battery can stay small and
>> safe,
>> > > while
>> > > > > > >> > > deployments
>> > > > > > >> > > > > that need forwarding, dashboards, or durable storage
>> can
>> > > opt
>> > > > > > into
>> > > > > > >> the
>> > > > > > >> > > > > implementation that matches their operational model.
>> > > > > > >> > > > >
>> > > > > > >> > > > > Thanks,
>> > > > > > >> > > > > -ej
>> > > > > > >> > > > >
>> > > > > > >> > > > > On Wed, Jun 10, 2026 at 11:43 AM Yufei Gu <
>> > > > > flyrain000@gmail.com
>> > > > > > >
>> > > > > > >> > > wrote:
>> > > > > > >> > > > >
>> > > > > > >> > > > > > Thanks EJ. I agree that the reporting/emitting
>> > boundary
>> > > > > feels
>> > > > > > >> like
>> > > > > > >> > > the
>> > > > > > >> > > > > > right SPI boundary, but I wonder if we can simplify
>> > this
>> > > > > even
>> > > > > > >> > > further.
>> > > > > > >> > > > > >
>> > > > > > >> > > > > > Iceberg scan and commit metrics look very similar
>> to
>> > > events
>> > > > > to
>> > > > > > >> me.
>> > > > > > >> > > They
>> > > > > > >> > > > > are
>> > > > > > >> > > > > > append only, asynchronously consumed, often
>> forwarded
>> > to
>> > > > > > >> external
>> > > > > > >> > > > > systems,
>> > > > > > >> > > > > > and have similar retention and cleanup
>> requirements.
>> > > More
>> > > > > > >> details
>> > > > > > >> > can
>> > > > > > >> > > > be
>> > > > > > >> > > > > > found in this thread [1]. In that case, Polaris may
>> > only
>> > > > > need
>> > > > > > to
>> > > > > > >> > > accept
>> > > > > > >> > > > > the
>> > > > > > >> > > > > > report and emit an event through the existing event
>> > > > > framework.
>> > > > > > >> > > > Filtering,
>> > > > > > >> > > > > > async delivery, custom sinks, and retention
>> mechanisms
>> > > could
>> > > > > > >> then
>> > > > > > >> > be
>> > > > > > >> > > > > shared
>> > > > > > >> > > > > > instead of introducing a separate metrics specific
>> > > extension
>> > > > > > >> layer
>> > > > > > >> > > and
>> > > > > > >> > > > > SPI.
>> > > > > > >> > > > > >
>> > > > > > >> > > > > > The main requirement I still see is querying
>> metrics
>> > > through
>> > > > > > >> > Polaris.
>> > > > > > >> > > > If
>> > > > > > >> > > > > we
>> > > > > > >> > > > > > want that battery included experience, we could
>> work
>> > on
>> > > a
>> > > > > > >> > persistence
>> > > > > > >> > > > > > listener implementation that selectively saves
>> metrics
>> > > and
>> > > > > > >> expose
>> > > > > > >> > it
>> > > > > > >> > > > via
>> > > > > > >> > > > > > the IRC event endpoint which seems pretty close in
>> > > Iceberg
>> > > > > > >> > > > > > community. Another option I like more is to allow
>> > users
>> > > to
>> > > > > > >> develop
>> > > > > > >> > > > their
>> > > > > > >> > > > > > own listener, so they can persist scan/commit
>> metrics
>> > > to any
>> > > > > > >> > systems
>> > > > > > >> > > > they
>> > > > > > >> > > > > > prefer. In general, that feels like an
>> implementation
>> > > choice
>> > > > > > on
>> > > > > > >> top
>> > > > > > >> > > of
>> > > > > > >> > > > > the
>> > > > > > >> > > > > > event framework rather than something that needs to
>> > > shape
>> > > > > the
>> > > > > > >> core
>> > > > > > >> > > > > > architecture.
>> > > > > > >> > > > > >
>> > > > > > >> > > > > > 1.
>> > > > > > >> >
>> > >
>> >
>> https://urldefense.com/v3/__https://lists.apache.org/thread/x9j8nscvy8hq61tyn01mj8yp6n9of0kp__;!!Iz9xO38YGHZK!4rnlVIqBQQQ6xpGe1PMV58edOS_cGrKoqdfhw4qSp_JQCavvEjaxXk0OwGPnxyEcO60AzKQK_XDtp7H3DdkFXcrXGcGDgks$
>> > > > > > >> > > > > >
>> > > > > > >> > > > > > Thanks,
>> > > > > > >> > > > > > Yufei
>> > > > > > >> > > > > >
>> > > > > > >> > > > > >
>> > > > > > >> > > > > > On Thu, Jun 4, 2026 at 5:14 PM EJ Wang <
>> > > > > > >> > > ej.wang.dev.interest@gmail.com
>> > > > > > >> > > > >
>> > > > > > >> > > > > > wrote:
>> > > > > > >> > > > > >
>> > > > > > >> > > > > > > Good point. I agree the existing
>> > > PolarisMetricsReporter is
>> > > > > > >> > already
>> > > > > > >> > > > very
>> > > > > > >> > > > > > > close to the right conceptual boundary. I am not
>> > > > > proposing a
>> > > > > > >> > second
>> > > > > > >> > > > > > > parallel reporter concept. The distinction I am
>> > > trying to
>> > > > > > >> make is
>> > > > > > >> > > > > > narrower:
>> > > > > > >> > > > > > >
>> > > > > > >> > > > > > > Current state:
>> > > > > > >> > > > > > >
>> > > > > > >> > > > > > >    - PolarisMetricsReporter is the existing
>> metrics
>> > > > > > reporting
>> > > > > > >> > hook.
>> > > > > > >> > > > > > >    - But it currently lives under
>> runtime/service.
>> > > > > > >> > > > > > >    - Its contract is documented around
>> Quarkus/CDI
>> > > > > discovery
>> > > > > > >> and
>> > > > > > >> > > > > > > @Identifier
>> > > > > > >> > > > > > >    selection.
>> > > > > > >> > > > > > >    - The default implementation is log-only.
>> > > > > > >> > > > > > >    - The durable implementation writes through
>> > > > > > >> > > PolarisMetricsManager
>> > > > > > >> > > > > and
>> > > > > > >> > > > > > >    MetricsPersistence.
>> > > > > > >> > > > > > >    - MetricsPersistence is currently inherited by
>> > > > > > >> > BasePersistence.
>> > > > > > >> > > > > > >
>> > > > > > >> > > > > > > My proposal:
>> > > > > > >> > > > > > >
>> > > > > > >> > > > > > >    - Keep the reporting/emitting boundary as the
>> > SPI.
>> > > > > > >> > > > > > >    - Revise that contract in a framework-agnostic
>> > > optional
>> > > > > > >> > Iceberg
>> > > > > > >> > > > > > *metrics
>> > > > > > >> > > > > > >    extension API layer*.
>> > > > > > >> > > > > > >    - Keep runtime/service responsible for REST
>> > > ingestion,
>> > > > > > >> > > table/authz
>> > > > > > >> > > > > > >    validation, and runtime wiring.
>> > > > > > >> > > > > > >    - Keep the battery implementation log-only in
>> the
>> > > > > metrics
>> > > > > > >> > > > extension
>> > > > > > >> > > > > > API
>> > > > > > >> > > > > > >    layer, *not under runtime/service*.
>> > > > > > >> > > > > > >    - Treat durable JDBC metrics as one
>> > implementation
>> > > of
>> > > > > the
>> > > > > > >> > > > reporting
>> > > > > > >> > > > > > SPI.
>> > > > > > >> > > > > > >    - Decompose the MetricsPersistence and
>> > > BasePersistence
>> > > > > > >> > coupling
>> > > > > > >> > > as
>> > > > > > >> > > > > > >    durable implementation detail.
>> > > > > > >> > > > > > >
>> > > > > > >> > > > > > > So the difference is not “new reporter concept vs
>> > old
>> > > > > > reporter
>> > > > > > >> > > > > concept.”
>> > > > > > >> > > > > > > but:
>> > > > > > >> > > > > > >
>> > > > > > >> > > > > > >
>> > > > > > >> > > > > > >    - PolarisMetricsReporter today =
>> runtime/service
>> > > > > > CDI-shaped
>> > > > > > >> > > hook.
>> > > > > > >> > > > > > >    - Target reporting SPI = same conceptual
>> > boundary,
>> > > but
>> > > > > > >> placed
>> > > > > > >> > in
>> > > > > > >> > > > the
>> > > > > > >> > > > > > >    right extension API layer and kept
>> > > framework-agnostic.
>> > > > > > >> > > > > > >
>> > > > > > >> > > > > > > For the current metrics PR, I think the minimal
>> > > > > > architectural
>> > > > > > >> > > cleanup
>> > > > > > >> > > > > > > before merge is to *put the reporting SPI
>> contract
>> > > and the
>> > > > > > >> > battery
>> > > > > > >> > > > > > default
>> > > > > > >> > > > > > > in the right place*. The durable JDBC
>> > implementation,
>> > > > > > >> > event-backed
>> > > > > > >> > > > > async
>> > > > > > >> > > > > > > implementation, external queue support, and
>> deeper
>> > > > > > persistence
>> > > > > > >> > > > cleanup
>> > > > > > >> > > > > > can
>> > > > > > >> > > > > > > continue as follow-ups, as long as we do not lock
>> > the
>> > > SPI
>> > > > > > >> > boundary
>> > > > > > >> > > to
>> > > > > > >> > > > > > > BasePersistence or to runtime/service wiring.
>> > > > > > >> > > > > > >
>> > > > > > >> > > > > > > -ej
>> > > > > > >> > > > > > >
>> > > > > > >> > > > > > > On Wed, Jun 3, 2026 at 3:00 PM Dmitri
>> Bourlatchkov <
>> > > > > > >> > > dimas@apache.org
>> > > > > > >> > > > >
>> > > > > > >> > > > > > > wrote:
>> > > > > > >> > > > > > >
>> > > > > > >> > > > > > > > Hi EJ,
>> > > > > > >> > > > > > > >
>> > > > > > >> > > > > > > > Thanks for the recap / summary!
>> > > > > > >> > > > > > > >
>> > > > > > >> > > > > > > > > Do folks agree that the stable SPI boundary
>> > > should be
>> > > > > > >> > > > > > > > metrics reporting/emitting, not metrics
>> > persistence?
>> > > > > > >> > > > > > > >
>> > > > > > >> > > > > > > > SGTM.
>> > > > > > >> > > > > > > >
>> > > > > > >> > > > > > > > > Does an optional Iceberg metrics extension
>> API
>> > > layer
>> > > > > > sound
>> > > > > > >> > like
>> > > > > > >> > > > the
>> > > > > > >> > > > > > > right
>> > > > > > >> > > > > > > > home for this SPI?
>> > > > > > >> > > > > > > >
>> > > > > > >> > > > > > > > How is that different from current
>> > > > > PolarisMetricsReporter?
>> > > > > > >> > > > > > > >
>> > > > > > >> > > > > > > > > Should the current durable metrics work be
>> > > reframed
>> > > > > as a
>> > > > > > >> > > durable
>> > > > > > >> > > > > > > > JDBC reference implementation of that SPI?
>> > > > > > >> > > > > > > >
>> > > > > > >> > > > > > > > SGTM.
>> > > > > > >> > > > > > > >
>> > > > > > >> > > > > > > > > What is the smallest PR sequence to get there
>> > > without
>> > > > > > >> > blocking
>> > > > > > >> > > > > > > > the current metrics work unnecessarily?
>> > > > > > >> > > > > > > >
>> > > > > > >> > > > > > > > Let's get
>> > >
>> >
>> https://urldefense.com/v3/__https://github.com/apache/polaris/pull/4397__;!!Iz9xO38YGHZK!4rnlVIqBQQQ6xpGe1PMV58edOS_cGrKoqdfhw4qSp_JQCavvEjaxXk0OwGPnxyEcO60AzKQK_XDtp7H3DdkFXcrXuu5jefU$
>> > > > > > >> merged
>> > > > > > >> > > > first.
>> > > > > > >> > > > > > > >
>> > > > > > >> > > > > > > > Then, I'd propose isolating the Metrics schema
>> > from
>> > > the
>> > > > > > >> > MetaStore
>> > > > > > >> > > > > > schema.
>> > > > > > >> > > > > > > > This will probably have some ripple effect into
>> > the
>> > > > > > >> bootstrap
>> > > > > > >> > > > > > workflows,
>> > > > > > >> > > > > > > so
>> > > > > > >> > > > > > > > it's not a trivial change.
>> > > > > > >> > > > > > > >
>> > > > > > >> > > > > > > > Then, let's reassess.
>> > > > > > >> > > > > > > >
>> > > > > > >> > > > > > > > Cheers,
>> > > > > > >> > > > > > > > Dmitri.
>> > > > > > >> > > > > > > >
>> > > > > > >> > > > > > > > On Wed, Jun 3, 2026 at 5:40 PM EJ Wang <
>> > > > > > >> > > > > ej.wang.dev.interest@gmail.com
>> > > > > > >> > > > > > >
>> > > > > > >> > > > > > > > wrote:
>> > > > > > >> > > > > > > >
>> > > > > > >> > > > > > > > > Hi folks,
>> > > > > > >> > > > > > > > >
>> > > > > > >> > > > > > > > > Thanks again for the discussion today. I
>> updated
>> > > the
>> > > > > > sync
>> > > > > > >> doc
>> > > > > > >> > > > with
>> > > > > > >> > > > > > > notes
>> > > > > > >> > > > > > > > > from the third metrics architecture sync:
>> > > > > > >> > > > > > > > >
>> > > > > > >> > > > > > > > >
>> > > > > > >> > > > > > > > >
>> > > > > > >> > > > > > > >
>> > > > > > >> > > > > > >
>> > > > > > >> > > > > >
>> > > > > > >> > > > >
>> > > > > > >> > > >
>> > > > > > >> > >
>> > > > > > >> >
>> > > > > > >>
>> > > > > >
>> > > > >
>> > >
>> >
>> https://urldefense.com/v3/__https://docs.google.com/document/d/100h7c4damrUzVuquYbBHM0EvA4LSWuW2IT2dN_7nYVA/edit?tab=t.k96s2xyqr5u1__;!!Iz9xO38YGHZK!4rnlVIqBQQQ6xpGe1PMV58edOS_cGrKoqdfhw4qSp_JQCavvEjaxXk0OwGPnxyEcO60AzKQK_XDtp7H3DdkFXcrXrqCL-QE$
>> > > > > > >> > > > > > > > >
>> > > > > > >> > > > > > > > > A few highlights from today’s discussion:
>> > > > > > >> > > > > > > > >
>> > > > > > >> > > > > > > > >    - We clarified that Iceberg metrics
>> reporting
>> > > can
>> > > > > be
>> > > > > > >> > > > interpreted
>> > > > > > >> > > > > > > > either
>> > > > > > >> > > > > > > > >    as sync or async handling from Polaris’s
>> > > > > perspective,
>> > > > > > >> so
>> > > > > > >> > > > Polaris
>> > > > > > >> > > > > > > > should
>> > > > > > >> > > > > > > > >    stay flexible as a platform instead of
>> baking
>> > > one
>> > > > > > >> handling
>> > > > > > >> > > > model
>> > > > > > >> > > > > > > into
>> > > > > > >> > > > > > > > > the
>> > > > > > >> > > > > > > > >    REST API behavior.
>> > > > > > >> > > > > > > > >    - We aligned that the built-in (battery)
>> > > behavior
>> > > > > for
>> > > > > > >> > > metrics
>> > > > > > >> > > > > > > emitting
>> > > > > > >> > > > > > > > >    can stay simple: no-op/log-only is enough
>> as
>> > > the
>> > > > > > >> default.
>> > > > > > >> > > > > > > > >    - Durable metrics persistence should be
>> > > treated as
>> > > > > an
>> > > > > > >> > > > > > implementation
>> > > > > > >> > > > > > > > of
>> > > > > > >> > > > > > > > >    the metrics reporting path, not as the
>> core
>> > SPI
>> > > > > > >> boundary.
>> > > > > > >> > > > > > > > >    - The existing durable metrics work can be
>> > > reviewed
>> > > > > > as
>> > > > > > >> a
>> > > > > > >> > > > > reference
>> > > > > > >> > > > > > > > >    implementation of the reporting SPI, with
>> > > > > > >> > > persistence-related
>> > > > > > >> > > > > > logic
>> > > > > > >> > > > > > > > kept
>> > > > > > >> > > > > > > > >    self-contained under a metrics durable
>> > > > > implementation
>> > > > > > >> > module
>> > > > > > >> > > > > > rather
>> > > > > > >> > > > > > > > than
>> > > > > > >> > > > > > > > >    scattered through core entity persistence.
>> > > > > > >> > > > > > > > >    - Dashboard/insights remains a real use
>> case,
>> > > but
>> > > > > we
>> > > > > > >> > agreed
>> > > > > > >> > > to
>> > > > > > >> > > > > > keep
>> > > > > > >> > > > > > > it
>> > > > > > >> > > > > > > > >    separate from the core metrics intake
>> > > discussion
>> > > > > for
>> > > > > > >> now.
>> > > > > > >> > > > > > > > >
>> > > > > > >> > > > > > > > > I also did a quick source check after the
>> > meeting
>> > > to
>> > > > > > make
>> > > > > > >> > sure
>> > > > > > >> > > we
>> > > > > > >> > > > > are
>> > > > > > >> > > > > > > > > describing the current state accurately.
>> > > > > > >> > > > > > > > >
>> > > > > > >> > > > > > > > > Current state:
>> > > > > > >> > > > > > > > >
>> > > > > > >> > > > > > > > >    - Polaris already has a metrics reporting
>> > hook:
>> > > > > > >> > > > > > > > PolarisMetricsReporter.
>> > > > > > >> > > > > > > > >    - The default implementation is
>> > > > > > DefaultMetricsReporter,
>> > > > > > >> > > > selected
>> > > > > > >> > > > > > by
>> > > > > > >> > > > > > > > >
>> > > polaris.iceberg-metrics.reporting.type=default. It
>> > > > > is
>> > > > > > >> > > > log-only,
>> > > > > > >> > > > > > > > >    effectively quiet unless metrics logging
>> is
>> > > > > enabled.
>> > > > > > >> > > > > > > > >    - There is also a
>> PersistingMetricsReporter,
>> > > > > selected
>> > > > > > >> by
>> > > > > > >> > > > > > > > >
>> > > polaris.iceberg-metrics.reporting.type=persisting,
>> > > > > > >> which
>> > > > > > >> > > > > converts
>> > > > > > >> > > > > > > > >    Iceberg scan/commit reports into Polaris
>> > > metrics
>> > > > > > >> records
>> > > > > > >> > and
>> > > > > > >> > > > > > writes
>> > > > > > >> > > > > > > > > through
>> > > > > > >> > > > > > > > >    PolarisMetricsManager ->
>> MetricsPersistence.
>> > > > > > >> > > > > > > > >    - MetricsPersistence currently exists in
>> the
>> > > > > > >> persistence
>> > > > > > >> > > layer
>> > > > > > >> > > > > and
>> > > > > > >> > > > > > > is
>> > > > > > >> > > > > > > > >    inherited by BasePersistence.
>> > > > > > >> > > > > > > > >
>> > > > > > >> > > > > > > > > My read from the discussion is that the
>> target
>> > > > > boundary
>> > > > > > >> > should
>> > > > > > >> > > > not
>> > > > > > >> > > > > be
>> > > > > > >> > > > > > > > > MetricsPersistence as inherited by
>> > > BasePersistence.
>> > > > > That
>> > > > > > >> path
>> > > > > > >> > > > > should
>> > > > > > >> > > > > > be
>> > > > > > >> > > > > > > > > decomposed as durable implementation detail
>> > (taken
>> > > > > care
>> > > > > > >> of by
>> > > > > > >> > > > > > > > >
>> >
>> https://urldefense.com/v3/__https://github.com/apache/polaris/pull/4397__;!!Iz9xO38YGHZK!4rnlVIqBQQQ6xpGe1PMV58edOS_cGrKoqdfhw4qSp_JQCavvEjaxXk0OwGPnxyEcO60AzKQK_XDtp7H3DdkFXcrXuu5jefU$
>> ).
>> > The
>> > > > > > stable
>> > > > > > >> > > > extension
>> > > > > > >> > > > > > > point
>> > > > > > >> > > > > > > > > should instead be the metrics
>> reporting/emitting
>> > > > > > boundary.
>> > > > > > >> > > > > > > > >
>> > > > > > >> > > > > > > > > Concretely, I think the next proposal should
>> be
>> > > shaped
>> > > > > > >> like
>> > > > > > >> > > this:
>> > > > > > >> > > > > > > > >
>> > > > > > >> > > > > > > > >    1.
>> > > > > > >> > > > > > > > >
>> > > > > > >> > > > > > > > >    Define the proper metrics reporting SPI
>> at an
>> > > > > > optional
>> > > > > > >> > > Iceberg
>> > > > > > >> > > > > > > metrics
>> > > > > > >> > > > > > > > >    extension API layer.
>> > > > > > >> > > > > > > > >
>> > > > > > >> > > > > > > > >    Example direction:
>> > > > > > >> > > > > > > > >
>> > > > > > >> > > > > > > > >    public interface IcebergMetricsReporter {
>> > > > > > >> > > > > > > > >      void
>> > reportMetric(IcebergMetricsReportContext
>> > > > > > >> context,
>> > > > > > >> > > > > > > > > MetricsReport report);
>> > > > > > >> > > > > > > > >    }
>> > > > > > >> > > > > > > > >
>> > > > > > >> > > > > > > > >    The context would carry the small
>> > > Polaris-resolved
>> > > > > > >> > envelope:
>> > > > > > >> > > > > > > > >    catalog/table identity, report type,
>> received
>> > > > > > >> timestamp,
>> > > > > > >> > and
>> > > > > > >> > > > > > > > > request/trace
>> > > > > > >> > > > > > > > >    context if available. The raw Iceberg
>> > > MetricsReport
>> > > > > > >> > remains
>> > > > > > >> > > > the
>> > > > > > >> > > > > > > > payload.
>> > > > > > >> > > > > > > > >    2.
>> > > > > > >> > > > > > > > >
>> > > > > > >> > > > > > > > >    Keep runtime/service as the REST ingestion
>> > and
>> > > > > wiring
>> > > > > > >> > layer.
>> > > > > > >> > > > > > > > >
>> > > > > > >> > > > > > > > >    The REST handler still resolves the table
>> and
>> > > > > > performs
>> > > > > > >> > authz
>> > > > > > >> > > > > > before
>> > > > > > >> > > > > > > > >    accepting the report. After that, it calls
>> > the
>> > > > > > selected
>> > > > > > >> > > > > reporting
>> > > > > > >> > > > > > > > >    implementation.
>> > > > > > >> > > > > > > > >    3.
>> > > > > > >> > > > > > > > >
>> > > > > > >> > > > > > > > >    Keep the battery default no-op/log-only.
>> > > > > > >> > > > > > > > >
>> > > > > > >> > > > > > > > >    This preserves an out-of-box safe default
>> and
>> > > > > avoids
>> > > > > > >> > > > requiring a
>> > > > > > >> > > > > > > > durable
>> > > > > > >> > > > > > > > >    metrics store for every Polaris
>> deployment.
>> > > > > > >> > > > > > > > >    4.
>> > > > > > >> > > > > > > > >
>> > > > > > >> > > > > > > > >    Move durable JDBC metrics into a
>> > self-contained
>> > > > > > >> > > implementation
>> > > > > > >> > > > > of
>> > > > > > >> > > > > > > the
>> > > > > > >> > > > > > > > >    reporting SPI.
>> > > > > > >> > > > > > > > >
>> > > > > > >> > > > > > > > >    That implementation can own its schema,
>> > > bootstrap,
>> > > > > > >> > > retention,
>> > > > > > >> > > > > and
>> > > > > > >> > > > > > > read
>> > > > > > >> > > > > > > > >    API support. It should not define the core
>> > > > > reporting
>> > > > > > >> SPI
>> > > > > > >> > > > > boundary,
>> > > > > > >> > > > > > > and
>> > > > > > >> > > > > > > > > it
>> > > > > > >> > > > > > > > >    should not require metrics persistence to
>> > > remain
>> > > > > > >> inherited
>> > > > > > >> > > > from
>> > > > > > >> > > > > > > > >    BasePersistence.
>> > > > > > >> > > > > > > > >    5.
>> > > > > > >> > > > > > > > >
>> > > > > > >> > > > > > > > >    Treat async/event-backed handling as
>> another
>> > > > > > >> > implementation
>> > > > > > >> > > of
>> > > > > > >> > > > > the
>> > > > > > >> > > > > > > > same
>> > > > > > >> > > > > > > > >    reporting SPI.
>> > > > > > >> > > > > > > > >
>> > > > > > >> > > > > > > > >    For example, an event-backed reporter
>> could
>> > > > > enqueue a
>> > > > > > >> > > metrics
>> > > > > > >> > > > > > event
>> > > > > > >> > > > > > > > and
>> > > > > > >> > > > > > > > >    let listeners handle durable storage or
>> other
>> > > > > sinks.
>> > > > > > >> If we
>> > > > > > >> > > > later
>> > > > > > >> > > > > > > need
>> > > > > > >> > > > > > > > a
>> > > > > > >> > > > > > > > >    replaceable queue engine, that seems like
>> a
>> > > shared
>> > > > > > >> > > > event/metrics
>> > > > > > >> > > > > > > > > substrate
>> > > > > > >> > > > > > > > >    topic rather than a metrics-only
>> requirement.
>> > > > > > >> > > > > > > > >
>> > > > > > >> > > > > > > > > This framing lets us keep the REST metrics
>> > > endpoint
>> > > > > > >> simple,
>> > > > > > >> > > > > preserve
>> > > > > > >> > > > > > > the
>> > > > > > >> > > > > > > > > current default behavior, support durable
>> > metrics
>> > > > > users,
>> > > > > > >> and
>> > > > > > >> > > > still
>> > > > > > >> > > > > > > leave
>> > > > > > >> > > > > > > > > room for async/event-backed or
>> > > external-queue-based
>> > > > > > >> > > > > implementations.
>> > > > > > >> > > > > > > > >
>> > > > > > >> > > > > > > > > I think the main follow-up questions are:
>> > > > > > >> > > > > > > > >
>> > > > > > >> > > > > > > > >    - Do folks agree that the stable SPI
>> boundary
>> > > > > should
>> > > > > > be
>> > > > > > >> > > > metrics
>> > > > > > >> > > > > > > > >    reporting/emitting, not metrics
>> persistence?
>> > > > > > >> > > > > > > > >    - Does an optional Iceberg metrics
>> extension
>> > > API
>> > > > > > layer
>> > > > > > >> > sound
>> > > > > > >> > > > > like
>> > > > > > >> > > > > > > the
>> > > > > > >> > > > > > > > >    right home for this SPI?
>> > > > > > >> > > > > > > > >    - Should the current durable metrics work
>> be
>> > > > > reframed
>> > > > > > >> as a
>> > > > > > >> > > > > durable
>> > > > > > >> > > > > > > > JDBC
>> > > > > > >> > > > > > > > >    reference implementation of that SPI?
>> > > > > > >> > > > > > > > >    - What is the smallest PR sequence to get
>> > there
>> > > > > > without
>> > > > > > >> > > > blocking
>> > > > > > >> > > > > > the
>> > > > > > >> > > > > > > > >    current metrics work unnecessarily?
>> > > > > > >> > > > > > > > >
>> > > > > > >> > > > > > > > > Thanks,
>> > > > > > >> > > > > > > > > -ej
>> > > > > > >> > > > > > > > >
>> > > > > > >> > > > > > > > > On Fri, May 15, 2026 at 5:16 PM Dmitri
>> > > Bourlatchkov <
>> > > > > > >> > > > > > dimas@apache.org>
>> > > > > > >> > > > > > > > > wrote:
>> > > > > > >> > > > > > > > >
>> > > > > > >> > > > > > > > > > Hi JB,
>> > > > > > >> > > > > > > > > >
>> > > > > > >> > > > > > > > > > Could you set up another meeting, please?
>> Same
>> > > time
>> > > > > on
>> > > > > > >> > > > Wednesday
>> > > > > > >> > > > > as
>> > > > > > >> > > > > > > > last
>> > > > > > >> > > > > > > > > > time... I hope it works for everyone.
>> > > > > > >> > > > > > > > > >
>> > > > > > >> > > > > > > > > > Cheers,
>> > > > > > >> > > > > > > > > > Dmitri.
>> > > > > > >> > > > > > > > > >
>> > > > > > >> > > > > > > > > > On Fri, May 15, 2026 at 8:06 PM Yufei Gu <
>> > > > > > >> > > flyrain000@gmail.com
>> > > > > > >> > > > >
>> > > > > > >> > > > > > > wrote:
>> > > > > > >> > > > > > > > > >
>> > > > > > >> > > > > > > > > > > +1 on another sync call next week.
>> > > > > > >> > > > > > > > > > >
>> > > > > > >> > > > > > > > > > > Yufei
>> > > > > > >> > > > > > > > > > >
>> > > > > > >> > > > > > > > > > >
>> > > > > > >> > > > > > > > > > > On Fri, May 15, 2026 at 4:52 PM Dmitri
>> > > > > Bourlatchkov
>> > > > > > <
>> > > > > > >> > > > > > > > dimas@apache.org>
>> > > > > > >> > > > > > > > > > > wrote:
>> > > > > > >> > > > > > > > > > >
>> > > > > > >> > > > > > > > > > > > Hi All,
>> > > > > > >> > > > > > > > > > > >
>> > > > > > >> > > > > > > > > > > > WDYT about another sync call next week?
>> > > > > > >> > > > > > > > > > > >
>> > > > > > >> > > > > > > > > > > > Thanks,
>> > > > > > >> > > > > > > > > > > > Dmitri.
>> > > > > > >> > > > > > > > > > > >
>> > > > > > >> > > > > > > > > > > > On Wed, May 6, 2026 at 5:29 PM Dmitri
>> > > > > > Bourlatchkov <
>> > > > > > >> > > > > > > > dimas@apache.org
>> > > > > > >> > > > > > > > > >
>> > > > > > >> > > > > > > > > > > > wrote:
>> > > > > > >> > > > > > > > > > > >
>> > > > > > >> > > > > > > > > > > > > Hi EJ,
>> > > > > > >> > > > > > > > > > > > >
>> > > > > > >> > > > > > > > > > > > > Thanks for the summary! It covers
>> what
>> > we
>> > > > > > >> discussed
>> > > > > > >> > in
>> > > > > > >> > > > the
>> > > > > > >> > > > > > > > meeting
>> > > > > > >> > > > > > > > > > very
>> > > > > > >> > > > > > > > > > > > > well, IMHO.
>> > > > > > >> > > > > > > > > > > > >
>> > > > > > >> > > > > > > > > > > > > Looking forward to concrete PRs :)
>> > > > > > >> > > > > > > > > > > > >
>> > > > > > >> > > > > > > > > > > > > Cheers,
>> > > > > > >> > > > > > > > > > > > > Dmitri.
>> > > > > > >> > > > > > > > > > > > >
>> > > > > > >> > > > > > > > > > > > > On Wed, May 6, 2026 at 5:08 PM EJ
>> Wang <
>> > > > > > >> > > > > > > > > > ej.wang.dev.interest@gmail.com
>> > > > > > >> > > > > > > > > > > >
>> > > > > > >> > > > > > > > > > > > > wrote:
>> > > > > > >> > > > > > > > > > > > >
>> > > > > > >> > > > > > > > > > > > >> Hi folks,
>> > > > > > >> > > > > > > > > > > > >>
>> > > > > > >> > > > > > > > > > > > >> We had a community sync earlier,
>> thanks
>> > > JB
>> > > > > for
>> > > > > > >> > > > scheduling
>> > > > > > >> > > > > > it.
>> > > > > > >> > > > > > > > > Notes
>> > > > > > >> > > > > > > > > > > from
>> > > > > > >> > > > > > > > > > > > >> the first metrics architecture sync
>> > (May
>> > > 6,
>> > > > > > >> 10-11am
>> > > > > > >> > > PT).
>> > > > > > >> > > > > > > > > Discussion
>> > > > > > >> > > > > > > > > > > doc
>> > > > > > >> > > > > > > > > > > > >> with per-section status:
>> > > > > > >> > > > > > > > > > > > >>
>> > > > > > >> > > > > > > > > > > > >>
>> > > > > > >> > > > > > > > > > > >
>> > > > > > >> > > > > > > > > > >
>> > > > > > >> > > > > > > > > >
>> > > > > > >> > > > > > > > >
>> > > > > > >> > > > > > > >
>> > > > > > >> > > > > > >
>> > > > > > >> > > > > >
>> > > > > > >> > > > >
>> > > > > > >> > > >
>> > > > > > >> > >
>> > > > > > >> >
>> > > > > > >>
>> > > > > >
>> > > > >
>> > >
>> >
>> https://urldefense.com/v3/__https://docs.google.com/document/d/100h7c4damrUzVuquYbBHM0EvA4LSWuW2IT2dN_7nYVA/edit?tab=t.0__;!!Iz9xO38YGHZK!4rnlVIqBQQQ6xpGe1PMV58edOS_cGrKoqdfhw4qSp_JQCavvEjaxXk0OwGPnxyEcO60AzKQK_XDtp7H3DdkFXcrXgIddeUs$
>> > > > > > >> > > > > > > > > > > > >>
>> > > > > > >> > > > > > > > > > > > >> *The meeting covered both topics
>> from
>> > the
>> > > > > doc.
>> > > > > > >> > > > > > Direction-level
>> > > > > > >> > > > > > > > > > > alignment
>> > > > > > >> > > > > > > > > > > > >> was reached on the headline pieces;
>> > > details
>> > > > > > >> remain
>> > > > > > >> > for
>> > > > > > >> > > > PR
>> > > > > > >> > > > > > > review
>> > > > > > >> > > > > > > > > or
>> > > > > > >> > > > > > > > > > > > >> follow-up sessions.*
>> > > > > > >> > > > > > > > > > > > >>
>> > > > > > >> > > > > > > > > > > > >> *Topic 1 — Persistence schema
>> redesign*
>> > > > > > >> > > > > > > > > > > > >> Idea-level alignment on
>> consolidating
>> > > > > per-type
>> > > > > > >> > tables
>> > > > > > >> > > > > > > > > > > > >> (scan_metrics_report,
>> > > > > > >> > > > > > > > > > > > >> commit_metrics_report) into a single
>> > > > > > >> metrics_report
>> > > > > > >> > > > table.
>> > > > > > >> > > > > > The
>> > > > > > >> > > > > > > > > > > > motivating
>> > > > > > >> > > > > > > > > > > > >> cost is the surface area added by
>> every
>> > > new
>> > > > > > >> metric
>> > > > > > >> > > type
>> > > > > > >> > > > > > today:
>> > > > > > >> > > > > > > > new
>> > > > > > >> > > > > > > > > > > > table,
>> > > > > > >> > > > > > > > > > > > >> SPI method, record class, model,
>> > > converter,
>> > > > > > >> schema
>> > > > > > >> > > > > > migration.
>> > > > > > >> > > > > > > > > > > > >>
>> > > > > > >> > > > > > > > > > > > >> Most schema details are deferred to
>> the
>> > > > > schema
>> > > > > > >> PR. A
>> > > > > > >> > > few
>> > > > > > >> > > > > > > > specific
>> > > > > > >> > > > > > > > > > > points
>> > > > > > >> > > > > > > > > > > > >> came up:
>> > > > > > >> > > > > > > > > > > > >> •   metric_schema_version: Yufei
>> > prefers
>> > > > > > dropping
>> > > > > > >> > it,
>> > > > > > >> > > > > since
>> > > > > > >> > > > > > > > there
>> > > > > > >> > > > > > > > > is
>> > > > > > >> > > > > > > > > > > no
>> > > > > > >> > > > > > > > > > > > >> spec-level concept of metrics
>> > versioning
>> > > > > today
>> > > > > > >> and
>> > > > > > >> > it
>> > > > > > >> > > is
>> > > > > > >> > > > > > hard
>> > > > > > >> > > > > > > to
>> > > > > > >> > > > > > > > > > > define
>> > > > > > >> > > > > > > > > > > > >> unilaterally. Robert prefers keeping
>> > it,
>> > > > > given
>> > > > > > >> IRC
>> > > > > > >> > v2
>> > > > > > >> > > is
>> > > > > > >> > > > > > > coming
>> > > > > > >> > > > > > > > > and
>> > > > > > >> > > > > > > > > > > the
>> > > > > > >> > > > > > > > > > > > >> schema should be considered against
>> its
>> > > > > likely
>> > > > > > >> > shape;
>> > > > > > >> > > > > Robert
>> > > > > > >> > > > > > > > also
>> > > > > > >> > > > > > > > > > > raised
>> > > > > > >> > > > > > > > > > > > >> how to differentiate various payload
>> > > formats
>> > > > > if
>> > > > > > >> any.
>> > > > > > >> > > > EJ's
>> > > > > > >> > > > > > read
>> > > > > > >> > > > > > > > is
>> > > > > > >> > > > > > > > > > that
>> > > > > > >> > > > > > > > > > > > >> this
>> > > > > > >> > > > > > > > > > > > >> is a two-way-door decision. We can
>> > start
>> > > > > > without
>> > > > > > >> the
>> > > > > > >> > > > > field,
>> > > > > > >> > > > > > > and
>> > > > > > >> > > > > > > > if
>> > > > > > >> > > > > > > > > > IRC
>> > > > > > >> > > > > > > > > > > > v2
>> > > > > > >> > > > > > > > > > > > >> changes the shape we would likely
>> roll
>> > a
>> > > > > > >> > corresponding
>> > > > > > >> > > > new
>> > > > > > >> > > > > > > > schema
>> > > > > > >> > > > > > > > > > > > anyway,
>> > > > > > >> > > > > > > > > > > > >> which is not particularly costly.
>> > > > > > >> > > > > > > > > > > > >> •   Payload format: Robert pointed
>> out
>> > > that
>> > > > > > >> future
>> > > > > > >> > > > formats
>> > > > > > >> > > > > > > > beyond
>> > > > > > >> > > > > > > > > > JSON
>> > > > > > >> > > > > > > > > > > > may
>> > > > > > >> > > > > > > > > > > > >> be worth supporting. The exact
>> shape is
>> > > > > > deferred
>> > > > > > >> to
>> > > > > > >> > > the
>> > > > > > >> > > > > > schema
>> > > > > > >> > > > > > > > > > > > discussion.
>> > > > > > >> > > > > > > > > > > > >> •   Partition strategy: Anand
>> suggested
>> > > > > monthly
>> > > > > > >> > > > > partitioning
>> > > > > > >> > > > > > > > based
>> > > > > > >> > > > > > > > > > on
>> > > > > > >> > > > > > > > > > > > his
>> > > > > > >> > > > > > > > > > > > >> experience as potentially helpful at
>> > > scale.
>> > > > > > >> > > > > > > > > > > > >>
>> > > > > > >> > > > > > > > > > > > >> *Topic 2 — Where metrics ingestion
>> and
>> > > > > storage
>> > > > > > >> > belong*
>> > > > > > >> > > > > > > > > > > > >> Idea-level alignment that metrics
>> > should
>> > > be a
>> > > > > > >> > > separated
>> > > > > > >> > > > > SPI
>> > > > > > >> > > > > > > from
>> > > > > > >> > > > > > > > > the
>> > > > > > >> > > > > > > > > > > > >> entity
>> > > > > > >> > > > > > > > > > > > >> persistence stack. Two reasons
>> > surfaced:
>> > > (a)
>> > > > > > >> > workloads
>> > > > > > >> > > > and
>> > > > > > >> > > > > > > > > > capability
>> > > > > > >> > > > > > > > > > > > >> requirements diverge enough that
>> > coupling
>> > > > > them
>> > > > > > >> > creates
>> > > > > > >> > > > > > > > artificial
>> > > > > > >> > > > > > > > > > > > >> constraints, and (b) admin
>> experience
>> > > > > improves
>> > > > > > >> when
>> > > > > > >> > > > > metrics
>> > > > > > >> > > > > > > has
>> > > > > > >> > > > > > > > > its
>> > > > > > >> > > > > > > > > > > own
>> > > > > > >> > > > > > > > > > > > >> bootstrap, retention, and lifecycle.
>> > > Dmitri
>> > > > > > noted
>> > > > > > >> > > > Polaris
>> > > > > > >> > > > > > > being
>> > > > > > >> > > > > > > > a
>> > > > > > >> > > > > > > > > > > > platform
>> > > > > > >> > > > > > > > > > > > >> should have the flexibility to
>> support
>> > > > > > different
>> > > > > > >> > > > > persistence
>> > > > > > >> > > > > > > > > > backends
>> > > > > > >> > > > > > > > > > > > per
>> > > > > > >> > > > > > > > > > > > >> concern, and pointed to a concrete
>> next
>> > > step
>> > > > > of
>> > > > > > >> > > > separating
>> > > > > > >> > > > > > the
>> > > > > > >> > > > > > > > > JDBC
>> > > > > > >> > > > > > > > > > > > >> bootstrap for metrics from the
>> > metastore
>> > > > > > >> bootstrap.
>> > > > > > >> > > > Robert
>> > > > > > >> > > > > > > > > proposed
>> > > > > > >> > > > > > > > > > an
>> > > > > > >> > > > > > > > > > > > >> additional UX extension: detect an
>> > > > > > unbootstrapped
>> > > > > > >> > > > metrics
>> > > > > > >> > > > > > > store
>> > > > > > >> > > > > > > > on
>> > > > > > >> > > > > > > > > > > first
>> > > > > > >> > > > > > > > > > > > >> use and auto-bootstrap rather than
>> > > requiring
>> > > > > an
>> > > > > > >> > > explicit
>> > > > > > >> > > > > > > manual
>> > > > > > >> > > > > > > > > > > > bootstrap
>> > > > > > >> > > > > > > > > > > > >> step.
>> > > > > > >> > > > > > > > > > > > >> The meeting also confirmed that
>> Polaris
>> > > > > metrics
>> > > > > > >> can
>> > > > > > >> > > > start
>> > > > > > >> > > > > > > small
>> > > > > > >> > > > > > > > > and
>> > > > > > >> > > > > > > > > > > stay
>> > > > > > >> > > > > > > > > > > > >> Iceberg-focused. Naming and
>> persistence
>> > > > > schema
>> > > > > > >> can
>> > > > > > >> > > lean
>> > > > > > >> > > > > > > > > > > > Iceberg-specific.
>> > > > > > >> > > > > > > > > > > > >> If a future expansion to
>> generic-table
>> > > > > metrics
>> > > > > > or
>> > > > > > >> > > > > > operational
>> > > > > > >> > > > > > > > > > metrics
>> > > > > > >> > > > > > > > > > > > >> arrives, an abstraction layer can be
>> > > built on
>> > > > > > >> top of
>> > > > > > >> > > the
>> > > > > > >> > > > > > > Iceberg
>> > > > > > >> > > > > > > > > > > metrics
>> > > > > > >> > > > > > > > > > > > >> reporter at that point. Robert
>> remains
>> > > on the
>> > > > > > >> fence
>> > > > > > >> > > and
>> > > > > > >> > > > > > would
>> > > > > > >> > > > > > > > > prefer
>> > > > > > >> > > > > > > > > > > > >> something more generic but did not
>> > block
>> > > the
>> > > > > > >> > > direction;
>> > > > > > >> > > > > > > Dmitri's
>> > > > > > >> > > > > > > > > > read
>> > > > > > >> > > > > > > > > > > > was
>> > > > > > >> > > > > > > > > > > > >> that the proposed framework already
>> has
>> > > > > enough
>> > > > > > >> > > > flexibility
>> > > > > > >> > > > > > to
>> > > > > > >> > > > > > > > > absorb
>> > > > > > >> > > > > > > > > > > > >> future
>> > > > > > >> > > > > > > > > > > > >> expansion.
>> > > > > > >> > > > > > > > > > > > >>
>> > > > > > >> > > > > > > > > > > > >> The Trade-offs and Proposed
>> structure
>> > > > > sections
>> > > > > > in
>> > > > > > >> > the
>> > > > > > >> > > > doc
>> > > > > > >> > > > > > were
>> > > > > > >> > > > > > > > not
>> > > > > > >> > > > > > > > > > > > >> reviewed
>> > > > > > >> > > > > > > > > > > > >> in detail. They remain open for
>> either
>> > > the
>> > > > > next
>> > > > > > >> sync
>> > > > > > >> > > or
>> > > > > > >> > > > PR
>> > > > > > >> > > > > > > > review.
>> > > > > > >> > > > > > > > > > > > >>
>> > > > > > >> > > > > > > > > > > > >> *Cross-cutting alignment —
>> > > battery-included
>> > > > > > plus
>> > > > > > >> > > > > pluggable*
>> > > > > > >> > > > > > > > > > > > >> A common philosophy emerged from the
>> > > > > > discussion.
>> > > > > > >> EJ
>> > > > > > >> > > > > > summarized
>> > > > > > >> > > > > > > > it
>> > > > > > >> > > > > > > > > > as:
>> > > > > > >> > > > > > > > > > > > >> Polaris should provide a
>> > > battery-included UX
>> > > > > > for
>> > > > > > >> > > > beginners
>> > > > > > >> > > > > > and
>> > > > > > >> > > > > > > > the
>> > > > > > >> > > > > > > > > > > > >> flexibility for advanced users to
>> swap
>> > > the
>> > > > > > >> included
>> > > > > > >> > > > > battery
>> > > > > > >> > > > > > > for
>> > > > > > >> > > > > > > > > > > > something
>> > > > > > >> > > > > > > > > > > > >> more powerful or tailored to their
>> use
>> > > case.
>> > > > > > The
>> > > > > > >> SPI
>> > > > > > >> > > > > design
>> > > > > > >> > > > > > > > needs
>> > > > > > >> > > > > > > > > to
>> > > > > > >> > > > > > > > > > > > >> enable
>> > > > > > >> > > > > > > > > > > > >> both.
>> > > > > > >> > > > > > > > > > > > >>
>> > > > > > >> > > > > > > > > > > > >> The inputs that shaped this framing:
>> > > > > > >> > > > > > > > > > > > >> •   Anand described how his team
>> uses
>> > the
>> > > > > > current
>> > > > > > >> > > > metrics
>> > > > > > >> > > > > > > > > > persistence
>> > > > > > >> > > > > > > > > > > > >> (three metrics consumers in v1.4).
>> > > > > > >> > > > > > > > > > > > >> •   Yufei raised Grafana and
>> dashboard
>> > > > > > >> integrations
>> > > > > > >> > > as a
>> > > > > > >> > > > > > > > > destination
>> > > > > > >> > > > > > > > > > > use
>> > > > > > >> > > > > > > > > > > > >> case beyond the default.
>> > > > > > >> > > > > > > > > > > > >> •   Robert called out that the
>> current
>> > > design
>> > > > > > is
>> > > > > > >> > more
>> > > > > > >> > > > > > > > > JDBC-focused.
>> > > > > > >> > > > > > > > > > > > >>
>> > > > > > >> > > > > > > > > > > > >> Two concrete instances:
>> > > > > > >> > > > > > > > > > > > >> •   Async metrics intake: Yufei's
>> > initial
>> > > > > > >> position
>> > > > > > >> > was
>> > > > > > >> > > > > that
>> > > > > > >> > > > > > > > async
>> > > > > > >> > > > > > > > > > > should
>> > > > > > >> > > > > > > > > > > > >> largely live on the producer side
>> and
>> > > there
>> > > > > is
>> > > > > > >> not
>> > > > > > >> > > much
>> > > > > > >> > > > > > > Polaris
>> > > > > > >> > > > > > > > > can
>> > > > > > >> > > > > > > > > > > do.
>> > > > > > >> > > > > > > > > > > > >> Robert suggested a Polaris-side
>> default
>> > > is
>> > > > > > doable
>> > > > > > >> > via
>> > > > > > >> > > > > > Vert.x.
>> > > > > > >> > > > > > > > > Dmitri
>> > > > > > >> > > > > > > > > > > > >> agreed
>> > > > > > >> > > > > > > > > > > > >> the direction is worth exploring.
>> The
>> > > meeting
>> > > > > > >> > > converged
>> > > > > > >> > > > > on a
>> > > > > > >> > > > > > > > > > > > >> battery-included default (likely
>> > > > > Vert.x-backed)
>> > > > > > >> with
>> > > > > > >> > > an
>> > > > > > >> > > > > SPI
>> > > > > > >> > > > > > > > shape
>> > > > > > >> > > > > > > > > > that
>> > > > > > >> > > > > > > > > > > > >> lets
>> > > > > > >> > > > > > > > > > > > >> power users route to a more scalable
>> > > backend
>> > > > > > >> > > (k8s-hosted
>> > > > > > >> > > > > > > queue,
>> > > > > > >> > > > > > > > > AWS
>> > > > > > >> > > > > > > > > > > SQS,
>> > > > > > >> > > > > > > > > > > > >> etc.).
>> > > > > > >> > > > > > > > > > > > >> •   Pluggable destinations:
>> combining
>> > > Yufei's
>> > > > > > >> > > dashboard
>> > > > > > >> > > > > use
>> > > > > > >> > > > > > > case
>> > > > > > >> > > > > > > > > > with
>> > > > > > >> > > > > > > > > > > > >> Robert's JDBC-focused call-out, the
>> > > meeting
>> > > > > > >> agreed
>> > > > > > >> > the
>> > > > > > >> > > > SPI
>> > > > > > >> > > > > > > > should
>> > > > > > >> > > > > > > > > be
>> > > > > > >> > > > > > > > > > > > >> structured for multiple sinks so
>> > > integrations
>> > > > > > >> become
>> > > > > > >> > > > impl
>> > > > > > >> > > > > > > > choices
>> > > > > > >> > > > > > > > > > > rather
>> > > > > > >> > > > > > > > > > > > >> than architectural changes.
>> > > > > > >> > > > > > > > > > > > >>
>> > > > > > >> > > > > > > > > > > > >> The battery-included default is most
>> > > likely
>> > > > > to
>> > > > > > >> use
>> > > > > > >> > the
>> > > > > > >> > > > > > > existing
>> > > > > > >> > > > > > > > > > > > >> JDBC-backed
>> > > > > > >> > > > > > > > > > > > >> approach.
>> > > > > > >> > > > > > > > > > > > >>
>> > > > > > >> > > > > > > > > > > > >> *Direction (idea-level alignment)*
>> > > > > > >> > > > > > > > > > > > >> •   Single metrics_report table
>> > > consolidating
>> > > > > > >> > per-type
>> > > > > > >> > > > > > > metrics,
>> > > > > > >> > > > > > > > > > > > replacing
>> > > > > > >> > > > > > > > > > > > >> scan_metrics_report and
>> > > commit_metrics_report
>> > > > > > >> > > > > > > > > > > > >> •   Iceberg-focused naming and
>> schema
>> > for
>> > > > > now,
>> > > > > > >> > revisit
>> > > > > > >> > > > if
>> > > > > > >> > > > > > > > > > > generic-table
>> > > > > > >> > > > > > > > > > > > or
>> > > > > > >> > > > > > > > > > > > >> operational metrics arrive
>> > > > > > >> > > > > > > > > > > > >> •   Metrics persistence as a
>> separated
>> > > SPI,
>> > > > > not
>> > > > > > >> on
>> > > > > > >> > > > > > > > BasePersistence
>> > > > > > >> > > > > > > > > > > > >> •   Bootstrap path separated for
>> > metrics,
>> > > > > > >> > independent
>> > > > > > >> > > of
>> > > > > > >> > > > > > > > metastore
>> > > > > > >> > > > > > > > > > > > >> bootstrap
>> > > > > > >> > > > > > > > > > > > >> •   "Battery-included plus
>> pluggable"
>> > as
>> > > the
>> > > > > > SPI
>> > > > > > >> > > design
>> > > > > > >> > > > > > > > philosophy
>> > > > > > >> > > > > > > > > > > > >>
>> > > > > > >> > > > > > > > > > > > >> *Open items*
>> > > > > > >> > > > > > > > > > > > >> •   Schema details:
>> > > metric_schema_version,
>> > > > > > >> payload
>> > > > > > >> > > > format,
>> > > > > > >> > > > > > IRC
>> > > > > > >> > > > > > > > v2
>> > > > > > >> > > > > > > > > > > > >> forward-compat shape
>> > > > > > >> > > > > > > > > > > > >> •   SPI design details — full review
>> > > either
>> > > > > in
>> > > > > > >> the
>> > > > > > >> > > next
>> > > > > > >> > > > > sync
>> > > > > > >> > > > > > > or
>> > > > > > >> > > > > > > > in
>> > > > > > >> > > > > > > > > > the
>> > > > > > >> > > > > > > > > > > > >> corresponding PR
>> > > > > > >> > > > > > > > > > > > >> •   Schema refactor PR ownership
>> > > > > > >> > > > > > > > > > > > >>
>> > > > > > >> > > > > > > > > > > > >> *Action items*
>> > > > > > >> > > > > > > > > > > > >> •   EJ to take a first stab at the
>> SPI
>> > > design
>> > > > > > and
>> > > > > > >> > > > > > potentially
>> > > > > > >> > > > > > > > > > partner
>> > > > > > >> > > > > > > > > > > > with
>> > > > > > >> > > > > > > > > > > > >> Anand to incorporate the lessons
>> > learned
>> > > from
>> > > > > > the
>> > > > > > >> > > > existing
>> > > > > > >> > > > > > > > > reporter
>> > > > > > >> > > > > > > > > > > and
>> > > > > > >> > > > > > > > > > > > >> persistence work.
>> > > > > > >> > > > > > > > > > > > >> •   Schema refactor PR ownership is
>> not
>> > > yet
>> > > > > > >> decided.
>> > > > > > >> > > If
>> > > > > > >> > > > > > anyone
>> > > > > > >> > > > > > > > is
>> > > > > > >> > > > > > > > > > > > >> interested in driving it, reply on
>> this
>> > > > > thread.
>> > > > > > >> > > > > > > > > > > > >> •   JB to schedule the next sync,
>> > > tentatively
>> > > > > > in
>> > > > > > >> two
>> > > > > > >> > > > > weeks.
>> > > > > > >> > > > > > > > > > > > >>
>> > > > > > >> > > > > > > > > > > > >> -ej
>> > > > > > >> > > > > > > > > > > > >>
>> > > > > > >> > > > > > > > > > > > >> On Mon, Apr 27, 2026 at 3:07 PM EJ
>> > Wang <
>> > > > > > >> > > > > > > > > > > ej.wang.dev.interest@gmail.com
>> > > > > > >> > > > > > > > > > > > >
>> > > > > > >> > > > > > > > > > > > >> wrote:
>> > > > > > >> > > > > > > > > > > > >>
>> > > > > > >> > > > > > > > > > > > >> > Thanks Yufei for the +1.
>> > > > > > >> > > > > > > > > > > > >> >
>> > > > > > >> > > > > > > > > > > > >> > JB, could you help add a biweekly
>> > > metrics
>> > > > > > >> > > architecture
>> > > > > > >> > > > > > sync
>> > > > > > >> > > > > > > to
>> > > > > > >> > > > > > > > > the
>> > > > > > >> > > > > > > > > > > > >> Polaris
>> > > > > > >> > > > > > > > > > > > >> > community calendar? I'm thinking
>> > > Thursdays
>> > > > > at
>> > > > > > >> > 9-10am
>> > > > > > >> > > > PT,
>> > > > > > >> > > > > > on
>> > > > > > >> > > > > > > > the
>> > > > > > >> > > > > > > > > > > > >> off-weeks
>> > > > > > >> > > > > > > > > > > > >> > from the community meeting
>> (starting
>> > > May
>> > > > > 7),
>> > > > > > 60
>> > > > > > >> > > > minutes.
>> > > > > > >> > > > > > > > > > > > >> >
>> > > > > > >> > > > > > > > > > > > >> > Here's a rough agenda to work
>> through
>> > > over
>> > > > > > the
>> > > > > > >> > first
>> > > > > > >> > > > few
>> > > > > > >> > > > > > > > > sessions,
>> > > > > > >> > > > > > > > > > > > >> grouped
>> > > > > > >> > > > > > > > > > > > >> > by priority:
>> > > > > > >> > > > > > > > > > > > >> >
>> > > > > > >> > > > > > > > > > > > >> > *First: foundational direction*
>> > > > > > >> > > > > > > > > > > > >> >
>> > > > > > >> > > > > > > > > > > > >> > 1.  MetricsPersistence: public
>> SPI or
>> > > > > > internal
>> > > > > > >> > > > > > > implementation
>> > > > > > >> > > > > > > > > > > detail?
>> > > > > > >> > > > > > > > > > > > >> >    •   Marked @Beta, javadoc calls
>> > it a
>> > > > > > >> "Service
>> > > > > > >> > > > > Provider
>> > > > > > >> > > > > > > > > > > Interface",
>> > > > > > >> > > > > > > > > > > > >> but
>> > > > > > >> > > > > > > > > > > > >> > only one consumer
>> > > > > (JdbcBasePersistenceImpl),
>> > > > > > >> lives
>> > > > > > >> > > on
>> > > > > > >> > > > > > > > > > > BasePersistence.
>> > > > > > >> > > > > > > > > > > > >> If
>> > > > > > >> > > > > > > > > > > > >> > demoted to a private helper
>> inside a
>> > > > > > persisting
>> > > > > > >> > > > reporter
>> > > > > > >> > > > > > > impl,
>> > > > > > >> > > > > > > > > > most
>> > > > > > >> > > > > > > > > > > > >> > downstream design decisions become
>> > > > > > >> implementation
>> > > > > > >> > > > > details
>> > > > > > >> > > > > > > > rather
>> > > > > > >> > > > > > > > > > > than
>> > > > > > >> > > > > > > > > > > > >> > contract questions.
>> > > > > > >> > > > > > > > > > > > >> >
>> > > > > > >> > > > > > > > > > > > >> > 2.  Persistence schema redesign
>> > > > > > >> > > > > > > > > > > > >> >    •   Current two-table layout
>> > > > > > >> > > (scan_metrics_report,
>> > > > > > >> > > > > > > > > > > > >> > commit_metrics_report) with ~25
>> > > flattened
>> > > > > > >> columns
>> > > > > > >> > > > each.
>> > > > > > >> > > > > > > Every
>> > > > > > >> > > > > > > > > new
>> > > > > > >> > > > > > > > > > > > metric
>> > > > > > >> > > > > > > > > > > > >> > type requires a new table, SPI
>> > method,
>> > > > > record
>> > > > > > >> > class,
>> > > > > > >> > > > > > model,
>> > > > > > >> > > > > > > > > > > converter,
>> > > > > > >> > > > > > > > > > > > >> and
>> > > > > > >> > > > > > > > > > > > >> > schema migration. Direction to
>> > explore:
>> > > > > > single
>> > > > > > >> > table
>> > > > > > >> > > > > with
>> > > > > > >> > > > > > > > > > > metric_type
>> > > > > > >> > > > > > > > > > > > >> enum,
>> > > > > > >> > > > > > > > > > > > >> > schema_version, and JSON payload
>> > > column.
>> > > > > > >> > > > > > > > > > > > >> >
>> > > > > > >> > > > > > > > > > > > >> > *Second: design details once
>> > direction
>> > > is
>> > > > > > set*
>> > > > > > >> > > > > > > > > > > > >> >
>> > > > > > >> > > > > > > > > > > > >> > 3.  Partition key strategy
>> > > > > > >> > > > > > > > > > > > >> >    •   Single-table design means
>> scan
>> > > > > metrics
>> > > > > > >> at
>> > > > > > >> > > scale
>> > > > > > >> > > > > > will
>> > > > > > >> > > > > > > > have
>> > > > > > >> > > > > > > > > > > high
>> > > > > > >> > > > > > > > > > > > >> > write concurrency per table.
>> Schema
>> > > needs
>> > > > > to
>> > > > > > >> > expose
>> > > > > > >> > > > > enough
>> > > > > > >> > > > > > > > > > structure
>> > > > > > >> > > > > > > > > > > > for
>> > > > > > >> > > > > > > > > > > > >> > backends to shard by entity or
>> time
>> > > range.
>> > > > > > >> > > > > > > > > > > > >> >
>> > > > > > >> > > > > > > > > > > > >> > 4.  Read/write path consistency
>> > > > > > >> > > > > > > > > > > > >> >    •   Writes go through
>> > > > > > PolarisMetricsManager
>> > > > > > >> on
>> > > > > > >> > > > > > > > > > MetaStoreManager.
>> > > > > > >> > > > > > > > > > > > >> Reads
>> > > > > > >> > > > > > > > > > > > >> > bypass MetaStoreManager and go
>> > > straight to
>> > > > > > >> > > > > > BasePersistence,
>> > > > > > >> > > > > > > > > > > excluding
>> > > > > > >> > > > > > > > > > > > >> > non-JDBC backends from the read
>> API.
>> > > > > > >> > > > > > > > > > > > >> >
>> > > > > > >> > > > > > > > > > > > >> > *Third: cleanup and alignment*
>> > > > > > >> > > > > > > > > > > > >> >
>> > > > > > >> > > > > > > > > > > > >> > 5.  PolarisMetricsReporter naming
>> > > > > > >> > > > > > > > > > > > >> >    •   Only handles IRC
>> > > > > > >> (ScanReport/CommitReport),
>> > > > > > >> > > > > doesn't
>> > > > > > >> > > > > > > > cover
>> > > > > > >> > > > > > > > > > > > generic
>> > > > > > >> > > > > > > > > > > > >> > tables or operational metrics.
>> Name
>> > is
>> > > > > > broader
>> > > > > > >> > than
>> > > > > > >> > > > > scope.
>> > > > > > >> > > > > > > > > > > > >> >
>> > > > > > >> > > > > > > > > > > > >> > 6.  PolarisMetricsManager facade
>> > > > > passthrough
>> > > > > > >> > > > > > > > > > > > >> >    •   Entire default method is
>> > > > > > >> > > > > > > > > > > > >>
>> > callCtx.getMetaStore().writeScanReport().
>> > > > > > >> > > > > > > > > > > > >> > Zero logic, passes Level 1
>> straight
>> > > through
>> > > > > > to
>> > > > > > >> > Level
>> > > > > > >> > > > 3.
>> > > > > > >> > > > > > Same
>> > > > > > >> > > > > > > > > > > > >> anti-pattern
>> > > > > > >> > > > > > > > > > > > >> > as PolarisEventManager.
>> > > > > > >> > > > > > > > > > > > >> >
>> > > > > > >> > > > > > > > > > > > >> > 7.  Iceberg community alignment
>> > > > > > >> > > > > > > > > > > > >> >    •   Payload-type extension
>> needs
>> > > > > > discussion
>> > > > > > >> on
>> > > > > > >> > > > > > > dev@iceberg.
>> > > > > > >> > > > > > > > > > > > >> obelix74's
>> > > > > > >> > > > > > > > > > > > >> > Feb thread got zero replies.
>> Needs a
>> > > > > > committer
>> > > > > > >> > > voice.
>> > > > > > >> > > > > > > > > > > > >> >
>> > > > > > >> > > > > > > > > > > > >> > Lets confirm prioritization in the
>> > > first
>> > > > > > >> session.
>> > > > > > >> > > > > > > > > > > > >> >
>> > > > > > >> > > > > > > > > > > > >> > -ej
>> > > > > > >> > > > > > > > > > > > >> >
>> > > > > > >> > > > > > > > > > > > >> > On Tue, Apr 21, 2026 at 3:18 PM
>> Yufei
>> > > Gu <
>> > > > > > >> > > > > > > > flyrain000@gmail.com>
>> > > > > > >> > > > > > > > > > > > wrote:
>> > > > > > >> > > > > > > > > > > > >> >
>> > > > > > >> > > > > > > > > > > > >> >> Thanks everyone for continuing to
>> > > drive
>> > > > > this
>> > > > > > >> > > > forward. I
>> > > > > > >> > > > > > > agree
>> > > > > > >> > > > > > > > > > that
>> > > > > > >> > > > > > > > > > > > the
>> > > > > > >> > > > > > > > > > > > >> >> problem is getting complex enough
>> > > that a
>> > > > > > more
>> > > > > > >> > > > > structured
>> > > > > > >> > > > > > > > > > discussion
>> > > > > > >> > > > > > > > > > > > >> would
>> > > > > > >> > > > > > > > > > > > >> >> help.
>> > > > > > >> > > > > > > > > > > > >> >>
>> > > > > > >> > > > > > > > > > > > >> >> +1 on setting up a biweekly sync
>> for
>> > > the
>> > > > > > >> metrics
>> > > > > > >> > > > > > > > architecture.
>> > > > > > >> > > > > > > > > > I’m
>> > > > > > >> > > > > > > > > > > > >> happy
>> > > > > > >> > > > > > > > > > > > >> >> to
>> > > > > > >> > > > > > > > > > > > >> >> join.
>> > > > > > >> > > > > > > > > > > > >> >>
>> > > > > > >> > > > > > > > > > > > >> >> Yufei
>> > > > > > >> > > > > > > > > > > > >> >>
>> > > > > > >> > > > > > > > > > > > >> >>
>> > > > > > >> > > > > > > > > > > > >> >> On Tue, Apr 21, 2026 at 2:34 PM
>> EJ
>> > > Wang <
>> > > > > > >> > > > > > > > > > > > >> ej.wang.dev.interest@gmail.com>
>> > > > > > >> > > > > > > > > > > > >> >> wrote:
>> > > > > > >> > > > > > > > > > > > >> >>
>> > > > > > >> > > > > > > > > > > > >> >> > Also, I've been looking more
>> > > closely at
>> > > > > > the
>> > > > > > >> > > > > > *persistence
>> > > > > > >> > > > > > > > > schema
>> > > > > > >> > > > > > > > > > > in
>> > > > > > >> > > > > > > > > > > > >> the
>> > > > > > >> > > > > > > > > > > > >> >> > current metrics work*, and I
>> think
>> > > > > > there's a
>> > > > > > >> > > > > structural
>> > > > > > >> > > > > > > > > > rigidity
>> > > > > > >> > > > > > > > > > > > >> problem
>> > > > > > >> > > > > > > > > > > > >> >> > worth raising before the shape
>> > gets
>> > > > > locked
>> > > > > > >> in.
>> > > > > > >> > > > > > > > > > > > >> >> >
>> > > > > > >> > > > > > > > > > > > >> >> > Right now we have two separate
>> > > tables
>> > > > > > >> > > > > > > (scan_metrics_report
>> > > > > > >> > > > > > > > > and
>> > > > > > >> > > > > > > > > > > > >> >> > commit_metrics_report), each
>> with
>> > > ~25
>> > > > > > >> flattened
>> > > > > > >> > > > > columns
>> > > > > > >> > > > > > > > that
>> > > > > > >> > > > > > > > > > > > directly
>> > > > > > >> > > > > > > > > > > > >> >> > mirror the Iceberg report
>> fields.
>> > > The
>> > > > > SPI
>> > > > > > >> > follows
>> > > > > > >> > > > the
>> > > > > > >> > > > > > > same
>> > > > > > >> > > > > > > > > > split:
>> > > > > > >> > > > > > > > > > > > >> >> > writeScanReport and
>> > > writeCommitReport as
>> > > > > > >> > separate
>> > > > > > >> > > > > > > methods,
>> > > > > > >> > > > > > > > > with
>> > > > > > >> > > > > > > > > > > > >> per-type
>> > > > > > >> > > > > > > > > > > > >> >> > record classes, converters, and
>> > > model
>> > > > > > >> objects.
>> > > > > > >> > > *The
>> > > > > > >> > > > > > > > practical
>> > > > > > >> > > > > > > > > > > cost:
>> > > > > > >> > > > > > > > > > > > >> >> > adding a new metric type
>> > > (operational
>> > > > > > >> metrics,
>> > > > > > >> > > for
>> > > > > > >> > > > > > > example)
>> > > > > > >> > > > > > > > > > > > requires
>> > > > > > >> > > > > > > > > > > > >> a
>> > > > > > >> > > > > > > > > > > > >> >> new
>> > > > > > >> > > > > > > > > > > > >> >> > table, a new SPI method, a new
>> > > record
>> > > > > > >> class, a
>> > > > > > >> > > new
>> > > > > > >> > > > > > model
>> > > > > > >> > > > > > > > > > class, a
>> > > > > > >> > > > > > > > > > > > new
>> > > > > > >> > > > > > > > > > > > >> >> > converter branch, and a schema
>> > > > > migration*.
>> > > > > > >> > > That's a
>> > > > > > >> > > > > lot
>> > > > > > >> > > > > > > of
>> > > > > > >> > > > > > > > > > > surface
>> > > > > > >> > > > > > > > > > > > >> area
>> > > > > > >> > > > > > > > > > > > >> >> > for what should be "one more
>> kind
>> > of
>> > > > > > >> metric."
>> > > > > > >> > > > > > > > > > > > >> >> >
>> > > > > > >> > > > > > > > > > > > >> >> > *My bias* would be toward a
>> single
>> > > > > metrics
>> > > > > > >> > table
>> > > > > > >> > > > with
>> > > > > > >> > > > > > *a
>> > > > > > >> > > > > > > > > typed
>> > > > > > >> > > > > > > > > > > JSON
>> > > > > > >> > > > > > > > > > > > >> >> > payload*. Something like:
>> > > metric_type
>> > > > > > >> (enum),
>> > > > > > >> > > > > > entity_id,
>> > > > > > >> > > > > > > > > > > > >> >> > table_identifier, snapshot_id
>> > > > > (nullable),
>> > > > > > >> > > > > received_ts,
>> > > > > > >> > > > > > > > > > > > >> schema_version,
>> > > > > > >> > > > > > > > > > > > >> >> and
>> > > > > > >> > > > > > > > > > > > >> >> > a payload column for the
>> > > metric-specific
>> > > > > > >> data.
>> > > > > > >> > > The
>> > > > > > >> > > > > > > > > metric_type
>> > > > > > >> > > > > > > > > > +
>> > > > > > >> > > > > > > > > > > > >> >> > schema_version pair gives us a
>> > > > > > >> > forward-compatible
>> > > > > > >> > > > > > > contract
>> > > > > > >> > > > > > > > > for
>> > > > > > >> > > > > > > > > > > the
>> > > > > > >> > > > > > > > > > > > >> >> payload
>> > > > > > >> > > > > > > > > > > > >> >> > shape. Adding a new metric type
>> > > becomes
>> > > > > an
>> > > > > > >> enum
>> > > > > > >> > > > value
>> > > > > > >> > > > > > > and a
>> > > > > > >> > > > > > > > > > > payload
>> > > > > > >> > > > > > > > > > > > >> >> schema,
>> > > > > > >> > > > > > > > > > > > >> >> > not a schema migration. One
>> thing
>> > I
>> > > > > think
>> > > > > > we
>> > > > > > >> > need
>> > > > > > >> > > > to
>> > > > > > >> > > > > be
>> > > > > > >> > > > > > > > > > > deliberate
>> > > > > > >> > > > > > > > > > > > >> >> about is
>> > > > > > >> > > > > > > > > > > > >> >> > the partition key design. If
>> all
>> > > metric
>> > > > > > >> types
>> > > > > > >> > > land
>> > > > > > >> > > > in
>> > > > > > >> > > > > > one
>> > > > > > >> > > > > > > > > > table,
>> > > > > > >> > > > > > > > > > > > scan
>> > > > > > >> > > > > > > > > > > > >> >> > metrics at scale (high
>> > concurrency,
>> > > high
>> > > > > > >> > > frequency
>> > > > > > >> > > > > > across
>> > > > > > >> > > > > > > > > many
>> > > > > > >> > > > > > > > > > > > >> tables)
>> > > > > > >> > > > > > > > > > > > >> >> > could easily create hot
>> > partitions.
>> > > We'd
>> > > > > > >> want
>> > > > > > >> > the
>> > > > > > >> > > > > > > > persistence
>> > > > > > >> > > > > > > > > > > layer
>> > > > > > >> > > > > > > > > > > > >> to
>> > > > > > >> > > > > > > > > > > > >> >> be
>> > > > > > >> > > > > > > > > > > > >> >> > able to shard by entity or time
>> > > range,
>> > > > > and
>> > > > > > >> that
>> > > > > > >> > > > means
>> > > > > > >> > > > > > the
>> > > > > > >> > > > > > > > > > logical
>> > > > > > >> > > > > > > > > > > > >> schema
>> > > > > > >> > > > > > > > > > > > >> >> > needs to expose enough
>> structure
>> > for
>> > > > > > >> backends
>> > > > > > >> > to
>> > > > > > >> > > > > > > partition
>> > > > > > >> > > > > > > > > on.
>> > > > > > >> > > > > > > > > > I
>> > > > > > >> > > > > > > > > > > > >> don't
>> > > > > > >> > > > > > > > > > > > >> >> > think the current flattened
>> layout
>> > > gives
>> > > > > > us
>> > > > > > >> > that.
>> > > > > > >> > > > > > > > > > > > >> >> >
>> > > > > > >> > > > > > > > > > > > >> >> > This is getting complex enough
>> > that
>> > > I
>> > > > > > don't
>> > > > > > >> > think
>> > > > > > >> > > > > > ad-hoc
>> > > > > > >> > > > > > > > > PR/ML
>> > > > > > >> > > > > > > > > > > > >> threads
>> > > > > > >> > > > > > > > > > > > >> >> > will converge well. *Would
>> people
>> > be
>> > > > > open
>> > > > > > >> to a
>> > > > > > >> > > > > biweekly
>> > > > > > >> > > > > > > > sync
>> > > > > > >> > > > > > > > > > for
>> > > > > > >> > > > > > > > > > > > >> metrics
>> > > > > > >> > > > > > > > > > > > >> >> > architecture?* I think 30
>> minutes
>> > > every
>> > > > > > two
>> > > > > > >> > weeks
>> > > > > > >> > > > > with
>> > > > > > >> > > > > > > > > > interested
>> > > > > > >> > > > > > > > > > > > >> >> parties
>> > > > > > >> > > > > > > > > > > > >> >> > would be enough to work through
>> > the
>> > > > > > schema,
>> > > > > > >> SPI
>> > > > > > >> > > > > shape,
>> > > > > > >> > > > > > > and
>> > > > > > >> > > > > > > > > read
>> > > > > > >> > > > > > > > > > > API
>> > > > > > >> > > > > > > > > > > > >> >> design
>> > > > > > >> > > > > > > > > > > > >> >> > together. Happy to help set
>> that
>> > up.
>> > > > > > >> > > > > > > > > > > > >> >> >
>> > > > > > >> > > > > > > > > > > > >> >> > -ej
>> > > > > > >> > > > > > > > > > > > >> >> >
>> > > > > > >> > > > > > > > > > > > >> >> > On Mon, Apr 20, 2026 at
>> 2:19 PM EJ
>> > > Wang
>> > > > > <
>> > > > > > >> > > > > > > > > > > > >> ej.wang.dev.interest@gmail.com
>> > > > > > >> > > > > > > > > > > > >> >> >
>> > > > > > >> > > > > > > > > > > > >> >> > wrote:
>> > > > > > >> > > > > > > > > > > > >> >> >
>> > > > > > >> > > > > > > > > > > > >> >> >> Reviewed #4115, left a
>> comment on
>> > > the
>> > > > > > code
>> > > > > > >> > > > > > organization
>> > > > > > >> > > > > > > > > side.
>> > > > > > >> > > > > > > > > > > > >> >> >>
>> > > > > > >> > > > > > > > > > > > >> >> >> One thing stood out: the
>> metrics
>> > > write
>> > > > > > path
>> > > > > > >> > > enters
>> > > > > > >> > > > > > > through
>> > > > > > >> > > > > > > > > > > > >> >> >> PolarisMetricsManager on
>> > > > > > MetaStoreManager,
>> > > > > > >> but
>> > > > > > >> > > the
>> > > > > > >> > > > > new
>> > > > > > >> > > > > > > > read
>> > > > > > >> > > > > > > > > > path
>> > > > > > >> > > > > > > > > > > > >> >> bypasses
>> > > > > > >> > > > > > > > > > > > >> >> >> MetaStoreManager entirely and
>> > goes
>> > > > > > >> straight to
>> > > > > > >> > > > > > > > > BasePersistence
>> > > > > > >> > > > > > > > > > > via
>> > > > > > >> > > > > > > > > > > > >> >> >> callContext.getMetaStore().
>> That
>> > > means
>> > > > > > the
>> > > > > > >> > read
>> > > > > > >> > > > API
>> > > > > > >> > > > > > only
>> > > > > > >> > > > > > > > > works
>> > > > > > >> > > > > > > > > > > for
>> > > > > > >> > > > > > > > > > > > >> >> backends
>> > > > > > >> > > > > > > > > > > > >> >> >> that implement
>> BasePersistence.
>> > > NoSQL
>> > > > > and
>> > > > > > >> > remote
>> > > > > > >> > > > > > > backends
>> > > > > > >> > > > > > > > > > can't
>> > > > > > >> > > > > > > > > > > > >> >> participate.
>> > > > > > >> > > > > > > > > > > > >> >> >>
>> > > > > > >> > > > > > > > > > > > >> >> >> Stepping back, I think the
>> > metrics
>> > > > > > >> subsystem
>> > > > > > >> > is
>> > > > > > >> > > > > > growing
>> > > > > > >> > > > > > > > into
>> > > > > > >> > > > > > > > > > > > >> something
>> > > > > > >> > > > > > > > > > > > >> >> >> real (write + read + REST API
>> +
>> > > AuthZ +
>> > > > > > >> > > > pagination)
>> > > > > > >> > > > > > *but
>> > > > > > >> > > > > > > > the
>> > > > > > >> > > > > > > > > > > > >> >> persistence
>> > > > > > >> > > > > > > > > > > > >> >> >> side is split across two
>> layers
>> > in
>> > > a
>> > > > > way
>> > > > > > >> > that's
>> > > > > > >> > > > hard
>> > > > > > >> > > > > > to
>> > > > > > >> > > > > > > > > > > extend*. I
>> > > > > > >> > > > > > > > > > > > >> put
>> > > > > > >> > > > > > > > > > > > >> >> >> together two diagrams to show
>> > what
>> > > I
>> > > > > mean
>> > > > > > >> (my
>> > > > > > >> > > best
>> > > > > > >> > > > > > > > effort).
>> > > > > > >> > > > > > > > > > > > >> >> >>
>> > > > > > >> > > > > > > > > > > > >> >> >> *Current state* (Diagram 1):
>> > three
>> > > > > > >> interfaces
>> > > > > > >> > at
>> > > > > > >> > > > > three
>> > > > > > >> > > > > > > > > > different
>> > > > > > >> > > > > > > > > > > > >> >> levels.
>> > > > > > >> > > > > > > > > > > > >> >> >> The engine-facing SPI
>> > > > > > >> (PolarisMetricsReporter)
>> > > > > > >> > > is
>> > > > > > >> > > > > > clean.
>> > > > > > >> > > > > > > > But
>> > > > > > >> > > > > > > > > > > > >> >> >> PolarisMetricsManager on
>> > > > > MetaStoreManager
>> > > > > > >> is a
>> > > > > > >> > > > > > > passthrough
>> > > > > > >> > > > > > > > > to
>> > > > > > >> > > > > > > > > > > > >> >> >> MetricsPersistence on
>> > > BasePersistence.
>> > > > > > The
>> > > > > > >> > @Beta
>> > > > > > >> > > > > > > > annotation
>> > > > > > >> > > > > > > > > > and
>> > > > > > >> > > > > > > > > > > > SPI
>> > > > > > >> > > > > > > > > > > > >> >> javadoc
>> > > > > > >> > > > > > > > > > > > >> >> >> are on the BasePersistence
>> layer,
>> > > while
>> > > > > > the
>> > > > > > >> > > actual
>> > > > > > >> > > > > > > > extension
>> > > > > > >> > > > > > > > > > > > points
>> > > > > > >> > > > > > > > > > > > >> >> >> (PolarisMetricsReporter,
>> > > > > > >> > PolarisMetricsManager)
>> > > > > > >> > > > > carry
>> > > > > > >> > > > > > no
>> > > > > > >> > > > > > > > > > > stability
>> > > > > > >> > > > > > > > > > > > >> >> >> annotation. The write path
>> goes
>> > > through
>> > > > > > the
>> > > > > > >> > > > > > > > MetaStoreManager
>> > > > > > >> > > > > > > > > > > > layer,
>> > > > > > >> > > > > > > > > > > > >> the
>> > > > > > >> > > > > > > > > > > > >> >> >> read path doesn't.
>> > > > > > >> > > > > > > > > > > > >> >> >>
>> > > > > > >> > > > > > > > > > > > >> >> >> *What I envision* (Diagram 2):
>> > two
>> > > SPIs
>> > > > > > at
>> > > > > > >> two
>> > > > > > >> > > > > levels.
>> > > > > > >> > > > > > > > > > > > >> >> >> PolarisMetricsReporter stays
>> as
>> > the
>> > > > > > >> > > engine-facing
>> > > > > > >> > > > > SPI.
>> > > > > > >> > > > > > > > > > > > >> >> >> PolarisMetricsManager becomes
>> the
>> > > > > > >> > backend-facing
>> > > > > > >> > > > SPI
>> > > > > > >> > > > > > > with
>> > > > > > >> > > > > > > > > both
>> > > > > > >> > > > > > > > > > > > write
>> > > > > > >> > > > > > > > > > > > >> >> and
>> > > > > > >> > > > > > > > > > > > >> >> >> read methods at the
>> > > MetaStoreManager
>> > > > > > level,
>> > > > > > >> > > where
>> > > > > > >> > > > > any
>> > > > > > >> > > > > > > > > backend
>> > > > > > >> > > > > > > > > > > > (JDBC,
>> > > > > > >> > > > > > > > > > > > >> >> NoSQL,
>> > > > > > >> > > > > > > > > > > > >> >> >> remote) can implement them.
>> > > > > > >> MetricsPersistence
>> > > > > > >> > > on
>> > > > > > >> > > > > > > > > > > BasePersistence
>> > > > > > >> > > > > > > > > > > > >> goes
>> > > > > > >> > > > > > > > > > > > >> >> >> away. Where metrics actually
>> land
>> > > is an
>> > > > > > >> > > > > implementation
>> > > > > > >> > > > > > > > > detail,
>> > > > > > >> > > > > > > > > > > > not a
>> > > > > > >> > > > > > > > > > > > >> >> core
>> > > > > > >> > > > > > > > > > > > >> >> >> interface.
>> > > > > > >> > > > > > > > > > > > >> >> >>
>> > > > > > >> > > > > > > > > > > > >> >> >> *Minor naming thing*:
>> > > > > > >> PolarisMetricsReporter
>> > > > > > >> > is
>> > > > > > >> > > > > > broader
>> > > > > > >> > > > > > > > than
>> > > > > > >> > > > > > > > > > > what
>> > > > > > >> > > > > > > > > > > > it
>> > > > > > >> > > > > > > > > > > > >> >> >> actually handles. It only
>> accepts
>> > > > > Iceberg
>> > > > > > >> REST
>> > > > > > >> > > > > Catalog
>> > > > > > >> > > > > > > > > metrics
>> > > > > > >> > > > > > > > > > > > >> >> (ScanReport,
>> > > > > > >> > > > > > > > > > > > >> >> >> CommitReport via
>> MetricsReport).
>> > > > > Generic
>> > > > > > >> table
>> > > > > > >> > > > > metrics
>> > > > > > >> > > > > > > or
>> > > > > > >> > > > > > > > > > > > >> operational
>> > > > > > >> > > > > > > > > > > > >> >> >> metrics aren't in scope. Not
>> > > blocking,
>> > > > > > but
>> > > > > > >> > worth
>> > > > > > >> > > > > > noting
>> > > > > > >> > > > > > > if
>> > > > > > >> > > > > > > > > the
>> > > > > > >> > > > > > > > > > > > >> metrics
>> > > > > > >> > > > > > > > > > > > >> >> >> surface expands.
>> > > > > > >> > > > > > > > > > > > >> >> >>
>> > > > > > >> > > > > > > > > > > > >> >> >> *Rough sketch of how to get
>> > there*:
>> > > > > > >> > > > > > > > > > > > >> >> >>  1.  Add read methods to
>> > > > > > >> PolarisMetricsManager
>> > > > > > >> > > > > > > > > > (listScanReports,
>> > > > > > >> > > > > > > > > > > > >> >> >> listCommitReports) with
>> default
>> > > no-op,
>> > > > > > >> same as
>> > > > > > >> > > the
>> > > > > > >> > > > > > > > existing
>> > > > > > >> > > > > > > > > > > write
>> > > > > > >> > > > > > > > > > > > >> >> methods.
>> > > > > > >> > > > > > > > > > > > >> >> >> (Probably make
>> > > PolarisMetricsManager
>> > > > > more
>> > > > > > >> > > explicit
>> > > > > > >> > > > > on
>> > > > > > >> > > > > > > > being
>> > > > > > >> > > > > > > > > > > > Iceberg
>> > > > > > >> > > > > > > > > > > > >> >> >> specific like package name or
>> > class
>> > > > > name
>> > > > > > >> etc.)
>> > > > > > >> > > > > > > > > > > > >> >> >>  2.  Wire
>> MetricsReportsService
>> > > through
>> > > > > > >> > > > > > MetaStoreManager
>> > > > > > >> > > > > > > > > > instead
>> > > > > > >> > > > > > > > > > > > of
>> > > > > > >> > > > > > > > > > > > >> >> >> callContext.getMetaStore().
>> > > > > > >> > > > > > > > > > > > >> >> >>  3.  Extract metrics
>> persistence
>> > > from
>> > > > > > >> > > > > > > > > JdbcBasePersistenceImpl
>> > > > > > >> > > > > > > > > > > into
>> > > > > > >> > > > > > > > > > > > >> its
>> > > > > > >> > > > > > > > > > > > >> >> >> own class. That file carries
>> ~7
>> > > > > > >> > > responsibilities,
>> > > > > > >> > > > > > > metrics
>> > > > > > >> > > > > > > > > > being
>> > > > > > >> > > > > > > > > > > > one
>> > > > > > >> > > > > > > > > > > > >> of
>> > > > > > >> > > > > > > > > > > > >> >> them.
>> > > > > > >> > > > > > > > > > > > >> >> >>  4.  Remove MetricsPersistence
>> > from
>> > > > > > >> > > > BasePersistence.
>> > > > > > >> > > > > > > > > > > > >> >> >>
>> > > > > > >> > > > > > > > > > > > >> >> >> *None of this needs to happen
>> in
>> > > #4115.
>> > > > > > >> But if
>> > > > > > >> > > the
>> > > > > > >> > > > > > > > direction
>> > > > > > >> > > > > > > > > > > makes
>> > > > > > >> > > > > > > > > > > > >> >> sense,
>> > > > > > >> > > > > > > > > > > > >> >> >> it would be good to align
>> before
>> > > the
>> > > > > > >> metrics
>> > > > > > >> > > > surface
>> > > > > > >> > > > > > > grows
>> > > > > > >> > > > > > > > > > > > further.
>> > > > > > >> > > > > > > > > > > > >> >> Curious
>> > > > > > >> > > > > > > > > > > > >> >> >> what others think.*
>> > > > > > >> > > > > > > > > > > > >> >> >>
>> > > > > > >> > > > > > > > > > > > >> >> >> *My mental model note*: Level
>> 1
>> > > > > > >> > > MetaStoreManager;
>> > > > > > >> > > > > > level
>> > > > > > >> > > > > > > 2
>> > > > > > >> > > > > > > > > > > > >> transactional
>> > > > > > >> > > > > > > > > > > > >> >> >> persistence; level 3 base
>> > > persistence
>> > > > > > >> > > > > > > > > > > > >> >> >>
>> > > > > > >> > > > > > > > > > > > >> >> >> Diagram 1
>> > > > > > >> > > > > > > > > > > > >> >> >> <
>> > > > > > >> > > > > > > > > > > > >> >>
>> > > > > > >> > > > > > > > > > > > >>
>> > > > > > >> > > > > > > > > > > >
>> > > > > > >> > > > > > > > > > >
>> > > > > > >> > > > > > > > > >
>> > > > > > >> > > > > > > > >
>> > > > > > >> > > > > > > >
>> > > > > > >> > > > > > >
>> > > > > > >> > > > > >
>> > > > > > >> > > > >
>> > > > > > >> > > >
>> > > > > > >> > >
>> > > > > > >> >
>> > > > > > >>
>> > > > > >
>> > > > >
>> > >
>> >
>> https://urldefense.com/v3/__https://www.plantuml.com/plantuml/uml/bLHDR-Cs4BthLmpIYupw0zbkKQ1r3M-S7Bp8xhhM7WCOb3IM65EaGD9EX2RzxHrHb4CxRelwa4YSDu_lpOVcnZ9jzvM8BBS2uGjQpJC3dtHMSekPtMk44IpsMgEqa5XcCOhCZikQQLP1pR8TAp2n3ILhmZDP20m0fcIvUkAoW2qJXd9z1bpToO9BX3WXu0ucy5rpgGPNm0nW5_epUWtm2Ue3pn3kMOFQmKntGZW0BYtgBSi8k5A2QMwybJNMIbFiGSR9QZc4nUqIvikStF0jHprua5C-amge42aNt3R0f5JaaoivdV2Pkqbx4hee4ymOkBh5BTiB-_uIeGeo8zL8rPsPl4DktdEiK1jkB1NdZCRbrSTecDe_mlHbF0wvBmCkaOH5_S8a_TTTKI6-nmCAkEw4LpxsZ-LbYLKQFKMNOgf_wuM7_bV9gOer5SYMMksBSWXFcbi49KNZXNLicwfe3TETC7gPdPqI7uBcHMb1RSzYq34c6PDUM9mn8HRsUTZEiDBve3NjVZumBj0U7SS37mGO7vcwtiK-_pU7U7L_f-digo9YbhSwIfMRwIITKGXbxdIUTCGF1SeCJxloKsU-3k9ddRbX1eDq1q_fx1JbBGT0glVyXimDuP4TQ5qpCAmnGEj2s_6n5mtn1z-97-63itFQZLPO1Ev2tu_WF7Ju-VPc0Skg5bYXxBhkY1xpD7EM_7fyflSpIsqMgVth5xhVr4eQxWQ8enaSAJQSG16yFSDuJ798rrcXr_3n-lfdk7icQjEBmFujL7AodiP_Y4Z7-YxvtZNs4zMgpNTl6tF8sglyPsmqchrjvQ-m-aP94r-TwCA2Ka8upPJZwtvSpoYCXkYMZU2NXvRMBfq9P3i3Le4VAZUAlUZ_oPKsxPgY0Q_BSKLkyr9bhQhQrJjo_x3TPlIB0DPjnMfcIoYP0QaYw1a0fTKDr8fB6ntNuvmoL1ZGkXa69Njh43zf9GiGxHQrA_jDYWRSzF5--WmTVrN97_Sm8LbLUy_lGBmLanJjFkDlGkRqjA_4tm00__;!!Iz9xO38YGHZK!4rnlVIqBQQQ6xpGe1PMV58edOS_cGrKoqdfhw4qSp_JQCavvEjaxXk0OwGPnxyEcO60AzKQK_XDtp7H3DdkFXcrXuTBISHg$
>> > > > > > >> > > > > > > > > > > > >> >> >
>> > > > > > >> > > > > > > > > > > > >> >> >> :
>> > > > > > >> > > > > > > > > > > > >> >> >>
>> > > > > > >> > > > > > > > > > > > >> >> >> [image: image.png]
>> > > > > > >> > > > > > > > > > > > >> >> >>
>> > > > > > >> > > > > > > > > > > > >> >> >> Diagram 2
>> > > > > > >> > > > > > > > > > > > >> >> >> <
>> > > > > > >> > > > > > > > > > > > >> >>
>> > > > > > >> > > > > > > > > > > > >>
>> > > > > > >> > > > > > > > > > > >
>> > > > > > >> > > > > > > > > > >
>> > > > > > >> > > > > > > > > >
>> > > > > > >> > > > > > > > >
>> > > > > > >> > > > > > > >
>> > > > > > >> > > > > > >
>> > > > > > >> > > > > >
>> > > > > > >> > > > >
>> > > > > > >> > > >
>> > > > > > >> > >
>> > > > > > >> >
>> > > > > > >>
>> > > > > >
>> > > > >
>> > >
>> >
>> https://urldefense.com/v3/__https://www.plantuml.com/plantuml/uml/VLLDR-8m4BtdLupO2sWBLVU8AaGB7AXAbssGzb896SS9RXqxjKqBwkv_tt7iV43fdaZYDpFlpRmnOsE9jhjSH9PRmM31hERKm8scMsuPjJlDe0yheZDc8RR4iYWoBrmMH9CS2a9VICPYUy1OZN0YCy5Q0BCbYNhdCeEK28En8G8wCvbnoQ0R8_05Bc6bkLIz3X03p1zzH7zR-9ZfDquPt9C3qoNCX2yV4G2NbkcKu5jdgGJHt0GbZwnG6i-UP3TUpk5gM6Ldqke350eZUqzoCft3U9xWHvxoa5-7K4nF1J46EbEMafsmdrCBbQ44gVggy18IZrn_ph5asd1ZiIKdQSgueZvjXrQFSFrdC3YN-nXmBacxbGiYyLVxLaBtdhqn0LSzdBDhqQtQoOJeGyad3z0lUqnYgpGB6Ns8oVyta00Dy_WnX0tIOZ8v6SYxHll1TrH6aejAik-mh-AphVFCwSUQqFypElag5QRGFDjQKEd96K1P8QP41c9TzA_IIQyvdAWyv_RSiS3skb0_EzDDkK2v5xWF6MiGFlvhpFLcD2Dq2pml14gaF67eQkmd8gulDoC4kSOu6KVpkvlUJg1RTbWISU40RdBUUS_9XfRZ2dwxm_SW8LYFISgm_MnlDQ6M9P1gbKEc4X-2pH_FvJCkCqm9pbVjD6LrwdLeOrDWfOaqc8Wh9BE85oNKxkNQ6o4yGRy_Eae0G_G8tZv81d3bHDB23WOdisohVr3nh_j6lbSjbNaLRTc8UgtPbAU1J_tygOfZX9DWEJeHDvYx-qmSi5FgNLPZwHrHcUsncGQ5-skhUclpE5fo4ounpFauYrUbkU6ccfnxMvitwag4IyerhTxj8In_Oj1bDO4pQru674loYrGlULHLEGCjwJJ8gDoVZR8MxO4BT3IzRvIcAQKezC6xpziGnTyImrfEGyJI_OcKfgtxIvnTqFEMS17L9Z-jsARN5FmTheP7HtSdtOMT0B4GY2FYHXxgQmMtj2bRqiLFGapiVe1_QVKDrkqXcm83aFEXnMYCZ-xlyHy__;!!Iz9xO38YGHZK!4rnlVIqBQQQ6xpGe1PMV58edOS_cGrKoqdfhw4qSp_JQCavvEjaxXk0OwGPnxyEcO60AzKQK_XDtp7H3DdkFXcrXSpmqcNk$
>> > > > > > >> > > > > > > > > > > > >> >> >
>> > > > > > >> > > > > > > > > > > > >> >> >> :
>> > > > > > >> > > > > > > > > > > > >> >> >> [image: image.png]
>> > > > > > >> > > > > > > > > > > > >> >> >>
>> > > > > > >> > > > > > > > > > > > >> >> >>  -ej
>> > > > > > >> > > > > > > > > > > > >> >> >>
>> > > > > > >> > > > > > > > > > > > >> >> >> On Wed, Apr 15, 2026 at
>> 8:22 AM
>> > > Dmitri
>> > > > > > >> > > > Bourlatchkov
>> > > > > > >> > > > > <
>> > > > > > >> > > > > > > > > > > > >> dimas@apache.org>
>> > > > > > >> > > > > > > > > > > > >> >> >> wrote:
>> > > > > > >> > > > > > > > > > > > >> >> >>
>> > > > > > >> > > > > > > > > > > > >> >> >>> Hi All,
>> > > > > > >> > > > > > > > > > > > >> >> >>>
>> > > > > > >> > > > > > > > > > > > >> >> >>> Heads up: The current state
>> of
>> > PR
>> > > > > [4115]
>> > > > > > >> > looks
>> > > > > > >> > > > > pretty
>> > > > > > >> > > > > > > > solid
>> > > > > > >> > > > > > > > > > to
>> > > > > > >> > > > > > > > > > > > me.
>> > > > > > >> > > > > > > > > > > > >> I
>> > > > > > >> > > > > > > > > > > > >> >> >>> believe this PR is
>> approaching a
>> > > > > > mergeable
>> > > > > > >> > > > > condition.
>> > > > > > >> > > > > > > > > > > > >> >> >>>
>> > > > > > >> > > > > > > > > > > > >> >> >>> Please post your reviews if
>> you
>> > > have
>> > > > > any
>> > > > > > >> > > > comments.
>> > > > > > >> > > > > > > > > > > > >> >> >>>
>> > > > > > >> > > > > > > > > > > > >> >> >>> [4115]
>> > > > > > >> > > >
>> >
>> https://urldefense.com/v3/__https://github.com/apache/polaris/pull/4115__;!!Iz9xO38YGHZK!4rnlVIqBQQQ6xpGe1PMV58edOS_cGrKoqdfhw4qSp_JQCavvEjaxXk0OwGPnxyEcO60AzKQK_XDtp7H3DdkFXcrXtFRscKs$
>> > > > > > >> > > > > > > > > > > > >> >> >>>
>> > > > > > >> > > > > > > > > > > > >> >> >>> Thanks,
>> > > > > > >> > > > > > > > > > > > >> >> >>> Dmitri.
>> > > > > > >> > > > > > > > > > > > >> >> >>>
>> > > > > > >> > > > > > > > > > > > >> >> >>> On Tue, Mar 3, 2026 at
>> 3:29 PM
>> > > Anand
>> > > > > > Kumar
>> > > > > > >> > > > Sankaran
>> > > > > > >> > > > > > via
>> > > > > > >> > > > > > > > > dev <
>> > > > > > >> > > > > > > > > > > > >> >> >>> dev@polaris.apache.org>
>> wrote:
>> > > > > > >> > > > > > > > > > > > >> >> >>>
>> > > > > > >> > > > > > > > > > > > >> >> >>> > Hi Yufei and Dmitri,
>> > > > > > >> > > > > > > > > > > > >> >> >>> >
>> > > > > > >> > > > > > > > > > > > >> >> >>> > Here is a proposal for the
>> > REST
>> > > > > > >> endpoints
>> > > > > > >> > for
>> > > > > > >> > > > > > metrics
>> > > > > > >> > > > > > > > and
>> > > > > > >> > > > > > > > > > > > events.
>> > > > > > >> > > > > > > > > > > > >> >> >>> >
>> > > > > > >> > > > > > > > > > > > >> >> >>> >
>> > > > > > >> > > > > >
>> >
>> https://urldefense.com/v3/__https://github.com/apache/polaris/pull/3924/changes__;!!Iz9xO38YGHZK!4rnlVIqBQQQ6xpGe1PMV58edOS_cGrKoqdfhw4qSp_JQCavvEjaxXk0OwGPnxyEcO60AzKQK_XDtp7H3DdkFXcrXcjoPhSs$
>> > > > > > >> > > > > > > > > > > > >> >> >>> >
>> > > > > > >> > > > > > > > > > > > >> >> >>> > I did not see any
>> precursors
>> > for
>> > > > > > >> raising a
>> > > > > > >> > PR
>> > > > > > >> > > > for
>> > > > > > >> > > > > > > > > > proposals,
>> > > > > > >> > > > > > > > > > > so
>> > > > > > >> > > > > > > > > > > > >> >> trying
>> > > > > > >> > > > > > > > > > > > >> >> >>> > this.  Please let me know
>> what
>> > > you
>> > > > > > >> think.
>> > > > > > >> > > > > > > > > > > > >> >> >>> >
>> > > > > > >> > > > > > > > > > > > >> >> >>> > -
>> > > > > > >> > > > > > > > > > > > >> >> >>> > Anand
>> > > > > > >> > > > > > > > > > > > >> >> >>> >
>> > > > > > >> > > > > > > > > > > > >> >> >>> > From: Anand Kumar Sankaran
>> <
>> > > > > > >> > > > > > > anand.sankaran@workday.com
>> > > > > > >> > > > > > > > >
>> > > > > > >> > > > > > > > > > > > >> >> >>> > Date: Monday, March 2,
>> 2026 at
>> > > > > > 10:25 AM
>> > > > > > >> > > > > > > > > > > > >> >> >>> > To: dev@polaris.apache.org
>> <
>> > > > > > >> > > > > dev@polaris.apache.org
>> > > > > > >> > > > > > >
>> > > > > > >> > > > > > > > > > > > >> >> >>> > Subject: Re: Polaris
>> Telemetry
>> > > and
>> > > > > > Audit
>> > > > > > >> > > Trail
>> > > > > > >> > > > > > > > > > > > >> >> >>> >
>> > > > > > >> > > > > > > > > > > > >> >> >>> > About the REST API, based
>> on
>> > my
>> > > use
>> > > > > > >> cases:
>> > > > > > >> > > > > > > > > > > > >> >> >>> >
>> > > > > > >> > > > > > > > > > > > >> >> >>> >
>> > > > > > >> > > > > > > > > > > > >> >> >>> >   1.
>> > > > > > >> > > > > > > > > > > > >> >> >>> > I want to be able to query
>> > > commit
>> > > > > > >> metrics
>> > > > > > >> > to
>> > > > > > >> > > > > track
>> > > > > > >> > > > > > > > files
>> > > > > > >> > > > > > > > > > > added
>> > > > > > >> > > > > > > > > > > > /
>> > > > > > >> > > > > > > > > > > > >> >> >>> removed
>> > > > > > >> > > > > > > > > > > > >> >> >>> > per commit, along with
>> record
>> > > > > counts.
>> > > > > > >> The
>> > > > > > >> > > > > ingestion
>> > > > > > >> > > > > > > > > > pipeline
>> > > > > > >> > > > > > > > > > > > that
>> > > > > > >> > > > > > > > > > > > >> >> >>> writes
>> > > > > > >> > > > > > > > > > > > >> >> >>> > this data is owned by us
>> and
>> > we
>> > > are
>> > > > > > >> > > guaranteed
>> > > > > > >> > > > to
>> > > > > > >> > > > > > > write
>> > > > > > >> > > > > > > > > > this
>> > > > > > >> > > > > > > > > > > > >> >> >>> information
>> > > > > > >> > > > > > > > > > > > >> >> >>> > for each write.
>> > > > > > >> > > > > > > > > > > > >> >> >>> >   2.
>> > > > > > >> > > > > > > > > > > > >> >> >>> > I want to be able to query
>> > scan
>> > > > > > metrics
>> > > > > > >> for
>> > > > > > >> > > > > read. I
>> > > > > > >> > > > > > > > > > > understand
>> > > > > > >> > > > > > > > > > > > >> >> clients
>> > > > > > >> > > > > > > > > > > > >> >> >>> do
>> > > > > > >> > > > > > > > > > > > >> >> >>> > not fulfill this
>> requirement.
>> > > > > > >> > > > > > > > > > > > >> >> >>> >   3.
>> > > > > > >> > > > > > > > > > > > >> >> >>> > I want to be able to query
>> the
>> > > > > events
>> > > > > > >> table
>> > > > > > >> > > > > (events
>> > > > > > >> > > > > > > are
>> > > > > > >> > > > > > > > > > > > >> persisted) -
>> > > > > > >> > > > > > > > > > > > >> >> >>> this
>> > > > > > >> > > > > > > > > > > > >> >> >>> > may supersede #2, I am not
>> > sure
>> > > yet.
>> > > > > > >> > > > > > > > > > > > >> >> >>> >
>> > > > > > >> > > > > > > > > > > > >> >> >>> > All this information is in
>> the
>> > > JDBC
>> > > > > > >> based
>> > > > > > >> > > > > > persistence
>> > > > > > >> > > > > > > > > model
>> > > > > > >> > > > > > > > > > > and
>> > > > > > >> > > > > > > > > > > > >> is
>> > > > > > >> > > > > > > > > > > > >> >> >>> > persisted in the
>> metastore. I
>> > > > > > currently
>> > > > > > >> > don’t
>> > > > > > >> > > > > have
>> > > > > > >> > > > > > a
>> > > > > > >> > > > > > > > need
>> > > > > > >> > > > > > > > > > to
>> > > > > > >> > > > > > > > > > > > >> query
>> > > > > > >> > > > > > > > > > > > >> >> >>> > prometheus or open
>> telemetry.
>> > I
>> > > do
>> > > > > > >> publish
>> > > > > > >> > > some
>> > > > > > >> > > > > > > events
>> > > > > > >> > > > > > > > to
>> > > > > > >> > > > > > > > > > > > >> Prometheus
>> > > > > > >> > > > > > > > > > > > >> >> >>> and
>> > > > > > >> > > > > > > > > > > > >> >> >>> > they are forwarded to our
>> > > dashboards
>> > > > > > >> > > elsewhere.
>> > > > > > >> > > > > > > > > > > > >> >> >>> >
>> > > > > > >> > > > > > > > > > > > >> >> >>> > About the CLI utilities, I
>> > > meant the
>> > > > > > >> admin
>> > > > > > >> > > user
>> > > > > > >> > > > > > > > > utilities.
>> > > > > > >> > > > > > > > > > In
>> > > > > > >> > > > > > > > > > > > >> one of
>> > > > > > >> > > > > > > > > > > > >> >> >>> the
>> > > > > > >> > > > > > > > > > > > >> >> >>> > earliest drafts of my
>> > proposal,
>> > > > > > Prashant
>> > > > > > >> > > > > mentioned
>> > > > > > >> > > > > > > that
>> > > > > > >> > > > > > > > > the
>> > > > > > >> > > > > > > > > > > > >> metrics
>> > > > > > >> > > > > > > > > > > > >> >> >>> tables
>> > > > > > >> > > > > > > > > > > > >> >> >>> > can grow indefinitely and
>> > that a
>> > > > > > similar
>> > > > > > >> > > > problem
>> > > > > > >> > > > > > > exists
>> > > > > > >> > > > > > > > > > with
>> > > > > > >> > > > > > > > > > > > the
>> > > > > > >> > > > > > > > > > > > >> >> events
>> > > > > > >> > > > > > > > > > > > >> >> >>> > table as well. We discussed
>> > that
>> > > > > > >> cleaning
>> > > > > > >> > up
>> > > > > > >> > > of
>> > > > > > >> > > > > old
>> > > > > > >> > > > > > > > > records
>> > > > > > >> > > > > > > > > > > > from
>> > > > > > >> > > > > > > > > > > > >> >> both
>> > > > > > >> > > > > > > > > > > > >> >> >>> > metrics tables and events
>> > > tables can
>> > > > > > be
>> > > > > > >> > done
>> > > > > > >> > > > via
>> > > > > > >> > > > > a
>> > > > > > >> > > > > > > CLI
>> > > > > > >> > > > > > > > > > > utility.
>> > > > > > >> > > > > > > > > > > > >> >> >>> >
>> > > > > > >> > > > > > > > > > > > >> >> >>> > I see that Yufei has
>> covered
>> > the
>> > > > > > >> discussion
>> > > > > > >> > > > about
>> > > > > > >> > > > > > > > > > > datasources.
>> > > > > > >> > > > > > > > > > > > >> >> >>> >
>> > > > > > >> > > > > > > > > > > > >> >> >>> > -
>> > > > > > >> > > > > > > > > > > > >> >> >>> > Anand
>> > > > > > >> > > > > > > > > > > > >> >> >>> >
>> > > > > > >> > > > > > > > > > > > >> >> >>> >
>> > > > > > >> > > > > > > > > > > > >> >> >>> >
>> > > > > > >> > > > > > > > > > > > >> >> >>> > From: Yufei Gu <
>> > > > > flyrain000@gmail.com>
>> > > > > > >> > > > > > > > > > > > >> >> >>> > Date: Friday, February 27,
>> > 2026
>> > > at
>> > > > > > >> 9:54 PM
>> > > > > > >> > > > > > > > > > > > >> >> >>> > To: dev@polaris.apache.org
>> <
>> > > > > > >> > > > > dev@polaris.apache.org
>> > > > > > >> > > > > > >
>> > > > > > >> > > > > > > > > > > > >> >> >>> > Subject: Re: Polaris
>> Telemetry
>> > > and
>> > > > > > Audit
>> > > > > > >> > > Trail
>> > > > > > >> > > > > > > > > > > > >> >> >>> >
>> > > > > > >> > > > > > > > > > > > >> >> >>> > This Message Is From an
>> > External
>> > > > > > Sender
>> > > > > > >> > > > > > > > > > > > >> >> >>> > This message came from
>> outside
>> > > your
>> > > > > > >> > > > organization.
>> > > > > > >> > > > > > > > > > > > >> >> >>> > Report Suspicious<
>> > > > > > >> > > > > > > > > > > > >> >> >>> >
>> > > > > > >> > > > > > > > > > > > >> >> >>>
>> > > > > > >> > > > > > > > > > > > >> >>
>> > > > > > >> > > > > > > > > > > > >>
>> > > > > > >> > > > > > > > > > > >
>> > > > > > >> > > > > > > > > > >
>> > > > > > >> > > > > > > > > >
>> > > > > > >> > > > > > > > >
>> > > > > > >> > > > > > > >
>> > > > > > >> > > > > > >
>> > > > > > >> > > > > >
>> > > > > > >> > > > >
>> > > > > > >> > > >
>> > > > > > >> > >
>> > > > > > >> >
>> > > > > > >>
>> > > > > >
>> > > > >
>> > >
>> >
>> https://us-phishalarm-ewt.proofpoint.com/EWT/v1/Iz9xO38YGHZK!YhNDZABkHi1B699ote2uMwpOZw8i0QMCGO2Szc-HshuABGhGvwPJcymE6G2oUUxtS8xDkSrtGTPm_I3QnVDHoLMk50m9v8z_nZKTkd-bnVUbreF1u0WnfV_X5eYevZl_$
>> > > > > > >> > > > > > > > > > > > >> >> >>> > >
>> > > > > > >> > > > > > > > > > > > >> >> >>> >
>> > > > > > >> > > > > > > > > > > > >> >> >>> >
>> > > > > > >> > > > > > > > > > > > >> >> >>> > As I mentioned in
>> > > > > > >> > > > > > > > > > > > >> >> >>> >
>> > > > > > >> > > > > > > > > > > > >> >> >>>
>> > > > > > >> > > > > > > > > > > > >> >>
>> > > > > > >> > > > > > > > > > > > >>
>> > > > > > >> > > > > > > > > > > >
>> > > > > > >> > > > > > > > > > >
>> > > > > > >> > > > > > > > > >
>> > > > > > >> > > > > > > > >
>> > > > > > >> > > > > > > >
>> > > > > > >> > > > > > >
>> > > > > > >> > > > > >
>> > > > > > >> > > > >
>> > > > > > >> > > >
>> > > > > > >> > >
>> > > > > > >> >
>> > > > > > >>
>> > > > > >
>> > > > >
>> > >
>> >
>> https://urldefense.com/v3/__https://github.com/apache/polaris/issues/3890__;!!Iz9xO38YGHZK!5EuyFFkk3vhRWVIRvQAWBSQfpJkTMA9HxugzDwXmN0LPPqhEFxYkFRGVhtb8AqUwXtDh2OplcMnbMDHKOxrvDU0$
>> > >> > > >> > > > > > > > > > > > >> >> >>> ,
>> > > > > > >> > > > > > > > > > > > >> >> >>> > supporting
>> > > > > > >> > > > > > > > > > > > >> >> >>> > multiple data sources is
>> not a
>> > > > > trivial
>> > > > > > >> > > change.
>> > > > > > >> > > > I
>> > > > > > >> > > > > > > would
>> > > > > > >> > > > > > > > > > > strongly
>> > > > > > >> > > > > > > > > > > > >> >> >>> recommend
>> > > > > > >> > > > > > > > > > > > >> >> >>> > starting with a design
>> > document
>> > > to
>> > > > > > >> > carefully
>> > > > > > >> > > > > > evaluate
>> > > > > > >> > > > > > > > the
>> > > > > > >> > > > > > > > > > > > >> >> architectural
>> > > > > > >> > > > > > > > > > > > >> >> >>> > implications and long term
>> > > impact.
>> > > > > > >> > > > > > > > > > > > >> >> >>> >
>> > > > > > >> > > > > > > > > > > > >> >> >>> > A REST endpoint to query
>> > metrics
>> > > > > seems
>> > > > > > >> > > > reasonable
>> > > > > > >> > > > > > > given
>> > > > > > >> > > > > > > > > the
>> > > > > > >> > > > > > > > > > > > >> current
>> > > > > > >> > > > > > > > > > > > >> >> >>> JDBC
>> > > > > > >> > > > > > > > > > > > >> >> >>> > based persistence model.
>> That
>> > > said,
>> > > > > we
>> > > > > > >> may
>> > > > > > >> > > also
>> > > > > > >> > > > > > > > consider
>> > > > > > >> > > > > > > > > > > > >> alternative
>> > > > > > >> > > > > > > > > > > > >> >> >>> > storage models. For
>> example,
>> > if
>> > > we
>> > > > > > later
>> > > > > > >> > > adopt
>> > > > > > >> > > > a
>> > > > > > >> > > > > > time
>> > > > > > >> > > > > > > > > > series
>> > > > > > >> > > > > > > > > > > > >> system
>> > > > > > >> > > > > > > > > > > > >> >> >>> such as
>> > > > > > >> > > > > > > > > > > > >> >> >>> > Prometheus to store
>> metrics,
>> > the
>> > > > > query
>> > > > > > >> > model
>> > > > > > >> > > > and
>> > > > > > >> > > > > > > access
>> > > > > > >> > > > > > > > > > > > patterns
>> > > > > > >> > > > > > > > > > > > >> >> would
>> > > > > > >> > > > > > > > > > > > >> >> >>> be
>> > > > > > >> > > > > > > > > > > > >> >> >>> > fundamentally different.
>> > > Designing
>> > > > > the
>> > > > > > >> REST
>> > > > > > >> > > API
>> > > > > > >> > > > > > > without
>> > > > > > >> > > > > > > > > > > > >> considering
>> > > > > > >> > > > > > > > > > > > >> >> >>> these
>> > > > > > >> > > > > > > > > > > > >> >> >>> > potential evolutions may
>> limit
>> > > > > > >> flexibility.
>> > > > > > >> > > I'd
>> > > > > > >> > > > > > > suggest
>> > > > > > >> > > > > > > > > to
>> > > > > > >> > > > > > > > > > > > start
>> > > > > > >> > > > > > > > > > > > >> >> with
>> > > > > > >> > > > > > > > > > > > >> >> >>> the
>> > > > > > >> > > > > > > > > > > > >> >> >>> > use case.
>> > > > > > >> > > > > > > > > > > > >> >> >>> >
>> > > > > > >> > > > > > > > > > > > >> >> >>> > Yufei
>> > > > > > >> > > > > > > > > > > > >> >> >>> >
>> > > > > > >> > > > > > > > > > > > >> >> >>> >
>> > > > > > >> > > > > > > > > > > > >> >> >>> > On Fri, Feb 27, 2026 at
>> > 3:42 PM
>> > > > > Dmitri
>> > > > > > >> > > > > > Bourlatchkov <
>> > > > > > >> > > > > > > > > > > > >> >> dimas@apache.org>
>> > > > > > >> > > > > > > > > > > > >> >> >>> > wrote:
>> > > > > > >> > > > > > > > > > > > >> >> >>> >
>> > > > > > >> > > > > > > > > > > > >> >> >>> > > Hi Anand,
>> > > > > > >> > > > > > > > > > > > >> >> >>> > >
>> > > > > > >> > > > > > > > > > > > >> >> >>> > > Sharing my view...
>> subject
>> > to
>> > > > > > >> discussion:
>> > > > > > >> > > > > > > > > > > > >> >> >>> > >
>> > > > > > >> > > > > > > > > > > > >> >> >>> > > 1. Adding non-IRC REST
>> API
>> > to
>> > > > > > Polaris
>> > > > > > >> is
>> > > > > > >> > > > > > perfectly
>> > > > > > >> > > > > > > > > fine.
>> > > > > > >> > > > > > > > > > > > >> >> >>> > >
>> > > > > > >> > > > > > > > > > > > >> >> >>> > > Figuring out specific
>> > endpoint
>> > > > > URIs
>> > > > > > >> and
>> > > > > > >> > > > > payloads
>> > > > > > >> > > > > > > > might
>> > > > > > >> > > > > > > > > > > > require
>> > > > > > >> > > > > > > > > > > > >> a
>> > > > > > >> > > > > > > > > > > > >> >> few
>> > > > > > >> > > > > > > > > > > > >> >> >>> > > roundtrips, so opening a
>> > > separate
>> > > > > > >> thread
>> > > > > > >> > > for
>> > > > > > >> > > > > that
>> > > > > > >> > > > > > > > might
>> > > > > > >> > > > > > > > > > be
>> > > > > > >> > > > > > > > > > > > >> best.
>> > > > > > >> > > > > > > > > > > > >> >> >>> > > Contributors commonly
>> create
>> > > > > Google
>> > > > > > >> Docs
>> > > > > > >> > > for
>> > > > > > >> > > > > new
>> > > > > > >> > > > > > > API
>> > > > > > >> > > > > > > > > > > > proposals
>> > > > > > >> > > > > > > > > > > > >> too
>> > > > > > >> > > > > > > > > > > > >> >> >>> (they
>> > > > > > >> > > > > > > > > > > > >> >> >>> > > fairly easy to update as
>> the
>> > > email
>> > > > > > >> > > discussion
>> > > > > > >> > > > > > > > > > progresses).
>> > > > > > >> > > > > > > > > > > > >> >> >>> > >
>> > > > > > >> > > > > > > > > > > > >> >> >>> > > There was a suggestion to
>> > try
>> > > > > > Markdown
>> > > > > > >> > > (with
>> > > > > > >> > > > > PRs)
>> > > > > > >> > > > > > > for
>> > > > > > >> > > > > > > > > > > > proposals
>> > > > > > >> > > > > > > > > > > > >> >> [1]
>> > > > > > >> > > > > > > > > > > > >> >> >>> ...
>> > > > > > >> > > > > > > > > > > > >> >> >>> > > feel free to give it a
>> try
>> > if
>> > > you
>> > > > > > are
>> > > > > > >> > > > > comfortable
>> > > > > > >> > > > > > > > with
>> > > > > > >> > > > > > > > > > > that.
>> > > > > > >> > > > > > > > > > > > >> >> >>> > >
>> > > > > > >> > > > > > > > > > > > >> >> >>> > > 2. Could you clarify
>> whether
>> > > you
>> > > > > > mean
>> > > > > > >> end
>> > > > > > >> > > > user
>> > > > > > >> > > > > > > > > utilities
>> > > > > > >> > > > > > > > > > or
>> > > > > > >> > > > > > > > > > > > >> admin
>> > > > > > >> > > > > > > > > > > > >> >> >>> user
>> > > > > > >> > > > > > > > > > > > >> >> >>> > > utilities? In the latter
>> > case
>> > > > > those
>> > > > > > >> might
>> > > > > > >> > > be
>> > > > > > >> > > > > more
>> > > > > > >> > > > > > > > > > suitable
>> > > > > > >> > > > > > > > > > > > for
>> > > > > > >> > > > > > > > > > > > >> the
>> > > > > > >> > > > > > > > > > > > >> >> >>> Admin
>> > > > > > >> > > > > > > > > > > > >> >> >>> > > CLI (java) not the Python
>> > CLI,
>> > > > > IMHO.
>> > > > > > >> > > > > > > > > > > > >> >> >>> > >
>> > > > > > >> > > > > > > > > > > > >> >> >>> > > Why would these
>> utilities be
>> > > > > common
>> > > > > > >> with
>> > > > > > >> > > > > events?
>> > > > > > >> > > > > > > > IMHO,
>> > > > > > >> > > > > > > > > > > event
>> > > > > > >> > > > > > > > > > > > >> use
>> > > > > > >> > > > > > > > > > > > >> >> >>> cases
>> > > > > > >> > > > > > > > > > > > >> >> >>> > are
>> > > > > > >> > > > > > > > > > > > >> >> >>> > > distinct from scan/commit
>> > > metrics.
>> > > > > > >> > > > > > > > > > > > >> >> >>> > >
>> > > > > > >> > > > > > > > > > > > >> >> >>> > > 3. I'd prefer separating
>> > > metrics
>> > > > > > >> > > persistence
>> > > > > > >> > > > > from
>> > > > > > >> > > > > > > > > > MetaStore
>> > > > > > >> > > > > > > > > > > > >> >> >>> persistence
>> > > > > > >> > > > > > > > > > > > >> >> >>> > at
>> > > > > > >> > > > > > > > > > > > >> >> >>> > > the code level, so that
>> they
>> > > could
>> > > > > > be
>> > > > > > >> > mixed
>> > > > > > >> > > > and
>> > > > > > >> > > > > > > > matched
>> > > > > > >> > > > > > > > > > > > >> >> >>> independently.
>> > > > > > >> > > > > > > > > > > > >> >> >>> > The
>> > > > > > >> > > > > > > > > > > > >> >> >>> > > separate datasource
>> question
>> > > will
>> > > > > > >> become
>> > > > > > >> > a
>> > > > > > >> > > > > > > non-issue
>> > > > > > >> > > > > > > > > with
>> > > > > > >> > > > > > > > > > > > that
>> > > > > > >> > > > > > > > > > > > >> >> >>> approach,
>> > > > > > >> > > > > > > > > > > > >> >> >>> > I
>> > > > > > >> > > > > > > > > > > > >> >> >>> > > guess.
>> > > > > > >> > > > > > > > > > > > >> >> >>> > >
>> > > > > > >> > > > > > > > > > > > >> >> >>> > > The rationale for
>> separating
>> > > scan
>> > > > > > >> metrics
>> > > > > > >> > > and
>> > > > > > >> > > > > > > > metastore
>> > > > > > >> > > > > > > > > > > > >> >> persistence
>> > > > > > >> > > > > > > > > > > > >> >> >>> is
>> > > > > > >> > > > > > > > > > > > >> >> >>> > that
>> > > > > > >> > > > > > > > > > > > >> >> >>> > > "cascading deletes"
>> between
>> > > them
>> > > > > are
>> > > > > > >> > hardly
>> > > > > > >> > > > > ever
>> > > > > > >> > > > > > > > > > required.
>> > > > > > >> > > > > > > > > > > > >> >> >>> Furthermore,
>> > > > > > >> > > > > > > > > > > > >> >> >>> > the
>> > > > > > >> > > > > > > > > > > > >> >> >>> > > data and query patterns
>> are
>> > > very
>> > > > > > >> > different
>> > > > > > >> > > so
>> > > > > > >> > > > > > > > different
>> > > > > > >> > > > > > > > > > > > >> >> technologies
>> > > > > > >> > > > > > > > > > > > >> >> >>> > might
>> > > > > > >> > > > > > > > > > > > >> >> >>> > > be beneficial in each
>> case.
>> > > > > > >> > > > > > > > > > > > >> >> >>> > >
>> > > > > > >> > > > > > > > > > > > >> >> >>> > > [1]
>> > > > > > >> > > > > > > > > > > > >> >> >>> >
>> > > > > > >> > > > > > > > > > > > >> >> >>>
>> > > > > > >> > > > > > > > > > > > >> >>
>> > > > > > >> > > > > > > > > > > > >>
>> > > > > > >> > > > > > > > > > > >
>> > > > > > >> > > > > > > > > > >
>> > > > > > >> > > > > > > > > >
>> > > > > > >> > > > > > > > >
>> > > > > > >> > > > > > > >
>> > > > > > >> > > > > > >
>> > > > > > >> > > > > >
>> > > > > > >> > > > >
>> > > > > > >> > > >
>> > > > > > >> > >
>> > > > > > >> >
>> > > > > > >>
>> > > > > >
>> > > > >
>> > >
>> >
>> https://urldefense.com/v3/__https://lists.apache.org/thread/yto2wp982t43h1mqjwnslswhws5z47cy__;!!Iz9xO38YGHZK!5EuyFFkk3vhRWVIRvQAWBSQfpJkTMA9HxugzDwXmN0LPPqhEFxYkFRGVhtb8AqUwXtDh2OplcMnbMDHKxYDakNU$
>> > >> > > >> > > > > > > > > > > > >> >> >>> > >
>> > > > > > >> > > > > > > > > > > > >> >> >>> > > Cheers,
>> > > > > > >> > > > > > > > > > > > >> >> >>> > > Dmitri.
>> > > > > > >> > > > > > > > > > > > >> >> >>> > >
>> > > > > > >> > > > > > > > > > > > >> >> >>> > > On Fri, Feb 27, 2026 at
>> > > 6:19 PM
>> > > > > > Anand
>> > > > > > >> > Kumar
>> > > > > > >> > > > > > > Sankaran
>> > > > > > >> > > > > > > > > via
>> > > > > > >> > > > > > > > > > > dev
>> > > > > > >> > > > > > > > > > > > <
>> > > > > > >> > > > > > > > > > > > >> >> >>> > > dev@polaris.apache.org>
>> > > wrote:
>> > > > > > >> > > > > > > > > > > > >> >> >>> > >
>> > > > > > >> > > > > > > > > > > > >> >> >>> > > > Thanks all. This PR is
>> > > merged
>> > > > > now.
>> > > > > > >> > > > > > > > > > > > >> >> >>> > > >
>> > > > > > >> > > > > > > > > > > > >> >> >>> > > > Here are the follow-up
>> > > features
>> > > > > /
>> > > > > > >> work
>> > > > > > >> > > > > needed.
>> > > > > > >> > > > > > > > These
>> > > > > > >> > > > > > > > > > > were
>> > > > > > >> > > > > > > > > > > > >> all
>> > > > > > >> > > > > > > > > > > > >> >> >>> part of
>> > > > > > >> > > > > > > > > > > > >> >> >>> > > the
>> > > > > > >> > > > > > > > > > > > >> >> >>> > > > merged PR at some
>> point in
>> > > time
>> > > > > > and
>> > > > > > >> > were
>> > > > > > >> > > > > > removed
>> > > > > > >> > > > > > > to
>> > > > > > >> > > > > > > > > > > reduce
>> > > > > > >> > > > > > > > > > > > >> >> scope.
>> > > > > > >> > > > > > > > > > > > >> >> >>> > > >
>> > > > > > >> > > > > > > > > > > > >> >> >>> > > > Please let me know what
>> > you
>> > > > > think.
>> > > > > > >> > > > > > > > > > > > >> >> >>> > > >
>> > > > > > >> > > > > > > > > > > > >> >> >>> > > >
>> > > > > > >> > > > > > > > > > > > >> >> >>> > > >   1.  A REST API to
>> > paginate
>> > > > > > through
>> > > > > > >> > > table
>> > > > > > >> > > > > > > metrics.
>> > > > > > >> > > > > > > > > > This
>> > > > > > >> > > > > > > > > > > > >> will be
>> > > > > > >> > > > > > > > > > > > >> >> >>> > non-IRC
>> > > > > > >> > > > > > > > > > > > >> >> >>> > > > standard addition.
>> > > > > > >> > > > > > > > > > > > >> >> >>> > > >   2.  Utilities for
>> > > managing old
>> > > > > > >> > records,
>> > > > > > >> > > > > > should
>> > > > > > >> > > > > > > be
>> > > > > > >> > > > > > > > > > > common
>> > > > > > >> > > > > > > > > > > > >> with
>> > > > > > >> > > > > > > > > > > > >> >> >>> events.
>> > > > > > >> > > > > > > > > > > > >> >> >>> > > > There was some
>> discussion
>> > > that
>> > > > > it
>> > > > > > >> > belongs
>> > > > > > >> > > > to
>> > > > > > >> > > > > > the
>> > > > > > >> > > > > > > > CLI.
>> > > > > > >> > > > > > > > > > > > >> >> >>> > > >   3.  Separate
>> datasource
>> > > > > > (metrics,
>> > > > > > >> > > events,
>> > > > > > >> > > > > > even
>> > > > > > >> > > > > > > > > other
>> > > > > > >> > > > > > > > > > > > >> tables?).
>> > > > > > >> > > > > > > > > > > > >> >> >>> > > >
>> > > > > > >> > > > > > > > > > > > >> >> >>> > > >
>> > > > > > >> > > > > > > > > > > > >> >> >>> > > > Anything else?
>> > > > > > >> > > > > > > > > > > > >> >> >>> > > >
>> > > > > > >> > > > > > > > > > > > >> >> >>> > > > -
>> > > > > > >> > > > > > > > > > > > >> >> >>> > > > Anand
>> > > > > > >> > > > > > > > > > > > >> >> >>> > > >
>> > > > > > >> > > > > > > > > > > > >> >> >>> > > >
>> > > > > > >> > > > > > > > > > > > >> >> >>> > >
>> > > > > > >> > > > > > > > > > > > >> >> >>> >
>> > > > > > >> > > > > > > > > > > > >> >> >>> >
>> > > > > > >> > > > > > > > > > > > >> >> >>>
>> > > > > > >> > > > > > > > > > > > >> >> >>
>> > > > > > >> > > > > > > > > > > > >> >>
>> > > > > > >> > > > > > > > > > > > >> >
>> > > > > > >> > > > > > > > > > > > >>
>> > > > > > >> > > > > > > > > > > > >
>> > > > > > >> > > > > > > > > > > >
>> > > > > > >> > > > > > > > > > >
>> > > > > > >> > > > > > > > > >
>> > > > > > >> > > > > > > > >
>> > > > > > >> > > > > > > >
>> > > > > > >> > > > > > >
>> > > > > > >> > > > > >
>> > > > > > >> > > > >
>> > > > > > >> > > >
>> > > > > > >> > >
>> > > > > > >> >
>> > > > > > >>
>> > > > > > >
>> > > > > >
>> > > > >
>> > >
>> >
>> >
>>
>

Re: Proposal for REST endpoints for table metrics and events

[Jean-Baptiste Onofré <jb...@nanthrax.net>]

Great, thanks for the update!

I will take a look.

Regards
JB

On Thu, Oct 8, 2026 at 2:37 AM Dmitri Bourlatchkov <di...@apache.org> wrote:

> Hi All,
>
> Metrics grants have been merged as a separate PR.
>
> Now [4115] is ready for one more review round. IMHO, it is ready to merge.
>
> [4115] https://github.com/apache/polaris/pull/4115
>
> Cheers,
> Dmitri.
>
> On Fri, Oct 2, 2026 at 1:01 AM Jean-Baptiste Onofré <jb...@nanthrax.net>
> wrote:
>
> > Hi all,
> >
> > I agree with Yufei: I would keep TABLE_READ_METRICS as a separate,
> > explicit grant, not implied by TABLE_FULL_METADATA.
> >
> > I believe metrics reports are closer to an activity log than to table
> > metadata. I also think it's consistent with what we have today, as
> > reporting scan and commit metrics is already with TABLE_READ_DATA and
> > TABLE_WRITE_DATA, not by "metadata" privilege.
> >
> > For the troubleshooting use case, the cost is just one additional
> > grant, I think it's acceptable :) I would still let
> > CATALOG_MANAGE_CONTENT implies it, as that privilege already implies
> > TABLE_READ_DATA.
> >
> > Concretely for PR #5669, I propose:
> > - to remove the "Implied by TABLE_FULL_METADATA" statement from the
> > javadoc and CHANGELOG
> > - to add SUPER_PRIVILEGES entry in the PR, so the grant change is
> > self-contained
> > - to drop ICEBERG_TABLE restriction.
> > - to update access-control.md documenting TABLE_READ_METRICS, etc
> >
> > I will comment directly in the PR.
> >
> > Regards
> > JB
> >
> > On Fri, Oct 2, 2026 at 3:40 AM Yufei Gu <fl...@gmail.com> wrote:
> > >
> > > Thanks Dmitri for the reminder.
> > >
> > > PR #5659 <https://github.com/apache/polaris/pull/5659> adds
> > > TABLE_READ_METRICS and says it is implied by TABLE_FULL_METADATA.
> Should
> > > that be the default?
> > >
> > > I’m leaning toward keeping it separate. Metrics describe workloads
> (Spark
> > > ETL, Flink streaming, etc) using a table and can include query filters,
> > so
> > > someone managing table metadata may not need access to other users’
> query
> > > activity.
> > >
> > > Including it could make troubleshooting easier for table admins,
> though.
> > > Are there use cases that favor automatic access over an explicit grant?
> > > Yufei
> > >
> > >
> > > On Wed, Sep 30, 2026 at 1:57 PM Dmitri Bourlatchkov <di...@apache.org>
> > > wrote:
> > >
> > > > Hi Yufei,
> > > >
> > > > Here's the separate PR for Metrics-related grants:
> > > > https://github.com/apache/polaris/pull/5659
> > > >
> > > > Please review it to unblock this contribution.
> > > >
> > > > Thanks,
> > > > Dmitri.
> > > >
> > > > On Wed, Sep 30, 2026 at 1:05 PM Yufei Gu <fl...@gmail.com>
> wrote:
> > > >
> > > > > From the previous email, we agreed to split the REST change from
> the
> > > > grant
> > > > > change. I think the PR is ready to go if we only keep the grant
> > changes.
> > > > >
> > > > > Yufei
> > > > >
> > > > >
> > > > > On Tue, Sep 29, 2026 at 9:26 PM Jean-Baptiste Onofré <
> > jb@nanthrax.net>
> > > > > wrote:
> > > > >
> > > > > > Hi guys,
> > > > > >
> > > > > > I agree with Dmitri: +1 to merge PR #4115 on Oct 1 (tomorrow).
> > > > > >
> > > > > > From my side, all outstanding feedback has been addressed,
> > including
> > > > > > the final documentation points raised by EJ regarding the
> empty-200
> > > > > > NoOp behavior and reconstructed payload description. The branch
> is
> > > > > > clean, up to date with main, and CI is green.
> > > > > >
> > > > > > Once this lands, we can follow up with PR #4756 for the JDBC
> query
> > SPI
> > > > > > implementation.
> > > > > >
> > > > > > Regards
> > > > > > JB
> > > > > >
> > > > > > On Wed, Sep 30, 2026 at 1:53 AM Dmitri Bourlatchkov <
> > dimas@apache.org>
> > > > > > wrote:
> > > > > > >
> > > > > > > Hi All,
> > > > > > >
> > > > > > > Any remaining blockers on [4115]?
> > > > > > >
> > > > > > > Given that the review was very long, please restate any
> remaining
> > > > > > concerns
> > > > > > > (here or in GH).
> > > > > > >
> > > > > > > Otherwise, I propose merging [4115] on Oct 1.
> > > > > > >
> > > > > > > [4115] https://github.com/apache/polaris/pull/4115
> > > > > > >
> > > > > > > Thanks,
> > > > > > > Dmitri.
> > > > > > >
> > > > > > > On Wed, Sep 23, 2026 at 7:53 PM Dmitri Bourlatchkov <
> > > > dimas@apache.org>
> > > > > > > wrote:
> > > > > > >
> > > > > > > > Hi EJ,
> > > > > > > >
> > > > > > > > Thanks for the meeting recap!
> > > > > > > >
> > > > > > > > All:
> > > > > > > >
> > > > > > > > Worth noting: PR [4115] also contains Management API changes
> > that
> > > > > were
> > > > > > not
> > > > > > > > anticipated initially and did not get an API change review
> > until
> > > > now.
> > > > > > These
> > > > > > > > are small changes to add new grant types for metrics RBAC..
> > > > > > > >
> > > > > > > > I've reviewed and approved PR [4115] in GH.
> > > > > > > >
> > > > > > > > [4115] https://github.com/apache/polaris/pull/4115
> > > > > > > >
> > > > > > > > Cheers,
> > > > > > > > Dmitri.
> > > > > > > >
> > > > > > > > On Wed, Aug 12, 2026 at 5:21 PM EJ Wang <
> > > > > > ej.wang.dev.interest@gmail.com>
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > >> Hi folks,
> > > > > > > >>
> > > > > > > >> Notes from today's metrics architecture sync are in the
> usual
> > doc:
> > > > > > > >>
> > > > > > > >>
> > > > > >
> > > > >
> > > >
> >
> https://docs.google.com/document/d/100h7c4damrUzVuquYbBHM0EvA4LSWuW2IT2dN_7nYVA/edit?pli=1&tab=t.18iv1m7p6i6b
> > > > > > > >>
> > > > > > > >> Highlights:
> > > > > > > >>
> > > > > > > >> Module placement. core/ now holds the entity data model
> only.
> > A
> > > > new
> > > > > > spi/
> > > > > > > >> takes every other shared contract: feature SPIs, feature
> > non-SPI
> > > > > > > >> contracts,
> > > > > > > >> substrate SPIs, substrate non-SPI contracts, and
> > contract-required
> > > > > > types.
> > > > > > > >> That settles the question PR #5204 reopened, by placing into
> > a new
> > > > > SPI
> > > > > > > >> module. Note that it changes slightly towards the referenced
> > WIP
> > > > RFC
> > > > > > [1]
> > > > > > > >> recommendations: what proposed to place in core/, now should
> > go to
> > > > > > spi/
> > > > > > > >> except for the entity data model.
> > > > > > > >>
> > > > > > > >> Metrics PRs. #5068 merged. #5204 needs reshaping onto the
> > split.
> > > > > > #4115 and
> > > > > > > >> #4756 stay open with action on the author to follow up on
> > review
> > > > > > comments.
> > > > > > > >>
> > > > > > > >> [1]
> > > > > > > >>
> > > > > > > >>
> > > > > >
> > > > >
> > > >
> >
> https://docs.google.com/document/d/1mj3vGpix9zTDTFFJKtWT5ZxI3PTGI1YNb5xFqyWrOkc/edit?tab=t.0
> > > > > > > >>
> > > > > > > >> On Wed, Jul 15, 2026 at 11:01 AM Anand Kumar Sankaran via
> dev
> > <
> > > > > > > >> dev@polaris.apache.org> wrote:
> > > > > > > >>
> > > > > > > >> > Thanks EJ.  My plan is to introduce PR0 in a new branch /
> > PR and
> > > > > > > >> repurpose
> > > > > > > >> > #4115 and #4756 for PR1 and PR2 to maintain all the
> valuable
> > > > > > discussions
> > > > > > > >> > that happened there.
> > > > > > > >> >
> > > > > > > >> > From: EJ Wang <ej...@gmail.com>
> > > > > > > >> > Date: Wednesday, July 15, 2026 at 10:41 AM
> > > > > > > >> > To: dev@polaris.apache.org <de...@polaris.apache.org>
> > > > > > > >> > Subject: Re: Proposal for REST endpoints for table metrics
> > and
> > > > > > events
> > > > > > > >> >
> > > > > > > >> > Hi folks, Thanks again for the discussion today. I updated
> > the
> > > > > sync
> > > > > > doc
> > > > > > > >> > with notes from today's metrics sync: https: //urldefense.
> > > > > > > >> com/v3/__https:
> > > > > > > >> > //docs. google.
> > > > > > > >> >
> > > > > > > >>
> > > > > >
> > > > >
> > > >
> >
> com/document/d/100h7c4damrUzVuquYbBHM0EvA4LSWuW2IT2dN_7nYVA/edit?pli=1&tab=t.
> > > > > > > >> >
> > > > > > > >>
> > > > > >
> > > > >
> > > >
> >
> 2ahd7wze5f__;!!Iz9xO38YGHZK!4rnlVIqBQQQ6xpGe1PMV58edOS_cGrKoqdfhw4qSp_JQCavvEjaxXk0OwGPnxyEcO60AzKQK_XDtp7H3DdkFXcrXJwAz1g0$
> > > > > > > >> >
> > > > > > > >> >
> > > > > > > >> > Hi folks,
> > > > > > > >> >
> > > > > > > >> > Thanks again for the discussion today. I updated the sync
> > doc
> > > > with
> > > > > > notes
> > > > > > > >> > from today's metrics sync:
> > > > > > > >> >
> > > > > > > >> >
> > > > > > > >>
> > > > > >
> > > > >
> > > >
> >
> https://urldefense.com/v3/__https://docs.google.com/document/d/100h7c4damrUzVuquYbBHM0EvA4LSWuW2IT2dN_7nYVA/edit?pli=1&tab=t.2ahd7wze5f__;!!Iz9xO38YGHZK!4rnlVIqBQQQ6xpGe1PMV58edOS_cGrKoqdfhw4qSp_JQCavvEjaxXk0OwGPnxyEcO60AzKQK_XDtp7H3DdkFXcrXJwAz1g0$
> > > > > > > >> >
> > > > > > > >> > See highlights below:
> > > > > > > >> >
> > > > > > > >> > In previous meetings, EJ proposed a two prs framing to
> push
> > > > > forward
> > > > > > the
> > > > > > > >> > metrics query work in polaris. Dmitri pointed out a subtle
> > point
> > > > > > that
> > > > > > > >> pr2
> > > > > > > >> > is currently also trying to do the persistence related
> code
> > > > > > placement
> > > > > > > >> > refactor. For the sake of straightforward review process,
> > EJ,
> > > > > > Dmitri and
> > > > > > > >> > Anand (the author) agreed to separate the persistence
> > > > refactoring
> > > > > > into
> > > > > > > >> pr0,
> > > > > > > >> > and reframing the pr scopes as follow:
> > > > > > > >> >
> > > > > > > >> >    1.
> > > > > > > >> >
> > > > > > > >> >    Pr0: refactor irc metrics spi
> > > > > > > >> >    1.
> > > > > > > >> >
> > > > > > > >> >       SPI should stay in core/ with metric type enveloped
> > > > > > > >> >       2.
> > > > > > > >> >
> > > > > > > >> >       default spi impl, no-op, in extensions/ - default
> > > > selection
> > > > > at
> > > > > > > >> >       runtime/
> > > > > > > >> >       3.
> > > > > > > >> >
> > > > > > > >> >       move metrics persistence and correspondings to
> > extensions/
> > > > > as
> > > > > > an
> > > > > > > >> >       alternative SPI impl
> > > > > > > >> >       2.
> > > > > > > >> >
> > > > > > > >> >    Pr1: rest api for query
> > > > > > > >> >
> > > > > > > >> >
> > > > > > > >>
> > > > > >
> > > > >
> > > >
> >
> https://urldefense.com/v3/__https://github.com/apache/polaris/pull/4115/changes__;!!Iz9xO38YGHZK!4rnlVIqBQQQ6xpGe1PMV58edOS_cGrKoqdfhw4qSp_JQCavvEjaxXk0OwGPnxyEcO60AzKQK_XDtp7H3DdkFXcrXFCbEbRk$
> > > > > > > >> >    1.
> > > > > > > >> >
> > > > > > > >> >       rest api spec in extensions/ as optional spec
> > > > > > > >> >       2.
> > > > > > > >> >
> > > > > > > >> >       SPI in extensions/ as optional spec spi
> > > > > > > >> >       3.
> > > > > > > >> >
> > > > > > > >> >       default spi impl in extensions/ no-op
> > > > > > > >> >       4.
> > > > > > > >> >
> > > > > > > >> >       handler related in runtime/ - thin adaptor from http
> > shape
> > > > > to
> > > > > > spi
> > > > > > > >> >       shape
> > > > > > > >> >       3.
> > > > > > > >> >
> > > > > > > >> >    Pr2: jdbc impl for query
> > > > > > > >> >
> > > > > > > >>
> > > > > >
> > > > >
> > > >
> >
> https://urldefense.com/v3/__https://github.com/apache/polaris/pull/4756__;!!Iz9xO38YGHZK!4rnlVIqBQQQ6xpGe1PMV58edOS_cGrKoqdfhw4qSp_JQCavvEjaxXk0OwGPnxyEcO60AzKQK_XDtp7H3DdkFXcrXRVMBzqE$
> > > > > > > >> >    1.
> > > > > > > >> >
> > > > > > > >> >       alternative spi impl in extensions/ -> query with
> jdbc
> > > > > > > >> >
> > > > > > > >> >
> > > > > > > >> > Thanks,
> > > > > > > >> > -ej
> > > > > > > >> >
> > > > > > > >> > On Wed, Jul 1, 2026 at 8:01 AM Alexandre Dutra <
> > > > adutra@apache.org
> > > > > >
> > > > > > > >> wrote:
> > > > > > > >> >
> > > > > > > >> > > Hi all,
> > > > > > > >> > >
> > > > > > > >> > > > While the current implementation stores metrics in the
> > > > > > metastore,
> > > > > > > >> other
> > > > > > > >> > > deployments
> > > > > > > >> > > may choose a KV store, a dedicated metrics warehouse, or
> > > > another
> > > > > > > >> backend
> > > > > > > >> > > entirely. In those cases, exposing a Polaris REST API
> > over a
> > > > > > specific
> > > > > > > >> > > schema feels like coupling the API to an implementation
> > > > detail.
> > > > > > > >> > >
> > > > > > > >> > > I wouldn't say to an implementation detail, but rather
> to
> > a
> > > > > > particular
> > > > > > > >> > > setup / configuration. I think that's fine. If you don't
> > store
> > > > > > > >> > > metrics, you don't read metrics. Similar situations will
> > > > > certainly
> > > > > > > >> > > happen in the future, e.g. if we ever introduce an API
> to
> > read
> > > > > > events,
> > > > > > > >> > > that would only work if events are being persisted in
> the
> > > > first
> > > > > > place.
> > > > > > > >> > > Again I think that's a fair trade-off.
> > > > > > > >> > >
> > > > > > > >> > > Thanks,
> > > > > > > >> > > Alex
> > > > > > > >> > >
> > > > > > > >> > > On Wed, Jul 1, 2026 at 7:23 AM Dmitri Bourlatchkov <
> > > > > > dimas@apache.org>
> > > > > > > >> > > wrote:
> > > > > > > >> > > >
> > > > > > > >> > > > Hi Yufei,
> > > > > > > >> > > >
> > > > > > > >> > > > From my POV, it is fine for Polaris to have multiple
> > REST
> > > > API
> > > > > > > >> surfaces
> > > > > > > >> > > with
> > > > > > > >> > > > different features and different degrees of
> "readiness".
> > > > Each
> > > > > > API
> > > > > > > >> can
> > > > > > > >> > > > evolve fairly independently. With proper
> modularization
> > > > (which
> > > > > > Anand
> > > > > > > >> > > did),
> > > > > > > >> > > > changes to the main Polaris code and these extra
> modules
> > > > > should
> > > > > > > >> > normally
> > > > > > > >> > > be
> > > > > > > >> > > > non-intrusive to each other.
> > > > > > > >> > > >
> > > > > > > >> > > > I assume Anand is going to be the first user of this
> > API.
> > > > > > However,
> > > > > > > >> if
> > > > > > > >> > we
> > > > > > > >> > > do
> > > > > > > >> > > > not merge it we'll never know whether other users are
> > > > > > interested. I
> > > > > > > >> do
> > > > > > > >> > > not
> > > > > > > >> > > > think many end users read the dev ML, but they
> certainly
> > > > read
> > > > > > the
> > > > > > > >> docs,
> > > > > > > >> > > > code and release notes.
> > > > > > > >> > > >
> > > > > > > >> > > > There is no risk for Polaris as a project in exposing
> > this
> > > > > > > >> experimental
> > > > > > > >> > > > API. If usage dwindles over time, we can remove it. If
> > it
> > > > > gains
> > > > > > a
> > > > > > > >> > > > considerable user / contributor base, we will promote
> > it to
> > > > a
> > > > > > > >> > > "first-class"
> > > > > > > >> > > > (non-beta) API.
> > > > > > > >> > > >
> > > > > > > >> > > > Cheers,
> > > > > > > >> > > > Dmitri.
> > > > > > > >> > > >
> > > > > > > >> > > > On Tue, Jun 30, 2026 at 7:53 PM Yufei Gu <
> > > > > flyrain000@gmail.com>
> > > > > > > >> wrote:
> > > > > > > >> > > >
> > > > > > > >> > > > > Hi all,
> > > > > > > >> > > > >
> > > > > > > >> > > > > Thanks for driving this work forward. Building this
> > as a
> > > > POC
> > > > > > makes
> > > > > > > >> > > perfect
> > > > > > > >> > > > > sense if the goal is to validate the idea and gather
> > > > > feedback.
> > > > > > > >> Should
> > > > > > > >> > > we
> > > > > > > >> > > > > merge a POC into the main branch before agreeing on
> > the
> > > > > > overall
> > > > > > > >> > > direction?
> > > > > > > >> > > > > To me, that feels like a separate discussion from
> > whether
> > > > > the
> > > > > > > >> > > > > implementation itself looks reasonable.
> > > > > > > >> > > > >
> > > > > > > >> > > > > One concern that I don't think has been fully
> > addressed is
> > > > > > > >> whether we
> > > > > > > >> > > need
> > > > > > > >> > > > > to expose metrics consumption through a Polaris REST
> > > > > endpoint
> > > > > > at
> > > > > > > >> all.
> > > > > > > >> > > > >
> > > > > > > >> > > > > My concern is less about the API itself and more
> > about the
> > > > > > > >> boundary
> > > > > > > >> > > Polaris
> > > > > > > >> > > > > should own. The proposed REST API is tied to a
> > particular
> > > > > > > >> persistence
> > > > > > > >> > > > > schema, but that schema is just one possible
> > > > implementation.
> > > > > > While
> > > > > > > >> > the
> > > > > > > >> > > > > current implementation stores metrics in the
> > metastore,
> > > > > other
> > > > > > > >> > > deployments
> > > > > > > >> > > > > may choose a KV store, a dedicated metrics
> warehouse,
> > or
> > > > > > another
> > > > > > > >> > > backend
> > > > > > > >> > > > > entirely. In those cases, exposing a Polaris REST
> API
> > > > over a
> > > > > > > >> specific
> > > > > > > >> > > > > schema feels like coupling the API to an
> > implementation
> > > > > > detail.
> > > > > > > >> > > > >
> > > > > > > >> > > > > Polaris isn't intended to become a full metrics
> > system.
> > > > I'd
> > > > > > > >> expect it
> > > > > > > >> > > to
> > > > > > > >> > > > > focus on producing metrics and let deployments
> choose
> > the
> > > > > most
> > > > > > > >> > > appropriate
> > > > > > > >> > > > > storage and consumption model for their environment.
> > > > > > > >> > > > >
> > > > > > > >> > > > > I'm not opposed to exploring this direction, but we
> > should
> > > > > > first
> > > > > > > >> > align
> > > > > > > >> > > on
> > > > > > > >> > > > > whether exposing a REST API through Polaris is the
> > right
> > > > > long
> > > > > > term
> > > > > > > >> > > approach
> > > > > > > >> > > > > before merging it into the main branch.
> > > > > > > >> > > > >
> > > > > > > >> > > > > Thanks,
> > > > > > > >> > > > > Yufei
> > > > > > > >> > > > >
> > > > > > > >> > > > > Yufei
> > > > > > > >> > > > >
> > > > > > > >> > > > >
> > > > > > > >> > > > > On Mon, Jun 29, 2026 at 9:27 AM Dmitri Bourlatchkov
> <
> > > > > > > >> > dimas@apache.org>
> > > > > > > >> > > > > wrote:
> > > > > > > >> > > > >
> > > > > > > >> > > > > > Hi All,
> > > > > > > >> > > > > >
> > > > > > > >> > > > > > It looks like Anand addressed all action items in
> > > > [4115].
> > > > > > > >> > > > > >
> > > > > > > >> > > > > > Please review the latest state of the code. The PR
> > LGTM
> > > > > and
> > > > > > I
> > > > > > > >> > > approved in
> > > > > > > >> > > > > > GH from my side.
> > > > > > > >> > > > > >
> > > > > > > >> > > > > > [4115]
> > > > > > > >> >
> > > > > > > >>
> > > > > >
> > > > >
> > > >
> >
> https://urldefense.com/v3/__https://github.com/apache/polaris/pull/4115__;!!Iz9xO38YGHZK!4rnlVIqBQQQ6xpGe1PMV58edOS_cGrKoqdfhw4qSp_JQCavvEjaxXk0OwGPnxyEcO60AzKQK_XDtp7H3DdkFXcrXtFRscKs$
> > > > > > > >> > > > > >
> > > > > > > >> > > > > > Thanks,
> > > > > > > >> > > > > > Dmitri.
> > > > > > > >> > > > > >
> > > > > > > >> > > > > > On Thu, Jun 25, 2026 at 1:56 PM Dmitri
> Bourlatchkov
> > <
> > > > > > > >> > > dimas@apache.org>
> > > > > > > >> > > > > > wrote:
> > > > > > > >> > > > > >
> > > > > > > >> > > > > > > Hi All,
> > > > > > > >> > > > > > >
> > > > > > > >> > > > > > > Recap of the discussion in today's Community
> Sync:
> > > > > > > >> > > > > > >
> > > > > > > >> > > > > > > * The modular approach for Metrics SPI(s) and
> > > > > > implementation
> > > > > > > >> > seems
> > > > > > > >> > > to
> > > > > > > >> > > > > > have
> > > > > > > >> > > > > > > community consensus (please comment if you still
> > have
> > > > > > > >> concerns)
> > > > > > > >> > > > > > >
> > > > > > > >> > > > > > > * What modules / feature flags are enabled in
> > Apache
> > > > > > Polaris
> > > > > > > >> > > images by
> > > > > > > >> > > > > > > default will be discussed later (when the
> feature
> > is
> > > > > > close to
> > > > > > > >> > > > > > completion).
> > > > > > > >> > > > > > >
> > > > > > > >> > > > > > > In any case, users will be able to turn the full
> > > > feature
> > > > > > > >> on/off
> > > > > > > >> > at
> > > > > > > >> > > > > their
> > > > > > > >> > > > > > > discretion, regardless of the default flag
> values.
> > > > > > > >> > > > > > >
> > > > > > > >> > > > > > > * It is generally preferable to have smaller PRs
> > and
> > > > > > introduce
> > > > > > > >> > new
> > > > > > > >> > > > > > > features gradually to simplify reviews.
> > > > > > > >> > > > > > >
> > > > > > > >> > > > > > > What this means for the Metrics API, IMHO is:
> > > > > > > >> > > > > > >
> > > > > > > >> > > > > > > a) The scope of PR 4115 is probably ok as it
> > stands
> > > > > right
> > > > > > > >> now. If
> > > > > > > >> > > > > someone
> > > > > > > >> > > > > > > wants to propose splitting it further, please
> > comment.
> > > > > > > >> > > > > > >
> > > > > > > >> > > > > > > b) After 4115 is merged, Metrics Persistence PRs
> > will
> > > > > > follow,
> > > > > > > >> > with
> > > > > > > >> > > > > > smaller
> > > > > > > >> > > > > > > scope per PR as appropriate for the nature of
> the
> > code
> > > > > > > >> changes.
> > > > > > > >> > > > > > >
> > > > > > > >> > > > > > > * The major concern was about exposing the new
> > Metrics
> > > > > > REST
> > > > > > > >> API
> > > > > > > >> > to
> > > > > > > >> > > end
> > > > > > > >> > > > > > > users. The nature of this concern was about
> > avoiding
> > > > the
> > > > > > > >> (false)
> > > > > > > >> > > > > > impression
> > > > > > > >> > > > > > > that the new API is a firm and adopted Polaris
> > > > > > specification.
> > > > > > > >> > > > > > >
> > > > > > > >> > > > > > > I think this concern is reasonable.
> > > > > > > >> > > > > > >
> > > > > > > >> > > > > > > I propose that Anand update the documentation
> > sections
> > > > > of
> > > > > > PR
> > > > > > > >> 4115
> > > > > > > >> > > to
> > > > > > > >> > > > > > > emphasise that the API is currently an
> > experimental /
> > > > > POC
> > > > > > API
> > > > > > > >> at
> > > > > > > >> > > this
> > > > > > > >> > > > > > time
> > > > > > > >> > > > > > > and is subject to change (including breaking
> > > > changes). I
> > > > > > > >> believe
> > > > > > > >> > it
> > > > > > > >> > > > > > already
> > > > > > > >> > > > > > > has the "beta" label. So this request is mostly
> > for
> > > > doc
> > > > > > > >> changes
> > > > > > > >> > to
> > > > > > > >> > > > > > clarify
> > > > > > > >> > > > > > > user expectations, since the "beta" label alone
> > may
> > > > not
> > > > > be
> > > > > > > >> > > interpreted
> > > > > > > >> > > > > > the
> > > > > > > >> > > > > > > same way by all users.
> > > > > > > >> > > > > > >
> > > > > > > >> > > > > > > I hope this update will resolve all remaining
> > concerns
> > > > > > with
> > > > > > > >> 4115
> > > > > > > >> > > and
> > > > > > > >> > > > > > allow
> > > > > > > >> > > > > > > for it to be merged so that the community will
> be
> > able
> > > > > to
> > > > > > > >> proceed
> > > > > > > >> > > to
> > > > > > > >> > > > > > > follow-up PRs (as noted above).
> > > > > > > >> > > > > > >
> > > > > > > >> > > > > > > Please add/correct me if I missed anything
> > relevant.
> > > > > > > >> > > > > > >
> > > > > > > >> > > > > > > Thanks,
> > > > > > > >> > > > > > > Dmitri.
> > > > > > > >> > > > > > >
> > > > > > > >> > > > > > > On Wed, Jun 24, 2026 at 8:11 PM Yufei Gu <
> > > > > > > >> flyrain000@gmail.com>
> > > > > > > >> > > wrote:
> > > > > > > >> > > > > > >
> > > > > > > >> > > > > > >> I do not think we have agreement on the query
> > REST
> > > > APIs
> > > > > > yet.
> > > > > > > >> > > > > > >>
> > > > > > > >> > > > > > >> In particular, the current query API shape
> seems
> > > > highly
> > > > > > > >> coupled
> > > > > > > >> > > to the
> > > > > > > >> > > > > > >> *example[1]* metrics in the Iceberg REST spec.
> I
> > do
> > > > not
> > > > > > think
> > > > > > > >> > > that is
> > > > > > > >> > > > > a
> > > > > > > >> > > > > > >> good way to design a REST API contract. For
> > example,
> > > > > > fields
> > > > > > > >> like
> > > > > > > >> > > > > > >> resultDataFiles, resultDeleteFiles,
> > > > totalFileSizeBytes,
> > > > > > > >> > > > > > >> totalDataManifests,
> > > > > > > >> > > > > > >> totalDeleteManifests, and scannedDataManifests
> > are
> > > > very
> > > > > > > >> specific
> > > > > > > >> > > to
> > > > > > > >> > > > > one
> > > > > > > >> > > > > > >> style of scan metric. That makes the API look
> > more
> > > > > like a
> > > > > > > >> direct
> > > > > > > >> > > > > > >> projection
> > > > > > > >> > > > > > >> of the current example payload than a general
> > query
> > > > > > model.
> > > > > > > >> > > > > > >>
> > > > > > > >> > > > > > >> My preference is that Polaris should not define
> > or
> > > > > > strongly
> > > > > > > >> > opine
> > > > > > > >> > > on
> > > > > > > >> > > > > the
> > > > > > > >> > > > > > >> Iceberg scan or commit metrics query REST API
> at
> > this
> > > > > > point.
> > > > > > > >> > > Polaris
> > > > > > > >> > > > > can
> > > > > > > >> > > > > > >> resolve and authorize the table, accept the
> > metrics
> > > > > > report,
> > > > > > > >> and
> > > > > > > >> > > > > delegate
> > > > > > > >> > > > > > >> it
> > > > > > > >> > > > > > >> to the configured reporting path. Durable
> > storage,
> > > > > > filtering,
> > > > > > > >> > > > > > dashboards,
> > > > > > > >> > > > > > >> and query APIs should remain downstream impl.
> > choices
> > > > > > unless
> > > > > > > >> and
> > > > > > > >> > > until
> > > > > > > >> > > > > > the
> > > > > > > >> > > > > > >> Iceberg community standardizes a query
> contract,
> > > > which
> > > > > I
> > > > > > > >> doubt
> > > > > > > >> > > will
> > > > > > > >> > > > > > >> happen.
> > > > > > > >> > > > > > >>
> > > > > > > >> > > > > > >> So I do not think the intake path has to be
> > blocked,
> > > > > but
> > > > > > I
> > > > > > > >> would
> > > > > > > >> > > > > prefer
> > > > > > > >> > > > > > >> not
> > > > > > > >> > > > > > >> to merge a Polaris specific query API that may
> be
> > > > hard
> > > > > to
> > > > > > > >> evolve
> > > > > > > >> > > > > later.
> > > > > > > >> > > > > > >>
> > > > > > > >> > > > > > >> 1.
> > > > > > > >> > > > > > >>
> > > > > > > >> > > > > > >>
> > > > > > > >> > > > > >
> > > > > > > >> > > > >
> > > > > > > >> > >
> > > > > > > >> >
> > > > > > > >>
> > > > > >
> > > > >
> > > >
> >
> https://urldefense.com/v3/__https://github.com/apache/iceberg/blob/2a6c556c0c883c04c6d3fbd68e6aeda36b91e0aa/open-api/rest-catalog-open-api.yaml*L4152__;Iw!!Iz9xO38YGHZK!4rnlVIqBQQQ6xpGe1PMV58edOS_cGrKoqdfhw4qSp_JQCavvEjaxXk0OwGPnxyEcO60AzKQK_XDtp7H3DdkFXcrXCDsqYGo$
> > > > > > > >> > > > > > >>
> > > > > > > >> > > > > > >> Yufei
> > > > > > > >> > > > > > >>
> > > > > > > >> > > > > > >>
> > > > > > > >> > > > > > >> On Tue, Jun 23, 2026 at 9:30 AM Dmitri
> > Bourlatchkov <
> > > > > > > >> > > dimas@apache.org
> > > > > > > >> > > > > >
> > > > > > > >> > > > > > >> wrote:
> > > > > > > >> > > > > > >>
> > > > > > > >> > > > > > >> > Hi All,
> > > > > > > >> > > > > > >> >
> > > > > > > >> > > > > > >> > So, what are the blockers for merging [4115],
> > if
> > > > any?
> > > > > > > >> > > > > > >> >
> > > > > > > >> > > > > > >> > [4115]
> > > > > > > >> >
> > > > > > > >>
> > > > > >
> > > > >
> > > >
> >
> https://urldefense.com/v3/__https://github.com/apache/polaris/pull/4115__;!!Iz9xO38YGHZK!4rnlVIqBQQQ6xpGe1PMV58edOS_cGrKoqdfhw4qSp_JQCavvEjaxXk0OwGPnxyEcO60AzKQK_XDtp7H3DdkFXcrXtFRscKs$
> > > > > > > >> > > > > > >> >
> > > > > > > >> > > > > > >> > Thanks,
> > > > > > > >> > > > > > >> > Dmitri.
> > > > > > > >> > > > > > >> >
> > > > > > > >> > > > > > >> > On Wed, Jun 17, 2026 at 5:28 PM EJ Wang <
> > > > > > > >> > > > > > ej.wang.dev.interest@gmail.com
> > > > > > > >> > > > > > >> >
> > > > > > > >> > > > > > >> > wrote:
> > > > > > > >> > > > > > >> >
> > > > > > > >> > > > > > >> > > Hi folks,
> > > > > > > >> > > > > > >> > >
> > > > > > > >> > > > > > >> > > Thanks again for the discussion today. I
> > updated
> > > > > the
> > > > > > sync
> > > > > > > >> > doc
> > > > > > > >> > > with
> > > > > > > >> > > > > > >> notes
> > > > > > > >> > > > > > >> > > from today's metrics sync:
> > > > > > > >> > > > > > >> > >
> > > > > > > >> > > > > > >> > >
> > > > > > > >> > > > > > >> > >
> > > > > > > >> > > > > > >> >
> > > > > > > >> > > > > > >>
> > > > > > > >> > > > > >
> > > > > > > >> > > > >
> > > > > > > >> > >
> > > > > > > >> >
> > > > > > > >>
> > > > > >
> > > > >
> > > >
> >
> https://urldefense.com/v3/__https://docs.google.com/document/d/100h7c4damrUzVuquYbBHM0EvA4LSWuW2IT2dN_7nYVA/edit?tab=t.ezk0rgdx0c6m__;!!Iz9xO38YGHZK!4rnlVIqBQQQ6xpGe1PMV58edOS_cGrKoqdfhw4qSp_JQCavvEjaxXk0OwGPnxyEcO60AzKQK_XDtp7H3DdkFXcrX83IZkm0$
> > > > > > > >> > > > > > >> > >
> > > > > > > >> > > > > > >> > > A few highlights:
> > > > > > > >> > > > > > >> > >
> > > > > > > >> > > > > > >> > >    - No objection to keeping the default
> > metrics
> > > > > > behavior
> > > > > > > >> > > small:
> > > > > > > >> > > > > > >> > >    no-op/log-only is enough as the built-in
> > > > > default.
> > > > > > > >> > > > > > >> > >    - Durable storage, event forwarding,
> > external
> > > > > > queues,
> > > > > > > >> > > > > dashboards,
> > > > > > > >> > > > > > >> and
> > > > > > > >> > > > > > >> > >    custom filtering should be
> implementation
> > > > > choices
> > > > > > > >> behind
> > > > > > > >> > > the
> > > > > > > >> > > > > > >> metrics
> > > > > > > >> > > > > > >> > >    reporting path.
> > > > > > > >> > > > > > >> > >    - For PR #4115, the REST layer should
> > > > > > > >> > resolve/authz/accept
> > > > > > > >> > > the
> > > > > > > >> > > > > > >> report,
> > > > > > > >> > > > > > >> > >    then delegate to the selected metrics
> > > > > > implementation.
> > > > > > > >> It
> > > > > > > >> > > should
> > > > > > > >> > > > > > not
> > > > > > > >> > > > > > >> > own
> > > > > > > >> > > > > > >> > >    durable storage semantics.
> > > > > > > >> > > > > > >> > >    - The rough boundary we discussed is:
> spec
> > > > owns
> > > > > > the
> > > > > > > >> wire
> > > > > > > >> > > > > > contract,
> > > > > > > >> > > > > > >> > >    runtime owns framework wiring,
> > API/contract
> > > > > > modules
> > > > > > > >> own
> > > > > > > >> > > > > > >> > provider-facing
> > > > > > > >> > > > > > >> > >    contracts, and extensions own
> replaceable
> > > > > > > >> > implementations.
> > > > > > > >> > > > > > >> > >    - OpenLineage has a similar path-naming
> > > > > question.
> > > > > > We
> > > > > > > >> > should
> > > > > > > >> > > > > avoid
> > > > > > > >> > > > > > >> > >    occupying a generic lineage path too
> > early if
> > > > we
> > > > > > want
> > > > > > > >> > room
> > > > > > > >> > > for
> > > > > > > >> > > > > > >> other
> > > > > > > >> > > > > > >> > >    lineage systems later.
> > > > > > > >> > > > > > >> > >
> > > > > > > >> > > > > > >> > > I believe the boundary rules previewed in
> > sync
> > > > > > meeting
> > > > > > > >> are
> > > > > > > >> > > useful
> > > > > > > >> > > > > > for
> > > > > > > >> > > > > > >> > > reviewing PR #4115, but it is not
> > > > > > > >> > > > > > >> > > a final module-layout proposal.
> > > > > > > >> > > > > > >> > >
> > > > > > > >> > > > > > >> > > For metrics specifically, I think the
> target
> > > > > remains:
> > > > > > > >> > > > > > >> > >    runtime
> > > > > > > >> > > > > > >> > >      -> metrics reporting/emitting contract
> > > > > > > >> > > > > > >> > >      -> default implementation
> > > > > > > >> > > > > > >> > >           -> lower implementation details,
> if
> > > > > needed
> > > > > > > >> > > > > > >> > >
> > > > > > > >> > > > > > >> > > That keeps the default simple, while
> allowing
> > > > > durable
> > > > > > > >> JDBC,
> > > > > > > >> > > > > > >> event-backed,
> > > > > > > >> > > > > > >> > > or external-queue implementations to be
> added
> > > > > > separately.
> > > > > > > >> > > > > > >> > >
> > > > > > > >> > > > > > >> > > We also touched on the split between
> Iceberg
> > > > > metrics
> > > > > > > >> > emitting
> > > > > > > >> > > and
> > > > > > > >> > > > > > >> > > Polaris-owned metrics querying. The
> querying
> > side
> > > > > can
> > > > > > > >> keep
> > > > > > > >> > > > > evolving,
> > > > > > > >> > > > > > >> > > especially for dashboard or generic-table
> use
> > > > > cases,
> > > > > > but
> > > > > > > >> I
> > > > > > > >> > do
> > > > > > > >> > > not
> > > > > > > >> > > > > > >> think
> > > > > > > >> > > > > > >> > > that needs to block the intake path in this
> > PR.
> > > > > > > >> > > > > > >> > >
> > > > > > > >> > > > > > >> > > Thanks,
> > > > > > > >> > > > > > >> > > -ej
> > > > > > > >> > > > > > >> > >
> > > > > > > >> > > > > > >> > > On Tue, Jun 16, 2026 at 2:17 PM Yufei Gu <
> > > > > > > >> > > flyrain000@gmail.com>
> > > > > > > >> > > > > > >> wrote:
> > > > > > > >> > > > > > >> > >
> > > > > > > >> > > > > > >> > > > Thanks for chiming in, EJ. Agreed that
> the
> > > > > default
> > > > > > > >> battery
> > > > > > > >> > > > > should
> > > > > > > >> > > > > > >> stay
> > > > > > > >> > > > > > >> > > > small. I'm leaning toward not including
> > metrics
> > > > > > > >> > persistence
> > > > > > > >> > > in
> > > > > > > >> > > > > the
> > > > > > > >> > > > > > >> > > default
> > > > > > > >> > > > > > >> > > > battery.
> > > > > > > >> > > > > > >> > > >
> > > > > > > >> > > > > > >> > > > Yufei
> > > > > > > >> > > > > > >> > > >
> > > > > > > >> > > > > > >> > > >
> > > > > > > >> > > > > > >> > > > On Thu, Jun 11, 2026 at 8:30 PM EJ Wang <
> > > > > > > >> > > > > > >> > ej.wang.dev.interest@gmail.com>
> > > > > > > >> > > > > > >> > > > wrote:
> > > > > > > >> > > > > > >> > > >
> > > > > > > >> > > > > > >> > > > > Hi Yufei,
> > > > > > > >> > > > > > >> > > > >
> > > > > > > >> > > > > > >> > > > > Thanks for connecting this back to the
> > REST
> > > > > > endpoint
> > > > > > > >> > > > > proposal. I
> > > > > > > >> > > > > > >> > > replied
> > > > > > > >> > > > > > >> > > > > with the fuller version on the
> > > > event-forwarding
> > > > > > > >> thread,
> > > > > > > >> > > but
> > > > > > > >> > > > > > >> wanted to
> > > > > > > >> > > > > > >> > > add
> > > > > > > >> > > > > > >> > > > > the shorter version here too.
> > > > > > > >> > > > > > >> > > > >
> > > > > > > >> > > > > > >> > > > > I agree that metrics reporting/emitting
> > is
> > > > the
> > > > > > right
> > > > > > > >> > > > > conceptual
> > > > > > > >> > > > > > >> > > boundary,
> > > > > > > >> > > > > > >> > > > > and I also think the event/listener
> path
> > is a
> > > > > > good
> > > > > > > >> > > > > > implementation
> > > > > > > >> > > > > > >> > > > direction
> > > > > > > >> > > > > > >> > > > > to explore. The distinction I want to
> > keep
> > > > > clear
> > > > > > is
> > > > > > > >> > > default
> > > > > > > >> > > > > > >> battery
> > > > > > > >> > > > > > >> > vs
> > > > > > > >> > > > > > >> > > > > extension implementation.
> > > > > > > >> > > > > > >> > > > >
> > > > > > > >> > > > > > >> > > > > For the REST/API side, I would keep the
> > > > > semantics
> > > > > > > >> > narrow:
> > > > > > > >> > > > > > Polaris
> > > > > > > >> > > > > > >> > > > resolves
> > > > > > > >> > > > > > >> > > > > and authorizes the table, accepts the
> > Iceberg
> > > > > > > >> > scan/commit
> > > > > > > >> > > > > > metrics
> > > > > > > >> > > > > > >> > > report
> > > > > > > >> > > > > > >> > > > > into the configured reporting/emitting
> > path,
> > > > > and
> > > > > > > >> returns
> > > > > > > >> > > 204.
> > > > > > > >> > > > > > That
> > > > > > > >> > > > > > >> > 204
> > > > > > > >> > > > > > >> > > > > should mean Polaris accepted the report
> > into
> > > > > the
> > > > > > > >> > ingestion
> > > > > > > >> > > > > path;
> > > > > > > >> > > > > > >> it
> > > > > > > >> > > > > > >> > > > should
> > > > > > > >> > > > > > >> > > > > not imply durable storage.
> > > > > > > >> > > > > > >> > > > >
> > > > > > > >> > > > > > >> > > > > Durable storage, event forwarding,
> > external
> > > > > > telemetry
> > > > > > > >> > > routing,
> > > > > > > >> > > > > > >> > > filtering,
> > > > > > > >> > > > > > >> > > > > and retention should sit behind that
> > > > > > > >> reporting/emitting
> > > > > > > >> > > > > boundary
> > > > > > > >> > > > > > >> as
> > > > > > > >> > > > > > >> > > > > implementation-layer behavior. A
> durable
> > JDBC
> > > > > > path
> > > > > > > >> can
> > > > > > > >> > be
> > > > > > > >> > > one
> > > > > > > >> > > > > > >> named
> > > > > > > >> > > > > > >> > > > > extension implementation. An
> > event/listener
> > > > > > > >> forwarding
> > > > > > > >> > > path
> > > > > > > >> > > > > can
> > > > > > > >> > > > > > be
> > > > > > > >> > > > > > >> > > > another
> > > > > > > >> > > > > > >> > > > > named extension implementation.
> > > > > > > >> > > > > > >> > > > >
> > > > > > > >> > > > > > >> > > > > So I am +1 on the event-forwarding idea
> > as a
> > > > > > > >> non-default
> > > > > > > >> > > > > > extension
> > > > > > > >> > > > > > >> > > > > implementation. I just would not make
> it
> > the
> > > > > > default
> > > > > > > >> > > battery
> > > > > > > >> > > > > or
> > > > > > > >> > > > > > >> the
> > > > > > > >> > > > > > >> > > core
> > > > > > > >> > > > > > >> > > > > API shape. The default battery can stay
> > small
> > > > > and
> > > > > > > >> safe,
> > > > > > > >> > > while
> > > > > > > >> > > > > > >> > > deployments
> > > > > > > >> > > > > > >> > > > > that need forwarding, dashboards, or
> > durable
> > > > > > storage
> > > > > > > >> can
> > > > > > > >> > > opt
> > > > > > > >> > > > > > into
> > > > > > > >> > > > > > >> the
> > > > > > > >> > > > > > >> > > > > implementation that matches their
> > operational
> > > > > > model.
> > > > > > > >> > > > > > >> > > > >
> > > > > > > >> > > > > > >> > > > > Thanks,
> > > > > > > >> > > > > > >> > > > > -ej
> > > > > > > >> > > > > > >> > > > >
> > > > > > > >> > > > > > >> > > > > On Wed, Jun 10, 2026 at 11:43 AM Yufei
> > Gu <
> > > > > > > >> > > > > flyrain000@gmail.com
> > > > > > > >> > > > > > >
> > > > > > > >> > > > > > >> > > wrote:
> > > > > > > >> > > > > > >> > > > >
> > > > > > > >> > > > > > >> > > > > > Thanks EJ. I agree that the
> > > > > reporting/emitting
> > > > > > > >> > boundary
> > > > > > > >> > > > > feels
> > > > > > > >> > > > > > >> like
> > > > > > > >> > > > > > >> > > the
> > > > > > > >> > > > > > >> > > > > > right SPI boundary, but I wonder if
> we
> > can
> > > > > > simplify
> > > > > > > >> > this
> > > > > > > >> > > > > even
> > > > > > > >> > > > > > >> > > further.
> > > > > > > >> > > > > > >> > > > > >
> > > > > > > >> > > > > > >> > > > > > Iceberg scan and commit metrics look
> > very
> > > > > > similar
> > > > > > > >> to
> > > > > > > >> > > events
> > > > > > > >> > > > > to
> > > > > > > >> > > > > > >> me.
> > > > > > > >> > > > > > >> > > They
> > > > > > > >> > > > > > >> > > > > are
> > > > > > > >> > > > > > >> > > > > > append only, asynchronously consumed,
> > often
> > > > > > > >> forwarded
> > > > > > > >> > to
> > > > > > > >> > > > > > >> external
> > > > > > > >> > > > > > >> > > > > systems,
> > > > > > > >> > > > > > >> > > > > > and have similar retention and
> cleanup
> > > > > > > >> requirements.
> > > > > > > >> > > More
> > > > > > > >> > > > > > >> details
> > > > > > > >> > > > > > >> > can
> > > > > > > >> > > > > > >> > > > be
> > > > > > > >> > > > > > >> > > > > > found in this thread [1]. In that
> case,
> > > > > > Polaris may
> > > > > > > >> > only
> > > > > > > >> > > > > need
> > > > > > > >> > > > > > to
> > > > > > > >> > > > > > >> > > accept
> > > > > > > >> > > > > > >> > > > > the
> > > > > > > >> > > > > > >> > > > > > report and emit an event through the
> > > > existing
> > > > > > event
> > > > > > > >> > > > > framework.
> > > > > > > >> > > > > > >> > > > Filtering,
> > > > > > > >> > > > > > >> > > > > > async delivery, custom sinks, and
> > retention
> > > > > > > >> mechanisms
> > > > > > > >> > > could
> > > > > > > >> > > > > > >> then
> > > > > > > >> > > > > > >> > be
> > > > > > > >> > > > > > >> > > > > shared
> > > > > > > >> > > > > > >> > > > > > instead of introducing a separate
> > metrics
> > > > > > specific
> > > > > > > >> > > extension
> > > > > > > >> > > > > > >> layer
> > > > > > > >> > > > > > >> > > and
> > > > > > > >> > > > > > >> > > > > SPI.
> > > > > > > >> > > > > > >> > > > > >
> > > > > > > >> > > > > > >> > > > > > The main requirement I still see is
> > > > querying
> > > > > > > >> metrics
> > > > > > > >> > > through
> > > > > > > >> > > > > > >> > Polaris.
> > > > > > > >> > > > > > >> > > > If
> > > > > > > >> > > > > > >> > > > > we
> > > > > > > >> > > > > > >> > > > > > want that battery included
> experience,
> > we
> > > > > could
> > > > > > > >> work
> > > > > > > >> > on
> > > > > > > >> > > a
> > > > > > > >> > > > > > >> > persistence
> > > > > > > >> > > > > > >> > > > > > listener implementation that
> > selectively
> > > > > saves
> > > > > > > >> metrics
> > > > > > > >> > > and
> > > > > > > >> > > > > > >> expose
> > > > > > > >> > > > > > >> > it
> > > > > > > >> > > > > > >> > > > via
> > > > > > > >> > > > > > >> > > > > > the IRC event endpoint which seems
> > pretty
> > > > > > close in
> > > > > > > >> > > Iceberg
> > > > > > > >> > > > > > >> > > > > > community. Another option I like more
> > is to
> > > > > > allow
> > > > > > > >> > users
> > > > > > > >> > > to
> > > > > > > >> > > > > > >> develop
> > > > > > > >> > > > > > >> > > > their
> > > > > > > >> > > > > > >> > > > > > own listener, so they can persist
> > > > scan/commit
> > > > > > > >> metrics
> > > > > > > >> > > to any
> > > > > > > >> > > > > > >> > systems
> > > > > > > >> > > > > > >> > > > they
> > > > > > > >> > > > > > >> > > > > > prefer. In general, that feels like
> an
> > > > > > > >> implementation
> > > > > > > >> > > choice
> > > > > > > >> > > > > > on
> > > > > > > >> > > > > > >> top
> > > > > > > >> > > > > > >> > > of
> > > > > > > >> > > > > > >> > > > > the
> > > > > > > >> > > > > > >> > > > > > event framework rather than something
> > that
> > > > > > needs to
> > > > > > > >> > > shape
> > > > > > > >> > > > > the
> > > > > > > >> > > > > > >> core
> > > > > > > >> > > > > > >> > > > > > architecture.
> > > > > > > >> > > > > > >> > > > > >
> > > > > > > >> > > > > > >> > > > > > 1.
> > > > > > > >> > > > > > >> >
> > > > > > > >> > >
> > > > > > > >> >
> > > > > > > >>
> > > > > >
> > > > >
> > > >
> >
> https://urldefense.com/v3/__https://lists.apache.org/thread/x9j8nscvy8hq61tyn01mj8yp6n9of0kp__;!!Iz9xO38YGHZK!4rnlVIqBQQQ6xpGe1PMV58edOS_cGrKoqdfhw4qSp_JQCavvEjaxXk0OwGPnxyEcO60AzKQK_XDtp7H3DdkFXcrXGcGDgks$
> > > > > > > >> > > > > > >> > > > > >
> > > > > > > >> > > > > > >> > > > > > Thanks,
> > > > > > > >> > > > > > >> > > > > > Yufei
> > > > > > > >> > > > > > >> > > > > >
> > > > > > > >> > > > > > >> > > > > >
> > > > > > > >> > > > > > >> > > > > > On Thu, Jun 4, 2026 at 5:14 PM EJ
> Wang
> > <
> > > > > > > >> > > > > > >> > > ej.wang.dev.interest@gmail.com
> > > > > > > >> > > > > > >> > > > >
> > > > > > > >> > > > > > >> > > > > > wrote:
> > > > > > > >> > > > > > >> > > > > >
> > > > > > > >> > > > > > >> > > > > > > Good point. I agree the existing
> > > > > > > >> > > PolarisMetricsReporter is
> > > > > > > >> > > > > > >> > already
> > > > > > > >> > > > > > >> > > > very
> > > > > > > >> > > > > > >> > > > > > > close to the right conceptual
> > boundary. I
> > > > > am
> > > > > > not
> > > > > > > >> > > > > proposing a
> > > > > > > >> > > > > > >> > second
> > > > > > > >> > > > > > >> > > > > > > parallel reporter concept. The
> > > > distinction
> > > > > I
> > > > > > am
> > > > > > > >> > > trying to
> > > > > > > >> > > > > > >> make is
> > > > > > > >> > > > > > >> > > > > > narrower:
> > > > > > > >> > > > > > >> > > > > > >
> > > > > > > >> > > > > > >> > > > > > > Current state:
> > > > > > > >> > > > > > >> > > > > > >
> > > > > > > >> > > > > > >> > > > > > >    - PolarisMetricsReporter is the
> > > > existing
> > > > > > > >> metrics
> > > > > > > >> > > > > > reporting
> > > > > > > >> > > > > > >> > hook.
> > > > > > > >> > > > > > >> > > > > > >    - But it currently lives under
> > > > > > > >> runtime/service.
> > > > > > > >> > > > > > >> > > > > > >    - Its contract is documented
> > around
> > > > > > > >> Quarkus/CDI
> > > > > > > >> > > > > discovery
> > > > > > > >> > > > > > >> and
> > > > > > > >> > > > > > >> > > > > > > @Identifier
> > > > > > > >> > > > > > >> > > > > > >    selection.
> > > > > > > >> > > > > > >> > > > > > >    - The default implementation is
> > > > > log-only.
> > > > > > > >> > > > > > >> > > > > > >    - The durable implementation
> > writes
> > > > > > through
> > > > > > > >> > > > > > >> > > PolarisMetricsManager
> > > > > > > >> > > > > > >> > > > > and
> > > > > > > >> > > > > > >> > > > > > >    MetricsPersistence.
> > > > > > > >> > > > > > >> > > > > > >    - MetricsPersistence is
> currently
> > > > > > inherited by
> > > > > > > >> > > > > > >> > BasePersistence.
> > > > > > > >> > > > > > >> > > > > > >
> > > > > > > >> > > > > > >> > > > > > > My proposal:
> > > > > > > >> > > > > > >> > > > > > >
> > > > > > > >> > > > > > >> > > > > > >    - Keep the reporting/emitting
> > boundary
> > > > > as
> > > > > > the
> > > > > > > >> > SPI.
> > > > > > > >> > > > > > >> > > > > > >    - Revise that contract in a
> > > > > > framework-agnostic
> > > > > > > >> > > optional
> > > > > > > >> > > > > > >> > Iceberg
> > > > > > > >> > > > > > >> > > > > > *metrics
> > > > > > > >> > > > > > >> > > > > > >    extension API layer*.
> > > > > > > >> > > > > > >> > > > > > >    - Keep runtime/service
> > responsible for
> > > > > > REST
> > > > > > > >> > > ingestion,
> > > > > > > >> > > > > > >> > > table/authz
> > > > > > > >> > > > > > >> > > > > > >    validation, and runtime wiring.
> > > > > > > >> > > > > > >> > > > > > >    - Keep the battery
> implementation
> > > > > > log-only in
> > > > > > > >> the
> > > > > > > >> > > > > metrics
> > > > > > > >> > > > > > >> > > > extension
> > > > > > > >> > > > > > >> > > > > > API
> > > > > > > >> > > > > > >> > > > > > >    layer, *not under
> > runtime/service*.
> > > > > > > >> > > > > > >> > > > > > >    - Treat durable JDBC metrics as
> > one
> > > > > > > >> > implementation
> > > > > > > >> > > of
> > > > > > > >> > > > > the
> > > > > > > >> > > > > > >> > > > reporting
> > > > > > > >> > > > > > >> > > > > > SPI.
> > > > > > > >> > > > > > >> > > > > > >    - Decompose the
> > MetricsPersistence and
> > > > > > > >> > > BasePersistence
> > > > > > > >> > > > > > >> > coupling
> > > > > > > >> > > > > > >> > > as
> > > > > > > >> > > > > > >> > > > > > >    durable implementation detail.
> > > > > > > >> > > > > > >> > > > > > >
> > > > > > > >> > > > > > >> > > > > > > So the difference is not “new
> > reporter
> > > > > > concept vs
> > > > > > > >> > old
> > > > > > > >> > > > > > reporter
> > > > > > > >> > > > > > >> > > > > concept.”
> > > > > > > >> > > > > > >> > > > > > > but:
> > > > > > > >> > > > > > >> > > > > > >
> > > > > > > >> > > > > > >> > > > > > >
> > > > > > > >> > > > > > >> > > > > > >    - PolarisMetricsReporter today =
> > > > > > > >> runtime/service
> > > > > > > >> > > > > > CDI-shaped
> > > > > > > >> > > > > > >> > > hook.
> > > > > > > >> > > > > > >> > > > > > >    - Target reporting SPI = same
> > > > conceptual
> > > > > > > >> > boundary,
> > > > > > > >> > > but
> > > > > > > >> > > > > > >> placed
> > > > > > > >> > > > > > >> > in
> > > > > > > >> > > > > > >> > > > the
> > > > > > > >> > > > > > >> > > > > > >    right extension API layer and
> kept
> > > > > > > >> > > framework-agnostic.
> > > > > > > >> > > > > > >> > > > > > >
> > > > > > > >> > > > > > >> > > > > > > For the current metrics PR, I think
> > the
> > > > > > minimal
> > > > > > > >> > > > > > architectural
> > > > > > > >> > > > > > >> > > cleanup
> > > > > > > >> > > > > > >> > > > > > > before merge is to *put the
> > reporting SPI
> > > > > > > >> contract
> > > > > > > >> > > and the
> > > > > > > >> > > > > > >> > battery
> > > > > > > >> > > > > > >> > > > > > default
> > > > > > > >> > > > > > >> > > > > > > in the right place*. The durable
> JDBC
> > > > > > > >> > implementation,
> > > > > > > >> > > > > > >> > event-backed
> > > > > > > >> > > > > > >> > > > > async
> > > > > > > >> > > > > > >> > > > > > > implementation, external queue
> > support,
> > > > and
> > > > > > > >> deeper
> > > > > > > >> > > > > > persistence
> > > > > > > >> > > > > > >> > > > cleanup
> > > > > > > >> > > > > > >> > > > > > can
> > > > > > > >> > > > > > >> > > > > > > continue as follow-ups, as long as
> > we do
> > > > > not
> > > > > > lock
> > > > > > > >> > the
> > > > > > > >> > > SPI
> > > > > > > >> > > > > > >> > boundary
> > > > > > > >> > > > > > >> > > to
> > > > > > > >> > > > > > >> > > > > > > BasePersistence or to
> runtime/service
> > > > > wiring.
> > > > > > > >> > > > > > >> > > > > > >
> > > > > > > >> > > > > > >> > > > > > > -ej
> > > > > > > >> > > > > > >> > > > > > >
> > > > > > > >> > > > > > >> > > > > > > On Wed, Jun 3, 2026 at 3:00 PM
> Dmitri
> > > > > > > >> Bourlatchkov <
> > > > > > > >> > > > > > >> > > dimas@apache.org
> > > > > > > >> > > > > > >> > > > >
> > > > > > > >> > > > > > >> > > > > > > wrote:
> > > > > > > >> > > > > > >> > > > > > >
> > > > > > > >> > > > > > >> > > > > > > > Hi EJ,
> > > > > > > >> > > > > > >> > > > > > > >
> > > > > > > >> > > > > > >> > > > > > > > Thanks for the recap / summary!
> > > > > > > >> > > > > > >> > > > > > > >
> > > > > > > >> > > > > > >> > > > > > > > > Do folks agree that the stable
> > SPI
> > > > > > boundary
> > > > > > > >> > > should be
> > > > > > > >> > > > > > >> > > > > > > > metrics reporting/emitting, not
> > metrics
> > > > > > > >> > persistence?
> > > > > > > >> > > > > > >> > > > > > > >
> > > > > > > >> > > > > > >> > > > > > > > SGTM.
> > > > > > > >> > > > > > >> > > > > > > >
> > > > > > > >> > > > > > >> > > > > > > > > Does an optional Iceberg
> metrics
> > > > > > extension
> > > > > > > >> API
> > > > > > > >> > > layer
> > > > > > > >> > > > > > sound
> > > > > > > >> > > > > > >> > like
> > > > > > > >> > > > > > >> > > > the
> > > > > > > >> > > > > > >> > > > > > > right
> > > > > > > >> > > > > > >> > > > > > > > home for this SPI?
> > > > > > > >> > > > > > >> > > > > > > >
> > > > > > > >> > > > > > >> > > > > > > > How is that different from
> current
> > > > > > > >> > > > > PolarisMetricsReporter?
> > > > > > > >> > > > > > >> > > > > > > >
> > > > > > > >> > > > > > >> > > > > > > > > Should the current durable
> > metrics
> > > > work
> > > > > > be
> > > > > > > >> > > reframed
> > > > > > > >> > > > > as a
> > > > > > > >> > > > > > >> > > durable
> > > > > > > >> > > > > > >> > > > > > > > JDBC reference implementation of
> > that
> > > > > SPI?
> > > > > > > >> > > > > > >> > > > > > > >
> > > > > > > >> > > > > > >> > > > > > > > SGTM.
> > > > > > > >> > > > > > >> > > > > > > >
> > > > > > > >> > > > > > >> > > > > > > > > What is the smallest PR
> sequence
> > to
> > > > get
> > > > > > there
> > > > > > > >> > > without
> > > > > > > >> > > > > > >> > blocking
> > > > > > > >> > > > > > >> > > > > > > > the current metrics work
> > unnecessarily?
> > > > > > > >> > > > > > >> > > > > > > >
> > > > > > > >> > > > > > >> > > > > > > > Let's get
> > > > > > > >> > >
> > > > > > > >> >
> > > > > > > >>
> > > > > >
> > > > >
> > > >
> >
> https://urldefense.com/v3/__https://github.com/apache/polaris/pull/4397__;!!Iz9xO38YGHZK!4rnlVIqBQQQ6xpGe1PMV58edOS_cGrKoqdfhw4qSp_JQCavvEjaxXk0OwGPnxyEcO60AzKQK_XDtp7H3DdkFXcrXuu5jefU$
> > > > > > > >> > > > > > >> merged
> > > > > > > >> > > > > > >> > > > first.
> > > > > > > >> > > > > > >> > > > > > > >
> > > > > > > >> > > > > > >> > > > > > > > Then, I'd propose isolating the
> > Metrics
> > > > > > schema
> > > > > > > >> > from
> > > > > > > >> > > the
> > > > > > > >> > > > > > >> > MetaStore
> > > > > > > >> > > > > > >> > > > > > schema.
> > > > > > > >> > > > > > >> > > > > > > > This will probably have some
> ripple
> > > > > effect
> > > > > > into
> > > > > > > >> > the
> > > > > > > >> > > > > > >> bootstrap
> > > > > > > >> > > > > > >> > > > > > workflows,
> > > > > > > >> > > > > > >> > > > > > > so
> > > > > > > >> > > > > > >> > > > > > > > it's not a trivial change.
> > > > > > > >> > > > > > >> > > > > > > >
> > > > > > > >> > > > > > >> > > > > > > > Then, let's reassess.
> > > > > > > >> > > > > > >> > > > > > > >
> > > > > > > >> > > > > > >> > > > > > > > Cheers,
> > > > > > > >> > > > > > >> > > > > > > > Dmitri.
> > > > > > > >> > > > > > >> > > > > > > >
> > > > > > > >> > > > > > >> > > > > > > > On Wed, Jun 3, 2026 at 5:40 PM EJ
> > Wang
> > > > <
> > > > > > > >> > > > > > >> > > > > ej.wang.dev.interest@gmail.com
> > > > > > > >> > > > > > >> > > > > > >
> > > > > > > >> > > > > > >> > > > > > > > wrote:
> > > > > > > >> > > > > > >> > > > > > > >
> > > > > > > >> > > > > > >> > > > > > > > > Hi folks,
> > > > > > > >> > > > > > >> > > > > > > > >
> > > > > > > >> > > > > > >> > > > > > > > > Thanks again for the discussion
> > > > today.
> > > > > I
> > > > > > > >> updated
> > > > > > > >> > > the
> > > > > > > >> > > > > > sync
> > > > > > > >> > > > > > >> doc
> > > > > > > >> > > > > > >> > > > with
> > > > > > > >> > > > > > >> > > > > > > notes
> > > > > > > >> > > > > > >> > > > > > > > > from the third metrics
> > architecture
> > > > > sync:
> > > > > > > >> > > > > > >> > > > > > > > >
> > > > > > > >> > > > > > >> > > > > > > > >
> > > > > > > >> > > > > > >> > > > > > > > >
> > > > > > > >> > > > > > >> > > > > > > >
> > > > > > > >> > > > > > >> > > > > > >
> > > > > > > >> > > > > > >> > > > > >
> > > > > > > >> > > > > > >> > > > >
> > > > > > > >> > > > > > >> > > >
> > > > > > > >> > > > > > >> > >
> > > > > > > >> > > > > > >> >
> > > > > > > >> > > > > > >>
> > > > > > > >> > > > > >
> > > > > > > >> > > > >
> > > > > > > >> > >
> > > > > > > >> >
> > > > > > > >>
> > > > > >
> > > > >
> > > >
> >
> https://urldefense.com/v3/__https://docs.google.com/document/d/100h7c4damrUzVuquYbBHM0EvA4LSWuW2IT2dN_7nYVA/edit?tab=t.k96s2xyqr5u1__;!!Iz9xO38YGHZK!4rnlVIqBQQQ6xpGe1PMV58edOS_cGrKoqdfhw4qSp_JQCavvEjaxXk0OwGPnxyEcO60AzKQK_XDtp7H3DdkFXcrXrqCL-QE$
> > > > > > > >> > > > > > >> > > > > > > > >
> > > > > > > >> > > > > > >> > > > > > > > > A few highlights from today’s
> > > > > discussion:
> > > > > > > >> > > > > > >> > > > > > > > >
> > > > > > > >> > > > > > >> > > > > > > > >    - We clarified that Iceberg
> > > > metrics
> > > > > > > >> reporting
> > > > > > > >> > > can
> > > > > > > >> > > > > be
> > > > > > > >> > > > > > >> > > > interpreted
> > > > > > > >> > > > > > >> > > > > > > > either
> > > > > > > >> > > > > > >> > > > > > > > >    as sync or async handling
> from
> > > > > > Polaris’s
> > > > > > > >> > > > > perspective,
> > > > > > > >> > > > > > >> so
> > > > > > > >> > > > > > >> > > > Polaris
> > > > > > > >> > > > > > >> > > > > > > > should
> > > > > > > >> > > > > > >> > > > > > > > >    stay flexible as a platform
> > > > instead
> > > > > of
> > > > > > > >> baking
> > > > > > > >> > > one
> > > > > > > >> > > > > > >> handling
> > > > > > > >> > > > > > >> > > > model
> > > > > > > >> > > > > > >> > > > > > > into
> > > > > > > >> > > > > > >> > > > > > > > > the
> > > > > > > >> > > > > > >> > > > > > > > >    REST API behavior.
> > > > > > > >> > > > > > >> > > > > > > > >    - We aligned that the
> built-in
> > > > > > (battery)
> > > > > > > >> > > behavior
> > > > > > > >> > > > > for
> > > > > > > >> > > > > > >> > > metrics
> > > > > > > >> > > > > > >> > > > > > > emitting
> > > > > > > >> > > > > > >> > > > > > > > >    can stay simple:
> > no-op/log-only is
> > > > > > enough
> > > > > > > >> as
> > > > > > > >> > > the
> > > > > > > >> > > > > > >> default.
> > > > > > > >> > > > > > >> > > > > > > > >    - Durable metrics
> persistence
> > > > should
> > > > > > be
> > > > > > > >> > > treated as
> > > > > > > >> > > > > an
> > > > > > > >> > > > > > >> > > > > > implementation
> > > > > > > >> > > > > > >> > > > > > > > of
> > > > > > > >> > > > > > >> > > > > > > > >    the metrics reporting path,
> > not as
> > > > > the
> > > > > > > >> core
> > > > > > > >> > SPI
> > > > > > > >> > > > > > >> boundary.
> > > > > > > >> > > > > > >> > > > > > > > >    - The existing durable
> metrics
> > > > work
> > > > > > can be
> > > > > > > >> > > reviewed
> > > > > > > >> > > > > > as
> > > > > > > >> > > > > > >> a
> > > > > > > >> > > > > > >> > > > > reference
> > > > > > > >> > > > > > >> > > > > > > > >    implementation of the
> > reporting
> > > > SPI,
> > > > > > with
> > > > > > > >> > > > > > >> > > persistence-related
> > > > > > > >> > > > > > >> > > > > > logic
> > > > > > > >> > > > > > >> > > > > > > > kept
> > > > > > > >> > > > > > >> > > > > > > > >    self-contained under a
> metrics
> > > > > durable
> > > > > > > >> > > > > implementation
> > > > > > > >> > > > > > >> > module
> > > > > > > >> > > > > > >> > > > > > rather
> > > > > > > >> > > > > > >> > > > > > > > than
> > > > > > > >> > > > > > >> > > > > > > > >    scattered through core
> entity
> > > > > > persistence.
> > > > > > > >> > > > > > >> > > > > > > > >    - Dashboard/insights
> remains a
> > > > real
> > > > > > use
> > > > > > > >> case,
> > > > > > > >> > > but
> > > > > > > >> > > > > we
> > > > > > > >> > > > > > >> > agreed
> > > > > > > >> > > > > > >> > > to
> > > > > > > >> > > > > > >> > > > > > keep
> > > > > > > >> > > > > > >> > > > > > > it
> > > > > > > >> > > > > > >> > > > > > > > >    separate from the core
> metrics
> > > > > intake
> > > > > > > >> > > discussion
> > > > > > > >> > > > > for
> > > > > > > >> > > > > > >> now.
> > > > > > > >> > > > > > >> > > > > > > > >
> > > > > > > >> > > > > > >> > > > > > > > > I also did a quick source check
> > after
> > > > > the
> > > > > > > >> > meeting
> > > > > > > >> > > to
> > > > > > > >> > > > > > make
> > > > > > > >> > > > > > >> > sure
> > > > > > > >> > > > > > >> > > we
> > > > > > > >> > > > > > >> > > > > are
> > > > > > > >> > > > > > >> > > > > > > > > describing the current state
> > > > > accurately.
> > > > > > > >> > > > > > >> > > > > > > > >
> > > > > > > >> > > > > > >> > > > > > > > > Current state:
> > > > > > > >> > > > > > >> > > > > > > > >
> > > > > > > >> > > > > > >> > > > > > > > >    - Polaris already has a
> > metrics
> > > > > > reporting
> > > > > > > >> > hook:
> > > > > > > >> > > > > > >> > > > > > > > PolarisMetricsReporter.
> > > > > > > >> > > > > > >> > > > > > > > >    - The default implementation
> > is
> > > > > > > >> > > > > > DefaultMetricsReporter,
> > > > > > > >> > > > > > >> > > > selected
> > > > > > > >> > > > > > >> > > > > > by
> > > > > > > >> > > > > > >> > > > > > > > >
> > > > > > > >> > > polaris.iceberg-metrics.reporting.type=default. It
> > > > > > > >> > > > > is
> > > > > > > >> > > > > > >> > > > log-only,
> > > > > > > >> > > > > > >> > > > > > > > >    effectively quiet unless
> > metrics
> > > > > > logging
> > > > > > > >> is
> > > > > > > >> > > > > enabled.
> > > > > > > >> > > > > > >> > > > > > > > >    - There is also a
> > > > > > > >> PersistingMetricsReporter,
> > > > > > > >> > > > > selected
> > > > > > > >> > > > > > >> by
> > > > > > > >> > > > > > >> > > > > > > > >
> > > > > > > >> > > polaris.iceberg-metrics.reporting.type=persisting,
> > > > > > > >> > > > > > >> which
> > > > > > > >> > > > > > >> > > > > converts
> > > > > > > >> > > > > > >> > > > > > > > >    Iceberg scan/commit reports
> > into
> > > > > > Polaris
> > > > > > > >> > > metrics
> > > > > > > >> > > > > > >> records
> > > > > > > >> > > > > > >> > and
> > > > > > > >> > > > > > >> > > > > > writes
> > > > > > > >> > > > > > >> > > > > > > > > through
> > > > > > > >> > > > > > >> > > > > > > > >    PolarisMetricsManager ->
> > > > > > > >> MetricsPersistence.
> > > > > > > >> > > > > > >> > > > > > > > >    - MetricsPersistence
> currently
> > > > > exists
> > > > > > in
> > > > > > > >> the
> > > > > > > >> > > > > > >> persistence
> > > > > > > >> > > > > > >> > > layer
> > > > > > > >> > > > > > >> > > > > and
> > > > > > > >> > > > > > >> > > > > > > is
> > > > > > > >> > > > > > >> > > > > > > > >    inherited by
> BasePersistence.
> > > > > > > >> > > > > > >> > > > > > > > >
> > > > > > > >> > > > > > >> > > > > > > > > My read from the discussion is
> > that
> > > > the
> > > > > > > >> target
> > > > > > > >> > > > > boundary
> > > > > > > >> > > > > > >> > should
> > > > > > > >> > > > > > >> > > > not
> > > > > > > >> > > > > > >> > > > > be
> > > > > > > >> > > > > > >> > > > > > > > > MetricsPersistence as inherited
> > by
> > > > > > > >> > > BasePersistence.
> > > > > > > >> > > > > That
> > > > > > > >> > > > > > >> path
> > > > > > > >> > > > > > >> > > > > should
> > > > > > > >> > > > > > >> > > > > > be
> > > > > > > >> > > > > > >> > > > > > > > > decomposed as durable
> > implementation
> > > > > > detail
> > > > > > > >> > (taken
> > > > > > > >> > > > > care
> > > > > > > >> > > > > > >> of by
> > > > > > > >> > > > > > >> > > > > > > > >
> > > > > > > >> >
> > > > > > > >>
> > > > > >
> > > > >
> > > >
> >
> https://urldefense.com/v3/__https://github.com/apache/polaris/pull/4397__;!!Iz9xO38YGHZK!4rnlVIqBQQQ6xpGe1PMV58edOS_cGrKoqdfhw4qSp_JQCavvEjaxXk0OwGPnxyEcO60AzKQK_XDtp7H3DdkFXcrXuu5jefU$
> > > > > > > >> ).
> > > > > > > >> > The
> > > > > > > >> > > > > > stable
> > > > > > > >> > > > > > >> > > > extension
> > > > > > > >> > > > > > >> > > > > > > point
> > > > > > > >> > > > > > >> > > > > > > > > should instead be the metrics
> > > > > > > >> reporting/emitting
> > > > > > > >> > > > > > boundary.
> > > > > > > >> > > > > > >> > > > > > > > >
> > > > > > > >> > > > > > >> > > > > > > > > Concretely, I think the next
> > proposal
> > > > > > should
> > > > > > > >> be
> > > > > > > >> > > shaped
> > > > > > > >> > > > > > >> like
> > > > > > > >> > > > > > >> > > this:
> > > > > > > >> > > > > > >> > > > > > > > >
> > > > > > > >> > > > > > >> > > > > > > > >    1.
> > > > > > > >> > > > > > >> > > > > > > > >
> > > > > > > >> > > > > > >> > > > > > > > >    Define the proper metrics
> > > > reporting
> > > > > > SPI
> > > > > > > >> at an
> > > > > > > >> > > > > > optional
> > > > > > > >> > > > > > >> > > Iceberg
> > > > > > > >> > > > > > >> > > > > > > metrics
> > > > > > > >> > > > > > >> > > > > > > > >    extension API layer.
> > > > > > > >> > > > > > >> > > > > > > > >
> > > > > > > >> > > > > > >> > > > > > > > >    Example direction:
> > > > > > > >> > > > > > >> > > > > > > > >
> > > > > > > >> > > > > > >> > > > > > > > >    public interface
> > > > > > IcebergMetricsReporter {
> > > > > > > >> > > > > > >> > > > > > > > >      void
> > > > > > > >> > reportMetric(IcebergMetricsReportContext
> > > > > > > >> > > > > > >> context,
> > > > > > > >> > > > > > >> > > > > > > > > MetricsReport report);
> > > > > > > >> > > > > > >> > > > > > > > >    }
> > > > > > > >> > > > > > >> > > > > > > > >
> > > > > > > >> > > > > > >> > > > > > > > >    The context would carry the
> > small
> > > > > > > >> > > Polaris-resolved
> > > > > > > >> > > > > > >> > envelope:
> > > > > > > >> > > > > > >> > > > > > > > >    catalog/table identity,
> report
> > > > type,
> > > > > > > >> received
> > > > > > > >> > > > > > >> timestamp,
> > > > > > > >> > > > > > >> > and
> > > > > > > >> > > > > > >> > > > > > > > > request/trace
> > > > > > > >> > > > > > >> > > > > > > > >    context if available. The
> raw
> > > > > Iceberg
> > > > > > > >> > > MetricsReport
> > > > > > > >> > > > > > >> > remains
> > > > > > > >> > > > > > >> > > > the
> > > > > > > >> > > > > > >> > > > > > > > payload.
> > > > > > > >> > > > > > >> > > > > > > > >    2.
> > > > > > > >> > > > > > >> > > > > > > > >
> > > > > > > >> > > > > > >> > > > > > > > >    Keep runtime/service as the
> > REST
> > > > > > ingestion
> > > > > > > >> > and
> > > > > > > >> > > > > wiring
> > > > > > > >> > > > > > >> > layer.
> > > > > > > >> > > > > > >> > > > > > > > >
> > > > > > > >> > > > > > >> > > > > > > > >    The REST handler still
> > resolves
> > > > the
> > > > > > table
> > > > > > > >> and
> > > > > > > >> > > > > > performs
> > > > > > > >> > > > > > >> > authz
> > > > > > > >> > > > > > >> > > > > > before
> > > > > > > >> > > > > > >> > > > > > > > >    accepting the report. After
> > that,
> > > > it
> > > > > > calls
> > > > > > > >> > the
> > > > > > > >> > > > > > selected
> > > > > > > >> > > > > > >> > > > > reporting
> > > > > > > >> > > > > > >> > > > > > > > >    implementation.
> > > > > > > >> > > > > > >> > > > > > > > >    3.
> > > > > > > >> > > > > > >> > > > > > > > >
> > > > > > > >> > > > > > >> > > > > > > > >    Keep the battery default
> > > > > > no-op/log-only.
> > > > > > > >> > > > > > >> > > > > > > > >
> > > > > > > >> > > > > > >> > > > > > > > >    This preserves an out-of-box
> > safe
> > > > > > default
> > > > > > > >> and
> > > > > > > >> > > > > avoids
> > > > > > > >> > > > > > >> > > > requiring a
> > > > > > > >> > > > > > >> > > > > > > > durable
> > > > > > > >> > > > > > >> > > > > > > > >    metrics store for every
> > Polaris
> > > > > > > >> deployment.
> > > > > > > >> > > > > > >> > > > > > > > >    4.
> > > > > > > >> > > > > > >> > > > > > > > >
> > > > > > > >> > > > > > >> > > > > > > > >    Move durable JDBC metrics
> > into a
> > > > > > > >> > self-contained
> > > > > > > >> > > > > > >> > > implementation
> > > > > > > >> > > > > > >> > > > > of
> > > > > > > >> > > > > > >> > > > > > > the
> > > > > > > >> > > > > > >> > > > > > > > >    reporting SPI.
> > > > > > > >> > > > > > >> > > > > > > > >
> > > > > > > >> > > > > > >> > > > > > > > >    That implementation can own
> > its
> > > > > > schema,
> > > > > > > >> > > bootstrap,
> > > > > > > >> > > > > > >> > > retention,
> > > > > > > >> > > > > > >> > > > > and
> > > > > > > >> > > > > > >> > > > > > > read
> > > > > > > >> > > > > > >> > > > > > > > >    API support. It should not
> > define
> > > > > the
> > > > > > core
> > > > > > > >> > > > > reporting
> > > > > > > >> > > > > > >> SPI
> > > > > > > >> > > > > > >> > > > > boundary,
> > > > > > > >> > > > > > >> > > > > > > and
> > > > > > > >> > > > > > >> > > > > > > > > it
> > > > > > > >> > > > > > >> > > > > > > > >    should not require metrics
> > > > > > persistence to
> > > > > > > >> > > remain
> > > > > > > >> > > > > > >> inherited
> > > > > > > >> > > > > > >> > > > from
> > > > > > > >> > > > > > >> > > > > > > > >    BasePersistence.
> > > > > > > >> > > > > > >> > > > > > > > >    5.
> > > > > > > >> > > > > > >> > > > > > > > >
> > > > > > > >> > > > > > >> > > > > > > > >    Treat async/event-backed
> > handling
> > > > as
> > > > > > > >> another
> > > > > > > >> > > > > > >> > implementation
> > > > > > > >> > > > > > >> > > of
> > > > > > > >> > > > > > >> > > > > the
> > > > > > > >> > > > > > >> > > > > > > > same
> > > > > > > >> > > > > > >> > > > > > > > >    reporting SPI.
> > > > > > > >> > > > > > >> > > > > > > > >
> > > > > > > >> > > > > > >> > > > > > > > >    For example, an event-backed
> > > > > reporter
> > > > > > > >> could
> > > > > > > >> > > > > enqueue a
> > > > > > > >> > > > > > >> > > metrics
> > > > > > > >> > > > > > >> > > > > > event
> > > > > > > >> > > > > > >> > > > > > > > and
> > > > > > > >> > > > > > >> > > > > > > > >    let listeners handle durable
> > > > storage
> > > > > > or
> > > > > > > >> other
> > > > > > > >> > > > > sinks.
> > > > > > > >> > > > > > >> If we
> > > > > > > >> > > > > > >> > > > later
> > > > > > > >> > > > > > >> > > > > > > need
> > > > > > > >> > > > > > >> > > > > > > > a
> > > > > > > >> > > > > > >> > > > > > > > >    replaceable queue engine,
> that
> > > > seems
> > > > > > like
> > > > > > > >> a
> > > > > > > >> > > shared
> > > > > > > >> > > > > > >> > > > event/metrics
> > > > > > > >> > > > > > >> > > > > > > > > substrate
> > > > > > > >> > > > > > >> > > > > > > > >    topic rather than a
> > metrics-only
> > > > > > > >> requirement.
> > > > > > > >> > > > > > >> > > > > > > > >
> > > > > > > >> > > > > > >> > > > > > > > > This framing lets us keep the
> > REST
> > > > > > metrics
> > > > > > > >> > > endpoint
> > > > > > > >> > > > > > >> simple,
> > > > > > > >> > > > > > >> > > > > preserve
> > > > > > > >> > > > > > >> > > > > > > the
> > > > > > > >> > > > > > >> > > > > > > > > current default behavior,
> support
> > > > > durable
> > > > > > > >> > metrics
> > > > > > > >> > > > > users,
> > > > > > > >> > > > > > >> and
> > > > > > > >> > > > > > >> > > > still
> > > > > > > >> > > > > > >> > > > > > > leave
> > > > > > > >> > > > > > >> > > > > > > > > room for async/event-backed or
> > > > > > > >> > > external-queue-based
> > > > > > > >> > > > > > >> > > > > implementations.
> > > > > > > >> > > > > > >> > > > > > > > >
> > > > > > > >> > > > > > >> > > > > > > > > I think the main follow-up
> > questions
> > > > > are:
> > > > > > > >> > > > > > >> > > > > > > > >
> > > > > > > >> > > > > > >> > > > > > > > >    - Do folks agree that the
> > stable
> > > > SPI
> > > > > > > >> boundary
> > > > > > > >> > > > > should
> > > > > > > >> > > > > > be
> > > > > > > >> > > > > > >> > > > metrics
> > > > > > > >> > > > > > >> > > > > > > > >    reporting/emitting, not
> > metrics
> > > > > > > >> persistence?
> > > > > > > >> > > > > > >> > > > > > > > >    - Does an optional Iceberg
> > metrics
> > > > > > > >> extension
> > > > > > > >> > > API
> > > > > > > >> > > > > > layer
> > > > > > > >> > > > > > >> > sound
> > > > > > > >> > > > > > >> > > > > like
> > > > > > > >> > > > > > >> > > > > > > the
> > > > > > > >> > > > > > >> > > > > > > > >    right home for this SPI?
> > > > > > > >> > > > > > >> > > > > > > > >    - Should the current durable
> > > > metrics
> > > > > > work
> > > > > > > >> be
> > > > > > > >> > > > > reframed
> > > > > > > >> > > > > > >> as a
> > > > > > > >> > > > > > >> > > > > durable
> > > > > > > >> > > > > > >> > > > > > > > JDBC
> > > > > > > >> > > > > > >> > > > > > > > >    reference implementation of
> > that
> > > > > SPI?
> > > > > > > >> > > > > > >> > > > > > > > >    - What is the smallest PR
> > sequence
> > > > > to
> > > > > > get
> > > > > > > >> > there
> > > > > > > >> > > > > > without
> > > > > > > >> > > > > > >> > > > blocking
> > > > > > > >> > > > > > >> > > > > > the
> > > > > > > >> > > > > > >> > > > > > > > >    current metrics work
> > > > unnecessarily?
> > > > > > > >> > > > > > >> > > > > > > > >
> > > > > > > >> > > > > > >> > > > > > > > > Thanks,
> > > > > > > >> > > > > > >> > > > > > > > > -ej
> > > > > > > >> > > > > > >> > > > > > > > >
> > > > > > > >> > > > > > >> > > > > > > > > On Fri, May 15, 2026 at 5:16 PM
> > > > Dmitri
> > > > > > > >> > > Bourlatchkov <
> > > > > > > >> > > > > > >> > > > > > dimas@apache.org>
> > > > > > > >> > > > > > >> > > > > > > > > wrote:
> > > > > > > >> > > > > > >> > > > > > > > >
> > > > > > > >> > > > > > >> > > > > > > > > > Hi JB,
> > > > > > > >> > > > > > >> > > > > > > > > >
> > > > > > > >> > > > > > >> > > > > > > > > > Could you set up another
> > meeting,
> > > > > > please?
> > > > > > > >> Same
> > > > > > > >> > > time
> > > > > > > >> > > > > on
> > > > > > > >> > > > > > >> > > > Wednesday
> > > > > > > >> > > > > > >> > > > > as
> > > > > > > >> > > > > > >> > > > > > > > last
> > > > > > > >> > > > > > >> > > > > > > > > > time... I hope it works for
> > > > everyone.
> > > > > > > >> > > > > > >> > > > > > > > > >
> > > > > > > >> > > > > > >> > > > > > > > > > Cheers,
> > > > > > > >> > > > > > >> > > > > > > > > > Dmitri.
> > > > > > > >> > > > > > >> > > > > > > > > >
> > > > > > > >> > > > > > >> > > > > > > > > > On Fri, May 15, 2026 at
> 8:06 PM
> > > > Yufei
> > > > > > Gu <
> > > > > > > >> > > > > > >> > > flyrain000@gmail.com
> > > > > > > >> > > > > > >> > > > >
> > > > > > > >> > > > > > >> > > > > > > wrote:
> > > > > > > >> > > > > > >> > > > > > > > > >
> > > > > > > >> > > > > > >> > > > > > > > > > > +1 on another sync call
> next
> > > > week.
> > > > > > > >> > > > > > >> > > > > > > > > > >
> > > > > > > >> > > > > > >> > > > > > > > > > > Yufei
> > > > > > > >> > > > > > >> > > > > > > > > > >
> > > > > > > >> > > > > > >> > > > > > > > > > >
> > > > > > > >> > > > > > >> > > > > > > > > > > On Fri, May 15, 2026 at
> > 4:52 PM
> > > > > > Dmitri
> > > > > > > >> > > > > Bourlatchkov
> > > > > > > >> > > > > > <
> > > > > > > >> > > > > > >> > > > > > > > dimas@apache.org>
> > > > > > > >> > > > > > >> > > > > > > > > > > wrote:
> > > > > > > >> > > > > > >> > > > > > > > > > >
> > > > > > > >> > > > > > >> > > > > > > > > > > > Hi All,
> > > > > > > >> > > > > > >> > > > > > > > > > > >
> > > > > > > >> > > > > > >> > > > > > > > > > > > WDYT about another sync
> > call
> > > > next
> > > > > > week?
> > > > > > > >> > > > > > >> > > > > > > > > > > >
> > > > > > > >> > > > > > >> > > > > > > > > > > > Thanks,
> > > > > > > >> > > > > > >> > > > > > > > > > > > Dmitri.
> > > > > > > >> > > > > > >> > > > > > > > > > > >
> > > > > > > >> > > > > > >> > > > > > > > > > > > On Wed, May 6, 2026 at
> > 5:29 PM
> > > > > > Dmitri
> > > > > > > >> > > > > > Bourlatchkov <
> > > > > > > >> > > > > > >> > > > > > > > dimas@apache.org
> > > > > > > >> > > > > > >> > > > > > > > > >
> > > > > > > >> > > > > > >> > > > > > > > > > > > wrote:
> > > > > > > >> > > > > > >> > > > > > > > > > > >
> > > > > > > >> > > > > > >> > > > > > > > > > > > > Hi EJ,
> > > > > > > >> > > > > > >> > > > > > > > > > > > >
> > > > > > > >> > > > > > >> > > > > > > > > > > > > Thanks for the summary!
> > It
> > > > > covers
> > > > > > > >> what
> > > > > > > >> > we
> > > > > > > >> > > > > > >> discussed
> > > > > > > >> > > > > > >> > in
> > > > > > > >> > > > > > >> > > > the
> > > > > > > >> > > > > > >> > > > > > > > meeting
> > > > > > > >> > > > > > >> > > > > > > > > > very
> > > > > > > >> > > > > > >> > > > > > > > > > > > > well, IMHO.
> > > > > > > >> > > > > > >> > > > > > > > > > > > >
> > > > > > > >> > > > > > >> > > > > > > > > > > > > Looking forward to
> > concrete
> > > > PRs
> > > > > > :)
> > > > > > > >> > > > > > >> > > > > > > > > > > > >
> > > > > > > >> > > > > > >> > > > > > > > > > > > > Cheers,
> > > > > > > >> > > > > > >> > > > > > > > > > > > > Dmitri.
> > > > > > > >> > > > > > >> > > > > > > > > > > > >
> > > > > > > >> > > > > > >> > > > > > > > > > > > > On Wed, May 6, 2026 at
> > > > 5:08 PM
> > > > > EJ
> > > > > > > >> Wang <
> > > > > > > >> > > > > > >> > > > > > > > > >
> ej.wang.dev.interest@gmail.com
> > > > > > > >> > > > > > >> > > > > > > > > > > >
> > > > > > > >> > > > > > >> > > > > > > > > > > > > wrote:
> > > > > > > >> > > > > > >> > > > > > > > > > > > >
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> Hi folks,
> > > > > > > >> > > > > > >> > > > > > > > > > > > >>
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> We had a community
> sync
> > > > > earlier,
> > > > > > > >> thanks
> > > > > > > >> > > JB
> > > > > > > >> > > > > for
> > > > > > > >> > > > > > >> > > > scheduling
> > > > > > > >> > > > > > >> > > > > > it.
> > > > > > > >> > > > > > >> > > > > > > > > Notes
> > > > > > > >> > > > > > >> > > > > > > > > > > from
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> the first metrics
> > > > architecture
> > > > > > sync
> > > > > > > >> > (May
> > > > > > > >> > > 6,
> > > > > > > >> > > > > > >> 10-11am
> > > > > > > >> > > > > > >> > > PT).
> > > > > > > >> > > > > > >> > > > > > > > > Discussion
> > > > > > > >> > > > > > >> > > > > > > > > > > doc
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> with per-section
> status:
> > > > > > > >> > > > > > >> > > > > > > > > > > > >>
> > > > > > > >> > > > > > >> > > > > > > > > > > > >>
> > > > > > > >> > > > > > >> > > > > > > > > > > >
> > > > > > > >> > > > > > >> > > > > > > > > > >
> > > > > > > >> > > > > > >> > > > > > > > > >
> > > > > > > >> > > > > > >> > > > > > > > >
> > > > > > > >> > > > > > >> > > > > > > >
> > > > > > > >> > > > > > >> > > > > > >
> > > > > > > >> > > > > > >> > > > > >
> > > > > > > >> > > > > > >> > > > >
> > > > > > > >> > > > > > >> > > >
> > > > > > > >> > > > > > >> > >
> > > > > > > >> > > > > > >> >
> > > > > > > >> > > > > > >>
> > > > > > > >> > > > > >
> > > > > > > >> > > > >
> > > > > > > >> > >
> > > > > > > >> >
> > > > > > > >>
> > > > > >
> > > > >
> > > >
> >
> https://urldefense.com/v3/__https://docs.google.com/document/d/100h7c4damrUzVuquYbBHM0EvA4LSWuW2IT2dN_7nYVA/edit?tab=t.0__;!!Iz9xO38YGHZK!4rnlVIqBQQQ6xpGe1PMV58edOS_cGrKoqdfhw4qSp_JQCavvEjaxXk0OwGPnxyEcO60AzKQK_XDtp7H3DdkFXcrXgIddeUs$
> > > > > > > >> > > > > > >> > > > > > > > > > > > >>
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> *The meeting covered
> > both
> > > > > topics
> > > > > > > >> from
> > > > > > > >> > the
> > > > > > > >> > > > > doc.
> > > > > > > >> > > > > > >> > > > > > Direction-level
> > > > > > > >> > > > > > >> > > > > > > > > > > alignment
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> was reached on the
> > headline
> > > > > > pieces;
> > > > > > > >> > > details
> > > > > > > >> > > > > > >> remain
> > > > > > > >> > > > > > >> > for
> > > > > > > >> > > > > > >> > > > PR
> > > > > > > >> > > > > > >> > > > > > > review
> > > > > > > >> > > > > > >> > > > > > > > > or
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> follow-up sessions.*
> > > > > > > >> > > > > > >> > > > > > > > > > > > >>
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> *Topic 1 — Persistence
> > > > schema
> > > > > > > >> redesign*
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> Idea-level alignment
> on
> > > > > > > >> consolidating
> > > > > > > >> > > > > per-type
> > > > > > > >> > > > > > >> > tables
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> (scan_metrics_report,
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> commit_metrics_report)
> > into
> > > > a
> > > > > > single
> > > > > > > >> > > > > > >> metrics_report
> > > > > > > >> > > > > > >> > > > table.
> > > > > > > >> > > > > > >> > > > > > The
> > > > > > > >> > > > > > >> > > > > > > > > > > > motivating
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> cost is the surface
> area
> > > > added
> > > > > > by
> > > > > > > >> every
> > > > > > > >> > > new
> > > > > > > >> > > > > > >> metric
> > > > > > > >> > > > > > >> > > type
> > > > > > > >> > > > > > >> > > > > > today:
> > > > > > > >> > > > > > >> > > > > > > > new
> > > > > > > >> > > > > > >> > > > > > > > > > > > table,
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> SPI method, record
> > class,
> > > > > model,
> > > > > > > >> > > converter,
> > > > > > > >> > > > > > >> schema
> > > > > > > >> > > > > > >> > > > > > migration.
> > > > > > > >> > > > > > >> > > > > > > > > > > > >>
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> Most schema details
> are
> > > > > > deferred to
> > > > > > > >> the
> > > > > > > >> > > > > schema
> > > > > > > >> > > > > > >> PR. A
> > > > > > > >> > > > > > >> > > few
> > > > > > > >> > > > > > >> > > > > > > > specific
> > > > > > > >> > > > > > >> > > > > > > > > > > points
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> came up:
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> •
> >  metric_schema_version:
> > > > > Yufei
> > > > > > > >> > prefers
> > > > > > > >> > > > > > dropping
> > > > > > > >> > > > > > >> > it,
> > > > > > > >> > > > > > >> > > > > since
> > > > > > > >> > > > > > >> > > > > > > > there
> > > > > > > >> > > > > > >> > > > > > > > > is
> > > > > > > >> > > > > > >> > > > > > > > > > > no
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> spec-level concept of
> > > > metrics
> > > > > > > >> > versioning
> > > > > > > >> > > > > today
> > > > > > > >> > > > > > >> and
> > > > > > > >> > > > > > >> > it
> > > > > > > >> > > > > > >> > > is
> > > > > > > >> > > > > > >> > > > > > hard
> > > > > > > >> > > > > > >> > > > > > > to
> > > > > > > >> > > > > > >> > > > > > > > > > > define
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> unilaterally. Robert
> > prefers
> > > > > > keeping
> > > > > > > >> > it,
> > > > > > > >> > > > > given
> > > > > > > >> > > > > > >> IRC
> > > > > > > >> > > > > > >> > v2
> > > > > > > >> > > > > > >> > > is
> > > > > > > >> > > > > > >> > > > > > > coming
> > > > > > > >> > > > > > >> > > > > > > > > and
> > > > > > > >> > > > > > >> > > > > > > > > > > the
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> schema should be
> > considered
> > > > > > against
> > > > > > > >> its
> > > > > > > >> > > > > likely
> > > > > > > >> > > > > > >> > shape;
> > > > > > > >> > > > > > >> > > > > Robert
> > > > > > > >> > > > > > >> > > > > > > > also
> > > > > > > >> > > > > > >> > > > > > > > > > > raised
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> how to differentiate
> > various
> > > > > > payload
> > > > > > > >> > > formats
> > > > > > > >> > > > > if
> > > > > > > >> > > > > > >> any.
> > > > > > > >> > > > > > >> > > > EJ's
> > > > > > > >> > > > > > >> > > > > > read
> > > > > > > >> > > > > > >> > > > > > > > is
> > > > > > > >> > > > > > >> > > > > > > > > > that
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> this
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> is a two-way-door
> > decision.
> > > > We
> > > > > > can
> > > > > > > >> > start
> > > > > > > >> > > > > > without
> > > > > > > >> > > > > > >> the
> > > > > > > >> > > > > > >> > > > > field,
> > > > > > > >> > > > > > >> > > > > > > and
> > > > > > > >> > > > > > >> > > > > > > > if
> > > > > > > >> > > > > > >> > > > > > > > > > IRC
> > > > > > > >> > > > > > >> > > > > > > > > > > > v2
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> changes the shape we
> > would
> > > > > > likely
> > > > > > > >> roll
> > > > > > > >> > a
> > > > > > > >> > > > > > >> > corresponding
> > > > > > > >> > > > > > >> > > > new
> > > > > > > >> > > > > > >> > > > > > > > schema
> > > > > > > >> > > > > > >> > > > > > > > > > > > anyway,
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> which is not
> > particularly
> > > > > > costly.
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> •   Payload format:
> > Robert
> > > > > > pointed
> > > > > > > >> out
> > > > > > > >> > > that
> > > > > > > >> > > > > > >> future
> > > > > > > >> > > > > > >> > > > formats
> > > > > > > >> > > > > > >> > > > > > > > beyond
> > > > > > > >> > > > > > >> > > > > > > > > > JSON
> > > > > > > >> > > > > > >> > > > > > > > > > > > may
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> be worth supporting.
> The
> > > > exact
> > > > > > > >> shape is
> > > > > > > >> > > > > > deferred
> > > > > > > >> > > > > > >> to
> > > > > > > >> > > > > > >> > > the
> > > > > > > >> > > > > > >> > > > > > schema
> > > > > > > >> > > > > > >> > > > > > > > > > > > discussion.
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> •   Partition
> strategy:
> > > > Anand
> > > > > > > >> suggested
> > > > > > > >> > > > > monthly
> > > > > > > >> > > > > > >> > > > > partitioning
> > > > > > > >> > > > > > >> > > > > > > > based
> > > > > > > >> > > > > > >> > > > > > > > > > on
> > > > > > > >> > > > > > >> > > > > > > > > > > > his
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> experience as
> > potentially
> > > > > > helpful at
> > > > > > > >> > > scale.
> > > > > > > >> > > > > > >> > > > > > > > > > > > >>
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> *Topic 2 — Where
> metrics
> > > > > > ingestion
> > > > > > > >> and
> > > > > > > >> > > > > storage
> > > > > > > >> > > > > > >> > belong*
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> Idea-level alignment
> > that
> > > > > > metrics
> > > > > > > >> > should
> > > > > > > >> > > be a
> > > > > > > >> > > > > > >> > > separated
> > > > > > > >> > > > > > >> > > > > SPI
> > > > > > > >> > > > > > >> > > > > > > from
> > > > > > > >> > > > > > >> > > > > > > > > the
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> entity
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> persistence stack. Two
> > > > reasons
> > > > > > > >> > surfaced:
> > > > > > > >> > > (a)
> > > > > > > >> > > > > > >> > workloads
> > > > > > > >> > > > > > >> > > > and
> > > > > > > >> > > > > > >> > > > > > > > > > capability
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> requirements diverge
> > enough
> > > > > that
> > > > > > > >> > coupling
> > > > > > > >> > > > > them
> > > > > > > >> > > > > > >> > creates
> > > > > > > >> > > > > > >> > > > > > > > artificial
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> constraints, and (b)
> > admin
> > > > > > > >> experience
> > > > > > > >> > > > > improves
> > > > > > > >> > > > > > >> when
> > > > > > > >> > > > > > >> > > > > metrics
> > > > > > > >> > > > > > >> > > > > > > has
> > > > > > > >> > > > > > >> > > > > > > > > its
> > > > > > > >> > > > > > >> > > > > > > > > > > own
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> bootstrap, retention,
> > and
> > > > > > lifecycle.
> > > > > > > >> > > Dmitri
> > > > > > > >> > > > > > noted
> > > > > > > >> > > > > > >> > > > Polaris
> > > > > > > >> > > > > > >> > > > > > > being
> > > > > > > >> > > > > > >> > > > > > > > a
> > > > > > > >> > > > > > >> > > > > > > > > > > > platform
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> should have the
> > flexibility
> > > > to
> > > > > > > >> support
> > > > > > > >> > > > > > different
> > > > > > > >> > > > > > >> > > > > persistence
> > > > > > > >> > > > > > >> > > > > > > > > > backends
> > > > > > > >> > > > > > >> > > > > > > > > > > > per
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> concern, and pointed
> to
> > a
> > > > > > concrete
> > > > > > > >> next
> > > > > > > >> > > step
> > > > > > > >> > > > > of
> > > > > > > >> > > > > > >> > > > separating
> > > > > > > >> > > > > > >> > > > > > the
> > > > > > > >> > > > > > >> > > > > > > > > JDBC
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> bootstrap for metrics
> > from
> > > > the
> > > > > > > >> > metastore
> > > > > > > >> > > > > > >> bootstrap.
> > > > > > > >> > > > > > >> > > > Robert
> > > > > > > >> > > > > > >> > > > > > > > > proposed
> > > > > > > >> > > > > > >> > > > > > > > > > an
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> additional UX
> extension:
> > > > > detect
> > > > > > an
> > > > > > > >> > > > > > unbootstrapped
> > > > > > > >> > > > > > >> > > > metrics
> > > > > > > >> > > > > > >> > > > > > > store
> > > > > > > >> > > > > > >> > > > > > > > on
> > > > > > > >> > > > > > >> > > > > > > > > > > first
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> use and auto-bootstrap
> > > > rather
> > > > > > than
> > > > > > > >> > > requiring
> > > > > > > >> > > > > an
> > > > > > > >> > > > > > >> > > explicit
> > > > > > > >> > > > > > >> > > > > > > manual
> > > > > > > >> > > > > > >> > > > > > > > > > > > bootstrap
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> step.
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> The meeting also
> > confirmed
> > > > > that
> > > > > > > >> Polaris
> > > > > > > >> > > > > metrics
> > > > > > > >> > > > > > >> can
> > > > > > > >> > > > > > >> > > > start
> > > > > > > >> > > > > > >> > > > > > > small
> > > > > > > >> > > > > > >> > > > > > > > > and
> > > > > > > >> > > > > > >> > > > > > > > > > > stay
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> Iceberg-focused.
> Naming
> > and
> > > > > > > >> persistence
> > > > > > > >> > > > > schema
> > > > > > > >> > > > > > >> can
> > > > > > > >> > > > > > >> > > lean
> > > > > > > >> > > > > > >> > > > > > > > > > > > Iceberg-specific.
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> If a future expansion
> to
> > > > > > > >> generic-table
> > > > > > > >> > > > > metrics
> > > > > > > >> > > > > > or
> > > > > > > >> > > > > > >> > > > > > operational
> > > > > > > >> > > > > > >> > > > > > > > > > metrics
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> arrives, an
> abstraction
> > > > layer
> > > > > > can be
> > > > > > > >> > > built on
> > > > > > > >> > > > > > >> top of
> > > > > > > >> > > > > > >> > > the
> > > > > > > >> > > > > > >> > > > > > > Iceberg
> > > > > > > >> > > > > > >> > > > > > > > > > > metrics
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> reporter at that
> point.
> > > > Robert
> > > > > > > >> remains
> > > > > > > >> > > on the
> > > > > > > >> > > > > > >> fence
> > > > > > > >> > > > > > >> > > and
> > > > > > > >> > > > > > >> > > > > > would
> > > > > > > >> > > > > > >> > > > > > > > > prefer
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> something more generic
> > but
> > > > did
> > > > > > not
> > > > > > > >> > block
> > > > > > > >> > > the
> > > > > > > >> > > > > > >> > > direction;
> > > > > > > >> > > > > > >> > > > > > > Dmitri's
> > > > > > > >> > > > > > >> > > > > > > > > > read
> > > > > > > >> > > > > > >> > > > > > > > > > > > was
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> that the proposed
> > framework
> > > > > > already
> > > > > > > >> has
> > > > > > > >> > > > > enough
> > > > > > > >> > > > > > >> > > > flexibility
> > > > > > > >> > > > > > >> > > > > > to
> > > > > > > >> > > > > > >> > > > > > > > > absorb
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> future
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> expansion.
> > > > > > > >> > > > > > >> > > > > > > > > > > > >>
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> The Trade-offs and
> > Proposed
> > > > > > > >> structure
> > > > > > > >> > > > > sections
> > > > > > > >> > > > > > in
> > > > > > > >> > > > > > >> > the
> > > > > > > >> > > > > > >> > > > doc
> > > > > > > >> > > > > > >> > > > > > were
> > > > > > > >> > > > > > >> > > > > > > > not
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> reviewed
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> in detail. They remain
> > open
> > > > > for
> > > > > > > >> either
> > > > > > > >> > > the
> > > > > > > >> > > > > next
> > > > > > > >> > > > > > >> sync
> > > > > > > >> > > > > > >> > > or
> > > > > > > >> > > > > > >> > > > PR
> > > > > > > >> > > > > > >> > > > > > > > review.
> > > > > > > >> > > > > > >> > > > > > > > > > > > >>
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> *Cross-cutting
> > alignment —
> > > > > > > >> > > battery-included
> > > > > > > >> > > > > > plus
> > > > > > > >> > > > > > >> > > > > pluggable*
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> A common philosophy
> > emerged
> > > > > > from the
> > > > > > > >> > > > > > discussion.
> > > > > > > >> > > > > > >> EJ
> > > > > > > >> > > > > > >> > > > > > summarized
> > > > > > > >> > > > > > >> > > > > > > > it
> > > > > > > >> > > > > > >> > > > > > > > > > as:
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> Polaris should
> provide a
> > > > > > > >> > > battery-included UX
> > > > > > > >> > > > > > for
> > > > > > > >> > > > > > >> > > > beginners
> > > > > > > >> > > > > > >> > > > > > and
> > > > > > > >> > > > > > >> > > > > > > > the
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> flexibility for
> advanced
> > > > users
> > > > > > to
> > > > > > > >> swap
> > > > > > > >> > > the
> > > > > > > >> > > > > > >> included
> > > > > > > >> > > > > > >> > > > > battery
> > > > > > > >> > > > > > >> > > > > > > for
> > > > > > > >> > > > > > >> > > > > > > > > > > > something
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> more powerful or
> > tailored to
> > > > > > their
> > > > > > > >> use
> > > > > > > >> > > case.
> > > > > > > >> > > > > > The
> > > > > > > >> > > > > > >> SPI
> > > > > > > >> > > > > > >> > > > > design
> > > > > > > >> > > > > > >> > > > > > > > needs
> > > > > > > >> > > > > > >> > > > > > > > > to
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> enable
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> both.
> > > > > > > >> > > > > > >> > > > > > > > > > > > >>
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> The inputs that shaped
> > this
> > > > > > framing:
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> •   Anand described
> how
> > his
> > > > > team
> > > > > > > >> uses
> > > > > > > >> > the
> > > > > > > >> > > > > > current
> > > > > > > >> > > > > > >> > > > metrics
> > > > > > > >> > > > > > >> > > > > > > > > > persistence
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> (three metrics
> > consumers in
> > > > > > v1.4).
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> •   Yufei raised
> > Grafana and
> > > > > > > >> dashboard
> > > > > > > >> > > > > > >> integrations
> > > > > > > >> > > > > > >> > > as a
> > > > > > > >> > > > > > >> > > > > > > > > destination
> > > > > > > >> > > > > > >> > > > > > > > > > > use
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> case beyond the
> default.
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> •   Robert called out
> > that
> > > > the
> > > > > > > >> current
> > > > > > > >> > > design
> > > > > > > >> > > > > > is
> > > > > > > >> > > > > > >> > more
> > > > > > > >> > > > > > >> > > > > > > > > JDBC-focused.
> > > > > > > >> > > > > > >> > > > > > > > > > > > >>
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> Two concrete
> instances:
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> •   Async metrics
> > intake:
> > > > > > Yufei's
> > > > > > > >> > initial
> > > > > > > >> > > > > > >> position
> > > > > > > >> > > > > > >> > was
> > > > > > > >> > > > > > >> > > > > that
> > > > > > > >> > > > > > >> > > > > > > > async
> > > > > > > >> > > > > > >> > > > > > > > > > > should
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> largely live on the
> > producer
> > > > > > side
> > > > > > > >> and
> > > > > > > >> > > there
> > > > > > > >> > > > > is
> > > > > > > >> > > > > > >> not
> > > > > > > >> > > > > > >> > > much
> > > > > > > >> > > > > > >> > > > > > > Polaris
> > > > > > > >> > > > > > >> > > > > > > > > can
> > > > > > > >> > > > > > >> > > > > > > > > > > do.
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> Robert suggested a
> > > > > Polaris-side
> > > > > > > >> default
> > > > > > > >> > > is
> > > > > > > >> > > > > > doable
> > > > > > > >> > > > > > >> > via
> > > > > > > >> > > > > > >> > > > > > Vert.x.
> > > > > > > >> > > > > > >> > > > > > > > > Dmitri
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> agreed
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> the direction is worth
> > > > > > exploring.
> > > > > > > >> The
> > > > > > > >> > > meeting
> > > > > > > >> > > > > > >> > > converged
> > > > > > > >> > > > > > >> > > > > on a
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> battery-included
> default
> > > > > (likely
> > > > > > > >> > > > > Vert.x-backed)
> > > > > > > >> > > > > > >> with
> > > > > > > >> > > > > > >> > > an
> > > > > > > >> > > > > > >> > > > > SPI
> > > > > > > >> > > > > > >> > > > > > > > shape
> > > > > > > >> > > > > > >> > > > > > > > > > that
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> lets
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> power users route to a
> > more
> > > > > > scalable
> > > > > > > >> > > backend
> > > > > > > >> > > > > > >> > > (k8s-hosted
> > > > > > > >> > > > > > >> > > > > > > queue,
> > > > > > > >> > > > > > >> > > > > > > > > AWS
> > > > > > > >> > > > > > >> > > > > > > > > > > SQS,
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> etc.).
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> •   Pluggable
> > destinations:
> > > > > > > >> combining
> > > > > > > >> > > Yufei's
> > > > > > > >> > > > > > >> > > dashboard
> > > > > > > >> > > > > > >> > > > > use
> > > > > > > >> > > > > > >> > > > > > > case
> > > > > > > >> > > > > > >> > > > > > > > > > with
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> Robert's JDBC-focused
> > > > > call-out,
> > > > > > the
> > > > > > > >> > > meeting
> > > > > > > >> > > > > > >> agreed
> > > > > > > >> > > > > > >> > the
> > > > > > > >> > > > > > >> > > > SPI
> > > > > > > >> > > > > > >> > > > > > > > should
> > > > > > > >> > > > > > >> > > > > > > > > be
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> structured for
> multiple
> > > > sinks
> > > > > so
> > > > > > > >> > > integrations
> > > > > > > >> > > > > > >> become
> > > > > > > >> > > > > > >> > > > impl
> > > > > > > >> > > > > > >> > > > > > > > choices
> > > > > > > >> > > > > > >> > > > > > > > > > > rather
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> than architectural
> > changes.
> > > > > > > >> > > > > > >> > > > > > > > > > > > >>
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> The battery-included
> > default
> > > > > is
> > > > > > most
> > > > > > > >> > > likely
> > > > > > > >> > > > > to
> > > > > > > >> > > > > > >> use
> > > > > > > >> > > > > > >> > the
> > > > > > > >> > > > > > >> > > > > > > existing
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> JDBC-backed
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> approach.
> > > > > > > >> > > > > > >> > > > > > > > > > > > >>
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> *Direction (idea-level
> > > > > > alignment)*
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> •   Single
> > metrics_report
> > > > > table
> > > > > > > >> > > consolidating
> > > > > > > >> > > > > > >> > per-type
> > > > > > > >> > > > > > >> > > > > > > metrics,
> > > > > > > >> > > > > > >> > > > > > > > > > > > replacing
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> scan_metrics_report
> and
> > > > > > > >> > > commit_metrics_report
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> •   Iceberg-focused
> > naming
> > > > and
> > > > > > > >> schema
> > > > > > > >> > for
> > > > > > > >> > > > > now,
> > > > > > > >> > > > > > >> > revisit
> > > > > > > >> > > > > > >> > > > if
> > > > > > > >> > > > > > >> > > > > > > > > > > generic-table
> > > > > > > >> > > > > > >> > > > > > > > > > > > or
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> operational metrics
> > arrive
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> •   Metrics
> persistence
> > as a
> > > > > > > >> separated
> > > > > > > >> > > SPI,
> > > > > > > >> > > > > not
> > > > > > > >> > > > > > >> on
> > > > > > > >> > > > > > >> > > > > > > > BasePersistence
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> •   Bootstrap path
> > separated
> > > > > for
> > > > > > > >> > metrics,
> > > > > > > >> > > > > > >> > independent
> > > > > > > >> > > > > > >> > > of
> > > > > > > >> > > > > > >> > > > > > > > metastore
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> bootstrap
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> •   "Battery-included
> > plus
> > > > > > > >> pluggable"
> > > > > > > >> > as
> > > > > > > >> > > the
> > > > > > > >> > > > > > SPI
> > > > > > > >> > > > > > >> > > design
> > > > > > > >> > > > > > >> > > > > > > > philosophy
> > > > > > > >> > > > > > >> > > > > > > > > > > > >>
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> *Open items*
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> •   Schema details:
> > > > > > > >> > > metric_schema_version,
> > > > > > > >> > > > > > >> payload
> > > > > > > >> > > > > > >> > > > format,
> > > > > > > >> > > > > > >> > > > > > IRC
> > > > > > > >> > > > > > >> > > > > > > > v2
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> forward-compat shape
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> •   SPI design
> details —
> > > > full
> > > > > > review
> > > > > > > >> > > either
> > > > > > > >> > > > > in
> > > > > > > >> > > > > > >> the
> > > > > > > >> > > > > > >> > > next
> > > > > > > >> > > > > > >> > > > > sync
> > > > > > > >> > > > > > >> > > > > > > or
> > > > > > > >> > > > > > >> > > > > > > > in
> > > > > > > >> > > > > > >> > > > > > > > > > the
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> corresponding PR
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> •   Schema refactor PR
> > > > > ownership
> > > > > > > >> > > > > > >> > > > > > > > > > > > >>
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> *Action items*
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> •   EJ to take a first
> > stab
> > > > at
> > > > > > the
> > > > > > > >> SPI
> > > > > > > >> > > design
> > > > > > > >> > > > > > and
> > > > > > > >> > > > > > >> > > > > > potentially
> > > > > > > >> > > > > > >> > > > > > > > > > partner
> > > > > > > >> > > > > > >> > > > > > > > > > > > with
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> Anand to incorporate
> the
> > > > > lessons
> > > > > > > >> > learned
> > > > > > > >> > > from
> > > > > > > >> > > > > > the
> > > > > > > >> > > > > > >> > > > existing
> > > > > > > >> > > > > > >> > > > > > > > > reporter
> > > > > > > >> > > > > > >> > > > > > > > > > > and
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> persistence work.
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> •   Schema refactor PR
> > > > > > ownership is
> > > > > > > >> not
> > > > > > > >> > > yet
> > > > > > > >> > > > > > >> decided.
> > > > > > > >> > > > > > >> > > If
> > > > > > > >> > > > > > >> > > > > > anyone
> > > > > > > >> > > > > > >> > > > > > > > is
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> interested in driving
> > it,
> > > > > reply
> > > > > > on
> > > > > > > >> this
> > > > > > > >> > > > > thread.
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> •   JB to schedule the
> > next
> > > > > > sync,
> > > > > > > >> > > tentatively
> > > > > > > >> > > > > > in
> > > > > > > >> > > > > > >> two
> > > > > > > >> > > > > > >> > > > > weeks.
> > > > > > > >> > > > > > >> > > > > > > > > > > > >>
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> -ej
> > > > > > > >> > > > > > >> > > > > > > > > > > > >>
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> On Mon, Apr 27, 2026
> at
> > > > > 3:07 PM
> > > > > > EJ
> > > > > > > >> > Wang <
> > > > > > > >> > > > > > >> > > > > > > > > > >
> > ej.wang.dev.interest@gmail.com
> > > > > > > >> > > > > > >> > > > > > > > > > > > >
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> wrote:
> > > > > > > >> > > > > > >> > > > > > > > > > > > >>
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> > Thanks Yufei for the
> > +1.
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> > JB, could you help
> > add a
> > > > > > biweekly
> > > > > > > >> > > metrics
> > > > > > > >> > > > > > >> > > architecture
> > > > > > > >> > > > > > >> > > > > > sync
> > > > > > > >> > > > > > >> > > > > > > to
> > > > > > > >> > > > > > >> > > > > > > > > the
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> Polaris
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> > community calendar?
> > I'm
> > > > > > thinking
> > > > > > > >> > > Thursdays
> > > > > > > >> > > > > at
> > > > > > > >> > > > > > >> > 9-10am
> > > > > > > >> > > > > > >> > > > PT,
> > > > > > > >> > > > > > >> > > > > > on
> > > > > > > >> > > > > > >> > > > > > > > the
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> off-weeks
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> > from the community
> > meeting
> > > > > > > >> (starting
> > > > > > > >> > > May
> > > > > > > >> > > > > 7),
> > > > > > > >> > > > > > 60
> > > > > > > >> > > > > > >> > > > minutes.
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> > Here's a rough
> agenda
> > to
> > > > > work
> > > > > > > >> through
> > > > > > > >> > > over
> > > > > > > >> > > > > > the
> > > > > > > >> > > > > > >> > first
> > > > > > > >> > > > > > >> > > > few
> > > > > > > >> > > > > > >> > > > > > > > > sessions,
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> grouped
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> > by priority:
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> > *First: foundational
> > > > > > direction*
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> > 1.
> > MetricsPersistence:
> > > > > public
> > > > > > > >> SPI or
> > > > > > > >> > > > > > internal
> > > > > > > >> > > > > > >> > > > > > > implementation
> > > > > > > >> > > > > > >> > > > > > > > > > > detail?
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >    •   Marked @Beta,
> > > > javadoc
> > > > > > calls
> > > > > > > >> > it a
> > > > > > > >> > > > > > >> "Service
> > > > > > > >> > > > > > >> > > > > Provider
> > > > > > > >> > > > > > >> > > > > > > > > > > Interface",
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> but
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> > only one consumer
> > > > > > > >> > > > > (JdbcBasePersistenceImpl),
> > > > > > > >> > > > > > >> lives
> > > > > > > >> > > > > > >> > > on
> > > > > > > >> > > > > > >> > > > > > > > > > > BasePersistence.
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> If
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> > demoted to a private
> > > > helper
> > > > > > > >> inside a
> > > > > > > >> > > > > > persisting
> > > > > > > >> > > > > > >> > > > reporter
> > > > > > > >> > > > > > >> > > > > > > impl,
> > > > > > > >> > > > > > >> > > > > > > > > > most
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> > downstream design
> > > > decisions
> > > > > > become
> > > > > > > >> > > > > > >> implementation
> > > > > > > >> > > > > > >> > > > > details
> > > > > > > >> > > > > > >> > > > > > > > rather
> > > > > > > >> > > > > > >> > > > > > > > > > > than
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> > contract questions.
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> > 2.  Persistence
> schema
> > > > > > redesign
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >    •   Current
> > two-table
> > > > > > layout
> > > > > > > >> > > > > > >> > > (scan_metrics_report,
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >
> commit_metrics_report)
> > > > with
> > > > > > ~25
> > > > > > > >> > > flattened
> > > > > > > >> > > > > > >> columns
> > > > > > > >> > > > > > >> > > > each.
> > > > > > > >> > > > > > >> > > > > > > Every
> > > > > > > >> > > > > > >> > > > > > > > > new
> > > > > > > >> > > > > > >> > > > > > > > > > > > metric
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> > type requires a new
> > table,
> > > > > SPI
> > > > > > > >> > method,
> > > > > > > >> > > > > record
> > > > > > > >> > > > > > >> > class,
> > > > > > > >> > > > > > >> > > > > > model,
> > > > > > > >> > > > > > >> > > > > > > > > > > converter,
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> and
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> > schema migration.
> > > > Direction
> > > > > to
> > > > > > > >> > explore:
> > > > > > > >> > > > > > single
> > > > > > > >> > > > > > >> > table
> > > > > > > >> > > > > > >> > > > > with
> > > > > > > >> > > > > > >> > > > > > > > > > > metric_type
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> enum,
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> > schema_version, and
> > JSON
> > > > > > payload
> > > > > > > >> > > column.
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> > *Second: design
> > details
> > > > once
> > > > > > > >> > direction
> > > > > > > >> > > is
> > > > > > > >> > > > > > set*
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> > 3.  Partition key
> > strategy
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >    •   Single-table
> > design
> > > > > > means
> > > > > > > >> scan
> > > > > > > >> > > > > metrics
> > > > > > > >> > > > > > >> at
> > > > > > > >> > > > > > >> > > scale
> > > > > > > >> > > > > > >> > > > > > will
> > > > > > > >> > > > > > >> > > > > > > > have
> > > > > > > >> > > > > > >> > > > > > > > > > > high
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> > write concurrency
> per
> > > > table.
> > > > > > > >> Schema
> > > > > > > >> > > needs
> > > > > > > >> > > > > to
> > > > > > > >> > > > > > >> > expose
> > > > > > > >> > > > > > >> > > > > enough
> > > > > > > >> > > > > > >> > > > > > > > > > structure
> > > > > > > >> > > > > > >> > > > > > > > > > > > for
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> > backends to shard by
> > > > entity
> > > > > or
> > > > > > > >> time
> > > > > > > >> > > range.
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> > 4.  Read/write path
> > > > > > consistency
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >    •   Writes go
> > through
> > > > > > > >> > > > > > PolarisMetricsManager
> > > > > > > >> > > > > > >> on
> > > > > > > >> > > > > > >> > > > > > > > > > MetaStoreManager.
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> Reads
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> > bypass
> > MetaStoreManager
> > > > and
> > > > > go
> > > > > > > >> > > straight to
> > > > > > > >> > > > > > >> > > > > > BasePersistence,
> > > > > > > >> > > > > > >> > > > > > > > > > > excluding
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> > non-JDBC backends
> > from the
> > > > > > read
> > > > > > > >> API.
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> > *Third: cleanup and
> > > > > alignment*
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> > 5.
> > PolarisMetricsReporter
> > > > > > naming
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >    •   Only handles
> > IRC
> > > > > > > >> > > > > > >> (ScanReport/CommitReport),
> > > > > > > >> > > > > > >> > > > > doesn't
> > > > > > > >> > > > > > >> > > > > > > > cover
> > > > > > > >> > > > > > >> > > > > > > > > > > > generic
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> > tables or
> operational
> > > > > metrics.
> > > > > > > >> Name
> > > > > > > >> > is
> > > > > > > >> > > > > > broader
> > > > > > > >> > > > > > >> > than
> > > > > > > >> > > > > > >> > > > > scope.
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> > 6.
> > PolarisMetricsManager
> > > > > > facade
> > > > > > > >> > > > > passthrough
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >    •   Entire
> default
> > > > method
> > > > > > is
> > > > > > > >> > > > > > >> > > > > > > > > > > > >>
> > > > > > > >> > callCtx.getMetaStore().writeScanReport().
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> > Zero logic, passes
> > Level 1
> > > > > > > >> straight
> > > > > > > >> > > through
> > > > > > > >> > > > > > to
> > > > > > > >> > > > > > >> > Level
> > > > > > > >> > > > > > >> > > > 3.
> > > > > > > >> > > > > > >> > > > > > Same
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> anti-pattern
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> > as
> > PolarisEventManager.
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> > 7.  Iceberg
> community
> > > > > > alignment
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >    •   Payload-type
> > > > > extension
> > > > > > > >> needs
> > > > > > > >> > > > > > discussion
> > > > > > > >> > > > > > >> on
> > > > > > > >> > > > > > >> > > > > > > dev@iceberg.
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> obelix74's
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> > Feb thread got zero
> > > > replies.
> > > > > > > >> Needs a
> > > > > > > >> > > > > > committer
> > > > > > > >> > > > > > >> > > voice.
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> > Lets confirm
> > > > prioritization
> > > > > > in the
> > > > > > > >> > > first
> > > > > > > >> > > > > > >> session.
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> > -ej
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> > On Tue, Apr 21, 2026
> > at
> > > > > > 3:18 PM
> > > > > > > >> Yufei
> > > > > > > >> > > Gu <
> > > > > > > >> > > > > > >> > > > > > > > flyrain000@gmail.com>
> > > > > > > >> > > > > > >> > > > > > > > > > > > wrote:
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> Thanks everyone for
> > > > > > continuing to
> > > > > > > >> > > drive
> > > > > > > >> > > > > this
> > > > > > > >> > > > > > >> > > > forward. I
> > > > > > > >> > > > > > >> > > > > > > agree
> > > > > > > >> > > > > > >> > > > > > > > > > that
> > > > > > > >> > > > > > >> > > > > > > > > > > > the
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> problem is getting
> > > > complex
> > > > > > enough
> > > > > > > >> > > that a
> > > > > > > >> > > > > > more
> > > > > > > >> > > > > > >> > > > > structured
> > > > > > > >> > > > > > >> > > > > > > > > > discussion
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> would
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> help.
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >>
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> +1 on setting up a
> > > > biweekly
> > > > > > sync
> > > > > > > >> for
> > > > > > > >> > > the
> > > > > > > >> > > > > > >> metrics
> > > > > > > >> > > > > > >> > > > > > > > architecture.
> > > > > > > >> > > > > > >> > > > > > > > > > I’m
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> happy
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> to
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> join.
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >>
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> Yufei
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >>
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >>
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> On Tue, Apr 21,
> 2026
> > at
> > > > > > 2:34 PM
> > > > > > > >> EJ
> > > > > > > >> > > Wang <
> > > > > > > >> > > > > > >> > > > > > > > > > > > >>
> > > > > ej.wang.dev.interest@gmail.com>
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> wrote:
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >>
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> > Also, I've been
> > looking
> > > > > > more
> > > > > > > >> > > closely at
> > > > > > > >> > > > > > the
> > > > > > > >> > > > > > >> > > > > > *persistence
> > > > > > > >> > > > > > >> > > > > > > > > schema
> > > > > > > >> > > > > > >> > > > > > > > > > > in
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> the
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> > current metrics
> > work*,
> > > > > and
> > > > > > I
> > > > > > > >> think
> > > > > > > >> > > > > > there's a
> > > > > > > >> > > > > > >> > > > > structural
> > > > > > > >> > > > > > >> > > > > > > > > > rigidity
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> problem
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> > worth raising
> > before
> > > > the
> > > > > > shape
> > > > > > > >> > gets
> > > > > > > >> > > > > locked
> > > > > > > >> > > > > > >> in.
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> > Right now we have
> > two
> > > > > > separate
> > > > > > > >> > > tables
> > > > > > > >> > > > > > >> > > > > > > (scan_metrics_report
> > > > > > > >> > > > > > >> > > > > > > > > and
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >
> > commit_metrics_report),
> > > > > > each
> > > > > > > >> with
> > > > > > > >> > > ~25
> > > > > > > >> > > > > > >> flattened
> > > > > > > >> > > > > > >> > > > > columns
> > > > > > > >> > > > > > >> > > > > > > > that
> > > > > > > >> > > > > > >> > > > > > > > > > > > directly
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> > mirror the
> Iceberg
> > > > report
> > > > > > > >> fields.
> > > > > > > >> > > The
> > > > > > > >> > > > > SPI
> > > > > > > >> > > > > > >> > follows
> > > > > > > >> > > > > > >> > > > the
> > > > > > > >> > > > > > >> > > > > > > same
> > > > > > > >> > > > > > >> > > > > > > > > > split:
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> > writeScanReport
> and
> > > > > > > >> > > writeCommitReport as
> > > > > > > >> > > > > > >> > separate
> > > > > > > >> > > > > > >> > > > > > > methods,
> > > > > > > >> > > > > > >> > > > > > > > > with
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> per-type
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> > record classes,
> > > > > > converters, and
> > > > > > > >> > > model
> > > > > > > >> > > > > > >> objects.
> > > > > > > >> > > > > > >> > > *The
> > > > > > > >> > > > > > >> > > > > > > > practical
> > > > > > > >> > > > > > >> > > > > > > > > > > cost:
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> > adding a new
> metric
> > > > type
> > > > > > > >> > > (operational
> > > > > > > >> > > > > > >> metrics,
> > > > > > > >> > > > > > >> > > for
> > > > > > > >> > > > > > >> > > > > > > example)
> > > > > > > >> > > > > > >> > > > > > > > > > > > requires
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> a
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> new
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> > table, a new SPI
> > > > method,
> > > > > a
> > > > > > new
> > > > > > > >> > > record
> > > > > > > >> > > > > > >> class, a
> > > > > > > >> > > > > > >> > > new
> > > > > > > >> > > > > > >> > > > > > model
> > > > > > > >> > > > > > >> > > > > > > > > > class, a
> > > > > > > >> > > > > > >> > > > > > > > > > > > new
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> > converter branch,
> > and a
> > > > > > schema
> > > > > > > >> > > > > migration*.
> > > > > > > >> > > > > > >> > > That's a
> > > > > > > >> > > > > > >> > > > > lot
> > > > > > > >> > > > > > >> > > > > > > of
> > > > > > > >> > > > > > >> > > > > > > > > > > surface
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> area
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> > for what should
> be
> > "one
> > > > > > more
> > > > > > > >> kind
> > > > > > > >> > of
> > > > > > > >> > > > > > >> metric."
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> > *My bias* would
> be
> > > > > toward a
> > > > > > > >> single
> > > > > > > >> > > > > metrics
> > > > > > > >> > > > > > >> > table
> > > > > > > >> > > > > > >> > > > with
> > > > > > > >> > > > > > >> > > > > > *a
> > > > > > > >> > > > > > >> > > > > > > > > typed
> > > > > > > >> > > > > > >> > > > > > > > > > > JSON
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> > payload*.
> Something
> > > > like:
> > > > > > > >> > > metric_type
> > > > > > > >> > > > > > >> (enum),
> > > > > > > >> > > > > > >> > > > > > entity_id,
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> > table_identifier,
> > > > > > snapshot_id
> > > > > > > >> > > > > (nullable),
> > > > > > > >> > > > > > >> > > > > received_ts,
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> schema_version,
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> and
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> > a payload column
> > for
> > > > the
> > > > > > > >> > > metric-specific
> > > > > > > >> > > > > > >> data.
> > > > > > > >> > > > > > >> > > The
> > > > > > > >> > > > > > >> > > > > > > > > metric_type
> > > > > > > >> > > > > > >> > > > > > > > > > +
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> > schema_version
> pair
> > > > gives
> > > > > > us a
> > > > > > > >> > > > > > >> > forward-compatible
> > > > > > > >> > > > > > >> > > > > > > contract
> > > > > > > >> > > > > > >> > > > > > > > > for
> > > > > > > >> > > > > > >> > > > > > > > > > > the
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> payload
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> > shape. Adding a
> new
> > > > > metric
> > > > > > type
> > > > > > > >> > > becomes
> > > > > > > >> > > > > an
> > > > > > > >> > > > > > >> enum
> > > > > > > >> > > > > > >> > > > value
> > > > > > > >> > > > > > >> > > > > > > and a
> > > > > > > >> > > > > > >> > > > > > > > > > > payload
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> schema,
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> > not a schema
> > migration.
> > > > > One
> > > > > > > >> thing
> > > > > > > >> > I
> > > > > > > >> > > > > think
> > > > > > > >> > > > > > we
> > > > > > > >> > > > > > >> > need
> > > > > > > >> > > > > > >> > > > to
> > > > > > > >> > > > > > >> > > > > be
> > > > > > > >> > > > > > >> > > > > > > > > > > deliberate
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> about is
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> > the partition key
> > > > design.
> > > > > > If
> > > > > > > >> all
> > > > > > > >> > > metric
> > > > > > > >> > > > > > >> types
> > > > > > > >> > > > > > >> > > land
> > > > > > > >> > > > > > >> > > > in
> > > > > > > >> > > > > > >> > > > > > one
> > > > > > > >> > > > > > >> > > > > > > > > > table,
> > > > > > > >> > > > > > >> > > > > > > > > > > > scan
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> > metrics at scale
> > (high
> > > > > > > >> > concurrency,
> > > > > > > >> > > high
> > > > > > > >> > > > > > >> > > frequency
> > > > > > > >> > > > > > >> > > > > > across
> > > > > > > >> > > > > > >> > > > > > > > > many
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> tables)
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> > could easily
> > create hot
> > > > > > > >> > partitions.
> > > > > > > >> > > We'd
> > > > > > > >> > > > > > >> want
> > > > > > > >> > > > > > >> > the
> > > > > > > >> > > > > > >> > > > > > > > persistence
> > > > > > > >> > > > > > >> > > > > > > > > > > layer
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> to
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> be
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> > able to shard by
> > entity
> > > > > or
> > > > > > time
> > > > > > > >> > > range,
> > > > > > > >> > > > > and
> > > > > > > >> > > > > > >> that
> > > > > > > >> > > > > > >> > > > means
> > > > > > > >> > > > > > >> > > > > > the
> > > > > > > >> > > > > > >> > > > > > > > > > logical
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> schema
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> > needs to expose
> > enough
> > > > > > > >> structure
> > > > > > > >> > for
> > > > > > > >> > > > > > >> backends
> > > > > > > >> > > > > > >> > to
> > > > > > > >> > > > > > >> > > > > > > partition
> > > > > > > >> > > > > > >> > > > > > > > > on.
> > > > > > > >> > > > > > >> > > > > > > > > > I
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> don't
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> > think the current
> > > > > flattened
> > > > > > > >> layout
> > > > > > > >> > > gives
> > > > > > > >> > > > > > us
> > > > > > > >> > > > > > >> > that.
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> > This is getting
> > complex
> > > > > > enough
> > > > > > > >> > that
> > > > > > > >> > > I
> > > > > > > >> > > > > > don't
> > > > > > > >> > > > > > >> > think
> > > > > > > >> > > > > > >> > > > > > ad-hoc
> > > > > > > >> > > > > > >> > > > > > > > > PR/ML
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> threads
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> > will converge
> well.
> > > > > *Would
> > > > > > > >> people
> > > > > > > >> > be
> > > > > > > >> > > > > open
> > > > > > > >> > > > > > >> to a
> > > > > > > >> > > > > > >> > > > > biweekly
> > > > > > > >> > > > > > >> > > > > > > > sync
> > > > > > > >> > > > > > >> > > > > > > > > > for
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> metrics
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> > architecture?* I
> > think
> > > > 30
> > > > > > > >> minutes
> > > > > > > >> > > every
> > > > > > > >> > > > > > two
> > > > > > > >> > > > > > >> > weeks
> > > > > > > >> > > > > > >> > > > > with
> > > > > > > >> > > > > > >> > > > > > > > > > interested
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> parties
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> > would be enough
> to
> > work
> > > > > > through
> > > > > > > >> > the
> > > > > > > >> > > > > > schema,
> > > > > > > >> > > > > > >> SPI
> > > > > > > >> > > > > > >> > > > > shape,
> > > > > > > >> > > > > > >> > > > > > > and
> > > > > > > >> > > > > > >> > > > > > > > > read
> > > > > > > >> > > > > > >> > > > > > > > > > > API
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> design
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> > together. Happy
> to
> > help
> > > > > set
> > > > > > > >> that
> > > > > > > >> > up.
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> > -ej
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> > On Mon, Apr 20,
> > 2026 at
> > > > > > > >> 2:19 PM EJ
> > > > > > > >> > > Wang
> > > > > > > >> > > > > <
> > > > > > > >> > > > > > >> > > > > > > > > > > > >>
> > > > > ej.wang.dev.interest@gmail.com
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> > wrote:
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >> Reviewed #4115,
> > left a
> > > > > > > >> comment on
> > > > > > > >> > > the
> > > > > > > >> > > > > > code
> > > > > > > >> > > > > > >> > > > > > organization
> > > > > > > >> > > > > > >> > > > > > > > > side.
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >> One thing stood
> > out:
> > > > the
> > > > > > > >> metrics
> > > > > > > >> > > write
> > > > > > > >> > > > > > path
> > > > > > > >> > > > > > >> > > enters
> > > > > > > >> > > > > > >> > > > > > > through
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>
> > PolarisMetricsManager
> > > > on
> > > > > > > >> > > > > > MetaStoreManager,
> > > > > > > >> > > > > > >> but
> > > > > > > >> > > > > > >> > > the
> > > > > > > >> > > > > > >> > > > > new
> > > > > > > >> > > > > > >> > > > > > > > read
> > > > > > > >> > > > > > >> > > > > > > > > > path
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> bypasses
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >> MetaStoreManager
> > > > > entirely
> > > > > > and
> > > > > > > >> > goes
> > > > > > > >> > > > > > >> straight to
> > > > > > > >> > > > > > >> > > > > > > > > BasePersistence
> > > > > > > >> > > > > > >> > > > > > > > > > > via
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>
> > > > > > callContext.getMetaStore().
> > > > > > > >> That
> > > > > > > >> > > means
> > > > > > > >> > > > > > the
> > > > > > > >> > > > > > >> > read
> > > > > > > >> > > > > > >> > > > API
> > > > > > > >> > > > > > >> > > > > > only
> > > > > > > >> > > > > > >> > > > > > > > > works
> > > > > > > >> > > > > > >> > > > > > > > > > > for
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> backends
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >> that implement
> > > > > > > >> BasePersistence.
> > > > > > > >> > > NoSQL
> > > > > > > >> > > > > and
> > > > > > > >> > > > > > >> > remote
> > > > > > > >> > > > > > >> > > > > > > backends
> > > > > > > >> > > > > > >> > > > > > > > > > can't
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> participate.
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >> Stepping back, I
> > think
> > > > > the
> > > > > > > >> > metrics
> > > > > > > >> > > > > > >> subsystem
> > > > > > > >> > > > > > >> > is
> > > > > > > >> > > > > > >> > > > > > growing
> > > > > > > >> > > > > > >> > > > > > > > into
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> something
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >> real (write +
> > read +
> > > > > REST
> > > > > > API
> > > > > > > >> +
> > > > > > > >> > > AuthZ +
> > > > > > > >> > > > > > >> > > > pagination)
> > > > > > > >> > > > > > >> > > > > > *but
> > > > > > > >> > > > > > >> > > > > > > > the
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> persistence
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >> side is split
> > across
> > > > two
> > > > > > > >> layers
> > > > > > > >> > in
> > > > > > > >> > > a
> > > > > > > >> > > > > way
> > > > > > > >> > > > > > >> > that's
> > > > > > > >> > > > > > >> > > > hard
> > > > > > > >> > > > > > >> > > > > > to
> > > > > > > >> > > > > > >> > > > > > > > > > > extend*. I
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> put
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >> together two
> > diagrams
> > > > to
> > > > > > show
> > > > > > > >> > what
> > > > > > > >> > > I
> > > > > > > >> > > > > mean
> > > > > > > >> > > > > > >> (my
> > > > > > > >> > > > > > >> > > best
> > > > > > > >> > > > > > >> > > > > > > > effort).
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >> *Current state*
> > > > (Diagram
> > > > > > 1):
> > > > > > > >> > three
> > > > > > > >> > > > > > >> interfaces
> > > > > > > >> > > > > > >> > at
> > > > > > > >> > > > > > >> > > > > three
> > > > > > > >> > > > > > >> > > > > > > > > > different
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> levels.
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >> The
> engine-facing
> > SPI
> > > > > > > >> > > > > > >> (PolarisMetricsReporter)
> > > > > > > >> > > > > > >> > > is
> > > > > > > >> > > > > > >> > > > > > clean.
> > > > > > > >> > > > > > >> > > > > > > > But
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>
> > PolarisMetricsManager
> > > > on
> > > > > > > >> > > > > MetaStoreManager
> > > > > > > >> > > > > > >> is a
> > > > > > > >> > > > > > >> > > > > > > passthrough
> > > > > > > >> > > > > > >> > > > > > > > > to
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>
> > MetricsPersistence on
> > > > > > > >> > > BasePersistence.
> > > > > > > >> > > > > > The
> > > > > > > >> > > > > > >> > @Beta
> > > > > > > >> > > > > > >> > > > > > > > annotation
> > > > > > > >> > > > > > >> > > > > > > > > > and
> > > > > > > >> > > > > > >> > > > > > > > > > > > SPI
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> javadoc
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >> are on the
> > > > > BasePersistence
> > > > > > > >> layer,
> > > > > > > >> > > while
> > > > > > > >> > > > > > the
> > > > > > > >> > > > > > >> > > actual
> > > > > > > >> > > > > > >> > > > > > > > extension
> > > > > > > >> > > > > > >> > > > > > > > > > > > points
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>
> > > > (PolarisMetricsReporter,
> > > > > > > >> > > > > > >> > PolarisMetricsManager)
> > > > > > > >> > > > > > >> > > > > carry
> > > > > > > >> > > > > > >> > > > > > no
> > > > > > > >> > > > > > >> > > > > > > > > > > stability
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >> annotation. The
> > write
> > > > > path
> > > > > > > >> goes
> > > > > > > >> > > through
> > > > > > > >> > > > > > the
> > > > > > > >> > > > > > >> > > > > > > > MetaStoreManager
> > > > > > > >> > > > > > >> > > > > > > > > > > > layer,
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> the
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >> read path
> doesn't.
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >> *What I
> envision*
> > > > > > (Diagram 2):
> > > > > > > >> > two
> > > > > > > >> > > SPIs
> > > > > > > >> > > > > > at
> > > > > > > >> > > > > > >> two
> > > > > > > >> > > > > > >> > > > > levels.
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>
> > PolarisMetricsReporter
> > > > > > stays
> > > > > > > >> as
> > > > > > > >> > the
> > > > > > > >> > > > > > >> > > engine-facing
> > > > > > > >> > > > > > >> > > > > SPI.
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>
> > PolarisMetricsManager
> > > > > > becomes
> > > > > > > >> the
> > > > > > > >> > > > > > >> > backend-facing
> > > > > > > >> > > > > > >> > > > SPI
> > > > > > > >> > > > > > >> > > > > > > with
> > > > > > > >> > > > > > >> > > > > > > > > both
> > > > > > > >> > > > > > >> > > > > > > > > > > > write
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> and
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >> read methods at
> > the
> > > > > > > >> > > MetaStoreManager
> > > > > > > >> > > > > > level,
> > > > > > > >> > > > > > >> > > where
> > > > > > > >> > > > > > >> > > > > any
> > > > > > > >> > > > > > >> > > > > > > > > backend
> > > > > > > >> > > > > > >> > > > > > > > > > > > (JDBC,
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> NoSQL,
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >> remote) can
> > implement
> > > > > > them.
> > > > > > > >> > > > > > >> MetricsPersistence
> > > > > > > >> > > > > > >> > > on
> > > > > > > >> > > > > > >> > > > > > > > > > > BasePersistence
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> goes
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >> away. Where
> > metrics
> > > > > > actually
> > > > > > > >> land
> > > > > > > >> > > is an
> > > > > > > >> > > > > > >> > > > > implementation
> > > > > > > >> > > > > > >> > > > > > > > > detail,
> > > > > > > >> > > > > > >> > > > > > > > > > > > not a
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> core
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >> interface.
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >> *Minor naming
> > thing*:
> > > > > > > >> > > > > > >> PolarisMetricsReporter
> > > > > > > >> > > > > > >> > is
> > > > > > > >> > > > > > >> > > > > > broader
> > > > > > > >> > > > > > >> > > > > > > > than
> > > > > > > >> > > > > > >> > > > > > > > > > > what
> > > > > > > >> > > > > > >> > > > > > > > > > > > it
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >> actually
> handles.
> > It
> > > > > only
> > > > > > > >> accepts
> > > > > > > >> > > > > Iceberg
> > > > > > > >> > > > > > >> REST
> > > > > > > >> > > > > > >> > > > > Catalog
> > > > > > > >> > > > > > >> > > > > > > > > metrics
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> (ScanReport,
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >> CommitReport via
> > > > > > > >> MetricsReport).
> > > > > > > >> > > > > Generic
> > > > > > > >> > > > > > >> table
> > > > > > > >> > > > > > >> > > > > metrics
> > > > > > > >> > > > > > >> > > > > > > or
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> operational
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >> metrics aren't
> in
> > > > scope.
> > > > > > Not
> > > > > > > >> > > blocking,
> > > > > > > >> > > > > > but
> > > > > > > >> > > > > > >> > worth
> > > > > > > >> > > > > > >> > > > > > noting
> > > > > > > >> > > > > > >> > > > > > > if
> > > > > > > >> > > > > > >> > > > > > > > > the
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> metrics
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >> surface expands.
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >> *Rough sketch of
> > how
> > > > to
> > > > > > get
> > > > > > > >> > there*:
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>  1.  Add read
> > methods
> > > > to
> > > > > > > >> > > > > > >> PolarisMetricsManager
> > > > > > > >> > > > > > >> > > > > > > > > > (listScanReports,
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>
> listCommitReports)
> > > > with
> > > > > > > >> default
> > > > > > > >> > > no-op,
> > > > > > > >> > > > > > >> same as
> > > > > > > >> > > > > > >> > > the
> > > > > > > >> > > > > > >> > > > > > > > existing
> > > > > > > >> > > > > > >> > > > > > > > > > > write
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> methods.
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >> (Probably make
> > > > > > > >> > > PolarisMetricsManager
> > > > > > > >> > > > > more
> > > > > > > >> > > > > > >> > > explicit
> > > > > > > >> > > > > > >> > > > > on
> > > > > > > >> > > > > > >> > > > > > > > being
> > > > > > > >> > > > > > >> > > > > > > > > > > > Iceberg
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >> specific like
> > package
> > > > > > name or
> > > > > > > >> > class
> > > > > > > >> > > > > name
> > > > > > > >> > > > > > >> etc.)
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>  2.  Wire
> > > > > > > >> MetricsReportsService
> > > > > > > >> > > through
> > > > > > > >> > > > > > >> > > > > > MetaStoreManager
> > > > > > > >> > > > > > >> > > > > > > > > > instead
> > > > > > > >> > > > > > >> > > > > > > > > > > > of
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>
> > > > > > callContext.getMetaStore().
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>  3.  Extract
> > metrics
> > > > > > > >> persistence
> > > > > > > >> > > from
> > > > > > > >> > > > > > >> > > > > > > > > JdbcBasePersistenceImpl
> > > > > > > >> > > > > > >> > > > > > > > > > > into
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> its
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >> own class. That
> > file
> > > > > > carries
> > > > > > > >> ~7
> > > > > > > >> > > > > > >> > > responsibilities,
> > > > > > > >> > > > > > >> > > > > > > metrics
> > > > > > > >> > > > > > >> > > > > > > > > > being
> > > > > > > >> > > > > > >> > > > > > > > > > > > one
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> of
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> them.
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>  4.  Remove
> > > > > > MetricsPersistence
> > > > > > > >> > from
> > > > > > > >> > > > > > >> > > > BasePersistence.
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >> *None of this
> > needs to
> > > > > > happen
> > > > > > > >> in
> > > > > > > >> > > #4115.
> > > > > > > >> > > > > > >> But if
> > > > > > > >> > > > > > >> > > the
> > > > > > > >> > > > > > >> > > > > > > > direction
> > > > > > > >> > > > > > >> > > > > > > > > > > makes
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> sense,
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >> it would be good
> > to
> > > > > align
> > > > > > > >> before
> > > > > > > >> > > the
> > > > > > > >> > > > > > >> metrics
> > > > > > > >> > > > > > >> > > > surface
> > > > > > > >> > > > > > >> > > > > > > grows
> > > > > > > >> > > > > > >> > > > > > > > > > > > further.
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> Curious
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >> what others
> > think.*
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >> *My mental model
> > > > note*:
> > > > > > Level
> > > > > > > >> 1
> > > > > > > >> > > > > > >> > > MetaStoreManager;
> > > > > > > >> > > > > > >> > > > > > level
> > > > > > > >> > > > > > >> > > > > > > 2
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> transactional
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >> persistence;
> > level 3
> > > > > base
> > > > > > > >> > > persistence
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >> Diagram 1
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >> <
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >>
> > > > > > > >> > > > > > >> > > > > > > > > > > > >>
> > > > > > > >> > > > > > >> > > > > > > > > > > >
> > > > > > > >> > > > > > >> > > > > > > > > > >
> > > > > > > >> > > > > > >> > > > > > > > > >
> > > > > > > >> > > > > > >> > > > > > > > >
> > > > > > > >> > > > > > >> > > > > > > >
> > > > > > > >> > > > > > >> > > > > > >
> > > > > > > >> > > > > > >> > > > > >
> > > > > > > >> > > > > > >> > > > >
> > > > > > > >> > > > > > >> > > >
> > > > > > > >> > > > > > >> > >
> > > > > > > >> > > > > > >> >
> > > > > > > >> > > > > > >>
> > > > > > > >> > > > > >
> > > > > > > >> > > > >
> > > > > > > >> > >
> > > > > > > >> >
> > > > > > > >>
> > > > > >
> > > > >
> > > >
> >
> https://urldefense.com/v3/__https://www.plantuml.com/plantuml/uml/bLHDR-Cs4BthLmpIYupw0zbkKQ1r3M-S7Bp8xhhM7WCOb3IM65EaGD9EX2RzxHrHb4CxRelwa4YSDu_lpOVcnZ9jzvM8BBS2uGjQpJC3dtHMSekPtMk44IpsMgEqa5XcCOhCZikQQLP1pR8TAp2n3ILhmZDP20m0fcIvUkAoW2qJXd9z1bpToO9BX3WXu0ucy5rpgGPNm0nW5_epUWtm2Ue3pn3kMOFQmKntGZW0BYtgBSi8k5A2QMwybJNMIbFiGSR9QZc4nUqIvikStF0jHprua5C-amge42aNt3R0f5JaaoivdV2Pkqbx4hee4ymOkBh5BTiB-_uIeGeo8zL8rPsPl4DktdEiK1jkB1NdZCRbrSTecDe_mlHbF0wvBmCkaOH5_S8a_TTTKI6-nmCAkEw4LpxsZ-LbYLKQFKMNOgf_wuM7_bV9gOer5SYMMksBSWXFcbi49KNZXNLicwfe3TETC7gPdPqI7uBcHMb1RSzYq34c6PDUM9mn8HRsUTZEiDBve3NjVZumBj0U7SS37mGO7vcwtiK-_pU7U7L_f-digo9YbhSwIfMRwIITKGXbxdIUTCGF1SeCJxloKsU-3k9ddRbX1eDq1q_fx1JbBGT0glVyXimDuP4TQ5qpCAmnGEj2s_6n5mtn1z-97-63itFQZLPO1Ev2tu_WF7Ju-VPc0Skg5bYXxBhkY1xpD7EM_7fyflSpIsqMgVth5xhVr4eQxWQ8enaSAJQSG16yFSDuJ798rrcXr_3n-lfdk7icQjEBmFujL7AodiP_Y4Z7-YxvtZNs4zMgpNTl6tF8sglyPsmqchrjvQ-m-aP94r-TwCA2Ka8upPJZwtvSpoYCXkYMZU2NXvRMBfq9P3i3Le4VAZUAlUZ_oPKsxPgY0Q_BSKLkyr9bhQhQrJjo_x3TPlIB0DPjnMfcIoYP0QaYw1a0fTKDr8fB6ntNuvmoL1ZGkXa69Njh43zf9GiGxHQrA_jDYWRSzF5--WmTVrN97_Sm8LbLUy_lGBmLanJjFkDlGkRqjA_4tm00__;!!Iz9xO38YGHZK!4rnlVIqBQQQ6xpGe1PMV58edOS_cGrKoqdfhw4qSp_JQCavvEjaxXk0OwGPnxyEcO60AzKQK_XDtp7H3DdkFXcrXuTBISHg$
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >> :
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >> [image:
> image.png]
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >> Diagram 2
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >> <
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >>
> > > > > > > >> > > > > > >> > > > > > > > > > > > >>
> > > > > > > >> > > > > > >> > > > > > > > > > > >
> > > > > > > >> > > > > > >> > > > > > > > > > >
> > > > > > > >> > > > > > >> > > > > > > > > >
> > > > > > > >> > > > > > >> > > > > > > > >
> > > > > > > >> > > > > > >> > > > > > > >
> > > > > > > >> > > > > > >> > > > > > >
> > > > > > > >> > > > > > >> > > > > >
> > > > > > > >> > > > > > >> > > > >
> > > > > > > >> > > > > > >> > > >
> > > > > > > >> > > > > > >> > >
> > > > > > > >> > > > > > >> >
> > > > > > > >> > > > > > >>
> > > > > > > >> > > > > >
> > > > > > > >> > > > >
> > > > > > > >> > >
> > > > > > > >> >
> > > > > > > >>
> > > > > >
> > > > >
> > > >
> >
> https://urldefense.com/v3/__https://www.plantuml.com/plantuml/uml/VLLDR-8m4BtdLupO2sWBLVU8AaGB7AXAbssGzb896SS9RXqxjKqBwkv_tt7iV43fdaZYDpFlpRmnOsE9jhjSH9PRmM31hERKm8scMsuPjJlDe0yheZDc8RR4iYWoBrmMH9CS2a9VICPYUy1OZN0YCy5Q0BCbYNhdCeEK28En8G8wCvbnoQ0R8_05Bc6bkLIz3X03p1zzH7zR-9ZfDquPt9C3qoNCX2yV4G2NbkcKu5jdgGJHt0GbZwnG6i-UP3TUpk5gM6Ldqke350eZUqzoCft3U9xWHvxoa5-7K4nF1J46EbEMafsmdrCBbQ44gVggy18IZrn_ph5asd1ZiIKdQSgueZvjXrQFSFrdC3YN-nXmBacxbGiYyLVxLaBtdhqn0LSzdBDhqQtQoOJeGyad3z0lUqnYgpGB6Ns8oVyta00Dy_WnX0tIOZ8v6SYxHll1TrH6aejAik-mh-AphVFCwSUQqFypElag5QRGFDjQKEd96K1P8QP41c9TzA_IIQyvdAWyv_RSiS3skb0_EzDDkK2v5xWF6MiGFlvhpFLcD2Dq2pml14gaF67eQkmd8gulDoC4kSOu6KVpkvlUJg1RTbWISU40RdBUUS_9XfRZ2dwxm_SW8LYFISgm_MnlDQ6M9P1gbKEc4X-2pH_FvJCkCqm9pbVjD6LrwdLeOrDWfOaqc8Wh9BE85oNKxkNQ6o4yGRy_Eae0G_G8tZv81d3bHDB23WOdisohVr3nh_j6lbSjbNaLRTc8UgtPbAU1J_tygOfZX9DWEJeHDvYx-qmSi5FgNLPZwHrHcUsncGQ5-skhUclpE5fo4ounpFauYrUbkU6ccfnxMvitwag4IyerhTxj8In_Oj1bDO4pQru674loYrGlULHLEGCjwJJ8gDoVZR8MxO4BT3IzRvIcAQKezC6xpziGnTyImrfEGyJI_OcKfgtxIvnTqFEMS17L9Z-jsARN5FmTheP7HtSdtOMT0B4GY2FYHXxgQmMtj2bRqiLFGapiVe1_QVKDrkqXcm83aFEXnMYCZ-xlyHy__;!!Iz9xO38YGHZK!4rnlVIqBQQQ6xpGe1PMV58edOS_cGrKoqdfhw4qSp_JQCavvEjaxXk0OwGPnxyEcO60AzKQK_XDtp7H3DdkFXcrXSpmqcNk$
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >> :
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >> [image:
> image.png]
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>  -ej
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >> On Wed, Apr 15,
> > 2026
> > > > at
> > > > > > > >> 8:22 AM
> > > > > > > >> > > Dmitri
> > > > > > > >> > > > > > >> > > > Bourlatchkov
> > > > > > > >> > > > > > >> > > > > <
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> dimas@apache.org>
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >> wrote:
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> Hi All,
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>>
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> Heads up: The
> > current
> > > > > > state
> > > > > > > >> of
> > > > > > > >> > PR
> > > > > > > >> > > > > [4115]
> > > > > > > >> > > > > > >> > looks
> > > > > > > >> > > > > > >> > > > > pretty
> > > > > > > >> > > > > > >> > > > > > > > solid
> > > > > > > >> > > > > > >> > > > > > > > > > to
> > > > > > > >> > > > > > >> > > > > > > > > > > > me.
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> I
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> believe this PR
> > is
> > > > > > > >> approaching a
> > > > > > > >> > > > > > mergeable
> > > > > > > >> > > > > > >> > > > > condition.
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>>
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> Please post
> your
> > > > > reviews
> > > > > > if
> > > > > > > >> you
> > > > > > > >> > > have
> > > > > > > >> > > > > any
> > > > > > > >> > > > > > >> > > > comments.
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>>
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> [4115]
> > > > > > > >> > > > > > >> > > >
> > > > > > > >> >
> > > > > > > >>
> > > > > >
> > > > >
> > > >
> >
> https://urldefense.com/v3/__https://github.com/apache/polaris/pull/4115__;!!Iz9xO38YGHZK!4rnlVIqBQQQ6xpGe1PMV58edOS_cGrKoqdfhw4qSp_JQCavvEjaxXk0OwGPnxyEcO60AzKQK_XDtp7H3DdkFXcrXtFRscKs$
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>>
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> Thanks,
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> Dmitri.
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>>
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> On Tue, Mar 3,
> > 2026
> > > > at
> > > > > > > >> 3:29 PM
> > > > > > > >> > > Anand
> > > > > > > >> > > > > > Kumar
> > > > > > > >> > > > > > >> > > > Sankaran
> > > > > > > >> > > > > > >> > > > > > via
> > > > > > > >> > > > > > >> > > > > > > > > dev <
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>>
> > > > dev@polaris.apache.org
> > > > > >
> > > > > > > >> wrote:
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>>
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > Hi Yufei and
> > > > Dmitri,
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> >
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > Here is a
> > proposal
> > > > > for
> > > > > > the
> > > > > > > >> > REST
> > > > > > > >> > > > > > >> endpoints
> > > > > > > >> > > > > > >> > for
> > > > > > > >> > > > > > >> > > > > > metrics
> > > > > > > >> > > > > > >> > > > > > > > and
> > > > > > > >> > > > > > >> > > > > > > > > > > > events.
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> >
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> >
> > > > > > > >> > > > > > >> > > > > >
> > > > > > > >> >
> > > > > > > >>
> > > > > >
> > > > >
> > > >
> >
> https://urldefense.com/v3/__https://github.com/apache/polaris/pull/3924/changes__;!!Iz9xO38YGHZK!4rnlVIqBQQQ6xpGe1PMV58edOS_cGrKoqdfhw4qSp_JQCavvEjaxXk0OwGPnxyEcO60AzKQK_XDtp7H3DdkFXcrXcjoPhSs$
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> >
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > I did not see
> > any
> > > > > > > >> precursors
> > > > > > > >> > for
> > > > > > > >> > > > > > >> raising a
> > > > > > > >> > > > > > >> > PR
> > > > > > > >> > > > > > >> > > > for
> > > > > > > >> > > > > > >> > > > > > > > > > proposals,
> > > > > > > >> > > > > > >> > > > > > > > > > > so
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> trying
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > this.  Please
> > let
> > > > me
> > > > > > know
> > > > > > > >> what
> > > > > > > >> > > you
> > > > > > > >> > > > > > >> think.
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> >
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > -
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > Anand
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> >
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > From: Anand
> > Kumar
> > > > > > Sankaran
> > > > > > > >> <
> > > > > > > >> > > > > > >> > > > > > > anand.sankaran@workday.com
> > > > > > > >> > > > > > >> > > > > > > > >
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > Date: Monday,
> > March
> > > > > 2,
> > > > > > > >> 2026 at
> > > > > > > >> > > > > > 10:25 AM
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > To:
> > > > > > dev@polaris.apache.org
> > > > > > > >> <
> > > > > > > >> > > > > > >> > > > > dev@polaris.apache.org
> > > > > > > >> > > > > > >> > > > > > >
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > Subject: Re:
> > > > Polaris
> > > > > > > >> Telemetry
> > > > > > > >> > > and
> > > > > > > >> > > > > > Audit
> > > > > > > >> > > > > > >> > > Trail
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> >
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > About the
> REST
> > API,
> > > > > > based
> > > > > > > >> on
> > > > > > > >> > my
> > > > > > > >> > > use
> > > > > > > >> > > > > > >> cases:
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> >
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> >
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> >   1.
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > I want to be
> > able
> > > > to
> > > > > > query
> > > > > > > >> > > commit
> > > > > > > >> > > > > > >> metrics
> > > > > > > >> > > > > > >> > to
> > > > > > > >> > > > > > >> > > > > track
> > > > > > > >> > > > > > >> > > > > > > > files
> > > > > > > >> > > > > > >> > > > > > > > > > > added
> > > > > > > >> > > > > > >> > > > > > > > > > > > /
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> removed
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > per commit,
> > along
> > > > > with
> > > > > > > >> record
> > > > > > > >> > > > > counts.
> > > > > > > >> > > > > > >> The
> > > > > > > >> > > > > > >> > > > > ingestion
> > > > > > > >> > > > > > >> > > > > > > > > > pipeline
> > > > > > > >> > > > > > >> > > > > > > > > > > > that
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> writes
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > this data is
> > owned
> > > > by
> > > > > > us
> > > > > > > >> and
> > > > > > > >> > we
> > > > > > > >> > > are
> > > > > > > >> > > > > > >> > > guaranteed
> > > > > > > >> > > > > > >> > > > to
> > > > > > > >> > > > > > >> > > > > > > write
> > > > > > > >> > > > > > >> > > > > > > > > > this
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> information
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > for each
> write.
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> >   2.
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > I want to be
> > able
> > > > to
> > > > > > query
> > > > > > > >> > scan
> > > > > > > >> > > > > > metrics
> > > > > > > >> > > > > > >> for
> > > > > > > >> > > > > > >> > > > > read. I
> > > > > > > >> > > > > > >> > > > > > > > > > > understand
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> clients
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> do
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > not fulfill
> > this
> > > > > > > >> requirement.
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> >   3.
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > I want to be
> > able
> > > > to
> > > > > > query
> > > > > > > >> the
> > > > > > > >> > > > > events
> > > > > > > >> > > > > > >> table
> > > > > > > >> > > > > > >> > > > > (events
> > > > > > > >> > > > > > >> > > > > > > are
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> persisted) -
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> this
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > may supersede
> > #2, I
> > > > > am
> > > > > > not
> > > > > > > >> > sure
> > > > > > > >> > > yet.
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> >
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > All this
> > > > information
> > > > > > is in
> > > > > > > >> the
> > > > > > > >> > > JDBC
> > > > > > > >> > > > > > >> based
> > > > > > > >> > > > > > >> > > > > > persistence
> > > > > > > >> > > > > > >> > > > > > > > > model
> > > > > > > >> > > > > > >> > > > > > > > > > > and
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> is
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > persisted in
> > the
> > > > > > > >> metastore. I
> > > > > > > >> > > > > > currently
> > > > > > > >> > > > > > >> > don’t
> > > > > > > >> > > > > > >> > > > > have
> > > > > > > >> > > > > > >> > > > > > a
> > > > > > > >> > > > > > >> > > > > > > > need
> > > > > > > >> > > > > > >> > > > > > > > > > to
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> query
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > prometheus or
> > open
> > > > > > > >> telemetry.
> > > > > > > >> > I
> > > > > > > >> > > do
> > > > > > > >> > > > > > >> publish
> > > > > > > >> > > > > > >> > > some
> > > > > > > >> > > > > > >> > > > > > > events
> > > > > > > >> > > > > > >> > > > > > > > to
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> Prometheus
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> and
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > they are
> > forwarded
> > > > to
> > > > > > our
> > > > > > > >> > > dashboards
> > > > > > > >> > > > > > >> > > elsewhere.
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> >
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > About the CLI
> > > > > > utilities, I
> > > > > > > >> > > meant the
> > > > > > > >> > > > > > >> admin
> > > > > > > >> > > > > > >> > > user
> > > > > > > >> > > > > > >> > > > > > > > > utilities.
> > > > > > > >> > > > > > >> > > > > > > > > > In
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> one of
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> the
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > earliest
> > drafts of
> > > > my
> > > > > > > >> > proposal,
> > > > > > > >> > > > > > Prashant
> > > > > > > >> > > > > > >> > > > > mentioned
> > > > > > > >> > > > > > >> > > > > > > that
> > > > > > > >> > > > > > >> > > > > > > > > the
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> metrics
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> tables
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > can grow
> > > > indefinitely
> > > > > > and
> > > > > > > >> > that a
> > > > > > > >> > > > > > similar
> > > > > > > >> > > > > > >> > > > problem
> > > > > > > >> > > > > > >> > > > > > > exists
> > > > > > > >> > > > > > >> > > > > > > > > > with
> > > > > > > >> > > > > > >> > > > > > > > > > > > the
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> events
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > table as
> well.
> > We
> > > > > > discussed
> > > > > > > >> > that
> > > > > > > >> > > > > > >> cleaning
> > > > > > > >> > > > > > >> > up
> > > > > > > >> > > > > > >> > > of
> > > > > > > >> > > > > > >> > > > > old
> > > > > > > >> > > > > > >> > > > > > > > > records
> > > > > > > >> > > > > > >> > > > > > > > > > > > from
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> both
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > metrics
> tables
> > and
> > > > > > events
> > > > > > > >> > > tables can
> > > > > > > >> > > > > > be
> > > > > > > >> > > > > > >> > done
> > > > > > > >> > > > > > >> > > > via
> > > > > > > >> > > > > > >> > > > > a
> > > > > > > >> > > > > > >> > > > > > > CLI
> > > > > > > >> > > > > > >> > > > > > > > > > > utility.
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> >
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > I see that
> > Yufei
> > > > has
> > > > > > > >> covered
> > > > > > > >> > the
> > > > > > > >> > > > > > >> discussion
> > > > > > > >> > > > > > >> > > > about
> > > > > > > >> > > > > > >> > > > > > > > > > > datasources.
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> >
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > -
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > Anand
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> >
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> >
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> >
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > From: Yufei
> Gu
> > <
> > > > > > > >> > > > > flyrain000@gmail.com>
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > Date: Friday,
> > > > > February
> > > > > > 27,
> > > > > > > >> > 2026
> > > > > > > >> > > at
> > > > > > > >> > > > > > >> 9:54 PM
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > To:
> > > > > > dev@polaris.apache.org
> > > > > > > >> <
> > > > > > > >> > > > > > >> > > > > dev@polaris.apache.org
> > > > > > > >> > > > > > >> > > > > > >
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > Subject: Re:
> > > > Polaris
> > > > > > > >> Telemetry
> > > > > > > >> > > and
> > > > > > > >> > > > > > Audit
> > > > > > > >> > > > > > >> > > Trail
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> >
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > This Message
> Is
> > > > From
> > > > > an
> > > > > > > >> > External
> > > > > > > >> > > > > > Sender
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > This message
> > came
> > > > > from
> > > > > > > >> outside
> > > > > > > >> > > your
> > > > > > > >> > > > > > >> > > > organization.
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > Report
> > Suspicious<
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> >
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>>
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >>
> > > > > > > >> > > > > > >> > > > > > > > > > > > >>
> > > > > > > >> > > > > > >> > > > > > > > > > > >
> > > > > > > >> > > > > > >> > > > > > > > > > >
> > > > > > > >> > > > > > >> > > > > > > > > >
> > > > > > > >> > > > > > >> > > > > > > > >
> > > > > > > >> > > > > > >> > > > > > > >
> > > > > > > >> > > > > > >> > > > > > >
> > > > > > > >> > > > > > >> > > > > >
> > > > > > > >> > > > > > >> > > > >
> > > > > > > >> > > > > > >> > > >
> > > > > > > >> > > > > > >> > >
> > > > > > > >> > > > > > >> >
> > > > > > > >> > > > > > >>
> > > > > > > >> > > > > >
> > > > > > > >> > > > >
> > > > > > > >> > >
> > > > > > > >> >
> > > > > > > >>
> > > > > >
> > > > >
> > > >
> >
> https://us-phishalarm-ewt.proofpoint.com/EWT/v1/Iz9xO38YGHZK!YhNDZABkHi1B699ote2uMwpOZw8i0QMCGO2Szc-HshuABGhGvwPJcymE6G2oUUxtS8xDkSrtGTPm_I3QnVDHoLMk50m9v8z_nZKTkd-bnVUbreF1u0WnfV_X5eYevZl_$
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > >
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> >
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> >
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > As I
> mentioned
> > in
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> >
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>>
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >>
> > > > > > > >> > > > > > >> > > > > > > > > > > > >>
> > > > > > > >> > > > > > >> > > > > > > > > > > >
> > > > > > > >> > > > > > >> > > > > > > > > > >
> > > > > > > >> > > > > > >> > > > > > > > > >
> > > > > > > >> > > > > > >> > > > > > > > >
> > > > > > > >> > > > > > >> > > > > > > >
> > > > > > > >> > > > > > >> > > > > > >
> > > > > > > >> > > > > > >> > > > > >
> > > > > > > >> > > > > > >> > > > >
> > > > > > > >> > > > > > >> > > >
> > > > > > > >> > > > > > >> > >
> > > > > > > >> > > > > > >> >
> > > > > > > >> > > > > > >>
> > > > > > > >> > > > > >
> > > > > > > >> > > > >
> > > > > > > >> > >
> > > > > > > >> >
> > > > > > > >>
> > > > > >
> > > > >
> > > >
> >
> https://urldefense.com/v3/__https://github.com/apache/polaris/issues/3890__;!!Iz9xO38YGHZK!5EuyFFkk3vhRWVIRvQAWBSQfpJkTMA9HxugzDwXmN0LPPqhEFxYkFRGVhtb8AqUwXtDh2OplcMnbMDHKOxrvDU0$
> > > > > > > >> > >> > > >> > > > > > > > > > > > >> >> >>> ,
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > supporting
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > multiple data
> > > > sources
> > > > > > is
> > > > > > > >> not a
> > > > > > > >> > > > > trivial
> > > > > > > >> > > > > > >> > > change.
> > > > > > > >> > > > > > >> > > > I
> > > > > > > >> > > > > > >> > > > > > > would
> > > > > > > >> > > > > > >> > > > > > > > > > > strongly
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> recommend
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > starting
> with a
> > > > > design
> > > > > > > >> > document
> > > > > > > >> > > to
> > > > > > > >> > > > > > >> > carefully
> > > > > > > >> > > > > > >> > > > > > evaluate
> > > > > > > >> > > > > > >> > > > > > > > the
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> architectural
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > implications
> > and
> > > > long
> > > > > > term
> > > > > > > >> > > impact.
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> >
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > A REST
> > endpoint to
> > > > > > query
> > > > > > > >> > metrics
> > > > > > > >> > > > > seems
> > > > > > > >> > > > > > >> > > > reasonable
> > > > > > > >> > > > > > >> > > > > > > given
> > > > > > > >> > > > > > >> > > > > > > > > the
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> current
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> JDBC
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > based
> > persistence
> > > > > > model.
> > > > > > > >> That
> > > > > > > >> > > said,
> > > > > > > >> > > > > we
> > > > > > > >> > > > > > >> may
> > > > > > > >> > > > > > >> > > also
> > > > > > > >> > > > > > >> > > > > > > > consider
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> alternative
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > storage
> > models. For
> > > > > > > >> example,
> > > > > > > >> > if
> > > > > > > >> > > we
> > > > > > > >> > > > > > later
> > > > > > > >> > > > > > >> > > adopt
> > > > > > > >> > > > > > >> > > > a
> > > > > > > >> > > > > > >> > > > > > time
> > > > > > > >> > > > > > >> > > > > > > > > > series
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> system
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> such as
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > Prometheus to
> > store
> > > > > > > >> metrics,
> > > > > > > >> > the
> > > > > > > >> > > > > query
> > > > > > > >> > > > > > >> > model
> > > > > > > >> > > > > > >> > > > and
> > > > > > > >> > > > > > >> > > > > > > access
> > > > > > > >> > > > > > >> > > > > > > > > > > > patterns
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> would
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> be
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > fundamentally
> > > > > > different.
> > > > > > > >> > > Designing
> > > > > > > >> > > > > the
> > > > > > > >> > > > > > >> REST
> > > > > > > >> > > > > > >> > > API
> > > > > > > >> > > > > > >> > > > > > > without
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> considering
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> these
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > potential
> > > > evolutions
> > > > > > may
> > > > > > > >> limit
> > > > > > > >> > > > > > >> flexibility.
> > > > > > > >> > > > > > >> > > I'd
> > > > > > > >> > > > > > >> > > > > > > suggest
> > > > > > > >> > > > > > >> > > > > > > > > to
> > > > > > > >> > > > > > >> > > > > > > > > > > > start
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> with
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> the
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > use case.
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> >
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > Yufei
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> >
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> >
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > On Fri, Feb
> 27,
> > > > 2026
> > > > > at
> > > > > > > >> > 3:42 PM
> > > > > > > >> > > > > Dmitri
> > > > > > > >> > > > > > >> > > > > > Bourlatchkov <
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> dimas@apache.org>
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > wrote:
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> >
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > > Hi Anand,
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > >
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > > Sharing my
> > > > view...
> > > > > > > >> subject
> > > > > > > >> > to
> > > > > > > >> > > > > > >> discussion:
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > >
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > > 1. Adding
> > non-IRC
> > > > > > REST
> > > > > > > >> API
> > > > > > > >> > to
> > > > > > > >> > > > > > Polaris
> > > > > > > >> > > > > > >> is
> > > > > > > >> > > > > > >> > > > > > perfectly
> > > > > > > >> > > > > > >> > > > > > > > > fine.
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > >
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > > Figuring
> out
> > > > > specific
> > > > > > > >> > endpoint
> > > > > > > >> > > > > URIs
> > > > > > > >> > > > > > >> and
> > > > > > > >> > > > > > >> > > > > payloads
> > > > > > > >> > > > > > >> > > > > > > > might
> > > > > > > >> > > > > > >> > > > > > > > > > > > require
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> a
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> few
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > > roundtrips,
> > so
> > > > > > opening a
> > > > > > > >> > > separate
> > > > > > > >> > > > > > >> thread
> > > > > > > >> > > > > > >> > > for
> > > > > > > >> > > > > > >> > > > > that
> > > > > > > >> > > > > > >> > > > > > > > might
> > > > > > > >> > > > > > >> > > > > > > > > > be
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> best.
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > >
> Contributors
> > > > > commonly
> > > > > > > >> create
> > > > > > > >> > > > > Google
> > > > > > > >> > > > > > >> Docs
> > > > > > > >> > > > > > >> > > for
> > > > > > > >> > > > > > >> > > > > new
> > > > > > > >> > > > > > >> > > > > > > API
> > > > > > > >> > > > > > >> > > > > > > > > > > > proposals
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> too
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> (they
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > > fairly easy
> > to
> > > > > > update as
> > > > > > > >> the
> > > > > > > >> > > email
> > > > > > > >> > > > > > >> > > discussion
> > > > > > > >> > > > > > >> > > > > > > > > > progresses).
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > >
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > > There was a
> > > > > > suggestion to
> > > > > > > >> > try
> > > > > > > >> > > > > > Markdown
> > > > > > > >> > > > > > >> > > (with
> > > > > > > >> > > > > > >> > > > > PRs)
> > > > > > > >> > > > > > >> > > > > > > for
> > > > > > > >> > > > > > >> > > > > > > > > > > > proposals
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> [1]
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> ...
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > > feel free
> to
> > give
> > > > > it
> > > > > > a
> > > > > > > >> try
> > > > > > > >> > if
> > > > > > > >> > > you
> > > > > > > >> > > > > > are
> > > > > > > >> > > > > > >> > > > > comfortable
> > > > > > > >> > > > > > >> > > > > > > > with
> > > > > > > >> > > > > > >> > > > > > > > > > > that.
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > >
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > > 2. Could
> you
> > > > > clarify
> > > > > > > >> whether
> > > > > > > >> > > you
> > > > > > > >> > > > > > mean
> > > > > > > >> > > > > > >> end
> > > > > > > >> > > > > > >> > > > user
> > > > > > > >> > > > > > >> > > > > > > > > utilities
> > > > > > > >> > > > > > >> > > > > > > > > > or
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> admin
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> user
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > > utilities?
> > In the
> > > > > > latter
> > > > > > > >> > case
> > > > > > > >> > > > > those
> > > > > > > >> > > > > > >> might
> > > > > > > >> > > > > > >> > > be
> > > > > > > >> > > > > > >> > > > > more
> > > > > > > >> > > > > > >> > > > > > > > > > suitable
> > > > > > > >> > > > > > >> > > > > > > > > > > > for
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> the
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> Admin
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > > CLI (java)
> > not
> > > > the
> > > > > > Python
> > > > > > > >> > CLI,
> > > > > > > >> > > > > IMHO.
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > >
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > > Why would
> > these
> > > > > > > >> utilities be
> > > > > > > >> > > > > common
> > > > > > > >> > > > > > >> with
> > > > > > > >> > > > > > >> > > > > events?
> > > > > > > >> > > > > > >> > > > > > > > IMHO,
> > > > > > > >> > > > > > >> > > > > > > > > > > event
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> use
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> cases
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > are
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > > distinct
> from
> > > > > > scan/commit
> > > > > > > >> > > metrics.
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > >
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > > 3. I'd
> prefer
> > > > > > separating
> > > > > > > >> > > metrics
> > > > > > > >> > > > > > >> > > persistence
> > > > > > > >> > > > > > >> > > > > from
> > > > > > > >> > > > > > >> > > > > > > > > > MetaStore
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> persistence
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > at
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > > the code
> > level,
> > > > so
> > > > > > that
> > > > > > > >> they
> > > > > > > >> > > could
> > > > > > > >> > > > > > be
> > > > > > > >> > > > > > >> > mixed
> > > > > > > >> > > > > > >> > > > and
> > > > > > > >> > > > > > >> > > > > > > > matched
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> independently.
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > The
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > > separate
> > > > datasource
> > > > > > > >> question
> > > > > > > >> > > will
> > > > > > > >> > > > > > >> become
> > > > > > > >> > > > > > >> > a
> > > > > > > >> > > > > > >> > > > > > > non-issue
> > > > > > > >> > > > > > >> > > > > > > > > with
> > > > > > > >> > > > > > >> > > > > > > > > > > > that
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> approach,
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > I
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > > guess.
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > >
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > > The
> > rationale for
> > > > > > > >> separating
> > > > > > > >> > > scan
> > > > > > > >> > > > > > >> metrics
> > > > > > > >> > > > > > >> > > and
> > > > > > > >> > > > > > >> > > > > > > > metastore
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> persistence
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> is
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > that
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > > "cascading
> > > > deletes"
> > > > > > > >> between
> > > > > > > >> > > them
> > > > > > > >> > > > > are
> > > > > > > >> > > > > > >> > hardly
> > > > > > > >> > > > > > >> > > > > ever
> > > > > > > >> > > > > > >> > > > > > > > > > required.
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> Furthermore,
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > the
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > > data and
> > query
> > > > > > patterns
> > > > > > > >> are
> > > > > > > >> > > very
> > > > > > > >> > > > > > >> > different
> > > > > > > >> > > > > > >> > > so
> > > > > > > >> > > > > > >> > > > > > > > different
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> technologies
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > might
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > > be
> > beneficial in
> > > > > each
> > > > > > > >> case.
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > >
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > > [1]
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> >
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>>
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >>
> > > > > > > >> > > > > > >> > > > > > > > > > > > >>
> > > > > > > >> > > > > > >> > > > > > > > > > > >
> > > > > > > >> > > > > > >> > > > > > > > > > >
> > > > > > > >> > > > > > >> > > > > > > > > >
> > > > > > > >> > > > > > >> > > > > > > > >
> > > > > > > >> > > > > > >> > > > > > > >
> > > > > > > >> > > > > > >> > > > > > >
> > > > > > > >> > > > > > >> > > > > >
> > > > > > > >> > > > > > >> > > > >
> > > > > > > >> > > > > > >> > > >
> > > > > > > >> > > > > > >> > >
> > > > > > > >> > > > > > >> >
> > > > > > > >> > > > > > >>
> > > > > > > >> > > > > >
> > > > > > > >> > > > >
> > > > > > > >> > >
> > > > > > > >> >
> > > > > > > >>
> > > > > >
> > > > >
> > > >
> >
> https://urldefense.com/v3/__https://lists.apache.org/thread/yto2wp982t43h1mqjwnslswhws5z47cy__;!!Iz9xO38YGHZK!5EuyFFkk3vhRWVIRvQAWBSQfpJkTMA9HxugzDwXmN0LPPqhEFxYkFRGVhtb8AqUwXtDh2OplcMnbMDHKxYDakNU$
> > > > > > > >> > >> > > >> > > > > > > > > > > > >> >> >>> > >
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > > Cheers,
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > > Dmitri.
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > >
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > > On Fri, Feb
> > 27,
> > > > > 2026
> > > > > > at
> > > > > > > >> > > 6:19 PM
> > > > > > > >> > > > > > Anand
> > > > > > > >> > > > > > >> > Kumar
> > > > > > > >> > > > > > >> > > > > > > Sankaran
> > > > > > > >> > > > > > >> > > > > > > > > via
> > > > > > > >> > > > > > >> > > > > > > > > > > dev
> > > > > > > >> > > > > > >> > > > > > > > > > > > <
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > >
> > > > > > dev@polaris.apache.org>
> > > > > > > >> > > wrote:
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > >
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > > > Thanks
> all.
> > > > This
> > > > > > PR is
> > > > > > > >> > > merged
> > > > > > > >> > > > > now.
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > > >
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > > > Here are
> > the
> > > > > > follow-up
> > > > > > > >> > > features
> > > > > > > >> > > > > /
> > > > > > > >> > > > > > >> work
> > > > > > > >> > > > > > >> > > > > needed.
> > > > > > > >> > > > > > >> > > > > > > > These
> > > > > > > >> > > > > > >> > > > > > > > > > > were
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> all
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> part of
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > > the
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > > > merged PR
> > at
> > > > some
> > > > > > > >> point in
> > > > > > > >> > > time
> > > > > > > >> > > > > > and
> > > > > > > >> > > > > > >> > were
> > > > > > > >> > > > > > >> > > > > > removed
> > > > > > > >> > > > > > >> > > > > > > to
> > > > > > > >> > > > > > >> > > > > > > > > > > reduce
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> scope.
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > > >
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > > > Please
> let
> > me
> > > > > know
> > > > > > what
> > > > > > > >> > you
> > > > > > > >> > > > > think.
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > > >
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > > >
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > > >   1.  A
> > REST
> > > > API
> > > > > to
> > > > > > > >> > paginate
> > > > > > > >> > > > > > through
> > > > > > > >> > > > > > >> > > table
> > > > > > > >> > > > > > >> > > > > > > metrics.
> > > > > > > >> > > > > > >> > > > > > > > > > This
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> will be
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > non-IRC
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > > > standard
> > > > > addition.
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > > >   2.
> > Utilities
> > > > > for
> > > > > > > >> > > managing old
> > > > > > > >> > > > > > >> > records,
> > > > > > > >> > > > > > >> > > > > > should
> > > > > > > >> > > > > > >> > > > > > > be
> > > > > > > >> > > > > > >> > > > > > > > > > > common
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> with
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> events.
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > > > There was
> > some
> > > > > > > >> discussion
> > > > > > > >> > > that
> > > > > > > >> > > > > it
> > > > > > > >> > > > > > >> > belongs
> > > > > > > >> > > > > > >> > > > to
> > > > > > > >> > > > > > >> > > > > > the
> > > > > > > >> > > > > > >> > > > > > > > CLI.
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > > >   3.
> > Separate
> > > > > > > >> datasource
> > > > > > > >> > > > > > (metrics,
> > > > > > > >> > > > > > >> > > events,
> > > > > > > >> > > > > > >> > > > > > even
> > > > > > > >> > > > > > >> > > > > > > > > other
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> tables?).
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > > >
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > > >
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > > > Anything
> > else?
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > > >
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > > > -
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > > > Anand
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > > >
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > > >
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > >
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> >
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> >
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>>
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >>
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >
> > > > > > > >> > > > > > >> > > > > > > > > > > > >>
> > > > > > > >> > > > > > >> > > > > > > > > > > > >
> > > > > > > >> > > > > > >> > > > > > > > > > > >
> > > > > > > >> > > > > > >> > > > > > > > > > >
> > > > > > > >> > > > > > >> > > > > > > > > >
> > > > > > > >> > > > > > >> > > > > > > > >
> > > > > > > >> > > > > > >> > > > > > > >
> > > > > > > >> > > > > > >> > > > > > >
> > > > > > > >> > > > > > >> > > > > >
> > > > > > > >> > > > > > >> > > > >
> > > > > > > >> > > > > > >> > > >
> > > > > > > >> > > > > > >> > >
> > > > > > > >> > > > > > >> >
> > > > > > > >> > > > > > >>
> > > > > > > >> > > > > > >
> > > > > > > >> > > > > >
> > > > > > > >> > > > >
> > > > > > > >> > >
> > > > > > > >> >
> > > > > > > >> >
> > > > > > > >>
> > > > > > > >
> > > > > >
> > > > >
> > > >
> >
>

Re: Proposal for REST endpoints for table metrics and events

[Yufei Gu <fl...@gmail.com>]

I’ve added another round of comments on PR #4115
<https://github.com/apache/polaris/pull/4115>. I don’t think this PR is
ready to merge yet. The metrics query spec still needs to align with the
existing Polaris catalog API conventions:

   -

   Update it for OpenAPI 3.1.1, following #5819.
   -

   Move it under spec/polaris-catalog-apis and follow the existing specs’
   structure, including shared prefix handling rather than a separate server
   definition.
   -

   Align the 400, 403, and 404 responses with the existing catalog APIs’
   error response definitions.
   -

   Clarify the separate 0.1.0 version and whether the additional license
   metadata is needed.

I also asked about the access-control documentation change and the use case
for actor. The latter is non-blocking, but its current description could
confuse the report-submitting principal with the user who initiated the
operation. I’d prefer omitting it unless there’s a concrete need, or
defining its meaning precisely.
Yufei


On Wed, Oct 21, 2026 at 9:13 AM Yufei Gu <fl...@gmail.com> wrote:

> Agreed. Thanks Dmitri! I will take a look at the PR today.
>
> Yufei
>
>
> On Wed, Oct 21, 2026 at 8:59 AM Dmitri Bourlatchkov <di...@apache.org>
> wrote:
>
>> Hi All,
>>
>> Heads up: it looks like the "Metrics Architecture" meeting is no longer
>> necessary. It will be removed from the Polaris calendar presently.
>>
>> Cheers,
>> Dmitri.
>>
>> On Tue, Oct 20, 2026 at 12:16 PM Dmitri Bourlatchkov <di...@apache.org>
>> wrote:
>>
>> > Hi All,
>> >
>> > PR [4115] has been approved for two weeks without any follow-up
>> comments.
>> >
>> > I'm going to merge it on Oct 22 unless fresh concerns are raised.
>> >
>> > [4115] https://github.com/apache/polaris/pull/4115
>> >
>> > Cheers,
>> > Dmitri.
>> >
>> > I
>> >
>> > On Wed, Oct 7, 2026 at 9:45 PM Yufei Gu <fl...@gmail.com> wrote:
>> >
>> >> Thanks for the reminder, Dmitri! I will take a look.
>> >>
>> >> Yufei
>> >>
>> >>
>> >> On Wed, Oct 7, 2026 at 5:37 PM Dmitri Bourlatchkov <di...@apache.org>
>> >> wrote:
>> >>
>> >> > Hi All,
>> >> >
>> >> > Metrics grants have been merged as a separate PR.
>> >> >
>> >> > Now [4115] is ready for one more review round. IMHO, it is ready to
>> >> merge.
>> >> >
>> >> > [4115] https://github.com/apache/polaris/pull/4115
>> >> >
>> >> > Cheers,
>> >> > Dmitri.
>> >> >
>> >> > On Fri, Oct 2, 2026 at 1:01 AM Jean-Baptiste Onofré <jb@nanthrax.net
>> >
>> >> > wrote:
>> >> >
>> >> > > Hi all,
>> >> > >
>> >> > > I agree with Yufei: I would keep TABLE_READ_METRICS as a separate,
>> >> > > explicit grant, not implied by TABLE_FULL_METADATA.
>> >> > >
>> >> > > I believe metrics reports are closer to an activity log than to
>> table
>> >> > > metadata. I also think it's consistent with what we have today, as
>> >> > > reporting scan and commit metrics is already with TABLE_READ_DATA
>> and
>> >> > > TABLE_WRITE_DATA, not by "metadata" privilege.
>> >> > >
>> >> > > For the troubleshooting use case, the cost is just one additional
>> >> > > grant, I think it's acceptable :) I would still let
>> >> > > CATALOG_MANAGE_CONTENT implies it, as that privilege already
>> implies
>> >> > > TABLE_READ_DATA.
>> >> > >
>> >> > > Concretely for PR #5669, I propose:
>> >> > > - to remove the "Implied by TABLE_FULL_METADATA" statement from the
>> >> > > javadoc and CHANGELOG
>> >> > > - to add SUPER_PRIVILEGES entry in the PR, so the grant change is
>> >> > > self-contained
>> >> > > - to drop ICEBERG_TABLE restriction.
>> >> > > - to update access-control.md documenting TABLE_READ_METRICS, etc
>> >> > >
>> >> > > I will comment directly in the PR.
>> >> > >
>> >> > > Regards
>> >> > > JB
>> >> > >
>> >> > > On Fri, Oct 2, 2026 at 3:40 AM Yufei Gu <fl...@gmail.com>
>> wrote:
>> >> > > >
>> >> > > > Thanks Dmitri for the reminder.
>> >> > > >
>> >> > > > PR #5659 <https://github.com/apache/polaris/pull/5659> adds
>> >> > > > TABLE_READ_METRICS and says it is implied by TABLE_FULL_METADATA.
>> >> > Should
>> >> > > > that be the default?
>> >> > > >
>> >> > > > I’m leaning toward keeping it separate. Metrics describe
>> workloads
>> >> > (Spark
>> >> > > > ETL, Flink streaming, etc) using a table and can include query
>> >> filters,
>> >> > > so
>> >> > > > someone managing table metadata may not need access to other
>> users’
>> >> > query
>> >> > > > activity.
>> >> > > >
>> >> > > > Including it could make troubleshooting easier for table admins,
>> >> > though.
>> >> > > > Are there use cases that favor automatic access over an explicit
>> >> grant?
>> >> > > > Yufei
>> >> > > >
>> >> > > >
>> >> > > > On Wed, Sep 30, 2026 at 1:57 PM Dmitri Bourlatchkov <
>> >> dimas@apache.org>
>> >> > > > wrote:
>> >> > > >
>> >> > > > > Hi Yufei,
>> >> > > > >
>> >> > > > > Here's the separate PR for Metrics-related grants:
>> >> > > > > https://github.com/apache/polaris/pull/5659
>> >> > > > >
>> >> > > > > Please review it to unblock this contribution.
>> >> > > > >
>> >> > > > > Thanks,
>> >> > > > > Dmitri.
>> >> > > > >
>> >> > > > > On Wed, Sep 30, 2026 at 1:05 PM Yufei Gu <flyrain000@gmail.com
>> >
>> >> > wrote:
>> >> > > > >
>> >> > > > > > From the previous email, we agreed to split the REST change
>> from
>> >> > the
>> >> > > > > grant
>> >> > > > > > change. I think the PR is ready to go if we only keep the
>> grant
>> >> > > changes.
>> >> > > > > >
>> >> > > > > > Yufei
>> >> > > > > >
>> >> > > > > >
>> >> > > > > > On Tue, Sep 29, 2026 at 9:26 PM Jean-Baptiste Onofré <
>> >> > > jb@nanthrax.net>
>> >> > > > > > wrote:
>> >> > > > > >
>> >> > > > > > > Hi guys,
>> >> > > > > > >
>> >> > > > > > > I agree with Dmitri: +1 to merge PR #4115 on Oct 1
>> (tomorrow).
>> >> > > > > > >
>> >> > > > > > > From my side, all outstanding feedback has been addressed,
>> >> > > including
>> >> > > > > > > the final documentation points raised by EJ regarding the
>> >> > empty-200
>> >> > > > > > > NoOp behavior and reconstructed payload description. The
>> >> branch
>> >> > is
>> >> > > > > > > clean, up to date with main, and CI is green.
>> >> > > > > > >
>> >> > > > > > > Once this lands, we can follow up with PR #4756 for the
>> JDBC
>> >> > query
>> >> > > SPI
>> >> > > > > > > implementation.
>> >> > > > > > >
>> >> > > > > > > Regards
>> >> > > > > > > JB
>> >> > > > > > >
>> >> > > > > > > On Wed, Sep 30, 2026 at 1:53 AM Dmitri Bourlatchkov <
>> >> > > dimas@apache.org>
>> >> > > > > > > wrote:
>> >> > > > > > > >
>> >> > > > > > > > Hi All,
>> >> > > > > > > >
>> >> > > > > > > > Any remaining blockers on [4115]?
>> >> > > > > > > >
>> >> > > > > > > > Given that the review was very long, please restate any
>> >> > remaining
>> >> > > > > > > concerns
>> >> > > > > > > > (here or in GH).
>> >> > > > > > > >
>> >> > > > > > > > Otherwise, I propose merging [4115] on Oct 1.
>> >> > > > > > > >
>> >> > > > > > > > [4115] https://github.com/apache/polaris/pull/4115
>> >> > > > > > > >
>> >> > > > > > > > Thanks,
>> >> > > > > > > > Dmitri.
>> >> > > > > > > >
>> >> > > > > > > > On Wed, Sep 23, 2026 at 7:53 PM Dmitri Bourlatchkov <
>> >> > > > > dimas@apache.org>
>> >> > > > > > > > wrote:
>> >> > > > > > > >
>> >> > > > > > > > > Hi EJ,
>> >> > > > > > > > >
>> >> > > > > > > > > Thanks for the meeting recap!
>> >> > > > > > > > >
>> >> > > > > > > > > All:
>> >> > > > > > > > >
>> >> > > > > > > > > Worth noting: PR [4115] also contains Management API
>> >> changes
>> >> > > that
>> >> > > > > > were
>> >> > > > > > > not
>> >> > > > > > > > > anticipated initially and did not get an API change
>> review
>> >> > > until
>> >> > > > > now.
>> >> > > > > > > These
>> >> > > > > > > > > are small changes to add new grant types for metrics
>> >> RBAC..
>> >> > > > > > > > >
>> >> > > > > > > > > I've reviewed and approved PR [4115] in GH.
>> >> > > > > > > > >
>> >> > > > > > > > > [4115] https://github.com/apache/polaris/pull/4115
>> >> > > > > > > > >
>> >> > > > > > > > > Cheers,
>> >> > > > > > > > > Dmitri.
>> >> > > > > > > > >
>> >> > > > > > > > > On Wed, Aug 12, 2026 at 5:21 PM EJ Wang <
>> >> > > > > > > ej.wang.dev.interest@gmail.com>
>> >> > > > > > > > > wrote:
>> >> > > > > > > > >
>> >> > > > > > > > >> Hi folks,
>> >> > > > > > > > >>
>> >> > > > > > > > >> Notes from today's metrics architecture sync are in
>> the
>> >> > usual
>> >> > > doc:
>> >> > > > > > > > >>
>> >> > > > > > > > >>
>> >> > > > > > >
>> >> > > > > >
>> >> > > > >
>> >> > >
>> >> >
>> >>
>> https://docs.google.com/document/d/100h7c4damrUzVuquYbBHM0EvA4LSWuW2IT2dN_7nYVA/edit?pli=1&tab=t.18iv1m7p6i6b
>> >> > > > > > > > >>
>> >> > > > > > > > >> Highlights:
>> >> > > > > > > > >>
>> >> > > > > > > > >> Module placement. core/ now holds the entity data
>> model
>> >> > only.
>> >> > > A
>> >> > > > > new
>> >> > > > > > > spi/
>> >> > > > > > > > >> takes every other shared contract: feature SPIs,
>> feature
>> >> > > non-SPI
>> >> > > > > > > > >> contracts,
>> >> > > > > > > > >> substrate SPIs, substrate non-SPI contracts, and
>> >> > > contract-required
>> >> > > > > > > types.
>> >> > > > > > > > >> That settles the question PR #5204 reopened, by
>> placing
>> >> into
>> >> > > a new
>> >> > > > > > SPI
>> >> > > > > > > > >> module. Note that it changes slightly towards the
>> >> referenced
>> >> > > WIP
>> >> > > > > RFC
>> >> > > > > > > [1]
>> >> > > > > > > > >> recommendations: what proposed to place in core/, now
>> >> should
>> >> > > go to
>> >> > > > > > > spi/
>> >> > > > > > > > >> except for the entity data model.
>> >> > > > > > > > >>
>> >> > > > > > > > >> Metrics PRs. #5068 merged. #5204 needs reshaping onto
>> the
>> >> > > split.
>> >> > > > > > > #4115 and
>> >> > > > > > > > >> #4756 stay open with action on the author to follow
>> up on
>> >> > > review
>> >> > > > > > > comments.
>> >> > > > > > > > >>
>> >> > > > > > > > >> [1]
>> >> > > > > > > > >>
>> >> > > > > > > > >>
>> >> > > > > > >
>> >> > > > > >
>> >> > > > >
>> >> > >
>> >> >
>> >>
>> https://docs.google.com/document/d/1mj3vGpix9zTDTFFJKtWT5ZxI3PTGI1YNb5xFqyWrOkc/edit?tab=t.0
>> >> > > > > > > > >>
>> >> > > > > > > > >> On Wed, Jul 15, 2026 at 11:01 AM Anand Kumar Sankaran
>> via
>> >> > dev
>> >> > > <
>> >> > > > > > > > >> dev@polaris.apache.org> wrote:
>> >> > > > > > > > >>
>> >> > > > > > > > >> > Thanks EJ.  My plan is to introduce PR0 in a new
>> >> branch /
>> >> > > PR and
>> >> > > > > > > > >> repurpose
>> >> > > > > > > > >> > #4115 and #4756 for PR1 and PR2 to maintain all the
>> >> > valuable
>> >> > > > > > > discussions
>> >> > > > > > > > >> > that happened there.
>> >> > > > > > > > >> >
>> >> > > > > > > > >> > From: EJ Wang <ej...@gmail.com>
>> >> > > > > > > > >> > Date: Wednesday, July 15, 2026 at 10:41 AM
>> >> > > > > > > > >> > To: dev@polaris.apache.org <de...@polaris.apache.org>
>> >> > > > > > > > >> > Subject: Re: Proposal for REST endpoints for table
>> >> metrics
>> >> > > and
>> >> > > > > > > events
>> >> > > > > > > > >> >
>> >> > > > > > > > >> > Hi folks, Thanks again for the discussion today. I
>> >> updated
>> >> > > the
>> >> > > > > > sync
>> >> > > > > > > doc
>> >> > > > > > > > >> > with notes from today's metrics sync: https:
>> >> //urldefense.
>> >> > > > > > > > >> com/v3/__https:
>> >> > > > > > > > >> > //docs. google.
>> >> > > > > > > > >> >
>> >> > > > > > > > >>
>> >> > > > > > >
>> >> > > > > >
>> >> > > > >
>> >> > >
>> >> >
>> >>
>> com/document/d/100h7c4damrUzVuquYbBHM0EvA4LSWuW2IT2dN_7nYVA/edit?pli=1&tab=t.
>> >> > > > > > > > >> >
>> >> > > > > > > > >>
>> >> > > > > > >
>> >> > > > > >
>> >> > > > >
>> >> > >
>> >> >
>> >>
>> 2ahd7wze5f__;!!Iz9xO38YGHZK!4rnlVIqBQQQ6xpGe1PMV58edOS_cGrKoqdfhw4qSp_JQCavvEjaxXk0OwGPnxyEcO60AzKQK_XDtp7H3DdkFXcrXJwAz1g0$
>> >> > > > > > > > >> >
>> >> > > > > > > > >> >
>> >> > > > > > > > >> > Hi folks,
>> >> > > > > > > > >> >
>> >> > > > > > > > >> > Thanks again for the discussion today. I updated the
>> >> sync
>> >> > > doc
>> >> > > > > with
>> >> > > > > > > notes
>> >> > > > > > > > >> > from today's metrics sync:
>> >> > > > > > > > >> >
>> >> > > > > > > > >> >
>> >> > > > > > > > >>
>> >> > > > > > >
>> >> > > > > >
>> >> > > > >
>> >> > >
>> >> >
>> >>
>> https://urldefense.com/v3/__https://docs.google.com/document/d/100h7c4damrUzVuquYbBHM0EvA4LSWuW2IT2dN_7nYVA/edit?pli=1&tab=t.2ahd7wze5f__;!!Iz9xO38YGHZK!4rnlVIqBQQQ6xpGe1PMV58edOS_cGrKoqdfhw4qSp_JQCavvEjaxXk0OwGPnxyEcO60AzKQK_XDtp7H3DdkFXcrXJwAz1g0$
>> >> > > > > > > > >> >
>> >> > > > > > > > >> > See highlights below:
>> >> > > > > > > > >> >
>> >> > > > > > > > >> > In previous meetings, EJ proposed a two prs framing
>> to
>> >> > push
>> >> > > > > > forward
>> >> > > > > > > the
>> >> > > > > > > > >> > metrics query work in polaris. Dmitri pointed out a
>> >> subtle
>> >> > > point
>> >> > > > > > > that
>> >> > > > > > > > >> pr2
>> >> > > > > > > > >> > is currently also trying to do the persistence
>> related
>> >> > code
>> >> > > > > > > placement
>> >> > > > > > > > >> > refactor. For the sake of straightforward review
>> >> process,
>> >> > > EJ,
>> >> > > > > > > Dmitri and
>> >> > > > > > > > >> > Anand (the author) agreed to separate the
>> persistence
>> >> > > > > refactoring
>> >> > > > > > > into
>> >> > > > > > > > >> pr0,
>> >> > > > > > > > >> > and reframing the pr scopes as follow:
>> >> > > > > > > > >> >
>> >> > > > > > > > >> >    1.
>> >> > > > > > > > >> >
>> >> > > > > > > > >> >    Pr0: refactor irc metrics spi
>> >> > > > > > > > >> >    1.
>> >> > > > > > > > >> >
>> >> > > > > > > > >> >       SPI should stay in core/ with metric type
>> >> enveloped
>> >> > > > > > > > >> >       2.
>> >> > > > > > > > >> >
>> >> > > > > > > > >> >       default spi impl, no-op, in extensions/ -
>> default
>> >> > > > > selection
>> >> > > > > > at
>> >> > > > > > > > >> >       runtime/
>> >> > > > > > > > >> >       3.
>> >> > > > > > > > >> >
>> >> > > > > > > > >> >       move metrics persistence and correspondings to
>> >> > > extensions/
>> >> > > > > > as
>> >> > > > > > > an
>> >> > > > > > > > >> >       alternative SPI impl
>> >> > > > > > > > >> >       2.
>> >> > > > > > > > >> >
>> >> > > > > > > > >> >    Pr1: rest api for query
>> >> > > > > > > > >> >
>> >> > > > > > > > >> >
>> >> > > > > > > > >>
>> >> > > > > > >
>> >> > > > > >
>> >> > > > >
>> >> > >
>> >> >
>> >>
>> https://urldefense.com/v3/__https://github.com/apache/polaris/pull/4115/changes__;!!Iz9xO38YGHZK!4rnlVIqBQQQ6xpGe1PMV58edOS_cGrKoqdfhw4qSp_JQCavvEjaxXk0OwGPnxyEcO60AzKQK_XDtp7H3DdkFXcrXFCbEbRk$
>> >> > > > > > > > >> >    1.
>> >> > > > > > > > >> >
>> >> > > > > > > > >> >       rest api spec in extensions/ as optional spec
>> >> > > > > > > > >> >       2.
>> >> > > > > > > > >> >
>> >> > > > > > > > >> >       SPI in extensions/ as optional spec spi
>> >> > > > > > > > >> >       3.
>> >> > > > > > > > >> >
>> >> > > > > > > > >> >       default spi impl in extensions/ no-op
>> >> > > > > > > > >> >       4.
>> >> > > > > > > > >> >
>> >> > > > > > > > >> >       handler related in runtime/ - thin adaptor
>> from
>> >> http
>> >> > > shape
>> >> > > > > > to
>> >> > > > > > > spi
>> >> > > > > > > > >> >       shape
>> >> > > > > > > > >> >       3.
>> >> > > > > > > > >> >
>> >> > > > > > > > >> >    Pr2: jdbc impl for query
>> >> > > > > > > > >> >
>> >> > > > > > > > >>
>> >> > > > > > >
>> >> > > > > >
>> >> > > > >
>> >> > >
>> >> >
>> >>
>> https://urldefense.com/v3/__https://github.com/apache/polaris/pull/4756__;!!Iz9xO38YGHZK!4rnlVIqBQQQ6xpGe1PMV58edOS_cGrKoqdfhw4qSp_JQCavvEjaxXk0OwGPnxyEcO60AzKQK_XDtp7H3DdkFXcrXRVMBzqE$
>> >> > > > > > > > >> >    1.
>> >> > > > > > > > >> >
>> >> > > > > > > > >> >       alternative spi impl in extensions/ -> query
>> with
>> >> > jdbc
>> >> > > > > > > > >> >
>> >> > > > > > > > >> >
>> >> > > > > > > > >> > Thanks,
>> >> > > > > > > > >> > -ej
>> >> > > > > > > > >> >
>> >> > > > > > > > >> > On Wed, Jul 1, 2026 at 8:01 AM Alexandre Dutra <
>> >> > > > > adutra@apache.org
>> >> > > > > > >
>> >> > > > > > > > >> wrote:
>> >> > > > > > > > >> >
>> >> > > > > > > > >> > > Hi all,
>> >> > > > > > > > >> > >
>> >> > > > > > > > >> > > > While the current implementation stores metrics
>> in
>> >> the
>> >> > > > > > > metastore,
>> >> > > > > > > > >> other
>> >> > > > > > > > >> > > deployments
>> >> > > > > > > > >> > > may choose a KV store, a dedicated metrics
>> >> warehouse, or
>> >> > > > > another
>> >> > > > > > > > >> backend
>> >> > > > > > > > >> > > entirely. In those cases, exposing a Polaris REST
>> API
>> >> > > over a
>> >> > > > > > > specific
>> >> > > > > > > > >> > > schema feels like coupling the API to an
>> >> implementation
>> >> > > > > detail.
>> >> > > > > > > > >> > >
>> >> > > > > > > > >> > > I wouldn't say to an implementation detail, but
>> >> rather
>> >> > to
>> >> > > a
>> >> > > > > > > particular
>> >> > > > > > > > >> > > setup / configuration. I think that's fine. If you
>> >> don't
>> >> > > store
>> >> > > > > > > > >> > > metrics, you don't read metrics. Similar
>> situations
>> >> will
>> >> > > > > > certainly
>> >> > > > > > > > >> > > happen in the future, e.g. if we ever introduce an
>> >> API
>> >> > to
>> >> > > read
>> >> > > > > > > events,
>> >> > > > > > > > >> > > that would only work if events are being
>> persisted in
>> >> > the
>> >> > > > > first
>> >> > > > > > > place.
>> >> > > > > > > > >> > > Again I think that's a fair trade-off.
>> >> > > > > > > > >> > >
>> >> > > > > > > > >> > > Thanks,
>> >> > > > > > > > >> > > Alex
>> >> > > > > > > > >> > >
>> >> > > > > > > > >> > > On Wed, Jul 1, 2026 at 7:23 AM Dmitri
>> Bourlatchkov <
>> >> > > > > > > dimas@apache.org>
>> >> > > > > > > > >> > > wrote:
>> >> > > > > > > > >> > > >
>> >> > > > > > > > >> > > > Hi Yufei,
>> >> > > > > > > > >> > > >
>> >> > > > > > > > >> > > > From my POV, it is fine for Polaris to have
>> >> multiple
>> >> > > REST
>> >> > > > > API
>> >> > > > > > > > >> surfaces
>> >> > > > > > > > >> > > with
>> >> > > > > > > > >> > > > different features and different degrees of
>> >> > "readiness".
>> >> > > > > Each
>> >> > > > > > > API
>> >> > > > > > > > >> can
>> >> > > > > > > > >> > > > evolve fairly independently. With proper
>> >> > modularization
>> >> > > > > (which
>> >> > > > > > > Anand
>> >> > > > > > > > >> > > did),
>> >> > > > > > > > >> > > > changes to the main Polaris code and these extra
>> >> > modules
>> >> > > > > > should
>> >> > > > > > > > >> > normally
>> >> > > > > > > > >> > > be
>> >> > > > > > > > >> > > > non-intrusive to each other.
>> >> > > > > > > > >> > > >
>> >> > > > > > > > >> > > > I assume Anand is going to be the first user of
>> >> this
>> >> > > API.
>> >> > > > > > > However,
>> >> > > > > > > > >> if
>> >> > > > > > > > >> > we
>> >> > > > > > > > >> > > do
>> >> > > > > > > > >> > > > not merge it we'll never know whether other
>> users
>> >> are
>> >> > > > > > > interested. I
>> >> > > > > > > > >> do
>> >> > > > > > > > >> > > not
>> >> > > > > > > > >> > > > think many end users read the dev ML, but they
>> >> > certainly
>> >> > > > > read
>> >> > > > > > > the
>> >> > > > > > > > >> docs,
>> >> > > > > > > > >> > > > code and release notes.
>> >> > > > > > > > >> > > >
>> >> > > > > > > > >> > > > There is no risk for Polaris as a project in
>> >> exposing
>> >> > > this
>> >> > > > > > > > >> experimental
>> >> > > > > > > > >> > > > API. If usage dwindles over time, we can remove
>> >> it. If
>> >> > > it
>> >> > > > > > gains
>> >> > > > > > > a
>> >> > > > > > > > >> > > > considerable user / contributor base, we will
>> >> promote
>> >> > > it to
>> >> > > > > a
>> >> > > > > > > > >> > > "first-class"
>> >> > > > > > > > >> > > > (non-beta) API.
>> >> > > > > > > > >> > > >
>> >> > > > > > > > >> > > > Cheers,
>> >> > > > > > > > >> > > > Dmitri.
>> >> > > > > > > > >> > > >
>> >> > > > > > > > >> > > > On Tue, Jun 30, 2026 at 7:53 PM Yufei Gu <
>> >> > > > > > flyrain000@gmail.com>
>> >> > > > > > > > >> wrote:
>> >> > > > > > > > >> > > >
>> >> > > > > > > > >> > > > > Hi all,
>> >> > > > > > > > >> > > > >
>> >> > > > > > > > >> > > > > Thanks for driving this work forward. Building
>> >> this
>> >> > > as a
>> >> > > > > POC
>> >> > > > > > > makes
>> >> > > > > > > > >> > > perfect
>> >> > > > > > > > >> > > > > sense if the goal is to validate the idea and
>> >> gather
>> >> > > > > > feedback.
>> >> > > > > > > > >> Should
>> >> > > > > > > > >> > > we
>> >> > > > > > > > >> > > > > merge a POC into the main branch before
>> agreeing
>> >> on
>> >> > > the
>> >> > > > > > > overall
>> >> > > > > > > > >> > > direction?
>> >> > > > > > > > >> > > > > To me, that feels like a separate discussion
>> from
>> >> > > whether
>> >> > > > > > the
>> >> > > > > > > > >> > > > > implementation itself looks reasonable.
>> >> > > > > > > > >> > > > >
>> >> > > > > > > > >> > > > > One concern that I don't think has been fully
>> >> > > addressed is
>> >> > > > > > > > >> whether we
>> >> > > > > > > > >> > > need
>> >> > > > > > > > >> > > > > to expose metrics consumption through a
>> Polaris
>> >> REST
>> >> > > > > > endpoint
>> >> > > > > > > at
>> >> > > > > > > > >> all.
>> >> > > > > > > > >> > > > >
>> >> > > > > > > > >> > > > > My concern is less about the API itself and
>> more
>> >> > > about the
>> >> > > > > > > > >> boundary
>> >> > > > > > > > >> > > Polaris
>> >> > > > > > > > >> > > > > should own. The proposed REST API is tied to a
>> >> > > particular
>> >> > > > > > > > >> persistence
>> >> > > > > > > > >> > > > > schema, but that schema is just one possible
>> >> > > > > implementation.
>> >> > > > > > > While
>> >> > > > > > > > >> > the
>> >> > > > > > > > >> > > > > current implementation stores metrics in the
>> >> > > metastore,
>> >> > > > > > other
>> >> > > > > > > > >> > > deployments
>> >> > > > > > > > >> > > > > may choose a KV store, a dedicated metrics
>> >> > warehouse,
>> >> > > or
>> >> > > > > > > another
>> >> > > > > > > > >> > > backend
>> >> > > > > > > > >> > > > > entirely. In those cases, exposing a Polaris
>> REST
>> >> > API
>> >> > > > > over a
>> >> > > > > > > > >> specific
>> >> > > > > > > > >> > > > > schema feels like coupling the API to an
>> >> > > implementation
>> >> > > > > > > detail.
>> >> > > > > > > > >> > > > >
>> >> > > > > > > > >> > > > > Polaris isn't intended to become a full
>> metrics
>> >> > > system.
>> >> > > > > I'd
>> >> > > > > > > > >> expect it
>> >> > > > > > > > >> > > to
>> >> > > > > > > > >> > > > > focus on producing metrics and let deployments
>> >> > choose
>> >> > > the
>> >> > > > > > most
>> >> > > > > > > > >> > > appropriate
>> >> > > > > > > > >> > > > > storage and consumption model for their
>> >> environment.
>> >> > > > > > > > >> > > > >
>> >> > > > > > > > >> > > > > I'm not opposed to exploring this direction,
>> but
>> >> we
>> >> > > should
>> >> > > > > > > first
>> >> > > > > > > > >> > align
>> >> > > > > > > > >> > > on
>> >> > > > > > > > >> > > > > whether exposing a REST API through Polaris is
>> >> the
>> >> > > right
>> >> > > > > > long
>> >> > > > > > > term
>> >> > > > > > > > >> > > approach
>> >> > > > > > > > >> > > > > before merging it into the main branch.
>> >> > > > > > > > >> > > > >
>> >> > > > > > > > >> > > > > Thanks,
>> >> > > > > > > > >> > > > > Yufei
>> >> > > > > > > > >> > > > >
>> >> > > > > > > > >> > > > > Yufei
>> >> > > > > > > > >> > > > >
>> >> > > > > > > > >> > > > >
>> >> > > > > > > > >> > > > > On Mon, Jun 29, 2026 at 9:27 AM Dmitri
>> >> Bourlatchkov
>> >> > <
>> >> > > > > > > > >> > dimas@apache.org>
>> >> > > > > > > > >> > > > > wrote:
>> >> > > > > > > > >> > > > >
>> >> > > > > > > > >> > > > > > Hi All,
>> >> > > > > > > > >> > > > > >
>> >> > > > > > > > >> > > > > > It looks like Anand addressed all action
>> items
>> >> in
>> >> > > > > [4115].
>> >> > > > > > > > >> > > > > >
>> >> > > > > > > > >> > > > > > Please review the latest state of the code.
>> >> The PR
>> >> > > LGTM
>> >> > > > > > and
>> >> > > > > > > I
>> >> > > > > > > > >> > > approved in
>> >> > > > > > > > >> > > > > > GH from my side.
>> >> > > > > > > > >> > > > > >
>> >> > > > > > > > >> > > > > > [4115]
>> >> > > > > > > > >> >
>> >> > > > > > > > >>
>> >> > > > > > >
>> >> > > > > >
>> >> > > > >
>> >> > >
>> >> >
>> >>
>> https://urldefense.com/v3/__https://github.com/apache/polaris/pull/4115__;!!Iz9xO38YGHZK!4rnlVIqBQQQ6xpGe1PMV58edOS_cGrKoqdfhw4qSp_JQCavvEjaxXk0OwGPnxyEcO60AzKQK_XDtp7H3DdkFXcrXtFRscKs$
>> >> > > > > > > > >> > > > > >
>> >> > > > > > > > >> > > > > > Thanks,
>> >> > > > > > > > >> > > > > > Dmitri.
>> >> > > > > > > > >> > > > > >
>> >> > > > > > > > >> > > > > > On Thu, Jun 25, 2026 at 1:56 PM Dmitri
>> >> > Bourlatchkov
>> >> > > <
>> >> > > > > > > > >> > > dimas@apache.org>
>> >> > > > > > > > >> > > > > > wrote:
>> >> > > > > > > > >> > > > > >
>> >> > > > > > > > >> > > > > > > Hi All,
>> >> > > > > > > > >> > > > > > >
>> >> > > > > > > > >> > > > > > > Recap of the discussion in today's
>> Community
>> >> > Sync:
>> >> > > > > > > > >> > > > > > >
>> >> > > > > > > > >> > > > > > > * The modular approach for Metrics SPI(s)
>> and
>> >> > > > > > > implementation
>> >> > > > > > > > >> > seems
>> >> > > > > > > > >> > > to
>> >> > > > > > > > >> > > > > > have
>> >> > > > > > > > >> > > > > > > community consensus (please comment if you
>> >> still
>> >> > > have
>> >> > > > > > > > >> concerns)
>> >> > > > > > > > >> > > > > > >
>> >> > > > > > > > >> > > > > > > * What modules / feature flags are
>> enabled in
>> >> > > Apache
>> >> > > > > > > Polaris
>> >> > > > > > > > >> > > images by
>> >> > > > > > > > >> > > > > > > default will be discussed later (when the
>> >> > feature
>> >> > > is
>> >> > > > > > > close to
>> >> > > > > > > > >> > > > > > completion).
>> >> > > > > > > > >> > > > > > >
>> >> > > > > > > > >> > > > > > > In any case, users will be able to turn
>> the
>> >> full
>> >> > > > > feature
>> >> > > > > > > > >> on/off
>> >> > > > > > > > >> > at
>> >> > > > > > > > >> > > > > their
>> >> > > > > > > > >> > > > > > > discretion, regardless of the default flag
>> >> > values.
>> >> > > > > > > > >> > > > > > >
>> >> > > > > > > > >> > > > > > > * It is generally preferable to have
>> smaller
>> >> PRs
>> >> > > and
>> >> > > > > > > introduce
>> >> > > > > > > > >> > new
>> >> > > > > > > > >> > > > > > > features gradually to simplify reviews.
>> >> > > > > > > > >> > > > > > >
>> >> > > > > > > > >> > > > > > > What this means for the Metrics API, IMHO
>> is:
>> >> > > > > > > > >> > > > > > >
>> >> > > > > > > > >> > > > > > > a) The scope of PR 4115 is probably ok as
>> it
>> >> > > stands
>> >> > > > > > right
>> >> > > > > > > > >> now. If
>> >> > > > > > > > >> > > > > someone
>> >> > > > > > > > >> > > > > > > wants to propose splitting it further,
>> please
>> >> > > comment.
>> >> > > > > > > > >> > > > > > >
>> >> > > > > > > > >> > > > > > > b) After 4115 is merged, Metrics
>> Persistence
>> >> PRs
>> >> > > will
>> >> > > > > > > follow,
>> >> > > > > > > > >> > with
>> >> > > > > > > > >> > > > > > smaller
>> >> > > > > > > > >> > > > > > > scope per PR as appropriate for the
>> nature of
>> >> > the
>> >> > > code
>> >> > > > > > > > >> changes.
>> >> > > > > > > > >> > > > > > >
>> >> > > > > > > > >> > > > > > > * The major concern was about exposing the
>> >> new
>> >> > > Metrics
>> >> > > > > > > REST
>> >> > > > > > > > >> API
>> >> > > > > > > > >> > to
>> >> > > > > > > > >> > > end
>> >> > > > > > > > >> > > > > > > users. The nature of this concern was
>> about
>> >> > > avoiding
>> >> > > > > the
>> >> > > > > > > > >> (false)
>> >> > > > > > > > >> > > > > > impression
>> >> > > > > > > > >> > > > > > > that the new API is a firm and adopted
>> >> Polaris
>> >> > > > > > > specification.
>> >> > > > > > > > >> > > > > > >
>> >> > > > > > > > >> > > > > > > I think this concern is reasonable.
>> >> > > > > > > > >> > > > > > >
>> >> > > > > > > > >> > > > > > > I propose that Anand update the
>> documentation
>> >> > > sections
>> >> > > > > > of
>> >> > > > > > > PR
>> >> > > > > > > > >> 4115
>> >> > > > > > > > >> > > to
>> >> > > > > > > > >> > > > > > > emphasise that the API is currently an
>> >> > > experimental /
>> >> > > > > > POC
>> >> > > > > > > API
>> >> > > > > > > > >> at
>> >> > > > > > > > >> > > this
>> >> > > > > > > > >> > > > > > time
>> >> > > > > > > > >> > > > > > > and is subject to change (including
>> breaking
>> >> > > > > changes). I
>> >> > > > > > > > >> believe
>> >> > > > > > > > >> > it
>> >> > > > > > > > >> > > > > > already
>> >> > > > > > > > >> > > > > > > has the "beta" label. So this request is
>> >> mostly
>> >> > > for
>> >> > > > > doc
>> >> > > > > > > > >> changes
>> >> > > > > > > > >> > to
>> >> > > > > > > > >> > > > > > clarify
>> >> > > > > > > > >> > > > > > > user expectations, since the "beta" label
>> >> alone
>> >> > > may
>> >> > > > > not
>> >> > > > > > be
>> >> > > > > > > > >> > > interpreted
>> >> > > > > > > > >> > > > > > the
>> >> > > > > > > > >> > > > > > > same way by all users.
>> >> > > > > > > > >> > > > > > >
>> >> > > > > > > > >> > > > > > > I hope this update will resolve all
>> remaining
>> >> > > concerns
>> >> > > > > > > with
>> >> > > > > > > > >> 4115
>> >> > > > > > > > >> > > and
>> >> > > > > > > > >> > > > > > allow
>> >> > > > > > > > >> > > > > > > for it to be merged so that the community
>> >> will
>> >> > be
>> >> > > able
>> >> > > > > > to
>> >> > > > > > > > >> proceed
>> >> > > > > > > > >> > > to
>> >> > > > > > > > >> > > > > > > follow-up PRs (as noted above).
>> >> > > > > > > > >> > > > > > >
>> >> > > > > > > > >> > > > > > > Please add/correct me if I missed anything
>> >> > > relevant.
>> >> > > > > > > > >> > > > > > >
>> >> > > > > > > > >> > > > > > > Thanks,
>> >> > > > > > > > >> > > > > > > Dmitri.
>> >> > > > > > > > >> > > > > > >
>> >> > > > > > > > >> > > > > > > On Wed, Jun 24, 2026 at 8:11 PM Yufei Gu <
>> >> > > > > > > > >> flyrain000@gmail.com>
>> >> > > > > > > > >> > > wrote:
>> >> > > > > > > > >> > > > > > >
>> >> > > > > > > > >> > > > > > >> I do not think we have agreement on the
>> >> query
>> >> > > REST
>> >> > > > > APIs
>> >> > > > > > > yet.
>> >> > > > > > > > >> > > > > > >>
>> >> > > > > > > > >> > > > > > >> In particular, the current query API
>> shape
>> >> > seems
>> >> > > > > highly
>> >> > > > > > > > >> coupled
>> >> > > > > > > > >> > > to the
>> >> > > > > > > > >> > > > > > >> *example[1]* metrics in the Iceberg REST
>> >> spec.
>> >> > I
>> >> > > do
>> >> > > > > not
>> >> > > > > > > think
>> >> > > > > > > > >> > > that is
>> >> > > > > > > > >> > > > > a
>> >> > > > > > > > >> > > > > > >> good way to design a REST API contract.
>> For
>> >> > > example,
>> >> > > > > > > fields
>> >> > > > > > > > >> like
>> >> > > > > > > > >> > > > > > >> resultDataFiles, resultDeleteFiles,
>> >> > > > > totalFileSizeBytes,
>> >> > > > > > > > >> > > > > > >> totalDataManifests,
>> >> > > > > > > > >> > > > > > >> totalDeleteManifests, and
>> >> scannedDataManifests
>> >> > > are
>> >> > > > > very
>> >> > > > > > > > >> specific
>> >> > > > > > > > >> > > to
>> >> > > > > > > > >> > > > > one
>> >> > > > > > > > >> > > > > > >> style of scan metric. That makes the API
>> >> look
>> >> > > more
>> >> > > > > > like a
>> >> > > > > > > > >> direct
>> >> > > > > > > > >> > > > > > >> projection
>> >> > > > > > > > >> > > > > > >> of the current example payload than a
>> >> general
>> >> > > query
>> >> > > > > > > model.
>> >> > > > > > > > >> > > > > > >>
>> >> > > > > > > > >> > > > > > >> My preference is that Polaris should not
>> >> define
>> >> > > or
>> >> > > > > > > strongly
>> >> > > > > > > > >> > opine
>> >> > > > > > > > >> > > on
>> >> > > > > > > > >> > > > > the
>> >> > > > > > > > >> > > > > > >> Iceberg scan or commit metrics query REST
>> >> API
>> >> > at
>> >> > > this
>> >> > > > > > > point.
>> >> > > > > > > > >> > > Polaris
>> >> > > > > > > > >> > > > > can
>> >> > > > > > > > >> > > > > > >> resolve and authorize the table, accept
>> the
>> >> > > metrics
>> >> > > > > > > report,
>> >> > > > > > > > >> and
>> >> > > > > > > > >> > > > > delegate
>> >> > > > > > > > >> > > > > > >> it
>> >> > > > > > > > >> > > > > > >> to the configured reporting path. Durable
>> >> > > storage,
>> >> > > > > > > filtering,
>> >> > > > > > > > >> > > > > > dashboards,
>> >> > > > > > > > >> > > > > > >> and query APIs should remain downstream
>> >> impl.
>> >> > > choices
>> >> > > > > > > unless
>> >> > > > > > > > >> and
>> >> > > > > > > > >> > > until
>> >> > > > > > > > >> > > > > > the
>> >> > > > > > > > >> > > > > > >> Iceberg community standardizes a query
>> >> > contract,
>> >> > > > > which
>> >> > > > > > I
>> >> > > > > > > > >> doubt
>> >> > > > > > > > >> > > will
>> >> > > > > > > > >> > > > > > >> happen.
>> >> > > > > > > > >> > > > > > >>
>> >> > > > > > > > >> > > > > > >> So I do not think the intake path has to
>> be
>> >> > > blocked,
>> >> > > > > > but
>> >> > > > > > > I
>> >> > > > > > > > >> would
>> >> > > > > > > > >> > > > > prefer
>> >> > > > > > > > >> > > > > > >> not
>> >> > > > > > > > >> > > > > > >> to merge a Polaris specific query API
>> that
>> >> may
>> >> > be
>> >> > > > > hard
>> >> > > > > > to
>> >> > > > > > > > >> evolve
>> >> > > > > > > > >> > > > > later.
>> >> > > > > > > > >> > > > > > >>
>> >> > > > > > > > >> > > > > > >> 1.
>> >> > > > > > > > >> > > > > > >>
>> >> > > > > > > > >> > > > > > >>
>> >> > > > > > > > >> > > > > >
>> >> > > > > > > > >> > > > >
>> >> > > > > > > > >> > >
>> >> > > > > > > > >> >
>> >> > > > > > > > >>
>> >> > > > > > >
>> >> > > > > >
>> >> > > > >
>> >> > >
>> >> >
>> >>
>> https://urldefense.com/v3/__https://github.com/apache/iceberg/blob/2a6c556c0c883c04c6d3fbd68e6aeda36b91e0aa/open-api/rest-catalog-open-api.yaml*L4152__;Iw!!Iz9xO38YGHZK!4rnlVIqBQQQ6xpGe1PMV58edOS_cGrKoqdfhw4qSp_JQCavvEjaxXk0OwGPnxyEcO60AzKQK_XDtp7H3DdkFXcrXCDsqYGo$
>> >> > > > > > > > >> > > > > > >>
>> >> > > > > > > > >> > > > > > >> Yufei
>> >> > > > > > > > >> > > > > > >>
>> >> > > > > > > > >> > > > > > >>
>> >> > > > > > > > >> > > > > > >> On Tue, Jun 23, 2026 at 9:30 AM Dmitri
>> >> > > Bourlatchkov <
>> >> > > > > > > > >> > > dimas@apache.org
>> >> > > > > > > > >> > > > > >
>> >> > > > > > > > >> > > > > > >> wrote:
>> >> > > > > > > > >> > > > > > >>
>> >> > > > > > > > >> > > > > > >> > Hi All,
>> >> > > > > > > > >> > > > > > >> >
>> >> > > > > > > > >> > > > > > >> > So, what are the blockers for merging
>> >> [4115],
>> >> > > if
>> >> > > > > any?
>> >> > > > > > > > >> > > > > > >> >
>> >> > > > > > > > >> > > > > > >> > [4115]
>> >> > > > > > > > >> >
>> >> > > > > > > > >>
>> >> > > > > > >
>> >> > > > > >
>> >> > > > >
>> >> > >
>> >> >
>> >>
>> https://urldefense.com/v3/__https://github.com/apache/polaris/pull/4115__;!!Iz9xO38YGHZK!4rnlVIqBQQQ6xpGe1PMV58edOS_cGrKoqdfhw4qSp_JQCavvEjaxXk0OwGPnxyEcO60AzKQK_XDtp7H3DdkFXcrXtFRscKs$
>> >> > > > > > > > >> > > > > > >> >
>> >> > > > > > > > >> > > > > > >> > Thanks,
>> >> > > > > > > > >> > > > > > >> > Dmitri.
>> >> > > > > > > > >> > > > > > >> >
>> >> > > > > > > > >> > > > > > >> > On Wed, Jun 17, 2026 at 5:28 PM EJ
>> Wang <
>> >> > > > > > > > >> > > > > > ej.wang.dev.interest@gmail.com
>> >> > > > > > > > >> > > > > > >> >
>> >> > > > > > > > >> > > > > > >> > wrote:
>> >> > > > > > > > >> > > > > > >> >
>> >> > > > > > > > >> > > > > > >> > > Hi folks,
>> >> > > > > > > > >> > > > > > >> > >
>> >> > > > > > > > >> > > > > > >> > > Thanks again for the discussion
>> today. I
>> >> > > updated
>> >> > > > > > the
>> >> > > > > > > sync
>> >> > > > > > > > >> > doc
>> >> > > > > > > > >> > > with
>> >> > > > > > > > >> > > > > > >> notes
>> >> > > > > > > > >> > > > > > >> > > from today's metrics sync:
>> >> > > > > > > > >> > > > > > >> > >
>> >> > > > > > > > >> > > > > > >> > >
>> >> > > > > > > > >> > > > > > >> > >
>> >> > > > > > > > >> > > > > > >> >
>> >> > > > > > > > >> > > > > > >>
>> >> > > > > > > > >> > > > > >
>> >> > > > > > > > >> > > > >
>> >> > > > > > > > >> > >
>> >> > > > > > > > >> >
>> >> > > > > > > > >>
>> >> > > > > > >
>> >> > > > > >
>> >> > > > >
>> >> > >
>> >> >
>> >>
>> https://urldefense.com/v3/__https://docs.google.com/document/d/100h7c4damrUzVuquYbBHM0EvA4LSWuW2IT2dN_7nYVA/edit?tab=t.ezk0rgdx0c6m__;!!Iz9xO38YGHZK!4rnlVIqBQQQ6xpGe1PMV58edOS_cGrKoqdfhw4qSp_JQCavvEjaxXk0OwGPnxyEcO60AzKQK_XDtp7H3DdkFXcrX83IZkm0$
>> >> > > > > > > > >> > > > > > >> > >
>> >> > > > > > > > >> > > > > > >> > > A few highlights:
>> >> > > > > > > > >> > > > > > >> > >
>> >> > > > > > > > >> > > > > > >> > >    - No objection to keeping the
>> default
>> >> > > metrics
>> >> > > > > > > behavior
>> >> > > > > > > > >> > > small:
>> >> > > > > > > > >> > > > > > >> > >    no-op/log-only is enough as the
>> >> built-in
>> >> > > > > > default.
>> >> > > > > > > > >> > > > > > >> > >    - Durable storage, event
>> forwarding,
>> >> > > external
>> >> > > > > > > queues,
>> >> > > > > > > > >> > > > > dashboards,
>> >> > > > > > > > >> > > > > > >> and
>> >> > > > > > > > >> > > > > > >> > >    custom filtering should be
>> >> > implementation
>> >> > > > > > choices
>> >> > > > > > > > >> behind
>> >> > > > > > > > >> > > the
>> >> > > > > > > > >> > > > > > >> metrics
>> >> > > > > > > > >> > > > > > >> > >    reporting path.
>> >> > > > > > > > >> > > > > > >> > >    - For PR #4115, the REST layer
>> should
>> >> > > > > > > > >> > resolve/authz/accept
>> >> > > > > > > > >> > > the
>> >> > > > > > > > >> > > > > > >> report,
>> >> > > > > > > > >> > > > > > >> > >    then delegate to the selected
>> metrics
>> >> > > > > > > implementation.
>> >> > > > > > > > >> It
>> >> > > > > > > > >> > > should
>> >> > > > > > > > >> > > > > > not
>> >> > > > > > > > >> > > > > > >> > own
>> >> > > > > > > > >> > > > > > >> > >    durable storage semantics.
>> >> > > > > > > > >> > > > > > >> > >    - The rough boundary we discussed
>> is:
>> >> > spec
>> >> > > > > owns
>> >> > > > > > > the
>> >> > > > > > > > >> wire
>> >> > > > > > > > >> > > > > > contract,
>> >> > > > > > > > >> > > > > > >> > >    runtime owns framework wiring,
>> >> > > API/contract
>> >> > > > > > > modules
>> >> > > > > > > > >> own
>> >> > > > > > > > >> > > > > > >> > provider-facing
>> >> > > > > > > > >> > > > > > >> > >    contracts, and extensions own
>> >> > replaceable
>> >> > > > > > > > >> > implementations.
>> >> > > > > > > > >> > > > > > >> > >    - OpenLineage has a similar
>> >> path-naming
>> >> > > > > > question.
>> >> > > > > > > We
>> >> > > > > > > > >> > should
>> >> > > > > > > > >> > > > > avoid
>> >> > > > > > > > >> > > > > > >> > >    occupying a generic lineage path
>> too
>> >> > > early if
>> >> > > > > we
>> >> > > > > > > want
>> >> > > > > > > > >> > room
>> >> > > > > > > > >> > > for
>> >> > > > > > > > >> > > > > > >> other
>> >> > > > > > > > >> > > > > > >> > >    lineage systems later.
>> >> > > > > > > > >> > > > > > >> > >
>> >> > > > > > > > >> > > > > > >> > > I believe the boundary rules
>> previewed
>> >> in
>> >> > > sync
>> >> > > > > > > meeting
>> >> > > > > > > > >> are
>> >> > > > > > > > >> > > useful
>> >> > > > > > > > >> > > > > > for
>> >> > > > > > > > >> > > > > > >> > > reviewing PR #4115, but it is not
>> >> > > > > > > > >> > > > > > >> > > a final module-layout proposal.
>> >> > > > > > > > >> > > > > > >> > >
>> >> > > > > > > > >> > > > > > >> > > For metrics specifically, I think the
>> >> > target
>> >> > > > > > remains:
>> >> > > > > > > > >> > > > > > >> > >    runtime
>> >> > > > > > > > >> > > > > > >> > >      -> metrics reporting/emitting
>> >> contract
>> >> > > > > > > > >> > > > > > >> > >      -> default implementation
>> >> > > > > > > > >> > > > > > >> > >           -> lower implementation
>> >> details,
>> >> > if
>> >> > > > > > needed
>> >> > > > > > > > >> > > > > > >> > >
>> >> > > > > > > > >> > > > > > >> > > That keeps the default simple, while
>> >> > allowing
>> >> > > > > > durable
>> >> > > > > > > > >> JDBC,
>> >> > > > > > > > >> > > > > > >> event-backed,
>> >> > > > > > > > >> > > > > > >> > > or external-queue implementations to
>> be
>> >> > added
>> >> > > > > > > separately.
>> >> > > > > > > > >> > > > > > >> > >
>> >> > > > > > > > >> > > > > > >> > > We also touched on the split between
>> >> > Iceberg
>> >> > > > > > metrics
>> >> > > > > > > > >> > emitting
>> >> > > > > > > > >> > > and
>> >> > > > > > > > >> > > > > > >> > > Polaris-owned metrics querying. The
>> >> > querying
>> >> > > side
>> >> > > > > > can
>> >> > > > > > > > >> keep
>> >> > > > > > > > >> > > > > evolving,
>> >> > > > > > > > >> > > > > > >> > > especially for dashboard or
>> >> generic-table
>> >> > use
>> >> > > > > > cases,
>> >> > > > > > > but
>> >> > > > > > > > >> I
>> >> > > > > > > > >> > do
>> >> > > > > > > > >> > > not
>> >> > > > > > > > >> > > > > > >> think
>> >> > > > > > > > >> > > > > > >> > > that needs to block the intake path
>> in
>> >> this
>> >> > > PR.
>> >> > > > > > > > >> > > > > > >> > >
>> >> > > > > > > > >> > > > > > >> > > Thanks,
>> >> > > > > > > > >> > > > > > >> > > -ej
>> >> > > > > > > > >> > > > > > >> > >
>> >> > > > > > > > >> > > > > > >> > > On Tue, Jun 16, 2026 at 2:17 PM Yufei
>> >> Gu <
>> >> > > > > > > > >> > > flyrain000@gmail.com>
>> >> > > > > > > > >> > > > > > >> wrote:
>> >> > > > > > > > >> > > > > > >> > >
>> >> > > > > > > > >> > > > > > >> > > > Thanks for chiming in, EJ. Agreed
>> that
>> >> > the
>> >> > > > > > default
>> >> > > > > > > > >> battery
>> >> > > > > > > > >> > > > > should
>> >> > > > > > > > >> > > > > > >> stay
>> >> > > > > > > > >> > > > > > >> > > > small. I'm leaning toward not
>> >> including
>> >> > > metrics
>> >> > > > > > > > >> > persistence
>> >> > > > > > > > >> > > in
>> >> > > > > > > > >> > > > > the
>> >> > > > > > > > >> > > > > > >> > > default
>> >> > > > > > > > >> > > > > > >> > > > battery.
>> >> > > > > > > > >> > > > > > >> > > >
>> >> > > > > > > > >> > > > > > >> > > > Yufei
>> >> > > > > > > > >> > > > > > >> > > >
>> >> > > > > > > > >> > > > > > >> > > >
>> >> > > > > > > > >> > > > > > >> > > > On Thu, Jun 11, 2026 at 8:30 PM EJ
>> >> Wang <
>> >> > > > > > > > >> > > > > > >> > ej.wang.dev.interest@gmail.com>
>> >> > > > > > > > >> > > > > > >> > > > wrote:
>> >> > > > > > > > >> > > > > > >> > > >
>> >> > > > > > > > >> > > > > > >> > > > > Hi Yufei,
>> >> > > > > > > > >> > > > > > >> > > > >
>> >> > > > > > > > >> > > > > > >> > > > > Thanks for connecting this back
>> to
>> >> the
>> >> > > REST
>> >> > > > > > > endpoint
>> >> > > > > > > > >> > > > > proposal. I
>> >> > > > > > > > >> > > > > > >> > > replied
>> >> > > > > > > > >> > > > > > >> > > > > with the fuller version on the
>> >> > > > > event-forwarding
>> >> > > > > > > > >> thread,
>> >> > > > > > > > >> > > but
>> >> > > > > > > > >> > > > > > >> wanted to
>> >> > > > > > > > >> > > > > > >> > > add
>> >> > > > > > > > >> > > > > > >> > > > > the shorter version here too.
>> >> > > > > > > > >> > > > > > >> > > > >
>> >> > > > > > > > >> > > > > > >> > > > > I agree that metrics
>> >> reporting/emitting
>> >> > > is
>> >> > > > > the
>> >> > > > > > > right
>> >> > > > > > > > >> > > > > conceptual
>> >> > > > > > > > >> > > > > > >> > > boundary,
>> >> > > > > > > > >> > > > > > >> > > > > and I also think the
>> event/listener
>> >> > path
>> >> > > is a
>> >> > > > > > > good
>> >> > > > > > > > >> > > > > > implementation
>> >> > > > > > > > >> > > > > > >> > > > direction
>> >> > > > > > > > >> > > > > > >> > > > > to explore. The distinction I
>> want
>> >> to
>> >> > > keep
>> >> > > > > > clear
>> >> > > > > > > is
>> >> > > > > > > > >> > > default
>> >> > > > > > > > >> > > > > > >> battery
>> >> > > > > > > > >> > > > > > >> > vs
>> >> > > > > > > > >> > > > > > >> > > > > extension implementation.
>> >> > > > > > > > >> > > > > > >> > > > >
>> >> > > > > > > > >> > > > > > >> > > > > For the REST/API side, I would
>> keep
>> >> the
>> >> > > > > > semantics
>> >> > > > > > > > >> > narrow:
>> >> > > > > > > > >> > > > > > Polaris
>> >> > > > > > > > >> > > > > > >> > > > resolves
>> >> > > > > > > > >> > > > > > >> > > > > and authorizes the table, accepts
>> >> the
>> >> > > Iceberg
>> >> > > > > > > > >> > scan/commit
>> >> > > > > > > > >> > > > > > metrics
>> >> > > > > > > > >> > > > > > >> > > report
>> >> > > > > > > > >> > > > > > >> > > > > into the configured
>> >> reporting/emitting
>> >> > > path,
>> >> > > > > > and
>> >> > > > > > > > >> returns
>> >> > > > > > > > >> > > 204.
>> >> > > > > > > > >> > > > > > That
>> >> > > > > > > > >> > > > > > >> > 204
>> >> > > > > > > > >> > > > > > >> > > > > should mean Polaris accepted the
>> >> report
>> >> > > into
>> >> > > > > > the
>> >> > > > > > > > >> > ingestion
>> >> > > > > > > > >> > > > > path;
>> >> > > > > > > > >> > > > > > >> it
>> >> > > > > > > > >> > > > > > >> > > > should
>> >> > > > > > > > >> > > > > > >> > > > > not imply durable storage.
>> >> > > > > > > > >> > > > > > >> > > > >
>> >> > > > > > > > >> > > > > > >> > > > > Durable storage, event
>> forwarding,
>> >> > > external
>> >> > > > > > > telemetry
>> >> > > > > > > > >> > > routing,
>> >> > > > > > > > >> > > > > > >> > > filtering,
>> >> > > > > > > > >> > > > > > >> > > > > and retention should sit behind
>> that
>> >> > > > > > > > >> reporting/emitting
>> >> > > > > > > > >> > > > > boundary
>> >> > > > > > > > >> > > > > > >> as
>> >> > > > > > > > >> > > > > > >> > > > > implementation-layer behavior. A
>> >> > durable
>> >> > > JDBC
>> >> > > > > > > path
>> >> > > > > > > > >> can
>> >> > > > > > > > >> > be
>> >> > > > > > > > >> > > one
>> >> > > > > > > > >> > > > > > >> named
>> >> > > > > > > > >> > > > > > >> > > > > extension implementation. An
>> >> > > event/listener
>> >> > > > > > > > >> forwarding
>> >> > > > > > > > >> > > path
>> >> > > > > > > > >> > > > > can
>> >> > > > > > > > >> > > > > > be
>> >> > > > > > > > >> > > > > > >> > > > another
>> >> > > > > > > > >> > > > > > >> > > > > named extension implementation.
>> >> > > > > > > > >> > > > > > >> > > > >
>> >> > > > > > > > >> > > > > > >> > > > > So I am +1 on the
>> event-forwarding
>> >> idea
>> >> > > as a
>> >> > > > > > > > >> non-default
>> >> > > > > > > > >> > > > > > extension
>> >> > > > > > > > >> > > > > > >> > > > > implementation. I just would not
>> >> make
>> >> > it
>> >> > > the
>> >> > > > > > > default
>> >> > > > > > > > >> > > battery
>> >> > > > > > > > >> > > > > or
>> >> > > > > > > > >> > > > > > >> the
>> >> > > > > > > > >> > > > > > >> > > core
>> >> > > > > > > > >> > > > > > >> > > > > API shape. The default battery
>> can
>> >> stay
>> >> > > small
>> >> > > > > > and
>> >> > > > > > > > >> safe,
>> >> > > > > > > > >> > > while
>> >> > > > > > > > >> > > > > > >> > > deployments
>> >> > > > > > > > >> > > > > > >> > > > > that need forwarding,
>> dashboards, or
>> >> > > durable
>> >> > > > > > > storage
>> >> > > > > > > > >> can
>> >> > > > > > > > >> > > opt
>> >> > > > > > > > >> > > > > > into
>> >> > > > > > > > >> > > > > > >> the
>> >> > > > > > > > >> > > > > > >> > > > > implementation that matches their
>> >> > > operational
>> >> > > > > > > model.
>> >> > > > > > > > >> > > > > > >> > > > >
>> >> > > > > > > > >> > > > > > >> > > > > Thanks,
>> >> > > > > > > > >> > > > > > >> > > > > -ej
>> >> > > > > > > > >> > > > > > >> > > > >
>> >> > > > > > > > >> > > > > > >> > > > > On Wed, Jun 10, 2026 at 11:43 AM
>> >> Yufei
>> >> > > Gu <
>> >> > > > > > > > >> > > > > flyrain000@gmail.com
>> >> > > > > > > > >> > > > > > >
>> >> > > > > > > > >> > > > > > >> > > wrote:
>> >> > > > > > > > >> > > > > > >> > > > >
>> >> > > > > > > > >> > > > > > >> > > > > > Thanks EJ. I agree that the
>> >> > > > > > reporting/emitting
>> >> > > > > > > > >> > boundary
>> >> > > > > > > > >> > > > > feels
>> >> > > > > > > > >> > > > > > >> like
>> >> > > > > > > > >> > > > > > >> > > the
>> >> > > > > > > > >> > > > > > >> > > > > > right SPI boundary, but I
>> wonder
>> >> if
>> >> > we
>> >> > > can
>> >> > > > > > > simplify
>> >> > > > > > > > >> > this
>> >> > > > > > > > >> > > > > even
>> >> > > > > > > > >> > > > > > >> > > further.
>> >> > > > > > > > >> > > > > > >> > > > > >
>> >> > > > > > > > >> > > > > > >> > > > > > Iceberg scan and commit metrics
>> >> look
>> >> > > very
>> >> > > > > > > similar
>> >> > > > > > > > >> to
>> >> > > > > > > > >> > > events
>> >> > > > > > > > >> > > > > to
>> >> > > > > > > > >> > > > > > >> me.
>> >> > > > > > > > >> > > > > > >> > > They
>> >> > > > > > > > >> > > > > > >> > > > > are
>> >> > > > > > > > >> > > > > > >> > > > > > append only, asynchronously
>> >> consumed,
>> >> > > often
>> >> > > > > > > > >> forwarded
>> >> > > > > > > > >> > to
>> >> > > > > > > > >> > > > > > >> external
>> >> > > > > > > > >> > > > > > >> > > > > systems,
>> >> > > > > > > > >> > > > > > >> > > > > > and have similar retention and
>> >> > cleanup
>> >> > > > > > > > >> requirements.
>> >> > > > > > > > >> > > More
>> >> > > > > > > > >> > > > > > >> details
>> >> > > > > > > > >> > > > > > >> > can
>> >> > > > > > > > >> > > > > > >> > > > be
>> >> > > > > > > > >> > > > > > >> > > > > > found in this thread [1]. In
>> that
>> >> > case,
>> >> > > > > > > Polaris may
>> >> > > > > > > > >> > only
>> >> > > > > > > > >> > > > > need
>> >> > > > > > > > >> > > > > > to
>> >> > > > > > > > >> > > > > > >> > > accept
>> >> > > > > > > > >> > > > > > >> > > > > the
>> >> > > > > > > > >> > > > > > >> > > > > > report and emit an event
>> through
>> >> the
>> >> > > > > existing
>> >> > > > > > > event
>> >> > > > > > > > >> > > > > framework.
>> >> > > > > > > > >> > > > > > >> > > > Filtering,
>> >> > > > > > > > >> > > > > > >> > > > > > async delivery, custom sinks,
>> and
>> >> > > retention
>> >> > > > > > > > >> mechanisms
>> >> > > > > > > > >> > > could
>> >> > > > > > > > >> > > > > > >> then
>> >> > > > > > > > >> > > > > > >> > be
>> >> > > > > > > > >> > > > > > >> > > > > shared
>> >> > > > > > > > >> > > > > > >> > > > > > instead of introducing a
>> separate
>> >> > > metrics
>> >> > > > > > > specific
>> >> > > > > > > > >> > > extension
>> >> > > > > > > > >> > > > > > >> layer
>> >> > > > > > > > >> > > > > > >> > > and
>> >> > > > > > > > >> > > > > > >> > > > > SPI.
>> >> > > > > > > > >> > > > > > >> > > > > >
>> >> > > > > > > > >> > > > > > >> > > > > > The main requirement I still
>> see
>> >> is
>> >> > > > > querying
>> >> > > > > > > > >> metrics
>> >> > > > > > > > >> > > through
>> >> > > > > > > > >> > > > > > >> > Polaris.
>> >> > > > > > > > >> > > > > > >> > > > If
>> >> > > > > > > > >> > > > > > >> > > > > we
>> >> > > > > > > > >> > > > > > >> > > > > > want that battery included
>> >> > experience,
>> >> > > we
>> >> > > > > > could
>> >> > > > > > > > >> work
>> >> > > > > > > > >> > on
>> >> > > > > > > > >> > > a
>> >> > > > > > > > >> > > > > > >> > persistence
>> >> > > > > > > > >> > > > > > >> > > > > > listener implementation that
>> >> > > selectively
>> >> > > > > > saves
>> >> > > > > > > > >> metrics
>> >> > > > > > > > >> > > and
>> >> > > > > > > > >> > > > > > >> expose
>> >> > > > > > > > >> > > > > > >> > it
>> >> > > > > > > > >> > > > > > >> > > > via
>> >> > > > > > > > >> > > > > > >> > > > > > the IRC event endpoint which
>> seems
>> >> > > pretty
>> >> > > > > > > close in
>> >> > > > > > > > >> > > Iceberg
>> >> > > > > > > > >> > > > > > >> > > > > > community. Another option I
>> like
>> >> more
>> >> > > is to
>> >> > > > > > > allow
>> >> > > > > > > > >> > users
>> >> > > > > > > > >> > > to
>> >> > > > > > > > >> > > > > > >> develop
>> >> > > > > > > > >> > > > > > >> > > > their
>> >> > > > > > > > >> > > > > > >> > > > > > own listener, so they can
>> persist
>> >> > > > > scan/commit
>> >> > > > > > > > >> metrics
>> >> > > > > > > > >> > > to any
>> >> > > > > > > > >> > > > > > >> > systems
>> >> > > > > > > > >> > > > > > >> > > > they
>> >> > > > > > > > >> > > > > > >> > > > > > prefer. In general, that feels
>> >> like
>> >> > an
>> >> > > > > > > > >> implementation
>> >> > > > > > > > >> > > choice
>> >> > > > > > > > >> > > > > > on
>> >> > > > > > > > >> > > > > > >> top
>> >> > > > > > > > >> > > > > > >> > > of
>> >> > > > > > > > >> > > > > > >> > > > > the
>> >> > > > > > > > >> > > > > > >> > > > > > event framework rather than
>> >> something
>> >> > > that
>> >> > > > > > > needs to
>> >> > > > > > > > >> > > shape
>> >> > > > > > > > >> > > > > the
>> >> > > > > > > > >> > > > > > >> core
>> >> > > > > > > > >> > > > > > >> > > > > > architecture.
>> >> > > > > > > > >> > > > > > >> > > > > >
>> >> > > > > > > > >> > > > > > >> > > > > > 1.
>> >> > > > > > > > >> > > > > > >> >
>> >> > > > > > > > >> > >
>> >> > > > > > > > >> >
>> >> > > > > > > > >>
>> >> > > > > > >
>> >> > > > > >
>> >> > > > >
>> >> > >
>> >> >
>> >>
>> https://urldefense.com/v3/__https://lists.apache.org/thread/x9j8nscvy8hq61tyn01mj8yp6n9of0kp__;!!Iz9xO38YGHZK!4rnlVIqBQQQ6xpGe1PMV58edOS_cGrKoqdfhw4qSp_JQCavvEjaxXk0OwGPnxyEcO60AzKQK_XDtp7H3DdkFXcrXGcGDgks$
>> >> > > > > > > > >> > > > > > >> > > > > >
>> >> > > > > > > > >> > > > > > >> > > > > > Thanks,
>> >> > > > > > > > >> > > > > > >> > > > > > Yufei
>> >> > > > > > > > >> > > > > > >> > > > > >
>> >> > > > > > > > >> > > > > > >> > > > > >
>> >> > > > > > > > >> > > > > > >> > > > > > On Thu, Jun 4, 2026 at 5:14 PM
>> EJ
>> >> > Wang
>> >> > > <
>> >> > > > > > > > >> > > > > > >> > > ej.wang.dev.interest@gmail.com
>> >> > > > > > > > >> > > > > > >> > > > >
>> >> > > > > > > > >> > > > > > >> > > > > > wrote:
>> >> > > > > > > > >> > > > > > >> > > > > >
>> >> > > > > > > > >> > > > > > >> > > > > > > Good point. I agree the
>> existing
>> >> > > > > > > > >> > > PolarisMetricsReporter is
>> >> > > > > > > > >> > > > > > >> > already
>> >> > > > > > > > >> > > > > > >> > > > very
>> >> > > > > > > > >> > > > > > >> > > > > > > close to the right conceptual
>> >> > > boundary. I
>> >> > > > > > am
>> >> > > > > > > not
>> >> > > > > > > > >> > > > > proposing a
>> >> > > > > > > > >> > > > > > >> > second
>> >> > > > > > > > >> > > > > > >> > > > > > > parallel reporter concept.
>> The
>> >> > > > > distinction
>> >> > > > > > I
>> >> > > > > > > am
>> >> > > > > > > > >> > > trying to
>> >> > > > > > > > >> > > > > > >> make is
>> >> > > > > > > > >> > > > > > >> > > > > > narrower:
>> >> > > > > > > > >> > > > > > >> > > > > > >
>> >> > > > > > > > >> > > > > > >> > > > > > > Current state:
>> >> > > > > > > > >> > > > > > >> > > > > > >
>> >> > > > > > > > >> > > > > > >> > > > > > >    - PolarisMetricsReporter
>> is
>> >> the
>> >> > > > > existing
>> >> > > > > > > > >> metrics
>> >> > > > > > > > >> > > > > > reporting
>> >> > > > > > > > >> > > > > > >> > hook.
>> >> > > > > > > > >> > > > > > >> > > > > > >    - But it currently lives
>> >> under
>> >> > > > > > > > >> runtime/service.
>> >> > > > > > > > >> > > > > > >> > > > > > >    - Its contract is
>> documented
>> >> > > around
>> >> > > > > > > > >> Quarkus/CDI
>> >> > > > > > > > >> > > > > discovery
>> >> > > > > > > > >> > > > > > >> and
>> >> > > > > > > > >> > > > > > >> > > > > > > @Identifier
>> >> > > > > > > > >> > > > > > >> > > > > > >    selection.
>> >> > > > > > > > >> > > > > > >> > > > > > >    - The default
>> implementation
>> >> is
>> >> > > > > > log-only.
>> >> > > > > > > > >> > > > > > >> > > > > > >    - The durable
>> implementation
>> >> > > writes
>> >> > > > > > > through
>> >> > > > > > > > >> > > > > > >> > > PolarisMetricsManager
>> >> > > > > > > > >> > > > > > >> > > > > and
>> >> > > > > > > > >> > > > > > >> > > > > > >    MetricsPersistence.
>> >> > > > > > > > >> > > > > > >> > > > > > >    - MetricsPersistence is
>> >> > currently
>> >> > > > > > > inherited by
>> >> > > > > > > > >> > > > > > >> > BasePersistence.
>> >> > > > > > > > >> > > > > > >> > > > > > >
>> >> > > > > > > > >> > > > > > >> > > > > > > My proposal:
>> >> > > > > > > > >> > > > > > >> > > > > > >
>> >> > > > > > > > >> > > > > > >> > > > > > >    - Keep the
>> reporting/emitting
>> >> > > boundary
>> >> > > > > > as
>> >> > > > > > > the
>> >> > > > > > > > >> > SPI.
>> >> > > > > > > > >> > > > > > >> > > > > > >    - Revise that contract in
>> a
>> >> > > > > > > framework-agnostic
>> >> > > > > > > > >> > > optional
>> >> > > > > > > > >> > > > > > >> > Iceberg
>> >> > > > > > > > >> > > > > > >> > > > > > *metrics
>> >> > > > > > > > >> > > > > > >> > > > > > >    extension API layer*.
>> >> > > > > > > > >> > > > > > >> > > > > > >    - Keep runtime/service
>> >> > > responsible for
>> >> > > > > > > REST
>> >> > > > > > > > >> > > ingestion,
>> >> > > > > > > > >> > > > > > >> > > table/authz
>> >> > > > > > > > >> > > > > > >> > > > > > >    validation, and runtime
>> >> wiring.
>> >> > > > > > > > >> > > > > > >> > > > > > >    - Keep the battery
>> >> > implementation
>> >> > > > > > > log-only in
>> >> > > > > > > > >> the
>> >> > > > > > > > >> > > > > metrics
>> >> > > > > > > > >> > > > > > >> > > > extension
>> >> > > > > > > > >> > > > > > >> > > > > > API
>> >> > > > > > > > >> > > > > > >> > > > > > >    layer, *not under
>> >> > > runtime/service*.
>> >> > > > > > > > >> > > > > > >> > > > > > >    - Treat durable JDBC
>> metrics
>> >> as
>> >> > > one
>> >> > > > > > > > >> > implementation
>> >> > > > > > > > >> > > of
>> >> > > > > > > > >> > > > > the
>> >> > > > > > > > >> > > > > > >> > > > reporting
>> >> > > > > > > > >> > > > > > >> > > > > > SPI.
>> >> > > > > > > > >> > > > > > >> > > > > > >    - Decompose the
>> >> > > MetricsPersistence and
>> >> > > > > > > > >> > > BasePersistence
>> >> > > > > > > > >> > > > > > >> > coupling
>> >> > > > > > > > >> > > > > > >> > > as
>> >> > > > > > > > >> > > > > > >> > > > > > >    durable implementation
>> >> detail.
>> >> > > > > > > > >> > > > > > >> > > > > > >
>> >> > > > > > > > >> > > > > > >> > > > > > > So the difference is not “new
>> >> > > reporter
>> >> > > > > > > concept vs
>> >> > > > > > > > >> > old
>> >> > > > > > > > >> > > > > > reporter
>> >> > > > > > > > >> > > > > > >> > > > > concept.”
>> >> > > > > > > > >> > > > > > >> > > > > > > but:
>> >> > > > > > > > >> > > > > > >> > > > > > >
>> >> > > > > > > > >> > > > > > >> > > > > > >
>> >> > > > > > > > >> > > > > > >> > > > > > >    - PolarisMetricsReporter
>> >> today =
>> >> > > > > > > > >> runtime/service
>> >> > > > > > > > >> > > > > > CDI-shaped
>> >> > > > > > > > >> > > > > > >> > > hook.
>> >> > > > > > > > >> > > > > > >> > > > > > >    - Target reporting SPI =
>> same
>> >> > > > > conceptual
>> >> > > > > > > > >> > boundary,
>> >> > > > > > > > >> > > but
>> >> > > > > > > > >> > > > > > >> placed
>> >> > > > > > > > >> > > > > > >> > in
>> >> > > > > > > > >> > > > > > >> > > > the
>> >> > > > > > > > >> > > > > > >> > > > > > >    right extension API layer
>> and
>> >> > kept
>> >> > > > > > > > >> > > framework-agnostic.
>> >> > > > > > > > >> > > > > > >> > > > > > >
>> >> > > > > > > > >> > > > > > >> > > > > > > For the current metrics PR, I
>> >> think
>> >> > > the
>> >> > > > > > > minimal
>> >> > > > > > > > >> > > > > > architectural
>> >> > > > > > > > >> > > > > > >> > > cleanup
>> >> > > > > > > > >> > > > > > >> > > > > > > before merge is to *put the
>> >> > > reporting SPI
>> >> > > > > > > > >> contract
>> >> > > > > > > > >> > > and the
>> >> > > > > > > > >> > > > > > >> > battery
>> >> > > > > > > > >> > > > > > >> > > > > > default
>> >> > > > > > > > >> > > > > > >> > > > > > > in the right place*. The
>> durable
>> >> > JDBC
>> >> > > > > > > > >> > implementation,
>> >> > > > > > > > >> > > > > > >> > event-backed
>> >> > > > > > > > >> > > > > > >> > > > > async
>> >> > > > > > > > >> > > > > > >> > > > > > > implementation, external
>> queue
>> >> > > support,
>> >> > > > > and
>> >> > > > > > > > >> deeper
>> >> > > > > > > > >> > > > > > persistence
>> >> > > > > > > > >> > > > > > >> > > > cleanup
>> >> > > > > > > > >> > > > > > >> > > > > > can
>> >> > > > > > > > >> > > > > > >> > > > > > > continue as follow-ups, as
>> long
>> >> as
>> >> > > we do
>> >> > > > > > not
>> >> > > > > > > lock
>> >> > > > > > > > >> > the
>> >> > > > > > > > >> > > SPI
>> >> > > > > > > > >> > > > > > >> > boundary
>> >> > > > > > > > >> > > > > > >> > > to
>> >> > > > > > > > >> > > > > > >> > > > > > > BasePersistence or to
>> >> > runtime/service
>> >> > > > > > wiring.
>> >> > > > > > > > >> > > > > > >> > > > > > >
>> >> > > > > > > > >> > > > > > >> > > > > > > -ej
>> >> > > > > > > > >> > > > > > >> > > > > > >
>> >> > > > > > > > >> > > > > > >> > > > > > > On Wed, Jun 3, 2026 at
>> 3:00 PM
>> >> > Dmitri
>> >> > > > > > > > >> Bourlatchkov <
>> >> > > > > > > > >> > > > > > >> > > dimas@apache.org
>> >> > > > > > > > >> > > > > > >> > > > >
>> >> > > > > > > > >> > > > > > >> > > > > > > wrote:
>> >> > > > > > > > >> > > > > > >> > > > > > >
>> >> > > > > > > > >> > > > > > >> > > > > > > > Hi EJ,
>> >> > > > > > > > >> > > > > > >> > > > > > > >
>> >> > > > > > > > >> > > > > > >> > > > > > > > Thanks for the recap /
>> >> summary!
>> >> > > > > > > > >> > > > > > >> > > > > > > >
>> >> > > > > > > > >> > > > > > >> > > > > > > > > Do folks agree that the
>> >> stable
>> >> > > SPI
>> >> > > > > > > boundary
>> >> > > > > > > > >> > > should be
>> >> > > > > > > > >> > > > > > >> > > > > > > > metrics reporting/emitting,
>> >> not
>> >> > > metrics
>> >> > > > > > > > >> > persistence?
>> >> > > > > > > > >> > > > > > >> > > > > > > >
>> >> > > > > > > > >> > > > > > >> > > > > > > > SGTM.
>> >> > > > > > > > >> > > > > > >> > > > > > > >
>> >> > > > > > > > >> > > > > > >> > > > > > > > > Does an optional Iceberg
>> >> > metrics
>> >> > > > > > > extension
>> >> > > > > > > > >> API
>> >> > > > > > > > >> > > layer
>> >> > > > > > > > >> > > > > > sound
>> >> > > > > > > > >> > > > > > >> > like
>> >> > > > > > > > >> > > > > > >> > > > the
>> >> > > > > > > > >> > > > > > >> > > > > > > right
>> >> > > > > > > > >> > > > > > >> > > > > > > > home for this SPI?
>> >> > > > > > > > >> > > > > > >> > > > > > > >
>> >> > > > > > > > >> > > > > > >> > > > > > > > How is that different from
>> >> > current
>> >> > > > > > > > >> > > > > PolarisMetricsReporter?
>> >> > > > > > > > >> > > > > > >> > > > > > > >
>> >> > > > > > > > >> > > > > > >> > > > > > > > > Should the current
>> durable
>> >> > > metrics
>> >> > > > > work
>> >> > > > > > > be
>> >> > > > > > > > >> > > reframed
>> >> > > > > > > > >> > > > > as a
>> >> > > > > > > > >> > > > > > >> > > durable
>> >> > > > > > > > >> > > > > > >> > > > > > > > JDBC reference
>> implementation
>> >> of
>> >> > > that
>> >> > > > > > SPI?
>> >> > > > > > > > >> > > > > > >> > > > > > > >
>> >> > > > > > > > >> > > > > > >> > > > > > > > SGTM.
>> >> > > > > > > > >> > > > > > >> > > > > > > >
>> >> > > > > > > > >> > > > > > >> > > > > > > > > What is the smallest PR
>> >> > sequence
>> >> > > to
>> >> > > > > get
>> >> > > > > > > there
>> >> > > > > > > > >> > > without
>> >> > > > > > > > >> > > > > > >> > blocking
>> >> > > > > > > > >> > > > > > >> > > > > > > > the current metrics work
>> >> > > unnecessarily?
>> >> > > > > > > > >> > > > > > >> > > > > > > >
>> >> > > > > > > > >> > > > > > >> > > > > > > > Let's get
>> >> > > > > > > > >> > >
>> >> > > > > > > > >> >
>> >> > > > > > > > >>
>> >> > > > > > >
>> >> > > > > >
>> >> > > > >
>> >> > >
>> >> >
>> >>
>> https://urldefense.com/v3/__https://github.com/apache/polaris/pull/4397__;!!Iz9xO38YGHZK!4rnlVIqBQQQ6xpGe1PMV58edOS_cGrKoqdfhw4qSp_JQCavvEjaxXk0OwGPnxyEcO60AzKQK_XDtp7H3DdkFXcrXuu5jefU$
>> >> > > > > > > > >> > > > > > >> merged
>> >> > > > > > > > >> > > > > > >> > > > first.
>> >> > > > > > > > >> > > > > > >> > > > > > > >
>> >> > > > > > > > >> > > > > > >> > > > > > > > Then, I'd propose isolating
>> >> the
>> >> > > Metrics
>> >> > > > > > > schema
>> >> > > > > > > > >> > from
>> >> > > > > > > > >> > > the
>> >> > > > > > > > >> > > > > > >> > MetaStore
>> >> > > > > > > > >> > > > > > >> > > > > > schema.
>> >> > > > > > > > >> > > > > > >> > > > > > > > This will probably have
>> some
>> >> > ripple
>> >> > > > > > effect
>> >> > > > > > > into
>> >> > > > > > > > >> > the
>> >> > > > > > > > >> > > > > > >> bootstrap
>> >> > > > > > > > >> > > > > > >> > > > > > workflows,
>> >> > > > > > > > >> > > > > > >> > > > > > > so
>> >> > > > > > > > >> > > > > > >> > > > > > > > it's not a trivial change.
>> >> > > > > > > > >> > > > > > >> > > > > > > >
>> >> > > > > > > > >> > > > > > >> > > > > > > > Then, let's reassess.
>> >> > > > > > > > >> > > > > > >> > > > > > > >
>> >> > > > > > > > >> > > > > > >> > > > > > > > Cheers,
>> >> > > > > > > > >> > > > > > >> > > > > > > > Dmitri.
>> >> > > > > > > > >> > > > > > >> > > > > > > >
>> >> > > > > > > > >> > > > > > >> > > > > > > > On Wed, Jun 3, 2026 at
>> >> 5:40 PM EJ
>> >> > > Wang
>> >> > > > > <
>> >> > > > > > > > >> > > > > > >> > > > > ej.wang.dev.interest@gmail.com
>> >> > > > > > > > >> > > > > > >> > > > > > >
>> >> > > > > > > > >> > > > > > >> > > > > > > > wrote:
>> >> > > > > > > > >> > > > > > >> > > > > > > >
>> >> > > > > > > > >> > > > > > >> > > > > > > > > Hi folks,
>> >> > > > > > > > >> > > > > > >> > > > > > > > >
>> >> > > > > > > > >> > > > > > >> > > > > > > > > Thanks again for the
>> >> discussion
>> >> > > > > today.
>> >> > > > > > I
>> >> > > > > > > > >> updated
>> >> > > > > > > > >> > > the
>> >> > > > > > > > >> > > > > > sync
>> >> > > > > > > > >> > > > > > >> doc
>> >> > > > > > > > >> > > > > > >> > > > with
>> >> > > > > > > > >> > > > > > >> > > > > > > notes
>> >> > > > > > > > >> > > > > > >> > > > > > > > > from the third metrics
>> >> > > architecture
>> >> > > > > > sync:
>> >> > > > > > > > >> > > > > > >> > > > > > > > >
>> >> > > > > > > > >> > > > > > >> > > > > > > > >
>> >> > > > > > > > >> > > > > > >> > > > > > > > >
>> >> > > > > > > > >> > > > > > >> > > > > > > >
>> >> > > > > > > > >> > > > > > >> > > > > > >
>> >> > > > > > > > >> > > > > > >> > > > > >
>> >> > > > > > > > >> > > > > > >> > > > >
>> >> > > > > > > > >> > > > > > >> > > >
>> >> > > > > > > > >> > > > > > >> > >
>> >> > > > > > > > >> > > > > > >> >
>> >> > > > > > > > >> > > > > > >>
>> >> > > > > > > > >> > > > > >
>> >> > > > > > > > >> > > > >
>> >> > > > > > > > >> > >
>> >> > > > > > > > >> >
>> >> > > > > > > > >>
>> >> > > > > > >
>> >> > > > > >
>> >> > > > >
>> >> > >
>> >> >
>> >>
>> https://urldefense.com/v3/__https://docs.google.com/document/d/100h7c4damrUzVuquYbBHM0EvA4LSWuW2IT2dN_7nYVA/edit?tab=t.k96s2xyqr5u1__;!!Iz9xO38YGHZK!4rnlVIqBQQQ6xpGe1PMV58edOS_cGrKoqdfhw4qSp_JQCavvEjaxXk0OwGPnxyEcO60AzKQK_XDtp7H3DdkFXcrXrqCL-QE$
>> >> > > > > > > > >> > > > > > >> > > > > > > > >
>> >> > > > > > > > >> > > > > > >> > > > > > > > > A few highlights from
>> >> today’s
>> >> > > > > > discussion:
>> >> > > > > > > > >> > > > > > >> > > > > > > > >
>> >> > > > > > > > >> > > > > > >> > > > > > > > >    - We clarified that
>> >> Iceberg
>> >> > > > > metrics
>> >> > > > > > > > >> reporting
>> >> > > > > > > > >> > > can
>> >> > > > > > > > >> > > > > be
>> >> > > > > > > > >> > > > > > >> > > > interpreted
>> >> > > > > > > > >> > > > > > >> > > > > > > > either
>> >> > > > > > > > >> > > > > > >> > > > > > > > >    as sync or async
>> handling
>> >> > from
>> >> > > > > > > Polaris’s
>> >> > > > > > > > >> > > > > perspective,
>> >> > > > > > > > >> > > > > > >> so
>> >> > > > > > > > >> > > > > > >> > > > Polaris
>> >> > > > > > > > >> > > > > > >> > > > > > > > should
>> >> > > > > > > > >> > > > > > >> > > > > > > > >    stay flexible as a
>> >> platform
>> >> > > > > instead
>> >> > > > > > of
>> >> > > > > > > > >> baking
>> >> > > > > > > > >> > > one
>> >> > > > > > > > >> > > > > > >> handling
>> >> > > > > > > > >> > > > > > >> > > > model
>> >> > > > > > > > >> > > > > > >> > > > > > > into
>> >> > > > > > > > >> > > > > > >> > > > > > > > > the
>> >> > > > > > > > >> > > > > > >> > > > > > > > >    REST API behavior.
>> >> > > > > > > > >> > > > > > >> > > > > > > > >    - We aligned that the
>> >> > built-in
>> >> > > > > > > (battery)
>> >> > > > > > > > >> > > behavior
>> >> > > > > > > > >> > > > > for
>> >> > > > > > > > >> > > > > > >> > > metrics
>> >> > > > > > > > >> > > > > > >> > > > > > > emitting
>> >> > > > > > > > >> > > > > > >> > > > > > > > >    can stay simple:
>> >> > > no-op/log-only is
>> >> > > > > > > enough
>> >> > > > > > > > >> as
>> >> > > > > > > > >> > > the
>> >> > > > > > > > >> > > > > > >> default.
>> >> > > > > > > > >> > > > > > >> > > > > > > > >    - Durable metrics
>> >> > persistence
>> >> > > > > should
>> >> > > > > > > be
>> >> > > > > > > > >> > > treated as
>> >> > > > > > > > >> > > > > an
>> >> > > > > > > > >> > > > > > >> > > > > > implementation
>> >> > > > > > > > >> > > > > > >> > > > > > > > of
>> >> > > > > > > > >> > > > > > >> > > > > > > > >    the metrics reporting
>> >> path,
>> >> > > not as
>> >> > > > > > the
>> >> > > > > > > > >> core
>> >> > > > > > > > >> > SPI
>> >> > > > > > > > >> > > > > > >> boundary.
>> >> > > > > > > > >> > > > > > >> > > > > > > > >    - The existing durable
>> >> > metrics
>> >> > > > > work
>> >> > > > > > > can be
>> >> > > > > > > > >> > > reviewed
>> >> > > > > > > > >> > > > > > as
>> >> > > > > > > > >> > > > > > >> a
>> >> > > > > > > > >> > > > > > >> > > > > reference
>> >> > > > > > > > >> > > > > > >> > > > > > > > >    implementation of the
>> >> > > reporting
>> >> > > > > SPI,
>> >> > > > > > > with
>> >> > > > > > > > >> > > > > > >> > > persistence-related
>> >> > > > > > > > >> > > > > > >> > > > > > logic
>> >> > > > > > > > >> > > > > > >> > > > > > > > kept
>> >> > > > > > > > >> > > > > > >> > > > > > > > >    self-contained under a
>> >> > metrics
>> >> > > > > > durable
>> >> > > > > > > > >> > > > > implementation
>> >> > > > > > > > >> > > > > > >> > module
>> >> > > > > > > > >> > > > > > >> > > > > > rather
>> >> > > > > > > > >> > > > > > >> > > > > > > > than
>> >> > > > > > > > >> > > > > > >> > > > > > > > >    scattered through core
>> >> > entity
>> >> > > > > > > persistence.
>> >> > > > > > > > >> > > > > > >> > > > > > > > >    - Dashboard/insights
>> >> > remains a
>> >> > > > > real
>> >> > > > > > > use
>> >> > > > > > > > >> case,
>> >> > > > > > > > >> > > but
>> >> > > > > > > > >> > > > > we
>> >> > > > > > > > >> > > > > > >> > agreed
>> >> > > > > > > > >> > > > > > >> > > to
>> >> > > > > > > > >> > > > > > >> > > > > > keep
>> >> > > > > > > > >> > > > > > >> > > > > > > it
>> >> > > > > > > > >> > > > > > >> > > > > > > > >    separate from the core
>> >> > metrics
>> >> > > > > > intake
>> >> > > > > > > > >> > > discussion
>> >> > > > > > > > >> > > > > for
>> >> > > > > > > > >> > > > > > >> now.
>> >> > > > > > > > >> > > > > > >> > > > > > > > >
>> >> > > > > > > > >> > > > > > >> > > > > > > > > I also did a quick source
>> >> check
>> >> > > after
>> >> > > > > > the
>> >> > > > > > > > >> > meeting
>> >> > > > > > > > >> > > to
>> >> > > > > > > > >> > > > > > make
>> >> > > > > > > > >> > > > > > >> > sure
>> >> > > > > > > > >> > > > > > >> > > we
>> >> > > > > > > > >> > > > > > >> > > > > are
>> >> > > > > > > > >> > > > > > >> > > > > > > > > describing the current
>> state
>> >> > > > > > accurately.
>> >> > > > > > > > >> > > > > > >> > > > > > > > >
>> >> > > > > > > > >> > > > > > >> > > > > > > > > Current state:
>> >> > > > > > > > >> > > > > > >> > > > > > > > >
>> >> > > > > > > > >> > > > > > >> > > > > > > > >    - Polaris already has
>> a
>> >> > > metrics
>> >> > > > > > > reporting
>> >> > > > > > > > >> > hook:
>> >> > > > > > > > >> > > > > > >> > > > > > > > PolarisMetricsReporter.
>> >> > > > > > > > >> > > > > > >> > > > > > > > >    - The default
>> >> implementation
>> >> > > is
>> >> > > > > > > > >> > > > > > DefaultMetricsReporter,
>> >> > > > > > > > >> > > > > > >> > > > selected
>> >> > > > > > > > >> > > > > > >> > > > > > by
>> >> > > > > > > > >> > > > > > >> > > > > > > > >
>> >> > > > > > > > >> > > polaris.iceberg-metrics.reporting.type=default. It
>> >> > > > > > > > >> > > > > is
>> >> > > > > > > > >> > > > > > >> > > > log-only,
>> >> > > > > > > > >> > > > > > >> > > > > > > > >    effectively quiet
>> unless
>> >> > > metrics
>> >> > > > > > > logging
>> >> > > > > > > > >> is
>> >> > > > > > > > >> > > > > enabled.
>> >> > > > > > > > >> > > > > > >> > > > > > > > >    - There is also a
>> >> > > > > > > > >> PersistingMetricsReporter,
>> >> > > > > > > > >> > > > > selected
>> >> > > > > > > > >> > > > > > >> by
>> >> > > > > > > > >> > > > > > >> > > > > > > > >
>> >> > > > > > > > >> > > polaris.iceberg-metrics.reporting.type=persisting,
>> >> > > > > > > > >> > > > > > >> which
>> >> > > > > > > > >> > > > > > >> > > > > converts
>> >> > > > > > > > >> > > > > > >> > > > > > > > >    Iceberg scan/commit
>> >> reports
>> >> > > into
>> >> > > > > > > Polaris
>> >> > > > > > > > >> > > metrics
>> >> > > > > > > > >> > > > > > >> records
>> >> > > > > > > > >> > > > > > >> > and
>> >> > > > > > > > >> > > > > > >> > > > > > writes
>> >> > > > > > > > >> > > > > > >> > > > > > > > > through
>> >> > > > > > > > >> > > > > > >> > > > > > > > >    PolarisMetricsManager
>> ->
>> >> > > > > > > > >> MetricsPersistence.
>> >> > > > > > > > >> > > > > > >> > > > > > > > >    - MetricsPersistence
>> >> > currently
>> >> > > > > > exists
>> >> > > > > > > in
>> >> > > > > > > > >> the
>> >> > > > > > > > >> > > > > > >> persistence
>> >> > > > > > > > >> > > > > > >> > > layer
>> >> > > > > > > > >> > > > > > >> > > > > and
>> >> > > > > > > > >> > > > > > >> > > > > > > is
>> >> > > > > > > > >> > > > > > >> > > > > > > > >    inherited by
>> >> > BasePersistence.
>> >> > > > > > > > >> > > > > > >> > > > > > > > >
>> >> > > > > > > > >> > > > > > >> > > > > > > > > My read from the
>> discussion
>> >> is
>> >> > > that
>> >> > > > > the
>> >> > > > > > > > >> target
>> >> > > > > > > > >> > > > > boundary
>> >> > > > > > > > >> > > > > > >> > should
>> >> > > > > > > > >> > > > > > >> > > > not
>> >> > > > > > > > >> > > > > > >> > > > > be
>> >> > > > > > > > >> > > > > > >> > > > > > > > > MetricsPersistence as
>> >> inherited
>> >> > > by
>> >> > > > > > > > >> > > BasePersistence.
>> >> > > > > > > > >> > > > > That
>> >> > > > > > > > >> > > > > > >> path
>> >> > > > > > > > >> > > > > > >> > > > > should
>> >> > > > > > > > >> > > > > > >> > > > > > be
>> >> > > > > > > > >> > > > > > >> > > > > > > > > decomposed as durable
>> >> > > implementation
>> >> > > > > > > detail
>> >> > > > > > > > >> > (taken
>> >> > > > > > > > >> > > > > care
>> >> > > > > > > > >> > > > > > >> of by
>> >> > > > > > > > >> > > > > > >> > > > > > > > >
>> >> > > > > > > > >> >
>> >> > > > > > > > >>
>> >> > > > > > >
>> >> > > > > >
>> >> > > > >
>> >> > >
>> >> >
>> >>
>> https://urldefense.com/v3/__https://github.com/apache/polaris/pull/4397__;!!Iz9xO38YGHZK!4rnlVIqBQQQ6xpGe1PMV58edOS_cGrKoqdfhw4qSp_JQCavvEjaxXk0OwGPnxyEcO60AzKQK_XDtp7H3DdkFXcrXuu5jefU$
>> >> > > > > > > > >> ).
>> >> > > > > > > > >> > The
>> >> > > > > > > > >> > > > > > stable
>> >> > > > > > > > >> > > > > > >> > > > extension
>> >> > > > > > > > >> > > > > > >> > > > > > > point
>> >> > > > > > > > >> > > > > > >> > > > > > > > > should instead be the
>> >> metrics
>> >> > > > > > > > >> reporting/emitting
>> >> > > > > > > > >> > > > > > boundary.
>> >> > > > > > > > >> > > > > > >> > > > > > > > >
>> >> > > > > > > > >> > > > > > >> > > > > > > > > Concretely, I think the
>> next
>> >> > > proposal
>> >> > > > > > > should
>> >> > > > > > > > >> be
>> >> > > > > > > > >> > > shaped
>> >> > > > > > > > >> > > > > > >> like
>> >> > > > > > > > >> > > > > > >> > > this:
>> >> > > > > > > > >> > > > > > >> > > > > > > > >
>> >> > > > > > > > >> > > > > > >> > > > > > > > >    1.
>> >> > > > > > > > >> > > > > > >> > > > > > > > >
>> >> > > > > > > > >> > > > > > >> > > > > > > > >    Define the proper
>> metrics
>> >> > > > > reporting
>> >> > > > > > > SPI
>> >> > > > > > > > >> at an
>> >> > > > > > > > >> > > > > > optional
>> >> > > > > > > > >> > > > > > >> > > Iceberg
>> >> > > > > > > > >> > > > > > >> > > > > > > metrics
>> >> > > > > > > > >> > > > > > >> > > > > > > > >    extension API layer.
>> >> > > > > > > > >> > > > > > >> > > > > > > > >
>> >> > > > > > > > >> > > > > > >> > > > > > > > >    Example direction:
>> >> > > > > > > > >> > > > > > >> > > > > > > > >
>> >> > > > > > > > >> > > > > > >> > > > > > > > >    public interface
>> >> > > > > > > IcebergMetricsReporter {
>> >> > > > > > > > >> > > > > > >> > > > > > > > >      void
>> >> > > > > > > > >> > reportMetric(IcebergMetricsReportContext
>> >> > > > > > > > >> > > > > > >> context,
>> >> > > > > > > > >> > > > > > >> > > > > > > > > MetricsReport report);
>> >> > > > > > > > >> > > > > > >> > > > > > > > >    }
>> >> > > > > > > > >> > > > > > >> > > > > > > > >
>> >> > > > > > > > >> > > > > > >> > > > > > > > >    The context would
>> carry
>> >> the
>> >> > > small
>> >> > > > > > > > >> > > Polaris-resolved
>> >> > > > > > > > >> > > > > > >> > envelope:
>> >> > > > > > > > >> > > > > > >> > > > > > > > >    catalog/table
>> identity,
>> >> > report
>> >> > > > > type,
>> >> > > > > > > > >> received
>> >> > > > > > > > >> > > > > > >> timestamp,
>> >> > > > > > > > >> > > > > > >> > and
>> >> > > > > > > > >> > > > > > >> > > > > > > > > request/trace
>> >> > > > > > > > >> > > > > > >> > > > > > > > >    context if available.
>> The
>> >> > raw
>> >> > > > > > Iceberg
>> >> > > > > > > > >> > > MetricsReport
>> >> > > > > > > > >> > > > > > >> > remains
>> >> > > > > > > > >> > > > > > >> > > > the
>> >> > > > > > > > >> > > > > > >> > > > > > > > payload.
>> >> > > > > > > > >> > > > > > >> > > > > > > > >    2.
>> >> > > > > > > > >> > > > > > >> > > > > > > > >
>> >> > > > > > > > >> > > > > > >> > > > > > > > >    Keep runtime/service
>> as
>> >> the
>> >> > > REST
>> >> > > > > > > ingestion
>> >> > > > > > > > >> > and
>> >> > > > > > > > >> > > > > wiring
>> >> > > > > > > > >> > > > > > >> > layer.
>> >> > > > > > > > >> > > > > > >> > > > > > > > >
>> >> > > > > > > > >> > > > > > >> > > > > > > > >    The REST handler still
>> >> > > resolves
>> >> > > > > the
>> >> > > > > > > table
>> >> > > > > > > > >> and
>> >> > > > > > > > >> > > > > > performs
>> >> > > > > > > > >> > > > > > >> > authz
>> >> > > > > > > > >> > > > > > >> > > > > > before
>> >> > > > > > > > >> > > > > > >> > > > > > > > >    accepting the report.
>> >> After
>> >> > > that,
>> >> > > > > it
>> >> > > > > > > calls
>> >> > > > > > > > >> > the
>> >> > > > > > > > >> > > > > > selected
>> >> > > > > > > > >> > > > > > >> > > > > reporting
>> >> > > > > > > > >> > > > > > >> > > > > > > > >    implementation.
>> >> > > > > > > > >> > > > > > >> > > > > > > > >    3.
>> >> > > > > > > > >> > > > > > >> > > > > > > > >
>> >> > > > > > > > >> > > > > > >> > > > > > > > >    Keep the battery
>> default
>> >> > > > > > > no-op/log-only.
>> >> > > > > > > > >> > > > > > >> > > > > > > > >
>> >> > > > > > > > >> > > > > > >> > > > > > > > >    This preserves an
>> >> out-of-box
>> >> > > safe
>> >> > > > > > > default
>> >> > > > > > > > >> and
>> >> > > > > > > > >> > > > > avoids
>> >> > > > > > > > >> > > > > > >> > > > requiring a
>> >> > > > > > > > >> > > > > > >> > > > > > > > durable
>> >> > > > > > > > >> > > > > > >> > > > > > > > >    metrics store for
>> every
>> >> > > Polaris
>> >> > > > > > > > >> deployment.
>> >> > > > > > > > >> > > > > > >> > > > > > > > >    4.
>> >> > > > > > > > >> > > > > > >> > > > > > > > >
>> >> > > > > > > > >> > > > > > >> > > > > > > > >    Move durable JDBC
>> metrics
>> >> > > into a
>> >> > > > > > > > >> > self-contained
>> >> > > > > > > > >> > > > > > >> > > implementation
>> >> > > > > > > > >> > > > > > >> > > > > of
>> >> > > > > > > > >> > > > > > >> > > > > > > the
>> >> > > > > > > > >> > > > > > >> > > > > > > > >    reporting SPI.
>> >> > > > > > > > >> > > > > > >> > > > > > > > >
>> >> > > > > > > > >> > > > > > >> > > > > > > > >    That implementation
>> can
>> >> own
>> >> > > its
>> >> > > > > > > schema,
>> >> > > > > > > > >> > > bootstrap,
>> >> > > > > > > > >> > > > > > >> > > retention,
>> >> > > > > > > > >> > > > > > >> > > > > and
>> >> > > > > > > > >> > > > > > >> > > > > > > read
>> >> > > > > > > > >> > > > > > >> > > > > > > > >    API support. It should
>> >> not
>> >> > > define
>> >> > > > > > the
>> >> > > > > > > core
>> >> > > > > > > > >> > > > > reporting
>> >> > > > > > > > >> > > > > > >> SPI
>> >> > > > > > > > >> > > > > > >> > > > > boundary,
>> >> > > > > > > > >> > > > > > >> > > > > > > and
>> >> > > > > > > > >> > > > > > >> > > > > > > > > it
>> >> > > > > > > > >> > > > > > >> > > > > > > > >    should not require
>> >> metrics
>> >> > > > > > > persistence to
>> >> > > > > > > > >> > > remain
>> >> > > > > > > > >> > > > > > >> inherited
>> >> > > > > > > > >> > > > > > >> > > > from
>> >> > > > > > > > >> > > > > > >> > > > > > > > >    BasePersistence.
>> >> > > > > > > > >> > > > > > >> > > > > > > > >    5.
>> >> > > > > > > > >> > > > > > >> > > > > > > > >
>> >> > > > > > > > >> > > > > > >> > > > > > > > >    Treat
>> async/event-backed
>> >> > > handling
>> >> > > > > as
>> >> > > > > > > > >> another
>> >> > > > > > > > >> > > > > > >> > implementation
>> >> > > > > > > > >> > > > > > >> > > of
>> >> > > > > > > > >> > > > > > >> > > > > the
>> >> > > > > > > > >> > > > > > >> > > > > > > > same
>> >> > > > > > > > >> > > > > > >> > > > > > > > >    reporting SPI.
>> >> > > > > > > > >> > > > > > >> > > > > > > > >
>> >> > > > > > > > >> > > > > > >> > > > > > > > >    For example, an
>> >> event-backed
>> >> > > > > > reporter
>> >> > > > > > > > >> could
>> >> > > > > > > > >> > > > > enqueue a
>> >> > > > > > > > >> > > > > > >> > > metrics
>> >> > > > > > > > >> > > > > > >> > > > > > event
>> >> > > > > > > > >> > > > > > >> > > > > > > > and
>> >> > > > > > > > >> > > > > > >> > > > > > > > >    let listeners handle
>> >> durable
>> >> > > > > storage
>> >> > > > > > > or
>> >> > > > > > > > >> other
>> >> > > > > > > > >> > > > > sinks.
>> >> > > > > > > > >> > > > > > >> If we
>> >> > > > > > > > >> > > > > > >> > > > later
>> >> > > > > > > > >> > > > > > >> > > > > > > need
>> >> > > > > > > > >> > > > > > >> > > > > > > > a
>> >> > > > > > > > >> > > > > > >> > > > > > > > >    replaceable queue
>> engine,
>> >> > that
>> >> > > > > seems
>> >> > > > > > > like
>> >> > > > > > > > >> a
>> >> > > > > > > > >> > > shared
>> >> > > > > > > > >> > > > > > >> > > > event/metrics
>> >> > > > > > > > >> > > > > > >> > > > > > > > > substrate
>> >> > > > > > > > >> > > > > > >> > > > > > > > >    topic rather than a
>> >> > > metrics-only
>> >> > > > > > > > >> requirement.
>> >> > > > > > > > >> > > > > > >> > > > > > > > >
>> >> > > > > > > > >> > > > > > >> > > > > > > > > This framing lets us keep
>> >> the
>> >> > > REST
>> >> > > > > > > metrics
>> >> > > > > > > > >> > > endpoint
>> >> > > > > > > > >> > > > > > >> simple,
>> >> > > > > > > > >> > > > > > >> > > > > preserve
>> >> > > > > > > > >> > > > > > >> > > > > > > the
>> >> > > > > > > > >> > > > > > >> > > > > > > > > current default behavior,
>> >> > support
>> >> > > > > > durable
>> >> > > > > > > > >> > metrics
>> >> > > > > > > > >> > > > > users,
>> >> > > > > > > > >> > > > > > >> and
>> >> > > > > > > > >> > > > > > >> > > > still
>> >> > > > > > > > >> > > > > > >> > > > > > > leave
>> >> > > > > > > > >> > > > > > >> > > > > > > > > room for
>> async/event-backed
>> >> or
>> >> > > > > > > > >> > > external-queue-based
>> >> > > > > > > > >> > > > > > >> > > > > implementations.
>> >> > > > > > > > >> > > > > > >> > > > > > > > >
>> >> > > > > > > > >> > > > > > >> > > > > > > > > I think the main
>> follow-up
>> >> > > questions
>> >> > > > > > are:
>> >> > > > > > > > >> > > > > > >> > > > > > > > >
>> >> > > > > > > > >> > > > > > >> > > > > > > > >    - Do folks agree that
>> the
>> >> > > stable
>> >> > > > > SPI
>> >> > > > > > > > >> boundary
>> >> > > > > > > > >> > > > > should
>> >> > > > > > > > >> > > > > > be
>> >> > > > > > > > >> > > > > > >> > > > metrics
>> >> > > > > > > > >> > > > > > >> > > > > > > > >    reporting/emitting,
>> not
>> >> > > metrics
>> >> > > > > > > > >> persistence?
>> >> > > > > > > > >> > > > > > >> > > > > > > > >    - Does an optional
>> >> Iceberg
>> >> > > metrics
>> >> > > > > > > > >> extension
>> >> > > > > > > > >> > > API
>> >> > > > > > > > >> > > > > > layer
>> >> > > > > > > > >> > > > > > >> > sound
>> >> > > > > > > > >> > > > > > >> > > > > like
>> >> > > > > > > > >> > > > > > >> > > > > > > the
>> >> > > > > > > > >> > > > > > >> > > > > > > > >    right home for this
>> SPI?
>> >> > > > > > > > >> > > > > > >> > > > > > > > >    - Should the current
>> >> durable
>> >> > > > > metrics
>> >> > > > > > > work
>> >> > > > > > > > >> be
>> >> > > > > > > > >> > > > > reframed
>> >> > > > > > > > >> > > > > > >> as a
>> >> > > > > > > > >> > > > > > >> > > > > durable
>> >> > > > > > > > >> > > > > > >> > > > > > > > JDBC
>> >> > > > > > > > >> > > > > > >> > > > > > > > >    reference
>> implementation
>> >> of
>> >> > > that
>> >> > > > > > SPI?
>> >> > > > > > > > >> > > > > > >> > > > > > > > >    - What is the
>> smallest PR
>> >> > > sequence
>> >> > > > > > to
>> >> > > > > > > get
>> >> > > > > > > > >> > there
>> >> > > > > > > > >> > > > > > without
>> >> > > > > > > > >> > > > > > >> > > > blocking
>> >> > > > > > > > >> > > > > > >> > > > > > the
>> >> > > > > > > > >> > > > > > >> > > > > > > > >    current metrics work
>> >> > > > > unnecessarily?
>> >> > > > > > > > >> > > > > > >> > > > > > > > >
>> >> > > > > > > > >> > > > > > >> > > > > > > > > Thanks,
>> >> > > > > > > > >> > > > > > >> > > > > > > > > -ej
>> >> > > > > > > > >> > > > > > >> > > > > > > > >
>> >> > > > > > > > >> > > > > > >> > > > > > > > > On Fri, May 15, 2026 at
>> >> 5:16 PM
>> >> > > > > Dmitri
>> >> > > > > > > > >> > > Bourlatchkov <
>> >> > > > > > > > >> > > > > > >> > > > > > dimas@apache.org>
>> >> > > > > > > > >> > > > > > >> > > > > > > > > wrote:
>> >> > > > > > > > >> > > > > > >> > > > > > > > >
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > Hi JB,
>> >> > > > > > > > >> > > > > > >> > > > > > > > > >
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > Could you set up
>> another
>> >> > > meeting,
>> >> > > > > > > please?
>> >> > > > > > > > >> Same
>> >> > > > > > > > >> > > time
>> >> > > > > > > > >> > > > > on
>> >> > > > > > > > >> > > > > > >> > > > Wednesday
>> >> > > > > > > > >> > > > > > >> > > > > as
>> >> > > > > > > > >> > > > > > >> > > > > > > > last
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > time... I hope it works
>> >> for
>> >> > > > > everyone.
>> >> > > > > > > > >> > > > > > >> > > > > > > > > >
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > Cheers,
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > Dmitri.
>> >> > > > > > > > >> > > > > > >> > > > > > > > > >
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > On Fri, May 15, 2026 at
>> >> > 8:06 PM
>> >> > > > > Yufei
>> >> > > > > > > Gu <
>> >> > > > > > > > >> > > > > > >> > > flyrain000@gmail.com
>> >> > > > > > > > >> > > > > > >> > > > >
>> >> > > > > > > > >> > > > > > >> > > > > > > wrote:
>> >> > > > > > > > >> > > > > > >> > > > > > > > > >
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > +1 on another sync
>> call
>> >> > next
>> >> > > > > week.
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > >
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > Yufei
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > >
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > >
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > On Fri, May 15, 2026
>> at
>> >> > > 4:52 PM
>> >> > > > > > > Dmitri
>> >> > > > > > > > >> > > > > Bourlatchkov
>> >> > > > > > > > >> > > > > > <
>> >> > > > > > > > >> > > > > > >> > > > > > > > dimas@apache.org>
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > wrote:
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > >
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > Hi All,
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > >
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > WDYT about another
>> >> sync
>> >> > > call
>> >> > > > > next
>> >> > > > > > > week?
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > >
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > Thanks,
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > Dmitri.
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > >
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > On Wed, May 6,
>> 2026 at
>> >> > > 5:29 PM
>> >> > > > > > > Dmitri
>> >> > > > > > > > >> > > > > > Bourlatchkov <
>> >> > > > > > > > >> > > > > > >> > > > > > > > dimas@apache.org
>> >> > > > > > > > >> > > > > > >> > > > > > > > > >
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > wrote:
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > >
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > > Hi EJ,
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > > Thanks for the
>> >> summary!
>> >> > > It
>> >> > > > > > covers
>> >> > > > > > > > >> what
>> >> > > > > > > > >> > we
>> >> > > > > > > > >> > > > > > >> discussed
>> >> > > > > > > > >> > > > > > >> > in
>> >> > > > > > > > >> > > > > > >> > > > the
>> >> > > > > > > > >> > > > > > >> > > > > > > > meeting
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > very
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > > well, IMHO.
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > > Looking forward
>> to
>> >> > > concrete
>> >> > > > > PRs
>> >> > > > > > > :)
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > > Cheers,
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > > Dmitri.
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > > On Wed, May 6,
>> 2026
>> >> at
>> >> > > > > 5:08 PM
>> >> > > > > > EJ
>> >> > > > > > > > >> Wang <
>> >> > > > > > > > >> > > > > > >> > > > > > > > > >
>> >> > ej.wang.dev.interest@gmail.com
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > >
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > > wrote:
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> Hi folks,
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >>
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> We had a
>> community
>> >> > sync
>> >> > > > > > earlier,
>> >> > > > > > > > >> thanks
>> >> > > > > > > > >> > > JB
>> >> > > > > > > > >> > > > > for
>> >> > > > > > > > >> > > > > > >> > > > scheduling
>> >> > > > > > > > >> > > > > > >> > > > > > it.
>> >> > > > > > > > >> > > > > > >> > > > > > > > > Notes
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > from
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> the first
>> metrics
>> >> > > > > architecture
>> >> > > > > > > sync
>> >> > > > > > > > >> > (May
>> >> > > > > > > > >> > > 6,
>> >> > > > > > > > >> > > > > > >> 10-11am
>> >> > > > > > > > >> > > > > > >> > > PT).
>> >> > > > > > > > >> > > > > > >> > > > > > > > > Discussion
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > doc
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> with per-section
>> >> > status:
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >>
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >>
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > >
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > >
>> >> > > > > > > > >> > > > > > >> > > > > > > > > >
>> >> > > > > > > > >> > > > > > >> > > > > > > > >
>> >> > > > > > > > >> > > > > > >> > > > > > > >
>> >> > > > > > > > >> > > > > > >> > > > > > >
>> >> > > > > > > > >> > > > > > >> > > > > >
>> >> > > > > > > > >> > > > > > >> > > > >
>> >> > > > > > > > >> > > > > > >> > > >
>> >> > > > > > > > >> > > > > > >> > >
>> >> > > > > > > > >> > > > > > >> >
>> >> > > > > > > > >> > > > > > >>
>> >> > > > > > > > >> > > > > >
>> >> > > > > > > > >> > > > >
>> >> > > > > > > > >> > >
>> >> > > > > > > > >> >
>> >> > > > > > > > >>
>> >> > > > > > >
>> >> > > > > >
>> >> > > > >
>> >> > >
>> >> >
>> >>
>> https://urldefense.com/v3/__https://docs.google.com/document/d/100h7c4damrUzVuquYbBHM0EvA4LSWuW2IT2dN_7nYVA/edit?tab=t.0__;!!Iz9xO38YGHZK!4rnlVIqBQQQ6xpGe1PMV58edOS_cGrKoqdfhw4qSp_JQCavvEjaxXk0OwGPnxyEcO60AzKQK_XDtp7H3DdkFXcrXgIddeUs$
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >>
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> *The meeting
>> >> covered
>> >> > > both
>> >> > > > > > topics
>> >> > > > > > > > >> from
>> >> > > > > > > > >> > the
>> >> > > > > > > > >> > > > > doc.
>> >> > > > > > > > >> > > > > > >> > > > > > Direction-level
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > alignment
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> was reached on
>> the
>> >> > > headline
>> >> > > > > > > pieces;
>> >> > > > > > > > >> > > details
>> >> > > > > > > > >> > > > > > >> remain
>> >> > > > > > > > >> > > > > > >> > for
>> >> > > > > > > > >> > > > > > >> > > > PR
>> >> > > > > > > > >> > > > > > >> > > > > > > review
>> >> > > > > > > > >> > > > > > >> > > > > > > > > or
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> follow-up
>> >> sessions.*
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >>
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> *Topic 1 —
>> >> Persistence
>> >> > > > > schema
>> >> > > > > > > > >> redesign*
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> Idea-level
>> >> alignment
>> >> > on
>> >> > > > > > > > >> consolidating
>> >> > > > > > > > >> > > > > per-type
>> >> > > > > > > > >> > > > > > >> > tables
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >>
>> >> (scan_metrics_report,
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >>
>> >> commit_metrics_report)
>> >> > > into
>> >> > > > > a
>> >> > > > > > > single
>> >> > > > > > > > >> > > > > > >> metrics_report
>> >> > > > > > > > >> > > > > > >> > > > table.
>> >> > > > > > > > >> > > > > > >> > > > > > The
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > motivating
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> cost is the
>> surface
>> >> > area
>> >> > > > > added
>> >> > > > > > > by
>> >> > > > > > > > >> every
>> >> > > > > > > > >> > > new
>> >> > > > > > > > >> > > > > > >> metric
>> >> > > > > > > > >> > > > > > >> > > type
>> >> > > > > > > > >> > > > > > >> > > > > > today:
>> >> > > > > > > > >> > > > > > >> > > > > > > > new
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > table,
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> SPI method,
>> record
>> >> > > class,
>> >> > > > > > model,
>> >> > > > > > > > >> > > converter,
>> >> > > > > > > > >> > > > > > >> schema
>> >> > > > > > > > >> > > > > > >> > > > > > migration.
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >>
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> Most schema
>> details
>> >> > are
>> >> > > > > > > deferred to
>> >> > > > > > > > >> the
>> >> > > > > > > > >> > > > > schema
>> >> > > > > > > > >> > > > > > >> PR. A
>> >> > > > > > > > >> > > > > > >> > > few
>> >> > > > > > > > >> > > > > > >> > > > > > > > specific
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > points
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> came up:
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> •
>> >> > >  metric_schema_version:
>> >> > > > > > Yufei
>> >> > > > > > > > >> > prefers
>> >> > > > > > > > >> > > > > > dropping
>> >> > > > > > > > >> > > > > > >> > it,
>> >> > > > > > > > >> > > > > > >> > > > > since
>> >> > > > > > > > >> > > > > > >> > > > > > > > there
>> >> > > > > > > > >> > > > > > >> > > > > > > > > is
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > no
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> spec-level
>> concept
>> >> of
>> >> > > > > metrics
>> >> > > > > > > > >> > versioning
>> >> > > > > > > > >> > > > > today
>> >> > > > > > > > >> > > > > > >> and
>> >> > > > > > > > >> > > > > > >> > it
>> >> > > > > > > > >> > > > > > >> > > is
>> >> > > > > > > > >> > > > > > >> > > > > > hard
>> >> > > > > > > > >> > > > > > >> > > > > > > to
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > define
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> unilaterally.
>> >> Robert
>> >> > > prefers
>> >> > > > > > > keeping
>> >> > > > > > > > >> > it,
>> >> > > > > > > > >> > > > > given
>> >> > > > > > > > >> > > > > > >> IRC
>> >> > > > > > > > >> > > > > > >> > v2
>> >> > > > > > > > >> > > > > > >> > > is
>> >> > > > > > > > >> > > > > > >> > > > > > > coming
>> >> > > > > > > > >> > > > > > >> > > > > > > > > and
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > the
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> schema should be
>> >> > > considered
>> >> > > > > > > against
>> >> > > > > > > > >> its
>> >> > > > > > > > >> > > > > likely
>> >> > > > > > > > >> > > > > > >> > shape;
>> >> > > > > > > > >> > > > > > >> > > > > Robert
>> >> > > > > > > > >> > > > > > >> > > > > > > > also
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > raised
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> how to
>> >> differentiate
>> >> > > various
>> >> > > > > > > payload
>> >> > > > > > > > >> > > formats
>> >> > > > > > > > >> > > > > if
>> >> > > > > > > > >> > > > > > >> any.
>> >> > > > > > > > >> > > > > > >> > > > EJ's
>> >> > > > > > > > >> > > > > > >> > > > > > read
>> >> > > > > > > > >> > > > > > >> > > > > > > > is
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > that
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> this
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> is a
>> two-way-door
>> >> > > decision.
>> >> > > > > We
>> >> > > > > > > can
>> >> > > > > > > > >> > start
>> >> > > > > > > > >> > > > > > without
>> >> > > > > > > > >> > > > > > >> the
>> >> > > > > > > > >> > > > > > >> > > > > field,
>> >> > > > > > > > >> > > > > > >> > > > > > > and
>> >> > > > > > > > >> > > > > > >> > > > > > > > if
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > IRC
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > v2
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> changes the
>> shape
>> >> we
>> >> > > would
>> >> > > > > > > likely
>> >> > > > > > > > >> roll
>> >> > > > > > > > >> > a
>> >> > > > > > > > >> > > > > > >> > corresponding
>> >> > > > > > > > >> > > > > > >> > > > new
>> >> > > > > > > > >> > > > > > >> > > > > > > > schema
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > anyway,
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> which is not
>> >> > > particularly
>> >> > > > > > > costly.
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> •   Payload
>> format:
>> >> > > Robert
>> >> > > > > > > pointed
>> >> > > > > > > > >> out
>> >> > > > > > > > >> > > that
>> >> > > > > > > > >> > > > > > >> future
>> >> > > > > > > > >> > > > > > >> > > > formats
>> >> > > > > > > > >> > > > > > >> > > > > > > > beyond
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > JSON
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > may
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> be worth
>> >> supporting.
>> >> > The
>> >> > > > > exact
>> >> > > > > > > > >> shape is
>> >> > > > > > > > >> > > > > > deferred
>> >> > > > > > > > >> > > > > > >> to
>> >> > > > > > > > >> > > > > > >> > > the
>> >> > > > > > > > >> > > > > > >> > > > > > schema
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > discussion.
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> •   Partition
>> >> > strategy:
>> >> > > > > Anand
>> >> > > > > > > > >> suggested
>> >> > > > > > > > >> > > > > monthly
>> >> > > > > > > > >> > > > > > >> > > > > partitioning
>> >> > > > > > > > >> > > > > > >> > > > > > > > based
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > on
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > his
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> experience as
>> >> > > potentially
>> >> > > > > > > helpful at
>> >> > > > > > > > >> > > scale.
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >>
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> *Topic 2 — Where
>> >> > metrics
>> >> > > > > > > ingestion
>> >> > > > > > > > >> and
>> >> > > > > > > > >> > > > > storage
>> >> > > > > > > > >> > > > > > >> > belong*
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> Idea-level
>> >> alignment
>> >> > > that
>> >> > > > > > > metrics
>> >> > > > > > > > >> > should
>> >> > > > > > > > >> > > be a
>> >> > > > > > > > >> > > > > > >> > > separated
>> >> > > > > > > > >> > > > > > >> > > > > SPI
>> >> > > > > > > > >> > > > > > >> > > > > > > from
>> >> > > > > > > > >> > > > > > >> > > > > > > > > the
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> entity
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> persistence
>> stack.
>> >> Two
>> >> > > > > reasons
>> >> > > > > > > > >> > surfaced:
>> >> > > > > > > > >> > > (a)
>> >> > > > > > > > >> > > > > > >> > workloads
>> >> > > > > > > > >> > > > > > >> > > > and
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > capability
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> requirements
>> >> diverge
>> >> > > enough
>> >> > > > > > that
>> >> > > > > > > > >> > coupling
>> >> > > > > > > > >> > > > > them
>> >> > > > > > > > >> > > > > > >> > creates
>> >> > > > > > > > >> > > > > > >> > > > > > > > artificial
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> constraints, and
>> >> (b)
>> >> > > admin
>> >> > > > > > > > >> experience
>> >> > > > > > > > >> > > > > improves
>> >> > > > > > > > >> > > > > > >> when
>> >> > > > > > > > >> > > > > > >> > > > > metrics
>> >> > > > > > > > >> > > > > > >> > > > > > > has
>> >> > > > > > > > >> > > > > > >> > > > > > > > > its
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > own
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> bootstrap,
>> >> retention,
>> >> > > and
>> >> > > > > > > lifecycle.
>> >> > > > > > > > >> > > Dmitri
>> >> > > > > > > > >> > > > > > noted
>> >> > > > > > > > >> > > > > > >> > > > Polaris
>> >> > > > > > > > >> > > > > > >> > > > > > > being
>> >> > > > > > > > >> > > > > > >> > > > > > > > a
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > platform
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> should have the
>> >> > > flexibility
>> >> > > > > to
>> >> > > > > > > > >> support
>> >> > > > > > > > >> > > > > > different
>> >> > > > > > > > >> > > > > > >> > > > > persistence
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > backends
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > per
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> concern, and
>> >> pointed
>> >> > to
>> >> > > a
>> >> > > > > > > concrete
>> >> > > > > > > > >> next
>> >> > > > > > > > >> > > step
>> >> > > > > > > > >> > > > > of
>> >> > > > > > > > >> > > > > > >> > > > separating
>> >> > > > > > > > >> > > > > > >> > > > > > the
>> >> > > > > > > > >> > > > > > >> > > > > > > > > JDBC
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> bootstrap for
>> >> metrics
>> >> > > from
>> >> > > > > the
>> >> > > > > > > > >> > metastore
>> >> > > > > > > > >> > > > > > >> bootstrap.
>> >> > > > > > > > >> > > > > > >> > > > Robert
>> >> > > > > > > > >> > > > > > >> > > > > > > > > proposed
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > an
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> additional UX
>> >> > extension:
>> >> > > > > > detect
>> >> > > > > > > an
>> >> > > > > > > > >> > > > > > unbootstrapped
>> >> > > > > > > > >> > > > > > >> > > > metrics
>> >> > > > > > > > >> > > > > > >> > > > > > > store
>> >> > > > > > > > >> > > > > > >> > > > > > > > on
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > first
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> use and
>> >> auto-bootstrap
>> >> > > > > rather
>> >> > > > > > > than
>> >> > > > > > > > >> > > requiring
>> >> > > > > > > > >> > > > > an
>> >> > > > > > > > >> > > > > > >> > > explicit
>> >> > > > > > > > >> > > > > > >> > > > > > > manual
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > bootstrap
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> step.
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> The meeting also
>> >> > > confirmed
>> >> > > > > > that
>> >> > > > > > > > >> Polaris
>> >> > > > > > > > >> > > > > metrics
>> >> > > > > > > > >> > > > > > >> can
>> >> > > > > > > > >> > > > > > >> > > > start
>> >> > > > > > > > >> > > > > > >> > > > > > > small
>> >> > > > > > > > >> > > > > > >> > > > > > > > > and
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > stay
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> Iceberg-focused.
>> >> > Naming
>> >> > > and
>> >> > > > > > > > >> persistence
>> >> > > > > > > > >> > > > > schema
>> >> > > > > > > > >> > > > > > >> can
>> >> > > > > > > > >> > > > > > >> > > lean
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > Iceberg-specific.
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> If a future
>> >> expansion
>> >> > to
>> >> > > > > > > > >> generic-table
>> >> > > > > > > > >> > > > > metrics
>> >> > > > > > > > >> > > > > > or
>> >> > > > > > > > >> > > > > > >> > > > > > operational
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > metrics
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> arrives, an
>> >> > abstraction
>> >> > > > > layer
>> >> > > > > > > can be
>> >> > > > > > > > >> > > built on
>> >> > > > > > > > >> > > > > > >> top of
>> >> > > > > > > > >> > > > > > >> > > the
>> >> > > > > > > > >> > > > > > >> > > > > > > Iceberg
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > metrics
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> reporter at that
>> >> > point.
>> >> > > > > Robert
>> >> > > > > > > > >> remains
>> >> > > > > > > > >> > > on the
>> >> > > > > > > > >> > > > > > >> fence
>> >> > > > > > > > >> > > > > > >> > > and
>> >> > > > > > > > >> > > > > > >> > > > > > would
>> >> > > > > > > > >> > > > > > >> > > > > > > > > prefer
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> something more
>> >> generic
>> >> > > but
>> >> > > > > did
>> >> > > > > > > not
>> >> > > > > > > > >> > block
>> >> > > > > > > > >> > > the
>> >> > > > > > > > >> > > > > > >> > > direction;
>> >> > > > > > > > >> > > > > > >> > > > > > > Dmitri's
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > read
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > was
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> that the
>> proposed
>> >> > > framework
>> >> > > > > > > already
>> >> > > > > > > > >> has
>> >> > > > > > > > >> > > > > enough
>> >> > > > > > > > >> > > > > > >> > > > flexibility
>> >> > > > > > > > >> > > > > > >> > > > > > to
>> >> > > > > > > > >> > > > > > >> > > > > > > > > absorb
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> future
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> expansion.
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >>
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> The Trade-offs
>> and
>> >> > > Proposed
>> >> > > > > > > > >> structure
>> >> > > > > > > > >> > > > > sections
>> >> > > > > > > > >> > > > > > in
>> >> > > > > > > > >> > > > > > >> > the
>> >> > > > > > > > >> > > > > > >> > > > doc
>> >> > > > > > > > >> > > > > > >> > > > > > were
>> >> > > > > > > > >> > > > > > >> > > > > > > > not
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> reviewed
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> in detail. They
>> >> remain
>> >> > > open
>> >> > > > > > for
>> >> > > > > > > > >> either
>> >> > > > > > > > >> > > the
>> >> > > > > > > > >> > > > > next
>> >> > > > > > > > >> > > > > > >> sync
>> >> > > > > > > > >> > > > > > >> > > or
>> >> > > > > > > > >> > > > > > >> > > > PR
>> >> > > > > > > > >> > > > > > >> > > > > > > > review.
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >>
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> *Cross-cutting
>> >> > > alignment —
>> >> > > > > > > > >> > > battery-included
>> >> > > > > > > > >> > > > > > plus
>> >> > > > > > > > >> > > > > > >> > > > > pluggable*
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> A common
>> philosophy
>> >> > > emerged
>> >> > > > > > > from the
>> >> > > > > > > > >> > > > > > discussion.
>> >> > > > > > > > >> > > > > > >> EJ
>> >> > > > > > > > >> > > > > > >> > > > > > summarized
>> >> > > > > > > > >> > > > > > >> > > > > > > > it
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > as:
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> Polaris should
>> >> > provide a
>> >> > > > > > > > >> > > battery-included UX
>> >> > > > > > > > >> > > > > > for
>> >> > > > > > > > >> > > > > > >> > > > beginners
>> >> > > > > > > > >> > > > > > >> > > > > > and
>> >> > > > > > > > >> > > > > > >> > > > > > > > the
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> flexibility for
>> >> > advanced
>> >> > > > > users
>> >> > > > > > > to
>> >> > > > > > > > >> swap
>> >> > > > > > > > >> > > the
>> >> > > > > > > > >> > > > > > >> included
>> >> > > > > > > > >> > > > > > >> > > > > battery
>> >> > > > > > > > >> > > > > > >> > > > > > > for
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > something
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> more powerful or
>> >> > > tailored to
>> >> > > > > > > their
>> >> > > > > > > > >> use
>> >> > > > > > > > >> > > case.
>> >> > > > > > > > >> > > > > > The
>> >> > > > > > > > >> > > > > > >> SPI
>> >> > > > > > > > >> > > > > > >> > > > > design
>> >> > > > > > > > >> > > > > > >> > > > > > > > needs
>> >> > > > > > > > >> > > > > > >> > > > > > > > > to
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> enable
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> both.
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >>
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> The inputs that
>> >> shaped
>> >> > > this
>> >> > > > > > > framing:
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> •   Anand
>> described
>> >> > how
>> >> > > his
>> >> > > > > > team
>> >> > > > > > > > >> uses
>> >> > > > > > > > >> > the
>> >> > > > > > > > >> > > > > > current
>> >> > > > > > > > >> > > > > > >> > > > metrics
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > persistence
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> (three metrics
>> >> > > consumers in
>> >> > > > > > > v1.4).
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> •   Yufei raised
>> >> > > Grafana and
>> >> > > > > > > > >> dashboard
>> >> > > > > > > > >> > > > > > >> integrations
>> >> > > > > > > > >> > > > > > >> > > as a
>> >> > > > > > > > >> > > > > > >> > > > > > > > > destination
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > use
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> case beyond the
>> >> > default.
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> •   Robert
>> called
>> >> out
>> >> > > that
>> >> > > > > the
>> >> > > > > > > > >> current
>> >> > > > > > > > >> > > design
>> >> > > > > > > > >> > > > > > is
>> >> > > > > > > > >> > > > > > >> > more
>> >> > > > > > > > >> > > > > > >> > > > > > > > > JDBC-focused.
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >>
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> Two concrete
>> >> > instances:
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> •   Async
>> metrics
>> >> > > intake:
>> >> > > > > > > Yufei's
>> >> > > > > > > > >> > initial
>> >> > > > > > > > >> > > > > > >> position
>> >> > > > > > > > >> > > > > > >> > was
>> >> > > > > > > > >> > > > > > >> > > > > that
>> >> > > > > > > > >> > > > > > >> > > > > > > > async
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > should
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> largely live on
>> the
>> >> > > producer
>> >> > > > > > > side
>> >> > > > > > > > >> and
>> >> > > > > > > > >> > > there
>> >> > > > > > > > >> > > > > is
>> >> > > > > > > > >> > > > > > >> not
>> >> > > > > > > > >> > > > > > >> > > much
>> >> > > > > > > > >> > > > > > >> > > > > > > Polaris
>> >> > > > > > > > >> > > > > > >> > > > > > > > > can
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > do.
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> Robert
>> suggested a
>> >> > > > > > Polaris-side
>> >> > > > > > > > >> default
>> >> > > > > > > > >> > > is
>> >> > > > > > > > >> > > > > > doable
>> >> > > > > > > > >> > > > > > >> > via
>> >> > > > > > > > >> > > > > > >> > > > > > Vert.x.
>> >> > > > > > > > >> > > > > > >> > > > > > > > > Dmitri
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> agreed
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> the direction is
>> >> worth
>> >> > > > > > > exploring.
>> >> > > > > > > > >> The
>> >> > > > > > > > >> > > meeting
>> >> > > > > > > > >> > > > > > >> > > converged
>> >> > > > > > > > >> > > > > > >> > > > > on a
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> battery-included
>> >> > default
>> >> > > > > > (likely
>> >> > > > > > > > >> > > > > Vert.x-backed)
>> >> > > > > > > > >> > > > > > >> with
>> >> > > > > > > > >> > > > > > >> > > an
>> >> > > > > > > > >> > > > > > >> > > > > SPI
>> >> > > > > > > > >> > > > > > >> > > > > > > > shape
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > that
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> lets
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> power users
>> route
>> >> to a
>> >> > > more
>> >> > > > > > > scalable
>> >> > > > > > > > >> > > backend
>> >> > > > > > > > >> > > > > > >> > > (k8s-hosted
>> >> > > > > > > > >> > > > > > >> > > > > > > queue,
>> >> > > > > > > > >> > > > > > >> > > > > > > > > AWS
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > SQS,
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> etc.).
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> •   Pluggable
>> >> > > destinations:
>> >> > > > > > > > >> combining
>> >> > > > > > > > >> > > Yufei's
>> >> > > > > > > > >> > > > > > >> > > dashboard
>> >> > > > > > > > >> > > > > > >> > > > > use
>> >> > > > > > > > >> > > > > > >> > > > > > > case
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > with
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> Robert's
>> >> JDBC-focused
>> >> > > > > > call-out,
>> >> > > > > > > the
>> >> > > > > > > > >> > > meeting
>> >> > > > > > > > >> > > > > > >> agreed
>> >> > > > > > > > >> > > > > > >> > the
>> >> > > > > > > > >> > > > > > >> > > > SPI
>> >> > > > > > > > >> > > > > > >> > > > > > > > should
>> >> > > > > > > > >> > > > > > >> > > > > > > > > be
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> structured for
>> >> > multiple
>> >> > > > > sinks
>> >> > > > > > so
>> >> > > > > > > > >> > > integrations
>> >> > > > > > > > >> > > > > > >> become
>> >> > > > > > > > >> > > > > > >> > > > impl
>> >> > > > > > > > >> > > > > > >> > > > > > > > choices
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > rather
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> than
>> architectural
>> >> > > changes.
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >>
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> The
>> >> battery-included
>> >> > > default
>> >> > > > > > is
>> >> > > > > > > most
>> >> > > > > > > > >> > > likely
>> >> > > > > > > > >> > > > > to
>> >> > > > > > > > >> > > > > > >> use
>> >> > > > > > > > >> > > > > > >> > the
>> >> > > > > > > > >> > > > > > >> > > > > > > existing
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> JDBC-backed
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> approach.
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >>
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> *Direction
>> >> (idea-level
>> >> > > > > > > alignment)*
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> •   Single
>> >> > > metrics_report
>> >> > > > > > table
>> >> > > > > > > > >> > > consolidating
>> >> > > > > > > > >> > > > > > >> > per-type
>> >> > > > > > > > >> > > > > > >> > > > > > > metrics,
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > replacing
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >>
>> scan_metrics_report
>> >> > and
>> >> > > > > > > > >> > > commit_metrics_report
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> •
>>  Iceberg-focused
>> >> > > naming
>> >> > > > > and
>> >> > > > > > > > >> schema
>> >> > > > > > > > >> > for
>> >> > > > > > > > >> > > > > now,
>> >> > > > > > > > >> > > > > > >> > revisit
>> >> > > > > > > > >> > > > > > >> > > > if
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > generic-table
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > or
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> operational
>> metrics
>> >> > > arrive
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> •   Metrics
>> >> > persistence
>> >> > > as a
>> >> > > > > > > > >> separated
>> >> > > > > > > > >> > > SPI,
>> >> > > > > > > > >> > > > > not
>> >> > > > > > > > >> > > > > > >> on
>> >> > > > > > > > >> > > > > > >> > > > > > > > BasePersistence
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> •   Bootstrap
>> path
>> >> > > separated
>> >> > > > > > for
>> >> > > > > > > > >> > metrics,
>> >> > > > > > > > >> > > > > > >> > independent
>> >> > > > > > > > >> > > > > > >> > > of
>> >> > > > > > > > >> > > > > > >> > > > > > > > metastore
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> bootstrap
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> •
>> >>  "Battery-included
>> >> > > plus
>> >> > > > > > > > >> pluggable"
>> >> > > > > > > > >> > as
>> >> > > > > > > > >> > > the
>> >> > > > > > > > >> > > > > > SPI
>> >> > > > > > > > >> > > > > > >> > > design
>> >> > > > > > > > >> > > > > > >> > > > > > > > philosophy
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >>
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> *Open items*
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> •   Schema
>> details:
>> >> > > > > > > > >> > > metric_schema_version,
>> >> > > > > > > > >> > > > > > >> payload
>> >> > > > > > > > >> > > > > > >> > > > format,
>> >> > > > > > > > >> > > > > > >> > > > > > IRC
>> >> > > > > > > > >> > > > > > >> > > > > > > > v2
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> forward-compat
>> >> shape
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> •   SPI design
>> >> > details —
>> >> > > > > full
>> >> > > > > > > review
>> >> > > > > > > > >> > > either
>> >> > > > > > > > >> > > > > in
>> >> > > > > > > > >> > > > > > >> the
>> >> > > > > > > > >> > > > > > >> > > next
>> >> > > > > > > > >> > > > > > >> > > > > sync
>> >> > > > > > > > >> > > > > > >> > > > > > > or
>> >> > > > > > > > >> > > > > > >> > > > > > > > in
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > the
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> corresponding PR
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> •   Schema
>> >> refactor PR
>> >> > > > > > ownership
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >>
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> *Action items*
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> •   EJ to take a
>> >> first
>> >> > > stab
>> >> > > > > at
>> >> > > > > > > the
>> >> > > > > > > > >> SPI
>> >> > > > > > > > >> > > design
>> >> > > > > > > > >> > > > > > and
>> >> > > > > > > > >> > > > > > >> > > > > > potentially
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > partner
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > with
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> Anand to
>> >> incorporate
>> >> > the
>> >> > > > > > lessons
>> >> > > > > > > > >> > learned
>> >> > > > > > > > >> > > from
>> >> > > > > > > > >> > > > > > the
>> >> > > > > > > > >> > > > > > >> > > > existing
>> >> > > > > > > > >> > > > > > >> > > > > > > > > reporter
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > and
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> persistence
>> work.
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> •   Schema
>> >> refactor PR
>> >> > > > > > > ownership is
>> >> > > > > > > > >> not
>> >> > > > > > > > >> > > yet
>> >> > > > > > > > >> > > > > > >> decided.
>> >> > > > > > > > >> > > > > > >> > > If
>> >> > > > > > > > >> > > > > > >> > > > > > anyone
>> >> > > > > > > > >> > > > > > >> > > > > > > > is
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> interested in
>> >> driving
>> >> > > it,
>> >> > > > > > reply
>> >> > > > > > > on
>> >> > > > > > > > >> this
>> >> > > > > > > > >> > > > > thread.
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> •   JB to
>> schedule
>> >> the
>> >> > > next
>> >> > > > > > > sync,
>> >> > > > > > > > >> > > tentatively
>> >> > > > > > > > >> > > > > > in
>> >> > > > > > > > >> > > > > > >> two
>> >> > > > > > > > >> > > > > > >> > > > > weeks.
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >>
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> -ej
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >>
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> On Mon, Apr 27,
>> >> 2026
>> >> > at
>> >> > > > > > 3:07 PM
>> >> > > > > > > EJ
>> >> > > > > > > > >> > Wang <
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > >
>> >> > > ej.wang.dev.interest@gmail.com
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> wrote:
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >>
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> > Thanks Yufei
>> for
>> >> the
>> >> > > +1.
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> > JB, could you
>> >> help
>> >> > > add a
>> >> > > > > > > biweekly
>> >> > > > > > > > >> > > metrics
>> >> > > > > > > > >> > > > > > >> > > architecture
>> >> > > > > > > > >> > > > > > >> > > > > > sync
>> >> > > > > > > > >> > > > > > >> > > > > > > to
>> >> > > > > > > > >> > > > > > >> > > > > > > > > the
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> Polaris
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> > community
>> >> calendar?
>> >> > > I'm
>> >> > > > > > > thinking
>> >> > > > > > > > >> > > Thursdays
>> >> > > > > > > > >> > > > > at
>> >> > > > > > > > >> > > > > > >> > 9-10am
>> >> > > > > > > > >> > > > > > >> > > > PT,
>> >> > > > > > > > >> > > > > > >> > > > > > on
>> >> > > > > > > > >> > > > > > >> > > > > > > > the
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> off-weeks
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> > from the
>> >> community
>> >> > > meeting
>> >> > > > > > > > >> (starting
>> >> > > > > > > > >> > > May
>> >> > > > > > > > >> > > > > 7),
>> >> > > > > > > > >> > > > > > 60
>> >> > > > > > > > >> > > > > > >> > > > minutes.
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> > Here's a rough
>> >> > agenda
>> >> > > to
>> >> > > > > > work
>> >> > > > > > > > >> through
>> >> > > > > > > > >> > > over
>> >> > > > > > > > >> > > > > > the
>> >> > > > > > > > >> > > > > > >> > first
>> >> > > > > > > > >> > > > > > >> > > > few
>> >> > > > > > > > >> > > > > > >> > > > > > > > > sessions,
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> grouped
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> > by priority:
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> > *First:
>> >> foundational
>> >> > > > > > > direction*
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> > 1.
>> >> > > MetricsPersistence:
>> >> > > > > > public
>> >> > > > > > > > >> SPI or
>> >> > > > > > > > >> > > > > > internal
>> >> > > > > > > > >> > > > > > >> > > > > > > implementation
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > detail?
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >    •   Marked
>> >> @Beta,
>> >> > > > > javadoc
>> >> > > > > > > calls
>> >> > > > > > > > >> > it a
>> >> > > > > > > > >> > > > > > >> "Service
>> >> > > > > > > > >> > > > > > >> > > > > Provider
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > Interface",
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> but
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> > only one
>> consumer
>> >> > > > > > > > >> > > > > (JdbcBasePersistenceImpl),
>> >> > > > > > > > >> > > > > > >> lives
>> >> > > > > > > > >> > > > > > >> > > on
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > BasePersistence.
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> If
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> > demoted to a
>> >> private
>> >> > > > > helper
>> >> > > > > > > > >> inside a
>> >> > > > > > > > >> > > > > > persisting
>> >> > > > > > > > >> > > > > > >> > > > reporter
>> >> > > > > > > > >> > > > > > >> > > > > > > impl,
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > most
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> > downstream
>> design
>> >> > > > > decisions
>> >> > > > > > > become
>> >> > > > > > > > >> > > > > > >> implementation
>> >> > > > > > > > >> > > > > > >> > > > > details
>> >> > > > > > > > >> > > > > > >> > > > > > > > rather
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > than
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> > contract
>> >> questions.
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> > 2.
>> Persistence
>> >> > schema
>> >> > > > > > > redesign
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >    •   Current
>> >> > > two-table
>> >> > > > > > > layout
>> >> > > > > > > > >> > > > > > >> > > (scan_metrics_report,
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >
>> >> > commit_metrics_report)
>> >> > > > > with
>> >> > > > > > > ~25
>> >> > > > > > > > >> > > flattened
>> >> > > > > > > > >> > > > > > >> columns
>> >> > > > > > > > >> > > > > > >> > > > each.
>> >> > > > > > > > >> > > > > > >> > > > > > > Every
>> >> > > > > > > > >> > > > > > >> > > > > > > > > new
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > metric
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> > type requires
>> a
>> >> new
>> >> > > table,
>> >> > > > > > SPI
>> >> > > > > > > > >> > method,
>> >> > > > > > > > >> > > > > record
>> >> > > > > > > > >> > > > > > >> > class,
>> >> > > > > > > > >> > > > > > >> > > > > > model,
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > converter,
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> and
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> > schema
>> migration.
>> >> > > > > Direction
>> >> > > > > > to
>> >> > > > > > > > >> > explore:
>> >> > > > > > > > >> > > > > > single
>> >> > > > > > > > >> > > > > > >> > table
>> >> > > > > > > > >> > > > > > >> > > > > with
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > metric_type
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> enum,
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >
>> schema_version,
>> >> and
>> >> > > JSON
>> >> > > > > > > payload
>> >> > > > > > > > >> > > column.
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> > *Second:
>> design
>> >> > > details
>> >> > > > > once
>> >> > > > > > > > >> > direction
>> >> > > > > > > > >> > > is
>> >> > > > > > > > >> > > > > > set*
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> > 3.  Partition
>> key
>> >> > > strategy
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >    •
>> >>  Single-table
>> >> > > design
>> >> > > > > > > means
>> >> > > > > > > > >> scan
>> >> > > > > > > > >> > > > > metrics
>> >> > > > > > > > >> > > > > > >> at
>> >> > > > > > > > >> > > > > > >> > > scale
>> >> > > > > > > > >> > > > > > >> > > > > > will
>> >> > > > > > > > >> > > > > > >> > > > > > > > have
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > high
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> > write
>> concurrency
>> >> > per
>> >> > > > > table.
>> >> > > > > > > > >> Schema
>> >> > > > > > > > >> > > needs
>> >> > > > > > > > >> > > > > to
>> >> > > > > > > > >> > > > > > >> > expose
>> >> > > > > > > > >> > > > > > >> > > > > enough
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > structure
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > for
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> > backends to
>> >> shard by
>> >> > > > > entity
>> >> > > > > > or
>> >> > > > > > > > >> time
>> >> > > > > > > > >> > > range.
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> > 4.  Read/write
>> >> path
>> >> > > > > > > consistency
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >    •   Writes
>> go
>> >> > > through
>> >> > > > > > > > >> > > > > > PolarisMetricsManager
>> >> > > > > > > > >> > > > > > >> on
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > MetaStoreManager.
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> Reads
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> > bypass
>> >> > > MetaStoreManager
>> >> > > > > and
>> >> > > > > > go
>> >> > > > > > > > >> > > straight to
>> >> > > > > > > > >> > > > > > >> > > > > > BasePersistence,
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > excluding
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> > non-JDBC
>> backends
>> >> > > from the
>> >> > > > > > > read
>> >> > > > > > > > >> API.
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> > *Third:
>> cleanup
>> >> and
>> >> > > > > > alignment*
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> > 5.
>> >> > > PolarisMetricsReporter
>> >> > > > > > > naming
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >    •   Only
>> >> handles
>> >> > > IRC
>> >> > > > > > > > >> > > > > > >> (ScanReport/CommitReport),
>> >> > > > > > > > >> > > > > > >> > > > > doesn't
>> >> > > > > > > > >> > > > > > >> > > > > > > > cover
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > generic
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> > tables or
>> >> > operational
>> >> > > > > > metrics.
>> >> > > > > > > > >> Name
>> >> > > > > > > > >> > is
>> >> > > > > > > > >> > > > > > broader
>> >> > > > > > > > >> > > > > > >> > than
>> >> > > > > > > > >> > > > > > >> > > > > scope.
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> > 6.
>> >> > > PolarisMetricsManager
>> >> > > > > > > facade
>> >> > > > > > > > >> > > > > passthrough
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >    •   Entire
>> >> > default
>> >> > > > > method
>> >> > > > > > > is
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >>
>> >> > > > > > > > >> > callCtx.getMetaStore().writeScanReport().
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> > Zero logic,
>> >> passes
>> >> > > Level 1
>> >> > > > > > > > >> straight
>> >> > > > > > > > >> > > through
>> >> > > > > > > > >> > > > > > to
>> >> > > > > > > > >> > > > > > >> > Level
>> >> > > > > > > > >> > > > > > >> > > > 3.
>> >> > > > > > > > >> > > > > > >> > > > > > Same
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> anti-pattern
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> > as
>> >> > > PolarisEventManager.
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> > 7.  Iceberg
>> >> > community
>> >> > > > > > > alignment
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >    •
>> >>  Payload-type
>> >> > > > > > extension
>> >> > > > > > > > >> needs
>> >> > > > > > > > >> > > > > > discussion
>> >> > > > > > > > >> > > > > > >> on
>> >> > > > > > > > >> > > > > > >> > > > > > > dev@iceberg.
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> obelix74's
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> > Feb thread got
>> >> zero
>> >> > > > > replies.
>> >> > > > > > > > >> Needs a
>> >> > > > > > > > >> > > > > > committer
>> >> > > > > > > > >> > > > > > >> > > voice.
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> > Lets confirm
>> >> > > > > prioritization
>> >> > > > > > > in the
>> >> > > > > > > > >> > > first
>> >> > > > > > > > >> > > > > > >> session.
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> > -ej
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> > On Tue, Apr
>> 21,
>> >> 2026
>> >> > > at
>> >> > > > > > > 3:18 PM
>> >> > > > > > > > >> Yufei
>> >> > > > > > > > >> > > Gu <
>> >> > > > > > > > >> > > > > > >> > > > > > > > flyrain000@gmail.com>
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > wrote:
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> Thanks
>> everyone
>> >> for
>> >> > > > > > > continuing to
>> >> > > > > > > > >> > > drive
>> >> > > > > > > > >> > > > > this
>> >> > > > > > > > >> > > > > > >> > > > forward. I
>> >> > > > > > > > >> > > > > > >> > > > > > > agree
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > that
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > the
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> problem is
>> >> getting
>> >> > > > > complex
>> >> > > > > > > enough
>> >> > > > > > > > >> > > that a
>> >> > > > > > > > >> > > > > > more
>> >> > > > > > > > >> > > > > > >> > > > > structured
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > discussion
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> would
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> help.
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >>
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> +1 on setting
>> >> up a
>> >> > > > > biweekly
>> >> > > > > > > sync
>> >> > > > > > > > >> for
>> >> > > > > > > > >> > > the
>> >> > > > > > > > >> > > > > > >> metrics
>> >> > > > > > > > >> > > > > > >> > > > > > > > architecture.
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > I’m
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> happy
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> to
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> join.
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >>
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> Yufei
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >>
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >>
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> On Tue, Apr
>> 21,
>> >> > 2026
>> >> > > at
>> >> > > > > > > 2:34 PM
>> >> > > > > > > > >> EJ
>> >> > > > > > > > >> > > Wang <
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >>
>> >> > > > > > ej.wang.dev.interest@gmail.com>
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> wrote:
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >>
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> > Also, I've
>> >> been
>> >> > > looking
>> >> > > > > > > more
>> >> > > > > > > > >> > > closely at
>> >> > > > > > > > >> > > > > > the
>> >> > > > > > > > >> > > > > > >> > > > > > *persistence
>> >> > > > > > > > >> > > > > > >> > > > > > > > > schema
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > in
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> the
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> > current
>> >> metrics
>> >> > > work*,
>> >> > > > > > and
>> >> > > > > > > I
>> >> > > > > > > > >> think
>> >> > > > > > > > >> > > > > > there's a
>> >> > > > > > > > >> > > > > > >> > > > > structural
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > rigidity
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> problem
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> > worth
>> raising
>> >> > > before
>> >> > > > > the
>> >> > > > > > > shape
>> >> > > > > > > > >> > gets
>> >> > > > > > > > >> > > > > locked
>> >> > > > > > > > >> > > > > > >> in.
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> > Right now
>> we
>> >> have
>> >> > > two
>> >> > > > > > > separate
>> >> > > > > > > > >> > > tables
>> >> > > > > > > > >> > > > > > >> > > > > > > (scan_metrics_report
>> >> > > > > > > > >> > > > > > >> > > > > > > > > and
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >
>> >> > > commit_metrics_report),
>> >> > > > > > > each
>> >> > > > > > > > >> with
>> >> > > > > > > > >> > > ~25
>> >> > > > > > > > >> > > > > > >> flattened
>> >> > > > > > > > >> > > > > > >> > > > > columns
>> >> > > > > > > > >> > > > > > >> > > > > > > > that
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > directly
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> > mirror the
>> >> > Iceberg
>> >> > > > > report
>> >> > > > > > > > >> fields.
>> >> > > > > > > > >> > > The
>> >> > > > > > > > >> > > > > SPI
>> >> > > > > > > > >> > > > > > >> > follows
>> >> > > > > > > > >> > > > > > >> > > > the
>> >> > > > > > > > >> > > > > > >> > > > > > > same
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > split:
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >
>> >> writeScanReport
>> >> > and
>> >> > > > > > > > >> > > writeCommitReport as
>> >> > > > > > > > >> > > > > > >> > separate
>> >> > > > > > > > >> > > > > > >> > > > > > > methods,
>> >> > > > > > > > >> > > > > > >> > > > > > > > > with
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> per-type
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> > record
>> >> classes,
>> >> > > > > > > converters, and
>> >> > > > > > > > >> > > model
>> >> > > > > > > > >> > > > > > >> objects.
>> >> > > > > > > > >> > > > > > >> > > *The
>> >> > > > > > > > >> > > > > > >> > > > > > > > practical
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > cost:
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> > adding a
>> new
>> >> > metric
>> >> > > > > type
>> >> > > > > > > > >> > > (operational
>> >> > > > > > > > >> > > > > > >> metrics,
>> >> > > > > > > > >> > > > > > >> > > for
>> >> > > > > > > > >> > > > > > >> > > > > > > example)
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > requires
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> a
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> new
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> > table, a
>> new
>> >> SPI
>> >> > > > > method,
>> >> > > > > > a
>> >> > > > > > > new
>> >> > > > > > > > >> > > record
>> >> > > > > > > > >> > > > > > >> class, a
>> >> > > > > > > > >> > > > > > >> > > new
>> >> > > > > > > > >> > > > > > >> > > > > > model
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > class, a
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > new
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> > converter
>> >> branch,
>> >> > > and a
>> >> > > > > > > schema
>> >> > > > > > > > >> > > > > migration*.
>> >> > > > > > > > >> > > > > > >> > > That's a
>> >> > > > > > > > >> > > > > > >> > > > > lot
>> >> > > > > > > > >> > > > > > >> > > > > > > of
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > surface
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> area
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> > for what
>> >> should
>> >> > be
>> >> > > "one
>> >> > > > > > > more
>> >> > > > > > > > >> kind
>> >> > > > > > > > >> > of
>> >> > > > > > > > >> > > > > > >> metric."
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> > *My bias*
>> >> would
>> >> > be
>> >> > > > > > toward a
>> >> > > > > > > > >> single
>> >> > > > > > > > >> > > > > metrics
>> >> > > > > > > > >> > > > > > >> > table
>> >> > > > > > > > >> > > > > > >> > > > with
>> >> > > > > > > > >> > > > > > >> > > > > > *a
>> >> > > > > > > > >> > > > > > >> > > > > > > > > typed
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > JSON
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> > payload*.
>> >> > Something
>> >> > > > > like:
>> >> > > > > > > > >> > > metric_type
>> >> > > > > > > > >> > > > > > >> (enum),
>> >> > > > > > > > >> > > > > > >> > > > > > entity_id,
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >
>> >> table_identifier,
>> >> > > > > > > snapshot_id
>> >> > > > > > > > >> > > > > (nullable),
>> >> > > > > > > > >> > > > > > >> > > > > received_ts,
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> schema_version,
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> and
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> > a payload
>> >> column
>> >> > > for
>> >> > > > > the
>> >> > > > > > > > >> > > metric-specific
>> >> > > > > > > > >> > > > > > >> data.
>> >> > > > > > > > >> > > > > > >> > > The
>> >> > > > > > > > >> > > > > > >> > > > > > > > > metric_type
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > +
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >
>> schema_version
>> >> > pair
>> >> > > > > gives
>> >> > > > > > > us a
>> >> > > > > > > > >> > > > > > >> > forward-compatible
>> >> > > > > > > > >> > > > > > >> > > > > > > contract
>> >> > > > > > > > >> > > > > > >> > > > > > > > > for
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > the
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> payload
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> > shape.
>> Adding
>> >> a
>> >> > new
>> >> > > > > > metric
>> >> > > > > > > type
>> >> > > > > > > > >> > > becomes
>> >> > > > > > > > >> > > > > an
>> >> > > > > > > > >> > > > > > >> enum
>> >> > > > > > > > >> > > > > > >> > > > value
>> >> > > > > > > > >> > > > > > >> > > > > > > and a
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > payload
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> schema,
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> > not a
>> schema
>> >> > > migration.
>> >> > > > > > One
>> >> > > > > > > > >> thing
>> >> > > > > > > > >> > I
>> >> > > > > > > > >> > > > > think
>> >> > > > > > > > >> > > > > > we
>> >> > > > > > > > >> > > > > > >> > need
>> >> > > > > > > > >> > > > > > >> > > > to
>> >> > > > > > > > >> > > > > > >> > > > > be
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > deliberate
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> about is
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> > the
>> partition
>> >> key
>> >> > > > > design.
>> >> > > > > > > If
>> >> > > > > > > > >> all
>> >> > > > > > > > >> > > metric
>> >> > > > > > > > >> > > > > > >> types
>> >> > > > > > > > >> > > > > > >> > > land
>> >> > > > > > > > >> > > > > > >> > > > in
>> >> > > > > > > > >> > > > > > >> > > > > > one
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > table,
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > scan
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> > metrics at
>> >> scale
>> >> > > (high
>> >> > > > > > > > >> > concurrency,
>> >> > > > > > > > >> > > high
>> >> > > > > > > > >> > > > > > >> > > frequency
>> >> > > > > > > > >> > > > > > >> > > > > > across
>> >> > > > > > > > >> > > > > > >> > > > > > > > > many
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> tables)
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> > could
>> easily
>> >> > > create hot
>> >> > > > > > > > >> > partitions.
>> >> > > > > > > > >> > > We'd
>> >> > > > > > > > >> > > > > > >> want
>> >> > > > > > > > >> > > > > > >> > the
>> >> > > > > > > > >> > > > > > >> > > > > > > > persistence
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > layer
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> to
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> be
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> > able to
>> shard
>> >> by
>> >> > > entity
>> >> > > > > > or
>> >> > > > > > > time
>> >> > > > > > > > >> > > range,
>> >> > > > > > > > >> > > > > and
>> >> > > > > > > > >> > > > > > >> that
>> >> > > > > > > > >> > > > > > >> > > > means
>> >> > > > > > > > >> > > > > > >> > > > > > the
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > logical
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> schema
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> > needs to
>> >> expose
>> >> > > enough
>> >> > > > > > > > >> structure
>> >> > > > > > > > >> > for
>> >> > > > > > > > >> > > > > > >> backends
>> >> > > > > > > > >> > > > > > >> > to
>> >> > > > > > > > >> > > > > > >> > > > > > > partition
>> >> > > > > > > > >> > > > > > >> > > > > > > > > on.
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > I
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> don't
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> > think the
>> >> current
>> >> > > > > > flattened
>> >> > > > > > > > >> layout
>> >> > > > > > > > >> > > gives
>> >> > > > > > > > >> > > > > > us
>> >> > > > > > > > >> > > > > > >> > that.
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> > This is
>> >> getting
>> >> > > complex
>> >> > > > > > > enough
>> >> > > > > > > > >> > that
>> >> > > > > > > > >> > > I
>> >> > > > > > > > >> > > > > > don't
>> >> > > > > > > > >> > > > > > >> > think
>> >> > > > > > > > >> > > > > > >> > > > > > ad-hoc
>> >> > > > > > > > >> > > > > > >> > > > > > > > > PR/ML
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> threads
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> > will
>> converge
>> >> > well.
>> >> > > > > > *Would
>> >> > > > > > > > >> people
>> >> > > > > > > > >> > be
>> >> > > > > > > > >> > > > > open
>> >> > > > > > > > >> > > > > > >> to a
>> >> > > > > > > > >> > > > > > >> > > > > biweekly
>> >> > > > > > > > >> > > > > > >> > > > > > > > sync
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > for
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> metrics
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >
>> >> architecture?* I
>> >> > > think
>> >> > > > > 30
>> >> > > > > > > > >> minutes
>> >> > > > > > > > >> > > every
>> >> > > > > > > > >> > > > > > two
>> >> > > > > > > > >> > > > > > >> > weeks
>> >> > > > > > > > >> > > > > > >> > > > > with
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > interested
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> parties
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> > would be
>> >> enough
>> >> > to
>> >> > > work
>> >> > > > > > > through
>> >> > > > > > > > >> > the
>> >> > > > > > > > >> > > > > > schema,
>> >> > > > > > > > >> > > > > > >> SPI
>> >> > > > > > > > >> > > > > > >> > > > > shape,
>> >> > > > > > > > >> > > > > > >> > > > > > > and
>> >> > > > > > > > >> > > > > > >> > > > > > > > > read
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > API
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> design
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> > together.
>> >> Happy
>> >> > to
>> >> > > help
>> >> > > > > > set
>> >> > > > > > > > >> that
>> >> > > > > > > > >> > up.
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> > -ej
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> > On Mon, Apr
>> >> 20,
>> >> > > 2026 at
>> >> > > > > > > > >> 2:19 PM EJ
>> >> > > > > > > > >> > > Wang
>> >> > > > > > > > >> > > > > <
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >>
>> >> > > > > > ej.wang.dev.interest@gmail.com
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> > wrote:
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >> Reviewed
>> >> #4115,
>> >> > > left a
>> >> > > > > > > > >> comment on
>> >> > > > > > > > >> > > the
>> >> > > > > > > > >> > > > > > code
>> >> > > > > > > > >> > > > > > >> > > > > > organization
>> >> > > > > > > > >> > > > > > >> > > > > > > > > side.
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >> One thing
>> >> stood
>> >> > > out:
>> >> > > > > the
>> >> > > > > > > > >> metrics
>> >> > > > > > > > >> > > write
>> >> > > > > > > > >> > > > > > path
>> >> > > > > > > > >> > > > > > >> > > enters
>> >> > > > > > > > >> > > > > > >> > > > > > > through
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>
>> >> > > PolarisMetricsManager
>> >> > > > > on
>> >> > > > > > > > >> > > > > > MetaStoreManager,
>> >> > > > > > > > >> > > > > > >> but
>> >> > > > > > > > >> > > > > > >> > > the
>> >> > > > > > > > >> > > > > > >> > > > > new
>> >> > > > > > > > >> > > > > > >> > > > > > > > read
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > path
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> bypasses
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>
>> >> MetaStoreManager
>> >> > > > > > entirely
>> >> > > > > > > and
>> >> > > > > > > > >> > goes
>> >> > > > > > > > >> > > > > > >> straight to
>> >> > > > > > > > >> > > > > > >> > > > > > > > > BasePersistence
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > via
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>
>> >> > > > > > > callContext.getMetaStore().
>> >> > > > > > > > >> That
>> >> > > > > > > > >> > > means
>> >> > > > > > > > >> > > > > > the
>> >> > > > > > > > >> > > > > > >> > read
>> >> > > > > > > > >> > > > > > >> > > > API
>> >> > > > > > > > >> > > > > > >> > > > > > only
>> >> > > > > > > > >> > > > > > >> > > > > > > > > works
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > for
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> backends
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >> that
>> >> implement
>> >> > > > > > > > >> BasePersistence.
>> >> > > > > > > > >> > > NoSQL
>> >> > > > > > > > >> > > > > and
>> >> > > > > > > > >> > > > > > >> > remote
>> >> > > > > > > > >> > > > > > >> > > > > > > backends
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > can't
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> participate.
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >> Stepping
>> >> back, I
>> >> > > think
>> >> > > > > > the
>> >> > > > > > > > >> > metrics
>> >> > > > > > > > >> > > > > > >> subsystem
>> >> > > > > > > > >> > > > > > >> > is
>> >> > > > > > > > >> > > > > > >> > > > > > growing
>> >> > > > > > > > >> > > > > > >> > > > > > > > into
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> something
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >> real
>> (write +
>> >> > > read +
>> >> > > > > > REST
>> >> > > > > > > API
>> >> > > > > > > > >> +
>> >> > > > > > > > >> > > AuthZ +
>> >> > > > > > > > >> > > > > > >> > > > pagination)
>> >> > > > > > > > >> > > > > > >> > > > > > *but
>> >> > > > > > > > >> > > > > > >> > > > > > > > the
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> persistence
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >> side is
>> split
>> >> > > across
>> >> > > > > two
>> >> > > > > > > > >> layers
>> >> > > > > > > > >> > in
>> >> > > > > > > > >> > > a
>> >> > > > > > > > >> > > > > way
>> >> > > > > > > > >> > > > > > >> > that's
>> >> > > > > > > > >> > > > > > >> > > > hard
>> >> > > > > > > > >> > > > > > >> > > > > > to
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > extend*. I
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> put
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >> together
>> two
>> >> > > diagrams
>> >> > > > > to
>> >> > > > > > > show
>> >> > > > > > > > >> > what
>> >> > > > > > > > >> > > I
>> >> > > > > > > > >> > > > > mean
>> >> > > > > > > > >> > > > > > >> (my
>> >> > > > > > > > >> > > > > > >> > > best
>> >> > > > > > > > >> > > > > > >> > > > > > > > effort).
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >> *Current
>> >> state*
>> >> > > > > (Diagram
>> >> > > > > > > 1):
>> >> > > > > > > > >> > three
>> >> > > > > > > > >> > > > > > >> interfaces
>> >> > > > > > > > >> > > > > > >> > at
>> >> > > > > > > > >> > > > > > >> > > > > three
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > different
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> levels.
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >> The
>> >> > engine-facing
>> >> > > SPI
>> >> > > > > > > > >> > > > > > >> (PolarisMetricsReporter)
>> >> > > > > > > > >> > > > > > >> > > is
>> >> > > > > > > > >> > > > > > >> > > > > > clean.
>> >> > > > > > > > >> > > > > > >> > > > > > > > But
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>
>> >> > > PolarisMetricsManager
>> >> > > > > on
>> >> > > > > > > > >> > > > > MetaStoreManager
>> >> > > > > > > > >> > > > > > >> is a
>> >> > > > > > > > >> > > > > > >> > > > > > > passthrough
>> >> > > > > > > > >> > > > > > >> > > > > > > > > to
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>
>> >> > > MetricsPersistence on
>> >> > > > > > > > >> > > BasePersistence.
>> >> > > > > > > > >> > > > > > The
>> >> > > > > > > > >> > > > > > >> > @Beta
>> >> > > > > > > > >> > > > > > >> > > > > > > > annotation
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > and
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > SPI
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> javadoc
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >> are on the
>> >> > > > > > BasePersistence
>> >> > > > > > > > >> layer,
>> >> > > > > > > > >> > > while
>> >> > > > > > > > >> > > > > > the
>> >> > > > > > > > >> > > > > > >> > > actual
>> >> > > > > > > > >> > > > > > >> > > > > > > > extension
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > points
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>
>> >> > > > > (PolarisMetricsReporter,
>> >> > > > > > > > >> > > > > > >> > PolarisMetricsManager)
>> >> > > > > > > > >> > > > > > >> > > > > carry
>> >> > > > > > > > >> > > > > > >> > > > > > no
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > stability
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>
>> annotation.
>> >> The
>> >> > > write
>> >> > > > > > path
>> >> > > > > > > > >> goes
>> >> > > > > > > > >> > > through
>> >> > > > > > > > >> > > > > > the
>> >> > > > > > > > >> > > > > > >> > > > > > > > MetaStoreManager
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > layer,
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> the
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >> read path
>> >> > doesn't.
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >> *What I
>> >> > envision*
>> >> > > > > > > (Diagram 2):
>> >> > > > > > > > >> > two
>> >> > > > > > > > >> > > SPIs
>> >> > > > > > > > >> > > > > > at
>> >> > > > > > > > >> > > > > > >> two
>> >> > > > > > > > >> > > > > > >> > > > > levels.
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>
>> >> > > PolarisMetricsReporter
>> >> > > > > > > stays
>> >> > > > > > > > >> as
>> >> > > > > > > > >> > the
>> >> > > > > > > > >> > > > > > >> > > engine-facing
>> >> > > > > > > > >> > > > > > >> > > > > SPI.
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>
>> >> > > PolarisMetricsManager
>> >> > > > > > > becomes
>> >> > > > > > > > >> the
>> >> > > > > > > > >> > > > > > >> > backend-facing
>> >> > > > > > > > >> > > > > > >> > > > SPI
>> >> > > > > > > > >> > > > > > >> > > > > > > with
>> >> > > > > > > > >> > > > > > >> > > > > > > > > both
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > write
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> and
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >> read
>> methods
>> >> at
>> >> > > the
>> >> > > > > > > > >> > > MetaStoreManager
>> >> > > > > > > > >> > > > > > level,
>> >> > > > > > > > >> > > > > > >> > > where
>> >> > > > > > > > >> > > > > > >> > > > > any
>> >> > > > > > > > >> > > > > > >> > > > > > > > > backend
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > (JDBC,
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> NoSQL,
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >> remote)
>> can
>> >> > > implement
>> >> > > > > > > them.
>> >> > > > > > > > >> > > > > > >> MetricsPersistence
>> >> > > > > > > > >> > > > > > >> > > on
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > BasePersistence
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> goes
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >> away.
>> Where
>> >> > > metrics
>> >> > > > > > > actually
>> >> > > > > > > > >> land
>> >> > > > > > > > >> > > is an
>> >> > > > > > > > >> > > > > > >> > > > > implementation
>> >> > > > > > > > >> > > > > > >> > > > > > > > > detail,
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > not a
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> core
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >> interface.
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >> *Minor
>> naming
>> >> > > thing*:
>> >> > > > > > > > >> > > > > > >> PolarisMetricsReporter
>> >> > > > > > > > >> > > > > > >> > is
>> >> > > > > > > > >> > > > > > >> > > > > > broader
>> >> > > > > > > > >> > > > > > >> > > > > > > > than
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > what
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > it
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >> actually
>> >> > handles.
>> >> > > It
>> >> > > > > > only
>> >> > > > > > > > >> accepts
>> >> > > > > > > > >> > > > > Iceberg
>> >> > > > > > > > >> > > > > > >> REST
>> >> > > > > > > > >> > > > > > >> > > > > Catalog
>> >> > > > > > > > >> > > > > > >> > > > > > > > > metrics
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> (ScanReport,
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>
>> CommitReport
>> >> via
>> >> > > > > > > > >> MetricsReport).
>> >> > > > > > > > >> > > > > Generic
>> >> > > > > > > > >> > > > > > >> table
>> >> > > > > > > > >> > > > > > >> > > > > metrics
>> >> > > > > > > > >> > > > > > >> > > > > > > or
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> operational
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >> metrics
>> >> aren't
>> >> > in
>> >> > > > > scope.
>> >> > > > > > > Not
>> >> > > > > > > > >> > > blocking,
>> >> > > > > > > > >> > > > > > but
>> >> > > > > > > > >> > > > > > >> > worth
>> >> > > > > > > > >> > > > > > >> > > > > > noting
>> >> > > > > > > > >> > > > > > >> > > > > > > if
>> >> > > > > > > > >> > > > > > >> > > > > > > > > the
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> metrics
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >> surface
>> >> expands.
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >> *Rough
>> >> sketch of
>> >> > > how
>> >> > > > > to
>> >> > > > > > > get
>> >> > > > > > > > >> > there*:
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>  1.  Add
>> read
>> >> > > methods
>> >> > > > > to
>> >> > > > > > > > >> > > > > > >> PolarisMetricsManager
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > (listScanReports,
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>
>> >> > listCommitReports)
>> >> > > > > with
>> >> > > > > > > > >> default
>> >> > > > > > > > >> > > no-op,
>> >> > > > > > > > >> > > > > > >> same as
>> >> > > > > > > > >> > > > > > >> > > the
>> >> > > > > > > > >> > > > > > >> > > > > > > > existing
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > write
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> methods.
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >> (Probably
>> >> make
>> >> > > > > > > > >> > > PolarisMetricsManager
>> >> > > > > > > > >> > > > > more
>> >> > > > > > > > >> > > > > > >> > > explicit
>> >> > > > > > > > >> > > > > > >> > > > > on
>> >> > > > > > > > >> > > > > > >> > > > > > > > being
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > Iceberg
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >> specific
>> like
>> >> > > package
>> >> > > > > > > name or
>> >> > > > > > > > >> > class
>> >> > > > > > > > >> > > > > name
>> >> > > > > > > > >> > > > > > >> etc.)
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>  2.  Wire
>> >> > > > > > > > >> MetricsReportsService
>> >> > > > > > > > >> > > through
>> >> > > > > > > > >> > > > > > >> > > > > > MetaStoreManager
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > instead
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > of
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>
>> >> > > > > > > callContext.getMetaStore().
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>  3.
>> Extract
>> >> > > metrics
>> >> > > > > > > > >> persistence
>> >> > > > > > > > >> > > from
>> >> > > > > > > > >> > > > > > >> > > > > > > > > JdbcBasePersistenceImpl
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > into
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> its
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >> own class.
>> >> That
>> >> > > file
>> >> > > > > > > carries
>> >> > > > > > > > >> ~7
>> >> > > > > > > > >> > > > > > >> > > responsibilities,
>> >> > > > > > > > >> > > > > > >> > > > > > > metrics
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > being
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > one
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> of
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> them.
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>  4.
>> Remove
>> >> > > > > > > MetricsPersistence
>> >> > > > > > > > >> > from
>> >> > > > > > > > >> > > > > > >> > > > BasePersistence.
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >> *None of
>> this
>> >> > > needs to
>> >> > > > > > > happen
>> >> > > > > > > > >> in
>> >> > > > > > > > >> > > #4115.
>> >> > > > > > > > >> > > > > > >> But if
>> >> > > > > > > > >> > > > > > >> > > the
>> >> > > > > > > > >> > > > > > >> > > > > > > > direction
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > makes
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> sense,
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >> it would
>> be
>> >> good
>> >> > > to
>> >> > > > > > align
>> >> > > > > > > > >> before
>> >> > > > > > > > >> > > the
>> >> > > > > > > > >> > > > > > >> metrics
>> >> > > > > > > > >> > > > > > >> > > > surface
>> >> > > > > > > > >> > > > > > >> > > > > > > grows
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > further.
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> Curious
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >> what
>> others
>> >> > > think.*
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >> *My mental
>> >> model
>> >> > > > > note*:
>> >> > > > > > > Level
>> >> > > > > > > > >> 1
>> >> > > > > > > > >> > > > > > >> > > MetaStoreManager;
>> >> > > > > > > > >> > > > > > >> > > > > > level
>> >> > > > > > > > >> > > > > > >> > > > > > > 2
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> transactional
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>
>> persistence;
>> >> > > level 3
>> >> > > > > > base
>> >> > > > > > > > >> > > persistence
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >> Diagram 1
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >> <
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >>
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >>
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > >
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > >
>> >> > > > > > > > >> > > > > > >> > > > > > > > > >
>> >> > > > > > > > >> > > > > > >> > > > > > > > >
>> >> > > > > > > > >> > > > > > >> > > > > > > >
>> >> > > > > > > > >> > > > > > >> > > > > > >
>> >> > > > > > > > >> > > > > > >> > > > > >
>> >> > > > > > > > >> > > > > > >> > > > >
>> >> > > > > > > > >> > > > > > >> > > >
>> >> > > > > > > > >> > > > > > >> > >
>> >> > > > > > > > >> > > > > > >> >
>> >> > > > > > > > >> > > > > > >>
>> >> > > > > > > > >> > > > > >
>> >> > > > > > > > >> > > > >
>> >> > > > > > > > >> > >
>> >> > > > > > > > >> >
>> >> > > > > > > > >>
>> >> > > > > > >
>> >> > > > > >
>> >> > > > >
>> >> > >
>> >> >
>> >>
>> https://urldefense.com/v3/__https://www.plantuml.com/plantuml/uml/bLHDR-Cs4BthLmpIYupw0zbkKQ1r3M-S7Bp8xhhM7WCOb3IM65EaGD9EX2RzxHrHb4CxRelwa4YSDu_lpOVcnZ9jzvM8BBS2uGjQpJC3dtHMSekPtMk44IpsMgEqa5XcCOhCZikQQLP1pR8TAp2n3ILhmZDP20m0fcIvUkAoW2qJXd9z1bpToO9BX3WXu0ucy5rpgGPNm0nW5_epUWtm2Ue3pn3kMOFQmKntGZW0BYtgBSi8k5A2QMwybJNMIbFiGSR9QZc4nUqIvikStF0jHprua5C-amge42aNt3R0f5JaaoivdV2Pkqbx4hee4ymOkBh5BTiB-_uIeGeo8zL8rPsPl4DktdEiK1jkB1NdZCRbrSTecDe_mlHbF0wvBmCkaOH5_S8a_TTTKI6-nmCAkEw4LpxsZ-LbYLKQFKMNOgf_wuM7_bV9gOer5SYMMksBSWXFcbi49KNZXNLicwfe3TETC7gPdPqI7uBcHMb1RSzYq34c6PDUM9mn8HRsUTZEiDBve3NjVZumBj0U7SS37mGO7vcwtiK-_pU7U7L_f-digo9YbhSwIfMRwIITKGXbxdIUTCGF1SeCJxloKsU-3k9ddRbX1eDq1q_fx1JbBGT0glVyXimDuP4TQ5qpCAmnGEj2s_6n5mtn1z-97-63itFQZLPO1Ev2tu_WF7Ju-VPc0Skg5bYXxBhkY1xpD7EM_7fyflSpIsqMgVth5xhVr4eQxWQ8enaSAJQSG16yFSDuJ798rrcXr_3n-lfdk7icQjEBmFujL7AodiP_Y4Z7-YxvtZNs4zMgpNTl6tF8sglyPsmqchrjvQ-m-aP94r-TwCA2Ka8upPJZwtvSpoYCXkYMZU2NXvRMBfq9P3i3Le4VAZUAlUZ_oPKsxPgY0Q_BSKLkyr9bhQhQrJjo_x3TPlIB0DPjnMfcIoYP0QaYw1a0fTKDr8fB6ntNuvmoL1ZGkXa69Njh43zf9GiGxHQrA_jDYWRSzF5--WmTVrN97_Sm8LbLUy_lGBmLanJjFkDlGkRqjA_4tm00__;!!Iz9xO38YGHZK!4rnlVIqBQQQ6xpGe1PMV58edOS_cGrKoqdfhw4qSp_JQCavvEjaxXk0OwGPnxyEcO60AzKQK_XDtp7H3DdkFXcrXuTBISHg$
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >> :
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >> [image:
>> >> > image.png]
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >> Diagram 2
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >> <
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >>
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >>
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > >
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > >
>> >> > > > > > > > >> > > > > > >> > > > > > > > > >
>> >> > > > > > > > >> > > > > > >> > > > > > > > >
>> >> > > > > > > > >> > > > > > >> > > > > > > >
>> >> > > > > > > > >> > > > > > >> > > > > > >
>> >> > > > > > > > >> > > > > > >> > > > > >
>> >> > > > > > > > >> > > > > > >> > > > >
>> >> > > > > > > > >> > > > > > >> > > >
>> >> > > > > > > > >> > > > > > >> > >
>> >> > > > > > > > >> > > > > > >> >
>> >> > > > > > > > >> > > > > > >>
>> >> > > > > > > > >> > > > > >
>> >> > > > > > > > >> > > > >
>> >> > > > > > > > >> > >
>> >> > > > > > > > >> >
>> >> > > > > > > > >>
>> >> > > > > > >
>> >> > > > > >
>> >> > > > >
>> >> > >
>> >> >
>> >>
>> https://urldefense.com/v3/__https://www.plantuml.com/plantuml/uml/VLLDR-8m4BtdLupO2sWBLVU8AaGB7AXAbssGzb896SS9RXqxjKqBwkv_tt7iV43fdaZYDpFlpRmnOsE9jhjSH9PRmM31hERKm8scMsuPjJlDe0yheZDc8RR4iYWoBrmMH9CS2a9VICPYUy1OZN0YCy5Q0BCbYNhdCeEK28En8G8wCvbnoQ0R8_05Bc6bkLIz3X03p1zzH7zR-9ZfDquPt9C3qoNCX2yV4G2NbkcKu5jdgGJHt0GbZwnG6i-UP3TUpk5gM6Ldqke350eZUqzoCft3U9xWHvxoa5-7K4nF1J46EbEMafsmdrCBbQ44gVggy18IZrn_ph5asd1ZiIKdQSgueZvjXrQFSFrdC3YN-nXmBacxbGiYyLVxLaBtdhqn0LSzdBDhqQtQoOJeGyad3z0lUqnYgpGB6Ns8oVyta00Dy_WnX0tIOZ8v6SYxHll1TrH6aejAik-mh-AphVFCwSUQqFypElag5QRGFDjQKEd96K1P8QP41c9TzA_IIQyvdAWyv_RSiS3skb0_EzDDkK2v5xWF6MiGFlvhpFLcD2Dq2pml14gaF67eQkmd8gulDoC4kSOu6KVpkvlUJg1RTbWISU40RdBUUS_9XfRZ2dwxm_SW8LYFISgm_MnlDQ6M9P1gbKEc4X-2pH_FvJCkCqm9pbVjD6LrwdLeOrDWfOaqc8Wh9BE85oNKxkNQ6o4yGRy_Eae0G_G8tZv81d3bHDB23WOdisohVr3nh_j6lbSjbNaLRTc8UgtPbAU1J_tygOfZX9DWEJeHDvYx-qmSi5FgNLPZwHrHcUsncGQ5-skhUclpE5fo4ounpFauYrUbkU6ccfnxMvitwag4IyerhTxj8In_Oj1bDO4pQru674loYrGlULHLEGCjwJJ8gDoVZR8MxO4BT3IzRvIcAQKezC6xpziGnTyImrfEGyJI_OcKfgtxIvnTqFEMS17L9Z-jsARN5FmTheP7HtSdtOMT0B4GY2FYHXxgQmMtj2bRqiLFGapiVe1_QVKDrkqXcm83aFEXnMYCZ-xlyHy__;!!Iz9xO38YGHZK!4rnlVIqBQQQ6xpGe1PMV58edOS_cGrKoqdfhw4qSp_JQCavvEjaxXk0OwGPnxyEcO60AzKQK_XDtp7H3DdkFXcrXSpmqcNk$
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >> :
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >> [image:
>> >> > image.png]
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>  -ej
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >> On Wed,
>> Apr
>> >> 15,
>> >> > > 2026
>> >> > > > > at
>> >> > > > > > > > >> 8:22 AM
>> >> > > > > > > > >> > > Dmitri
>> >> > > > > > > > >> > > > > > >> > > > Bourlatchkov
>> >> > > > > > > > >> > > > > > >> > > > > <
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >>
>> dimas@apache.org>
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >> wrote:
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> Hi All,
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>>
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> Heads up:
>> >> The
>> >> > > current
>> >> > > > > > > state
>> >> > > > > > > > >> of
>> >> > > > > > > > >> > PR
>> >> > > > > > > > >> > > > > [4115]
>> >> > > > > > > > >> > > > > > >> > looks
>> >> > > > > > > > >> > > > > > >> > > > > pretty
>> >> > > > > > > > >> > > > > > >> > > > > > > > solid
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > to
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > me.
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> I
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> believe
>> >> this PR
>> >> > > is
>> >> > > > > > > > >> approaching a
>> >> > > > > > > > >> > > > > > mergeable
>> >> > > > > > > > >> > > > > > >> > > > > condition.
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>>
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> Please
>> post
>> >> > your
>> >> > > > > > reviews
>> >> > > > > > > if
>> >> > > > > > > > >> you
>> >> > > > > > > > >> > > have
>> >> > > > > > > > >> > > > > any
>> >> > > > > > > > >> > > > > > >> > > > comments.
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>>
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> [4115]
>> >> > > > > > > > >> > > > > > >> > > >
>> >> > > > > > > > >> >
>> >> > > > > > > > >>
>> >> > > > > > >
>> >> > > > > >
>> >> > > > >
>> >> > >
>> >> >
>> >>
>> https://urldefense.com/v3/__https://github.com/apache/polaris/pull/4115__;!!Iz9xO38YGHZK!4rnlVIqBQQQ6xpGe1PMV58edOS_cGrKoqdfhw4qSp_JQCavvEjaxXk0OwGPnxyEcO60AzKQK_XDtp7H3DdkFXcrXtFRscKs$
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>>
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> Thanks,
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> Dmitri.
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>>
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> On Tue,
>> Mar
>> >> 3,
>> >> > > 2026
>> >> > > > > at
>> >> > > > > > > > >> 3:29 PM
>> >> > > > > > > > >> > > Anand
>> >> > > > > > > > >> > > > > > Kumar
>> >> > > > > > > > >> > > > > > >> > > > Sankaran
>> >> > > > > > > > >> > > > > > >> > > > > > via
>> >> > > > > > > > >> > > > > > >> > > > > > > > > dev <
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>>
>> >> > > > > dev@polaris.apache.org
>> >> > > > > > >
>> >> > > > > > > > >> wrote:
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>>
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > Hi
>> Yufei
>> >> and
>> >> > > > > Dmitri,
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> >
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > Here
>> is a
>> >> > > proposal
>> >> > > > > > for
>> >> > > > > > > the
>> >> > > > > > > > >> > REST
>> >> > > > > > > > >> > > > > > >> endpoints
>> >> > > > > > > > >> > > > > > >> > for
>> >> > > > > > > > >> > > > > > >> > > > > > metrics
>> >> > > > > > > > >> > > > > > >> > > > > > > > and
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > events.
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> >
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> >
>> >> > > > > > > > >> > > > > > >> > > > > >
>> >> > > > > > > > >> >
>> >> > > > > > > > >>
>> >> > > > > > >
>> >> > > > > >
>> >> > > > >
>> >> > >
>> >> >
>> >>
>> https://urldefense.com/v3/__https://github.com/apache/polaris/pull/3924/changes__;!!Iz9xO38YGHZK!4rnlVIqBQQQ6xpGe1PMV58edOS_cGrKoqdfhw4qSp_JQCavvEjaxXk0OwGPnxyEcO60AzKQK_XDtp7H3DdkFXcrXcjoPhSs$
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> >
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > I did
>> not
>> >> see
>> >> > > any
>> >> > > > > > > > >> precursors
>> >> > > > > > > > >> > for
>> >> > > > > > > > >> > > > > > >> raising a
>> >> > > > > > > > >> > > > > > >> > PR
>> >> > > > > > > > >> > > > > > >> > > > for
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > proposals,
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > so
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> trying
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > this.
>> >> Please
>> >> > > let
>> >> > > > > me
>> >> > > > > > > know
>> >> > > > > > > > >> what
>> >> > > > > > > > >> > > you
>> >> > > > > > > > >> > > > > > >> think.
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> >
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > -
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > Anand
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> >
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > From:
>> >> Anand
>> >> > > Kumar
>> >> > > > > > > Sankaran
>> >> > > > > > > > >> <
>> >> > > > > > > > >> > > > > > >> > > > > > > anand.sankaran@workday.com
>> >> > > > > > > > >> > > > > > >> > > > > > > > >
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > Date:
>> >> Monday,
>> >> > > March
>> >> > > > > > 2,
>> >> > > > > > > > >> 2026 at
>> >> > > > > > > > >> > > > > > 10:25 AM
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > To:
>> >> > > > > > > dev@polaris.apache.org
>> >> > > > > > > > >> <
>> >> > > > > > > > >> > > > > > >> > > > > dev@polaris.apache.org
>> >> > > > > > > > >> > > > > > >> > > > > > >
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> >
>> Subject:
>> >> Re:
>> >> > > > > Polaris
>> >> > > > > > > > >> Telemetry
>> >> > > > > > > > >> > > and
>> >> > > > > > > > >> > > > > > Audit
>> >> > > > > > > > >> > > > > > >> > > Trail
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> >
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > About
>> the
>> >> > REST
>> >> > > API,
>> >> > > > > > > based
>> >> > > > > > > > >> on
>> >> > > > > > > > >> > my
>> >> > > > > > > > >> > > use
>> >> > > > > > > > >> > > > > > >> cases:
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> >
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> >
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> >   1.
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > I want
>> to
>> >> be
>> >> > > able
>> >> > > > > to
>> >> > > > > > > query
>> >> > > > > > > > >> > > commit
>> >> > > > > > > > >> > > > > > >> metrics
>> >> > > > > > > > >> > > > > > >> > to
>> >> > > > > > > > >> > > > > > >> > > > > track
>> >> > > > > > > > >> > > > > > >> > > > > > > > files
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > added
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > /
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> removed
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > per
>> >> commit,
>> >> > > along
>> >> > > > > > with
>> >> > > > > > > > >> record
>> >> > > > > > > > >> > > > > counts.
>> >> > > > > > > > >> > > > > > >> The
>> >> > > > > > > > >> > > > > > >> > > > > ingestion
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > pipeline
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > that
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> writes
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > this
>> data
>> >> is
>> >> > > owned
>> >> > > > > by
>> >> > > > > > > us
>> >> > > > > > > > >> and
>> >> > > > > > > > >> > we
>> >> > > > > > > > >> > > are
>> >> > > > > > > > >> > > > > > >> > > guaranteed
>> >> > > > > > > > >> > > > > > >> > > > to
>> >> > > > > > > > >> > > > > > >> > > > > > > write
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > this
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>>
>> information
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > for
>> each
>> >> > write.
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> >   2.
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > I want
>> to
>> >> be
>> >> > > able
>> >> > > > > to
>> >> > > > > > > query
>> >> > > > > > > > >> > scan
>> >> > > > > > > > >> > > > > > metrics
>> >> > > > > > > > >> > > > > > >> for
>> >> > > > > > > > >> > > > > > >> > > > > read. I
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > understand
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> clients
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> do
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > not
>> >> fulfill
>> >> > > this
>> >> > > > > > > > >> requirement.
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> >   3.
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > I want
>> to
>> >> be
>> >> > > able
>> >> > > > > to
>> >> > > > > > > query
>> >> > > > > > > > >> the
>> >> > > > > > > > >> > > > > events
>> >> > > > > > > > >> > > > > > >> table
>> >> > > > > > > > >> > > > > > >> > > > > (events
>> >> > > > > > > > >> > > > > > >> > > > > > > are
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> persisted) -
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> this
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > may
>> >> supersede
>> >> > > #2, I
>> >> > > > > > am
>> >> > > > > > > not
>> >> > > > > > > > >> > sure
>> >> > > > > > > > >> > > yet.
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> >
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > All
>> this
>> >> > > > > information
>> >> > > > > > > is in
>> >> > > > > > > > >> the
>> >> > > > > > > > >> > > JDBC
>> >> > > > > > > > >> > > > > > >> based
>> >> > > > > > > > >> > > > > > >> > > > > > persistence
>> >> > > > > > > > >> > > > > > >> > > > > > > > > model
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > and
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> is
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> >
>> persisted
>> >> in
>> >> > > the
>> >> > > > > > > > >> metastore. I
>> >> > > > > > > > >> > > > > > currently
>> >> > > > > > > > >> > > > > > >> > don’t
>> >> > > > > > > > >> > > > > > >> > > > > have
>> >> > > > > > > > >> > > > > > >> > > > > > a
>> >> > > > > > > > >> > > > > > >> > > > > > > > need
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > to
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> query
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> >
>> >> prometheus or
>> >> > > open
>> >> > > > > > > > >> telemetry.
>> >> > > > > > > > >> > I
>> >> > > > > > > > >> > > do
>> >> > > > > > > > >> > > > > > >> publish
>> >> > > > > > > > >> > > > > > >> > > some
>> >> > > > > > > > >> > > > > > >> > > > > > > events
>> >> > > > > > > > >> > > > > > >> > > > > > > > to
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> Prometheus
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> and
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > they
>> are
>> >> > > forwarded
>> >> > > > > to
>> >> > > > > > > our
>> >> > > > > > > > >> > > dashboards
>> >> > > > > > > > >> > > > > > >> > > elsewhere.
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> >
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > About
>> the
>> >> CLI
>> >> > > > > > > utilities, I
>> >> > > > > > > > >> > > meant the
>> >> > > > > > > > >> > > > > > >> admin
>> >> > > > > > > > >> > > > > > >> > > user
>> >> > > > > > > > >> > > > > > >> > > > > > > > > utilities.
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > In
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> one of
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> the
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> >
>> earliest
>> >> > > drafts of
>> >> > > > > my
>> >> > > > > > > > >> > proposal,
>> >> > > > > > > > >> > > > > > Prashant
>> >> > > > > > > > >> > > > > > >> > > > > mentioned
>> >> > > > > > > > >> > > > > > >> > > > > > > that
>> >> > > > > > > > >> > > > > > >> > > > > > > > > the
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> metrics
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> tables
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > can
>> grow
>> >> > > > > indefinitely
>> >> > > > > > > and
>> >> > > > > > > > >> > that a
>> >> > > > > > > > >> > > > > > similar
>> >> > > > > > > > >> > > > > > >> > > > problem
>> >> > > > > > > > >> > > > > > >> > > > > > > exists
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > with
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > the
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> events
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > table
>> as
>> >> > well.
>> >> > > We
>> >> > > > > > > discussed
>> >> > > > > > > > >> > that
>> >> > > > > > > > >> > > > > > >> cleaning
>> >> > > > > > > > >> > > > > > >> > up
>> >> > > > > > > > >> > > > > > >> > > of
>> >> > > > > > > > >> > > > > > >> > > > > old
>> >> > > > > > > > >> > > > > > >> > > > > > > > > records
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > from
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> both
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > metrics
>> >> > tables
>> >> > > and
>> >> > > > > > > events
>> >> > > > > > > > >> > > tables can
>> >> > > > > > > > >> > > > > > be
>> >> > > > > > > > >> > > > > > >> > done
>> >> > > > > > > > >> > > > > > >> > > > via
>> >> > > > > > > > >> > > > > > >> > > > > a
>> >> > > > > > > > >> > > > > > >> > > > > > > CLI
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > utility.
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> >
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > I see
>> that
>> >> > > Yufei
>> >> > > > > has
>> >> > > > > > > > >> covered
>> >> > > > > > > > >> > the
>> >> > > > > > > > >> > > > > > >> discussion
>> >> > > > > > > > >> > > > > > >> > > > about
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > datasources.
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> >
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > -
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > Anand
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> >
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> >
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> >
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > From:
>> >> Yufei
>> >> > Gu
>> >> > > <
>> >> > > > > > > > >> > > > > flyrain000@gmail.com>
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > Date:
>> >> Friday,
>> >> > > > > > February
>> >> > > > > > > 27,
>> >> > > > > > > > >> > 2026
>> >> > > > > > > > >> > > at
>> >> > > > > > > > >> > > > > > >> 9:54 PM
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > To:
>> >> > > > > > > dev@polaris.apache.org
>> >> > > > > > > > >> <
>> >> > > > > > > > >> > > > > > >> > > > > dev@polaris.apache.org
>> >> > > > > > > > >> > > > > > >> > > > > > >
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> >
>> Subject:
>> >> Re:
>> >> > > > > Polaris
>> >> > > > > > > > >> Telemetry
>> >> > > > > > > > >> > > and
>> >> > > > > > > > >> > > > > > Audit
>> >> > > > > > > > >> > > > > > >> > > Trail
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> >
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > This
>> >> Message
>> >> > Is
>> >> > > > > From
>> >> > > > > > an
>> >> > > > > > > > >> > External
>> >> > > > > > > > >> > > > > > Sender
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > This
>> >> message
>> >> > > came
>> >> > > > > > from
>> >> > > > > > > > >> outside
>> >> > > > > > > > >> > > your
>> >> > > > > > > > >> > > > > > >> > > > organization.
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > Report
>> >> > > Suspicious<
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> >
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>>
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >>
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >>
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > >
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > >
>> >> > > > > > > > >> > > > > > >> > > > > > > > > >
>> >> > > > > > > > >> > > > > > >> > > > > > > > >
>> >> > > > > > > > >> > > > > > >> > > > > > > >
>> >> > > > > > > > >> > > > > > >> > > > > > >
>> >> > > > > > > > >> > > > > > >> > > > > >
>> >> > > > > > > > >> > > > > > >> > > > >
>> >> > > > > > > > >> > > > > > >> > > >
>> >> > > > > > > > >> > > > > > >> > >
>> >> > > > > > > > >> > > > > > >> >
>> >> > > > > > > > >> > > > > > >>
>> >> > > > > > > > >> > > > > >
>> >> > > > > > > > >> > > > >
>> >> > > > > > > > >> > >
>> >> > > > > > > > >> >
>> >> > > > > > > > >>
>> >> > > > > > >
>> >> > > > > >
>> >> > > > >
>> >> > >
>> >> >
>> >>
>> https://us-phishalarm-ewt.proofpoint.com/EWT/v1/Iz9xO38YGHZK!YhNDZABkHi1B699ote2uMwpOZw8i0QMCGO2Szc-HshuABGhGvwPJcymE6G2oUUxtS8xDkSrtGTPm_I3QnVDHoLMk50m9v8z_nZKTkd-bnVUbreF1u0WnfV_X5eYevZl_$
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > >
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> >
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> >
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > As I
>> >> > mentioned
>> >> > > in
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> >
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>>
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >>
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >>
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > >
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > >
>> >> > > > > > > > >> > > > > > >> > > > > > > > > >
>> >> > > > > > > > >> > > > > > >> > > > > > > > >
>> >> > > > > > > > >> > > > > > >> > > > > > > >
>> >> > > > > > > > >> > > > > > >> > > > > > >
>> >> > > > > > > > >> > > > > > >> > > > > >
>> >> > > > > > > > >> > > > > > >> > > > >
>> >> > > > > > > > >> > > > > > >> > > >
>> >> > > > > > > > >> > > > > > >> > >
>> >> > > > > > > > >> > > > > > >> >
>> >> > > > > > > > >> > > > > > >>
>> >> > > > > > > > >> > > > > >
>> >> > > > > > > > >> > > > >
>> >> > > > > > > > >> > >
>> >> > > > > > > > >> >
>> >> > > > > > > > >>
>> >> > > > > > >
>> >> > > > > >
>> >> > > > >
>> >> > >
>> >> >
>> >>
>> https://urldefense.com/v3/__https://github.com/apache/polaris/issues/3890__;!!Iz9xO38YGHZK!5EuyFFkk3vhRWVIRvQAWBSQfpJkTMA9HxugzDwXmN0LPPqhEFxYkFRGVhtb8AqUwXtDh2OplcMnbMDHKOxrvDU0$
>> >> > > > > > > > >> > >> > > >> > > > > > > > > > > > >> >> >>> ,
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> >
>> supporting
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> >
>> multiple
>> >> data
>> >> > > > > sources
>> >> > > > > > > is
>> >> > > > > > > > >> not a
>> >> > > > > > > > >> > > > > trivial
>> >> > > > > > > > >> > > > > > >> > > change.
>> >> > > > > > > > >> > > > > > >> > > > I
>> >> > > > > > > > >> > > > > > >> > > > > > > would
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > strongly
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> recommend
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> >
>> starting
>> >> > with a
>> >> > > > > > design
>> >> > > > > > > > >> > document
>> >> > > > > > > > >> > > to
>> >> > > > > > > > >> > > > > > >> > carefully
>> >> > > > > > > > >> > > > > > >> > > > > > evaluate
>> >> > > > > > > > >> > > > > > >> > > > > > > > the
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> architectural
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> >
>> >> implications
>> >> > > and
>> >> > > > > long
>> >> > > > > > > term
>> >> > > > > > > > >> > > impact.
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> >
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > A REST
>> >> > > endpoint to
>> >> > > > > > > query
>> >> > > > > > > > >> > metrics
>> >> > > > > > > > >> > > > > seems
>> >> > > > > > > > >> > > > > > >> > > > reasonable
>> >> > > > > > > > >> > > > > > >> > > > > > > given
>> >> > > > > > > > >> > > > > > >> > > > > > > > > the
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> current
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> JDBC
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > based
>> >> > > persistence
>> >> > > > > > > model.
>> >> > > > > > > > >> That
>> >> > > > > > > > >> > > said,
>> >> > > > > > > > >> > > > > we
>> >> > > > > > > > >> > > > > > >> may
>> >> > > > > > > > >> > > > > > >> > > also
>> >> > > > > > > > >> > > > > > >> > > > > > > > consider
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> alternative
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > storage
>> >> > > models. For
>> >> > > > > > > > >> example,
>> >> > > > > > > > >> > if
>> >> > > > > > > > >> > > we
>> >> > > > > > > > >> > > > > > later
>> >> > > > > > > > >> > > > > > >> > > adopt
>> >> > > > > > > > >> > > > > > >> > > > a
>> >> > > > > > > > >> > > > > > >> > > > > > time
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > series
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> system
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> such as
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> >
>> >> Prometheus to
>> >> > > store
>> >> > > > > > > > >> metrics,
>> >> > > > > > > > >> > the
>> >> > > > > > > > >> > > > > query
>> >> > > > > > > > >> > > > > > >> > model
>> >> > > > > > > > >> > > > > > >> > > > and
>> >> > > > > > > > >> > > > > > >> > > > > > > access
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > patterns
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> would
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> be
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> >
>> >> fundamentally
>> >> > > > > > > different.
>> >> > > > > > > > >> > > Designing
>> >> > > > > > > > >> > > > > the
>> >> > > > > > > > >> > > > > > >> REST
>> >> > > > > > > > >> > > > > > >> > > API
>> >> > > > > > > > >> > > > > > >> > > > > > > without
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> considering
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> these
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> >
>> potential
>> >> > > > > evolutions
>> >> > > > > > > may
>> >> > > > > > > > >> limit
>> >> > > > > > > > >> > > > > > >> flexibility.
>> >> > > > > > > > >> > > > > > >> > > I'd
>> >> > > > > > > > >> > > > > > >> > > > > > > suggest
>> >> > > > > > > > >> > > > > > >> > > > > > > > > to
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > start
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> with
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> the
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > use
>> case.
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> >
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > Yufei
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> >
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> >
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > On Fri,
>> >> Feb
>> >> > 27,
>> >> > > > > 2026
>> >> > > > > > at
>> >> > > > > > > > >> > 3:42 PM
>> >> > > > > > > > >> > > > > Dmitri
>> >> > > > > > > > >> > > > > > >> > > > > > Bourlatchkov <
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >>
>> >> dimas@apache.org>
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > wrote:
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> >
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > > Hi
>> >> Anand,
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > >
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > >
>> Sharing
>> >> my
>> >> > > > > view...
>> >> > > > > > > > >> subject
>> >> > > > > > > > >> > to
>> >> > > > > > > > >> > > > > > >> discussion:
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > >
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > > 1.
>> >> Adding
>> >> > > non-IRC
>> >> > > > > > > REST
>> >> > > > > > > > >> API
>> >> > > > > > > > >> > to
>> >> > > > > > > > >> > > > > > Polaris
>> >> > > > > > > > >> > > > > > >> is
>> >> > > > > > > > >> > > > > > >> > > > > > perfectly
>> >> > > > > > > > >> > > > > > >> > > > > > > > > fine.
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > >
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > >
>> Figuring
>> >> > out
>> >> > > > > > specific
>> >> > > > > > > > >> > endpoint
>> >> > > > > > > > >> > > > > URIs
>> >> > > > > > > > >> > > > > > >> and
>> >> > > > > > > > >> > > > > > >> > > > > payloads
>> >> > > > > > > > >> > > > > > >> > > > > > > > might
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > require
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> a
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> few
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > >
>> >> roundtrips,
>> >> > > so
>> >> > > > > > > opening a
>> >> > > > > > > > >> > > separate
>> >> > > > > > > > >> > > > > > >> thread
>> >> > > > > > > > >> > > > > > >> > > for
>> >> > > > > > > > >> > > > > > >> > > > > that
>> >> > > > > > > > >> > > > > > >> > > > > > > > might
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > be
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> best.
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > >
>> >> > Contributors
>> >> > > > > > commonly
>> >> > > > > > > > >> create
>> >> > > > > > > > >> > > > > Google
>> >> > > > > > > > >> > > > > > >> Docs
>> >> > > > > > > > >> > > > > > >> > > for
>> >> > > > > > > > >> > > > > > >> > > > > new
>> >> > > > > > > > >> > > > > > >> > > > > > > API
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > proposals
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> too
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> (they
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > >
>> fairly
>> >> easy
>> >> > > to
>> >> > > > > > > update as
>> >> > > > > > > > >> the
>> >> > > > > > > > >> > > email
>> >> > > > > > > > >> > > > > > >> > > discussion
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > progresses).
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > >
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > > There
>> >> was a
>> >> > > > > > > suggestion to
>> >> > > > > > > > >> > try
>> >> > > > > > > > >> > > > > > Markdown
>> >> > > > > > > > >> > > > > > >> > > (with
>> >> > > > > > > > >> > > > > > >> > > > > PRs)
>> >> > > > > > > > >> > > > > > >> > > > > > > for
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > proposals
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> [1]
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> ...
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > > feel
>> >> free
>> >> > to
>> >> > > give
>> >> > > > > > it
>> >> > > > > > > a
>> >> > > > > > > > >> try
>> >> > > > > > > > >> > if
>> >> > > > > > > > >> > > you
>> >> > > > > > > > >> > > > > > are
>> >> > > > > > > > >> > > > > > >> > > > > comfortable
>> >> > > > > > > > >> > > > > > >> > > > > > > > with
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > that.
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > >
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > > 2.
>> Could
>> >> > you
>> >> > > > > > clarify
>> >> > > > > > > > >> whether
>> >> > > > > > > > >> > > you
>> >> > > > > > > > >> > > > > > mean
>> >> > > > > > > > >> > > > > > >> end
>> >> > > > > > > > >> > > > > > >> > > > user
>> >> > > > > > > > >> > > > > > >> > > > > > > > > utilities
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > or
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> admin
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> user
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > >
>> >> utilities?
>> >> > > In the
>> >> > > > > > > latter
>> >> > > > > > > > >> > case
>> >> > > > > > > > >> > > > > those
>> >> > > > > > > > >> > > > > > >> might
>> >> > > > > > > > >> > > > > > >> > > be
>> >> > > > > > > > >> > > > > > >> > > > > more
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > suitable
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > for
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> the
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> Admin
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > > CLI
>> >> (java)
>> >> > > not
>> >> > > > > the
>> >> > > > > > > Python
>> >> > > > > > > > >> > CLI,
>> >> > > > > > > > >> > > > > IMHO.
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > >
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > > Why
>> >> would
>> >> > > these
>> >> > > > > > > > >> utilities be
>> >> > > > > > > > >> > > > > common
>> >> > > > > > > > >> > > > > > >> with
>> >> > > > > > > > >> > > > > > >> > > > > events?
>> >> > > > > > > > >> > > > > > >> > > > > > > > IMHO,
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > event
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> use
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> cases
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > are
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > >
>> distinct
>> >> > from
>> >> > > > > > > scan/commit
>> >> > > > > > > > >> > > metrics.
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > >
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > > 3.
>> I'd
>> >> > prefer
>> >> > > > > > > separating
>> >> > > > > > > > >> > > metrics
>> >> > > > > > > > >> > > > > > >> > > persistence
>> >> > > > > > > > >> > > > > > >> > > > > from
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > MetaStore
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>>
>> persistence
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > at
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > > the
>> code
>> >> > > level,
>> >> > > > > so
>> >> > > > > > > that
>> >> > > > > > > > >> they
>> >> > > > > > > > >> > > could
>> >> > > > > > > > >> > > > > > be
>> >> > > > > > > > >> > > > > > >> > mixed
>> >> > > > > > > > >> > > > > > >> > > > and
>> >> > > > > > > > >> > > > > > >> > > > > > > > matched
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>>
>> >> independently.
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > The
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > >
>> separate
>> >> > > > > datasource
>> >> > > > > > > > >> question
>> >> > > > > > > > >> > > will
>> >> > > > > > > > >> > > > > > >> become
>> >> > > > > > > > >> > > > > > >> > a
>> >> > > > > > > > >> > > > > > >> > > > > > > non-issue
>> >> > > > > > > > >> > > > > > >> > > > > > > > > with
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > that
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> approach,
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > I
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > >
>> guess.
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > >
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > > The
>> >> > > rationale for
>> >> > > > > > > > >> separating
>> >> > > > > > > > >> > > scan
>> >> > > > > > > > >> > > > > > >> metrics
>> >> > > > > > > > >> > > > > > >> > > and
>> >> > > > > > > > >> > > > > > >> > > > > > > > metastore
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> persistence
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> is
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > that
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > >
>> >> "cascading
>> >> > > > > deletes"
>> >> > > > > > > > >> between
>> >> > > > > > > > >> > > them
>> >> > > > > > > > >> > > > > are
>> >> > > > > > > > >> > > > > > >> > hardly
>> >> > > > > > > > >> > > > > > >> > > > > ever
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > required.
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>>
>> Furthermore,
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > the
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > > data
>> and
>> >> > > query
>> >> > > > > > > patterns
>> >> > > > > > > > >> are
>> >> > > > > > > > >> > > very
>> >> > > > > > > > >> > > > > > >> > different
>> >> > > > > > > > >> > > > > > >> > > so
>> >> > > > > > > > >> > > > > > >> > > > > > > > different
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> technologies
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > might
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > > be
>> >> > > beneficial in
>> >> > > > > > each
>> >> > > > > > > > >> case.
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > >
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > > [1]
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> >
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>>
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >>
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >>
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > >
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > >
>> >> > > > > > > > >> > > > > > >> > > > > > > > > >
>> >> > > > > > > > >> > > > > > >> > > > > > > > >
>> >> > > > > > > > >> > > > > > >> > > > > > > >
>> >> > > > > > > > >> > > > > > >> > > > > > >
>> >> > > > > > > > >> > > > > > >> > > > > >
>> >> > > > > > > > >> > > > > > >> > > > >
>> >> > > > > > > > >> > > > > > >> > > >
>> >> > > > > > > > >> > > > > > >> > >
>> >> > > > > > > > >> > > > > > >> >
>> >> > > > > > > > >> > > > > > >>
>> >> > > > > > > > >> > > > > >
>> >> > > > > > > > >> > > > >
>> >> > > > > > > > >> > >
>> >> > > > > > > > >> >
>> >> > > > > > > > >>
>> >> > > > > > >
>> >> > > > > >
>> >> > > > >
>> >> > >
>> >> >
>> >>
>> https://urldefense.com/v3/__https://lists.apache.org/thread/yto2wp982t43h1mqjwnslswhws5z47cy__;!!Iz9xO38YGHZK!5EuyFFkk3vhRWVIRvQAWBSQfpJkTMA9HxugzDwXmN0LPPqhEFxYkFRGVhtb8AqUwXtDh2OplcMnbMDHKxYDakNU$
>> >> > > > > > > > >> > >> > > >> > > > > > > > > > > > >> >> >>> > >
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > >
>> Cheers,
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > >
>> Dmitri.
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > >
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > > On
>> Fri,
>> >> Feb
>> >> > > 27,
>> >> > > > > > 2026
>> >> > > > > > > at
>> >> > > > > > > > >> > > 6:19 PM
>> >> > > > > > > > >> > > > > > Anand
>> >> > > > > > > > >> > > > > > >> > Kumar
>> >> > > > > > > > >> > > > > > >> > > > > > > Sankaran
>> >> > > > > > > > >> > > > > > >> > > > > > > > > via
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > dev
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > <
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > >
>> >> > > > > > > dev@polaris.apache.org>
>> >> > > > > > > > >> > > wrote:
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > >
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > > >
>> Thanks
>> >> > all.
>> >> > > > > This
>> >> > > > > > > PR is
>> >> > > > > > > > >> > > merged
>> >> > > > > > > > >> > > > > now.
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > > >
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > > >
>> Here
>> >> are
>> >> > > the
>> >> > > > > > > follow-up
>> >> > > > > > > > >> > > features
>> >> > > > > > > > >> > > > > /
>> >> > > > > > > > >> > > > > > >> work
>> >> > > > > > > > >> > > > > > >> > > > > needed.
>> >> > > > > > > > >> > > > > > >> > > > > > > > These
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > were
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> all
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> part of
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > > the
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > > >
>> >> merged PR
>> >> > > at
>> >> > > > > some
>> >> > > > > > > > >> point in
>> >> > > > > > > > >> > > time
>> >> > > > > > > > >> > > > > > and
>> >> > > > > > > > >> > > > > > >> > were
>> >> > > > > > > > >> > > > > > >> > > > > > removed
>> >> > > > > > > > >> > > > > > >> > > > > > > to
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > reduce
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> scope.
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > > >
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > > >
>> Please
>> >> > let
>> >> > > me
>> >> > > > > > know
>> >> > > > > > > what
>> >> > > > > > > > >> > you
>> >> > > > > > > > >> > > > > think.
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > > >
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > > >
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > > >
>>  1.
>> >> A
>> >> > > REST
>> >> > > > > API
>> >> > > > > > to
>> >> > > > > > > > >> > paginate
>> >> > > > > > > > >> > > > > > through
>> >> > > > > > > > >> > > > > > >> > > table
>> >> > > > > > > > >> > > > > > >> > > > > > > metrics.
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > This
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> will be
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > non-IRC
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > > >
>> >> standard
>> >> > > > > > addition.
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > > >
>>  2.
>> >> > > Utilities
>> >> > > > > > for
>> >> > > > > > > > >> > > managing old
>> >> > > > > > > > >> > > > > > >> > records,
>> >> > > > > > > > >> > > > > > >> > > > > > should
>> >> > > > > > > > >> > > > > > >> > > > > > > be
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > common
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> with
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> events.
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > > >
>> There
>> >> was
>> >> > > some
>> >> > > > > > > > >> discussion
>> >> > > > > > > > >> > > that
>> >> > > > > > > > >> > > > > it
>> >> > > > > > > > >> > > > > > >> > belongs
>> >> > > > > > > > >> > > > > > >> > > > to
>> >> > > > > > > > >> > > > > > >> > > > > > the
>> >> > > > > > > > >> > > > > > >> > > > > > > > CLI.
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > > >
>>  3.
>> >> > > Separate
>> >> > > > > > > > >> datasource
>> >> > > > > > > > >> > > > > > (metrics,
>> >> > > > > > > > >> > > > > > >> > > events,
>> >> > > > > > > > >> > > > > > >> > > > > > even
>> >> > > > > > > > >> > > > > > >> > > > > > > > > other
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> tables?).
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > > >
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > > >
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > > >
>> >> Anything
>> >> > > else?
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > > >
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > > > -
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > > >
>> Anand
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > > >
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > > >
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > >
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> >
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> >
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>>
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >>
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >>
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > > >
>> >> > > > > > > > >> > > > > > >> > > > > > > > > > >
>> >> > > > > > > > >> > > > > > >> > > > > > > > > >
>> >> > > > > > > > >> > > > > > >> > > > > > > > >
>> >> > > > > > > > >> > > > > > >> > > > > > > >
>> >> > > > > > > > >> > > > > > >> > > > > > >
>> >> > > > > > > > >> > > > > > >> > > > > >
>> >> > > > > > > > >> > > > > > >> > > > >
>> >> > > > > > > > >> > > > > > >> > > >
>> >> > > > > > > > >> > > > > > >> > >
>> >> > > > > > > > >> > > > > > >> >
>> >> > > > > > > > >> > > > > > >>
>> >> > > > > > > > >> > > > > > >
>> >> > > > > > > > >> > > > > >
>> >> > > > > > > > >> > > > >
>> >> > > > > > > > >> > >
>> >> > > > > > > > >> >
>> >> > > > > > > > >> >
>> >> > > > > > > > >>
>> >> > > > > > > > >
>> >> > > > > > >
>> >> > > > > >
>> >> > > > >
>> >> > >
>> >> >
>> >>
>> >
>>
>

Re: Proposal for REST endpoints for table metrics and events

[Yufei Gu <fl...@gmail.com>]

Agreed. Thanks Dmitri! I will take a look at the PR today.

Yufei


On Wed, Oct 21, 2026 at 8:59 AM Dmitri Bourlatchkov <di...@apache.org>
wrote:

> Hi All,
>
> Heads up: it looks like the "Metrics Architecture" meeting is no longer
> necessary. It will be removed from the Polaris calendar presently.
>
> Cheers,
> Dmitri.
>
> On Tue, Oct 20, 2026 at 12:16 PM Dmitri Bourlatchkov <di...@apache.org>
> wrote:
>
> > Hi All,
> >
> > PR [4115] has been approved for two weeks without any follow-up comments.
> >
> > I'm going to merge it on Oct 22 unless fresh concerns are raised.
> >
> > [4115] https://github.com/apache/polaris/pull/4115
> >
> > Cheers,
> > Dmitri.
> >
> > I
> >
> > On Wed, Oct 7, 2026 at 9:45 PM Yufei Gu <fl...@gmail.com> wrote:
> >
> >> Thanks for the reminder, Dmitri! I will take a look.
> >>
> >> Yufei
> >>
> >>
> >> On Wed, Oct 7, 2026 at 5:37 PM Dmitri Bourlatchkov <di...@apache.org>
> >> wrote:
> >>
> >> > Hi All,
> >> >
> >> > Metrics grants have been merged as a separate PR.
> >> >
> >> > Now [4115] is ready for one more review round. IMHO, it is ready to
> >> merge.
> >> >
> >> > [4115] https://github.com/apache/polaris/pull/4115
> >> >
> >> > Cheers,
> >> > Dmitri.
> >> >
> >> > On Fri, Oct 2, 2026 at 1:01 AM Jean-Baptiste Onofré <jb...@nanthrax.net>
> >> > wrote:
> >> >
> >> > > Hi all,
> >> > >
> >> > > I agree with Yufei: I would keep TABLE_READ_METRICS as a separate,
> >> > > explicit grant, not implied by TABLE_FULL_METADATA.
> >> > >
> >> > > I believe metrics reports are closer to an activity log than to
> table
> >> > > metadata. I also think it's consistent with what we have today, as
> >> > > reporting scan and commit metrics is already with TABLE_READ_DATA
> and
> >> > > TABLE_WRITE_DATA, not by "metadata" privilege.
> >> > >
> >> > > For the troubleshooting use case, the cost is just one additional
> >> > > grant, I think it's acceptable :) I would still let
> >> > > CATALOG_MANAGE_CONTENT implies it, as that privilege already implies
> >> > > TABLE_READ_DATA.
> >> > >
> >> > > Concretely for PR #5669, I propose:
> >> > > - to remove the "Implied by TABLE_FULL_METADATA" statement from the
> >> > > javadoc and CHANGELOG
> >> > > - to add SUPER_PRIVILEGES entry in the PR, so the grant change is
> >> > > self-contained
> >> > > - to drop ICEBERG_TABLE restriction.
> >> > > - to update access-control.md documenting TABLE_READ_METRICS, etc
> >> > >
> >> > > I will comment directly in the PR.
> >> > >
> >> > > Regards
> >> > > JB
> >> > >
> >> > > On Fri, Oct 2, 2026 at 3:40 AM Yufei Gu <fl...@gmail.com>
> wrote:
> >> > > >
> >> > > > Thanks Dmitri for the reminder.
> >> > > >
> >> > > > PR #5659 <https://github.com/apache/polaris/pull/5659> adds
> >> > > > TABLE_READ_METRICS and says it is implied by TABLE_FULL_METADATA.
> >> > Should
> >> > > > that be the default?
> >> > > >
> >> > > > I’m leaning toward keeping it separate. Metrics describe workloads
> >> > (Spark
> >> > > > ETL, Flink streaming, etc) using a table and can include query
> >> filters,
> >> > > so
> >> > > > someone managing table metadata may not need access to other
> users’
> >> > query
> >> > > > activity.
> >> > > >
> >> > > > Including it could make troubleshooting easier for table admins,
> >> > though.
> >> > > > Are there use cases that favor automatic access over an explicit
> >> grant?
> >> > > > Yufei
> >> > > >
> >> > > >
> >> > > > On Wed, Sep 30, 2026 at 1:57 PM Dmitri Bourlatchkov <
> >> dimas@apache.org>
> >> > > > wrote:
> >> > > >
> >> > > > > Hi Yufei,
> >> > > > >
> >> > > > > Here's the separate PR for Metrics-related grants:
> >> > > > > https://github.com/apache/polaris/pull/5659
> >> > > > >
> >> > > > > Please review it to unblock this contribution.
> >> > > > >
> >> > > > > Thanks,
> >> > > > > Dmitri.
> >> > > > >
> >> > > > > On Wed, Sep 30, 2026 at 1:05 PM Yufei Gu <fl...@gmail.com>
> >> > wrote:
> >> > > > >
> >> > > > > > From the previous email, we agreed to split the REST change
> from
> >> > the
> >> > > > > grant
> >> > > > > > change. I think the PR is ready to go if we only keep the
> grant
> >> > > changes.
> >> > > > > >
> >> > > > > > Yufei
> >> > > > > >
> >> > > > > >
> >> > > > > > On Tue, Sep 29, 2026 at 9:26 PM Jean-Baptiste Onofré <
> >> > > jb@nanthrax.net>
> >> > > > > > wrote:
> >> > > > > >
> >> > > > > > > Hi guys,
> >> > > > > > >
> >> > > > > > > I agree with Dmitri: +1 to merge PR #4115 on Oct 1
> (tomorrow).
> >> > > > > > >
> >> > > > > > > From my side, all outstanding feedback has been addressed,
> >> > > including
> >> > > > > > > the final documentation points raised by EJ regarding the
> >> > empty-200
> >> > > > > > > NoOp behavior and reconstructed payload description. The
> >> branch
> >> > is
> >> > > > > > > clean, up to date with main, and CI is green.
> >> > > > > > >
> >> > > > > > > Once this lands, we can follow up with PR #4756 for the JDBC
> >> > query
> >> > > SPI
> >> > > > > > > implementation.
> >> > > > > > >
> >> > > > > > > Regards
> >> > > > > > > JB
> >> > > > > > >
> >> > > > > > > On Wed, Sep 30, 2026 at 1:53 AM Dmitri Bourlatchkov <
> >> > > dimas@apache.org>
> >> > > > > > > wrote:
> >> > > > > > > >
> >> > > > > > > > Hi All,
> >> > > > > > > >
> >> > > > > > > > Any remaining blockers on [4115]?
> >> > > > > > > >
> >> > > > > > > > Given that the review was very long, please restate any
> >> > remaining
> >> > > > > > > concerns
> >> > > > > > > > (here or in GH).
> >> > > > > > > >
> >> > > > > > > > Otherwise, I propose merging [4115] on Oct 1.
> >> > > > > > > >
> >> > > > > > > > [4115] https://github.com/apache/polaris/pull/4115
> >> > > > > > > >
> >> > > > > > > > Thanks,
> >> > > > > > > > Dmitri.
> >> > > > > > > >
> >> > > > > > > > On Wed, Sep 23, 2026 at 7:53 PM Dmitri Bourlatchkov <
> >> > > > > dimas@apache.org>
> >> > > > > > > > wrote:
> >> > > > > > > >
> >> > > > > > > > > Hi EJ,
> >> > > > > > > > >
> >> > > > > > > > > Thanks for the meeting recap!
> >> > > > > > > > >
> >> > > > > > > > > All:
> >> > > > > > > > >
> >> > > > > > > > > Worth noting: PR [4115] also contains Management API
> >> changes
> >> > > that
> >> > > > > > were
> >> > > > > > > not
> >> > > > > > > > > anticipated initially and did not get an API change
> review
> >> > > until
> >> > > > > now.
> >> > > > > > > These
> >> > > > > > > > > are small changes to add new grant types for metrics
> >> RBAC..
> >> > > > > > > > >
> >> > > > > > > > > I've reviewed and approved PR [4115] in GH.
> >> > > > > > > > >
> >> > > > > > > > > [4115] https://github.com/apache/polaris/pull/4115
> >> > > > > > > > >
> >> > > > > > > > > Cheers,
> >> > > > > > > > > Dmitri.
> >> > > > > > > > >
> >> > > > > > > > > On Wed, Aug 12, 2026 at 5:21 PM EJ Wang <
> >> > > > > > > ej.wang.dev.interest@gmail.com>
> >> > > > > > > > > wrote:
> >> > > > > > > > >
> >> > > > > > > > >> Hi folks,
> >> > > > > > > > >>
> >> > > > > > > > >> Notes from today's metrics architecture sync are in the
> >> > usual
> >> > > doc:
> >> > > > > > > > >>
> >> > > > > > > > >>
> >> > > > > > >
> >> > > > > >
> >> > > > >
> >> > >
> >> >
> >>
> https://docs.google.com/document/d/100h7c4damrUzVuquYbBHM0EvA4LSWuW2IT2dN_7nYVA/edit?pli=1&tab=t.18iv1m7p6i6b
> >> > > > > > > > >>
> >> > > > > > > > >> Highlights:
> >> > > > > > > > >>
> >> > > > > > > > >> Module placement. core/ now holds the entity data model
> >> > only.
> >> > > A
> >> > > > > new
> >> > > > > > > spi/
> >> > > > > > > > >> takes every other shared contract: feature SPIs,
> feature
> >> > > non-SPI
> >> > > > > > > > >> contracts,
> >> > > > > > > > >> substrate SPIs, substrate non-SPI contracts, and
> >> > > contract-required
> >> > > > > > > types.
> >> > > > > > > > >> That settles the question PR #5204 reopened, by placing
> >> into
> >> > > a new
> >> > > > > > SPI
> >> > > > > > > > >> module. Note that it changes slightly towards the
> >> referenced
> >> > > WIP
> >> > > > > RFC
> >> > > > > > > [1]
> >> > > > > > > > >> recommendations: what proposed to place in core/, now
> >> should
> >> > > go to
> >> > > > > > > spi/
> >> > > > > > > > >> except for the entity data model.
> >> > > > > > > > >>
> >> > > > > > > > >> Metrics PRs. #5068 merged. #5204 needs reshaping onto
> the
> >> > > split.
> >> > > > > > > #4115 and
> >> > > > > > > > >> #4756 stay open with action on the author to follow up
> on
> >> > > review
> >> > > > > > > comments.
> >> > > > > > > > >>
> >> > > > > > > > >> [1]
> >> > > > > > > > >>
> >> > > > > > > > >>
> >> > > > > > >
> >> > > > > >
> >> > > > >
> >> > >
> >> >
> >>
> https://docs.google.com/document/d/1mj3vGpix9zTDTFFJKtWT5ZxI3PTGI1YNb5xFqyWrOkc/edit?tab=t.0
> >> > > > > > > > >>
> >> > > > > > > > >> On Wed, Jul 15, 2026 at 11:01 AM Anand Kumar Sankaran
> via
> >> > dev
> >> > > <
> >> > > > > > > > >> dev@polaris.apache.org> wrote:
> >> > > > > > > > >>
> >> > > > > > > > >> > Thanks EJ.  My plan is to introduce PR0 in a new
> >> branch /
> >> > > PR and
> >> > > > > > > > >> repurpose
> >> > > > > > > > >> > #4115 and #4756 for PR1 and PR2 to maintain all the
> >> > valuable
> >> > > > > > > discussions
> >> > > > > > > > >> > that happened there.
> >> > > > > > > > >> >
> >> > > > > > > > >> > From: EJ Wang <ej...@gmail.com>
> >> > > > > > > > >> > Date: Wednesday, July 15, 2026 at 10:41 AM
> >> > > > > > > > >> > To: dev@polaris.apache.org <de...@polaris.apache.org>
> >> > > > > > > > >> > Subject: Re: Proposal for REST endpoints for table
> >> metrics
> >> > > and
> >> > > > > > > events
> >> > > > > > > > >> >
> >> > > > > > > > >> > Hi folks, Thanks again for the discussion today. I
> >> updated
> >> > > the
> >> > > > > > sync
> >> > > > > > > doc
> >> > > > > > > > >> > with notes from today's metrics sync: https:
> >> //urldefense.
> >> > > > > > > > >> com/v3/__https:
> >> > > > > > > > >> > //docs. google.
> >> > > > > > > > >> >
> >> > > > > > > > >>
> >> > > > > > >
> >> > > > > >
> >> > > > >
> >> > >
> >> >
> >>
> com/document/d/100h7c4damrUzVuquYbBHM0EvA4LSWuW2IT2dN_7nYVA/edit?pli=1&tab=t.
> >> > > > > > > > >> >
> >> > > > > > > > >>
> >> > > > > > >
> >> > > > > >
> >> > > > >
> >> > >
> >> >
> >>
> 2ahd7wze5f__;!!Iz9xO38YGHZK!4rnlVIqBQQQ6xpGe1PMV58edOS_cGrKoqdfhw4qSp_JQCavvEjaxXk0OwGPnxyEcO60AzKQK_XDtp7H3DdkFXcrXJwAz1g0$
> >> > > > > > > > >> >
> >> > > > > > > > >> >
> >> > > > > > > > >> > Hi folks,
> >> > > > > > > > >> >
> >> > > > > > > > >> > Thanks again for the discussion today. I updated the
> >> sync
> >> > > doc
> >> > > > > with
> >> > > > > > > notes
> >> > > > > > > > >> > from today's metrics sync:
> >> > > > > > > > >> >
> >> > > > > > > > >> >
> >> > > > > > > > >>
> >> > > > > > >
> >> > > > > >
> >> > > > >
> >> > >
> >> >
> >>
> https://urldefense.com/v3/__https://docs.google.com/document/d/100h7c4damrUzVuquYbBHM0EvA4LSWuW2IT2dN_7nYVA/edit?pli=1&tab=t.2ahd7wze5f__;!!Iz9xO38YGHZK!4rnlVIqBQQQ6xpGe1PMV58edOS_cGrKoqdfhw4qSp_JQCavvEjaxXk0OwGPnxyEcO60AzKQK_XDtp7H3DdkFXcrXJwAz1g0$
> >> > > > > > > > >> >
> >> > > > > > > > >> > See highlights below:
> >> > > > > > > > >> >
> >> > > > > > > > >> > In previous meetings, EJ proposed a two prs framing
> to
> >> > push
> >> > > > > > forward
> >> > > > > > > the
> >> > > > > > > > >> > metrics query work in polaris. Dmitri pointed out a
> >> subtle
> >> > > point
> >> > > > > > > that
> >> > > > > > > > >> pr2
> >> > > > > > > > >> > is currently also trying to do the persistence
> related
> >> > code
> >> > > > > > > placement
> >> > > > > > > > >> > refactor. For the sake of straightforward review
> >> process,
> >> > > EJ,
> >> > > > > > > Dmitri and
> >> > > > > > > > >> > Anand (the author) agreed to separate the persistence
> >> > > > > refactoring
> >> > > > > > > into
> >> > > > > > > > >> pr0,
> >> > > > > > > > >> > and reframing the pr scopes as follow:
> >> > > > > > > > >> >
> >> > > > > > > > >> >    1.
> >> > > > > > > > >> >
> >> > > > > > > > >> >    Pr0: refactor irc metrics spi
> >> > > > > > > > >> >    1.
> >> > > > > > > > >> >
> >> > > > > > > > >> >       SPI should stay in core/ with metric type
> >> enveloped
> >> > > > > > > > >> >       2.
> >> > > > > > > > >> >
> >> > > > > > > > >> >       default spi impl, no-op, in extensions/ -
> default
> >> > > > > selection
> >> > > > > > at
> >> > > > > > > > >> >       runtime/
> >> > > > > > > > >> >       3.
> >> > > > > > > > >> >
> >> > > > > > > > >> >       move metrics persistence and correspondings to
> >> > > extensions/
> >> > > > > > as
> >> > > > > > > an
> >> > > > > > > > >> >       alternative SPI impl
> >> > > > > > > > >> >       2.
> >> > > > > > > > >> >
> >> > > > > > > > >> >    Pr1: rest api for query
> >> > > > > > > > >> >
> >> > > > > > > > >> >
> >> > > > > > > > >>
> >> > > > > > >
> >> > > > > >
> >> > > > >
> >> > >
> >> >
> >>
> https://urldefense.com/v3/__https://github.com/apache/polaris/pull/4115/changes__;!!Iz9xO38YGHZK!4rnlVIqBQQQ6xpGe1PMV58edOS_cGrKoqdfhw4qSp_JQCavvEjaxXk0OwGPnxyEcO60AzKQK_XDtp7H3DdkFXcrXFCbEbRk$
> >> > > > > > > > >> >    1.
> >> > > > > > > > >> >
> >> > > > > > > > >> >       rest api spec in extensions/ as optional spec
> >> > > > > > > > >> >       2.
> >> > > > > > > > >> >
> >> > > > > > > > >> >       SPI in extensions/ as optional spec spi
> >> > > > > > > > >> >       3.
> >> > > > > > > > >> >
> >> > > > > > > > >> >       default spi impl in extensions/ no-op
> >> > > > > > > > >> >       4.
> >> > > > > > > > >> >
> >> > > > > > > > >> >       handler related in runtime/ - thin adaptor from
> >> http
> >> > > shape
> >> > > > > > to
> >> > > > > > > spi
> >> > > > > > > > >> >       shape
> >> > > > > > > > >> >       3.
> >> > > > > > > > >> >
> >> > > > > > > > >> >    Pr2: jdbc impl for query
> >> > > > > > > > >> >
> >> > > > > > > > >>
> >> > > > > > >
> >> > > > > >
> >> > > > >
> >> > >
> >> >
> >>
> https://urldefense.com/v3/__https://github.com/apache/polaris/pull/4756__;!!Iz9xO38YGHZK!4rnlVIqBQQQ6xpGe1PMV58edOS_cGrKoqdfhw4qSp_JQCavvEjaxXk0OwGPnxyEcO60AzKQK_XDtp7H3DdkFXcrXRVMBzqE$
> >> > > > > > > > >> >    1.
> >> > > > > > > > >> >
> >> > > > > > > > >> >       alternative spi impl in extensions/ -> query
> with
> >> > jdbc
> >> > > > > > > > >> >
> >> > > > > > > > >> >
> >> > > > > > > > >> > Thanks,
> >> > > > > > > > >> > -ej
> >> > > > > > > > >> >
> >> > > > > > > > >> > On Wed, Jul 1, 2026 at 8:01 AM Alexandre Dutra <
> >> > > > > adutra@apache.org
> >> > > > > > >
> >> > > > > > > > >> wrote:
> >> > > > > > > > >> >
> >> > > > > > > > >> > > Hi all,
> >> > > > > > > > >> > >
> >> > > > > > > > >> > > > While the current implementation stores metrics
> in
> >> the
> >> > > > > > > metastore,
> >> > > > > > > > >> other
> >> > > > > > > > >> > > deployments
> >> > > > > > > > >> > > may choose a KV store, a dedicated metrics
> >> warehouse, or
> >> > > > > another
> >> > > > > > > > >> backend
> >> > > > > > > > >> > > entirely. In those cases, exposing a Polaris REST
> API
> >> > > over a
> >> > > > > > > specific
> >> > > > > > > > >> > > schema feels like coupling the API to an
> >> implementation
> >> > > > > detail.
> >> > > > > > > > >> > >
> >> > > > > > > > >> > > I wouldn't say to an implementation detail, but
> >> rather
> >> > to
> >> > > a
> >> > > > > > > particular
> >> > > > > > > > >> > > setup / configuration. I think that's fine. If you
> >> don't
> >> > > store
> >> > > > > > > > >> > > metrics, you don't read metrics. Similar situations
> >> will
> >> > > > > > certainly
> >> > > > > > > > >> > > happen in the future, e.g. if we ever introduce an
> >> API
> >> > to
> >> > > read
> >> > > > > > > events,
> >> > > > > > > > >> > > that would only work if events are being persisted
> in
> >> > the
> >> > > > > first
> >> > > > > > > place.
> >> > > > > > > > >> > > Again I think that's a fair trade-off.
> >> > > > > > > > >> > >
> >> > > > > > > > >> > > Thanks,
> >> > > > > > > > >> > > Alex
> >> > > > > > > > >> > >
> >> > > > > > > > >> > > On Wed, Jul 1, 2026 at 7:23 AM Dmitri Bourlatchkov
> <
> >> > > > > > > dimas@apache.org>
> >> > > > > > > > >> > > wrote:
> >> > > > > > > > >> > > >
> >> > > > > > > > >> > > > Hi Yufei,
> >> > > > > > > > >> > > >
> >> > > > > > > > >> > > > From my POV, it is fine for Polaris to have
> >> multiple
> >> > > REST
> >> > > > > API
> >> > > > > > > > >> surfaces
> >> > > > > > > > >> > > with
> >> > > > > > > > >> > > > different features and different degrees of
> >> > "readiness".
> >> > > > > Each
> >> > > > > > > API
> >> > > > > > > > >> can
> >> > > > > > > > >> > > > evolve fairly independently. With proper
> >> > modularization
> >> > > > > (which
> >> > > > > > > Anand
> >> > > > > > > > >> > > did),
> >> > > > > > > > >> > > > changes to the main Polaris code and these extra
> >> > modules
> >> > > > > > should
> >> > > > > > > > >> > normally
> >> > > > > > > > >> > > be
> >> > > > > > > > >> > > > non-intrusive to each other.
> >> > > > > > > > >> > > >
> >> > > > > > > > >> > > > I assume Anand is going to be the first user of
> >> this
> >> > > API.
> >> > > > > > > However,
> >> > > > > > > > >> if
> >> > > > > > > > >> > we
> >> > > > > > > > >> > > do
> >> > > > > > > > >> > > > not merge it we'll never know whether other users
> >> are
> >> > > > > > > interested. I
> >> > > > > > > > >> do
> >> > > > > > > > >> > > not
> >> > > > > > > > >> > > > think many end users read the dev ML, but they
> >> > certainly
> >> > > > > read
> >> > > > > > > the
> >> > > > > > > > >> docs,
> >> > > > > > > > >> > > > code and release notes.
> >> > > > > > > > >> > > >
> >> > > > > > > > >> > > > There is no risk for Polaris as a project in
> >> exposing
> >> > > this
> >> > > > > > > > >> experimental
> >> > > > > > > > >> > > > API. If usage dwindles over time, we can remove
> >> it. If
> >> > > it
> >> > > > > > gains
> >> > > > > > > a
> >> > > > > > > > >> > > > considerable user / contributor base, we will
> >> promote
> >> > > it to
> >> > > > > a
> >> > > > > > > > >> > > "first-class"
> >> > > > > > > > >> > > > (non-beta) API.
> >> > > > > > > > >> > > >
> >> > > > > > > > >> > > > Cheers,
> >> > > > > > > > >> > > > Dmitri.
> >> > > > > > > > >> > > >
> >> > > > > > > > >> > > > On Tue, Jun 30, 2026 at 7:53 PM Yufei Gu <
> >> > > > > > flyrain000@gmail.com>
> >> > > > > > > > >> wrote:
> >> > > > > > > > >> > > >
> >> > > > > > > > >> > > > > Hi all,
> >> > > > > > > > >> > > > >
> >> > > > > > > > >> > > > > Thanks for driving this work forward. Building
> >> this
> >> > > as a
> >> > > > > POC
> >> > > > > > > makes
> >> > > > > > > > >> > > perfect
> >> > > > > > > > >> > > > > sense if the goal is to validate the idea and
> >> gather
> >> > > > > > feedback.
> >> > > > > > > > >> Should
> >> > > > > > > > >> > > we
> >> > > > > > > > >> > > > > merge a POC into the main branch before
> agreeing
> >> on
> >> > > the
> >> > > > > > > overall
> >> > > > > > > > >> > > direction?
> >> > > > > > > > >> > > > > To me, that feels like a separate discussion
> from
> >> > > whether
> >> > > > > > the
> >> > > > > > > > >> > > > > implementation itself looks reasonable.
> >> > > > > > > > >> > > > >
> >> > > > > > > > >> > > > > One concern that I don't think has been fully
> >> > > addressed is
> >> > > > > > > > >> whether we
> >> > > > > > > > >> > > need
> >> > > > > > > > >> > > > > to expose metrics consumption through a Polaris
> >> REST
> >> > > > > > endpoint
> >> > > > > > > at
> >> > > > > > > > >> all.
> >> > > > > > > > >> > > > >
> >> > > > > > > > >> > > > > My concern is less about the API itself and
> more
> >> > > about the
> >> > > > > > > > >> boundary
> >> > > > > > > > >> > > Polaris
> >> > > > > > > > >> > > > > should own. The proposed REST API is tied to a
> >> > > particular
> >> > > > > > > > >> persistence
> >> > > > > > > > >> > > > > schema, but that schema is just one possible
> >> > > > > implementation.
> >> > > > > > > While
> >> > > > > > > > >> > the
> >> > > > > > > > >> > > > > current implementation stores metrics in the
> >> > > metastore,
> >> > > > > > other
> >> > > > > > > > >> > > deployments
> >> > > > > > > > >> > > > > may choose a KV store, a dedicated metrics
> >> > warehouse,
> >> > > or
> >> > > > > > > another
> >> > > > > > > > >> > > backend
> >> > > > > > > > >> > > > > entirely. In those cases, exposing a Polaris
> REST
> >> > API
> >> > > > > over a
> >> > > > > > > > >> specific
> >> > > > > > > > >> > > > > schema feels like coupling the API to an
> >> > > implementation
> >> > > > > > > detail.
> >> > > > > > > > >> > > > >
> >> > > > > > > > >> > > > > Polaris isn't intended to become a full metrics
> >> > > system.
> >> > > > > I'd
> >> > > > > > > > >> expect it
> >> > > > > > > > >> > > to
> >> > > > > > > > >> > > > > focus on producing metrics and let deployments
> >> > choose
> >> > > the
> >> > > > > > most
> >> > > > > > > > >> > > appropriate
> >> > > > > > > > >> > > > > storage and consumption model for their
> >> environment.
> >> > > > > > > > >> > > > >
> >> > > > > > > > >> > > > > I'm not opposed to exploring this direction,
> but
> >> we
> >> > > should
> >> > > > > > > first
> >> > > > > > > > >> > align
> >> > > > > > > > >> > > on
> >> > > > > > > > >> > > > > whether exposing a REST API through Polaris is
> >> the
> >> > > right
> >> > > > > > long
> >> > > > > > > term
> >> > > > > > > > >> > > approach
> >> > > > > > > > >> > > > > before merging it into the main branch.
> >> > > > > > > > >> > > > >
> >> > > > > > > > >> > > > > Thanks,
> >> > > > > > > > >> > > > > Yufei
> >> > > > > > > > >> > > > >
> >> > > > > > > > >> > > > > Yufei
> >> > > > > > > > >> > > > >
> >> > > > > > > > >> > > > >
> >> > > > > > > > >> > > > > On Mon, Jun 29, 2026 at 9:27 AM Dmitri
> >> Bourlatchkov
> >> > <
> >> > > > > > > > >> > dimas@apache.org>
> >> > > > > > > > >> > > > > wrote:
> >> > > > > > > > >> > > > >
> >> > > > > > > > >> > > > > > Hi All,
> >> > > > > > > > >> > > > > >
> >> > > > > > > > >> > > > > > It looks like Anand addressed all action
> items
> >> in
> >> > > > > [4115].
> >> > > > > > > > >> > > > > >
> >> > > > > > > > >> > > > > > Please review the latest state of the code.
> >> The PR
> >> > > LGTM
> >> > > > > > and
> >> > > > > > > I
> >> > > > > > > > >> > > approved in
> >> > > > > > > > >> > > > > > GH from my side.
> >> > > > > > > > >> > > > > >
> >> > > > > > > > >> > > > > > [4115]
> >> > > > > > > > >> >
> >> > > > > > > > >>
> >> > > > > > >
> >> > > > > >
> >> > > > >
> >> > >
> >> >
> >>
> https://urldefense.com/v3/__https://github.com/apache/polaris/pull/4115__;!!Iz9xO38YGHZK!4rnlVIqBQQQ6xpGe1PMV58edOS_cGrKoqdfhw4qSp_JQCavvEjaxXk0OwGPnxyEcO60AzKQK_XDtp7H3DdkFXcrXtFRscKs$
> >> > > > > > > > >> > > > > >
> >> > > > > > > > >> > > > > > Thanks,
> >> > > > > > > > >> > > > > > Dmitri.
> >> > > > > > > > >> > > > > >
> >> > > > > > > > >> > > > > > On Thu, Jun 25, 2026 at 1:56 PM Dmitri
> >> > Bourlatchkov
> >> > > <
> >> > > > > > > > >> > > dimas@apache.org>
> >> > > > > > > > >> > > > > > wrote:
> >> > > > > > > > >> > > > > >
> >> > > > > > > > >> > > > > > > Hi All,
> >> > > > > > > > >> > > > > > >
> >> > > > > > > > >> > > > > > > Recap of the discussion in today's
> Community
> >> > Sync:
> >> > > > > > > > >> > > > > > >
> >> > > > > > > > >> > > > > > > * The modular approach for Metrics SPI(s)
> and
> >> > > > > > > implementation
> >> > > > > > > > >> > seems
> >> > > > > > > > >> > > to
> >> > > > > > > > >> > > > > > have
> >> > > > > > > > >> > > > > > > community consensus (please comment if you
> >> still
> >> > > have
> >> > > > > > > > >> concerns)
> >> > > > > > > > >> > > > > > >
> >> > > > > > > > >> > > > > > > * What modules / feature flags are enabled
> in
> >> > > Apache
> >> > > > > > > Polaris
> >> > > > > > > > >> > > images by
> >> > > > > > > > >> > > > > > > default will be discussed later (when the
> >> > feature
> >> > > is
> >> > > > > > > close to
> >> > > > > > > > >> > > > > > completion).
> >> > > > > > > > >> > > > > > >
> >> > > > > > > > >> > > > > > > In any case, users will be able to turn the
> >> full
> >> > > > > feature
> >> > > > > > > > >> on/off
> >> > > > > > > > >> > at
> >> > > > > > > > >> > > > > their
> >> > > > > > > > >> > > > > > > discretion, regardless of the default flag
> >> > values.
> >> > > > > > > > >> > > > > > >
> >> > > > > > > > >> > > > > > > * It is generally preferable to have
> smaller
> >> PRs
> >> > > and
> >> > > > > > > introduce
> >> > > > > > > > >> > new
> >> > > > > > > > >> > > > > > > features gradually to simplify reviews.
> >> > > > > > > > >> > > > > > >
> >> > > > > > > > >> > > > > > > What this means for the Metrics API, IMHO
> is:
> >> > > > > > > > >> > > > > > >
> >> > > > > > > > >> > > > > > > a) The scope of PR 4115 is probably ok as
> it
> >> > > stands
> >> > > > > > right
> >> > > > > > > > >> now. If
> >> > > > > > > > >> > > > > someone
> >> > > > > > > > >> > > > > > > wants to propose splitting it further,
> please
> >> > > comment.
> >> > > > > > > > >> > > > > > >
> >> > > > > > > > >> > > > > > > b) After 4115 is merged, Metrics
> Persistence
> >> PRs
> >> > > will
> >> > > > > > > follow,
> >> > > > > > > > >> > with
> >> > > > > > > > >> > > > > > smaller
> >> > > > > > > > >> > > > > > > scope per PR as appropriate for the nature
> of
> >> > the
> >> > > code
> >> > > > > > > > >> changes.
> >> > > > > > > > >> > > > > > >
> >> > > > > > > > >> > > > > > > * The major concern was about exposing the
> >> new
> >> > > Metrics
> >> > > > > > > REST
> >> > > > > > > > >> API
> >> > > > > > > > >> > to
> >> > > > > > > > >> > > end
> >> > > > > > > > >> > > > > > > users. The nature of this concern was about
> >> > > avoiding
> >> > > > > the
> >> > > > > > > > >> (false)
> >> > > > > > > > >> > > > > > impression
> >> > > > > > > > >> > > > > > > that the new API is a firm and adopted
> >> Polaris
> >> > > > > > > specification.
> >> > > > > > > > >> > > > > > >
> >> > > > > > > > >> > > > > > > I think this concern is reasonable.
> >> > > > > > > > >> > > > > > >
> >> > > > > > > > >> > > > > > > I propose that Anand update the
> documentation
> >> > > sections
> >> > > > > > of
> >> > > > > > > PR
> >> > > > > > > > >> 4115
> >> > > > > > > > >> > > to
> >> > > > > > > > >> > > > > > > emphasise that the API is currently an
> >> > > experimental /
> >> > > > > > POC
> >> > > > > > > API
> >> > > > > > > > >> at
> >> > > > > > > > >> > > this
> >> > > > > > > > >> > > > > > time
> >> > > > > > > > >> > > > > > > and is subject to change (including
> breaking
> >> > > > > changes). I
> >> > > > > > > > >> believe
> >> > > > > > > > >> > it
> >> > > > > > > > >> > > > > > already
> >> > > > > > > > >> > > > > > > has the "beta" label. So this request is
> >> mostly
> >> > > for
> >> > > > > doc
> >> > > > > > > > >> changes
> >> > > > > > > > >> > to
> >> > > > > > > > >> > > > > > clarify
> >> > > > > > > > >> > > > > > > user expectations, since the "beta" label
> >> alone
> >> > > may
> >> > > > > not
> >> > > > > > be
> >> > > > > > > > >> > > interpreted
> >> > > > > > > > >> > > > > > the
> >> > > > > > > > >> > > > > > > same way by all users.
> >> > > > > > > > >> > > > > > >
> >> > > > > > > > >> > > > > > > I hope this update will resolve all
> remaining
> >> > > concerns
> >> > > > > > > with
> >> > > > > > > > >> 4115
> >> > > > > > > > >> > > and
> >> > > > > > > > >> > > > > > allow
> >> > > > > > > > >> > > > > > > for it to be merged so that the community
> >> will
> >> > be
> >> > > able
> >> > > > > > to
> >> > > > > > > > >> proceed
> >> > > > > > > > >> > > to
> >> > > > > > > > >> > > > > > > follow-up PRs (as noted above).
> >> > > > > > > > >> > > > > > >
> >> > > > > > > > >> > > > > > > Please add/correct me if I missed anything
> >> > > relevant.
> >> > > > > > > > >> > > > > > >
> >> > > > > > > > >> > > > > > > Thanks,
> >> > > > > > > > >> > > > > > > Dmitri.
> >> > > > > > > > >> > > > > > >
> >> > > > > > > > >> > > > > > > On Wed, Jun 24, 2026 at 8:11 PM Yufei Gu <
> >> > > > > > > > >> flyrain000@gmail.com>
> >> > > > > > > > >> > > wrote:
> >> > > > > > > > >> > > > > > >
> >> > > > > > > > >> > > > > > >> I do not think we have agreement on the
> >> query
> >> > > REST
> >> > > > > APIs
> >> > > > > > > yet.
> >> > > > > > > > >> > > > > > >>
> >> > > > > > > > >> > > > > > >> In particular, the current query API shape
> >> > seems
> >> > > > > highly
> >> > > > > > > > >> coupled
> >> > > > > > > > >> > > to the
> >> > > > > > > > >> > > > > > >> *example[1]* metrics in the Iceberg REST
> >> spec.
> >> > I
> >> > > do
> >> > > > > not
> >> > > > > > > think
> >> > > > > > > > >> > > that is
> >> > > > > > > > >> > > > > a
> >> > > > > > > > >> > > > > > >> good way to design a REST API contract.
> For
> >> > > example,
> >> > > > > > > fields
> >> > > > > > > > >> like
> >> > > > > > > > >> > > > > > >> resultDataFiles, resultDeleteFiles,
> >> > > > > totalFileSizeBytes,
> >> > > > > > > > >> > > > > > >> totalDataManifests,
> >> > > > > > > > >> > > > > > >> totalDeleteManifests, and
> >> scannedDataManifests
> >> > > are
> >> > > > > very
> >> > > > > > > > >> specific
> >> > > > > > > > >> > > to
> >> > > > > > > > >> > > > > one
> >> > > > > > > > >> > > > > > >> style of scan metric. That makes the API
> >> look
> >> > > more
> >> > > > > > like a
> >> > > > > > > > >> direct
> >> > > > > > > > >> > > > > > >> projection
> >> > > > > > > > >> > > > > > >> of the current example payload than a
> >> general
> >> > > query
> >> > > > > > > model.
> >> > > > > > > > >> > > > > > >>
> >> > > > > > > > >> > > > > > >> My preference is that Polaris should not
> >> define
> >> > > or
> >> > > > > > > strongly
> >> > > > > > > > >> > opine
> >> > > > > > > > >> > > on
> >> > > > > > > > >> > > > > the
> >> > > > > > > > >> > > > > > >> Iceberg scan or commit metrics query REST
> >> API
> >> > at
> >> > > this
> >> > > > > > > point.
> >> > > > > > > > >> > > Polaris
> >> > > > > > > > >> > > > > can
> >> > > > > > > > >> > > > > > >> resolve and authorize the table, accept
> the
> >> > > metrics
> >> > > > > > > report,
> >> > > > > > > > >> and
> >> > > > > > > > >> > > > > delegate
> >> > > > > > > > >> > > > > > >> it
> >> > > > > > > > >> > > > > > >> to the configured reporting path. Durable
> >> > > storage,
> >> > > > > > > filtering,
> >> > > > > > > > >> > > > > > dashboards,
> >> > > > > > > > >> > > > > > >> and query APIs should remain downstream
> >> impl.
> >> > > choices
> >> > > > > > > unless
> >> > > > > > > > >> and
> >> > > > > > > > >> > > until
> >> > > > > > > > >> > > > > > the
> >> > > > > > > > >> > > > > > >> Iceberg community standardizes a query
> >> > contract,
> >> > > > > which
> >> > > > > > I
> >> > > > > > > > >> doubt
> >> > > > > > > > >> > > will
> >> > > > > > > > >> > > > > > >> happen.
> >> > > > > > > > >> > > > > > >>
> >> > > > > > > > >> > > > > > >> So I do not think the intake path has to
> be
> >> > > blocked,
> >> > > > > > but
> >> > > > > > > I
> >> > > > > > > > >> would
> >> > > > > > > > >> > > > > prefer
> >> > > > > > > > >> > > > > > >> not
> >> > > > > > > > >> > > > > > >> to merge a Polaris specific query API that
> >> may
> >> > be
> >> > > > > hard
> >> > > > > > to
> >> > > > > > > > >> evolve
> >> > > > > > > > >> > > > > later.
> >> > > > > > > > >> > > > > > >>
> >> > > > > > > > >> > > > > > >> 1.
> >> > > > > > > > >> > > > > > >>
> >> > > > > > > > >> > > > > > >>
> >> > > > > > > > >> > > > > >
> >> > > > > > > > >> > > > >
> >> > > > > > > > >> > >
> >> > > > > > > > >> >
> >> > > > > > > > >>
> >> > > > > > >
> >> > > > > >
> >> > > > >
> >> > >
> >> >
> >>
> https://urldefense.com/v3/__https://github.com/apache/iceberg/blob/2a6c556c0c883c04c6d3fbd68e6aeda36b91e0aa/open-api/rest-catalog-open-api.yaml*L4152__;Iw!!Iz9xO38YGHZK!4rnlVIqBQQQ6xpGe1PMV58edOS_cGrKoqdfhw4qSp_JQCavvEjaxXk0OwGPnxyEcO60AzKQK_XDtp7H3DdkFXcrXCDsqYGo$
> >> > > > > > > > >> > > > > > >>
> >> > > > > > > > >> > > > > > >> Yufei
> >> > > > > > > > >> > > > > > >>
> >> > > > > > > > >> > > > > > >>
> >> > > > > > > > >> > > > > > >> On Tue, Jun 23, 2026 at 9:30 AM Dmitri
> >> > > Bourlatchkov <
> >> > > > > > > > >> > > dimas@apache.org
> >> > > > > > > > >> > > > > >
> >> > > > > > > > >> > > > > > >> wrote:
> >> > > > > > > > >> > > > > > >>
> >> > > > > > > > >> > > > > > >> > Hi All,
> >> > > > > > > > >> > > > > > >> >
> >> > > > > > > > >> > > > > > >> > So, what are the blockers for merging
> >> [4115],
> >> > > if
> >> > > > > any?
> >> > > > > > > > >> > > > > > >> >
> >> > > > > > > > >> > > > > > >> > [4115]
> >> > > > > > > > >> >
> >> > > > > > > > >>
> >> > > > > > >
> >> > > > > >
> >> > > > >
> >> > >
> >> >
> >>
> https://urldefense.com/v3/__https://github.com/apache/polaris/pull/4115__;!!Iz9xO38YGHZK!4rnlVIqBQQQ6xpGe1PMV58edOS_cGrKoqdfhw4qSp_JQCavvEjaxXk0OwGPnxyEcO60AzKQK_XDtp7H3DdkFXcrXtFRscKs$
> >> > > > > > > > >> > > > > > >> >
> >> > > > > > > > >> > > > > > >> > Thanks,
> >> > > > > > > > >> > > > > > >> > Dmitri.
> >> > > > > > > > >> > > > > > >> >
> >> > > > > > > > >> > > > > > >> > On Wed, Jun 17, 2026 at 5:28 PM EJ Wang
> <
> >> > > > > > > > >> > > > > > ej.wang.dev.interest@gmail.com
> >> > > > > > > > >> > > > > > >> >
> >> > > > > > > > >> > > > > > >> > wrote:
> >> > > > > > > > >> > > > > > >> >
> >> > > > > > > > >> > > > > > >> > > Hi folks,
> >> > > > > > > > >> > > > > > >> > >
> >> > > > > > > > >> > > > > > >> > > Thanks again for the discussion
> today. I
> >> > > updated
> >> > > > > > the
> >> > > > > > > sync
> >> > > > > > > > >> > doc
> >> > > > > > > > >> > > with
> >> > > > > > > > >> > > > > > >> notes
> >> > > > > > > > >> > > > > > >> > > from today's metrics sync:
> >> > > > > > > > >> > > > > > >> > >
> >> > > > > > > > >> > > > > > >> > >
> >> > > > > > > > >> > > > > > >> > >
> >> > > > > > > > >> > > > > > >> >
> >> > > > > > > > >> > > > > > >>
> >> > > > > > > > >> > > > > >
> >> > > > > > > > >> > > > >
> >> > > > > > > > >> > >
> >> > > > > > > > >> >
> >> > > > > > > > >>
> >> > > > > > >
> >> > > > > >
> >> > > > >
> >> > >
> >> >
> >>
> https://urldefense.com/v3/__https://docs.google.com/document/d/100h7c4damrUzVuquYbBHM0EvA4LSWuW2IT2dN_7nYVA/edit?tab=t.ezk0rgdx0c6m__;!!Iz9xO38YGHZK!4rnlVIqBQQQ6xpGe1PMV58edOS_cGrKoqdfhw4qSp_JQCavvEjaxXk0OwGPnxyEcO60AzKQK_XDtp7H3DdkFXcrX83IZkm0$
> >> > > > > > > > >> > > > > > >> > >
> >> > > > > > > > >> > > > > > >> > > A few highlights:
> >> > > > > > > > >> > > > > > >> > >
> >> > > > > > > > >> > > > > > >> > >    - No objection to keeping the
> default
> >> > > metrics
> >> > > > > > > behavior
> >> > > > > > > > >> > > small:
> >> > > > > > > > >> > > > > > >> > >    no-op/log-only is enough as the
> >> built-in
> >> > > > > > default.
> >> > > > > > > > >> > > > > > >> > >    - Durable storage, event
> forwarding,
> >> > > external
> >> > > > > > > queues,
> >> > > > > > > > >> > > > > dashboards,
> >> > > > > > > > >> > > > > > >> and
> >> > > > > > > > >> > > > > > >> > >    custom filtering should be
> >> > implementation
> >> > > > > > choices
> >> > > > > > > > >> behind
> >> > > > > > > > >> > > the
> >> > > > > > > > >> > > > > > >> metrics
> >> > > > > > > > >> > > > > > >> > >    reporting path.
> >> > > > > > > > >> > > > > > >> > >    - For PR #4115, the REST layer
> should
> >> > > > > > > > >> > resolve/authz/accept
> >> > > > > > > > >> > > the
> >> > > > > > > > >> > > > > > >> report,
> >> > > > > > > > >> > > > > > >> > >    then delegate to the selected
> metrics
> >> > > > > > > implementation.
> >> > > > > > > > >> It
> >> > > > > > > > >> > > should
> >> > > > > > > > >> > > > > > not
> >> > > > > > > > >> > > > > > >> > own
> >> > > > > > > > >> > > > > > >> > >    durable storage semantics.
> >> > > > > > > > >> > > > > > >> > >    - The rough boundary we discussed
> is:
> >> > spec
> >> > > > > owns
> >> > > > > > > the
> >> > > > > > > > >> wire
> >> > > > > > > > >> > > > > > contract,
> >> > > > > > > > >> > > > > > >> > >    runtime owns framework wiring,
> >> > > API/contract
> >> > > > > > > modules
> >> > > > > > > > >> own
> >> > > > > > > > >> > > > > > >> > provider-facing
> >> > > > > > > > >> > > > > > >> > >    contracts, and extensions own
> >> > replaceable
> >> > > > > > > > >> > implementations.
> >> > > > > > > > >> > > > > > >> > >    - OpenLineage has a similar
> >> path-naming
> >> > > > > > question.
> >> > > > > > > We
> >> > > > > > > > >> > should
> >> > > > > > > > >> > > > > avoid
> >> > > > > > > > >> > > > > > >> > >    occupying a generic lineage path
> too
> >> > > early if
> >> > > > > we
> >> > > > > > > want
> >> > > > > > > > >> > room
> >> > > > > > > > >> > > for
> >> > > > > > > > >> > > > > > >> other
> >> > > > > > > > >> > > > > > >> > >    lineage systems later.
> >> > > > > > > > >> > > > > > >> > >
> >> > > > > > > > >> > > > > > >> > > I believe the boundary rules previewed
> >> in
> >> > > sync
> >> > > > > > > meeting
> >> > > > > > > > >> are
> >> > > > > > > > >> > > useful
> >> > > > > > > > >> > > > > > for
> >> > > > > > > > >> > > > > > >> > > reviewing PR #4115, but it is not
> >> > > > > > > > >> > > > > > >> > > a final module-layout proposal.
> >> > > > > > > > >> > > > > > >> > >
> >> > > > > > > > >> > > > > > >> > > For metrics specifically, I think the
> >> > target
> >> > > > > > remains:
> >> > > > > > > > >> > > > > > >> > >    runtime
> >> > > > > > > > >> > > > > > >> > >      -> metrics reporting/emitting
> >> contract
> >> > > > > > > > >> > > > > > >> > >      -> default implementation
> >> > > > > > > > >> > > > > > >> > >           -> lower implementation
> >> details,
> >> > if
> >> > > > > > needed
> >> > > > > > > > >> > > > > > >> > >
> >> > > > > > > > >> > > > > > >> > > That keeps the default simple, while
> >> > allowing
> >> > > > > > durable
> >> > > > > > > > >> JDBC,
> >> > > > > > > > >> > > > > > >> event-backed,
> >> > > > > > > > >> > > > > > >> > > or external-queue implementations to
> be
> >> > added
> >> > > > > > > separately.
> >> > > > > > > > >> > > > > > >> > >
> >> > > > > > > > >> > > > > > >> > > We also touched on the split between
> >> > Iceberg
> >> > > > > > metrics
> >> > > > > > > > >> > emitting
> >> > > > > > > > >> > > and
> >> > > > > > > > >> > > > > > >> > > Polaris-owned metrics querying. The
> >> > querying
> >> > > side
> >> > > > > > can
> >> > > > > > > > >> keep
> >> > > > > > > > >> > > > > evolving,
> >> > > > > > > > >> > > > > > >> > > especially for dashboard or
> >> generic-table
> >> > use
> >> > > > > > cases,
> >> > > > > > > but
> >> > > > > > > > >> I
> >> > > > > > > > >> > do
> >> > > > > > > > >> > > not
> >> > > > > > > > >> > > > > > >> think
> >> > > > > > > > >> > > > > > >> > > that needs to block the intake path in
> >> this
> >> > > PR.
> >> > > > > > > > >> > > > > > >> > >
> >> > > > > > > > >> > > > > > >> > > Thanks,
> >> > > > > > > > >> > > > > > >> > > -ej
> >> > > > > > > > >> > > > > > >> > >
> >> > > > > > > > >> > > > > > >> > > On Tue, Jun 16, 2026 at 2:17 PM Yufei
> >> Gu <
> >> > > > > > > > >> > > flyrain000@gmail.com>
> >> > > > > > > > >> > > > > > >> wrote:
> >> > > > > > > > >> > > > > > >> > >
> >> > > > > > > > >> > > > > > >> > > > Thanks for chiming in, EJ. Agreed
> that
> >> > the
> >> > > > > > default
> >> > > > > > > > >> battery
> >> > > > > > > > >> > > > > should
> >> > > > > > > > >> > > > > > >> stay
> >> > > > > > > > >> > > > > > >> > > > small. I'm leaning toward not
> >> including
> >> > > metrics
> >> > > > > > > > >> > persistence
> >> > > > > > > > >> > > in
> >> > > > > > > > >> > > > > the
> >> > > > > > > > >> > > > > > >> > > default
> >> > > > > > > > >> > > > > > >> > > > battery.
> >> > > > > > > > >> > > > > > >> > > >
> >> > > > > > > > >> > > > > > >> > > > Yufei
> >> > > > > > > > >> > > > > > >> > > >
> >> > > > > > > > >> > > > > > >> > > >
> >> > > > > > > > >> > > > > > >> > > > On Thu, Jun 11, 2026 at 8:30 PM EJ
> >> Wang <
> >> > > > > > > > >> > > > > > >> > ej.wang.dev.interest@gmail.com>
> >> > > > > > > > >> > > > > > >> > > > wrote:
> >> > > > > > > > >> > > > > > >> > > >
> >> > > > > > > > >> > > > > > >> > > > > Hi Yufei,
> >> > > > > > > > >> > > > > > >> > > > >
> >> > > > > > > > >> > > > > > >> > > > > Thanks for connecting this back to
> >> the
> >> > > REST
> >> > > > > > > endpoint
> >> > > > > > > > >> > > > > proposal. I
> >> > > > > > > > >> > > > > > >> > > replied
> >> > > > > > > > >> > > > > > >> > > > > with the fuller version on the
> >> > > > > event-forwarding
> >> > > > > > > > >> thread,
> >> > > > > > > > >> > > but
> >> > > > > > > > >> > > > > > >> wanted to
> >> > > > > > > > >> > > > > > >> > > add
> >> > > > > > > > >> > > > > > >> > > > > the shorter version here too.
> >> > > > > > > > >> > > > > > >> > > > >
> >> > > > > > > > >> > > > > > >> > > > > I agree that metrics
> >> reporting/emitting
> >> > > is
> >> > > > > the
> >> > > > > > > right
> >> > > > > > > > >> > > > > conceptual
> >> > > > > > > > >> > > > > > >> > > boundary,
> >> > > > > > > > >> > > > > > >> > > > > and I also think the
> event/listener
> >> > path
> >> > > is a
> >> > > > > > > good
> >> > > > > > > > >> > > > > > implementation
> >> > > > > > > > >> > > > > > >> > > > direction
> >> > > > > > > > >> > > > > > >> > > > > to explore. The distinction I want
> >> to
> >> > > keep
> >> > > > > > clear
> >> > > > > > > is
> >> > > > > > > > >> > > default
> >> > > > > > > > >> > > > > > >> battery
> >> > > > > > > > >> > > > > > >> > vs
> >> > > > > > > > >> > > > > > >> > > > > extension implementation.
> >> > > > > > > > >> > > > > > >> > > > >
> >> > > > > > > > >> > > > > > >> > > > > For the REST/API side, I would
> keep
> >> the
> >> > > > > > semantics
> >> > > > > > > > >> > narrow:
> >> > > > > > > > >> > > > > > Polaris
> >> > > > > > > > >> > > > > > >> > > > resolves
> >> > > > > > > > >> > > > > > >> > > > > and authorizes the table, accepts
> >> the
> >> > > Iceberg
> >> > > > > > > > >> > scan/commit
> >> > > > > > > > >> > > > > > metrics
> >> > > > > > > > >> > > > > > >> > > report
> >> > > > > > > > >> > > > > > >> > > > > into the configured
> >> reporting/emitting
> >> > > path,
> >> > > > > > and
> >> > > > > > > > >> returns
> >> > > > > > > > >> > > 204.
> >> > > > > > > > >> > > > > > That
> >> > > > > > > > >> > > > > > >> > 204
> >> > > > > > > > >> > > > > > >> > > > > should mean Polaris accepted the
> >> report
> >> > > into
> >> > > > > > the
> >> > > > > > > > >> > ingestion
> >> > > > > > > > >> > > > > path;
> >> > > > > > > > >> > > > > > >> it
> >> > > > > > > > >> > > > > > >> > > > should
> >> > > > > > > > >> > > > > > >> > > > > not imply durable storage.
> >> > > > > > > > >> > > > > > >> > > > >
> >> > > > > > > > >> > > > > > >> > > > > Durable storage, event forwarding,
> >> > > external
> >> > > > > > > telemetry
> >> > > > > > > > >> > > routing,
> >> > > > > > > > >> > > > > > >> > > filtering,
> >> > > > > > > > >> > > > > > >> > > > > and retention should sit behind
> that
> >> > > > > > > > >> reporting/emitting
> >> > > > > > > > >> > > > > boundary
> >> > > > > > > > >> > > > > > >> as
> >> > > > > > > > >> > > > > > >> > > > > implementation-layer behavior. A
> >> > durable
> >> > > JDBC
> >> > > > > > > path
> >> > > > > > > > >> can
> >> > > > > > > > >> > be
> >> > > > > > > > >> > > one
> >> > > > > > > > >> > > > > > >> named
> >> > > > > > > > >> > > > > > >> > > > > extension implementation. An
> >> > > event/listener
> >> > > > > > > > >> forwarding
> >> > > > > > > > >> > > path
> >> > > > > > > > >> > > > > can
> >> > > > > > > > >> > > > > > be
> >> > > > > > > > >> > > > > > >> > > > another
> >> > > > > > > > >> > > > > > >> > > > > named extension implementation.
> >> > > > > > > > >> > > > > > >> > > > >
> >> > > > > > > > >> > > > > > >> > > > > So I am +1 on the event-forwarding
> >> idea
> >> > > as a
> >> > > > > > > > >> non-default
> >> > > > > > > > >> > > > > > extension
> >> > > > > > > > >> > > > > > >> > > > > implementation. I just would not
> >> make
> >> > it
> >> > > the
> >> > > > > > > default
> >> > > > > > > > >> > > battery
> >> > > > > > > > >> > > > > or
> >> > > > > > > > >> > > > > > >> the
> >> > > > > > > > >> > > > > > >> > > core
> >> > > > > > > > >> > > > > > >> > > > > API shape. The default battery can
> >> stay
> >> > > small
> >> > > > > > and
> >> > > > > > > > >> safe,
> >> > > > > > > > >> > > while
> >> > > > > > > > >> > > > > > >> > > deployments
> >> > > > > > > > >> > > > > > >> > > > > that need forwarding, dashboards,
> or
> >> > > durable
> >> > > > > > > storage
> >> > > > > > > > >> can
> >> > > > > > > > >> > > opt
> >> > > > > > > > >> > > > > > into
> >> > > > > > > > >> > > > > > >> the
> >> > > > > > > > >> > > > > > >> > > > > implementation that matches their
> >> > > operational
> >> > > > > > > model.
> >> > > > > > > > >> > > > > > >> > > > >
> >> > > > > > > > >> > > > > > >> > > > > Thanks,
> >> > > > > > > > >> > > > > > >> > > > > -ej
> >> > > > > > > > >> > > > > > >> > > > >
> >> > > > > > > > >> > > > > > >> > > > > On Wed, Jun 10, 2026 at 11:43 AM
> >> Yufei
> >> > > Gu <
> >> > > > > > > > >> > > > > flyrain000@gmail.com
> >> > > > > > > > >> > > > > > >
> >> > > > > > > > >> > > > > > >> > > wrote:
> >> > > > > > > > >> > > > > > >> > > > >
> >> > > > > > > > >> > > > > > >> > > > > > Thanks EJ. I agree that the
> >> > > > > > reporting/emitting
> >> > > > > > > > >> > boundary
> >> > > > > > > > >> > > > > feels
> >> > > > > > > > >> > > > > > >> like
> >> > > > > > > > >> > > > > > >> > > the
> >> > > > > > > > >> > > > > > >> > > > > > right SPI boundary, but I wonder
> >> if
> >> > we
> >> > > can
> >> > > > > > > simplify
> >> > > > > > > > >> > this
> >> > > > > > > > >> > > > > even
> >> > > > > > > > >> > > > > > >> > > further.
> >> > > > > > > > >> > > > > > >> > > > > >
> >> > > > > > > > >> > > > > > >> > > > > > Iceberg scan and commit metrics
> >> look
> >> > > very
> >> > > > > > > similar
> >> > > > > > > > >> to
> >> > > > > > > > >> > > events
> >> > > > > > > > >> > > > > to
> >> > > > > > > > >> > > > > > >> me.
> >> > > > > > > > >> > > > > > >> > > They
> >> > > > > > > > >> > > > > > >> > > > > are
> >> > > > > > > > >> > > > > > >> > > > > > append only, asynchronously
> >> consumed,
> >> > > often
> >> > > > > > > > >> forwarded
> >> > > > > > > > >> > to
> >> > > > > > > > >> > > > > > >> external
> >> > > > > > > > >> > > > > > >> > > > > systems,
> >> > > > > > > > >> > > > > > >> > > > > > and have similar retention and
> >> > cleanup
> >> > > > > > > > >> requirements.
> >> > > > > > > > >> > > More
> >> > > > > > > > >> > > > > > >> details
> >> > > > > > > > >> > > > > > >> > can
> >> > > > > > > > >> > > > > > >> > > > be
> >> > > > > > > > >> > > > > > >> > > > > > found in this thread [1]. In
> that
> >> > case,
> >> > > > > > > Polaris may
> >> > > > > > > > >> > only
> >> > > > > > > > >> > > > > need
> >> > > > > > > > >> > > > > > to
> >> > > > > > > > >> > > > > > >> > > accept
> >> > > > > > > > >> > > > > > >> > > > > the
> >> > > > > > > > >> > > > > > >> > > > > > report and emit an event through
> >> the
> >> > > > > existing
> >> > > > > > > event
> >> > > > > > > > >> > > > > framework.
> >> > > > > > > > >> > > > > > >> > > > Filtering,
> >> > > > > > > > >> > > > > > >> > > > > > async delivery, custom sinks,
> and
> >> > > retention
> >> > > > > > > > >> mechanisms
> >> > > > > > > > >> > > could
> >> > > > > > > > >> > > > > > >> then
> >> > > > > > > > >> > > > > > >> > be
> >> > > > > > > > >> > > > > > >> > > > > shared
> >> > > > > > > > >> > > > > > >> > > > > > instead of introducing a
> separate
> >> > > metrics
> >> > > > > > > specific
> >> > > > > > > > >> > > extension
> >> > > > > > > > >> > > > > > >> layer
> >> > > > > > > > >> > > > > > >> > > and
> >> > > > > > > > >> > > > > > >> > > > > SPI.
> >> > > > > > > > >> > > > > > >> > > > > >
> >> > > > > > > > >> > > > > > >> > > > > > The main requirement I still see
> >> is
> >> > > > > querying
> >> > > > > > > > >> metrics
> >> > > > > > > > >> > > through
> >> > > > > > > > >> > > > > > >> > Polaris.
> >> > > > > > > > >> > > > > > >> > > > If
> >> > > > > > > > >> > > > > > >> > > > > we
> >> > > > > > > > >> > > > > > >> > > > > > want that battery included
> >> > experience,
> >> > > we
> >> > > > > > could
> >> > > > > > > > >> work
> >> > > > > > > > >> > on
> >> > > > > > > > >> > > a
> >> > > > > > > > >> > > > > > >> > persistence
> >> > > > > > > > >> > > > > > >> > > > > > listener implementation that
> >> > > selectively
> >> > > > > > saves
> >> > > > > > > > >> metrics
> >> > > > > > > > >> > > and
> >> > > > > > > > >> > > > > > >> expose
> >> > > > > > > > >> > > > > > >> > it
> >> > > > > > > > >> > > > > > >> > > > via
> >> > > > > > > > >> > > > > > >> > > > > > the IRC event endpoint which
> seems
> >> > > pretty
> >> > > > > > > close in
> >> > > > > > > > >> > > Iceberg
> >> > > > > > > > >> > > > > > >> > > > > > community. Another option I like
> >> more
> >> > > is to
> >> > > > > > > allow
> >> > > > > > > > >> > users
> >> > > > > > > > >> > > to
> >> > > > > > > > >> > > > > > >> develop
> >> > > > > > > > >> > > > > > >> > > > their
> >> > > > > > > > >> > > > > > >> > > > > > own listener, so they can
> persist
> >> > > > > scan/commit
> >> > > > > > > > >> metrics
> >> > > > > > > > >> > > to any
> >> > > > > > > > >> > > > > > >> > systems
> >> > > > > > > > >> > > > > > >> > > > they
> >> > > > > > > > >> > > > > > >> > > > > > prefer. In general, that feels
> >> like
> >> > an
> >> > > > > > > > >> implementation
> >> > > > > > > > >> > > choice
> >> > > > > > > > >> > > > > > on
> >> > > > > > > > >> > > > > > >> top
> >> > > > > > > > >> > > > > > >> > > of
> >> > > > > > > > >> > > > > > >> > > > > the
> >> > > > > > > > >> > > > > > >> > > > > > event framework rather than
> >> something
> >> > > that
> >> > > > > > > needs to
> >> > > > > > > > >> > > shape
> >> > > > > > > > >> > > > > the
> >> > > > > > > > >> > > > > > >> core
> >> > > > > > > > >> > > > > > >> > > > > > architecture.
> >> > > > > > > > >> > > > > > >> > > > > >
> >> > > > > > > > >> > > > > > >> > > > > > 1.
> >> > > > > > > > >> > > > > > >> >
> >> > > > > > > > >> > >
> >> > > > > > > > >> >
> >> > > > > > > > >>
> >> > > > > > >
> >> > > > > >
> >> > > > >
> >> > >
> >> >
> >>
> https://urldefense.com/v3/__https://lists.apache.org/thread/x9j8nscvy8hq61tyn01mj8yp6n9of0kp__;!!Iz9xO38YGHZK!4rnlVIqBQQQ6xpGe1PMV58edOS_cGrKoqdfhw4qSp_JQCavvEjaxXk0OwGPnxyEcO60AzKQK_XDtp7H3DdkFXcrXGcGDgks$
> >> > > > > > > > >> > > > > > >> > > > > >
> >> > > > > > > > >> > > > > > >> > > > > > Thanks,
> >> > > > > > > > >> > > > > > >> > > > > > Yufei
> >> > > > > > > > >> > > > > > >> > > > > >
> >> > > > > > > > >> > > > > > >> > > > > >
> >> > > > > > > > >> > > > > > >> > > > > > On Thu, Jun 4, 2026 at 5:14 PM
> EJ
> >> > Wang
> >> > > <
> >> > > > > > > > >> > > > > > >> > > ej.wang.dev.interest@gmail.com
> >> > > > > > > > >> > > > > > >> > > > >
> >> > > > > > > > >> > > > > > >> > > > > > wrote:
> >> > > > > > > > >> > > > > > >> > > > > >
> >> > > > > > > > >> > > > > > >> > > > > > > Good point. I agree the
> existing
> >> > > > > > > > >> > > PolarisMetricsReporter is
> >> > > > > > > > >> > > > > > >> > already
> >> > > > > > > > >> > > > > > >> > > > very
> >> > > > > > > > >> > > > > > >> > > > > > > close to the right conceptual
> >> > > boundary. I
> >> > > > > > am
> >> > > > > > > not
> >> > > > > > > > >> > > > > proposing a
> >> > > > > > > > >> > > > > > >> > second
> >> > > > > > > > >> > > > > > >> > > > > > > parallel reporter concept. The
> >> > > > > distinction
> >> > > > > > I
> >> > > > > > > am
> >> > > > > > > > >> > > trying to
> >> > > > > > > > >> > > > > > >> make is
> >> > > > > > > > >> > > > > > >> > > > > > narrower:
> >> > > > > > > > >> > > > > > >> > > > > > >
> >> > > > > > > > >> > > > > > >> > > > > > > Current state:
> >> > > > > > > > >> > > > > > >> > > > > > >
> >> > > > > > > > >> > > > > > >> > > > > > >    - PolarisMetricsReporter is
> >> the
> >> > > > > existing
> >> > > > > > > > >> metrics
> >> > > > > > > > >> > > > > > reporting
> >> > > > > > > > >> > > > > > >> > hook.
> >> > > > > > > > >> > > > > > >> > > > > > >    - But it currently lives
> >> under
> >> > > > > > > > >> runtime/service.
> >> > > > > > > > >> > > > > > >> > > > > > >    - Its contract is
> documented
> >> > > around
> >> > > > > > > > >> Quarkus/CDI
> >> > > > > > > > >> > > > > discovery
> >> > > > > > > > >> > > > > > >> and
> >> > > > > > > > >> > > > > > >> > > > > > > @Identifier
> >> > > > > > > > >> > > > > > >> > > > > > >    selection.
> >> > > > > > > > >> > > > > > >> > > > > > >    - The default
> implementation
> >> is
> >> > > > > > log-only.
> >> > > > > > > > >> > > > > > >> > > > > > >    - The durable
> implementation
> >> > > writes
> >> > > > > > > through
> >> > > > > > > > >> > > > > > >> > > PolarisMetricsManager
> >> > > > > > > > >> > > > > > >> > > > > and
> >> > > > > > > > >> > > > > > >> > > > > > >    MetricsPersistence.
> >> > > > > > > > >> > > > > > >> > > > > > >    - MetricsPersistence is
> >> > currently
> >> > > > > > > inherited by
> >> > > > > > > > >> > > > > > >> > BasePersistence.
> >> > > > > > > > >> > > > > > >> > > > > > >
> >> > > > > > > > >> > > > > > >> > > > > > > My proposal:
> >> > > > > > > > >> > > > > > >> > > > > > >
> >> > > > > > > > >> > > > > > >> > > > > > >    - Keep the
> reporting/emitting
> >> > > boundary
> >> > > > > > as
> >> > > > > > > the
> >> > > > > > > > >> > SPI.
> >> > > > > > > > >> > > > > > >> > > > > > >    - Revise that contract in a
> >> > > > > > > framework-agnostic
> >> > > > > > > > >> > > optional
> >> > > > > > > > >> > > > > > >> > Iceberg
> >> > > > > > > > >> > > > > > >> > > > > > *metrics
> >> > > > > > > > >> > > > > > >> > > > > > >    extension API layer*.
> >> > > > > > > > >> > > > > > >> > > > > > >    - Keep runtime/service
> >> > > responsible for
> >> > > > > > > REST
> >> > > > > > > > >> > > ingestion,
> >> > > > > > > > >> > > > > > >> > > table/authz
> >> > > > > > > > >> > > > > > >> > > > > > >    validation, and runtime
> >> wiring.
> >> > > > > > > > >> > > > > > >> > > > > > >    - Keep the battery
> >> > implementation
> >> > > > > > > log-only in
> >> > > > > > > > >> the
> >> > > > > > > > >> > > > > metrics
> >> > > > > > > > >> > > > > > >> > > > extension
> >> > > > > > > > >> > > > > > >> > > > > > API
> >> > > > > > > > >> > > > > > >> > > > > > >    layer, *not under
> >> > > runtime/service*.
> >> > > > > > > > >> > > > > > >> > > > > > >    - Treat durable JDBC
> metrics
> >> as
> >> > > one
> >> > > > > > > > >> > implementation
> >> > > > > > > > >> > > of
> >> > > > > > > > >> > > > > the
> >> > > > > > > > >> > > > > > >> > > > reporting
> >> > > > > > > > >> > > > > > >> > > > > > SPI.
> >> > > > > > > > >> > > > > > >> > > > > > >    - Decompose the
> >> > > MetricsPersistence and
> >> > > > > > > > >> > > BasePersistence
> >> > > > > > > > >> > > > > > >> > coupling
> >> > > > > > > > >> > > > > > >> > > as
> >> > > > > > > > >> > > > > > >> > > > > > >    durable implementation
> >> detail.
> >> > > > > > > > >> > > > > > >> > > > > > >
> >> > > > > > > > >> > > > > > >> > > > > > > So the difference is not “new
> >> > > reporter
> >> > > > > > > concept vs
> >> > > > > > > > >> > old
> >> > > > > > > > >> > > > > > reporter
> >> > > > > > > > >> > > > > > >> > > > > concept.”
> >> > > > > > > > >> > > > > > >> > > > > > > but:
> >> > > > > > > > >> > > > > > >> > > > > > >
> >> > > > > > > > >> > > > > > >> > > > > > >
> >> > > > > > > > >> > > > > > >> > > > > > >    - PolarisMetricsReporter
> >> today =
> >> > > > > > > > >> runtime/service
> >> > > > > > > > >> > > > > > CDI-shaped
> >> > > > > > > > >> > > > > > >> > > hook.
> >> > > > > > > > >> > > > > > >> > > > > > >    - Target reporting SPI =
> same
> >> > > > > conceptual
> >> > > > > > > > >> > boundary,
> >> > > > > > > > >> > > but
> >> > > > > > > > >> > > > > > >> placed
> >> > > > > > > > >> > > > > > >> > in
> >> > > > > > > > >> > > > > > >> > > > the
> >> > > > > > > > >> > > > > > >> > > > > > >    right extension API layer
> and
> >> > kept
> >> > > > > > > > >> > > framework-agnostic.
> >> > > > > > > > >> > > > > > >> > > > > > >
> >> > > > > > > > >> > > > > > >> > > > > > > For the current metrics PR, I
> >> think
> >> > > the
> >> > > > > > > minimal
> >> > > > > > > > >> > > > > > architectural
> >> > > > > > > > >> > > > > > >> > > cleanup
> >> > > > > > > > >> > > > > > >> > > > > > > before merge is to *put the
> >> > > reporting SPI
> >> > > > > > > > >> contract
> >> > > > > > > > >> > > and the
> >> > > > > > > > >> > > > > > >> > battery
> >> > > > > > > > >> > > > > > >> > > > > > default
> >> > > > > > > > >> > > > > > >> > > > > > > in the right place*. The
> durable
> >> > JDBC
> >> > > > > > > > >> > implementation,
> >> > > > > > > > >> > > > > > >> > event-backed
> >> > > > > > > > >> > > > > > >> > > > > async
> >> > > > > > > > >> > > > > > >> > > > > > > implementation, external queue
> >> > > support,
> >> > > > > and
> >> > > > > > > > >> deeper
> >> > > > > > > > >> > > > > > persistence
> >> > > > > > > > >> > > > > > >> > > > cleanup
> >> > > > > > > > >> > > > > > >> > > > > > can
> >> > > > > > > > >> > > > > > >> > > > > > > continue as follow-ups, as
> long
> >> as
> >> > > we do
> >> > > > > > not
> >> > > > > > > lock
> >> > > > > > > > >> > the
> >> > > > > > > > >> > > SPI
> >> > > > > > > > >> > > > > > >> > boundary
> >> > > > > > > > >> > > > > > >> > > to
> >> > > > > > > > >> > > > > > >> > > > > > > BasePersistence or to
> >> > runtime/service
> >> > > > > > wiring.
> >> > > > > > > > >> > > > > > >> > > > > > >
> >> > > > > > > > >> > > > > > >> > > > > > > -ej
> >> > > > > > > > >> > > > > > >> > > > > > >
> >> > > > > > > > >> > > > > > >> > > > > > > On Wed, Jun 3, 2026 at 3:00 PM
> >> > Dmitri
> >> > > > > > > > >> Bourlatchkov <
> >> > > > > > > > >> > > > > > >> > > dimas@apache.org
> >> > > > > > > > >> > > > > > >> > > > >
> >> > > > > > > > >> > > > > > >> > > > > > > wrote:
> >> > > > > > > > >> > > > > > >> > > > > > >
> >> > > > > > > > >> > > > > > >> > > > > > > > Hi EJ,
> >> > > > > > > > >> > > > > > >> > > > > > > >
> >> > > > > > > > >> > > > > > >> > > > > > > > Thanks for the recap /
> >> summary!
> >> > > > > > > > >> > > > > > >> > > > > > > >
> >> > > > > > > > >> > > > > > >> > > > > > > > > Do folks agree that the
> >> stable
> >> > > SPI
> >> > > > > > > boundary
> >> > > > > > > > >> > > should be
> >> > > > > > > > >> > > > > > >> > > > > > > > metrics reporting/emitting,
> >> not
> >> > > metrics
> >> > > > > > > > >> > persistence?
> >> > > > > > > > >> > > > > > >> > > > > > > >
> >> > > > > > > > >> > > > > > >> > > > > > > > SGTM.
> >> > > > > > > > >> > > > > > >> > > > > > > >
> >> > > > > > > > >> > > > > > >> > > > > > > > > Does an optional Iceberg
> >> > metrics
> >> > > > > > > extension
> >> > > > > > > > >> API
> >> > > > > > > > >> > > layer
> >> > > > > > > > >> > > > > > sound
> >> > > > > > > > >> > > > > > >> > like
> >> > > > > > > > >> > > > > > >> > > > the
> >> > > > > > > > >> > > > > > >> > > > > > > right
> >> > > > > > > > >> > > > > > >> > > > > > > > home for this SPI?
> >> > > > > > > > >> > > > > > >> > > > > > > >
> >> > > > > > > > >> > > > > > >> > > > > > > > How is that different from
> >> > current
> >> > > > > > > > >> > > > > PolarisMetricsReporter?
> >> > > > > > > > >> > > > > > >> > > > > > > >
> >> > > > > > > > >> > > > > > >> > > > > > > > > Should the current durable
> >> > > metrics
> >> > > > > work
> >> > > > > > > be
> >> > > > > > > > >> > > reframed
> >> > > > > > > > >> > > > > as a
> >> > > > > > > > >> > > > > > >> > > durable
> >> > > > > > > > >> > > > > > >> > > > > > > > JDBC reference
> implementation
> >> of
> >> > > that
> >> > > > > > SPI?
> >> > > > > > > > >> > > > > > >> > > > > > > >
> >> > > > > > > > >> > > > > > >> > > > > > > > SGTM.
> >> > > > > > > > >> > > > > > >> > > > > > > >
> >> > > > > > > > >> > > > > > >> > > > > > > > > What is the smallest PR
> >> > sequence
> >> > > to
> >> > > > > get
> >> > > > > > > there
> >> > > > > > > > >> > > without
> >> > > > > > > > >> > > > > > >> > blocking
> >> > > > > > > > >> > > > > > >> > > > > > > > the current metrics work
> >> > > unnecessarily?
> >> > > > > > > > >> > > > > > >> > > > > > > >
> >> > > > > > > > >> > > > > > >> > > > > > > > Let's get
> >> > > > > > > > >> > >
> >> > > > > > > > >> >
> >> > > > > > > > >>
> >> > > > > > >
> >> > > > > >
> >> > > > >
> >> > >
> >> >
> >>
> https://urldefense.com/v3/__https://github.com/apache/polaris/pull/4397__;!!Iz9xO38YGHZK!4rnlVIqBQQQ6xpGe1PMV58edOS_cGrKoqdfhw4qSp_JQCavvEjaxXk0OwGPnxyEcO60AzKQK_XDtp7H3DdkFXcrXuu5jefU$
> >> > > > > > > > >> > > > > > >> merged
> >> > > > > > > > >> > > > > > >> > > > first.
> >> > > > > > > > >> > > > > > >> > > > > > > >
> >> > > > > > > > >> > > > > > >> > > > > > > > Then, I'd propose isolating
> >> the
> >> > > Metrics
> >> > > > > > > schema
> >> > > > > > > > >> > from
> >> > > > > > > > >> > > the
> >> > > > > > > > >> > > > > > >> > MetaStore
> >> > > > > > > > >> > > > > > >> > > > > > schema.
> >> > > > > > > > >> > > > > > >> > > > > > > > This will probably have some
> >> > ripple
> >> > > > > > effect
> >> > > > > > > into
> >> > > > > > > > >> > the
> >> > > > > > > > >> > > > > > >> bootstrap
> >> > > > > > > > >> > > > > > >> > > > > > workflows,
> >> > > > > > > > >> > > > > > >> > > > > > > so
> >> > > > > > > > >> > > > > > >> > > > > > > > it's not a trivial change.
> >> > > > > > > > >> > > > > > >> > > > > > > >
> >> > > > > > > > >> > > > > > >> > > > > > > > Then, let's reassess.
> >> > > > > > > > >> > > > > > >> > > > > > > >
> >> > > > > > > > >> > > > > > >> > > > > > > > Cheers,
> >> > > > > > > > >> > > > > > >> > > > > > > > Dmitri.
> >> > > > > > > > >> > > > > > >> > > > > > > >
> >> > > > > > > > >> > > > > > >> > > > > > > > On Wed, Jun 3, 2026 at
> >> 5:40 PM EJ
> >> > > Wang
> >> > > > > <
> >> > > > > > > > >> > > > > > >> > > > > ej.wang.dev.interest@gmail.com
> >> > > > > > > > >> > > > > > >> > > > > > >
> >> > > > > > > > >> > > > > > >> > > > > > > > wrote:
> >> > > > > > > > >> > > > > > >> > > > > > > >
> >> > > > > > > > >> > > > > > >> > > > > > > > > Hi folks,
> >> > > > > > > > >> > > > > > >> > > > > > > > >
> >> > > > > > > > >> > > > > > >> > > > > > > > > Thanks again for the
> >> discussion
> >> > > > > today.
> >> > > > > > I
> >> > > > > > > > >> updated
> >> > > > > > > > >> > > the
> >> > > > > > > > >> > > > > > sync
> >> > > > > > > > >> > > > > > >> doc
> >> > > > > > > > >> > > > > > >> > > > with
> >> > > > > > > > >> > > > > > >> > > > > > > notes
> >> > > > > > > > >> > > > > > >> > > > > > > > > from the third metrics
> >> > > architecture
> >> > > > > > sync:
> >> > > > > > > > >> > > > > > >> > > > > > > > >
> >> > > > > > > > >> > > > > > >> > > > > > > > >
> >> > > > > > > > >> > > > > > >> > > > > > > > >
> >> > > > > > > > >> > > > > > >> > > > > > > >
> >> > > > > > > > >> > > > > > >> > > > > > >
> >> > > > > > > > >> > > > > > >> > > > > >
> >> > > > > > > > >> > > > > > >> > > > >
> >> > > > > > > > >> > > > > > >> > > >
> >> > > > > > > > >> > > > > > >> > >
> >> > > > > > > > >> > > > > > >> >
> >> > > > > > > > >> > > > > > >>
> >> > > > > > > > >> > > > > >
> >> > > > > > > > >> > > > >
> >> > > > > > > > >> > >
> >> > > > > > > > >> >
> >> > > > > > > > >>
> >> > > > > > >
> >> > > > > >
> >> > > > >
> >> > >
> >> >
> >>
> https://urldefense.com/v3/__https://docs.google.com/document/d/100h7c4damrUzVuquYbBHM0EvA4LSWuW2IT2dN_7nYVA/edit?tab=t.k96s2xyqr5u1__;!!Iz9xO38YGHZK!4rnlVIqBQQQ6xpGe1PMV58edOS_cGrKoqdfhw4qSp_JQCavvEjaxXk0OwGPnxyEcO60AzKQK_XDtp7H3DdkFXcrXrqCL-QE$
> >> > > > > > > > >> > > > > > >> > > > > > > > >
> >> > > > > > > > >> > > > > > >> > > > > > > > > A few highlights from
> >> today’s
> >> > > > > > discussion:
> >> > > > > > > > >> > > > > > >> > > > > > > > >
> >> > > > > > > > >> > > > > > >> > > > > > > > >    - We clarified that
> >> Iceberg
> >> > > > > metrics
> >> > > > > > > > >> reporting
> >> > > > > > > > >> > > can
> >> > > > > > > > >> > > > > be
> >> > > > > > > > >> > > > > > >> > > > interpreted
> >> > > > > > > > >> > > > > > >> > > > > > > > either
> >> > > > > > > > >> > > > > > >> > > > > > > > >    as sync or async
> handling
> >> > from
> >> > > > > > > Polaris’s
> >> > > > > > > > >> > > > > perspective,
> >> > > > > > > > >> > > > > > >> so
> >> > > > > > > > >> > > > > > >> > > > Polaris
> >> > > > > > > > >> > > > > > >> > > > > > > > should
> >> > > > > > > > >> > > > > > >> > > > > > > > >    stay flexible as a
> >> platform
> >> > > > > instead
> >> > > > > > of
> >> > > > > > > > >> baking
> >> > > > > > > > >> > > one
> >> > > > > > > > >> > > > > > >> handling
> >> > > > > > > > >> > > > > > >> > > > model
> >> > > > > > > > >> > > > > > >> > > > > > > into
> >> > > > > > > > >> > > > > > >> > > > > > > > > the
> >> > > > > > > > >> > > > > > >> > > > > > > > >    REST API behavior.
> >> > > > > > > > >> > > > > > >> > > > > > > > >    - We aligned that the
> >> > built-in
> >> > > > > > > (battery)
> >> > > > > > > > >> > > behavior
> >> > > > > > > > >> > > > > for
> >> > > > > > > > >> > > > > > >> > > metrics
> >> > > > > > > > >> > > > > > >> > > > > > > emitting
> >> > > > > > > > >> > > > > > >> > > > > > > > >    can stay simple:
> >> > > no-op/log-only is
> >> > > > > > > enough
> >> > > > > > > > >> as
> >> > > > > > > > >> > > the
> >> > > > > > > > >> > > > > > >> default.
> >> > > > > > > > >> > > > > > >> > > > > > > > >    - Durable metrics
> >> > persistence
> >> > > > > should
> >> > > > > > > be
> >> > > > > > > > >> > > treated as
> >> > > > > > > > >> > > > > an
> >> > > > > > > > >> > > > > > >> > > > > > implementation
> >> > > > > > > > >> > > > > > >> > > > > > > > of
> >> > > > > > > > >> > > > > > >> > > > > > > > >    the metrics reporting
> >> path,
> >> > > not as
> >> > > > > > the
> >> > > > > > > > >> core
> >> > > > > > > > >> > SPI
> >> > > > > > > > >> > > > > > >> boundary.
> >> > > > > > > > >> > > > > > >> > > > > > > > >    - The existing durable
> >> > metrics
> >> > > > > work
> >> > > > > > > can be
> >> > > > > > > > >> > > reviewed
> >> > > > > > > > >> > > > > > as
> >> > > > > > > > >> > > > > > >> a
> >> > > > > > > > >> > > > > > >> > > > > reference
> >> > > > > > > > >> > > > > > >> > > > > > > > >    implementation of the
> >> > > reporting
> >> > > > > SPI,
> >> > > > > > > with
> >> > > > > > > > >> > > > > > >> > > persistence-related
> >> > > > > > > > >> > > > > > >> > > > > > logic
> >> > > > > > > > >> > > > > > >> > > > > > > > kept
> >> > > > > > > > >> > > > > > >> > > > > > > > >    self-contained under a
> >> > metrics
> >> > > > > > durable
> >> > > > > > > > >> > > > > implementation
> >> > > > > > > > >> > > > > > >> > module
> >> > > > > > > > >> > > > > > >> > > > > > rather
> >> > > > > > > > >> > > > > > >> > > > > > > > than
> >> > > > > > > > >> > > > > > >> > > > > > > > >    scattered through core
> >> > entity
> >> > > > > > > persistence.
> >> > > > > > > > >> > > > > > >> > > > > > > > >    - Dashboard/insights
> >> > remains a
> >> > > > > real
> >> > > > > > > use
> >> > > > > > > > >> case,
> >> > > > > > > > >> > > but
> >> > > > > > > > >> > > > > we
> >> > > > > > > > >> > > > > > >> > agreed
> >> > > > > > > > >> > > > > > >> > > to
> >> > > > > > > > >> > > > > > >> > > > > > keep
> >> > > > > > > > >> > > > > > >> > > > > > > it
> >> > > > > > > > >> > > > > > >> > > > > > > > >    separate from the core
> >> > metrics
> >> > > > > > intake
> >> > > > > > > > >> > > discussion
> >> > > > > > > > >> > > > > for
> >> > > > > > > > >> > > > > > >> now.
> >> > > > > > > > >> > > > > > >> > > > > > > > >
> >> > > > > > > > >> > > > > > >> > > > > > > > > I also did a quick source
> >> check
> >> > > after
> >> > > > > > the
> >> > > > > > > > >> > meeting
> >> > > > > > > > >> > > to
> >> > > > > > > > >> > > > > > make
> >> > > > > > > > >> > > > > > >> > sure
> >> > > > > > > > >> > > > > > >> > > we
> >> > > > > > > > >> > > > > > >> > > > > are
> >> > > > > > > > >> > > > > > >> > > > > > > > > describing the current
> state
> >> > > > > > accurately.
> >> > > > > > > > >> > > > > > >> > > > > > > > >
> >> > > > > > > > >> > > > > > >> > > > > > > > > Current state:
> >> > > > > > > > >> > > > > > >> > > > > > > > >
> >> > > > > > > > >> > > > > > >> > > > > > > > >    - Polaris already has a
> >> > > metrics
> >> > > > > > > reporting
> >> > > > > > > > >> > hook:
> >> > > > > > > > >> > > > > > >> > > > > > > > PolarisMetricsReporter.
> >> > > > > > > > >> > > > > > >> > > > > > > > >    - The default
> >> implementation
> >> > > is
> >> > > > > > > > >> > > > > > DefaultMetricsReporter,
> >> > > > > > > > >> > > > > > >> > > > selected
> >> > > > > > > > >> > > > > > >> > > > > > by
> >> > > > > > > > >> > > > > > >> > > > > > > > >
> >> > > > > > > > >> > > polaris.iceberg-metrics.reporting.type=default. It
> >> > > > > > > > >> > > > > is
> >> > > > > > > > >> > > > > > >> > > > log-only,
> >> > > > > > > > >> > > > > > >> > > > > > > > >    effectively quiet
> unless
> >> > > metrics
> >> > > > > > > logging
> >> > > > > > > > >> is
> >> > > > > > > > >> > > > > enabled.
> >> > > > > > > > >> > > > > > >> > > > > > > > >    - There is also a
> >> > > > > > > > >> PersistingMetricsReporter,
> >> > > > > > > > >> > > > > selected
> >> > > > > > > > >> > > > > > >> by
> >> > > > > > > > >> > > > > > >> > > > > > > > >
> >> > > > > > > > >> > > polaris.iceberg-metrics.reporting.type=persisting,
> >> > > > > > > > >> > > > > > >> which
> >> > > > > > > > >> > > > > > >> > > > > converts
> >> > > > > > > > >> > > > > > >> > > > > > > > >    Iceberg scan/commit
> >> reports
> >> > > into
> >> > > > > > > Polaris
> >> > > > > > > > >> > > metrics
> >> > > > > > > > >> > > > > > >> records
> >> > > > > > > > >> > > > > > >> > and
> >> > > > > > > > >> > > > > > >> > > > > > writes
> >> > > > > > > > >> > > > > > >> > > > > > > > > through
> >> > > > > > > > >> > > > > > >> > > > > > > > >    PolarisMetricsManager
> ->
> >> > > > > > > > >> MetricsPersistence.
> >> > > > > > > > >> > > > > > >> > > > > > > > >    - MetricsPersistence
> >> > currently
> >> > > > > > exists
> >> > > > > > > in
> >> > > > > > > > >> the
> >> > > > > > > > >> > > > > > >> persistence
> >> > > > > > > > >> > > > > > >> > > layer
> >> > > > > > > > >> > > > > > >> > > > > and
> >> > > > > > > > >> > > > > > >> > > > > > > is
> >> > > > > > > > >> > > > > > >> > > > > > > > >    inherited by
> >> > BasePersistence.
> >> > > > > > > > >> > > > > > >> > > > > > > > >
> >> > > > > > > > >> > > > > > >> > > > > > > > > My read from the
> discussion
> >> is
> >> > > that
> >> > > > > the
> >> > > > > > > > >> target
> >> > > > > > > > >> > > > > boundary
> >> > > > > > > > >> > > > > > >> > should
> >> > > > > > > > >> > > > > > >> > > > not
> >> > > > > > > > >> > > > > > >> > > > > be
> >> > > > > > > > >> > > > > > >> > > > > > > > > MetricsPersistence as
> >> inherited
> >> > > by
> >> > > > > > > > >> > > BasePersistence.
> >> > > > > > > > >> > > > > That
> >> > > > > > > > >> > > > > > >> path
> >> > > > > > > > >> > > > > > >> > > > > should
> >> > > > > > > > >> > > > > > >> > > > > > be
> >> > > > > > > > >> > > > > > >> > > > > > > > > decomposed as durable
> >> > > implementation
> >> > > > > > > detail
> >> > > > > > > > >> > (taken
> >> > > > > > > > >> > > > > care
> >> > > > > > > > >> > > > > > >> of by
> >> > > > > > > > >> > > > > > >> > > > > > > > >
> >> > > > > > > > >> >
> >> > > > > > > > >>
> >> > > > > > >
> >> > > > > >
> >> > > > >
> >> > >
> >> >
> >>
> https://urldefense.com/v3/__https://github.com/apache/polaris/pull/4397__;!!Iz9xO38YGHZK!4rnlVIqBQQQ6xpGe1PMV58edOS_cGrKoqdfhw4qSp_JQCavvEjaxXk0OwGPnxyEcO60AzKQK_XDtp7H3DdkFXcrXuu5jefU$
> >> > > > > > > > >> ).
> >> > > > > > > > >> > The
> >> > > > > > > > >> > > > > > stable
> >> > > > > > > > >> > > > > > >> > > > extension
> >> > > > > > > > >> > > > > > >> > > > > > > point
> >> > > > > > > > >> > > > > > >> > > > > > > > > should instead be the
> >> metrics
> >> > > > > > > > >> reporting/emitting
> >> > > > > > > > >> > > > > > boundary.
> >> > > > > > > > >> > > > > > >> > > > > > > > >
> >> > > > > > > > >> > > > > > >> > > > > > > > > Concretely, I think the
> next
> >> > > proposal
> >> > > > > > > should
> >> > > > > > > > >> be
> >> > > > > > > > >> > > shaped
> >> > > > > > > > >> > > > > > >> like
> >> > > > > > > > >> > > > > > >> > > this:
> >> > > > > > > > >> > > > > > >> > > > > > > > >
> >> > > > > > > > >> > > > > > >> > > > > > > > >    1.
> >> > > > > > > > >> > > > > > >> > > > > > > > >
> >> > > > > > > > >> > > > > > >> > > > > > > > >    Define the proper
> metrics
> >> > > > > reporting
> >> > > > > > > SPI
> >> > > > > > > > >> at an
> >> > > > > > > > >> > > > > > optional
> >> > > > > > > > >> > > > > > >> > > Iceberg
> >> > > > > > > > >> > > > > > >> > > > > > > metrics
> >> > > > > > > > >> > > > > > >> > > > > > > > >    extension API layer.
> >> > > > > > > > >> > > > > > >> > > > > > > > >
> >> > > > > > > > >> > > > > > >> > > > > > > > >    Example direction:
> >> > > > > > > > >> > > > > > >> > > > > > > > >
> >> > > > > > > > >> > > > > > >> > > > > > > > >    public interface
> >> > > > > > > IcebergMetricsReporter {
> >> > > > > > > > >> > > > > > >> > > > > > > > >      void
> >> > > > > > > > >> > reportMetric(IcebergMetricsReportContext
> >> > > > > > > > >> > > > > > >> context,
> >> > > > > > > > >> > > > > > >> > > > > > > > > MetricsReport report);
> >> > > > > > > > >> > > > > > >> > > > > > > > >    }
> >> > > > > > > > >> > > > > > >> > > > > > > > >
> >> > > > > > > > >> > > > > > >> > > > > > > > >    The context would carry
> >> the
> >> > > small
> >> > > > > > > > >> > > Polaris-resolved
> >> > > > > > > > >> > > > > > >> > envelope:
> >> > > > > > > > >> > > > > > >> > > > > > > > >    catalog/table identity,
> >> > report
> >> > > > > type,
> >> > > > > > > > >> received
> >> > > > > > > > >> > > > > > >> timestamp,
> >> > > > > > > > >> > > > > > >> > and
> >> > > > > > > > >> > > > > > >> > > > > > > > > request/trace
> >> > > > > > > > >> > > > > > >> > > > > > > > >    context if available.
> The
> >> > raw
> >> > > > > > Iceberg
> >> > > > > > > > >> > > MetricsReport
> >> > > > > > > > >> > > > > > >> > remains
> >> > > > > > > > >> > > > > > >> > > > the
> >> > > > > > > > >> > > > > > >> > > > > > > > payload.
> >> > > > > > > > >> > > > > > >> > > > > > > > >    2.
> >> > > > > > > > >> > > > > > >> > > > > > > > >
> >> > > > > > > > >> > > > > > >> > > > > > > > >    Keep runtime/service as
> >> the
> >> > > REST
> >> > > > > > > ingestion
> >> > > > > > > > >> > and
> >> > > > > > > > >> > > > > wiring
> >> > > > > > > > >> > > > > > >> > layer.
> >> > > > > > > > >> > > > > > >> > > > > > > > >
> >> > > > > > > > >> > > > > > >> > > > > > > > >    The REST handler still
> >> > > resolves
> >> > > > > the
> >> > > > > > > table
> >> > > > > > > > >> and
> >> > > > > > > > >> > > > > > performs
> >> > > > > > > > >> > > > > > >> > authz
> >> > > > > > > > >> > > > > > >> > > > > > before
> >> > > > > > > > >> > > > > > >> > > > > > > > >    accepting the report.
> >> After
> >> > > that,
> >> > > > > it
> >> > > > > > > calls
> >> > > > > > > > >> > the
> >> > > > > > > > >> > > > > > selected
> >> > > > > > > > >> > > > > > >> > > > > reporting
> >> > > > > > > > >> > > > > > >> > > > > > > > >    implementation.
> >> > > > > > > > >> > > > > > >> > > > > > > > >    3.
> >> > > > > > > > >> > > > > > >> > > > > > > > >
> >> > > > > > > > >> > > > > > >> > > > > > > > >    Keep the battery
> default
> >> > > > > > > no-op/log-only.
> >> > > > > > > > >> > > > > > >> > > > > > > > >
> >> > > > > > > > >> > > > > > >> > > > > > > > >    This preserves an
> >> out-of-box
> >> > > safe
> >> > > > > > > default
> >> > > > > > > > >> and
> >> > > > > > > > >> > > > > avoids
> >> > > > > > > > >> > > > > > >> > > > requiring a
> >> > > > > > > > >> > > > > > >> > > > > > > > durable
> >> > > > > > > > >> > > > > > >> > > > > > > > >    metrics store for every
> >> > > Polaris
> >> > > > > > > > >> deployment.
> >> > > > > > > > >> > > > > > >> > > > > > > > >    4.
> >> > > > > > > > >> > > > > > >> > > > > > > > >
> >> > > > > > > > >> > > > > > >> > > > > > > > >    Move durable JDBC
> metrics
> >> > > into a
> >> > > > > > > > >> > self-contained
> >> > > > > > > > >> > > > > > >> > > implementation
> >> > > > > > > > >> > > > > > >> > > > > of
> >> > > > > > > > >> > > > > > >> > > > > > > the
> >> > > > > > > > >> > > > > > >> > > > > > > > >    reporting SPI.
> >> > > > > > > > >> > > > > > >> > > > > > > > >
> >> > > > > > > > >> > > > > > >> > > > > > > > >    That implementation can
> >> own
> >> > > its
> >> > > > > > > schema,
> >> > > > > > > > >> > > bootstrap,
> >> > > > > > > > >> > > > > > >> > > retention,
> >> > > > > > > > >> > > > > > >> > > > > and
> >> > > > > > > > >> > > > > > >> > > > > > > read
> >> > > > > > > > >> > > > > > >> > > > > > > > >    API support. It should
> >> not
> >> > > define
> >> > > > > > the
> >> > > > > > > core
> >> > > > > > > > >> > > > > reporting
> >> > > > > > > > >> > > > > > >> SPI
> >> > > > > > > > >> > > > > > >> > > > > boundary,
> >> > > > > > > > >> > > > > > >> > > > > > > and
> >> > > > > > > > >> > > > > > >> > > > > > > > > it
> >> > > > > > > > >> > > > > > >> > > > > > > > >    should not require
> >> metrics
> >> > > > > > > persistence to
> >> > > > > > > > >> > > remain
> >> > > > > > > > >> > > > > > >> inherited
> >> > > > > > > > >> > > > > > >> > > > from
> >> > > > > > > > >> > > > > > >> > > > > > > > >    BasePersistence.
> >> > > > > > > > >> > > > > > >> > > > > > > > >    5.
> >> > > > > > > > >> > > > > > >> > > > > > > > >
> >> > > > > > > > >> > > > > > >> > > > > > > > >    Treat
> async/event-backed
> >> > > handling
> >> > > > > as
> >> > > > > > > > >> another
> >> > > > > > > > >> > > > > > >> > implementation
> >> > > > > > > > >> > > > > > >> > > of
> >> > > > > > > > >> > > > > > >> > > > > the
> >> > > > > > > > >> > > > > > >> > > > > > > > same
> >> > > > > > > > >> > > > > > >> > > > > > > > >    reporting SPI.
> >> > > > > > > > >> > > > > > >> > > > > > > > >
> >> > > > > > > > >> > > > > > >> > > > > > > > >    For example, an
> >> event-backed
> >> > > > > > reporter
> >> > > > > > > > >> could
> >> > > > > > > > >> > > > > enqueue a
> >> > > > > > > > >> > > > > > >> > > metrics
> >> > > > > > > > >> > > > > > >> > > > > > event
> >> > > > > > > > >> > > > > > >> > > > > > > > and
> >> > > > > > > > >> > > > > > >> > > > > > > > >    let listeners handle
> >> durable
> >> > > > > storage
> >> > > > > > > or
> >> > > > > > > > >> other
> >> > > > > > > > >> > > > > sinks.
> >> > > > > > > > >> > > > > > >> If we
> >> > > > > > > > >> > > > > > >> > > > later
> >> > > > > > > > >> > > > > > >> > > > > > > need
> >> > > > > > > > >> > > > > > >> > > > > > > > a
> >> > > > > > > > >> > > > > > >> > > > > > > > >    replaceable queue
> engine,
> >> > that
> >> > > > > seems
> >> > > > > > > like
> >> > > > > > > > >> a
> >> > > > > > > > >> > > shared
> >> > > > > > > > >> > > > > > >> > > > event/metrics
> >> > > > > > > > >> > > > > > >> > > > > > > > > substrate
> >> > > > > > > > >> > > > > > >> > > > > > > > >    topic rather than a
> >> > > metrics-only
> >> > > > > > > > >> requirement.
> >> > > > > > > > >> > > > > > >> > > > > > > > >
> >> > > > > > > > >> > > > > > >> > > > > > > > > This framing lets us keep
> >> the
> >> > > REST
> >> > > > > > > metrics
> >> > > > > > > > >> > > endpoint
> >> > > > > > > > >> > > > > > >> simple,
> >> > > > > > > > >> > > > > > >> > > > > preserve
> >> > > > > > > > >> > > > > > >> > > > > > > the
> >> > > > > > > > >> > > > > > >> > > > > > > > > current default behavior,
> >> > support
> >> > > > > > durable
> >> > > > > > > > >> > metrics
> >> > > > > > > > >> > > > > users,
> >> > > > > > > > >> > > > > > >> and
> >> > > > > > > > >> > > > > > >> > > > still
> >> > > > > > > > >> > > > > > >> > > > > > > leave
> >> > > > > > > > >> > > > > > >> > > > > > > > > room for
> async/event-backed
> >> or
> >> > > > > > > > >> > > external-queue-based
> >> > > > > > > > >> > > > > > >> > > > > implementations.
> >> > > > > > > > >> > > > > > >> > > > > > > > >
> >> > > > > > > > >> > > > > > >> > > > > > > > > I think the main follow-up
> >> > > questions
> >> > > > > > are:
> >> > > > > > > > >> > > > > > >> > > > > > > > >
> >> > > > > > > > >> > > > > > >> > > > > > > > >    - Do folks agree that
> the
> >> > > stable
> >> > > > > SPI
> >> > > > > > > > >> boundary
> >> > > > > > > > >> > > > > should
> >> > > > > > > > >> > > > > > be
> >> > > > > > > > >> > > > > > >> > > > metrics
> >> > > > > > > > >> > > > > > >> > > > > > > > >    reporting/emitting, not
> >> > > metrics
> >> > > > > > > > >> persistence?
> >> > > > > > > > >> > > > > > >> > > > > > > > >    - Does an optional
> >> Iceberg
> >> > > metrics
> >> > > > > > > > >> extension
> >> > > > > > > > >> > > API
> >> > > > > > > > >> > > > > > layer
> >> > > > > > > > >> > > > > > >> > sound
> >> > > > > > > > >> > > > > > >> > > > > like
> >> > > > > > > > >> > > > > > >> > > > > > > the
> >> > > > > > > > >> > > > > > >> > > > > > > > >    right home for this
> SPI?
> >> > > > > > > > >> > > > > > >> > > > > > > > >    - Should the current
> >> durable
> >> > > > > metrics
> >> > > > > > > work
> >> > > > > > > > >> be
> >> > > > > > > > >> > > > > reframed
> >> > > > > > > > >> > > > > > >> as a
> >> > > > > > > > >> > > > > > >> > > > > durable
> >> > > > > > > > >> > > > > > >> > > > > > > > JDBC
> >> > > > > > > > >> > > > > > >> > > > > > > > >    reference
> implementation
> >> of
> >> > > that
> >> > > > > > SPI?
> >> > > > > > > > >> > > > > > >> > > > > > > > >    - What is the smallest
> PR
> >> > > sequence
> >> > > > > > to
> >> > > > > > > get
> >> > > > > > > > >> > there
> >> > > > > > > > >> > > > > > without
> >> > > > > > > > >> > > > > > >> > > > blocking
> >> > > > > > > > >> > > > > > >> > > > > > the
> >> > > > > > > > >> > > > > > >> > > > > > > > >    current metrics work
> >> > > > > unnecessarily?
> >> > > > > > > > >> > > > > > >> > > > > > > > >
> >> > > > > > > > >> > > > > > >> > > > > > > > > Thanks,
> >> > > > > > > > >> > > > > > >> > > > > > > > > -ej
> >> > > > > > > > >> > > > > > >> > > > > > > > >
> >> > > > > > > > >> > > > > > >> > > > > > > > > On Fri, May 15, 2026 at
> >> 5:16 PM
> >> > > > > Dmitri
> >> > > > > > > > >> > > Bourlatchkov <
> >> > > > > > > > >> > > > > > >> > > > > > dimas@apache.org>
> >> > > > > > > > >> > > > > > >> > > > > > > > > wrote:
> >> > > > > > > > >> > > > > > >> > > > > > > > >
> >> > > > > > > > >> > > > > > >> > > > > > > > > > Hi JB,
> >> > > > > > > > >> > > > > > >> > > > > > > > > >
> >> > > > > > > > >> > > > > > >> > > > > > > > > > Could you set up another
> >> > > meeting,
> >> > > > > > > please?
> >> > > > > > > > >> Same
> >> > > > > > > > >> > > time
> >> > > > > > > > >> > > > > on
> >> > > > > > > > >> > > > > > >> > > > Wednesday
> >> > > > > > > > >> > > > > > >> > > > > as
> >> > > > > > > > >> > > > > > >> > > > > > > > last
> >> > > > > > > > >> > > > > > >> > > > > > > > > > time... I hope it works
> >> for
> >> > > > > everyone.
> >> > > > > > > > >> > > > > > >> > > > > > > > > >
> >> > > > > > > > >> > > > > > >> > > > > > > > > > Cheers,
> >> > > > > > > > >> > > > > > >> > > > > > > > > > Dmitri.
> >> > > > > > > > >> > > > > > >> > > > > > > > > >
> >> > > > > > > > >> > > > > > >> > > > > > > > > > On Fri, May 15, 2026 at
> >> > 8:06 PM
> >> > > > > Yufei
> >> > > > > > > Gu <
> >> > > > > > > > >> > > > > > >> > > flyrain000@gmail.com
> >> > > > > > > > >> > > > > > >> > > > >
> >> > > > > > > > >> > > > > > >> > > > > > > wrote:
> >> > > > > > > > >> > > > > > >> > > > > > > > > >
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > +1 on another sync
> call
> >> > next
> >> > > > > week.
> >> > > > > > > > >> > > > > > >> > > > > > > > > > >
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > Yufei
> >> > > > > > > > >> > > > > > >> > > > > > > > > > >
> >> > > > > > > > >> > > > > > >> > > > > > > > > > >
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > On Fri, May 15, 2026
> at
> >> > > 4:52 PM
> >> > > > > > > Dmitri
> >> > > > > > > > >> > > > > Bourlatchkov
> >> > > > > > > > >> > > > > > <
> >> > > > > > > > >> > > > > > >> > > > > > > > dimas@apache.org>
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > wrote:
> >> > > > > > > > >> > > > > > >> > > > > > > > > > >
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > Hi All,
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > >
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > WDYT about another
> >> sync
> >> > > call
> >> > > > > next
> >> > > > > > > week?
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > >
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > Thanks,
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > Dmitri.
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > >
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > On Wed, May 6, 2026
> at
> >> > > 5:29 PM
> >> > > > > > > Dmitri
> >> > > > > > > > >> > > > > > Bourlatchkov <
> >> > > > > > > > >> > > > > > >> > > > > > > > dimas@apache.org
> >> > > > > > > > >> > > > > > >> > > > > > > > > >
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > wrote:
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > >
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > > Hi EJ,
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > > Thanks for the
> >> summary!
> >> > > It
> >> > > > > > covers
> >> > > > > > > > >> what
> >> > > > > > > > >> > we
> >> > > > > > > > >> > > > > > >> discussed
> >> > > > > > > > >> > > > > > >> > in
> >> > > > > > > > >> > > > > > >> > > > the
> >> > > > > > > > >> > > > > > >> > > > > > > > meeting
> >> > > > > > > > >> > > > > > >> > > > > > > > > > very
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > > well, IMHO.
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > > Looking forward to
> >> > > concrete
> >> > > > > PRs
> >> > > > > > > :)
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > > Cheers,
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > > Dmitri.
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > > On Wed, May 6,
> 2026
> >> at
> >> > > > > 5:08 PM
> >> > > > > > EJ
> >> > > > > > > > >> Wang <
> >> > > > > > > > >> > > > > > >> > > > > > > > > >
> >> > ej.wang.dev.interest@gmail.com
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > >
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > > wrote:
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> Hi folks,
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >>
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> We had a
> community
> >> > sync
> >> > > > > > earlier,
> >> > > > > > > > >> thanks
> >> > > > > > > > >> > > JB
> >> > > > > > > > >> > > > > for
> >> > > > > > > > >> > > > > > >> > > > scheduling
> >> > > > > > > > >> > > > > > >> > > > > > it.
> >> > > > > > > > >> > > > > > >> > > > > > > > > Notes
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > from
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> the first metrics
> >> > > > > architecture
> >> > > > > > > sync
> >> > > > > > > > >> > (May
> >> > > > > > > > >> > > 6,
> >> > > > > > > > >> > > > > > >> 10-11am
> >> > > > > > > > >> > > > > > >> > > PT).
> >> > > > > > > > >> > > > > > >> > > > > > > > > Discussion
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > doc
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> with per-section
> >> > status:
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >>
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >>
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > >
> >> > > > > > > > >> > > > > > >> > > > > > > > > > >
> >> > > > > > > > >> > > > > > >> > > > > > > > > >
> >> > > > > > > > >> > > > > > >> > > > > > > > >
> >> > > > > > > > >> > > > > > >> > > > > > > >
> >> > > > > > > > >> > > > > > >> > > > > > >
> >> > > > > > > > >> > > > > > >> > > > > >
> >> > > > > > > > >> > > > > > >> > > > >
> >> > > > > > > > >> > > > > > >> > > >
> >> > > > > > > > >> > > > > > >> > >
> >> > > > > > > > >> > > > > > >> >
> >> > > > > > > > >> > > > > > >>
> >> > > > > > > > >> > > > > >
> >> > > > > > > > >> > > > >
> >> > > > > > > > >> > >
> >> > > > > > > > >> >
> >> > > > > > > > >>
> >> > > > > > >
> >> > > > > >
> >> > > > >
> >> > >
> >> >
> >>
> https://urldefense.com/v3/__https://docs.google.com/document/d/100h7c4damrUzVuquYbBHM0EvA4LSWuW2IT2dN_7nYVA/edit?tab=t.0__;!!Iz9xO38YGHZK!4rnlVIqBQQQ6xpGe1PMV58edOS_cGrKoqdfhw4qSp_JQCavvEjaxXk0OwGPnxyEcO60AzKQK_XDtp7H3DdkFXcrXgIddeUs$
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >>
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> *The meeting
> >> covered
> >> > > both
> >> > > > > > topics
> >> > > > > > > > >> from
> >> > > > > > > > >> > the
> >> > > > > > > > >> > > > > doc.
> >> > > > > > > > >> > > > > > >> > > > > > Direction-level
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > alignment
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> was reached on
> the
> >> > > headline
> >> > > > > > > pieces;
> >> > > > > > > > >> > > details
> >> > > > > > > > >> > > > > > >> remain
> >> > > > > > > > >> > > > > > >> > for
> >> > > > > > > > >> > > > > > >> > > > PR
> >> > > > > > > > >> > > > > > >> > > > > > > review
> >> > > > > > > > >> > > > > > >> > > > > > > > > or
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> follow-up
> >> sessions.*
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >>
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> *Topic 1 —
> >> Persistence
> >> > > > > schema
> >> > > > > > > > >> redesign*
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> Idea-level
> >> alignment
> >> > on
> >> > > > > > > > >> consolidating
> >> > > > > > > > >> > > > > per-type
> >> > > > > > > > >> > > > > > >> > tables
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >>
> >> (scan_metrics_report,
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >>
> >> commit_metrics_report)
> >> > > into
> >> > > > > a
> >> > > > > > > single
> >> > > > > > > > >> > > > > > >> metrics_report
> >> > > > > > > > >> > > > > > >> > > > table.
> >> > > > > > > > >> > > > > > >> > > > > > The
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > motivating
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> cost is the
> surface
> >> > area
> >> > > > > added
> >> > > > > > > by
> >> > > > > > > > >> every
> >> > > > > > > > >> > > new
> >> > > > > > > > >> > > > > > >> metric
> >> > > > > > > > >> > > > > > >> > > type
> >> > > > > > > > >> > > > > > >> > > > > > today:
> >> > > > > > > > >> > > > > > >> > > > > > > > new
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > table,
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> SPI method,
> record
> >> > > class,
> >> > > > > > model,
> >> > > > > > > > >> > > converter,
> >> > > > > > > > >> > > > > > >> schema
> >> > > > > > > > >> > > > > > >> > > > > > migration.
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >>
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> Most schema
> details
> >> > are
> >> > > > > > > deferred to
> >> > > > > > > > >> the
> >> > > > > > > > >> > > > > schema
> >> > > > > > > > >> > > > > > >> PR. A
> >> > > > > > > > >> > > > > > >> > > few
> >> > > > > > > > >> > > > > > >> > > > > > > > specific
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > points
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> came up:
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> •
> >> > >  metric_schema_version:
> >> > > > > > Yufei
> >> > > > > > > > >> > prefers
> >> > > > > > > > >> > > > > > dropping
> >> > > > > > > > >> > > > > > >> > it,
> >> > > > > > > > >> > > > > > >> > > > > since
> >> > > > > > > > >> > > > > > >> > > > > > > > there
> >> > > > > > > > >> > > > > > >> > > > > > > > > is
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > no
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> spec-level
> concept
> >> of
> >> > > > > metrics
> >> > > > > > > > >> > versioning
> >> > > > > > > > >> > > > > today
> >> > > > > > > > >> > > > > > >> and
> >> > > > > > > > >> > > > > > >> > it
> >> > > > > > > > >> > > > > > >> > > is
> >> > > > > > > > >> > > > > > >> > > > > > hard
> >> > > > > > > > >> > > > > > >> > > > > > > to
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > define
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> unilaterally.
> >> Robert
> >> > > prefers
> >> > > > > > > keeping
> >> > > > > > > > >> > it,
> >> > > > > > > > >> > > > > given
> >> > > > > > > > >> > > > > > >> IRC
> >> > > > > > > > >> > > > > > >> > v2
> >> > > > > > > > >> > > > > > >> > > is
> >> > > > > > > > >> > > > > > >> > > > > > > coming
> >> > > > > > > > >> > > > > > >> > > > > > > > > and
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > the
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> schema should be
> >> > > considered
> >> > > > > > > against
> >> > > > > > > > >> its
> >> > > > > > > > >> > > > > likely
> >> > > > > > > > >> > > > > > >> > shape;
> >> > > > > > > > >> > > > > > >> > > > > Robert
> >> > > > > > > > >> > > > > > >> > > > > > > > also
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > raised
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> how to
> >> differentiate
> >> > > various
> >> > > > > > > payload
> >> > > > > > > > >> > > formats
> >> > > > > > > > >> > > > > if
> >> > > > > > > > >> > > > > > >> any.
> >> > > > > > > > >> > > > > > >> > > > EJ's
> >> > > > > > > > >> > > > > > >> > > > > > read
> >> > > > > > > > >> > > > > > >> > > > > > > > is
> >> > > > > > > > >> > > > > > >> > > > > > > > > > that
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> this
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> is a two-way-door
> >> > > decision.
> >> > > > > We
> >> > > > > > > can
> >> > > > > > > > >> > start
> >> > > > > > > > >> > > > > > without
> >> > > > > > > > >> > > > > > >> the
> >> > > > > > > > >> > > > > > >> > > > > field,
> >> > > > > > > > >> > > > > > >> > > > > > > and
> >> > > > > > > > >> > > > > > >> > > > > > > > if
> >> > > > > > > > >> > > > > > >> > > > > > > > > > IRC
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > v2
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> changes the shape
> >> we
> >> > > would
> >> > > > > > > likely
> >> > > > > > > > >> roll
> >> > > > > > > > >> > a
> >> > > > > > > > >> > > > > > >> > corresponding
> >> > > > > > > > >> > > > > > >> > > > new
> >> > > > > > > > >> > > > > > >> > > > > > > > schema
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > anyway,
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> which is not
> >> > > particularly
> >> > > > > > > costly.
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> •   Payload
> format:
> >> > > Robert
> >> > > > > > > pointed
> >> > > > > > > > >> out
> >> > > > > > > > >> > > that
> >> > > > > > > > >> > > > > > >> future
> >> > > > > > > > >> > > > > > >> > > > formats
> >> > > > > > > > >> > > > > > >> > > > > > > > beyond
> >> > > > > > > > >> > > > > > >> > > > > > > > > > JSON
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > may
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> be worth
> >> supporting.
> >> > The
> >> > > > > exact
> >> > > > > > > > >> shape is
> >> > > > > > > > >> > > > > > deferred
> >> > > > > > > > >> > > > > > >> to
> >> > > > > > > > >> > > > > > >> > > the
> >> > > > > > > > >> > > > > > >> > > > > > schema
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > discussion.
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> •   Partition
> >> > strategy:
> >> > > > > Anand
> >> > > > > > > > >> suggested
> >> > > > > > > > >> > > > > monthly
> >> > > > > > > > >> > > > > > >> > > > > partitioning
> >> > > > > > > > >> > > > > > >> > > > > > > > based
> >> > > > > > > > >> > > > > > >> > > > > > > > > > on
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > his
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> experience as
> >> > > potentially
> >> > > > > > > helpful at
> >> > > > > > > > >> > > scale.
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >>
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> *Topic 2 — Where
> >> > metrics
> >> > > > > > > ingestion
> >> > > > > > > > >> and
> >> > > > > > > > >> > > > > storage
> >> > > > > > > > >> > > > > > >> > belong*
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> Idea-level
> >> alignment
> >> > > that
> >> > > > > > > metrics
> >> > > > > > > > >> > should
> >> > > > > > > > >> > > be a
> >> > > > > > > > >> > > > > > >> > > separated
> >> > > > > > > > >> > > > > > >> > > > > SPI
> >> > > > > > > > >> > > > > > >> > > > > > > from
> >> > > > > > > > >> > > > > > >> > > > > > > > > the
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> entity
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> persistence
> stack.
> >> Two
> >> > > > > reasons
> >> > > > > > > > >> > surfaced:
> >> > > > > > > > >> > > (a)
> >> > > > > > > > >> > > > > > >> > workloads
> >> > > > > > > > >> > > > > > >> > > > and
> >> > > > > > > > >> > > > > > >> > > > > > > > > > capability
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> requirements
> >> diverge
> >> > > enough
> >> > > > > > that
> >> > > > > > > > >> > coupling
> >> > > > > > > > >> > > > > them
> >> > > > > > > > >> > > > > > >> > creates
> >> > > > > > > > >> > > > > > >> > > > > > > > artificial
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> constraints, and
> >> (b)
> >> > > admin
> >> > > > > > > > >> experience
> >> > > > > > > > >> > > > > improves
> >> > > > > > > > >> > > > > > >> when
> >> > > > > > > > >> > > > > > >> > > > > metrics
> >> > > > > > > > >> > > > > > >> > > > > > > has
> >> > > > > > > > >> > > > > > >> > > > > > > > > its
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > own
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> bootstrap,
> >> retention,
> >> > > and
> >> > > > > > > lifecycle.
> >> > > > > > > > >> > > Dmitri
> >> > > > > > > > >> > > > > > noted
> >> > > > > > > > >> > > > > > >> > > > Polaris
> >> > > > > > > > >> > > > > > >> > > > > > > being
> >> > > > > > > > >> > > > > > >> > > > > > > > a
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > platform
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> should have the
> >> > > flexibility
> >> > > > > to
> >> > > > > > > > >> support
> >> > > > > > > > >> > > > > > different
> >> > > > > > > > >> > > > > > >> > > > > persistence
> >> > > > > > > > >> > > > > > >> > > > > > > > > > backends
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > per
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> concern, and
> >> pointed
> >> > to
> >> > > a
> >> > > > > > > concrete
> >> > > > > > > > >> next
> >> > > > > > > > >> > > step
> >> > > > > > > > >> > > > > of
> >> > > > > > > > >> > > > > > >> > > > separating
> >> > > > > > > > >> > > > > > >> > > > > > the
> >> > > > > > > > >> > > > > > >> > > > > > > > > JDBC
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> bootstrap for
> >> metrics
> >> > > from
> >> > > > > the
> >> > > > > > > > >> > metastore
> >> > > > > > > > >> > > > > > >> bootstrap.
> >> > > > > > > > >> > > > > > >> > > > Robert
> >> > > > > > > > >> > > > > > >> > > > > > > > > proposed
> >> > > > > > > > >> > > > > > >> > > > > > > > > > an
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> additional UX
> >> > extension:
> >> > > > > > detect
> >> > > > > > > an
> >> > > > > > > > >> > > > > > unbootstrapped
> >> > > > > > > > >> > > > > > >> > > > metrics
> >> > > > > > > > >> > > > > > >> > > > > > > store
> >> > > > > > > > >> > > > > > >> > > > > > > > on
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > first
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> use and
> >> auto-bootstrap
> >> > > > > rather
> >> > > > > > > than
> >> > > > > > > > >> > > requiring
> >> > > > > > > > >> > > > > an
> >> > > > > > > > >> > > > > > >> > > explicit
> >> > > > > > > > >> > > > > > >> > > > > > > manual
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > bootstrap
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> step.
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> The meeting also
> >> > > confirmed
> >> > > > > > that
> >> > > > > > > > >> Polaris
> >> > > > > > > > >> > > > > metrics
> >> > > > > > > > >> > > > > > >> can
> >> > > > > > > > >> > > > > > >> > > > start
> >> > > > > > > > >> > > > > > >> > > > > > > small
> >> > > > > > > > >> > > > > > >> > > > > > > > > and
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > stay
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> Iceberg-focused.
> >> > Naming
> >> > > and
> >> > > > > > > > >> persistence
> >> > > > > > > > >> > > > > schema
> >> > > > > > > > >> > > > > > >> can
> >> > > > > > > > >> > > > > > >> > > lean
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > Iceberg-specific.
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> If a future
> >> expansion
> >> > to
> >> > > > > > > > >> generic-table
> >> > > > > > > > >> > > > > metrics
> >> > > > > > > > >> > > > > > or
> >> > > > > > > > >> > > > > > >> > > > > > operational
> >> > > > > > > > >> > > > > > >> > > > > > > > > > metrics
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> arrives, an
> >> > abstraction
> >> > > > > layer
> >> > > > > > > can be
> >> > > > > > > > >> > > built on
> >> > > > > > > > >> > > > > > >> top of
> >> > > > > > > > >> > > > > > >> > > the
> >> > > > > > > > >> > > > > > >> > > > > > > Iceberg
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > metrics
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> reporter at that
> >> > point.
> >> > > > > Robert
> >> > > > > > > > >> remains
> >> > > > > > > > >> > > on the
> >> > > > > > > > >> > > > > > >> fence
> >> > > > > > > > >> > > > > > >> > > and
> >> > > > > > > > >> > > > > > >> > > > > > would
> >> > > > > > > > >> > > > > > >> > > > > > > > > prefer
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> something more
> >> generic
> >> > > but
> >> > > > > did
> >> > > > > > > not
> >> > > > > > > > >> > block
> >> > > > > > > > >> > > the
> >> > > > > > > > >> > > > > > >> > > direction;
> >> > > > > > > > >> > > > > > >> > > > > > > Dmitri's
> >> > > > > > > > >> > > > > > >> > > > > > > > > > read
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > was
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> that the proposed
> >> > > framework
> >> > > > > > > already
> >> > > > > > > > >> has
> >> > > > > > > > >> > > > > enough
> >> > > > > > > > >> > > > > > >> > > > flexibility
> >> > > > > > > > >> > > > > > >> > > > > > to
> >> > > > > > > > >> > > > > > >> > > > > > > > > absorb
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> future
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> expansion.
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >>
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> The Trade-offs
> and
> >> > > Proposed
> >> > > > > > > > >> structure
> >> > > > > > > > >> > > > > sections
> >> > > > > > > > >> > > > > > in
> >> > > > > > > > >> > > > > > >> > the
> >> > > > > > > > >> > > > > > >> > > > doc
> >> > > > > > > > >> > > > > > >> > > > > > were
> >> > > > > > > > >> > > > > > >> > > > > > > > not
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> reviewed
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> in detail. They
> >> remain
> >> > > open
> >> > > > > > for
> >> > > > > > > > >> either
> >> > > > > > > > >> > > the
> >> > > > > > > > >> > > > > next
> >> > > > > > > > >> > > > > > >> sync
> >> > > > > > > > >> > > > > > >> > > or
> >> > > > > > > > >> > > > > > >> > > > PR
> >> > > > > > > > >> > > > > > >> > > > > > > > review.
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >>
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> *Cross-cutting
> >> > > alignment —
> >> > > > > > > > >> > > battery-included
> >> > > > > > > > >> > > > > > plus
> >> > > > > > > > >> > > > > > >> > > > > pluggable*
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> A common
> philosophy
> >> > > emerged
> >> > > > > > > from the
> >> > > > > > > > >> > > > > > discussion.
> >> > > > > > > > >> > > > > > >> EJ
> >> > > > > > > > >> > > > > > >> > > > > > summarized
> >> > > > > > > > >> > > > > > >> > > > > > > > it
> >> > > > > > > > >> > > > > > >> > > > > > > > > > as:
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> Polaris should
> >> > provide a
> >> > > > > > > > >> > > battery-included UX
> >> > > > > > > > >> > > > > > for
> >> > > > > > > > >> > > > > > >> > > > beginners
> >> > > > > > > > >> > > > > > >> > > > > > and
> >> > > > > > > > >> > > > > > >> > > > > > > > the
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> flexibility for
> >> > advanced
> >> > > > > users
> >> > > > > > > to
> >> > > > > > > > >> swap
> >> > > > > > > > >> > > the
> >> > > > > > > > >> > > > > > >> included
> >> > > > > > > > >> > > > > > >> > > > > battery
> >> > > > > > > > >> > > > > > >> > > > > > > for
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > something
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> more powerful or
> >> > > tailored to
> >> > > > > > > their
> >> > > > > > > > >> use
> >> > > > > > > > >> > > case.
> >> > > > > > > > >> > > > > > The
> >> > > > > > > > >> > > > > > >> SPI
> >> > > > > > > > >> > > > > > >> > > > > design
> >> > > > > > > > >> > > > > > >> > > > > > > > needs
> >> > > > > > > > >> > > > > > >> > > > > > > > > to
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> enable
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> both.
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >>
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> The inputs that
> >> shaped
> >> > > this
> >> > > > > > > framing:
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> •   Anand
> described
> >> > how
> >> > > his
> >> > > > > > team
> >> > > > > > > > >> uses
> >> > > > > > > > >> > the
> >> > > > > > > > >> > > > > > current
> >> > > > > > > > >> > > > > > >> > > > metrics
> >> > > > > > > > >> > > > > > >> > > > > > > > > > persistence
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> (three metrics
> >> > > consumers in
> >> > > > > > > v1.4).
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> •   Yufei raised
> >> > > Grafana and
> >> > > > > > > > >> dashboard
> >> > > > > > > > >> > > > > > >> integrations
> >> > > > > > > > >> > > > > > >> > > as a
> >> > > > > > > > >> > > > > > >> > > > > > > > > destination
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > use
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> case beyond the
> >> > default.
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> •   Robert called
> >> out
> >> > > that
> >> > > > > the
> >> > > > > > > > >> current
> >> > > > > > > > >> > > design
> >> > > > > > > > >> > > > > > is
> >> > > > > > > > >> > > > > > >> > more
> >> > > > > > > > >> > > > > > >> > > > > > > > > JDBC-focused.
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >>
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> Two concrete
> >> > instances:
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> •   Async metrics
> >> > > intake:
> >> > > > > > > Yufei's
> >> > > > > > > > >> > initial
> >> > > > > > > > >> > > > > > >> position
> >> > > > > > > > >> > > > > > >> > was
> >> > > > > > > > >> > > > > > >> > > > > that
> >> > > > > > > > >> > > > > > >> > > > > > > > async
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > should
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> largely live on
> the
> >> > > producer
> >> > > > > > > side
> >> > > > > > > > >> and
> >> > > > > > > > >> > > there
> >> > > > > > > > >> > > > > is
> >> > > > > > > > >> > > > > > >> not
> >> > > > > > > > >> > > > > > >> > > much
> >> > > > > > > > >> > > > > > >> > > > > > > Polaris
> >> > > > > > > > >> > > > > > >> > > > > > > > > can
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > do.
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> Robert suggested
> a
> >> > > > > > Polaris-side
> >> > > > > > > > >> default
> >> > > > > > > > >> > > is
> >> > > > > > > > >> > > > > > doable
> >> > > > > > > > >> > > > > > >> > via
> >> > > > > > > > >> > > > > > >> > > > > > Vert.x.
> >> > > > > > > > >> > > > > > >> > > > > > > > > Dmitri
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> agreed
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> the direction is
> >> worth
> >> > > > > > > exploring.
> >> > > > > > > > >> The
> >> > > > > > > > >> > > meeting
> >> > > > > > > > >> > > > > > >> > > converged
> >> > > > > > > > >> > > > > > >> > > > > on a
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> battery-included
> >> > default
> >> > > > > > (likely
> >> > > > > > > > >> > > > > Vert.x-backed)
> >> > > > > > > > >> > > > > > >> with
> >> > > > > > > > >> > > > > > >> > > an
> >> > > > > > > > >> > > > > > >> > > > > SPI
> >> > > > > > > > >> > > > > > >> > > > > > > > shape
> >> > > > > > > > >> > > > > > >> > > > > > > > > > that
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> lets
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> power users route
> >> to a
> >> > > more
> >> > > > > > > scalable
> >> > > > > > > > >> > > backend
> >> > > > > > > > >> > > > > > >> > > (k8s-hosted
> >> > > > > > > > >> > > > > > >> > > > > > > queue,
> >> > > > > > > > >> > > > > > >> > > > > > > > > AWS
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > SQS,
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> etc.).
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> •   Pluggable
> >> > > destinations:
> >> > > > > > > > >> combining
> >> > > > > > > > >> > > Yufei's
> >> > > > > > > > >> > > > > > >> > > dashboard
> >> > > > > > > > >> > > > > > >> > > > > use
> >> > > > > > > > >> > > > > > >> > > > > > > case
> >> > > > > > > > >> > > > > > >> > > > > > > > > > with
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> Robert's
> >> JDBC-focused
> >> > > > > > call-out,
> >> > > > > > > the
> >> > > > > > > > >> > > meeting
> >> > > > > > > > >> > > > > > >> agreed
> >> > > > > > > > >> > > > > > >> > the
> >> > > > > > > > >> > > > > > >> > > > SPI
> >> > > > > > > > >> > > > > > >> > > > > > > > should
> >> > > > > > > > >> > > > > > >> > > > > > > > > be
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> structured for
> >> > multiple
> >> > > > > sinks
> >> > > > > > so
> >> > > > > > > > >> > > integrations
> >> > > > > > > > >> > > > > > >> become
> >> > > > > > > > >> > > > > > >> > > > impl
> >> > > > > > > > >> > > > > > >> > > > > > > > choices
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > rather
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> than
> architectural
> >> > > changes.
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >>
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> The
> >> battery-included
> >> > > default
> >> > > > > > is
> >> > > > > > > most
> >> > > > > > > > >> > > likely
> >> > > > > > > > >> > > > > to
> >> > > > > > > > >> > > > > > >> use
> >> > > > > > > > >> > > > > > >> > the
> >> > > > > > > > >> > > > > > >> > > > > > > existing
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> JDBC-backed
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> approach.
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >>
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> *Direction
> >> (idea-level
> >> > > > > > > alignment)*
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> •   Single
> >> > > metrics_report
> >> > > > > > table
> >> > > > > > > > >> > > consolidating
> >> > > > > > > > >> > > > > > >> > per-type
> >> > > > > > > > >> > > > > > >> > > > > > > metrics,
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > replacing
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >>
> scan_metrics_report
> >> > and
> >> > > > > > > > >> > > commit_metrics_report
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> •
>  Iceberg-focused
> >> > > naming
> >> > > > > and
> >> > > > > > > > >> schema
> >> > > > > > > > >> > for
> >> > > > > > > > >> > > > > now,
> >> > > > > > > > >> > > > > > >> > revisit
> >> > > > > > > > >> > > > > > >> > > > if
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > generic-table
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > or
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> operational
> metrics
> >> > > arrive
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> •   Metrics
> >> > persistence
> >> > > as a
> >> > > > > > > > >> separated
> >> > > > > > > > >> > > SPI,
> >> > > > > > > > >> > > > > not
> >> > > > > > > > >> > > > > > >> on
> >> > > > > > > > >> > > > > > >> > > > > > > > BasePersistence
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> •   Bootstrap
> path
> >> > > separated
> >> > > > > > for
> >> > > > > > > > >> > metrics,
> >> > > > > > > > >> > > > > > >> > independent
> >> > > > > > > > >> > > > > > >> > > of
> >> > > > > > > > >> > > > > > >> > > > > > > > metastore
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> bootstrap
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> •
> >>  "Battery-included
> >> > > plus
> >> > > > > > > > >> pluggable"
> >> > > > > > > > >> > as
> >> > > > > > > > >> > > the
> >> > > > > > > > >> > > > > > SPI
> >> > > > > > > > >> > > > > > >> > > design
> >> > > > > > > > >> > > > > > >> > > > > > > > philosophy
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >>
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> *Open items*
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> •   Schema
> details:
> >> > > > > > > > >> > > metric_schema_version,
> >> > > > > > > > >> > > > > > >> payload
> >> > > > > > > > >> > > > > > >> > > > format,
> >> > > > > > > > >> > > > > > >> > > > > > IRC
> >> > > > > > > > >> > > > > > >> > > > > > > > v2
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> forward-compat
> >> shape
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> •   SPI design
> >> > details —
> >> > > > > full
> >> > > > > > > review
> >> > > > > > > > >> > > either
> >> > > > > > > > >> > > > > in
> >> > > > > > > > >> > > > > > >> the
> >> > > > > > > > >> > > > > > >> > > next
> >> > > > > > > > >> > > > > > >> > > > > sync
> >> > > > > > > > >> > > > > > >> > > > > > > or
> >> > > > > > > > >> > > > > > >> > > > > > > > in
> >> > > > > > > > >> > > > > > >> > > > > > > > > > the
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> corresponding PR
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> •   Schema
> >> refactor PR
> >> > > > > > ownership
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >>
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> *Action items*
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> •   EJ to take a
> >> first
> >> > > stab
> >> > > > > at
> >> > > > > > > the
> >> > > > > > > > >> SPI
> >> > > > > > > > >> > > design
> >> > > > > > > > >> > > > > > and
> >> > > > > > > > >> > > > > > >> > > > > > potentially
> >> > > > > > > > >> > > > > > >> > > > > > > > > > partner
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > with
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> Anand to
> >> incorporate
> >> > the
> >> > > > > > lessons
> >> > > > > > > > >> > learned
> >> > > > > > > > >> > > from
> >> > > > > > > > >> > > > > > the
> >> > > > > > > > >> > > > > > >> > > > existing
> >> > > > > > > > >> > > > > > >> > > > > > > > > reporter
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > and
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> persistence work.
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> •   Schema
> >> refactor PR
> >> > > > > > > ownership is
> >> > > > > > > > >> not
> >> > > > > > > > >> > > yet
> >> > > > > > > > >> > > > > > >> decided.
> >> > > > > > > > >> > > > > > >> > > If
> >> > > > > > > > >> > > > > > >> > > > > > anyone
> >> > > > > > > > >> > > > > > >> > > > > > > > is
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> interested in
> >> driving
> >> > > it,
> >> > > > > > reply
> >> > > > > > > on
> >> > > > > > > > >> this
> >> > > > > > > > >> > > > > thread.
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> •   JB to
> schedule
> >> the
> >> > > next
> >> > > > > > > sync,
> >> > > > > > > > >> > > tentatively
> >> > > > > > > > >> > > > > > in
> >> > > > > > > > >> > > > > > >> two
> >> > > > > > > > >> > > > > > >> > > > > weeks.
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >>
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> -ej
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >>
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> On Mon, Apr 27,
> >> 2026
> >> > at
> >> > > > > > 3:07 PM
> >> > > > > > > EJ
> >> > > > > > > > >> > Wang <
> >> > > > > > > > >> > > > > > >> > > > > > > > > > >
> >> > > ej.wang.dev.interest@gmail.com
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> wrote:
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >>
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> > Thanks Yufei
> for
> >> the
> >> > > +1.
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> > JB, could you
> >> help
> >> > > add a
> >> > > > > > > biweekly
> >> > > > > > > > >> > > metrics
> >> > > > > > > > >> > > > > > >> > > architecture
> >> > > > > > > > >> > > > > > >> > > > > > sync
> >> > > > > > > > >> > > > > > >> > > > > > > to
> >> > > > > > > > >> > > > > > >> > > > > > > > > the
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> Polaris
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> > community
> >> calendar?
> >> > > I'm
> >> > > > > > > thinking
> >> > > > > > > > >> > > Thursdays
> >> > > > > > > > >> > > > > at
> >> > > > > > > > >> > > > > > >> > 9-10am
> >> > > > > > > > >> > > > > > >> > > > PT,
> >> > > > > > > > >> > > > > > >> > > > > > on
> >> > > > > > > > >> > > > > > >> > > > > > > > the
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> off-weeks
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> > from the
> >> community
> >> > > meeting
> >> > > > > > > > >> (starting
> >> > > > > > > > >> > > May
> >> > > > > > > > >> > > > > 7),
> >> > > > > > > > >> > > > > > 60
> >> > > > > > > > >> > > > > > >> > > > minutes.
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> > Here's a rough
> >> > agenda
> >> > > to
> >> > > > > > work
> >> > > > > > > > >> through
> >> > > > > > > > >> > > over
> >> > > > > > > > >> > > > > > the
> >> > > > > > > > >> > > > > > >> > first
> >> > > > > > > > >> > > > > > >> > > > few
> >> > > > > > > > >> > > > > > >> > > > > > > > > sessions,
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> grouped
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> > by priority:
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> > *First:
> >> foundational
> >> > > > > > > direction*
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> > 1.
> >> > > MetricsPersistence:
> >> > > > > > public
> >> > > > > > > > >> SPI or
> >> > > > > > > > >> > > > > > internal
> >> > > > > > > > >> > > > > > >> > > > > > > implementation
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > detail?
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >    •   Marked
> >> @Beta,
> >> > > > > javadoc
> >> > > > > > > calls
> >> > > > > > > > >> > it a
> >> > > > > > > > >> > > > > > >> "Service
> >> > > > > > > > >> > > > > > >> > > > > Provider
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > Interface",
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> but
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> > only one
> consumer
> >> > > > > > > > >> > > > > (JdbcBasePersistenceImpl),
> >> > > > > > > > >> > > > > > >> lives
> >> > > > > > > > >> > > > > > >> > > on
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > BasePersistence.
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> If
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> > demoted to a
> >> private
> >> > > > > helper
> >> > > > > > > > >> inside a
> >> > > > > > > > >> > > > > > persisting
> >> > > > > > > > >> > > > > > >> > > > reporter
> >> > > > > > > > >> > > > > > >> > > > > > > impl,
> >> > > > > > > > >> > > > > > >> > > > > > > > > > most
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> > downstream
> design
> >> > > > > decisions
> >> > > > > > > become
> >> > > > > > > > >> > > > > > >> implementation
> >> > > > > > > > >> > > > > > >> > > > > details
> >> > > > > > > > >> > > > > > >> > > > > > > > rather
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > than
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> > contract
> >> questions.
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> > 2.  Persistence
> >> > schema
> >> > > > > > > redesign
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >    •   Current
> >> > > two-table
> >> > > > > > > layout
> >> > > > > > > > >> > > > > > >> > > (scan_metrics_report,
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >
> >> > commit_metrics_report)
> >> > > > > with
> >> > > > > > > ~25
> >> > > > > > > > >> > > flattened
> >> > > > > > > > >> > > > > > >> columns
> >> > > > > > > > >> > > > > > >> > > > each.
> >> > > > > > > > >> > > > > > >> > > > > > > Every
> >> > > > > > > > >> > > > > > >> > > > > > > > > new
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > metric
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> > type requires a
> >> new
> >> > > table,
> >> > > > > > SPI
> >> > > > > > > > >> > method,
> >> > > > > > > > >> > > > > record
> >> > > > > > > > >> > > > > > >> > class,
> >> > > > > > > > >> > > > > > >> > > > > > model,
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > converter,
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> and
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> > schema
> migration.
> >> > > > > Direction
> >> > > > > > to
> >> > > > > > > > >> > explore:
> >> > > > > > > > >> > > > > > single
> >> > > > > > > > >> > > > > > >> > table
> >> > > > > > > > >> > > > > > >> > > > > with
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > metric_type
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> enum,
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> > schema_version,
> >> and
> >> > > JSON
> >> > > > > > > payload
> >> > > > > > > > >> > > column.
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> > *Second: design
> >> > > details
> >> > > > > once
> >> > > > > > > > >> > direction
> >> > > > > > > > >> > > is
> >> > > > > > > > >> > > > > > set*
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> > 3.  Partition
> key
> >> > > strategy
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >    •
> >>  Single-table
> >> > > design
> >> > > > > > > means
> >> > > > > > > > >> scan
> >> > > > > > > > >> > > > > metrics
> >> > > > > > > > >> > > > > > >> at
> >> > > > > > > > >> > > > > > >> > > scale
> >> > > > > > > > >> > > > > > >> > > > > > will
> >> > > > > > > > >> > > > > > >> > > > > > > > have
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > high
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> > write
> concurrency
> >> > per
> >> > > > > table.
> >> > > > > > > > >> Schema
> >> > > > > > > > >> > > needs
> >> > > > > > > > >> > > > > to
> >> > > > > > > > >> > > > > > >> > expose
> >> > > > > > > > >> > > > > > >> > > > > enough
> >> > > > > > > > >> > > > > > >> > > > > > > > > > structure
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > for
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> > backends to
> >> shard by
> >> > > > > entity
> >> > > > > > or
> >> > > > > > > > >> time
> >> > > > > > > > >> > > range.
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> > 4.  Read/write
> >> path
> >> > > > > > > consistency
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >    •   Writes
> go
> >> > > through
> >> > > > > > > > >> > > > > > PolarisMetricsManager
> >> > > > > > > > >> > > > > > >> on
> >> > > > > > > > >> > > > > > >> > > > > > > > > > MetaStoreManager.
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> Reads
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> > bypass
> >> > > MetaStoreManager
> >> > > > > and
> >> > > > > > go
> >> > > > > > > > >> > > straight to
> >> > > > > > > > >> > > > > > >> > > > > > BasePersistence,
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > excluding
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> > non-JDBC
> backends
> >> > > from the
> >> > > > > > > read
> >> > > > > > > > >> API.
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> > *Third: cleanup
> >> and
> >> > > > > > alignment*
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> > 5.
> >> > > PolarisMetricsReporter
> >> > > > > > > naming
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >    •   Only
> >> handles
> >> > > IRC
> >> > > > > > > > >> > > > > > >> (ScanReport/CommitReport),
> >> > > > > > > > >> > > > > > >> > > > > doesn't
> >> > > > > > > > >> > > > > > >> > > > > > > > cover
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > generic
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> > tables or
> >> > operational
> >> > > > > > metrics.
> >> > > > > > > > >> Name
> >> > > > > > > > >> > is
> >> > > > > > > > >> > > > > > broader
> >> > > > > > > > >> > > > > > >> > than
> >> > > > > > > > >> > > > > > >> > > > > scope.
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> > 6.
> >> > > PolarisMetricsManager
> >> > > > > > > facade
> >> > > > > > > > >> > > > > passthrough
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >    •   Entire
> >> > default
> >> > > > > method
> >> > > > > > > is
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >>
> >> > > > > > > > >> > callCtx.getMetaStore().writeScanReport().
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> > Zero logic,
> >> passes
> >> > > Level 1
> >> > > > > > > > >> straight
> >> > > > > > > > >> > > through
> >> > > > > > > > >> > > > > > to
> >> > > > > > > > >> > > > > > >> > Level
> >> > > > > > > > >> > > > > > >> > > > 3.
> >> > > > > > > > >> > > > > > >> > > > > > Same
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> anti-pattern
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> > as
> >> > > PolarisEventManager.
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> > 7.  Iceberg
> >> > community
> >> > > > > > > alignment
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >    •
> >>  Payload-type
> >> > > > > > extension
> >> > > > > > > > >> needs
> >> > > > > > > > >> > > > > > discussion
> >> > > > > > > > >> > > > > > >> on
> >> > > > > > > > >> > > > > > >> > > > > > > dev@iceberg.
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> obelix74's
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> > Feb thread got
> >> zero
> >> > > > > replies.
> >> > > > > > > > >> Needs a
> >> > > > > > > > >> > > > > > committer
> >> > > > > > > > >> > > > > > >> > > voice.
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> > Lets confirm
> >> > > > > prioritization
> >> > > > > > > in the
> >> > > > > > > > >> > > first
> >> > > > > > > > >> > > > > > >> session.
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> > -ej
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> > On Tue, Apr 21,
> >> 2026
> >> > > at
> >> > > > > > > 3:18 PM
> >> > > > > > > > >> Yufei
> >> > > > > > > > >> > > Gu <
> >> > > > > > > > >> > > > > > >> > > > > > > > flyrain000@gmail.com>
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > wrote:
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> Thanks
> everyone
> >> for
> >> > > > > > > continuing to
> >> > > > > > > > >> > > drive
> >> > > > > > > > >> > > > > this
> >> > > > > > > > >> > > > > > >> > > > forward. I
> >> > > > > > > > >> > > > > > >> > > > > > > agree
> >> > > > > > > > >> > > > > > >> > > > > > > > > > that
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > the
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> problem is
> >> getting
> >> > > > > complex
> >> > > > > > > enough
> >> > > > > > > > >> > > that a
> >> > > > > > > > >> > > > > > more
> >> > > > > > > > >> > > > > > >> > > > > structured
> >> > > > > > > > >> > > > > > >> > > > > > > > > > discussion
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> would
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> help.
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >>
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> +1 on setting
> >> up a
> >> > > > > biweekly
> >> > > > > > > sync
> >> > > > > > > > >> for
> >> > > > > > > > >> > > the
> >> > > > > > > > >> > > > > > >> metrics
> >> > > > > > > > >> > > > > > >> > > > > > > > architecture.
> >> > > > > > > > >> > > > > > >> > > > > > > > > > I’m
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> happy
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> to
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> join.
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >>
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> Yufei
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >>
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >>
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> On Tue, Apr
> 21,
> >> > 2026
> >> > > at
> >> > > > > > > 2:34 PM
> >> > > > > > > > >> EJ
> >> > > > > > > > >> > > Wang <
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >>
> >> > > > > > ej.wang.dev.interest@gmail.com>
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> wrote:
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >>
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> > Also, I've
> >> been
> >> > > looking
> >> > > > > > > more
> >> > > > > > > > >> > > closely at
> >> > > > > > > > >> > > > > > the
> >> > > > > > > > >> > > > > > >> > > > > > *persistence
> >> > > > > > > > >> > > > > > >> > > > > > > > > schema
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > in
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> the
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> > current
> >> metrics
> >> > > work*,
> >> > > > > > and
> >> > > > > > > I
> >> > > > > > > > >> think
> >> > > > > > > > >> > > > > > there's a
> >> > > > > > > > >> > > > > > >> > > > > structural
> >> > > > > > > > >> > > > > > >> > > > > > > > > > rigidity
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> problem
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> > worth
> raising
> >> > > before
> >> > > > > the
> >> > > > > > > shape
> >> > > > > > > > >> > gets
> >> > > > > > > > >> > > > > locked
> >> > > > > > > > >> > > > > > >> in.
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> > Right now we
> >> have
> >> > > two
> >> > > > > > > separate
> >> > > > > > > > >> > > tables
> >> > > > > > > > >> > > > > > >> > > > > > > (scan_metrics_report
> >> > > > > > > > >> > > > > > >> > > > > > > > > and
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >
> >> > > commit_metrics_report),
> >> > > > > > > each
> >> > > > > > > > >> with
> >> > > > > > > > >> > > ~25
> >> > > > > > > > >> > > > > > >> flattened
> >> > > > > > > > >> > > > > > >> > > > > columns
> >> > > > > > > > >> > > > > > >> > > > > > > > that
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > directly
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> > mirror the
> >> > Iceberg
> >> > > > > report
> >> > > > > > > > >> fields.
> >> > > > > > > > >> > > The
> >> > > > > > > > >> > > > > SPI
> >> > > > > > > > >> > > > > > >> > follows
> >> > > > > > > > >> > > > > > >> > > > the
> >> > > > > > > > >> > > > > > >> > > > > > > same
> >> > > > > > > > >> > > > > > >> > > > > > > > > > split:
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >
> >> writeScanReport
> >> > and
> >> > > > > > > > >> > > writeCommitReport as
> >> > > > > > > > >> > > > > > >> > separate
> >> > > > > > > > >> > > > > > >> > > > > > > methods,
> >> > > > > > > > >> > > > > > >> > > > > > > > > with
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> per-type
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> > record
> >> classes,
> >> > > > > > > converters, and
> >> > > > > > > > >> > > model
> >> > > > > > > > >> > > > > > >> objects.
> >> > > > > > > > >> > > > > > >> > > *The
> >> > > > > > > > >> > > > > > >> > > > > > > > practical
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > cost:
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> > adding a new
> >> > metric
> >> > > > > type
> >> > > > > > > > >> > > (operational
> >> > > > > > > > >> > > > > > >> metrics,
> >> > > > > > > > >> > > > > > >> > > for
> >> > > > > > > > >> > > > > > >> > > > > > > example)
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > requires
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> a
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> new
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> > table, a new
> >> SPI
> >> > > > > method,
> >> > > > > > a
> >> > > > > > > new
> >> > > > > > > > >> > > record
> >> > > > > > > > >> > > > > > >> class, a
> >> > > > > > > > >> > > > > > >> > > new
> >> > > > > > > > >> > > > > > >> > > > > > model
> >> > > > > > > > >> > > > > > >> > > > > > > > > > class, a
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > new
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> > converter
> >> branch,
> >> > > and a
> >> > > > > > > schema
> >> > > > > > > > >> > > > > migration*.
> >> > > > > > > > >> > > > > > >> > > That's a
> >> > > > > > > > >> > > > > > >> > > > > lot
> >> > > > > > > > >> > > > > > >> > > > > > > of
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > surface
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> area
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> > for what
> >> should
> >> > be
> >> > > "one
> >> > > > > > > more
> >> > > > > > > > >> kind
> >> > > > > > > > >> > of
> >> > > > > > > > >> > > > > > >> metric."
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> > *My bias*
> >> would
> >> > be
> >> > > > > > toward a
> >> > > > > > > > >> single
> >> > > > > > > > >> > > > > metrics
> >> > > > > > > > >> > > > > > >> > table
> >> > > > > > > > >> > > > > > >> > > > with
> >> > > > > > > > >> > > > > > >> > > > > > *a
> >> > > > > > > > >> > > > > > >> > > > > > > > > typed
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > JSON
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> > payload*.
> >> > Something
> >> > > > > like:
> >> > > > > > > > >> > > metric_type
> >> > > > > > > > >> > > > > > >> (enum),
> >> > > > > > > > >> > > > > > >> > > > > > entity_id,
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >
> >> table_identifier,
> >> > > > > > > snapshot_id
> >> > > > > > > > >> > > > > (nullable),
> >> > > > > > > > >> > > > > > >> > > > > received_ts,
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> schema_version,
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> and
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> > a payload
> >> column
> >> > > for
> >> > > > > the
> >> > > > > > > > >> > > metric-specific
> >> > > > > > > > >> > > > > > >> data.
> >> > > > > > > > >> > > > > > >> > > The
> >> > > > > > > > >> > > > > > >> > > > > > > > > metric_type
> >> > > > > > > > >> > > > > > >> > > > > > > > > > +
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >
> schema_version
> >> > pair
> >> > > > > gives
> >> > > > > > > us a
> >> > > > > > > > >> > > > > > >> > forward-compatible
> >> > > > > > > > >> > > > > > >> > > > > > > contract
> >> > > > > > > > >> > > > > > >> > > > > > > > > for
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > the
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> payload
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> > shape.
> Adding
> >> a
> >> > new
> >> > > > > > metric
> >> > > > > > > type
> >> > > > > > > > >> > > becomes
> >> > > > > > > > >> > > > > an
> >> > > > > > > > >> > > > > > >> enum
> >> > > > > > > > >> > > > > > >> > > > value
> >> > > > > > > > >> > > > > > >> > > > > > > and a
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > payload
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> schema,
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> > not a schema
> >> > > migration.
> >> > > > > > One
> >> > > > > > > > >> thing
> >> > > > > > > > >> > I
> >> > > > > > > > >> > > > > think
> >> > > > > > > > >> > > > > > we
> >> > > > > > > > >> > > > > > >> > need
> >> > > > > > > > >> > > > > > >> > > > to
> >> > > > > > > > >> > > > > > >> > > > > be
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > deliberate
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> about is
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> > the
> partition
> >> key
> >> > > > > design.
> >> > > > > > > If
> >> > > > > > > > >> all
> >> > > > > > > > >> > > metric
> >> > > > > > > > >> > > > > > >> types
> >> > > > > > > > >> > > > > > >> > > land
> >> > > > > > > > >> > > > > > >> > > > in
> >> > > > > > > > >> > > > > > >> > > > > > one
> >> > > > > > > > >> > > > > > >> > > > > > > > > > table,
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > scan
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> > metrics at
> >> scale
> >> > > (high
> >> > > > > > > > >> > concurrency,
> >> > > > > > > > >> > > high
> >> > > > > > > > >> > > > > > >> > > frequency
> >> > > > > > > > >> > > > > > >> > > > > > across
> >> > > > > > > > >> > > > > > >> > > > > > > > > many
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> tables)
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> > could easily
> >> > > create hot
> >> > > > > > > > >> > partitions.
> >> > > > > > > > >> > > We'd
> >> > > > > > > > >> > > > > > >> want
> >> > > > > > > > >> > > > > > >> > the
> >> > > > > > > > >> > > > > > >> > > > > > > > persistence
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > layer
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> to
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> be
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> > able to
> shard
> >> by
> >> > > entity
> >> > > > > > or
> >> > > > > > > time
> >> > > > > > > > >> > > range,
> >> > > > > > > > >> > > > > and
> >> > > > > > > > >> > > > > > >> that
> >> > > > > > > > >> > > > > > >> > > > means
> >> > > > > > > > >> > > > > > >> > > > > > the
> >> > > > > > > > >> > > > > > >> > > > > > > > > > logical
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> schema
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> > needs to
> >> expose
> >> > > enough
> >> > > > > > > > >> structure
> >> > > > > > > > >> > for
> >> > > > > > > > >> > > > > > >> backends
> >> > > > > > > > >> > > > > > >> > to
> >> > > > > > > > >> > > > > > >> > > > > > > partition
> >> > > > > > > > >> > > > > > >> > > > > > > > > on.
> >> > > > > > > > >> > > > > > >> > > > > > > > > > I
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> don't
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> > think the
> >> current
> >> > > > > > flattened
> >> > > > > > > > >> layout
> >> > > > > > > > >> > > gives
> >> > > > > > > > >> > > > > > us
> >> > > > > > > > >> > > > > > >> > that.
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> > This is
> >> getting
> >> > > complex
> >> > > > > > > enough
> >> > > > > > > > >> > that
> >> > > > > > > > >> > > I
> >> > > > > > > > >> > > > > > don't
> >> > > > > > > > >> > > > > > >> > think
> >> > > > > > > > >> > > > > > >> > > > > > ad-hoc
> >> > > > > > > > >> > > > > > >> > > > > > > > > PR/ML
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> threads
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> > will
> converge
> >> > well.
> >> > > > > > *Would
> >> > > > > > > > >> people
> >> > > > > > > > >> > be
> >> > > > > > > > >> > > > > open
> >> > > > > > > > >> > > > > > >> to a
> >> > > > > > > > >> > > > > > >> > > > > biweekly
> >> > > > > > > > >> > > > > > >> > > > > > > > sync
> >> > > > > > > > >> > > > > > >> > > > > > > > > > for
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> metrics
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >
> >> architecture?* I
> >> > > think
> >> > > > > 30
> >> > > > > > > > >> minutes
> >> > > > > > > > >> > > every
> >> > > > > > > > >> > > > > > two
> >> > > > > > > > >> > > > > > >> > weeks
> >> > > > > > > > >> > > > > > >> > > > > with
> >> > > > > > > > >> > > > > > >> > > > > > > > > > interested
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> parties
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> > would be
> >> enough
> >> > to
> >> > > work
> >> > > > > > > through
> >> > > > > > > > >> > the
> >> > > > > > > > >> > > > > > schema,
> >> > > > > > > > >> > > > > > >> SPI
> >> > > > > > > > >> > > > > > >> > > > > shape,
> >> > > > > > > > >> > > > > > >> > > > > > > and
> >> > > > > > > > >> > > > > > >> > > > > > > > > read
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > API
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> design
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> > together.
> >> Happy
> >> > to
> >> > > help
> >> > > > > > set
> >> > > > > > > > >> that
> >> > > > > > > > >> > up.
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> > -ej
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> > On Mon, Apr
> >> 20,
> >> > > 2026 at
> >> > > > > > > > >> 2:19 PM EJ
> >> > > > > > > > >> > > Wang
> >> > > > > > > > >> > > > > <
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >>
> >> > > > > > ej.wang.dev.interest@gmail.com
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> > wrote:
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >> Reviewed
> >> #4115,
> >> > > left a
> >> > > > > > > > >> comment on
> >> > > > > > > > >> > > the
> >> > > > > > > > >> > > > > > code
> >> > > > > > > > >> > > > > > >> > > > > > organization
> >> > > > > > > > >> > > > > > >> > > > > > > > > side.
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >> One thing
> >> stood
> >> > > out:
> >> > > > > the
> >> > > > > > > > >> metrics
> >> > > > > > > > >> > > write
> >> > > > > > > > >> > > > > > path
> >> > > > > > > > >> > > > > > >> > > enters
> >> > > > > > > > >> > > > > > >> > > > > > > through
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>
> >> > > PolarisMetricsManager
> >> > > > > on
> >> > > > > > > > >> > > > > > MetaStoreManager,
> >> > > > > > > > >> > > > > > >> but
> >> > > > > > > > >> > > > > > >> > > the
> >> > > > > > > > >> > > > > > >> > > > > new
> >> > > > > > > > >> > > > > > >> > > > > > > > read
> >> > > > > > > > >> > > > > > >> > > > > > > > > > path
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> bypasses
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>
> >> MetaStoreManager
> >> > > > > > entirely
> >> > > > > > > and
> >> > > > > > > > >> > goes
> >> > > > > > > > >> > > > > > >> straight to
> >> > > > > > > > >> > > > > > >> > > > > > > > > BasePersistence
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > via
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>
> >> > > > > > > callContext.getMetaStore().
> >> > > > > > > > >> That
> >> > > > > > > > >> > > means
> >> > > > > > > > >> > > > > > the
> >> > > > > > > > >> > > > > > >> > read
> >> > > > > > > > >> > > > > > >> > > > API
> >> > > > > > > > >> > > > > > >> > > > > > only
> >> > > > > > > > >> > > > > > >> > > > > > > > > works
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > for
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> backends
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >> that
> >> implement
> >> > > > > > > > >> BasePersistence.
> >> > > > > > > > >> > > NoSQL
> >> > > > > > > > >> > > > > and
> >> > > > > > > > >> > > > > > >> > remote
> >> > > > > > > > >> > > > > > >> > > > > > > backends
> >> > > > > > > > >> > > > > > >> > > > > > > > > > can't
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> participate.
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >> Stepping
> >> back, I
> >> > > think
> >> > > > > > the
> >> > > > > > > > >> > metrics
> >> > > > > > > > >> > > > > > >> subsystem
> >> > > > > > > > >> > > > > > >> > is
> >> > > > > > > > >> > > > > > >> > > > > > growing
> >> > > > > > > > >> > > > > > >> > > > > > > > into
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> something
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >> real
> (write +
> >> > > read +
> >> > > > > > REST
> >> > > > > > > API
> >> > > > > > > > >> +
> >> > > > > > > > >> > > AuthZ +
> >> > > > > > > > >> > > > > > >> > > > pagination)
> >> > > > > > > > >> > > > > > >> > > > > > *but
> >> > > > > > > > >> > > > > > >> > > > > > > > the
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> persistence
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >> side is
> split
> >> > > across
> >> > > > > two
> >> > > > > > > > >> layers
> >> > > > > > > > >> > in
> >> > > > > > > > >> > > a
> >> > > > > > > > >> > > > > way
> >> > > > > > > > >> > > > > > >> > that's
> >> > > > > > > > >> > > > > > >> > > > hard
> >> > > > > > > > >> > > > > > >> > > > > > to
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > extend*. I
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> put
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >> together
> two
> >> > > diagrams
> >> > > > > to
> >> > > > > > > show
> >> > > > > > > > >> > what
> >> > > > > > > > >> > > I
> >> > > > > > > > >> > > > > mean
> >> > > > > > > > >> > > > > > >> (my
> >> > > > > > > > >> > > > > > >> > > best
> >> > > > > > > > >> > > > > > >> > > > > > > > effort).
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >> *Current
> >> state*
> >> > > > > (Diagram
> >> > > > > > > 1):
> >> > > > > > > > >> > three
> >> > > > > > > > >> > > > > > >> interfaces
> >> > > > > > > > >> > > > > > >> > at
> >> > > > > > > > >> > > > > > >> > > > > three
> >> > > > > > > > >> > > > > > >> > > > > > > > > > different
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> levels.
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >> The
> >> > engine-facing
> >> > > SPI
> >> > > > > > > > >> > > > > > >> (PolarisMetricsReporter)
> >> > > > > > > > >> > > > > > >> > > is
> >> > > > > > > > >> > > > > > >> > > > > > clean.
> >> > > > > > > > >> > > > > > >> > > > > > > > But
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>
> >> > > PolarisMetricsManager
> >> > > > > on
> >> > > > > > > > >> > > > > MetaStoreManager
> >> > > > > > > > >> > > > > > >> is a
> >> > > > > > > > >> > > > > > >> > > > > > > passthrough
> >> > > > > > > > >> > > > > > >> > > > > > > > > to
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>
> >> > > MetricsPersistence on
> >> > > > > > > > >> > > BasePersistence.
> >> > > > > > > > >> > > > > > The
> >> > > > > > > > >> > > > > > >> > @Beta
> >> > > > > > > > >> > > > > > >> > > > > > > > annotation
> >> > > > > > > > >> > > > > > >> > > > > > > > > > and
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > SPI
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> javadoc
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >> are on the
> >> > > > > > BasePersistence
> >> > > > > > > > >> layer,
> >> > > > > > > > >> > > while
> >> > > > > > > > >> > > > > > the
> >> > > > > > > > >> > > > > > >> > > actual
> >> > > > > > > > >> > > > > > >> > > > > > > > extension
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > points
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>
> >> > > > > (PolarisMetricsReporter,
> >> > > > > > > > >> > > > > > >> > PolarisMetricsManager)
> >> > > > > > > > >> > > > > > >> > > > > carry
> >> > > > > > > > >> > > > > > >> > > > > > no
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > stability
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >> annotation.
> >> The
> >> > > write
> >> > > > > > path
> >> > > > > > > > >> goes
> >> > > > > > > > >> > > through
> >> > > > > > > > >> > > > > > the
> >> > > > > > > > >> > > > > > >> > > > > > > > MetaStoreManager
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > layer,
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> the
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >> read path
> >> > doesn't.
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >> *What I
> >> > envision*
> >> > > > > > > (Diagram 2):
> >> > > > > > > > >> > two
> >> > > > > > > > >> > > SPIs
> >> > > > > > > > >> > > > > > at
> >> > > > > > > > >> > > > > > >> two
> >> > > > > > > > >> > > > > > >> > > > > levels.
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>
> >> > > PolarisMetricsReporter
> >> > > > > > > stays
> >> > > > > > > > >> as
> >> > > > > > > > >> > the
> >> > > > > > > > >> > > > > > >> > > engine-facing
> >> > > > > > > > >> > > > > > >> > > > > SPI.
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>
> >> > > PolarisMetricsManager
> >> > > > > > > becomes
> >> > > > > > > > >> the
> >> > > > > > > > >> > > > > > >> > backend-facing
> >> > > > > > > > >> > > > > > >> > > > SPI
> >> > > > > > > > >> > > > > > >> > > > > > > with
> >> > > > > > > > >> > > > > > >> > > > > > > > > both
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > write
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> and
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >> read
> methods
> >> at
> >> > > the
> >> > > > > > > > >> > > MetaStoreManager
> >> > > > > > > > >> > > > > > level,
> >> > > > > > > > >> > > > > > >> > > where
> >> > > > > > > > >> > > > > > >> > > > > any
> >> > > > > > > > >> > > > > > >> > > > > > > > > backend
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > (JDBC,
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> NoSQL,
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >> remote) can
> >> > > implement
> >> > > > > > > them.
> >> > > > > > > > >> > > > > > >> MetricsPersistence
> >> > > > > > > > >> > > > > > >> > > on
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > BasePersistence
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> goes
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >> away. Where
> >> > > metrics
> >> > > > > > > actually
> >> > > > > > > > >> land
> >> > > > > > > > >> > > is an
> >> > > > > > > > >> > > > > > >> > > > > implementation
> >> > > > > > > > >> > > > > > >> > > > > > > > > detail,
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > not a
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> core
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >> interface.
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >> *Minor
> naming
> >> > > thing*:
> >> > > > > > > > >> > > > > > >> PolarisMetricsReporter
> >> > > > > > > > >> > > > > > >> > is
> >> > > > > > > > >> > > > > > >> > > > > > broader
> >> > > > > > > > >> > > > > > >> > > > > > > > than
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > what
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > it
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >> actually
> >> > handles.
> >> > > It
> >> > > > > > only
> >> > > > > > > > >> accepts
> >> > > > > > > > >> > > > > Iceberg
> >> > > > > > > > >> > > > > > >> REST
> >> > > > > > > > >> > > > > > >> > > > > Catalog
> >> > > > > > > > >> > > > > > >> > > > > > > > > metrics
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> (ScanReport,
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>
> CommitReport
> >> via
> >> > > > > > > > >> MetricsReport).
> >> > > > > > > > >> > > > > Generic
> >> > > > > > > > >> > > > > > >> table
> >> > > > > > > > >> > > > > > >> > > > > metrics
> >> > > > > > > > >> > > > > > >> > > > > > > or
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> operational
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >> metrics
> >> aren't
> >> > in
> >> > > > > scope.
> >> > > > > > > Not
> >> > > > > > > > >> > > blocking,
> >> > > > > > > > >> > > > > > but
> >> > > > > > > > >> > > > > > >> > worth
> >> > > > > > > > >> > > > > > >> > > > > > noting
> >> > > > > > > > >> > > > > > >> > > > > > > if
> >> > > > > > > > >> > > > > > >> > > > > > > > > the
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> metrics
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >> surface
> >> expands.
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >> *Rough
> >> sketch of
> >> > > how
> >> > > > > to
> >> > > > > > > get
> >> > > > > > > > >> > there*:
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>  1.  Add
> read
> >> > > methods
> >> > > > > to
> >> > > > > > > > >> > > > > > >> PolarisMetricsManager
> >> > > > > > > > >> > > > > > >> > > > > > > > > > (listScanReports,
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>
> >> > listCommitReports)
> >> > > > > with
> >> > > > > > > > >> default
> >> > > > > > > > >> > > no-op,
> >> > > > > > > > >> > > > > > >> same as
> >> > > > > > > > >> > > > > > >> > > the
> >> > > > > > > > >> > > > > > >> > > > > > > > existing
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > write
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> methods.
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >> (Probably
> >> make
> >> > > > > > > > >> > > PolarisMetricsManager
> >> > > > > > > > >> > > > > more
> >> > > > > > > > >> > > > > > >> > > explicit
> >> > > > > > > > >> > > > > > >> > > > > on
> >> > > > > > > > >> > > > > > >> > > > > > > > being
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > Iceberg
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >> specific
> like
> >> > > package
> >> > > > > > > name or
> >> > > > > > > > >> > class
> >> > > > > > > > >> > > > > name
> >> > > > > > > > >> > > > > > >> etc.)
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>  2.  Wire
> >> > > > > > > > >> MetricsReportsService
> >> > > > > > > > >> > > through
> >> > > > > > > > >> > > > > > >> > > > > > MetaStoreManager
> >> > > > > > > > >> > > > > > >> > > > > > > > > > instead
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > of
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>
> >> > > > > > > callContext.getMetaStore().
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>  3.
> Extract
> >> > > metrics
> >> > > > > > > > >> persistence
> >> > > > > > > > >> > > from
> >> > > > > > > > >> > > > > > >> > > > > > > > > JdbcBasePersistenceImpl
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > into
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> its
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >> own class.
> >> That
> >> > > file
> >> > > > > > > carries
> >> > > > > > > > >> ~7
> >> > > > > > > > >> > > > > > >> > > responsibilities,
> >> > > > > > > > >> > > > > > >> > > > > > > metrics
> >> > > > > > > > >> > > > > > >> > > > > > > > > > being
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > one
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> of
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> them.
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>  4.  Remove
> >> > > > > > > MetricsPersistence
> >> > > > > > > > >> > from
> >> > > > > > > > >> > > > > > >> > > > BasePersistence.
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >> *None of
> this
> >> > > needs to
> >> > > > > > > happen
> >> > > > > > > > >> in
> >> > > > > > > > >> > > #4115.
> >> > > > > > > > >> > > > > > >> But if
> >> > > > > > > > >> > > > > > >> > > the
> >> > > > > > > > >> > > > > > >> > > > > > > > direction
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > makes
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> sense,
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >> it would be
> >> good
> >> > > to
> >> > > > > > align
> >> > > > > > > > >> before
> >> > > > > > > > >> > > the
> >> > > > > > > > >> > > > > > >> metrics
> >> > > > > > > > >> > > > > > >> > > > surface
> >> > > > > > > > >> > > > > > >> > > > > > > grows
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > further.
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> Curious
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >> what others
> >> > > think.*
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >> *My mental
> >> model
> >> > > > > note*:
> >> > > > > > > Level
> >> > > > > > > > >> 1
> >> > > > > > > > >> > > > > > >> > > MetaStoreManager;
> >> > > > > > > > >> > > > > > >> > > > > > level
> >> > > > > > > > >> > > > > > >> > > > > > > 2
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> transactional
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>
> persistence;
> >> > > level 3
> >> > > > > > base
> >> > > > > > > > >> > > persistence
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >> Diagram 1
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >> <
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >>
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >>
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > >
> >> > > > > > > > >> > > > > > >> > > > > > > > > > >
> >> > > > > > > > >> > > > > > >> > > > > > > > > >
> >> > > > > > > > >> > > > > > >> > > > > > > > >
> >> > > > > > > > >> > > > > > >> > > > > > > >
> >> > > > > > > > >> > > > > > >> > > > > > >
> >> > > > > > > > >> > > > > > >> > > > > >
> >> > > > > > > > >> > > > > > >> > > > >
> >> > > > > > > > >> > > > > > >> > > >
> >> > > > > > > > >> > > > > > >> > >
> >> > > > > > > > >> > > > > > >> >
> >> > > > > > > > >> > > > > > >>
> >> > > > > > > > >> > > > > >
> >> > > > > > > > >> > > > >
> >> > > > > > > > >> > >
> >> > > > > > > > >> >
> >> > > > > > > > >>
> >> > > > > > >
> >> > > > > >
> >> > > > >
> >> > >
> >> >
> >>
> https://urldefense.com/v3/__https://www.plantuml.com/plantuml/uml/bLHDR-Cs4BthLmpIYupw0zbkKQ1r3M-S7Bp8xhhM7WCOb3IM65EaGD9EX2RzxHrHb4CxRelwa4YSDu_lpOVcnZ9jzvM8BBS2uGjQpJC3dtHMSekPtMk44IpsMgEqa5XcCOhCZikQQLP1pR8TAp2n3ILhmZDP20m0fcIvUkAoW2qJXd9z1bpToO9BX3WXu0ucy5rpgGPNm0nW5_epUWtm2Ue3pn3kMOFQmKntGZW0BYtgBSi8k5A2QMwybJNMIbFiGSR9QZc4nUqIvikStF0jHprua5C-amge42aNt3R0f5JaaoivdV2Pkqbx4hee4ymOkBh5BTiB-_uIeGeo8zL8rPsPl4DktdEiK1jkB1NdZCRbrSTecDe_mlHbF0wvBmCkaOH5_S8a_TTTKI6-nmCAkEw4LpxsZ-LbYLKQFKMNOgf_wuM7_bV9gOer5SYMMksBSWXFcbi49KNZXNLicwfe3TETC7gPdPqI7uBcHMb1RSzYq34c6PDUM9mn8HRsUTZEiDBve3NjVZumBj0U7SS37mGO7vcwtiK-_pU7U7L_f-digo9YbhSwIfMRwIITKGXbxdIUTCGF1SeCJxloKsU-3k9ddRbX1eDq1q_fx1JbBGT0glVyXimDuP4TQ5qpCAmnGEj2s_6n5mtn1z-97-63itFQZLPO1Ev2tu_WF7Ju-VPc0Skg5bYXxBhkY1xpD7EM_7fyflSpIsqMgVth5xhVr4eQxWQ8enaSAJQSG16yFSDuJ798rrcXr_3n-lfdk7icQjEBmFujL7AodiP_Y4Z7-YxvtZNs4zMgpNTl6tF8sglyPsmqchrjvQ-m-aP94r-TwCA2Ka8upPJZwtvSpoYCXkYMZU2NXvRMBfq9P3i3Le4VAZUAlUZ_oPKsxPgY0Q_BSKLkyr9bhQhQrJjo_x3TPlIB0DPjnMfcIoYP0QaYw1a0fTKDr8fB6ntNuvmoL1ZGkXa69Njh43zf9GiGxHQrA_jDYWRSzF5--WmTVrN97_Sm8LbLUy_lGBmLanJjFkDlGkRqjA_4tm00__;!!Iz9xO38YGHZK!4rnlVIqBQQQ6xpGe1PMV58edOS_cGrKoqdfhw4qSp_JQCavvEjaxXk0OwGPnxyEcO60AzKQK_XDtp7H3DdkFXcrXuTBISHg$
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >> :
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >> [image:
> >> > image.png]
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >> Diagram 2
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >> <
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >>
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >>
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > >
> >> > > > > > > > >> > > > > > >> > > > > > > > > > >
> >> > > > > > > > >> > > > > > >> > > > > > > > > >
> >> > > > > > > > >> > > > > > >> > > > > > > > >
> >> > > > > > > > >> > > > > > >> > > > > > > >
> >> > > > > > > > >> > > > > > >> > > > > > >
> >> > > > > > > > >> > > > > > >> > > > > >
> >> > > > > > > > >> > > > > > >> > > > >
> >> > > > > > > > >> > > > > > >> > > >
> >> > > > > > > > >> > > > > > >> > >
> >> > > > > > > > >> > > > > > >> >
> >> > > > > > > > >> > > > > > >>
> >> > > > > > > > >> > > > > >
> >> > > > > > > > >> > > > >
> >> > > > > > > > >> > >
> >> > > > > > > > >> >
> >> > > > > > > > >>
> >> > > > > > >
> >> > > > > >
> >> > > > >
> >> > >
> >> >
> >>
> https://urldefense.com/v3/__https://www.plantuml.com/plantuml/uml/VLLDR-8m4BtdLupO2sWBLVU8AaGB7AXAbssGzb896SS9RXqxjKqBwkv_tt7iV43fdaZYDpFlpRmnOsE9jhjSH9PRmM31hERKm8scMsuPjJlDe0yheZDc8RR4iYWoBrmMH9CS2a9VICPYUy1OZN0YCy5Q0BCbYNhdCeEK28En8G8wCvbnoQ0R8_05Bc6bkLIz3X03p1zzH7zR-9ZfDquPt9C3qoNCX2yV4G2NbkcKu5jdgGJHt0GbZwnG6i-UP3TUpk5gM6Ldqke350eZUqzoCft3U9xWHvxoa5-7K4nF1J46EbEMafsmdrCBbQ44gVggy18IZrn_ph5asd1ZiIKdQSgueZvjXrQFSFrdC3YN-nXmBacxbGiYyLVxLaBtdhqn0LSzdBDhqQtQoOJeGyad3z0lUqnYgpGB6Ns8oVyta00Dy_WnX0tIOZ8v6SYxHll1TrH6aejAik-mh-AphVFCwSUQqFypElag5QRGFDjQKEd96K1P8QP41c9TzA_IIQyvdAWyv_RSiS3skb0_EzDDkK2v5xWF6MiGFlvhpFLcD2Dq2pml14gaF67eQkmd8gulDoC4kSOu6KVpkvlUJg1RTbWISU40RdBUUS_9XfRZ2dwxm_SW8LYFISgm_MnlDQ6M9P1gbKEc4X-2pH_FvJCkCqm9pbVjD6LrwdLeOrDWfOaqc8Wh9BE85oNKxkNQ6o4yGRy_Eae0G_G8tZv81d3bHDB23WOdisohVr3nh_j6lbSjbNaLRTc8UgtPbAU1J_tygOfZX9DWEJeHDvYx-qmSi5FgNLPZwHrHcUsncGQ5-skhUclpE5fo4ounpFauYrUbkU6ccfnxMvitwag4IyerhTxj8In_Oj1bDO4pQru674loYrGlULHLEGCjwJJ8gDoVZR8MxO4BT3IzRvIcAQKezC6xpziGnTyImrfEGyJI_OcKfgtxIvnTqFEMS17L9Z-jsARN5FmTheP7HtSdtOMT0B4GY2FYHXxgQmMtj2bRqiLFGapiVe1_QVKDrkqXcm83aFEXnMYCZ-xlyHy__;!!Iz9xO38YGHZK!4rnlVIqBQQQ6xpGe1PMV58edOS_cGrKoqdfhw4qSp_JQCavvEjaxXk0OwGPnxyEcO60AzKQK_XDtp7H3DdkFXcrXSpmqcNk$
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >> :
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >> [image:
> >> > image.png]
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>  -ej
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >> On Wed, Apr
> >> 15,
> >> > > 2026
> >> > > > > at
> >> > > > > > > > >> 8:22 AM
> >> > > > > > > > >> > > Dmitri
> >> > > > > > > > >> > > > > > >> > > > Bourlatchkov
> >> > > > > > > > >> > > > > > >> > > > > <
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> dimas@apache.org
> >
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >> wrote:
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> Hi All,
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>>
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> Heads up:
> >> The
> >> > > current
> >> > > > > > > state
> >> > > > > > > > >> of
> >> > > > > > > > >> > PR
> >> > > > > > > > >> > > > > [4115]
> >> > > > > > > > >> > > > > > >> > looks
> >> > > > > > > > >> > > > > > >> > > > > pretty
> >> > > > > > > > >> > > > > > >> > > > > > > > solid
> >> > > > > > > > >> > > > > > >> > > > > > > > > > to
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > me.
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> I
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> believe
> >> this PR
> >> > > is
> >> > > > > > > > >> approaching a
> >> > > > > > > > >> > > > > > mergeable
> >> > > > > > > > >> > > > > > >> > > > > condition.
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>>
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> Please
> post
> >> > your
> >> > > > > > reviews
> >> > > > > > > if
> >> > > > > > > > >> you
> >> > > > > > > > >> > > have
> >> > > > > > > > >> > > > > any
> >> > > > > > > > >> > > > > > >> > > > comments.
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>>
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> [4115]
> >> > > > > > > > >> > > > > > >> > > >
> >> > > > > > > > >> >
> >> > > > > > > > >>
> >> > > > > > >
> >> > > > > >
> >> > > > >
> >> > >
> >> >
> >>
> https://urldefense.com/v3/__https://github.com/apache/polaris/pull/4115__;!!Iz9xO38YGHZK!4rnlVIqBQQQ6xpGe1PMV58edOS_cGrKoqdfhw4qSp_JQCavvEjaxXk0OwGPnxyEcO60AzKQK_XDtp7H3DdkFXcrXtFRscKs$
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>>
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> Thanks,
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> Dmitri.
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>>
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> On Tue,
> Mar
> >> 3,
> >> > > 2026
> >> > > > > at
> >> > > > > > > > >> 3:29 PM
> >> > > > > > > > >> > > Anand
> >> > > > > > > > >> > > > > > Kumar
> >> > > > > > > > >> > > > > > >> > > > Sankaran
> >> > > > > > > > >> > > > > > >> > > > > > via
> >> > > > > > > > >> > > > > > >> > > > > > > > > dev <
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>>
> >> > > > > dev@polaris.apache.org
> >> > > > > > >
> >> > > > > > > > >> wrote:
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>>
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > Hi Yufei
> >> and
> >> > > > > Dmitri,
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> >
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > Here is
> a
> >> > > proposal
> >> > > > > > for
> >> > > > > > > the
> >> > > > > > > > >> > REST
> >> > > > > > > > >> > > > > > >> endpoints
> >> > > > > > > > >> > > > > > >> > for
> >> > > > > > > > >> > > > > > >> > > > > > metrics
> >> > > > > > > > >> > > > > > >> > > > > > > > and
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > events.
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> >
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> >
> >> > > > > > > > >> > > > > > >> > > > > >
> >> > > > > > > > >> >
> >> > > > > > > > >>
> >> > > > > > >
> >> > > > > >
> >> > > > >
> >> > >
> >> >
> >>
> https://urldefense.com/v3/__https://github.com/apache/polaris/pull/3924/changes__;!!Iz9xO38YGHZK!4rnlVIqBQQQ6xpGe1PMV58edOS_cGrKoqdfhw4qSp_JQCavvEjaxXk0OwGPnxyEcO60AzKQK_XDtp7H3DdkFXcrXcjoPhSs$
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> >
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > I did
> not
> >> see
> >> > > any
> >> > > > > > > > >> precursors
> >> > > > > > > > >> > for
> >> > > > > > > > >> > > > > > >> raising a
> >> > > > > > > > >> > > > > > >> > PR
> >> > > > > > > > >> > > > > > >> > > > for
> >> > > > > > > > >> > > > > > >> > > > > > > > > > proposals,
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > so
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> trying
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > this.
> >> Please
> >> > > let
> >> > > > > me
> >> > > > > > > know
> >> > > > > > > > >> what
> >> > > > > > > > >> > > you
> >> > > > > > > > >> > > > > > >> think.
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> >
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > -
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > Anand
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> >
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > From:
> >> Anand
> >> > > Kumar
> >> > > > > > > Sankaran
> >> > > > > > > > >> <
> >> > > > > > > > >> > > > > > >> > > > > > > anand.sankaran@workday.com
> >> > > > > > > > >> > > > > > >> > > > > > > > >
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > Date:
> >> Monday,
> >> > > March
> >> > > > > > 2,
> >> > > > > > > > >> 2026 at
> >> > > > > > > > >> > > > > > 10:25 AM
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > To:
> >> > > > > > > dev@polaris.apache.org
> >> > > > > > > > >> <
> >> > > > > > > > >> > > > > > >> > > > > dev@polaris.apache.org
> >> > > > > > > > >> > > > > > >> > > > > > >
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > Subject:
> >> Re:
> >> > > > > Polaris
> >> > > > > > > > >> Telemetry
> >> > > > > > > > >> > > and
> >> > > > > > > > >> > > > > > Audit
> >> > > > > > > > >> > > > > > >> > > Trail
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> >
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > About
> the
> >> > REST
> >> > > API,
> >> > > > > > > based
> >> > > > > > > > >> on
> >> > > > > > > > >> > my
> >> > > > > > > > >> > > use
> >> > > > > > > > >> > > > > > >> cases:
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> >
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> >
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> >   1.
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > I want
> to
> >> be
> >> > > able
> >> > > > > to
> >> > > > > > > query
> >> > > > > > > > >> > > commit
> >> > > > > > > > >> > > > > > >> metrics
> >> > > > > > > > >> > > > > > >> > to
> >> > > > > > > > >> > > > > > >> > > > > track
> >> > > > > > > > >> > > > > > >> > > > > > > > files
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > added
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > /
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> removed
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > per
> >> commit,
> >> > > along
> >> > > > > > with
> >> > > > > > > > >> record
> >> > > > > > > > >> > > > > counts.
> >> > > > > > > > >> > > > > > >> The
> >> > > > > > > > >> > > > > > >> > > > > ingestion
> >> > > > > > > > >> > > > > > >> > > > > > > > > > pipeline
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > that
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> writes
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > this
> data
> >> is
> >> > > owned
> >> > > > > by
> >> > > > > > > us
> >> > > > > > > > >> and
> >> > > > > > > > >> > we
> >> > > > > > > > >> > > are
> >> > > > > > > > >> > > > > > >> > > guaranteed
> >> > > > > > > > >> > > > > > >> > > > to
> >> > > > > > > > >> > > > > > >> > > > > > > write
> >> > > > > > > > >> > > > > > >> > > > > > > > > > this
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>>
> information
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > for each
> >> > write.
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> >   2.
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > I want
> to
> >> be
> >> > > able
> >> > > > > to
> >> > > > > > > query
> >> > > > > > > > >> > scan
> >> > > > > > > > >> > > > > > metrics
> >> > > > > > > > >> > > > > > >> for
> >> > > > > > > > >> > > > > > >> > > > > read. I
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > understand
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> clients
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> do
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > not
> >> fulfill
> >> > > this
> >> > > > > > > > >> requirement.
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> >   3.
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > I want
> to
> >> be
> >> > > able
> >> > > > > to
> >> > > > > > > query
> >> > > > > > > > >> the
> >> > > > > > > > >> > > > > events
> >> > > > > > > > >> > > > > > >> table
> >> > > > > > > > >> > > > > > >> > > > > (events
> >> > > > > > > > >> > > > > > >> > > > > > > are
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> persisted) -
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> this
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > may
> >> supersede
> >> > > #2, I
> >> > > > > > am
> >> > > > > > > not
> >> > > > > > > > >> > sure
> >> > > > > > > > >> > > yet.
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> >
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > All this
> >> > > > > information
> >> > > > > > > is in
> >> > > > > > > > >> the
> >> > > > > > > > >> > > JDBC
> >> > > > > > > > >> > > > > > >> based
> >> > > > > > > > >> > > > > > >> > > > > > persistence
> >> > > > > > > > >> > > > > > >> > > > > > > > > model
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > and
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> is
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> >
> persisted
> >> in
> >> > > the
> >> > > > > > > > >> metastore. I
> >> > > > > > > > >> > > > > > currently
> >> > > > > > > > >> > > > > > >> > don’t
> >> > > > > > > > >> > > > > > >> > > > > have
> >> > > > > > > > >> > > > > > >> > > > > > a
> >> > > > > > > > >> > > > > > >> > > > > > > > need
> >> > > > > > > > >> > > > > > >> > > > > > > > > > to
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> query
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> >
> >> prometheus or
> >> > > open
> >> > > > > > > > >> telemetry.
> >> > > > > > > > >> > I
> >> > > > > > > > >> > > do
> >> > > > > > > > >> > > > > > >> publish
> >> > > > > > > > >> > > > > > >> > > some
> >> > > > > > > > >> > > > > > >> > > > > > > events
> >> > > > > > > > >> > > > > > >> > > > > > > > to
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> Prometheus
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> and
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > they are
> >> > > forwarded
> >> > > > > to
> >> > > > > > > our
> >> > > > > > > > >> > > dashboards
> >> > > > > > > > >> > > > > > >> > > elsewhere.
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> >
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > About
> the
> >> CLI
> >> > > > > > > utilities, I
> >> > > > > > > > >> > > meant the
> >> > > > > > > > >> > > > > > >> admin
> >> > > > > > > > >> > > > > > >> > > user
> >> > > > > > > > >> > > > > > >> > > > > > > > > utilities.
> >> > > > > > > > >> > > > > > >> > > > > > > > > > In
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> one of
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> the
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > earliest
> >> > > drafts of
> >> > > > > my
> >> > > > > > > > >> > proposal,
> >> > > > > > > > >> > > > > > Prashant
> >> > > > > > > > >> > > > > > >> > > > > mentioned
> >> > > > > > > > >> > > > > > >> > > > > > > that
> >> > > > > > > > >> > > > > > >> > > > > > > > > the
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> metrics
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> tables
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > can grow
> >> > > > > indefinitely
> >> > > > > > > and
> >> > > > > > > > >> > that a
> >> > > > > > > > >> > > > > > similar
> >> > > > > > > > >> > > > > > >> > > > problem
> >> > > > > > > > >> > > > > > >> > > > > > > exists
> >> > > > > > > > >> > > > > > >> > > > > > > > > > with
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > the
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> events
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > table as
> >> > well.
> >> > > We
> >> > > > > > > discussed
> >> > > > > > > > >> > that
> >> > > > > > > > >> > > > > > >> cleaning
> >> > > > > > > > >> > > > > > >> > up
> >> > > > > > > > >> > > > > > >> > > of
> >> > > > > > > > >> > > > > > >> > > > > old
> >> > > > > > > > >> > > > > > >> > > > > > > > > records
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > from
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> both
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > metrics
> >> > tables
> >> > > and
> >> > > > > > > events
> >> > > > > > > > >> > > tables can
> >> > > > > > > > >> > > > > > be
> >> > > > > > > > >> > > > > > >> > done
> >> > > > > > > > >> > > > > > >> > > > via
> >> > > > > > > > >> > > > > > >> > > > > a
> >> > > > > > > > >> > > > > > >> > > > > > > CLI
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > utility.
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> >
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > I see
> that
> >> > > Yufei
> >> > > > > has
> >> > > > > > > > >> covered
> >> > > > > > > > >> > the
> >> > > > > > > > >> > > > > > >> discussion
> >> > > > > > > > >> > > > > > >> > > > about
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > datasources.
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> >
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > -
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > Anand
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> >
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> >
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> >
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > From:
> >> Yufei
> >> > Gu
> >> > > <
> >> > > > > > > > >> > > > > flyrain000@gmail.com>
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > Date:
> >> Friday,
> >> > > > > > February
> >> > > > > > > 27,
> >> > > > > > > > >> > 2026
> >> > > > > > > > >> > > at
> >> > > > > > > > >> > > > > > >> 9:54 PM
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > To:
> >> > > > > > > dev@polaris.apache.org
> >> > > > > > > > >> <
> >> > > > > > > > >> > > > > > >> > > > > dev@polaris.apache.org
> >> > > > > > > > >> > > > > > >> > > > > > >
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > Subject:
> >> Re:
> >> > > > > Polaris
> >> > > > > > > > >> Telemetry
> >> > > > > > > > >> > > and
> >> > > > > > > > >> > > > > > Audit
> >> > > > > > > > >> > > > > > >> > > Trail
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> >
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > This
> >> Message
> >> > Is
> >> > > > > From
> >> > > > > > an
> >> > > > > > > > >> > External
> >> > > > > > > > >> > > > > > Sender
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > This
> >> message
> >> > > came
> >> > > > > > from
> >> > > > > > > > >> outside
> >> > > > > > > > >> > > your
> >> > > > > > > > >> > > > > > >> > > > organization.
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > Report
> >> > > Suspicious<
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> >
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>>
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >>
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >>
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > >
> >> > > > > > > > >> > > > > > >> > > > > > > > > > >
> >> > > > > > > > >> > > > > > >> > > > > > > > > >
> >> > > > > > > > >> > > > > > >> > > > > > > > >
> >> > > > > > > > >> > > > > > >> > > > > > > >
> >> > > > > > > > >> > > > > > >> > > > > > >
> >> > > > > > > > >> > > > > > >> > > > > >
> >> > > > > > > > >> > > > > > >> > > > >
> >> > > > > > > > >> > > > > > >> > > >
> >> > > > > > > > >> > > > > > >> > >
> >> > > > > > > > >> > > > > > >> >
> >> > > > > > > > >> > > > > > >>
> >> > > > > > > > >> > > > > >
> >> > > > > > > > >> > > > >
> >> > > > > > > > >> > >
> >> > > > > > > > >> >
> >> > > > > > > > >>
> >> > > > > > >
> >> > > > > >
> >> > > > >
> >> > >
> >> >
> >>
> https://us-phishalarm-ewt.proofpoint.com/EWT/v1/Iz9xO38YGHZK!YhNDZABkHi1B699ote2uMwpOZw8i0QMCGO2Szc-HshuABGhGvwPJcymE6G2oUUxtS8xDkSrtGTPm_I3QnVDHoLMk50m9v8z_nZKTkd-bnVUbreF1u0WnfV_X5eYevZl_$
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > >
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> >
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> >
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > As I
> >> > mentioned
> >> > > in
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> >
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>>
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >>
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >>
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > >
> >> > > > > > > > >> > > > > > >> > > > > > > > > > >
> >> > > > > > > > >> > > > > > >> > > > > > > > > >
> >> > > > > > > > >> > > > > > >> > > > > > > > >
> >> > > > > > > > >> > > > > > >> > > > > > > >
> >> > > > > > > > >> > > > > > >> > > > > > >
> >> > > > > > > > >> > > > > > >> > > > > >
> >> > > > > > > > >> > > > > > >> > > > >
> >> > > > > > > > >> > > > > > >> > > >
> >> > > > > > > > >> > > > > > >> > >
> >> > > > > > > > >> > > > > > >> >
> >> > > > > > > > >> > > > > > >>
> >> > > > > > > > >> > > > > >
> >> > > > > > > > >> > > > >
> >> > > > > > > > >> > >
> >> > > > > > > > >> >
> >> > > > > > > > >>
> >> > > > > > >
> >> > > > > >
> >> > > > >
> >> > >
> >> >
> >>
> https://urldefense.com/v3/__https://github.com/apache/polaris/issues/3890__;!!Iz9xO38YGHZK!5EuyFFkk3vhRWVIRvQAWBSQfpJkTMA9HxugzDwXmN0LPPqhEFxYkFRGVhtb8AqUwXtDh2OplcMnbMDHKOxrvDU0$
> >> > > > > > > > >> > >> > > >> > > > > > > > > > > > >> >> >>> ,
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> >
> supporting
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > multiple
> >> data
> >> > > > > sources
> >> > > > > > > is
> >> > > > > > > > >> not a
> >> > > > > > > > >> > > > > trivial
> >> > > > > > > > >> > > > > > >> > > change.
> >> > > > > > > > >> > > > > > >> > > > I
> >> > > > > > > > >> > > > > > >> > > > > > > would
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > strongly
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> recommend
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > starting
> >> > with a
> >> > > > > > design
> >> > > > > > > > >> > document
> >> > > > > > > > >> > > to
> >> > > > > > > > >> > > > > > >> > carefully
> >> > > > > > > > >> > > > > > >> > > > > > evaluate
> >> > > > > > > > >> > > > > > >> > > > > > > > the
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> architectural
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> >
> >> implications
> >> > > and
> >> > > > > long
> >> > > > > > > term
> >> > > > > > > > >> > > impact.
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> >
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > A REST
> >> > > endpoint to
> >> > > > > > > query
> >> > > > > > > > >> > metrics
> >> > > > > > > > >> > > > > seems
> >> > > > > > > > >> > > > > > >> > > > reasonable
> >> > > > > > > > >> > > > > > >> > > > > > > given
> >> > > > > > > > >> > > > > > >> > > > > > > > > the
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> current
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> JDBC
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > based
> >> > > persistence
> >> > > > > > > model.
> >> > > > > > > > >> That
> >> > > > > > > > >> > > said,
> >> > > > > > > > >> > > > > we
> >> > > > > > > > >> > > > > > >> may
> >> > > > > > > > >> > > > > > >> > > also
> >> > > > > > > > >> > > > > > >> > > > > > > > consider
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> alternative
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > storage
> >> > > models. For
> >> > > > > > > > >> example,
> >> > > > > > > > >> > if
> >> > > > > > > > >> > > we
> >> > > > > > > > >> > > > > > later
> >> > > > > > > > >> > > > > > >> > > adopt
> >> > > > > > > > >> > > > > > >> > > > a
> >> > > > > > > > >> > > > > > >> > > > > > time
> >> > > > > > > > >> > > > > > >> > > > > > > > > > series
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> system
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> such as
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> >
> >> Prometheus to
> >> > > store
> >> > > > > > > > >> metrics,
> >> > > > > > > > >> > the
> >> > > > > > > > >> > > > > query
> >> > > > > > > > >> > > > > > >> > model
> >> > > > > > > > >> > > > > > >> > > > and
> >> > > > > > > > >> > > > > > >> > > > > > > access
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > patterns
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> would
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> be
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> >
> >> fundamentally
> >> > > > > > > different.
> >> > > > > > > > >> > > Designing
> >> > > > > > > > >> > > > > the
> >> > > > > > > > >> > > > > > >> REST
> >> > > > > > > > >> > > > > > >> > > API
> >> > > > > > > > >> > > > > > >> > > > > > > without
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> considering
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> these
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> >
> potential
> >> > > > > evolutions
> >> > > > > > > may
> >> > > > > > > > >> limit
> >> > > > > > > > >> > > > > > >> flexibility.
> >> > > > > > > > >> > > > > > >> > > I'd
> >> > > > > > > > >> > > > > > >> > > > > > > suggest
> >> > > > > > > > >> > > > > > >> > > > > > > > > to
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > start
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> with
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> the
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > use
> case.
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> >
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > Yufei
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> >
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> >
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > On Fri,
> >> Feb
> >> > 27,
> >> > > > > 2026
> >> > > > > > at
> >> > > > > > > > >> > 3:42 PM
> >> > > > > > > > >> > > > > Dmitri
> >> > > > > > > > >> > > > > > >> > > > > > Bourlatchkov <
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >>
> >> dimas@apache.org>
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > wrote:
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> >
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > > Hi
> >> Anand,
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > >
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > >
> Sharing
> >> my
> >> > > > > view...
> >> > > > > > > > >> subject
> >> > > > > > > > >> > to
> >> > > > > > > > >> > > > > > >> discussion:
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > >
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > > 1.
> >> Adding
> >> > > non-IRC
> >> > > > > > > REST
> >> > > > > > > > >> API
> >> > > > > > > > >> > to
> >> > > > > > > > >> > > > > > Polaris
> >> > > > > > > > >> > > > > > >> is
> >> > > > > > > > >> > > > > > >> > > > > > perfectly
> >> > > > > > > > >> > > > > > >> > > > > > > > > fine.
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > >
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > >
> Figuring
> >> > out
> >> > > > > > specific
> >> > > > > > > > >> > endpoint
> >> > > > > > > > >> > > > > URIs
> >> > > > > > > > >> > > > > > >> and
> >> > > > > > > > >> > > > > > >> > > > > payloads
> >> > > > > > > > >> > > > > > >> > > > > > > > might
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > require
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> a
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> few
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > >
> >> roundtrips,
> >> > > so
> >> > > > > > > opening a
> >> > > > > > > > >> > > separate
> >> > > > > > > > >> > > > > > >> thread
> >> > > > > > > > >> > > > > > >> > > for
> >> > > > > > > > >> > > > > > >> > > > > that
> >> > > > > > > > >> > > > > > >> > > > > > > > might
> >> > > > > > > > >> > > > > > >> > > > > > > > > > be
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> best.
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > >
> >> > Contributors
> >> > > > > > commonly
> >> > > > > > > > >> create
> >> > > > > > > > >> > > > > Google
> >> > > > > > > > >> > > > > > >> Docs
> >> > > > > > > > >> > > > > > >> > > for
> >> > > > > > > > >> > > > > > >> > > > > new
> >> > > > > > > > >> > > > > > >> > > > > > > API
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > proposals
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> too
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> (they
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > > fairly
> >> easy
> >> > > to
> >> > > > > > > update as
> >> > > > > > > > >> the
> >> > > > > > > > >> > > email
> >> > > > > > > > >> > > > > > >> > > discussion
> >> > > > > > > > >> > > > > > >> > > > > > > > > > progresses).
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > >
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > > There
> >> was a
> >> > > > > > > suggestion to
> >> > > > > > > > >> > try
> >> > > > > > > > >> > > > > > Markdown
> >> > > > > > > > >> > > > > > >> > > (with
> >> > > > > > > > >> > > > > > >> > > > > PRs)
> >> > > > > > > > >> > > > > > >> > > > > > > for
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > proposals
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> [1]
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> ...
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > > feel
> >> free
> >> > to
> >> > > give
> >> > > > > > it
> >> > > > > > > a
> >> > > > > > > > >> try
> >> > > > > > > > >> > if
> >> > > > > > > > >> > > you
> >> > > > > > > > >> > > > > > are
> >> > > > > > > > >> > > > > > >> > > > > comfortable
> >> > > > > > > > >> > > > > > >> > > > > > > > with
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > that.
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > >
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > > 2.
> Could
> >> > you
> >> > > > > > clarify
> >> > > > > > > > >> whether
> >> > > > > > > > >> > > you
> >> > > > > > > > >> > > > > > mean
> >> > > > > > > > >> > > > > > >> end
> >> > > > > > > > >> > > > > > >> > > > user
> >> > > > > > > > >> > > > > > >> > > > > > > > > utilities
> >> > > > > > > > >> > > > > > >> > > > > > > > > > or
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> admin
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> user
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > >
> >> utilities?
> >> > > In the
> >> > > > > > > latter
> >> > > > > > > > >> > case
> >> > > > > > > > >> > > > > those
> >> > > > > > > > >> > > > > > >> might
> >> > > > > > > > >> > > > > > >> > > be
> >> > > > > > > > >> > > > > > >> > > > > more
> >> > > > > > > > >> > > > > > >> > > > > > > > > > suitable
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > for
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> the
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> Admin
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > > CLI
> >> (java)
> >> > > not
> >> > > > > the
> >> > > > > > > Python
> >> > > > > > > > >> > CLI,
> >> > > > > > > > >> > > > > IMHO.
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > >
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > > Why
> >> would
> >> > > these
> >> > > > > > > > >> utilities be
> >> > > > > > > > >> > > > > common
> >> > > > > > > > >> > > > > > >> with
> >> > > > > > > > >> > > > > > >> > > > > events?
> >> > > > > > > > >> > > > > > >> > > > > > > > IMHO,
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > event
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> use
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> cases
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > are
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > >
> distinct
> >> > from
> >> > > > > > > scan/commit
> >> > > > > > > > >> > > metrics.
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > >
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > > 3. I'd
> >> > prefer
> >> > > > > > > separating
> >> > > > > > > > >> > > metrics
> >> > > > > > > > >> > > > > > >> > > persistence
> >> > > > > > > > >> > > > > > >> > > > > from
> >> > > > > > > > >> > > > > > >> > > > > > > > > > MetaStore
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>>
> persistence
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > at
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > > the
> code
> >> > > level,
> >> > > > > so
> >> > > > > > > that
> >> > > > > > > > >> they
> >> > > > > > > > >> > > could
> >> > > > > > > > >> > > > > > be
> >> > > > > > > > >> > > > > > >> > mixed
> >> > > > > > > > >> > > > > > >> > > > and
> >> > > > > > > > >> > > > > > >> > > > > > > > matched
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>>
> >> independently.
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > The
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > >
> separate
> >> > > > > datasource
> >> > > > > > > > >> question
> >> > > > > > > > >> > > will
> >> > > > > > > > >> > > > > > >> become
> >> > > > > > > > >> > > > > > >> > a
> >> > > > > > > > >> > > > > > >> > > > > > > non-issue
> >> > > > > > > > >> > > > > > >> > > > > > > > > with
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > that
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> approach,
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > I
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > > guess.
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > >
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > > The
> >> > > rationale for
> >> > > > > > > > >> separating
> >> > > > > > > > >> > > scan
> >> > > > > > > > >> > > > > > >> metrics
> >> > > > > > > > >> > > > > > >> > > and
> >> > > > > > > > >> > > > > > >> > > > > > > > metastore
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> persistence
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> is
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > that
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > >
> >> "cascading
> >> > > > > deletes"
> >> > > > > > > > >> between
> >> > > > > > > > >> > > them
> >> > > > > > > > >> > > > > are
> >> > > > > > > > >> > > > > > >> > hardly
> >> > > > > > > > >> > > > > > >> > > > > ever
> >> > > > > > > > >> > > > > > >> > > > > > > > > > required.
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>>
> Furthermore,
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > the
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > > data
> and
> >> > > query
> >> > > > > > > patterns
> >> > > > > > > > >> are
> >> > > > > > > > >> > > very
> >> > > > > > > > >> > > > > > >> > different
> >> > > > > > > > >> > > > > > >> > > so
> >> > > > > > > > >> > > > > > >> > > > > > > > different
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> technologies
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > might
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > > be
> >> > > beneficial in
> >> > > > > > each
> >> > > > > > > > >> case.
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > >
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > > [1]
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> >
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>>
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >>
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >>
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > >
> >> > > > > > > > >> > > > > > >> > > > > > > > > > >
> >> > > > > > > > >> > > > > > >> > > > > > > > > >
> >> > > > > > > > >> > > > > > >> > > > > > > > >
> >> > > > > > > > >> > > > > > >> > > > > > > >
> >> > > > > > > > >> > > > > > >> > > > > > >
> >> > > > > > > > >> > > > > > >> > > > > >
> >> > > > > > > > >> > > > > > >> > > > >
> >> > > > > > > > >> > > > > > >> > > >
> >> > > > > > > > >> > > > > > >> > >
> >> > > > > > > > >> > > > > > >> >
> >> > > > > > > > >> > > > > > >>
> >> > > > > > > > >> > > > > >
> >> > > > > > > > >> > > > >
> >> > > > > > > > >> > >
> >> > > > > > > > >> >
> >> > > > > > > > >>
> >> > > > > > >
> >> > > > > >
> >> > > > >
> >> > >
> >> >
> >>
> https://urldefense.com/v3/__https://lists.apache.org/thread/yto2wp982t43h1mqjwnslswhws5z47cy__;!!Iz9xO38YGHZK!5EuyFFkk3vhRWVIRvQAWBSQfpJkTMA9HxugzDwXmN0LPPqhEFxYkFRGVhtb8AqUwXtDh2OplcMnbMDHKxYDakNU$
> >> > > > > > > > >> > >> > > >> > > > > > > > > > > > >> >> >>> > >
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > >
> Cheers,
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > >
> Dmitri.
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > >
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > > On
> Fri,
> >> Feb
> >> > > 27,
> >> > > > > > 2026
> >> > > > > > > at
> >> > > > > > > > >> > > 6:19 PM
> >> > > > > > > > >> > > > > > Anand
> >> > > > > > > > >> > > > > > >> > Kumar
> >> > > > > > > > >> > > > > > >> > > > > > > Sankaran
> >> > > > > > > > >> > > > > > >> > > > > > > > > via
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > dev
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > <
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > >
> >> > > > > > > dev@polaris.apache.org>
> >> > > > > > > > >> > > wrote:
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > >
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > > >
> Thanks
> >> > all.
> >> > > > > This
> >> > > > > > > PR is
> >> > > > > > > > >> > > merged
> >> > > > > > > > >> > > > > now.
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > > >
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > > > Here
> >> are
> >> > > the
> >> > > > > > > follow-up
> >> > > > > > > > >> > > features
> >> > > > > > > > >> > > > > /
> >> > > > > > > > >> > > > > > >> work
> >> > > > > > > > >> > > > > > >> > > > > needed.
> >> > > > > > > > >> > > > > > >> > > > > > > > These
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > were
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> all
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> part of
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > > the
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > > >
> >> merged PR
> >> > > at
> >> > > > > some
> >> > > > > > > > >> point in
> >> > > > > > > > >> > > time
> >> > > > > > > > >> > > > > > and
> >> > > > > > > > >> > > > > > >> > were
> >> > > > > > > > >> > > > > > >> > > > > > removed
> >> > > > > > > > >> > > > > > >> > > > > > > to
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > reduce
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> scope.
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > > >
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > > >
> Please
> >> > let
> >> > > me
> >> > > > > > know
> >> > > > > > > what
> >> > > > > > > > >> > you
> >> > > > > > > > >> > > > > think.
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > > >
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > > >
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > > >   1.
> >> A
> >> > > REST
> >> > > > > API
> >> > > > > > to
> >> > > > > > > > >> > paginate
> >> > > > > > > > >> > > > > > through
> >> > > > > > > > >> > > > > > >> > > table
> >> > > > > > > > >> > > > > > >> > > > > > > metrics.
> >> > > > > > > > >> > > > > > >> > > > > > > > > > This
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> will be
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > non-IRC
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > > >
> >> standard
> >> > > > > > addition.
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > > >   2.
> >> > > Utilities
> >> > > > > > for
> >> > > > > > > > >> > > managing old
> >> > > > > > > > >> > > > > > >> > records,
> >> > > > > > > > >> > > > > > >> > > > > > should
> >> > > > > > > > >> > > > > > >> > > > > > > be
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > common
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> with
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> events.
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > > >
> There
> >> was
> >> > > some
> >> > > > > > > > >> discussion
> >> > > > > > > > >> > > that
> >> > > > > > > > >> > > > > it
> >> > > > > > > > >> > > > > > >> > belongs
> >> > > > > > > > >> > > > > > >> > > > to
> >> > > > > > > > >> > > > > > >> > > > > > the
> >> > > > > > > > >> > > > > > >> > > > > > > > CLI.
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > > >   3.
> >> > > Separate
> >> > > > > > > > >> datasource
> >> > > > > > > > >> > > > > > (metrics,
> >> > > > > > > > >> > > > > > >> > > events,
> >> > > > > > > > >> > > > > > >> > > > > > even
> >> > > > > > > > >> > > > > > >> > > > > > > > > other
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> tables?).
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > > >
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > > >
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > > >
> >> Anything
> >> > > else?
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > > >
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > > > -
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > > >
> Anand
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > > >
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > > >
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > >
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> >
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> >
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>>
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >>
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >>
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > > >
> >> > > > > > > > >> > > > > > >> > > > > > > > > > > >
> >> > > > > > > > >> > > > > > >> > > > > > > > > > >
> >> > > > > > > > >> > > > > > >> > > > > > > > > >
> >> > > > > > > > >> > > > > > >> > > > > > > > >
> >> > > > > > > > >> > > > > > >> > > > > > > >
> >> > > > > > > > >> > > > > > >> > > > > > >
> >> > > > > > > > >> > > > > > >> > > > > >
> >> > > > > > > > >> > > > > > >> > > > >
> >> > > > > > > > >> > > > > > >> > > >
> >> > > > > > > > >> > > > > > >> > >
> >> > > > > > > > >> > > > > > >> >
> >> > > > > > > > >> > > > > > >>
> >> > > > > > > > >> > > > > > >
> >> > > > > > > > >> > > > > >
> >> > > > > > > > >> > > > >
> >> > > > > > > > >> > >
> >> > > > > > > > >> >
> >> > > > > > > > >> >
> >> > > > > > > > >>
> >> > > > > > > > >
> >> > > > > > >
> >> > > > > >
> >> > > > >
> >> > >
> >> >
> >>
> >
>

Re: Proposal for REST endpoints for table metrics and events

[Dmitri Bourlatchkov <di...@apache.org>]

Hi All,

Heads up: it looks like the "Metrics Architecture" meeting is no longer
necessary. It will be removed from the Polaris calendar presently.

Cheers,
Dmitri.

On Tue, Oct 20, 2026 at 12:16 PM Dmitri Bourlatchkov <di...@apache.org>
wrote:

> Hi All,
>
> PR [4115] has been approved for two weeks without any follow-up comments.
>
> I'm going to merge it on Oct 22 unless fresh concerns are raised.
>
> [4115] https://github.com/apache/polaris/pull/4115
>
> Cheers,
> Dmitri.
>
> I
>
> On Wed, Oct 7, 2026 at 9:45 PM Yufei Gu <fl...@gmail.com> wrote:
>
>> Thanks for the reminder, Dmitri! I will take a look.
>>
>> Yufei
>>
>>
>> On Wed, Oct 7, 2026 at 5:37 PM Dmitri Bourlatchkov <di...@apache.org>
>> wrote:
>>
>> > Hi All,
>> >
>> > Metrics grants have been merged as a separate PR.
>> >
>> > Now [4115] is ready for one more review round. IMHO, it is ready to
>> merge.
>> >
>> > [4115] https://github.com/apache/polaris/pull/4115
>> >
>> > Cheers,
>> > Dmitri.
>> >
>> > On Fri, Oct 2, 2026 at 1:01 AM Jean-Baptiste Onofré <jb...@nanthrax.net>
>> > wrote:
>> >
>> > > Hi all,
>> > >
>> > > I agree with Yufei: I would keep TABLE_READ_METRICS as a separate,
>> > > explicit grant, not implied by TABLE_FULL_METADATA.
>> > >
>> > > I believe metrics reports are closer to an activity log than to table
>> > > metadata. I also think it's consistent with what we have today, as
>> > > reporting scan and commit metrics is already with TABLE_READ_DATA and
>> > > TABLE_WRITE_DATA, not by "metadata" privilege.
>> > >
>> > > For the troubleshooting use case, the cost is just one additional
>> > > grant, I think it's acceptable :) I would still let
>> > > CATALOG_MANAGE_CONTENT implies it, as that privilege already implies
>> > > TABLE_READ_DATA.
>> > >
>> > > Concretely for PR #5669, I propose:
>> > > - to remove the "Implied by TABLE_FULL_METADATA" statement from the
>> > > javadoc and CHANGELOG
>> > > - to add SUPER_PRIVILEGES entry in the PR, so the grant change is
>> > > self-contained
>> > > - to drop ICEBERG_TABLE restriction.
>> > > - to update access-control.md documenting TABLE_READ_METRICS, etc
>> > >
>> > > I will comment directly in the PR.
>> > >
>> > > Regards
>> > > JB
>> > >
>> > > On Fri, Oct 2, 2026 at 3:40 AM Yufei Gu <fl...@gmail.com> wrote:
>> > > >
>> > > > Thanks Dmitri for the reminder.
>> > > >
>> > > > PR #5659 <https://github.com/apache/polaris/pull/5659> adds
>> > > > TABLE_READ_METRICS and says it is implied by TABLE_FULL_METADATA.
>> > Should
>> > > > that be the default?
>> > > >
>> > > > I’m leaning toward keeping it separate. Metrics describe workloads
>> > (Spark
>> > > > ETL, Flink streaming, etc) using a table and can include query
>> filters,
>> > > so
>> > > > someone managing table metadata may not need access to other users’
>> > query
>> > > > activity.
>> > > >
>> > > > Including it could make troubleshooting easier for table admins,
>> > though.
>> > > > Are there use cases that favor automatic access over an explicit
>> grant?
>> > > > Yufei
>> > > >
>> > > >
>> > > > On Wed, Sep 30, 2026 at 1:57 PM Dmitri Bourlatchkov <
>> dimas@apache.org>
>> > > > wrote:
>> > > >
>> > > > > Hi Yufei,
>> > > > >
>> > > > > Here's the separate PR for Metrics-related grants:
>> > > > > https://github.com/apache/polaris/pull/5659
>> > > > >
>> > > > > Please review it to unblock this contribution.
>> > > > >
>> > > > > Thanks,
>> > > > > Dmitri.
>> > > > >
>> > > > > On Wed, Sep 30, 2026 at 1:05 PM Yufei Gu <fl...@gmail.com>
>> > wrote:
>> > > > >
>> > > > > > From the previous email, we agreed to split the REST change from
>> > the
>> > > > > grant
>> > > > > > change. I think the PR is ready to go if we only keep the grant
>> > > changes.
>> > > > > >
>> > > > > > Yufei
>> > > > > >
>> > > > > >
>> > > > > > On Tue, Sep 29, 2026 at 9:26 PM Jean-Baptiste Onofré <
>> > > jb@nanthrax.net>
>> > > > > > wrote:
>> > > > > >
>> > > > > > > Hi guys,
>> > > > > > >
>> > > > > > > I agree with Dmitri: +1 to merge PR #4115 on Oct 1 (tomorrow).
>> > > > > > >
>> > > > > > > From my side, all outstanding feedback has been addressed,
>> > > including
>> > > > > > > the final documentation points raised by EJ regarding the
>> > empty-200
>> > > > > > > NoOp behavior and reconstructed payload description. The
>> branch
>> > is
>> > > > > > > clean, up to date with main, and CI is green.
>> > > > > > >
>> > > > > > > Once this lands, we can follow up with PR #4756 for the JDBC
>> > query
>> > > SPI
>> > > > > > > implementation.
>> > > > > > >
>> > > > > > > Regards
>> > > > > > > JB
>> > > > > > >
>> > > > > > > On Wed, Sep 30, 2026 at 1:53 AM Dmitri Bourlatchkov <
>> > > dimas@apache.org>
>> > > > > > > wrote:
>> > > > > > > >
>> > > > > > > > Hi All,
>> > > > > > > >
>> > > > > > > > Any remaining blockers on [4115]?
>> > > > > > > >
>> > > > > > > > Given that the review was very long, please restate any
>> > remaining
>> > > > > > > concerns
>> > > > > > > > (here or in GH).
>> > > > > > > >
>> > > > > > > > Otherwise, I propose merging [4115] on Oct 1.
>> > > > > > > >
>> > > > > > > > [4115] https://github.com/apache/polaris/pull/4115
>> > > > > > > >
>> > > > > > > > Thanks,
>> > > > > > > > Dmitri.
>> > > > > > > >
>> > > > > > > > On Wed, Sep 23, 2026 at 7:53 PM Dmitri Bourlatchkov <
>> > > > > dimas@apache.org>
>> > > > > > > > wrote:
>> > > > > > > >
>> > > > > > > > > Hi EJ,
>> > > > > > > > >
>> > > > > > > > > Thanks for the meeting recap!
>> > > > > > > > >
>> > > > > > > > > All:
>> > > > > > > > >
>> > > > > > > > > Worth noting: PR [4115] also contains Management API
>> changes
>> > > that
>> > > > > > were
>> > > > > > > not
>> > > > > > > > > anticipated initially and did not get an API change review
>> > > until
>> > > > > now.
>> > > > > > > These
>> > > > > > > > > are small changes to add new grant types for metrics
>> RBAC..
>> > > > > > > > >
>> > > > > > > > > I've reviewed and approved PR [4115] in GH.
>> > > > > > > > >
>> > > > > > > > > [4115] https://github.com/apache/polaris/pull/4115
>> > > > > > > > >
>> > > > > > > > > Cheers,
>> > > > > > > > > Dmitri.
>> > > > > > > > >
>> > > > > > > > > On Wed, Aug 12, 2026 at 5:21 PM EJ Wang <
>> > > > > > > ej.wang.dev.interest@gmail.com>
>> > > > > > > > > wrote:
>> > > > > > > > >
>> > > > > > > > >> Hi folks,
>> > > > > > > > >>
>> > > > > > > > >> Notes from today's metrics architecture sync are in the
>> > usual
>> > > doc:
>> > > > > > > > >>
>> > > > > > > > >>
>> > > > > > >
>> > > > > >
>> > > > >
>> > >
>> >
>> https://docs.google.com/document/d/100h7c4damrUzVuquYbBHM0EvA4LSWuW2IT2dN_7nYVA/edit?pli=1&tab=t.18iv1m7p6i6b
>> > > > > > > > >>
>> > > > > > > > >> Highlights:
>> > > > > > > > >>
>> > > > > > > > >> Module placement. core/ now holds the entity data model
>> > only.
>> > > A
>> > > > > new
>> > > > > > > spi/
>> > > > > > > > >> takes every other shared contract: feature SPIs, feature
>> > > non-SPI
>> > > > > > > > >> contracts,
>> > > > > > > > >> substrate SPIs, substrate non-SPI contracts, and
>> > > contract-required
>> > > > > > > types.
>> > > > > > > > >> That settles the question PR #5204 reopened, by placing
>> into
>> > > a new
>> > > > > > SPI
>> > > > > > > > >> module. Note that it changes slightly towards the
>> referenced
>> > > WIP
>> > > > > RFC
>> > > > > > > [1]
>> > > > > > > > >> recommendations: what proposed to place in core/, now
>> should
>> > > go to
>> > > > > > > spi/
>> > > > > > > > >> except for the entity data model.
>> > > > > > > > >>
>> > > > > > > > >> Metrics PRs. #5068 merged. #5204 needs reshaping onto the
>> > > split.
>> > > > > > > #4115 and
>> > > > > > > > >> #4756 stay open with action on the author to follow up on
>> > > review
>> > > > > > > comments.
>> > > > > > > > >>
>> > > > > > > > >> [1]
>> > > > > > > > >>
>> > > > > > > > >>
>> > > > > > >
>> > > > > >
>> > > > >
>> > >
>> >
>> https://docs.google.com/document/d/1mj3vGpix9zTDTFFJKtWT5ZxI3PTGI1YNb5xFqyWrOkc/edit?tab=t.0
>> > > > > > > > >>
>> > > > > > > > >> On Wed, Jul 15, 2026 at 11:01 AM Anand Kumar Sankaran via
>> > dev
>> > > <
>> > > > > > > > >> dev@polaris.apache.org> wrote:
>> > > > > > > > >>
>> > > > > > > > >> > Thanks EJ.  My plan is to introduce PR0 in a new
>> branch /
>> > > PR and
>> > > > > > > > >> repurpose
>> > > > > > > > >> > #4115 and #4756 for PR1 and PR2 to maintain all the
>> > valuable
>> > > > > > > discussions
>> > > > > > > > >> > that happened there.
>> > > > > > > > >> >
>> > > > > > > > >> > From: EJ Wang <ej...@gmail.com>
>> > > > > > > > >> > Date: Wednesday, July 15, 2026 at 10:41 AM
>> > > > > > > > >> > To: dev@polaris.apache.org <de...@polaris.apache.org>
>> > > > > > > > >> > Subject: Re: Proposal for REST endpoints for table
>> metrics
>> > > and
>> > > > > > > events
>> > > > > > > > >> >
>> > > > > > > > >> > Hi folks, Thanks again for the discussion today. I
>> updated
>> > > the
>> > > > > > sync
>> > > > > > > doc
>> > > > > > > > >> > with notes from today's metrics sync: https:
>> //urldefense.
>> > > > > > > > >> com/v3/__https:
>> > > > > > > > >> > //docs. google.
>> > > > > > > > >> >
>> > > > > > > > >>
>> > > > > > >
>> > > > > >
>> > > > >
>> > >
>> >
>> com/document/d/100h7c4damrUzVuquYbBHM0EvA4LSWuW2IT2dN_7nYVA/edit?pli=1&tab=t.
>> > > > > > > > >> >
>> > > > > > > > >>
>> > > > > > >
>> > > > > >
>> > > > >
>> > >
>> >
>> 2ahd7wze5f__;!!Iz9xO38YGHZK!4rnlVIqBQQQ6xpGe1PMV58edOS_cGrKoqdfhw4qSp_JQCavvEjaxXk0OwGPnxyEcO60AzKQK_XDtp7H3DdkFXcrXJwAz1g0$
>> > > > > > > > >> >
>> > > > > > > > >> >
>> > > > > > > > >> > Hi folks,
>> > > > > > > > >> >
>> > > > > > > > >> > Thanks again for the discussion today. I updated the
>> sync
>> > > doc
>> > > > > with
>> > > > > > > notes
>> > > > > > > > >> > from today's metrics sync:
>> > > > > > > > >> >
>> > > > > > > > >> >
>> > > > > > > > >>
>> > > > > > >
>> > > > > >
>> > > > >
>> > >
>> >
>> https://urldefense.com/v3/__https://docs.google.com/document/d/100h7c4damrUzVuquYbBHM0EvA4LSWuW2IT2dN_7nYVA/edit?pli=1&tab=t.2ahd7wze5f__;!!Iz9xO38YGHZK!4rnlVIqBQQQ6xpGe1PMV58edOS_cGrKoqdfhw4qSp_JQCavvEjaxXk0OwGPnxyEcO60AzKQK_XDtp7H3DdkFXcrXJwAz1g0$
>> > > > > > > > >> >
>> > > > > > > > >> > See highlights below:
>> > > > > > > > >> >
>> > > > > > > > >> > In previous meetings, EJ proposed a two prs framing to
>> > push
>> > > > > > forward
>> > > > > > > the
>> > > > > > > > >> > metrics query work in polaris. Dmitri pointed out a
>> subtle
>> > > point
>> > > > > > > that
>> > > > > > > > >> pr2
>> > > > > > > > >> > is currently also trying to do the persistence related
>> > code
>> > > > > > > placement
>> > > > > > > > >> > refactor. For the sake of straightforward review
>> process,
>> > > EJ,
>> > > > > > > Dmitri and
>> > > > > > > > >> > Anand (the author) agreed to separate the persistence
>> > > > > refactoring
>> > > > > > > into
>> > > > > > > > >> pr0,
>> > > > > > > > >> > and reframing the pr scopes as follow:
>> > > > > > > > >> >
>> > > > > > > > >> >    1.
>> > > > > > > > >> >
>> > > > > > > > >> >    Pr0: refactor irc metrics spi
>> > > > > > > > >> >    1.
>> > > > > > > > >> >
>> > > > > > > > >> >       SPI should stay in core/ with metric type
>> enveloped
>> > > > > > > > >> >       2.
>> > > > > > > > >> >
>> > > > > > > > >> >       default spi impl, no-op, in extensions/ - default
>> > > > > selection
>> > > > > > at
>> > > > > > > > >> >       runtime/
>> > > > > > > > >> >       3.
>> > > > > > > > >> >
>> > > > > > > > >> >       move metrics persistence and correspondings to
>> > > extensions/
>> > > > > > as
>> > > > > > > an
>> > > > > > > > >> >       alternative SPI impl
>> > > > > > > > >> >       2.
>> > > > > > > > >> >
>> > > > > > > > >> >    Pr1: rest api for query
>> > > > > > > > >> >
>> > > > > > > > >> >
>> > > > > > > > >>
>> > > > > > >
>> > > > > >
>> > > > >
>> > >
>> >
>> https://urldefense.com/v3/__https://github.com/apache/polaris/pull/4115/changes__;!!Iz9xO38YGHZK!4rnlVIqBQQQ6xpGe1PMV58edOS_cGrKoqdfhw4qSp_JQCavvEjaxXk0OwGPnxyEcO60AzKQK_XDtp7H3DdkFXcrXFCbEbRk$
>> > > > > > > > >> >    1.
>> > > > > > > > >> >
>> > > > > > > > >> >       rest api spec in extensions/ as optional spec
>> > > > > > > > >> >       2.
>> > > > > > > > >> >
>> > > > > > > > >> >       SPI in extensions/ as optional spec spi
>> > > > > > > > >> >       3.
>> > > > > > > > >> >
>> > > > > > > > >> >       default spi impl in extensions/ no-op
>> > > > > > > > >> >       4.
>> > > > > > > > >> >
>> > > > > > > > >> >       handler related in runtime/ - thin adaptor from
>> http
>> > > shape
>> > > > > > to
>> > > > > > > spi
>> > > > > > > > >> >       shape
>> > > > > > > > >> >       3.
>> > > > > > > > >> >
>> > > > > > > > >> >    Pr2: jdbc impl for query
>> > > > > > > > >> >
>> > > > > > > > >>
>> > > > > > >
>> > > > > >
>> > > > >
>> > >
>> >
>> https://urldefense.com/v3/__https://github.com/apache/polaris/pull/4756__;!!Iz9xO38YGHZK!4rnlVIqBQQQ6xpGe1PMV58edOS_cGrKoqdfhw4qSp_JQCavvEjaxXk0OwGPnxyEcO60AzKQK_XDtp7H3DdkFXcrXRVMBzqE$
>> > > > > > > > >> >    1.
>> > > > > > > > >> >
>> > > > > > > > >> >       alternative spi impl in extensions/ -> query with
>> > jdbc
>> > > > > > > > >> >
>> > > > > > > > >> >
>> > > > > > > > >> > Thanks,
>> > > > > > > > >> > -ej
>> > > > > > > > >> >
>> > > > > > > > >> > On Wed, Jul 1, 2026 at 8:01 AM Alexandre Dutra <
>> > > > > adutra@apache.org
>> > > > > > >
>> > > > > > > > >> wrote:
>> > > > > > > > >> >
>> > > > > > > > >> > > Hi all,
>> > > > > > > > >> > >
>> > > > > > > > >> > > > While the current implementation stores metrics in
>> the
>> > > > > > > metastore,
>> > > > > > > > >> other
>> > > > > > > > >> > > deployments
>> > > > > > > > >> > > may choose a KV store, a dedicated metrics
>> warehouse, or
>> > > > > another
>> > > > > > > > >> backend
>> > > > > > > > >> > > entirely. In those cases, exposing a Polaris REST API
>> > > over a
>> > > > > > > specific
>> > > > > > > > >> > > schema feels like coupling the API to an
>> implementation
>> > > > > detail.
>> > > > > > > > >> > >
>> > > > > > > > >> > > I wouldn't say to an implementation detail, but
>> rather
>> > to
>> > > a
>> > > > > > > particular
>> > > > > > > > >> > > setup / configuration. I think that's fine. If you
>> don't
>> > > store
>> > > > > > > > >> > > metrics, you don't read metrics. Similar situations
>> will
>> > > > > > certainly
>> > > > > > > > >> > > happen in the future, e.g. if we ever introduce an
>> API
>> > to
>> > > read
>> > > > > > > events,
>> > > > > > > > >> > > that would only work if events are being persisted in
>> > the
>> > > > > first
>> > > > > > > place.
>> > > > > > > > >> > > Again I think that's a fair trade-off.
>> > > > > > > > >> > >
>> > > > > > > > >> > > Thanks,
>> > > > > > > > >> > > Alex
>> > > > > > > > >> > >
>> > > > > > > > >> > > On Wed, Jul 1, 2026 at 7:23 AM Dmitri Bourlatchkov <
>> > > > > > > dimas@apache.org>
>> > > > > > > > >> > > wrote:
>> > > > > > > > >> > > >
>> > > > > > > > >> > > > Hi Yufei,
>> > > > > > > > >> > > >
>> > > > > > > > >> > > > From my POV, it is fine for Polaris to have
>> multiple
>> > > REST
>> > > > > API
>> > > > > > > > >> surfaces
>> > > > > > > > >> > > with
>> > > > > > > > >> > > > different features and different degrees of
>> > "readiness".
>> > > > > Each
>> > > > > > > API
>> > > > > > > > >> can
>> > > > > > > > >> > > > evolve fairly independently. With proper
>> > modularization
>> > > > > (which
>> > > > > > > Anand
>> > > > > > > > >> > > did),
>> > > > > > > > >> > > > changes to the main Polaris code and these extra
>> > modules
>> > > > > > should
>> > > > > > > > >> > normally
>> > > > > > > > >> > > be
>> > > > > > > > >> > > > non-intrusive to each other.
>> > > > > > > > >> > > >
>> > > > > > > > >> > > > I assume Anand is going to be the first user of
>> this
>> > > API.
>> > > > > > > However,
>> > > > > > > > >> if
>> > > > > > > > >> > we
>> > > > > > > > >> > > do
>> > > > > > > > >> > > > not merge it we'll never know whether other users
>> are
>> > > > > > > interested. I
>> > > > > > > > >> do
>> > > > > > > > >> > > not
>> > > > > > > > >> > > > think many end users read the dev ML, but they
>> > certainly
>> > > > > read
>> > > > > > > the
>> > > > > > > > >> docs,
>> > > > > > > > >> > > > code and release notes.
>> > > > > > > > >> > > >
>> > > > > > > > >> > > > There is no risk for Polaris as a project in
>> exposing
>> > > this
>> > > > > > > > >> experimental
>> > > > > > > > >> > > > API. If usage dwindles over time, we can remove
>> it. If
>> > > it
>> > > > > > gains
>> > > > > > > a
>> > > > > > > > >> > > > considerable user / contributor base, we will
>> promote
>> > > it to
>> > > > > a
>> > > > > > > > >> > > "first-class"
>> > > > > > > > >> > > > (non-beta) API.
>> > > > > > > > >> > > >
>> > > > > > > > >> > > > Cheers,
>> > > > > > > > >> > > > Dmitri.
>> > > > > > > > >> > > >
>> > > > > > > > >> > > > On Tue, Jun 30, 2026 at 7:53 PM Yufei Gu <
>> > > > > > flyrain000@gmail.com>
>> > > > > > > > >> wrote:
>> > > > > > > > >> > > >
>> > > > > > > > >> > > > > Hi all,
>> > > > > > > > >> > > > >
>> > > > > > > > >> > > > > Thanks for driving this work forward. Building
>> this
>> > > as a
>> > > > > POC
>> > > > > > > makes
>> > > > > > > > >> > > perfect
>> > > > > > > > >> > > > > sense if the goal is to validate the idea and
>> gather
>> > > > > > feedback.
>> > > > > > > > >> Should
>> > > > > > > > >> > > we
>> > > > > > > > >> > > > > merge a POC into the main branch before agreeing
>> on
>> > > the
>> > > > > > > overall
>> > > > > > > > >> > > direction?
>> > > > > > > > >> > > > > To me, that feels like a separate discussion from
>> > > whether
>> > > > > > the
>> > > > > > > > >> > > > > implementation itself looks reasonable.
>> > > > > > > > >> > > > >
>> > > > > > > > >> > > > > One concern that I don't think has been fully
>> > > addressed is
>> > > > > > > > >> whether we
>> > > > > > > > >> > > need
>> > > > > > > > >> > > > > to expose metrics consumption through a Polaris
>> REST
>> > > > > > endpoint
>> > > > > > > at
>> > > > > > > > >> all.
>> > > > > > > > >> > > > >
>> > > > > > > > >> > > > > My concern is less about the API itself and more
>> > > about the
>> > > > > > > > >> boundary
>> > > > > > > > >> > > Polaris
>> > > > > > > > >> > > > > should own. The proposed REST API is tied to a
>> > > particular
>> > > > > > > > >> persistence
>> > > > > > > > >> > > > > schema, but that schema is just one possible
>> > > > > implementation.
>> > > > > > > While
>> > > > > > > > >> > the
>> > > > > > > > >> > > > > current implementation stores metrics in the
>> > > metastore,
>> > > > > > other
>> > > > > > > > >> > > deployments
>> > > > > > > > >> > > > > may choose a KV store, a dedicated metrics
>> > warehouse,
>> > > or
>> > > > > > > another
>> > > > > > > > >> > > backend
>> > > > > > > > >> > > > > entirely. In those cases, exposing a Polaris REST
>> > API
>> > > > > over a
>> > > > > > > > >> specific
>> > > > > > > > >> > > > > schema feels like coupling the API to an
>> > > implementation
>> > > > > > > detail.
>> > > > > > > > >> > > > >
>> > > > > > > > >> > > > > Polaris isn't intended to become a full metrics
>> > > system.
>> > > > > I'd
>> > > > > > > > >> expect it
>> > > > > > > > >> > > to
>> > > > > > > > >> > > > > focus on producing metrics and let deployments
>> > choose
>> > > the
>> > > > > > most
>> > > > > > > > >> > > appropriate
>> > > > > > > > >> > > > > storage and consumption model for their
>> environment.
>> > > > > > > > >> > > > >
>> > > > > > > > >> > > > > I'm not opposed to exploring this direction, but
>> we
>> > > should
>> > > > > > > first
>> > > > > > > > >> > align
>> > > > > > > > >> > > on
>> > > > > > > > >> > > > > whether exposing a REST API through Polaris is
>> the
>> > > right
>> > > > > > long
>> > > > > > > term
>> > > > > > > > >> > > approach
>> > > > > > > > >> > > > > before merging it into the main branch.
>> > > > > > > > >> > > > >
>> > > > > > > > >> > > > > Thanks,
>> > > > > > > > >> > > > > Yufei
>> > > > > > > > >> > > > >
>> > > > > > > > >> > > > > Yufei
>> > > > > > > > >> > > > >
>> > > > > > > > >> > > > >
>> > > > > > > > >> > > > > On Mon, Jun 29, 2026 at 9:27 AM Dmitri
>> Bourlatchkov
>> > <
>> > > > > > > > >> > dimas@apache.org>
>> > > > > > > > >> > > > > wrote:
>> > > > > > > > >> > > > >
>> > > > > > > > >> > > > > > Hi All,
>> > > > > > > > >> > > > > >
>> > > > > > > > >> > > > > > It looks like Anand addressed all action items
>> in
>> > > > > [4115].
>> > > > > > > > >> > > > > >
>> > > > > > > > >> > > > > > Please review the latest state of the code.
>> The PR
>> > > LGTM
>> > > > > > and
>> > > > > > > I
>> > > > > > > > >> > > approved in
>> > > > > > > > >> > > > > > GH from my side.
>> > > > > > > > >> > > > > >
>> > > > > > > > >> > > > > > [4115]
>> > > > > > > > >> >
>> > > > > > > > >>
>> > > > > > >
>> > > > > >
>> > > > >
>> > >
>> >
>> https://urldefense.com/v3/__https://github.com/apache/polaris/pull/4115__;!!Iz9xO38YGHZK!4rnlVIqBQQQ6xpGe1PMV58edOS_cGrKoqdfhw4qSp_JQCavvEjaxXk0OwGPnxyEcO60AzKQK_XDtp7H3DdkFXcrXtFRscKs$
>> > > > > > > > >> > > > > >
>> > > > > > > > >> > > > > > Thanks,
>> > > > > > > > >> > > > > > Dmitri.
>> > > > > > > > >> > > > > >
>> > > > > > > > >> > > > > > On Thu, Jun 25, 2026 at 1:56 PM Dmitri
>> > Bourlatchkov
>> > > <
>> > > > > > > > >> > > dimas@apache.org>
>> > > > > > > > >> > > > > > wrote:
>> > > > > > > > >> > > > > >
>> > > > > > > > >> > > > > > > Hi All,
>> > > > > > > > >> > > > > > >
>> > > > > > > > >> > > > > > > Recap of the discussion in today's Community
>> > Sync:
>> > > > > > > > >> > > > > > >
>> > > > > > > > >> > > > > > > * The modular approach for Metrics SPI(s) and
>> > > > > > > implementation
>> > > > > > > > >> > seems
>> > > > > > > > >> > > to
>> > > > > > > > >> > > > > > have
>> > > > > > > > >> > > > > > > community consensus (please comment if you
>> still
>> > > have
>> > > > > > > > >> concerns)
>> > > > > > > > >> > > > > > >
>> > > > > > > > >> > > > > > > * What modules / feature flags are enabled in
>> > > Apache
>> > > > > > > Polaris
>> > > > > > > > >> > > images by
>> > > > > > > > >> > > > > > > default will be discussed later (when the
>> > feature
>> > > is
>> > > > > > > close to
>> > > > > > > > >> > > > > > completion).
>> > > > > > > > >> > > > > > >
>> > > > > > > > >> > > > > > > In any case, users will be able to turn the
>> full
>> > > > > feature
>> > > > > > > > >> on/off
>> > > > > > > > >> > at
>> > > > > > > > >> > > > > their
>> > > > > > > > >> > > > > > > discretion, regardless of the default flag
>> > values.
>> > > > > > > > >> > > > > > >
>> > > > > > > > >> > > > > > > * It is generally preferable to have smaller
>> PRs
>> > > and
>> > > > > > > introduce
>> > > > > > > > >> > new
>> > > > > > > > >> > > > > > > features gradually to simplify reviews.
>> > > > > > > > >> > > > > > >
>> > > > > > > > >> > > > > > > What this means for the Metrics API, IMHO is:
>> > > > > > > > >> > > > > > >
>> > > > > > > > >> > > > > > > a) The scope of PR 4115 is probably ok as it
>> > > stands
>> > > > > > right
>> > > > > > > > >> now. If
>> > > > > > > > >> > > > > someone
>> > > > > > > > >> > > > > > > wants to propose splitting it further, please
>> > > comment.
>> > > > > > > > >> > > > > > >
>> > > > > > > > >> > > > > > > b) After 4115 is merged, Metrics Persistence
>> PRs
>> > > will
>> > > > > > > follow,
>> > > > > > > > >> > with
>> > > > > > > > >> > > > > > smaller
>> > > > > > > > >> > > > > > > scope per PR as appropriate for the nature of
>> > the
>> > > code
>> > > > > > > > >> changes.
>> > > > > > > > >> > > > > > >
>> > > > > > > > >> > > > > > > * The major concern was about exposing the
>> new
>> > > Metrics
>> > > > > > > REST
>> > > > > > > > >> API
>> > > > > > > > >> > to
>> > > > > > > > >> > > end
>> > > > > > > > >> > > > > > > users. The nature of this concern was about
>> > > avoiding
>> > > > > the
>> > > > > > > > >> (false)
>> > > > > > > > >> > > > > > impression
>> > > > > > > > >> > > > > > > that the new API is a firm and adopted
>> Polaris
>> > > > > > > specification.
>> > > > > > > > >> > > > > > >
>> > > > > > > > >> > > > > > > I think this concern is reasonable.
>> > > > > > > > >> > > > > > >
>> > > > > > > > >> > > > > > > I propose that Anand update the documentation
>> > > sections
>> > > > > > of
>> > > > > > > PR
>> > > > > > > > >> 4115
>> > > > > > > > >> > > to
>> > > > > > > > >> > > > > > > emphasise that the API is currently an
>> > > experimental /
>> > > > > > POC
>> > > > > > > API
>> > > > > > > > >> at
>> > > > > > > > >> > > this
>> > > > > > > > >> > > > > > time
>> > > > > > > > >> > > > > > > and is subject to change (including breaking
>> > > > > changes). I
>> > > > > > > > >> believe
>> > > > > > > > >> > it
>> > > > > > > > >> > > > > > already
>> > > > > > > > >> > > > > > > has the "beta" label. So this request is
>> mostly
>> > > for
>> > > > > doc
>> > > > > > > > >> changes
>> > > > > > > > >> > to
>> > > > > > > > >> > > > > > clarify
>> > > > > > > > >> > > > > > > user expectations, since the "beta" label
>> alone
>> > > may
>> > > > > not
>> > > > > > be
>> > > > > > > > >> > > interpreted
>> > > > > > > > >> > > > > > the
>> > > > > > > > >> > > > > > > same way by all users.
>> > > > > > > > >> > > > > > >
>> > > > > > > > >> > > > > > > I hope this update will resolve all remaining
>> > > concerns
>> > > > > > > with
>> > > > > > > > >> 4115
>> > > > > > > > >> > > and
>> > > > > > > > >> > > > > > allow
>> > > > > > > > >> > > > > > > for it to be merged so that the community
>> will
>> > be
>> > > able
>> > > > > > to
>> > > > > > > > >> proceed
>> > > > > > > > >> > > to
>> > > > > > > > >> > > > > > > follow-up PRs (as noted above).
>> > > > > > > > >> > > > > > >
>> > > > > > > > >> > > > > > > Please add/correct me if I missed anything
>> > > relevant.
>> > > > > > > > >> > > > > > >
>> > > > > > > > >> > > > > > > Thanks,
>> > > > > > > > >> > > > > > > Dmitri.
>> > > > > > > > >> > > > > > >
>> > > > > > > > >> > > > > > > On Wed, Jun 24, 2026 at 8:11 PM Yufei Gu <
>> > > > > > > > >> flyrain000@gmail.com>
>> > > > > > > > >> > > wrote:
>> > > > > > > > >> > > > > > >
>> > > > > > > > >> > > > > > >> I do not think we have agreement on the
>> query
>> > > REST
>> > > > > APIs
>> > > > > > > yet.
>> > > > > > > > >> > > > > > >>
>> > > > > > > > >> > > > > > >> In particular, the current query API shape
>> > seems
>> > > > > highly
>> > > > > > > > >> coupled
>> > > > > > > > >> > > to the
>> > > > > > > > >> > > > > > >> *example[1]* metrics in the Iceberg REST
>> spec.
>> > I
>> > > do
>> > > > > not
>> > > > > > > think
>> > > > > > > > >> > > that is
>> > > > > > > > >> > > > > a
>> > > > > > > > >> > > > > > >> good way to design a REST API contract. For
>> > > example,
>> > > > > > > fields
>> > > > > > > > >> like
>> > > > > > > > >> > > > > > >> resultDataFiles, resultDeleteFiles,
>> > > > > totalFileSizeBytes,
>> > > > > > > > >> > > > > > >> totalDataManifests,
>> > > > > > > > >> > > > > > >> totalDeleteManifests, and
>> scannedDataManifests
>> > > are
>> > > > > very
>> > > > > > > > >> specific
>> > > > > > > > >> > > to
>> > > > > > > > >> > > > > one
>> > > > > > > > >> > > > > > >> style of scan metric. That makes the API
>> look
>> > > more
>> > > > > > like a
>> > > > > > > > >> direct
>> > > > > > > > >> > > > > > >> projection
>> > > > > > > > >> > > > > > >> of the current example payload than a
>> general
>> > > query
>> > > > > > > model.
>> > > > > > > > >> > > > > > >>
>> > > > > > > > >> > > > > > >> My preference is that Polaris should not
>> define
>> > > or
>> > > > > > > strongly
>> > > > > > > > >> > opine
>> > > > > > > > >> > > on
>> > > > > > > > >> > > > > the
>> > > > > > > > >> > > > > > >> Iceberg scan or commit metrics query REST
>> API
>> > at
>> > > this
>> > > > > > > point.
>> > > > > > > > >> > > Polaris
>> > > > > > > > >> > > > > can
>> > > > > > > > >> > > > > > >> resolve and authorize the table, accept the
>> > > metrics
>> > > > > > > report,
>> > > > > > > > >> and
>> > > > > > > > >> > > > > delegate
>> > > > > > > > >> > > > > > >> it
>> > > > > > > > >> > > > > > >> to the configured reporting path. Durable
>> > > storage,
>> > > > > > > filtering,
>> > > > > > > > >> > > > > > dashboards,
>> > > > > > > > >> > > > > > >> and query APIs should remain downstream
>> impl.
>> > > choices
>> > > > > > > unless
>> > > > > > > > >> and
>> > > > > > > > >> > > until
>> > > > > > > > >> > > > > > the
>> > > > > > > > >> > > > > > >> Iceberg community standardizes a query
>> > contract,
>> > > > > which
>> > > > > > I
>> > > > > > > > >> doubt
>> > > > > > > > >> > > will
>> > > > > > > > >> > > > > > >> happen.
>> > > > > > > > >> > > > > > >>
>> > > > > > > > >> > > > > > >> So I do not think the intake path has to be
>> > > blocked,
>> > > > > > but
>> > > > > > > I
>> > > > > > > > >> would
>> > > > > > > > >> > > > > prefer
>> > > > > > > > >> > > > > > >> not
>> > > > > > > > >> > > > > > >> to merge a Polaris specific query API that
>> may
>> > be
>> > > > > hard
>> > > > > > to
>> > > > > > > > >> evolve
>> > > > > > > > >> > > > > later.
>> > > > > > > > >> > > > > > >>
>> > > > > > > > >> > > > > > >> 1.
>> > > > > > > > >> > > > > > >>
>> > > > > > > > >> > > > > > >>
>> > > > > > > > >> > > > > >
>> > > > > > > > >> > > > >
>> > > > > > > > >> > >
>> > > > > > > > >> >
>> > > > > > > > >>
>> > > > > > >
>> > > > > >
>> > > > >
>> > >
>> >
>> https://urldefense.com/v3/__https://github.com/apache/iceberg/blob/2a6c556c0c883c04c6d3fbd68e6aeda36b91e0aa/open-api/rest-catalog-open-api.yaml*L4152__;Iw!!Iz9xO38YGHZK!4rnlVIqBQQQ6xpGe1PMV58edOS_cGrKoqdfhw4qSp_JQCavvEjaxXk0OwGPnxyEcO60AzKQK_XDtp7H3DdkFXcrXCDsqYGo$
>> > > > > > > > >> > > > > > >>
>> > > > > > > > >> > > > > > >> Yufei
>> > > > > > > > >> > > > > > >>
>> > > > > > > > >> > > > > > >>
>> > > > > > > > >> > > > > > >> On Tue, Jun 23, 2026 at 9:30 AM Dmitri
>> > > Bourlatchkov <
>> > > > > > > > >> > > dimas@apache.org
>> > > > > > > > >> > > > > >
>> > > > > > > > >> > > > > > >> wrote:
>> > > > > > > > >> > > > > > >>
>> > > > > > > > >> > > > > > >> > Hi All,
>> > > > > > > > >> > > > > > >> >
>> > > > > > > > >> > > > > > >> > So, what are the blockers for merging
>> [4115],
>> > > if
>> > > > > any?
>> > > > > > > > >> > > > > > >> >
>> > > > > > > > >> > > > > > >> > [4115]
>> > > > > > > > >> >
>> > > > > > > > >>
>> > > > > > >
>> > > > > >
>> > > > >
>> > >
>> >
>> https://urldefense.com/v3/__https://github.com/apache/polaris/pull/4115__;!!Iz9xO38YGHZK!4rnlVIqBQQQ6xpGe1PMV58edOS_cGrKoqdfhw4qSp_JQCavvEjaxXk0OwGPnxyEcO60AzKQK_XDtp7H3DdkFXcrXtFRscKs$
>> > > > > > > > >> > > > > > >> >
>> > > > > > > > >> > > > > > >> > Thanks,
>> > > > > > > > >> > > > > > >> > Dmitri.
>> > > > > > > > >> > > > > > >> >
>> > > > > > > > >> > > > > > >> > On Wed, Jun 17, 2026 at 5:28 PM EJ Wang <
>> > > > > > > > >> > > > > > ej.wang.dev.interest@gmail.com
>> > > > > > > > >> > > > > > >> >
>> > > > > > > > >> > > > > > >> > wrote:
>> > > > > > > > >> > > > > > >> >
>> > > > > > > > >> > > > > > >> > > Hi folks,
>> > > > > > > > >> > > > > > >> > >
>> > > > > > > > >> > > > > > >> > > Thanks again for the discussion today. I
>> > > updated
>> > > > > > the
>> > > > > > > sync
>> > > > > > > > >> > doc
>> > > > > > > > >> > > with
>> > > > > > > > >> > > > > > >> notes
>> > > > > > > > >> > > > > > >> > > from today's metrics sync:
>> > > > > > > > >> > > > > > >> > >
>> > > > > > > > >> > > > > > >> > >
>> > > > > > > > >> > > > > > >> > >
>> > > > > > > > >> > > > > > >> >
>> > > > > > > > >> > > > > > >>
>> > > > > > > > >> > > > > >
>> > > > > > > > >> > > > >
>> > > > > > > > >> > >
>> > > > > > > > >> >
>> > > > > > > > >>
>> > > > > > >
>> > > > > >
>> > > > >
>> > >
>> >
>> https://urldefense.com/v3/__https://docs.google.com/document/d/100h7c4damrUzVuquYbBHM0EvA4LSWuW2IT2dN_7nYVA/edit?tab=t.ezk0rgdx0c6m__;!!Iz9xO38YGHZK!4rnlVIqBQQQ6xpGe1PMV58edOS_cGrKoqdfhw4qSp_JQCavvEjaxXk0OwGPnxyEcO60AzKQK_XDtp7H3DdkFXcrX83IZkm0$
>> > > > > > > > >> > > > > > >> > >
>> > > > > > > > >> > > > > > >> > > A few highlights:
>> > > > > > > > >> > > > > > >> > >
>> > > > > > > > >> > > > > > >> > >    - No objection to keeping the default
>> > > metrics
>> > > > > > > behavior
>> > > > > > > > >> > > small:
>> > > > > > > > >> > > > > > >> > >    no-op/log-only is enough as the
>> built-in
>> > > > > > default.
>> > > > > > > > >> > > > > > >> > >    - Durable storage, event forwarding,
>> > > external
>> > > > > > > queues,
>> > > > > > > > >> > > > > dashboards,
>> > > > > > > > >> > > > > > >> and
>> > > > > > > > >> > > > > > >> > >    custom filtering should be
>> > implementation
>> > > > > > choices
>> > > > > > > > >> behind
>> > > > > > > > >> > > the
>> > > > > > > > >> > > > > > >> metrics
>> > > > > > > > >> > > > > > >> > >    reporting path.
>> > > > > > > > >> > > > > > >> > >    - For PR #4115, the REST layer should
>> > > > > > > > >> > resolve/authz/accept
>> > > > > > > > >> > > the
>> > > > > > > > >> > > > > > >> report,
>> > > > > > > > >> > > > > > >> > >    then delegate to the selected metrics
>> > > > > > > implementation.
>> > > > > > > > >> It
>> > > > > > > > >> > > should
>> > > > > > > > >> > > > > > not
>> > > > > > > > >> > > > > > >> > own
>> > > > > > > > >> > > > > > >> > >    durable storage semantics.
>> > > > > > > > >> > > > > > >> > >    - The rough boundary we discussed is:
>> > spec
>> > > > > owns
>> > > > > > > the
>> > > > > > > > >> wire
>> > > > > > > > >> > > > > > contract,
>> > > > > > > > >> > > > > > >> > >    runtime owns framework wiring,
>> > > API/contract
>> > > > > > > modules
>> > > > > > > > >> own
>> > > > > > > > >> > > > > > >> > provider-facing
>> > > > > > > > >> > > > > > >> > >    contracts, and extensions own
>> > replaceable
>> > > > > > > > >> > implementations.
>> > > > > > > > >> > > > > > >> > >    - OpenLineage has a similar
>> path-naming
>> > > > > > question.
>> > > > > > > We
>> > > > > > > > >> > should
>> > > > > > > > >> > > > > avoid
>> > > > > > > > >> > > > > > >> > >    occupying a generic lineage path too
>> > > early if
>> > > > > we
>> > > > > > > want
>> > > > > > > > >> > room
>> > > > > > > > >> > > for
>> > > > > > > > >> > > > > > >> other
>> > > > > > > > >> > > > > > >> > >    lineage systems later.
>> > > > > > > > >> > > > > > >> > >
>> > > > > > > > >> > > > > > >> > > I believe the boundary rules previewed
>> in
>> > > sync
>> > > > > > > meeting
>> > > > > > > > >> are
>> > > > > > > > >> > > useful
>> > > > > > > > >> > > > > > for
>> > > > > > > > >> > > > > > >> > > reviewing PR #4115, but it is not
>> > > > > > > > >> > > > > > >> > > a final module-layout proposal.
>> > > > > > > > >> > > > > > >> > >
>> > > > > > > > >> > > > > > >> > > For metrics specifically, I think the
>> > target
>> > > > > > remains:
>> > > > > > > > >> > > > > > >> > >    runtime
>> > > > > > > > >> > > > > > >> > >      -> metrics reporting/emitting
>> contract
>> > > > > > > > >> > > > > > >> > >      -> default implementation
>> > > > > > > > >> > > > > > >> > >           -> lower implementation
>> details,
>> > if
>> > > > > > needed
>> > > > > > > > >> > > > > > >> > >
>> > > > > > > > >> > > > > > >> > > That keeps the default simple, while
>> > allowing
>> > > > > > durable
>> > > > > > > > >> JDBC,
>> > > > > > > > >> > > > > > >> event-backed,
>> > > > > > > > >> > > > > > >> > > or external-queue implementations to be
>> > added
>> > > > > > > separately.
>> > > > > > > > >> > > > > > >> > >
>> > > > > > > > >> > > > > > >> > > We also touched on the split between
>> > Iceberg
>> > > > > > metrics
>> > > > > > > > >> > emitting
>> > > > > > > > >> > > and
>> > > > > > > > >> > > > > > >> > > Polaris-owned metrics querying. The
>> > querying
>> > > side
>> > > > > > can
>> > > > > > > > >> keep
>> > > > > > > > >> > > > > evolving,
>> > > > > > > > >> > > > > > >> > > especially for dashboard or
>> generic-table
>> > use
>> > > > > > cases,
>> > > > > > > but
>> > > > > > > > >> I
>> > > > > > > > >> > do
>> > > > > > > > >> > > not
>> > > > > > > > >> > > > > > >> think
>> > > > > > > > >> > > > > > >> > > that needs to block the intake path in
>> this
>> > > PR.
>> > > > > > > > >> > > > > > >> > >
>> > > > > > > > >> > > > > > >> > > Thanks,
>> > > > > > > > >> > > > > > >> > > -ej
>> > > > > > > > >> > > > > > >> > >
>> > > > > > > > >> > > > > > >> > > On Tue, Jun 16, 2026 at 2:17 PM Yufei
>> Gu <
>> > > > > > > > >> > > flyrain000@gmail.com>
>> > > > > > > > >> > > > > > >> wrote:
>> > > > > > > > >> > > > > > >> > >
>> > > > > > > > >> > > > > > >> > > > Thanks for chiming in, EJ. Agreed that
>> > the
>> > > > > > default
>> > > > > > > > >> battery
>> > > > > > > > >> > > > > should
>> > > > > > > > >> > > > > > >> stay
>> > > > > > > > >> > > > > > >> > > > small. I'm leaning toward not
>> including
>> > > metrics
>> > > > > > > > >> > persistence
>> > > > > > > > >> > > in
>> > > > > > > > >> > > > > the
>> > > > > > > > >> > > > > > >> > > default
>> > > > > > > > >> > > > > > >> > > > battery.
>> > > > > > > > >> > > > > > >> > > >
>> > > > > > > > >> > > > > > >> > > > Yufei
>> > > > > > > > >> > > > > > >> > > >
>> > > > > > > > >> > > > > > >> > > >
>> > > > > > > > >> > > > > > >> > > > On Thu, Jun 11, 2026 at 8:30 PM EJ
>> Wang <
>> > > > > > > > >> > > > > > >> > ej.wang.dev.interest@gmail.com>
>> > > > > > > > >> > > > > > >> > > > wrote:
>> > > > > > > > >> > > > > > >> > > >
>> > > > > > > > >> > > > > > >> > > > > Hi Yufei,
>> > > > > > > > >> > > > > > >> > > > >
>> > > > > > > > >> > > > > > >> > > > > Thanks for connecting this back to
>> the
>> > > REST
>> > > > > > > endpoint
>> > > > > > > > >> > > > > proposal. I
>> > > > > > > > >> > > > > > >> > > replied
>> > > > > > > > >> > > > > > >> > > > > with the fuller version on the
>> > > > > event-forwarding
>> > > > > > > > >> thread,
>> > > > > > > > >> > > but
>> > > > > > > > >> > > > > > >> wanted to
>> > > > > > > > >> > > > > > >> > > add
>> > > > > > > > >> > > > > > >> > > > > the shorter version here too.
>> > > > > > > > >> > > > > > >> > > > >
>> > > > > > > > >> > > > > > >> > > > > I agree that metrics
>> reporting/emitting
>> > > is
>> > > > > the
>> > > > > > > right
>> > > > > > > > >> > > > > conceptual
>> > > > > > > > >> > > > > > >> > > boundary,
>> > > > > > > > >> > > > > > >> > > > > and I also think the event/listener
>> > path
>> > > is a
>> > > > > > > good
>> > > > > > > > >> > > > > > implementation
>> > > > > > > > >> > > > > > >> > > > direction
>> > > > > > > > >> > > > > > >> > > > > to explore. The distinction I want
>> to
>> > > keep
>> > > > > > clear
>> > > > > > > is
>> > > > > > > > >> > > default
>> > > > > > > > >> > > > > > >> battery
>> > > > > > > > >> > > > > > >> > vs
>> > > > > > > > >> > > > > > >> > > > > extension implementation.
>> > > > > > > > >> > > > > > >> > > > >
>> > > > > > > > >> > > > > > >> > > > > For the REST/API side, I would keep
>> the
>> > > > > > semantics
>> > > > > > > > >> > narrow:
>> > > > > > > > >> > > > > > Polaris
>> > > > > > > > >> > > > > > >> > > > resolves
>> > > > > > > > >> > > > > > >> > > > > and authorizes the table, accepts
>> the
>> > > Iceberg
>> > > > > > > > >> > scan/commit
>> > > > > > > > >> > > > > > metrics
>> > > > > > > > >> > > > > > >> > > report
>> > > > > > > > >> > > > > > >> > > > > into the configured
>> reporting/emitting
>> > > path,
>> > > > > > and
>> > > > > > > > >> returns
>> > > > > > > > >> > > 204.
>> > > > > > > > >> > > > > > That
>> > > > > > > > >> > > > > > >> > 204
>> > > > > > > > >> > > > > > >> > > > > should mean Polaris accepted the
>> report
>> > > into
>> > > > > > the
>> > > > > > > > >> > ingestion
>> > > > > > > > >> > > > > path;
>> > > > > > > > >> > > > > > >> it
>> > > > > > > > >> > > > > > >> > > > should
>> > > > > > > > >> > > > > > >> > > > > not imply durable storage.
>> > > > > > > > >> > > > > > >> > > > >
>> > > > > > > > >> > > > > > >> > > > > Durable storage, event forwarding,
>> > > external
>> > > > > > > telemetry
>> > > > > > > > >> > > routing,
>> > > > > > > > >> > > > > > >> > > filtering,
>> > > > > > > > >> > > > > > >> > > > > and retention should sit behind that
>> > > > > > > > >> reporting/emitting
>> > > > > > > > >> > > > > boundary
>> > > > > > > > >> > > > > > >> as
>> > > > > > > > >> > > > > > >> > > > > implementation-layer behavior. A
>> > durable
>> > > JDBC
>> > > > > > > path
>> > > > > > > > >> can
>> > > > > > > > >> > be
>> > > > > > > > >> > > one
>> > > > > > > > >> > > > > > >> named
>> > > > > > > > >> > > > > > >> > > > > extension implementation. An
>> > > event/listener
>> > > > > > > > >> forwarding
>> > > > > > > > >> > > path
>> > > > > > > > >> > > > > can
>> > > > > > > > >> > > > > > be
>> > > > > > > > >> > > > > > >> > > > another
>> > > > > > > > >> > > > > > >> > > > > named extension implementation.
>> > > > > > > > >> > > > > > >> > > > >
>> > > > > > > > >> > > > > > >> > > > > So I am +1 on the event-forwarding
>> idea
>> > > as a
>> > > > > > > > >> non-default
>> > > > > > > > >> > > > > > extension
>> > > > > > > > >> > > > > > >> > > > > implementation. I just would not
>> make
>> > it
>> > > the
>> > > > > > > default
>> > > > > > > > >> > > battery
>> > > > > > > > >> > > > > or
>> > > > > > > > >> > > > > > >> the
>> > > > > > > > >> > > > > > >> > > core
>> > > > > > > > >> > > > > > >> > > > > API shape. The default battery can
>> stay
>> > > small
>> > > > > > and
>> > > > > > > > >> safe,
>> > > > > > > > >> > > while
>> > > > > > > > >> > > > > > >> > > deployments
>> > > > > > > > >> > > > > > >> > > > > that need forwarding, dashboards, or
>> > > durable
>> > > > > > > storage
>> > > > > > > > >> can
>> > > > > > > > >> > > opt
>> > > > > > > > >> > > > > > into
>> > > > > > > > >> > > > > > >> the
>> > > > > > > > >> > > > > > >> > > > > implementation that matches their
>> > > operational
>> > > > > > > model.
>> > > > > > > > >> > > > > > >> > > > >
>> > > > > > > > >> > > > > > >> > > > > Thanks,
>> > > > > > > > >> > > > > > >> > > > > -ej
>> > > > > > > > >> > > > > > >> > > > >
>> > > > > > > > >> > > > > > >> > > > > On Wed, Jun 10, 2026 at 11:43 AM
>> Yufei
>> > > Gu <
>> > > > > > > > >> > > > > flyrain000@gmail.com
>> > > > > > > > >> > > > > > >
>> > > > > > > > >> > > > > > >> > > wrote:
>> > > > > > > > >> > > > > > >> > > > >
>> > > > > > > > >> > > > > > >> > > > > > Thanks EJ. I agree that the
>> > > > > > reporting/emitting
>> > > > > > > > >> > boundary
>> > > > > > > > >> > > > > feels
>> > > > > > > > >> > > > > > >> like
>> > > > > > > > >> > > > > > >> > > the
>> > > > > > > > >> > > > > > >> > > > > > right SPI boundary, but I wonder
>> if
>> > we
>> > > can
>> > > > > > > simplify
>> > > > > > > > >> > this
>> > > > > > > > >> > > > > even
>> > > > > > > > >> > > > > > >> > > further.
>> > > > > > > > >> > > > > > >> > > > > >
>> > > > > > > > >> > > > > > >> > > > > > Iceberg scan and commit metrics
>> look
>> > > very
>> > > > > > > similar
>> > > > > > > > >> to
>> > > > > > > > >> > > events
>> > > > > > > > >> > > > > to
>> > > > > > > > >> > > > > > >> me.
>> > > > > > > > >> > > > > > >> > > They
>> > > > > > > > >> > > > > > >> > > > > are
>> > > > > > > > >> > > > > > >> > > > > > append only, asynchronously
>> consumed,
>> > > often
>> > > > > > > > >> forwarded
>> > > > > > > > >> > to
>> > > > > > > > >> > > > > > >> external
>> > > > > > > > >> > > > > > >> > > > > systems,
>> > > > > > > > >> > > > > > >> > > > > > and have similar retention and
>> > cleanup
>> > > > > > > > >> requirements.
>> > > > > > > > >> > > More
>> > > > > > > > >> > > > > > >> details
>> > > > > > > > >> > > > > > >> > can
>> > > > > > > > >> > > > > > >> > > > be
>> > > > > > > > >> > > > > > >> > > > > > found in this thread [1]. In that
>> > case,
>> > > > > > > Polaris may
>> > > > > > > > >> > only
>> > > > > > > > >> > > > > need
>> > > > > > > > >> > > > > > to
>> > > > > > > > >> > > > > > >> > > accept
>> > > > > > > > >> > > > > > >> > > > > the
>> > > > > > > > >> > > > > > >> > > > > > report and emit an event through
>> the
>> > > > > existing
>> > > > > > > event
>> > > > > > > > >> > > > > framework.
>> > > > > > > > >> > > > > > >> > > > Filtering,
>> > > > > > > > >> > > > > > >> > > > > > async delivery, custom sinks, and
>> > > retention
>> > > > > > > > >> mechanisms
>> > > > > > > > >> > > could
>> > > > > > > > >> > > > > > >> then
>> > > > > > > > >> > > > > > >> > be
>> > > > > > > > >> > > > > > >> > > > > shared
>> > > > > > > > >> > > > > > >> > > > > > instead of introducing a separate
>> > > metrics
>> > > > > > > specific
>> > > > > > > > >> > > extension
>> > > > > > > > >> > > > > > >> layer
>> > > > > > > > >> > > > > > >> > > and
>> > > > > > > > >> > > > > > >> > > > > SPI.
>> > > > > > > > >> > > > > > >> > > > > >
>> > > > > > > > >> > > > > > >> > > > > > The main requirement I still see
>> is
>> > > > > querying
>> > > > > > > > >> metrics
>> > > > > > > > >> > > through
>> > > > > > > > >> > > > > > >> > Polaris.
>> > > > > > > > >> > > > > > >> > > > If
>> > > > > > > > >> > > > > > >> > > > > we
>> > > > > > > > >> > > > > > >> > > > > > want that battery included
>> > experience,
>> > > we
>> > > > > > could
>> > > > > > > > >> work
>> > > > > > > > >> > on
>> > > > > > > > >> > > a
>> > > > > > > > >> > > > > > >> > persistence
>> > > > > > > > >> > > > > > >> > > > > > listener implementation that
>> > > selectively
>> > > > > > saves
>> > > > > > > > >> metrics
>> > > > > > > > >> > > and
>> > > > > > > > >> > > > > > >> expose
>> > > > > > > > >> > > > > > >> > it
>> > > > > > > > >> > > > > > >> > > > via
>> > > > > > > > >> > > > > > >> > > > > > the IRC event endpoint which seems
>> > > pretty
>> > > > > > > close in
>> > > > > > > > >> > > Iceberg
>> > > > > > > > >> > > > > > >> > > > > > community. Another option I like
>> more
>> > > is to
>> > > > > > > allow
>> > > > > > > > >> > users
>> > > > > > > > >> > > to
>> > > > > > > > >> > > > > > >> develop
>> > > > > > > > >> > > > > > >> > > > their
>> > > > > > > > >> > > > > > >> > > > > > own listener, so they can persist
>> > > > > scan/commit
>> > > > > > > > >> metrics
>> > > > > > > > >> > > to any
>> > > > > > > > >> > > > > > >> > systems
>> > > > > > > > >> > > > > > >> > > > they
>> > > > > > > > >> > > > > > >> > > > > > prefer. In general, that feels
>> like
>> > an
>> > > > > > > > >> implementation
>> > > > > > > > >> > > choice
>> > > > > > > > >> > > > > > on
>> > > > > > > > >> > > > > > >> top
>> > > > > > > > >> > > > > > >> > > of
>> > > > > > > > >> > > > > > >> > > > > the
>> > > > > > > > >> > > > > > >> > > > > > event framework rather than
>> something
>> > > that
>> > > > > > > needs to
>> > > > > > > > >> > > shape
>> > > > > > > > >> > > > > the
>> > > > > > > > >> > > > > > >> core
>> > > > > > > > >> > > > > > >> > > > > > architecture.
>> > > > > > > > >> > > > > > >> > > > > >
>> > > > > > > > >> > > > > > >> > > > > > 1.
>> > > > > > > > >> > > > > > >> >
>> > > > > > > > >> > >
>> > > > > > > > >> >
>> > > > > > > > >>
>> > > > > > >
>> > > > > >
>> > > > >
>> > >
>> >
>> https://urldefense.com/v3/__https://lists.apache.org/thread/x9j8nscvy8hq61tyn01mj8yp6n9of0kp__;!!Iz9xO38YGHZK!4rnlVIqBQQQ6xpGe1PMV58edOS_cGrKoqdfhw4qSp_JQCavvEjaxXk0OwGPnxyEcO60AzKQK_XDtp7H3DdkFXcrXGcGDgks$
>> > > > > > > > >> > > > > > >> > > > > >
>> > > > > > > > >> > > > > > >> > > > > > Thanks,
>> > > > > > > > >> > > > > > >> > > > > > Yufei
>> > > > > > > > >> > > > > > >> > > > > >
>> > > > > > > > >> > > > > > >> > > > > >
>> > > > > > > > >> > > > > > >> > > > > > On Thu, Jun 4, 2026 at 5:14 PM EJ
>> > Wang
>> > > <
>> > > > > > > > >> > > > > > >> > > ej.wang.dev.interest@gmail.com
>> > > > > > > > >> > > > > > >> > > > >
>> > > > > > > > >> > > > > > >> > > > > > wrote:
>> > > > > > > > >> > > > > > >> > > > > >
>> > > > > > > > >> > > > > > >> > > > > > > Good point. I agree the existing
>> > > > > > > > >> > > PolarisMetricsReporter is
>> > > > > > > > >> > > > > > >> > already
>> > > > > > > > >> > > > > > >> > > > very
>> > > > > > > > >> > > > > > >> > > > > > > close to the right conceptual
>> > > boundary. I
>> > > > > > am
>> > > > > > > not
>> > > > > > > > >> > > > > proposing a
>> > > > > > > > >> > > > > > >> > second
>> > > > > > > > >> > > > > > >> > > > > > > parallel reporter concept. The
>> > > > > distinction
>> > > > > > I
>> > > > > > > am
>> > > > > > > > >> > > trying to
>> > > > > > > > >> > > > > > >> make is
>> > > > > > > > >> > > > > > >> > > > > > narrower:
>> > > > > > > > >> > > > > > >> > > > > > >
>> > > > > > > > >> > > > > > >> > > > > > > Current state:
>> > > > > > > > >> > > > > > >> > > > > > >
>> > > > > > > > >> > > > > > >> > > > > > >    - PolarisMetricsReporter is
>> the
>> > > > > existing
>> > > > > > > > >> metrics
>> > > > > > > > >> > > > > > reporting
>> > > > > > > > >> > > > > > >> > hook.
>> > > > > > > > >> > > > > > >> > > > > > >    - But it currently lives
>> under
>> > > > > > > > >> runtime/service.
>> > > > > > > > >> > > > > > >> > > > > > >    - Its contract is documented
>> > > around
>> > > > > > > > >> Quarkus/CDI
>> > > > > > > > >> > > > > discovery
>> > > > > > > > >> > > > > > >> and
>> > > > > > > > >> > > > > > >> > > > > > > @Identifier
>> > > > > > > > >> > > > > > >> > > > > > >    selection.
>> > > > > > > > >> > > > > > >> > > > > > >    - The default implementation
>> is
>> > > > > > log-only.
>> > > > > > > > >> > > > > > >> > > > > > >    - The durable implementation
>> > > writes
>> > > > > > > through
>> > > > > > > > >> > > > > > >> > > PolarisMetricsManager
>> > > > > > > > >> > > > > > >> > > > > and
>> > > > > > > > >> > > > > > >> > > > > > >    MetricsPersistence.
>> > > > > > > > >> > > > > > >> > > > > > >    - MetricsPersistence is
>> > currently
>> > > > > > > inherited by
>> > > > > > > > >> > > > > > >> > BasePersistence.
>> > > > > > > > >> > > > > > >> > > > > > >
>> > > > > > > > >> > > > > > >> > > > > > > My proposal:
>> > > > > > > > >> > > > > > >> > > > > > >
>> > > > > > > > >> > > > > > >> > > > > > >    - Keep the reporting/emitting
>> > > boundary
>> > > > > > as
>> > > > > > > the
>> > > > > > > > >> > SPI.
>> > > > > > > > >> > > > > > >> > > > > > >    - Revise that contract in a
>> > > > > > > framework-agnostic
>> > > > > > > > >> > > optional
>> > > > > > > > >> > > > > > >> > Iceberg
>> > > > > > > > >> > > > > > >> > > > > > *metrics
>> > > > > > > > >> > > > > > >> > > > > > >    extension API layer*.
>> > > > > > > > >> > > > > > >> > > > > > >    - Keep runtime/service
>> > > responsible for
>> > > > > > > REST
>> > > > > > > > >> > > ingestion,
>> > > > > > > > >> > > > > > >> > > table/authz
>> > > > > > > > >> > > > > > >> > > > > > >    validation, and runtime
>> wiring.
>> > > > > > > > >> > > > > > >> > > > > > >    - Keep the battery
>> > implementation
>> > > > > > > log-only in
>> > > > > > > > >> the
>> > > > > > > > >> > > > > metrics
>> > > > > > > > >> > > > > > >> > > > extension
>> > > > > > > > >> > > > > > >> > > > > > API
>> > > > > > > > >> > > > > > >> > > > > > >    layer, *not under
>> > > runtime/service*.
>> > > > > > > > >> > > > > > >> > > > > > >    - Treat durable JDBC metrics
>> as
>> > > one
>> > > > > > > > >> > implementation
>> > > > > > > > >> > > of
>> > > > > > > > >> > > > > the
>> > > > > > > > >> > > > > > >> > > > reporting
>> > > > > > > > >> > > > > > >> > > > > > SPI.
>> > > > > > > > >> > > > > > >> > > > > > >    - Decompose the
>> > > MetricsPersistence and
>> > > > > > > > >> > > BasePersistence
>> > > > > > > > >> > > > > > >> > coupling
>> > > > > > > > >> > > > > > >> > > as
>> > > > > > > > >> > > > > > >> > > > > > >    durable implementation
>> detail.
>> > > > > > > > >> > > > > > >> > > > > > >
>> > > > > > > > >> > > > > > >> > > > > > > So the difference is not “new
>> > > reporter
>> > > > > > > concept vs
>> > > > > > > > >> > old
>> > > > > > > > >> > > > > > reporter
>> > > > > > > > >> > > > > > >> > > > > concept.”
>> > > > > > > > >> > > > > > >> > > > > > > but:
>> > > > > > > > >> > > > > > >> > > > > > >
>> > > > > > > > >> > > > > > >> > > > > > >
>> > > > > > > > >> > > > > > >> > > > > > >    - PolarisMetricsReporter
>> today =
>> > > > > > > > >> runtime/service
>> > > > > > > > >> > > > > > CDI-shaped
>> > > > > > > > >> > > > > > >> > > hook.
>> > > > > > > > >> > > > > > >> > > > > > >    - Target reporting SPI = same
>> > > > > conceptual
>> > > > > > > > >> > boundary,
>> > > > > > > > >> > > but
>> > > > > > > > >> > > > > > >> placed
>> > > > > > > > >> > > > > > >> > in
>> > > > > > > > >> > > > > > >> > > > the
>> > > > > > > > >> > > > > > >> > > > > > >    right extension API layer and
>> > kept
>> > > > > > > > >> > > framework-agnostic.
>> > > > > > > > >> > > > > > >> > > > > > >
>> > > > > > > > >> > > > > > >> > > > > > > For the current metrics PR, I
>> think
>> > > the
>> > > > > > > minimal
>> > > > > > > > >> > > > > > architectural
>> > > > > > > > >> > > > > > >> > > cleanup
>> > > > > > > > >> > > > > > >> > > > > > > before merge is to *put the
>> > > reporting SPI
>> > > > > > > > >> contract
>> > > > > > > > >> > > and the
>> > > > > > > > >> > > > > > >> > battery
>> > > > > > > > >> > > > > > >> > > > > > default
>> > > > > > > > >> > > > > > >> > > > > > > in the right place*. The durable
>> > JDBC
>> > > > > > > > >> > implementation,
>> > > > > > > > >> > > > > > >> > event-backed
>> > > > > > > > >> > > > > > >> > > > > async
>> > > > > > > > >> > > > > > >> > > > > > > implementation, external queue
>> > > support,
>> > > > > and
>> > > > > > > > >> deeper
>> > > > > > > > >> > > > > > persistence
>> > > > > > > > >> > > > > > >> > > > cleanup
>> > > > > > > > >> > > > > > >> > > > > > can
>> > > > > > > > >> > > > > > >> > > > > > > continue as follow-ups, as long
>> as
>> > > we do
>> > > > > > not
>> > > > > > > lock
>> > > > > > > > >> > the
>> > > > > > > > >> > > SPI
>> > > > > > > > >> > > > > > >> > boundary
>> > > > > > > > >> > > > > > >> > > to
>> > > > > > > > >> > > > > > >> > > > > > > BasePersistence or to
>> > runtime/service
>> > > > > > wiring.
>> > > > > > > > >> > > > > > >> > > > > > >
>> > > > > > > > >> > > > > > >> > > > > > > -ej
>> > > > > > > > >> > > > > > >> > > > > > >
>> > > > > > > > >> > > > > > >> > > > > > > On Wed, Jun 3, 2026 at 3:00 PM
>> > Dmitri
>> > > > > > > > >> Bourlatchkov <
>> > > > > > > > >> > > > > > >> > > dimas@apache.org
>> > > > > > > > >> > > > > > >> > > > >
>> > > > > > > > >> > > > > > >> > > > > > > wrote:
>> > > > > > > > >> > > > > > >> > > > > > >
>> > > > > > > > >> > > > > > >> > > > > > > > Hi EJ,
>> > > > > > > > >> > > > > > >> > > > > > > >
>> > > > > > > > >> > > > > > >> > > > > > > > Thanks for the recap /
>> summary!
>> > > > > > > > >> > > > > > >> > > > > > > >
>> > > > > > > > >> > > > > > >> > > > > > > > > Do folks agree that the
>> stable
>> > > SPI
>> > > > > > > boundary
>> > > > > > > > >> > > should be
>> > > > > > > > >> > > > > > >> > > > > > > > metrics reporting/emitting,
>> not
>> > > metrics
>> > > > > > > > >> > persistence?
>> > > > > > > > >> > > > > > >> > > > > > > >
>> > > > > > > > >> > > > > > >> > > > > > > > SGTM.
>> > > > > > > > >> > > > > > >> > > > > > > >
>> > > > > > > > >> > > > > > >> > > > > > > > > Does an optional Iceberg
>> > metrics
>> > > > > > > extension
>> > > > > > > > >> API
>> > > > > > > > >> > > layer
>> > > > > > > > >> > > > > > sound
>> > > > > > > > >> > > > > > >> > like
>> > > > > > > > >> > > > > > >> > > > the
>> > > > > > > > >> > > > > > >> > > > > > > right
>> > > > > > > > >> > > > > > >> > > > > > > > home for this SPI?
>> > > > > > > > >> > > > > > >> > > > > > > >
>> > > > > > > > >> > > > > > >> > > > > > > > How is that different from
>> > current
>> > > > > > > > >> > > > > PolarisMetricsReporter?
>> > > > > > > > >> > > > > > >> > > > > > > >
>> > > > > > > > >> > > > > > >> > > > > > > > > Should the current durable
>> > > metrics
>> > > > > work
>> > > > > > > be
>> > > > > > > > >> > > reframed
>> > > > > > > > >> > > > > as a
>> > > > > > > > >> > > > > > >> > > durable
>> > > > > > > > >> > > > > > >> > > > > > > > JDBC reference implementation
>> of
>> > > that
>> > > > > > SPI?
>> > > > > > > > >> > > > > > >> > > > > > > >
>> > > > > > > > >> > > > > > >> > > > > > > > SGTM.
>> > > > > > > > >> > > > > > >> > > > > > > >
>> > > > > > > > >> > > > > > >> > > > > > > > > What is the smallest PR
>> > sequence
>> > > to
>> > > > > get
>> > > > > > > there
>> > > > > > > > >> > > without
>> > > > > > > > >> > > > > > >> > blocking
>> > > > > > > > >> > > > > > >> > > > > > > > the current metrics work
>> > > unnecessarily?
>> > > > > > > > >> > > > > > >> > > > > > > >
>> > > > > > > > >> > > > > > >> > > > > > > > Let's get
>> > > > > > > > >> > >
>> > > > > > > > >> >
>> > > > > > > > >>
>> > > > > > >
>> > > > > >
>> > > > >
>> > >
>> >
>> https://urldefense.com/v3/__https://github.com/apache/polaris/pull/4397__;!!Iz9xO38YGHZK!4rnlVIqBQQQ6xpGe1PMV58edOS_cGrKoqdfhw4qSp_JQCavvEjaxXk0OwGPnxyEcO60AzKQK_XDtp7H3DdkFXcrXuu5jefU$
>> > > > > > > > >> > > > > > >> merged
>> > > > > > > > >> > > > > > >> > > > first.
>> > > > > > > > >> > > > > > >> > > > > > > >
>> > > > > > > > >> > > > > > >> > > > > > > > Then, I'd propose isolating
>> the
>> > > Metrics
>> > > > > > > schema
>> > > > > > > > >> > from
>> > > > > > > > >> > > the
>> > > > > > > > >> > > > > > >> > MetaStore
>> > > > > > > > >> > > > > > >> > > > > > schema.
>> > > > > > > > >> > > > > > >> > > > > > > > This will probably have some
>> > ripple
>> > > > > > effect
>> > > > > > > into
>> > > > > > > > >> > the
>> > > > > > > > >> > > > > > >> bootstrap
>> > > > > > > > >> > > > > > >> > > > > > workflows,
>> > > > > > > > >> > > > > > >> > > > > > > so
>> > > > > > > > >> > > > > > >> > > > > > > > it's not a trivial change.
>> > > > > > > > >> > > > > > >> > > > > > > >
>> > > > > > > > >> > > > > > >> > > > > > > > Then, let's reassess.
>> > > > > > > > >> > > > > > >> > > > > > > >
>> > > > > > > > >> > > > > > >> > > > > > > > Cheers,
>> > > > > > > > >> > > > > > >> > > > > > > > Dmitri.
>> > > > > > > > >> > > > > > >> > > > > > > >
>> > > > > > > > >> > > > > > >> > > > > > > > On Wed, Jun 3, 2026 at
>> 5:40 PM EJ
>> > > Wang
>> > > > > <
>> > > > > > > > >> > > > > > >> > > > > ej.wang.dev.interest@gmail.com
>> > > > > > > > >> > > > > > >> > > > > > >
>> > > > > > > > >> > > > > > >> > > > > > > > wrote:
>> > > > > > > > >> > > > > > >> > > > > > > >
>> > > > > > > > >> > > > > > >> > > > > > > > > Hi folks,
>> > > > > > > > >> > > > > > >> > > > > > > > >
>> > > > > > > > >> > > > > > >> > > > > > > > > Thanks again for the
>> discussion
>> > > > > today.
>> > > > > > I
>> > > > > > > > >> updated
>> > > > > > > > >> > > the
>> > > > > > > > >> > > > > > sync
>> > > > > > > > >> > > > > > >> doc
>> > > > > > > > >> > > > > > >> > > > with
>> > > > > > > > >> > > > > > >> > > > > > > notes
>> > > > > > > > >> > > > > > >> > > > > > > > > from the third metrics
>> > > architecture
>> > > > > > sync:
>> > > > > > > > >> > > > > > >> > > > > > > > >
>> > > > > > > > >> > > > > > >> > > > > > > > >
>> > > > > > > > >> > > > > > >> > > > > > > > >
>> > > > > > > > >> > > > > > >> > > > > > > >
>> > > > > > > > >> > > > > > >> > > > > > >
>> > > > > > > > >> > > > > > >> > > > > >
>> > > > > > > > >> > > > > > >> > > > >
>> > > > > > > > >> > > > > > >> > > >
>> > > > > > > > >> > > > > > >> > >
>> > > > > > > > >> > > > > > >> >
>> > > > > > > > >> > > > > > >>
>> > > > > > > > >> > > > > >
>> > > > > > > > >> > > > >
>> > > > > > > > >> > >
>> > > > > > > > >> >
>> > > > > > > > >>
>> > > > > > >
>> > > > > >
>> > > > >
>> > >
>> >
>> https://urldefense.com/v3/__https://docs.google.com/document/d/100h7c4damrUzVuquYbBHM0EvA4LSWuW2IT2dN_7nYVA/edit?tab=t.k96s2xyqr5u1__;!!Iz9xO38YGHZK!4rnlVIqBQQQ6xpGe1PMV58edOS_cGrKoqdfhw4qSp_JQCavvEjaxXk0OwGPnxyEcO60AzKQK_XDtp7H3DdkFXcrXrqCL-QE$
>> > > > > > > > >> > > > > > >> > > > > > > > >
>> > > > > > > > >> > > > > > >> > > > > > > > > A few highlights from
>> today’s
>> > > > > > discussion:
>> > > > > > > > >> > > > > > >> > > > > > > > >
>> > > > > > > > >> > > > > > >> > > > > > > > >    - We clarified that
>> Iceberg
>> > > > > metrics
>> > > > > > > > >> reporting
>> > > > > > > > >> > > can
>> > > > > > > > >> > > > > be
>> > > > > > > > >> > > > > > >> > > > interpreted
>> > > > > > > > >> > > > > > >> > > > > > > > either
>> > > > > > > > >> > > > > > >> > > > > > > > >    as sync or async handling
>> > from
>> > > > > > > Polaris’s
>> > > > > > > > >> > > > > perspective,
>> > > > > > > > >> > > > > > >> so
>> > > > > > > > >> > > > > > >> > > > Polaris
>> > > > > > > > >> > > > > > >> > > > > > > > should
>> > > > > > > > >> > > > > > >> > > > > > > > >    stay flexible as a
>> platform
>> > > > > instead
>> > > > > > of
>> > > > > > > > >> baking
>> > > > > > > > >> > > one
>> > > > > > > > >> > > > > > >> handling
>> > > > > > > > >> > > > > > >> > > > model
>> > > > > > > > >> > > > > > >> > > > > > > into
>> > > > > > > > >> > > > > > >> > > > > > > > > the
>> > > > > > > > >> > > > > > >> > > > > > > > >    REST API behavior.
>> > > > > > > > >> > > > > > >> > > > > > > > >    - We aligned that the
>> > built-in
>> > > > > > > (battery)
>> > > > > > > > >> > > behavior
>> > > > > > > > >> > > > > for
>> > > > > > > > >> > > > > > >> > > metrics
>> > > > > > > > >> > > > > > >> > > > > > > emitting
>> > > > > > > > >> > > > > > >> > > > > > > > >    can stay simple:
>> > > no-op/log-only is
>> > > > > > > enough
>> > > > > > > > >> as
>> > > > > > > > >> > > the
>> > > > > > > > >> > > > > > >> default.
>> > > > > > > > >> > > > > > >> > > > > > > > >    - Durable metrics
>> > persistence
>> > > > > should
>> > > > > > > be
>> > > > > > > > >> > > treated as
>> > > > > > > > >> > > > > an
>> > > > > > > > >> > > > > > >> > > > > > implementation
>> > > > > > > > >> > > > > > >> > > > > > > > of
>> > > > > > > > >> > > > > > >> > > > > > > > >    the metrics reporting
>> path,
>> > > not as
>> > > > > > the
>> > > > > > > > >> core
>> > > > > > > > >> > SPI
>> > > > > > > > >> > > > > > >> boundary.
>> > > > > > > > >> > > > > > >> > > > > > > > >    - The existing durable
>> > metrics
>> > > > > work
>> > > > > > > can be
>> > > > > > > > >> > > reviewed
>> > > > > > > > >> > > > > > as
>> > > > > > > > >> > > > > > >> a
>> > > > > > > > >> > > > > > >> > > > > reference
>> > > > > > > > >> > > > > > >> > > > > > > > >    implementation of the
>> > > reporting
>> > > > > SPI,
>> > > > > > > with
>> > > > > > > > >> > > > > > >> > > persistence-related
>> > > > > > > > >> > > > > > >> > > > > > logic
>> > > > > > > > >> > > > > > >> > > > > > > > kept
>> > > > > > > > >> > > > > > >> > > > > > > > >    self-contained under a
>> > metrics
>> > > > > > durable
>> > > > > > > > >> > > > > implementation
>> > > > > > > > >> > > > > > >> > module
>> > > > > > > > >> > > > > > >> > > > > > rather
>> > > > > > > > >> > > > > > >> > > > > > > > than
>> > > > > > > > >> > > > > > >> > > > > > > > >    scattered through core
>> > entity
>> > > > > > > persistence.
>> > > > > > > > >> > > > > > >> > > > > > > > >    - Dashboard/insights
>> > remains a
>> > > > > real
>> > > > > > > use
>> > > > > > > > >> case,
>> > > > > > > > >> > > but
>> > > > > > > > >> > > > > we
>> > > > > > > > >> > > > > > >> > agreed
>> > > > > > > > >> > > > > > >> > > to
>> > > > > > > > >> > > > > > >> > > > > > keep
>> > > > > > > > >> > > > > > >> > > > > > > it
>> > > > > > > > >> > > > > > >> > > > > > > > >    separate from the core
>> > metrics
>> > > > > > intake
>> > > > > > > > >> > > discussion
>> > > > > > > > >> > > > > for
>> > > > > > > > >> > > > > > >> now.
>> > > > > > > > >> > > > > > >> > > > > > > > >
>> > > > > > > > >> > > > > > >> > > > > > > > > I also did a quick source
>> check
>> > > after
>> > > > > > the
>> > > > > > > > >> > meeting
>> > > > > > > > >> > > to
>> > > > > > > > >> > > > > > make
>> > > > > > > > >> > > > > > >> > sure
>> > > > > > > > >> > > > > > >> > > we
>> > > > > > > > >> > > > > > >> > > > > are
>> > > > > > > > >> > > > > > >> > > > > > > > > describing the current state
>> > > > > > accurately.
>> > > > > > > > >> > > > > > >> > > > > > > > >
>> > > > > > > > >> > > > > > >> > > > > > > > > Current state:
>> > > > > > > > >> > > > > > >> > > > > > > > >
>> > > > > > > > >> > > > > > >> > > > > > > > >    - Polaris already has a
>> > > metrics
>> > > > > > > reporting
>> > > > > > > > >> > hook:
>> > > > > > > > >> > > > > > >> > > > > > > > PolarisMetricsReporter.
>> > > > > > > > >> > > > > > >> > > > > > > > >    - The default
>> implementation
>> > > is
>> > > > > > > > >> > > > > > DefaultMetricsReporter,
>> > > > > > > > >> > > > > > >> > > > selected
>> > > > > > > > >> > > > > > >> > > > > > by
>> > > > > > > > >> > > > > > >> > > > > > > > >
>> > > > > > > > >> > > polaris.iceberg-metrics.reporting.type=default. It
>> > > > > > > > >> > > > > is
>> > > > > > > > >> > > > > > >> > > > log-only,
>> > > > > > > > >> > > > > > >> > > > > > > > >    effectively quiet unless
>> > > metrics
>> > > > > > > logging
>> > > > > > > > >> is
>> > > > > > > > >> > > > > enabled.
>> > > > > > > > >> > > > > > >> > > > > > > > >    - There is also a
>> > > > > > > > >> PersistingMetricsReporter,
>> > > > > > > > >> > > > > selected
>> > > > > > > > >> > > > > > >> by
>> > > > > > > > >> > > > > > >> > > > > > > > >
>> > > > > > > > >> > > polaris.iceberg-metrics.reporting.type=persisting,
>> > > > > > > > >> > > > > > >> which
>> > > > > > > > >> > > > > > >> > > > > converts
>> > > > > > > > >> > > > > > >> > > > > > > > >    Iceberg scan/commit
>> reports
>> > > into
>> > > > > > > Polaris
>> > > > > > > > >> > > metrics
>> > > > > > > > >> > > > > > >> records
>> > > > > > > > >> > > > > > >> > and
>> > > > > > > > >> > > > > > >> > > > > > writes
>> > > > > > > > >> > > > > > >> > > > > > > > > through
>> > > > > > > > >> > > > > > >> > > > > > > > >    PolarisMetricsManager ->
>> > > > > > > > >> MetricsPersistence.
>> > > > > > > > >> > > > > > >> > > > > > > > >    - MetricsPersistence
>> > currently
>> > > > > > exists
>> > > > > > > in
>> > > > > > > > >> the
>> > > > > > > > >> > > > > > >> persistence
>> > > > > > > > >> > > > > > >> > > layer
>> > > > > > > > >> > > > > > >> > > > > and
>> > > > > > > > >> > > > > > >> > > > > > > is
>> > > > > > > > >> > > > > > >> > > > > > > > >    inherited by
>> > BasePersistence.
>> > > > > > > > >> > > > > > >> > > > > > > > >
>> > > > > > > > >> > > > > > >> > > > > > > > > My read from the discussion
>> is
>> > > that
>> > > > > the
>> > > > > > > > >> target
>> > > > > > > > >> > > > > boundary
>> > > > > > > > >> > > > > > >> > should
>> > > > > > > > >> > > > > > >> > > > not
>> > > > > > > > >> > > > > > >> > > > > be
>> > > > > > > > >> > > > > > >> > > > > > > > > MetricsPersistence as
>> inherited
>> > > by
>> > > > > > > > >> > > BasePersistence.
>> > > > > > > > >> > > > > That
>> > > > > > > > >> > > > > > >> path
>> > > > > > > > >> > > > > > >> > > > > should
>> > > > > > > > >> > > > > > >> > > > > > be
>> > > > > > > > >> > > > > > >> > > > > > > > > decomposed as durable
>> > > implementation
>> > > > > > > detail
>> > > > > > > > >> > (taken
>> > > > > > > > >> > > > > care
>> > > > > > > > >> > > > > > >> of by
>> > > > > > > > >> > > > > > >> > > > > > > > >
>> > > > > > > > >> >
>> > > > > > > > >>
>> > > > > > >
>> > > > > >
>> > > > >
>> > >
>> >
>> https://urldefense.com/v3/__https://github.com/apache/polaris/pull/4397__;!!Iz9xO38YGHZK!4rnlVIqBQQQ6xpGe1PMV58edOS_cGrKoqdfhw4qSp_JQCavvEjaxXk0OwGPnxyEcO60AzKQK_XDtp7H3DdkFXcrXuu5jefU$
>> > > > > > > > >> ).
>> > > > > > > > >> > The
>> > > > > > > > >> > > > > > stable
>> > > > > > > > >> > > > > > >> > > > extension
>> > > > > > > > >> > > > > > >> > > > > > > point
>> > > > > > > > >> > > > > > >> > > > > > > > > should instead be the
>> metrics
>> > > > > > > > >> reporting/emitting
>> > > > > > > > >> > > > > > boundary.
>> > > > > > > > >> > > > > > >> > > > > > > > >
>> > > > > > > > >> > > > > > >> > > > > > > > > Concretely, I think the next
>> > > proposal
>> > > > > > > should
>> > > > > > > > >> be
>> > > > > > > > >> > > shaped
>> > > > > > > > >> > > > > > >> like
>> > > > > > > > >> > > > > > >> > > this:
>> > > > > > > > >> > > > > > >> > > > > > > > >
>> > > > > > > > >> > > > > > >> > > > > > > > >    1.
>> > > > > > > > >> > > > > > >> > > > > > > > >
>> > > > > > > > >> > > > > > >> > > > > > > > >    Define the proper metrics
>> > > > > reporting
>> > > > > > > SPI
>> > > > > > > > >> at an
>> > > > > > > > >> > > > > > optional
>> > > > > > > > >> > > > > > >> > > Iceberg
>> > > > > > > > >> > > > > > >> > > > > > > metrics
>> > > > > > > > >> > > > > > >> > > > > > > > >    extension API layer.
>> > > > > > > > >> > > > > > >> > > > > > > > >
>> > > > > > > > >> > > > > > >> > > > > > > > >    Example direction:
>> > > > > > > > >> > > > > > >> > > > > > > > >
>> > > > > > > > >> > > > > > >> > > > > > > > >    public interface
>> > > > > > > IcebergMetricsReporter {
>> > > > > > > > >> > > > > > >> > > > > > > > >      void
>> > > > > > > > >> > reportMetric(IcebergMetricsReportContext
>> > > > > > > > >> > > > > > >> context,
>> > > > > > > > >> > > > > > >> > > > > > > > > MetricsReport report);
>> > > > > > > > >> > > > > > >> > > > > > > > >    }
>> > > > > > > > >> > > > > > >> > > > > > > > >
>> > > > > > > > >> > > > > > >> > > > > > > > >    The context would carry
>> the
>> > > small
>> > > > > > > > >> > > Polaris-resolved
>> > > > > > > > >> > > > > > >> > envelope:
>> > > > > > > > >> > > > > > >> > > > > > > > >    catalog/table identity,
>> > report
>> > > > > type,
>> > > > > > > > >> received
>> > > > > > > > >> > > > > > >> timestamp,
>> > > > > > > > >> > > > > > >> > and
>> > > > > > > > >> > > > > > >> > > > > > > > > request/trace
>> > > > > > > > >> > > > > > >> > > > > > > > >    context if available. The
>> > raw
>> > > > > > Iceberg
>> > > > > > > > >> > > MetricsReport
>> > > > > > > > >> > > > > > >> > remains
>> > > > > > > > >> > > > > > >> > > > the
>> > > > > > > > >> > > > > > >> > > > > > > > payload.
>> > > > > > > > >> > > > > > >> > > > > > > > >    2.
>> > > > > > > > >> > > > > > >> > > > > > > > >
>> > > > > > > > >> > > > > > >> > > > > > > > >    Keep runtime/service as
>> the
>> > > REST
>> > > > > > > ingestion
>> > > > > > > > >> > and
>> > > > > > > > >> > > > > wiring
>> > > > > > > > >> > > > > > >> > layer.
>> > > > > > > > >> > > > > > >> > > > > > > > >
>> > > > > > > > >> > > > > > >> > > > > > > > >    The REST handler still
>> > > resolves
>> > > > > the
>> > > > > > > table
>> > > > > > > > >> and
>> > > > > > > > >> > > > > > performs
>> > > > > > > > >> > > > > > >> > authz
>> > > > > > > > >> > > > > > >> > > > > > before
>> > > > > > > > >> > > > > > >> > > > > > > > >    accepting the report.
>> After
>> > > that,
>> > > > > it
>> > > > > > > calls
>> > > > > > > > >> > the
>> > > > > > > > >> > > > > > selected
>> > > > > > > > >> > > > > > >> > > > > reporting
>> > > > > > > > >> > > > > > >> > > > > > > > >    implementation.
>> > > > > > > > >> > > > > > >> > > > > > > > >    3.
>> > > > > > > > >> > > > > > >> > > > > > > > >
>> > > > > > > > >> > > > > > >> > > > > > > > >    Keep the battery default
>> > > > > > > no-op/log-only.
>> > > > > > > > >> > > > > > >> > > > > > > > >
>> > > > > > > > >> > > > > > >> > > > > > > > >    This preserves an
>> out-of-box
>> > > safe
>> > > > > > > default
>> > > > > > > > >> and
>> > > > > > > > >> > > > > avoids
>> > > > > > > > >> > > > > > >> > > > requiring a
>> > > > > > > > >> > > > > > >> > > > > > > > durable
>> > > > > > > > >> > > > > > >> > > > > > > > >    metrics store for every
>> > > Polaris
>> > > > > > > > >> deployment.
>> > > > > > > > >> > > > > > >> > > > > > > > >    4.
>> > > > > > > > >> > > > > > >> > > > > > > > >
>> > > > > > > > >> > > > > > >> > > > > > > > >    Move durable JDBC metrics
>> > > into a
>> > > > > > > > >> > self-contained
>> > > > > > > > >> > > > > > >> > > implementation
>> > > > > > > > >> > > > > > >> > > > > of
>> > > > > > > > >> > > > > > >> > > > > > > the
>> > > > > > > > >> > > > > > >> > > > > > > > >    reporting SPI.
>> > > > > > > > >> > > > > > >> > > > > > > > >
>> > > > > > > > >> > > > > > >> > > > > > > > >    That implementation can
>> own
>> > > its
>> > > > > > > schema,
>> > > > > > > > >> > > bootstrap,
>> > > > > > > > >> > > > > > >> > > retention,
>> > > > > > > > >> > > > > > >> > > > > and
>> > > > > > > > >> > > > > > >> > > > > > > read
>> > > > > > > > >> > > > > > >> > > > > > > > >    API support. It should
>> not
>> > > define
>> > > > > > the
>> > > > > > > core
>> > > > > > > > >> > > > > reporting
>> > > > > > > > >> > > > > > >> SPI
>> > > > > > > > >> > > > > > >> > > > > boundary,
>> > > > > > > > >> > > > > > >> > > > > > > and
>> > > > > > > > >> > > > > > >> > > > > > > > > it
>> > > > > > > > >> > > > > > >> > > > > > > > >    should not require
>> metrics
>> > > > > > > persistence to
>> > > > > > > > >> > > remain
>> > > > > > > > >> > > > > > >> inherited
>> > > > > > > > >> > > > > > >> > > > from
>> > > > > > > > >> > > > > > >> > > > > > > > >    BasePersistence.
>> > > > > > > > >> > > > > > >> > > > > > > > >    5.
>> > > > > > > > >> > > > > > >> > > > > > > > >
>> > > > > > > > >> > > > > > >> > > > > > > > >    Treat async/event-backed
>> > > handling
>> > > > > as
>> > > > > > > > >> another
>> > > > > > > > >> > > > > > >> > implementation
>> > > > > > > > >> > > > > > >> > > of
>> > > > > > > > >> > > > > > >> > > > > the
>> > > > > > > > >> > > > > > >> > > > > > > > same
>> > > > > > > > >> > > > > > >> > > > > > > > >    reporting SPI.
>> > > > > > > > >> > > > > > >> > > > > > > > >
>> > > > > > > > >> > > > > > >> > > > > > > > >    For example, an
>> event-backed
>> > > > > > reporter
>> > > > > > > > >> could
>> > > > > > > > >> > > > > enqueue a
>> > > > > > > > >> > > > > > >> > > metrics
>> > > > > > > > >> > > > > > >> > > > > > event
>> > > > > > > > >> > > > > > >> > > > > > > > and
>> > > > > > > > >> > > > > > >> > > > > > > > >    let listeners handle
>> durable
>> > > > > storage
>> > > > > > > or
>> > > > > > > > >> other
>> > > > > > > > >> > > > > sinks.
>> > > > > > > > >> > > > > > >> If we
>> > > > > > > > >> > > > > > >> > > > later
>> > > > > > > > >> > > > > > >> > > > > > > need
>> > > > > > > > >> > > > > > >> > > > > > > > a
>> > > > > > > > >> > > > > > >> > > > > > > > >    replaceable queue engine,
>> > that
>> > > > > seems
>> > > > > > > like
>> > > > > > > > >> a
>> > > > > > > > >> > > shared
>> > > > > > > > >> > > > > > >> > > > event/metrics
>> > > > > > > > >> > > > > > >> > > > > > > > > substrate
>> > > > > > > > >> > > > > > >> > > > > > > > >    topic rather than a
>> > > metrics-only
>> > > > > > > > >> requirement.
>> > > > > > > > >> > > > > > >> > > > > > > > >
>> > > > > > > > >> > > > > > >> > > > > > > > > This framing lets us keep
>> the
>> > > REST
>> > > > > > > metrics
>> > > > > > > > >> > > endpoint
>> > > > > > > > >> > > > > > >> simple,
>> > > > > > > > >> > > > > > >> > > > > preserve
>> > > > > > > > >> > > > > > >> > > > > > > the
>> > > > > > > > >> > > > > > >> > > > > > > > > current default behavior,
>> > support
>> > > > > > durable
>> > > > > > > > >> > metrics
>> > > > > > > > >> > > > > users,
>> > > > > > > > >> > > > > > >> and
>> > > > > > > > >> > > > > > >> > > > still
>> > > > > > > > >> > > > > > >> > > > > > > leave
>> > > > > > > > >> > > > > > >> > > > > > > > > room for async/event-backed
>> or
>> > > > > > > > >> > > external-queue-based
>> > > > > > > > >> > > > > > >> > > > > implementations.
>> > > > > > > > >> > > > > > >> > > > > > > > >
>> > > > > > > > >> > > > > > >> > > > > > > > > I think the main follow-up
>> > > questions
>> > > > > > are:
>> > > > > > > > >> > > > > > >> > > > > > > > >
>> > > > > > > > >> > > > > > >> > > > > > > > >    - Do folks agree that the
>> > > stable
>> > > > > SPI
>> > > > > > > > >> boundary
>> > > > > > > > >> > > > > should
>> > > > > > > > >> > > > > > be
>> > > > > > > > >> > > > > > >> > > > metrics
>> > > > > > > > >> > > > > > >> > > > > > > > >    reporting/emitting, not
>> > > metrics
>> > > > > > > > >> persistence?
>> > > > > > > > >> > > > > > >> > > > > > > > >    - Does an optional
>> Iceberg
>> > > metrics
>> > > > > > > > >> extension
>> > > > > > > > >> > > API
>> > > > > > > > >> > > > > > layer
>> > > > > > > > >> > > > > > >> > sound
>> > > > > > > > >> > > > > > >> > > > > like
>> > > > > > > > >> > > > > > >> > > > > > > the
>> > > > > > > > >> > > > > > >> > > > > > > > >    right home for this SPI?
>> > > > > > > > >> > > > > > >> > > > > > > > >    - Should the current
>> durable
>> > > > > metrics
>> > > > > > > work
>> > > > > > > > >> be
>> > > > > > > > >> > > > > reframed
>> > > > > > > > >> > > > > > >> as a
>> > > > > > > > >> > > > > > >> > > > > durable
>> > > > > > > > >> > > > > > >> > > > > > > > JDBC
>> > > > > > > > >> > > > > > >> > > > > > > > >    reference implementation
>> of
>> > > that
>> > > > > > SPI?
>> > > > > > > > >> > > > > > >> > > > > > > > >    - What is the smallest PR
>> > > sequence
>> > > > > > to
>> > > > > > > get
>> > > > > > > > >> > there
>> > > > > > > > >> > > > > > without
>> > > > > > > > >> > > > > > >> > > > blocking
>> > > > > > > > >> > > > > > >> > > > > > the
>> > > > > > > > >> > > > > > >> > > > > > > > >    current metrics work
>> > > > > unnecessarily?
>> > > > > > > > >> > > > > > >> > > > > > > > >
>> > > > > > > > >> > > > > > >> > > > > > > > > Thanks,
>> > > > > > > > >> > > > > > >> > > > > > > > > -ej
>> > > > > > > > >> > > > > > >> > > > > > > > >
>> > > > > > > > >> > > > > > >> > > > > > > > > On Fri, May 15, 2026 at
>> 5:16 PM
>> > > > > Dmitri
>> > > > > > > > >> > > Bourlatchkov <
>> > > > > > > > >> > > > > > >> > > > > > dimas@apache.org>
>> > > > > > > > >> > > > > > >> > > > > > > > > wrote:
>> > > > > > > > >> > > > > > >> > > > > > > > >
>> > > > > > > > >> > > > > > >> > > > > > > > > > Hi JB,
>> > > > > > > > >> > > > > > >> > > > > > > > > >
>> > > > > > > > >> > > > > > >> > > > > > > > > > Could you set up another
>> > > meeting,
>> > > > > > > please?
>> > > > > > > > >> Same
>> > > > > > > > >> > > time
>> > > > > > > > >> > > > > on
>> > > > > > > > >> > > > > > >> > > > Wednesday
>> > > > > > > > >> > > > > > >> > > > > as
>> > > > > > > > >> > > > > > >> > > > > > > > last
>> > > > > > > > >> > > > > > >> > > > > > > > > > time... I hope it works
>> for
>> > > > > everyone.
>> > > > > > > > >> > > > > > >> > > > > > > > > >
>> > > > > > > > >> > > > > > >> > > > > > > > > > Cheers,
>> > > > > > > > >> > > > > > >> > > > > > > > > > Dmitri.
>> > > > > > > > >> > > > > > >> > > > > > > > > >
>> > > > > > > > >> > > > > > >> > > > > > > > > > On Fri, May 15, 2026 at
>> > 8:06 PM
>> > > > > Yufei
>> > > > > > > Gu <
>> > > > > > > > >> > > > > > >> > > flyrain000@gmail.com
>> > > > > > > > >> > > > > > >> > > > >
>> > > > > > > > >> > > > > > >> > > > > > > wrote:
>> > > > > > > > >> > > > > > >> > > > > > > > > >
>> > > > > > > > >> > > > > > >> > > > > > > > > > > +1 on another sync call
>> > next
>> > > > > week.
>> > > > > > > > >> > > > > > >> > > > > > > > > > >
>> > > > > > > > >> > > > > > >> > > > > > > > > > > Yufei
>> > > > > > > > >> > > > > > >> > > > > > > > > > >
>> > > > > > > > >> > > > > > >> > > > > > > > > > >
>> > > > > > > > >> > > > > > >> > > > > > > > > > > On Fri, May 15, 2026 at
>> > > 4:52 PM
>> > > > > > > Dmitri
>> > > > > > > > >> > > > > Bourlatchkov
>> > > > > > > > >> > > > > > <
>> > > > > > > > >> > > > > > >> > > > > > > > dimas@apache.org>
>> > > > > > > > >> > > > > > >> > > > > > > > > > > wrote:
>> > > > > > > > >> > > > > > >> > > > > > > > > > >
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > Hi All,
>> > > > > > > > >> > > > > > >> > > > > > > > > > > >
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > WDYT about another
>> sync
>> > > call
>> > > > > next
>> > > > > > > week?
>> > > > > > > > >> > > > > > >> > > > > > > > > > > >
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > Thanks,
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > Dmitri.
>> > > > > > > > >> > > > > > >> > > > > > > > > > > >
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > On Wed, May 6, 2026 at
>> > > 5:29 PM
>> > > > > > > Dmitri
>> > > > > > > > >> > > > > > Bourlatchkov <
>> > > > > > > > >> > > > > > >> > > > > > > > dimas@apache.org
>> > > > > > > > >> > > > > > >> > > > > > > > > >
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > wrote:
>> > > > > > > > >> > > > > > >> > > > > > > > > > > >
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > > Hi EJ,
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > > Thanks for the
>> summary!
>> > > It
>> > > > > > covers
>> > > > > > > > >> what
>> > > > > > > > >> > we
>> > > > > > > > >> > > > > > >> discussed
>> > > > > > > > >> > > > > > >> > in
>> > > > > > > > >> > > > > > >> > > > the
>> > > > > > > > >> > > > > > >> > > > > > > > meeting
>> > > > > > > > >> > > > > > >> > > > > > > > > > very
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > > well, IMHO.
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > > Looking forward to
>> > > concrete
>> > > > > PRs
>> > > > > > > :)
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > > Cheers,
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > > Dmitri.
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > > On Wed, May 6, 2026
>> at
>> > > > > 5:08 PM
>> > > > > > EJ
>> > > > > > > > >> Wang <
>> > > > > > > > >> > > > > > >> > > > > > > > > >
>> > ej.wang.dev.interest@gmail.com
>> > > > > > > > >> > > > > > >> > > > > > > > > > > >
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > > wrote:
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> Hi folks,
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >>
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> We had a community
>> > sync
>> > > > > > earlier,
>> > > > > > > > >> thanks
>> > > > > > > > >> > > JB
>> > > > > > > > >> > > > > for
>> > > > > > > > >> > > > > > >> > > > scheduling
>> > > > > > > > >> > > > > > >> > > > > > it.
>> > > > > > > > >> > > > > > >> > > > > > > > > Notes
>> > > > > > > > >> > > > > > >> > > > > > > > > > > from
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> the first metrics
>> > > > > architecture
>> > > > > > > sync
>> > > > > > > > >> > (May
>> > > > > > > > >> > > 6,
>> > > > > > > > >> > > > > > >> 10-11am
>> > > > > > > > >> > > > > > >> > > PT).
>> > > > > > > > >> > > > > > >> > > > > > > > > Discussion
>> > > > > > > > >> > > > > > >> > > > > > > > > > > doc
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> with per-section
>> > status:
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >>
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >>
>> > > > > > > > >> > > > > > >> > > > > > > > > > > >
>> > > > > > > > >> > > > > > >> > > > > > > > > > >
>> > > > > > > > >> > > > > > >> > > > > > > > > >
>> > > > > > > > >> > > > > > >> > > > > > > > >
>> > > > > > > > >> > > > > > >> > > > > > > >
>> > > > > > > > >> > > > > > >> > > > > > >
>> > > > > > > > >> > > > > > >> > > > > >
>> > > > > > > > >> > > > > > >> > > > >
>> > > > > > > > >> > > > > > >> > > >
>> > > > > > > > >> > > > > > >> > >
>> > > > > > > > >> > > > > > >> >
>> > > > > > > > >> > > > > > >>
>> > > > > > > > >> > > > > >
>> > > > > > > > >> > > > >
>> > > > > > > > >> > >
>> > > > > > > > >> >
>> > > > > > > > >>
>> > > > > > >
>> > > > > >
>> > > > >
>> > >
>> >
>> https://urldefense.com/v3/__https://docs.google.com/document/d/100h7c4damrUzVuquYbBHM0EvA4LSWuW2IT2dN_7nYVA/edit?tab=t.0__;!!Iz9xO38YGHZK!4rnlVIqBQQQ6xpGe1PMV58edOS_cGrKoqdfhw4qSp_JQCavvEjaxXk0OwGPnxyEcO60AzKQK_XDtp7H3DdkFXcrXgIddeUs$
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >>
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> *The meeting
>> covered
>> > > both
>> > > > > > topics
>> > > > > > > > >> from
>> > > > > > > > >> > the
>> > > > > > > > >> > > > > doc.
>> > > > > > > > >> > > > > > >> > > > > > Direction-level
>> > > > > > > > >> > > > > > >> > > > > > > > > > > alignment
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> was reached on the
>> > > headline
>> > > > > > > pieces;
>> > > > > > > > >> > > details
>> > > > > > > > >> > > > > > >> remain
>> > > > > > > > >> > > > > > >> > for
>> > > > > > > > >> > > > > > >> > > > PR
>> > > > > > > > >> > > > > > >> > > > > > > review
>> > > > > > > > >> > > > > > >> > > > > > > > > or
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> follow-up
>> sessions.*
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >>
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> *Topic 1 —
>> Persistence
>> > > > > schema
>> > > > > > > > >> redesign*
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> Idea-level
>> alignment
>> > on
>> > > > > > > > >> consolidating
>> > > > > > > > >> > > > > per-type
>> > > > > > > > >> > > > > > >> > tables
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >>
>> (scan_metrics_report,
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >>
>> commit_metrics_report)
>> > > into
>> > > > > a
>> > > > > > > single
>> > > > > > > > >> > > > > > >> metrics_report
>> > > > > > > > >> > > > > > >> > > > table.
>> > > > > > > > >> > > > > > >> > > > > > The
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > motivating
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> cost is the surface
>> > area
>> > > > > added
>> > > > > > > by
>> > > > > > > > >> every
>> > > > > > > > >> > > new
>> > > > > > > > >> > > > > > >> metric
>> > > > > > > > >> > > > > > >> > > type
>> > > > > > > > >> > > > > > >> > > > > > today:
>> > > > > > > > >> > > > > > >> > > > > > > > new
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > table,
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> SPI method, record
>> > > class,
>> > > > > > model,
>> > > > > > > > >> > > converter,
>> > > > > > > > >> > > > > > >> schema
>> > > > > > > > >> > > > > > >> > > > > > migration.
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >>
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> Most schema details
>> > are
>> > > > > > > deferred to
>> > > > > > > > >> the
>> > > > > > > > >> > > > > schema
>> > > > > > > > >> > > > > > >> PR. A
>> > > > > > > > >> > > > > > >> > > few
>> > > > > > > > >> > > > > > >> > > > > > > > specific
>> > > > > > > > >> > > > > > >> > > > > > > > > > > points
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> came up:
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> •
>> > >  metric_schema_version:
>> > > > > > Yufei
>> > > > > > > > >> > prefers
>> > > > > > > > >> > > > > > dropping
>> > > > > > > > >> > > > > > >> > it,
>> > > > > > > > >> > > > > > >> > > > > since
>> > > > > > > > >> > > > > > >> > > > > > > > there
>> > > > > > > > >> > > > > > >> > > > > > > > > is
>> > > > > > > > >> > > > > > >> > > > > > > > > > > no
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> spec-level concept
>> of
>> > > > > metrics
>> > > > > > > > >> > versioning
>> > > > > > > > >> > > > > today
>> > > > > > > > >> > > > > > >> and
>> > > > > > > > >> > > > > > >> > it
>> > > > > > > > >> > > > > > >> > > is
>> > > > > > > > >> > > > > > >> > > > > > hard
>> > > > > > > > >> > > > > > >> > > > > > > to
>> > > > > > > > >> > > > > > >> > > > > > > > > > > define
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> unilaterally.
>> Robert
>> > > prefers
>> > > > > > > keeping
>> > > > > > > > >> > it,
>> > > > > > > > >> > > > > given
>> > > > > > > > >> > > > > > >> IRC
>> > > > > > > > >> > > > > > >> > v2
>> > > > > > > > >> > > > > > >> > > is
>> > > > > > > > >> > > > > > >> > > > > > > coming
>> > > > > > > > >> > > > > > >> > > > > > > > > and
>> > > > > > > > >> > > > > > >> > > > > > > > > > > the
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> schema should be
>> > > considered
>> > > > > > > against
>> > > > > > > > >> its
>> > > > > > > > >> > > > > likely
>> > > > > > > > >> > > > > > >> > shape;
>> > > > > > > > >> > > > > > >> > > > > Robert
>> > > > > > > > >> > > > > > >> > > > > > > > also
>> > > > > > > > >> > > > > > >> > > > > > > > > > > raised
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> how to
>> differentiate
>> > > various
>> > > > > > > payload
>> > > > > > > > >> > > formats
>> > > > > > > > >> > > > > if
>> > > > > > > > >> > > > > > >> any.
>> > > > > > > > >> > > > > > >> > > > EJ's
>> > > > > > > > >> > > > > > >> > > > > > read
>> > > > > > > > >> > > > > > >> > > > > > > > is
>> > > > > > > > >> > > > > > >> > > > > > > > > > that
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> this
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> is a two-way-door
>> > > decision.
>> > > > > We
>> > > > > > > can
>> > > > > > > > >> > start
>> > > > > > > > >> > > > > > without
>> > > > > > > > >> > > > > > >> the
>> > > > > > > > >> > > > > > >> > > > > field,
>> > > > > > > > >> > > > > > >> > > > > > > and
>> > > > > > > > >> > > > > > >> > > > > > > > if
>> > > > > > > > >> > > > > > >> > > > > > > > > > IRC
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > v2
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> changes the shape
>> we
>> > > would
>> > > > > > > likely
>> > > > > > > > >> roll
>> > > > > > > > >> > a
>> > > > > > > > >> > > > > > >> > corresponding
>> > > > > > > > >> > > > > > >> > > > new
>> > > > > > > > >> > > > > > >> > > > > > > > schema
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > anyway,
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> which is not
>> > > particularly
>> > > > > > > costly.
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> •   Payload format:
>> > > Robert
>> > > > > > > pointed
>> > > > > > > > >> out
>> > > > > > > > >> > > that
>> > > > > > > > >> > > > > > >> future
>> > > > > > > > >> > > > > > >> > > > formats
>> > > > > > > > >> > > > > > >> > > > > > > > beyond
>> > > > > > > > >> > > > > > >> > > > > > > > > > JSON
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > may
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> be worth
>> supporting.
>> > The
>> > > > > exact
>> > > > > > > > >> shape is
>> > > > > > > > >> > > > > > deferred
>> > > > > > > > >> > > > > > >> to
>> > > > > > > > >> > > > > > >> > > the
>> > > > > > > > >> > > > > > >> > > > > > schema
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > discussion.
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> •   Partition
>> > strategy:
>> > > > > Anand
>> > > > > > > > >> suggested
>> > > > > > > > >> > > > > monthly
>> > > > > > > > >> > > > > > >> > > > > partitioning
>> > > > > > > > >> > > > > > >> > > > > > > > based
>> > > > > > > > >> > > > > > >> > > > > > > > > > on
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > his
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> experience as
>> > > potentially
>> > > > > > > helpful at
>> > > > > > > > >> > > scale.
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >>
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> *Topic 2 — Where
>> > metrics
>> > > > > > > ingestion
>> > > > > > > > >> and
>> > > > > > > > >> > > > > storage
>> > > > > > > > >> > > > > > >> > belong*
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> Idea-level
>> alignment
>> > > that
>> > > > > > > metrics
>> > > > > > > > >> > should
>> > > > > > > > >> > > be a
>> > > > > > > > >> > > > > > >> > > separated
>> > > > > > > > >> > > > > > >> > > > > SPI
>> > > > > > > > >> > > > > > >> > > > > > > from
>> > > > > > > > >> > > > > > >> > > > > > > > > the
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> entity
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> persistence stack.
>> Two
>> > > > > reasons
>> > > > > > > > >> > surfaced:
>> > > > > > > > >> > > (a)
>> > > > > > > > >> > > > > > >> > workloads
>> > > > > > > > >> > > > > > >> > > > and
>> > > > > > > > >> > > > > > >> > > > > > > > > > capability
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> requirements
>> diverge
>> > > enough
>> > > > > > that
>> > > > > > > > >> > coupling
>> > > > > > > > >> > > > > them
>> > > > > > > > >> > > > > > >> > creates
>> > > > > > > > >> > > > > > >> > > > > > > > artificial
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> constraints, and
>> (b)
>> > > admin
>> > > > > > > > >> experience
>> > > > > > > > >> > > > > improves
>> > > > > > > > >> > > > > > >> when
>> > > > > > > > >> > > > > > >> > > > > metrics
>> > > > > > > > >> > > > > > >> > > > > > > has
>> > > > > > > > >> > > > > > >> > > > > > > > > its
>> > > > > > > > >> > > > > > >> > > > > > > > > > > own
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> bootstrap,
>> retention,
>> > > and
>> > > > > > > lifecycle.
>> > > > > > > > >> > > Dmitri
>> > > > > > > > >> > > > > > noted
>> > > > > > > > >> > > > > > >> > > > Polaris
>> > > > > > > > >> > > > > > >> > > > > > > being
>> > > > > > > > >> > > > > > >> > > > > > > > a
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > platform
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> should have the
>> > > flexibility
>> > > > > to
>> > > > > > > > >> support
>> > > > > > > > >> > > > > > different
>> > > > > > > > >> > > > > > >> > > > > persistence
>> > > > > > > > >> > > > > > >> > > > > > > > > > backends
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > per
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> concern, and
>> pointed
>> > to
>> > > a
>> > > > > > > concrete
>> > > > > > > > >> next
>> > > > > > > > >> > > step
>> > > > > > > > >> > > > > of
>> > > > > > > > >> > > > > > >> > > > separating
>> > > > > > > > >> > > > > > >> > > > > > the
>> > > > > > > > >> > > > > > >> > > > > > > > > JDBC
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> bootstrap for
>> metrics
>> > > from
>> > > > > the
>> > > > > > > > >> > metastore
>> > > > > > > > >> > > > > > >> bootstrap.
>> > > > > > > > >> > > > > > >> > > > Robert
>> > > > > > > > >> > > > > > >> > > > > > > > > proposed
>> > > > > > > > >> > > > > > >> > > > > > > > > > an
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> additional UX
>> > extension:
>> > > > > > detect
>> > > > > > > an
>> > > > > > > > >> > > > > > unbootstrapped
>> > > > > > > > >> > > > > > >> > > > metrics
>> > > > > > > > >> > > > > > >> > > > > > > store
>> > > > > > > > >> > > > > > >> > > > > > > > on
>> > > > > > > > >> > > > > > >> > > > > > > > > > > first
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> use and
>> auto-bootstrap
>> > > > > rather
>> > > > > > > than
>> > > > > > > > >> > > requiring
>> > > > > > > > >> > > > > an
>> > > > > > > > >> > > > > > >> > > explicit
>> > > > > > > > >> > > > > > >> > > > > > > manual
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > bootstrap
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> step.
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> The meeting also
>> > > confirmed
>> > > > > > that
>> > > > > > > > >> Polaris
>> > > > > > > > >> > > > > metrics
>> > > > > > > > >> > > > > > >> can
>> > > > > > > > >> > > > > > >> > > > start
>> > > > > > > > >> > > > > > >> > > > > > > small
>> > > > > > > > >> > > > > > >> > > > > > > > > and
>> > > > > > > > >> > > > > > >> > > > > > > > > > > stay
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> Iceberg-focused.
>> > Naming
>> > > and
>> > > > > > > > >> persistence
>> > > > > > > > >> > > > > schema
>> > > > > > > > >> > > > > > >> can
>> > > > > > > > >> > > > > > >> > > lean
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > Iceberg-specific.
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> If a future
>> expansion
>> > to
>> > > > > > > > >> generic-table
>> > > > > > > > >> > > > > metrics
>> > > > > > > > >> > > > > > or
>> > > > > > > > >> > > > > > >> > > > > > operational
>> > > > > > > > >> > > > > > >> > > > > > > > > > metrics
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> arrives, an
>> > abstraction
>> > > > > layer
>> > > > > > > can be
>> > > > > > > > >> > > built on
>> > > > > > > > >> > > > > > >> top of
>> > > > > > > > >> > > > > > >> > > the
>> > > > > > > > >> > > > > > >> > > > > > > Iceberg
>> > > > > > > > >> > > > > > >> > > > > > > > > > > metrics
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> reporter at that
>> > point.
>> > > > > Robert
>> > > > > > > > >> remains
>> > > > > > > > >> > > on the
>> > > > > > > > >> > > > > > >> fence
>> > > > > > > > >> > > > > > >> > > and
>> > > > > > > > >> > > > > > >> > > > > > would
>> > > > > > > > >> > > > > > >> > > > > > > > > prefer
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> something more
>> generic
>> > > but
>> > > > > did
>> > > > > > > not
>> > > > > > > > >> > block
>> > > > > > > > >> > > the
>> > > > > > > > >> > > > > > >> > > direction;
>> > > > > > > > >> > > > > > >> > > > > > > Dmitri's
>> > > > > > > > >> > > > > > >> > > > > > > > > > read
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > was
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> that the proposed
>> > > framework
>> > > > > > > already
>> > > > > > > > >> has
>> > > > > > > > >> > > > > enough
>> > > > > > > > >> > > > > > >> > > > flexibility
>> > > > > > > > >> > > > > > >> > > > > > to
>> > > > > > > > >> > > > > > >> > > > > > > > > absorb
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> future
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> expansion.
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >>
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> The Trade-offs and
>> > > Proposed
>> > > > > > > > >> structure
>> > > > > > > > >> > > > > sections
>> > > > > > > > >> > > > > > in
>> > > > > > > > >> > > > > > >> > the
>> > > > > > > > >> > > > > > >> > > > doc
>> > > > > > > > >> > > > > > >> > > > > > were
>> > > > > > > > >> > > > > > >> > > > > > > > not
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> reviewed
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> in detail. They
>> remain
>> > > open
>> > > > > > for
>> > > > > > > > >> either
>> > > > > > > > >> > > the
>> > > > > > > > >> > > > > next
>> > > > > > > > >> > > > > > >> sync
>> > > > > > > > >> > > > > > >> > > or
>> > > > > > > > >> > > > > > >> > > > PR
>> > > > > > > > >> > > > > > >> > > > > > > > review.
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >>
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> *Cross-cutting
>> > > alignment —
>> > > > > > > > >> > > battery-included
>> > > > > > > > >> > > > > > plus
>> > > > > > > > >> > > > > > >> > > > > pluggable*
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> A common philosophy
>> > > emerged
>> > > > > > > from the
>> > > > > > > > >> > > > > > discussion.
>> > > > > > > > >> > > > > > >> EJ
>> > > > > > > > >> > > > > > >> > > > > > summarized
>> > > > > > > > >> > > > > > >> > > > > > > > it
>> > > > > > > > >> > > > > > >> > > > > > > > > > as:
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> Polaris should
>> > provide a
>> > > > > > > > >> > > battery-included UX
>> > > > > > > > >> > > > > > for
>> > > > > > > > >> > > > > > >> > > > beginners
>> > > > > > > > >> > > > > > >> > > > > > and
>> > > > > > > > >> > > > > > >> > > > > > > > the
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> flexibility for
>> > advanced
>> > > > > users
>> > > > > > > to
>> > > > > > > > >> swap
>> > > > > > > > >> > > the
>> > > > > > > > >> > > > > > >> included
>> > > > > > > > >> > > > > > >> > > > > battery
>> > > > > > > > >> > > > > > >> > > > > > > for
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > something
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> more powerful or
>> > > tailored to
>> > > > > > > their
>> > > > > > > > >> use
>> > > > > > > > >> > > case.
>> > > > > > > > >> > > > > > The
>> > > > > > > > >> > > > > > >> SPI
>> > > > > > > > >> > > > > > >> > > > > design
>> > > > > > > > >> > > > > > >> > > > > > > > needs
>> > > > > > > > >> > > > > > >> > > > > > > > > to
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> enable
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> both.
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >>
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> The inputs that
>> shaped
>> > > this
>> > > > > > > framing:
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> •   Anand described
>> > how
>> > > his
>> > > > > > team
>> > > > > > > > >> uses
>> > > > > > > > >> > the
>> > > > > > > > >> > > > > > current
>> > > > > > > > >> > > > > > >> > > > metrics
>> > > > > > > > >> > > > > > >> > > > > > > > > > persistence
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> (three metrics
>> > > consumers in
>> > > > > > > v1.4).
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> •   Yufei raised
>> > > Grafana and
>> > > > > > > > >> dashboard
>> > > > > > > > >> > > > > > >> integrations
>> > > > > > > > >> > > > > > >> > > as a
>> > > > > > > > >> > > > > > >> > > > > > > > > destination
>> > > > > > > > >> > > > > > >> > > > > > > > > > > use
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> case beyond the
>> > default.
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> •   Robert called
>> out
>> > > that
>> > > > > the
>> > > > > > > > >> current
>> > > > > > > > >> > > design
>> > > > > > > > >> > > > > > is
>> > > > > > > > >> > > > > > >> > more
>> > > > > > > > >> > > > > > >> > > > > > > > > JDBC-focused.
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >>
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> Two concrete
>> > instances:
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> •   Async metrics
>> > > intake:
>> > > > > > > Yufei's
>> > > > > > > > >> > initial
>> > > > > > > > >> > > > > > >> position
>> > > > > > > > >> > > > > > >> > was
>> > > > > > > > >> > > > > > >> > > > > that
>> > > > > > > > >> > > > > > >> > > > > > > > async
>> > > > > > > > >> > > > > > >> > > > > > > > > > > should
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> largely live on the
>> > > producer
>> > > > > > > side
>> > > > > > > > >> and
>> > > > > > > > >> > > there
>> > > > > > > > >> > > > > is
>> > > > > > > > >> > > > > > >> not
>> > > > > > > > >> > > > > > >> > > much
>> > > > > > > > >> > > > > > >> > > > > > > Polaris
>> > > > > > > > >> > > > > > >> > > > > > > > > can
>> > > > > > > > >> > > > > > >> > > > > > > > > > > do.
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> Robert suggested a
>> > > > > > Polaris-side
>> > > > > > > > >> default
>> > > > > > > > >> > > is
>> > > > > > > > >> > > > > > doable
>> > > > > > > > >> > > > > > >> > via
>> > > > > > > > >> > > > > > >> > > > > > Vert.x.
>> > > > > > > > >> > > > > > >> > > > > > > > > Dmitri
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> agreed
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> the direction is
>> worth
>> > > > > > > exploring.
>> > > > > > > > >> The
>> > > > > > > > >> > > meeting
>> > > > > > > > >> > > > > > >> > > converged
>> > > > > > > > >> > > > > > >> > > > > on a
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> battery-included
>> > default
>> > > > > > (likely
>> > > > > > > > >> > > > > Vert.x-backed)
>> > > > > > > > >> > > > > > >> with
>> > > > > > > > >> > > > > > >> > > an
>> > > > > > > > >> > > > > > >> > > > > SPI
>> > > > > > > > >> > > > > > >> > > > > > > > shape
>> > > > > > > > >> > > > > > >> > > > > > > > > > that
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> lets
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> power users route
>> to a
>> > > more
>> > > > > > > scalable
>> > > > > > > > >> > > backend
>> > > > > > > > >> > > > > > >> > > (k8s-hosted
>> > > > > > > > >> > > > > > >> > > > > > > queue,
>> > > > > > > > >> > > > > > >> > > > > > > > > AWS
>> > > > > > > > >> > > > > > >> > > > > > > > > > > SQS,
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> etc.).
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> •   Pluggable
>> > > destinations:
>> > > > > > > > >> combining
>> > > > > > > > >> > > Yufei's
>> > > > > > > > >> > > > > > >> > > dashboard
>> > > > > > > > >> > > > > > >> > > > > use
>> > > > > > > > >> > > > > > >> > > > > > > case
>> > > > > > > > >> > > > > > >> > > > > > > > > > with
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> Robert's
>> JDBC-focused
>> > > > > > call-out,
>> > > > > > > the
>> > > > > > > > >> > > meeting
>> > > > > > > > >> > > > > > >> agreed
>> > > > > > > > >> > > > > > >> > the
>> > > > > > > > >> > > > > > >> > > > SPI
>> > > > > > > > >> > > > > > >> > > > > > > > should
>> > > > > > > > >> > > > > > >> > > > > > > > > be
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> structured for
>> > multiple
>> > > > > sinks
>> > > > > > so
>> > > > > > > > >> > > integrations
>> > > > > > > > >> > > > > > >> become
>> > > > > > > > >> > > > > > >> > > > impl
>> > > > > > > > >> > > > > > >> > > > > > > > choices
>> > > > > > > > >> > > > > > >> > > > > > > > > > > rather
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> than architectural
>> > > changes.
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >>
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> The
>> battery-included
>> > > default
>> > > > > > is
>> > > > > > > most
>> > > > > > > > >> > > likely
>> > > > > > > > >> > > > > to
>> > > > > > > > >> > > > > > >> use
>> > > > > > > > >> > > > > > >> > the
>> > > > > > > > >> > > > > > >> > > > > > > existing
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> JDBC-backed
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> approach.
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >>
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> *Direction
>> (idea-level
>> > > > > > > alignment)*
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> •   Single
>> > > metrics_report
>> > > > > > table
>> > > > > > > > >> > > consolidating
>> > > > > > > > >> > > > > > >> > per-type
>> > > > > > > > >> > > > > > >> > > > > > > metrics,
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > replacing
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> scan_metrics_report
>> > and
>> > > > > > > > >> > > commit_metrics_report
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> •   Iceberg-focused
>> > > naming
>> > > > > and
>> > > > > > > > >> schema
>> > > > > > > > >> > for
>> > > > > > > > >> > > > > now,
>> > > > > > > > >> > > > > > >> > revisit
>> > > > > > > > >> > > > > > >> > > > if
>> > > > > > > > >> > > > > > >> > > > > > > > > > > generic-table
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > or
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> operational metrics
>> > > arrive
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> •   Metrics
>> > persistence
>> > > as a
>> > > > > > > > >> separated
>> > > > > > > > >> > > SPI,
>> > > > > > > > >> > > > > not
>> > > > > > > > >> > > > > > >> on
>> > > > > > > > >> > > > > > >> > > > > > > > BasePersistence
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> •   Bootstrap path
>> > > separated
>> > > > > > for
>> > > > > > > > >> > metrics,
>> > > > > > > > >> > > > > > >> > independent
>> > > > > > > > >> > > > > > >> > > of
>> > > > > > > > >> > > > > > >> > > > > > > > metastore
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> bootstrap
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> •
>>  "Battery-included
>> > > plus
>> > > > > > > > >> pluggable"
>> > > > > > > > >> > as
>> > > > > > > > >> > > the
>> > > > > > > > >> > > > > > SPI
>> > > > > > > > >> > > > > > >> > > design
>> > > > > > > > >> > > > > > >> > > > > > > > philosophy
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >>
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> *Open items*
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> •   Schema details:
>> > > > > > > > >> > > metric_schema_version,
>> > > > > > > > >> > > > > > >> payload
>> > > > > > > > >> > > > > > >> > > > format,
>> > > > > > > > >> > > > > > >> > > > > > IRC
>> > > > > > > > >> > > > > > >> > > > > > > > v2
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> forward-compat
>> shape
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> •   SPI design
>> > details —
>> > > > > full
>> > > > > > > review
>> > > > > > > > >> > > either
>> > > > > > > > >> > > > > in
>> > > > > > > > >> > > > > > >> the
>> > > > > > > > >> > > > > > >> > > next
>> > > > > > > > >> > > > > > >> > > > > sync
>> > > > > > > > >> > > > > > >> > > > > > > or
>> > > > > > > > >> > > > > > >> > > > > > > > in
>> > > > > > > > >> > > > > > >> > > > > > > > > > the
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> corresponding PR
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> •   Schema
>> refactor PR
>> > > > > > ownership
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >>
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> *Action items*
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> •   EJ to take a
>> first
>> > > stab
>> > > > > at
>> > > > > > > the
>> > > > > > > > >> SPI
>> > > > > > > > >> > > design
>> > > > > > > > >> > > > > > and
>> > > > > > > > >> > > > > > >> > > > > > potentially
>> > > > > > > > >> > > > > > >> > > > > > > > > > partner
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > with
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> Anand to
>> incorporate
>> > the
>> > > > > > lessons
>> > > > > > > > >> > learned
>> > > > > > > > >> > > from
>> > > > > > > > >> > > > > > the
>> > > > > > > > >> > > > > > >> > > > existing
>> > > > > > > > >> > > > > > >> > > > > > > > > reporter
>> > > > > > > > >> > > > > > >> > > > > > > > > > > and
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> persistence work.
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> •   Schema
>> refactor PR
>> > > > > > > ownership is
>> > > > > > > > >> not
>> > > > > > > > >> > > yet
>> > > > > > > > >> > > > > > >> decided.
>> > > > > > > > >> > > > > > >> > > If
>> > > > > > > > >> > > > > > >> > > > > > anyone
>> > > > > > > > >> > > > > > >> > > > > > > > is
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> interested in
>> driving
>> > > it,
>> > > > > > reply
>> > > > > > > on
>> > > > > > > > >> this
>> > > > > > > > >> > > > > thread.
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> •   JB to schedule
>> the
>> > > next
>> > > > > > > sync,
>> > > > > > > > >> > > tentatively
>> > > > > > > > >> > > > > > in
>> > > > > > > > >> > > > > > >> two
>> > > > > > > > >> > > > > > >> > > > > weeks.
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >>
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> -ej
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >>
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> On Mon, Apr 27,
>> 2026
>> > at
>> > > > > > 3:07 PM
>> > > > > > > EJ
>> > > > > > > > >> > Wang <
>> > > > > > > > >> > > > > > >> > > > > > > > > > >
>> > > ej.wang.dev.interest@gmail.com
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> wrote:
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >>
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> > Thanks Yufei for
>> the
>> > > +1.
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> > JB, could you
>> help
>> > > add a
>> > > > > > > biweekly
>> > > > > > > > >> > > metrics
>> > > > > > > > >> > > > > > >> > > architecture
>> > > > > > > > >> > > > > > >> > > > > > sync
>> > > > > > > > >> > > > > > >> > > > > > > to
>> > > > > > > > >> > > > > > >> > > > > > > > > the
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> Polaris
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> > community
>> calendar?
>> > > I'm
>> > > > > > > thinking
>> > > > > > > > >> > > Thursdays
>> > > > > > > > >> > > > > at
>> > > > > > > > >> > > > > > >> > 9-10am
>> > > > > > > > >> > > > > > >> > > > PT,
>> > > > > > > > >> > > > > > >> > > > > > on
>> > > > > > > > >> > > > > > >> > > > > > > > the
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> off-weeks
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> > from the
>> community
>> > > meeting
>> > > > > > > > >> (starting
>> > > > > > > > >> > > May
>> > > > > > > > >> > > > > 7),
>> > > > > > > > >> > > > > > 60
>> > > > > > > > >> > > > > > >> > > > minutes.
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> > Here's a rough
>> > agenda
>> > > to
>> > > > > > work
>> > > > > > > > >> through
>> > > > > > > > >> > > over
>> > > > > > > > >> > > > > > the
>> > > > > > > > >> > > > > > >> > first
>> > > > > > > > >> > > > > > >> > > > few
>> > > > > > > > >> > > > > > >> > > > > > > > > sessions,
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> grouped
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> > by priority:
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> > *First:
>> foundational
>> > > > > > > direction*
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> > 1.
>> > > MetricsPersistence:
>> > > > > > public
>> > > > > > > > >> SPI or
>> > > > > > > > >> > > > > > internal
>> > > > > > > > >> > > > > > >> > > > > > > implementation
>> > > > > > > > >> > > > > > >> > > > > > > > > > > detail?
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >    •   Marked
>> @Beta,
>> > > > > javadoc
>> > > > > > > calls
>> > > > > > > > >> > it a
>> > > > > > > > >> > > > > > >> "Service
>> > > > > > > > >> > > > > > >> > > > > Provider
>> > > > > > > > >> > > > > > >> > > > > > > > > > > Interface",
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> but
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> > only one consumer
>> > > > > > > > >> > > > > (JdbcBasePersistenceImpl),
>> > > > > > > > >> > > > > > >> lives
>> > > > > > > > >> > > > > > >> > > on
>> > > > > > > > >> > > > > > >> > > > > > > > > > > BasePersistence.
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> If
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> > demoted to a
>> private
>> > > > > helper
>> > > > > > > > >> inside a
>> > > > > > > > >> > > > > > persisting
>> > > > > > > > >> > > > > > >> > > > reporter
>> > > > > > > > >> > > > > > >> > > > > > > impl,
>> > > > > > > > >> > > > > > >> > > > > > > > > > most
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> > downstream design
>> > > > > decisions
>> > > > > > > become
>> > > > > > > > >> > > > > > >> implementation
>> > > > > > > > >> > > > > > >> > > > > details
>> > > > > > > > >> > > > > > >> > > > > > > > rather
>> > > > > > > > >> > > > > > >> > > > > > > > > > > than
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> > contract
>> questions.
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> > 2.  Persistence
>> > schema
>> > > > > > > redesign
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >    •   Current
>> > > two-table
>> > > > > > > layout
>> > > > > > > > >> > > > > > >> > > (scan_metrics_report,
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >
>> > commit_metrics_report)
>> > > > > with
>> > > > > > > ~25
>> > > > > > > > >> > > flattened
>> > > > > > > > >> > > > > > >> columns
>> > > > > > > > >> > > > > > >> > > > each.
>> > > > > > > > >> > > > > > >> > > > > > > Every
>> > > > > > > > >> > > > > > >> > > > > > > > > new
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > metric
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> > type requires a
>> new
>> > > table,
>> > > > > > SPI
>> > > > > > > > >> > method,
>> > > > > > > > >> > > > > record
>> > > > > > > > >> > > > > > >> > class,
>> > > > > > > > >> > > > > > >> > > > > > model,
>> > > > > > > > >> > > > > > >> > > > > > > > > > > converter,
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> and
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> > schema migration.
>> > > > > Direction
>> > > > > > to
>> > > > > > > > >> > explore:
>> > > > > > > > >> > > > > > single
>> > > > > > > > >> > > > > > >> > table
>> > > > > > > > >> > > > > > >> > > > > with
>> > > > > > > > >> > > > > > >> > > > > > > > > > > metric_type
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> enum,
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> > schema_version,
>> and
>> > > JSON
>> > > > > > > payload
>> > > > > > > > >> > > column.
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> > *Second: design
>> > > details
>> > > > > once
>> > > > > > > > >> > direction
>> > > > > > > > >> > > is
>> > > > > > > > >> > > > > > set*
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> > 3.  Partition key
>> > > strategy
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >    •
>>  Single-table
>> > > design
>> > > > > > > means
>> > > > > > > > >> scan
>> > > > > > > > >> > > > > metrics
>> > > > > > > > >> > > > > > >> at
>> > > > > > > > >> > > > > > >> > > scale
>> > > > > > > > >> > > > > > >> > > > > > will
>> > > > > > > > >> > > > > > >> > > > > > > > have
>> > > > > > > > >> > > > > > >> > > > > > > > > > > high
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> > write concurrency
>> > per
>> > > > > table.
>> > > > > > > > >> Schema
>> > > > > > > > >> > > needs
>> > > > > > > > >> > > > > to
>> > > > > > > > >> > > > > > >> > expose
>> > > > > > > > >> > > > > > >> > > > > enough
>> > > > > > > > >> > > > > > >> > > > > > > > > > structure
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > for
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> > backends to
>> shard by
>> > > > > entity
>> > > > > > or
>> > > > > > > > >> time
>> > > > > > > > >> > > range.
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> > 4.  Read/write
>> path
>> > > > > > > consistency
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >    •   Writes go
>> > > through
>> > > > > > > > >> > > > > > PolarisMetricsManager
>> > > > > > > > >> > > > > > >> on
>> > > > > > > > >> > > > > > >> > > > > > > > > > MetaStoreManager.
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> Reads
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> > bypass
>> > > MetaStoreManager
>> > > > > and
>> > > > > > go
>> > > > > > > > >> > > straight to
>> > > > > > > > >> > > > > > >> > > > > > BasePersistence,
>> > > > > > > > >> > > > > > >> > > > > > > > > > > excluding
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> > non-JDBC backends
>> > > from the
>> > > > > > > read
>> > > > > > > > >> API.
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> > *Third: cleanup
>> and
>> > > > > > alignment*
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> > 5.
>> > > PolarisMetricsReporter
>> > > > > > > naming
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >    •   Only
>> handles
>> > > IRC
>> > > > > > > > >> > > > > > >> (ScanReport/CommitReport),
>> > > > > > > > >> > > > > > >> > > > > doesn't
>> > > > > > > > >> > > > > > >> > > > > > > > cover
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > generic
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> > tables or
>> > operational
>> > > > > > metrics.
>> > > > > > > > >> Name
>> > > > > > > > >> > is
>> > > > > > > > >> > > > > > broader
>> > > > > > > > >> > > > > > >> > than
>> > > > > > > > >> > > > > > >> > > > > scope.
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> > 6.
>> > > PolarisMetricsManager
>> > > > > > > facade
>> > > > > > > > >> > > > > passthrough
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >    •   Entire
>> > default
>> > > > > method
>> > > > > > > is
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >>
>> > > > > > > > >> > callCtx.getMetaStore().writeScanReport().
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> > Zero logic,
>> passes
>> > > Level 1
>> > > > > > > > >> straight
>> > > > > > > > >> > > through
>> > > > > > > > >> > > > > > to
>> > > > > > > > >> > > > > > >> > Level
>> > > > > > > > >> > > > > > >> > > > 3.
>> > > > > > > > >> > > > > > >> > > > > > Same
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> anti-pattern
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> > as
>> > > PolarisEventManager.
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> > 7.  Iceberg
>> > community
>> > > > > > > alignment
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >    •
>>  Payload-type
>> > > > > > extension
>> > > > > > > > >> needs
>> > > > > > > > >> > > > > > discussion
>> > > > > > > > >> > > > > > >> on
>> > > > > > > > >> > > > > > >> > > > > > > dev@iceberg.
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> obelix74's
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> > Feb thread got
>> zero
>> > > > > replies.
>> > > > > > > > >> Needs a
>> > > > > > > > >> > > > > > committer
>> > > > > > > > >> > > > > > >> > > voice.
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> > Lets confirm
>> > > > > prioritization
>> > > > > > > in the
>> > > > > > > > >> > > first
>> > > > > > > > >> > > > > > >> session.
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> > -ej
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> > On Tue, Apr 21,
>> 2026
>> > > at
>> > > > > > > 3:18 PM
>> > > > > > > > >> Yufei
>> > > > > > > > >> > > Gu <
>> > > > > > > > >> > > > > > >> > > > > > > > flyrain000@gmail.com>
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > wrote:
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> Thanks everyone
>> for
>> > > > > > > continuing to
>> > > > > > > > >> > > drive
>> > > > > > > > >> > > > > this
>> > > > > > > > >> > > > > > >> > > > forward. I
>> > > > > > > > >> > > > > > >> > > > > > > agree
>> > > > > > > > >> > > > > > >> > > > > > > > > > that
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > the
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> problem is
>> getting
>> > > > > complex
>> > > > > > > enough
>> > > > > > > > >> > > that a
>> > > > > > > > >> > > > > > more
>> > > > > > > > >> > > > > > >> > > > > structured
>> > > > > > > > >> > > > > > >> > > > > > > > > > discussion
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> would
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> help.
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >>
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> +1 on setting
>> up a
>> > > > > biweekly
>> > > > > > > sync
>> > > > > > > > >> for
>> > > > > > > > >> > > the
>> > > > > > > > >> > > > > > >> metrics
>> > > > > > > > >> > > > > > >> > > > > > > > architecture.
>> > > > > > > > >> > > > > > >> > > > > > > > > > I’m
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> happy
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> to
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> join.
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >>
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> Yufei
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >>
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >>
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> On Tue, Apr 21,
>> > 2026
>> > > at
>> > > > > > > 2:34 PM
>> > > > > > > > >> EJ
>> > > > > > > > >> > > Wang <
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >>
>> > > > > > ej.wang.dev.interest@gmail.com>
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> wrote:
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >>
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> > Also, I've
>> been
>> > > looking
>> > > > > > > more
>> > > > > > > > >> > > closely at
>> > > > > > > > >> > > > > > the
>> > > > > > > > >> > > > > > >> > > > > > *persistence
>> > > > > > > > >> > > > > > >> > > > > > > > > schema
>> > > > > > > > >> > > > > > >> > > > > > > > > > > in
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> the
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> > current
>> metrics
>> > > work*,
>> > > > > > and
>> > > > > > > I
>> > > > > > > > >> think
>> > > > > > > > >> > > > > > there's a
>> > > > > > > > >> > > > > > >> > > > > structural
>> > > > > > > > >> > > > > > >> > > > > > > > > > rigidity
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> problem
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> > worth raising
>> > > before
>> > > > > the
>> > > > > > > shape
>> > > > > > > > >> > gets
>> > > > > > > > >> > > > > locked
>> > > > > > > > >> > > > > > >> in.
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> > Right now we
>> have
>> > > two
>> > > > > > > separate
>> > > > > > > > >> > > tables
>> > > > > > > > >> > > > > > >> > > > > > > (scan_metrics_report
>> > > > > > > > >> > > > > > >> > > > > > > > > and
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >
>> > > commit_metrics_report),
>> > > > > > > each
>> > > > > > > > >> with
>> > > > > > > > >> > > ~25
>> > > > > > > > >> > > > > > >> flattened
>> > > > > > > > >> > > > > > >> > > > > columns
>> > > > > > > > >> > > > > > >> > > > > > > > that
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > directly
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> > mirror the
>> > Iceberg
>> > > > > report
>> > > > > > > > >> fields.
>> > > > > > > > >> > > The
>> > > > > > > > >> > > > > SPI
>> > > > > > > > >> > > > > > >> > follows
>> > > > > > > > >> > > > > > >> > > > the
>> > > > > > > > >> > > > > > >> > > > > > > same
>> > > > > > > > >> > > > > > >> > > > > > > > > > split:
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >
>> writeScanReport
>> > and
>> > > > > > > > >> > > writeCommitReport as
>> > > > > > > > >> > > > > > >> > separate
>> > > > > > > > >> > > > > > >> > > > > > > methods,
>> > > > > > > > >> > > > > > >> > > > > > > > > with
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> per-type
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> > record
>> classes,
>> > > > > > > converters, and
>> > > > > > > > >> > > model
>> > > > > > > > >> > > > > > >> objects.
>> > > > > > > > >> > > > > > >> > > *The
>> > > > > > > > >> > > > > > >> > > > > > > > practical
>> > > > > > > > >> > > > > > >> > > > > > > > > > > cost:
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> > adding a new
>> > metric
>> > > > > type
>> > > > > > > > >> > > (operational
>> > > > > > > > >> > > > > > >> metrics,
>> > > > > > > > >> > > > > > >> > > for
>> > > > > > > > >> > > > > > >> > > > > > > example)
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > requires
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> a
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> new
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> > table, a new
>> SPI
>> > > > > method,
>> > > > > > a
>> > > > > > > new
>> > > > > > > > >> > > record
>> > > > > > > > >> > > > > > >> class, a
>> > > > > > > > >> > > > > > >> > > new
>> > > > > > > > >> > > > > > >> > > > > > model
>> > > > > > > > >> > > > > > >> > > > > > > > > > class, a
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > new
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> > converter
>> branch,
>> > > and a
>> > > > > > > schema
>> > > > > > > > >> > > > > migration*.
>> > > > > > > > >> > > > > > >> > > That's a
>> > > > > > > > >> > > > > > >> > > > > lot
>> > > > > > > > >> > > > > > >> > > > > > > of
>> > > > > > > > >> > > > > > >> > > > > > > > > > > surface
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> area
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> > for what
>> should
>> > be
>> > > "one
>> > > > > > > more
>> > > > > > > > >> kind
>> > > > > > > > >> > of
>> > > > > > > > >> > > > > > >> metric."
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> > *My bias*
>> would
>> > be
>> > > > > > toward a
>> > > > > > > > >> single
>> > > > > > > > >> > > > > metrics
>> > > > > > > > >> > > > > > >> > table
>> > > > > > > > >> > > > > > >> > > > with
>> > > > > > > > >> > > > > > >> > > > > > *a
>> > > > > > > > >> > > > > > >> > > > > > > > > typed
>> > > > > > > > >> > > > > > >> > > > > > > > > > > JSON
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> > payload*.
>> > Something
>> > > > > like:
>> > > > > > > > >> > > metric_type
>> > > > > > > > >> > > > > > >> (enum),
>> > > > > > > > >> > > > > > >> > > > > > entity_id,
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >
>> table_identifier,
>> > > > > > > snapshot_id
>> > > > > > > > >> > > > > (nullable),
>> > > > > > > > >> > > > > > >> > > > > received_ts,
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> schema_version,
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> and
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> > a payload
>> column
>> > > for
>> > > > > the
>> > > > > > > > >> > > metric-specific
>> > > > > > > > >> > > > > > >> data.
>> > > > > > > > >> > > > > > >> > > The
>> > > > > > > > >> > > > > > >> > > > > > > > > metric_type
>> > > > > > > > >> > > > > > >> > > > > > > > > > +
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> > schema_version
>> > pair
>> > > > > gives
>> > > > > > > us a
>> > > > > > > > >> > > > > > >> > forward-compatible
>> > > > > > > > >> > > > > > >> > > > > > > contract
>> > > > > > > > >> > > > > > >> > > > > > > > > for
>> > > > > > > > >> > > > > > >> > > > > > > > > > > the
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> payload
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> > shape. Adding
>> a
>> > new
>> > > > > > metric
>> > > > > > > type
>> > > > > > > > >> > > becomes
>> > > > > > > > >> > > > > an
>> > > > > > > > >> > > > > > >> enum
>> > > > > > > > >> > > > > > >> > > > value
>> > > > > > > > >> > > > > > >> > > > > > > and a
>> > > > > > > > >> > > > > > >> > > > > > > > > > > payload
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> schema,
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> > not a schema
>> > > migration.
>> > > > > > One
>> > > > > > > > >> thing
>> > > > > > > > >> > I
>> > > > > > > > >> > > > > think
>> > > > > > > > >> > > > > > we
>> > > > > > > > >> > > > > > >> > need
>> > > > > > > > >> > > > > > >> > > > to
>> > > > > > > > >> > > > > > >> > > > > be
>> > > > > > > > >> > > > > > >> > > > > > > > > > > deliberate
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> about is
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> > the partition
>> key
>> > > > > design.
>> > > > > > > If
>> > > > > > > > >> all
>> > > > > > > > >> > > metric
>> > > > > > > > >> > > > > > >> types
>> > > > > > > > >> > > > > > >> > > land
>> > > > > > > > >> > > > > > >> > > > in
>> > > > > > > > >> > > > > > >> > > > > > one
>> > > > > > > > >> > > > > > >> > > > > > > > > > table,
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > scan
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> > metrics at
>> scale
>> > > (high
>> > > > > > > > >> > concurrency,
>> > > > > > > > >> > > high
>> > > > > > > > >> > > > > > >> > > frequency
>> > > > > > > > >> > > > > > >> > > > > > across
>> > > > > > > > >> > > > > > >> > > > > > > > > many
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> tables)
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> > could easily
>> > > create hot
>> > > > > > > > >> > partitions.
>> > > > > > > > >> > > We'd
>> > > > > > > > >> > > > > > >> want
>> > > > > > > > >> > > > > > >> > the
>> > > > > > > > >> > > > > > >> > > > > > > > persistence
>> > > > > > > > >> > > > > > >> > > > > > > > > > > layer
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> to
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> be
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> > able to shard
>> by
>> > > entity
>> > > > > > or
>> > > > > > > time
>> > > > > > > > >> > > range,
>> > > > > > > > >> > > > > and
>> > > > > > > > >> > > > > > >> that
>> > > > > > > > >> > > > > > >> > > > means
>> > > > > > > > >> > > > > > >> > > > > > the
>> > > > > > > > >> > > > > > >> > > > > > > > > > logical
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> schema
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> > needs to
>> expose
>> > > enough
>> > > > > > > > >> structure
>> > > > > > > > >> > for
>> > > > > > > > >> > > > > > >> backends
>> > > > > > > > >> > > > > > >> > to
>> > > > > > > > >> > > > > > >> > > > > > > partition
>> > > > > > > > >> > > > > > >> > > > > > > > > on.
>> > > > > > > > >> > > > > > >> > > > > > > > > > I
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> don't
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> > think the
>> current
>> > > > > > flattened
>> > > > > > > > >> layout
>> > > > > > > > >> > > gives
>> > > > > > > > >> > > > > > us
>> > > > > > > > >> > > > > > >> > that.
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> > This is
>> getting
>> > > complex
>> > > > > > > enough
>> > > > > > > > >> > that
>> > > > > > > > >> > > I
>> > > > > > > > >> > > > > > don't
>> > > > > > > > >> > > > > > >> > think
>> > > > > > > > >> > > > > > >> > > > > > ad-hoc
>> > > > > > > > >> > > > > > >> > > > > > > > > PR/ML
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> threads
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> > will converge
>> > well.
>> > > > > > *Would
>> > > > > > > > >> people
>> > > > > > > > >> > be
>> > > > > > > > >> > > > > open
>> > > > > > > > >> > > > > > >> to a
>> > > > > > > > >> > > > > > >> > > > > biweekly
>> > > > > > > > >> > > > > > >> > > > > > > > sync
>> > > > > > > > >> > > > > > >> > > > > > > > > > for
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> metrics
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >
>> architecture?* I
>> > > think
>> > > > > 30
>> > > > > > > > >> minutes
>> > > > > > > > >> > > every
>> > > > > > > > >> > > > > > two
>> > > > > > > > >> > > > > > >> > weeks
>> > > > > > > > >> > > > > > >> > > > > with
>> > > > > > > > >> > > > > > >> > > > > > > > > > interested
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> parties
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> > would be
>> enough
>> > to
>> > > work
>> > > > > > > through
>> > > > > > > > >> > the
>> > > > > > > > >> > > > > > schema,
>> > > > > > > > >> > > > > > >> SPI
>> > > > > > > > >> > > > > > >> > > > > shape,
>> > > > > > > > >> > > > > > >> > > > > > > and
>> > > > > > > > >> > > > > > >> > > > > > > > > read
>> > > > > > > > >> > > > > > >> > > > > > > > > > > API
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> design
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> > together.
>> Happy
>> > to
>> > > help
>> > > > > > set
>> > > > > > > > >> that
>> > > > > > > > >> > up.
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> > -ej
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> > On Mon, Apr
>> 20,
>> > > 2026 at
>> > > > > > > > >> 2:19 PM EJ
>> > > > > > > > >> > > Wang
>> > > > > > > > >> > > > > <
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >>
>> > > > > > ej.wang.dev.interest@gmail.com
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> > wrote:
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >> Reviewed
>> #4115,
>> > > left a
>> > > > > > > > >> comment on
>> > > > > > > > >> > > the
>> > > > > > > > >> > > > > > code
>> > > > > > > > >> > > > > > >> > > > > > organization
>> > > > > > > > >> > > > > > >> > > > > > > > > side.
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >> One thing
>> stood
>> > > out:
>> > > > > the
>> > > > > > > > >> metrics
>> > > > > > > > >> > > write
>> > > > > > > > >> > > > > > path
>> > > > > > > > >> > > > > > >> > > enters
>> > > > > > > > >> > > > > > >> > > > > > > through
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>
>> > > PolarisMetricsManager
>> > > > > on
>> > > > > > > > >> > > > > > MetaStoreManager,
>> > > > > > > > >> > > > > > >> but
>> > > > > > > > >> > > > > > >> > > the
>> > > > > > > > >> > > > > > >> > > > > new
>> > > > > > > > >> > > > > > >> > > > > > > > read
>> > > > > > > > >> > > > > > >> > > > > > > > > > path
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> bypasses
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>
>> MetaStoreManager
>> > > > > > entirely
>> > > > > > > and
>> > > > > > > > >> > goes
>> > > > > > > > >> > > > > > >> straight to
>> > > > > > > > >> > > > > > >> > > > > > > > > BasePersistence
>> > > > > > > > >> > > > > > >> > > > > > > > > > > via
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>
>> > > > > > > callContext.getMetaStore().
>> > > > > > > > >> That
>> > > > > > > > >> > > means
>> > > > > > > > >> > > > > > the
>> > > > > > > > >> > > > > > >> > read
>> > > > > > > > >> > > > > > >> > > > API
>> > > > > > > > >> > > > > > >> > > > > > only
>> > > > > > > > >> > > > > > >> > > > > > > > > works
>> > > > > > > > >> > > > > > >> > > > > > > > > > > for
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> backends
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >> that
>> implement
>> > > > > > > > >> BasePersistence.
>> > > > > > > > >> > > NoSQL
>> > > > > > > > >> > > > > and
>> > > > > > > > >> > > > > > >> > remote
>> > > > > > > > >> > > > > > >> > > > > > > backends
>> > > > > > > > >> > > > > > >> > > > > > > > > > can't
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> participate.
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >> Stepping
>> back, I
>> > > think
>> > > > > > the
>> > > > > > > > >> > metrics
>> > > > > > > > >> > > > > > >> subsystem
>> > > > > > > > >> > > > > > >> > is
>> > > > > > > > >> > > > > > >> > > > > > growing
>> > > > > > > > >> > > > > > >> > > > > > > > into
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> something
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >> real (write +
>> > > read +
>> > > > > > REST
>> > > > > > > API
>> > > > > > > > >> +
>> > > > > > > > >> > > AuthZ +
>> > > > > > > > >> > > > > > >> > > > pagination)
>> > > > > > > > >> > > > > > >> > > > > > *but
>> > > > > > > > >> > > > > > >> > > > > > > > the
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> persistence
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >> side is split
>> > > across
>> > > > > two
>> > > > > > > > >> layers
>> > > > > > > > >> > in
>> > > > > > > > >> > > a
>> > > > > > > > >> > > > > way
>> > > > > > > > >> > > > > > >> > that's
>> > > > > > > > >> > > > > > >> > > > hard
>> > > > > > > > >> > > > > > >> > > > > > to
>> > > > > > > > >> > > > > > >> > > > > > > > > > > extend*. I
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> put
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >> together two
>> > > diagrams
>> > > > > to
>> > > > > > > show
>> > > > > > > > >> > what
>> > > > > > > > >> > > I
>> > > > > > > > >> > > > > mean
>> > > > > > > > >> > > > > > >> (my
>> > > > > > > > >> > > > > > >> > > best
>> > > > > > > > >> > > > > > >> > > > > > > > effort).
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >> *Current
>> state*
>> > > > > (Diagram
>> > > > > > > 1):
>> > > > > > > > >> > three
>> > > > > > > > >> > > > > > >> interfaces
>> > > > > > > > >> > > > > > >> > at
>> > > > > > > > >> > > > > > >> > > > > three
>> > > > > > > > >> > > > > > >> > > > > > > > > > different
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> levels.
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >> The
>> > engine-facing
>> > > SPI
>> > > > > > > > >> > > > > > >> (PolarisMetricsReporter)
>> > > > > > > > >> > > > > > >> > > is
>> > > > > > > > >> > > > > > >> > > > > > clean.
>> > > > > > > > >> > > > > > >> > > > > > > > But
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>
>> > > PolarisMetricsManager
>> > > > > on
>> > > > > > > > >> > > > > MetaStoreManager
>> > > > > > > > >> > > > > > >> is a
>> > > > > > > > >> > > > > > >> > > > > > > passthrough
>> > > > > > > > >> > > > > > >> > > > > > > > > to
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>
>> > > MetricsPersistence on
>> > > > > > > > >> > > BasePersistence.
>> > > > > > > > >> > > > > > The
>> > > > > > > > >> > > > > > >> > @Beta
>> > > > > > > > >> > > > > > >> > > > > > > > annotation
>> > > > > > > > >> > > > > > >> > > > > > > > > > and
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > SPI
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> javadoc
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >> are on the
>> > > > > > BasePersistence
>> > > > > > > > >> layer,
>> > > > > > > > >> > > while
>> > > > > > > > >> > > > > > the
>> > > > > > > > >> > > > > > >> > > actual
>> > > > > > > > >> > > > > > >> > > > > > > > extension
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > points
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>
>> > > > > (PolarisMetricsReporter,
>> > > > > > > > >> > > > > > >> > PolarisMetricsManager)
>> > > > > > > > >> > > > > > >> > > > > carry
>> > > > > > > > >> > > > > > >> > > > > > no
>> > > > > > > > >> > > > > > >> > > > > > > > > > > stability
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >> annotation.
>> The
>> > > write
>> > > > > > path
>> > > > > > > > >> goes
>> > > > > > > > >> > > through
>> > > > > > > > >> > > > > > the
>> > > > > > > > >> > > > > > >> > > > > > > > MetaStoreManager
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > layer,
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> the
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >> read path
>> > doesn't.
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >> *What I
>> > envision*
>> > > > > > > (Diagram 2):
>> > > > > > > > >> > two
>> > > > > > > > >> > > SPIs
>> > > > > > > > >> > > > > > at
>> > > > > > > > >> > > > > > >> two
>> > > > > > > > >> > > > > > >> > > > > levels.
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>
>> > > PolarisMetricsReporter
>> > > > > > > stays
>> > > > > > > > >> as
>> > > > > > > > >> > the
>> > > > > > > > >> > > > > > >> > > engine-facing
>> > > > > > > > >> > > > > > >> > > > > SPI.
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>
>> > > PolarisMetricsManager
>> > > > > > > becomes
>> > > > > > > > >> the
>> > > > > > > > >> > > > > > >> > backend-facing
>> > > > > > > > >> > > > > > >> > > > SPI
>> > > > > > > > >> > > > > > >> > > > > > > with
>> > > > > > > > >> > > > > > >> > > > > > > > > both
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > write
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> and
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >> read methods
>> at
>> > > the
>> > > > > > > > >> > > MetaStoreManager
>> > > > > > > > >> > > > > > level,
>> > > > > > > > >> > > > > > >> > > where
>> > > > > > > > >> > > > > > >> > > > > any
>> > > > > > > > >> > > > > > >> > > > > > > > > backend
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > (JDBC,
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> NoSQL,
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >> remote) can
>> > > implement
>> > > > > > > them.
>> > > > > > > > >> > > > > > >> MetricsPersistence
>> > > > > > > > >> > > > > > >> > > on
>> > > > > > > > >> > > > > > >> > > > > > > > > > > BasePersistence
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> goes
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >> away. Where
>> > > metrics
>> > > > > > > actually
>> > > > > > > > >> land
>> > > > > > > > >> > > is an
>> > > > > > > > >> > > > > > >> > > > > implementation
>> > > > > > > > >> > > > > > >> > > > > > > > > detail,
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > not a
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> core
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >> interface.
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >> *Minor naming
>> > > thing*:
>> > > > > > > > >> > > > > > >> PolarisMetricsReporter
>> > > > > > > > >> > > > > > >> > is
>> > > > > > > > >> > > > > > >> > > > > > broader
>> > > > > > > > >> > > > > > >> > > > > > > > than
>> > > > > > > > >> > > > > > >> > > > > > > > > > > what
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > it
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >> actually
>> > handles.
>> > > It
>> > > > > > only
>> > > > > > > > >> accepts
>> > > > > > > > >> > > > > Iceberg
>> > > > > > > > >> > > > > > >> REST
>> > > > > > > > >> > > > > > >> > > > > Catalog
>> > > > > > > > >> > > > > > >> > > > > > > > > metrics
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> (ScanReport,
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >> CommitReport
>> via
>> > > > > > > > >> MetricsReport).
>> > > > > > > > >> > > > > Generic
>> > > > > > > > >> > > > > > >> table
>> > > > > > > > >> > > > > > >> > > > > metrics
>> > > > > > > > >> > > > > > >> > > > > > > or
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> operational
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >> metrics
>> aren't
>> > in
>> > > > > scope.
>> > > > > > > Not
>> > > > > > > > >> > > blocking,
>> > > > > > > > >> > > > > > but
>> > > > > > > > >> > > > > > >> > worth
>> > > > > > > > >> > > > > > >> > > > > > noting
>> > > > > > > > >> > > > > > >> > > > > > > if
>> > > > > > > > >> > > > > > >> > > > > > > > > the
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> metrics
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >> surface
>> expands.
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >> *Rough
>> sketch of
>> > > how
>> > > > > to
>> > > > > > > get
>> > > > > > > > >> > there*:
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>  1.  Add read
>> > > methods
>> > > > > to
>> > > > > > > > >> > > > > > >> PolarisMetricsManager
>> > > > > > > > >> > > > > > >> > > > > > > > > > (listScanReports,
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>
>> > listCommitReports)
>> > > > > with
>> > > > > > > > >> default
>> > > > > > > > >> > > no-op,
>> > > > > > > > >> > > > > > >> same as
>> > > > > > > > >> > > > > > >> > > the
>> > > > > > > > >> > > > > > >> > > > > > > > existing
>> > > > > > > > >> > > > > > >> > > > > > > > > > > write
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> methods.
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >> (Probably
>> make
>> > > > > > > > >> > > PolarisMetricsManager
>> > > > > > > > >> > > > > more
>> > > > > > > > >> > > > > > >> > > explicit
>> > > > > > > > >> > > > > > >> > > > > on
>> > > > > > > > >> > > > > > >> > > > > > > > being
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > Iceberg
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >> specific like
>> > > package
>> > > > > > > name or
>> > > > > > > > >> > class
>> > > > > > > > >> > > > > name
>> > > > > > > > >> > > > > > >> etc.)
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>  2.  Wire
>> > > > > > > > >> MetricsReportsService
>> > > > > > > > >> > > through
>> > > > > > > > >> > > > > > >> > > > > > MetaStoreManager
>> > > > > > > > >> > > > > > >> > > > > > > > > > instead
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > of
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>
>> > > > > > > callContext.getMetaStore().
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>  3.  Extract
>> > > metrics
>> > > > > > > > >> persistence
>> > > > > > > > >> > > from
>> > > > > > > > >> > > > > > >> > > > > > > > > JdbcBasePersistenceImpl
>> > > > > > > > >> > > > > > >> > > > > > > > > > > into
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> its
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >> own class.
>> That
>> > > file
>> > > > > > > carries
>> > > > > > > > >> ~7
>> > > > > > > > >> > > > > > >> > > responsibilities,
>> > > > > > > > >> > > > > > >> > > > > > > metrics
>> > > > > > > > >> > > > > > >> > > > > > > > > > being
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > one
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> of
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> them.
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>  4.  Remove
>> > > > > > > MetricsPersistence
>> > > > > > > > >> > from
>> > > > > > > > >> > > > > > >> > > > BasePersistence.
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >> *None of this
>> > > needs to
>> > > > > > > happen
>> > > > > > > > >> in
>> > > > > > > > >> > > #4115.
>> > > > > > > > >> > > > > > >> But if
>> > > > > > > > >> > > > > > >> > > the
>> > > > > > > > >> > > > > > >> > > > > > > > direction
>> > > > > > > > >> > > > > > >> > > > > > > > > > > makes
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> sense,
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >> it would be
>> good
>> > > to
>> > > > > > align
>> > > > > > > > >> before
>> > > > > > > > >> > > the
>> > > > > > > > >> > > > > > >> metrics
>> > > > > > > > >> > > > > > >> > > > surface
>> > > > > > > > >> > > > > > >> > > > > > > grows
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > further.
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> Curious
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >> what others
>> > > think.*
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >> *My mental
>> model
>> > > > > note*:
>> > > > > > > Level
>> > > > > > > > >> 1
>> > > > > > > > >> > > > > > >> > > MetaStoreManager;
>> > > > > > > > >> > > > > > >> > > > > > level
>> > > > > > > > >> > > > > > >> > > > > > > 2
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> transactional
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >> persistence;
>> > > level 3
>> > > > > > base
>> > > > > > > > >> > > persistence
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >> Diagram 1
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >> <
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >>
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >>
>> > > > > > > > >> > > > > > >> > > > > > > > > > > >
>> > > > > > > > >> > > > > > >> > > > > > > > > > >
>> > > > > > > > >> > > > > > >> > > > > > > > > >
>> > > > > > > > >> > > > > > >> > > > > > > > >
>> > > > > > > > >> > > > > > >> > > > > > > >
>> > > > > > > > >> > > > > > >> > > > > > >
>> > > > > > > > >> > > > > > >> > > > > >
>> > > > > > > > >> > > > > > >> > > > >
>> > > > > > > > >> > > > > > >> > > >
>> > > > > > > > >> > > > > > >> > >
>> > > > > > > > >> > > > > > >> >
>> > > > > > > > >> > > > > > >>
>> > > > > > > > >> > > > > >
>> > > > > > > > >> > > > >
>> > > > > > > > >> > >
>> > > > > > > > >> >
>> > > > > > > > >>
>> > > > > > >
>> > > > > >
>> > > > >
>> > >
>> >
>> https://urldefense.com/v3/__https://www.plantuml.com/plantuml/uml/bLHDR-Cs4BthLmpIYupw0zbkKQ1r3M-S7Bp8xhhM7WCOb3IM65EaGD9EX2RzxHrHb4CxRelwa4YSDu_lpOVcnZ9jzvM8BBS2uGjQpJC3dtHMSekPtMk44IpsMgEqa5XcCOhCZikQQLP1pR8TAp2n3ILhmZDP20m0fcIvUkAoW2qJXd9z1bpToO9BX3WXu0ucy5rpgGPNm0nW5_epUWtm2Ue3pn3kMOFQmKntGZW0BYtgBSi8k5A2QMwybJNMIbFiGSR9QZc4nUqIvikStF0jHprua5C-amge42aNt3R0f5JaaoivdV2Pkqbx4hee4ymOkBh5BTiB-_uIeGeo8zL8rPsPl4DktdEiK1jkB1NdZCRbrSTecDe_mlHbF0wvBmCkaOH5_S8a_TTTKI6-nmCAkEw4LpxsZ-LbYLKQFKMNOgf_wuM7_bV9gOer5SYMMksBSWXFcbi49KNZXNLicwfe3TETC7gPdPqI7uBcHMb1RSzYq34c6PDUM9mn8HRsUTZEiDBve3NjVZumBj0U7SS37mGO7vcwtiK-_pU7U7L_f-digo9YbhSwIfMRwIITKGXbxdIUTCGF1SeCJxloKsU-3k9ddRbX1eDq1q_fx1JbBGT0glVyXimDuP4TQ5qpCAmnGEj2s_6n5mtn1z-97-63itFQZLPO1Ev2tu_WF7Ju-VPc0Skg5bYXxBhkY1xpD7EM_7fyflSpIsqMgVth5xhVr4eQxWQ8enaSAJQSG16yFSDuJ798rrcXr_3n-lfdk7icQjEBmFujL7AodiP_Y4Z7-YxvtZNs4zMgpNTl6tF8sglyPsmqchrjvQ-m-aP94r-TwCA2Ka8upPJZwtvSpoYCXkYMZU2NXvRMBfq9P3i3Le4VAZUAlUZ_oPKsxPgY0Q_BSKLkyr9bhQhQrJjo_x3TPlIB0DPjnMfcIoYP0QaYw1a0fTKDr8fB6ntNuvmoL1ZGkXa69Njh43zf9GiGxHQrA_jDYWRSzF5--WmTVrN97_Sm8LbLUy_lGBmLanJjFkDlGkRqjA_4tm00__;!!Iz9xO38YGHZK!4rnlVIqBQQQ6xpGe1PMV58edOS_cGrKoqdfhw4qSp_JQCavvEjaxXk0OwGPnxyEcO60AzKQK_XDtp7H3DdkFXcrXuTBISHg$
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >> :
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >> [image:
>> > image.png]
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >> Diagram 2
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >> <
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >>
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >>
>> > > > > > > > >> > > > > > >> > > > > > > > > > > >
>> > > > > > > > >> > > > > > >> > > > > > > > > > >
>> > > > > > > > >> > > > > > >> > > > > > > > > >
>> > > > > > > > >> > > > > > >> > > > > > > > >
>> > > > > > > > >> > > > > > >> > > > > > > >
>> > > > > > > > >> > > > > > >> > > > > > >
>> > > > > > > > >> > > > > > >> > > > > >
>> > > > > > > > >> > > > > > >> > > > >
>> > > > > > > > >> > > > > > >> > > >
>> > > > > > > > >> > > > > > >> > >
>> > > > > > > > >> > > > > > >> >
>> > > > > > > > >> > > > > > >>
>> > > > > > > > >> > > > > >
>> > > > > > > > >> > > > >
>> > > > > > > > >> > >
>> > > > > > > > >> >
>> > > > > > > > >>
>> > > > > > >
>> > > > > >
>> > > > >
>> > >
>> >
>> https://urldefense.com/v3/__https://www.plantuml.com/plantuml/uml/VLLDR-8m4BtdLupO2sWBLVU8AaGB7AXAbssGzb896SS9RXqxjKqBwkv_tt7iV43fdaZYDpFlpRmnOsE9jhjSH9PRmM31hERKm8scMsuPjJlDe0yheZDc8RR4iYWoBrmMH9CS2a9VICPYUy1OZN0YCy5Q0BCbYNhdCeEK28En8G8wCvbnoQ0R8_05Bc6bkLIz3X03p1zzH7zR-9ZfDquPt9C3qoNCX2yV4G2NbkcKu5jdgGJHt0GbZwnG6i-UP3TUpk5gM6Ldqke350eZUqzoCft3U9xWHvxoa5-7K4nF1J46EbEMafsmdrCBbQ44gVggy18IZrn_ph5asd1ZiIKdQSgueZvjXrQFSFrdC3YN-nXmBacxbGiYyLVxLaBtdhqn0LSzdBDhqQtQoOJeGyad3z0lUqnYgpGB6Ns8oVyta00Dy_WnX0tIOZ8v6SYxHll1TrH6aejAik-mh-AphVFCwSUQqFypElag5QRGFDjQKEd96K1P8QP41c9TzA_IIQyvdAWyv_RSiS3skb0_EzDDkK2v5xWF6MiGFlvhpFLcD2Dq2pml14gaF67eQkmd8gulDoC4kSOu6KVpkvlUJg1RTbWISU40RdBUUS_9XfRZ2dwxm_SW8LYFISgm_MnlDQ6M9P1gbKEc4X-2pH_FvJCkCqm9pbVjD6LrwdLeOrDWfOaqc8Wh9BE85oNKxkNQ6o4yGRy_Eae0G_G8tZv81d3bHDB23WOdisohVr3nh_j6lbSjbNaLRTc8UgtPbAU1J_tygOfZX9DWEJeHDvYx-qmSi5FgNLPZwHrHcUsncGQ5-skhUclpE5fo4ounpFauYrUbkU6ccfnxMvitwag4IyerhTxj8In_Oj1bDO4pQru674loYrGlULHLEGCjwJJ8gDoVZR8MxO4BT3IzRvIcAQKezC6xpziGnTyImrfEGyJI_OcKfgtxIvnTqFEMS17L9Z-jsARN5FmTheP7HtSdtOMT0B4GY2FYHXxgQmMtj2bRqiLFGapiVe1_QVKDrkqXcm83aFEXnMYCZ-xlyHy__;!!Iz9xO38YGHZK!4rnlVIqBQQQ6xpGe1PMV58edOS_cGrKoqdfhw4qSp_JQCavvEjaxXk0OwGPnxyEcO60AzKQK_XDtp7H3DdkFXcrXSpmqcNk$
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >> :
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >> [image:
>> > image.png]
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>  -ej
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >> On Wed, Apr
>> 15,
>> > > 2026
>> > > > > at
>> > > > > > > > >> 8:22 AM
>> > > > > > > > >> > > Dmitri
>> > > > > > > > >> > > > > > >> > > > Bourlatchkov
>> > > > > > > > >> > > > > > >> > > > > <
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> dimas@apache.org>
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >> wrote:
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> Hi All,
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>>
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> Heads up:
>> The
>> > > current
>> > > > > > > state
>> > > > > > > > >> of
>> > > > > > > > >> > PR
>> > > > > > > > >> > > > > [4115]
>> > > > > > > > >> > > > > > >> > looks
>> > > > > > > > >> > > > > > >> > > > > pretty
>> > > > > > > > >> > > > > > >> > > > > > > > solid
>> > > > > > > > >> > > > > > >> > > > > > > > > > to
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > me.
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> I
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> believe
>> this PR
>> > > is
>> > > > > > > > >> approaching a
>> > > > > > > > >> > > > > > mergeable
>> > > > > > > > >> > > > > > >> > > > > condition.
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>>
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> Please post
>> > your
>> > > > > > reviews
>> > > > > > > if
>> > > > > > > > >> you
>> > > > > > > > >> > > have
>> > > > > > > > >> > > > > any
>> > > > > > > > >> > > > > > >> > > > comments.
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>>
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> [4115]
>> > > > > > > > >> > > > > > >> > > >
>> > > > > > > > >> >
>> > > > > > > > >>
>> > > > > > >
>> > > > > >
>> > > > >
>> > >
>> >
>> https://urldefense.com/v3/__https://github.com/apache/polaris/pull/4115__;!!Iz9xO38YGHZK!4rnlVIqBQQQ6xpGe1PMV58edOS_cGrKoqdfhw4qSp_JQCavvEjaxXk0OwGPnxyEcO60AzKQK_XDtp7H3DdkFXcrXtFRscKs$
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>>
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> Thanks,
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> Dmitri.
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>>
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> On Tue, Mar
>> 3,
>> > > 2026
>> > > > > at
>> > > > > > > > >> 3:29 PM
>> > > > > > > > >> > > Anand
>> > > > > > > > >> > > > > > Kumar
>> > > > > > > > >> > > > > > >> > > > Sankaran
>> > > > > > > > >> > > > > > >> > > > > > via
>> > > > > > > > >> > > > > > >> > > > > > > > > dev <
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>>
>> > > > > dev@polaris.apache.org
>> > > > > > >
>> > > > > > > > >> wrote:
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>>
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > Hi Yufei
>> and
>> > > > > Dmitri,
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> >
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > Here is a
>> > > proposal
>> > > > > > for
>> > > > > > > the
>> > > > > > > > >> > REST
>> > > > > > > > >> > > > > > >> endpoints
>> > > > > > > > >> > > > > > >> > for
>> > > > > > > > >> > > > > > >> > > > > > metrics
>> > > > > > > > >> > > > > > >> > > > > > > > and
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > events.
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> >
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> >
>> > > > > > > > >> > > > > > >> > > > > >
>> > > > > > > > >> >
>> > > > > > > > >>
>> > > > > > >
>> > > > > >
>> > > > >
>> > >
>> >
>> https://urldefense.com/v3/__https://github.com/apache/polaris/pull/3924/changes__;!!Iz9xO38YGHZK!4rnlVIqBQQQ6xpGe1PMV58edOS_cGrKoqdfhw4qSp_JQCavvEjaxXk0OwGPnxyEcO60AzKQK_XDtp7H3DdkFXcrXcjoPhSs$
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> >
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > I did not
>> see
>> > > any
>> > > > > > > > >> precursors
>> > > > > > > > >> > for
>> > > > > > > > >> > > > > > >> raising a
>> > > > > > > > >> > > > > > >> > PR
>> > > > > > > > >> > > > > > >> > > > for
>> > > > > > > > >> > > > > > >> > > > > > > > > > proposals,
>> > > > > > > > >> > > > > > >> > > > > > > > > > > so
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> trying
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > this.
>> Please
>> > > let
>> > > > > me
>> > > > > > > know
>> > > > > > > > >> what
>> > > > > > > > >> > > you
>> > > > > > > > >> > > > > > >> think.
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> >
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > -
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > Anand
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> >
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > From:
>> Anand
>> > > Kumar
>> > > > > > > Sankaran
>> > > > > > > > >> <
>> > > > > > > > >> > > > > > >> > > > > > > anand.sankaran@workday.com
>> > > > > > > > >> > > > > > >> > > > > > > > >
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > Date:
>> Monday,
>> > > March
>> > > > > > 2,
>> > > > > > > > >> 2026 at
>> > > > > > > > >> > > > > > 10:25 AM
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > To:
>> > > > > > > dev@polaris.apache.org
>> > > > > > > > >> <
>> > > > > > > > >> > > > > > >> > > > > dev@polaris.apache.org
>> > > > > > > > >> > > > > > >> > > > > > >
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > Subject:
>> Re:
>> > > > > Polaris
>> > > > > > > > >> Telemetry
>> > > > > > > > >> > > and
>> > > > > > > > >> > > > > > Audit
>> > > > > > > > >> > > > > > >> > > Trail
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> >
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > About the
>> > REST
>> > > API,
>> > > > > > > based
>> > > > > > > > >> on
>> > > > > > > > >> > my
>> > > > > > > > >> > > use
>> > > > > > > > >> > > > > > >> cases:
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> >
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> >
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> >   1.
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > I want to
>> be
>> > > able
>> > > > > to
>> > > > > > > query
>> > > > > > > > >> > > commit
>> > > > > > > > >> > > > > > >> metrics
>> > > > > > > > >> > > > > > >> > to
>> > > > > > > > >> > > > > > >> > > > > track
>> > > > > > > > >> > > > > > >> > > > > > > > files
>> > > > > > > > >> > > > > > >> > > > > > > > > > > added
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > /
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> removed
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > per
>> commit,
>> > > along
>> > > > > > with
>> > > > > > > > >> record
>> > > > > > > > >> > > > > counts.
>> > > > > > > > >> > > > > > >> The
>> > > > > > > > >> > > > > > >> > > > > ingestion
>> > > > > > > > >> > > > > > >> > > > > > > > > > pipeline
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > that
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> writes
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > this data
>> is
>> > > owned
>> > > > > by
>> > > > > > > us
>> > > > > > > > >> and
>> > > > > > > > >> > we
>> > > > > > > > >> > > are
>> > > > > > > > >> > > > > > >> > > guaranteed
>> > > > > > > > >> > > > > > >> > > > to
>> > > > > > > > >> > > > > > >> > > > > > > write
>> > > > > > > > >> > > > > > >> > > > > > > > > > this
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> information
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > for each
>> > write.
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> >   2.
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > I want to
>> be
>> > > able
>> > > > > to
>> > > > > > > query
>> > > > > > > > >> > scan
>> > > > > > > > >> > > > > > metrics
>> > > > > > > > >> > > > > > >> for
>> > > > > > > > >> > > > > > >> > > > > read. I
>> > > > > > > > >> > > > > > >> > > > > > > > > > > understand
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> clients
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> do
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > not
>> fulfill
>> > > this
>> > > > > > > > >> requirement.
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> >   3.
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > I want to
>> be
>> > > able
>> > > > > to
>> > > > > > > query
>> > > > > > > > >> the
>> > > > > > > > >> > > > > events
>> > > > > > > > >> > > > > > >> table
>> > > > > > > > >> > > > > > >> > > > > (events
>> > > > > > > > >> > > > > > >> > > > > > > are
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> persisted) -
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> this
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > may
>> supersede
>> > > #2, I
>> > > > > > am
>> > > > > > > not
>> > > > > > > > >> > sure
>> > > > > > > > >> > > yet.
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> >
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > All this
>> > > > > information
>> > > > > > > is in
>> > > > > > > > >> the
>> > > > > > > > >> > > JDBC
>> > > > > > > > >> > > > > > >> based
>> > > > > > > > >> > > > > > >> > > > > > persistence
>> > > > > > > > >> > > > > > >> > > > > > > > > model
>> > > > > > > > >> > > > > > >> > > > > > > > > > > and
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> is
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > persisted
>> in
>> > > the
>> > > > > > > > >> metastore. I
>> > > > > > > > >> > > > > > currently
>> > > > > > > > >> > > > > > >> > don’t
>> > > > > > > > >> > > > > > >> > > > > have
>> > > > > > > > >> > > > > > >> > > > > > a
>> > > > > > > > >> > > > > > >> > > > > > > > need
>> > > > > > > > >> > > > > > >> > > > > > > > > > to
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> query
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> >
>> prometheus or
>> > > open
>> > > > > > > > >> telemetry.
>> > > > > > > > >> > I
>> > > > > > > > >> > > do
>> > > > > > > > >> > > > > > >> publish
>> > > > > > > > >> > > > > > >> > > some
>> > > > > > > > >> > > > > > >> > > > > > > events
>> > > > > > > > >> > > > > > >> > > > > > > > to
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> Prometheus
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> and
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > they are
>> > > forwarded
>> > > > > to
>> > > > > > > our
>> > > > > > > > >> > > dashboards
>> > > > > > > > >> > > > > > >> > > elsewhere.
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> >
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > About the
>> CLI
>> > > > > > > utilities, I
>> > > > > > > > >> > > meant the
>> > > > > > > > >> > > > > > >> admin
>> > > > > > > > >> > > > > > >> > > user
>> > > > > > > > >> > > > > > >> > > > > > > > > utilities.
>> > > > > > > > >> > > > > > >> > > > > > > > > > In
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> one of
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> the
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > earliest
>> > > drafts of
>> > > > > my
>> > > > > > > > >> > proposal,
>> > > > > > > > >> > > > > > Prashant
>> > > > > > > > >> > > > > > >> > > > > mentioned
>> > > > > > > > >> > > > > > >> > > > > > > that
>> > > > > > > > >> > > > > > >> > > > > > > > > the
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> metrics
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> tables
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > can grow
>> > > > > indefinitely
>> > > > > > > and
>> > > > > > > > >> > that a
>> > > > > > > > >> > > > > > similar
>> > > > > > > > >> > > > > > >> > > > problem
>> > > > > > > > >> > > > > > >> > > > > > > exists
>> > > > > > > > >> > > > > > >> > > > > > > > > > with
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > the
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> events
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > table as
>> > well.
>> > > We
>> > > > > > > discussed
>> > > > > > > > >> > that
>> > > > > > > > >> > > > > > >> cleaning
>> > > > > > > > >> > > > > > >> > up
>> > > > > > > > >> > > > > > >> > > of
>> > > > > > > > >> > > > > > >> > > > > old
>> > > > > > > > >> > > > > > >> > > > > > > > > records
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > from
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> both
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > metrics
>> > tables
>> > > and
>> > > > > > > events
>> > > > > > > > >> > > tables can
>> > > > > > > > >> > > > > > be
>> > > > > > > > >> > > > > > >> > done
>> > > > > > > > >> > > > > > >> > > > via
>> > > > > > > > >> > > > > > >> > > > > a
>> > > > > > > > >> > > > > > >> > > > > > > CLI
>> > > > > > > > >> > > > > > >> > > > > > > > > > > utility.
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> >
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > I see that
>> > > Yufei
>> > > > > has
>> > > > > > > > >> covered
>> > > > > > > > >> > the
>> > > > > > > > >> > > > > > >> discussion
>> > > > > > > > >> > > > > > >> > > > about
>> > > > > > > > >> > > > > > >> > > > > > > > > > > datasources.
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> >
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > -
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > Anand
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> >
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> >
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> >
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > From:
>> Yufei
>> > Gu
>> > > <
>> > > > > > > > >> > > > > flyrain000@gmail.com>
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > Date:
>> Friday,
>> > > > > > February
>> > > > > > > 27,
>> > > > > > > > >> > 2026
>> > > > > > > > >> > > at
>> > > > > > > > >> > > > > > >> 9:54 PM
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > To:
>> > > > > > > dev@polaris.apache.org
>> > > > > > > > >> <
>> > > > > > > > >> > > > > > >> > > > > dev@polaris.apache.org
>> > > > > > > > >> > > > > > >> > > > > > >
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > Subject:
>> Re:
>> > > > > Polaris
>> > > > > > > > >> Telemetry
>> > > > > > > > >> > > and
>> > > > > > > > >> > > > > > Audit
>> > > > > > > > >> > > > > > >> > > Trail
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> >
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > This
>> Message
>> > Is
>> > > > > From
>> > > > > > an
>> > > > > > > > >> > External
>> > > > > > > > >> > > > > > Sender
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > This
>> message
>> > > came
>> > > > > > from
>> > > > > > > > >> outside
>> > > > > > > > >> > > your
>> > > > > > > > >> > > > > > >> > > > organization.
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > Report
>> > > Suspicious<
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> >
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>>
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >>
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >>
>> > > > > > > > >> > > > > > >> > > > > > > > > > > >
>> > > > > > > > >> > > > > > >> > > > > > > > > > >
>> > > > > > > > >> > > > > > >> > > > > > > > > >
>> > > > > > > > >> > > > > > >> > > > > > > > >
>> > > > > > > > >> > > > > > >> > > > > > > >
>> > > > > > > > >> > > > > > >> > > > > > >
>> > > > > > > > >> > > > > > >> > > > > >
>> > > > > > > > >> > > > > > >> > > > >
>> > > > > > > > >> > > > > > >> > > >
>> > > > > > > > >> > > > > > >> > >
>> > > > > > > > >> > > > > > >> >
>> > > > > > > > >> > > > > > >>
>> > > > > > > > >> > > > > >
>> > > > > > > > >> > > > >
>> > > > > > > > >> > >
>> > > > > > > > >> >
>> > > > > > > > >>
>> > > > > > >
>> > > > > >
>> > > > >
>> > >
>> >
>> https://us-phishalarm-ewt.proofpoint.com/EWT/v1/Iz9xO38YGHZK!YhNDZABkHi1B699ote2uMwpOZw8i0QMCGO2Szc-HshuABGhGvwPJcymE6G2oUUxtS8xDkSrtGTPm_I3QnVDHoLMk50m9v8z_nZKTkd-bnVUbreF1u0WnfV_X5eYevZl_$
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > >
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> >
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> >
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > As I
>> > mentioned
>> > > in
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> >
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>>
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >>
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >>
>> > > > > > > > >> > > > > > >> > > > > > > > > > > >
>> > > > > > > > >> > > > > > >> > > > > > > > > > >
>> > > > > > > > >> > > > > > >> > > > > > > > > >
>> > > > > > > > >> > > > > > >> > > > > > > > >
>> > > > > > > > >> > > > > > >> > > > > > > >
>> > > > > > > > >> > > > > > >> > > > > > >
>> > > > > > > > >> > > > > > >> > > > > >
>> > > > > > > > >> > > > > > >> > > > >
>> > > > > > > > >> > > > > > >> > > >
>> > > > > > > > >> > > > > > >> > >
>> > > > > > > > >> > > > > > >> >
>> > > > > > > > >> > > > > > >>
>> > > > > > > > >> > > > > >
>> > > > > > > > >> > > > >
>> > > > > > > > >> > >
>> > > > > > > > >> >
>> > > > > > > > >>
>> > > > > > >
>> > > > > >
>> > > > >
>> > >
>> >
>> https://urldefense.com/v3/__https://github.com/apache/polaris/issues/3890__;!!Iz9xO38YGHZK!5EuyFFkk3vhRWVIRvQAWBSQfpJkTMA9HxugzDwXmN0LPPqhEFxYkFRGVhtb8AqUwXtDh2OplcMnbMDHKOxrvDU0$
>> > > > > > > > >> > >> > > >> > > > > > > > > > > > >> >> >>> ,
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > supporting
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > multiple
>> data
>> > > > > sources
>> > > > > > > is
>> > > > > > > > >> not a
>> > > > > > > > >> > > > > trivial
>> > > > > > > > >> > > > > > >> > > change.
>> > > > > > > > >> > > > > > >> > > > I
>> > > > > > > > >> > > > > > >> > > > > > > would
>> > > > > > > > >> > > > > > >> > > > > > > > > > > strongly
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> recommend
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > starting
>> > with a
>> > > > > > design
>> > > > > > > > >> > document
>> > > > > > > > >> > > to
>> > > > > > > > >> > > > > > >> > carefully
>> > > > > > > > >> > > > > > >> > > > > > evaluate
>> > > > > > > > >> > > > > > >> > > > > > > > the
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> architectural
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> >
>> implications
>> > > and
>> > > > > long
>> > > > > > > term
>> > > > > > > > >> > > impact.
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> >
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > A REST
>> > > endpoint to
>> > > > > > > query
>> > > > > > > > >> > metrics
>> > > > > > > > >> > > > > seems
>> > > > > > > > >> > > > > > >> > > > reasonable
>> > > > > > > > >> > > > > > >> > > > > > > given
>> > > > > > > > >> > > > > > >> > > > > > > > > the
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> current
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> JDBC
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > based
>> > > persistence
>> > > > > > > model.
>> > > > > > > > >> That
>> > > > > > > > >> > > said,
>> > > > > > > > >> > > > > we
>> > > > > > > > >> > > > > > >> may
>> > > > > > > > >> > > > > > >> > > also
>> > > > > > > > >> > > > > > >> > > > > > > > consider
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> alternative
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > storage
>> > > models. For
>> > > > > > > > >> example,
>> > > > > > > > >> > if
>> > > > > > > > >> > > we
>> > > > > > > > >> > > > > > later
>> > > > > > > > >> > > > > > >> > > adopt
>> > > > > > > > >> > > > > > >> > > > a
>> > > > > > > > >> > > > > > >> > > > > > time
>> > > > > > > > >> > > > > > >> > > > > > > > > > series
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> system
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> such as
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> >
>> Prometheus to
>> > > store
>> > > > > > > > >> metrics,
>> > > > > > > > >> > the
>> > > > > > > > >> > > > > query
>> > > > > > > > >> > > > > > >> > model
>> > > > > > > > >> > > > > > >> > > > and
>> > > > > > > > >> > > > > > >> > > > > > > access
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > patterns
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> would
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> be
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> >
>> fundamentally
>> > > > > > > different.
>> > > > > > > > >> > > Designing
>> > > > > > > > >> > > > > the
>> > > > > > > > >> > > > > > >> REST
>> > > > > > > > >> > > > > > >> > > API
>> > > > > > > > >> > > > > > >> > > > > > > without
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> considering
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> these
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > potential
>> > > > > evolutions
>> > > > > > > may
>> > > > > > > > >> limit
>> > > > > > > > >> > > > > > >> flexibility.
>> > > > > > > > >> > > > > > >> > > I'd
>> > > > > > > > >> > > > > > >> > > > > > > suggest
>> > > > > > > > >> > > > > > >> > > > > > > > > to
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > start
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> with
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> the
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > use case.
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> >
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > Yufei
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> >
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> >
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > On Fri,
>> Feb
>> > 27,
>> > > > > 2026
>> > > > > > at
>> > > > > > > > >> > 3:42 PM
>> > > > > > > > >> > > > > Dmitri
>> > > > > > > > >> > > > > > >> > > > > > Bourlatchkov <
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >>
>> dimas@apache.org>
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > wrote:
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> >
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > > Hi
>> Anand,
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > >
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > > Sharing
>> my
>> > > > > view...
>> > > > > > > > >> subject
>> > > > > > > > >> > to
>> > > > > > > > >> > > > > > >> discussion:
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > >
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > > 1.
>> Adding
>> > > non-IRC
>> > > > > > > REST
>> > > > > > > > >> API
>> > > > > > > > >> > to
>> > > > > > > > >> > > > > > Polaris
>> > > > > > > > >> > > > > > >> is
>> > > > > > > > >> > > > > > >> > > > > > perfectly
>> > > > > > > > >> > > > > > >> > > > > > > > > fine.
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > >
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > > Figuring
>> > out
>> > > > > > specific
>> > > > > > > > >> > endpoint
>> > > > > > > > >> > > > > URIs
>> > > > > > > > >> > > > > > >> and
>> > > > > > > > >> > > > > > >> > > > > payloads
>> > > > > > > > >> > > > > > >> > > > > > > > might
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > require
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> a
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> few
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > >
>> roundtrips,
>> > > so
>> > > > > > > opening a
>> > > > > > > > >> > > separate
>> > > > > > > > >> > > > > > >> thread
>> > > > > > > > >> > > > > > >> > > for
>> > > > > > > > >> > > > > > >> > > > > that
>> > > > > > > > >> > > > > > >> > > > > > > > might
>> > > > > > > > >> > > > > > >> > > > > > > > > > be
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> best.
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > >
>> > Contributors
>> > > > > > commonly
>> > > > > > > > >> create
>> > > > > > > > >> > > > > Google
>> > > > > > > > >> > > > > > >> Docs
>> > > > > > > > >> > > > > > >> > > for
>> > > > > > > > >> > > > > > >> > > > > new
>> > > > > > > > >> > > > > > >> > > > > > > API
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > proposals
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> too
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> (they
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > > fairly
>> easy
>> > > to
>> > > > > > > update as
>> > > > > > > > >> the
>> > > > > > > > >> > > email
>> > > > > > > > >> > > > > > >> > > discussion
>> > > > > > > > >> > > > > > >> > > > > > > > > > progresses).
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > >
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > > There
>> was a
>> > > > > > > suggestion to
>> > > > > > > > >> > try
>> > > > > > > > >> > > > > > Markdown
>> > > > > > > > >> > > > > > >> > > (with
>> > > > > > > > >> > > > > > >> > > > > PRs)
>> > > > > > > > >> > > > > > >> > > > > > > for
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > proposals
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> [1]
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> ...
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > > feel
>> free
>> > to
>> > > give
>> > > > > > it
>> > > > > > > a
>> > > > > > > > >> try
>> > > > > > > > >> > if
>> > > > > > > > >> > > you
>> > > > > > > > >> > > > > > are
>> > > > > > > > >> > > > > > >> > > > > comfortable
>> > > > > > > > >> > > > > > >> > > > > > > > with
>> > > > > > > > >> > > > > > >> > > > > > > > > > > that.
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > >
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > > 2. Could
>> > you
>> > > > > > clarify
>> > > > > > > > >> whether
>> > > > > > > > >> > > you
>> > > > > > > > >> > > > > > mean
>> > > > > > > > >> > > > > > >> end
>> > > > > > > > >> > > > > > >> > > > user
>> > > > > > > > >> > > > > > >> > > > > > > > > utilities
>> > > > > > > > >> > > > > > >> > > > > > > > > > or
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> admin
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> user
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > >
>> utilities?
>> > > In the
>> > > > > > > latter
>> > > > > > > > >> > case
>> > > > > > > > >> > > > > those
>> > > > > > > > >> > > > > > >> might
>> > > > > > > > >> > > > > > >> > > be
>> > > > > > > > >> > > > > > >> > > > > more
>> > > > > > > > >> > > > > > >> > > > > > > > > > suitable
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > for
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> the
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> Admin
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > > CLI
>> (java)
>> > > not
>> > > > > the
>> > > > > > > Python
>> > > > > > > > >> > CLI,
>> > > > > > > > >> > > > > IMHO.
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > >
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > > Why
>> would
>> > > these
>> > > > > > > > >> utilities be
>> > > > > > > > >> > > > > common
>> > > > > > > > >> > > > > > >> with
>> > > > > > > > >> > > > > > >> > > > > events?
>> > > > > > > > >> > > > > > >> > > > > > > > IMHO,
>> > > > > > > > >> > > > > > >> > > > > > > > > > > event
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> use
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> cases
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > are
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > > distinct
>> > from
>> > > > > > > scan/commit
>> > > > > > > > >> > > metrics.
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > >
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > > 3. I'd
>> > prefer
>> > > > > > > separating
>> > > > > > > > >> > > metrics
>> > > > > > > > >> > > > > > >> > > persistence
>> > > > > > > > >> > > > > > >> > > > > from
>> > > > > > > > >> > > > > > >> > > > > > > > > > MetaStore
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> persistence
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > at
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > > the code
>> > > level,
>> > > > > so
>> > > > > > > that
>> > > > > > > > >> they
>> > > > > > > > >> > > could
>> > > > > > > > >> > > > > > be
>> > > > > > > > >> > > > > > >> > mixed
>> > > > > > > > >> > > > > > >> > > > and
>> > > > > > > > >> > > > > > >> > > > > > > > matched
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>>
>> independently.
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > The
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > > separate
>> > > > > datasource
>> > > > > > > > >> question
>> > > > > > > > >> > > will
>> > > > > > > > >> > > > > > >> become
>> > > > > > > > >> > > > > > >> > a
>> > > > > > > > >> > > > > > >> > > > > > > non-issue
>> > > > > > > > >> > > > > > >> > > > > > > > > with
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > that
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> approach,
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > I
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > > guess.
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > >
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > > The
>> > > rationale for
>> > > > > > > > >> separating
>> > > > > > > > >> > > scan
>> > > > > > > > >> > > > > > >> metrics
>> > > > > > > > >> > > > > > >> > > and
>> > > > > > > > >> > > > > > >> > > > > > > > metastore
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> persistence
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> is
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > that
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > >
>> "cascading
>> > > > > deletes"
>> > > > > > > > >> between
>> > > > > > > > >> > > them
>> > > > > > > > >> > > > > are
>> > > > > > > > >> > > > > > >> > hardly
>> > > > > > > > >> > > > > > >> > > > > ever
>> > > > > > > > >> > > > > > >> > > > > > > > > > required.
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> Furthermore,
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > the
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > > data and
>> > > query
>> > > > > > > patterns
>> > > > > > > > >> are
>> > > > > > > > >> > > very
>> > > > > > > > >> > > > > > >> > different
>> > > > > > > > >> > > > > > >> > > so
>> > > > > > > > >> > > > > > >> > > > > > > > different
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> technologies
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > might
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > > be
>> > > beneficial in
>> > > > > > each
>> > > > > > > > >> case.
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > >
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > > [1]
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> >
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>>
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >>
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >>
>> > > > > > > > >> > > > > > >> > > > > > > > > > > >
>> > > > > > > > >> > > > > > >> > > > > > > > > > >
>> > > > > > > > >> > > > > > >> > > > > > > > > >
>> > > > > > > > >> > > > > > >> > > > > > > > >
>> > > > > > > > >> > > > > > >> > > > > > > >
>> > > > > > > > >> > > > > > >> > > > > > >
>> > > > > > > > >> > > > > > >> > > > > >
>> > > > > > > > >> > > > > > >> > > > >
>> > > > > > > > >> > > > > > >> > > >
>> > > > > > > > >> > > > > > >> > >
>> > > > > > > > >> > > > > > >> >
>> > > > > > > > >> > > > > > >>
>> > > > > > > > >> > > > > >
>> > > > > > > > >> > > > >
>> > > > > > > > >> > >
>> > > > > > > > >> >
>> > > > > > > > >>
>> > > > > > >
>> > > > > >
>> > > > >
>> > >
>> >
>> https://urldefense.com/v3/__https://lists.apache.org/thread/yto2wp982t43h1mqjwnslswhws5z47cy__;!!Iz9xO38YGHZK!5EuyFFkk3vhRWVIRvQAWBSQfpJkTMA9HxugzDwXmN0LPPqhEFxYkFRGVhtb8AqUwXtDh2OplcMnbMDHKxYDakNU$
>> > > > > > > > >> > >> > > >> > > > > > > > > > > > >> >> >>> > >
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > > Cheers,
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > > Dmitri.
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > >
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > > On Fri,
>> Feb
>> > > 27,
>> > > > > > 2026
>> > > > > > > at
>> > > > > > > > >> > > 6:19 PM
>> > > > > > > > >> > > > > > Anand
>> > > > > > > > >> > > > > > >> > Kumar
>> > > > > > > > >> > > > > > >> > > > > > > Sankaran
>> > > > > > > > >> > > > > > >> > > > > > > > > via
>> > > > > > > > >> > > > > > >> > > > > > > > > > > dev
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > <
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > >
>> > > > > > > dev@polaris.apache.org>
>> > > > > > > > >> > > wrote:
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > >
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > > > Thanks
>> > all.
>> > > > > This
>> > > > > > > PR is
>> > > > > > > > >> > > merged
>> > > > > > > > >> > > > > now.
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > > >
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > > > Here
>> are
>> > > the
>> > > > > > > follow-up
>> > > > > > > > >> > > features
>> > > > > > > > >> > > > > /
>> > > > > > > > >> > > > > > >> work
>> > > > > > > > >> > > > > > >> > > > > needed.
>> > > > > > > > >> > > > > > >> > > > > > > > These
>> > > > > > > > >> > > > > > >> > > > > > > > > > > were
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> all
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> part of
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > > the
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > > >
>> merged PR
>> > > at
>> > > > > some
>> > > > > > > > >> point in
>> > > > > > > > >> > > time
>> > > > > > > > >> > > > > > and
>> > > > > > > > >> > > > > > >> > were
>> > > > > > > > >> > > > > > >> > > > > > removed
>> > > > > > > > >> > > > > > >> > > > > > > to
>> > > > > > > > >> > > > > > >> > > > > > > > > > > reduce
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> scope.
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > > >
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > > > Please
>> > let
>> > > me
>> > > > > > know
>> > > > > > > what
>> > > > > > > > >> > you
>> > > > > > > > >> > > > > think.
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > > >
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > > >
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > > >   1.
>> A
>> > > REST
>> > > > > API
>> > > > > > to
>> > > > > > > > >> > paginate
>> > > > > > > > >> > > > > > through
>> > > > > > > > >> > > > > > >> > > table
>> > > > > > > > >> > > > > > >> > > > > > > metrics.
>> > > > > > > > >> > > > > > >> > > > > > > > > > This
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> will be
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > non-IRC
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > > >
>> standard
>> > > > > > addition.
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > > >   2.
>> > > Utilities
>> > > > > > for
>> > > > > > > > >> > > managing old
>> > > > > > > > >> > > > > > >> > records,
>> > > > > > > > >> > > > > > >> > > > > > should
>> > > > > > > > >> > > > > > >> > > > > > > be
>> > > > > > > > >> > > > > > >> > > > > > > > > > > common
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> with
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> events.
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > > > There
>> was
>> > > some
>> > > > > > > > >> discussion
>> > > > > > > > >> > > that
>> > > > > > > > >> > > > > it
>> > > > > > > > >> > > > > > >> > belongs
>> > > > > > > > >> > > > > > >> > > > to
>> > > > > > > > >> > > > > > >> > > > > > the
>> > > > > > > > >> > > > > > >> > > > > > > > CLI.
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > > >   3.
>> > > Separate
>> > > > > > > > >> datasource
>> > > > > > > > >> > > > > > (metrics,
>> > > > > > > > >> > > > > > >> > > events,
>> > > > > > > > >> > > > > > >> > > > > > even
>> > > > > > > > >> > > > > > >> > > > > > > > > other
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> tables?).
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > > >
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > > >
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > > >
>> Anything
>> > > else?
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > > >
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > > > -
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > > > Anand
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > > >
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > > >
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > >
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> >
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> >
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>>
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >>
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >>
>> > > > > > > > >> > > > > > >> > > > > > > > > > > > >
>> > > > > > > > >> > > > > > >> > > > > > > > > > > >
>> > > > > > > > >> > > > > > >> > > > > > > > > > >
>> > > > > > > > >> > > > > > >> > > > > > > > > >
>> > > > > > > > >> > > > > > >> > > > > > > > >
>> > > > > > > > >> > > > > > >> > > > > > > >
>> > > > > > > > >> > > > > > >> > > > > > >
>> > > > > > > > >> > > > > > >> > > > > >
>> > > > > > > > >> > > > > > >> > > > >
>> > > > > > > > >> > > > > > >> > > >
>> > > > > > > > >> > > > > > >> > >
>> > > > > > > > >> > > > > > >> >
>> > > > > > > > >> > > > > > >>
>> > > > > > > > >> > > > > > >
>> > > > > > > > >> > > > > >
>> > > > > > > > >> > > > >
>> > > > > > > > >> > >
>> > > > > > > > >> >
>> > > > > > > > >> >
>> > > > > > > > >>
>> > > > > > > > >
>> > > > > > >
>> > > > > >
>> > > > >
>> > >
>> >
>>
>

Re: Proposal for REST endpoints for table metrics and events

[Dmitri Bourlatchkov <di...@apache.org>]

Hi All,

PR [4115] has been approved for two weeks without any follow-up comments.

I'm going to merge it on Oct 22 unless fresh concerns are raised.

[4115] https://github.com/apache/polaris/pull/4115

Cheers,
Dmitri.

I

On Wed, Oct 7, 2026 at 9:45 PM Yufei Gu <fl...@gmail.com> wrote:

> Thanks for the reminder, Dmitri! I will take a look.
>
> Yufei
>
>
> On Wed, Oct 7, 2026 at 5:37 PM Dmitri Bourlatchkov <di...@apache.org>
> wrote:
>
> > Hi All,
> >
> > Metrics grants have been merged as a separate PR.
> >
> > Now [4115] is ready for one more review round. IMHO, it is ready to
> merge.
> >
> > [4115] https://github.com/apache/polaris/pull/4115
> >
> > Cheers,
> > Dmitri.
> >
> > On Fri, Oct 2, 2026 at 1:01 AM Jean-Baptiste Onofré <jb...@nanthrax.net>
> > wrote:
> >
> > > Hi all,
> > >
> > > I agree with Yufei: I would keep TABLE_READ_METRICS as a separate,
> > > explicit grant, not implied by TABLE_FULL_METADATA.
> > >
> > > I believe metrics reports are closer to an activity log than to table
> > > metadata. I also think it's consistent with what we have today, as
> > > reporting scan and commit metrics is already with TABLE_READ_DATA and
> > > TABLE_WRITE_DATA, not by "metadata" privilege.
> > >
> > > For the troubleshooting use case, the cost is just one additional
> > > grant, I think it's acceptable :) I would still let
> > > CATALOG_MANAGE_CONTENT implies it, as that privilege already implies
> > > TABLE_READ_DATA.
> > >
> > > Concretely for PR #5669, I propose:
> > > - to remove the "Implied by TABLE_FULL_METADATA" statement from the
> > > javadoc and CHANGELOG
> > > - to add SUPER_PRIVILEGES entry in the PR, so the grant change is
> > > self-contained
> > > - to drop ICEBERG_TABLE restriction.
> > > - to update access-control.md documenting TABLE_READ_METRICS, etc
> > >
> > > I will comment directly in the PR.
> > >
> > > Regards
> > > JB
> > >
> > > On Fri, Oct 2, 2026 at 3:40 AM Yufei Gu <fl...@gmail.com> wrote:
> > > >
> > > > Thanks Dmitri for the reminder.
> > > >
> > > > PR #5659 <https://github.com/apache/polaris/pull/5659> adds
> > > > TABLE_READ_METRICS and says it is implied by TABLE_FULL_METADATA.
> > Should
> > > > that be the default?
> > > >
> > > > I’m leaning toward keeping it separate. Metrics describe workloads
> > (Spark
> > > > ETL, Flink streaming, etc) using a table and can include query
> filters,
> > > so
> > > > someone managing table metadata may not need access to other users’
> > query
> > > > activity.
> > > >
> > > > Including it could make troubleshooting easier for table admins,
> > though.
> > > > Are there use cases that favor automatic access over an explicit
> grant?
> > > > Yufei
> > > >
> > > >
> > > > On Wed, Sep 30, 2026 at 1:57 PM Dmitri Bourlatchkov <
> dimas@apache.org>
> > > > wrote:
> > > >
> > > > > Hi Yufei,
> > > > >
> > > > > Here's the separate PR for Metrics-related grants:
> > > > > https://github.com/apache/polaris/pull/5659
> > > > >
> > > > > Please review it to unblock this contribution.
> > > > >
> > > > > Thanks,
> > > > > Dmitri.
> > > > >
> > > > > On Wed, Sep 30, 2026 at 1:05 PM Yufei Gu <fl...@gmail.com>
> > wrote:
> > > > >
> > > > > > From the previous email, we agreed to split the REST change from
> > the
> > > > > grant
> > > > > > change. I think the PR is ready to go if we only keep the grant
> > > changes.
> > > > > >
> > > > > > Yufei
> > > > > >
> > > > > >
> > > > > > On Tue, Sep 29, 2026 at 9:26 PM Jean-Baptiste Onofré <
> > > jb@nanthrax.net>
> > > > > > wrote:
> > > > > >
> > > > > > > Hi guys,
> > > > > > >
> > > > > > > I agree with Dmitri: +1 to merge PR #4115 on Oct 1 (tomorrow).
> > > > > > >
> > > > > > > From my side, all outstanding feedback has been addressed,
> > > including
> > > > > > > the final documentation points raised by EJ regarding the
> > empty-200
> > > > > > > NoOp behavior and reconstructed payload description. The branch
> > is
> > > > > > > clean, up to date with main, and CI is green.
> > > > > > >
> > > > > > > Once this lands, we can follow up with PR #4756 for the JDBC
> > query
> > > SPI
> > > > > > > implementation.
> > > > > > >
> > > > > > > Regards
> > > > > > > JB
> > > > > > >
> > > > > > > On Wed, Sep 30, 2026 at 1:53 AM Dmitri Bourlatchkov <
> > > dimas@apache.org>
> > > > > > > wrote:
> > > > > > > >
> > > > > > > > Hi All,
> > > > > > > >
> > > > > > > > Any remaining blockers on [4115]?
> > > > > > > >
> > > > > > > > Given that the review was very long, please restate any
> > remaining
> > > > > > > concerns
> > > > > > > > (here or in GH).
> > > > > > > >
> > > > > > > > Otherwise, I propose merging [4115] on Oct 1.
> > > > > > > >
> > > > > > > > [4115] https://github.com/apache/polaris/pull/4115
> > > > > > > >
> > > > > > > > Thanks,
> > > > > > > > Dmitri.
> > > > > > > >
> > > > > > > > On Wed, Sep 23, 2026 at 7:53 PM Dmitri Bourlatchkov <
> > > > > dimas@apache.org>
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > > > Hi EJ,
> > > > > > > > >
> > > > > > > > > Thanks for the meeting recap!
> > > > > > > > >
> > > > > > > > > All:
> > > > > > > > >
> > > > > > > > > Worth noting: PR [4115] also contains Management API
> changes
> > > that
> > > > > > were
> > > > > > > not
> > > > > > > > > anticipated initially and did not get an API change review
> > > until
> > > > > now.
> > > > > > > These
> > > > > > > > > are small changes to add new grant types for metrics RBAC..
> > > > > > > > >
> > > > > > > > > I've reviewed and approved PR [4115] in GH.
> > > > > > > > >
> > > > > > > > > [4115] https://github.com/apache/polaris/pull/4115
> > > > > > > > >
> > > > > > > > > Cheers,
> > > > > > > > > Dmitri.
> > > > > > > > >
> > > > > > > > > On Wed, Aug 12, 2026 at 5:21 PM EJ Wang <
> > > > > > > ej.wang.dev.interest@gmail.com>
> > > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > >> Hi folks,
> > > > > > > > >>
> > > > > > > > >> Notes from today's metrics architecture sync are in the
> > usual
> > > doc:
> > > > > > > > >>
> > > > > > > > >>
> > > > > > >
> > > > > >
> > > > >
> > >
> >
> https://docs.google.com/document/d/100h7c4damrUzVuquYbBHM0EvA4LSWuW2IT2dN_7nYVA/edit?pli=1&tab=t.18iv1m7p6i6b
> > > > > > > > >>
> > > > > > > > >> Highlights:
> > > > > > > > >>
> > > > > > > > >> Module placement. core/ now holds the entity data model
> > only.
> > > A
> > > > > new
> > > > > > > spi/
> > > > > > > > >> takes every other shared contract: feature SPIs, feature
> > > non-SPI
> > > > > > > > >> contracts,
> > > > > > > > >> substrate SPIs, substrate non-SPI contracts, and
> > > contract-required
> > > > > > > types.
> > > > > > > > >> That settles the question PR #5204 reopened, by placing
> into
> > > a new
> > > > > > SPI
> > > > > > > > >> module. Note that it changes slightly towards the
> referenced
> > > WIP
> > > > > RFC
> > > > > > > [1]
> > > > > > > > >> recommendations: what proposed to place in core/, now
> should
> > > go to
> > > > > > > spi/
> > > > > > > > >> except for the entity data model.
> > > > > > > > >>
> > > > > > > > >> Metrics PRs. #5068 merged. #5204 needs reshaping onto the
> > > split.
> > > > > > > #4115 and
> > > > > > > > >> #4756 stay open with action on the author to follow up on
> > > review
> > > > > > > comments.
> > > > > > > > >>
> > > > > > > > >> [1]
> > > > > > > > >>
> > > > > > > > >>
> > > > > > >
> > > > > >
> > > > >
> > >
> >
> https://docs.google.com/document/d/1mj3vGpix9zTDTFFJKtWT5ZxI3PTGI1YNb5xFqyWrOkc/edit?tab=t.0
> > > > > > > > >>
> > > > > > > > >> On Wed, Jul 15, 2026 at 11:01 AM Anand Kumar Sankaran via
> > dev
> > > <
> > > > > > > > >> dev@polaris.apache.org> wrote:
> > > > > > > > >>
> > > > > > > > >> > Thanks EJ.  My plan is to introduce PR0 in a new branch
> /
> > > PR and
> > > > > > > > >> repurpose
> > > > > > > > >> > #4115 and #4756 for PR1 and PR2 to maintain all the
> > valuable
> > > > > > > discussions
> > > > > > > > >> > that happened there.
> > > > > > > > >> >
> > > > > > > > >> > From: EJ Wang <ej...@gmail.com>
> > > > > > > > >> > Date: Wednesday, July 15, 2026 at 10:41 AM
> > > > > > > > >> > To: dev@polaris.apache.org <de...@polaris.apache.org>
> > > > > > > > >> > Subject: Re: Proposal for REST endpoints for table
> metrics
> > > and
> > > > > > > events
> > > > > > > > >> >
> > > > > > > > >> > Hi folks, Thanks again for the discussion today. I
> updated
> > > the
> > > > > > sync
> > > > > > > doc
> > > > > > > > >> > with notes from today's metrics sync: https:
> //urldefense.
> > > > > > > > >> com/v3/__https:
> > > > > > > > >> > //docs. google.
> > > > > > > > >> >
> > > > > > > > >>
> > > > > > >
> > > > > >
> > > > >
> > >
> >
> com/document/d/100h7c4damrUzVuquYbBHM0EvA4LSWuW2IT2dN_7nYVA/edit?pli=1&tab=t.
> > > > > > > > >> >
> > > > > > > > >>
> > > > > > >
> > > > > >
> > > > >
> > >
> >
> 2ahd7wze5f__;!!Iz9xO38YGHZK!4rnlVIqBQQQ6xpGe1PMV58edOS_cGrKoqdfhw4qSp_JQCavvEjaxXk0OwGPnxyEcO60AzKQK_XDtp7H3DdkFXcrXJwAz1g0$
> > > > > > > > >> >
> > > > > > > > >> >
> > > > > > > > >> > Hi folks,
> > > > > > > > >> >
> > > > > > > > >> > Thanks again for the discussion today. I updated the
> sync
> > > doc
> > > > > with
> > > > > > > notes
> > > > > > > > >> > from today's metrics sync:
> > > > > > > > >> >
> > > > > > > > >> >
> > > > > > > > >>
> > > > > > >
> > > > > >
> > > > >
> > >
> >
> https://urldefense.com/v3/__https://docs.google.com/document/d/100h7c4damrUzVuquYbBHM0EvA4LSWuW2IT2dN_7nYVA/edit?pli=1&tab=t.2ahd7wze5f__;!!Iz9xO38YGHZK!4rnlVIqBQQQ6xpGe1PMV58edOS_cGrKoqdfhw4qSp_JQCavvEjaxXk0OwGPnxyEcO60AzKQK_XDtp7H3DdkFXcrXJwAz1g0$
> > > > > > > > >> >
> > > > > > > > >> > See highlights below:
> > > > > > > > >> >
> > > > > > > > >> > In previous meetings, EJ proposed a two prs framing to
> > push
> > > > > > forward
> > > > > > > the
> > > > > > > > >> > metrics query work in polaris. Dmitri pointed out a
> subtle
> > > point
> > > > > > > that
> > > > > > > > >> pr2
> > > > > > > > >> > is currently also trying to do the persistence related
> > code
> > > > > > > placement
> > > > > > > > >> > refactor. For the sake of straightforward review
> process,
> > > EJ,
> > > > > > > Dmitri and
> > > > > > > > >> > Anand (the author) agreed to separate the persistence
> > > > > refactoring
> > > > > > > into
> > > > > > > > >> pr0,
> > > > > > > > >> > and reframing the pr scopes as follow:
> > > > > > > > >> >
> > > > > > > > >> >    1.
> > > > > > > > >> >
> > > > > > > > >> >    Pr0: refactor irc metrics spi
> > > > > > > > >> >    1.
> > > > > > > > >> >
> > > > > > > > >> >       SPI should stay in core/ with metric type
> enveloped
> > > > > > > > >> >       2.
> > > > > > > > >> >
> > > > > > > > >> >       default spi impl, no-op, in extensions/ - default
> > > > > selection
> > > > > > at
> > > > > > > > >> >       runtime/
> > > > > > > > >> >       3.
> > > > > > > > >> >
> > > > > > > > >> >       move metrics persistence and correspondings to
> > > extensions/
> > > > > > as
> > > > > > > an
> > > > > > > > >> >       alternative SPI impl
> > > > > > > > >> >       2.
> > > > > > > > >> >
> > > > > > > > >> >    Pr1: rest api for query
> > > > > > > > >> >
> > > > > > > > >> >
> > > > > > > > >>
> > > > > > >
> > > > > >
> > > > >
> > >
> >
> https://urldefense.com/v3/__https://github.com/apache/polaris/pull/4115/changes__;!!Iz9xO38YGHZK!4rnlVIqBQQQ6xpGe1PMV58edOS_cGrKoqdfhw4qSp_JQCavvEjaxXk0OwGPnxyEcO60AzKQK_XDtp7H3DdkFXcrXFCbEbRk$
> > > > > > > > >> >    1.
> > > > > > > > >> >
> > > > > > > > >> >       rest api spec in extensions/ as optional spec
> > > > > > > > >> >       2.
> > > > > > > > >> >
> > > > > > > > >> >       SPI in extensions/ as optional spec spi
> > > > > > > > >> >       3.
> > > > > > > > >> >
> > > > > > > > >> >       default spi impl in extensions/ no-op
> > > > > > > > >> >       4.
> > > > > > > > >> >
> > > > > > > > >> >       handler related in runtime/ - thin adaptor from
> http
> > > shape
> > > > > > to
> > > > > > > spi
> > > > > > > > >> >       shape
> > > > > > > > >> >       3.
> > > > > > > > >> >
> > > > > > > > >> >    Pr2: jdbc impl for query
> > > > > > > > >> >
> > > > > > > > >>
> > > > > > >
> > > > > >
> > > > >
> > >
> >
> https://urldefense.com/v3/__https://github.com/apache/polaris/pull/4756__;!!Iz9xO38YGHZK!4rnlVIqBQQQ6xpGe1PMV58edOS_cGrKoqdfhw4qSp_JQCavvEjaxXk0OwGPnxyEcO60AzKQK_XDtp7H3DdkFXcrXRVMBzqE$
> > > > > > > > >> >    1.
> > > > > > > > >> >
> > > > > > > > >> >       alternative spi impl in extensions/ -> query with
> > jdbc
> > > > > > > > >> >
> > > > > > > > >> >
> > > > > > > > >> > Thanks,
> > > > > > > > >> > -ej
> > > > > > > > >> >
> > > > > > > > >> > On Wed, Jul 1, 2026 at 8:01 AM Alexandre Dutra <
> > > > > adutra@apache.org
> > > > > > >
> > > > > > > > >> wrote:
> > > > > > > > >> >
> > > > > > > > >> > > Hi all,
> > > > > > > > >> > >
> > > > > > > > >> > > > While the current implementation stores metrics in
> the
> > > > > > > metastore,
> > > > > > > > >> other
> > > > > > > > >> > > deployments
> > > > > > > > >> > > may choose a KV store, a dedicated metrics warehouse,
> or
> > > > > another
> > > > > > > > >> backend
> > > > > > > > >> > > entirely. In those cases, exposing a Polaris REST API
> > > over a
> > > > > > > specific
> > > > > > > > >> > > schema feels like coupling the API to an
> implementation
> > > > > detail.
> > > > > > > > >> > >
> > > > > > > > >> > > I wouldn't say to an implementation detail, but rather
> > to
> > > a
> > > > > > > particular
> > > > > > > > >> > > setup / configuration. I think that's fine. If you
> don't
> > > store
> > > > > > > > >> > > metrics, you don't read metrics. Similar situations
> will
> > > > > > certainly
> > > > > > > > >> > > happen in the future, e.g. if we ever introduce an API
> > to
> > > read
> > > > > > > events,
> > > > > > > > >> > > that would only work if events are being persisted in
> > the
> > > > > first
> > > > > > > place.
> > > > > > > > >> > > Again I think that's a fair trade-off.
> > > > > > > > >> > >
> > > > > > > > >> > > Thanks,
> > > > > > > > >> > > Alex
> > > > > > > > >> > >
> > > > > > > > >> > > On Wed, Jul 1, 2026 at 7:23 AM Dmitri Bourlatchkov <
> > > > > > > dimas@apache.org>
> > > > > > > > >> > > wrote:
> > > > > > > > >> > > >
> > > > > > > > >> > > > Hi Yufei,
> > > > > > > > >> > > >
> > > > > > > > >> > > > From my POV, it is fine for Polaris to have multiple
> > > REST
> > > > > API
> > > > > > > > >> surfaces
> > > > > > > > >> > > with
> > > > > > > > >> > > > different features and different degrees of
> > "readiness".
> > > > > Each
> > > > > > > API
> > > > > > > > >> can
> > > > > > > > >> > > > evolve fairly independently. With proper
> > modularization
> > > > > (which
> > > > > > > Anand
> > > > > > > > >> > > did),
> > > > > > > > >> > > > changes to the main Polaris code and these extra
> > modules
> > > > > > should
> > > > > > > > >> > normally
> > > > > > > > >> > > be
> > > > > > > > >> > > > non-intrusive to each other.
> > > > > > > > >> > > >
> > > > > > > > >> > > > I assume Anand is going to be the first user of this
> > > API.
> > > > > > > However,
> > > > > > > > >> if
> > > > > > > > >> > we
> > > > > > > > >> > > do
> > > > > > > > >> > > > not merge it we'll never know whether other users
> are
> > > > > > > interested. I
> > > > > > > > >> do
> > > > > > > > >> > > not
> > > > > > > > >> > > > think many end users read the dev ML, but they
> > certainly
> > > > > read
> > > > > > > the
> > > > > > > > >> docs,
> > > > > > > > >> > > > code and release notes.
> > > > > > > > >> > > >
> > > > > > > > >> > > > There is no risk for Polaris as a project in
> exposing
> > > this
> > > > > > > > >> experimental
> > > > > > > > >> > > > API. If usage dwindles over time, we can remove it.
> If
> > > it
> > > > > > gains
> > > > > > > a
> > > > > > > > >> > > > considerable user / contributor base, we will
> promote
> > > it to
> > > > > a
> > > > > > > > >> > > "first-class"
> > > > > > > > >> > > > (non-beta) API.
> > > > > > > > >> > > >
> > > > > > > > >> > > > Cheers,
> > > > > > > > >> > > > Dmitri.
> > > > > > > > >> > > >
> > > > > > > > >> > > > On Tue, Jun 30, 2026 at 7:53 PM Yufei Gu <
> > > > > > flyrain000@gmail.com>
> > > > > > > > >> wrote:
> > > > > > > > >> > > >
> > > > > > > > >> > > > > Hi all,
> > > > > > > > >> > > > >
> > > > > > > > >> > > > > Thanks for driving this work forward. Building
> this
> > > as a
> > > > > POC
> > > > > > > makes
> > > > > > > > >> > > perfect
> > > > > > > > >> > > > > sense if the goal is to validate the idea and
> gather
> > > > > > feedback.
> > > > > > > > >> Should
> > > > > > > > >> > > we
> > > > > > > > >> > > > > merge a POC into the main branch before agreeing
> on
> > > the
> > > > > > > overall
> > > > > > > > >> > > direction?
> > > > > > > > >> > > > > To me, that feels like a separate discussion from
> > > whether
> > > > > > the
> > > > > > > > >> > > > > implementation itself looks reasonable.
> > > > > > > > >> > > > >
> > > > > > > > >> > > > > One concern that I don't think has been fully
> > > addressed is
> > > > > > > > >> whether we
> > > > > > > > >> > > need
> > > > > > > > >> > > > > to expose metrics consumption through a Polaris
> REST
> > > > > > endpoint
> > > > > > > at
> > > > > > > > >> all.
> > > > > > > > >> > > > >
> > > > > > > > >> > > > > My concern is less about the API itself and more
> > > about the
> > > > > > > > >> boundary
> > > > > > > > >> > > Polaris
> > > > > > > > >> > > > > should own. The proposed REST API is tied to a
> > > particular
> > > > > > > > >> persistence
> > > > > > > > >> > > > > schema, but that schema is just one possible
> > > > > implementation.
> > > > > > > While
> > > > > > > > >> > the
> > > > > > > > >> > > > > current implementation stores metrics in the
> > > metastore,
> > > > > > other
> > > > > > > > >> > > deployments
> > > > > > > > >> > > > > may choose a KV store, a dedicated metrics
> > warehouse,
> > > or
> > > > > > > another
> > > > > > > > >> > > backend
> > > > > > > > >> > > > > entirely. In those cases, exposing a Polaris REST
> > API
> > > > > over a
> > > > > > > > >> specific
> > > > > > > > >> > > > > schema feels like coupling the API to an
> > > implementation
> > > > > > > detail.
> > > > > > > > >> > > > >
> > > > > > > > >> > > > > Polaris isn't intended to become a full metrics
> > > system.
> > > > > I'd
> > > > > > > > >> expect it
> > > > > > > > >> > > to
> > > > > > > > >> > > > > focus on producing metrics and let deployments
> > choose
> > > the
> > > > > > most
> > > > > > > > >> > > appropriate
> > > > > > > > >> > > > > storage and consumption model for their
> environment.
> > > > > > > > >> > > > >
> > > > > > > > >> > > > > I'm not opposed to exploring this direction, but
> we
> > > should
> > > > > > > first
> > > > > > > > >> > align
> > > > > > > > >> > > on
> > > > > > > > >> > > > > whether exposing a REST API through Polaris is the
> > > right
> > > > > > long
> > > > > > > term
> > > > > > > > >> > > approach
> > > > > > > > >> > > > > before merging it into the main branch.
> > > > > > > > >> > > > >
> > > > > > > > >> > > > > Thanks,
> > > > > > > > >> > > > > Yufei
> > > > > > > > >> > > > >
> > > > > > > > >> > > > > Yufei
> > > > > > > > >> > > > >
> > > > > > > > >> > > > >
> > > > > > > > >> > > > > On Mon, Jun 29, 2026 at 9:27 AM Dmitri
> Bourlatchkov
> > <
> > > > > > > > >> > dimas@apache.org>
> > > > > > > > >> > > > > wrote:
> > > > > > > > >> > > > >
> > > > > > > > >> > > > > > Hi All,
> > > > > > > > >> > > > > >
> > > > > > > > >> > > > > > It looks like Anand addressed all action items
> in
> > > > > [4115].
> > > > > > > > >> > > > > >
> > > > > > > > >> > > > > > Please review the latest state of the code. The
> PR
> > > LGTM
> > > > > > and
> > > > > > > I
> > > > > > > > >> > > approved in
> > > > > > > > >> > > > > > GH from my side.
> > > > > > > > >> > > > > >
> > > > > > > > >> > > > > > [4115]
> > > > > > > > >> >
> > > > > > > > >>
> > > > > > >
> > > > > >
> > > > >
> > >
> >
> https://urldefense.com/v3/__https://github.com/apache/polaris/pull/4115__;!!Iz9xO38YGHZK!4rnlVIqBQQQ6xpGe1PMV58edOS_cGrKoqdfhw4qSp_JQCavvEjaxXk0OwGPnxyEcO60AzKQK_XDtp7H3DdkFXcrXtFRscKs$
> > > > > > > > >> > > > > >
> > > > > > > > >> > > > > > Thanks,
> > > > > > > > >> > > > > > Dmitri.
> > > > > > > > >> > > > > >
> > > > > > > > >> > > > > > On Thu, Jun 25, 2026 at 1:56 PM Dmitri
> > Bourlatchkov
> > > <
> > > > > > > > >> > > dimas@apache.org>
> > > > > > > > >> > > > > > wrote:
> > > > > > > > >> > > > > >
> > > > > > > > >> > > > > > > Hi All,
> > > > > > > > >> > > > > > >
> > > > > > > > >> > > > > > > Recap of the discussion in today's Community
> > Sync:
> > > > > > > > >> > > > > > >
> > > > > > > > >> > > > > > > * The modular approach for Metrics SPI(s) and
> > > > > > > implementation
> > > > > > > > >> > seems
> > > > > > > > >> > > to
> > > > > > > > >> > > > > > have
> > > > > > > > >> > > > > > > community consensus (please comment if you
> still
> > > have
> > > > > > > > >> concerns)
> > > > > > > > >> > > > > > >
> > > > > > > > >> > > > > > > * What modules / feature flags are enabled in
> > > Apache
> > > > > > > Polaris
> > > > > > > > >> > > images by
> > > > > > > > >> > > > > > > default will be discussed later (when the
> > feature
> > > is
> > > > > > > close to
> > > > > > > > >> > > > > > completion).
> > > > > > > > >> > > > > > >
> > > > > > > > >> > > > > > > In any case, users will be able to turn the
> full
> > > > > feature
> > > > > > > > >> on/off
> > > > > > > > >> > at
> > > > > > > > >> > > > > their
> > > > > > > > >> > > > > > > discretion, regardless of the default flag
> > values.
> > > > > > > > >> > > > > > >
> > > > > > > > >> > > > > > > * It is generally preferable to have smaller
> PRs
> > > and
> > > > > > > introduce
> > > > > > > > >> > new
> > > > > > > > >> > > > > > > features gradually to simplify reviews.
> > > > > > > > >> > > > > > >
> > > > > > > > >> > > > > > > What this means for the Metrics API, IMHO is:
> > > > > > > > >> > > > > > >
> > > > > > > > >> > > > > > > a) The scope of PR 4115 is probably ok as it
> > > stands
> > > > > > right
> > > > > > > > >> now. If
> > > > > > > > >> > > > > someone
> > > > > > > > >> > > > > > > wants to propose splitting it further, please
> > > comment.
> > > > > > > > >> > > > > > >
> > > > > > > > >> > > > > > > b) After 4115 is merged, Metrics Persistence
> PRs
> > > will
> > > > > > > follow,
> > > > > > > > >> > with
> > > > > > > > >> > > > > > smaller
> > > > > > > > >> > > > > > > scope per PR as appropriate for the nature of
> > the
> > > code
> > > > > > > > >> changes.
> > > > > > > > >> > > > > > >
> > > > > > > > >> > > > > > > * The major concern was about exposing the new
> > > Metrics
> > > > > > > REST
> > > > > > > > >> API
> > > > > > > > >> > to
> > > > > > > > >> > > end
> > > > > > > > >> > > > > > > users. The nature of this concern was about
> > > avoiding
> > > > > the
> > > > > > > > >> (false)
> > > > > > > > >> > > > > > impression
> > > > > > > > >> > > > > > > that the new API is a firm and adopted Polaris
> > > > > > > specification.
> > > > > > > > >> > > > > > >
> > > > > > > > >> > > > > > > I think this concern is reasonable.
> > > > > > > > >> > > > > > >
> > > > > > > > >> > > > > > > I propose that Anand update the documentation
> > > sections
> > > > > > of
> > > > > > > PR
> > > > > > > > >> 4115
> > > > > > > > >> > > to
> > > > > > > > >> > > > > > > emphasise that the API is currently an
> > > experimental /
> > > > > > POC
> > > > > > > API
> > > > > > > > >> at
> > > > > > > > >> > > this
> > > > > > > > >> > > > > > time
> > > > > > > > >> > > > > > > and is subject to change (including breaking
> > > > > changes). I
> > > > > > > > >> believe
> > > > > > > > >> > it
> > > > > > > > >> > > > > > already
> > > > > > > > >> > > > > > > has the "beta" label. So this request is
> mostly
> > > for
> > > > > doc
> > > > > > > > >> changes
> > > > > > > > >> > to
> > > > > > > > >> > > > > > clarify
> > > > > > > > >> > > > > > > user expectations, since the "beta" label
> alone
> > > may
> > > > > not
> > > > > > be
> > > > > > > > >> > > interpreted
> > > > > > > > >> > > > > > the
> > > > > > > > >> > > > > > > same way by all users.
> > > > > > > > >> > > > > > >
> > > > > > > > >> > > > > > > I hope this update will resolve all remaining
> > > concerns
> > > > > > > with
> > > > > > > > >> 4115
> > > > > > > > >> > > and
> > > > > > > > >> > > > > > allow
> > > > > > > > >> > > > > > > for it to be merged so that the community will
> > be
> > > able
> > > > > > to
> > > > > > > > >> proceed
> > > > > > > > >> > > to
> > > > > > > > >> > > > > > > follow-up PRs (as noted above).
> > > > > > > > >> > > > > > >
> > > > > > > > >> > > > > > > Please add/correct me if I missed anything
> > > relevant.
> > > > > > > > >> > > > > > >
> > > > > > > > >> > > > > > > Thanks,
> > > > > > > > >> > > > > > > Dmitri.
> > > > > > > > >> > > > > > >
> > > > > > > > >> > > > > > > On Wed, Jun 24, 2026 at 8:11 PM Yufei Gu <
> > > > > > > > >> flyrain000@gmail.com>
> > > > > > > > >> > > wrote:
> > > > > > > > >> > > > > > >
> > > > > > > > >> > > > > > >> I do not think we have agreement on the query
> > > REST
> > > > > APIs
> > > > > > > yet.
> > > > > > > > >> > > > > > >>
> > > > > > > > >> > > > > > >> In particular, the current query API shape
> > seems
> > > > > highly
> > > > > > > > >> coupled
> > > > > > > > >> > > to the
> > > > > > > > >> > > > > > >> *example[1]* metrics in the Iceberg REST
> spec.
> > I
> > > do
> > > > > not
> > > > > > > think
> > > > > > > > >> > > that is
> > > > > > > > >> > > > > a
> > > > > > > > >> > > > > > >> good way to design a REST API contract. For
> > > example,
> > > > > > > fields
> > > > > > > > >> like
> > > > > > > > >> > > > > > >> resultDataFiles, resultDeleteFiles,
> > > > > totalFileSizeBytes,
> > > > > > > > >> > > > > > >> totalDataManifests,
> > > > > > > > >> > > > > > >> totalDeleteManifests, and
> scannedDataManifests
> > > are
> > > > > very
> > > > > > > > >> specific
> > > > > > > > >> > > to
> > > > > > > > >> > > > > one
> > > > > > > > >> > > > > > >> style of scan metric. That makes the API look
> > > more
> > > > > > like a
> > > > > > > > >> direct
> > > > > > > > >> > > > > > >> projection
> > > > > > > > >> > > > > > >> of the current example payload than a general
> > > query
> > > > > > > model.
> > > > > > > > >> > > > > > >>
> > > > > > > > >> > > > > > >> My preference is that Polaris should not
> define
> > > or
> > > > > > > strongly
> > > > > > > > >> > opine
> > > > > > > > >> > > on
> > > > > > > > >> > > > > the
> > > > > > > > >> > > > > > >> Iceberg scan or commit metrics query REST API
> > at
> > > this
> > > > > > > point.
> > > > > > > > >> > > Polaris
> > > > > > > > >> > > > > can
> > > > > > > > >> > > > > > >> resolve and authorize the table, accept the
> > > metrics
> > > > > > > report,
> > > > > > > > >> and
> > > > > > > > >> > > > > delegate
> > > > > > > > >> > > > > > >> it
> > > > > > > > >> > > > > > >> to the configured reporting path. Durable
> > > storage,
> > > > > > > filtering,
> > > > > > > > >> > > > > > dashboards,
> > > > > > > > >> > > > > > >> and query APIs should remain downstream impl.
> > > choices
> > > > > > > unless
> > > > > > > > >> and
> > > > > > > > >> > > until
> > > > > > > > >> > > > > > the
> > > > > > > > >> > > > > > >> Iceberg community standardizes a query
> > contract,
> > > > > which
> > > > > > I
> > > > > > > > >> doubt
> > > > > > > > >> > > will
> > > > > > > > >> > > > > > >> happen.
> > > > > > > > >> > > > > > >>
> > > > > > > > >> > > > > > >> So I do not think the intake path has to be
> > > blocked,
> > > > > > but
> > > > > > > I
> > > > > > > > >> would
> > > > > > > > >> > > > > prefer
> > > > > > > > >> > > > > > >> not
> > > > > > > > >> > > > > > >> to merge a Polaris specific query API that
> may
> > be
> > > > > hard
> > > > > > to
> > > > > > > > >> evolve
> > > > > > > > >> > > > > later.
> > > > > > > > >> > > > > > >>
> > > > > > > > >> > > > > > >> 1.
> > > > > > > > >> > > > > > >>
> > > > > > > > >> > > > > > >>
> > > > > > > > >> > > > > >
> > > > > > > > >> > > > >
> > > > > > > > >> > >
> > > > > > > > >> >
> > > > > > > > >>
> > > > > > >
> > > > > >
> > > > >
> > >
> >
> https://urldefense.com/v3/__https://github.com/apache/iceberg/blob/2a6c556c0c883c04c6d3fbd68e6aeda36b91e0aa/open-api/rest-catalog-open-api.yaml*L4152__;Iw!!Iz9xO38YGHZK!4rnlVIqBQQQ6xpGe1PMV58edOS_cGrKoqdfhw4qSp_JQCavvEjaxXk0OwGPnxyEcO60AzKQK_XDtp7H3DdkFXcrXCDsqYGo$
> > > > > > > > >> > > > > > >>
> > > > > > > > >> > > > > > >> Yufei
> > > > > > > > >> > > > > > >>
> > > > > > > > >> > > > > > >>
> > > > > > > > >> > > > > > >> On Tue, Jun 23, 2026 at 9:30 AM Dmitri
> > > Bourlatchkov <
> > > > > > > > >> > > dimas@apache.org
> > > > > > > > >> > > > > >
> > > > > > > > >> > > > > > >> wrote:
> > > > > > > > >> > > > > > >>
> > > > > > > > >> > > > > > >> > Hi All,
> > > > > > > > >> > > > > > >> >
> > > > > > > > >> > > > > > >> > So, what are the blockers for merging
> [4115],
> > > if
> > > > > any?
> > > > > > > > >> > > > > > >> >
> > > > > > > > >> > > > > > >> > [4115]
> > > > > > > > >> >
> > > > > > > > >>
> > > > > > >
> > > > > >
> > > > >
> > >
> >
> https://urldefense.com/v3/__https://github.com/apache/polaris/pull/4115__;!!Iz9xO38YGHZK!4rnlVIqBQQQ6xpGe1PMV58edOS_cGrKoqdfhw4qSp_JQCavvEjaxXk0OwGPnxyEcO60AzKQK_XDtp7H3DdkFXcrXtFRscKs$
> > > > > > > > >> > > > > > >> >
> > > > > > > > >> > > > > > >> > Thanks,
> > > > > > > > >> > > > > > >> > Dmitri.
> > > > > > > > >> > > > > > >> >
> > > > > > > > >> > > > > > >> > On Wed, Jun 17, 2026 at 5:28 PM EJ Wang <
> > > > > > > > >> > > > > > ej.wang.dev.interest@gmail.com
> > > > > > > > >> > > > > > >> >
> > > > > > > > >> > > > > > >> > wrote:
> > > > > > > > >> > > > > > >> >
> > > > > > > > >> > > > > > >> > > Hi folks,
> > > > > > > > >> > > > > > >> > >
> > > > > > > > >> > > > > > >> > > Thanks again for the discussion today. I
> > > updated
> > > > > > the
> > > > > > > sync
> > > > > > > > >> > doc
> > > > > > > > >> > > with
> > > > > > > > >> > > > > > >> notes
> > > > > > > > >> > > > > > >> > > from today's metrics sync:
> > > > > > > > >> > > > > > >> > >
> > > > > > > > >> > > > > > >> > >
> > > > > > > > >> > > > > > >> > >
> > > > > > > > >> > > > > > >> >
> > > > > > > > >> > > > > > >>
> > > > > > > > >> > > > > >
> > > > > > > > >> > > > >
> > > > > > > > >> > >
> > > > > > > > >> >
> > > > > > > > >>
> > > > > > >
> > > > > >
> > > > >
> > >
> >
> https://urldefense.com/v3/__https://docs.google.com/document/d/100h7c4damrUzVuquYbBHM0EvA4LSWuW2IT2dN_7nYVA/edit?tab=t.ezk0rgdx0c6m__;!!Iz9xO38YGHZK!4rnlVIqBQQQ6xpGe1PMV58edOS_cGrKoqdfhw4qSp_JQCavvEjaxXk0OwGPnxyEcO60AzKQK_XDtp7H3DdkFXcrX83IZkm0$
> > > > > > > > >> > > > > > >> > >
> > > > > > > > >> > > > > > >> > > A few highlights:
> > > > > > > > >> > > > > > >> > >
> > > > > > > > >> > > > > > >> > >    - No objection to keeping the default
> > > metrics
> > > > > > > behavior
> > > > > > > > >> > > small:
> > > > > > > > >> > > > > > >> > >    no-op/log-only is enough as the
> built-in
> > > > > > default.
> > > > > > > > >> > > > > > >> > >    - Durable storage, event forwarding,
> > > external
> > > > > > > queues,
> > > > > > > > >> > > > > dashboards,
> > > > > > > > >> > > > > > >> and
> > > > > > > > >> > > > > > >> > >    custom filtering should be
> > implementation
> > > > > > choices
> > > > > > > > >> behind
> > > > > > > > >> > > the
> > > > > > > > >> > > > > > >> metrics
> > > > > > > > >> > > > > > >> > >    reporting path.
> > > > > > > > >> > > > > > >> > >    - For PR #4115, the REST layer should
> > > > > > > > >> > resolve/authz/accept
> > > > > > > > >> > > the
> > > > > > > > >> > > > > > >> report,
> > > > > > > > >> > > > > > >> > >    then delegate to the selected metrics
> > > > > > > implementation.
> > > > > > > > >> It
> > > > > > > > >> > > should
> > > > > > > > >> > > > > > not
> > > > > > > > >> > > > > > >> > own
> > > > > > > > >> > > > > > >> > >    durable storage semantics.
> > > > > > > > >> > > > > > >> > >    - The rough boundary we discussed is:
> > spec
> > > > > owns
> > > > > > > the
> > > > > > > > >> wire
> > > > > > > > >> > > > > > contract,
> > > > > > > > >> > > > > > >> > >    runtime owns framework wiring,
> > > API/contract
> > > > > > > modules
> > > > > > > > >> own
> > > > > > > > >> > > > > > >> > provider-facing
> > > > > > > > >> > > > > > >> > >    contracts, and extensions own
> > replaceable
> > > > > > > > >> > implementations.
> > > > > > > > >> > > > > > >> > >    - OpenLineage has a similar
> path-naming
> > > > > > question.
> > > > > > > We
> > > > > > > > >> > should
> > > > > > > > >> > > > > avoid
> > > > > > > > >> > > > > > >> > >    occupying a generic lineage path too
> > > early if
> > > > > we
> > > > > > > want
> > > > > > > > >> > room
> > > > > > > > >> > > for
> > > > > > > > >> > > > > > >> other
> > > > > > > > >> > > > > > >> > >    lineage systems later.
> > > > > > > > >> > > > > > >> > >
> > > > > > > > >> > > > > > >> > > I believe the boundary rules previewed in
> > > sync
> > > > > > > meeting
> > > > > > > > >> are
> > > > > > > > >> > > useful
> > > > > > > > >> > > > > > for
> > > > > > > > >> > > > > > >> > > reviewing PR #4115, but it is not
> > > > > > > > >> > > > > > >> > > a final module-layout proposal.
> > > > > > > > >> > > > > > >> > >
> > > > > > > > >> > > > > > >> > > For metrics specifically, I think the
> > target
> > > > > > remains:
> > > > > > > > >> > > > > > >> > >    runtime
> > > > > > > > >> > > > > > >> > >      -> metrics reporting/emitting
> contract
> > > > > > > > >> > > > > > >> > >      -> default implementation
> > > > > > > > >> > > > > > >> > >           -> lower implementation
> details,
> > if
> > > > > > needed
> > > > > > > > >> > > > > > >> > >
> > > > > > > > >> > > > > > >> > > That keeps the default simple, while
> > allowing
> > > > > > durable
> > > > > > > > >> JDBC,
> > > > > > > > >> > > > > > >> event-backed,
> > > > > > > > >> > > > > > >> > > or external-queue implementations to be
> > added
> > > > > > > separately.
> > > > > > > > >> > > > > > >> > >
> > > > > > > > >> > > > > > >> > > We also touched on the split between
> > Iceberg
> > > > > > metrics
> > > > > > > > >> > emitting
> > > > > > > > >> > > and
> > > > > > > > >> > > > > > >> > > Polaris-owned metrics querying. The
> > querying
> > > side
> > > > > > can
> > > > > > > > >> keep
> > > > > > > > >> > > > > evolving,
> > > > > > > > >> > > > > > >> > > especially for dashboard or generic-table
> > use
> > > > > > cases,
> > > > > > > but
> > > > > > > > >> I
> > > > > > > > >> > do
> > > > > > > > >> > > not
> > > > > > > > >> > > > > > >> think
> > > > > > > > >> > > > > > >> > > that needs to block the intake path in
> this
> > > PR.
> > > > > > > > >> > > > > > >> > >
> > > > > > > > >> > > > > > >> > > Thanks,
> > > > > > > > >> > > > > > >> > > -ej
> > > > > > > > >> > > > > > >> > >
> > > > > > > > >> > > > > > >> > > On Tue, Jun 16, 2026 at 2:17 PM Yufei Gu
> <
> > > > > > > > >> > > flyrain000@gmail.com>
> > > > > > > > >> > > > > > >> wrote:
> > > > > > > > >> > > > > > >> > >
> > > > > > > > >> > > > > > >> > > > Thanks for chiming in, EJ. Agreed that
> > the
> > > > > > default
> > > > > > > > >> battery
> > > > > > > > >> > > > > should
> > > > > > > > >> > > > > > >> stay
> > > > > > > > >> > > > > > >> > > > small. I'm leaning toward not including
> > > metrics
> > > > > > > > >> > persistence
> > > > > > > > >> > > in
> > > > > > > > >> > > > > the
> > > > > > > > >> > > > > > >> > > default
> > > > > > > > >> > > > > > >> > > > battery.
> > > > > > > > >> > > > > > >> > > >
> > > > > > > > >> > > > > > >> > > > Yufei
> > > > > > > > >> > > > > > >> > > >
> > > > > > > > >> > > > > > >> > > >
> > > > > > > > >> > > > > > >> > > > On Thu, Jun 11, 2026 at 8:30 PM EJ
> Wang <
> > > > > > > > >> > > > > > >> > ej.wang.dev.interest@gmail.com>
> > > > > > > > >> > > > > > >> > > > wrote:
> > > > > > > > >> > > > > > >> > > >
> > > > > > > > >> > > > > > >> > > > > Hi Yufei,
> > > > > > > > >> > > > > > >> > > > >
> > > > > > > > >> > > > > > >> > > > > Thanks for connecting this back to
> the
> > > REST
> > > > > > > endpoint
> > > > > > > > >> > > > > proposal. I
> > > > > > > > >> > > > > > >> > > replied
> > > > > > > > >> > > > > > >> > > > > with the fuller version on the
> > > > > event-forwarding
> > > > > > > > >> thread,
> > > > > > > > >> > > but
> > > > > > > > >> > > > > > >> wanted to
> > > > > > > > >> > > > > > >> > > add
> > > > > > > > >> > > > > > >> > > > > the shorter version here too.
> > > > > > > > >> > > > > > >> > > > >
> > > > > > > > >> > > > > > >> > > > > I agree that metrics
> reporting/emitting
> > > is
> > > > > the
> > > > > > > right
> > > > > > > > >> > > > > conceptual
> > > > > > > > >> > > > > > >> > > boundary,
> > > > > > > > >> > > > > > >> > > > > and I also think the event/listener
> > path
> > > is a
> > > > > > > good
> > > > > > > > >> > > > > > implementation
> > > > > > > > >> > > > > > >> > > > direction
> > > > > > > > >> > > > > > >> > > > > to explore. The distinction I want to
> > > keep
> > > > > > clear
> > > > > > > is
> > > > > > > > >> > > default
> > > > > > > > >> > > > > > >> battery
> > > > > > > > >> > > > > > >> > vs
> > > > > > > > >> > > > > > >> > > > > extension implementation.
> > > > > > > > >> > > > > > >> > > > >
> > > > > > > > >> > > > > > >> > > > > For the REST/API side, I would keep
> the
> > > > > > semantics
> > > > > > > > >> > narrow:
> > > > > > > > >> > > > > > Polaris
> > > > > > > > >> > > > > > >> > > > resolves
> > > > > > > > >> > > > > > >> > > > > and authorizes the table, accepts the
> > > Iceberg
> > > > > > > > >> > scan/commit
> > > > > > > > >> > > > > > metrics
> > > > > > > > >> > > > > > >> > > report
> > > > > > > > >> > > > > > >> > > > > into the configured
> reporting/emitting
> > > path,
> > > > > > and
> > > > > > > > >> returns
> > > > > > > > >> > > 204.
> > > > > > > > >> > > > > > That
> > > > > > > > >> > > > > > >> > 204
> > > > > > > > >> > > > > > >> > > > > should mean Polaris accepted the
> report
> > > into
> > > > > > the
> > > > > > > > >> > ingestion
> > > > > > > > >> > > > > path;
> > > > > > > > >> > > > > > >> it
> > > > > > > > >> > > > > > >> > > > should
> > > > > > > > >> > > > > > >> > > > > not imply durable storage.
> > > > > > > > >> > > > > > >> > > > >
> > > > > > > > >> > > > > > >> > > > > Durable storage, event forwarding,
> > > external
> > > > > > > telemetry
> > > > > > > > >> > > routing,
> > > > > > > > >> > > > > > >> > > filtering,
> > > > > > > > >> > > > > > >> > > > > and retention should sit behind that
> > > > > > > > >> reporting/emitting
> > > > > > > > >> > > > > boundary
> > > > > > > > >> > > > > > >> as
> > > > > > > > >> > > > > > >> > > > > implementation-layer behavior. A
> > durable
> > > JDBC
> > > > > > > path
> > > > > > > > >> can
> > > > > > > > >> > be
> > > > > > > > >> > > one
> > > > > > > > >> > > > > > >> named
> > > > > > > > >> > > > > > >> > > > > extension implementation. An
> > > event/listener
> > > > > > > > >> forwarding
> > > > > > > > >> > > path
> > > > > > > > >> > > > > can
> > > > > > > > >> > > > > > be
> > > > > > > > >> > > > > > >> > > > another
> > > > > > > > >> > > > > > >> > > > > named extension implementation.
> > > > > > > > >> > > > > > >> > > > >
> > > > > > > > >> > > > > > >> > > > > So I am +1 on the event-forwarding
> idea
> > > as a
> > > > > > > > >> non-default
> > > > > > > > >> > > > > > extension
> > > > > > > > >> > > > > > >> > > > > implementation. I just would not make
> > it
> > > the
> > > > > > > default
> > > > > > > > >> > > battery
> > > > > > > > >> > > > > or
> > > > > > > > >> > > > > > >> the
> > > > > > > > >> > > > > > >> > > core
> > > > > > > > >> > > > > > >> > > > > API shape. The default battery can
> stay
> > > small
> > > > > > and
> > > > > > > > >> safe,
> > > > > > > > >> > > while
> > > > > > > > >> > > > > > >> > > deployments
> > > > > > > > >> > > > > > >> > > > > that need forwarding, dashboards, or
> > > durable
> > > > > > > storage
> > > > > > > > >> can
> > > > > > > > >> > > opt
> > > > > > > > >> > > > > > into
> > > > > > > > >> > > > > > >> the
> > > > > > > > >> > > > > > >> > > > > implementation that matches their
> > > operational
> > > > > > > model.
> > > > > > > > >> > > > > > >> > > > >
> > > > > > > > >> > > > > > >> > > > > Thanks,
> > > > > > > > >> > > > > > >> > > > > -ej
> > > > > > > > >> > > > > > >> > > > >
> > > > > > > > >> > > > > > >> > > > > On Wed, Jun 10, 2026 at 11:43 AM
> Yufei
> > > Gu <
> > > > > > > > >> > > > > flyrain000@gmail.com
> > > > > > > > >> > > > > > >
> > > > > > > > >> > > > > > >> > > wrote:
> > > > > > > > >> > > > > > >> > > > >
> > > > > > > > >> > > > > > >> > > > > > Thanks EJ. I agree that the
> > > > > > reporting/emitting
> > > > > > > > >> > boundary
> > > > > > > > >> > > > > feels
> > > > > > > > >> > > > > > >> like
> > > > > > > > >> > > > > > >> > > the
> > > > > > > > >> > > > > > >> > > > > > right SPI boundary, but I wonder if
> > we
> > > can
> > > > > > > simplify
> > > > > > > > >> > this
> > > > > > > > >> > > > > even
> > > > > > > > >> > > > > > >> > > further.
> > > > > > > > >> > > > > > >> > > > > >
> > > > > > > > >> > > > > > >> > > > > > Iceberg scan and commit metrics
> look
> > > very
> > > > > > > similar
> > > > > > > > >> to
> > > > > > > > >> > > events
> > > > > > > > >> > > > > to
> > > > > > > > >> > > > > > >> me.
> > > > > > > > >> > > > > > >> > > They
> > > > > > > > >> > > > > > >> > > > > are
> > > > > > > > >> > > > > > >> > > > > > append only, asynchronously
> consumed,
> > > often
> > > > > > > > >> forwarded
> > > > > > > > >> > to
> > > > > > > > >> > > > > > >> external
> > > > > > > > >> > > > > > >> > > > > systems,
> > > > > > > > >> > > > > > >> > > > > > and have similar retention and
> > cleanup
> > > > > > > > >> requirements.
> > > > > > > > >> > > More
> > > > > > > > >> > > > > > >> details
> > > > > > > > >> > > > > > >> > can
> > > > > > > > >> > > > > > >> > > > be
> > > > > > > > >> > > > > > >> > > > > > found in this thread [1]. In that
> > case,
> > > > > > > Polaris may
> > > > > > > > >> > only
> > > > > > > > >> > > > > need
> > > > > > > > >> > > > > > to
> > > > > > > > >> > > > > > >> > > accept
> > > > > > > > >> > > > > > >> > > > > the
> > > > > > > > >> > > > > > >> > > > > > report and emit an event through
> the
> > > > > existing
> > > > > > > event
> > > > > > > > >> > > > > framework.
> > > > > > > > >> > > > > > >> > > > Filtering,
> > > > > > > > >> > > > > > >> > > > > > async delivery, custom sinks, and
> > > retention
> > > > > > > > >> mechanisms
> > > > > > > > >> > > could
> > > > > > > > >> > > > > > >> then
> > > > > > > > >> > > > > > >> > be
> > > > > > > > >> > > > > > >> > > > > shared
> > > > > > > > >> > > > > > >> > > > > > instead of introducing a separate
> > > metrics
> > > > > > > specific
> > > > > > > > >> > > extension
> > > > > > > > >> > > > > > >> layer
> > > > > > > > >> > > > > > >> > > and
> > > > > > > > >> > > > > > >> > > > > SPI.
> > > > > > > > >> > > > > > >> > > > > >
> > > > > > > > >> > > > > > >> > > > > > The main requirement I still see is
> > > > > querying
> > > > > > > > >> metrics
> > > > > > > > >> > > through
> > > > > > > > >> > > > > > >> > Polaris.
> > > > > > > > >> > > > > > >> > > > If
> > > > > > > > >> > > > > > >> > > > > we
> > > > > > > > >> > > > > > >> > > > > > want that battery included
> > experience,
> > > we
> > > > > > could
> > > > > > > > >> work
> > > > > > > > >> > on
> > > > > > > > >> > > a
> > > > > > > > >> > > > > > >> > persistence
> > > > > > > > >> > > > > > >> > > > > > listener implementation that
> > > selectively
> > > > > > saves
> > > > > > > > >> metrics
> > > > > > > > >> > > and
> > > > > > > > >> > > > > > >> expose
> > > > > > > > >> > > > > > >> > it
> > > > > > > > >> > > > > > >> > > > via
> > > > > > > > >> > > > > > >> > > > > > the IRC event endpoint which seems
> > > pretty
> > > > > > > close in
> > > > > > > > >> > > Iceberg
> > > > > > > > >> > > > > > >> > > > > > community. Another option I like
> more
> > > is to
> > > > > > > allow
> > > > > > > > >> > users
> > > > > > > > >> > > to
> > > > > > > > >> > > > > > >> develop
> > > > > > > > >> > > > > > >> > > > their
> > > > > > > > >> > > > > > >> > > > > > own listener, so they can persist
> > > > > scan/commit
> > > > > > > > >> metrics
> > > > > > > > >> > > to any
> > > > > > > > >> > > > > > >> > systems
> > > > > > > > >> > > > > > >> > > > they
> > > > > > > > >> > > > > > >> > > > > > prefer. In general, that feels like
> > an
> > > > > > > > >> implementation
> > > > > > > > >> > > choice
> > > > > > > > >> > > > > > on
> > > > > > > > >> > > > > > >> top
> > > > > > > > >> > > > > > >> > > of
> > > > > > > > >> > > > > > >> > > > > the
> > > > > > > > >> > > > > > >> > > > > > event framework rather than
> something
> > > that
> > > > > > > needs to
> > > > > > > > >> > > shape
> > > > > > > > >> > > > > the
> > > > > > > > >> > > > > > >> core
> > > > > > > > >> > > > > > >> > > > > > architecture.
> > > > > > > > >> > > > > > >> > > > > >
> > > > > > > > >> > > > > > >> > > > > > 1.
> > > > > > > > >> > > > > > >> >
> > > > > > > > >> > >
> > > > > > > > >> >
> > > > > > > > >>
> > > > > > >
> > > > > >
> > > > >
> > >
> >
> https://urldefense.com/v3/__https://lists.apache.org/thread/x9j8nscvy8hq61tyn01mj8yp6n9of0kp__;!!Iz9xO38YGHZK!4rnlVIqBQQQ6xpGe1PMV58edOS_cGrKoqdfhw4qSp_JQCavvEjaxXk0OwGPnxyEcO60AzKQK_XDtp7H3DdkFXcrXGcGDgks$
> > > > > > > > >> > > > > > >> > > > > >
> > > > > > > > >> > > > > > >> > > > > > Thanks,
> > > > > > > > >> > > > > > >> > > > > > Yufei
> > > > > > > > >> > > > > > >> > > > > >
> > > > > > > > >> > > > > > >> > > > > >
> > > > > > > > >> > > > > > >> > > > > > On Thu, Jun 4, 2026 at 5:14 PM EJ
> > Wang
> > > <
> > > > > > > > >> > > > > > >> > > ej.wang.dev.interest@gmail.com
> > > > > > > > >> > > > > > >> > > > >
> > > > > > > > >> > > > > > >> > > > > > wrote:
> > > > > > > > >> > > > > > >> > > > > >
> > > > > > > > >> > > > > > >> > > > > > > Good point. I agree the existing
> > > > > > > > >> > > PolarisMetricsReporter is
> > > > > > > > >> > > > > > >> > already
> > > > > > > > >> > > > > > >> > > > very
> > > > > > > > >> > > > > > >> > > > > > > close to the right conceptual
> > > boundary. I
> > > > > > am
> > > > > > > not
> > > > > > > > >> > > > > proposing a
> > > > > > > > >> > > > > > >> > second
> > > > > > > > >> > > > > > >> > > > > > > parallel reporter concept. The
> > > > > distinction
> > > > > > I
> > > > > > > am
> > > > > > > > >> > > trying to
> > > > > > > > >> > > > > > >> make is
> > > > > > > > >> > > > > > >> > > > > > narrower:
> > > > > > > > >> > > > > > >> > > > > > >
> > > > > > > > >> > > > > > >> > > > > > > Current state:
> > > > > > > > >> > > > > > >> > > > > > >
> > > > > > > > >> > > > > > >> > > > > > >    - PolarisMetricsReporter is
> the
> > > > > existing
> > > > > > > > >> metrics
> > > > > > > > >> > > > > > reporting
> > > > > > > > >> > > > > > >> > hook.
> > > > > > > > >> > > > > > >> > > > > > >    - But it currently lives under
> > > > > > > > >> runtime/service.
> > > > > > > > >> > > > > > >> > > > > > >    - Its contract is documented
> > > around
> > > > > > > > >> Quarkus/CDI
> > > > > > > > >> > > > > discovery
> > > > > > > > >> > > > > > >> and
> > > > > > > > >> > > > > > >> > > > > > > @Identifier
> > > > > > > > >> > > > > > >> > > > > > >    selection.
> > > > > > > > >> > > > > > >> > > > > > >    - The default implementation
> is
> > > > > > log-only.
> > > > > > > > >> > > > > > >> > > > > > >    - The durable implementation
> > > writes
> > > > > > > through
> > > > > > > > >> > > > > > >> > > PolarisMetricsManager
> > > > > > > > >> > > > > > >> > > > > and
> > > > > > > > >> > > > > > >> > > > > > >    MetricsPersistence.
> > > > > > > > >> > > > > > >> > > > > > >    - MetricsPersistence is
> > currently
> > > > > > > inherited by
> > > > > > > > >> > > > > > >> > BasePersistence.
> > > > > > > > >> > > > > > >> > > > > > >
> > > > > > > > >> > > > > > >> > > > > > > My proposal:
> > > > > > > > >> > > > > > >> > > > > > >
> > > > > > > > >> > > > > > >> > > > > > >    - Keep the reporting/emitting
> > > boundary
> > > > > > as
> > > > > > > the
> > > > > > > > >> > SPI.
> > > > > > > > >> > > > > > >> > > > > > >    - Revise that contract in a
> > > > > > > framework-agnostic
> > > > > > > > >> > > optional
> > > > > > > > >> > > > > > >> > Iceberg
> > > > > > > > >> > > > > > >> > > > > > *metrics
> > > > > > > > >> > > > > > >> > > > > > >    extension API layer*.
> > > > > > > > >> > > > > > >> > > > > > >    - Keep runtime/service
> > > responsible for
> > > > > > > REST
> > > > > > > > >> > > ingestion,
> > > > > > > > >> > > > > > >> > > table/authz
> > > > > > > > >> > > > > > >> > > > > > >    validation, and runtime
> wiring.
> > > > > > > > >> > > > > > >> > > > > > >    - Keep the battery
> > implementation
> > > > > > > log-only in
> > > > > > > > >> the
> > > > > > > > >> > > > > metrics
> > > > > > > > >> > > > > > >> > > > extension
> > > > > > > > >> > > > > > >> > > > > > API
> > > > > > > > >> > > > > > >> > > > > > >    layer, *not under
> > > runtime/service*.
> > > > > > > > >> > > > > > >> > > > > > >    - Treat durable JDBC metrics
> as
> > > one
> > > > > > > > >> > implementation
> > > > > > > > >> > > of
> > > > > > > > >> > > > > the
> > > > > > > > >> > > > > > >> > > > reporting
> > > > > > > > >> > > > > > >> > > > > > SPI.
> > > > > > > > >> > > > > > >> > > > > > >    - Decompose the
> > > MetricsPersistence and
> > > > > > > > >> > > BasePersistence
> > > > > > > > >> > > > > > >> > coupling
> > > > > > > > >> > > > > > >> > > as
> > > > > > > > >> > > > > > >> > > > > > >    durable implementation detail.
> > > > > > > > >> > > > > > >> > > > > > >
> > > > > > > > >> > > > > > >> > > > > > > So the difference is not “new
> > > reporter
> > > > > > > concept vs
> > > > > > > > >> > old
> > > > > > > > >> > > > > > reporter
> > > > > > > > >> > > > > > >> > > > > concept.”
> > > > > > > > >> > > > > > >> > > > > > > but:
> > > > > > > > >> > > > > > >> > > > > > >
> > > > > > > > >> > > > > > >> > > > > > >
> > > > > > > > >> > > > > > >> > > > > > >    - PolarisMetricsReporter
> today =
> > > > > > > > >> runtime/service
> > > > > > > > >> > > > > > CDI-shaped
> > > > > > > > >> > > > > > >> > > hook.
> > > > > > > > >> > > > > > >> > > > > > >    - Target reporting SPI = same
> > > > > conceptual
> > > > > > > > >> > boundary,
> > > > > > > > >> > > but
> > > > > > > > >> > > > > > >> placed
> > > > > > > > >> > > > > > >> > in
> > > > > > > > >> > > > > > >> > > > the
> > > > > > > > >> > > > > > >> > > > > > >    right extension API layer and
> > kept
> > > > > > > > >> > > framework-agnostic.
> > > > > > > > >> > > > > > >> > > > > > >
> > > > > > > > >> > > > > > >> > > > > > > For the current metrics PR, I
> think
> > > the
> > > > > > > minimal
> > > > > > > > >> > > > > > architectural
> > > > > > > > >> > > > > > >> > > cleanup
> > > > > > > > >> > > > > > >> > > > > > > before merge is to *put the
> > > reporting SPI
> > > > > > > > >> contract
> > > > > > > > >> > > and the
> > > > > > > > >> > > > > > >> > battery
> > > > > > > > >> > > > > > >> > > > > > default
> > > > > > > > >> > > > > > >> > > > > > > in the right place*. The durable
> > JDBC
> > > > > > > > >> > implementation,
> > > > > > > > >> > > > > > >> > event-backed
> > > > > > > > >> > > > > > >> > > > > async
> > > > > > > > >> > > > > > >> > > > > > > implementation, external queue
> > > support,
> > > > > and
> > > > > > > > >> deeper
> > > > > > > > >> > > > > > persistence
> > > > > > > > >> > > > > > >> > > > cleanup
> > > > > > > > >> > > > > > >> > > > > > can
> > > > > > > > >> > > > > > >> > > > > > > continue as follow-ups, as long
> as
> > > we do
> > > > > > not
> > > > > > > lock
> > > > > > > > >> > the
> > > > > > > > >> > > SPI
> > > > > > > > >> > > > > > >> > boundary
> > > > > > > > >> > > > > > >> > > to
> > > > > > > > >> > > > > > >> > > > > > > BasePersistence or to
> > runtime/service
> > > > > > wiring.
> > > > > > > > >> > > > > > >> > > > > > >
> > > > > > > > >> > > > > > >> > > > > > > -ej
> > > > > > > > >> > > > > > >> > > > > > >
> > > > > > > > >> > > > > > >> > > > > > > On Wed, Jun 3, 2026 at 3:00 PM
> > Dmitri
> > > > > > > > >> Bourlatchkov <
> > > > > > > > >> > > > > > >> > > dimas@apache.org
> > > > > > > > >> > > > > > >> > > > >
> > > > > > > > >> > > > > > >> > > > > > > wrote:
> > > > > > > > >> > > > > > >> > > > > > >
> > > > > > > > >> > > > > > >> > > > > > > > Hi EJ,
> > > > > > > > >> > > > > > >> > > > > > > >
> > > > > > > > >> > > > > > >> > > > > > > > Thanks for the recap / summary!
> > > > > > > > >> > > > > > >> > > > > > > >
> > > > > > > > >> > > > > > >> > > > > > > > > Do folks agree that the
> stable
> > > SPI
> > > > > > > boundary
> > > > > > > > >> > > should be
> > > > > > > > >> > > > > > >> > > > > > > > metrics reporting/emitting, not
> > > metrics
> > > > > > > > >> > persistence?
> > > > > > > > >> > > > > > >> > > > > > > >
> > > > > > > > >> > > > > > >> > > > > > > > SGTM.
> > > > > > > > >> > > > > > >> > > > > > > >
> > > > > > > > >> > > > > > >> > > > > > > > > Does an optional Iceberg
> > metrics
> > > > > > > extension
> > > > > > > > >> API
> > > > > > > > >> > > layer
> > > > > > > > >> > > > > > sound
> > > > > > > > >> > > > > > >> > like
> > > > > > > > >> > > > > > >> > > > the
> > > > > > > > >> > > > > > >> > > > > > > right
> > > > > > > > >> > > > > > >> > > > > > > > home for this SPI?
> > > > > > > > >> > > > > > >> > > > > > > >
> > > > > > > > >> > > > > > >> > > > > > > > How is that different from
> > current
> > > > > > > > >> > > > > PolarisMetricsReporter?
> > > > > > > > >> > > > > > >> > > > > > > >
> > > > > > > > >> > > > > > >> > > > > > > > > Should the current durable
> > > metrics
> > > > > work
> > > > > > > be
> > > > > > > > >> > > reframed
> > > > > > > > >> > > > > as a
> > > > > > > > >> > > > > > >> > > durable
> > > > > > > > >> > > > > > >> > > > > > > > JDBC reference implementation
> of
> > > that
> > > > > > SPI?
> > > > > > > > >> > > > > > >> > > > > > > >
> > > > > > > > >> > > > > > >> > > > > > > > SGTM.
> > > > > > > > >> > > > > > >> > > > > > > >
> > > > > > > > >> > > > > > >> > > > > > > > > What is the smallest PR
> > sequence
> > > to
> > > > > get
> > > > > > > there
> > > > > > > > >> > > without
> > > > > > > > >> > > > > > >> > blocking
> > > > > > > > >> > > > > > >> > > > > > > > the current metrics work
> > > unnecessarily?
> > > > > > > > >> > > > > > >> > > > > > > >
> > > > > > > > >> > > > > > >> > > > > > > > Let's get
> > > > > > > > >> > >
> > > > > > > > >> >
> > > > > > > > >>
> > > > > > >
> > > > > >
> > > > >
> > >
> >
> https://urldefense.com/v3/__https://github.com/apache/polaris/pull/4397__;!!Iz9xO38YGHZK!4rnlVIqBQQQ6xpGe1PMV58edOS_cGrKoqdfhw4qSp_JQCavvEjaxXk0OwGPnxyEcO60AzKQK_XDtp7H3DdkFXcrXuu5jefU$
> > > > > > > > >> > > > > > >> merged
> > > > > > > > >> > > > > > >> > > > first.
> > > > > > > > >> > > > > > >> > > > > > > >
> > > > > > > > >> > > > > > >> > > > > > > > Then, I'd propose isolating the
> > > Metrics
> > > > > > > schema
> > > > > > > > >> > from
> > > > > > > > >> > > the
> > > > > > > > >> > > > > > >> > MetaStore
> > > > > > > > >> > > > > > >> > > > > > schema.
> > > > > > > > >> > > > > > >> > > > > > > > This will probably have some
> > ripple
> > > > > > effect
> > > > > > > into
> > > > > > > > >> > the
> > > > > > > > >> > > > > > >> bootstrap
> > > > > > > > >> > > > > > >> > > > > > workflows,
> > > > > > > > >> > > > > > >> > > > > > > so
> > > > > > > > >> > > > > > >> > > > > > > > it's not a trivial change.
> > > > > > > > >> > > > > > >> > > > > > > >
> > > > > > > > >> > > > > > >> > > > > > > > Then, let's reassess.
> > > > > > > > >> > > > > > >> > > > > > > >
> > > > > > > > >> > > > > > >> > > > > > > > Cheers,
> > > > > > > > >> > > > > > >> > > > > > > > Dmitri.
> > > > > > > > >> > > > > > >> > > > > > > >
> > > > > > > > >> > > > > > >> > > > > > > > On Wed, Jun 3, 2026 at 5:40 PM
> EJ
> > > Wang
> > > > > <
> > > > > > > > >> > > > > > >> > > > > ej.wang.dev.interest@gmail.com
> > > > > > > > >> > > > > > >> > > > > > >
> > > > > > > > >> > > > > > >> > > > > > > > wrote:
> > > > > > > > >> > > > > > >> > > > > > > >
> > > > > > > > >> > > > > > >> > > > > > > > > Hi folks,
> > > > > > > > >> > > > > > >> > > > > > > > >
> > > > > > > > >> > > > > > >> > > > > > > > > Thanks again for the
> discussion
> > > > > today.
> > > > > > I
> > > > > > > > >> updated
> > > > > > > > >> > > the
> > > > > > > > >> > > > > > sync
> > > > > > > > >> > > > > > >> doc
> > > > > > > > >> > > > > > >> > > > with
> > > > > > > > >> > > > > > >> > > > > > > notes
> > > > > > > > >> > > > > > >> > > > > > > > > from the third metrics
> > > architecture
> > > > > > sync:
> > > > > > > > >> > > > > > >> > > > > > > > >
> > > > > > > > >> > > > > > >> > > > > > > > >
> > > > > > > > >> > > > > > >> > > > > > > > >
> > > > > > > > >> > > > > > >> > > > > > > >
> > > > > > > > >> > > > > > >> > > > > > >
> > > > > > > > >> > > > > > >> > > > > >
> > > > > > > > >> > > > > > >> > > > >
> > > > > > > > >> > > > > > >> > > >
> > > > > > > > >> > > > > > >> > >
> > > > > > > > >> > > > > > >> >
> > > > > > > > >> > > > > > >>
> > > > > > > > >> > > > > >
> > > > > > > > >> > > > >
> > > > > > > > >> > >
> > > > > > > > >> >
> > > > > > > > >>
> > > > > > >
> > > > > >
> > > > >
> > >
> >
> https://urldefense.com/v3/__https://docs.google.com/document/d/100h7c4damrUzVuquYbBHM0EvA4LSWuW2IT2dN_7nYVA/edit?tab=t.k96s2xyqr5u1__;!!Iz9xO38YGHZK!4rnlVIqBQQQ6xpGe1PMV58edOS_cGrKoqdfhw4qSp_JQCavvEjaxXk0OwGPnxyEcO60AzKQK_XDtp7H3DdkFXcrXrqCL-QE$
> > > > > > > > >> > > > > > >> > > > > > > > >
> > > > > > > > >> > > > > > >> > > > > > > > > A few highlights from today’s
> > > > > > discussion:
> > > > > > > > >> > > > > > >> > > > > > > > >
> > > > > > > > >> > > > > > >> > > > > > > > >    - We clarified that
> Iceberg
> > > > > metrics
> > > > > > > > >> reporting
> > > > > > > > >> > > can
> > > > > > > > >> > > > > be
> > > > > > > > >> > > > > > >> > > > interpreted
> > > > > > > > >> > > > > > >> > > > > > > > either
> > > > > > > > >> > > > > > >> > > > > > > > >    as sync or async handling
> > from
> > > > > > > Polaris’s
> > > > > > > > >> > > > > perspective,
> > > > > > > > >> > > > > > >> so
> > > > > > > > >> > > > > > >> > > > Polaris
> > > > > > > > >> > > > > > >> > > > > > > > should
> > > > > > > > >> > > > > > >> > > > > > > > >    stay flexible as a
> platform
> > > > > instead
> > > > > > of
> > > > > > > > >> baking
> > > > > > > > >> > > one
> > > > > > > > >> > > > > > >> handling
> > > > > > > > >> > > > > > >> > > > model
> > > > > > > > >> > > > > > >> > > > > > > into
> > > > > > > > >> > > > > > >> > > > > > > > > the
> > > > > > > > >> > > > > > >> > > > > > > > >    REST API behavior.
> > > > > > > > >> > > > > > >> > > > > > > > >    - We aligned that the
> > built-in
> > > > > > > (battery)
> > > > > > > > >> > > behavior
> > > > > > > > >> > > > > for
> > > > > > > > >> > > > > > >> > > metrics
> > > > > > > > >> > > > > > >> > > > > > > emitting
> > > > > > > > >> > > > > > >> > > > > > > > >    can stay simple:
> > > no-op/log-only is
> > > > > > > enough
> > > > > > > > >> as
> > > > > > > > >> > > the
> > > > > > > > >> > > > > > >> default.
> > > > > > > > >> > > > > > >> > > > > > > > >    - Durable metrics
> > persistence
> > > > > should
> > > > > > > be
> > > > > > > > >> > > treated as
> > > > > > > > >> > > > > an
> > > > > > > > >> > > > > > >> > > > > > implementation
> > > > > > > > >> > > > > > >> > > > > > > > of
> > > > > > > > >> > > > > > >> > > > > > > > >    the metrics reporting
> path,
> > > not as
> > > > > > the
> > > > > > > > >> core
> > > > > > > > >> > SPI
> > > > > > > > >> > > > > > >> boundary.
> > > > > > > > >> > > > > > >> > > > > > > > >    - The existing durable
> > metrics
> > > > > work
> > > > > > > can be
> > > > > > > > >> > > reviewed
> > > > > > > > >> > > > > > as
> > > > > > > > >> > > > > > >> a
> > > > > > > > >> > > > > > >> > > > > reference
> > > > > > > > >> > > > > > >> > > > > > > > >    implementation of the
> > > reporting
> > > > > SPI,
> > > > > > > with
> > > > > > > > >> > > > > > >> > > persistence-related
> > > > > > > > >> > > > > > >> > > > > > logic
> > > > > > > > >> > > > > > >> > > > > > > > kept
> > > > > > > > >> > > > > > >> > > > > > > > >    self-contained under a
> > metrics
> > > > > > durable
> > > > > > > > >> > > > > implementation
> > > > > > > > >> > > > > > >> > module
> > > > > > > > >> > > > > > >> > > > > > rather
> > > > > > > > >> > > > > > >> > > > > > > > than
> > > > > > > > >> > > > > > >> > > > > > > > >    scattered through core
> > entity
> > > > > > > persistence.
> > > > > > > > >> > > > > > >> > > > > > > > >    - Dashboard/insights
> > remains a
> > > > > real
> > > > > > > use
> > > > > > > > >> case,
> > > > > > > > >> > > but
> > > > > > > > >> > > > > we
> > > > > > > > >> > > > > > >> > agreed
> > > > > > > > >> > > > > > >> > > to
> > > > > > > > >> > > > > > >> > > > > > keep
> > > > > > > > >> > > > > > >> > > > > > > it
> > > > > > > > >> > > > > > >> > > > > > > > >    separate from the core
> > metrics
> > > > > > intake
> > > > > > > > >> > > discussion
> > > > > > > > >> > > > > for
> > > > > > > > >> > > > > > >> now.
> > > > > > > > >> > > > > > >> > > > > > > > >
> > > > > > > > >> > > > > > >> > > > > > > > > I also did a quick source
> check
> > > after
> > > > > > the
> > > > > > > > >> > meeting
> > > > > > > > >> > > to
> > > > > > > > >> > > > > > make
> > > > > > > > >> > > > > > >> > sure
> > > > > > > > >> > > > > > >> > > we
> > > > > > > > >> > > > > > >> > > > > are
> > > > > > > > >> > > > > > >> > > > > > > > > describing the current state
> > > > > > accurately.
> > > > > > > > >> > > > > > >> > > > > > > > >
> > > > > > > > >> > > > > > >> > > > > > > > > Current state:
> > > > > > > > >> > > > > > >> > > > > > > > >
> > > > > > > > >> > > > > > >> > > > > > > > >    - Polaris already has a
> > > metrics
> > > > > > > reporting
> > > > > > > > >> > hook:
> > > > > > > > >> > > > > > >> > > > > > > > PolarisMetricsReporter.
> > > > > > > > >> > > > > > >> > > > > > > > >    - The default
> implementation
> > > is
> > > > > > > > >> > > > > > DefaultMetricsReporter,
> > > > > > > > >> > > > > > >> > > > selected
> > > > > > > > >> > > > > > >> > > > > > by
> > > > > > > > >> > > > > > >> > > > > > > > >
> > > > > > > > >> > > polaris.iceberg-metrics.reporting.type=default. It
> > > > > > > > >> > > > > is
> > > > > > > > >> > > > > > >> > > > log-only,
> > > > > > > > >> > > > > > >> > > > > > > > >    effectively quiet unless
> > > metrics
> > > > > > > logging
> > > > > > > > >> is
> > > > > > > > >> > > > > enabled.
> > > > > > > > >> > > > > > >> > > > > > > > >    - There is also a
> > > > > > > > >> PersistingMetricsReporter,
> > > > > > > > >> > > > > selected
> > > > > > > > >> > > > > > >> by
> > > > > > > > >> > > > > > >> > > > > > > > >
> > > > > > > > >> > > polaris.iceberg-metrics.reporting.type=persisting,
> > > > > > > > >> > > > > > >> which
> > > > > > > > >> > > > > > >> > > > > converts
> > > > > > > > >> > > > > > >> > > > > > > > >    Iceberg scan/commit
> reports
> > > into
> > > > > > > Polaris
> > > > > > > > >> > > metrics
> > > > > > > > >> > > > > > >> records
> > > > > > > > >> > > > > > >> > and
> > > > > > > > >> > > > > > >> > > > > > writes
> > > > > > > > >> > > > > > >> > > > > > > > > through
> > > > > > > > >> > > > > > >> > > > > > > > >    PolarisMetricsManager ->
> > > > > > > > >> MetricsPersistence.
> > > > > > > > >> > > > > > >> > > > > > > > >    - MetricsPersistence
> > currently
> > > > > > exists
> > > > > > > in
> > > > > > > > >> the
> > > > > > > > >> > > > > > >> persistence
> > > > > > > > >> > > > > > >> > > layer
> > > > > > > > >> > > > > > >> > > > > and
> > > > > > > > >> > > > > > >> > > > > > > is
> > > > > > > > >> > > > > > >> > > > > > > > >    inherited by
> > BasePersistence.
> > > > > > > > >> > > > > > >> > > > > > > > >
> > > > > > > > >> > > > > > >> > > > > > > > > My read from the discussion
> is
> > > that
> > > > > the
> > > > > > > > >> target
> > > > > > > > >> > > > > boundary
> > > > > > > > >> > > > > > >> > should
> > > > > > > > >> > > > > > >> > > > not
> > > > > > > > >> > > > > > >> > > > > be
> > > > > > > > >> > > > > > >> > > > > > > > > MetricsPersistence as
> inherited
> > > by
> > > > > > > > >> > > BasePersistence.
> > > > > > > > >> > > > > That
> > > > > > > > >> > > > > > >> path
> > > > > > > > >> > > > > > >> > > > > should
> > > > > > > > >> > > > > > >> > > > > > be
> > > > > > > > >> > > > > > >> > > > > > > > > decomposed as durable
> > > implementation
> > > > > > > detail
> > > > > > > > >> > (taken
> > > > > > > > >> > > > > care
> > > > > > > > >> > > > > > >> of by
> > > > > > > > >> > > > > > >> > > > > > > > >
> > > > > > > > >> >
> > > > > > > > >>
> > > > > > >
> > > > > >
> > > > >
> > >
> >
> https://urldefense.com/v3/__https://github.com/apache/polaris/pull/4397__;!!Iz9xO38YGHZK!4rnlVIqBQQQ6xpGe1PMV58edOS_cGrKoqdfhw4qSp_JQCavvEjaxXk0OwGPnxyEcO60AzKQK_XDtp7H3DdkFXcrXuu5jefU$
> > > > > > > > >> ).
> > > > > > > > >> > The
> > > > > > > > >> > > > > > stable
> > > > > > > > >> > > > > > >> > > > extension
> > > > > > > > >> > > > > > >> > > > > > > point
> > > > > > > > >> > > > > > >> > > > > > > > > should instead be the metrics
> > > > > > > > >> reporting/emitting
> > > > > > > > >> > > > > > boundary.
> > > > > > > > >> > > > > > >> > > > > > > > >
> > > > > > > > >> > > > > > >> > > > > > > > > Concretely, I think the next
> > > proposal
> > > > > > > should
> > > > > > > > >> be
> > > > > > > > >> > > shaped
> > > > > > > > >> > > > > > >> like
> > > > > > > > >> > > > > > >> > > this:
> > > > > > > > >> > > > > > >> > > > > > > > >
> > > > > > > > >> > > > > > >> > > > > > > > >    1.
> > > > > > > > >> > > > > > >> > > > > > > > >
> > > > > > > > >> > > > > > >> > > > > > > > >    Define the proper metrics
> > > > > reporting
> > > > > > > SPI
> > > > > > > > >> at an
> > > > > > > > >> > > > > > optional
> > > > > > > > >> > > > > > >> > > Iceberg
> > > > > > > > >> > > > > > >> > > > > > > metrics
> > > > > > > > >> > > > > > >> > > > > > > > >    extension API layer.
> > > > > > > > >> > > > > > >> > > > > > > > >
> > > > > > > > >> > > > > > >> > > > > > > > >    Example direction:
> > > > > > > > >> > > > > > >> > > > > > > > >
> > > > > > > > >> > > > > > >> > > > > > > > >    public interface
> > > > > > > IcebergMetricsReporter {
> > > > > > > > >> > > > > > >> > > > > > > > >      void
> > > > > > > > >> > reportMetric(IcebergMetricsReportContext
> > > > > > > > >> > > > > > >> context,
> > > > > > > > >> > > > > > >> > > > > > > > > MetricsReport report);
> > > > > > > > >> > > > > > >> > > > > > > > >    }
> > > > > > > > >> > > > > > >> > > > > > > > >
> > > > > > > > >> > > > > > >> > > > > > > > >    The context would carry
> the
> > > small
> > > > > > > > >> > > Polaris-resolved
> > > > > > > > >> > > > > > >> > envelope:
> > > > > > > > >> > > > > > >> > > > > > > > >    catalog/table identity,
> > report
> > > > > type,
> > > > > > > > >> received
> > > > > > > > >> > > > > > >> timestamp,
> > > > > > > > >> > > > > > >> > and
> > > > > > > > >> > > > > > >> > > > > > > > > request/trace
> > > > > > > > >> > > > > > >> > > > > > > > >    context if available. The
> > raw
> > > > > > Iceberg
> > > > > > > > >> > > MetricsReport
> > > > > > > > >> > > > > > >> > remains
> > > > > > > > >> > > > > > >> > > > the
> > > > > > > > >> > > > > > >> > > > > > > > payload.
> > > > > > > > >> > > > > > >> > > > > > > > >    2.
> > > > > > > > >> > > > > > >> > > > > > > > >
> > > > > > > > >> > > > > > >> > > > > > > > >    Keep runtime/service as
> the
> > > REST
> > > > > > > ingestion
> > > > > > > > >> > and
> > > > > > > > >> > > > > wiring
> > > > > > > > >> > > > > > >> > layer.
> > > > > > > > >> > > > > > >> > > > > > > > >
> > > > > > > > >> > > > > > >> > > > > > > > >    The REST handler still
> > > resolves
> > > > > the
> > > > > > > table
> > > > > > > > >> and
> > > > > > > > >> > > > > > performs
> > > > > > > > >> > > > > > >> > authz
> > > > > > > > >> > > > > > >> > > > > > before
> > > > > > > > >> > > > > > >> > > > > > > > >    accepting the report.
> After
> > > that,
> > > > > it
> > > > > > > calls
> > > > > > > > >> > the
> > > > > > > > >> > > > > > selected
> > > > > > > > >> > > > > > >> > > > > reporting
> > > > > > > > >> > > > > > >> > > > > > > > >    implementation.
> > > > > > > > >> > > > > > >> > > > > > > > >    3.
> > > > > > > > >> > > > > > >> > > > > > > > >
> > > > > > > > >> > > > > > >> > > > > > > > >    Keep the battery default
> > > > > > > no-op/log-only.
> > > > > > > > >> > > > > > >> > > > > > > > >
> > > > > > > > >> > > > > > >> > > > > > > > >    This preserves an
> out-of-box
> > > safe
> > > > > > > default
> > > > > > > > >> and
> > > > > > > > >> > > > > avoids
> > > > > > > > >> > > > > > >> > > > requiring a
> > > > > > > > >> > > > > > >> > > > > > > > durable
> > > > > > > > >> > > > > > >> > > > > > > > >    metrics store for every
> > > Polaris
> > > > > > > > >> deployment.
> > > > > > > > >> > > > > > >> > > > > > > > >    4.
> > > > > > > > >> > > > > > >> > > > > > > > >
> > > > > > > > >> > > > > > >> > > > > > > > >    Move durable JDBC metrics
> > > into a
> > > > > > > > >> > self-contained
> > > > > > > > >> > > > > > >> > > implementation
> > > > > > > > >> > > > > > >> > > > > of
> > > > > > > > >> > > > > > >> > > > > > > the
> > > > > > > > >> > > > > > >> > > > > > > > >    reporting SPI.
> > > > > > > > >> > > > > > >> > > > > > > > >
> > > > > > > > >> > > > > > >> > > > > > > > >    That implementation can
> own
> > > its
> > > > > > > schema,
> > > > > > > > >> > > bootstrap,
> > > > > > > > >> > > > > > >> > > retention,
> > > > > > > > >> > > > > > >> > > > > and
> > > > > > > > >> > > > > > >> > > > > > > read
> > > > > > > > >> > > > > > >> > > > > > > > >    API support. It should not
> > > define
> > > > > > the
> > > > > > > core
> > > > > > > > >> > > > > reporting
> > > > > > > > >> > > > > > >> SPI
> > > > > > > > >> > > > > > >> > > > > boundary,
> > > > > > > > >> > > > > > >> > > > > > > and
> > > > > > > > >> > > > > > >> > > > > > > > > it
> > > > > > > > >> > > > > > >> > > > > > > > >    should not require metrics
> > > > > > > persistence to
> > > > > > > > >> > > remain
> > > > > > > > >> > > > > > >> inherited
> > > > > > > > >> > > > > > >> > > > from
> > > > > > > > >> > > > > > >> > > > > > > > >    BasePersistence.
> > > > > > > > >> > > > > > >> > > > > > > > >    5.
> > > > > > > > >> > > > > > >> > > > > > > > >
> > > > > > > > >> > > > > > >> > > > > > > > >    Treat async/event-backed
> > > handling
> > > > > as
> > > > > > > > >> another
> > > > > > > > >> > > > > > >> > implementation
> > > > > > > > >> > > > > > >> > > of
> > > > > > > > >> > > > > > >> > > > > the
> > > > > > > > >> > > > > > >> > > > > > > > same
> > > > > > > > >> > > > > > >> > > > > > > > >    reporting SPI.
> > > > > > > > >> > > > > > >> > > > > > > > >
> > > > > > > > >> > > > > > >> > > > > > > > >    For example, an
> event-backed
> > > > > > reporter
> > > > > > > > >> could
> > > > > > > > >> > > > > enqueue a
> > > > > > > > >> > > > > > >> > > metrics
> > > > > > > > >> > > > > > >> > > > > > event
> > > > > > > > >> > > > > > >> > > > > > > > and
> > > > > > > > >> > > > > > >> > > > > > > > >    let listeners handle
> durable
> > > > > storage
> > > > > > > or
> > > > > > > > >> other
> > > > > > > > >> > > > > sinks.
> > > > > > > > >> > > > > > >> If we
> > > > > > > > >> > > > > > >> > > > later
> > > > > > > > >> > > > > > >> > > > > > > need
> > > > > > > > >> > > > > > >> > > > > > > > a
> > > > > > > > >> > > > > > >> > > > > > > > >    replaceable queue engine,
> > that
> > > > > seems
> > > > > > > like
> > > > > > > > >> a
> > > > > > > > >> > > shared
> > > > > > > > >> > > > > > >> > > > event/metrics
> > > > > > > > >> > > > > > >> > > > > > > > > substrate
> > > > > > > > >> > > > > > >> > > > > > > > >    topic rather than a
> > > metrics-only
> > > > > > > > >> requirement.
> > > > > > > > >> > > > > > >> > > > > > > > >
> > > > > > > > >> > > > > > >> > > > > > > > > This framing lets us keep the
> > > REST
> > > > > > > metrics
> > > > > > > > >> > > endpoint
> > > > > > > > >> > > > > > >> simple,
> > > > > > > > >> > > > > > >> > > > > preserve
> > > > > > > > >> > > > > > >> > > > > > > the
> > > > > > > > >> > > > > > >> > > > > > > > > current default behavior,
> > support
> > > > > > durable
> > > > > > > > >> > metrics
> > > > > > > > >> > > > > users,
> > > > > > > > >> > > > > > >> and
> > > > > > > > >> > > > > > >> > > > still
> > > > > > > > >> > > > > > >> > > > > > > leave
> > > > > > > > >> > > > > > >> > > > > > > > > room for async/event-backed
> or
> > > > > > > > >> > > external-queue-based
> > > > > > > > >> > > > > > >> > > > > implementations.
> > > > > > > > >> > > > > > >> > > > > > > > >
> > > > > > > > >> > > > > > >> > > > > > > > > I think the main follow-up
> > > questions
> > > > > > are:
> > > > > > > > >> > > > > > >> > > > > > > > >
> > > > > > > > >> > > > > > >> > > > > > > > >    - Do folks agree that the
> > > stable
> > > > > SPI
> > > > > > > > >> boundary
> > > > > > > > >> > > > > should
> > > > > > > > >> > > > > > be
> > > > > > > > >> > > > > > >> > > > metrics
> > > > > > > > >> > > > > > >> > > > > > > > >    reporting/emitting, not
> > > metrics
> > > > > > > > >> persistence?
> > > > > > > > >> > > > > > >> > > > > > > > >    - Does an optional Iceberg
> > > metrics
> > > > > > > > >> extension
> > > > > > > > >> > > API
> > > > > > > > >> > > > > > layer
> > > > > > > > >> > > > > > >> > sound
> > > > > > > > >> > > > > > >> > > > > like
> > > > > > > > >> > > > > > >> > > > > > > the
> > > > > > > > >> > > > > > >> > > > > > > > >    right home for this SPI?
> > > > > > > > >> > > > > > >> > > > > > > > >    - Should the current
> durable
> > > > > metrics
> > > > > > > work
> > > > > > > > >> be
> > > > > > > > >> > > > > reframed
> > > > > > > > >> > > > > > >> as a
> > > > > > > > >> > > > > > >> > > > > durable
> > > > > > > > >> > > > > > >> > > > > > > > JDBC
> > > > > > > > >> > > > > > >> > > > > > > > >    reference implementation
> of
> > > that
> > > > > > SPI?
> > > > > > > > >> > > > > > >> > > > > > > > >    - What is the smallest PR
> > > sequence
> > > > > > to
> > > > > > > get
> > > > > > > > >> > there
> > > > > > > > >> > > > > > without
> > > > > > > > >> > > > > > >> > > > blocking
> > > > > > > > >> > > > > > >> > > > > > the
> > > > > > > > >> > > > > > >> > > > > > > > >    current metrics work
> > > > > unnecessarily?
> > > > > > > > >> > > > > > >> > > > > > > > >
> > > > > > > > >> > > > > > >> > > > > > > > > Thanks,
> > > > > > > > >> > > > > > >> > > > > > > > > -ej
> > > > > > > > >> > > > > > >> > > > > > > > >
> > > > > > > > >> > > > > > >> > > > > > > > > On Fri, May 15, 2026 at
> 5:16 PM
> > > > > Dmitri
> > > > > > > > >> > > Bourlatchkov <
> > > > > > > > >> > > > > > >> > > > > > dimas@apache.org>
> > > > > > > > >> > > > > > >> > > > > > > > > wrote:
> > > > > > > > >> > > > > > >> > > > > > > > >
> > > > > > > > >> > > > > > >> > > > > > > > > > Hi JB,
> > > > > > > > >> > > > > > >> > > > > > > > > >
> > > > > > > > >> > > > > > >> > > > > > > > > > Could you set up another
> > > meeting,
> > > > > > > please?
> > > > > > > > >> Same
> > > > > > > > >> > > time
> > > > > > > > >> > > > > on
> > > > > > > > >> > > > > > >> > > > Wednesday
> > > > > > > > >> > > > > > >> > > > > as
> > > > > > > > >> > > > > > >> > > > > > > > last
> > > > > > > > >> > > > > > >> > > > > > > > > > time... I hope it works for
> > > > > everyone.
> > > > > > > > >> > > > > > >> > > > > > > > > >
> > > > > > > > >> > > > > > >> > > > > > > > > > Cheers,
> > > > > > > > >> > > > > > >> > > > > > > > > > Dmitri.
> > > > > > > > >> > > > > > >> > > > > > > > > >
> > > > > > > > >> > > > > > >> > > > > > > > > > On Fri, May 15, 2026 at
> > 8:06 PM
> > > > > Yufei
> > > > > > > Gu <
> > > > > > > > >> > > > > > >> > > flyrain000@gmail.com
> > > > > > > > >> > > > > > >> > > > >
> > > > > > > > >> > > > > > >> > > > > > > wrote:
> > > > > > > > >> > > > > > >> > > > > > > > > >
> > > > > > > > >> > > > > > >> > > > > > > > > > > +1 on another sync call
> > next
> > > > > week.
> > > > > > > > >> > > > > > >> > > > > > > > > > >
> > > > > > > > >> > > > > > >> > > > > > > > > > > Yufei
> > > > > > > > >> > > > > > >> > > > > > > > > > >
> > > > > > > > >> > > > > > >> > > > > > > > > > >
> > > > > > > > >> > > > > > >> > > > > > > > > > > On Fri, May 15, 2026 at
> > > 4:52 PM
> > > > > > > Dmitri
> > > > > > > > >> > > > > Bourlatchkov
> > > > > > > > >> > > > > > <
> > > > > > > > >> > > > > > >> > > > > > > > dimas@apache.org>
> > > > > > > > >> > > > > > >> > > > > > > > > > > wrote:
> > > > > > > > >> > > > > > >> > > > > > > > > > >
> > > > > > > > >> > > > > > >> > > > > > > > > > > > Hi All,
> > > > > > > > >> > > > > > >> > > > > > > > > > > >
> > > > > > > > >> > > > > > >> > > > > > > > > > > > WDYT about another sync
> > > call
> > > > > next
> > > > > > > week?
> > > > > > > > >> > > > > > >> > > > > > > > > > > >
> > > > > > > > >> > > > > > >> > > > > > > > > > > > Thanks,
> > > > > > > > >> > > > > > >> > > > > > > > > > > > Dmitri.
> > > > > > > > >> > > > > > >> > > > > > > > > > > >
> > > > > > > > >> > > > > > >> > > > > > > > > > > > On Wed, May 6, 2026 at
> > > 5:29 PM
> > > > > > > Dmitri
> > > > > > > > >> > > > > > Bourlatchkov <
> > > > > > > > >> > > > > > >> > > > > > > > dimas@apache.org
> > > > > > > > >> > > > > > >> > > > > > > > > >
> > > > > > > > >> > > > > > >> > > > > > > > > > > > wrote:
> > > > > > > > >> > > > > > >> > > > > > > > > > > >
> > > > > > > > >> > > > > > >> > > > > > > > > > > > > Hi EJ,
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >
> > > > > > > > >> > > > > > >> > > > > > > > > > > > > Thanks for the
> summary!
> > > It
> > > > > > covers
> > > > > > > > >> what
> > > > > > > > >> > we
> > > > > > > > >> > > > > > >> discussed
> > > > > > > > >> > > > > > >> > in
> > > > > > > > >> > > > > > >> > > > the
> > > > > > > > >> > > > > > >> > > > > > > > meeting
> > > > > > > > >> > > > > > >> > > > > > > > > > very
> > > > > > > > >> > > > > > >> > > > > > > > > > > > > well, IMHO.
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >
> > > > > > > > >> > > > > > >> > > > > > > > > > > > > Looking forward to
> > > concrete
> > > > > PRs
> > > > > > > :)
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >
> > > > > > > > >> > > > > > >> > > > > > > > > > > > > Cheers,
> > > > > > > > >> > > > > > >> > > > > > > > > > > > > Dmitri.
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >
> > > > > > > > >> > > > > > >> > > > > > > > > > > > > On Wed, May 6, 2026
> at
> > > > > 5:08 PM
> > > > > > EJ
> > > > > > > > >> Wang <
> > > > > > > > >> > > > > > >> > > > > > > > > >
> > ej.wang.dev.interest@gmail.com
> > > > > > > > >> > > > > > >> > > > > > > > > > > >
> > > > > > > > >> > > > > > >> > > > > > > > > > > > > wrote:
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> Hi folks,
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >>
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> We had a community
> > sync
> > > > > > earlier,
> > > > > > > > >> thanks
> > > > > > > > >> > > JB
> > > > > > > > >> > > > > for
> > > > > > > > >> > > > > > >> > > > scheduling
> > > > > > > > >> > > > > > >> > > > > > it.
> > > > > > > > >> > > > > > >> > > > > > > > > Notes
> > > > > > > > >> > > > > > >> > > > > > > > > > > from
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> the first metrics
> > > > > architecture
> > > > > > > sync
> > > > > > > > >> > (May
> > > > > > > > >> > > 6,
> > > > > > > > >> > > > > > >> 10-11am
> > > > > > > > >> > > > > > >> > > PT).
> > > > > > > > >> > > > > > >> > > > > > > > > Discussion
> > > > > > > > >> > > > > > >> > > > > > > > > > > doc
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> with per-section
> > status:
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >>
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >>
> > > > > > > > >> > > > > > >> > > > > > > > > > > >
> > > > > > > > >> > > > > > >> > > > > > > > > > >
> > > > > > > > >> > > > > > >> > > > > > > > > >
> > > > > > > > >> > > > > > >> > > > > > > > >
> > > > > > > > >> > > > > > >> > > > > > > >
> > > > > > > > >> > > > > > >> > > > > > >
> > > > > > > > >> > > > > > >> > > > > >
> > > > > > > > >> > > > > > >> > > > >
> > > > > > > > >> > > > > > >> > > >
> > > > > > > > >> > > > > > >> > >
> > > > > > > > >> > > > > > >> >
> > > > > > > > >> > > > > > >>
> > > > > > > > >> > > > > >
> > > > > > > > >> > > > >
> > > > > > > > >> > >
> > > > > > > > >> >
> > > > > > > > >>
> > > > > > >
> > > > > >
> > > > >
> > >
> >
> https://urldefense.com/v3/__https://docs.google.com/document/d/100h7c4damrUzVuquYbBHM0EvA4LSWuW2IT2dN_7nYVA/edit?tab=t.0__;!!Iz9xO38YGHZK!4rnlVIqBQQQ6xpGe1PMV58edOS_cGrKoqdfhw4qSp_JQCavvEjaxXk0OwGPnxyEcO60AzKQK_XDtp7H3DdkFXcrXgIddeUs$
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >>
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> *The meeting covered
> > > both
> > > > > > topics
> > > > > > > > >> from
> > > > > > > > >> > the
> > > > > > > > >> > > > > doc.
> > > > > > > > >> > > > > > >> > > > > > Direction-level
> > > > > > > > >> > > > > > >> > > > > > > > > > > alignment
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> was reached on the
> > > headline
> > > > > > > pieces;
> > > > > > > > >> > > details
> > > > > > > > >> > > > > > >> remain
> > > > > > > > >> > > > > > >> > for
> > > > > > > > >> > > > > > >> > > > PR
> > > > > > > > >> > > > > > >> > > > > > > review
> > > > > > > > >> > > > > > >> > > > > > > > > or
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> follow-up sessions.*
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >>
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> *Topic 1 —
> Persistence
> > > > > schema
> > > > > > > > >> redesign*
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> Idea-level alignment
> > on
> > > > > > > > >> consolidating
> > > > > > > > >> > > > > per-type
> > > > > > > > >> > > > > > >> > tables
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >>
> (scan_metrics_report,
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >>
> commit_metrics_report)
> > > into
> > > > > a
> > > > > > > single
> > > > > > > > >> > > > > > >> metrics_report
> > > > > > > > >> > > > > > >> > > > table.
> > > > > > > > >> > > > > > >> > > > > > The
> > > > > > > > >> > > > > > >> > > > > > > > > > > > motivating
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> cost is the surface
> > area
> > > > > added
> > > > > > > by
> > > > > > > > >> every
> > > > > > > > >> > > new
> > > > > > > > >> > > > > > >> metric
> > > > > > > > >> > > > > > >> > > type
> > > > > > > > >> > > > > > >> > > > > > today:
> > > > > > > > >> > > > > > >> > > > > > > > new
> > > > > > > > >> > > > > > >> > > > > > > > > > > > table,
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> SPI method, record
> > > class,
> > > > > > model,
> > > > > > > > >> > > converter,
> > > > > > > > >> > > > > > >> schema
> > > > > > > > >> > > > > > >> > > > > > migration.
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >>
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> Most schema details
> > are
> > > > > > > deferred to
> > > > > > > > >> the
> > > > > > > > >> > > > > schema
> > > > > > > > >> > > > > > >> PR. A
> > > > > > > > >> > > > > > >> > > few
> > > > > > > > >> > > > > > >> > > > > > > > specific
> > > > > > > > >> > > > > > >> > > > > > > > > > > points
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> came up:
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> •
> > >  metric_schema_version:
> > > > > > Yufei
> > > > > > > > >> > prefers
> > > > > > > > >> > > > > > dropping
> > > > > > > > >> > > > > > >> > it,
> > > > > > > > >> > > > > > >> > > > > since
> > > > > > > > >> > > > > > >> > > > > > > > there
> > > > > > > > >> > > > > > >> > > > > > > > > is
> > > > > > > > >> > > > > > >> > > > > > > > > > > no
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> spec-level concept
> of
> > > > > metrics
> > > > > > > > >> > versioning
> > > > > > > > >> > > > > today
> > > > > > > > >> > > > > > >> and
> > > > > > > > >> > > > > > >> > it
> > > > > > > > >> > > > > > >> > > is
> > > > > > > > >> > > > > > >> > > > > > hard
> > > > > > > > >> > > > > > >> > > > > > > to
> > > > > > > > >> > > > > > >> > > > > > > > > > > define
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> unilaterally. Robert
> > > prefers
> > > > > > > keeping
> > > > > > > > >> > it,
> > > > > > > > >> > > > > given
> > > > > > > > >> > > > > > >> IRC
> > > > > > > > >> > > > > > >> > v2
> > > > > > > > >> > > > > > >> > > is
> > > > > > > > >> > > > > > >> > > > > > > coming
> > > > > > > > >> > > > > > >> > > > > > > > > and
> > > > > > > > >> > > > > > >> > > > > > > > > > > the
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> schema should be
> > > considered
> > > > > > > against
> > > > > > > > >> its
> > > > > > > > >> > > > > likely
> > > > > > > > >> > > > > > >> > shape;
> > > > > > > > >> > > > > > >> > > > > Robert
> > > > > > > > >> > > > > > >> > > > > > > > also
> > > > > > > > >> > > > > > >> > > > > > > > > > > raised
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> how to differentiate
> > > various
> > > > > > > payload
> > > > > > > > >> > > formats
> > > > > > > > >> > > > > if
> > > > > > > > >> > > > > > >> any.
> > > > > > > > >> > > > > > >> > > > EJ's
> > > > > > > > >> > > > > > >> > > > > > read
> > > > > > > > >> > > > > > >> > > > > > > > is
> > > > > > > > >> > > > > > >> > > > > > > > > > that
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> this
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> is a two-way-door
> > > decision.
> > > > > We
> > > > > > > can
> > > > > > > > >> > start
> > > > > > > > >> > > > > > without
> > > > > > > > >> > > > > > >> the
> > > > > > > > >> > > > > > >> > > > > field,
> > > > > > > > >> > > > > > >> > > > > > > and
> > > > > > > > >> > > > > > >> > > > > > > > if
> > > > > > > > >> > > > > > >> > > > > > > > > > IRC
> > > > > > > > >> > > > > > >> > > > > > > > > > > > v2
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> changes the shape we
> > > would
> > > > > > > likely
> > > > > > > > >> roll
> > > > > > > > >> > a
> > > > > > > > >> > > > > > >> > corresponding
> > > > > > > > >> > > > > > >> > > > new
> > > > > > > > >> > > > > > >> > > > > > > > schema
> > > > > > > > >> > > > > > >> > > > > > > > > > > > anyway,
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> which is not
> > > particularly
> > > > > > > costly.
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> •   Payload format:
> > > Robert
> > > > > > > pointed
> > > > > > > > >> out
> > > > > > > > >> > > that
> > > > > > > > >> > > > > > >> future
> > > > > > > > >> > > > > > >> > > > formats
> > > > > > > > >> > > > > > >> > > > > > > > beyond
> > > > > > > > >> > > > > > >> > > > > > > > > > JSON
> > > > > > > > >> > > > > > >> > > > > > > > > > > > may
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> be worth supporting.
> > The
> > > > > exact
> > > > > > > > >> shape is
> > > > > > > > >> > > > > > deferred
> > > > > > > > >> > > > > > >> to
> > > > > > > > >> > > > > > >> > > the
> > > > > > > > >> > > > > > >> > > > > > schema
> > > > > > > > >> > > > > > >> > > > > > > > > > > > discussion.
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> •   Partition
> > strategy:
> > > > > Anand
> > > > > > > > >> suggested
> > > > > > > > >> > > > > monthly
> > > > > > > > >> > > > > > >> > > > > partitioning
> > > > > > > > >> > > > > > >> > > > > > > > based
> > > > > > > > >> > > > > > >> > > > > > > > > > on
> > > > > > > > >> > > > > > >> > > > > > > > > > > > his
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> experience as
> > > potentially
> > > > > > > helpful at
> > > > > > > > >> > > scale.
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >>
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> *Topic 2 — Where
> > metrics
> > > > > > > ingestion
> > > > > > > > >> and
> > > > > > > > >> > > > > storage
> > > > > > > > >> > > > > > >> > belong*
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> Idea-level alignment
> > > that
> > > > > > > metrics
> > > > > > > > >> > should
> > > > > > > > >> > > be a
> > > > > > > > >> > > > > > >> > > separated
> > > > > > > > >> > > > > > >> > > > > SPI
> > > > > > > > >> > > > > > >> > > > > > > from
> > > > > > > > >> > > > > > >> > > > > > > > > the
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> entity
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> persistence stack.
> Two
> > > > > reasons
> > > > > > > > >> > surfaced:
> > > > > > > > >> > > (a)
> > > > > > > > >> > > > > > >> > workloads
> > > > > > > > >> > > > > > >> > > > and
> > > > > > > > >> > > > > > >> > > > > > > > > > capability
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> requirements diverge
> > > enough
> > > > > > that
> > > > > > > > >> > coupling
> > > > > > > > >> > > > > them
> > > > > > > > >> > > > > > >> > creates
> > > > > > > > >> > > > > > >> > > > > > > > artificial
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> constraints, and (b)
> > > admin
> > > > > > > > >> experience
> > > > > > > > >> > > > > improves
> > > > > > > > >> > > > > > >> when
> > > > > > > > >> > > > > > >> > > > > metrics
> > > > > > > > >> > > > > > >> > > > > > > has
> > > > > > > > >> > > > > > >> > > > > > > > > its
> > > > > > > > >> > > > > > >> > > > > > > > > > > own
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> bootstrap,
> retention,
> > > and
> > > > > > > lifecycle.
> > > > > > > > >> > > Dmitri
> > > > > > > > >> > > > > > noted
> > > > > > > > >> > > > > > >> > > > Polaris
> > > > > > > > >> > > > > > >> > > > > > > being
> > > > > > > > >> > > > > > >> > > > > > > > a
> > > > > > > > >> > > > > > >> > > > > > > > > > > > platform
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> should have the
> > > flexibility
> > > > > to
> > > > > > > > >> support
> > > > > > > > >> > > > > > different
> > > > > > > > >> > > > > > >> > > > > persistence
> > > > > > > > >> > > > > > >> > > > > > > > > > backends
> > > > > > > > >> > > > > > >> > > > > > > > > > > > per
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> concern, and pointed
> > to
> > > a
> > > > > > > concrete
> > > > > > > > >> next
> > > > > > > > >> > > step
> > > > > > > > >> > > > > of
> > > > > > > > >> > > > > > >> > > > separating
> > > > > > > > >> > > > > > >> > > > > > the
> > > > > > > > >> > > > > > >> > > > > > > > > JDBC
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> bootstrap for
> metrics
> > > from
> > > > > the
> > > > > > > > >> > metastore
> > > > > > > > >> > > > > > >> bootstrap.
> > > > > > > > >> > > > > > >> > > > Robert
> > > > > > > > >> > > > > > >> > > > > > > > > proposed
> > > > > > > > >> > > > > > >> > > > > > > > > > an
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> additional UX
> > extension:
> > > > > > detect
> > > > > > > an
> > > > > > > > >> > > > > > unbootstrapped
> > > > > > > > >> > > > > > >> > > > metrics
> > > > > > > > >> > > > > > >> > > > > > > store
> > > > > > > > >> > > > > > >> > > > > > > > on
> > > > > > > > >> > > > > > >> > > > > > > > > > > first
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> use and
> auto-bootstrap
> > > > > rather
> > > > > > > than
> > > > > > > > >> > > requiring
> > > > > > > > >> > > > > an
> > > > > > > > >> > > > > > >> > > explicit
> > > > > > > > >> > > > > > >> > > > > > > manual
> > > > > > > > >> > > > > > >> > > > > > > > > > > > bootstrap
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> step.
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> The meeting also
> > > confirmed
> > > > > > that
> > > > > > > > >> Polaris
> > > > > > > > >> > > > > metrics
> > > > > > > > >> > > > > > >> can
> > > > > > > > >> > > > > > >> > > > start
> > > > > > > > >> > > > > > >> > > > > > > small
> > > > > > > > >> > > > > > >> > > > > > > > > and
> > > > > > > > >> > > > > > >> > > > > > > > > > > stay
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> Iceberg-focused.
> > Naming
> > > and
> > > > > > > > >> persistence
> > > > > > > > >> > > > > schema
> > > > > > > > >> > > > > > >> can
> > > > > > > > >> > > > > > >> > > lean
> > > > > > > > >> > > > > > >> > > > > > > > > > > > Iceberg-specific.
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> If a future
> expansion
> > to
> > > > > > > > >> generic-table
> > > > > > > > >> > > > > metrics
> > > > > > > > >> > > > > > or
> > > > > > > > >> > > > > > >> > > > > > operational
> > > > > > > > >> > > > > > >> > > > > > > > > > metrics
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> arrives, an
> > abstraction
> > > > > layer
> > > > > > > can be
> > > > > > > > >> > > built on
> > > > > > > > >> > > > > > >> top of
> > > > > > > > >> > > > > > >> > > the
> > > > > > > > >> > > > > > >> > > > > > > Iceberg
> > > > > > > > >> > > > > > >> > > > > > > > > > > metrics
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> reporter at that
> > point.
> > > > > Robert
> > > > > > > > >> remains
> > > > > > > > >> > > on the
> > > > > > > > >> > > > > > >> fence
> > > > > > > > >> > > > > > >> > > and
> > > > > > > > >> > > > > > >> > > > > > would
> > > > > > > > >> > > > > > >> > > > > > > > > prefer
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> something more
> generic
> > > but
> > > > > did
> > > > > > > not
> > > > > > > > >> > block
> > > > > > > > >> > > the
> > > > > > > > >> > > > > > >> > > direction;
> > > > > > > > >> > > > > > >> > > > > > > Dmitri's
> > > > > > > > >> > > > > > >> > > > > > > > > > read
> > > > > > > > >> > > > > > >> > > > > > > > > > > > was
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> that the proposed
> > > framework
> > > > > > > already
> > > > > > > > >> has
> > > > > > > > >> > > > > enough
> > > > > > > > >> > > > > > >> > > > flexibility
> > > > > > > > >> > > > > > >> > > > > > to
> > > > > > > > >> > > > > > >> > > > > > > > > absorb
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> future
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> expansion.
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >>
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> The Trade-offs and
> > > Proposed
> > > > > > > > >> structure
> > > > > > > > >> > > > > sections
> > > > > > > > >> > > > > > in
> > > > > > > > >> > > > > > >> > the
> > > > > > > > >> > > > > > >> > > > doc
> > > > > > > > >> > > > > > >> > > > > > were
> > > > > > > > >> > > > > > >> > > > > > > > not
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> reviewed
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> in detail. They
> remain
> > > open
> > > > > > for
> > > > > > > > >> either
> > > > > > > > >> > > the
> > > > > > > > >> > > > > next
> > > > > > > > >> > > > > > >> sync
> > > > > > > > >> > > > > > >> > > or
> > > > > > > > >> > > > > > >> > > > PR
> > > > > > > > >> > > > > > >> > > > > > > > review.
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >>
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> *Cross-cutting
> > > alignment —
> > > > > > > > >> > > battery-included
> > > > > > > > >> > > > > > plus
> > > > > > > > >> > > > > > >> > > > > pluggable*
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> A common philosophy
> > > emerged
> > > > > > > from the
> > > > > > > > >> > > > > > discussion.
> > > > > > > > >> > > > > > >> EJ
> > > > > > > > >> > > > > > >> > > > > > summarized
> > > > > > > > >> > > > > > >> > > > > > > > it
> > > > > > > > >> > > > > > >> > > > > > > > > > as:
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> Polaris should
> > provide a
> > > > > > > > >> > > battery-included UX
> > > > > > > > >> > > > > > for
> > > > > > > > >> > > > > > >> > > > beginners
> > > > > > > > >> > > > > > >> > > > > > and
> > > > > > > > >> > > > > > >> > > > > > > > the
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> flexibility for
> > advanced
> > > > > users
> > > > > > > to
> > > > > > > > >> swap
> > > > > > > > >> > > the
> > > > > > > > >> > > > > > >> included
> > > > > > > > >> > > > > > >> > > > > battery
> > > > > > > > >> > > > > > >> > > > > > > for
> > > > > > > > >> > > > > > >> > > > > > > > > > > > something
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> more powerful or
> > > tailored to
> > > > > > > their
> > > > > > > > >> use
> > > > > > > > >> > > case.
> > > > > > > > >> > > > > > The
> > > > > > > > >> > > > > > >> SPI
> > > > > > > > >> > > > > > >> > > > > design
> > > > > > > > >> > > > > > >> > > > > > > > needs
> > > > > > > > >> > > > > > >> > > > > > > > > to
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> enable
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> both.
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >>
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> The inputs that
> shaped
> > > this
> > > > > > > framing:
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> •   Anand described
> > how
> > > his
> > > > > > team
> > > > > > > > >> uses
> > > > > > > > >> > the
> > > > > > > > >> > > > > > current
> > > > > > > > >> > > > > > >> > > > metrics
> > > > > > > > >> > > > > > >> > > > > > > > > > persistence
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> (three metrics
> > > consumers in
> > > > > > > v1.4).
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> •   Yufei raised
> > > Grafana and
> > > > > > > > >> dashboard
> > > > > > > > >> > > > > > >> integrations
> > > > > > > > >> > > > > > >> > > as a
> > > > > > > > >> > > > > > >> > > > > > > > > destination
> > > > > > > > >> > > > > > >> > > > > > > > > > > use
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> case beyond the
> > default.
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> •   Robert called
> out
> > > that
> > > > > the
> > > > > > > > >> current
> > > > > > > > >> > > design
> > > > > > > > >> > > > > > is
> > > > > > > > >> > > > > > >> > more
> > > > > > > > >> > > > > > >> > > > > > > > > JDBC-focused.
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >>
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> Two concrete
> > instances:
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> •   Async metrics
> > > intake:
> > > > > > > Yufei's
> > > > > > > > >> > initial
> > > > > > > > >> > > > > > >> position
> > > > > > > > >> > > > > > >> > was
> > > > > > > > >> > > > > > >> > > > > that
> > > > > > > > >> > > > > > >> > > > > > > > async
> > > > > > > > >> > > > > > >> > > > > > > > > > > should
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> largely live on the
> > > producer
> > > > > > > side
> > > > > > > > >> and
> > > > > > > > >> > > there
> > > > > > > > >> > > > > is
> > > > > > > > >> > > > > > >> not
> > > > > > > > >> > > > > > >> > > much
> > > > > > > > >> > > > > > >> > > > > > > Polaris
> > > > > > > > >> > > > > > >> > > > > > > > > can
> > > > > > > > >> > > > > > >> > > > > > > > > > > do.
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> Robert suggested a
> > > > > > Polaris-side
> > > > > > > > >> default
> > > > > > > > >> > > is
> > > > > > > > >> > > > > > doable
> > > > > > > > >> > > > > > >> > via
> > > > > > > > >> > > > > > >> > > > > > Vert.x.
> > > > > > > > >> > > > > > >> > > > > > > > > Dmitri
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> agreed
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> the direction is
> worth
> > > > > > > exploring.
> > > > > > > > >> The
> > > > > > > > >> > > meeting
> > > > > > > > >> > > > > > >> > > converged
> > > > > > > > >> > > > > > >> > > > > on a
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> battery-included
> > default
> > > > > > (likely
> > > > > > > > >> > > > > Vert.x-backed)
> > > > > > > > >> > > > > > >> with
> > > > > > > > >> > > > > > >> > > an
> > > > > > > > >> > > > > > >> > > > > SPI
> > > > > > > > >> > > > > > >> > > > > > > > shape
> > > > > > > > >> > > > > > >> > > > > > > > > > that
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> lets
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> power users route
> to a
> > > more
> > > > > > > scalable
> > > > > > > > >> > > backend
> > > > > > > > >> > > > > > >> > > (k8s-hosted
> > > > > > > > >> > > > > > >> > > > > > > queue,
> > > > > > > > >> > > > > > >> > > > > > > > > AWS
> > > > > > > > >> > > > > > >> > > > > > > > > > > SQS,
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> etc.).
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> •   Pluggable
> > > destinations:
> > > > > > > > >> combining
> > > > > > > > >> > > Yufei's
> > > > > > > > >> > > > > > >> > > dashboard
> > > > > > > > >> > > > > > >> > > > > use
> > > > > > > > >> > > > > > >> > > > > > > case
> > > > > > > > >> > > > > > >> > > > > > > > > > with
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> Robert's
> JDBC-focused
> > > > > > call-out,
> > > > > > > the
> > > > > > > > >> > > meeting
> > > > > > > > >> > > > > > >> agreed
> > > > > > > > >> > > > > > >> > the
> > > > > > > > >> > > > > > >> > > > SPI
> > > > > > > > >> > > > > > >> > > > > > > > should
> > > > > > > > >> > > > > > >> > > > > > > > > be
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> structured for
> > multiple
> > > > > sinks
> > > > > > so
> > > > > > > > >> > > integrations
> > > > > > > > >> > > > > > >> become
> > > > > > > > >> > > > > > >> > > > impl
> > > > > > > > >> > > > > > >> > > > > > > > choices
> > > > > > > > >> > > > > > >> > > > > > > > > > > rather
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> than architectural
> > > changes.
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >>
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> The battery-included
> > > default
> > > > > > is
> > > > > > > most
> > > > > > > > >> > > likely
> > > > > > > > >> > > > > to
> > > > > > > > >> > > > > > >> use
> > > > > > > > >> > > > > > >> > the
> > > > > > > > >> > > > > > >> > > > > > > existing
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> JDBC-backed
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> approach.
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >>
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> *Direction
> (idea-level
> > > > > > > alignment)*
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> •   Single
> > > metrics_report
> > > > > > table
> > > > > > > > >> > > consolidating
> > > > > > > > >> > > > > > >> > per-type
> > > > > > > > >> > > > > > >> > > > > > > metrics,
> > > > > > > > >> > > > > > >> > > > > > > > > > > > replacing
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> scan_metrics_report
> > and
> > > > > > > > >> > > commit_metrics_report
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> •   Iceberg-focused
> > > naming
> > > > > and
> > > > > > > > >> schema
> > > > > > > > >> > for
> > > > > > > > >> > > > > now,
> > > > > > > > >> > > > > > >> > revisit
> > > > > > > > >> > > > > > >> > > > if
> > > > > > > > >> > > > > > >> > > > > > > > > > > generic-table
> > > > > > > > >> > > > > > >> > > > > > > > > > > > or
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> operational metrics
> > > arrive
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> •   Metrics
> > persistence
> > > as a
> > > > > > > > >> separated
> > > > > > > > >> > > SPI,
> > > > > > > > >> > > > > not
> > > > > > > > >> > > > > > >> on
> > > > > > > > >> > > > > > >> > > > > > > > BasePersistence
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> •   Bootstrap path
> > > separated
> > > > > > for
> > > > > > > > >> > metrics,
> > > > > > > > >> > > > > > >> > independent
> > > > > > > > >> > > > > > >> > > of
> > > > > > > > >> > > > > > >> > > > > > > > metastore
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> bootstrap
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> •
>  "Battery-included
> > > plus
> > > > > > > > >> pluggable"
> > > > > > > > >> > as
> > > > > > > > >> > > the
> > > > > > > > >> > > > > > SPI
> > > > > > > > >> > > > > > >> > > design
> > > > > > > > >> > > > > > >> > > > > > > > philosophy
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >>
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> *Open items*
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> •   Schema details:
> > > > > > > > >> > > metric_schema_version,
> > > > > > > > >> > > > > > >> payload
> > > > > > > > >> > > > > > >> > > > format,
> > > > > > > > >> > > > > > >> > > > > > IRC
> > > > > > > > >> > > > > > >> > > > > > > > v2
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> forward-compat shape
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> •   SPI design
> > details —
> > > > > full
> > > > > > > review
> > > > > > > > >> > > either
> > > > > > > > >> > > > > in
> > > > > > > > >> > > > > > >> the
> > > > > > > > >> > > > > > >> > > next
> > > > > > > > >> > > > > > >> > > > > sync
> > > > > > > > >> > > > > > >> > > > > > > or
> > > > > > > > >> > > > > > >> > > > > > > > in
> > > > > > > > >> > > > > > >> > > > > > > > > > the
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> corresponding PR
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> •   Schema refactor
> PR
> > > > > > ownership
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >>
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> *Action items*
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> •   EJ to take a
> first
> > > stab
> > > > > at
> > > > > > > the
> > > > > > > > >> SPI
> > > > > > > > >> > > design
> > > > > > > > >> > > > > > and
> > > > > > > > >> > > > > > >> > > > > > potentially
> > > > > > > > >> > > > > > >> > > > > > > > > > partner
> > > > > > > > >> > > > > > >> > > > > > > > > > > > with
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> Anand to incorporate
> > the
> > > > > > lessons
> > > > > > > > >> > learned
> > > > > > > > >> > > from
> > > > > > > > >> > > > > > the
> > > > > > > > >> > > > > > >> > > > existing
> > > > > > > > >> > > > > > >> > > > > > > > > reporter
> > > > > > > > >> > > > > > >> > > > > > > > > > > and
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> persistence work.
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> •   Schema refactor
> PR
> > > > > > > ownership is
> > > > > > > > >> not
> > > > > > > > >> > > yet
> > > > > > > > >> > > > > > >> decided.
> > > > > > > > >> > > > > > >> > > If
> > > > > > > > >> > > > > > >> > > > > > anyone
> > > > > > > > >> > > > > > >> > > > > > > > is
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> interested in
> driving
> > > it,
> > > > > > reply
> > > > > > > on
> > > > > > > > >> this
> > > > > > > > >> > > > > thread.
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> •   JB to schedule
> the
> > > next
> > > > > > > sync,
> > > > > > > > >> > > tentatively
> > > > > > > > >> > > > > > in
> > > > > > > > >> > > > > > >> two
> > > > > > > > >> > > > > > >> > > > > weeks.
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >>
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> -ej
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >>
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> On Mon, Apr 27, 2026
> > at
> > > > > > 3:07 PM
> > > > > > > EJ
> > > > > > > > >> > Wang <
> > > > > > > > >> > > > > > >> > > > > > > > > > >
> > > ej.wang.dev.interest@gmail.com
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> wrote:
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >>
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> > Thanks Yufei for
> the
> > > +1.
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> > JB, could you help
> > > add a
> > > > > > > biweekly
> > > > > > > > >> > > metrics
> > > > > > > > >> > > > > > >> > > architecture
> > > > > > > > >> > > > > > >> > > > > > sync
> > > > > > > > >> > > > > > >> > > > > > > to
> > > > > > > > >> > > > > > >> > > > > > > > > the
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> Polaris
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> > community
> calendar?
> > > I'm
> > > > > > > thinking
> > > > > > > > >> > > Thursdays
> > > > > > > > >> > > > > at
> > > > > > > > >> > > > > > >> > 9-10am
> > > > > > > > >> > > > > > >> > > > PT,
> > > > > > > > >> > > > > > >> > > > > > on
> > > > > > > > >> > > > > > >> > > > > > > > the
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> off-weeks
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> > from the community
> > > meeting
> > > > > > > > >> (starting
> > > > > > > > >> > > May
> > > > > > > > >> > > > > 7),
> > > > > > > > >> > > > > > 60
> > > > > > > > >> > > > > > >> > > > minutes.
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> > Here's a rough
> > agenda
> > > to
> > > > > > work
> > > > > > > > >> through
> > > > > > > > >> > > over
> > > > > > > > >> > > > > > the
> > > > > > > > >> > > > > > >> > first
> > > > > > > > >> > > > > > >> > > > few
> > > > > > > > >> > > > > > >> > > > > > > > > sessions,
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> grouped
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> > by priority:
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> > *First:
> foundational
> > > > > > > direction*
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> > 1.
> > > MetricsPersistence:
> > > > > > public
> > > > > > > > >> SPI or
> > > > > > > > >> > > > > > internal
> > > > > > > > >> > > > > > >> > > > > > > implementation
> > > > > > > > >> > > > > > >> > > > > > > > > > > detail?
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >    •   Marked
> @Beta,
> > > > > javadoc
> > > > > > > calls
> > > > > > > > >> > it a
> > > > > > > > >> > > > > > >> "Service
> > > > > > > > >> > > > > > >> > > > > Provider
> > > > > > > > >> > > > > > >> > > > > > > > > > > Interface",
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> but
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> > only one consumer
> > > > > > > > >> > > > > (JdbcBasePersistenceImpl),
> > > > > > > > >> > > > > > >> lives
> > > > > > > > >> > > > > > >> > > on
> > > > > > > > >> > > > > > >> > > > > > > > > > > BasePersistence.
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> If
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> > demoted to a
> private
> > > > > helper
> > > > > > > > >> inside a
> > > > > > > > >> > > > > > persisting
> > > > > > > > >> > > > > > >> > > > reporter
> > > > > > > > >> > > > > > >> > > > > > > impl,
> > > > > > > > >> > > > > > >> > > > > > > > > > most
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> > downstream design
> > > > > decisions
> > > > > > > become
> > > > > > > > >> > > > > > >> implementation
> > > > > > > > >> > > > > > >> > > > > details
> > > > > > > > >> > > > > > >> > > > > > > > rather
> > > > > > > > >> > > > > > >> > > > > > > > > > > than
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> > contract
> questions.
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> > 2.  Persistence
> > schema
> > > > > > > redesign
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >    •   Current
> > > two-table
> > > > > > > layout
> > > > > > > > >> > > > > > >> > > (scan_metrics_report,
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >
> > commit_metrics_report)
> > > > > with
> > > > > > > ~25
> > > > > > > > >> > > flattened
> > > > > > > > >> > > > > > >> columns
> > > > > > > > >> > > > > > >> > > > each.
> > > > > > > > >> > > > > > >> > > > > > > Every
> > > > > > > > >> > > > > > >> > > > > > > > > new
> > > > > > > > >> > > > > > >> > > > > > > > > > > > metric
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> > type requires a
> new
> > > table,
> > > > > > SPI
> > > > > > > > >> > method,
> > > > > > > > >> > > > > record
> > > > > > > > >> > > > > > >> > class,
> > > > > > > > >> > > > > > >> > > > > > model,
> > > > > > > > >> > > > > > >> > > > > > > > > > > converter,
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> and
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> > schema migration.
> > > > > Direction
> > > > > > to
> > > > > > > > >> > explore:
> > > > > > > > >> > > > > > single
> > > > > > > > >> > > > > > >> > table
> > > > > > > > >> > > > > > >> > > > > with
> > > > > > > > >> > > > > > >> > > > > > > > > > > metric_type
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> enum,
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> > schema_version,
> and
> > > JSON
> > > > > > > payload
> > > > > > > > >> > > column.
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> > *Second: design
> > > details
> > > > > once
> > > > > > > > >> > direction
> > > > > > > > >> > > is
> > > > > > > > >> > > > > > set*
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> > 3.  Partition key
> > > strategy
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >    •
>  Single-table
> > > design
> > > > > > > means
> > > > > > > > >> scan
> > > > > > > > >> > > > > metrics
> > > > > > > > >> > > > > > >> at
> > > > > > > > >> > > > > > >> > > scale
> > > > > > > > >> > > > > > >> > > > > > will
> > > > > > > > >> > > > > > >> > > > > > > > have
> > > > > > > > >> > > > > > >> > > > > > > > > > > high
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> > write concurrency
> > per
> > > > > table.
> > > > > > > > >> Schema
> > > > > > > > >> > > needs
> > > > > > > > >> > > > > to
> > > > > > > > >> > > > > > >> > expose
> > > > > > > > >> > > > > > >> > > > > enough
> > > > > > > > >> > > > > > >> > > > > > > > > > structure
> > > > > > > > >> > > > > > >> > > > > > > > > > > > for
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> > backends to shard
> by
> > > > > entity
> > > > > > or
> > > > > > > > >> time
> > > > > > > > >> > > range.
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> > 4.  Read/write
> path
> > > > > > > consistency
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >    •   Writes go
> > > through
> > > > > > > > >> > > > > > PolarisMetricsManager
> > > > > > > > >> > > > > > >> on
> > > > > > > > >> > > > > > >> > > > > > > > > > MetaStoreManager.
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> Reads
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> > bypass
> > > MetaStoreManager
> > > > > and
> > > > > > go
> > > > > > > > >> > > straight to
> > > > > > > > >> > > > > > >> > > > > > BasePersistence,
> > > > > > > > >> > > > > > >> > > > > > > > > > > excluding
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> > non-JDBC backends
> > > from the
> > > > > > > read
> > > > > > > > >> API.
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> > *Third: cleanup
> and
> > > > > > alignment*
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> > 5.
> > > PolarisMetricsReporter
> > > > > > > naming
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >    •   Only
> handles
> > > IRC
> > > > > > > > >> > > > > > >> (ScanReport/CommitReport),
> > > > > > > > >> > > > > > >> > > > > doesn't
> > > > > > > > >> > > > > > >> > > > > > > > cover
> > > > > > > > >> > > > > > >> > > > > > > > > > > > generic
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> > tables or
> > operational
> > > > > > metrics.
> > > > > > > > >> Name
> > > > > > > > >> > is
> > > > > > > > >> > > > > > broader
> > > > > > > > >> > > > > > >> > than
> > > > > > > > >> > > > > > >> > > > > scope.
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> > 6.
> > > PolarisMetricsManager
> > > > > > > facade
> > > > > > > > >> > > > > passthrough
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >    •   Entire
> > default
> > > > > method
> > > > > > > is
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >>
> > > > > > > > >> > callCtx.getMetaStore().writeScanReport().
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> > Zero logic, passes
> > > Level 1
> > > > > > > > >> straight
> > > > > > > > >> > > through
> > > > > > > > >> > > > > > to
> > > > > > > > >> > > > > > >> > Level
> > > > > > > > >> > > > > > >> > > > 3.
> > > > > > > > >> > > > > > >> > > > > > Same
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> anti-pattern
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> > as
> > > PolarisEventManager.
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> > 7.  Iceberg
> > community
> > > > > > > alignment
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >    •
>  Payload-type
> > > > > > extension
> > > > > > > > >> needs
> > > > > > > > >> > > > > > discussion
> > > > > > > > >> > > > > > >> on
> > > > > > > > >> > > > > > >> > > > > > > dev@iceberg.
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> obelix74's
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> > Feb thread got
> zero
> > > > > replies.
> > > > > > > > >> Needs a
> > > > > > > > >> > > > > > committer
> > > > > > > > >> > > > > > >> > > voice.
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> > Lets confirm
> > > > > prioritization
> > > > > > > in the
> > > > > > > > >> > > first
> > > > > > > > >> > > > > > >> session.
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> > -ej
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> > On Tue, Apr 21,
> 2026
> > > at
> > > > > > > 3:18 PM
> > > > > > > > >> Yufei
> > > > > > > > >> > > Gu <
> > > > > > > > >> > > > > > >> > > > > > > > flyrain000@gmail.com>
> > > > > > > > >> > > > > > >> > > > > > > > > > > > wrote:
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> Thanks everyone
> for
> > > > > > > continuing to
> > > > > > > > >> > > drive
> > > > > > > > >> > > > > this
> > > > > > > > >> > > > > > >> > > > forward. I
> > > > > > > > >> > > > > > >> > > > > > > agree
> > > > > > > > >> > > > > > >> > > > > > > > > > that
> > > > > > > > >> > > > > > >> > > > > > > > > > > > the
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> problem is
> getting
> > > > > complex
> > > > > > > enough
> > > > > > > > >> > > that a
> > > > > > > > >> > > > > > more
> > > > > > > > >> > > > > > >> > > > > structured
> > > > > > > > >> > > > > > >> > > > > > > > > > discussion
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> would
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> help.
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >>
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> +1 on setting up
> a
> > > > > biweekly
> > > > > > > sync
> > > > > > > > >> for
> > > > > > > > >> > > the
> > > > > > > > >> > > > > > >> metrics
> > > > > > > > >> > > > > > >> > > > > > > > architecture.
> > > > > > > > >> > > > > > >> > > > > > > > > > I’m
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> happy
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> to
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> join.
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >>
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> Yufei
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >>
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >>
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> On Tue, Apr 21,
> > 2026
> > > at
> > > > > > > 2:34 PM
> > > > > > > > >> EJ
> > > > > > > > >> > > Wang <
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >>
> > > > > > ej.wang.dev.interest@gmail.com>
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> wrote:
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >>
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> > Also, I've been
> > > looking
> > > > > > > more
> > > > > > > > >> > > closely at
> > > > > > > > >> > > > > > the
> > > > > > > > >> > > > > > >> > > > > > *persistence
> > > > > > > > >> > > > > > >> > > > > > > > > schema
> > > > > > > > >> > > > > > >> > > > > > > > > > > in
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> the
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> > current metrics
> > > work*,
> > > > > > and
> > > > > > > I
> > > > > > > > >> think
> > > > > > > > >> > > > > > there's a
> > > > > > > > >> > > > > > >> > > > > structural
> > > > > > > > >> > > > > > >> > > > > > > > > > rigidity
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> problem
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> > worth raising
> > > before
> > > > > the
> > > > > > > shape
> > > > > > > > >> > gets
> > > > > > > > >> > > > > locked
> > > > > > > > >> > > > > > >> in.
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> > Right now we
> have
> > > two
> > > > > > > separate
> > > > > > > > >> > > tables
> > > > > > > > >> > > > > > >> > > > > > > (scan_metrics_report
> > > > > > > > >> > > > > > >> > > > > > > > > and
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >
> > > commit_metrics_report),
> > > > > > > each
> > > > > > > > >> with
> > > > > > > > >> > > ~25
> > > > > > > > >> > > > > > >> flattened
> > > > > > > > >> > > > > > >> > > > > columns
> > > > > > > > >> > > > > > >> > > > > > > > that
> > > > > > > > >> > > > > > >> > > > > > > > > > > > directly
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> > mirror the
> > Iceberg
> > > > > report
> > > > > > > > >> fields.
> > > > > > > > >> > > The
> > > > > > > > >> > > > > SPI
> > > > > > > > >> > > > > > >> > follows
> > > > > > > > >> > > > > > >> > > > the
> > > > > > > > >> > > > > > >> > > > > > > same
> > > > > > > > >> > > > > > >> > > > > > > > > > split:
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> > writeScanReport
> > and
> > > > > > > > >> > > writeCommitReport as
> > > > > > > > >> > > > > > >> > separate
> > > > > > > > >> > > > > > >> > > > > > > methods,
> > > > > > > > >> > > > > > >> > > > > > > > > with
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> per-type
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> > record classes,
> > > > > > > converters, and
> > > > > > > > >> > > model
> > > > > > > > >> > > > > > >> objects.
> > > > > > > > >> > > > > > >> > > *The
> > > > > > > > >> > > > > > >> > > > > > > > practical
> > > > > > > > >> > > > > > >> > > > > > > > > > > cost:
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> > adding a new
> > metric
> > > > > type
> > > > > > > > >> > > (operational
> > > > > > > > >> > > > > > >> metrics,
> > > > > > > > >> > > > > > >> > > for
> > > > > > > > >> > > > > > >> > > > > > > example)
> > > > > > > > >> > > > > > >> > > > > > > > > > > > requires
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> a
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> new
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> > table, a new
> SPI
> > > > > method,
> > > > > > a
> > > > > > > new
> > > > > > > > >> > > record
> > > > > > > > >> > > > > > >> class, a
> > > > > > > > >> > > > > > >> > > new
> > > > > > > > >> > > > > > >> > > > > > model
> > > > > > > > >> > > > > > >> > > > > > > > > > class, a
> > > > > > > > >> > > > > > >> > > > > > > > > > > > new
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> > converter
> branch,
> > > and a
> > > > > > > schema
> > > > > > > > >> > > > > migration*.
> > > > > > > > >> > > > > > >> > > That's a
> > > > > > > > >> > > > > > >> > > > > lot
> > > > > > > > >> > > > > > >> > > > > > > of
> > > > > > > > >> > > > > > >> > > > > > > > > > > surface
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> area
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> > for what should
> > be
> > > "one
> > > > > > > more
> > > > > > > > >> kind
> > > > > > > > >> > of
> > > > > > > > >> > > > > > >> metric."
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> > *My bias* would
> > be
> > > > > > toward a
> > > > > > > > >> single
> > > > > > > > >> > > > > metrics
> > > > > > > > >> > > > > > >> > table
> > > > > > > > >> > > > > > >> > > > with
> > > > > > > > >> > > > > > >> > > > > > *a
> > > > > > > > >> > > > > > >> > > > > > > > > typed
> > > > > > > > >> > > > > > >> > > > > > > > > > > JSON
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> > payload*.
> > Something
> > > > > like:
> > > > > > > > >> > > metric_type
> > > > > > > > >> > > > > > >> (enum),
> > > > > > > > >> > > > > > >> > > > > > entity_id,
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >
> table_identifier,
> > > > > > > snapshot_id
> > > > > > > > >> > > > > (nullable),
> > > > > > > > >> > > > > > >> > > > > received_ts,
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> schema_version,
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> and
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> > a payload
> column
> > > for
> > > > > the
> > > > > > > > >> > > metric-specific
> > > > > > > > >> > > > > > >> data.
> > > > > > > > >> > > > > > >> > > The
> > > > > > > > >> > > > > > >> > > > > > > > > metric_type
> > > > > > > > >> > > > > > >> > > > > > > > > > +
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> > schema_version
> > pair
> > > > > gives
> > > > > > > us a
> > > > > > > > >> > > > > > >> > forward-compatible
> > > > > > > > >> > > > > > >> > > > > > > contract
> > > > > > > > >> > > > > > >> > > > > > > > > for
> > > > > > > > >> > > > > > >> > > > > > > > > > > the
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> payload
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> > shape. Adding a
> > new
> > > > > > metric
> > > > > > > type
> > > > > > > > >> > > becomes
> > > > > > > > >> > > > > an
> > > > > > > > >> > > > > > >> enum
> > > > > > > > >> > > > > > >> > > > value
> > > > > > > > >> > > > > > >> > > > > > > and a
> > > > > > > > >> > > > > > >> > > > > > > > > > > payload
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> schema,
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> > not a schema
> > > migration.
> > > > > > One
> > > > > > > > >> thing
> > > > > > > > >> > I
> > > > > > > > >> > > > > think
> > > > > > > > >> > > > > > we
> > > > > > > > >> > > > > > >> > need
> > > > > > > > >> > > > > > >> > > > to
> > > > > > > > >> > > > > > >> > > > > be
> > > > > > > > >> > > > > > >> > > > > > > > > > > deliberate
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> about is
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> > the partition
> key
> > > > > design.
> > > > > > > If
> > > > > > > > >> all
> > > > > > > > >> > > metric
> > > > > > > > >> > > > > > >> types
> > > > > > > > >> > > > > > >> > > land
> > > > > > > > >> > > > > > >> > > > in
> > > > > > > > >> > > > > > >> > > > > > one
> > > > > > > > >> > > > > > >> > > > > > > > > > table,
> > > > > > > > >> > > > > > >> > > > > > > > > > > > scan
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> > metrics at
> scale
> > > (high
> > > > > > > > >> > concurrency,
> > > > > > > > >> > > high
> > > > > > > > >> > > > > > >> > > frequency
> > > > > > > > >> > > > > > >> > > > > > across
> > > > > > > > >> > > > > > >> > > > > > > > > many
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> tables)
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> > could easily
> > > create hot
> > > > > > > > >> > partitions.
> > > > > > > > >> > > We'd
> > > > > > > > >> > > > > > >> want
> > > > > > > > >> > > > > > >> > the
> > > > > > > > >> > > > > > >> > > > > > > > persistence
> > > > > > > > >> > > > > > >> > > > > > > > > > > layer
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> to
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> be
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> > able to shard
> by
> > > entity
> > > > > > or
> > > > > > > time
> > > > > > > > >> > > range,
> > > > > > > > >> > > > > and
> > > > > > > > >> > > > > > >> that
> > > > > > > > >> > > > > > >> > > > means
> > > > > > > > >> > > > > > >> > > > > > the
> > > > > > > > >> > > > > > >> > > > > > > > > > logical
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> schema
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> > needs to expose
> > > enough
> > > > > > > > >> structure
> > > > > > > > >> > for
> > > > > > > > >> > > > > > >> backends
> > > > > > > > >> > > > > > >> > to
> > > > > > > > >> > > > > > >> > > > > > > partition
> > > > > > > > >> > > > > > >> > > > > > > > > on.
> > > > > > > > >> > > > > > >> > > > > > > > > > I
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> don't
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> > think the
> current
> > > > > > flattened
> > > > > > > > >> layout
> > > > > > > > >> > > gives
> > > > > > > > >> > > > > > us
> > > > > > > > >> > > > > > >> > that.
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> > This is getting
> > > complex
> > > > > > > enough
> > > > > > > > >> > that
> > > > > > > > >> > > I
> > > > > > > > >> > > > > > don't
> > > > > > > > >> > > > > > >> > think
> > > > > > > > >> > > > > > >> > > > > > ad-hoc
> > > > > > > > >> > > > > > >> > > > > > > > > PR/ML
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> threads
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> > will converge
> > well.
> > > > > > *Would
> > > > > > > > >> people
> > > > > > > > >> > be
> > > > > > > > >> > > > > open
> > > > > > > > >> > > > > > >> to a
> > > > > > > > >> > > > > > >> > > > > biweekly
> > > > > > > > >> > > > > > >> > > > > > > > sync
> > > > > > > > >> > > > > > >> > > > > > > > > > for
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> metrics
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> > architecture?*
> I
> > > think
> > > > > 30
> > > > > > > > >> minutes
> > > > > > > > >> > > every
> > > > > > > > >> > > > > > two
> > > > > > > > >> > > > > > >> > weeks
> > > > > > > > >> > > > > > >> > > > > with
> > > > > > > > >> > > > > > >> > > > > > > > > > interested
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> parties
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> > would be enough
> > to
> > > work
> > > > > > > through
> > > > > > > > >> > the
> > > > > > > > >> > > > > > schema,
> > > > > > > > >> > > > > > >> SPI
> > > > > > > > >> > > > > > >> > > > > shape,
> > > > > > > > >> > > > > > >> > > > > > > and
> > > > > > > > >> > > > > > >> > > > > > > > > read
> > > > > > > > >> > > > > > >> > > > > > > > > > > API
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> design
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> > together. Happy
> > to
> > > help
> > > > > > set
> > > > > > > > >> that
> > > > > > > > >> > up.
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> > -ej
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> > On Mon, Apr 20,
> > > 2026 at
> > > > > > > > >> 2:19 PM EJ
> > > > > > > > >> > > Wang
> > > > > > > > >> > > > > <
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >>
> > > > > > ej.wang.dev.interest@gmail.com
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> > wrote:
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >> Reviewed
> #4115,
> > > left a
> > > > > > > > >> comment on
> > > > > > > > >> > > the
> > > > > > > > >> > > > > > code
> > > > > > > > >> > > > > > >> > > > > > organization
> > > > > > > > >> > > > > > >> > > > > > > > > side.
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >> One thing
> stood
> > > out:
> > > > > the
> > > > > > > > >> metrics
> > > > > > > > >> > > write
> > > > > > > > >> > > > > > path
> > > > > > > > >> > > > > > >> > > enters
> > > > > > > > >> > > > > > >> > > > > > > through
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>
> > > PolarisMetricsManager
> > > > > on
> > > > > > > > >> > > > > > MetaStoreManager,
> > > > > > > > >> > > > > > >> but
> > > > > > > > >> > > > > > >> > > the
> > > > > > > > >> > > > > > >> > > > > new
> > > > > > > > >> > > > > > >> > > > > > > > read
> > > > > > > > >> > > > > > >> > > > > > > > > > path
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> bypasses
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>
> MetaStoreManager
> > > > > > entirely
> > > > > > > and
> > > > > > > > >> > goes
> > > > > > > > >> > > > > > >> straight to
> > > > > > > > >> > > > > > >> > > > > > > > > BasePersistence
> > > > > > > > >> > > > > > >> > > > > > > > > > > via
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>
> > > > > > > callContext.getMetaStore().
> > > > > > > > >> That
> > > > > > > > >> > > means
> > > > > > > > >> > > > > > the
> > > > > > > > >> > > > > > >> > read
> > > > > > > > >> > > > > > >> > > > API
> > > > > > > > >> > > > > > >> > > > > > only
> > > > > > > > >> > > > > > >> > > > > > > > > works
> > > > > > > > >> > > > > > >> > > > > > > > > > > for
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> backends
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >> that implement
> > > > > > > > >> BasePersistence.
> > > > > > > > >> > > NoSQL
> > > > > > > > >> > > > > and
> > > > > > > > >> > > > > > >> > remote
> > > > > > > > >> > > > > > >> > > > > > > backends
> > > > > > > > >> > > > > > >> > > > > > > > > > can't
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> participate.
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >> Stepping
> back, I
> > > think
> > > > > > the
> > > > > > > > >> > metrics
> > > > > > > > >> > > > > > >> subsystem
> > > > > > > > >> > > > > > >> > is
> > > > > > > > >> > > > > > >> > > > > > growing
> > > > > > > > >> > > > > > >> > > > > > > > into
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> something
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >> real (write +
> > > read +
> > > > > > REST
> > > > > > > API
> > > > > > > > >> +
> > > > > > > > >> > > AuthZ +
> > > > > > > > >> > > > > > >> > > > pagination)
> > > > > > > > >> > > > > > >> > > > > > *but
> > > > > > > > >> > > > > > >> > > > > > > > the
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> persistence
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >> side is split
> > > across
> > > > > two
> > > > > > > > >> layers
> > > > > > > > >> > in
> > > > > > > > >> > > a
> > > > > > > > >> > > > > way
> > > > > > > > >> > > > > > >> > that's
> > > > > > > > >> > > > > > >> > > > hard
> > > > > > > > >> > > > > > >> > > > > > to
> > > > > > > > >> > > > > > >> > > > > > > > > > > extend*. I
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> put
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >> together two
> > > diagrams
> > > > > to
> > > > > > > show
> > > > > > > > >> > what
> > > > > > > > >> > > I
> > > > > > > > >> > > > > mean
> > > > > > > > >> > > > > > >> (my
> > > > > > > > >> > > > > > >> > > best
> > > > > > > > >> > > > > > >> > > > > > > > effort).
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >> *Current
> state*
> > > > > (Diagram
> > > > > > > 1):
> > > > > > > > >> > three
> > > > > > > > >> > > > > > >> interfaces
> > > > > > > > >> > > > > > >> > at
> > > > > > > > >> > > > > > >> > > > > three
> > > > > > > > >> > > > > > >> > > > > > > > > > different
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> levels.
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >> The
> > engine-facing
> > > SPI
> > > > > > > > >> > > > > > >> (PolarisMetricsReporter)
> > > > > > > > >> > > > > > >> > > is
> > > > > > > > >> > > > > > >> > > > > > clean.
> > > > > > > > >> > > > > > >> > > > > > > > But
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>
> > > PolarisMetricsManager
> > > > > on
> > > > > > > > >> > > > > MetaStoreManager
> > > > > > > > >> > > > > > >> is a
> > > > > > > > >> > > > > > >> > > > > > > passthrough
> > > > > > > > >> > > > > > >> > > > > > > > > to
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>
> > > MetricsPersistence on
> > > > > > > > >> > > BasePersistence.
> > > > > > > > >> > > > > > The
> > > > > > > > >> > > > > > >> > @Beta
> > > > > > > > >> > > > > > >> > > > > > > > annotation
> > > > > > > > >> > > > > > >> > > > > > > > > > and
> > > > > > > > >> > > > > > >> > > > > > > > > > > > SPI
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> javadoc
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >> are on the
> > > > > > BasePersistence
> > > > > > > > >> layer,
> > > > > > > > >> > > while
> > > > > > > > >> > > > > > the
> > > > > > > > >> > > > > > >> > > actual
> > > > > > > > >> > > > > > >> > > > > > > > extension
> > > > > > > > >> > > > > > >> > > > > > > > > > > > points
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>
> > > > > (PolarisMetricsReporter,
> > > > > > > > >> > > > > > >> > PolarisMetricsManager)
> > > > > > > > >> > > > > > >> > > > > carry
> > > > > > > > >> > > > > > >> > > > > > no
> > > > > > > > >> > > > > > >> > > > > > > > > > > stability
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >> annotation.
> The
> > > write
> > > > > > path
> > > > > > > > >> goes
> > > > > > > > >> > > through
> > > > > > > > >> > > > > > the
> > > > > > > > >> > > > > > >> > > > > > > > MetaStoreManager
> > > > > > > > >> > > > > > >> > > > > > > > > > > > layer,
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> the
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >> read path
> > doesn't.
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >> *What I
> > envision*
> > > > > > > (Diagram 2):
> > > > > > > > >> > two
> > > > > > > > >> > > SPIs
> > > > > > > > >> > > > > > at
> > > > > > > > >> > > > > > >> two
> > > > > > > > >> > > > > > >> > > > > levels.
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>
> > > PolarisMetricsReporter
> > > > > > > stays
> > > > > > > > >> as
> > > > > > > > >> > the
> > > > > > > > >> > > > > > >> > > engine-facing
> > > > > > > > >> > > > > > >> > > > > SPI.
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>
> > > PolarisMetricsManager
> > > > > > > becomes
> > > > > > > > >> the
> > > > > > > > >> > > > > > >> > backend-facing
> > > > > > > > >> > > > > > >> > > > SPI
> > > > > > > > >> > > > > > >> > > > > > > with
> > > > > > > > >> > > > > > >> > > > > > > > > both
> > > > > > > > >> > > > > > >> > > > > > > > > > > > write
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> and
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >> read methods
> at
> > > the
> > > > > > > > >> > > MetaStoreManager
> > > > > > > > >> > > > > > level,
> > > > > > > > >> > > > > > >> > > where
> > > > > > > > >> > > > > > >> > > > > any
> > > > > > > > >> > > > > > >> > > > > > > > > backend
> > > > > > > > >> > > > > > >> > > > > > > > > > > > (JDBC,
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> NoSQL,
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >> remote) can
> > > implement
> > > > > > > them.
> > > > > > > > >> > > > > > >> MetricsPersistence
> > > > > > > > >> > > > > > >> > > on
> > > > > > > > >> > > > > > >> > > > > > > > > > > BasePersistence
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> goes
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >> away. Where
> > > metrics
> > > > > > > actually
> > > > > > > > >> land
> > > > > > > > >> > > is an
> > > > > > > > >> > > > > > >> > > > > implementation
> > > > > > > > >> > > > > > >> > > > > > > > > detail,
> > > > > > > > >> > > > > > >> > > > > > > > > > > > not a
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> core
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >> interface.
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >> *Minor naming
> > > thing*:
> > > > > > > > >> > > > > > >> PolarisMetricsReporter
> > > > > > > > >> > > > > > >> > is
> > > > > > > > >> > > > > > >> > > > > > broader
> > > > > > > > >> > > > > > >> > > > > > > > than
> > > > > > > > >> > > > > > >> > > > > > > > > > > what
> > > > > > > > >> > > > > > >> > > > > > > > > > > > it
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >> actually
> > handles.
> > > It
> > > > > > only
> > > > > > > > >> accepts
> > > > > > > > >> > > > > Iceberg
> > > > > > > > >> > > > > > >> REST
> > > > > > > > >> > > > > > >> > > > > Catalog
> > > > > > > > >> > > > > > >> > > > > > > > > metrics
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> (ScanReport,
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >> CommitReport
> via
> > > > > > > > >> MetricsReport).
> > > > > > > > >> > > > > Generic
> > > > > > > > >> > > > > > >> table
> > > > > > > > >> > > > > > >> > > > > metrics
> > > > > > > > >> > > > > > >> > > > > > > or
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> operational
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >> metrics aren't
> > in
> > > > > scope.
> > > > > > > Not
> > > > > > > > >> > > blocking,
> > > > > > > > >> > > > > > but
> > > > > > > > >> > > > > > >> > worth
> > > > > > > > >> > > > > > >> > > > > > noting
> > > > > > > > >> > > > > > >> > > > > > > if
> > > > > > > > >> > > > > > >> > > > > > > > > the
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> metrics
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >> surface
> expands.
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >> *Rough sketch
> of
> > > how
> > > > > to
> > > > > > > get
> > > > > > > > >> > there*:
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>  1.  Add read
> > > methods
> > > > > to
> > > > > > > > >> > > > > > >> PolarisMetricsManager
> > > > > > > > >> > > > > > >> > > > > > > > > > (listScanReports,
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>
> > listCommitReports)
> > > > > with
> > > > > > > > >> default
> > > > > > > > >> > > no-op,
> > > > > > > > >> > > > > > >> same as
> > > > > > > > >> > > > > > >> > > the
> > > > > > > > >> > > > > > >> > > > > > > > existing
> > > > > > > > >> > > > > > >> > > > > > > > > > > write
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> methods.
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >> (Probably make
> > > > > > > > >> > > PolarisMetricsManager
> > > > > > > > >> > > > > more
> > > > > > > > >> > > > > > >> > > explicit
> > > > > > > > >> > > > > > >> > > > > on
> > > > > > > > >> > > > > > >> > > > > > > > being
> > > > > > > > >> > > > > > >> > > > > > > > > > > > Iceberg
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >> specific like
> > > package
> > > > > > > name or
> > > > > > > > >> > class
> > > > > > > > >> > > > > name
> > > > > > > > >> > > > > > >> etc.)
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>  2.  Wire
> > > > > > > > >> MetricsReportsService
> > > > > > > > >> > > through
> > > > > > > > >> > > > > > >> > > > > > MetaStoreManager
> > > > > > > > >> > > > > > >> > > > > > > > > > instead
> > > > > > > > >> > > > > > >> > > > > > > > > > > > of
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>
> > > > > > > callContext.getMetaStore().
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>  3.  Extract
> > > metrics
> > > > > > > > >> persistence
> > > > > > > > >> > > from
> > > > > > > > >> > > > > > >> > > > > > > > > JdbcBasePersistenceImpl
> > > > > > > > >> > > > > > >> > > > > > > > > > > into
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> its
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >> own class.
> That
> > > file
> > > > > > > carries
> > > > > > > > >> ~7
> > > > > > > > >> > > > > > >> > > responsibilities,
> > > > > > > > >> > > > > > >> > > > > > > metrics
> > > > > > > > >> > > > > > >> > > > > > > > > > being
> > > > > > > > >> > > > > > >> > > > > > > > > > > > one
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> of
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> them.
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>  4.  Remove
> > > > > > > MetricsPersistence
> > > > > > > > >> > from
> > > > > > > > >> > > > > > >> > > > BasePersistence.
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >> *None of this
> > > needs to
> > > > > > > happen
> > > > > > > > >> in
> > > > > > > > >> > > #4115.
> > > > > > > > >> > > > > > >> But if
> > > > > > > > >> > > > > > >> > > the
> > > > > > > > >> > > > > > >> > > > > > > > direction
> > > > > > > > >> > > > > > >> > > > > > > > > > > makes
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> sense,
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >> it would be
> good
> > > to
> > > > > > align
> > > > > > > > >> before
> > > > > > > > >> > > the
> > > > > > > > >> > > > > > >> metrics
> > > > > > > > >> > > > > > >> > > > surface
> > > > > > > > >> > > > > > >> > > > > > > grows
> > > > > > > > >> > > > > > >> > > > > > > > > > > > further.
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> Curious
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >> what others
> > > think.*
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >> *My mental
> model
> > > > > note*:
> > > > > > > Level
> > > > > > > > >> 1
> > > > > > > > >> > > > > > >> > > MetaStoreManager;
> > > > > > > > >> > > > > > >> > > > > > level
> > > > > > > > >> > > > > > >> > > > > > > 2
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> transactional
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >> persistence;
> > > level 3
> > > > > > base
> > > > > > > > >> > > persistence
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >> Diagram 1
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >> <
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >>
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >>
> > > > > > > > >> > > > > > >> > > > > > > > > > > >
> > > > > > > > >> > > > > > >> > > > > > > > > > >
> > > > > > > > >> > > > > > >> > > > > > > > > >
> > > > > > > > >> > > > > > >> > > > > > > > >
> > > > > > > > >> > > > > > >> > > > > > > >
> > > > > > > > >> > > > > > >> > > > > > >
> > > > > > > > >> > > > > > >> > > > > >
> > > > > > > > >> > > > > > >> > > > >
> > > > > > > > >> > > > > > >> > > >
> > > > > > > > >> > > > > > >> > >
> > > > > > > > >> > > > > > >> >
> > > > > > > > >> > > > > > >>
> > > > > > > > >> > > > > >
> > > > > > > > >> > > > >
> > > > > > > > >> > >
> > > > > > > > >> >
> > > > > > > > >>
> > > > > > >
> > > > > >
> > > > >
> > >
> >
> https://urldefense.com/v3/__https://www.plantuml.com/plantuml/uml/bLHDR-Cs4BthLmpIYupw0zbkKQ1r3M-S7Bp8xhhM7WCOb3IM65EaGD9EX2RzxHrHb4CxRelwa4YSDu_lpOVcnZ9jzvM8BBS2uGjQpJC3dtHMSekPtMk44IpsMgEqa5XcCOhCZikQQLP1pR8TAp2n3ILhmZDP20m0fcIvUkAoW2qJXd9z1bpToO9BX3WXu0ucy5rpgGPNm0nW5_epUWtm2Ue3pn3kMOFQmKntGZW0BYtgBSi8k5A2QMwybJNMIbFiGSR9QZc4nUqIvikStF0jHprua5C-amge42aNt3R0f5JaaoivdV2Pkqbx4hee4ymOkBh5BTiB-_uIeGeo8zL8rPsPl4DktdEiK1jkB1NdZCRbrSTecDe_mlHbF0wvBmCkaOH5_S8a_TTTKI6-nmCAkEw4LpxsZ-LbYLKQFKMNOgf_wuM7_bV9gOer5SYMMksBSWXFcbi49KNZXNLicwfe3TETC7gPdPqI7uBcHMb1RSzYq34c6PDUM9mn8HRsUTZEiDBve3NjVZumBj0U7SS37mGO7vcwtiK-_pU7U7L_f-digo9YbhSwIfMRwIITKGXbxdIUTCGF1SeCJxloKsU-3k9ddRbX1eDq1q_fx1JbBGT0glVyXimDuP4TQ5qpCAmnGEj2s_6n5mtn1z-97-63itFQZLPO1Ev2tu_WF7Ju-VPc0Skg5bYXxBhkY1xpD7EM_7fyflSpIsqMgVth5xhVr4eQxWQ8enaSAJQSG16yFSDuJ798rrcXr_3n-lfdk7icQjEBmFujL7AodiP_Y4Z7-YxvtZNs4zMgpNTl6tF8sglyPsmqchrjvQ-m-aP94r-TwCA2Ka8upPJZwtvSpoYCXkYMZU2NXvRMBfq9P3i3Le4VAZUAlUZ_oPKsxPgY0Q_BSKLkyr9bhQhQrJjo_x3TPlIB0DPjnMfcIoYP0QaYw1a0fTKDr8fB6ntNuvmoL1ZGkXa69Njh43zf9GiGxHQrA_jDYWRSzF5--WmTVrN97_Sm8LbLUy_lGBmLanJjFkDlGkRqjA_4tm00__;!!Iz9xO38YGHZK!4rnlVIqBQQQ6xpGe1PMV58edOS_cGrKoqdfhw4qSp_JQCavvEjaxXk0OwGPnxyEcO60AzKQK_XDtp7H3DdkFXcrXuTBISHg$
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >> :
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >> [image:
> > image.png]
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >> Diagram 2
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >> <
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >>
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >>
> > > > > > > > >> > > > > > >> > > > > > > > > > > >
> > > > > > > > >> > > > > > >> > > > > > > > > > >
> > > > > > > > >> > > > > > >> > > > > > > > > >
> > > > > > > > >> > > > > > >> > > > > > > > >
> > > > > > > > >> > > > > > >> > > > > > > >
> > > > > > > > >> > > > > > >> > > > > > >
> > > > > > > > >> > > > > > >> > > > > >
> > > > > > > > >> > > > > > >> > > > >
> > > > > > > > >> > > > > > >> > > >
> > > > > > > > >> > > > > > >> > >
> > > > > > > > >> > > > > > >> >
> > > > > > > > >> > > > > > >>
> > > > > > > > >> > > > > >
> > > > > > > > >> > > > >
> > > > > > > > >> > >
> > > > > > > > >> >
> > > > > > > > >>
> > > > > > >
> > > > > >
> > > > >
> > >
> >
> https://urldefense.com/v3/__https://www.plantuml.com/plantuml/uml/VLLDR-8m4BtdLupO2sWBLVU8AaGB7AXAbssGzb896SS9RXqxjKqBwkv_tt7iV43fdaZYDpFlpRmnOsE9jhjSH9PRmM31hERKm8scMsuPjJlDe0yheZDc8RR4iYWoBrmMH9CS2a9VICPYUy1OZN0YCy5Q0BCbYNhdCeEK28En8G8wCvbnoQ0R8_05Bc6bkLIz3X03p1zzH7zR-9ZfDquPt9C3qoNCX2yV4G2NbkcKu5jdgGJHt0GbZwnG6i-UP3TUpk5gM6Ldqke350eZUqzoCft3U9xWHvxoa5-7K4nF1J46EbEMafsmdrCBbQ44gVggy18IZrn_ph5asd1ZiIKdQSgueZvjXrQFSFrdC3YN-nXmBacxbGiYyLVxLaBtdhqn0LSzdBDhqQtQoOJeGyad3z0lUqnYgpGB6Ns8oVyta00Dy_WnX0tIOZ8v6SYxHll1TrH6aejAik-mh-AphVFCwSUQqFypElag5QRGFDjQKEd96K1P8QP41c9TzA_IIQyvdAWyv_RSiS3skb0_EzDDkK2v5xWF6MiGFlvhpFLcD2Dq2pml14gaF67eQkmd8gulDoC4kSOu6KVpkvlUJg1RTbWISU40RdBUUS_9XfRZ2dwxm_SW8LYFISgm_MnlDQ6M9P1gbKEc4X-2pH_FvJCkCqm9pbVjD6LrwdLeOrDWfOaqc8Wh9BE85oNKxkNQ6o4yGRy_Eae0G_G8tZv81d3bHDB23WOdisohVr3nh_j6lbSjbNaLRTc8UgtPbAU1J_tygOfZX9DWEJeHDvYx-qmSi5FgNLPZwHrHcUsncGQ5-skhUclpE5fo4ounpFauYrUbkU6ccfnxMvitwag4IyerhTxj8In_Oj1bDO4pQru674loYrGlULHLEGCjwJJ8gDoVZR8MxO4BT3IzRvIcAQKezC6xpziGnTyImrfEGyJI_OcKfgtxIvnTqFEMS17L9Z-jsARN5FmTheP7HtSdtOMT0B4GY2FYHXxgQmMtj2bRqiLFGapiVe1_QVKDrkqXcm83aFEXnMYCZ-xlyHy__;!!Iz9xO38YGHZK!4rnlVIqBQQQ6xpGe1PMV58edOS_cGrKoqdfhw4qSp_JQCavvEjaxXk0OwGPnxyEcO60AzKQK_XDtp7H3DdkFXcrXSpmqcNk$
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >> :
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >> [image:
> > image.png]
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>  -ej
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >> On Wed, Apr
> 15,
> > > 2026
> > > > > at
> > > > > > > > >> 8:22 AM
> > > > > > > > >> > > Dmitri
> > > > > > > > >> > > > > > >> > > > Bourlatchkov
> > > > > > > > >> > > > > > >> > > > > <
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> dimas@apache.org>
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >> wrote:
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> Hi All,
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>>
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> Heads up: The
> > > current
> > > > > > > state
> > > > > > > > >> of
> > > > > > > > >> > PR
> > > > > > > > >> > > > > [4115]
> > > > > > > > >> > > > > > >> > looks
> > > > > > > > >> > > > > > >> > > > > pretty
> > > > > > > > >> > > > > > >> > > > > > > > solid
> > > > > > > > >> > > > > > >> > > > > > > > > > to
> > > > > > > > >> > > > > > >> > > > > > > > > > > > me.
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> I
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> believe this
> PR
> > > is
> > > > > > > > >> approaching a
> > > > > > > > >> > > > > > mergeable
> > > > > > > > >> > > > > > >> > > > > condition.
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>>
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> Please post
> > your
> > > > > > reviews
> > > > > > > if
> > > > > > > > >> you
> > > > > > > > >> > > have
> > > > > > > > >> > > > > any
> > > > > > > > >> > > > > > >> > > > comments.
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>>
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> [4115]
> > > > > > > > >> > > > > > >> > > >
> > > > > > > > >> >
> > > > > > > > >>
> > > > > > >
> > > > > >
> > > > >
> > >
> >
> https://urldefense.com/v3/__https://github.com/apache/polaris/pull/4115__;!!Iz9xO38YGHZK!4rnlVIqBQQQ6xpGe1PMV58edOS_cGrKoqdfhw4qSp_JQCavvEjaxXk0OwGPnxyEcO60AzKQK_XDtp7H3DdkFXcrXtFRscKs$
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>>
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> Thanks,
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> Dmitri.
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>>
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> On Tue, Mar
> 3,
> > > 2026
> > > > > at
> > > > > > > > >> 3:29 PM
> > > > > > > > >> > > Anand
> > > > > > > > >> > > > > > Kumar
> > > > > > > > >> > > > > > >> > > > Sankaran
> > > > > > > > >> > > > > > >> > > > > > via
> > > > > > > > >> > > > > > >> > > > > > > > > dev <
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>>
> > > > > dev@polaris.apache.org
> > > > > > >
> > > > > > > > >> wrote:
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>>
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > Hi Yufei
> and
> > > > > Dmitri,
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> >
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > Here is a
> > > proposal
> > > > > > for
> > > > > > > the
> > > > > > > > >> > REST
> > > > > > > > >> > > > > > >> endpoints
> > > > > > > > >> > > > > > >> > for
> > > > > > > > >> > > > > > >> > > > > > metrics
> > > > > > > > >> > > > > > >> > > > > > > > and
> > > > > > > > >> > > > > > >> > > > > > > > > > > > events.
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> >
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> >
> > > > > > > > >> > > > > > >> > > > > >
> > > > > > > > >> >
> > > > > > > > >>
> > > > > > >
> > > > > >
> > > > >
> > >
> >
> https://urldefense.com/v3/__https://github.com/apache/polaris/pull/3924/changes__;!!Iz9xO38YGHZK!4rnlVIqBQQQ6xpGe1PMV58edOS_cGrKoqdfhw4qSp_JQCavvEjaxXk0OwGPnxyEcO60AzKQK_XDtp7H3DdkFXcrXcjoPhSs$
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> >
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > I did not
> see
> > > any
> > > > > > > > >> precursors
> > > > > > > > >> > for
> > > > > > > > >> > > > > > >> raising a
> > > > > > > > >> > > > > > >> > PR
> > > > > > > > >> > > > > > >> > > > for
> > > > > > > > >> > > > > > >> > > > > > > > > > proposals,
> > > > > > > > >> > > > > > >> > > > > > > > > > > so
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> trying
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > this.
> Please
> > > let
> > > > > me
> > > > > > > know
> > > > > > > > >> what
> > > > > > > > >> > > you
> > > > > > > > >> > > > > > >> think.
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> >
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > -
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > Anand
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> >
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > From: Anand
> > > Kumar
> > > > > > > Sankaran
> > > > > > > > >> <
> > > > > > > > >> > > > > > >> > > > > > > anand.sankaran@workday.com
> > > > > > > > >> > > > > > >> > > > > > > > >
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > Date:
> Monday,
> > > March
> > > > > > 2,
> > > > > > > > >> 2026 at
> > > > > > > > >> > > > > > 10:25 AM
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > To:
> > > > > > > dev@polaris.apache.org
> > > > > > > > >> <
> > > > > > > > >> > > > > > >> > > > > dev@polaris.apache.org
> > > > > > > > >> > > > > > >> > > > > > >
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > Subject:
> Re:
> > > > > Polaris
> > > > > > > > >> Telemetry
> > > > > > > > >> > > and
> > > > > > > > >> > > > > > Audit
> > > > > > > > >> > > > > > >> > > Trail
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> >
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > About the
> > REST
> > > API,
> > > > > > > based
> > > > > > > > >> on
> > > > > > > > >> > my
> > > > > > > > >> > > use
> > > > > > > > >> > > > > > >> cases:
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> >
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> >
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> >   1.
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > I want to
> be
> > > able
> > > > > to
> > > > > > > query
> > > > > > > > >> > > commit
> > > > > > > > >> > > > > > >> metrics
> > > > > > > > >> > > > > > >> > to
> > > > > > > > >> > > > > > >> > > > > track
> > > > > > > > >> > > > > > >> > > > > > > > files
> > > > > > > > >> > > > > > >> > > > > > > > > > > added
> > > > > > > > >> > > > > > >> > > > > > > > > > > > /
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> removed
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > per commit,
> > > along
> > > > > > with
> > > > > > > > >> record
> > > > > > > > >> > > > > counts.
> > > > > > > > >> > > > > > >> The
> > > > > > > > >> > > > > > >> > > > > ingestion
> > > > > > > > >> > > > > > >> > > > > > > > > > pipeline
> > > > > > > > >> > > > > > >> > > > > > > > > > > > that
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> writes
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > this data
> is
> > > owned
> > > > > by
> > > > > > > us
> > > > > > > > >> and
> > > > > > > > >> > we
> > > > > > > > >> > > are
> > > > > > > > >> > > > > > >> > > guaranteed
> > > > > > > > >> > > > > > >> > > > to
> > > > > > > > >> > > > > > >> > > > > > > write
> > > > > > > > >> > > > > > >> > > > > > > > > > this
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> information
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > for each
> > write.
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> >   2.
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > I want to
> be
> > > able
> > > > > to
> > > > > > > query
> > > > > > > > >> > scan
> > > > > > > > >> > > > > > metrics
> > > > > > > > >> > > > > > >> for
> > > > > > > > >> > > > > > >> > > > > read. I
> > > > > > > > >> > > > > > >> > > > > > > > > > > understand
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> clients
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> do
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > not fulfill
> > > this
> > > > > > > > >> requirement.
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> >   3.
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > I want to
> be
> > > able
> > > > > to
> > > > > > > query
> > > > > > > > >> the
> > > > > > > > >> > > > > events
> > > > > > > > >> > > > > > >> table
> > > > > > > > >> > > > > > >> > > > > (events
> > > > > > > > >> > > > > > >> > > > > > > are
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> persisted) -
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> this
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > may
> supersede
> > > #2, I
> > > > > > am
> > > > > > > not
> > > > > > > > >> > sure
> > > > > > > > >> > > yet.
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> >
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > All this
> > > > > information
> > > > > > > is in
> > > > > > > > >> the
> > > > > > > > >> > > JDBC
> > > > > > > > >> > > > > > >> based
> > > > > > > > >> > > > > > >> > > > > > persistence
> > > > > > > > >> > > > > > >> > > > > > > > > model
> > > > > > > > >> > > > > > >> > > > > > > > > > > and
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> is
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > persisted
> in
> > > the
> > > > > > > > >> metastore. I
> > > > > > > > >> > > > > > currently
> > > > > > > > >> > > > > > >> > don’t
> > > > > > > > >> > > > > > >> > > > > have
> > > > > > > > >> > > > > > >> > > > > > a
> > > > > > > > >> > > > > > >> > > > > > > > need
> > > > > > > > >> > > > > > >> > > > > > > > > > to
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> query
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > prometheus
> or
> > > open
> > > > > > > > >> telemetry.
> > > > > > > > >> > I
> > > > > > > > >> > > do
> > > > > > > > >> > > > > > >> publish
> > > > > > > > >> > > > > > >> > > some
> > > > > > > > >> > > > > > >> > > > > > > events
> > > > > > > > >> > > > > > >> > > > > > > > to
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> Prometheus
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> and
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > they are
> > > forwarded
> > > > > to
> > > > > > > our
> > > > > > > > >> > > dashboards
> > > > > > > > >> > > > > > >> > > elsewhere.
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> >
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > About the
> CLI
> > > > > > > utilities, I
> > > > > > > > >> > > meant the
> > > > > > > > >> > > > > > >> admin
> > > > > > > > >> > > > > > >> > > user
> > > > > > > > >> > > > > > >> > > > > > > > > utilities.
> > > > > > > > >> > > > > > >> > > > > > > > > > In
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> one of
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> the
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > earliest
> > > drafts of
> > > > > my
> > > > > > > > >> > proposal,
> > > > > > > > >> > > > > > Prashant
> > > > > > > > >> > > > > > >> > > > > mentioned
> > > > > > > > >> > > > > > >> > > > > > > that
> > > > > > > > >> > > > > > >> > > > > > > > > the
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> metrics
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> tables
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > can grow
> > > > > indefinitely
> > > > > > > and
> > > > > > > > >> > that a
> > > > > > > > >> > > > > > similar
> > > > > > > > >> > > > > > >> > > > problem
> > > > > > > > >> > > > > > >> > > > > > > exists
> > > > > > > > >> > > > > > >> > > > > > > > > > with
> > > > > > > > >> > > > > > >> > > > > > > > > > > > the
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> events
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > table as
> > well.
> > > We
> > > > > > > discussed
> > > > > > > > >> > that
> > > > > > > > >> > > > > > >> cleaning
> > > > > > > > >> > > > > > >> > up
> > > > > > > > >> > > > > > >> > > of
> > > > > > > > >> > > > > > >> > > > > old
> > > > > > > > >> > > > > > >> > > > > > > > > records
> > > > > > > > >> > > > > > >> > > > > > > > > > > > from
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> both
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > metrics
> > tables
> > > and
> > > > > > > events
> > > > > > > > >> > > tables can
> > > > > > > > >> > > > > > be
> > > > > > > > >> > > > > > >> > done
> > > > > > > > >> > > > > > >> > > > via
> > > > > > > > >> > > > > > >> > > > > a
> > > > > > > > >> > > > > > >> > > > > > > CLI
> > > > > > > > >> > > > > > >> > > > > > > > > > > utility.
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> >
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > I see that
> > > Yufei
> > > > > has
> > > > > > > > >> covered
> > > > > > > > >> > the
> > > > > > > > >> > > > > > >> discussion
> > > > > > > > >> > > > > > >> > > > about
> > > > > > > > >> > > > > > >> > > > > > > > > > > datasources.
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> >
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > -
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > Anand
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> >
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> >
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> >
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > From: Yufei
> > Gu
> > > <
> > > > > > > > >> > > > > flyrain000@gmail.com>
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > Date:
> Friday,
> > > > > > February
> > > > > > > 27,
> > > > > > > > >> > 2026
> > > > > > > > >> > > at
> > > > > > > > >> > > > > > >> 9:54 PM
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > To:
> > > > > > > dev@polaris.apache.org
> > > > > > > > >> <
> > > > > > > > >> > > > > > >> > > > > dev@polaris.apache.org
> > > > > > > > >> > > > > > >> > > > > > >
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > Subject:
> Re:
> > > > > Polaris
> > > > > > > > >> Telemetry
> > > > > > > > >> > > and
> > > > > > > > >> > > > > > Audit
> > > > > > > > >> > > > > > >> > > Trail
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> >
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > This
> Message
> > Is
> > > > > From
> > > > > > an
> > > > > > > > >> > External
> > > > > > > > >> > > > > > Sender
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > This
> message
> > > came
> > > > > > from
> > > > > > > > >> outside
> > > > > > > > >> > > your
> > > > > > > > >> > > > > > >> > > > organization.
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > Report
> > > Suspicious<
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> >
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>>
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >>
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >>
> > > > > > > > >> > > > > > >> > > > > > > > > > > >
> > > > > > > > >> > > > > > >> > > > > > > > > > >
> > > > > > > > >> > > > > > >> > > > > > > > > >
> > > > > > > > >> > > > > > >> > > > > > > > >
> > > > > > > > >> > > > > > >> > > > > > > >
> > > > > > > > >> > > > > > >> > > > > > >
> > > > > > > > >> > > > > > >> > > > > >
> > > > > > > > >> > > > > > >> > > > >
> > > > > > > > >> > > > > > >> > > >
> > > > > > > > >> > > > > > >> > >
> > > > > > > > >> > > > > > >> >
> > > > > > > > >> > > > > > >>
> > > > > > > > >> > > > > >
> > > > > > > > >> > > > >
> > > > > > > > >> > >
> > > > > > > > >> >
> > > > > > > > >>
> > > > > > >
> > > > > >
> > > > >
> > >
> >
> https://us-phishalarm-ewt.proofpoint.com/EWT/v1/Iz9xO38YGHZK!YhNDZABkHi1B699ote2uMwpOZw8i0QMCGO2Szc-HshuABGhGvwPJcymE6G2oUUxtS8xDkSrtGTPm_I3QnVDHoLMk50m9v8z_nZKTkd-bnVUbreF1u0WnfV_X5eYevZl_$
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > >
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> >
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> >
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > As I
> > mentioned
> > > in
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> >
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>>
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >>
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >>
> > > > > > > > >> > > > > > >> > > > > > > > > > > >
> > > > > > > > >> > > > > > >> > > > > > > > > > >
> > > > > > > > >> > > > > > >> > > > > > > > > >
> > > > > > > > >> > > > > > >> > > > > > > > >
> > > > > > > > >> > > > > > >> > > > > > > >
> > > > > > > > >> > > > > > >> > > > > > >
> > > > > > > > >> > > > > > >> > > > > >
> > > > > > > > >> > > > > > >> > > > >
> > > > > > > > >> > > > > > >> > > >
> > > > > > > > >> > > > > > >> > >
> > > > > > > > >> > > > > > >> >
> > > > > > > > >> > > > > > >>
> > > > > > > > >> > > > > >
> > > > > > > > >> > > > >
> > > > > > > > >> > >
> > > > > > > > >> >
> > > > > > > > >>
> > > > > > >
> > > > > >
> > > > >
> > >
> >
> https://urldefense.com/v3/__https://github.com/apache/polaris/issues/3890__;!!Iz9xO38YGHZK!5EuyFFkk3vhRWVIRvQAWBSQfpJkTMA9HxugzDwXmN0LPPqhEFxYkFRGVhtb8AqUwXtDh2OplcMnbMDHKOxrvDU0$
> > > > > > > > >> > >> > > >> > > > > > > > > > > > >> >> >>> ,
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > supporting
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > multiple
> data
> > > > > sources
> > > > > > > is
> > > > > > > > >> not a
> > > > > > > > >> > > > > trivial
> > > > > > > > >> > > > > > >> > > change.
> > > > > > > > >> > > > > > >> > > > I
> > > > > > > > >> > > > > > >> > > > > > > would
> > > > > > > > >> > > > > > >> > > > > > > > > > > strongly
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> recommend
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > starting
> > with a
> > > > > > design
> > > > > > > > >> > document
> > > > > > > > >> > > to
> > > > > > > > >> > > > > > >> > carefully
> > > > > > > > >> > > > > > >> > > > > > evaluate
> > > > > > > > >> > > > > > >> > > > > > > > the
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> architectural
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> >
> implications
> > > and
> > > > > long
> > > > > > > term
> > > > > > > > >> > > impact.
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> >
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > A REST
> > > endpoint to
> > > > > > > query
> > > > > > > > >> > metrics
> > > > > > > > >> > > > > seems
> > > > > > > > >> > > > > > >> > > > reasonable
> > > > > > > > >> > > > > > >> > > > > > > given
> > > > > > > > >> > > > > > >> > > > > > > > > the
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> current
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> JDBC
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > based
> > > persistence
> > > > > > > model.
> > > > > > > > >> That
> > > > > > > > >> > > said,
> > > > > > > > >> > > > > we
> > > > > > > > >> > > > > > >> may
> > > > > > > > >> > > > > > >> > > also
> > > > > > > > >> > > > > > >> > > > > > > > consider
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> alternative
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > storage
> > > models. For
> > > > > > > > >> example,
> > > > > > > > >> > if
> > > > > > > > >> > > we
> > > > > > > > >> > > > > > later
> > > > > > > > >> > > > > > >> > > adopt
> > > > > > > > >> > > > > > >> > > > a
> > > > > > > > >> > > > > > >> > > > > > time
> > > > > > > > >> > > > > > >> > > > > > > > > > series
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> system
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> such as
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > Prometheus
> to
> > > store
> > > > > > > > >> metrics,
> > > > > > > > >> > the
> > > > > > > > >> > > > > query
> > > > > > > > >> > > > > > >> > model
> > > > > > > > >> > > > > > >> > > > and
> > > > > > > > >> > > > > > >> > > > > > > access
> > > > > > > > >> > > > > > >> > > > > > > > > > > > patterns
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> would
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> be
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> >
> fundamentally
> > > > > > > different.
> > > > > > > > >> > > Designing
> > > > > > > > >> > > > > the
> > > > > > > > >> > > > > > >> REST
> > > > > > > > >> > > > > > >> > > API
> > > > > > > > >> > > > > > >> > > > > > > without
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> considering
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> these
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > potential
> > > > > evolutions
> > > > > > > may
> > > > > > > > >> limit
> > > > > > > > >> > > > > > >> flexibility.
> > > > > > > > >> > > > > > >> > > I'd
> > > > > > > > >> > > > > > >> > > > > > > suggest
> > > > > > > > >> > > > > > >> > > > > > > > > to
> > > > > > > > >> > > > > > >> > > > > > > > > > > > start
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> with
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> the
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > use case.
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> >
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > Yufei
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> >
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> >
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > On Fri, Feb
> > 27,
> > > > > 2026
> > > > > > at
> > > > > > > > >> > 3:42 PM
> > > > > > > > >> > > > > Dmitri
> > > > > > > > >> > > > > > >> > > > > > Bourlatchkov <
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> dimas@apache.org
> >
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > wrote:
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> >
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > > Hi Anand,
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > >
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > > Sharing
> my
> > > > > view...
> > > > > > > > >> subject
> > > > > > > > >> > to
> > > > > > > > >> > > > > > >> discussion:
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > >
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > > 1. Adding
> > > non-IRC
> > > > > > > REST
> > > > > > > > >> API
> > > > > > > > >> > to
> > > > > > > > >> > > > > > Polaris
> > > > > > > > >> > > > > > >> is
> > > > > > > > >> > > > > > >> > > > > > perfectly
> > > > > > > > >> > > > > > >> > > > > > > > > fine.
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > >
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > > Figuring
> > out
> > > > > > specific
> > > > > > > > >> > endpoint
> > > > > > > > >> > > > > URIs
> > > > > > > > >> > > > > > >> and
> > > > > > > > >> > > > > > >> > > > > payloads
> > > > > > > > >> > > > > > >> > > > > > > > might
> > > > > > > > >> > > > > > >> > > > > > > > > > > > require
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> a
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> few
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > >
> roundtrips,
> > > so
> > > > > > > opening a
> > > > > > > > >> > > separate
> > > > > > > > >> > > > > > >> thread
> > > > > > > > >> > > > > > >> > > for
> > > > > > > > >> > > > > > >> > > > > that
> > > > > > > > >> > > > > > >> > > > > > > > might
> > > > > > > > >> > > > > > >> > > > > > > > > > be
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> best.
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > >
> > Contributors
> > > > > > commonly
> > > > > > > > >> create
> > > > > > > > >> > > > > Google
> > > > > > > > >> > > > > > >> Docs
> > > > > > > > >> > > > > > >> > > for
> > > > > > > > >> > > > > > >> > > > > new
> > > > > > > > >> > > > > > >> > > > > > > API
> > > > > > > > >> > > > > > >> > > > > > > > > > > > proposals
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> too
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> (they
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > > fairly
> easy
> > > to
> > > > > > > update as
> > > > > > > > >> the
> > > > > > > > >> > > email
> > > > > > > > >> > > > > > >> > > discussion
> > > > > > > > >> > > > > > >> > > > > > > > > > progresses).
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > >
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > > There
> was a
> > > > > > > suggestion to
> > > > > > > > >> > try
> > > > > > > > >> > > > > > Markdown
> > > > > > > > >> > > > > > >> > > (with
> > > > > > > > >> > > > > > >> > > > > PRs)
> > > > > > > > >> > > > > > >> > > > > > > for
> > > > > > > > >> > > > > > >> > > > > > > > > > > > proposals
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> [1]
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> ...
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > > feel free
> > to
> > > give
> > > > > > it
> > > > > > > a
> > > > > > > > >> try
> > > > > > > > >> > if
> > > > > > > > >> > > you
> > > > > > > > >> > > > > > are
> > > > > > > > >> > > > > > >> > > > > comfortable
> > > > > > > > >> > > > > > >> > > > > > > > with
> > > > > > > > >> > > > > > >> > > > > > > > > > > that.
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > >
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > > 2. Could
> > you
> > > > > > clarify
> > > > > > > > >> whether
> > > > > > > > >> > > you
> > > > > > > > >> > > > > > mean
> > > > > > > > >> > > > > > >> end
> > > > > > > > >> > > > > > >> > > > user
> > > > > > > > >> > > > > > >> > > > > > > > > utilities
> > > > > > > > >> > > > > > >> > > > > > > > > > or
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> admin
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> user
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > >
> utilities?
> > > In the
> > > > > > > latter
> > > > > > > > >> > case
> > > > > > > > >> > > > > those
> > > > > > > > >> > > > > > >> might
> > > > > > > > >> > > > > > >> > > be
> > > > > > > > >> > > > > > >> > > > > more
> > > > > > > > >> > > > > > >> > > > > > > > > > suitable
> > > > > > > > >> > > > > > >> > > > > > > > > > > > for
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> the
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> Admin
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > > CLI
> (java)
> > > not
> > > > > the
> > > > > > > Python
> > > > > > > > >> > CLI,
> > > > > > > > >> > > > > IMHO.
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > >
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > > Why would
> > > these
> > > > > > > > >> utilities be
> > > > > > > > >> > > > > common
> > > > > > > > >> > > > > > >> with
> > > > > > > > >> > > > > > >> > > > > events?
> > > > > > > > >> > > > > > >> > > > > > > > IMHO,
> > > > > > > > >> > > > > > >> > > > > > > > > > > event
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> use
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> cases
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > are
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > > distinct
> > from
> > > > > > > scan/commit
> > > > > > > > >> > > metrics.
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > >
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > > 3. I'd
> > prefer
> > > > > > > separating
> > > > > > > > >> > > metrics
> > > > > > > > >> > > > > > >> > > persistence
> > > > > > > > >> > > > > > >> > > > > from
> > > > > > > > >> > > > > > >> > > > > > > > > > MetaStore
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> persistence
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > at
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > > the code
> > > level,
> > > > > so
> > > > > > > that
> > > > > > > > >> they
> > > > > > > > >> > > could
> > > > > > > > >> > > > > > be
> > > > > > > > >> > > > > > >> > mixed
> > > > > > > > >> > > > > > >> > > > and
> > > > > > > > >> > > > > > >> > > > > > > > matched
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>>
> independently.
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > The
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > > separate
> > > > > datasource
> > > > > > > > >> question
> > > > > > > > >> > > will
> > > > > > > > >> > > > > > >> become
> > > > > > > > >> > > > > > >> > a
> > > > > > > > >> > > > > > >> > > > > > > non-issue
> > > > > > > > >> > > > > > >> > > > > > > > > with
> > > > > > > > >> > > > > > >> > > > > > > > > > > > that
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> approach,
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > I
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > > guess.
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > >
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > > The
> > > rationale for
> > > > > > > > >> separating
> > > > > > > > >> > > scan
> > > > > > > > >> > > > > > >> metrics
> > > > > > > > >> > > > > > >> > > and
> > > > > > > > >> > > > > > >> > > > > > > > metastore
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> persistence
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> is
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > that
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > >
> "cascading
> > > > > deletes"
> > > > > > > > >> between
> > > > > > > > >> > > them
> > > > > > > > >> > > > > are
> > > > > > > > >> > > > > > >> > hardly
> > > > > > > > >> > > > > > >> > > > > ever
> > > > > > > > >> > > > > > >> > > > > > > > > > required.
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> Furthermore,
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > the
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > > data and
> > > query
> > > > > > > patterns
> > > > > > > > >> are
> > > > > > > > >> > > very
> > > > > > > > >> > > > > > >> > different
> > > > > > > > >> > > > > > >> > > so
> > > > > > > > >> > > > > > >> > > > > > > > different
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> technologies
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > might
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > > be
> > > beneficial in
> > > > > > each
> > > > > > > > >> case.
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > >
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > > [1]
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> >
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>>
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >>
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >>
> > > > > > > > >> > > > > > >> > > > > > > > > > > >
> > > > > > > > >> > > > > > >> > > > > > > > > > >
> > > > > > > > >> > > > > > >> > > > > > > > > >
> > > > > > > > >> > > > > > >> > > > > > > > >
> > > > > > > > >> > > > > > >> > > > > > > >
> > > > > > > > >> > > > > > >> > > > > > >
> > > > > > > > >> > > > > > >> > > > > >
> > > > > > > > >> > > > > > >> > > > >
> > > > > > > > >> > > > > > >> > > >
> > > > > > > > >> > > > > > >> > >
> > > > > > > > >> > > > > > >> >
> > > > > > > > >> > > > > > >>
> > > > > > > > >> > > > > >
> > > > > > > > >> > > > >
> > > > > > > > >> > >
> > > > > > > > >> >
> > > > > > > > >>
> > > > > > >
> > > > > >
> > > > >
> > >
> >
> https://urldefense.com/v3/__https://lists.apache.org/thread/yto2wp982t43h1mqjwnslswhws5z47cy__;!!Iz9xO38YGHZK!5EuyFFkk3vhRWVIRvQAWBSQfpJkTMA9HxugzDwXmN0LPPqhEFxYkFRGVhtb8AqUwXtDh2OplcMnbMDHKxYDakNU$
> > > > > > > > >> > >> > > >> > > > > > > > > > > > >> >> >>> > >
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > > Cheers,
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > > Dmitri.
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > >
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > > On Fri,
> Feb
> > > 27,
> > > > > > 2026
> > > > > > > at
> > > > > > > > >> > > 6:19 PM
> > > > > > > > >> > > > > > Anand
> > > > > > > > >> > > > > > >> > Kumar
> > > > > > > > >> > > > > > >> > > > > > > Sankaran
> > > > > > > > >> > > > > > >> > > > > > > > > via
> > > > > > > > >> > > > > > >> > > > > > > > > > > dev
> > > > > > > > >> > > > > > >> > > > > > > > > > > > <
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > >
> > > > > > > dev@polaris.apache.org>
> > > > > > > > >> > > wrote:
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > >
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > > > Thanks
> > all.
> > > > > This
> > > > > > > PR is
> > > > > > > > >> > > merged
> > > > > > > > >> > > > > now.
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > > >
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > > > Here
> are
> > > the
> > > > > > > follow-up
> > > > > > > > >> > > features
> > > > > > > > >> > > > > /
> > > > > > > > >> > > > > > >> work
> > > > > > > > >> > > > > > >> > > > > needed.
> > > > > > > > >> > > > > > >> > > > > > > > These
> > > > > > > > >> > > > > > >> > > > > > > > > > > were
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> all
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> part of
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > > the
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > > > merged
> PR
> > > at
> > > > > some
> > > > > > > > >> point in
> > > > > > > > >> > > time
> > > > > > > > >> > > > > > and
> > > > > > > > >> > > > > > >> > were
> > > > > > > > >> > > > > > >> > > > > > removed
> > > > > > > > >> > > > > > >> > > > > > > to
> > > > > > > > >> > > > > > >> > > > > > > > > > > reduce
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> scope.
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > > >
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > > > Please
> > let
> > > me
> > > > > > know
> > > > > > > what
> > > > > > > > >> > you
> > > > > > > > >> > > > > think.
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > > >
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > > >
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > > >   1.  A
> > > REST
> > > > > API
> > > > > > to
> > > > > > > > >> > paginate
> > > > > > > > >> > > > > > through
> > > > > > > > >> > > > > > >> > > table
> > > > > > > > >> > > > > > >> > > > > > > metrics.
> > > > > > > > >> > > > > > >> > > > > > > > > > This
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> will be
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > non-IRC
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > > >
> standard
> > > > > > addition.
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > > >   2.
> > > Utilities
> > > > > > for
> > > > > > > > >> > > managing old
> > > > > > > > >> > > > > > >> > records,
> > > > > > > > >> > > > > > >> > > > > > should
> > > > > > > > >> > > > > > >> > > > > > > be
> > > > > > > > >> > > > > > >> > > > > > > > > > > common
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> with
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> events.
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > > > There
> was
> > > some
> > > > > > > > >> discussion
> > > > > > > > >> > > that
> > > > > > > > >> > > > > it
> > > > > > > > >> > > > > > >> > belongs
> > > > > > > > >> > > > > > >> > > > to
> > > > > > > > >> > > > > > >> > > > > > the
> > > > > > > > >> > > > > > >> > > > > > > > CLI.
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > > >   3.
> > > Separate
> > > > > > > > >> datasource
> > > > > > > > >> > > > > > (metrics,
> > > > > > > > >> > > > > > >> > > events,
> > > > > > > > >> > > > > > >> > > > > > even
> > > > > > > > >> > > > > > >> > > > > > > > > other
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> tables?).
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > > >
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > > >
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > > >
> Anything
> > > else?
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > > >
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > > > -
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > > > Anand
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > > >
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > > >
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > >
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> >
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> >
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>>
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >>
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >>
> > > > > > > > >> > > > > > >> > > > > > > > > > > > >
> > > > > > > > >> > > > > > >> > > > > > > > > > > >
> > > > > > > > >> > > > > > >> > > > > > > > > > >
> > > > > > > > >> > > > > > >> > > > > > > > > >
> > > > > > > > >> > > > > > >> > > > > > > > >
> > > > > > > > >> > > > > > >> > > > > > > >
> > > > > > > > >> > > > > > >> > > > > > >
> > > > > > > > >> > > > > > >> > > > > >
> > > > > > > > >> > > > > > >> > > > >
> > > > > > > > >> > > > > > >> > > >
> > > > > > > > >> > > > > > >> > >
> > > > > > > > >> > > > > > >> >
> > > > > > > > >> > > > > > >>
> > > > > > > > >> > > > > > >
> > > > > > > > >> > > > > >
> > > > > > > > >> > > > >
> > > > > > > > >> > >
> > > > > > > > >> >
> > > > > > > > >> >
> > > > > > > > >>
> > > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > >
> >
>

Re: Proposal for REST endpoints for table metrics and events

[Yufei Gu <fl...@gmail.com>]

Thanks for the reminder, Dmitri! I will take a look.

Yufei


On Wed, Oct 7, 2026 at 5:37 PM Dmitri Bourlatchkov <di...@apache.org> wrote:

> Hi All,
>
> Metrics grants have been merged as a separate PR.
>
> Now [4115] is ready for one more review round. IMHO, it is ready to merge.
>
> [4115] https://github.com/apache/polaris/pull/4115
>
> Cheers,
> Dmitri.
>
> On Fri, Oct 2, 2026 at 1:01 AM Jean-Baptiste Onofré <jb...@nanthrax.net>
> wrote:
>
> > Hi all,
> >
> > I agree with Yufei: I would keep TABLE_READ_METRICS as a separate,
> > explicit grant, not implied by TABLE_FULL_METADATA.
> >
> > I believe metrics reports are closer to an activity log than to table
> > metadata. I also think it's consistent with what we have today, as
> > reporting scan and commit metrics is already with TABLE_READ_DATA and
> > TABLE_WRITE_DATA, not by "metadata" privilege.
> >
> > For the troubleshooting use case, the cost is just one additional
> > grant, I think it's acceptable :) I would still let
> > CATALOG_MANAGE_CONTENT implies it, as that privilege already implies
> > TABLE_READ_DATA.
> >
> > Concretely for PR #5669, I propose:
> > - to remove the "Implied by TABLE_FULL_METADATA" statement from the
> > javadoc and CHANGELOG
> > - to add SUPER_PRIVILEGES entry in the PR, so the grant change is
> > self-contained
> > - to drop ICEBERG_TABLE restriction.
> > - to update access-control.md documenting TABLE_READ_METRICS, etc
> >
> > I will comment directly in the PR.
> >
> > Regards
> > JB
> >
> > On Fri, Oct 2, 2026 at 3:40 AM Yufei Gu <fl...@gmail.com> wrote:
> > >
> > > Thanks Dmitri for the reminder.
> > >
> > > PR #5659 <https://github.com/apache/polaris/pull/5659> adds
> > > TABLE_READ_METRICS and says it is implied by TABLE_FULL_METADATA.
> Should
> > > that be the default?
> > >
> > > I’m leaning toward keeping it separate. Metrics describe workloads
> (Spark
> > > ETL, Flink streaming, etc) using a table and can include query filters,
> > so
> > > someone managing table metadata may not need access to other users’
> query
> > > activity.
> > >
> > > Including it could make troubleshooting easier for table admins,
> though.
> > > Are there use cases that favor automatic access over an explicit grant?
> > > Yufei
> > >
> > >
> > > On Wed, Sep 30, 2026 at 1:57 PM Dmitri Bourlatchkov <di...@apache.org>
> > > wrote:
> > >
> > > > Hi Yufei,
> > > >
> > > > Here's the separate PR for Metrics-related grants:
> > > > https://github.com/apache/polaris/pull/5659
> > > >
> > > > Please review it to unblock this contribution.
> > > >
> > > > Thanks,
> > > > Dmitri.
> > > >
> > > > On Wed, Sep 30, 2026 at 1:05 PM Yufei Gu <fl...@gmail.com>
> wrote:
> > > >
> > > > > From the previous email, we agreed to split the REST change from
> the
> > > > grant
> > > > > change. I think the PR is ready to go if we only keep the grant
> > changes.
> > > > >
> > > > > Yufei
> > > > >
> > > > >
> > > > > On Tue, Sep 29, 2026 at 9:26 PM Jean-Baptiste Onofré <
> > jb@nanthrax.net>
> > > > > wrote:
> > > > >
> > > > > > Hi guys,
> > > > > >
> > > > > > I agree with Dmitri: +1 to merge PR #4115 on Oct 1 (tomorrow).
> > > > > >
> > > > > > From my side, all outstanding feedback has been addressed,
> > including
> > > > > > the final documentation points raised by EJ regarding the
> empty-200
> > > > > > NoOp behavior and reconstructed payload description. The branch
> is
> > > > > > clean, up to date with main, and CI is green.
> > > > > >
> > > > > > Once this lands, we can follow up with PR #4756 for the JDBC
> query
> > SPI
> > > > > > implementation.
> > > > > >
> > > > > > Regards
> > > > > > JB
> > > > > >
> > > > > > On Wed, Sep 30, 2026 at 1:53 AM Dmitri Bourlatchkov <
> > dimas@apache.org>
> > > > > > wrote:
> > > > > > >
> > > > > > > Hi All,
> > > > > > >
> > > > > > > Any remaining blockers on [4115]?
> > > > > > >
> > > > > > > Given that the review was very long, please restate any
> remaining
> > > > > > concerns
> > > > > > > (here or in GH).
> > > > > > >
> > > > > > > Otherwise, I propose merging [4115] on Oct 1.
> > > > > > >
> > > > > > > [4115] https://github.com/apache/polaris/pull/4115
> > > > > > >
> > > > > > > Thanks,
> > > > > > > Dmitri.
> > > > > > >
> > > > > > > On Wed, Sep 23, 2026 at 7:53 PM Dmitri Bourlatchkov <
> > > > dimas@apache.org>
> > > > > > > wrote:
> > > > > > >
> > > > > > > > Hi EJ,
> > > > > > > >
> > > > > > > > Thanks for the meeting recap!
> > > > > > > >
> > > > > > > > All:
> > > > > > > >
> > > > > > > > Worth noting: PR [4115] also contains Management API changes
> > that
> > > > > were
> > > > > > not
> > > > > > > > anticipated initially and did not get an API change review
> > until
> > > > now.
> > > > > > These
> > > > > > > > are small changes to add new grant types for metrics RBAC..
> > > > > > > >
> > > > > > > > I've reviewed and approved PR [4115] in GH.
> > > > > > > >
> > > > > > > > [4115] https://github.com/apache/polaris/pull/4115
> > > > > > > >
> > > > > > > > Cheers,
> > > > > > > > Dmitri.
> > > > > > > >
> > > > > > > > On Wed, Aug 12, 2026 at 5:21 PM EJ Wang <
> > > > > > ej.wang.dev.interest@gmail.com>
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > >> Hi folks,
> > > > > > > >>
> > > > > > > >> Notes from today's metrics architecture sync are in the
> usual
> > doc:
> > > > > > > >>
> > > > > > > >>
> > > > > >
> > > > >
> > > >
> >
> https://docs.google.com/document/d/100h7c4damrUzVuquYbBHM0EvA4LSWuW2IT2dN_7nYVA/edit?pli=1&tab=t.18iv1m7p6i6b
> > > > > > > >>
> > > > > > > >> Highlights:
> > > > > > > >>
> > > > > > > >> Module placement. core/ now holds the entity data model
> only.
> > A
> > > > new
> > > > > > spi/
> > > > > > > >> takes every other shared contract: feature SPIs, feature
> > non-SPI
> > > > > > > >> contracts,
> > > > > > > >> substrate SPIs, substrate non-SPI contracts, and
> > contract-required
> > > > > > types.
> > > > > > > >> That settles the question PR #5204 reopened, by placing into
> > a new
> > > > > SPI
> > > > > > > >> module. Note that it changes slightly towards the referenced
> > WIP
> > > > RFC
> > > > > > [1]
> > > > > > > >> recommendations: what proposed to place in core/, now should
> > go to
> > > > > > spi/
> > > > > > > >> except for the entity data model.
> > > > > > > >>
> > > > > > > >> Metrics PRs. #5068 merged. #5204 needs reshaping onto the
> > split.
> > > > > > #4115 and
> > > > > > > >> #4756 stay open with action on the author to follow up on
> > review
> > > > > > comments.
> > > > > > > >>
> > > > > > > >> [1]
> > > > > > > >>
> > > > > > > >>
> > > > > >
> > > > >
> > > >
> >
> https://docs.google.com/document/d/1mj3vGpix9zTDTFFJKtWT5ZxI3PTGI1YNb5xFqyWrOkc/edit?tab=t.0
> > > > > > > >>
> > > > > > > >> On Wed, Jul 15, 2026 at 11:01 AM Anand Kumar Sankaran via
> dev
> > <
> > > > > > > >> dev@polaris.apache.org> wrote:
> > > > > > > >>
> > > > > > > >> > Thanks EJ.  My plan is to introduce PR0 in a new branch /
> > PR and
> > > > > > > >> repurpose
> > > > > > > >> > #4115 and #4756 for PR1 and PR2 to maintain all the
> valuable
> > > > > > discussions
> > > > > > > >> > that happened there.
> > > > > > > >> >
> > > > > > > >> > From: EJ Wang <ej...@gmail.com>
> > > > > > > >> > Date: Wednesday, July 15, 2026 at 10:41 AM
> > > > > > > >> > To: dev@polaris.apache.org <de...@polaris.apache.org>
> > > > > > > >> > Subject: Re: Proposal for REST endpoints for table metrics
> > and
> > > > > > events
> > > > > > > >> >
> > > > > > > >> > Hi folks, Thanks again for the discussion today. I updated
> > the
> > > > > sync
> > > > > > doc
> > > > > > > >> > with notes from today's metrics sync: https: //urldefense.
> > > > > > > >> com/v3/__https:
> > > > > > > >> > //docs. google.
> > > > > > > >> >
> > > > > > > >>
> > > > > >
> > > > >
> > > >
> >
> com/document/d/100h7c4damrUzVuquYbBHM0EvA4LSWuW2IT2dN_7nYVA/edit?pli=1&tab=t.
> > > > > > > >> >
> > > > > > > >>
> > > > > >
> > > > >
> > > >
> >
> 2ahd7wze5f__;!!Iz9xO38YGHZK!4rnlVIqBQQQ6xpGe1PMV58edOS_cGrKoqdfhw4qSp_JQCavvEjaxXk0OwGPnxyEcO60AzKQK_XDtp7H3DdkFXcrXJwAz1g0$
> > > > > > > >> >
> > > > > > > >> >
> > > > > > > >> > Hi folks,
> > > > > > > >> >
> > > > > > > >> > Thanks again for the discussion today. I updated the sync
> > doc
> > > > with
> > > > > > notes
> > > > > > > >> > from today's metrics sync:
> > > > > > > >> >
> > > > > > > >> >
> > > > > > > >>
> > > > > >
> > > > >
> > > >
> >
> https://urldefense.com/v3/__https://docs.google.com/document/d/100h7c4damrUzVuquYbBHM0EvA4LSWuW2IT2dN_7nYVA/edit?pli=1&tab=t.2ahd7wze5f__;!!Iz9xO38YGHZK!4rnlVIqBQQQ6xpGe1PMV58edOS_cGrKoqdfhw4qSp_JQCavvEjaxXk0OwGPnxyEcO60AzKQK_XDtp7H3DdkFXcrXJwAz1g0$
> > > > > > > >> >
> > > > > > > >> > See highlights below:
> > > > > > > >> >
> > > > > > > >> > In previous meetings, EJ proposed a two prs framing to
> push
> > > > > forward
> > > > > > the
> > > > > > > >> > metrics query work in polaris. Dmitri pointed out a subtle
> > point
> > > > > > that
> > > > > > > >> pr2
> > > > > > > >> > is currently also trying to do the persistence related
> code
> > > > > > placement
> > > > > > > >> > refactor. For the sake of straightforward review process,
> > EJ,
> > > > > > Dmitri and
> > > > > > > >> > Anand (the author) agreed to separate the persistence
> > > > refactoring
> > > > > > into
> > > > > > > >> pr0,
> > > > > > > >> > and reframing the pr scopes as follow:
> > > > > > > >> >
> > > > > > > >> >    1.
> > > > > > > >> >
> > > > > > > >> >    Pr0: refactor irc metrics spi
> > > > > > > >> >    1.
> > > > > > > >> >
> > > > > > > >> >       SPI should stay in core/ with metric type enveloped
> > > > > > > >> >       2.
> > > > > > > >> >
> > > > > > > >> >       default spi impl, no-op, in extensions/ - default
> > > > selection
> > > > > at
> > > > > > > >> >       runtime/
> > > > > > > >> >       3.
> > > > > > > >> >
> > > > > > > >> >       move metrics persistence and correspondings to
> > extensions/
> > > > > as
> > > > > > an
> > > > > > > >> >       alternative SPI impl
> > > > > > > >> >       2.
> > > > > > > >> >
> > > > > > > >> >    Pr1: rest api for query
> > > > > > > >> >
> > > > > > > >> >
> > > > > > > >>
> > > > > >
> > > > >
> > > >
> >
> https://urldefense.com/v3/__https://github.com/apache/polaris/pull/4115/changes__;!!Iz9xO38YGHZK!4rnlVIqBQQQ6xpGe1PMV58edOS_cGrKoqdfhw4qSp_JQCavvEjaxXk0OwGPnxyEcO60AzKQK_XDtp7H3DdkFXcrXFCbEbRk$
> > > > > > > >> >    1.
> > > > > > > >> >
> > > > > > > >> >       rest api spec in extensions/ as optional spec
> > > > > > > >> >       2.
> > > > > > > >> >
> > > > > > > >> >       SPI in extensions/ as optional spec spi
> > > > > > > >> >       3.
> > > > > > > >> >
> > > > > > > >> >       default spi impl in extensions/ no-op
> > > > > > > >> >       4.
> > > > > > > >> >
> > > > > > > >> >       handler related in runtime/ - thin adaptor from http
> > shape
> > > > > to
> > > > > > spi
> > > > > > > >> >       shape
> > > > > > > >> >       3.
> > > > > > > >> >
> > > > > > > >> >    Pr2: jdbc impl for query
> > > > > > > >> >
> > > > > > > >>
> > > > > >
> > > > >
> > > >
> >
> https://urldefense.com/v3/__https://github.com/apache/polaris/pull/4756__;!!Iz9xO38YGHZK!4rnlVIqBQQQ6xpGe1PMV58edOS_cGrKoqdfhw4qSp_JQCavvEjaxXk0OwGPnxyEcO60AzKQK_XDtp7H3DdkFXcrXRVMBzqE$
> > > > > > > >> >    1.
> > > > > > > >> >
> > > > > > > >> >       alternative spi impl in extensions/ -> query with
> jdbc
> > > > > > > >> >
> > > > > > > >> >
> > > > > > > >> > Thanks,
> > > > > > > >> > -ej
> > > > > > > >> >
> > > > > > > >> > On Wed, Jul 1, 2026 at 8:01 AM Alexandre Dutra <
> > > > adutra@apache.org
> > > > > >
> > > > > > > >> wrote:
> > > > > > > >> >
> > > > > > > >> > > Hi all,
> > > > > > > >> > >
> > > > > > > >> > > > While the current implementation stores metrics in the
> > > > > > metastore,
> > > > > > > >> other
> > > > > > > >> > > deployments
> > > > > > > >> > > may choose a KV store, a dedicated metrics warehouse, or
> > > > another
> > > > > > > >> backend
> > > > > > > >> > > entirely. In those cases, exposing a Polaris REST API
> > over a
> > > > > > specific
> > > > > > > >> > > schema feels like coupling the API to an implementation
> > > > detail.
> > > > > > > >> > >
> > > > > > > >> > > I wouldn't say to an implementation detail, but rather
> to
> > a
> > > > > > particular
> > > > > > > >> > > setup / configuration. I think that's fine. If you don't
> > store
> > > > > > > >> > > metrics, you don't read metrics. Similar situations will
> > > > > certainly
> > > > > > > >> > > happen in the future, e.g. if we ever introduce an API
> to
> > read
> > > > > > events,
> > > > > > > >> > > that would only work if events are being persisted in
> the
> > > > first
> > > > > > place.
> > > > > > > >> > > Again I think that's a fair trade-off.
> > > > > > > >> > >
> > > > > > > >> > > Thanks,
> > > > > > > >> > > Alex
> > > > > > > >> > >
> > > > > > > >> > > On Wed, Jul 1, 2026 at 7:23 AM Dmitri Bourlatchkov <
> > > > > > dimas@apache.org>
> > > > > > > >> > > wrote:
> > > > > > > >> > > >
> > > > > > > >> > > > Hi Yufei,
> > > > > > > >> > > >
> > > > > > > >> > > > From my POV, it is fine for Polaris to have multiple
> > REST
> > > > API
> > > > > > > >> surfaces
> > > > > > > >> > > with
> > > > > > > >> > > > different features and different degrees of
> "readiness".
> > > > Each
> > > > > > API
> > > > > > > >> can
> > > > > > > >> > > > evolve fairly independently. With proper
> modularization
> > > > (which
> > > > > > Anand
> > > > > > > >> > > did),
> > > > > > > >> > > > changes to the main Polaris code and these extra
> modules
> > > > > should
> > > > > > > >> > normally
> > > > > > > >> > > be
> > > > > > > >> > > > non-intrusive to each other.
> > > > > > > >> > > >
> > > > > > > >> > > > I assume Anand is going to be the first user of this
> > API.
> > > > > > However,
> > > > > > > >> if
> > > > > > > >> > we
> > > > > > > >> > > do
> > > > > > > >> > > > not merge it we'll never know whether other users are
> > > > > > interested. I
> > > > > > > >> do
> > > > > > > >> > > not
> > > > > > > >> > > > think many end users read the dev ML, but they
> certainly
> > > > read
> > > > > > the
> > > > > > > >> docs,
> > > > > > > >> > > > code and release notes.
> > > > > > > >> > > >
> > > > > > > >> > > > There is no risk for Polaris as a project in exposing
> > this
> > > > > > > >> experimental
> > > > > > > >> > > > API. If usage dwindles over time, we can remove it. If
> > it
> > > > > gains
> > > > > > a
> > > > > > > >> > > > considerable user / contributor base, we will promote
> > it to
> > > > a
> > > > > > > >> > > "first-class"
> > > > > > > >> > > > (non-beta) API.
> > > > > > > >> > > >
> > > > > > > >> > > > Cheers,
> > > > > > > >> > > > Dmitri.
> > > > > > > >> > > >
> > > > > > > >> > > > On Tue, Jun 30, 2026 at 7:53 PM Yufei Gu <
> > > > > flyrain000@gmail.com>
> > > > > > > >> wrote:
> > > > > > > >> > > >
> > > > > > > >> > > > > Hi all,
> > > > > > > >> > > > >
> > > > > > > >> > > > > Thanks for driving this work forward. Building this
> > as a
> > > > POC
> > > > > > makes
> > > > > > > >> > > perfect
> > > > > > > >> > > > > sense if the goal is to validate the idea and gather
> > > > > feedback.
> > > > > > > >> Should
> > > > > > > >> > > we
> > > > > > > >> > > > > merge a POC into the main branch before agreeing on
> > the
> > > > > > overall
> > > > > > > >> > > direction?
> > > > > > > >> > > > > To me, that feels like a separate discussion from
> > whether
> > > > > the
> > > > > > > >> > > > > implementation itself looks reasonable.
> > > > > > > >> > > > >
> > > > > > > >> > > > > One concern that I don't think has been fully
> > addressed is
> > > > > > > >> whether we
> > > > > > > >> > > need
> > > > > > > >> > > > > to expose metrics consumption through a Polaris REST
> > > > > endpoint
> > > > > > at
> > > > > > > >> all.
> > > > > > > >> > > > >
> > > > > > > >> > > > > My concern is less about the API itself and more
> > about the
> > > > > > > >> boundary
> > > > > > > >> > > Polaris
> > > > > > > >> > > > > should own. The proposed REST API is tied to a
> > particular
> > > > > > > >> persistence
> > > > > > > >> > > > > schema, but that schema is just one possible
> > > > implementation.
> > > > > > While
> > > > > > > >> > the
> > > > > > > >> > > > > current implementation stores metrics in the
> > metastore,
> > > > > other
> > > > > > > >> > > deployments
> > > > > > > >> > > > > may choose a KV store, a dedicated metrics
> warehouse,
> > or
> > > > > > another
> > > > > > > >> > > backend
> > > > > > > >> > > > > entirely. In those cases, exposing a Polaris REST
> API
> > > > over a
> > > > > > > >> specific
> > > > > > > >> > > > > schema feels like coupling the API to an
> > implementation
> > > > > > detail.
> > > > > > > >> > > > >
> > > > > > > >> > > > > Polaris isn't intended to become a full metrics
> > system.
> > > > I'd
> > > > > > > >> expect it
> > > > > > > >> > > to
> > > > > > > >> > > > > focus on producing metrics and let deployments
> choose
> > the
> > > > > most
> > > > > > > >> > > appropriate
> > > > > > > >> > > > > storage and consumption model for their environment.
> > > > > > > >> > > > >
> > > > > > > >> > > > > I'm not opposed to exploring this direction, but we
> > should
> > > > > > first
> > > > > > > >> > align
> > > > > > > >> > > on
> > > > > > > >> > > > > whether exposing a REST API through Polaris is the
> > right
> > > > > long
> > > > > > term
> > > > > > > >> > > approach
> > > > > > > >> > > > > before merging it into the main branch.
> > > > > > > >> > > > >
> > > > > > > >> > > > > Thanks,
> > > > > > > >> > > > > Yufei
> > > > > > > >> > > > >
> > > > > > > >> > > > > Yufei
> > > > > > > >> > > > >
> > > > > > > >> > > > >
> > > > > > > >> > > > > On Mon, Jun 29, 2026 at 9:27 AM Dmitri Bourlatchkov
> <
> > > > > > > >> > dimas@apache.org>
> > > > > > > >> > > > > wrote:
> > > > > > > >> > > > >
> > > > > > > >> > > > > > Hi All,
> > > > > > > >> > > > > >
> > > > > > > >> > > > > > It looks like Anand addressed all action items in
> > > > [4115].
> > > > > > > >> > > > > >
> > > > > > > >> > > > > > Please review the latest state of the code. The PR
> > LGTM
> > > > > and
> > > > > > I
> > > > > > > >> > > approved in
> > > > > > > >> > > > > > GH from my side.
> > > > > > > >> > > > > >
> > > > > > > >> > > > > > [4115]
> > > > > > > >> >
> > > > > > > >>
> > > > > >
> > > > >
> > > >
> >
> https://urldefense.com/v3/__https://github.com/apache/polaris/pull/4115__;!!Iz9xO38YGHZK!4rnlVIqBQQQ6xpGe1PMV58edOS_cGrKoqdfhw4qSp_JQCavvEjaxXk0OwGPnxyEcO60AzKQK_XDtp7H3DdkFXcrXtFRscKs$
> > > > > > > >> > > > > >
> > > > > > > >> > > > > > Thanks,
> > > > > > > >> > > > > > Dmitri.
> > > > > > > >> > > > > >
> > > > > > > >> > > > > > On Thu, Jun 25, 2026 at 1:56 PM Dmitri
> Bourlatchkov
> > <
> > > > > > > >> > > dimas@apache.org>
> > > > > > > >> > > > > > wrote:
> > > > > > > >> > > > > >
> > > > > > > >> > > > > > > Hi All,
> > > > > > > >> > > > > > >
> > > > > > > >> > > > > > > Recap of the discussion in today's Community
> Sync:
> > > > > > > >> > > > > > >
> > > > > > > >> > > > > > > * The modular approach for Metrics SPI(s) and
> > > > > > implementation
> > > > > > > >> > seems
> > > > > > > >> > > to
> > > > > > > >> > > > > > have
> > > > > > > >> > > > > > > community consensus (please comment if you still
> > have
> > > > > > > >> concerns)
> > > > > > > >> > > > > > >
> > > > > > > >> > > > > > > * What modules / feature flags are enabled in
> > Apache
> > > > > > Polaris
> > > > > > > >> > > images by
> > > > > > > >> > > > > > > default will be discussed later (when the
> feature
> > is
> > > > > > close to
> > > > > > > >> > > > > > completion).
> > > > > > > >> > > > > > >
> > > > > > > >> > > > > > > In any case, users will be able to turn the full
> > > > feature
> > > > > > > >> on/off
> > > > > > > >> > at
> > > > > > > >> > > > > their
> > > > > > > >> > > > > > > discretion, regardless of the default flag
> values.
> > > > > > > >> > > > > > >
> > > > > > > >> > > > > > > * It is generally preferable to have smaller PRs
> > and
> > > > > > introduce
> > > > > > > >> > new
> > > > > > > >> > > > > > > features gradually to simplify reviews.
> > > > > > > >> > > > > > >
> > > > > > > >> > > > > > > What this means for the Metrics API, IMHO is:
> > > > > > > >> > > > > > >
> > > > > > > >> > > > > > > a) The scope of PR 4115 is probably ok as it
> > stands
> > > > > right
> > > > > > > >> now. If
> > > > > > > >> > > > > someone
> > > > > > > >> > > > > > > wants to propose splitting it further, please
> > comment.
> > > > > > > >> > > > > > >
> > > > > > > >> > > > > > > b) After 4115 is merged, Metrics Persistence PRs
> > will
> > > > > > follow,
> > > > > > > >> > with
> > > > > > > >> > > > > > smaller
> > > > > > > >> > > > > > > scope per PR as appropriate for the nature of
> the
> > code
> > > > > > > >> changes.
> > > > > > > >> > > > > > >
> > > > > > > >> > > > > > > * The major concern was about exposing the new
> > Metrics
> > > > > > REST
> > > > > > > >> API
> > > > > > > >> > to
> > > > > > > >> > > end
> > > > > > > >> > > > > > > users. The nature of this concern was about
> > avoiding
> > > > the
> > > > > > > >> (false)
> > > > > > > >> > > > > > impression
> > > > > > > >> > > > > > > that the new API is a firm and adopted Polaris
> > > > > > specification.
> > > > > > > >> > > > > > >
> > > > > > > >> > > > > > > I think this concern is reasonable.
> > > > > > > >> > > > > > >
> > > > > > > >> > > > > > > I propose that Anand update the documentation
> > sections
> > > > > of
> > > > > > PR
> > > > > > > >> 4115
> > > > > > > >> > > to
> > > > > > > >> > > > > > > emphasise that the API is currently an
> > experimental /
> > > > > POC
> > > > > > API
> > > > > > > >> at
> > > > > > > >> > > this
> > > > > > > >> > > > > > time
> > > > > > > >> > > > > > > and is subject to change (including breaking
> > > > changes). I
> > > > > > > >> believe
> > > > > > > >> > it
> > > > > > > >> > > > > > already
> > > > > > > >> > > > > > > has the "beta" label. So this request is mostly
> > for
> > > > doc
> > > > > > > >> changes
> > > > > > > >> > to
> > > > > > > >> > > > > > clarify
> > > > > > > >> > > > > > > user expectations, since the "beta" label alone
> > may
> > > > not
> > > > > be
> > > > > > > >> > > interpreted
> > > > > > > >> > > > > > the
> > > > > > > >> > > > > > > same way by all users.
> > > > > > > >> > > > > > >
> > > > > > > >> > > > > > > I hope this update will resolve all remaining
> > concerns
> > > > > > with
> > > > > > > >> 4115
> > > > > > > >> > > and
> > > > > > > >> > > > > > allow
> > > > > > > >> > > > > > > for it to be merged so that the community will
> be
> > able
> > > > > to
> > > > > > > >> proceed
> > > > > > > >> > > to
> > > > > > > >> > > > > > > follow-up PRs (as noted above).
> > > > > > > >> > > > > > >
> > > > > > > >> > > > > > > Please add/correct me if I missed anything
> > relevant.
> > > > > > > >> > > > > > >
> > > > > > > >> > > > > > > Thanks,
> > > > > > > >> > > > > > > Dmitri.
> > > > > > > >> > > > > > >
> > > > > > > >> > > > > > > On Wed, Jun 24, 2026 at 8:11 PM Yufei Gu <
> > > > > > > >> flyrain000@gmail.com>
> > > > > > > >> > > wrote:
> > > > > > > >> > > > > > >
> > > > > > > >> > > > > > >> I do not think we have agreement on the query
> > REST
> > > > APIs
> > > > > > yet.
> > > > > > > >> > > > > > >>
> > > > > > > >> > > > > > >> In particular, the current query API shape
> seems
> > > > highly
> > > > > > > >> coupled
> > > > > > > >> > > to the
> > > > > > > >> > > > > > >> *example[1]* metrics in the Iceberg REST spec.
> I
> > do
> > > > not
> > > > > > think
> > > > > > > >> > > that is
> > > > > > > >> > > > > a
> > > > > > > >> > > > > > >> good way to design a REST API contract. For
> > example,
> > > > > > fields
> > > > > > > >> like
> > > > > > > >> > > > > > >> resultDataFiles, resultDeleteFiles,
> > > > totalFileSizeBytes,
> > > > > > > >> > > > > > >> totalDataManifests,
> > > > > > > >> > > > > > >> totalDeleteManifests, and scannedDataManifests
> > are
> > > > very
> > > > > > > >> specific
> > > > > > > >> > > to
> > > > > > > >> > > > > one
> > > > > > > >> > > > > > >> style of scan metric. That makes the API look
> > more
> > > > > like a
> > > > > > > >> direct
> > > > > > > >> > > > > > >> projection
> > > > > > > >> > > > > > >> of the current example payload than a general
> > query
> > > > > > model.
> > > > > > > >> > > > > > >>
> > > > > > > >> > > > > > >> My preference is that Polaris should not define
> > or
> > > > > > strongly
> > > > > > > >> > opine
> > > > > > > >> > > on
> > > > > > > >> > > > > the
> > > > > > > >> > > > > > >> Iceberg scan or commit metrics query REST API
> at
> > this
> > > > > > point.
> > > > > > > >> > > Polaris
> > > > > > > >> > > > > can
> > > > > > > >> > > > > > >> resolve and authorize the table, accept the
> > metrics
> > > > > > report,
> > > > > > > >> and
> > > > > > > >> > > > > delegate
> > > > > > > >> > > > > > >> it
> > > > > > > >> > > > > > >> to the configured reporting path. Durable
> > storage,
> > > > > > filtering,
> > > > > > > >> > > > > > dashboards,
> > > > > > > >> > > > > > >> and query APIs should remain downstream impl.
> > choices
> > > > > > unless
> > > > > > > >> and
> > > > > > > >> > > until
> > > > > > > >> > > > > > the
> > > > > > > >> > > > > > >> Iceberg community standardizes a query
> contract,
> > > > which
> > > > > I
> > > > > > > >> doubt
> > > > > > > >> > > will
> > > > > > > >> > > > > > >> happen.
> > > > > > > >> > > > > > >>
> > > > > > > >> > > > > > >> So I do not think the intake path has to be
> > blocked,
> > > > > but
> > > > > > I
> > > > > > > >> would
> > > > > > > >> > > > > prefer
> > > > > > > >> > > > > > >> not
> > > > > > > >> > > > > > >> to merge a Polaris specific query API that may
> be
> > > > hard
> > > > > to
> > > > > > > >> evolve
> > > > > > > >> > > > > later.
> > > > > > > >> > > > > > >>
> > > > > > > >> > > > > > >> 1.
> > > > > > > >> > > > > > >>
> > > > > > > >> > > > > > >>
> > > > > > > >> > > > > >
> > > > > > > >> > > > >
> > > > > > > >> > >
> > > > > > > >> >
> > > > > > > >>
> > > > > >
> > > > >
> > > >
> >
> https://urldefense.com/v3/__https://github.com/apache/iceberg/blob/2a6c556c0c883c04c6d3fbd68e6aeda36b91e0aa/open-api/rest-catalog-open-api.yaml*L4152__;Iw!!Iz9xO38YGHZK!4rnlVIqBQQQ6xpGe1PMV58edOS_cGrKoqdfhw4qSp_JQCavvEjaxXk0OwGPnxyEcO60AzKQK_XDtp7H3DdkFXcrXCDsqYGo$
> > > > > > > >> > > > > > >>
> > > > > > > >> > > > > > >> Yufei
> > > > > > > >> > > > > > >>
> > > > > > > >> > > > > > >>
> > > > > > > >> > > > > > >> On Tue, Jun 23, 2026 at 9:30 AM Dmitri
> > Bourlatchkov <
> > > > > > > >> > > dimas@apache.org
> > > > > > > >> > > > > >
> > > > > > > >> > > > > > >> wrote:
> > > > > > > >> > > > > > >>
> > > > > > > >> > > > > > >> > Hi All,
> > > > > > > >> > > > > > >> >
> > > > > > > >> > > > > > >> > So, what are the blockers for merging [4115],
> > if
> > > > any?
> > > > > > > >> > > > > > >> >
> > > > > > > >> > > > > > >> > [4115]
> > > > > > > >> >
> > > > > > > >>
> > > > > >
> > > > >
> > > >
> >
> https://urldefense.com/v3/__https://github.com/apache/polaris/pull/4115__;!!Iz9xO38YGHZK!4rnlVIqBQQQ6xpGe1PMV58edOS_cGrKoqdfhw4qSp_JQCavvEjaxXk0OwGPnxyEcO60AzKQK_XDtp7H3DdkFXcrXtFRscKs$
> > > > > > > >> > > > > > >> >
> > > > > > > >> > > > > > >> > Thanks,
> > > > > > > >> > > > > > >> > Dmitri.
> > > > > > > >> > > > > > >> >
> > > > > > > >> > > > > > >> > On Wed, Jun 17, 2026 at 5:28 PM EJ Wang <
> > > > > > > >> > > > > > ej.wang.dev.interest@gmail.com
> > > > > > > >> > > > > > >> >
> > > > > > > >> > > > > > >> > wrote:
> > > > > > > >> > > > > > >> >
> > > > > > > >> > > > > > >> > > Hi folks,
> > > > > > > >> > > > > > >> > >
> > > > > > > >> > > > > > >> > > Thanks again for the discussion today. I
> > updated
> > > > > the
> > > > > > sync
> > > > > > > >> > doc
> > > > > > > >> > > with
> > > > > > > >> > > > > > >> notes
> > > > > > > >> > > > > > >> > > from today's metrics sync:
> > > > > > > >> > > > > > >> > >
> > > > > > > >> > > > > > >> > >
> > > > > > > >> > > > > > >> > >
> > > > > > > >> > > > > > >> >
> > > > > > > >> > > > > > >>
> > > > > > > >> > > > > >
> > > > > > > >> > > > >
> > > > > > > >> > >
> > > > > > > >> >
> > > > > > > >>
> > > > > >
> > > > >
> > > >
> >
> https://urldefense.com/v3/__https://docs.google.com/document/d/100h7c4damrUzVuquYbBHM0EvA4LSWuW2IT2dN_7nYVA/edit?tab=t.ezk0rgdx0c6m__;!!Iz9xO38YGHZK!4rnlVIqBQQQ6xpGe1PMV58edOS_cGrKoqdfhw4qSp_JQCavvEjaxXk0OwGPnxyEcO60AzKQK_XDtp7H3DdkFXcrX83IZkm0$
> > > > > > > >> > > > > > >> > >
> > > > > > > >> > > > > > >> > > A few highlights:
> > > > > > > >> > > > > > >> > >
> > > > > > > >> > > > > > >> > >    - No objection to keeping the default
> > metrics
> > > > > > behavior
> > > > > > > >> > > small:
> > > > > > > >> > > > > > >> > >    no-op/log-only is enough as the built-in
> > > > > default.
> > > > > > > >> > > > > > >> > >    - Durable storage, event forwarding,
> > external
> > > > > > queues,
> > > > > > > >> > > > > dashboards,
> > > > > > > >> > > > > > >> and
> > > > > > > >> > > > > > >> > >    custom filtering should be
> implementation
> > > > > choices
> > > > > > > >> behind
> > > > > > > >> > > the
> > > > > > > >> > > > > > >> metrics
> > > > > > > >> > > > > > >> > >    reporting path.
> > > > > > > >> > > > > > >> > >    - For PR #4115, the REST layer should
> > > > > > > >> > resolve/authz/accept
> > > > > > > >> > > the
> > > > > > > >> > > > > > >> report,
> > > > > > > >> > > > > > >> > >    then delegate to the selected metrics
> > > > > > implementation.
> > > > > > > >> It
> > > > > > > >> > > should
> > > > > > > >> > > > > > not
> > > > > > > >> > > > > > >> > own
> > > > > > > >> > > > > > >> > >    durable storage semantics.
> > > > > > > >> > > > > > >> > >    - The rough boundary we discussed is:
> spec
> > > > owns
> > > > > > the
> > > > > > > >> wire
> > > > > > > >> > > > > > contract,
> > > > > > > >> > > > > > >> > >    runtime owns framework wiring,
> > API/contract
> > > > > > modules
> > > > > > > >> own
> > > > > > > >> > > > > > >> > provider-facing
> > > > > > > >> > > > > > >> > >    contracts, and extensions own
> replaceable
> > > > > > > >> > implementations.
> > > > > > > >> > > > > > >> > >    - OpenLineage has a similar path-naming
> > > > > question.
> > > > > > We
> > > > > > > >> > should
> > > > > > > >> > > > > avoid
> > > > > > > >> > > > > > >> > >    occupying a generic lineage path too
> > early if
> > > > we
> > > > > > want
> > > > > > > >> > room
> > > > > > > >> > > for
> > > > > > > >> > > > > > >> other
> > > > > > > >> > > > > > >> > >    lineage systems later.
> > > > > > > >> > > > > > >> > >
> > > > > > > >> > > > > > >> > > I believe the boundary rules previewed in
> > sync
> > > > > > meeting
> > > > > > > >> are
> > > > > > > >> > > useful
> > > > > > > >> > > > > > for
> > > > > > > >> > > > > > >> > > reviewing PR #4115, but it is not
> > > > > > > >> > > > > > >> > > a final module-layout proposal.
> > > > > > > >> > > > > > >> > >
> > > > > > > >> > > > > > >> > > For metrics specifically, I think the
> target
> > > > > remains:
> > > > > > > >> > > > > > >> > >    runtime
> > > > > > > >> > > > > > >> > >      -> metrics reporting/emitting contract
> > > > > > > >> > > > > > >> > >      -> default implementation
> > > > > > > >> > > > > > >> > >           -> lower implementation details,
> if
> > > > > needed
> > > > > > > >> > > > > > >> > >
> > > > > > > >> > > > > > >> > > That keeps the default simple, while
> allowing
> > > > > durable
> > > > > > > >> JDBC,
> > > > > > > >> > > > > > >> event-backed,
> > > > > > > >> > > > > > >> > > or external-queue implementations to be
> added
> > > > > > separately.
> > > > > > > >> > > > > > >> > >
> > > > > > > >> > > > > > >> > > We also touched on the split between
> Iceberg
> > > > > metrics
> > > > > > > >> > emitting
> > > > > > > >> > > and
> > > > > > > >> > > > > > >> > > Polaris-owned metrics querying. The
> querying
> > side
> > > > > can
> > > > > > > >> keep
> > > > > > > >> > > > > evolving,
> > > > > > > >> > > > > > >> > > especially for dashboard or generic-table
> use
> > > > > cases,
> > > > > > but
> > > > > > > >> I
> > > > > > > >> > do
> > > > > > > >> > > not
> > > > > > > >> > > > > > >> think
> > > > > > > >> > > > > > >> > > that needs to block the intake path in this
> > PR.
> > > > > > > >> > > > > > >> > >
> > > > > > > >> > > > > > >> > > Thanks,
> > > > > > > >> > > > > > >> > > -ej
> > > > > > > >> > > > > > >> > >
> > > > > > > >> > > > > > >> > > On Tue, Jun 16, 2026 at 2:17 PM Yufei Gu <
> > > > > > > >> > > flyrain000@gmail.com>
> > > > > > > >> > > > > > >> wrote:
> > > > > > > >> > > > > > >> > >
> > > > > > > >> > > > > > >> > > > Thanks for chiming in, EJ. Agreed that
> the
> > > > > default
> > > > > > > >> battery
> > > > > > > >> > > > > should
> > > > > > > >> > > > > > >> stay
> > > > > > > >> > > > > > >> > > > small. I'm leaning toward not including
> > metrics
> > > > > > > >> > persistence
> > > > > > > >> > > in
> > > > > > > >> > > > > the
> > > > > > > >> > > > > > >> > > default
> > > > > > > >> > > > > > >> > > > battery.
> > > > > > > >> > > > > > >> > > >
> > > > > > > >> > > > > > >> > > > Yufei
> > > > > > > >> > > > > > >> > > >
> > > > > > > >> > > > > > >> > > >
> > > > > > > >> > > > > > >> > > > On Thu, Jun 11, 2026 at 8:30 PM EJ Wang <
> > > > > > > >> > > > > > >> > ej.wang.dev.interest@gmail.com>
> > > > > > > >> > > > > > >> > > > wrote:
> > > > > > > >> > > > > > >> > > >
> > > > > > > >> > > > > > >> > > > > Hi Yufei,
> > > > > > > >> > > > > > >> > > > >
> > > > > > > >> > > > > > >> > > > > Thanks for connecting this back to the
> > REST
> > > > > > endpoint
> > > > > > > >> > > > > proposal. I
> > > > > > > >> > > > > > >> > > replied
> > > > > > > >> > > > > > >> > > > > with the fuller version on the
> > > > event-forwarding
> > > > > > > >> thread,
> > > > > > > >> > > but
> > > > > > > >> > > > > > >> wanted to
> > > > > > > >> > > > > > >> > > add
> > > > > > > >> > > > > > >> > > > > the shorter version here too.
> > > > > > > >> > > > > > >> > > > >
> > > > > > > >> > > > > > >> > > > > I agree that metrics reporting/emitting
> > is
> > > > the
> > > > > > right
> > > > > > > >> > > > > conceptual
> > > > > > > >> > > > > > >> > > boundary,
> > > > > > > >> > > > > > >> > > > > and I also think the event/listener
> path
> > is a
> > > > > > good
> > > > > > > >> > > > > > implementation
> > > > > > > >> > > > > > >> > > > direction
> > > > > > > >> > > > > > >> > > > > to explore. The distinction I want to
> > keep
> > > > > clear
> > > > > > is
> > > > > > > >> > > default
> > > > > > > >> > > > > > >> battery
> > > > > > > >> > > > > > >> > vs
> > > > > > > >> > > > > > >> > > > > extension implementation.
> > > > > > > >> > > > > > >> > > > >
> > > > > > > >> > > > > > >> > > > > For the REST/API side, I would keep the
> > > > > semantics
> > > > > > > >> > narrow:
> > > > > > > >> > > > > > Polaris
> > > > > > > >> > > > > > >> > > > resolves
> > > > > > > >> > > > > > >> > > > > and authorizes the table, accepts the
> > Iceberg
> > > > > > > >> > scan/commit
> > > > > > > >> > > > > > metrics
> > > > > > > >> > > > > > >> > > report
> > > > > > > >> > > > > > >> > > > > into the configured reporting/emitting
> > path,
> > > > > and
> > > > > > > >> returns
> > > > > > > >> > > 204.
> > > > > > > >> > > > > > That
> > > > > > > >> > > > > > >> > 204
> > > > > > > >> > > > > > >> > > > > should mean Polaris accepted the report
> > into
> > > > > the
> > > > > > > >> > ingestion
> > > > > > > >> > > > > path;
> > > > > > > >> > > > > > >> it
> > > > > > > >> > > > > > >> > > > should
> > > > > > > >> > > > > > >> > > > > not imply durable storage.
> > > > > > > >> > > > > > >> > > > >
> > > > > > > >> > > > > > >> > > > > Durable storage, event forwarding,
> > external
> > > > > > telemetry
> > > > > > > >> > > routing,
> > > > > > > >> > > > > > >> > > filtering,
> > > > > > > >> > > > > > >> > > > > and retention should sit behind that
> > > > > > > >> reporting/emitting
> > > > > > > >> > > > > boundary
> > > > > > > >> > > > > > >> as
> > > > > > > >> > > > > > >> > > > > implementation-layer behavior. A
> durable
> > JDBC
> > > > > > path
> > > > > > > >> can
> > > > > > > >> > be
> > > > > > > >> > > one
> > > > > > > >> > > > > > >> named
> > > > > > > >> > > > > > >> > > > > extension implementation. An
> > event/listener
> > > > > > > >> forwarding
> > > > > > > >> > > path
> > > > > > > >> > > > > can
> > > > > > > >> > > > > > be
> > > > > > > >> > > > > > >> > > > another
> > > > > > > >> > > > > > >> > > > > named extension implementation.
> > > > > > > >> > > > > > >> > > > >
> > > > > > > >> > > > > > >> > > > > So I am +1 on the event-forwarding idea
> > as a
> > > > > > > >> non-default
> > > > > > > >> > > > > > extension
> > > > > > > >> > > > > > >> > > > > implementation. I just would not make
> it
> > the
> > > > > > default
> > > > > > > >> > > battery
> > > > > > > >> > > > > or
> > > > > > > >> > > > > > >> the
> > > > > > > >> > > > > > >> > > core
> > > > > > > >> > > > > > >> > > > > API shape. The default battery can stay
> > small
> > > > > and
> > > > > > > >> safe,
> > > > > > > >> > > while
> > > > > > > >> > > > > > >> > > deployments
> > > > > > > >> > > > > > >> > > > > that need forwarding, dashboards, or
> > durable
> > > > > > storage
> > > > > > > >> can
> > > > > > > >> > > opt
> > > > > > > >> > > > > > into
> > > > > > > >> > > > > > >> the
> > > > > > > >> > > > > > >> > > > > implementation that matches their
> > operational
> > > > > > model.
> > > > > > > >> > > > > > >> > > > >
> > > > > > > >> > > > > > >> > > > > Thanks,
> > > > > > > >> > > > > > >> > > > > -ej
> > > > > > > >> > > > > > >> > > > >
> > > > > > > >> > > > > > >> > > > > On Wed, Jun 10, 2026 at 11:43 AM Yufei
> > Gu <
> > > > > > > >> > > > > flyrain000@gmail.com
> > > > > > > >> > > > > > >
> > > > > > > >> > > > > > >> > > wrote:
> > > > > > > >> > > > > > >> > > > >
> > > > > > > >> > > > > > >> > > > > > Thanks EJ. I agree that the
> > > > > reporting/emitting
> > > > > > > >> > boundary
> > > > > > > >> > > > > feels
> > > > > > > >> > > > > > >> like
> > > > > > > >> > > > > > >> > > the
> > > > > > > >> > > > > > >> > > > > > right SPI boundary, but I wonder if
> we
> > can
> > > > > > simplify
> > > > > > > >> > this
> > > > > > > >> > > > > even
> > > > > > > >> > > > > > >> > > further.
> > > > > > > >> > > > > > >> > > > > >
> > > > > > > >> > > > > > >> > > > > > Iceberg scan and commit metrics look
> > very
> > > > > > similar
> > > > > > > >> to
> > > > > > > >> > > events
> > > > > > > >> > > > > to
> > > > > > > >> > > > > > >> me.
> > > > > > > >> > > > > > >> > > They
> > > > > > > >> > > > > > >> > > > > are
> > > > > > > >> > > > > > >> > > > > > append only, asynchronously consumed,
> > often
> > > > > > > >> forwarded
> > > > > > > >> > to
> > > > > > > >> > > > > > >> external
> > > > > > > >> > > > > > >> > > > > systems,
> > > > > > > >> > > > > > >> > > > > > and have similar retention and
> cleanup
> > > > > > > >> requirements.
> > > > > > > >> > > More
> > > > > > > >> > > > > > >> details
> > > > > > > >> > > > > > >> > can
> > > > > > > >> > > > > > >> > > > be
> > > > > > > >> > > > > > >> > > > > > found in this thread [1]. In that
> case,
> > > > > > Polaris may
> > > > > > > >> > only
> > > > > > > >> > > > > need
> > > > > > > >> > > > > > to
> > > > > > > >> > > > > > >> > > accept
> > > > > > > >> > > > > > >> > > > > the
> > > > > > > >> > > > > > >> > > > > > report and emit an event through the
> > > > existing
> > > > > > event
> > > > > > > >> > > > > framework.
> > > > > > > >> > > > > > >> > > > Filtering,
> > > > > > > >> > > > > > >> > > > > > async delivery, custom sinks, and
> > retention
> > > > > > > >> mechanisms
> > > > > > > >> > > could
> > > > > > > >> > > > > > >> then
> > > > > > > >> > > > > > >> > be
> > > > > > > >> > > > > > >> > > > > shared
> > > > > > > >> > > > > > >> > > > > > instead of introducing a separate
> > metrics
> > > > > > specific
> > > > > > > >> > > extension
> > > > > > > >> > > > > > >> layer
> > > > > > > >> > > > > > >> > > and
> > > > > > > >> > > > > > >> > > > > SPI.
> > > > > > > >> > > > > > >> > > > > >
> > > > > > > >> > > > > > >> > > > > > The main requirement I still see is
> > > > querying
> > > > > > > >> metrics
> > > > > > > >> > > through
> > > > > > > >> > > > > > >> > Polaris.
> > > > > > > >> > > > > > >> > > > If
> > > > > > > >> > > > > > >> > > > > we
> > > > > > > >> > > > > > >> > > > > > want that battery included
> experience,
> > we
> > > > > could
> > > > > > > >> work
> > > > > > > >> > on
> > > > > > > >> > > a
> > > > > > > >> > > > > > >> > persistence
> > > > > > > >> > > > > > >> > > > > > listener implementation that
> > selectively
> > > > > saves
> > > > > > > >> metrics
> > > > > > > >> > > and
> > > > > > > >> > > > > > >> expose
> > > > > > > >> > > > > > >> > it
> > > > > > > >> > > > > > >> > > > via
> > > > > > > >> > > > > > >> > > > > > the IRC event endpoint which seems
> > pretty
> > > > > > close in
> > > > > > > >> > > Iceberg
> > > > > > > >> > > > > > >> > > > > > community. Another option I like more
> > is to
> > > > > > allow
> > > > > > > >> > users
> > > > > > > >> > > to
> > > > > > > >> > > > > > >> develop
> > > > > > > >> > > > > > >> > > > their
> > > > > > > >> > > > > > >> > > > > > own listener, so they can persist
> > > > scan/commit
> > > > > > > >> metrics
> > > > > > > >> > > to any
> > > > > > > >> > > > > > >> > systems
> > > > > > > >> > > > > > >> > > > they
> > > > > > > >> > > > > > >> > > > > > prefer. In general, that feels like
> an
> > > > > > > >> implementation
> > > > > > > >> > > choice
> > > > > > > >> > > > > > on
> > > > > > > >> > > > > > >> top
> > > > > > > >> > > > > > >> > > of
> > > > > > > >> > > > > > >> > > > > the
> > > > > > > >> > > > > > >> > > > > > event framework rather than something
> > that
> > > > > > needs to
> > > > > > > >> > > shape
> > > > > > > >> > > > > the
> > > > > > > >> > > > > > >> core
> > > > > > > >> > > > > > >> > > > > > architecture.
> > > > > > > >> > > > > > >> > > > > >
> > > > > > > >> > > > > > >> > > > > > 1.
> > > > > > > >> > > > > > >> >
> > > > > > > >> > >
> > > > > > > >> >
> > > > > > > >>
> > > > > >
> > > > >
> > > >
> >
> https://urldefense.com/v3/__https://lists.apache.org/thread/x9j8nscvy8hq61tyn01mj8yp6n9of0kp__;!!Iz9xO38YGHZK!4rnlVIqBQQQ6xpGe1PMV58edOS_cGrKoqdfhw4qSp_JQCavvEjaxXk0OwGPnxyEcO60AzKQK_XDtp7H3DdkFXcrXGcGDgks$
> > > > > > > >> > > > > > >> > > > > >
> > > > > > > >> > > > > > >> > > > > > Thanks,
> > > > > > > >> > > > > > >> > > > > > Yufei
> > > > > > > >> > > > > > >> > > > > >
> > > > > > > >> > > > > > >> > > > > >
> > > > > > > >> > > > > > >> > > > > > On Thu, Jun 4, 2026 at 5:14 PM EJ
> Wang
> > <
> > > > > > > >> > > > > > >> > > ej.wang.dev.interest@gmail.com
> > > > > > > >> > > > > > >> > > > >
> > > > > > > >> > > > > > >> > > > > > wrote:
> > > > > > > >> > > > > > >> > > > > >
> > > > > > > >> > > > > > >> > > > > > > Good point. I agree the existing
> > > > > > > >> > > PolarisMetricsReporter is
> > > > > > > >> > > > > > >> > already
> > > > > > > >> > > > > > >> > > > very
> > > > > > > >> > > > > > >> > > > > > > close to the right conceptual
> > boundary. I
> > > > > am
> > > > > > not
> > > > > > > >> > > > > proposing a
> > > > > > > >> > > > > > >> > second
> > > > > > > >> > > > > > >> > > > > > > parallel reporter concept. The
> > > > distinction
> > > > > I
> > > > > > am
> > > > > > > >> > > trying to
> > > > > > > >> > > > > > >> make is
> > > > > > > >> > > > > > >> > > > > > narrower:
> > > > > > > >> > > > > > >> > > > > > >
> > > > > > > >> > > > > > >> > > > > > > Current state:
> > > > > > > >> > > > > > >> > > > > > >
> > > > > > > >> > > > > > >> > > > > > >    - PolarisMetricsReporter is the
> > > > existing
> > > > > > > >> metrics
> > > > > > > >> > > > > > reporting
> > > > > > > >> > > > > > >> > hook.
> > > > > > > >> > > > > > >> > > > > > >    - But it currently lives under
> > > > > > > >> runtime/service.
> > > > > > > >> > > > > > >> > > > > > >    - Its contract is documented
> > around
> > > > > > > >> Quarkus/CDI
> > > > > > > >> > > > > discovery
> > > > > > > >> > > > > > >> and
> > > > > > > >> > > > > > >> > > > > > > @Identifier
> > > > > > > >> > > > > > >> > > > > > >    selection.
> > > > > > > >> > > > > > >> > > > > > >    - The default implementation is
> > > > > log-only.
> > > > > > > >> > > > > > >> > > > > > >    - The durable implementation
> > writes
> > > > > > through
> > > > > > > >> > > > > > >> > > PolarisMetricsManager
> > > > > > > >> > > > > > >> > > > > and
> > > > > > > >> > > > > > >> > > > > > >    MetricsPersistence.
> > > > > > > >> > > > > > >> > > > > > >    - MetricsPersistence is
> currently
> > > > > > inherited by
> > > > > > > >> > > > > > >> > BasePersistence.
> > > > > > > >> > > > > > >> > > > > > >
> > > > > > > >> > > > > > >> > > > > > > My proposal:
> > > > > > > >> > > > > > >> > > > > > >
> > > > > > > >> > > > > > >> > > > > > >    - Keep the reporting/emitting
> > boundary
> > > > > as
> > > > > > the
> > > > > > > >> > SPI.
> > > > > > > >> > > > > > >> > > > > > >    - Revise that contract in a
> > > > > > framework-agnostic
> > > > > > > >> > > optional
> > > > > > > >> > > > > > >> > Iceberg
> > > > > > > >> > > > > > >> > > > > > *metrics
> > > > > > > >> > > > > > >> > > > > > >    extension API layer*.
> > > > > > > >> > > > > > >> > > > > > >    - Keep runtime/service
> > responsible for
> > > > > > REST
> > > > > > > >> > > ingestion,
> > > > > > > >> > > > > > >> > > table/authz
> > > > > > > >> > > > > > >> > > > > > >    validation, and runtime wiring.
> > > > > > > >> > > > > > >> > > > > > >    - Keep the battery
> implementation
> > > > > > log-only in
> > > > > > > >> the
> > > > > > > >> > > > > metrics
> > > > > > > >> > > > > > >> > > > extension
> > > > > > > >> > > > > > >> > > > > > API
> > > > > > > >> > > > > > >> > > > > > >    layer, *not under
> > runtime/service*.
> > > > > > > >> > > > > > >> > > > > > >    - Treat durable JDBC metrics as
> > one
> > > > > > > >> > implementation
> > > > > > > >> > > of
> > > > > > > >> > > > > the
> > > > > > > >> > > > > > >> > > > reporting
> > > > > > > >> > > > > > >> > > > > > SPI.
> > > > > > > >> > > > > > >> > > > > > >    - Decompose the
> > MetricsPersistence and
> > > > > > > >> > > BasePersistence
> > > > > > > >> > > > > > >> > coupling
> > > > > > > >> > > > > > >> > > as
> > > > > > > >> > > > > > >> > > > > > >    durable implementation detail.
> > > > > > > >> > > > > > >> > > > > > >
> > > > > > > >> > > > > > >> > > > > > > So the difference is not “new
> > reporter
> > > > > > concept vs
> > > > > > > >> > old
> > > > > > > >> > > > > > reporter
> > > > > > > >> > > > > > >> > > > > concept.”
> > > > > > > >> > > > > > >> > > > > > > but:
> > > > > > > >> > > > > > >> > > > > > >
> > > > > > > >> > > > > > >> > > > > > >
> > > > > > > >> > > > > > >> > > > > > >    - PolarisMetricsReporter today =
> > > > > > > >> runtime/service
> > > > > > > >> > > > > > CDI-shaped
> > > > > > > >> > > > > > >> > > hook.
> > > > > > > >> > > > > > >> > > > > > >    - Target reporting SPI = same
> > > > conceptual
> > > > > > > >> > boundary,
> > > > > > > >> > > but
> > > > > > > >> > > > > > >> placed
> > > > > > > >> > > > > > >> > in
> > > > > > > >> > > > > > >> > > > the
> > > > > > > >> > > > > > >> > > > > > >    right extension API layer and
> kept
> > > > > > > >> > > framework-agnostic.
> > > > > > > >> > > > > > >> > > > > > >
> > > > > > > >> > > > > > >> > > > > > > For the current metrics PR, I think
> > the
> > > > > > minimal
> > > > > > > >> > > > > > architectural
> > > > > > > >> > > > > > >> > > cleanup
> > > > > > > >> > > > > > >> > > > > > > before merge is to *put the
> > reporting SPI
> > > > > > > >> contract
> > > > > > > >> > > and the
> > > > > > > >> > > > > > >> > battery
> > > > > > > >> > > > > > >> > > > > > default
> > > > > > > >> > > > > > >> > > > > > > in the right place*. The durable
> JDBC
> > > > > > > >> > implementation,
> > > > > > > >> > > > > > >> > event-backed
> > > > > > > >> > > > > > >> > > > > async
> > > > > > > >> > > > > > >> > > > > > > implementation, external queue
> > support,
> > > > and
> > > > > > > >> deeper
> > > > > > > >> > > > > > persistence
> > > > > > > >> > > > > > >> > > > cleanup
> > > > > > > >> > > > > > >> > > > > > can
> > > > > > > >> > > > > > >> > > > > > > continue as follow-ups, as long as
> > we do
> > > > > not
> > > > > > lock
> > > > > > > >> > the
> > > > > > > >> > > SPI
> > > > > > > >> > > > > > >> > boundary
> > > > > > > >> > > > > > >> > > to
> > > > > > > >> > > > > > >> > > > > > > BasePersistence or to
> runtime/service
> > > > > wiring.
> > > > > > > >> > > > > > >> > > > > > >
> > > > > > > >> > > > > > >> > > > > > > -ej
> > > > > > > >> > > > > > >> > > > > > >
> > > > > > > >> > > > > > >> > > > > > > On Wed, Jun 3, 2026 at 3:00 PM
> Dmitri
> > > > > > > >> Bourlatchkov <
> > > > > > > >> > > > > > >> > > dimas@apache.org
> > > > > > > >> > > > > > >> > > > >
> > > > > > > >> > > > > > >> > > > > > > wrote:
> > > > > > > >> > > > > > >> > > > > > >
> > > > > > > >> > > > > > >> > > > > > > > Hi EJ,
> > > > > > > >> > > > > > >> > > > > > > >
> > > > > > > >> > > > > > >> > > > > > > > Thanks for the recap / summary!
> > > > > > > >> > > > > > >> > > > > > > >
> > > > > > > >> > > > > > >> > > > > > > > > Do folks agree that the stable
> > SPI
> > > > > > boundary
> > > > > > > >> > > should be
> > > > > > > >> > > > > > >> > > > > > > > metrics reporting/emitting, not
> > metrics
> > > > > > > >> > persistence?
> > > > > > > >> > > > > > >> > > > > > > >
> > > > > > > >> > > > > > >> > > > > > > > SGTM.
> > > > > > > >> > > > > > >> > > > > > > >
> > > > > > > >> > > > > > >> > > > > > > > > Does an optional Iceberg
> metrics
> > > > > > extension
> > > > > > > >> API
> > > > > > > >> > > layer
> > > > > > > >> > > > > > sound
> > > > > > > >> > > > > > >> > like
> > > > > > > >> > > > > > >> > > > the
> > > > > > > >> > > > > > >> > > > > > > right
> > > > > > > >> > > > > > >> > > > > > > > home for this SPI?
> > > > > > > >> > > > > > >> > > > > > > >
> > > > > > > >> > > > > > >> > > > > > > > How is that different from
> current
> > > > > > > >> > > > > PolarisMetricsReporter?
> > > > > > > >> > > > > > >> > > > > > > >
> > > > > > > >> > > > > > >> > > > > > > > > Should the current durable
> > metrics
> > > > work
> > > > > > be
> > > > > > > >> > > reframed
> > > > > > > >> > > > > as a
> > > > > > > >> > > > > > >> > > durable
> > > > > > > >> > > > > > >> > > > > > > > JDBC reference implementation of
> > that
> > > > > SPI?
> > > > > > > >> > > > > > >> > > > > > > >
> > > > > > > >> > > > > > >> > > > > > > > SGTM.
> > > > > > > >> > > > > > >> > > > > > > >
> > > > > > > >> > > > > > >> > > > > > > > > What is the smallest PR
> sequence
> > to
> > > > get
> > > > > > there
> > > > > > > >> > > without
> > > > > > > >> > > > > > >> > blocking
> > > > > > > >> > > > > > >> > > > > > > > the current metrics work
> > unnecessarily?
> > > > > > > >> > > > > > >> > > > > > > >
> > > > > > > >> > > > > > >> > > > > > > > Let's get
> > > > > > > >> > >
> > > > > > > >> >
> > > > > > > >>
> > > > > >
> > > > >
> > > >
> >
> https://urldefense.com/v3/__https://github.com/apache/polaris/pull/4397__;!!Iz9xO38YGHZK!4rnlVIqBQQQ6xpGe1PMV58edOS_cGrKoqdfhw4qSp_JQCavvEjaxXk0OwGPnxyEcO60AzKQK_XDtp7H3DdkFXcrXuu5jefU$
> > > > > > > >> > > > > > >> merged
> > > > > > > >> > > > > > >> > > > first.
> > > > > > > >> > > > > > >> > > > > > > >
> > > > > > > >> > > > > > >> > > > > > > > Then, I'd propose isolating the
> > Metrics
> > > > > > schema
> > > > > > > >> > from
> > > > > > > >> > > the
> > > > > > > >> > > > > > >> > MetaStore
> > > > > > > >> > > > > > >> > > > > > schema.
> > > > > > > >> > > > > > >> > > > > > > > This will probably have some
> ripple
> > > > > effect
> > > > > > into
> > > > > > > >> > the
> > > > > > > >> > > > > > >> bootstrap
> > > > > > > >> > > > > > >> > > > > > workflows,
> > > > > > > >> > > > > > >> > > > > > > so
> > > > > > > >> > > > > > >> > > > > > > > it's not a trivial change.
> > > > > > > >> > > > > > >> > > > > > > >
> > > > > > > >> > > > > > >> > > > > > > > Then, let's reassess.
> > > > > > > >> > > > > > >> > > > > > > >
> > > > > > > >> > > > > > >> > > > > > > > Cheers,
> > > > > > > >> > > > > > >> > > > > > > > Dmitri.
> > > > > > > >> > > > > > >> > > > > > > >
> > > > > > > >> > > > > > >> > > > > > > > On Wed, Jun 3, 2026 at 5:40 PM EJ
> > Wang
> > > > <
> > > > > > > >> > > > > > >> > > > > ej.wang.dev.interest@gmail.com
> > > > > > > >> > > > > > >> > > > > > >
> > > > > > > >> > > > > > >> > > > > > > > wrote:
> > > > > > > >> > > > > > >> > > > > > > >
> > > > > > > >> > > > > > >> > > > > > > > > Hi folks,
> > > > > > > >> > > > > > >> > > > > > > > >
> > > > > > > >> > > > > > >> > > > > > > > > Thanks again for the discussion
> > > > today.
> > > > > I
> > > > > > > >> updated
> > > > > > > >> > > the
> > > > > > > >> > > > > > sync
> > > > > > > >> > > > > > >> doc
> > > > > > > >> > > > > > >> > > > with
> > > > > > > >> > > > > > >> > > > > > > notes
> > > > > > > >> > > > > > >> > > > > > > > > from the third metrics
> > architecture
> > > > > sync:
> > > > > > > >> > > > > > >> > > > > > > > >
> > > > > > > >> > > > > > >> > > > > > > > >
> > > > > > > >> > > > > > >> > > > > > > > >
> > > > > > > >> > > > > > >> > > > > > > >
> > > > > > > >> > > > > > >> > > > > > >
> > > > > > > >> > > > > > >> > > > > >
> > > > > > > >> > > > > > >> > > > >
> > > > > > > >> > > > > > >> > > >
> > > > > > > >> > > > > > >> > >
> > > > > > > >> > > > > > >> >
> > > > > > > >> > > > > > >>
> > > > > > > >> > > > > >
> > > > > > > >> > > > >
> > > > > > > >> > >
> > > > > > > >> >
> > > > > > > >>
> > > > > >
> > > > >
> > > >
> >
> https://urldefense.com/v3/__https://docs.google.com/document/d/100h7c4damrUzVuquYbBHM0EvA4LSWuW2IT2dN_7nYVA/edit?tab=t.k96s2xyqr5u1__;!!Iz9xO38YGHZK!4rnlVIqBQQQ6xpGe1PMV58edOS_cGrKoqdfhw4qSp_JQCavvEjaxXk0OwGPnxyEcO60AzKQK_XDtp7H3DdkFXcrXrqCL-QE$
> > > > > > > >> > > > > > >> > > > > > > > >
> > > > > > > >> > > > > > >> > > > > > > > > A few highlights from today’s
> > > > > discussion:
> > > > > > > >> > > > > > >> > > > > > > > >
> > > > > > > >> > > > > > >> > > > > > > > >    - We clarified that Iceberg
> > > > metrics
> > > > > > > >> reporting
> > > > > > > >> > > can
> > > > > > > >> > > > > be
> > > > > > > >> > > > > > >> > > > interpreted
> > > > > > > >> > > > > > >> > > > > > > > either
> > > > > > > >> > > > > > >> > > > > > > > >    as sync or async handling
> from
> > > > > > Polaris’s
> > > > > > > >> > > > > perspective,
> > > > > > > >> > > > > > >> so
> > > > > > > >> > > > > > >> > > > Polaris
> > > > > > > >> > > > > > >> > > > > > > > should
> > > > > > > >> > > > > > >> > > > > > > > >    stay flexible as a platform
> > > > instead
> > > > > of
> > > > > > > >> baking
> > > > > > > >> > > one
> > > > > > > >> > > > > > >> handling
> > > > > > > >> > > > > > >> > > > model
> > > > > > > >> > > > > > >> > > > > > > into
> > > > > > > >> > > > > > >> > > > > > > > > the
> > > > > > > >> > > > > > >> > > > > > > > >    REST API behavior.
> > > > > > > >> > > > > > >> > > > > > > > >    - We aligned that the
> built-in
> > > > > > (battery)
> > > > > > > >> > > behavior
> > > > > > > >> > > > > for
> > > > > > > >> > > > > > >> > > metrics
> > > > > > > >> > > > > > >> > > > > > > emitting
> > > > > > > >> > > > > > >> > > > > > > > >    can stay simple:
> > no-op/log-only is
> > > > > > enough
> > > > > > > >> as
> > > > > > > >> > > the
> > > > > > > >> > > > > > >> default.
> > > > > > > >> > > > > > >> > > > > > > > >    - Durable metrics
> persistence
> > > > should
> > > > > > be
> > > > > > > >> > > treated as
> > > > > > > >> > > > > an
> > > > > > > >> > > > > > >> > > > > > implementation
> > > > > > > >> > > > > > >> > > > > > > > of
> > > > > > > >> > > > > > >> > > > > > > > >    the metrics reporting path,
> > not as
> > > > > the
> > > > > > > >> core
> > > > > > > >> > SPI
> > > > > > > >> > > > > > >> boundary.
> > > > > > > >> > > > > > >> > > > > > > > >    - The existing durable
> metrics
> > > > work
> > > > > > can be
> > > > > > > >> > > reviewed
> > > > > > > >> > > > > > as
> > > > > > > >> > > > > > >> a
> > > > > > > >> > > > > > >> > > > > reference
> > > > > > > >> > > > > > >> > > > > > > > >    implementation of the
> > reporting
> > > > SPI,
> > > > > > with
> > > > > > > >> > > > > > >> > > persistence-related
> > > > > > > >> > > > > > >> > > > > > logic
> > > > > > > >> > > > > > >> > > > > > > > kept
> > > > > > > >> > > > > > >> > > > > > > > >    self-contained under a
> metrics
> > > > > durable
> > > > > > > >> > > > > implementation
> > > > > > > >> > > > > > >> > module
> > > > > > > >> > > > > > >> > > > > > rather
> > > > > > > >> > > > > > >> > > > > > > > than
> > > > > > > >> > > > > > >> > > > > > > > >    scattered through core
> entity
> > > > > > persistence.
> > > > > > > >> > > > > > >> > > > > > > > >    - Dashboard/insights
> remains a
> > > > real
> > > > > > use
> > > > > > > >> case,
> > > > > > > >> > > but
> > > > > > > >> > > > > we
> > > > > > > >> > > > > > >> > agreed
> > > > > > > >> > > > > > >> > > to
> > > > > > > >> > > > > > >> > > > > > keep
> > > > > > > >> > > > > > >> > > > > > > it
> > > > > > > >> > > > > > >> > > > > > > > >    separate from the core
> metrics
> > > > > intake
> > > > > > > >> > > discussion
> > > > > > > >> > > > > for
> > > > > > > >> > > > > > >> now.
> > > > > > > >> > > > > > >> > > > > > > > >
> > > > > > > >> > > > > > >> > > > > > > > > I also did a quick source check
> > after
> > > > > the
> > > > > > > >> > meeting
> > > > > > > >> > > to
> > > > > > > >> > > > > > make
> > > > > > > >> > > > > > >> > sure
> > > > > > > >> > > > > > >> > > we
> > > > > > > >> > > > > > >> > > > > are
> > > > > > > >> > > > > > >> > > > > > > > > describing the current state
> > > > > accurately.
> > > > > > > >> > > > > > >> > > > > > > > >
> > > > > > > >> > > > > > >> > > > > > > > > Current state:
> > > > > > > >> > > > > > >> > > > > > > > >
> > > > > > > >> > > > > > >> > > > > > > > >    - Polaris already has a
> > metrics
> > > > > > reporting
> > > > > > > >> > hook:
> > > > > > > >> > > > > > >> > > > > > > > PolarisMetricsReporter.
> > > > > > > >> > > > > > >> > > > > > > > >    - The default implementation
> > is
> > > > > > > >> > > > > > DefaultMetricsReporter,
> > > > > > > >> > > > > > >> > > > selected
> > > > > > > >> > > > > > >> > > > > > by
> > > > > > > >> > > > > > >> > > > > > > > >
> > > > > > > >> > > polaris.iceberg-metrics.reporting.type=default. It
> > > > > > > >> > > > > is
> > > > > > > >> > > > > > >> > > > log-only,
> > > > > > > >> > > > > > >> > > > > > > > >    effectively quiet unless
> > metrics
> > > > > > logging
> > > > > > > >> is
> > > > > > > >> > > > > enabled.
> > > > > > > >> > > > > > >> > > > > > > > >    - There is also a
> > > > > > > >> PersistingMetricsReporter,
> > > > > > > >> > > > > selected
> > > > > > > >> > > > > > >> by
> > > > > > > >> > > > > > >> > > > > > > > >
> > > > > > > >> > > polaris.iceberg-metrics.reporting.type=persisting,
> > > > > > > >> > > > > > >> which
> > > > > > > >> > > > > > >> > > > > converts
> > > > > > > >> > > > > > >> > > > > > > > >    Iceberg scan/commit reports
> > into
> > > > > > Polaris
> > > > > > > >> > > metrics
> > > > > > > >> > > > > > >> records
> > > > > > > >> > > > > > >> > and
> > > > > > > >> > > > > > >> > > > > > writes
> > > > > > > >> > > > > > >> > > > > > > > > through
> > > > > > > >> > > > > > >> > > > > > > > >    PolarisMetricsManager ->
> > > > > > > >> MetricsPersistence.
> > > > > > > >> > > > > > >> > > > > > > > >    - MetricsPersistence
> currently
> > > > > exists
> > > > > > in
> > > > > > > >> the
> > > > > > > >> > > > > > >> persistence
> > > > > > > >> > > > > > >> > > layer
> > > > > > > >> > > > > > >> > > > > and
> > > > > > > >> > > > > > >> > > > > > > is
> > > > > > > >> > > > > > >> > > > > > > > >    inherited by
> BasePersistence.
> > > > > > > >> > > > > > >> > > > > > > > >
> > > > > > > >> > > > > > >> > > > > > > > > My read from the discussion is
> > that
> > > > the
> > > > > > > >> target
> > > > > > > >> > > > > boundary
> > > > > > > >> > > > > > >> > should
> > > > > > > >> > > > > > >> > > > not
> > > > > > > >> > > > > > >> > > > > be
> > > > > > > >> > > > > > >> > > > > > > > > MetricsPersistence as inherited
> > by
> > > > > > > >> > > BasePersistence.
> > > > > > > >> > > > > That
> > > > > > > >> > > > > > >> path
> > > > > > > >> > > > > > >> > > > > should
> > > > > > > >> > > > > > >> > > > > > be
> > > > > > > >> > > > > > >> > > > > > > > > decomposed as durable
> > implementation
> > > > > > detail
> > > > > > > >> > (taken
> > > > > > > >> > > > > care
> > > > > > > >> > > > > > >> of by
> > > > > > > >> > > > > > >> > > > > > > > >
> > > > > > > >> >
> > > > > > > >>
> > > > > >
> > > > >
> > > >
> >
> https://urldefense.com/v3/__https://github.com/apache/polaris/pull/4397__;!!Iz9xO38YGHZK!4rnlVIqBQQQ6xpGe1PMV58edOS_cGrKoqdfhw4qSp_JQCavvEjaxXk0OwGPnxyEcO60AzKQK_XDtp7H3DdkFXcrXuu5jefU$
> > > > > > > >> ).
> > > > > > > >> > The
> > > > > > > >> > > > > > stable
> > > > > > > >> > > > > > >> > > > extension
> > > > > > > >> > > > > > >> > > > > > > point
> > > > > > > >> > > > > > >> > > > > > > > > should instead be the metrics
> > > > > > > >> reporting/emitting
> > > > > > > >> > > > > > boundary.
> > > > > > > >> > > > > > >> > > > > > > > >
> > > > > > > >> > > > > > >> > > > > > > > > Concretely, I think the next
> > proposal
> > > > > > should
> > > > > > > >> be
> > > > > > > >> > > shaped
> > > > > > > >> > > > > > >> like
> > > > > > > >> > > > > > >> > > this:
> > > > > > > >> > > > > > >> > > > > > > > >
> > > > > > > >> > > > > > >> > > > > > > > >    1.
> > > > > > > >> > > > > > >> > > > > > > > >
> > > > > > > >> > > > > > >> > > > > > > > >    Define the proper metrics
> > > > reporting
> > > > > > SPI
> > > > > > > >> at an
> > > > > > > >> > > > > > optional
> > > > > > > >> > > > > > >> > > Iceberg
> > > > > > > >> > > > > > >> > > > > > > metrics
> > > > > > > >> > > > > > >> > > > > > > > >    extension API layer.
> > > > > > > >> > > > > > >> > > > > > > > >
> > > > > > > >> > > > > > >> > > > > > > > >    Example direction:
> > > > > > > >> > > > > > >> > > > > > > > >
> > > > > > > >> > > > > > >> > > > > > > > >    public interface
> > > > > > IcebergMetricsReporter {
> > > > > > > >> > > > > > >> > > > > > > > >      void
> > > > > > > >> > reportMetric(IcebergMetricsReportContext
> > > > > > > >> > > > > > >> context,
> > > > > > > >> > > > > > >> > > > > > > > > MetricsReport report);
> > > > > > > >> > > > > > >> > > > > > > > >    }
> > > > > > > >> > > > > > >> > > > > > > > >
> > > > > > > >> > > > > > >> > > > > > > > >    The context would carry the
> > small
> > > > > > > >> > > Polaris-resolved
> > > > > > > >> > > > > > >> > envelope:
> > > > > > > >> > > > > > >> > > > > > > > >    catalog/table identity,
> report
> > > > type,
> > > > > > > >> received
> > > > > > > >> > > > > > >> timestamp,
> > > > > > > >> > > > > > >> > and
> > > > > > > >> > > > > > >> > > > > > > > > request/trace
> > > > > > > >> > > > > > >> > > > > > > > >    context if available. The
> raw
> > > > > Iceberg
> > > > > > > >> > > MetricsReport
> > > > > > > >> > > > > > >> > remains
> > > > > > > >> > > > > > >> > > > the
> > > > > > > >> > > > > > >> > > > > > > > payload.
> > > > > > > >> > > > > > >> > > > > > > > >    2.
> > > > > > > >> > > > > > >> > > > > > > > >
> > > > > > > >> > > > > > >> > > > > > > > >    Keep runtime/service as the
> > REST
> > > > > > ingestion
> > > > > > > >> > and
> > > > > > > >> > > > > wiring
> > > > > > > >> > > > > > >> > layer.
> > > > > > > >> > > > > > >> > > > > > > > >
> > > > > > > >> > > > > > >> > > > > > > > >    The REST handler still
> > resolves
> > > > the
> > > > > > table
> > > > > > > >> and
> > > > > > > >> > > > > > performs
> > > > > > > >> > > > > > >> > authz
> > > > > > > >> > > > > > >> > > > > > before
> > > > > > > >> > > > > > >> > > > > > > > >    accepting the report. After
> > that,
> > > > it
> > > > > > calls
> > > > > > > >> > the
> > > > > > > >> > > > > > selected
> > > > > > > >> > > > > > >> > > > > reporting
> > > > > > > >> > > > > > >> > > > > > > > >    implementation.
> > > > > > > >> > > > > > >> > > > > > > > >    3.
> > > > > > > >> > > > > > >> > > > > > > > >
> > > > > > > >> > > > > > >> > > > > > > > >    Keep the battery default
> > > > > > no-op/log-only.
> > > > > > > >> > > > > > >> > > > > > > > >
> > > > > > > >> > > > > > >> > > > > > > > >    This preserves an out-of-box
> > safe
> > > > > > default
> > > > > > > >> and
> > > > > > > >> > > > > avoids
> > > > > > > >> > > > > > >> > > > requiring a
> > > > > > > >> > > > > > >> > > > > > > > durable
> > > > > > > >> > > > > > >> > > > > > > > >    metrics store for every
> > Polaris
> > > > > > > >> deployment.
> > > > > > > >> > > > > > >> > > > > > > > >    4.
> > > > > > > >> > > > > > >> > > > > > > > >
> > > > > > > >> > > > > > >> > > > > > > > >    Move durable JDBC metrics
> > into a
> > > > > > > >> > self-contained
> > > > > > > >> > > > > > >> > > implementation
> > > > > > > >> > > > > > >> > > > > of
> > > > > > > >> > > > > > >> > > > > > > the
> > > > > > > >> > > > > > >> > > > > > > > >    reporting SPI.
> > > > > > > >> > > > > > >> > > > > > > > >
> > > > > > > >> > > > > > >> > > > > > > > >    That implementation can own
> > its
> > > > > > schema,
> > > > > > > >> > > bootstrap,
> > > > > > > >> > > > > > >> > > retention,
> > > > > > > >> > > > > > >> > > > > and
> > > > > > > >> > > > > > >> > > > > > > read
> > > > > > > >> > > > > > >> > > > > > > > >    API support. It should not
> > define
> > > > > the
> > > > > > core
> > > > > > > >> > > > > reporting
> > > > > > > >> > > > > > >> SPI
> > > > > > > >> > > > > > >> > > > > boundary,
> > > > > > > >> > > > > > >> > > > > > > and
> > > > > > > >> > > > > > >> > > > > > > > > it
> > > > > > > >> > > > > > >> > > > > > > > >    should not require metrics
> > > > > > persistence to
> > > > > > > >> > > remain
> > > > > > > >> > > > > > >> inherited
> > > > > > > >> > > > > > >> > > > from
> > > > > > > >> > > > > > >> > > > > > > > >    BasePersistence.
> > > > > > > >> > > > > > >> > > > > > > > >    5.
> > > > > > > >> > > > > > >> > > > > > > > >
> > > > > > > >> > > > > > >> > > > > > > > >    Treat async/event-backed
> > handling
> > > > as
> > > > > > > >> another
> > > > > > > >> > > > > > >> > implementation
> > > > > > > >> > > > > > >> > > of
> > > > > > > >> > > > > > >> > > > > the
> > > > > > > >> > > > > > >> > > > > > > > same
> > > > > > > >> > > > > > >> > > > > > > > >    reporting SPI.
> > > > > > > >> > > > > > >> > > > > > > > >
> > > > > > > >> > > > > > >> > > > > > > > >    For example, an event-backed
> > > > > reporter
> > > > > > > >> could
> > > > > > > >> > > > > enqueue a
> > > > > > > >> > > > > > >> > > metrics
> > > > > > > >> > > > > > >> > > > > > event
> > > > > > > >> > > > > > >> > > > > > > > and
> > > > > > > >> > > > > > >> > > > > > > > >    let listeners handle durable
> > > > storage
> > > > > > or
> > > > > > > >> other
> > > > > > > >> > > > > sinks.
> > > > > > > >> > > > > > >> If we
> > > > > > > >> > > > > > >> > > > later
> > > > > > > >> > > > > > >> > > > > > > need
> > > > > > > >> > > > > > >> > > > > > > > a
> > > > > > > >> > > > > > >> > > > > > > > >    replaceable queue engine,
> that
> > > > seems
> > > > > > like
> > > > > > > >> a
> > > > > > > >> > > shared
> > > > > > > >> > > > > > >> > > > event/metrics
> > > > > > > >> > > > > > >> > > > > > > > > substrate
> > > > > > > >> > > > > > >> > > > > > > > >    topic rather than a
> > metrics-only
> > > > > > > >> requirement.
> > > > > > > >> > > > > > >> > > > > > > > >
> > > > > > > >> > > > > > >> > > > > > > > > This framing lets us keep the
> > REST
> > > > > > metrics
> > > > > > > >> > > endpoint
> > > > > > > >> > > > > > >> simple,
> > > > > > > >> > > > > > >> > > > > preserve
> > > > > > > >> > > > > > >> > > > > > > the
> > > > > > > >> > > > > > >> > > > > > > > > current default behavior,
> support
> > > > > durable
> > > > > > > >> > metrics
> > > > > > > >> > > > > users,
> > > > > > > >> > > > > > >> and
> > > > > > > >> > > > > > >> > > > still
> > > > > > > >> > > > > > >> > > > > > > leave
> > > > > > > >> > > > > > >> > > > > > > > > room for async/event-backed or
> > > > > > > >> > > external-queue-based
> > > > > > > >> > > > > > >> > > > > implementations.
> > > > > > > >> > > > > > >> > > > > > > > >
> > > > > > > >> > > > > > >> > > > > > > > > I think the main follow-up
> > questions
> > > > > are:
> > > > > > > >> > > > > > >> > > > > > > > >
> > > > > > > >> > > > > > >> > > > > > > > >    - Do folks agree that the
> > stable
> > > > SPI
> > > > > > > >> boundary
> > > > > > > >> > > > > should
> > > > > > > >> > > > > > be
> > > > > > > >> > > > > > >> > > > metrics
> > > > > > > >> > > > > > >> > > > > > > > >    reporting/emitting, not
> > metrics
> > > > > > > >> persistence?
> > > > > > > >> > > > > > >> > > > > > > > >    - Does an optional Iceberg
> > metrics
> > > > > > > >> extension
> > > > > > > >> > > API
> > > > > > > >> > > > > > layer
> > > > > > > >> > > > > > >> > sound
> > > > > > > >> > > > > > >> > > > > like
> > > > > > > >> > > > > > >> > > > > > > the
> > > > > > > >> > > > > > >> > > > > > > > >    right home for this SPI?
> > > > > > > >> > > > > > >> > > > > > > > >    - Should the current durable
> > > > metrics
> > > > > > work
> > > > > > > >> be
> > > > > > > >> > > > > reframed
> > > > > > > >> > > > > > >> as a
> > > > > > > >> > > > > > >> > > > > durable
> > > > > > > >> > > > > > >> > > > > > > > JDBC
> > > > > > > >> > > > > > >> > > > > > > > >    reference implementation of
> > that
> > > > > SPI?
> > > > > > > >> > > > > > >> > > > > > > > >    - What is the smallest PR
> > sequence
> > > > > to
> > > > > > get
> > > > > > > >> > there
> > > > > > > >> > > > > > without
> > > > > > > >> > > > > > >> > > > blocking
> > > > > > > >> > > > > > >> > > > > > the
> > > > > > > >> > > > > > >> > > > > > > > >    current metrics work
> > > > unnecessarily?
> > > > > > > >> > > > > > >> > > > > > > > >
> > > > > > > >> > > > > > >> > > > > > > > > Thanks,
> > > > > > > >> > > > > > >> > > > > > > > > -ej
> > > > > > > >> > > > > > >> > > > > > > > >
> > > > > > > >> > > > > > >> > > > > > > > > On Fri, May 15, 2026 at 5:16 PM
> > > > Dmitri
> > > > > > > >> > > Bourlatchkov <
> > > > > > > >> > > > > > >> > > > > > dimas@apache.org>
> > > > > > > >> > > > > > >> > > > > > > > > wrote:
> > > > > > > >> > > > > > >> > > > > > > > >
> > > > > > > >> > > > > > >> > > > > > > > > > Hi JB,
> > > > > > > >> > > > > > >> > > > > > > > > >
> > > > > > > >> > > > > > >> > > > > > > > > > Could you set up another
> > meeting,
> > > > > > please?
> > > > > > > >> Same
> > > > > > > >> > > time
> > > > > > > >> > > > > on
> > > > > > > >> > > > > > >> > > > Wednesday
> > > > > > > >> > > > > > >> > > > > as
> > > > > > > >> > > > > > >> > > > > > > > last
> > > > > > > >> > > > > > >> > > > > > > > > > time... I hope it works for
> > > > everyone.
> > > > > > > >> > > > > > >> > > > > > > > > >
> > > > > > > >> > > > > > >> > > > > > > > > > Cheers,
> > > > > > > >> > > > > > >> > > > > > > > > > Dmitri.
> > > > > > > >> > > > > > >> > > > > > > > > >
> > > > > > > >> > > > > > >> > > > > > > > > > On Fri, May 15, 2026 at
> 8:06 PM
> > > > Yufei
> > > > > > Gu <
> > > > > > > >> > > > > > >> > > flyrain000@gmail.com
> > > > > > > >> > > > > > >> > > > >
> > > > > > > >> > > > > > >> > > > > > > wrote:
> > > > > > > >> > > > > > >> > > > > > > > > >
> > > > > > > >> > > > > > >> > > > > > > > > > > +1 on another sync call
> next
> > > > week.
> > > > > > > >> > > > > > >> > > > > > > > > > >
> > > > > > > >> > > > > > >> > > > > > > > > > > Yufei
> > > > > > > >> > > > > > >> > > > > > > > > > >
> > > > > > > >> > > > > > >> > > > > > > > > > >
> > > > > > > >> > > > > > >> > > > > > > > > > > On Fri, May 15, 2026 at
> > 4:52 PM
> > > > > > Dmitri
> > > > > > > >> > > > > Bourlatchkov
> > > > > > > >> > > > > > <
> > > > > > > >> > > > > > >> > > > > > > > dimas@apache.org>
> > > > > > > >> > > > > > >> > > > > > > > > > > wrote:
> > > > > > > >> > > > > > >> > > > > > > > > > >
> > > > > > > >> > > > > > >> > > > > > > > > > > > Hi All,
> > > > > > > >> > > > > > >> > > > > > > > > > > >
> > > > > > > >> > > > > > >> > > > > > > > > > > > WDYT about another sync
> > call
> > > > next
> > > > > > week?
> > > > > > > >> > > > > > >> > > > > > > > > > > >
> > > > > > > >> > > > > > >> > > > > > > > > > > > Thanks,
> > > > > > > >> > > > > > >> > > > > > > > > > > > Dmitri.
> > > > > > > >> > > > > > >> > > > > > > > > > > >
> > > > > > > >> > > > > > >> > > > > > > > > > > > On Wed, May 6, 2026 at
> > 5:29 PM
> > > > > > Dmitri
> > > > > > > >> > > > > > Bourlatchkov <
> > > > > > > >> > > > > > >> > > > > > > > dimas@apache.org
> > > > > > > >> > > > > > >> > > > > > > > > >
> > > > > > > >> > > > > > >> > > > > > > > > > > > wrote:
> > > > > > > >> > > > > > >> > > > > > > > > > > >
> > > > > > > >> > > > > > >> > > > > > > > > > > > > Hi EJ,
> > > > > > > >> > > > > > >> > > > > > > > > > > > >
> > > > > > > >> > > > > > >> > > > > > > > > > > > > Thanks for the summary!
> > It
> > > > > covers
> > > > > > > >> what
> > > > > > > >> > we
> > > > > > > >> > > > > > >> discussed
> > > > > > > >> > > > > > >> > in
> > > > > > > >> > > > > > >> > > > the
> > > > > > > >> > > > > > >> > > > > > > > meeting
> > > > > > > >> > > > > > >> > > > > > > > > > very
> > > > > > > >> > > > > > >> > > > > > > > > > > > > well, IMHO.
> > > > > > > >> > > > > > >> > > > > > > > > > > > >
> > > > > > > >> > > > > > >> > > > > > > > > > > > > Looking forward to
> > concrete
> > > > PRs
> > > > > > :)
> > > > > > > >> > > > > > >> > > > > > > > > > > > >
> > > > > > > >> > > > > > >> > > > > > > > > > > > > Cheers,
> > > > > > > >> > > > > > >> > > > > > > > > > > > > Dmitri.
> > > > > > > >> > > > > > >> > > > > > > > > > > > >
> > > > > > > >> > > > > > >> > > > > > > > > > > > > On Wed, May 6, 2026 at
> > > > 5:08 PM
> > > > > EJ
> > > > > > > >> Wang <
> > > > > > > >> > > > > > >> > > > > > > > > >
> ej.wang.dev.interest@gmail.com
> > > > > > > >> > > > > > >> > > > > > > > > > > >
> > > > > > > >> > > > > > >> > > > > > > > > > > > > wrote:
> > > > > > > >> > > > > > >> > > > > > > > > > > > >
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> Hi folks,
> > > > > > > >> > > > > > >> > > > > > > > > > > > >>
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> We had a community
> sync
> > > > > earlier,
> > > > > > > >> thanks
> > > > > > > >> > > JB
> > > > > > > >> > > > > for
> > > > > > > >> > > > > > >> > > > scheduling
> > > > > > > >> > > > > > >> > > > > > it.
> > > > > > > >> > > > > > >> > > > > > > > > Notes
> > > > > > > >> > > > > > >> > > > > > > > > > > from
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> the first metrics
> > > > architecture
> > > > > > sync
> > > > > > > >> > (May
> > > > > > > >> > > 6,
> > > > > > > >> > > > > > >> 10-11am
> > > > > > > >> > > > > > >> > > PT).
> > > > > > > >> > > > > > >> > > > > > > > > Discussion
> > > > > > > >> > > > > > >> > > > > > > > > > > doc
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> with per-section
> status:
> > > > > > > >> > > > > > >> > > > > > > > > > > > >>
> > > > > > > >> > > > > > >> > > > > > > > > > > > >>
> > > > > > > >> > > > > > >> > > > > > > > > > > >
> > > > > > > >> > > > > > >> > > > > > > > > > >
> > > > > > > >> > > > > > >> > > > > > > > > >
> > > > > > > >> > > > > > >> > > > > > > > >
> > > > > > > >> > > > > > >> > > > > > > >
> > > > > > > >> > > > > > >> > > > > > >
> > > > > > > >> > > > > > >> > > > > >
> > > > > > > >> > > > > > >> > > > >
> > > > > > > >> > > > > > >> > > >
> > > > > > > >> > > > > > >> > >
> > > > > > > >> > > > > > >> >
> > > > > > > >> > > > > > >>
> > > > > > > >> > > > > >
> > > > > > > >> > > > >
> > > > > > > >> > >
> > > > > > > >> >
> > > > > > > >>
> > > > > >
> > > > >
> > > >
> >
> https://urldefense.com/v3/__https://docs.google.com/document/d/100h7c4damrUzVuquYbBHM0EvA4LSWuW2IT2dN_7nYVA/edit?tab=t.0__;!!Iz9xO38YGHZK!4rnlVIqBQQQ6xpGe1PMV58edOS_cGrKoqdfhw4qSp_JQCavvEjaxXk0OwGPnxyEcO60AzKQK_XDtp7H3DdkFXcrXgIddeUs$
> > > > > > > >> > > > > > >> > > > > > > > > > > > >>
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> *The meeting covered
> > both
> > > > > topics
> > > > > > > >> from
> > > > > > > >> > the
> > > > > > > >> > > > > doc.
> > > > > > > >> > > > > > >> > > > > > Direction-level
> > > > > > > >> > > > > > >> > > > > > > > > > > alignment
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> was reached on the
> > headline
> > > > > > pieces;
> > > > > > > >> > > details
> > > > > > > >> > > > > > >> remain
> > > > > > > >> > > > > > >> > for
> > > > > > > >> > > > > > >> > > > PR
> > > > > > > >> > > > > > >> > > > > > > review
> > > > > > > >> > > > > > >> > > > > > > > > or
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> follow-up sessions.*
> > > > > > > >> > > > > > >> > > > > > > > > > > > >>
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> *Topic 1 — Persistence
> > > > schema
> > > > > > > >> redesign*
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> Idea-level alignment
> on
> > > > > > > >> consolidating
> > > > > > > >> > > > > per-type
> > > > > > > >> > > > > > >> > tables
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> (scan_metrics_report,
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> commit_metrics_report)
> > into
> > > > a
> > > > > > single
> > > > > > > >> > > > > > >> metrics_report
> > > > > > > >> > > > > > >> > > > table.
> > > > > > > >> > > > > > >> > > > > > The
> > > > > > > >> > > > > > >> > > > > > > > > > > > motivating
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> cost is the surface
> area
> > > > added
> > > > > > by
> > > > > > > >> every
> > > > > > > >> > > new
> > > > > > > >> > > > > > >> metric
> > > > > > > >> > > > > > >> > > type
> > > > > > > >> > > > > > >> > > > > > today:
> > > > > > > >> > > > > > >> > > > > > > > new
> > > > > > > >> > > > > > >> > > > > > > > > > > > table,
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> SPI method, record
> > class,
> > > > > model,
> > > > > > > >> > > converter,
> > > > > > > >> > > > > > >> schema
> > > > > > > >> > > > > > >> > > > > > migration.
> > > > > > > >> > > > > > >> > > > > > > > > > > > >>
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> Most schema details
> are
> > > > > > deferred to
> > > > > > > >> the
> > > > > > > >> > > > > schema
> > > > > > > >> > > > > > >> PR. A
> > > > > > > >> > > > > > >> > > few
> > > > > > > >> > > > > > >> > > > > > > > specific
> > > > > > > >> > > > > > >> > > > > > > > > > > points
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> came up:
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> •
> >  metric_schema_version:
> > > > > Yufei
> > > > > > > >> > prefers
> > > > > > > >> > > > > > dropping
> > > > > > > >> > > > > > >> > it,
> > > > > > > >> > > > > > >> > > > > since
> > > > > > > >> > > > > > >> > > > > > > > there
> > > > > > > >> > > > > > >> > > > > > > > > is
> > > > > > > >> > > > > > >> > > > > > > > > > > no
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> spec-level concept of
> > > > metrics
> > > > > > > >> > versioning
> > > > > > > >> > > > > today
> > > > > > > >> > > > > > >> and
> > > > > > > >> > > > > > >> > it
> > > > > > > >> > > > > > >> > > is
> > > > > > > >> > > > > > >> > > > > > hard
> > > > > > > >> > > > > > >> > > > > > > to
> > > > > > > >> > > > > > >> > > > > > > > > > > define
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> unilaterally. Robert
> > prefers
> > > > > > keeping
> > > > > > > >> > it,
> > > > > > > >> > > > > given
> > > > > > > >> > > > > > >> IRC
> > > > > > > >> > > > > > >> > v2
> > > > > > > >> > > > > > >> > > is
> > > > > > > >> > > > > > >> > > > > > > coming
> > > > > > > >> > > > > > >> > > > > > > > > and
> > > > > > > >> > > > > > >> > > > > > > > > > > the
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> schema should be
> > considered
> > > > > > against
> > > > > > > >> its
> > > > > > > >> > > > > likely
> > > > > > > >> > > > > > >> > shape;
> > > > > > > >> > > > > > >> > > > > Robert
> > > > > > > >> > > > > > >> > > > > > > > also
> > > > > > > >> > > > > > >> > > > > > > > > > > raised
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> how to differentiate
> > various
> > > > > > payload
> > > > > > > >> > > formats
> > > > > > > >> > > > > if
> > > > > > > >> > > > > > >> any.
> > > > > > > >> > > > > > >> > > > EJ's
> > > > > > > >> > > > > > >> > > > > > read
> > > > > > > >> > > > > > >> > > > > > > > is
> > > > > > > >> > > > > > >> > > > > > > > > > that
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> this
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> is a two-way-door
> > decision.
> > > > We
> > > > > > can
> > > > > > > >> > start
> > > > > > > >> > > > > > without
> > > > > > > >> > > > > > >> the
> > > > > > > >> > > > > > >> > > > > field,
> > > > > > > >> > > > > > >> > > > > > > and
> > > > > > > >> > > > > > >> > > > > > > > if
> > > > > > > >> > > > > > >> > > > > > > > > > IRC
> > > > > > > >> > > > > > >> > > > > > > > > > > > v2
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> changes the shape we
> > would
> > > > > > likely
> > > > > > > >> roll
> > > > > > > >> > a
> > > > > > > >> > > > > > >> > corresponding
> > > > > > > >> > > > > > >> > > > new
> > > > > > > >> > > > > > >> > > > > > > > schema
> > > > > > > >> > > > > > >> > > > > > > > > > > > anyway,
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> which is not
> > particularly
> > > > > > costly.
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> •   Payload format:
> > Robert
> > > > > > pointed
> > > > > > > >> out
> > > > > > > >> > > that
> > > > > > > >> > > > > > >> future
> > > > > > > >> > > > > > >> > > > formats
> > > > > > > >> > > > > > >> > > > > > > > beyond
> > > > > > > >> > > > > > >> > > > > > > > > > JSON
> > > > > > > >> > > > > > >> > > > > > > > > > > > may
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> be worth supporting.
> The
> > > > exact
> > > > > > > >> shape is
> > > > > > > >> > > > > > deferred
> > > > > > > >> > > > > > >> to
> > > > > > > >> > > > > > >> > > the
> > > > > > > >> > > > > > >> > > > > > schema
> > > > > > > >> > > > > > >> > > > > > > > > > > > discussion.
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> •   Partition
> strategy:
> > > > Anand
> > > > > > > >> suggested
> > > > > > > >> > > > > monthly
> > > > > > > >> > > > > > >> > > > > partitioning
> > > > > > > >> > > > > > >> > > > > > > > based
> > > > > > > >> > > > > > >> > > > > > > > > > on
> > > > > > > >> > > > > > >> > > > > > > > > > > > his
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> experience as
> > potentially
> > > > > > helpful at
> > > > > > > >> > > scale.
> > > > > > > >> > > > > > >> > > > > > > > > > > > >>
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> *Topic 2 — Where
> metrics
> > > > > > ingestion
> > > > > > > >> and
> > > > > > > >> > > > > storage
> > > > > > > >> > > > > > >> > belong*
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> Idea-level alignment
> > that
> > > > > > metrics
> > > > > > > >> > should
> > > > > > > >> > > be a
> > > > > > > >> > > > > > >> > > separated
> > > > > > > >> > > > > > >> > > > > SPI
> > > > > > > >> > > > > > >> > > > > > > from
> > > > > > > >> > > > > > >> > > > > > > > > the
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> entity
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> persistence stack. Two
> > > > reasons
> > > > > > > >> > surfaced:
> > > > > > > >> > > (a)
> > > > > > > >> > > > > > >> > workloads
> > > > > > > >> > > > > > >> > > > and
> > > > > > > >> > > > > > >> > > > > > > > > > capability
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> requirements diverge
> > enough
> > > > > that
> > > > > > > >> > coupling
> > > > > > > >> > > > > them
> > > > > > > >> > > > > > >> > creates
> > > > > > > >> > > > > > >> > > > > > > > artificial
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> constraints, and (b)
> > admin
> > > > > > > >> experience
> > > > > > > >> > > > > improves
> > > > > > > >> > > > > > >> when
> > > > > > > >> > > > > > >> > > > > metrics
> > > > > > > >> > > > > > >> > > > > > > has
> > > > > > > >> > > > > > >> > > > > > > > > its
> > > > > > > >> > > > > > >> > > > > > > > > > > own
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> bootstrap, retention,
> > and
> > > > > > lifecycle.
> > > > > > > >> > > Dmitri
> > > > > > > >> > > > > > noted
> > > > > > > >> > > > > > >> > > > Polaris
> > > > > > > >> > > > > > >> > > > > > > being
> > > > > > > >> > > > > > >> > > > > > > > a
> > > > > > > >> > > > > > >> > > > > > > > > > > > platform
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> should have the
> > flexibility
> > > > to
> > > > > > > >> support
> > > > > > > >> > > > > > different
> > > > > > > >> > > > > > >> > > > > persistence
> > > > > > > >> > > > > > >> > > > > > > > > > backends
> > > > > > > >> > > > > > >> > > > > > > > > > > > per
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> concern, and pointed
> to
> > a
> > > > > > concrete
> > > > > > > >> next
> > > > > > > >> > > step
> > > > > > > >> > > > > of
> > > > > > > >> > > > > > >> > > > separating
> > > > > > > >> > > > > > >> > > > > > the
> > > > > > > >> > > > > > >> > > > > > > > > JDBC
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> bootstrap for metrics
> > from
> > > > the
> > > > > > > >> > metastore
> > > > > > > >> > > > > > >> bootstrap.
> > > > > > > >> > > > > > >> > > > Robert
> > > > > > > >> > > > > > >> > > > > > > > > proposed
> > > > > > > >> > > > > > >> > > > > > > > > > an
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> additional UX
> extension:
> > > > > detect
> > > > > > an
> > > > > > > >> > > > > > unbootstrapped
> > > > > > > >> > > > > > >> > > > metrics
> > > > > > > >> > > > > > >> > > > > > > store
> > > > > > > >> > > > > > >> > > > > > > > on
> > > > > > > >> > > > > > >> > > > > > > > > > > first
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> use and auto-bootstrap
> > > > rather
> > > > > > than
> > > > > > > >> > > requiring
> > > > > > > >> > > > > an
> > > > > > > >> > > > > > >> > > explicit
> > > > > > > >> > > > > > >> > > > > > > manual
> > > > > > > >> > > > > > >> > > > > > > > > > > > bootstrap
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> step.
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> The meeting also
> > confirmed
> > > > > that
> > > > > > > >> Polaris
> > > > > > > >> > > > > metrics
> > > > > > > >> > > > > > >> can
> > > > > > > >> > > > > > >> > > > start
> > > > > > > >> > > > > > >> > > > > > > small
> > > > > > > >> > > > > > >> > > > > > > > > and
> > > > > > > >> > > > > > >> > > > > > > > > > > stay
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> Iceberg-focused.
> Naming
> > and
> > > > > > > >> persistence
> > > > > > > >> > > > > schema
> > > > > > > >> > > > > > >> can
> > > > > > > >> > > > > > >> > > lean
> > > > > > > >> > > > > > >> > > > > > > > > > > > Iceberg-specific.
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> If a future expansion
> to
> > > > > > > >> generic-table
> > > > > > > >> > > > > metrics
> > > > > > > >> > > > > > or
> > > > > > > >> > > > > > >> > > > > > operational
> > > > > > > >> > > > > > >> > > > > > > > > > metrics
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> arrives, an
> abstraction
> > > > layer
> > > > > > can be
> > > > > > > >> > > built on
> > > > > > > >> > > > > > >> top of
> > > > > > > >> > > > > > >> > > the
> > > > > > > >> > > > > > >> > > > > > > Iceberg
> > > > > > > >> > > > > > >> > > > > > > > > > > metrics
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> reporter at that
> point.
> > > > Robert
> > > > > > > >> remains
> > > > > > > >> > > on the
> > > > > > > >> > > > > > >> fence
> > > > > > > >> > > > > > >> > > and
> > > > > > > >> > > > > > >> > > > > > would
> > > > > > > >> > > > > > >> > > > > > > > > prefer
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> something more generic
> > but
> > > > did
> > > > > > not
> > > > > > > >> > block
> > > > > > > >> > > the
> > > > > > > >> > > > > > >> > > direction;
> > > > > > > >> > > > > > >> > > > > > > Dmitri's
> > > > > > > >> > > > > > >> > > > > > > > > > read
> > > > > > > >> > > > > > >> > > > > > > > > > > > was
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> that the proposed
> > framework
> > > > > > already
> > > > > > > >> has
> > > > > > > >> > > > > enough
> > > > > > > >> > > > > > >> > > > flexibility
> > > > > > > >> > > > > > >> > > > > > to
> > > > > > > >> > > > > > >> > > > > > > > > absorb
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> future
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> expansion.
> > > > > > > >> > > > > > >> > > > > > > > > > > > >>
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> The Trade-offs and
> > Proposed
> > > > > > > >> structure
> > > > > > > >> > > > > sections
> > > > > > > >> > > > > > in
> > > > > > > >> > > > > > >> > the
> > > > > > > >> > > > > > >> > > > doc
> > > > > > > >> > > > > > >> > > > > > were
> > > > > > > >> > > > > > >> > > > > > > > not
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> reviewed
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> in detail. They remain
> > open
> > > > > for
> > > > > > > >> either
> > > > > > > >> > > the
> > > > > > > >> > > > > next
> > > > > > > >> > > > > > >> sync
> > > > > > > >> > > > > > >> > > or
> > > > > > > >> > > > > > >> > > > PR
> > > > > > > >> > > > > > >> > > > > > > > review.
> > > > > > > >> > > > > > >> > > > > > > > > > > > >>
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> *Cross-cutting
> > alignment —
> > > > > > > >> > > battery-included
> > > > > > > >> > > > > > plus
> > > > > > > >> > > > > > >> > > > > pluggable*
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> A common philosophy
> > emerged
> > > > > > from the
> > > > > > > >> > > > > > discussion.
> > > > > > > >> > > > > > >> EJ
> > > > > > > >> > > > > > >> > > > > > summarized
> > > > > > > >> > > > > > >> > > > > > > > it
> > > > > > > >> > > > > > >> > > > > > > > > > as:
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> Polaris should
> provide a
> > > > > > > >> > > battery-included UX
> > > > > > > >> > > > > > for
> > > > > > > >> > > > > > >> > > > beginners
> > > > > > > >> > > > > > >> > > > > > and
> > > > > > > >> > > > > > >> > > > > > > > the
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> flexibility for
> advanced
> > > > users
> > > > > > to
> > > > > > > >> swap
> > > > > > > >> > > the
> > > > > > > >> > > > > > >> included
> > > > > > > >> > > > > > >> > > > > battery
> > > > > > > >> > > > > > >> > > > > > > for
> > > > > > > >> > > > > > >> > > > > > > > > > > > something
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> more powerful or
> > tailored to
> > > > > > their
> > > > > > > >> use
> > > > > > > >> > > case.
> > > > > > > >> > > > > > The
> > > > > > > >> > > > > > >> SPI
> > > > > > > >> > > > > > >> > > > > design
> > > > > > > >> > > > > > >> > > > > > > > needs
> > > > > > > >> > > > > > >> > > > > > > > > to
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> enable
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> both.
> > > > > > > >> > > > > > >> > > > > > > > > > > > >>
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> The inputs that shaped
> > this
> > > > > > framing:
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> •   Anand described
> how
> > his
> > > > > team
> > > > > > > >> uses
> > > > > > > >> > the
> > > > > > > >> > > > > > current
> > > > > > > >> > > > > > >> > > > metrics
> > > > > > > >> > > > > > >> > > > > > > > > > persistence
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> (three metrics
> > consumers in
> > > > > > v1.4).
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> •   Yufei raised
> > Grafana and
> > > > > > > >> dashboard
> > > > > > > >> > > > > > >> integrations
> > > > > > > >> > > > > > >> > > as a
> > > > > > > >> > > > > > >> > > > > > > > > destination
> > > > > > > >> > > > > > >> > > > > > > > > > > use
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> case beyond the
> default.
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> •   Robert called out
> > that
> > > > the
> > > > > > > >> current
> > > > > > > >> > > design
> > > > > > > >> > > > > > is
> > > > > > > >> > > > > > >> > more
> > > > > > > >> > > > > > >> > > > > > > > > JDBC-focused.
> > > > > > > >> > > > > > >> > > > > > > > > > > > >>
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> Two concrete
> instances:
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> •   Async metrics
> > intake:
> > > > > > Yufei's
> > > > > > > >> > initial
> > > > > > > >> > > > > > >> position
> > > > > > > >> > > > > > >> > was
> > > > > > > >> > > > > > >> > > > > that
> > > > > > > >> > > > > > >> > > > > > > > async
> > > > > > > >> > > > > > >> > > > > > > > > > > should
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> largely live on the
> > producer
> > > > > > side
> > > > > > > >> and
> > > > > > > >> > > there
> > > > > > > >> > > > > is
> > > > > > > >> > > > > > >> not
> > > > > > > >> > > > > > >> > > much
> > > > > > > >> > > > > > >> > > > > > > Polaris
> > > > > > > >> > > > > > >> > > > > > > > > can
> > > > > > > >> > > > > > >> > > > > > > > > > > do.
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> Robert suggested a
> > > > > Polaris-side
> > > > > > > >> default
> > > > > > > >> > > is
> > > > > > > >> > > > > > doable
> > > > > > > >> > > > > > >> > via
> > > > > > > >> > > > > > >> > > > > > Vert.x.
> > > > > > > >> > > > > > >> > > > > > > > > Dmitri
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> agreed
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> the direction is worth
> > > > > > exploring.
> > > > > > > >> The
> > > > > > > >> > > meeting
> > > > > > > >> > > > > > >> > > converged
> > > > > > > >> > > > > > >> > > > > on a
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> battery-included
> default
> > > > > (likely
> > > > > > > >> > > > > Vert.x-backed)
> > > > > > > >> > > > > > >> with
> > > > > > > >> > > > > > >> > > an
> > > > > > > >> > > > > > >> > > > > SPI
> > > > > > > >> > > > > > >> > > > > > > > shape
> > > > > > > >> > > > > > >> > > > > > > > > > that
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> lets
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> power users route to a
> > more
> > > > > > scalable
> > > > > > > >> > > backend
> > > > > > > >> > > > > > >> > > (k8s-hosted
> > > > > > > >> > > > > > >> > > > > > > queue,
> > > > > > > >> > > > > > >> > > > > > > > > AWS
> > > > > > > >> > > > > > >> > > > > > > > > > > SQS,
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> etc.).
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> •   Pluggable
> > destinations:
> > > > > > > >> combining
> > > > > > > >> > > Yufei's
> > > > > > > >> > > > > > >> > > dashboard
> > > > > > > >> > > > > > >> > > > > use
> > > > > > > >> > > > > > >> > > > > > > case
> > > > > > > >> > > > > > >> > > > > > > > > > with
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> Robert's JDBC-focused
> > > > > call-out,
> > > > > > the
> > > > > > > >> > > meeting
> > > > > > > >> > > > > > >> agreed
> > > > > > > >> > > > > > >> > the
> > > > > > > >> > > > > > >> > > > SPI
> > > > > > > >> > > > > > >> > > > > > > > should
> > > > > > > >> > > > > > >> > > > > > > > > be
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> structured for
> multiple
> > > > sinks
> > > > > so
> > > > > > > >> > > integrations
> > > > > > > >> > > > > > >> become
> > > > > > > >> > > > > > >> > > > impl
> > > > > > > >> > > > > > >> > > > > > > > choices
> > > > > > > >> > > > > > >> > > > > > > > > > > rather
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> than architectural
> > changes.
> > > > > > > >> > > > > > >> > > > > > > > > > > > >>
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> The battery-included
> > default
> > > > > is
> > > > > > most
> > > > > > > >> > > likely
> > > > > > > >> > > > > to
> > > > > > > >> > > > > > >> use
> > > > > > > >> > > > > > >> > the
> > > > > > > >> > > > > > >> > > > > > > existing
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> JDBC-backed
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> approach.
> > > > > > > >> > > > > > >> > > > > > > > > > > > >>
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> *Direction (idea-level
> > > > > > alignment)*
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> •   Single
> > metrics_report
> > > > > table
> > > > > > > >> > > consolidating
> > > > > > > >> > > > > > >> > per-type
> > > > > > > >> > > > > > >> > > > > > > metrics,
> > > > > > > >> > > > > > >> > > > > > > > > > > > replacing
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> scan_metrics_report
> and
> > > > > > > >> > > commit_metrics_report
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> •   Iceberg-focused
> > naming
> > > > and
> > > > > > > >> schema
> > > > > > > >> > for
> > > > > > > >> > > > > now,
> > > > > > > >> > > > > > >> > revisit
> > > > > > > >> > > > > > >> > > > if
> > > > > > > >> > > > > > >> > > > > > > > > > > generic-table
> > > > > > > >> > > > > > >> > > > > > > > > > > > or
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> operational metrics
> > arrive
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> •   Metrics
> persistence
> > as a
> > > > > > > >> separated
> > > > > > > >> > > SPI,
> > > > > > > >> > > > > not
> > > > > > > >> > > > > > >> on
> > > > > > > >> > > > > > >> > > > > > > > BasePersistence
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> •   Bootstrap path
> > separated
> > > > > for
> > > > > > > >> > metrics,
> > > > > > > >> > > > > > >> > independent
> > > > > > > >> > > > > > >> > > of
> > > > > > > >> > > > > > >> > > > > > > > metastore
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> bootstrap
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> •   "Battery-included
> > plus
> > > > > > > >> pluggable"
> > > > > > > >> > as
> > > > > > > >> > > the
> > > > > > > >> > > > > > SPI
> > > > > > > >> > > > > > >> > > design
> > > > > > > >> > > > > > >> > > > > > > > philosophy
> > > > > > > >> > > > > > >> > > > > > > > > > > > >>
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> *Open items*
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> •   Schema details:
> > > > > > > >> > > metric_schema_version,
> > > > > > > >> > > > > > >> payload
> > > > > > > >> > > > > > >> > > > format,
> > > > > > > >> > > > > > >> > > > > > IRC
> > > > > > > >> > > > > > >> > > > > > > > v2
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> forward-compat shape
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> •   SPI design
> details —
> > > > full
> > > > > > review
> > > > > > > >> > > either
> > > > > > > >> > > > > in
> > > > > > > >> > > > > > >> the
> > > > > > > >> > > > > > >> > > next
> > > > > > > >> > > > > > >> > > > > sync
> > > > > > > >> > > > > > >> > > > > > > or
> > > > > > > >> > > > > > >> > > > > > > > in
> > > > > > > >> > > > > > >> > > > > > > > > > the
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> corresponding PR
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> •   Schema refactor PR
> > > > > ownership
> > > > > > > >> > > > > > >> > > > > > > > > > > > >>
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> *Action items*
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> •   EJ to take a first
> > stab
> > > > at
> > > > > > the
> > > > > > > >> SPI
> > > > > > > >> > > design
> > > > > > > >> > > > > > and
> > > > > > > >> > > > > > >> > > > > > potentially
> > > > > > > >> > > > > > >> > > > > > > > > > partner
> > > > > > > >> > > > > > >> > > > > > > > > > > > with
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> Anand to incorporate
> the
> > > > > lessons
> > > > > > > >> > learned
> > > > > > > >> > > from
> > > > > > > >> > > > > > the
> > > > > > > >> > > > > > >> > > > existing
> > > > > > > >> > > > > > >> > > > > > > > > reporter
> > > > > > > >> > > > > > >> > > > > > > > > > > and
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> persistence work.
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> •   Schema refactor PR
> > > > > > ownership is
> > > > > > > >> not
> > > > > > > >> > > yet
> > > > > > > >> > > > > > >> decided.
> > > > > > > >> > > > > > >> > > If
> > > > > > > >> > > > > > >> > > > > > anyone
> > > > > > > >> > > > > > >> > > > > > > > is
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> interested in driving
> > it,
> > > > > reply
> > > > > > on
> > > > > > > >> this
> > > > > > > >> > > > > thread.
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> •   JB to schedule the
> > next
> > > > > > sync,
> > > > > > > >> > > tentatively
> > > > > > > >> > > > > > in
> > > > > > > >> > > > > > >> two
> > > > > > > >> > > > > > >> > > > > weeks.
> > > > > > > >> > > > > > >> > > > > > > > > > > > >>
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> -ej
> > > > > > > >> > > > > > >> > > > > > > > > > > > >>
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> On Mon, Apr 27, 2026
> at
> > > > > 3:07 PM
> > > > > > EJ
> > > > > > > >> > Wang <
> > > > > > > >> > > > > > >> > > > > > > > > > >
> > ej.wang.dev.interest@gmail.com
> > > > > > > >> > > > > > >> > > > > > > > > > > > >
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> wrote:
> > > > > > > >> > > > > > >> > > > > > > > > > > > >>
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> > Thanks Yufei for the
> > +1.
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> > JB, could you help
> > add a
> > > > > > biweekly
> > > > > > > >> > > metrics
> > > > > > > >> > > > > > >> > > architecture
> > > > > > > >> > > > > > >> > > > > > sync
> > > > > > > >> > > > > > >> > > > > > > to
> > > > > > > >> > > > > > >> > > > > > > > > the
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> Polaris
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> > community calendar?
> > I'm
> > > > > > thinking
> > > > > > > >> > > Thursdays
> > > > > > > >> > > > > at
> > > > > > > >> > > > > > >> > 9-10am
> > > > > > > >> > > > > > >> > > > PT,
> > > > > > > >> > > > > > >> > > > > > on
> > > > > > > >> > > > > > >> > > > > > > > the
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> off-weeks
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> > from the community
> > meeting
> > > > > > > >> (starting
> > > > > > > >> > > May
> > > > > > > >> > > > > 7),
> > > > > > > >> > > > > > 60
> > > > > > > >> > > > > > >> > > > minutes.
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> > Here's a rough
> agenda
> > to
> > > > > work
> > > > > > > >> through
> > > > > > > >> > > over
> > > > > > > >> > > > > > the
> > > > > > > >> > > > > > >> > first
> > > > > > > >> > > > > > >> > > > few
> > > > > > > >> > > > > > >> > > > > > > > > sessions,
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> grouped
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> > by priority:
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> > *First: foundational
> > > > > > direction*
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> > 1.
> > MetricsPersistence:
> > > > > public
> > > > > > > >> SPI or
> > > > > > > >> > > > > > internal
> > > > > > > >> > > > > > >> > > > > > > implementation
> > > > > > > >> > > > > > >> > > > > > > > > > > detail?
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >    •   Marked @Beta,
> > > > javadoc
> > > > > > calls
> > > > > > > >> > it a
> > > > > > > >> > > > > > >> "Service
> > > > > > > >> > > > > > >> > > > > Provider
> > > > > > > >> > > > > > >> > > > > > > > > > > Interface",
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> but
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> > only one consumer
> > > > > > > >> > > > > (JdbcBasePersistenceImpl),
> > > > > > > >> > > > > > >> lives
> > > > > > > >> > > > > > >> > > on
> > > > > > > >> > > > > > >> > > > > > > > > > > BasePersistence.
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> If
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> > demoted to a private
> > > > helper
> > > > > > > >> inside a
> > > > > > > >> > > > > > persisting
> > > > > > > >> > > > > > >> > > > reporter
> > > > > > > >> > > > > > >> > > > > > > impl,
> > > > > > > >> > > > > > >> > > > > > > > > > most
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> > downstream design
> > > > decisions
> > > > > > become
> > > > > > > >> > > > > > >> implementation
> > > > > > > >> > > > > > >> > > > > details
> > > > > > > >> > > > > > >> > > > > > > > rather
> > > > > > > >> > > > > > >> > > > > > > > > > > than
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> > contract questions.
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> > 2.  Persistence
> schema
> > > > > > redesign
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >    •   Current
> > two-table
> > > > > > layout
> > > > > > > >> > > > > > >> > > (scan_metrics_report,
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >
> commit_metrics_report)
> > > > with
> > > > > > ~25
> > > > > > > >> > > flattened
> > > > > > > >> > > > > > >> columns
> > > > > > > >> > > > > > >> > > > each.
> > > > > > > >> > > > > > >> > > > > > > Every
> > > > > > > >> > > > > > >> > > > > > > > > new
> > > > > > > >> > > > > > >> > > > > > > > > > > > metric
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> > type requires a new
> > table,
> > > > > SPI
> > > > > > > >> > method,
> > > > > > > >> > > > > record
> > > > > > > >> > > > > > >> > class,
> > > > > > > >> > > > > > >> > > > > > model,
> > > > > > > >> > > > > > >> > > > > > > > > > > converter,
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> and
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> > schema migration.
> > > > Direction
> > > > > to
> > > > > > > >> > explore:
> > > > > > > >> > > > > > single
> > > > > > > >> > > > > > >> > table
> > > > > > > >> > > > > > >> > > > > with
> > > > > > > >> > > > > > >> > > > > > > > > > > metric_type
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> enum,
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> > schema_version, and
> > JSON
> > > > > > payload
> > > > > > > >> > > column.
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> > *Second: design
> > details
> > > > once
> > > > > > > >> > direction
> > > > > > > >> > > is
> > > > > > > >> > > > > > set*
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> > 3.  Partition key
> > strategy
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >    •   Single-table
> > design
> > > > > > means
> > > > > > > >> scan
> > > > > > > >> > > > > metrics
> > > > > > > >> > > > > > >> at
> > > > > > > >> > > > > > >> > > scale
> > > > > > > >> > > > > > >> > > > > > will
> > > > > > > >> > > > > > >> > > > > > > > have
> > > > > > > >> > > > > > >> > > > > > > > > > > high
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> > write concurrency
> per
> > > > table.
> > > > > > > >> Schema
> > > > > > > >> > > needs
> > > > > > > >> > > > > to
> > > > > > > >> > > > > > >> > expose
> > > > > > > >> > > > > > >> > > > > enough
> > > > > > > >> > > > > > >> > > > > > > > > > structure
> > > > > > > >> > > > > > >> > > > > > > > > > > > for
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> > backends to shard by
> > > > entity
> > > > > or
> > > > > > > >> time
> > > > > > > >> > > range.
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> > 4.  Read/write path
> > > > > > consistency
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >    •   Writes go
> > through
> > > > > > > >> > > > > > PolarisMetricsManager
> > > > > > > >> > > > > > >> on
> > > > > > > >> > > > > > >> > > > > > > > > > MetaStoreManager.
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> Reads
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> > bypass
> > MetaStoreManager
> > > > and
> > > > > go
> > > > > > > >> > > straight to
> > > > > > > >> > > > > > >> > > > > > BasePersistence,
> > > > > > > >> > > > > > >> > > > > > > > > > > excluding
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> > non-JDBC backends
> > from the
> > > > > > read
> > > > > > > >> API.
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> > *Third: cleanup and
> > > > > alignment*
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> > 5.
> > PolarisMetricsReporter
> > > > > > naming
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >    •   Only handles
> > IRC
> > > > > > > >> > > > > > >> (ScanReport/CommitReport),
> > > > > > > >> > > > > > >> > > > > doesn't
> > > > > > > >> > > > > > >> > > > > > > > cover
> > > > > > > >> > > > > > >> > > > > > > > > > > > generic
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> > tables or
> operational
> > > > > metrics.
> > > > > > > >> Name
> > > > > > > >> > is
> > > > > > > >> > > > > > broader
> > > > > > > >> > > > > > >> > than
> > > > > > > >> > > > > > >> > > > > scope.
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> > 6.
> > PolarisMetricsManager
> > > > > > facade
> > > > > > > >> > > > > passthrough
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >    •   Entire
> default
> > > > method
> > > > > > is
> > > > > > > >> > > > > > >> > > > > > > > > > > > >>
> > > > > > > >> > callCtx.getMetaStore().writeScanReport().
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> > Zero logic, passes
> > Level 1
> > > > > > > >> straight
> > > > > > > >> > > through
> > > > > > > >> > > > > > to
> > > > > > > >> > > > > > >> > Level
> > > > > > > >> > > > > > >> > > > 3.
> > > > > > > >> > > > > > >> > > > > > Same
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> anti-pattern
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> > as
> > PolarisEventManager.
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> > 7.  Iceberg
> community
> > > > > > alignment
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >    •   Payload-type
> > > > > extension
> > > > > > > >> needs
> > > > > > > >> > > > > > discussion
> > > > > > > >> > > > > > >> on
> > > > > > > >> > > > > > >> > > > > > > dev@iceberg.
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> obelix74's
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> > Feb thread got zero
> > > > replies.
> > > > > > > >> Needs a
> > > > > > > >> > > > > > committer
> > > > > > > >> > > > > > >> > > voice.
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> > Lets confirm
> > > > prioritization
> > > > > > in the
> > > > > > > >> > > first
> > > > > > > >> > > > > > >> session.
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> > -ej
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> > On Tue, Apr 21, 2026
> > at
> > > > > > 3:18 PM
> > > > > > > >> Yufei
> > > > > > > >> > > Gu <
> > > > > > > >> > > > > > >> > > > > > > > flyrain000@gmail.com>
> > > > > > > >> > > > > > >> > > > > > > > > > > > wrote:
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> Thanks everyone for
> > > > > > continuing to
> > > > > > > >> > > drive
> > > > > > > >> > > > > this
> > > > > > > >> > > > > > >> > > > forward. I
> > > > > > > >> > > > > > >> > > > > > > agree
> > > > > > > >> > > > > > >> > > > > > > > > > that
> > > > > > > >> > > > > > >> > > > > > > > > > > > the
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> problem is getting
> > > > complex
> > > > > > enough
> > > > > > > >> > > that a
> > > > > > > >> > > > > > more
> > > > > > > >> > > > > > >> > > > > structured
> > > > > > > >> > > > > > >> > > > > > > > > > discussion
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> would
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> help.
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >>
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> +1 on setting up a
> > > > biweekly
> > > > > > sync
> > > > > > > >> for
> > > > > > > >> > > the
> > > > > > > >> > > > > > >> metrics
> > > > > > > >> > > > > > >> > > > > > > > architecture.
> > > > > > > >> > > > > > >> > > > > > > > > > I’m
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> happy
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> to
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> join.
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >>
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> Yufei
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >>
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >>
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> On Tue, Apr 21,
> 2026
> > at
> > > > > > 2:34 PM
> > > > > > > >> EJ
> > > > > > > >> > > Wang <
> > > > > > > >> > > > > > >> > > > > > > > > > > > >>
> > > > > ej.wang.dev.interest@gmail.com>
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> wrote:
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >>
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> > Also, I've been
> > looking
> > > > > > more
> > > > > > > >> > > closely at
> > > > > > > >> > > > > > the
> > > > > > > >> > > > > > >> > > > > > *persistence
> > > > > > > >> > > > > > >> > > > > > > > > schema
> > > > > > > >> > > > > > >> > > > > > > > > > > in
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> the
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> > current metrics
> > work*,
> > > > > and
> > > > > > I
> > > > > > > >> think
> > > > > > > >> > > > > > there's a
> > > > > > > >> > > > > > >> > > > > structural
> > > > > > > >> > > > > > >> > > > > > > > > > rigidity
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> problem
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> > worth raising
> > before
> > > > the
> > > > > > shape
> > > > > > > >> > gets
> > > > > > > >> > > > > locked
> > > > > > > >> > > > > > >> in.
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> > Right now we have
> > two
> > > > > > separate
> > > > > > > >> > > tables
> > > > > > > >> > > > > > >> > > > > > > (scan_metrics_report
> > > > > > > >> > > > > > >> > > > > > > > > and
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >
> > commit_metrics_report),
> > > > > > each
> > > > > > > >> with
> > > > > > > >> > > ~25
> > > > > > > >> > > > > > >> flattened
> > > > > > > >> > > > > > >> > > > > columns
> > > > > > > >> > > > > > >> > > > > > > > that
> > > > > > > >> > > > > > >> > > > > > > > > > > > directly
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> > mirror the
> Iceberg
> > > > report
> > > > > > > >> fields.
> > > > > > > >> > > The
> > > > > > > >> > > > > SPI
> > > > > > > >> > > > > > >> > follows
> > > > > > > >> > > > > > >> > > > the
> > > > > > > >> > > > > > >> > > > > > > same
> > > > > > > >> > > > > > >> > > > > > > > > > split:
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> > writeScanReport
> and
> > > > > > > >> > > writeCommitReport as
> > > > > > > >> > > > > > >> > separate
> > > > > > > >> > > > > > >> > > > > > > methods,
> > > > > > > >> > > > > > >> > > > > > > > > with
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> per-type
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> > record classes,
> > > > > > converters, and
> > > > > > > >> > > model
> > > > > > > >> > > > > > >> objects.
> > > > > > > >> > > > > > >> > > *The
> > > > > > > >> > > > > > >> > > > > > > > practical
> > > > > > > >> > > > > > >> > > > > > > > > > > cost:
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> > adding a new
> metric
> > > > type
> > > > > > > >> > > (operational
> > > > > > > >> > > > > > >> metrics,
> > > > > > > >> > > > > > >> > > for
> > > > > > > >> > > > > > >> > > > > > > example)
> > > > > > > >> > > > > > >> > > > > > > > > > > > requires
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> a
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> new
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> > table, a new SPI
> > > > method,
> > > > > a
> > > > > > new
> > > > > > > >> > > record
> > > > > > > >> > > > > > >> class, a
> > > > > > > >> > > > > > >> > > new
> > > > > > > >> > > > > > >> > > > > > model
> > > > > > > >> > > > > > >> > > > > > > > > > class, a
> > > > > > > >> > > > > > >> > > > > > > > > > > > new
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> > converter branch,
> > and a
> > > > > > schema
> > > > > > > >> > > > > migration*.
> > > > > > > >> > > > > > >> > > That's a
> > > > > > > >> > > > > > >> > > > > lot
> > > > > > > >> > > > > > >> > > > > > > of
> > > > > > > >> > > > > > >> > > > > > > > > > > surface
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> area
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> > for what should
> be
> > "one
> > > > > > more
> > > > > > > >> kind
> > > > > > > >> > of
> > > > > > > >> > > > > > >> metric."
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> > *My bias* would
> be
> > > > > toward a
> > > > > > > >> single
> > > > > > > >> > > > > metrics
> > > > > > > >> > > > > > >> > table
> > > > > > > >> > > > > > >> > > > with
> > > > > > > >> > > > > > >> > > > > > *a
> > > > > > > >> > > > > > >> > > > > > > > > typed
> > > > > > > >> > > > > > >> > > > > > > > > > > JSON
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> > payload*.
> Something
> > > > like:
> > > > > > > >> > > metric_type
> > > > > > > >> > > > > > >> (enum),
> > > > > > > >> > > > > > >> > > > > > entity_id,
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> > table_identifier,
> > > > > > snapshot_id
> > > > > > > >> > > > > (nullable),
> > > > > > > >> > > > > > >> > > > > received_ts,
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> schema_version,
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> and
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> > a payload column
> > for
> > > > the
> > > > > > > >> > > metric-specific
> > > > > > > >> > > > > > >> data.
> > > > > > > >> > > > > > >> > > The
> > > > > > > >> > > > > > >> > > > > > > > > metric_type
> > > > > > > >> > > > > > >> > > > > > > > > > +
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> > schema_version
> pair
> > > > gives
> > > > > > us a
> > > > > > > >> > > > > > >> > forward-compatible
> > > > > > > >> > > > > > >> > > > > > > contract
> > > > > > > >> > > > > > >> > > > > > > > > for
> > > > > > > >> > > > > > >> > > > > > > > > > > the
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> payload
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> > shape. Adding a
> new
> > > > > metric
> > > > > > type
> > > > > > > >> > > becomes
> > > > > > > >> > > > > an
> > > > > > > >> > > > > > >> enum
> > > > > > > >> > > > > > >> > > > value
> > > > > > > >> > > > > > >> > > > > > > and a
> > > > > > > >> > > > > > >> > > > > > > > > > > payload
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> schema,
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> > not a schema
> > migration.
> > > > > One
> > > > > > > >> thing
> > > > > > > >> > I
> > > > > > > >> > > > > think
> > > > > > > >> > > > > > we
> > > > > > > >> > > > > > >> > need
> > > > > > > >> > > > > > >> > > > to
> > > > > > > >> > > > > > >> > > > > be
> > > > > > > >> > > > > > >> > > > > > > > > > > deliberate
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> about is
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> > the partition key
> > > > design.
> > > > > > If
> > > > > > > >> all
> > > > > > > >> > > metric
> > > > > > > >> > > > > > >> types
> > > > > > > >> > > > > > >> > > land
> > > > > > > >> > > > > > >> > > > in
> > > > > > > >> > > > > > >> > > > > > one
> > > > > > > >> > > > > > >> > > > > > > > > > table,
> > > > > > > >> > > > > > >> > > > > > > > > > > > scan
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> > metrics at scale
> > (high
> > > > > > > >> > concurrency,
> > > > > > > >> > > high
> > > > > > > >> > > > > > >> > > frequency
> > > > > > > >> > > > > > >> > > > > > across
> > > > > > > >> > > > > > >> > > > > > > > > many
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> tables)
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> > could easily
> > create hot
> > > > > > > >> > partitions.
> > > > > > > >> > > We'd
> > > > > > > >> > > > > > >> want
> > > > > > > >> > > > > > >> > the
> > > > > > > >> > > > > > >> > > > > > > > persistence
> > > > > > > >> > > > > > >> > > > > > > > > > > layer
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> to
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> be
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> > able to shard by
> > entity
> > > > > or
> > > > > > time
> > > > > > > >> > > range,
> > > > > > > >> > > > > and
> > > > > > > >> > > > > > >> that
> > > > > > > >> > > > > > >> > > > means
> > > > > > > >> > > > > > >> > > > > > the
> > > > > > > >> > > > > > >> > > > > > > > > > logical
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> schema
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> > needs to expose
> > enough
> > > > > > > >> structure
> > > > > > > >> > for
> > > > > > > >> > > > > > >> backends
> > > > > > > >> > > > > > >> > to
> > > > > > > >> > > > > > >> > > > > > > partition
> > > > > > > >> > > > > > >> > > > > > > > > on.
> > > > > > > >> > > > > > >> > > > > > > > > > I
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> don't
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> > think the current
> > > > > flattened
> > > > > > > >> layout
> > > > > > > >> > > gives
> > > > > > > >> > > > > > us
> > > > > > > >> > > > > > >> > that.
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> > This is getting
> > complex
> > > > > > enough
> > > > > > > >> > that
> > > > > > > >> > > I
> > > > > > > >> > > > > > don't
> > > > > > > >> > > > > > >> > think
> > > > > > > >> > > > > > >> > > > > > ad-hoc
> > > > > > > >> > > > > > >> > > > > > > > > PR/ML
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> threads
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> > will converge
> well.
> > > > > *Would
> > > > > > > >> people
> > > > > > > >> > be
> > > > > > > >> > > > > open
> > > > > > > >> > > > > > >> to a
> > > > > > > >> > > > > > >> > > > > biweekly
> > > > > > > >> > > > > > >> > > > > > > > sync
> > > > > > > >> > > > > > >> > > > > > > > > > for
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> metrics
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> > architecture?* I
> > think
> > > > 30
> > > > > > > >> minutes
> > > > > > > >> > > every
> > > > > > > >> > > > > > two
> > > > > > > >> > > > > > >> > weeks
> > > > > > > >> > > > > > >> > > > > with
> > > > > > > >> > > > > > >> > > > > > > > > > interested
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> parties
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> > would be enough
> to
> > work
> > > > > > through
> > > > > > > >> > the
> > > > > > > >> > > > > > schema,
> > > > > > > >> > > > > > >> SPI
> > > > > > > >> > > > > > >> > > > > shape,
> > > > > > > >> > > > > > >> > > > > > > and
> > > > > > > >> > > > > > >> > > > > > > > > read
> > > > > > > >> > > > > > >> > > > > > > > > > > API
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> design
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> > together. Happy
> to
> > help
> > > > > set
> > > > > > > >> that
> > > > > > > >> > up.
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> > -ej
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> > On Mon, Apr 20,
> > 2026 at
> > > > > > > >> 2:19 PM EJ
> > > > > > > >> > > Wang
> > > > > > > >> > > > > <
> > > > > > > >> > > > > > >> > > > > > > > > > > > >>
> > > > > ej.wang.dev.interest@gmail.com
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> > wrote:
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >> Reviewed #4115,
> > left a
> > > > > > > >> comment on
> > > > > > > >> > > the
> > > > > > > >> > > > > > code
> > > > > > > >> > > > > > >> > > > > > organization
> > > > > > > >> > > > > > >> > > > > > > > > side.
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >> One thing stood
> > out:
> > > > the
> > > > > > > >> metrics
> > > > > > > >> > > write
> > > > > > > >> > > > > > path
> > > > > > > >> > > > > > >> > > enters
> > > > > > > >> > > > > > >> > > > > > > through
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>
> > PolarisMetricsManager
> > > > on
> > > > > > > >> > > > > > MetaStoreManager,
> > > > > > > >> > > > > > >> but
> > > > > > > >> > > > > > >> > > the
> > > > > > > >> > > > > > >> > > > > new
> > > > > > > >> > > > > > >> > > > > > > > read
> > > > > > > >> > > > > > >> > > > > > > > > > path
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> bypasses
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >> MetaStoreManager
> > > > > entirely
> > > > > > and
> > > > > > > >> > goes
> > > > > > > >> > > > > > >> straight to
> > > > > > > >> > > > > > >> > > > > > > > > BasePersistence
> > > > > > > >> > > > > > >> > > > > > > > > > > via
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>
> > > > > > callContext.getMetaStore().
> > > > > > > >> That
> > > > > > > >> > > means
> > > > > > > >> > > > > > the
> > > > > > > >> > > > > > >> > read
> > > > > > > >> > > > > > >> > > > API
> > > > > > > >> > > > > > >> > > > > > only
> > > > > > > >> > > > > > >> > > > > > > > > works
> > > > > > > >> > > > > > >> > > > > > > > > > > for
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> backends
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >> that implement
> > > > > > > >> BasePersistence.
> > > > > > > >> > > NoSQL
> > > > > > > >> > > > > and
> > > > > > > >> > > > > > >> > remote
> > > > > > > >> > > > > > >> > > > > > > backends
> > > > > > > >> > > > > > >> > > > > > > > > > can't
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> participate.
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >> Stepping back, I
> > think
> > > > > the
> > > > > > > >> > metrics
> > > > > > > >> > > > > > >> subsystem
> > > > > > > >> > > > > > >> > is
> > > > > > > >> > > > > > >> > > > > > growing
> > > > > > > >> > > > > > >> > > > > > > > into
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> something
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >> real (write +
> > read +
> > > > > REST
> > > > > > API
> > > > > > > >> +
> > > > > > > >> > > AuthZ +
> > > > > > > >> > > > > > >> > > > pagination)
> > > > > > > >> > > > > > >> > > > > > *but
> > > > > > > >> > > > > > >> > > > > > > > the
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> persistence
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >> side is split
> > across
> > > > two
> > > > > > > >> layers
> > > > > > > >> > in
> > > > > > > >> > > a
> > > > > > > >> > > > > way
> > > > > > > >> > > > > > >> > that's
> > > > > > > >> > > > > > >> > > > hard
> > > > > > > >> > > > > > >> > > > > > to
> > > > > > > >> > > > > > >> > > > > > > > > > > extend*. I
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> put
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >> together two
> > diagrams
> > > > to
> > > > > > show
> > > > > > > >> > what
> > > > > > > >> > > I
> > > > > > > >> > > > > mean
> > > > > > > >> > > > > > >> (my
> > > > > > > >> > > > > > >> > > best
> > > > > > > >> > > > > > >> > > > > > > > effort).
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >> *Current state*
> > > > (Diagram
> > > > > > 1):
> > > > > > > >> > three
> > > > > > > >> > > > > > >> interfaces
> > > > > > > >> > > > > > >> > at
> > > > > > > >> > > > > > >> > > > > three
> > > > > > > >> > > > > > >> > > > > > > > > > different
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> levels.
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >> The
> engine-facing
> > SPI
> > > > > > > >> > > > > > >> (PolarisMetricsReporter)
> > > > > > > >> > > > > > >> > > is
> > > > > > > >> > > > > > >> > > > > > clean.
> > > > > > > >> > > > > > >> > > > > > > > But
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>
> > PolarisMetricsManager
> > > > on
> > > > > > > >> > > > > MetaStoreManager
> > > > > > > >> > > > > > >> is a
> > > > > > > >> > > > > > >> > > > > > > passthrough
> > > > > > > >> > > > > > >> > > > > > > > > to
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>
> > MetricsPersistence on
> > > > > > > >> > > BasePersistence.
> > > > > > > >> > > > > > The
> > > > > > > >> > > > > > >> > @Beta
> > > > > > > >> > > > > > >> > > > > > > > annotation
> > > > > > > >> > > > > > >> > > > > > > > > > and
> > > > > > > >> > > > > > >> > > > > > > > > > > > SPI
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> javadoc
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >> are on the
> > > > > BasePersistence
> > > > > > > >> layer,
> > > > > > > >> > > while
> > > > > > > >> > > > > > the
> > > > > > > >> > > > > > >> > > actual
> > > > > > > >> > > > > > >> > > > > > > > extension
> > > > > > > >> > > > > > >> > > > > > > > > > > > points
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>
> > > > (PolarisMetricsReporter,
> > > > > > > >> > > > > > >> > PolarisMetricsManager)
> > > > > > > >> > > > > > >> > > > > carry
> > > > > > > >> > > > > > >> > > > > > no
> > > > > > > >> > > > > > >> > > > > > > > > > > stability
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >> annotation. The
> > write
> > > > > path
> > > > > > > >> goes
> > > > > > > >> > > through
> > > > > > > >> > > > > > the
> > > > > > > >> > > > > > >> > > > > > > > MetaStoreManager
> > > > > > > >> > > > > > >> > > > > > > > > > > > layer,
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> the
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >> read path
> doesn't.
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >> *What I
> envision*
> > > > > > (Diagram 2):
> > > > > > > >> > two
> > > > > > > >> > > SPIs
> > > > > > > >> > > > > > at
> > > > > > > >> > > > > > >> two
> > > > > > > >> > > > > > >> > > > > levels.
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>
> > PolarisMetricsReporter
> > > > > > stays
> > > > > > > >> as
> > > > > > > >> > the
> > > > > > > >> > > > > > >> > > engine-facing
> > > > > > > >> > > > > > >> > > > > SPI.
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>
> > PolarisMetricsManager
> > > > > > becomes
> > > > > > > >> the
> > > > > > > >> > > > > > >> > backend-facing
> > > > > > > >> > > > > > >> > > > SPI
> > > > > > > >> > > > > > >> > > > > > > with
> > > > > > > >> > > > > > >> > > > > > > > > both
> > > > > > > >> > > > > > >> > > > > > > > > > > > write
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> and
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >> read methods at
> > the
> > > > > > > >> > > MetaStoreManager
> > > > > > > >> > > > > > level,
> > > > > > > >> > > > > > >> > > where
> > > > > > > >> > > > > > >> > > > > any
> > > > > > > >> > > > > > >> > > > > > > > > backend
> > > > > > > >> > > > > > >> > > > > > > > > > > > (JDBC,
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> NoSQL,
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >> remote) can
> > implement
> > > > > > them.
> > > > > > > >> > > > > > >> MetricsPersistence
> > > > > > > >> > > > > > >> > > on
> > > > > > > >> > > > > > >> > > > > > > > > > > BasePersistence
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> goes
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >> away. Where
> > metrics
> > > > > > actually
> > > > > > > >> land
> > > > > > > >> > > is an
> > > > > > > >> > > > > > >> > > > > implementation
> > > > > > > >> > > > > > >> > > > > > > > > detail,
> > > > > > > >> > > > > > >> > > > > > > > > > > > not a
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> core
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >> interface.
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >> *Minor naming
> > thing*:
> > > > > > > >> > > > > > >> PolarisMetricsReporter
> > > > > > > >> > > > > > >> > is
> > > > > > > >> > > > > > >> > > > > > broader
> > > > > > > >> > > > > > >> > > > > > > > than
> > > > > > > >> > > > > > >> > > > > > > > > > > what
> > > > > > > >> > > > > > >> > > > > > > > > > > > it
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >> actually
> handles.
> > It
> > > > > only
> > > > > > > >> accepts
> > > > > > > >> > > > > Iceberg
> > > > > > > >> > > > > > >> REST
> > > > > > > >> > > > > > >> > > > > Catalog
> > > > > > > >> > > > > > >> > > > > > > > > metrics
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> (ScanReport,
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >> CommitReport via
> > > > > > > >> MetricsReport).
> > > > > > > >> > > > > Generic
> > > > > > > >> > > > > > >> table
> > > > > > > >> > > > > > >> > > > > metrics
> > > > > > > >> > > > > > >> > > > > > > or
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> operational
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >> metrics aren't
> in
> > > > scope.
> > > > > > Not
> > > > > > > >> > > blocking,
> > > > > > > >> > > > > > but
> > > > > > > >> > > > > > >> > worth
> > > > > > > >> > > > > > >> > > > > > noting
> > > > > > > >> > > > > > >> > > > > > > if
> > > > > > > >> > > > > > >> > > > > > > > > the
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> metrics
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >> surface expands.
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >> *Rough sketch of
> > how
> > > > to
> > > > > > get
> > > > > > > >> > there*:
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>  1.  Add read
> > methods
> > > > to
> > > > > > > >> > > > > > >> PolarisMetricsManager
> > > > > > > >> > > > > > >> > > > > > > > > > (listScanReports,
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>
> listCommitReports)
> > > > with
> > > > > > > >> default
> > > > > > > >> > > no-op,
> > > > > > > >> > > > > > >> same as
> > > > > > > >> > > > > > >> > > the
> > > > > > > >> > > > > > >> > > > > > > > existing
> > > > > > > >> > > > > > >> > > > > > > > > > > write
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> methods.
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >> (Probably make
> > > > > > > >> > > PolarisMetricsManager
> > > > > > > >> > > > > more
> > > > > > > >> > > > > > >> > > explicit
> > > > > > > >> > > > > > >> > > > > on
> > > > > > > >> > > > > > >> > > > > > > > being
> > > > > > > >> > > > > > >> > > > > > > > > > > > Iceberg
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >> specific like
> > package
> > > > > > name or
> > > > > > > >> > class
> > > > > > > >> > > > > name
> > > > > > > >> > > > > > >> etc.)
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>  2.  Wire
> > > > > > > >> MetricsReportsService
> > > > > > > >> > > through
> > > > > > > >> > > > > > >> > > > > > MetaStoreManager
> > > > > > > >> > > > > > >> > > > > > > > > > instead
> > > > > > > >> > > > > > >> > > > > > > > > > > > of
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>
> > > > > > callContext.getMetaStore().
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>  3.  Extract
> > metrics
> > > > > > > >> persistence
> > > > > > > >> > > from
> > > > > > > >> > > > > > >> > > > > > > > > JdbcBasePersistenceImpl
> > > > > > > >> > > > > > >> > > > > > > > > > > into
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> its
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >> own class. That
> > file
> > > > > > carries
> > > > > > > >> ~7
> > > > > > > >> > > > > > >> > > responsibilities,
> > > > > > > >> > > > > > >> > > > > > > metrics
> > > > > > > >> > > > > > >> > > > > > > > > > being
> > > > > > > >> > > > > > >> > > > > > > > > > > > one
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> of
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> them.
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>  4.  Remove
> > > > > > MetricsPersistence
> > > > > > > >> > from
> > > > > > > >> > > > > > >> > > > BasePersistence.
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >> *None of this
> > needs to
> > > > > > happen
> > > > > > > >> in
> > > > > > > >> > > #4115.
> > > > > > > >> > > > > > >> But if
> > > > > > > >> > > > > > >> > > the
> > > > > > > >> > > > > > >> > > > > > > > direction
> > > > > > > >> > > > > > >> > > > > > > > > > > makes
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> sense,
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >> it would be good
> > to
> > > > > align
> > > > > > > >> before
> > > > > > > >> > > the
> > > > > > > >> > > > > > >> metrics
> > > > > > > >> > > > > > >> > > > surface
> > > > > > > >> > > > > > >> > > > > > > grows
> > > > > > > >> > > > > > >> > > > > > > > > > > > further.
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> Curious
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >> what others
> > think.*
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >> *My mental model
> > > > note*:
> > > > > > Level
> > > > > > > >> 1
> > > > > > > >> > > > > > >> > > MetaStoreManager;
> > > > > > > >> > > > > > >> > > > > > level
> > > > > > > >> > > > > > >> > > > > > > 2
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> transactional
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >> persistence;
> > level 3
> > > > > base
> > > > > > > >> > > persistence
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >> Diagram 1
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >> <
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >>
> > > > > > > >> > > > > > >> > > > > > > > > > > > >>
> > > > > > > >> > > > > > >> > > > > > > > > > > >
> > > > > > > >> > > > > > >> > > > > > > > > > >
> > > > > > > >> > > > > > >> > > > > > > > > >
> > > > > > > >> > > > > > >> > > > > > > > >
> > > > > > > >> > > > > > >> > > > > > > >
> > > > > > > >> > > > > > >> > > > > > >
> > > > > > > >> > > > > > >> > > > > >
> > > > > > > >> > > > > > >> > > > >
> > > > > > > >> > > > > > >> > > >
> > > > > > > >> > > > > > >> > >
> > > > > > > >> > > > > > >> >
> > > > > > > >> > > > > > >>
> > > > > > > >> > > > > >
> > > > > > > >> > > > >
> > > > > > > >> > >
> > > > > > > >> >
> > > > > > > >>
> > > > > >
> > > > >
> > > >
> >
> https://urldefense.com/v3/__https://www.plantuml.com/plantuml/uml/bLHDR-Cs4BthLmpIYupw0zbkKQ1r3M-S7Bp8xhhM7WCOb3IM65EaGD9EX2RzxHrHb4CxRelwa4YSDu_lpOVcnZ9jzvM8BBS2uGjQpJC3dtHMSekPtMk44IpsMgEqa5XcCOhCZikQQLP1pR8TAp2n3ILhmZDP20m0fcIvUkAoW2qJXd9z1bpToO9BX3WXu0ucy5rpgGPNm0nW5_epUWtm2Ue3pn3kMOFQmKntGZW0BYtgBSi8k5A2QMwybJNMIbFiGSR9QZc4nUqIvikStF0jHprua5C-amge42aNt3R0f5JaaoivdV2Pkqbx4hee4ymOkBh5BTiB-_uIeGeo8zL8rPsPl4DktdEiK1jkB1NdZCRbrSTecDe_mlHbF0wvBmCkaOH5_S8a_TTTKI6-nmCAkEw4LpxsZ-LbYLKQFKMNOgf_wuM7_bV9gOer5SYMMksBSWXFcbi49KNZXNLicwfe3TETC7gPdPqI7uBcHMb1RSzYq34c6PDUM9mn8HRsUTZEiDBve3NjVZumBj0U7SS37mGO7vcwtiK-_pU7U7L_f-digo9YbhSwIfMRwIITKGXbxdIUTCGF1SeCJxloKsU-3k9ddRbX1eDq1q_fx1JbBGT0glVyXimDuP4TQ5qpCAmnGEj2s_6n5mtn1z-97-63itFQZLPO1Ev2tu_WF7Ju-VPc0Skg5bYXxBhkY1xpD7EM_7fyflSpIsqMgVth5xhVr4eQxWQ8enaSAJQSG16yFSDuJ798rrcXr_3n-lfdk7icQjEBmFujL7AodiP_Y4Z7-YxvtZNs4zMgpNTl6tF8sglyPsmqchrjvQ-m-aP94r-TwCA2Ka8upPJZwtvSpoYCXkYMZU2NXvRMBfq9P3i3Le4VAZUAlUZ_oPKsxPgY0Q_BSKLkyr9bhQhQrJjo_x3TPlIB0DPjnMfcIoYP0QaYw1a0fTKDr8fB6ntNuvmoL1ZGkXa69Njh43zf9GiGxHQrA_jDYWRSzF5--WmTVrN97_Sm8LbLUy_lGBmLanJjFkDlGkRqjA_4tm00__;!!Iz9xO38YGHZK!4rnlVIqBQQQ6xpGe1PMV58edOS_cGrKoqdfhw4qSp_JQCavvEjaxXk0OwGPnxyEcO60AzKQK_XDtp7H3DdkFXcrXuTBISHg$
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >> :
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >> [image:
> image.png]
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >> Diagram 2
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >> <
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >>
> > > > > > > >> > > > > > >> > > > > > > > > > > > >>
> > > > > > > >> > > > > > >> > > > > > > > > > > >
> > > > > > > >> > > > > > >> > > > > > > > > > >
> > > > > > > >> > > > > > >> > > > > > > > > >
> > > > > > > >> > > > > > >> > > > > > > > >
> > > > > > > >> > > > > > >> > > > > > > >
> > > > > > > >> > > > > > >> > > > > > >
> > > > > > > >> > > > > > >> > > > > >
> > > > > > > >> > > > > > >> > > > >
> > > > > > > >> > > > > > >> > > >
> > > > > > > >> > > > > > >> > >
> > > > > > > >> > > > > > >> >
> > > > > > > >> > > > > > >>
> > > > > > > >> > > > > >
> > > > > > > >> > > > >
> > > > > > > >> > >
> > > > > > > >> >
> > > > > > > >>
> > > > > >
> > > > >
> > > >
> >
> https://urldefense.com/v3/__https://www.plantuml.com/plantuml/uml/VLLDR-8m4BtdLupO2sWBLVU8AaGB7AXAbssGzb896SS9RXqxjKqBwkv_tt7iV43fdaZYDpFlpRmnOsE9jhjSH9PRmM31hERKm8scMsuPjJlDe0yheZDc8RR4iYWoBrmMH9CS2a9VICPYUy1OZN0YCy5Q0BCbYNhdCeEK28En8G8wCvbnoQ0R8_05Bc6bkLIz3X03p1zzH7zR-9ZfDquPt9C3qoNCX2yV4G2NbkcKu5jdgGJHt0GbZwnG6i-UP3TUpk5gM6Ldqke350eZUqzoCft3U9xWHvxoa5-7K4nF1J46EbEMafsmdrCBbQ44gVggy18IZrn_ph5asd1ZiIKdQSgueZvjXrQFSFrdC3YN-nXmBacxbGiYyLVxLaBtdhqn0LSzdBDhqQtQoOJeGyad3z0lUqnYgpGB6Ns8oVyta00Dy_WnX0tIOZ8v6SYxHll1TrH6aejAik-mh-AphVFCwSUQqFypElag5QRGFDjQKEd96K1P8QP41c9TzA_IIQyvdAWyv_RSiS3skb0_EzDDkK2v5xWF6MiGFlvhpFLcD2Dq2pml14gaF67eQkmd8gulDoC4kSOu6KVpkvlUJg1RTbWISU40RdBUUS_9XfRZ2dwxm_SW8LYFISgm_MnlDQ6M9P1gbKEc4X-2pH_FvJCkCqm9pbVjD6LrwdLeOrDWfOaqc8Wh9BE85oNKxkNQ6o4yGRy_Eae0G_G8tZv81d3bHDB23WOdisohVr3nh_j6lbSjbNaLRTc8UgtPbAU1J_tygOfZX9DWEJeHDvYx-qmSi5FgNLPZwHrHcUsncGQ5-skhUclpE5fo4ounpFauYrUbkU6ccfnxMvitwag4IyerhTxj8In_Oj1bDO4pQru674loYrGlULHLEGCjwJJ8gDoVZR8MxO4BT3IzRvIcAQKezC6xpziGnTyImrfEGyJI_OcKfgtxIvnTqFEMS17L9Z-jsARN5FmTheP7HtSdtOMT0B4GY2FYHXxgQmMtj2bRqiLFGapiVe1_QVKDrkqXcm83aFEXnMYCZ-xlyHy__;!!Iz9xO38YGHZK!4rnlVIqBQQQ6xpGe1PMV58edOS_cGrKoqdfhw4qSp_JQCavvEjaxXk0OwGPnxyEcO60AzKQK_XDtp7H3DdkFXcrXSpmqcNk$
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >> :
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >> [image:
> image.png]
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>  -ej
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >> On Wed, Apr 15,
> > 2026
> > > > at
> > > > > > > >> 8:22 AM
> > > > > > > >> > > Dmitri
> > > > > > > >> > > > > > >> > > > Bourlatchkov
> > > > > > > >> > > > > > >> > > > > <
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> dimas@apache.org>
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >> wrote:
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> Hi All,
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>>
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> Heads up: The
> > current
> > > > > > state
> > > > > > > >> of
> > > > > > > >> > PR
> > > > > > > >> > > > > [4115]
> > > > > > > >> > > > > > >> > looks
> > > > > > > >> > > > > > >> > > > > pretty
> > > > > > > >> > > > > > >> > > > > > > > solid
> > > > > > > >> > > > > > >> > > > > > > > > > to
> > > > > > > >> > > > > > >> > > > > > > > > > > > me.
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> I
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> believe this PR
> > is
> > > > > > > >> approaching a
> > > > > > > >> > > > > > mergeable
> > > > > > > >> > > > > > >> > > > > condition.
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>>
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> Please post
> your
> > > > > reviews
> > > > > > if
> > > > > > > >> you
> > > > > > > >> > > have
> > > > > > > >> > > > > any
> > > > > > > >> > > > > > >> > > > comments.
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>>
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> [4115]
> > > > > > > >> > > > > > >> > > >
> > > > > > > >> >
> > > > > > > >>
> > > > > >
> > > > >
> > > >
> >
> https://urldefense.com/v3/__https://github.com/apache/polaris/pull/4115__;!!Iz9xO38YGHZK!4rnlVIqBQQQ6xpGe1PMV58edOS_cGrKoqdfhw4qSp_JQCavvEjaxXk0OwGPnxyEcO60AzKQK_XDtp7H3DdkFXcrXtFRscKs$
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>>
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> Thanks,
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> Dmitri.
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>>
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> On Tue, Mar 3,
> > 2026
> > > > at
> > > > > > > >> 3:29 PM
> > > > > > > >> > > Anand
> > > > > > > >> > > > > > Kumar
> > > > > > > >> > > > > > >> > > > Sankaran
> > > > > > > >> > > > > > >> > > > > > via
> > > > > > > >> > > > > > >> > > > > > > > > dev <
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>>
> > > > dev@polaris.apache.org
> > > > > >
> > > > > > > >> wrote:
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>>
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > Hi Yufei and
> > > > Dmitri,
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> >
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > Here is a
> > proposal
> > > > > for
> > > > > > the
> > > > > > > >> > REST
> > > > > > > >> > > > > > >> endpoints
> > > > > > > >> > > > > > >> > for
> > > > > > > >> > > > > > >> > > > > > metrics
> > > > > > > >> > > > > > >> > > > > > > > and
> > > > > > > >> > > > > > >> > > > > > > > > > > > events.
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> >
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> >
> > > > > > > >> > > > > > >> > > > > >
> > > > > > > >> >
> > > > > > > >>
> > > > > >
> > > > >
> > > >
> >
> https://urldefense.com/v3/__https://github.com/apache/polaris/pull/3924/changes__;!!Iz9xO38YGHZK!4rnlVIqBQQQ6xpGe1PMV58edOS_cGrKoqdfhw4qSp_JQCavvEjaxXk0OwGPnxyEcO60AzKQK_XDtp7H3DdkFXcrXcjoPhSs$
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> >
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > I did not see
> > any
> > > > > > > >> precursors
> > > > > > > >> > for
> > > > > > > >> > > > > > >> raising a
> > > > > > > >> > > > > > >> > PR
> > > > > > > >> > > > > > >> > > > for
> > > > > > > >> > > > > > >> > > > > > > > > > proposals,
> > > > > > > >> > > > > > >> > > > > > > > > > > so
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> trying
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > this.  Please
> > let
> > > > me
> > > > > > know
> > > > > > > >> what
> > > > > > > >> > > you
> > > > > > > >> > > > > > >> think.
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> >
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > -
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > Anand
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> >
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > From: Anand
> > Kumar
> > > > > > Sankaran
> > > > > > > >> <
> > > > > > > >> > > > > > >> > > > > > > anand.sankaran@workday.com
> > > > > > > >> > > > > > >> > > > > > > > >
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > Date: Monday,
> > March
> > > > > 2,
> > > > > > > >> 2026 at
> > > > > > > >> > > > > > 10:25 AM
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > To:
> > > > > > dev@polaris.apache.org
> > > > > > > >> <
> > > > > > > >> > > > > > >> > > > > dev@polaris.apache.org
> > > > > > > >> > > > > > >> > > > > > >
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > Subject: Re:
> > > > Polaris
> > > > > > > >> Telemetry
> > > > > > > >> > > and
> > > > > > > >> > > > > > Audit
> > > > > > > >> > > > > > >> > > Trail
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> >
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > About the
> REST
> > API,
> > > > > > based
> > > > > > > >> on
> > > > > > > >> > my
> > > > > > > >> > > use
> > > > > > > >> > > > > > >> cases:
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> >
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> >
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> >   1.
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > I want to be
> > able
> > > > to
> > > > > > query
> > > > > > > >> > > commit
> > > > > > > >> > > > > > >> metrics
> > > > > > > >> > > > > > >> > to
> > > > > > > >> > > > > > >> > > > > track
> > > > > > > >> > > > > > >> > > > > > > > files
> > > > > > > >> > > > > > >> > > > > > > > > > > added
> > > > > > > >> > > > > > >> > > > > > > > > > > > /
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> removed
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > per commit,
> > along
> > > > > with
> > > > > > > >> record
> > > > > > > >> > > > > counts.
> > > > > > > >> > > > > > >> The
> > > > > > > >> > > > > > >> > > > > ingestion
> > > > > > > >> > > > > > >> > > > > > > > > > pipeline
> > > > > > > >> > > > > > >> > > > > > > > > > > > that
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> writes
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > this data is
> > owned
> > > > by
> > > > > > us
> > > > > > > >> and
> > > > > > > >> > we
> > > > > > > >> > > are
> > > > > > > >> > > > > > >> > > guaranteed
> > > > > > > >> > > > > > >> > > > to
> > > > > > > >> > > > > > >> > > > > > > write
> > > > > > > >> > > > > > >> > > > > > > > > > this
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> information
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > for each
> write.
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> >   2.
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > I want to be
> > able
> > > > to
> > > > > > query
> > > > > > > >> > scan
> > > > > > > >> > > > > > metrics
> > > > > > > >> > > > > > >> for
> > > > > > > >> > > > > > >> > > > > read. I
> > > > > > > >> > > > > > >> > > > > > > > > > > understand
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> clients
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> do
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > not fulfill
> > this
> > > > > > > >> requirement.
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> >   3.
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > I want to be
> > able
> > > > to
> > > > > > query
> > > > > > > >> the
> > > > > > > >> > > > > events
> > > > > > > >> > > > > > >> table
> > > > > > > >> > > > > > >> > > > > (events
> > > > > > > >> > > > > > >> > > > > > > are
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> persisted) -
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> this
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > may supersede
> > #2, I
> > > > > am
> > > > > > not
> > > > > > > >> > sure
> > > > > > > >> > > yet.
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> >
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > All this
> > > > information
> > > > > > is in
> > > > > > > >> the
> > > > > > > >> > > JDBC
> > > > > > > >> > > > > > >> based
> > > > > > > >> > > > > > >> > > > > > persistence
> > > > > > > >> > > > > > >> > > > > > > > > model
> > > > > > > >> > > > > > >> > > > > > > > > > > and
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> is
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > persisted in
> > the
> > > > > > > >> metastore. I
> > > > > > > >> > > > > > currently
> > > > > > > >> > > > > > >> > don’t
> > > > > > > >> > > > > > >> > > > > have
> > > > > > > >> > > > > > >> > > > > > a
> > > > > > > >> > > > > > >> > > > > > > > need
> > > > > > > >> > > > > > >> > > > > > > > > > to
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> query
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > prometheus or
> > open
> > > > > > > >> telemetry.
> > > > > > > >> > I
> > > > > > > >> > > do
> > > > > > > >> > > > > > >> publish
> > > > > > > >> > > > > > >> > > some
> > > > > > > >> > > > > > >> > > > > > > events
> > > > > > > >> > > > > > >> > > > > > > > to
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> Prometheus
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> and
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > they are
> > forwarded
> > > > to
> > > > > > our
> > > > > > > >> > > dashboards
> > > > > > > >> > > > > > >> > > elsewhere.
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> >
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > About the CLI
> > > > > > utilities, I
> > > > > > > >> > > meant the
> > > > > > > >> > > > > > >> admin
> > > > > > > >> > > > > > >> > > user
> > > > > > > >> > > > > > >> > > > > > > > > utilities.
> > > > > > > >> > > > > > >> > > > > > > > > > In
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> one of
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> the
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > earliest
> > drafts of
> > > > my
> > > > > > > >> > proposal,
> > > > > > > >> > > > > > Prashant
> > > > > > > >> > > > > > >> > > > > mentioned
> > > > > > > >> > > > > > >> > > > > > > that
> > > > > > > >> > > > > > >> > > > > > > > > the
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> metrics
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> tables
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > can grow
> > > > indefinitely
> > > > > > and
> > > > > > > >> > that a
> > > > > > > >> > > > > > similar
> > > > > > > >> > > > > > >> > > > problem
> > > > > > > >> > > > > > >> > > > > > > exists
> > > > > > > >> > > > > > >> > > > > > > > > > with
> > > > > > > >> > > > > > >> > > > > > > > > > > > the
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> events
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > table as
> well.
> > We
> > > > > > discussed
> > > > > > > >> > that
> > > > > > > >> > > > > > >> cleaning
> > > > > > > >> > > > > > >> > up
> > > > > > > >> > > > > > >> > > of
> > > > > > > >> > > > > > >> > > > > old
> > > > > > > >> > > > > > >> > > > > > > > > records
> > > > > > > >> > > > > > >> > > > > > > > > > > > from
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> both
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > metrics
> tables
> > and
> > > > > > events
> > > > > > > >> > > tables can
> > > > > > > >> > > > > > be
> > > > > > > >> > > > > > >> > done
> > > > > > > >> > > > > > >> > > > via
> > > > > > > >> > > > > > >> > > > > a
> > > > > > > >> > > > > > >> > > > > > > CLI
> > > > > > > >> > > > > > >> > > > > > > > > > > utility.
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> >
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > I see that
> > Yufei
> > > > has
> > > > > > > >> covered
> > > > > > > >> > the
> > > > > > > >> > > > > > >> discussion
> > > > > > > >> > > > > > >> > > > about
> > > > > > > >> > > > > > >> > > > > > > > > > > datasources.
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> >
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > -
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > Anand
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> >
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> >
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> >
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > From: Yufei
> Gu
> > <
> > > > > > > >> > > > > flyrain000@gmail.com>
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > Date: Friday,
> > > > > February
> > > > > > 27,
> > > > > > > >> > 2026
> > > > > > > >> > > at
> > > > > > > >> > > > > > >> 9:54 PM
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > To:
> > > > > > dev@polaris.apache.org
> > > > > > > >> <
> > > > > > > >> > > > > > >> > > > > dev@polaris.apache.org
> > > > > > > >> > > > > > >> > > > > > >
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > Subject: Re:
> > > > Polaris
> > > > > > > >> Telemetry
> > > > > > > >> > > and
> > > > > > > >> > > > > > Audit
> > > > > > > >> > > > > > >> > > Trail
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> >
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > This Message
> Is
> > > > From
> > > > > an
> > > > > > > >> > External
> > > > > > > >> > > > > > Sender
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > This message
> > came
> > > > > from
> > > > > > > >> outside
> > > > > > > >> > > your
> > > > > > > >> > > > > > >> > > > organization.
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > Report
> > Suspicious<
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> >
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>>
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >>
> > > > > > > >> > > > > > >> > > > > > > > > > > > >>
> > > > > > > >> > > > > > >> > > > > > > > > > > >
> > > > > > > >> > > > > > >> > > > > > > > > > >
> > > > > > > >> > > > > > >> > > > > > > > > >
> > > > > > > >> > > > > > >> > > > > > > > >
> > > > > > > >> > > > > > >> > > > > > > >
> > > > > > > >> > > > > > >> > > > > > >
> > > > > > > >> > > > > > >> > > > > >
> > > > > > > >> > > > > > >> > > > >
> > > > > > > >> > > > > > >> > > >
> > > > > > > >> > > > > > >> > >
> > > > > > > >> > > > > > >> >
> > > > > > > >> > > > > > >>
> > > > > > > >> > > > > >
> > > > > > > >> > > > >
> > > > > > > >> > >
> > > > > > > >> >
> > > > > > > >>
> > > > > >
> > > > >
> > > >
> >
> https://us-phishalarm-ewt.proofpoint.com/EWT/v1/Iz9xO38YGHZK!YhNDZABkHi1B699ote2uMwpOZw8i0QMCGO2Szc-HshuABGhGvwPJcymE6G2oUUxtS8xDkSrtGTPm_I3QnVDHoLMk50m9v8z_nZKTkd-bnVUbreF1u0WnfV_X5eYevZl_$
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > >
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> >
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> >
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > As I
> mentioned
> > in
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> >
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>>
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >>
> > > > > > > >> > > > > > >> > > > > > > > > > > > >>
> > > > > > > >> > > > > > >> > > > > > > > > > > >
> > > > > > > >> > > > > > >> > > > > > > > > > >
> > > > > > > >> > > > > > >> > > > > > > > > >
> > > > > > > >> > > > > > >> > > > > > > > >
> > > > > > > >> > > > > > >> > > > > > > >
> > > > > > > >> > > > > > >> > > > > > >
> > > > > > > >> > > > > > >> > > > > >
> > > > > > > >> > > > > > >> > > > >
> > > > > > > >> > > > > > >> > > >
> > > > > > > >> > > > > > >> > >
> > > > > > > >> > > > > > >> >
> > > > > > > >> > > > > > >>
> > > > > > > >> > > > > >
> > > > > > > >> > > > >
> > > > > > > >> > >
> > > > > > > >> >
> > > > > > > >>
> > > > > >
> > > > >
> > > >
> >
> https://urldefense.com/v3/__https://github.com/apache/polaris/issues/3890__;!!Iz9xO38YGHZK!5EuyFFkk3vhRWVIRvQAWBSQfpJkTMA9HxugzDwXmN0LPPqhEFxYkFRGVhtb8AqUwXtDh2OplcMnbMDHKOxrvDU0$
> > > > > > > >> > >> > > >> > > > > > > > > > > > >> >> >>> ,
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > supporting
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > multiple data
> > > > sources
> > > > > > is
> > > > > > > >> not a
> > > > > > > >> > > > > trivial
> > > > > > > >> > > > > > >> > > change.
> > > > > > > >> > > > > > >> > > > I
> > > > > > > >> > > > > > >> > > > > > > would
> > > > > > > >> > > > > > >> > > > > > > > > > > strongly
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> recommend
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > starting
> with a
> > > > > design
> > > > > > > >> > document
> > > > > > > >> > > to
> > > > > > > >> > > > > > >> > carefully
> > > > > > > >> > > > > > >> > > > > > evaluate
> > > > > > > >> > > > > > >> > > > > > > > the
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> architectural
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > implications
> > and
> > > > long
> > > > > > term
> > > > > > > >> > > impact.
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> >
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > A REST
> > endpoint to
> > > > > > query
> > > > > > > >> > metrics
> > > > > > > >> > > > > seems
> > > > > > > >> > > > > > >> > > > reasonable
> > > > > > > >> > > > > > >> > > > > > > given
> > > > > > > >> > > > > > >> > > > > > > > > the
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> current
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> JDBC
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > based
> > persistence
> > > > > > model.
> > > > > > > >> That
> > > > > > > >> > > said,
> > > > > > > >> > > > > we
> > > > > > > >> > > > > > >> may
> > > > > > > >> > > > > > >> > > also
> > > > > > > >> > > > > > >> > > > > > > > consider
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> alternative
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > storage
> > models. For
> > > > > > > >> example,
> > > > > > > >> > if
> > > > > > > >> > > we
> > > > > > > >> > > > > > later
> > > > > > > >> > > > > > >> > > adopt
> > > > > > > >> > > > > > >> > > > a
> > > > > > > >> > > > > > >> > > > > > time
> > > > > > > >> > > > > > >> > > > > > > > > > series
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> system
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> such as
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > Prometheus to
> > store
> > > > > > > >> metrics,
> > > > > > > >> > the
> > > > > > > >> > > > > query
> > > > > > > >> > > > > > >> > model
> > > > > > > >> > > > > > >> > > > and
> > > > > > > >> > > > > > >> > > > > > > access
> > > > > > > >> > > > > > >> > > > > > > > > > > > patterns
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> would
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> be
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > fundamentally
> > > > > > different.
> > > > > > > >> > > Designing
> > > > > > > >> > > > > the
> > > > > > > >> > > > > > >> REST
> > > > > > > >> > > > > > >> > > API
> > > > > > > >> > > > > > >> > > > > > > without
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> considering
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> these
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > potential
> > > > evolutions
> > > > > > may
> > > > > > > >> limit
> > > > > > > >> > > > > > >> flexibility.
> > > > > > > >> > > > > > >> > > I'd
> > > > > > > >> > > > > > >> > > > > > > suggest
> > > > > > > >> > > > > > >> > > > > > > > > to
> > > > > > > >> > > > > > >> > > > > > > > > > > > start
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> with
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> the
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > use case.
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> >
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > Yufei
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> >
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> >
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > On Fri, Feb
> 27,
> > > > 2026
> > > > > at
> > > > > > > >> > 3:42 PM
> > > > > > > >> > > > > Dmitri
> > > > > > > >> > > > > > >> > > > > > Bourlatchkov <
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> dimas@apache.org>
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > wrote:
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> >
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > > Hi Anand,
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > >
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > > Sharing my
> > > > view...
> > > > > > > >> subject
> > > > > > > >> > to
> > > > > > > >> > > > > > >> discussion:
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > >
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > > 1. Adding
> > non-IRC
> > > > > > REST
> > > > > > > >> API
> > > > > > > >> > to
> > > > > > > >> > > > > > Polaris
> > > > > > > >> > > > > > >> is
> > > > > > > >> > > > > > >> > > > > > perfectly
> > > > > > > >> > > > > > >> > > > > > > > > fine.
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > >
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > > Figuring
> out
> > > > > specific
> > > > > > > >> > endpoint
> > > > > > > >> > > > > URIs
> > > > > > > >> > > > > > >> and
> > > > > > > >> > > > > > >> > > > > payloads
> > > > > > > >> > > > > > >> > > > > > > > might
> > > > > > > >> > > > > > >> > > > > > > > > > > > require
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> a
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> few
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > > roundtrips,
> > so
> > > > > > opening a
> > > > > > > >> > > separate
> > > > > > > >> > > > > > >> thread
> > > > > > > >> > > > > > >> > > for
> > > > > > > >> > > > > > >> > > > > that
> > > > > > > >> > > > > > >> > > > > > > > might
> > > > > > > >> > > > > > >> > > > > > > > > > be
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> best.
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > >
> Contributors
> > > > > commonly
> > > > > > > >> create
> > > > > > > >> > > > > Google
> > > > > > > >> > > > > > >> Docs
> > > > > > > >> > > > > > >> > > for
> > > > > > > >> > > > > > >> > > > > new
> > > > > > > >> > > > > > >> > > > > > > API
> > > > > > > >> > > > > > >> > > > > > > > > > > > proposals
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> too
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> (they
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > > fairly easy
> > to
> > > > > > update as
> > > > > > > >> the
> > > > > > > >> > > email
> > > > > > > >> > > > > > >> > > discussion
> > > > > > > >> > > > > > >> > > > > > > > > > progresses).
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > >
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > > There was a
> > > > > > suggestion to
> > > > > > > >> > try
> > > > > > > >> > > > > > Markdown
> > > > > > > >> > > > > > >> > > (with
> > > > > > > >> > > > > > >> > > > > PRs)
> > > > > > > >> > > > > > >> > > > > > > for
> > > > > > > >> > > > > > >> > > > > > > > > > > > proposals
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> [1]
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> ...
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > > feel free
> to
> > give
> > > > > it
> > > > > > a
> > > > > > > >> try
> > > > > > > >> > if
> > > > > > > >> > > you
> > > > > > > >> > > > > > are
> > > > > > > >> > > > > > >> > > > > comfortable
> > > > > > > >> > > > > > >> > > > > > > > with
> > > > > > > >> > > > > > >> > > > > > > > > > > that.
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > >
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > > 2. Could
> you
> > > > > clarify
> > > > > > > >> whether
> > > > > > > >> > > you
> > > > > > > >> > > > > > mean
> > > > > > > >> > > > > > >> end
> > > > > > > >> > > > > > >> > > > user
> > > > > > > >> > > > > > >> > > > > > > > > utilities
> > > > > > > >> > > > > > >> > > > > > > > > > or
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> admin
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> user
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > > utilities?
> > In the
> > > > > > latter
> > > > > > > >> > case
> > > > > > > >> > > > > those
> > > > > > > >> > > > > > >> might
> > > > > > > >> > > > > > >> > > be
> > > > > > > >> > > > > > >> > > > > more
> > > > > > > >> > > > > > >> > > > > > > > > > suitable
> > > > > > > >> > > > > > >> > > > > > > > > > > > for
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> the
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> Admin
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > > CLI (java)
> > not
> > > > the
> > > > > > Python
> > > > > > > >> > CLI,
> > > > > > > >> > > > > IMHO.
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > >
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > > Why would
> > these
> > > > > > > >> utilities be
> > > > > > > >> > > > > common
> > > > > > > >> > > > > > >> with
> > > > > > > >> > > > > > >> > > > > events?
> > > > > > > >> > > > > > >> > > > > > > > IMHO,
> > > > > > > >> > > > > > >> > > > > > > > > > > event
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> use
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> cases
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > are
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > > distinct
> from
> > > > > > scan/commit
> > > > > > > >> > > metrics.
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > >
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > > 3. I'd
> prefer
> > > > > > separating
> > > > > > > >> > > metrics
> > > > > > > >> > > > > > >> > > persistence
> > > > > > > >> > > > > > >> > > > > from
> > > > > > > >> > > > > > >> > > > > > > > > > MetaStore
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> persistence
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > at
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > > the code
> > level,
> > > > so
> > > > > > that
> > > > > > > >> they
> > > > > > > >> > > could
> > > > > > > >> > > > > > be
> > > > > > > >> > > > > > >> > mixed
> > > > > > > >> > > > > > >> > > > and
> > > > > > > >> > > > > > >> > > > > > > > matched
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> independently.
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > The
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > > separate
> > > > datasource
> > > > > > > >> question
> > > > > > > >> > > will
> > > > > > > >> > > > > > >> become
> > > > > > > >> > > > > > >> > a
> > > > > > > >> > > > > > >> > > > > > > non-issue
> > > > > > > >> > > > > > >> > > > > > > > > with
> > > > > > > >> > > > > > >> > > > > > > > > > > > that
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> approach,
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > I
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > > guess.
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > >
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > > The
> > rationale for
> > > > > > > >> separating
> > > > > > > >> > > scan
> > > > > > > >> > > > > > >> metrics
> > > > > > > >> > > > > > >> > > and
> > > > > > > >> > > > > > >> > > > > > > > metastore
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> persistence
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> is
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > that
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > > "cascading
> > > > deletes"
> > > > > > > >> between
> > > > > > > >> > > them
> > > > > > > >> > > > > are
> > > > > > > >> > > > > > >> > hardly
> > > > > > > >> > > > > > >> > > > > ever
> > > > > > > >> > > > > > >> > > > > > > > > > required.
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> Furthermore,
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > the
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > > data and
> > query
> > > > > > patterns
> > > > > > > >> are
> > > > > > > >> > > very
> > > > > > > >> > > > > > >> > different
> > > > > > > >> > > > > > >> > > so
> > > > > > > >> > > > > > >> > > > > > > > different
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> technologies
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > might
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > > be
> > beneficial in
> > > > > each
> > > > > > > >> case.
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > >
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > > [1]
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> >
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>>
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >>
> > > > > > > >> > > > > > >> > > > > > > > > > > > >>
> > > > > > > >> > > > > > >> > > > > > > > > > > >
> > > > > > > >> > > > > > >> > > > > > > > > > >
> > > > > > > >> > > > > > >> > > > > > > > > >
> > > > > > > >> > > > > > >> > > > > > > > >
> > > > > > > >> > > > > > >> > > > > > > >
> > > > > > > >> > > > > > >> > > > > > >
> > > > > > > >> > > > > > >> > > > > >
> > > > > > > >> > > > > > >> > > > >
> > > > > > > >> > > > > > >> > > >
> > > > > > > >> > > > > > >> > >
> > > > > > > >> > > > > > >> >
> > > > > > > >> > > > > > >>
> > > > > > > >> > > > > >
> > > > > > > >> > > > >
> > > > > > > >> > >
> > > > > > > >> >
> > > > > > > >>
> > > > > >
> > > > >
> > > >
> >
> https://urldefense.com/v3/__https://lists.apache.org/thread/yto2wp982t43h1mqjwnslswhws5z47cy__;!!Iz9xO38YGHZK!5EuyFFkk3vhRWVIRvQAWBSQfpJkTMA9HxugzDwXmN0LPPqhEFxYkFRGVhtb8AqUwXtDh2OplcMnbMDHKxYDakNU$
> > > > > > > >> > >> > > >> > > > > > > > > > > > >> >> >>> > >
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > > Cheers,
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > > Dmitri.
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > >
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > > On Fri, Feb
> > 27,
> > > > > 2026
> > > > > > at
> > > > > > > >> > > 6:19 PM
> > > > > > > >> > > > > > Anand
> > > > > > > >> > > > > > >> > Kumar
> > > > > > > >> > > > > > >> > > > > > > Sankaran
> > > > > > > >> > > > > > >> > > > > > > > > via
> > > > > > > >> > > > > > >> > > > > > > > > > > dev
> > > > > > > >> > > > > > >> > > > > > > > > > > > <
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > >
> > > > > > dev@polaris.apache.org>
> > > > > > > >> > > wrote:
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > >
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > > > Thanks
> all.
> > > > This
> > > > > > PR is
> > > > > > > >> > > merged
> > > > > > > >> > > > > now.
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > > >
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > > > Here are
> > the
> > > > > > follow-up
> > > > > > > >> > > features
> > > > > > > >> > > > > /
> > > > > > > >> > > > > > >> work
> > > > > > > >> > > > > > >> > > > > needed.
> > > > > > > >> > > > > > >> > > > > > > > These
> > > > > > > >> > > > > > >> > > > > > > > > > > were
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> all
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> part of
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > > the
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > > > merged PR
> > at
> > > > some
> > > > > > > >> point in
> > > > > > > >> > > time
> > > > > > > >> > > > > > and
> > > > > > > >> > > > > > >> > were
> > > > > > > >> > > > > > >> > > > > > removed
> > > > > > > >> > > > > > >> > > > > > > to
> > > > > > > >> > > > > > >> > > > > > > > > > > reduce
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> scope.
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > > >
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > > > Please
> let
> > me
> > > > > know
> > > > > > what
> > > > > > > >> > you
> > > > > > > >> > > > > think.
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > > >
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > > >
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > > >   1.  A
> > REST
> > > > API
> > > > > to
> > > > > > > >> > paginate
> > > > > > > >> > > > > > through
> > > > > > > >> > > > > > >> > > table
> > > > > > > >> > > > > > >> > > > > > > metrics.
> > > > > > > >> > > > > > >> > > > > > > > > > This
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> will be
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > non-IRC
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > > > standard
> > > > > addition.
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > > >   2.
> > Utilities
> > > > > for
> > > > > > > >> > > managing old
> > > > > > > >> > > > > > >> > records,
> > > > > > > >> > > > > > >> > > > > > should
> > > > > > > >> > > > > > >> > > > > > > be
> > > > > > > >> > > > > > >> > > > > > > > > > > common
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> with
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> events.
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > > > There was
> > some
> > > > > > > >> discussion
> > > > > > > >> > > that
> > > > > > > >> > > > > it
> > > > > > > >> > > > > > >> > belongs
> > > > > > > >> > > > > > >> > > > to
> > > > > > > >> > > > > > >> > > > > > the
> > > > > > > >> > > > > > >> > > > > > > > CLI.
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > > >   3.
> > Separate
> > > > > > > >> datasource
> > > > > > > >> > > > > > (metrics,
> > > > > > > >> > > > > > >> > > events,
> > > > > > > >> > > > > > >> > > > > > even
> > > > > > > >> > > > > > >> > > > > > > > > other
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> tables?).
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > > >
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > > >
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > > > Anything
> > else?
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > > >
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > > > -
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > > > Anand
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > > >
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > > >
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > >
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> >
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> >
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>>
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >>
> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >
> > > > > > > >> > > > > > >> > > > > > > > > > > > >>
> > > > > > > >> > > > > > >> > > > > > > > > > > > >
> > > > > > > >> > > > > > >> > > > > > > > > > > >
> > > > > > > >> > > > > > >> > > > > > > > > > >
> > > > > > > >> > > > > > >> > > > > > > > > >
> > > > > > > >> > > > > > >> > > > > > > > >
> > > > > > > >> > > > > > >> > > > > > > >
> > > > > > > >> > > > > > >> > > > > > >
> > > > > > > >> > > > > > >> > > > > >
> > > > > > > >> > > > > > >> > > > >
> > > > > > > >> > > > > > >> > > >
> > > > > > > >> > > > > > >> > >
> > > > > > > >> > > > > > >> >
> > > > > > > >> > > > > > >>
> > > > > > > >> > > > > > >
> > > > > > > >> > > > > >
> > > > > > > >> > > > >
> > > > > > > >> > >
> > > > > > > >> >
> > > > > > > >> >
> > > > > > > >>
> > > > > > > >
> > > > > >
> > > > >
> > > >
> >
>

Re: Proposal for REST endpoints for table metrics and events

[Dmitri Bourlatchkov <di...@apache.org>]

Hi All,

Metrics grants have been merged as a separate PR.

Now [4115] is ready for one more review round. IMHO, it is ready to merge.

[4115] https://github.com/apache/polaris/pull/4115

Cheers,
Dmitri.

On Fri, Oct 2, 2026 at 1:01 AM Jean-Baptiste Onofré <jb...@nanthrax.net> wrote:

> Hi all,
>
> I agree with Yufei: I would keep TABLE_READ_METRICS as a separate,
> explicit grant, not implied by TABLE_FULL_METADATA.
>
> I believe metrics reports are closer to an activity log than to table
> metadata. I also think it's consistent with what we have today, as
> reporting scan and commit metrics is already with TABLE_READ_DATA and
> TABLE_WRITE_DATA, not by "metadata" privilege.
>
> For the troubleshooting use case, the cost is just one additional
> grant, I think it's acceptable :) I would still let
> CATALOG_MANAGE_CONTENT implies it, as that privilege already implies
> TABLE_READ_DATA.
>
> Concretely for PR #5669, I propose:
> - to remove the "Implied by TABLE_FULL_METADATA" statement from the
> javadoc and CHANGELOG
> - to add SUPER_PRIVILEGES entry in the PR, so the grant change is
> self-contained
> - to drop ICEBERG_TABLE restriction.
> - to update access-control.md documenting TABLE_READ_METRICS, etc
>
> I will comment directly in the PR.
>
> Regards
> JB
>
> On Fri, Oct 2, 2026 at 3:40 AM Yufei Gu <fl...@gmail.com> wrote:
> >
> > Thanks Dmitri for the reminder.
> >
> > PR #5659 <https://github.com/apache/polaris/pull/5659> adds
> > TABLE_READ_METRICS and says it is implied by TABLE_FULL_METADATA. Should
> > that be the default?
> >
> > I’m leaning toward keeping it separate. Metrics describe workloads (Spark
> > ETL, Flink streaming, etc) using a table and can include query filters,
> so
> > someone managing table metadata may not need access to other users’ query
> > activity.
> >
> > Including it could make troubleshooting easier for table admins, though.
> > Are there use cases that favor automatic access over an explicit grant?
> > Yufei
> >
> >
> > On Wed, Sep 30, 2026 at 1:57 PM Dmitri Bourlatchkov <di...@apache.org>
> > wrote:
> >
> > > Hi Yufei,
> > >
> > > Here's the separate PR for Metrics-related grants:
> > > https://github.com/apache/polaris/pull/5659
> > >
> > > Please review it to unblock this contribution.
> > >
> > > Thanks,
> > > Dmitri.
> > >
> > > On Wed, Sep 30, 2026 at 1:05 PM Yufei Gu <fl...@gmail.com> wrote:
> > >
> > > > From the previous email, we agreed to split the REST change from the
> > > grant
> > > > change. I think the PR is ready to go if we only keep the grant
> changes.
> > > >
> > > > Yufei
> > > >
> > > >
> > > > On Tue, Sep 29, 2026 at 9:26 PM Jean-Baptiste Onofré <
> jb@nanthrax.net>
> > > > wrote:
> > > >
> > > > > Hi guys,
> > > > >
> > > > > I agree with Dmitri: +1 to merge PR #4115 on Oct 1 (tomorrow).
> > > > >
> > > > > From my side, all outstanding feedback has been addressed,
> including
> > > > > the final documentation points raised by EJ regarding the empty-200
> > > > > NoOp behavior and reconstructed payload description. The branch is
> > > > > clean, up to date with main, and CI is green.
> > > > >
> > > > > Once this lands, we can follow up with PR #4756 for the JDBC query
> SPI
> > > > > implementation.
> > > > >
> > > > > Regards
> > > > > JB
> > > > >
> > > > > On Wed, Sep 30, 2026 at 1:53 AM Dmitri Bourlatchkov <
> dimas@apache.org>
> > > > > wrote:
> > > > > >
> > > > > > Hi All,
> > > > > >
> > > > > > Any remaining blockers on [4115]?
> > > > > >
> > > > > > Given that the review was very long, please restate any remaining
> > > > > concerns
> > > > > > (here or in GH).
> > > > > >
> > > > > > Otherwise, I propose merging [4115] on Oct 1.
> > > > > >
> > > > > > [4115] https://github.com/apache/polaris/pull/4115
> > > > > >
> > > > > > Thanks,
> > > > > > Dmitri.
> > > > > >
> > > > > > On Wed, Sep 23, 2026 at 7:53 PM Dmitri Bourlatchkov <
> > > dimas@apache.org>
> > > > > > wrote:
> > > > > >
> > > > > > > Hi EJ,
> > > > > > >
> > > > > > > Thanks for the meeting recap!
> > > > > > >
> > > > > > > All:
> > > > > > >
> > > > > > > Worth noting: PR [4115] also contains Management API changes
> that
> > > > were
> > > > > not
> > > > > > > anticipated initially and did not get an API change review
> until
> > > now.
> > > > > These
> > > > > > > are small changes to add new grant types for metrics RBAC..
> > > > > > >
> > > > > > > I've reviewed and approved PR [4115] in GH.
> > > > > > >
> > > > > > > [4115] https://github.com/apache/polaris/pull/4115
> > > > > > >
> > > > > > > Cheers,
> > > > > > > Dmitri.
> > > > > > >
> > > > > > > On Wed, Aug 12, 2026 at 5:21 PM EJ Wang <
> > > > > ej.wang.dev.interest@gmail.com>
> > > > > > > wrote:
> > > > > > >
> > > > > > >> Hi folks,
> > > > > > >>
> > > > > > >> Notes from today's metrics architecture sync are in the usual
> doc:
> > > > > > >>
> > > > > > >>
> > > > >
> > > >
> > >
> https://docs.google.com/document/d/100h7c4damrUzVuquYbBHM0EvA4LSWuW2IT2dN_7nYVA/edit?pli=1&tab=t.18iv1m7p6i6b
> > > > > > >>
> > > > > > >> Highlights:
> > > > > > >>
> > > > > > >> Module placement. core/ now holds the entity data model only.
> A
> > > new
> > > > > spi/
> > > > > > >> takes every other shared contract: feature SPIs, feature
> non-SPI
> > > > > > >> contracts,
> > > > > > >> substrate SPIs, substrate non-SPI contracts, and
> contract-required
> > > > > types.
> > > > > > >> That settles the question PR #5204 reopened, by placing into
> a new
> > > > SPI
> > > > > > >> module. Note that it changes slightly towards the referenced
> WIP
> > > RFC
> > > > > [1]
> > > > > > >> recommendations: what proposed to place in core/, now should
> go to
> > > > > spi/
> > > > > > >> except for the entity data model.
> > > > > > >>
> > > > > > >> Metrics PRs. #5068 merged. #5204 needs reshaping onto the
> split.
> > > > > #4115 and
> > > > > > >> #4756 stay open with action on the author to follow up on
> review
> > > > > comments.
> > > > > > >>
> > > > > > >> [1]
> > > > > > >>
> > > > > > >>
> > > > >
> > > >
> > >
> https://docs.google.com/document/d/1mj3vGpix9zTDTFFJKtWT5ZxI3PTGI1YNb5xFqyWrOkc/edit?tab=t.0
> > > > > > >>
> > > > > > >> On Wed, Jul 15, 2026 at 11:01 AM Anand Kumar Sankaran via dev
> <
> > > > > > >> dev@polaris.apache.org> wrote:
> > > > > > >>
> > > > > > >> > Thanks EJ.  My plan is to introduce PR0 in a new branch /
> PR and
> > > > > > >> repurpose
> > > > > > >> > #4115 and #4756 for PR1 and PR2 to maintain all the valuable
> > > > > discussions
> > > > > > >> > that happened there.
> > > > > > >> >
> > > > > > >> > From: EJ Wang <ej...@gmail.com>
> > > > > > >> > Date: Wednesday, July 15, 2026 at 10:41 AM
> > > > > > >> > To: dev@polaris.apache.org <de...@polaris.apache.org>
> > > > > > >> > Subject: Re: Proposal for REST endpoints for table metrics
> and
> > > > > events
> > > > > > >> >
> > > > > > >> > Hi folks, Thanks again for the discussion today. I updated
> the
> > > > sync
> > > > > doc
> > > > > > >> > with notes from today's metrics sync: https: //urldefense.
> > > > > > >> com/v3/__https:
> > > > > > >> > //docs. google.
> > > > > > >> >
> > > > > > >>
> > > > >
> > > >
> > >
> com/document/d/100h7c4damrUzVuquYbBHM0EvA4LSWuW2IT2dN_7nYVA/edit?pli=1&tab=t.
> > > > > > >> >
> > > > > > >>
> > > > >
> > > >
> > >
> 2ahd7wze5f__;!!Iz9xO38YGHZK!4rnlVIqBQQQ6xpGe1PMV58edOS_cGrKoqdfhw4qSp_JQCavvEjaxXk0OwGPnxyEcO60AzKQK_XDtp7H3DdkFXcrXJwAz1g0$
> > > > > > >> >
> > > > > > >> >
> > > > > > >> > Hi folks,
> > > > > > >> >
> > > > > > >> > Thanks again for the discussion today. I updated the sync
> doc
> > > with
> > > > > notes
> > > > > > >> > from today's metrics sync:
> > > > > > >> >
> > > > > > >> >
> > > > > > >>
> > > > >
> > > >
> > >
> https://urldefense.com/v3/__https://docs.google.com/document/d/100h7c4damrUzVuquYbBHM0EvA4LSWuW2IT2dN_7nYVA/edit?pli=1&tab=t.2ahd7wze5f__;!!Iz9xO38YGHZK!4rnlVIqBQQQ6xpGe1PMV58edOS_cGrKoqdfhw4qSp_JQCavvEjaxXk0OwGPnxyEcO60AzKQK_XDtp7H3DdkFXcrXJwAz1g0$
> > > > > > >> >
> > > > > > >> > See highlights below:
> > > > > > >> >
> > > > > > >> > In previous meetings, EJ proposed a two prs framing to push
> > > > forward
> > > > > the
> > > > > > >> > metrics query work in polaris. Dmitri pointed out a subtle
> point
> > > > > that
> > > > > > >> pr2
> > > > > > >> > is currently also trying to do the persistence related code
> > > > > placement
> > > > > > >> > refactor. For the sake of straightforward review process,
> EJ,
> > > > > Dmitri and
> > > > > > >> > Anand (the author) agreed to separate the persistence
> > > refactoring
> > > > > into
> > > > > > >> pr0,
> > > > > > >> > and reframing the pr scopes as follow:
> > > > > > >> >
> > > > > > >> >    1.
> > > > > > >> >
> > > > > > >> >    Pr0: refactor irc metrics spi
> > > > > > >> >    1.
> > > > > > >> >
> > > > > > >> >       SPI should stay in core/ with metric type enveloped
> > > > > > >> >       2.
> > > > > > >> >
> > > > > > >> >       default spi impl, no-op, in extensions/ - default
> > > selection
> > > > at
> > > > > > >> >       runtime/
> > > > > > >> >       3.
> > > > > > >> >
> > > > > > >> >       move metrics persistence and correspondings to
> extensions/
> > > > as
> > > > > an
> > > > > > >> >       alternative SPI impl
> > > > > > >> >       2.
> > > > > > >> >
> > > > > > >> >    Pr1: rest api for query
> > > > > > >> >
> > > > > > >> >
> > > > > > >>
> > > > >
> > > >
> > >
> https://urldefense.com/v3/__https://github.com/apache/polaris/pull/4115/changes__;!!Iz9xO38YGHZK!4rnlVIqBQQQ6xpGe1PMV58edOS_cGrKoqdfhw4qSp_JQCavvEjaxXk0OwGPnxyEcO60AzKQK_XDtp7H3DdkFXcrXFCbEbRk$
> > > > > > >> >    1.
> > > > > > >> >
> > > > > > >> >       rest api spec in extensions/ as optional spec
> > > > > > >> >       2.
> > > > > > >> >
> > > > > > >> >       SPI in extensions/ as optional spec spi
> > > > > > >> >       3.
> > > > > > >> >
> > > > > > >> >       default spi impl in extensions/ no-op
> > > > > > >> >       4.
> > > > > > >> >
> > > > > > >> >       handler related in runtime/ - thin adaptor from http
> shape
> > > > to
> > > > > spi
> > > > > > >> >       shape
> > > > > > >> >       3.
> > > > > > >> >
> > > > > > >> >    Pr2: jdbc impl for query
> > > > > > >> >
> > > > > > >>
> > > > >
> > > >
> > >
> https://urldefense.com/v3/__https://github.com/apache/polaris/pull/4756__;!!Iz9xO38YGHZK!4rnlVIqBQQQ6xpGe1PMV58edOS_cGrKoqdfhw4qSp_JQCavvEjaxXk0OwGPnxyEcO60AzKQK_XDtp7H3DdkFXcrXRVMBzqE$
> > > > > > >> >    1.
> > > > > > >> >
> > > > > > >> >       alternative spi impl in extensions/ -> query with jdbc
> > > > > > >> >
> > > > > > >> >
> > > > > > >> > Thanks,
> > > > > > >> > -ej
> > > > > > >> >
> > > > > > >> > On Wed, Jul 1, 2026 at 8:01 AM Alexandre Dutra <
> > > adutra@apache.org
> > > > >
> > > > > > >> wrote:
> > > > > > >> >
> > > > > > >> > > Hi all,
> > > > > > >> > >
> > > > > > >> > > > While the current implementation stores metrics in the
> > > > > metastore,
> > > > > > >> other
> > > > > > >> > > deployments
> > > > > > >> > > may choose a KV store, a dedicated metrics warehouse, or
> > > another
> > > > > > >> backend
> > > > > > >> > > entirely. In those cases, exposing a Polaris REST API
> over a
> > > > > specific
> > > > > > >> > > schema feels like coupling the API to an implementation
> > > detail.
> > > > > > >> > >
> > > > > > >> > > I wouldn't say to an implementation detail, but rather to
> a
> > > > > particular
> > > > > > >> > > setup / configuration. I think that's fine. If you don't
> store
> > > > > > >> > > metrics, you don't read metrics. Similar situations will
> > > > certainly
> > > > > > >> > > happen in the future, e.g. if we ever introduce an API to
> read
> > > > > events,
> > > > > > >> > > that would only work if events are being persisted in the
> > > first
> > > > > place.
> > > > > > >> > > Again I think that's a fair trade-off.
> > > > > > >> > >
> > > > > > >> > > Thanks,
> > > > > > >> > > Alex
> > > > > > >> > >
> > > > > > >> > > On Wed, Jul 1, 2026 at 7:23 AM Dmitri Bourlatchkov <
> > > > > dimas@apache.org>
> > > > > > >> > > wrote:
> > > > > > >> > > >
> > > > > > >> > > > Hi Yufei,
> > > > > > >> > > >
> > > > > > >> > > > From my POV, it is fine for Polaris to have multiple
> REST
> > > API
> > > > > > >> surfaces
> > > > > > >> > > with
> > > > > > >> > > > different features and different degrees of "readiness".
> > > Each
> > > > > API
> > > > > > >> can
> > > > > > >> > > > evolve fairly independently. With proper modularization
> > > (which
> > > > > Anand
> > > > > > >> > > did),
> > > > > > >> > > > changes to the main Polaris code and these extra modules
> > > > should
> > > > > > >> > normally
> > > > > > >> > > be
> > > > > > >> > > > non-intrusive to each other.
> > > > > > >> > > >
> > > > > > >> > > > I assume Anand is going to be the first user of this
> API.
> > > > > However,
> > > > > > >> if
> > > > > > >> > we
> > > > > > >> > > do
> > > > > > >> > > > not merge it we'll never know whether other users are
> > > > > interested. I
> > > > > > >> do
> > > > > > >> > > not
> > > > > > >> > > > think many end users read the dev ML, but they certainly
> > > read
> > > > > the
> > > > > > >> docs,
> > > > > > >> > > > code and release notes.
> > > > > > >> > > >
> > > > > > >> > > > There is no risk for Polaris as a project in exposing
> this
> > > > > > >> experimental
> > > > > > >> > > > API. If usage dwindles over time, we can remove it. If
> it
> > > > gains
> > > > > a
> > > > > > >> > > > considerable user / contributor base, we will promote
> it to
> > > a
> > > > > > >> > > "first-class"
> > > > > > >> > > > (non-beta) API.
> > > > > > >> > > >
> > > > > > >> > > > Cheers,
> > > > > > >> > > > Dmitri.
> > > > > > >> > > >
> > > > > > >> > > > On Tue, Jun 30, 2026 at 7:53 PM Yufei Gu <
> > > > flyrain000@gmail.com>
> > > > > > >> wrote:
> > > > > > >> > > >
> > > > > > >> > > > > Hi all,
> > > > > > >> > > > >
> > > > > > >> > > > > Thanks for driving this work forward. Building this
> as a
> > > POC
> > > > > makes
> > > > > > >> > > perfect
> > > > > > >> > > > > sense if the goal is to validate the idea and gather
> > > > feedback.
> > > > > > >> Should
> > > > > > >> > > we
> > > > > > >> > > > > merge a POC into the main branch before agreeing on
> the
> > > > > overall
> > > > > > >> > > direction?
> > > > > > >> > > > > To me, that feels like a separate discussion from
> whether
> > > > the
> > > > > > >> > > > > implementation itself looks reasonable.
> > > > > > >> > > > >
> > > > > > >> > > > > One concern that I don't think has been fully
> addressed is
> > > > > > >> whether we
> > > > > > >> > > need
> > > > > > >> > > > > to expose metrics consumption through a Polaris REST
> > > > endpoint
> > > > > at
> > > > > > >> all.
> > > > > > >> > > > >
> > > > > > >> > > > > My concern is less about the API itself and more
> about the
> > > > > > >> boundary
> > > > > > >> > > Polaris
> > > > > > >> > > > > should own. The proposed REST API is tied to a
> particular
> > > > > > >> persistence
> > > > > > >> > > > > schema, but that schema is just one possible
> > > implementation.
> > > > > While
> > > > > > >> > the
> > > > > > >> > > > > current implementation stores metrics in the
> metastore,
> > > > other
> > > > > > >> > > deployments
> > > > > > >> > > > > may choose a KV store, a dedicated metrics warehouse,
> or
> > > > > another
> > > > > > >> > > backend
> > > > > > >> > > > > entirely. In those cases, exposing a Polaris REST API
> > > over a
> > > > > > >> specific
> > > > > > >> > > > > schema feels like coupling the API to an
> implementation
> > > > > detail.
> > > > > > >> > > > >
> > > > > > >> > > > > Polaris isn't intended to become a full metrics
> system.
> > > I'd
> > > > > > >> expect it
> > > > > > >> > > to
> > > > > > >> > > > > focus on producing metrics and let deployments choose
> the
> > > > most
> > > > > > >> > > appropriate
> > > > > > >> > > > > storage and consumption model for their environment.
> > > > > > >> > > > >
> > > > > > >> > > > > I'm not opposed to exploring this direction, but we
> should
> > > > > first
> > > > > > >> > align
> > > > > > >> > > on
> > > > > > >> > > > > whether exposing a REST API through Polaris is the
> right
> > > > long
> > > > > term
> > > > > > >> > > approach
> > > > > > >> > > > > before merging it into the main branch.
> > > > > > >> > > > >
> > > > > > >> > > > > Thanks,
> > > > > > >> > > > > Yufei
> > > > > > >> > > > >
> > > > > > >> > > > > Yufei
> > > > > > >> > > > >
> > > > > > >> > > > >
> > > > > > >> > > > > On Mon, Jun 29, 2026 at 9:27 AM Dmitri Bourlatchkov <
> > > > > > >> > dimas@apache.org>
> > > > > > >> > > > > wrote:
> > > > > > >> > > > >
> > > > > > >> > > > > > Hi All,
> > > > > > >> > > > > >
> > > > > > >> > > > > > It looks like Anand addressed all action items in
> > > [4115].
> > > > > > >> > > > > >
> > > > > > >> > > > > > Please review the latest state of the code. The PR
> LGTM
> > > > and
> > > > > I
> > > > > > >> > > approved in
> > > > > > >> > > > > > GH from my side.
> > > > > > >> > > > > >
> > > > > > >> > > > > > [4115]
> > > > > > >> >
> > > > > > >>
> > > > >
> > > >
> > >
> https://urldefense.com/v3/__https://github.com/apache/polaris/pull/4115__;!!Iz9xO38YGHZK!4rnlVIqBQQQ6xpGe1PMV58edOS_cGrKoqdfhw4qSp_JQCavvEjaxXk0OwGPnxyEcO60AzKQK_XDtp7H3DdkFXcrXtFRscKs$
> > > > > > >> > > > > >
> > > > > > >> > > > > > Thanks,
> > > > > > >> > > > > > Dmitri.
> > > > > > >> > > > > >
> > > > > > >> > > > > > On Thu, Jun 25, 2026 at 1:56 PM Dmitri Bourlatchkov
> <
> > > > > > >> > > dimas@apache.org>
> > > > > > >> > > > > > wrote:
> > > > > > >> > > > > >
> > > > > > >> > > > > > > Hi All,
> > > > > > >> > > > > > >
> > > > > > >> > > > > > > Recap of the discussion in today's Community Sync:
> > > > > > >> > > > > > >
> > > > > > >> > > > > > > * The modular approach for Metrics SPI(s) and
> > > > > implementation
> > > > > > >> > seems
> > > > > > >> > > to
> > > > > > >> > > > > > have
> > > > > > >> > > > > > > community consensus (please comment if you still
> have
> > > > > > >> concerns)
> > > > > > >> > > > > > >
> > > > > > >> > > > > > > * What modules / feature flags are enabled in
> Apache
> > > > > Polaris
> > > > > > >> > > images by
> > > > > > >> > > > > > > default will be discussed later (when the feature
> is
> > > > > close to
> > > > > > >> > > > > > completion).
> > > > > > >> > > > > > >
> > > > > > >> > > > > > > In any case, users will be able to turn the full
> > > feature
> > > > > > >> on/off
> > > > > > >> > at
> > > > > > >> > > > > their
> > > > > > >> > > > > > > discretion, regardless of the default flag values.
> > > > > > >> > > > > > >
> > > > > > >> > > > > > > * It is generally preferable to have smaller PRs
> and
> > > > > introduce
> > > > > > >> > new
> > > > > > >> > > > > > > features gradually to simplify reviews.
> > > > > > >> > > > > > >
> > > > > > >> > > > > > > What this means for the Metrics API, IMHO is:
> > > > > > >> > > > > > >
> > > > > > >> > > > > > > a) The scope of PR 4115 is probably ok as it
> stands
> > > > right
> > > > > > >> now. If
> > > > > > >> > > > > someone
> > > > > > >> > > > > > > wants to propose splitting it further, please
> comment.
> > > > > > >> > > > > > >
> > > > > > >> > > > > > > b) After 4115 is merged, Metrics Persistence PRs
> will
> > > > > follow,
> > > > > > >> > with
> > > > > > >> > > > > > smaller
> > > > > > >> > > > > > > scope per PR as appropriate for the nature of the
> code
> > > > > > >> changes.
> > > > > > >> > > > > > >
> > > > > > >> > > > > > > * The major concern was about exposing the new
> Metrics
> > > > > REST
> > > > > > >> API
> > > > > > >> > to
> > > > > > >> > > end
> > > > > > >> > > > > > > users. The nature of this concern was about
> avoiding
> > > the
> > > > > > >> (false)
> > > > > > >> > > > > > impression
> > > > > > >> > > > > > > that the new API is a firm and adopted Polaris
> > > > > specification.
> > > > > > >> > > > > > >
> > > > > > >> > > > > > > I think this concern is reasonable.
> > > > > > >> > > > > > >
> > > > > > >> > > > > > > I propose that Anand update the documentation
> sections
> > > > of
> > > > > PR
> > > > > > >> 4115
> > > > > > >> > > to
> > > > > > >> > > > > > > emphasise that the API is currently an
> experimental /
> > > > POC
> > > > > API
> > > > > > >> at
> > > > > > >> > > this
> > > > > > >> > > > > > time
> > > > > > >> > > > > > > and is subject to change (including breaking
> > > changes). I
> > > > > > >> believe
> > > > > > >> > it
> > > > > > >> > > > > > already
> > > > > > >> > > > > > > has the "beta" label. So this request is mostly
> for
> > > doc
> > > > > > >> changes
> > > > > > >> > to
> > > > > > >> > > > > > clarify
> > > > > > >> > > > > > > user expectations, since the "beta" label alone
> may
> > > not
> > > > be
> > > > > > >> > > interpreted
> > > > > > >> > > > > > the
> > > > > > >> > > > > > > same way by all users.
> > > > > > >> > > > > > >
> > > > > > >> > > > > > > I hope this update will resolve all remaining
> concerns
> > > > > with
> > > > > > >> 4115
> > > > > > >> > > and
> > > > > > >> > > > > > allow
> > > > > > >> > > > > > > for it to be merged so that the community will be
> able
> > > > to
> > > > > > >> proceed
> > > > > > >> > > to
> > > > > > >> > > > > > > follow-up PRs (as noted above).
> > > > > > >> > > > > > >
> > > > > > >> > > > > > > Please add/correct me if I missed anything
> relevant.
> > > > > > >> > > > > > >
> > > > > > >> > > > > > > Thanks,
> > > > > > >> > > > > > > Dmitri.
> > > > > > >> > > > > > >
> > > > > > >> > > > > > > On Wed, Jun 24, 2026 at 8:11 PM Yufei Gu <
> > > > > > >> flyrain000@gmail.com>
> > > > > > >> > > wrote:
> > > > > > >> > > > > > >
> > > > > > >> > > > > > >> I do not think we have agreement on the query
> REST
> > > APIs
> > > > > yet.
> > > > > > >> > > > > > >>
> > > > > > >> > > > > > >> In particular, the current query API shape seems
> > > highly
> > > > > > >> coupled
> > > > > > >> > > to the
> > > > > > >> > > > > > >> *example[1]* metrics in the Iceberg REST spec. I
> do
> > > not
> > > > > think
> > > > > > >> > > that is
> > > > > > >> > > > > a
> > > > > > >> > > > > > >> good way to design a REST API contract. For
> example,
> > > > > fields
> > > > > > >> like
> > > > > > >> > > > > > >> resultDataFiles, resultDeleteFiles,
> > > totalFileSizeBytes,
> > > > > > >> > > > > > >> totalDataManifests,
> > > > > > >> > > > > > >> totalDeleteManifests, and scannedDataManifests
> are
> > > very
> > > > > > >> specific
> > > > > > >> > > to
> > > > > > >> > > > > one
> > > > > > >> > > > > > >> style of scan metric. That makes the API look
> more
> > > > like a
> > > > > > >> direct
> > > > > > >> > > > > > >> projection
> > > > > > >> > > > > > >> of the current example payload than a general
> query
> > > > > model.
> > > > > > >> > > > > > >>
> > > > > > >> > > > > > >> My preference is that Polaris should not define
> or
> > > > > strongly
> > > > > > >> > opine
> > > > > > >> > > on
> > > > > > >> > > > > the
> > > > > > >> > > > > > >> Iceberg scan or commit metrics query REST API at
> this
> > > > > point.
> > > > > > >> > > Polaris
> > > > > > >> > > > > can
> > > > > > >> > > > > > >> resolve and authorize the table, accept the
> metrics
> > > > > report,
> > > > > > >> and
> > > > > > >> > > > > delegate
> > > > > > >> > > > > > >> it
> > > > > > >> > > > > > >> to the configured reporting path. Durable
> storage,
> > > > > filtering,
> > > > > > >> > > > > > dashboards,
> > > > > > >> > > > > > >> and query APIs should remain downstream impl.
> choices
> > > > > unless
> > > > > > >> and
> > > > > > >> > > until
> > > > > > >> > > > > > the
> > > > > > >> > > > > > >> Iceberg community standardizes a query contract,
> > > which
> > > > I
> > > > > > >> doubt
> > > > > > >> > > will
> > > > > > >> > > > > > >> happen.
> > > > > > >> > > > > > >>
> > > > > > >> > > > > > >> So I do not think the intake path has to be
> blocked,
> > > > but
> > > > > I
> > > > > > >> would
> > > > > > >> > > > > prefer
> > > > > > >> > > > > > >> not
> > > > > > >> > > > > > >> to merge a Polaris specific query API that may be
> > > hard
> > > > to
> > > > > > >> evolve
> > > > > > >> > > > > later.
> > > > > > >> > > > > > >>
> > > > > > >> > > > > > >> 1.
> > > > > > >> > > > > > >>
> > > > > > >> > > > > > >>
> > > > > > >> > > > > >
> > > > > > >> > > > >
> > > > > > >> > >
> > > > > > >> >
> > > > > > >>
> > > > >
> > > >
> > >
> https://urldefense.com/v3/__https://github.com/apache/iceberg/blob/2a6c556c0c883c04c6d3fbd68e6aeda36b91e0aa/open-api/rest-catalog-open-api.yaml*L4152__;Iw!!Iz9xO38YGHZK!4rnlVIqBQQQ6xpGe1PMV58edOS_cGrKoqdfhw4qSp_JQCavvEjaxXk0OwGPnxyEcO60AzKQK_XDtp7H3DdkFXcrXCDsqYGo$
> > > > > > >> > > > > > >>
> > > > > > >> > > > > > >> Yufei
> > > > > > >> > > > > > >>
> > > > > > >> > > > > > >>
> > > > > > >> > > > > > >> On Tue, Jun 23, 2026 at 9:30 AM Dmitri
> Bourlatchkov <
> > > > > > >> > > dimas@apache.org
> > > > > > >> > > > > >
> > > > > > >> > > > > > >> wrote:
> > > > > > >> > > > > > >>
> > > > > > >> > > > > > >> > Hi All,
> > > > > > >> > > > > > >> >
> > > > > > >> > > > > > >> > So, what are the blockers for merging [4115],
> if
> > > any?
> > > > > > >> > > > > > >> >
> > > > > > >> > > > > > >> > [4115]
> > > > > > >> >
> > > > > > >>
> > > > >
> > > >
> > >
> https://urldefense.com/v3/__https://github.com/apache/polaris/pull/4115__;!!Iz9xO38YGHZK!4rnlVIqBQQQ6xpGe1PMV58edOS_cGrKoqdfhw4qSp_JQCavvEjaxXk0OwGPnxyEcO60AzKQK_XDtp7H3DdkFXcrXtFRscKs$
> > > > > > >> > > > > > >> >
> > > > > > >> > > > > > >> > Thanks,
> > > > > > >> > > > > > >> > Dmitri.
> > > > > > >> > > > > > >> >
> > > > > > >> > > > > > >> > On Wed, Jun 17, 2026 at 5:28 PM EJ Wang <
> > > > > > >> > > > > > ej.wang.dev.interest@gmail.com
> > > > > > >> > > > > > >> >
> > > > > > >> > > > > > >> > wrote:
> > > > > > >> > > > > > >> >
> > > > > > >> > > > > > >> > > Hi folks,
> > > > > > >> > > > > > >> > >
> > > > > > >> > > > > > >> > > Thanks again for the discussion today. I
> updated
> > > > the
> > > > > sync
> > > > > > >> > doc
> > > > > > >> > > with
> > > > > > >> > > > > > >> notes
> > > > > > >> > > > > > >> > > from today's metrics sync:
> > > > > > >> > > > > > >> > >
> > > > > > >> > > > > > >> > >
> > > > > > >> > > > > > >> > >
> > > > > > >> > > > > > >> >
> > > > > > >> > > > > > >>
> > > > > > >> > > > > >
> > > > > > >> > > > >
> > > > > > >> > >
> > > > > > >> >
> > > > > > >>
> > > > >
> > > >
> > >
> https://urldefense.com/v3/__https://docs.google.com/document/d/100h7c4damrUzVuquYbBHM0EvA4LSWuW2IT2dN_7nYVA/edit?tab=t.ezk0rgdx0c6m__;!!Iz9xO38YGHZK!4rnlVIqBQQQ6xpGe1PMV58edOS_cGrKoqdfhw4qSp_JQCavvEjaxXk0OwGPnxyEcO60AzKQK_XDtp7H3DdkFXcrX83IZkm0$
> > > > > > >> > > > > > >> > >
> > > > > > >> > > > > > >> > > A few highlights:
> > > > > > >> > > > > > >> > >
> > > > > > >> > > > > > >> > >    - No objection to keeping the default
> metrics
> > > > > behavior
> > > > > > >> > > small:
> > > > > > >> > > > > > >> > >    no-op/log-only is enough as the built-in
> > > > default.
> > > > > > >> > > > > > >> > >    - Durable storage, event forwarding,
> external
> > > > > queues,
> > > > > > >> > > > > dashboards,
> > > > > > >> > > > > > >> and
> > > > > > >> > > > > > >> > >    custom filtering should be implementation
> > > > choices
> > > > > > >> behind
> > > > > > >> > > the
> > > > > > >> > > > > > >> metrics
> > > > > > >> > > > > > >> > >    reporting path.
> > > > > > >> > > > > > >> > >    - For PR #4115, the REST layer should
> > > > > > >> > resolve/authz/accept
> > > > > > >> > > the
> > > > > > >> > > > > > >> report,
> > > > > > >> > > > > > >> > >    then delegate to the selected metrics
> > > > > implementation.
> > > > > > >> It
> > > > > > >> > > should
> > > > > > >> > > > > > not
> > > > > > >> > > > > > >> > own
> > > > > > >> > > > > > >> > >    durable storage semantics.
> > > > > > >> > > > > > >> > >    - The rough boundary we discussed is: spec
> > > owns
> > > > > the
> > > > > > >> wire
> > > > > > >> > > > > > contract,
> > > > > > >> > > > > > >> > >    runtime owns framework wiring,
> API/contract
> > > > > modules
> > > > > > >> own
> > > > > > >> > > > > > >> > provider-facing
> > > > > > >> > > > > > >> > >    contracts, and extensions own replaceable
> > > > > > >> > implementations.
> > > > > > >> > > > > > >> > >    - OpenLineage has a similar path-naming
> > > > question.
> > > > > We
> > > > > > >> > should
> > > > > > >> > > > > avoid
> > > > > > >> > > > > > >> > >    occupying a generic lineage path too
> early if
> > > we
> > > > > want
> > > > > > >> > room
> > > > > > >> > > for
> > > > > > >> > > > > > >> other
> > > > > > >> > > > > > >> > >    lineage systems later.
> > > > > > >> > > > > > >> > >
> > > > > > >> > > > > > >> > > I believe the boundary rules previewed in
> sync
> > > > > meeting
> > > > > > >> are
> > > > > > >> > > useful
> > > > > > >> > > > > > for
> > > > > > >> > > > > > >> > > reviewing PR #4115, but it is not
> > > > > > >> > > > > > >> > > a final module-layout proposal.
> > > > > > >> > > > > > >> > >
> > > > > > >> > > > > > >> > > For metrics specifically, I think the target
> > > > remains:
> > > > > > >> > > > > > >> > >    runtime
> > > > > > >> > > > > > >> > >      -> metrics reporting/emitting contract
> > > > > > >> > > > > > >> > >      -> default implementation
> > > > > > >> > > > > > >> > >           -> lower implementation details, if
> > > > needed
> > > > > > >> > > > > > >> > >
> > > > > > >> > > > > > >> > > That keeps the default simple, while allowing
> > > > durable
> > > > > > >> JDBC,
> > > > > > >> > > > > > >> event-backed,
> > > > > > >> > > > > > >> > > or external-queue implementations to be added
> > > > > separately.
> > > > > > >> > > > > > >> > >
> > > > > > >> > > > > > >> > > We also touched on the split between Iceberg
> > > > metrics
> > > > > > >> > emitting
> > > > > > >> > > and
> > > > > > >> > > > > > >> > > Polaris-owned metrics querying. The querying
> side
> > > > can
> > > > > > >> keep
> > > > > > >> > > > > evolving,
> > > > > > >> > > > > > >> > > especially for dashboard or generic-table use
> > > > cases,
> > > > > but
> > > > > > >> I
> > > > > > >> > do
> > > > > > >> > > not
> > > > > > >> > > > > > >> think
> > > > > > >> > > > > > >> > > that needs to block the intake path in this
> PR.
> > > > > > >> > > > > > >> > >
> > > > > > >> > > > > > >> > > Thanks,
> > > > > > >> > > > > > >> > > -ej
> > > > > > >> > > > > > >> > >
> > > > > > >> > > > > > >> > > On Tue, Jun 16, 2026 at 2:17 PM Yufei Gu <
> > > > > > >> > > flyrain000@gmail.com>
> > > > > > >> > > > > > >> wrote:
> > > > > > >> > > > > > >> > >
> > > > > > >> > > > > > >> > > > Thanks for chiming in, EJ. Agreed that the
> > > > default
> > > > > > >> battery
> > > > > > >> > > > > should
> > > > > > >> > > > > > >> stay
> > > > > > >> > > > > > >> > > > small. I'm leaning toward not including
> metrics
> > > > > > >> > persistence
> > > > > > >> > > in
> > > > > > >> > > > > the
> > > > > > >> > > > > > >> > > default
> > > > > > >> > > > > > >> > > > battery.
> > > > > > >> > > > > > >> > > >
> > > > > > >> > > > > > >> > > > Yufei
> > > > > > >> > > > > > >> > > >
> > > > > > >> > > > > > >> > > >
> > > > > > >> > > > > > >> > > > On Thu, Jun 11, 2026 at 8:30 PM EJ Wang <
> > > > > > >> > > > > > >> > ej.wang.dev.interest@gmail.com>
> > > > > > >> > > > > > >> > > > wrote:
> > > > > > >> > > > > > >> > > >
> > > > > > >> > > > > > >> > > > > Hi Yufei,
> > > > > > >> > > > > > >> > > > >
> > > > > > >> > > > > > >> > > > > Thanks for connecting this back to the
> REST
> > > > > endpoint
> > > > > > >> > > > > proposal. I
> > > > > > >> > > > > > >> > > replied
> > > > > > >> > > > > > >> > > > > with the fuller version on the
> > > event-forwarding
> > > > > > >> thread,
> > > > > > >> > > but
> > > > > > >> > > > > > >> wanted to
> > > > > > >> > > > > > >> > > add
> > > > > > >> > > > > > >> > > > > the shorter version here too.
> > > > > > >> > > > > > >> > > > >
> > > > > > >> > > > > > >> > > > > I agree that metrics reporting/emitting
> is
> > > the
> > > > > right
> > > > > > >> > > > > conceptual
> > > > > > >> > > > > > >> > > boundary,
> > > > > > >> > > > > > >> > > > > and I also think the event/listener path
> is a
> > > > > good
> > > > > > >> > > > > > implementation
> > > > > > >> > > > > > >> > > > direction
> > > > > > >> > > > > > >> > > > > to explore. The distinction I want to
> keep
> > > > clear
> > > > > is
> > > > > > >> > > default
> > > > > > >> > > > > > >> battery
> > > > > > >> > > > > > >> > vs
> > > > > > >> > > > > > >> > > > > extension implementation.
> > > > > > >> > > > > > >> > > > >
> > > > > > >> > > > > > >> > > > > For the REST/API side, I would keep the
> > > > semantics
> > > > > > >> > narrow:
> > > > > > >> > > > > > Polaris
> > > > > > >> > > > > > >> > > > resolves
> > > > > > >> > > > > > >> > > > > and authorizes the table, accepts the
> Iceberg
> > > > > > >> > scan/commit
> > > > > > >> > > > > > metrics
> > > > > > >> > > > > > >> > > report
> > > > > > >> > > > > > >> > > > > into the configured reporting/emitting
> path,
> > > > and
> > > > > > >> returns
> > > > > > >> > > 204.
> > > > > > >> > > > > > That
> > > > > > >> > > > > > >> > 204
> > > > > > >> > > > > > >> > > > > should mean Polaris accepted the report
> into
> > > > the
> > > > > > >> > ingestion
> > > > > > >> > > > > path;
> > > > > > >> > > > > > >> it
> > > > > > >> > > > > > >> > > > should
> > > > > > >> > > > > > >> > > > > not imply durable storage.
> > > > > > >> > > > > > >> > > > >
> > > > > > >> > > > > > >> > > > > Durable storage, event forwarding,
> external
> > > > > telemetry
> > > > > > >> > > routing,
> > > > > > >> > > > > > >> > > filtering,
> > > > > > >> > > > > > >> > > > > and retention should sit behind that
> > > > > > >> reporting/emitting
> > > > > > >> > > > > boundary
> > > > > > >> > > > > > >> as
> > > > > > >> > > > > > >> > > > > implementation-layer behavior. A durable
> JDBC
> > > > > path
> > > > > > >> can
> > > > > > >> > be
> > > > > > >> > > one
> > > > > > >> > > > > > >> named
> > > > > > >> > > > > > >> > > > > extension implementation. An
> event/listener
> > > > > > >> forwarding
> > > > > > >> > > path
> > > > > > >> > > > > can
> > > > > > >> > > > > > be
> > > > > > >> > > > > > >> > > > another
> > > > > > >> > > > > > >> > > > > named extension implementation.
> > > > > > >> > > > > > >> > > > >
> > > > > > >> > > > > > >> > > > > So I am +1 on the event-forwarding idea
> as a
> > > > > > >> non-default
> > > > > > >> > > > > > extension
> > > > > > >> > > > > > >> > > > > implementation. I just would not make it
> the
> > > > > default
> > > > > > >> > > battery
> > > > > > >> > > > > or
> > > > > > >> > > > > > >> the
> > > > > > >> > > > > > >> > > core
> > > > > > >> > > > > > >> > > > > API shape. The default battery can stay
> small
> > > > and
> > > > > > >> safe,
> > > > > > >> > > while
> > > > > > >> > > > > > >> > > deployments
> > > > > > >> > > > > > >> > > > > that need forwarding, dashboards, or
> durable
> > > > > storage
> > > > > > >> can
> > > > > > >> > > opt
> > > > > > >> > > > > > into
> > > > > > >> > > > > > >> the
> > > > > > >> > > > > > >> > > > > implementation that matches their
> operational
> > > > > model.
> > > > > > >> > > > > > >> > > > >
> > > > > > >> > > > > > >> > > > > Thanks,
> > > > > > >> > > > > > >> > > > > -ej
> > > > > > >> > > > > > >> > > > >
> > > > > > >> > > > > > >> > > > > On Wed, Jun 10, 2026 at 11:43 AM Yufei
> Gu <
> > > > > > >> > > > > flyrain000@gmail.com
> > > > > > >> > > > > > >
> > > > > > >> > > > > > >> > > wrote:
> > > > > > >> > > > > > >> > > > >
> > > > > > >> > > > > > >> > > > > > Thanks EJ. I agree that the
> > > > reporting/emitting
> > > > > > >> > boundary
> > > > > > >> > > > > feels
> > > > > > >> > > > > > >> like
> > > > > > >> > > > > > >> > > the
> > > > > > >> > > > > > >> > > > > > right SPI boundary, but I wonder if we
> can
> > > > > simplify
> > > > > > >> > this
> > > > > > >> > > > > even
> > > > > > >> > > > > > >> > > further.
> > > > > > >> > > > > > >> > > > > >
> > > > > > >> > > > > > >> > > > > > Iceberg scan and commit metrics look
> very
> > > > > similar
> > > > > > >> to
> > > > > > >> > > events
> > > > > > >> > > > > to
> > > > > > >> > > > > > >> me.
> > > > > > >> > > > > > >> > > They
> > > > > > >> > > > > > >> > > > > are
> > > > > > >> > > > > > >> > > > > > append only, asynchronously consumed,
> often
> > > > > > >> forwarded
> > > > > > >> > to
> > > > > > >> > > > > > >> external
> > > > > > >> > > > > > >> > > > > systems,
> > > > > > >> > > > > > >> > > > > > and have similar retention and cleanup
> > > > > > >> requirements.
> > > > > > >> > > More
> > > > > > >> > > > > > >> details
> > > > > > >> > > > > > >> > can
> > > > > > >> > > > > > >> > > > be
> > > > > > >> > > > > > >> > > > > > found in this thread [1]. In that case,
> > > > > Polaris may
> > > > > > >> > only
> > > > > > >> > > > > need
> > > > > > >> > > > > > to
> > > > > > >> > > > > > >> > > accept
> > > > > > >> > > > > > >> > > > > the
> > > > > > >> > > > > > >> > > > > > report and emit an event through the
> > > existing
> > > > > event
> > > > > > >> > > > > framework.
> > > > > > >> > > > > > >> > > > Filtering,
> > > > > > >> > > > > > >> > > > > > async delivery, custom sinks, and
> retention
> > > > > > >> mechanisms
> > > > > > >> > > could
> > > > > > >> > > > > > >> then
> > > > > > >> > > > > > >> > be
> > > > > > >> > > > > > >> > > > > shared
> > > > > > >> > > > > > >> > > > > > instead of introducing a separate
> metrics
> > > > > specific
> > > > > > >> > > extension
> > > > > > >> > > > > > >> layer
> > > > > > >> > > > > > >> > > and
> > > > > > >> > > > > > >> > > > > SPI.
> > > > > > >> > > > > > >> > > > > >
> > > > > > >> > > > > > >> > > > > > The main requirement I still see is
> > > querying
> > > > > > >> metrics
> > > > > > >> > > through
> > > > > > >> > > > > > >> > Polaris.
> > > > > > >> > > > > > >> > > > If
> > > > > > >> > > > > > >> > > > > we
> > > > > > >> > > > > > >> > > > > > want that battery included experience,
> we
> > > > could
> > > > > > >> work
> > > > > > >> > on
> > > > > > >> > > a
> > > > > > >> > > > > > >> > persistence
> > > > > > >> > > > > > >> > > > > > listener implementation that
> selectively
> > > > saves
> > > > > > >> metrics
> > > > > > >> > > and
> > > > > > >> > > > > > >> expose
> > > > > > >> > > > > > >> > it
> > > > > > >> > > > > > >> > > > via
> > > > > > >> > > > > > >> > > > > > the IRC event endpoint which seems
> pretty
> > > > > close in
> > > > > > >> > > Iceberg
> > > > > > >> > > > > > >> > > > > > community. Another option I like more
> is to
> > > > > allow
> > > > > > >> > users
> > > > > > >> > > to
> > > > > > >> > > > > > >> develop
> > > > > > >> > > > > > >> > > > their
> > > > > > >> > > > > > >> > > > > > own listener, so they can persist
> > > scan/commit
> > > > > > >> metrics
> > > > > > >> > > to any
> > > > > > >> > > > > > >> > systems
> > > > > > >> > > > > > >> > > > they
> > > > > > >> > > > > > >> > > > > > prefer. In general, that feels like an
> > > > > > >> implementation
> > > > > > >> > > choice
> > > > > > >> > > > > > on
> > > > > > >> > > > > > >> top
> > > > > > >> > > > > > >> > > of
> > > > > > >> > > > > > >> > > > > the
> > > > > > >> > > > > > >> > > > > > event framework rather than something
> that
> > > > > needs to
> > > > > > >> > > shape
> > > > > > >> > > > > the
> > > > > > >> > > > > > >> core
> > > > > > >> > > > > > >> > > > > > architecture.
> > > > > > >> > > > > > >> > > > > >
> > > > > > >> > > > > > >> > > > > > 1.
> > > > > > >> > > > > > >> >
> > > > > > >> > >
> > > > > > >> >
> > > > > > >>
> > > > >
> > > >
> > >
> https://urldefense.com/v3/__https://lists.apache.org/thread/x9j8nscvy8hq61tyn01mj8yp6n9of0kp__;!!Iz9xO38YGHZK!4rnlVIqBQQQ6xpGe1PMV58edOS_cGrKoqdfhw4qSp_JQCavvEjaxXk0OwGPnxyEcO60AzKQK_XDtp7H3DdkFXcrXGcGDgks$
> > > > > > >> > > > > > >> > > > > >
> > > > > > >> > > > > > >> > > > > > Thanks,
> > > > > > >> > > > > > >> > > > > > Yufei
> > > > > > >> > > > > > >> > > > > >
> > > > > > >> > > > > > >> > > > > >
> > > > > > >> > > > > > >> > > > > > On Thu, Jun 4, 2026 at 5:14 PM EJ Wang
> <
> > > > > > >> > > > > > >> > > ej.wang.dev.interest@gmail.com
> > > > > > >> > > > > > >> > > > >
> > > > > > >> > > > > > >> > > > > > wrote:
> > > > > > >> > > > > > >> > > > > >
> > > > > > >> > > > > > >> > > > > > > Good point. I agree the existing
> > > > > > >> > > PolarisMetricsReporter is
> > > > > > >> > > > > > >> > already
> > > > > > >> > > > > > >> > > > very
> > > > > > >> > > > > > >> > > > > > > close to the right conceptual
> boundary. I
> > > > am
> > > > > not
> > > > > > >> > > > > proposing a
> > > > > > >> > > > > > >> > second
> > > > > > >> > > > > > >> > > > > > > parallel reporter concept. The
> > > distinction
> > > > I
> > > > > am
> > > > > > >> > > trying to
> > > > > > >> > > > > > >> make is
> > > > > > >> > > > > > >> > > > > > narrower:
> > > > > > >> > > > > > >> > > > > > >
> > > > > > >> > > > > > >> > > > > > > Current state:
> > > > > > >> > > > > > >> > > > > > >
> > > > > > >> > > > > > >> > > > > > >    - PolarisMetricsReporter is the
> > > existing
> > > > > > >> metrics
> > > > > > >> > > > > > reporting
> > > > > > >> > > > > > >> > hook.
> > > > > > >> > > > > > >> > > > > > >    - But it currently lives under
> > > > > > >> runtime/service.
> > > > > > >> > > > > > >> > > > > > >    - Its contract is documented
> around
> > > > > > >> Quarkus/CDI
> > > > > > >> > > > > discovery
> > > > > > >> > > > > > >> and
> > > > > > >> > > > > > >> > > > > > > @Identifier
> > > > > > >> > > > > > >> > > > > > >    selection.
> > > > > > >> > > > > > >> > > > > > >    - The default implementation is
> > > > log-only.
> > > > > > >> > > > > > >> > > > > > >    - The durable implementation
> writes
> > > > > through
> > > > > > >> > > > > > >> > > PolarisMetricsManager
> > > > > > >> > > > > > >> > > > > and
> > > > > > >> > > > > > >> > > > > > >    MetricsPersistence.
> > > > > > >> > > > > > >> > > > > > >    - MetricsPersistence is currently
> > > > > inherited by
> > > > > > >> > > > > > >> > BasePersistence.
> > > > > > >> > > > > > >> > > > > > >
> > > > > > >> > > > > > >> > > > > > > My proposal:
> > > > > > >> > > > > > >> > > > > > >
> > > > > > >> > > > > > >> > > > > > >    - Keep the reporting/emitting
> boundary
> > > > as
> > > > > the
> > > > > > >> > SPI.
> > > > > > >> > > > > > >> > > > > > >    - Revise that contract in a
> > > > > framework-agnostic
> > > > > > >> > > optional
> > > > > > >> > > > > > >> > Iceberg
> > > > > > >> > > > > > >> > > > > > *metrics
> > > > > > >> > > > > > >> > > > > > >    extension API layer*.
> > > > > > >> > > > > > >> > > > > > >    - Keep runtime/service
> responsible for
> > > > > REST
> > > > > > >> > > ingestion,
> > > > > > >> > > > > > >> > > table/authz
> > > > > > >> > > > > > >> > > > > > >    validation, and runtime wiring.
> > > > > > >> > > > > > >> > > > > > >    - Keep the battery implementation
> > > > > log-only in
> > > > > > >> the
> > > > > > >> > > > > metrics
> > > > > > >> > > > > > >> > > > extension
> > > > > > >> > > > > > >> > > > > > API
> > > > > > >> > > > > > >> > > > > > >    layer, *not under
> runtime/service*.
> > > > > > >> > > > > > >> > > > > > >    - Treat durable JDBC metrics as
> one
> > > > > > >> > implementation
> > > > > > >> > > of
> > > > > > >> > > > > the
> > > > > > >> > > > > > >> > > > reporting
> > > > > > >> > > > > > >> > > > > > SPI.
> > > > > > >> > > > > > >> > > > > > >    - Decompose the
> MetricsPersistence and
> > > > > > >> > > BasePersistence
> > > > > > >> > > > > > >> > coupling
> > > > > > >> > > > > > >> > > as
> > > > > > >> > > > > > >> > > > > > >    durable implementation detail.
> > > > > > >> > > > > > >> > > > > > >
> > > > > > >> > > > > > >> > > > > > > So the difference is not “new
> reporter
> > > > > concept vs
> > > > > > >> > old
> > > > > > >> > > > > > reporter
> > > > > > >> > > > > > >> > > > > concept.”
> > > > > > >> > > > > > >> > > > > > > but:
> > > > > > >> > > > > > >> > > > > > >
> > > > > > >> > > > > > >> > > > > > >
> > > > > > >> > > > > > >> > > > > > >    - PolarisMetricsReporter today =
> > > > > > >> runtime/service
> > > > > > >> > > > > > CDI-shaped
> > > > > > >> > > > > > >> > > hook.
> > > > > > >> > > > > > >> > > > > > >    - Target reporting SPI = same
> > > conceptual
> > > > > > >> > boundary,
> > > > > > >> > > but
> > > > > > >> > > > > > >> placed
> > > > > > >> > > > > > >> > in
> > > > > > >> > > > > > >> > > > the
> > > > > > >> > > > > > >> > > > > > >    right extension API layer and kept
> > > > > > >> > > framework-agnostic.
> > > > > > >> > > > > > >> > > > > > >
> > > > > > >> > > > > > >> > > > > > > For the current metrics PR, I think
> the
> > > > > minimal
> > > > > > >> > > > > > architectural
> > > > > > >> > > > > > >> > > cleanup
> > > > > > >> > > > > > >> > > > > > > before merge is to *put the
> reporting SPI
> > > > > > >> contract
> > > > > > >> > > and the
> > > > > > >> > > > > > >> > battery
> > > > > > >> > > > > > >> > > > > > default
> > > > > > >> > > > > > >> > > > > > > in the right place*. The durable JDBC
> > > > > > >> > implementation,
> > > > > > >> > > > > > >> > event-backed
> > > > > > >> > > > > > >> > > > > async
> > > > > > >> > > > > > >> > > > > > > implementation, external queue
> support,
> > > and
> > > > > > >> deeper
> > > > > > >> > > > > > persistence
> > > > > > >> > > > > > >> > > > cleanup
> > > > > > >> > > > > > >> > > > > > can
> > > > > > >> > > > > > >> > > > > > > continue as follow-ups, as long as
> we do
> > > > not
> > > > > lock
> > > > > > >> > the
> > > > > > >> > > SPI
> > > > > > >> > > > > > >> > boundary
> > > > > > >> > > > > > >> > > to
> > > > > > >> > > > > > >> > > > > > > BasePersistence or to runtime/service
> > > > wiring.
> > > > > > >> > > > > > >> > > > > > >
> > > > > > >> > > > > > >> > > > > > > -ej
> > > > > > >> > > > > > >> > > > > > >
> > > > > > >> > > > > > >> > > > > > > On Wed, Jun 3, 2026 at 3:00 PM Dmitri
> > > > > > >> Bourlatchkov <
> > > > > > >> > > > > > >> > > dimas@apache.org
> > > > > > >> > > > > > >> > > > >
> > > > > > >> > > > > > >> > > > > > > wrote:
> > > > > > >> > > > > > >> > > > > > >
> > > > > > >> > > > > > >> > > > > > > > Hi EJ,
> > > > > > >> > > > > > >> > > > > > > >
> > > > > > >> > > > > > >> > > > > > > > Thanks for the recap / summary!
> > > > > > >> > > > > > >> > > > > > > >
> > > > > > >> > > > > > >> > > > > > > > > Do folks agree that the stable
> SPI
> > > > > boundary
> > > > > > >> > > should be
> > > > > > >> > > > > > >> > > > > > > > metrics reporting/emitting, not
> metrics
> > > > > > >> > persistence?
> > > > > > >> > > > > > >> > > > > > > >
> > > > > > >> > > > > > >> > > > > > > > SGTM.
> > > > > > >> > > > > > >> > > > > > > >
> > > > > > >> > > > > > >> > > > > > > > > Does an optional Iceberg metrics
> > > > > extension
> > > > > > >> API
> > > > > > >> > > layer
> > > > > > >> > > > > > sound
> > > > > > >> > > > > > >> > like
> > > > > > >> > > > > > >> > > > the
> > > > > > >> > > > > > >> > > > > > > right
> > > > > > >> > > > > > >> > > > > > > > home for this SPI?
> > > > > > >> > > > > > >> > > > > > > >
> > > > > > >> > > > > > >> > > > > > > > How is that different from current
> > > > > > >> > > > > PolarisMetricsReporter?
> > > > > > >> > > > > > >> > > > > > > >
> > > > > > >> > > > > > >> > > > > > > > > Should the current durable
> metrics
> > > work
> > > > > be
> > > > > > >> > > reframed
> > > > > > >> > > > > as a
> > > > > > >> > > > > > >> > > durable
> > > > > > >> > > > > > >> > > > > > > > JDBC reference implementation of
> that
> > > > SPI?
> > > > > > >> > > > > > >> > > > > > > >
> > > > > > >> > > > > > >> > > > > > > > SGTM.
> > > > > > >> > > > > > >> > > > > > > >
> > > > > > >> > > > > > >> > > > > > > > > What is the smallest PR sequence
> to
> > > get
> > > > > there
> > > > > > >> > > without
> > > > > > >> > > > > > >> > blocking
> > > > > > >> > > > > > >> > > > > > > > the current metrics work
> unnecessarily?
> > > > > > >> > > > > > >> > > > > > > >
> > > > > > >> > > > > > >> > > > > > > > Let's get
> > > > > > >> > >
> > > > > > >> >
> > > > > > >>
> > > > >
> > > >
> > >
> https://urldefense.com/v3/__https://github.com/apache/polaris/pull/4397__;!!Iz9xO38YGHZK!4rnlVIqBQQQ6xpGe1PMV58edOS_cGrKoqdfhw4qSp_JQCavvEjaxXk0OwGPnxyEcO60AzKQK_XDtp7H3DdkFXcrXuu5jefU$
> > > > > > >> > > > > > >> merged
> > > > > > >> > > > > > >> > > > first.
> > > > > > >> > > > > > >> > > > > > > >
> > > > > > >> > > > > > >> > > > > > > > Then, I'd propose isolating the
> Metrics
> > > > > schema
> > > > > > >> > from
> > > > > > >> > > the
> > > > > > >> > > > > > >> > MetaStore
> > > > > > >> > > > > > >> > > > > > schema.
> > > > > > >> > > > > > >> > > > > > > > This will probably have some ripple
> > > > effect
> > > > > into
> > > > > > >> > the
> > > > > > >> > > > > > >> bootstrap
> > > > > > >> > > > > > >> > > > > > workflows,
> > > > > > >> > > > > > >> > > > > > > so
> > > > > > >> > > > > > >> > > > > > > > it's not a trivial change.
> > > > > > >> > > > > > >> > > > > > > >
> > > > > > >> > > > > > >> > > > > > > > Then, let's reassess.
> > > > > > >> > > > > > >> > > > > > > >
> > > > > > >> > > > > > >> > > > > > > > Cheers,
> > > > > > >> > > > > > >> > > > > > > > Dmitri.
> > > > > > >> > > > > > >> > > > > > > >
> > > > > > >> > > > > > >> > > > > > > > On Wed, Jun 3, 2026 at 5:40 PM EJ
> Wang
> > > <
> > > > > > >> > > > > > >> > > > > ej.wang.dev.interest@gmail.com
> > > > > > >> > > > > > >> > > > > > >
> > > > > > >> > > > > > >> > > > > > > > wrote:
> > > > > > >> > > > > > >> > > > > > > >
> > > > > > >> > > > > > >> > > > > > > > > Hi folks,
> > > > > > >> > > > > > >> > > > > > > > >
> > > > > > >> > > > > > >> > > > > > > > > Thanks again for the discussion
> > > today.
> > > > I
> > > > > > >> updated
> > > > > > >> > > the
> > > > > > >> > > > > > sync
> > > > > > >> > > > > > >> doc
> > > > > > >> > > > > > >> > > > with
> > > > > > >> > > > > > >> > > > > > > notes
> > > > > > >> > > > > > >> > > > > > > > > from the third metrics
> architecture
> > > > sync:
> > > > > > >> > > > > > >> > > > > > > > >
> > > > > > >> > > > > > >> > > > > > > > >
> > > > > > >> > > > > > >> > > > > > > > >
> > > > > > >> > > > > > >> > > > > > > >
> > > > > > >> > > > > > >> > > > > > >
> > > > > > >> > > > > > >> > > > > >
> > > > > > >> > > > > > >> > > > >
> > > > > > >> > > > > > >> > > >
> > > > > > >> > > > > > >> > >
> > > > > > >> > > > > > >> >
> > > > > > >> > > > > > >>
> > > > > > >> > > > > >
> > > > > > >> > > > >
> > > > > > >> > >
> > > > > > >> >
> > > > > > >>
> > > > >
> > > >
> > >
> https://urldefense.com/v3/__https://docs.google.com/document/d/100h7c4damrUzVuquYbBHM0EvA4LSWuW2IT2dN_7nYVA/edit?tab=t.k96s2xyqr5u1__;!!Iz9xO38YGHZK!4rnlVIqBQQQ6xpGe1PMV58edOS_cGrKoqdfhw4qSp_JQCavvEjaxXk0OwGPnxyEcO60AzKQK_XDtp7H3DdkFXcrXrqCL-QE$
> > > > > > >> > > > > > >> > > > > > > > >
> > > > > > >> > > > > > >> > > > > > > > > A few highlights from today’s
> > > > discussion:
> > > > > > >> > > > > > >> > > > > > > > >
> > > > > > >> > > > > > >> > > > > > > > >    - We clarified that Iceberg
> > > metrics
> > > > > > >> reporting
> > > > > > >> > > can
> > > > > > >> > > > > be
> > > > > > >> > > > > > >> > > > interpreted
> > > > > > >> > > > > > >> > > > > > > > either
> > > > > > >> > > > > > >> > > > > > > > >    as sync or async handling from
> > > > > Polaris’s
> > > > > > >> > > > > perspective,
> > > > > > >> > > > > > >> so
> > > > > > >> > > > > > >> > > > Polaris
> > > > > > >> > > > > > >> > > > > > > > should
> > > > > > >> > > > > > >> > > > > > > > >    stay flexible as a platform
> > > instead
> > > > of
> > > > > > >> baking
> > > > > > >> > > one
> > > > > > >> > > > > > >> handling
> > > > > > >> > > > > > >> > > > model
> > > > > > >> > > > > > >> > > > > > > into
> > > > > > >> > > > > > >> > > > > > > > > the
> > > > > > >> > > > > > >> > > > > > > > >    REST API behavior.
> > > > > > >> > > > > > >> > > > > > > > >    - We aligned that the built-in
> > > > > (battery)
> > > > > > >> > > behavior
> > > > > > >> > > > > for
> > > > > > >> > > > > > >> > > metrics
> > > > > > >> > > > > > >> > > > > > > emitting
> > > > > > >> > > > > > >> > > > > > > > >    can stay simple:
> no-op/log-only is
> > > > > enough
> > > > > > >> as
> > > > > > >> > > the
> > > > > > >> > > > > > >> default.
> > > > > > >> > > > > > >> > > > > > > > >    - Durable metrics persistence
> > > should
> > > > > be
> > > > > > >> > > treated as
> > > > > > >> > > > > an
> > > > > > >> > > > > > >> > > > > > implementation
> > > > > > >> > > > > > >> > > > > > > > of
> > > > > > >> > > > > > >> > > > > > > > >    the metrics reporting path,
> not as
> > > > the
> > > > > > >> core
> > > > > > >> > SPI
> > > > > > >> > > > > > >> boundary.
> > > > > > >> > > > > > >> > > > > > > > >    - The existing durable metrics
> > > work
> > > > > can be
> > > > > > >> > > reviewed
> > > > > > >> > > > > > as
> > > > > > >> > > > > > >> a
> > > > > > >> > > > > > >> > > > > reference
> > > > > > >> > > > > > >> > > > > > > > >    implementation of the
> reporting
> > > SPI,
> > > > > with
> > > > > > >> > > > > > >> > > persistence-related
> > > > > > >> > > > > > >> > > > > > logic
> > > > > > >> > > > > > >> > > > > > > > kept
> > > > > > >> > > > > > >> > > > > > > > >    self-contained under a metrics
> > > > durable
> > > > > > >> > > > > implementation
> > > > > > >> > > > > > >> > module
> > > > > > >> > > > > > >> > > > > > rather
> > > > > > >> > > > > > >> > > > > > > > than
> > > > > > >> > > > > > >> > > > > > > > >    scattered through core entity
> > > > > persistence.
> > > > > > >> > > > > > >> > > > > > > > >    - Dashboard/insights remains a
> > > real
> > > > > use
> > > > > > >> case,
> > > > > > >> > > but
> > > > > > >> > > > > we
> > > > > > >> > > > > > >> > agreed
> > > > > > >> > > > > > >> > > to
> > > > > > >> > > > > > >> > > > > > keep
> > > > > > >> > > > > > >> > > > > > > it
> > > > > > >> > > > > > >> > > > > > > > >    separate from the core metrics
> > > > intake
> > > > > > >> > > discussion
> > > > > > >> > > > > for
> > > > > > >> > > > > > >> now.
> > > > > > >> > > > > > >> > > > > > > > >
> > > > > > >> > > > > > >> > > > > > > > > I also did a quick source check
> after
> > > > the
> > > > > > >> > meeting
> > > > > > >> > > to
> > > > > > >> > > > > > make
> > > > > > >> > > > > > >> > sure
> > > > > > >> > > > > > >> > > we
> > > > > > >> > > > > > >> > > > > are
> > > > > > >> > > > > > >> > > > > > > > > describing the current state
> > > > accurately.
> > > > > > >> > > > > > >> > > > > > > > >
> > > > > > >> > > > > > >> > > > > > > > > Current state:
> > > > > > >> > > > > > >> > > > > > > > >
> > > > > > >> > > > > > >> > > > > > > > >    - Polaris already has a
> metrics
> > > > > reporting
> > > > > > >> > hook:
> > > > > > >> > > > > > >> > > > > > > > PolarisMetricsReporter.
> > > > > > >> > > > > > >> > > > > > > > >    - The default implementation
> is
> > > > > > >> > > > > > DefaultMetricsReporter,
> > > > > > >> > > > > > >> > > > selected
> > > > > > >> > > > > > >> > > > > > by
> > > > > > >> > > > > > >> > > > > > > > >
> > > > > > >> > > polaris.iceberg-metrics.reporting.type=default. It
> > > > > > >> > > > > is
> > > > > > >> > > > > > >> > > > log-only,
> > > > > > >> > > > > > >> > > > > > > > >    effectively quiet unless
> metrics
> > > > > logging
> > > > > > >> is
> > > > > > >> > > > > enabled.
> > > > > > >> > > > > > >> > > > > > > > >    - There is also a
> > > > > > >> PersistingMetricsReporter,
> > > > > > >> > > > > selected
> > > > > > >> > > > > > >> by
> > > > > > >> > > > > > >> > > > > > > > >
> > > > > > >> > > polaris.iceberg-metrics.reporting.type=persisting,
> > > > > > >> > > > > > >> which
> > > > > > >> > > > > > >> > > > > converts
> > > > > > >> > > > > > >> > > > > > > > >    Iceberg scan/commit reports
> into
> > > > > Polaris
> > > > > > >> > > metrics
> > > > > > >> > > > > > >> records
> > > > > > >> > > > > > >> > and
> > > > > > >> > > > > > >> > > > > > writes
> > > > > > >> > > > > > >> > > > > > > > > through
> > > > > > >> > > > > > >> > > > > > > > >    PolarisMetricsManager ->
> > > > > > >> MetricsPersistence.
> > > > > > >> > > > > > >> > > > > > > > >    - MetricsPersistence currently
> > > > exists
> > > > > in
> > > > > > >> the
> > > > > > >> > > > > > >> persistence
> > > > > > >> > > > > > >> > > layer
> > > > > > >> > > > > > >> > > > > and
> > > > > > >> > > > > > >> > > > > > > is
> > > > > > >> > > > > > >> > > > > > > > >    inherited by BasePersistence.
> > > > > > >> > > > > > >> > > > > > > > >
> > > > > > >> > > > > > >> > > > > > > > > My read from the discussion is
> that
> > > the
> > > > > > >> target
> > > > > > >> > > > > boundary
> > > > > > >> > > > > > >> > should
> > > > > > >> > > > > > >> > > > not
> > > > > > >> > > > > > >> > > > > be
> > > > > > >> > > > > > >> > > > > > > > > MetricsPersistence as inherited
> by
> > > > > > >> > > BasePersistence.
> > > > > > >> > > > > That
> > > > > > >> > > > > > >> path
> > > > > > >> > > > > > >> > > > > should
> > > > > > >> > > > > > >> > > > > > be
> > > > > > >> > > > > > >> > > > > > > > > decomposed as durable
> implementation
> > > > > detail
> > > > > > >> > (taken
> > > > > > >> > > > > care
> > > > > > >> > > > > > >> of by
> > > > > > >> > > > > > >> > > > > > > > >
> > > > > > >> >
> > > > > > >>
> > > > >
> > > >
> > >
> https://urldefense.com/v3/__https://github.com/apache/polaris/pull/4397__;!!Iz9xO38YGHZK!4rnlVIqBQQQ6xpGe1PMV58edOS_cGrKoqdfhw4qSp_JQCavvEjaxXk0OwGPnxyEcO60AzKQK_XDtp7H3DdkFXcrXuu5jefU$
> > > > > > >> ).
> > > > > > >> > The
> > > > > > >> > > > > > stable
> > > > > > >> > > > > > >> > > > extension
> > > > > > >> > > > > > >> > > > > > > point
> > > > > > >> > > > > > >> > > > > > > > > should instead be the metrics
> > > > > > >> reporting/emitting
> > > > > > >> > > > > > boundary.
> > > > > > >> > > > > > >> > > > > > > > >
> > > > > > >> > > > > > >> > > > > > > > > Concretely, I think the next
> proposal
> > > > > should
> > > > > > >> be
> > > > > > >> > > shaped
> > > > > > >> > > > > > >> like
> > > > > > >> > > > > > >> > > this:
> > > > > > >> > > > > > >> > > > > > > > >
> > > > > > >> > > > > > >> > > > > > > > >    1.
> > > > > > >> > > > > > >> > > > > > > > >
> > > > > > >> > > > > > >> > > > > > > > >    Define the proper metrics
> > > reporting
> > > > > SPI
> > > > > > >> at an
> > > > > > >> > > > > > optional
> > > > > > >> > > > > > >> > > Iceberg
> > > > > > >> > > > > > >> > > > > > > metrics
> > > > > > >> > > > > > >> > > > > > > > >    extension API layer.
> > > > > > >> > > > > > >> > > > > > > > >
> > > > > > >> > > > > > >> > > > > > > > >    Example direction:
> > > > > > >> > > > > > >> > > > > > > > >
> > > > > > >> > > > > > >> > > > > > > > >    public interface
> > > > > IcebergMetricsReporter {
> > > > > > >> > > > > > >> > > > > > > > >      void
> > > > > > >> > reportMetric(IcebergMetricsReportContext
> > > > > > >> > > > > > >> context,
> > > > > > >> > > > > > >> > > > > > > > > MetricsReport report);
> > > > > > >> > > > > > >> > > > > > > > >    }
> > > > > > >> > > > > > >> > > > > > > > >
> > > > > > >> > > > > > >> > > > > > > > >    The context would carry the
> small
> > > > > > >> > > Polaris-resolved
> > > > > > >> > > > > > >> > envelope:
> > > > > > >> > > > > > >> > > > > > > > >    catalog/table identity, report
> > > type,
> > > > > > >> received
> > > > > > >> > > > > > >> timestamp,
> > > > > > >> > > > > > >> > and
> > > > > > >> > > > > > >> > > > > > > > > request/trace
> > > > > > >> > > > > > >> > > > > > > > >    context if available. The raw
> > > > Iceberg
> > > > > > >> > > MetricsReport
> > > > > > >> > > > > > >> > remains
> > > > > > >> > > > > > >> > > > the
> > > > > > >> > > > > > >> > > > > > > > payload.
> > > > > > >> > > > > > >> > > > > > > > >    2.
> > > > > > >> > > > > > >> > > > > > > > >
> > > > > > >> > > > > > >> > > > > > > > >    Keep runtime/service as the
> REST
> > > > > ingestion
> > > > > > >> > and
> > > > > > >> > > > > wiring
> > > > > > >> > > > > > >> > layer.
> > > > > > >> > > > > > >> > > > > > > > >
> > > > > > >> > > > > > >> > > > > > > > >    The REST handler still
> resolves
> > > the
> > > > > table
> > > > > > >> and
> > > > > > >> > > > > > performs
> > > > > > >> > > > > > >> > authz
> > > > > > >> > > > > > >> > > > > > before
> > > > > > >> > > > > > >> > > > > > > > >    accepting the report. After
> that,
> > > it
> > > > > calls
> > > > > > >> > the
> > > > > > >> > > > > > selected
> > > > > > >> > > > > > >> > > > > reporting
> > > > > > >> > > > > > >> > > > > > > > >    implementation.
> > > > > > >> > > > > > >> > > > > > > > >    3.
> > > > > > >> > > > > > >> > > > > > > > >
> > > > > > >> > > > > > >> > > > > > > > >    Keep the battery default
> > > > > no-op/log-only.
> > > > > > >> > > > > > >> > > > > > > > >
> > > > > > >> > > > > > >> > > > > > > > >    This preserves an out-of-box
> safe
> > > > > default
> > > > > > >> and
> > > > > > >> > > > > avoids
> > > > > > >> > > > > > >> > > > requiring a
> > > > > > >> > > > > > >> > > > > > > > durable
> > > > > > >> > > > > > >> > > > > > > > >    metrics store for every
> Polaris
> > > > > > >> deployment.
> > > > > > >> > > > > > >> > > > > > > > >    4.
> > > > > > >> > > > > > >> > > > > > > > >
> > > > > > >> > > > > > >> > > > > > > > >    Move durable JDBC metrics
> into a
> > > > > > >> > self-contained
> > > > > > >> > > > > > >> > > implementation
> > > > > > >> > > > > > >> > > > > of
> > > > > > >> > > > > > >> > > > > > > the
> > > > > > >> > > > > > >> > > > > > > > >    reporting SPI.
> > > > > > >> > > > > > >> > > > > > > > >
> > > > > > >> > > > > > >> > > > > > > > >    That implementation can own
> its
> > > > > schema,
> > > > > > >> > > bootstrap,
> > > > > > >> > > > > > >> > > retention,
> > > > > > >> > > > > > >> > > > > and
> > > > > > >> > > > > > >> > > > > > > read
> > > > > > >> > > > > > >> > > > > > > > >    API support. It should not
> define
> > > > the
> > > > > core
> > > > > > >> > > > > reporting
> > > > > > >> > > > > > >> SPI
> > > > > > >> > > > > > >> > > > > boundary,
> > > > > > >> > > > > > >> > > > > > > and
> > > > > > >> > > > > > >> > > > > > > > > it
> > > > > > >> > > > > > >> > > > > > > > >    should not require metrics
> > > > > persistence to
> > > > > > >> > > remain
> > > > > > >> > > > > > >> inherited
> > > > > > >> > > > > > >> > > > from
> > > > > > >> > > > > > >> > > > > > > > >    BasePersistence.
> > > > > > >> > > > > > >> > > > > > > > >    5.
> > > > > > >> > > > > > >> > > > > > > > >
> > > > > > >> > > > > > >> > > > > > > > >    Treat async/event-backed
> handling
> > > as
> > > > > > >> another
> > > > > > >> > > > > > >> > implementation
> > > > > > >> > > > > > >> > > of
> > > > > > >> > > > > > >> > > > > the
> > > > > > >> > > > > > >> > > > > > > > same
> > > > > > >> > > > > > >> > > > > > > > >    reporting SPI.
> > > > > > >> > > > > > >> > > > > > > > >
> > > > > > >> > > > > > >> > > > > > > > >    For example, an event-backed
> > > > reporter
> > > > > > >> could
> > > > > > >> > > > > enqueue a
> > > > > > >> > > > > > >> > > metrics
> > > > > > >> > > > > > >> > > > > > event
> > > > > > >> > > > > > >> > > > > > > > and
> > > > > > >> > > > > > >> > > > > > > > >    let listeners handle durable
> > > storage
> > > > > or
> > > > > > >> other
> > > > > > >> > > > > sinks.
> > > > > > >> > > > > > >> If we
> > > > > > >> > > > > > >> > > > later
> > > > > > >> > > > > > >> > > > > > > need
> > > > > > >> > > > > > >> > > > > > > > a
> > > > > > >> > > > > > >> > > > > > > > >    replaceable queue engine, that
> > > seems
> > > > > like
> > > > > > >> a
> > > > > > >> > > shared
> > > > > > >> > > > > > >> > > > event/metrics
> > > > > > >> > > > > > >> > > > > > > > > substrate
> > > > > > >> > > > > > >> > > > > > > > >    topic rather than a
> metrics-only
> > > > > > >> requirement.
> > > > > > >> > > > > > >> > > > > > > > >
> > > > > > >> > > > > > >> > > > > > > > > This framing lets us keep the
> REST
> > > > > metrics
> > > > > > >> > > endpoint
> > > > > > >> > > > > > >> simple,
> > > > > > >> > > > > > >> > > > > preserve
> > > > > > >> > > > > > >> > > > > > > the
> > > > > > >> > > > > > >> > > > > > > > > current default behavior, support
> > > > durable
> > > > > > >> > metrics
> > > > > > >> > > > > users,
> > > > > > >> > > > > > >> and
> > > > > > >> > > > > > >> > > > still
> > > > > > >> > > > > > >> > > > > > > leave
> > > > > > >> > > > > > >> > > > > > > > > room for async/event-backed or
> > > > > > >> > > external-queue-based
> > > > > > >> > > > > > >> > > > > implementations.
> > > > > > >> > > > > > >> > > > > > > > >
> > > > > > >> > > > > > >> > > > > > > > > I think the main follow-up
> questions
> > > > are:
> > > > > > >> > > > > > >> > > > > > > > >
> > > > > > >> > > > > > >> > > > > > > > >    - Do folks agree that the
> stable
> > > SPI
> > > > > > >> boundary
> > > > > > >> > > > > should
> > > > > > >> > > > > > be
> > > > > > >> > > > > > >> > > > metrics
> > > > > > >> > > > > > >> > > > > > > > >    reporting/emitting, not
> metrics
> > > > > > >> persistence?
> > > > > > >> > > > > > >> > > > > > > > >    - Does an optional Iceberg
> metrics
> > > > > > >> extension
> > > > > > >> > > API
> > > > > > >> > > > > > layer
> > > > > > >> > > > > > >> > sound
> > > > > > >> > > > > > >> > > > > like
> > > > > > >> > > > > > >> > > > > > > the
> > > > > > >> > > > > > >> > > > > > > > >    right home for this SPI?
> > > > > > >> > > > > > >> > > > > > > > >    - Should the current durable
> > > metrics
> > > > > work
> > > > > > >> be
> > > > > > >> > > > > reframed
> > > > > > >> > > > > > >> as a
> > > > > > >> > > > > > >> > > > > durable
> > > > > > >> > > > > > >> > > > > > > > JDBC
> > > > > > >> > > > > > >> > > > > > > > >    reference implementation of
> that
> > > > SPI?
> > > > > > >> > > > > > >> > > > > > > > >    - What is the smallest PR
> sequence
> > > > to
> > > > > get
> > > > > > >> > there
> > > > > > >> > > > > > without
> > > > > > >> > > > > > >> > > > blocking
> > > > > > >> > > > > > >> > > > > > the
> > > > > > >> > > > > > >> > > > > > > > >    current metrics work
> > > unnecessarily?
> > > > > > >> > > > > > >> > > > > > > > >
> > > > > > >> > > > > > >> > > > > > > > > Thanks,
> > > > > > >> > > > > > >> > > > > > > > > -ej
> > > > > > >> > > > > > >> > > > > > > > >
> > > > > > >> > > > > > >> > > > > > > > > On Fri, May 15, 2026 at 5:16 PM
> > > Dmitri
> > > > > > >> > > Bourlatchkov <
> > > > > > >> > > > > > >> > > > > > dimas@apache.org>
> > > > > > >> > > > > > >> > > > > > > > > wrote:
> > > > > > >> > > > > > >> > > > > > > > >
> > > > > > >> > > > > > >> > > > > > > > > > Hi JB,
> > > > > > >> > > > > > >> > > > > > > > > >
> > > > > > >> > > > > > >> > > > > > > > > > Could you set up another
> meeting,
> > > > > please?
> > > > > > >> Same
> > > > > > >> > > time
> > > > > > >> > > > > on
> > > > > > >> > > > > > >> > > > Wednesday
> > > > > > >> > > > > > >> > > > > as
> > > > > > >> > > > > > >> > > > > > > > last
> > > > > > >> > > > > > >> > > > > > > > > > time... I hope it works for
> > > everyone.
> > > > > > >> > > > > > >> > > > > > > > > >
> > > > > > >> > > > > > >> > > > > > > > > > Cheers,
> > > > > > >> > > > > > >> > > > > > > > > > Dmitri.
> > > > > > >> > > > > > >> > > > > > > > > >
> > > > > > >> > > > > > >> > > > > > > > > > On Fri, May 15, 2026 at 8:06 PM
> > > Yufei
> > > > > Gu <
> > > > > > >> > > > > > >> > > flyrain000@gmail.com
> > > > > > >> > > > > > >> > > > >
> > > > > > >> > > > > > >> > > > > > > wrote:
> > > > > > >> > > > > > >> > > > > > > > > >
> > > > > > >> > > > > > >> > > > > > > > > > > +1 on another sync call next
> > > week.
> > > > > > >> > > > > > >> > > > > > > > > > >
> > > > > > >> > > > > > >> > > > > > > > > > > Yufei
> > > > > > >> > > > > > >> > > > > > > > > > >
> > > > > > >> > > > > > >> > > > > > > > > > >
> > > > > > >> > > > > > >> > > > > > > > > > > On Fri, May 15, 2026 at
> 4:52 PM
> > > > > Dmitri
> > > > > > >> > > > > Bourlatchkov
> > > > > > >> > > > > > <
> > > > > > >> > > > > > >> > > > > > > > dimas@apache.org>
> > > > > > >> > > > > > >> > > > > > > > > > > wrote:
> > > > > > >> > > > > > >> > > > > > > > > > >
> > > > > > >> > > > > > >> > > > > > > > > > > > Hi All,
> > > > > > >> > > > > > >> > > > > > > > > > > >
> > > > > > >> > > > > > >> > > > > > > > > > > > WDYT about another sync
> call
> > > next
> > > > > week?
> > > > > > >> > > > > > >> > > > > > > > > > > >
> > > > > > >> > > > > > >> > > > > > > > > > > > Thanks,
> > > > > > >> > > > > > >> > > > > > > > > > > > Dmitri.
> > > > > > >> > > > > > >> > > > > > > > > > > >
> > > > > > >> > > > > > >> > > > > > > > > > > > On Wed, May 6, 2026 at
> 5:29 PM
> > > > > Dmitri
> > > > > > >> > > > > > Bourlatchkov <
> > > > > > >> > > > > > >> > > > > > > > dimas@apache.org
> > > > > > >> > > > > > >> > > > > > > > > >
> > > > > > >> > > > > > >> > > > > > > > > > > > wrote:
> > > > > > >> > > > > > >> > > > > > > > > > > >
> > > > > > >> > > > > > >> > > > > > > > > > > > > Hi EJ,
> > > > > > >> > > > > > >> > > > > > > > > > > > >
> > > > > > >> > > > > > >> > > > > > > > > > > > > Thanks for the summary!
> It
> > > > covers
> > > > > > >> what
> > > > > > >> > we
> > > > > > >> > > > > > >> discussed
> > > > > > >> > > > > > >> > in
> > > > > > >> > > > > > >> > > > the
> > > > > > >> > > > > > >> > > > > > > > meeting
> > > > > > >> > > > > > >> > > > > > > > > > very
> > > > > > >> > > > > > >> > > > > > > > > > > > > well, IMHO.
> > > > > > >> > > > > > >> > > > > > > > > > > > >
> > > > > > >> > > > > > >> > > > > > > > > > > > > Looking forward to
> concrete
> > > PRs
> > > > > :)
> > > > > > >> > > > > > >> > > > > > > > > > > > >
> > > > > > >> > > > > > >> > > > > > > > > > > > > Cheers,
> > > > > > >> > > > > > >> > > > > > > > > > > > > Dmitri.
> > > > > > >> > > > > > >> > > > > > > > > > > > >
> > > > > > >> > > > > > >> > > > > > > > > > > > > On Wed, May 6, 2026 at
> > > 5:08 PM
> > > > EJ
> > > > > > >> Wang <
> > > > > > >> > > > > > >> > > > > > > > > > ej.wang.dev.interest@gmail.com
> > > > > > >> > > > > > >> > > > > > > > > > > >
> > > > > > >> > > > > > >> > > > > > > > > > > > > wrote:
> > > > > > >> > > > > > >> > > > > > > > > > > > >
> > > > > > >> > > > > > >> > > > > > > > > > > > >> Hi folks,
> > > > > > >> > > > > > >> > > > > > > > > > > > >>
> > > > > > >> > > > > > >> > > > > > > > > > > > >> We had a community sync
> > > > earlier,
> > > > > > >> thanks
> > > > > > >> > > JB
> > > > > > >> > > > > for
> > > > > > >> > > > > > >> > > > scheduling
> > > > > > >> > > > > > >> > > > > > it.
> > > > > > >> > > > > > >> > > > > > > > > Notes
> > > > > > >> > > > > > >> > > > > > > > > > > from
> > > > > > >> > > > > > >> > > > > > > > > > > > >> the first metrics
> > > architecture
> > > > > sync
> > > > > > >> > (May
> > > > > > >> > > 6,
> > > > > > >> > > > > > >> 10-11am
> > > > > > >> > > > > > >> > > PT).
> > > > > > >> > > > > > >> > > > > > > > > Discussion
> > > > > > >> > > > > > >> > > > > > > > > > > doc
> > > > > > >> > > > > > >> > > > > > > > > > > > >> with per-section status:
> > > > > > >> > > > > > >> > > > > > > > > > > > >>
> > > > > > >> > > > > > >> > > > > > > > > > > > >>
> > > > > > >> > > > > > >> > > > > > > > > > > >
> > > > > > >> > > > > > >> > > > > > > > > > >
> > > > > > >> > > > > > >> > > > > > > > > >
> > > > > > >> > > > > > >> > > > > > > > >
> > > > > > >> > > > > > >> > > > > > > >
> > > > > > >> > > > > > >> > > > > > >
> > > > > > >> > > > > > >> > > > > >
> > > > > > >> > > > > > >> > > > >
> > > > > > >> > > > > > >> > > >
> > > > > > >> > > > > > >> > >
> > > > > > >> > > > > > >> >
> > > > > > >> > > > > > >>
> > > > > > >> > > > > >
> > > > > > >> > > > >
> > > > > > >> > >
> > > > > > >> >
> > > > > > >>
> > > > >
> > > >
> > >
> https://urldefense.com/v3/__https://docs.google.com/document/d/100h7c4damrUzVuquYbBHM0EvA4LSWuW2IT2dN_7nYVA/edit?tab=t.0__;!!Iz9xO38YGHZK!4rnlVIqBQQQ6xpGe1PMV58edOS_cGrKoqdfhw4qSp_JQCavvEjaxXk0OwGPnxyEcO60AzKQK_XDtp7H3DdkFXcrXgIddeUs$
> > > > > > >> > > > > > >> > > > > > > > > > > > >>
> > > > > > >> > > > > > >> > > > > > > > > > > > >> *The meeting covered
> both
> > > > topics
> > > > > > >> from
> > > > > > >> > the
> > > > > > >> > > > > doc.
> > > > > > >> > > > > > >> > > > > > Direction-level
> > > > > > >> > > > > > >> > > > > > > > > > > alignment
> > > > > > >> > > > > > >> > > > > > > > > > > > >> was reached on the
> headline
> > > > > pieces;
> > > > > > >> > > details
> > > > > > >> > > > > > >> remain
> > > > > > >> > > > > > >> > for
> > > > > > >> > > > > > >> > > > PR
> > > > > > >> > > > > > >> > > > > > > review
> > > > > > >> > > > > > >> > > > > > > > > or
> > > > > > >> > > > > > >> > > > > > > > > > > > >> follow-up sessions.*
> > > > > > >> > > > > > >> > > > > > > > > > > > >>
> > > > > > >> > > > > > >> > > > > > > > > > > > >> *Topic 1 — Persistence
> > > schema
> > > > > > >> redesign*
> > > > > > >> > > > > > >> > > > > > > > > > > > >> Idea-level alignment on
> > > > > > >> consolidating
> > > > > > >> > > > > per-type
> > > > > > >> > > > > > >> > tables
> > > > > > >> > > > > > >> > > > > > > > > > > > >> (scan_metrics_report,
> > > > > > >> > > > > > >> > > > > > > > > > > > >> commit_metrics_report)
> into
> > > a
> > > > > single
> > > > > > >> > > > > > >> metrics_report
> > > > > > >> > > > > > >> > > > table.
> > > > > > >> > > > > > >> > > > > > The
> > > > > > >> > > > > > >> > > > > > > > > > > > motivating
> > > > > > >> > > > > > >> > > > > > > > > > > > >> cost is the surface area
> > > added
> > > > > by
> > > > > > >> every
> > > > > > >> > > new
> > > > > > >> > > > > > >> metric
> > > > > > >> > > > > > >> > > type
> > > > > > >> > > > > > >> > > > > > today:
> > > > > > >> > > > > > >> > > > > > > > new
> > > > > > >> > > > > > >> > > > > > > > > > > > table,
> > > > > > >> > > > > > >> > > > > > > > > > > > >> SPI method, record
> class,
> > > > model,
> > > > > > >> > > converter,
> > > > > > >> > > > > > >> schema
> > > > > > >> > > > > > >> > > > > > migration.
> > > > > > >> > > > > > >> > > > > > > > > > > > >>
> > > > > > >> > > > > > >> > > > > > > > > > > > >> Most schema details are
> > > > > deferred to
> > > > > > >> the
> > > > > > >> > > > > schema
> > > > > > >> > > > > > >> PR. A
> > > > > > >> > > > > > >> > > few
> > > > > > >> > > > > > >> > > > > > > > specific
> > > > > > >> > > > > > >> > > > > > > > > > > points
> > > > > > >> > > > > > >> > > > > > > > > > > > >> came up:
> > > > > > >> > > > > > >> > > > > > > > > > > > >> •
>  metric_schema_version:
> > > > Yufei
> > > > > > >> > prefers
> > > > > > >> > > > > > dropping
> > > > > > >> > > > > > >> > it,
> > > > > > >> > > > > > >> > > > > since
> > > > > > >> > > > > > >> > > > > > > > there
> > > > > > >> > > > > > >> > > > > > > > > is
> > > > > > >> > > > > > >> > > > > > > > > > > no
> > > > > > >> > > > > > >> > > > > > > > > > > > >> spec-level concept of
> > > metrics
> > > > > > >> > versioning
> > > > > > >> > > > > today
> > > > > > >> > > > > > >> and
> > > > > > >> > > > > > >> > it
> > > > > > >> > > > > > >> > > is
> > > > > > >> > > > > > >> > > > > > hard
> > > > > > >> > > > > > >> > > > > > > to
> > > > > > >> > > > > > >> > > > > > > > > > > define
> > > > > > >> > > > > > >> > > > > > > > > > > > >> unilaterally. Robert
> prefers
> > > > > keeping
> > > > > > >> > it,
> > > > > > >> > > > > given
> > > > > > >> > > > > > >> IRC
> > > > > > >> > > > > > >> > v2
> > > > > > >> > > > > > >> > > is
> > > > > > >> > > > > > >> > > > > > > coming
> > > > > > >> > > > > > >> > > > > > > > > and
> > > > > > >> > > > > > >> > > > > > > > > > > the
> > > > > > >> > > > > > >> > > > > > > > > > > > >> schema should be
> considered
> > > > > against
> > > > > > >> its
> > > > > > >> > > > > likely
> > > > > > >> > > > > > >> > shape;
> > > > > > >> > > > > > >> > > > > Robert
> > > > > > >> > > > > > >> > > > > > > > also
> > > > > > >> > > > > > >> > > > > > > > > > > raised
> > > > > > >> > > > > > >> > > > > > > > > > > > >> how to differentiate
> various
> > > > > payload
> > > > > > >> > > formats
> > > > > > >> > > > > if
> > > > > > >> > > > > > >> any.
> > > > > > >> > > > > > >> > > > EJ's
> > > > > > >> > > > > > >> > > > > > read
> > > > > > >> > > > > > >> > > > > > > > is
> > > > > > >> > > > > > >> > > > > > > > > > that
> > > > > > >> > > > > > >> > > > > > > > > > > > >> this
> > > > > > >> > > > > > >> > > > > > > > > > > > >> is a two-way-door
> decision.
> > > We
> > > > > can
> > > > > > >> > start
> > > > > > >> > > > > > without
> > > > > > >> > > > > > >> the
> > > > > > >> > > > > > >> > > > > field,
> > > > > > >> > > > > > >> > > > > > > and
> > > > > > >> > > > > > >> > > > > > > > if
> > > > > > >> > > > > > >> > > > > > > > > > IRC
> > > > > > >> > > > > > >> > > > > > > > > > > > v2
> > > > > > >> > > > > > >> > > > > > > > > > > > >> changes the shape we
> would
> > > > > likely
> > > > > > >> roll
> > > > > > >> > a
> > > > > > >> > > > > > >> > corresponding
> > > > > > >> > > > > > >> > > > new
> > > > > > >> > > > > > >> > > > > > > > schema
> > > > > > >> > > > > > >> > > > > > > > > > > > anyway,
> > > > > > >> > > > > > >> > > > > > > > > > > > >> which is not
> particularly
> > > > > costly.
> > > > > > >> > > > > > >> > > > > > > > > > > > >> •   Payload format:
> Robert
> > > > > pointed
> > > > > > >> out
> > > > > > >> > > that
> > > > > > >> > > > > > >> future
> > > > > > >> > > > > > >> > > > formats
> > > > > > >> > > > > > >> > > > > > > > beyond
> > > > > > >> > > > > > >> > > > > > > > > > JSON
> > > > > > >> > > > > > >> > > > > > > > > > > > may
> > > > > > >> > > > > > >> > > > > > > > > > > > >> be worth supporting. The
> > > exact
> > > > > > >> shape is
> > > > > > >> > > > > > deferred
> > > > > > >> > > > > > >> to
> > > > > > >> > > > > > >> > > the
> > > > > > >> > > > > > >> > > > > > schema
> > > > > > >> > > > > > >> > > > > > > > > > > > discussion.
> > > > > > >> > > > > > >> > > > > > > > > > > > >> •   Partition strategy:
> > > Anand
> > > > > > >> suggested
> > > > > > >> > > > > monthly
> > > > > > >> > > > > > >> > > > > partitioning
> > > > > > >> > > > > > >> > > > > > > > based
> > > > > > >> > > > > > >> > > > > > > > > > on
> > > > > > >> > > > > > >> > > > > > > > > > > > his
> > > > > > >> > > > > > >> > > > > > > > > > > > >> experience as
> potentially
> > > > > helpful at
> > > > > > >> > > scale.
> > > > > > >> > > > > > >> > > > > > > > > > > > >>
> > > > > > >> > > > > > >> > > > > > > > > > > > >> *Topic 2 — Where metrics
> > > > > ingestion
> > > > > > >> and
> > > > > > >> > > > > storage
> > > > > > >> > > > > > >> > belong*
> > > > > > >> > > > > > >> > > > > > > > > > > > >> Idea-level alignment
> that
> > > > > metrics
> > > > > > >> > should
> > > > > > >> > > be a
> > > > > > >> > > > > > >> > > separated
> > > > > > >> > > > > > >> > > > > SPI
> > > > > > >> > > > > > >> > > > > > > from
> > > > > > >> > > > > > >> > > > > > > > > the
> > > > > > >> > > > > > >> > > > > > > > > > > > >> entity
> > > > > > >> > > > > > >> > > > > > > > > > > > >> persistence stack. Two
> > > reasons
> > > > > > >> > surfaced:
> > > > > > >> > > (a)
> > > > > > >> > > > > > >> > workloads
> > > > > > >> > > > > > >> > > > and
> > > > > > >> > > > > > >> > > > > > > > > > capability
> > > > > > >> > > > > > >> > > > > > > > > > > > >> requirements diverge
> enough
> > > > that
> > > > > > >> > coupling
> > > > > > >> > > > > them
> > > > > > >> > > > > > >> > creates
> > > > > > >> > > > > > >> > > > > > > > artificial
> > > > > > >> > > > > > >> > > > > > > > > > > > >> constraints, and (b)
> admin
> > > > > > >> experience
> > > > > > >> > > > > improves
> > > > > > >> > > > > > >> when
> > > > > > >> > > > > > >> > > > > metrics
> > > > > > >> > > > > > >> > > > > > > has
> > > > > > >> > > > > > >> > > > > > > > > its
> > > > > > >> > > > > > >> > > > > > > > > > > own
> > > > > > >> > > > > > >> > > > > > > > > > > > >> bootstrap, retention,
> and
> > > > > lifecycle.
> > > > > > >> > > Dmitri
> > > > > > >> > > > > > noted
> > > > > > >> > > > > > >> > > > Polaris
> > > > > > >> > > > > > >> > > > > > > being
> > > > > > >> > > > > > >> > > > > > > > a
> > > > > > >> > > > > > >> > > > > > > > > > > > platform
> > > > > > >> > > > > > >> > > > > > > > > > > > >> should have the
> flexibility
> > > to
> > > > > > >> support
> > > > > > >> > > > > > different
> > > > > > >> > > > > > >> > > > > persistence
> > > > > > >> > > > > > >> > > > > > > > > > backends
> > > > > > >> > > > > > >> > > > > > > > > > > > per
> > > > > > >> > > > > > >> > > > > > > > > > > > >> concern, and pointed to
> a
> > > > > concrete
> > > > > > >> next
> > > > > > >> > > step
> > > > > > >> > > > > of
> > > > > > >> > > > > > >> > > > separating
> > > > > > >> > > > > > >> > > > > > the
> > > > > > >> > > > > > >> > > > > > > > > JDBC
> > > > > > >> > > > > > >> > > > > > > > > > > > >> bootstrap for metrics
> from
> > > the
> > > > > > >> > metastore
> > > > > > >> > > > > > >> bootstrap.
> > > > > > >> > > > > > >> > > > Robert
> > > > > > >> > > > > > >> > > > > > > > > proposed
> > > > > > >> > > > > > >> > > > > > > > > > an
> > > > > > >> > > > > > >> > > > > > > > > > > > >> additional UX extension:
> > > > detect
> > > > > an
> > > > > > >> > > > > > unbootstrapped
> > > > > > >> > > > > > >> > > > metrics
> > > > > > >> > > > > > >> > > > > > > store
> > > > > > >> > > > > > >> > > > > > > > on
> > > > > > >> > > > > > >> > > > > > > > > > > first
> > > > > > >> > > > > > >> > > > > > > > > > > > >> use and auto-bootstrap
> > > rather
> > > > > than
> > > > > > >> > > requiring
> > > > > > >> > > > > an
> > > > > > >> > > > > > >> > > explicit
> > > > > > >> > > > > > >> > > > > > > manual
> > > > > > >> > > > > > >> > > > > > > > > > > > bootstrap
> > > > > > >> > > > > > >> > > > > > > > > > > > >> step.
> > > > > > >> > > > > > >> > > > > > > > > > > > >> The meeting also
> confirmed
> > > > that
> > > > > > >> Polaris
> > > > > > >> > > > > metrics
> > > > > > >> > > > > > >> can
> > > > > > >> > > > > > >> > > > start
> > > > > > >> > > > > > >> > > > > > > small
> > > > > > >> > > > > > >> > > > > > > > > and
> > > > > > >> > > > > > >> > > > > > > > > > > stay
> > > > > > >> > > > > > >> > > > > > > > > > > > >> Iceberg-focused. Naming
> and
> > > > > > >> persistence
> > > > > > >> > > > > schema
> > > > > > >> > > > > > >> can
> > > > > > >> > > > > > >> > > lean
> > > > > > >> > > > > > >> > > > > > > > > > > > Iceberg-specific.
> > > > > > >> > > > > > >> > > > > > > > > > > > >> If a future expansion to
> > > > > > >> generic-table
> > > > > > >> > > > > metrics
> > > > > > >> > > > > > or
> > > > > > >> > > > > > >> > > > > > operational
> > > > > > >> > > > > > >> > > > > > > > > > metrics
> > > > > > >> > > > > > >> > > > > > > > > > > > >> arrives, an abstraction
> > > layer
> > > > > can be
> > > > > > >> > > built on
> > > > > > >> > > > > > >> top of
> > > > > > >> > > > > > >> > > the
> > > > > > >> > > > > > >> > > > > > > Iceberg
> > > > > > >> > > > > > >> > > > > > > > > > > metrics
> > > > > > >> > > > > > >> > > > > > > > > > > > >> reporter at that point.
> > > Robert
> > > > > > >> remains
> > > > > > >> > > on the
> > > > > > >> > > > > > >> fence
> > > > > > >> > > > > > >> > > and
> > > > > > >> > > > > > >> > > > > > would
> > > > > > >> > > > > > >> > > > > > > > > prefer
> > > > > > >> > > > > > >> > > > > > > > > > > > >> something more generic
> but
> > > did
> > > > > not
> > > > > > >> > block
> > > > > > >> > > the
> > > > > > >> > > > > > >> > > direction;
> > > > > > >> > > > > > >> > > > > > > Dmitri's
> > > > > > >> > > > > > >> > > > > > > > > > read
> > > > > > >> > > > > > >> > > > > > > > > > > > was
> > > > > > >> > > > > > >> > > > > > > > > > > > >> that the proposed
> framework
> > > > > already
> > > > > > >> has
> > > > > > >> > > > > enough
> > > > > > >> > > > > > >> > > > flexibility
> > > > > > >> > > > > > >> > > > > > to
> > > > > > >> > > > > > >> > > > > > > > > absorb
> > > > > > >> > > > > > >> > > > > > > > > > > > >> future
> > > > > > >> > > > > > >> > > > > > > > > > > > >> expansion.
> > > > > > >> > > > > > >> > > > > > > > > > > > >>
> > > > > > >> > > > > > >> > > > > > > > > > > > >> The Trade-offs and
> Proposed
> > > > > > >> structure
> > > > > > >> > > > > sections
> > > > > > >> > > > > > in
> > > > > > >> > > > > > >> > the
> > > > > > >> > > > > > >> > > > doc
> > > > > > >> > > > > > >> > > > > > were
> > > > > > >> > > > > > >> > > > > > > > not
> > > > > > >> > > > > > >> > > > > > > > > > > > >> reviewed
> > > > > > >> > > > > > >> > > > > > > > > > > > >> in detail. They remain
> open
> > > > for
> > > > > > >> either
> > > > > > >> > > the
> > > > > > >> > > > > next
> > > > > > >> > > > > > >> sync
> > > > > > >> > > > > > >> > > or
> > > > > > >> > > > > > >> > > > PR
> > > > > > >> > > > > > >> > > > > > > > review.
> > > > > > >> > > > > > >> > > > > > > > > > > > >>
> > > > > > >> > > > > > >> > > > > > > > > > > > >> *Cross-cutting
> alignment —
> > > > > > >> > > battery-included
> > > > > > >> > > > > > plus
> > > > > > >> > > > > > >> > > > > pluggable*
> > > > > > >> > > > > > >> > > > > > > > > > > > >> A common philosophy
> emerged
> > > > > from the
> > > > > > >> > > > > > discussion.
> > > > > > >> > > > > > >> EJ
> > > > > > >> > > > > > >> > > > > > summarized
> > > > > > >> > > > > > >> > > > > > > > it
> > > > > > >> > > > > > >> > > > > > > > > > as:
> > > > > > >> > > > > > >> > > > > > > > > > > > >> Polaris should provide a
> > > > > > >> > > battery-included UX
> > > > > > >> > > > > > for
> > > > > > >> > > > > > >> > > > beginners
> > > > > > >> > > > > > >> > > > > > and
> > > > > > >> > > > > > >> > > > > > > > the
> > > > > > >> > > > > > >> > > > > > > > > > > > >> flexibility for advanced
> > > users
> > > > > to
> > > > > > >> swap
> > > > > > >> > > the
> > > > > > >> > > > > > >> included
> > > > > > >> > > > > > >> > > > > battery
> > > > > > >> > > > > > >> > > > > > > for
> > > > > > >> > > > > > >> > > > > > > > > > > > something
> > > > > > >> > > > > > >> > > > > > > > > > > > >> more powerful or
> tailored to
> > > > > their
> > > > > > >> use
> > > > > > >> > > case.
> > > > > > >> > > > > > The
> > > > > > >> > > > > > >> SPI
> > > > > > >> > > > > > >> > > > > design
> > > > > > >> > > > > > >> > > > > > > > needs
> > > > > > >> > > > > > >> > > > > > > > > to
> > > > > > >> > > > > > >> > > > > > > > > > > > >> enable
> > > > > > >> > > > > > >> > > > > > > > > > > > >> both.
> > > > > > >> > > > > > >> > > > > > > > > > > > >>
> > > > > > >> > > > > > >> > > > > > > > > > > > >> The inputs that shaped
> this
> > > > > framing:
> > > > > > >> > > > > > >> > > > > > > > > > > > >> •   Anand described how
> his
> > > > team
> > > > > > >> uses
> > > > > > >> > the
> > > > > > >> > > > > > current
> > > > > > >> > > > > > >> > > > metrics
> > > > > > >> > > > > > >> > > > > > > > > > persistence
> > > > > > >> > > > > > >> > > > > > > > > > > > >> (three metrics
> consumers in
> > > > > v1.4).
> > > > > > >> > > > > > >> > > > > > > > > > > > >> •   Yufei raised
> Grafana and
> > > > > > >> dashboard
> > > > > > >> > > > > > >> integrations
> > > > > > >> > > > > > >> > > as a
> > > > > > >> > > > > > >> > > > > > > > > destination
> > > > > > >> > > > > > >> > > > > > > > > > > use
> > > > > > >> > > > > > >> > > > > > > > > > > > >> case beyond the default.
> > > > > > >> > > > > > >> > > > > > > > > > > > >> •   Robert called out
> that
> > > the
> > > > > > >> current
> > > > > > >> > > design
> > > > > > >> > > > > > is
> > > > > > >> > > > > > >> > more
> > > > > > >> > > > > > >> > > > > > > > > JDBC-focused.
> > > > > > >> > > > > > >> > > > > > > > > > > > >>
> > > > > > >> > > > > > >> > > > > > > > > > > > >> Two concrete instances:
> > > > > > >> > > > > > >> > > > > > > > > > > > >> •   Async metrics
> intake:
> > > > > Yufei's
> > > > > > >> > initial
> > > > > > >> > > > > > >> position
> > > > > > >> > > > > > >> > was
> > > > > > >> > > > > > >> > > > > that
> > > > > > >> > > > > > >> > > > > > > > async
> > > > > > >> > > > > > >> > > > > > > > > > > should
> > > > > > >> > > > > > >> > > > > > > > > > > > >> largely live on the
> producer
> > > > > side
> > > > > > >> and
> > > > > > >> > > there
> > > > > > >> > > > > is
> > > > > > >> > > > > > >> not
> > > > > > >> > > > > > >> > > much
> > > > > > >> > > > > > >> > > > > > > Polaris
> > > > > > >> > > > > > >> > > > > > > > > can
> > > > > > >> > > > > > >> > > > > > > > > > > do.
> > > > > > >> > > > > > >> > > > > > > > > > > > >> Robert suggested a
> > > > Polaris-side
> > > > > > >> default
> > > > > > >> > > is
> > > > > > >> > > > > > doable
> > > > > > >> > > > > > >> > via
> > > > > > >> > > > > > >> > > > > > Vert.x.
> > > > > > >> > > > > > >> > > > > > > > > Dmitri
> > > > > > >> > > > > > >> > > > > > > > > > > > >> agreed
> > > > > > >> > > > > > >> > > > > > > > > > > > >> the direction is worth
> > > > > exploring.
> > > > > > >> The
> > > > > > >> > > meeting
> > > > > > >> > > > > > >> > > converged
> > > > > > >> > > > > > >> > > > > on a
> > > > > > >> > > > > > >> > > > > > > > > > > > >> battery-included default
> > > > (likely
> > > > > > >> > > > > Vert.x-backed)
> > > > > > >> > > > > > >> with
> > > > > > >> > > > > > >> > > an
> > > > > > >> > > > > > >> > > > > SPI
> > > > > > >> > > > > > >> > > > > > > > shape
> > > > > > >> > > > > > >> > > > > > > > > > that
> > > > > > >> > > > > > >> > > > > > > > > > > > >> lets
> > > > > > >> > > > > > >> > > > > > > > > > > > >> power users route to a
> more
> > > > > scalable
> > > > > > >> > > backend
> > > > > > >> > > > > > >> > > (k8s-hosted
> > > > > > >> > > > > > >> > > > > > > queue,
> > > > > > >> > > > > > >> > > > > > > > > AWS
> > > > > > >> > > > > > >> > > > > > > > > > > SQS,
> > > > > > >> > > > > > >> > > > > > > > > > > > >> etc.).
> > > > > > >> > > > > > >> > > > > > > > > > > > >> •   Pluggable
> destinations:
> > > > > > >> combining
> > > > > > >> > > Yufei's
> > > > > > >> > > > > > >> > > dashboard
> > > > > > >> > > > > > >> > > > > use
> > > > > > >> > > > > > >> > > > > > > case
> > > > > > >> > > > > > >> > > > > > > > > > with
> > > > > > >> > > > > > >> > > > > > > > > > > > >> Robert's JDBC-focused
> > > > call-out,
> > > > > the
> > > > > > >> > > meeting
> > > > > > >> > > > > > >> agreed
> > > > > > >> > > > > > >> > the
> > > > > > >> > > > > > >> > > > SPI
> > > > > > >> > > > > > >> > > > > > > > should
> > > > > > >> > > > > > >> > > > > > > > > be
> > > > > > >> > > > > > >> > > > > > > > > > > > >> structured for multiple
> > > sinks
> > > > so
> > > > > > >> > > integrations
> > > > > > >> > > > > > >> become
> > > > > > >> > > > > > >> > > > impl
> > > > > > >> > > > > > >> > > > > > > > choices
> > > > > > >> > > > > > >> > > > > > > > > > > rather
> > > > > > >> > > > > > >> > > > > > > > > > > > >> than architectural
> changes.
> > > > > > >> > > > > > >> > > > > > > > > > > > >>
> > > > > > >> > > > > > >> > > > > > > > > > > > >> The battery-included
> default
> > > > is
> > > > > most
> > > > > > >> > > likely
> > > > > > >> > > > > to
> > > > > > >> > > > > > >> use
> > > > > > >> > > > > > >> > the
> > > > > > >> > > > > > >> > > > > > > existing
> > > > > > >> > > > > > >> > > > > > > > > > > > >> JDBC-backed
> > > > > > >> > > > > > >> > > > > > > > > > > > >> approach.
> > > > > > >> > > > > > >> > > > > > > > > > > > >>
> > > > > > >> > > > > > >> > > > > > > > > > > > >> *Direction (idea-level
> > > > > alignment)*
> > > > > > >> > > > > > >> > > > > > > > > > > > >> •   Single
> metrics_report
> > > > table
> > > > > > >> > > consolidating
> > > > > > >> > > > > > >> > per-type
> > > > > > >> > > > > > >> > > > > > > metrics,
> > > > > > >> > > > > > >> > > > > > > > > > > > replacing
> > > > > > >> > > > > > >> > > > > > > > > > > > >> scan_metrics_report and
> > > > > > >> > > commit_metrics_report
> > > > > > >> > > > > > >> > > > > > > > > > > > >> •   Iceberg-focused
> naming
> > > and
> > > > > > >> schema
> > > > > > >> > for
> > > > > > >> > > > > now,
> > > > > > >> > > > > > >> > revisit
> > > > > > >> > > > > > >> > > > if
> > > > > > >> > > > > > >> > > > > > > > > > > generic-table
> > > > > > >> > > > > > >> > > > > > > > > > > > or
> > > > > > >> > > > > > >> > > > > > > > > > > > >> operational metrics
> arrive
> > > > > > >> > > > > > >> > > > > > > > > > > > >> •   Metrics persistence
> as a
> > > > > > >> separated
> > > > > > >> > > SPI,
> > > > > > >> > > > > not
> > > > > > >> > > > > > >> on
> > > > > > >> > > > > > >> > > > > > > > BasePersistence
> > > > > > >> > > > > > >> > > > > > > > > > > > >> •   Bootstrap path
> separated
> > > > for
> > > > > > >> > metrics,
> > > > > > >> > > > > > >> > independent
> > > > > > >> > > > > > >> > > of
> > > > > > >> > > > > > >> > > > > > > > metastore
> > > > > > >> > > > > > >> > > > > > > > > > > > >> bootstrap
> > > > > > >> > > > > > >> > > > > > > > > > > > >> •   "Battery-included
> plus
> > > > > > >> pluggable"
> > > > > > >> > as
> > > > > > >> > > the
> > > > > > >> > > > > > SPI
> > > > > > >> > > > > > >> > > design
> > > > > > >> > > > > > >> > > > > > > > philosophy
> > > > > > >> > > > > > >> > > > > > > > > > > > >>
> > > > > > >> > > > > > >> > > > > > > > > > > > >> *Open items*
> > > > > > >> > > > > > >> > > > > > > > > > > > >> •   Schema details:
> > > > > > >> > > metric_schema_version,
> > > > > > >> > > > > > >> payload
> > > > > > >> > > > > > >> > > > format,
> > > > > > >> > > > > > >> > > > > > IRC
> > > > > > >> > > > > > >> > > > > > > > v2
> > > > > > >> > > > > > >> > > > > > > > > > > > >> forward-compat shape
> > > > > > >> > > > > > >> > > > > > > > > > > > >> •   SPI design details —
> > > full
> > > > > review
> > > > > > >> > > either
> > > > > > >> > > > > in
> > > > > > >> > > > > > >> the
> > > > > > >> > > > > > >> > > next
> > > > > > >> > > > > > >> > > > > sync
> > > > > > >> > > > > > >> > > > > > > or
> > > > > > >> > > > > > >> > > > > > > > in
> > > > > > >> > > > > > >> > > > > > > > > > the
> > > > > > >> > > > > > >> > > > > > > > > > > > >> corresponding PR
> > > > > > >> > > > > > >> > > > > > > > > > > > >> •   Schema refactor PR
> > > > ownership
> > > > > > >> > > > > > >> > > > > > > > > > > > >>
> > > > > > >> > > > > > >> > > > > > > > > > > > >> *Action items*
> > > > > > >> > > > > > >> > > > > > > > > > > > >> •   EJ to take a first
> stab
> > > at
> > > > > the
> > > > > > >> SPI
> > > > > > >> > > design
> > > > > > >> > > > > > and
> > > > > > >> > > > > > >> > > > > > potentially
> > > > > > >> > > > > > >> > > > > > > > > > partner
> > > > > > >> > > > > > >> > > > > > > > > > > > with
> > > > > > >> > > > > > >> > > > > > > > > > > > >> Anand to incorporate the
> > > > lessons
> > > > > > >> > learned
> > > > > > >> > > from
> > > > > > >> > > > > > the
> > > > > > >> > > > > > >> > > > existing
> > > > > > >> > > > > > >> > > > > > > > > reporter
> > > > > > >> > > > > > >> > > > > > > > > > > and
> > > > > > >> > > > > > >> > > > > > > > > > > > >> persistence work.
> > > > > > >> > > > > > >> > > > > > > > > > > > >> •   Schema refactor PR
> > > > > ownership is
> > > > > > >> not
> > > > > > >> > > yet
> > > > > > >> > > > > > >> decided.
> > > > > > >> > > > > > >> > > If
> > > > > > >> > > > > > >> > > > > > anyone
> > > > > > >> > > > > > >> > > > > > > > is
> > > > > > >> > > > > > >> > > > > > > > > > > > >> interested in driving
> it,
> > > > reply
> > > > > on
> > > > > > >> this
> > > > > > >> > > > > thread.
> > > > > > >> > > > > > >> > > > > > > > > > > > >> •   JB to schedule the
> next
> > > > > sync,
> > > > > > >> > > tentatively
> > > > > > >> > > > > > in
> > > > > > >> > > > > > >> two
> > > > > > >> > > > > > >> > > > > weeks.
> > > > > > >> > > > > > >> > > > > > > > > > > > >>
> > > > > > >> > > > > > >> > > > > > > > > > > > >> -ej
> > > > > > >> > > > > > >> > > > > > > > > > > > >>
> > > > > > >> > > > > > >> > > > > > > > > > > > >> On Mon, Apr 27, 2026 at
> > > > 3:07 PM
> > > > > EJ
> > > > > > >> > Wang <
> > > > > > >> > > > > > >> > > > > > > > > > >
> ej.wang.dev.interest@gmail.com
> > > > > > >> > > > > > >> > > > > > > > > > > > >
> > > > > > >> > > > > > >> > > > > > > > > > > > >> wrote:
> > > > > > >> > > > > > >> > > > > > > > > > > > >>
> > > > > > >> > > > > > >> > > > > > > > > > > > >> > Thanks Yufei for the
> +1.
> > > > > > >> > > > > > >> > > > > > > > > > > > >> >
> > > > > > >> > > > > > >> > > > > > > > > > > > >> > JB, could you help
> add a
> > > > > biweekly
> > > > > > >> > > metrics
> > > > > > >> > > > > > >> > > architecture
> > > > > > >> > > > > > >> > > > > > sync
> > > > > > >> > > > > > >> > > > > > > to
> > > > > > >> > > > > > >> > > > > > > > > the
> > > > > > >> > > > > > >> > > > > > > > > > > > >> Polaris
> > > > > > >> > > > > > >> > > > > > > > > > > > >> > community calendar?
> I'm
> > > > > thinking
> > > > > > >> > > Thursdays
> > > > > > >> > > > > at
> > > > > > >> > > > > > >> > 9-10am
> > > > > > >> > > > > > >> > > > PT,
> > > > > > >> > > > > > >> > > > > > on
> > > > > > >> > > > > > >> > > > > > > > the
> > > > > > >> > > > > > >> > > > > > > > > > > > >> off-weeks
> > > > > > >> > > > > > >> > > > > > > > > > > > >> > from the community
> meeting
> > > > > > >> (starting
> > > > > > >> > > May
> > > > > > >> > > > > 7),
> > > > > > >> > > > > > 60
> > > > > > >> > > > > > >> > > > minutes.
> > > > > > >> > > > > > >> > > > > > > > > > > > >> >
> > > > > > >> > > > > > >> > > > > > > > > > > > >> > Here's a rough agenda
> to
> > > > work
> > > > > > >> through
> > > > > > >> > > over
> > > > > > >> > > > > > the
> > > > > > >> > > > > > >> > first
> > > > > > >> > > > > > >> > > > few
> > > > > > >> > > > > > >> > > > > > > > > sessions,
> > > > > > >> > > > > > >> > > > > > > > > > > > >> grouped
> > > > > > >> > > > > > >> > > > > > > > > > > > >> > by priority:
> > > > > > >> > > > > > >> > > > > > > > > > > > >> >
> > > > > > >> > > > > > >> > > > > > > > > > > > >> > *First: foundational
> > > > > direction*
> > > > > > >> > > > > > >> > > > > > > > > > > > >> >
> > > > > > >> > > > > > >> > > > > > > > > > > > >> > 1.
> MetricsPersistence:
> > > > public
> > > > > > >> SPI or
> > > > > > >> > > > > > internal
> > > > > > >> > > > > > >> > > > > > > implementation
> > > > > > >> > > > > > >> > > > > > > > > > > detail?
> > > > > > >> > > > > > >> > > > > > > > > > > > >> >    •   Marked @Beta,
> > > javadoc
> > > > > calls
> > > > > > >> > it a
> > > > > > >> > > > > > >> "Service
> > > > > > >> > > > > > >> > > > > Provider
> > > > > > >> > > > > > >> > > > > > > > > > > Interface",
> > > > > > >> > > > > > >> > > > > > > > > > > > >> but
> > > > > > >> > > > > > >> > > > > > > > > > > > >> > only one consumer
> > > > > > >> > > > > (JdbcBasePersistenceImpl),
> > > > > > >> > > > > > >> lives
> > > > > > >> > > > > > >> > > on
> > > > > > >> > > > > > >> > > > > > > > > > > BasePersistence.
> > > > > > >> > > > > > >> > > > > > > > > > > > >> If
> > > > > > >> > > > > > >> > > > > > > > > > > > >> > demoted to a private
> > > helper
> > > > > > >> inside a
> > > > > > >> > > > > > persisting
> > > > > > >> > > > > > >> > > > reporter
> > > > > > >> > > > > > >> > > > > > > impl,
> > > > > > >> > > > > > >> > > > > > > > > > most
> > > > > > >> > > > > > >> > > > > > > > > > > > >> > downstream design
> > > decisions
> > > > > become
> > > > > > >> > > > > > >> implementation
> > > > > > >> > > > > > >> > > > > details
> > > > > > >> > > > > > >> > > > > > > > rather
> > > > > > >> > > > > > >> > > > > > > > > > > than
> > > > > > >> > > > > > >> > > > > > > > > > > > >> > contract questions.
> > > > > > >> > > > > > >> > > > > > > > > > > > >> >
> > > > > > >> > > > > > >> > > > > > > > > > > > >> > 2.  Persistence schema
> > > > > redesign
> > > > > > >> > > > > > >> > > > > > > > > > > > >> >    •   Current
> two-table
> > > > > layout
> > > > > > >> > > > > > >> > > (scan_metrics_report,
> > > > > > >> > > > > > >> > > > > > > > > > > > >> > commit_metrics_report)
> > > with
> > > > > ~25
> > > > > > >> > > flattened
> > > > > > >> > > > > > >> columns
> > > > > > >> > > > > > >> > > > each.
> > > > > > >> > > > > > >> > > > > > > Every
> > > > > > >> > > > > > >> > > > > > > > > new
> > > > > > >> > > > > > >> > > > > > > > > > > > metric
> > > > > > >> > > > > > >> > > > > > > > > > > > >> > type requires a new
> table,
> > > > SPI
> > > > > > >> > method,
> > > > > > >> > > > > record
> > > > > > >> > > > > > >> > class,
> > > > > > >> > > > > > >> > > > > > model,
> > > > > > >> > > > > > >> > > > > > > > > > > converter,
> > > > > > >> > > > > > >> > > > > > > > > > > > >> and
> > > > > > >> > > > > > >> > > > > > > > > > > > >> > schema migration.
> > > Direction
> > > > to
> > > > > > >> > explore:
> > > > > > >> > > > > > single
> > > > > > >> > > > > > >> > table
> > > > > > >> > > > > > >> > > > > with
> > > > > > >> > > > > > >> > > > > > > > > > > metric_type
> > > > > > >> > > > > > >> > > > > > > > > > > > >> enum,
> > > > > > >> > > > > > >> > > > > > > > > > > > >> > schema_version, and
> JSON
> > > > > payload
> > > > > > >> > > column.
> > > > > > >> > > > > > >> > > > > > > > > > > > >> >
> > > > > > >> > > > > > >> > > > > > > > > > > > >> > *Second: design
> details
> > > once
> > > > > > >> > direction
> > > > > > >> > > is
> > > > > > >> > > > > > set*
> > > > > > >> > > > > > >> > > > > > > > > > > > >> >
> > > > > > >> > > > > > >> > > > > > > > > > > > >> > 3.  Partition key
> strategy
> > > > > > >> > > > > > >> > > > > > > > > > > > >> >    •   Single-table
> design
> > > > > means
> > > > > > >> scan
> > > > > > >> > > > > metrics
> > > > > > >> > > > > > >> at
> > > > > > >> > > > > > >> > > scale
> > > > > > >> > > > > > >> > > > > > will
> > > > > > >> > > > > > >> > > > > > > > have
> > > > > > >> > > > > > >> > > > > > > > > > > high
> > > > > > >> > > > > > >> > > > > > > > > > > > >> > write concurrency per
> > > table.
> > > > > > >> Schema
> > > > > > >> > > needs
> > > > > > >> > > > > to
> > > > > > >> > > > > > >> > expose
> > > > > > >> > > > > > >> > > > > enough
> > > > > > >> > > > > > >> > > > > > > > > > structure
> > > > > > >> > > > > > >> > > > > > > > > > > > for
> > > > > > >> > > > > > >> > > > > > > > > > > > >> > backends to shard by
> > > entity
> > > > or
> > > > > > >> time
> > > > > > >> > > range.
> > > > > > >> > > > > > >> > > > > > > > > > > > >> >
> > > > > > >> > > > > > >> > > > > > > > > > > > >> > 4.  Read/write path
> > > > > consistency
> > > > > > >> > > > > > >> > > > > > > > > > > > >> >    •   Writes go
> through
> > > > > > >> > > > > > PolarisMetricsManager
> > > > > > >> > > > > > >> on
> > > > > > >> > > > > > >> > > > > > > > > > MetaStoreManager.
> > > > > > >> > > > > > >> > > > > > > > > > > > >> Reads
> > > > > > >> > > > > > >> > > > > > > > > > > > >> > bypass
> MetaStoreManager
> > > and
> > > > go
> > > > > > >> > > straight to
> > > > > > >> > > > > > >> > > > > > BasePersistence,
> > > > > > >> > > > > > >> > > > > > > > > > > excluding
> > > > > > >> > > > > > >> > > > > > > > > > > > >> > non-JDBC backends
> from the
> > > > > read
> > > > > > >> API.
> > > > > > >> > > > > > >> > > > > > > > > > > > >> >
> > > > > > >> > > > > > >> > > > > > > > > > > > >> > *Third: cleanup and
> > > > alignment*
> > > > > > >> > > > > > >> > > > > > > > > > > > >> >
> > > > > > >> > > > > > >> > > > > > > > > > > > >> > 5.
> PolarisMetricsReporter
> > > > > naming
> > > > > > >> > > > > > >> > > > > > > > > > > > >> >    •   Only handles
> IRC
> > > > > > >> > > > > > >> (ScanReport/CommitReport),
> > > > > > >> > > > > > >> > > > > doesn't
> > > > > > >> > > > > > >> > > > > > > > cover
> > > > > > >> > > > > > >> > > > > > > > > > > > generic
> > > > > > >> > > > > > >> > > > > > > > > > > > >> > tables or operational
> > > > metrics.
> > > > > > >> Name
> > > > > > >> > is
> > > > > > >> > > > > > broader
> > > > > > >> > > > > > >> > than
> > > > > > >> > > > > > >> > > > > scope.
> > > > > > >> > > > > > >> > > > > > > > > > > > >> >
> > > > > > >> > > > > > >> > > > > > > > > > > > >> > 6.
> PolarisMetricsManager
> > > > > facade
> > > > > > >> > > > > passthrough
> > > > > > >> > > > > > >> > > > > > > > > > > > >> >    •   Entire default
> > > method
> > > > > is
> > > > > > >> > > > > > >> > > > > > > > > > > > >>
> > > > > > >> > callCtx.getMetaStore().writeScanReport().
> > > > > > >> > > > > > >> > > > > > > > > > > > >> > Zero logic, passes
> Level 1
> > > > > > >> straight
> > > > > > >> > > through
> > > > > > >> > > > > > to
> > > > > > >> > > > > > >> > Level
> > > > > > >> > > > > > >> > > > 3.
> > > > > > >> > > > > > >> > > > > > Same
> > > > > > >> > > > > > >> > > > > > > > > > > > >> anti-pattern
> > > > > > >> > > > > > >> > > > > > > > > > > > >> > as
> PolarisEventManager.
> > > > > > >> > > > > > >> > > > > > > > > > > > >> >
> > > > > > >> > > > > > >> > > > > > > > > > > > >> > 7.  Iceberg community
> > > > > alignment
> > > > > > >> > > > > > >> > > > > > > > > > > > >> >    •   Payload-type
> > > > extension
> > > > > > >> needs
> > > > > > >> > > > > > discussion
> > > > > > >> > > > > > >> on
> > > > > > >> > > > > > >> > > > > > > dev@iceberg.
> > > > > > >> > > > > > >> > > > > > > > > > > > >> obelix74's
> > > > > > >> > > > > > >> > > > > > > > > > > > >> > Feb thread got zero
> > > replies.
> > > > > > >> Needs a
> > > > > > >> > > > > > committer
> > > > > > >> > > > > > >> > > voice.
> > > > > > >> > > > > > >> > > > > > > > > > > > >> >
> > > > > > >> > > > > > >> > > > > > > > > > > > >> > Lets confirm
> > > prioritization
> > > > > in the
> > > > > > >> > > first
> > > > > > >> > > > > > >> session.
> > > > > > >> > > > > > >> > > > > > > > > > > > >> >
> > > > > > >> > > > > > >> > > > > > > > > > > > >> > -ej
> > > > > > >> > > > > > >> > > > > > > > > > > > >> >
> > > > > > >> > > > > > >> > > > > > > > > > > > >> > On Tue, Apr 21, 2026
> at
> > > > > 3:18 PM
> > > > > > >> Yufei
> > > > > > >> > > Gu <
> > > > > > >> > > > > > >> > > > > > > > flyrain000@gmail.com>
> > > > > > >> > > > > > >> > > > > > > > > > > > wrote:
> > > > > > >> > > > > > >> > > > > > > > > > > > >> >
> > > > > > >> > > > > > >> > > > > > > > > > > > >> >> Thanks everyone for
> > > > > continuing to
> > > > > > >> > > drive
> > > > > > >> > > > > this
> > > > > > >> > > > > > >> > > > forward. I
> > > > > > >> > > > > > >> > > > > > > agree
> > > > > > >> > > > > > >> > > > > > > > > > that
> > > > > > >> > > > > > >> > > > > > > > > > > > the
> > > > > > >> > > > > > >> > > > > > > > > > > > >> >> problem is getting
> > > complex
> > > > > enough
> > > > > > >> > > that a
> > > > > > >> > > > > > more
> > > > > > >> > > > > > >> > > > > structured
> > > > > > >> > > > > > >> > > > > > > > > > discussion
> > > > > > >> > > > > > >> > > > > > > > > > > > >> would
> > > > > > >> > > > > > >> > > > > > > > > > > > >> >> help.
> > > > > > >> > > > > > >> > > > > > > > > > > > >> >>
> > > > > > >> > > > > > >> > > > > > > > > > > > >> >> +1 on setting up a
> > > biweekly
> > > > > sync
> > > > > > >> for
> > > > > > >> > > the
> > > > > > >> > > > > > >> metrics
> > > > > > >> > > > > > >> > > > > > > > architecture.
> > > > > > >> > > > > > >> > > > > > > > > > I’m
> > > > > > >> > > > > > >> > > > > > > > > > > > >> happy
> > > > > > >> > > > > > >> > > > > > > > > > > > >> >> to
> > > > > > >> > > > > > >> > > > > > > > > > > > >> >> join.
> > > > > > >> > > > > > >> > > > > > > > > > > > >> >>
> > > > > > >> > > > > > >> > > > > > > > > > > > >> >> Yufei
> > > > > > >> > > > > > >> > > > > > > > > > > > >> >>
> > > > > > >> > > > > > >> > > > > > > > > > > > >> >>
> > > > > > >> > > > > > >> > > > > > > > > > > > >> >> On Tue, Apr 21, 2026
> at
> > > > > 2:34 PM
> > > > > > >> EJ
> > > > > > >> > > Wang <
> > > > > > >> > > > > > >> > > > > > > > > > > > >>
> > > > ej.wang.dev.interest@gmail.com>
> > > > > > >> > > > > > >> > > > > > > > > > > > >> >> wrote:
> > > > > > >> > > > > > >> > > > > > > > > > > > >> >>
> > > > > > >> > > > > > >> > > > > > > > > > > > >> >> > Also, I've been
> looking
> > > > > more
> > > > > > >> > > closely at
> > > > > > >> > > > > > the
> > > > > > >> > > > > > >> > > > > > *persistence
> > > > > > >> > > > > > >> > > > > > > > > schema
> > > > > > >> > > > > > >> > > > > > > > > > > in
> > > > > > >> > > > > > >> > > > > > > > > > > > >> the
> > > > > > >> > > > > > >> > > > > > > > > > > > >> >> > current metrics
> work*,
> > > > and
> > > > > I
> > > > > > >> think
> > > > > > >> > > > > > there's a
> > > > > > >> > > > > > >> > > > > structural
> > > > > > >> > > > > > >> > > > > > > > > > rigidity
> > > > > > >> > > > > > >> > > > > > > > > > > > >> problem
> > > > > > >> > > > > > >> > > > > > > > > > > > >> >> > worth raising
> before
> > > the
> > > > > shape
> > > > > > >> > gets
> > > > > > >> > > > > locked
> > > > > > >> > > > > > >> in.
> > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >
> > > > > > >> > > > > > >> > > > > > > > > > > > >> >> > Right now we have
> two
> > > > > separate
> > > > > > >> > > tables
> > > > > > >> > > > > > >> > > > > > > (scan_metrics_report
> > > > > > >> > > > > > >> > > > > > > > > and
> > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >
> commit_metrics_report),
> > > > > each
> > > > > > >> with
> > > > > > >> > > ~25
> > > > > > >> > > > > > >> flattened
> > > > > > >> > > > > > >> > > > > columns
> > > > > > >> > > > > > >> > > > > > > > that
> > > > > > >> > > > > > >> > > > > > > > > > > > directly
> > > > > > >> > > > > > >> > > > > > > > > > > > >> >> > mirror the Iceberg
> > > report
> > > > > > >> fields.
> > > > > > >> > > The
> > > > > > >> > > > > SPI
> > > > > > >> > > > > > >> > follows
> > > > > > >> > > > > > >> > > > the
> > > > > > >> > > > > > >> > > > > > > same
> > > > > > >> > > > > > >> > > > > > > > > > split:
> > > > > > >> > > > > > >> > > > > > > > > > > > >> >> > writeScanReport and
> > > > > > >> > > writeCommitReport as
> > > > > > >> > > > > > >> > separate
> > > > > > >> > > > > > >> > > > > > > methods,
> > > > > > >> > > > > > >> > > > > > > > > with
> > > > > > >> > > > > > >> > > > > > > > > > > > >> per-type
> > > > > > >> > > > > > >> > > > > > > > > > > > >> >> > record classes,
> > > > > converters, and
> > > > > > >> > > model
> > > > > > >> > > > > > >> objects.
> > > > > > >> > > > > > >> > > *The
> > > > > > >> > > > > > >> > > > > > > > practical
> > > > > > >> > > > > > >> > > > > > > > > > > cost:
> > > > > > >> > > > > > >> > > > > > > > > > > > >> >> > adding a new metric
> > > type
> > > > > > >> > > (operational
> > > > > > >> > > > > > >> metrics,
> > > > > > >> > > > > > >> > > for
> > > > > > >> > > > > > >> > > > > > > example)
> > > > > > >> > > > > > >> > > > > > > > > > > > requires
> > > > > > >> > > > > > >> > > > > > > > > > > > >> a
> > > > > > >> > > > > > >> > > > > > > > > > > > >> >> new
> > > > > > >> > > > > > >> > > > > > > > > > > > >> >> > table, a new SPI
> > > method,
> > > > a
> > > > > new
> > > > > > >> > > record
> > > > > > >> > > > > > >> class, a
> > > > > > >> > > > > > >> > > new
> > > > > > >> > > > > > >> > > > > > model
> > > > > > >> > > > > > >> > > > > > > > > > class, a
> > > > > > >> > > > > > >> > > > > > > > > > > > new
> > > > > > >> > > > > > >> > > > > > > > > > > > >> >> > converter branch,
> and a
> > > > > schema
> > > > > > >> > > > > migration*.
> > > > > > >> > > > > > >> > > That's a
> > > > > > >> > > > > > >> > > > > lot
> > > > > > >> > > > > > >> > > > > > > of
> > > > > > >> > > > > > >> > > > > > > > > > > surface
> > > > > > >> > > > > > >> > > > > > > > > > > > >> area
> > > > > > >> > > > > > >> > > > > > > > > > > > >> >> > for what should be
> "one
> > > > > more
> > > > > > >> kind
> > > > > > >> > of
> > > > > > >> > > > > > >> metric."
> > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >
> > > > > > >> > > > > > >> > > > > > > > > > > > >> >> > *My bias* would be
> > > > toward a
> > > > > > >> single
> > > > > > >> > > > > metrics
> > > > > > >> > > > > > >> > table
> > > > > > >> > > > > > >> > > > with
> > > > > > >> > > > > > >> > > > > > *a
> > > > > > >> > > > > > >> > > > > > > > > typed
> > > > > > >> > > > > > >> > > > > > > > > > > JSON
> > > > > > >> > > > > > >> > > > > > > > > > > > >> >> > payload*. Something
> > > like:
> > > > > > >> > > metric_type
> > > > > > >> > > > > > >> (enum),
> > > > > > >> > > > > > >> > > > > > entity_id,
> > > > > > >> > > > > > >> > > > > > > > > > > > >> >> > table_identifier,
> > > > > snapshot_id
> > > > > > >> > > > > (nullable),
> > > > > > >> > > > > > >> > > > > received_ts,
> > > > > > >> > > > > > >> > > > > > > > > > > > >> schema_version,
> > > > > > >> > > > > > >> > > > > > > > > > > > >> >> and
> > > > > > >> > > > > > >> > > > > > > > > > > > >> >> > a payload column
> for
> > > the
> > > > > > >> > > metric-specific
> > > > > > >> > > > > > >> data.
> > > > > > >> > > > > > >> > > The
> > > > > > >> > > > > > >> > > > > > > > > metric_type
> > > > > > >> > > > > > >> > > > > > > > > > +
> > > > > > >> > > > > > >> > > > > > > > > > > > >> >> > schema_version pair
> > > gives
> > > > > us a
> > > > > > >> > > > > > >> > forward-compatible
> > > > > > >> > > > > > >> > > > > > > contract
> > > > > > >> > > > > > >> > > > > > > > > for
> > > > > > >> > > > > > >> > > > > > > > > > > the
> > > > > > >> > > > > > >> > > > > > > > > > > > >> >> payload
> > > > > > >> > > > > > >> > > > > > > > > > > > >> >> > shape. Adding a new
> > > > metric
> > > > > type
> > > > > > >> > > becomes
> > > > > > >> > > > > an
> > > > > > >> > > > > > >> enum
> > > > > > >> > > > > > >> > > > value
> > > > > > >> > > > > > >> > > > > > > and a
> > > > > > >> > > > > > >> > > > > > > > > > > payload
> > > > > > >> > > > > > >> > > > > > > > > > > > >> >> schema,
> > > > > > >> > > > > > >> > > > > > > > > > > > >> >> > not a schema
> migration.
> > > > One
> > > > > > >> thing
> > > > > > >> > I
> > > > > > >> > > > > think
> > > > > > >> > > > > > we
> > > > > > >> > > > > > >> > need
> > > > > > >> > > > > > >> > > > to
> > > > > > >> > > > > > >> > > > > be
> > > > > > >> > > > > > >> > > > > > > > > > > deliberate
> > > > > > >> > > > > > >> > > > > > > > > > > > >> >> about is
> > > > > > >> > > > > > >> > > > > > > > > > > > >> >> > the partition key
> > > design.
> > > > > If
> > > > > > >> all
> > > > > > >> > > metric
> > > > > > >> > > > > > >> types
> > > > > > >> > > > > > >> > > land
> > > > > > >> > > > > > >> > > > in
> > > > > > >> > > > > > >> > > > > > one
> > > > > > >> > > > > > >> > > > > > > > > > table,
> > > > > > >> > > > > > >> > > > > > > > > > > > scan
> > > > > > >> > > > > > >> > > > > > > > > > > > >> >> > metrics at scale
> (high
> > > > > > >> > concurrency,
> > > > > > >> > > high
> > > > > > >> > > > > > >> > > frequency
> > > > > > >> > > > > > >> > > > > > across
> > > > > > >> > > > > > >> > > > > > > > > many
> > > > > > >> > > > > > >> > > > > > > > > > > > >> tables)
> > > > > > >> > > > > > >> > > > > > > > > > > > >> >> > could easily
> create hot
> > > > > > >> > partitions.
> > > > > > >> > > We'd
> > > > > > >> > > > > > >> want
> > > > > > >> > > > > > >> > the
> > > > > > >> > > > > > >> > > > > > > > persistence
> > > > > > >> > > > > > >> > > > > > > > > > > layer
> > > > > > >> > > > > > >> > > > > > > > > > > > >> to
> > > > > > >> > > > > > >> > > > > > > > > > > > >> >> be
> > > > > > >> > > > > > >> > > > > > > > > > > > >> >> > able to shard by
> entity
> > > > or
> > > > > time
> > > > > > >> > > range,
> > > > > > >> > > > > and
> > > > > > >> > > > > > >> that
> > > > > > >> > > > > > >> > > > means
> > > > > > >> > > > > > >> > > > > > the
> > > > > > >> > > > > > >> > > > > > > > > > logical
> > > > > > >> > > > > > >> > > > > > > > > > > > >> schema
> > > > > > >> > > > > > >> > > > > > > > > > > > >> >> > needs to expose
> enough
> > > > > > >> structure
> > > > > > >> > for
> > > > > > >> > > > > > >> backends
> > > > > > >> > > > > > >> > to
> > > > > > >> > > > > > >> > > > > > > partition
> > > > > > >> > > > > > >> > > > > > > > > on.
> > > > > > >> > > > > > >> > > > > > > > > > I
> > > > > > >> > > > > > >> > > > > > > > > > > > >> don't
> > > > > > >> > > > > > >> > > > > > > > > > > > >> >> > think the current
> > > > flattened
> > > > > > >> layout
> > > > > > >> > > gives
> > > > > > >> > > > > > us
> > > > > > >> > > > > > >> > that.
> > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >
> > > > > > >> > > > > > >> > > > > > > > > > > > >> >> > This is getting
> complex
> > > > > enough
> > > > > > >> > that
> > > > > > >> > > I
> > > > > > >> > > > > > don't
> > > > > > >> > > > > > >> > think
> > > > > > >> > > > > > >> > > > > > ad-hoc
> > > > > > >> > > > > > >> > > > > > > > > PR/ML
> > > > > > >> > > > > > >> > > > > > > > > > > > >> threads
> > > > > > >> > > > > > >> > > > > > > > > > > > >> >> > will converge well.
> > > > *Would
> > > > > > >> people
> > > > > > >> > be
> > > > > > >> > > > > open
> > > > > > >> > > > > > >> to a
> > > > > > >> > > > > > >> > > > > biweekly
> > > > > > >> > > > > > >> > > > > > > > sync
> > > > > > >> > > > > > >> > > > > > > > > > for
> > > > > > >> > > > > > >> > > > > > > > > > > > >> metrics
> > > > > > >> > > > > > >> > > > > > > > > > > > >> >> > architecture?* I
> think
> > > 30
> > > > > > >> minutes
> > > > > > >> > > every
> > > > > > >> > > > > > two
> > > > > > >> > > > > > >> > weeks
> > > > > > >> > > > > > >> > > > > with
> > > > > > >> > > > > > >> > > > > > > > > > interested
> > > > > > >> > > > > > >> > > > > > > > > > > > >> >> parties
> > > > > > >> > > > > > >> > > > > > > > > > > > >> >> > would be enough to
> work
> > > > > through
> > > > > > >> > the
> > > > > > >> > > > > > schema,
> > > > > > >> > > > > > >> SPI
> > > > > > >> > > > > > >> > > > > shape,
> > > > > > >> > > > > > >> > > > > > > and
> > > > > > >> > > > > > >> > > > > > > > > read
> > > > > > >> > > > > > >> > > > > > > > > > > API
> > > > > > >> > > > > > >> > > > > > > > > > > > >> >> design
> > > > > > >> > > > > > >> > > > > > > > > > > > >> >> > together. Happy to
> help
> > > > set
> > > > > > >> that
> > > > > > >> > up.
> > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >
> > > > > > >> > > > > > >> > > > > > > > > > > > >> >> > -ej
> > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >
> > > > > > >> > > > > > >> > > > > > > > > > > > >> >> > On Mon, Apr 20,
> 2026 at
> > > > > > >> 2:19 PM EJ
> > > > > > >> > > Wang
> > > > > > >> > > > > <
> > > > > > >> > > > > > >> > > > > > > > > > > > >>
> > > > ej.wang.dev.interest@gmail.com
> > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >
> > > > > > >> > > > > > >> > > > > > > > > > > > >> >> > wrote:
> > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >
> > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >> Reviewed #4115,
> left a
> > > > > > >> comment on
> > > > > > >> > > the
> > > > > > >> > > > > > code
> > > > > > >> > > > > > >> > > > > > organization
> > > > > > >> > > > > > >> > > > > > > > > side.
> > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>
> > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >> One thing stood
> out:
> > > the
> > > > > > >> metrics
> > > > > > >> > > write
> > > > > > >> > > > > > path
> > > > > > >> > > > > > >> > > enters
> > > > > > >> > > > > > >> > > > > > > through
> > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>
> PolarisMetricsManager
> > > on
> > > > > > >> > > > > > MetaStoreManager,
> > > > > > >> > > > > > >> but
> > > > > > >> > > > > > >> > > the
> > > > > > >> > > > > > >> > > > > new
> > > > > > >> > > > > > >> > > > > > > > read
> > > > > > >> > > > > > >> > > > > > > > > > path
> > > > > > >> > > > > > >> > > > > > > > > > > > >> >> bypasses
> > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >> MetaStoreManager
> > > > entirely
> > > > > and
> > > > > > >> > goes
> > > > > > >> > > > > > >> straight to
> > > > > > >> > > > > > >> > > > > > > > > BasePersistence
> > > > > > >> > > > > > >> > > > > > > > > > > via
> > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>
> > > > > callContext.getMetaStore().
> > > > > > >> That
> > > > > > >> > > means
> > > > > > >> > > > > > the
> > > > > > >> > > > > > >> > read
> > > > > > >> > > > > > >> > > > API
> > > > > > >> > > > > > >> > > > > > only
> > > > > > >> > > > > > >> > > > > > > > > works
> > > > > > >> > > > > > >> > > > > > > > > > > for
> > > > > > >> > > > > > >> > > > > > > > > > > > >> >> backends
> > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >> that implement
> > > > > > >> BasePersistence.
> > > > > > >> > > NoSQL
> > > > > > >> > > > > and
> > > > > > >> > > > > > >> > remote
> > > > > > >> > > > > > >> > > > > > > backends
> > > > > > >> > > > > > >> > > > > > > > > > can't
> > > > > > >> > > > > > >> > > > > > > > > > > > >> >> participate.
> > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>
> > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >> Stepping back, I
> think
> > > > the
> > > > > > >> > metrics
> > > > > > >> > > > > > >> subsystem
> > > > > > >> > > > > > >> > is
> > > > > > >> > > > > > >> > > > > > growing
> > > > > > >> > > > > > >> > > > > > > > into
> > > > > > >> > > > > > >> > > > > > > > > > > > >> something
> > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >> real (write +
> read +
> > > > REST
> > > > > API
> > > > > > >> +
> > > > > > >> > > AuthZ +
> > > > > > >> > > > > > >> > > > pagination)
> > > > > > >> > > > > > >> > > > > > *but
> > > > > > >> > > > > > >> > > > > > > > the
> > > > > > >> > > > > > >> > > > > > > > > > > > >> >> persistence
> > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >> side is split
> across
> > > two
> > > > > > >> layers
> > > > > > >> > in
> > > > > > >> > > a
> > > > > > >> > > > > way
> > > > > > >> > > > > > >> > that's
> > > > > > >> > > > > > >> > > > hard
> > > > > > >> > > > > > >> > > > > > to
> > > > > > >> > > > > > >> > > > > > > > > > > extend*. I
> > > > > > >> > > > > > >> > > > > > > > > > > > >> put
> > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >> together two
> diagrams
> > > to
> > > > > show
> > > > > > >> > what
> > > > > > >> > > I
> > > > > > >> > > > > mean
> > > > > > >> > > > > > >> (my
> > > > > > >> > > > > > >> > > best
> > > > > > >> > > > > > >> > > > > > > > effort).
> > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>
> > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >> *Current state*
> > > (Diagram
> > > > > 1):
> > > > > > >> > three
> > > > > > >> > > > > > >> interfaces
> > > > > > >> > > > > > >> > at
> > > > > > >> > > > > > >> > > > > three
> > > > > > >> > > > > > >> > > > > > > > > > different
> > > > > > >> > > > > > >> > > > > > > > > > > > >> >> levels.
> > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >> The engine-facing
> SPI
> > > > > > >> > > > > > >> (PolarisMetricsReporter)
> > > > > > >> > > > > > >> > > is
> > > > > > >> > > > > > >> > > > > > clean.
> > > > > > >> > > > > > >> > > > > > > > But
> > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>
> PolarisMetricsManager
> > > on
> > > > > > >> > > > > MetaStoreManager
> > > > > > >> > > > > > >> is a
> > > > > > >> > > > > > >> > > > > > > passthrough
> > > > > > >> > > > > > >> > > > > > > > > to
> > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>
> MetricsPersistence on
> > > > > > >> > > BasePersistence.
> > > > > > >> > > > > > The
> > > > > > >> > > > > > >> > @Beta
> > > > > > >> > > > > > >> > > > > > > > annotation
> > > > > > >> > > > > > >> > > > > > > > > > and
> > > > > > >> > > > > > >> > > > > > > > > > > > SPI
> > > > > > >> > > > > > >> > > > > > > > > > > > >> >> javadoc
> > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >> are on the
> > > > BasePersistence
> > > > > > >> layer,
> > > > > > >> > > while
> > > > > > >> > > > > > the
> > > > > > >> > > > > > >> > > actual
> > > > > > >> > > > > > >> > > > > > > > extension
> > > > > > >> > > > > > >> > > > > > > > > > > > points
> > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>
> > > (PolarisMetricsReporter,
> > > > > > >> > > > > > >> > PolarisMetricsManager)
> > > > > > >> > > > > > >> > > > > carry
> > > > > > >> > > > > > >> > > > > > no
> > > > > > >> > > > > > >> > > > > > > > > > > stability
> > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >> annotation. The
> write
> > > > path
> > > > > > >> goes
> > > > > > >> > > through
> > > > > > >> > > > > > the
> > > > > > >> > > > > > >> > > > > > > > MetaStoreManager
> > > > > > >> > > > > > >> > > > > > > > > > > > layer,
> > > > > > >> > > > > > >> > > > > > > > > > > > >> the
> > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >> read path doesn't.
> > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>
> > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >> *What I envision*
> > > > > (Diagram 2):
> > > > > > >> > two
> > > > > > >> > > SPIs
> > > > > > >> > > > > > at
> > > > > > >> > > > > > >> two
> > > > > > >> > > > > > >> > > > > levels.
> > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>
> PolarisMetricsReporter
> > > > > stays
> > > > > > >> as
> > > > > > >> > the
> > > > > > >> > > > > > >> > > engine-facing
> > > > > > >> > > > > > >> > > > > SPI.
> > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>
> PolarisMetricsManager
> > > > > becomes
> > > > > > >> the
> > > > > > >> > > > > > >> > backend-facing
> > > > > > >> > > > > > >> > > > SPI
> > > > > > >> > > > > > >> > > > > > > with
> > > > > > >> > > > > > >> > > > > > > > > both
> > > > > > >> > > > > > >> > > > > > > > > > > > write
> > > > > > >> > > > > > >> > > > > > > > > > > > >> >> and
> > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >> read methods at
> the
> > > > > > >> > > MetaStoreManager
> > > > > > >> > > > > > level,
> > > > > > >> > > > > > >> > > where
> > > > > > >> > > > > > >> > > > > any
> > > > > > >> > > > > > >> > > > > > > > > backend
> > > > > > >> > > > > > >> > > > > > > > > > > > (JDBC,
> > > > > > >> > > > > > >> > > > > > > > > > > > >> >> NoSQL,
> > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >> remote) can
> implement
> > > > > them.
> > > > > > >> > > > > > >> MetricsPersistence
> > > > > > >> > > > > > >> > > on
> > > > > > >> > > > > > >> > > > > > > > > > > BasePersistence
> > > > > > >> > > > > > >> > > > > > > > > > > > >> goes
> > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >> away. Where
> metrics
> > > > > actually
> > > > > > >> land
> > > > > > >> > > is an
> > > > > > >> > > > > > >> > > > > implementation
> > > > > > >> > > > > > >> > > > > > > > > detail,
> > > > > > >> > > > > > >> > > > > > > > > > > > not a
> > > > > > >> > > > > > >> > > > > > > > > > > > >> >> core
> > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >> interface.
> > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>
> > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >> *Minor naming
> thing*:
> > > > > > >> > > > > > >> PolarisMetricsReporter
> > > > > > >> > > > > > >> > is
> > > > > > >> > > > > > >> > > > > > broader
> > > > > > >> > > > > > >> > > > > > > > than
> > > > > > >> > > > > > >> > > > > > > > > > > what
> > > > > > >> > > > > > >> > > > > > > > > > > > it
> > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >> actually handles.
> It
> > > > only
> > > > > > >> accepts
> > > > > > >> > > > > Iceberg
> > > > > > >> > > > > > >> REST
> > > > > > >> > > > > > >> > > > > Catalog
> > > > > > >> > > > > > >> > > > > > > > > metrics
> > > > > > >> > > > > > >> > > > > > > > > > > > >> >> (ScanReport,
> > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >> CommitReport via
> > > > > > >> MetricsReport).
> > > > > > >> > > > > Generic
> > > > > > >> > > > > > >> table
> > > > > > >> > > > > > >> > > > > metrics
> > > > > > >> > > > > > >> > > > > > > or
> > > > > > >> > > > > > >> > > > > > > > > > > > >> operational
> > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >> metrics aren't in
> > > scope.
> > > > > Not
> > > > > > >> > > blocking,
> > > > > > >> > > > > > but
> > > > > > >> > > > > > >> > worth
> > > > > > >> > > > > > >> > > > > > noting
> > > > > > >> > > > > > >> > > > > > > if
> > > > > > >> > > > > > >> > > > > > > > > the
> > > > > > >> > > > > > >> > > > > > > > > > > > >> metrics
> > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >> surface expands.
> > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>
> > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >> *Rough sketch of
> how
> > > to
> > > > > get
> > > > > > >> > there*:
> > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>  1.  Add read
> methods
> > > to
> > > > > > >> > > > > > >> PolarisMetricsManager
> > > > > > >> > > > > > >> > > > > > > > > > (listScanReports,
> > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >> listCommitReports)
> > > with
> > > > > > >> default
> > > > > > >> > > no-op,
> > > > > > >> > > > > > >> same as
> > > > > > >> > > > > > >> > > the
> > > > > > >> > > > > > >> > > > > > > > existing
> > > > > > >> > > > > > >> > > > > > > > > > > write
> > > > > > >> > > > > > >> > > > > > > > > > > > >> >> methods.
> > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >> (Probably make
> > > > > > >> > > PolarisMetricsManager
> > > > > > >> > > > > more
> > > > > > >> > > > > > >> > > explicit
> > > > > > >> > > > > > >> > > > > on
> > > > > > >> > > > > > >> > > > > > > > being
> > > > > > >> > > > > > >> > > > > > > > > > > > Iceberg
> > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >> specific like
> package
> > > > > name or
> > > > > > >> > class
> > > > > > >> > > > > name
> > > > > > >> > > > > > >> etc.)
> > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>  2.  Wire
> > > > > > >> MetricsReportsService
> > > > > > >> > > through
> > > > > > >> > > > > > >> > > > > > MetaStoreManager
> > > > > > >> > > > > > >> > > > > > > > > > instead
> > > > > > >> > > > > > >> > > > > > > > > > > > of
> > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>
> > > > > callContext.getMetaStore().
> > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>  3.  Extract
> metrics
> > > > > > >> persistence
> > > > > > >> > > from
> > > > > > >> > > > > > >> > > > > > > > > JdbcBasePersistenceImpl
> > > > > > >> > > > > > >> > > > > > > > > > > into
> > > > > > >> > > > > > >> > > > > > > > > > > > >> its
> > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >> own class. That
> file
> > > > > carries
> > > > > > >> ~7
> > > > > > >> > > > > > >> > > responsibilities,
> > > > > > >> > > > > > >> > > > > > > metrics
> > > > > > >> > > > > > >> > > > > > > > > > being
> > > > > > >> > > > > > >> > > > > > > > > > > > one
> > > > > > >> > > > > > >> > > > > > > > > > > > >> of
> > > > > > >> > > > > > >> > > > > > > > > > > > >> >> them.
> > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>  4.  Remove
> > > > > MetricsPersistence
> > > > > > >> > from
> > > > > > >> > > > > > >> > > > BasePersistence.
> > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>
> > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >> *None of this
> needs to
> > > > > happen
> > > > > > >> in
> > > > > > >> > > #4115.
> > > > > > >> > > > > > >> But if
> > > > > > >> > > > > > >> > > the
> > > > > > >> > > > > > >> > > > > > > > direction
> > > > > > >> > > > > > >> > > > > > > > > > > makes
> > > > > > >> > > > > > >> > > > > > > > > > > > >> >> sense,
> > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >> it would be good
> to
> > > > align
> > > > > > >> before
> > > > > > >> > > the
> > > > > > >> > > > > > >> metrics
> > > > > > >> > > > > > >> > > > surface
> > > > > > >> > > > > > >> > > > > > > grows
> > > > > > >> > > > > > >> > > > > > > > > > > > further.
> > > > > > >> > > > > > >> > > > > > > > > > > > >> >> Curious
> > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >> what others
> think.*
> > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>
> > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >> *My mental model
> > > note*:
> > > > > Level
> > > > > > >> 1
> > > > > > >> > > > > > >> > > MetaStoreManager;
> > > > > > >> > > > > > >> > > > > > level
> > > > > > >> > > > > > >> > > > > > > 2
> > > > > > >> > > > > > >> > > > > > > > > > > > >> transactional
> > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >> persistence;
> level 3
> > > > base
> > > > > > >> > > persistence
> > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>
> > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >> Diagram 1
> > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >> <
> > > > > > >> > > > > > >> > > > > > > > > > > > >> >>
> > > > > > >> > > > > > >> > > > > > > > > > > > >>
> > > > > > >> > > > > > >> > > > > > > > > > > >
> > > > > > >> > > > > > >> > > > > > > > > > >
> > > > > > >> > > > > > >> > > > > > > > > >
> > > > > > >> > > > > > >> > > > > > > > >
> > > > > > >> > > > > > >> > > > > > > >
> > > > > > >> > > > > > >> > > > > > >
> > > > > > >> > > > > > >> > > > > >
> > > > > > >> > > > > > >> > > > >
> > > > > > >> > > > > > >> > > >
> > > > > > >> > > > > > >> > >
> > > > > > >> > > > > > >> >
> > > > > > >> > > > > > >>
> > > > > > >> > > > > >
> > > > > > >> > > > >
> > > > > > >> > >
> > > > > > >> >
> > > > > > >>
> > > > >
> > > >
> > >
> https://urldefense.com/v3/__https://www.plantuml.com/plantuml/uml/bLHDR-Cs4BthLmpIYupw0zbkKQ1r3M-S7Bp8xhhM7WCOb3IM65EaGD9EX2RzxHrHb4CxRelwa4YSDu_lpOVcnZ9jzvM8BBS2uGjQpJC3dtHMSekPtMk44IpsMgEqa5XcCOhCZikQQLP1pR8TAp2n3ILhmZDP20m0fcIvUkAoW2qJXd9z1bpToO9BX3WXu0ucy5rpgGPNm0nW5_epUWtm2Ue3pn3kMOFQmKntGZW0BYtgBSi8k5A2QMwybJNMIbFiGSR9QZc4nUqIvikStF0jHprua5C-amge42aNt3R0f5JaaoivdV2Pkqbx4hee4ymOkBh5BTiB-_uIeGeo8zL8rPsPl4DktdEiK1jkB1NdZCRbrSTecDe_mlHbF0wvBmCkaOH5_S8a_TTTKI6-nmCAkEw4LpxsZ-LbYLKQFKMNOgf_wuM7_bV9gOer5SYMMksBSWXFcbi49KNZXNLicwfe3TETC7gPdPqI7uBcHMb1RSzYq34c6PDUM9mn8HRsUTZEiDBve3NjVZumBj0U7SS37mGO7vcwtiK-_pU7U7L_f-digo9YbhSwIfMRwIITKGXbxdIUTCGF1SeCJxloKsU-3k9ddRbX1eDq1q_fx1JbBGT0glVyXimDuP4TQ5qpCAmnGEj2s_6n5mtn1z-97-63itFQZLPO1Ev2tu_WF7Ju-VPc0Skg5bYXxBhkY1xpD7EM_7fyflSpIsqMgVth5xhVr4eQxWQ8enaSAJQSG16yFSDuJ798rrcXr_3n-lfdk7icQjEBmFujL7AodiP_Y4Z7-YxvtZNs4zMgpNTl6tF8sglyPsmqchrjvQ-m-aP94r-TwCA2Ka8upPJZwtvSpoYCXkYMZU2NXvRMBfq9P3i3Le4VAZUAlUZ_oPKsxPgY0Q_BSKLkyr9bhQhQrJjo_x3TPlIB0DPjnMfcIoYP0QaYw1a0fTKDr8fB6ntNuvmoL1ZGkXa69Njh43zf9GiGxHQrA_jDYWRSzF5--WmTVrN97_Sm8LbLUy_lGBmLanJjFkDlGkRqjA_4tm00__;!!Iz9xO38YGHZK!4rnlVIqBQQQ6xpGe1PMV58edOS_cGrKoqdfhw4qSp_JQCavvEjaxXk0OwGPnxyEcO60AzKQK_XDtp7H3DdkFXcrXuTBISHg$
> > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >
> > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >> :
> > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>
> > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >> [image: image.png]
> > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>
> > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >> Diagram 2
> > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >> <
> > > > > > >> > > > > > >> > > > > > > > > > > > >> >>
> > > > > > >> > > > > > >> > > > > > > > > > > > >>
> > > > > > >> > > > > > >> > > > > > > > > > > >
> > > > > > >> > > > > > >> > > > > > > > > > >
> > > > > > >> > > > > > >> > > > > > > > > >
> > > > > > >> > > > > > >> > > > > > > > >
> > > > > > >> > > > > > >> > > > > > > >
> > > > > > >> > > > > > >> > > > > > >
> > > > > > >> > > > > > >> > > > > >
> > > > > > >> > > > > > >> > > > >
> > > > > > >> > > > > > >> > > >
> > > > > > >> > > > > > >> > >
> > > > > > >> > > > > > >> >
> > > > > > >> > > > > > >>
> > > > > > >> > > > > >
> > > > > > >> > > > >
> > > > > > >> > >
> > > > > > >> >
> > > > > > >>
> > > > >
> > > >
> > >
> https://urldefense.com/v3/__https://www.plantuml.com/plantuml/uml/VLLDR-8m4BtdLupO2sWBLVU8AaGB7AXAbssGzb896SS9RXqxjKqBwkv_tt7iV43fdaZYDpFlpRmnOsE9jhjSH9PRmM31hERKm8scMsuPjJlDe0yheZDc8RR4iYWoBrmMH9CS2a9VICPYUy1OZN0YCy5Q0BCbYNhdCeEK28En8G8wCvbnoQ0R8_05Bc6bkLIz3X03p1zzH7zR-9ZfDquPt9C3qoNCX2yV4G2NbkcKu5jdgGJHt0GbZwnG6i-UP3TUpk5gM6Ldqke350eZUqzoCft3U9xWHvxoa5-7K4nF1J46EbEMafsmdrCBbQ44gVggy18IZrn_ph5asd1ZiIKdQSgueZvjXrQFSFrdC3YN-nXmBacxbGiYyLVxLaBtdhqn0LSzdBDhqQtQoOJeGyad3z0lUqnYgpGB6Ns8oVyta00Dy_WnX0tIOZ8v6SYxHll1TrH6aejAik-mh-AphVFCwSUQqFypElag5QRGFDjQKEd96K1P8QP41c9TzA_IIQyvdAWyv_RSiS3skb0_EzDDkK2v5xWF6MiGFlvhpFLcD2Dq2pml14gaF67eQkmd8gulDoC4kSOu6KVpkvlUJg1RTbWISU40RdBUUS_9XfRZ2dwxm_SW8LYFISgm_MnlDQ6M9P1gbKEc4X-2pH_FvJCkCqm9pbVjD6LrwdLeOrDWfOaqc8Wh9BE85oNKxkNQ6o4yGRy_Eae0G_G8tZv81d3bHDB23WOdisohVr3nh_j6lbSjbNaLRTc8UgtPbAU1J_tygOfZX9DWEJeHDvYx-qmSi5FgNLPZwHrHcUsncGQ5-skhUclpE5fo4ounpFauYrUbkU6ccfnxMvitwag4IyerhTxj8In_Oj1bDO4pQru674loYrGlULHLEGCjwJJ8gDoVZR8MxO4BT3IzRvIcAQKezC6xpziGnTyImrfEGyJI_OcKfgtxIvnTqFEMS17L9Z-jsARN5FmTheP7HtSdtOMT0B4GY2FYHXxgQmMtj2bRqiLFGapiVe1_QVKDrkqXcm83aFEXnMYCZ-xlyHy__;!!Iz9xO38YGHZK!4rnlVIqBQQQ6xpGe1PMV58edOS_cGrKoqdfhw4qSp_JQCavvEjaxXk0OwGPnxyEcO60AzKQK_XDtp7H3DdkFXcrXSpmqcNk$
> > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >
> > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >> :
> > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >> [image: image.png]
> > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>
> > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>  -ej
> > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>
> > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >> On Wed, Apr 15,
> 2026
> > > at
> > > > > > >> 8:22 AM
> > > > > > >> > > Dmitri
> > > > > > >> > > > > > >> > > > Bourlatchkov
> > > > > > >> > > > > > >> > > > > <
> > > > > > >> > > > > > >> > > > > > > > > > > > >> dimas@apache.org>
> > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >> wrote:
> > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>
> > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> Hi All,
> > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>>
> > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> Heads up: The
> current
> > > > > state
> > > > > > >> of
> > > > > > >> > PR
> > > > > > >> > > > > [4115]
> > > > > > >> > > > > > >> > looks
> > > > > > >> > > > > > >> > > > > pretty
> > > > > > >> > > > > > >> > > > > > > > solid
> > > > > > >> > > > > > >> > > > > > > > > > to
> > > > > > >> > > > > > >> > > > > > > > > > > > me.
> > > > > > >> > > > > > >> > > > > > > > > > > > >> I
> > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> believe this PR
> is
> > > > > > >> approaching a
> > > > > > >> > > > > > mergeable
> > > > > > >> > > > > > >> > > > > condition.
> > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>>
> > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> Please post your
> > > > reviews
> > > > > if
> > > > > > >> you
> > > > > > >> > > have
> > > > > > >> > > > > any
> > > > > > >> > > > > > >> > > > comments.
> > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>>
> > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> [4115]
> > > > > > >> > > > > > >> > > >
> > > > > > >> >
> > > > > > >>
> > > > >
> > > >
> > >
> https://urldefense.com/v3/__https://github.com/apache/polaris/pull/4115__;!!Iz9xO38YGHZK!4rnlVIqBQQQ6xpGe1PMV58edOS_cGrKoqdfhw4qSp_JQCavvEjaxXk0OwGPnxyEcO60AzKQK_XDtp7H3DdkFXcrXtFRscKs$
> > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>>
> > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> Thanks,
> > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> Dmitri.
> > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>>
> > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> On Tue, Mar 3,
> 2026
> > > at
> > > > > > >> 3:29 PM
> > > > > > >> > > Anand
> > > > > > >> > > > > > Kumar
> > > > > > >> > > > > > >> > > > Sankaran
> > > > > > >> > > > > > >> > > > > > via
> > > > > > >> > > > > > >> > > > > > > > > dev <
> > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>>
> > > dev@polaris.apache.org
> > > > >
> > > > > > >> wrote:
> > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>>
> > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > Hi Yufei and
> > > Dmitri,
> > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> >
> > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > Here is a
> proposal
> > > > for
> > > > > the
> > > > > > >> > REST
> > > > > > >> > > > > > >> endpoints
> > > > > > >> > > > > > >> > for
> > > > > > >> > > > > > >> > > > > > metrics
> > > > > > >> > > > > > >> > > > > > > > and
> > > > > > >> > > > > > >> > > > > > > > > > > > events.
> > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> >
> > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> >
> > > > > > >> > > > > > >> > > > > >
> > > > > > >> >
> > > > > > >>
> > > > >
> > > >
> > >
> https://urldefense.com/v3/__https://github.com/apache/polaris/pull/3924/changes__;!!Iz9xO38YGHZK!4rnlVIqBQQQ6xpGe1PMV58edOS_cGrKoqdfhw4qSp_JQCavvEjaxXk0OwGPnxyEcO60AzKQK_XDtp7H3DdkFXcrXcjoPhSs$
> > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> >
> > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > I did not see
> any
> > > > > > >> precursors
> > > > > > >> > for
> > > > > > >> > > > > > >> raising a
> > > > > > >> > > > > > >> > PR
> > > > > > >> > > > > > >> > > > for
> > > > > > >> > > > > > >> > > > > > > > > > proposals,
> > > > > > >> > > > > > >> > > > > > > > > > > so
> > > > > > >> > > > > > >> > > > > > > > > > > > >> >> trying
> > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > this.  Please
> let
> > > me
> > > > > know
> > > > > > >> what
> > > > > > >> > > you
> > > > > > >> > > > > > >> think.
> > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> >
> > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > -
> > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > Anand
> > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> >
> > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > From: Anand
> Kumar
> > > > > Sankaran
> > > > > > >> <
> > > > > > >> > > > > > >> > > > > > > anand.sankaran@workday.com
> > > > > > >> > > > > > >> > > > > > > > >
> > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > Date: Monday,
> March
> > > > 2,
> > > > > > >> 2026 at
> > > > > > >> > > > > > 10:25 AM
> > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > To:
> > > > > dev@polaris.apache.org
> > > > > > >> <
> > > > > > >> > > > > > >> > > > > dev@polaris.apache.org
> > > > > > >> > > > > > >> > > > > > >
> > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > Subject: Re:
> > > Polaris
> > > > > > >> Telemetry
> > > > > > >> > > and
> > > > > > >> > > > > > Audit
> > > > > > >> > > > > > >> > > Trail
> > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> >
> > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > About the REST
> API,
> > > > > based
> > > > > > >> on
> > > > > > >> > my
> > > > > > >> > > use
> > > > > > >> > > > > > >> cases:
> > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> >
> > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> >
> > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> >   1.
> > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > I want to be
> able
> > > to
> > > > > query
> > > > > > >> > > commit
> > > > > > >> > > > > > >> metrics
> > > > > > >> > > > > > >> > to
> > > > > > >> > > > > > >> > > > > track
> > > > > > >> > > > > > >> > > > > > > > files
> > > > > > >> > > > > > >> > > > > > > > > > > added
> > > > > > >> > > > > > >> > > > > > > > > > > > /
> > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> removed
> > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > per commit,
> along
> > > > with
> > > > > > >> record
> > > > > > >> > > > > counts.
> > > > > > >> > > > > > >> The
> > > > > > >> > > > > > >> > > > > ingestion
> > > > > > >> > > > > > >> > > > > > > > > > pipeline
> > > > > > >> > > > > > >> > > > > > > > > > > > that
> > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> writes
> > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > this data is
> owned
> > > by
> > > > > us
> > > > > > >> and
> > > > > > >> > we
> > > > > > >> > > are
> > > > > > >> > > > > > >> > > guaranteed
> > > > > > >> > > > > > >> > > > to
> > > > > > >> > > > > > >> > > > > > > write
> > > > > > >> > > > > > >> > > > > > > > > > this
> > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> information
> > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > for each write.
> > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> >   2.
> > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > I want to be
> able
> > > to
> > > > > query
> > > > > > >> > scan
> > > > > > >> > > > > > metrics
> > > > > > >> > > > > > >> for
> > > > > > >> > > > > > >> > > > > read. I
> > > > > > >> > > > > > >> > > > > > > > > > > understand
> > > > > > >> > > > > > >> > > > > > > > > > > > >> >> clients
> > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> do
> > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > not fulfill
> this
> > > > > > >> requirement.
> > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> >   3.
> > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > I want to be
> able
> > > to
> > > > > query
> > > > > > >> the
> > > > > > >> > > > > events
> > > > > > >> > > > > > >> table
> > > > > > >> > > > > > >> > > > > (events
> > > > > > >> > > > > > >> > > > > > > are
> > > > > > >> > > > > > >> > > > > > > > > > > > >> persisted) -
> > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> this
> > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > may supersede
> #2, I
> > > > am
> > > > > not
> > > > > > >> > sure
> > > > > > >> > > yet.
> > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> >
> > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > All this
> > > information
> > > > > is in
> > > > > > >> the
> > > > > > >> > > JDBC
> > > > > > >> > > > > > >> based
> > > > > > >> > > > > > >> > > > > > persistence
> > > > > > >> > > > > > >> > > > > > > > > model
> > > > > > >> > > > > > >> > > > > > > > > > > and
> > > > > > >> > > > > > >> > > > > > > > > > > > >> is
> > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > persisted in
> the
> > > > > > >> metastore. I
> > > > > > >> > > > > > currently
> > > > > > >> > > > > > >> > don’t
> > > > > > >> > > > > > >> > > > > have
> > > > > > >> > > > > > >> > > > > > a
> > > > > > >> > > > > > >> > > > > > > > need
> > > > > > >> > > > > > >> > > > > > > > > > to
> > > > > > >> > > > > > >> > > > > > > > > > > > >> query
> > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > prometheus or
> open
> > > > > > >> telemetry.
> > > > > > >> > I
> > > > > > >> > > do
> > > > > > >> > > > > > >> publish
> > > > > > >> > > > > > >> > > some
> > > > > > >> > > > > > >> > > > > > > events
> > > > > > >> > > > > > >> > > > > > > > to
> > > > > > >> > > > > > >> > > > > > > > > > > > >> Prometheus
> > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> and
> > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > they are
> forwarded
> > > to
> > > > > our
> > > > > > >> > > dashboards
> > > > > > >> > > > > > >> > > elsewhere.
> > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> >
> > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > About the CLI
> > > > > utilities, I
> > > > > > >> > > meant the
> > > > > > >> > > > > > >> admin
> > > > > > >> > > > > > >> > > user
> > > > > > >> > > > > > >> > > > > > > > > utilities.
> > > > > > >> > > > > > >> > > > > > > > > > In
> > > > > > >> > > > > > >> > > > > > > > > > > > >> one of
> > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> the
> > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > earliest
> drafts of
> > > my
> > > > > > >> > proposal,
> > > > > > >> > > > > > Prashant
> > > > > > >> > > > > > >> > > > > mentioned
> > > > > > >> > > > > > >> > > > > > > that
> > > > > > >> > > > > > >> > > > > > > > > the
> > > > > > >> > > > > > >> > > > > > > > > > > > >> metrics
> > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> tables
> > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > can grow
> > > indefinitely
> > > > > and
> > > > > > >> > that a
> > > > > > >> > > > > > similar
> > > > > > >> > > > > > >> > > > problem
> > > > > > >> > > > > > >> > > > > > > exists
> > > > > > >> > > > > > >> > > > > > > > > > with
> > > > > > >> > > > > > >> > > > > > > > > > > > the
> > > > > > >> > > > > > >> > > > > > > > > > > > >> >> events
> > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > table as well.
> We
> > > > > discussed
> > > > > > >> > that
> > > > > > >> > > > > > >> cleaning
> > > > > > >> > > > > > >> > up
> > > > > > >> > > > > > >> > > of
> > > > > > >> > > > > > >> > > > > old
> > > > > > >> > > > > > >> > > > > > > > > records
> > > > > > >> > > > > > >> > > > > > > > > > > > from
> > > > > > >> > > > > > >> > > > > > > > > > > > >> >> both
> > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > metrics tables
> and
> > > > > events
> > > > > > >> > > tables can
> > > > > > >> > > > > > be
> > > > > > >> > > > > > >> > done
> > > > > > >> > > > > > >> > > > via
> > > > > > >> > > > > > >> > > > > a
> > > > > > >> > > > > > >> > > > > > > CLI
> > > > > > >> > > > > > >> > > > > > > > > > > utility.
> > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> >
> > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > I see that
> Yufei
> > > has
> > > > > > >> covered
> > > > > > >> > the
> > > > > > >> > > > > > >> discussion
> > > > > > >> > > > > > >> > > > about
> > > > > > >> > > > > > >> > > > > > > > > > > datasources.
> > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> >
> > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > -
> > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > Anand
> > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> >
> > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> >
> > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> >
> > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > From: Yufei Gu
> <
> > > > > > >> > > > > flyrain000@gmail.com>
> > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > Date: Friday,
> > > > February
> > > > > 27,
> > > > > > >> > 2026
> > > > > > >> > > at
> > > > > > >> > > > > > >> 9:54 PM
> > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > To:
> > > > > dev@polaris.apache.org
> > > > > > >> <
> > > > > > >> > > > > > >> > > > > dev@polaris.apache.org
> > > > > > >> > > > > > >> > > > > > >
> > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > Subject: Re:
> > > Polaris
> > > > > > >> Telemetry
> > > > > > >> > > and
> > > > > > >> > > > > > Audit
> > > > > > >> > > > > > >> > > Trail
> > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> >
> > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > This Message Is
> > > From
> > > > an
> > > > > > >> > External
> > > > > > >> > > > > > Sender
> > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > This message
> came
> > > > from
> > > > > > >> outside
> > > > > > >> > > your
> > > > > > >> > > > > > >> > > > organization.
> > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > Report
> Suspicious<
> > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> >
> > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>>
> > > > > > >> > > > > > >> > > > > > > > > > > > >> >>
> > > > > > >> > > > > > >> > > > > > > > > > > > >>
> > > > > > >> > > > > > >> > > > > > > > > > > >
> > > > > > >> > > > > > >> > > > > > > > > > >
> > > > > > >> > > > > > >> > > > > > > > > >
> > > > > > >> > > > > > >> > > > > > > > >
> > > > > > >> > > > > > >> > > > > > > >
> > > > > > >> > > > > > >> > > > > > >
> > > > > > >> > > > > > >> > > > > >
> > > > > > >> > > > > > >> > > > >
> > > > > > >> > > > > > >> > > >
> > > > > > >> > > > > > >> > >
> > > > > > >> > > > > > >> >
> > > > > > >> > > > > > >>
> > > > > > >> > > > > >
> > > > > > >> > > > >
> > > > > > >> > >
> > > > > > >> >
> > > > > > >>
> > > > >
> > > >
> > >
> https://us-phishalarm-ewt.proofpoint.com/EWT/v1/Iz9xO38YGHZK!YhNDZABkHi1B699ote2uMwpOZw8i0QMCGO2Szc-HshuABGhGvwPJcymE6G2oUUxtS8xDkSrtGTPm_I3QnVDHoLMk50m9v8z_nZKTkd-bnVUbreF1u0WnfV_X5eYevZl_$
> > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > >
> > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> >
> > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> >
> > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > As I mentioned
> in
> > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> >
> > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>>
> > > > > > >> > > > > > >> > > > > > > > > > > > >> >>
> > > > > > >> > > > > > >> > > > > > > > > > > > >>
> > > > > > >> > > > > > >> > > > > > > > > > > >
> > > > > > >> > > > > > >> > > > > > > > > > >
> > > > > > >> > > > > > >> > > > > > > > > >
> > > > > > >> > > > > > >> > > > > > > > >
> > > > > > >> > > > > > >> > > > > > > >
> > > > > > >> > > > > > >> > > > > > >
> > > > > > >> > > > > > >> > > > > >
> > > > > > >> > > > > > >> > > > >
> > > > > > >> > > > > > >> > > >
> > > > > > >> > > > > > >> > >
> > > > > > >> > > > > > >> >
> > > > > > >> > > > > > >>
> > > > > > >> > > > > >
> > > > > > >> > > > >
> > > > > > >> > >
> > > > > > >> >
> > > > > > >>
> > > > >
> > > >
> > >
> https://urldefense.com/v3/__https://github.com/apache/polaris/issues/3890__;!!Iz9xO38YGHZK!5EuyFFkk3vhRWVIRvQAWBSQfpJkTMA9HxugzDwXmN0LPPqhEFxYkFRGVhtb8AqUwXtDh2OplcMnbMDHKOxrvDU0$
> > > > > > >> > >> > > >> > > > > > > > > > > > >> >> >>> ,
> > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > supporting
> > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > multiple data
> > > sources
> > > > > is
> > > > > > >> not a
> > > > > > >> > > > > trivial
> > > > > > >> > > > > > >> > > change.
> > > > > > >> > > > > > >> > > > I
> > > > > > >> > > > > > >> > > > > > > would
> > > > > > >> > > > > > >> > > > > > > > > > > strongly
> > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> recommend
> > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > starting with a
> > > > design
> > > > > > >> > document
> > > > > > >> > > to
> > > > > > >> > > > > > >> > carefully
> > > > > > >> > > > > > >> > > > > > evaluate
> > > > > > >> > > > > > >> > > > > > > > the
> > > > > > >> > > > > > >> > > > > > > > > > > > >> >> architectural
> > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > implications
> and
> > > long
> > > > > term
> > > > > > >> > > impact.
> > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> >
> > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > A REST
> endpoint to
> > > > > query
> > > > > > >> > metrics
> > > > > > >> > > > > seems
> > > > > > >> > > > > > >> > > > reasonable
> > > > > > >> > > > > > >> > > > > > > given
> > > > > > >> > > > > > >> > > > > > > > > the
> > > > > > >> > > > > > >> > > > > > > > > > > > >> current
> > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> JDBC
> > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > based
> persistence
> > > > > model.
> > > > > > >> That
> > > > > > >> > > said,
> > > > > > >> > > > > we
> > > > > > >> > > > > > >> may
> > > > > > >> > > > > > >> > > also
> > > > > > >> > > > > > >> > > > > > > > consider
> > > > > > >> > > > > > >> > > > > > > > > > > > >> alternative
> > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > storage
> models. For
> > > > > > >> example,
> > > > > > >> > if
> > > > > > >> > > we
> > > > > > >> > > > > > later
> > > > > > >> > > > > > >> > > adopt
> > > > > > >> > > > > > >> > > > a
> > > > > > >> > > > > > >> > > > > > time
> > > > > > >> > > > > > >> > > > > > > > > > series
> > > > > > >> > > > > > >> > > > > > > > > > > > >> system
> > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> such as
> > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > Prometheus to
> store
> > > > > > >> metrics,
> > > > > > >> > the
> > > > > > >> > > > > query
> > > > > > >> > > > > > >> > model
> > > > > > >> > > > > > >> > > > and
> > > > > > >> > > > > > >> > > > > > > access
> > > > > > >> > > > > > >> > > > > > > > > > > > patterns
> > > > > > >> > > > > > >> > > > > > > > > > > > >> >> would
> > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> be
> > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > fundamentally
> > > > > different.
> > > > > > >> > > Designing
> > > > > > >> > > > > the
> > > > > > >> > > > > > >> REST
> > > > > > >> > > > > > >> > > API
> > > > > > >> > > > > > >> > > > > > > without
> > > > > > >> > > > > > >> > > > > > > > > > > > >> considering
> > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> these
> > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > potential
> > > evolutions
> > > > > may
> > > > > > >> limit
> > > > > > >> > > > > > >> flexibility.
> > > > > > >> > > > > > >> > > I'd
> > > > > > >> > > > > > >> > > > > > > suggest
> > > > > > >> > > > > > >> > > > > > > > > to
> > > > > > >> > > > > > >> > > > > > > > > > > > start
> > > > > > >> > > > > > >> > > > > > > > > > > > >> >> with
> > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> the
> > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > use case.
> > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> >
> > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > Yufei
> > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> >
> > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> >
> > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > On Fri, Feb 27,
> > > 2026
> > > > at
> > > > > > >> > 3:42 PM
> > > > > > >> > > > > Dmitri
> > > > > > >> > > > > > >> > > > > > Bourlatchkov <
> > > > > > >> > > > > > >> > > > > > > > > > > > >> >> dimas@apache.org>
> > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > wrote:
> > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> >
> > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > > Hi Anand,
> > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > >
> > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > > Sharing my
> > > view...
> > > > > > >> subject
> > > > > > >> > to
> > > > > > >> > > > > > >> discussion:
> > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > >
> > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > > 1. Adding
> non-IRC
> > > > > REST
> > > > > > >> API
> > > > > > >> > to
> > > > > > >> > > > > > Polaris
> > > > > > >> > > > > > >> is
> > > > > > >> > > > > > >> > > > > > perfectly
> > > > > > >> > > > > > >> > > > > > > > > fine.
> > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > >
> > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > > Figuring out
> > > > specific
> > > > > > >> > endpoint
> > > > > > >> > > > > URIs
> > > > > > >> > > > > > >> and
> > > > > > >> > > > > > >> > > > > payloads
> > > > > > >> > > > > > >> > > > > > > > might
> > > > > > >> > > > > > >> > > > > > > > > > > > require
> > > > > > >> > > > > > >> > > > > > > > > > > > >> a
> > > > > > >> > > > > > >> > > > > > > > > > > > >> >> few
> > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > > roundtrips,
> so
> > > > > opening a
> > > > > > >> > > separate
> > > > > > >> > > > > > >> thread
> > > > > > >> > > > > > >> > > for
> > > > > > >> > > > > > >> > > > > that
> > > > > > >> > > > > > >> > > > > > > > might
> > > > > > >> > > > > > >> > > > > > > > > > be
> > > > > > >> > > > > > >> > > > > > > > > > > > >> best.
> > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > > Contributors
> > > > commonly
> > > > > > >> create
> > > > > > >> > > > > Google
> > > > > > >> > > > > > >> Docs
> > > > > > >> > > > > > >> > > for
> > > > > > >> > > > > > >> > > > > new
> > > > > > >> > > > > > >> > > > > > > API
> > > > > > >> > > > > > >> > > > > > > > > > > > proposals
> > > > > > >> > > > > > >> > > > > > > > > > > > >> too
> > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> (they
> > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > > fairly easy
> to
> > > > > update as
> > > > > > >> the
> > > > > > >> > > email
> > > > > > >> > > > > > >> > > discussion
> > > > > > >> > > > > > >> > > > > > > > > > progresses).
> > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > >
> > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > > There was a
> > > > > suggestion to
> > > > > > >> > try
> > > > > > >> > > > > > Markdown
> > > > > > >> > > > > > >> > > (with
> > > > > > >> > > > > > >> > > > > PRs)
> > > > > > >> > > > > > >> > > > > > > for
> > > > > > >> > > > > > >> > > > > > > > > > > > proposals
> > > > > > >> > > > > > >> > > > > > > > > > > > >> >> [1]
> > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> ...
> > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > > feel free to
> give
> > > > it
> > > > > a
> > > > > > >> try
> > > > > > >> > if
> > > > > > >> > > you
> > > > > > >> > > > > > are
> > > > > > >> > > > > > >> > > > > comfortable
> > > > > > >> > > > > > >> > > > > > > > with
> > > > > > >> > > > > > >> > > > > > > > > > > that.
> > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > >
> > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > > 2. Could you
> > > > clarify
> > > > > > >> whether
> > > > > > >> > > you
> > > > > > >> > > > > > mean
> > > > > > >> > > > > > >> end
> > > > > > >> > > > > > >> > > > user
> > > > > > >> > > > > > >> > > > > > > > > utilities
> > > > > > >> > > > > > >> > > > > > > > > > or
> > > > > > >> > > > > > >> > > > > > > > > > > > >> admin
> > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> user
> > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > > utilities?
> In the
> > > > > latter
> > > > > > >> > case
> > > > > > >> > > > > those
> > > > > > >> > > > > > >> might
> > > > > > >> > > > > > >> > > be
> > > > > > >> > > > > > >> > > > > more
> > > > > > >> > > > > > >> > > > > > > > > > suitable
> > > > > > >> > > > > > >> > > > > > > > > > > > for
> > > > > > >> > > > > > >> > > > > > > > > > > > >> the
> > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> Admin
> > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > > CLI (java)
> not
> > > the
> > > > > Python
> > > > > > >> > CLI,
> > > > > > >> > > > > IMHO.
> > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > >
> > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > > Why would
> these
> > > > > > >> utilities be
> > > > > > >> > > > > common
> > > > > > >> > > > > > >> with
> > > > > > >> > > > > > >> > > > > events?
> > > > > > >> > > > > > >> > > > > > > > IMHO,
> > > > > > >> > > > > > >> > > > > > > > > > > event
> > > > > > >> > > > > > >> > > > > > > > > > > > >> use
> > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> cases
> > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > are
> > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > > distinct from
> > > > > scan/commit
> > > > > > >> > > metrics.
> > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > >
> > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > > 3. I'd prefer
> > > > > separating
> > > > > > >> > > metrics
> > > > > > >> > > > > > >> > > persistence
> > > > > > >> > > > > > >> > > > > from
> > > > > > >> > > > > > >> > > > > > > > > > MetaStore
> > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> persistence
> > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > at
> > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > > the code
> level,
> > > so
> > > > > that
> > > > > > >> they
> > > > > > >> > > could
> > > > > > >> > > > > > be
> > > > > > >> > > > > > >> > mixed
> > > > > > >> > > > > > >> > > > and
> > > > > > >> > > > > > >> > > > > > > > matched
> > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> independently.
> > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > The
> > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > > separate
> > > datasource
> > > > > > >> question
> > > > > > >> > > will
> > > > > > >> > > > > > >> become
> > > > > > >> > > > > > >> > a
> > > > > > >> > > > > > >> > > > > > > non-issue
> > > > > > >> > > > > > >> > > > > > > > > with
> > > > > > >> > > > > > >> > > > > > > > > > > > that
> > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> approach,
> > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > I
> > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > > guess.
> > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > >
> > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > > The
> rationale for
> > > > > > >> separating
> > > > > > >> > > scan
> > > > > > >> > > > > > >> metrics
> > > > > > >> > > > > > >> > > and
> > > > > > >> > > > > > >> > > > > > > > metastore
> > > > > > >> > > > > > >> > > > > > > > > > > > >> >> persistence
> > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> is
> > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > that
> > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > > "cascading
> > > deletes"
> > > > > > >> between
> > > > > > >> > > them
> > > > > > >> > > > > are
> > > > > > >> > > > > > >> > hardly
> > > > > > >> > > > > > >> > > > > ever
> > > > > > >> > > > > > >> > > > > > > > > > required.
> > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> Furthermore,
> > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > the
> > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > > data and
> query
> > > > > patterns
> > > > > > >> are
> > > > > > >> > > very
> > > > > > >> > > > > > >> > different
> > > > > > >> > > > > > >> > > so
> > > > > > >> > > > > > >> > > > > > > > different
> > > > > > >> > > > > > >> > > > > > > > > > > > >> >> technologies
> > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > might
> > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > > be
> beneficial in
> > > > each
> > > > > > >> case.
> > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > >
> > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > > [1]
> > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> >
> > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>>
> > > > > > >> > > > > > >> > > > > > > > > > > > >> >>
> > > > > > >> > > > > > >> > > > > > > > > > > > >>
> > > > > > >> > > > > > >> > > > > > > > > > > >
> > > > > > >> > > > > > >> > > > > > > > > > >
> > > > > > >> > > > > > >> > > > > > > > > >
> > > > > > >> > > > > > >> > > > > > > > >
> > > > > > >> > > > > > >> > > > > > > >
> > > > > > >> > > > > > >> > > > > > >
> > > > > > >> > > > > > >> > > > > >
> > > > > > >> > > > > > >> > > > >
> > > > > > >> > > > > > >> > > >
> > > > > > >> > > > > > >> > >
> > > > > > >> > > > > > >> >
> > > > > > >> > > > > > >>
> > > > > > >> > > > > >
> > > > > > >> > > > >
> > > > > > >> > >
> > > > > > >> >
> > > > > > >>
> > > > >
> > > >
> > >
> https://urldefense.com/v3/__https://lists.apache.org/thread/yto2wp982t43h1mqjwnslswhws5z47cy__;!!Iz9xO38YGHZK!5EuyFFkk3vhRWVIRvQAWBSQfpJkTMA9HxugzDwXmN0LPPqhEFxYkFRGVhtb8AqUwXtDh2OplcMnbMDHKxYDakNU$
> > > > > > >> > >> > > >> > > > > > > > > > > > >> >> >>> > >
> > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > > Cheers,
> > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > > Dmitri.
> > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > >
> > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > > On Fri, Feb
> 27,
> > > > 2026
> > > > > at
> > > > > > >> > > 6:19 PM
> > > > > > >> > > > > > Anand
> > > > > > >> > > > > > >> > Kumar
> > > > > > >> > > > > > >> > > > > > > Sankaran
> > > > > > >> > > > > > >> > > > > > > > > via
> > > > > > >> > > > > > >> > > > > > > > > > > dev
> > > > > > >> > > > > > >> > > > > > > > > > > > <
> > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > >
> > > > > dev@polaris.apache.org>
> > > > > > >> > > wrote:
> > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > >
> > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > > > Thanks all.
> > > This
> > > > > PR is
> > > > > > >> > > merged
> > > > > > >> > > > > now.
> > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > > >
> > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > > > Here are
> the
> > > > > follow-up
> > > > > > >> > > features
> > > > > > >> > > > > /
> > > > > > >> > > > > > >> work
> > > > > > >> > > > > > >> > > > > needed.
> > > > > > >> > > > > > >> > > > > > > > These
> > > > > > >> > > > > > >> > > > > > > > > > > were
> > > > > > >> > > > > > >> > > > > > > > > > > > >> all
> > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> part of
> > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > > the
> > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > > > merged PR
> at
> > > some
> > > > > > >> point in
> > > > > > >> > > time
> > > > > > >> > > > > > and
> > > > > > >> > > > > > >> > were
> > > > > > >> > > > > > >> > > > > > removed
> > > > > > >> > > > > > >> > > > > > > to
> > > > > > >> > > > > > >> > > > > > > > > > > reduce
> > > > > > >> > > > > > >> > > > > > > > > > > > >> >> scope.
> > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > > >
> > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > > > Please let
> me
> > > > know
> > > > > what
> > > > > > >> > you
> > > > > > >> > > > > think.
> > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > > >
> > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > > >
> > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > > >   1.  A
> REST
> > > API
> > > > to
> > > > > > >> > paginate
> > > > > > >> > > > > > through
> > > > > > >> > > > > > >> > > table
> > > > > > >> > > > > > >> > > > > > > metrics.
> > > > > > >> > > > > > >> > > > > > > > > > This
> > > > > > >> > > > > > >> > > > > > > > > > > > >> will be
> > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > non-IRC
> > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > > > standard
> > > > addition.
> > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > > >   2.
> Utilities
> > > > for
> > > > > > >> > > managing old
> > > > > > >> > > > > > >> > records,
> > > > > > >> > > > > > >> > > > > > should
> > > > > > >> > > > > > >> > > > > > > be
> > > > > > >> > > > > > >> > > > > > > > > > > common
> > > > > > >> > > > > > >> > > > > > > > > > > > >> with
> > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> events.
> > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > > > There was
> some
> > > > > > >> discussion
> > > > > > >> > > that
> > > > > > >> > > > > it
> > > > > > >> > > > > > >> > belongs
> > > > > > >> > > > > > >> > > > to
> > > > > > >> > > > > > >> > > > > > the
> > > > > > >> > > > > > >> > > > > > > > CLI.
> > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > > >   3.
> Separate
> > > > > > >> datasource
> > > > > > >> > > > > > (metrics,
> > > > > > >> > > > > > >> > > events,
> > > > > > >> > > > > > >> > > > > > even
> > > > > > >> > > > > > >> > > > > > > > > other
> > > > > > >> > > > > > >> > > > > > > > > > > > >> tables?).
> > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > > >
> > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > > >
> > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > > > Anything
> else?
> > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > > >
> > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > > > -
> > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > > > Anand
> > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > > >
> > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > > >
> > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > >
> > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> >
> > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> >
> > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>>
> > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>
> > > > > > >> > > > > > >> > > > > > > > > > > > >> >>
> > > > > > >> > > > > > >> > > > > > > > > > > > >> >
> > > > > > >> > > > > > >> > > > > > > > > > > > >>
> > > > > > >> > > > > > >> > > > > > > > > > > > >
> > > > > > >> > > > > > >> > > > > > > > > > > >
> > > > > > >> > > > > > >> > > > > > > > > > >
> > > > > > >> > > > > > >> > > > > > > > > >
> > > > > > >> > > > > > >> > > > > > > > >
> > > > > > >> > > > > > >> > > > > > > >
> > > > > > >> > > > > > >> > > > > > >
> > > > > > >> > > > > > >> > > > > >
> > > > > > >> > > > > > >> > > > >
> > > > > > >> > > > > > >> > > >
> > > > > > >> > > > > > >> > >
> > > > > > >> > > > > > >> >
> > > > > > >> > > > > > >>
> > > > > > >> > > > > > >
> > > > > > >> > > > > >
> > > > > > >> > > > >
> > > > > > >> > >
> > > > > > >> >
> > > > > > >> >
> > > > > > >>
> > > > > > >
> > > > >
> > > >
> > >
>

Re: Proposal for REST endpoints for table metrics and events

[Jean-Baptiste Onofré <jb...@nanthrax.net>]

Hi all,

I agree with Yufei: I would keep TABLE_READ_METRICS as a separate,
explicit grant, not implied by TABLE_FULL_METADATA.

I believe metrics reports are closer to an activity log than to table
metadata. I also think it's consistent with what we have today, as
reporting scan and commit metrics is already with TABLE_READ_DATA and
TABLE_WRITE_DATA, not by "metadata" privilege.

For the troubleshooting use case, the cost is just one additional
grant, I think it's acceptable :) I would still let
CATALOG_MANAGE_CONTENT implies it, as that privilege already implies
TABLE_READ_DATA.

Concretely for PR #5669, I propose:
- to remove the "Implied by TABLE_FULL_METADATA" statement from the
javadoc and CHANGELOG
- to add SUPER_PRIVILEGES entry in the PR, so the grant change is self-contained
- to drop ICEBERG_TABLE restriction.
- to update access-control.md documenting TABLE_READ_METRICS, etc

I will comment directly in the PR.

Regards
JB

On Fri, Oct 2, 2026 at 3:40 AM Yufei Gu <fl...@gmail.com> wrote:
>
> Thanks Dmitri for the reminder.
>
> PR #5659 <https://github.com/apache/polaris/pull/5659> adds
> TABLE_READ_METRICS and says it is implied by TABLE_FULL_METADATA. Should
> that be the default?
>
> I’m leaning toward keeping it separate. Metrics describe workloads (Spark
> ETL, Flink streaming, etc) using a table and can include query filters, so
> someone managing table metadata may not need access to other users’ query
> activity.
>
> Including it could make troubleshooting easier for table admins, though.
> Are there use cases that favor automatic access over an explicit grant?
> Yufei
>
>
> On Wed, Sep 30, 2026 at 1:57 PM Dmitri Bourlatchkov <di...@apache.org>
> wrote:
>
> > Hi Yufei,
> >
> > Here's the separate PR for Metrics-related grants:
> > https://github.com/apache/polaris/pull/5659
> >
> > Please review it to unblock this contribution.
> >
> > Thanks,
> > Dmitri.
> >
> > On Wed, Sep 30, 2026 at 1:05 PM Yufei Gu <fl...@gmail.com> wrote:
> >
> > > From the previous email, we agreed to split the REST change from the
> > grant
> > > change. I think the PR is ready to go if we only keep the grant changes.
> > >
> > > Yufei
> > >
> > >
> > > On Tue, Sep 29, 2026 at 9:26 PM Jean-Baptiste Onofré <jb...@nanthrax.net>
> > > wrote:
> > >
> > > > Hi guys,
> > > >
> > > > I agree with Dmitri: +1 to merge PR #4115 on Oct 1 (tomorrow).
> > > >
> > > > From my side, all outstanding feedback has been addressed, including
> > > > the final documentation points raised by EJ regarding the empty-200
> > > > NoOp behavior and reconstructed payload description. The branch is
> > > > clean, up to date with main, and CI is green.
> > > >
> > > > Once this lands, we can follow up with PR #4756 for the JDBC query SPI
> > > > implementation.
> > > >
> > > > Regards
> > > > JB
> > > >
> > > > On Wed, Sep 30, 2026 at 1:53 AM Dmitri Bourlatchkov <di...@apache.org>
> > > > wrote:
> > > > >
> > > > > Hi All,
> > > > >
> > > > > Any remaining blockers on [4115]?
> > > > >
> > > > > Given that the review was very long, please restate any remaining
> > > > concerns
> > > > > (here or in GH).
> > > > >
> > > > > Otherwise, I propose merging [4115] on Oct 1.
> > > > >
> > > > > [4115] https://github.com/apache/polaris/pull/4115
> > > > >
> > > > > Thanks,
> > > > > Dmitri.
> > > > >
> > > > > On Wed, Sep 23, 2026 at 7:53 PM Dmitri Bourlatchkov <
> > dimas@apache.org>
> > > > > wrote:
> > > > >
> > > > > > Hi EJ,
> > > > > >
> > > > > > Thanks for the meeting recap!
> > > > > >
> > > > > > All:
> > > > > >
> > > > > > Worth noting: PR [4115] also contains Management API changes that
> > > were
> > > > not
> > > > > > anticipated initially and did not get an API change review until
> > now.
> > > > These
> > > > > > are small changes to add new grant types for metrics RBAC..
> > > > > >
> > > > > > I've reviewed and approved PR [4115] in GH.
> > > > > >
> > > > > > [4115] https://github.com/apache/polaris/pull/4115
> > > > > >
> > > > > > Cheers,
> > > > > > Dmitri.
> > > > > >
> > > > > > On Wed, Aug 12, 2026 at 5:21 PM EJ Wang <
> > > > ej.wang.dev.interest@gmail.com>
> > > > > > wrote:
> > > > > >
> > > > > >> Hi folks,
> > > > > >>
> > > > > >> Notes from today's metrics architecture sync are in the usual doc:
> > > > > >>
> > > > > >>
> > > >
> > >
> > https://docs.google.com/document/d/100h7c4damrUzVuquYbBHM0EvA4LSWuW2IT2dN_7nYVA/edit?pli=1&tab=t.18iv1m7p6i6b
> > > > > >>
> > > > > >> Highlights:
> > > > > >>
> > > > > >> Module placement. core/ now holds the entity data model only. A
> > new
> > > > spi/
> > > > > >> takes every other shared contract: feature SPIs, feature non-SPI
> > > > > >> contracts,
> > > > > >> substrate SPIs, substrate non-SPI contracts, and contract-required
> > > > types.
> > > > > >> That settles the question PR #5204 reopened, by placing into a new
> > > SPI
> > > > > >> module. Note that it changes slightly towards the referenced WIP
> > RFC
> > > > [1]
> > > > > >> recommendations: what proposed to place in core/, now should go to
> > > > spi/
> > > > > >> except for the entity data model.
> > > > > >>
> > > > > >> Metrics PRs. #5068 merged. #5204 needs reshaping onto the split.
> > > > #4115 and
> > > > > >> #4756 stay open with action on the author to follow up on review
> > > > comments.
> > > > > >>
> > > > > >> [1]
> > > > > >>
> > > > > >>
> > > >
> > >
> > https://docs.google.com/document/d/1mj3vGpix9zTDTFFJKtWT5ZxI3PTGI1YNb5xFqyWrOkc/edit?tab=t.0
> > > > > >>
> > > > > >> On Wed, Jul 15, 2026 at 11:01 AM Anand Kumar Sankaran via dev <
> > > > > >> dev@polaris.apache.org> wrote:
> > > > > >>
> > > > > >> > Thanks EJ.  My plan is to introduce PR0 in a new branch / PR and
> > > > > >> repurpose
> > > > > >> > #4115 and #4756 for PR1 and PR2 to maintain all the valuable
> > > > discussions
> > > > > >> > that happened there.
> > > > > >> >
> > > > > >> > From: EJ Wang <ej...@gmail.com>
> > > > > >> > Date: Wednesday, July 15, 2026 at 10:41 AM
> > > > > >> > To: dev@polaris.apache.org <de...@polaris.apache.org>
> > > > > >> > Subject: Re: Proposal for REST endpoints for table metrics and
> > > > events
> > > > > >> >
> > > > > >> > Hi folks, Thanks again for the discussion today. I updated the
> > > sync
> > > > doc
> > > > > >> > with notes from today's metrics sync: https: //urldefense.
> > > > > >> com/v3/__https:
> > > > > >> > //docs. google.
> > > > > >> >
> > > > > >>
> > > >
> > >
> > com/document/d/100h7c4damrUzVuquYbBHM0EvA4LSWuW2IT2dN_7nYVA/edit?pli=1&tab=t.
> > > > > >> >
> > > > > >>
> > > >
> > >
> > 2ahd7wze5f__;!!Iz9xO38YGHZK!4rnlVIqBQQQ6xpGe1PMV58edOS_cGrKoqdfhw4qSp_JQCavvEjaxXk0OwGPnxyEcO60AzKQK_XDtp7H3DdkFXcrXJwAz1g0$
> > > > > >> >
> > > > > >> >
> > > > > >> > Hi folks,
> > > > > >> >
> > > > > >> > Thanks again for the discussion today. I updated the sync doc
> > with
> > > > notes
> > > > > >> > from today's metrics sync:
> > > > > >> >
> > > > > >> >
> > > > > >>
> > > >
> > >
> > https://urldefense.com/v3/__https://docs.google.com/document/d/100h7c4damrUzVuquYbBHM0EvA4LSWuW2IT2dN_7nYVA/edit?pli=1&tab=t.2ahd7wze5f__;!!Iz9xO38YGHZK!4rnlVIqBQQQ6xpGe1PMV58edOS_cGrKoqdfhw4qSp_JQCavvEjaxXk0OwGPnxyEcO60AzKQK_XDtp7H3DdkFXcrXJwAz1g0$
> > > > > >> >
> > > > > >> > See highlights below:
> > > > > >> >
> > > > > >> > In previous meetings, EJ proposed a two prs framing to push
> > > forward
> > > > the
> > > > > >> > metrics query work in polaris. Dmitri pointed out a subtle point
> > > > that
> > > > > >> pr2
> > > > > >> > is currently also trying to do the persistence related code
> > > > placement
> > > > > >> > refactor. For the sake of straightforward review process, EJ,
> > > > Dmitri and
> > > > > >> > Anand (the author) agreed to separate the persistence
> > refactoring
> > > > into
> > > > > >> pr0,
> > > > > >> > and reframing the pr scopes as follow:
> > > > > >> >
> > > > > >> >    1.
> > > > > >> >
> > > > > >> >    Pr0: refactor irc metrics spi
> > > > > >> >    1.
> > > > > >> >
> > > > > >> >       SPI should stay in core/ with metric type enveloped
> > > > > >> >       2.
> > > > > >> >
> > > > > >> >       default spi impl, no-op, in extensions/ - default
> > selection
> > > at
> > > > > >> >       runtime/
> > > > > >> >       3.
> > > > > >> >
> > > > > >> >       move metrics persistence and correspondings to extensions/
> > > as
> > > > an
> > > > > >> >       alternative SPI impl
> > > > > >> >       2.
> > > > > >> >
> > > > > >> >    Pr1: rest api for query
> > > > > >> >
> > > > > >> >
> > > > > >>
> > > >
> > >
> > https://urldefense.com/v3/__https://github.com/apache/polaris/pull/4115/changes__;!!Iz9xO38YGHZK!4rnlVIqBQQQ6xpGe1PMV58edOS_cGrKoqdfhw4qSp_JQCavvEjaxXk0OwGPnxyEcO60AzKQK_XDtp7H3DdkFXcrXFCbEbRk$
> > > > > >> >    1.
> > > > > >> >
> > > > > >> >       rest api spec in extensions/ as optional spec
> > > > > >> >       2.
> > > > > >> >
> > > > > >> >       SPI in extensions/ as optional spec spi
> > > > > >> >       3.
> > > > > >> >
> > > > > >> >       default spi impl in extensions/ no-op
> > > > > >> >       4.
> > > > > >> >
> > > > > >> >       handler related in runtime/ - thin adaptor from http shape
> > > to
> > > > spi
> > > > > >> >       shape
> > > > > >> >       3.
> > > > > >> >
> > > > > >> >    Pr2: jdbc impl for query
> > > > > >> >
> > > > > >>
> > > >
> > >
> > https://urldefense.com/v3/__https://github.com/apache/polaris/pull/4756__;!!Iz9xO38YGHZK!4rnlVIqBQQQ6xpGe1PMV58edOS_cGrKoqdfhw4qSp_JQCavvEjaxXk0OwGPnxyEcO60AzKQK_XDtp7H3DdkFXcrXRVMBzqE$
> > > > > >> >    1.
> > > > > >> >
> > > > > >> >       alternative spi impl in extensions/ -> query with jdbc
> > > > > >> >
> > > > > >> >
> > > > > >> > Thanks,
> > > > > >> > -ej
> > > > > >> >
> > > > > >> > On Wed, Jul 1, 2026 at 8:01 AM Alexandre Dutra <
> > adutra@apache.org
> > > >
> > > > > >> wrote:
> > > > > >> >
> > > > > >> > > Hi all,
> > > > > >> > >
> > > > > >> > > > While the current implementation stores metrics in the
> > > > metastore,
> > > > > >> other
> > > > > >> > > deployments
> > > > > >> > > may choose a KV store, a dedicated metrics warehouse, or
> > another
> > > > > >> backend
> > > > > >> > > entirely. In those cases, exposing a Polaris REST API over a
> > > > specific
> > > > > >> > > schema feels like coupling the API to an implementation
> > detail.
> > > > > >> > >
> > > > > >> > > I wouldn't say to an implementation detail, but rather to a
> > > > particular
> > > > > >> > > setup / configuration. I think that's fine. If you don't store
> > > > > >> > > metrics, you don't read metrics. Similar situations will
> > > certainly
> > > > > >> > > happen in the future, e.g. if we ever introduce an API to read
> > > > events,
> > > > > >> > > that would only work if events are being persisted in the
> > first
> > > > place.
> > > > > >> > > Again I think that's a fair trade-off.
> > > > > >> > >
> > > > > >> > > Thanks,
> > > > > >> > > Alex
> > > > > >> > >
> > > > > >> > > On Wed, Jul 1, 2026 at 7:23 AM Dmitri Bourlatchkov <
> > > > dimas@apache.org>
> > > > > >> > > wrote:
> > > > > >> > > >
> > > > > >> > > > Hi Yufei,
> > > > > >> > > >
> > > > > >> > > > From my POV, it is fine for Polaris to have multiple REST
> > API
> > > > > >> surfaces
> > > > > >> > > with
> > > > > >> > > > different features and different degrees of "readiness".
> > Each
> > > > API
> > > > > >> can
> > > > > >> > > > evolve fairly independently. With proper modularization
> > (which
> > > > Anand
> > > > > >> > > did),
> > > > > >> > > > changes to the main Polaris code and these extra modules
> > > should
> > > > > >> > normally
> > > > > >> > > be
> > > > > >> > > > non-intrusive to each other.
> > > > > >> > > >
> > > > > >> > > > I assume Anand is going to be the first user of this API.
> > > > However,
> > > > > >> if
> > > > > >> > we
> > > > > >> > > do
> > > > > >> > > > not merge it we'll never know whether other users are
> > > > interested. I
> > > > > >> do
> > > > > >> > > not
> > > > > >> > > > think many end users read the dev ML, but they certainly
> > read
> > > > the
> > > > > >> docs,
> > > > > >> > > > code and release notes.
> > > > > >> > > >
> > > > > >> > > > There is no risk for Polaris as a project in exposing this
> > > > > >> experimental
> > > > > >> > > > API. If usage dwindles over time, we can remove it. If it
> > > gains
> > > > a
> > > > > >> > > > considerable user / contributor base, we will promote it to
> > a
> > > > > >> > > "first-class"
> > > > > >> > > > (non-beta) API.
> > > > > >> > > >
> > > > > >> > > > Cheers,
> > > > > >> > > > Dmitri.
> > > > > >> > > >
> > > > > >> > > > On Tue, Jun 30, 2026 at 7:53 PM Yufei Gu <
> > > flyrain000@gmail.com>
> > > > > >> wrote:
> > > > > >> > > >
> > > > > >> > > > > Hi all,
> > > > > >> > > > >
> > > > > >> > > > > Thanks for driving this work forward. Building this as a
> > POC
> > > > makes
> > > > > >> > > perfect
> > > > > >> > > > > sense if the goal is to validate the idea and gather
> > > feedback.
> > > > > >> Should
> > > > > >> > > we
> > > > > >> > > > > merge a POC into the main branch before agreeing on the
> > > > overall
> > > > > >> > > direction?
> > > > > >> > > > > To me, that feels like a separate discussion from whether
> > > the
> > > > > >> > > > > implementation itself looks reasonable.
> > > > > >> > > > >
> > > > > >> > > > > One concern that I don't think has been fully addressed is
> > > > > >> whether we
> > > > > >> > > need
> > > > > >> > > > > to expose metrics consumption through a Polaris REST
> > > endpoint
> > > > at
> > > > > >> all.
> > > > > >> > > > >
> > > > > >> > > > > My concern is less about the API itself and more about the
> > > > > >> boundary
> > > > > >> > > Polaris
> > > > > >> > > > > should own. The proposed REST API is tied to a particular
> > > > > >> persistence
> > > > > >> > > > > schema, but that schema is just one possible
> > implementation.
> > > > While
> > > > > >> > the
> > > > > >> > > > > current implementation stores metrics in the metastore,
> > > other
> > > > > >> > > deployments
> > > > > >> > > > > may choose a KV store, a dedicated metrics warehouse, or
> > > > another
> > > > > >> > > backend
> > > > > >> > > > > entirely. In those cases, exposing a Polaris REST API
> > over a
> > > > > >> specific
> > > > > >> > > > > schema feels like coupling the API to an implementation
> > > > detail.
> > > > > >> > > > >
> > > > > >> > > > > Polaris isn't intended to become a full metrics system.
> > I'd
> > > > > >> expect it
> > > > > >> > > to
> > > > > >> > > > > focus on producing metrics and let deployments choose the
> > > most
> > > > > >> > > appropriate
> > > > > >> > > > > storage and consumption model for their environment.
> > > > > >> > > > >
> > > > > >> > > > > I'm not opposed to exploring this direction, but we should
> > > > first
> > > > > >> > align
> > > > > >> > > on
> > > > > >> > > > > whether exposing a REST API through Polaris is the right
> > > long
> > > > term
> > > > > >> > > approach
> > > > > >> > > > > before merging it into the main branch.
> > > > > >> > > > >
> > > > > >> > > > > Thanks,
> > > > > >> > > > > Yufei
> > > > > >> > > > >
> > > > > >> > > > > Yufei
> > > > > >> > > > >
> > > > > >> > > > >
> > > > > >> > > > > On Mon, Jun 29, 2026 at 9:27 AM Dmitri Bourlatchkov <
> > > > > >> > dimas@apache.org>
> > > > > >> > > > > wrote:
> > > > > >> > > > >
> > > > > >> > > > > > Hi All,
> > > > > >> > > > > >
> > > > > >> > > > > > It looks like Anand addressed all action items in
> > [4115].
> > > > > >> > > > > >
> > > > > >> > > > > > Please review the latest state of the code. The PR LGTM
> > > and
> > > > I
> > > > > >> > > approved in
> > > > > >> > > > > > GH from my side.
> > > > > >> > > > > >
> > > > > >> > > > > > [4115]
> > > > > >> >
> > > > > >>
> > > >
> > >
> > https://urldefense.com/v3/__https://github.com/apache/polaris/pull/4115__;!!Iz9xO38YGHZK!4rnlVIqBQQQ6xpGe1PMV58edOS_cGrKoqdfhw4qSp_JQCavvEjaxXk0OwGPnxyEcO60AzKQK_XDtp7H3DdkFXcrXtFRscKs$
> > > > > >> > > > > >
> > > > > >> > > > > > Thanks,
> > > > > >> > > > > > Dmitri.
> > > > > >> > > > > >
> > > > > >> > > > > > On Thu, Jun 25, 2026 at 1:56 PM Dmitri Bourlatchkov <
> > > > > >> > > dimas@apache.org>
> > > > > >> > > > > > wrote:
> > > > > >> > > > > >
> > > > > >> > > > > > > Hi All,
> > > > > >> > > > > > >
> > > > > >> > > > > > > Recap of the discussion in today's Community Sync:
> > > > > >> > > > > > >
> > > > > >> > > > > > > * The modular approach for Metrics SPI(s) and
> > > > implementation
> > > > > >> > seems
> > > > > >> > > to
> > > > > >> > > > > > have
> > > > > >> > > > > > > community consensus (please comment if you still have
> > > > > >> concerns)
> > > > > >> > > > > > >
> > > > > >> > > > > > > * What modules / feature flags are enabled in Apache
> > > > Polaris
> > > > > >> > > images by
> > > > > >> > > > > > > default will be discussed later (when the feature is
> > > > close to
> > > > > >> > > > > > completion).
> > > > > >> > > > > > >
> > > > > >> > > > > > > In any case, users will be able to turn the full
> > feature
> > > > > >> on/off
> > > > > >> > at
> > > > > >> > > > > their
> > > > > >> > > > > > > discretion, regardless of the default flag values.
> > > > > >> > > > > > >
> > > > > >> > > > > > > * It is generally preferable to have smaller PRs and
> > > > introduce
> > > > > >> > new
> > > > > >> > > > > > > features gradually to simplify reviews.
> > > > > >> > > > > > >
> > > > > >> > > > > > > What this means for the Metrics API, IMHO is:
> > > > > >> > > > > > >
> > > > > >> > > > > > > a) The scope of PR 4115 is probably ok as it stands
> > > right
> > > > > >> now. If
> > > > > >> > > > > someone
> > > > > >> > > > > > > wants to propose splitting it further, please comment.
> > > > > >> > > > > > >
> > > > > >> > > > > > > b) After 4115 is merged, Metrics Persistence PRs will
> > > > follow,
> > > > > >> > with
> > > > > >> > > > > > smaller
> > > > > >> > > > > > > scope per PR as appropriate for the nature of the code
> > > > > >> changes.
> > > > > >> > > > > > >
> > > > > >> > > > > > > * The major concern was about exposing the new Metrics
> > > > REST
> > > > > >> API
> > > > > >> > to
> > > > > >> > > end
> > > > > >> > > > > > > users. The nature of this concern was about avoiding
> > the
> > > > > >> (false)
> > > > > >> > > > > > impression
> > > > > >> > > > > > > that the new API is a firm and adopted Polaris
> > > > specification.
> > > > > >> > > > > > >
> > > > > >> > > > > > > I think this concern is reasonable.
> > > > > >> > > > > > >
> > > > > >> > > > > > > I propose that Anand update the documentation sections
> > > of
> > > > PR
> > > > > >> 4115
> > > > > >> > > to
> > > > > >> > > > > > > emphasise that the API is currently an experimental /
> > > POC
> > > > API
> > > > > >> at
> > > > > >> > > this
> > > > > >> > > > > > time
> > > > > >> > > > > > > and is subject to change (including breaking
> > changes). I
> > > > > >> believe
> > > > > >> > it
> > > > > >> > > > > > already
> > > > > >> > > > > > > has the "beta" label. So this request is mostly for
> > doc
> > > > > >> changes
> > > > > >> > to
> > > > > >> > > > > > clarify
> > > > > >> > > > > > > user expectations, since the "beta" label alone may
> > not
> > > be
> > > > > >> > > interpreted
> > > > > >> > > > > > the
> > > > > >> > > > > > > same way by all users.
> > > > > >> > > > > > >
> > > > > >> > > > > > > I hope this update will resolve all remaining concerns
> > > > with
> > > > > >> 4115
> > > > > >> > > and
> > > > > >> > > > > > allow
> > > > > >> > > > > > > for it to be merged so that the community will be able
> > > to
> > > > > >> proceed
> > > > > >> > > to
> > > > > >> > > > > > > follow-up PRs (as noted above).
> > > > > >> > > > > > >
> > > > > >> > > > > > > Please add/correct me if I missed anything relevant.
> > > > > >> > > > > > >
> > > > > >> > > > > > > Thanks,
> > > > > >> > > > > > > Dmitri.
> > > > > >> > > > > > >
> > > > > >> > > > > > > On Wed, Jun 24, 2026 at 8:11 PM Yufei Gu <
> > > > > >> flyrain000@gmail.com>
> > > > > >> > > wrote:
> > > > > >> > > > > > >
> > > > > >> > > > > > >> I do not think we have agreement on the query REST
> > APIs
> > > > yet.
> > > > > >> > > > > > >>
> > > > > >> > > > > > >> In particular, the current query API shape seems
> > highly
> > > > > >> coupled
> > > > > >> > > to the
> > > > > >> > > > > > >> *example[1]* metrics in the Iceberg REST spec. I do
> > not
> > > > think
> > > > > >> > > that is
> > > > > >> > > > > a
> > > > > >> > > > > > >> good way to design a REST API contract. For example,
> > > > fields
> > > > > >> like
> > > > > >> > > > > > >> resultDataFiles, resultDeleteFiles,
> > totalFileSizeBytes,
> > > > > >> > > > > > >> totalDataManifests,
> > > > > >> > > > > > >> totalDeleteManifests, and scannedDataManifests are
> > very
> > > > > >> specific
> > > > > >> > > to
> > > > > >> > > > > one
> > > > > >> > > > > > >> style of scan metric. That makes the API look more
> > > like a
> > > > > >> direct
> > > > > >> > > > > > >> projection
> > > > > >> > > > > > >> of the current example payload than a general query
> > > > model.
> > > > > >> > > > > > >>
> > > > > >> > > > > > >> My preference is that Polaris should not define or
> > > > strongly
> > > > > >> > opine
> > > > > >> > > on
> > > > > >> > > > > the
> > > > > >> > > > > > >> Iceberg scan or commit metrics query REST API at this
> > > > point.
> > > > > >> > > Polaris
> > > > > >> > > > > can
> > > > > >> > > > > > >> resolve and authorize the table, accept the metrics
> > > > report,
> > > > > >> and
> > > > > >> > > > > delegate
> > > > > >> > > > > > >> it
> > > > > >> > > > > > >> to the configured reporting path. Durable storage,
> > > > filtering,
> > > > > >> > > > > > dashboards,
> > > > > >> > > > > > >> and query APIs should remain downstream impl. choices
> > > > unless
> > > > > >> and
> > > > > >> > > until
> > > > > >> > > > > > the
> > > > > >> > > > > > >> Iceberg community standardizes a query contract,
> > which
> > > I
> > > > > >> doubt
> > > > > >> > > will
> > > > > >> > > > > > >> happen.
> > > > > >> > > > > > >>
> > > > > >> > > > > > >> So I do not think the intake path has to be blocked,
> > > but
> > > > I
> > > > > >> would
> > > > > >> > > > > prefer
> > > > > >> > > > > > >> not
> > > > > >> > > > > > >> to merge a Polaris specific query API that may be
> > hard
> > > to
> > > > > >> evolve
> > > > > >> > > > > later.
> > > > > >> > > > > > >>
> > > > > >> > > > > > >> 1.
> > > > > >> > > > > > >>
> > > > > >> > > > > > >>
> > > > > >> > > > > >
> > > > > >> > > > >
> > > > > >> > >
> > > > > >> >
> > > > > >>
> > > >
> > >
> > https://urldefense.com/v3/__https://github.com/apache/iceberg/blob/2a6c556c0c883c04c6d3fbd68e6aeda36b91e0aa/open-api/rest-catalog-open-api.yaml*L4152__;Iw!!Iz9xO38YGHZK!4rnlVIqBQQQ6xpGe1PMV58edOS_cGrKoqdfhw4qSp_JQCavvEjaxXk0OwGPnxyEcO60AzKQK_XDtp7H3DdkFXcrXCDsqYGo$
> > > > > >> > > > > > >>
> > > > > >> > > > > > >> Yufei
> > > > > >> > > > > > >>
> > > > > >> > > > > > >>
> > > > > >> > > > > > >> On Tue, Jun 23, 2026 at 9:30 AM Dmitri Bourlatchkov <
> > > > > >> > > dimas@apache.org
> > > > > >> > > > > >
> > > > > >> > > > > > >> wrote:
> > > > > >> > > > > > >>
> > > > > >> > > > > > >> > Hi All,
> > > > > >> > > > > > >> >
> > > > > >> > > > > > >> > So, what are the blockers for merging [4115], if
> > any?
> > > > > >> > > > > > >> >
> > > > > >> > > > > > >> > [4115]
> > > > > >> >
> > > > > >>
> > > >
> > >
> > https://urldefense.com/v3/__https://github.com/apache/polaris/pull/4115__;!!Iz9xO38YGHZK!4rnlVIqBQQQ6xpGe1PMV58edOS_cGrKoqdfhw4qSp_JQCavvEjaxXk0OwGPnxyEcO60AzKQK_XDtp7H3DdkFXcrXtFRscKs$
> > > > > >> > > > > > >> >
> > > > > >> > > > > > >> > Thanks,
> > > > > >> > > > > > >> > Dmitri.
> > > > > >> > > > > > >> >
> > > > > >> > > > > > >> > On Wed, Jun 17, 2026 at 5:28 PM EJ Wang <
> > > > > >> > > > > > ej.wang.dev.interest@gmail.com
> > > > > >> > > > > > >> >
> > > > > >> > > > > > >> > wrote:
> > > > > >> > > > > > >> >
> > > > > >> > > > > > >> > > Hi folks,
> > > > > >> > > > > > >> > >
> > > > > >> > > > > > >> > > Thanks again for the discussion today. I updated
> > > the
> > > > sync
> > > > > >> > doc
> > > > > >> > > with
> > > > > >> > > > > > >> notes
> > > > > >> > > > > > >> > > from today's metrics sync:
> > > > > >> > > > > > >> > >
> > > > > >> > > > > > >> > >
> > > > > >> > > > > > >> > >
> > > > > >> > > > > > >> >
> > > > > >> > > > > > >>
> > > > > >> > > > > >
> > > > > >> > > > >
> > > > > >> > >
> > > > > >> >
> > > > > >>
> > > >
> > >
> > https://urldefense.com/v3/__https://docs.google.com/document/d/100h7c4damrUzVuquYbBHM0EvA4LSWuW2IT2dN_7nYVA/edit?tab=t.ezk0rgdx0c6m__;!!Iz9xO38YGHZK!4rnlVIqBQQQ6xpGe1PMV58edOS_cGrKoqdfhw4qSp_JQCavvEjaxXk0OwGPnxyEcO60AzKQK_XDtp7H3DdkFXcrX83IZkm0$
> > > > > >> > > > > > >> > >
> > > > > >> > > > > > >> > > A few highlights:
> > > > > >> > > > > > >> > >
> > > > > >> > > > > > >> > >    - No objection to keeping the default metrics
> > > > behavior
> > > > > >> > > small:
> > > > > >> > > > > > >> > >    no-op/log-only is enough as the built-in
> > > default.
> > > > > >> > > > > > >> > >    - Durable storage, event forwarding, external
> > > > queues,
> > > > > >> > > > > dashboards,
> > > > > >> > > > > > >> and
> > > > > >> > > > > > >> > >    custom filtering should be implementation
> > > choices
> > > > > >> behind
> > > > > >> > > the
> > > > > >> > > > > > >> metrics
> > > > > >> > > > > > >> > >    reporting path.
> > > > > >> > > > > > >> > >    - For PR #4115, the REST layer should
> > > > > >> > resolve/authz/accept
> > > > > >> > > the
> > > > > >> > > > > > >> report,
> > > > > >> > > > > > >> > >    then delegate to the selected metrics
> > > > implementation.
> > > > > >> It
> > > > > >> > > should
> > > > > >> > > > > > not
> > > > > >> > > > > > >> > own
> > > > > >> > > > > > >> > >    durable storage semantics.
> > > > > >> > > > > > >> > >    - The rough boundary we discussed is: spec
> > owns
> > > > the
> > > > > >> wire
> > > > > >> > > > > > contract,
> > > > > >> > > > > > >> > >    runtime owns framework wiring, API/contract
> > > > modules
> > > > > >> own
> > > > > >> > > > > > >> > provider-facing
> > > > > >> > > > > > >> > >    contracts, and extensions own replaceable
> > > > > >> > implementations.
> > > > > >> > > > > > >> > >    - OpenLineage has a similar path-naming
> > > question.
> > > > We
> > > > > >> > should
> > > > > >> > > > > avoid
> > > > > >> > > > > > >> > >    occupying a generic lineage path too early if
> > we
> > > > want
> > > > > >> > room
> > > > > >> > > for
> > > > > >> > > > > > >> other
> > > > > >> > > > > > >> > >    lineage systems later.
> > > > > >> > > > > > >> > >
> > > > > >> > > > > > >> > > I believe the boundary rules previewed in sync
> > > > meeting
> > > > > >> are
> > > > > >> > > useful
> > > > > >> > > > > > for
> > > > > >> > > > > > >> > > reviewing PR #4115, but it is not
> > > > > >> > > > > > >> > > a final module-layout proposal.
> > > > > >> > > > > > >> > >
> > > > > >> > > > > > >> > > For metrics specifically, I think the target
> > > remains:
> > > > > >> > > > > > >> > >    runtime
> > > > > >> > > > > > >> > >      -> metrics reporting/emitting contract
> > > > > >> > > > > > >> > >      -> default implementation
> > > > > >> > > > > > >> > >           -> lower implementation details, if
> > > needed
> > > > > >> > > > > > >> > >
> > > > > >> > > > > > >> > > That keeps the default simple, while allowing
> > > durable
> > > > > >> JDBC,
> > > > > >> > > > > > >> event-backed,
> > > > > >> > > > > > >> > > or external-queue implementations to be added
> > > > separately.
> > > > > >> > > > > > >> > >
> > > > > >> > > > > > >> > > We also touched on the split between Iceberg
> > > metrics
> > > > > >> > emitting
> > > > > >> > > and
> > > > > >> > > > > > >> > > Polaris-owned metrics querying. The querying side
> > > can
> > > > > >> keep
> > > > > >> > > > > evolving,
> > > > > >> > > > > > >> > > especially for dashboard or generic-table use
> > > cases,
> > > > but
> > > > > >> I
> > > > > >> > do
> > > > > >> > > not
> > > > > >> > > > > > >> think
> > > > > >> > > > > > >> > > that needs to block the intake path in this PR.
> > > > > >> > > > > > >> > >
> > > > > >> > > > > > >> > > Thanks,
> > > > > >> > > > > > >> > > -ej
> > > > > >> > > > > > >> > >
> > > > > >> > > > > > >> > > On Tue, Jun 16, 2026 at 2:17 PM Yufei Gu <
> > > > > >> > > flyrain000@gmail.com>
> > > > > >> > > > > > >> wrote:
> > > > > >> > > > > > >> > >
> > > > > >> > > > > > >> > > > Thanks for chiming in, EJ. Agreed that the
> > > default
> > > > > >> battery
> > > > > >> > > > > should
> > > > > >> > > > > > >> stay
> > > > > >> > > > > > >> > > > small. I'm leaning toward not including metrics
> > > > > >> > persistence
> > > > > >> > > in
> > > > > >> > > > > the
> > > > > >> > > > > > >> > > default
> > > > > >> > > > > > >> > > > battery.
> > > > > >> > > > > > >> > > >
> > > > > >> > > > > > >> > > > Yufei
> > > > > >> > > > > > >> > > >
> > > > > >> > > > > > >> > > >
> > > > > >> > > > > > >> > > > On Thu, Jun 11, 2026 at 8:30 PM EJ Wang <
> > > > > >> > > > > > >> > ej.wang.dev.interest@gmail.com>
> > > > > >> > > > > > >> > > > wrote:
> > > > > >> > > > > > >> > > >
> > > > > >> > > > > > >> > > > > Hi Yufei,
> > > > > >> > > > > > >> > > > >
> > > > > >> > > > > > >> > > > > Thanks for connecting this back to the REST
> > > > endpoint
> > > > > >> > > > > proposal. I
> > > > > >> > > > > > >> > > replied
> > > > > >> > > > > > >> > > > > with the fuller version on the
> > event-forwarding
> > > > > >> thread,
> > > > > >> > > but
> > > > > >> > > > > > >> wanted to
> > > > > >> > > > > > >> > > add
> > > > > >> > > > > > >> > > > > the shorter version here too.
> > > > > >> > > > > > >> > > > >
> > > > > >> > > > > > >> > > > > I agree that metrics reporting/emitting is
> > the
> > > > right
> > > > > >> > > > > conceptual
> > > > > >> > > > > > >> > > boundary,
> > > > > >> > > > > > >> > > > > and I also think the event/listener path is a
> > > > good
> > > > > >> > > > > > implementation
> > > > > >> > > > > > >> > > > direction
> > > > > >> > > > > > >> > > > > to explore. The distinction I want to keep
> > > clear
> > > > is
> > > > > >> > > default
> > > > > >> > > > > > >> battery
> > > > > >> > > > > > >> > vs
> > > > > >> > > > > > >> > > > > extension implementation.
> > > > > >> > > > > > >> > > > >
> > > > > >> > > > > > >> > > > > For the REST/API side, I would keep the
> > > semantics
> > > > > >> > narrow:
> > > > > >> > > > > > Polaris
> > > > > >> > > > > > >> > > > resolves
> > > > > >> > > > > > >> > > > > and authorizes the table, accepts the Iceberg
> > > > > >> > scan/commit
> > > > > >> > > > > > metrics
> > > > > >> > > > > > >> > > report
> > > > > >> > > > > > >> > > > > into the configured reporting/emitting path,
> > > and
> > > > > >> returns
> > > > > >> > > 204.
> > > > > >> > > > > > That
> > > > > >> > > > > > >> > 204
> > > > > >> > > > > > >> > > > > should mean Polaris accepted the report into
> > > the
> > > > > >> > ingestion
> > > > > >> > > > > path;
> > > > > >> > > > > > >> it
> > > > > >> > > > > > >> > > > should
> > > > > >> > > > > > >> > > > > not imply durable storage.
> > > > > >> > > > > > >> > > > >
> > > > > >> > > > > > >> > > > > Durable storage, event forwarding, external
> > > > telemetry
> > > > > >> > > routing,
> > > > > >> > > > > > >> > > filtering,
> > > > > >> > > > > > >> > > > > and retention should sit behind that
> > > > > >> reporting/emitting
> > > > > >> > > > > boundary
> > > > > >> > > > > > >> as
> > > > > >> > > > > > >> > > > > implementation-layer behavior. A durable JDBC
> > > > path
> > > > > >> can
> > > > > >> > be
> > > > > >> > > one
> > > > > >> > > > > > >> named
> > > > > >> > > > > > >> > > > > extension implementation. An event/listener
> > > > > >> forwarding
> > > > > >> > > path
> > > > > >> > > > > can
> > > > > >> > > > > > be
> > > > > >> > > > > > >> > > > another
> > > > > >> > > > > > >> > > > > named extension implementation.
> > > > > >> > > > > > >> > > > >
> > > > > >> > > > > > >> > > > > So I am +1 on the event-forwarding idea as a
> > > > > >> non-default
> > > > > >> > > > > > extension
> > > > > >> > > > > > >> > > > > implementation. I just would not make it the
> > > > default
> > > > > >> > > battery
> > > > > >> > > > > or
> > > > > >> > > > > > >> the
> > > > > >> > > > > > >> > > core
> > > > > >> > > > > > >> > > > > API shape. The default battery can stay small
> > > and
> > > > > >> safe,
> > > > > >> > > while
> > > > > >> > > > > > >> > > deployments
> > > > > >> > > > > > >> > > > > that need forwarding, dashboards, or durable
> > > > storage
> > > > > >> can
> > > > > >> > > opt
> > > > > >> > > > > > into
> > > > > >> > > > > > >> the
> > > > > >> > > > > > >> > > > > implementation that matches their operational
> > > > model.
> > > > > >> > > > > > >> > > > >
> > > > > >> > > > > > >> > > > > Thanks,
> > > > > >> > > > > > >> > > > > -ej
> > > > > >> > > > > > >> > > > >
> > > > > >> > > > > > >> > > > > On Wed, Jun 10, 2026 at 11:43 AM Yufei Gu <
> > > > > >> > > > > flyrain000@gmail.com
> > > > > >> > > > > > >
> > > > > >> > > > > > >> > > wrote:
> > > > > >> > > > > > >> > > > >
> > > > > >> > > > > > >> > > > > > Thanks EJ. I agree that the
> > > reporting/emitting
> > > > > >> > boundary
> > > > > >> > > > > feels
> > > > > >> > > > > > >> like
> > > > > >> > > > > > >> > > the
> > > > > >> > > > > > >> > > > > > right SPI boundary, but I wonder if we can
> > > > simplify
> > > > > >> > this
> > > > > >> > > > > even
> > > > > >> > > > > > >> > > further.
> > > > > >> > > > > > >> > > > > >
> > > > > >> > > > > > >> > > > > > Iceberg scan and commit metrics look very
> > > > similar
> > > > > >> to
> > > > > >> > > events
> > > > > >> > > > > to
> > > > > >> > > > > > >> me.
> > > > > >> > > > > > >> > > They
> > > > > >> > > > > > >> > > > > are
> > > > > >> > > > > > >> > > > > > append only, asynchronously consumed, often
> > > > > >> forwarded
> > > > > >> > to
> > > > > >> > > > > > >> external
> > > > > >> > > > > > >> > > > > systems,
> > > > > >> > > > > > >> > > > > > and have similar retention and cleanup
> > > > > >> requirements.
> > > > > >> > > More
> > > > > >> > > > > > >> details
> > > > > >> > > > > > >> > can
> > > > > >> > > > > > >> > > > be
> > > > > >> > > > > > >> > > > > > found in this thread [1]. In that case,
> > > > Polaris may
> > > > > >> > only
> > > > > >> > > > > need
> > > > > >> > > > > > to
> > > > > >> > > > > > >> > > accept
> > > > > >> > > > > > >> > > > > the
> > > > > >> > > > > > >> > > > > > report and emit an event through the
> > existing
> > > > event
> > > > > >> > > > > framework.
> > > > > >> > > > > > >> > > > Filtering,
> > > > > >> > > > > > >> > > > > > async delivery, custom sinks, and retention
> > > > > >> mechanisms
> > > > > >> > > could
> > > > > >> > > > > > >> then
> > > > > >> > > > > > >> > be
> > > > > >> > > > > > >> > > > > shared
> > > > > >> > > > > > >> > > > > > instead of introducing a separate metrics
> > > > specific
> > > > > >> > > extension
> > > > > >> > > > > > >> layer
> > > > > >> > > > > > >> > > and
> > > > > >> > > > > > >> > > > > SPI.
> > > > > >> > > > > > >> > > > > >
> > > > > >> > > > > > >> > > > > > The main requirement I still see is
> > querying
> > > > > >> metrics
> > > > > >> > > through
> > > > > >> > > > > > >> > Polaris.
> > > > > >> > > > > > >> > > > If
> > > > > >> > > > > > >> > > > > we
> > > > > >> > > > > > >> > > > > > want that battery included experience, we
> > > could
> > > > > >> work
> > > > > >> > on
> > > > > >> > > a
> > > > > >> > > > > > >> > persistence
> > > > > >> > > > > > >> > > > > > listener implementation that selectively
> > > saves
> > > > > >> metrics
> > > > > >> > > and
> > > > > >> > > > > > >> expose
> > > > > >> > > > > > >> > it
> > > > > >> > > > > > >> > > > via
> > > > > >> > > > > > >> > > > > > the IRC event endpoint which seems pretty
> > > > close in
> > > > > >> > > Iceberg
> > > > > >> > > > > > >> > > > > > community. Another option I like more is to
> > > > allow
> > > > > >> > users
> > > > > >> > > to
> > > > > >> > > > > > >> develop
> > > > > >> > > > > > >> > > > their
> > > > > >> > > > > > >> > > > > > own listener, so they can persist
> > scan/commit
> > > > > >> metrics
> > > > > >> > > to any
> > > > > >> > > > > > >> > systems
> > > > > >> > > > > > >> > > > they
> > > > > >> > > > > > >> > > > > > prefer. In general, that feels like an
> > > > > >> implementation
> > > > > >> > > choice
> > > > > >> > > > > > on
> > > > > >> > > > > > >> top
> > > > > >> > > > > > >> > > of
> > > > > >> > > > > > >> > > > > the
> > > > > >> > > > > > >> > > > > > event framework rather than something that
> > > > needs to
> > > > > >> > > shape
> > > > > >> > > > > the
> > > > > >> > > > > > >> core
> > > > > >> > > > > > >> > > > > > architecture.
> > > > > >> > > > > > >> > > > > >
> > > > > >> > > > > > >> > > > > > 1.
> > > > > >> > > > > > >> >
> > > > > >> > >
> > > > > >> >
> > > > > >>
> > > >
> > >
> > https://urldefense.com/v3/__https://lists.apache.org/thread/x9j8nscvy8hq61tyn01mj8yp6n9of0kp__;!!Iz9xO38YGHZK!4rnlVIqBQQQ6xpGe1PMV58edOS_cGrKoqdfhw4qSp_JQCavvEjaxXk0OwGPnxyEcO60AzKQK_XDtp7H3DdkFXcrXGcGDgks$
> > > > > >> > > > > > >> > > > > >
> > > > > >> > > > > > >> > > > > > Thanks,
> > > > > >> > > > > > >> > > > > > Yufei
> > > > > >> > > > > > >> > > > > >
> > > > > >> > > > > > >> > > > > >
> > > > > >> > > > > > >> > > > > > On Thu, Jun 4, 2026 at 5:14 PM EJ Wang <
> > > > > >> > > > > > >> > > ej.wang.dev.interest@gmail.com
> > > > > >> > > > > > >> > > > >
> > > > > >> > > > > > >> > > > > > wrote:
> > > > > >> > > > > > >> > > > > >
> > > > > >> > > > > > >> > > > > > > Good point. I agree the existing
> > > > > >> > > PolarisMetricsReporter is
> > > > > >> > > > > > >> > already
> > > > > >> > > > > > >> > > > very
> > > > > >> > > > > > >> > > > > > > close to the right conceptual boundary. I
> > > am
> > > > not
> > > > > >> > > > > proposing a
> > > > > >> > > > > > >> > second
> > > > > >> > > > > > >> > > > > > > parallel reporter concept. The
> > distinction
> > > I
> > > > am
> > > > > >> > > trying to
> > > > > >> > > > > > >> make is
> > > > > >> > > > > > >> > > > > > narrower:
> > > > > >> > > > > > >> > > > > > >
> > > > > >> > > > > > >> > > > > > > Current state:
> > > > > >> > > > > > >> > > > > > >
> > > > > >> > > > > > >> > > > > > >    - PolarisMetricsReporter is the
> > existing
> > > > > >> metrics
> > > > > >> > > > > > reporting
> > > > > >> > > > > > >> > hook.
> > > > > >> > > > > > >> > > > > > >    - But it currently lives under
> > > > > >> runtime/service.
> > > > > >> > > > > > >> > > > > > >    - Its contract is documented around
> > > > > >> Quarkus/CDI
> > > > > >> > > > > discovery
> > > > > >> > > > > > >> and
> > > > > >> > > > > > >> > > > > > > @Identifier
> > > > > >> > > > > > >> > > > > > >    selection.
> > > > > >> > > > > > >> > > > > > >    - The default implementation is
> > > log-only.
> > > > > >> > > > > > >> > > > > > >    - The durable implementation writes
> > > > through
> > > > > >> > > > > > >> > > PolarisMetricsManager
> > > > > >> > > > > > >> > > > > and
> > > > > >> > > > > > >> > > > > > >    MetricsPersistence.
> > > > > >> > > > > > >> > > > > > >    - MetricsPersistence is currently
> > > > inherited by
> > > > > >> > > > > > >> > BasePersistence.
> > > > > >> > > > > > >> > > > > > >
> > > > > >> > > > > > >> > > > > > > My proposal:
> > > > > >> > > > > > >> > > > > > >
> > > > > >> > > > > > >> > > > > > >    - Keep the reporting/emitting boundary
> > > as
> > > > the
> > > > > >> > SPI.
> > > > > >> > > > > > >> > > > > > >    - Revise that contract in a
> > > > framework-agnostic
> > > > > >> > > optional
> > > > > >> > > > > > >> > Iceberg
> > > > > >> > > > > > >> > > > > > *metrics
> > > > > >> > > > > > >> > > > > > >    extension API layer*.
> > > > > >> > > > > > >> > > > > > >    - Keep runtime/service responsible for
> > > > REST
> > > > > >> > > ingestion,
> > > > > >> > > > > > >> > > table/authz
> > > > > >> > > > > > >> > > > > > >    validation, and runtime wiring.
> > > > > >> > > > > > >> > > > > > >    - Keep the battery implementation
> > > > log-only in
> > > > > >> the
> > > > > >> > > > > metrics
> > > > > >> > > > > > >> > > > extension
> > > > > >> > > > > > >> > > > > > API
> > > > > >> > > > > > >> > > > > > >    layer, *not under runtime/service*.
> > > > > >> > > > > > >> > > > > > >    - Treat durable JDBC metrics as one
> > > > > >> > implementation
> > > > > >> > > of
> > > > > >> > > > > the
> > > > > >> > > > > > >> > > > reporting
> > > > > >> > > > > > >> > > > > > SPI.
> > > > > >> > > > > > >> > > > > > >    - Decompose the MetricsPersistence and
> > > > > >> > > BasePersistence
> > > > > >> > > > > > >> > coupling
> > > > > >> > > > > > >> > > as
> > > > > >> > > > > > >> > > > > > >    durable implementation detail.
> > > > > >> > > > > > >> > > > > > >
> > > > > >> > > > > > >> > > > > > > So the difference is not “new reporter
> > > > concept vs
> > > > > >> > old
> > > > > >> > > > > > reporter
> > > > > >> > > > > > >> > > > > concept.”
> > > > > >> > > > > > >> > > > > > > but:
> > > > > >> > > > > > >> > > > > > >
> > > > > >> > > > > > >> > > > > > >
> > > > > >> > > > > > >> > > > > > >    - PolarisMetricsReporter today =
> > > > > >> runtime/service
> > > > > >> > > > > > CDI-shaped
> > > > > >> > > > > > >> > > hook.
> > > > > >> > > > > > >> > > > > > >    - Target reporting SPI = same
> > conceptual
> > > > > >> > boundary,
> > > > > >> > > but
> > > > > >> > > > > > >> placed
> > > > > >> > > > > > >> > in
> > > > > >> > > > > > >> > > > the
> > > > > >> > > > > > >> > > > > > >    right extension API layer and kept
> > > > > >> > > framework-agnostic.
> > > > > >> > > > > > >> > > > > > >
> > > > > >> > > > > > >> > > > > > > For the current metrics PR, I think the
> > > > minimal
> > > > > >> > > > > > architectural
> > > > > >> > > > > > >> > > cleanup
> > > > > >> > > > > > >> > > > > > > before merge is to *put the reporting SPI
> > > > > >> contract
> > > > > >> > > and the
> > > > > >> > > > > > >> > battery
> > > > > >> > > > > > >> > > > > > default
> > > > > >> > > > > > >> > > > > > > in the right place*. The durable JDBC
> > > > > >> > implementation,
> > > > > >> > > > > > >> > event-backed
> > > > > >> > > > > > >> > > > > async
> > > > > >> > > > > > >> > > > > > > implementation, external queue support,
> > and
> > > > > >> deeper
> > > > > >> > > > > > persistence
> > > > > >> > > > > > >> > > > cleanup
> > > > > >> > > > > > >> > > > > > can
> > > > > >> > > > > > >> > > > > > > continue as follow-ups, as long as we do
> > > not
> > > > lock
> > > > > >> > the
> > > > > >> > > SPI
> > > > > >> > > > > > >> > boundary
> > > > > >> > > > > > >> > > to
> > > > > >> > > > > > >> > > > > > > BasePersistence or to runtime/service
> > > wiring.
> > > > > >> > > > > > >> > > > > > >
> > > > > >> > > > > > >> > > > > > > -ej
> > > > > >> > > > > > >> > > > > > >
> > > > > >> > > > > > >> > > > > > > On Wed, Jun 3, 2026 at 3:00 PM Dmitri
> > > > > >> Bourlatchkov <
> > > > > >> > > > > > >> > > dimas@apache.org
> > > > > >> > > > > > >> > > > >
> > > > > >> > > > > > >> > > > > > > wrote:
> > > > > >> > > > > > >> > > > > > >
> > > > > >> > > > > > >> > > > > > > > Hi EJ,
> > > > > >> > > > > > >> > > > > > > >
> > > > > >> > > > > > >> > > > > > > > Thanks for the recap / summary!
> > > > > >> > > > > > >> > > > > > > >
> > > > > >> > > > > > >> > > > > > > > > Do folks agree that the stable SPI
> > > > boundary
> > > > > >> > > should be
> > > > > >> > > > > > >> > > > > > > > metrics reporting/emitting, not metrics
> > > > > >> > persistence?
> > > > > >> > > > > > >> > > > > > > >
> > > > > >> > > > > > >> > > > > > > > SGTM.
> > > > > >> > > > > > >> > > > > > > >
> > > > > >> > > > > > >> > > > > > > > > Does an optional Iceberg metrics
> > > > extension
> > > > > >> API
> > > > > >> > > layer
> > > > > >> > > > > > sound
> > > > > >> > > > > > >> > like
> > > > > >> > > > > > >> > > > the
> > > > > >> > > > > > >> > > > > > > right
> > > > > >> > > > > > >> > > > > > > > home for this SPI?
> > > > > >> > > > > > >> > > > > > > >
> > > > > >> > > > > > >> > > > > > > > How is that different from current
> > > > > >> > > > > PolarisMetricsReporter?
> > > > > >> > > > > > >> > > > > > > >
> > > > > >> > > > > > >> > > > > > > > > Should the current durable metrics
> > work
> > > > be
> > > > > >> > > reframed
> > > > > >> > > > > as a
> > > > > >> > > > > > >> > > durable
> > > > > >> > > > > > >> > > > > > > > JDBC reference implementation of that
> > > SPI?
> > > > > >> > > > > > >> > > > > > > >
> > > > > >> > > > > > >> > > > > > > > SGTM.
> > > > > >> > > > > > >> > > > > > > >
> > > > > >> > > > > > >> > > > > > > > > What is the smallest PR sequence to
> > get
> > > > there
> > > > > >> > > without
> > > > > >> > > > > > >> > blocking
> > > > > >> > > > > > >> > > > > > > > the current metrics work unnecessarily?
> > > > > >> > > > > > >> > > > > > > >
> > > > > >> > > > > > >> > > > > > > > Let's get
> > > > > >> > >
> > > > > >> >
> > > > > >>
> > > >
> > >
> > https://urldefense.com/v3/__https://github.com/apache/polaris/pull/4397__;!!Iz9xO38YGHZK!4rnlVIqBQQQ6xpGe1PMV58edOS_cGrKoqdfhw4qSp_JQCavvEjaxXk0OwGPnxyEcO60AzKQK_XDtp7H3DdkFXcrXuu5jefU$
> > > > > >> > > > > > >> merged
> > > > > >> > > > > > >> > > > first.
> > > > > >> > > > > > >> > > > > > > >
> > > > > >> > > > > > >> > > > > > > > Then, I'd propose isolating the Metrics
> > > > schema
> > > > > >> > from
> > > > > >> > > the
> > > > > >> > > > > > >> > MetaStore
> > > > > >> > > > > > >> > > > > > schema.
> > > > > >> > > > > > >> > > > > > > > This will probably have some ripple
> > > effect
> > > > into
> > > > > >> > the
> > > > > >> > > > > > >> bootstrap
> > > > > >> > > > > > >> > > > > > workflows,
> > > > > >> > > > > > >> > > > > > > so
> > > > > >> > > > > > >> > > > > > > > it's not a trivial change.
> > > > > >> > > > > > >> > > > > > > >
> > > > > >> > > > > > >> > > > > > > > Then, let's reassess.
> > > > > >> > > > > > >> > > > > > > >
> > > > > >> > > > > > >> > > > > > > > Cheers,
> > > > > >> > > > > > >> > > > > > > > Dmitri.
> > > > > >> > > > > > >> > > > > > > >
> > > > > >> > > > > > >> > > > > > > > On Wed, Jun 3, 2026 at 5:40 PM EJ Wang
> > <
> > > > > >> > > > > > >> > > > > ej.wang.dev.interest@gmail.com
> > > > > >> > > > > > >> > > > > > >
> > > > > >> > > > > > >> > > > > > > > wrote:
> > > > > >> > > > > > >> > > > > > > >
> > > > > >> > > > > > >> > > > > > > > > Hi folks,
> > > > > >> > > > > > >> > > > > > > > >
> > > > > >> > > > > > >> > > > > > > > > Thanks again for the discussion
> > today.
> > > I
> > > > > >> updated
> > > > > >> > > the
> > > > > >> > > > > > sync
> > > > > >> > > > > > >> doc
> > > > > >> > > > > > >> > > > with
> > > > > >> > > > > > >> > > > > > > notes
> > > > > >> > > > > > >> > > > > > > > > from the third metrics architecture
> > > sync:
> > > > > >> > > > > > >> > > > > > > > >
> > > > > >> > > > > > >> > > > > > > > >
> > > > > >> > > > > > >> > > > > > > > >
> > > > > >> > > > > > >> > > > > > > >
> > > > > >> > > > > > >> > > > > > >
> > > > > >> > > > > > >> > > > > >
> > > > > >> > > > > > >> > > > >
> > > > > >> > > > > > >> > > >
> > > > > >> > > > > > >> > >
> > > > > >> > > > > > >> >
> > > > > >> > > > > > >>
> > > > > >> > > > > >
> > > > > >> > > > >
> > > > > >> > >
> > > > > >> >
> > > > > >>
> > > >
> > >
> > https://urldefense.com/v3/__https://docs.google.com/document/d/100h7c4damrUzVuquYbBHM0EvA4LSWuW2IT2dN_7nYVA/edit?tab=t.k96s2xyqr5u1__;!!Iz9xO38YGHZK!4rnlVIqBQQQ6xpGe1PMV58edOS_cGrKoqdfhw4qSp_JQCavvEjaxXk0OwGPnxyEcO60AzKQK_XDtp7H3DdkFXcrXrqCL-QE$
> > > > > >> > > > > > >> > > > > > > > >
> > > > > >> > > > > > >> > > > > > > > > A few highlights from today’s
> > > discussion:
> > > > > >> > > > > > >> > > > > > > > >
> > > > > >> > > > > > >> > > > > > > > >    - We clarified that Iceberg
> > metrics
> > > > > >> reporting
> > > > > >> > > can
> > > > > >> > > > > be
> > > > > >> > > > > > >> > > > interpreted
> > > > > >> > > > > > >> > > > > > > > either
> > > > > >> > > > > > >> > > > > > > > >    as sync or async handling from
> > > > Polaris’s
> > > > > >> > > > > perspective,
> > > > > >> > > > > > >> so
> > > > > >> > > > > > >> > > > Polaris
> > > > > >> > > > > > >> > > > > > > > should
> > > > > >> > > > > > >> > > > > > > > >    stay flexible as a platform
> > instead
> > > of
> > > > > >> baking
> > > > > >> > > one
> > > > > >> > > > > > >> handling
> > > > > >> > > > > > >> > > > model
> > > > > >> > > > > > >> > > > > > > into
> > > > > >> > > > > > >> > > > > > > > > the
> > > > > >> > > > > > >> > > > > > > > >    REST API behavior.
> > > > > >> > > > > > >> > > > > > > > >    - We aligned that the built-in
> > > > (battery)
> > > > > >> > > behavior
> > > > > >> > > > > for
> > > > > >> > > > > > >> > > metrics
> > > > > >> > > > > > >> > > > > > > emitting
> > > > > >> > > > > > >> > > > > > > > >    can stay simple: no-op/log-only is
> > > > enough
> > > > > >> as
> > > > > >> > > the
> > > > > >> > > > > > >> default.
> > > > > >> > > > > > >> > > > > > > > >    - Durable metrics persistence
> > should
> > > > be
> > > > > >> > > treated as
> > > > > >> > > > > an
> > > > > >> > > > > > >> > > > > > implementation
> > > > > >> > > > > > >> > > > > > > > of
> > > > > >> > > > > > >> > > > > > > > >    the metrics reporting path, not as
> > > the
> > > > > >> core
> > > > > >> > SPI
> > > > > >> > > > > > >> boundary.
> > > > > >> > > > > > >> > > > > > > > >    - The existing durable metrics
> > work
> > > > can be
> > > > > >> > > reviewed
> > > > > >> > > > > > as
> > > > > >> > > > > > >> a
> > > > > >> > > > > > >> > > > > reference
> > > > > >> > > > > > >> > > > > > > > >    implementation of the reporting
> > SPI,
> > > > with
> > > > > >> > > > > > >> > > persistence-related
> > > > > >> > > > > > >> > > > > > logic
> > > > > >> > > > > > >> > > > > > > > kept
> > > > > >> > > > > > >> > > > > > > > >    self-contained under a metrics
> > > durable
> > > > > >> > > > > implementation
> > > > > >> > > > > > >> > module
> > > > > >> > > > > > >> > > > > > rather
> > > > > >> > > > > > >> > > > > > > > than
> > > > > >> > > > > > >> > > > > > > > >    scattered through core entity
> > > > persistence.
> > > > > >> > > > > > >> > > > > > > > >    - Dashboard/insights remains a
> > real
> > > > use
> > > > > >> case,
> > > > > >> > > but
> > > > > >> > > > > we
> > > > > >> > > > > > >> > agreed
> > > > > >> > > > > > >> > > to
> > > > > >> > > > > > >> > > > > > keep
> > > > > >> > > > > > >> > > > > > > it
> > > > > >> > > > > > >> > > > > > > > >    separate from the core metrics
> > > intake
> > > > > >> > > discussion
> > > > > >> > > > > for
> > > > > >> > > > > > >> now.
> > > > > >> > > > > > >> > > > > > > > >
> > > > > >> > > > > > >> > > > > > > > > I also did a quick source check after
> > > the
> > > > > >> > meeting
> > > > > >> > > to
> > > > > >> > > > > > make
> > > > > >> > > > > > >> > sure
> > > > > >> > > > > > >> > > we
> > > > > >> > > > > > >> > > > > are
> > > > > >> > > > > > >> > > > > > > > > describing the current state
> > > accurately.
> > > > > >> > > > > > >> > > > > > > > >
> > > > > >> > > > > > >> > > > > > > > > Current state:
> > > > > >> > > > > > >> > > > > > > > >
> > > > > >> > > > > > >> > > > > > > > >    - Polaris already has a metrics
> > > > reporting
> > > > > >> > hook:
> > > > > >> > > > > > >> > > > > > > > PolarisMetricsReporter.
> > > > > >> > > > > > >> > > > > > > > >    - The default implementation is
> > > > > >> > > > > > DefaultMetricsReporter,
> > > > > >> > > > > > >> > > > selected
> > > > > >> > > > > > >> > > > > > by
> > > > > >> > > > > > >> > > > > > > > >
> > > > > >> > > polaris.iceberg-metrics.reporting.type=default. It
> > > > > >> > > > > is
> > > > > >> > > > > > >> > > > log-only,
> > > > > >> > > > > > >> > > > > > > > >    effectively quiet unless metrics
> > > > logging
> > > > > >> is
> > > > > >> > > > > enabled.
> > > > > >> > > > > > >> > > > > > > > >    - There is also a
> > > > > >> PersistingMetricsReporter,
> > > > > >> > > > > selected
> > > > > >> > > > > > >> by
> > > > > >> > > > > > >> > > > > > > > >
> > > > > >> > > polaris.iceberg-metrics.reporting.type=persisting,
> > > > > >> > > > > > >> which
> > > > > >> > > > > > >> > > > > converts
> > > > > >> > > > > > >> > > > > > > > >    Iceberg scan/commit reports into
> > > > Polaris
> > > > > >> > > metrics
> > > > > >> > > > > > >> records
> > > > > >> > > > > > >> > and
> > > > > >> > > > > > >> > > > > > writes
> > > > > >> > > > > > >> > > > > > > > > through
> > > > > >> > > > > > >> > > > > > > > >    PolarisMetricsManager ->
> > > > > >> MetricsPersistence.
> > > > > >> > > > > > >> > > > > > > > >    - MetricsPersistence currently
> > > exists
> > > > in
> > > > > >> the
> > > > > >> > > > > > >> persistence
> > > > > >> > > > > > >> > > layer
> > > > > >> > > > > > >> > > > > and
> > > > > >> > > > > > >> > > > > > > is
> > > > > >> > > > > > >> > > > > > > > >    inherited by BasePersistence.
> > > > > >> > > > > > >> > > > > > > > >
> > > > > >> > > > > > >> > > > > > > > > My read from the discussion is that
> > the
> > > > > >> target
> > > > > >> > > > > boundary
> > > > > >> > > > > > >> > should
> > > > > >> > > > > > >> > > > not
> > > > > >> > > > > > >> > > > > be
> > > > > >> > > > > > >> > > > > > > > > MetricsPersistence as inherited by
> > > > > >> > > BasePersistence.
> > > > > >> > > > > That
> > > > > >> > > > > > >> path
> > > > > >> > > > > > >> > > > > should
> > > > > >> > > > > > >> > > > > > be
> > > > > >> > > > > > >> > > > > > > > > decomposed as durable implementation
> > > > detail
> > > > > >> > (taken
> > > > > >> > > > > care
> > > > > >> > > > > > >> of by
> > > > > >> > > > > > >> > > > > > > > >
> > > > > >> >
> > > > > >>
> > > >
> > >
> > https://urldefense.com/v3/__https://github.com/apache/polaris/pull/4397__;!!Iz9xO38YGHZK!4rnlVIqBQQQ6xpGe1PMV58edOS_cGrKoqdfhw4qSp_JQCavvEjaxXk0OwGPnxyEcO60AzKQK_XDtp7H3DdkFXcrXuu5jefU$
> > > > > >> ).
> > > > > >> > The
> > > > > >> > > > > > stable
> > > > > >> > > > > > >> > > > extension
> > > > > >> > > > > > >> > > > > > > point
> > > > > >> > > > > > >> > > > > > > > > should instead be the metrics
> > > > > >> reporting/emitting
> > > > > >> > > > > > boundary.
> > > > > >> > > > > > >> > > > > > > > >
> > > > > >> > > > > > >> > > > > > > > > Concretely, I think the next proposal
> > > > should
> > > > > >> be
> > > > > >> > > shaped
> > > > > >> > > > > > >> like
> > > > > >> > > > > > >> > > this:
> > > > > >> > > > > > >> > > > > > > > >
> > > > > >> > > > > > >> > > > > > > > >    1.
> > > > > >> > > > > > >> > > > > > > > >
> > > > > >> > > > > > >> > > > > > > > >    Define the proper metrics
> > reporting
> > > > SPI
> > > > > >> at an
> > > > > >> > > > > > optional
> > > > > >> > > > > > >> > > Iceberg
> > > > > >> > > > > > >> > > > > > > metrics
> > > > > >> > > > > > >> > > > > > > > >    extension API layer.
> > > > > >> > > > > > >> > > > > > > > >
> > > > > >> > > > > > >> > > > > > > > >    Example direction:
> > > > > >> > > > > > >> > > > > > > > >
> > > > > >> > > > > > >> > > > > > > > >    public interface
> > > > IcebergMetricsReporter {
> > > > > >> > > > > > >> > > > > > > > >      void
> > > > > >> > reportMetric(IcebergMetricsReportContext
> > > > > >> > > > > > >> context,
> > > > > >> > > > > > >> > > > > > > > > MetricsReport report);
> > > > > >> > > > > > >> > > > > > > > >    }
> > > > > >> > > > > > >> > > > > > > > >
> > > > > >> > > > > > >> > > > > > > > >    The context would carry the small
> > > > > >> > > Polaris-resolved
> > > > > >> > > > > > >> > envelope:
> > > > > >> > > > > > >> > > > > > > > >    catalog/table identity, report
> > type,
> > > > > >> received
> > > > > >> > > > > > >> timestamp,
> > > > > >> > > > > > >> > and
> > > > > >> > > > > > >> > > > > > > > > request/trace
> > > > > >> > > > > > >> > > > > > > > >    context if available. The raw
> > > Iceberg
> > > > > >> > > MetricsReport
> > > > > >> > > > > > >> > remains
> > > > > >> > > > > > >> > > > the
> > > > > >> > > > > > >> > > > > > > > payload.
> > > > > >> > > > > > >> > > > > > > > >    2.
> > > > > >> > > > > > >> > > > > > > > >
> > > > > >> > > > > > >> > > > > > > > >    Keep runtime/service as the REST
> > > > ingestion
> > > > > >> > and
> > > > > >> > > > > wiring
> > > > > >> > > > > > >> > layer.
> > > > > >> > > > > > >> > > > > > > > >
> > > > > >> > > > > > >> > > > > > > > >    The REST handler still resolves
> > the
> > > > table
> > > > > >> and
> > > > > >> > > > > > performs
> > > > > >> > > > > > >> > authz
> > > > > >> > > > > > >> > > > > > before
> > > > > >> > > > > > >> > > > > > > > >    accepting the report. After that,
> > it
> > > > calls
> > > > > >> > the
> > > > > >> > > > > > selected
> > > > > >> > > > > > >> > > > > reporting
> > > > > >> > > > > > >> > > > > > > > >    implementation.
> > > > > >> > > > > > >> > > > > > > > >    3.
> > > > > >> > > > > > >> > > > > > > > >
> > > > > >> > > > > > >> > > > > > > > >    Keep the battery default
> > > > no-op/log-only.
> > > > > >> > > > > > >> > > > > > > > >
> > > > > >> > > > > > >> > > > > > > > >    This preserves an out-of-box safe
> > > > default
> > > > > >> and
> > > > > >> > > > > avoids
> > > > > >> > > > > > >> > > > requiring a
> > > > > >> > > > > > >> > > > > > > > durable
> > > > > >> > > > > > >> > > > > > > > >    metrics store for every Polaris
> > > > > >> deployment.
> > > > > >> > > > > > >> > > > > > > > >    4.
> > > > > >> > > > > > >> > > > > > > > >
> > > > > >> > > > > > >> > > > > > > > >    Move durable JDBC metrics into a
> > > > > >> > self-contained
> > > > > >> > > > > > >> > > implementation
> > > > > >> > > > > > >> > > > > of
> > > > > >> > > > > > >> > > > > > > the
> > > > > >> > > > > > >> > > > > > > > >    reporting SPI.
> > > > > >> > > > > > >> > > > > > > > >
> > > > > >> > > > > > >> > > > > > > > >    That implementation can own its
> > > > schema,
> > > > > >> > > bootstrap,
> > > > > >> > > > > > >> > > retention,
> > > > > >> > > > > > >> > > > > and
> > > > > >> > > > > > >> > > > > > > read
> > > > > >> > > > > > >> > > > > > > > >    API support. It should not define
> > > the
> > > > core
> > > > > >> > > > > reporting
> > > > > >> > > > > > >> SPI
> > > > > >> > > > > > >> > > > > boundary,
> > > > > >> > > > > > >> > > > > > > and
> > > > > >> > > > > > >> > > > > > > > > it
> > > > > >> > > > > > >> > > > > > > > >    should not require metrics
> > > > persistence to
> > > > > >> > > remain
> > > > > >> > > > > > >> inherited
> > > > > >> > > > > > >> > > > from
> > > > > >> > > > > > >> > > > > > > > >    BasePersistence.
> > > > > >> > > > > > >> > > > > > > > >    5.
> > > > > >> > > > > > >> > > > > > > > >
> > > > > >> > > > > > >> > > > > > > > >    Treat async/event-backed handling
> > as
> > > > > >> another
> > > > > >> > > > > > >> > implementation
> > > > > >> > > > > > >> > > of
> > > > > >> > > > > > >> > > > > the
> > > > > >> > > > > > >> > > > > > > > same
> > > > > >> > > > > > >> > > > > > > > >    reporting SPI.
> > > > > >> > > > > > >> > > > > > > > >
> > > > > >> > > > > > >> > > > > > > > >    For example, an event-backed
> > > reporter
> > > > > >> could
> > > > > >> > > > > enqueue a
> > > > > >> > > > > > >> > > metrics
> > > > > >> > > > > > >> > > > > > event
> > > > > >> > > > > > >> > > > > > > > and
> > > > > >> > > > > > >> > > > > > > > >    let listeners handle durable
> > storage
> > > > or
> > > > > >> other
> > > > > >> > > > > sinks.
> > > > > >> > > > > > >> If we
> > > > > >> > > > > > >> > > > later
> > > > > >> > > > > > >> > > > > > > need
> > > > > >> > > > > > >> > > > > > > > a
> > > > > >> > > > > > >> > > > > > > > >    replaceable queue engine, that
> > seems
> > > > like
> > > > > >> a
> > > > > >> > > shared
> > > > > >> > > > > > >> > > > event/metrics
> > > > > >> > > > > > >> > > > > > > > > substrate
> > > > > >> > > > > > >> > > > > > > > >    topic rather than a metrics-only
> > > > > >> requirement.
> > > > > >> > > > > > >> > > > > > > > >
> > > > > >> > > > > > >> > > > > > > > > This framing lets us keep the REST
> > > > metrics
> > > > > >> > > endpoint
> > > > > >> > > > > > >> simple,
> > > > > >> > > > > > >> > > > > preserve
> > > > > >> > > > > > >> > > > > > > the
> > > > > >> > > > > > >> > > > > > > > > current default behavior, support
> > > durable
> > > > > >> > metrics
> > > > > >> > > > > users,
> > > > > >> > > > > > >> and
> > > > > >> > > > > > >> > > > still
> > > > > >> > > > > > >> > > > > > > leave
> > > > > >> > > > > > >> > > > > > > > > room for async/event-backed or
> > > > > >> > > external-queue-based
> > > > > >> > > > > > >> > > > > implementations.
> > > > > >> > > > > > >> > > > > > > > >
> > > > > >> > > > > > >> > > > > > > > > I think the main follow-up questions
> > > are:
> > > > > >> > > > > > >> > > > > > > > >
> > > > > >> > > > > > >> > > > > > > > >    - Do folks agree that the stable
> > SPI
> > > > > >> boundary
> > > > > >> > > > > should
> > > > > >> > > > > > be
> > > > > >> > > > > > >> > > > metrics
> > > > > >> > > > > > >> > > > > > > > >    reporting/emitting, not metrics
> > > > > >> persistence?
> > > > > >> > > > > > >> > > > > > > > >    - Does an optional Iceberg metrics
> > > > > >> extension
> > > > > >> > > API
> > > > > >> > > > > > layer
> > > > > >> > > > > > >> > sound
> > > > > >> > > > > > >> > > > > like
> > > > > >> > > > > > >> > > > > > > the
> > > > > >> > > > > > >> > > > > > > > >    right home for this SPI?
> > > > > >> > > > > > >> > > > > > > > >    - Should the current durable
> > metrics
> > > > work
> > > > > >> be
> > > > > >> > > > > reframed
> > > > > >> > > > > > >> as a
> > > > > >> > > > > > >> > > > > durable
> > > > > >> > > > > > >> > > > > > > > JDBC
> > > > > >> > > > > > >> > > > > > > > >    reference implementation of that
> > > SPI?
> > > > > >> > > > > > >> > > > > > > > >    - What is the smallest PR sequence
> > > to
> > > > get
> > > > > >> > there
> > > > > >> > > > > > without
> > > > > >> > > > > > >> > > > blocking
> > > > > >> > > > > > >> > > > > > the
> > > > > >> > > > > > >> > > > > > > > >    current metrics work
> > unnecessarily?
> > > > > >> > > > > > >> > > > > > > > >
> > > > > >> > > > > > >> > > > > > > > > Thanks,
> > > > > >> > > > > > >> > > > > > > > > -ej
> > > > > >> > > > > > >> > > > > > > > >
> > > > > >> > > > > > >> > > > > > > > > On Fri, May 15, 2026 at 5:16 PM
> > Dmitri
> > > > > >> > > Bourlatchkov <
> > > > > >> > > > > > >> > > > > > dimas@apache.org>
> > > > > >> > > > > > >> > > > > > > > > wrote:
> > > > > >> > > > > > >> > > > > > > > >
> > > > > >> > > > > > >> > > > > > > > > > Hi JB,
> > > > > >> > > > > > >> > > > > > > > > >
> > > > > >> > > > > > >> > > > > > > > > > Could you set up another meeting,
> > > > please?
> > > > > >> Same
> > > > > >> > > time
> > > > > >> > > > > on
> > > > > >> > > > > > >> > > > Wednesday
> > > > > >> > > > > > >> > > > > as
> > > > > >> > > > > > >> > > > > > > > last
> > > > > >> > > > > > >> > > > > > > > > > time... I hope it works for
> > everyone.
> > > > > >> > > > > > >> > > > > > > > > >
> > > > > >> > > > > > >> > > > > > > > > > Cheers,
> > > > > >> > > > > > >> > > > > > > > > > Dmitri.
> > > > > >> > > > > > >> > > > > > > > > >
> > > > > >> > > > > > >> > > > > > > > > > On Fri, May 15, 2026 at 8:06 PM
> > Yufei
> > > > Gu <
> > > > > >> > > > > > >> > > flyrain000@gmail.com
> > > > > >> > > > > > >> > > > >
> > > > > >> > > > > > >> > > > > > > wrote:
> > > > > >> > > > > > >> > > > > > > > > >
> > > > > >> > > > > > >> > > > > > > > > > > +1 on another sync call next
> > week.
> > > > > >> > > > > > >> > > > > > > > > > >
> > > > > >> > > > > > >> > > > > > > > > > > Yufei
> > > > > >> > > > > > >> > > > > > > > > > >
> > > > > >> > > > > > >> > > > > > > > > > >
> > > > > >> > > > > > >> > > > > > > > > > > On Fri, May 15, 2026 at 4:52 PM
> > > > Dmitri
> > > > > >> > > > > Bourlatchkov
> > > > > >> > > > > > <
> > > > > >> > > > > > >> > > > > > > > dimas@apache.org>
> > > > > >> > > > > > >> > > > > > > > > > > wrote:
> > > > > >> > > > > > >> > > > > > > > > > >
> > > > > >> > > > > > >> > > > > > > > > > > > Hi All,
> > > > > >> > > > > > >> > > > > > > > > > > >
> > > > > >> > > > > > >> > > > > > > > > > > > WDYT about another sync call
> > next
> > > > week?
> > > > > >> > > > > > >> > > > > > > > > > > >
> > > > > >> > > > > > >> > > > > > > > > > > > Thanks,
> > > > > >> > > > > > >> > > > > > > > > > > > Dmitri.
> > > > > >> > > > > > >> > > > > > > > > > > >
> > > > > >> > > > > > >> > > > > > > > > > > > On Wed, May 6, 2026 at 5:29 PM
> > > > Dmitri
> > > > > >> > > > > > Bourlatchkov <
> > > > > >> > > > > > >> > > > > > > > dimas@apache.org
> > > > > >> > > > > > >> > > > > > > > > >
> > > > > >> > > > > > >> > > > > > > > > > > > wrote:
> > > > > >> > > > > > >> > > > > > > > > > > >
> > > > > >> > > > > > >> > > > > > > > > > > > > Hi EJ,
> > > > > >> > > > > > >> > > > > > > > > > > > >
> > > > > >> > > > > > >> > > > > > > > > > > > > Thanks for the summary! It
> > > covers
> > > > > >> what
> > > > > >> > we
> > > > > >> > > > > > >> discussed
> > > > > >> > > > > > >> > in
> > > > > >> > > > > > >> > > > the
> > > > > >> > > > > > >> > > > > > > > meeting
> > > > > >> > > > > > >> > > > > > > > > > very
> > > > > >> > > > > > >> > > > > > > > > > > > > well, IMHO.
> > > > > >> > > > > > >> > > > > > > > > > > > >
> > > > > >> > > > > > >> > > > > > > > > > > > > Looking forward to concrete
> > PRs
> > > > :)
> > > > > >> > > > > > >> > > > > > > > > > > > >
> > > > > >> > > > > > >> > > > > > > > > > > > > Cheers,
> > > > > >> > > > > > >> > > > > > > > > > > > > Dmitri.
> > > > > >> > > > > > >> > > > > > > > > > > > >
> > > > > >> > > > > > >> > > > > > > > > > > > > On Wed, May 6, 2026 at
> > 5:08 PM
> > > EJ
> > > > > >> Wang <
> > > > > >> > > > > > >> > > > > > > > > > ej.wang.dev.interest@gmail.com
> > > > > >> > > > > > >> > > > > > > > > > > >
> > > > > >> > > > > > >> > > > > > > > > > > > > wrote:
> > > > > >> > > > > > >> > > > > > > > > > > > >
> > > > > >> > > > > > >> > > > > > > > > > > > >> Hi folks,
> > > > > >> > > > > > >> > > > > > > > > > > > >>
> > > > > >> > > > > > >> > > > > > > > > > > > >> We had a community sync
> > > earlier,
> > > > > >> thanks
> > > > > >> > > JB
> > > > > >> > > > > for
> > > > > >> > > > > > >> > > > scheduling
> > > > > >> > > > > > >> > > > > > it.
> > > > > >> > > > > > >> > > > > > > > > Notes
> > > > > >> > > > > > >> > > > > > > > > > > from
> > > > > >> > > > > > >> > > > > > > > > > > > >> the first metrics
> > architecture
> > > > sync
> > > > > >> > (May
> > > > > >> > > 6,
> > > > > >> > > > > > >> 10-11am
> > > > > >> > > > > > >> > > PT).
> > > > > >> > > > > > >> > > > > > > > > Discussion
> > > > > >> > > > > > >> > > > > > > > > > > doc
> > > > > >> > > > > > >> > > > > > > > > > > > >> with per-section status:
> > > > > >> > > > > > >> > > > > > > > > > > > >>
> > > > > >> > > > > > >> > > > > > > > > > > > >>
> > > > > >> > > > > > >> > > > > > > > > > > >
> > > > > >> > > > > > >> > > > > > > > > > >
> > > > > >> > > > > > >> > > > > > > > > >
> > > > > >> > > > > > >> > > > > > > > >
> > > > > >> > > > > > >> > > > > > > >
> > > > > >> > > > > > >> > > > > > >
> > > > > >> > > > > > >> > > > > >
> > > > > >> > > > > > >> > > > >
> > > > > >> > > > > > >> > > >
> > > > > >> > > > > > >> > >
> > > > > >> > > > > > >> >
> > > > > >> > > > > > >>
> > > > > >> > > > > >
> > > > > >> > > > >
> > > > > >> > >
> > > > > >> >
> > > > > >>
> > > >
> > >
> > https://urldefense.com/v3/__https://docs.google.com/document/d/100h7c4damrUzVuquYbBHM0EvA4LSWuW2IT2dN_7nYVA/edit?tab=t.0__;!!Iz9xO38YGHZK!4rnlVIqBQQQ6xpGe1PMV58edOS_cGrKoqdfhw4qSp_JQCavvEjaxXk0OwGPnxyEcO60AzKQK_XDtp7H3DdkFXcrXgIddeUs$
> > > > > >> > > > > > >> > > > > > > > > > > > >>
> > > > > >> > > > > > >> > > > > > > > > > > > >> *The meeting covered both
> > > topics
> > > > > >> from
> > > > > >> > the
> > > > > >> > > > > doc.
> > > > > >> > > > > > >> > > > > > Direction-level
> > > > > >> > > > > > >> > > > > > > > > > > alignment
> > > > > >> > > > > > >> > > > > > > > > > > > >> was reached on the headline
> > > > pieces;
> > > > > >> > > details
> > > > > >> > > > > > >> remain
> > > > > >> > > > > > >> > for
> > > > > >> > > > > > >> > > > PR
> > > > > >> > > > > > >> > > > > > > review
> > > > > >> > > > > > >> > > > > > > > > or
> > > > > >> > > > > > >> > > > > > > > > > > > >> follow-up sessions.*
> > > > > >> > > > > > >> > > > > > > > > > > > >>
> > > > > >> > > > > > >> > > > > > > > > > > > >> *Topic 1 — Persistence
> > schema
> > > > > >> redesign*
> > > > > >> > > > > > >> > > > > > > > > > > > >> Idea-level alignment on
> > > > > >> consolidating
> > > > > >> > > > > per-type
> > > > > >> > > > > > >> > tables
> > > > > >> > > > > > >> > > > > > > > > > > > >> (scan_metrics_report,
> > > > > >> > > > > > >> > > > > > > > > > > > >> commit_metrics_report) into
> > a
> > > > single
> > > > > >> > > > > > >> metrics_report
> > > > > >> > > > > > >> > > > table.
> > > > > >> > > > > > >> > > > > > The
> > > > > >> > > > > > >> > > > > > > > > > > > motivating
> > > > > >> > > > > > >> > > > > > > > > > > > >> cost is the surface area
> > added
> > > > by
> > > > > >> every
> > > > > >> > > new
> > > > > >> > > > > > >> metric
> > > > > >> > > > > > >> > > type
> > > > > >> > > > > > >> > > > > > today:
> > > > > >> > > > > > >> > > > > > > > new
> > > > > >> > > > > > >> > > > > > > > > > > > table,
> > > > > >> > > > > > >> > > > > > > > > > > > >> SPI method, record class,
> > > model,
> > > > > >> > > converter,
> > > > > >> > > > > > >> schema
> > > > > >> > > > > > >> > > > > > migration.
> > > > > >> > > > > > >> > > > > > > > > > > > >>
> > > > > >> > > > > > >> > > > > > > > > > > > >> Most schema details are
> > > > deferred to
> > > > > >> the
> > > > > >> > > > > schema
> > > > > >> > > > > > >> PR. A
> > > > > >> > > > > > >> > > few
> > > > > >> > > > > > >> > > > > > > > specific
> > > > > >> > > > > > >> > > > > > > > > > > points
> > > > > >> > > > > > >> > > > > > > > > > > > >> came up:
> > > > > >> > > > > > >> > > > > > > > > > > > >> •   metric_schema_version:
> > > Yufei
> > > > > >> > prefers
> > > > > >> > > > > > dropping
> > > > > >> > > > > > >> > it,
> > > > > >> > > > > > >> > > > > since
> > > > > >> > > > > > >> > > > > > > > there
> > > > > >> > > > > > >> > > > > > > > > is
> > > > > >> > > > > > >> > > > > > > > > > > no
> > > > > >> > > > > > >> > > > > > > > > > > > >> spec-level concept of
> > metrics
> > > > > >> > versioning
> > > > > >> > > > > today
> > > > > >> > > > > > >> and
> > > > > >> > > > > > >> > it
> > > > > >> > > > > > >> > > is
> > > > > >> > > > > > >> > > > > > hard
> > > > > >> > > > > > >> > > > > > > to
> > > > > >> > > > > > >> > > > > > > > > > > define
> > > > > >> > > > > > >> > > > > > > > > > > > >> unilaterally. Robert prefers
> > > > keeping
> > > > > >> > it,
> > > > > >> > > > > given
> > > > > >> > > > > > >> IRC
> > > > > >> > > > > > >> > v2
> > > > > >> > > > > > >> > > is
> > > > > >> > > > > > >> > > > > > > coming
> > > > > >> > > > > > >> > > > > > > > > and
> > > > > >> > > > > > >> > > > > > > > > > > the
> > > > > >> > > > > > >> > > > > > > > > > > > >> schema should be considered
> > > > against
> > > > > >> its
> > > > > >> > > > > likely
> > > > > >> > > > > > >> > shape;
> > > > > >> > > > > > >> > > > > Robert
> > > > > >> > > > > > >> > > > > > > > also
> > > > > >> > > > > > >> > > > > > > > > > > raised
> > > > > >> > > > > > >> > > > > > > > > > > > >> how to differentiate various
> > > > payload
> > > > > >> > > formats
> > > > > >> > > > > if
> > > > > >> > > > > > >> any.
> > > > > >> > > > > > >> > > > EJ's
> > > > > >> > > > > > >> > > > > > read
> > > > > >> > > > > > >> > > > > > > > is
> > > > > >> > > > > > >> > > > > > > > > > that
> > > > > >> > > > > > >> > > > > > > > > > > > >> this
> > > > > >> > > > > > >> > > > > > > > > > > > >> is a two-way-door decision.
> > We
> > > > can
> > > > > >> > start
> > > > > >> > > > > > without
> > > > > >> > > > > > >> the
> > > > > >> > > > > > >> > > > > field,
> > > > > >> > > > > > >> > > > > > > and
> > > > > >> > > > > > >> > > > > > > > if
> > > > > >> > > > > > >> > > > > > > > > > IRC
> > > > > >> > > > > > >> > > > > > > > > > > > v2
> > > > > >> > > > > > >> > > > > > > > > > > > >> changes the shape we would
> > > > likely
> > > > > >> roll
> > > > > >> > a
> > > > > >> > > > > > >> > corresponding
> > > > > >> > > > > > >> > > > new
> > > > > >> > > > > > >> > > > > > > > schema
> > > > > >> > > > > > >> > > > > > > > > > > > anyway,
> > > > > >> > > > > > >> > > > > > > > > > > > >> which is not particularly
> > > > costly.
> > > > > >> > > > > > >> > > > > > > > > > > > >> •   Payload format: Robert
> > > > pointed
> > > > > >> out
> > > > > >> > > that
> > > > > >> > > > > > >> future
> > > > > >> > > > > > >> > > > formats
> > > > > >> > > > > > >> > > > > > > > beyond
> > > > > >> > > > > > >> > > > > > > > > > JSON
> > > > > >> > > > > > >> > > > > > > > > > > > may
> > > > > >> > > > > > >> > > > > > > > > > > > >> be worth supporting. The
> > exact
> > > > > >> shape is
> > > > > >> > > > > > deferred
> > > > > >> > > > > > >> to
> > > > > >> > > > > > >> > > the
> > > > > >> > > > > > >> > > > > > schema
> > > > > >> > > > > > >> > > > > > > > > > > > discussion.
> > > > > >> > > > > > >> > > > > > > > > > > > >> •   Partition strategy:
> > Anand
> > > > > >> suggested
> > > > > >> > > > > monthly
> > > > > >> > > > > > >> > > > > partitioning
> > > > > >> > > > > > >> > > > > > > > based
> > > > > >> > > > > > >> > > > > > > > > > on
> > > > > >> > > > > > >> > > > > > > > > > > > his
> > > > > >> > > > > > >> > > > > > > > > > > > >> experience as potentially
> > > > helpful at
> > > > > >> > > scale.
> > > > > >> > > > > > >> > > > > > > > > > > > >>
> > > > > >> > > > > > >> > > > > > > > > > > > >> *Topic 2 — Where metrics
> > > > ingestion
> > > > > >> and
> > > > > >> > > > > storage
> > > > > >> > > > > > >> > belong*
> > > > > >> > > > > > >> > > > > > > > > > > > >> Idea-level alignment that
> > > > metrics
> > > > > >> > should
> > > > > >> > > be a
> > > > > >> > > > > > >> > > separated
> > > > > >> > > > > > >> > > > > SPI
> > > > > >> > > > > > >> > > > > > > from
> > > > > >> > > > > > >> > > > > > > > > the
> > > > > >> > > > > > >> > > > > > > > > > > > >> entity
> > > > > >> > > > > > >> > > > > > > > > > > > >> persistence stack. Two
> > reasons
> > > > > >> > surfaced:
> > > > > >> > > (a)
> > > > > >> > > > > > >> > workloads
> > > > > >> > > > > > >> > > > and
> > > > > >> > > > > > >> > > > > > > > > > capability
> > > > > >> > > > > > >> > > > > > > > > > > > >> requirements diverge enough
> > > that
> > > > > >> > coupling
> > > > > >> > > > > them
> > > > > >> > > > > > >> > creates
> > > > > >> > > > > > >> > > > > > > > artificial
> > > > > >> > > > > > >> > > > > > > > > > > > >> constraints, and (b) admin
> > > > > >> experience
> > > > > >> > > > > improves
> > > > > >> > > > > > >> when
> > > > > >> > > > > > >> > > > > metrics
> > > > > >> > > > > > >> > > > > > > has
> > > > > >> > > > > > >> > > > > > > > > its
> > > > > >> > > > > > >> > > > > > > > > > > own
> > > > > >> > > > > > >> > > > > > > > > > > > >> bootstrap, retention, and
> > > > lifecycle.
> > > > > >> > > Dmitri
> > > > > >> > > > > > noted
> > > > > >> > > > > > >> > > > Polaris
> > > > > >> > > > > > >> > > > > > > being
> > > > > >> > > > > > >> > > > > > > > a
> > > > > >> > > > > > >> > > > > > > > > > > > platform
> > > > > >> > > > > > >> > > > > > > > > > > > >> should have the flexibility
> > to
> > > > > >> support
> > > > > >> > > > > > different
> > > > > >> > > > > > >> > > > > persistence
> > > > > >> > > > > > >> > > > > > > > > > backends
> > > > > >> > > > > > >> > > > > > > > > > > > per
> > > > > >> > > > > > >> > > > > > > > > > > > >> concern, and pointed to a
> > > > concrete
> > > > > >> next
> > > > > >> > > step
> > > > > >> > > > > of
> > > > > >> > > > > > >> > > > separating
> > > > > >> > > > > > >> > > > > > the
> > > > > >> > > > > > >> > > > > > > > > JDBC
> > > > > >> > > > > > >> > > > > > > > > > > > >> bootstrap for metrics from
> > the
> > > > > >> > metastore
> > > > > >> > > > > > >> bootstrap.
> > > > > >> > > > > > >> > > > Robert
> > > > > >> > > > > > >> > > > > > > > > proposed
> > > > > >> > > > > > >> > > > > > > > > > an
> > > > > >> > > > > > >> > > > > > > > > > > > >> additional UX extension:
> > > detect
> > > > an
> > > > > >> > > > > > unbootstrapped
> > > > > >> > > > > > >> > > > metrics
> > > > > >> > > > > > >> > > > > > > store
> > > > > >> > > > > > >> > > > > > > > on
> > > > > >> > > > > > >> > > > > > > > > > > first
> > > > > >> > > > > > >> > > > > > > > > > > > >> use and auto-bootstrap
> > rather
> > > > than
> > > > > >> > > requiring
> > > > > >> > > > > an
> > > > > >> > > > > > >> > > explicit
> > > > > >> > > > > > >> > > > > > > manual
> > > > > >> > > > > > >> > > > > > > > > > > > bootstrap
> > > > > >> > > > > > >> > > > > > > > > > > > >> step.
> > > > > >> > > > > > >> > > > > > > > > > > > >> The meeting also confirmed
> > > that
> > > > > >> Polaris
> > > > > >> > > > > metrics
> > > > > >> > > > > > >> can
> > > > > >> > > > > > >> > > > start
> > > > > >> > > > > > >> > > > > > > small
> > > > > >> > > > > > >> > > > > > > > > and
> > > > > >> > > > > > >> > > > > > > > > > > stay
> > > > > >> > > > > > >> > > > > > > > > > > > >> Iceberg-focused. Naming and
> > > > > >> persistence
> > > > > >> > > > > schema
> > > > > >> > > > > > >> can
> > > > > >> > > > > > >> > > lean
> > > > > >> > > > > > >> > > > > > > > > > > > Iceberg-specific.
> > > > > >> > > > > > >> > > > > > > > > > > > >> If a future expansion to
> > > > > >> generic-table
> > > > > >> > > > > metrics
> > > > > >> > > > > > or
> > > > > >> > > > > > >> > > > > > operational
> > > > > >> > > > > > >> > > > > > > > > > metrics
> > > > > >> > > > > > >> > > > > > > > > > > > >> arrives, an abstraction
> > layer
> > > > can be
> > > > > >> > > built on
> > > > > >> > > > > > >> top of
> > > > > >> > > > > > >> > > the
> > > > > >> > > > > > >> > > > > > > Iceberg
> > > > > >> > > > > > >> > > > > > > > > > > metrics
> > > > > >> > > > > > >> > > > > > > > > > > > >> reporter at that point.
> > Robert
> > > > > >> remains
> > > > > >> > > on the
> > > > > >> > > > > > >> fence
> > > > > >> > > > > > >> > > and
> > > > > >> > > > > > >> > > > > > would
> > > > > >> > > > > > >> > > > > > > > > prefer
> > > > > >> > > > > > >> > > > > > > > > > > > >> something more generic but
> > did
> > > > not
> > > > > >> > block
> > > > > >> > > the
> > > > > >> > > > > > >> > > direction;
> > > > > >> > > > > > >> > > > > > > Dmitri's
> > > > > >> > > > > > >> > > > > > > > > > read
> > > > > >> > > > > > >> > > > > > > > > > > > was
> > > > > >> > > > > > >> > > > > > > > > > > > >> that the proposed framework
> > > > already
> > > > > >> has
> > > > > >> > > > > enough
> > > > > >> > > > > > >> > > > flexibility
> > > > > >> > > > > > >> > > > > > to
> > > > > >> > > > > > >> > > > > > > > > absorb
> > > > > >> > > > > > >> > > > > > > > > > > > >> future
> > > > > >> > > > > > >> > > > > > > > > > > > >> expansion.
> > > > > >> > > > > > >> > > > > > > > > > > > >>
> > > > > >> > > > > > >> > > > > > > > > > > > >> The Trade-offs and Proposed
> > > > > >> structure
> > > > > >> > > > > sections
> > > > > >> > > > > > in
> > > > > >> > > > > > >> > the
> > > > > >> > > > > > >> > > > doc
> > > > > >> > > > > > >> > > > > > were
> > > > > >> > > > > > >> > > > > > > > not
> > > > > >> > > > > > >> > > > > > > > > > > > >> reviewed
> > > > > >> > > > > > >> > > > > > > > > > > > >> in detail. They remain open
> > > for
> > > > > >> either
> > > > > >> > > the
> > > > > >> > > > > next
> > > > > >> > > > > > >> sync
> > > > > >> > > > > > >> > > or
> > > > > >> > > > > > >> > > > PR
> > > > > >> > > > > > >> > > > > > > > review.
> > > > > >> > > > > > >> > > > > > > > > > > > >>
> > > > > >> > > > > > >> > > > > > > > > > > > >> *Cross-cutting alignment —
> > > > > >> > > battery-included
> > > > > >> > > > > > plus
> > > > > >> > > > > > >> > > > > pluggable*
> > > > > >> > > > > > >> > > > > > > > > > > > >> A common philosophy emerged
> > > > from the
> > > > > >> > > > > > discussion.
> > > > > >> > > > > > >> EJ
> > > > > >> > > > > > >> > > > > > summarized
> > > > > >> > > > > > >> > > > > > > > it
> > > > > >> > > > > > >> > > > > > > > > > as:
> > > > > >> > > > > > >> > > > > > > > > > > > >> Polaris should provide a
> > > > > >> > > battery-included UX
> > > > > >> > > > > > for
> > > > > >> > > > > > >> > > > beginners
> > > > > >> > > > > > >> > > > > > and
> > > > > >> > > > > > >> > > > > > > > the
> > > > > >> > > > > > >> > > > > > > > > > > > >> flexibility for advanced
> > users
> > > > to
> > > > > >> swap
> > > > > >> > > the
> > > > > >> > > > > > >> included
> > > > > >> > > > > > >> > > > > battery
> > > > > >> > > > > > >> > > > > > > for
> > > > > >> > > > > > >> > > > > > > > > > > > something
> > > > > >> > > > > > >> > > > > > > > > > > > >> more powerful or tailored to
> > > > their
> > > > > >> use
> > > > > >> > > case.
> > > > > >> > > > > > The
> > > > > >> > > > > > >> SPI
> > > > > >> > > > > > >> > > > > design
> > > > > >> > > > > > >> > > > > > > > needs
> > > > > >> > > > > > >> > > > > > > > > to
> > > > > >> > > > > > >> > > > > > > > > > > > >> enable
> > > > > >> > > > > > >> > > > > > > > > > > > >> both.
> > > > > >> > > > > > >> > > > > > > > > > > > >>
> > > > > >> > > > > > >> > > > > > > > > > > > >> The inputs that shaped this
> > > > framing:
> > > > > >> > > > > > >> > > > > > > > > > > > >> •   Anand described how his
> > > team
> > > > > >> uses
> > > > > >> > the
> > > > > >> > > > > > current
> > > > > >> > > > > > >> > > > metrics
> > > > > >> > > > > > >> > > > > > > > > > persistence
> > > > > >> > > > > > >> > > > > > > > > > > > >> (three metrics consumers in
> > > > v1.4).
> > > > > >> > > > > > >> > > > > > > > > > > > >> •   Yufei raised Grafana and
> > > > > >> dashboard
> > > > > >> > > > > > >> integrations
> > > > > >> > > > > > >> > > as a
> > > > > >> > > > > > >> > > > > > > > > destination
> > > > > >> > > > > > >> > > > > > > > > > > use
> > > > > >> > > > > > >> > > > > > > > > > > > >> case beyond the default.
> > > > > >> > > > > > >> > > > > > > > > > > > >> •   Robert called out that
> > the
> > > > > >> current
> > > > > >> > > design
> > > > > >> > > > > > is
> > > > > >> > > > > > >> > more
> > > > > >> > > > > > >> > > > > > > > > JDBC-focused.
> > > > > >> > > > > > >> > > > > > > > > > > > >>
> > > > > >> > > > > > >> > > > > > > > > > > > >> Two concrete instances:
> > > > > >> > > > > > >> > > > > > > > > > > > >> •   Async metrics intake:
> > > > Yufei's
> > > > > >> > initial
> > > > > >> > > > > > >> position
> > > > > >> > > > > > >> > was
> > > > > >> > > > > > >> > > > > that
> > > > > >> > > > > > >> > > > > > > > async
> > > > > >> > > > > > >> > > > > > > > > > > should
> > > > > >> > > > > > >> > > > > > > > > > > > >> largely live on the producer
> > > > side
> > > > > >> and
> > > > > >> > > there
> > > > > >> > > > > is
> > > > > >> > > > > > >> not
> > > > > >> > > > > > >> > > much
> > > > > >> > > > > > >> > > > > > > Polaris
> > > > > >> > > > > > >> > > > > > > > > can
> > > > > >> > > > > > >> > > > > > > > > > > do.
> > > > > >> > > > > > >> > > > > > > > > > > > >> Robert suggested a
> > > Polaris-side
> > > > > >> default
> > > > > >> > > is
> > > > > >> > > > > > doable
> > > > > >> > > > > > >> > via
> > > > > >> > > > > > >> > > > > > Vert.x.
> > > > > >> > > > > > >> > > > > > > > > Dmitri
> > > > > >> > > > > > >> > > > > > > > > > > > >> agreed
> > > > > >> > > > > > >> > > > > > > > > > > > >> the direction is worth
> > > > exploring.
> > > > > >> The
> > > > > >> > > meeting
> > > > > >> > > > > > >> > > converged
> > > > > >> > > > > > >> > > > > on a
> > > > > >> > > > > > >> > > > > > > > > > > > >> battery-included default
> > > (likely
> > > > > >> > > > > Vert.x-backed)
> > > > > >> > > > > > >> with
> > > > > >> > > > > > >> > > an
> > > > > >> > > > > > >> > > > > SPI
> > > > > >> > > > > > >> > > > > > > > shape
> > > > > >> > > > > > >> > > > > > > > > > that
> > > > > >> > > > > > >> > > > > > > > > > > > >> lets
> > > > > >> > > > > > >> > > > > > > > > > > > >> power users route to a more
> > > > scalable
> > > > > >> > > backend
> > > > > >> > > > > > >> > > (k8s-hosted
> > > > > >> > > > > > >> > > > > > > queue,
> > > > > >> > > > > > >> > > > > > > > > AWS
> > > > > >> > > > > > >> > > > > > > > > > > SQS,
> > > > > >> > > > > > >> > > > > > > > > > > > >> etc.).
> > > > > >> > > > > > >> > > > > > > > > > > > >> •   Pluggable destinations:
> > > > > >> combining
> > > > > >> > > Yufei's
> > > > > >> > > > > > >> > > dashboard
> > > > > >> > > > > > >> > > > > use
> > > > > >> > > > > > >> > > > > > > case
> > > > > >> > > > > > >> > > > > > > > > > with
> > > > > >> > > > > > >> > > > > > > > > > > > >> Robert's JDBC-focused
> > > call-out,
> > > > the
> > > > > >> > > meeting
> > > > > >> > > > > > >> agreed
> > > > > >> > > > > > >> > the
> > > > > >> > > > > > >> > > > SPI
> > > > > >> > > > > > >> > > > > > > > should
> > > > > >> > > > > > >> > > > > > > > > be
> > > > > >> > > > > > >> > > > > > > > > > > > >> structured for multiple
> > sinks
> > > so
> > > > > >> > > integrations
> > > > > >> > > > > > >> become
> > > > > >> > > > > > >> > > > impl
> > > > > >> > > > > > >> > > > > > > > choices
> > > > > >> > > > > > >> > > > > > > > > > > rather
> > > > > >> > > > > > >> > > > > > > > > > > > >> than architectural changes.
> > > > > >> > > > > > >> > > > > > > > > > > > >>
> > > > > >> > > > > > >> > > > > > > > > > > > >> The battery-included default
> > > is
> > > > most
> > > > > >> > > likely
> > > > > >> > > > > to
> > > > > >> > > > > > >> use
> > > > > >> > > > > > >> > the
> > > > > >> > > > > > >> > > > > > > existing
> > > > > >> > > > > > >> > > > > > > > > > > > >> JDBC-backed
> > > > > >> > > > > > >> > > > > > > > > > > > >> approach.
> > > > > >> > > > > > >> > > > > > > > > > > > >>
> > > > > >> > > > > > >> > > > > > > > > > > > >> *Direction (idea-level
> > > > alignment)*
> > > > > >> > > > > > >> > > > > > > > > > > > >> •   Single metrics_report
> > > table
> > > > > >> > > consolidating
> > > > > >> > > > > > >> > per-type
> > > > > >> > > > > > >> > > > > > > metrics,
> > > > > >> > > > > > >> > > > > > > > > > > > replacing
> > > > > >> > > > > > >> > > > > > > > > > > > >> scan_metrics_report and
> > > > > >> > > commit_metrics_report
> > > > > >> > > > > > >> > > > > > > > > > > > >> •   Iceberg-focused naming
> > and
> > > > > >> schema
> > > > > >> > for
> > > > > >> > > > > now,
> > > > > >> > > > > > >> > revisit
> > > > > >> > > > > > >> > > > if
> > > > > >> > > > > > >> > > > > > > > > > > generic-table
> > > > > >> > > > > > >> > > > > > > > > > > > or
> > > > > >> > > > > > >> > > > > > > > > > > > >> operational metrics arrive
> > > > > >> > > > > > >> > > > > > > > > > > > >> •   Metrics persistence as a
> > > > > >> separated
> > > > > >> > > SPI,
> > > > > >> > > > > not
> > > > > >> > > > > > >> on
> > > > > >> > > > > > >> > > > > > > > BasePersistence
> > > > > >> > > > > > >> > > > > > > > > > > > >> •   Bootstrap path separated
> > > for
> > > > > >> > metrics,
> > > > > >> > > > > > >> > independent
> > > > > >> > > > > > >> > > of
> > > > > >> > > > > > >> > > > > > > > metastore
> > > > > >> > > > > > >> > > > > > > > > > > > >> bootstrap
> > > > > >> > > > > > >> > > > > > > > > > > > >> •   "Battery-included plus
> > > > > >> pluggable"
> > > > > >> > as
> > > > > >> > > the
> > > > > >> > > > > > SPI
> > > > > >> > > > > > >> > > design
> > > > > >> > > > > > >> > > > > > > > philosophy
> > > > > >> > > > > > >> > > > > > > > > > > > >>
> > > > > >> > > > > > >> > > > > > > > > > > > >> *Open items*
> > > > > >> > > > > > >> > > > > > > > > > > > >> •   Schema details:
> > > > > >> > > metric_schema_version,
> > > > > >> > > > > > >> payload
> > > > > >> > > > > > >> > > > format,
> > > > > >> > > > > > >> > > > > > IRC
> > > > > >> > > > > > >> > > > > > > > v2
> > > > > >> > > > > > >> > > > > > > > > > > > >> forward-compat shape
> > > > > >> > > > > > >> > > > > > > > > > > > >> •   SPI design details —
> > full
> > > > review
> > > > > >> > > either
> > > > > >> > > > > in
> > > > > >> > > > > > >> the
> > > > > >> > > > > > >> > > next
> > > > > >> > > > > > >> > > > > sync
> > > > > >> > > > > > >> > > > > > > or
> > > > > >> > > > > > >> > > > > > > > in
> > > > > >> > > > > > >> > > > > > > > > > the
> > > > > >> > > > > > >> > > > > > > > > > > > >> corresponding PR
> > > > > >> > > > > > >> > > > > > > > > > > > >> •   Schema refactor PR
> > > ownership
> > > > > >> > > > > > >> > > > > > > > > > > > >>
> > > > > >> > > > > > >> > > > > > > > > > > > >> *Action items*
> > > > > >> > > > > > >> > > > > > > > > > > > >> •   EJ to take a first stab
> > at
> > > > the
> > > > > >> SPI
> > > > > >> > > design
> > > > > >> > > > > > and
> > > > > >> > > > > > >> > > > > > potentially
> > > > > >> > > > > > >> > > > > > > > > > partner
> > > > > >> > > > > > >> > > > > > > > > > > > with
> > > > > >> > > > > > >> > > > > > > > > > > > >> Anand to incorporate the
> > > lessons
> > > > > >> > learned
> > > > > >> > > from
> > > > > >> > > > > > the
> > > > > >> > > > > > >> > > > existing
> > > > > >> > > > > > >> > > > > > > > > reporter
> > > > > >> > > > > > >> > > > > > > > > > > and
> > > > > >> > > > > > >> > > > > > > > > > > > >> persistence work.
> > > > > >> > > > > > >> > > > > > > > > > > > >> •   Schema refactor PR
> > > > ownership is
> > > > > >> not
> > > > > >> > > yet
> > > > > >> > > > > > >> decided.
> > > > > >> > > > > > >> > > If
> > > > > >> > > > > > >> > > > > > anyone
> > > > > >> > > > > > >> > > > > > > > is
> > > > > >> > > > > > >> > > > > > > > > > > > >> interested in driving it,
> > > reply
> > > > on
> > > > > >> this
> > > > > >> > > > > thread.
> > > > > >> > > > > > >> > > > > > > > > > > > >> •   JB to schedule the next
> > > > sync,
> > > > > >> > > tentatively
> > > > > >> > > > > > in
> > > > > >> > > > > > >> two
> > > > > >> > > > > > >> > > > > weeks.
> > > > > >> > > > > > >> > > > > > > > > > > > >>
> > > > > >> > > > > > >> > > > > > > > > > > > >> -ej
> > > > > >> > > > > > >> > > > > > > > > > > > >>
> > > > > >> > > > > > >> > > > > > > > > > > > >> On Mon, Apr 27, 2026 at
> > > 3:07 PM
> > > > EJ
> > > > > >> > Wang <
> > > > > >> > > > > > >> > > > > > > > > > > ej.wang.dev.interest@gmail.com
> > > > > >> > > > > > >> > > > > > > > > > > > >
> > > > > >> > > > > > >> > > > > > > > > > > > >> wrote:
> > > > > >> > > > > > >> > > > > > > > > > > > >>
> > > > > >> > > > > > >> > > > > > > > > > > > >> > Thanks Yufei for the +1.
> > > > > >> > > > > > >> > > > > > > > > > > > >> >
> > > > > >> > > > > > >> > > > > > > > > > > > >> > JB, could you help add a
> > > > biweekly
> > > > > >> > > metrics
> > > > > >> > > > > > >> > > architecture
> > > > > >> > > > > > >> > > > > > sync
> > > > > >> > > > > > >> > > > > > > to
> > > > > >> > > > > > >> > > > > > > > > the
> > > > > >> > > > > > >> > > > > > > > > > > > >> Polaris
> > > > > >> > > > > > >> > > > > > > > > > > > >> > community calendar? I'm
> > > > thinking
> > > > > >> > > Thursdays
> > > > > >> > > > > at
> > > > > >> > > > > > >> > 9-10am
> > > > > >> > > > > > >> > > > PT,
> > > > > >> > > > > > >> > > > > > on
> > > > > >> > > > > > >> > > > > > > > the
> > > > > >> > > > > > >> > > > > > > > > > > > >> off-weeks
> > > > > >> > > > > > >> > > > > > > > > > > > >> > from the community meeting
> > > > > >> (starting
> > > > > >> > > May
> > > > > >> > > > > 7),
> > > > > >> > > > > > 60
> > > > > >> > > > > > >> > > > minutes.
> > > > > >> > > > > > >> > > > > > > > > > > > >> >
> > > > > >> > > > > > >> > > > > > > > > > > > >> > Here's a rough agenda to
> > > work
> > > > > >> through
> > > > > >> > > over
> > > > > >> > > > > > the
> > > > > >> > > > > > >> > first
> > > > > >> > > > > > >> > > > few
> > > > > >> > > > > > >> > > > > > > > > sessions,
> > > > > >> > > > > > >> > > > > > > > > > > > >> grouped
> > > > > >> > > > > > >> > > > > > > > > > > > >> > by priority:
> > > > > >> > > > > > >> > > > > > > > > > > > >> >
> > > > > >> > > > > > >> > > > > > > > > > > > >> > *First: foundational
> > > > direction*
> > > > > >> > > > > > >> > > > > > > > > > > > >> >
> > > > > >> > > > > > >> > > > > > > > > > > > >> > 1.  MetricsPersistence:
> > > public
> > > > > >> SPI or
> > > > > >> > > > > > internal
> > > > > >> > > > > > >> > > > > > > implementation
> > > > > >> > > > > > >> > > > > > > > > > > detail?
> > > > > >> > > > > > >> > > > > > > > > > > > >> >    •   Marked @Beta,
> > javadoc
> > > > calls
> > > > > >> > it a
> > > > > >> > > > > > >> "Service
> > > > > >> > > > > > >> > > > > Provider
> > > > > >> > > > > > >> > > > > > > > > > > Interface",
> > > > > >> > > > > > >> > > > > > > > > > > > >> but
> > > > > >> > > > > > >> > > > > > > > > > > > >> > only one consumer
> > > > > >> > > > > (JdbcBasePersistenceImpl),
> > > > > >> > > > > > >> lives
> > > > > >> > > > > > >> > > on
> > > > > >> > > > > > >> > > > > > > > > > > BasePersistence.
> > > > > >> > > > > > >> > > > > > > > > > > > >> If
> > > > > >> > > > > > >> > > > > > > > > > > > >> > demoted to a private
> > helper
> > > > > >> inside a
> > > > > >> > > > > > persisting
> > > > > >> > > > > > >> > > > reporter
> > > > > >> > > > > > >> > > > > > > impl,
> > > > > >> > > > > > >> > > > > > > > > > most
> > > > > >> > > > > > >> > > > > > > > > > > > >> > downstream design
> > decisions
> > > > become
> > > > > >> > > > > > >> implementation
> > > > > >> > > > > > >> > > > > details
> > > > > >> > > > > > >> > > > > > > > rather
> > > > > >> > > > > > >> > > > > > > > > > > than
> > > > > >> > > > > > >> > > > > > > > > > > > >> > contract questions.
> > > > > >> > > > > > >> > > > > > > > > > > > >> >
> > > > > >> > > > > > >> > > > > > > > > > > > >> > 2.  Persistence schema
> > > > redesign
> > > > > >> > > > > > >> > > > > > > > > > > > >> >    •   Current two-table
> > > > layout
> > > > > >> > > > > > >> > > (scan_metrics_report,
> > > > > >> > > > > > >> > > > > > > > > > > > >> > commit_metrics_report)
> > with
> > > > ~25
> > > > > >> > > flattened
> > > > > >> > > > > > >> columns
> > > > > >> > > > > > >> > > > each.
> > > > > >> > > > > > >> > > > > > > Every
> > > > > >> > > > > > >> > > > > > > > > new
> > > > > >> > > > > > >> > > > > > > > > > > > metric
> > > > > >> > > > > > >> > > > > > > > > > > > >> > type requires a new table,
> > > SPI
> > > > > >> > method,
> > > > > >> > > > > record
> > > > > >> > > > > > >> > class,
> > > > > >> > > > > > >> > > > > > model,
> > > > > >> > > > > > >> > > > > > > > > > > converter,
> > > > > >> > > > > > >> > > > > > > > > > > > >> and
> > > > > >> > > > > > >> > > > > > > > > > > > >> > schema migration.
> > Direction
> > > to
> > > > > >> > explore:
> > > > > >> > > > > > single
> > > > > >> > > > > > >> > table
> > > > > >> > > > > > >> > > > > with
> > > > > >> > > > > > >> > > > > > > > > > > metric_type
> > > > > >> > > > > > >> > > > > > > > > > > > >> enum,
> > > > > >> > > > > > >> > > > > > > > > > > > >> > schema_version, and JSON
> > > > payload
> > > > > >> > > column.
> > > > > >> > > > > > >> > > > > > > > > > > > >> >
> > > > > >> > > > > > >> > > > > > > > > > > > >> > *Second: design details
> > once
> > > > > >> > direction
> > > > > >> > > is
> > > > > >> > > > > > set*
> > > > > >> > > > > > >> > > > > > > > > > > > >> >
> > > > > >> > > > > > >> > > > > > > > > > > > >> > 3.  Partition key strategy
> > > > > >> > > > > > >> > > > > > > > > > > > >> >    •   Single-table design
> > > > means
> > > > > >> scan
> > > > > >> > > > > metrics
> > > > > >> > > > > > >> at
> > > > > >> > > > > > >> > > scale
> > > > > >> > > > > > >> > > > > > will
> > > > > >> > > > > > >> > > > > > > > have
> > > > > >> > > > > > >> > > > > > > > > > > high
> > > > > >> > > > > > >> > > > > > > > > > > > >> > write concurrency per
> > table.
> > > > > >> Schema
> > > > > >> > > needs
> > > > > >> > > > > to
> > > > > >> > > > > > >> > expose
> > > > > >> > > > > > >> > > > > enough
> > > > > >> > > > > > >> > > > > > > > > > structure
> > > > > >> > > > > > >> > > > > > > > > > > > for
> > > > > >> > > > > > >> > > > > > > > > > > > >> > backends to shard by
> > entity
> > > or
> > > > > >> time
> > > > > >> > > range.
> > > > > >> > > > > > >> > > > > > > > > > > > >> >
> > > > > >> > > > > > >> > > > > > > > > > > > >> > 4.  Read/write path
> > > > consistency
> > > > > >> > > > > > >> > > > > > > > > > > > >> >    •   Writes go through
> > > > > >> > > > > > PolarisMetricsManager
> > > > > >> > > > > > >> on
> > > > > >> > > > > > >> > > > > > > > > > MetaStoreManager.
> > > > > >> > > > > > >> > > > > > > > > > > > >> Reads
> > > > > >> > > > > > >> > > > > > > > > > > > >> > bypass MetaStoreManager
> > and
> > > go
> > > > > >> > > straight to
> > > > > >> > > > > > >> > > > > > BasePersistence,
> > > > > >> > > > > > >> > > > > > > > > > > excluding
> > > > > >> > > > > > >> > > > > > > > > > > > >> > non-JDBC backends from the
> > > > read
> > > > > >> API.
> > > > > >> > > > > > >> > > > > > > > > > > > >> >
> > > > > >> > > > > > >> > > > > > > > > > > > >> > *Third: cleanup and
> > > alignment*
> > > > > >> > > > > > >> > > > > > > > > > > > >> >
> > > > > >> > > > > > >> > > > > > > > > > > > >> > 5.  PolarisMetricsReporter
> > > > naming
> > > > > >> > > > > > >> > > > > > > > > > > > >> >    •   Only handles IRC
> > > > > >> > > > > > >> (ScanReport/CommitReport),
> > > > > >> > > > > > >> > > > > doesn't
> > > > > >> > > > > > >> > > > > > > > cover
> > > > > >> > > > > > >> > > > > > > > > > > > generic
> > > > > >> > > > > > >> > > > > > > > > > > > >> > tables or operational
> > > metrics.
> > > > > >> Name
> > > > > >> > is
> > > > > >> > > > > > broader
> > > > > >> > > > > > >> > than
> > > > > >> > > > > > >> > > > > scope.
> > > > > >> > > > > > >> > > > > > > > > > > > >> >
> > > > > >> > > > > > >> > > > > > > > > > > > >> > 6.  PolarisMetricsManager
> > > > facade
> > > > > >> > > > > passthrough
> > > > > >> > > > > > >> > > > > > > > > > > > >> >    •   Entire default
> > method
> > > > is
> > > > > >> > > > > > >> > > > > > > > > > > > >>
> > > > > >> > callCtx.getMetaStore().writeScanReport().
> > > > > >> > > > > > >> > > > > > > > > > > > >> > Zero logic, passes Level 1
> > > > > >> straight
> > > > > >> > > through
> > > > > >> > > > > > to
> > > > > >> > > > > > >> > Level
> > > > > >> > > > > > >> > > > 3.
> > > > > >> > > > > > >> > > > > > Same
> > > > > >> > > > > > >> > > > > > > > > > > > >> anti-pattern
> > > > > >> > > > > > >> > > > > > > > > > > > >> > as PolarisEventManager.
> > > > > >> > > > > > >> > > > > > > > > > > > >> >
> > > > > >> > > > > > >> > > > > > > > > > > > >> > 7.  Iceberg community
> > > > alignment
> > > > > >> > > > > > >> > > > > > > > > > > > >> >    •   Payload-type
> > > extension
> > > > > >> needs
> > > > > >> > > > > > discussion
> > > > > >> > > > > > >> on
> > > > > >> > > > > > >> > > > > > > dev@iceberg.
> > > > > >> > > > > > >> > > > > > > > > > > > >> obelix74's
> > > > > >> > > > > > >> > > > > > > > > > > > >> > Feb thread got zero
> > replies.
> > > > > >> Needs a
> > > > > >> > > > > > committer
> > > > > >> > > > > > >> > > voice.
> > > > > >> > > > > > >> > > > > > > > > > > > >> >
> > > > > >> > > > > > >> > > > > > > > > > > > >> > Lets confirm
> > prioritization
> > > > in the
> > > > > >> > > first
> > > > > >> > > > > > >> session.
> > > > > >> > > > > > >> > > > > > > > > > > > >> >
> > > > > >> > > > > > >> > > > > > > > > > > > >> > -ej
> > > > > >> > > > > > >> > > > > > > > > > > > >> >
> > > > > >> > > > > > >> > > > > > > > > > > > >> > On Tue, Apr 21, 2026 at
> > > > 3:18 PM
> > > > > >> Yufei
> > > > > >> > > Gu <
> > > > > >> > > > > > >> > > > > > > > flyrain000@gmail.com>
> > > > > >> > > > > > >> > > > > > > > > > > > wrote:
> > > > > >> > > > > > >> > > > > > > > > > > > >> >
> > > > > >> > > > > > >> > > > > > > > > > > > >> >> Thanks everyone for
> > > > continuing to
> > > > > >> > > drive
> > > > > >> > > > > this
> > > > > >> > > > > > >> > > > forward. I
> > > > > >> > > > > > >> > > > > > > agree
> > > > > >> > > > > > >> > > > > > > > > > that
> > > > > >> > > > > > >> > > > > > > > > > > > the
> > > > > >> > > > > > >> > > > > > > > > > > > >> >> problem is getting
> > complex
> > > > enough
> > > > > >> > > that a
> > > > > >> > > > > > more
> > > > > >> > > > > > >> > > > > structured
> > > > > >> > > > > > >> > > > > > > > > > discussion
> > > > > >> > > > > > >> > > > > > > > > > > > >> would
> > > > > >> > > > > > >> > > > > > > > > > > > >> >> help.
> > > > > >> > > > > > >> > > > > > > > > > > > >> >>
> > > > > >> > > > > > >> > > > > > > > > > > > >> >> +1 on setting up a
> > biweekly
> > > > sync
> > > > > >> for
> > > > > >> > > the
> > > > > >> > > > > > >> metrics
> > > > > >> > > > > > >> > > > > > > > architecture.
> > > > > >> > > > > > >> > > > > > > > > > I’m
> > > > > >> > > > > > >> > > > > > > > > > > > >> happy
> > > > > >> > > > > > >> > > > > > > > > > > > >> >> to
> > > > > >> > > > > > >> > > > > > > > > > > > >> >> join.
> > > > > >> > > > > > >> > > > > > > > > > > > >> >>
> > > > > >> > > > > > >> > > > > > > > > > > > >> >> Yufei
> > > > > >> > > > > > >> > > > > > > > > > > > >> >>
> > > > > >> > > > > > >> > > > > > > > > > > > >> >>
> > > > > >> > > > > > >> > > > > > > > > > > > >> >> On Tue, Apr 21, 2026 at
> > > > 2:34 PM
> > > > > >> EJ
> > > > > >> > > Wang <
> > > > > >> > > > > > >> > > > > > > > > > > > >>
> > > ej.wang.dev.interest@gmail.com>
> > > > > >> > > > > > >> > > > > > > > > > > > >> >> wrote:
> > > > > >> > > > > > >> > > > > > > > > > > > >> >>
> > > > > >> > > > > > >> > > > > > > > > > > > >> >> > Also, I've been looking
> > > > more
> > > > > >> > > closely at
> > > > > >> > > > > > the
> > > > > >> > > > > > >> > > > > > *persistence
> > > > > >> > > > > > >> > > > > > > > > schema
> > > > > >> > > > > > >> > > > > > > > > > > in
> > > > > >> > > > > > >> > > > > > > > > > > > >> the
> > > > > >> > > > > > >> > > > > > > > > > > > >> >> > current metrics work*,
> > > and
> > > > I
> > > > > >> think
> > > > > >> > > > > > there's a
> > > > > >> > > > > > >> > > > > structural
> > > > > >> > > > > > >> > > > > > > > > > rigidity
> > > > > >> > > > > > >> > > > > > > > > > > > >> problem
> > > > > >> > > > > > >> > > > > > > > > > > > >> >> > worth raising before
> > the
> > > > shape
> > > > > >> > gets
> > > > > >> > > > > locked
> > > > > >> > > > > > >> in.
> > > > > >> > > > > > >> > > > > > > > > > > > >> >> >
> > > > > >> > > > > > >> > > > > > > > > > > > >> >> > Right now we have two
> > > > separate
> > > > > >> > > tables
> > > > > >> > > > > > >> > > > > > > (scan_metrics_report
> > > > > >> > > > > > >> > > > > > > > > and
> > > > > >> > > > > > >> > > > > > > > > > > > >> >> > commit_metrics_report),
> > > > each
> > > > > >> with
> > > > > >> > > ~25
> > > > > >> > > > > > >> flattened
> > > > > >> > > > > > >> > > > > columns
> > > > > >> > > > > > >> > > > > > > > that
> > > > > >> > > > > > >> > > > > > > > > > > > directly
> > > > > >> > > > > > >> > > > > > > > > > > > >> >> > mirror the Iceberg
> > report
> > > > > >> fields.
> > > > > >> > > The
> > > > > >> > > > > SPI
> > > > > >> > > > > > >> > follows
> > > > > >> > > > > > >> > > > the
> > > > > >> > > > > > >> > > > > > > same
> > > > > >> > > > > > >> > > > > > > > > > split:
> > > > > >> > > > > > >> > > > > > > > > > > > >> >> > writeScanReport and
> > > > > >> > > writeCommitReport as
> > > > > >> > > > > > >> > separate
> > > > > >> > > > > > >> > > > > > > methods,
> > > > > >> > > > > > >> > > > > > > > > with
> > > > > >> > > > > > >> > > > > > > > > > > > >> per-type
> > > > > >> > > > > > >> > > > > > > > > > > > >> >> > record classes,
> > > > converters, and
> > > > > >> > > model
> > > > > >> > > > > > >> objects.
> > > > > >> > > > > > >> > > *The
> > > > > >> > > > > > >> > > > > > > > practical
> > > > > >> > > > > > >> > > > > > > > > > > cost:
> > > > > >> > > > > > >> > > > > > > > > > > > >> >> > adding a new metric
> > type
> > > > > >> > > (operational
> > > > > >> > > > > > >> metrics,
> > > > > >> > > > > > >> > > for
> > > > > >> > > > > > >> > > > > > > example)
> > > > > >> > > > > > >> > > > > > > > > > > > requires
> > > > > >> > > > > > >> > > > > > > > > > > > >> a
> > > > > >> > > > > > >> > > > > > > > > > > > >> >> new
> > > > > >> > > > > > >> > > > > > > > > > > > >> >> > table, a new SPI
> > method,
> > > a
> > > > new
> > > > > >> > > record
> > > > > >> > > > > > >> class, a
> > > > > >> > > > > > >> > > new
> > > > > >> > > > > > >> > > > > > model
> > > > > >> > > > > > >> > > > > > > > > > class, a
> > > > > >> > > > > > >> > > > > > > > > > > > new
> > > > > >> > > > > > >> > > > > > > > > > > > >> >> > converter branch, and a
> > > > schema
> > > > > >> > > > > migration*.
> > > > > >> > > > > > >> > > That's a
> > > > > >> > > > > > >> > > > > lot
> > > > > >> > > > > > >> > > > > > > of
> > > > > >> > > > > > >> > > > > > > > > > > surface
> > > > > >> > > > > > >> > > > > > > > > > > > >> area
> > > > > >> > > > > > >> > > > > > > > > > > > >> >> > for what should be "one
> > > > more
> > > > > >> kind
> > > > > >> > of
> > > > > >> > > > > > >> metric."
> > > > > >> > > > > > >> > > > > > > > > > > > >> >> >
> > > > > >> > > > > > >> > > > > > > > > > > > >> >> > *My bias* would be
> > > toward a
> > > > > >> single
> > > > > >> > > > > metrics
> > > > > >> > > > > > >> > table
> > > > > >> > > > > > >> > > > with
> > > > > >> > > > > > >> > > > > > *a
> > > > > >> > > > > > >> > > > > > > > > typed
> > > > > >> > > > > > >> > > > > > > > > > > JSON
> > > > > >> > > > > > >> > > > > > > > > > > > >> >> > payload*. Something
> > like:
> > > > > >> > > metric_type
> > > > > >> > > > > > >> (enum),
> > > > > >> > > > > > >> > > > > > entity_id,
> > > > > >> > > > > > >> > > > > > > > > > > > >> >> > table_identifier,
> > > > snapshot_id
> > > > > >> > > > > (nullable),
> > > > > >> > > > > > >> > > > > received_ts,
> > > > > >> > > > > > >> > > > > > > > > > > > >> schema_version,
> > > > > >> > > > > > >> > > > > > > > > > > > >> >> and
> > > > > >> > > > > > >> > > > > > > > > > > > >> >> > a payload column for
> > the
> > > > > >> > > metric-specific
> > > > > >> > > > > > >> data.
> > > > > >> > > > > > >> > > The
> > > > > >> > > > > > >> > > > > > > > > metric_type
> > > > > >> > > > > > >> > > > > > > > > > +
> > > > > >> > > > > > >> > > > > > > > > > > > >> >> > schema_version pair
> > gives
> > > > us a
> > > > > >> > > > > > >> > forward-compatible
> > > > > >> > > > > > >> > > > > > > contract
> > > > > >> > > > > > >> > > > > > > > > for
> > > > > >> > > > > > >> > > > > > > > > > > the
> > > > > >> > > > > > >> > > > > > > > > > > > >> >> payload
> > > > > >> > > > > > >> > > > > > > > > > > > >> >> > shape. Adding a new
> > > metric
> > > > type
> > > > > >> > > becomes
> > > > > >> > > > > an
> > > > > >> > > > > > >> enum
> > > > > >> > > > > > >> > > > value
> > > > > >> > > > > > >> > > > > > > and a
> > > > > >> > > > > > >> > > > > > > > > > > payload
> > > > > >> > > > > > >> > > > > > > > > > > > >> >> schema,
> > > > > >> > > > > > >> > > > > > > > > > > > >> >> > not a schema migration.
> > > One
> > > > > >> thing
> > > > > >> > I
> > > > > >> > > > > think
> > > > > >> > > > > > we
> > > > > >> > > > > > >> > need
> > > > > >> > > > > > >> > > > to
> > > > > >> > > > > > >> > > > > be
> > > > > >> > > > > > >> > > > > > > > > > > deliberate
> > > > > >> > > > > > >> > > > > > > > > > > > >> >> about is
> > > > > >> > > > > > >> > > > > > > > > > > > >> >> > the partition key
> > design.
> > > > If
> > > > > >> all
> > > > > >> > > metric
> > > > > >> > > > > > >> types
> > > > > >> > > > > > >> > > land
> > > > > >> > > > > > >> > > > in
> > > > > >> > > > > > >> > > > > > one
> > > > > >> > > > > > >> > > > > > > > > > table,
> > > > > >> > > > > > >> > > > > > > > > > > > scan
> > > > > >> > > > > > >> > > > > > > > > > > > >> >> > metrics at scale (high
> > > > > >> > concurrency,
> > > > > >> > > high
> > > > > >> > > > > > >> > > frequency
> > > > > >> > > > > > >> > > > > > across
> > > > > >> > > > > > >> > > > > > > > > many
> > > > > >> > > > > > >> > > > > > > > > > > > >> tables)
> > > > > >> > > > > > >> > > > > > > > > > > > >> >> > could easily create hot
> > > > > >> > partitions.
> > > > > >> > > We'd
> > > > > >> > > > > > >> want
> > > > > >> > > > > > >> > the
> > > > > >> > > > > > >> > > > > > > > persistence
> > > > > >> > > > > > >> > > > > > > > > > > layer
> > > > > >> > > > > > >> > > > > > > > > > > > >> to
> > > > > >> > > > > > >> > > > > > > > > > > > >> >> be
> > > > > >> > > > > > >> > > > > > > > > > > > >> >> > able to shard by entity
> > > or
> > > > time
> > > > > >> > > range,
> > > > > >> > > > > and
> > > > > >> > > > > > >> that
> > > > > >> > > > > > >> > > > means
> > > > > >> > > > > > >> > > > > > the
> > > > > >> > > > > > >> > > > > > > > > > logical
> > > > > >> > > > > > >> > > > > > > > > > > > >> schema
> > > > > >> > > > > > >> > > > > > > > > > > > >> >> > needs to expose enough
> > > > > >> structure
> > > > > >> > for
> > > > > >> > > > > > >> backends
> > > > > >> > > > > > >> > to
> > > > > >> > > > > > >> > > > > > > partition
> > > > > >> > > > > > >> > > > > > > > > on.
> > > > > >> > > > > > >> > > > > > > > > > I
> > > > > >> > > > > > >> > > > > > > > > > > > >> don't
> > > > > >> > > > > > >> > > > > > > > > > > > >> >> > think the current
> > > flattened
> > > > > >> layout
> > > > > >> > > gives
> > > > > >> > > > > > us
> > > > > >> > > > > > >> > that.
> > > > > >> > > > > > >> > > > > > > > > > > > >> >> >
> > > > > >> > > > > > >> > > > > > > > > > > > >> >> > This is getting complex
> > > > enough
> > > > > >> > that
> > > > > >> > > I
> > > > > >> > > > > > don't
> > > > > >> > > > > > >> > think
> > > > > >> > > > > > >> > > > > > ad-hoc
> > > > > >> > > > > > >> > > > > > > > > PR/ML
> > > > > >> > > > > > >> > > > > > > > > > > > >> threads
> > > > > >> > > > > > >> > > > > > > > > > > > >> >> > will converge well.
> > > *Would
> > > > > >> people
> > > > > >> > be
> > > > > >> > > > > open
> > > > > >> > > > > > >> to a
> > > > > >> > > > > > >> > > > > biweekly
> > > > > >> > > > > > >> > > > > > > > sync
> > > > > >> > > > > > >> > > > > > > > > > for
> > > > > >> > > > > > >> > > > > > > > > > > > >> metrics
> > > > > >> > > > > > >> > > > > > > > > > > > >> >> > architecture?* I think
> > 30
> > > > > >> minutes
> > > > > >> > > every
> > > > > >> > > > > > two
> > > > > >> > > > > > >> > weeks
> > > > > >> > > > > > >> > > > > with
> > > > > >> > > > > > >> > > > > > > > > > interested
> > > > > >> > > > > > >> > > > > > > > > > > > >> >> parties
> > > > > >> > > > > > >> > > > > > > > > > > > >> >> > would be enough to work
> > > > through
> > > > > >> > the
> > > > > >> > > > > > schema,
> > > > > >> > > > > > >> SPI
> > > > > >> > > > > > >> > > > > shape,
> > > > > >> > > > > > >> > > > > > > and
> > > > > >> > > > > > >> > > > > > > > > read
> > > > > >> > > > > > >> > > > > > > > > > > API
> > > > > >> > > > > > >> > > > > > > > > > > > >> >> design
> > > > > >> > > > > > >> > > > > > > > > > > > >> >> > together. Happy to help
> > > set
> > > > > >> that
> > > > > >> > up.
> > > > > >> > > > > > >> > > > > > > > > > > > >> >> >
> > > > > >> > > > > > >> > > > > > > > > > > > >> >> > -ej
> > > > > >> > > > > > >> > > > > > > > > > > > >> >> >
> > > > > >> > > > > > >> > > > > > > > > > > > >> >> > On Mon, Apr 20, 2026 at
> > > > > >> 2:19 PM EJ
> > > > > >> > > Wang
> > > > > >> > > > > <
> > > > > >> > > > > > >> > > > > > > > > > > > >>
> > > ej.wang.dev.interest@gmail.com
> > > > > >> > > > > > >> > > > > > > > > > > > >> >> >
> > > > > >> > > > > > >> > > > > > > > > > > > >> >> > wrote:
> > > > > >> > > > > > >> > > > > > > > > > > > >> >> >
> > > > > >> > > > > > >> > > > > > > > > > > > >> >> >> Reviewed #4115, left a
> > > > > >> comment on
> > > > > >> > > the
> > > > > >> > > > > > code
> > > > > >> > > > > > >> > > > > > organization
> > > > > >> > > > > > >> > > > > > > > > side.
> > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>
> > > > > >> > > > > > >> > > > > > > > > > > > >> >> >> One thing stood out:
> > the
> > > > > >> metrics
> > > > > >> > > write
> > > > > >> > > > > > path
> > > > > >> > > > > > >> > > enters
> > > > > >> > > > > > >> > > > > > > through
> > > > > >> > > > > > >> > > > > > > > > > > > >> >> >> PolarisMetricsManager
> > on
> > > > > >> > > > > > MetaStoreManager,
> > > > > >> > > > > > >> but
> > > > > >> > > > > > >> > > the
> > > > > >> > > > > > >> > > > > new
> > > > > >> > > > > > >> > > > > > > > read
> > > > > >> > > > > > >> > > > > > > > > > path
> > > > > >> > > > > > >> > > > > > > > > > > > >> >> bypasses
> > > > > >> > > > > > >> > > > > > > > > > > > >> >> >> MetaStoreManager
> > > entirely
> > > > and
> > > > > >> > goes
> > > > > >> > > > > > >> straight to
> > > > > >> > > > > > >> > > > > > > > > BasePersistence
> > > > > >> > > > > > >> > > > > > > > > > > via
> > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>
> > > > callContext.getMetaStore().
> > > > > >> That
> > > > > >> > > means
> > > > > >> > > > > > the
> > > > > >> > > > > > >> > read
> > > > > >> > > > > > >> > > > API
> > > > > >> > > > > > >> > > > > > only
> > > > > >> > > > > > >> > > > > > > > > works
> > > > > >> > > > > > >> > > > > > > > > > > for
> > > > > >> > > > > > >> > > > > > > > > > > > >> >> backends
> > > > > >> > > > > > >> > > > > > > > > > > > >> >> >> that implement
> > > > > >> BasePersistence.
> > > > > >> > > NoSQL
> > > > > >> > > > > and
> > > > > >> > > > > > >> > remote
> > > > > >> > > > > > >> > > > > > > backends
> > > > > >> > > > > > >> > > > > > > > > > can't
> > > > > >> > > > > > >> > > > > > > > > > > > >> >> participate.
> > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>
> > > > > >> > > > > > >> > > > > > > > > > > > >> >> >> Stepping back, I think
> > > the
> > > > > >> > metrics
> > > > > >> > > > > > >> subsystem
> > > > > >> > > > > > >> > is
> > > > > >> > > > > > >> > > > > > growing
> > > > > >> > > > > > >> > > > > > > > into
> > > > > >> > > > > > >> > > > > > > > > > > > >> something
> > > > > >> > > > > > >> > > > > > > > > > > > >> >> >> real (write + read +
> > > REST
> > > > API
> > > > > >> +
> > > > > >> > > AuthZ +
> > > > > >> > > > > > >> > > > pagination)
> > > > > >> > > > > > >> > > > > > *but
> > > > > >> > > > > > >> > > > > > > > the
> > > > > >> > > > > > >> > > > > > > > > > > > >> >> persistence
> > > > > >> > > > > > >> > > > > > > > > > > > >> >> >> side is split across
> > two
> > > > > >> layers
> > > > > >> > in
> > > > > >> > > a
> > > > > >> > > > > way
> > > > > >> > > > > > >> > that's
> > > > > >> > > > > > >> > > > hard
> > > > > >> > > > > > >> > > > > > to
> > > > > >> > > > > > >> > > > > > > > > > > extend*. I
> > > > > >> > > > > > >> > > > > > > > > > > > >> put
> > > > > >> > > > > > >> > > > > > > > > > > > >> >> >> together two diagrams
> > to
> > > > show
> > > > > >> > what
> > > > > >> > > I
> > > > > >> > > > > mean
> > > > > >> > > > > > >> (my
> > > > > >> > > > > > >> > > best
> > > > > >> > > > > > >> > > > > > > > effort).
> > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>
> > > > > >> > > > > > >> > > > > > > > > > > > >> >> >> *Current state*
> > (Diagram
> > > > 1):
> > > > > >> > three
> > > > > >> > > > > > >> interfaces
> > > > > >> > > > > > >> > at
> > > > > >> > > > > > >> > > > > three
> > > > > >> > > > > > >> > > > > > > > > > different
> > > > > >> > > > > > >> > > > > > > > > > > > >> >> levels.
> > > > > >> > > > > > >> > > > > > > > > > > > >> >> >> The engine-facing SPI
> > > > > >> > > > > > >> (PolarisMetricsReporter)
> > > > > >> > > > > > >> > > is
> > > > > >> > > > > > >> > > > > > clean.
> > > > > >> > > > > > >> > > > > > > > But
> > > > > >> > > > > > >> > > > > > > > > > > > >> >> >> PolarisMetricsManager
> > on
> > > > > >> > > > > MetaStoreManager
> > > > > >> > > > > > >> is a
> > > > > >> > > > > > >> > > > > > > passthrough
> > > > > >> > > > > > >> > > > > > > > > to
> > > > > >> > > > > > >> > > > > > > > > > > > >> >> >> MetricsPersistence on
> > > > > >> > > BasePersistence.
> > > > > >> > > > > > The
> > > > > >> > > > > > >> > @Beta
> > > > > >> > > > > > >> > > > > > > > annotation
> > > > > >> > > > > > >> > > > > > > > > > and
> > > > > >> > > > > > >> > > > > > > > > > > > SPI
> > > > > >> > > > > > >> > > > > > > > > > > > >> >> javadoc
> > > > > >> > > > > > >> > > > > > > > > > > > >> >> >> are on the
> > > BasePersistence
> > > > > >> layer,
> > > > > >> > > while
> > > > > >> > > > > > the
> > > > > >> > > > > > >> > > actual
> > > > > >> > > > > > >> > > > > > > > extension
> > > > > >> > > > > > >> > > > > > > > > > > > points
> > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>
> > (PolarisMetricsReporter,
> > > > > >> > > > > > >> > PolarisMetricsManager)
> > > > > >> > > > > > >> > > > > carry
> > > > > >> > > > > > >> > > > > > no
> > > > > >> > > > > > >> > > > > > > > > > > stability
> > > > > >> > > > > > >> > > > > > > > > > > > >> >> >> annotation. The write
> > > path
> > > > > >> goes
> > > > > >> > > through
> > > > > >> > > > > > the
> > > > > >> > > > > > >> > > > > > > > MetaStoreManager
> > > > > >> > > > > > >> > > > > > > > > > > > layer,
> > > > > >> > > > > > >> > > > > > > > > > > > >> the
> > > > > >> > > > > > >> > > > > > > > > > > > >> >> >> read path doesn't.
> > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>
> > > > > >> > > > > > >> > > > > > > > > > > > >> >> >> *What I envision*
> > > > (Diagram 2):
> > > > > >> > two
> > > > > >> > > SPIs
> > > > > >> > > > > > at
> > > > > >> > > > > > >> two
> > > > > >> > > > > > >> > > > > levels.
> > > > > >> > > > > > >> > > > > > > > > > > > >> >> >> PolarisMetricsReporter
> > > > stays
> > > > > >> as
> > > > > >> > the
> > > > > >> > > > > > >> > > engine-facing
> > > > > >> > > > > > >> > > > > SPI.
> > > > > >> > > > > > >> > > > > > > > > > > > >> >> >> PolarisMetricsManager
> > > > becomes
> > > > > >> the
> > > > > >> > > > > > >> > backend-facing
> > > > > >> > > > > > >> > > > SPI
> > > > > >> > > > > > >> > > > > > > with
> > > > > >> > > > > > >> > > > > > > > > both
> > > > > >> > > > > > >> > > > > > > > > > > > write
> > > > > >> > > > > > >> > > > > > > > > > > > >> >> and
> > > > > >> > > > > > >> > > > > > > > > > > > >> >> >> read methods at the
> > > > > >> > > MetaStoreManager
> > > > > >> > > > > > level,
> > > > > >> > > > > > >> > > where
> > > > > >> > > > > > >> > > > > any
> > > > > >> > > > > > >> > > > > > > > > backend
> > > > > >> > > > > > >> > > > > > > > > > > > (JDBC,
> > > > > >> > > > > > >> > > > > > > > > > > > >> >> NoSQL,
> > > > > >> > > > > > >> > > > > > > > > > > > >> >> >> remote) can implement
> > > > them.
> > > > > >> > > > > > >> MetricsPersistence
> > > > > >> > > > > > >> > > on
> > > > > >> > > > > > >> > > > > > > > > > > BasePersistence
> > > > > >> > > > > > >> > > > > > > > > > > > >> goes
> > > > > >> > > > > > >> > > > > > > > > > > > >> >> >> away. Where metrics
> > > > actually
> > > > > >> land
> > > > > >> > > is an
> > > > > >> > > > > > >> > > > > implementation
> > > > > >> > > > > > >> > > > > > > > > detail,
> > > > > >> > > > > > >> > > > > > > > > > > > not a
> > > > > >> > > > > > >> > > > > > > > > > > > >> >> core
> > > > > >> > > > > > >> > > > > > > > > > > > >> >> >> interface.
> > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>
> > > > > >> > > > > > >> > > > > > > > > > > > >> >> >> *Minor naming thing*:
> > > > > >> > > > > > >> PolarisMetricsReporter
> > > > > >> > > > > > >> > is
> > > > > >> > > > > > >> > > > > > broader
> > > > > >> > > > > > >> > > > > > > > than
> > > > > >> > > > > > >> > > > > > > > > > > what
> > > > > >> > > > > > >> > > > > > > > > > > > it
> > > > > >> > > > > > >> > > > > > > > > > > > >> >> >> actually handles. It
> > > only
> > > > > >> accepts
> > > > > >> > > > > Iceberg
> > > > > >> > > > > > >> REST
> > > > > >> > > > > > >> > > > > Catalog
> > > > > >> > > > > > >> > > > > > > > > metrics
> > > > > >> > > > > > >> > > > > > > > > > > > >> >> (ScanReport,
> > > > > >> > > > > > >> > > > > > > > > > > > >> >> >> CommitReport via
> > > > > >> MetricsReport).
> > > > > >> > > > > Generic
> > > > > >> > > > > > >> table
> > > > > >> > > > > > >> > > > > metrics
> > > > > >> > > > > > >> > > > > > > or
> > > > > >> > > > > > >> > > > > > > > > > > > >> operational
> > > > > >> > > > > > >> > > > > > > > > > > > >> >> >> metrics aren't in
> > scope.
> > > > Not
> > > > > >> > > blocking,
> > > > > >> > > > > > but
> > > > > >> > > > > > >> > worth
> > > > > >> > > > > > >> > > > > > noting
> > > > > >> > > > > > >> > > > > > > if
> > > > > >> > > > > > >> > > > > > > > > the
> > > > > >> > > > > > >> > > > > > > > > > > > >> metrics
> > > > > >> > > > > > >> > > > > > > > > > > > >> >> >> surface expands.
> > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>
> > > > > >> > > > > > >> > > > > > > > > > > > >> >> >> *Rough sketch of how
> > to
> > > > get
> > > > > >> > there*:
> > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>  1.  Add read methods
> > to
> > > > > >> > > > > > >> PolarisMetricsManager
> > > > > >> > > > > > >> > > > > > > > > > (listScanReports,
> > > > > >> > > > > > >> > > > > > > > > > > > >> >> >> listCommitReports)
> > with
> > > > > >> default
> > > > > >> > > no-op,
> > > > > >> > > > > > >> same as
> > > > > >> > > > > > >> > > the
> > > > > >> > > > > > >> > > > > > > > existing
> > > > > >> > > > > > >> > > > > > > > > > > write
> > > > > >> > > > > > >> > > > > > > > > > > > >> >> methods.
> > > > > >> > > > > > >> > > > > > > > > > > > >> >> >> (Probably make
> > > > > >> > > PolarisMetricsManager
> > > > > >> > > > > more
> > > > > >> > > > > > >> > > explicit
> > > > > >> > > > > > >> > > > > on
> > > > > >> > > > > > >> > > > > > > > being
> > > > > >> > > > > > >> > > > > > > > > > > > Iceberg
> > > > > >> > > > > > >> > > > > > > > > > > > >> >> >> specific like package
> > > > name or
> > > > > >> > class
> > > > > >> > > > > name
> > > > > >> > > > > > >> etc.)
> > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>  2.  Wire
> > > > > >> MetricsReportsService
> > > > > >> > > through
> > > > > >> > > > > > >> > > > > > MetaStoreManager
> > > > > >> > > > > > >> > > > > > > > > > instead
> > > > > >> > > > > > >> > > > > > > > > > > > of
> > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>
> > > > callContext.getMetaStore().
> > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>  3.  Extract metrics
> > > > > >> persistence
> > > > > >> > > from
> > > > > >> > > > > > >> > > > > > > > > JdbcBasePersistenceImpl
> > > > > >> > > > > > >> > > > > > > > > > > into
> > > > > >> > > > > > >> > > > > > > > > > > > >> its
> > > > > >> > > > > > >> > > > > > > > > > > > >> >> >> own class. That file
> > > > carries
> > > > > >> ~7
> > > > > >> > > > > > >> > > responsibilities,
> > > > > >> > > > > > >> > > > > > > metrics
> > > > > >> > > > > > >> > > > > > > > > > being
> > > > > >> > > > > > >> > > > > > > > > > > > one
> > > > > >> > > > > > >> > > > > > > > > > > > >> of
> > > > > >> > > > > > >> > > > > > > > > > > > >> >> them.
> > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>  4.  Remove
> > > > MetricsPersistence
> > > > > >> > from
> > > > > >> > > > > > >> > > > BasePersistence.
> > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>
> > > > > >> > > > > > >> > > > > > > > > > > > >> >> >> *None of this needs to
> > > > happen
> > > > > >> in
> > > > > >> > > #4115.
> > > > > >> > > > > > >> But if
> > > > > >> > > > > > >> > > the
> > > > > >> > > > > > >> > > > > > > > direction
> > > > > >> > > > > > >> > > > > > > > > > > makes
> > > > > >> > > > > > >> > > > > > > > > > > > >> >> sense,
> > > > > >> > > > > > >> > > > > > > > > > > > >> >> >> it would be good to
> > > align
> > > > > >> before
> > > > > >> > > the
> > > > > >> > > > > > >> metrics
> > > > > >> > > > > > >> > > > surface
> > > > > >> > > > > > >> > > > > > > grows
> > > > > >> > > > > > >> > > > > > > > > > > > further.
> > > > > >> > > > > > >> > > > > > > > > > > > >> >> Curious
> > > > > >> > > > > > >> > > > > > > > > > > > >> >> >> what others think.*
> > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>
> > > > > >> > > > > > >> > > > > > > > > > > > >> >> >> *My mental model
> > note*:
> > > > Level
> > > > > >> 1
> > > > > >> > > > > > >> > > MetaStoreManager;
> > > > > >> > > > > > >> > > > > > level
> > > > > >> > > > > > >> > > > > > > 2
> > > > > >> > > > > > >> > > > > > > > > > > > >> transactional
> > > > > >> > > > > > >> > > > > > > > > > > > >> >> >> persistence; level 3
> > > base
> > > > > >> > > persistence
> > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>
> > > > > >> > > > > > >> > > > > > > > > > > > >> >> >> Diagram 1
> > > > > >> > > > > > >> > > > > > > > > > > > >> >> >> <
> > > > > >> > > > > > >> > > > > > > > > > > > >> >>
> > > > > >> > > > > > >> > > > > > > > > > > > >>
> > > > > >> > > > > > >> > > > > > > > > > > >
> > > > > >> > > > > > >> > > > > > > > > > >
> > > > > >> > > > > > >> > > > > > > > > >
> > > > > >> > > > > > >> > > > > > > > >
> > > > > >> > > > > > >> > > > > > > >
> > > > > >> > > > > > >> > > > > > >
> > > > > >> > > > > > >> > > > > >
> > > > > >> > > > > > >> > > > >
> > > > > >> > > > > > >> > > >
> > > > > >> > > > > > >> > >
> > > > > >> > > > > > >> >
> > > > > >> > > > > > >>
> > > > > >> > > > > >
> > > > > >> > > > >
> > > > > >> > >
> > > > > >> >
> > > > > >>
> > > >
> > >
> > https://urldefense.com/v3/__https://www.plantuml.com/plantuml/uml/bLHDR-Cs4BthLmpIYupw0zbkKQ1r3M-S7Bp8xhhM7WCOb3IM65EaGD9EX2RzxHrHb4CxRelwa4YSDu_lpOVcnZ9jzvM8BBS2uGjQpJC3dtHMSekPtMk44IpsMgEqa5XcCOhCZikQQLP1pR8TAp2n3ILhmZDP20m0fcIvUkAoW2qJXd9z1bpToO9BX3WXu0ucy5rpgGPNm0nW5_epUWtm2Ue3pn3kMOFQmKntGZW0BYtgBSi8k5A2QMwybJNMIbFiGSR9QZc4nUqIvikStF0jHprua5C-amge42aNt3R0f5JaaoivdV2Pkqbx4hee4ymOkBh5BTiB-_uIeGeo8zL8rPsPl4DktdEiK1jkB1NdZCRbrSTecDe_mlHbF0wvBmCkaOH5_S8a_TTTKI6-nmCAkEw4LpxsZ-LbYLKQFKMNOgf_wuM7_bV9gOer5SYMMksBSWXFcbi49KNZXNLicwfe3TETC7gPdPqI7uBcHMb1RSzYq34c6PDUM9mn8HRsUTZEiDBve3NjVZumBj0U7SS37mGO7vcwtiK-_pU7U7L_f-digo9YbhSwIfMRwIITKGXbxdIUTCGF1SeCJxloKsU-3k9ddRbX1eDq1q_fx1JbBGT0glVyXimDuP4TQ5qpCAmnGEj2s_6n5mtn1z-97-63itFQZLPO1Ev2tu_WF7Ju-VPc0Skg5bYXxBhkY1xpD7EM_7fyflSpIsqMgVth5xhVr4eQxWQ8enaSAJQSG16yFSDuJ798rrcXr_3n-lfdk7icQjEBmFujL7AodiP_Y4Z7-YxvtZNs4zMgpNTl6tF8sglyPsmqchrjvQ-m-aP94r-TwCA2Ka8upPJZwtvSpoYCXkYMZU2NXvRMBfq9P3i3Le4VAZUAlUZ_oPKsxPgY0Q_BSKLkyr9bhQhQrJjo_x3TPlIB0DPjnMfcIoYP0QaYw1a0fTKDr8fB6ntNuvmoL1ZGkXa69Njh43zf9GiGxHQrA_jDYWRSzF5--WmTVrN97_Sm8LbLUy_lGBmLanJjFkDlGkRqjA_4tm00__;!!Iz9xO38YGHZK!4rnlVIqBQQQ6xpGe1PMV58edOS_cGrKoqdfhw4qSp_JQCavvEjaxXk0OwGPnxyEcO60AzKQK_XDtp7H3DdkFXcrXuTBISHg$
> > > > > >> > > > > > >> > > > > > > > > > > > >> >> >
> > > > > >> > > > > > >> > > > > > > > > > > > >> >> >> :
> > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>
> > > > > >> > > > > > >> > > > > > > > > > > > >> >> >> [image: image.png]
> > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>
> > > > > >> > > > > > >> > > > > > > > > > > > >> >> >> Diagram 2
> > > > > >> > > > > > >> > > > > > > > > > > > >> >> >> <
> > > > > >> > > > > > >> > > > > > > > > > > > >> >>
> > > > > >> > > > > > >> > > > > > > > > > > > >>
> > > > > >> > > > > > >> > > > > > > > > > > >
> > > > > >> > > > > > >> > > > > > > > > > >
> > > > > >> > > > > > >> > > > > > > > > >
> > > > > >> > > > > > >> > > > > > > > >
> > > > > >> > > > > > >> > > > > > > >
> > > > > >> > > > > > >> > > > > > >
> > > > > >> > > > > > >> > > > > >
> > > > > >> > > > > > >> > > > >
> > > > > >> > > > > > >> > > >
> > > > > >> > > > > > >> > >
> > > > > >> > > > > > >> >
> > > > > >> > > > > > >>
> > > > > >> > > > > >
> > > > > >> > > > >
> > > > > >> > >
> > > > > >> >
> > > > > >>
> > > >
> > >
> > https://urldefense.com/v3/__https://www.plantuml.com/plantuml/uml/VLLDR-8m4BtdLupO2sWBLVU8AaGB7AXAbssGzb896SS9RXqxjKqBwkv_tt7iV43fdaZYDpFlpRmnOsE9jhjSH9PRmM31hERKm8scMsuPjJlDe0yheZDc8RR4iYWoBrmMH9CS2a9VICPYUy1OZN0YCy5Q0BCbYNhdCeEK28En8G8wCvbnoQ0R8_05Bc6bkLIz3X03p1zzH7zR-9ZfDquPt9C3qoNCX2yV4G2NbkcKu5jdgGJHt0GbZwnG6i-UP3TUpk5gM6Ldqke350eZUqzoCft3U9xWHvxoa5-7K4nF1J46EbEMafsmdrCBbQ44gVggy18IZrn_ph5asd1ZiIKdQSgueZvjXrQFSFrdC3YN-nXmBacxbGiYyLVxLaBtdhqn0LSzdBDhqQtQoOJeGyad3z0lUqnYgpGB6Ns8oVyta00Dy_WnX0tIOZ8v6SYxHll1TrH6aejAik-mh-AphVFCwSUQqFypElag5QRGFDjQKEd96K1P8QP41c9TzA_IIQyvdAWyv_RSiS3skb0_EzDDkK2v5xWF6MiGFlvhpFLcD2Dq2pml14gaF67eQkmd8gulDoC4kSOu6KVpkvlUJg1RTbWISU40RdBUUS_9XfRZ2dwxm_SW8LYFISgm_MnlDQ6M9P1gbKEc4X-2pH_FvJCkCqm9pbVjD6LrwdLeOrDWfOaqc8Wh9BE85oNKxkNQ6o4yGRy_Eae0G_G8tZv81d3bHDB23WOdisohVr3nh_j6lbSjbNaLRTc8UgtPbAU1J_tygOfZX9DWEJeHDvYx-qmSi5FgNLPZwHrHcUsncGQ5-skhUclpE5fo4ounpFauYrUbkU6ccfnxMvitwag4IyerhTxj8In_Oj1bDO4pQru674loYrGlULHLEGCjwJJ8gDoVZR8MxO4BT3IzRvIcAQKezC6xpziGnTyImrfEGyJI_OcKfgtxIvnTqFEMS17L9Z-jsARN5FmTheP7HtSdtOMT0B4GY2FYHXxgQmMtj2bRqiLFGapiVe1_QVKDrkqXcm83aFEXnMYCZ-xlyHy__;!!Iz9xO38YGHZK!4rnlVIqBQQQ6xpGe1PMV58edOS_cGrKoqdfhw4qSp_JQCavvEjaxXk0OwGPnxyEcO60AzKQK_XDtp7H3DdkFXcrXSpmqcNk$
> > > > > >> > > > > > >> > > > > > > > > > > > >> >> >
> > > > > >> > > > > > >> > > > > > > > > > > > >> >> >> :
> > > > > >> > > > > > >> > > > > > > > > > > > >> >> >> [image: image.png]
> > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>
> > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>  -ej
> > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>
> > > > > >> > > > > > >> > > > > > > > > > > > >> >> >> On Wed, Apr 15, 2026
> > at
> > > > > >> 8:22 AM
> > > > > >> > > Dmitri
> > > > > >> > > > > > >> > > > Bourlatchkov
> > > > > >> > > > > > >> > > > > <
> > > > > >> > > > > > >> > > > > > > > > > > > >> dimas@apache.org>
> > > > > >> > > > > > >> > > > > > > > > > > > >> >> >> wrote:
> > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>
> > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> Hi All,
> > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>>
> > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> Heads up: The current
> > > > state
> > > > > >> of
> > > > > >> > PR
> > > > > >> > > > > [4115]
> > > > > >> > > > > > >> > looks
> > > > > >> > > > > > >> > > > > pretty
> > > > > >> > > > > > >> > > > > > > > solid
> > > > > >> > > > > > >> > > > > > > > > > to
> > > > > >> > > > > > >> > > > > > > > > > > > me.
> > > > > >> > > > > > >> > > > > > > > > > > > >> I
> > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> believe this PR is
> > > > > >> approaching a
> > > > > >> > > > > > mergeable
> > > > > >> > > > > > >> > > > > condition.
> > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>>
> > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> Please post your
> > > reviews
> > > > if
> > > > > >> you
> > > > > >> > > have
> > > > > >> > > > > any
> > > > > >> > > > > > >> > > > comments.
> > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>>
> > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> [4115]
> > > > > >> > > > > > >> > > >
> > > > > >> >
> > > > > >>
> > > >
> > >
> > https://urldefense.com/v3/__https://github.com/apache/polaris/pull/4115__;!!Iz9xO38YGHZK!4rnlVIqBQQQ6xpGe1PMV58edOS_cGrKoqdfhw4qSp_JQCavvEjaxXk0OwGPnxyEcO60AzKQK_XDtp7H3DdkFXcrXtFRscKs$
> > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>>
> > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> Thanks,
> > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> Dmitri.
> > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>>
> > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> On Tue, Mar 3, 2026
> > at
> > > > > >> 3:29 PM
> > > > > >> > > Anand
> > > > > >> > > > > > Kumar
> > > > > >> > > > > > >> > > > Sankaran
> > > > > >> > > > > > >> > > > > > via
> > > > > >> > > > > > >> > > > > > > > > dev <
> > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>>
> > dev@polaris.apache.org
> > > >
> > > > > >> wrote:
> > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>>
> > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > Hi Yufei and
> > Dmitri,
> > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> >
> > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > Here is a proposal
> > > for
> > > > the
> > > > > >> > REST
> > > > > >> > > > > > >> endpoints
> > > > > >> > > > > > >> > for
> > > > > >> > > > > > >> > > > > > metrics
> > > > > >> > > > > > >> > > > > > > > and
> > > > > >> > > > > > >> > > > > > > > > > > > events.
> > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> >
> > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> >
> > > > > >> > > > > > >> > > > > >
> > > > > >> >
> > > > > >>
> > > >
> > >
> > https://urldefense.com/v3/__https://github.com/apache/polaris/pull/3924/changes__;!!Iz9xO38YGHZK!4rnlVIqBQQQ6xpGe1PMV58edOS_cGrKoqdfhw4qSp_JQCavvEjaxXk0OwGPnxyEcO60AzKQK_XDtp7H3DdkFXcrXcjoPhSs$
> > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> >
> > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > I did not see any
> > > > > >> precursors
> > > > > >> > for
> > > > > >> > > > > > >> raising a
> > > > > >> > > > > > >> > PR
> > > > > >> > > > > > >> > > > for
> > > > > >> > > > > > >> > > > > > > > > > proposals,
> > > > > >> > > > > > >> > > > > > > > > > > so
> > > > > >> > > > > > >> > > > > > > > > > > > >> >> trying
> > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > this.  Please let
> > me
> > > > know
> > > > > >> what
> > > > > >> > > you
> > > > > >> > > > > > >> think.
> > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> >
> > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > -
> > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > Anand
> > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> >
> > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > From: Anand Kumar
> > > > Sankaran
> > > > > >> <
> > > > > >> > > > > > >> > > > > > > anand.sankaran@workday.com
> > > > > >> > > > > > >> > > > > > > > >
> > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > Date: Monday, March
> > > 2,
> > > > > >> 2026 at
> > > > > >> > > > > > 10:25 AM
> > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > To:
> > > > dev@polaris.apache.org
> > > > > >> <
> > > > > >> > > > > > >> > > > > dev@polaris.apache.org
> > > > > >> > > > > > >> > > > > > >
> > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > Subject: Re:
> > Polaris
> > > > > >> Telemetry
> > > > > >> > > and
> > > > > >> > > > > > Audit
> > > > > >> > > > > > >> > > Trail
> > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> >
> > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > About the REST API,
> > > > based
> > > > > >> on
> > > > > >> > my
> > > > > >> > > use
> > > > > >> > > > > > >> cases:
> > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> >
> > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> >
> > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> >   1.
> > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > I want to be able
> > to
> > > > query
> > > > > >> > > commit
> > > > > >> > > > > > >> metrics
> > > > > >> > > > > > >> > to
> > > > > >> > > > > > >> > > > > track
> > > > > >> > > > > > >> > > > > > > > files
> > > > > >> > > > > > >> > > > > > > > > > > added
> > > > > >> > > > > > >> > > > > > > > > > > > /
> > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> removed
> > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > per commit, along
> > > with
> > > > > >> record
> > > > > >> > > > > counts.
> > > > > >> > > > > > >> The
> > > > > >> > > > > > >> > > > > ingestion
> > > > > >> > > > > > >> > > > > > > > > > pipeline
> > > > > >> > > > > > >> > > > > > > > > > > > that
> > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> writes
> > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > this data is owned
> > by
> > > > us
> > > > > >> and
> > > > > >> > we
> > > > > >> > > are
> > > > > >> > > > > > >> > > guaranteed
> > > > > >> > > > > > >> > > > to
> > > > > >> > > > > > >> > > > > > > write
> > > > > >> > > > > > >> > > > > > > > > > this
> > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> information
> > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > for each write.
> > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> >   2.
> > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > I want to be able
> > to
> > > > query
> > > > > >> > scan
> > > > > >> > > > > > metrics
> > > > > >> > > > > > >> for
> > > > > >> > > > > > >> > > > > read. I
> > > > > >> > > > > > >> > > > > > > > > > > understand
> > > > > >> > > > > > >> > > > > > > > > > > > >> >> clients
> > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> do
> > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > not fulfill this
> > > > > >> requirement.
> > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> >   3.
> > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > I want to be able
> > to
> > > > query
> > > > > >> the
> > > > > >> > > > > events
> > > > > >> > > > > > >> table
> > > > > >> > > > > > >> > > > > (events
> > > > > >> > > > > > >> > > > > > > are
> > > > > >> > > > > > >> > > > > > > > > > > > >> persisted) -
> > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> this
> > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > may supersede #2, I
> > > am
> > > > not
> > > > > >> > sure
> > > > > >> > > yet.
> > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> >
> > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > All this
> > information
> > > > is in
> > > > > >> the
> > > > > >> > > JDBC
> > > > > >> > > > > > >> based
> > > > > >> > > > > > >> > > > > > persistence
> > > > > >> > > > > > >> > > > > > > > > model
> > > > > >> > > > > > >> > > > > > > > > > > and
> > > > > >> > > > > > >> > > > > > > > > > > > >> is
> > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > persisted in the
> > > > > >> metastore. I
> > > > > >> > > > > > currently
> > > > > >> > > > > > >> > don’t
> > > > > >> > > > > > >> > > > > have
> > > > > >> > > > > > >> > > > > > a
> > > > > >> > > > > > >> > > > > > > > need
> > > > > >> > > > > > >> > > > > > > > > > to
> > > > > >> > > > > > >> > > > > > > > > > > > >> query
> > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > prometheus or open
> > > > > >> telemetry.
> > > > > >> > I
> > > > > >> > > do
> > > > > >> > > > > > >> publish
> > > > > >> > > > > > >> > > some
> > > > > >> > > > > > >> > > > > > > events
> > > > > >> > > > > > >> > > > > > > > to
> > > > > >> > > > > > >> > > > > > > > > > > > >> Prometheus
> > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> and
> > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > they are forwarded
> > to
> > > > our
> > > > > >> > > dashboards
> > > > > >> > > > > > >> > > elsewhere.
> > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> >
> > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > About the CLI
> > > > utilities, I
> > > > > >> > > meant the
> > > > > >> > > > > > >> admin
> > > > > >> > > > > > >> > > user
> > > > > >> > > > > > >> > > > > > > > > utilities.
> > > > > >> > > > > > >> > > > > > > > > > In
> > > > > >> > > > > > >> > > > > > > > > > > > >> one of
> > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> the
> > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > earliest drafts of
> > my
> > > > > >> > proposal,
> > > > > >> > > > > > Prashant
> > > > > >> > > > > > >> > > > > mentioned
> > > > > >> > > > > > >> > > > > > > that
> > > > > >> > > > > > >> > > > > > > > > the
> > > > > >> > > > > > >> > > > > > > > > > > > >> metrics
> > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> tables
> > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > can grow
> > indefinitely
> > > > and
> > > > > >> > that a
> > > > > >> > > > > > similar
> > > > > >> > > > > > >> > > > problem
> > > > > >> > > > > > >> > > > > > > exists
> > > > > >> > > > > > >> > > > > > > > > > with
> > > > > >> > > > > > >> > > > > > > > > > > > the
> > > > > >> > > > > > >> > > > > > > > > > > > >> >> events
> > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > table as well. We
> > > > discussed
> > > > > >> > that
> > > > > >> > > > > > >> cleaning
> > > > > >> > > > > > >> > up
> > > > > >> > > > > > >> > > of
> > > > > >> > > > > > >> > > > > old
> > > > > >> > > > > > >> > > > > > > > > records
> > > > > >> > > > > > >> > > > > > > > > > > > from
> > > > > >> > > > > > >> > > > > > > > > > > > >> >> both
> > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > metrics tables and
> > > > events
> > > > > >> > > tables can
> > > > > >> > > > > > be
> > > > > >> > > > > > >> > done
> > > > > >> > > > > > >> > > > via
> > > > > >> > > > > > >> > > > > a
> > > > > >> > > > > > >> > > > > > > CLI
> > > > > >> > > > > > >> > > > > > > > > > > utility.
> > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> >
> > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > I see that Yufei
> > has
> > > > > >> covered
> > > > > >> > the
> > > > > >> > > > > > >> discussion
> > > > > >> > > > > > >> > > > about
> > > > > >> > > > > > >> > > > > > > > > > > datasources.
> > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> >
> > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > -
> > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > Anand
> > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> >
> > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> >
> > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> >
> > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > From: Yufei Gu <
> > > > > >> > > > > flyrain000@gmail.com>
> > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > Date: Friday,
> > > February
> > > > 27,
> > > > > >> > 2026
> > > > > >> > > at
> > > > > >> > > > > > >> 9:54 PM
> > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > To:
> > > > dev@polaris.apache.org
> > > > > >> <
> > > > > >> > > > > > >> > > > > dev@polaris.apache.org
> > > > > >> > > > > > >> > > > > > >
> > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > Subject: Re:
> > Polaris
> > > > > >> Telemetry
> > > > > >> > > and
> > > > > >> > > > > > Audit
> > > > > >> > > > > > >> > > Trail
> > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> >
> > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > This Message Is
> > From
> > > an
> > > > > >> > External
> > > > > >> > > > > > Sender
> > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > This message came
> > > from
> > > > > >> outside
> > > > > >> > > your
> > > > > >> > > > > > >> > > > organization.
> > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > Report Suspicious<
> > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> >
> > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>>
> > > > > >> > > > > > >> > > > > > > > > > > > >> >>
> > > > > >> > > > > > >> > > > > > > > > > > > >>
> > > > > >> > > > > > >> > > > > > > > > > > >
> > > > > >> > > > > > >> > > > > > > > > > >
> > > > > >> > > > > > >> > > > > > > > > >
> > > > > >> > > > > > >> > > > > > > > >
> > > > > >> > > > > > >> > > > > > > >
> > > > > >> > > > > > >> > > > > > >
> > > > > >> > > > > > >> > > > > >
> > > > > >> > > > > > >> > > > >
> > > > > >> > > > > > >> > > >
> > > > > >> > > > > > >> > >
> > > > > >> > > > > > >> >
> > > > > >> > > > > > >>
> > > > > >> > > > > >
> > > > > >> > > > >
> > > > > >> > >
> > > > > >> >
> > > > > >>
> > > >
> > >
> > https://us-phishalarm-ewt.proofpoint.com/EWT/v1/Iz9xO38YGHZK!YhNDZABkHi1B699ote2uMwpOZw8i0QMCGO2Szc-HshuABGhGvwPJcymE6G2oUUxtS8xDkSrtGTPm_I3QnVDHoLMk50m9v8z_nZKTkd-bnVUbreF1u0WnfV_X5eYevZl_$
> > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > >
> > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> >
> > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> >
> > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > As I mentioned in
> > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> >
> > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>>
> > > > > >> > > > > > >> > > > > > > > > > > > >> >>
> > > > > >> > > > > > >> > > > > > > > > > > > >>
> > > > > >> > > > > > >> > > > > > > > > > > >
> > > > > >> > > > > > >> > > > > > > > > > >
> > > > > >> > > > > > >> > > > > > > > > >
> > > > > >> > > > > > >> > > > > > > > >
> > > > > >> > > > > > >> > > > > > > >
> > > > > >> > > > > > >> > > > > > >
> > > > > >> > > > > > >> > > > > >
> > > > > >> > > > > > >> > > > >
> > > > > >> > > > > > >> > > >
> > > > > >> > > > > > >> > >
> > > > > >> > > > > > >> >
> > > > > >> > > > > > >>
> > > > > >> > > > > >
> > > > > >> > > > >
> > > > > >> > >
> > > > > >> >
> > > > > >>
> > > >
> > >
> > https://urldefense.com/v3/__https://github.com/apache/polaris/issues/3890__;!!Iz9xO38YGHZK!5EuyFFkk3vhRWVIRvQAWBSQfpJkTMA9HxugzDwXmN0LPPqhEFxYkFRGVhtb8AqUwXtDh2OplcMnbMDHKOxrvDU0$
> > > > > >> > >> > > >> > > > > > > > > > > > >> >> >>> ,
> > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > supporting
> > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > multiple data
> > sources
> > > > is
> > > > > >> not a
> > > > > >> > > > > trivial
> > > > > >> > > > > > >> > > change.
> > > > > >> > > > > > >> > > > I
> > > > > >> > > > > > >> > > > > > > would
> > > > > >> > > > > > >> > > > > > > > > > > strongly
> > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> recommend
> > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > starting with a
> > > design
> > > > > >> > document
> > > > > >> > > to
> > > > > >> > > > > > >> > carefully
> > > > > >> > > > > > >> > > > > > evaluate
> > > > > >> > > > > > >> > > > > > > > the
> > > > > >> > > > > > >> > > > > > > > > > > > >> >> architectural
> > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > implications and
> > long
> > > > term
> > > > > >> > > impact.
> > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> >
> > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > A REST endpoint to
> > > > query
> > > > > >> > metrics
> > > > > >> > > > > seems
> > > > > >> > > > > > >> > > > reasonable
> > > > > >> > > > > > >> > > > > > > given
> > > > > >> > > > > > >> > > > > > > > > the
> > > > > >> > > > > > >> > > > > > > > > > > > >> current
> > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> JDBC
> > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > based persistence
> > > > model.
> > > > > >> That
> > > > > >> > > said,
> > > > > >> > > > > we
> > > > > >> > > > > > >> may
> > > > > >> > > > > > >> > > also
> > > > > >> > > > > > >> > > > > > > > consider
> > > > > >> > > > > > >> > > > > > > > > > > > >> alternative
> > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > storage models. For
> > > > > >> example,
> > > > > >> > if
> > > > > >> > > we
> > > > > >> > > > > > later
> > > > > >> > > > > > >> > > adopt
> > > > > >> > > > > > >> > > > a
> > > > > >> > > > > > >> > > > > > time
> > > > > >> > > > > > >> > > > > > > > > > series
> > > > > >> > > > > > >> > > > > > > > > > > > >> system
> > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> such as
> > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > Prometheus to store
> > > > > >> metrics,
> > > > > >> > the
> > > > > >> > > > > query
> > > > > >> > > > > > >> > model
> > > > > >> > > > > > >> > > > and
> > > > > >> > > > > > >> > > > > > > access
> > > > > >> > > > > > >> > > > > > > > > > > > patterns
> > > > > >> > > > > > >> > > > > > > > > > > > >> >> would
> > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> be
> > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > fundamentally
> > > > different.
> > > > > >> > > Designing
> > > > > >> > > > > the
> > > > > >> > > > > > >> REST
> > > > > >> > > > > > >> > > API
> > > > > >> > > > > > >> > > > > > > without
> > > > > >> > > > > > >> > > > > > > > > > > > >> considering
> > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> these
> > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > potential
> > evolutions
> > > > may
> > > > > >> limit
> > > > > >> > > > > > >> flexibility.
> > > > > >> > > > > > >> > > I'd
> > > > > >> > > > > > >> > > > > > > suggest
> > > > > >> > > > > > >> > > > > > > > > to
> > > > > >> > > > > > >> > > > > > > > > > > > start
> > > > > >> > > > > > >> > > > > > > > > > > > >> >> with
> > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> the
> > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > use case.
> > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> >
> > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > Yufei
> > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> >
> > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> >
> > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > On Fri, Feb 27,
> > 2026
> > > at
> > > > > >> > 3:42 PM
> > > > > >> > > > > Dmitri
> > > > > >> > > > > > >> > > > > > Bourlatchkov <
> > > > > >> > > > > > >> > > > > > > > > > > > >> >> dimas@apache.org>
> > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > wrote:
> > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> >
> > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > > Hi Anand,
> > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > >
> > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > > Sharing my
> > view...
> > > > > >> subject
> > > > > >> > to
> > > > > >> > > > > > >> discussion:
> > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > >
> > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > > 1. Adding non-IRC
> > > > REST
> > > > > >> API
> > > > > >> > to
> > > > > >> > > > > > Polaris
> > > > > >> > > > > > >> is
> > > > > >> > > > > > >> > > > > > perfectly
> > > > > >> > > > > > >> > > > > > > > > fine.
> > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > >
> > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > > Figuring out
> > > specific
> > > > > >> > endpoint
> > > > > >> > > > > URIs
> > > > > >> > > > > > >> and
> > > > > >> > > > > > >> > > > > payloads
> > > > > >> > > > > > >> > > > > > > > might
> > > > > >> > > > > > >> > > > > > > > > > > > require
> > > > > >> > > > > > >> > > > > > > > > > > > >> a
> > > > > >> > > > > > >> > > > > > > > > > > > >> >> few
> > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > > roundtrips, so
> > > > opening a
> > > > > >> > > separate
> > > > > >> > > > > > >> thread
> > > > > >> > > > > > >> > > for
> > > > > >> > > > > > >> > > > > that
> > > > > >> > > > > > >> > > > > > > > might
> > > > > >> > > > > > >> > > > > > > > > > be
> > > > > >> > > > > > >> > > > > > > > > > > > >> best.
> > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > > Contributors
> > > commonly
> > > > > >> create
> > > > > >> > > > > Google
> > > > > >> > > > > > >> Docs
> > > > > >> > > > > > >> > > for
> > > > > >> > > > > > >> > > > > new
> > > > > >> > > > > > >> > > > > > > API
> > > > > >> > > > > > >> > > > > > > > > > > > proposals
> > > > > >> > > > > > >> > > > > > > > > > > > >> too
> > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> (they
> > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > > fairly easy to
> > > > update as
> > > > > >> the
> > > > > >> > > email
> > > > > >> > > > > > >> > > discussion
> > > > > >> > > > > > >> > > > > > > > > > progresses).
> > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > >
> > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > > There was a
> > > > suggestion to
> > > > > >> > try
> > > > > >> > > > > > Markdown
> > > > > >> > > > > > >> > > (with
> > > > > >> > > > > > >> > > > > PRs)
> > > > > >> > > > > > >> > > > > > > for
> > > > > >> > > > > > >> > > > > > > > > > > > proposals
> > > > > >> > > > > > >> > > > > > > > > > > > >> >> [1]
> > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> ...
> > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > > feel free to give
> > > it
> > > > a
> > > > > >> try
> > > > > >> > if
> > > > > >> > > you
> > > > > >> > > > > > are
> > > > > >> > > > > > >> > > > > comfortable
> > > > > >> > > > > > >> > > > > > > > with
> > > > > >> > > > > > >> > > > > > > > > > > that.
> > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > >
> > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > > 2. Could you
> > > clarify
> > > > > >> whether
> > > > > >> > > you
> > > > > >> > > > > > mean
> > > > > >> > > > > > >> end
> > > > > >> > > > > > >> > > > user
> > > > > >> > > > > > >> > > > > > > > > utilities
> > > > > >> > > > > > >> > > > > > > > > > or
> > > > > >> > > > > > >> > > > > > > > > > > > >> admin
> > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> user
> > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > > utilities? In the
> > > > latter
> > > > > >> > case
> > > > > >> > > > > those
> > > > > >> > > > > > >> might
> > > > > >> > > > > > >> > > be
> > > > > >> > > > > > >> > > > > more
> > > > > >> > > > > > >> > > > > > > > > > suitable
> > > > > >> > > > > > >> > > > > > > > > > > > for
> > > > > >> > > > > > >> > > > > > > > > > > > >> the
> > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> Admin
> > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > > CLI (java) not
> > the
> > > > Python
> > > > > >> > CLI,
> > > > > >> > > > > IMHO.
> > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > >
> > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > > Why would these
> > > > > >> utilities be
> > > > > >> > > > > common
> > > > > >> > > > > > >> with
> > > > > >> > > > > > >> > > > > events?
> > > > > >> > > > > > >> > > > > > > > IMHO,
> > > > > >> > > > > > >> > > > > > > > > > > event
> > > > > >> > > > > > >> > > > > > > > > > > > >> use
> > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> cases
> > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > are
> > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > > distinct from
> > > > scan/commit
> > > > > >> > > metrics.
> > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > >
> > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > > 3. I'd prefer
> > > > separating
> > > > > >> > > metrics
> > > > > >> > > > > > >> > > persistence
> > > > > >> > > > > > >> > > > > from
> > > > > >> > > > > > >> > > > > > > > > > MetaStore
> > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> persistence
> > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > at
> > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > > the code level,
> > so
> > > > that
> > > > > >> they
> > > > > >> > > could
> > > > > >> > > > > > be
> > > > > >> > > > > > >> > mixed
> > > > > >> > > > > > >> > > > and
> > > > > >> > > > > > >> > > > > > > > matched
> > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> independently.
> > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > The
> > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > > separate
> > datasource
> > > > > >> question
> > > > > >> > > will
> > > > > >> > > > > > >> become
> > > > > >> > > > > > >> > a
> > > > > >> > > > > > >> > > > > > > non-issue
> > > > > >> > > > > > >> > > > > > > > > with
> > > > > >> > > > > > >> > > > > > > > > > > > that
> > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> approach,
> > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > I
> > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > > guess.
> > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > >
> > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > > The rationale for
> > > > > >> separating
> > > > > >> > > scan
> > > > > >> > > > > > >> metrics
> > > > > >> > > > > > >> > > and
> > > > > >> > > > > > >> > > > > > > > metastore
> > > > > >> > > > > > >> > > > > > > > > > > > >> >> persistence
> > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> is
> > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > that
> > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > > "cascading
> > deletes"
> > > > > >> between
> > > > > >> > > them
> > > > > >> > > > > are
> > > > > >> > > > > > >> > hardly
> > > > > >> > > > > > >> > > > > ever
> > > > > >> > > > > > >> > > > > > > > > > required.
> > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> Furthermore,
> > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > the
> > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > > data and query
> > > > patterns
> > > > > >> are
> > > > > >> > > very
> > > > > >> > > > > > >> > different
> > > > > >> > > > > > >> > > so
> > > > > >> > > > > > >> > > > > > > > different
> > > > > >> > > > > > >> > > > > > > > > > > > >> >> technologies
> > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > might
> > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > > be beneficial in
> > > each
> > > > > >> case.
> > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > >
> > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > > [1]
> > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> >
> > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>>
> > > > > >> > > > > > >> > > > > > > > > > > > >> >>
> > > > > >> > > > > > >> > > > > > > > > > > > >>
> > > > > >> > > > > > >> > > > > > > > > > > >
> > > > > >> > > > > > >> > > > > > > > > > >
> > > > > >> > > > > > >> > > > > > > > > >
> > > > > >> > > > > > >> > > > > > > > >
> > > > > >> > > > > > >> > > > > > > >
> > > > > >> > > > > > >> > > > > > >
> > > > > >> > > > > > >> > > > > >
> > > > > >> > > > > > >> > > > >
> > > > > >> > > > > > >> > > >
> > > > > >> > > > > > >> > >
> > > > > >> > > > > > >> >
> > > > > >> > > > > > >>
> > > > > >> > > > > >
> > > > > >> > > > >
> > > > > >> > >
> > > > > >> >
> > > > > >>
> > > >
> > >
> > https://urldefense.com/v3/__https://lists.apache.org/thread/yto2wp982t43h1mqjwnslswhws5z47cy__;!!Iz9xO38YGHZK!5EuyFFkk3vhRWVIRvQAWBSQfpJkTMA9HxugzDwXmN0LPPqhEFxYkFRGVhtb8AqUwXtDh2OplcMnbMDHKxYDakNU$
> > > > > >> > >> > > >> > > > > > > > > > > > >> >> >>> > >
> > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > > Cheers,
> > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > > Dmitri.
> > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > >
> > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > > On Fri, Feb 27,
> > > 2026
> > > > at
> > > > > >> > > 6:19 PM
> > > > > >> > > > > > Anand
> > > > > >> > > > > > >> > Kumar
> > > > > >> > > > > > >> > > > > > > Sankaran
> > > > > >> > > > > > >> > > > > > > > > via
> > > > > >> > > > > > >> > > > > > > > > > > dev
> > > > > >> > > > > > >> > > > > > > > > > > > <
> > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > >
> > > > dev@polaris.apache.org>
> > > > > >> > > wrote:
> > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > >
> > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > > > Thanks all.
> > This
> > > > PR is
> > > > > >> > > merged
> > > > > >> > > > > now.
> > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > > >
> > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > > > Here are the
> > > > follow-up
> > > > > >> > > features
> > > > > >> > > > > /
> > > > > >> > > > > > >> work
> > > > > >> > > > > > >> > > > > needed.
> > > > > >> > > > > > >> > > > > > > > These
> > > > > >> > > > > > >> > > > > > > > > > > were
> > > > > >> > > > > > >> > > > > > > > > > > > >> all
> > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> part of
> > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > > the
> > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > > > merged PR at
> > some
> > > > > >> point in
> > > > > >> > > time
> > > > > >> > > > > > and
> > > > > >> > > > > > >> > were
> > > > > >> > > > > > >> > > > > > removed
> > > > > >> > > > > > >> > > > > > > to
> > > > > >> > > > > > >> > > > > > > > > > > reduce
> > > > > >> > > > > > >> > > > > > > > > > > > >> >> scope.
> > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > > >
> > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > > > Please let me
> > > know
> > > > what
> > > > > >> > you
> > > > > >> > > > > think.
> > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > > >
> > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > > >
> > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > > >   1.  A REST
> > API
> > > to
> > > > > >> > paginate
> > > > > >> > > > > > through
> > > > > >> > > > > > >> > > table
> > > > > >> > > > > > >> > > > > > > metrics.
> > > > > >> > > > > > >> > > > > > > > > > This
> > > > > >> > > > > > >> > > > > > > > > > > > >> will be
> > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > non-IRC
> > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > > > standard
> > > addition.
> > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > > >   2.  Utilities
> > > for
> > > > > >> > > managing old
> > > > > >> > > > > > >> > records,
> > > > > >> > > > > > >> > > > > > should
> > > > > >> > > > > > >> > > > > > > be
> > > > > >> > > > > > >> > > > > > > > > > > common
> > > > > >> > > > > > >> > > > > > > > > > > > >> with
> > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> events.
> > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > > > There was some
> > > > > >> discussion
> > > > > >> > > that
> > > > > >> > > > > it
> > > > > >> > > > > > >> > belongs
> > > > > >> > > > > > >> > > > to
> > > > > >> > > > > > >> > > > > > the
> > > > > >> > > > > > >> > > > > > > > CLI.
> > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > > >   3.  Separate
> > > > > >> datasource
> > > > > >> > > > > > (metrics,
> > > > > >> > > > > > >> > > events,
> > > > > >> > > > > > >> > > > > > even
> > > > > >> > > > > > >> > > > > > > > > other
> > > > > >> > > > > > >> > > > > > > > > > > > >> tables?).
> > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > > >
> > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > > >
> > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > > > Anything else?
> > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > > >
> > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > > > -
> > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > > > Anand
> > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > > >
> > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > > >
> > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > >
> > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> >
> > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> >
> > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>>
> > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>
> > > > > >> > > > > > >> > > > > > > > > > > > >> >>
> > > > > >> > > > > > >> > > > > > > > > > > > >> >
> > > > > >> > > > > > >> > > > > > > > > > > > >>
> > > > > >> > > > > > >> > > > > > > > > > > > >
> > > > > >> > > > > > >> > > > > > > > > > > >
> > > > > >> > > > > > >> > > > > > > > > > >
> > > > > >> > > > > > >> > > > > > > > > >
> > > > > >> > > > > > >> > > > > > > > >
> > > > > >> > > > > > >> > > > > > > >
> > > > > >> > > > > > >> > > > > > >
> > > > > >> > > > > > >> > > > > >
> > > > > >> > > > > > >> > > > >
> > > > > >> > > > > > >> > > >
> > > > > >> > > > > > >> > >
> > > > > >> > > > > > >> >
> > > > > >> > > > > > >>
> > > > > >> > > > > > >
> > > > > >> > > > > >
> > > > > >> > > > >
> > > > > >> > >
> > > > > >> >
> > > > > >> >
> > > > > >>
> > > > > >
> > > >
> > >
> >

Re: Proposal for REST endpoints for table metrics and events

[Yufei Gu <fl...@gmail.com>]

Thanks Dmitri for the reminder.

PR #5659 <https://github.com/apache/polaris/pull/5659> adds
TABLE_READ_METRICS and says it is implied by TABLE_FULL_METADATA. Should
that be the default?

I’m leaning toward keeping it separate. Metrics describe workloads (Spark
ETL, Flink streaming, etc) using a table and can include query filters, so
someone managing table metadata may not need access to other users’ query
activity.

Including it could make troubleshooting easier for table admins, though.
Are there use cases that favor automatic access over an explicit grant?
Yufei


On Wed, Sep 30, 2026 at 1:57 PM Dmitri Bourlatchkov <di...@apache.org>
wrote:

> Hi Yufei,
>
> Here's the separate PR for Metrics-related grants:
> https://github.com/apache/polaris/pull/5659
>
> Please review it to unblock this contribution.
>
> Thanks,
> Dmitri.
>
> On Wed, Sep 30, 2026 at 1:05 PM Yufei Gu <fl...@gmail.com> wrote:
>
> > From the previous email, we agreed to split the REST change from the
> grant
> > change. I think the PR is ready to go if we only keep the grant changes.
> >
> > Yufei
> >
> >
> > On Tue, Sep 29, 2026 at 9:26 PM Jean-Baptiste Onofré <jb...@nanthrax.net>
> > wrote:
> >
> > > Hi guys,
> > >
> > > I agree with Dmitri: +1 to merge PR #4115 on Oct 1 (tomorrow).
> > >
> > > From my side, all outstanding feedback has been addressed, including
> > > the final documentation points raised by EJ regarding the empty-200
> > > NoOp behavior and reconstructed payload description. The branch is
> > > clean, up to date with main, and CI is green.
> > >
> > > Once this lands, we can follow up with PR #4756 for the JDBC query SPI
> > > implementation.
> > >
> > > Regards
> > > JB
> > >
> > > On Wed, Sep 30, 2026 at 1:53 AM Dmitri Bourlatchkov <di...@apache.org>
> > > wrote:
> > > >
> > > > Hi All,
> > > >
> > > > Any remaining blockers on [4115]?
> > > >
> > > > Given that the review was very long, please restate any remaining
> > > concerns
> > > > (here or in GH).
> > > >
> > > > Otherwise, I propose merging [4115] on Oct 1.
> > > >
> > > > [4115] https://github.com/apache/polaris/pull/4115
> > > >
> > > > Thanks,
> > > > Dmitri.
> > > >
> > > > On Wed, Sep 23, 2026 at 7:53 PM Dmitri Bourlatchkov <
> dimas@apache.org>
> > > > wrote:
> > > >
> > > > > Hi EJ,
> > > > >
> > > > > Thanks for the meeting recap!
> > > > >
> > > > > All:
> > > > >
> > > > > Worth noting: PR [4115] also contains Management API changes that
> > were
> > > not
> > > > > anticipated initially and did not get an API change review until
> now.
> > > These
> > > > > are small changes to add new grant types for metrics RBAC..
> > > > >
> > > > > I've reviewed and approved PR [4115] in GH.
> > > > >
> > > > > [4115] https://github.com/apache/polaris/pull/4115
> > > > >
> > > > > Cheers,
> > > > > Dmitri.
> > > > >
> > > > > On Wed, Aug 12, 2026 at 5:21 PM EJ Wang <
> > > ej.wang.dev.interest@gmail.com>
> > > > > wrote:
> > > > >
> > > > >> Hi folks,
> > > > >>
> > > > >> Notes from today's metrics architecture sync are in the usual doc:
> > > > >>
> > > > >>
> > >
> >
> https://docs.google.com/document/d/100h7c4damrUzVuquYbBHM0EvA4LSWuW2IT2dN_7nYVA/edit?pli=1&tab=t.18iv1m7p6i6b
> > > > >>
> > > > >> Highlights:
> > > > >>
> > > > >> Module placement. core/ now holds the entity data model only. A
> new
> > > spi/
> > > > >> takes every other shared contract: feature SPIs, feature non-SPI
> > > > >> contracts,
> > > > >> substrate SPIs, substrate non-SPI contracts, and contract-required
> > > types.
> > > > >> That settles the question PR #5204 reopened, by placing into a new
> > SPI
> > > > >> module. Note that it changes slightly towards the referenced WIP
> RFC
> > > [1]
> > > > >> recommendations: what proposed to place in core/, now should go to
> > > spi/
> > > > >> except for the entity data model.
> > > > >>
> > > > >> Metrics PRs. #5068 merged. #5204 needs reshaping onto the split.
> > > #4115 and
> > > > >> #4756 stay open with action on the author to follow up on review
> > > comments.
> > > > >>
> > > > >> [1]
> > > > >>
> > > > >>
> > >
> >
> https://docs.google.com/document/d/1mj3vGpix9zTDTFFJKtWT5ZxI3PTGI1YNb5xFqyWrOkc/edit?tab=t.0
> > > > >>
> > > > >> On Wed, Jul 15, 2026 at 11:01 AM Anand Kumar Sankaran via dev <
> > > > >> dev@polaris.apache.org> wrote:
> > > > >>
> > > > >> > Thanks EJ.  My plan is to introduce PR0 in a new branch / PR and
> > > > >> repurpose
> > > > >> > #4115 and #4756 for PR1 and PR2 to maintain all the valuable
> > > discussions
> > > > >> > that happened there.
> > > > >> >
> > > > >> > From: EJ Wang <ej...@gmail.com>
> > > > >> > Date: Wednesday, July 15, 2026 at 10:41 AM
> > > > >> > To: dev@polaris.apache.org <de...@polaris.apache.org>
> > > > >> > Subject: Re: Proposal for REST endpoints for table metrics and
> > > events
> > > > >> >
> > > > >> > Hi folks, Thanks again for the discussion today. I updated the
> > sync
> > > doc
> > > > >> > with notes from today's metrics sync: https: //urldefense.
> > > > >> com/v3/__https:
> > > > >> > //docs. google.
> > > > >> >
> > > > >>
> > >
> >
> com/document/d/100h7c4damrUzVuquYbBHM0EvA4LSWuW2IT2dN_7nYVA/edit?pli=1&tab=t.
> > > > >> >
> > > > >>
> > >
> >
> 2ahd7wze5f__;!!Iz9xO38YGHZK!4rnlVIqBQQQ6xpGe1PMV58edOS_cGrKoqdfhw4qSp_JQCavvEjaxXk0OwGPnxyEcO60AzKQK_XDtp7H3DdkFXcrXJwAz1g0$
> > > > >> >
> > > > >> >
> > > > >> > Hi folks,
> > > > >> >
> > > > >> > Thanks again for the discussion today. I updated the sync doc
> with
> > > notes
> > > > >> > from today's metrics sync:
> > > > >> >
> > > > >> >
> > > > >>
> > >
> >
> https://urldefense.com/v3/__https://docs.google.com/document/d/100h7c4damrUzVuquYbBHM0EvA4LSWuW2IT2dN_7nYVA/edit?pli=1&tab=t.2ahd7wze5f__;!!Iz9xO38YGHZK!4rnlVIqBQQQ6xpGe1PMV58edOS_cGrKoqdfhw4qSp_JQCavvEjaxXk0OwGPnxyEcO60AzKQK_XDtp7H3DdkFXcrXJwAz1g0$
> > > > >> >
> > > > >> > See highlights below:
> > > > >> >
> > > > >> > In previous meetings, EJ proposed a two prs framing to push
> > forward
> > > the
> > > > >> > metrics query work in polaris. Dmitri pointed out a subtle point
> > > that
> > > > >> pr2
> > > > >> > is currently also trying to do the persistence related code
> > > placement
> > > > >> > refactor. For the sake of straightforward review process, EJ,
> > > Dmitri and
> > > > >> > Anand (the author) agreed to separate the persistence
> refactoring
> > > into
> > > > >> pr0,
> > > > >> > and reframing the pr scopes as follow:
> > > > >> >
> > > > >> >    1.
> > > > >> >
> > > > >> >    Pr0: refactor irc metrics spi
> > > > >> >    1.
> > > > >> >
> > > > >> >       SPI should stay in core/ with metric type enveloped
> > > > >> >       2.
> > > > >> >
> > > > >> >       default spi impl, no-op, in extensions/ - default
> selection
> > at
> > > > >> >       runtime/
> > > > >> >       3.
> > > > >> >
> > > > >> >       move metrics persistence and correspondings to extensions/
> > as
> > > an
> > > > >> >       alternative SPI impl
> > > > >> >       2.
> > > > >> >
> > > > >> >    Pr1: rest api for query
> > > > >> >
> > > > >> >
> > > > >>
> > >
> >
> https://urldefense.com/v3/__https://github.com/apache/polaris/pull/4115/changes__;!!Iz9xO38YGHZK!4rnlVIqBQQQ6xpGe1PMV58edOS_cGrKoqdfhw4qSp_JQCavvEjaxXk0OwGPnxyEcO60AzKQK_XDtp7H3DdkFXcrXFCbEbRk$
> > > > >> >    1.
> > > > >> >
> > > > >> >       rest api spec in extensions/ as optional spec
> > > > >> >       2.
> > > > >> >
> > > > >> >       SPI in extensions/ as optional spec spi
> > > > >> >       3.
> > > > >> >
> > > > >> >       default spi impl in extensions/ no-op
> > > > >> >       4.
> > > > >> >
> > > > >> >       handler related in runtime/ - thin adaptor from http shape
> > to
> > > spi
> > > > >> >       shape
> > > > >> >       3.
> > > > >> >
> > > > >> >    Pr2: jdbc impl for query
> > > > >> >
> > > > >>
> > >
> >
> https://urldefense.com/v3/__https://github.com/apache/polaris/pull/4756__;!!Iz9xO38YGHZK!4rnlVIqBQQQ6xpGe1PMV58edOS_cGrKoqdfhw4qSp_JQCavvEjaxXk0OwGPnxyEcO60AzKQK_XDtp7H3DdkFXcrXRVMBzqE$
> > > > >> >    1.
> > > > >> >
> > > > >> >       alternative spi impl in extensions/ -> query with jdbc
> > > > >> >
> > > > >> >
> > > > >> > Thanks,
> > > > >> > -ej
> > > > >> >
> > > > >> > On Wed, Jul 1, 2026 at 8:01 AM Alexandre Dutra <
> adutra@apache.org
> > >
> > > > >> wrote:
> > > > >> >
> > > > >> > > Hi all,
> > > > >> > >
> > > > >> > > > While the current implementation stores metrics in the
> > > metastore,
> > > > >> other
> > > > >> > > deployments
> > > > >> > > may choose a KV store, a dedicated metrics warehouse, or
> another
> > > > >> backend
> > > > >> > > entirely. In those cases, exposing a Polaris REST API over a
> > > specific
> > > > >> > > schema feels like coupling the API to an implementation
> detail.
> > > > >> > >
> > > > >> > > I wouldn't say to an implementation detail, but rather to a
> > > particular
> > > > >> > > setup / configuration. I think that's fine. If you don't store
> > > > >> > > metrics, you don't read metrics. Similar situations will
> > certainly
> > > > >> > > happen in the future, e.g. if we ever introduce an API to read
> > > events,
> > > > >> > > that would only work if events are being persisted in the
> first
> > > place.
> > > > >> > > Again I think that's a fair trade-off.
> > > > >> > >
> > > > >> > > Thanks,
> > > > >> > > Alex
> > > > >> > >
> > > > >> > > On Wed, Jul 1, 2026 at 7:23 AM Dmitri Bourlatchkov <
> > > dimas@apache.org>
> > > > >> > > wrote:
> > > > >> > > >
> > > > >> > > > Hi Yufei,
> > > > >> > > >
> > > > >> > > > From my POV, it is fine for Polaris to have multiple REST
> API
> > > > >> surfaces
> > > > >> > > with
> > > > >> > > > different features and different degrees of "readiness".
> Each
> > > API
> > > > >> can
> > > > >> > > > evolve fairly independently. With proper modularization
> (which
> > > Anand
> > > > >> > > did),
> > > > >> > > > changes to the main Polaris code and these extra modules
> > should
> > > > >> > normally
> > > > >> > > be
> > > > >> > > > non-intrusive to each other.
> > > > >> > > >
> > > > >> > > > I assume Anand is going to be the first user of this API.
> > > However,
> > > > >> if
> > > > >> > we
> > > > >> > > do
> > > > >> > > > not merge it we'll never know whether other users are
> > > interested. I
> > > > >> do
> > > > >> > > not
> > > > >> > > > think many end users read the dev ML, but they certainly
> read
> > > the
> > > > >> docs,
> > > > >> > > > code and release notes.
> > > > >> > > >
> > > > >> > > > There is no risk for Polaris as a project in exposing this
> > > > >> experimental
> > > > >> > > > API. If usage dwindles over time, we can remove it. If it
> > gains
> > > a
> > > > >> > > > considerable user / contributor base, we will promote it to
> a
> > > > >> > > "first-class"
> > > > >> > > > (non-beta) API.
> > > > >> > > >
> > > > >> > > > Cheers,
> > > > >> > > > Dmitri.
> > > > >> > > >
> > > > >> > > > On Tue, Jun 30, 2026 at 7:53 PM Yufei Gu <
> > flyrain000@gmail.com>
> > > > >> wrote:
> > > > >> > > >
> > > > >> > > > > Hi all,
> > > > >> > > > >
> > > > >> > > > > Thanks for driving this work forward. Building this as a
> POC
> > > makes
> > > > >> > > perfect
> > > > >> > > > > sense if the goal is to validate the idea and gather
> > feedback.
> > > > >> Should
> > > > >> > > we
> > > > >> > > > > merge a POC into the main branch before agreeing on the
> > > overall
> > > > >> > > direction?
> > > > >> > > > > To me, that feels like a separate discussion from whether
> > the
> > > > >> > > > > implementation itself looks reasonable.
> > > > >> > > > >
> > > > >> > > > > One concern that I don't think has been fully addressed is
> > > > >> whether we
> > > > >> > > need
> > > > >> > > > > to expose metrics consumption through a Polaris REST
> > endpoint
> > > at
> > > > >> all.
> > > > >> > > > >
> > > > >> > > > > My concern is less about the API itself and more about the
> > > > >> boundary
> > > > >> > > Polaris
> > > > >> > > > > should own. The proposed REST API is tied to a particular
> > > > >> persistence
> > > > >> > > > > schema, but that schema is just one possible
> implementation.
> > > While
> > > > >> > the
> > > > >> > > > > current implementation stores metrics in the metastore,
> > other
> > > > >> > > deployments
> > > > >> > > > > may choose a KV store, a dedicated metrics warehouse, or
> > > another
> > > > >> > > backend
> > > > >> > > > > entirely. In those cases, exposing a Polaris REST API
> over a
> > > > >> specific
> > > > >> > > > > schema feels like coupling the API to an implementation
> > > detail.
> > > > >> > > > >
> > > > >> > > > > Polaris isn't intended to become a full metrics system.
> I'd
> > > > >> expect it
> > > > >> > > to
> > > > >> > > > > focus on producing metrics and let deployments choose the
> > most
> > > > >> > > appropriate
> > > > >> > > > > storage and consumption model for their environment.
> > > > >> > > > >
> > > > >> > > > > I'm not opposed to exploring this direction, but we should
> > > first
> > > > >> > align
> > > > >> > > on
> > > > >> > > > > whether exposing a REST API through Polaris is the right
> > long
> > > term
> > > > >> > > approach
> > > > >> > > > > before merging it into the main branch.
> > > > >> > > > >
> > > > >> > > > > Thanks,
> > > > >> > > > > Yufei
> > > > >> > > > >
> > > > >> > > > > Yufei
> > > > >> > > > >
> > > > >> > > > >
> > > > >> > > > > On Mon, Jun 29, 2026 at 9:27 AM Dmitri Bourlatchkov <
> > > > >> > dimas@apache.org>
> > > > >> > > > > wrote:
> > > > >> > > > >
> > > > >> > > > > > Hi All,
> > > > >> > > > > >
> > > > >> > > > > > It looks like Anand addressed all action items in
> [4115].
> > > > >> > > > > >
> > > > >> > > > > > Please review the latest state of the code. The PR LGTM
> > and
> > > I
> > > > >> > > approved in
> > > > >> > > > > > GH from my side.
> > > > >> > > > > >
> > > > >> > > > > > [4115]
> > > > >> >
> > > > >>
> > >
> >
> https://urldefense.com/v3/__https://github.com/apache/polaris/pull/4115__;!!Iz9xO38YGHZK!4rnlVIqBQQQ6xpGe1PMV58edOS_cGrKoqdfhw4qSp_JQCavvEjaxXk0OwGPnxyEcO60AzKQK_XDtp7H3DdkFXcrXtFRscKs$
> > > > >> > > > > >
> > > > >> > > > > > Thanks,
> > > > >> > > > > > Dmitri.
> > > > >> > > > > >
> > > > >> > > > > > On Thu, Jun 25, 2026 at 1:56 PM Dmitri Bourlatchkov <
> > > > >> > > dimas@apache.org>
> > > > >> > > > > > wrote:
> > > > >> > > > > >
> > > > >> > > > > > > Hi All,
> > > > >> > > > > > >
> > > > >> > > > > > > Recap of the discussion in today's Community Sync:
> > > > >> > > > > > >
> > > > >> > > > > > > * The modular approach for Metrics SPI(s) and
> > > implementation
> > > > >> > seems
> > > > >> > > to
> > > > >> > > > > > have
> > > > >> > > > > > > community consensus (please comment if you still have
> > > > >> concerns)
> > > > >> > > > > > >
> > > > >> > > > > > > * What modules / feature flags are enabled in Apache
> > > Polaris
> > > > >> > > images by
> > > > >> > > > > > > default will be discussed later (when the feature is
> > > close to
> > > > >> > > > > > completion).
> > > > >> > > > > > >
> > > > >> > > > > > > In any case, users will be able to turn the full
> feature
> > > > >> on/off
> > > > >> > at
> > > > >> > > > > their
> > > > >> > > > > > > discretion, regardless of the default flag values.
> > > > >> > > > > > >
> > > > >> > > > > > > * It is generally preferable to have smaller PRs and
> > > introduce
> > > > >> > new
> > > > >> > > > > > > features gradually to simplify reviews.
> > > > >> > > > > > >
> > > > >> > > > > > > What this means for the Metrics API, IMHO is:
> > > > >> > > > > > >
> > > > >> > > > > > > a) The scope of PR 4115 is probably ok as it stands
> > right
> > > > >> now. If
> > > > >> > > > > someone
> > > > >> > > > > > > wants to propose splitting it further, please comment.
> > > > >> > > > > > >
> > > > >> > > > > > > b) After 4115 is merged, Metrics Persistence PRs will
> > > follow,
> > > > >> > with
> > > > >> > > > > > smaller
> > > > >> > > > > > > scope per PR as appropriate for the nature of the code
> > > > >> changes.
> > > > >> > > > > > >
> > > > >> > > > > > > * The major concern was about exposing the new Metrics
> > > REST
> > > > >> API
> > > > >> > to
> > > > >> > > end
> > > > >> > > > > > > users. The nature of this concern was about avoiding
> the
> > > > >> (false)
> > > > >> > > > > > impression
> > > > >> > > > > > > that the new API is a firm and adopted Polaris
> > > specification.
> > > > >> > > > > > >
> > > > >> > > > > > > I think this concern is reasonable.
> > > > >> > > > > > >
> > > > >> > > > > > > I propose that Anand update the documentation sections
> > of
> > > PR
> > > > >> 4115
> > > > >> > > to
> > > > >> > > > > > > emphasise that the API is currently an experimental /
> > POC
> > > API
> > > > >> at
> > > > >> > > this
> > > > >> > > > > > time
> > > > >> > > > > > > and is subject to change (including breaking
> changes). I
> > > > >> believe
> > > > >> > it
> > > > >> > > > > > already
> > > > >> > > > > > > has the "beta" label. So this request is mostly for
> doc
> > > > >> changes
> > > > >> > to
> > > > >> > > > > > clarify
> > > > >> > > > > > > user expectations, since the "beta" label alone may
> not
> > be
> > > > >> > > interpreted
> > > > >> > > > > > the
> > > > >> > > > > > > same way by all users.
> > > > >> > > > > > >
> > > > >> > > > > > > I hope this update will resolve all remaining concerns
> > > with
> > > > >> 4115
> > > > >> > > and
> > > > >> > > > > > allow
> > > > >> > > > > > > for it to be merged so that the community will be able
> > to
> > > > >> proceed
> > > > >> > > to
> > > > >> > > > > > > follow-up PRs (as noted above).
> > > > >> > > > > > >
> > > > >> > > > > > > Please add/correct me if I missed anything relevant.
> > > > >> > > > > > >
> > > > >> > > > > > > Thanks,
> > > > >> > > > > > > Dmitri.
> > > > >> > > > > > >
> > > > >> > > > > > > On Wed, Jun 24, 2026 at 8:11 PM Yufei Gu <
> > > > >> flyrain000@gmail.com>
> > > > >> > > wrote:
> > > > >> > > > > > >
> > > > >> > > > > > >> I do not think we have agreement on the query REST
> APIs
> > > yet.
> > > > >> > > > > > >>
> > > > >> > > > > > >> In particular, the current query API shape seems
> highly
> > > > >> coupled
> > > > >> > > to the
> > > > >> > > > > > >> *example[1]* metrics in the Iceberg REST spec. I do
> not
> > > think
> > > > >> > > that is
> > > > >> > > > > a
> > > > >> > > > > > >> good way to design a REST API contract. For example,
> > > fields
> > > > >> like
> > > > >> > > > > > >> resultDataFiles, resultDeleteFiles,
> totalFileSizeBytes,
> > > > >> > > > > > >> totalDataManifests,
> > > > >> > > > > > >> totalDeleteManifests, and scannedDataManifests are
> very
> > > > >> specific
> > > > >> > > to
> > > > >> > > > > one
> > > > >> > > > > > >> style of scan metric. That makes the API look more
> > like a
> > > > >> direct
> > > > >> > > > > > >> projection
> > > > >> > > > > > >> of the current example payload than a general query
> > > model.
> > > > >> > > > > > >>
> > > > >> > > > > > >> My preference is that Polaris should not define or
> > > strongly
> > > > >> > opine
> > > > >> > > on
> > > > >> > > > > the
> > > > >> > > > > > >> Iceberg scan or commit metrics query REST API at this
> > > point.
> > > > >> > > Polaris
> > > > >> > > > > can
> > > > >> > > > > > >> resolve and authorize the table, accept the metrics
> > > report,
> > > > >> and
> > > > >> > > > > delegate
> > > > >> > > > > > >> it
> > > > >> > > > > > >> to the configured reporting path. Durable storage,
> > > filtering,
> > > > >> > > > > > dashboards,
> > > > >> > > > > > >> and query APIs should remain downstream impl. choices
> > > unless
> > > > >> and
> > > > >> > > until
> > > > >> > > > > > the
> > > > >> > > > > > >> Iceberg community standardizes a query contract,
> which
> > I
> > > > >> doubt
> > > > >> > > will
> > > > >> > > > > > >> happen.
> > > > >> > > > > > >>
> > > > >> > > > > > >> So I do not think the intake path has to be blocked,
> > but
> > > I
> > > > >> would
> > > > >> > > > > prefer
> > > > >> > > > > > >> not
> > > > >> > > > > > >> to merge a Polaris specific query API that may be
> hard
> > to
> > > > >> evolve
> > > > >> > > > > later.
> > > > >> > > > > > >>
> > > > >> > > > > > >> 1.
> > > > >> > > > > > >>
> > > > >> > > > > > >>
> > > > >> > > > > >
> > > > >> > > > >
> > > > >> > >
> > > > >> >
> > > > >>
> > >
> >
> https://urldefense.com/v3/__https://github.com/apache/iceberg/blob/2a6c556c0c883c04c6d3fbd68e6aeda36b91e0aa/open-api/rest-catalog-open-api.yaml*L4152__;Iw!!Iz9xO38YGHZK!4rnlVIqBQQQ6xpGe1PMV58edOS_cGrKoqdfhw4qSp_JQCavvEjaxXk0OwGPnxyEcO60AzKQK_XDtp7H3DdkFXcrXCDsqYGo$
> > > > >> > > > > > >>
> > > > >> > > > > > >> Yufei
> > > > >> > > > > > >>
> > > > >> > > > > > >>
> > > > >> > > > > > >> On Tue, Jun 23, 2026 at 9:30 AM Dmitri Bourlatchkov <
> > > > >> > > dimas@apache.org
> > > > >> > > > > >
> > > > >> > > > > > >> wrote:
> > > > >> > > > > > >>
> > > > >> > > > > > >> > Hi All,
> > > > >> > > > > > >> >
> > > > >> > > > > > >> > So, what are the blockers for merging [4115], if
> any?
> > > > >> > > > > > >> >
> > > > >> > > > > > >> > [4115]
> > > > >> >
> > > > >>
> > >
> >
> https://urldefense.com/v3/__https://github.com/apache/polaris/pull/4115__;!!Iz9xO38YGHZK!4rnlVIqBQQQ6xpGe1PMV58edOS_cGrKoqdfhw4qSp_JQCavvEjaxXk0OwGPnxyEcO60AzKQK_XDtp7H3DdkFXcrXtFRscKs$
> > > > >> > > > > > >> >
> > > > >> > > > > > >> > Thanks,
> > > > >> > > > > > >> > Dmitri.
> > > > >> > > > > > >> >
> > > > >> > > > > > >> > On Wed, Jun 17, 2026 at 5:28 PM EJ Wang <
> > > > >> > > > > > ej.wang.dev.interest@gmail.com
> > > > >> > > > > > >> >
> > > > >> > > > > > >> > wrote:
> > > > >> > > > > > >> >
> > > > >> > > > > > >> > > Hi folks,
> > > > >> > > > > > >> > >
> > > > >> > > > > > >> > > Thanks again for the discussion today. I updated
> > the
> > > sync
> > > > >> > doc
> > > > >> > > with
> > > > >> > > > > > >> notes
> > > > >> > > > > > >> > > from today's metrics sync:
> > > > >> > > > > > >> > >
> > > > >> > > > > > >> > >
> > > > >> > > > > > >> > >
> > > > >> > > > > > >> >
> > > > >> > > > > > >>
> > > > >> > > > > >
> > > > >> > > > >
> > > > >> > >
> > > > >> >
> > > > >>
> > >
> >
> https://urldefense.com/v3/__https://docs.google.com/document/d/100h7c4damrUzVuquYbBHM0EvA4LSWuW2IT2dN_7nYVA/edit?tab=t.ezk0rgdx0c6m__;!!Iz9xO38YGHZK!4rnlVIqBQQQ6xpGe1PMV58edOS_cGrKoqdfhw4qSp_JQCavvEjaxXk0OwGPnxyEcO60AzKQK_XDtp7H3DdkFXcrX83IZkm0$
> > > > >> > > > > > >> > >
> > > > >> > > > > > >> > > A few highlights:
> > > > >> > > > > > >> > >
> > > > >> > > > > > >> > >    - No objection to keeping the default metrics
> > > behavior
> > > > >> > > small:
> > > > >> > > > > > >> > >    no-op/log-only is enough as the built-in
> > default.
> > > > >> > > > > > >> > >    - Durable storage, event forwarding, external
> > > queues,
> > > > >> > > > > dashboards,
> > > > >> > > > > > >> and
> > > > >> > > > > > >> > >    custom filtering should be implementation
> > choices
> > > > >> behind
> > > > >> > > the
> > > > >> > > > > > >> metrics
> > > > >> > > > > > >> > >    reporting path.
> > > > >> > > > > > >> > >    - For PR #4115, the REST layer should
> > > > >> > resolve/authz/accept
> > > > >> > > the
> > > > >> > > > > > >> report,
> > > > >> > > > > > >> > >    then delegate to the selected metrics
> > > implementation.
> > > > >> It
> > > > >> > > should
> > > > >> > > > > > not
> > > > >> > > > > > >> > own
> > > > >> > > > > > >> > >    durable storage semantics.
> > > > >> > > > > > >> > >    - The rough boundary we discussed is: spec
> owns
> > > the
> > > > >> wire
> > > > >> > > > > > contract,
> > > > >> > > > > > >> > >    runtime owns framework wiring, API/contract
> > > modules
> > > > >> own
> > > > >> > > > > > >> > provider-facing
> > > > >> > > > > > >> > >    contracts, and extensions own replaceable
> > > > >> > implementations.
> > > > >> > > > > > >> > >    - OpenLineage has a similar path-naming
> > question.
> > > We
> > > > >> > should
> > > > >> > > > > avoid
> > > > >> > > > > > >> > >    occupying a generic lineage path too early if
> we
> > > want
> > > > >> > room
> > > > >> > > for
> > > > >> > > > > > >> other
> > > > >> > > > > > >> > >    lineage systems later.
> > > > >> > > > > > >> > >
> > > > >> > > > > > >> > > I believe the boundary rules previewed in sync
> > > meeting
> > > > >> are
> > > > >> > > useful
> > > > >> > > > > > for
> > > > >> > > > > > >> > > reviewing PR #4115, but it is not
> > > > >> > > > > > >> > > a final module-layout proposal.
> > > > >> > > > > > >> > >
> > > > >> > > > > > >> > > For metrics specifically, I think the target
> > remains:
> > > > >> > > > > > >> > >    runtime
> > > > >> > > > > > >> > >      -> metrics reporting/emitting contract
> > > > >> > > > > > >> > >      -> default implementation
> > > > >> > > > > > >> > >           -> lower implementation details, if
> > needed
> > > > >> > > > > > >> > >
> > > > >> > > > > > >> > > That keeps the default simple, while allowing
> > durable
> > > > >> JDBC,
> > > > >> > > > > > >> event-backed,
> > > > >> > > > > > >> > > or external-queue implementations to be added
> > > separately.
> > > > >> > > > > > >> > >
> > > > >> > > > > > >> > > We also touched on the split between Iceberg
> > metrics
> > > > >> > emitting
> > > > >> > > and
> > > > >> > > > > > >> > > Polaris-owned metrics querying. The querying side
> > can
> > > > >> keep
> > > > >> > > > > evolving,
> > > > >> > > > > > >> > > especially for dashboard or generic-table use
> > cases,
> > > but
> > > > >> I
> > > > >> > do
> > > > >> > > not
> > > > >> > > > > > >> think
> > > > >> > > > > > >> > > that needs to block the intake path in this PR.
> > > > >> > > > > > >> > >
> > > > >> > > > > > >> > > Thanks,
> > > > >> > > > > > >> > > -ej
> > > > >> > > > > > >> > >
> > > > >> > > > > > >> > > On Tue, Jun 16, 2026 at 2:17 PM Yufei Gu <
> > > > >> > > flyrain000@gmail.com>
> > > > >> > > > > > >> wrote:
> > > > >> > > > > > >> > >
> > > > >> > > > > > >> > > > Thanks for chiming in, EJ. Agreed that the
> > default
> > > > >> battery
> > > > >> > > > > should
> > > > >> > > > > > >> stay
> > > > >> > > > > > >> > > > small. I'm leaning toward not including metrics
> > > > >> > persistence
> > > > >> > > in
> > > > >> > > > > the
> > > > >> > > > > > >> > > default
> > > > >> > > > > > >> > > > battery.
> > > > >> > > > > > >> > > >
> > > > >> > > > > > >> > > > Yufei
> > > > >> > > > > > >> > > >
> > > > >> > > > > > >> > > >
> > > > >> > > > > > >> > > > On Thu, Jun 11, 2026 at 8:30 PM EJ Wang <
> > > > >> > > > > > >> > ej.wang.dev.interest@gmail.com>
> > > > >> > > > > > >> > > > wrote:
> > > > >> > > > > > >> > > >
> > > > >> > > > > > >> > > > > Hi Yufei,
> > > > >> > > > > > >> > > > >
> > > > >> > > > > > >> > > > > Thanks for connecting this back to the REST
> > > endpoint
> > > > >> > > > > proposal. I
> > > > >> > > > > > >> > > replied
> > > > >> > > > > > >> > > > > with the fuller version on the
> event-forwarding
> > > > >> thread,
> > > > >> > > but
> > > > >> > > > > > >> wanted to
> > > > >> > > > > > >> > > add
> > > > >> > > > > > >> > > > > the shorter version here too.
> > > > >> > > > > > >> > > > >
> > > > >> > > > > > >> > > > > I agree that metrics reporting/emitting is
> the
> > > right
> > > > >> > > > > conceptual
> > > > >> > > > > > >> > > boundary,
> > > > >> > > > > > >> > > > > and I also think the event/listener path is a
> > > good
> > > > >> > > > > > implementation
> > > > >> > > > > > >> > > > direction
> > > > >> > > > > > >> > > > > to explore. The distinction I want to keep
> > clear
> > > is
> > > > >> > > default
> > > > >> > > > > > >> battery
> > > > >> > > > > > >> > vs
> > > > >> > > > > > >> > > > > extension implementation.
> > > > >> > > > > > >> > > > >
> > > > >> > > > > > >> > > > > For the REST/API side, I would keep the
> > semantics
> > > > >> > narrow:
> > > > >> > > > > > Polaris
> > > > >> > > > > > >> > > > resolves
> > > > >> > > > > > >> > > > > and authorizes the table, accepts the Iceberg
> > > > >> > scan/commit
> > > > >> > > > > > metrics
> > > > >> > > > > > >> > > report
> > > > >> > > > > > >> > > > > into the configured reporting/emitting path,
> > and
> > > > >> returns
> > > > >> > > 204.
> > > > >> > > > > > That
> > > > >> > > > > > >> > 204
> > > > >> > > > > > >> > > > > should mean Polaris accepted the report into
> > the
> > > > >> > ingestion
> > > > >> > > > > path;
> > > > >> > > > > > >> it
> > > > >> > > > > > >> > > > should
> > > > >> > > > > > >> > > > > not imply durable storage.
> > > > >> > > > > > >> > > > >
> > > > >> > > > > > >> > > > > Durable storage, event forwarding, external
> > > telemetry
> > > > >> > > routing,
> > > > >> > > > > > >> > > filtering,
> > > > >> > > > > > >> > > > > and retention should sit behind that
> > > > >> reporting/emitting
> > > > >> > > > > boundary
> > > > >> > > > > > >> as
> > > > >> > > > > > >> > > > > implementation-layer behavior. A durable JDBC
> > > path
> > > > >> can
> > > > >> > be
> > > > >> > > one
> > > > >> > > > > > >> named
> > > > >> > > > > > >> > > > > extension implementation. An event/listener
> > > > >> forwarding
> > > > >> > > path
> > > > >> > > > > can
> > > > >> > > > > > be
> > > > >> > > > > > >> > > > another
> > > > >> > > > > > >> > > > > named extension implementation.
> > > > >> > > > > > >> > > > >
> > > > >> > > > > > >> > > > > So I am +1 on the event-forwarding idea as a
> > > > >> non-default
> > > > >> > > > > > extension
> > > > >> > > > > > >> > > > > implementation. I just would not make it the
> > > default
> > > > >> > > battery
> > > > >> > > > > or
> > > > >> > > > > > >> the
> > > > >> > > > > > >> > > core
> > > > >> > > > > > >> > > > > API shape. The default battery can stay small
> > and
> > > > >> safe,
> > > > >> > > while
> > > > >> > > > > > >> > > deployments
> > > > >> > > > > > >> > > > > that need forwarding, dashboards, or durable
> > > storage
> > > > >> can
> > > > >> > > opt
> > > > >> > > > > > into
> > > > >> > > > > > >> the
> > > > >> > > > > > >> > > > > implementation that matches their operational
> > > model.
> > > > >> > > > > > >> > > > >
> > > > >> > > > > > >> > > > > Thanks,
> > > > >> > > > > > >> > > > > -ej
> > > > >> > > > > > >> > > > >
> > > > >> > > > > > >> > > > > On Wed, Jun 10, 2026 at 11:43 AM Yufei Gu <
> > > > >> > > > > flyrain000@gmail.com
> > > > >> > > > > > >
> > > > >> > > > > > >> > > wrote:
> > > > >> > > > > > >> > > > >
> > > > >> > > > > > >> > > > > > Thanks EJ. I agree that the
> > reporting/emitting
> > > > >> > boundary
> > > > >> > > > > feels
> > > > >> > > > > > >> like
> > > > >> > > > > > >> > > the
> > > > >> > > > > > >> > > > > > right SPI boundary, but I wonder if we can
> > > simplify
> > > > >> > this
> > > > >> > > > > even
> > > > >> > > > > > >> > > further.
> > > > >> > > > > > >> > > > > >
> > > > >> > > > > > >> > > > > > Iceberg scan and commit metrics look very
> > > similar
> > > > >> to
> > > > >> > > events
> > > > >> > > > > to
> > > > >> > > > > > >> me.
> > > > >> > > > > > >> > > They
> > > > >> > > > > > >> > > > > are
> > > > >> > > > > > >> > > > > > append only, asynchronously consumed, often
> > > > >> forwarded
> > > > >> > to
> > > > >> > > > > > >> external
> > > > >> > > > > > >> > > > > systems,
> > > > >> > > > > > >> > > > > > and have similar retention and cleanup
> > > > >> requirements.
> > > > >> > > More
> > > > >> > > > > > >> details
> > > > >> > > > > > >> > can
> > > > >> > > > > > >> > > > be
> > > > >> > > > > > >> > > > > > found in this thread [1]. In that case,
> > > Polaris may
> > > > >> > only
> > > > >> > > > > need
> > > > >> > > > > > to
> > > > >> > > > > > >> > > accept
> > > > >> > > > > > >> > > > > the
> > > > >> > > > > > >> > > > > > report and emit an event through the
> existing
> > > event
> > > > >> > > > > framework.
> > > > >> > > > > > >> > > > Filtering,
> > > > >> > > > > > >> > > > > > async delivery, custom sinks, and retention
> > > > >> mechanisms
> > > > >> > > could
> > > > >> > > > > > >> then
> > > > >> > > > > > >> > be
> > > > >> > > > > > >> > > > > shared
> > > > >> > > > > > >> > > > > > instead of introducing a separate metrics
> > > specific
> > > > >> > > extension
> > > > >> > > > > > >> layer
> > > > >> > > > > > >> > > and
> > > > >> > > > > > >> > > > > SPI.
> > > > >> > > > > > >> > > > > >
> > > > >> > > > > > >> > > > > > The main requirement I still see is
> querying
> > > > >> metrics
> > > > >> > > through
> > > > >> > > > > > >> > Polaris.
> > > > >> > > > > > >> > > > If
> > > > >> > > > > > >> > > > > we
> > > > >> > > > > > >> > > > > > want that battery included experience, we
> > could
> > > > >> work
> > > > >> > on
> > > > >> > > a
> > > > >> > > > > > >> > persistence
> > > > >> > > > > > >> > > > > > listener implementation that selectively
> > saves
> > > > >> metrics
> > > > >> > > and
> > > > >> > > > > > >> expose
> > > > >> > > > > > >> > it
> > > > >> > > > > > >> > > > via
> > > > >> > > > > > >> > > > > > the IRC event endpoint which seems pretty
> > > close in
> > > > >> > > Iceberg
> > > > >> > > > > > >> > > > > > community. Another option I like more is to
> > > allow
> > > > >> > users
> > > > >> > > to
> > > > >> > > > > > >> develop
> > > > >> > > > > > >> > > > their
> > > > >> > > > > > >> > > > > > own listener, so they can persist
> scan/commit
> > > > >> metrics
> > > > >> > > to any
> > > > >> > > > > > >> > systems
> > > > >> > > > > > >> > > > they
> > > > >> > > > > > >> > > > > > prefer. In general, that feels like an
> > > > >> implementation
> > > > >> > > choice
> > > > >> > > > > > on
> > > > >> > > > > > >> top
> > > > >> > > > > > >> > > of
> > > > >> > > > > > >> > > > > the
> > > > >> > > > > > >> > > > > > event framework rather than something that
> > > needs to
> > > > >> > > shape
> > > > >> > > > > the
> > > > >> > > > > > >> core
> > > > >> > > > > > >> > > > > > architecture.
> > > > >> > > > > > >> > > > > >
> > > > >> > > > > > >> > > > > > 1.
> > > > >> > > > > > >> >
> > > > >> > >
> > > > >> >
> > > > >>
> > >
> >
> https://urldefense.com/v3/__https://lists.apache.org/thread/x9j8nscvy8hq61tyn01mj8yp6n9of0kp__;!!Iz9xO38YGHZK!4rnlVIqBQQQ6xpGe1PMV58edOS_cGrKoqdfhw4qSp_JQCavvEjaxXk0OwGPnxyEcO60AzKQK_XDtp7H3DdkFXcrXGcGDgks$
> > > > >> > > > > > >> > > > > >
> > > > >> > > > > > >> > > > > > Thanks,
> > > > >> > > > > > >> > > > > > Yufei
> > > > >> > > > > > >> > > > > >
> > > > >> > > > > > >> > > > > >
> > > > >> > > > > > >> > > > > > On Thu, Jun 4, 2026 at 5:14 PM EJ Wang <
> > > > >> > > > > > >> > > ej.wang.dev.interest@gmail.com
> > > > >> > > > > > >> > > > >
> > > > >> > > > > > >> > > > > > wrote:
> > > > >> > > > > > >> > > > > >
> > > > >> > > > > > >> > > > > > > Good point. I agree the existing
> > > > >> > > PolarisMetricsReporter is
> > > > >> > > > > > >> > already
> > > > >> > > > > > >> > > > very
> > > > >> > > > > > >> > > > > > > close to the right conceptual boundary. I
> > am
> > > not
> > > > >> > > > > proposing a
> > > > >> > > > > > >> > second
> > > > >> > > > > > >> > > > > > > parallel reporter concept. The
> distinction
> > I
> > > am
> > > > >> > > trying to
> > > > >> > > > > > >> make is
> > > > >> > > > > > >> > > > > > narrower:
> > > > >> > > > > > >> > > > > > >
> > > > >> > > > > > >> > > > > > > Current state:
> > > > >> > > > > > >> > > > > > >
> > > > >> > > > > > >> > > > > > >    - PolarisMetricsReporter is the
> existing
> > > > >> metrics
> > > > >> > > > > > reporting
> > > > >> > > > > > >> > hook.
> > > > >> > > > > > >> > > > > > >    - But it currently lives under
> > > > >> runtime/service.
> > > > >> > > > > > >> > > > > > >    - Its contract is documented around
> > > > >> Quarkus/CDI
> > > > >> > > > > discovery
> > > > >> > > > > > >> and
> > > > >> > > > > > >> > > > > > > @Identifier
> > > > >> > > > > > >> > > > > > >    selection.
> > > > >> > > > > > >> > > > > > >    - The default implementation is
> > log-only.
> > > > >> > > > > > >> > > > > > >    - The durable implementation writes
> > > through
> > > > >> > > > > > >> > > PolarisMetricsManager
> > > > >> > > > > > >> > > > > and
> > > > >> > > > > > >> > > > > > >    MetricsPersistence.
> > > > >> > > > > > >> > > > > > >    - MetricsPersistence is currently
> > > inherited by
> > > > >> > > > > > >> > BasePersistence.
> > > > >> > > > > > >> > > > > > >
> > > > >> > > > > > >> > > > > > > My proposal:
> > > > >> > > > > > >> > > > > > >
> > > > >> > > > > > >> > > > > > >    - Keep the reporting/emitting boundary
> > as
> > > the
> > > > >> > SPI.
> > > > >> > > > > > >> > > > > > >    - Revise that contract in a
> > > framework-agnostic
> > > > >> > > optional
> > > > >> > > > > > >> > Iceberg
> > > > >> > > > > > >> > > > > > *metrics
> > > > >> > > > > > >> > > > > > >    extension API layer*.
> > > > >> > > > > > >> > > > > > >    - Keep runtime/service responsible for
> > > REST
> > > > >> > > ingestion,
> > > > >> > > > > > >> > > table/authz
> > > > >> > > > > > >> > > > > > >    validation, and runtime wiring.
> > > > >> > > > > > >> > > > > > >    - Keep the battery implementation
> > > log-only in
> > > > >> the
> > > > >> > > > > metrics
> > > > >> > > > > > >> > > > extension
> > > > >> > > > > > >> > > > > > API
> > > > >> > > > > > >> > > > > > >    layer, *not under runtime/service*.
> > > > >> > > > > > >> > > > > > >    - Treat durable JDBC metrics as one
> > > > >> > implementation
> > > > >> > > of
> > > > >> > > > > the
> > > > >> > > > > > >> > > > reporting
> > > > >> > > > > > >> > > > > > SPI.
> > > > >> > > > > > >> > > > > > >    - Decompose the MetricsPersistence and
> > > > >> > > BasePersistence
> > > > >> > > > > > >> > coupling
> > > > >> > > > > > >> > > as
> > > > >> > > > > > >> > > > > > >    durable implementation detail.
> > > > >> > > > > > >> > > > > > >
> > > > >> > > > > > >> > > > > > > So the difference is not “new reporter
> > > concept vs
> > > > >> > old
> > > > >> > > > > > reporter
> > > > >> > > > > > >> > > > > concept.”
> > > > >> > > > > > >> > > > > > > but:
> > > > >> > > > > > >> > > > > > >
> > > > >> > > > > > >> > > > > > >
> > > > >> > > > > > >> > > > > > >    - PolarisMetricsReporter today =
> > > > >> runtime/service
> > > > >> > > > > > CDI-shaped
> > > > >> > > > > > >> > > hook.
> > > > >> > > > > > >> > > > > > >    - Target reporting SPI = same
> conceptual
> > > > >> > boundary,
> > > > >> > > but
> > > > >> > > > > > >> placed
> > > > >> > > > > > >> > in
> > > > >> > > > > > >> > > > the
> > > > >> > > > > > >> > > > > > >    right extension API layer and kept
> > > > >> > > framework-agnostic.
> > > > >> > > > > > >> > > > > > >
> > > > >> > > > > > >> > > > > > > For the current metrics PR, I think the
> > > minimal
> > > > >> > > > > > architectural
> > > > >> > > > > > >> > > cleanup
> > > > >> > > > > > >> > > > > > > before merge is to *put the reporting SPI
> > > > >> contract
> > > > >> > > and the
> > > > >> > > > > > >> > battery
> > > > >> > > > > > >> > > > > > default
> > > > >> > > > > > >> > > > > > > in the right place*. The durable JDBC
> > > > >> > implementation,
> > > > >> > > > > > >> > event-backed
> > > > >> > > > > > >> > > > > async
> > > > >> > > > > > >> > > > > > > implementation, external queue support,
> and
> > > > >> deeper
> > > > >> > > > > > persistence
> > > > >> > > > > > >> > > > cleanup
> > > > >> > > > > > >> > > > > > can
> > > > >> > > > > > >> > > > > > > continue as follow-ups, as long as we do
> > not
> > > lock
> > > > >> > the
> > > > >> > > SPI
> > > > >> > > > > > >> > boundary
> > > > >> > > > > > >> > > to
> > > > >> > > > > > >> > > > > > > BasePersistence or to runtime/service
> > wiring.
> > > > >> > > > > > >> > > > > > >
> > > > >> > > > > > >> > > > > > > -ej
> > > > >> > > > > > >> > > > > > >
> > > > >> > > > > > >> > > > > > > On Wed, Jun 3, 2026 at 3:00 PM Dmitri
> > > > >> Bourlatchkov <
> > > > >> > > > > > >> > > dimas@apache.org
> > > > >> > > > > > >> > > > >
> > > > >> > > > > > >> > > > > > > wrote:
> > > > >> > > > > > >> > > > > > >
> > > > >> > > > > > >> > > > > > > > Hi EJ,
> > > > >> > > > > > >> > > > > > > >
> > > > >> > > > > > >> > > > > > > > Thanks for the recap / summary!
> > > > >> > > > > > >> > > > > > > >
> > > > >> > > > > > >> > > > > > > > > Do folks agree that the stable SPI
> > > boundary
> > > > >> > > should be
> > > > >> > > > > > >> > > > > > > > metrics reporting/emitting, not metrics
> > > > >> > persistence?
> > > > >> > > > > > >> > > > > > > >
> > > > >> > > > > > >> > > > > > > > SGTM.
> > > > >> > > > > > >> > > > > > > >
> > > > >> > > > > > >> > > > > > > > > Does an optional Iceberg metrics
> > > extension
> > > > >> API
> > > > >> > > layer
> > > > >> > > > > > sound
> > > > >> > > > > > >> > like
> > > > >> > > > > > >> > > > the
> > > > >> > > > > > >> > > > > > > right
> > > > >> > > > > > >> > > > > > > > home for this SPI?
> > > > >> > > > > > >> > > > > > > >
> > > > >> > > > > > >> > > > > > > > How is that different from current
> > > > >> > > > > PolarisMetricsReporter?
> > > > >> > > > > > >> > > > > > > >
> > > > >> > > > > > >> > > > > > > > > Should the current durable metrics
> work
> > > be
> > > > >> > > reframed
> > > > >> > > > > as a
> > > > >> > > > > > >> > > durable
> > > > >> > > > > > >> > > > > > > > JDBC reference implementation of that
> > SPI?
> > > > >> > > > > > >> > > > > > > >
> > > > >> > > > > > >> > > > > > > > SGTM.
> > > > >> > > > > > >> > > > > > > >
> > > > >> > > > > > >> > > > > > > > > What is the smallest PR sequence to
> get
> > > there
> > > > >> > > without
> > > > >> > > > > > >> > blocking
> > > > >> > > > > > >> > > > > > > > the current metrics work unnecessarily?
> > > > >> > > > > > >> > > > > > > >
> > > > >> > > > > > >> > > > > > > > Let's get
> > > > >> > >
> > > > >> >
> > > > >>
> > >
> >
> https://urldefense.com/v3/__https://github.com/apache/polaris/pull/4397__;!!Iz9xO38YGHZK!4rnlVIqBQQQ6xpGe1PMV58edOS_cGrKoqdfhw4qSp_JQCavvEjaxXk0OwGPnxyEcO60AzKQK_XDtp7H3DdkFXcrXuu5jefU$
> > > > >> > > > > > >> merged
> > > > >> > > > > > >> > > > first.
> > > > >> > > > > > >> > > > > > > >
> > > > >> > > > > > >> > > > > > > > Then, I'd propose isolating the Metrics
> > > schema
> > > > >> > from
> > > > >> > > the
> > > > >> > > > > > >> > MetaStore
> > > > >> > > > > > >> > > > > > schema.
> > > > >> > > > > > >> > > > > > > > This will probably have some ripple
> > effect
> > > into
> > > > >> > the
> > > > >> > > > > > >> bootstrap
> > > > >> > > > > > >> > > > > > workflows,
> > > > >> > > > > > >> > > > > > > so
> > > > >> > > > > > >> > > > > > > > it's not a trivial change.
> > > > >> > > > > > >> > > > > > > >
> > > > >> > > > > > >> > > > > > > > Then, let's reassess.
> > > > >> > > > > > >> > > > > > > >
> > > > >> > > > > > >> > > > > > > > Cheers,
> > > > >> > > > > > >> > > > > > > > Dmitri.
> > > > >> > > > > > >> > > > > > > >
> > > > >> > > > > > >> > > > > > > > On Wed, Jun 3, 2026 at 5:40 PM EJ Wang
> <
> > > > >> > > > > > >> > > > > ej.wang.dev.interest@gmail.com
> > > > >> > > > > > >> > > > > > >
> > > > >> > > > > > >> > > > > > > > wrote:
> > > > >> > > > > > >> > > > > > > >
> > > > >> > > > > > >> > > > > > > > > Hi folks,
> > > > >> > > > > > >> > > > > > > > >
> > > > >> > > > > > >> > > > > > > > > Thanks again for the discussion
> today.
> > I
> > > > >> updated
> > > > >> > > the
> > > > >> > > > > > sync
> > > > >> > > > > > >> doc
> > > > >> > > > > > >> > > > with
> > > > >> > > > > > >> > > > > > > notes
> > > > >> > > > > > >> > > > > > > > > from the third metrics architecture
> > sync:
> > > > >> > > > > > >> > > > > > > > >
> > > > >> > > > > > >> > > > > > > > >
> > > > >> > > > > > >> > > > > > > > >
> > > > >> > > > > > >> > > > > > > >
> > > > >> > > > > > >> > > > > > >
> > > > >> > > > > > >> > > > > >
> > > > >> > > > > > >> > > > >
> > > > >> > > > > > >> > > >
> > > > >> > > > > > >> > >
> > > > >> > > > > > >> >
> > > > >> > > > > > >>
> > > > >> > > > > >
> > > > >> > > > >
> > > > >> > >
> > > > >> >
> > > > >>
> > >
> >
> https://urldefense.com/v3/__https://docs.google.com/document/d/100h7c4damrUzVuquYbBHM0EvA4LSWuW2IT2dN_7nYVA/edit?tab=t.k96s2xyqr5u1__;!!Iz9xO38YGHZK!4rnlVIqBQQQ6xpGe1PMV58edOS_cGrKoqdfhw4qSp_JQCavvEjaxXk0OwGPnxyEcO60AzKQK_XDtp7H3DdkFXcrXrqCL-QE$
> > > > >> > > > > > >> > > > > > > > >
> > > > >> > > > > > >> > > > > > > > > A few highlights from today’s
> > discussion:
> > > > >> > > > > > >> > > > > > > > >
> > > > >> > > > > > >> > > > > > > > >    - We clarified that Iceberg
> metrics
> > > > >> reporting
> > > > >> > > can
> > > > >> > > > > be
> > > > >> > > > > > >> > > > interpreted
> > > > >> > > > > > >> > > > > > > > either
> > > > >> > > > > > >> > > > > > > > >    as sync or async handling from
> > > Polaris’s
> > > > >> > > > > perspective,
> > > > >> > > > > > >> so
> > > > >> > > > > > >> > > > Polaris
> > > > >> > > > > > >> > > > > > > > should
> > > > >> > > > > > >> > > > > > > > >    stay flexible as a platform
> instead
> > of
> > > > >> baking
> > > > >> > > one
> > > > >> > > > > > >> handling
> > > > >> > > > > > >> > > > model
> > > > >> > > > > > >> > > > > > > into
> > > > >> > > > > > >> > > > > > > > > the
> > > > >> > > > > > >> > > > > > > > >    REST API behavior.
> > > > >> > > > > > >> > > > > > > > >    - We aligned that the built-in
> > > (battery)
> > > > >> > > behavior
> > > > >> > > > > for
> > > > >> > > > > > >> > > metrics
> > > > >> > > > > > >> > > > > > > emitting
> > > > >> > > > > > >> > > > > > > > >    can stay simple: no-op/log-only is
> > > enough
> > > > >> as
> > > > >> > > the
> > > > >> > > > > > >> default.
> > > > >> > > > > > >> > > > > > > > >    - Durable metrics persistence
> should
> > > be
> > > > >> > > treated as
> > > > >> > > > > an
> > > > >> > > > > > >> > > > > > implementation
> > > > >> > > > > > >> > > > > > > > of
> > > > >> > > > > > >> > > > > > > > >    the metrics reporting path, not as
> > the
> > > > >> core
> > > > >> > SPI
> > > > >> > > > > > >> boundary.
> > > > >> > > > > > >> > > > > > > > >    - The existing durable metrics
> work
> > > can be
> > > > >> > > reviewed
> > > > >> > > > > > as
> > > > >> > > > > > >> a
> > > > >> > > > > > >> > > > > reference
> > > > >> > > > > > >> > > > > > > > >    implementation of the reporting
> SPI,
> > > with
> > > > >> > > > > > >> > > persistence-related
> > > > >> > > > > > >> > > > > > logic
> > > > >> > > > > > >> > > > > > > > kept
> > > > >> > > > > > >> > > > > > > > >    self-contained under a metrics
> > durable
> > > > >> > > > > implementation
> > > > >> > > > > > >> > module
> > > > >> > > > > > >> > > > > > rather
> > > > >> > > > > > >> > > > > > > > than
> > > > >> > > > > > >> > > > > > > > >    scattered through core entity
> > > persistence.
> > > > >> > > > > > >> > > > > > > > >    - Dashboard/insights remains a
> real
> > > use
> > > > >> case,
> > > > >> > > but
> > > > >> > > > > we
> > > > >> > > > > > >> > agreed
> > > > >> > > > > > >> > > to
> > > > >> > > > > > >> > > > > > keep
> > > > >> > > > > > >> > > > > > > it
> > > > >> > > > > > >> > > > > > > > >    separate from the core metrics
> > intake
> > > > >> > > discussion
> > > > >> > > > > for
> > > > >> > > > > > >> now.
> > > > >> > > > > > >> > > > > > > > >
> > > > >> > > > > > >> > > > > > > > > I also did a quick source check after
> > the
> > > > >> > meeting
> > > > >> > > to
> > > > >> > > > > > make
> > > > >> > > > > > >> > sure
> > > > >> > > > > > >> > > we
> > > > >> > > > > > >> > > > > are
> > > > >> > > > > > >> > > > > > > > > describing the current state
> > accurately.
> > > > >> > > > > > >> > > > > > > > >
> > > > >> > > > > > >> > > > > > > > > Current state:
> > > > >> > > > > > >> > > > > > > > >
> > > > >> > > > > > >> > > > > > > > >    - Polaris already has a metrics
> > > reporting
> > > > >> > hook:
> > > > >> > > > > > >> > > > > > > > PolarisMetricsReporter.
> > > > >> > > > > > >> > > > > > > > >    - The default implementation is
> > > > >> > > > > > DefaultMetricsReporter,
> > > > >> > > > > > >> > > > selected
> > > > >> > > > > > >> > > > > > by
> > > > >> > > > > > >> > > > > > > > >
> > > > >> > > polaris.iceberg-metrics.reporting.type=default. It
> > > > >> > > > > is
> > > > >> > > > > > >> > > > log-only,
> > > > >> > > > > > >> > > > > > > > >    effectively quiet unless metrics
> > > logging
> > > > >> is
> > > > >> > > > > enabled.
> > > > >> > > > > > >> > > > > > > > >    - There is also a
> > > > >> PersistingMetricsReporter,
> > > > >> > > > > selected
> > > > >> > > > > > >> by
> > > > >> > > > > > >> > > > > > > > >
> > > > >> > > polaris.iceberg-metrics.reporting.type=persisting,
> > > > >> > > > > > >> which
> > > > >> > > > > > >> > > > > converts
> > > > >> > > > > > >> > > > > > > > >    Iceberg scan/commit reports into
> > > Polaris
> > > > >> > > metrics
> > > > >> > > > > > >> records
> > > > >> > > > > > >> > and
> > > > >> > > > > > >> > > > > > writes
> > > > >> > > > > > >> > > > > > > > > through
> > > > >> > > > > > >> > > > > > > > >    PolarisMetricsManager ->
> > > > >> MetricsPersistence.
> > > > >> > > > > > >> > > > > > > > >    - MetricsPersistence currently
> > exists
> > > in
> > > > >> the
> > > > >> > > > > > >> persistence
> > > > >> > > > > > >> > > layer
> > > > >> > > > > > >> > > > > and
> > > > >> > > > > > >> > > > > > > is
> > > > >> > > > > > >> > > > > > > > >    inherited by BasePersistence.
> > > > >> > > > > > >> > > > > > > > >
> > > > >> > > > > > >> > > > > > > > > My read from the discussion is that
> the
> > > > >> target
> > > > >> > > > > boundary
> > > > >> > > > > > >> > should
> > > > >> > > > > > >> > > > not
> > > > >> > > > > > >> > > > > be
> > > > >> > > > > > >> > > > > > > > > MetricsPersistence as inherited by
> > > > >> > > BasePersistence.
> > > > >> > > > > That
> > > > >> > > > > > >> path
> > > > >> > > > > > >> > > > > should
> > > > >> > > > > > >> > > > > > be
> > > > >> > > > > > >> > > > > > > > > decomposed as durable implementation
> > > detail
> > > > >> > (taken
> > > > >> > > > > care
> > > > >> > > > > > >> of by
> > > > >> > > > > > >> > > > > > > > >
> > > > >> >
> > > > >>
> > >
> >
> https://urldefense.com/v3/__https://github.com/apache/polaris/pull/4397__;!!Iz9xO38YGHZK!4rnlVIqBQQQ6xpGe1PMV58edOS_cGrKoqdfhw4qSp_JQCavvEjaxXk0OwGPnxyEcO60AzKQK_XDtp7H3DdkFXcrXuu5jefU$
> > > > >> ).
> > > > >> > The
> > > > >> > > > > > stable
> > > > >> > > > > > >> > > > extension
> > > > >> > > > > > >> > > > > > > point
> > > > >> > > > > > >> > > > > > > > > should instead be the metrics
> > > > >> reporting/emitting
> > > > >> > > > > > boundary.
> > > > >> > > > > > >> > > > > > > > >
> > > > >> > > > > > >> > > > > > > > > Concretely, I think the next proposal
> > > should
> > > > >> be
> > > > >> > > shaped
> > > > >> > > > > > >> like
> > > > >> > > > > > >> > > this:
> > > > >> > > > > > >> > > > > > > > >
> > > > >> > > > > > >> > > > > > > > >    1.
> > > > >> > > > > > >> > > > > > > > >
> > > > >> > > > > > >> > > > > > > > >    Define the proper metrics
> reporting
> > > SPI
> > > > >> at an
> > > > >> > > > > > optional
> > > > >> > > > > > >> > > Iceberg
> > > > >> > > > > > >> > > > > > > metrics
> > > > >> > > > > > >> > > > > > > > >    extension API layer.
> > > > >> > > > > > >> > > > > > > > >
> > > > >> > > > > > >> > > > > > > > >    Example direction:
> > > > >> > > > > > >> > > > > > > > >
> > > > >> > > > > > >> > > > > > > > >    public interface
> > > IcebergMetricsReporter {
> > > > >> > > > > > >> > > > > > > > >      void
> > > > >> > reportMetric(IcebergMetricsReportContext
> > > > >> > > > > > >> context,
> > > > >> > > > > > >> > > > > > > > > MetricsReport report);
> > > > >> > > > > > >> > > > > > > > >    }
> > > > >> > > > > > >> > > > > > > > >
> > > > >> > > > > > >> > > > > > > > >    The context would carry the small
> > > > >> > > Polaris-resolved
> > > > >> > > > > > >> > envelope:
> > > > >> > > > > > >> > > > > > > > >    catalog/table identity, report
> type,
> > > > >> received
> > > > >> > > > > > >> timestamp,
> > > > >> > > > > > >> > and
> > > > >> > > > > > >> > > > > > > > > request/trace
> > > > >> > > > > > >> > > > > > > > >    context if available. The raw
> > Iceberg
> > > > >> > > MetricsReport
> > > > >> > > > > > >> > remains
> > > > >> > > > > > >> > > > the
> > > > >> > > > > > >> > > > > > > > payload.
> > > > >> > > > > > >> > > > > > > > >    2.
> > > > >> > > > > > >> > > > > > > > >
> > > > >> > > > > > >> > > > > > > > >    Keep runtime/service as the REST
> > > ingestion
> > > > >> > and
> > > > >> > > > > wiring
> > > > >> > > > > > >> > layer.
> > > > >> > > > > > >> > > > > > > > >
> > > > >> > > > > > >> > > > > > > > >    The REST handler still resolves
> the
> > > table
> > > > >> and
> > > > >> > > > > > performs
> > > > >> > > > > > >> > authz
> > > > >> > > > > > >> > > > > > before
> > > > >> > > > > > >> > > > > > > > >    accepting the report. After that,
> it
> > > calls
> > > > >> > the
> > > > >> > > > > > selected
> > > > >> > > > > > >> > > > > reporting
> > > > >> > > > > > >> > > > > > > > >    implementation.
> > > > >> > > > > > >> > > > > > > > >    3.
> > > > >> > > > > > >> > > > > > > > >
> > > > >> > > > > > >> > > > > > > > >    Keep the battery default
> > > no-op/log-only.
> > > > >> > > > > > >> > > > > > > > >
> > > > >> > > > > > >> > > > > > > > >    This preserves an out-of-box safe
> > > default
> > > > >> and
> > > > >> > > > > avoids
> > > > >> > > > > > >> > > > requiring a
> > > > >> > > > > > >> > > > > > > > durable
> > > > >> > > > > > >> > > > > > > > >    metrics store for every Polaris
> > > > >> deployment.
> > > > >> > > > > > >> > > > > > > > >    4.
> > > > >> > > > > > >> > > > > > > > >
> > > > >> > > > > > >> > > > > > > > >    Move durable JDBC metrics into a
> > > > >> > self-contained
> > > > >> > > > > > >> > > implementation
> > > > >> > > > > > >> > > > > of
> > > > >> > > > > > >> > > > > > > the
> > > > >> > > > > > >> > > > > > > > >    reporting SPI.
> > > > >> > > > > > >> > > > > > > > >
> > > > >> > > > > > >> > > > > > > > >    That implementation can own its
> > > schema,
> > > > >> > > bootstrap,
> > > > >> > > > > > >> > > retention,
> > > > >> > > > > > >> > > > > and
> > > > >> > > > > > >> > > > > > > read
> > > > >> > > > > > >> > > > > > > > >    API support. It should not define
> > the
> > > core
> > > > >> > > > > reporting
> > > > >> > > > > > >> SPI
> > > > >> > > > > > >> > > > > boundary,
> > > > >> > > > > > >> > > > > > > and
> > > > >> > > > > > >> > > > > > > > > it
> > > > >> > > > > > >> > > > > > > > >    should not require metrics
> > > persistence to
> > > > >> > > remain
> > > > >> > > > > > >> inherited
> > > > >> > > > > > >> > > > from
> > > > >> > > > > > >> > > > > > > > >    BasePersistence.
> > > > >> > > > > > >> > > > > > > > >    5.
> > > > >> > > > > > >> > > > > > > > >
> > > > >> > > > > > >> > > > > > > > >    Treat async/event-backed handling
> as
> > > > >> another
> > > > >> > > > > > >> > implementation
> > > > >> > > > > > >> > > of
> > > > >> > > > > > >> > > > > the
> > > > >> > > > > > >> > > > > > > > same
> > > > >> > > > > > >> > > > > > > > >    reporting SPI.
> > > > >> > > > > > >> > > > > > > > >
> > > > >> > > > > > >> > > > > > > > >    For example, an event-backed
> > reporter
> > > > >> could
> > > > >> > > > > enqueue a
> > > > >> > > > > > >> > > metrics
> > > > >> > > > > > >> > > > > > event
> > > > >> > > > > > >> > > > > > > > and
> > > > >> > > > > > >> > > > > > > > >    let listeners handle durable
> storage
> > > or
> > > > >> other
> > > > >> > > > > sinks.
> > > > >> > > > > > >> If we
> > > > >> > > > > > >> > > > later
> > > > >> > > > > > >> > > > > > > need
> > > > >> > > > > > >> > > > > > > > a
> > > > >> > > > > > >> > > > > > > > >    replaceable queue engine, that
> seems
> > > like
> > > > >> a
> > > > >> > > shared
> > > > >> > > > > > >> > > > event/metrics
> > > > >> > > > > > >> > > > > > > > > substrate
> > > > >> > > > > > >> > > > > > > > >    topic rather than a metrics-only
> > > > >> requirement.
> > > > >> > > > > > >> > > > > > > > >
> > > > >> > > > > > >> > > > > > > > > This framing lets us keep the REST
> > > metrics
> > > > >> > > endpoint
> > > > >> > > > > > >> simple,
> > > > >> > > > > > >> > > > > preserve
> > > > >> > > > > > >> > > > > > > the
> > > > >> > > > > > >> > > > > > > > > current default behavior, support
> > durable
> > > > >> > metrics
> > > > >> > > > > users,
> > > > >> > > > > > >> and
> > > > >> > > > > > >> > > > still
> > > > >> > > > > > >> > > > > > > leave
> > > > >> > > > > > >> > > > > > > > > room for async/event-backed or
> > > > >> > > external-queue-based
> > > > >> > > > > > >> > > > > implementations.
> > > > >> > > > > > >> > > > > > > > >
> > > > >> > > > > > >> > > > > > > > > I think the main follow-up questions
> > are:
> > > > >> > > > > > >> > > > > > > > >
> > > > >> > > > > > >> > > > > > > > >    - Do folks agree that the stable
> SPI
> > > > >> boundary
> > > > >> > > > > should
> > > > >> > > > > > be
> > > > >> > > > > > >> > > > metrics
> > > > >> > > > > > >> > > > > > > > >    reporting/emitting, not metrics
> > > > >> persistence?
> > > > >> > > > > > >> > > > > > > > >    - Does an optional Iceberg metrics
> > > > >> extension
> > > > >> > > API
> > > > >> > > > > > layer
> > > > >> > > > > > >> > sound
> > > > >> > > > > > >> > > > > like
> > > > >> > > > > > >> > > > > > > the
> > > > >> > > > > > >> > > > > > > > >    right home for this SPI?
> > > > >> > > > > > >> > > > > > > > >    - Should the current durable
> metrics
> > > work
> > > > >> be
> > > > >> > > > > reframed
> > > > >> > > > > > >> as a
> > > > >> > > > > > >> > > > > durable
> > > > >> > > > > > >> > > > > > > > JDBC
> > > > >> > > > > > >> > > > > > > > >    reference implementation of that
> > SPI?
> > > > >> > > > > > >> > > > > > > > >    - What is the smallest PR sequence
> > to
> > > get
> > > > >> > there
> > > > >> > > > > > without
> > > > >> > > > > > >> > > > blocking
> > > > >> > > > > > >> > > > > > the
> > > > >> > > > > > >> > > > > > > > >    current metrics work
> unnecessarily?
> > > > >> > > > > > >> > > > > > > > >
> > > > >> > > > > > >> > > > > > > > > Thanks,
> > > > >> > > > > > >> > > > > > > > > -ej
> > > > >> > > > > > >> > > > > > > > >
> > > > >> > > > > > >> > > > > > > > > On Fri, May 15, 2026 at 5:16 PM
> Dmitri
> > > > >> > > Bourlatchkov <
> > > > >> > > > > > >> > > > > > dimas@apache.org>
> > > > >> > > > > > >> > > > > > > > > wrote:
> > > > >> > > > > > >> > > > > > > > >
> > > > >> > > > > > >> > > > > > > > > > Hi JB,
> > > > >> > > > > > >> > > > > > > > > >
> > > > >> > > > > > >> > > > > > > > > > Could you set up another meeting,
> > > please?
> > > > >> Same
> > > > >> > > time
> > > > >> > > > > on
> > > > >> > > > > > >> > > > Wednesday
> > > > >> > > > > > >> > > > > as
> > > > >> > > > > > >> > > > > > > > last
> > > > >> > > > > > >> > > > > > > > > > time... I hope it works for
> everyone.
> > > > >> > > > > > >> > > > > > > > > >
> > > > >> > > > > > >> > > > > > > > > > Cheers,
> > > > >> > > > > > >> > > > > > > > > > Dmitri.
> > > > >> > > > > > >> > > > > > > > > >
> > > > >> > > > > > >> > > > > > > > > > On Fri, May 15, 2026 at 8:06 PM
> Yufei
> > > Gu <
> > > > >> > > > > > >> > > flyrain000@gmail.com
> > > > >> > > > > > >> > > > >
> > > > >> > > > > > >> > > > > > > wrote:
> > > > >> > > > > > >> > > > > > > > > >
> > > > >> > > > > > >> > > > > > > > > > > +1 on another sync call next
> week.
> > > > >> > > > > > >> > > > > > > > > > >
> > > > >> > > > > > >> > > > > > > > > > > Yufei
> > > > >> > > > > > >> > > > > > > > > > >
> > > > >> > > > > > >> > > > > > > > > > >
> > > > >> > > > > > >> > > > > > > > > > > On Fri, May 15, 2026 at 4:52 PM
> > > Dmitri
> > > > >> > > > > Bourlatchkov
> > > > >> > > > > > <
> > > > >> > > > > > >> > > > > > > > dimas@apache.org>
> > > > >> > > > > > >> > > > > > > > > > > wrote:
> > > > >> > > > > > >> > > > > > > > > > >
> > > > >> > > > > > >> > > > > > > > > > > > Hi All,
> > > > >> > > > > > >> > > > > > > > > > > >
> > > > >> > > > > > >> > > > > > > > > > > > WDYT about another sync call
> next
> > > week?
> > > > >> > > > > > >> > > > > > > > > > > >
> > > > >> > > > > > >> > > > > > > > > > > > Thanks,
> > > > >> > > > > > >> > > > > > > > > > > > Dmitri.
> > > > >> > > > > > >> > > > > > > > > > > >
> > > > >> > > > > > >> > > > > > > > > > > > On Wed, May 6, 2026 at 5:29 PM
> > > Dmitri
> > > > >> > > > > > Bourlatchkov <
> > > > >> > > > > > >> > > > > > > > dimas@apache.org
> > > > >> > > > > > >> > > > > > > > > >
> > > > >> > > > > > >> > > > > > > > > > > > wrote:
> > > > >> > > > > > >> > > > > > > > > > > >
> > > > >> > > > > > >> > > > > > > > > > > > > Hi EJ,
> > > > >> > > > > > >> > > > > > > > > > > > >
> > > > >> > > > > > >> > > > > > > > > > > > > Thanks for the summary! It
> > covers
> > > > >> what
> > > > >> > we
> > > > >> > > > > > >> discussed
> > > > >> > > > > > >> > in
> > > > >> > > > > > >> > > > the
> > > > >> > > > > > >> > > > > > > > meeting
> > > > >> > > > > > >> > > > > > > > > > very
> > > > >> > > > > > >> > > > > > > > > > > > > well, IMHO.
> > > > >> > > > > > >> > > > > > > > > > > > >
> > > > >> > > > > > >> > > > > > > > > > > > > Looking forward to concrete
> PRs
> > > :)
> > > > >> > > > > > >> > > > > > > > > > > > >
> > > > >> > > > > > >> > > > > > > > > > > > > Cheers,
> > > > >> > > > > > >> > > > > > > > > > > > > Dmitri.
> > > > >> > > > > > >> > > > > > > > > > > > >
> > > > >> > > > > > >> > > > > > > > > > > > > On Wed, May 6, 2026 at
> 5:08 PM
> > EJ
> > > > >> Wang <
> > > > >> > > > > > >> > > > > > > > > > ej.wang.dev.interest@gmail.com
> > > > >> > > > > > >> > > > > > > > > > > >
> > > > >> > > > > > >> > > > > > > > > > > > > wrote:
> > > > >> > > > > > >> > > > > > > > > > > > >
> > > > >> > > > > > >> > > > > > > > > > > > >> Hi folks,
> > > > >> > > > > > >> > > > > > > > > > > > >>
> > > > >> > > > > > >> > > > > > > > > > > > >> We had a community sync
> > earlier,
> > > > >> thanks
> > > > >> > > JB
> > > > >> > > > > for
> > > > >> > > > > > >> > > > scheduling
> > > > >> > > > > > >> > > > > > it.
> > > > >> > > > > > >> > > > > > > > > Notes
> > > > >> > > > > > >> > > > > > > > > > > from
> > > > >> > > > > > >> > > > > > > > > > > > >> the first metrics
> architecture
> > > sync
> > > > >> > (May
> > > > >> > > 6,
> > > > >> > > > > > >> 10-11am
> > > > >> > > > > > >> > > PT).
> > > > >> > > > > > >> > > > > > > > > Discussion
> > > > >> > > > > > >> > > > > > > > > > > doc
> > > > >> > > > > > >> > > > > > > > > > > > >> with per-section status:
> > > > >> > > > > > >> > > > > > > > > > > > >>
> > > > >> > > > > > >> > > > > > > > > > > > >>
> > > > >> > > > > > >> > > > > > > > > > > >
> > > > >> > > > > > >> > > > > > > > > > >
> > > > >> > > > > > >> > > > > > > > > >
> > > > >> > > > > > >> > > > > > > > >
> > > > >> > > > > > >> > > > > > > >
> > > > >> > > > > > >> > > > > > >
> > > > >> > > > > > >> > > > > >
> > > > >> > > > > > >> > > > >
> > > > >> > > > > > >> > > >
> > > > >> > > > > > >> > >
> > > > >> > > > > > >> >
> > > > >> > > > > > >>
> > > > >> > > > > >
> > > > >> > > > >
> > > > >> > >
> > > > >> >
> > > > >>
> > >
> >
> https://urldefense.com/v3/__https://docs.google.com/document/d/100h7c4damrUzVuquYbBHM0EvA4LSWuW2IT2dN_7nYVA/edit?tab=t.0__;!!Iz9xO38YGHZK!4rnlVIqBQQQ6xpGe1PMV58edOS_cGrKoqdfhw4qSp_JQCavvEjaxXk0OwGPnxyEcO60AzKQK_XDtp7H3DdkFXcrXgIddeUs$
> > > > >> > > > > > >> > > > > > > > > > > > >>
> > > > >> > > > > > >> > > > > > > > > > > > >> *The meeting covered both
> > topics
> > > > >> from
> > > > >> > the
> > > > >> > > > > doc.
> > > > >> > > > > > >> > > > > > Direction-level
> > > > >> > > > > > >> > > > > > > > > > > alignment
> > > > >> > > > > > >> > > > > > > > > > > > >> was reached on the headline
> > > pieces;
> > > > >> > > details
> > > > >> > > > > > >> remain
> > > > >> > > > > > >> > for
> > > > >> > > > > > >> > > > PR
> > > > >> > > > > > >> > > > > > > review
> > > > >> > > > > > >> > > > > > > > > or
> > > > >> > > > > > >> > > > > > > > > > > > >> follow-up sessions.*
> > > > >> > > > > > >> > > > > > > > > > > > >>
> > > > >> > > > > > >> > > > > > > > > > > > >> *Topic 1 — Persistence
> schema
> > > > >> redesign*
> > > > >> > > > > > >> > > > > > > > > > > > >> Idea-level alignment on
> > > > >> consolidating
> > > > >> > > > > per-type
> > > > >> > > > > > >> > tables
> > > > >> > > > > > >> > > > > > > > > > > > >> (scan_metrics_report,
> > > > >> > > > > > >> > > > > > > > > > > > >> commit_metrics_report) into
> a
> > > single
> > > > >> > > > > > >> metrics_report
> > > > >> > > > > > >> > > > table.
> > > > >> > > > > > >> > > > > > The
> > > > >> > > > > > >> > > > > > > > > > > > motivating
> > > > >> > > > > > >> > > > > > > > > > > > >> cost is the surface area
> added
> > > by
> > > > >> every
> > > > >> > > new
> > > > >> > > > > > >> metric
> > > > >> > > > > > >> > > type
> > > > >> > > > > > >> > > > > > today:
> > > > >> > > > > > >> > > > > > > > new
> > > > >> > > > > > >> > > > > > > > > > > > table,
> > > > >> > > > > > >> > > > > > > > > > > > >> SPI method, record class,
> > model,
> > > > >> > > converter,
> > > > >> > > > > > >> schema
> > > > >> > > > > > >> > > > > > migration.
> > > > >> > > > > > >> > > > > > > > > > > > >>
> > > > >> > > > > > >> > > > > > > > > > > > >> Most schema details are
> > > deferred to
> > > > >> the
> > > > >> > > > > schema
> > > > >> > > > > > >> PR. A
> > > > >> > > > > > >> > > few
> > > > >> > > > > > >> > > > > > > > specific
> > > > >> > > > > > >> > > > > > > > > > > points
> > > > >> > > > > > >> > > > > > > > > > > > >> came up:
> > > > >> > > > > > >> > > > > > > > > > > > >> •   metric_schema_version:
> > Yufei
> > > > >> > prefers
> > > > >> > > > > > dropping
> > > > >> > > > > > >> > it,
> > > > >> > > > > > >> > > > > since
> > > > >> > > > > > >> > > > > > > > there
> > > > >> > > > > > >> > > > > > > > > is
> > > > >> > > > > > >> > > > > > > > > > > no
> > > > >> > > > > > >> > > > > > > > > > > > >> spec-level concept of
> metrics
> > > > >> > versioning
> > > > >> > > > > today
> > > > >> > > > > > >> and
> > > > >> > > > > > >> > it
> > > > >> > > > > > >> > > is
> > > > >> > > > > > >> > > > > > hard
> > > > >> > > > > > >> > > > > > > to
> > > > >> > > > > > >> > > > > > > > > > > define
> > > > >> > > > > > >> > > > > > > > > > > > >> unilaterally. Robert prefers
> > > keeping
> > > > >> > it,
> > > > >> > > > > given
> > > > >> > > > > > >> IRC
> > > > >> > > > > > >> > v2
> > > > >> > > > > > >> > > is
> > > > >> > > > > > >> > > > > > > coming
> > > > >> > > > > > >> > > > > > > > > and
> > > > >> > > > > > >> > > > > > > > > > > the
> > > > >> > > > > > >> > > > > > > > > > > > >> schema should be considered
> > > against
> > > > >> its
> > > > >> > > > > likely
> > > > >> > > > > > >> > shape;
> > > > >> > > > > > >> > > > > Robert
> > > > >> > > > > > >> > > > > > > > also
> > > > >> > > > > > >> > > > > > > > > > > raised
> > > > >> > > > > > >> > > > > > > > > > > > >> how to differentiate various
> > > payload
> > > > >> > > formats
> > > > >> > > > > if
> > > > >> > > > > > >> any.
> > > > >> > > > > > >> > > > EJ's
> > > > >> > > > > > >> > > > > > read
> > > > >> > > > > > >> > > > > > > > is
> > > > >> > > > > > >> > > > > > > > > > that
> > > > >> > > > > > >> > > > > > > > > > > > >> this
> > > > >> > > > > > >> > > > > > > > > > > > >> is a two-way-door decision.
> We
> > > can
> > > > >> > start
> > > > >> > > > > > without
> > > > >> > > > > > >> the
> > > > >> > > > > > >> > > > > field,
> > > > >> > > > > > >> > > > > > > and
> > > > >> > > > > > >> > > > > > > > if
> > > > >> > > > > > >> > > > > > > > > > IRC
> > > > >> > > > > > >> > > > > > > > > > > > v2
> > > > >> > > > > > >> > > > > > > > > > > > >> changes the shape we would
> > > likely
> > > > >> roll
> > > > >> > a
> > > > >> > > > > > >> > corresponding
> > > > >> > > > > > >> > > > new
> > > > >> > > > > > >> > > > > > > > schema
> > > > >> > > > > > >> > > > > > > > > > > > anyway,
> > > > >> > > > > > >> > > > > > > > > > > > >> which is not particularly
> > > costly.
> > > > >> > > > > > >> > > > > > > > > > > > >> •   Payload format: Robert
> > > pointed
> > > > >> out
> > > > >> > > that
> > > > >> > > > > > >> future
> > > > >> > > > > > >> > > > formats
> > > > >> > > > > > >> > > > > > > > beyond
> > > > >> > > > > > >> > > > > > > > > > JSON
> > > > >> > > > > > >> > > > > > > > > > > > may
> > > > >> > > > > > >> > > > > > > > > > > > >> be worth supporting. The
> exact
> > > > >> shape is
> > > > >> > > > > > deferred
> > > > >> > > > > > >> to
> > > > >> > > > > > >> > > the
> > > > >> > > > > > >> > > > > > schema
> > > > >> > > > > > >> > > > > > > > > > > > discussion.
> > > > >> > > > > > >> > > > > > > > > > > > >> •   Partition strategy:
> Anand
> > > > >> suggested
> > > > >> > > > > monthly
> > > > >> > > > > > >> > > > > partitioning
> > > > >> > > > > > >> > > > > > > > based
> > > > >> > > > > > >> > > > > > > > > > on
> > > > >> > > > > > >> > > > > > > > > > > > his
> > > > >> > > > > > >> > > > > > > > > > > > >> experience as potentially
> > > helpful at
> > > > >> > > scale.
> > > > >> > > > > > >> > > > > > > > > > > > >>
> > > > >> > > > > > >> > > > > > > > > > > > >> *Topic 2 — Where metrics
> > > ingestion
> > > > >> and
> > > > >> > > > > storage
> > > > >> > > > > > >> > belong*
> > > > >> > > > > > >> > > > > > > > > > > > >> Idea-level alignment that
> > > metrics
> > > > >> > should
> > > > >> > > be a
> > > > >> > > > > > >> > > separated
> > > > >> > > > > > >> > > > > SPI
> > > > >> > > > > > >> > > > > > > from
> > > > >> > > > > > >> > > > > > > > > the
> > > > >> > > > > > >> > > > > > > > > > > > >> entity
> > > > >> > > > > > >> > > > > > > > > > > > >> persistence stack. Two
> reasons
> > > > >> > surfaced:
> > > > >> > > (a)
> > > > >> > > > > > >> > workloads
> > > > >> > > > > > >> > > > and
> > > > >> > > > > > >> > > > > > > > > > capability
> > > > >> > > > > > >> > > > > > > > > > > > >> requirements diverge enough
> > that
> > > > >> > coupling
> > > > >> > > > > them
> > > > >> > > > > > >> > creates
> > > > >> > > > > > >> > > > > > > > artificial
> > > > >> > > > > > >> > > > > > > > > > > > >> constraints, and (b) admin
> > > > >> experience
> > > > >> > > > > improves
> > > > >> > > > > > >> when
> > > > >> > > > > > >> > > > > metrics
> > > > >> > > > > > >> > > > > > > has
> > > > >> > > > > > >> > > > > > > > > its
> > > > >> > > > > > >> > > > > > > > > > > own
> > > > >> > > > > > >> > > > > > > > > > > > >> bootstrap, retention, and
> > > lifecycle.
> > > > >> > > Dmitri
> > > > >> > > > > > noted
> > > > >> > > > > > >> > > > Polaris
> > > > >> > > > > > >> > > > > > > being
> > > > >> > > > > > >> > > > > > > > a
> > > > >> > > > > > >> > > > > > > > > > > > platform
> > > > >> > > > > > >> > > > > > > > > > > > >> should have the flexibility
> to
> > > > >> support
> > > > >> > > > > > different
> > > > >> > > > > > >> > > > > persistence
> > > > >> > > > > > >> > > > > > > > > > backends
> > > > >> > > > > > >> > > > > > > > > > > > per
> > > > >> > > > > > >> > > > > > > > > > > > >> concern, and pointed to a
> > > concrete
> > > > >> next
> > > > >> > > step
> > > > >> > > > > of
> > > > >> > > > > > >> > > > separating
> > > > >> > > > > > >> > > > > > the
> > > > >> > > > > > >> > > > > > > > > JDBC
> > > > >> > > > > > >> > > > > > > > > > > > >> bootstrap for metrics from
> the
> > > > >> > metastore
> > > > >> > > > > > >> bootstrap.
> > > > >> > > > > > >> > > > Robert
> > > > >> > > > > > >> > > > > > > > > proposed
> > > > >> > > > > > >> > > > > > > > > > an
> > > > >> > > > > > >> > > > > > > > > > > > >> additional UX extension:
> > detect
> > > an
> > > > >> > > > > > unbootstrapped
> > > > >> > > > > > >> > > > metrics
> > > > >> > > > > > >> > > > > > > store
> > > > >> > > > > > >> > > > > > > > on
> > > > >> > > > > > >> > > > > > > > > > > first
> > > > >> > > > > > >> > > > > > > > > > > > >> use and auto-bootstrap
> rather
> > > than
> > > > >> > > requiring
> > > > >> > > > > an
> > > > >> > > > > > >> > > explicit
> > > > >> > > > > > >> > > > > > > manual
> > > > >> > > > > > >> > > > > > > > > > > > bootstrap
> > > > >> > > > > > >> > > > > > > > > > > > >> step.
> > > > >> > > > > > >> > > > > > > > > > > > >> The meeting also confirmed
> > that
> > > > >> Polaris
> > > > >> > > > > metrics
> > > > >> > > > > > >> can
> > > > >> > > > > > >> > > > start
> > > > >> > > > > > >> > > > > > > small
> > > > >> > > > > > >> > > > > > > > > and
> > > > >> > > > > > >> > > > > > > > > > > stay
> > > > >> > > > > > >> > > > > > > > > > > > >> Iceberg-focused. Naming and
> > > > >> persistence
> > > > >> > > > > schema
> > > > >> > > > > > >> can
> > > > >> > > > > > >> > > lean
> > > > >> > > > > > >> > > > > > > > > > > > Iceberg-specific.
> > > > >> > > > > > >> > > > > > > > > > > > >> If a future expansion to
> > > > >> generic-table
> > > > >> > > > > metrics
> > > > >> > > > > > or
> > > > >> > > > > > >> > > > > > operational
> > > > >> > > > > > >> > > > > > > > > > metrics
> > > > >> > > > > > >> > > > > > > > > > > > >> arrives, an abstraction
> layer
> > > can be
> > > > >> > > built on
> > > > >> > > > > > >> top of
> > > > >> > > > > > >> > > the
> > > > >> > > > > > >> > > > > > > Iceberg
> > > > >> > > > > > >> > > > > > > > > > > metrics
> > > > >> > > > > > >> > > > > > > > > > > > >> reporter at that point.
> Robert
> > > > >> remains
> > > > >> > > on the
> > > > >> > > > > > >> fence
> > > > >> > > > > > >> > > and
> > > > >> > > > > > >> > > > > > would
> > > > >> > > > > > >> > > > > > > > > prefer
> > > > >> > > > > > >> > > > > > > > > > > > >> something more generic but
> did
> > > not
> > > > >> > block
> > > > >> > > the
> > > > >> > > > > > >> > > direction;
> > > > >> > > > > > >> > > > > > > Dmitri's
> > > > >> > > > > > >> > > > > > > > > > read
> > > > >> > > > > > >> > > > > > > > > > > > was
> > > > >> > > > > > >> > > > > > > > > > > > >> that the proposed framework
> > > already
> > > > >> has
> > > > >> > > > > enough
> > > > >> > > > > > >> > > > flexibility
> > > > >> > > > > > >> > > > > > to
> > > > >> > > > > > >> > > > > > > > > absorb
> > > > >> > > > > > >> > > > > > > > > > > > >> future
> > > > >> > > > > > >> > > > > > > > > > > > >> expansion.
> > > > >> > > > > > >> > > > > > > > > > > > >>
> > > > >> > > > > > >> > > > > > > > > > > > >> The Trade-offs and Proposed
> > > > >> structure
> > > > >> > > > > sections
> > > > >> > > > > > in
> > > > >> > > > > > >> > the
> > > > >> > > > > > >> > > > doc
> > > > >> > > > > > >> > > > > > were
> > > > >> > > > > > >> > > > > > > > not
> > > > >> > > > > > >> > > > > > > > > > > > >> reviewed
> > > > >> > > > > > >> > > > > > > > > > > > >> in detail. They remain open
> > for
> > > > >> either
> > > > >> > > the
> > > > >> > > > > next
> > > > >> > > > > > >> sync
> > > > >> > > > > > >> > > or
> > > > >> > > > > > >> > > > PR
> > > > >> > > > > > >> > > > > > > > review.
> > > > >> > > > > > >> > > > > > > > > > > > >>
> > > > >> > > > > > >> > > > > > > > > > > > >> *Cross-cutting alignment —
> > > > >> > > battery-included
> > > > >> > > > > > plus
> > > > >> > > > > > >> > > > > pluggable*
> > > > >> > > > > > >> > > > > > > > > > > > >> A common philosophy emerged
> > > from the
> > > > >> > > > > > discussion.
> > > > >> > > > > > >> EJ
> > > > >> > > > > > >> > > > > > summarized
> > > > >> > > > > > >> > > > > > > > it
> > > > >> > > > > > >> > > > > > > > > > as:
> > > > >> > > > > > >> > > > > > > > > > > > >> Polaris should provide a
> > > > >> > > battery-included UX
> > > > >> > > > > > for
> > > > >> > > > > > >> > > > beginners
> > > > >> > > > > > >> > > > > > and
> > > > >> > > > > > >> > > > > > > > the
> > > > >> > > > > > >> > > > > > > > > > > > >> flexibility for advanced
> users
> > > to
> > > > >> swap
> > > > >> > > the
> > > > >> > > > > > >> included
> > > > >> > > > > > >> > > > > battery
> > > > >> > > > > > >> > > > > > > for
> > > > >> > > > > > >> > > > > > > > > > > > something
> > > > >> > > > > > >> > > > > > > > > > > > >> more powerful or tailored to
> > > their
> > > > >> use
> > > > >> > > case.
> > > > >> > > > > > The
> > > > >> > > > > > >> SPI
> > > > >> > > > > > >> > > > > design
> > > > >> > > > > > >> > > > > > > > needs
> > > > >> > > > > > >> > > > > > > > > to
> > > > >> > > > > > >> > > > > > > > > > > > >> enable
> > > > >> > > > > > >> > > > > > > > > > > > >> both.
> > > > >> > > > > > >> > > > > > > > > > > > >>
> > > > >> > > > > > >> > > > > > > > > > > > >> The inputs that shaped this
> > > framing:
> > > > >> > > > > > >> > > > > > > > > > > > >> •   Anand described how his
> > team
> > > > >> uses
> > > > >> > the
> > > > >> > > > > > current
> > > > >> > > > > > >> > > > metrics
> > > > >> > > > > > >> > > > > > > > > > persistence
> > > > >> > > > > > >> > > > > > > > > > > > >> (three metrics consumers in
> > > v1.4).
> > > > >> > > > > > >> > > > > > > > > > > > >> •   Yufei raised Grafana and
> > > > >> dashboard
> > > > >> > > > > > >> integrations
> > > > >> > > > > > >> > > as a
> > > > >> > > > > > >> > > > > > > > > destination
> > > > >> > > > > > >> > > > > > > > > > > use
> > > > >> > > > > > >> > > > > > > > > > > > >> case beyond the default.
> > > > >> > > > > > >> > > > > > > > > > > > >> •   Robert called out that
> the
> > > > >> current
> > > > >> > > design
> > > > >> > > > > > is
> > > > >> > > > > > >> > more
> > > > >> > > > > > >> > > > > > > > > JDBC-focused.
> > > > >> > > > > > >> > > > > > > > > > > > >>
> > > > >> > > > > > >> > > > > > > > > > > > >> Two concrete instances:
> > > > >> > > > > > >> > > > > > > > > > > > >> •   Async metrics intake:
> > > Yufei's
> > > > >> > initial
> > > > >> > > > > > >> position
> > > > >> > > > > > >> > was
> > > > >> > > > > > >> > > > > that
> > > > >> > > > > > >> > > > > > > > async
> > > > >> > > > > > >> > > > > > > > > > > should
> > > > >> > > > > > >> > > > > > > > > > > > >> largely live on the producer
> > > side
> > > > >> and
> > > > >> > > there
> > > > >> > > > > is
> > > > >> > > > > > >> not
> > > > >> > > > > > >> > > much
> > > > >> > > > > > >> > > > > > > Polaris
> > > > >> > > > > > >> > > > > > > > > can
> > > > >> > > > > > >> > > > > > > > > > > do.
> > > > >> > > > > > >> > > > > > > > > > > > >> Robert suggested a
> > Polaris-side
> > > > >> default
> > > > >> > > is
> > > > >> > > > > > doable
> > > > >> > > > > > >> > via
> > > > >> > > > > > >> > > > > > Vert.x.
> > > > >> > > > > > >> > > > > > > > > Dmitri
> > > > >> > > > > > >> > > > > > > > > > > > >> agreed
> > > > >> > > > > > >> > > > > > > > > > > > >> the direction is worth
> > > exploring.
> > > > >> The
> > > > >> > > meeting
> > > > >> > > > > > >> > > converged
> > > > >> > > > > > >> > > > > on a
> > > > >> > > > > > >> > > > > > > > > > > > >> battery-included default
> > (likely
> > > > >> > > > > Vert.x-backed)
> > > > >> > > > > > >> with
> > > > >> > > > > > >> > > an
> > > > >> > > > > > >> > > > > SPI
> > > > >> > > > > > >> > > > > > > > shape
> > > > >> > > > > > >> > > > > > > > > > that
> > > > >> > > > > > >> > > > > > > > > > > > >> lets
> > > > >> > > > > > >> > > > > > > > > > > > >> power users route to a more
> > > scalable
> > > > >> > > backend
> > > > >> > > > > > >> > > (k8s-hosted
> > > > >> > > > > > >> > > > > > > queue,
> > > > >> > > > > > >> > > > > > > > > AWS
> > > > >> > > > > > >> > > > > > > > > > > SQS,
> > > > >> > > > > > >> > > > > > > > > > > > >> etc.).
> > > > >> > > > > > >> > > > > > > > > > > > >> •   Pluggable destinations:
> > > > >> combining
> > > > >> > > Yufei's
> > > > >> > > > > > >> > > dashboard
> > > > >> > > > > > >> > > > > use
> > > > >> > > > > > >> > > > > > > case
> > > > >> > > > > > >> > > > > > > > > > with
> > > > >> > > > > > >> > > > > > > > > > > > >> Robert's JDBC-focused
> > call-out,
> > > the
> > > > >> > > meeting
> > > > >> > > > > > >> agreed
> > > > >> > > > > > >> > the
> > > > >> > > > > > >> > > > SPI
> > > > >> > > > > > >> > > > > > > > should
> > > > >> > > > > > >> > > > > > > > > be
> > > > >> > > > > > >> > > > > > > > > > > > >> structured for multiple
> sinks
> > so
> > > > >> > > integrations
> > > > >> > > > > > >> become
> > > > >> > > > > > >> > > > impl
> > > > >> > > > > > >> > > > > > > > choices
> > > > >> > > > > > >> > > > > > > > > > > rather
> > > > >> > > > > > >> > > > > > > > > > > > >> than architectural changes.
> > > > >> > > > > > >> > > > > > > > > > > > >>
> > > > >> > > > > > >> > > > > > > > > > > > >> The battery-included default
> > is
> > > most
> > > > >> > > likely
> > > > >> > > > > to
> > > > >> > > > > > >> use
> > > > >> > > > > > >> > the
> > > > >> > > > > > >> > > > > > > existing
> > > > >> > > > > > >> > > > > > > > > > > > >> JDBC-backed
> > > > >> > > > > > >> > > > > > > > > > > > >> approach.
> > > > >> > > > > > >> > > > > > > > > > > > >>
> > > > >> > > > > > >> > > > > > > > > > > > >> *Direction (idea-level
> > > alignment)*
> > > > >> > > > > > >> > > > > > > > > > > > >> •   Single metrics_report
> > table
> > > > >> > > consolidating
> > > > >> > > > > > >> > per-type
> > > > >> > > > > > >> > > > > > > metrics,
> > > > >> > > > > > >> > > > > > > > > > > > replacing
> > > > >> > > > > > >> > > > > > > > > > > > >> scan_metrics_report and
> > > > >> > > commit_metrics_report
> > > > >> > > > > > >> > > > > > > > > > > > >> •   Iceberg-focused naming
> and
> > > > >> schema
> > > > >> > for
> > > > >> > > > > now,
> > > > >> > > > > > >> > revisit
> > > > >> > > > > > >> > > > if
> > > > >> > > > > > >> > > > > > > > > > > generic-table
> > > > >> > > > > > >> > > > > > > > > > > > or
> > > > >> > > > > > >> > > > > > > > > > > > >> operational metrics arrive
> > > > >> > > > > > >> > > > > > > > > > > > >> •   Metrics persistence as a
> > > > >> separated
> > > > >> > > SPI,
> > > > >> > > > > not
> > > > >> > > > > > >> on
> > > > >> > > > > > >> > > > > > > > BasePersistence
> > > > >> > > > > > >> > > > > > > > > > > > >> •   Bootstrap path separated
> > for
> > > > >> > metrics,
> > > > >> > > > > > >> > independent
> > > > >> > > > > > >> > > of
> > > > >> > > > > > >> > > > > > > > metastore
> > > > >> > > > > > >> > > > > > > > > > > > >> bootstrap
> > > > >> > > > > > >> > > > > > > > > > > > >> •   "Battery-included plus
> > > > >> pluggable"
> > > > >> > as
> > > > >> > > the
> > > > >> > > > > > SPI
> > > > >> > > > > > >> > > design
> > > > >> > > > > > >> > > > > > > > philosophy
> > > > >> > > > > > >> > > > > > > > > > > > >>
> > > > >> > > > > > >> > > > > > > > > > > > >> *Open items*
> > > > >> > > > > > >> > > > > > > > > > > > >> •   Schema details:
> > > > >> > > metric_schema_version,
> > > > >> > > > > > >> payload
> > > > >> > > > > > >> > > > format,
> > > > >> > > > > > >> > > > > > IRC
> > > > >> > > > > > >> > > > > > > > v2
> > > > >> > > > > > >> > > > > > > > > > > > >> forward-compat shape
> > > > >> > > > > > >> > > > > > > > > > > > >> •   SPI design details —
> full
> > > review
> > > > >> > > either
> > > > >> > > > > in
> > > > >> > > > > > >> the
> > > > >> > > > > > >> > > next
> > > > >> > > > > > >> > > > > sync
> > > > >> > > > > > >> > > > > > > or
> > > > >> > > > > > >> > > > > > > > in
> > > > >> > > > > > >> > > > > > > > > > the
> > > > >> > > > > > >> > > > > > > > > > > > >> corresponding PR
> > > > >> > > > > > >> > > > > > > > > > > > >> •   Schema refactor PR
> > ownership
> > > > >> > > > > > >> > > > > > > > > > > > >>
> > > > >> > > > > > >> > > > > > > > > > > > >> *Action items*
> > > > >> > > > > > >> > > > > > > > > > > > >> •   EJ to take a first stab
> at
> > > the
> > > > >> SPI
> > > > >> > > design
> > > > >> > > > > > and
> > > > >> > > > > > >> > > > > > potentially
> > > > >> > > > > > >> > > > > > > > > > partner
> > > > >> > > > > > >> > > > > > > > > > > > with
> > > > >> > > > > > >> > > > > > > > > > > > >> Anand to incorporate the
> > lessons
> > > > >> > learned
> > > > >> > > from
> > > > >> > > > > > the
> > > > >> > > > > > >> > > > existing
> > > > >> > > > > > >> > > > > > > > > reporter
> > > > >> > > > > > >> > > > > > > > > > > and
> > > > >> > > > > > >> > > > > > > > > > > > >> persistence work.
> > > > >> > > > > > >> > > > > > > > > > > > >> •   Schema refactor PR
> > > ownership is
> > > > >> not
> > > > >> > > yet
> > > > >> > > > > > >> decided.
> > > > >> > > > > > >> > > If
> > > > >> > > > > > >> > > > > > anyone
> > > > >> > > > > > >> > > > > > > > is
> > > > >> > > > > > >> > > > > > > > > > > > >> interested in driving it,
> > reply
> > > on
> > > > >> this
> > > > >> > > > > thread.
> > > > >> > > > > > >> > > > > > > > > > > > >> •   JB to schedule the next
> > > sync,
> > > > >> > > tentatively
> > > > >> > > > > > in
> > > > >> > > > > > >> two
> > > > >> > > > > > >> > > > > weeks.
> > > > >> > > > > > >> > > > > > > > > > > > >>
> > > > >> > > > > > >> > > > > > > > > > > > >> -ej
> > > > >> > > > > > >> > > > > > > > > > > > >>
> > > > >> > > > > > >> > > > > > > > > > > > >> On Mon, Apr 27, 2026 at
> > 3:07 PM
> > > EJ
> > > > >> > Wang <
> > > > >> > > > > > >> > > > > > > > > > > ej.wang.dev.interest@gmail.com
> > > > >> > > > > > >> > > > > > > > > > > > >
> > > > >> > > > > > >> > > > > > > > > > > > >> wrote:
> > > > >> > > > > > >> > > > > > > > > > > > >>
> > > > >> > > > > > >> > > > > > > > > > > > >> > Thanks Yufei for the +1.
> > > > >> > > > > > >> > > > > > > > > > > > >> >
> > > > >> > > > > > >> > > > > > > > > > > > >> > JB, could you help add a
> > > biweekly
> > > > >> > > metrics
> > > > >> > > > > > >> > > architecture
> > > > >> > > > > > >> > > > > > sync
> > > > >> > > > > > >> > > > > > > to
> > > > >> > > > > > >> > > > > > > > > the
> > > > >> > > > > > >> > > > > > > > > > > > >> Polaris
> > > > >> > > > > > >> > > > > > > > > > > > >> > community calendar? I'm
> > > thinking
> > > > >> > > Thursdays
> > > > >> > > > > at
> > > > >> > > > > > >> > 9-10am
> > > > >> > > > > > >> > > > PT,
> > > > >> > > > > > >> > > > > > on
> > > > >> > > > > > >> > > > > > > > the
> > > > >> > > > > > >> > > > > > > > > > > > >> off-weeks
> > > > >> > > > > > >> > > > > > > > > > > > >> > from the community meeting
> > > > >> (starting
> > > > >> > > May
> > > > >> > > > > 7),
> > > > >> > > > > > 60
> > > > >> > > > > > >> > > > minutes.
> > > > >> > > > > > >> > > > > > > > > > > > >> >
> > > > >> > > > > > >> > > > > > > > > > > > >> > Here's a rough agenda to
> > work
> > > > >> through
> > > > >> > > over
> > > > >> > > > > > the
> > > > >> > > > > > >> > first
> > > > >> > > > > > >> > > > few
> > > > >> > > > > > >> > > > > > > > > sessions,
> > > > >> > > > > > >> > > > > > > > > > > > >> grouped
> > > > >> > > > > > >> > > > > > > > > > > > >> > by priority:
> > > > >> > > > > > >> > > > > > > > > > > > >> >
> > > > >> > > > > > >> > > > > > > > > > > > >> > *First: foundational
> > > direction*
> > > > >> > > > > > >> > > > > > > > > > > > >> >
> > > > >> > > > > > >> > > > > > > > > > > > >> > 1.  MetricsPersistence:
> > public
> > > > >> SPI or
> > > > >> > > > > > internal
> > > > >> > > > > > >> > > > > > > implementation
> > > > >> > > > > > >> > > > > > > > > > > detail?
> > > > >> > > > > > >> > > > > > > > > > > > >> >    •   Marked @Beta,
> javadoc
> > > calls
> > > > >> > it a
> > > > >> > > > > > >> "Service
> > > > >> > > > > > >> > > > > Provider
> > > > >> > > > > > >> > > > > > > > > > > Interface",
> > > > >> > > > > > >> > > > > > > > > > > > >> but
> > > > >> > > > > > >> > > > > > > > > > > > >> > only one consumer
> > > > >> > > > > (JdbcBasePersistenceImpl),
> > > > >> > > > > > >> lives
> > > > >> > > > > > >> > > on
> > > > >> > > > > > >> > > > > > > > > > > BasePersistence.
> > > > >> > > > > > >> > > > > > > > > > > > >> If
> > > > >> > > > > > >> > > > > > > > > > > > >> > demoted to a private
> helper
> > > > >> inside a
> > > > >> > > > > > persisting
> > > > >> > > > > > >> > > > reporter
> > > > >> > > > > > >> > > > > > > impl,
> > > > >> > > > > > >> > > > > > > > > > most
> > > > >> > > > > > >> > > > > > > > > > > > >> > downstream design
> decisions
> > > become
> > > > >> > > > > > >> implementation
> > > > >> > > > > > >> > > > > details
> > > > >> > > > > > >> > > > > > > > rather
> > > > >> > > > > > >> > > > > > > > > > > than
> > > > >> > > > > > >> > > > > > > > > > > > >> > contract questions.
> > > > >> > > > > > >> > > > > > > > > > > > >> >
> > > > >> > > > > > >> > > > > > > > > > > > >> > 2.  Persistence schema
> > > redesign
> > > > >> > > > > > >> > > > > > > > > > > > >> >    •   Current two-table
> > > layout
> > > > >> > > > > > >> > > (scan_metrics_report,
> > > > >> > > > > > >> > > > > > > > > > > > >> > commit_metrics_report)
> with
> > > ~25
> > > > >> > > flattened
> > > > >> > > > > > >> columns
> > > > >> > > > > > >> > > > each.
> > > > >> > > > > > >> > > > > > > Every
> > > > >> > > > > > >> > > > > > > > > new
> > > > >> > > > > > >> > > > > > > > > > > > metric
> > > > >> > > > > > >> > > > > > > > > > > > >> > type requires a new table,
> > SPI
> > > > >> > method,
> > > > >> > > > > record
> > > > >> > > > > > >> > class,
> > > > >> > > > > > >> > > > > > model,
> > > > >> > > > > > >> > > > > > > > > > > converter,
> > > > >> > > > > > >> > > > > > > > > > > > >> and
> > > > >> > > > > > >> > > > > > > > > > > > >> > schema migration.
> Direction
> > to
> > > > >> > explore:
> > > > >> > > > > > single
> > > > >> > > > > > >> > table
> > > > >> > > > > > >> > > > > with
> > > > >> > > > > > >> > > > > > > > > > > metric_type
> > > > >> > > > > > >> > > > > > > > > > > > >> enum,
> > > > >> > > > > > >> > > > > > > > > > > > >> > schema_version, and JSON
> > > payload
> > > > >> > > column.
> > > > >> > > > > > >> > > > > > > > > > > > >> >
> > > > >> > > > > > >> > > > > > > > > > > > >> > *Second: design details
> once
> > > > >> > direction
> > > > >> > > is
> > > > >> > > > > > set*
> > > > >> > > > > > >> > > > > > > > > > > > >> >
> > > > >> > > > > > >> > > > > > > > > > > > >> > 3.  Partition key strategy
> > > > >> > > > > > >> > > > > > > > > > > > >> >    •   Single-table design
> > > means
> > > > >> scan
> > > > >> > > > > metrics
> > > > >> > > > > > >> at
> > > > >> > > > > > >> > > scale
> > > > >> > > > > > >> > > > > > will
> > > > >> > > > > > >> > > > > > > > have
> > > > >> > > > > > >> > > > > > > > > > > high
> > > > >> > > > > > >> > > > > > > > > > > > >> > write concurrency per
> table.
> > > > >> Schema
> > > > >> > > needs
> > > > >> > > > > to
> > > > >> > > > > > >> > expose
> > > > >> > > > > > >> > > > > enough
> > > > >> > > > > > >> > > > > > > > > > structure
> > > > >> > > > > > >> > > > > > > > > > > > for
> > > > >> > > > > > >> > > > > > > > > > > > >> > backends to shard by
> entity
> > or
> > > > >> time
> > > > >> > > range.
> > > > >> > > > > > >> > > > > > > > > > > > >> >
> > > > >> > > > > > >> > > > > > > > > > > > >> > 4.  Read/write path
> > > consistency
> > > > >> > > > > > >> > > > > > > > > > > > >> >    •   Writes go through
> > > > >> > > > > > PolarisMetricsManager
> > > > >> > > > > > >> on
> > > > >> > > > > > >> > > > > > > > > > MetaStoreManager.
> > > > >> > > > > > >> > > > > > > > > > > > >> Reads
> > > > >> > > > > > >> > > > > > > > > > > > >> > bypass MetaStoreManager
> and
> > go
> > > > >> > > straight to
> > > > >> > > > > > >> > > > > > BasePersistence,
> > > > >> > > > > > >> > > > > > > > > > > excluding
> > > > >> > > > > > >> > > > > > > > > > > > >> > non-JDBC backends from the
> > > read
> > > > >> API.
> > > > >> > > > > > >> > > > > > > > > > > > >> >
> > > > >> > > > > > >> > > > > > > > > > > > >> > *Third: cleanup and
> > alignment*
> > > > >> > > > > > >> > > > > > > > > > > > >> >
> > > > >> > > > > > >> > > > > > > > > > > > >> > 5.  PolarisMetricsReporter
> > > naming
> > > > >> > > > > > >> > > > > > > > > > > > >> >    •   Only handles IRC
> > > > >> > > > > > >> (ScanReport/CommitReport),
> > > > >> > > > > > >> > > > > doesn't
> > > > >> > > > > > >> > > > > > > > cover
> > > > >> > > > > > >> > > > > > > > > > > > generic
> > > > >> > > > > > >> > > > > > > > > > > > >> > tables or operational
> > metrics.
> > > > >> Name
> > > > >> > is
> > > > >> > > > > > broader
> > > > >> > > > > > >> > than
> > > > >> > > > > > >> > > > > scope.
> > > > >> > > > > > >> > > > > > > > > > > > >> >
> > > > >> > > > > > >> > > > > > > > > > > > >> > 6.  PolarisMetricsManager
> > > facade
> > > > >> > > > > passthrough
> > > > >> > > > > > >> > > > > > > > > > > > >> >    •   Entire default
> method
> > > is
> > > > >> > > > > > >> > > > > > > > > > > > >>
> > > > >> > callCtx.getMetaStore().writeScanReport().
> > > > >> > > > > > >> > > > > > > > > > > > >> > Zero logic, passes Level 1
> > > > >> straight
> > > > >> > > through
> > > > >> > > > > > to
> > > > >> > > > > > >> > Level
> > > > >> > > > > > >> > > > 3.
> > > > >> > > > > > >> > > > > > Same
> > > > >> > > > > > >> > > > > > > > > > > > >> anti-pattern
> > > > >> > > > > > >> > > > > > > > > > > > >> > as PolarisEventManager.
> > > > >> > > > > > >> > > > > > > > > > > > >> >
> > > > >> > > > > > >> > > > > > > > > > > > >> > 7.  Iceberg community
> > > alignment
> > > > >> > > > > > >> > > > > > > > > > > > >> >    •   Payload-type
> > extension
> > > > >> needs
> > > > >> > > > > > discussion
> > > > >> > > > > > >> on
> > > > >> > > > > > >> > > > > > > dev@iceberg.
> > > > >> > > > > > >> > > > > > > > > > > > >> obelix74's
> > > > >> > > > > > >> > > > > > > > > > > > >> > Feb thread got zero
> replies.
> > > > >> Needs a
> > > > >> > > > > > committer
> > > > >> > > > > > >> > > voice.
> > > > >> > > > > > >> > > > > > > > > > > > >> >
> > > > >> > > > > > >> > > > > > > > > > > > >> > Lets confirm
> prioritization
> > > in the
> > > > >> > > first
> > > > >> > > > > > >> session.
> > > > >> > > > > > >> > > > > > > > > > > > >> >
> > > > >> > > > > > >> > > > > > > > > > > > >> > -ej
> > > > >> > > > > > >> > > > > > > > > > > > >> >
> > > > >> > > > > > >> > > > > > > > > > > > >> > On Tue, Apr 21, 2026 at
> > > 3:18 PM
> > > > >> Yufei
> > > > >> > > Gu <
> > > > >> > > > > > >> > > > > > > > flyrain000@gmail.com>
> > > > >> > > > > > >> > > > > > > > > > > > wrote:
> > > > >> > > > > > >> > > > > > > > > > > > >> >
> > > > >> > > > > > >> > > > > > > > > > > > >> >> Thanks everyone for
> > > continuing to
> > > > >> > > drive
> > > > >> > > > > this
> > > > >> > > > > > >> > > > forward. I
> > > > >> > > > > > >> > > > > > > agree
> > > > >> > > > > > >> > > > > > > > > > that
> > > > >> > > > > > >> > > > > > > > > > > > the
> > > > >> > > > > > >> > > > > > > > > > > > >> >> problem is getting
> complex
> > > enough
> > > > >> > > that a
> > > > >> > > > > > more
> > > > >> > > > > > >> > > > > structured
> > > > >> > > > > > >> > > > > > > > > > discussion
> > > > >> > > > > > >> > > > > > > > > > > > >> would
> > > > >> > > > > > >> > > > > > > > > > > > >> >> help.
> > > > >> > > > > > >> > > > > > > > > > > > >> >>
> > > > >> > > > > > >> > > > > > > > > > > > >> >> +1 on setting up a
> biweekly
> > > sync
> > > > >> for
> > > > >> > > the
> > > > >> > > > > > >> metrics
> > > > >> > > > > > >> > > > > > > > architecture.
> > > > >> > > > > > >> > > > > > > > > > I’m
> > > > >> > > > > > >> > > > > > > > > > > > >> happy
> > > > >> > > > > > >> > > > > > > > > > > > >> >> to
> > > > >> > > > > > >> > > > > > > > > > > > >> >> join.
> > > > >> > > > > > >> > > > > > > > > > > > >> >>
> > > > >> > > > > > >> > > > > > > > > > > > >> >> Yufei
> > > > >> > > > > > >> > > > > > > > > > > > >> >>
> > > > >> > > > > > >> > > > > > > > > > > > >> >>
> > > > >> > > > > > >> > > > > > > > > > > > >> >> On Tue, Apr 21, 2026 at
> > > 2:34 PM
> > > > >> EJ
> > > > >> > > Wang <
> > > > >> > > > > > >> > > > > > > > > > > > >>
> > ej.wang.dev.interest@gmail.com>
> > > > >> > > > > > >> > > > > > > > > > > > >> >> wrote:
> > > > >> > > > > > >> > > > > > > > > > > > >> >>
> > > > >> > > > > > >> > > > > > > > > > > > >> >> > Also, I've been looking
> > > more
> > > > >> > > closely at
> > > > >> > > > > > the
> > > > >> > > > > > >> > > > > > *persistence
> > > > >> > > > > > >> > > > > > > > > schema
> > > > >> > > > > > >> > > > > > > > > > > in
> > > > >> > > > > > >> > > > > > > > > > > > >> the
> > > > >> > > > > > >> > > > > > > > > > > > >> >> > current metrics work*,
> > and
> > > I
> > > > >> think
> > > > >> > > > > > there's a
> > > > >> > > > > > >> > > > > structural
> > > > >> > > > > > >> > > > > > > > > > rigidity
> > > > >> > > > > > >> > > > > > > > > > > > >> problem
> > > > >> > > > > > >> > > > > > > > > > > > >> >> > worth raising before
> the
> > > shape
> > > > >> > gets
> > > > >> > > > > locked
> > > > >> > > > > > >> in.
> > > > >> > > > > > >> > > > > > > > > > > > >> >> >
> > > > >> > > > > > >> > > > > > > > > > > > >> >> > Right now we have two
> > > separate
> > > > >> > > tables
> > > > >> > > > > > >> > > > > > > (scan_metrics_report
> > > > >> > > > > > >> > > > > > > > > and
> > > > >> > > > > > >> > > > > > > > > > > > >> >> > commit_metrics_report),
> > > each
> > > > >> with
> > > > >> > > ~25
> > > > >> > > > > > >> flattened
> > > > >> > > > > > >> > > > > columns
> > > > >> > > > > > >> > > > > > > > that
> > > > >> > > > > > >> > > > > > > > > > > > directly
> > > > >> > > > > > >> > > > > > > > > > > > >> >> > mirror the Iceberg
> report
> > > > >> fields.
> > > > >> > > The
> > > > >> > > > > SPI
> > > > >> > > > > > >> > follows
> > > > >> > > > > > >> > > > the
> > > > >> > > > > > >> > > > > > > same
> > > > >> > > > > > >> > > > > > > > > > split:
> > > > >> > > > > > >> > > > > > > > > > > > >> >> > writeScanReport and
> > > > >> > > writeCommitReport as
> > > > >> > > > > > >> > separate
> > > > >> > > > > > >> > > > > > > methods,
> > > > >> > > > > > >> > > > > > > > > with
> > > > >> > > > > > >> > > > > > > > > > > > >> per-type
> > > > >> > > > > > >> > > > > > > > > > > > >> >> > record classes,
> > > converters, and
> > > > >> > > model
> > > > >> > > > > > >> objects.
> > > > >> > > > > > >> > > *The
> > > > >> > > > > > >> > > > > > > > practical
> > > > >> > > > > > >> > > > > > > > > > > cost:
> > > > >> > > > > > >> > > > > > > > > > > > >> >> > adding a new metric
> type
> > > > >> > > (operational
> > > > >> > > > > > >> metrics,
> > > > >> > > > > > >> > > for
> > > > >> > > > > > >> > > > > > > example)
> > > > >> > > > > > >> > > > > > > > > > > > requires
> > > > >> > > > > > >> > > > > > > > > > > > >> a
> > > > >> > > > > > >> > > > > > > > > > > > >> >> new
> > > > >> > > > > > >> > > > > > > > > > > > >> >> > table, a new SPI
> method,
> > a
> > > new
> > > > >> > > record
> > > > >> > > > > > >> class, a
> > > > >> > > > > > >> > > new
> > > > >> > > > > > >> > > > > > model
> > > > >> > > > > > >> > > > > > > > > > class, a
> > > > >> > > > > > >> > > > > > > > > > > > new
> > > > >> > > > > > >> > > > > > > > > > > > >> >> > converter branch, and a
> > > schema
> > > > >> > > > > migration*.
> > > > >> > > > > > >> > > That's a
> > > > >> > > > > > >> > > > > lot
> > > > >> > > > > > >> > > > > > > of
> > > > >> > > > > > >> > > > > > > > > > > surface
> > > > >> > > > > > >> > > > > > > > > > > > >> area
> > > > >> > > > > > >> > > > > > > > > > > > >> >> > for what should be "one
> > > more
> > > > >> kind
> > > > >> > of
> > > > >> > > > > > >> metric."
> > > > >> > > > > > >> > > > > > > > > > > > >> >> >
> > > > >> > > > > > >> > > > > > > > > > > > >> >> > *My bias* would be
> > toward a
> > > > >> single
> > > > >> > > > > metrics
> > > > >> > > > > > >> > table
> > > > >> > > > > > >> > > > with
> > > > >> > > > > > >> > > > > > *a
> > > > >> > > > > > >> > > > > > > > > typed
> > > > >> > > > > > >> > > > > > > > > > > JSON
> > > > >> > > > > > >> > > > > > > > > > > > >> >> > payload*. Something
> like:
> > > > >> > > metric_type
> > > > >> > > > > > >> (enum),
> > > > >> > > > > > >> > > > > > entity_id,
> > > > >> > > > > > >> > > > > > > > > > > > >> >> > table_identifier,
> > > snapshot_id
> > > > >> > > > > (nullable),
> > > > >> > > > > > >> > > > > received_ts,
> > > > >> > > > > > >> > > > > > > > > > > > >> schema_version,
> > > > >> > > > > > >> > > > > > > > > > > > >> >> and
> > > > >> > > > > > >> > > > > > > > > > > > >> >> > a payload column for
> the
> > > > >> > > metric-specific
> > > > >> > > > > > >> data.
> > > > >> > > > > > >> > > The
> > > > >> > > > > > >> > > > > > > > > metric_type
> > > > >> > > > > > >> > > > > > > > > > +
> > > > >> > > > > > >> > > > > > > > > > > > >> >> > schema_version pair
> gives
> > > us a
> > > > >> > > > > > >> > forward-compatible
> > > > >> > > > > > >> > > > > > > contract
> > > > >> > > > > > >> > > > > > > > > for
> > > > >> > > > > > >> > > > > > > > > > > the
> > > > >> > > > > > >> > > > > > > > > > > > >> >> payload
> > > > >> > > > > > >> > > > > > > > > > > > >> >> > shape. Adding a new
> > metric
> > > type
> > > > >> > > becomes
> > > > >> > > > > an
> > > > >> > > > > > >> enum
> > > > >> > > > > > >> > > > value
> > > > >> > > > > > >> > > > > > > and a
> > > > >> > > > > > >> > > > > > > > > > > payload
> > > > >> > > > > > >> > > > > > > > > > > > >> >> schema,
> > > > >> > > > > > >> > > > > > > > > > > > >> >> > not a schema migration.
> > One
> > > > >> thing
> > > > >> > I
> > > > >> > > > > think
> > > > >> > > > > > we
> > > > >> > > > > > >> > need
> > > > >> > > > > > >> > > > to
> > > > >> > > > > > >> > > > > be
> > > > >> > > > > > >> > > > > > > > > > > deliberate
> > > > >> > > > > > >> > > > > > > > > > > > >> >> about is
> > > > >> > > > > > >> > > > > > > > > > > > >> >> > the partition key
> design.
> > > If
> > > > >> all
> > > > >> > > metric
> > > > >> > > > > > >> types
> > > > >> > > > > > >> > > land
> > > > >> > > > > > >> > > > in
> > > > >> > > > > > >> > > > > > one
> > > > >> > > > > > >> > > > > > > > > > table,
> > > > >> > > > > > >> > > > > > > > > > > > scan
> > > > >> > > > > > >> > > > > > > > > > > > >> >> > metrics at scale (high
> > > > >> > concurrency,
> > > > >> > > high
> > > > >> > > > > > >> > > frequency
> > > > >> > > > > > >> > > > > > across
> > > > >> > > > > > >> > > > > > > > > many
> > > > >> > > > > > >> > > > > > > > > > > > >> tables)
> > > > >> > > > > > >> > > > > > > > > > > > >> >> > could easily create hot
> > > > >> > partitions.
> > > > >> > > We'd
> > > > >> > > > > > >> want
> > > > >> > > > > > >> > the
> > > > >> > > > > > >> > > > > > > > persistence
> > > > >> > > > > > >> > > > > > > > > > > layer
> > > > >> > > > > > >> > > > > > > > > > > > >> to
> > > > >> > > > > > >> > > > > > > > > > > > >> >> be
> > > > >> > > > > > >> > > > > > > > > > > > >> >> > able to shard by entity
> > or
> > > time
> > > > >> > > range,
> > > > >> > > > > and
> > > > >> > > > > > >> that
> > > > >> > > > > > >> > > > means
> > > > >> > > > > > >> > > > > > the
> > > > >> > > > > > >> > > > > > > > > > logical
> > > > >> > > > > > >> > > > > > > > > > > > >> schema
> > > > >> > > > > > >> > > > > > > > > > > > >> >> > needs to expose enough
> > > > >> structure
> > > > >> > for
> > > > >> > > > > > >> backends
> > > > >> > > > > > >> > to
> > > > >> > > > > > >> > > > > > > partition
> > > > >> > > > > > >> > > > > > > > > on.
> > > > >> > > > > > >> > > > > > > > > > I
> > > > >> > > > > > >> > > > > > > > > > > > >> don't
> > > > >> > > > > > >> > > > > > > > > > > > >> >> > think the current
> > flattened
> > > > >> layout
> > > > >> > > gives
> > > > >> > > > > > us
> > > > >> > > > > > >> > that.
> > > > >> > > > > > >> > > > > > > > > > > > >> >> >
> > > > >> > > > > > >> > > > > > > > > > > > >> >> > This is getting complex
> > > enough
> > > > >> > that
> > > > >> > > I
> > > > >> > > > > > don't
> > > > >> > > > > > >> > think
> > > > >> > > > > > >> > > > > > ad-hoc
> > > > >> > > > > > >> > > > > > > > > PR/ML
> > > > >> > > > > > >> > > > > > > > > > > > >> threads
> > > > >> > > > > > >> > > > > > > > > > > > >> >> > will converge well.
> > *Would
> > > > >> people
> > > > >> > be
> > > > >> > > > > open
> > > > >> > > > > > >> to a
> > > > >> > > > > > >> > > > > biweekly
> > > > >> > > > > > >> > > > > > > > sync
> > > > >> > > > > > >> > > > > > > > > > for
> > > > >> > > > > > >> > > > > > > > > > > > >> metrics
> > > > >> > > > > > >> > > > > > > > > > > > >> >> > architecture?* I think
> 30
> > > > >> minutes
> > > > >> > > every
> > > > >> > > > > > two
> > > > >> > > > > > >> > weeks
> > > > >> > > > > > >> > > > > with
> > > > >> > > > > > >> > > > > > > > > > interested
> > > > >> > > > > > >> > > > > > > > > > > > >> >> parties
> > > > >> > > > > > >> > > > > > > > > > > > >> >> > would be enough to work
> > > through
> > > > >> > the
> > > > >> > > > > > schema,
> > > > >> > > > > > >> SPI
> > > > >> > > > > > >> > > > > shape,
> > > > >> > > > > > >> > > > > > > and
> > > > >> > > > > > >> > > > > > > > > read
> > > > >> > > > > > >> > > > > > > > > > > API
> > > > >> > > > > > >> > > > > > > > > > > > >> >> design
> > > > >> > > > > > >> > > > > > > > > > > > >> >> > together. Happy to help
> > set
> > > > >> that
> > > > >> > up.
> > > > >> > > > > > >> > > > > > > > > > > > >> >> >
> > > > >> > > > > > >> > > > > > > > > > > > >> >> > -ej
> > > > >> > > > > > >> > > > > > > > > > > > >> >> >
> > > > >> > > > > > >> > > > > > > > > > > > >> >> > On Mon, Apr 20, 2026 at
> > > > >> 2:19 PM EJ
> > > > >> > > Wang
> > > > >> > > > > <
> > > > >> > > > > > >> > > > > > > > > > > > >>
> > ej.wang.dev.interest@gmail.com
> > > > >> > > > > > >> > > > > > > > > > > > >> >> >
> > > > >> > > > > > >> > > > > > > > > > > > >> >> > wrote:
> > > > >> > > > > > >> > > > > > > > > > > > >> >> >
> > > > >> > > > > > >> > > > > > > > > > > > >> >> >> Reviewed #4115, left a
> > > > >> comment on
> > > > >> > > the
> > > > >> > > > > > code
> > > > >> > > > > > >> > > > > > organization
> > > > >> > > > > > >> > > > > > > > > side.
> > > > >> > > > > > >> > > > > > > > > > > > >> >> >>
> > > > >> > > > > > >> > > > > > > > > > > > >> >> >> One thing stood out:
> the
> > > > >> metrics
> > > > >> > > write
> > > > >> > > > > > path
> > > > >> > > > > > >> > > enters
> > > > >> > > > > > >> > > > > > > through
> > > > >> > > > > > >> > > > > > > > > > > > >> >> >> PolarisMetricsManager
> on
> > > > >> > > > > > MetaStoreManager,
> > > > >> > > > > > >> but
> > > > >> > > > > > >> > > the
> > > > >> > > > > > >> > > > > new
> > > > >> > > > > > >> > > > > > > > read
> > > > >> > > > > > >> > > > > > > > > > path
> > > > >> > > > > > >> > > > > > > > > > > > >> >> bypasses
> > > > >> > > > > > >> > > > > > > > > > > > >> >> >> MetaStoreManager
> > entirely
> > > and
> > > > >> > goes
> > > > >> > > > > > >> straight to
> > > > >> > > > > > >> > > > > > > > > BasePersistence
> > > > >> > > > > > >> > > > > > > > > > > via
> > > > >> > > > > > >> > > > > > > > > > > > >> >> >>
> > > callContext.getMetaStore().
> > > > >> That
> > > > >> > > means
> > > > >> > > > > > the
> > > > >> > > > > > >> > read
> > > > >> > > > > > >> > > > API
> > > > >> > > > > > >> > > > > > only
> > > > >> > > > > > >> > > > > > > > > works
> > > > >> > > > > > >> > > > > > > > > > > for
> > > > >> > > > > > >> > > > > > > > > > > > >> >> backends
> > > > >> > > > > > >> > > > > > > > > > > > >> >> >> that implement
> > > > >> BasePersistence.
> > > > >> > > NoSQL
> > > > >> > > > > and
> > > > >> > > > > > >> > remote
> > > > >> > > > > > >> > > > > > > backends
> > > > >> > > > > > >> > > > > > > > > > can't
> > > > >> > > > > > >> > > > > > > > > > > > >> >> participate.
> > > > >> > > > > > >> > > > > > > > > > > > >> >> >>
> > > > >> > > > > > >> > > > > > > > > > > > >> >> >> Stepping back, I think
> > the
> > > > >> > metrics
> > > > >> > > > > > >> subsystem
> > > > >> > > > > > >> > is
> > > > >> > > > > > >> > > > > > growing
> > > > >> > > > > > >> > > > > > > > into
> > > > >> > > > > > >> > > > > > > > > > > > >> something
> > > > >> > > > > > >> > > > > > > > > > > > >> >> >> real (write + read +
> > REST
> > > API
> > > > >> +
> > > > >> > > AuthZ +
> > > > >> > > > > > >> > > > pagination)
> > > > >> > > > > > >> > > > > > *but
> > > > >> > > > > > >> > > > > > > > the
> > > > >> > > > > > >> > > > > > > > > > > > >> >> persistence
> > > > >> > > > > > >> > > > > > > > > > > > >> >> >> side is split across
> two
> > > > >> layers
> > > > >> > in
> > > > >> > > a
> > > > >> > > > > way
> > > > >> > > > > > >> > that's
> > > > >> > > > > > >> > > > hard
> > > > >> > > > > > >> > > > > > to
> > > > >> > > > > > >> > > > > > > > > > > extend*. I
> > > > >> > > > > > >> > > > > > > > > > > > >> put
> > > > >> > > > > > >> > > > > > > > > > > > >> >> >> together two diagrams
> to
> > > show
> > > > >> > what
> > > > >> > > I
> > > > >> > > > > mean
> > > > >> > > > > > >> (my
> > > > >> > > > > > >> > > best
> > > > >> > > > > > >> > > > > > > > effort).
> > > > >> > > > > > >> > > > > > > > > > > > >> >> >>
> > > > >> > > > > > >> > > > > > > > > > > > >> >> >> *Current state*
> (Diagram
> > > 1):
> > > > >> > three
> > > > >> > > > > > >> interfaces
> > > > >> > > > > > >> > at
> > > > >> > > > > > >> > > > > three
> > > > >> > > > > > >> > > > > > > > > > different
> > > > >> > > > > > >> > > > > > > > > > > > >> >> levels.
> > > > >> > > > > > >> > > > > > > > > > > > >> >> >> The engine-facing SPI
> > > > >> > > > > > >> (PolarisMetricsReporter)
> > > > >> > > > > > >> > > is
> > > > >> > > > > > >> > > > > > clean.
> > > > >> > > > > > >> > > > > > > > But
> > > > >> > > > > > >> > > > > > > > > > > > >> >> >> PolarisMetricsManager
> on
> > > > >> > > > > MetaStoreManager
> > > > >> > > > > > >> is a
> > > > >> > > > > > >> > > > > > > passthrough
> > > > >> > > > > > >> > > > > > > > > to
> > > > >> > > > > > >> > > > > > > > > > > > >> >> >> MetricsPersistence on
> > > > >> > > BasePersistence.
> > > > >> > > > > > The
> > > > >> > > > > > >> > @Beta
> > > > >> > > > > > >> > > > > > > > annotation
> > > > >> > > > > > >> > > > > > > > > > and
> > > > >> > > > > > >> > > > > > > > > > > > SPI
> > > > >> > > > > > >> > > > > > > > > > > > >> >> javadoc
> > > > >> > > > > > >> > > > > > > > > > > > >> >> >> are on the
> > BasePersistence
> > > > >> layer,
> > > > >> > > while
> > > > >> > > > > > the
> > > > >> > > > > > >> > > actual
> > > > >> > > > > > >> > > > > > > > extension
> > > > >> > > > > > >> > > > > > > > > > > > points
> > > > >> > > > > > >> > > > > > > > > > > > >> >> >>
> (PolarisMetricsReporter,
> > > > >> > > > > > >> > PolarisMetricsManager)
> > > > >> > > > > > >> > > > > carry
> > > > >> > > > > > >> > > > > > no
> > > > >> > > > > > >> > > > > > > > > > > stability
> > > > >> > > > > > >> > > > > > > > > > > > >> >> >> annotation. The write
> > path
> > > > >> goes
> > > > >> > > through
> > > > >> > > > > > the
> > > > >> > > > > > >> > > > > > > > MetaStoreManager
> > > > >> > > > > > >> > > > > > > > > > > > layer,
> > > > >> > > > > > >> > > > > > > > > > > > >> the
> > > > >> > > > > > >> > > > > > > > > > > > >> >> >> read path doesn't.
> > > > >> > > > > > >> > > > > > > > > > > > >> >> >>
> > > > >> > > > > > >> > > > > > > > > > > > >> >> >> *What I envision*
> > > (Diagram 2):
> > > > >> > two
> > > > >> > > SPIs
> > > > >> > > > > > at
> > > > >> > > > > > >> two
> > > > >> > > > > > >> > > > > levels.
> > > > >> > > > > > >> > > > > > > > > > > > >> >> >> PolarisMetricsReporter
> > > stays
> > > > >> as
> > > > >> > the
> > > > >> > > > > > >> > > engine-facing
> > > > >> > > > > > >> > > > > SPI.
> > > > >> > > > > > >> > > > > > > > > > > > >> >> >> PolarisMetricsManager
> > > becomes
> > > > >> the
> > > > >> > > > > > >> > backend-facing
> > > > >> > > > > > >> > > > SPI
> > > > >> > > > > > >> > > > > > > with
> > > > >> > > > > > >> > > > > > > > > both
> > > > >> > > > > > >> > > > > > > > > > > > write
> > > > >> > > > > > >> > > > > > > > > > > > >> >> and
> > > > >> > > > > > >> > > > > > > > > > > > >> >> >> read methods at the
> > > > >> > > MetaStoreManager
> > > > >> > > > > > level,
> > > > >> > > > > > >> > > where
> > > > >> > > > > > >> > > > > any
> > > > >> > > > > > >> > > > > > > > > backend
> > > > >> > > > > > >> > > > > > > > > > > > (JDBC,
> > > > >> > > > > > >> > > > > > > > > > > > >> >> NoSQL,
> > > > >> > > > > > >> > > > > > > > > > > > >> >> >> remote) can implement
> > > them.
> > > > >> > > > > > >> MetricsPersistence
> > > > >> > > > > > >> > > on
> > > > >> > > > > > >> > > > > > > > > > > BasePersistence
> > > > >> > > > > > >> > > > > > > > > > > > >> goes
> > > > >> > > > > > >> > > > > > > > > > > > >> >> >> away. Where metrics
> > > actually
> > > > >> land
> > > > >> > > is an
> > > > >> > > > > > >> > > > > implementation
> > > > >> > > > > > >> > > > > > > > > detail,
> > > > >> > > > > > >> > > > > > > > > > > > not a
> > > > >> > > > > > >> > > > > > > > > > > > >> >> core
> > > > >> > > > > > >> > > > > > > > > > > > >> >> >> interface.
> > > > >> > > > > > >> > > > > > > > > > > > >> >> >>
> > > > >> > > > > > >> > > > > > > > > > > > >> >> >> *Minor naming thing*:
> > > > >> > > > > > >> PolarisMetricsReporter
> > > > >> > > > > > >> > is
> > > > >> > > > > > >> > > > > > broader
> > > > >> > > > > > >> > > > > > > > than
> > > > >> > > > > > >> > > > > > > > > > > what
> > > > >> > > > > > >> > > > > > > > > > > > it
> > > > >> > > > > > >> > > > > > > > > > > > >> >> >> actually handles. It
> > only
> > > > >> accepts
> > > > >> > > > > Iceberg
> > > > >> > > > > > >> REST
> > > > >> > > > > > >> > > > > Catalog
> > > > >> > > > > > >> > > > > > > > > metrics
> > > > >> > > > > > >> > > > > > > > > > > > >> >> (ScanReport,
> > > > >> > > > > > >> > > > > > > > > > > > >> >> >> CommitReport via
> > > > >> MetricsReport).
> > > > >> > > > > Generic
> > > > >> > > > > > >> table
> > > > >> > > > > > >> > > > > metrics
> > > > >> > > > > > >> > > > > > > or
> > > > >> > > > > > >> > > > > > > > > > > > >> operational
> > > > >> > > > > > >> > > > > > > > > > > > >> >> >> metrics aren't in
> scope.
> > > Not
> > > > >> > > blocking,
> > > > >> > > > > > but
> > > > >> > > > > > >> > worth
> > > > >> > > > > > >> > > > > > noting
> > > > >> > > > > > >> > > > > > > if
> > > > >> > > > > > >> > > > > > > > > the
> > > > >> > > > > > >> > > > > > > > > > > > >> metrics
> > > > >> > > > > > >> > > > > > > > > > > > >> >> >> surface expands.
> > > > >> > > > > > >> > > > > > > > > > > > >> >> >>
> > > > >> > > > > > >> > > > > > > > > > > > >> >> >> *Rough sketch of how
> to
> > > get
> > > > >> > there*:
> > > > >> > > > > > >> > > > > > > > > > > > >> >> >>  1.  Add read methods
> to
> > > > >> > > > > > >> PolarisMetricsManager
> > > > >> > > > > > >> > > > > > > > > > (listScanReports,
> > > > >> > > > > > >> > > > > > > > > > > > >> >> >> listCommitReports)
> with
> > > > >> default
> > > > >> > > no-op,
> > > > >> > > > > > >> same as
> > > > >> > > > > > >> > > the
> > > > >> > > > > > >> > > > > > > > existing
> > > > >> > > > > > >> > > > > > > > > > > write
> > > > >> > > > > > >> > > > > > > > > > > > >> >> methods.
> > > > >> > > > > > >> > > > > > > > > > > > >> >> >> (Probably make
> > > > >> > > PolarisMetricsManager
> > > > >> > > > > more
> > > > >> > > > > > >> > > explicit
> > > > >> > > > > > >> > > > > on
> > > > >> > > > > > >> > > > > > > > being
> > > > >> > > > > > >> > > > > > > > > > > > Iceberg
> > > > >> > > > > > >> > > > > > > > > > > > >> >> >> specific like package
> > > name or
> > > > >> > class
> > > > >> > > > > name
> > > > >> > > > > > >> etc.)
> > > > >> > > > > > >> > > > > > > > > > > > >> >> >>  2.  Wire
> > > > >> MetricsReportsService
> > > > >> > > through
> > > > >> > > > > > >> > > > > > MetaStoreManager
> > > > >> > > > > > >> > > > > > > > > > instead
> > > > >> > > > > > >> > > > > > > > > > > > of
> > > > >> > > > > > >> > > > > > > > > > > > >> >> >>
> > > callContext.getMetaStore().
> > > > >> > > > > > >> > > > > > > > > > > > >> >> >>  3.  Extract metrics
> > > > >> persistence
> > > > >> > > from
> > > > >> > > > > > >> > > > > > > > > JdbcBasePersistenceImpl
> > > > >> > > > > > >> > > > > > > > > > > into
> > > > >> > > > > > >> > > > > > > > > > > > >> its
> > > > >> > > > > > >> > > > > > > > > > > > >> >> >> own class. That file
> > > carries
> > > > >> ~7
> > > > >> > > > > > >> > > responsibilities,
> > > > >> > > > > > >> > > > > > > metrics
> > > > >> > > > > > >> > > > > > > > > > being
> > > > >> > > > > > >> > > > > > > > > > > > one
> > > > >> > > > > > >> > > > > > > > > > > > >> of
> > > > >> > > > > > >> > > > > > > > > > > > >> >> them.
> > > > >> > > > > > >> > > > > > > > > > > > >> >> >>  4.  Remove
> > > MetricsPersistence
> > > > >> > from
> > > > >> > > > > > >> > > > BasePersistence.
> > > > >> > > > > > >> > > > > > > > > > > > >> >> >>
> > > > >> > > > > > >> > > > > > > > > > > > >> >> >> *None of this needs to
> > > happen
> > > > >> in
> > > > >> > > #4115.
> > > > >> > > > > > >> But if
> > > > >> > > > > > >> > > the
> > > > >> > > > > > >> > > > > > > > direction
> > > > >> > > > > > >> > > > > > > > > > > makes
> > > > >> > > > > > >> > > > > > > > > > > > >> >> sense,
> > > > >> > > > > > >> > > > > > > > > > > > >> >> >> it would be good to
> > align
> > > > >> before
> > > > >> > > the
> > > > >> > > > > > >> metrics
> > > > >> > > > > > >> > > > surface
> > > > >> > > > > > >> > > > > > > grows
> > > > >> > > > > > >> > > > > > > > > > > > further.
> > > > >> > > > > > >> > > > > > > > > > > > >> >> Curious
> > > > >> > > > > > >> > > > > > > > > > > > >> >> >> what others think.*
> > > > >> > > > > > >> > > > > > > > > > > > >> >> >>
> > > > >> > > > > > >> > > > > > > > > > > > >> >> >> *My mental model
> note*:
> > > Level
> > > > >> 1
> > > > >> > > > > > >> > > MetaStoreManager;
> > > > >> > > > > > >> > > > > > level
> > > > >> > > > > > >> > > > > > > 2
> > > > >> > > > > > >> > > > > > > > > > > > >> transactional
> > > > >> > > > > > >> > > > > > > > > > > > >> >> >> persistence; level 3
> > base
> > > > >> > > persistence
> > > > >> > > > > > >> > > > > > > > > > > > >> >> >>
> > > > >> > > > > > >> > > > > > > > > > > > >> >> >> Diagram 1
> > > > >> > > > > > >> > > > > > > > > > > > >> >> >> <
> > > > >> > > > > > >> > > > > > > > > > > > >> >>
> > > > >> > > > > > >> > > > > > > > > > > > >>
> > > > >> > > > > > >> > > > > > > > > > > >
> > > > >> > > > > > >> > > > > > > > > > >
> > > > >> > > > > > >> > > > > > > > > >
> > > > >> > > > > > >> > > > > > > > >
> > > > >> > > > > > >> > > > > > > >
> > > > >> > > > > > >> > > > > > >
> > > > >> > > > > > >> > > > > >
> > > > >> > > > > > >> > > > >
> > > > >> > > > > > >> > > >
> > > > >> > > > > > >> > >
> > > > >> > > > > > >> >
> > > > >> > > > > > >>
> > > > >> > > > > >
> > > > >> > > > >
> > > > >> > >
> > > > >> >
> > > > >>
> > >
> >
> https://urldefense.com/v3/__https://www.plantuml.com/plantuml/uml/bLHDR-Cs4BthLmpIYupw0zbkKQ1r3M-S7Bp8xhhM7WCOb3IM65EaGD9EX2RzxHrHb4CxRelwa4YSDu_lpOVcnZ9jzvM8BBS2uGjQpJC3dtHMSekPtMk44IpsMgEqa5XcCOhCZikQQLP1pR8TAp2n3ILhmZDP20m0fcIvUkAoW2qJXd9z1bpToO9BX3WXu0ucy5rpgGPNm0nW5_epUWtm2Ue3pn3kMOFQmKntGZW0BYtgBSi8k5A2QMwybJNMIbFiGSR9QZc4nUqIvikStF0jHprua5C-amge42aNt3R0f5JaaoivdV2Pkqbx4hee4ymOkBh5BTiB-_uIeGeo8zL8rPsPl4DktdEiK1jkB1NdZCRbrSTecDe_mlHbF0wvBmCkaOH5_S8a_TTTKI6-nmCAkEw4LpxsZ-LbYLKQFKMNOgf_wuM7_bV9gOer5SYMMksBSWXFcbi49KNZXNLicwfe3TETC7gPdPqI7uBcHMb1RSzYq34c6PDUM9mn8HRsUTZEiDBve3NjVZumBj0U7SS37mGO7vcwtiK-_pU7U7L_f-digo9YbhSwIfMRwIITKGXbxdIUTCGF1SeCJxloKsU-3k9ddRbX1eDq1q_fx1JbBGT0glVyXimDuP4TQ5qpCAmnGEj2s_6n5mtn1z-97-63itFQZLPO1Ev2tu_WF7Ju-VPc0Skg5bYXxBhkY1xpD7EM_7fyflSpIsqMgVth5xhVr4eQxWQ8enaSAJQSG16yFSDuJ798rrcXr_3n-lfdk7icQjEBmFujL7AodiP_Y4Z7-YxvtZNs4zMgpNTl6tF8sglyPsmqchrjvQ-m-aP94r-TwCA2Ka8upPJZwtvSpoYCXkYMZU2NXvRMBfq9P3i3Le4VAZUAlUZ_oPKsxPgY0Q_BSKLkyr9bhQhQrJjo_x3TPlIB0DPjnMfcIoYP0QaYw1a0fTKDr8fB6ntNuvmoL1ZGkXa69Njh43zf9GiGxHQrA_jDYWRSzF5--WmTVrN97_Sm8LbLUy_lGBmLanJjFkDlGkRqjA_4tm00__;!!Iz9xO38YGHZK!4rnlVIqBQQQ6xpGe1PMV58edOS_cGrKoqdfhw4qSp_JQCavvEjaxXk0OwGPnxyEcO60AzKQK_XDtp7H3DdkFXcrXuTBISHg$
> > > > >> > > > > > >> > > > > > > > > > > > >> >> >
> > > > >> > > > > > >> > > > > > > > > > > > >> >> >> :
> > > > >> > > > > > >> > > > > > > > > > > > >> >> >>
> > > > >> > > > > > >> > > > > > > > > > > > >> >> >> [image: image.png]
> > > > >> > > > > > >> > > > > > > > > > > > >> >> >>
> > > > >> > > > > > >> > > > > > > > > > > > >> >> >> Diagram 2
> > > > >> > > > > > >> > > > > > > > > > > > >> >> >> <
> > > > >> > > > > > >> > > > > > > > > > > > >> >>
> > > > >> > > > > > >> > > > > > > > > > > > >>
> > > > >> > > > > > >> > > > > > > > > > > >
> > > > >> > > > > > >> > > > > > > > > > >
> > > > >> > > > > > >> > > > > > > > > >
> > > > >> > > > > > >> > > > > > > > >
> > > > >> > > > > > >> > > > > > > >
> > > > >> > > > > > >> > > > > > >
> > > > >> > > > > > >> > > > > >
> > > > >> > > > > > >> > > > >
> > > > >> > > > > > >> > > >
> > > > >> > > > > > >> > >
> > > > >> > > > > > >> >
> > > > >> > > > > > >>
> > > > >> > > > > >
> > > > >> > > > >
> > > > >> > >
> > > > >> >
> > > > >>
> > >
> >
> https://urldefense.com/v3/__https://www.plantuml.com/plantuml/uml/VLLDR-8m4BtdLupO2sWBLVU8AaGB7AXAbssGzb896SS9RXqxjKqBwkv_tt7iV43fdaZYDpFlpRmnOsE9jhjSH9PRmM31hERKm8scMsuPjJlDe0yheZDc8RR4iYWoBrmMH9CS2a9VICPYUy1OZN0YCy5Q0BCbYNhdCeEK28En8G8wCvbnoQ0R8_05Bc6bkLIz3X03p1zzH7zR-9ZfDquPt9C3qoNCX2yV4G2NbkcKu5jdgGJHt0GbZwnG6i-UP3TUpk5gM6Ldqke350eZUqzoCft3U9xWHvxoa5-7K4nF1J46EbEMafsmdrCBbQ44gVggy18IZrn_ph5asd1ZiIKdQSgueZvjXrQFSFrdC3YN-nXmBacxbGiYyLVxLaBtdhqn0LSzdBDhqQtQoOJeGyad3z0lUqnYgpGB6Ns8oVyta00Dy_WnX0tIOZ8v6SYxHll1TrH6aejAik-mh-AphVFCwSUQqFypElag5QRGFDjQKEd96K1P8QP41c9TzA_IIQyvdAWyv_RSiS3skb0_EzDDkK2v5xWF6MiGFlvhpFLcD2Dq2pml14gaF67eQkmd8gulDoC4kSOu6KVpkvlUJg1RTbWISU40RdBUUS_9XfRZ2dwxm_SW8LYFISgm_MnlDQ6M9P1gbKEc4X-2pH_FvJCkCqm9pbVjD6LrwdLeOrDWfOaqc8Wh9BE85oNKxkNQ6o4yGRy_Eae0G_G8tZv81d3bHDB23WOdisohVr3nh_j6lbSjbNaLRTc8UgtPbAU1J_tygOfZX9DWEJeHDvYx-qmSi5FgNLPZwHrHcUsncGQ5-skhUclpE5fo4ounpFauYrUbkU6ccfnxMvitwag4IyerhTxj8In_Oj1bDO4pQru674loYrGlULHLEGCjwJJ8gDoVZR8MxO4BT3IzRvIcAQKezC6xpziGnTyImrfEGyJI_OcKfgtxIvnTqFEMS17L9Z-jsARN5FmTheP7HtSdtOMT0B4GY2FYHXxgQmMtj2bRqiLFGapiVe1_QVKDrkqXcm83aFEXnMYCZ-xlyHy__;!!Iz9xO38YGHZK!4rnlVIqBQQQ6xpGe1PMV58edOS_cGrKoqdfhw4qSp_JQCavvEjaxXk0OwGPnxyEcO60AzKQK_XDtp7H3DdkFXcrXSpmqcNk$
> > > > >> > > > > > >> > > > > > > > > > > > >> >> >
> > > > >> > > > > > >> > > > > > > > > > > > >> >> >> :
> > > > >> > > > > > >> > > > > > > > > > > > >> >> >> [image: image.png]
> > > > >> > > > > > >> > > > > > > > > > > > >> >> >>
> > > > >> > > > > > >> > > > > > > > > > > > >> >> >>  -ej
> > > > >> > > > > > >> > > > > > > > > > > > >> >> >>
> > > > >> > > > > > >> > > > > > > > > > > > >> >> >> On Wed, Apr 15, 2026
> at
> > > > >> 8:22 AM
> > > > >> > > Dmitri
> > > > >> > > > > > >> > > > Bourlatchkov
> > > > >> > > > > > >> > > > > <
> > > > >> > > > > > >> > > > > > > > > > > > >> dimas@apache.org>
> > > > >> > > > > > >> > > > > > > > > > > > >> >> >> wrote:
> > > > >> > > > > > >> > > > > > > > > > > > >> >> >>
> > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> Hi All,
> > > > >> > > > > > >> > > > > > > > > > > > >> >> >>>
> > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> Heads up: The current
> > > state
> > > > >> of
> > > > >> > PR
> > > > >> > > > > [4115]
> > > > >> > > > > > >> > looks
> > > > >> > > > > > >> > > > > pretty
> > > > >> > > > > > >> > > > > > > > solid
> > > > >> > > > > > >> > > > > > > > > > to
> > > > >> > > > > > >> > > > > > > > > > > > me.
> > > > >> > > > > > >> > > > > > > > > > > > >> I
> > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> believe this PR is
> > > > >> approaching a
> > > > >> > > > > > mergeable
> > > > >> > > > > > >> > > > > condition.
> > > > >> > > > > > >> > > > > > > > > > > > >> >> >>>
> > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> Please post your
> > reviews
> > > if
> > > > >> you
> > > > >> > > have
> > > > >> > > > > any
> > > > >> > > > > > >> > > > comments.
> > > > >> > > > > > >> > > > > > > > > > > > >> >> >>>
> > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> [4115]
> > > > >> > > > > > >> > > >
> > > > >> >
> > > > >>
> > >
> >
> https://urldefense.com/v3/__https://github.com/apache/polaris/pull/4115__;!!Iz9xO38YGHZK!4rnlVIqBQQQ6xpGe1PMV58edOS_cGrKoqdfhw4qSp_JQCavvEjaxXk0OwGPnxyEcO60AzKQK_XDtp7H3DdkFXcrXtFRscKs$
> > > > >> > > > > > >> > > > > > > > > > > > >> >> >>>
> > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> Thanks,
> > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> Dmitri.
> > > > >> > > > > > >> > > > > > > > > > > > >> >> >>>
> > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> On Tue, Mar 3, 2026
> at
> > > > >> 3:29 PM
> > > > >> > > Anand
> > > > >> > > > > > Kumar
> > > > >> > > > > > >> > > > Sankaran
> > > > >> > > > > > >> > > > > > via
> > > > >> > > > > > >> > > > > > > > > dev <
> > > > >> > > > > > >> > > > > > > > > > > > >> >> >>>
> dev@polaris.apache.org
> > >
> > > > >> wrote:
> > > > >> > > > > > >> > > > > > > > > > > > >> >> >>>
> > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > Hi Yufei and
> Dmitri,
> > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> >
> > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > Here is a proposal
> > for
> > > the
> > > > >> > REST
> > > > >> > > > > > >> endpoints
> > > > >> > > > > > >> > for
> > > > >> > > > > > >> > > > > > metrics
> > > > >> > > > > > >> > > > > > > > and
> > > > >> > > > > > >> > > > > > > > > > > > events.
> > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> >
> > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> >
> > > > >> > > > > > >> > > > > >
> > > > >> >
> > > > >>
> > >
> >
> https://urldefense.com/v3/__https://github.com/apache/polaris/pull/3924/changes__;!!Iz9xO38YGHZK!4rnlVIqBQQQ6xpGe1PMV58edOS_cGrKoqdfhw4qSp_JQCavvEjaxXk0OwGPnxyEcO60AzKQK_XDtp7H3DdkFXcrXcjoPhSs$
> > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> >
> > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > I did not see any
> > > > >> precursors
> > > > >> > for
> > > > >> > > > > > >> raising a
> > > > >> > > > > > >> > PR
> > > > >> > > > > > >> > > > for
> > > > >> > > > > > >> > > > > > > > > > proposals,
> > > > >> > > > > > >> > > > > > > > > > > so
> > > > >> > > > > > >> > > > > > > > > > > > >> >> trying
> > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > this.  Please let
> me
> > > know
> > > > >> what
> > > > >> > > you
> > > > >> > > > > > >> think.
> > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> >
> > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > -
> > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > Anand
> > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> >
> > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > From: Anand Kumar
> > > Sankaran
> > > > >> <
> > > > >> > > > > > >> > > > > > > anand.sankaran@workday.com
> > > > >> > > > > > >> > > > > > > > >
> > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > Date: Monday, March
> > 2,
> > > > >> 2026 at
> > > > >> > > > > > 10:25 AM
> > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > To:
> > > dev@polaris.apache.org
> > > > >> <
> > > > >> > > > > > >> > > > > dev@polaris.apache.org
> > > > >> > > > > > >> > > > > > >
> > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > Subject: Re:
> Polaris
> > > > >> Telemetry
> > > > >> > > and
> > > > >> > > > > > Audit
> > > > >> > > > > > >> > > Trail
> > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> >
> > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > About the REST API,
> > > based
> > > > >> on
> > > > >> > my
> > > > >> > > use
> > > > >> > > > > > >> cases:
> > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> >
> > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> >
> > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> >   1.
> > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > I want to be able
> to
> > > query
> > > > >> > > commit
> > > > >> > > > > > >> metrics
> > > > >> > > > > > >> > to
> > > > >> > > > > > >> > > > > track
> > > > >> > > > > > >> > > > > > > > files
> > > > >> > > > > > >> > > > > > > > > > > added
> > > > >> > > > > > >> > > > > > > > > > > > /
> > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> removed
> > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > per commit, along
> > with
> > > > >> record
> > > > >> > > > > counts.
> > > > >> > > > > > >> The
> > > > >> > > > > > >> > > > > ingestion
> > > > >> > > > > > >> > > > > > > > > > pipeline
> > > > >> > > > > > >> > > > > > > > > > > > that
> > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> writes
> > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > this data is owned
> by
> > > us
> > > > >> and
> > > > >> > we
> > > > >> > > are
> > > > >> > > > > > >> > > guaranteed
> > > > >> > > > > > >> > > > to
> > > > >> > > > > > >> > > > > > > write
> > > > >> > > > > > >> > > > > > > > > > this
> > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> information
> > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > for each write.
> > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> >   2.
> > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > I want to be able
> to
> > > query
> > > > >> > scan
> > > > >> > > > > > metrics
> > > > >> > > > > > >> for
> > > > >> > > > > > >> > > > > read. I
> > > > >> > > > > > >> > > > > > > > > > > understand
> > > > >> > > > > > >> > > > > > > > > > > > >> >> clients
> > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> do
> > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > not fulfill this
> > > > >> requirement.
> > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> >   3.
> > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > I want to be able
> to
> > > query
> > > > >> the
> > > > >> > > > > events
> > > > >> > > > > > >> table
> > > > >> > > > > > >> > > > > (events
> > > > >> > > > > > >> > > > > > > are
> > > > >> > > > > > >> > > > > > > > > > > > >> persisted) -
> > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> this
> > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > may supersede #2, I
> > am
> > > not
> > > > >> > sure
> > > > >> > > yet.
> > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> >
> > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > All this
> information
> > > is in
> > > > >> the
> > > > >> > > JDBC
> > > > >> > > > > > >> based
> > > > >> > > > > > >> > > > > > persistence
> > > > >> > > > > > >> > > > > > > > > model
> > > > >> > > > > > >> > > > > > > > > > > and
> > > > >> > > > > > >> > > > > > > > > > > > >> is
> > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > persisted in the
> > > > >> metastore. I
> > > > >> > > > > > currently
> > > > >> > > > > > >> > don’t
> > > > >> > > > > > >> > > > > have
> > > > >> > > > > > >> > > > > > a
> > > > >> > > > > > >> > > > > > > > need
> > > > >> > > > > > >> > > > > > > > > > to
> > > > >> > > > > > >> > > > > > > > > > > > >> query
> > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > prometheus or open
> > > > >> telemetry.
> > > > >> > I
> > > > >> > > do
> > > > >> > > > > > >> publish
> > > > >> > > > > > >> > > some
> > > > >> > > > > > >> > > > > > > events
> > > > >> > > > > > >> > > > > > > > to
> > > > >> > > > > > >> > > > > > > > > > > > >> Prometheus
> > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> and
> > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > they are forwarded
> to
> > > our
> > > > >> > > dashboards
> > > > >> > > > > > >> > > elsewhere.
> > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> >
> > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > About the CLI
> > > utilities, I
> > > > >> > > meant the
> > > > >> > > > > > >> admin
> > > > >> > > > > > >> > > user
> > > > >> > > > > > >> > > > > > > > > utilities.
> > > > >> > > > > > >> > > > > > > > > > In
> > > > >> > > > > > >> > > > > > > > > > > > >> one of
> > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> the
> > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > earliest drafts of
> my
> > > > >> > proposal,
> > > > >> > > > > > Prashant
> > > > >> > > > > > >> > > > > mentioned
> > > > >> > > > > > >> > > > > > > that
> > > > >> > > > > > >> > > > > > > > > the
> > > > >> > > > > > >> > > > > > > > > > > > >> metrics
> > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> tables
> > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > can grow
> indefinitely
> > > and
> > > > >> > that a
> > > > >> > > > > > similar
> > > > >> > > > > > >> > > > problem
> > > > >> > > > > > >> > > > > > > exists
> > > > >> > > > > > >> > > > > > > > > > with
> > > > >> > > > > > >> > > > > > > > > > > > the
> > > > >> > > > > > >> > > > > > > > > > > > >> >> events
> > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > table as well. We
> > > discussed
> > > > >> > that
> > > > >> > > > > > >> cleaning
> > > > >> > > > > > >> > up
> > > > >> > > > > > >> > > of
> > > > >> > > > > > >> > > > > old
> > > > >> > > > > > >> > > > > > > > > records
> > > > >> > > > > > >> > > > > > > > > > > > from
> > > > >> > > > > > >> > > > > > > > > > > > >> >> both
> > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > metrics tables and
> > > events
> > > > >> > > tables can
> > > > >> > > > > > be
> > > > >> > > > > > >> > done
> > > > >> > > > > > >> > > > via
> > > > >> > > > > > >> > > > > a
> > > > >> > > > > > >> > > > > > > CLI
> > > > >> > > > > > >> > > > > > > > > > > utility.
> > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> >
> > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > I see that Yufei
> has
> > > > >> covered
> > > > >> > the
> > > > >> > > > > > >> discussion
> > > > >> > > > > > >> > > > about
> > > > >> > > > > > >> > > > > > > > > > > datasources.
> > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> >
> > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > -
> > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > Anand
> > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> >
> > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> >
> > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> >
> > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > From: Yufei Gu <
> > > > >> > > > > flyrain000@gmail.com>
> > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > Date: Friday,
> > February
> > > 27,
> > > > >> > 2026
> > > > >> > > at
> > > > >> > > > > > >> 9:54 PM
> > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > To:
> > > dev@polaris.apache.org
> > > > >> <
> > > > >> > > > > > >> > > > > dev@polaris.apache.org
> > > > >> > > > > > >> > > > > > >
> > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > Subject: Re:
> Polaris
> > > > >> Telemetry
> > > > >> > > and
> > > > >> > > > > > Audit
> > > > >> > > > > > >> > > Trail
> > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> >
> > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > This Message Is
> From
> > an
> > > > >> > External
> > > > >> > > > > > Sender
> > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > This message came
> > from
> > > > >> outside
> > > > >> > > your
> > > > >> > > > > > >> > > > organization.
> > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > Report Suspicious<
> > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> >
> > > > >> > > > > > >> > > > > > > > > > > > >> >> >>>
> > > > >> > > > > > >> > > > > > > > > > > > >> >>
> > > > >> > > > > > >> > > > > > > > > > > > >>
> > > > >> > > > > > >> > > > > > > > > > > >
> > > > >> > > > > > >> > > > > > > > > > >
> > > > >> > > > > > >> > > > > > > > > >
> > > > >> > > > > > >> > > > > > > > >
> > > > >> > > > > > >> > > > > > > >
> > > > >> > > > > > >> > > > > > >
> > > > >> > > > > > >> > > > > >
> > > > >> > > > > > >> > > > >
> > > > >> > > > > > >> > > >
> > > > >> > > > > > >> > >
> > > > >> > > > > > >> >
> > > > >> > > > > > >>
> > > > >> > > > > >
> > > > >> > > > >
> > > > >> > >
> > > > >> >
> > > > >>
> > >
> >
> https://us-phishalarm-ewt.proofpoint.com/EWT/v1/Iz9xO38YGHZK!YhNDZABkHi1B699ote2uMwpOZw8i0QMCGO2Szc-HshuABGhGvwPJcymE6G2oUUxtS8xDkSrtGTPm_I3QnVDHoLMk50m9v8z_nZKTkd-bnVUbreF1u0WnfV_X5eYevZl_$
> > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > >
> > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> >
> > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> >
> > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > As I mentioned in
> > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> >
> > > > >> > > > > > >> > > > > > > > > > > > >> >> >>>
> > > > >> > > > > > >> > > > > > > > > > > > >> >>
> > > > >> > > > > > >> > > > > > > > > > > > >>
> > > > >> > > > > > >> > > > > > > > > > > >
> > > > >> > > > > > >> > > > > > > > > > >
> > > > >> > > > > > >> > > > > > > > > >
> > > > >> > > > > > >> > > > > > > > >
> > > > >> > > > > > >> > > > > > > >
> > > > >> > > > > > >> > > > > > >
> > > > >> > > > > > >> > > > > >
> > > > >> > > > > > >> > > > >
> > > > >> > > > > > >> > > >
> > > > >> > > > > > >> > >
> > > > >> > > > > > >> >
> > > > >> > > > > > >>
> > > > >> > > > > >
> > > > >> > > > >
> > > > >> > >
> > > > >> >
> > > > >>
> > >
> >
> https://urldefense.com/v3/__https://github.com/apache/polaris/issues/3890__;!!Iz9xO38YGHZK!5EuyFFkk3vhRWVIRvQAWBSQfpJkTMA9HxugzDwXmN0LPPqhEFxYkFRGVhtb8AqUwXtDh2OplcMnbMDHKOxrvDU0$
> > > > >> > >> > > >> > > > > > > > > > > > >> >> >>> ,
> > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > supporting
> > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > multiple data
> sources
> > > is
> > > > >> not a
> > > > >> > > > > trivial
> > > > >> > > > > > >> > > change.
> > > > >> > > > > > >> > > > I
> > > > >> > > > > > >> > > > > > > would
> > > > >> > > > > > >> > > > > > > > > > > strongly
> > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> recommend
> > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > starting with a
> > design
> > > > >> > document
> > > > >> > > to
> > > > >> > > > > > >> > carefully
> > > > >> > > > > > >> > > > > > evaluate
> > > > >> > > > > > >> > > > > > > > the
> > > > >> > > > > > >> > > > > > > > > > > > >> >> architectural
> > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > implications and
> long
> > > term
> > > > >> > > impact.
> > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> >
> > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > A REST endpoint to
> > > query
> > > > >> > metrics
> > > > >> > > > > seems
> > > > >> > > > > > >> > > > reasonable
> > > > >> > > > > > >> > > > > > > given
> > > > >> > > > > > >> > > > > > > > > the
> > > > >> > > > > > >> > > > > > > > > > > > >> current
> > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> JDBC
> > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > based persistence
> > > model.
> > > > >> That
> > > > >> > > said,
> > > > >> > > > > we
> > > > >> > > > > > >> may
> > > > >> > > > > > >> > > also
> > > > >> > > > > > >> > > > > > > > consider
> > > > >> > > > > > >> > > > > > > > > > > > >> alternative
> > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > storage models. For
> > > > >> example,
> > > > >> > if
> > > > >> > > we
> > > > >> > > > > > later
> > > > >> > > > > > >> > > adopt
> > > > >> > > > > > >> > > > a
> > > > >> > > > > > >> > > > > > time
> > > > >> > > > > > >> > > > > > > > > > series
> > > > >> > > > > > >> > > > > > > > > > > > >> system
> > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> such as
> > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > Prometheus to store
> > > > >> metrics,
> > > > >> > the
> > > > >> > > > > query
> > > > >> > > > > > >> > model
> > > > >> > > > > > >> > > > and
> > > > >> > > > > > >> > > > > > > access
> > > > >> > > > > > >> > > > > > > > > > > > patterns
> > > > >> > > > > > >> > > > > > > > > > > > >> >> would
> > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> be
> > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > fundamentally
> > > different.
> > > > >> > > Designing
> > > > >> > > > > the
> > > > >> > > > > > >> REST
> > > > >> > > > > > >> > > API
> > > > >> > > > > > >> > > > > > > without
> > > > >> > > > > > >> > > > > > > > > > > > >> considering
> > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> these
> > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > potential
> evolutions
> > > may
> > > > >> limit
> > > > >> > > > > > >> flexibility.
> > > > >> > > > > > >> > > I'd
> > > > >> > > > > > >> > > > > > > suggest
> > > > >> > > > > > >> > > > > > > > > to
> > > > >> > > > > > >> > > > > > > > > > > > start
> > > > >> > > > > > >> > > > > > > > > > > > >> >> with
> > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> the
> > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > use case.
> > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> >
> > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > Yufei
> > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> >
> > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> >
> > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > On Fri, Feb 27,
> 2026
> > at
> > > > >> > 3:42 PM
> > > > >> > > > > Dmitri
> > > > >> > > > > > >> > > > > > Bourlatchkov <
> > > > >> > > > > > >> > > > > > > > > > > > >> >> dimas@apache.org>
> > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > wrote:
> > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> >
> > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > > Hi Anand,
> > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > >
> > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > > Sharing my
> view...
> > > > >> subject
> > > > >> > to
> > > > >> > > > > > >> discussion:
> > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > >
> > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > > 1. Adding non-IRC
> > > REST
> > > > >> API
> > > > >> > to
> > > > >> > > > > > Polaris
> > > > >> > > > > > >> is
> > > > >> > > > > > >> > > > > > perfectly
> > > > >> > > > > > >> > > > > > > > > fine.
> > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > >
> > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > > Figuring out
> > specific
> > > > >> > endpoint
> > > > >> > > > > URIs
> > > > >> > > > > > >> and
> > > > >> > > > > > >> > > > > payloads
> > > > >> > > > > > >> > > > > > > > might
> > > > >> > > > > > >> > > > > > > > > > > > require
> > > > >> > > > > > >> > > > > > > > > > > > >> a
> > > > >> > > > > > >> > > > > > > > > > > > >> >> few
> > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > > roundtrips, so
> > > opening a
> > > > >> > > separate
> > > > >> > > > > > >> thread
> > > > >> > > > > > >> > > for
> > > > >> > > > > > >> > > > > that
> > > > >> > > > > > >> > > > > > > > might
> > > > >> > > > > > >> > > > > > > > > > be
> > > > >> > > > > > >> > > > > > > > > > > > >> best.
> > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > > Contributors
> > commonly
> > > > >> create
> > > > >> > > > > Google
> > > > >> > > > > > >> Docs
> > > > >> > > > > > >> > > for
> > > > >> > > > > > >> > > > > new
> > > > >> > > > > > >> > > > > > > API
> > > > >> > > > > > >> > > > > > > > > > > > proposals
> > > > >> > > > > > >> > > > > > > > > > > > >> too
> > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> (they
> > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > > fairly easy to
> > > update as
> > > > >> the
> > > > >> > > email
> > > > >> > > > > > >> > > discussion
> > > > >> > > > > > >> > > > > > > > > > progresses).
> > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > >
> > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > > There was a
> > > suggestion to
> > > > >> > try
> > > > >> > > > > > Markdown
> > > > >> > > > > > >> > > (with
> > > > >> > > > > > >> > > > > PRs)
> > > > >> > > > > > >> > > > > > > for
> > > > >> > > > > > >> > > > > > > > > > > > proposals
> > > > >> > > > > > >> > > > > > > > > > > > >> >> [1]
> > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> ...
> > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > > feel free to give
> > it
> > > a
> > > > >> try
> > > > >> > if
> > > > >> > > you
> > > > >> > > > > > are
> > > > >> > > > > > >> > > > > comfortable
> > > > >> > > > > > >> > > > > > > > with
> > > > >> > > > > > >> > > > > > > > > > > that.
> > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > >
> > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > > 2. Could you
> > clarify
> > > > >> whether
> > > > >> > > you
> > > > >> > > > > > mean
> > > > >> > > > > > >> end
> > > > >> > > > > > >> > > > user
> > > > >> > > > > > >> > > > > > > > > utilities
> > > > >> > > > > > >> > > > > > > > > > or
> > > > >> > > > > > >> > > > > > > > > > > > >> admin
> > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> user
> > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > > utilities? In the
> > > latter
> > > > >> > case
> > > > >> > > > > those
> > > > >> > > > > > >> might
> > > > >> > > > > > >> > > be
> > > > >> > > > > > >> > > > > more
> > > > >> > > > > > >> > > > > > > > > > suitable
> > > > >> > > > > > >> > > > > > > > > > > > for
> > > > >> > > > > > >> > > > > > > > > > > > >> the
> > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> Admin
> > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > > CLI (java) not
> the
> > > Python
> > > > >> > CLI,
> > > > >> > > > > IMHO.
> > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > >
> > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > > Why would these
> > > > >> utilities be
> > > > >> > > > > common
> > > > >> > > > > > >> with
> > > > >> > > > > > >> > > > > events?
> > > > >> > > > > > >> > > > > > > > IMHO,
> > > > >> > > > > > >> > > > > > > > > > > event
> > > > >> > > > > > >> > > > > > > > > > > > >> use
> > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> cases
> > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > are
> > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > > distinct from
> > > scan/commit
> > > > >> > > metrics.
> > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > >
> > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > > 3. I'd prefer
> > > separating
> > > > >> > > metrics
> > > > >> > > > > > >> > > persistence
> > > > >> > > > > > >> > > > > from
> > > > >> > > > > > >> > > > > > > > > > MetaStore
> > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> persistence
> > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > at
> > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > > the code level,
> so
> > > that
> > > > >> they
> > > > >> > > could
> > > > >> > > > > > be
> > > > >> > > > > > >> > mixed
> > > > >> > > > > > >> > > > and
> > > > >> > > > > > >> > > > > > > > matched
> > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> independently.
> > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > The
> > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > > separate
> datasource
> > > > >> question
> > > > >> > > will
> > > > >> > > > > > >> become
> > > > >> > > > > > >> > a
> > > > >> > > > > > >> > > > > > > non-issue
> > > > >> > > > > > >> > > > > > > > > with
> > > > >> > > > > > >> > > > > > > > > > > > that
> > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> approach,
> > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > I
> > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > > guess.
> > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > >
> > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > > The rationale for
> > > > >> separating
> > > > >> > > scan
> > > > >> > > > > > >> metrics
> > > > >> > > > > > >> > > and
> > > > >> > > > > > >> > > > > > > > metastore
> > > > >> > > > > > >> > > > > > > > > > > > >> >> persistence
> > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> is
> > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > that
> > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > > "cascading
> deletes"
> > > > >> between
> > > > >> > > them
> > > > >> > > > > are
> > > > >> > > > > > >> > hardly
> > > > >> > > > > > >> > > > > ever
> > > > >> > > > > > >> > > > > > > > > > required.
> > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> Furthermore,
> > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > the
> > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > > data and query
> > > patterns
> > > > >> are
> > > > >> > > very
> > > > >> > > > > > >> > different
> > > > >> > > > > > >> > > so
> > > > >> > > > > > >> > > > > > > > different
> > > > >> > > > > > >> > > > > > > > > > > > >> >> technologies
> > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > might
> > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > > be beneficial in
> > each
> > > > >> case.
> > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > >
> > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > > [1]
> > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> >
> > > > >> > > > > > >> > > > > > > > > > > > >> >> >>>
> > > > >> > > > > > >> > > > > > > > > > > > >> >>
> > > > >> > > > > > >> > > > > > > > > > > > >>
> > > > >> > > > > > >> > > > > > > > > > > >
> > > > >> > > > > > >> > > > > > > > > > >
> > > > >> > > > > > >> > > > > > > > > >
> > > > >> > > > > > >> > > > > > > > >
> > > > >> > > > > > >> > > > > > > >
> > > > >> > > > > > >> > > > > > >
> > > > >> > > > > > >> > > > > >
> > > > >> > > > > > >> > > > >
> > > > >> > > > > > >> > > >
> > > > >> > > > > > >> > >
> > > > >> > > > > > >> >
> > > > >> > > > > > >>
> > > > >> > > > > >
> > > > >> > > > >
> > > > >> > >
> > > > >> >
> > > > >>
> > >
> >
> https://urldefense.com/v3/__https://lists.apache.org/thread/yto2wp982t43h1mqjwnslswhws5z47cy__;!!Iz9xO38YGHZK!5EuyFFkk3vhRWVIRvQAWBSQfpJkTMA9HxugzDwXmN0LPPqhEFxYkFRGVhtb8AqUwXtDh2OplcMnbMDHKxYDakNU$
> > > > >> > >> > > >> > > > > > > > > > > > >> >> >>> > >
> > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > > Cheers,
> > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > > Dmitri.
> > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > >
> > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > > On Fri, Feb 27,
> > 2026
> > > at
> > > > >> > > 6:19 PM
> > > > >> > > > > > Anand
> > > > >> > > > > > >> > Kumar
> > > > >> > > > > > >> > > > > > > Sankaran
> > > > >> > > > > > >> > > > > > > > > via
> > > > >> > > > > > >> > > > > > > > > > > dev
> > > > >> > > > > > >> > > > > > > > > > > > <
> > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > >
> > > dev@polaris.apache.org>
> > > > >> > > wrote:
> > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > >
> > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > > > Thanks all.
> This
> > > PR is
> > > > >> > > merged
> > > > >> > > > > now.
> > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > > >
> > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > > > Here are the
> > > follow-up
> > > > >> > > features
> > > > >> > > > > /
> > > > >> > > > > > >> work
> > > > >> > > > > > >> > > > > needed.
> > > > >> > > > > > >> > > > > > > > These
> > > > >> > > > > > >> > > > > > > > > > > were
> > > > >> > > > > > >> > > > > > > > > > > > >> all
> > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> part of
> > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > > the
> > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > > > merged PR at
> some
> > > > >> point in
> > > > >> > > time
> > > > >> > > > > > and
> > > > >> > > > > > >> > were
> > > > >> > > > > > >> > > > > > removed
> > > > >> > > > > > >> > > > > > > to
> > > > >> > > > > > >> > > > > > > > > > > reduce
> > > > >> > > > > > >> > > > > > > > > > > > >> >> scope.
> > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > > >
> > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > > > Please let me
> > know
> > > what
> > > > >> > you
> > > > >> > > > > think.
> > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > > >
> > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > > >
> > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > > >   1.  A REST
> API
> > to
> > > > >> > paginate
> > > > >> > > > > > through
> > > > >> > > > > > >> > > table
> > > > >> > > > > > >> > > > > > > metrics.
> > > > >> > > > > > >> > > > > > > > > > This
> > > > >> > > > > > >> > > > > > > > > > > > >> will be
> > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > non-IRC
> > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > > > standard
> > addition.
> > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > > >   2.  Utilities
> > for
> > > > >> > > managing old
> > > > >> > > > > > >> > records,
> > > > >> > > > > > >> > > > > > should
> > > > >> > > > > > >> > > > > > > be
> > > > >> > > > > > >> > > > > > > > > > > common
> > > > >> > > > > > >> > > > > > > > > > > > >> with
> > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> events.
> > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > > > There was some
> > > > >> discussion
> > > > >> > > that
> > > > >> > > > > it
> > > > >> > > > > > >> > belongs
> > > > >> > > > > > >> > > > to
> > > > >> > > > > > >> > > > > > the
> > > > >> > > > > > >> > > > > > > > CLI.
> > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > > >   3.  Separate
> > > > >> datasource
> > > > >> > > > > > (metrics,
> > > > >> > > > > > >> > > events,
> > > > >> > > > > > >> > > > > > even
> > > > >> > > > > > >> > > > > > > > > other
> > > > >> > > > > > >> > > > > > > > > > > > >> tables?).
> > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > > >
> > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > > >
> > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > > > Anything else?
> > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > > >
> > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > > > -
> > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > > > Anand
> > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > > >
> > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > > >
> > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > >
> > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> >
> > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> >
> > > > >> > > > > > >> > > > > > > > > > > > >> >> >>>
> > > > >> > > > > > >> > > > > > > > > > > > >> >> >>
> > > > >> > > > > > >> > > > > > > > > > > > >> >>
> > > > >> > > > > > >> > > > > > > > > > > > >> >
> > > > >> > > > > > >> > > > > > > > > > > > >>
> > > > >> > > > > > >> > > > > > > > > > > > >
> > > > >> > > > > > >> > > > > > > > > > > >
> > > > >> > > > > > >> > > > > > > > > > >
> > > > >> > > > > > >> > > > > > > > > >
> > > > >> > > > > > >> > > > > > > > >
> > > > >> > > > > > >> > > > > > > >
> > > > >> > > > > > >> > > > > > >
> > > > >> > > > > > >> > > > > >
> > > > >> > > > > > >> > > > >
> > > > >> > > > > > >> > > >
> > > > >> > > > > > >> > >
> > > > >> > > > > > >> >
> > > > >> > > > > > >>
> > > > >> > > > > > >
> > > > >> > > > > >
> > > > >> > > > >
> > > > >> > >
> > > > >> >
> > > > >> >
> > > > >>
> > > > >
> > >
> >
>

Re: Proposal for REST endpoints for table metrics and events

[Dmitri Bourlatchkov <di...@apache.org>]

Hi Yufei,

Here's the separate PR for Metrics-related grants:
https://github.com/apache/polaris/pull/5659

Please review it to unblock this contribution.

Thanks,
Dmitri.

On Wed, Sep 30, 2026 at 1:05 PM Yufei Gu <fl...@gmail.com> wrote:

> From the previous email, we agreed to split the REST change from the grant
> change. I think the PR is ready to go if we only keep the grant changes.
>
> Yufei
>
>
> On Tue, Sep 29, 2026 at 9:26 PM Jean-Baptiste Onofré <jb...@nanthrax.net>
> wrote:
>
> > Hi guys,
> >
> > I agree with Dmitri: +1 to merge PR #4115 on Oct 1 (tomorrow).
> >
> > From my side, all outstanding feedback has been addressed, including
> > the final documentation points raised by EJ regarding the empty-200
> > NoOp behavior and reconstructed payload description. The branch is
> > clean, up to date with main, and CI is green.
> >
> > Once this lands, we can follow up with PR #4756 for the JDBC query SPI
> > implementation.
> >
> > Regards
> > JB
> >
> > On Wed, Sep 30, 2026 at 1:53 AM Dmitri Bourlatchkov <di...@apache.org>
> > wrote:
> > >
> > > Hi All,
> > >
> > > Any remaining blockers on [4115]?
> > >
> > > Given that the review was very long, please restate any remaining
> > concerns
> > > (here or in GH).
> > >
> > > Otherwise, I propose merging [4115] on Oct 1.
> > >
> > > [4115] https://github.com/apache/polaris/pull/4115
> > >
> > > Thanks,
> > > Dmitri.
> > >
> > > On Wed, Sep 23, 2026 at 7:53 PM Dmitri Bourlatchkov <di...@apache.org>
> > > wrote:
> > >
> > > > Hi EJ,
> > > >
> > > > Thanks for the meeting recap!
> > > >
> > > > All:
> > > >
> > > > Worth noting: PR [4115] also contains Management API changes that
> were
> > not
> > > > anticipated initially and did not get an API change review until now.
> > These
> > > > are small changes to add new grant types for metrics RBAC..
> > > >
> > > > I've reviewed and approved PR [4115] in GH.
> > > >
> > > > [4115] https://github.com/apache/polaris/pull/4115
> > > >
> > > > Cheers,
> > > > Dmitri.
> > > >
> > > > On Wed, Aug 12, 2026 at 5:21 PM EJ Wang <
> > ej.wang.dev.interest@gmail.com>
> > > > wrote:
> > > >
> > > >> Hi folks,
> > > >>
> > > >> Notes from today's metrics architecture sync are in the usual doc:
> > > >>
> > > >>
> >
> https://docs.google.com/document/d/100h7c4damrUzVuquYbBHM0EvA4LSWuW2IT2dN_7nYVA/edit?pli=1&tab=t.18iv1m7p6i6b
> > > >>
> > > >> Highlights:
> > > >>
> > > >> Module placement. core/ now holds the entity data model only. A new
> > spi/
> > > >> takes every other shared contract: feature SPIs, feature non-SPI
> > > >> contracts,
> > > >> substrate SPIs, substrate non-SPI contracts, and contract-required
> > types.
> > > >> That settles the question PR #5204 reopened, by placing into a new
> SPI
> > > >> module. Note that it changes slightly towards the referenced WIP RFC
> > [1]
> > > >> recommendations: what proposed to place in core/, now should go to
> > spi/
> > > >> except for the entity data model.
> > > >>
> > > >> Metrics PRs. #5068 merged. #5204 needs reshaping onto the split.
> > #4115 and
> > > >> #4756 stay open with action on the author to follow up on review
> > comments.
> > > >>
> > > >> [1]
> > > >>
> > > >>
> >
> https://docs.google.com/document/d/1mj3vGpix9zTDTFFJKtWT5ZxI3PTGI1YNb5xFqyWrOkc/edit?tab=t.0
> > > >>
> > > >> On Wed, Jul 15, 2026 at 11:01 AM Anand Kumar Sankaran via dev <
> > > >> dev@polaris.apache.org> wrote:
> > > >>
> > > >> > Thanks EJ.  My plan is to introduce PR0 in a new branch / PR and
> > > >> repurpose
> > > >> > #4115 and #4756 for PR1 and PR2 to maintain all the valuable
> > discussions
> > > >> > that happened there.
> > > >> >
> > > >> > From: EJ Wang <ej...@gmail.com>
> > > >> > Date: Wednesday, July 15, 2026 at 10:41 AM
> > > >> > To: dev@polaris.apache.org <de...@polaris.apache.org>
> > > >> > Subject: Re: Proposal for REST endpoints for table metrics and
> > events
> > > >> >
> > > >> > Hi folks, Thanks again for the discussion today. I updated the
> sync
> > doc
> > > >> > with notes from today's metrics sync: https: //urldefense.
> > > >> com/v3/__https:
> > > >> > //docs. google.
> > > >> >
> > > >>
> >
> com/document/d/100h7c4damrUzVuquYbBHM0EvA4LSWuW2IT2dN_7nYVA/edit?pli=1&tab=t.
> > > >> >
> > > >>
> >
> 2ahd7wze5f__;!!Iz9xO38YGHZK!4rnlVIqBQQQ6xpGe1PMV58edOS_cGrKoqdfhw4qSp_JQCavvEjaxXk0OwGPnxyEcO60AzKQK_XDtp7H3DdkFXcrXJwAz1g0$
> > > >> >
> > > >> >
> > > >> > Hi folks,
> > > >> >
> > > >> > Thanks again for the discussion today. I updated the sync doc with
> > notes
> > > >> > from today's metrics sync:
> > > >> >
> > > >> >
> > > >>
> >
> https://urldefense.com/v3/__https://docs.google.com/document/d/100h7c4damrUzVuquYbBHM0EvA4LSWuW2IT2dN_7nYVA/edit?pli=1&tab=t.2ahd7wze5f__;!!Iz9xO38YGHZK!4rnlVIqBQQQ6xpGe1PMV58edOS_cGrKoqdfhw4qSp_JQCavvEjaxXk0OwGPnxyEcO60AzKQK_XDtp7H3DdkFXcrXJwAz1g0$
> > > >> >
> > > >> > See highlights below:
> > > >> >
> > > >> > In previous meetings, EJ proposed a two prs framing to push
> forward
> > the
> > > >> > metrics query work in polaris. Dmitri pointed out a subtle point
> > that
> > > >> pr2
> > > >> > is currently also trying to do the persistence related code
> > placement
> > > >> > refactor. For the sake of straightforward review process, EJ,
> > Dmitri and
> > > >> > Anand (the author) agreed to separate the persistence refactoring
> > into
> > > >> pr0,
> > > >> > and reframing the pr scopes as follow:
> > > >> >
> > > >> >    1.
> > > >> >
> > > >> >    Pr0: refactor irc metrics spi
> > > >> >    1.
> > > >> >
> > > >> >       SPI should stay in core/ with metric type enveloped
> > > >> >       2.
> > > >> >
> > > >> >       default spi impl, no-op, in extensions/ - default selection
> at
> > > >> >       runtime/
> > > >> >       3.
> > > >> >
> > > >> >       move metrics persistence and correspondings to extensions/
> as
> > an
> > > >> >       alternative SPI impl
> > > >> >       2.
> > > >> >
> > > >> >    Pr1: rest api for query
> > > >> >
> > > >> >
> > > >>
> >
> https://urldefense.com/v3/__https://github.com/apache/polaris/pull/4115/changes__;!!Iz9xO38YGHZK!4rnlVIqBQQQ6xpGe1PMV58edOS_cGrKoqdfhw4qSp_JQCavvEjaxXk0OwGPnxyEcO60AzKQK_XDtp7H3DdkFXcrXFCbEbRk$
> > > >> >    1.
> > > >> >
> > > >> >       rest api spec in extensions/ as optional spec
> > > >> >       2.
> > > >> >
> > > >> >       SPI in extensions/ as optional spec spi
> > > >> >       3.
> > > >> >
> > > >> >       default spi impl in extensions/ no-op
> > > >> >       4.
> > > >> >
> > > >> >       handler related in runtime/ - thin adaptor from http shape
> to
> > spi
> > > >> >       shape
> > > >> >       3.
> > > >> >
> > > >> >    Pr2: jdbc impl for query
> > > >> >
> > > >>
> >
> https://urldefense.com/v3/__https://github.com/apache/polaris/pull/4756__;!!Iz9xO38YGHZK!4rnlVIqBQQQ6xpGe1PMV58edOS_cGrKoqdfhw4qSp_JQCavvEjaxXk0OwGPnxyEcO60AzKQK_XDtp7H3DdkFXcrXRVMBzqE$
> > > >> >    1.
> > > >> >
> > > >> >       alternative spi impl in extensions/ -> query with jdbc
> > > >> >
> > > >> >
> > > >> > Thanks,
> > > >> > -ej
> > > >> >
> > > >> > On Wed, Jul 1, 2026 at 8:01 AM Alexandre Dutra <adutra@apache.org
> >
> > > >> wrote:
> > > >> >
> > > >> > > Hi all,
> > > >> > >
> > > >> > > > While the current implementation stores metrics in the
> > metastore,
> > > >> other
> > > >> > > deployments
> > > >> > > may choose a KV store, a dedicated metrics warehouse, or another
> > > >> backend
> > > >> > > entirely. In those cases, exposing a Polaris REST API over a
> > specific
> > > >> > > schema feels like coupling the API to an implementation detail.
> > > >> > >
> > > >> > > I wouldn't say to an implementation detail, but rather to a
> > particular
> > > >> > > setup / configuration. I think that's fine. If you don't store
> > > >> > > metrics, you don't read metrics. Similar situations will
> certainly
> > > >> > > happen in the future, e.g. if we ever introduce an API to read
> > events,
> > > >> > > that would only work if events are being persisted in the first
> > place.
> > > >> > > Again I think that's a fair trade-off.
> > > >> > >
> > > >> > > Thanks,
> > > >> > > Alex
> > > >> > >
> > > >> > > On Wed, Jul 1, 2026 at 7:23 AM Dmitri Bourlatchkov <
> > dimas@apache.org>
> > > >> > > wrote:
> > > >> > > >
> > > >> > > > Hi Yufei,
> > > >> > > >
> > > >> > > > From my POV, it is fine for Polaris to have multiple REST API
> > > >> surfaces
> > > >> > > with
> > > >> > > > different features and different degrees of "readiness". Each
> > API
> > > >> can
> > > >> > > > evolve fairly independently. With proper modularization (which
> > Anand
> > > >> > > did),
> > > >> > > > changes to the main Polaris code and these extra modules
> should
> > > >> > normally
> > > >> > > be
> > > >> > > > non-intrusive to each other.
> > > >> > > >
> > > >> > > > I assume Anand is going to be the first user of this API.
> > However,
> > > >> if
> > > >> > we
> > > >> > > do
> > > >> > > > not merge it we'll never know whether other users are
> > interested. I
> > > >> do
> > > >> > > not
> > > >> > > > think many end users read the dev ML, but they certainly read
> > the
> > > >> docs,
> > > >> > > > code and release notes.
> > > >> > > >
> > > >> > > > There is no risk for Polaris as a project in exposing this
> > > >> experimental
> > > >> > > > API. If usage dwindles over time, we can remove it. If it
> gains
> > a
> > > >> > > > considerable user / contributor base, we will promote it to a
> > > >> > > "first-class"
> > > >> > > > (non-beta) API.
> > > >> > > >
> > > >> > > > Cheers,
> > > >> > > > Dmitri.
> > > >> > > >
> > > >> > > > On Tue, Jun 30, 2026 at 7:53 PM Yufei Gu <
> flyrain000@gmail.com>
> > > >> wrote:
> > > >> > > >
> > > >> > > > > Hi all,
> > > >> > > > >
> > > >> > > > > Thanks for driving this work forward. Building this as a POC
> > makes
> > > >> > > perfect
> > > >> > > > > sense if the goal is to validate the idea and gather
> feedback.
> > > >> Should
> > > >> > > we
> > > >> > > > > merge a POC into the main branch before agreeing on the
> > overall
> > > >> > > direction?
> > > >> > > > > To me, that feels like a separate discussion from whether
> the
> > > >> > > > > implementation itself looks reasonable.
> > > >> > > > >
> > > >> > > > > One concern that I don't think has been fully addressed is
> > > >> whether we
> > > >> > > need
> > > >> > > > > to expose metrics consumption through a Polaris REST
> endpoint
> > at
> > > >> all.
> > > >> > > > >
> > > >> > > > > My concern is less about the API itself and more about the
> > > >> boundary
> > > >> > > Polaris
> > > >> > > > > should own. The proposed REST API is tied to a particular
> > > >> persistence
> > > >> > > > > schema, but that schema is just one possible implementation.
> > While
> > > >> > the
> > > >> > > > > current implementation stores metrics in the metastore,
> other
> > > >> > > deployments
> > > >> > > > > may choose a KV store, a dedicated metrics warehouse, or
> > another
> > > >> > > backend
> > > >> > > > > entirely. In those cases, exposing a Polaris REST API over a
> > > >> specific
> > > >> > > > > schema feels like coupling the API to an implementation
> > detail.
> > > >> > > > >
> > > >> > > > > Polaris isn't intended to become a full metrics system. I'd
> > > >> expect it
> > > >> > > to
> > > >> > > > > focus on producing metrics and let deployments choose the
> most
> > > >> > > appropriate
> > > >> > > > > storage and consumption model for their environment.
> > > >> > > > >
> > > >> > > > > I'm not opposed to exploring this direction, but we should
> > first
> > > >> > align
> > > >> > > on
> > > >> > > > > whether exposing a REST API through Polaris is the right
> long
> > term
> > > >> > > approach
> > > >> > > > > before merging it into the main branch.
> > > >> > > > >
> > > >> > > > > Thanks,
> > > >> > > > > Yufei
> > > >> > > > >
> > > >> > > > > Yufei
> > > >> > > > >
> > > >> > > > >
> > > >> > > > > On Mon, Jun 29, 2026 at 9:27 AM Dmitri Bourlatchkov <
> > > >> > dimas@apache.org>
> > > >> > > > > wrote:
> > > >> > > > >
> > > >> > > > > > Hi All,
> > > >> > > > > >
> > > >> > > > > > It looks like Anand addressed all action items in [4115].
> > > >> > > > > >
> > > >> > > > > > Please review the latest state of the code. The PR LGTM
> and
> > I
> > > >> > > approved in
> > > >> > > > > > GH from my side.
> > > >> > > > > >
> > > >> > > > > > [4115]
> > > >> >
> > > >>
> >
> https://urldefense.com/v3/__https://github.com/apache/polaris/pull/4115__;!!Iz9xO38YGHZK!4rnlVIqBQQQ6xpGe1PMV58edOS_cGrKoqdfhw4qSp_JQCavvEjaxXk0OwGPnxyEcO60AzKQK_XDtp7H3DdkFXcrXtFRscKs$
> > > >> > > > > >
> > > >> > > > > > Thanks,
> > > >> > > > > > Dmitri.
> > > >> > > > > >
> > > >> > > > > > On Thu, Jun 25, 2026 at 1:56 PM Dmitri Bourlatchkov <
> > > >> > > dimas@apache.org>
> > > >> > > > > > wrote:
> > > >> > > > > >
> > > >> > > > > > > Hi All,
> > > >> > > > > > >
> > > >> > > > > > > Recap of the discussion in today's Community Sync:
> > > >> > > > > > >
> > > >> > > > > > > * The modular approach for Metrics SPI(s) and
> > implementation
> > > >> > seems
> > > >> > > to
> > > >> > > > > > have
> > > >> > > > > > > community consensus (please comment if you still have
> > > >> concerns)
> > > >> > > > > > >
> > > >> > > > > > > * What modules / feature flags are enabled in Apache
> > Polaris
> > > >> > > images by
> > > >> > > > > > > default will be discussed later (when the feature is
> > close to
> > > >> > > > > > completion).
> > > >> > > > > > >
> > > >> > > > > > > In any case, users will be able to turn the full feature
> > > >> on/off
> > > >> > at
> > > >> > > > > their
> > > >> > > > > > > discretion, regardless of the default flag values.
> > > >> > > > > > >
> > > >> > > > > > > * It is generally preferable to have smaller PRs and
> > introduce
> > > >> > new
> > > >> > > > > > > features gradually to simplify reviews.
> > > >> > > > > > >
> > > >> > > > > > > What this means for the Metrics API, IMHO is:
> > > >> > > > > > >
> > > >> > > > > > > a) The scope of PR 4115 is probably ok as it stands
> right
> > > >> now. If
> > > >> > > > > someone
> > > >> > > > > > > wants to propose splitting it further, please comment.
> > > >> > > > > > >
> > > >> > > > > > > b) After 4115 is merged, Metrics Persistence PRs will
> > follow,
> > > >> > with
> > > >> > > > > > smaller
> > > >> > > > > > > scope per PR as appropriate for the nature of the code
> > > >> changes.
> > > >> > > > > > >
> > > >> > > > > > > * The major concern was about exposing the new Metrics
> > REST
> > > >> API
> > > >> > to
> > > >> > > end
> > > >> > > > > > > users. The nature of this concern was about avoiding the
> > > >> (false)
> > > >> > > > > > impression
> > > >> > > > > > > that the new API is a firm and adopted Polaris
> > specification.
> > > >> > > > > > >
> > > >> > > > > > > I think this concern is reasonable.
> > > >> > > > > > >
> > > >> > > > > > > I propose that Anand update the documentation sections
> of
> > PR
> > > >> 4115
> > > >> > > to
> > > >> > > > > > > emphasise that the API is currently an experimental /
> POC
> > API
> > > >> at
> > > >> > > this
> > > >> > > > > > time
> > > >> > > > > > > and is subject to change (including breaking changes). I
> > > >> believe
> > > >> > it
> > > >> > > > > > already
> > > >> > > > > > > has the "beta" label. So this request is mostly for doc
> > > >> changes
> > > >> > to
> > > >> > > > > > clarify
> > > >> > > > > > > user expectations, since the "beta" label alone may not
> be
> > > >> > > interpreted
> > > >> > > > > > the
> > > >> > > > > > > same way by all users.
> > > >> > > > > > >
> > > >> > > > > > > I hope this update will resolve all remaining concerns
> > with
> > > >> 4115
> > > >> > > and
> > > >> > > > > > allow
> > > >> > > > > > > for it to be merged so that the community will be able
> to
> > > >> proceed
> > > >> > > to
> > > >> > > > > > > follow-up PRs (as noted above).
> > > >> > > > > > >
> > > >> > > > > > > Please add/correct me if I missed anything relevant.
> > > >> > > > > > >
> > > >> > > > > > > Thanks,
> > > >> > > > > > > Dmitri.
> > > >> > > > > > >
> > > >> > > > > > > On Wed, Jun 24, 2026 at 8:11 PM Yufei Gu <
> > > >> flyrain000@gmail.com>
> > > >> > > wrote:
> > > >> > > > > > >
> > > >> > > > > > >> I do not think we have agreement on the query REST APIs
> > yet.
> > > >> > > > > > >>
> > > >> > > > > > >> In particular, the current query API shape seems highly
> > > >> coupled
> > > >> > > to the
> > > >> > > > > > >> *example[1]* metrics in the Iceberg REST spec. I do not
> > think
> > > >> > > that is
> > > >> > > > > a
> > > >> > > > > > >> good way to design a REST API contract. For example,
> > fields
> > > >> like
> > > >> > > > > > >> resultDataFiles, resultDeleteFiles, totalFileSizeBytes,
> > > >> > > > > > >> totalDataManifests,
> > > >> > > > > > >> totalDeleteManifests, and scannedDataManifests are very
> > > >> specific
> > > >> > > to
> > > >> > > > > one
> > > >> > > > > > >> style of scan metric. That makes the API look more
> like a
> > > >> direct
> > > >> > > > > > >> projection
> > > >> > > > > > >> of the current example payload than a general query
> > model.
> > > >> > > > > > >>
> > > >> > > > > > >> My preference is that Polaris should not define or
> > strongly
> > > >> > opine
> > > >> > > on
> > > >> > > > > the
> > > >> > > > > > >> Iceberg scan or commit metrics query REST API at this
> > point.
> > > >> > > Polaris
> > > >> > > > > can
> > > >> > > > > > >> resolve and authorize the table, accept the metrics
> > report,
> > > >> and
> > > >> > > > > delegate
> > > >> > > > > > >> it
> > > >> > > > > > >> to the configured reporting path. Durable storage,
> > filtering,
> > > >> > > > > > dashboards,
> > > >> > > > > > >> and query APIs should remain downstream impl. choices
> > unless
> > > >> and
> > > >> > > until
> > > >> > > > > > the
> > > >> > > > > > >> Iceberg community standardizes a query contract, which
> I
> > > >> doubt
> > > >> > > will
> > > >> > > > > > >> happen.
> > > >> > > > > > >>
> > > >> > > > > > >> So I do not think the intake path has to be blocked,
> but
> > I
> > > >> would
> > > >> > > > > prefer
> > > >> > > > > > >> not
> > > >> > > > > > >> to merge a Polaris specific query API that may be hard
> to
> > > >> evolve
> > > >> > > > > later.
> > > >> > > > > > >>
> > > >> > > > > > >> 1.
> > > >> > > > > > >>
> > > >> > > > > > >>
> > > >> > > > > >
> > > >> > > > >
> > > >> > >
> > > >> >
> > > >>
> >
> https://urldefense.com/v3/__https://github.com/apache/iceberg/blob/2a6c556c0c883c04c6d3fbd68e6aeda36b91e0aa/open-api/rest-catalog-open-api.yaml*L4152__;Iw!!Iz9xO38YGHZK!4rnlVIqBQQQ6xpGe1PMV58edOS_cGrKoqdfhw4qSp_JQCavvEjaxXk0OwGPnxyEcO60AzKQK_XDtp7H3DdkFXcrXCDsqYGo$
> > > >> > > > > > >>
> > > >> > > > > > >> Yufei
> > > >> > > > > > >>
> > > >> > > > > > >>
> > > >> > > > > > >> On Tue, Jun 23, 2026 at 9:30 AM Dmitri Bourlatchkov <
> > > >> > > dimas@apache.org
> > > >> > > > > >
> > > >> > > > > > >> wrote:
> > > >> > > > > > >>
> > > >> > > > > > >> > Hi All,
> > > >> > > > > > >> >
> > > >> > > > > > >> > So, what are the blockers for merging [4115], if any?
> > > >> > > > > > >> >
> > > >> > > > > > >> > [4115]
> > > >> >
> > > >>
> >
> https://urldefense.com/v3/__https://github.com/apache/polaris/pull/4115__;!!Iz9xO38YGHZK!4rnlVIqBQQQ6xpGe1PMV58edOS_cGrKoqdfhw4qSp_JQCavvEjaxXk0OwGPnxyEcO60AzKQK_XDtp7H3DdkFXcrXtFRscKs$
> > > >> > > > > > >> >
> > > >> > > > > > >> > Thanks,
> > > >> > > > > > >> > Dmitri.
> > > >> > > > > > >> >
> > > >> > > > > > >> > On Wed, Jun 17, 2026 at 5:28 PM EJ Wang <
> > > >> > > > > > ej.wang.dev.interest@gmail.com
> > > >> > > > > > >> >
> > > >> > > > > > >> > wrote:
> > > >> > > > > > >> >
> > > >> > > > > > >> > > Hi folks,
> > > >> > > > > > >> > >
> > > >> > > > > > >> > > Thanks again for the discussion today. I updated
> the
> > sync
> > > >> > doc
> > > >> > > with
> > > >> > > > > > >> notes
> > > >> > > > > > >> > > from today's metrics sync:
> > > >> > > > > > >> > >
> > > >> > > > > > >> > >
> > > >> > > > > > >> > >
> > > >> > > > > > >> >
> > > >> > > > > > >>
> > > >> > > > > >
> > > >> > > > >
> > > >> > >
> > > >> >
> > > >>
> >
> https://urldefense.com/v3/__https://docs.google.com/document/d/100h7c4damrUzVuquYbBHM0EvA4LSWuW2IT2dN_7nYVA/edit?tab=t.ezk0rgdx0c6m__;!!Iz9xO38YGHZK!4rnlVIqBQQQ6xpGe1PMV58edOS_cGrKoqdfhw4qSp_JQCavvEjaxXk0OwGPnxyEcO60AzKQK_XDtp7H3DdkFXcrX83IZkm0$
> > > >> > > > > > >> > >
> > > >> > > > > > >> > > A few highlights:
> > > >> > > > > > >> > >
> > > >> > > > > > >> > >    - No objection to keeping the default metrics
> > behavior
> > > >> > > small:
> > > >> > > > > > >> > >    no-op/log-only is enough as the built-in
> default.
> > > >> > > > > > >> > >    - Durable storage, event forwarding, external
> > queues,
> > > >> > > > > dashboards,
> > > >> > > > > > >> and
> > > >> > > > > > >> > >    custom filtering should be implementation
> choices
> > > >> behind
> > > >> > > the
> > > >> > > > > > >> metrics
> > > >> > > > > > >> > >    reporting path.
> > > >> > > > > > >> > >    - For PR #4115, the REST layer should
> > > >> > resolve/authz/accept
> > > >> > > the
> > > >> > > > > > >> report,
> > > >> > > > > > >> > >    then delegate to the selected metrics
> > implementation.
> > > >> It
> > > >> > > should
> > > >> > > > > > not
> > > >> > > > > > >> > own
> > > >> > > > > > >> > >    durable storage semantics.
> > > >> > > > > > >> > >    - The rough boundary we discussed is: spec owns
> > the
> > > >> wire
> > > >> > > > > > contract,
> > > >> > > > > > >> > >    runtime owns framework wiring, API/contract
> > modules
> > > >> own
> > > >> > > > > > >> > provider-facing
> > > >> > > > > > >> > >    contracts, and extensions own replaceable
> > > >> > implementations.
> > > >> > > > > > >> > >    - OpenLineage has a similar path-naming
> question.
> > We
> > > >> > should
> > > >> > > > > avoid
> > > >> > > > > > >> > >    occupying a generic lineage path too early if we
> > want
> > > >> > room
> > > >> > > for
> > > >> > > > > > >> other
> > > >> > > > > > >> > >    lineage systems later.
> > > >> > > > > > >> > >
> > > >> > > > > > >> > > I believe the boundary rules previewed in sync
> > meeting
> > > >> are
> > > >> > > useful
> > > >> > > > > > for
> > > >> > > > > > >> > > reviewing PR #4115, but it is not
> > > >> > > > > > >> > > a final module-layout proposal.
> > > >> > > > > > >> > >
> > > >> > > > > > >> > > For metrics specifically, I think the target
> remains:
> > > >> > > > > > >> > >    runtime
> > > >> > > > > > >> > >      -> metrics reporting/emitting contract
> > > >> > > > > > >> > >      -> default implementation
> > > >> > > > > > >> > >           -> lower implementation details, if
> needed
> > > >> > > > > > >> > >
> > > >> > > > > > >> > > That keeps the default simple, while allowing
> durable
> > > >> JDBC,
> > > >> > > > > > >> event-backed,
> > > >> > > > > > >> > > or external-queue implementations to be added
> > separately.
> > > >> > > > > > >> > >
> > > >> > > > > > >> > > We also touched on the split between Iceberg
> metrics
> > > >> > emitting
> > > >> > > and
> > > >> > > > > > >> > > Polaris-owned metrics querying. The querying side
> can
> > > >> keep
> > > >> > > > > evolving,
> > > >> > > > > > >> > > especially for dashboard or generic-table use
> cases,
> > but
> > > >> I
> > > >> > do
> > > >> > > not
> > > >> > > > > > >> think
> > > >> > > > > > >> > > that needs to block the intake path in this PR.
> > > >> > > > > > >> > >
> > > >> > > > > > >> > > Thanks,
> > > >> > > > > > >> > > -ej
> > > >> > > > > > >> > >
> > > >> > > > > > >> > > On Tue, Jun 16, 2026 at 2:17 PM Yufei Gu <
> > > >> > > flyrain000@gmail.com>
> > > >> > > > > > >> wrote:
> > > >> > > > > > >> > >
> > > >> > > > > > >> > > > Thanks for chiming in, EJ. Agreed that the
> default
> > > >> battery
> > > >> > > > > should
> > > >> > > > > > >> stay
> > > >> > > > > > >> > > > small. I'm leaning toward not including metrics
> > > >> > persistence
> > > >> > > in
> > > >> > > > > the
> > > >> > > > > > >> > > default
> > > >> > > > > > >> > > > battery.
> > > >> > > > > > >> > > >
> > > >> > > > > > >> > > > Yufei
> > > >> > > > > > >> > > >
> > > >> > > > > > >> > > >
> > > >> > > > > > >> > > > On Thu, Jun 11, 2026 at 8:30 PM EJ Wang <
> > > >> > > > > > >> > ej.wang.dev.interest@gmail.com>
> > > >> > > > > > >> > > > wrote:
> > > >> > > > > > >> > > >
> > > >> > > > > > >> > > > > Hi Yufei,
> > > >> > > > > > >> > > > >
> > > >> > > > > > >> > > > > Thanks for connecting this back to the REST
> > endpoint
> > > >> > > > > proposal. I
> > > >> > > > > > >> > > replied
> > > >> > > > > > >> > > > > with the fuller version on the event-forwarding
> > > >> thread,
> > > >> > > but
> > > >> > > > > > >> wanted to
> > > >> > > > > > >> > > add
> > > >> > > > > > >> > > > > the shorter version here too.
> > > >> > > > > > >> > > > >
> > > >> > > > > > >> > > > > I agree that metrics reporting/emitting is the
> > right
> > > >> > > > > conceptual
> > > >> > > > > > >> > > boundary,
> > > >> > > > > > >> > > > > and I also think the event/listener path is a
> > good
> > > >> > > > > > implementation
> > > >> > > > > > >> > > > direction
> > > >> > > > > > >> > > > > to explore. The distinction I want to keep
> clear
> > is
> > > >> > > default
> > > >> > > > > > >> battery
> > > >> > > > > > >> > vs
> > > >> > > > > > >> > > > > extension implementation.
> > > >> > > > > > >> > > > >
> > > >> > > > > > >> > > > > For the REST/API side, I would keep the
> semantics
> > > >> > narrow:
> > > >> > > > > > Polaris
> > > >> > > > > > >> > > > resolves
> > > >> > > > > > >> > > > > and authorizes the table, accepts the Iceberg
> > > >> > scan/commit
> > > >> > > > > > metrics
> > > >> > > > > > >> > > report
> > > >> > > > > > >> > > > > into the configured reporting/emitting path,
> and
> > > >> returns
> > > >> > > 204.
> > > >> > > > > > That
> > > >> > > > > > >> > 204
> > > >> > > > > > >> > > > > should mean Polaris accepted the report into
> the
> > > >> > ingestion
> > > >> > > > > path;
> > > >> > > > > > >> it
> > > >> > > > > > >> > > > should
> > > >> > > > > > >> > > > > not imply durable storage.
> > > >> > > > > > >> > > > >
> > > >> > > > > > >> > > > > Durable storage, event forwarding, external
> > telemetry
> > > >> > > routing,
> > > >> > > > > > >> > > filtering,
> > > >> > > > > > >> > > > > and retention should sit behind that
> > > >> reporting/emitting
> > > >> > > > > boundary
> > > >> > > > > > >> as
> > > >> > > > > > >> > > > > implementation-layer behavior. A durable JDBC
> > path
> > > >> can
> > > >> > be
> > > >> > > one
> > > >> > > > > > >> named
> > > >> > > > > > >> > > > > extension implementation. An event/listener
> > > >> forwarding
> > > >> > > path
> > > >> > > > > can
> > > >> > > > > > be
> > > >> > > > > > >> > > > another
> > > >> > > > > > >> > > > > named extension implementation.
> > > >> > > > > > >> > > > >
> > > >> > > > > > >> > > > > So I am +1 on the event-forwarding idea as a
> > > >> non-default
> > > >> > > > > > extension
> > > >> > > > > > >> > > > > implementation. I just would not make it the
> > default
> > > >> > > battery
> > > >> > > > > or
> > > >> > > > > > >> the
> > > >> > > > > > >> > > core
> > > >> > > > > > >> > > > > API shape. The default battery can stay small
> and
> > > >> safe,
> > > >> > > while
> > > >> > > > > > >> > > deployments
> > > >> > > > > > >> > > > > that need forwarding, dashboards, or durable
> > storage
> > > >> can
> > > >> > > opt
> > > >> > > > > > into
> > > >> > > > > > >> the
> > > >> > > > > > >> > > > > implementation that matches their operational
> > model.
> > > >> > > > > > >> > > > >
> > > >> > > > > > >> > > > > Thanks,
> > > >> > > > > > >> > > > > -ej
> > > >> > > > > > >> > > > >
> > > >> > > > > > >> > > > > On Wed, Jun 10, 2026 at 11:43 AM Yufei Gu <
> > > >> > > > > flyrain000@gmail.com
> > > >> > > > > > >
> > > >> > > > > > >> > > wrote:
> > > >> > > > > > >> > > > >
> > > >> > > > > > >> > > > > > Thanks EJ. I agree that the
> reporting/emitting
> > > >> > boundary
> > > >> > > > > feels
> > > >> > > > > > >> like
> > > >> > > > > > >> > > the
> > > >> > > > > > >> > > > > > right SPI boundary, but I wonder if we can
> > simplify
> > > >> > this
> > > >> > > > > even
> > > >> > > > > > >> > > further.
> > > >> > > > > > >> > > > > >
> > > >> > > > > > >> > > > > > Iceberg scan and commit metrics look very
> > similar
> > > >> to
> > > >> > > events
> > > >> > > > > to
> > > >> > > > > > >> me.
> > > >> > > > > > >> > > They
> > > >> > > > > > >> > > > > are
> > > >> > > > > > >> > > > > > append only, asynchronously consumed, often
> > > >> forwarded
> > > >> > to
> > > >> > > > > > >> external
> > > >> > > > > > >> > > > > systems,
> > > >> > > > > > >> > > > > > and have similar retention and cleanup
> > > >> requirements.
> > > >> > > More
> > > >> > > > > > >> details
> > > >> > > > > > >> > can
> > > >> > > > > > >> > > > be
> > > >> > > > > > >> > > > > > found in this thread [1]. In that case,
> > Polaris may
> > > >> > only
> > > >> > > > > need
> > > >> > > > > > to
> > > >> > > > > > >> > > accept
> > > >> > > > > > >> > > > > the
> > > >> > > > > > >> > > > > > report and emit an event through the existing
> > event
> > > >> > > > > framework.
> > > >> > > > > > >> > > > Filtering,
> > > >> > > > > > >> > > > > > async delivery, custom sinks, and retention
> > > >> mechanisms
> > > >> > > could
> > > >> > > > > > >> then
> > > >> > > > > > >> > be
> > > >> > > > > > >> > > > > shared
> > > >> > > > > > >> > > > > > instead of introducing a separate metrics
> > specific
> > > >> > > extension
> > > >> > > > > > >> layer
> > > >> > > > > > >> > > and
> > > >> > > > > > >> > > > > SPI.
> > > >> > > > > > >> > > > > >
> > > >> > > > > > >> > > > > > The main requirement I still see is querying
> > > >> metrics
> > > >> > > through
> > > >> > > > > > >> > Polaris.
> > > >> > > > > > >> > > > If
> > > >> > > > > > >> > > > > we
> > > >> > > > > > >> > > > > > want that battery included experience, we
> could
> > > >> work
> > > >> > on
> > > >> > > a
> > > >> > > > > > >> > persistence
> > > >> > > > > > >> > > > > > listener implementation that selectively
> saves
> > > >> metrics
> > > >> > > and
> > > >> > > > > > >> expose
> > > >> > > > > > >> > it
> > > >> > > > > > >> > > > via
> > > >> > > > > > >> > > > > > the IRC event endpoint which seems pretty
> > close in
> > > >> > > Iceberg
> > > >> > > > > > >> > > > > > community. Another option I like more is to
> > allow
> > > >> > users
> > > >> > > to
> > > >> > > > > > >> develop
> > > >> > > > > > >> > > > their
> > > >> > > > > > >> > > > > > own listener, so they can persist scan/commit
> > > >> metrics
> > > >> > > to any
> > > >> > > > > > >> > systems
> > > >> > > > > > >> > > > they
> > > >> > > > > > >> > > > > > prefer. In general, that feels like an
> > > >> implementation
> > > >> > > choice
> > > >> > > > > > on
> > > >> > > > > > >> top
> > > >> > > > > > >> > > of
> > > >> > > > > > >> > > > > the
> > > >> > > > > > >> > > > > > event framework rather than something that
> > needs to
> > > >> > > shape
> > > >> > > > > the
> > > >> > > > > > >> core
> > > >> > > > > > >> > > > > > architecture.
> > > >> > > > > > >> > > > > >
> > > >> > > > > > >> > > > > > 1.
> > > >> > > > > > >> >
> > > >> > >
> > > >> >
> > > >>
> >
> https://urldefense.com/v3/__https://lists.apache.org/thread/x9j8nscvy8hq61tyn01mj8yp6n9of0kp__;!!Iz9xO38YGHZK!4rnlVIqBQQQ6xpGe1PMV58edOS_cGrKoqdfhw4qSp_JQCavvEjaxXk0OwGPnxyEcO60AzKQK_XDtp7H3DdkFXcrXGcGDgks$
> > > >> > > > > > >> > > > > >
> > > >> > > > > > >> > > > > > Thanks,
> > > >> > > > > > >> > > > > > Yufei
> > > >> > > > > > >> > > > > >
> > > >> > > > > > >> > > > > >
> > > >> > > > > > >> > > > > > On Thu, Jun 4, 2026 at 5:14 PM EJ Wang <
> > > >> > > > > > >> > > ej.wang.dev.interest@gmail.com
> > > >> > > > > > >> > > > >
> > > >> > > > > > >> > > > > > wrote:
> > > >> > > > > > >> > > > > >
> > > >> > > > > > >> > > > > > > Good point. I agree the existing
> > > >> > > PolarisMetricsReporter is
> > > >> > > > > > >> > already
> > > >> > > > > > >> > > > very
> > > >> > > > > > >> > > > > > > close to the right conceptual boundary. I
> am
> > not
> > > >> > > > > proposing a
> > > >> > > > > > >> > second
> > > >> > > > > > >> > > > > > > parallel reporter concept. The distinction
> I
> > am
> > > >> > > trying to
> > > >> > > > > > >> make is
> > > >> > > > > > >> > > > > > narrower:
> > > >> > > > > > >> > > > > > >
> > > >> > > > > > >> > > > > > > Current state:
> > > >> > > > > > >> > > > > > >
> > > >> > > > > > >> > > > > > >    - PolarisMetricsReporter is the existing
> > > >> metrics
> > > >> > > > > > reporting
> > > >> > > > > > >> > hook.
> > > >> > > > > > >> > > > > > >    - But it currently lives under
> > > >> runtime/service.
> > > >> > > > > > >> > > > > > >    - Its contract is documented around
> > > >> Quarkus/CDI
> > > >> > > > > discovery
> > > >> > > > > > >> and
> > > >> > > > > > >> > > > > > > @Identifier
> > > >> > > > > > >> > > > > > >    selection.
> > > >> > > > > > >> > > > > > >    - The default implementation is
> log-only.
> > > >> > > > > > >> > > > > > >    - The durable implementation writes
> > through
> > > >> > > > > > >> > > PolarisMetricsManager
> > > >> > > > > > >> > > > > and
> > > >> > > > > > >> > > > > > >    MetricsPersistence.
> > > >> > > > > > >> > > > > > >    - MetricsPersistence is currently
> > inherited by
> > > >> > > > > > >> > BasePersistence.
> > > >> > > > > > >> > > > > > >
> > > >> > > > > > >> > > > > > > My proposal:
> > > >> > > > > > >> > > > > > >
> > > >> > > > > > >> > > > > > >    - Keep the reporting/emitting boundary
> as
> > the
> > > >> > SPI.
> > > >> > > > > > >> > > > > > >    - Revise that contract in a
> > framework-agnostic
> > > >> > > optional
> > > >> > > > > > >> > Iceberg
> > > >> > > > > > >> > > > > > *metrics
> > > >> > > > > > >> > > > > > >    extension API layer*.
> > > >> > > > > > >> > > > > > >    - Keep runtime/service responsible for
> > REST
> > > >> > > ingestion,
> > > >> > > > > > >> > > table/authz
> > > >> > > > > > >> > > > > > >    validation, and runtime wiring.
> > > >> > > > > > >> > > > > > >    - Keep the battery implementation
> > log-only in
> > > >> the
> > > >> > > > > metrics
> > > >> > > > > > >> > > > extension
> > > >> > > > > > >> > > > > > API
> > > >> > > > > > >> > > > > > >    layer, *not under runtime/service*.
> > > >> > > > > > >> > > > > > >    - Treat durable JDBC metrics as one
> > > >> > implementation
> > > >> > > of
> > > >> > > > > the
> > > >> > > > > > >> > > > reporting
> > > >> > > > > > >> > > > > > SPI.
> > > >> > > > > > >> > > > > > >    - Decompose the MetricsPersistence and
> > > >> > > BasePersistence
> > > >> > > > > > >> > coupling
> > > >> > > > > > >> > > as
> > > >> > > > > > >> > > > > > >    durable implementation detail.
> > > >> > > > > > >> > > > > > >
> > > >> > > > > > >> > > > > > > So the difference is not “new reporter
> > concept vs
> > > >> > old
> > > >> > > > > > reporter
> > > >> > > > > > >> > > > > concept.”
> > > >> > > > > > >> > > > > > > but:
> > > >> > > > > > >> > > > > > >
> > > >> > > > > > >> > > > > > >
> > > >> > > > > > >> > > > > > >    - PolarisMetricsReporter today =
> > > >> runtime/service
> > > >> > > > > > CDI-shaped
> > > >> > > > > > >> > > hook.
> > > >> > > > > > >> > > > > > >    - Target reporting SPI = same conceptual
> > > >> > boundary,
> > > >> > > but
> > > >> > > > > > >> placed
> > > >> > > > > > >> > in
> > > >> > > > > > >> > > > the
> > > >> > > > > > >> > > > > > >    right extension API layer and kept
> > > >> > > framework-agnostic.
> > > >> > > > > > >> > > > > > >
> > > >> > > > > > >> > > > > > > For the current metrics PR, I think the
> > minimal
> > > >> > > > > > architectural
> > > >> > > > > > >> > > cleanup
> > > >> > > > > > >> > > > > > > before merge is to *put the reporting SPI
> > > >> contract
> > > >> > > and the
> > > >> > > > > > >> > battery
> > > >> > > > > > >> > > > > > default
> > > >> > > > > > >> > > > > > > in the right place*. The durable JDBC
> > > >> > implementation,
> > > >> > > > > > >> > event-backed
> > > >> > > > > > >> > > > > async
> > > >> > > > > > >> > > > > > > implementation, external queue support, and
> > > >> deeper
> > > >> > > > > > persistence
> > > >> > > > > > >> > > > cleanup
> > > >> > > > > > >> > > > > > can
> > > >> > > > > > >> > > > > > > continue as follow-ups, as long as we do
> not
> > lock
> > > >> > the
> > > >> > > SPI
> > > >> > > > > > >> > boundary
> > > >> > > > > > >> > > to
> > > >> > > > > > >> > > > > > > BasePersistence or to runtime/service
> wiring.
> > > >> > > > > > >> > > > > > >
> > > >> > > > > > >> > > > > > > -ej
> > > >> > > > > > >> > > > > > >
> > > >> > > > > > >> > > > > > > On Wed, Jun 3, 2026 at 3:00 PM Dmitri
> > > >> Bourlatchkov <
> > > >> > > > > > >> > > dimas@apache.org
> > > >> > > > > > >> > > > >
> > > >> > > > > > >> > > > > > > wrote:
> > > >> > > > > > >> > > > > > >
> > > >> > > > > > >> > > > > > > > Hi EJ,
> > > >> > > > > > >> > > > > > > >
> > > >> > > > > > >> > > > > > > > Thanks for the recap / summary!
> > > >> > > > > > >> > > > > > > >
> > > >> > > > > > >> > > > > > > > > Do folks agree that the stable SPI
> > boundary
> > > >> > > should be
> > > >> > > > > > >> > > > > > > > metrics reporting/emitting, not metrics
> > > >> > persistence?
> > > >> > > > > > >> > > > > > > >
> > > >> > > > > > >> > > > > > > > SGTM.
> > > >> > > > > > >> > > > > > > >
> > > >> > > > > > >> > > > > > > > > Does an optional Iceberg metrics
> > extension
> > > >> API
> > > >> > > layer
> > > >> > > > > > sound
> > > >> > > > > > >> > like
> > > >> > > > > > >> > > > the
> > > >> > > > > > >> > > > > > > right
> > > >> > > > > > >> > > > > > > > home for this SPI?
> > > >> > > > > > >> > > > > > > >
> > > >> > > > > > >> > > > > > > > How is that different from current
> > > >> > > > > PolarisMetricsReporter?
> > > >> > > > > > >> > > > > > > >
> > > >> > > > > > >> > > > > > > > > Should the current durable metrics work
> > be
> > > >> > > reframed
> > > >> > > > > as a
> > > >> > > > > > >> > > durable
> > > >> > > > > > >> > > > > > > > JDBC reference implementation of that
> SPI?
> > > >> > > > > > >> > > > > > > >
> > > >> > > > > > >> > > > > > > > SGTM.
> > > >> > > > > > >> > > > > > > >
> > > >> > > > > > >> > > > > > > > > What is the smallest PR sequence to get
> > there
> > > >> > > without
> > > >> > > > > > >> > blocking
> > > >> > > > > > >> > > > > > > > the current metrics work unnecessarily?
> > > >> > > > > > >> > > > > > > >
> > > >> > > > > > >> > > > > > > > Let's get
> > > >> > >
> > > >> >
> > > >>
> >
> https://urldefense.com/v3/__https://github.com/apache/polaris/pull/4397__;!!Iz9xO38YGHZK!4rnlVIqBQQQ6xpGe1PMV58edOS_cGrKoqdfhw4qSp_JQCavvEjaxXk0OwGPnxyEcO60AzKQK_XDtp7H3DdkFXcrXuu5jefU$
> > > >> > > > > > >> merged
> > > >> > > > > > >> > > > first.
> > > >> > > > > > >> > > > > > > >
> > > >> > > > > > >> > > > > > > > Then, I'd propose isolating the Metrics
> > schema
> > > >> > from
> > > >> > > the
> > > >> > > > > > >> > MetaStore
> > > >> > > > > > >> > > > > > schema.
> > > >> > > > > > >> > > > > > > > This will probably have some ripple
> effect
> > into
> > > >> > the
> > > >> > > > > > >> bootstrap
> > > >> > > > > > >> > > > > > workflows,
> > > >> > > > > > >> > > > > > > so
> > > >> > > > > > >> > > > > > > > it's not a trivial change.
> > > >> > > > > > >> > > > > > > >
> > > >> > > > > > >> > > > > > > > Then, let's reassess.
> > > >> > > > > > >> > > > > > > >
> > > >> > > > > > >> > > > > > > > Cheers,
> > > >> > > > > > >> > > > > > > > Dmitri.
> > > >> > > > > > >> > > > > > > >
> > > >> > > > > > >> > > > > > > > On Wed, Jun 3, 2026 at 5:40 PM EJ Wang <
> > > >> > > > > > >> > > > > ej.wang.dev.interest@gmail.com
> > > >> > > > > > >> > > > > > >
> > > >> > > > > > >> > > > > > > > wrote:
> > > >> > > > > > >> > > > > > > >
> > > >> > > > > > >> > > > > > > > > Hi folks,
> > > >> > > > > > >> > > > > > > > >
> > > >> > > > > > >> > > > > > > > > Thanks again for the discussion today.
> I
> > > >> updated
> > > >> > > the
> > > >> > > > > > sync
> > > >> > > > > > >> doc
> > > >> > > > > > >> > > > with
> > > >> > > > > > >> > > > > > > notes
> > > >> > > > > > >> > > > > > > > > from the third metrics architecture
> sync:
> > > >> > > > > > >> > > > > > > > >
> > > >> > > > > > >> > > > > > > > >
> > > >> > > > > > >> > > > > > > > >
> > > >> > > > > > >> > > > > > > >
> > > >> > > > > > >> > > > > > >
> > > >> > > > > > >> > > > > >
> > > >> > > > > > >> > > > >
> > > >> > > > > > >> > > >
> > > >> > > > > > >> > >
> > > >> > > > > > >> >
> > > >> > > > > > >>
> > > >> > > > > >
> > > >> > > > >
> > > >> > >
> > > >> >
> > > >>
> >
> https://urldefense.com/v3/__https://docs.google.com/document/d/100h7c4damrUzVuquYbBHM0EvA4LSWuW2IT2dN_7nYVA/edit?tab=t.k96s2xyqr5u1__;!!Iz9xO38YGHZK!4rnlVIqBQQQ6xpGe1PMV58edOS_cGrKoqdfhw4qSp_JQCavvEjaxXk0OwGPnxyEcO60AzKQK_XDtp7H3DdkFXcrXrqCL-QE$
> > > >> > > > > > >> > > > > > > > >
> > > >> > > > > > >> > > > > > > > > A few highlights from today’s
> discussion:
> > > >> > > > > > >> > > > > > > > >
> > > >> > > > > > >> > > > > > > > >    - We clarified that Iceberg metrics
> > > >> reporting
> > > >> > > can
> > > >> > > > > be
> > > >> > > > > > >> > > > interpreted
> > > >> > > > > > >> > > > > > > > either
> > > >> > > > > > >> > > > > > > > >    as sync or async handling from
> > Polaris’s
> > > >> > > > > perspective,
> > > >> > > > > > >> so
> > > >> > > > > > >> > > > Polaris
> > > >> > > > > > >> > > > > > > > should
> > > >> > > > > > >> > > > > > > > >    stay flexible as a platform instead
> of
> > > >> baking
> > > >> > > one
> > > >> > > > > > >> handling
> > > >> > > > > > >> > > > model
> > > >> > > > > > >> > > > > > > into
> > > >> > > > > > >> > > > > > > > > the
> > > >> > > > > > >> > > > > > > > >    REST API behavior.
> > > >> > > > > > >> > > > > > > > >    - We aligned that the built-in
> > (battery)
> > > >> > > behavior
> > > >> > > > > for
> > > >> > > > > > >> > > metrics
> > > >> > > > > > >> > > > > > > emitting
> > > >> > > > > > >> > > > > > > > >    can stay simple: no-op/log-only is
> > enough
> > > >> as
> > > >> > > the
> > > >> > > > > > >> default.
> > > >> > > > > > >> > > > > > > > >    - Durable metrics persistence should
> > be
> > > >> > > treated as
> > > >> > > > > an
> > > >> > > > > > >> > > > > > implementation
> > > >> > > > > > >> > > > > > > > of
> > > >> > > > > > >> > > > > > > > >    the metrics reporting path, not as
> the
> > > >> core
> > > >> > SPI
> > > >> > > > > > >> boundary.
> > > >> > > > > > >> > > > > > > > >    - The existing durable metrics work
> > can be
> > > >> > > reviewed
> > > >> > > > > > as
> > > >> > > > > > >> a
> > > >> > > > > > >> > > > > reference
> > > >> > > > > > >> > > > > > > > >    implementation of the reporting SPI,
> > with
> > > >> > > > > > >> > > persistence-related
> > > >> > > > > > >> > > > > > logic
> > > >> > > > > > >> > > > > > > > kept
> > > >> > > > > > >> > > > > > > > >    self-contained under a metrics
> durable
> > > >> > > > > implementation
> > > >> > > > > > >> > module
> > > >> > > > > > >> > > > > > rather
> > > >> > > > > > >> > > > > > > > than
> > > >> > > > > > >> > > > > > > > >    scattered through core entity
> > persistence.
> > > >> > > > > > >> > > > > > > > >    - Dashboard/insights remains a real
> > use
> > > >> case,
> > > >> > > but
> > > >> > > > > we
> > > >> > > > > > >> > agreed
> > > >> > > > > > >> > > to
> > > >> > > > > > >> > > > > > keep
> > > >> > > > > > >> > > > > > > it
> > > >> > > > > > >> > > > > > > > >    separate from the core metrics
> intake
> > > >> > > discussion
> > > >> > > > > for
> > > >> > > > > > >> now.
> > > >> > > > > > >> > > > > > > > >
> > > >> > > > > > >> > > > > > > > > I also did a quick source check after
> the
> > > >> > meeting
> > > >> > > to
> > > >> > > > > > make
> > > >> > > > > > >> > sure
> > > >> > > > > > >> > > we
> > > >> > > > > > >> > > > > are
> > > >> > > > > > >> > > > > > > > > describing the current state
> accurately.
> > > >> > > > > > >> > > > > > > > >
> > > >> > > > > > >> > > > > > > > > Current state:
> > > >> > > > > > >> > > > > > > > >
> > > >> > > > > > >> > > > > > > > >    - Polaris already has a metrics
> > reporting
> > > >> > hook:
> > > >> > > > > > >> > > > > > > > PolarisMetricsReporter.
> > > >> > > > > > >> > > > > > > > >    - The default implementation is
> > > >> > > > > > DefaultMetricsReporter,
> > > >> > > > > > >> > > > selected
> > > >> > > > > > >> > > > > > by
> > > >> > > > > > >> > > > > > > > >
> > > >> > > polaris.iceberg-metrics.reporting.type=default. It
> > > >> > > > > is
> > > >> > > > > > >> > > > log-only,
> > > >> > > > > > >> > > > > > > > >    effectively quiet unless metrics
> > logging
> > > >> is
> > > >> > > > > enabled.
> > > >> > > > > > >> > > > > > > > >    - There is also a
> > > >> PersistingMetricsReporter,
> > > >> > > > > selected
> > > >> > > > > > >> by
> > > >> > > > > > >> > > > > > > > >
> > > >> > > polaris.iceberg-metrics.reporting.type=persisting,
> > > >> > > > > > >> which
> > > >> > > > > > >> > > > > converts
> > > >> > > > > > >> > > > > > > > >    Iceberg scan/commit reports into
> > Polaris
> > > >> > > metrics
> > > >> > > > > > >> records
> > > >> > > > > > >> > and
> > > >> > > > > > >> > > > > > writes
> > > >> > > > > > >> > > > > > > > > through
> > > >> > > > > > >> > > > > > > > >    PolarisMetricsManager ->
> > > >> MetricsPersistence.
> > > >> > > > > > >> > > > > > > > >    - MetricsPersistence currently
> exists
> > in
> > > >> the
> > > >> > > > > > >> persistence
> > > >> > > > > > >> > > layer
> > > >> > > > > > >> > > > > and
> > > >> > > > > > >> > > > > > > is
> > > >> > > > > > >> > > > > > > > >    inherited by BasePersistence.
> > > >> > > > > > >> > > > > > > > >
> > > >> > > > > > >> > > > > > > > > My read from the discussion is that the
> > > >> target
> > > >> > > > > boundary
> > > >> > > > > > >> > should
> > > >> > > > > > >> > > > not
> > > >> > > > > > >> > > > > be
> > > >> > > > > > >> > > > > > > > > MetricsPersistence as inherited by
> > > >> > > BasePersistence.
> > > >> > > > > That
> > > >> > > > > > >> path
> > > >> > > > > > >> > > > > should
> > > >> > > > > > >> > > > > > be
> > > >> > > > > > >> > > > > > > > > decomposed as durable implementation
> > detail
> > > >> > (taken
> > > >> > > > > care
> > > >> > > > > > >> of by
> > > >> > > > > > >> > > > > > > > >
> > > >> >
> > > >>
> >
> https://urldefense.com/v3/__https://github.com/apache/polaris/pull/4397__;!!Iz9xO38YGHZK!4rnlVIqBQQQ6xpGe1PMV58edOS_cGrKoqdfhw4qSp_JQCavvEjaxXk0OwGPnxyEcO60AzKQK_XDtp7H3DdkFXcrXuu5jefU$
> > > >> ).
> > > >> > The
> > > >> > > > > > stable
> > > >> > > > > > >> > > > extension
> > > >> > > > > > >> > > > > > > point
> > > >> > > > > > >> > > > > > > > > should instead be the metrics
> > > >> reporting/emitting
> > > >> > > > > > boundary.
> > > >> > > > > > >> > > > > > > > >
> > > >> > > > > > >> > > > > > > > > Concretely, I think the next proposal
> > should
> > > >> be
> > > >> > > shaped
> > > >> > > > > > >> like
> > > >> > > > > > >> > > this:
> > > >> > > > > > >> > > > > > > > >
> > > >> > > > > > >> > > > > > > > >    1.
> > > >> > > > > > >> > > > > > > > >
> > > >> > > > > > >> > > > > > > > >    Define the proper metrics reporting
> > SPI
> > > >> at an
> > > >> > > > > > optional
> > > >> > > > > > >> > > Iceberg
> > > >> > > > > > >> > > > > > > metrics
> > > >> > > > > > >> > > > > > > > >    extension API layer.
> > > >> > > > > > >> > > > > > > > >
> > > >> > > > > > >> > > > > > > > >    Example direction:
> > > >> > > > > > >> > > > > > > > >
> > > >> > > > > > >> > > > > > > > >    public interface
> > IcebergMetricsReporter {
> > > >> > > > > > >> > > > > > > > >      void
> > > >> > reportMetric(IcebergMetricsReportContext
> > > >> > > > > > >> context,
> > > >> > > > > > >> > > > > > > > > MetricsReport report);
> > > >> > > > > > >> > > > > > > > >    }
> > > >> > > > > > >> > > > > > > > >
> > > >> > > > > > >> > > > > > > > >    The context would carry the small
> > > >> > > Polaris-resolved
> > > >> > > > > > >> > envelope:
> > > >> > > > > > >> > > > > > > > >    catalog/table identity, report type,
> > > >> received
> > > >> > > > > > >> timestamp,
> > > >> > > > > > >> > and
> > > >> > > > > > >> > > > > > > > > request/trace
> > > >> > > > > > >> > > > > > > > >    context if available. The raw
> Iceberg
> > > >> > > MetricsReport
> > > >> > > > > > >> > remains
> > > >> > > > > > >> > > > the
> > > >> > > > > > >> > > > > > > > payload.
> > > >> > > > > > >> > > > > > > > >    2.
> > > >> > > > > > >> > > > > > > > >
> > > >> > > > > > >> > > > > > > > >    Keep runtime/service as the REST
> > ingestion
> > > >> > and
> > > >> > > > > wiring
> > > >> > > > > > >> > layer.
> > > >> > > > > > >> > > > > > > > >
> > > >> > > > > > >> > > > > > > > >    The REST handler still resolves the
> > table
> > > >> and
> > > >> > > > > > performs
> > > >> > > > > > >> > authz
> > > >> > > > > > >> > > > > > before
> > > >> > > > > > >> > > > > > > > >    accepting the report. After that, it
> > calls
> > > >> > the
> > > >> > > > > > selected
> > > >> > > > > > >> > > > > reporting
> > > >> > > > > > >> > > > > > > > >    implementation.
> > > >> > > > > > >> > > > > > > > >    3.
> > > >> > > > > > >> > > > > > > > >
> > > >> > > > > > >> > > > > > > > >    Keep the battery default
> > no-op/log-only.
> > > >> > > > > > >> > > > > > > > >
> > > >> > > > > > >> > > > > > > > >    This preserves an out-of-box safe
> > default
> > > >> and
> > > >> > > > > avoids
> > > >> > > > > > >> > > > requiring a
> > > >> > > > > > >> > > > > > > > durable
> > > >> > > > > > >> > > > > > > > >    metrics store for every Polaris
> > > >> deployment.
> > > >> > > > > > >> > > > > > > > >    4.
> > > >> > > > > > >> > > > > > > > >
> > > >> > > > > > >> > > > > > > > >    Move durable JDBC metrics into a
> > > >> > self-contained
> > > >> > > > > > >> > > implementation
> > > >> > > > > > >> > > > > of
> > > >> > > > > > >> > > > > > > the
> > > >> > > > > > >> > > > > > > > >    reporting SPI.
> > > >> > > > > > >> > > > > > > > >
> > > >> > > > > > >> > > > > > > > >    That implementation can own its
> > schema,
> > > >> > > bootstrap,
> > > >> > > > > > >> > > retention,
> > > >> > > > > > >> > > > > and
> > > >> > > > > > >> > > > > > > read
> > > >> > > > > > >> > > > > > > > >    API support. It should not define
> the
> > core
> > > >> > > > > reporting
> > > >> > > > > > >> SPI
> > > >> > > > > > >> > > > > boundary,
> > > >> > > > > > >> > > > > > > and
> > > >> > > > > > >> > > > > > > > > it
> > > >> > > > > > >> > > > > > > > >    should not require metrics
> > persistence to
> > > >> > > remain
> > > >> > > > > > >> inherited
> > > >> > > > > > >> > > > from
> > > >> > > > > > >> > > > > > > > >    BasePersistence.
> > > >> > > > > > >> > > > > > > > >    5.
> > > >> > > > > > >> > > > > > > > >
> > > >> > > > > > >> > > > > > > > >    Treat async/event-backed handling as
> > > >> another
> > > >> > > > > > >> > implementation
> > > >> > > > > > >> > > of
> > > >> > > > > > >> > > > > the
> > > >> > > > > > >> > > > > > > > same
> > > >> > > > > > >> > > > > > > > >    reporting SPI.
> > > >> > > > > > >> > > > > > > > >
> > > >> > > > > > >> > > > > > > > >    For example, an event-backed
> reporter
> > > >> could
> > > >> > > > > enqueue a
> > > >> > > > > > >> > > metrics
> > > >> > > > > > >> > > > > > event
> > > >> > > > > > >> > > > > > > > and
> > > >> > > > > > >> > > > > > > > >    let listeners handle durable storage
> > or
> > > >> other
> > > >> > > > > sinks.
> > > >> > > > > > >> If we
> > > >> > > > > > >> > > > later
> > > >> > > > > > >> > > > > > > need
> > > >> > > > > > >> > > > > > > > a
> > > >> > > > > > >> > > > > > > > >    replaceable queue engine, that seems
> > like
> > > >> a
> > > >> > > shared
> > > >> > > > > > >> > > > event/metrics
> > > >> > > > > > >> > > > > > > > > substrate
> > > >> > > > > > >> > > > > > > > >    topic rather than a metrics-only
> > > >> requirement.
> > > >> > > > > > >> > > > > > > > >
> > > >> > > > > > >> > > > > > > > > This framing lets us keep the REST
> > metrics
> > > >> > > endpoint
> > > >> > > > > > >> simple,
> > > >> > > > > > >> > > > > preserve
> > > >> > > > > > >> > > > > > > the
> > > >> > > > > > >> > > > > > > > > current default behavior, support
> durable
> > > >> > metrics
> > > >> > > > > users,
> > > >> > > > > > >> and
> > > >> > > > > > >> > > > still
> > > >> > > > > > >> > > > > > > leave
> > > >> > > > > > >> > > > > > > > > room for async/event-backed or
> > > >> > > external-queue-based
> > > >> > > > > > >> > > > > implementations.
> > > >> > > > > > >> > > > > > > > >
> > > >> > > > > > >> > > > > > > > > I think the main follow-up questions
> are:
> > > >> > > > > > >> > > > > > > > >
> > > >> > > > > > >> > > > > > > > >    - Do folks agree that the stable SPI
> > > >> boundary
> > > >> > > > > should
> > > >> > > > > > be
> > > >> > > > > > >> > > > metrics
> > > >> > > > > > >> > > > > > > > >    reporting/emitting, not metrics
> > > >> persistence?
> > > >> > > > > > >> > > > > > > > >    - Does an optional Iceberg metrics
> > > >> extension
> > > >> > > API
> > > >> > > > > > layer
> > > >> > > > > > >> > sound
> > > >> > > > > > >> > > > > like
> > > >> > > > > > >> > > > > > > the
> > > >> > > > > > >> > > > > > > > >    right home for this SPI?
> > > >> > > > > > >> > > > > > > > >    - Should the current durable metrics
> > work
> > > >> be
> > > >> > > > > reframed
> > > >> > > > > > >> as a
> > > >> > > > > > >> > > > > durable
> > > >> > > > > > >> > > > > > > > JDBC
> > > >> > > > > > >> > > > > > > > >    reference implementation of that
> SPI?
> > > >> > > > > > >> > > > > > > > >    - What is the smallest PR sequence
> to
> > get
> > > >> > there
> > > >> > > > > > without
> > > >> > > > > > >> > > > blocking
> > > >> > > > > > >> > > > > > the
> > > >> > > > > > >> > > > > > > > >    current metrics work unnecessarily?
> > > >> > > > > > >> > > > > > > > >
> > > >> > > > > > >> > > > > > > > > Thanks,
> > > >> > > > > > >> > > > > > > > > -ej
> > > >> > > > > > >> > > > > > > > >
> > > >> > > > > > >> > > > > > > > > On Fri, May 15, 2026 at 5:16 PM Dmitri
> > > >> > > Bourlatchkov <
> > > >> > > > > > >> > > > > > dimas@apache.org>
> > > >> > > > > > >> > > > > > > > > wrote:
> > > >> > > > > > >> > > > > > > > >
> > > >> > > > > > >> > > > > > > > > > Hi JB,
> > > >> > > > > > >> > > > > > > > > >
> > > >> > > > > > >> > > > > > > > > > Could you set up another meeting,
> > please?
> > > >> Same
> > > >> > > time
> > > >> > > > > on
> > > >> > > > > > >> > > > Wednesday
> > > >> > > > > > >> > > > > as
> > > >> > > > > > >> > > > > > > > last
> > > >> > > > > > >> > > > > > > > > > time... I hope it works for everyone.
> > > >> > > > > > >> > > > > > > > > >
> > > >> > > > > > >> > > > > > > > > > Cheers,
> > > >> > > > > > >> > > > > > > > > > Dmitri.
> > > >> > > > > > >> > > > > > > > > >
> > > >> > > > > > >> > > > > > > > > > On Fri, May 15, 2026 at 8:06 PM Yufei
> > Gu <
> > > >> > > > > > >> > > flyrain000@gmail.com
> > > >> > > > > > >> > > > >
> > > >> > > > > > >> > > > > > > wrote:
> > > >> > > > > > >> > > > > > > > > >
> > > >> > > > > > >> > > > > > > > > > > +1 on another sync call next week.
> > > >> > > > > > >> > > > > > > > > > >
> > > >> > > > > > >> > > > > > > > > > > Yufei
> > > >> > > > > > >> > > > > > > > > > >
> > > >> > > > > > >> > > > > > > > > > >
> > > >> > > > > > >> > > > > > > > > > > On Fri, May 15, 2026 at 4:52 PM
> > Dmitri
> > > >> > > > > Bourlatchkov
> > > >> > > > > > <
> > > >> > > > > > >> > > > > > > > dimas@apache.org>
> > > >> > > > > > >> > > > > > > > > > > wrote:
> > > >> > > > > > >> > > > > > > > > > >
> > > >> > > > > > >> > > > > > > > > > > > Hi All,
> > > >> > > > > > >> > > > > > > > > > > >
> > > >> > > > > > >> > > > > > > > > > > > WDYT about another sync call next
> > week?
> > > >> > > > > > >> > > > > > > > > > > >
> > > >> > > > > > >> > > > > > > > > > > > Thanks,
> > > >> > > > > > >> > > > > > > > > > > > Dmitri.
> > > >> > > > > > >> > > > > > > > > > > >
> > > >> > > > > > >> > > > > > > > > > > > On Wed, May 6, 2026 at 5:29 PM
> > Dmitri
> > > >> > > > > > Bourlatchkov <
> > > >> > > > > > >> > > > > > > > dimas@apache.org
> > > >> > > > > > >> > > > > > > > > >
> > > >> > > > > > >> > > > > > > > > > > > wrote:
> > > >> > > > > > >> > > > > > > > > > > >
> > > >> > > > > > >> > > > > > > > > > > > > Hi EJ,
> > > >> > > > > > >> > > > > > > > > > > > >
> > > >> > > > > > >> > > > > > > > > > > > > Thanks for the summary! It
> covers
> > > >> what
> > > >> > we
> > > >> > > > > > >> discussed
> > > >> > > > > > >> > in
> > > >> > > > > > >> > > > the
> > > >> > > > > > >> > > > > > > > meeting
> > > >> > > > > > >> > > > > > > > > > very
> > > >> > > > > > >> > > > > > > > > > > > > well, IMHO.
> > > >> > > > > > >> > > > > > > > > > > > >
> > > >> > > > > > >> > > > > > > > > > > > > Looking forward to concrete PRs
> > :)
> > > >> > > > > > >> > > > > > > > > > > > >
> > > >> > > > > > >> > > > > > > > > > > > > Cheers,
> > > >> > > > > > >> > > > > > > > > > > > > Dmitri.
> > > >> > > > > > >> > > > > > > > > > > > >
> > > >> > > > > > >> > > > > > > > > > > > > On Wed, May 6, 2026 at 5:08 PM
> EJ
> > > >> Wang <
> > > >> > > > > > >> > > > > > > > > > ej.wang.dev.interest@gmail.com
> > > >> > > > > > >> > > > > > > > > > > >
> > > >> > > > > > >> > > > > > > > > > > > > wrote:
> > > >> > > > > > >> > > > > > > > > > > > >
> > > >> > > > > > >> > > > > > > > > > > > >> Hi folks,
> > > >> > > > > > >> > > > > > > > > > > > >>
> > > >> > > > > > >> > > > > > > > > > > > >> We had a community sync
> earlier,
> > > >> thanks
> > > >> > > JB
> > > >> > > > > for
> > > >> > > > > > >> > > > scheduling
> > > >> > > > > > >> > > > > > it.
> > > >> > > > > > >> > > > > > > > > Notes
> > > >> > > > > > >> > > > > > > > > > > from
> > > >> > > > > > >> > > > > > > > > > > > >> the first metrics architecture
> > sync
> > > >> > (May
> > > >> > > 6,
> > > >> > > > > > >> 10-11am
> > > >> > > > > > >> > > PT).
> > > >> > > > > > >> > > > > > > > > Discussion
> > > >> > > > > > >> > > > > > > > > > > doc
> > > >> > > > > > >> > > > > > > > > > > > >> with per-section status:
> > > >> > > > > > >> > > > > > > > > > > > >>
> > > >> > > > > > >> > > > > > > > > > > > >>
> > > >> > > > > > >> > > > > > > > > > > >
> > > >> > > > > > >> > > > > > > > > > >
> > > >> > > > > > >> > > > > > > > > >
> > > >> > > > > > >> > > > > > > > >
> > > >> > > > > > >> > > > > > > >
> > > >> > > > > > >> > > > > > >
> > > >> > > > > > >> > > > > >
> > > >> > > > > > >> > > > >
> > > >> > > > > > >> > > >
> > > >> > > > > > >> > >
> > > >> > > > > > >> >
> > > >> > > > > > >>
> > > >> > > > > >
> > > >> > > > >
> > > >> > >
> > > >> >
> > > >>
> >
> https://urldefense.com/v3/__https://docs.google.com/document/d/100h7c4damrUzVuquYbBHM0EvA4LSWuW2IT2dN_7nYVA/edit?tab=t.0__;!!Iz9xO38YGHZK!4rnlVIqBQQQ6xpGe1PMV58edOS_cGrKoqdfhw4qSp_JQCavvEjaxXk0OwGPnxyEcO60AzKQK_XDtp7H3DdkFXcrXgIddeUs$
> > > >> > > > > > >> > > > > > > > > > > > >>
> > > >> > > > > > >> > > > > > > > > > > > >> *The meeting covered both
> topics
> > > >> from
> > > >> > the
> > > >> > > > > doc.
> > > >> > > > > > >> > > > > > Direction-level
> > > >> > > > > > >> > > > > > > > > > > alignment
> > > >> > > > > > >> > > > > > > > > > > > >> was reached on the headline
> > pieces;
> > > >> > > details
> > > >> > > > > > >> remain
> > > >> > > > > > >> > for
> > > >> > > > > > >> > > > PR
> > > >> > > > > > >> > > > > > > review
> > > >> > > > > > >> > > > > > > > > or
> > > >> > > > > > >> > > > > > > > > > > > >> follow-up sessions.*
> > > >> > > > > > >> > > > > > > > > > > > >>
> > > >> > > > > > >> > > > > > > > > > > > >> *Topic 1 — Persistence schema
> > > >> redesign*
> > > >> > > > > > >> > > > > > > > > > > > >> Idea-level alignment on
> > > >> consolidating
> > > >> > > > > per-type
> > > >> > > > > > >> > tables
> > > >> > > > > > >> > > > > > > > > > > > >> (scan_metrics_report,
> > > >> > > > > > >> > > > > > > > > > > > >> commit_metrics_report) into a
> > single
> > > >> > > > > > >> metrics_report
> > > >> > > > > > >> > > > table.
> > > >> > > > > > >> > > > > > The
> > > >> > > > > > >> > > > > > > > > > > > motivating
> > > >> > > > > > >> > > > > > > > > > > > >> cost is the surface area added
> > by
> > > >> every
> > > >> > > new
> > > >> > > > > > >> metric
> > > >> > > > > > >> > > type
> > > >> > > > > > >> > > > > > today:
> > > >> > > > > > >> > > > > > > > new
> > > >> > > > > > >> > > > > > > > > > > > table,
> > > >> > > > > > >> > > > > > > > > > > > >> SPI method, record class,
> model,
> > > >> > > converter,
> > > >> > > > > > >> schema
> > > >> > > > > > >> > > > > > migration.
> > > >> > > > > > >> > > > > > > > > > > > >>
> > > >> > > > > > >> > > > > > > > > > > > >> Most schema details are
> > deferred to
> > > >> the
> > > >> > > > > schema
> > > >> > > > > > >> PR. A
> > > >> > > > > > >> > > few
> > > >> > > > > > >> > > > > > > > specific
> > > >> > > > > > >> > > > > > > > > > > points
> > > >> > > > > > >> > > > > > > > > > > > >> came up:
> > > >> > > > > > >> > > > > > > > > > > > >> •   metric_schema_version:
> Yufei
> > > >> > prefers
> > > >> > > > > > dropping
> > > >> > > > > > >> > it,
> > > >> > > > > > >> > > > > since
> > > >> > > > > > >> > > > > > > > there
> > > >> > > > > > >> > > > > > > > > is
> > > >> > > > > > >> > > > > > > > > > > no
> > > >> > > > > > >> > > > > > > > > > > > >> spec-level concept of metrics
> > > >> > versioning
> > > >> > > > > today
> > > >> > > > > > >> and
> > > >> > > > > > >> > it
> > > >> > > > > > >> > > is
> > > >> > > > > > >> > > > > > hard
> > > >> > > > > > >> > > > > > > to
> > > >> > > > > > >> > > > > > > > > > > define
> > > >> > > > > > >> > > > > > > > > > > > >> unilaterally. Robert prefers
> > keeping
> > > >> > it,
> > > >> > > > > given
> > > >> > > > > > >> IRC
> > > >> > > > > > >> > v2
> > > >> > > > > > >> > > is
> > > >> > > > > > >> > > > > > > coming
> > > >> > > > > > >> > > > > > > > > and
> > > >> > > > > > >> > > > > > > > > > > the
> > > >> > > > > > >> > > > > > > > > > > > >> schema should be considered
> > against
> > > >> its
> > > >> > > > > likely
> > > >> > > > > > >> > shape;
> > > >> > > > > > >> > > > > Robert
> > > >> > > > > > >> > > > > > > > also
> > > >> > > > > > >> > > > > > > > > > > raised
> > > >> > > > > > >> > > > > > > > > > > > >> how to differentiate various
> > payload
> > > >> > > formats
> > > >> > > > > if
> > > >> > > > > > >> any.
> > > >> > > > > > >> > > > EJ's
> > > >> > > > > > >> > > > > > read
> > > >> > > > > > >> > > > > > > > is
> > > >> > > > > > >> > > > > > > > > > that
> > > >> > > > > > >> > > > > > > > > > > > >> this
> > > >> > > > > > >> > > > > > > > > > > > >> is a two-way-door decision. We
> > can
> > > >> > start
> > > >> > > > > > without
> > > >> > > > > > >> the
> > > >> > > > > > >> > > > > field,
> > > >> > > > > > >> > > > > > > and
> > > >> > > > > > >> > > > > > > > if
> > > >> > > > > > >> > > > > > > > > > IRC
> > > >> > > > > > >> > > > > > > > > > > > v2
> > > >> > > > > > >> > > > > > > > > > > > >> changes the shape we would
> > likely
> > > >> roll
> > > >> > a
> > > >> > > > > > >> > corresponding
> > > >> > > > > > >> > > > new
> > > >> > > > > > >> > > > > > > > schema
> > > >> > > > > > >> > > > > > > > > > > > anyway,
> > > >> > > > > > >> > > > > > > > > > > > >> which is not particularly
> > costly.
> > > >> > > > > > >> > > > > > > > > > > > >> •   Payload format: Robert
> > pointed
> > > >> out
> > > >> > > that
> > > >> > > > > > >> future
> > > >> > > > > > >> > > > formats
> > > >> > > > > > >> > > > > > > > beyond
> > > >> > > > > > >> > > > > > > > > > JSON
> > > >> > > > > > >> > > > > > > > > > > > may
> > > >> > > > > > >> > > > > > > > > > > > >> be worth supporting. The exact
> > > >> shape is
> > > >> > > > > > deferred
> > > >> > > > > > >> to
> > > >> > > > > > >> > > the
> > > >> > > > > > >> > > > > > schema
> > > >> > > > > > >> > > > > > > > > > > > discussion.
> > > >> > > > > > >> > > > > > > > > > > > >> •   Partition strategy: Anand
> > > >> suggested
> > > >> > > > > monthly
> > > >> > > > > > >> > > > > partitioning
> > > >> > > > > > >> > > > > > > > based
> > > >> > > > > > >> > > > > > > > > > on
> > > >> > > > > > >> > > > > > > > > > > > his
> > > >> > > > > > >> > > > > > > > > > > > >> experience as potentially
> > helpful at
> > > >> > > scale.
> > > >> > > > > > >> > > > > > > > > > > > >>
> > > >> > > > > > >> > > > > > > > > > > > >> *Topic 2 — Where metrics
> > ingestion
> > > >> and
> > > >> > > > > storage
> > > >> > > > > > >> > belong*
> > > >> > > > > > >> > > > > > > > > > > > >> Idea-level alignment that
> > metrics
> > > >> > should
> > > >> > > be a
> > > >> > > > > > >> > > separated
> > > >> > > > > > >> > > > > SPI
> > > >> > > > > > >> > > > > > > from
> > > >> > > > > > >> > > > > > > > > the
> > > >> > > > > > >> > > > > > > > > > > > >> entity
> > > >> > > > > > >> > > > > > > > > > > > >> persistence stack. Two reasons
> > > >> > surfaced:
> > > >> > > (a)
> > > >> > > > > > >> > workloads
> > > >> > > > > > >> > > > and
> > > >> > > > > > >> > > > > > > > > > capability
> > > >> > > > > > >> > > > > > > > > > > > >> requirements diverge enough
> that
> > > >> > coupling
> > > >> > > > > them
> > > >> > > > > > >> > creates
> > > >> > > > > > >> > > > > > > > artificial
> > > >> > > > > > >> > > > > > > > > > > > >> constraints, and (b) admin
> > > >> experience
> > > >> > > > > improves
> > > >> > > > > > >> when
> > > >> > > > > > >> > > > > metrics
> > > >> > > > > > >> > > > > > > has
> > > >> > > > > > >> > > > > > > > > its
> > > >> > > > > > >> > > > > > > > > > > own
> > > >> > > > > > >> > > > > > > > > > > > >> bootstrap, retention, and
> > lifecycle.
> > > >> > > Dmitri
> > > >> > > > > > noted
> > > >> > > > > > >> > > > Polaris
> > > >> > > > > > >> > > > > > > being
> > > >> > > > > > >> > > > > > > > a
> > > >> > > > > > >> > > > > > > > > > > > platform
> > > >> > > > > > >> > > > > > > > > > > > >> should have the flexibility to
> > > >> support
> > > >> > > > > > different
> > > >> > > > > > >> > > > > persistence
> > > >> > > > > > >> > > > > > > > > > backends
> > > >> > > > > > >> > > > > > > > > > > > per
> > > >> > > > > > >> > > > > > > > > > > > >> concern, and pointed to a
> > concrete
> > > >> next
> > > >> > > step
> > > >> > > > > of
> > > >> > > > > > >> > > > separating
> > > >> > > > > > >> > > > > > the
> > > >> > > > > > >> > > > > > > > > JDBC
> > > >> > > > > > >> > > > > > > > > > > > >> bootstrap for metrics from the
> > > >> > metastore
> > > >> > > > > > >> bootstrap.
> > > >> > > > > > >> > > > Robert
> > > >> > > > > > >> > > > > > > > > proposed
> > > >> > > > > > >> > > > > > > > > > an
> > > >> > > > > > >> > > > > > > > > > > > >> additional UX extension:
> detect
> > an
> > > >> > > > > > unbootstrapped
> > > >> > > > > > >> > > > metrics
> > > >> > > > > > >> > > > > > > store
> > > >> > > > > > >> > > > > > > > on
> > > >> > > > > > >> > > > > > > > > > > first
> > > >> > > > > > >> > > > > > > > > > > > >> use and auto-bootstrap rather
> > than
> > > >> > > requiring
> > > >> > > > > an
> > > >> > > > > > >> > > explicit
> > > >> > > > > > >> > > > > > > manual
> > > >> > > > > > >> > > > > > > > > > > > bootstrap
> > > >> > > > > > >> > > > > > > > > > > > >> step.
> > > >> > > > > > >> > > > > > > > > > > > >> The meeting also confirmed
> that
> > > >> Polaris
> > > >> > > > > metrics
> > > >> > > > > > >> can
> > > >> > > > > > >> > > > start
> > > >> > > > > > >> > > > > > > small
> > > >> > > > > > >> > > > > > > > > and
> > > >> > > > > > >> > > > > > > > > > > stay
> > > >> > > > > > >> > > > > > > > > > > > >> Iceberg-focused. Naming and
> > > >> persistence
> > > >> > > > > schema
> > > >> > > > > > >> can
> > > >> > > > > > >> > > lean
> > > >> > > > > > >> > > > > > > > > > > > Iceberg-specific.
> > > >> > > > > > >> > > > > > > > > > > > >> If a future expansion to
> > > >> generic-table
> > > >> > > > > metrics
> > > >> > > > > > or
> > > >> > > > > > >> > > > > > operational
> > > >> > > > > > >> > > > > > > > > > metrics
> > > >> > > > > > >> > > > > > > > > > > > >> arrives, an abstraction layer
> > can be
> > > >> > > built on
> > > >> > > > > > >> top of
> > > >> > > > > > >> > > the
> > > >> > > > > > >> > > > > > > Iceberg
> > > >> > > > > > >> > > > > > > > > > > metrics
> > > >> > > > > > >> > > > > > > > > > > > >> reporter at that point. Robert
> > > >> remains
> > > >> > > on the
> > > >> > > > > > >> fence
> > > >> > > > > > >> > > and
> > > >> > > > > > >> > > > > > would
> > > >> > > > > > >> > > > > > > > > prefer
> > > >> > > > > > >> > > > > > > > > > > > >> something more generic but did
> > not
> > > >> > block
> > > >> > > the
> > > >> > > > > > >> > > direction;
> > > >> > > > > > >> > > > > > > Dmitri's
> > > >> > > > > > >> > > > > > > > > > read
> > > >> > > > > > >> > > > > > > > > > > > was
> > > >> > > > > > >> > > > > > > > > > > > >> that the proposed framework
> > already
> > > >> has
> > > >> > > > > enough
> > > >> > > > > > >> > > > flexibility
> > > >> > > > > > >> > > > > > to
> > > >> > > > > > >> > > > > > > > > absorb
> > > >> > > > > > >> > > > > > > > > > > > >> future
> > > >> > > > > > >> > > > > > > > > > > > >> expansion.
> > > >> > > > > > >> > > > > > > > > > > > >>
> > > >> > > > > > >> > > > > > > > > > > > >> The Trade-offs and Proposed
> > > >> structure
> > > >> > > > > sections
> > > >> > > > > > in
> > > >> > > > > > >> > the
> > > >> > > > > > >> > > > doc
> > > >> > > > > > >> > > > > > were
> > > >> > > > > > >> > > > > > > > not
> > > >> > > > > > >> > > > > > > > > > > > >> reviewed
> > > >> > > > > > >> > > > > > > > > > > > >> in detail. They remain open
> for
> > > >> either
> > > >> > > the
> > > >> > > > > next
> > > >> > > > > > >> sync
> > > >> > > > > > >> > > or
> > > >> > > > > > >> > > > PR
> > > >> > > > > > >> > > > > > > > review.
> > > >> > > > > > >> > > > > > > > > > > > >>
> > > >> > > > > > >> > > > > > > > > > > > >> *Cross-cutting alignment —
> > > >> > > battery-included
> > > >> > > > > > plus
> > > >> > > > > > >> > > > > pluggable*
> > > >> > > > > > >> > > > > > > > > > > > >> A common philosophy emerged
> > from the
> > > >> > > > > > discussion.
> > > >> > > > > > >> EJ
> > > >> > > > > > >> > > > > > summarized
> > > >> > > > > > >> > > > > > > > it
> > > >> > > > > > >> > > > > > > > > > as:
> > > >> > > > > > >> > > > > > > > > > > > >> Polaris should provide a
> > > >> > > battery-included UX
> > > >> > > > > > for
> > > >> > > > > > >> > > > beginners
> > > >> > > > > > >> > > > > > and
> > > >> > > > > > >> > > > > > > > the
> > > >> > > > > > >> > > > > > > > > > > > >> flexibility for advanced users
> > to
> > > >> swap
> > > >> > > the
> > > >> > > > > > >> included
> > > >> > > > > > >> > > > > battery
> > > >> > > > > > >> > > > > > > for
> > > >> > > > > > >> > > > > > > > > > > > something
> > > >> > > > > > >> > > > > > > > > > > > >> more powerful or tailored to
> > their
> > > >> use
> > > >> > > case.
> > > >> > > > > > The
> > > >> > > > > > >> SPI
> > > >> > > > > > >> > > > > design
> > > >> > > > > > >> > > > > > > > needs
> > > >> > > > > > >> > > > > > > > > to
> > > >> > > > > > >> > > > > > > > > > > > >> enable
> > > >> > > > > > >> > > > > > > > > > > > >> both.
> > > >> > > > > > >> > > > > > > > > > > > >>
> > > >> > > > > > >> > > > > > > > > > > > >> The inputs that shaped this
> > framing:
> > > >> > > > > > >> > > > > > > > > > > > >> •   Anand described how his
> team
> > > >> uses
> > > >> > the
> > > >> > > > > > current
> > > >> > > > > > >> > > > metrics
> > > >> > > > > > >> > > > > > > > > > persistence
> > > >> > > > > > >> > > > > > > > > > > > >> (three metrics consumers in
> > v1.4).
> > > >> > > > > > >> > > > > > > > > > > > >> •   Yufei raised Grafana and
> > > >> dashboard
> > > >> > > > > > >> integrations
> > > >> > > > > > >> > > as a
> > > >> > > > > > >> > > > > > > > > destination
> > > >> > > > > > >> > > > > > > > > > > use
> > > >> > > > > > >> > > > > > > > > > > > >> case beyond the default.
> > > >> > > > > > >> > > > > > > > > > > > >> •   Robert called out that the
> > > >> current
> > > >> > > design
> > > >> > > > > > is
> > > >> > > > > > >> > more
> > > >> > > > > > >> > > > > > > > > JDBC-focused.
> > > >> > > > > > >> > > > > > > > > > > > >>
> > > >> > > > > > >> > > > > > > > > > > > >> Two concrete instances:
> > > >> > > > > > >> > > > > > > > > > > > >> •   Async metrics intake:
> > Yufei's
> > > >> > initial
> > > >> > > > > > >> position
> > > >> > > > > > >> > was
> > > >> > > > > > >> > > > > that
> > > >> > > > > > >> > > > > > > > async
> > > >> > > > > > >> > > > > > > > > > > should
> > > >> > > > > > >> > > > > > > > > > > > >> largely live on the producer
> > side
> > > >> and
> > > >> > > there
> > > >> > > > > is
> > > >> > > > > > >> not
> > > >> > > > > > >> > > much
> > > >> > > > > > >> > > > > > > Polaris
> > > >> > > > > > >> > > > > > > > > can
> > > >> > > > > > >> > > > > > > > > > > do.
> > > >> > > > > > >> > > > > > > > > > > > >> Robert suggested a
> Polaris-side
> > > >> default
> > > >> > > is
> > > >> > > > > > doable
> > > >> > > > > > >> > via
> > > >> > > > > > >> > > > > > Vert.x.
> > > >> > > > > > >> > > > > > > > > Dmitri
> > > >> > > > > > >> > > > > > > > > > > > >> agreed
> > > >> > > > > > >> > > > > > > > > > > > >> the direction is worth
> > exploring.
> > > >> The
> > > >> > > meeting
> > > >> > > > > > >> > > converged
> > > >> > > > > > >> > > > > on a
> > > >> > > > > > >> > > > > > > > > > > > >> battery-included default
> (likely
> > > >> > > > > Vert.x-backed)
> > > >> > > > > > >> with
> > > >> > > > > > >> > > an
> > > >> > > > > > >> > > > > SPI
> > > >> > > > > > >> > > > > > > > shape
> > > >> > > > > > >> > > > > > > > > > that
> > > >> > > > > > >> > > > > > > > > > > > >> lets
> > > >> > > > > > >> > > > > > > > > > > > >> power users route to a more
> > scalable
> > > >> > > backend
> > > >> > > > > > >> > > (k8s-hosted
> > > >> > > > > > >> > > > > > > queue,
> > > >> > > > > > >> > > > > > > > > AWS
> > > >> > > > > > >> > > > > > > > > > > SQS,
> > > >> > > > > > >> > > > > > > > > > > > >> etc.).
> > > >> > > > > > >> > > > > > > > > > > > >> •   Pluggable destinations:
> > > >> combining
> > > >> > > Yufei's
> > > >> > > > > > >> > > dashboard
> > > >> > > > > > >> > > > > use
> > > >> > > > > > >> > > > > > > case
> > > >> > > > > > >> > > > > > > > > > with
> > > >> > > > > > >> > > > > > > > > > > > >> Robert's JDBC-focused
> call-out,
> > the
> > > >> > > meeting
> > > >> > > > > > >> agreed
> > > >> > > > > > >> > the
> > > >> > > > > > >> > > > SPI
> > > >> > > > > > >> > > > > > > > should
> > > >> > > > > > >> > > > > > > > > be
> > > >> > > > > > >> > > > > > > > > > > > >> structured for multiple sinks
> so
> > > >> > > integrations
> > > >> > > > > > >> become
> > > >> > > > > > >> > > > impl
> > > >> > > > > > >> > > > > > > > choices
> > > >> > > > > > >> > > > > > > > > > > rather
> > > >> > > > > > >> > > > > > > > > > > > >> than architectural changes.
> > > >> > > > > > >> > > > > > > > > > > > >>
> > > >> > > > > > >> > > > > > > > > > > > >> The battery-included default
> is
> > most
> > > >> > > likely
> > > >> > > > > to
> > > >> > > > > > >> use
> > > >> > > > > > >> > the
> > > >> > > > > > >> > > > > > > existing
> > > >> > > > > > >> > > > > > > > > > > > >> JDBC-backed
> > > >> > > > > > >> > > > > > > > > > > > >> approach.
> > > >> > > > > > >> > > > > > > > > > > > >>
> > > >> > > > > > >> > > > > > > > > > > > >> *Direction (idea-level
> > alignment)*
> > > >> > > > > > >> > > > > > > > > > > > >> •   Single metrics_report
> table
> > > >> > > consolidating
> > > >> > > > > > >> > per-type
> > > >> > > > > > >> > > > > > > metrics,
> > > >> > > > > > >> > > > > > > > > > > > replacing
> > > >> > > > > > >> > > > > > > > > > > > >> scan_metrics_report and
> > > >> > > commit_metrics_report
> > > >> > > > > > >> > > > > > > > > > > > >> •   Iceberg-focused naming and
> > > >> schema
> > > >> > for
> > > >> > > > > now,
> > > >> > > > > > >> > revisit
> > > >> > > > > > >> > > > if
> > > >> > > > > > >> > > > > > > > > > > generic-table
> > > >> > > > > > >> > > > > > > > > > > > or
> > > >> > > > > > >> > > > > > > > > > > > >> operational metrics arrive
> > > >> > > > > > >> > > > > > > > > > > > >> •   Metrics persistence as a
> > > >> separated
> > > >> > > SPI,
> > > >> > > > > not
> > > >> > > > > > >> on
> > > >> > > > > > >> > > > > > > > BasePersistence
> > > >> > > > > > >> > > > > > > > > > > > >> •   Bootstrap path separated
> for
> > > >> > metrics,
> > > >> > > > > > >> > independent
> > > >> > > > > > >> > > of
> > > >> > > > > > >> > > > > > > > metastore
> > > >> > > > > > >> > > > > > > > > > > > >> bootstrap
> > > >> > > > > > >> > > > > > > > > > > > >> •   "Battery-included plus
> > > >> pluggable"
> > > >> > as
> > > >> > > the
> > > >> > > > > > SPI
> > > >> > > > > > >> > > design
> > > >> > > > > > >> > > > > > > > philosophy
> > > >> > > > > > >> > > > > > > > > > > > >>
> > > >> > > > > > >> > > > > > > > > > > > >> *Open items*
> > > >> > > > > > >> > > > > > > > > > > > >> •   Schema details:
> > > >> > > metric_schema_version,
> > > >> > > > > > >> payload
> > > >> > > > > > >> > > > format,
> > > >> > > > > > >> > > > > > IRC
> > > >> > > > > > >> > > > > > > > v2
> > > >> > > > > > >> > > > > > > > > > > > >> forward-compat shape
> > > >> > > > > > >> > > > > > > > > > > > >> •   SPI design details — full
> > review
> > > >> > > either
> > > >> > > > > in
> > > >> > > > > > >> the
> > > >> > > > > > >> > > next
> > > >> > > > > > >> > > > > sync
> > > >> > > > > > >> > > > > > > or
> > > >> > > > > > >> > > > > > > > in
> > > >> > > > > > >> > > > > > > > > > the
> > > >> > > > > > >> > > > > > > > > > > > >> corresponding PR
> > > >> > > > > > >> > > > > > > > > > > > >> •   Schema refactor PR
> ownership
> > > >> > > > > > >> > > > > > > > > > > > >>
> > > >> > > > > > >> > > > > > > > > > > > >> *Action items*
> > > >> > > > > > >> > > > > > > > > > > > >> •   EJ to take a first stab at
> > the
> > > >> SPI
> > > >> > > design
> > > >> > > > > > and
> > > >> > > > > > >> > > > > > potentially
> > > >> > > > > > >> > > > > > > > > > partner
> > > >> > > > > > >> > > > > > > > > > > > with
> > > >> > > > > > >> > > > > > > > > > > > >> Anand to incorporate the
> lessons
> > > >> > learned
> > > >> > > from
> > > >> > > > > > the
> > > >> > > > > > >> > > > existing
> > > >> > > > > > >> > > > > > > > > reporter
> > > >> > > > > > >> > > > > > > > > > > and
> > > >> > > > > > >> > > > > > > > > > > > >> persistence work.
> > > >> > > > > > >> > > > > > > > > > > > >> •   Schema refactor PR
> > ownership is
> > > >> not
> > > >> > > yet
> > > >> > > > > > >> decided.
> > > >> > > > > > >> > > If
> > > >> > > > > > >> > > > > > anyone
> > > >> > > > > > >> > > > > > > > is
> > > >> > > > > > >> > > > > > > > > > > > >> interested in driving it,
> reply
> > on
> > > >> this
> > > >> > > > > thread.
> > > >> > > > > > >> > > > > > > > > > > > >> •   JB to schedule the next
> > sync,
> > > >> > > tentatively
> > > >> > > > > > in
> > > >> > > > > > >> two
> > > >> > > > > > >> > > > > weeks.
> > > >> > > > > > >> > > > > > > > > > > > >>
> > > >> > > > > > >> > > > > > > > > > > > >> -ej
> > > >> > > > > > >> > > > > > > > > > > > >>
> > > >> > > > > > >> > > > > > > > > > > > >> On Mon, Apr 27, 2026 at
> 3:07 PM
> > EJ
> > > >> > Wang <
> > > >> > > > > > >> > > > > > > > > > > ej.wang.dev.interest@gmail.com
> > > >> > > > > > >> > > > > > > > > > > > >
> > > >> > > > > > >> > > > > > > > > > > > >> wrote:
> > > >> > > > > > >> > > > > > > > > > > > >>
> > > >> > > > > > >> > > > > > > > > > > > >> > Thanks Yufei for the +1.
> > > >> > > > > > >> > > > > > > > > > > > >> >
> > > >> > > > > > >> > > > > > > > > > > > >> > JB, could you help add a
> > biweekly
> > > >> > > metrics
> > > >> > > > > > >> > > architecture
> > > >> > > > > > >> > > > > > sync
> > > >> > > > > > >> > > > > > > to
> > > >> > > > > > >> > > > > > > > > the
> > > >> > > > > > >> > > > > > > > > > > > >> Polaris
> > > >> > > > > > >> > > > > > > > > > > > >> > community calendar? I'm
> > thinking
> > > >> > > Thursdays
> > > >> > > > > at
> > > >> > > > > > >> > 9-10am
> > > >> > > > > > >> > > > PT,
> > > >> > > > > > >> > > > > > on
> > > >> > > > > > >> > > > > > > > the
> > > >> > > > > > >> > > > > > > > > > > > >> off-weeks
> > > >> > > > > > >> > > > > > > > > > > > >> > from the community meeting
> > > >> (starting
> > > >> > > May
> > > >> > > > > 7),
> > > >> > > > > > 60
> > > >> > > > > > >> > > > minutes.
> > > >> > > > > > >> > > > > > > > > > > > >> >
> > > >> > > > > > >> > > > > > > > > > > > >> > Here's a rough agenda to
> work
> > > >> through
> > > >> > > over
> > > >> > > > > > the
> > > >> > > > > > >> > first
> > > >> > > > > > >> > > > few
> > > >> > > > > > >> > > > > > > > > sessions,
> > > >> > > > > > >> > > > > > > > > > > > >> grouped
> > > >> > > > > > >> > > > > > > > > > > > >> > by priority:
> > > >> > > > > > >> > > > > > > > > > > > >> >
> > > >> > > > > > >> > > > > > > > > > > > >> > *First: foundational
> > direction*
> > > >> > > > > > >> > > > > > > > > > > > >> >
> > > >> > > > > > >> > > > > > > > > > > > >> > 1.  MetricsPersistence:
> public
> > > >> SPI or
> > > >> > > > > > internal
> > > >> > > > > > >> > > > > > > implementation
> > > >> > > > > > >> > > > > > > > > > > detail?
> > > >> > > > > > >> > > > > > > > > > > > >> >    •   Marked @Beta, javadoc
> > calls
> > > >> > it a
> > > >> > > > > > >> "Service
> > > >> > > > > > >> > > > > Provider
> > > >> > > > > > >> > > > > > > > > > > Interface",
> > > >> > > > > > >> > > > > > > > > > > > >> but
> > > >> > > > > > >> > > > > > > > > > > > >> > only one consumer
> > > >> > > > > (JdbcBasePersistenceImpl),
> > > >> > > > > > >> lives
> > > >> > > > > > >> > > on
> > > >> > > > > > >> > > > > > > > > > > BasePersistence.
> > > >> > > > > > >> > > > > > > > > > > > >> If
> > > >> > > > > > >> > > > > > > > > > > > >> > demoted to a private helper
> > > >> inside a
> > > >> > > > > > persisting
> > > >> > > > > > >> > > > reporter
> > > >> > > > > > >> > > > > > > impl,
> > > >> > > > > > >> > > > > > > > > > most
> > > >> > > > > > >> > > > > > > > > > > > >> > downstream design decisions
> > become
> > > >> > > > > > >> implementation
> > > >> > > > > > >> > > > > details
> > > >> > > > > > >> > > > > > > > rather
> > > >> > > > > > >> > > > > > > > > > > than
> > > >> > > > > > >> > > > > > > > > > > > >> > contract questions.
> > > >> > > > > > >> > > > > > > > > > > > >> >
> > > >> > > > > > >> > > > > > > > > > > > >> > 2.  Persistence schema
> > redesign
> > > >> > > > > > >> > > > > > > > > > > > >> >    •   Current two-table
> > layout
> > > >> > > > > > >> > > (scan_metrics_report,
> > > >> > > > > > >> > > > > > > > > > > > >> > commit_metrics_report) with
> > ~25
> > > >> > > flattened
> > > >> > > > > > >> columns
> > > >> > > > > > >> > > > each.
> > > >> > > > > > >> > > > > > > Every
> > > >> > > > > > >> > > > > > > > > new
> > > >> > > > > > >> > > > > > > > > > > > metric
> > > >> > > > > > >> > > > > > > > > > > > >> > type requires a new table,
> SPI
> > > >> > method,
> > > >> > > > > record
> > > >> > > > > > >> > class,
> > > >> > > > > > >> > > > > > model,
> > > >> > > > > > >> > > > > > > > > > > converter,
> > > >> > > > > > >> > > > > > > > > > > > >> and
> > > >> > > > > > >> > > > > > > > > > > > >> > schema migration. Direction
> to
> > > >> > explore:
> > > >> > > > > > single
> > > >> > > > > > >> > table
> > > >> > > > > > >> > > > > with
> > > >> > > > > > >> > > > > > > > > > > metric_type
> > > >> > > > > > >> > > > > > > > > > > > >> enum,
> > > >> > > > > > >> > > > > > > > > > > > >> > schema_version, and JSON
> > payload
> > > >> > > column.
> > > >> > > > > > >> > > > > > > > > > > > >> >
> > > >> > > > > > >> > > > > > > > > > > > >> > *Second: design details once
> > > >> > direction
> > > >> > > is
> > > >> > > > > > set*
> > > >> > > > > > >> > > > > > > > > > > > >> >
> > > >> > > > > > >> > > > > > > > > > > > >> > 3.  Partition key strategy
> > > >> > > > > > >> > > > > > > > > > > > >> >    •   Single-table design
> > means
> > > >> scan
> > > >> > > > > metrics
> > > >> > > > > > >> at
> > > >> > > > > > >> > > scale
> > > >> > > > > > >> > > > > > will
> > > >> > > > > > >> > > > > > > > have
> > > >> > > > > > >> > > > > > > > > > > high
> > > >> > > > > > >> > > > > > > > > > > > >> > write concurrency per table.
> > > >> Schema
> > > >> > > needs
> > > >> > > > > to
> > > >> > > > > > >> > expose
> > > >> > > > > > >> > > > > enough
> > > >> > > > > > >> > > > > > > > > > structure
> > > >> > > > > > >> > > > > > > > > > > > for
> > > >> > > > > > >> > > > > > > > > > > > >> > backends to shard by entity
> or
> > > >> time
> > > >> > > range.
> > > >> > > > > > >> > > > > > > > > > > > >> >
> > > >> > > > > > >> > > > > > > > > > > > >> > 4.  Read/write path
> > consistency
> > > >> > > > > > >> > > > > > > > > > > > >> >    •   Writes go through
> > > >> > > > > > PolarisMetricsManager
> > > >> > > > > > >> on
> > > >> > > > > > >> > > > > > > > > > MetaStoreManager.
> > > >> > > > > > >> > > > > > > > > > > > >> Reads
> > > >> > > > > > >> > > > > > > > > > > > >> > bypass MetaStoreManager and
> go
> > > >> > > straight to
> > > >> > > > > > >> > > > > > BasePersistence,
> > > >> > > > > > >> > > > > > > > > > > excluding
> > > >> > > > > > >> > > > > > > > > > > > >> > non-JDBC backends from the
> > read
> > > >> API.
> > > >> > > > > > >> > > > > > > > > > > > >> >
> > > >> > > > > > >> > > > > > > > > > > > >> > *Third: cleanup and
> alignment*
> > > >> > > > > > >> > > > > > > > > > > > >> >
> > > >> > > > > > >> > > > > > > > > > > > >> > 5.  PolarisMetricsReporter
> > naming
> > > >> > > > > > >> > > > > > > > > > > > >> >    •   Only handles IRC
> > > >> > > > > > >> (ScanReport/CommitReport),
> > > >> > > > > > >> > > > > doesn't
> > > >> > > > > > >> > > > > > > > cover
> > > >> > > > > > >> > > > > > > > > > > > generic
> > > >> > > > > > >> > > > > > > > > > > > >> > tables or operational
> metrics.
> > > >> Name
> > > >> > is
> > > >> > > > > > broader
> > > >> > > > > > >> > than
> > > >> > > > > > >> > > > > scope.
> > > >> > > > > > >> > > > > > > > > > > > >> >
> > > >> > > > > > >> > > > > > > > > > > > >> > 6.  PolarisMetricsManager
> > facade
> > > >> > > > > passthrough
> > > >> > > > > > >> > > > > > > > > > > > >> >    •   Entire default method
> > is
> > > >> > > > > > >> > > > > > > > > > > > >>
> > > >> > callCtx.getMetaStore().writeScanReport().
> > > >> > > > > > >> > > > > > > > > > > > >> > Zero logic, passes Level 1
> > > >> straight
> > > >> > > through
> > > >> > > > > > to
> > > >> > > > > > >> > Level
> > > >> > > > > > >> > > > 3.
> > > >> > > > > > >> > > > > > Same
> > > >> > > > > > >> > > > > > > > > > > > >> anti-pattern
> > > >> > > > > > >> > > > > > > > > > > > >> > as PolarisEventManager.
> > > >> > > > > > >> > > > > > > > > > > > >> >
> > > >> > > > > > >> > > > > > > > > > > > >> > 7.  Iceberg community
> > alignment
> > > >> > > > > > >> > > > > > > > > > > > >> >    •   Payload-type
> extension
> > > >> needs
> > > >> > > > > > discussion
> > > >> > > > > > >> on
> > > >> > > > > > >> > > > > > > dev@iceberg.
> > > >> > > > > > >> > > > > > > > > > > > >> obelix74's
> > > >> > > > > > >> > > > > > > > > > > > >> > Feb thread got zero replies.
> > > >> Needs a
> > > >> > > > > > committer
> > > >> > > > > > >> > > voice.
> > > >> > > > > > >> > > > > > > > > > > > >> >
> > > >> > > > > > >> > > > > > > > > > > > >> > Lets confirm prioritization
> > in the
> > > >> > > first
> > > >> > > > > > >> session.
> > > >> > > > > > >> > > > > > > > > > > > >> >
> > > >> > > > > > >> > > > > > > > > > > > >> > -ej
> > > >> > > > > > >> > > > > > > > > > > > >> >
> > > >> > > > > > >> > > > > > > > > > > > >> > On Tue, Apr 21, 2026 at
> > 3:18 PM
> > > >> Yufei
> > > >> > > Gu <
> > > >> > > > > > >> > > > > > > > flyrain000@gmail.com>
> > > >> > > > > > >> > > > > > > > > > > > wrote:
> > > >> > > > > > >> > > > > > > > > > > > >> >
> > > >> > > > > > >> > > > > > > > > > > > >> >> Thanks everyone for
> > continuing to
> > > >> > > drive
> > > >> > > > > this
> > > >> > > > > > >> > > > forward. I
> > > >> > > > > > >> > > > > > > agree
> > > >> > > > > > >> > > > > > > > > > that
> > > >> > > > > > >> > > > > > > > > > > > the
> > > >> > > > > > >> > > > > > > > > > > > >> >> problem is getting complex
> > enough
> > > >> > > that a
> > > >> > > > > > more
> > > >> > > > > > >> > > > > structured
> > > >> > > > > > >> > > > > > > > > > discussion
> > > >> > > > > > >> > > > > > > > > > > > >> would
> > > >> > > > > > >> > > > > > > > > > > > >> >> help.
> > > >> > > > > > >> > > > > > > > > > > > >> >>
> > > >> > > > > > >> > > > > > > > > > > > >> >> +1 on setting up a biweekly
> > sync
> > > >> for
> > > >> > > the
> > > >> > > > > > >> metrics
> > > >> > > > > > >> > > > > > > > architecture.
> > > >> > > > > > >> > > > > > > > > > I’m
> > > >> > > > > > >> > > > > > > > > > > > >> happy
> > > >> > > > > > >> > > > > > > > > > > > >> >> to
> > > >> > > > > > >> > > > > > > > > > > > >> >> join.
> > > >> > > > > > >> > > > > > > > > > > > >> >>
> > > >> > > > > > >> > > > > > > > > > > > >> >> Yufei
> > > >> > > > > > >> > > > > > > > > > > > >> >>
> > > >> > > > > > >> > > > > > > > > > > > >> >>
> > > >> > > > > > >> > > > > > > > > > > > >> >> On Tue, Apr 21, 2026 at
> > 2:34 PM
> > > >> EJ
> > > >> > > Wang <
> > > >> > > > > > >> > > > > > > > > > > > >>
> ej.wang.dev.interest@gmail.com>
> > > >> > > > > > >> > > > > > > > > > > > >> >> wrote:
> > > >> > > > > > >> > > > > > > > > > > > >> >>
> > > >> > > > > > >> > > > > > > > > > > > >> >> > Also, I've been looking
> > more
> > > >> > > closely at
> > > >> > > > > > the
> > > >> > > > > > >> > > > > > *persistence
> > > >> > > > > > >> > > > > > > > > schema
> > > >> > > > > > >> > > > > > > > > > > in
> > > >> > > > > > >> > > > > > > > > > > > >> the
> > > >> > > > > > >> > > > > > > > > > > > >> >> > current metrics work*,
> and
> > I
> > > >> think
> > > >> > > > > > there's a
> > > >> > > > > > >> > > > > structural
> > > >> > > > > > >> > > > > > > > > > rigidity
> > > >> > > > > > >> > > > > > > > > > > > >> problem
> > > >> > > > > > >> > > > > > > > > > > > >> >> > worth raising before the
> > shape
> > > >> > gets
> > > >> > > > > locked
> > > >> > > > > > >> in.
> > > >> > > > > > >> > > > > > > > > > > > >> >> >
> > > >> > > > > > >> > > > > > > > > > > > >> >> > Right now we have two
> > separate
> > > >> > > tables
> > > >> > > > > > >> > > > > > > (scan_metrics_report
> > > >> > > > > > >> > > > > > > > > and
> > > >> > > > > > >> > > > > > > > > > > > >> >> > commit_metrics_report),
> > each
> > > >> with
> > > >> > > ~25
> > > >> > > > > > >> flattened
> > > >> > > > > > >> > > > > columns
> > > >> > > > > > >> > > > > > > > that
> > > >> > > > > > >> > > > > > > > > > > > directly
> > > >> > > > > > >> > > > > > > > > > > > >> >> > mirror the Iceberg report
> > > >> fields.
> > > >> > > The
> > > >> > > > > SPI
> > > >> > > > > > >> > follows
> > > >> > > > > > >> > > > the
> > > >> > > > > > >> > > > > > > same
> > > >> > > > > > >> > > > > > > > > > split:
> > > >> > > > > > >> > > > > > > > > > > > >> >> > writeScanReport and
> > > >> > > writeCommitReport as
> > > >> > > > > > >> > separate
> > > >> > > > > > >> > > > > > > methods,
> > > >> > > > > > >> > > > > > > > > with
> > > >> > > > > > >> > > > > > > > > > > > >> per-type
> > > >> > > > > > >> > > > > > > > > > > > >> >> > record classes,
> > converters, and
> > > >> > > model
> > > >> > > > > > >> objects.
> > > >> > > > > > >> > > *The
> > > >> > > > > > >> > > > > > > > practical
> > > >> > > > > > >> > > > > > > > > > > cost:
> > > >> > > > > > >> > > > > > > > > > > > >> >> > adding a new metric type
> > > >> > > (operational
> > > >> > > > > > >> metrics,
> > > >> > > > > > >> > > for
> > > >> > > > > > >> > > > > > > example)
> > > >> > > > > > >> > > > > > > > > > > > requires
> > > >> > > > > > >> > > > > > > > > > > > >> a
> > > >> > > > > > >> > > > > > > > > > > > >> >> new
> > > >> > > > > > >> > > > > > > > > > > > >> >> > table, a new SPI method,
> a
> > new
> > > >> > > record
> > > >> > > > > > >> class, a
> > > >> > > > > > >> > > new
> > > >> > > > > > >> > > > > > model
> > > >> > > > > > >> > > > > > > > > > class, a
> > > >> > > > > > >> > > > > > > > > > > > new
> > > >> > > > > > >> > > > > > > > > > > > >> >> > converter branch, and a
> > schema
> > > >> > > > > migration*.
> > > >> > > > > > >> > > That's a
> > > >> > > > > > >> > > > > lot
> > > >> > > > > > >> > > > > > > of
> > > >> > > > > > >> > > > > > > > > > > surface
> > > >> > > > > > >> > > > > > > > > > > > >> area
> > > >> > > > > > >> > > > > > > > > > > > >> >> > for what should be "one
> > more
> > > >> kind
> > > >> > of
> > > >> > > > > > >> metric."
> > > >> > > > > > >> > > > > > > > > > > > >> >> >
> > > >> > > > > > >> > > > > > > > > > > > >> >> > *My bias* would be
> toward a
> > > >> single
> > > >> > > > > metrics
> > > >> > > > > > >> > table
> > > >> > > > > > >> > > > with
> > > >> > > > > > >> > > > > > *a
> > > >> > > > > > >> > > > > > > > > typed
> > > >> > > > > > >> > > > > > > > > > > JSON
> > > >> > > > > > >> > > > > > > > > > > > >> >> > payload*. Something like:
> > > >> > > metric_type
> > > >> > > > > > >> (enum),
> > > >> > > > > > >> > > > > > entity_id,
> > > >> > > > > > >> > > > > > > > > > > > >> >> > table_identifier,
> > snapshot_id
> > > >> > > > > (nullable),
> > > >> > > > > > >> > > > > received_ts,
> > > >> > > > > > >> > > > > > > > > > > > >> schema_version,
> > > >> > > > > > >> > > > > > > > > > > > >> >> and
> > > >> > > > > > >> > > > > > > > > > > > >> >> > a payload column for the
> > > >> > > metric-specific
> > > >> > > > > > >> data.
> > > >> > > > > > >> > > The
> > > >> > > > > > >> > > > > > > > > metric_type
> > > >> > > > > > >> > > > > > > > > > +
> > > >> > > > > > >> > > > > > > > > > > > >> >> > schema_version pair gives
> > us a
> > > >> > > > > > >> > forward-compatible
> > > >> > > > > > >> > > > > > > contract
> > > >> > > > > > >> > > > > > > > > for
> > > >> > > > > > >> > > > > > > > > > > the
> > > >> > > > > > >> > > > > > > > > > > > >> >> payload
> > > >> > > > > > >> > > > > > > > > > > > >> >> > shape. Adding a new
> metric
> > type
> > > >> > > becomes
> > > >> > > > > an
> > > >> > > > > > >> enum
> > > >> > > > > > >> > > > value
> > > >> > > > > > >> > > > > > > and a
> > > >> > > > > > >> > > > > > > > > > > payload
> > > >> > > > > > >> > > > > > > > > > > > >> >> schema,
> > > >> > > > > > >> > > > > > > > > > > > >> >> > not a schema migration.
> One
> > > >> thing
> > > >> > I
> > > >> > > > > think
> > > >> > > > > > we
> > > >> > > > > > >> > need
> > > >> > > > > > >> > > > to
> > > >> > > > > > >> > > > > be
> > > >> > > > > > >> > > > > > > > > > > deliberate
> > > >> > > > > > >> > > > > > > > > > > > >> >> about is
> > > >> > > > > > >> > > > > > > > > > > > >> >> > the partition key design.
> > If
> > > >> all
> > > >> > > metric
> > > >> > > > > > >> types
> > > >> > > > > > >> > > land
> > > >> > > > > > >> > > > in
> > > >> > > > > > >> > > > > > one
> > > >> > > > > > >> > > > > > > > > > table,
> > > >> > > > > > >> > > > > > > > > > > > scan
> > > >> > > > > > >> > > > > > > > > > > > >> >> > metrics at scale (high
> > > >> > concurrency,
> > > >> > > high
> > > >> > > > > > >> > > frequency
> > > >> > > > > > >> > > > > > across
> > > >> > > > > > >> > > > > > > > > many
> > > >> > > > > > >> > > > > > > > > > > > >> tables)
> > > >> > > > > > >> > > > > > > > > > > > >> >> > could easily create hot
> > > >> > partitions.
> > > >> > > We'd
> > > >> > > > > > >> want
> > > >> > > > > > >> > the
> > > >> > > > > > >> > > > > > > > persistence
> > > >> > > > > > >> > > > > > > > > > > layer
> > > >> > > > > > >> > > > > > > > > > > > >> to
> > > >> > > > > > >> > > > > > > > > > > > >> >> be
> > > >> > > > > > >> > > > > > > > > > > > >> >> > able to shard by entity
> or
> > time
> > > >> > > range,
> > > >> > > > > and
> > > >> > > > > > >> that
> > > >> > > > > > >> > > > means
> > > >> > > > > > >> > > > > > the
> > > >> > > > > > >> > > > > > > > > > logical
> > > >> > > > > > >> > > > > > > > > > > > >> schema
> > > >> > > > > > >> > > > > > > > > > > > >> >> > needs to expose enough
> > > >> structure
> > > >> > for
> > > >> > > > > > >> backends
> > > >> > > > > > >> > to
> > > >> > > > > > >> > > > > > > partition
> > > >> > > > > > >> > > > > > > > > on.
> > > >> > > > > > >> > > > > > > > > > I
> > > >> > > > > > >> > > > > > > > > > > > >> don't
> > > >> > > > > > >> > > > > > > > > > > > >> >> > think the current
> flattened
> > > >> layout
> > > >> > > gives
> > > >> > > > > > us
> > > >> > > > > > >> > that.
> > > >> > > > > > >> > > > > > > > > > > > >> >> >
> > > >> > > > > > >> > > > > > > > > > > > >> >> > This is getting complex
> > enough
> > > >> > that
> > > >> > > I
> > > >> > > > > > don't
> > > >> > > > > > >> > think
> > > >> > > > > > >> > > > > > ad-hoc
> > > >> > > > > > >> > > > > > > > > PR/ML
> > > >> > > > > > >> > > > > > > > > > > > >> threads
> > > >> > > > > > >> > > > > > > > > > > > >> >> > will converge well.
> *Would
> > > >> people
> > > >> > be
> > > >> > > > > open
> > > >> > > > > > >> to a
> > > >> > > > > > >> > > > > biweekly
> > > >> > > > > > >> > > > > > > > sync
> > > >> > > > > > >> > > > > > > > > > for
> > > >> > > > > > >> > > > > > > > > > > > >> metrics
> > > >> > > > > > >> > > > > > > > > > > > >> >> > architecture?* I think 30
> > > >> minutes
> > > >> > > every
> > > >> > > > > > two
> > > >> > > > > > >> > weeks
> > > >> > > > > > >> > > > > with
> > > >> > > > > > >> > > > > > > > > > interested
> > > >> > > > > > >> > > > > > > > > > > > >> >> parties
> > > >> > > > > > >> > > > > > > > > > > > >> >> > would be enough to work
> > through
> > > >> > the
> > > >> > > > > > schema,
> > > >> > > > > > >> SPI
> > > >> > > > > > >> > > > > shape,
> > > >> > > > > > >> > > > > > > and
> > > >> > > > > > >> > > > > > > > > read
> > > >> > > > > > >> > > > > > > > > > > API
> > > >> > > > > > >> > > > > > > > > > > > >> >> design
> > > >> > > > > > >> > > > > > > > > > > > >> >> > together. Happy to help
> set
> > > >> that
> > > >> > up.
> > > >> > > > > > >> > > > > > > > > > > > >> >> >
> > > >> > > > > > >> > > > > > > > > > > > >> >> > -ej
> > > >> > > > > > >> > > > > > > > > > > > >> >> >
> > > >> > > > > > >> > > > > > > > > > > > >> >> > On Mon, Apr 20, 2026 at
> > > >> 2:19 PM EJ
> > > >> > > Wang
> > > >> > > > > <
> > > >> > > > > > >> > > > > > > > > > > > >>
> ej.wang.dev.interest@gmail.com
> > > >> > > > > > >> > > > > > > > > > > > >> >> >
> > > >> > > > > > >> > > > > > > > > > > > >> >> > wrote:
> > > >> > > > > > >> > > > > > > > > > > > >> >> >
> > > >> > > > > > >> > > > > > > > > > > > >> >> >> Reviewed #4115, left a
> > > >> comment on
> > > >> > > the
> > > >> > > > > > code
> > > >> > > > > > >> > > > > > organization
> > > >> > > > > > >> > > > > > > > > side.
> > > >> > > > > > >> > > > > > > > > > > > >> >> >>
> > > >> > > > > > >> > > > > > > > > > > > >> >> >> One thing stood out: the
> > > >> metrics
> > > >> > > write
> > > >> > > > > > path
> > > >> > > > > > >> > > enters
> > > >> > > > > > >> > > > > > > through
> > > >> > > > > > >> > > > > > > > > > > > >> >> >> PolarisMetricsManager on
> > > >> > > > > > MetaStoreManager,
> > > >> > > > > > >> but
> > > >> > > > > > >> > > the
> > > >> > > > > > >> > > > > new
> > > >> > > > > > >> > > > > > > > read
> > > >> > > > > > >> > > > > > > > > > path
> > > >> > > > > > >> > > > > > > > > > > > >> >> bypasses
> > > >> > > > > > >> > > > > > > > > > > > >> >> >> MetaStoreManager
> entirely
> > and
> > > >> > goes
> > > >> > > > > > >> straight to
> > > >> > > > > > >> > > > > > > > > BasePersistence
> > > >> > > > > > >> > > > > > > > > > > via
> > > >> > > > > > >> > > > > > > > > > > > >> >> >>
> > callContext.getMetaStore().
> > > >> That
> > > >> > > means
> > > >> > > > > > the
> > > >> > > > > > >> > read
> > > >> > > > > > >> > > > API
> > > >> > > > > > >> > > > > > only
> > > >> > > > > > >> > > > > > > > > works
> > > >> > > > > > >> > > > > > > > > > > for
> > > >> > > > > > >> > > > > > > > > > > > >> >> backends
> > > >> > > > > > >> > > > > > > > > > > > >> >> >> that implement
> > > >> BasePersistence.
> > > >> > > NoSQL
> > > >> > > > > and
> > > >> > > > > > >> > remote
> > > >> > > > > > >> > > > > > > backends
> > > >> > > > > > >> > > > > > > > > > can't
> > > >> > > > > > >> > > > > > > > > > > > >> >> participate.
> > > >> > > > > > >> > > > > > > > > > > > >> >> >>
> > > >> > > > > > >> > > > > > > > > > > > >> >> >> Stepping back, I think
> the
> > > >> > metrics
> > > >> > > > > > >> subsystem
> > > >> > > > > > >> > is
> > > >> > > > > > >> > > > > > growing
> > > >> > > > > > >> > > > > > > > into
> > > >> > > > > > >> > > > > > > > > > > > >> something
> > > >> > > > > > >> > > > > > > > > > > > >> >> >> real (write + read +
> REST
> > API
> > > >> +
> > > >> > > AuthZ +
> > > >> > > > > > >> > > > pagination)
> > > >> > > > > > >> > > > > > *but
> > > >> > > > > > >> > > > > > > > the
> > > >> > > > > > >> > > > > > > > > > > > >> >> persistence
> > > >> > > > > > >> > > > > > > > > > > > >> >> >> side is split across two
> > > >> layers
> > > >> > in
> > > >> > > a
> > > >> > > > > way
> > > >> > > > > > >> > that's
> > > >> > > > > > >> > > > hard
> > > >> > > > > > >> > > > > > to
> > > >> > > > > > >> > > > > > > > > > > extend*. I
> > > >> > > > > > >> > > > > > > > > > > > >> put
> > > >> > > > > > >> > > > > > > > > > > > >> >> >> together two diagrams to
> > show
> > > >> > what
> > > >> > > I
> > > >> > > > > mean
> > > >> > > > > > >> (my
> > > >> > > > > > >> > > best
> > > >> > > > > > >> > > > > > > > effort).
> > > >> > > > > > >> > > > > > > > > > > > >> >> >>
> > > >> > > > > > >> > > > > > > > > > > > >> >> >> *Current state* (Diagram
> > 1):
> > > >> > three
> > > >> > > > > > >> interfaces
> > > >> > > > > > >> > at
> > > >> > > > > > >> > > > > three
> > > >> > > > > > >> > > > > > > > > > different
> > > >> > > > > > >> > > > > > > > > > > > >> >> levels.
> > > >> > > > > > >> > > > > > > > > > > > >> >> >> The engine-facing SPI
> > > >> > > > > > >> (PolarisMetricsReporter)
> > > >> > > > > > >> > > is
> > > >> > > > > > >> > > > > > clean.
> > > >> > > > > > >> > > > > > > > But
> > > >> > > > > > >> > > > > > > > > > > > >> >> >> PolarisMetricsManager on
> > > >> > > > > MetaStoreManager
> > > >> > > > > > >> is a
> > > >> > > > > > >> > > > > > > passthrough
> > > >> > > > > > >> > > > > > > > > to
> > > >> > > > > > >> > > > > > > > > > > > >> >> >> MetricsPersistence on
> > > >> > > BasePersistence.
> > > >> > > > > > The
> > > >> > > > > > >> > @Beta
> > > >> > > > > > >> > > > > > > > annotation
> > > >> > > > > > >> > > > > > > > > > and
> > > >> > > > > > >> > > > > > > > > > > > SPI
> > > >> > > > > > >> > > > > > > > > > > > >> >> javadoc
> > > >> > > > > > >> > > > > > > > > > > > >> >> >> are on the
> BasePersistence
> > > >> layer,
> > > >> > > while
> > > >> > > > > > the
> > > >> > > > > > >> > > actual
> > > >> > > > > > >> > > > > > > > extension
> > > >> > > > > > >> > > > > > > > > > > > points
> > > >> > > > > > >> > > > > > > > > > > > >> >> >> (PolarisMetricsReporter,
> > > >> > > > > > >> > PolarisMetricsManager)
> > > >> > > > > > >> > > > > carry
> > > >> > > > > > >> > > > > > no
> > > >> > > > > > >> > > > > > > > > > > stability
> > > >> > > > > > >> > > > > > > > > > > > >> >> >> annotation. The write
> path
> > > >> goes
> > > >> > > through
> > > >> > > > > > the
> > > >> > > > > > >> > > > > > > > MetaStoreManager
> > > >> > > > > > >> > > > > > > > > > > > layer,
> > > >> > > > > > >> > > > > > > > > > > > >> the
> > > >> > > > > > >> > > > > > > > > > > > >> >> >> read path doesn't.
> > > >> > > > > > >> > > > > > > > > > > > >> >> >>
> > > >> > > > > > >> > > > > > > > > > > > >> >> >> *What I envision*
> > (Diagram 2):
> > > >> > two
> > > >> > > SPIs
> > > >> > > > > > at
> > > >> > > > > > >> two
> > > >> > > > > > >> > > > > levels.
> > > >> > > > > > >> > > > > > > > > > > > >> >> >> PolarisMetricsReporter
> > stays
> > > >> as
> > > >> > the
> > > >> > > > > > >> > > engine-facing
> > > >> > > > > > >> > > > > SPI.
> > > >> > > > > > >> > > > > > > > > > > > >> >> >> PolarisMetricsManager
> > becomes
> > > >> the
> > > >> > > > > > >> > backend-facing
> > > >> > > > > > >> > > > SPI
> > > >> > > > > > >> > > > > > > with
> > > >> > > > > > >> > > > > > > > > both
> > > >> > > > > > >> > > > > > > > > > > > write
> > > >> > > > > > >> > > > > > > > > > > > >> >> and
> > > >> > > > > > >> > > > > > > > > > > > >> >> >> read methods at the
> > > >> > > MetaStoreManager
> > > >> > > > > > level,
> > > >> > > > > > >> > > where
> > > >> > > > > > >> > > > > any
> > > >> > > > > > >> > > > > > > > > backend
> > > >> > > > > > >> > > > > > > > > > > > (JDBC,
> > > >> > > > > > >> > > > > > > > > > > > >> >> NoSQL,
> > > >> > > > > > >> > > > > > > > > > > > >> >> >> remote) can implement
> > them.
> > > >> > > > > > >> MetricsPersistence
> > > >> > > > > > >> > > on
> > > >> > > > > > >> > > > > > > > > > > BasePersistence
> > > >> > > > > > >> > > > > > > > > > > > >> goes
> > > >> > > > > > >> > > > > > > > > > > > >> >> >> away. Where metrics
> > actually
> > > >> land
> > > >> > > is an
> > > >> > > > > > >> > > > > implementation
> > > >> > > > > > >> > > > > > > > > detail,
> > > >> > > > > > >> > > > > > > > > > > > not a
> > > >> > > > > > >> > > > > > > > > > > > >> >> core
> > > >> > > > > > >> > > > > > > > > > > > >> >> >> interface.
> > > >> > > > > > >> > > > > > > > > > > > >> >> >>
> > > >> > > > > > >> > > > > > > > > > > > >> >> >> *Minor naming thing*:
> > > >> > > > > > >> PolarisMetricsReporter
> > > >> > > > > > >> > is
> > > >> > > > > > >> > > > > > broader
> > > >> > > > > > >> > > > > > > > than
> > > >> > > > > > >> > > > > > > > > > > what
> > > >> > > > > > >> > > > > > > > > > > > it
> > > >> > > > > > >> > > > > > > > > > > > >> >> >> actually handles. It
> only
> > > >> accepts
> > > >> > > > > Iceberg
> > > >> > > > > > >> REST
> > > >> > > > > > >> > > > > Catalog
> > > >> > > > > > >> > > > > > > > > metrics
> > > >> > > > > > >> > > > > > > > > > > > >> >> (ScanReport,
> > > >> > > > > > >> > > > > > > > > > > > >> >> >> CommitReport via
> > > >> MetricsReport).
> > > >> > > > > Generic
> > > >> > > > > > >> table
> > > >> > > > > > >> > > > > metrics
> > > >> > > > > > >> > > > > > > or
> > > >> > > > > > >> > > > > > > > > > > > >> operational
> > > >> > > > > > >> > > > > > > > > > > > >> >> >> metrics aren't in scope.
> > Not
> > > >> > > blocking,
> > > >> > > > > > but
> > > >> > > > > > >> > worth
> > > >> > > > > > >> > > > > > noting
> > > >> > > > > > >> > > > > > > if
> > > >> > > > > > >> > > > > > > > > the
> > > >> > > > > > >> > > > > > > > > > > > >> metrics
> > > >> > > > > > >> > > > > > > > > > > > >> >> >> surface expands.
> > > >> > > > > > >> > > > > > > > > > > > >> >> >>
> > > >> > > > > > >> > > > > > > > > > > > >> >> >> *Rough sketch of how to
> > get
> > > >> > there*:
> > > >> > > > > > >> > > > > > > > > > > > >> >> >>  1.  Add read methods to
> > > >> > > > > > >> PolarisMetricsManager
> > > >> > > > > > >> > > > > > > > > > (listScanReports,
> > > >> > > > > > >> > > > > > > > > > > > >> >> >> listCommitReports) with
> > > >> default
> > > >> > > no-op,
> > > >> > > > > > >> same as
> > > >> > > > > > >> > > the
> > > >> > > > > > >> > > > > > > > existing
> > > >> > > > > > >> > > > > > > > > > > write
> > > >> > > > > > >> > > > > > > > > > > > >> >> methods.
> > > >> > > > > > >> > > > > > > > > > > > >> >> >> (Probably make
> > > >> > > PolarisMetricsManager
> > > >> > > > > more
> > > >> > > > > > >> > > explicit
> > > >> > > > > > >> > > > > on
> > > >> > > > > > >> > > > > > > > being
> > > >> > > > > > >> > > > > > > > > > > > Iceberg
> > > >> > > > > > >> > > > > > > > > > > > >> >> >> specific like package
> > name or
> > > >> > class
> > > >> > > > > name
> > > >> > > > > > >> etc.)
> > > >> > > > > > >> > > > > > > > > > > > >> >> >>  2.  Wire
> > > >> MetricsReportsService
> > > >> > > through
> > > >> > > > > > >> > > > > > MetaStoreManager
> > > >> > > > > > >> > > > > > > > > > instead
> > > >> > > > > > >> > > > > > > > > > > > of
> > > >> > > > > > >> > > > > > > > > > > > >> >> >>
> > callContext.getMetaStore().
> > > >> > > > > > >> > > > > > > > > > > > >> >> >>  3.  Extract metrics
> > > >> persistence
> > > >> > > from
> > > >> > > > > > >> > > > > > > > > JdbcBasePersistenceImpl
> > > >> > > > > > >> > > > > > > > > > > into
> > > >> > > > > > >> > > > > > > > > > > > >> its
> > > >> > > > > > >> > > > > > > > > > > > >> >> >> own class. That file
> > carries
> > > >> ~7
> > > >> > > > > > >> > > responsibilities,
> > > >> > > > > > >> > > > > > > metrics
> > > >> > > > > > >> > > > > > > > > > being
> > > >> > > > > > >> > > > > > > > > > > > one
> > > >> > > > > > >> > > > > > > > > > > > >> of
> > > >> > > > > > >> > > > > > > > > > > > >> >> them.
> > > >> > > > > > >> > > > > > > > > > > > >> >> >>  4.  Remove
> > MetricsPersistence
> > > >> > from
> > > >> > > > > > >> > > > BasePersistence.
> > > >> > > > > > >> > > > > > > > > > > > >> >> >>
> > > >> > > > > > >> > > > > > > > > > > > >> >> >> *None of this needs to
> > happen
> > > >> in
> > > >> > > #4115.
> > > >> > > > > > >> But if
> > > >> > > > > > >> > > the
> > > >> > > > > > >> > > > > > > > direction
> > > >> > > > > > >> > > > > > > > > > > makes
> > > >> > > > > > >> > > > > > > > > > > > >> >> sense,
> > > >> > > > > > >> > > > > > > > > > > > >> >> >> it would be good to
> align
> > > >> before
> > > >> > > the
> > > >> > > > > > >> metrics
> > > >> > > > > > >> > > > surface
> > > >> > > > > > >> > > > > > > grows
> > > >> > > > > > >> > > > > > > > > > > > further.
> > > >> > > > > > >> > > > > > > > > > > > >> >> Curious
> > > >> > > > > > >> > > > > > > > > > > > >> >> >> what others think.*
> > > >> > > > > > >> > > > > > > > > > > > >> >> >>
> > > >> > > > > > >> > > > > > > > > > > > >> >> >> *My mental model note*:
> > Level
> > > >> 1
> > > >> > > > > > >> > > MetaStoreManager;
> > > >> > > > > > >> > > > > > level
> > > >> > > > > > >> > > > > > > 2
> > > >> > > > > > >> > > > > > > > > > > > >> transactional
> > > >> > > > > > >> > > > > > > > > > > > >> >> >> persistence; level 3
> base
> > > >> > > persistence
> > > >> > > > > > >> > > > > > > > > > > > >> >> >>
> > > >> > > > > > >> > > > > > > > > > > > >> >> >> Diagram 1
> > > >> > > > > > >> > > > > > > > > > > > >> >> >> <
> > > >> > > > > > >> > > > > > > > > > > > >> >>
> > > >> > > > > > >> > > > > > > > > > > > >>
> > > >> > > > > > >> > > > > > > > > > > >
> > > >> > > > > > >> > > > > > > > > > >
> > > >> > > > > > >> > > > > > > > > >
> > > >> > > > > > >> > > > > > > > >
> > > >> > > > > > >> > > > > > > >
> > > >> > > > > > >> > > > > > >
> > > >> > > > > > >> > > > > >
> > > >> > > > > > >> > > > >
> > > >> > > > > > >> > > >
> > > >> > > > > > >> > >
> > > >> > > > > > >> >
> > > >> > > > > > >>
> > > >> > > > > >
> > > >> > > > >
> > > >> > >
> > > >> >
> > > >>
> >
> https://urldefense.com/v3/__https://www.plantuml.com/plantuml/uml/bLHDR-Cs4BthLmpIYupw0zbkKQ1r3M-S7Bp8xhhM7WCOb3IM65EaGD9EX2RzxHrHb4CxRelwa4YSDu_lpOVcnZ9jzvM8BBS2uGjQpJC3dtHMSekPtMk44IpsMgEqa5XcCOhCZikQQLP1pR8TAp2n3ILhmZDP20m0fcIvUkAoW2qJXd9z1bpToO9BX3WXu0ucy5rpgGPNm0nW5_epUWtm2Ue3pn3kMOFQmKntGZW0BYtgBSi8k5A2QMwybJNMIbFiGSR9QZc4nUqIvikStF0jHprua5C-amge42aNt3R0f5JaaoivdV2Pkqbx4hee4ymOkBh5BTiB-_uIeGeo8zL8rPsPl4DktdEiK1jkB1NdZCRbrSTecDe_mlHbF0wvBmCkaOH5_S8a_TTTKI6-nmCAkEw4LpxsZ-LbYLKQFKMNOgf_wuM7_bV9gOer5SYMMksBSWXFcbi49KNZXNLicwfe3TETC7gPdPqI7uBcHMb1RSzYq34c6PDUM9mn8HRsUTZEiDBve3NjVZumBj0U7SS37mGO7vcwtiK-_pU7U7L_f-digo9YbhSwIfMRwIITKGXbxdIUTCGF1SeCJxloKsU-3k9ddRbX1eDq1q_fx1JbBGT0glVyXimDuP4TQ5qpCAmnGEj2s_6n5mtn1z-97-63itFQZLPO1Ev2tu_WF7Ju-VPc0Skg5bYXxBhkY1xpD7EM_7fyflSpIsqMgVth5xhVr4eQxWQ8enaSAJQSG16yFSDuJ798rrcXr_3n-lfdk7icQjEBmFujL7AodiP_Y4Z7-YxvtZNs4zMgpNTl6tF8sglyPsmqchrjvQ-m-aP94r-TwCA2Ka8upPJZwtvSpoYCXkYMZU2NXvRMBfq9P3i3Le4VAZUAlUZ_oPKsxPgY0Q_BSKLkyr9bhQhQrJjo_x3TPlIB0DPjnMfcIoYP0QaYw1a0fTKDr8fB6ntNuvmoL1ZGkXa69Njh43zf9GiGxHQrA_jDYWRSzF5--WmTVrN97_Sm8LbLUy_lGBmLanJjFkDlGkRqjA_4tm00__;!!Iz9xO38YGHZK!4rnlVIqBQQQ6xpGe1PMV58edOS_cGrKoqdfhw4qSp_JQCavvEjaxXk0OwGPnxyEcO60AzKQK_XDtp7H3DdkFXcrXuTBISHg$
> > > >> > > > > > >> > > > > > > > > > > > >> >> >
> > > >> > > > > > >> > > > > > > > > > > > >> >> >> :
> > > >> > > > > > >> > > > > > > > > > > > >> >> >>
> > > >> > > > > > >> > > > > > > > > > > > >> >> >> [image: image.png]
> > > >> > > > > > >> > > > > > > > > > > > >> >> >>
> > > >> > > > > > >> > > > > > > > > > > > >> >> >> Diagram 2
> > > >> > > > > > >> > > > > > > > > > > > >> >> >> <
> > > >> > > > > > >> > > > > > > > > > > > >> >>
> > > >> > > > > > >> > > > > > > > > > > > >>
> > > >> > > > > > >> > > > > > > > > > > >
> > > >> > > > > > >> > > > > > > > > > >
> > > >> > > > > > >> > > > > > > > > >
> > > >> > > > > > >> > > > > > > > >
> > > >> > > > > > >> > > > > > > >
> > > >> > > > > > >> > > > > > >
> > > >> > > > > > >> > > > > >
> > > >> > > > > > >> > > > >
> > > >> > > > > > >> > > >
> > > >> > > > > > >> > >
> > > >> > > > > > >> >
> > > >> > > > > > >>
> > > >> > > > > >
> > > >> > > > >
> > > >> > >
> > > >> >
> > > >>
> >
> https://urldefense.com/v3/__https://www.plantuml.com/plantuml/uml/VLLDR-8m4BtdLupO2sWBLVU8AaGB7AXAbssGzb896SS9RXqxjKqBwkv_tt7iV43fdaZYDpFlpRmnOsE9jhjSH9PRmM31hERKm8scMsuPjJlDe0yheZDc8RR4iYWoBrmMH9CS2a9VICPYUy1OZN0YCy5Q0BCbYNhdCeEK28En8G8wCvbnoQ0R8_05Bc6bkLIz3X03p1zzH7zR-9ZfDquPt9C3qoNCX2yV4G2NbkcKu5jdgGJHt0GbZwnG6i-UP3TUpk5gM6Ldqke350eZUqzoCft3U9xWHvxoa5-7K4nF1J46EbEMafsmdrCBbQ44gVggy18IZrn_ph5asd1ZiIKdQSgueZvjXrQFSFrdC3YN-nXmBacxbGiYyLVxLaBtdhqn0LSzdBDhqQtQoOJeGyad3z0lUqnYgpGB6Ns8oVyta00Dy_WnX0tIOZ8v6SYxHll1TrH6aejAik-mh-AphVFCwSUQqFypElag5QRGFDjQKEd96K1P8QP41c9TzA_IIQyvdAWyv_RSiS3skb0_EzDDkK2v5xWF6MiGFlvhpFLcD2Dq2pml14gaF67eQkmd8gulDoC4kSOu6KVpkvlUJg1RTbWISU40RdBUUS_9XfRZ2dwxm_SW8LYFISgm_MnlDQ6M9P1gbKEc4X-2pH_FvJCkCqm9pbVjD6LrwdLeOrDWfOaqc8Wh9BE85oNKxkNQ6o4yGRy_Eae0G_G8tZv81d3bHDB23WOdisohVr3nh_j6lbSjbNaLRTc8UgtPbAU1J_tygOfZX9DWEJeHDvYx-qmSi5FgNLPZwHrHcUsncGQ5-skhUclpE5fo4ounpFauYrUbkU6ccfnxMvitwag4IyerhTxj8In_Oj1bDO4pQru674loYrGlULHLEGCjwJJ8gDoVZR8MxO4BT3IzRvIcAQKezC6xpziGnTyImrfEGyJI_OcKfgtxIvnTqFEMS17L9Z-jsARN5FmTheP7HtSdtOMT0B4GY2FYHXxgQmMtj2bRqiLFGapiVe1_QVKDrkqXcm83aFEXnMYCZ-xlyHy__;!!Iz9xO38YGHZK!4rnlVIqBQQQ6xpGe1PMV58edOS_cGrKoqdfhw4qSp_JQCavvEjaxXk0OwGPnxyEcO60AzKQK_XDtp7H3DdkFXcrXSpmqcNk$
> > > >> > > > > > >> > > > > > > > > > > > >> >> >
> > > >> > > > > > >> > > > > > > > > > > > >> >> >> :
> > > >> > > > > > >> > > > > > > > > > > > >> >> >> [image: image.png]
> > > >> > > > > > >> > > > > > > > > > > > >> >> >>
> > > >> > > > > > >> > > > > > > > > > > > >> >> >>  -ej
> > > >> > > > > > >> > > > > > > > > > > > >> >> >>
> > > >> > > > > > >> > > > > > > > > > > > >> >> >> On Wed, Apr 15, 2026 at
> > > >> 8:22 AM
> > > >> > > Dmitri
> > > >> > > > > > >> > > > Bourlatchkov
> > > >> > > > > > >> > > > > <
> > > >> > > > > > >> > > > > > > > > > > > >> dimas@apache.org>
> > > >> > > > > > >> > > > > > > > > > > > >> >> >> wrote:
> > > >> > > > > > >> > > > > > > > > > > > >> >> >>
> > > >> > > > > > >> > > > > > > > > > > > >> >> >>> Hi All,
> > > >> > > > > > >> > > > > > > > > > > > >> >> >>>
> > > >> > > > > > >> > > > > > > > > > > > >> >> >>> Heads up: The current
> > state
> > > >> of
> > > >> > PR
> > > >> > > > > [4115]
> > > >> > > > > > >> > looks
> > > >> > > > > > >> > > > > pretty
> > > >> > > > > > >> > > > > > > > solid
> > > >> > > > > > >> > > > > > > > > > to
> > > >> > > > > > >> > > > > > > > > > > > me.
> > > >> > > > > > >> > > > > > > > > > > > >> I
> > > >> > > > > > >> > > > > > > > > > > > >> >> >>> believe this PR is
> > > >> approaching a
> > > >> > > > > > mergeable
> > > >> > > > > > >> > > > > condition.
> > > >> > > > > > >> > > > > > > > > > > > >> >> >>>
> > > >> > > > > > >> > > > > > > > > > > > >> >> >>> Please post your
> reviews
> > if
> > > >> you
> > > >> > > have
> > > >> > > > > any
> > > >> > > > > > >> > > > comments.
> > > >> > > > > > >> > > > > > > > > > > > >> >> >>>
> > > >> > > > > > >> > > > > > > > > > > > >> >> >>> [4115]
> > > >> > > > > > >> > > >
> > > >> >
> > > >>
> >
> https://urldefense.com/v3/__https://github.com/apache/polaris/pull/4115__;!!Iz9xO38YGHZK!4rnlVIqBQQQ6xpGe1PMV58edOS_cGrKoqdfhw4qSp_JQCavvEjaxXk0OwGPnxyEcO60AzKQK_XDtp7H3DdkFXcrXtFRscKs$
> > > >> > > > > > >> > > > > > > > > > > > >> >> >>>
> > > >> > > > > > >> > > > > > > > > > > > >> >> >>> Thanks,
> > > >> > > > > > >> > > > > > > > > > > > >> >> >>> Dmitri.
> > > >> > > > > > >> > > > > > > > > > > > >> >> >>>
> > > >> > > > > > >> > > > > > > > > > > > >> >> >>> On Tue, Mar 3, 2026 at
> > > >> 3:29 PM
> > > >> > > Anand
> > > >> > > > > > Kumar
> > > >> > > > > > >> > > > Sankaran
> > > >> > > > > > >> > > > > > via
> > > >> > > > > > >> > > > > > > > > dev <
> > > >> > > > > > >> > > > > > > > > > > > >> >> >>> dev@polaris.apache.org
> >
> > > >> wrote:
> > > >> > > > > > >> > > > > > > > > > > > >> >> >>>
> > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > Hi Yufei and Dmitri,
> > > >> > > > > > >> > > > > > > > > > > > >> >> >>> >
> > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > Here is a proposal
> for
> > the
> > > >> > REST
> > > >> > > > > > >> endpoints
> > > >> > > > > > >> > for
> > > >> > > > > > >> > > > > > metrics
> > > >> > > > > > >> > > > > > > > and
> > > >> > > > > > >> > > > > > > > > > > > events.
> > > >> > > > > > >> > > > > > > > > > > > >> >> >>> >
> > > >> > > > > > >> > > > > > > > > > > > >> >> >>> >
> > > >> > > > > > >> > > > > >
> > > >> >
> > > >>
> >
> https://urldefense.com/v3/__https://github.com/apache/polaris/pull/3924/changes__;!!Iz9xO38YGHZK!4rnlVIqBQQQ6xpGe1PMV58edOS_cGrKoqdfhw4qSp_JQCavvEjaxXk0OwGPnxyEcO60AzKQK_XDtp7H3DdkFXcrXcjoPhSs$
> > > >> > > > > > >> > > > > > > > > > > > >> >> >>> >
> > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > I did not see any
> > > >> precursors
> > > >> > for
> > > >> > > > > > >> raising a
> > > >> > > > > > >> > PR
> > > >> > > > > > >> > > > for
> > > >> > > > > > >> > > > > > > > > > proposals,
> > > >> > > > > > >> > > > > > > > > > > so
> > > >> > > > > > >> > > > > > > > > > > > >> >> trying
> > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > this.  Please let me
> > know
> > > >> what
> > > >> > > you
> > > >> > > > > > >> think.
> > > >> > > > > > >> > > > > > > > > > > > >> >> >>> >
> > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > -
> > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > Anand
> > > >> > > > > > >> > > > > > > > > > > > >> >> >>> >
> > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > From: Anand Kumar
> > Sankaran
> > > >> <
> > > >> > > > > > >> > > > > > > anand.sankaran@workday.com
> > > >> > > > > > >> > > > > > > > >
> > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > Date: Monday, March
> 2,
> > > >> 2026 at
> > > >> > > > > > 10:25 AM
> > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > To:
> > dev@polaris.apache.org
> > > >> <
> > > >> > > > > > >> > > > > dev@polaris.apache.org
> > > >> > > > > > >> > > > > > >
> > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > Subject: Re: Polaris
> > > >> Telemetry
> > > >> > > and
> > > >> > > > > > Audit
> > > >> > > > > > >> > > Trail
> > > >> > > > > > >> > > > > > > > > > > > >> >> >>> >
> > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > About the REST API,
> > based
> > > >> on
> > > >> > my
> > > >> > > use
> > > >> > > > > > >> cases:
> > > >> > > > > > >> > > > > > > > > > > > >> >> >>> >
> > > >> > > > > > >> > > > > > > > > > > > >> >> >>> >
> > > >> > > > > > >> > > > > > > > > > > > >> >> >>> >   1.
> > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > I want to be able to
> > query
> > > >> > > commit
> > > >> > > > > > >> metrics
> > > >> > > > > > >> > to
> > > >> > > > > > >> > > > > track
> > > >> > > > > > >> > > > > > > > files
> > > >> > > > > > >> > > > > > > > > > > added
> > > >> > > > > > >> > > > > > > > > > > > /
> > > >> > > > > > >> > > > > > > > > > > > >> >> >>> removed
> > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > per commit, along
> with
> > > >> record
> > > >> > > > > counts.
> > > >> > > > > > >> The
> > > >> > > > > > >> > > > > ingestion
> > > >> > > > > > >> > > > > > > > > > pipeline
> > > >> > > > > > >> > > > > > > > > > > > that
> > > >> > > > > > >> > > > > > > > > > > > >> >> >>> writes
> > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > this data is owned by
> > us
> > > >> and
> > > >> > we
> > > >> > > are
> > > >> > > > > > >> > > guaranteed
> > > >> > > > > > >> > > > to
> > > >> > > > > > >> > > > > > > write
> > > >> > > > > > >> > > > > > > > > > this
> > > >> > > > > > >> > > > > > > > > > > > >> >> >>> information
> > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > for each write.
> > > >> > > > > > >> > > > > > > > > > > > >> >> >>> >   2.
> > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > I want to be able to
> > query
> > > >> > scan
> > > >> > > > > > metrics
> > > >> > > > > > >> for
> > > >> > > > > > >> > > > > read. I
> > > >> > > > > > >> > > > > > > > > > > understand
> > > >> > > > > > >> > > > > > > > > > > > >> >> clients
> > > >> > > > > > >> > > > > > > > > > > > >> >> >>> do
> > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > not fulfill this
> > > >> requirement.
> > > >> > > > > > >> > > > > > > > > > > > >> >> >>> >   3.
> > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > I want to be able to
> > query
> > > >> the
> > > >> > > > > events
> > > >> > > > > > >> table
> > > >> > > > > > >> > > > > (events
> > > >> > > > > > >> > > > > > > are
> > > >> > > > > > >> > > > > > > > > > > > >> persisted) -
> > > >> > > > > > >> > > > > > > > > > > > >> >> >>> this
> > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > may supersede #2, I
> am
> > not
> > > >> > sure
> > > >> > > yet.
> > > >> > > > > > >> > > > > > > > > > > > >> >> >>> >
> > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > All this information
> > is in
> > > >> the
> > > >> > > JDBC
> > > >> > > > > > >> based
> > > >> > > > > > >> > > > > > persistence
> > > >> > > > > > >> > > > > > > > > model
> > > >> > > > > > >> > > > > > > > > > > and
> > > >> > > > > > >> > > > > > > > > > > > >> is
> > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > persisted in the
> > > >> metastore. I
> > > >> > > > > > currently
> > > >> > > > > > >> > don’t
> > > >> > > > > > >> > > > > have
> > > >> > > > > > >> > > > > > a
> > > >> > > > > > >> > > > > > > > need
> > > >> > > > > > >> > > > > > > > > > to
> > > >> > > > > > >> > > > > > > > > > > > >> query
> > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > prometheus or open
> > > >> telemetry.
> > > >> > I
> > > >> > > do
> > > >> > > > > > >> publish
> > > >> > > > > > >> > > some
> > > >> > > > > > >> > > > > > > events
> > > >> > > > > > >> > > > > > > > to
> > > >> > > > > > >> > > > > > > > > > > > >> Prometheus
> > > >> > > > > > >> > > > > > > > > > > > >> >> >>> and
> > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > they are forwarded to
> > our
> > > >> > > dashboards
> > > >> > > > > > >> > > elsewhere.
> > > >> > > > > > >> > > > > > > > > > > > >> >> >>> >
> > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > About the CLI
> > utilities, I
> > > >> > > meant the
> > > >> > > > > > >> admin
> > > >> > > > > > >> > > user
> > > >> > > > > > >> > > > > > > > > utilities.
> > > >> > > > > > >> > > > > > > > > > In
> > > >> > > > > > >> > > > > > > > > > > > >> one of
> > > >> > > > > > >> > > > > > > > > > > > >> >> >>> the
> > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > earliest drafts of my
> > > >> > proposal,
> > > >> > > > > > Prashant
> > > >> > > > > > >> > > > > mentioned
> > > >> > > > > > >> > > > > > > that
> > > >> > > > > > >> > > > > > > > > the
> > > >> > > > > > >> > > > > > > > > > > > >> metrics
> > > >> > > > > > >> > > > > > > > > > > > >> >> >>> tables
> > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > can grow indefinitely
> > and
> > > >> > that a
> > > >> > > > > > similar
> > > >> > > > > > >> > > > problem
> > > >> > > > > > >> > > > > > > exists
> > > >> > > > > > >> > > > > > > > > > with
> > > >> > > > > > >> > > > > > > > > > > > the
> > > >> > > > > > >> > > > > > > > > > > > >> >> events
> > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > table as well. We
> > discussed
> > > >> > that
> > > >> > > > > > >> cleaning
> > > >> > > > > > >> > up
> > > >> > > > > > >> > > of
> > > >> > > > > > >> > > > > old
> > > >> > > > > > >> > > > > > > > > records
> > > >> > > > > > >> > > > > > > > > > > > from
> > > >> > > > > > >> > > > > > > > > > > > >> >> both
> > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > metrics tables and
> > events
> > > >> > > tables can
> > > >> > > > > > be
> > > >> > > > > > >> > done
> > > >> > > > > > >> > > > via
> > > >> > > > > > >> > > > > a
> > > >> > > > > > >> > > > > > > CLI
> > > >> > > > > > >> > > > > > > > > > > utility.
> > > >> > > > > > >> > > > > > > > > > > > >> >> >>> >
> > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > I see that Yufei has
> > > >> covered
> > > >> > the
> > > >> > > > > > >> discussion
> > > >> > > > > > >> > > > about
> > > >> > > > > > >> > > > > > > > > > > datasources.
> > > >> > > > > > >> > > > > > > > > > > > >> >> >>> >
> > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > -
> > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > Anand
> > > >> > > > > > >> > > > > > > > > > > > >> >> >>> >
> > > >> > > > > > >> > > > > > > > > > > > >> >> >>> >
> > > >> > > > > > >> > > > > > > > > > > > >> >> >>> >
> > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > From: Yufei Gu <
> > > >> > > > > flyrain000@gmail.com>
> > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > Date: Friday,
> February
> > 27,
> > > >> > 2026
> > > >> > > at
> > > >> > > > > > >> 9:54 PM
> > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > To:
> > dev@polaris.apache.org
> > > >> <
> > > >> > > > > > >> > > > > dev@polaris.apache.org
> > > >> > > > > > >> > > > > > >
> > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > Subject: Re: Polaris
> > > >> Telemetry
> > > >> > > and
> > > >> > > > > > Audit
> > > >> > > > > > >> > > Trail
> > > >> > > > > > >> > > > > > > > > > > > >> >> >>> >
> > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > This Message Is From
> an
> > > >> > External
> > > >> > > > > > Sender
> > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > This message came
> from
> > > >> outside
> > > >> > > your
> > > >> > > > > > >> > > > organization.
> > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > Report Suspicious<
> > > >> > > > > > >> > > > > > > > > > > > >> >> >>> >
> > > >> > > > > > >> > > > > > > > > > > > >> >> >>>
> > > >> > > > > > >> > > > > > > > > > > > >> >>
> > > >> > > > > > >> > > > > > > > > > > > >>
> > > >> > > > > > >> > > > > > > > > > > >
> > > >> > > > > > >> > > > > > > > > > >
> > > >> > > > > > >> > > > > > > > > >
> > > >> > > > > > >> > > > > > > > >
> > > >> > > > > > >> > > > > > > >
> > > >> > > > > > >> > > > > > >
> > > >> > > > > > >> > > > > >
> > > >> > > > > > >> > > > >
> > > >> > > > > > >> > > >
> > > >> > > > > > >> > >
> > > >> > > > > > >> >
> > > >> > > > > > >>
> > > >> > > > > >
> > > >> > > > >
> > > >> > >
> > > >> >
> > > >>
> >
> https://us-phishalarm-ewt.proofpoint.com/EWT/v1/Iz9xO38YGHZK!YhNDZABkHi1B699ote2uMwpOZw8i0QMCGO2Szc-HshuABGhGvwPJcymE6G2oUUxtS8xDkSrtGTPm_I3QnVDHoLMk50m9v8z_nZKTkd-bnVUbreF1u0WnfV_X5eYevZl_$
> > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > >
> > > >> > > > > > >> > > > > > > > > > > > >> >> >>> >
> > > >> > > > > > >> > > > > > > > > > > > >> >> >>> >
> > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > As I mentioned in
> > > >> > > > > > >> > > > > > > > > > > > >> >> >>> >
> > > >> > > > > > >> > > > > > > > > > > > >> >> >>>
> > > >> > > > > > >> > > > > > > > > > > > >> >>
> > > >> > > > > > >> > > > > > > > > > > > >>
> > > >> > > > > > >> > > > > > > > > > > >
> > > >> > > > > > >> > > > > > > > > > >
> > > >> > > > > > >> > > > > > > > > >
> > > >> > > > > > >> > > > > > > > >
> > > >> > > > > > >> > > > > > > >
> > > >> > > > > > >> > > > > > >
> > > >> > > > > > >> > > > > >
> > > >> > > > > > >> > > > >
> > > >> > > > > > >> > > >
> > > >> > > > > > >> > >
> > > >> > > > > > >> >
> > > >> > > > > > >>
> > > >> > > > > >
> > > >> > > > >
> > > >> > >
> > > >> >
> > > >>
> >
> https://urldefense.com/v3/__https://github.com/apache/polaris/issues/3890__;!!Iz9xO38YGHZK!5EuyFFkk3vhRWVIRvQAWBSQfpJkTMA9HxugzDwXmN0LPPqhEFxYkFRGVhtb8AqUwXtDh2OplcMnbMDHKOxrvDU0$
> > > >> > >> > > >> > > > > > > > > > > > >> >> >>> ,
> > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > supporting
> > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > multiple data sources
> > is
> > > >> not a
> > > >> > > > > trivial
> > > >> > > > > > >> > > change.
> > > >> > > > > > >> > > > I
> > > >> > > > > > >> > > > > > > would
> > > >> > > > > > >> > > > > > > > > > > strongly
> > > >> > > > > > >> > > > > > > > > > > > >> >> >>> recommend
> > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > starting with a
> design
> > > >> > document
> > > >> > > to
> > > >> > > > > > >> > carefully
> > > >> > > > > > >> > > > > > evaluate
> > > >> > > > > > >> > > > > > > > the
> > > >> > > > > > >> > > > > > > > > > > > >> >> architectural
> > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > implications and long
> > term
> > > >> > > impact.
> > > >> > > > > > >> > > > > > > > > > > > >> >> >>> >
> > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > A REST endpoint to
> > query
> > > >> > metrics
> > > >> > > > > seems
> > > >> > > > > > >> > > > reasonable
> > > >> > > > > > >> > > > > > > given
> > > >> > > > > > >> > > > > > > > > the
> > > >> > > > > > >> > > > > > > > > > > > >> current
> > > >> > > > > > >> > > > > > > > > > > > >> >> >>> JDBC
> > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > based persistence
> > model.
> > > >> That
> > > >> > > said,
> > > >> > > > > we
> > > >> > > > > > >> may
> > > >> > > > > > >> > > also
> > > >> > > > > > >> > > > > > > > consider
> > > >> > > > > > >> > > > > > > > > > > > >> alternative
> > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > storage models. For
> > > >> example,
> > > >> > if
> > > >> > > we
> > > >> > > > > > later
> > > >> > > > > > >> > > adopt
> > > >> > > > > > >> > > > a
> > > >> > > > > > >> > > > > > time
> > > >> > > > > > >> > > > > > > > > > series
> > > >> > > > > > >> > > > > > > > > > > > >> system
> > > >> > > > > > >> > > > > > > > > > > > >> >> >>> such as
> > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > Prometheus to store
> > > >> metrics,
> > > >> > the
> > > >> > > > > query
> > > >> > > > > > >> > model
> > > >> > > > > > >> > > > and
> > > >> > > > > > >> > > > > > > access
> > > >> > > > > > >> > > > > > > > > > > > patterns
> > > >> > > > > > >> > > > > > > > > > > > >> >> would
> > > >> > > > > > >> > > > > > > > > > > > >> >> >>> be
> > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > fundamentally
> > different.
> > > >> > > Designing
> > > >> > > > > the
> > > >> > > > > > >> REST
> > > >> > > > > > >> > > API
> > > >> > > > > > >> > > > > > > without
> > > >> > > > > > >> > > > > > > > > > > > >> considering
> > > >> > > > > > >> > > > > > > > > > > > >> >> >>> these
> > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > potential evolutions
> > may
> > > >> limit
> > > >> > > > > > >> flexibility.
> > > >> > > > > > >> > > I'd
> > > >> > > > > > >> > > > > > > suggest
> > > >> > > > > > >> > > > > > > > > to
> > > >> > > > > > >> > > > > > > > > > > > start
> > > >> > > > > > >> > > > > > > > > > > > >> >> with
> > > >> > > > > > >> > > > > > > > > > > > >> >> >>> the
> > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > use case.
> > > >> > > > > > >> > > > > > > > > > > > >> >> >>> >
> > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > Yufei
> > > >> > > > > > >> > > > > > > > > > > > >> >> >>> >
> > > >> > > > > > >> > > > > > > > > > > > >> >> >>> >
> > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > On Fri, Feb 27, 2026
> at
> > > >> > 3:42 PM
> > > >> > > > > Dmitri
> > > >> > > > > > >> > > > > > Bourlatchkov <
> > > >> > > > > > >> > > > > > > > > > > > >> >> dimas@apache.org>
> > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > wrote:
> > > >> > > > > > >> > > > > > > > > > > > >> >> >>> >
> > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > > Hi Anand,
> > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > >
> > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > > Sharing my view...
> > > >> subject
> > > >> > to
> > > >> > > > > > >> discussion:
> > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > >
> > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > > 1. Adding non-IRC
> > REST
> > > >> API
> > > >> > to
> > > >> > > > > > Polaris
> > > >> > > > > > >> is
> > > >> > > > > > >> > > > > > perfectly
> > > >> > > > > > >> > > > > > > > > fine.
> > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > >
> > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > > Figuring out
> specific
> > > >> > endpoint
> > > >> > > > > URIs
> > > >> > > > > > >> and
> > > >> > > > > > >> > > > > payloads
> > > >> > > > > > >> > > > > > > > might
> > > >> > > > > > >> > > > > > > > > > > > require
> > > >> > > > > > >> > > > > > > > > > > > >> a
> > > >> > > > > > >> > > > > > > > > > > > >> >> few
> > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > > roundtrips, so
> > opening a
> > > >> > > separate
> > > >> > > > > > >> thread
> > > >> > > > > > >> > > for
> > > >> > > > > > >> > > > > that
> > > >> > > > > > >> > > > > > > > might
> > > >> > > > > > >> > > > > > > > > > be
> > > >> > > > > > >> > > > > > > > > > > > >> best.
> > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > > Contributors
> commonly
> > > >> create
> > > >> > > > > Google
> > > >> > > > > > >> Docs
> > > >> > > > > > >> > > for
> > > >> > > > > > >> > > > > new
> > > >> > > > > > >> > > > > > > API
> > > >> > > > > > >> > > > > > > > > > > > proposals
> > > >> > > > > > >> > > > > > > > > > > > >> too
> > > >> > > > > > >> > > > > > > > > > > > >> >> >>> (they
> > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > > fairly easy to
> > update as
> > > >> the
> > > >> > > email
> > > >> > > > > > >> > > discussion
> > > >> > > > > > >> > > > > > > > > > progresses).
> > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > >
> > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > > There was a
> > suggestion to
> > > >> > try
> > > >> > > > > > Markdown
> > > >> > > > > > >> > > (with
> > > >> > > > > > >> > > > > PRs)
> > > >> > > > > > >> > > > > > > for
> > > >> > > > > > >> > > > > > > > > > > > proposals
> > > >> > > > > > >> > > > > > > > > > > > >> >> [1]
> > > >> > > > > > >> > > > > > > > > > > > >> >> >>> ...
> > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > > feel free to give
> it
> > a
> > > >> try
> > > >> > if
> > > >> > > you
> > > >> > > > > > are
> > > >> > > > > > >> > > > > comfortable
> > > >> > > > > > >> > > > > > > > with
> > > >> > > > > > >> > > > > > > > > > > that.
> > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > >
> > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > > 2. Could you
> clarify
> > > >> whether
> > > >> > > you
> > > >> > > > > > mean
> > > >> > > > > > >> end
> > > >> > > > > > >> > > > user
> > > >> > > > > > >> > > > > > > > > utilities
> > > >> > > > > > >> > > > > > > > > > or
> > > >> > > > > > >> > > > > > > > > > > > >> admin
> > > >> > > > > > >> > > > > > > > > > > > >> >> >>> user
> > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > > utilities? In the
> > latter
> > > >> > case
> > > >> > > > > those
> > > >> > > > > > >> might
> > > >> > > > > > >> > > be
> > > >> > > > > > >> > > > > more
> > > >> > > > > > >> > > > > > > > > > suitable
> > > >> > > > > > >> > > > > > > > > > > > for
> > > >> > > > > > >> > > > > > > > > > > > >> the
> > > >> > > > > > >> > > > > > > > > > > > >> >> >>> Admin
> > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > > CLI (java) not the
> > Python
> > > >> > CLI,
> > > >> > > > > IMHO.
> > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > >
> > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > > Why would these
> > > >> utilities be
> > > >> > > > > common
> > > >> > > > > > >> with
> > > >> > > > > > >> > > > > events?
> > > >> > > > > > >> > > > > > > > IMHO,
> > > >> > > > > > >> > > > > > > > > > > event
> > > >> > > > > > >> > > > > > > > > > > > >> use
> > > >> > > > > > >> > > > > > > > > > > > >> >> >>> cases
> > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > are
> > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > > distinct from
> > scan/commit
> > > >> > > metrics.
> > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > >
> > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > > 3. I'd prefer
> > separating
> > > >> > > metrics
> > > >> > > > > > >> > > persistence
> > > >> > > > > > >> > > > > from
> > > >> > > > > > >> > > > > > > > > > MetaStore
> > > >> > > > > > >> > > > > > > > > > > > >> >> >>> persistence
> > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > at
> > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > > the code level, so
> > that
> > > >> they
> > > >> > > could
> > > >> > > > > > be
> > > >> > > > > > >> > mixed
> > > >> > > > > > >> > > > and
> > > >> > > > > > >> > > > > > > > matched
> > > >> > > > > > >> > > > > > > > > > > > >> >> >>> independently.
> > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > The
> > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > > separate datasource
> > > >> question
> > > >> > > will
> > > >> > > > > > >> become
> > > >> > > > > > >> > a
> > > >> > > > > > >> > > > > > > non-issue
> > > >> > > > > > >> > > > > > > > > with
> > > >> > > > > > >> > > > > > > > > > > > that
> > > >> > > > > > >> > > > > > > > > > > > >> >> >>> approach,
> > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > I
> > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > > guess.
> > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > >
> > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > > The rationale for
> > > >> separating
> > > >> > > scan
> > > >> > > > > > >> metrics
> > > >> > > > > > >> > > and
> > > >> > > > > > >> > > > > > > > metastore
> > > >> > > > > > >> > > > > > > > > > > > >> >> persistence
> > > >> > > > > > >> > > > > > > > > > > > >> >> >>> is
> > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > that
> > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > > "cascading deletes"
> > > >> between
> > > >> > > them
> > > >> > > > > are
> > > >> > > > > > >> > hardly
> > > >> > > > > > >> > > > > ever
> > > >> > > > > > >> > > > > > > > > > required.
> > > >> > > > > > >> > > > > > > > > > > > >> >> >>> Furthermore,
> > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > the
> > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > > data and query
> > patterns
> > > >> are
> > > >> > > very
> > > >> > > > > > >> > different
> > > >> > > > > > >> > > so
> > > >> > > > > > >> > > > > > > > different
> > > >> > > > > > >> > > > > > > > > > > > >> >> technologies
> > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > might
> > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > > be beneficial in
> each
> > > >> case.
> > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > >
> > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > > [1]
> > > >> > > > > > >> > > > > > > > > > > > >> >> >>> >
> > > >> > > > > > >> > > > > > > > > > > > >> >> >>>
> > > >> > > > > > >> > > > > > > > > > > > >> >>
> > > >> > > > > > >> > > > > > > > > > > > >>
> > > >> > > > > > >> > > > > > > > > > > >
> > > >> > > > > > >> > > > > > > > > > >
> > > >> > > > > > >> > > > > > > > > >
> > > >> > > > > > >> > > > > > > > >
> > > >> > > > > > >> > > > > > > >
> > > >> > > > > > >> > > > > > >
> > > >> > > > > > >> > > > > >
> > > >> > > > > > >> > > > >
> > > >> > > > > > >> > > >
> > > >> > > > > > >> > >
> > > >> > > > > > >> >
> > > >> > > > > > >>
> > > >> > > > > >
> > > >> > > > >
> > > >> > >
> > > >> >
> > > >>
> >
> https://urldefense.com/v3/__https://lists.apache.org/thread/yto2wp982t43h1mqjwnslswhws5z47cy__;!!Iz9xO38YGHZK!5EuyFFkk3vhRWVIRvQAWBSQfpJkTMA9HxugzDwXmN0LPPqhEFxYkFRGVhtb8AqUwXtDh2OplcMnbMDHKxYDakNU$
> > > >> > >> > > >> > > > > > > > > > > > >> >> >>> > >
> > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > > Cheers,
> > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > > Dmitri.
> > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > >
> > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > > On Fri, Feb 27,
> 2026
> > at
> > > >> > > 6:19 PM
> > > >> > > > > > Anand
> > > >> > > > > > >> > Kumar
> > > >> > > > > > >> > > > > > > Sankaran
> > > >> > > > > > >> > > > > > > > > via
> > > >> > > > > > >> > > > > > > > > > > dev
> > > >> > > > > > >> > > > > > > > > > > > <
> > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > >
> > dev@polaris.apache.org>
> > > >> > > wrote:
> > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > >
> > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > > > Thanks all. This
> > PR is
> > > >> > > merged
> > > >> > > > > now.
> > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > > >
> > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > > > Here are the
> > follow-up
> > > >> > > features
> > > >> > > > > /
> > > >> > > > > > >> work
> > > >> > > > > > >> > > > > needed.
> > > >> > > > > > >> > > > > > > > These
> > > >> > > > > > >> > > > > > > > > > > were
> > > >> > > > > > >> > > > > > > > > > > > >> all
> > > >> > > > > > >> > > > > > > > > > > > >> >> >>> part of
> > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > > the
> > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > > > merged PR at some
> > > >> point in
> > > >> > > time
> > > >> > > > > > and
> > > >> > > > > > >> > were
> > > >> > > > > > >> > > > > > removed
> > > >> > > > > > >> > > > > > > to
> > > >> > > > > > >> > > > > > > > > > > reduce
> > > >> > > > > > >> > > > > > > > > > > > >> >> scope.
> > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > > >
> > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > > > Please let me
> know
> > what
> > > >> > you
> > > >> > > > > think.
> > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > > >
> > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > > >
> > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > > >   1.  A REST API
> to
> > > >> > paginate
> > > >> > > > > > through
> > > >> > > > > > >> > > table
> > > >> > > > > > >> > > > > > > metrics.
> > > >> > > > > > >> > > > > > > > > > This
> > > >> > > > > > >> > > > > > > > > > > > >> will be
> > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > non-IRC
> > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > > > standard
> addition.
> > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > > >   2.  Utilities
> for
> > > >> > > managing old
> > > >> > > > > > >> > records,
> > > >> > > > > > >> > > > > > should
> > > >> > > > > > >> > > > > > > be
> > > >> > > > > > >> > > > > > > > > > > common
> > > >> > > > > > >> > > > > > > > > > > > >> with
> > > >> > > > > > >> > > > > > > > > > > > >> >> >>> events.
> > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > > > There was some
> > > >> discussion
> > > >> > > that
> > > >> > > > > it
> > > >> > > > > > >> > belongs
> > > >> > > > > > >> > > > to
> > > >> > > > > > >> > > > > > the
> > > >> > > > > > >> > > > > > > > CLI.
> > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > > >   3.  Separate
> > > >> datasource
> > > >> > > > > > (metrics,
> > > >> > > > > > >> > > events,
> > > >> > > > > > >> > > > > > even
> > > >> > > > > > >> > > > > > > > > other
> > > >> > > > > > >> > > > > > > > > > > > >> tables?).
> > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > > >
> > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > > >
> > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > > > Anything else?
> > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > > >
> > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > > > -
> > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > > > Anand
> > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > > >
> > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > > >
> > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > >
> > > >> > > > > > >> > > > > > > > > > > > >> >> >>> >
> > > >> > > > > > >> > > > > > > > > > > > >> >> >>> >
> > > >> > > > > > >> > > > > > > > > > > > >> >> >>>
> > > >> > > > > > >> > > > > > > > > > > > >> >> >>
> > > >> > > > > > >> > > > > > > > > > > > >> >>
> > > >> > > > > > >> > > > > > > > > > > > >> >
> > > >> > > > > > >> > > > > > > > > > > > >>
> > > >> > > > > > >> > > > > > > > > > > > >
> > > >> > > > > > >> > > > > > > > > > > >
> > > >> > > > > > >> > > > > > > > > > >
> > > >> > > > > > >> > > > > > > > > >
> > > >> > > > > > >> > > > > > > > >
> > > >> > > > > > >> > > > > > > >
> > > >> > > > > > >> > > > > > >
> > > >> > > > > > >> > > > > >
> > > >> > > > > > >> > > > >
> > > >> > > > > > >> > > >
> > > >> > > > > > >> > >
> > > >> > > > > > >> >
> > > >> > > > > > >>
> > > >> > > > > > >
> > > >> > > > > >
> > > >> > > > >
> > > >> > >
> > > >> >
> > > >> >
> > > >>
> > > >
> >
>

Re: Proposal for REST endpoints for table metrics and events

[Yufei Gu <fl...@gmail.com>]

From the previous email, we agreed to split the REST change from the grant
change. I think the PR is ready to go if we only keep the grant changes.

Yufei


On Tue, Sep 29, 2026 at 9:26 PM Jean-Baptiste Onofré <jb...@nanthrax.net>
wrote:

> Hi guys,
>
> I agree with Dmitri: +1 to merge PR #4115 on Oct 1 (tomorrow).
>
> From my side, all outstanding feedback has been addressed, including
> the final documentation points raised by EJ regarding the empty-200
> NoOp behavior and reconstructed payload description. The branch is
> clean, up to date with main, and CI is green.
>
> Once this lands, we can follow up with PR #4756 for the JDBC query SPI
> implementation.
>
> Regards
> JB
>
> On Wed, Sep 30, 2026 at 1:53 AM Dmitri Bourlatchkov <di...@apache.org>
> wrote:
> >
> > Hi All,
> >
> > Any remaining blockers on [4115]?
> >
> > Given that the review was very long, please restate any remaining
> concerns
> > (here or in GH).
> >
> > Otherwise, I propose merging [4115] on Oct 1.
> >
> > [4115] https://github.com/apache/polaris/pull/4115
> >
> > Thanks,
> > Dmitri.
> >
> > On Wed, Sep 23, 2026 at 7:53 PM Dmitri Bourlatchkov <di...@apache.org>
> > wrote:
> >
> > > Hi EJ,
> > >
> > > Thanks for the meeting recap!
> > >
> > > All:
> > >
> > > Worth noting: PR [4115] also contains Management API changes that were
> not
> > > anticipated initially and did not get an API change review until now.
> These
> > > are small changes to add new grant types for metrics RBAC..
> > >
> > > I've reviewed and approved PR [4115] in GH.
> > >
> > > [4115] https://github.com/apache/polaris/pull/4115
> > >
> > > Cheers,
> > > Dmitri.
> > >
> > > On Wed, Aug 12, 2026 at 5:21 PM EJ Wang <
> ej.wang.dev.interest@gmail.com>
> > > wrote:
> > >
> > >> Hi folks,
> > >>
> > >> Notes from today's metrics architecture sync are in the usual doc:
> > >>
> > >>
> https://docs.google.com/document/d/100h7c4damrUzVuquYbBHM0EvA4LSWuW2IT2dN_7nYVA/edit?pli=1&tab=t.18iv1m7p6i6b
> > >>
> > >> Highlights:
> > >>
> > >> Module placement. core/ now holds the entity data model only. A new
> spi/
> > >> takes every other shared contract: feature SPIs, feature non-SPI
> > >> contracts,
> > >> substrate SPIs, substrate non-SPI contracts, and contract-required
> types.
> > >> That settles the question PR #5204 reopened, by placing into a new SPI
> > >> module. Note that it changes slightly towards the referenced WIP RFC
> [1]
> > >> recommendations: what proposed to place in core/, now should go to
> spi/
> > >> except for the entity data model.
> > >>
> > >> Metrics PRs. #5068 merged. #5204 needs reshaping onto the split.
> #4115 and
> > >> #4756 stay open with action on the author to follow up on review
> comments.
> > >>
> > >> [1]
> > >>
> > >>
> https://docs.google.com/document/d/1mj3vGpix9zTDTFFJKtWT5ZxI3PTGI1YNb5xFqyWrOkc/edit?tab=t.0
> > >>
> > >> On Wed, Jul 15, 2026 at 11:01 AM Anand Kumar Sankaran via dev <
> > >> dev@polaris.apache.org> wrote:
> > >>
> > >> > Thanks EJ.  My plan is to introduce PR0 in a new branch / PR and
> > >> repurpose
> > >> > #4115 and #4756 for PR1 and PR2 to maintain all the valuable
> discussions
> > >> > that happened there.
> > >> >
> > >> > From: EJ Wang <ej...@gmail.com>
> > >> > Date: Wednesday, July 15, 2026 at 10:41 AM
> > >> > To: dev@polaris.apache.org <de...@polaris.apache.org>
> > >> > Subject: Re: Proposal for REST endpoints for table metrics and
> events
> > >> >
> > >> > Hi folks, Thanks again for the discussion today. I updated the sync
> doc
> > >> > with notes from today's metrics sync: https: //urldefense.
> > >> com/v3/__https:
> > >> > //docs. google.
> > >> >
> > >>
> com/document/d/100h7c4damrUzVuquYbBHM0EvA4LSWuW2IT2dN_7nYVA/edit?pli=1&tab=t.
> > >> >
> > >>
> 2ahd7wze5f__;!!Iz9xO38YGHZK!4rnlVIqBQQQ6xpGe1PMV58edOS_cGrKoqdfhw4qSp_JQCavvEjaxXk0OwGPnxyEcO60AzKQK_XDtp7H3DdkFXcrXJwAz1g0$
> > >> >
> > >> >
> > >> > Hi folks,
> > >> >
> > >> > Thanks again for the discussion today. I updated the sync doc with
> notes
> > >> > from today's metrics sync:
> > >> >
> > >> >
> > >>
> https://urldefense.com/v3/__https://docs.google.com/document/d/100h7c4damrUzVuquYbBHM0EvA4LSWuW2IT2dN_7nYVA/edit?pli=1&tab=t.2ahd7wze5f__;!!Iz9xO38YGHZK!4rnlVIqBQQQ6xpGe1PMV58edOS_cGrKoqdfhw4qSp_JQCavvEjaxXk0OwGPnxyEcO60AzKQK_XDtp7H3DdkFXcrXJwAz1g0$
> > >> >
> > >> > See highlights below:
> > >> >
> > >> > In previous meetings, EJ proposed a two prs framing to push forward
> the
> > >> > metrics query work in polaris. Dmitri pointed out a subtle point
> that
> > >> pr2
> > >> > is currently also trying to do the persistence related code
> placement
> > >> > refactor. For the sake of straightforward review process, EJ,
> Dmitri and
> > >> > Anand (the author) agreed to separate the persistence refactoring
> into
> > >> pr0,
> > >> > and reframing the pr scopes as follow:
> > >> >
> > >> >    1.
> > >> >
> > >> >    Pr0: refactor irc metrics spi
> > >> >    1.
> > >> >
> > >> >       SPI should stay in core/ with metric type enveloped
> > >> >       2.
> > >> >
> > >> >       default spi impl, no-op, in extensions/ - default selection at
> > >> >       runtime/
> > >> >       3.
> > >> >
> > >> >       move metrics persistence and correspondings to extensions/ as
> an
> > >> >       alternative SPI impl
> > >> >       2.
> > >> >
> > >> >    Pr1: rest api for query
> > >> >
> > >> >
> > >>
> https://urldefense.com/v3/__https://github.com/apache/polaris/pull/4115/changes__;!!Iz9xO38YGHZK!4rnlVIqBQQQ6xpGe1PMV58edOS_cGrKoqdfhw4qSp_JQCavvEjaxXk0OwGPnxyEcO60AzKQK_XDtp7H3DdkFXcrXFCbEbRk$
> > >> >    1.
> > >> >
> > >> >       rest api spec in extensions/ as optional spec
> > >> >       2.
> > >> >
> > >> >       SPI in extensions/ as optional spec spi
> > >> >       3.
> > >> >
> > >> >       default spi impl in extensions/ no-op
> > >> >       4.
> > >> >
> > >> >       handler related in runtime/ - thin adaptor from http shape to
> spi
> > >> >       shape
> > >> >       3.
> > >> >
> > >> >    Pr2: jdbc impl for query
> > >> >
> > >>
> https://urldefense.com/v3/__https://github.com/apache/polaris/pull/4756__;!!Iz9xO38YGHZK!4rnlVIqBQQQ6xpGe1PMV58edOS_cGrKoqdfhw4qSp_JQCavvEjaxXk0OwGPnxyEcO60AzKQK_XDtp7H3DdkFXcrXRVMBzqE$
> > >> >    1.
> > >> >
> > >> >       alternative spi impl in extensions/ -> query with jdbc
> > >> >
> > >> >
> > >> > Thanks,
> > >> > -ej
> > >> >
> > >> > On Wed, Jul 1, 2026 at 8:01 AM Alexandre Dutra <ad...@apache.org>
> > >> wrote:
> > >> >
> > >> > > Hi all,
> > >> > >
> > >> > > > While the current implementation stores metrics in the
> metastore,
> > >> other
> > >> > > deployments
> > >> > > may choose a KV store, a dedicated metrics warehouse, or another
> > >> backend
> > >> > > entirely. In those cases, exposing a Polaris REST API over a
> specific
> > >> > > schema feels like coupling the API to an implementation detail.
> > >> > >
> > >> > > I wouldn't say to an implementation detail, but rather to a
> particular
> > >> > > setup / configuration. I think that's fine. If you don't store
> > >> > > metrics, you don't read metrics. Similar situations will certainly
> > >> > > happen in the future, e.g. if we ever introduce an API to read
> events,
> > >> > > that would only work if events are being persisted in the first
> place.
> > >> > > Again I think that's a fair trade-off.
> > >> > >
> > >> > > Thanks,
> > >> > > Alex
> > >> > >
> > >> > > On Wed, Jul 1, 2026 at 7:23 AM Dmitri Bourlatchkov <
> dimas@apache.org>
> > >> > > wrote:
> > >> > > >
> > >> > > > Hi Yufei,
> > >> > > >
> > >> > > > From my POV, it is fine for Polaris to have multiple REST API
> > >> surfaces
> > >> > > with
> > >> > > > different features and different degrees of "readiness". Each
> API
> > >> can
> > >> > > > evolve fairly independently. With proper modularization (which
> Anand
> > >> > > did),
> > >> > > > changes to the main Polaris code and these extra modules should
> > >> > normally
> > >> > > be
> > >> > > > non-intrusive to each other.
> > >> > > >
> > >> > > > I assume Anand is going to be the first user of this API.
> However,
> > >> if
> > >> > we
> > >> > > do
> > >> > > > not merge it we'll never know whether other users are
> interested. I
> > >> do
> > >> > > not
> > >> > > > think many end users read the dev ML, but they certainly read
> the
> > >> docs,
> > >> > > > code and release notes.
> > >> > > >
> > >> > > > There is no risk for Polaris as a project in exposing this
> > >> experimental
> > >> > > > API. If usage dwindles over time, we can remove it. If it gains
> a
> > >> > > > considerable user / contributor base, we will promote it to a
> > >> > > "first-class"
> > >> > > > (non-beta) API.
> > >> > > >
> > >> > > > Cheers,
> > >> > > > Dmitri.
> > >> > > >
> > >> > > > On Tue, Jun 30, 2026 at 7:53 PM Yufei Gu <fl...@gmail.com>
> > >> wrote:
> > >> > > >
> > >> > > > > Hi all,
> > >> > > > >
> > >> > > > > Thanks for driving this work forward. Building this as a POC
> makes
> > >> > > perfect
> > >> > > > > sense if the goal is to validate the idea and gather feedback.
> > >> Should
> > >> > > we
> > >> > > > > merge a POC into the main branch before agreeing on the
> overall
> > >> > > direction?
> > >> > > > > To me, that feels like a separate discussion from whether the
> > >> > > > > implementation itself looks reasonable.
> > >> > > > >
> > >> > > > > One concern that I don't think has been fully addressed is
> > >> whether we
> > >> > > need
> > >> > > > > to expose metrics consumption through a Polaris REST endpoint
> at
> > >> all.
> > >> > > > >
> > >> > > > > My concern is less about the API itself and more about the
> > >> boundary
> > >> > > Polaris
> > >> > > > > should own. The proposed REST API is tied to a particular
> > >> persistence
> > >> > > > > schema, but that schema is just one possible implementation.
> While
> > >> > the
> > >> > > > > current implementation stores metrics in the metastore, other
> > >> > > deployments
> > >> > > > > may choose a KV store, a dedicated metrics warehouse, or
> another
> > >> > > backend
> > >> > > > > entirely. In those cases, exposing a Polaris REST API over a
> > >> specific
> > >> > > > > schema feels like coupling the API to an implementation
> detail.
> > >> > > > >
> > >> > > > > Polaris isn't intended to become a full metrics system. I'd
> > >> expect it
> > >> > > to
> > >> > > > > focus on producing metrics and let deployments choose the most
> > >> > > appropriate
> > >> > > > > storage and consumption model for their environment.
> > >> > > > >
> > >> > > > > I'm not opposed to exploring this direction, but we should
> first
> > >> > align
> > >> > > on
> > >> > > > > whether exposing a REST API through Polaris is the right long
> term
> > >> > > approach
> > >> > > > > before merging it into the main branch.
> > >> > > > >
> > >> > > > > Thanks,
> > >> > > > > Yufei
> > >> > > > >
> > >> > > > > Yufei
> > >> > > > >
> > >> > > > >
> > >> > > > > On Mon, Jun 29, 2026 at 9:27 AM Dmitri Bourlatchkov <
> > >> > dimas@apache.org>
> > >> > > > > wrote:
> > >> > > > >
> > >> > > > > > Hi All,
> > >> > > > > >
> > >> > > > > > It looks like Anand addressed all action items in [4115].
> > >> > > > > >
> > >> > > > > > Please review the latest state of the code. The PR LGTM and
> I
> > >> > > approved in
> > >> > > > > > GH from my side.
> > >> > > > > >
> > >> > > > > > [4115]
> > >> >
> > >>
> https://urldefense.com/v3/__https://github.com/apache/polaris/pull/4115__;!!Iz9xO38YGHZK!4rnlVIqBQQQ6xpGe1PMV58edOS_cGrKoqdfhw4qSp_JQCavvEjaxXk0OwGPnxyEcO60AzKQK_XDtp7H3DdkFXcrXtFRscKs$
> > >> > > > > >
> > >> > > > > > Thanks,
> > >> > > > > > Dmitri.
> > >> > > > > >
> > >> > > > > > On Thu, Jun 25, 2026 at 1:56 PM Dmitri Bourlatchkov <
> > >> > > dimas@apache.org>
> > >> > > > > > wrote:
> > >> > > > > >
> > >> > > > > > > Hi All,
> > >> > > > > > >
> > >> > > > > > > Recap of the discussion in today's Community Sync:
> > >> > > > > > >
> > >> > > > > > > * The modular approach for Metrics SPI(s) and
> implementation
> > >> > seems
> > >> > > to
> > >> > > > > > have
> > >> > > > > > > community consensus (please comment if you still have
> > >> concerns)
> > >> > > > > > >
> > >> > > > > > > * What modules / feature flags are enabled in Apache
> Polaris
> > >> > > images by
> > >> > > > > > > default will be discussed later (when the feature is
> close to
> > >> > > > > > completion).
> > >> > > > > > >
> > >> > > > > > > In any case, users will be able to turn the full feature
> > >> on/off
> > >> > at
> > >> > > > > their
> > >> > > > > > > discretion, regardless of the default flag values.
> > >> > > > > > >
> > >> > > > > > > * It is generally preferable to have smaller PRs and
> introduce
> > >> > new
> > >> > > > > > > features gradually to simplify reviews.
> > >> > > > > > >
> > >> > > > > > > What this means for the Metrics API, IMHO is:
> > >> > > > > > >
> > >> > > > > > > a) The scope of PR 4115 is probably ok as it stands right
> > >> now. If
> > >> > > > > someone
> > >> > > > > > > wants to propose splitting it further, please comment.
> > >> > > > > > >
> > >> > > > > > > b) After 4115 is merged, Metrics Persistence PRs will
> follow,
> > >> > with
> > >> > > > > > smaller
> > >> > > > > > > scope per PR as appropriate for the nature of the code
> > >> changes.
> > >> > > > > > >
> > >> > > > > > > * The major concern was about exposing the new Metrics
> REST
> > >> API
> > >> > to
> > >> > > end
> > >> > > > > > > users. The nature of this concern was about avoiding the
> > >> (false)
> > >> > > > > > impression
> > >> > > > > > > that the new API is a firm and adopted Polaris
> specification.
> > >> > > > > > >
> > >> > > > > > > I think this concern is reasonable.
> > >> > > > > > >
> > >> > > > > > > I propose that Anand update the documentation sections of
> PR
> > >> 4115
> > >> > > to
> > >> > > > > > > emphasise that the API is currently an experimental / POC
> API
> > >> at
> > >> > > this
> > >> > > > > > time
> > >> > > > > > > and is subject to change (including breaking changes). I
> > >> believe
> > >> > it
> > >> > > > > > already
> > >> > > > > > > has the "beta" label. So this request is mostly for doc
> > >> changes
> > >> > to
> > >> > > > > > clarify
> > >> > > > > > > user expectations, since the "beta" label alone may not be
> > >> > > interpreted
> > >> > > > > > the
> > >> > > > > > > same way by all users.
> > >> > > > > > >
> > >> > > > > > > I hope this update will resolve all remaining concerns
> with
> > >> 4115
> > >> > > and
> > >> > > > > > allow
> > >> > > > > > > for it to be merged so that the community will be able to
> > >> proceed
> > >> > > to
> > >> > > > > > > follow-up PRs (as noted above).
> > >> > > > > > >
> > >> > > > > > > Please add/correct me if I missed anything relevant.
> > >> > > > > > >
> > >> > > > > > > Thanks,
> > >> > > > > > > Dmitri.
> > >> > > > > > >
> > >> > > > > > > On Wed, Jun 24, 2026 at 8:11 PM Yufei Gu <
> > >> flyrain000@gmail.com>
> > >> > > wrote:
> > >> > > > > > >
> > >> > > > > > >> I do not think we have agreement on the query REST APIs
> yet.
> > >> > > > > > >>
> > >> > > > > > >> In particular, the current query API shape seems highly
> > >> coupled
> > >> > > to the
> > >> > > > > > >> *example[1]* metrics in the Iceberg REST spec. I do not
> think
> > >> > > that is
> > >> > > > > a
> > >> > > > > > >> good way to design a REST API contract. For example,
> fields
> > >> like
> > >> > > > > > >> resultDataFiles, resultDeleteFiles, totalFileSizeBytes,
> > >> > > > > > >> totalDataManifests,
> > >> > > > > > >> totalDeleteManifests, and scannedDataManifests are very
> > >> specific
> > >> > > to
> > >> > > > > one
> > >> > > > > > >> style of scan metric. That makes the API look more like a
> > >> direct
> > >> > > > > > >> projection
> > >> > > > > > >> of the current example payload than a general query
> model.
> > >> > > > > > >>
> > >> > > > > > >> My preference is that Polaris should not define or
> strongly
> > >> > opine
> > >> > > on
> > >> > > > > the
> > >> > > > > > >> Iceberg scan or commit metrics query REST API at this
> point.
> > >> > > Polaris
> > >> > > > > can
> > >> > > > > > >> resolve and authorize the table, accept the metrics
> report,
> > >> and
> > >> > > > > delegate
> > >> > > > > > >> it
> > >> > > > > > >> to the configured reporting path. Durable storage,
> filtering,
> > >> > > > > > dashboards,
> > >> > > > > > >> and query APIs should remain downstream impl. choices
> unless
> > >> and
> > >> > > until
> > >> > > > > > the
> > >> > > > > > >> Iceberg community standardizes a query contract, which I
> > >> doubt
> > >> > > will
> > >> > > > > > >> happen.
> > >> > > > > > >>
> > >> > > > > > >> So I do not think the intake path has to be blocked, but
> I
> > >> would
> > >> > > > > prefer
> > >> > > > > > >> not
> > >> > > > > > >> to merge a Polaris specific query API that may be hard to
> > >> evolve
> > >> > > > > later.
> > >> > > > > > >>
> > >> > > > > > >> 1.
> > >> > > > > > >>
> > >> > > > > > >>
> > >> > > > > >
> > >> > > > >
> > >> > >
> > >> >
> > >>
> https://urldefense.com/v3/__https://github.com/apache/iceberg/blob/2a6c556c0c883c04c6d3fbd68e6aeda36b91e0aa/open-api/rest-catalog-open-api.yaml*L4152__;Iw!!Iz9xO38YGHZK!4rnlVIqBQQQ6xpGe1PMV58edOS_cGrKoqdfhw4qSp_JQCavvEjaxXk0OwGPnxyEcO60AzKQK_XDtp7H3DdkFXcrXCDsqYGo$
> > >> > > > > > >>
> > >> > > > > > >> Yufei
> > >> > > > > > >>
> > >> > > > > > >>
> > >> > > > > > >> On Tue, Jun 23, 2026 at 9:30 AM Dmitri Bourlatchkov <
> > >> > > dimas@apache.org
> > >> > > > > >
> > >> > > > > > >> wrote:
> > >> > > > > > >>
> > >> > > > > > >> > Hi All,
> > >> > > > > > >> >
> > >> > > > > > >> > So, what are the blockers for merging [4115], if any?
> > >> > > > > > >> >
> > >> > > > > > >> > [4115]
> > >> >
> > >>
> https://urldefense.com/v3/__https://github.com/apache/polaris/pull/4115__;!!Iz9xO38YGHZK!4rnlVIqBQQQ6xpGe1PMV58edOS_cGrKoqdfhw4qSp_JQCavvEjaxXk0OwGPnxyEcO60AzKQK_XDtp7H3DdkFXcrXtFRscKs$
> > >> > > > > > >> >
> > >> > > > > > >> > Thanks,
> > >> > > > > > >> > Dmitri.
> > >> > > > > > >> >
> > >> > > > > > >> > On Wed, Jun 17, 2026 at 5:28 PM EJ Wang <
> > >> > > > > > ej.wang.dev.interest@gmail.com
> > >> > > > > > >> >
> > >> > > > > > >> > wrote:
> > >> > > > > > >> >
> > >> > > > > > >> > > Hi folks,
> > >> > > > > > >> > >
> > >> > > > > > >> > > Thanks again for the discussion today. I updated the
> sync
> > >> > doc
> > >> > > with
> > >> > > > > > >> notes
> > >> > > > > > >> > > from today's metrics sync:
> > >> > > > > > >> > >
> > >> > > > > > >> > >
> > >> > > > > > >> > >
> > >> > > > > > >> >
> > >> > > > > > >>
> > >> > > > > >
> > >> > > > >
> > >> > >
> > >> >
> > >>
> https://urldefense.com/v3/__https://docs.google.com/document/d/100h7c4damrUzVuquYbBHM0EvA4LSWuW2IT2dN_7nYVA/edit?tab=t.ezk0rgdx0c6m__;!!Iz9xO38YGHZK!4rnlVIqBQQQ6xpGe1PMV58edOS_cGrKoqdfhw4qSp_JQCavvEjaxXk0OwGPnxyEcO60AzKQK_XDtp7H3DdkFXcrX83IZkm0$
> > >> > > > > > >> > >
> > >> > > > > > >> > > A few highlights:
> > >> > > > > > >> > >
> > >> > > > > > >> > >    - No objection to keeping the default metrics
> behavior
> > >> > > small:
> > >> > > > > > >> > >    no-op/log-only is enough as the built-in default.
> > >> > > > > > >> > >    - Durable storage, event forwarding, external
> queues,
> > >> > > > > dashboards,
> > >> > > > > > >> and
> > >> > > > > > >> > >    custom filtering should be implementation choices
> > >> behind
> > >> > > the
> > >> > > > > > >> metrics
> > >> > > > > > >> > >    reporting path.
> > >> > > > > > >> > >    - For PR #4115, the REST layer should
> > >> > resolve/authz/accept
> > >> > > the
> > >> > > > > > >> report,
> > >> > > > > > >> > >    then delegate to the selected metrics
> implementation.
> > >> It
> > >> > > should
> > >> > > > > > not
> > >> > > > > > >> > own
> > >> > > > > > >> > >    durable storage semantics.
> > >> > > > > > >> > >    - The rough boundary we discussed is: spec owns
> the
> > >> wire
> > >> > > > > > contract,
> > >> > > > > > >> > >    runtime owns framework wiring, API/contract
> modules
> > >> own
> > >> > > > > > >> > provider-facing
> > >> > > > > > >> > >    contracts, and extensions own replaceable
> > >> > implementations.
> > >> > > > > > >> > >    - OpenLineage has a similar path-naming question.
> We
> > >> > should
> > >> > > > > avoid
> > >> > > > > > >> > >    occupying a generic lineage path too early if we
> want
> > >> > room
> > >> > > for
> > >> > > > > > >> other
> > >> > > > > > >> > >    lineage systems later.
> > >> > > > > > >> > >
> > >> > > > > > >> > > I believe the boundary rules previewed in sync
> meeting
> > >> are
> > >> > > useful
> > >> > > > > > for
> > >> > > > > > >> > > reviewing PR #4115, but it is not
> > >> > > > > > >> > > a final module-layout proposal.
> > >> > > > > > >> > >
> > >> > > > > > >> > > For metrics specifically, I think the target remains:
> > >> > > > > > >> > >    runtime
> > >> > > > > > >> > >      -> metrics reporting/emitting contract
> > >> > > > > > >> > >      -> default implementation
> > >> > > > > > >> > >           -> lower implementation details, if needed
> > >> > > > > > >> > >
> > >> > > > > > >> > > That keeps the default simple, while allowing durable
> > >> JDBC,
> > >> > > > > > >> event-backed,
> > >> > > > > > >> > > or external-queue implementations to be added
> separately.
> > >> > > > > > >> > >
> > >> > > > > > >> > > We also touched on the split between Iceberg metrics
> > >> > emitting
> > >> > > and
> > >> > > > > > >> > > Polaris-owned metrics querying. The querying side can
> > >> keep
> > >> > > > > evolving,
> > >> > > > > > >> > > especially for dashboard or generic-table use cases,
> but
> > >> I
> > >> > do
> > >> > > not
> > >> > > > > > >> think
> > >> > > > > > >> > > that needs to block the intake path in this PR.
> > >> > > > > > >> > >
> > >> > > > > > >> > > Thanks,
> > >> > > > > > >> > > -ej
> > >> > > > > > >> > >
> > >> > > > > > >> > > On Tue, Jun 16, 2026 at 2:17 PM Yufei Gu <
> > >> > > flyrain000@gmail.com>
> > >> > > > > > >> wrote:
> > >> > > > > > >> > >
> > >> > > > > > >> > > > Thanks for chiming in, EJ. Agreed that the default
> > >> battery
> > >> > > > > should
> > >> > > > > > >> stay
> > >> > > > > > >> > > > small. I'm leaning toward not including metrics
> > >> > persistence
> > >> > > in
> > >> > > > > the
> > >> > > > > > >> > > default
> > >> > > > > > >> > > > battery.
> > >> > > > > > >> > > >
> > >> > > > > > >> > > > Yufei
> > >> > > > > > >> > > >
> > >> > > > > > >> > > >
> > >> > > > > > >> > > > On Thu, Jun 11, 2026 at 8:30 PM EJ Wang <
> > >> > > > > > >> > ej.wang.dev.interest@gmail.com>
> > >> > > > > > >> > > > wrote:
> > >> > > > > > >> > > >
> > >> > > > > > >> > > > > Hi Yufei,
> > >> > > > > > >> > > > >
> > >> > > > > > >> > > > > Thanks for connecting this back to the REST
> endpoint
> > >> > > > > proposal. I
> > >> > > > > > >> > > replied
> > >> > > > > > >> > > > > with the fuller version on the event-forwarding
> > >> thread,
> > >> > > but
> > >> > > > > > >> wanted to
> > >> > > > > > >> > > add
> > >> > > > > > >> > > > > the shorter version here too.
> > >> > > > > > >> > > > >
> > >> > > > > > >> > > > > I agree that metrics reporting/emitting is the
> right
> > >> > > > > conceptual
> > >> > > > > > >> > > boundary,
> > >> > > > > > >> > > > > and I also think the event/listener path is a
> good
> > >> > > > > > implementation
> > >> > > > > > >> > > > direction
> > >> > > > > > >> > > > > to explore. The distinction I want to keep clear
> is
> > >> > > default
> > >> > > > > > >> battery
> > >> > > > > > >> > vs
> > >> > > > > > >> > > > > extension implementation.
> > >> > > > > > >> > > > >
> > >> > > > > > >> > > > > For the REST/API side, I would keep the semantics
> > >> > narrow:
> > >> > > > > > Polaris
> > >> > > > > > >> > > > resolves
> > >> > > > > > >> > > > > and authorizes the table, accepts the Iceberg
> > >> > scan/commit
> > >> > > > > > metrics
> > >> > > > > > >> > > report
> > >> > > > > > >> > > > > into the configured reporting/emitting path, and
> > >> returns
> > >> > > 204.
> > >> > > > > > That
> > >> > > > > > >> > 204
> > >> > > > > > >> > > > > should mean Polaris accepted the report into the
> > >> > ingestion
> > >> > > > > path;
> > >> > > > > > >> it
> > >> > > > > > >> > > > should
> > >> > > > > > >> > > > > not imply durable storage.
> > >> > > > > > >> > > > >
> > >> > > > > > >> > > > > Durable storage, event forwarding, external
> telemetry
> > >> > > routing,
> > >> > > > > > >> > > filtering,
> > >> > > > > > >> > > > > and retention should sit behind that
> > >> reporting/emitting
> > >> > > > > boundary
> > >> > > > > > >> as
> > >> > > > > > >> > > > > implementation-layer behavior. A durable JDBC
> path
> > >> can
> > >> > be
> > >> > > one
> > >> > > > > > >> named
> > >> > > > > > >> > > > > extension implementation. An event/listener
> > >> forwarding
> > >> > > path
> > >> > > > > can
> > >> > > > > > be
> > >> > > > > > >> > > > another
> > >> > > > > > >> > > > > named extension implementation.
> > >> > > > > > >> > > > >
> > >> > > > > > >> > > > > So I am +1 on the event-forwarding idea as a
> > >> non-default
> > >> > > > > > extension
> > >> > > > > > >> > > > > implementation. I just would not make it the
> default
> > >> > > battery
> > >> > > > > or
> > >> > > > > > >> the
> > >> > > > > > >> > > core
> > >> > > > > > >> > > > > API shape. The default battery can stay small and
> > >> safe,
> > >> > > while
> > >> > > > > > >> > > deployments
> > >> > > > > > >> > > > > that need forwarding, dashboards, or durable
> storage
> > >> can
> > >> > > opt
> > >> > > > > > into
> > >> > > > > > >> the
> > >> > > > > > >> > > > > implementation that matches their operational
> model.
> > >> > > > > > >> > > > >
> > >> > > > > > >> > > > > Thanks,
> > >> > > > > > >> > > > > -ej
> > >> > > > > > >> > > > >
> > >> > > > > > >> > > > > On Wed, Jun 10, 2026 at 11:43 AM Yufei Gu <
> > >> > > > > flyrain000@gmail.com
> > >> > > > > > >
> > >> > > > > > >> > > wrote:
> > >> > > > > > >> > > > >
> > >> > > > > > >> > > > > > Thanks EJ. I agree that the reporting/emitting
> > >> > boundary
> > >> > > > > feels
> > >> > > > > > >> like
> > >> > > > > > >> > > the
> > >> > > > > > >> > > > > > right SPI boundary, but I wonder if we can
> simplify
> > >> > this
> > >> > > > > even
> > >> > > > > > >> > > further.
> > >> > > > > > >> > > > > >
> > >> > > > > > >> > > > > > Iceberg scan and commit metrics look very
> similar
> > >> to
> > >> > > events
> > >> > > > > to
> > >> > > > > > >> me.
> > >> > > > > > >> > > They
> > >> > > > > > >> > > > > are
> > >> > > > > > >> > > > > > append only, asynchronously consumed, often
> > >> forwarded
> > >> > to
> > >> > > > > > >> external
> > >> > > > > > >> > > > > systems,
> > >> > > > > > >> > > > > > and have similar retention and cleanup
> > >> requirements.
> > >> > > More
> > >> > > > > > >> details
> > >> > > > > > >> > can
> > >> > > > > > >> > > > be
> > >> > > > > > >> > > > > > found in this thread [1]. In that case,
> Polaris may
> > >> > only
> > >> > > > > need
> > >> > > > > > to
> > >> > > > > > >> > > accept
> > >> > > > > > >> > > > > the
> > >> > > > > > >> > > > > > report and emit an event through the existing
> event
> > >> > > > > framework.
> > >> > > > > > >> > > > Filtering,
> > >> > > > > > >> > > > > > async delivery, custom sinks, and retention
> > >> mechanisms
> > >> > > could
> > >> > > > > > >> then
> > >> > > > > > >> > be
> > >> > > > > > >> > > > > shared
> > >> > > > > > >> > > > > > instead of introducing a separate metrics
> specific
> > >> > > extension
> > >> > > > > > >> layer
> > >> > > > > > >> > > and
> > >> > > > > > >> > > > > SPI.
> > >> > > > > > >> > > > > >
> > >> > > > > > >> > > > > > The main requirement I still see is querying
> > >> metrics
> > >> > > through
> > >> > > > > > >> > Polaris.
> > >> > > > > > >> > > > If
> > >> > > > > > >> > > > > we
> > >> > > > > > >> > > > > > want that battery included experience, we could
> > >> work
> > >> > on
> > >> > > a
> > >> > > > > > >> > persistence
> > >> > > > > > >> > > > > > listener implementation that selectively saves
> > >> metrics
> > >> > > and
> > >> > > > > > >> expose
> > >> > > > > > >> > it
> > >> > > > > > >> > > > via
> > >> > > > > > >> > > > > > the IRC event endpoint which seems pretty
> close in
> > >> > > Iceberg
> > >> > > > > > >> > > > > > community. Another option I like more is to
> allow
> > >> > users
> > >> > > to
> > >> > > > > > >> develop
> > >> > > > > > >> > > > their
> > >> > > > > > >> > > > > > own listener, so they can persist scan/commit
> > >> metrics
> > >> > > to any
> > >> > > > > > >> > systems
> > >> > > > > > >> > > > they
> > >> > > > > > >> > > > > > prefer. In general, that feels like an
> > >> implementation
> > >> > > choice
> > >> > > > > > on
> > >> > > > > > >> top
> > >> > > > > > >> > > of
> > >> > > > > > >> > > > > the
> > >> > > > > > >> > > > > > event framework rather than something that
> needs to
> > >> > > shape
> > >> > > > > the
> > >> > > > > > >> core
> > >> > > > > > >> > > > > > architecture.
> > >> > > > > > >> > > > > >
> > >> > > > > > >> > > > > > 1.
> > >> > > > > > >> >
> > >> > >
> > >> >
> > >>
> https://urldefense.com/v3/__https://lists.apache.org/thread/x9j8nscvy8hq61tyn01mj8yp6n9of0kp__;!!Iz9xO38YGHZK!4rnlVIqBQQQ6xpGe1PMV58edOS_cGrKoqdfhw4qSp_JQCavvEjaxXk0OwGPnxyEcO60AzKQK_XDtp7H3DdkFXcrXGcGDgks$
> > >> > > > > > >> > > > > >
> > >> > > > > > >> > > > > > Thanks,
> > >> > > > > > >> > > > > > Yufei
> > >> > > > > > >> > > > > >
> > >> > > > > > >> > > > > >
> > >> > > > > > >> > > > > > On Thu, Jun 4, 2026 at 5:14 PM EJ Wang <
> > >> > > > > > >> > > ej.wang.dev.interest@gmail.com
> > >> > > > > > >> > > > >
> > >> > > > > > >> > > > > > wrote:
> > >> > > > > > >> > > > > >
> > >> > > > > > >> > > > > > > Good point. I agree the existing
> > >> > > PolarisMetricsReporter is
> > >> > > > > > >> > already
> > >> > > > > > >> > > > very
> > >> > > > > > >> > > > > > > close to the right conceptual boundary. I am
> not
> > >> > > > > proposing a
> > >> > > > > > >> > second
> > >> > > > > > >> > > > > > > parallel reporter concept. The distinction I
> am
> > >> > > trying to
> > >> > > > > > >> make is
> > >> > > > > > >> > > > > > narrower:
> > >> > > > > > >> > > > > > >
> > >> > > > > > >> > > > > > > Current state:
> > >> > > > > > >> > > > > > >
> > >> > > > > > >> > > > > > >    - PolarisMetricsReporter is the existing
> > >> metrics
> > >> > > > > > reporting
> > >> > > > > > >> > hook.
> > >> > > > > > >> > > > > > >    - But it currently lives under
> > >> runtime/service.
> > >> > > > > > >> > > > > > >    - Its contract is documented around
> > >> Quarkus/CDI
> > >> > > > > discovery
> > >> > > > > > >> and
> > >> > > > > > >> > > > > > > @Identifier
> > >> > > > > > >> > > > > > >    selection.
> > >> > > > > > >> > > > > > >    - The default implementation is log-only.
> > >> > > > > > >> > > > > > >    - The durable implementation writes
> through
> > >> > > > > > >> > > PolarisMetricsManager
> > >> > > > > > >> > > > > and
> > >> > > > > > >> > > > > > >    MetricsPersistence.
> > >> > > > > > >> > > > > > >    - MetricsPersistence is currently
> inherited by
> > >> > > > > > >> > BasePersistence.
> > >> > > > > > >> > > > > > >
> > >> > > > > > >> > > > > > > My proposal:
> > >> > > > > > >> > > > > > >
> > >> > > > > > >> > > > > > >    - Keep the reporting/emitting boundary as
> the
> > >> > SPI.
> > >> > > > > > >> > > > > > >    - Revise that contract in a
> framework-agnostic
> > >> > > optional
> > >> > > > > > >> > Iceberg
> > >> > > > > > >> > > > > > *metrics
> > >> > > > > > >> > > > > > >    extension API layer*.
> > >> > > > > > >> > > > > > >    - Keep runtime/service responsible for
> REST
> > >> > > ingestion,
> > >> > > > > > >> > > table/authz
> > >> > > > > > >> > > > > > >    validation, and runtime wiring.
> > >> > > > > > >> > > > > > >    - Keep the battery implementation
> log-only in
> > >> the
> > >> > > > > metrics
> > >> > > > > > >> > > > extension
> > >> > > > > > >> > > > > > API
> > >> > > > > > >> > > > > > >    layer, *not under runtime/service*.
> > >> > > > > > >> > > > > > >    - Treat durable JDBC metrics as one
> > >> > implementation
> > >> > > of
> > >> > > > > the
> > >> > > > > > >> > > > reporting
> > >> > > > > > >> > > > > > SPI.
> > >> > > > > > >> > > > > > >    - Decompose the MetricsPersistence and
> > >> > > BasePersistence
> > >> > > > > > >> > coupling
> > >> > > > > > >> > > as
> > >> > > > > > >> > > > > > >    durable implementation detail.
> > >> > > > > > >> > > > > > >
> > >> > > > > > >> > > > > > > So the difference is not “new reporter
> concept vs
> > >> > old
> > >> > > > > > reporter
> > >> > > > > > >> > > > > concept.”
> > >> > > > > > >> > > > > > > but:
> > >> > > > > > >> > > > > > >
> > >> > > > > > >> > > > > > >
> > >> > > > > > >> > > > > > >    - PolarisMetricsReporter today =
> > >> runtime/service
> > >> > > > > > CDI-shaped
> > >> > > > > > >> > > hook.
> > >> > > > > > >> > > > > > >    - Target reporting SPI = same conceptual
> > >> > boundary,
> > >> > > but
> > >> > > > > > >> placed
> > >> > > > > > >> > in
> > >> > > > > > >> > > > the
> > >> > > > > > >> > > > > > >    right extension API layer and kept
> > >> > > framework-agnostic.
> > >> > > > > > >> > > > > > >
> > >> > > > > > >> > > > > > > For the current metrics PR, I think the
> minimal
> > >> > > > > > architectural
> > >> > > > > > >> > > cleanup
> > >> > > > > > >> > > > > > > before merge is to *put the reporting SPI
> > >> contract
> > >> > > and the
> > >> > > > > > >> > battery
> > >> > > > > > >> > > > > > default
> > >> > > > > > >> > > > > > > in the right place*. The durable JDBC
> > >> > implementation,
> > >> > > > > > >> > event-backed
> > >> > > > > > >> > > > > async
> > >> > > > > > >> > > > > > > implementation, external queue support, and
> > >> deeper
> > >> > > > > > persistence
> > >> > > > > > >> > > > cleanup
> > >> > > > > > >> > > > > > can
> > >> > > > > > >> > > > > > > continue as follow-ups, as long as we do not
> lock
> > >> > the
> > >> > > SPI
> > >> > > > > > >> > boundary
> > >> > > > > > >> > > to
> > >> > > > > > >> > > > > > > BasePersistence or to runtime/service wiring.
> > >> > > > > > >> > > > > > >
> > >> > > > > > >> > > > > > > -ej
> > >> > > > > > >> > > > > > >
> > >> > > > > > >> > > > > > > On Wed, Jun 3, 2026 at 3:00 PM Dmitri
> > >> Bourlatchkov <
> > >> > > > > > >> > > dimas@apache.org
> > >> > > > > > >> > > > >
> > >> > > > > > >> > > > > > > wrote:
> > >> > > > > > >> > > > > > >
> > >> > > > > > >> > > > > > > > Hi EJ,
> > >> > > > > > >> > > > > > > >
> > >> > > > > > >> > > > > > > > Thanks for the recap / summary!
> > >> > > > > > >> > > > > > > >
> > >> > > > > > >> > > > > > > > > Do folks agree that the stable SPI
> boundary
> > >> > > should be
> > >> > > > > > >> > > > > > > > metrics reporting/emitting, not metrics
> > >> > persistence?
> > >> > > > > > >> > > > > > > >
> > >> > > > > > >> > > > > > > > SGTM.
> > >> > > > > > >> > > > > > > >
> > >> > > > > > >> > > > > > > > > Does an optional Iceberg metrics
> extension
> > >> API
> > >> > > layer
> > >> > > > > > sound
> > >> > > > > > >> > like
> > >> > > > > > >> > > > the
> > >> > > > > > >> > > > > > > right
> > >> > > > > > >> > > > > > > > home for this SPI?
> > >> > > > > > >> > > > > > > >
> > >> > > > > > >> > > > > > > > How is that different from current
> > >> > > > > PolarisMetricsReporter?
> > >> > > > > > >> > > > > > > >
> > >> > > > > > >> > > > > > > > > Should the current durable metrics work
> be
> > >> > > reframed
> > >> > > > > as a
> > >> > > > > > >> > > durable
> > >> > > > > > >> > > > > > > > JDBC reference implementation of that SPI?
> > >> > > > > > >> > > > > > > >
> > >> > > > > > >> > > > > > > > SGTM.
> > >> > > > > > >> > > > > > > >
> > >> > > > > > >> > > > > > > > > What is the smallest PR sequence to get
> there
> > >> > > without
> > >> > > > > > >> > blocking
> > >> > > > > > >> > > > > > > > the current metrics work unnecessarily?
> > >> > > > > > >> > > > > > > >
> > >> > > > > > >> > > > > > > > Let's get
> > >> > >
> > >> >
> > >>
> https://urldefense.com/v3/__https://github.com/apache/polaris/pull/4397__;!!Iz9xO38YGHZK!4rnlVIqBQQQ6xpGe1PMV58edOS_cGrKoqdfhw4qSp_JQCavvEjaxXk0OwGPnxyEcO60AzKQK_XDtp7H3DdkFXcrXuu5jefU$
> > >> > > > > > >> merged
> > >> > > > > > >> > > > first.
> > >> > > > > > >> > > > > > > >
> > >> > > > > > >> > > > > > > > Then, I'd propose isolating the Metrics
> schema
> > >> > from
> > >> > > the
> > >> > > > > > >> > MetaStore
> > >> > > > > > >> > > > > > schema.
> > >> > > > > > >> > > > > > > > This will probably have some ripple effect
> into
> > >> > the
> > >> > > > > > >> bootstrap
> > >> > > > > > >> > > > > > workflows,
> > >> > > > > > >> > > > > > > so
> > >> > > > > > >> > > > > > > > it's not a trivial change.
> > >> > > > > > >> > > > > > > >
> > >> > > > > > >> > > > > > > > Then, let's reassess.
> > >> > > > > > >> > > > > > > >
> > >> > > > > > >> > > > > > > > Cheers,
> > >> > > > > > >> > > > > > > > Dmitri.
> > >> > > > > > >> > > > > > > >
> > >> > > > > > >> > > > > > > > On Wed, Jun 3, 2026 at 5:40 PM EJ Wang <
> > >> > > > > > >> > > > > ej.wang.dev.interest@gmail.com
> > >> > > > > > >> > > > > > >
> > >> > > > > > >> > > > > > > > wrote:
> > >> > > > > > >> > > > > > > >
> > >> > > > > > >> > > > > > > > > Hi folks,
> > >> > > > > > >> > > > > > > > >
> > >> > > > > > >> > > > > > > > > Thanks again for the discussion today. I
> > >> updated
> > >> > > the
> > >> > > > > > sync
> > >> > > > > > >> doc
> > >> > > > > > >> > > > with
> > >> > > > > > >> > > > > > > notes
> > >> > > > > > >> > > > > > > > > from the third metrics architecture sync:
> > >> > > > > > >> > > > > > > > >
> > >> > > > > > >> > > > > > > > >
> > >> > > > > > >> > > > > > > > >
> > >> > > > > > >> > > > > > > >
> > >> > > > > > >> > > > > > >
> > >> > > > > > >> > > > > >
> > >> > > > > > >> > > > >
> > >> > > > > > >> > > >
> > >> > > > > > >> > >
> > >> > > > > > >> >
> > >> > > > > > >>
> > >> > > > > >
> > >> > > > >
> > >> > >
> > >> >
> > >>
> https://urldefense.com/v3/__https://docs.google.com/document/d/100h7c4damrUzVuquYbBHM0EvA4LSWuW2IT2dN_7nYVA/edit?tab=t.k96s2xyqr5u1__;!!Iz9xO38YGHZK!4rnlVIqBQQQ6xpGe1PMV58edOS_cGrKoqdfhw4qSp_JQCavvEjaxXk0OwGPnxyEcO60AzKQK_XDtp7H3DdkFXcrXrqCL-QE$
> > >> > > > > > >> > > > > > > > >
> > >> > > > > > >> > > > > > > > > A few highlights from today’s discussion:
> > >> > > > > > >> > > > > > > > >
> > >> > > > > > >> > > > > > > > >    - We clarified that Iceberg metrics
> > >> reporting
> > >> > > can
> > >> > > > > be
> > >> > > > > > >> > > > interpreted
> > >> > > > > > >> > > > > > > > either
> > >> > > > > > >> > > > > > > > >    as sync or async handling from
> Polaris’s
> > >> > > > > perspective,
> > >> > > > > > >> so
> > >> > > > > > >> > > > Polaris
> > >> > > > > > >> > > > > > > > should
> > >> > > > > > >> > > > > > > > >    stay flexible as a platform instead of
> > >> baking
> > >> > > one
> > >> > > > > > >> handling
> > >> > > > > > >> > > > model
> > >> > > > > > >> > > > > > > into
> > >> > > > > > >> > > > > > > > > the
> > >> > > > > > >> > > > > > > > >    REST API behavior.
> > >> > > > > > >> > > > > > > > >    - We aligned that the built-in
> (battery)
> > >> > > behavior
> > >> > > > > for
> > >> > > > > > >> > > metrics
> > >> > > > > > >> > > > > > > emitting
> > >> > > > > > >> > > > > > > > >    can stay simple: no-op/log-only is
> enough
> > >> as
> > >> > > the
> > >> > > > > > >> default.
> > >> > > > > > >> > > > > > > > >    - Durable metrics persistence should
> be
> > >> > > treated as
> > >> > > > > an
> > >> > > > > > >> > > > > > implementation
> > >> > > > > > >> > > > > > > > of
> > >> > > > > > >> > > > > > > > >    the metrics reporting path, not as the
> > >> core
> > >> > SPI
> > >> > > > > > >> boundary.
> > >> > > > > > >> > > > > > > > >    - The existing durable metrics work
> can be
> > >> > > reviewed
> > >> > > > > > as
> > >> > > > > > >> a
> > >> > > > > > >> > > > > reference
> > >> > > > > > >> > > > > > > > >    implementation of the reporting SPI,
> with
> > >> > > > > > >> > > persistence-related
> > >> > > > > > >> > > > > > logic
> > >> > > > > > >> > > > > > > > kept
> > >> > > > > > >> > > > > > > > >    self-contained under a metrics durable
> > >> > > > > implementation
> > >> > > > > > >> > module
> > >> > > > > > >> > > > > > rather
> > >> > > > > > >> > > > > > > > than
> > >> > > > > > >> > > > > > > > >    scattered through core entity
> persistence.
> > >> > > > > > >> > > > > > > > >    - Dashboard/insights remains a real
> use
> > >> case,
> > >> > > but
> > >> > > > > we
> > >> > > > > > >> > agreed
> > >> > > > > > >> > > to
> > >> > > > > > >> > > > > > keep
> > >> > > > > > >> > > > > > > it
> > >> > > > > > >> > > > > > > > >    separate from the core metrics intake
> > >> > > discussion
> > >> > > > > for
> > >> > > > > > >> now.
> > >> > > > > > >> > > > > > > > >
> > >> > > > > > >> > > > > > > > > I also did a quick source check after the
> > >> > meeting
> > >> > > to
> > >> > > > > > make
> > >> > > > > > >> > sure
> > >> > > > > > >> > > we
> > >> > > > > > >> > > > > are
> > >> > > > > > >> > > > > > > > > describing the current state accurately.
> > >> > > > > > >> > > > > > > > >
> > >> > > > > > >> > > > > > > > > Current state:
> > >> > > > > > >> > > > > > > > >
> > >> > > > > > >> > > > > > > > >    - Polaris already has a metrics
> reporting
> > >> > hook:
> > >> > > > > > >> > > > > > > > PolarisMetricsReporter.
> > >> > > > > > >> > > > > > > > >    - The default implementation is
> > >> > > > > > DefaultMetricsReporter,
> > >> > > > > > >> > > > selected
> > >> > > > > > >> > > > > > by
> > >> > > > > > >> > > > > > > > >
> > >> > > polaris.iceberg-metrics.reporting.type=default. It
> > >> > > > > is
> > >> > > > > > >> > > > log-only,
> > >> > > > > > >> > > > > > > > >    effectively quiet unless metrics
> logging
> > >> is
> > >> > > > > enabled.
> > >> > > > > > >> > > > > > > > >    - There is also a
> > >> PersistingMetricsReporter,
> > >> > > > > selected
> > >> > > > > > >> by
> > >> > > > > > >> > > > > > > > >
> > >> > > polaris.iceberg-metrics.reporting.type=persisting,
> > >> > > > > > >> which
> > >> > > > > > >> > > > > converts
> > >> > > > > > >> > > > > > > > >    Iceberg scan/commit reports into
> Polaris
> > >> > > metrics
> > >> > > > > > >> records
> > >> > > > > > >> > and
> > >> > > > > > >> > > > > > writes
> > >> > > > > > >> > > > > > > > > through
> > >> > > > > > >> > > > > > > > >    PolarisMetricsManager ->
> > >> MetricsPersistence.
> > >> > > > > > >> > > > > > > > >    - MetricsPersistence currently exists
> in
> > >> the
> > >> > > > > > >> persistence
> > >> > > > > > >> > > layer
> > >> > > > > > >> > > > > and
> > >> > > > > > >> > > > > > > is
> > >> > > > > > >> > > > > > > > >    inherited by BasePersistence.
> > >> > > > > > >> > > > > > > > >
> > >> > > > > > >> > > > > > > > > My read from the discussion is that the
> > >> target
> > >> > > > > boundary
> > >> > > > > > >> > should
> > >> > > > > > >> > > > not
> > >> > > > > > >> > > > > be
> > >> > > > > > >> > > > > > > > > MetricsPersistence as inherited by
> > >> > > BasePersistence.
> > >> > > > > That
> > >> > > > > > >> path
> > >> > > > > > >> > > > > should
> > >> > > > > > >> > > > > > be
> > >> > > > > > >> > > > > > > > > decomposed as durable implementation
> detail
> > >> > (taken
> > >> > > > > care
> > >> > > > > > >> of by
> > >> > > > > > >> > > > > > > > >
> > >> >
> > >>
> https://urldefense.com/v3/__https://github.com/apache/polaris/pull/4397__;!!Iz9xO38YGHZK!4rnlVIqBQQQ6xpGe1PMV58edOS_cGrKoqdfhw4qSp_JQCavvEjaxXk0OwGPnxyEcO60AzKQK_XDtp7H3DdkFXcrXuu5jefU$
> > >> ).
> > >> > The
> > >> > > > > > stable
> > >> > > > > > >> > > > extension
> > >> > > > > > >> > > > > > > point
> > >> > > > > > >> > > > > > > > > should instead be the metrics
> > >> reporting/emitting
> > >> > > > > > boundary.
> > >> > > > > > >> > > > > > > > >
> > >> > > > > > >> > > > > > > > > Concretely, I think the next proposal
> should
> > >> be
> > >> > > shaped
> > >> > > > > > >> like
> > >> > > > > > >> > > this:
> > >> > > > > > >> > > > > > > > >
> > >> > > > > > >> > > > > > > > >    1.
> > >> > > > > > >> > > > > > > > >
> > >> > > > > > >> > > > > > > > >    Define the proper metrics reporting
> SPI
> > >> at an
> > >> > > > > > optional
> > >> > > > > > >> > > Iceberg
> > >> > > > > > >> > > > > > > metrics
> > >> > > > > > >> > > > > > > > >    extension API layer.
> > >> > > > > > >> > > > > > > > >
> > >> > > > > > >> > > > > > > > >    Example direction:
> > >> > > > > > >> > > > > > > > >
> > >> > > > > > >> > > > > > > > >    public interface
> IcebergMetricsReporter {
> > >> > > > > > >> > > > > > > > >      void
> > >> > reportMetric(IcebergMetricsReportContext
> > >> > > > > > >> context,
> > >> > > > > > >> > > > > > > > > MetricsReport report);
> > >> > > > > > >> > > > > > > > >    }
> > >> > > > > > >> > > > > > > > >
> > >> > > > > > >> > > > > > > > >    The context would carry the small
> > >> > > Polaris-resolved
> > >> > > > > > >> > envelope:
> > >> > > > > > >> > > > > > > > >    catalog/table identity, report type,
> > >> received
> > >> > > > > > >> timestamp,
> > >> > > > > > >> > and
> > >> > > > > > >> > > > > > > > > request/trace
> > >> > > > > > >> > > > > > > > >    context if available. The raw Iceberg
> > >> > > MetricsReport
> > >> > > > > > >> > remains
> > >> > > > > > >> > > > the
> > >> > > > > > >> > > > > > > > payload.
> > >> > > > > > >> > > > > > > > >    2.
> > >> > > > > > >> > > > > > > > >
> > >> > > > > > >> > > > > > > > >    Keep runtime/service as the REST
> ingestion
> > >> > and
> > >> > > > > wiring
> > >> > > > > > >> > layer.
> > >> > > > > > >> > > > > > > > >
> > >> > > > > > >> > > > > > > > >    The REST handler still resolves the
> table
> > >> and
> > >> > > > > > performs
> > >> > > > > > >> > authz
> > >> > > > > > >> > > > > > before
> > >> > > > > > >> > > > > > > > >    accepting the report. After that, it
> calls
> > >> > the
> > >> > > > > > selected
> > >> > > > > > >> > > > > reporting
> > >> > > > > > >> > > > > > > > >    implementation.
> > >> > > > > > >> > > > > > > > >    3.
> > >> > > > > > >> > > > > > > > >
> > >> > > > > > >> > > > > > > > >    Keep the battery default
> no-op/log-only.
> > >> > > > > > >> > > > > > > > >
> > >> > > > > > >> > > > > > > > >    This preserves an out-of-box safe
> default
> > >> and
> > >> > > > > avoids
> > >> > > > > > >> > > > requiring a
> > >> > > > > > >> > > > > > > > durable
> > >> > > > > > >> > > > > > > > >    metrics store for every Polaris
> > >> deployment.
> > >> > > > > > >> > > > > > > > >    4.
> > >> > > > > > >> > > > > > > > >
> > >> > > > > > >> > > > > > > > >    Move durable JDBC metrics into a
> > >> > self-contained
> > >> > > > > > >> > > implementation
> > >> > > > > > >> > > > > of
> > >> > > > > > >> > > > > > > the
> > >> > > > > > >> > > > > > > > >    reporting SPI.
> > >> > > > > > >> > > > > > > > >
> > >> > > > > > >> > > > > > > > >    That implementation can own its
> schema,
> > >> > > bootstrap,
> > >> > > > > > >> > > retention,
> > >> > > > > > >> > > > > and
> > >> > > > > > >> > > > > > > read
> > >> > > > > > >> > > > > > > > >    API support. It should not define the
> core
> > >> > > > > reporting
> > >> > > > > > >> SPI
> > >> > > > > > >> > > > > boundary,
> > >> > > > > > >> > > > > > > and
> > >> > > > > > >> > > > > > > > > it
> > >> > > > > > >> > > > > > > > >    should not require metrics
> persistence to
> > >> > > remain
> > >> > > > > > >> inherited
> > >> > > > > > >> > > > from
> > >> > > > > > >> > > > > > > > >    BasePersistence.
> > >> > > > > > >> > > > > > > > >    5.
> > >> > > > > > >> > > > > > > > >
> > >> > > > > > >> > > > > > > > >    Treat async/event-backed handling as
> > >> another
> > >> > > > > > >> > implementation
> > >> > > > > > >> > > of
> > >> > > > > > >> > > > > the
> > >> > > > > > >> > > > > > > > same
> > >> > > > > > >> > > > > > > > >    reporting SPI.
> > >> > > > > > >> > > > > > > > >
> > >> > > > > > >> > > > > > > > >    For example, an event-backed reporter
> > >> could
> > >> > > > > enqueue a
> > >> > > > > > >> > > metrics
> > >> > > > > > >> > > > > > event
> > >> > > > > > >> > > > > > > > and
> > >> > > > > > >> > > > > > > > >    let listeners handle durable storage
> or
> > >> other
> > >> > > > > sinks.
> > >> > > > > > >> If we
> > >> > > > > > >> > > > later
> > >> > > > > > >> > > > > > > need
> > >> > > > > > >> > > > > > > > a
> > >> > > > > > >> > > > > > > > >    replaceable queue engine, that seems
> like
> > >> a
> > >> > > shared
> > >> > > > > > >> > > > event/metrics
> > >> > > > > > >> > > > > > > > > substrate
> > >> > > > > > >> > > > > > > > >    topic rather than a metrics-only
> > >> requirement.
> > >> > > > > > >> > > > > > > > >
> > >> > > > > > >> > > > > > > > > This framing lets us keep the REST
> metrics
> > >> > > endpoint
> > >> > > > > > >> simple,
> > >> > > > > > >> > > > > preserve
> > >> > > > > > >> > > > > > > the
> > >> > > > > > >> > > > > > > > > current default behavior, support durable
> > >> > metrics
> > >> > > > > users,
> > >> > > > > > >> and
> > >> > > > > > >> > > > still
> > >> > > > > > >> > > > > > > leave
> > >> > > > > > >> > > > > > > > > room for async/event-backed or
> > >> > > external-queue-based
> > >> > > > > > >> > > > > implementations.
> > >> > > > > > >> > > > > > > > >
> > >> > > > > > >> > > > > > > > > I think the main follow-up questions are:
> > >> > > > > > >> > > > > > > > >
> > >> > > > > > >> > > > > > > > >    - Do folks agree that the stable SPI
> > >> boundary
> > >> > > > > should
> > >> > > > > > be
> > >> > > > > > >> > > > metrics
> > >> > > > > > >> > > > > > > > >    reporting/emitting, not metrics
> > >> persistence?
> > >> > > > > > >> > > > > > > > >    - Does an optional Iceberg metrics
> > >> extension
> > >> > > API
> > >> > > > > > layer
> > >> > > > > > >> > sound
> > >> > > > > > >> > > > > like
> > >> > > > > > >> > > > > > > the
> > >> > > > > > >> > > > > > > > >    right home for this SPI?
> > >> > > > > > >> > > > > > > > >    - Should the current durable metrics
> work
> > >> be
> > >> > > > > reframed
> > >> > > > > > >> as a
> > >> > > > > > >> > > > > durable
> > >> > > > > > >> > > > > > > > JDBC
> > >> > > > > > >> > > > > > > > >    reference implementation of that SPI?
> > >> > > > > > >> > > > > > > > >    - What is the smallest PR sequence to
> get
> > >> > there
> > >> > > > > > without
> > >> > > > > > >> > > > blocking
> > >> > > > > > >> > > > > > the
> > >> > > > > > >> > > > > > > > >    current metrics work unnecessarily?
> > >> > > > > > >> > > > > > > > >
> > >> > > > > > >> > > > > > > > > Thanks,
> > >> > > > > > >> > > > > > > > > -ej
> > >> > > > > > >> > > > > > > > >
> > >> > > > > > >> > > > > > > > > On Fri, May 15, 2026 at 5:16 PM Dmitri
> > >> > > Bourlatchkov <
> > >> > > > > > >> > > > > > dimas@apache.org>
> > >> > > > > > >> > > > > > > > > wrote:
> > >> > > > > > >> > > > > > > > >
> > >> > > > > > >> > > > > > > > > > Hi JB,
> > >> > > > > > >> > > > > > > > > >
> > >> > > > > > >> > > > > > > > > > Could you set up another meeting,
> please?
> > >> Same
> > >> > > time
> > >> > > > > on
> > >> > > > > > >> > > > Wednesday
> > >> > > > > > >> > > > > as
> > >> > > > > > >> > > > > > > > last
> > >> > > > > > >> > > > > > > > > > time... I hope it works for everyone.
> > >> > > > > > >> > > > > > > > > >
> > >> > > > > > >> > > > > > > > > > Cheers,
> > >> > > > > > >> > > > > > > > > > Dmitri.
> > >> > > > > > >> > > > > > > > > >
> > >> > > > > > >> > > > > > > > > > On Fri, May 15, 2026 at 8:06 PM Yufei
> Gu <
> > >> > > > > > >> > > flyrain000@gmail.com
> > >> > > > > > >> > > > >
> > >> > > > > > >> > > > > > > wrote:
> > >> > > > > > >> > > > > > > > > >
> > >> > > > > > >> > > > > > > > > > > +1 on another sync call next week.
> > >> > > > > > >> > > > > > > > > > >
> > >> > > > > > >> > > > > > > > > > > Yufei
> > >> > > > > > >> > > > > > > > > > >
> > >> > > > > > >> > > > > > > > > > >
> > >> > > > > > >> > > > > > > > > > > On Fri, May 15, 2026 at 4:52 PM
> Dmitri
> > >> > > > > Bourlatchkov
> > >> > > > > > <
> > >> > > > > > >> > > > > > > > dimas@apache.org>
> > >> > > > > > >> > > > > > > > > > > wrote:
> > >> > > > > > >> > > > > > > > > > >
> > >> > > > > > >> > > > > > > > > > > > Hi All,
> > >> > > > > > >> > > > > > > > > > > >
> > >> > > > > > >> > > > > > > > > > > > WDYT about another sync call next
> week?
> > >> > > > > > >> > > > > > > > > > > >
> > >> > > > > > >> > > > > > > > > > > > Thanks,
> > >> > > > > > >> > > > > > > > > > > > Dmitri.
> > >> > > > > > >> > > > > > > > > > > >
> > >> > > > > > >> > > > > > > > > > > > On Wed, May 6, 2026 at 5:29 PM
> Dmitri
> > >> > > > > > Bourlatchkov <
> > >> > > > > > >> > > > > > > > dimas@apache.org
> > >> > > > > > >> > > > > > > > > >
> > >> > > > > > >> > > > > > > > > > > > wrote:
> > >> > > > > > >> > > > > > > > > > > >
> > >> > > > > > >> > > > > > > > > > > > > Hi EJ,
> > >> > > > > > >> > > > > > > > > > > > >
> > >> > > > > > >> > > > > > > > > > > > > Thanks for the summary! It covers
> > >> what
> > >> > we
> > >> > > > > > >> discussed
> > >> > > > > > >> > in
> > >> > > > > > >> > > > the
> > >> > > > > > >> > > > > > > > meeting
> > >> > > > > > >> > > > > > > > > > very
> > >> > > > > > >> > > > > > > > > > > > > well, IMHO.
> > >> > > > > > >> > > > > > > > > > > > >
> > >> > > > > > >> > > > > > > > > > > > > Looking forward to concrete PRs
> :)
> > >> > > > > > >> > > > > > > > > > > > >
> > >> > > > > > >> > > > > > > > > > > > > Cheers,
> > >> > > > > > >> > > > > > > > > > > > > Dmitri.
> > >> > > > > > >> > > > > > > > > > > > >
> > >> > > > > > >> > > > > > > > > > > > > On Wed, May 6, 2026 at 5:08 PM EJ
> > >> Wang <
> > >> > > > > > >> > > > > > > > > > ej.wang.dev.interest@gmail.com
> > >> > > > > > >> > > > > > > > > > > >
> > >> > > > > > >> > > > > > > > > > > > > wrote:
> > >> > > > > > >> > > > > > > > > > > > >
> > >> > > > > > >> > > > > > > > > > > > >> Hi folks,
> > >> > > > > > >> > > > > > > > > > > > >>
> > >> > > > > > >> > > > > > > > > > > > >> We had a community sync earlier,
> > >> thanks
> > >> > > JB
> > >> > > > > for
> > >> > > > > > >> > > > scheduling
> > >> > > > > > >> > > > > > it.
> > >> > > > > > >> > > > > > > > > Notes
> > >> > > > > > >> > > > > > > > > > > from
> > >> > > > > > >> > > > > > > > > > > > >> the first metrics architecture
> sync
> > >> > (May
> > >> > > 6,
> > >> > > > > > >> 10-11am
> > >> > > > > > >> > > PT).
> > >> > > > > > >> > > > > > > > > Discussion
> > >> > > > > > >> > > > > > > > > > > doc
> > >> > > > > > >> > > > > > > > > > > > >> with per-section status:
> > >> > > > > > >> > > > > > > > > > > > >>
> > >> > > > > > >> > > > > > > > > > > > >>
> > >> > > > > > >> > > > > > > > > > > >
> > >> > > > > > >> > > > > > > > > > >
> > >> > > > > > >> > > > > > > > > >
> > >> > > > > > >> > > > > > > > >
> > >> > > > > > >> > > > > > > >
> > >> > > > > > >> > > > > > >
> > >> > > > > > >> > > > > >
> > >> > > > > > >> > > > >
> > >> > > > > > >> > > >
> > >> > > > > > >> > >
> > >> > > > > > >> >
> > >> > > > > > >>
> > >> > > > > >
> > >> > > > >
> > >> > >
> > >> >
> > >>
> https://urldefense.com/v3/__https://docs.google.com/document/d/100h7c4damrUzVuquYbBHM0EvA4LSWuW2IT2dN_7nYVA/edit?tab=t.0__;!!Iz9xO38YGHZK!4rnlVIqBQQQ6xpGe1PMV58edOS_cGrKoqdfhw4qSp_JQCavvEjaxXk0OwGPnxyEcO60AzKQK_XDtp7H3DdkFXcrXgIddeUs$
> > >> > > > > > >> > > > > > > > > > > > >>
> > >> > > > > > >> > > > > > > > > > > > >> *The meeting covered both topics
> > >> from
> > >> > the
> > >> > > > > doc.
> > >> > > > > > >> > > > > > Direction-level
> > >> > > > > > >> > > > > > > > > > > alignment
> > >> > > > > > >> > > > > > > > > > > > >> was reached on the headline
> pieces;
> > >> > > details
> > >> > > > > > >> remain
> > >> > > > > > >> > for
> > >> > > > > > >> > > > PR
> > >> > > > > > >> > > > > > > review
> > >> > > > > > >> > > > > > > > > or
> > >> > > > > > >> > > > > > > > > > > > >> follow-up sessions.*
> > >> > > > > > >> > > > > > > > > > > > >>
> > >> > > > > > >> > > > > > > > > > > > >> *Topic 1 — Persistence schema
> > >> redesign*
> > >> > > > > > >> > > > > > > > > > > > >> Idea-level alignment on
> > >> consolidating
> > >> > > > > per-type
> > >> > > > > > >> > tables
> > >> > > > > > >> > > > > > > > > > > > >> (scan_metrics_report,
> > >> > > > > > >> > > > > > > > > > > > >> commit_metrics_report) into a
> single
> > >> > > > > > >> metrics_report
> > >> > > > > > >> > > > table.
> > >> > > > > > >> > > > > > The
> > >> > > > > > >> > > > > > > > > > > > motivating
> > >> > > > > > >> > > > > > > > > > > > >> cost is the surface area added
> by
> > >> every
> > >> > > new
> > >> > > > > > >> metric
> > >> > > > > > >> > > type
> > >> > > > > > >> > > > > > today:
> > >> > > > > > >> > > > > > > > new
> > >> > > > > > >> > > > > > > > > > > > table,
> > >> > > > > > >> > > > > > > > > > > > >> SPI method, record class, model,
> > >> > > converter,
> > >> > > > > > >> schema
> > >> > > > > > >> > > > > > migration.
> > >> > > > > > >> > > > > > > > > > > > >>
> > >> > > > > > >> > > > > > > > > > > > >> Most schema details are
> deferred to
> > >> the
> > >> > > > > schema
> > >> > > > > > >> PR. A
> > >> > > > > > >> > > few
> > >> > > > > > >> > > > > > > > specific
> > >> > > > > > >> > > > > > > > > > > points
> > >> > > > > > >> > > > > > > > > > > > >> came up:
> > >> > > > > > >> > > > > > > > > > > > >> •   metric_schema_version: Yufei
> > >> > prefers
> > >> > > > > > dropping
> > >> > > > > > >> > it,
> > >> > > > > > >> > > > > since
> > >> > > > > > >> > > > > > > > there
> > >> > > > > > >> > > > > > > > > is
> > >> > > > > > >> > > > > > > > > > > no
> > >> > > > > > >> > > > > > > > > > > > >> spec-level concept of metrics
> > >> > versioning
> > >> > > > > today
> > >> > > > > > >> and
> > >> > > > > > >> > it
> > >> > > > > > >> > > is
> > >> > > > > > >> > > > > > hard
> > >> > > > > > >> > > > > > > to
> > >> > > > > > >> > > > > > > > > > > define
> > >> > > > > > >> > > > > > > > > > > > >> unilaterally. Robert prefers
> keeping
> > >> > it,
> > >> > > > > given
> > >> > > > > > >> IRC
> > >> > > > > > >> > v2
> > >> > > > > > >> > > is
> > >> > > > > > >> > > > > > > coming
> > >> > > > > > >> > > > > > > > > and
> > >> > > > > > >> > > > > > > > > > > the
> > >> > > > > > >> > > > > > > > > > > > >> schema should be considered
> against
> > >> its
> > >> > > > > likely
> > >> > > > > > >> > shape;
> > >> > > > > > >> > > > > Robert
> > >> > > > > > >> > > > > > > > also
> > >> > > > > > >> > > > > > > > > > > raised
> > >> > > > > > >> > > > > > > > > > > > >> how to differentiate various
> payload
> > >> > > formats
> > >> > > > > if
> > >> > > > > > >> any.
> > >> > > > > > >> > > > EJ's
> > >> > > > > > >> > > > > > read
> > >> > > > > > >> > > > > > > > is
> > >> > > > > > >> > > > > > > > > > that
> > >> > > > > > >> > > > > > > > > > > > >> this
> > >> > > > > > >> > > > > > > > > > > > >> is a two-way-door decision. We
> can
> > >> > start
> > >> > > > > > without
> > >> > > > > > >> the
> > >> > > > > > >> > > > > field,
> > >> > > > > > >> > > > > > > and
> > >> > > > > > >> > > > > > > > if
> > >> > > > > > >> > > > > > > > > > IRC
> > >> > > > > > >> > > > > > > > > > > > v2
> > >> > > > > > >> > > > > > > > > > > > >> changes the shape we would
> likely
> > >> roll
> > >> > a
> > >> > > > > > >> > corresponding
> > >> > > > > > >> > > > new
> > >> > > > > > >> > > > > > > > schema
> > >> > > > > > >> > > > > > > > > > > > anyway,
> > >> > > > > > >> > > > > > > > > > > > >> which is not particularly
> costly.
> > >> > > > > > >> > > > > > > > > > > > >> •   Payload format: Robert
> pointed
> > >> out
> > >> > > that
> > >> > > > > > >> future
> > >> > > > > > >> > > > formats
> > >> > > > > > >> > > > > > > > beyond
> > >> > > > > > >> > > > > > > > > > JSON
> > >> > > > > > >> > > > > > > > > > > > may
> > >> > > > > > >> > > > > > > > > > > > >> be worth supporting. The exact
> > >> shape is
> > >> > > > > > deferred
> > >> > > > > > >> to
> > >> > > > > > >> > > the
> > >> > > > > > >> > > > > > schema
> > >> > > > > > >> > > > > > > > > > > > discussion.
> > >> > > > > > >> > > > > > > > > > > > >> •   Partition strategy: Anand
> > >> suggested
> > >> > > > > monthly
> > >> > > > > > >> > > > > partitioning
> > >> > > > > > >> > > > > > > > based
> > >> > > > > > >> > > > > > > > > > on
> > >> > > > > > >> > > > > > > > > > > > his
> > >> > > > > > >> > > > > > > > > > > > >> experience as potentially
> helpful at
> > >> > > scale.
> > >> > > > > > >> > > > > > > > > > > > >>
> > >> > > > > > >> > > > > > > > > > > > >> *Topic 2 — Where metrics
> ingestion
> > >> and
> > >> > > > > storage
> > >> > > > > > >> > belong*
> > >> > > > > > >> > > > > > > > > > > > >> Idea-level alignment that
> metrics
> > >> > should
> > >> > > be a
> > >> > > > > > >> > > separated
> > >> > > > > > >> > > > > SPI
> > >> > > > > > >> > > > > > > from
> > >> > > > > > >> > > > > > > > > the
> > >> > > > > > >> > > > > > > > > > > > >> entity
> > >> > > > > > >> > > > > > > > > > > > >> persistence stack. Two reasons
> > >> > surfaced:
> > >> > > (a)
> > >> > > > > > >> > workloads
> > >> > > > > > >> > > > and
> > >> > > > > > >> > > > > > > > > > capability
> > >> > > > > > >> > > > > > > > > > > > >> requirements diverge enough that
> > >> > coupling
> > >> > > > > them
> > >> > > > > > >> > creates
> > >> > > > > > >> > > > > > > > artificial
> > >> > > > > > >> > > > > > > > > > > > >> constraints, and (b) admin
> > >> experience
> > >> > > > > improves
> > >> > > > > > >> when
> > >> > > > > > >> > > > > metrics
> > >> > > > > > >> > > > > > > has
> > >> > > > > > >> > > > > > > > > its
> > >> > > > > > >> > > > > > > > > > > own
> > >> > > > > > >> > > > > > > > > > > > >> bootstrap, retention, and
> lifecycle.
> > >> > > Dmitri
> > >> > > > > > noted
> > >> > > > > > >> > > > Polaris
> > >> > > > > > >> > > > > > > being
> > >> > > > > > >> > > > > > > > a
> > >> > > > > > >> > > > > > > > > > > > platform
> > >> > > > > > >> > > > > > > > > > > > >> should have the flexibility to
> > >> support
> > >> > > > > > different
> > >> > > > > > >> > > > > persistence
> > >> > > > > > >> > > > > > > > > > backends
> > >> > > > > > >> > > > > > > > > > > > per
> > >> > > > > > >> > > > > > > > > > > > >> concern, and pointed to a
> concrete
> > >> next
> > >> > > step
> > >> > > > > of
> > >> > > > > > >> > > > separating
> > >> > > > > > >> > > > > > the
> > >> > > > > > >> > > > > > > > > JDBC
> > >> > > > > > >> > > > > > > > > > > > >> bootstrap for metrics from the
> > >> > metastore
> > >> > > > > > >> bootstrap.
> > >> > > > > > >> > > > Robert
> > >> > > > > > >> > > > > > > > > proposed
> > >> > > > > > >> > > > > > > > > > an
> > >> > > > > > >> > > > > > > > > > > > >> additional UX extension: detect
> an
> > >> > > > > > unbootstrapped
> > >> > > > > > >> > > > metrics
> > >> > > > > > >> > > > > > > store
> > >> > > > > > >> > > > > > > > on
> > >> > > > > > >> > > > > > > > > > > first
> > >> > > > > > >> > > > > > > > > > > > >> use and auto-bootstrap rather
> than
> > >> > > requiring
> > >> > > > > an
> > >> > > > > > >> > > explicit
> > >> > > > > > >> > > > > > > manual
> > >> > > > > > >> > > > > > > > > > > > bootstrap
> > >> > > > > > >> > > > > > > > > > > > >> step.
> > >> > > > > > >> > > > > > > > > > > > >> The meeting also confirmed that
> > >> Polaris
> > >> > > > > metrics
> > >> > > > > > >> can
> > >> > > > > > >> > > > start
> > >> > > > > > >> > > > > > > small
> > >> > > > > > >> > > > > > > > > and
> > >> > > > > > >> > > > > > > > > > > stay
> > >> > > > > > >> > > > > > > > > > > > >> Iceberg-focused. Naming and
> > >> persistence
> > >> > > > > schema
> > >> > > > > > >> can
> > >> > > > > > >> > > lean
> > >> > > > > > >> > > > > > > > > > > > Iceberg-specific.
> > >> > > > > > >> > > > > > > > > > > > >> If a future expansion to
> > >> generic-table
> > >> > > > > metrics
> > >> > > > > > or
> > >> > > > > > >> > > > > > operational
> > >> > > > > > >> > > > > > > > > > metrics
> > >> > > > > > >> > > > > > > > > > > > >> arrives, an abstraction layer
> can be
> > >> > > built on
> > >> > > > > > >> top of
> > >> > > > > > >> > > the
> > >> > > > > > >> > > > > > > Iceberg
> > >> > > > > > >> > > > > > > > > > > metrics
> > >> > > > > > >> > > > > > > > > > > > >> reporter at that point. Robert
> > >> remains
> > >> > > on the
> > >> > > > > > >> fence
> > >> > > > > > >> > > and
> > >> > > > > > >> > > > > > would
> > >> > > > > > >> > > > > > > > > prefer
> > >> > > > > > >> > > > > > > > > > > > >> something more generic but did
> not
> > >> > block
> > >> > > the
> > >> > > > > > >> > > direction;
> > >> > > > > > >> > > > > > > Dmitri's
> > >> > > > > > >> > > > > > > > > > read
> > >> > > > > > >> > > > > > > > > > > > was
> > >> > > > > > >> > > > > > > > > > > > >> that the proposed framework
> already
> > >> has
> > >> > > > > enough
> > >> > > > > > >> > > > flexibility
> > >> > > > > > >> > > > > > to
> > >> > > > > > >> > > > > > > > > absorb
> > >> > > > > > >> > > > > > > > > > > > >> future
> > >> > > > > > >> > > > > > > > > > > > >> expansion.
> > >> > > > > > >> > > > > > > > > > > > >>
> > >> > > > > > >> > > > > > > > > > > > >> The Trade-offs and Proposed
> > >> structure
> > >> > > > > sections
> > >> > > > > > in
> > >> > > > > > >> > the
> > >> > > > > > >> > > > doc
> > >> > > > > > >> > > > > > were
> > >> > > > > > >> > > > > > > > not
> > >> > > > > > >> > > > > > > > > > > > >> reviewed
> > >> > > > > > >> > > > > > > > > > > > >> in detail. They remain open for
> > >> either
> > >> > > the
> > >> > > > > next
> > >> > > > > > >> sync
> > >> > > > > > >> > > or
> > >> > > > > > >> > > > PR
> > >> > > > > > >> > > > > > > > review.
> > >> > > > > > >> > > > > > > > > > > > >>
> > >> > > > > > >> > > > > > > > > > > > >> *Cross-cutting alignment —
> > >> > > battery-included
> > >> > > > > > plus
> > >> > > > > > >> > > > > pluggable*
> > >> > > > > > >> > > > > > > > > > > > >> A common philosophy emerged
> from the
> > >> > > > > > discussion.
> > >> > > > > > >> EJ
> > >> > > > > > >> > > > > > summarized
> > >> > > > > > >> > > > > > > > it
> > >> > > > > > >> > > > > > > > > > as:
> > >> > > > > > >> > > > > > > > > > > > >> Polaris should provide a
> > >> > > battery-included UX
> > >> > > > > > for
> > >> > > > > > >> > > > beginners
> > >> > > > > > >> > > > > > and
> > >> > > > > > >> > > > > > > > the
> > >> > > > > > >> > > > > > > > > > > > >> flexibility for advanced users
> to
> > >> swap
> > >> > > the
> > >> > > > > > >> included
> > >> > > > > > >> > > > > battery
> > >> > > > > > >> > > > > > > for
> > >> > > > > > >> > > > > > > > > > > > something
> > >> > > > > > >> > > > > > > > > > > > >> more powerful or tailored to
> their
> > >> use
> > >> > > case.
> > >> > > > > > The
> > >> > > > > > >> SPI
> > >> > > > > > >> > > > > design
> > >> > > > > > >> > > > > > > > needs
> > >> > > > > > >> > > > > > > > > to
> > >> > > > > > >> > > > > > > > > > > > >> enable
> > >> > > > > > >> > > > > > > > > > > > >> both.
> > >> > > > > > >> > > > > > > > > > > > >>
> > >> > > > > > >> > > > > > > > > > > > >> The inputs that shaped this
> framing:
> > >> > > > > > >> > > > > > > > > > > > >> •   Anand described how his team
> > >> uses
> > >> > the
> > >> > > > > > current
> > >> > > > > > >> > > > metrics
> > >> > > > > > >> > > > > > > > > > persistence
> > >> > > > > > >> > > > > > > > > > > > >> (three metrics consumers in
> v1.4).
> > >> > > > > > >> > > > > > > > > > > > >> •   Yufei raised Grafana and
> > >> dashboard
> > >> > > > > > >> integrations
> > >> > > > > > >> > > as a
> > >> > > > > > >> > > > > > > > > destination
> > >> > > > > > >> > > > > > > > > > > use
> > >> > > > > > >> > > > > > > > > > > > >> case beyond the default.
> > >> > > > > > >> > > > > > > > > > > > >> •   Robert called out that the
> > >> current
> > >> > > design
> > >> > > > > > is
> > >> > > > > > >> > more
> > >> > > > > > >> > > > > > > > > JDBC-focused.
> > >> > > > > > >> > > > > > > > > > > > >>
> > >> > > > > > >> > > > > > > > > > > > >> Two concrete instances:
> > >> > > > > > >> > > > > > > > > > > > >> •   Async metrics intake:
> Yufei's
> > >> > initial
> > >> > > > > > >> position
> > >> > > > > > >> > was
> > >> > > > > > >> > > > > that
> > >> > > > > > >> > > > > > > > async
> > >> > > > > > >> > > > > > > > > > > should
> > >> > > > > > >> > > > > > > > > > > > >> largely live on the producer
> side
> > >> and
> > >> > > there
> > >> > > > > is
> > >> > > > > > >> not
> > >> > > > > > >> > > much
> > >> > > > > > >> > > > > > > Polaris
> > >> > > > > > >> > > > > > > > > can
> > >> > > > > > >> > > > > > > > > > > do.
> > >> > > > > > >> > > > > > > > > > > > >> Robert suggested a Polaris-side
> > >> default
> > >> > > is
> > >> > > > > > doable
> > >> > > > > > >> > via
> > >> > > > > > >> > > > > > Vert.x.
> > >> > > > > > >> > > > > > > > > Dmitri
> > >> > > > > > >> > > > > > > > > > > > >> agreed
> > >> > > > > > >> > > > > > > > > > > > >> the direction is worth
> exploring.
> > >> The
> > >> > > meeting
> > >> > > > > > >> > > converged
> > >> > > > > > >> > > > > on a
> > >> > > > > > >> > > > > > > > > > > > >> battery-included default (likely
> > >> > > > > Vert.x-backed)
> > >> > > > > > >> with
> > >> > > > > > >> > > an
> > >> > > > > > >> > > > > SPI
> > >> > > > > > >> > > > > > > > shape
> > >> > > > > > >> > > > > > > > > > that
> > >> > > > > > >> > > > > > > > > > > > >> lets
> > >> > > > > > >> > > > > > > > > > > > >> power users route to a more
> scalable
> > >> > > backend
> > >> > > > > > >> > > (k8s-hosted
> > >> > > > > > >> > > > > > > queue,
> > >> > > > > > >> > > > > > > > > AWS
> > >> > > > > > >> > > > > > > > > > > SQS,
> > >> > > > > > >> > > > > > > > > > > > >> etc.).
> > >> > > > > > >> > > > > > > > > > > > >> •   Pluggable destinations:
> > >> combining
> > >> > > Yufei's
> > >> > > > > > >> > > dashboard
> > >> > > > > > >> > > > > use
> > >> > > > > > >> > > > > > > case
> > >> > > > > > >> > > > > > > > > > with
> > >> > > > > > >> > > > > > > > > > > > >> Robert's JDBC-focused call-out,
> the
> > >> > > meeting
> > >> > > > > > >> agreed
> > >> > > > > > >> > the
> > >> > > > > > >> > > > SPI
> > >> > > > > > >> > > > > > > > should
> > >> > > > > > >> > > > > > > > > be
> > >> > > > > > >> > > > > > > > > > > > >> structured for multiple sinks so
> > >> > > integrations
> > >> > > > > > >> become
> > >> > > > > > >> > > > impl
> > >> > > > > > >> > > > > > > > choices
> > >> > > > > > >> > > > > > > > > > > rather
> > >> > > > > > >> > > > > > > > > > > > >> than architectural changes.
> > >> > > > > > >> > > > > > > > > > > > >>
> > >> > > > > > >> > > > > > > > > > > > >> The battery-included default is
> most
> > >> > > likely
> > >> > > > > to
> > >> > > > > > >> use
> > >> > > > > > >> > the
> > >> > > > > > >> > > > > > > existing
> > >> > > > > > >> > > > > > > > > > > > >> JDBC-backed
> > >> > > > > > >> > > > > > > > > > > > >> approach.
> > >> > > > > > >> > > > > > > > > > > > >>
> > >> > > > > > >> > > > > > > > > > > > >> *Direction (idea-level
> alignment)*
> > >> > > > > > >> > > > > > > > > > > > >> •   Single metrics_report table
> > >> > > consolidating
> > >> > > > > > >> > per-type
> > >> > > > > > >> > > > > > > metrics,
> > >> > > > > > >> > > > > > > > > > > > replacing
> > >> > > > > > >> > > > > > > > > > > > >> scan_metrics_report and
> > >> > > commit_metrics_report
> > >> > > > > > >> > > > > > > > > > > > >> •   Iceberg-focused naming and
> > >> schema
> > >> > for
> > >> > > > > now,
> > >> > > > > > >> > revisit
> > >> > > > > > >> > > > if
> > >> > > > > > >> > > > > > > > > > > generic-table
> > >> > > > > > >> > > > > > > > > > > > or
> > >> > > > > > >> > > > > > > > > > > > >> operational metrics arrive
> > >> > > > > > >> > > > > > > > > > > > >> •   Metrics persistence as a
> > >> separated
> > >> > > SPI,
> > >> > > > > not
> > >> > > > > > >> on
> > >> > > > > > >> > > > > > > > BasePersistence
> > >> > > > > > >> > > > > > > > > > > > >> •   Bootstrap path separated for
> > >> > metrics,
> > >> > > > > > >> > independent
> > >> > > > > > >> > > of
> > >> > > > > > >> > > > > > > > metastore
> > >> > > > > > >> > > > > > > > > > > > >> bootstrap
> > >> > > > > > >> > > > > > > > > > > > >> •   "Battery-included plus
> > >> pluggable"
> > >> > as
> > >> > > the
> > >> > > > > > SPI
> > >> > > > > > >> > > design
> > >> > > > > > >> > > > > > > > philosophy
> > >> > > > > > >> > > > > > > > > > > > >>
> > >> > > > > > >> > > > > > > > > > > > >> *Open items*
> > >> > > > > > >> > > > > > > > > > > > >> •   Schema details:
> > >> > > metric_schema_version,
> > >> > > > > > >> payload
> > >> > > > > > >> > > > format,
> > >> > > > > > >> > > > > > IRC
> > >> > > > > > >> > > > > > > > v2
> > >> > > > > > >> > > > > > > > > > > > >> forward-compat shape
> > >> > > > > > >> > > > > > > > > > > > >> •   SPI design details — full
> review
> > >> > > either
> > >> > > > > in
> > >> > > > > > >> the
> > >> > > > > > >> > > next
> > >> > > > > > >> > > > > sync
> > >> > > > > > >> > > > > > > or
> > >> > > > > > >> > > > > > > > in
> > >> > > > > > >> > > > > > > > > > the
> > >> > > > > > >> > > > > > > > > > > > >> corresponding PR
> > >> > > > > > >> > > > > > > > > > > > >> •   Schema refactor PR ownership
> > >> > > > > > >> > > > > > > > > > > > >>
> > >> > > > > > >> > > > > > > > > > > > >> *Action items*
> > >> > > > > > >> > > > > > > > > > > > >> •   EJ to take a first stab at
> the
> > >> SPI
> > >> > > design
> > >> > > > > > and
> > >> > > > > > >> > > > > > potentially
> > >> > > > > > >> > > > > > > > > > partner
> > >> > > > > > >> > > > > > > > > > > > with
> > >> > > > > > >> > > > > > > > > > > > >> Anand to incorporate the lessons
> > >> > learned
> > >> > > from
> > >> > > > > > the
> > >> > > > > > >> > > > existing
> > >> > > > > > >> > > > > > > > > reporter
> > >> > > > > > >> > > > > > > > > > > and
> > >> > > > > > >> > > > > > > > > > > > >> persistence work.
> > >> > > > > > >> > > > > > > > > > > > >> •   Schema refactor PR
> ownership is
> > >> not
> > >> > > yet
> > >> > > > > > >> decided.
> > >> > > > > > >> > > If
> > >> > > > > > >> > > > > > anyone
> > >> > > > > > >> > > > > > > > is
> > >> > > > > > >> > > > > > > > > > > > >> interested in driving it, reply
> on
> > >> this
> > >> > > > > thread.
> > >> > > > > > >> > > > > > > > > > > > >> •   JB to schedule the next
> sync,
> > >> > > tentatively
> > >> > > > > > in
> > >> > > > > > >> two
> > >> > > > > > >> > > > > weeks.
> > >> > > > > > >> > > > > > > > > > > > >>
> > >> > > > > > >> > > > > > > > > > > > >> -ej
> > >> > > > > > >> > > > > > > > > > > > >>
> > >> > > > > > >> > > > > > > > > > > > >> On Mon, Apr 27, 2026 at 3:07 PM
> EJ
> > >> > Wang <
> > >> > > > > > >> > > > > > > > > > > ej.wang.dev.interest@gmail.com
> > >> > > > > > >> > > > > > > > > > > > >
> > >> > > > > > >> > > > > > > > > > > > >> wrote:
> > >> > > > > > >> > > > > > > > > > > > >>
> > >> > > > > > >> > > > > > > > > > > > >> > Thanks Yufei for the +1.
> > >> > > > > > >> > > > > > > > > > > > >> >
> > >> > > > > > >> > > > > > > > > > > > >> > JB, could you help add a
> biweekly
> > >> > > metrics
> > >> > > > > > >> > > architecture
> > >> > > > > > >> > > > > > sync
> > >> > > > > > >> > > > > > > to
> > >> > > > > > >> > > > > > > > > the
> > >> > > > > > >> > > > > > > > > > > > >> Polaris
> > >> > > > > > >> > > > > > > > > > > > >> > community calendar? I'm
> thinking
> > >> > > Thursdays
> > >> > > > > at
> > >> > > > > > >> > 9-10am
> > >> > > > > > >> > > > PT,
> > >> > > > > > >> > > > > > on
> > >> > > > > > >> > > > > > > > the
> > >> > > > > > >> > > > > > > > > > > > >> off-weeks
> > >> > > > > > >> > > > > > > > > > > > >> > from the community meeting
> > >> (starting
> > >> > > May
> > >> > > > > 7),
> > >> > > > > > 60
> > >> > > > > > >> > > > minutes.
> > >> > > > > > >> > > > > > > > > > > > >> >
> > >> > > > > > >> > > > > > > > > > > > >> > Here's a rough agenda to work
> > >> through
> > >> > > over
> > >> > > > > > the
> > >> > > > > > >> > first
> > >> > > > > > >> > > > few
> > >> > > > > > >> > > > > > > > > sessions,
> > >> > > > > > >> > > > > > > > > > > > >> grouped
> > >> > > > > > >> > > > > > > > > > > > >> > by priority:
> > >> > > > > > >> > > > > > > > > > > > >> >
> > >> > > > > > >> > > > > > > > > > > > >> > *First: foundational
> direction*
> > >> > > > > > >> > > > > > > > > > > > >> >
> > >> > > > > > >> > > > > > > > > > > > >> > 1.  MetricsPersistence: public
> > >> SPI or
> > >> > > > > > internal
> > >> > > > > > >> > > > > > > implementation
> > >> > > > > > >> > > > > > > > > > > detail?
> > >> > > > > > >> > > > > > > > > > > > >> >    •   Marked @Beta, javadoc
> calls
> > >> > it a
> > >> > > > > > >> "Service
> > >> > > > > > >> > > > > Provider
> > >> > > > > > >> > > > > > > > > > > Interface",
> > >> > > > > > >> > > > > > > > > > > > >> but
> > >> > > > > > >> > > > > > > > > > > > >> > only one consumer
> > >> > > > > (JdbcBasePersistenceImpl),
> > >> > > > > > >> lives
> > >> > > > > > >> > > on
> > >> > > > > > >> > > > > > > > > > > BasePersistence.
> > >> > > > > > >> > > > > > > > > > > > >> If
> > >> > > > > > >> > > > > > > > > > > > >> > demoted to a private helper
> > >> inside a
> > >> > > > > > persisting
> > >> > > > > > >> > > > reporter
> > >> > > > > > >> > > > > > > impl,
> > >> > > > > > >> > > > > > > > > > most
> > >> > > > > > >> > > > > > > > > > > > >> > downstream design decisions
> become
> > >> > > > > > >> implementation
> > >> > > > > > >> > > > > details
> > >> > > > > > >> > > > > > > > rather
> > >> > > > > > >> > > > > > > > > > > than
> > >> > > > > > >> > > > > > > > > > > > >> > contract questions.
> > >> > > > > > >> > > > > > > > > > > > >> >
> > >> > > > > > >> > > > > > > > > > > > >> > 2.  Persistence schema
> redesign
> > >> > > > > > >> > > > > > > > > > > > >> >    •   Current two-table
> layout
> > >> > > > > > >> > > (scan_metrics_report,
> > >> > > > > > >> > > > > > > > > > > > >> > commit_metrics_report) with
> ~25
> > >> > > flattened
> > >> > > > > > >> columns
> > >> > > > > > >> > > > each.
> > >> > > > > > >> > > > > > > Every
> > >> > > > > > >> > > > > > > > > new
> > >> > > > > > >> > > > > > > > > > > > metric
> > >> > > > > > >> > > > > > > > > > > > >> > type requires a new table, SPI
> > >> > method,
> > >> > > > > record
> > >> > > > > > >> > class,
> > >> > > > > > >> > > > > > model,
> > >> > > > > > >> > > > > > > > > > > converter,
> > >> > > > > > >> > > > > > > > > > > > >> and
> > >> > > > > > >> > > > > > > > > > > > >> > schema migration. Direction to
> > >> > explore:
> > >> > > > > > single
> > >> > > > > > >> > table
> > >> > > > > > >> > > > > with
> > >> > > > > > >> > > > > > > > > > > metric_type
> > >> > > > > > >> > > > > > > > > > > > >> enum,
> > >> > > > > > >> > > > > > > > > > > > >> > schema_version, and JSON
> payload
> > >> > > column.
> > >> > > > > > >> > > > > > > > > > > > >> >
> > >> > > > > > >> > > > > > > > > > > > >> > *Second: design details once
> > >> > direction
> > >> > > is
> > >> > > > > > set*
> > >> > > > > > >> > > > > > > > > > > > >> >
> > >> > > > > > >> > > > > > > > > > > > >> > 3.  Partition key strategy
> > >> > > > > > >> > > > > > > > > > > > >> >    •   Single-table design
> means
> > >> scan
> > >> > > > > metrics
> > >> > > > > > >> at
> > >> > > > > > >> > > scale
> > >> > > > > > >> > > > > > will
> > >> > > > > > >> > > > > > > > have
> > >> > > > > > >> > > > > > > > > > > high
> > >> > > > > > >> > > > > > > > > > > > >> > write concurrency per table.
> > >> Schema
> > >> > > needs
> > >> > > > > to
> > >> > > > > > >> > expose
> > >> > > > > > >> > > > > enough
> > >> > > > > > >> > > > > > > > > > structure
> > >> > > > > > >> > > > > > > > > > > > for
> > >> > > > > > >> > > > > > > > > > > > >> > backends to shard by entity or
> > >> time
> > >> > > range.
> > >> > > > > > >> > > > > > > > > > > > >> >
> > >> > > > > > >> > > > > > > > > > > > >> > 4.  Read/write path
> consistency
> > >> > > > > > >> > > > > > > > > > > > >> >    •   Writes go through
> > >> > > > > > PolarisMetricsManager
> > >> > > > > > >> on
> > >> > > > > > >> > > > > > > > > > MetaStoreManager.
> > >> > > > > > >> > > > > > > > > > > > >> Reads
> > >> > > > > > >> > > > > > > > > > > > >> > bypass MetaStoreManager and go
> > >> > > straight to
> > >> > > > > > >> > > > > > BasePersistence,
> > >> > > > > > >> > > > > > > > > > > excluding
> > >> > > > > > >> > > > > > > > > > > > >> > non-JDBC backends from the
> read
> > >> API.
> > >> > > > > > >> > > > > > > > > > > > >> >
> > >> > > > > > >> > > > > > > > > > > > >> > *Third: cleanup and alignment*
> > >> > > > > > >> > > > > > > > > > > > >> >
> > >> > > > > > >> > > > > > > > > > > > >> > 5.  PolarisMetricsReporter
> naming
> > >> > > > > > >> > > > > > > > > > > > >> >    •   Only handles IRC
> > >> > > > > > >> (ScanReport/CommitReport),
> > >> > > > > > >> > > > > doesn't
> > >> > > > > > >> > > > > > > > cover
> > >> > > > > > >> > > > > > > > > > > > generic
> > >> > > > > > >> > > > > > > > > > > > >> > tables or operational metrics.
> > >> Name
> > >> > is
> > >> > > > > > broader
> > >> > > > > > >> > than
> > >> > > > > > >> > > > > scope.
> > >> > > > > > >> > > > > > > > > > > > >> >
> > >> > > > > > >> > > > > > > > > > > > >> > 6.  PolarisMetricsManager
> facade
> > >> > > > > passthrough
> > >> > > > > > >> > > > > > > > > > > > >> >    •   Entire default method
> is
> > >> > > > > > >> > > > > > > > > > > > >>
> > >> > callCtx.getMetaStore().writeScanReport().
> > >> > > > > > >> > > > > > > > > > > > >> > Zero logic, passes Level 1
> > >> straight
> > >> > > through
> > >> > > > > > to
> > >> > > > > > >> > Level
> > >> > > > > > >> > > > 3.
> > >> > > > > > >> > > > > > Same
> > >> > > > > > >> > > > > > > > > > > > >> anti-pattern
> > >> > > > > > >> > > > > > > > > > > > >> > as PolarisEventManager.
> > >> > > > > > >> > > > > > > > > > > > >> >
> > >> > > > > > >> > > > > > > > > > > > >> > 7.  Iceberg community
> alignment
> > >> > > > > > >> > > > > > > > > > > > >> >    •   Payload-type extension
> > >> needs
> > >> > > > > > discussion
> > >> > > > > > >> on
> > >> > > > > > >> > > > > > > dev@iceberg.
> > >> > > > > > >> > > > > > > > > > > > >> obelix74's
> > >> > > > > > >> > > > > > > > > > > > >> > Feb thread got zero replies.
> > >> Needs a
> > >> > > > > > committer
> > >> > > > > > >> > > voice.
> > >> > > > > > >> > > > > > > > > > > > >> >
> > >> > > > > > >> > > > > > > > > > > > >> > Lets confirm prioritization
> in the
> > >> > > first
> > >> > > > > > >> session.
> > >> > > > > > >> > > > > > > > > > > > >> >
> > >> > > > > > >> > > > > > > > > > > > >> > -ej
> > >> > > > > > >> > > > > > > > > > > > >> >
> > >> > > > > > >> > > > > > > > > > > > >> > On Tue, Apr 21, 2026 at
> 3:18 PM
> > >> Yufei
> > >> > > Gu <
> > >> > > > > > >> > > > > > > > flyrain000@gmail.com>
> > >> > > > > > >> > > > > > > > > > > > wrote:
> > >> > > > > > >> > > > > > > > > > > > >> >
> > >> > > > > > >> > > > > > > > > > > > >> >> Thanks everyone for
> continuing to
> > >> > > drive
> > >> > > > > this
> > >> > > > > > >> > > > forward. I
> > >> > > > > > >> > > > > > > agree
> > >> > > > > > >> > > > > > > > > > that
> > >> > > > > > >> > > > > > > > > > > > the
> > >> > > > > > >> > > > > > > > > > > > >> >> problem is getting complex
> enough
> > >> > > that a
> > >> > > > > > more
> > >> > > > > > >> > > > > structured
> > >> > > > > > >> > > > > > > > > > discussion
> > >> > > > > > >> > > > > > > > > > > > >> would
> > >> > > > > > >> > > > > > > > > > > > >> >> help.
> > >> > > > > > >> > > > > > > > > > > > >> >>
> > >> > > > > > >> > > > > > > > > > > > >> >> +1 on setting up a biweekly
> sync
> > >> for
> > >> > > the
> > >> > > > > > >> metrics
> > >> > > > > > >> > > > > > > > architecture.
> > >> > > > > > >> > > > > > > > > > I’m
> > >> > > > > > >> > > > > > > > > > > > >> happy
> > >> > > > > > >> > > > > > > > > > > > >> >> to
> > >> > > > > > >> > > > > > > > > > > > >> >> join.
> > >> > > > > > >> > > > > > > > > > > > >> >>
> > >> > > > > > >> > > > > > > > > > > > >> >> Yufei
> > >> > > > > > >> > > > > > > > > > > > >> >>
> > >> > > > > > >> > > > > > > > > > > > >> >>
> > >> > > > > > >> > > > > > > > > > > > >> >> On Tue, Apr 21, 2026 at
> 2:34 PM
> > >> EJ
> > >> > > Wang <
> > >> > > > > > >> > > > > > > > > > > > >> ej.wang.dev.interest@gmail.com>
> > >> > > > > > >> > > > > > > > > > > > >> >> wrote:
> > >> > > > > > >> > > > > > > > > > > > >> >>
> > >> > > > > > >> > > > > > > > > > > > >> >> > Also, I've been looking
> more
> > >> > > closely at
> > >> > > > > > the
> > >> > > > > > >> > > > > > *persistence
> > >> > > > > > >> > > > > > > > > schema
> > >> > > > > > >> > > > > > > > > > > in
> > >> > > > > > >> > > > > > > > > > > > >> the
> > >> > > > > > >> > > > > > > > > > > > >> >> > current metrics work*, and
> I
> > >> think
> > >> > > > > > there's a
> > >> > > > > > >> > > > > structural
> > >> > > > > > >> > > > > > > > > > rigidity
> > >> > > > > > >> > > > > > > > > > > > >> problem
> > >> > > > > > >> > > > > > > > > > > > >> >> > worth raising before the
> shape
> > >> > gets
> > >> > > > > locked
> > >> > > > > > >> in.
> > >> > > > > > >> > > > > > > > > > > > >> >> >
> > >> > > > > > >> > > > > > > > > > > > >> >> > Right now we have two
> separate
> > >> > > tables
> > >> > > > > > >> > > > > > > (scan_metrics_report
> > >> > > > > > >> > > > > > > > > and
> > >> > > > > > >> > > > > > > > > > > > >> >> > commit_metrics_report),
> each
> > >> with
> > >> > > ~25
> > >> > > > > > >> flattened
> > >> > > > > > >> > > > > columns
> > >> > > > > > >> > > > > > > > that
> > >> > > > > > >> > > > > > > > > > > > directly
> > >> > > > > > >> > > > > > > > > > > > >> >> > mirror the Iceberg report
> > >> fields.
> > >> > > The
> > >> > > > > SPI
> > >> > > > > > >> > follows
> > >> > > > > > >> > > > the
> > >> > > > > > >> > > > > > > same
> > >> > > > > > >> > > > > > > > > > split:
> > >> > > > > > >> > > > > > > > > > > > >> >> > writeScanReport and
> > >> > > writeCommitReport as
> > >> > > > > > >> > separate
> > >> > > > > > >> > > > > > > methods,
> > >> > > > > > >> > > > > > > > > with
> > >> > > > > > >> > > > > > > > > > > > >> per-type
> > >> > > > > > >> > > > > > > > > > > > >> >> > record classes,
> converters, and
> > >> > > model
> > >> > > > > > >> objects.
> > >> > > > > > >> > > *The
> > >> > > > > > >> > > > > > > > practical
> > >> > > > > > >> > > > > > > > > > > cost:
> > >> > > > > > >> > > > > > > > > > > > >> >> > adding a new metric type
> > >> > > (operational
> > >> > > > > > >> metrics,
> > >> > > > > > >> > > for
> > >> > > > > > >> > > > > > > example)
> > >> > > > > > >> > > > > > > > > > > > requires
> > >> > > > > > >> > > > > > > > > > > > >> a
> > >> > > > > > >> > > > > > > > > > > > >> >> new
> > >> > > > > > >> > > > > > > > > > > > >> >> > table, a new SPI method, a
> new
> > >> > > record
> > >> > > > > > >> class, a
> > >> > > > > > >> > > new
> > >> > > > > > >> > > > > > model
> > >> > > > > > >> > > > > > > > > > class, a
> > >> > > > > > >> > > > > > > > > > > > new
> > >> > > > > > >> > > > > > > > > > > > >> >> > converter branch, and a
> schema
> > >> > > > > migration*.
> > >> > > > > > >> > > That's a
> > >> > > > > > >> > > > > lot
> > >> > > > > > >> > > > > > > of
> > >> > > > > > >> > > > > > > > > > > surface
> > >> > > > > > >> > > > > > > > > > > > >> area
> > >> > > > > > >> > > > > > > > > > > > >> >> > for what should be "one
> more
> > >> kind
> > >> > of
> > >> > > > > > >> metric."
> > >> > > > > > >> > > > > > > > > > > > >> >> >
> > >> > > > > > >> > > > > > > > > > > > >> >> > *My bias* would be toward a
> > >> single
> > >> > > > > metrics
> > >> > > > > > >> > table
> > >> > > > > > >> > > > with
> > >> > > > > > >> > > > > > *a
> > >> > > > > > >> > > > > > > > > typed
> > >> > > > > > >> > > > > > > > > > > JSON
> > >> > > > > > >> > > > > > > > > > > > >> >> > payload*. Something like:
> > >> > > metric_type
> > >> > > > > > >> (enum),
> > >> > > > > > >> > > > > > entity_id,
> > >> > > > > > >> > > > > > > > > > > > >> >> > table_identifier,
> snapshot_id
> > >> > > > > (nullable),
> > >> > > > > > >> > > > > received_ts,
> > >> > > > > > >> > > > > > > > > > > > >> schema_version,
> > >> > > > > > >> > > > > > > > > > > > >> >> and
> > >> > > > > > >> > > > > > > > > > > > >> >> > a payload column for the
> > >> > > metric-specific
> > >> > > > > > >> data.
> > >> > > > > > >> > > The
> > >> > > > > > >> > > > > > > > > metric_type
> > >> > > > > > >> > > > > > > > > > +
> > >> > > > > > >> > > > > > > > > > > > >> >> > schema_version pair gives
> us a
> > >> > > > > > >> > forward-compatible
> > >> > > > > > >> > > > > > > contract
> > >> > > > > > >> > > > > > > > > for
> > >> > > > > > >> > > > > > > > > > > the
> > >> > > > > > >> > > > > > > > > > > > >> >> payload
> > >> > > > > > >> > > > > > > > > > > > >> >> > shape. Adding a new metric
> type
> > >> > > becomes
> > >> > > > > an
> > >> > > > > > >> enum
> > >> > > > > > >> > > > value
> > >> > > > > > >> > > > > > > and a
> > >> > > > > > >> > > > > > > > > > > payload
> > >> > > > > > >> > > > > > > > > > > > >> >> schema,
> > >> > > > > > >> > > > > > > > > > > > >> >> > not a schema migration. One
> > >> thing
> > >> > I
> > >> > > > > think
> > >> > > > > > we
> > >> > > > > > >> > need
> > >> > > > > > >> > > > to
> > >> > > > > > >> > > > > be
> > >> > > > > > >> > > > > > > > > > > deliberate
> > >> > > > > > >> > > > > > > > > > > > >> >> about is
> > >> > > > > > >> > > > > > > > > > > > >> >> > the partition key design.
> If
> > >> all
> > >> > > metric
> > >> > > > > > >> types
> > >> > > > > > >> > > land
> > >> > > > > > >> > > > in
> > >> > > > > > >> > > > > > one
> > >> > > > > > >> > > > > > > > > > table,
> > >> > > > > > >> > > > > > > > > > > > scan
> > >> > > > > > >> > > > > > > > > > > > >> >> > metrics at scale (high
> > >> > concurrency,
> > >> > > high
> > >> > > > > > >> > > frequency
> > >> > > > > > >> > > > > > across
> > >> > > > > > >> > > > > > > > > many
> > >> > > > > > >> > > > > > > > > > > > >> tables)
> > >> > > > > > >> > > > > > > > > > > > >> >> > could easily create hot
> > >> > partitions.
> > >> > > We'd
> > >> > > > > > >> want
> > >> > > > > > >> > the
> > >> > > > > > >> > > > > > > > persistence
> > >> > > > > > >> > > > > > > > > > > layer
> > >> > > > > > >> > > > > > > > > > > > >> to
> > >> > > > > > >> > > > > > > > > > > > >> >> be
> > >> > > > > > >> > > > > > > > > > > > >> >> > able to shard by entity or
> time
> > >> > > range,
> > >> > > > > and
> > >> > > > > > >> that
> > >> > > > > > >> > > > means
> > >> > > > > > >> > > > > > the
> > >> > > > > > >> > > > > > > > > > logical
> > >> > > > > > >> > > > > > > > > > > > >> schema
> > >> > > > > > >> > > > > > > > > > > > >> >> > needs to expose enough
> > >> structure
> > >> > for
> > >> > > > > > >> backends
> > >> > > > > > >> > to
> > >> > > > > > >> > > > > > > partition
> > >> > > > > > >> > > > > > > > > on.
> > >> > > > > > >> > > > > > > > > > I
> > >> > > > > > >> > > > > > > > > > > > >> don't
> > >> > > > > > >> > > > > > > > > > > > >> >> > think the current flattened
> > >> layout
> > >> > > gives
> > >> > > > > > us
> > >> > > > > > >> > that.
> > >> > > > > > >> > > > > > > > > > > > >> >> >
> > >> > > > > > >> > > > > > > > > > > > >> >> > This is getting complex
> enough
> > >> > that
> > >> > > I
> > >> > > > > > don't
> > >> > > > > > >> > think
> > >> > > > > > >> > > > > > ad-hoc
> > >> > > > > > >> > > > > > > > > PR/ML
> > >> > > > > > >> > > > > > > > > > > > >> threads
> > >> > > > > > >> > > > > > > > > > > > >> >> > will converge well. *Would
> > >> people
> > >> > be
> > >> > > > > open
> > >> > > > > > >> to a
> > >> > > > > > >> > > > > biweekly
> > >> > > > > > >> > > > > > > > sync
> > >> > > > > > >> > > > > > > > > > for
> > >> > > > > > >> > > > > > > > > > > > >> metrics
> > >> > > > > > >> > > > > > > > > > > > >> >> > architecture?* I think 30
> > >> minutes
> > >> > > every
> > >> > > > > > two
> > >> > > > > > >> > weeks
> > >> > > > > > >> > > > > with
> > >> > > > > > >> > > > > > > > > > interested
> > >> > > > > > >> > > > > > > > > > > > >> >> parties
> > >> > > > > > >> > > > > > > > > > > > >> >> > would be enough to work
> through
> > >> > the
> > >> > > > > > schema,
> > >> > > > > > >> SPI
> > >> > > > > > >> > > > > shape,
> > >> > > > > > >> > > > > > > and
> > >> > > > > > >> > > > > > > > > read
> > >> > > > > > >> > > > > > > > > > > API
> > >> > > > > > >> > > > > > > > > > > > >> >> design
> > >> > > > > > >> > > > > > > > > > > > >> >> > together. Happy to help set
> > >> that
> > >> > up.
> > >> > > > > > >> > > > > > > > > > > > >> >> >
> > >> > > > > > >> > > > > > > > > > > > >> >> > -ej
> > >> > > > > > >> > > > > > > > > > > > >> >> >
> > >> > > > > > >> > > > > > > > > > > > >> >> > On Mon, Apr 20, 2026 at
> > >> 2:19 PM EJ
> > >> > > Wang
> > >> > > > > <
> > >> > > > > > >> > > > > > > > > > > > >> ej.wang.dev.interest@gmail.com
> > >> > > > > > >> > > > > > > > > > > > >> >> >
> > >> > > > > > >> > > > > > > > > > > > >> >> > wrote:
> > >> > > > > > >> > > > > > > > > > > > >> >> >
> > >> > > > > > >> > > > > > > > > > > > >> >> >> Reviewed #4115, left a
> > >> comment on
> > >> > > the
> > >> > > > > > code
> > >> > > > > > >> > > > > > organization
> > >> > > > > > >> > > > > > > > > side.
> > >> > > > > > >> > > > > > > > > > > > >> >> >>
> > >> > > > > > >> > > > > > > > > > > > >> >> >> One thing stood out: the
> > >> metrics
> > >> > > write
> > >> > > > > > path
> > >> > > > > > >> > > enters
> > >> > > > > > >> > > > > > > through
> > >> > > > > > >> > > > > > > > > > > > >> >> >> PolarisMetricsManager on
> > >> > > > > > MetaStoreManager,
> > >> > > > > > >> but
> > >> > > > > > >> > > the
> > >> > > > > > >> > > > > new
> > >> > > > > > >> > > > > > > > read
> > >> > > > > > >> > > > > > > > > > path
> > >> > > > > > >> > > > > > > > > > > > >> >> bypasses
> > >> > > > > > >> > > > > > > > > > > > >> >> >> MetaStoreManager entirely
> and
> > >> > goes
> > >> > > > > > >> straight to
> > >> > > > > > >> > > > > > > > > BasePersistence
> > >> > > > > > >> > > > > > > > > > > via
> > >> > > > > > >> > > > > > > > > > > > >> >> >>
> callContext.getMetaStore().
> > >> That
> > >> > > means
> > >> > > > > > the
> > >> > > > > > >> > read
> > >> > > > > > >> > > > API
> > >> > > > > > >> > > > > > only
> > >> > > > > > >> > > > > > > > > works
> > >> > > > > > >> > > > > > > > > > > for
> > >> > > > > > >> > > > > > > > > > > > >> >> backends
> > >> > > > > > >> > > > > > > > > > > > >> >> >> that implement
> > >> BasePersistence.
> > >> > > NoSQL
> > >> > > > > and
> > >> > > > > > >> > remote
> > >> > > > > > >> > > > > > > backends
> > >> > > > > > >> > > > > > > > > > can't
> > >> > > > > > >> > > > > > > > > > > > >> >> participate.
> > >> > > > > > >> > > > > > > > > > > > >> >> >>
> > >> > > > > > >> > > > > > > > > > > > >> >> >> Stepping back, I think the
> > >> > metrics
> > >> > > > > > >> subsystem
> > >> > > > > > >> > is
> > >> > > > > > >> > > > > > growing
> > >> > > > > > >> > > > > > > > into
> > >> > > > > > >> > > > > > > > > > > > >> something
> > >> > > > > > >> > > > > > > > > > > > >> >> >> real (write + read + REST
> API
> > >> +
> > >> > > AuthZ +
> > >> > > > > > >> > > > pagination)
> > >> > > > > > >> > > > > > *but
> > >> > > > > > >> > > > > > > > the
> > >> > > > > > >> > > > > > > > > > > > >> >> persistence
> > >> > > > > > >> > > > > > > > > > > > >> >> >> side is split across two
> > >> layers
> > >> > in
> > >> > > a
> > >> > > > > way
> > >> > > > > > >> > that's
> > >> > > > > > >> > > > hard
> > >> > > > > > >> > > > > > to
> > >> > > > > > >> > > > > > > > > > > extend*. I
> > >> > > > > > >> > > > > > > > > > > > >> put
> > >> > > > > > >> > > > > > > > > > > > >> >> >> together two diagrams to
> show
> > >> > what
> > >> > > I
> > >> > > > > mean
> > >> > > > > > >> (my
> > >> > > > > > >> > > best
> > >> > > > > > >> > > > > > > > effort).
> > >> > > > > > >> > > > > > > > > > > > >> >> >>
> > >> > > > > > >> > > > > > > > > > > > >> >> >> *Current state* (Diagram
> 1):
> > >> > three
> > >> > > > > > >> interfaces
> > >> > > > > > >> > at
> > >> > > > > > >> > > > > three
> > >> > > > > > >> > > > > > > > > > different
> > >> > > > > > >> > > > > > > > > > > > >> >> levels.
> > >> > > > > > >> > > > > > > > > > > > >> >> >> The engine-facing SPI
> > >> > > > > > >> (PolarisMetricsReporter)
> > >> > > > > > >> > > is
> > >> > > > > > >> > > > > > clean.
> > >> > > > > > >> > > > > > > > But
> > >> > > > > > >> > > > > > > > > > > > >> >> >> PolarisMetricsManager on
> > >> > > > > MetaStoreManager
> > >> > > > > > >> is a
> > >> > > > > > >> > > > > > > passthrough
> > >> > > > > > >> > > > > > > > > to
> > >> > > > > > >> > > > > > > > > > > > >> >> >> MetricsPersistence on
> > >> > > BasePersistence.
> > >> > > > > > The
> > >> > > > > > >> > @Beta
> > >> > > > > > >> > > > > > > > annotation
> > >> > > > > > >> > > > > > > > > > and
> > >> > > > > > >> > > > > > > > > > > > SPI
> > >> > > > > > >> > > > > > > > > > > > >> >> javadoc
> > >> > > > > > >> > > > > > > > > > > > >> >> >> are on the BasePersistence
> > >> layer,
> > >> > > while
> > >> > > > > > the
> > >> > > > > > >> > > actual
> > >> > > > > > >> > > > > > > > extension
> > >> > > > > > >> > > > > > > > > > > > points
> > >> > > > > > >> > > > > > > > > > > > >> >> >> (PolarisMetricsReporter,
> > >> > > > > > >> > PolarisMetricsManager)
> > >> > > > > > >> > > > > carry
> > >> > > > > > >> > > > > > no
> > >> > > > > > >> > > > > > > > > > > stability
> > >> > > > > > >> > > > > > > > > > > > >> >> >> annotation. The write path
> > >> goes
> > >> > > through
> > >> > > > > > the
> > >> > > > > > >> > > > > > > > MetaStoreManager
> > >> > > > > > >> > > > > > > > > > > > layer,
> > >> > > > > > >> > > > > > > > > > > > >> the
> > >> > > > > > >> > > > > > > > > > > > >> >> >> read path doesn't.
> > >> > > > > > >> > > > > > > > > > > > >> >> >>
> > >> > > > > > >> > > > > > > > > > > > >> >> >> *What I envision*
> (Diagram 2):
> > >> > two
> > >> > > SPIs
> > >> > > > > > at
> > >> > > > > > >> two
> > >> > > > > > >> > > > > levels.
> > >> > > > > > >> > > > > > > > > > > > >> >> >> PolarisMetricsReporter
> stays
> > >> as
> > >> > the
> > >> > > > > > >> > > engine-facing
> > >> > > > > > >> > > > > SPI.
> > >> > > > > > >> > > > > > > > > > > > >> >> >> PolarisMetricsManager
> becomes
> > >> the
> > >> > > > > > >> > backend-facing
> > >> > > > > > >> > > > SPI
> > >> > > > > > >> > > > > > > with
> > >> > > > > > >> > > > > > > > > both
> > >> > > > > > >> > > > > > > > > > > > write
> > >> > > > > > >> > > > > > > > > > > > >> >> and
> > >> > > > > > >> > > > > > > > > > > > >> >> >> read methods at the
> > >> > > MetaStoreManager
> > >> > > > > > level,
> > >> > > > > > >> > > where
> > >> > > > > > >> > > > > any
> > >> > > > > > >> > > > > > > > > backend
> > >> > > > > > >> > > > > > > > > > > > (JDBC,
> > >> > > > > > >> > > > > > > > > > > > >> >> NoSQL,
> > >> > > > > > >> > > > > > > > > > > > >> >> >> remote) can implement
> them.
> > >> > > > > > >> MetricsPersistence
> > >> > > > > > >> > > on
> > >> > > > > > >> > > > > > > > > > > BasePersistence
> > >> > > > > > >> > > > > > > > > > > > >> goes
> > >> > > > > > >> > > > > > > > > > > > >> >> >> away. Where metrics
> actually
> > >> land
> > >> > > is an
> > >> > > > > > >> > > > > implementation
> > >> > > > > > >> > > > > > > > > detail,
> > >> > > > > > >> > > > > > > > > > > > not a
> > >> > > > > > >> > > > > > > > > > > > >> >> core
> > >> > > > > > >> > > > > > > > > > > > >> >> >> interface.
> > >> > > > > > >> > > > > > > > > > > > >> >> >>
> > >> > > > > > >> > > > > > > > > > > > >> >> >> *Minor naming thing*:
> > >> > > > > > >> PolarisMetricsReporter
> > >> > > > > > >> > is
> > >> > > > > > >> > > > > > broader
> > >> > > > > > >> > > > > > > > than
> > >> > > > > > >> > > > > > > > > > > what
> > >> > > > > > >> > > > > > > > > > > > it
> > >> > > > > > >> > > > > > > > > > > > >> >> >> actually handles. It only
> > >> accepts
> > >> > > > > Iceberg
> > >> > > > > > >> REST
> > >> > > > > > >> > > > > Catalog
> > >> > > > > > >> > > > > > > > > metrics
> > >> > > > > > >> > > > > > > > > > > > >> >> (ScanReport,
> > >> > > > > > >> > > > > > > > > > > > >> >> >> CommitReport via
> > >> MetricsReport).
> > >> > > > > Generic
> > >> > > > > > >> table
> > >> > > > > > >> > > > > metrics
> > >> > > > > > >> > > > > > > or
> > >> > > > > > >> > > > > > > > > > > > >> operational
> > >> > > > > > >> > > > > > > > > > > > >> >> >> metrics aren't in scope.
> Not
> > >> > > blocking,
> > >> > > > > > but
> > >> > > > > > >> > worth
> > >> > > > > > >> > > > > > noting
> > >> > > > > > >> > > > > > > if
> > >> > > > > > >> > > > > > > > > the
> > >> > > > > > >> > > > > > > > > > > > >> metrics
> > >> > > > > > >> > > > > > > > > > > > >> >> >> surface expands.
> > >> > > > > > >> > > > > > > > > > > > >> >> >>
> > >> > > > > > >> > > > > > > > > > > > >> >> >> *Rough sketch of how to
> get
> > >> > there*:
> > >> > > > > > >> > > > > > > > > > > > >> >> >>  1.  Add read methods to
> > >> > > > > > >> PolarisMetricsManager
> > >> > > > > > >> > > > > > > > > > (listScanReports,
> > >> > > > > > >> > > > > > > > > > > > >> >> >> listCommitReports) with
> > >> default
> > >> > > no-op,
> > >> > > > > > >> same as
> > >> > > > > > >> > > the
> > >> > > > > > >> > > > > > > > existing
> > >> > > > > > >> > > > > > > > > > > write
> > >> > > > > > >> > > > > > > > > > > > >> >> methods.
> > >> > > > > > >> > > > > > > > > > > > >> >> >> (Probably make
> > >> > > PolarisMetricsManager
> > >> > > > > more
> > >> > > > > > >> > > explicit
> > >> > > > > > >> > > > > on
> > >> > > > > > >> > > > > > > > being
> > >> > > > > > >> > > > > > > > > > > > Iceberg
> > >> > > > > > >> > > > > > > > > > > > >> >> >> specific like package
> name or
> > >> > class
> > >> > > > > name
> > >> > > > > > >> etc.)
> > >> > > > > > >> > > > > > > > > > > > >> >> >>  2.  Wire
> > >> MetricsReportsService
> > >> > > through
> > >> > > > > > >> > > > > > MetaStoreManager
> > >> > > > > > >> > > > > > > > > > instead
> > >> > > > > > >> > > > > > > > > > > > of
> > >> > > > > > >> > > > > > > > > > > > >> >> >>
> callContext.getMetaStore().
> > >> > > > > > >> > > > > > > > > > > > >> >> >>  3.  Extract metrics
> > >> persistence
> > >> > > from
> > >> > > > > > >> > > > > > > > > JdbcBasePersistenceImpl
> > >> > > > > > >> > > > > > > > > > > into
> > >> > > > > > >> > > > > > > > > > > > >> its
> > >> > > > > > >> > > > > > > > > > > > >> >> >> own class. That file
> carries
> > >> ~7
> > >> > > > > > >> > > responsibilities,
> > >> > > > > > >> > > > > > > metrics
> > >> > > > > > >> > > > > > > > > > being
> > >> > > > > > >> > > > > > > > > > > > one
> > >> > > > > > >> > > > > > > > > > > > >> of
> > >> > > > > > >> > > > > > > > > > > > >> >> them.
> > >> > > > > > >> > > > > > > > > > > > >> >> >>  4.  Remove
> MetricsPersistence
> > >> > from
> > >> > > > > > >> > > > BasePersistence.
> > >> > > > > > >> > > > > > > > > > > > >> >> >>
> > >> > > > > > >> > > > > > > > > > > > >> >> >> *None of this needs to
> happen
> > >> in
> > >> > > #4115.
> > >> > > > > > >> But if
> > >> > > > > > >> > > the
> > >> > > > > > >> > > > > > > > direction
> > >> > > > > > >> > > > > > > > > > > makes
> > >> > > > > > >> > > > > > > > > > > > >> >> sense,
> > >> > > > > > >> > > > > > > > > > > > >> >> >> it would be good to align
> > >> before
> > >> > > the
> > >> > > > > > >> metrics
> > >> > > > > > >> > > > surface
> > >> > > > > > >> > > > > > > grows
> > >> > > > > > >> > > > > > > > > > > > further.
> > >> > > > > > >> > > > > > > > > > > > >> >> Curious
> > >> > > > > > >> > > > > > > > > > > > >> >> >> what others think.*
> > >> > > > > > >> > > > > > > > > > > > >> >> >>
> > >> > > > > > >> > > > > > > > > > > > >> >> >> *My mental model note*:
> Level
> > >> 1
> > >> > > > > > >> > > MetaStoreManager;
> > >> > > > > > >> > > > > > level
> > >> > > > > > >> > > > > > > 2
> > >> > > > > > >> > > > > > > > > > > > >> transactional
> > >> > > > > > >> > > > > > > > > > > > >> >> >> persistence; level 3 base
> > >> > > persistence
> > >> > > > > > >> > > > > > > > > > > > >> >> >>
> > >> > > > > > >> > > > > > > > > > > > >> >> >> Diagram 1
> > >> > > > > > >> > > > > > > > > > > > >> >> >> <
> > >> > > > > > >> > > > > > > > > > > > >> >>
> > >> > > > > > >> > > > > > > > > > > > >>
> > >> > > > > > >> > > > > > > > > > > >
> > >> > > > > > >> > > > > > > > > > >
> > >> > > > > > >> > > > > > > > > >
> > >> > > > > > >> > > > > > > > >
> > >> > > > > > >> > > > > > > >
> > >> > > > > > >> > > > > > >
> > >> > > > > > >> > > > > >
> > >> > > > > > >> > > > >
> > >> > > > > > >> > > >
> > >> > > > > > >> > >
> > >> > > > > > >> >
> > >> > > > > > >>
> > >> > > > > >
> > >> > > > >
> > >> > >
> > >> >
> > >>
> https://urldefense.com/v3/__https://www.plantuml.com/plantuml/uml/bLHDR-Cs4BthLmpIYupw0zbkKQ1r3M-S7Bp8xhhM7WCOb3IM65EaGD9EX2RzxHrHb4CxRelwa4YSDu_lpOVcnZ9jzvM8BBS2uGjQpJC3dtHMSekPtMk44IpsMgEqa5XcCOhCZikQQLP1pR8TAp2n3ILhmZDP20m0fcIvUkAoW2qJXd9z1bpToO9BX3WXu0ucy5rpgGPNm0nW5_epUWtm2Ue3pn3kMOFQmKntGZW0BYtgBSi8k5A2QMwybJNMIbFiGSR9QZc4nUqIvikStF0jHprua5C-amge42aNt3R0f5JaaoivdV2Pkqbx4hee4ymOkBh5BTiB-_uIeGeo8zL8rPsPl4DktdEiK1jkB1NdZCRbrSTecDe_mlHbF0wvBmCkaOH5_S8a_TTTKI6-nmCAkEw4LpxsZ-LbYLKQFKMNOgf_wuM7_bV9gOer5SYMMksBSWXFcbi49KNZXNLicwfe3TETC7gPdPqI7uBcHMb1RSzYq34c6PDUM9mn8HRsUTZEiDBve3NjVZumBj0U7SS37mGO7vcwtiK-_pU7U7L_f-digo9YbhSwIfMRwIITKGXbxdIUTCGF1SeCJxloKsU-3k9ddRbX1eDq1q_fx1JbBGT0glVyXimDuP4TQ5qpCAmnGEj2s_6n5mtn1z-97-63itFQZLPO1Ev2tu_WF7Ju-VPc0Skg5bYXxBhkY1xpD7EM_7fyflSpIsqMgVth5xhVr4eQxWQ8enaSAJQSG16yFSDuJ798rrcXr_3n-lfdk7icQjEBmFujL7AodiP_Y4Z7-YxvtZNs4zMgpNTl6tF8sglyPsmqchrjvQ-m-aP94r-TwCA2Ka8upPJZwtvSpoYCXkYMZU2NXvRMBfq9P3i3Le4VAZUAlUZ_oPKsxPgY0Q_BSKLkyr9bhQhQrJjo_x3TPlIB0DPjnMfcIoYP0QaYw1a0fTKDr8fB6ntNuvmoL1ZGkXa69Njh43zf9GiGxHQrA_jDYWRSzF5--WmTVrN97_Sm8LbLUy_lGBmLanJjFkDlGkRqjA_4tm00__;!!Iz9xO38YGHZK!4rnlVIqBQQQ6xpGe1PMV58edOS_cGrKoqdfhw4qSp_JQCavvEjaxXk0OwGPnxyEcO60AzKQK_XDtp7H3DdkFXcrXuTBISHg$
> > >> > > > > > >> > > > > > > > > > > > >> >> >
> > >> > > > > > >> > > > > > > > > > > > >> >> >> :
> > >> > > > > > >> > > > > > > > > > > > >> >> >>
> > >> > > > > > >> > > > > > > > > > > > >> >> >> [image: image.png]
> > >> > > > > > >> > > > > > > > > > > > >> >> >>
> > >> > > > > > >> > > > > > > > > > > > >> >> >> Diagram 2
> > >> > > > > > >> > > > > > > > > > > > >> >> >> <
> > >> > > > > > >> > > > > > > > > > > > >> >>
> > >> > > > > > >> > > > > > > > > > > > >>
> > >> > > > > > >> > > > > > > > > > > >
> > >> > > > > > >> > > > > > > > > > >
> > >> > > > > > >> > > > > > > > > >
> > >> > > > > > >> > > > > > > > >
> > >> > > > > > >> > > > > > > >
> > >> > > > > > >> > > > > > >
> > >> > > > > > >> > > > > >
> > >> > > > > > >> > > > >
> > >> > > > > > >> > > >
> > >> > > > > > >> > >
> > >> > > > > > >> >
> > >> > > > > > >>
> > >> > > > > >
> > >> > > > >
> > >> > >
> > >> >
> > >>
> https://urldefense.com/v3/__https://www.plantuml.com/plantuml/uml/VLLDR-8m4BtdLupO2sWBLVU8AaGB7AXAbssGzb896SS9RXqxjKqBwkv_tt7iV43fdaZYDpFlpRmnOsE9jhjSH9PRmM31hERKm8scMsuPjJlDe0yheZDc8RR4iYWoBrmMH9CS2a9VICPYUy1OZN0YCy5Q0BCbYNhdCeEK28En8G8wCvbnoQ0R8_05Bc6bkLIz3X03p1zzH7zR-9ZfDquPt9C3qoNCX2yV4G2NbkcKu5jdgGJHt0GbZwnG6i-UP3TUpk5gM6Ldqke350eZUqzoCft3U9xWHvxoa5-7K4nF1J46EbEMafsmdrCBbQ44gVggy18IZrn_ph5asd1ZiIKdQSgueZvjXrQFSFrdC3YN-nXmBacxbGiYyLVxLaBtdhqn0LSzdBDhqQtQoOJeGyad3z0lUqnYgpGB6Ns8oVyta00Dy_WnX0tIOZ8v6SYxHll1TrH6aejAik-mh-AphVFCwSUQqFypElag5QRGFDjQKEd96K1P8QP41c9TzA_IIQyvdAWyv_RSiS3skb0_EzDDkK2v5xWF6MiGFlvhpFLcD2Dq2pml14gaF67eQkmd8gulDoC4kSOu6KVpkvlUJg1RTbWISU40RdBUUS_9XfRZ2dwxm_SW8LYFISgm_MnlDQ6M9P1gbKEc4X-2pH_FvJCkCqm9pbVjD6LrwdLeOrDWfOaqc8Wh9BE85oNKxkNQ6o4yGRy_Eae0G_G8tZv81d3bHDB23WOdisohVr3nh_j6lbSjbNaLRTc8UgtPbAU1J_tygOfZX9DWEJeHDvYx-qmSi5FgNLPZwHrHcUsncGQ5-skhUclpE5fo4ounpFauYrUbkU6ccfnxMvitwag4IyerhTxj8In_Oj1bDO4pQru674loYrGlULHLEGCjwJJ8gDoVZR8MxO4BT3IzRvIcAQKezC6xpziGnTyImrfEGyJI_OcKfgtxIvnTqFEMS17L9Z-jsARN5FmTheP7HtSdtOMT0B4GY2FYHXxgQmMtj2bRqiLFGapiVe1_QVKDrkqXcm83aFEXnMYCZ-xlyHy__;!!Iz9xO38YGHZK!4rnlVIqBQQQ6xpGe1PMV58edOS_cGrKoqdfhw4qSp_JQCavvEjaxXk0OwGPnxyEcO60AzKQK_XDtp7H3DdkFXcrXSpmqcNk$
> > >> > > > > > >> > > > > > > > > > > > >> >> >
> > >> > > > > > >> > > > > > > > > > > > >> >> >> :
> > >> > > > > > >> > > > > > > > > > > > >> >> >> [image: image.png]
> > >> > > > > > >> > > > > > > > > > > > >> >> >>
> > >> > > > > > >> > > > > > > > > > > > >> >> >>  -ej
> > >> > > > > > >> > > > > > > > > > > > >> >> >>
> > >> > > > > > >> > > > > > > > > > > > >> >> >> On Wed, Apr 15, 2026 at
> > >> 8:22 AM
> > >> > > Dmitri
> > >> > > > > > >> > > > Bourlatchkov
> > >> > > > > > >> > > > > <
> > >> > > > > > >> > > > > > > > > > > > >> dimas@apache.org>
> > >> > > > > > >> > > > > > > > > > > > >> >> >> wrote:
> > >> > > > > > >> > > > > > > > > > > > >> >> >>
> > >> > > > > > >> > > > > > > > > > > > >> >> >>> Hi All,
> > >> > > > > > >> > > > > > > > > > > > >> >> >>>
> > >> > > > > > >> > > > > > > > > > > > >> >> >>> Heads up: The current
> state
> > >> of
> > >> > PR
> > >> > > > > [4115]
> > >> > > > > > >> > looks
> > >> > > > > > >> > > > > pretty
> > >> > > > > > >> > > > > > > > solid
> > >> > > > > > >> > > > > > > > > > to
> > >> > > > > > >> > > > > > > > > > > > me.
> > >> > > > > > >> > > > > > > > > > > > >> I
> > >> > > > > > >> > > > > > > > > > > > >> >> >>> believe this PR is
> > >> approaching a
> > >> > > > > > mergeable
> > >> > > > > > >> > > > > condition.
> > >> > > > > > >> > > > > > > > > > > > >> >> >>>
> > >> > > > > > >> > > > > > > > > > > > >> >> >>> Please post your reviews
> if
> > >> you
> > >> > > have
> > >> > > > > any
> > >> > > > > > >> > > > comments.
> > >> > > > > > >> > > > > > > > > > > > >> >> >>>
> > >> > > > > > >> > > > > > > > > > > > >> >> >>> [4115]
> > >> > > > > > >> > > >
> > >> >
> > >>
> https://urldefense.com/v3/__https://github.com/apache/polaris/pull/4115__;!!Iz9xO38YGHZK!4rnlVIqBQQQ6xpGe1PMV58edOS_cGrKoqdfhw4qSp_JQCavvEjaxXk0OwGPnxyEcO60AzKQK_XDtp7H3DdkFXcrXtFRscKs$
> > >> > > > > > >> > > > > > > > > > > > >> >> >>>
> > >> > > > > > >> > > > > > > > > > > > >> >> >>> Thanks,
> > >> > > > > > >> > > > > > > > > > > > >> >> >>> Dmitri.
> > >> > > > > > >> > > > > > > > > > > > >> >> >>>
> > >> > > > > > >> > > > > > > > > > > > >> >> >>> On Tue, Mar 3, 2026 at
> > >> 3:29 PM
> > >> > > Anand
> > >> > > > > > Kumar
> > >> > > > > > >> > > > Sankaran
> > >> > > > > > >> > > > > > via
> > >> > > > > > >> > > > > > > > > dev <
> > >> > > > > > >> > > > > > > > > > > > >> >> >>> dev@polaris.apache.org>
> > >> wrote:
> > >> > > > > > >> > > > > > > > > > > > >> >> >>>
> > >> > > > > > >> > > > > > > > > > > > >> >> >>> > Hi Yufei and Dmitri,
> > >> > > > > > >> > > > > > > > > > > > >> >> >>> >
> > >> > > > > > >> > > > > > > > > > > > >> >> >>> > Here is a proposal for
> the
> > >> > REST
> > >> > > > > > >> endpoints
> > >> > > > > > >> > for
> > >> > > > > > >> > > > > > metrics
> > >> > > > > > >> > > > > > > > and
> > >> > > > > > >> > > > > > > > > > > > events.
> > >> > > > > > >> > > > > > > > > > > > >> >> >>> >
> > >> > > > > > >> > > > > > > > > > > > >> >> >>> >
> > >> > > > > > >> > > > > >
> > >> >
> > >>
> https://urldefense.com/v3/__https://github.com/apache/polaris/pull/3924/changes__;!!Iz9xO38YGHZK!4rnlVIqBQQQ6xpGe1PMV58edOS_cGrKoqdfhw4qSp_JQCavvEjaxXk0OwGPnxyEcO60AzKQK_XDtp7H3DdkFXcrXcjoPhSs$
> > >> > > > > > >> > > > > > > > > > > > >> >> >>> >
> > >> > > > > > >> > > > > > > > > > > > >> >> >>> > I did not see any
> > >> precursors
> > >> > for
> > >> > > > > > >> raising a
> > >> > > > > > >> > PR
> > >> > > > > > >> > > > for
> > >> > > > > > >> > > > > > > > > > proposals,
> > >> > > > > > >> > > > > > > > > > > so
> > >> > > > > > >> > > > > > > > > > > > >> >> trying
> > >> > > > > > >> > > > > > > > > > > > >> >> >>> > this.  Please let me
> know
> > >> what
> > >> > > you
> > >> > > > > > >> think.
> > >> > > > > > >> > > > > > > > > > > > >> >> >>> >
> > >> > > > > > >> > > > > > > > > > > > >> >> >>> > -
> > >> > > > > > >> > > > > > > > > > > > >> >> >>> > Anand
> > >> > > > > > >> > > > > > > > > > > > >> >> >>> >
> > >> > > > > > >> > > > > > > > > > > > >> >> >>> > From: Anand Kumar
> Sankaran
> > >> <
> > >> > > > > > >> > > > > > > anand.sankaran@workday.com
> > >> > > > > > >> > > > > > > > >
> > >> > > > > > >> > > > > > > > > > > > >> >> >>> > Date: Monday, March 2,
> > >> 2026 at
> > >> > > > > > 10:25 AM
> > >> > > > > > >> > > > > > > > > > > > >> >> >>> > To:
> dev@polaris.apache.org
> > >> <
> > >> > > > > > >> > > > > dev@polaris.apache.org
> > >> > > > > > >> > > > > > >
> > >> > > > > > >> > > > > > > > > > > > >> >> >>> > Subject: Re: Polaris
> > >> Telemetry
> > >> > > and
> > >> > > > > > Audit
> > >> > > > > > >> > > Trail
> > >> > > > > > >> > > > > > > > > > > > >> >> >>> >
> > >> > > > > > >> > > > > > > > > > > > >> >> >>> > About the REST API,
> based
> > >> on
> > >> > my
> > >> > > use
> > >> > > > > > >> cases:
> > >> > > > > > >> > > > > > > > > > > > >> >> >>> >
> > >> > > > > > >> > > > > > > > > > > > >> >> >>> >
> > >> > > > > > >> > > > > > > > > > > > >> >> >>> >   1.
> > >> > > > > > >> > > > > > > > > > > > >> >> >>> > I want to be able to
> query
> > >> > > commit
> > >> > > > > > >> metrics
> > >> > > > > > >> > to
> > >> > > > > > >> > > > > track
> > >> > > > > > >> > > > > > > > files
> > >> > > > > > >> > > > > > > > > > > added
> > >> > > > > > >> > > > > > > > > > > > /
> > >> > > > > > >> > > > > > > > > > > > >> >> >>> removed
> > >> > > > > > >> > > > > > > > > > > > >> >> >>> > per commit, along with
> > >> record
> > >> > > > > counts.
> > >> > > > > > >> The
> > >> > > > > > >> > > > > ingestion
> > >> > > > > > >> > > > > > > > > > pipeline
> > >> > > > > > >> > > > > > > > > > > > that
> > >> > > > > > >> > > > > > > > > > > > >> >> >>> writes
> > >> > > > > > >> > > > > > > > > > > > >> >> >>> > this data is owned by
> us
> > >> and
> > >> > we
> > >> > > are
> > >> > > > > > >> > > guaranteed
> > >> > > > > > >> > > > to
> > >> > > > > > >> > > > > > > write
> > >> > > > > > >> > > > > > > > > > this
> > >> > > > > > >> > > > > > > > > > > > >> >> >>> information
> > >> > > > > > >> > > > > > > > > > > > >> >> >>> > for each write.
> > >> > > > > > >> > > > > > > > > > > > >> >> >>> >   2.
> > >> > > > > > >> > > > > > > > > > > > >> >> >>> > I want to be able to
> query
> > >> > scan
> > >> > > > > > metrics
> > >> > > > > > >> for
> > >> > > > > > >> > > > > read. I
> > >> > > > > > >> > > > > > > > > > > understand
> > >> > > > > > >> > > > > > > > > > > > >> >> clients
> > >> > > > > > >> > > > > > > > > > > > >> >> >>> do
> > >> > > > > > >> > > > > > > > > > > > >> >> >>> > not fulfill this
> > >> requirement.
> > >> > > > > > >> > > > > > > > > > > > >> >> >>> >   3.
> > >> > > > > > >> > > > > > > > > > > > >> >> >>> > I want to be able to
> query
> > >> the
> > >> > > > > events
> > >> > > > > > >> table
> > >> > > > > > >> > > > > (events
> > >> > > > > > >> > > > > > > are
> > >> > > > > > >> > > > > > > > > > > > >> persisted) -
> > >> > > > > > >> > > > > > > > > > > > >> >> >>> this
> > >> > > > > > >> > > > > > > > > > > > >> >> >>> > may supersede #2, I am
> not
> > >> > sure
> > >> > > yet.
> > >> > > > > > >> > > > > > > > > > > > >> >> >>> >
> > >> > > > > > >> > > > > > > > > > > > >> >> >>> > All this information
> is in
> > >> the
> > >> > > JDBC
> > >> > > > > > >> based
> > >> > > > > > >> > > > > > persistence
> > >> > > > > > >> > > > > > > > > model
> > >> > > > > > >> > > > > > > > > > > and
> > >> > > > > > >> > > > > > > > > > > > >> is
> > >> > > > > > >> > > > > > > > > > > > >> >> >>> > persisted in the
> > >> metastore. I
> > >> > > > > > currently
> > >> > > > > > >> > don’t
> > >> > > > > > >> > > > > have
> > >> > > > > > >> > > > > > a
> > >> > > > > > >> > > > > > > > need
> > >> > > > > > >> > > > > > > > > > to
> > >> > > > > > >> > > > > > > > > > > > >> query
> > >> > > > > > >> > > > > > > > > > > > >> >> >>> > prometheus or open
> > >> telemetry.
> > >> > I
> > >> > > do
> > >> > > > > > >> publish
> > >> > > > > > >> > > some
> > >> > > > > > >> > > > > > > events
> > >> > > > > > >> > > > > > > > to
> > >> > > > > > >> > > > > > > > > > > > >> Prometheus
> > >> > > > > > >> > > > > > > > > > > > >> >> >>> and
> > >> > > > > > >> > > > > > > > > > > > >> >> >>> > they are forwarded to
> our
> > >> > > dashboards
> > >> > > > > > >> > > elsewhere.
> > >> > > > > > >> > > > > > > > > > > > >> >> >>> >
> > >> > > > > > >> > > > > > > > > > > > >> >> >>> > About the CLI
> utilities, I
> > >> > > meant the
> > >> > > > > > >> admin
> > >> > > > > > >> > > user
> > >> > > > > > >> > > > > > > > > utilities.
> > >> > > > > > >> > > > > > > > > > In
> > >> > > > > > >> > > > > > > > > > > > >> one of
> > >> > > > > > >> > > > > > > > > > > > >> >> >>> the
> > >> > > > > > >> > > > > > > > > > > > >> >> >>> > earliest drafts of my
> > >> > proposal,
> > >> > > > > > Prashant
> > >> > > > > > >> > > > > mentioned
> > >> > > > > > >> > > > > > > that
> > >> > > > > > >> > > > > > > > > the
> > >> > > > > > >> > > > > > > > > > > > >> metrics
> > >> > > > > > >> > > > > > > > > > > > >> >> >>> tables
> > >> > > > > > >> > > > > > > > > > > > >> >> >>> > can grow indefinitely
> and
> > >> > that a
> > >> > > > > > similar
> > >> > > > > > >> > > > problem
> > >> > > > > > >> > > > > > > exists
> > >> > > > > > >> > > > > > > > > > with
> > >> > > > > > >> > > > > > > > > > > > the
> > >> > > > > > >> > > > > > > > > > > > >> >> events
> > >> > > > > > >> > > > > > > > > > > > >> >> >>> > table as well. We
> discussed
> > >> > that
> > >> > > > > > >> cleaning
> > >> > > > > > >> > up
> > >> > > > > > >> > > of
> > >> > > > > > >> > > > > old
> > >> > > > > > >> > > > > > > > > records
> > >> > > > > > >> > > > > > > > > > > > from
> > >> > > > > > >> > > > > > > > > > > > >> >> both
> > >> > > > > > >> > > > > > > > > > > > >> >> >>> > metrics tables and
> events
> > >> > > tables can
> > >> > > > > > be
> > >> > > > > > >> > done
> > >> > > > > > >> > > > via
> > >> > > > > > >> > > > > a
> > >> > > > > > >> > > > > > > CLI
> > >> > > > > > >> > > > > > > > > > > utility.
> > >> > > > > > >> > > > > > > > > > > > >> >> >>> >
> > >> > > > > > >> > > > > > > > > > > > >> >> >>> > I see that Yufei has
> > >> covered
> > >> > the
> > >> > > > > > >> discussion
> > >> > > > > > >> > > > about
> > >> > > > > > >> > > > > > > > > > > datasources.
> > >> > > > > > >> > > > > > > > > > > > >> >> >>> >
> > >> > > > > > >> > > > > > > > > > > > >> >> >>> > -
> > >> > > > > > >> > > > > > > > > > > > >> >> >>> > Anand
> > >> > > > > > >> > > > > > > > > > > > >> >> >>> >
> > >> > > > > > >> > > > > > > > > > > > >> >> >>> >
> > >> > > > > > >> > > > > > > > > > > > >> >> >>> >
> > >> > > > > > >> > > > > > > > > > > > >> >> >>> > From: Yufei Gu <
> > >> > > > > flyrain000@gmail.com>
> > >> > > > > > >> > > > > > > > > > > > >> >> >>> > Date: Friday, February
> 27,
> > >> > 2026
> > >> > > at
> > >> > > > > > >> 9:54 PM
> > >> > > > > > >> > > > > > > > > > > > >> >> >>> > To:
> dev@polaris.apache.org
> > >> <
> > >> > > > > > >> > > > > dev@polaris.apache.org
> > >> > > > > > >> > > > > > >
> > >> > > > > > >> > > > > > > > > > > > >> >> >>> > Subject: Re: Polaris
> > >> Telemetry
> > >> > > and
> > >> > > > > > Audit
> > >> > > > > > >> > > Trail
> > >> > > > > > >> > > > > > > > > > > > >> >> >>> >
> > >> > > > > > >> > > > > > > > > > > > >> >> >>> > This Message Is From an
> > >> > External
> > >> > > > > > Sender
> > >> > > > > > >> > > > > > > > > > > > >> >> >>> > This message came from
> > >> outside
> > >> > > your
> > >> > > > > > >> > > > organization.
> > >> > > > > > >> > > > > > > > > > > > >> >> >>> > Report Suspicious<
> > >> > > > > > >> > > > > > > > > > > > >> >> >>> >
> > >> > > > > > >> > > > > > > > > > > > >> >> >>>
> > >> > > > > > >> > > > > > > > > > > > >> >>
> > >> > > > > > >> > > > > > > > > > > > >>
> > >> > > > > > >> > > > > > > > > > > >
> > >> > > > > > >> > > > > > > > > > >
> > >> > > > > > >> > > > > > > > > >
> > >> > > > > > >> > > > > > > > >
> > >> > > > > > >> > > > > > > >
> > >> > > > > > >> > > > > > >
> > >> > > > > > >> > > > > >
> > >> > > > > > >> > > > >
> > >> > > > > > >> > > >
> > >> > > > > > >> > >
> > >> > > > > > >> >
> > >> > > > > > >>
> > >> > > > > >
> > >> > > > >
> > >> > >
> > >> >
> > >>
> https://us-phishalarm-ewt.proofpoint.com/EWT/v1/Iz9xO38YGHZK!YhNDZABkHi1B699ote2uMwpOZw8i0QMCGO2Szc-HshuABGhGvwPJcymE6G2oUUxtS8xDkSrtGTPm_I3QnVDHoLMk50m9v8z_nZKTkd-bnVUbreF1u0WnfV_X5eYevZl_$
> > >> > > > > > >> > > > > > > > > > > > >> >> >>> > >
> > >> > > > > > >> > > > > > > > > > > > >> >> >>> >
> > >> > > > > > >> > > > > > > > > > > > >> >> >>> >
> > >> > > > > > >> > > > > > > > > > > > >> >> >>> > As I mentioned in
> > >> > > > > > >> > > > > > > > > > > > >> >> >>> >
> > >> > > > > > >> > > > > > > > > > > > >> >> >>>
> > >> > > > > > >> > > > > > > > > > > > >> >>
> > >> > > > > > >> > > > > > > > > > > > >>
> > >> > > > > > >> > > > > > > > > > > >
> > >> > > > > > >> > > > > > > > > > >
> > >> > > > > > >> > > > > > > > > >
> > >> > > > > > >> > > > > > > > >
> > >> > > > > > >> > > > > > > >
> > >> > > > > > >> > > > > > >
> > >> > > > > > >> > > > > >
> > >> > > > > > >> > > > >
> > >> > > > > > >> > > >
> > >> > > > > > >> > >
> > >> > > > > > >> >
> > >> > > > > > >>
> > >> > > > > >
> > >> > > > >
> > >> > >
> > >> >
> > >>
> https://urldefense.com/v3/__https://github.com/apache/polaris/issues/3890__;!!Iz9xO38YGHZK!5EuyFFkk3vhRWVIRvQAWBSQfpJkTMA9HxugzDwXmN0LPPqhEFxYkFRGVhtb8AqUwXtDh2OplcMnbMDHKOxrvDU0$
> > >> > >> > > >> > > > > > > > > > > > >> >> >>> ,
> > >> > > > > > >> > > > > > > > > > > > >> >> >>> > supporting
> > >> > > > > > >> > > > > > > > > > > > >> >> >>> > multiple data sources
> is
> > >> not a
> > >> > > > > trivial
> > >> > > > > > >> > > change.
> > >> > > > > > >> > > > I
> > >> > > > > > >> > > > > > > would
> > >> > > > > > >> > > > > > > > > > > strongly
> > >> > > > > > >> > > > > > > > > > > > >> >> >>> recommend
> > >> > > > > > >> > > > > > > > > > > > >> >> >>> > starting with a design
> > >> > document
> > >> > > to
> > >> > > > > > >> > carefully
> > >> > > > > > >> > > > > > evaluate
> > >> > > > > > >> > > > > > > > the
> > >> > > > > > >> > > > > > > > > > > > >> >> architectural
> > >> > > > > > >> > > > > > > > > > > > >> >> >>> > implications and long
> term
> > >> > > impact.
> > >> > > > > > >> > > > > > > > > > > > >> >> >>> >
> > >> > > > > > >> > > > > > > > > > > > >> >> >>> > A REST endpoint to
> query
> > >> > metrics
> > >> > > > > seems
> > >> > > > > > >> > > > reasonable
> > >> > > > > > >> > > > > > > given
> > >> > > > > > >> > > > > > > > > the
> > >> > > > > > >> > > > > > > > > > > > >> current
> > >> > > > > > >> > > > > > > > > > > > >> >> >>> JDBC
> > >> > > > > > >> > > > > > > > > > > > >> >> >>> > based persistence
> model.
> > >> That
> > >> > > said,
> > >> > > > > we
> > >> > > > > > >> may
> > >> > > > > > >> > > also
> > >> > > > > > >> > > > > > > > consider
> > >> > > > > > >> > > > > > > > > > > > >> alternative
> > >> > > > > > >> > > > > > > > > > > > >> >> >>> > storage models. For
> > >> example,
> > >> > if
> > >> > > we
> > >> > > > > > later
> > >> > > > > > >> > > adopt
> > >> > > > > > >> > > > a
> > >> > > > > > >> > > > > > time
> > >> > > > > > >> > > > > > > > > > series
> > >> > > > > > >> > > > > > > > > > > > >> system
> > >> > > > > > >> > > > > > > > > > > > >> >> >>> such as
> > >> > > > > > >> > > > > > > > > > > > >> >> >>> > Prometheus to store
> > >> metrics,
> > >> > the
> > >> > > > > query
> > >> > > > > > >> > model
> > >> > > > > > >> > > > and
> > >> > > > > > >> > > > > > > access
> > >> > > > > > >> > > > > > > > > > > > patterns
> > >> > > > > > >> > > > > > > > > > > > >> >> would
> > >> > > > > > >> > > > > > > > > > > > >> >> >>> be
> > >> > > > > > >> > > > > > > > > > > > >> >> >>> > fundamentally
> different.
> > >> > > Designing
> > >> > > > > the
> > >> > > > > > >> REST
> > >> > > > > > >> > > API
> > >> > > > > > >> > > > > > > without
> > >> > > > > > >> > > > > > > > > > > > >> considering
> > >> > > > > > >> > > > > > > > > > > > >> >> >>> these
> > >> > > > > > >> > > > > > > > > > > > >> >> >>> > potential evolutions
> may
> > >> limit
> > >> > > > > > >> flexibility.
> > >> > > > > > >> > > I'd
> > >> > > > > > >> > > > > > > suggest
> > >> > > > > > >> > > > > > > > > to
> > >> > > > > > >> > > > > > > > > > > > start
> > >> > > > > > >> > > > > > > > > > > > >> >> with
> > >> > > > > > >> > > > > > > > > > > > >> >> >>> the
> > >> > > > > > >> > > > > > > > > > > > >> >> >>> > use case.
> > >> > > > > > >> > > > > > > > > > > > >> >> >>> >
> > >> > > > > > >> > > > > > > > > > > > >> >> >>> > Yufei
> > >> > > > > > >> > > > > > > > > > > > >> >> >>> >
> > >> > > > > > >> > > > > > > > > > > > >> >> >>> >
> > >> > > > > > >> > > > > > > > > > > > >> >> >>> > On Fri, Feb 27, 2026 at
> > >> > 3:42 PM
> > >> > > > > Dmitri
> > >> > > > > > >> > > > > > Bourlatchkov <
> > >> > > > > > >> > > > > > > > > > > > >> >> dimas@apache.org>
> > >> > > > > > >> > > > > > > > > > > > >> >> >>> > wrote:
> > >> > > > > > >> > > > > > > > > > > > >> >> >>> >
> > >> > > > > > >> > > > > > > > > > > > >> >> >>> > > Hi Anand,
> > >> > > > > > >> > > > > > > > > > > > >> >> >>> > >
> > >> > > > > > >> > > > > > > > > > > > >> >> >>> > > Sharing my view...
> > >> subject
> > >> > to
> > >> > > > > > >> discussion:
> > >> > > > > > >> > > > > > > > > > > > >> >> >>> > >
> > >> > > > > > >> > > > > > > > > > > > >> >> >>> > > 1. Adding non-IRC
> REST
> > >> API
> > >> > to
> > >> > > > > > Polaris
> > >> > > > > > >> is
> > >> > > > > > >> > > > > > perfectly
> > >> > > > > > >> > > > > > > > > fine.
> > >> > > > > > >> > > > > > > > > > > > >> >> >>> > >
> > >> > > > > > >> > > > > > > > > > > > >> >> >>> > > Figuring out specific
> > >> > endpoint
> > >> > > > > URIs
> > >> > > > > > >> and
> > >> > > > > > >> > > > > payloads
> > >> > > > > > >> > > > > > > > might
> > >> > > > > > >> > > > > > > > > > > > require
> > >> > > > > > >> > > > > > > > > > > > >> a
> > >> > > > > > >> > > > > > > > > > > > >> >> few
> > >> > > > > > >> > > > > > > > > > > > >> >> >>> > > roundtrips, so
> opening a
> > >> > > separate
> > >> > > > > > >> thread
> > >> > > > > > >> > > for
> > >> > > > > > >> > > > > that
> > >> > > > > > >> > > > > > > > might
> > >> > > > > > >> > > > > > > > > > be
> > >> > > > > > >> > > > > > > > > > > > >> best.
> > >> > > > > > >> > > > > > > > > > > > >> >> >>> > > Contributors commonly
> > >> create
> > >> > > > > Google
> > >> > > > > > >> Docs
> > >> > > > > > >> > > for
> > >> > > > > > >> > > > > new
> > >> > > > > > >> > > > > > > API
> > >> > > > > > >> > > > > > > > > > > > proposals
> > >> > > > > > >> > > > > > > > > > > > >> too
> > >> > > > > > >> > > > > > > > > > > > >> >> >>> (they
> > >> > > > > > >> > > > > > > > > > > > >> >> >>> > > fairly easy to
> update as
> > >> the
> > >> > > email
> > >> > > > > > >> > > discussion
> > >> > > > > > >> > > > > > > > > > progresses).
> > >> > > > > > >> > > > > > > > > > > > >> >> >>> > >
> > >> > > > > > >> > > > > > > > > > > > >> >> >>> > > There was a
> suggestion to
> > >> > try
> > >> > > > > > Markdown
> > >> > > > > > >> > > (with
> > >> > > > > > >> > > > > PRs)
> > >> > > > > > >> > > > > > > for
> > >> > > > > > >> > > > > > > > > > > > proposals
> > >> > > > > > >> > > > > > > > > > > > >> >> [1]
> > >> > > > > > >> > > > > > > > > > > > >> >> >>> ...
> > >> > > > > > >> > > > > > > > > > > > >> >> >>> > > feel free to give it
> a
> > >> try
> > >> > if
> > >> > > you
> > >> > > > > > are
> > >> > > > > > >> > > > > comfortable
> > >> > > > > > >> > > > > > > > with
> > >> > > > > > >> > > > > > > > > > > that.
> > >> > > > > > >> > > > > > > > > > > > >> >> >>> > >
> > >> > > > > > >> > > > > > > > > > > > >> >> >>> > > 2. Could you clarify
> > >> whether
> > >> > > you
> > >> > > > > > mean
> > >> > > > > > >> end
> > >> > > > > > >> > > > user
> > >> > > > > > >> > > > > > > > > utilities
> > >> > > > > > >> > > > > > > > > > or
> > >> > > > > > >> > > > > > > > > > > > >> admin
> > >> > > > > > >> > > > > > > > > > > > >> >> >>> user
> > >> > > > > > >> > > > > > > > > > > > >> >> >>> > > utilities? In the
> latter
> > >> > case
> > >> > > > > those
> > >> > > > > > >> might
> > >> > > > > > >> > > be
> > >> > > > > > >> > > > > more
> > >> > > > > > >> > > > > > > > > > suitable
> > >> > > > > > >> > > > > > > > > > > > for
> > >> > > > > > >> > > > > > > > > > > > >> the
> > >> > > > > > >> > > > > > > > > > > > >> >> >>> Admin
> > >> > > > > > >> > > > > > > > > > > > >> >> >>> > > CLI (java) not the
> Python
> > >> > CLI,
> > >> > > > > IMHO.
> > >> > > > > > >> > > > > > > > > > > > >> >> >>> > >
> > >> > > > > > >> > > > > > > > > > > > >> >> >>> > > Why would these
> > >> utilities be
> > >> > > > > common
> > >> > > > > > >> with
> > >> > > > > > >> > > > > events?
> > >> > > > > > >> > > > > > > > IMHO,
> > >> > > > > > >> > > > > > > > > > > event
> > >> > > > > > >> > > > > > > > > > > > >> use
> > >> > > > > > >> > > > > > > > > > > > >> >> >>> cases
> > >> > > > > > >> > > > > > > > > > > > >> >> >>> > are
> > >> > > > > > >> > > > > > > > > > > > >> >> >>> > > distinct from
> scan/commit
> > >> > > metrics.
> > >> > > > > > >> > > > > > > > > > > > >> >> >>> > >
> > >> > > > > > >> > > > > > > > > > > > >> >> >>> > > 3. I'd prefer
> separating
> > >> > > metrics
> > >> > > > > > >> > > persistence
> > >> > > > > > >> > > > > from
> > >> > > > > > >> > > > > > > > > > MetaStore
> > >> > > > > > >> > > > > > > > > > > > >> >> >>> persistence
> > >> > > > > > >> > > > > > > > > > > > >> >> >>> > at
> > >> > > > > > >> > > > > > > > > > > > >> >> >>> > > the code level, so
> that
> > >> they
> > >> > > could
> > >> > > > > > be
> > >> > > > > > >> > mixed
> > >> > > > > > >> > > > and
> > >> > > > > > >> > > > > > > > matched
> > >> > > > > > >> > > > > > > > > > > > >> >> >>> independently.
> > >> > > > > > >> > > > > > > > > > > > >> >> >>> > The
> > >> > > > > > >> > > > > > > > > > > > >> >> >>> > > separate datasource
> > >> question
> > >> > > will
> > >> > > > > > >> become
> > >> > > > > > >> > a
> > >> > > > > > >> > > > > > > non-issue
> > >> > > > > > >> > > > > > > > > with
> > >> > > > > > >> > > > > > > > > > > > that
> > >> > > > > > >> > > > > > > > > > > > >> >> >>> approach,
> > >> > > > > > >> > > > > > > > > > > > >> >> >>> > I
> > >> > > > > > >> > > > > > > > > > > > >> >> >>> > > guess.
> > >> > > > > > >> > > > > > > > > > > > >> >> >>> > >
> > >> > > > > > >> > > > > > > > > > > > >> >> >>> > > The rationale for
> > >> separating
> > >> > > scan
> > >> > > > > > >> metrics
> > >> > > > > > >> > > and
> > >> > > > > > >> > > > > > > > metastore
> > >> > > > > > >> > > > > > > > > > > > >> >> persistence
> > >> > > > > > >> > > > > > > > > > > > >> >> >>> is
> > >> > > > > > >> > > > > > > > > > > > >> >> >>> > that
> > >> > > > > > >> > > > > > > > > > > > >> >> >>> > > "cascading deletes"
> > >> between
> > >> > > them
> > >> > > > > are
> > >> > > > > > >> > hardly
> > >> > > > > > >> > > > > ever
> > >> > > > > > >> > > > > > > > > > required.
> > >> > > > > > >> > > > > > > > > > > > >> >> >>> Furthermore,
> > >> > > > > > >> > > > > > > > > > > > >> >> >>> > the
> > >> > > > > > >> > > > > > > > > > > > >> >> >>> > > data and query
> patterns
> > >> are
> > >> > > very
> > >> > > > > > >> > different
> > >> > > > > > >> > > so
> > >> > > > > > >> > > > > > > > different
> > >> > > > > > >> > > > > > > > > > > > >> >> technologies
> > >> > > > > > >> > > > > > > > > > > > >> >> >>> > might
> > >> > > > > > >> > > > > > > > > > > > >> >> >>> > > be beneficial in each
> > >> case.
> > >> > > > > > >> > > > > > > > > > > > >> >> >>> > >
> > >> > > > > > >> > > > > > > > > > > > >> >> >>> > > [1]
> > >> > > > > > >> > > > > > > > > > > > >> >> >>> >
> > >> > > > > > >> > > > > > > > > > > > >> >> >>>
> > >> > > > > > >> > > > > > > > > > > > >> >>
> > >> > > > > > >> > > > > > > > > > > > >>
> > >> > > > > > >> > > > > > > > > > > >
> > >> > > > > > >> > > > > > > > > > >
> > >> > > > > > >> > > > > > > > > >
> > >> > > > > > >> > > > > > > > >
> > >> > > > > > >> > > > > > > >
> > >> > > > > > >> > > > > > >
> > >> > > > > > >> > > > > >
> > >> > > > > > >> > > > >
> > >> > > > > > >> > > >
> > >> > > > > > >> > >
> > >> > > > > > >> >
> > >> > > > > > >>
> > >> > > > > >
> > >> > > > >
> > >> > >
> > >> >
> > >>
> https://urldefense.com/v3/__https://lists.apache.org/thread/yto2wp982t43h1mqjwnslswhws5z47cy__;!!Iz9xO38YGHZK!5EuyFFkk3vhRWVIRvQAWBSQfpJkTMA9HxugzDwXmN0LPPqhEFxYkFRGVhtb8AqUwXtDh2OplcMnbMDHKxYDakNU$
> > >> > >> > > >> > > > > > > > > > > > >> >> >>> > >
> > >> > > > > > >> > > > > > > > > > > > >> >> >>> > > Cheers,
> > >> > > > > > >> > > > > > > > > > > > >> >> >>> > > Dmitri.
> > >> > > > > > >> > > > > > > > > > > > >> >> >>> > >
> > >> > > > > > >> > > > > > > > > > > > >> >> >>> > > On Fri, Feb 27, 2026
> at
> > >> > > 6:19 PM
> > >> > > > > > Anand
> > >> > > > > > >> > Kumar
> > >> > > > > > >> > > > > > > Sankaran
> > >> > > > > > >> > > > > > > > > via
> > >> > > > > > >> > > > > > > > > > > dev
> > >> > > > > > >> > > > > > > > > > > > <
> > >> > > > > > >> > > > > > > > > > > > >> >> >>> > >
> dev@polaris.apache.org>
> > >> > > wrote:
> > >> > > > > > >> > > > > > > > > > > > >> >> >>> > >
> > >> > > > > > >> > > > > > > > > > > > >> >> >>> > > > Thanks all. This
> PR is
> > >> > > merged
> > >> > > > > now.
> > >> > > > > > >> > > > > > > > > > > > >> >> >>> > > >
> > >> > > > > > >> > > > > > > > > > > > >> >> >>> > > > Here are the
> follow-up
> > >> > > features
> > >> > > > > /
> > >> > > > > > >> work
> > >> > > > > > >> > > > > needed.
> > >> > > > > > >> > > > > > > > These
> > >> > > > > > >> > > > > > > > > > > were
> > >> > > > > > >> > > > > > > > > > > > >> all
> > >> > > > > > >> > > > > > > > > > > > >> >> >>> part of
> > >> > > > > > >> > > > > > > > > > > > >> >> >>> > > the
> > >> > > > > > >> > > > > > > > > > > > >> >> >>> > > > merged PR at some
> > >> point in
> > >> > > time
> > >> > > > > > and
> > >> > > > > > >> > were
> > >> > > > > > >> > > > > > removed
> > >> > > > > > >> > > > > > > to
> > >> > > > > > >> > > > > > > > > > > reduce
> > >> > > > > > >> > > > > > > > > > > > >> >> scope.
> > >> > > > > > >> > > > > > > > > > > > >> >> >>> > > >
> > >> > > > > > >> > > > > > > > > > > > >> >> >>> > > > Please let me know
> what
> > >> > you
> > >> > > > > think.
> > >> > > > > > >> > > > > > > > > > > > >> >> >>> > > >
> > >> > > > > > >> > > > > > > > > > > > >> >> >>> > > >
> > >> > > > > > >> > > > > > > > > > > > >> >> >>> > > >   1.  A REST API to
> > >> > paginate
> > >> > > > > > through
> > >> > > > > > >> > > table
> > >> > > > > > >> > > > > > > metrics.
> > >> > > > > > >> > > > > > > > > > This
> > >> > > > > > >> > > > > > > > > > > > >> will be
> > >> > > > > > >> > > > > > > > > > > > >> >> >>> > non-IRC
> > >> > > > > > >> > > > > > > > > > > > >> >> >>> > > > standard addition.
> > >> > > > > > >> > > > > > > > > > > > >> >> >>> > > >   2.  Utilities for
> > >> > > managing old
> > >> > > > > > >> > records,
> > >> > > > > > >> > > > > > should
> > >> > > > > > >> > > > > > > be
> > >> > > > > > >> > > > > > > > > > > common
> > >> > > > > > >> > > > > > > > > > > > >> with
> > >> > > > > > >> > > > > > > > > > > > >> >> >>> events.
> > >> > > > > > >> > > > > > > > > > > > >> >> >>> > > > There was some
> > >> discussion
> > >> > > that
> > >> > > > > it
> > >> > > > > > >> > belongs
> > >> > > > > > >> > > > to
> > >> > > > > > >> > > > > > the
> > >> > > > > > >> > > > > > > > CLI.
> > >> > > > > > >> > > > > > > > > > > > >> >> >>> > > >   3.  Separate
> > >> datasource
> > >> > > > > > (metrics,
> > >> > > > > > >> > > events,
> > >> > > > > > >> > > > > > even
> > >> > > > > > >> > > > > > > > > other
> > >> > > > > > >> > > > > > > > > > > > >> tables?).
> > >> > > > > > >> > > > > > > > > > > > >> >> >>> > > >
> > >> > > > > > >> > > > > > > > > > > > >> >> >>> > > >
> > >> > > > > > >> > > > > > > > > > > > >> >> >>> > > > Anything else?
> > >> > > > > > >> > > > > > > > > > > > >> >> >>> > > >
> > >> > > > > > >> > > > > > > > > > > > >> >> >>> > > > -
> > >> > > > > > >> > > > > > > > > > > > >> >> >>> > > > Anand
> > >> > > > > > >> > > > > > > > > > > > >> >> >>> > > >
> > >> > > > > > >> > > > > > > > > > > > >> >> >>> > > >
> > >> > > > > > >> > > > > > > > > > > > >> >> >>> > >
> > >> > > > > > >> > > > > > > > > > > > >> >> >>> >
> > >> > > > > > >> > > > > > > > > > > > >> >> >>> >
> > >> > > > > > >> > > > > > > > > > > > >> >> >>>
> > >> > > > > > >> > > > > > > > > > > > >> >> >>
> > >> > > > > > >> > > > > > > > > > > > >> >>
> > >> > > > > > >> > > > > > > > > > > > >> >
> > >> > > > > > >> > > > > > > > > > > > >>
> > >> > > > > > >> > > > > > > > > > > > >
> > >> > > > > > >> > > > > > > > > > > >
> > >> > > > > > >> > > > > > > > > > >
> > >> > > > > > >> > > > > > > > > >
> > >> > > > > > >> > > > > > > > >
> > >> > > > > > >> > > > > > > >
> > >> > > > > > >> > > > > > >
> > >> > > > > > >> > > > > >
> > >> > > > > > >> > > > >
> > >> > > > > > >> > > >
> > >> > > > > > >> > >
> > >> > > > > > >> >
> > >> > > > > > >>
> > >> > > > > > >
> > >> > > > > >
> > >> > > > >
> > >> > >
> > >> >
> > >> >
> > >>
> > >
>

Re: Proposal for REST endpoints for table metrics and events

[Jean-Baptiste Onofré <jb...@nanthrax.net>]

Hi guys,

I agree with Dmitri: +1 to merge PR #4115 on Oct 1 (tomorrow).

From my side, all outstanding feedback has been addressed, including
the final documentation points raised by EJ regarding the empty-200
NoOp behavior and reconstructed payload description. The branch is
clean, up to date with main, and CI is green.

Once this lands, we can follow up with PR #4756 for the JDBC query SPI
implementation.

Regards
JB

On Wed, Sep 30, 2026 at 1:53 AM Dmitri Bourlatchkov <di...@apache.org> wrote:
>
> Hi All,
>
> Any remaining blockers on [4115]?
>
> Given that the review was very long, please restate any remaining concerns
> (here or in GH).
>
> Otherwise, I propose merging [4115] on Oct 1.
>
> [4115] https://github.com/apache/polaris/pull/4115
>
> Thanks,
> Dmitri.
>
> On Wed, Sep 23, 2026 at 7:53 PM Dmitri Bourlatchkov <di...@apache.org>
> wrote:
>
> > Hi EJ,
> >
> > Thanks for the meeting recap!
> >
> > All:
> >
> > Worth noting: PR [4115] also contains Management API changes that were not
> > anticipated initially and did not get an API change review until now. These
> > are small changes to add new grant types for metrics RBAC..
> >
> > I've reviewed and approved PR [4115] in GH.
> >
> > [4115] https://github.com/apache/polaris/pull/4115
> >
> > Cheers,
> > Dmitri.
> >
> > On Wed, Aug 12, 2026 at 5:21 PM EJ Wang <ej...@gmail.com>
> > wrote:
> >
> >> Hi folks,
> >>
> >> Notes from today's metrics architecture sync are in the usual doc:
> >>
> >> https://docs.google.com/document/d/100h7c4damrUzVuquYbBHM0EvA4LSWuW2IT2dN_7nYVA/edit?pli=1&tab=t.18iv1m7p6i6b
> >>
> >> Highlights:
> >>
> >> Module placement. core/ now holds the entity data model only. A new spi/
> >> takes every other shared contract: feature SPIs, feature non-SPI
> >> contracts,
> >> substrate SPIs, substrate non-SPI contracts, and contract-required types.
> >> That settles the question PR #5204 reopened, by placing into a new SPI
> >> module. Note that it changes slightly towards the referenced WIP RFC [1]
> >> recommendations: what proposed to place in core/, now should go to spi/
> >> except for the entity data model.
> >>
> >> Metrics PRs. #5068 merged. #5204 needs reshaping onto the split. #4115 and
> >> #4756 stay open with action on the author to follow up on review comments.
> >>
> >> [1]
> >>
> >> https://docs.google.com/document/d/1mj3vGpix9zTDTFFJKtWT5ZxI3PTGI1YNb5xFqyWrOkc/edit?tab=t.0
> >>
> >> On Wed, Jul 15, 2026 at 11:01 AM Anand Kumar Sankaran via dev <
> >> dev@polaris.apache.org> wrote:
> >>
> >> > Thanks EJ.  My plan is to introduce PR0 in a new branch / PR and
> >> repurpose
> >> > #4115 and #4756 for PR1 and PR2 to maintain all the valuable discussions
> >> > that happened there.
> >> >
> >> > From: EJ Wang <ej...@gmail.com>
> >> > Date: Wednesday, July 15, 2026 at 10:41 AM
> >> > To: dev@polaris.apache.org <de...@polaris.apache.org>
> >> > Subject: Re: Proposal for REST endpoints for table metrics and events
> >> >
> >> > Hi folks, Thanks again for the discussion today. I updated the sync doc
> >> > with notes from today's metrics sync: https: //urldefense.
> >> com/v3/__https:
> >> > //docs. google.
> >> >
> >> com/document/d/100h7c4damrUzVuquYbBHM0EvA4LSWuW2IT2dN_7nYVA/edit?pli=1&tab=t.
> >> >
> >> 2ahd7wze5f__;!!Iz9xO38YGHZK!4rnlVIqBQQQ6xpGe1PMV58edOS_cGrKoqdfhw4qSp_JQCavvEjaxXk0OwGPnxyEcO60AzKQK_XDtp7H3DdkFXcrXJwAz1g0$
> >> >
> >> >
> >> > Hi folks,
> >> >
> >> > Thanks again for the discussion today. I updated the sync doc with notes
> >> > from today's metrics sync:
> >> >
> >> >
> >> https://urldefense.com/v3/__https://docs.google.com/document/d/100h7c4damrUzVuquYbBHM0EvA4LSWuW2IT2dN_7nYVA/edit?pli=1&tab=t.2ahd7wze5f__;!!Iz9xO38YGHZK!4rnlVIqBQQQ6xpGe1PMV58edOS_cGrKoqdfhw4qSp_JQCavvEjaxXk0OwGPnxyEcO60AzKQK_XDtp7H3DdkFXcrXJwAz1g0$
> >> >
> >> > See highlights below:
> >> >
> >> > In previous meetings, EJ proposed a two prs framing to push forward the
> >> > metrics query work in polaris. Dmitri pointed out a subtle point that
> >> pr2
> >> > is currently also trying to do the persistence related code placement
> >> > refactor. For the sake of straightforward review process, EJ, Dmitri and
> >> > Anand (the author) agreed to separate the persistence refactoring into
> >> pr0,
> >> > and reframing the pr scopes as follow:
> >> >
> >> >    1.
> >> >
> >> >    Pr0: refactor irc metrics spi
> >> >    1.
> >> >
> >> >       SPI should stay in core/ with metric type enveloped
> >> >       2.
> >> >
> >> >       default spi impl, no-op, in extensions/ - default selection at
> >> >       runtime/
> >> >       3.
> >> >
> >> >       move metrics persistence and correspondings to extensions/ as an
> >> >       alternative SPI impl
> >> >       2.
> >> >
> >> >    Pr1: rest api for query
> >> >
> >> >
> >> https://urldefense.com/v3/__https://github.com/apache/polaris/pull/4115/changes__;!!Iz9xO38YGHZK!4rnlVIqBQQQ6xpGe1PMV58edOS_cGrKoqdfhw4qSp_JQCavvEjaxXk0OwGPnxyEcO60AzKQK_XDtp7H3DdkFXcrXFCbEbRk$
> >> >    1.
> >> >
> >> >       rest api spec in extensions/ as optional spec
> >> >       2.
> >> >
> >> >       SPI in extensions/ as optional spec spi
> >> >       3.
> >> >
> >> >       default spi impl in extensions/ no-op
> >> >       4.
> >> >
> >> >       handler related in runtime/ - thin adaptor from http shape to spi
> >> >       shape
> >> >       3.
> >> >
> >> >    Pr2: jdbc impl for query
> >> >
> >> https://urldefense.com/v3/__https://github.com/apache/polaris/pull/4756__;!!Iz9xO38YGHZK!4rnlVIqBQQQ6xpGe1PMV58edOS_cGrKoqdfhw4qSp_JQCavvEjaxXk0OwGPnxyEcO60AzKQK_XDtp7H3DdkFXcrXRVMBzqE$
> >> >    1.
> >> >
> >> >       alternative spi impl in extensions/ -> query with jdbc
> >> >
> >> >
> >> > Thanks,
> >> > -ej
> >> >
> >> > On Wed, Jul 1, 2026 at 8:01 AM Alexandre Dutra <ad...@apache.org>
> >> wrote:
> >> >
> >> > > Hi all,
> >> > >
> >> > > > While the current implementation stores metrics in the metastore,
> >> other
> >> > > deployments
> >> > > may choose a KV store, a dedicated metrics warehouse, or another
> >> backend
> >> > > entirely. In those cases, exposing a Polaris REST API over a specific
> >> > > schema feels like coupling the API to an implementation detail.
> >> > >
> >> > > I wouldn't say to an implementation detail, but rather to a particular
> >> > > setup / configuration. I think that's fine. If you don't store
> >> > > metrics, you don't read metrics. Similar situations will certainly
> >> > > happen in the future, e.g. if we ever introduce an API to read events,
> >> > > that would only work if events are being persisted in the first place.
> >> > > Again I think that's a fair trade-off.
> >> > >
> >> > > Thanks,
> >> > > Alex
> >> > >
> >> > > On Wed, Jul 1, 2026 at 7:23 AM Dmitri Bourlatchkov <di...@apache.org>
> >> > > wrote:
> >> > > >
> >> > > > Hi Yufei,
> >> > > >
> >> > > > From my POV, it is fine for Polaris to have multiple REST API
> >> surfaces
> >> > > with
> >> > > > different features and different degrees of "readiness". Each API
> >> can
> >> > > > evolve fairly independently. With proper modularization (which Anand
> >> > > did),
> >> > > > changes to the main Polaris code and these extra modules should
> >> > normally
> >> > > be
> >> > > > non-intrusive to each other.
> >> > > >
> >> > > > I assume Anand is going to be the first user of this API. However,
> >> if
> >> > we
> >> > > do
> >> > > > not merge it we'll never know whether other users are interested. I
> >> do
> >> > > not
> >> > > > think many end users read the dev ML, but they certainly read the
> >> docs,
> >> > > > code and release notes.
> >> > > >
> >> > > > There is no risk for Polaris as a project in exposing this
> >> experimental
> >> > > > API. If usage dwindles over time, we can remove it. If it gains a
> >> > > > considerable user / contributor base, we will promote it to a
> >> > > "first-class"
> >> > > > (non-beta) API.
> >> > > >
> >> > > > Cheers,
> >> > > > Dmitri.
> >> > > >
> >> > > > On Tue, Jun 30, 2026 at 7:53 PM Yufei Gu <fl...@gmail.com>
> >> wrote:
> >> > > >
> >> > > > > Hi all,
> >> > > > >
> >> > > > > Thanks for driving this work forward. Building this as a POC makes
> >> > > perfect
> >> > > > > sense if the goal is to validate the idea and gather feedback.
> >> Should
> >> > > we
> >> > > > > merge a POC into the main branch before agreeing on the overall
> >> > > direction?
> >> > > > > To me, that feels like a separate discussion from whether the
> >> > > > > implementation itself looks reasonable.
> >> > > > >
> >> > > > > One concern that I don't think has been fully addressed is
> >> whether we
> >> > > need
> >> > > > > to expose metrics consumption through a Polaris REST endpoint at
> >> all.
> >> > > > >
> >> > > > > My concern is less about the API itself and more about the
> >> boundary
> >> > > Polaris
> >> > > > > should own. The proposed REST API is tied to a particular
> >> persistence
> >> > > > > schema, but that schema is just one possible implementation. While
> >> > the
> >> > > > > current implementation stores metrics in the metastore, other
> >> > > deployments
> >> > > > > may choose a KV store, a dedicated metrics warehouse, or another
> >> > > backend
> >> > > > > entirely. In those cases, exposing a Polaris REST API over a
> >> specific
> >> > > > > schema feels like coupling the API to an implementation detail.
> >> > > > >
> >> > > > > Polaris isn't intended to become a full metrics system. I'd
> >> expect it
> >> > > to
> >> > > > > focus on producing metrics and let deployments choose the most
> >> > > appropriate
> >> > > > > storage and consumption model for their environment.
> >> > > > >
> >> > > > > I'm not opposed to exploring this direction, but we should first
> >> > align
> >> > > on
> >> > > > > whether exposing a REST API through Polaris is the right long term
> >> > > approach
> >> > > > > before merging it into the main branch.
> >> > > > >
> >> > > > > Thanks,
> >> > > > > Yufei
> >> > > > >
> >> > > > > Yufei
> >> > > > >
> >> > > > >
> >> > > > > On Mon, Jun 29, 2026 at 9:27 AM Dmitri Bourlatchkov <
> >> > dimas@apache.org>
> >> > > > > wrote:
> >> > > > >
> >> > > > > > Hi All,
> >> > > > > >
> >> > > > > > It looks like Anand addressed all action items in [4115].
> >> > > > > >
> >> > > > > > Please review the latest state of the code. The PR LGTM and I
> >> > > approved in
> >> > > > > > GH from my side.
> >> > > > > >
> >> > > > > > [4115]
> >> >
> >> https://urldefense.com/v3/__https://github.com/apache/polaris/pull/4115__;!!Iz9xO38YGHZK!4rnlVIqBQQQ6xpGe1PMV58edOS_cGrKoqdfhw4qSp_JQCavvEjaxXk0OwGPnxyEcO60AzKQK_XDtp7H3DdkFXcrXtFRscKs$
> >> > > > > >
> >> > > > > > Thanks,
> >> > > > > > Dmitri.
> >> > > > > >
> >> > > > > > On Thu, Jun 25, 2026 at 1:56 PM Dmitri Bourlatchkov <
> >> > > dimas@apache.org>
> >> > > > > > wrote:
> >> > > > > >
> >> > > > > > > Hi All,
> >> > > > > > >
> >> > > > > > > Recap of the discussion in today's Community Sync:
> >> > > > > > >
> >> > > > > > > * The modular approach for Metrics SPI(s) and implementation
> >> > seems
> >> > > to
> >> > > > > > have
> >> > > > > > > community consensus (please comment if you still have
> >> concerns)
> >> > > > > > >
> >> > > > > > > * What modules / feature flags are enabled in Apache Polaris
> >> > > images by
> >> > > > > > > default will be discussed later (when the feature is close to
> >> > > > > > completion).
> >> > > > > > >
> >> > > > > > > In any case, users will be able to turn the full feature
> >> on/off
> >> > at
> >> > > > > their
> >> > > > > > > discretion, regardless of the default flag values.
> >> > > > > > >
> >> > > > > > > * It is generally preferable to have smaller PRs and introduce
> >> > new
> >> > > > > > > features gradually to simplify reviews.
> >> > > > > > >
> >> > > > > > > What this means for the Metrics API, IMHO is:
> >> > > > > > >
> >> > > > > > > a) The scope of PR 4115 is probably ok as it stands right
> >> now. If
> >> > > > > someone
> >> > > > > > > wants to propose splitting it further, please comment.
> >> > > > > > >
> >> > > > > > > b) After 4115 is merged, Metrics Persistence PRs will follow,
> >> > with
> >> > > > > > smaller
> >> > > > > > > scope per PR as appropriate for the nature of the code
> >> changes.
> >> > > > > > >
> >> > > > > > > * The major concern was about exposing the new Metrics REST
> >> API
> >> > to
> >> > > end
> >> > > > > > > users. The nature of this concern was about avoiding the
> >> (false)
> >> > > > > > impression
> >> > > > > > > that the new API is a firm and adopted Polaris specification.
> >> > > > > > >
> >> > > > > > > I think this concern is reasonable.
> >> > > > > > >
> >> > > > > > > I propose that Anand update the documentation sections of PR
> >> 4115
> >> > > to
> >> > > > > > > emphasise that the API is currently an experimental / POC API
> >> at
> >> > > this
> >> > > > > > time
> >> > > > > > > and is subject to change (including breaking changes). I
> >> believe
> >> > it
> >> > > > > > already
> >> > > > > > > has the "beta" label. So this request is mostly for doc
> >> changes
> >> > to
> >> > > > > > clarify
> >> > > > > > > user expectations, since the "beta" label alone may not be
> >> > > interpreted
> >> > > > > > the
> >> > > > > > > same way by all users.
> >> > > > > > >
> >> > > > > > > I hope this update will resolve all remaining concerns with
> >> 4115
> >> > > and
> >> > > > > > allow
> >> > > > > > > for it to be merged so that the community will be able to
> >> proceed
> >> > > to
> >> > > > > > > follow-up PRs (as noted above).
> >> > > > > > >
> >> > > > > > > Please add/correct me if I missed anything relevant.
> >> > > > > > >
> >> > > > > > > Thanks,
> >> > > > > > > Dmitri.
> >> > > > > > >
> >> > > > > > > On Wed, Jun 24, 2026 at 8:11 PM Yufei Gu <
> >> flyrain000@gmail.com>
> >> > > wrote:
> >> > > > > > >
> >> > > > > > >> I do not think we have agreement on the query REST APIs yet.
> >> > > > > > >>
> >> > > > > > >> In particular, the current query API shape seems highly
> >> coupled
> >> > > to the
> >> > > > > > >> *example[1]* metrics in the Iceberg REST spec. I do not think
> >> > > that is
> >> > > > > a
> >> > > > > > >> good way to design a REST API contract. For example, fields
> >> like
> >> > > > > > >> resultDataFiles, resultDeleteFiles, totalFileSizeBytes,
> >> > > > > > >> totalDataManifests,
> >> > > > > > >> totalDeleteManifests, and scannedDataManifests are very
> >> specific
> >> > > to
> >> > > > > one
> >> > > > > > >> style of scan metric. That makes the API look more like a
> >> direct
> >> > > > > > >> projection
> >> > > > > > >> of the current example payload than a general query model.
> >> > > > > > >>
> >> > > > > > >> My preference is that Polaris should not define or strongly
> >> > opine
> >> > > on
> >> > > > > the
> >> > > > > > >> Iceberg scan or commit metrics query REST API at this point.
> >> > > Polaris
> >> > > > > can
> >> > > > > > >> resolve and authorize the table, accept the metrics report,
> >> and
> >> > > > > delegate
> >> > > > > > >> it
> >> > > > > > >> to the configured reporting path. Durable storage, filtering,
> >> > > > > > dashboards,
> >> > > > > > >> and query APIs should remain downstream impl. choices unless
> >> and
> >> > > until
> >> > > > > > the
> >> > > > > > >> Iceberg community standardizes a query contract, which I
> >> doubt
> >> > > will
> >> > > > > > >> happen.
> >> > > > > > >>
> >> > > > > > >> So I do not think the intake path has to be blocked, but I
> >> would
> >> > > > > prefer
> >> > > > > > >> not
> >> > > > > > >> to merge a Polaris specific query API that may be hard to
> >> evolve
> >> > > > > later.
> >> > > > > > >>
> >> > > > > > >> 1.
> >> > > > > > >>
> >> > > > > > >>
> >> > > > > >
> >> > > > >
> >> > >
> >> >
> >> https://urldefense.com/v3/__https://github.com/apache/iceberg/blob/2a6c556c0c883c04c6d3fbd68e6aeda36b91e0aa/open-api/rest-catalog-open-api.yaml*L4152__;Iw!!Iz9xO38YGHZK!4rnlVIqBQQQ6xpGe1PMV58edOS_cGrKoqdfhw4qSp_JQCavvEjaxXk0OwGPnxyEcO60AzKQK_XDtp7H3DdkFXcrXCDsqYGo$
> >> > > > > > >>
> >> > > > > > >> Yufei
> >> > > > > > >>
> >> > > > > > >>
> >> > > > > > >> On Tue, Jun 23, 2026 at 9:30 AM Dmitri Bourlatchkov <
> >> > > dimas@apache.org
> >> > > > > >
> >> > > > > > >> wrote:
> >> > > > > > >>
> >> > > > > > >> > Hi All,
> >> > > > > > >> >
> >> > > > > > >> > So, what are the blockers for merging [4115], if any?
> >> > > > > > >> >
> >> > > > > > >> > [4115]
> >> >
> >> https://urldefense.com/v3/__https://github.com/apache/polaris/pull/4115__;!!Iz9xO38YGHZK!4rnlVIqBQQQ6xpGe1PMV58edOS_cGrKoqdfhw4qSp_JQCavvEjaxXk0OwGPnxyEcO60AzKQK_XDtp7H3DdkFXcrXtFRscKs$
> >> > > > > > >> >
> >> > > > > > >> > Thanks,
> >> > > > > > >> > Dmitri.
> >> > > > > > >> >
> >> > > > > > >> > On Wed, Jun 17, 2026 at 5:28 PM EJ Wang <
> >> > > > > > ej.wang.dev.interest@gmail.com
> >> > > > > > >> >
> >> > > > > > >> > wrote:
> >> > > > > > >> >
> >> > > > > > >> > > Hi folks,
> >> > > > > > >> > >
> >> > > > > > >> > > Thanks again for the discussion today. I updated the sync
> >> > doc
> >> > > with
> >> > > > > > >> notes
> >> > > > > > >> > > from today's metrics sync:
> >> > > > > > >> > >
> >> > > > > > >> > >
> >> > > > > > >> > >
> >> > > > > > >> >
> >> > > > > > >>
> >> > > > > >
> >> > > > >
> >> > >
> >> >
> >> https://urldefense.com/v3/__https://docs.google.com/document/d/100h7c4damrUzVuquYbBHM0EvA4LSWuW2IT2dN_7nYVA/edit?tab=t.ezk0rgdx0c6m__;!!Iz9xO38YGHZK!4rnlVIqBQQQ6xpGe1PMV58edOS_cGrKoqdfhw4qSp_JQCavvEjaxXk0OwGPnxyEcO60AzKQK_XDtp7H3DdkFXcrX83IZkm0$
> >> > > > > > >> > >
> >> > > > > > >> > > A few highlights:
> >> > > > > > >> > >
> >> > > > > > >> > >    - No objection to keeping the default metrics behavior
> >> > > small:
> >> > > > > > >> > >    no-op/log-only is enough as the built-in default.
> >> > > > > > >> > >    - Durable storage, event forwarding, external queues,
> >> > > > > dashboards,
> >> > > > > > >> and
> >> > > > > > >> > >    custom filtering should be implementation choices
> >> behind
> >> > > the
> >> > > > > > >> metrics
> >> > > > > > >> > >    reporting path.
> >> > > > > > >> > >    - For PR #4115, the REST layer should
> >> > resolve/authz/accept
> >> > > the
> >> > > > > > >> report,
> >> > > > > > >> > >    then delegate to the selected metrics implementation.
> >> It
> >> > > should
> >> > > > > > not
> >> > > > > > >> > own
> >> > > > > > >> > >    durable storage semantics.
> >> > > > > > >> > >    - The rough boundary we discussed is: spec owns the
> >> wire
> >> > > > > > contract,
> >> > > > > > >> > >    runtime owns framework wiring, API/contract modules
> >> own
> >> > > > > > >> > provider-facing
> >> > > > > > >> > >    contracts, and extensions own replaceable
> >> > implementations.
> >> > > > > > >> > >    - OpenLineage has a similar path-naming question. We
> >> > should
> >> > > > > avoid
> >> > > > > > >> > >    occupying a generic lineage path too early if we want
> >> > room
> >> > > for
> >> > > > > > >> other
> >> > > > > > >> > >    lineage systems later.
> >> > > > > > >> > >
> >> > > > > > >> > > I believe the boundary rules previewed in sync meeting
> >> are
> >> > > useful
> >> > > > > > for
> >> > > > > > >> > > reviewing PR #4115, but it is not
> >> > > > > > >> > > a final module-layout proposal.
> >> > > > > > >> > >
> >> > > > > > >> > > For metrics specifically, I think the target remains:
> >> > > > > > >> > >    runtime
> >> > > > > > >> > >      -> metrics reporting/emitting contract
> >> > > > > > >> > >      -> default implementation
> >> > > > > > >> > >           -> lower implementation details, if needed
> >> > > > > > >> > >
> >> > > > > > >> > > That keeps the default simple, while allowing durable
> >> JDBC,
> >> > > > > > >> event-backed,
> >> > > > > > >> > > or external-queue implementations to be added separately.
> >> > > > > > >> > >
> >> > > > > > >> > > We also touched on the split between Iceberg metrics
> >> > emitting
> >> > > and
> >> > > > > > >> > > Polaris-owned metrics querying. The querying side can
> >> keep
> >> > > > > evolving,
> >> > > > > > >> > > especially for dashboard or generic-table use cases, but
> >> I
> >> > do
> >> > > not
> >> > > > > > >> think
> >> > > > > > >> > > that needs to block the intake path in this PR.
> >> > > > > > >> > >
> >> > > > > > >> > > Thanks,
> >> > > > > > >> > > -ej
> >> > > > > > >> > >
> >> > > > > > >> > > On Tue, Jun 16, 2026 at 2:17 PM Yufei Gu <
> >> > > flyrain000@gmail.com>
> >> > > > > > >> wrote:
> >> > > > > > >> > >
> >> > > > > > >> > > > Thanks for chiming in, EJ. Agreed that the default
> >> battery
> >> > > > > should
> >> > > > > > >> stay
> >> > > > > > >> > > > small. I'm leaning toward not including metrics
> >> > persistence
> >> > > in
> >> > > > > the
> >> > > > > > >> > > default
> >> > > > > > >> > > > battery.
> >> > > > > > >> > > >
> >> > > > > > >> > > > Yufei
> >> > > > > > >> > > >
> >> > > > > > >> > > >
> >> > > > > > >> > > > On Thu, Jun 11, 2026 at 8:30 PM EJ Wang <
> >> > > > > > >> > ej.wang.dev.interest@gmail.com>
> >> > > > > > >> > > > wrote:
> >> > > > > > >> > > >
> >> > > > > > >> > > > > Hi Yufei,
> >> > > > > > >> > > > >
> >> > > > > > >> > > > > Thanks for connecting this back to the REST endpoint
> >> > > > > proposal. I
> >> > > > > > >> > > replied
> >> > > > > > >> > > > > with the fuller version on the event-forwarding
> >> thread,
> >> > > but
> >> > > > > > >> wanted to
> >> > > > > > >> > > add
> >> > > > > > >> > > > > the shorter version here too.
> >> > > > > > >> > > > >
> >> > > > > > >> > > > > I agree that metrics reporting/emitting is the right
> >> > > > > conceptual
> >> > > > > > >> > > boundary,
> >> > > > > > >> > > > > and I also think the event/listener path is a good
> >> > > > > > implementation
> >> > > > > > >> > > > direction
> >> > > > > > >> > > > > to explore. The distinction I want to keep clear is
> >> > > default
> >> > > > > > >> battery
> >> > > > > > >> > vs
> >> > > > > > >> > > > > extension implementation.
> >> > > > > > >> > > > >
> >> > > > > > >> > > > > For the REST/API side, I would keep the semantics
> >> > narrow:
> >> > > > > > Polaris
> >> > > > > > >> > > > resolves
> >> > > > > > >> > > > > and authorizes the table, accepts the Iceberg
> >> > scan/commit
> >> > > > > > metrics
> >> > > > > > >> > > report
> >> > > > > > >> > > > > into the configured reporting/emitting path, and
> >> returns
> >> > > 204.
> >> > > > > > That
> >> > > > > > >> > 204
> >> > > > > > >> > > > > should mean Polaris accepted the report into the
> >> > ingestion
> >> > > > > path;
> >> > > > > > >> it
> >> > > > > > >> > > > should
> >> > > > > > >> > > > > not imply durable storage.
> >> > > > > > >> > > > >
> >> > > > > > >> > > > > Durable storage, event forwarding, external telemetry
> >> > > routing,
> >> > > > > > >> > > filtering,
> >> > > > > > >> > > > > and retention should sit behind that
> >> reporting/emitting
> >> > > > > boundary
> >> > > > > > >> as
> >> > > > > > >> > > > > implementation-layer behavior. A durable JDBC path
> >> can
> >> > be
> >> > > one
> >> > > > > > >> named
> >> > > > > > >> > > > > extension implementation. An event/listener
> >> forwarding
> >> > > path
> >> > > > > can
> >> > > > > > be
> >> > > > > > >> > > > another
> >> > > > > > >> > > > > named extension implementation.
> >> > > > > > >> > > > >
> >> > > > > > >> > > > > So I am +1 on the event-forwarding idea as a
> >> non-default
> >> > > > > > extension
> >> > > > > > >> > > > > implementation. I just would not make it the default
> >> > > battery
> >> > > > > or
> >> > > > > > >> the
> >> > > > > > >> > > core
> >> > > > > > >> > > > > API shape. The default battery can stay small and
> >> safe,
> >> > > while
> >> > > > > > >> > > deployments
> >> > > > > > >> > > > > that need forwarding, dashboards, or durable storage
> >> can
> >> > > opt
> >> > > > > > into
> >> > > > > > >> the
> >> > > > > > >> > > > > implementation that matches their operational model.
> >> > > > > > >> > > > >
> >> > > > > > >> > > > > Thanks,
> >> > > > > > >> > > > > -ej
> >> > > > > > >> > > > >
> >> > > > > > >> > > > > On Wed, Jun 10, 2026 at 11:43 AM Yufei Gu <
> >> > > > > flyrain000@gmail.com
> >> > > > > > >
> >> > > > > > >> > > wrote:
> >> > > > > > >> > > > >
> >> > > > > > >> > > > > > Thanks EJ. I agree that the reporting/emitting
> >> > boundary
> >> > > > > feels
> >> > > > > > >> like
> >> > > > > > >> > > the
> >> > > > > > >> > > > > > right SPI boundary, but I wonder if we can simplify
> >> > this
> >> > > > > even
> >> > > > > > >> > > further.
> >> > > > > > >> > > > > >
> >> > > > > > >> > > > > > Iceberg scan and commit metrics look very similar
> >> to
> >> > > events
> >> > > > > to
> >> > > > > > >> me.
> >> > > > > > >> > > They
> >> > > > > > >> > > > > are
> >> > > > > > >> > > > > > append only, asynchronously consumed, often
> >> forwarded
> >> > to
> >> > > > > > >> external
> >> > > > > > >> > > > > systems,
> >> > > > > > >> > > > > > and have similar retention and cleanup
> >> requirements.
> >> > > More
> >> > > > > > >> details
> >> > > > > > >> > can
> >> > > > > > >> > > > be
> >> > > > > > >> > > > > > found in this thread [1]. In that case, Polaris may
> >> > only
> >> > > > > need
> >> > > > > > to
> >> > > > > > >> > > accept
> >> > > > > > >> > > > > the
> >> > > > > > >> > > > > > report and emit an event through the existing event
> >> > > > > framework.
> >> > > > > > >> > > > Filtering,
> >> > > > > > >> > > > > > async delivery, custom sinks, and retention
> >> mechanisms
> >> > > could
> >> > > > > > >> then
> >> > > > > > >> > be
> >> > > > > > >> > > > > shared
> >> > > > > > >> > > > > > instead of introducing a separate metrics specific
> >> > > extension
> >> > > > > > >> layer
> >> > > > > > >> > > and
> >> > > > > > >> > > > > SPI.
> >> > > > > > >> > > > > >
> >> > > > > > >> > > > > > The main requirement I still see is querying
> >> metrics
> >> > > through
> >> > > > > > >> > Polaris.
> >> > > > > > >> > > > If
> >> > > > > > >> > > > > we
> >> > > > > > >> > > > > > want that battery included experience, we could
> >> work
> >> > on
> >> > > a
> >> > > > > > >> > persistence
> >> > > > > > >> > > > > > listener implementation that selectively saves
> >> metrics
> >> > > and
> >> > > > > > >> expose
> >> > > > > > >> > it
> >> > > > > > >> > > > via
> >> > > > > > >> > > > > > the IRC event endpoint which seems pretty close in
> >> > > Iceberg
> >> > > > > > >> > > > > > community. Another option I like more is to allow
> >> > users
> >> > > to
> >> > > > > > >> develop
> >> > > > > > >> > > > their
> >> > > > > > >> > > > > > own listener, so they can persist scan/commit
> >> metrics
> >> > > to any
> >> > > > > > >> > systems
> >> > > > > > >> > > > they
> >> > > > > > >> > > > > > prefer. In general, that feels like an
> >> implementation
> >> > > choice
> >> > > > > > on
> >> > > > > > >> top
> >> > > > > > >> > > of
> >> > > > > > >> > > > > the
> >> > > > > > >> > > > > > event framework rather than something that needs to
> >> > > shape
> >> > > > > the
> >> > > > > > >> core
> >> > > > > > >> > > > > > architecture.
> >> > > > > > >> > > > > >
> >> > > > > > >> > > > > > 1.
> >> > > > > > >> >
> >> > >
> >> >
> >> https://urldefense.com/v3/__https://lists.apache.org/thread/x9j8nscvy8hq61tyn01mj8yp6n9of0kp__;!!Iz9xO38YGHZK!4rnlVIqBQQQ6xpGe1PMV58edOS_cGrKoqdfhw4qSp_JQCavvEjaxXk0OwGPnxyEcO60AzKQK_XDtp7H3DdkFXcrXGcGDgks$
> >> > > > > > >> > > > > >
> >> > > > > > >> > > > > > Thanks,
> >> > > > > > >> > > > > > Yufei
> >> > > > > > >> > > > > >
> >> > > > > > >> > > > > >
> >> > > > > > >> > > > > > On Thu, Jun 4, 2026 at 5:14 PM EJ Wang <
> >> > > > > > >> > > ej.wang.dev.interest@gmail.com
> >> > > > > > >> > > > >
> >> > > > > > >> > > > > > wrote:
> >> > > > > > >> > > > > >
> >> > > > > > >> > > > > > > Good point. I agree the existing
> >> > > PolarisMetricsReporter is
> >> > > > > > >> > already
> >> > > > > > >> > > > very
> >> > > > > > >> > > > > > > close to the right conceptual boundary. I am not
> >> > > > > proposing a
> >> > > > > > >> > second
> >> > > > > > >> > > > > > > parallel reporter concept. The distinction I am
> >> > > trying to
> >> > > > > > >> make is
> >> > > > > > >> > > > > > narrower:
> >> > > > > > >> > > > > > >
> >> > > > > > >> > > > > > > Current state:
> >> > > > > > >> > > > > > >
> >> > > > > > >> > > > > > >    - PolarisMetricsReporter is the existing
> >> metrics
> >> > > > > > reporting
> >> > > > > > >> > hook.
> >> > > > > > >> > > > > > >    - But it currently lives under
> >> runtime/service.
> >> > > > > > >> > > > > > >    - Its contract is documented around
> >> Quarkus/CDI
> >> > > > > discovery
> >> > > > > > >> and
> >> > > > > > >> > > > > > > @Identifier
> >> > > > > > >> > > > > > >    selection.
> >> > > > > > >> > > > > > >    - The default implementation is log-only.
> >> > > > > > >> > > > > > >    - The durable implementation writes through
> >> > > > > > >> > > PolarisMetricsManager
> >> > > > > > >> > > > > and
> >> > > > > > >> > > > > > >    MetricsPersistence.
> >> > > > > > >> > > > > > >    - MetricsPersistence is currently inherited by
> >> > > > > > >> > BasePersistence.
> >> > > > > > >> > > > > > >
> >> > > > > > >> > > > > > > My proposal:
> >> > > > > > >> > > > > > >
> >> > > > > > >> > > > > > >    - Keep the reporting/emitting boundary as the
> >> > SPI.
> >> > > > > > >> > > > > > >    - Revise that contract in a framework-agnostic
> >> > > optional
> >> > > > > > >> > Iceberg
> >> > > > > > >> > > > > > *metrics
> >> > > > > > >> > > > > > >    extension API layer*.
> >> > > > > > >> > > > > > >    - Keep runtime/service responsible for REST
> >> > > ingestion,
> >> > > > > > >> > > table/authz
> >> > > > > > >> > > > > > >    validation, and runtime wiring.
> >> > > > > > >> > > > > > >    - Keep the battery implementation log-only in
> >> the
> >> > > > > metrics
> >> > > > > > >> > > > extension
> >> > > > > > >> > > > > > API
> >> > > > > > >> > > > > > >    layer, *not under runtime/service*.
> >> > > > > > >> > > > > > >    - Treat durable JDBC metrics as one
> >> > implementation
> >> > > of
> >> > > > > the
> >> > > > > > >> > > > reporting
> >> > > > > > >> > > > > > SPI.
> >> > > > > > >> > > > > > >    - Decompose the MetricsPersistence and
> >> > > BasePersistence
> >> > > > > > >> > coupling
> >> > > > > > >> > > as
> >> > > > > > >> > > > > > >    durable implementation detail.
> >> > > > > > >> > > > > > >
> >> > > > > > >> > > > > > > So the difference is not “new reporter concept vs
> >> > old
> >> > > > > > reporter
> >> > > > > > >> > > > > concept.”
> >> > > > > > >> > > > > > > but:
> >> > > > > > >> > > > > > >
> >> > > > > > >> > > > > > >
> >> > > > > > >> > > > > > >    - PolarisMetricsReporter today =
> >> runtime/service
> >> > > > > > CDI-shaped
> >> > > > > > >> > > hook.
> >> > > > > > >> > > > > > >    - Target reporting SPI = same conceptual
> >> > boundary,
> >> > > but
> >> > > > > > >> placed
> >> > > > > > >> > in
> >> > > > > > >> > > > the
> >> > > > > > >> > > > > > >    right extension API layer and kept
> >> > > framework-agnostic.
> >> > > > > > >> > > > > > >
> >> > > > > > >> > > > > > > For the current metrics PR, I think the minimal
> >> > > > > > architectural
> >> > > > > > >> > > cleanup
> >> > > > > > >> > > > > > > before merge is to *put the reporting SPI
> >> contract
> >> > > and the
> >> > > > > > >> > battery
> >> > > > > > >> > > > > > default
> >> > > > > > >> > > > > > > in the right place*. The durable JDBC
> >> > implementation,
> >> > > > > > >> > event-backed
> >> > > > > > >> > > > > async
> >> > > > > > >> > > > > > > implementation, external queue support, and
> >> deeper
> >> > > > > > persistence
> >> > > > > > >> > > > cleanup
> >> > > > > > >> > > > > > can
> >> > > > > > >> > > > > > > continue as follow-ups, as long as we do not lock
> >> > the
> >> > > SPI
> >> > > > > > >> > boundary
> >> > > > > > >> > > to
> >> > > > > > >> > > > > > > BasePersistence or to runtime/service wiring.
> >> > > > > > >> > > > > > >
> >> > > > > > >> > > > > > > -ej
> >> > > > > > >> > > > > > >
> >> > > > > > >> > > > > > > On Wed, Jun 3, 2026 at 3:00 PM Dmitri
> >> Bourlatchkov <
> >> > > > > > >> > > dimas@apache.org
> >> > > > > > >> > > > >
> >> > > > > > >> > > > > > > wrote:
> >> > > > > > >> > > > > > >
> >> > > > > > >> > > > > > > > Hi EJ,
> >> > > > > > >> > > > > > > >
> >> > > > > > >> > > > > > > > Thanks for the recap / summary!
> >> > > > > > >> > > > > > > >
> >> > > > > > >> > > > > > > > > Do folks agree that the stable SPI boundary
> >> > > should be
> >> > > > > > >> > > > > > > > metrics reporting/emitting, not metrics
> >> > persistence?
> >> > > > > > >> > > > > > > >
> >> > > > > > >> > > > > > > > SGTM.
> >> > > > > > >> > > > > > > >
> >> > > > > > >> > > > > > > > > Does an optional Iceberg metrics extension
> >> API
> >> > > layer
> >> > > > > > sound
> >> > > > > > >> > like
> >> > > > > > >> > > > the
> >> > > > > > >> > > > > > > right
> >> > > > > > >> > > > > > > > home for this SPI?
> >> > > > > > >> > > > > > > >
> >> > > > > > >> > > > > > > > How is that different from current
> >> > > > > PolarisMetricsReporter?
> >> > > > > > >> > > > > > > >
> >> > > > > > >> > > > > > > > > Should the current durable metrics work be
> >> > > reframed
> >> > > > > as a
> >> > > > > > >> > > durable
> >> > > > > > >> > > > > > > > JDBC reference implementation of that SPI?
> >> > > > > > >> > > > > > > >
> >> > > > > > >> > > > > > > > SGTM.
> >> > > > > > >> > > > > > > >
> >> > > > > > >> > > > > > > > > What is the smallest PR sequence to get there
> >> > > without
> >> > > > > > >> > blocking
> >> > > > > > >> > > > > > > > the current metrics work unnecessarily?
> >> > > > > > >> > > > > > > >
> >> > > > > > >> > > > > > > > Let's get
> >> > >
> >> >
> >> https://urldefense.com/v3/__https://github.com/apache/polaris/pull/4397__;!!Iz9xO38YGHZK!4rnlVIqBQQQ6xpGe1PMV58edOS_cGrKoqdfhw4qSp_JQCavvEjaxXk0OwGPnxyEcO60AzKQK_XDtp7H3DdkFXcrXuu5jefU$
> >> > > > > > >> merged
> >> > > > > > >> > > > first.
> >> > > > > > >> > > > > > > >
> >> > > > > > >> > > > > > > > Then, I'd propose isolating the Metrics schema
> >> > from
> >> > > the
> >> > > > > > >> > MetaStore
> >> > > > > > >> > > > > > schema.
> >> > > > > > >> > > > > > > > This will probably have some ripple effect into
> >> > the
> >> > > > > > >> bootstrap
> >> > > > > > >> > > > > > workflows,
> >> > > > > > >> > > > > > > so
> >> > > > > > >> > > > > > > > it's not a trivial change.
> >> > > > > > >> > > > > > > >
> >> > > > > > >> > > > > > > > Then, let's reassess.
> >> > > > > > >> > > > > > > >
> >> > > > > > >> > > > > > > > Cheers,
> >> > > > > > >> > > > > > > > Dmitri.
> >> > > > > > >> > > > > > > >
> >> > > > > > >> > > > > > > > On Wed, Jun 3, 2026 at 5:40 PM EJ Wang <
> >> > > > > > >> > > > > ej.wang.dev.interest@gmail.com
> >> > > > > > >> > > > > > >
> >> > > > > > >> > > > > > > > wrote:
> >> > > > > > >> > > > > > > >
> >> > > > > > >> > > > > > > > > Hi folks,
> >> > > > > > >> > > > > > > > >
> >> > > > > > >> > > > > > > > > Thanks again for the discussion today. I
> >> updated
> >> > > the
> >> > > > > > sync
> >> > > > > > >> doc
> >> > > > > > >> > > > with
> >> > > > > > >> > > > > > > notes
> >> > > > > > >> > > > > > > > > from the third metrics architecture sync:
> >> > > > > > >> > > > > > > > >
> >> > > > > > >> > > > > > > > >
> >> > > > > > >> > > > > > > > >
> >> > > > > > >> > > > > > > >
> >> > > > > > >> > > > > > >
> >> > > > > > >> > > > > >
> >> > > > > > >> > > > >
> >> > > > > > >> > > >
> >> > > > > > >> > >
> >> > > > > > >> >
> >> > > > > > >>
> >> > > > > >
> >> > > > >
> >> > >
> >> >
> >> https://urldefense.com/v3/__https://docs.google.com/document/d/100h7c4damrUzVuquYbBHM0EvA4LSWuW2IT2dN_7nYVA/edit?tab=t.k96s2xyqr5u1__;!!Iz9xO38YGHZK!4rnlVIqBQQQ6xpGe1PMV58edOS_cGrKoqdfhw4qSp_JQCavvEjaxXk0OwGPnxyEcO60AzKQK_XDtp7H3DdkFXcrXrqCL-QE$
> >> > > > > > >> > > > > > > > >
> >> > > > > > >> > > > > > > > > A few highlights from today’s discussion:
> >> > > > > > >> > > > > > > > >
> >> > > > > > >> > > > > > > > >    - We clarified that Iceberg metrics
> >> reporting
> >> > > can
> >> > > > > be
> >> > > > > > >> > > > interpreted
> >> > > > > > >> > > > > > > > either
> >> > > > > > >> > > > > > > > >    as sync or async handling from Polaris’s
> >> > > > > perspective,
> >> > > > > > >> so
> >> > > > > > >> > > > Polaris
> >> > > > > > >> > > > > > > > should
> >> > > > > > >> > > > > > > > >    stay flexible as a platform instead of
> >> baking
> >> > > one
> >> > > > > > >> handling
> >> > > > > > >> > > > model
> >> > > > > > >> > > > > > > into
> >> > > > > > >> > > > > > > > > the
> >> > > > > > >> > > > > > > > >    REST API behavior.
> >> > > > > > >> > > > > > > > >    - We aligned that the built-in (battery)
> >> > > behavior
> >> > > > > for
> >> > > > > > >> > > metrics
> >> > > > > > >> > > > > > > emitting
> >> > > > > > >> > > > > > > > >    can stay simple: no-op/log-only is enough
> >> as
> >> > > the
> >> > > > > > >> default.
> >> > > > > > >> > > > > > > > >    - Durable metrics persistence should be
> >> > > treated as
> >> > > > > an
> >> > > > > > >> > > > > > implementation
> >> > > > > > >> > > > > > > > of
> >> > > > > > >> > > > > > > > >    the metrics reporting path, not as the
> >> core
> >> > SPI
> >> > > > > > >> boundary.
> >> > > > > > >> > > > > > > > >    - The existing durable metrics work can be
> >> > > reviewed
> >> > > > > > as
> >> > > > > > >> a
> >> > > > > > >> > > > > reference
> >> > > > > > >> > > > > > > > >    implementation of the reporting SPI, with
> >> > > > > > >> > > persistence-related
> >> > > > > > >> > > > > > logic
> >> > > > > > >> > > > > > > > kept
> >> > > > > > >> > > > > > > > >    self-contained under a metrics durable
> >> > > > > implementation
> >> > > > > > >> > module
> >> > > > > > >> > > > > > rather
> >> > > > > > >> > > > > > > > than
> >> > > > > > >> > > > > > > > >    scattered through core entity persistence.
> >> > > > > > >> > > > > > > > >    - Dashboard/insights remains a real use
> >> case,
> >> > > but
> >> > > > > we
> >> > > > > > >> > agreed
> >> > > > > > >> > > to
> >> > > > > > >> > > > > > keep
> >> > > > > > >> > > > > > > it
> >> > > > > > >> > > > > > > > >    separate from the core metrics intake
> >> > > discussion
> >> > > > > for
> >> > > > > > >> now.
> >> > > > > > >> > > > > > > > >
> >> > > > > > >> > > > > > > > > I also did a quick source check after the
> >> > meeting
> >> > > to
> >> > > > > > make
> >> > > > > > >> > sure
> >> > > > > > >> > > we
> >> > > > > > >> > > > > are
> >> > > > > > >> > > > > > > > > describing the current state accurately.
> >> > > > > > >> > > > > > > > >
> >> > > > > > >> > > > > > > > > Current state:
> >> > > > > > >> > > > > > > > >
> >> > > > > > >> > > > > > > > >    - Polaris already has a metrics reporting
> >> > hook:
> >> > > > > > >> > > > > > > > PolarisMetricsReporter.
> >> > > > > > >> > > > > > > > >    - The default implementation is
> >> > > > > > DefaultMetricsReporter,
> >> > > > > > >> > > > selected
> >> > > > > > >> > > > > > by
> >> > > > > > >> > > > > > > > >
> >> > > polaris.iceberg-metrics.reporting.type=default. It
> >> > > > > is
> >> > > > > > >> > > > log-only,
> >> > > > > > >> > > > > > > > >    effectively quiet unless metrics logging
> >> is
> >> > > > > enabled.
> >> > > > > > >> > > > > > > > >    - There is also a
> >> PersistingMetricsReporter,
> >> > > > > selected
> >> > > > > > >> by
> >> > > > > > >> > > > > > > > >
> >> > > polaris.iceberg-metrics.reporting.type=persisting,
> >> > > > > > >> which
> >> > > > > > >> > > > > converts
> >> > > > > > >> > > > > > > > >    Iceberg scan/commit reports into Polaris
> >> > > metrics
> >> > > > > > >> records
> >> > > > > > >> > and
> >> > > > > > >> > > > > > writes
> >> > > > > > >> > > > > > > > > through
> >> > > > > > >> > > > > > > > >    PolarisMetricsManager ->
> >> MetricsPersistence.
> >> > > > > > >> > > > > > > > >    - MetricsPersistence currently exists in
> >> the
> >> > > > > > >> persistence
> >> > > > > > >> > > layer
> >> > > > > > >> > > > > and
> >> > > > > > >> > > > > > > is
> >> > > > > > >> > > > > > > > >    inherited by BasePersistence.
> >> > > > > > >> > > > > > > > >
> >> > > > > > >> > > > > > > > > My read from the discussion is that the
> >> target
> >> > > > > boundary
> >> > > > > > >> > should
> >> > > > > > >> > > > not
> >> > > > > > >> > > > > be
> >> > > > > > >> > > > > > > > > MetricsPersistence as inherited by
> >> > > BasePersistence.
> >> > > > > That
> >> > > > > > >> path
> >> > > > > > >> > > > > should
> >> > > > > > >> > > > > > be
> >> > > > > > >> > > > > > > > > decomposed as durable implementation detail
> >> > (taken
> >> > > > > care
> >> > > > > > >> of by
> >> > > > > > >> > > > > > > > >
> >> >
> >> https://urldefense.com/v3/__https://github.com/apache/polaris/pull/4397__;!!Iz9xO38YGHZK!4rnlVIqBQQQ6xpGe1PMV58edOS_cGrKoqdfhw4qSp_JQCavvEjaxXk0OwGPnxyEcO60AzKQK_XDtp7H3DdkFXcrXuu5jefU$
> >> ).
> >> > The
> >> > > > > > stable
> >> > > > > > >> > > > extension
> >> > > > > > >> > > > > > > point
> >> > > > > > >> > > > > > > > > should instead be the metrics
> >> reporting/emitting
> >> > > > > > boundary.
> >> > > > > > >> > > > > > > > >
> >> > > > > > >> > > > > > > > > Concretely, I think the next proposal should
> >> be
> >> > > shaped
> >> > > > > > >> like
> >> > > > > > >> > > this:
> >> > > > > > >> > > > > > > > >
> >> > > > > > >> > > > > > > > >    1.
> >> > > > > > >> > > > > > > > >
> >> > > > > > >> > > > > > > > >    Define the proper metrics reporting SPI
> >> at an
> >> > > > > > optional
> >> > > > > > >> > > Iceberg
> >> > > > > > >> > > > > > > metrics
> >> > > > > > >> > > > > > > > >    extension API layer.
> >> > > > > > >> > > > > > > > >
> >> > > > > > >> > > > > > > > >    Example direction:
> >> > > > > > >> > > > > > > > >
> >> > > > > > >> > > > > > > > >    public interface IcebergMetricsReporter {
> >> > > > > > >> > > > > > > > >      void
> >> > reportMetric(IcebergMetricsReportContext
> >> > > > > > >> context,
> >> > > > > > >> > > > > > > > > MetricsReport report);
> >> > > > > > >> > > > > > > > >    }
> >> > > > > > >> > > > > > > > >
> >> > > > > > >> > > > > > > > >    The context would carry the small
> >> > > Polaris-resolved
> >> > > > > > >> > envelope:
> >> > > > > > >> > > > > > > > >    catalog/table identity, report type,
> >> received
> >> > > > > > >> timestamp,
> >> > > > > > >> > and
> >> > > > > > >> > > > > > > > > request/trace
> >> > > > > > >> > > > > > > > >    context if available. The raw Iceberg
> >> > > MetricsReport
> >> > > > > > >> > remains
> >> > > > > > >> > > > the
> >> > > > > > >> > > > > > > > payload.
> >> > > > > > >> > > > > > > > >    2.
> >> > > > > > >> > > > > > > > >
> >> > > > > > >> > > > > > > > >    Keep runtime/service as the REST ingestion
> >> > and
> >> > > > > wiring
> >> > > > > > >> > layer.
> >> > > > > > >> > > > > > > > >
> >> > > > > > >> > > > > > > > >    The REST handler still resolves the table
> >> and
> >> > > > > > performs
> >> > > > > > >> > authz
> >> > > > > > >> > > > > > before
> >> > > > > > >> > > > > > > > >    accepting the report. After that, it calls
> >> > the
> >> > > > > > selected
> >> > > > > > >> > > > > reporting
> >> > > > > > >> > > > > > > > >    implementation.
> >> > > > > > >> > > > > > > > >    3.
> >> > > > > > >> > > > > > > > >
> >> > > > > > >> > > > > > > > >    Keep the battery default no-op/log-only.
> >> > > > > > >> > > > > > > > >
> >> > > > > > >> > > > > > > > >    This preserves an out-of-box safe default
> >> and
> >> > > > > avoids
> >> > > > > > >> > > > requiring a
> >> > > > > > >> > > > > > > > durable
> >> > > > > > >> > > > > > > > >    metrics store for every Polaris
> >> deployment.
> >> > > > > > >> > > > > > > > >    4.
> >> > > > > > >> > > > > > > > >
> >> > > > > > >> > > > > > > > >    Move durable JDBC metrics into a
> >> > self-contained
> >> > > > > > >> > > implementation
> >> > > > > > >> > > > > of
> >> > > > > > >> > > > > > > the
> >> > > > > > >> > > > > > > > >    reporting SPI.
> >> > > > > > >> > > > > > > > >
> >> > > > > > >> > > > > > > > >    That implementation can own its schema,
> >> > > bootstrap,
> >> > > > > > >> > > retention,
> >> > > > > > >> > > > > and
> >> > > > > > >> > > > > > > read
> >> > > > > > >> > > > > > > > >    API support. It should not define the core
> >> > > > > reporting
> >> > > > > > >> SPI
> >> > > > > > >> > > > > boundary,
> >> > > > > > >> > > > > > > and
> >> > > > > > >> > > > > > > > > it
> >> > > > > > >> > > > > > > > >    should not require metrics persistence to
> >> > > remain
> >> > > > > > >> inherited
> >> > > > > > >> > > > from
> >> > > > > > >> > > > > > > > >    BasePersistence.
> >> > > > > > >> > > > > > > > >    5.
> >> > > > > > >> > > > > > > > >
> >> > > > > > >> > > > > > > > >    Treat async/event-backed handling as
> >> another
> >> > > > > > >> > implementation
> >> > > > > > >> > > of
> >> > > > > > >> > > > > the
> >> > > > > > >> > > > > > > > same
> >> > > > > > >> > > > > > > > >    reporting SPI.
> >> > > > > > >> > > > > > > > >
> >> > > > > > >> > > > > > > > >    For example, an event-backed reporter
> >> could
> >> > > > > enqueue a
> >> > > > > > >> > > metrics
> >> > > > > > >> > > > > > event
> >> > > > > > >> > > > > > > > and
> >> > > > > > >> > > > > > > > >    let listeners handle durable storage or
> >> other
> >> > > > > sinks.
> >> > > > > > >> If we
> >> > > > > > >> > > > later
> >> > > > > > >> > > > > > > need
> >> > > > > > >> > > > > > > > a
> >> > > > > > >> > > > > > > > >    replaceable queue engine, that seems like
> >> a
> >> > > shared
> >> > > > > > >> > > > event/metrics
> >> > > > > > >> > > > > > > > > substrate
> >> > > > > > >> > > > > > > > >    topic rather than a metrics-only
> >> requirement.
> >> > > > > > >> > > > > > > > >
> >> > > > > > >> > > > > > > > > This framing lets us keep the REST metrics
> >> > > endpoint
> >> > > > > > >> simple,
> >> > > > > > >> > > > > preserve
> >> > > > > > >> > > > > > > the
> >> > > > > > >> > > > > > > > > current default behavior, support durable
> >> > metrics
> >> > > > > users,
> >> > > > > > >> and
> >> > > > > > >> > > > still
> >> > > > > > >> > > > > > > leave
> >> > > > > > >> > > > > > > > > room for async/event-backed or
> >> > > external-queue-based
> >> > > > > > >> > > > > implementations.
> >> > > > > > >> > > > > > > > >
> >> > > > > > >> > > > > > > > > I think the main follow-up questions are:
> >> > > > > > >> > > > > > > > >
> >> > > > > > >> > > > > > > > >    - Do folks agree that the stable SPI
> >> boundary
> >> > > > > should
> >> > > > > > be
> >> > > > > > >> > > > metrics
> >> > > > > > >> > > > > > > > >    reporting/emitting, not metrics
> >> persistence?
> >> > > > > > >> > > > > > > > >    - Does an optional Iceberg metrics
> >> extension
> >> > > API
> >> > > > > > layer
> >> > > > > > >> > sound
> >> > > > > > >> > > > > like
> >> > > > > > >> > > > > > > the
> >> > > > > > >> > > > > > > > >    right home for this SPI?
> >> > > > > > >> > > > > > > > >    - Should the current durable metrics work
> >> be
> >> > > > > reframed
> >> > > > > > >> as a
> >> > > > > > >> > > > > durable
> >> > > > > > >> > > > > > > > JDBC
> >> > > > > > >> > > > > > > > >    reference implementation of that SPI?
> >> > > > > > >> > > > > > > > >    - What is the smallest PR sequence to get
> >> > there
> >> > > > > > without
> >> > > > > > >> > > > blocking
> >> > > > > > >> > > > > > the
> >> > > > > > >> > > > > > > > >    current metrics work unnecessarily?
> >> > > > > > >> > > > > > > > >
> >> > > > > > >> > > > > > > > > Thanks,
> >> > > > > > >> > > > > > > > > -ej
> >> > > > > > >> > > > > > > > >
> >> > > > > > >> > > > > > > > > On Fri, May 15, 2026 at 5:16 PM Dmitri
> >> > > Bourlatchkov <
> >> > > > > > >> > > > > > dimas@apache.org>
> >> > > > > > >> > > > > > > > > wrote:
> >> > > > > > >> > > > > > > > >
> >> > > > > > >> > > > > > > > > > Hi JB,
> >> > > > > > >> > > > > > > > > >
> >> > > > > > >> > > > > > > > > > Could you set up another meeting, please?
> >> Same
> >> > > time
> >> > > > > on
> >> > > > > > >> > > > Wednesday
> >> > > > > > >> > > > > as
> >> > > > > > >> > > > > > > > last
> >> > > > > > >> > > > > > > > > > time... I hope it works for everyone.
> >> > > > > > >> > > > > > > > > >
> >> > > > > > >> > > > > > > > > > Cheers,
> >> > > > > > >> > > > > > > > > > Dmitri.
> >> > > > > > >> > > > > > > > > >
> >> > > > > > >> > > > > > > > > > On Fri, May 15, 2026 at 8:06 PM Yufei Gu <
> >> > > > > > >> > > flyrain000@gmail.com
> >> > > > > > >> > > > >
> >> > > > > > >> > > > > > > wrote:
> >> > > > > > >> > > > > > > > > >
> >> > > > > > >> > > > > > > > > > > +1 on another sync call next week.
> >> > > > > > >> > > > > > > > > > >
> >> > > > > > >> > > > > > > > > > > Yufei
> >> > > > > > >> > > > > > > > > > >
> >> > > > > > >> > > > > > > > > > >
> >> > > > > > >> > > > > > > > > > > On Fri, May 15, 2026 at 4:52 PM Dmitri
> >> > > > > Bourlatchkov
> >> > > > > > <
> >> > > > > > >> > > > > > > > dimas@apache.org>
> >> > > > > > >> > > > > > > > > > > wrote:
> >> > > > > > >> > > > > > > > > > >
> >> > > > > > >> > > > > > > > > > > > Hi All,
> >> > > > > > >> > > > > > > > > > > >
> >> > > > > > >> > > > > > > > > > > > WDYT about another sync call next week?
> >> > > > > > >> > > > > > > > > > > >
> >> > > > > > >> > > > > > > > > > > > Thanks,
> >> > > > > > >> > > > > > > > > > > > Dmitri.
> >> > > > > > >> > > > > > > > > > > >
> >> > > > > > >> > > > > > > > > > > > On Wed, May 6, 2026 at 5:29 PM Dmitri
> >> > > > > > Bourlatchkov <
> >> > > > > > >> > > > > > > > dimas@apache.org
> >> > > > > > >> > > > > > > > > >
> >> > > > > > >> > > > > > > > > > > > wrote:
> >> > > > > > >> > > > > > > > > > > >
> >> > > > > > >> > > > > > > > > > > > > Hi EJ,
> >> > > > > > >> > > > > > > > > > > > >
> >> > > > > > >> > > > > > > > > > > > > Thanks for the summary! It covers
> >> what
> >> > we
> >> > > > > > >> discussed
> >> > > > > > >> > in
> >> > > > > > >> > > > the
> >> > > > > > >> > > > > > > > meeting
> >> > > > > > >> > > > > > > > > > very
> >> > > > > > >> > > > > > > > > > > > > well, IMHO.
> >> > > > > > >> > > > > > > > > > > > >
> >> > > > > > >> > > > > > > > > > > > > Looking forward to concrete PRs :)
> >> > > > > > >> > > > > > > > > > > > >
> >> > > > > > >> > > > > > > > > > > > > Cheers,
> >> > > > > > >> > > > > > > > > > > > > Dmitri.
> >> > > > > > >> > > > > > > > > > > > >
> >> > > > > > >> > > > > > > > > > > > > On Wed, May 6, 2026 at 5:08 PM EJ
> >> Wang <
> >> > > > > > >> > > > > > > > > > ej.wang.dev.interest@gmail.com
> >> > > > > > >> > > > > > > > > > > >
> >> > > > > > >> > > > > > > > > > > > > wrote:
> >> > > > > > >> > > > > > > > > > > > >
> >> > > > > > >> > > > > > > > > > > > >> Hi folks,
> >> > > > > > >> > > > > > > > > > > > >>
> >> > > > > > >> > > > > > > > > > > > >> We had a community sync earlier,
> >> thanks
> >> > > JB
> >> > > > > for
> >> > > > > > >> > > > scheduling
> >> > > > > > >> > > > > > it.
> >> > > > > > >> > > > > > > > > Notes
> >> > > > > > >> > > > > > > > > > > from
> >> > > > > > >> > > > > > > > > > > > >> the first metrics architecture sync
> >> > (May
> >> > > 6,
> >> > > > > > >> 10-11am
> >> > > > > > >> > > PT).
> >> > > > > > >> > > > > > > > > Discussion
> >> > > > > > >> > > > > > > > > > > doc
> >> > > > > > >> > > > > > > > > > > > >> with per-section status:
> >> > > > > > >> > > > > > > > > > > > >>
> >> > > > > > >> > > > > > > > > > > > >>
> >> > > > > > >> > > > > > > > > > > >
> >> > > > > > >> > > > > > > > > > >
> >> > > > > > >> > > > > > > > > >
> >> > > > > > >> > > > > > > > >
> >> > > > > > >> > > > > > > >
> >> > > > > > >> > > > > > >
> >> > > > > > >> > > > > >
> >> > > > > > >> > > > >
> >> > > > > > >> > > >
> >> > > > > > >> > >
> >> > > > > > >> >
> >> > > > > > >>
> >> > > > > >
> >> > > > >
> >> > >
> >> >
> >> https://urldefense.com/v3/__https://docs.google.com/document/d/100h7c4damrUzVuquYbBHM0EvA4LSWuW2IT2dN_7nYVA/edit?tab=t.0__;!!Iz9xO38YGHZK!4rnlVIqBQQQ6xpGe1PMV58edOS_cGrKoqdfhw4qSp_JQCavvEjaxXk0OwGPnxyEcO60AzKQK_XDtp7H3DdkFXcrXgIddeUs$
> >> > > > > > >> > > > > > > > > > > > >>
> >> > > > > > >> > > > > > > > > > > > >> *The meeting covered both topics
> >> from
> >> > the
> >> > > > > doc.
> >> > > > > > >> > > > > > Direction-level
> >> > > > > > >> > > > > > > > > > > alignment
> >> > > > > > >> > > > > > > > > > > > >> was reached on the headline pieces;
> >> > > details
> >> > > > > > >> remain
> >> > > > > > >> > for
> >> > > > > > >> > > > PR
> >> > > > > > >> > > > > > > review
> >> > > > > > >> > > > > > > > > or
> >> > > > > > >> > > > > > > > > > > > >> follow-up sessions.*
> >> > > > > > >> > > > > > > > > > > > >>
> >> > > > > > >> > > > > > > > > > > > >> *Topic 1 — Persistence schema
> >> redesign*
> >> > > > > > >> > > > > > > > > > > > >> Idea-level alignment on
> >> consolidating
> >> > > > > per-type
> >> > > > > > >> > tables
> >> > > > > > >> > > > > > > > > > > > >> (scan_metrics_report,
> >> > > > > > >> > > > > > > > > > > > >> commit_metrics_report) into a single
> >> > > > > > >> metrics_report
> >> > > > > > >> > > > table.
> >> > > > > > >> > > > > > The
> >> > > > > > >> > > > > > > > > > > > motivating
> >> > > > > > >> > > > > > > > > > > > >> cost is the surface area added by
> >> every
> >> > > new
> >> > > > > > >> metric
> >> > > > > > >> > > type
> >> > > > > > >> > > > > > today:
> >> > > > > > >> > > > > > > > new
> >> > > > > > >> > > > > > > > > > > > table,
> >> > > > > > >> > > > > > > > > > > > >> SPI method, record class, model,
> >> > > converter,
> >> > > > > > >> schema
> >> > > > > > >> > > > > > migration.
> >> > > > > > >> > > > > > > > > > > > >>
> >> > > > > > >> > > > > > > > > > > > >> Most schema details are deferred to
> >> the
> >> > > > > schema
> >> > > > > > >> PR. A
> >> > > > > > >> > > few
> >> > > > > > >> > > > > > > > specific
> >> > > > > > >> > > > > > > > > > > points
> >> > > > > > >> > > > > > > > > > > > >> came up:
> >> > > > > > >> > > > > > > > > > > > >> •   metric_schema_version: Yufei
> >> > prefers
> >> > > > > > dropping
> >> > > > > > >> > it,
> >> > > > > > >> > > > > since
> >> > > > > > >> > > > > > > > there
> >> > > > > > >> > > > > > > > > is
> >> > > > > > >> > > > > > > > > > > no
> >> > > > > > >> > > > > > > > > > > > >> spec-level concept of metrics
> >> > versioning
> >> > > > > today
> >> > > > > > >> and
> >> > > > > > >> > it
> >> > > > > > >> > > is
> >> > > > > > >> > > > > > hard
> >> > > > > > >> > > > > > > to
> >> > > > > > >> > > > > > > > > > > define
> >> > > > > > >> > > > > > > > > > > > >> unilaterally. Robert prefers keeping
> >> > it,
> >> > > > > given
> >> > > > > > >> IRC
> >> > > > > > >> > v2
> >> > > > > > >> > > is
> >> > > > > > >> > > > > > > coming
> >> > > > > > >> > > > > > > > > and
> >> > > > > > >> > > > > > > > > > > the
> >> > > > > > >> > > > > > > > > > > > >> schema should be considered against
> >> its
> >> > > > > likely
> >> > > > > > >> > shape;
> >> > > > > > >> > > > > Robert
> >> > > > > > >> > > > > > > > also
> >> > > > > > >> > > > > > > > > > > raised
> >> > > > > > >> > > > > > > > > > > > >> how to differentiate various payload
> >> > > formats
> >> > > > > if
> >> > > > > > >> any.
> >> > > > > > >> > > > EJ's
> >> > > > > > >> > > > > > read
> >> > > > > > >> > > > > > > > is
> >> > > > > > >> > > > > > > > > > that
> >> > > > > > >> > > > > > > > > > > > >> this
> >> > > > > > >> > > > > > > > > > > > >> is a two-way-door decision. We can
> >> > start
> >> > > > > > without
> >> > > > > > >> the
> >> > > > > > >> > > > > field,
> >> > > > > > >> > > > > > > and
> >> > > > > > >> > > > > > > > if
> >> > > > > > >> > > > > > > > > > IRC
> >> > > > > > >> > > > > > > > > > > > v2
> >> > > > > > >> > > > > > > > > > > > >> changes the shape we would likely
> >> roll
> >> > a
> >> > > > > > >> > corresponding
> >> > > > > > >> > > > new
> >> > > > > > >> > > > > > > > schema
> >> > > > > > >> > > > > > > > > > > > anyway,
> >> > > > > > >> > > > > > > > > > > > >> which is not particularly costly.
> >> > > > > > >> > > > > > > > > > > > >> •   Payload format: Robert pointed
> >> out
> >> > > that
> >> > > > > > >> future
> >> > > > > > >> > > > formats
> >> > > > > > >> > > > > > > > beyond
> >> > > > > > >> > > > > > > > > > JSON
> >> > > > > > >> > > > > > > > > > > > may
> >> > > > > > >> > > > > > > > > > > > >> be worth supporting. The exact
> >> shape is
> >> > > > > > deferred
> >> > > > > > >> to
> >> > > > > > >> > > the
> >> > > > > > >> > > > > > schema
> >> > > > > > >> > > > > > > > > > > > discussion.
> >> > > > > > >> > > > > > > > > > > > >> •   Partition strategy: Anand
> >> suggested
> >> > > > > monthly
> >> > > > > > >> > > > > partitioning
> >> > > > > > >> > > > > > > > based
> >> > > > > > >> > > > > > > > > > on
> >> > > > > > >> > > > > > > > > > > > his
> >> > > > > > >> > > > > > > > > > > > >> experience as potentially helpful at
> >> > > scale.
> >> > > > > > >> > > > > > > > > > > > >>
> >> > > > > > >> > > > > > > > > > > > >> *Topic 2 — Where metrics ingestion
> >> and
> >> > > > > storage
> >> > > > > > >> > belong*
> >> > > > > > >> > > > > > > > > > > > >> Idea-level alignment that metrics
> >> > should
> >> > > be a
> >> > > > > > >> > > separated
> >> > > > > > >> > > > > SPI
> >> > > > > > >> > > > > > > from
> >> > > > > > >> > > > > > > > > the
> >> > > > > > >> > > > > > > > > > > > >> entity
> >> > > > > > >> > > > > > > > > > > > >> persistence stack. Two reasons
> >> > surfaced:
> >> > > (a)
> >> > > > > > >> > workloads
> >> > > > > > >> > > > and
> >> > > > > > >> > > > > > > > > > capability
> >> > > > > > >> > > > > > > > > > > > >> requirements diverge enough that
> >> > coupling
> >> > > > > them
> >> > > > > > >> > creates
> >> > > > > > >> > > > > > > > artificial
> >> > > > > > >> > > > > > > > > > > > >> constraints, and (b) admin
> >> experience
> >> > > > > improves
> >> > > > > > >> when
> >> > > > > > >> > > > > metrics
> >> > > > > > >> > > > > > > has
> >> > > > > > >> > > > > > > > > its
> >> > > > > > >> > > > > > > > > > > own
> >> > > > > > >> > > > > > > > > > > > >> bootstrap, retention, and lifecycle.
> >> > > Dmitri
> >> > > > > > noted
> >> > > > > > >> > > > Polaris
> >> > > > > > >> > > > > > > being
> >> > > > > > >> > > > > > > > a
> >> > > > > > >> > > > > > > > > > > > platform
> >> > > > > > >> > > > > > > > > > > > >> should have the flexibility to
> >> support
> >> > > > > > different
> >> > > > > > >> > > > > persistence
> >> > > > > > >> > > > > > > > > > backends
> >> > > > > > >> > > > > > > > > > > > per
> >> > > > > > >> > > > > > > > > > > > >> concern, and pointed to a concrete
> >> next
> >> > > step
> >> > > > > of
> >> > > > > > >> > > > separating
> >> > > > > > >> > > > > > the
> >> > > > > > >> > > > > > > > > JDBC
> >> > > > > > >> > > > > > > > > > > > >> bootstrap for metrics from the
> >> > metastore
> >> > > > > > >> bootstrap.
> >> > > > > > >> > > > Robert
> >> > > > > > >> > > > > > > > > proposed
> >> > > > > > >> > > > > > > > > > an
> >> > > > > > >> > > > > > > > > > > > >> additional UX extension: detect an
> >> > > > > > unbootstrapped
> >> > > > > > >> > > > metrics
> >> > > > > > >> > > > > > > store
> >> > > > > > >> > > > > > > > on
> >> > > > > > >> > > > > > > > > > > first
> >> > > > > > >> > > > > > > > > > > > >> use and auto-bootstrap rather than
> >> > > requiring
> >> > > > > an
> >> > > > > > >> > > explicit
> >> > > > > > >> > > > > > > manual
> >> > > > > > >> > > > > > > > > > > > bootstrap
> >> > > > > > >> > > > > > > > > > > > >> step.
> >> > > > > > >> > > > > > > > > > > > >> The meeting also confirmed that
> >> Polaris
> >> > > > > metrics
> >> > > > > > >> can
> >> > > > > > >> > > > start
> >> > > > > > >> > > > > > > small
> >> > > > > > >> > > > > > > > > and
> >> > > > > > >> > > > > > > > > > > stay
> >> > > > > > >> > > > > > > > > > > > >> Iceberg-focused. Naming and
> >> persistence
> >> > > > > schema
> >> > > > > > >> can
> >> > > > > > >> > > lean
> >> > > > > > >> > > > > > > > > > > > Iceberg-specific.
> >> > > > > > >> > > > > > > > > > > > >> If a future expansion to
> >> generic-table
> >> > > > > metrics
> >> > > > > > or
> >> > > > > > >> > > > > > operational
> >> > > > > > >> > > > > > > > > > metrics
> >> > > > > > >> > > > > > > > > > > > >> arrives, an abstraction layer can be
> >> > > built on
> >> > > > > > >> top of
> >> > > > > > >> > > the
> >> > > > > > >> > > > > > > Iceberg
> >> > > > > > >> > > > > > > > > > > metrics
> >> > > > > > >> > > > > > > > > > > > >> reporter at that point. Robert
> >> remains
> >> > > on the
> >> > > > > > >> fence
> >> > > > > > >> > > and
> >> > > > > > >> > > > > > would
> >> > > > > > >> > > > > > > > > prefer
> >> > > > > > >> > > > > > > > > > > > >> something more generic but did not
> >> > block
> >> > > the
> >> > > > > > >> > > direction;
> >> > > > > > >> > > > > > > Dmitri's
> >> > > > > > >> > > > > > > > > > read
> >> > > > > > >> > > > > > > > > > > > was
> >> > > > > > >> > > > > > > > > > > > >> that the proposed framework already
> >> has
> >> > > > > enough
> >> > > > > > >> > > > flexibility
> >> > > > > > >> > > > > > to
> >> > > > > > >> > > > > > > > > absorb
> >> > > > > > >> > > > > > > > > > > > >> future
> >> > > > > > >> > > > > > > > > > > > >> expansion.
> >> > > > > > >> > > > > > > > > > > > >>
> >> > > > > > >> > > > > > > > > > > > >> The Trade-offs and Proposed
> >> structure
> >> > > > > sections
> >> > > > > > in
> >> > > > > > >> > the
> >> > > > > > >> > > > doc
> >> > > > > > >> > > > > > were
> >> > > > > > >> > > > > > > > not
> >> > > > > > >> > > > > > > > > > > > >> reviewed
> >> > > > > > >> > > > > > > > > > > > >> in detail. They remain open for
> >> either
> >> > > the
> >> > > > > next
> >> > > > > > >> sync
> >> > > > > > >> > > or
> >> > > > > > >> > > > PR
> >> > > > > > >> > > > > > > > review.
> >> > > > > > >> > > > > > > > > > > > >>
> >> > > > > > >> > > > > > > > > > > > >> *Cross-cutting alignment —
> >> > > battery-included
> >> > > > > > plus
> >> > > > > > >> > > > > pluggable*
> >> > > > > > >> > > > > > > > > > > > >> A common philosophy emerged from the
> >> > > > > > discussion.
> >> > > > > > >> EJ
> >> > > > > > >> > > > > > summarized
> >> > > > > > >> > > > > > > > it
> >> > > > > > >> > > > > > > > > > as:
> >> > > > > > >> > > > > > > > > > > > >> Polaris should provide a
> >> > > battery-included UX
> >> > > > > > for
> >> > > > > > >> > > > beginners
> >> > > > > > >> > > > > > and
> >> > > > > > >> > > > > > > > the
> >> > > > > > >> > > > > > > > > > > > >> flexibility for advanced users to
> >> swap
> >> > > the
> >> > > > > > >> included
> >> > > > > > >> > > > > battery
> >> > > > > > >> > > > > > > for
> >> > > > > > >> > > > > > > > > > > > something
> >> > > > > > >> > > > > > > > > > > > >> more powerful or tailored to their
> >> use
> >> > > case.
> >> > > > > > The
> >> > > > > > >> SPI
> >> > > > > > >> > > > > design
> >> > > > > > >> > > > > > > > needs
> >> > > > > > >> > > > > > > > > to
> >> > > > > > >> > > > > > > > > > > > >> enable
> >> > > > > > >> > > > > > > > > > > > >> both.
> >> > > > > > >> > > > > > > > > > > > >>
> >> > > > > > >> > > > > > > > > > > > >> The inputs that shaped this framing:
> >> > > > > > >> > > > > > > > > > > > >> •   Anand described how his team
> >> uses
> >> > the
> >> > > > > > current
> >> > > > > > >> > > > metrics
> >> > > > > > >> > > > > > > > > > persistence
> >> > > > > > >> > > > > > > > > > > > >> (three metrics consumers in v1.4).
> >> > > > > > >> > > > > > > > > > > > >> •   Yufei raised Grafana and
> >> dashboard
> >> > > > > > >> integrations
> >> > > > > > >> > > as a
> >> > > > > > >> > > > > > > > > destination
> >> > > > > > >> > > > > > > > > > > use
> >> > > > > > >> > > > > > > > > > > > >> case beyond the default.
> >> > > > > > >> > > > > > > > > > > > >> •   Robert called out that the
> >> current
> >> > > design
> >> > > > > > is
> >> > > > > > >> > more
> >> > > > > > >> > > > > > > > > JDBC-focused.
> >> > > > > > >> > > > > > > > > > > > >>
> >> > > > > > >> > > > > > > > > > > > >> Two concrete instances:
> >> > > > > > >> > > > > > > > > > > > >> •   Async metrics intake: Yufei's
> >> > initial
> >> > > > > > >> position
> >> > > > > > >> > was
> >> > > > > > >> > > > > that
> >> > > > > > >> > > > > > > > async
> >> > > > > > >> > > > > > > > > > > should
> >> > > > > > >> > > > > > > > > > > > >> largely live on the producer side
> >> and
> >> > > there
> >> > > > > is
> >> > > > > > >> not
> >> > > > > > >> > > much
> >> > > > > > >> > > > > > > Polaris
> >> > > > > > >> > > > > > > > > can
> >> > > > > > >> > > > > > > > > > > do.
> >> > > > > > >> > > > > > > > > > > > >> Robert suggested a Polaris-side
> >> default
> >> > > is
> >> > > > > > doable
> >> > > > > > >> > via
> >> > > > > > >> > > > > > Vert.x.
> >> > > > > > >> > > > > > > > > Dmitri
> >> > > > > > >> > > > > > > > > > > > >> agreed
> >> > > > > > >> > > > > > > > > > > > >> the direction is worth exploring.
> >> The
> >> > > meeting
> >> > > > > > >> > > converged
> >> > > > > > >> > > > > on a
> >> > > > > > >> > > > > > > > > > > > >> battery-included default (likely
> >> > > > > Vert.x-backed)
> >> > > > > > >> with
> >> > > > > > >> > > an
> >> > > > > > >> > > > > SPI
> >> > > > > > >> > > > > > > > shape
> >> > > > > > >> > > > > > > > > > that
> >> > > > > > >> > > > > > > > > > > > >> lets
> >> > > > > > >> > > > > > > > > > > > >> power users route to a more scalable
> >> > > backend
> >> > > > > > >> > > (k8s-hosted
> >> > > > > > >> > > > > > > queue,
> >> > > > > > >> > > > > > > > > AWS
> >> > > > > > >> > > > > > > > > > > SQS,
> >> > > > > > >> > > > > > > > > > > > >> etc.).
> >> > > > > > >> > > > > > > > > > > > >> •   Pluggable destinations:
> >> combining
> >> > > Yufei's
> >> > > > > > >> > > dashboard
> >> > > > > > >> > > > > use
> >> > > > > > >> > > > > > > case
> >> > > > > > >> > > > > > > > > > with
> >> > > > > > >> > > > > > > > > > > > >> Robert's JDBC-focused call-out, the
> >> > > meeting
> >> > > > > > >> agreed
> >> > > > > > >> > the
> >> > > > > > >> > > > SPI
> >> > > > > > >> > > > > > > > should
> >> > > > > > >> > > > > > > > > be
> >> > > > > > >> > > > > > > > > > > > >> structured for multiple sinks so
> >> > > integrations
> >> > > > > > >> become
> >> > > > > > >> > > > impl
> >> > > > > > >> > > > > > > > choices
> >> > > > > > >> > > > > > > > > > > rather
> >> > > > > > >> > > > > > > > > > > > >> than architectural changes.
> >> > > > > > >> > > > > > > > > > > > >>
> >> > > > > > >> > > > > > > > > > > > >> The battery-included default is most
> >> > > likely
> >> > > > > to
> >> > > > > > >> use
> >> > > > > > >> > the
> >> > > > > > >> > > > > > > existing
> >> > > > > > >> > > > > > > > > > > > >> JDBC-backed
> >> > > > > > >> > > > > > > > > > > > >> approach.
> >> > > > > > >> > > > > > > > > > > > >>
> >> > > > > > >> > > > > > > > > > > > >> *Direction (idea-level alignment)*
> >> > > > > > >> > > > > > > > > > > > >> •   Single metrics_report table
> >> > > consolidating
> >> > > > > > >> > per-type
> >> > > > > > >> > > > > > > metrics,
> >> > > > > > >> > > > > > > > > > > > replacing
> >> > > > > > >> > > > > > > > > > > > >> scan_metrics_report and
> >> > > commit_metrics_report
> >> > > > > > >> > > > > > > > > > > > >> •   Iceberg-focused naming and
> >> schema
> >> > for
> >> > > > > now,
> >> > > > > > >> > revisit
> >> > > > > > >> > > > if
> >> > > > > > >> > > > > > > > > > > generic-table
> >> > > > > > >> > > > > > > > > > > > or
> >> > > > > > >> > > > > > > > > > > > >> operational metrics arrive
> >> > > > > > >> > > > > > > > > > > > >> •   Metrics persistence as a
> >> separated
> >> > > SPI,
> >> > > > > not
> >> > > > > > >> on
> >> > > > > > >> > > > > > > > BasePersistence
> >> > > > > > >> > > > > > > > > > > > >> •   Bootstrap path separated for
> >> > metrics,
> >> > > > > > >> > independent
> >> > > > > > >> > > of
> >> > > > > > >> > > > > > > > metastore
> >> > > > > > >> > > > > > > > > > > > >> bootstrap
> >> > > > > > >> > > > > > > > > > > > >> •   "Battery-included plus
> >> pluggable"
> >> > as
> >> > > the
> >> > > > > > SPI
> >> > > > > > >> > > design
> >> > > > > > >> > > > > > > > philosophy
> >> > > > > > >> > > > > > > > > > > > >>
> >> > > > > > >> > > > > > > > > > > > >> *Open items*
> >> > > > > > >> > > > > > > > > > > > >> •   Schema details:
> >> > > metric_schema_version,
> >> > > > > > >> payload
> >> > > > > > >> > > > format,
> >> > > > > > >> > > > > > IRC
> >> > > > > > >> > > > > > > > v2
> >> > > > > > >> > > > > > > > > > > > >> forward-compat shape
> >> > > > > > >> > > > > > > > > > > > >> •   SPI design details — full review
> >> > > either
> >> > > > > in
> >> > > > > > >> the
> >> > > > > > >> > > next
> >> > > > > > >> > > > > sync
> >> > > > > > >> > > > > > > or
> >> > > > > > >> > > > > > > > in
> >> > > > > > >> > > > > > > > > > the
> >> > > > > > >> > > > > > > > > > > > >> corresponding PR
> >> > > > > > >> > > > > > > > > > > > >> •   Schema refactor PR ownership
> >> > > > > > >> > > > > > > > > > > > >>
> >> > > > > > >> > > > > > > > > > > > >> *Action items*
> >> > > > > > >> > > > > > > > > > > > >> •   EJ to take a first stab at the
> >> SPI
> >> > > design
> >> > > > > > and
> >> > > > > > >> > > > > > potentially
> >> > > > > > >> > > > > > > > > > partner
> >> > > > > > >> > > > > > > > > > > > with
> >> > > > > > >> > > > > > > > > > > > >> Anand to incorporate the lessons
> >> > learned
> >> > > from
> >> > > > > > the
> >> > > > > > >> > > > existing
> >> > > > > > >> > > > > > > > > reporter
> >> > > > > > >> > > > > > > > > > > and
> >> > > > > > >> > > > > > > > > > > > >> persistence work.
> >> > > > > > >> > > > > > > > > > > > >> •   Schema refactor PR ownership is
> >> not
> >> > > yet
> >> > > > > > >> decided.
> >> > > > > > >> > > If
> >> > > > > > >> > > > > > anyone
> >> > > > > > >> > > > > > > > is
> >> > > > > > >> > > > > > > > > > > > >> interested in driving it, reply on
> >> this
> >> > > > > thread.
> >> > > > > > >> > > > > > > > > > > > >> •   JB to schedule the next sync,
> >> > > tentatively
> >> > > > > > in
> >> > > > > > >> two
> >> > > > > > >> > > > > weeks.
> >> > > > > > >> > > > > > > > > > > > >>
> >> > > > > > >> > > > > > > > > > > > >> -ej
> >> > > > > > >> > > > > > > > > > > > >>
> >> > > > > > >> > > > > > > > > > > > >> On Mon, Apr 27, 2026 at 3:07 PM EJ
> >> > Wang <
> >> > > > > > >> > > > > > > > > > > ej.wang.dev.interest@gmail.com
> >> > > > > > >> > > > > > > > > > > > >
> >> > > > > > >> > > > > > > > > > > > >> wrote:
> >> > > > > > >> > > > > > > > > > > > >>
> >> > > > > > >> > > > > > > > > > > > >> > Thanks Yufei for the +1.
> >> > > > > > >> > > > > > > > > > > > >> >
> >> > > > > > >> > > > > > > > > > > > >> > JB, could you help add a biweekly
> >> > > metrics
> >> > > > > > >> > > architecture
> >> > > > > > >> > > > > > sync
> >> > > > > > >> > > > > > > to
> >> > > > > > >> > > > > > > > > the
> >> > > > > > >> > > > > > > > > > > > >> Polaris
> >> > > > > > >> > > > > > > > > > > > >> > community calendar? I'm thinking
> >> > > Thursdays
> >> > > > > at
> >> > > > > > >> > 9-10am
> >> > > > > > >> > > > PT,
> >> > > > > > >> > > > > > on
> >> > > > > > >> > > > > > > > the
> >> > > > > > >> > > > > > > > > > > > >> off-weeks
> >> > > > > > >> > > > > > > > > > > > >> > from the community meeting
> >> (starting
> >> > > May
> >> > > > > 7),
> >> > > > > > 60
> >> > > > > > >> > > > minutes.
> >> > > > > > >> > > > > > > > > > > > >> >
> >> > > > > > >> > > > > > > > > > > > >> > Here's a rough agenda to work
> >> through
> >> > > over
> >> > > > > > the
> >> > > > > > >> > first
> >> > > > > > >> > > > few
> >> > > > > > >> > > > > > > > > sessions,
> >> > > > > > >> > > > > > > > > > > > >> grouped
> >> > > > > > >> > > > > > > > > > > > >> > by priority:
> >> > > > > > >> > > > > > > > > > > > >> >
> >> > > > > > >> > > > > > > > > > > > >> > *First: foundational direction*
> >> > > > > > >> > > > > > > > > > > > >> >
> >> > > > > > >> > > > > > > > > > > > >> > 1.  MetricsPersistence: public
> >> SPI or
> >> > > > > > internal
> >> > > > > > >> > > > > > > implementation
> >> > > > > > >> > > > > > > > > > > detail?
> >> > > > > > >> > > > > > > > > > > > >> >    •   Marked @Beta, javadoc calls
> >> > it a
> >> > > > > > >> "Service
> >> > > > > > >> > > > > Provider
> >> > > > > > >> > > > > > > > > > > Interface",
> >> > > > > > >> > > > > > > > > > > > >> but
> >> > > > > > >> > > > > > > > > > > > >> > only one consumer
> >> > > > > (JdbcBasePersistenceImpl),
> >> > > > > > >> lives
> >> > > > > > >> > > on
> >> > > > > > >> > > > > > > > > > > BasePersistence.
> >> > > > > > >> > > > > > > > > > > > >> If
> >> > > > > > >> > > > > > > > > > > > >> > demoted to a private helper
> >> inside a
> >> > > > > > persisting
> >> > > > > > >> > > > reporter
> >> > > > > > >> > > > > > > impl,
> >> > > > > > >> > > > > > > > > > most
> >> > > > > > >> > > > > > > > > > > > >> > downstream design decisions become
> >> > > > > > >> implementation
> >> > > > > > >> > > > > details
> >> > > > > > >> > > > > > > > rather
> >> > > > > > >> > > > > > > > > > > than
> >> > > > > > >> > > > > > > > > > > > >> > contract questions.
> >> > > > > > >> > > > > > > > > > > > >> >
> >> > > > > > >> > > > > > > > > > > > >> > 2.  Persistence schema redesign
> >> > > > > > >> > > > > > > > > > > > >> >    •   Current two-table layout
> >> > > > > > >> > > (scan_metrics_report,
> >> > > > > > >> > > > > > > > > > > > >> > commit_metrics_report) with ~25
> >> > > flattened
> >> > > > > > >> columns
> >> > > > > > >> > > > each.
> >> > > > > > >> > > > > > > Every
> >> > > > > > >> > > > > > > > > new
> >> > > > > > >> > > > > > > > > > > > metric
> >> > > > > > >> > > > > > > > > > > > >> > type requires a new table, SPI
> >> > method,
> >> > > > > record
> >> > > > > > >> > class,
> >> > > > > > >> > > > > > model,
> >> > > > > > >> > > > > > > > > > > converter,
> >> > > > > > >> > > > > > > > > > > > >> and
> >> > > > > > >> > > > > > > > > > > > >> > schema migration. Direction to
> >> > explore:
> >> > > > > > single
> >> > > > > > >> > table
> >> > > > > > >> > > > > with
> >> > > > > > >> > > > > > > > > > > metric_type
> >> > > > > > >> > > > > > > > > > > > >> enum,
> >> > > > > > >> > > > > > > > > > > > >> > schema_version, and JSON payload
> >> > > column.
> >> > > > > > >> > > > > > > > > > > > >> >
> >> > > > > > >> > > > > > > > > > > > >> > *Second: design details once
> >> > direction
> >> > > is
> >> > > > > > set*
> >> > > > > > >> > > > > > > > > > > > >> >
> >> > > > > > >> > > > > > > > > > > > >> > 3.  Partition key strategy
> >> > > > > > >> > > > > > > > > > > > >> >    •   Single-table design means
> >> scan
> >> > > > > metrics
> >> > > > > > >> at
> >> > > > > > >> > > scale
> >> > > > > > >> > > > > > will
> >> > > > > > >> > > > > > > > have
> >> > > > > > >> > > > > > > > > > > high
> >> > > > > > >> > > > > > > > > > > > >> > write concurrency per table.
> >> Schema
> >> > > needs
> >> > > > > to
> >> > > > > > >> > expose
> >> > > > > > >> > > > > enough
> >> > > > > > >> > > > > > > > > > structure
> >> > > > > > >> > > > > > > > > > > > for
> >> > > > > > >> > > > > > > > > > > > >> > backends to shard by entity or
> >> time
> >> > > range.
> >> > > > > > >> > > > > > > > > > > > >> >
> >> > > > > > >> > > > > > > > > > > > >> > 4.  Read/write path consistency
> >> > > > > > >> > > > > > > > > > > > >> >    •   Writes go through
> >> > > > > > PolarisMetricsManager
> >> > > > > > >> on
> >> > > > > > >> > > > > > > > > > MetaStoreManager.
> >> > > > > > >> > > > > > > > > > > > >> Reads
> >> > > > > > >> > > > > > > > > > > > >> > bypass MetaStoreManager and go
> >> > > straight to
> >> > > > > > >> > > > > > BasePersistence,
> >> > > > > > >> > > > > > > > > > > excluding
> >> > > > > > >> > > > > > > > > > > > >> > non-JDBC backends from the read
> >> API.
> >> > > > > > >> > > > > > > > > > > > >> >
> >> > > > > > >> > > > > > > > > > > > >> > *Third: cleanup and alignment*
> >> > > > > > >> > > > > > > > > > > > >> >
> >> > > > > > >> > > > > > > > > > > > >> > 5.  PolarisMetricsReporter naming
> >> > > > > > >> > > > > > > > > > > > >> >    •   Only handles IRC
> >> > > > > > >> (ScanReport/CommitReport),
> >> > > > > > >> > > > > doesn't
> >> > > > > > >> > > > > > > > cover
> >> > > > > > >> > > > > > > > > > > > generic
> >> > > > > > >> > > > > > > > > > > > >> > tables or operational metrics.
> >> Name
> >> > is
> >> > > > > > broader
> >> > > > > > >> > than
> >> > > > > > >> > > > > scope.
> >> > > > > > >> > > > > > > > > > > > >> >
> >> > > > > > >> > > > > > > > > > > > >> > 6.  PolarisMetricsManager facade
> >> > > > > passthrough
> >> > > > > > >> > > > > > > > > > > > >> >    •   Entire default method is
> >> > > > > > >> > > > > > > > > > > > >>
> >> > callCtx.getMetaStore().writeScanReport().
> >> > > > > > >> > > > > > > > > > > > >> > Zero logic, passes Level 1
> >> straight
> >> > > through
> >> > > > > > to
> >> > > > > > >> > Level
> >> > > > > > >> > > > 3.
> >> > > > > > >> > > > > > Same
> >> > > > > > >> > > > > > > > > > > > >> anti-pattern
> >> > > > > > >> > > > > > > > > > > > >> > as PolarisEventManager.
> >> > > > > > >> > > > > > > > > > > > >> >
> >> > > > > > >> > > > > > > > > > > > >> > 7.  Iceberg community alignment
> >> > > > > > >> > > > > > > > > > > > >> >    •   Payload-type extension
> >> needs
> >> > > > > > discussion
> >> > > > > > >> on
> >> > > > > > >> > > > > > > dev@iceberg.
> >> > > > > > >> > > > > > > > > > > > >> obelix74's
> >> > > > > > >> > > > > > > > > > > > >> > Feb thread got zero replies.
> >> Needs a
> >> > > > > > committer
> >> > > > > > >> > > voice.
> >> > > > > > >> > > > > > > > > > > > >> >
> >> > > > > > >> > > > > > > > > > > > >> > Lets confirm prioritization in the
> >> > > first
> >> > > > > > >> session.
> >> > > > > > >> > > > > > > > > > > > >> >
> >> > > > > > >> > > > > > > > > > > > >> > -ej
> >> > > > > > >> > > > > > > > > > > > >> >
> >> > > > > > >> > > > > > > > > > > > >> > On Tue, Apr 21, 2026 at 3:18 PM
> >> Yufei
> >> > > Gu <
> >> > > > > > >> > > > > > > > flyrain000@gmail.com>
> >> > > > > > >> > > > > > > > > > > > wrote:
> >> > > > > > >> > > > > > > > > > > > >> >
> >> > > > > > >> > > > > > > > > > > > >> >> Thanks everyone for continuing to
> >> > > drive
> >> > > > > this
> >> > > > > > >> > > > forward. I
> >> > > > > > >> > > > > > > agree
> >> > > > > > >> > > > > > > > > > that
> >> > > > > > >> > > > > > > > > > > > the
> >> > > > > > >> > > > > > > > > > > > >> >> problem is getting complex enough
> >> > > that a
> >> > > > > > more
> >> > > > > > >> > > > > structured
> >> > > > > > >> > > > > > > > > > discussion
> >> > > > > > >> > > > > > > > > > > > >> would
> >> > > > > > >> > > > > > > > > > > > >> >> help.
> >> > > > > > >> > > > > > > > > > > > >> >>
> >> > > > > > >> > > > > > > > > > > > >> >> +1 on setting up a biweekly sync
> >> for
> >> > > the
> >> > > > > > >> metrics
> >> > > > > > >> > > > > > > > architecture.
> >> > > > > > >> > > > > > > > > > I’m
> >> > > > > > >> > > > > > > > > > > > >> happy
> >> > > > > > >> > > > > > > > > > > > >> >> to
> >> > > > > > >> > > > > > > > > > > > >> >> join.
> >> > > > > > >> > > > > > > > > > > > >> >>
> >> > > > > > >> > > > > > > > > > > > >> >> Yufei
> >> > > > > > >> > > > > > > > > > > > >> >>
> >> > > > > > >> > > > > > > > > > > > >> >>
> >> > > > > > >> > > > > > > > > > > > >> >> On Tue, Apr 21, 2026 at 2:34 PM
> >> EJ
> >> > > Wang <
> >> > > > > > >> > > > > > > > > > > > >> ej.wang.dev.interest@gmail.com>
> >> > > > > > >> > > > > > > > > > > > >> >> wrote:
> >> > > > > > >> > > > > > > > > > > > >> >>
> >> > > > > > >> > > > > > > > > > > > >> >> > Also, I've been looking more
> >> > > closely at
> >> > > > > > the
> >> > > > > > >> > > > > > *persistence
> >> > > > > > >> > > > > > > > > schema
> >> > > > > > >> > > > > > > > > > > in
> >> > > > > > >> > > > > > > > > > > > >> the
> >> > > > > > >> > > > > > > > > > > > >> >> > current metrics work*, and I
> >> think
> >> > > > > > there's a
> >> > > > > > >> > > > > structural
> >> > > > > > >> > > > > > > > > > rigidity
> >> > > > > > >> > > > > > > > > > > > >> problem
> >> > > > > > >> > > > > > > > > > > > >> >> > worth raising before the shape
> >> > gets
> >> > > > > locked
> >> > > > > > >> in.
> >> > > > > > >> > > > > > > > > > > > >> >> >
> >> > > > > > >> > > > > > > > > > > > >> >> > Right now we have two separate
> >> > > tables
> >> > > > > > >> > > > > > > (scan_metrics_report
> >> > > > > > >> > > > > > > > > and
> >> > > > > > >> > > > > > > > > > > > >> >> > commit_metrics_report), each
> >> with
> >> > > ~25
> >> > > > > > >> flattened
> >> > > > > > >> > > > > columns
> >> > > > > > >> > > > > > > > that
> >> > > > > > >> > > > > > > > > > > > directly
> >> > > > > > >> > > > > > > > > > > > >> >> > mirror the Iceberg report
> >> fields.
> >> > > The
> >> > > > > SPI
> >> > > > > > >> > follows
> >> > > > > > >> > > > the
> >> > > > > > >> > > > > > > same
> >> > > > > > >> > > > > > > > > > split:
> >> > > > > > >> > > > > > > > > > > > >> >> > writeScanReport and
> >> > > writeCommitReport as
> >> > > > > > >> > separate
> >> > > > > > >> > > > > > > methods,
> >> > > > > > >> > > > > > > > > with
> >> > > > > > >> > > > > > > > > > > > >> per-type
> >> > > > > > >> > > > > > > > > > > > >> >> > record classes, converters, and
> >> > > model
> >> > > > > > >> objects.
> >> > > > > > >> > > *The
> >> > > > > > >> > > > > > > > practical
> >> > > > > > >> > > > > > > > > > > cost:
> >> > > > > > >> > > > > > > > > > > > >> >> > adding a new metric type
> >> > > (operational
> >> > > > > > >> metrics,
> >> > > > > > >> > > for
> >> > > > > > >> > > > > > > example)
> >> > > > > > >> > > > > > > > > > > > requires
> >> > > > > > >> > > > > > > > > > > > >> a
> >> > > > > > >> > > > > > > > > > > > >> >> new
> >> > > > > > >> > > > > > > > > > > > >> >> > table, a new SPI method, a new
> >> > > record
> >> > > > > > >> class, a
> >> > > > > > >> > > new
> >> > > > > > >> > > > > > model
> >> > > > > > >> > > > > > > > > > class, a
> >> > > > > > >> > > > > > > > > > > > new
> >> > > > > > >> > > > > > > > > > > > >> >> > converter branch, and a schema
> >> > > > > migration*.
> >> > > > > > >> > > That's a
> >> > > > > > >> > > > > lot
> >> > > > > > >> > > > > > > of
> >> > > > > > >> > > > > > > > > > > surface
> >> > > > > > >> > > > > > > > > > > > >> area
> >> > > > > > >> > > > > > > > > > > > >> >> > for what should be "one more
> >> kind
> >> > of
> >> > > > > > >> metric."
> >> > > > > > >> > > > > > > > > > > > >> >> >
> >> > > > > > >> > > > > > > > > > > > >> >> > *My bias* would be toward a
> >> single
> >> > > > > metrics
> >> > > > > > >> > table
> >> > > > > > >> > > > with
> >> > > > > > >> > > > > > *a
> >> > > > > > >> > > > > > > > > typed
> >> > > > > > >> > > > > > > > > > > JSON
> >> > > > > > >> > > > > > > > > > > > >> >> > payload*. Something like:
> >> > > metric_type
> >> > > > > > >> (enum),
> >> > > > > > >> > > > > > entity_id,
> >> > > > > > >> > > > > > > > > > > > >> >> > table_identifier, snapshot_id
> >> > > > > (nullable),
> >> > > > > > >> > > > > received_ts,
> >> > > > > > >> > > > > > > > > > > > >> schema_version,
> >> > > > > > >> > > > > > > > > > > > >> >> and
> >> > > > > > >> > > > > > > > > > > > >> >> > a payload column for the
> >> > > metric-specific
> >> > > > > > >> data.
> >> > > > > > >> > > The
> >> > > > > > >> > > > > > > > > metric_type
> >> > > > > > >> > > > > > > > > > +
> >> > > > > > >> > > > > > > > > > > > >> >> > schema_version pair gives us a
> >> > > > > > >> > forward-compatible
> >> > > > > > >> > > > > > > contract
> >> > > > > > >> > > > > > > > > for
> >> > > > > > >> > > > > > > > > > > the
> >> > > > > > >> > > > > > > > > > > > >> >> payload
> >> > > > > > >> > > > > > > > > > > > >> >> > shape. Adding a new metric type
> >> > > becomes
> >> > > > > an
> >> > > > > > >> enum
> >> > > > > > >> > > > value
> >> > > > > > >> > > > > > > and a
> >> > > > > > >> > > > > > > > > > > payload
> >> > > > > > >> > > > > > > > > > > > >> >> schema,
> >> > > > > > >> > > > > > > > > > > > >> >> > not a schema migration. One
> >> thing
> >> > I
> >> > > > > think
> >> > > > > > we
> >> > > > > > >> > need
> >> > > > > > >> > > > to
> >> > > > > > >> > > > > be
> >> > > > > > >> > > > > > > > > > > deliberate
> >> > > > > > >> > > > > > > > > > > > >> >> about is
> >> > > > > > >> > > > > > > > > > > > >> >> > the partition key design. If
> >> all
> >> > > metric
> >> > > > > > >> types
> >> > > > > > >> > > land
> >> > > > > > >> > > > in
> >> > > > > > >> > > > > > one
> >> > > > > > >> > > > > > > > > > table,
> >> > > > > > >> > > > > > > > > > > > scan
> >> > > > > > >> > > > > > > > > > > > >> >> > metrics at scale (high
> >> > concurrency,
> >> > > high
> >> > > > > > >> > > frequency
> >> > > > > > >> > > > > > across
> >> > > > > > >> > > > > > > > > many
> >> > > > > > >> > > > > > > > > > > > >> tables)
> >> > > > > > >> > > > > > > > > > > > >> >> > could easily create hot
> >> > partitions.
> >> > > We'd
> >> > > > > > >> want
> >> > > > > > >> > the
> >> > > > > > >> > > > > > > > persistence
> >> > > > > > >> > > > > > > > > > > layer
> >> > > > > > >> > > > > > > > > > > > >> to
> >> > > > > > >> > > > > > > > > > > > >> >> be
> >> > > > > > >> > > > > > > > > > > > >> >> > able to shard by entity or time
> >> > > range,
> >> > > > > and
> >> > > > > > >> that
> >> > > > > > >> > > > means
> >> > > > > > >> > > > > > the
> >> > > > > > >> > > > > > > > > > logical
> >> > > > > > >> > > > > > > > > > > > >> schema
> >> > > > > > >> > > > > > > > > > > > >> >> > needs to expose enough
> >> structure
> >> > for
> >> > > > > > >> backends
> >> > > > > > >> > to
> >> > > > > > >> > > > > > > partition
> >> > > > > > >> > > > > > > > > on.
> >> > > > > > >> > > > > > > > > > I
> >> > > > > > >> > > > > > > > > > > > >> don't
> >> > > > > > >> > > > > > > > > > > > >> >> > think the current flattened
> >> layout
> >> > > gives
> >> > > > > > us
> >> > > > > > >> > that.
> >> > > > > > >> > > > > > > > > > > > >> >> >
> >> > > > > > >> > > > > > > > > > > > >> >> > This is getting complex enough
> >> > that
> >> > > I
> >> > > > > > don't
> >> > > > > > >> > think
> >> > > > > > >> > > > > > ad-hoc
> >> > > > > > >> > > > > > > > > PR/ML
> >> > > > > > >> > > > > > > > > > > > >> threads
> >> > > > > > >> > > > > > > > > > > > >> >> > will converge well. *Would
> >> people
> >> > be
> >> > > > > open
> >> > > > > > >> to a
> >> > > > > > >> > > > > biweekly
> >> > > > > > >> > > > > > > > sync
> >> > > > > > >> > > > > > > > > > for
> >> > > > > > >> > > > > > > > > > > > >> metrics
> >> > > > > > >> > > > > > > > > > > > >> >> > architecture?* I think 30
> >> minutes
> >> > > every
> >> > > > > > two
> >> > > > > > >> > weeks
> >> > > > > > >> > > > > with
> >> > > > > > >> > > > > > > > > > interested
> >> > > > > > >> > > > > > > > > > > > >> >> parties
> >> > > > > > >> > > > > > > > > > > > >> >> > would be enough to work through
> >> > the
> >> > > > > > schema,
> >> > > > > > >> SPI
> >> > > > > > >> > > > > shape,
> >> > > > > > >> > > > > > > and
> >> > > > > > >> > > > > > > > > read
> >> > > > > > >> > > > > > > > > > > API
> >> > > > > > >> > > > > > > > > > > > >> >> design
> >> > > > > > >> > > > > > > > > > > > >> >> > together. Happy to help set
> >> that
> >> > up.
> >> > > > > > >> > > > > > > > > > > > >> >> >
> >> > > > > > >> > > > > > > > > > > > >> >> > -ej
> >> > > > > > >> > > > > > > > > > > > >> >> >
> >> > > > > > >> > > > > > > > > > > > >> >> > On Mon, Apr 20, 2026 at
> >> 2:19 PM EJ
> >> > > Wang
> >> > > > > <
> >> > > > > > >> > > > > > > > > > > > >> ej.wang.dev.interest@gmail.com
> >> > > > > > >> > > > > > > > > > > > >> >> >
> >> > > > > > >> > > > > > > > > > > > >> >> > wrote:
> >> > > > > > >> > > > > > > > > > > > >> >> >
> >> > > > > > >> > > > > > > > > > > > >> >> >> Reviewed #4115, left a
> >> comment on
> >> > > the
> >> > > > > > code
> >> > > > > > >> > > > > > organization
> >> > > > > > >> > > > > > > > > side.
> >> > > > > > >> > > > > > > > > > > > >> >> >>
> >> > > > > > >> > > > > > > > > > > > >> >> >> One thing stood out: the
> >> metrics
> >> > > write
> >> > > > > > path
> >> > > > > > >> > > enters
> >> > > > > > >> > > > > > > through
> >> > > > > > >> > > > > > > > > > > > >> >> >> PolarisMetricsManager on
> >> > > > > > MetaStoreManager,
> >> > > > > > >> but
> >> > > > > > >> > > the
> >> > > > > > >> > > > > new
> >> > > > > > >> > > > > > > > read
> >> > > > > > >> > > > > > > > > > path
> >> > > > > > >> > > > > > > > > > > > >> >> bypasses
> >> > > > > > >> > > > > > > > > > > > >> >> >> MetaStoreManager entirely and
> >> > goes
> >> > > > > > >> straight to
> >> > > > > > >> > > > > > > > > BasePersistence
> >> > > > > > >> > > > > > > > > > > via
> >> > > > > > >> > > > > > > > > > > > >> >> >> callContext.getMetaStore().
> >> That
> >> > > means
> >> > > > > > the
> >> > > > > > >> > read
> >> > > > > > >> > > > API
> >> > > > > > >> > > > > > only
> >> > > > > > >> > > > > > > > > works
> >> > > > > > >> > > > > > > > > > > for
> >> > > > > > >> > > > > > > > > > > > >> >> backends
> >> > > > > > >> > > > > > > > > > > > >> >> >> that implement
> >> BasePersistence.
> >> > > NoSQL
> >> > > > > and
> >> > > > > > >> > remote
> >> > > > > > >> > > > > > > backends
> >> > > > > > >> > > > > > > > > > can't
> >> > > > > > >> > > > > > > > > > > > >> >> participate.
> >> > > > > > >> > > > > > > > > > > > >> >> >>
> >> > > > > > >> > > > > > > > > > > > >> >> >> Stepping back, I think the
> >> > metrics
> >> > > > > > >> subsystem
> >> > > > > > >> > is
> >> > > > > > >> > > > > > growing
> >> > > > > > >> > > > > > > > into
> >> > > > > > >> > > > > > > > > > > > >> something
> >> > > > > > >> > > > > > > > > > > > >> >> >> real (write + read + REST API
> >> +
> >> > > AuthZ +
> >> > > > > > >> > > > pagination)
> >> > > > > > >> > > > > > *but
> >> > > > > > >> > > > > > > > the
> >> > > > > > >> > > > > > > > > > > > >> >> persistence
> >> > > > > > >> > > > > > > > > > > > >> >> >> side is split across two
> >> layers
> >> > in
> >> > > a
> >> > > > > way
> >> > > > > > >> > that's
> >> > > > > > >> > > > hard
> >> > > > > > >> > > > > > to
> >> > > > > > >> > > > > > > > > > > extend*. I
> >> > > > > > >> > > > > > > > > > > > >> put
> >> > > > > > >> > > > > > > > > > > > >> >> >> together two diagrams to show
> >> > what
> >> > > I
> >> > > > > mean
> >> > > > > > >> (my
> >> > > > > > >> > > best
> >> > > > > > >> > > > > > > > effort).
> >> > > > > > >> > > > > > > > > > > > >> >> >>
> >> > > > > > >> > > > > > > > > > > > >> >> >> *Current state* (Diagram 1):
> >> > three
> >> > > > > > >> interfaces
> >> > > > > > >> > at
> >> > > > > > >> > > > > three
> >> > > > > > >> > > > > > > > > > different
> >> > > > > > >> > > > > > > > > > > > >> >> levels.
> >> > > > > > >> > > > > > > > > > > > >> >> >> The engine-facing SPI
> >> > > > > > >> (PolarisMetricsReporter)
> >> > > > > > >> > > is
> >> > > > > > >> > > > > > clean.
> >> > > > > > >> > > > > > > > But
> >> > > > > > >> > > > > > > > > > > > >> >> >> PolarisMetricsManager on
> >> > > > > MetaStoreManager
> >> > > > > > >> is a
> >> > > > > > >> > > > > > > passthrough
> >> > > > > > >> > > > > > > > > to
> >> > > > > > >> > > > > > > > > > > > >> >> >> MetricsPersistence on
> >> > > BasePersistence.
> >> > > > > > The
> >> > > > > > >> > @Beta
> >> > > > > > >> > > > > > > > annotation
> >> > > > > > >> > > > > > > > > > and
> >> > > > > > >> > > > > > > > > > > > SPI
> >> > > > > > >> > > > > > > > > > > > >> >> javadoc
> >> > > > > > >> > > > > > > > > > > > >> >> >> are on the BasePersistence
> >> layer,
> >> > > while
> >> > > > > > the
> >> > > > > > >> > > actual
> >> > > > > > >> > > > > > > > extension
> >> > > > > > >> > > > > > > > > > > > points
> >> > > > > > >> > > > > > > > > > > > >> >> >> (PolarisMetricsReporter,
> >> > > > > > >> > PolarisMetricsManager)
> >> > > > > > >> > > > > carry
> >> > > > > > >> > > > > > no
> >> > > > > > >> > > > > > > > > > > stability
> >> > > > > > >> > > > > > > > > > > > >> >> >> annotation. The write path
> >> goes
> >> > > through
> >> > > > > > the
> >> > > > > > >> > > > > > > > MetaStoreManager
> >> > > > > > >> > > > > > > > > > > > layer,
> >> > > > > > >> > > > > > > > > > > > >> the
> >> > > > > > >> > > > > > > > > > > > >> >> >> read path doesn't.
> >> > > > > > >> > > > > > > > > > > > >> >> >>
> >> > > > > > >> > > > > > > > > > > > >> >> >> *What I envision* (Diagram 2):
> >> > two
> >> > > SPIs
> >> > > > > > at
> >> > > > > > >> two
> >> > > > > > >> > > > > levels.
> >> > > > > > >> > > > > > > > > > > > >> >> >> PolarisMetricsReporter stays
> >> as
> >> > the
> >> > > > > > >> > > engine-facing
> >> > > > > > >> > > > > SPI.
> >> > > > > > >> > > > > > > > > > > > >> >> >> PolarisMetricsManager becomes
> >> the
> >> > > > > > >> > backend-facing
> >> > > > > > >> > > > SPI
> >> > > > > > >> > > > > > > with
> >> > > > > > >> > > > > > > > > both
> >> > > > > > >> > > > > > > > > > > > write
> >> > > > > > >> > > > > > > > > > > > >> >> and
> >> > > > > > >> > > > > > > > > > > > >> >> >> read methods at the
> >> > > MetaStoreManager
> >> > > > > > level,
> >> > > > > > >> > > where
> >> > > > > > >> > > > > any
> >> > > > > > >> > > > > > > > > backend
> >> > > > > > >> > > > > > > > > > > > (JDBC,
> >> > > > > > >> > > > > > > > > > > > >> >> NoSQL,
> >> > > > > > >> > > > > > > > > > > > >> >> >> remote) can implement them.
> >> > > > > > >> MetricsPersistence
> >> > > > > > >> > > on
> >> > > > > > >> > > > > > > > > > > BasePersistence
> >> > > > > > >> > > > > > > > > > > > >> goes
> >> > > > > > >> > > > > > > > > > > > >> >> >> away. Where metrics actually
> >> land
> >> > > is an
> >> > > > > > >> > > > > implementation
> >> > > > > > >> > > > > > > > > detail,
> >> > > > > > >> > > > > > > > > > > > not a
> >> > > > > > >> > > > > > > > > > > > >> >> core
> >> > > > > > >> > > > > > > > > > > > >> >> >> interface.
> >> > > > > > >> > > > > > > > > > > > >> >> >>
> >> > > > > > >> > > > > > > > > > > > >> >> >> *Minor naming thing*:
> >> > > > > > >> PolarisMetricsReporter
> >> > > > > > >> > is
> >> > > > > > >> > > > > > broader
> >> > > > > > >> > > > > > > > than
> >> > > > > > >> > > > > > > > > > > what
> >> > > > > > >> > > > > > > > > > > > it
> >> > > > > > >> > > > > > > > > > > > >> >> >> actually handles. It only
> >> accepts
> >> > > > > Iceberg
> >> > > > > > >> REST
> >> > > > > > >> > > > > Catalog
> >> > > > > > >> > > > > > > > > metrics
> >> > > > > > >> > > > > > > > > > > > >> >> (ScanReport,
> >> > > > > > >> > > > > > > > > > > > >> >> >> CommitReport via
> >> MetricsReport).
> >> > > > > Generic
> >> > > > > > >> table
> >> > > > > > >> > > > > metrics
> >> > > > > > >> > > > > > > or
> >> > > > > > >> > > > > > > > > > > > >> operational
> >> > > > > > >> > > > > > > > > > > > >> >> >> metrics aren't in scope. Not
> >> > > blocking,
> >> > > > > > but
> >> > > > > > >> > worth
> >> > > > > > >> > > > > > noting
> >> > > > > > >> > > > > > > if
> >> > > > > > >> > > > > > > > > the
> >> > > > > > >> > > > > > > > > > > > >> metrics
> >> > > > > > >> > > > > > > > > > > > >> >> >> surface expands.
> >> > > > > > >> > > > > > > > > > > > >> >> >>
> >> > > > > > >> > > > > > > > > > > > >> >> >> *Rough sketch of how to get
> >> > there*:
> >> > > > > > >> > > > > > > > > > > > >> >> >>  1.  Add read methods to
> >> > > > > > >> PolarisMetricsManager
> >> > > > > > >> > > > > > > > > > (listScanReports,
> >> > > > > > >> > > > > > > > > > > > >> >> >> listCommitReports) with
> >> default
> >> > > no-op,
> >> > > > > > >> same as
> >> > > > > > >> > > the
> >> > > > > > >> > > > > > > > existing
> >> > > > > > >> > > > > > > > > > > write
> >> > > > > > >> > > > > > > > > > > > >> >> methods.
> >> > > > > > >> > > > > > > > > > > > >> >> >> (Probably make
> >> > > PolarisMetricsManager
> >> > > > > more
> >> > > > > > >> > > explicit
> >> > > > > > >> > > > > on
> >> > > > > > >> > > > > > > > being
> >> > > > > > >> > > > > > > > > > > > Iceberg
> >> > > > > > >> > > > > > > > > > > > >> >> >> specific like package name or
> >> > class
> >> > > > > name
> >> > > > > > >> etc.)
> >> > > > > > >> > > > > > > > > > > > >> >> >>  2.  Wire
> >> MetricsReportsService
> >> > > through
> >> > > > > > >> > > > > > MetaStoreManager
> >> > > > > > >> > > > > > > > > > instead
> >> > > > > > >> > > > > > > > > > > > of
> >> > > > > > >> > > > > > > > > > > > >> >> >> callContext.getMetaStore().
> >> > > > > > >> > > > > > > > > > > > >> >> >>  3.  Extract metrics
> >> persistence
> >> > > from
> >> > > > > > >> > > > > > > > > JdbcBasePersistenceImpl
> >> > > > > > >> > > > > > > > > > > into
> >> > > > > > >> > > > > > > > > > > > >> its
> >> > > > > > >> > > > > > > > > > > > >> >> >> own class. That file carries
> >> ~7
> >> > > > > > >> > > responsibilities,
> >> > > > > > >> > > > > > > metrics
> >> > > > > > >> > > > > > > > > > being
> >> > > > > > >> > > > > > > > > > > > one
> >> > > > > > >> > > > > > > > > > > > >> of
> >> > > > > > >> > > > > > > > > > > > >> >> them.
> >> > > > > > >> > > > > > > > > > > > >> >> >>  4.  Remove MetricsPersistence
> >> > from
> >> > > > > > >> > > > BasePersistence.
> >> > > > > > >> > > > > > > > > > > > >> >> >>
> >> > > > > > >> > > > > > > > > > > > >> >> >> *None of this needs to happen
> >> in
> >> > > #4115.
> >> > > > > > >> But if
> >> > > > > > >> > > the
> >> > > > > > >> > > > > > > > direction
> >> > > > > > >> > > > > > > > > > > makes
> >> > > > > > >> > > > > > > > > > > > >> >> sense,
> >> > > > > > >> > > > > > > > > > > > >> >> >> it would be good to align
> >> before
> >> > > the
> >> > > > > > >> metrics
> >> > > > > > >> > > > surface
> >> > > > > > >> > > > > > > grows
> >> > > > > > >> > > > > > > > > > > > further.
> >> > > > > > >> > > > > > > > > > > > >> >> Curious
> >> > > > > > >> > > > > > > > > > > > >> >> >> what others think.*
> >> > > > > > >> > > > > > > > > > > > >> >> >>
> >> > > > > > >> > > > > > > > > > > > >> >> >> *My mental model note*: Level
> >> 1
> >> > > > > > >> > > MetaStoreManager;
> >> > > > > > >> > > > > > level
> >> > > > > > >> > > > > > > 2
> >> > > > > > >> > > > > > > > > > > > >> transactional
> >> > > > > > >> > > > > > > > > > > > >> >> >> persistence; level 3 base
> >> > > persistence
> >> > > > > > >> > > > > > > > > > > > >> >> >>
> >> > > > > > >> > > > > > > > > > > > >> >> >> Diagram 1
> >> > > > > > >> > > > > > > > > > > > >> >> >> <
> >> > > > > > >> > > > > > > > > > > > >> >>
> >> > > > > > >> > > > > > > > > > > > >>
> >> > > > > > >> > > > > > > > > > > >
> >> > > > > > >> > > > > > > > > > >
> >> > > > > > >> > > > > > > > > >
> >> > > > > > >> > > > > > > > >
> >> > > > > > >> > > > > > > >
> >> > > > > > >> > > > > > >
> >> > > > > > >> > > > > >
> >> > > > > > >> > > > >
> >> > > > > > >> > > >
> >> > > > > > >> > >
> >> > > > > > >> >
> >> > > > > > >>
> >> > > > > >
> >> > > > >
> >> > >
> >> >
> >> https://urldefense.com/v3/__https://www.plantuml.com/plantuml/uml/bLHDR-Cs4BthLmpIYupw0zbkKQ1r3M-S7Bp8xhhM7WCOb3IM65EaGD9EX2RzxHrHb4CxRelwa4YSDu_lpOVcnZ9jzvM8BBS2uGjQpJC3dtHMSekPtMk44IpsMgEqa5XcCOhCZikQQLP1pR8TAp2n3ILhmZDP20m0fcIvUkAoW2qJXd9z1bpToO9BX3WXu0ucy5rpgGPNm0nW5_epUWtm2Ue3pn3kMOFQmKntGZW0BYtgBSi8k5A2QMwybJNMIbFiGSR9QZc4nUqIvikStF0jHprua5C-amge42aNt3R0f5JaaoivdV2Pkqbx4hee4ymOkBh5BTiB-_uIeGeo8zL8rPsPl4DktdEiK1jkB1NdZCRbrSTecDe_mlHbF0wvBmCkaOH5_S8a_TTTKI6-nmCAkEw4LpxsZ-LbYLKQFKMNOgf_wuM7_bV9gOer5SYMMksBSWXFcbi49KNZXNLicwfe3TETC7gPdPqI7uBcHMb1RSzYq34c6PDUM9mn8HRsUTZEiDBve3NjVZumBj0U7SS37mGO7vcwtiK-_pU7U7L_f-digo9YbhSwIfMRwIITKGXbxdIUTCGF1SeCJxloKsU-3k9ddRbX1eDq1q_fx1JbBGT0glVyXimDuP4TQ5qpCAmnGEj2s_6n5mtn1z-97-63itFQZLPO1Ev2tu_WF7Ju-VPc0Skg5bYXxBhkY1xpD7EM_7fyflSpIsqMgVth5xhVr4eQxWQ8enaSAJQSG16yFSDuJ798rrcXr_3n-lfdk7icQjEBmFujL7AodiP_Y4Z7-YxvtZNs4zMgpNTl6tF8sglyPsmqchrjvQ-m-aP94r-TwCA2Ka8upPJZwtvSpoYCXkYMZU2NXvRMBfq9P3i3Le4VAZUAlUZ_oPKsxPgY0Q_BSKLkyr9bhQhQrJjo_x3TPlIB0DPjnMfcIoYP0QaYw1a0fTKDr8fB6ntNuvmoL1ZGkXa69Njh43zf9GiGxHQrA_jDYWRSzF5--WmTVrN97_Sm8LbLUy_lGBmLanJjFkDlGkRqjA_4tm00__;!!Iz9xO38YGHZK!4rnlVIqBQQQ6xpGe1PMV58edOS_cGrKoqdfhw4qSp_JQCavvEjaxXk0OwGPnxyEcO60AzKQK_XDtp7H3DdkFXcrXuTBISHg$
> >> > > > > > >> > > > > > > > > > > > >> >> >
> >> > > > > > >> > > > > > > > > > > > >> >> >> :
> >> > > > > > >> > > > > > > > > > > > >> >> >>
> >> > > > > > >> > > > > > > > > > > > >> >> >> [image: image.png]
> >> > > > > > >> > > > > > > > > > > > >> >> >>
> >> > > > > > >> > > > > > > > > > > > >> >> >> Diagram 2
> >> > > > > > >> > > > > > > > > > > > >> >> >> <
> >> > > > > > >> > > > > > > > > > > > >> >>
> >> > > > > > >> > > > > > > > > > > > >>
> >> > > > > > >> > > > > > > > > > > >
> >> > > > > > >> > > > > > > > > > >
> >> > > > > > >> > > > > > > > > >
> >> > > > > > >> > > > > > > > >
> >> > > > > > >> > > > > > > >
> >> > > > > > >> > > > > > >
> >> > > > > > >> > > > > >
> >> > > > > > >> > > > >
> >> > > > > > >> > > >
> >> > > > > > >> > >
> >> > > > > > >> >
> >> > > > > > >>
> >> > > > > >
> >> > > > >
> >> > >
> >> >
> >> https://urldefense.com/v3/__https://www.plantuml.com/plantuml/uml/VLLDR-8m4BtdLupO2sWBLVU8AaGB7AXAbssGzb896SS9RXqxjKqBwkv_tt7iV43fdaZYDpFlpRmnOsE9jhjSH9PRmM31hERKm8scMsuPjJlDe0yheZDc8RR4iYWoBrmMH9CS2a9VICPYUy1OZN0YCy5Q0BCbYNhdCeEK28En8G8wCvbnoQ0R8_05Bc6bkLIz3X03p1zzH7zR-9ZfDquPt9C3qoNCX2yV4G2NbkcKu5jdgGJHt0GbZwnG6i-UP3TUpk5gM6Ldqke350eZUqzoCft3U9xWHvxoa5-7K4nF1J46EbEMafsmdrCBbQ44gVggy18IZrn_ph5asd1ZiIKdQSgueZvjXrQFSFrdC3YN-nXmBacxbGiYyLVxLaBtdhqn0LSzdBDhqQtQoOJeGyad3z0lUqnYgpGB6Ns8oVyta00Dy_WnX0tIOZ8v6SYxHll1TrH6aejAik-mh-AphVFCwSUQqFypElag5QRGFDjQKEd96K1P8QP41c9TzA_IIQyvdAWyv_RSiS3skb0_EzDDkK2v5xWF6MiGFlvhpFLcD2Dq2pml14gaF67eQkmd8gulDoC4kSOu6KVpkvlUJg1RTbWISU40RdBUUS_9XfRZ2dwxm_SW8LYFISgm_MnlDQ6M9P1gbKEc4X-2pH_FvJCkCqm9pbVjD6LrwdLeOrDWfOaqc8Wh9BE85oNKxkNQ6o4yGRy_Eae0G_G8tZv81d3bHDB23WOdisohVr3nh_j6lbSjbNaLRTc8UgtPbAU1J_tygOfZX9DWEJeHDvYx-qmSi5FgNLPZwHrHcUsncGQ5-skhUclpE5fo4ounpFauYrUbkU6ccfnxMvitwag4IyerhTxj8In_Oj1bDO4pQru674loYrGlULHLEGCjwJJ8gDoVZR8MxO4BT3IzRvIcAQKezC6xpziGnTyImrfEGyJI_OcKfgtxIvnTqFEMS17L9Z-jsARN5FmTheP7HtSdtOMT0B4GY2FYHXxgQmMtj2bRqiLFGapiVe1_QVKDrkqXcm83aFEXnMYCZ-xlyHy__;!!Iz9xO38YGHZK!4rnlVIqBQQQ6xpGe1PMV58edOS_cGrKoqdfhw4qSp_JQCavvEjaxXk0OwGPnxyEcO60AzKQK_XDtp7H3DdkFXcrXSpmqcNk$
> >> > > > > > >> > > > > > > > > > > > >> >> >
> >> > > > > > >> > > > > > > > > > > > >> >> >> :
> >> > > > > > >> > > > > > > > > > > > >> >> >> [image: image.png]
> >> > > > > > >> > > > > > > > > > > > >> >> >>
> >> > > > > > >> > > > > > > > > > > > >> >> >>  -ej
> >> > > > > > >> > > > > > > > > > > > >> >> >>
> >> > > > > > >> > > > > > > > > > > > >> >> >> On Wed, Apr 15, 2026 at
> >> 8:22 AM
> >> > > Dmitri
> >> > > > > > >> > > > Bourlatchkov
> >> > > > > > >> > > > > <
> >> > > > > > >> > > > > > > > > > > > >> dimas@apache.org>
> >> > > > > > >> > > > > > > > > > > > >> >> >> wrote:
> >> > > > > > >> > > > > > > > > > > > >> >> >>
> >> > > > > > >> > > > > > > > > > > > >> >> >>> Hi All,
> >> > > > > > >> > > > > > > > > > > > >> >> >>>
> >> > > > > > >> > > > > > > > > > > > >> >> >>> Heads up: The current state
> >> of
> >> > PR
> >> > > > > [4115]
> >> > > > > > >> > looks
> >> > > > > > >> > > > > pretty
> >> > > > > > >> > > > > > > > solid
> >> > > > > > >> > > > > > > > > > to
> >> > > > > > >> > > > > > > > > > > > me.
> >> > > > > > >> > > > > > > > > > > > >> I
> >> > > > > > >> > > > > > > > > > > > >> >> >>> believe this PR is
> >> approaching a
> >> > > > > > mergeable
> >> > > > > > >> > > > > condition.
> >> > > > > > >> > > > > > > > > > > > >> >> >>>
> >> > > > > > >> > > > > > > > > > > > >> >> >>> Please post your reviews if
> >> you
> >> > > have
> >> > > > > any
> >> > > > > > >> > > > comments.
> >> > > > > > >> > > > > > > > > > > > >> >> >>>
> >> > > > > > >> > > > > > > > > > > > >> >> >>> [4115]
> >> > > > > > >> > > >
> >> >
> >> https://urldefense.com/v3/__https://github.com/apache/polaris/pull/4115__;!!Iz9xO38YGHZK!4rnlVIqBQQQ6xpGe1PMV58edOS_cGrKoqdfhw4qSp_JQCavvEjaxXk0OwGPnxyEcO60AzKQK_XDtp7H3DdkFXcrXtFRscKs$
> >> > > > > > >> > > > > > > > > > > > >> >> >>>
> >> > > > > > >> > > > > > > > > > > > >> >> >>> Thanks,
> >> > > > > > >> > > > > > > > > > > > >> >> >>> Dmitri.
> >> > > > > > >> > > > > > > > > > > > >> >> >>>
> >> > > > > > >> > > > > > > > > > > > >> >> >>> On Tue, Mar 3, 2026 at
> >> 3:29 PM
> >> > > Anand
> >> > > > > > Kumar
> >> > > > > > >> > > > Sankaran
> >> > > > > > >> > > > > > via
> >> > > > > > >> > > > > > > > > dev <
> >> > > > > > >> > > > > > > > > > > > >> >> >>> dev@polaris.apache.org>
> >> wrote:
> >> > > > > > >> > > > > > > > > > > > >> >> >>>
> >> > > > > > >> > > > > > > > > > > > >> >> >>> > Hi Yufei and Dmitri,
> >> > > > > > >> > > > > > > > > > > > >> >> >>> >
> >> > > > > > >> > > > > > > > > > > > >> >> >>> > Here is a proposal for the
> >> > REST
> >> > > > > > >> endpoints
> >> > > > > > >> > for
> >> > > > > > >> > > > > > metrics
> >> > > > > > >> > > > > > > > and
> >> > > > > > >> > > > > > > > > > > > events.
> >> > > > > > >> > > > > > > > > > > > >> >> >>> >
> >> > > > > > >> > > > > > > > > > > > >> >> >>> >
> >> > > > > > >> > > > > >
> >> >
> >> https://urldefense.com/v3/__https://github.com/apache/polaris/pull/3924/changes__;!!Iz9xO38YGHZK!4rnlVIqBQQQ6xpGe1PMV58edOS_cGrKoqdfhw4qSp_JQCavvEjaxXk0OwGPnxyEcO60AzKQK_XDtp7H3DdkFXcrXcjoPhSs$
> >> > > > > > >> > > > > > > > > > > > >> >> >>> >
> >> > > > > > >> > > > > > > > > > > > >> >> >>> > I did not see any
> >> precursors
> >> > for
> >> > > > > > >> raising a
> >> > > > > > >> > PR
> >> > > > > > >> > > > for
> >> > > > > > >> > > > > > > > > > proposals,
> >> > > > > > >> > > > > > > > > > > so
> >> > > > > > >> > > > > > > > > > > > >> >> trying
> >> > > > > > >> > > > > > > > > > > > >> >> >>> > this.  Please let me know
> >> what
> >> > > you
> >> > > > > > >> think.
> >> > > > > > >> > > > > > > > > > > > >> >> >>> >
> >> > > > > > >> > > > > > > > > > > > >> >> >>> > -
> >> > > > > > >> > > > > > > > > > > > >> >> >>> > Anand
> >> > > > > > >> > > > > > > > > > > > >> >> >>> >
> >> > > > > > >> > > > > > > > > > > > >> >> >>> > From: Anand Kumar Sankaran
> >> <
> >> > > > > > >> > > > > > > anand.sankaran@workday.com
> >> > > > > > >> > > > > > > > >
> >> > > > > > >> > > > > > > > > > > > >> >> >>> > Date: Monday, March 2,
> >> 2026 at
> >> > > > > > 10:25 AM
> >> > > > > > >> > > > > > > > > > > > >> >> >>> > To: dev@polaris.apache.org
> >> <
> >> > > > > > >> > > > > dev@polaris.apache.org
> >> > > > > > >> > > > > > >
> >> > > > > > >> > > > > > > > > > > > >> >> >>> > Subject: Re: Polaris
> >> Telemetry
> >> > > and
> >> > > > > > Audit
> >> > > > > > >> > > Trail
> >> > > > > > >> > > > > > > > > > > > >> >> >>> >
> >> > > > > > >> > > > > > > > > > > > >> >> >>> > About the REST API, based
> >> on
> >> > my
> >> > > use
> >> > > > > > >> cases:
> >> > > > > > >> > > > > > > > > > > > >> >> >>> >
> >> > > > > > >> > > > > > > > > > > > >> >> >>> >
> >> > > > > > >> > > > > > > > > > > > >> >> >>> >   1.
> >> > > > > > >> > > > > > > > > > > > >> >> >>> > I want to be able to query
> >> > > commit
> >> > > > > > >> metrics
> >> > > > > > >> > to
> >> > > > > > >> > > > > track
> >> > > > > > >> > > > > > > > files
> >> > > > > > >> > > > > > > > > > > added
> >> > > > > > >> > > > > > > > > > > > /
> >> > > > > > >> > > > > > > > > > > > >> >> >>> removed
> >> > > > > > >> > > > > > > > > > > > >> >> >>> > per commit, along with
> >> record
> >> > > > > counts.
> >> > > > > > >> The
> >> > > > > > >> > > > > ingestion
> >> > > > > > >> > > > > > > > > > pipeline
> >> > > > > > >> > > > > > > > > > > > that
> >> > > > > > >> > > > > > > > > > > > >> >> >>> writes
> >> > > > > > >> > > > > > > > > > > > >> >> >>> > this data is owned by us
> >> and
> >> > we
> >> > > are
> >> > > > > > >> > > guaranteed
> >> > > > > > >> > > > to
> >> > > > > > >> > > > > > > write
> >> > > > > > >> > > > > > > > > > this
> >> > > > > > >> > > > > > > > > > > > >> >> >>> information
> >> > > > > > >> > > > > > > > > > > > >> >> >>> > for each write.
> >> > > > > > >> > > > > > > > > > > > >> >> >>> >   2.
> >> > > > > > >> > > > > > > > > > > > >> >> >>> > I want to be able to query
> >> > scan
> >> > > > > > metrics
> >> > > > > > >> for
> >> > > > > > >> > > > > read. I
> >> > > > > > >> > > > > > > > > > > understand
> >> > > > > > >> > > > > > > > > > > > >> >> clients
> >> > > > > > >> > > > > > > > > > > > >> >> >>> do
> >> > > > > > >> > > > > > > > > > > > >> >> >>> > not fulfill this
> >> requirement.
> >> > > > > > >> > > > > > > > > > > > >> >> >>> >   3.
> >> > > > > > >> > > > > > > > > > > > >> >> >>> > I want to be able to query
> >> the
> >> > > > > events
> >> > > > > > >> table
> >> > > > > > >> > > > > (events
> >> > > > > > >> > > > > > > are
> >> > > > > > >> > > > > > > > > > > > >> persisted) -
> >> > > > > > >> > > > > > > > > > > > >> >> >>> this
> >> > > > > > >> > > > > > > > > > > > >> >> >>> > may supersede #2, I am not
> >> > sure
> >> > > yet.
> >> > > > > > >> > > > > > > > > > > > >> >> >>> >
> >> > > > > > >> > > > > > > > > > > > >> >> >>> > All this information is in
> >> the
> >> > > JDBC
> >> > > > > > >> based
> >> > > > > > >> > > > > > persistence
> >> > > > > > >> > > > > > > > > model
> >> > > > > > >> > > > > > > > > > > and
> >> > > > > > >> > > > > > > > > > > > >> is
> >> > > > > > >> > > > > > > > > > > > >> >> >>> > persisted in the
> >> metastore. I
> >> > > > > > currently
> >> > > > > > >> > don’t
> >> > > > > > >> > > > > have
> >> > > > > > >> > > > > > a
> >> > > > > > >> > > > > > > > need
> >> > > > > > >> > > > > > > > > > to
> >> > > > > > >> > > > > > > > > > > > >> query
> >> > > > > > >> > > > > > > > > > > > >> >> >>> > prometheus or open
> >> telemetry.
> >> > I
> >> > > do
> >> > > > > > >> publish
> >> > > > > > >> > > some
> >> > > > > > >> > > > > > > events
> >> > > > > > >> > > > > > > > to
> >> > > > > > >> > > > > > > > > > > > >> Prometheus
> >> > > > > > >> > > > > > > > > > > > >> >> >>> and
> >> > > > > > >> > > > > > > > > > > > >> >> >>> > they are forwarded to our
> >> > > dashboards
> >> > > > > > >> > > elsewhere.
> >> > > > > > >> > > > > > > > > > > > >> >> >>> >
> >> > > > > > >> > > > > > > > > > > > >> >> >>> > About the CLI utilities, I
> >> > > meant the
> >> > > > > > >> admin
> >> > > > > > >> > > user
> >> > > > > > >> > > > > > > > > utilities.
> >> > > > > > >> > > > > > > > > > In
> >> > > > > > >> > > > > > > > > > > > >> one of
> >> > > > > > >> > > > > > > > > > > > >> >> >>> the
> >> > > > > > >> > > > > > > > > > > > >> >> >>> > earliest drafts of my
> >> > proposal,
> >> > > > > > Prashant
> >> > > > > > >> > > > > mentioned
> >> > > > > > >> > > > > > > that
> >> > > > > > >> > > > > > > > > the
> >> > > > > > >> > > > > > > > > > > > >> metrics
> >> > > > > > >> > > > > > > > > > > > >> >> >>> tables
> >> > > > > > >> > > > > > > > > > > > >> >> >>> > can grow indefinitely and
> >> > that a
> >> > > > > > similar
> >> > > > > > >> > > > problem
> >> > > > > > >> > > > > > > exists
> >> > > > > > >> > > > > > > > > > with
> >> > > > > > >> > > > > > > > > > > > the
> >> > > > > > >> > > > > > > > > > > > >> >> events
> >> > > > > > >> > > > > > > > > > > > >> >> >>> > table as well. We discussed
> >> > that
> >> > > > > > >> cleaning
> >> > > > > > >> > up
> >> > > > > > >> > > of
> >> > > > > > >> > > > > old
> >> > > > > > >> > > > > > > > > records
> >> > > > > > >> > > > > > > > > > > > from
> >> > > > > > >> > > > > > > > > > > > >> >> both
> >> > > > > > >> > > > > > > > > > > > >> >> >>> > metrics tables and events
> >> > > tables can
> >> > > > > > be
> >> > > > > > >> > done
> >> > > > > > >> > > > via
> >> > > > > > >> > > > > a
> >> > > > > > >> > > > > > > CLI
> >> > > > > > >> > > > > > > > > > > utility.
> >> > > > > > >> > > > > > > > > > > > >> >> >>> >
> >> > > > > > >> > > > > > > > > > > > >> >> >>> > I see that Yufei has
> >> covered
> >> > the
> >> > > > > > >> discussion
> >> > > > > > >> > > > about
> >> > > > > > >> > > > > > > > > > > datasources.
> >> > > > > > >> > > > > > > > > > > > >> >> >>> >
> >> > > > > > >> > > > > > > > > > > > >> >> >>> > -
> >> > > > > > >> > > > > > > > > > > > >> >> >>> > Anand
> >> > > > > > >> > > > > > > > > > > > >> >> >>> >
> >> > > > > > >> > > > > > > > > > > > >> >> >>> >
> >> > > > > > >> > > > > > > > > > > > >> >> >>> >
> >> > > > > > >> > > > > > > > > > > > >> >> >>> > From: Yufei Gu <
> >> > > > > flyrain000@gmail.com>
> >> > > > > > >> > > > > > > > > > > > >> >> >>> > Date: Friday, February 27,
> >> > 2026
> >> > > at
> >> > > > > > >> 9:54 PM
> >> > > > > > >> > > > > > > > > > > > >> >> >>> > To: dev@polaris.apache.org
> >> <
> >> > > > > > >> > > > > dev@polaris.apache.org
> >> > > > > > >> > > > > > >
> >> > > > > > >> > > > > > > > > > > > >> >> >>> > Subject: Re: Polaris
> >> Telemetry
> >> > > and
> >> > > > > > Audit
> >> > > > > > >> > > Trail
> >> > > > > > >> > > > > > > > > > > > >> >> >>> >
> >> > > > > > >> > > > > > > > > > > > >> >> >>> > This Message Is From an
> >> > External
> >> > > > > > Sender
> >> > > > > > >> > > > > > > > > > > > >> >> >>> > This message came from
> >> outside
> >> > > your
> >> > > > > > >> > > > organization.
> >> > > > > > >> > > > > > > > > > > > >> >> >>> > Report Suspicious<
> >> > > > > > >> > > > > > > > > > > > >> >> >>> >
> >> > > > > > >> > > > > > > > > > > > >> >> >>>
> >> > > > > > >> > > > > > > > > > > > >> >>
> >> > > > > > >> > > > > > > > > > > > >>
> >> > > > > > >> > > > > > > > > > > >
> >> > > > > > >> > > > > > > > > > >
> >> > > > > > >> > > > > > > > > >
> >> > > > > > >> > > > > > > > >
> >> > > > > > >> > > > > > > >
> >> > > > > > >> > > > > > >
> >> > > > > > >> > > > > >
> >> > > > > > >> > > > >
> >> > > > > > >> > > >
> >> > > > > > >> > >
> >> > > > > > >> >
> >> > > > > > >>
> >> > > > > >
> >> > > > >
> >> > >
> >> >
> >> https://us-phishalarm-ewt.proofpoint.com/EWT/v1/Iz9xO38YGHZK!YhNDZABkHi1B699ote2uMwpOZw8i0QMCGO2Szc-HshuABGhGvwPJcymE6G2oUUxtS8xDkSrtGTPm_I3QnVDHoLMk50m9v8z_nZKTkd-bnVUbreF1u0WnfV_X5eYevZl_$
> >> > > > > > >> > > > > > > > > > > > >> >> >>> > >
> >> > > > > > >> > > > > > > > > > > > >> >> >>> >
> >> > > > > > >> > > > > > > > > > > > >> >> >>> >
> >> > > > > > >> > > > > > > > > > > > >> >> >>> > As I mentioned in
> >> > > > > > >> > > > > > > > > > > > >> >> >>> >
> >> > > > > > >> > > > > > > > > > > > >> >> >>>
> >> > > > > > >> > > > > > > > > > > > >> >>
> >> > > > > > >> > > > > > > > > > > > >>
> >> > > > > > >> > > > > > > > > > > >
> >> > > > > > >> > > > > > > > > > >
> >> > > > > > >> > > > > > > > > >
> >> > > > > > >> > > > > > > > >
> >> > > > > > >> > > > > > > >
> >> > > > > > >> > > > > > >
> >> > > > > > >> > > > > >
> >> > > > > > >> > > > >
> >> > > > > > >> > > >
> >> > > > > > >> > >
> >> > > > > > >> >
> >> > > > > > >>
> >> > > > > >
> >> > > > >
> >> > >
> >> >
> >> https://urldefense.com/v3/__https://github.com/apache/polaris/issues/3890__;!!Iz9xO38YGHZK!5EuyFFkk3vhRWVIRvQAWBSQfpJkTMA9HxugzDwXmN0LPPqhEFxYkFRGVhtb8AqUwXtDh2OplcMnbMDHKOxrvDU0$
> >> > >> > > >> > > > > > > > > > > > >> >> >>> ,
> >> > > > > > >> > > > > > > > > > > > >> >> >>> > supporting
> >> > > > > > >> > > > > > > > > > > > >> >> >>> > multiple data sources is
> >> not a
> >> > > > > trivial
> >> > > > > > >> > > change.
> >> > > > > > >> > > > I
> >> > > > > > >> > > > > > > would
> >> > > > > > >> > > > > > > > > > > strongly
> >> > > > > > >> > > > > > > > > > > > >> >> >>> recommend
> >> > > > > > >> > > > > > > > > > > > >> >> >>> > starting with a design
> >> > document
> >> > > to
> >> > > > > > >> > carefully
> >> > > > > > >> > > > > > evaluate
> >> > > > > > >> > > > > > > > the
> >> > > > > > >> > > > > > > > > > > > >> >> architectural
> >> > > > > > >> > > > > > > > > > > > >> >> >>> > implications and long term
> >> > > impact.
> >> > > > > > >> > > > > > > > > > > > >> >> >>> >
> >> > > > > > >> > > > > > > > > > > > >> >> >>> > A REST endpoint to query
> >> > metrics
> >> > > > > seems
> >> > > > > > >> > > > reasonable
> >> > > > > > >> > > > > > > given
> >> > > > > > >> > > > > > > > > the
> >> > > > > > >> > > > > > > > > > > > >> current
> >> > > > > > >> > > > > > > > > > > > >> >> >>> JDBC
> >> > > > > > >> > > > > > > > > > > > >> >> >>> > based persistence model.
> >> That
> >> > > said,
> >> > > > > we
> >> > > > > > >> may
> >> > > > > > >> > > also
> >> > > > > > >> > > > > > > > consider
> >> > > > > > >> > > > > > > > > > > > >> alternative
> >> > > > > > >> > > > > > > > > > > > >> >> >>> > storage models. For
> >> example,
> >> > if
> >> > > we
> >> > > > > > later
> >> > > > > > >> > > adopt
> >> > > > > > >> > > > a
> >> > > > > > >> > > > > > time
> >> > > > > > >> > > > > > > > > > series
> >> > > > > > >> > > > > > > > > > > > >> system
> >> > > > > > >> > > > > > > > > > > > >> >> >>> such as
> >> > > > > > >> > > > > > > > > > > > >> >> >>> > Prometheus to store
> >> metrics,
> >> > the
> >> > > > > query
> >> > > > > > >> > model
> >> > > > > > >> > > > and
> >> > > > > > >> > > > > > > access
> >> > > > > > >> > > > > > > > > > > > patterns
> >> > > > > > >> > > > > > > > > > > > >> >> would
> >> > > > > > >> > > > > > > > > > > > >> >> >>> be
> >> > > > > > >> > > > > > > > > > > > >> >> >>> > fundamentally different.
> >> > > Designing
> >> > > > > the
> >> > > > > > >> REST
> >> > > > > > >> > > API
> >> > > > > > >> > > > > > > without
> >> > > > > > >> > > > > > > > > > > > >> considering
> >> > > > > > >> > > > > > > > > > > > >> >> >>> these
> >> > > > > > >> > > > > > > > > > > > >> >> >>> > potential evolutions may
> >> limit
> >> > > > > > >> flexibility.
> >> > > > > > >> > > I'd
> >> > > > > > >> > > > > > > suggest
> >> > > > > > >> > > > > > > > > to
> >> > > > > > >> > > > > > > > > > > > start
> >> > > > > > >> > > > > > > > > > > > >> >> with
> >> > > > > > >> > > > > > > > > > > > >> >> >>> the
> >> > > > > > >> > > > > > > > > > > > >> >> >>> > use case.
> >> > > > > > >> > > > > > > > > > > > >> >> >>> >
> >> > > > > > >> > > > > > > > > > > > >> >> >>> > Yufei
> >> > > > > > >> > > > > > > > > > > > >> >> >>> >
> >> > > > > > >> > > > > > > > > > > > >> >> >>> >
> >> > > > > > >> > > > > > > > > > > > >> >> >>> > On Fri, Feb 27, 2026 at
> >> > 3:42 PM
> >> > > > > Dmitri
> >> > > > > > >> > > > > > Bourlatchkov <
> >> > > > > > >> > > > > > > > > > > > >> >> dimas@apache.org>
> >> > > > > > >> > > > > > > > > > > > >> >> >>> > wrote:
> >> > > > > > >> > > > > > > > > > > > >> >> >>> >
> >> > > > > > >> > > > > > > > > > > > >> >> >>> > > Hi Anand,
> >> > > > > > >> > > > > > > > > > > > >> >> >>> > >
> >> > > > > > >> > > > > > > > > > > > >> >> >>> > > Sharing my view...
> >> subject
> >> > to
> >> > > > > > >> discussion:
> >> > > > > > >> > > > > > > > > > > > >> >> >>> > >
> >> > > > > > >> > > > > > > > > > > > >> >> >>> > > 1. Adding non-IRC REST
> >> API
> >> > to
> >> > > > > > Polaris
> >> > > > > > >> is
> >> > > > > > >> > > > > > perfectly
> >> > > > > > >> > > > > > > > > fine.
> >> > > > > > >> > > > > > > > > > > > >> >> >>> > >
> >> > > > > > >> > > > > > > > > > > > >> >> >>> > > Figuring out specific
> >> > endpoint
> >> > > > > URIs
> >> > > > > > >> and
> >> > > > > > >> > > > > payloads
> >> > > > > > >> > > > > > > > might
> >> > > > > > >> > > > > > > > > > > > require
> >> > > > > > >> > > > > > > > > > > > >> a
> >> > > > > > >> > > > > > > > > > > > >> >> few
> >> > > > > > >> > > > > > > > > > > > >> >> >>> > > roundtrips, so opening a
> >> > > separate
> >> > > > > > >> thread
> >> > > > > > >> > > for
> >> > > > > > >> > > > > that
> >> > > > > > >> > > > > > > > might
> >> > > > > > >> > > > > > > > > > be
> >> > > > > > >> > > > > > > > > > > > >> best.
> >> > > > > > >> > > > > > > > > > > > >> >> >>> > > Contributors commonly
> >> create
> >> > > > > Google
> >> > > > > > >> Docs
> >> > > > > > >> > > for
> >> > > > > > >> > > > > new
> >> > > > > > >> > > > > > > API
> >> > > > > > >> > > > > > > > > > > > proposals
> >> > > > > > >> > > > > > > > > > > > >> too
> >> > > > > > >> > > > > > > > > > > > >> >> >>> (they
> >> > > > > > >> > > > > > > > > > > > >> >> >>> > > fairly easy to update as
> >> the
> >> > > email
> >> > > > > > >> > > discussion
> >> > > > > > >> > > > > > > > > > progresses).
> >> > > > > > >> > > > > > > > > > > > >> >> >>> > >
> >> > > > > > >> > > > > > > > > > > > >> >> >>> > > There was a suggestion to
> >> > try
> >> > > > > > Markdown
> >> > > > > > >> > > (with
> >> > > > > > >> > > > > PRs)
> >> > > > > > >> > > > > > > for
> >> > > > > > >> > > > > > > > > > > > proposals
> >> > > > > > >> > > > > > > > > > > > >> >> [1]
> >> > > > > > >> > > > > > > > > > > > >> >> >>> ...
> >> > > > > > >> > > > > > > > > > > > >> >> >>> > > feel free to give it a
> >> try
> >> > if
> >> > > you
> >> > > > > > are
> >> > > > > > >> > > > > comfortable
> >> > > > > > >> > > > > > > > with
> >> > > > > > >> > > > > > > > > > > that.
> >> > > > > > >> > > > > > > > > > > > >> >> >>> > >
> >> > > > > > >> > > > > > > > > > > > >> >> >>> > > 2. Could you clarify
> >> whether
> >> > > you
> >> > > > > > mean
> >> > > > > > >> end
> >> > > > > > >> > > > user
> >> > > > > > >> > > > > > > > > utilities
> >> > > > > > >> > > > > > > > > > or
> >> > > > > > >> > > > > > > > > > > > >> admin
> >> > > > > > >> > > > > > > > > > > > >> >> >>> user
> >> > > > > > >> > > > > > > > > > > > >> >> >>> > > utilities? In the latter
> >> > case
> >> > > > > those
> >> > > > > > >> might
> >> > > > > > >> > > be
> >> > > > > > >> > > > > more
> >> > > > > > >> > > > > > > > > > suitable
> >> > > > > > >> > > > > > > > > > > > for
> >> > > > > > >> > > > > > > > > > > > >> the
> >> > > > > > >> > > > > > > > > > > > >> >> >>> Admin
> >> > > > > > >> > > > > > > > > > > > >> >> >>> > > CLI (java) not the Python
> >> > CLI,
> >> > > > > IMHO.
> >> > > > > > >> > > > > > > > > > > > >> >> >>> > >
> >> > > > > > >> > > > > > > > > > > > >> >> >>> > > Why would these
> >> utilities be
> >> > > > > common
> >> > > > > > >> with
> >> > > > > > >> > > > > events?
> >> > > > > > >> > > > > > > > IMHO,
> >> > > > > > >> > > > > > > > > > > event
> >> > > > > > >> > > > > > > > > > > > >> use
> >> > > > > > >> > > > > > > > > > > > >> >> >>> cases
> >> > > > > > >> > > > > > > > > > > > >> >> >>> > are
> >> > > > > > >> > > > > > > > > > > > >> >> >>> > > distinct from scan/commit
> >> > > metrics.
> >> > > > > > >> > > > > > > > > > > > >> >> >>> > >
> >> > > > > > >> > > > > > > > > > > > >> >> >>> > > 3. I'd prefer separating
> >> > > metrics
> >> > > > > > >> > > persistence
> >> > > > > > >> > > > > from
> >> > > > > > >> > > > > > > > > > MetaStore
> >> > > > > > >> > > > > > > > > > > > >> >> >>> persistence
> >> > > > > > >> > > > > > > > > > > > >> >> >>> > at
> >> > > > > > >> > > > > > > > > > > > >> >> >>> > > the code level, so that
> >> they
> >> > > could
> >> > > > > > be
> >> > > > > > >> > mixed
> >> > > > > > >> > > > and
> >> > > > > > >> > > > > > > > matched
> >> > > > > > >> > > > > > > > > > > > >> >> >>> independently.
> >> > > > > > >> > > > > > > > > > > > >> >> >>> > The
> >> > > > > > >> > > > > > > > > > > > >> >> >>> > > separate datasource
> >> question
> >> > > will
> >> > > > > > >> become
> >> > > > > > >> > a
> >> > > > > > >> > > > > > > non-issue
> >> > > > > > >> > > > > > > > > with
> >> > > > > > >> > > > > > > > > > > > that
> >> > > > > > >> > > > > > > > > > > > >> >> >>> approach,
> >> > > > > > >> > > > > > > > > > > > >> >> >>> > I
> >> > > > > > >> > > > > > > > > > > > >> >> >>> > > guess.
> >> > > > > > >> > > > > > > > > > > > >> >> >>> > >
> >> > > > > > >> > > > > > > > > > > > >> >> >>> > > The rationale for
> >> separating
> >> > > scan
> >> > > > > > >> metrics
> >> > > > > > >> > > and
> >> > > > > > >> > > > > > > > metastore
> >> > > > > > >> > > > > > > > > > > > >> >> persistence
> >> > > > > > >> > > > > > > > > > > > >> >> >>> is
> >> > > > > > >> > > > > > > > > > > > >> >> >>> > that
> >> > > > > > >> > > > > > > > > > > > >> >> >>> > > "cascading deletes"
> >> between
> >> > > them
> >> > > > > are
> >> > > > > > >> > hardly
> >> > > > > > >> > > > > ever
> >> > > > > > >> > > > > > > > > > required.
> >> > > > > > >> > > > > > > > > > > > >> >> >>> Furthermore,
> >> > > > > > >> > > > > > > > > > > > >> >> >>> > the
> >> > > > > > >> > > > > > > > > > > > >> >> >>> > > data and query patterns
> >> are
> >> > > very
> >> > > > > > >> > different
> >> > > > > > >> > > so
> >> > > > > > >> > > > > > > > different
> >> > > > > > >> > > > > > > > > > > > >> >> technologies
> >> > > > > > >> > > > > > > > > > > > >> >> >>> > might
> >> > > > > > >> > > > > > > > > > > > >> >> >>> > > be beneficial in each
> >> case.
> >> > > > > > >> > > > > > > > > > > > >> >> >>> > >
> >> > > > > > >> > > > > > > > > > > > >> >> >>> > > [1]
> >> > > > > > >> > > > > > > > > > > > >> >> >>> >
> >> > > > > > >> > > > > > > > > > > > >> >> >>>
> >> > > > > > >> > > > > > > > > > > > >> >>
> >> > > > > > >> > > > > > > > > > > > >>
> >> > > > > > >> > > > > > > > > > > >
> >> > > > > > >> > > > > > > > > > >
> >> > > > > > >> > > > > > > > > >
> >> > > > > > >> > > > > > > > >
> >> > > > > > >> > > > > > > >
> >> > > > > > >> > > > > > >
> >> > > > > > >> > > > > >
> >> > > > > > >> > > > >
> >> > > > > > >> > > >
> >> > > > > > >> > >
> >> > > > > > >> >
> >> > > > > > >>
> >> > > > > >
> >> > > > >
> >> > >
> >> >
> >> https://urldefense.com/v3/__https://lists.apache.org/thread/yto2wp982t43h1mqjwnslswhws5z47cy__;!!Iz9xO38YGHZK!5EuyFFkk3vhRWVIRvQAWBSQfpJkTMA9HxugzDwXmN0LPPqhEFxYkFRGVhtb8AqUwXtDh2OplcMnbMDHKxYDakNU$
> >> > >> > > >> > > > > > > > > > > > >> >> >>> > >
> >> > > > > > >> > > > > > > > > > > > >> >> >>> > > Cheers,
> >> > > > > > >> > > > > > > > > > > > >> >> >>> > > Dmitri.
> >> > > > > > >> > > > > > > > > > > > >> >> >>> > >
> >> > > > > > >> > > > > > > > > > > > >> >> >>> > > On Fri, Feb 27, 2026 at
> >> > > 6:19 PM
> >> > > > > > Anand
> >> > > > > > >> > Kumar
> >> > > > > > >> > > > > > > Sankaran
> >> > > > > > >> > > > > > > > > via
> >> > > > > > >> > > > > > > > > > > dev
> >> > > > > > >> > > > > > > > > > > > <
> >> > > > > > >> > > > > > > > > > > > >> >> >>> > > dev@polaris.apache.org>
> >> > > wrote:
> >> > > > > > >> > > > > > > > > > > > >> >> >>> > >
> >> > > > > > >> > > > > > > > > > > > >> >> >>> > > > Thanks all. This PR is
> >> > > merged
> >> > > > > now.
> >> > > > > > >> > > > > > > > > > > > >> >> >>> > > >
> >> > > > > > >> > > > > > > > > > > > >> >> >>> > > > Here are the follow-up
> >> > > features
> >> > > > > /
> >> > > > > > >> work
> >> > > > > > >> > > > > needed.
> >> > > > > > >> > > > > > > > These
> >> > > > > > >> > > > > > > > > > > were
> >> > > > > > >> > > > > > > > > > > > >> all
> >> > > > > > >> > > > > > > > > > > > >> >> >>> part of
> >> > > > > > >> > > > > > > > > > > > >> >> >>> > > the
> >> > > > > > >> > > > > > > > > > > > >> >> >>> > > > merged PR at some
> >> point in
> >> > > time
> >> > > > > > and
> >> > > > > > >> > were
> >> > > > > > >> > > > > > removed
> >> > > > > > >> > > > > > > to
> >> > > > > > >> > > > > > > > > > > reduce
> >> > > > > > >> > > > > > > > > > > > >> >> scope.
> >> > > > > > >> > > > > > > > > > > > >> >> >>> > > >
> >> > > > > > >> > > > > > > > > > > > >> >> >>> > > > Please let me know what
> >> > you
> >> > > > > think.
> >> > > > > > >> > > > > > > > > > > > >> >> >>> > > >
> >> > > > > > >> > > > > > > > > > > > >> >> >>> > > >
> >> > > > > > >> > > > > > > > > > > > >> >> >>> > > >   1.  A REST API to
> >> > paginate
> >> > > > > > through
> >> > > > > > >> > > table
> >> > > > > > >> > > > > > > metrics.
> >> > > > > > >> > > > > > > > > > This
> >> > > > > > >> > > > > > > > > > > > >> will be
> >> > > > > > >> > > > > > > > > > > > >> >> >>> > non-IRC
> >> > > > > > >> > > > > > > > > > > > >> >> >>> > > > standard addition.
> >> > > > > > >> > > > > > > > > > > > >> >> >>> > > >   2.  Utilities for
> >> > > managing old
> >> > > > > > >> > records,
> >> > > > > > >> > > > > > should
> >> > > > > > >> > > > > > > be
> >> > > > > > >> > > > > > > > > > > common
> >> > > > > > >> > > > > > > > > > > > >> with
> >> > > > > > >> > > > > > > > > > > > >> >> >>> events.
> >> > > > > > >> > > > > > > > > > > > >> >> >>> > > > There was some
> >> discussion
> >> > > that
> >> > > > > it
> >> > > > > > >> > belongs
> >> > > > > > >> > > > to
> >> > > > > > >> > > > > > the
> >> > > > > > >> > > > > > > > CLI.
> >> > > > > > >> > > > > > > > > > > > >> >> >>> > > >   3.  Separate
> >> datasource
> >> > > > > > (metrics,
> >> > > > > > >> > > events,
> >> > > > > > >> > > > > > even
> >> > > > > > >> > > > > > > > > other
> >> > > > > > >> > > > > > > > > > > > >> tables?).
> >> > > > > > >> > > > > > > > > > > > >> >> >>> > > >
> >> > > > > > >> > > > > > > > > > > > >> >> >>> > > >
> >> > > > > > >> > > > > > > > > > > > >> >> >>> > > > Anything else?
> >> > > > > > >> > > > > > > > > > > > >> >> >>> > > >
> >> > > > > > >> > > > > > > > > > > > >> >> >>> > > > -
> >> > > > > > >> > > > > > > > > > > > >> >> >>> > > > Anand
> >> > > > > > >> > > > > > > > > > > > >> >> >>> > > >
> >> > > > > > >> > > > > > > > > > > > >> >> >>> > > >
> >> > > > > > >> > > > > > > > > > > > >> >> >>> > >
> >> > > > > > >> > > > > > > > > > > > >> >> >>> >
> >> > > > > > >> > > > > > > > > > > > >> >> >>> >
> >> > > > > > >> > > > > > > > > > > > >> >> >>>
> >> > > > > > >> > > > > > > > > > > > >> >> >>
> >> > > > > > >> > > > > > > > > > > > >> >>
> >> > > > > > >> > > > > > > > > > > > >> >
> >> > > > > > >> > > > > > > > > > > > >>
> >> > > > > > >> > > > > > > > > > > > >
> >> > > > > > >> > > > > > > > > > > >
> >> > > > > > >> > > > > > > > > > >
> >> > > > > > >> > > > > > > > > >
> >> > > > > > >> > > > > > > > >
> >> > > > > > >> > > > > > > >
> >> > > > > > >> > > > > > >
> >> > > > > > >> > > > > >
> >> > > > > > >> > > > >
> >> > > > > > >> > > >
> >> > > > > > >> > >
> >> > > > > > >> >
> >> > > > > > >>
> >> > > > > > >
> >> > > > > >
> >> > > > >
> >> > >
> >> >
> >> >
> >>
> >

Re: Proposal for REST endpoints for table metrics and events

[Dmitri Bourlatchkov <di...@apache.org>]

Hi All,

Any remaining blockers on [4115]?

Given that the review was very long, please restate any remaining concerns
(here or in GH).

Otherwise, I propose merging [4115] on Oct 1.

[4115] https://github.com/apache/polaris/pull/4115

Thanks,
Dmitri.

On Wed, Sep 23, 2026 at 7:53 PM Dmitri Bourlatchkov <di...@apache.org>
wrote:

> Hi EJ,
>
> Thanks for the meeting recap!
>
> All:
>
> Worth noting: PR [4115] also contains Management API changes that were not
> anticipated initially and did not get an API change review until now. These
> are small changes to add new grant types for metrics RBAC..
>
> I've reviewed and approved PR [4115] in GH.
>
> [4115] https://github.com/apache/polaris/pull/4115
>
> Cheers,
> Dmitri.
>
> On Wed, Aug 12, 2026 at 5:21 PM EJ Wang <ej...@gmail.com>
> wrote:
>
>> Hi folks,
>>
>> Notes from today's metrics architecture sync are in the usual doc:
>>
>> https://docs.google.com/document/d/100h7c4damrUzVuquYbBHM0EvA4LSWuW2IT2dN_7nYVA/edit?pli=1&tab=t.18iv1m7p6i6b
>>
>> Highlights:
>>
>> Module placement. core/ now holds the entity data model only. A new spi/
>> takes every other shared contract: feature SPIs, feature non-SPI
>> contracts,
>> substrate SPIs, substrate non-SPI contracts, and contract-required types.
>> That settles the question PR #5204 reopened, by placing into a new SPI
>> module. Note that it changes slightly towards the referenced WIP RFC [1]
>> recommendations: what proposed to place in core/, now should go to spi/
>> except for the entity data model.
>>
>> Metrics PRs. #5068 merged. #5204 needs reshaping onto the split. #4115 and
>> #4756 stay open with action on the author to follow up on review comments.
>>
>> [1]
>>
>> https://docs.google.com/document/d/1mj3vGpix9zTDTFFJKtWT5ZxI3PTGI1YNb5xFqyWrOkc/edit?tab=t.0
>>
>> On Wed, Jul 15, 2026 at 11:01 AM Anand Kumar Sankaran via dev <
>> dev@polaris.apache.org> wrote:
>>
>> > Thanks EJ.  My plan is to introduce PR0 in a new branch / PR and
>> repurpose
>> > #4115 and #4756 for PR1 and PR2 to maintain all the valuable discussions
>> > that happened there.
>> >
>> > From: EJ Wang <ej...@gmail.com>
>> > Date: Wednesday, July 15, 2026 at 10:41 AM
>> > To: dev@polaris.apache.org <de...@polaris.apache.org>
>> > Subject: Re: Proposal for REST endpoints for table metrics and events
>> >
>> > Hi folks, Thanks again for the discussion today. I updated the sync doc
>> > with notes from today's metrics sync: https: //urldefense.
>> com/v3/__https:
>> > //docs. google.
>> >
>> com/document/d/100h7c4damrUzVuquYbBHM0EvA4LSWuW2IT2dN_7nYVA/edit?pli=1&tab=t.
>> >
>> 2ahd7wze5f__;!!Iz9xO38YGHZK!4rnlVIqBQQQ6xpGe1PMV58edOS_cGrKoqdfhw4qSp_JQCavvEjaxXk0OwGPnxyEcO60AzKQK_XDtp7H3DdkFXcrXJwAz1g0$
>> >
>> >
>> > Hi folks,
>> >
>> > Thanks again for the discussion today. I updated the sync doc with notes
>> > from today's metrics sync:
>> >
>> >
>> https://urldefense.com/v3/__https://docs.google.com/document/d/100h7c4damrUzVuquYbBHM0EvA4LSWuW2IT2dN_7nYVA/edit?pli=1&tab=t.2ahd7wze5f__;!!Iz9xO38YGHZK!4rnlVIqBQQQ6xpGe1PMV58edOS_cGrKoqdfhw4qSp_JQCavvEjaxXk0OwGPnxyEcO60AzKQK_XDtp7H3DdkFXcrXJwAz1g0$
>> >
>> > See highlights below:
>> >
>> > In previous meetings, EJ proposed a two prs framing to push forward the
>> > metrics query work in polaris. Dmitri pointed out a subtle point that
>> pr2
>> > is currently also trying to do the persistence related code placement
>> > refactor. For the sake of straightforward review process, EJ, Dmitri and
>> > Anand (the author) agreed to separate the persistence refactoring into
>> pr0,
>> > and reframing the pr scopes as follow:
>> >
>> >    1.
>> >
>> >    Pr0: refactor irc metrics spi
>> >    1.
>> >
>> >       SPI should stay in core/ with metric type enveloped
>> >       2.
>> >
>> >       default spi impl, no-op, in extensions/ - default selection at
>> >       runtime/
>> >       3.
>> >
>> >       move metrics persistence and correspondings to extensions/ as an
>> >       alternative SPI impl
>> >       2.
>> >
>> >    Pr1: rest api for query
>> >
>> >
>> https://urldefense.com/v3/__https://github.com/apache/polaris/pull/4115/changes__;!!Iz9xO38YGHZK!4rnlVIqBQQQ6xpGe1PMV58edOS_cGrKoqdfhw4qSp_JQCavvEjaxXk0OwGPnxyEcO60AzKQK_XDtp7H3DdkFXcrXFCbEbRk$
>> >    1.
>> >
>> >       rest api spec in extensions/ as optional spec
>> >       2.
>> >
>> >       SPI in extensions/ as optional spec spi
>> >       3.
>> >
>> >       default spi impl in extensions/ no-op
>> >       4.
>> >
>> >       handler related in runtime/ - thin adaptor from http shape to spi
>> >       shape
>> >       3.
>> >
>> >    Pr2: jdbc impl for query
>> >
>> https://urldefense.com/v3/__https://github.com/apache/polaris/pull/4756__;!!Iz9xO38YGHZK!4rnlVIqBQQQ6xpGe1PMV58edOS_cGrKoqdfhw4qSp_JQCavvEjaxXk0OwGPnxyEcO60AzKQK_XDtp7H3DdkFXcrXRVMBzqE$
>> >    1.
>> >
>> >       alternative spi impl in extensions/ -> query with jdbc
>> >
>> >
>> > Thanks,
>> > -ej
>> >
>> > On Wed, Jul 1, 2026 at 8:01 AM Alexandre Dutra <ad...@apache.org>
>> wrote:
>> >
>> > > Hi all,
>> > >
>> > > > While the current implementation stores metrics in the metastore,
>> other
>> > > deployments
>> > > may choose a KV store, a dedicated metrics warehouse, or another
>> backend
>> > > entirely. In those cases, exposing a Polaris REST API over a specific
>> > > schema feels like coupling the API to an implementation detail.
>> > >
>> > > I wouldn't say to an implementation detail, but rather to a particular
>> > > setup / configuration. I think that's fine. If you don't store
>> > > metrics, you don't read metrics. Similar situations will certainly
>> > > happen in the future, e.g. if we ever introduce an API to read events,
>> > > that would only work if events are being persisted in the first place.
>> > > Again I think that's a fair trade-off.
>> > >
>> > > Thanks,
>> > > Alex
>> > >
>> > > On Wed, Jul 1, 2026 at 7:23 AM Dmitri Bourlatchkov <di...@apache.org>
>> > > wrote:
>> > > >
>> > > > Hi Yufei,
>> > > >
>> > > > From my POV, it is fine for Polaris to have multiple REST API
>> surfaces
>> > > with
>> > > > different features and different degrees of "readiness". Each API
>> can
>> > > > evolve fairly independently. With proper modularization (which Anand
>> > > did),
>> > > > changes to the main Polaris code and these extra modules should
>> > normally
>> > > be
>> > > > non-intrusive to each other.
>> > > >
>> > > > I assume Anand is going to be the first user of this API. However,
>> if
>> > we
>> > > do
>> > > > not merge it we'll never know whether other users are interested. I
>> do
>> > > not
>> > > > think many end users read the dev ML, but they certainly read the
>> docs,
>> > > > code and release notes.
>> > > >
>> > > > There is no risk for Polaris as a project in exposing this
>> experimental
>> > > > API. If usage dwindles over time, we can remove it. If it gains a
>> > > > considerable user / contributor base, we will promote it to a
>> > > "first-class"
>> > > > (non-beta) API.
>> > > >
>> > > > Cheers,
>> > > > Dmitri.
>> > > >
>> > > > On Tue, Jun 30, 2026 at 7:53 PM Yufei Gu <fl...@gmail.com>
>> wrote:
>> > > >
>> > > > > Hi all,
>> > > > >
>> > > > > Thanks for driving this work forward. Building this as a POC makes
>> > > perfect
>> > > > > sense if the goal is to validate the idea and gather feedback.
>> Should
>> > > we
>> > > > > merge a POC into the main branch before agreeing on the overall
>> > > direction?
>> > > > > To me, that feels like a separate discussion from whether the
>> > > > > implementation itself looks reasonable.
>> > > > >
>> > > > > One concern that I don't think has been fully addressed is
>> whether we
>> > > need
>> > > > > to expose metrics consumption through a Polaris REST endpoint at
>> all.
>> > > > >
>> > > > > My concern is less about the API itself and more about the
>> boundary
>> > > Polaris
>> > > > > should own. The proposed REST API is tied to a particular
>> persistence
>> > > > > schema, but that schema is just one possible implementation. While
>> > the
>> > > > > current implementation stores metrics in the metastore, other
>> > > deployments
>> > > > > may choose a KV store, a dedicated metrics warehouse, or another
>> > > backend
>> > > > > entirely. In those cases, exposing a Polaris REST API over a
>> specific
>> > > > > schema feels like coupling the API to an implementation detail.
>> > > > >
>> > > > > Polaris isn't intended to become a full metrics system. I'd
>> expect it
>> > > to
>> > > > > focus on producing metrics and let deployments choose the most
>> > > appropriate
>> > > > > storage and consumption model for their environment.
>> > > > >
>> > > > > I'm not opposed to exploring this direction, but we should first
>> > align
>> > > on
>> > > > > whether exposing a REST API through Polaris is the right long term
>> > > approach
>> > > > > before merging it into the main branch.
>> > > > >
>> > > > > Thanks,
>> > > > > Yufei
>> > > > >
>> > > > > Yufei
>> > > > >
>> > > > >
>> > > > > On Mon, Jun 29, 2026 at 9:27 AM Dmitri Bourlatchkov <
>> > dimas@apache.org>
>> > > > > wrote:
>> > > > >
>> > > > > > Hi All,
>> > > > > >
>> > > > > > It looks like Anand addressed all action items in [4115].
>> > > > > >
>> > > > > > Please review the latest state of the code. The PR LGTM and I
>> > > approved in
>> > > > > > GH from my side.
>> > > > > >
>> > > > > > [4115]
>> >
>> https://urldefense.com/v3/__https://github.com/apache/polaris/pull/4115__;!!Iz9xO38YGHZK!4rnlVIqBQQQ6xpGe1PMV58edOS_cGrKoqdfhw4qSp_JQCavvEjaxXk0OwGPnxyEcO60AzKQK_XDtp7H3DdkFXcrXtFRscKs$
>> > > > > >
>> > > > > > Thanks,
>> > > > > > Dmitri.
>> > > > > >
>> > > > > > On Thu, Jun 25, 2026 at 1:56 PM Dmitri Bourlatchkov <
>> > > dimas@apache.org>
>> > > > > > wrote:
>> > > > > >
>> > > > > > > Hi All,
>> > > > > > >
>> > > > > > > Recap of the discussion in today's Community Sync:
>> > > > > > >
>> > > > > > > * The modular approach for Metrics SPI(s) and implementation
>> > seems
>> > > to
>> > > > > > have
>> > > > > > > community consensus (please comment if you still have
>> concerns)
>> > > > > > >
>> > > > > > > * What modules / feature flags are enabled in Apache Polaris
>> > > images by
>> > > > > > > default will be discussed later (when the feature is close to
>> > > > > > completion).
>> > > > > > >
>> > > > > > > In any case, users will be able to turn the full feature
>> on/off
>> > at
>> > > > > their
>> > > > > > > discretion, regardless of the default flag values.
>> > > > > > >
>> > > > > > > * It is generally preferable to have smaller PRs and introduce
>> > new
>> > > > > > > features gradually to simplify reviews.
>> > > > > > >
>> > > > > > > What this means for the Metrics API, IMHO is:
>> > > > > > >
>> > > > > > > a) The scope of PR 4115 is probably ok as it stands right
>> now. If
>> > > > > someone
>> > > > > > > wants to propose splitting it further, please comment.
>> > > > > > >
>> > > > > > > b) After 4115 is merged, Metrics Persistence PRs will follow,
>> > with
>> > > > > > smaller
>> > > > > > > scope per PR as appropriate for the nature of the code
>> changes.
>> > > > > > >
>> > > > > > > * The major concern was about exposing the new Metrics REST
>> API
>> > to
>> > > end
>> > > > > > > users. The nature of this concern was about avoiding the
>> (false)
>> > > > > > impression
>> > > > > > > that the new API is a firm and adopted Polaris specification.
>> > > > > > >
>> > > > > > > I think this concern is reasonable.
>> > > > > > >
>> > > > > > > I propose that Anand update the documentation sections of PR
>> 4115
>> > > to
>> > > > > > > emphasise that the API is currently an experimental / POC API
>> at
>> > > this
>> > > > > > time
>> > > > > > > and is subject to change (including breaking changes). I
>> believe
>> > it
>> > > > > > already
>> > > > > > > has the "beta" label. So this request is mostly for doc
>> changes
>> > to
>> > > > > > clarify
>> > > > > > > user expectations, since the "beta" label alone may not be
>> > > interpreted
>> > > > > > the
>> > > > > > > same way by all users.
>> > > > > > >
>> > > > > > > I hope this update will resolve all remaining concerns with
>> 4115
>> > > and
>> > > > > > allow
>> > > > > > > for it to be merged so that the community will be able to
>> proceed
>> > > to
>> > > > > > > follow-up PRs (as noted above).
>> > > > > > >
>> > > > > > > Please add/correct me if I missed anything relevant.
>> > > > > > >
>> > > > > > > Thanks,
>> > > > > > > Dmitri.
>> > > > > > >
>> > > > > > > On Wed, Jun 24, 2026 at 8:11 PM Yufei Gu <
>> flyrain000@gmail.com>
>> > > wrote:
>> > > > > > >
>> > > > > > >> I do not think we have agreement on the query REST APIs yet.
>> > > > > > >>
>> > > > > > >> In particular, the current query API shape seems highly
>> coupled
>> > > to the
>> > > > > > >> *example[1]* metrics in the Iceberg REST spec. I do not think
>> > > that is
>> > > > > a
>> > > > > > >> good way to design a REST API contract. For example, fields
>> like
>> > > > > > >> resultDataFiles, resultDeleteFiles, totalFileSizeBytes,
>> > > > > > >> totalDataManifests,
>> > > > > > >> totalDeleteManifests, and scannedDataManifests are very
>> specific
>> > > to
>> > > > > one
>> > > > > > >> style of scan metric. That makes the API look more like a
>> direct
>> > > > > > >> projection
>> > > > > > >> of the current example payload than a general query model.
>> > > > > > >>
>> > > > > > >> My preference is that Polaris should not define or strongly
>> > opine
>> > > on
>> > > > > the
>> > > > > > >> Iceberg scan or commit metrics query REST API at this point.
>> > > Polaris
>> > > > > can
>> > > > > > >> resolve and authorize the table, accept the metrics report,
>> and
>> > > > > delegate
>> > > > > > >> it
>> > > > > > >> to the configured reporting path. Durable storage, filtering,
>> > > > > > dashboards,
>> > > > > > >> and query APIs should remain downstream impl. choices unless
>> and
>> > > until
>> > > > > > the
>> > > > > > >> Iceberg community standardizes a query contract, which I
>> doubt
>> > > will
>> > > > > > >> happen.
>> > > > > > >>
>> > > > > > >> So I do not think the intake path has to be blocked, but I
>> would
>> > > > > prefer
>> > > > > > >> not
>> > > > > > >> to merge a Polaris specific query API that may be hard to
>> evolve
>> > > > > later.
>> > > > > > >>
>> > > > > > >> 1.
>> > > > > > >>
>> > > > > > >>
>> > > > > >
>> > > > >
>> > >
>> >
>> https://urldefense.com/v3/__https://github.com/apache/iceberg/blob/2a6c556c0c883c04c6d3fbd68e6aeda36b91e0aa/open-api/rest-catalog-open-api.yaml*L4152__;Iw!!Iz9xO38YGHZK!4rnlVIqBQQQ6xpGe1PMV58edOS_cGrKoqdfhw4qSp_JQCavvEjaxXk0OwGPnxyEcO60AzKQK_XDtp7H3DdkFXcrXCDsqYGo$
>> > > > > > >>
>> > > > > > >> Yufei
>> > > > > > >>
>> > > > > > >>
>> > > > > > >> On Tue, Jun 23, 2026 at 9:30 AM Dmitri Bourlatchkov <
>> > > dimas@apache.org
>> > > > > >
>> > > > > > >> wrote:
>> > > > > > >>
>> > > > > > >> > Hi All,
>> > > > > > >> >
>> > > > > > >> > So, what are the blockers for merging [4115], if any?
>> > > > > > >> >
>> > > > > > >> > [4115]
>> >
>> https://urldefense.com/v3/__https://github.com/apache/polaris/pull/4115__;!!Iz9xO38YGHZK!4rnlVIqBQQQ6xpGe1PMV58edOS_cGrKoqdfhw4qSp_JQCavvEjaxXk0OwGPnxyEcO60AzKQK_XDtp7H3DdkFXcrXtFRscKs$
>> > > > > > >> >
>> > > > > > >> > Thanks,
>> > > > > > >> > Dmitri.
>> > > > > > >> >
>> > > > > > >> > On Wed, Jun 17, 2026 at 5:28 PM EJ Wang <
>> > > > > > ej.wang.dev.interest@gmail.com
>> > > > > > >> >
>> > > > > > >> > wrote:
>> > > > > > >> >
>> > > > > > >> > > Hi folks,
>> > > > > > >> > >
>> > > > > > >> > > Thanks again for the discussion today. I updated the sync
>> > doc
>> > > with
>> > > > > > >> notes
>> > > > > > >> > > from today's metrics sync:
>> > > > > > >> > >
>> > > > > > >> > >
>> > > > > > >> > >
>> > > > > > >> >
>> > > > > > >>
>> > > > > >
>> > > > >
>> > >
>> >
>> https://urldefense.com/v3/__https://docs.google.com/document/d/100h7c4damrUzVuquYbBHM0EvA4LSWuW2IT2dN_7nYVA/edit?tab=t.ezk0rgdx0c6m__;!!Iz9xO38YGHZK!4rnlVIqBQQQ6xpGe1PMV58edOS_cGrKoqdfhw4qSp_JQCavvEjaxXk0OwGPnxyEcO60AzKQK_XDtp7H3DdkFXcrX83IZkm0$
>> > > > > > >> > >
>> > > > > > >> > > A few highlights:
>> > > > > > >> > >
>> > > > > > >> > >    - No objection to keeping the default metrics behavior
>> > > small:
>> > > > > > >> > >    no-op/log-only is enough as the built-in default.
>> > > > > > >> > >    - Durable storage, event forwarding, external queues,
>> > > > > dashboards,
>> > > > > > >> and
>> > > > > > >> > >    custom filtering should be implementation choices
>> behind
>> > > the
>> > > > > > >> metrics
>> > > > > > >> > >    reporting path.
>> > > > > > >> > >    - For PR #4115, the REST layer should
>> > resolve/authz/accept
>> > > the
>> > > > > > >> report,
>> > > > > > >> > >    then delegate to the selected metrics implementation.
>> It
>> > > should
>> > > > > > not
>> > > > > > >> > own
>> > > > > > >> > >    durable storage semantics.
>> > > > > > >> > >    - The rough boundary we discussed is: spec owns the
>> wire
>> > > > > > contract,
>> > > > > > >> > >    runtime owns framework wiring, API/contract modules
>> own
>> > > > > > >> > provider-facing
>> > > > > > >> > >    contracts, and extensions own replaceable
>> > implementations.
>> > > > > > >> > >    - OpenLineage has a similar path-naming question. We
>> > should
>> > > > > avoid
>> > > > > > >> > >    occupying a generic lineage path too early if we want
>> > room
>> > > for
>> > > > > > >> other
>> > > > > > >> > >    lineage systems later.
>> > > > > > >> > >
>> > > > > > >> > > I believe the boundary rules previewed in sync meeting
>> are
>> > > useful
>> > > > > > for
>> > > > > > >> > > reviewing PR #4115, but it is not
>> > > > > > >> > > a final module-layout proposal.
>> > > > > > >> > >
>> > > > > > >> > > For metrics specifically, I think the target remains:
>> > > > > > >> > >    runtime
>> > > > > > >> > >      -> metrics reporting/emitting contract
>> > > > > > >> > >      -> default implementation
>> > > > > > >> > >           -> lower implementation details, if needed
>> > > > > > >> > >
>> > > > > > >> > > That keeps the default simple, while allowing durable
>> JDBC,
>> > > > > > >> event-backed,
>> > > > > > >> > > or external-queue implementations to be added separately.
>> > > > > > >> > >
>> > > > > > >> > > We also touched on the split between Iceberg metrics
>> > emitting
>> > > and
>> > > > > > >> > > Polaris-owned metrics querying. The querying side can
>> keep
>> > > > > evolving,
>> > > > > > >> > > especially for dashboard or generic-table use cases, but
>> I
>> > do
>> > > not
>> > > > > > >> think
>> > > > > > >> > > that needs to block the intake path in this PR.
>> > > > > > >> > >
>> > > > > > >> > > Thanks,
>> > > > > > >> > > -ej
>> > > > > > >> > >
>> > > > > > >> > > On Tue, Jun 16, 2026 at 2:17 PM Yufei Gu <
>> > > flyrain000@gmail.com>
>> > > > > > >> wrote:
>> > > > > > >> > >
>> > > > > > >> > > > Thanks for chiming in, EJ. Agreed that the default
>> battery
>> > > > > should
>> > > > > > >> stay
>> > > > > > >> > > > small. I'm leaning toward not including metrics
>> > persistence
>> > > in
>> > > > > the
>> > > > > > >> > > default
>> > > > > > >> > > > battery.
>> > > > > > >> > > >
>> > > > > > >> > > > Yufei
>> > > > > > >> > > >
>> > > > > > >> > > >
>> > > > > > >> > > > On Thu, Jun 11, 2026 at 8:30 PM EJ Wang <
>> > > > > > >> > ej.wang.dev.interest@gmail.com>
>> > > > > > >> > > > wrote:
>> > > > > > >> > > >
>> > > > > > >> > > > > Hi Yufei,
>> > > > > > >> > > > >
>> > > > > > >> > > > > Thanks for connecting this back to the REST endpoint
>> > > > > proposal. I
>> > > > > > >> > > replied
>> > > > > > >> > > > > with the fuller version on the event-forwarding
>> thread,
>> > > but
>> > > > > > >> wanted to
>> > > > > > >> > > add
>> > > > > > >> > > > > the shorter version here too.
>> > > > > > >> > > > >
>> > > > > > >> > > > > I agree that metrics reporting/emitting is the right
>> > > > > conceptual
>> > > > > > >> > > boundary,
>> > > > > > >> > > > > and I also think the event/listener path is a good
>> > > > > > implementation
>> > > > > > >> > > > direction
>> > > > > > >> > > > > to explore. The distinction I want to keep clear is
>> > > default
>> > > > > > >> battery
>> > > > > > >> > vs
>> > > > > > >> > > > > extension implementation.
>> > > > > > >> > > > >
>> > > > > > >> > > > > For the REST/API side, I would keep the semantics
>> > narrow:
>> > > > > > Polaris
>> > > > > > >> > > > resolves
>> > > > > > >> > > > > and authorizes the table, accepts the Iceberg
>> > scan/commit
>> > > > > > metrics
>> > > > > > >> > > report
>> > > > > > >> > > > > into the configured reporting/emitting path, and
>> returns
>> > > 204.
>> > > > > > That
>> > > > > > >> > 204
>> > > > > > >> > > > > should mean Polaris accepted the report into the
>> > ingestion
>> > > > > path;
>> > > > > > >> it
>> > > > > > >> > > > should
>> > > > > > >> > > > > not imply durable storage.
>> > > > > > >> > > > >
>> > > > > > >> > > > > Durable storage, event forwarding, external telemetry
>> > > routing,
>> > > > > > >> > > filtering,
>> > > > > > >> > > > > and retention should sit behind that
>> reporting/emitting
>> > > > > boundary
>> > > > > > >> as
>> > > > > > >> > > > > implementation-layer behavior. A durable JDBC path
>> can
>> > be
>> > > one
>> > > > > > >> named
>> > > > > > >> > > > > extension implementation. An event/listener
>> forwarding
>> > > path
>> > > > > can
>> > > > > > be
>> > > > > > >> > > > another
>> > > > > > >> > > > > named extension implementation.
>> > > > > > >> > > > >
>> > > > > > >> > > > > So I am +1 on the event-forwarding idea as a
>> non-default
>> > > > > > extension
>> > > > > > >> > > > > implementation. I just would not make it the default
>> > > battery
>> > > > > or
>> > > > > > >> the
>> > > > > > >> > > core
>> > > > > > >> > > > > API shape. The default battery can stay small and
>> safe,
>> > > while
>> > > > > > >> > > deployments
>> > > > > > >> > > > > that need forwarding, dashboards, or durable storage
>> can
>> > > opt
>> > > > > > into
>> > > > > > >> the
>> > > > > > >> > > > > implementation that matches their operational model.
>> > > > > > >> > > > >
>> > > > > > >> > > > > Thanks,
>> > > > > > >> > > > > -ej
>> > > > > > >> > > > >
>> > > > > > >> > > > > On Wed, Jun 10, 2026 at 11:43 AM Yufei Gu <
>> > > > > flyrain000@gmail.com
>> > > > > > >
>> > > > > > >> > > wrote:
>> > > > > > >> > > > >
>> > > > > > >> > > > > > Thanks EJ. I agree that the reporting/emitting
>> > boundary
>> > > > > feels
>> > > > > > >> like
>> > > > > > >> > > the
>> > > > > > >> > > > > > right SPI boundary, but I wonder if we can simplify
>> > this
>> > > > > even
>> > > > > > >> > > further.
>> > > > > > >> > > > > >
>> > > > > > >> > > > > > Iceberg scan and commit metrics look very similar
>> to
>> > > events
>> > > > > to
>> > > > > > >> me.
>> > > > > > >> > > They
>> > > > > > >> > > > > are
>> > > > > > >> > > > > > append only, asynchronously consumed, often
>> forwarded
>> > to
>> > > > > > >> external
>> > > > > > >> > > > > systems,
>> > > > > > >> > > > > > and have similar retention and cleanup
>> requirements.
>> > > More
>> > > > > > >> details
>> > > > > > >> > can
>> > > > > > >> > > > be
>> > > > > > >> > > > > > found in this thread [1]. In that case, Polaris may
>> > only
>> > > > > need
>> > > > > > to
>> > > > > > >> > > accept
>> > > > > > >> > > > > the
>> > > > > > >> > > > > > report and emit an event through the existing event
>> > > > > framework.
>> > > > > > >> > > > Filtering,
>> > > > > > >> > > > > > async delivery, custom sinks, and retention
>> mechanisms
>> > > could
>> > > > > > >> then
>> > > > > > >> > be
>> > > > > > >> > > > > shared
>> > > > > > >> > > > > > instead of introducing a separate metrics specific
>> > > extension
>> > > > > > >> layer
>> > > > > > >> > > and
>> > > > > > >> > > > > SPI.
>> > > > > > >> > > > > >
>> > > > > > >> > > > > > The main requirement I still see is querying
>> metrics
>> > > through
>> > > > > > >> > Polaris.
>> > > > > > >> > > > If
>> > > > > > >> > > > > we
>> > > > > > >> > > > > > want that battery included experience, we could
>> work
>> > on
>> > > a
>> > > > > > >> > persistence
>> > > > > > >> > > > > > listener implementation that selectively saves
>> metrics
>> > > and
>> > > > > > >> expose
>> > > > > > >> > it
>> > > > > > >> > > > via
>> > > > > > >> > > > > > the IRC event endpoint which seems pretty close in
>> > > Iceberg
>> > > > > > >> > > > > > community. Another option I like more is to allow
>> > users
>> > > to
>> > > > > > >> develop
>> > > > > > >> > > > their
>> > > > > > >> > > > > > own listener, so they can persist scan/commit
>> metrics
>> > > to any
>> > > > > > >> > systems
>> > > > > > >> > > > they
>> > > > > > >> > > > > > prefer. In general, that feels like an
>> implementation
>> > > choice
>> > > > > > on
>> > > > > > >> top
>> > > > > > >> > > of
>> > > > > > >> > > > > the
>> > > > > > >> > > > > > event framework rather than something that needs to
>> > > shape
>> > > > > the
>> > > > > > >> core
>> > > > > > >> > > > > > architecture.
>> > > > > > >> > > > > >
>> > > > > > >> > > > > > 1.
>> > > > > > >> >
>> > >
>> >
>> https://urldefense.com/v3/__https://lists.apache.org/thread/x9j8nscvy8hq61tyn01mj8yp6n9of0kp__;!!Iz9xO38YGHZK!4rnlVIqBQQQ6xpGe1PMV58edOS_cGrKoqdfhw4qSp_JQCavvEjaxXk0OwGPnxyEcO60AzKQK_XDtp7H3DdkFXcrXGcGDgks$
>> > > > > > >> > > > > >
>> > > > > > >> > > > > > Thanks,
>> > > > > > >> > > > > > Yufei
>> > > > > > >> > > > > >
>> > > > > > >> > > > > >
>> > > > > > >> > > > > > On Thu, Jun 4, 2026 at 5:14 PM EJ Wang <
>> > > > > > >> > > ej.wang.dev.interest@gmail.com
>> > > > > > >> > > > >
>> > > > > > >> > > > > > wrote:
>> > > > > > >> > > > > >
>> > > > > > >> > > > > > > Good point. I agree the existing
>> > > PolarisMetricsReporter is
>> > > > > > >> > already
>> > > > > > >> > > > very
>> > > > > > >> > > > > > > close to the right conceptual boundary. I am not
>> > > > > proposing a
>> > > > > > >> > second
>> > > > > > >> > > > > > > parallel reporter concept. The distinction I am
>> > > trying to
>> > > > > > >> make is
>> > > > > > >> > > > > > narrower:
>> > > > > > >> > > > > > >
>> > > > > > >> > > > > > > Current state:
>> > > > > > >> > > > > > >
>> > > > > > >> > > > > > >    - PolarisMetricsReporter is the existing
>> metrics
>> > > > > > reporting
>> > > > > > >> > hook.
>> > > > > > >> > > > > > >    - But it currently lives under
>> runtime/service.
>> > > > > > >> > > > > > >    - Its contract is documented around
>> Quarkus/CDI
>> > > > > discovery
>> > > > > > >> and
>> > > > > > >> > > > > > > @Identifier
>> > > > > > >> > > > > > >    selection.
>> > > > > > >> > > > > > >    - The default implementation is log-only.
>> > > > > > >> > > > > > >    - The durable implementation writes through
>> > > > > > >> > > PolarisMetricsManager
>> > > > > > >> > > > > and
>> > > > > > >> > > > > > >    MetricsPersistence.
>> > > > > > >> > > > > > >    - MetricsPersistence is currently inherited by
>> > > > > > >> > BasePersistence.
>> > > > > > >> > > > > > >
>> > > > > > >> > > > > > > My proposal:
>> > > > > > >> > > > > > >
>> > > > > > >> > > > > > >    - Keep the reporting/emitting boundary as the
>> > SPI.
>> > > > > > >> > > > > > >    - Revise that contract in a framework-agnostic
>> > > optional
>> > > > > > >> > Iceberg
>> > > > > > >> > > > > > *metrics
>> > > > > > >> > > > > > >    extension API layer*.
>> > > > > > >> > > > > > >    - Keep runtime/service responsible for REST
>> > > ingestion,
>> > > > > > >> > > table/authz
>> > > > > > >> > > > > > >    validation, and runtime wiring.
>> > > > > > >> > > > > > >    - Keep the battery implementation log-only in
>> the
>> > > > > metrics
>> > > > > > >> > > > extension
>> > > > > > >> > > > > > API
>> > > > > > >> > > > > > >    layer, *not under runtime/service*.
>> > > > > > >> > > > > > >    - Treat durable JDBC metrics as one
>> > implementation
>> > > of
>> > > > > the
>> > > > > > >> > > > reporting
>> > > > > > >> > > > > > SPI.
>> > > > > > >> > > > > > >    - Decompose the MetricsPersistence and
>> > > BasePersistence
>> > > > > > >> > coupling
>> > > > > > >> > > as
>> > > > > > >> > > > > > >    durable implementation detail.
>> > > > > > >> > > > > > >
>> > > > > > >> > > > > > > So the difference is not “new reporter concept vs
>> > old
>> > > > > > reporter
>> > > > > > >> > > > > concept.”
>> > > > > > >> > > > > > > but:
>> > > > > > >> > > > > > >
>> > > > > > >> > > > > > >
>> > > > > > >> > > > > > >    - PolarisMetricsReporter today =
>> runtime/service
>> > > > > > CDI-shaped
>> > > > > > >> > > hook.
>> > > > > > >> > > > > > >    - Target reporting SPI = same conceptual
>> > boundary,
>> > > but
>> > > > > > >> placed
>> > > > > > >> > in
>> > > > > > >> > > > the
>> > > > > > >> > > > > > >    right extension API layer and kept
>> > > framework-agnostic.
>> > > > > > >> > > > > > >
>> > > > > > >> > > > > > > For the current metrics PR, I think the minimal
>> > > > > > architectural
>> > > > > > >> > > cleanup
>> > > > > > >> > > > > > > before merge is to *put the reporting SPI
>> contract
>> > > and the
>> > > > > > >> > battery
>> > > > > > >> > > > > > default
>> > > > > > >> > > > > > > in the right place*. The durable JDBC
>> > implementation,
>> > > > > > >> > event-backed
>> > > > > > >> > > > > async
>> > > > > > >> > > > > > > implementation, external queue support, and
>> deeper
>> > > > > > persistence
>> > > > > > >> > > > cleanup
>> > > > > > >> > > > > > can
>> > > > > > >> > > > > > > continue as follow-ups, as long as we do not lock
>> > the
>> > > SPI
>> > > > > > >> > boundary
>> > > > > > >> > > to
>> > > > > > >> > > > > > > BasePersistence or to runtime/service wiring.
>> > > > > > >> > > > > > >
>> > > > > > >> > > > > > > -ej
>> > > > > > >> > > > > > >
>> > > > > > >> > > > > > > On Wed, Jun 3, 2026 at 3:00 PM Dmitri
>> Bourlatchkov <
>> > > > > > >> > > dimas@apache.org
>> > > > > > >> > > > >
>> > > > > > >> > > > > > > wrote:
>> > > > > > >> > > > > > >
>> > > > > > >> > > > > > > > Hi EJ,
>> > > > > > >> > > > > > > >
>> > > > > > >> > > > > > > > Thanks for the recap / summary!
>> > > > > > >> > > > > > > >
>> > > > > > >> > > > > > > > > Do folks agree that the stable SPI boundary
>> > > should be
>> > > > > > >> > > > > > > > metrics reporting/emitting, not metrics
>> > persistence?
>> > > > > > >> > > > > > > >
>> > > > > > >> > > > > > > > SGTM.
>> > > > > > >> > > > > > > >
>> > > > > > >> > > > > > > > > Does an optional Iceberg metrics extension
>> API
>> > > layer
>> > > > > > sound
>> > > > > > >> > like
>> > > > > > >> > > > the
>> > > > > > >> > > > > > > right
>> > > > > > >> > > > > > > > home for this SPI?
>> > > > > > >> > > > > > > >
>> > > > > > >> > > > > > > > How is that different from current
>> > > > > PolarisMetricsReporter?
>> > > > > > >> > > > > > > >
>> > > > > > >> > > > > > > > > Should the current durable metrics work be
>> > > reframed
>> > > > > as a
>> > > > > > >> > > durable
>> > > > > > >> > > > > > > > JDBC reference implementation of that SPI?
>> > > > > > >> > > > > > > >
>> > > > > > >> > > > > > > > SGTM.
>> > > > > > >> > > > > > > >
>> > > > > > >> > > > > > > > > What is the smallest PR sequence to get there
>> > > without
>> > > > > > >> > blocking
>> > > > > > >> > > > > > > > the current metrics work unnecessarily?
>> > > > > > >> > > > > > > >
>> > > > > > >> > > > > > > > Let's get
>> > >
>> >
>> https://urldefense.com/v3/__https://github.com/apache/polaris/pull/4397__;!!Iz9xO38YGHZK!4rnlVIqBQQQ6xpGe1PMV58edOS_cGrKoqdfhw4qSp_JQCavvEjaxXk0OwGPnxyEcO60AzKQK_XDtp7H3DdkFXcrXuu5jefU$
>> > > > > > >> merged
>> > > > > > >> > > > first.
>> > > > > > >> > > > > > > >
>> > > > > > >> > > > > > > > Then, I'd propose isolating the Metrics schema
>> > from
>> > > the
>> > > > > > >> > MetaStore
>> > > > > > >> > > > > > schema.
>> > > > > > >> > > > > > > > This will probably have some ripple effect into
>> > the
>> > > > > > >> bootstrap
>> > > > > > >> > > > > > workflows,
>> > > > > > >> > > > > > > so
>> > > > > > >> > > > > > > > it's not a trivial change.
>> > > > > > >> > > > > > > >
>> > > > > > >> > > > > > > > Then, let's reassess.
>> > > > > > >> > > > > > > >
>> > > > > > >> > > > > > > > Cheers,
>> > > > > > >> > > > > > > > Dmitri.
>> > > > > > >> > > > > > > >
>> > > > > > >> > > > > > > > On Wed, Jun 3, 2026 at 5:40 PM EJ Wang <
>> > > > > > >> > > > > ej.wang.dev.interest@gmail.com
>> > > > > > >> > > > > > >
>> > > > > > >> > > > > > > > wrote:
>> > > > > > >> > > > > > > >
>> > > > > > >> > > > > > > > > Hi folks,
>> > > > > > >> > > > > > > > >
>> > > > > > >> > > > > > > > > Thanks again for the discussion today. I
>> updated
>> > > the
>> > > > > > sync
>> > > > > > >> doc
>> > > > > > >> > > > with
>> > > > > > >> > > > > > > notes
>> > > > > > >> > > > > > > > > from the third metrics architecture sync:
>> > > > > > >> > > > > > > > >
>> > > > > > >> > > > > > > > >
>> > > > > > >> > > > > > > > >
>> > > > > > >> > > > > > > >
>> > > > > > >> > > > > > >
>> > > > > > >> > > > > >
>> > > > > > >> > > > >
>> > > > > > >> > > >
>> > > > > > >> > >
>> > > > > > >> >
>> > > > > > >>
>> > > > > >
>> > > > >
>> > >
>> >
>> https://urldefense.com/v3/__https://docs.google.com/document/d/100h7c4damrUzVuquYbBHM0EvA4LSWuW2IT2dN_7nYVA/edit?tab=t.k96s2xyqr5u1__;!!Iz9xO38YGHZK!4rnlVIqBQQQ6xpGe1PMV58edOS_cGrKoqdfhw4qSp_JQCavvEjaxXk0OwGPnxyEcO60AzKQK_XDtp7H3DdkFXcrXrqCL-QE$
>> > > > > > >> > > > > > > > >
>> > > > > > >> > > > > > > > > A few highlights from today’s discussion:
>> > > > > > >> > > > > > > > >
>> > > > > > >> > > > > > > > >    - We clarified that Iceberg metrics
>> reporting
>> > > can
>> > > > > be
>> > > > > > >> > > > interpreted
>> > > > > > >> > > > > > > > either
>> > > > > > >> > > > > > > > >    as sync or async handling from Polaris’s
>> > > > > perspective,
>> > > > > > >> so
>> > > > > > >> > > > Polaris
>> > > > > > >> > > > > > > > should
>> > > > > > >> > > > > > > > >    stay flexible as a platform instead of
>> baking
>> > > one
>> > > > > > >> handling
>> > > > > > >> > > > model
>> > > > > > >> > > > > > > into
>> > > > > > >> > > > > > > > > the
>> > > > > > >> > > > > > > > >    REST API behavior.
>> > > > > > >> > > > > > > > >    - We aligned that the built-in (battery)
>> > > behavior
>> > > > > for
>> > > > > > >> > > metrics
>> > > > > > >> > > > > > > emitting
>> > > > > > >> > > > > > > > >    can stay simple: no-op/log-only is enough
>> as
>> > > the
>> > > > > > >> default.
>> > > > > > >> > > > > > > > >    - Durable metrics persistence should be
>> > > treated as
>> > > > > an
>> > > > > > >> > > > > > implementation
>> > > > > > >> > > > > > > > of
>> > > > > > >> > > > > > > > >    the metrics reporting path, not as the
>> core
>> > SPI
>> > > > > > >> boundary.
>> > > > > > >> > > > > > > > >    - The existing durable metrics work can be
>> > > reviewed
>> > > > > > as
>> > > > > > >> a
>> > > > > > >> > > > > reference
>> > > > > > >> > > > > > > > >    implementation of the reporting SPI, with
>> > > > > > >> > > persistence-related
>> > > > > > >> > > > > > logic
>> > > > > > >> > > > > > > > kept
>> > > > > > >> > > > > > > > >    self-contained under a metrics durable
>> > > > > implementation
>> > > > > > >> > module
>> > > > > > >> > > > > > rather
>> > > > > > >> > > > > > > > than
>> > > > > > >> > > > > > > > >    scattered through core entity persistence.
>> > > > > > >> > > > > > > > >    - Dashboard/insights remains a real use
>> case,
>> > > but
>> > > > > we
>> > > > > > >> > agreed
>> > > > > > >> > > to
>> > > > > > >> > > > > > keep
>> > > > > > >> > > > > > > it
>> > > > > > >> > > > > > > > >    separate from the core metrics intake
>> > > discussion
>> > > > > for
>> > > > > > >> now.
>> > > > > > >> > > > > > > > >
>> > > > > > >> > > > > > > > > I also did a quick source check after the
>> > meeting
>> > > to
>> > > > > > make
>> > > > > > >> > sure
>> > > > > > >> > > we
>> > > > > > >> > > > > are
>> > > > > > >> > > > > > > > > describing the current state accurately.
>> > > > > > >> > > > > > > > >
>> > > > > > >> > > > > > > > > Current state:
>> > > > > > >> > > > > > > > >
>> > > > > > >> > > > > > > > >    - Polaris already has a metrics reporting
>> > hook:
>> > > > > > >> > > > > > > > PolarisMetricsReporter.
>> > > > > > >> > > > > > > > >    - The default implementation is
>> > > > > > DefaultMetricsReporter,
>> > > > > > >> > > > selected
>> > > > > > >> > > > > > by
>> > > > > > >> > > > > > > > >
>> > > polaris.iceberg-metrics.reporting.type=default. It
>> > > > > is
>> > > > > > >> > > > log-only,
>> > > > > > >> > > > > > > > >    effectively quiet unless metrics logging
>> is
>> > > > > enabled.
>> > > > > > >> > > > > > > > >    - There is also a
>> PersistingMetricsReporter,
>> > > > > selected
>> > > > > > >> by
>> > > > > > >> > > > > > > > >
>> > > polaris.iceberg-metrics.reporting.type=persisting,
>> > > > > > >> which
>> > > > > > >> > > > > converts
>> > > > > > >> > > > > > > > >    Iceberg scan/commit reports into Polaris
>> > > metrics
>> > > > > > >> records
>> > > > > > >> > and
>> > > > > > >> > > > > > writes
>> > > > > > >> > > > > > > > > through
>> > > > > > >> > > > > > > > >    PolarisMetricsManager ->
>> MetricsPersistence.
>> > > > > > >> > > > > > > > >    - MetricsPersistence currently exists in
>> the
>> > > > > > >> persistence
>> > > > > > >> > > layer
>> > > > > > >> > > > > and
>> > > > > > >> > > > > > > is
>> > > > > > >> > > > > > > > >    inherited by BasePersistence.
>> > > > > > >> > > > > > > > >
>> > > > > > >> > > > > > > > > My read from the discussion is that the
>> target
>> > > > > boundary
>> > > > > > >> > should
>> > > > > > >> > > > not
>> > > > > > >> > > > > be
>> > > > > > >> > > > > > > > > MetricsPersistence as inherited by
>> > > BasePersistence.
>> > > > > That
>> > > > > > >> path
>> > > > > > >> > > > > should
>> > > > > > >> > > > > > be
>> > > > > > >> > > > > > > > > decomposed as durable implementation detail
>> > (taken
>> > > > > care
>> > > > > > >> of by
>> > > > > > >> > > > > > > > >
>> >
>> https://urldefense.com/v3/__https://github.com/apache/polaris/pull/4397__;!!Iz9xO38YGHZK!4rnlVIqBQQQ6xpGe1PMV58edOS_cGrKoqdfhw4qSp_JQCavvEjaxXk0OwGPnxyEcO60AzKQK_XDtp7H3DdkFXcrXuu5jefU$
>> ).
>> > The
>> > > > > > stable
>> > > > > > >> > > > extension
>> > > > > > >> > > > > > > point
>> > > > > > >> > > > > > > > > should instead be the metrics
>> reporting/emitting
>> > > > > > boundary.
>> > > > > > >> > > > > > > > >
>> > > > > > >> > > > > > > > > Concretely, I think the next proposal should
>> be
>> > > shaped
>> > > > > > >> like
>> > > > > > >> > > this:
>> > > > > > >> > > > > > > > >
>> > > > > > >> > > > > > > > >    1.
>> > > > > > >> > > > > > > > >
>> > > > > > >> > > > > > > > >    Define the proper metrics reporting SPI
>> at an
>> > > > > > optional
>> > > > > > >> > > Iceberg
>> > > > > > >> > > > > > > metrics
>> > > > > > >> > > > > > > > >    extension API layer.
>> > > > > > >> > > > > > > > >
>> > > > > > >> > > > > > > > >    Example direction:
>> > > > > > >> > > > > > > > >
>> > > > > > >> > > > > > > > >    public interface IcebergMetricsReporter {
>> > > > > > >> > > > > > > > >      void
>> > reportMetric(IcebergMetricsReportContext
>> > > > > > >> context,
>> > > > > > >> > > > > > > > > MetricsReport report);
>> > > > > > >> > > > > > > > >    }
>> > > > > > >> > > > > > > > >
>> > > > > > >> > > > > > > > >    The context would carry the small
>> > > Polaris-resolved
>> > > > > > >> > envelope:
>> > > > > > >> > > > > > > > >    catalog/table identity, report type,
>> received
>> > > > > > >> timestamp,
>> > > > > > >> > and
>> > > > > > >> > > > > > > > > request/trace
>> > > > > > >> > > > > > > > >    context if available. The raw Iceberg
>> > > MetricsReport
>> > > > > > >> > remains
>> > > > > > >> > > > the
>> > > > > > >> > > > > > > > payload.
>> > > > > > >> > > > > > > > >    2.
>> > > > > > >> > > > > > > > >
>> > > > > > >> > > > > > > > >    Keep runtime/service as the REST ingestion
>> > and
>> > > > > wiring
>> > > > > > >> > layer.
>> > > > > > >> > > > > > > > >
>> > > > > > >> > > > > > > > >    The REST handler still resolves the table
>> and
>> > > > > > performs
>> > > > > > >> > authz
>> > > > > > >> > > > > > before
>> > > > > > >> > > > > > > > >    accepting the report. After that, it calls
>> > the
>> > > > > > selected
>> > > > > > >> > > > > reporting
>> > > > > > >> > > > > > > > >    implementation.
>> > > > > > >> > > > > > > > >    3.
>> > > > > > >> > > > > > > > >
>> > > > > > >> > > > > > > > >    Keep the battery default no-op/log-only.
>> > > > > > >> > > > > > > > >
>> > > > > > >> > > > > > > > >    This preserves an out-of-box safe default
>> and
>> > > > > avoids
>> > > > > > >> > > > requiring a
>> > > > > > >> > > > > > > > durable
>> > > > > > >> > > > > > > > >    metrics store for every Polaris
>> deployment.
>> > > > > > >> > > > > > > > >    4.
>> > > > > > >> > > > > > > > >
>> > > > > > >> > > > > > > > >    Move durable JDBC metrics into a
>> > self-contained
>> > > > > > >> > > implementation
>> > > > > > >> > > > > of
>> > > > > > >> > > > > > > the
>> > > > > > >> > > > > > > > >    reporting SPI.
>> > > > > > >> > > > > > > > >
>> > > > > > >> > > > > > > > >    That implementation can own its schema,
>> > > bootstrap,
>> > > > > > >> > > retention,
>> > > > > > >> > > > > and
>> > > > > > >> > > > > > > read
>> > > > > > >> > > > > > > > >    API support. It should not define the core
>> > > > > reporting
>> > > > > > >> SPI
>> > > > > > >> > > > > boundary,
>> > > > > > >> > > > > > > and
>> > > > > > >> > > > > > > > > it
>> > > > > > >> > > > > > > > >    should not require metrics persistence to
>> > > remain
>> > > > > > >> inherited
>> > > > > > >> > > > from
>> > > > > > >> > > > > > > > >    BasePersistence.
>> > > > > > >> > > > > > > > >    5.
>> > > > > > >> > > > > > > > >
>> > > > > > >> > > > > > > > >    Treat async/event-backed handling as
>> another
>> > > > > > >> > implementation
>> > > > > > >> > > of
>> > > > > > >> > > > > the
>> > > > > > >> > > > > > > > same
>> > > > > > >> > > > > > > > >    reporting SPI.
>> > > > > > >> > > > > > > > >
>> > > > > > >> > > > > > > > >    For example, an event-backed reporter
>> could
>> > > > > enqueue a
>> > > > > > >> > > metrics
>> > > > > > >> > > > > > event
>> > > > > > >> > > > > > > > and
>> > > > > > >> > > > > > > > >    let listeners handle durable storage or
>> other
>> > > > > sinks.
>> > > > > > >> If we
>> > > > > > >> > > > later
>> > > > > > >> > > > > > > need
>> > > > > > >> > > > > > > > a
>> > > > > > >> > > > > > > > >    replaceable queue engine, that seems like
>> a
>> > > shared
>> > > > > > >> > > > event/metrics
>> > > > > > >> > > > > > > > > substrate
>> > > > > > >> > > > > > > > >    topic rather than a metrics-only
>> requirement.
>> > > > > > >> > > > > > > > >
>> > > > > > >> > > > > > > > > This framing lets us keep the REST metrics
>> > > endpoint
>> > > > > > >> simple,
>> > > > > > >> > > > > preserve
>> > > > > > >> > > > > > > the
>> > > > > > >> > > > > > > > > current default behavior, support durable
>> > metrics
>> > > > > users,
>> > > > > > >> and
>> > > > > > >> > > > still
>> > > > > > >> > > > > > > leave
>> > > > > > >> > > > > > > > > room for async/event-backed or
>> > > external-queue-based
>> > > > > > >> > > > > implementations.
>> > > > > > >> > > > > > > > >
>> > > > > > >> > > > > > > > > I think the main follow-up questions are:
>> > > > > > >> > > > > > > > >
>> > > > > > >> > > > > > > > >    - Do folks agree that the stable SPI
>> boundary
>> > > > > should
>> > > > > > be
>> > > > > > >> > > > metrics
>> > > > > > >> > > > > > > > >    reporting/emitting, not metrics
>> persistence?
>> > > > > > >> > > > > > > > >    - Does an optional Iceberg metrics
>> extension
>> > > API
>> > > > > > layer
>> > > > > > >> > sound
>> > > > > > >> > > > > like
>> > > > > > >> > > > > > > the
>> > > > > > >> > > > > > > > >    right home for this SPI?
>> > > > > > >> > > > > > > > >    - Should the current durable metrics work
>> be
>> > > > > reframed
>> > > > > > >> as a
>> > > > > > >> > > > > durable
>> > > > > > >> > > > > > > > JDBC
>> > > > > > >> > > > > > > > >    reference implementation of that SPI?
>> > > > > > >> > > > > > > > >    - What is the smallest PR sequence to get
>> > there
>> > > > > > without
>> > > > > > >> > > > blocking
>> > > > > > >> > > > > > the
>> > > > > > >> > > > > > > > >    current metrics work unnecessarily?
>> > > > > > >> > > > > > > > >
>> > > > > > >> > > > > > > > > Thanks,
>> > > > > > >> > > > > > > > > -ej
>> > > > > > >> > > > > > > > >
>> > > > > > >> > > > > > > > > On Fri, May 15, 2026 at 5:16 PM Dmitri
>> > > Bourlatchkov <
>> > > > > > >> > > > > > dimas@apache.org>
>> > > > > > >> > > > > > > > > wrote:
>> > > > > > >> > > > > > > > >
>> > > > > > >> > > > > > > > > > Hi JB,
>> > > > > > >> > > > > > > > > >
>> > > > > > >> > > > > > > > > > Could you set up another meeting, please?
>> Same
>> > > time
>> > > > > on
>> > > > > > >> > > > Wednesday
>> > > > > > >> > > > > as
>> > > > > > >> > > > > > > > last
>> > > > > > >> > > > > > > > > > time... I hope it works for everyone.
>> > > > > > >> > > > > > > > > >
>> > > > > > >> > > > > > > > > > Cheers,
>> > > > > > >> > > > > > > > > > Dmitri.
>> > > > > > >> > > > > > > > > >
>> > > > > > >> > > > > > > > > > On Fri, May 15, 2026 at 8:06 PM Yufei Gu <
>> > > > > > >> > > flyrain000@gmail.com
>> > > > > > >> > > > >
>> > > > > > >> > > > > > > wrote:
>> > > > > > >> > > > > > > > > >
>> > > > > > >> > > > > > > > > > > +1 on another sync call next week.
>> > > > > > >> > > > > > > > > > >
>> > > > > > >> > > > > > > > > > > Yufei
>> > > > > > >> > > > > > > > > > >
>> > > > > > >> > > > > > > > > > >
>> > > > > > >> > > > > > > > > > > On Fri, May 15, 2026 at 4:52 PM Dmitri
>> > > > > Bourlatchkov
>> > > > > > <
>> > > > > > >> > > > > > > > dimas@apache.org>
>> > > > > > >> > > > > > > > > > > wrote:
>> > > > > > >> > > > > > > > > > >
>> > > > > > >> > > > > > > > > > > > Hi All,
>> > > > > > >> > > > > > > > > > > >
>> > > > > > >> > > > > > > > > > > > WDYT about another sync call next week?
>> > > > > > >> > > > > > > > > > > >
>> > > > > > >> > > > > > > > > > > > Thanks,
>> > > > > > >> > > > > > > > > > > > Dmitri.
>> > > > > > >> > > > > > > > > > > >
>> > > > > > >> > > > > > > > > > > > On Wed, May 6, 2026 at 5:29 PM Dmitri
>> > > > > > Bourlatchkov <
>> > > > > > >> > > > > > > > dimas@apache.org
>> > > > > > >> > > > > > > > > >
>> > > > > > >> > > > > > > > > > > > wrote:
>> > > > > > >> > > > > > > > > > > >
>> > > > > > >> > > > > > > > > > > > > Hi EJ,
>> > > > > > >> > > > > > > > > > > > >
>> > > > > > >> > > > > > > > > > > > > Thanks for the summary! It covers
>> what
>> > we
>> > > > > > >> discussed
>> > > > > > >> > in
>> > > > > > >> > > > the
>> > > > > > >> > > > > > > > meeting
>> > > > > > >> > > > > > > > > > very
>> > > > > > >> > > > > > > > > > > > > well, IMHO.
>> > > > > > >> > > > > > > > > > > > >
>> > > > > > >> > > > > > > > > > > > > Looking forward to concrete PRs :)
>> > > > > > >> > > > > > > > > > > > >
>> > > > > > >> > > > > > > > > > > > > Cheers,
>> > > > > > >> > > > > > > > > > > > > Dmitri.
>> > > > > > >> > > > > > > > > > > > >
>> > > > > > >> > > > > > > > > > > > > On Wed, May 6, 2026 at 5:08 PM EJ
>> Wang <
>> > > > > > >> > > > > > > > > > ej.wang.dev.interest@gmail.com
>> > > > > > >> > > > > > > > > > > >
>> > > > > > >> > > > > > > > > > > > > wrote:
>> > > > > > >> > > > > > > > > > > > >
>> > > > > > >> > > > > > > > > > > > >> Hi folks,
>> > > > > > >> > > > > > > > > > > > >>
>> > > > > > >> > > > > > > > > > > > >> We had a community sync earlier,
>> thanks
>> > > JB
>> > > > > for
>> > > > > > >> > > > scheduling
>> > > > > > >> > > > > > it.
>> > > > > > >> > > > > > > > > Notes
>> > > > > > >> > > > > > > > > > > from
>> > > > > > >> > > > > > > > > > > > >> the first metrics architecture sync
>> > (May
>> > > 6,
>> > > > > > >> 10-11am
>> > > > > > >> > > PT).
>> > > > > > >> > > > > > > > > Discussion
>> > > > > > >> > > > > > > > > > > doc
>> > > > > > >> > > > > > > > > > > > >> with per-section status:
>> > > > > > >> > > > > > > > > > > > >>
>> > > > > > >> > > > > > > > > > > > >>
>> > > > > > >> > > > > > > > > > > >
>> > > > > > >> > > > > > > > > > >
>> > > > > > >> > > > > > > > > >
>> > > > > > >> > > > > > > > >
>> > > > > > >> > > > > > > >
>> > > > > > >> > > > > > >
>> > > > > > >> > > > > >
>> > > > > > >> > > > >
>> > > > > > >> > > >
>> > > > > > >> > >
>> > > > > > >> >
>> > > > > > >>
>> > > > > >
>> > > > >
>> > >
>> >
>> https://urldefense.com/v3/__https://docs.google.com/document/d/100h7c4damrUzVuquYbBHM0EvA4LSWuW2IT2dN_7nYVA/edit?tab=t.0__;!!Iz9xO38YGHZK!4rnlVIqBQQQ6xpGe1PMV58edOS_cGrKoqdfhw4qSp_JQCavvEjaxXk0OwGPnxyEcO60AzKQK_XDtp7H3DdkFXcrXgIddeUs$
>> > > > > > >> > > > > > > > > > > > >>
>> > > > > > >> > > > > > > > > > > > >> *The meeting covered both topics
>> from
>> > the
>> > > > > doc.
>> > > > > > >> > > > > > Direction-level
>> > > > > > >> > > > > > > > > > > alignment
>> > > > > > >> > > > > > > > > > > > >> was reached on the headline pieces;
>> > > details
>> > > > > > >> remain
>> > > > > > >> > for
>> > > > > > >> > > > PR
>> > > > > > >> > > > > > > review
>> > > > > > >> > > > > > > > > or
>> > > > > > >> > > > > > > > > > > > >> follow-up sessions.*
>> > > > > > >> > > > > > > > > > > > >>
>> > > > > > >> > > > > > > > > > > > >> *Topic 1 — Persistence schema
>> redesign*
>> > > > > > >> > > > > > > > > > > > >> Idea-level alignment on
>> consolidating
>> > > > > per-type
>> > > > > > >> > tables
>> > > > > > >> > > > > > > > > > > > >> (scan_metrics_report,
>> > > > > > >> > > > > > > > > > > > >> commit_metrics_report) into a single
>> > > > > > >> metrics_report
>> > > > > > >> > > > table.
>> > > > > > >> > > > > > The
>> > > > > > >> > > > > > > > > > > > motivating
>> > > > > > >> > > > > > > > > > > > >> cost is the surface area added by
>> every
>> > > new
>> > > > > > >> metric
>> > > > > > >> > > type
>> > > > > > >> > > > > > today:
>> > > > > > >> > > > > > > > new
>> > > > > > >> > > > > > > > > > > > table,
>> > > > > > >> > > > > > > > > > > > >> SPI method, record class, model,
>> > > converter,
>> > > > > > >> schema
>> > > > > > >> > > > > > migration.
>> > > > > > >> > > > > > > > > > > > >>
>> > > > > > >> > > > > > > > > > > > >> Most schema details are deferred to
>> the
>> > > > > schema
>> > > > > > >> PR. A
>> > > > > > >> > > few
>> > > > > > >> > > > > > > > specific
>> > > > > > >> > > > > > > > > > > points
>> > > > > > >> > > > > > > > > > > > >> came up:
>> > > > > > >> > > > > > > > > > > > >> •   metric_schema_version: Yufei
>> > prefers
>> > > > > > dropping
>> > > > > > >> > it,
>> > > > > > >> > > > > since
>> > > > > > >> > > > > > > > there
>> > > > > > >> > > > > > > > > is
>> > > > > > >> > > > > > > > > > > no
>> > > > > > >> > > > > > > > > > > > >> spec-level concept of metrics
>> > versioning
>> > > > > today
>> > > > > > >> and
>> > > > > > >> > it
>> > > > > > >> > > is
>> > > > > > >> > > > > > hard
>> > > > > > >> > > > > > > to
>> > > > > > >> > > > > > > > > > > define
>> > > > > > >> > > > > > > > > > > > >> unilaterally. Robert prefers keeping
>> > it,
>> > > > > given
>> > > > > > >> IRC
>> > > > > > >> > v2
>> > > > > > >> > > is
>> > > > > > >> > > > > > > coming
>> > > > > > >> > > > > > > > > and
>> > > > > > >> > > > > > > > > > > the
>> > > > > > >> > > > > > > > > > > > >> schema should be considered against
>> its
>> > > > > likely
>> > > > > > >> > shape;
>> > > > > > >> > > > > Robert
>> > > > > > >> > > > > > > > also
>> > > > > > >> > > > > > > > > > > raised
>> > > > > > >> > > > > > > > > > > > >> how to differentiate various payload
>> > > formats
>> > > > > if
>> > > > > > >> any.
>> > > > > > >> > > > EJ's
>> > > > > > >> > > > > > read
>> > > > > > >> > > > > > > > is
>> > > > > > >> > > > > > > > > > that
>> > > > > > >> > > > > > > > > > > > >> this
>> > > > > > >> > > > > > > > > > > > >> is a two-way-door decision. We can
>> > start
>> > > > > > without
>> > > > > > >> the
>> > > > > > >> > > > > field,
>> > > > > > >> > > > > > > and
>> > > > > > >> > > > > > > > if
>> > > > > > >> > > > > > > > > > IRC
>> > > > > > >> > > > > > > > > > > > v2
>> > > > > > >> > > > > > > > > > > > >> changes the shape we would likely
>> roll
>> > a
>> > > > > > >> > corresponding
>> > > > > > >> > > > new
>> > > > > > >> > > > > > > > schema
>> > > > > > >> > > > > > > > > > > > anyway,
>> > > > > > >> > > > > > > > > > > > >> which is not particularly costly.
>> > > > > > >> > > > > > > > > > > > >> •   Payload format: Robert pointed
>> out
>> > > that
>> > > > > > >> future
>> > > > > > >> > > > formats
>> > > > > > >> > > > > > > > beyond
>> > > > > > >> > > > > > > > > > JSON
>> > > > > > >> > > > > > > > > > > > may
>> > > > > > >> > > > > > > > > > > > >> be worth supporting. The exact
>> shape is
>> > > > > > deferred
>> > > > > > >> to
>> > > > > > >> > > the
>> > > > > > >> > > > > > schema
>> > > > > > >> > > > > > > > > > > > discussion.
>> > > > > > >> > > > > > > > > > > > >> •   Partition strategy: Anand
>> suggested
>> > > > > monthly
>> > > > > > >> > > > > partitioning
>> > > > > > >> > > > > > > > based
>> > > > > > >> > > > > > > > > > on
>> > > > > > >> > > > > > > > > > > > his
>> > > > > > >> > > > > > > > > > > > >> experience as potentially helpful at
>> > > scale.
>> > > > > > >> > > > > > > > > > > > >>
>> > > > > > >> > > > > > > > > > > > >> *Topic 2 — Where metrics ingestion
>> and
>> > > > > storage
>> > > > > > >> > belong*
>> > > > > > >> > > > > > > > > > > > >> Idea-level alignment that metrics
>> > should
>> > > be a
>> > > > > > >> > > separated
>> > > > > > >> > > > > SPI
>> > > > > > >> > > > > > > from
>> > > > > > >> > > > > > > > > the
>> > > > > > >> > > > > > > > > > > > >> entity
>> > > > > > >> > > > > > > > > > > > >> persistence stack. Two reasons
>> > surfaced:
>> > > (a)
>> > > > > > >> > workloads
>> > > > > > >> > > > and
>> > > > > > >> > > > > > > > > > capability
>> > > > > > >> > > > > > > > > > > > >> requirements diverge enough that
>> > coupling
>> > > > > them
>> > > > > > >> > creates
>> > > > > > >> > > > > > > > artificial
>> > > > > > >> > > > > > > > > > > > >> constraints, and (b) admin
>> experience
>> > > > > improves
>> > > > > > >> when
>> > > > > > >> > > > > metrics
>> > > > > > >> > > > > > > has
>> > > > > > >> > > > > > > > > its
>> > > > > > >> > > > > > > > > > > own
>> > > > > > >> > > > > > > > > > > > >> bootstrap, retention, and lifecycle.
>> > > Dmitri
>> > > > > > noted
>> > > > > > >> > > > Polaris
>> > > > > > >> > > > > > > being
>> > > > > > >> > > > > > > > a
>> > > > > > >> > > > > > > > > > > > platform
>> > > > > > >> > > > > > > > > > > > >> should have the flexibility to
>> support
>> > > > > > different
>> > > > > > >> > > > > persistence
>> > > > > > >> > > > > > > > > > backends
>> > > > > > >> > > > > > > > > > > > per
>> > > > > > >> > > > > > > > > > > > >> concern, and pointed to a concrete
>> next
>> > > step
>> > > > > of
>> > > > > > >> > > > separating
>> > > > > > >> > > > > > the
>> > > > > > >> > > > > > > > > JDBC
>> > > > > > >> > > > > > > > > > > > >> bootstrap for metrics from the
>> > metastore
>> > > > > > >> bootstrap.
>> > > > > > >> > > > Robert
>> > > > > > >> > > > > > > > > proposed
>> > > > > > >> > > > > > > > > > an
>> > > > > > >> > > > > > > > > > > > >> additional UX extension: detect an
>> > > > > > unbootstrapped
>> > > > > > >> > > > metrics
>> > > > > > >> > > > > > > store
>> > > > > > >> > > > > > > > on
>> > > > > > >> > > > > > > > > > > first
>> > > > > > >> > > > > > > > > > > > >> use and auto-bootstrap rather than
>> > > requiring
>> > > > > an
>> > > > > > >> > > explicit
>> > > > > > >> > > > > > > manual
>> > > > > > >> > > > > > > > > > > > bootstrap
>> > > > > > >> > > > > > > > > > > > >> step.
>> > > > > > >> > > > > > > > > > > > >> The meeting also confirmed that
>> Polaris
>> > > > > metrics
>> > > > > > >> can
>> > > > > > >> > > > start
>> > > > > > >> > > > > > > small
>> > > > > > >> > > > > > > > > and
>> > > > > > >> > > > > > > > > > > stay
>> > > > > > >> > > > > > > > > > > > >> Iceberg-focused. Naming and
>> persistence
>> > > > > schema
>> > > > > > >> can
>> > > > > > >> > > lean
>> > > > > > >> > > > > > > > > > > > Iceberg-specific.
>> > > > > > >> > > > > > > > > > > > >> If a future expansion to
>> generic-table
>> > > > > metrics
>> > > > > > or
>> > > > > > >> > > > > > operational
>> > > > > > >> > > > > > > > > > metrics
>> > > > > > >> > > > > > > > > > > > >> arrives, an abstraction layer can be
>> > > built on
>> > > > > > >> top of
>> > > > > > >> > > the
>> > > > > > >> > > > > > > Iceberg
>> > > > > > >> > > > > > > > > > > metrics
>> > > > > > >> > > > > > > > > > > > >> reporter at that point. Robert
>> remains
>> > > on the
>> > > > > > >> fence
>> > > > > > >> > > and
>> > > > > > >> > > > > > would
>> > > > > > >> > > > > > > > > prefer
>> > > > > > >> > > > > > > > > > > > >> something more generic but did not
>> > block
>> > > the
>> > > > > > >> > > direction;
>> > > > > > >> > > > > > > Dmitri's
>> > > > > > >> > > > > > > > > > read
>> > > > > > >> > > > > > > > > > > > was
>> > > > > > >> > > > > > > > > > > > >> that the proposed framework already
>> has
>> > > > > enough
>> > > > > > >> > > > flexibility
>> > > > > > >> > > > > > to
>> > > > > > >> > > > > > > > > absorb
>> > > > > > >> > > > > > > > > > > > >> future
>> > > > > > >> > > > > > > > > > > > >> expansion.
>> > > > > > >> > > > > > > > > > > > >>
>> > > > > > >> > > > > > > > > > > > >> The Trade-offs and Proposed
>> structure
>> > > > > sections
>> > > > > > in
>> > > > > > >> > the
>> > > > > > >> > > > doc
>> > > > > > >> > > > > > were
>> > > > > > >> > > > > > > > not
>> > > > > > >> > > > > > > > > > > > >> reviewed
>> > > > > > >> > > > > > > > > > > > >> in detail. They remain open for
>> either
>> > > the
>> > > > > next
>> > > > > > >> sync
>> > > > > > >> > > or
>> > > > > > >> > > > PR
>> > > > > > >> > > > > > > > review.
>> > > > > > >> > > > > > > > > > > > >>
>> > > > > > >> > > > > > > > > > > > >> *Cross-cutting alignment —
>> > > battery-included
>> > > > > > plus
>> > > > > > >> > > > > pluggable*
>> > > > > > >> > > > > > > > > > > > >> A common philosophy emerged from the
>> > > > > > discussion.
>> > > > > > >> EJ
>> > > > > > >> > > > > > summarized
>> > > > > > >> > > > > > > > it
>> > > > > > >> > > > > > > > > > as:
>> > > > > > >> > > > > > > > > > > > >> Polaris should provide a
>> > > battery-included UX
>> > > > > > for
>> > > > > > >> > > > beginners
>> > > > > > >> > > > > > and
>> > > > > > >> > > > > > > > the
>> > > > > > >> > > > > > > > > > > > >> flexibility for advanced users to
>> swap
>> > > the
>> > > > > > >> included
>> > > > > > >> > > > > battery
>> > > > > > >> > > > > > > for
>> > > > > > >> > > > > > > > > > > > something
>> > > > > > >> > > > > > > > > > > > >> more powerful or tailored to their
>> use
>> > > case.
>> > > > > > The
>> > > > > > >> SPI
>> > > > > > >> > > > > design
>> > > > > > >> > > > > > > > needs
>> > > > > > >> > > > > > > > > to
>> > > > > > >> > > > > > > > > > > > >> enable
>> > > > > > >> > > > > > > > > > > > >> both.
>> > > > > > >> > > > > > > > > > > > >>
>> > > > > > >> > > > > > > > > > > > >> The inputs that shaped this framing:
>> > > > > > >> > > > > > > > > > > > >> •   Anand described how his team
>> uses
>> > the
>> > > > > > current
>> > > > > > >> > > > metrics
>> > > > > > >> > > > > > > > > > persistence
>> > > > > > >> > > > > > > > > > > > >> (three metrics consumers in v1.4).
>> > > > > > >> > > > > > > > > > > > >> •   Yufei raised Grafana and
>> dashboard
>> > > > > > >> integrations
>> > > > > > >> > > as a
>> > > > > > >> > > > > > > > > destination
>> > > > > > >> > > > > > > > > > > use
>> > > > > > >> > > > > > > > > > > > >> case beyond the default.
>> > > > > > >> > > > > > > > > > > > >> •   Robert called out that the
>> current
>> > > design
>> > > > > > is
>> > > > > > >> > more
>> > > > > > >> > > > > > > > > JDBC-focused.
>> > > > > > >> > > > > > > > > > > > >>
>> > > > > > >> > > > > > > > > > > > >> Two concrete instances:
>> > > > > > >> > > > > > > > > > > > >> •   Async metrics intake: Yufei's
>> > initial
>> > > > > > >> position
>> > > > > > >> > was
>> > > > > > >> > > > > that
>> > > > > > >> > > > > > > > async
>> > > > > > >> > > > > > > > > > > should
>> > > > > > >> > > > > > > > > > > > >> largely live on the producer side
>> and
>> > > there
>> > > > > is
>> > > > > > >> not
>> > > > > > >> > > much
>> > > > > > >> > > > > > > Polaris
>> > > > > > >> > > > > > > > > can
>> > > > > > >> > > > > > > > > > > do.
>> > > > > > >> > > > > > > > > > > > >> Robert suggested a Polaris-side
>> default
>> > > is
>> > > > > > doable
>> > > > > > >> > via
>> > > > > > >> > > > > > Vert.x.
>> > > > > > >> > > > > > > > > Dmitri
>> > > > > > >> > > > > > > > > > > > >> agreed
>> > > > > > >> > > > > > > > > > > > >> the direction is worth exploring.
>> The
>> > > meeting
>> > > > > > >> > > converged
>> > > > > > >> > > > > on a
>> > > > > > >> > > > > > > > > > > > >> battery-included default (likely
>> > > > > Vert.x-backed)
>> > > > > > >> with
>> > > > > > >> > > an
>> > > > > > >> > > > > SPI
>> > > > > > >> > > > > > > > shape
>> > > > > > >> > > > > > > > > > that
>> > > > > > >> > > > > > > > > > > > >> lets
>> > > > > > >> > > > > > > > > > > > >> power users route to a more scalable
>> > > backend
>> > > > > > >> > > (k8s-hosted
>> > > > > > >> > > > > > > queue,
>> > > > > > >> > > > > > > > > AWS
>> > > > > > >> > > > > > > > > > > SQS,
>> > > > > > >> > > > > > > > > > > > >> etc.).
>> > > > > > >> > > > > > > > > > > > >> •   Pluggable destinations:
>> combining
>> > > Yufei's
>> > > > > > >> > > dashboard
>> > > > > > >> > > > > use
>> > > > > > >> > > > > > > case
>> > > > > > >> > > > > > > > > > with
>> > > > > > >> > > > > > > > > > > > >> Robert's JDBC-focused call-out, the
>> > > meeting
>> > > > > > >> agreed
>> > > > > > >> > the
>> > > > > > >> > > > SPI
>> > > > > > >> > > > > > > > should
>> > > > > > >> > > > > > > > > be
>> > > > > > >> > > > > > > > > > > > >> structured for multiple sinks so
>> > > integrations
>> > > > > > >> become
>> > > > > > >> > > > impl
>> > > > > > >> > > > > > > > choices
>> > > > > > >> > > > > > > > > > > rather
>> > > > > > >> > > > > > > > > > > > >> than architectural changes.
>> > > > > > >> > > > > > > > > > > > >>
>> > > > > > >> > > > > > > > > > > > >> The battery-included default is most
>> > > likely
>> > > > > to
>> > > > > > >> use
>> > > > > > >> > the
>> > > > > > >> > > > > > > existing
>> > > > > > >> > > > > > > > > > > > >> JDBC-backed
>> > > > > > >> > > > > > > > > > > > >> approach.
>> > > > > > >> > > > > > > > > > > > >>
>> > > > > > >> > > > > > > > > > > > >> *Direction (idea-level alignment)*
>> > > > > > >> > > > > > > > > > > > >> •   Single metrics_report table
>> > > consolidating
>> > > > > > >> > per-type
>> > > > > > >> > > > > > > metrics,
>> > > > > > >> > > > > > > > > > > > replacing
>> > > > > > >> > > > > > > > > > > > >> scan_metrics_report and
>> > > commit_metrics_report
>> > > > > > >> > > > > > > > > > > > >> •   Iceberg-focused naming and
>> schema
>> > for
>> > > > > now,
>> > > > > > >> > revisit
>> > > > > > >> > > > if
>> > > > > > >> > > > > > > > > > > generic-table
>> > > > > > >> > > > > > > > > > > > or
>> > > > > > >> > > > > > > > > > > > >> operational metrics arrive
>> > > > > > >> > > > > > > > > > > > >> •   Metrics persistence as a
>> separated
>> > > SPI,
>> > > > > not
>> > > > > > >> on
>> > > > > > >> > > > > > > > BasePersistence
>> > > > > > >> > > > > > > > > > > > >> •   Bootstrap path separated for
>> > metrics,
>> > > > > > >> > independent
>> > > > > > >> > > of
>> > > > > > >> > > > > > > > metastore
>> > > > > > >> > > > > > > > > > > > >> bootstrap
>> > > > > > >> > > > > > > > > > > > >> •   "Battery-included plus
>> pluggable"
>> > as
>> > > the
>> > > > > > SPI
>> > > > > > >> > > design
>> > > > > > >> > > > > > > > philosophy
>> > > > > > >> > > > > > > > > > > > >>
>> > > > > > >> > > > > > > > > > > > >> *Open items*
>> > > > > > >> > > > > > > > > > > > >> •   Schema details:
>> > > metric_schema_version,
>> > > > > > >> payload
>> > > > > > >> > > > format,
>> > > > > > >> > > > > > IRC
>> > > > > > >> > > > > > > > v2
>> > > > > > >> > > > > > > > > > > > >> forward-compat shape
>> > > > > > >> > > > > > > > > > > > >> •   SPI design details — full review
>> > > either
>> > > > > in
>> > > > > > >> the
>> > > > > > >> > > next
>> > > > > > >> > > > > sync
>> > > > > > >> > > > > > > or
>> > > > > > >> > > > > > > > in
>> > > > > > >> > > > > > > > > > the
>> > > > > > >> > > > > > > > > > > > >> corresponding PR
>> > > > > > >> > > > > > > > > > > > >> •   Schema refactor PR ownership
>> > > > > > >> > > > > > > > > > > > >>
>> > > > > > >> > > > > > > > > > > > >> *Action items*
>> > > > > > >> > > > > > > > > > > > >> •   EJ to take a first stab at the
>> SPI
>> > > design
>> > > > > > and
>> > > > > > >> > > > > > potentially
>> > > > > > >> > > > > > > > > > partner
>> > > > > > >> > > > > > > > > > > > with
>> > > > > > >> > > > > > > > > > > > >> Anand to incorporate the lessons
>> > learned
>> > > from
>> > > > > > the
>> > > > > > >> > > > existing
>> > > > > > >> > > > > > > > > reporter
>> > > > > > >> > > > > > > > > > > and
>> > > > > > >> > > > > > > > > > > > >> persistence work.
>> > > > > > >> > > > > > > > > > > > >> •   Schema refactor PR ownership is
>> not
>> > > yet
>> > > > > > >> decided.
>> > > > > > >> > > If
>> > > > > > >> > > > > > anyone
>> > > > > > >> > > > > > > > is
>> > > > > > >> > > > > > > > > > > > >> interested in driving it, reply on
>> this
>> > > > > thread.
>> > > > > > >> > > > > > > > > > > > >> •   JB to schedule the next sync,
>> > > tentatively
>> > > > > > in
>> > > > > > >> two
>> > > > > > >> > > > > weeks.
>> > > > > > >> > > > > > > > > > > > >>
>> > > > > > >> > > > > > > > > > > > >> -ej
>> > > > > > >> > > > > > > > > > > > >>
>> > > > > > >> > > > > > > > > > > > >> On Mon, Apr 27, 2026 at 3:07 PM EJ
>> > Wang <
>> > > > > > >> > > > > > > > > > > ej.wang.dev.interest@gmail.com
>> > > > > > >> > > > > > > > > > > > >
>> > > > > > >> > > > > > > > > > > > >> wrote:
>> > > > > > >> > > > > > > > > > > > >>
>> > > > > > >> > > > > > > > > > > > >> > Thanks Yufei for the +1.
>> > > > > > >> > > > > > > > > > > > >> >
>> > > > > > >> > > > > > > > > > > > >> > JB, could you help add a biweekly
>> > > metrics
>> > > > > > >> > > architecture
>> > > > > > >> > > > > > sync
>> > > > > > >> > > > > > > to
>> > > > > > >> > > > > > > > > the
>> > > > > > >> > > > > > > > > > > > >> Polaris
>> > > > > > >> > > > > > > > > > > > >> > community calendar? I'm thinking
>> > > Thursdays
>> > > > > at
>> > > > > > >> > 9-10am
>> > > > > > >> > > > PT,
>> > > > > > >> > > > > > on
>> > > > > > >> > > > > > > > the
>> > > > > > >> > > > > > > > > > > > >> off-weeks
>> > > > > > >> > > > > > > > > > > > >> > from the community meeting
>> (starting
>> > > May
>> > > > > 7),
>> > > > > > 60
>> > > > > > >> > > > minutes.
>> > > > > > >> > > > > > > > > > > > >> >
>> > > > > > >> > > > > > > > > > > > >> > Here's a rough agenda to work
>> through
>> > > over
>> > > > > > the
>> > > > > > >> > first
>> > > > > > >> > > > few
>> > > > > > >> > > > > > > > > sessions,
>> > > > > > >> > > > > > > > > > > > >> grouped
>> > > > > > >> > > > > > > > > > > > >> > by priority:
>> > > > > > >> > > > > > > > > > > > >> >
>> > > > > > >> > > > > > > > > > > > >> > *First: foundational direction*
>> > > > > > >> > > > > > > > > > > > >> >
>> > > > > > >> > > > > > > > > > > > >> > 1.  MetricsPersistence: public
>> SPI or
>> > > > > > internal
>> > > > > > >> > > > > > > implementation
>> > > > > > >> > > > > > > > > > > detail?
>> > > > > > >> > > > > > > > > > > > >> >    •   Marked @Beta, javadoc calls
>> > it a
>> > > > > > >> "Service
>> > > > > > >> > > > > Provider
>> > > > > > >> > > > > > > > > > > Interface",
>> > > > > > >> > > > > > > > > > > > >> but
>> > > > > > >> > > > > > > > > > > > >> > only one consumer
>> > > > > (JdbcBasePersistenceImpl),
>> > > > > > >> lives
>> > > > > > >> > > on
>> > > > > > >> > > > > > > > > > > BasePersistence.
>> > > > > > >> > > > > > > > > > > > >> If
>> > > > > > >> > > > > > > > > > > > >> > demoted to a private helper
>> inside a
>> > > > > > persisting
>> > > > > > >> > > > reporter
>> > > > > > >> > > > > > > impl,
>> > > > > > >> > > > > > > > > > most
>> > > > > > >> > > > > > > > > > > > >> > downstream design decisions become
>> > > > > > >> implementation
>> > > > > > >> > > > > details
>> > > > > > >> > > > > > > > rather
>> > > > > > >> > > > > > > > > > > than
>> > > > > > >> > > > > > > > > > > > >> > contract questions.
>> > > > > > >> > > > > > > > > > > > >> >
>> > > > > > >> > > > > > > > > > > > >> > 2.  Persistence schema redesign
>> > > > > > >> > > > > > > > > > > > >> >    •   Current two-table layout
>> > > > > > >> > > (scan_metrics_report,
>> > > > > > >> > > > > > > > > > > > >> > commit_metrics_report) with ~25
>> > > flattened
>> > > > > > >> columns
>> > > > > > >> > > > each.
>> > > > > > >> > > > > > > Every
>> > > > > > >> > > > > > > > > new
>> > > > > > >> > > > > > > > > > > > metric
>> > > > > > >> > > > > > > > > > > > >> > type requires a new table, SPI
>> > method,
>> > > > > record
>> > > > > > >> > class,
>> > > > > > >> > > > > > model,
>> > > > > > >> > > > > > > > > > > converter,
>> > > > > > >> > > > > > > > > > > > >> and
>> > > > > > >> > > > > > > > > > > > >> > schema migration. Direction to
>> > explore:
>> > > > > > single
>> > > > > > >> > table
>> > > > > > >> > > > > with
>> > > > > > >> > > > > > > > > > > metric_type
>> > > > > > >> > > > > > > > > > > > >> enum,
>> > > > > > >> > > > > > > > > > > > >> > schema_version, and JSON payload
>> > > column.
>> > > > > > >> > > > > > > > > > > > >> >
>> > > > > > >> > > > > > > > > > > > >> > *Second: design details once
>> > direction
>> > > is
>> > > > > > set*
>> > > > > > >> > > > > > > > > > > > >> >
>> > > > > > >> > > > > > > > > > > > >> > 3.  Partition key strategy
>> > > > > > >> > > > > > > > > > > > >> >    •   Single-table design means
>> scan
>> > > > > metrics
>> > > > > > >> at
>> > > > > > >> > > scale
>> > > > > > >> > > > > > will
>> > > > > > >> > > > > > > > have
>> > > > > > >> > > > > > > > > > > high
>> > > > > > >> > > > > > > > > > > > >> > write concurrency per table.
>> Schema
>> > > needs
>> > > > > to
>> > > > > > >> > expose
>> > > > > > >> > > > > enough
>> > > > > > >> > > > > > > > > > structure
>> > > > > > >> > > > > > > > > > > > for
>> > > > > > >> > > > > > > > > > > > >> > backends to shard by entity or
>> time
>> > > range.
>> > > > > > >> > > > > > > > > > > > >> >
>> > > > > > >> > > > > > > > > > > > >> > 4.  Read/write path consistency
>> > > > > > >> > > > > > > > > > > > >> >    •   Writes go through
>> > > > > > PolarisMetricsManager
>> > > > > > >> on
>> > > > > > >> > > > > > > > > > MetaStoreManager.
>> > > > > > >> > > > > > > > > > > > >> Reads
>> > > > > > >> > > > > > > > > > > > >> > bypass MetaStoreManager and go
>> > > straight to
>> > > > > > >> > > > > > BasePersistence,
>> > > > > > >> > > > > > > > > > > excluding
>> > > > > > >> > > > > > > > > > > > >> > non-JDBC backends from the read
>> API.
>> > > > > > >> > > > > > > > > > > > >> >
>> > > > > > >> > > > > > > > > > > > >> > *Third: cleanup and alignment*
>> > > > > > >> > > > > > > > > > > > >> >
>> > > > > > >> > > > > > > > > > > > >> > 5.  PolarisMetricsReporter naming
>> > > > > > >> > > > > > > > > > > > >> >    •   Only handles IRC
>> > > > > > >> (ScanReport/CommitReport),
>> > > > > > >> > > > > doesn't
>> > > > > > >> > > > > > > > cover
>> > > > > > >> > > > > > > > > > > > generic
>> > > > > > >> > > > > > > > > > > > >> > tables or operational metrics.
>> Name
>> > is
>> > > > > > broader
>> > > > > > >> > than
>> > > > > > >> > > > > scope.
>> > > > > > >> > > > > > > > > > > > >> >
>> > > > > > >> > > > > > > > > > > > >> > 6.  PolarisMetricsManager facade
>> > > > > passthrough
>> > > > > > >> > > > > > > > > > > > >> >    •   Entire default method is
>> > > > > > >> > > > > > > > > > > > >>
>> > callCtx.getMetaStore().writeScanReport().
>> > > > > > >> > > > > > > > > > > > >> > Zero logic, passes Level 1
>> straight
>> > > through
>> > > > > > to
>> > > > > > >> > Level
>> > > > > > >> > > > 3.
>> > > > > > >> > > > > > Same
>> > > > > > >> > > > > > > > > > > > >> anti-pattern
>> > > > > > >> > > > > > > > > > > > >> > as PolarisEventManager.
>> > > > > > >> > > > > > > > > > > > >> >
>> > > > > > >> > > > > > > > > > > > >> > 7.  Iceberg community alignment
>> > > > > > >> > > > > > > > > > > > >> >    •   Payload-type extension
>> needs
>> > > > > > discussion
>> > > > > > >> on
>> > > > > > >> > > > > > > dev@iceberg.
>> > > > > > >> > > > > > > > > > > > >> obelix74's
>> > > > > > >> > > > > > > > > > > > >> > Feb thread got zero replies.
>> Needs a
>> > > > > > committer
>> > > > > > >> > > voice.
>> > > > > > >> > > > > > > > > > > > >> >
>> > > > > > >> > > > > > > > > > > > >> > Lets confirm prioritization in the
>> > > first
>> > > > > > >> session.
>> > > > > > >> > > > > > > > > > > > >> >
>> > > > > > >> > > > > > > > > > > > >> > -ej
>> > > > > > >> > > > > > > > > > > > >> >
>> > > > > > >> > > > > > > > > > > > >> > On Tue, Apr 21, 2026 at 3:18 PM
>> Yufei
>> > > Gu <
>> > > > > > >> > > > > > > > flyrain000@gmail.com>
>> > > > > > >> > > > > > > > > > > > wrote:
>> > > > > > >> > > > > > > > > > > > >> >
>> > > > > > >> > > > > > > > > > > > >> >> Thanks everyone for continuing to
>> > > drive
>> > > > > this
>> > > > > > >> > > > forward. I
>> > > > > > >> > > > > > > agree
>> > > > > > >> > > > > > > > > > that
>> > > > > > >> > > > > > > > > > > > the
>> > > > > > >> > > > > > > > > > > > >> >> problem is getting complex enough
>> > > that a
>> > > > > > more
>> > > > > > >> > > > > structured
>> > > > > > >> > > > > > > > > > discussion
>> > > > > > >> > > > > > > > > > > > >> would
>> > > > > > >> > > > > > > > > > > > >> >> help.
>> > > > > > >> > > > > > > > > > > > >> >>
>> > > > > > >> > > > > > > > > > > > >> >> +1 on setting up a biweekly sync
>> for
>> > > the
>> > > > > > >> metrics
>> > > > > > >> > > > > > > > architecture.
>> > > > > > >> > > > > > > > > > I’m
>> > > > > > >> > > > > > > > > > > > >> happy
>> > > > > > >> > > > > > > > > > > > >> >> to
>> > > > > > >> > > > > > > > > > > > >> >> join.
>> > > > > > >> > > > > > > > > > > > >> >>
>> > > > > > >> > > > > > > > > > > > >> >> Yufei
>> > > > > > >> > > > > > > > > > > > >> >>
>> > > > > > >> > > > > > > > > > > > >> >>
>> > > > > > >> > > > > > > > > > > > >> >> On Tue, Apr 21, 2026 at 2:34 PM
>> EJ
>> > > Wang <
>> > > > > > >> > > > > > > > > > > > >> ej.wang.dev.interest@gmail.com>
>> > > > > > >> > > > > > > > > > > > >> >> wrote:
>> > > > > > >> > > > > > > > > > > > >> >>
>> > > > > > >> > > > > > > > > > > > >> >> > Also, I've been looking more
>> > > closely at
>> > > > > > the
>> > > > > > >> > > > > > *persistence
>> > > > > > >> > > > > > > > > schema
>> > > > > > >> > > > > > > > > > > in
>> > > > > > >> > > > > > > > > > > > >> the
>> > > > > > >> > > > > > > > > > > > >> >> > current metrics work*, and I
>> think
>> > > > > > there's a
>> > > > > > >> > > > > structural
>> > > > > > >> > > > > > > > > > rigidity
>> > > > > > >> > > > > > > > > > > > >> problem
>> > > > > > >> > > > > > > > > > > > >> >> > worth raising before the shape
>> > gets
>> > > > > locked
>> > > > > > >> in.
>> > > > > > >> > > > > > > > > > > > >> >> >
>> > > > > > >> > > > > > > > > > > > >> >> > Right now we have two separate
>> > > tables
>> > > > > > >> > > > > > > (scan_metrics_report
>> > > > > > >> > > > > > > > > and
>> > > > > > >> > > > > > > > > > > > >> >> > commit_metrics_report), each
>> with
>> > > ~25
>> > > > > > >> flattened
>> > > > > > >> > > > > columns
>> > > > > > >> > > > > > > > that
>> > > > > > >> > > > > > > > > > > > directly
>> > > > > > >> > > > > > > > > > > > >> >> > mirror the Iceberg report
>> fields.
>> > > The
>> > > > > SPI
>> > > > > > >> > follows
>> > > > > > >> > > > the
>> > > > > > >> > > > > > > same
>> > > > > > >> > > > > > > > > > split:
>> > > > > > >> > > > > > > > > > > > >> >> > writeScanReport and
>> > > writeCommitReport as
>> > > > > > >> > separate
>> > > > > > >> > > > > > > methods,
>> > > > > > >> > > > > > > > > with
>> > > > > > >> > > > > > > > > > > > >> per-type
>> > > > > > >> > > > > > > > > > > > >> >> > record classes, converters, and
>> > > model
>> > > > > > >> objects.
>> > > > > > >> > > *The
>> > > > > > >> > > > > > > > practical
>> > > > > > >> > > > > > > > > > > cost:
>> > > > > > >> > > > > > > > > > > > >> >> > adding a new metric type
>> > > (operational
>> > > > > > >> metrics,
>> > > > > > >> > > for
>> > > > > > >> > > > > > > example)
>> > > > > > >> > > > > > > > > > > > requires
>> > > > > > >> > > > > > > > > > > > >> a
>> > > > > > >> > > > > > > > > > > > >> >> new
>> > > > > > >> > > > > > > > > > > > >> >> > table, a new SPI method, a new
>> > > record
>> > > > > > >> class, a
>> > > > > > >> > > new
>> > > > > > >> > > > > > model
>> > > > > > >> > > > > > > > > > class, a
>> > > > > > >> > > > > > > > > > > > new
>> > > > > > >> > > > > > > > > > > > >> >> > converter branch, and a schema
>> > > > > migration*.
>> > > > > > >> > > That's a
>> > > > > > >> > > > > lot
>> > > > > > >> > > > > > > of
>> > > > > > >> > > > > > > > > > > surface
>> > > > > > >> > > > > > > > > > > > >> area
>> > > > > > >> > > > > > > > > > > > >> >> > for what should be "one more
>> kind
>> > of
>> > > > > > >> metric."
>> > > > > > >> > > > > > > > > > > > >> >> >
>> > > > > > >> > > > > > > > > > > > >> >> > *My bias* would be toward a
>> single
>> > > > > metrics
>> > > > > > >> > table
>> > > > > > >> > > > with
>> > > > > > >> > > > > > *a
>> > > > > > >> > > > > > > > > typed
>> > > > > > >> > > > > > > > > > > JSON
>> > > > > > >> > > > > > > > > > > > >> >> > payload*. Something like:
>> > > metric_type
>> > > > > > >> (enum),
>> > > > > > >> > > > > > entity_id,
>> > > > > > >> > > > > > > > > > > > >> >> > table_identifier, snapshot_id
>> > > > > (nullable),
>> > > > > > >> > > > > received_ts,
>> > > > > > >> > > > > > > > > > > > >> schema_version,
>> > > > > > >> > > > > > > > > > > > >> >> and
>> > > > > > >> > > > > > > > > > > > >> >> > a payload column for the
>> > > metric-specific
>> > > > > > >> data.
>> > > > > > >> > > The
>> > > > > > >> > > > > > > > > metric_type
>> > > > > > >> > > > > > > > > > +
>> > > > > > >> > > > > > > > > > > > >> >> > schema_version pair gives us a
>> > > > > > >> > forward-compatible
>> > > > > > >> > > > > > > contract
>> > > > > > >> > > > > > > > > for
>> > > > > > >> > > > > > > > > > > the
>> > > > > > >> > > > > > > > > > > > >> >> payload
>> > > > > > >> > > > > > > > > > > > >> >> > shape. Adding a new metric type
>> > > becomes
>> > > > > an
>> > > > > > >> enum
>> > > > > > >> > > > value
>> > > > > > >> > > > > > > and a
>> > > > > > >> > > > > > > > > > > payload
>> > > > > > >> > > > > > > > > > > > >> >> schema,
>> > > > > > >> > > > > > > > > > > > >> >> > not a schema migration. One
>> thing
>> > I
>> > > > > think
>> > > > > > we
>> > > > > > >> > need
>> > > > > > >> > > > to
>> > > > > > >> > > > > be
>> > > > > > >> > > > > > > > > > > deliberate
>> > > > > > >> > > > > > > > > > > > >> >> about is
>> > > > > > >> > > > > > > > > > > > >> >> > the partition key design. If
>> all
>> > > metric
>> > > > > > >> types
>> > > > > > >> > > land
>> > > > > > >> > > > in
>> > > > > > >> > > > > > one
>> > > > > > >> > > > > > > > > > table,
>> > > > > > >> > > > > > > > > > > > scan
>> > > > > > >> > > > > > > > > > > > >> >> > metrics at scale (high
>> > concurrency,
>> > > high
>> > > > > > >> > > frequency
>> > > > > > >> > > > > > across
>> > > > > > >> > > > > > > > > many
>> > > > > > >> > > > > > > > > > > > >> tables)
>> > > > > > >> > > > > > > > > > > > >> >> > could easily create hot
>> > partitions.
>> > > We'd
>> > > > > > >> want
>> > > > > > >> > the
>> > > > > > >> > > > > > > > persistence
>> > > > > > >> > > > > > > > > > > layer
>> > > > > > >> > > > > > > > > > > > >> to
>> > > > > > >> > > > > > > > > > > > >> >> be
>> > > > > > >> > > > > > > > > > > > >> >> > able to shard by entity or time
>> > > range,
>> > > > > and
>> > > > > > >> that
>> > > > > > >> > > > means
>> > > > > > >> > > > > > the
>> > > > > > >> > > > > > > > > > logical
>> > > > > > >> > > > > > > > > > > > >> schema
>> > > > > > >> > > > > > > > > > > > >> >> > needs to expose enough
>> structure
>> > for
>> > > > > > >> backends
>> > > > > > >> > to
>> > > > > > >> > > > > > > partition
>> > > > > > >> > > > > > > > > on.
>> > > > > > >> > > > > > > > > > I
>> > > > > > >> > > > > > > > > > > > >> don't
>> > > > > > >> > > > > > > > > > > > >> >> > think the current flattened
>> layout
>> > > gives
>> > > > > > us
>> > > > > > >> > that.
>> > > > > > >> > > > > > > > > > > > >> >> >
>> > > > > > >> > > > > > > > > > > > >> >> > This is getting complex enough
>> > that
>> > > I
>> > > > > > don't
>> > > > > > >> > think
>> > > > > > >> > > > > > ad-hoc
>> > > > > > >> > > > > > > > > PR/ML
>> > > > > > >> > > > > > > > > > > > >> threads
>> > > > > > >> > > > > > > > > > > > >> >> > will converge well. *Would
>> people
>> > be
>> > > > > open
>> > > > > > >> to a
>> > > > > > >> > > > > biweekly
>> > > > > > >> > > > > > > > sync
>> > > > > > >> > > > > > > > > > for
>> > > > > > >> > > > > > > > > > > > >> metrics
>> > > > > > >> > > > > > > > > > > > >> >> > architecture?* I think 30
>> minutes
>> > > every
>> > > > > > two
>> > > > > > >> > weeks
>> > > > > > >> > > > > with
>> > > > > > >> > > > > > > > > > interested
>> > > > > > >> > > > > > > > > > > > >> >> parties
>> > > > > > >> > > > > > > > > > > > >> >> > would be enough to work through
>> > the
>> > > > > > schema,
>> > > > > > >> SPI
>> > > > > > >> > > > > shape,
>> > > > > > >> > > > > > > and
>> > > > > > >> > > > > > > > > read
>> > > > > > >> > > > > > > > > > > API
>> > > > > > >> > > > > > > > > > > > >> >> design
>> > > > > > >> > > > > > > > > > > > >> >> > together. Happy to help set
>> that
>> > up.
>> > > > > > >> > > > > > > > > > > > >> >> >
>> > > > > > >> > > > > > > > > > > > >> >> > -ej
>> > > > > > >> > > > > > > > > > > > >> >> >
>> > > > > > >> > > > > > > > > > > > >> >> > On Mon, Apr 20, 2026 at
>> 2:19 PM EJ
>> > > Wang
>> > > > > <
>> > > > > > >> > > > > > > > > > > > >> ej.wang.dev.interest@gmail.com
>> > > > > > >> > > > > > > > > > > > >> >> >
>> > > > > > >> > > > > > > > > > > > >> >> > wrote:
>> > > > > > >> > > > > > > > > > > > >> >> >
>> > > > > > >> > > > > > > > > > > > >> >> >> Reviewed #4115, left a
>> comment on
>> > > the
>> > > > > > code
>> > > > > > >> > > > > > organization
>> > > > > > >> > > > > > > > > side.
>> > > > > > >> > > > > > > > > > > > >> >> >>
>> > > > > > >> > > > > > > > > > > > >> >> >> One thing stood out: the
>> metrics
>> > > write
>> > > > > > path
>> > > > > > >> > > enters
>> > > > > > >> > > > > > > through
>> > > > > > >> > > > > > > > > > > > >> >> >> PolarisMetricsManager on
>> > > > > > MetaStoreManager,
>> > > > > > >> but
>> > > > > > >> > > the
>> > > > > > >> > > > > new
>> > > > > > >> > > > > > > > read
>> > > > > > >> > > > > > > > > > path
>> > > > > > >> > > > > > > > > > > > >> >> bypasses
>> > > > > > >> > > > > > > > > > > > >> >> >> MetaStoreManager entirely and
>> > goes
>> > > > > > >> straight to
>> > > > > > >> > > > > > > > > BasePersistence
>> > > > > > >> > > > > > > > > > > via
>> > > > > > >> > > > > > > > > > > > >> >> >> callContext.getMetaStore().
>> That
>> > > means
>> > > > > > the
>> > > > > > >> > read
>> > > > > > >> > > > API
>> > > > > > >> > > > > > only
>> > > > > > >> > > > > > > > > works
>> > > > > > >> > > > > > > > > > > for
>> > > > > > >> > > > > > > > > > > > >> >> backends
>> > > > > > >> > > > > > > > > > > > >> >> >> that implement
>> BasePersistence.
>> > > NoSQL
>> > > > > and
>> > > > > > >> > remote
>> > > > > > >> > > > > > > backends
>> > > > > > >> > > > > > > > > > can't
>> > > > > > >> > > > > > > > > > > > >> >> participate.
>> > > > > > >> > > > > > > > > > > > >> >> >>
>> > > > > > >> > > > > > > > > > > > >> >> >> Stepping back, I think the
>> > metrics
>> > > > > > >> subsystem
>> > > > > > >> > is
>> > > > > > >> > > > > > growing
>> > > > > > >> > > > > > > > into
>> > > > > > >> > > > > > > > > > > > >> something
>> > > > > > >> > > > > > > > > > > > >> >> >> real (write + read + REST API
>> +
>> > > AuthZ +
>> > > > > > >> > > > pagination)
>> > > > > > >> > > > > > *but
>> > > > > > >> > > > > > > > the
>> > > > > > >> > > > > > > > > > > > >> >> persistence
>> > > > > > >> > > > > > > > > > > > >> >> >> side is split across two
>> layers
>> > in
>> > > a
>> > > > > way
>> > > > > > >> > that's
>> > > > > > >> > > > hard
>> > > > > > >> > > > > > to
>> > > > > > >> > > > > > > > > > > extend*. I
>> > > > > > >> > > > > > > > > > > > >> put
>> > > > > > >> > > > > > > > > > > > >> >> >> together two diagrams to show
>> > what
>> > > I
>> > > > > mean
>> > > > > > >> (my
>> > > > > > >> > > best
>> > > > > > >> > > > > > > > effort).
>> > > > > > >> > > > > > > > > > > > >> >> >>
>> > > > > > >> > > > > > > > > > > > >> >> >> *Current state* (Diagram 1):
>> > three
>> > > > > > >> interfaces
>> > > > > > >> > at
>> > > > > > >> > > > > three
>> > > > > > >> > > > > > > > > > different
>> > > > > > >> > > > > > > > > > > > >> >> levels.
>> > > > > > >> > > > > > > > > > > > >> >> >> The engine-facing SPI
>> > > > > > >> (PolarisMetricsReporter)
>> > > > > > >> > > is
>> > > > > > >> > > > > > clean.
>> > > > > > >> > > > > > > > But
>> > > > > > >> > > > > > > > > > > > >> >> >> PolarisMetricsManager on
>> > > > > MetaStoreManager
>> > > > > > >> is a
>> > > > > > >> > > > > > > passthrough
>> > > > > > >> > > > > > > > > to
>> > > > > > >> > > > > > > > > > > > >> >> >> MetricsPersistence on
>> > > BasePersistence.
>> > > > > > The
>> > > > > > >> > @Beta
>> > > > > > >> > > > > > > > annotation
>> > > > > > >> > > > > > > > > > and
>> > > > > > >> > > > > > > > > > > > SPI
>> > > > > > >> > > > > > > > > > > > >> >> javadoc
>> > > > > > >> > > > > > > > > > > > >> >> >> are on the BasePersistence
>> layer,
>> > > while
>> > > > > > the
>> > > > > > >> > > actual
>> > > > > > >> > > > > > > > extension
>> > > > > > >> > > > > > > > > > > > points
>> > > > > > >> > > > > > > > > > > > >> >> >> (PolarisMetricsReporter,
>> > > > > > >> > PolarisMetricsManager)
>> > > > > > >> > > > > carry
>> > > > > > >> > > > > > no
>> > > > > > >> > > > > > > > > > > stability
>> > > > > > >> > > > > > > > > > > > >> >> >> annotation. The write path
>> goes
>> > > through
>> > > > > > the
>> > > > > > >> > > > > > > > MetaStoreManager
>> > > > > > >> > > > > > > > > > > > layer,
>> > > > > > >> > > > > > > > > > > > >> the
>> > > > > > >> > > > > > > > > > > > >> >> >> read path doesn't.
>> > > > > > >> > > > > > > > > > > > >> >> >>
>> > > > > > >> > > > > > > > > > > > >> >> >> *What I envision* (Diagram 2):
>> > two
>> > > SPIs
>> > > > > > at
>> > > > > > >> two
>> > > > > > >> > > > > levels.
>> > > > > > >> > > > > > > > > > > > >> >> >> PolarisMetricsReporter stays
>> as
>> > the
>> > > > > > >> > > engine-facing
>> > > > > > >> > > > > SPI.
>> > > > > > >> > > > > > > > > > > > >> >> >> PolarisMetricsManager becomes
>> the
>> > > > > > >> > backend-facing
>> > > > > > >> > > > SPI
>> > > > > > >> > > > > > > with
>> > > > > > >> > > > > > > > > both
>> > > > > > >> > > > > > > > > > > > write
>> > > > > > >> > > > > > > > > > > > >> >> and
>> > > > > > >> > > > > > > > > > > > >> >> >> read methods at the
>> > > MetaStoreManager
>> > > > > > level,
>> > > > > > >> > > where
>> > > > > > >> > > > > any
>> > > > > > >> > > > > > > > > backend
>> > > > > > >> > > > > > > > > > > > (JDBC,
>> > > > > > >> > > > > > > > > > > > >> >> NoSQL,
>> > > > > > >> > > > > > > > > > > > >> >> >> remote) can implement them.
>> > > > > > >> MetricsPersistence
>> > > > > > >> > > on
>> > > > > > >> > > > > > > > > > > BasePersistence
>> > > > > > >> > > > > > > > > > > > >> goes
>> > > > > > >> > > > > > > > > > > > >> >> >> away. Where metrics actually
>> land
>> > > is an
>> > > > > > >> > > > > implementation
>> > > > > > >> > > > > > > > > detail,
>> > > > > > >> > > > > > > > > > > > not a
>> > > > > > >> > > > > > > > > > > > >> >> core
>> > > > > > >> > > > > > > > > > > > >> >> >> interface.
>> > > > > > >> > > > > > > > > > > > >> >> >>
>> > > > > > >> > > > > > > > > > > > >> >> >> *Minor naming thing*:
>> > > > > > >> PolarisMetricsReporter
>> > > > > > >> > is
>> > > > > > >> > > > > > broader
>> > > > > > >> > > > > > > > than
>> > > > > > >> > > > > > > > > > > what
>> > > > > > >> > > > > > > > > > > > it
>> > > > > > >> > > > > > > > > > > > >> >> >> actually handles. It only
>> accepts
>> > > > > Iceberg
>> > > > > > >> REST
>> > > > > > >> > > > > Catalog
>> > > > > > >> > > > > > > > > metrics
>> > > > > > >> > > > > > > > > > > > >> >> (ScanReport,
>> > > > > > >> > > > > > > > > > > > >> >> >> CommitReport via
>> MetricsReport).
>> > > > > Generic
>> > > > > > >> table
>> > > > > > >> > > > > metrics
>> > > > > > >> > > > > > > or
>> > > > > > >> > > > > > > > > > > > >> operational
>> > > > > > >> > > > > > > > > > > > >> >> >> metrics aren't in scope. Not
>> > > blocking,
>> > > > > > but
>> > > > > > >> > worth
>> > > > > > >> > > > > > noting
>> > > > > > >> > > > > > > if
>> > > > > > >> > > > > > > > > the
>> > > > > > >> > > > > > > > > > > > >> metrics
>> > > > > > >> > > > > > > > > > > > >> >> >> surface expands.
>> > > > > > >> > > > > > > > > > > > >> >> >>
>> > > > > > >> > > > > > > > > > > > >> >> >> *Rough sketch of how to get
>> > there*:
>> > > > > > >> > > > > > > > > > > > >> >> >>  1.  Add read methods to
>> > > > > > >> PolarisMetricsManager
>> > > > > > >> > > > > > > > > > (listScanReports,
>> > > > > > >> > > > > > > > > > > > >> >> >> listCommitReports) with
>> default
>> > > no-op,
>> > > > > > >> same as
>> > > > > > >> > > the
>> > > > > > >> > > > > > > > existing
>> > > > > > >> > > > > > > > > > > write
>> > > > > > >> > > > > > > > > > > > >> >> methods.
>> > > > > > >> > > > > > > > > > > > >> >> >> (Probably make
>> > > PolarisMetricsManager
>> > > > > more
>> > > > > > >> > > explicit
>> > > > > > >> > > > > on
>> > > > > > >> > > > > > > > being
>> > > > > > >> > > > > > > > > > > > Iceberg
>> > > > > > >> > > > > > > > > > > > >> >> >> specific like package name or
>> > class
>> > > > > name
>> > > > > > >> etc.)
>> > > > > > >> > > > > > > > > > > > >> >> >>  2.  Wire
>> MetricsReportsService
>> > > through
>> > > > > > >> > > > > > MetaStoreManager
>> > > > > > >> > > > > > > > > > instead
>> > > > > > >> > > > > > > > > > > > of
>> > > > > > >> > > > > > > > > > > > >> >> >> callContext.getMetaStore().
>> > > > > > >> > > > > > > > > > > > >> >> >>  3.  Extract metrics
>> persistence
>> > > from
>> > > > > > >> > > > > > > > > JdbcBasePersistenceImpl
>> > > > > > >> > > > > > > > > > > into
>> > > > > > >> > > > > > > > > > > > >> its
>> > > > > > >> > > > > > > > > > > > >> >> >> own class. That file carries
>> ~7
>> > > > > > >> > > responsibilities,
>> > > > > > >> > > > > > > metrics
>> > > > > > >> > > > > > > > > > being
>> > > > > > >> > > > > > > > > > > > one
>> > > > > > >> > > > > > > > > > > > >> of
>> > > > > > >> > > > > > > > > > > > >> >> them.
>> > > > > > >> > > > > > > > > > > > >> >> >>  4.  Remove MetricsPersistence
>> > from
>> > > > > > >> > > > BasePersistence.
>> > > > > > >> > > > > > > > > > > > >> >> >>
>> > > > > > >> > > > > > > > > > > > >> >> >> *None of this needs to happen
>> in
>> > > #4115.
>> > > > > > >> But if
>> > > > > > >> > > the
>> > > > > > >> > > > > > > > direction
>> > > > > > >> > > > > > > > > > > makes
>> > > > > > >> > > > > > > > > > > > >> >> sense,
>> > > > > > >> > > > > > > > > > > > >> >> >> it would be good to align
>> before
>> > > the
>> > > > > > >> metrics
>> > > > > > >> > > > surface
>> > > > > > >> > > > > > > grows
>> > > > > > >> > > > > > > > > > > > further.
>> > > > > > >> > > > > > > > > > > > >> >> Curious
>> > > > > > >> > > > > > > > > > > > >> >> >> what others think.*
>> > > > > > >> > > > > > > > > > > > >> >> >>
>> > > > > > >> > > > > > > > > > > > >> >> >> *My mental model note*: Level
>> 1
>> > > > > > >> > > MetaStoreManager;
>> > > > > > >> > > > > > level
>> > > > > > >> > > > > > > 2
>> > > > > > >> > > > > > > > > > > > >> transactional
>> > > > > > >> > > > > > > > > > > > >> >> >> persistence; level 3 base
>> > > persistence
>> > > > > > >> > > > > > > > > > > > >> >> >>
>> > > > > > >> > > > > > > > > > > > >> >> >> Diagram 1
>> > > > > > >> > > > > > > > > > > > >> >> >> <
>> > > > > > >> > > > > > > > > > > > >> >>
>> > > > > > >> > > > > > > > > > > > >>
>> > > > > > >> > > > > > > > > > > >
>> > > > > > >> > > > > > > > > > >
>> > > > > > >> > > > > > > > > >
>> > > > > > >> > > > > > > > >
>> > > > > > >> > > > > > > >
>> > > > > > >> > > > > > >
>> > > > > > >> > > > > >
>> > > > > > >> > > > >
>> > > > > > >> > > >
>> > > > > > >> > >
>> > > > > > >> >
>> > > > > > >>
>> > > > > >
>> > > > >
>> > >
>> >
>> https://urldefense.com/v3/__https://www.plantuml.com/plantuml/uml/bLHDR-Cs4BthLmpIYupw0zbkKQ1r3M-S7Bp8xhhM7WCOb3IM65EaGD9EX2RzxHrHb4CxRelwa4YSDu_lpOVcnZ9jzvM8BBS2uGjQpJC3dtHMSekPtMk44IpsMgEqa5XcCOhCZikQQLP1pR8TAp2n3ILhmZDP20m0fcIvUkAoW2qJXd9z1bpToO9BX3WXu0ucy5rpgGPNm0nW5_epUWtm2Ue3pn3kMOFQmKntGZW0BYtgBSi8k5A2QMwybJNMIbFiGSR9QZc4nUqIvikStF0jHprua5C-amge42aNt3R0f5JaaoivdV2Pkqbx4hee4ymOkBh5BTiB-_uIeGeo8zL8rPsPl4DktdEiK1jkB1NdZCRbrSTecDe_mlHbF0wvBmCkaOH5_S8a_TTTKI6-nmCAkEw4LpxsZ-LbYLKQFKMNOgf_wuM7_bV9gOer5SYMMksBSWXFcbi49KNZXNLicwfe3TETC7gPdPqI7uBcHMb1RSzYq34c6PDUM9mn8HRsUTZEiDBve3NjVZumBj0U7SS37mGO7vcwtiK-_pU7U7L_f-digo9YbhSwIfMRwIITKGXbxdIUTCGF1SeCJxloKsU-3k9ddRbX1eDq1q_fx1JbBGT0glVyXimDuP4TQ5qpCAmnGEj2s_6n5mtn1z-97-63itFQZLPO1Ev2tu_WF7Ju-VPc0Skg5bYXxBhkY1xpD7EM_7fyflSpIsqMgVth5xhVr4eQxWQ8enaSAJQSG16yFSDuJ798rrcXr_3n-lfdk7icQjEBmFujL7AodiP_Y4Z7-YxvtZNs4zMgpNTl6tF8sglyPsmqchrjvQ-m-aP94r-TwCA2Ka8upPJZwtvSpoYCXkYMZU2NXvRMBfq9P3i3Le4VAZUAlUZ_oPKsxPgY0Q_BSKLkyr9bhQhQrJjo_x3TPlIB0DPjnMfcIoYP0QaYw1a0fTKDr8fB6ntNuvmoL1ZGkXa69Njh43zf9GiGxHQrA_jDYWRSzF5--WmTVrN97_Sm8LbLUy_lGBmLanJjFkDlGkRqjA_4tm00__;!!Iz9xO38YGHZK!4rnlVIqBQQQ6xpGe1PMV58edOS_cGrKoqdfhw4qSp_JQCavvEjaxXk0OwGPnxyEcO60AzKQK_XDtp7H3DdkFXcrXuTBISHg$
>> > > > > > >> > > > > > > > > > > > >> >> >
>> > > > > > >> > > > > > > > > > > > >> >> >> :
>> > > > > > >> > > > > > > > > > > > >> >> >>
>> > > > > > >> > > > > > > > > > > > >> >> >> [image: image.png]
>> > > > > > >> > > > > > > > > > > > >> >> >>
>> > > > > > >> > > > > > > > > > > > >> >> >> Diagram 2
>> > > > > > >> > > > > > > > > > > > >> >> >> <
>> > > > > > >> > > > > > > > > > > > >> >>
>> > > > > > >> > > > > > > > > > > > >>
>> > > > > > >> > > > > > > > > > > >
>> > > > > > >> > > > > > > > > > >
>> > > > > > >> > > > > > > > > >
>> > > > > > >> > > > > > > > >
>> > > > > > >> > > > > > > >
>> > > > > > >> > > > > > >
>> > > > > > >> > > > > >
>> > > > > > >> > > > >
>> > > > > > >> > > >
>> > > > > > >> > >
>> > > > > > >> >
>> > > > > > >>
>> > > > > >
>> > > > >
>> > >
>> >
>> https://urldefense.com/v3/__https://www.plantuml.com/plantuml/uml/VLLDR-8m4BtdLupO2sWBLVU8AaGB7AXAbssGzb896SS9RXqxjKqBwkv_tt7iV43fdaZYDpFlpRmnOsE9jhjSH9PRmM31hERKm8scMsuPjJlDe0yheZDc8RR4iYWoBrmMH9CS2a9VICPYUy1OZN0YCy5Q0BCbYNhdCeEK28En8G8wCvbnoQ0R8_05Bc6bkLIz3X03p1zzH7zR-9ZfDquPt9C3qoNCX2yV4G2NbkcKu5jdgGJHt0GbZwnG6i-UP3TUpk5gM6Ldqke350eZUqzoCft3U9xWHvxoa5-7K4nF1J46EbEMafsmdrCBbQ44gVggy18IZrn_ph5asd1ZiIKdQSgueZvjXrQFSFrdC3YN-nXmBacxbGiYyLVxLaBtdhqn0LSzdBDhqQtQoOJeGyad3z0lUqnYgpGB6Ns8oVyta00Dy_WnX0tIOZ8v6SYxHll1TrH6aejAik-mh-AphVFCwSUQqFypElag5QRGFDjQKEd96K1P8QP41c9TzA_IIQyvdAWyv_RSiS3skb0_EzDDkK2v5xWF6MiGFlvhpFLcD2Dq2pml14gaF67eQkmd8gulDoC4kSOu6KVpkvlUJg1RTbWISU40RdBUUS_9XfRZ2dwxm_SW8LYFISgm_MnlDQ6M9P1gbKEc4X-2pH_FvJCkCqm9pbVjD6LrwdLeOrDWfOaqc8Wh9BE85oNKxkNQ6o4yGRy_Eae0G_G8tZv81d3bHDB23WOdisohVr3nh_j6lbSjbNaLRTc8UgtPbAU1J_tygOfZX9DWEJeHDvYx-qmSi5FgNLPZwHrHcUsncGQ5-skhUclpE5fo4ounpFauYrUbkU6ccfnxMvitwag4IyerhTxj8In_Oj1bDO4pQru674loYrGlULHLEGCjwJJ8gDoVZR8MxO4BT3IzRvIcAQKezC6xpziGnTyImrfEGyJI_OcKfgtxIvnTqFEMS17L9Z-jsARN5FmTheP7HtSdtOMT0B4GY2FYHXxgQmMtj2bRqiLFGapiVe1_QVKDrkqXcm83aFEXnMYCZ-xlyHy__;!!Iz9xO38YGHZK!4rnlVIqBQQQ6xpGe1PMV58edOS_cGrKoqdfhw4qSp_JQCavvEjaxXk0OwGPnxyEcO60AzKQK_XDtp7H3DdkFXcrXSpmqcNk$
>> > > > > > >> > > > > > > > > > > > >> >> >
>> > > > > > >> > > > > > > > > > > > >> >> >> :
>> > > > > > >> > > > > > > > > > > > >> >> >> [image: image.png]
>> > > > > > >> > > > > > > > > > > > >> >> >>
>> > > > > > >> > > > > > > > > > > > >> >> >>  -ej
>> > > > > > >> > > > > > > > > > > > >> >> >>
>> > > > > > >> > > > > > > > > > > > >> >> >> On Wed, Apr 15, 2026 at
>> 8:22 AM
>> > > Dmitri
>> > > > > > >> > > > Bourlatchkov
>> > > > > > >> > > > > <
>> > > > > > >> > > > > > > > > > > > >> dimas@apache.org>
>> > > > > > >> > > > > > > > > > > > >> >> >> wrote:
>> > > > > > >> > > > > > > > > > > > >> >> >>
>> > > > > > >> > > > > > > > > > > > >> >> >>> Hi All,
>> > > > > > >> > > > > > > > > > > > >> >> >>>
>> > > > > > >> > > > > > > > > > > > >> >> >>> Heads up: The current state
>> of
>> > PR
>> > > > > [4115]
>> > > > > > >> > looks
>> > > > > > >> > > > > pretty
>> > > > > > >> > > > > > > > solid
>> > > > > > >> > > > > > > > > > to
>> > > > > > >> > > > > > > > > > > > me.
>> > > > > > >> > > > > > > > > > > > >> I
>> > > > > > >> > > > > > > > > > > > >> >> >>> believe this PR is
>> approaching a
>> > > > > > mergeable
>> > > > > > >> > > > > condition.
>> > > > > > >> > > > > > > > > > > > >> >> >>>
>> > > > > > >> > > > > > > > > > > > >> >> >>> Please post your reviews if
>> you
>> > > have
>> > > > > any
>> > > > > > >> > > > comments.
>> > > > > > >> > > > > > > > > > > > >> >> >>>
>> > > > > > >> > > > > > > > > > > > >> >> >>> [4115]
>> > > > > > >> > > >
>> >
>> https://urldefense.com/v3/__https://github.com/apache/polaris/pull/4115__;!!Iz9xO38YGHZK!4rnlVIqBQQQ6xpGe1PMV58edOS_cGrKoqdfhw4qSp_JQCavvEjaxXk0OwGPnxyEcO60AzKQK_XDtp7H3DdkFXcrXtFRscKs$
>> > > > > > >> > > > > > > > > > > > >> >> >>>
>> > > > > > >> > > > > > > > > > > > >> >> >>> Thanks,
>> > > > > > >> > > > > > > > > > > > >> >> >>> Dmitri.
>> > > > > > >> > > > > > > > > > > > >> >> >>>
>> > > > > > >> > > > > > > > > > > > >> >> >>> On Tue, Mar 3, 2026 at
>> 3:29 PM
>> > > Anand
>> > > > > > Kumar
>> > > > > > >> > > > Sankaran
>> > > > > > >> > > > > > via
>> > > > > > >> > > > > > > > > dev <
>> > > > > > >> > > > > > > > > > > > >> >> >>> dev@polaris.apache.org>
>> wrote:
>> > > > > > >> > > > > > > > > > > > >> >> >>>
>> > > > > > >> > > > > > > > > > > > >> >> >>> > Hi Yufei and Dmitri,
>> > > > > > >> > > > > > > > > > > > >> >> >>> >
>> > > > > > >> > > > > > > > > > > > >> >> >>> > Here is a proposal for the
>> > REST
>> > > > > > >> endpoints
>> > > > > > >> > for
>> > > > > > >> > > > > > metrics
>> > > > > > >> > > > > > > > and
>> > > > > > >> > > > > > > > > > > > events.
>> > > > > > >> > > > > > > > > > > > >> >> >>> >
>> > > > > > >> > > > > > > > > > > > >> >> >>> >
>> > > > > > >> > > > > >
>> >
>> https://urldefense.com/v3/__https://github.com/apache/polaris/pull/3924/changes__;!!Iz9xO38YGHZK!4rnlVIqBQQQ6xpGe1PMV58edOS_cGrKoqdfhw4qSp_JQCavvEjaxXk0OwGPnxyEcO60AzKQK_XDtp7H3DdkFXcrXcjoPhSs$
>> > > > > > >> > > > > > > > > > > > >> >> >>> >
>> > > > > > >> > > > > > > > > > > > >> >> >>> > I did not see any
>> precursors
>> > for
>> > > > > > >> raising a
>> > > > > > >> > PR
>> > > > > > >> > > > for
>> > > > > > >> > > > > > > > > > proposals,
>> > > > > > >> > > > > > > > > > > so
>> > > > > > >> > > > > > > > > > > > >> >> trying
>> > > > > > >> > > > > > > > > > > > >> >> >>> > this.  Please let me know
>> what
>> > > you
>> > > > > > >> think.
>> > > > > > >> > > > > > > > > > > > >> >> >>> >
>> > > > > > >> > > > > > > > > > > > >> >> >>> > -
>> > > > > > >> > > > > > > > > > > > >> >> >>> > Anand
>> > > > > > >> > > > > > > > > > > > >> >> >>> >
>> > > > > > >> > > > > > > > > > > > >> >> >>> > From: Anand Kumar Sankaran
>> <
>> > > > > > >> > > > > > > anand.sankaran@workday.com
>> > > > > > >> > > > > > > > >
>> > > > > > >> > > > > > > > > > > > >> >> >>> > Date: Monday, March 2,
>> 2026 at
>> > > > > > 10:25 AM
>> > > > > > >> > > > > > > > > > > > >> >> >>> > To: dev@polaris.apache.org
>> <
>> > > > > > >> > > > > dev@polaris.apache.org
>> > > > > > >> > > > > > >
>> > > > > > >> > > > > > > > > > > > >> >> >>> > Subject: Re: Polaris
>> Telemetry
>> > > and
>> > > > > > Audit
>> > > > > > >> > > Trail
>> > > > > > >> > > > > > > > > > > > >> >> >>> >
>> > > > > > >> > > > > > > > > > > > >> >> >>> > About the REST API, based
>> on
>> > my
>> > > use
>> > > > > > >> cases:
>> > > > > > >> > > > > > > > > > > > >> >> >>> >
>> > > > > > >> > > > > > > > > > > > >> >> >>> >
>> > > > > > >> > > > > > > > > > > > >> >> >>> >   1.
>> > > > > > >> > > > > > > > > > > > >> >> >>> > I want to be able to query
>> > > commit
>> > > > > > >> metrics
>> > > > > > >> > to
>> > > > > > >> > > > > track
>> > > > > > >> > > > > > > > files
>> > > > > > >> > > > > > > > > > > added
>> > > > > > >> > > > > > > > > > > > /
>> > > > > > >> > > > > > > > > > > > >> >> >>> removed
>> > > > > > >> > > > > > > > > > > > >> >> >>> > per commit, along with
>> record
>> > > > > counts.
>> > > > > > >> The
>> > > > > > >> > > > > ingestion
>> > > > > > >> > > > > > > > > > pipeline
>> > > > > > >> > > > > > > > > > > > that
>> > > > > > >> > > > > > > > > > > > >> >> >>> writes
>> > > > > > >> > > > > > > > > > > > >> >> >>> > this data is owned by us
>> and
>> > we
>> > > are
>> > > > > > >> > > guaranteed
>> > > > > > >> > > > to
>> > > > > > >> > > > > > > write
>> > > > > > >> > > > > > > > > > this
>> > > > > > >> > > > > > > > > > > > >> >> >>> information
>> > > > > > >> > > > > > > > > > > > >> >> >>> > for each write.
>> > > > > > >> > > > > > > > > > > > >> >> >>> >   2.
>> > > > > > >> > > > > > > > > > > > >> >> >>> > I want to be able to query
>> > scan
>> > > > > > metrics
>> > > > > > >> for
>> > > > > > >> > > > > read. I
>> > > > > > >> > > > > > > > > > > understand
>> > > > > > >> > > > > > > > > > > > >> >> clients
>> > > > > > >> > > > > > > > > > > > >> >> >>> do
>> > > > > > >> > > > > > > > > > > > >> >> >>> > not fulfill this
>> requirement.
>> > > > > > >> > > > > > > > > > > > >> >> >>> >   3.
>> > > > > > >> > > > > > > > > > > > >> >> >>> > I want to be able to query
>> the
>> > > > > events
>> > > > > > >> table
>> > > > > > >> > > > > (events
>> > > > > > >> > > > > > > are
>> > > > > > >> > > > > > > > > > > > >> persisted) -
>> > > > > > >> > > > > > > > > > > > >> >> >>> this
>> > > > > > >> > > > > > > > > > > > >> >> >>> > may supersede #2, I am not
>> > sure
>> > > yet.
>> > > > > > >> > > > > > > > > > > > >> >> >>> >
>> > > > > > >> > > > > > > > > > > > >> >> >>> > All this information is in
>> the
>> > > JDBC
>> > > > > > >> based
>> > > > > > >> > > > > > persistence
>> > > > > > >> > > > > > > > > model
>> > > > > > >> > > > > > > > > > > and
>> > > > > > >> > > > > > > > > > > > >> is
>> > > > > > >> > > > > > > > > > > > >> >> >>> > persisted in the
>> metastore. I
>> > > > > > currently
>> > > > > > >> > don’t
>> > > > > > >> > > > > have
>> > > > > > >> > > > > > a
>> > > > > > >> > > > > > > > need
>> > > > > > >> > > > > > > > > > to
>> > > > > > >> > > > > > > > > > > > >> query
>> > > > > > >> > > > > > > > > > > > >> >> >>> > prometheus or open
>> telemetry.
>> > I
>> > > do
>> > > > > > >> publish
>> > > > > > >> > > some
>> > > > > > >> > > > > > > events
>> > > > > > >> > > > > > > > to
>> > > > > > >> > > > > > > > > > > > >> Prometheus
>> > > > > > >> > > > > > > > > > > > >> >> >>> and
>> > > > > > >> > > > > > > > > > > > >> >> >>> > they are forwarded to our
>> > > dashboards
>> > > > > > >> > > elsewhere.
>> > > > > > >> > > > > > > > > > > > >> >> >>> >
>> > > > > > >> > > > > > > > > > > > >> >> >>> > About the CLI utilities, I
>> > > meant the
>> > > > > > >> admin
>> > > > > > >> > > user
>> > > > > > >> > > > > > > > > utilities.
>> > > > > > >> > > > > > > > > > In
>> > > > > > >> > > > > > > > > > > > >> one of
>> > > > > > >> > > > > > > > > > > > >> >> >>> the
>> > > > > > >> > > > > > > > > > > > >> >> >>> > earliest drafts of my
>> > proposal,
>> > > > > > Prashant
>> > > > > > >> > > > > mentioned
>> > > > > > >> > > > > > > that
>> > > > > > >> > > > > > > > > the
>> > > > > > >> > > > > > > > > > > > >> metrics
>> > > > > > >> > > > > > > > > > > > >> >> >>> tables
>> > > > > > >> > > > > > > > > > > > >> >> >>> > can grow indefinitely and
>> > that a
>> > > > > > similar
>> > > > > > >> > > > problem
>> > > > > > >> > > > > > > exists
>> > > > > > >> > > > > > > > > > with
>> > > > > > >> > > > > > > > > > > > the
>> > > > > > >> > > > > > > > > > > > >> >> events
>> > > > > > >> > > > > > > > > > > > >> >> >>> > table as well. We discussed
>> > that
>> > > > > > >> cleaning
>> > > > > > >> > up
>> > > > > > >> > > of
>> > > > > > >> > > > > old
>> > > > > > >> > > > > > > > > records
>> > > > > > >> > > > > > > > > > > > from
>> > > > > > >> > > > > > > > > > > > >> >> both
>> > > > > > >> > > > > > > > > > > > >> >> >>> > metrics tables and events
>> > > tables can
>> > > > > > be
>> > > > > > >> > done
>> > > > > > >> > > > via
>> > > > > > >> > > > > a
>> > > > > > >> > > > > > > CLI
>> > > > > > >> > > > > > > > > > > utility.
>> > > > > > >> > > > > > > > > > > > >> >> >>> >
>> > > > > > >> > > > > > > > > > > > >> >> >>> > I see that Yufei has
>> covered
>> > the
>> > > > > > >> discussion
>> > > > > > >> > > > about
>> > > > > > >> > > > > > > > > > > datasources.
>> > > > > > >> > > > > > > > > > > > >> >> >>> >
>> > > > > > >> > > > > > > > > > > > >> >> >>> > -
>> > > > > > >> > > > > > > > > > > > >> >> >>> > Anand
>> > > > > > >> > > > > > > > > > > > >> >> >>> >
>> > > > > > >> > > > > > > > > > > > >> >> >>> >
>> > > > > > >> > > > > > > > > > > > >> >> >>> >
>> > > > > > >> > > > > > > > > > > > >> >> >>> > From: Yufei Gu <
>> > > > > flyrain000@gmail.com>
>> > > > > > >> > > > > > > > > > > > >> >> >>> > Date: Friday, February 27,
>> > 2026
>> > > at
>> > > > > > >> 9:54 PM
>> > > > > > >> > > > > > > > > > > > >> >> >>> > To: dev@polaris.apache.org
>> <
>> > > > > > >> > > > > dev@polaris.apache.org
>> > > > > > >> > > > > > >
>> > > > > > >> > > > > > > > > > > > >> >> >>> > Subject: Re: Polaris
>> Telemetry
>> > > and
>> > > > > > Audit
>> > > > > > >> > > Trail
>> > > > > > >> > > > > > > > > > > > >> >> >>> >
>> > > > > > >> > > > > > > > > > > > >> >> >>> > This Message Is From an
>> > External
>> > > > > > Sender
>> > > > > > >> > > > > > > > > > > > >> >> >>> > This message came from
>> outside
>> > > your
>> > > > > > >> > > > organization.
>> > > > > > >> > > > > > > > > > > > >> >> >>> > Report Suspicious<
>> > > > > > >> > > > > > > > > > > > >> >> >>> >
>> > > > > > >> > > > > > > > > > > > >> >> >>>
>> > > > > > >> > > > > > > > > > > > >> >>
>> > > > > > >> > > > > > > > > > > > >>
>> > > > > > >> > > > > > > > > > > >
>> > > > > > >> > > > > > > > > > >
>> > > > > > >> > > > > > > > > >
>> > > > > > >> > > > > > > > >
>> > > > > > >> > > > > > > >
>> > > > > > >> > > > > > >
>> > > > > > >> > > > > >
>> > > > > > >> > > > >
>> > > > > > >> > > >
>> > > > > > >> > >
>> > > > > > >> >
>> > > > > > >>
>> > > > > >
>> > > > >
>> > >
>> >
>> https://us-phishalarm-ewt.proofpoint.com/EWT/v1/Iz9xO38YGHZK!YhNDZABkHi1B699ote2uMwpOZw8i0QMCGO2Szc-HshuABGhGvwPJcymE6G2oUUxtS8xDkSrtGTPm_I3QnVDHoLMk50m9v8z_nZKTkd-bnVUbreF1u0WnfV_X5eYevZl_$
>> > > > > > >> > > > > > > > > > > > >> >> >>> > >
>> > > > > > >> > > > > > > > > > > > >> >> >>> >
>> > > > > > >> > > > > > > > > > > > >> >> >>> >
>> > > > > > >> > > > > > > > > > > > >> >> >>> > As I mentioned in
>> > > > > > >> > > > > > > > > > > > >> >> >>> >
>> > > > > > >> > > > > > > > > > > > >> >> >>>
>> > > > > > >> > > > > > > > > > > > >> >>
>> > > > > > >> > > > > > > > > > > > >>
>> > > > > > >> > > > > > > > > > > >
>> > > > > > >> > > > > > > > > > >
>> > > > > > >> > > > > > > > > >
>> > > > > > >> > > > > > > > >
>> > > > > > >> > > > > > > >
>> > > > > > >> > > > > > >
>> > > > > > >> > > > > >
>> > > > > > >> > > > >
>> > > > > > >> > > >
>> > > > > > >> > >
>> > > > > > >> >
>> > > > > > >>
>> > > > > >
>> > > > >
>> > >
>> >
>> https://urldefense.com/v3/__https://github.com/apache/polaris/issues/3890__;!!Iz9xO38YGHZK!5EuyFFkk3vhRWVIRvQAWBSQfpJkTMA9HxugzDwXmN0LPPqhEFxYkFRGVhtb8AqUwXtDh2OplcMnbMDHKOxrvDU0$
>> > >> > > >> > > > > > > > > > > > >> >> >>> ,
>> > > > > > >> > > > > > > > > > > > >> >> >>> > supporting
>> > > > > > >> > > > > > > > > > > > >> >> >>> > multiple data sources is
>> not a
>> > > > > trivial
>> > > > > > >> > > change.
>> > > > > > >> > > > I
>> > > > > > >> > > > > > > would
>> > > > > > >> > > > > > > > > > > strongly
>> > > > > > >> > > > > > > > > > > > >> >> >>> recommend
>> > > > > > >> > > > > > > > > > > > >> >> >>> > starting with a design
>> > document
>> > > to
>> > > > > > >> > carefully
>> > > > > > >> > > > > > evaluate
>> > > > > > >> > > > > > > > the
>> > > > > > >> > > > > > > > > > > > >> >> architectural
>> > > > > > >> > > > > > > > > > > > >> >> >>> > implications and long term
>> > > impact.
>> > > > > > >> > > > > > > > > > > > >> >> >>> >
>> > > > > > >> > > > > > > > > > > > >> >> >>> > A REST endpoint to query
>> > metrics
>> > > > > seems
>> > > > > > >> > > > reasonable
>> > > > > > >> > > > > > > given
>> > > > > > >> > > > > > > > > the
>> > > > > > >> > > > > > > > > > > > >> current
>> > > > > > >> > > > > > > > > > > > >> >> >>> JDBC
>> > > > > > >> > > > > > > > > > > > >> >> >>> > based persistence model.
>> That
>> > > said,
>> > > > > we
>> > > > > > >> may
>> > > > > > >> > > also
>> > > > > > >> > > > > > > > consider
>> > > > > > >> > > > > > > > > > > > >> alternative
>> > > > > > >> > > > > > > > > > > > >> >> >>> > storage models. For
>> example,
>> > if
>> > > we
>> > > > > > later
>> > > > > > >> > > adopt
>> > > > > > >> > > > a
>> > > > > > >> > > > > > time
>> > > > > > >> > > > > > > > > > series
>> > > > > > >> > > > > > > > > > > > >> system
>> > > > > > >> > > > > > > > > > > > >> >> >>> such as
>> > > > > > >> > > > > > > > > > > > >> >> >>> > Prometheus to store
>> metrics,
>> > the
>> > > > > query
>> > > > > > >> > model
>> > > > > > >> > > > and
>> > > > > > >> > > > > > > access
>> > > > > > >> > > > > > > > > > > > patterns
>> > > > > > >> > > > > > > > > > > > >> >> would
>> > > > > > >> > > > > > > > > > > > >> >> >>> be
>> > > > > > >> > > > > > > > > > > > >> >> >>> > fundamentally different.
>> > > Designing
>> > > > > the
>> > > > > > >> REST
>> > > > > > >> > > API
>> > > > > > >> > > > > > > without
>> > > > > > >> > > > > > > > > > > > >> considering
>> > > > > > >> > > > > > > > > > > > >> >> >>> these
>> > > > > > >> > > > > > > > > > > > >> >> >>> > potential evolutions may
>> limit
>> > > > > > >> flexibility.
>> > > > > > >> > > I'd
>> > > > > > >> > > > > > > suggest
>> > > > > > >> > > > > > > > > to
>> > > > > > >> > > > > > > > > > > > start
>> > > > > > >> > > > > > > > > > > > >> >> with
>> > > > > > >> > > > > > > > > > > > >> >> >>> the
>> > > > > > >> > > > > > > > > > > > >> >> >>> > use case.
>> > > > > > >> > > > > > > > > > > > >> >> >>> >
>> > > > > > >> > > > > > > > > > > > >> >> >>> > Yufei
>> > > > > > >> > > > > > > > > > > > >> >> >>> >
>> > > > > > >> > > > > > > > > > > > >> >> >>> >
>> > > > > > >> > > > > > > > > > > > >> >> >>> > On Fri, Feb 27, 2026 at
>> > 3:42 PM
>> > > > > Dmitri
>> > > > > > >> > > > > > Bourlatchkov <
>> > > > > > >> > > > > > > > > > > > >> >> dimas@apache.org>
>> > > > > > >> > > > > > > > > > > > >> >> >>> > wrote:
>> > > > > > >> > > > > > > > > > > > >> >> >>> >
>> > > > > > >> > > > > > > > > > > > >> >> >>> > > Hi Anand,
>> > > > > > >> > > > > > > > > > > > >> >> >>> > >
>> > > > > > >> > > > > > > > > > > > >> >> >>> > > Sharing my view...
>> subject
>> > to
>> > > > > > >> discussion:
>> > > > > > >> > > > > > > > > > > > >> >> >>> > >
>> > > > > > >> > > > > > > > > > > > >> >> >>> > > 1. Adding non-IRC REST
>> API
>> > to
>> > > > > > Polaris
>> > > > > > >> is
>> > > > > > >> > > > > > perfectly
>> > > > > > >> > > > > > > > > fine.
>> > > > > > >> > > > > > > > > > > > >> >> >>> > >
>> > > > > > >> > > > > > > > > > > > >> >> >>> > > Figuring out specific
>> > endpoint
>> > > > > URIs
>> > > > > > >> and
>> > > > > > >> > > > > payloads
>> > > > > > >> > > > > > > > might
>> > > > > > >> > > > > > > > > > > > require
>> > > > > > >> > > > > > > > > > > > >> a
>> > > > > > >> > > > > > > > > > > > >> >> few
>> > > > > > >> > > > > > > > > > > > >> >> >>> > > roundtrips, so opening a
>> > > separate
>> > > > > > >> thread
>> > > > > > >> > > for
>> > > > > > >> > > > > that
>> > > > > > >> > > > > > > > might
>> > > > > > >> > > > > > > > > > be
>> > > > > > >> > > > > > > > > > > > >> best.
>> > > > > > >> > > > > > > > > > > > >> >> >>> > > Contributors commonly
>> create
>> > > > > Google
>> > > > > > >> Docs
>> > > > > > >> > > for
>> > > > > > >> > > > > new
>> > > > > > >> > > > > > > API
>> > > > > > >> > > > > > > > > > > > proposals
>> > > > > > >> > > > > > > > > > > > >> too
>> > > > > > >> > > > > > > > > > > > >> >> >>> (they
>> > > > > > >> > > > > > > > > > > > >> >> >>> > > fairly easy to update as
>> the
>> > > email
>> > > > > > >> > > discussion
>> > > > > > >> > > > > > > > > > progresses).
>> > > > > > >> > > > > > > > > > > > >> >> >>> > >
>> > > > > > >> > > > > > > > > > > > >> >> >>> > > There was a suggestion to
>> > try
>> > > > > > Markdown
>> > > > > > >> > > (with
>> > > > > > >> > > > > PRs)
>> > > > > > >> > > > > > > for
>> > > > > > >> > > > > > > > > > > > proposals
>> > > > > > >> > > > > > > > > > > > >> >> [1]
>> > > > > > >> > > > > > > > > > > > >> >> >>> ...
>> > > > > > >> > > > > > > > > > > > >> >> >>> > > feel free to give it a
>> try
>> > if
>> > > you
>> > > > > > are
>> > > > > > >> > > > > comfortable
>> > > > > > >> > > > > > > > with
>> > > > > > >> > > > > > > > > > > that.
>> > > > > > >> > > > > > > > > > > > >> >> >>> > >
>> > > > > > >> > > > > > > > > > > > >> >> >>> > > 2. Could you clarify
>> whether
>> > > you
>> > > > > > mean
>> > > > > > >> end
>> > > > > > >> > > > user
>> > > > > > >> > > > > > > > > utilities
>> > > > > > >> > > > > > > > > > or
>> > > > > > >> > > > > > > > > > > > >> admin
>> > > > > > >> > > > > > > > > > > > >> >> >>> user
>> > > > > > >> > > > > > > > > > > > >> >> >>> > > utilities? In the latter
>> > case
>> > > > > those
>> > > > > > >> might
>> > > > > > >> > > be
>> > > > > > >> > > > > more
>> > > > > > >> > > > > > > > > > suitable
>> > > > > > >> > > > > > > > > > > > for
>> > > > > > >> > > > > > > > > > > > >> the
>> > > > > > >> > > > > > > > > > > > >> >> >>> Admin
>> > > > > > >> > > > > > > > > > > > >> >> >>> > > CLI (java) not the Python
>> > CLI,
>> > > > > IMHO.
>> > > > > > >> > > > > > > > > > > > >> >> >>> > >
>> > > > > > >> > > > > > > > > > > > >> >> >>> > > Why would these
>> utilities be
>> > > > > common
>> > > > > > >> with
>> > > > > > >> > > > > events?
>> > > > > > >> > > > > > > > IMHO,
>> > > > > > >> > > > > > > > > > > event
>> > > > > > >> > > > > > > > > > > > >> use
>> > > > > > >> > > > > > > > > > > > >> >> >>> cases
>> > > > > > >> > > > > > > > > > > > >> >> >>> > are
>> > > > > > >> > > > > > > > > > > > >> >> >>> > > distinct from scan/commit
>> > > metrics.
>> > > > > > >> > > > > > > > > > > > >> >> >>> > >
>> > > > > > >> > > > > > > > > > > > >> >> >>> > > 3. I'd prefer separating
>> > > metrics
>> > > > > > >> > > persistence
>> > > > > > >> > > > > from
>> > > > > > >> > > > > > > > > > MetaStore
>> > > > > > >> > > > > > > > > > > > >> >> >>> persistence
>> > > > > > >> > > > > > > > > > > > >> >> >>> > at
>> > > > > > >> > > > > > > > > > > > >> >> >>> > > the code level, so that
>> they
>> > > could
>> > > > > > be
>> > > > > > >> > mixed
>> > > > > > >> > > > and
>> > > > > > >> > > > > > > > matched
>> > > > > > >> > > > > > > > > > > > >> >> >>> independently.
>> > > > > > >> > > > > > > > > > > > >> >> >>> > The
>> > > > > > >> > > > > > > > > > > > >> >> >>> > > separate datasource
>> question
>> > > will
>> > > > > > >> become
>> > > > > > >> > a
>> > > > > > >> > > > > > > non-issue
>> > > > > > >> > > > > > > > > with
>> > > > > > >> > > > > > > > > > > > that
>> > > > > > >> > > > > > > > > > > > >> >> >>> approach,
>> > > > > > >> > > > > > > > > > > > >> >> >>> > I
>> > > > > > >> > > > > > > > > > > > >> >> >>> > > guess.
>> > > > > > >> > > > > > > > > > > > >> >> >>> > >
>> > > > > > >> > > > > > > > > > > > >> >> >>> > > The rationale for
>> separating
>> > > scan
>> > > > > > >> metrics
>> > > > > > >> > > and
>> > > > > > >> > > > > > > > metastore
>> > > > > > >> > > > > > > > > > > > >> >> persistence
>> > > > > > >> > > > > > > > > > > > >> >> >>> is
>> > > > > > >> > > > > > > > > > > > >> >> >>> > that
>> > > > > > >> > > > > > > > > > > > >> >> >>> > > "cascading deletes"
>> between
>> > > them
>> > > > > are
>> > > > > > >> > hardly
>> > > > > > >> > > > > ever
>> > > > > > >> > > > > > > > > > required.
>> > > > > > >> > > > > > > > > > > > >> >> >>> Furthermore,
>> > > > > > >> > > > > > > > > > > > >> >> >>> > the
>> > > > > > >> > > > > > > > > > > > >> >> >>> > > data and query patterns
>> are
>> > > very
>> > > > > > >> > different
>> > > > > > >> > > so
>> > > > > > >> > > > > > > > different
>> > > > > > >> > > > > > > > > > > > >> >> technologies
>> > > > > > >> > > > > > > > > > > > >> >> >>> > might
>> > > > > > >> > > > > > > > > > > > >> >> >>> > > be beneficial in each
>> case.
>> > > > > > >> > > > > > > > > > > > >> >> >>> > >
>> > > > > > >> > > > > > > > > > > > >> >> >>> > > [1]
>> > > > > > >> > > > > > > > > > > > >> >> >>> >
>> > > > > > >> > > > > > > > > > > > >> >> >>>
>> > > > > > >> > > > > > > > > > > > >> >>
>> > > > > > >> > > > > > > > > > > > >>
>> > > > > > >> > > > > > > > > > > >
>> > > > > > >> > > > > > > > > > >
>> > > > > > >> > > > > > > > > >
>> > > > > > >> > > > > > > > >
>> > > > > > >> > > > > > > >
>> > > > > > >> > > > > > >
>> > > > > > >> > > > > >
>> > > > > > >> > > > >
>> > > > > > >> > > >
>> > > > > > >> > >
>> > > > > > >> >
>> > > > > > >>
>> > > > > >
>> > > > >
>> > >
>> >
>> https://urldefense.com/v3/__https://lists.apache.org/thread/yto2wp982t43h1mqjwnslswhws5z47cy__;!!Iz9xO38YGHZK!5EuyFFkk3vhRWVIRvQAWBSQfpJkTMA9HxugzDwXmN0LPPqhEFxYkFRGVhtb8AqUwXtDh2OplcMnbMDHKxYDakNU$
>> > >> > > >> > > > > > > > > > > > >> >> >>> > >
>> > > > > > >> > > > > > > > > > > > >> >> >>> > > Cheers,
>> > > > > > >> > > > > > > > > > > > >> >> >>> > > Dmitri.
>> > > > > > >> > > > > > > > > > > > >> >> >>> > >
>> > > > > > >> > > > > > > > > > > > >> >> >>> > > On Fri, Feb 27, 2026 at
>> > > 6:19 PM
>> > > > > > Anand
>> > > > > > >> > Kumar
>> > > > > > >> > > > > > > Sankaran
>> > > > > > >> > > > > > > > > via
>> > > > > > >> > > > > > > > > > > dev
>> > > > > > >> > > > > > > > > > > > <
>> > > > > > >> > > > > > > > > > > > >> >> >>> > > dev@polaris.apache.org>
>> > > wrote:
>> > > > > > >> > > > > > > > > > > > >> >> >>> > >
>> > > > > > >> > > > > > > > > > > > >> >> >>> > > > Thanks all. This PR is
>> > > merged
>> > > > > now.
>> > > > > > >> > > > > > > > > > > > >> >> >>> > > >
>> > > > > > >> > > > > > > > > > > > >> >> >>> > > > Here are the follow-up
>> > > features
>> > > > > /
>> > > > > > >> work
>> > > > > > >> > > > > needed.
>> > > > > > >> > > > > > > > These
>> > > > > > >> > > > > > > > > > > were
>> > > > > > >> > > > > > > > > > > > >> all
>> > > > > > >> > > > > > > > > > > > >> >> >>> part of
>> > > > > > >> > > > > > > > > > > > >> >> >>> > > the
>> > > > > > >> > > > > > > > > > > > >> >> >>> > > > merged PR at some
>> point in
>> > > time
>> > > > > > and
>> > > > > > >> > were
>> > > > > > >> > > > > > removed
>> > > > > > >> > > > > > > to
>> > > > > > >> > > > > > > > > > > reduce
>> > > > > > >> > > > > > > > > > > > >> >> scope.
>> > > > > > >> > > > > > > > > > > > >> >> >>> > > >
>> > > > > > >> > > > > > > > > > > > >> >> >>> > > > Please let me know what
>> > you
>> > > > > think.
>> > > > > > >> > > > > > > > > > > > >> >> >>> > > >
>> > > > > > >> > > > > > > > > > > > >> >> >>> > > >
>> > > > > > >> > > > > > > > > > > > >> >> >>> > > >   1.  A REST API to
>> > paginate
>> > > > > > through
>> > > > > > >> > > table
>> > > > > > >> > > > > > > metrics.
>> > > > > > >> > > > > > > > > > This
>> > > > > > >> > > > > > > > > > > > >> will be
>> > > > > > >> > > > > > > > > > > > >> >> >>> > non-IRC
>> > > > > > >> > > > > > > > > > > > >> >> >>> > > > standard addition.
>> > > > > > >> > > > > > > > > > > > >> >> >>> > > >   2.  Utilities for
>> > > managing old
>> > > > > > >> > records,
>> > > > > > >> > > > > > should
>> > > > > > >> > > > > > > be
>> > > > > > >> > > > > > > > > > > common
>> > > > > > >> > > > > > > > > > > > >> with
>> > > > > > >> > > > > > > > > > > > >> >> >>> events.
>> > > > > > >> > > > > > > > > > > > >> >> >>> > > > There was some
>> discussion
>> > > that
>> > > > > it
>> > > > > > >> > belongs
>> > > > > > >> > > > to
>> > > > > > >> > > > > > the
>> > > > > > >> > > > > > > > CLI.
>> > > > > > >> > > > > > > > > > > > >> >> >>> > > >   3.  Separate
>> datasource
>> > > > > > (metrics,
>> > > > > > >> > > events,
>> > > > > > >> > > > > > even
>> > > > > > >> > > > > > > > > other
>> > > > > > >> > > > > > > > > > > > >> tables?).
>> > > > > > >> > > > > > > > > > > > >> >> >>> > > >
>> > > > > > >> > > > > > > > > > > > >> >> >>> > > >
>> > > > > > >> > > > > > > > > > > > >> >> >>> > > > Anything else?
>> > > > > > >> > > > > > > > > > > > >> >> >>> > > >
>> > > > > > >> > > > > > > > > > > > >> >> >>> > > > -
>> > > > > > >> > > > > > > > > > > > >> >> >>> > > > Anand
>> > > > > > >> > > > > > > > > > > > >> >> >>> > > >
>> > > > > > >> > > > > > > > > > > > >> >> >>> > > >
>> > > > > > >> > > > > > > > > > > > >> >> >>> > >
>> > > > > > >> > > > > > > > > > > > >> >> >>> >
>> > > > > > >> > > > > > > > > > > > >> >> >>> >
>> > > > > > >> > > > > > > > > > > > >> >> >>>
>> > > > > > >> > > > > > > > > > > > >> >> >>
>> > > > > > >> > > > > > > > > > > > >> >>
>> > > > > > >> > > > > > > > > > > > >> >
>> > > > > > >> > > > > > > > > > > > >>
>> > > > > > >> > > > > > > > > > > > >
>> > > > > > >> > > > > > > > > > > >
>> > > > > > >> > > > > > > > > > >
>> > > > > > >> > > > > > > > > >
>> > > > > > >> > > > > > > > >
>> > > > > > >> > > > > > > >
>> > > > > > >> > > > > > >
>> > > > > > >> > > > > >
>> > > > > > >> > > > >
>> > > > > > >> > > >
>> > > > > > >> > >
>> > > > > > >> >
>> > > > > > >>
>> > > > > > >
>> > > > > >
>> > > > >
>> > >
>> >
>> >
>>
>

Re: Proposal for REST endpoints for table metrics and events

[Dmitri Bourlatchkov <di...@apache.org>]

Hi EJ,

Thanks for the meeting recap!

All:

Worth noting: PR [4115] also contains Management API changes that were not
anticipated initially and did not get an API change review until now. These
are small changes to add new grant types for metrics RBAC..

I've reviewed and approved PR [4115] in GH.

[4115] https://github.com/apache/polaris/pull/4115

Cheers,
Dmitri.

On Wed, Aug 12, 2026 at 5:21 PM EJ Wang <ej...@gmail.com>
wrote:

> Hi folks,
>
> Notes from today's metrics architecture sync are in the usual doc:
>
> https://docs.google.com/document/d/100h7c4damrUzVuquYbBHM0EvA4LSWuW2IT2dN_7nYVA/edit?pli=1&tab=t.18iv1m7p6i6b
>
> Highlights:
>
> Module placement. core/ now holds the entity data model only. A new spi/
> takes every other shared contract: feature SPIs, feature non-SPI contracts,
> substrate SPIs, substrate non-SPI contracts, and contract-required types.
> That settles the question PR #5204 reopened, by placing into a new SPI
> module. Note that it changes slightly towards the referenced WIP RFC [1]
> recommendations: what proposed to place in core/, now should go to spi/
> except for the entity data model.
>
> Metrics PRs. #5068 merged. #5204 needs reshaping onto the split. #4115 and
> #4756 stay open with action on the author to follow up on review comments.
>
> [1]
>
> https://docs.google.com/document/d/1mj3vGpix9zTDTFFJKtWT5ZxI3PTGI1YNb5xFqyWrOkc/edit?tab=t.0
>
> On Wed, Jul 15, 2026 at 11:01 AM Anand Kumar Sankaran via dev <
> dev@polaris.apache.org> wrote:
>
> > Thanks EJ.  My plan is to introduce PR0 in a new branch / PR and
> repurpose
> > #4115 and #4756 for PR1 and PR2 to maintain all the valuable discussions
> > that happened there.
> >
> > From: EJ Wang <ej...@gmail.com>
> > Date: Wednesday, July 15, 2026 at 10:41 AM
> > To: dev@polaris.apache.org <de...@polaris.apache.org>
> > Subject: Re: Proposal for REST endpoints for table metrics and events
> >
> > Hi folks, Thanks again for the discussion today. I updated the sync doc
> > with notes from today's metrics sync: https: //urldefense.
> com/v3/__https:
> > //docs. google.
> >
> com/document/d/100h7c4damrUzVuquYbBHM0EvA4LSWuW2IT2dN_7nYVA/edit?pli=1&tab=t.
> >
> 2ahd7wze5f__;!!Iz9xO38YGHZK!4rnlVIqBQQQ6xpGe1PMV58edOS_cGrKoqdfhw4qSp_JQCavvEjaxXk0OwGPnxyEcO60AzKQK_XDtp7H3DdkFXcrXJwAz1g0$
> >
> >
> > Hi folks,
> >
> > Thanks again for the discussion today. I updated the sync doc with notes
> > from today's metrics sync:
> >
> >
> https://urldefense.com/v3/__https://docs.google.com/document/d/100h7c4damrUzVuquYbBHM0EvA4LSWuW2IT2dN_7nYVA/edit?pli=1&tab=t.2ahd7wze5f__;!!Iz9xO38YGHZK!4rnlVIqBQQQ6xpGe1PMV58edOS_cGrKoqdfhw4qSp_JQCavvEjaxXk0OwGPnxyEcO60AzKQK_XDtp7H3DdkFXcrXJwAz1g0$
> >
> > See highlights below:
> >
> > In previous meetings, EJ proposed a two prs framing to push forward the
> > metrics query work in polaris. Dmitri pointed out a subtle point that pr2
> > is currently also trying to do the persistence related code placement
> > refactor. For the sake of straightforward review process, EJ, Dmitri and
> > Anand (the author) agreed to separate the persistence refactoring into
> pr0,
> > and reframing the pr scopes as follow:
> >
> >    1.
> >
> >    Pr0: refactor irc metrics spi
> >    1.
> >
> >       SPI should stay in core/ with metric type enveloped
> >       2.
> >
> >       default spi impl, no-op, in extensions/ - default selection at
> >       runtime/
> >       3.
> >
> >       move metrics persistence and correspondings to extensions/ as an
> >       alternative SPI impl
> >       2.
> >
> >    Pr1: rest api for query
> >
> >
> https://urldefense.com/v3/__https://github.com/apache/polaris/pull/4115/changes__;!!Iz9xO38YGHZK!4rnlVIqBQQQ6xpGe1PMV58edOS_cGrKoqdfhw4qSp_JQCavvEjaxXk0OwGPnxyEcO60AzKQK_XDtp7H3DdkFXcrXFCbEbRk$
> >    1.
> >
> >       rest api spec in extensions/ as optional spec
> >       2.
> >
> >       SPI in extensions/ as optional spec spi
> >       3.
> >
> >       default spi impl in extensions/ no-op
> >       4.
> >
> >       handler related in runtime/ - thin adaptor from http shape to spi
> >       shape
> >       3.
> >
> >    Pr2: jdbc impl for query
> >
> https://urldefense.com/v3/__https://github.com/apache/polaris/pull/4756__;!!Iz9xO38YGHZK!4rnlVIqBQQQ6xpGe1PMV58edOS_cGrKoqdfhw4qSp_JQCavvEjaxXk0OwGPnxyEcO60AzKQK_XDtp7H3DdkFXcrXRVMBzqE$
> >    1.
> >
> >       alternative spi impl in extensions/ -> query with jdbc
> >
> >
> > Thanks,
> > -ej
> >
> > On Wed, Jul 1, 2026 at 8:01 AM Alexandre Dutra <ad...@apache.org>
> wrote:
> >
> > > Hi all,
> > >
> > > > While the current implementation stores metrics in the metastore,
> other
> > > deployments
> > > may choose a KV store, a dedicated metrics warehouse, or another
> backend
> > > entirely. In those cases, exposing a Polaris REST API over a specific
> > > schema feels like coupling the API to an implementation detail.
> > >
> > > I wouldn't say to an implementation detail, but rather to a particular
> > > setup / configuration. I think that's fine. If you don't store
> > > metrics, you don't read metrics. Similar situations will certainly
> > > happen in the future, e.g. if we ever introduce an API to read events,
> > > that would only work if events are being persisted in the first place.
> > > Again I think that's a fair trade-off.
> > >
> > > Thanks,
> > > Alex
> > >
> > > On Wed, Jul 1, 2026 at 7:23 AM Dmitri Bourlatchkov <di...@apache.org>
> > > wrote:
> > > >
> > > > Hi Yufei,
> > > >
> > > > From my POV, it is fine for Polaris to have multiple REST API
> surfaces
> > > with
> > > > different features and different degrees of "readiness". Each API can
> > > > evolve fairly independently. With proper modularization (which Anand
> > > did),
> > > > changes to the main Polaris code and these extra modules should
> > normally
> > > be
> > > > non-intrusive to each other.
> > > >
> > > > I assume Anand is going to be the first user of this API. However, if
> > we
> > > do
> > > > not merge it we'll never know whether other users are interested. I
> do
> > > not
> > > > think many end users read the dev ML, but they certainly read the
> docs,
> > > > code and release notes.
> > > >
> > > > There is no risk for Polaris as a project in exposing this
> experimental
> > > > API. If usage dwindles over time, we can remove it. If it gains a
> > > > considerable user / contributor base, we will promote it to a
> > > "first-class"
> > > > (non-beta) API.
> > > >
> > > > Cheers,
> > > > Dmitri.
> > > >
> > > > On Tue, Jun 30, 2026 at 7:53 PM Yufei Gu <fl...@gmail.com>
> wrote:
> > > >
> > > > > Hi all,
> > > > >
> > > > > Thanks for driving this work forward. Building this as a POC makes
> > > perfect
> > > > > sense if the goal is to validate the idea and gather feedback.
> Should
> > > we
> > > > > merge a POC into the main branch before agreeing on the overall
> > > direction?
> > > > > To me, that feels like a separate discussion from whether the
> > > > > implementation itself looks reasonable.
> > > > >
> > > > > One concern that I don't think has been fully addressed is whether
> we
> > > need
> > > > > to expose metrics consumption through a Polaris REST endpoint at
> all.
> > > > >
> > > > > My concern is less about the API itself and more about the boundary
> > > Polaris
> > > > > should own. The proposed REST API is tied to a particular
> persistence
> > > > > schema, but that schema is just one possible implementation. While
> > the
> > > > > current implementation stores metrics in the metastore, other
> > > deployments
> > > > > may choose a KV store, a dedicated metrics warehouse, or another
> > > backend
> > > > > entirely. In those cases, exposing a Polaris REST API over a
> specific
> > > > > schema feels like coupling the API to an implementation detail.
> > > > >
> > > > > Polaris isn't intended to become a full metrics system. I'd expect
> it
> > > to
> > > > > focus on producing metrics and let deployments choose the most
> > > appropriate
> > > > > storage and consumption model for their environment.
> > > > >
> > > > > I'm not opposed to exploring this direction, but we should first
> > align
> > > on
> > > > > whether exposing a REST API through Polaris is the right long term
> > > approach
> > > > > before merging it into the main branch.
> > > > >
> > > > > Thanks,
> > > > > Yufei
> > > > >
> > > > > Yufei
> > > > >
> > > > >
> > > > > On Mon, Jun 29, 2026 at 9:27 AM Dmitri Bourlatchkov <
> > dimas@apache.org>
> > > > > wrote:
> > > > >
> > > > > > Hi All,
> > > > > >
> > > > > > It looks like Anand addressed all action items in [4115].
> > > > > >
> > > > > > Please review the latest state of the code. The PR LGTM and I
> > > approved in
> > > > > > GH from my side.
> > > > > >
> > > > > > [4115]
> >
> https://urldefense.com/v3/__https://github.com/apache/polaris/pull/4115__;!!Iz9xO38YGHZK!4rnlVIqBQQQ6xpGe1PMV58edOS_cGrKoqdfhw4qSp_JQCavvEjaxXk0OwGPnxyEcO60AzKQK_XDtp7H3DdkFXcrXtFRscKs$
> > > > > >
> > > > > > Thanks,
> > > > > > Dmitri.
> > > > > >
> > > > > > On Thu, Jun 25, 2026 at 1:56 PM Dmitri Bourlatchkov <
> > > dimas@apache.org>
> > > > > > wrote:
> > > > > >
> > > > > > > Hi All,
> > > > > > >
> > > > > > > Recap of the discussion in today's Community Sync:
> > > > > > >
> > > > > > > * The modular approach for Metrics SPI(s) and implementation
> > seems
> > > to
> > > > > > have
> > > > > > > community consensus (please comment if you still have concerns)
> > > > > > >
> > > > > > > * What modules / feature flags are enabled in Apache Polaris
> > > images by
> > > > > > > default will be discussed later (when the feature is close to
> > > > > > completion).
> > > > > > >
> > > > > > > In any case, users will be able to turn the full feature on/off
> > at
> > > > > their
> > > > > > > discretion, regardless of the default flag values.
> > > > > > >
> > > > > > > * It is generally preferable to have smaller PRs and introduce
> > new
> > > > > > > features gradually to simplify reviews.
> > > > > > >
> > > > > > > What this means for the Metrics API, IMHO is:
> > > > > > >
> > > > > > > a) The scope of PR 4115 is probably ok as it stands right now.
> If
> > > > > someone
> > > > > > > wants to propose splitting it further, please comment.
> > > > > > >
> > > > > > > b) After 4115 is merged, Metrics Persistence PRs will follow,
> > with
> > > > > > smaller
> > > > > > > scope per PR as appropriate for the nature of the code changes.
> > > > > > >
> > > > > > > * The major concern was about exposing the new Metrics REST API
> > to
> > > end
> > > > > > > users. The nature of this concern was about avoiding the
> (false)
> > > > > > impression
> > > > > > > that the new API is a firm and adopted Polaris specification.
> > > > > > >
> > > > > > > I think this concern is reasonable.
> > > > > > >
> > > > > > > I propose that Anand update the documentation sections of PR
> 4115
> > > to
> > > > > > > emphasise that the API is currently an experimental / POC API
> at
> > > this
> > > > > > time
> > > > > > > and is subject to change (including breaking changes). I
> believe
> > it
> > > > > > already
> > > > > > > has the "beta" label. So this request is mostly for doc changes
> > to
> > > > > > clarify
> > > > > > > user expectations, since the "beta" label alone may not be
> > > interpreted
> > > > > > the
> > > > > > > same way by all users.
> > > > > > >
> > > > > > > I hope this update will resolve all remaining concerns with
> 4115
> > > and
> > > > > > allow
> > > > > > > for it to be merged so that the community will be able to
> proceed
> > > to
> > > > > > > follow-up PRs (as noted above).
> > > > > > >
> > > > > > > Please add/correct me if I missed anything relevant.
> > > > > > >
> > > > > > > Thanks,
> > > > > > > Dmitri.
> > > > > > >
> > > > > > > On Wed, Jun 24, 2026 at 8:11 PM Yufei Gu <flyrain000@gmail.com
> >
> > > wrote:
> > > > > > >
> > > > > > >> I do not think we have agreement on the query REST APIs yet.
> > > > > > >>
> > > > > > >> In particular, the current query API shape seems highly
> coupled
> > > to the
> > > > > > >> *example[1]* metrics in the Iceberg REST spec. I do not think
> > > that is
> > > > > a
> > > > > > >> good way to design a REST API contract. For example, fields
> like
> > > > > > >> resultDataFiles, resultDeleteFiles, totalFileSizeBytes,
> > > > > > >> totalDataManifests,
> > > > > > >> totalDeleteManifests, and scannedDataManifests are very
> specific
> > > to
> > > > > one
> > > > > > >> style of scan metric. That makes the API look more like a
> direct
> > > > > > >> projection
> > > > > > >> of the current example payload than a general query model.
> > > > > > >>
> > > > > > >> My preference is that Polaris should not define or strongly
> > opine
> > > on
> > > > > the
> > > > > > >> Iceberg scan or commit metrics query REST API at this point.
> > > Polaris
> > > > > can
> > > > > > >> resolve and authorize the table, accept the metrics report,
> and
> > > > > delegate
> > > > > > >> it
> > > > > > >> to the configured reporting path. Durable storage, filtering,
> > > > > > dashboards,
> > > > > > >> and query APIs should remain downstream impl. choices unless
> and
> > > until
> > > > > > the
> > > > > > >> Iceberg community standardizes a query contract, which I doubt
> > > will
> > > > > > >> happen.
> > > > > > >>
> > > > > > >> So I do not think the intake path has to be blocked, but I
> would
> > > > > prefer
> > > > > > >> not
> > > > > > >> to merge a Polaris specific query API that may be hard to
> evolve
> > > > > later.
> > > > > > >>
> > > > > > >> 1.
> > > > > > >>
> > > > > > >>
> > > > > >
> > > > >
> > >
> >
> https://urldefense.com/v3/__https://github.com/apache/iceberg/blob/2a6c556c0c883c04c6d3fbd68e6aeda36b91e0aa/open-api/rest-catalog-open-api.yaml*L4152__;Iw!!Iz9xO38YGHZK!4rnlVIqBQQQ6xpGe1PMV58edOS_cGrKoqdfhw4qSp_JQCavvEjaxXk0OwGPnxyEcO60AzKQK_XDtp7H3DdkFXcrXCDsqYGo$
> > > > > > >>
> > > > > > >> Yufei
> > > > > > >>
> > > > > > >>
> > > > > > >> On Tue, Jun 23, 2026 at 9:30 AM Dmitri Bourlatchkov <
> > > dimas@apache.org
> > > > > >
> > > > > > >> wrote:
> > > > > > >>
> > > > > > >> > Hi All,
> > > > > > >> >
> > > > > > >> > So, what are the blockers for merging [4115], if any?
> > > > > > >> >
> > > > > > >> > [4115]
> >
> https://urldefense.com/v3/__https://github.com/apache/polaris/pull/4115__;!!Iz9xO38YGHZK!4rnlVIqBQQQ6xpGe1PMV58edOS_cGrKoqdfhw4qSp_JQCavvEjaxXk0OwGPnxyEcO60AzKQK_XDtp7H3DdkFXcrXtFRscKs$
> > > > > > >> >
> > > > > > >> > Thanks,
> > > > > > >> > Dmitri.
> > > > > > >> >
> > > > > > >> > On Wed, Jun 17, 2026 at 5:28 PM EJ Wang <
> > > > > > ej.wang.dev.interest@gmail.com
> > > > > > >> >
> > > > > > >> > wrote:
> > > > > > >> >
> > > > > > >> > > Hi folks,
> > > > > > >> > >
> > > > > > >> > > Thanks again for the discussion today. I updated the sync
> > doc
> > > with
> > > > > > >> notes
> > > > > > >> > > from today's metrics sync:
> > > > > > >> > >
> > > > > > >> > >
> > > > > > >> > >
> > > > > > >> >
> > > > > > >>
> > > > > >
> > > > >
> > >
> >
> https://urldefense.com/v3/__https://docs.google.com/document/d/100h7c4damrUzVuquYbBHM0EvA4LSWuW2IT2dN_7nYVA/edit?tab=t.ezk0rgdx0c6m__;!!Iz9xO38YGHZK!4rnlVIqBQQQ6xpGe1PMV58edOS_cGrKoqdfhw4qSp_JQCavvEjaxXk0OwGPnxyEcO60AzKQK_XDtp7H3DdkFXcrX83IZkm0$
> > > > > > >> > >
> > > > > > >> > > A few highlights:
> > > > > > >> > >
> > > > > > >> > >    - No objection to keeping the default metrics behavior
> > > small:
> > > > > > >> > >    no-op/log-only is enough as the built-in default.
> > > > > > >> > >    - Durable storage, event forwarding, external queues,
> > > > > dashboards,
> > > > > > >> and
> > > > > > >> > >    custom filtering should be implementation choices
> behind
> > > the
> > > > > > >> metrics
> > > > > > >> > >    reporting path.
> > > > > > >> > >    - For PR #4115, the REST layer should
> > resolve/authz/accept
> > > the
> > > > > > >> report,
> > > > > > >> > >    then delegate to the selected metrics implementation.
> It
> > > should
> > > > > > not
> > > > > > >> > own
> > > > > > >> > >    durable storage semantics.
> > > > > > >> > >    - The rough boundary we discussed is: spec owns the
> wire
> > > > > > contract,
> > > > > > >> > >    runtime owns framework wiring, API/contract modules own
> > > > > > >> > provider-facing
> > > > > > >> > >    contracts, and extensions own replaceable
> > implementations.
> > > > > > >> > >    - OpenLineage has a similar path-naming question. We
> > should
> > > > > avoid
> > > > > > >> > >    occupying a generic lineage path too early if we want
> > room
> > > for
> > > > > > >> other
> > > > > > >> > >    lineage systems later.
> > > > > > >> > >
> > > > > > >> > > I believe the boundary rules previewed in sync meeting are
> > > useful
> > > > > > for
> > > > > > >> > > reviewing PR #4115, but it is not
> > > > > > >> > > a final module-layout proposal.
> > > > > > >> > >
> > > > > > >> > > For metrics specifically, I think the target remains:
> > > > > > >> > >    runtime
> > > > > > >> > >      -> metrics reporting/emitting contract
> > > > > > >> > >      -> default implementation
> > > > > > >> > >           -> lower implementation details, if needed
> > > > > > >> > >
> > > > > > >> > > That keeps the default simple, while allowing durable
> JDBC,
> > > > > > >> event-backed,
> > > > > > >> > > or external-queue implementations to be added separately.
> > > > > > >> > >
> > > > > > >> > > We also touched on the split between Iceberg metrics
> > emitting
> > > and
> > > > > > >> > > Polaris-owned metrics querying. The querying side can keep
> > > > > evolving,
> > > > > > >> > > especially for dashboard or generic-table use cases, but I
> > do
> > > not
> > > > > > >> think
> > > > > > >> > > that needs to block the intake path in this PR.
> > > > > > >> > >
> > > > > > >> > > Thanks,
> > > > > > >> > > -ej
> > > > > > >> > >
> > > > > > >> > > On Tue, Jun 16, 2026 at 2:17 PM Yufei Gu <
> > > flyrain000@gmail.com>
> > > > > > >> wrote:
> > > > > > >> > >
> > > > > > >> > > > Thanks for chiming in, EJ. Agreed that the default
> battery
> > > > > should
> > > > > > >> stay
> > > > > > >> > > > small. I'm leaning toward not including metrics
> > persistence
> > > in
> > > > > the
> > > > > > >> > > default
> > > > > > >> > > > battery.
> > > > > > >> > > >
> > > > > > >> > > > Yufei
> > > > > > >> > > >
> > > > > > >> > > >
> > > > > > >> > > > On Thu, Jun 11, 2026 at 8:30 PM EJ Wang <
> > > > > > >> > ej.wang.dev.interest@gmail.com>
> > > > > > >> > > > wrote:
> > > > > > >> > > >
> > > > > > >> > > > > Hi Yufei,
> > > > > > >> > > > >
> > > > > > >> > > > > Thanks for connecting this back to the REST endpoint
> > > > > proposal. I
> > > > > > >> > > replied
> > > > > > >> > > > > with the fuller version on the event-forwarding
> thread,
> > > but
> > > > > > >> wanted to
> > > > > > >> > > add
> > > > > > >> > > > > the shorter version here too.
> > > > > > >> > > > >
> > > > > > >> > > > > I agree that metrics reporting/emitting is the right
> > > > > conceptual
> > > > > > >> > > boundary,
> > > > > > >> > > > > and I also think the event/listener path is a good
> > > > > > implementation
> > > > > > >> > > > direction
> > > > > > >> > > > > to explore. The distinction I want to keep clear is
> > > default
> > > > > > >> battery
> > > > > > >> > vs
> > > > > > >> > > > > extension implementation.
> > > > > > >> > > > >
> > > > > > >> > > > > For the REST/API side, I would keep the semantics
> > narrow:
> > > > > > Polaris
> > > > > > >> > > > resolves
> > > > > > >> > > > > and authorizes the table, accepts the Iceberg
> > scan/commit
> > > > > > metrics
> > > > > > >> > > report
> > > > > > >> > > > > into the configured reporting/emitting path, and
> returns
> > > 204.
> > > > > > That
> > > > > > >> > 204
> > > > > > >> > > > > should mean Polaris accepted the report into the
> > ingestion
> > > > > path;
> > > > > > >> it
> > > > > > >> > > > should
> > > > > > >> > > > > not imply durable storage.
> > > > > > >> > > > >
> > > > > > >> > > > > Durable storage, event forwarding, external telemetry
> > > routing,
> > > > > > >> > > filtering,
> > > > > > >> > > > > and retention should sit behind that
> reporting/emitting
> > > > > boundary
> > > > > > >> as
> > > > > > >> > > > > implementation-layer behavior. A durable JDBC path can
> > be
> > > one
> > > > > > >> named
> > > > > > >> > > > > extension implementation. An event/listener forwarding
> > > path
> > > > > can
> > > > > > be
> > > > > > >> > > > another
> > > > > > >> > > > > named extension implementation.
> > > > > > >> > > > >
> > > > > > >> > > > > So I am +1 on the event-forwarding idea as a
> non-default
> > > > > > extension
> > > > > > >> > > > > implementation. I just would not make it the default
> > > battery
> > > > > or
> > > > > > >> the
> > > > > > >> > > core
> > > > > > >> > > > > API shape. The default battery can stay small and
> safe,
> > > while
> > > > > > >> > > deployments
> > > > > > >> > > > > that need forwarding, dashboards, or durable storage
> can
> > > opt
> > > > > > into
> > > > > > >> the
> > > > > > >> > > > > implementation that matches their operational model.
> > > > > > >> > > > >
> > > > > > >> > > > > Thanks,
> > > > > > >> > > > > -ej
> > > > > > >> > > > >
> > > > > > >> > > > > On Wed, Jun 10, 2026 at 11:43 AM Yufei Gu <
> > > > > flyrain000@gmail.com
> > > > > > >
> > > > > > >> > > wrote:
> > > > > > >> > > > >
> > > > > > >> > > > > > Thanks EJ. I agree that the reporting/emitting
> > boundary
> > > > > feels
> > > > > > >> like
> > > > > > >> > > the
> > > > > > >> > > > > > right SPI boundary, but I wonder if we can simplify
> > this
> > > > > even
> > > > > > >> > > further.
> > > > > > >> > > > > >
> > > > > > >> > > > > > Iceberg scan and commit metrics look very similar to
> > > events
> > > > > to
> > > > > > >> me.
> > > > > > >> > > They
> > > > > > >> > > > > are
> > > > > > >> > > > > > append only, asynchronously consumed, often
> forwarded
> > to
> > > > > > >> external
> > > > > > >> > > > > systems,
> > > > > > >> > > > > > and have similar retention and cleanup requirements.
> > > More
> > > > > > >> details
> > > > > > >> > can
> > > > > > >> > > > be
> > > > > > >> > > > > > found in this thread [1]. In that case, Polaris may
> > only
> > > > > need
> > > > > > to
> > > > > > >> > > accept
> > > > > > >> > > > > the
> > > > > > >> > > > > > report and emit an event through the existing event
> > > > > framework.
> > > > > > >> > > > Filtering,
> > > > > > >> > > > > > async delivery, custom sinks, and retention
> mechanisms
> > > could
> > > > > > >> then
> > > > > > >> > be
> > > > > > >> > > > > shared
> > > > > > >> > > > > > instead of introducing a separate metrics specific
> > > extension
> > > > > > >> layer
> > > > > > >> > > and
> > > > > > >> > > > > SPI.
> > > > > > >> > > > > >
> > > > > > >> > > > > > The main requirement I still see is querying metrics
> > > through
> > > > > > >> > Polaris.
> > > > > > >> > > > If
> > > > > > >> > > > > we
> > > > > > >> > > > > > want that battery included experience, we could work
> > on
> > > a
> > > > > > >> > persistence
> > > > > > >> > > > > > listener implementation that selectively saves
> metrics
> > > and
> > > > > > >> expose
> > > > > > >> > it
> > > > > > >> > > > via
> > > > > > >> > > > > > the IRC event endpoint which seems pretty close in
> > > Iceberg
> > > > > > >> > > > > > community. Another option I like more is to allow
> > users
> > > to
> > > > > > >> develop
> > > > > > >> > > > their
> > > > > > >> > > > > > own listener, so they can persist scan/commit
> metrics
> > > to any
> > > > > > >> > systems
> > > > > > >> > > > they
> > > > > > >> > > > > > prefer. In general, that feels like an
> implementation
> > > choice
> > > > > > on
> > > > > > >> top
> > > > > > >> > > of
> > > > > > >> > > > > the
> > > > > > >> > > > > > event framework rather than something that needs to
> > > shape
> > > > > the
> > > > > > >> core
> > > > > > >> > > > > > architecture.
> > > > > > >> > > > > >
> > > > > > >> > > > > > 1.
> > > > > > >> >
> > >
> >
> https://urldefense.com/v3/__https://lists.apache.org/thread/x9j8nscvy8hq61tyn01mj8yp6n9of0kp__;!!Iz9xO38YGHZK!4rnlVIqBQQQ6xpGe1PMV58edOS_cGrKoqdfhw4qSp_JQCavvEjaxXk0OwGPnxyEcO60AzKQK_XDtp7H3DdkFXcrXGcGDgks$
> > > > > > >> > > > > >
> > > > > > >> > > > > > Thanks,
> > > > > > >> > > > > > Yufei
> > > > > > >> > > > > >
> > > > > > >> > > > > >
> > > > > > >> > > > > > On Thu, Jun 4, 2026 at 5:14 PM EJ Wang <
> > > > > > >> > > ej.wang.dev.interest@gmail.com
> > > > > > >> > > > >
> > > > > > >> > > > > > wrote:
> > > > > > >> > > > > >
> > > > > > >> > > > > > > Good point. I agree the existing
> > > PolarisMetricsReporter is
> > > > > > >> > already
> > > > > > >> > > > very
> > > > > > >> > > > > > > close to the right conceptual boundary. I am not
> > > > > proposing a
> > > > > > >> > second
> > > > > > >> > > > > > > parallel reporter concept. The distinction I am
> > > trying to
> > > > > > >> make is
> > > > > > >> > > > > > narrower:
> > > > > > >> > > > > > >
> > > > > > >> > > > > > > Current state:
> > > > > > >> > > > > > >
> > > > > > >> > > > > > >    - PolarisMetricsReporter is the existing
> metrics
> > > > > > reporting
> > > > > > >> > hook.
> > > > > > >> > > > > > >    - But it currently lives under runtime/service.
> > > > > > >> > > > > > >    - Its contract is documented around Quarkus/CDI
> > > > > discovery
> > > > > > >> and
> > > > > > >> > > > > > > @Identifier
> > > > > > >> > > > > > >    selection.
> > > > > > >> > > > > > >    - The default implementation is log-only.
> > > > > > >> > > > > > >    - The durable implementation writes through
> > > > > > >> > > PolarisMetricsManager
> > > > > > >> > > > > and
> > > > > > >> > > > > > >    MetricsPersistence.
> > > > > > >> > > > > > >    - MetricsPersistence is currently inherited by
> > > > > > >> > BasePersistence.
> > > > > > >> > > > > > >
> > > > > > >> > > > > > > My proposal:
> > > > > > >> > > > > > >
> > > > > > >> > > > > > >    - Keep the reporting/emitting boundary as the
> > SPI.
> > > > > > >> > > > > > >    - Revise that contract in a framework-agnostic
> > > optional
> > > > > > >> > Iceberg
> > > > > > >> > > > > > *metrics
> > > > > > >> > > > > > >    extension API layer*.
> > > > > > >> > > > > > >    - Keep runtime/service responsible for REST
> > > ingestion,
> > > > > > >> > > table/authz
> > > > > > >> > > > > > >    validation, and runtime wiring.
> > > > > > >> > > > > > >    - Keep the battery implementation log-only in
> the
> > > > > metrics
> > > > > > >> > > > extension
> > > > > > >> > > > > > API
> > > > > > >> > > > > > >    layer, *not under runtime/service*.
> > > > > > >> > > > > > >    - Treat durable JDBC metrics as one
> > implementation
> > > of
> > > > > the
> > > > > > >> > > > reporting
> > > > > > >> > > > > > SPI.
> > > > > > >> > > > > > >    - Decompose the MetricsPersistence and
> > > BasePersistence
> > > > > > >> > coupling
> > > > > > >> > > as
> > > > > > >> > > > > > >    durable implementation detail.
> > > > > > >> > > > > > >
> > > > > > >> > > > > > > So the difference is not “new reporter concept vs
> > old
> > > > > > reporter
> > > > > > >> > > > > concept.”
> > > > > > >> > > > > > > but:
> > > > > > >> > > > > > >
> > > > > > >> > > > > > >
> > > > > > >> > > > > > >    - PolarisMetricsReporter today =
> runtime/service
> > > > > > CDI-shaped
> > > > > > >> > > hook.
> > > > > > >> > > > > > >    - Target reporting SPI = same conceptual
> > boundary,
> > > but
> > > > > > >> placed
> > > > > > >> > in
> > > > > > >> > > > the
> > > > > > >> > > > > > >    right extension API layer and kept
> > > framework-agnostic.
> > > > > > >> > > > > > >
> > > > > > >> > > > > > > For the current metrics PR, I think the minimal
> > > > > > architectural
> > > > > > >> > > cleanup
> > > > > > >> > > > > > > before merge is to *put the reporting SPI contract
> > > and the
> > > > > > >> > battery
> > > > > > >> > > > > > default
> > > > > > >> > > > > > > in the right place*. The durable JDBC
> > implementation,
> > > > > > >> > event-backed
> > > > > > >> > > > > async
> > > > > > >> > > > > > > implementation, external queue support, and deeper
> > > > > > persistence
> > > > > > >> > > > cleanup
> > > > > > >> > > > > > can
> > > > > > >> > > > > > > continue as follow-ups, as long as we do not lock
> > the
> > > SPI
> > > > > > >> > boundary
> > > > > > >> > > to
> > > > > > >> > > > > > > BasePersistence or to runtime/service wiring.
> > > > > > >> > > > > > >
> > > > > > >> > > > > > > -ej
> > > > > > >> > > > > > >
> > > > > > >> > > > > > > On Wed, Jun 3, 2026 at 3:00 PM Dmitri
> Bourlatchkov <
> > > > > > >> > > dimas@apache.org
> > > > > > >> > > > >
> > > > > > >> > > > > > > wrote:
> > > > > > >> > > > > > >
> > > > > > >> > > > > > > > Hi EJ,
> > > > > > >> > > > > > > >
> > > > > > >> > > > > > > > Thanks for the recap / summary!
> > > > > > >> > > > > > > >
> > > > > > >> > > > > > > > > Do folks agree that the stable SPI boundary
> > > should be
> > > > > > >> > > > > > > > metrics reporting/emitting, not metrics
> > persistence?
> > > > > > >> > > > > > > >
> > > > > > >> > > > > > > > SGTM.
> > > > > > >> > > > > > > >
> > > > > > >> > > > > > > > > Does an optional Iceberg metrics extension API
> > > layer
> > > > > > sound
> > > > > > >> > like
> > > > > > >> > > > the
> > > > > > >> > > > > > > right
> > > > > > >> > > > > > > > home for this SPI?
> > > > > > >> > > > > > > >
> > > > > > >> > > > > > > > How is that different from current
> > > > > PolarisMetricsReporter?
> > > > > > >> > > > > > > >
> > > > > > >> > > > > > > > > Should the current durable metrics work be
> > > reframed
> > > > > as a
> > > > > > >> > > durable
> > > > > > >> > > > > > > > JDBC reference implementation of that SPI?
> > > > > > >> > > > > > > >
> > > > > > >> > > > > > > > SGTM.
> > > > > > >> > > > > > > >
> > > > > > >> > > > > > > > > What is the smallest PR sequence to get there
> > > without
> > > > > > >> > blocking
> > > > > > >> > > > > > > > the current metrics work unnecessarily?
> > > > > > >> > > > > > > >
> > > > > > >> > > > > > > > Let's get
> > >
> >
> https://urldefense.com/v3/__https://github.com/apache/polaris/pull/4397__;!!Iz9xO38YGHZK!4rnlVIqBQQQ6xpGe1PMV58edOS_cGrKoqdfhw4qSp_JQCavvEjaxXk0OwGPnxyEcO60AzKQK_XDtp7H3DdkFXcrXuu5jefU$
> > > > > > >> merged
> > > > > > >> > > > first.
> > > > > > >> > > > > > > >
> > > > > > >> > > > > > > > Then, I'd propose isolating the Metrics schema
> > from
> > > the
> > > > > > >> > MetaStore
> > > > > > >> > > > > > schema.
> > > > > > >> > > > > > > > This will probably have some ripple effect into
> > the
> > > > > > >> bootstrap
> > > > > > >> > > > > > workflows,
> > > > > > >> > > > > > > so
> > > > > > >> > > > > > > > it's not a trivial change.
> > > > > > >> > > > > > > >
> > > > > > >> > > > > > > > Then, let's reassess.
> > > > > > >> > > > > > > >
> > > > > > >> > > > > > > > Cheers,
> > > > > > >> > > > > > > > Dmitri.
> > > > > > >> > > > > > > >
> > > > > > >> > > > > > > > On Wed, Jun 3, 2026 at 5:40 PM EJ Wang <
> > > > > > >> > > > > ej.wang.dev.interest@gmail.com
> > > > > > >> > > > > > >
> > > > > > >> > > > > > > > wrote:
> > > > > > >> > > > > > > >
> > > > > > >> > > > > > > > > Hi folks,
> > > > > > >> > > > > > > > >
> > > > > > >> > > > > > > > > Thanks again for the discussion today. I
> updated
> > > the
> > > > > > sync
> > > > > > >> doc
> > > > > > >> > > > with
> > > > > > >> > > > > > > notes
> > > > > > >> > > > > > > > > from the third metrics architecture sync:
> > > > > > >> > > > > > > > >
> > > > > > >> > > > > > > > >
> > > > > > >> > > > > > > > >
> > > > > > >> > > > > > > >
> > > > > > >> > > > > > >
> > > > > > >> > > > > >
> > > > > > >> > > > >
> > > > > > >> > > >
> > > > > > >> > >
> > > > > > >> >
> > > > > > >>
> > > > > >
> > > > >
> > >
> >
> https://urldefense.com/v3/__https://docs.google.com/document/d/100h7c4damrUzVuquYbBHM0EvA4LSWuW2IT2dN_7nYVA/edit?tab=t.k96s2xyqr5u1__;!!Iz9xO38YGHZK!4rnlVIqBQQQ6xpGe1PMV58edOS_cGrKoqdfhw4qSp_JQCavvEjaxXk0OwGPnxyEcO60AzKQK_XDtp7H3DdkFXcrXrqCL-QE$
> > > > > > >> > > > > > > > >
> > > > > > >> > > > > > > > > A few highlights from today’s discussion:
> > > > > > >> > > > > > > > >
> > > > > > >> > > > > > > > >    - We clarified that Iceberg metrics
> reporting
> > > can
> > > > > be
> > > > > > >> > > > interpreted
> > > > > > >> > > > > > > > either
> > > > > > >> > > > > > > > >    as sync or async handling from Polaris’s
> > > > > perspective,
> > > > > > >> so
> > > > > > >> > > > Polaris
> > > > > > >> > > > > > > > should
> > > > > > >> > > > > > > > >    stay flexible as a platform instead of
> baking
> > > one
> > > > > > >> handling
> > > > > > >> > > > model
> > > > > > >> > > > > > > into
> > > > > > >> > > > > > > > > the
> > > > > > >> > > > > > > > >    REST API behavior.
> > > > > > >> > > > > > > > >    - We aligned that the built-in (battery)
> > > behavior
> > > > > for
> > > > > > >> > > metrics
> > > > > > >> > > > > > > emitting
> > > > > > >> > > > > > > > >    can stay simple: no-op/log-only is enough
> as
> > > the
> > > > > > >> default.
> > > > > > >> > > > > > > > >    - Durable metrics persistence should be
> > > treated as
> > > > > an
> > > > > > >> > > > > > implementation
> > > > > > >> > > > > > > > of
> > > > > > >> > > > > > > > >    the metrics reporting path, not as the core
> > SPI
> > > > > > >> boundary.
> > > > > > >> > > > > > > > >    - The existing durable metrics work can be
> > > reviewed
> > > > > > as
> > > > > > >> a
> > > > > > >> > > > > reference
> > > > > > >> > > > > > > > >    implementation of the reporting SPI, with
> > > > > > >> > > persistence-related
> > > > > > >> > > > > > logic
> > > > > > >> > > > > > > > kept
> > > > > > >> > > > > > > > >    self-contained under a metrics durable
> > > > > implementation
> > > > > > >> > module
> > > > > > >> > > > > > rather
> > > > > > >> > > > > > > > than
> > > > > > >> > > > > > > > >    scattered through core entity persistence.
> > > > > > >> > > > > > > > >    - Dashboard/insights remains a real use
> case,
> > > but
> > > > > we
> > > > > > >> > agreed
> > > > > > >> > > to
> > > > > > >> > > > > > keep
> > > > > > >> > > > > > > it
> > > > > > >> > > > > > > > >    separate from the core metrics intake
> > > discussion
> > > > > for
> > > > > > >> now.
> > > > > > >> > > > > > > > >
> > > > > > >> > > > > > > > > I also did a quick source check after the
> > meeting
> > > to
> > > > > > make
> > > > > > >> > sure
> > > > > > >> > > we
> > > > > > >> > > > > are
> > > > > > >> > > > > > > > > describing the current state accurately.
> > > > > > >> > > > > > > > >
> > > > > > >> > > > > > > > > Current state:
> > > > > > >> > > > > > > > >
> > > > > > >> > > > > > > > >    - Polaris already has a metrics reporting
> > hook:
> > > > > > >> > > > > > > > PolarisMetricsReporter.
> > > > > > >> > > > > > > > >    - The default implementation is
> > > > > > DefaultMetricsReporter,
> > > > > > >> > > > selected
> > > > > > >> > > > > > by
> > > > > > >> > > > > > > > >
> > > polaris.iceberg-metrics.reporting.type=default. It
> > > > > is
> > > > > > >> > > > log-only,
> > > > > > >> > > > > > > > >    effectively quiet unless metrics logging is
> > > > > enabled.
> > > > > > >> > > > > > > > >    - There is also a
> PersistingMetricsReporter,
> > > > > selected
> > > > > > >> by
> > > > > > >> > > > > > > > >
> > > polaris.iceberg-metrics.reporting.type=persisting,
> > > > > > >> which
> > > > > > >> > > > > converts
> > > > > > >> > > > > > > > >    Iceberg scan/commit reports into Polaris
> > > metrics
> > > > > > >> records
> > > > > > >> > and
> > > > > > >> > > > > > writes
> > > > > > >> > > > > > > > > through
> > > > > > >> > > > > > > > >    PolarisMetricsManager ->
> MetricsPersistence.
> > > > > > >> > > > > > > > >    - MetricsPersistence currently exists in
> the
> > > > > > >> persistence
> > > > > > >> > > layer
> > > > > > >> > > > > and
> > > > > > >> > > > > > > is
> > > > > > >> > > > > > > > >    inherited by BasePersistence.
> > > > > > >> > > > > > > > >
> > > > > > >> > > > > > > > > My read from the discussion is that the target
> > > > > boundary
> > > > > > >> > should
> > > > > > >> > > > not
> > > > > > >> > > > > be
> > > > > > >> > > > > > > > > MetricsPersistence as inherited by
> > > BasePersistence.
> > > > > That
> > > > > > >> path
> > > > > > >> > > > > should
> > > > > > >> > > > > > be
> > > > > > >> > > > > > > > > decomposed as durable implementation detail
> > (taken
> > > > > care
> > > > > > >> of by
> > > > > > >> > > > > > > > >
> >
> https://urldefense.com/v3/__https://github.com/apache/polaris/pull/4397__;!!Iz9xO38YGHZK!4rnlVIqBQQQ6xpGe1PMV58edOS_cGrKoqdfhw4qSp_JQCavvEjaxXk0OwGPnxyEcO60AzKQK_XDtp7H3DdkFXcrXuu5jefU$
> ).
> > The
> > > > > > stable
> > > > > > >> > > > extension
> > > > > > >> > > > > > > point
> > > > > > >> > > > > > > > > should instead be the metrics
> reporting/emitting
> > > > > > boundary.
> > > > > > >> > > > > > > > >
> > > > > > >> > > > > > > > > Concretely, I think the next proposal should
> be
> > > shaped
> > > > > > >> like
> > > > > > >> > > this:
> > > > > > >> > > > > > > > >
> > > > > > >> > > > > > > > >    1.
> > > > > > >> > > > > > > > >
> > > > > > >> > > > > > > > >    Define the proper metrics reporting SPI at
> an
> > > > > > optional
> > > > > > >> > > Iceberg
> > > > > > >> > > > > > > metrics
> > > > > > >> > > > > > > > >    extension API layer.
> > > > > > >> > > > > > > > >
> > > > > > >> > > > > > > > >    Example direction:
> > > > > > >> > > > > > > > >
> > > > > > >> > > > > > > > >    public interface IcebergMetricsReporter {
> > > > > > >> > > > > > > > >      void
> > reportMetric(IcebergMetricsReportContext
> > > > > > >> context,
> > > > > > >> > > > > > > > > MetricsReport report);
> > > > > > >> > > > > > > > >    }
> > > > > > >> > > > > > > > >
> > > > > > >> > > > > > > > >    The context would carry the small
> > > Polaris-resolved
> > > > > > >> > envelope:
> > > > > > >> > > > > > > > >    catalog/table identity, report type,
> received
> > > > > > >> timestamp,
> > > > > > >> > and
> > > > > > >> > > > > > > > > request/trace
> > > > > > >> > > > > > > > >    context if available. The raw Iceberg
> > > MetricsReport
> > > > > > >> > remains
> > > > > > >> > > > the
> > > > > > >> > > > > > > > payload.
> > > > > > >> > > > > > > > >    2.
> > > > > > >> > > > > > > > >
> > > > > > >> > > > > > > > >    Keep runtime/service as the REST ingestion
> > and
> > > > > wiring
> > > > > > >> > layer.
> > > > > > >> > > > > > > > >
> > > > > > >> > > > > > > > >    The REST handler still resolves the table
> and
> > > > > > performs
> > > > > > >> > authz
> > > > > > >> > > > > > before
> > > > > > >> > > > > > > > >    accepting the report. After that, it calls
> > the
> > > > > > selected
> > > > > > >> > > > > reporting
> > > > > > >> > > > > > > > >    implementation.
> > > > > > >> > > > > > > > >    3.
> > > > > > >> > > > > > > > >
> > > > > > >> > > > > > > > >    Keep the battery default no-op/log-only.
> > > > > > >> > > > > > > > >
> > > > > > >> > > > > > > > >    This preserves an out-of-box safe default
> and
> > > > > avoids
> > > > > > >> > > > requiring a
> > > > > > >> > > > > > > > durable
> > > > > > >> > > > > > > > >    metrics store for every Polaris deployment.
> > > > > > >> > > > > > > > >    4.
> > > > > > >> > > > > > > > >
> > > > > > >> > > > > > > > >    Move durable JDBC metrics into a
> > self-contained
> > > > > > >> > > implementation
> > > > > > >> > > > > of
> > > > > > >> > > > > > > the
> > > > > > >> > > > > > > > >    reporting SPI.
> > > > > > >> > > > > > > > >
> > > > > > >> > > > > > > > >    That implementation can own its schema,
> > > bootstrap,
> > > > > > >> > > retention,
> > > > > > >> > > > > and
> > > > > > >> > > > > > > read
> > > > > > >> > > > > > > > >    API support. It should not define the core
> > > > > reporting
> > > > > > >> SPI
> > > > > > >> > > > > boundary,
> > > > > > >> > > > > > > and
> > > > > > >> > > > > > > > > it
> > > > > > >> > > > > > > > >    should not require metrics persistence to
> > > remain
> > > > > > >> inherited
> > > > > > >> > > > from
> > > > > > >> > > > > > > > >    BasePersistence.
> > > > > > >> > > > > > > > >    5.
> > > > > > >> > > > > > > > >
> > > > > > >> > > > > > > > >    Treat async/event-backed handling as
> another
> > > > > > >> > implementation
> > > > > > >> > > of
> > > > > > >> > > > > the
> > > > > > >> > > > > > > > same
> > > > > > >> > > > > > > > >    reporting SPI.
> > > > > > >> > > > > > > > >
> > > > > > >> > > > > > > > >    For example, an event-backed reporter could
> > > > > enqueue a
> > > > > > >> > > metrics
> > > > > > >> > > > > > event
> > > > > > >> > > > > > > > and
> > > > > > >> > > > > > > > >    let listeners handle durable storage or
> other
> > > > > sinks.
> > > > > > >> If we
> > > > > > >> > > > later
> > > > > > >> > > > > > > need
> > > > > > >> > > > > > > > a
> > > > > > >> > > > > > > > >    replaceable queue engine, that seems like a
> > > shared
> > > > > > >> > > > event/metrics
> > > > > > >> > > > > > > > > substrate
> > > > > > >> > > > > > > > >    topic rather than a metrics-only
> requirement.
> > > > > > >> > > > > > > > >
> > > > > > >> > > > > > > > > This framing lets us keep the REST metrics
> > > endpoint
> > > > > > >> simple,
> > > > > > >> > > > > preserve
> > > > > > >> > > > > > > the
> > > > > > >> > > > > > > > > current default behavior, support durable
> > metrics
> > > > > users,
> > > > > > >> and
> > > > > > >> > > > still
> > > > > > >> > > > > > > leave
> > > > > > >> > > > > > > > > room for async/event-backed or
> > > external-queue-based
> > > > > > >> > > > > implementations.
> > > > > > >> > > > > > > > >
> > > > > > >> > > > > > > > > I think the main follow-up questions are:
> > > > > > >> > > > > > > > >
> > > > > > >> > > > > > > > >    - Do folks agree that the stable SPI
> boundary
> > > > > should
> > > > > > be
> > > > > > >> > > > metrics
> > > > > > >> > > > > > > > >    reporting/emitting, not metrics
> persistence?
> > > > > > >> > > > > > > > >    - Does an optional Iceberg metrics
> extension
> > > API
> > > > > > layer
> > > > > > >> > sound
> > > > > > >> > > > > like
> > > > > > >> > > > > > > the
> > > > > > >> > > > > > > > >    right home for this SPI?
> > > > > > >> > > > > > > > >    - Should the current durable metrics work
> be
> > > > > reframed
> > > > > > >> as a
> > > > > > >> > > > > durable
> > > > > > >> > > > > > > > JDBC
> > > > > > >> > > > > > > > >    reference implementation of that SPI?
> > > > > > >> > > > > > > > >    - What is the smallest PR sequence to get
> > there
> > > > > > without
> > > > > > >> > > > blocking
> > > > > > >> > > > > > the
> > > > > > >> > > > > > > > >    current metrics work unnecessarily?
> > > > > > >> > > > > > > > >
> > > > > > >> > > > > > > > > Thanks,
> > > > > > >> > > > > > > > > -ej
> > > > > > >> > > > > > > > >
> > > > > > >> > > > > > > > > On Fri, May 15, 2026 at 5:16 PM Dmitri
> > > Bourlatchkov <
> > > > > > >> > > > > > dimas@apache.org>
> > > > > > >> > > > > > > > > wrote:
> > > > > > >> > > > > > > > >
> > > > > > >> > > > > > > > > > Hi JB,
> > > > > > >> > > > > > > > > >
> > > > > > >> > > > > > > > > > Could you set up another meeting, please?
> Same
> > > time
> > > > > on
> > > > > > >> > > > Wednesday
> > > > > > >> > > > > as
> > > > > > >> > > > > > > > last
> > > > > > >> > > > > > > > > > time... I hope it works for everyone.
> > > > > > >> > > > > > > > > >
> > > > > > >> > > > > > > > > > Cheers,
> > > > > > >> > > > > > > > > > Dmitri.
> > > > > > >> > > > > > > > > >
> > > > > > >> > > > > > > > > > On Fri, May 15, 2026 at 8:06 PM Yufei Gu <
> > > > > > >> > > flyrain000@gmail.com
> > > > > > >> > > > >
> > > > > > >> > > > > > > wrote:
> > > > > > >> > > > > > > > > >
> > > > > > >> > > > > > > > > > > +1 on another sync call next week.
> > > > > > >> > > > > > > > > > >
> > > > > > >> > > > > > > > > > > Yufei
> > > > > > >> > > > > > > > > > >
> > > > > > >> > > > > > > > > > >
> > > > > > >> > > > > > > > > > > On Fri, May 15, 2026 at 4:52 PM Dmitri
> > > > > Bourlatchkov
> > > > > > <
> > > > > > >> > > > > > > > dimas@apache.org>
> > > > > > >> > > > > > > > > > > wrote:
> > > > > > >> > > > > > > > > > >
> > > > > > >> > > > > > > > > > > > Hi All,
> > > > > > >> > > > > > > > > > > >
> > > > > > >> > > > > > > > > > > > WDYT about another sync call next week?
> > > > > > >> > > > > > > > > > > >
> > > > > > >> > > > > > > > > > > > Thanks,
> > > > > > >> > > > > > > > > > > > Dmitri.
> > > > > > >> > > > > > > > > > > >
> > > > > > >> > > > > > > > > > > > On Wed, May 6, 2026 at 5:29 PM Dmitri
> > > > > > Bourlatchkov <
> > > > > > >> > > > > > > > dimas@apache.org
> > > > > > >> > > > > > > > > >
> > > > > > >> > > > > > > > > > > > wrote:
> > > > > > >> > > > > > > > > > > >
> > > > > > >> > > > > > > > > > > > > Hi EJ,
> > > > > > >> > > > > > > > > > > > >
> > > > > > >> > > > > > > > > > > > > Thanks for the summary! It covers what
> > we
> > > > > > >> discussed
> > > > > > >> > in
> > > > > > >> > > > the
> > > > > > >> > > > > > > > meeting
> > > > > > >> > > > > > > > > > very
> > > > > > >> > > > > > > > > > > > > well, IMHO.
> > > > > > >> > > > > > > > > > > > >
> > > > > > >> > > > > > > > > > > > > Looking forward to concrete PRs :)
> > > > > > >> > > > > > > > > > > > >
> > > > > > >> > > > > > > > > > > > > Cheers,
> > > > > > >> > > > > > > > > > > > > Dmitri.
> > > > > > >> > > > > > > > > > > > >
> > > > > > >> > > > > > > > > > > > > On Wed, May 6, 2026 at 5:08 PM EJ
> Wang <
> > > > > > >> > > > > > > > > > ej.wang.dev.interest@gmail.com
> > > > > > >> > > > > > > > > > > >
> > > > > > >> > > > > > > > > > > > > wrote:
> > > > > > >> > > > > > > > > > > > >
> > > > > > >> > > > > > > > > > > > >> Hi folks,
> > > > > > >> > > > > > > > > > > > >>
> > > > > > >> > > > > > > > > > > > >> We had a community sync earlier,
> thanks
> > > JB
> > > > > for
> > > > > > >> > > > scheduling
> > > > > > >> > > > > > it.
> > > > > > >> > > > > > > > > Notes
> > > > > > >> > > > > > > > > > > from
> > > > > > >> > > > > > > > > > > > >> the first metrics architecture sync
> > (May
> > > 6,
> > > > > > >> 10-11am
> > > > > > >> > > PT).
> > > > > > >> > > > > > > > > Discussion
> > > > > > >> > > > > > > > > > > doc
> > > > > > >> > > > > > > > > > > > >> with per-section status:
> > > > > > >> > > > > > > > > > > > >>
> > > > > > >> > > > > > > > > > > > >>
> > > > > > >> > > > > > > > > > > >
> > > > > > >> > > > > > > > > > >
> > > > > > >> > > > > > > > > >
> > > > > > >> > > > > > > > >
> > > > > > >> > > > > > > >
> > > > > > >> > > > > > >
> > > > > > >> > > > > >
> > > > > > >> > > > >
> > > > > > >> > > >
> > > > > > >> > >
> > > > > > >> >
> > > > > > >>
> > > > > >
> > > > >
> > >
> >
> https://urldefense.com/v3/__https://docs.google.com/document/d/100h7c4damrUzVuquYbBHM0EvA4LSWuW2IT2dN_7nYVA/edit?tab=t.0__;!!Iz9xO38YGHZK!4rnlVIqBQQQ6xpGe1PMV58edOS_cGrKoqdfhw4qSp_JQCavvEjaxXk0OwGPnxyEcO60AzKQK_XDtp7H3DdkFXcrXgIddeUs$
> > > > > > >> > > > > > > > > > > > >>
> > > > > > >> > > > > > > > > > > > >> *The meeting covered both topics from
> > the
> > > > > doc.
> > > > > > >> > > > > > Direction-level
> > > > > > >> > > > > > > > > > > alignment
> > > > > > >> > > > > > > > > > > > >> was reached on the headline pieces;
> > > details
> > > > > > >> remain
> > > > > > >> > for
> > > > > > >> > > > PR
> > > > > > >> > > > > > > review
> > > > > > >> > > > > > > > > or
> > > > > > >> > > > > > > > > > > > >> follow-up sessions.*
> > > > > > >> > > > > > > > > > > > >>
> > > > > > >> > > > > > > > > > > > >> *Topic 1 — Persistence schema
> redesign*
> > > > > > >> > > > > > > > > > > > >> Idea-level alignment on consolidating
> > > > > per-type
> > > > > > >> > tables
> > > > > > >> > > > > > > > > > > > >> (scan_metrics_report,
> > > > > > >> > > > > > > > > > > > >> commit_metrics_report) into a single
> > > > > > >> metrics_report
> > > > > > >> > > > table.
> > > > > > >> > > > > > The
> > > > > > >> > > > > > > > > > > > motivating
> > > > > > >> > > > > > > > > > > > >> cost is the surface area added by
> every
> > > new
> > > > > > >> metric
> > > > > > >> > > type
> > > > > > >> > > > > > today:
> > > > > > >> > > > > > > > new
> > > > > > >> > > > > > > > > > > > table,
> > > > > > >> > > > > > > > > > > > >> SPI method, record class, model,
> > > converter,
> > > > > > >> schema
> > > > > > >> > > > > > migration.
> > > > > > >> > > > > > > > > > > > >>
> > > > > > >> > > > > > > > > > > > >> Most schema details are deferred to
> the
> > > > > schema
> > > > > > >> PR. A
> > > > > > >> > > few
> > > > > > >> > > > > > > > specific
> > > > > > >> > > > > > > > > > > points
> > > > > > >> > > > > > > > > > > > >> came up:
> > > > > > >> > > > > > > > > > > > >> •   metric_schema_version: Yufei
> > prefers
> > > > > > dropping
> > > > > > >> > it,
> > > > > > >> > > > > since
> > > > > > >> > > > > > > > there
> > > > > > >> > > > > > > > > is
> > > > > > >> > > > > > > > > > > no
> > > > > > >> > > > > > > > > > > > >> spec-level concept of metrics
> > versioning
> > > > > today
> > > > > > >> and
> > > > > > >> > it
> > > > > > >> > > is
> > > > > > >> > > > > > hard
> > > > > > >> > > > > > > to
> > > > > > >> > > > > > > > > > > define
> > > > > > >> > > > > > > > > > > > >> unilaterally. Robert prefers keeping
> > it,
> > > > > given
> > > > > > >> IRC
> > > > > > >> > v2
> > > > > > >> > > is
> > > > > > >> > > > > > > coming
> > > > > > >> > > > > > > > > and
> > > > > > >> > > > > > > > > > > the
> > > > > > >> > > > > > > > > > > > >> schema should be considered against
> its
> > > > > likely
> > > > > > >> > shape;
> > > > > > >> > > > > Robert
> > > > > > >> > > > > > > > also
> > > > > > >> > > > > > > > > > > raised
> > > > > > >> > > > > > > > > > > > >> how to differentiate various payload
> > > formats
> > > > > if
> > > > > > >> any.
> > > > > > >> > > > EJ's
> > > > > > >> > > > > > read
> > > > > > >> > > > > > > > is
> > > > > > >> > > > > > > > > > that
> > > > > > >> > > > > > > > > > > > >> this
> > > > > > >> > > > > > > > > > > > >> is a two-way-door decision. We can
> > start
> > > > > > without
> > > > > > >> the
> > > > > > >> > > > > field,
> > > > > > >> > > > > > > and
> > > > > > >> > > > > > > > if
> > > > > > >> > > > > > > > > > IRC
> > > > > > >> > > > > > > > > > > > v2
> > > > > > >> > > > > > > > > > > > >> changes the shape we would likely
> roll
> > a
> > > > > > >> > corresponding
> > > > > > >> > > > new
> > > > > > >> > > > > > > > schema
> > > > > > >> > > > > > > > > > > > anyway,
> > > > > > >> > > > > > > > > > > > >> which is not particularly costly.
> > > > > > >> > > > > > > > > > > > >> •   Payload format: Robert pointed
> out
> > > that
> > > > > > >> future
> > > > > > >> > > > formats
> > > > > > >> > > > > > > > beyond
> > > > > > >> > > > > > > > > > JSON
> > > > > > >> > > > > > > > > > > > may
> > > > > > >> > > > > > > > > > > > >> be worth supporting. The exact shape
> is
> > > > > > deferred
> > > > > > >> to
> > > > > > >> > > the
> > > > > > >> > > > > > schema
> > > > > > >> > > > > > > > > > > > discussion.
> > > > > > >> > > > > > > > > > > > >> •   Partition strategy: Anand
> suggested
> > > > > monthly
> > > > > > >> > > > > partitioning
> > > > > > >> > > > > > > > based
> > > > > > >> > > > > > > > > > on
> > > > > > >> > > > > > > > > > > > his
> > > > > > >> > > > > > > > > > > > >> experience as potentially helpful at
> > > scale.
> > > > > > >> > > > > > > > > > > > >>
> > > > > > >> > > > > > > > > > > > >> *Topic 2 — Where metrics ingestion
> and
> > > > > storage
> > > > > > >> > belong*
> > > > > > >> > > > > > > > > > > > >> Idea-level alignment that metrics
> > should
> > > be a
> > > > > > >> > > separated
> > > > > > >> > > > > SPI
> > > > > > >> > > > > > > from
> > > > > > >> > > > > > > > > the
> > > > > > >> > > > > > > > > > > > >> entity
> > > > > > >> > > > > > > > > > > > >> persistence stack. Two reasons
> > surfaced:
> > > (a)
> > > > > > >> > workloads
> > > > > > >> > > > and
> > > > > > >> > > > > > > > > > capability
> > > > > > >> > > > > > > > > > > > >> requirements diverge enough that
> > coupling
> > > > > them
> > > > > > >> > creates
> > > > > > >> > > > > > > > artificial
> > > > > > >> > > > > > > > > > > > >> constraints, and (b) admin experience
> > > > > improves
> > > > > > >> when
> > > > > > >> > > > > metrics
> > > > > > >> > > > > > > has
> > > > > > >> > > > > > > > > its
> > > > > > >> > > > > > > > > > > own
> > > > > > >> > > > > > > > > > > > >> bootstrap, retention, and lifecycle.
> > > Dmitri
> > > > > > noted
> > > > > > >> > > > Polaris
> > > > > > >> > > > > > > being
> > > > > > >> > > > > > > > a
> > > > > > >> > > > > > > > > > > > platform
> > > > > > >> > > > > > > > > > > > >> should have the flexibility to
> support
> > > > > > different
> > > > > > >> > > > > persistence
> > > > > > >> > > > > > > > > > backends
> > > > > > >> > > > > > > > > > > > per
> > > > > > >> > > > > > > > > > > > >> concern, and pointed to a concrete
> next
> > > step
> > > > > of
> > > > > > >> > > > separating
> > > > > > >> > > > > > the
> > > > > > >> > > > > > > > > JDBC
> > > > > > >> > > > > > > > > > > > >> bootstrap for metrics from the
> > metastore
> > > > > > >> bootstrap.
> > > > > > >> > > > Robert
> > > > > > >> > > > > > > > > proposed
> > > > > > >> > > > > > > > > > an
> > > > > > >> > > > > > > > > > > > >> additional UX extension: detect an
> > > > > > unbootstrapped
> > > > > > >> > > > metrics
> > > > > > >> > > > > > > store
> > > > > > >> > > > > > > > on
> > > > > > >> > > > > > > > > > > first
> > > > > > >> > > > > > > > > > > > >> use and auto-bootstrap rather than
> > > requiring
> > > > > an
> > > > > > >> > > explicit
> > > > > > >> > > > > > > manual
> > > > > > >> > > > > > > > > > > > bootstrap
> > > > > > >> > > > > > > > > > > > >> step.
> > > > > > >> > > > > > > > > > > > >> The meeting also confirmed that
> Polaris
> > > > > metrics
> > > > > > >> can
> > > > > > >> > > > start
> > > > > > >> > > > > > > small
> > > > > > >> > > > > > > > > and
> > > > > > >> > > > > > > > > > > stay
> > > > > > >> > > > > > > > > > > > >> Iceberg-focused. Naming and
> persistence
> > > > > schema
> > > > > > >> can
> > > > > > >> > > lean
> > > > > > >> > > > > > > > > > > > Iceberg-specific.
> > > > > > >> > > > > > > > > > > > >> If a future expansion to
> generic-table
> > > > > metrics
> > > > > > or
> > > > > > >> > > > > > operational
> > > > > > >> > > > > > > > > > metrics
> > > > > > >> > > > > > > > > > > > >> arrives, an abstraction layer can be
> > > built on
> > > > > > >> top of
> > > > > > >> > > the
> > > > > > >> > > > > > > Iceberg
> > > > > > >> > > > > > > > > > > metrics
> > > > > > >> > > > > > > > > > > > >> reporter at that point. Robert
> remains
> > > on the
> > > > > > >> fence
> > > > > > >> > > and
> > > > > > >> > > > > > would
> > > > > > >> > > > > > > > > prefer
> > > > > > >> > > > > > > > > > > > >> something more generic but did not
> > block
> > > the
> > > > > > >> > > direction;
> > > > > > >> > > > > > > Dmitri's
> > > > > > >> > > > > > > > > > read
> > > > > > >> > > > > > > > > > > > was
> > > > > > >> > > > > > > > > > > > >> that the proposed framework already
> has
> > > > > enough
> > > > > > >> > > > flexibility
> > > > > > >> > > > > > to
> > > > > > >> > > > > > > > > absorb
> > > > > > >> > > > > > > > > > > > >> future
> > > > > > >> > > > > > > > > > > > >> expansion.
> > > > > > >> > > > > > > > > > > > >>
> > > > > > >> > > > > > > > > > > > >> The Trade-offs and Proposed structure
> > > > > sections
> > > > > > in
> > > > > > >> > the
> > > > > > >> > > > doc
> > > > > > >> > > > > > were
> > > > > > >> > > > > > > > not
> > > > > > >> > > > > > > > > > > > >> reviewed
> > > > > > >> > > > > > > > > > > > >> in detail. They remain open for
> either
> > > the
> > > > > next
> > > > > > >> sync
> > > > > > >> > > or
> > > > > > >> > > > PR
> > > > > > >> > > > > > > > review.
> > > > > > >> > > > > > > > > > > > >>
> > > > > > >> > > > > > > > > > > > >> *Cross-cutting alignment —
> > > battery-included
> > > > > > plus
> > > > > > >> > > > > pluggable*
> > > > > > >> > > > > > > > > > > > >> A common philosophy emerged from the
> > > > > > discussion.
> > > > > > >> EJ
> > > > > > >> > > > > > summarized
> > > > > > >> > > > > > > > it
> > > > > > >> > > > > > > > > > as:
> > > > > > >> > > > > > > > > > > > >> Polaris should provide a
> > > battery-included UX
> > > > > > for
> > > > > > >> > > > beginners
> > > > > > >> > > > > > and
> > > > > > >> > > > > > > > the
> > > > > > >> > > > > > > > > > > > >> flexibility for advanced users to
> swap
> > > the
> > > > > > >> included
> > > > > > >> > > > > battery
> > > > > > >> > > > > > > for
> > > > > > >> > > > > > > > > > > > something
> > > > > > >> > > > > > > > > > > > >> more powerful or tailored to their
> use
> > > case.
> > > > > > The
> > > > > > >> SPI
> > > > > > >> > > > > design
> > > > > > >> > > > > > > > needs
> > > > > > >> > > > > > > > > to
> > > > > > >> > > > > > > > > > > > >> enable
> > > > > > >> > > > > > > > > > > > >> both.
> > > > > > >> > > > > > > > > > > > >>
> > > > > > >> > > > > > > > > > > > >> The inputs that shaped this framing:
> > > > > > >> > > > > > > > > > > > >> •   Anand described how his team uses
> > the
> > > > > > current
> > > > > > >> > > > metrics
> > > > > > >> > > > > > > > > > persistence
> > > > > > >> > > > > > > > > > > > >> (three metrics consumers in v1.4).
> > > > > > >> > > > > > > > > > > > >> •   Yufei raised Grafana and
> dashboard
> > > > > > >> integrations
> > > > > > >> > > as a
> > > > > > >> > > > > > > > > destination
> > > > > > >> > > > > > > > > > > use
> > > > > > >> > > > > > > > > > > > >> case beyond the default.
> > > > > > >> > > > > > > > > > > > >> •   Robert called out that the
> current
> > > design
> > > > > > is
> > > > > > >> > more
> > > > > > >> > > > > > > > > JDBC-focused.
> > > > > > >> > > > > > > > > > > > >>
> > > > > > >> > > > > > > > > > > > >> Two concrete instances:
> > > > > > >> > > > > > > > > > > > >> •   Async metrics intake: Yufei's
> > initial
> > > > > > >> position
> > > > > > >> > was
> > > > > > >> > > > > that
> > > > > > >> > > > > > > > async
> > > > > > >> > > > > > > > > > > should
> > > > > > >> > > > > > > > > > > > >> largely live on the producer side and
> > > there
> > > > > is
> > > > > > >> not
> > > > > > >> > > much
> > > > > > >> > > > > > > Polaris
> > > > > > >> > > > > > > > > can
> > > > > > >> > > > > > > > > > > do.
> > > > > > >> > > > > > > > > > > > >> Robert suggested a Polaris-side
> default
> > > is
> > > > > > doable
> > > > > > >> > via
> > > > > > >> > > > > > Vert.x.
> > > > > > >> > > > > > > > > Dmitri
> > > > > > >> > > > > > > > > > > > >> agreed
> > > > > > >> > > > > > > > > > > > >> the direction is worth exploring. The
> > > meeting
> > > > > > >> > > converged
> > > > > > >> > > > > on a
> > > > > > >> > > > > > > > > > > > >> battery-included default (likely
> > > > > Vert.x-backed)
> > > > > > >> with
> > > > > > >> > > an
> > > > > > >> > > > > SPI
> > > > > > >> > > > > > > > shape
> > > > > > >> > > > > > > > > > that
> > > > > > >> > > > > > > > > > > > >> lets
> > > > > > >> > > > > > > > > > > > >> power users route to a more scalable
> > > backend
> > > > > > >> > > (k8s-hosted
> > > > > > >> > > > > > > queue,
> > > > > > >> > > > > > > > > AWS
> > > > > > >> > > > > > > > > > > SQS,
> > > > > > >> > > > > > > > > > > > >> etc.).
> > > > > > >> > > > > > > > > > > > >> •   Pluggable destinations: combining
> > > Yufei's
> > > > > > >> > > dashboard
> > > > > > >> > > > > use
> > > > > > >> > > > > > > case
> > > > > > >> > > > > > > > > > with
> > > > > > >> > > > > > > > > > > > >> Robert's JDBC-focused call-out, the
> > > meeting
> > > > > > >> agreed
> > > > > > >> > the
> > > > > > >> > > > SPI
> > > > > > >> > > > > > > > should
> > > > > > >> > > > > > > > > be
> > > > > > >> > > > > > > > > > > > >> structured for multiple sinks so
> > > integrations
> > > > > > >> become
> > > > > > >> > > > impl
> > > > > > >> > > > > > > > choices
> > > > > > >> > > > > > > > > > > rather
> > > > > > >> > > > > > > > > > > > >> than architectural changes.
> > > > > > >> > > > > > > > > > > > >>
> > > > > > >> > > > > > > > > > > > >> The battery-included default is most
> > > likely
> > > > > to
> > > > > > >> use
> > > > > > >> > the
> > > > > > >> > > > > > > existing
> > > > > > >> > > > > > > > > > > > >> JDBC-backed
> > > > > > >> > > > > > > > > > > > >> approach.
> > > > > > >> > > > > > > > > > > > >>
> > > > > > >> > > > > > > > > > > > >> *Direction (idea-level alignment)*
> > > > > > >> > > > > > > > > > > > >> •   Single metrics_report table
> > > consolidating
> > > > > > >> > per-type
> > > > > > >> > > > > > > metrics,
> > > > > > >> > > > > > > > > > > > replacing
> > > > > > >> > > > > > > > > > > > >> scan_metrics_report and
> > > commit_metrics_report
> > > > > > >> > > > > > > > > > > > >> •   Iceberg-focused naming and schema
> > for
> > > > > now,
> > > > > > >> > revisit
> > > > > > >> > > > if
> > > > > > >> > > > > > > > > > > generic-table
> > > > > > >> > > > > > > > > > > > or
> > > > > > >> > > > > > > > > > > > >> operational metrics arrive
> > > > > > >> > > > > > > > > > > > >> •   Metrics persistence as a
> separated
> > > SPI,
> > > > > not
> > > > > > >> on
> > > > > > >> > > > > > > > BasePersistence
> > > > > > >> > > > > > > > > > > > >> •   Bootstrap path separated for
> > metrics,
> > > > > > >> > independent
> > > > > > >> > > of
> > > > > > >> > > > > > > > metastore
> > > > > > >> > > > > > > > > > > > >> bootstrap
> > > > > > >> > > > > > > > > > > > >> •   "Battery-included plus pluggable"
> > as
> > > the
> > > > > > SPI
> > > > > > >> > > design
> > > > > > >> > > > > > > > philosophy
> > > > > > >> > > > > > > > > > > > >>
> > > > > > >> > > > > > > > > > > > >> *Open items*
> > > > > > >> > > > > > > > > > > > >> •   Schema details:
> > > metric_schema_version,
> > > > > > >> payload
> > > > > > >> > > > format,
> > > > > > >> > > > > > IRC
> > > > > > >> > > > > > > > v2
> > > > > > >> > > > > > > > > > > > >> forward-compat shape
> > > > > > >> > > > > > > > > > > > >> •   SPI design details — full review
> > > either
> > > > > in
> > > > > > >> the
> > > > > > >> > > next
> > > > > > >> > > > > sync
> > > > > > >> > > > > > > or
> > > > > > >> > > > > > > > in
> > > > > > >> > > > > > > > > > the
> > > > > > >> > > > > > > > > > > > >> corresponding PR
> > > > > > >> > > > > > > > > > > > >> •   Schema refactor PR ownership
> > > > > > >> > > > > > > > > > > > >>
> > > > > > >> > > > > > > > > > > > >> *Action items*
> > > > > > >> > > > > > > > > > > > >> •   EJ to take a first stab at the
> SPI
> > > design
> > > > > > and
> > > > > > >> > > > > > potentially
> > > > > > >> > > > > > > > > > partner
> > > > > > >> > > > > > > > > > > > with
> > > > > > >> > > > > > > > > > > > >> Anand to incorporate the lessons
> > learned
> > > from
> > > > > > the
> > > > > > >> > > > existing
> > > > > > >> > > > > > > > > reporter
> > > > > > >> > > > > > > > > > > and
> > > > > > >> > > > > > > > > > > > >> persistence work.
> > > > > > >> > > > > > > > > > > > >> •   Schema refactor PR ownership is
> not
> > > yet
> > > > > > >> decided.
> > > > > > >> > > If
> > > > > > >> > > > > > anyone
> > > > > > >> > > > > > > > is
> > > > > > >> > > > > > > > > > > > >> interested in driving it, reply on
> this
> > > > > thread.
> > > > > > >> > > > > > > > > > > > >> •   JB to schedule the next sync,
> > > tentatively
> > > > > > in
> > > > > > >> two
> > > > > > >> > > > > weeks.
> > > > > > >> > > > > > > > > > > > >>
> > > > > > >> > > > > > > > > > > > >> -ej
> > > > > > >> > > > > > > > > > > > >>
> > > > > > >> > > > > > > > > > > > >> On Mon, Apr 27, 2026 at 3:07 PM EJ
> > Wang <
> > > > > > >> > > > > > > > > > > ej.wang.dev.interest@gmail.com
> > > > > > >> > > > > > > > > > > > >
> > > > > > >> > > > > > > > > > > > >> wrote:
> > > > > > >> > > > > > > > > > > > >>
> > > > > > >> > > > > > > > > > > > >> > Thanks Yufei for the +1.
> > > > > > >> > > > > > > > > > > > >> >
> > > > > > >> > > > > > > > > > > > >> > JB, could you help add a biweekly
> > > metrics
> > > > > > >> > > architecture
> > > > > > >> > > > > > sync
> > > > > > >> > > > > > > to
> > > > > > >> > > > > > > > > the
> > > > > > >> > > > > > > > > > > > >> Polaris
> > > > > > >> > > > > > > > > > > > >> > community calendar? I'm thinking
> > > Thursdays
> > > > > at
> > > > > > >> > 9-10am
> > > > > > >> > > > PT,
> > > > > > >> > > > > > on
> > > > > > >> > > > > > > > the
> > > > > > >> > > > > > > > > > > > >> off-weeks
> > > > > > >> > > > > > > > > > > > >> > from the community meeting
> (starting
> > > May
> > > > > 7),
> > > > > > 60
> > > > > > >> > > > minutes.
> > > > > > >> > > > > > > > > > > > >> >
> > > > > > >> > > > > > > > > > > > >> > Here's a rough agenda to work
> through
> > > over
> > > > > > the
> > > > > > >> > first
> > > > > > >> > > > few
> > > > > > >> > > > > > > > > sessions,
> > > > > > >> > > > > > > > > > > > >> grouped
> > > > > > >> > > > > > > > > > > > >> > by priority:
> > > > > > >> > > > > > > > > > > > >> >
> > > > > > >> > > > > > > > > > > > >> > *First: foundational direction*
> > > > > > >> > > > > > > > > > > > >> >
> > > > > > >> > > > > > > > > > > > >> > 1.  MetricsPersistence: public SPI
> or
> > > > > > internal
> > > > > > >> > > > > > > implementation
> > > > > > >> > > > > > > > > > > detail?
> > > > > > >> > > > > > > > > > > > >> >    •   Marked @Beta, javadoc calls
> > it a
> > > > > > >> "Service
> > > > > > >> > > > > Provider
> > > > > > >> > > > > > > > > > > Interface",
> > > > > > >> > > > > > > > > > > > >> but
> > > > > > >> > > > > > > > > > > > >> > only one consumer
> > > > > (JdbcBasePersistenceImpl),
> > > > > > >> lives
> > > > > > >> > > on
> > > > > > >> > > > > > > > > > > BasePersistence.
> > > > > > >> > > > > > > > > > > > >> If
> > > > > > >> > > > > > > > > > > > >> > demoted to a private helper inside
> a
> > > > > > persisting
> > > > > > >> > > > reporter
> > > > > > >> > > > > > > impl,
> > > > > > >> > > > > > > > > > most
> > > > > > >> > > > > > > > > > > > >> > downstream design decisions become
> > > > > > >> implementation
> > > > > > >> > > > > details
> > > > > > >> > > > > > > > rather
> > > > > > >> > > > > > > > > > > than
> > > > > > >> > > > > > > > > > > > >> > contract questions.
> > > > > > >> > > > > > > > > > > > >> >
> > > > > > >> > > > > > > > > > > > >> > 2.  Persistence schema redesign
> > > > > > >> > > > > > > > > > > > >> >    •   Current two-table layout
> > > > > > >> > > (scan_metrics_report,
> > > > > > >> > > > > > > > > > > > >> > commit_metrics_report) with ~25
> > > flattened
> > > > > > >> columns
> > > > > > >> > > > each.
> > > > > > >> > > > > > > Every
> > > > > > >> > > > > > > > > new
> > > > > > >> > > > > > > > > > > > metric
> > > > > > >> > > > > > > > > > > > >> > type requires a new table, SPI
> > method,
> > > > > record
> > > > > > >> > class,
> > > > > > >> > > > > > model,
> > > > > > >> > > > > > > > > > > converter,
> > > > > > >> > > > > > > > > > > > >> and
> > > > > > >> > > > > > > > > > > > >> > schema migration. Direction to
> > explore:
> > > > > > single
> > > > > > >> > table
> > > > > > >> > > > > with
> > > > > > >> > > > > > > > > > > metric_type
> > > > > > >> > > > > > > > > > > > >> enum,
> > > > > > >> > > > > > > > > > > > >> > schema_version, and JSON payload
> > > column.
> > > > > > >> > > > > > > > > > > > >> >
> > > > > > >> > > > > > > > > > > > >> > *Second: design details once
> > direction
> > > is
> > > > > > set*
> > > > > > >> > > > > > > > > > > > >> >
> > > > > > >> > > > > > > > > > > > >> > 3.  Partition key strategy
> > > > > > >> > > > > > > > > > > > >> >    •   Single-table design means
> scan
> > > > > metrics
> > > > > > >> at
> > > > > > >> > > scale
> > > > > > >> > > > > > will
> > > > > > >> > > > > > > > have
> > > > > > >> > > > > > > > > > > high
> > > > > > >> > > > > > > > > > > > >> > write concurrency per table. Schema
> > > needs
> > > > > to
> > > > > > >> > expose
> > > > > > >> > > > > enough
> > > > > > >> > > > > > > > > > structure
> > > > > > >> > > > > > > > > > > > for
> > > > > > >> > > > > > > > > > > > >> > backends to shard by entity or time
> > > range.
> > > > > > >> > > > > > > > > > > > >> >
> > > > > > >> > > > > > > > > > > > >> > 4.  Read/write path consistency
> > > > > > >> > > > > > > > > > > > >> >    •   Writes go through
> > > > > > PolarisMetricsManager
> > > > > > >> on
> > > > > > >> > > > > > > > > > MetaStoreManager.
> > > > > > >> > > > > > > > > > > > >> Reads
> > > > > > >> > > > > > > > > > > > >> > bypass MetaStoreManager and go
> > > straight to
> > > > > > >> > > > > > BasePersistence,
> > > > > > >> > > > > > > > > > > excluding
> > > > > > >> > > > > > > > > > > > >> > non-JDBC backends from the read
> API.
> > > > > > >> > > > > > > > > > > > >> >
> > > > > > >> > > > > > > > > > > > >> > *Third: cleanup and alignment*
> > > > > > >> > > > > > > > > > > > >> >
> > > > > > >> > > > > > > > > > > > >> > 5.  PolarisMetricsReporter naming
> > > > > > >> > > > > > > > > > > > >> >    •   Only handles IRC
> > > > > > >> (ScanReport/CommitReport),
> > > > > > >> > > > > doesn't
> > > > > > >> > > > > > > > cover
> > > > > > >> > > > > > > > > > > > generic
> > > > > > >> > > > > > > > > > > > >> > tables or operational metrics. Name
> > is
> > > > > > broader
> > > > > > >> > than
> > > > > > >> > > > > scope.
> > > > > > >> > > > > > > > > > > > >> >
> > > > > > >> > > > > > > > > > > > >> > 6.  PolarisMetricsManager facade
> > > > > passthrough
> > > > > > >> > > > > > > > > > > > >> >    •   Entire default method is
> > > > > > >> > > > > > > > > > > > >>
> > callCtx.getMetaStore().writeScanReport().
> > > > > > >> > > > > > > > > > > > >> > Zero logic, passes Level 1 straight
> > > through
> > > > > > to
> > > > > > >> > Level
> > > > > > >> > > > 3.
> > > > > > >> > > > > > Same
> > > > > > >> > > > > > > > > > > > >> anti-pattern
> > > > > > >> > > > > > > > > > > > >> > as PolarisEventManager.
> > > > > > >> > > > > > > > > > > > >> >
> > > > > > >> > > > > > > > > > > > >> > 7.  Iceberg community alignment
> > > > > > >> > > > > > > > > > > > >> >    •   Payload-type extension needs
> > > > > > discussion
> > > > > > >> on
> > > > > > >> > > > > > > dev@iceberg.
> > > > > > >> > > > > > > > > > > > >> obelix74's
> > > > > > >> > > > > > > > > > > > >> > Feb thread got zero replies. Needs
> a
> > > > > > committer
> > > > > > >> > > voice.
> > > > > > >> > > > > > > > > > > > >> >
> > > > > > >> > > > > > > > > > > > >> > Lets confirm prioritization in the
> > > first
> > > > > > >> session.
> > > > > > >> > > > > > > > > > > > >> >
> > > > > > >> > > > > > > > > > > > >> > -ej
> > > > > > >> > > > > > > > > > > > >> >
> > > > > > >> > > > > > > > > > > > >> > On Tue, Apr 21, 2026 at 3:18 PM
> Yufei
> > > Gu <
> > > > > > >> > > > > > > > flyrain000@gmail.com>
> > > > > > >> > > > > > > > > > > > wrote:
> > > > > > >> > > > > > > > > > > > >> >
> > > > > > >> > > > > > > > > > > > >> >> Thanks everyone for continuing to
> > > drive
> > > > > this
> > > > > > >> > > > forward. I
> > > > > > >> > > > > > > agree
> > > > > > >> > > > > > > > > > that
> > > > > > >> > > > > > > > > > > > the
> > > > > > >> > > > > > > > > > > > >> >> problem is getting complex enough
> > > that a
> > > > > > more
> > > > > > >> > > > > structured
> > > > > > >> > > > > > > > > > discussion
> > > > > > >> > > > > > > > > > > > >> would
> > > > > > >> > > > > > > > > > > > >> >> help.
> > > > > > >> > > > > > > > > > > > >> >>
> > > > > > >> > > > > > > > > > > > >> >> +1 on setting up a biweekly sync
> for
> > > the
> > > > > > >> metrics
> > > > > > >> > > > > > > > architecture.
> > > > > > >> > > > > > > > > > I’m
> > > > > > >> > > > > > > > > > > > >> happy
> > > > > > >> > > > > > > > > > > > >> >> to
> > > > > > >> > > > > > > > > > > > >> >> join.
> > > > > > >> > > > > > > > > > > > >> >>
> > > > > > >> > > > > > > > > > > > >> >> Yufei
> > > > > > >> > > > > > > > > > > > >> >>
> > > > > > >> > > > > > > > > > > > >> >>
> > > > > > >> > > > > > > > > > > > >> >> On Tue, Apr 21, 2026 at 2:34 PM EJ
> > > Wang <
> > > > > > >> > > > > > > > > > > > >> ej.wang.dev.interest@gmail.com>
> > > > > > >> > > > > > > > > > > > >> >> wrote:
> > > > > > >> > > > > > > > > > > > >> >>
> > > > > > >> > > > > > > > > > > > >> >> > Also, I've been looking more
> > > closely at
> > > > > > the
> > > > > > >> > > > > > *persistence
> > > > > > >> > > > > > > > > schema
> > > > > > >> > > > > > > > > > > in
> > > > > > >> > > > > > > > > > > > >> the
> > > > > > >> > > > > > > > > > > > >> >> > current metrics work*, and I
> think
> > > > > > there's a
> > > > > > >> > > > > structural
> > > > > > >> > > > > > > > > > rigidity
> > > > > > >> > > > > > > > > > > > >> problem
> > > > > > >> > > > > > > > > > > > >> >> > worth raising before the shape
> > gets
> > > > > locked
> > > > > > >> in.
> > > > > > >> > > > > > > > > > > > >> >> >
> > > > > > >> > > > > > > > > > > > >> >> > Right now we have two separate
> > > tables
> > > > > > >> > > > > > > (scan_metrics_report
> > > > > > >> > > > > > > > > and
> > > > > > >> > > > > > > > > > > > >> >> > commit_metrics_report), each
> with
> > > ~25
> > > > > > >> flattened
> > > > > > >> > > > > columns
> > > > > > >> > > > > > > > that
> > > > > > >> > > > > > > > > > > > directly
> > > > > > >> > > > > > > > > > > > >> >> > mirror the Iceberg report
> fields.
> > > The
> > > > > SPI
> > > > > > >> > follows
> > > > > > >> > > > the
> > > > > > >> > > > > > > same
> > > > > > >> > > > > > > > > > split:
> > > > > > >> > > > > > > > > > > > >> >> > writeScanReport and
> > > writeCommitReport as
> > > > > > >> > separate
> > > > > > >> > > > > > > methods,
> > > > > > >> > > > > > > > > with
> > > > > > >> > > > > > > > > > > > >> per-type
> > > > > > >> > > > > > > > > > > > >> >> > record classes, converters, and
> > > model
> > > > > > >> objects.
> > > > > > >> > > *The
> > > > > > >> > > > > > > > practical
> > > > > > >> > > > > > > > > > > cost:
> > > > > > >> > > > > > > > > > > > >> >> > adding a new metric type
> > > (operational
> > > > > > >> metrics,
> > > > > > >> > > for
> > > > > > >> > > > > > > example)
> > > > > > >> > > > > > > > > > > > requires
> > > > > > >> > > > > > > > > > > > >> a
> > > > > > >> > > > > > > > > > > > >> >> new
> > > > > > >> > > > > > > > > > > > >> >> > table, a new SPI method, a new
> > > record
> > > > > > >> class, a
> > > > > > >> > > new
> > > > > > >> > > > > > model
> > > > > > >> > > > > > > > > > class, a
> > > > > > >> > > > > > > > > > > > new
> > > > > > >> > > > > > > > > > > > >> >> > converter branch, and a schema
> > > > > migration*.
> > > > > > >> > > That's a
> > > > > > >> > > > > lot
> > > > > > >> > > > > > > of
> > > > > > >> > > > > > > > > > > surface
> > > > > > >> > > > > > > > > > > > >> area
> > > > > > >> > > > > > > > > > > > >> >> > for what should be "one more
> kind
> > of
> > > > > > >> metric."
> > > > > > >> > > > > > > > > > > > >> >> >
> > > > > > >> > > > > > > > > > > > >> >> > *My bias* would be toward a
> single
> > > > > metrics
> > > > > > >> > table
> > > > > > >> > > > with
> > > > > > >> > > > > > *a
> > > > > > >> > > > > > > > > typed
> > > > > > >> > > > > > > > > > > JSON
> > > > > > >> > > > > > > > > > > > >> >> > payload*. Something like:
> > > metric_type
> > > > > > >> (enum),
> > > > > > >> > > > > > entity_id,
> > > > > > >> > > > > > > > > > > > >> >> > table_identifier, snapshot_id
> > > > > (nullable),
> > > > > > >> > > > > received_ts,
> > > > > > >> > > > > > > > > > > > >> schema_version,
> > > > > > >> > > > > > > > > > > > >> >> and
> > > > > > >> > > > > > > > > > > > >> >> > a payload column for the
> > > metric-specific
> > > > > > >> data.
> > > > > > >> > > The
> > > > > > >> > > > > > > > > metric_type
> > > > > > >> > > > > > > > > > +
> > > > > > >> > > > > > > > > > > > >> >> > schema_version pair gives us a
> > > > > > >> > forward-compatible
> > > > > > >> > > > > > > contract
> > > > > > >> > > > > > > > > for
> > > > > > >> > > > > > > > > > > the
> > > > > > >> > > > > > > > > > > > >> >> payload
> > > > > > >> > > > > > > > > > > > >> >> > shape. Adding a new metric type
> > > becomes
> > > > > an
> > > > > > >> enum
> > > > > > >> > > > value
> > > > > > >> > > > > > > and a
> > > > > > >> > > > > > > > > > > payload
> > > > > > >> > > > > > > > > > > > >> >> schema,
> > > > > > >> > > > > > > > > > > > >> >> > not a schema migration. One
> thing
> > I
> > > > > think
> > > > > > we
> > > > > > >> > need
> > > > > > >> > > > to
> > > > > > >> > > > > be
> > > > > > >> > > > > > > > > > > deliberate
> > > > > > >> > > > > > > > > > > > >> >> about is
> > > > > > >> > > > > > > > > > > > >> >> > the partition key design. If all
> > > metric
> > > > > > >> types
> > > > > > >> > > land
> > > > > > >> > > > in
> > > > > > >> > > > > > one
> > > > > > >> > > > > > > > > > table,
> > > > > > >> > > > > > > > > > > > scan
> > > > > > >> > > > > > > > > > > > >> >> > metrics at scale (high
> > concurrency,
> > > high
> > > > > > >> > > frequency
> > > > > > >> > > > > > across
> > > > > > >> > > > > > > > > many
> > > > > > >> > > > > > > > > > > > >> tables)
> > > > > > >> > > > > > > > > > > > >> >> > could easily create hot
> > partitions.
> > > We'd
> > > > > > >> want
> > > > > > >> > the
> > > > > > >> > > > > > > > persistence
> > > > > > >> > > > > > > > > > > layer
> > > > > > >> > > > > > > > > > > > >> to
> > > > > > >> > > > > > > > > > > > >> >> be
> > > > > > >> > > > > > > > > > > > >> >> > able to shard by entity or time
> > > range,
> > > > > and
> > > > > > >> that
> > > > > > >> > > > means
> > > > > > >> > > > > > the
> > > > > > >> > > > > > > > > > logical
> > > > > > >> > > > > > > > > > > > >> schema
> > > > > > >> > > > > > > > > > > > >> >> > needs to expose enough structure
> > for
> > > > > > >> backends
> > > > > > >> > to
> > > > > > >> > > > > > > partition
> > > > > > >> > > > > > > > > on.
> > > > > > >> > > > > > > > > > I
> > > > > > >> > > > > > > > > > > > >> don't
> > > > > > >> > > > > > > > > > > > >> >> > think the current flattened
> layout
> > > gives
> > > > > > us
> > > > > > >> > that.
> > > > > > >> > > > > > > > > > > > >> >> >
> > > > > > >> > > > > > > > > > > > >> >> > This is getting complex enough
> > that
> > > I
> > > > > > don't
> > > > > > >> > think
> > > > > > >> > > > > > ad-hoc
> > > > > > >> > > > > > > > > PR/ML
> > > > > > >> > > > > > > > > > > > >> threads
> > > > > > >> > > > > > > > > > > > >> >> > will converge well. *Would
> people
> > be
> > > > > open
> > > > > > >> to a
> > > > > > >> > > > > biweekly
> > > > > > >> > > > > > > > sync
> > > > > > >> > > > > > > > > > for
> > > > > > >> > > > > > > > > > > > >> metrics
> > > > > > >> > > > > > > > > > > > >> >> > architecture?* I think 30
> minutes
> > > every
> > > > > > two
> > > > > > >> > weeks
> > > > > > >> > > > > with
> > > > > > >> > > > > > > > > > interested
> > > > > > >> > > > > > > > > > > > >> >> parties
> > > > > > >> > > > > > > > > > > > >> >> > would be enough to work through
> > the
> > > > > > schema,
> > > > > > >> SPI
> > > > > > >> > > > > shape,
> > > > > > >> > > > > > > and
> > > > > > >> > > > > > > > > read
> > > > > > >> > > > > > > > > > > API
> > > > > > >> > > > > > > > > > > > >> >> design
> > > > > > >> > > > > > > > > > > > >> >> > together. Happy to help set that
> > up.
> > > > > > >> > > > > > > > > > > > >> >> >
> > > > > > >> > > > > > > > > > > > >> >> > -ej
> > > > > > >> > > > > > > > > > > > >> >> >
> > > > > > >> > > > > > > > > > > > >> >> > On Mon, Apr 20, 2026 at 2:19 PM
> EJ
> > > Wang
> > > > > <
> > > > > > >> > > > > > > > > > > > >> ej.wang.dev.interest@gmail.com
> > > > > > >> > > > > > > > > > > > >> >> >
> > > > > > >> > > > > > > > > > > > >> >> > wrote:
> > > > > > >> > > > > > > > > > > > >> >> >
> > > > > > >> > > > > > > > > > > > >> >> >> Reviewed #4115, left a comment
> on
> > > the
> > > > > > code
> > > > > > >> > > > > > organization
> > > > > > >> > > > > > > > > side.
> > > > > > >> > > > > > > > > > > > >> >> >>
> > > > > > >> > > > > > > > > > > > >> >> >> One thing stood out: the
> metrics
> > > write
> > > > > > path
> > > > > > >> > > enters
> > > > > > >> > > > > > > through
> > > > > > >> > > > > > > > > > > > >> >> >> PolarisMetricsManager on
> > > > > > MetaStoreManager,
> > > > > > >> but
> > > > > > >> > > the
> > > > > > >> > > > > new
> > > > > > >> > > > > > > > read
> > > > > > >> > > > > > > > > > path
> > > > > > >> > > > > > > > > > > > >> >> bypasses
> > > > > > >> > > > > > > > > > > > >> >> >> MetaStoreManager entirely and
> > goes
> > > > > > >> straight to
> > > > > > >> > > > > > > > > BasePersistence
> > > > > > >> > > > > > > > > > > via
> > > > > > >> > > > > > > > > > > > >> >> >> callContext.getMetaStore().
> That
> > > means
> > > > > > the
> > > > > > >> > read
> > > > > > >> > > > API
> > > > > > >> > > > > > only
> > > > > > >> > > > > > > > > works
> > > > > > >> > > > > > > > > > > for
> > > > > > >> > > > > > > > > > > > >> >> backends
> > > > > > >> > > > > > > > > > > > >> >> >> that implement BasePersistence.
> > > NoSQL
> > > > > and
> > > > > > >> > remote
> > > > > > >> > > > > > > backends
> > > > > > >> > > > > > > > > > can't
> > > > > > >> > > > > > > > > > > > >> >> participate.
> > > > > > >> > > > > > > > > > > > >> >> >>
> > > > > > >> > > > > > > > > > > > >> >> >> Stepping back, I think the
> > metrics
> > > > > > >> subsystem
> > > > > > >> > is
> > > > > > >> > > > > > growing
> > > > > > >> > > > > > > > into
> > > > > > >> > > > > > > > > > > > >> something
> > > > > > >> > > > > > > > > > > > >> >> >> real (write + read + REST API +
> > > AuthZ +
> > > > > > >> > > > pagination)
> > > > > > >> > > > > > *but
> > > > > > >> > > > > > > > the
> > > > > > >> > > > > > > > > > > > >> >> persistence
> > > > > > >> > > > > > > > > > > > >> >> >> side is split across two layers
> > in
> > > a
> > > > > way
> > > > > > >> > that's
> > > > > > >> > > > hard
> > > > > > >> > > > > > to
> > > > > > >> > > > > > > > > > > extend*. I
> > > > > > >> > > > > > > > > > > > >> put
> > > > > > >> > > > > > > > > > > > >> >> >> together two diagrams to show
> > what
> > > I
> > > > > mean
> > > > > > >> (my
> > > > > > >> > > best
> > > > > > >> > > > > > > > effort).
> > > > > > >> > > > > > > > > > > > >> >> >>
> > > > > > >> > > > > > > > > > > > >> >> >> *Current state* (Diagram 1):
> > three
> > > > > > >> interfaces
> > > > > > >> > at
> > > > > > >> > > > > three
> > > > > > >> > > > > > > > > > different
> > > > > > >> > > > > > > > > > > > >> >> levels.
> > > > > > >> > > > > > > > > > > > >> >> >> The engine-facing SPI
> > > > > > >> (PolarisMetricsReporter)
> > > > > > >> > > is
> > > > > > >> > > > > > clean.
> > > > > > >> > > > > > > > But
> > > > > > >> > > > > > > > > > > > >> >> >> PolarisMetricsManager on
> > > > > MetaStoreManager
> > > > > > >> is a
> > > > > > >> > > > > > > passthrough
> > > > > > >> > > > > > > > > to
> > > > > > >> > > > > > > > > > > > >> >> >> MetricsPersistence on
> > > BasePersistence.
> > > > > > The
> > > > > > >> > @Beta
> > > > > > >> > > > > > > > annotation
> > > > > > >> > > > > > > > > > and
> > > > > > >> > > > > > > > > > > > SPI
> > > > > > >> > > > > > > > > > > > >> >> javadoc
> > > > > > >> > > > > > > > > > > > >> >> >> are on the BasePersistence
> layer,
> > > while
> > > > > > the
> > > > > > >> > > actual
> > > > > > >> > > > > > > > extension
> > > > > > >> > > > > > > > > > > > points
> > > > > > >> > > > > > > > > > > > >> >> >> (PolarisMetricsReporter,
> > > > > > >> > PolarisMetricsManager)
> > > > > > >> > > > > carry
> > > > > > >> > > > > > no
> > > > > > >> > > > > > > > > > > stability
> > > > > > >> > > > > > > > > > > > >> >> >> annotation. The write path goes
> > > through
> > > > > > the
> > > > > > >> > > > > > > > MetaStoreManager
> > > > > > >> > > > > > > > > > > > layer,
> > > > > > >> > > > > > > > > > > > >> the
> > > > > > >> > > > > > > > > > > > >> >> >> read path doesn't.
> > > > > > >> > > > > > > > > > > > >> >> >>
> > > > > > >> > > > > > > > > > > > >> >> >> *What I envision* (Diagram 2):
> > two
> > > SPIs
> > > > > > at
> > > > > > >> two
> > > > > > >> > > > > levels.
> > > > > > >> > > > > > > > > > > > >> >> >> PolarisMetricsReporter stays as
> > the
> > > > > > >> > > engine-facing
> > > > > > >> > > > > SPI.
> > > > > > >> > > > > > > > > > > > >> >> >> PolarisMetricsManager becomes
> the
> > > > > > >> > backend-facing
> > > > > > >> > > > SPI
> > > > > > >> > > > > > > with
> > > > > > >> > > > > > > > > both
> > > > > > >> > > > > > > > > > > > write
> > > > > > >> > > > > > > > > > > > >> >> and
> > > > > > >> > > > > > > > > > > > >> >> >> read methods at the
> > > MetaStoreManager
> > > > > > level,
> > > > > > >> > > where
> > > > > > >> > > > > any
> > > > > > >> > > > > > > > > backend
> > > > > > >> > > > > > > > > > > > (JDBC,
> > > > > > >> > > > > > > > > > > > >> >> NoSQL,
> > > > > > >> > > > > > > > > > > > >> >> >> remote) can implement them.
> > > > > > >> MetricsPersistence
> > > > > > >> > > on
> > > > > > >> > > > > > > > > > > BasePersistence
> > > > > > >> > > > > > > > > > > > >> goes
> > > > > > >> > > > > > > > > > > > >> >> >> away. Where metrics actually
> land
> > > is an
> > > > > > >> > > > > implementation
> > > > > > >> > > > > > > > > detail,
> > > > > > >> > > > > > > > > > > > not a
> > > > > > >> > > > > > > > > > > > >> >> core
> > > > > > >> > > > > > > > > > > > >> >> >> interface.
> > > > > > >> > > > > > > > > > > > >> >> >>
> > > > > > >> > > > > > > > > > > > >> >> >> *Minor naming thing*:
> > > > > > >> PolarisMetricsReporter
> > > > > > >> > is
> > > > > > >> > > > > > broader
> > > > > > >> > > > > > > > than
> > > > > > >> > > > > > > > > > > what
> > > > > > >> > > > > > > > > > > > it
> > > > > > >> > > > > > > > > > > > >> >> >> actually handles. It only
> accepts
> > > > > Iceberg
> > > > > > >> REST
> > > > > > >> > > > > Catalog
> > > > > > >> > > > > > > > > metrics
> > > > > > >> > > > > > > > > > > > >> >> (ScanReport,
> > > > > > >> > > > > > > > > > > > >> >> >> CommitReport via
> MetricsReport).
> > > > > Generic
> > > > > > >> table
> > > > > > >> > > > > metrics
> > > > > > >> > > > > > > or
> > > > > > >> > > > > > > > > > > > >> operational
> > > > > > >> > > > > > > > > > > > >> >> >> metrics aren't in scope. Not
> > > blocking,
> > > > > > but
> > > > > > >> > worth
> > > > > > >> > > > > > noting
> > > > > > >> > > > > > > if
> > > > > > >> > > > > > > > > the
> > > > > > >> > > > > > > > > > > > >> metrics
> > > > > > >> > > > > > > > > > > > >> >> >> surface expands.
> > > > > > >> > > > > > > > > > > > >> >> >>
> > > > > > >> > > > > > > > > > > > >> >> >> *Rough sketch of how to get
> > there*:
> > > > > > >> > > > > > > > > > > > >> >> >>  1.  Add read methods to
> > > > > > >> PolarisMetricsManager
> > > > > > >> > > > > > > > > > (listScanReports,
> > > > > > >> > > > > > > > > > > > >> >> >> listCommitReports) with default
> > > no-op,
> > > > > > >> same as
> > > > > > >> > > the
> > > > > > >> > > > > > > > existing
> > > > > > >> > > > > > > > > > > write
> > > > > > >> > > > > > > > > > > > >> >> methods.
> > > > > > >> > > > > > > > > > > > >> >> >> (Probably make
> > > PolarisMetricsManager
> > > > > more
> > > > > > >> > > explicit
> > > > > > >> > > > > on
> > > > > > >> > > > > > > > being
> > > > > > >> > > > > > > > > > > > Iceberg
> > > > > > >> > > > > > > > > > > > >> >> >> specific like package name or
> > class
> > > > > name
> > > > > > >> etc.)
> > > > > > >> > > > > > > > > > > > >> >> >>  2.  Wire MetricsReportsService
> > > through
> > > > > > >> > > > > > MetaStoreManager
> > > > > > >> > > > > > > > > > instead
> > > > > > >> > > > > > > > > > > > of
> > > > > > >> > > > > > > > > > > > >> >> >> callContext.getMetaStore().
> > > > > > >> > > > > > > > > > > > >> >> >>  3.  Extract metrics
> persistence
> > > from
> > > > > > >> > > > > > > > > JdbcBasePersistenceImpl
> > > > > > >> > > > > > > > > > > into
> > > > > > >> > > > > > > > > > > > >> its
> > > > > > >> > > > > > > > > > > > >> >> >> own class. That file carries ~7
> > > > > > >> > > responsibilities,
> > > > > > >> > > > > > > metrics
> > > > > > >> > > > > > > > > > being
> > > > > > >> > > > > > > > > > > > one
> > > > > > >> > > > > > > > > > > > >> of
> > > > > > >> > > > > > > > > > > > >> >> them.
> > > > > > >> > > > > > > > > > > > >> >> >>  4.  Remove MetricsPersistence
> > from
> > > > > > >> > > > BasePersistence.
> > > > > > >> > > > > > > > > > > > >> >> >>
> > > > > > >> > > > > > > > > > > > >> >> >> *None of this needs to happen
> in
> > > #4115.
> > > > > > >> But if
> > > > > > >> > > the
> > > > > > >> > > > > > > > direction
> > > > > > >> > > > > > > > > > > makes
> > > > > > >> > > > > > > > > > > > >> >> sense,
> > > > > > >> > > > > > > > > > > > >> >> >> it would be good to align
> before
> > > the
> > > > > > >> metrics
> > > > > > >> > > > surface
> > > > > > >> > > > > > > grows
> > > > > > >> > > > > > > > > > > > further.
> > > > > > >> > > > > > > > > > > > >> >> Curious
> > > > > > >> > > > > > > > > > > > >> >> >> what others think.*
> > > > > > >> > > > > > > > > > > > >> >> >>
> > > > > > >> > > > > > > > > > > > >> >> >> *My mental model note*: Level 1
> > > > > > >> > > MetaStoreManager;
> > > > > > >> > > > > > level
> > > > > > >> > > > > > > 2
> > > > > > >> > > > > > > > > > > > >> transactional
> > > > > > >> > > > > > > > > > > > >> >> >> persistence; level 3 base
> > > persistence
> > > > > > >> > > > > > > > > > > > >> >> >>
> > > > > > >> > > > > > > > > > > > >> >> >> Diagram 1
> > > > > > >> > > > > > > > > > > > >> >> >> <
> > > > > > >> > > > > > > > > > > > >> >>
> > > > > > >> > > > > > > > > > > > >>
> > > > > > >> > > > > > > > > > > >
> > > > > > >> > > > > > > > > > >
> > > > > > >> > > > > > > > > >
> > > > > > >> > > > > > > > >
> > > > > > >> > > > > > > >
> > > > > > >> > > > > > >
> > > > > > >> > > > > >
> > > > > > >> > > > >
> > > > > > >> > > >
> > > > > > >> > >
> > > > > > >> >
> > > > > > >>
> > > > > >
> > > > >
> > >
> >
> https://urldefense.com/v3/__https://www.plantuml.com/plantuml/uml/bLHDR-Cs4BthLmpIYupw0zbkKQ1r3M-S7Bp8xhhM7WCOb3IM65EaGD9EX2RzxHrHb4CxRelwa4YSDu_lpOVcnZ9jzvM8BBS2uGjQpJC3dtHMSekPtMk44IpsMgEqa5XcCOhCZikQQLP1pR8TAp2n3ILhmZDP20m0fcIvUkAoW2qJXd9z1bpToO9BX3WXu0ucy5rpgGPNm0nW5_epUWtm2Ue3pn3kMOFQmKntGZW0BYtgBSi8k5A2QMwybJNMIbFiGSR9QZc4nUqIvikStF0jHprua5C-amge42aNt3R0f5JaaoivdV2Pkqbx4hee4ymOkBh5BTiB-_uIeGeo8zL8rPsPl4DktdEiK1jkB1NdZCRbrSTecDe_mlHbF0wvBmCkaOH5_S8a_TTTKI6-nmCAkEw4LpxsZ-LbYLKQFKMNOgf_wuM7_bV9gOer5SYMMksBSWXFcbi49KNZXNLicwfe3TETC7gPdPqI7uBcHMb1RSzYq34c6PDUM9mn8HRsUTZEiDBve3NjVZumBj0U7SS37mGO7vcwtiK-_pU7U7L_f-digo9YbhSwIfMRwIITKGXbxdIUTCGF1SeCJxloKsU-3k9ddRbX1eDq1q_fx1JbBGT0glVyXimDuP4TQ5qpCAmnGEj2s_6n5mtn1z-97-63itFQZLPO1Ev2tu_WF7Ju-VPc0Skg5bYXxBhkY1xpD7EM_7fyflSpIsqMgVth5xhVr4eQxWQ8enaSAJQSG16yFSDuJ798rrcXr_3n-lfdk7icQjEBmFujL7AodiP_Y4Z7-YxvtZNs4zMgpNTl6tF8sglyPsmqchrjvQ-m-aP94r-TwCA2Ka8upPJZwtvSpoYCXkYMZU2NXvRMBfq9P3i3Le4VAZUAlUZ_oPKsxPgY0Q_BSKLkyr9bhQhQrJjo_x3TPlIB0DPjnMfcIoYP0QaYw1a0fTKDr8fB6ntNuvmoL1ZGkXa69Njh43zf9GiGxHQrA_jDYWRSzF5--WmTVrN97_Sm8LbLUy_lGBmLanJjFkDlGkRqjA_4tm00__;!!Iz9xO38YGHZK!4rnlVIqBQQQ6xpGe1PMV58edOS_cGrKoqdfhw4qSp_JQCavvEjaxXk0OwGPnxyEcO60AzKQK_XDtp7H3DdkFXcrXuTBISHg$
> > > > > > >> > > > > > > > > > > > >> >> >
> > > > > > >> > > > > > > > > > > > >> >> >> :
> > > > > > >> > > > > > > > > > > > >> >> >>
> > > > > > >> > > > > > > > > > > > >> >> >> [image: image.png]
> > > > > > >> > > > > > > > > > > > >> >> >>
> > > > > > >> > > > > > > > > > > > >> >> >> Diagram 2
> > > > > > >> > > > > > > > > > > > >> >> >> <
> > > > > > >> > > > > > > > > > > > >> >>
> > > > > > >> > > > > > > > > > > > >>
> > > > > > >> > > > > > > > > > > >
> > > > > > >> > > > > > > > > > >
> > > > > > >> > > > > > > > > >
> > > > > > >> > > > > > > > >
> > > > > > >> > > > > > > >
> > > > > > >> > > > > > >
> > > > > > >> > > > > >
> > > > > > >> > > > >
> > > > > > >> > > >
> > > > > > >> > >
> > > > > > >> >
> > > > > > >>
> > > > > >
> > > > >
> > >
> >
> https://urldefense.com/v3/__https://www.plantuml.com/plantuml/uml/VLLDR-8m4BtdLupO2sWBLVU8AaGB7AXAbssGzb896SS9RXqxjKqBwkv_tt7iV43fdaZYDpFlpRmnOsE9jhjSH9PRmM31hERKm8scMsuPjJlDe0yheZDc8RR4iYWoBrmMH9CS2a9VICPYUy1OZN0YCy5Q0BCbYNhdCeEK28En8G8wCvbnoQ0R8_05Bc6bkLIz3X03p1zzH7zR-9ZfDquPt9C3qoNCX2yV4G2NbkcKu5jdgGJHt0GbZwnG6i-UP3TUpk5gM6Ldqke350eZUqzoCft3U9xWHvxoa5-7K4nF1J46EbEMafsmdrCBbQ44gVggy18IZrn_ph5asd1ZiIKdQSgueZvjXrQFSFrdC3YN-nXmBacxbGiYyLVxLaBtdhqn0LSzdBDhqQtQoOJeGyad3z0lUqnYgpGB6Ns8oVyta00Dy_WnX0tIOZ8v6SYxHll1TrH6aejAik-mh-AphVFCwSUQqFypElag5QRGFDjQKEd96K1P8QP41c9TzA_IIQyvdAWyv_RSiS3skb0_EzDDkK2v5xWF6MiGFlvhpFLcD2Dq2pml14gaF67eQkmd8gulDoC4kSOu6KVpkvlUJg1RTbWISU40RdBUUS_9XfRZ2dwxm_SW8LYFISgm_MnlDQ6M9P1gbKEc4X-2pH_FvJCkCqm9pbVjD6LrwdLeOrDWfOaqc8Wh9BE85oNKxkNQ6o4yGRy_Eae0G_G8tZv81d3bHDB23WOdisohVr3nh_j6lbSjbNaLRTc8UgtPbAU1J_tygOfZX9DWEJeHDvYx-qmSi5FgNLPZwHrHcUsncGQ5-skhUclpE5fo4ounpFauYrUbkU6ccfnxMvitwag4IyerhTxj8In_Oj1bDO4pQru674loYrGlULHLEGCjwJJ8gDoVZR8MxO4BT3IzRvIcAQKezC6xpziGnTyImrfEGyJI_OcKfgtxIvnTqFEMS17L9Z-jsARN5FmTheP7HtSdtOMT0B4GY2FYHXxgQmMtj2bRqiLFGapiVe1_QVKDrkqXcm83aFEXnMYCZ-xlyHy__;!!Iz9xO38YGHZK!4rnlVIqBQQQ6xpGe1PMV58edOS_cGrKoqdfhw4qSp_JQCavvEjaxXk0OwGPnxyEcO60AzKQK_XDtp7H3DdkFXcrXSpmqcNk$
> > > > > > >> > > > > > > > > > > > >> >> >
> > > > > > >> > > > > > > > > > > > >> >> >> :
> > > > > > >> > > > > > > > > > > > >> >> >> [image: image.png]
> > > > > > >> > > > > > > > > > > > >> >> >>
> > > > > > >> > > > > > > > > > > > >> >> >>  -ej
> > > > > > >> > > > > > > > > > > > >> >> >>
> > > > > > >> > > > > > > > > > > > >> >> >> On Wed, Apr 15, 2026 at 8:22 AM
> > > Dmitri
> > > > > > >> > > > Bourlatchkov
> > > > > > >> > > > > <
> > > > > > >> > > > > > > > > > > > >> dimas@apache.org>
> > > > > > >> > > > > > > > > > > > >> >> >> wrote:
> > > > > > >> > > > > > > > > > > > >> >> >>
> > > > > > >> > > > > > > > > > > > >> >> >>> Hi All,
> > > > > > >> > > > > > > > > > > > >> >> >>>
> > > > > > >> > > > > > > > > > > > >> >> >>> Heads up: The current state of
> > PR
> > > > > [4115]
> > > > > > >> > looks
> > > > > > >> > > > > pretty
> > > > > > >> > > > > > > > solid
> > > > > > >> > > > > > > > > > to
> > > > > > >> > > > > > > > > > > > me.
> > > > > > >> > > > > > > > > > > > >> I
> > > > > > >> > > > > > > > > > > > >> >> >>> believe this PR is
> approaching a
> > > > > > mergeable
> > > > > > >> > > > > condition.
> > > > > > >> > > > > > > > > > > > >> >> >>>
> > > > > > >> > > > > > > > > > > > >> >> >>> Please post your reviews if
> you
> > > have
> > > > > any
> > > > > > >> > > > comments.
> > > > > > >> > > > > > > > > > > > >> >> >>>
> > > > > > >> > > > > > > > > > > > >> >> >>> [4115]
> > > > > > >> > > >
> >
> https://urldefense.com/v3/__https://github.com/apache/polaris/pull/4115__;!!Iz9xO38YGHZK!4rnlVIqBQQQ6xpGe1PMV58edOS_cGrKoqdfhw4qSp_JQCavvEjaxXk0OwGPnxyEcO60AzKQK_XDtp7H3DdkFXcrXtFRscKs$
> > > > > > >> > > > > > > > > > > > >> >> >>>
> > > > > > >> > > > > > > > > > > > >> >> >>> Thanks,
> > > > > > >> > > > > > > > > > > > >> >> >>> Dmitri.
> > > > > > >> > > > > > > > > > > > >> >> >>>
> > > > > > >> > > > > > > > > > > > >> >> >>> On Tue, Mar 3, 2026 at 3:29 PM
> > > Anand
> > > > > > Kumar
> > > > > > >> > > > Sankaran
> > > > > > >> > > > > > via
> > > > > > >> > > > > > > > > dev <
> > > > > > >> > > > > > > > > > > > >> >> >>> dev@polaris.apache.org>
> wrote:
> > > > > > >> > > > > > > > > > > > >> >> >>>
> > > > > > >> > > > > > > > > > > > >> >> >>> > Hi Yufei and Dmitri,
> > > > > > >> > > > > > > > > > > > >> >> >>> >
> > > > > > >> > > > > > > > > > > > >> >> >>> > Here is a proposal for the
> > REST
> > > > > > >> endpoints
> > > > > > >> > for
> > > > > > >> > > > > > metrics
> > > > > > >> > > > > > > > and
> > > > > > >> > > > > > > > > > > > events.
> > > > > > >> > > > > > > > > > > > >> >> >>> >
> > > > > > >> > > > > > > > > > > > >> >> >>> >
> > > > > > >> > > > > >
> >
> https://urldefense.com/v3/__https://github.com/apache/polaris/pull/3924/changes__;!!Iz9xO38YGHZK!4rnlVIqBQQQ6xpGe1PMV58edOS_cGrKoqdfhw4qSp_JQCavvEjaxXk0OwGPnxyEcO60AzKQK_XDtp7H3DdkFXcrXcjoPhSs$
> > > > > > >> > > > > > > > > > > > >> >> >>> >
> > > > > > >> > > > > > > > > > > > >> >> >>> > I did not see any precursors
> > for
> > > > > > >> raising a
> > > > > > >> > PR
> > > > > > >> > > > for
> > > > > > >> > > > > > > > > > proposals,
> > > > > > >> > > > > > > > > > > so
> > > > > > >> > > > > > > > > > > > >> >> trying
> > > > > > >> > > > > > > > > > > > >> >> >>> > this.  Please let me know
> what
> > > you
> > > > > > >> think.
> > > > > > >> > > > > > > > > > > > >> >> >>> >
> > > > > > >> > > > > > > > > > > > >> >> >>> > -
> > > > > > >> > > > > > > > > > > > >> >> >>> > Anand
> > > > > > >> > > > > > > > > > > > >> >> >>> >
> > > > > > >> > > > > > > > > > > > >> >> >>> > From: Anand Kumar Sankaran <
> > > > > > >> > > > > > > anand.sankaran@workday.com
> > > > > > >> > > > > > > > >
> > > > > > >> > > > > > > > > > > > >> >> >>> > Date: Monday, March 2, 2026
> at
> > > > > > 10:25 AM
> > > > > > >> > > > > > > > > > > > >> >> >>> > To: dev@polaris.apache.org
> <
> > > > > > >> > > > > dev@polaris.apache.org
> > > > > > >> > > > > > >
> > > > > > >> > > > > > > > > > > > >> >> >>> > Subject: Re: Polaris
> Telemetry
> > > and
> > > > > > Audit
> > > > > > >> > > Trail
> > > > > > >> > > > > > > > > > > > >> >> >>> >
> > > > > > >> > > > > > > > > > > > >> >> >>> > About the REST API, based on
> > my
> > > use
> > > > > > >> cases:
> > > > > > >> > > > > > > > > > > > >> >> >>> >
> > > > > > >> > > > > > > > > > > > >> >> >>> >
> > > > > > >> > > > > > > > > > > > >> >> >>> >   1.
> > > > > > >> > > > > > > > > > > > >> >> >>> > I want to be able to query
> > > commit
> > > > > > >> metrics
> > > > > > >> > to
> > > > > > >> > > > > track
> > > > > > >> > > > > > > > files
> > > > > > >> > > > > > > > > > > added
> > > > > > >> > > > > > > > > > > > /
> > > > > > >> > > > > > > > > > > > >> >> >>> removed
> > > > > > >> > > > > > > > > > > > >> >> >>> > per commit, along with
> record
> > > > > counts.
> > > > > > >> The
> > > > > > >> > > > > ingestion
> > > > > > >> > > > > > > > > > pipeline
> > > > > > >> > > > > > > > > > > > that
> > > > > > >> > > > > > > > > > > > >> >> >>> writes
> > > > > > >> > > > > > > > > > > > >> >> >>> > this data is owned by us and
> > we
> > > are
> > > > > > >> > > guaranteed
> > > > > > >> > > > to
> > > > > > >> > > > > > > write
> > > > > > >> > > > > > > > > > this
> > > > > > >> > > > > > > > > > > > >> >> >>> information
> > > > > > >> > > > > > > > > > > > >> >> >>> > for each write.
> > > > > > >> > > > > > > > > > > > >> >> >>> >   2.
> > > > > > >> > > > > > > > > > > > >> >> >>> > I want to be able to query
> > scan
> > > > > > metrics
> > > > > > >> for
> > > > > > >> > > > > read. I
> > > > > > >> > > > > > > > > > > understand
> > > > > > >> > > > > > > > > > > > >> >> clients
> > > > > > >> > > > > > > > > > > > >> >> >>> do
> > > > > > >> > > > > > > > > > > > >> >> >>> > not fulfill this
> requirement.
> > > > > > >> > > > > > > > > > > > >> >> >>> >   3.
> > > > > > >> > > > > > > > > > > > >> >> >>> > I want to be able to query
> the
> > > > > events
> > > > > > >> table
> > > > > > >> > > > > (events
> > > > > > >> > > > > > > are
> > > > > > >> > > > > > > > > > > > >> persisted) -
> > > > > > >> > > > > > > > > > > > >> >> >>> this
> > > > > > >> > > > > > > > > > > > >> >> >>> > may supersede #2, I am not
> > sure
> > > yet.
> > > > > > >> > > > > > > > > > > > >> >> >>> >
> > > > > > >> > > > > > > > > > > > >> >> >>> > All this information is in
> the
> > > JDBC
> > > > > > >> based
> > > > > > >> > > > > > persistence
> > > > > > >> > > > > > > > > model
> > > > > > >> > > > > > > > > > > and
> > > > > > >> > > > > > > > > > > > >> is
> > > > > > >> > > > > > > > > > > > >> >> >>> > persisted in the metastore.
> I
> > > > > > currently
> > > > > > >> > don’t
> > > > > > >> > > > > have
> > > > > > >> > > > > > a
> > > > > > >> > > > > > > > need
> > > > > > >> > > > > > > > > > to
> > > > > > >> > > > > > > > > > > > >> query
> > > > > > >> > > > > > > > > > > > >> >> >>> > prometheus or open
> telemetry.
> > I
> > > do
> > > > > > >> publish
> > > > > > >> > > some
> > > > > > >> > > > > > > events
> > > > > > >> > > > > > > > to
> > > > > > >> > > > > > > > > > > > >> Prometheus
> > > > > > >> > > > > > > > > > > > >> >> >>> and
> > > > > > >> > > > > > > > > > > > >> >> >>> > they are forwarded to our
> > > dashboards
> > > > > > >> > > elsewhere.
> > > > > > >> > > > > > > > > > > > >> >> >>> >
> > > > > > >> > > > > > > > > > > > >> >> >>> > About the CLI utilities, I
> > > meant the
> > > > > > >> admin
> > > > > > >> > > user
> > > > > > >> > > > > > > > > utilities.
> > > > > > >> > > > > > > > > > In
> > > > > > >> > > > > > > > > > > > >> one of
> > > > > > >> > > > > > > > > > > > >> >> >>> the
> > > > > > >> > > > > > > > > > > > >> >> >>> > earliest drafts of my
> > proposal,
> > > > > > Prashant
> > > > > > >> > > > > mentioned
> > > > > > >> > > > > > > that
> > > > > > >> > > > > > > > > the
> > > > > > >> > > > > > > > > > > > >> metrics
> > > > > > >> > > > > > > > > > > > >> >> >>> tables
> > > > > > >> > > > > > > > > > > > >> >> >>> > can grow indefinitely and
> > that a
> > > > > > similar
> > > > > > >> > > > problem
> > > > > > >> > > > > > > exists
> > > > > > >> > > > > > > > > > with
> > > > > > >> > > > > > > > > > > > the
> > > > > > >> > > > > > > > > > > > >> >> events
> > > > > > >> > > > > > > > > > > > >> >> >>> > table as well. We discussed
> > that
> > > > > > >> cleaning
> > > > > > >> > up
> > > > > > >> > > of
> > > > > > >> > > > > old
> > > > > > >> > > > > > > > > records
> > > > > > >> > > > > > > > > > > > from
> > > > > > >> > > > > > > > > > > > >> >> both
> > > > > > >> > > > > > > > > > > > >> >> >>> > metrics tables and events
> > > tables can
> > > > > > be
> > > > > > >> > done
> > > > > > >> > > > via
> > > > > > >> > > > > a
> > > > > > >> > > > > > > CLI
> > > > > > >> > > > > > > > > > > utility.
> > > > > > >> > > > > > > > > > > > >> >> >>> >
> > > > > > >> > > > > > > > > > > > >> >> >>> > I see that Yufei has covered
> > the
> > > > > > >> discussion
> > > > > > >> > > > about
> > > > > > >> > > > > > > > > > > datasources.
> > > > > > >> > > > > > > > > > > > >> >> >>> >
> > > > > > >> > > > > > > > > > > > >> >> >>> > -
> > > > > > >> > > > > > > > > > > > >> >> >>> > Anand
> > > > > > >> > > > > > > > > > > > >> >> >>> >
> > > > > > >> > > > > > > > > > > > >> >> >>> >
> > > > > > >> > > > > > > > > > > > >> >> >>> >
> > > > > > >> > > > > > > > > > > > >> >> >>> > From: Yufei Gu <
> > > > > flyrain000@gmail.com>
> > > > > > >> > > > > > > > > > > > >> >> >>> > Date: Friday, February 27,
> > 2026
> > > at
> > > > > > >> 9:54 PM
> > > > > > >> > > > > > > > > > > > >> >> >>> > To: dev@polaris.apache.org
> <
> > > > > > >> > > > > dev@polaris.apache.org
> > > > > > >> > > > > > >
> > > > > > >> > > > > > > > > > > > >> >> >>> > Subject: Re: Polaris
> Telemetry
> > > and
> > > > > > Audit
> > > > > > >> > > Trail
> > > > > > >> > > > > > > > > > > > >> >> >>> >
> > > > > > >> > > > > > > > > > > > >> >> >>> > This Message Is From an
> > External
> > > > > > Sender
> > > > > > >> > > > > > > > > > > > >> >> >>> > This message came from
> outside
> > > your
> > > > > > >> > > > organization.
> > > > > > >> > > > > > > > > > > > >> >> >>> > Report Suspicious<
> > > > > > >> > > > > > > > > > > > >> >> >>> >
> > > > > > >> > > > > > > > > > > > >> >> >>>
> > > > > > >> > > > > > > > > > > > >> >>
> > > > > > >> > > > > > > > > > > > >>
> > > > > > >> > > > > > > > > > > >
> > > > > > >> > > > > > > > > > >
> > > > > > >> > > > > > > > > >
> > > > > > >> > > > > > > > >
> > > > > > >> > > > > > > >
> > > > > > >> > > > > > >
> > > > > > >> > > > > >
> > > > > > >> > > > >
> > > > > > >> > > >
> > > > > > >> > >
> > > > > > >> >
> > > > > > >>
> > > > > >
> > > > >
> > >
> >
> https://us-phishalarm-ewt.proofpoint.com/EWT/v1/Iz9xO38YGHZK!YhNDZABkHi1B699ote2uMwpOZw8i0QMCGO2Szc-HshuABGhGvwPJcymE6G2oUUxtS8xDkSrtGTPm_I3QnVDHoLMk50m9v8z_nZKTkd-bnVUbreF1u0WnfV_X5eYevZl_$
> > > > > > >> > > > > > > > > > > > >> >> >>> > >
> > > > > > >> > > > > > > > > > > > >> >> >>> >
> > > > > > >> > > > > > > > > > > > >> >> >>> >
> > > > > > >> > > > > > > > > > > > >> >> >>> > As I mentioned in
> > > > > > >> > > > > > > > > > > > >> >> >>> >
> > > > > > >> > > > > > > > > > > > >> >> >>>
> > > > > > >> > > > > > > > > > > > >> >>
> > > > > > >> > > > > > > > > > > > >>
> > > > > > >> > > > > > > > > > > >
> > > > > > >> > > > > > > > > > >
> > > > > > >> > > > > > > > > >
> > > > > > >> > > > > > > > >
> > > > > > >> > > > > > > >
> > > > > > >> > > > > > >
> > > > > > >> > > > > >
> > > > > > >> > > > >
> > > > > > >> > > >
> > > > > > >> > >
> > > > > > >> >
> > > > > > >>
> > > > > >
> > > > >
> > >
> >
> https://urldefense.com/v3/__https://github.com/apache/polaris/issues/3890__;!!Iz9xO38YGHZK!5EuyFFkk3vhRWVIRvQAWBSQfpJkTMA9HxugzDwXmN0LPPqhEFxYkFRGVhtb8AqUwXtDh2OplcMnbMDHKOxrvDU0$
> > >> > > >> > > > > > > > > > > > >> >> >>> ,
> > > > > > >> > > > > > > > > > > > >> >> >>> > supporting
> > > > > > >> > > > > > > > > > > > >> >> >>> > multiple data sources is
> not a
> > > > > trivial
> > > > > > >> > > change.
> > > > > > >> > > > I
> > > > > > >> > > > > > > would
> > > > > > >> > > > > > > > > > > strongly
> > > > > > >> > > > > > > > > > > > >> >> >>> recommend
> > > > > > >> > > > > > > > > > > > >> >> >>> > starting with a design
> > document
> > > to
> > > > > > >> > carefully
> > > > > > >> > > > > > evaluate
> > > > > > >> > > > > > > > the
> > > > > > >> > > > > > > > > > > > >> >> architectural
> > > > > > >> > > > > > > > > > > > >> >> >>> > implications and long term
> > > impact.
> > > > > > >> > > > > > > > > > > > >> >> >>> >
> > > > > > >> > > > > > > > > > > > >> >> >>> > A REST endpoint to query
> > metrics
> > > > > seems
> > > > > > >> > > > reasonable
> > > > > > >> > > > > > > given
> > > > > > >> > > > > > > > > the
> > > > > > >> > > > > > > > > > > > >> current
> > > > > > >> > > > > > > > > > > > >> >> >>> JDBC
> > > > > > >> > > > > > > > > > > > >> >> >>> > based persistence model.
> That
> > > said,
> > > > > we
> > > > > > >> may
> > > > > > >> > > also
> > > > > > >> > > > > > > > consider
> > > > > > >> > > > > > > > > > > > >> alternative
> > > > > > >> > > > > > > > > > > > >> >> >>> > storage models. For example,
> > if
> > > we
> > > > > > later
> > > > > > >> > > adopt
> > > > > > >> > > > a
> > > > > > >> > > > > > time
> > > > > > >> > > > > > > > > > series
> > > > > > >> > > > > > > > > > > > >> system
> > > > > > >> > > > > > > > > > > > >> >> >>> such as
> > > > > > >> > > > > > > > > > > > >> >> >>> > Prometheus to store metrics,
> > the
> > > > > query
> > > > > > >> > model
> > > > > > >> > > > and
> > > > > > >> > > > > > > access
> > > > > > >> > > > > > > > > > > > patterns
> > > > > > >> > > > > > > > > > > > >> >> would
> > > > > > >> > > > > > > > > > > > >> >> >>> be
> > > > > > >> > > > > > > > > > > > >> >> >>> > fundamentally different.
> > > Designing
> > > > > the
> > > > > > >> REST
> > > > > > >> > > API
> > > > > > >> > > > > > > without
> > > > > > >> > > > > > > > > > > > >> considering
> > > > > > >> > > > > > > > > > > > >> >> >>> these
> > > > > > >> > > > > > > > > > > > >> >> >>> > potential evolutions may
> limit
> > > > > > >> flexibility.
> > > > > > >> > > I'd
> > > > > > >> > > > > > > suggest
> > > > > > >> > > > > > > > > to
> > > > > > >> > > > > > > > > > > > start
> > > > > > >> > > > > > > > > > > > >> >> with
> > > > > > >> > > > > > > > > > > > >> >> >>> the
> > > > > > >> > > > > > > > > > > > >> >> >>> > use case.
> > > > > > >> > > > > > > > > > > > >> >> >>> >
> > > > > > >> > > > > > > > > > > > >> >> >>> > Yufei
> > > > > > >> > > > > > > > > > > > >> >> >>> >
> > > > > > >> > > > > > > > > > > > >> >> >>> >
> > > > > > >> > > > > > > > > > > > >> >> >>> > On Fri, Feb 27, 2026 at
> > 3:42 PM
> > > > > Dmitri
> > > > > > >> > > > > > Bourlatchkov <
> > > > > > >> > > > > > > > > > > > >> >> dimas@apache.org>
> > > > > > >> > > > > > > > > > > > >> >> >>> > wrote:
> > > > > > >> > > > > > > > > > > > >> >> >>> >
> > > > > > >> > > > > > > > > > > > >> >> >>> > > Hi Anand,
> > > > > > >> > > > > > > > > > > > >> >> >>> > >
> > > > > > >> > > > > > > > > > > > >> >> >>> > > Sharing my view... subject
> > to
> > > > > > >> discussion:
> > > > > > >> > > > > > > > > > > > >> >> >>> > >
> > > > > > >> > > > > > > > > > > > >> >> >>> > > 1. Adding non-IRC REST API
> > to
> > > > > > Polaris
> > > > > > >> is
> > > > > > >> > > > > > perfectly
> > > > > > >> > > > > > > > > fine.
> > > > > > >> > > > > > > > > > > > >> >> >>> > >
> > > > > > >> > > > > > > > > > > > >> >> >>> > > Figuring out specific
> > endpoint
> > > > > URIs
> > > > > > >> and
> > > > > > >> > > > > payloads
> > > > > > >> > > > > > > > might
> > > > > > >> > > > > > > > > > > > require
> > > > > > >> > > > > > > > > > > > >> a
> > > > > > >> > > > > > > > > > > > >> >> few
> > > > > > >> > > > > > > > > > > > >> >> >>> > > roundtrips, so opening a
> > > separate
> > > > > > >> thread
> > > > > > >> > > for
> > > > > > >> > > > > that
> > > > > > >> > > > > > > > might
> > > > > > >> > > > > > > > > > be
> > > > > > >> > > > > > > > > > > > >> best.
> > > > > > >> > > > > > > > > > > > >> >> >>> > > Contributors commonly
> create
> > > > > Google
> > > > > > >> Docs
> > > > > > >> > > for
> > > > > > >> > > > > new
> > > > > > >> > > > > > > API
> > > > > > >> > > > > > > > > > > > proposals
> > > > > > >> > > > > > > > > > > > >> too
> > > > > > >> > > > > > > > > > > > >> >> >>> (they
> > > > > > >> > > > > > > > > > > > >> >> >>> > > fairly easy to update as
> the
> > > email
> > > > > > >> > > discussion
> > > > > > >> > > > > > > > > > progresses).
> > > > > > >> > > > > > > > > > > > >> >> >>> > >
> > > > > > >> > > > > > > > > > > > >> >> >>> > > There was a suggestion to
> > try
> > > > > > Markdown
> > > > > > >> > > (with
> > > > > > >> > > > > PRs)
> > > > > > >> > > > > > > for
> > > > > > >> > > > > > > > > > > > proposals
> > > > > > >> > > > > > > > > > > > >> >> [1]
> > > > > > >> > > > > > > > > > > > >> >> >>> ...
> > > > > > >> > > > > > > > > > > > >> >> >>> > > feel free to give it a try
> > if
> > > you
> > > > > > are
> > > > > > >> > > > > comfortable
> > > > > > >> > > > > > > > with
> > > > > > >> > > > > > > > > > > that.
> > > > > > >> > > > > > > > > > > > >> >> >>> > >
> > > > > > >> > > > > > > > > > > > >> >> >>> > > 2. Could you clarify
> whether
> > > you
> > > > > > mean
> > > > > > >> end
> > > > > > >> > > > user
> > > > > > >> > > > > > > > > utilities
> > > > > > >> > > > > > > > > > or
> > > > > > >> > > > > > > > > > > > >> admin
> > > > > > >> > > > > > > > > > > > >> >> >>> user
> > > > > > >> > > > > > > > > > > > >> >> >>> > > utilities? In the latter
> > case
> > > > > those
> > > > > > >> might
> > > > > > >> > > be
> > > > > > >> > > > > more
> > > > > > >> > > > > > > > > > suitable
> > > > > > >> > > > > > > > > > > > for
> > > > > > >> > > > > > > > > > > > >> the
> > > > > > >> > > > > > > > > > > > >> >> >>> Admin
> > > > > > >> > > > > > > > > > > > >> >> >>> > > CLI (java) not the Python
> > CLI,
> > > > > IMHO.
> > > > > > >> > > > > > > > > > > > >> >> >>> > >
> > > > > > >> > > > > > > > > > > > >> >> >>> > > Why would these utilities
> be
> > > > > common
> > > > > > >> with
> > > > > > >> > > > > events?
> > > > > > >> > > > > > > > IMHO,
> > > > > > >> > > > > > > > > > > event
> > > > > > >> > > > > > > > > > > > >> use
> > > > > > >> > > > > > > > > > > > >> >> >>> cases
> > > > > > >> > > > > > > > > > > > >> >> >>> > are
> > > > > > >> > > > > > > > > > > > >> >> >>> > > distinct from scan/commit
> > > metrics.
> > > > > > >> > > > > > > > > > > > >> >> >>> > >
> > > > > > >> > > > > > > > > > > > >> >> >>> > > 3. I'd prefer separating
> > > metrics
> > > > > > >> > > persistence
> > > > > > >> > > > > from
> > > > > > >> > > > > > > > > > MetaStore
> > > > > > >> > > > > > > > > > > > >> >> >>> persistence
> > > > > > >> > > > > > > > > > > > >> >> >>> > at
> > > > > > >> > > > > > > > > > > > >> >> >>> > > the code level, so that
> they
> > > could
> > > > > > be
> > > > > > >> > mixed
> > > > > > >> > > > and
> > > > > > >> > > > > > > > matched
> > > > > > >> > > > > > > > > > > > >> >> >>> independently.
> > > > > > >> > > > > > > > > > > > >> >> >>> > The
> > > > > > >> > > > > > > > > > > > >> >> >>> > > separate datasource
> question
> > > will
> > > > > > >> become
> > > > > > >> > a
> > > > > > >> > > > > > > non-issue
> > > > > > >> > > > > > > > > with
> > > > > > >> > > > > > > > > > > > that
> > > > > > >> > > > > > > > > > > > >> >> >>> approach,
> > > > > > >> > > > > > > > > > > > >> >> >>> > I
> > > > > > >> > > > > > > > > > > > >> >> >>> > > guess.
> > > > > > >> > > > > > > > > > > > >> >> >>> > >
> > > > > > >> > > > > > > > > > > > >> >> >>> > > The rationale for
> separating
> > > scan
> > > > > > >> metrics
> > > > > > >> > > and
> > > > > > >> > > > > > > > metastore
> > > > > > >> > > > > > > > > > > > >> >> persistence
> > > > > > >> > > > > > > > > > > > >> >> >>> is
> > > > > > >> > > > > > > > > > > > >> >> >>> > that
> > > > > > >> > > > > > > > > > > > >> >> >>> > > "cascading deletes"
> between
> > > them
> > > > > are
> > > > > > >> > hardly
> > > > > > >> > > > > ever
> > > > > > >> > > > > > > > > > required.
> > > > > > >> > > > > > > > > > > > >> >> >>> Furthermore,
> > > > > > >> > > > > > > > > > > > >> >> >>> > the
> > > > > > >> > > > > > > > > > > > >> >> >>> > > data and query patterns
> are
> > > very
> > > > > > >> > different
> > > > > > >> > > so
> > > > > > >> > > > > > > > different
> > > > > > >> > > > > > > > > > > > >> >> technologies
> > > > > > >> > > > > > > > > > > > >> >> >>> > might
> > > > > > >> > > > > > > > > > > > >> >> >>> > > be beneficial in each
> case.
> > > > > > >> > > > > > > > > > > > >> >> >>> > >
> > > > > > >> > > > > > > > > > > > >> >> >>> > > [1]
> > > > > > >> > > > > > > > > > > > >> >> >>> >
> > > > > > >> > > > > > > > > > > > >> >> >>>
> > > > > > >> > > > > > > > > > > > >> >>
> > > > > > >> > > > > > > > > > > > >>
> > > > > > >> > > > > > > > > > > >
> > > > > > >> > > > > > > > > > >
> > > > > > >> > > > > > > > > >
> > > > > > >> > > > > > > > >
> > > > > > >> > > > > > > >
> > > > > > >> > > > > > >
> > > > > > >> > > > > >
> > > > > > >> > > > >
> > > > > > >> > > >
> > > > > > >> > >
> > > > > > >> >
> > > > > > >>
> > > > > >
> > > > >
> > >
> >
> https://urldefense.com/v3/__https://lists.apache.org/thread/yto2wp982t43h1mqjwnslswhws5z47cy__;!!Iz9xO38YGHZK!5EuyFFkk3vhRWVIRvQAWBSQfpJkTMA9HxugzDwXmN0LPPqhEFxYkFRGVhtb8AqUwXtDh2OplcMnbMDHKxYDakNU$
> > >> > > >> > > > > > > > > > > > >> >> >>> > >
> > > > > > >> > > > > > > > > > > > >> >> >>> > > Cheers,
> > > > > > >> > > > > > > > > > > > >> >> >>> > > Dmitri.
> > > > > > >> > > > > > > > > > > > >> >> >>> > >
> > > > > > >> > > > > > > > > > > > >> >> >>> > > On Fri, Feb 27, 2026 at
> > > 6:19 PM
> > > > > > Anand
> > > > > > >> > Kumar
> > > > > > >> > > > > > > Sankaran
> > > > > > >> > > > > > > > > via
> > > > > > >> > > > > > > > > > > dev
> > > > > > >> > > > > > > > > > > > <
> > > > > > >> > > > > > > > > > > > >> >> >>> > > dev@polaris.apache.org>
> > > wrote:
> > > > > > >> > > > > > > > > > > > >> >> >>> > >
> > > > > > >> > > > > > > > > > > > >> >> >>> > > > Thanks all. This PR is
> > > merged
> > > > > now.
> > > > > > >> > > > > > > > > > > > >> >> >>> > > >
> > > > > > >> > > > > > > > > > > > >> >> >>> > > > Here are the follow-up
> > > features
> > > > > /
> > > > > > >> work
> > > > > > >> > > > > needed.
> > > > > > >> > > > > > > > These
> > > > > > >> > > > > > > > > > > were
> > > > > > >> > > > > > > > > > > > >> all
> > > > > > >> > > > > > > > > > > > >> >> >>> part of
> > > > > > >> > > > > > > > > > > > >> >> >>> > > the
> > > > > > >> > > > > > > > > > > > >> >> >>> > > > merged PR at some point
> in
> > > time
> > > > > > and
> > > > > > >> > were
> > > > > > >> > > > > > removed
> > > > > > >> > > > > > > to
> > > > > > >> > > > > > > > > > > reduce
> > > > > > >> > > > > > > > > > > > >> >> scope.
> > > > > > >> > > > > > > > > > > > >> >> >>> > > >
> > > > > > >> > > > > > > > > > > > >> >> >>> > > > Please let me know what
> > you
> > > > > think.
> > > > > > >> > > > > > > > > > > > >> >> >>> > > >
> > > > > > >> > > > > > > > > > > > >> >> >>> > > >
> > > > > > >> > > > > > > > > > > > >> >> >>> > > >   1.  A REST API to
> > paginate
> > > > > > through
> > > > > > >> > > table
> > > > > > >> > > > > > > metrics.
> > > > > > >> > > > > > > > > > This
> > > > > > >> > > > > > > > > > > > >> will be
> > > > > > >> > > > > > > > > > > > >> >> >>> > non-IRC
> > > > > > >> > > > > > > > > > > > >> >> >>> > > > standard addition.
> > > > > > >> > > > > > > > > > > > >> >> >>> > > >   2.  Utilities for
> > > managing old
> > > > > > >> > records,
> > > > > > >> > > > > > should
> > > > > > >> > > > > > > be
> > > > > > >> > > > > > > > > > > common
> > > > > > >> > > > > > > > > > > > >> with
> > > > > > >> > > > > > > > > > > > >> >> >>> events.
> > > > > > >> > > > > > > > > > > > >> >> >>> > > > There was some
> discussion
> > > that
> > > > > it
> > > > > > >> > belongs
> > > > > > >> > > > to
> > > > > > >> > > > > > the
> > > > > > >> > > > > > > > CLI.
> > > > > > >> > > > > > > > > > > > >> >> >>> > > >   3.  Separate
> datasource
> > > > > > (metrics,
> > > > > > >> > > events,
> > > > > > >> > > > > > even
> > > > > > >> > > > > > > > > other
> > > > > > >> > > > > > > > > > > > >> tables?).
> > > > > > >> > > > > > > > > > > > >> >> >>> > > >
> > > > > > >> > > > > > > > > > > > >> >> >>> > > >
> > > > > > >> > > > > > > > > > > > >> >> >>> > > > Anything else?
> > > > > > >> > > > > > > > > > > > >> >> >>> > > >
> > > > > > >> > > > > > > > > > > > >> >> >>> > > > -
> > > > > > >> > > > > > > > > > > > >> >> >>> > > > Anand
> > > > > > >> > > > > > > > > > > > >> >> >>> > > >
> > > > > > >> > > > > > > > > > > > >> >> >>> > > >
> > > > > > >> > > > > > > > > > > > >> >> >>> > >
> > > > > > >> > > > > > > > > > > > >> >> >>> >
> > > > > > >> > > > > > > > > > > > >> >> >>> >
> > > > > > >> > > > > > > > > > > > >> >> >>>
> > > > > > >> > > > > > > > > > > > >> >> >>
> > > > > > >> > > > > > > > > > > > >> >>
> > > > > > >> > > > > > > > > > > > >> >
> > > > > > >> > > > > > > > > > > > >>
> > > > > > >> > > > > > > > > > > > >
> > > > > > >> > > > > > > > > > > >
> > > > > > >> > > > > > > > > > >
> > > > > > >> > > > > > > > > >
> > > > > > >> > > > > > > > >
> > > > > > >> > > > > > > >
> > > > > > >> > > > > > >
> > > > > > >> > > > > >
> > > > > > >> > > > >
> > > > > > >> > > >
> > > > > > >> > >
> > > > > > >> >
> > > > > > >>
> > > > > > >
> > > > > >
> > > > >
> > >
> >
> >
>

Re: Proposal for REST endpoints for table metrics and events

[EJ Wang <ej...@gmail.com>]

Hi folks,

Notes from today's metrics architecture sync are in the usual doc:
https://docs.google.com/document/d/100h7c4damrUzVuquYbBHM0EvA4LSWuW2IT2dN_7nYVA/edit?pli=1&tab=t.18iv1m7p6i6b

Highlights:

Module placement. core/ now holds the entity data model only. A new spi/
takes every other shared contract: feature SPIs, feature non-SPI contracts,
substrate SPIs, substrate non-SPI contracts, and contract-required types.
That settles the question PR #5204 reopened, by placing into a new SPI
module. Note that it changes slightly towards the referenced WIP RFC [1]
recommendations: what proposed to place in core/, now should go to spi/
except for the entity data model.

Metrics PRs. #5068 merged. #5204 needs reshaping onto the split. #4115 and
#4756 stay open with action on the author to follow up on review comments.

[1]
https://docs.google.com/document/d/1mj3vGpix9zTDTFFJKtWT5ZxI3PTGI1YNb5xFqyWrOkc/edit?tab=t.0

On Wed, Jul 15, 2026 at 11:01 AM Anand Kumar Sankaran via dev <
dev@polaris.apache.org> wrote:

> Thanks EJ.  My plan is to introduce PR0 in a new branch / PR and repurpose
> #4115 and #4756 for PR1 and PR2 to maintain all the valuable discussions
> that happened there.
>
> From: EJ Wang <ej...@gmail.com>
> Date: Wednesday, July 15, 2026 at 10:41 AM
> To: dev@polaris.apache.org <de...@polaris.apache.org>
> Subject: Re: Proposal for REST endpoints for table metrics and events
>
> Hi folks, Thanks again for the discussion today. I updated the sync doc
> with notes from today's metrics sync: https: //urldefense. com/v3/__https:
> //docs. google.
> com/document/d/100h7c4damrUzVuquYbBHM0EvA4LSWuW2IT2dN_7nYVA/edit?pli=1&tab=t.
> 2ahd7wze5f__;!!Iz9xO38YGHZK!4rnlVIqBQQQ6xpGe1PMV58edOS_cGrKoqdfhw4qSp_JQCavvEjaxXk0OwGPnxyEcO60AzKQK_XDtp7H3DdkFXcrXJwAz1g0$
>
>
> Hi folks,
>
> Thanks again for the discussion today. I updated the sync doc with notes
> from today's metrics sync:
>
> https://urldefense.com/v3/__https://docs.google.com/document/d/100h7c4damrUzVuquYbBHM0EvA4LSWuW2IT2dN_7nYVA/edit?pli=1&tab=t.2ahd7wze5f__;!!Iz9xO38YGHZK!4rnlVIqBQQQ6xpGe1PMV58edOS_cGrKoqdfhw4qSp_JQCavvEjaxXk0OwGPnxyEcO60AzKQK_XDtp7H3DdkFXcrXJwAz1g0$
>
> See highlights below:
>
> In previous meetings, EJ proposed a two prs framing to push forward the
> metrics query work in polaris. Dmitri pointed out a subtle point that pr2
> is currently also trying to do the persistence related code placement
> refactor. For the sake of straightforward review process, EJ, Dmitri and
> Anand (the author) agreed to separate the persistence refactoring into pr0,
> and reframing the pr scopes as follow:
>
>    1.
>
>    Pr0: refactor irc metrics spi
>    1.
>
>       SPI should stay in core/ with metric type enveloped
>       2.
>
>       default spi impl, no-op, in extensions/ - default selection at
>       runtime/
>       3.
>
>       move metrics persistence and correspondings to extensions/ as an
>       alternative SPI impl
>       2.
>
>    Pr1: rest api for query
>
> https://urldefense.com/v3/__https://github.com/apache/polaris/pull/4115/changes__;!!Iz9xO38YGHZK!4rnlVIqBQQQ6xpGe1PMV58edOS_cGrKoqdfhw4qSp_JQCavvEjaxXk0OwGPnxyEcO60AzKQK_XDtp7H3DdkFXcrXFCbEbRk$
>    1.
>
>       rest api spec in extensions/ as optional spec
>       2.
>
>       SPI in extensions/ as optional spec spi
>       3.
>
>       default spi impl in extensions/ no-op
>       4.
>
>       handler related in runtime/ - thin adaptor from http shape to spi
>       shape
>       3.
>
>    Pr2: jdbc impl for query
> https://urldefense.com/v3/__https://github.com/apache/polaris/pull/4756__;!!Iz9xO38YGHZK!4rnlVIqBQQQ6xpGe1PMV58edOS_cGrKoqdfhw4qSp_JQCavvEjaxXk0OwGPnxyEcO60AzKQK_XDtp7H3DdkFXcrXRVMBzqE$
>    1.
>
>       alternative spi impl in extensions/ -> query with jdbc
>
>
> Thanks,
> -ej
>
> On Wed, Jul 1, 2026 at 8:01 AM Alexandre Dutra <ad...@apache.org> wrote:
>
> > Hi all,
> >
> > > While the current implementation stores metrics in the metastore, other
> > deployments
> > may choose a KV store, a dedicated metrics warehouse, or another backend
> > entirely. In those cases, exposing a Polaris REST API over a specific
> > schema feels like coupling the API to an implementation detail.
> >
> > I wouldn't say to an implementation detail, but rather to a particular
> > setup / configuration. I think that's fine. If you don't store
> > metrics, you don't read metrics. Similar situations will certainly
> > happen in the future, e.g. if we ever introduce an API to read events,
> > that would only work if events are being persisted in the first place.
> > Again I think that's a fair trade-off.
> >
> > Thanks,
> > Alex
> >
> > On Wed, Jul 1, 2026 at 7:23 AM Dmitri Bourlatchkov <di...@apache.org>
> > wrote:
> > >
> > > Hi Yufei,
> > >
> > > From my POV, it is fine for Polaris to have multiple REST API surfaces
> > with
> > > different features and different degrees of "readiness". Each API can
> > > evolve fairly independently. With proper modularization (which Anand
> > did),
> > > changes to the main Polaris code and these extra modules should
> normally
> > be
> > > non-intrusive to each other.
> > >
> > > I assume Anand is going to be the first user of this API. However, if
> we
> > do
> > > not merge it we'll never know whether other users are interested. I do
> > not
> > > think many end users read the dev ML, but they certainly read the docs,
> > > code and release notes.
> > >
> > > There is no risk for Polaris as a project in exposing this experimental
> > > API. If usage dwindles over time, we can remove it. If it gains a
> > > considerable user / contributor base, we will promote it to a
> > "first-class"
> > > (non-beta) API.
> > >
> > > Cheers,
> > > Dmitri.
> > >
> > > On Tue, Jun 30, 2026 at 7:53 PM Yufei Gu <fl...@gmail.com> wrote:
> > >
> > > > Hi all,
> > > >
> > > > Thanks for driving this work forward. Building this as a POC makes
> > perfect
> > > > sense if the goal is to validate the idea and gather feedback. Should
> > we
> > > > merge a POC into the main branch before agreeing on the overall
> > direction?
> > > > To me, that feels like a separate discussion from whether the
> > > > implementation itself looks reasonable.
> > > >
> > > > One concern that I don't think has been fully addressed is whether we
> > need
> > > > to expose metrics consumption through a Polaris REST endpoint at all.
> > > >
> > > > My concern is less about the API itself and more about the boundary
> > Polaris
> > > > should own. The proposed REST API is tied to a particular persistence
> > > > schema, but that schema is just one possible implementation. While
> the
> > > > current implementation stores metrics in the metastore, other
> > deployments
> > > > may choose a KV store, a dedicated metrics warehouse, or another
> > backend
> > > > entirely. In those cases, exposing a Polaris REST API over a specific
> > > > schema feels like coupling the API to an implementation detail.
> > > >
> > > > Polaris isn't intended to become a full metrics system. I'd expect it
> > to
> > > > focus on producing metrics and let deployments choose the most
> > appropriate
> > > > storage and consumption model for their environment.
> > > >
> > > > I'm not opposed to exploring this direction, but we should first
> align
> > on
> > > > whether exposing a REST API through Polaris is the right long term
> > approach
> > > > before merging it into the main branch.
> > > >
> > > > Thanks,
> > > > Yufei
> > > >
> > > > Yufei
> > > >
> > > >
> > > > On Mon, Jun 29, 2026 at 9:27 AM Dmitri Bourlatchkov <
> dimas@apache.org>
> > > > wrote:
> > > >
> > > > > Hi All,
> > > > >
> > > > > It looks like Anand addressed all action items in [4115].
> > > > >
> > > > > Please review the latest state of the code. The PR LGTM and I
> > approved in
> > > > > GH from my side.
> > > > >
> > > > > [4115]
> https://urldefense.com/v3/__https://github.com/apache/polaris/pull/4115__;!!Iz9xO38YGHZK!4rnlVIqBQQQ6xpGe1PMV58edOS_cGrKoqdfhw4qSp_JQCavvEjaxXk0OwGPnxyEcO60AzKQK_XDtp7H3DdkFXcrXtFRscKs$
> > > > >
> > > > > Thanks,
> > > > > Dmitri.
> > > > >
> > > > > On Thu, Jun 25, 2026 at 1:56 PM Dmitri Bourlatchkov <
> > dimas@apache.org>
> > > > > wrote:
> > > > >
> > > > > > Hi All,
> > > > > >
> > > > > > Recap of the discussion in today's Community Sync:
> > > > > >
> > > > > > * The modular approach for Metrics SPI(s) and implementation
> seems
> > to
> > > > > have
> > > > > > community consensus (please comment if you still have concerns)
> > > > > >
> > > > > > * What modules / feature flags are enabled in Apache Polaris
> > images by
> > > > > > default will be discussed later (when the feature is close to
> > > > > completion).
> > > > > >
> > > > > > In any case, users will be able to turn the full feature on/off
> at
> > > > their
> > > > > > discretion, regardless of the default flag values.
> > > > > >
> > > > > > * It is generally preferable to have smaller PRs and introduce
> new
> > > > > > features gradually to simplify reviews.
> > > > > >
> > > > > > What this means for the Metrics API, IMHO is:
> > > > > >
> > > > > > a) The scope of PR 4115 is probably ok as it stands right now. If
> > > > someone
> > > > > > wants to propose splitting it further, please comment.
> > > > > >
> > > > > > b) After 4115 is merged, Metrics Persistence PRs will follow,
> with
> > > > > smaller
> > > > > > scope per PR as appropriate for the nature of the code changes.
> > > > > >
> > > > > > * The major concern was about exposing the new Metrics REST API
> to
> > end
> > > > > > users. The nature of this concern was about avoiding the (false)
> > > > > impression
> > > > > > that the new API is a firm and adopted Polaris specification.
> > > > > >
> > > > > > I think this concern is reasonable.
> > > > > >
> > > > > > I propose that Anand update the documentation sections of PR 4115
> > to
> > > > > > emphasise that the API is currently an experimental / POC API at
> > this
> > > > > time
> > > > > > and is subject to change (including breaking changes). I believe
> it
> > > > > already
> > > > > > has the "beta" label. So this request is mostly for doc changes
> to
> > > > > clarify
> > > > > > user expectations, since the "beta" label alone may not be
> > interpreted
> > > > > the
> > > > > > same way by all users.
> > > > > >
> > > > > > I hope this update will resolve all remaining concerns with 4115
> > and
> > > > > allow
> > > > > > for it to be merged so that the community will be able to proceed
> > to
> > > > > > follow-up PRs (as noted above).
> > > > > >
> > > > > > Please add/correct me if I missed anything relevant.
> > > > > >
> > > > > > Thanks,
> > > > > > Dmitri.
> > > > > >
> > > > > > On Wed, Jun 24, 2026 at 8:11 PM Yufei Gu <fl...@gmail.com>
> > wrote:
> > > > > >
> > > > > >> I do not think we have agreement on the query REST APIs yet.
> > > > > >>
> > > > > >> In particular, the current query API shape seems highly coupled
> > to the
> > > > > >> *example[1]* metrics in the Iceberg REST spec. I do not think
> > that is
> > > > a
> > > > > >> good way to design a REST API contract. For example, fields like
> > > > > >> resultDataFiles, resultDeleteFiles, totalFileSizeBytes,
> > > > > >> totalDataManifests,
> > > > > >> totalDeleteManifests, and scannedDataManifests are very specific
> > to
> > > > one
> > > > > >> style of scan metric. That makes the API look more like a direct
> > > > > >> projection
> > > > > >> of the current example payload than a general query model.
> > > > > >>
> > > > > >> My preference is that Polaris should not define or strongly
> opine
> > on
> > > > the
> > > > > >> Iceberg scan or commit metrics query REST API at this point.
> > Polaris
> > > > can
> > > > > >> resolve and authorize the table, accept the metrics report, and
> > > > delegate
> > > > > >> it
> > > > > >> to the configured reporting path. Durable storage, filtering,
> > > > > dashboards,
> > > > > >> and query APIs should remain downstream impl. choices unless and
> > until
> > > > > the
> > > > > >> Iceberg community standardizes a query contract, which I doubt
> > will
> > > > > >> happen.
> > > > > >>
> > > > > >> So I do not think the intake path has to be blocked, but I would
> > > > prefer
> > > > > >> not
> > > > > >> to merge a Polaris specific query API that may be hard to evolve
> > > > later.
> > > > > >>
> > > > > >> 1.
> > > > > >>
> > > > > >>
> > > > >
> > > >
> >
> https://urldefense.com/v3/__https://github.com/apache/iceberg/blob/2a6c556c0c883c04c6d3fbd68e6aeda36b91e0aa/open-api/rest-catalog-open-api.yaml*L4152__;Iw!!Iz9xO38YGHZK!4rnlVIqBQQQ6xpGe1PMV58edOS_cGrKoqdfhw4qSp_JQCavvEjaxXk0OwGPnxyEcO60AzKQK_XDtp7H3DdkFXcrXCDsqYGo$
> > > > > >>
> > > > > >> Yufei
> > > > > >>
> > > > > >>
> > > > > >> On Tue, Jun 23, 2026 at 9:30 AM Dmitri Bourlatchkov <
> > dimas@apache.org
> > > > >
> > > > > >> wrote:
> > > > > >>
> > > > > >> > Hi All,
> > > > > >> >
> > > > > >> > So, what are the blockers for merging [4115], if any?
> > > > > >> >
> > > > > >> > [4115]
> https://urldefense.com/v3/__https://github.com/apache/polaris/pull/4115__;!!Iz9xO38YGHZK!4rnlVIqBQQQ6xpGe1PMV58edOS_cGrKoqdfhw4qSp_JQCavvEjaxXk0OwGPnxyEcO60AzKQK_XDtp7H3DdkFXcrXtFRscKs$
> > > > > >> >
> > > > > >> > Thanks,
> > > > > >> > Dmitri.
> > > > > >> >
> > > > > >> > On Wed, Jun 17, 2026 at 5:28 PM EJ Wang <
> > > > > ej.wang.dev.interest@gmail.com
> > > > > >> >
> > > > > >> > wrote:
> > > > > >> >
> > > > > >> > > Hi folks,
> > > > > >> > >
> > > > > >> > > Thanks again for the discussion today. I updated the sync
> doc
> > with
> > > > > >> notes
> > > > > >> > > from today's metrics sync:
> > > > > >> > >
> > > > > >> > >
> > > > > >> > >
> > > > > >> >
> > > > > >>
> > > > >
> > > >
> >
> https://urldefense.com/v3/__https://docs.google.com/document/d/100h7c4damrUzVuquYbBHM0EvA4LSWuW2IT2dN_7nYVA/edit?tab=t.ezk0rgdx0c6m__;!!Iz9xO38YGHZK!4rnlVIqBQQQ6xpGe1PMV58edOS_cGrKoqdfhw4qSp_JQCavvEjaxXk0OwGPnxyEcO60AzKQK_XDtp7H3DdkFXcrX83IZkm0$
> > > > > >> > >
> > > > > >> > > A few highlights:
> > > > > >> > >
> > > > > >> > >    - No objection to keeping the default metrics behavior
> > small:
> > > > > >> > >    no-op/log-only is enough as the built-in default.
> > > > > >> > >    - Durable storage, event forwarding, external queues,
> > > > dashboards,
> > > > > >> and
> > > > > >> > >    custom filtering should be implementation choices behind
> > the
> > > > > >> metrics
> > > > > >> > >    reporting path.
> > > > > >> > >    - For PR #4115, the REST layer should
> resolve/authz/accept
> > the
> > > > > >> report,
> > > > > >> > >    then delegate to the selected metrics implementation. It
> > should
> > > > > not
> > > > > >> > own
> > > > > >> > >    durable storage semantics.
> > > > > >> > >    - The rough boundary we discussed is: spec owns the wire
> > > > > contract,
> > > > > >> > >    runtime owns framework wiring, API/contract modules own
> > > > > >> > provider-facing
> > > > > >> > >    contracts, and extensions own replaceable
> implementations.
> > > > > >> > >    - OpenLineage has a similar path-naming question. We
> should
> > > > avoid
> > > > > >> > >    occupying a generic lineage path too early if we want
> room
> > for
> > > > > >> other
> > > > > >> > >    lineage systems later.
> > > > > >> > >
> > > > > >> > > I believe the boundary rules previewed in sync meeting are
> > useful
> > > > > for
> > > > > >> > > reviewing PR #4115, but it is not
> > > > > >> > > a final module-layout proposal.
> > > > > >> > >
> > > > > >> > > For metrics specifically, I think the target remains:
> > > > > >> > >    runtime
> > > > > >> > >      -> metrics reporting/emitting contract
> > > > > >> > >      -> default implementation
> > > > > >> > >           -> lower implementation details, if needed
> > > > > >> > >
> > > > > >> > > That keeps the default simple, while allowing durable JDBC,
> > > > > >> event-backed,
> > > > > >> > > or external-queue implementations to be added separately.
> > > > > >> > >
> > > > > >> > > We also touched on the split between Iceberg metrics
> emitting
> > and
> > > > > >> > > Polaris-owned metrics querying. The querying side can keep
> > > > evolving,
> > > > > >> > > especially for dashboard or generic-table use cases, but I
> do
> > not
> > > > > >> think
> > > > > >> > > that needs to block the intake path in this PR.
> > > > > >> > >
> > > > > >> > > Thanks,
> > > > > >> > > -ej
> > > > > >> > >
> > > > > >> > > On Tue, Jun 16, 2026 at 2:17 PM Yufei Gu <
> > flyrain000@gmail.com>
> > > > > >> wrote:
> > > > > >> > >
> > > > > >> > > > Thanks for chiming in, EJ. Agreed that the default battery
> > > > should
> > > > > >> stay
> > > > > >> > > > small. I'm leaning toward not including metrics
> persistence
> > in
> > > > the
> > > > > >> > > default
> > > > > >> > > > battery.
> > > > > >> > > >
> > > > > >> > > > Yufei
> > > > > >> > > >
> > > > > >> > > >
> > > > > >> > > > On Thu, Jun 11, 2026 at 8:30 PM EJ Wang <
> > > > > >> > ej.wang.dev.interest@gmail.com>
> > > > > >> > > > wrote:
> > > > > >> > > >
> > > > > >> > > > > Hi Yufei,
> > > > > >> > > > >
> > > > > >> > > > > Thanks for connecting this back to the REST endpoint
> > > > proposal. I
> > > > > >> > > replied
> > > > > >> > > > > with the fuller version on the event-forwarding thread,
> > but
> > > > > >> wanted to
> > > > > >> > > add
> > > > > >> > > > > the shorter version here too.
> > > > > >> > > > >
> > > > > >> > > > > I agree that metrics reporting/emitting is the right
> > > > conceptual
> > > > > >> > > boundary,
> > > > > >> > > > > and I also think the event/listener path is a good
> > > > > implementation
> > > > > >> > > > direction
> > > > > >> > > > > to explore. The distinction I want to keep clear is
> > default
> > > > > >> battery
> > > > > >> > vs
> > > > > >> > > > > extension implementation.
> > > > > >> > > > >
> > > > > >> > > > > For the REST/API side, I would keep the semantics
> narrow:
> > > > > Polaris
> > > > > >> > > > resolves
> > > > > >> > > > > and authorizes the table, accepts the Iceberg
> scan/commit
> > > > > metrics
> > > > > >> > > report
> > > > > >> > > > > into the configured reporting/emitting path, and returns
> > 204.
> > > > > That
> > > > > >> > 204
> > > > > >> > > > > should mean Polaris accepted the report into the
> ingestion
> > > > path;
> > > > > >> it
> > > > > >> > > > should
> > > > > >> > > > > not imply durable storage.
> > > > > >> > > > >
> > > > > >> > > > > Durable storage, event forwarding, external telemetry
> > routing,
> > > > > >> > > filtering,
> > > > > >> > > > > and retention should sit behind that reporting/emitting
> > > > boundary
> > > > > >> as
> > > > > >> > > > > implementation-layer behavior. A durable JDBC path can
> be
> > one
> > > > > >> named
> > > > > >> > > > > extension implementation. An event/listener forwarding
> > path
> > > > can
> > > > > be
> > > > > >> > > > another
> > > > > >> > > > > named extension implementation.
> > > > > >> > > > >
> > > > > >> > > > > So I am +1 on the event-forwarding idea as a non-default
> > > > > extension
> > > > > >> > > > > implementation. I just would not make it the default
> > battery
> > > > or
> > > > > >> the
> > > > > >> > > core
> > > > > >> > > > > API shape. The default battery can stay small and safe,
> > while
> > > > > >> > > deployments
> > > > > >> > > > > that need forwarding, dashboards, or durable storage can
> > opt
> > > > > into
> > > > > >> the
> > > > > >> > > > > implementation that matches their operational model.
> > > > > >> > > > >
> > > > > >> > > > > Thanks,
> > > > > >> > > > > -ej
> > > > > >> > > > >
> > > > > >> > > > > On Wed, Jun 10, 2026 at 11:43 AM Yufei Gu <
> > > > flyrain000@gmail.com
> > > > > >
> > > > > >> > > wrote:
> > > > > >> > > > >
> > > > > >> > > > > > Thanks EJ. I agree that the reporting/emitting
> boundary
> > > > feels
> > > > > >> like
> > > > > >> > > the
> > > > > >> > > > > > right SPI boundary, but I wonder if we can simplify
> this
> > > > even
> > > > > >> > > further.
> > > > > >> > > > > >
> > > > > >> > > > > > Iceberg scan and commit metrics look very similar to
> > events
> > > > to
> > > > > >> me.
> > > > > >> > > They
> > > > > >> > > > > are
> > > > > >> > > > > > append only, asynchronously consumed, often forwarded
> to
> > > > > >> external
> > > > > >> > > > > systems,
> > > > > >> > > > > > and have similar retention and cleanup requirements.
> > More
> > > > > >> details
> > > > > >> > can
> > > > > >> > > > be
> > > > > >> > > > > > found in this thread [1]. In that case, Polaris may
> only
> > > > need
> > > > > to
> > > > > >> > > accept
> > > > > >> > > > > the
> > > > > >> > > > > > report and emit an event through the existing event
> > > > framework.
> > > > > >> > > > Filtering,
> > > > > >> > > > > > async delivery, custom sinks, and retention mechanisms
> > could
> > > > > >> then
> > > > > >> > be
> > > > > >> > > > > shared
> > > > > >> > > > > > instead of introducing a separate metrics specific
> > extension
> > > > > >> layer
> > > > > >> > > and
> > > > > >> > > > > SPI.
> > > > > >> > > > > >
> > > > > >> > > > > > The main requirement I still see is querying metrics
> > through
> > > > > >> > Polaris.
> > > > > >> > > > If
> > > > > >> > > > > we
> > > > > >> > > > > > want that battery included experience, we could work
> on
> > a
> > > > > >> > persistence
> > > > > >> > > > > > listener implementation that selectively saves metrics
> > and
> > > > > >> expose
> > > > > >> > it
> > > > > >> > > > via
> > > > > >> > > > > > the IRC event endpoint which seems pretty close in
> > Iceberg
> > > > > >> > > > > > community. Another option I like more is to allow
> users
> > to
> > > > > >> develop
> > > > > >> > > > their
> > > > > >> > > > > > own listener, so they can persist scan/commit metrics
> > to any
> > > > > >> > systems
> > > > > >> > > > they
> > > > > >> > > > > > prefer. In general, that feels like an implementation
> > choice
> > > > > on
> > > > > >> top
> > > > > >> > > of
> > > > > >> > > > > the
> > > > > >> > > > > > event framework rather than something that needs to
> > shape
> > > > the
> > > > > >> core
> > > > > >> > > > > > architecture.
> > > > > >> > > > > >
> > > > > >> > > > > > 1.
> > > > > >> >
> >
> https://urldefense.com/v3/__https://lists.apache.org/thread/x9j8nscvy8hq61tyn01mj8yp6n9of0kp__;!!Iz9xO38YGHZK!4rnlVIqBQQQ6xpGe1PMV58edOS_cGrKoqdfhw4qSp_JQCavvEjaxXk0OwGPnxyEcO60AzKQK_XDtp7H3DdkFXcrXGcGDgks$
> > > > > >> > > > > >
> > > > > >> > > > > > Thanks,
> > > > > >> > > > > > Yufei
> > > > > >> > > > > >
> > > > > >> > > > > >
> > > > > >> > > > > > On Thu, Jun 4, 2026 at 5:14 PM EJ Wang <
> > > > > >> > > ej.wang.dev.interest@gmail.com
> > > > > >> > > > >
> > > > > >> > > > > > wrote:
> > > > > >> > > > > >
> > > > > >> > > > > > > Good point. I agree the existing
> > PolarisMetricsReporter is
> > > > > >> > already
> > > > > >> > > > very
> > > > > >> > > > > > > close to the right conceptual boundary. I am not
> > > > proposing a
> > > > > >> > second
> > > > > >> > > > > > > parallel reporter concept. The distinction I am
> > trying to
> > > > > >> make is
> > > > > >> > > > > > narrower:
> > > > > >> > > > > > >
> > > > > >> > > > > > > Current state:
> > > > > >> > > > > > >
> > > > > >> > > > > > >    - PolarisMetricsReporter is the existing metrics
> > > > > reporting
> > > > > >> > hook.
> > > > > >> > > > > > >    - But it currently lives under runtime/service.
> > > > > >> > > > > > >    - Its contract is documented around Quarkus/CDI
> > > > discovery
> > > > > >> and
> > > > > >> > > > > > > @Identifier
> > > > > >> > > > > > >    selection.
> > > > > >> > > > > > >    - The default implementation is log-only.
> > > > > >> > > > > > >    - The durable implementation writes through
> > > > > >> > > PolarisMetricsManager
> > > > > >> > > > > and
> > > > > >> > > > > > >    MetricsPersistence.
> > > > > >> > > > > > >    - MetricsPersistence is currently inherited by
> > > > > >> > BasePersistence.
> > > > > >> > > > > > >
> > > > > >> > > > > > > My proposal:
> > > > > >> > > > > > >
> > > > > >> > > > > > >    - Keep the reporting/emitting boundary as the
> SPI.
> > > > > >> > > > > > >    - Revise that contract in a framework-agnostic
> > optional
> > > > > >> > Iceberg
> > > > > >> > > > > > *metrics
> > > > > >> > > > > > >    extension API layer*.
> > > > > >> > > > > > >    - Keep runtime/service responsible for REST
> > ingestion,
> > > > > >> > > table/authz
> > > > > >> > > > > > >    validation, and runtime wiring.
> > > > > >> > > > > > >    - Keep the battery implementation log-only in the
> > > > metrics
> > > > > >> > > > extension
> > > > > >> > > > > > API
> > > > > >> > > > > > >    layer, *not under runtime/service*.
> > > > > >> > > > > > >    - Treat durable JDBC metrics as one
> implementation
> > of
> > > > the
> > > > > >> > > > reporting
> > > > > >> > > > > > SPI.
> > > > > >> > > > > > >    - Decompose the MetricsPersistence and
> > BasePersistence
> > > > > >> > coupling
> > > > > >> > > as
> > > > > >> > > > > > >    durable implementation detail.
> > > > > >> > > > > > >
> > > > > >> > > > > > > So the difference is not “new reporter concept vs
> old
> > > > > reporter
> > > > > >> > > > > concept.”
> > > > > >> > > > > > > but:
> > > > > >> > > > > > >
> > > > > >> > > > > > >
> > > > > >> > > > > > >    - PolarisMetricsReporter today = runtime/service
> > > > > CDI-shaped
> > > > > >> > > hook.
> > > > > >> > > > > > >    - Target reporting SPI = same conceptual
> boundary,
> > but
> > > > > >> placed
> > > > > >> > in
> > > > > >> > > > the
> > > > > >> > > > > > >    right extension API layer and kept
> > framework-agnostic.
> > > > > >> > > > > > >
> > > > > >> > > > > > > For the current metrics PR, I think the minimal
> > > > > architectural
> > > > > >> > > cleanup
> > > > > >> > > > > > > before merge is to *put the reporting SPI contract
> > and the
> > > > > >> > battery
> > > > > >> > > > > > default
> > > > > >> > > > > > > in the right place*. The durable JDBC
> implementation,
> > > > > >> > event-backed
> > > > > >> > > > > async
> > > > > >> > > > > > > implementation, external queue support, and deeper
> > > > > persistence
> > > > > >> > > > cleanup
> > > > > >> > > > > > can
> > > > > >> > > > > > > continue as follow-ups, as long as we do not lock
> the
> > SPI
> > > > > >> > boundary
> > > > > >> > > to
> > > > > >> > > > > > > BasePersistence or to runtime/service wiring.
> > > > > >> > > > > > >
> > > > > >> > > > > > > -ej
> > > > > >> > > > > > >
> > > > > >> > > > > > > On Wed, Jun 3, 2026 at 3:00 PM Dmitri Bourlatchkov <
> > > > > >> > > dimas@apache.org
> > > > > >> > > > >
> > > > > >> > > > > > > wrote:
> > > > > >> > > > > > >
> > > > > >> > > > > > > > Hi EJ,
> > > > > >> > > > > > > >
> > > > > >> > > > > > > > Thanks for the recap / summary!
> > > > > >> > > > > > > >
> > > > > >> > > > > > > > > Do folks agree that the stable SPI boundary
> > should be
> > > > > >> > > > > > > > metrics reporting/emitting, not metrics
> persistence?
> > > > > >> > > > > > > >
> > > > > >> > > > > > > > SGTM.
> > > > > >> > > > > > > >
> > > > > >> > > > > > > > > Does an optional Iceberg metrics extension API
> > layer
> > > > > sound
> > > > > >> > like
> > > > > >> > > > the
> > > > > >> > > > > > > right
> > > > > >> > > > > > > > home for this SPI?
> > > > > >> > > > > > > >
> > > > > >> > > > > > > > How is that different from current
> > > > PolarisMetricsReporter?
> > > > > >> > > > > > > >
> > > > > >> > > > > > > > > Should the current durable metrics work be
> > reframed
> > > > as a
> > > > > >> > > durable
> > > > > >> > > > > > > > JDBC reference implementation of that SPI?
> > > > > >> > > > > > > >
> > > > > >> > > > > > > > SGTM.
> > > > > >> > > > > > > >
> > > > > >> > > > > > > > > What is the smallest PR sequence to get there
> > without
> > > > > >> > blocking
> > > > > >> > > > > > > > the current metrics work unnecessarily?
> > > > > >> > > > > > > >
> > > > > >> > > > > > > > Let's get
> >
> https://urldefense.com/v3/__https://github.com/apache/polaris/pull/4397__;!!Iz9xO38YGHZK!4rnlVIqBQQQ6xpGe1PMV58edOS_cGrKoqdfhw4qSp_JQCavvEjaxXk0OwGPnxyEcO60AzKQK_XDtp7H3DdkFXcrXuu5jefU$
> > > > > >> merged
> > > > > >> > > > first.
> > > > > >> > > > > > > >
> > > > > >> > > > > > > > Then, I'd propose isolating the Metrics schema
> from
> > the
> > > > > >> > MetaStore
> > > > > >> > > > > > schema.
> > > > > >> > > > > > > > This will probably have some ripple effect into
> the
> > > > > >> bootstrap
> > > > > >> > > > > > workflows,
> > > > > >> > > > > > > so
> > > > > >> > > > > > > > it's not a trivial change.
> > > > > >> > > > > > > >
> > > > > >> > > > > > > > Then, let's reassess.
> > > > > >> > > > > > > >
> > > > > >> > > > > > > > Cheers,
> > > > > >> > > > > > > > Dmitri.
> > > > > >> > > > > > > >
> > > > > >> > > > > > > > On Wed, Jun 3, 2026 at 5:40 PM EJ Wang <
> > > > > >> > > > > ej.wang.dev.interest@gmail.com
> > > > > >> > > > > > >
> > > > > >> > > > > > > > wrote:
> > > > > >> > > > > > > >
> > > > > >> > > > > > > > > Hi folks,
> > > > > >> > > > > > > > >
> > > > > >> > > > > > > > > Thanks again for the discussion today. I updated
> > the
> > > > > sync
> > > > > >> doc
> > > > > >> > > > with
> > > > > >> > > > > > > notes
> > > > > >> > > > > > > > > from the third metrics architecture sync:
> > > > > >> > > > > > > > >
> > > > > >> > > > > > > > >
> > > > > >> > > > > > > > >
> > > > > >> > > > > > > >
> > > > > >> > > > > > >
> > > > > >> > > > > >
> > > > > >> > > > >
> > > > > >> > > >
> > > > > >> > >
> > > > > >> >
> > > > > >>
> > > > >
> > > >
> >
> https://urldefense.com/v3/__https://docs.google.com/document/d/100h7c4damrUzVuquYbBHM0EvA4LSWuW2IT2dN_7nYVA/edit?tab=t.k96s2xyqr5u1__;!!Iz9xO38YGHZK!4rnlVIqBQQQ6xpGe1PMV58edOS_cGrKoqdfhw4qSp_JQCavvEjaxXk0OwGPnxyEcO60AzKQK_XDtp7H3DdkFXcrXrqCL-QE$
> > > > > >> > > > > > > > >
> > > > > >> > > > > > > > > A few highlights from today’s discussion:
> > > > > >> > > > > > > > >
> > > > > >> > > > > > > > >    - We clarified that Iceberg metrics reporting
> > can
> > > > be
> > > > > >> > > > interpreted
> > > > > >> > > > > > > > either
> > > > > >> > > > > > > > >    as sync or async handling from Polaris’s
> > > > perspective,
> > > > > >> so
> > > > > >> > > > Polaris
> > > > > >> > > > > > > > should
> > > > > >> > > > > > > > >    stay flexible as a platform instead of baking
> > one
> > > > > >> handling
> > > > > >> > > > model
> > > > > >> > > > > > > into
> > > > > >> > > > > > > > > the
> > > > > >> > > > > > > > >    REST API behavior.
> > > > > >> > > > > > > > >    - We aligned that the built-in (battery)
> > behavior
> > > > for
> > > > > >> > > metrics
> > > > > >> > > > > > > emitting
> > > > > >> > > > > > > > >    can stay simple: no-op/log-only is enough as
> > the
> > > > > >> default.
> > > > > >> > > > > > > > >    - Durable metrics persistence should be
> > treated as
> > > > an
> > > > > >> > > > > > implementation
> > > > > >> > > > > > > > of
> > > > > >> > > > > > > > >    the metrics reporting path, not as the core
> SPI
> > > > > >> boundary.
> > > > > >> > > > > > > > >    - The existing durable metrics work can be
> > reviewed
> > > > > as
> > > > > >> a
> > > > > >> > > > > reference
> > > > > >> > > > > > > > >    implementation of the reporting SPI, with
> > > > > >> > > persistence-related
> > > > > >> > > > > > logic
> > > > > >> > > > > > > > kept
> > > > > >> > > > > > > > >    self-contained under a metrics durable
> > > > implementation
> > > > > >> > module
> > > > > >> > > > > > rather
> > > > > >> > > > > > > > than
> > > > > >> > > > > > > > >    scattered through core entity persistence.
> > > > > >> > > > > > > > >    - Dashboard/insights remains a real use case,
> > but
> > > > we
> > > > > >> > agreed
> > > > > >> > > to
> > > > > >> > > > > > keep
> > > > > >> > > > > > > it
> > > > > >> > > > > > > > >    separate from the core metrics intake
> > discussion
> > > > for
> > > > > >> now.
> > > > > >> > > > > > > > >
> > > > > >> > > > > > > > > I also did a quick source check after the
> meeting
> > to
> > > > > make
> > > > > >> > sure
> > > > > >> > > we
> > > > > >> > > > > are
> > > > > >> > > > > > > > > describing the current state accurately.
> > > > > >> > > > > > > > >
> > > > > >> > > > > > > > > Current state:
> > > > > >> > > > > > > > >
> > > > > >> > > > > > > > >    - Polaris already has a metrics reporting
> hook:
> > > > > >> > > > > > > > PolarisMetricsReporter.
> > > > > >> > > > > > > > >    - The default implementation is
> > > > > DefaultMetricsReporter,
> > > > > >> > > > selected
> > > > > >> > > > > > by
> > > > > >> > > > > > > > >
> > polaris.iceberg-metrics.reporting.type=default. It
> > > > is
> > > > > >> > > > log-only,
> > > > > >> > > > > > > > >    effectively quiet unless metrics logging is
> > > > enabled.
> > > > > >> > > > > > > > >    - There is also a PersistingMetricsReporter,
> > > > selected
> > > > > >> by
> > > > > >> > > > > > > > >
> > polaris.iceberg-metrics.reporting.type=persisting,
> > > > > >> which
> > > > > >> > > > > converts
> > > > > >> > > > > > > > >    Iceberg scan/commit reports into Polaris
> > metrics
> > > > > >> records
> > > > > >> > and
> > > > > >> > > > > > writes
> > > > > >> > > > > > > > > through
> > > > > >> > > > > > > > >    PolarisMetricsManager -> MetricsPersistence.
> > > > > >> > > > > > > > >    - MetricsPersistence currently exists in the
> > > > > >> persistence
> > > > > >> > > layer
> > > > > >> > > > > and
> > > > > >> > > > > > > is
> > > > > >> > > > > > > > >    inherited by BasePersistence.
> > > > > >> > > > > > > > >
> > > > > >> > > > > > > > > My read from the discussion is that the target
> > > > boundary
> > > > > >> > should
> > > > > >> > > > not
> > > > > >> > > > > be
> > > > > >> > > > > > > > > MetricsPersistence as inherited by
> > BasePersistence.
> > > > That
> > > > > >> path
> > > > > >> > > > > should
> > > > > >> > > > > > be
> > > > > >> > > > > > > > > decomposed as durable implementation detail
> (taken
> > > > care
> > > > > >> of by
> > > > > >> > > > > > > > >
> https://urldefense.com/v3/__https://github.com/apache/polaris/pull/4397__;!!Iz9xO38YGHZK!4rnlVIqBQQQ6xpGe1PMV58edOS_cGrKoqdfhw4qSp_JQCavvEjaxXk0OwGPnxyEcO60AzKQK_XDtp7H3DdkFXcrXuu5jefU$).
> The
> > > > > stable
> > > > > >> > > > extension
> > > > > >> > > > > > > point
> > > > > >> > > > > > > > > should instead be the metrics reporting/emitting
> > > > > boundary.
> > > > > >> > > > > > > > >
> > > > > >> > > > > > > > > Concretely, I think the next proposal should be
> > shaped
> > > > > >> like
> > > > > >> > > this:
> > > > > >> > > > > > > > >
> > > > > >> > > > > > > > >    1.
> > > > > >> > > > > > > > >
> > > > > >> > > > > > > > >    Define the proper metrics reporting SPI at an
> > > > > optional
> > > > > >> > > Iceberg
> > > > > >> > > > > > > metrics
> > > > > >> > > > > > > > >    extension API layer.
> > > > > >> > > > > > > > >
> > > > > >> > > > > > > > >    Example direction:
> > > > > >> > > > > > > > >
> > > > > >> > > > > > > > >    public interface IcebergMetricsReporter {
> > > > > >> > > > > > > > >      void
> reportMetric(IcebergMetricsReportContext
> > > > > >> context,
> > > > > >> > > > > > > > > MetricsReport report);
> > > > > >> > > > > > > > >    }
> > > > > >> > > > > > > > >
> > > > > >> > > > > > > > >    The context would carry the small
> > Polaris-resolved
> > > > > >> > envelope:
> > > > > >> > > > > > > > >    catalog/table identity, report type, received
> > > > > >> timestamp,
> > > > > >> > and
> > > > > >> > > > > > > > > request/trace
> > > > > >> > > > > > > > >    context if available. The raw Iceberg
> > MetricsReport
> > > > > >> > remains
> > > > > >> > > > the
> > > > > >> > > > > > > > payload.
> > > > > >> > > > > > > > >    2.
> > > > > >> > > > > > > > >
> > > > > >> > > > > > > > >    Keep runtime/service as the REST ingestion
> and
> > > > wiring
> > > > > >> > layer.
> > > > > >> > > > > > > > >
> > > > > >> > > > > > > > >    The REST handler still resolves the table and
> > > > > performs
> > > > > >> > authz
> > > > > >> > > > > > before
> > > > > >> > > > > > > > >    accepting the report. After that, it calls
> the
> > > > > selected
> > > > > >> > > > > reporting
> > > > > >> > > > > > > > >    implementation.
> > > > > >> > > > > > > > >    3.
> > > > > >> > > > > > > > >
> > > > > >> > > > > > > > >    Keep the battery default no-op/log-only.
> > > > > >> > > > > > > > >
> > > > > >> > > > > > > > >    This preserves an out-of-box safe default and
> > > > avoids
> > > > > >> > > > requiring a
> > > > > >> > > > > > > > durable
> > > > > >> > > > > > > > >    metrics store for every Polaris deployment.
> > > > > >> > > > > > > > >    4.
> > > > > >> > > > > > > > >
> > > > > >> > > > > > > > >    Move durable JDBC metrics into a
> self-contained
> > > > > >> > > implementation
> > > > > >> > > > > of
> > > > > >> > > > > > > the
> > > > > >> > > > > > > > >    reporting SPI.
> > > > > >> > > > > > > > >
> > > > > >> > > > > > > > >    That implementation can own its schema,
> > bootstrap,
> > > > > >> > > retention,
> > > > > >> > > > > and
> > > > > >> > > > > > > read
> > > > > >> > > > > > > > >    API support. It should not define the core
> > > > reporting
> > > > > >> SPI
> > > > > >> > > > > boundary,
> > > > > >> > > > > > > and
> > > > > >> > > > > > > > > it
> > > > > >> > > > > > > > >    should not require metrics persistence to
> > remain
> > > > > >> inherited
> > > > > >> > > > from
> > > > > >> > > > > > > > >    BasePersistence.
> > > > > >> > > > > > > > >    5.
> > > > > >> > > > > > > > >
> > > > > >> > > > > > > > >    Treat async/event-backed handling as another
> > > > > >> > implementation
> > > > > >> > > of
> > > > > >> > > > > the
> > > > > >> > > > > > > > same
> > > > > >> > > > > > > > >    reporting SPI.
> > > > > >> > > > > > > > >
> > > > > >> > > > > > > > >    For example, an event-backed reporter could
> > > > enqueue a
> > > > > >> > > metrics
> > > > > >> > > > > > event
> > > > > >> > > > > > > > and
> > > > > >> > > > > > > > >    let listeners handle durable storage or other
> > > > sinks.
> > > > > >> If we
> > > > > >> > > > later
> > > > > >> > > > > > > need
> > > > > >> > > > > > > > a
> > > > > >> > > > > > > > >    replaceable queue engine, that seems like a
> > shared
> > > > > >> > > > event/metrics
> > > > > >> > > > > > > > > substrate
> > > > > >> > > > > > > > >    topic rather than a metrics-only requirement.
> > > > > >> > > > > > > > >
> > > > > >> > > > > > > > > This framing lets us keep the REST metrics
> > endpoint
> > > > > >> simple,
> > > > > >> > > > > preserve
> > > > > >> > > > > > > the
> > > > > >> > > > > > > > > current default behavior, support durable
> metrics
> > > > users,
> > > > > >> and
> > > > > >> > > > still
> > > > > >> > > > > > > leave
> > > > > >> > > > > > > > > room for async/event-backed or
> > external-queue-based
> > > > > >> > > > > implementations.
> > > > > >> > > > > > > > >
> > > > > >> > > > > > > > > I think the main follow-up questions are:
> > > > > >> > > > > > > > >
> > > > > >> > > > > > > > >    - Do folks agree that the stable SPI boundary
> > > > should
> > > > > be
> > > > > >> > > > metrics
> > > > > >> > > > > > > > >    reporting/emitting, not metrics persistence?
> > > > > >> > > > > > > > >    - Does an optional Iceberg metrics extension
> > API
> > > > > layer
> > > > > >> > sound
> > > > > >> > > > > like
> > > > > >> > > > > > > the
> > > > > >> > > > > > > > >    right home for this SPI?
> > > > > >> > > > > > > > >    - Should the current durable metrics work be
> > > > reframed
> > > > > >> as a
> > > > > >> > > > > durable
> > > > > >> > > > > > > > JDBC
> > > > > >> > > > > > > > >    reference implementation of that SPI?
> > > > > >> > > > > > > > >    - What is the smallest PR sequence to get
> there
> > > > > without
> > > > > >> > > > blocking
> > > > > >> > > > > > the
> > > > > >> > > > > > > > >    current metrics work unnecessarily?
> > > > > >> > > > > > > > >
> > > > > >> > > > > > > > > Thanks,
> > > > > >> > > > > > > > > -ej
> > > > > >> > > > > > > > >
> > > > > >> > > > > > > > > On Fri, May 15, 2026 at 5:16 PM Dmitri
> > Bourlatchkov <
> > > > > >> > > > > > dimas@apache.org>
> > > > > >> > > > > > > > > wrote:
> > > > > >> > > > > > > > >
> > > > > >> > > > > > > > > > Hi JB,
> > > > > >> > > > > > > > > >
> > > > > >> > > > > > > > > > Could you set up another meeting, please? Same
> > time
> > > > on
> > > > > >> > > > Wednesday
> > > > > >> > > > > as
> > > > > >> > > > > > > > last
> > > > > >> > > > > > > > > > time... I hope it works for everyone.
> > > > > >> > > > > > > > > >
> > > > > >> > > > > > > > > > Cheers,
> > > > > >> > > > > > > > > > Dmitri.
> > > > > >> > > > > > > > > >
> > > > > >> > > > > > > > > > On Fri, May 15, 2026 at 8:06 PM Yufei Gu <
> > > > > >> > > flyrain000@gmail.com
> > > > > >> > > > >
> > > > > >> > > > > > > wrote:
> > > > > >> > > > > > > > > >
> > > > > >> > > > > > > > > > > +1 on another sync call next week.
> > > > > >> > > > > > > > > > >
> > > > > >> > > > > > > > > > > Yufei
> > > > > >> > > > > > > > > > >
> > > > > >> > > > > > > > > > >
> > > > > >> > > > > > > > > > > On Fri, May 15, 2026 at 4:52 PM Dmitri
> > > > Bourlatchkov
> > > > > <
> > > > > >> > > > > > > > dimas@apache.org>
> > > > > >> > > > > > > > > > > wrote:
> > > > > >> > > > > > > > > > >
> > > > > >> > > > > > > > > > > > Hi All,
> > > > > >> > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > WDYT about another sync call next week?
> > > > > >> > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > Thanks,
> > > > > >> > > > > > > > > > > > Dmitri.
> > > > > >> > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > On Wed, May 6, 2026 at 5:29 PM Dmitri
> > > > > Bourlatchkov <
> > > > > >> > > > > > > > dimas@apache.org
> > > > > >> > > > > > > > > >
> > > > > >> > > > > > > > > > > > wrote:
> > > > > >> > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > > Hi EJ,
> > > > > >> > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > > Thanks for the summary! It covers what
> we
> > > > > >> discussed
> > > > > >> > in
> > > > > >> > > > the
> > > > > >> > > > > > > > meeting
> > > > > >> > > > > > > > > > very
> > > > > >> > > > > > > > > > > > > well, IMHO.
> > > > > >> > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > > Looking forward to concrete PRs :)
> > > > > >> > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > > Cheers,
> > > > > >> > > > > > > > > > > > > Dmitri.
> > > > > >> > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > > On Wed, May 6, 2026 at 5:08 PM EJ Wang <
> > > > > >> > > > > > > > > > ej.wang.dev.interest@gmail.com
> > > > > >> > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > > wrote:
> > > > > >> > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > >> Hi folks,
> > > > > >> > > > > > > > > > > > >>
> > > > > >> > > > > > > > > > > > >> We had a community sync earlier, thanks
> > JB
> > > > for
> > > > > >> > > > scheduling
> > > > > >> > > > > > it.
> > > > > >> > > > > > > > > Notes
> > > > > >> > > > > > > > > > > from
> > > > > >> > > > > > > > > > > > >> the first metrics architecture sync
> (May
> > 6,
> > > > > >> 10-11am
> > > > > >> > > PT).
> > > > > >> > > > > > > > > Discussion
> > > > > >> > > > > > > > > > > doc
> > > > > >> > > > > > > > > > > > >> with per-section status:
> > > > > >> > > > > > > > > > > > >>
> > > > > >> > > > > > > > > > > > >>
> > > > > >> > > > > > > > > > > >
> > > > > >> > > > > > > > > > >
> > > > > >> > > > > > > > > >
> > > > > >> > > > > > > > >
> > > > > >> > > > > > > >
> > > > > >> > > > > > >
> > > > > >> > > > > >
> > > > > >> > > > >
> > > > > >> > > >
> > > > > >> > >
> > > > > >> >
> > > > > >>
> > > > >
> > > >
> >
> https://urldefense.com/v3/__https://docs.google.com/document/d/100h7c4damrUzVuquYbBHM0EvA4LSWuW2IT2dN_7nYVA/edit?tab=t.0__;!!Iz9xO38YGHZK!4rnlVIqBQQQ6xpGe1PMV58edOS_cGrKoqdfhw4qSp_JQCavvEjaxXk0OwGPnxyEcO60AzKQK_XDtp7H3DdkFXcrXgIddeUs$
> > > > > >> > > > > > > > > > > > >>
> > > > > >> > > > > > > > > > > > >> *The meeting covered both topics from
> the
> > > > doc.
> > > > > >> > > > > > Direction-level
> > > > > >> > > > > > > > > > > alignment
> > > > > >> > > > > > > > > > > > >> was reached on the headline pieces;
> > details
> > > > > >> remain
> > > > > >> > for
> > > > > >> > > > PR
> > > > > >> > > > > > > review
> > > > > >> > > > > > > > > or
> > > > > >> > > > > > > > > > > > >> follow-up sessions.*
> > > > > >> > > > > > > > > > > > >>
> > > > > >> > > > > > > > > > > > >> *Topic 1 — Persistence schema redesign*
> > > > > >> > > > > > > > > > > > >> Idea-level alignment on consolidating
> > > > per-type
> > > > > >> > tables
> > > > > >> > > > > > > > > > > > >> (scan_metrics_report,
> > > > > >> > > > > > > > > > > > >> commit_metrics_report) into a single
> > > > > >> metrics_report
> > > > > >> > > > table.
> > > > > >> > > > > > The
> > > > > >> > > > > > > > > > > > motivating
> > > > > >> > > > > > > > > > > > >> cost is the surface area added by every
> > new
> > > > > >> metric
> > > > > >> > > type
> > > > > >> > > > > > today:
> > > > > >> > > > > > > > new
> > > > > >> > > > > > > > > > > > table,
> > > > > >> > > > > > > > > > > > >> SPI method, record class, model,
> > converter,
> > > > > >> schema
> > > > > >> > > > > > migration.
> > > > > >> > > > > > > > > > > > >>
> > > > > >> > > > > > > > > > > > >> Most schema details are deferred to the
> > > > schema
> > > > > >> PR. A
> > > > > >> > > few
> > > > > >> > > > > > > > specific
> > > > > >> > > > > > > > > > > points
> > > > > >> > > > > > > > > > > > >> came up:
> > > > > >> > > > > > > > > > > > >> •   metric_schema_version: Yufei
> prefers
> > > > > dropping
> > > > > >> > it,
> > > > > >> > > > > since
> > > > > >> > > > > > > > there
> > > > > >> > > > > > > > > is
> > > > > >> > > > > > > > > > > no
> > > > > >> > > > > > > > > > > > >> spec-level concept of metrics
> versioning
> > > > today
> > > > > >> and
> > > > > >> > it
> > > > > >> > > is
> > > > > >> > > > > > hard
> > > > > >> > > > > > > to
> > > > > >> > > > > > > > > > > define
> > > > > >> > > > > > > > > > > > >> unilaterally. Robert prefers keeping
> it,
> > > > given
> > > > > >> IRC
> > > > > >> > v2
> > > > > >> > > is
> > > > > >> > > > > > > coming
> > > > > >> > > > > > > > > and
> > > > > >> > > > > > > > > > > the
> > > > > >> > > > > > > > > > > > >> schema should be considered against its
> > > > likely
> > > > > >> > shape;
> > > > > >> > > > > Robert
> > > > > >> > > > > > > > also
> > > > > >> > > > > > > > > > > raised
> > > > > >> > > > > > > > > > > > >> how to differentiate various payload
> > formats
> > > > if
> > > > > >> any.
> > > > > >> > > > EJ's
> > > > > >> > > > > > read
> > > > > >> > > > > > > > is
> > > > > >> > > > > > > > > > that
> > > > > >> > > > > > > > > > > > >> this
> > > > > >> > > > > > > > > > > > >> is a two-way-door decision. We can
> start
> > > > > without
> > > > > >> the
> > > > > >> > > > > field,
> > > > > >> > > > > > > and
> > > > > >> > > > > > > > if
> > > > > >> > > > > > > > > > IRC
> > > > > >> > > > > > > > > > > > v2
> > > > > >> > > > > > > > > > > > >> changes the shape we would likely roll
> a
> > > > > >> > corresponding
> > > > > >> > > > new
> > > > > >> > > > > > > > schema
> > > > > >> > > > > > > > > > > > anyway,
> > > > > >> > > > > > > > > > > > >> which is not particularly costly.
> > > > > >> > > > > > > > > > > > >> •   Payload format: Robert pointed out
> > that
> > > > > >> future
> > > > > >> > > > formats
> > > > > >> > > > > > > > beyond
> > > > > >> > > > > > > > > > JSON
> > > > > >> > > > > > > > > > > > may
> > > > > >> > > > > > > > > > > > >> be worth supporting. The exact shape is
> > > > > deferred
> > > > > >> to
> > > > > >> > > the
> > > > > >> > > > > > schema
> > > > > >> > > > > > > > > > > > discussion.
> > > > > >> > > > > > > > > > > > >> •   Partition strategy: Anand suggested
> > > > monthly
> > > > > >> > > > > partitioning
> > > > > >> > > > > > > > based
> > > > > >> > > > > > > > > > on
> > > > > >> > > > > > > > > > > > his
> > > > > >> > > > > > > > > > > > >> experience as potentially helpful at
> > scale.
> > > > > >> > > > > > > > > > > > >>
> > > > > >> > > > > > > > > > > > >> *Topic 2 — Where metrics ingestion and
> > > > storage
> > > > > >> > belong*
> > > > > >> > > > > > > > > > > > >> Idea-level alignment that metrics
> should
> > be a
> > > > > >> > > separated
> > > > > >> > > > > SPI
> > > > > >> > > > > > > from
> > > > > >> > > > > > > > > the
> > > > > >> > > > > > > > > > > > >> entity
> > > > > >> > > > > > > > > > > > >> persistence stack. Two reasons
> surfaced:
> > (a)
> > > > > >> > workloads
> > > > > >> > > > and
> > > > > >> > > > > > > > > > capability
> > > > > >> > > > > > > > > > > > >> requirements diverge enough that
> coupling
> > > > them
> > > > > >> > creates
> > > > > >> > > > > > > > artificial
> > > > > >> > > > > > > > > > > > >> constraints, and (b) admin experience
> > > > improves
> > > > > >> when
> > > > > >> > > > > metrics
> > > > > >> > > > > > > has
> > > > > >> > > > > > > > > its
> > > > > >> > > > > > > > > > > own
> > > > > >> > > > > > > > > > > > >> bootstrap, retention, and lifecycle.
> > Dmitri
> > > > > noted
> > > > > >> > > > Polaris
> > > > > >> > > > > > > being
> > > > > >> > > > > > > > a
> > > > > >> > > > > > > > > > > > platform
> > > > > >> > > > > > > > > > > > >> should have the flexibility to support
> > > > > different
> > > > > >> > > > > persistence
> > > > > >> > > > > > > > > > backends
> > > > > >> > > > > > > > > > > > per
> > > > > >> > > > > > > > > > > > >> concern, and pointed to a concrete next
> > step
> > > > of
> > > > > >> > > > separating
> > > > > >> > > > > > the
> > > > > >> > > > > > > > > JDBC
> > > > > >> > > > > > > > > > > > >> bootstrap for metrics from the
> metastore
> > > > > >> bootstrap.
> > > > > >> > > > Robert
> > > > > >> > > > > > > > > proposed
> > > > > >> > > > > > > > > > an
> > > > > >> > > > > > > > > > > > >> additional UX extension: detect an
> > > > > unbootstrapped
> > > > > >> > > > metrics
> > > > > >> > > > > > > store
> > > > > >> > > > > > > > on
> > > > > >> > > > > > > > > > > first
> > > > > >> > > > > > > > > > > > >> use and auto-bootstrap rather than
> > requiring
> > > > an
> > > > > >> > > explicit
> > > > > >> > > > > > > manual
> > > > > >> > > > > > > > > > > > bootstrap
> > > > > >> > > > > > > > > > > > >> step.
> > > > > >> > > > > > > > > > > > >> The meeting also confirmed that Polaris
> > > > metrics
> > > > > >> can
> > > > > >> > > > start
> > > > > >> > > > > > > small
> > > > > >> > > > > > > > > and
> > > > > >> > > > > > > > > > > stay
> > > > > >> > > > > > > > > > > > >> Iceberg-focused. Naming and persistence
> > > > schema
> > > > > >> can
> > > > > >> > > lean
> > > > > >> > > > > > > > > > > > Iceberg-specific.
> > > > > >> > > > > > > > > > > > >> If a future expansion to generic-table
> > > > metrics
> > > > > or
> > > > > >> > > > > > operational
> > > > > >> > > > > > > > > > metrics
> > > > > >> > > > > > > > > > > > >> arrives, an abstraction layer can be
> > built on
> > > > > >> top of
> > > > > >> > > the
> > > > > >> > > > > > > Iceberg
> > > > > >> > > > > > > > > > > metrics
> > > > > >> > > > > > > > > > > > >> reporter at that point. Robert remains
> > on the
> > > > > >> fence
> > > > > >> > > and
> > > > > >> > > > > > would
> > > > > >> > > > > > > > > prefer
> > > > > >> > > > > > > > > > > > >> something more generic but did not
> block
> > the
> > > > > >> > > direction;
> > > > > >> > > > > > > Dmitri's
> > > > > >> > > > > > > > > > read
> > > > > >> > > > > > > > > > > > was
> > > > > >> > > > > > > > > > > > >> that the proposed framework already has
> > > > enough
> > > > > >> > > > flexibility
> > > > > >> > > > > > to
> > > > > >> > > > > > > > > absorb
> > > > > >> > > > > > > > > > > > >> future
> > > > > >> > > > > > > > > > > > >> expansion.
> > > > > >> > > > > > > > > > > > >>
> > > > > >> > > > > > > > > > > > >> The Trade-offs and Proposed structure
> > > > sections
> > > > > in
> > > > > >> > the
> > > > > >> > > > doc
> > > > > >> > > > > > were
> > > > > >> > > > > > > > not
> > > > > >> > > > > > > > > > > > >> reviewed
> > > > > >> > > > > > > > > > > > >> in detail. They remain open for either
> > the
> > > > next
> > > > > >> sync
> > > > > >> > > or
> > > > > >> > > > PR
> > > > > >> > > > > > > > review.
> > > > > >> > > > > > > > > > > > >>
> > > > > >> > > > > > > > > > > > >> *Cross-cutting alignment —
> > battery-included
> > > > > plus
> > > > > >> > > > > pluggable*
> > > > > >> > > > > > > > > > > > >> A common philosophy emerged from the
> > > > > discussion.
> > > > > >> EJ
> > > > > >> > > > > > summarized
> > > > > >> > > > > > > > it
> > > > > >> > > > > > > > > > as:
> > > > > >> > > > > > > > > > > > >> Polaris should provide a
> > battery-included UX
> > > > > for
> > > > > >> > > > beginners
> > > > > >> > > > > > and
> > > > > >> > > > > > > > the
> > > > > >> > > > > > > > > > > > >> flexibility for advanced users to swap
> > the
> > > > > >> included
> > > > > >> > > > > battery
> > > > > >> > > > > > > for
> > > > > >> > > > > > > > > > > > something
> > > > > >> > > > > > > > > > > > >> more powerful or tailored to their use
> > case.
> > > > > The
> > > > > >> SPI
> > > > > >> > > > > design
> > > > > >> > > > > > > > needs
> > > > > >> > > > > > > > > to
> > > > > >> > > > > > > > > > > > >> enable
> > > > > >> > > > > > > > > > > > >> both.
> > > > > >> > > > > > > > > > > > >>
> > > > > >> > > > > > > > > > > > >> The inputs that shaped this framing:
> > > > > >> > > > > > > > > > > > >> •   Anand described how his team uses
> the
> > > > > current
> > > > > >> > > > metrics
> > > > > >> > > > > > > > > > persistence
> > > > > >> > > > > > > > > > > > >> (three metrics consumers in v1.4).
> > > > > >> > > > > > > > > > > > >> •   Yufei raised Grafana and dashboard
> > > > > >> integrations
> > > > > >> > > as a
> > > > > >> > > > > > > > > destination
> > > > > >> > > > > > > > > > > use
> > > > > >> > > > > > > > > > > > >> case beyond the default.
> > > > > >> > > > > > > > > > > > >> •   Robert called out that the current
> > design
> > > > > is
> > > > > >> > more
> > > > > >> > > > > > > > > JDBC-focused.
> > > > > >> > > > > > > > > > > > >>
> > > > > >> > > > > > > > > > > > >> Two concrete instances:
> > > > > >> > > > > > > > > > > > >> •   Async metrics intake: Yufei's
> initial
> > > > > >> position
> > > > > >> > was
> > > > > >> > > > > that
> > > > > >> > > > > > > > async
> > > > > >> > > > > > > > > > > should
> > > > > >> > > > > > > > > > > > >> largely live on the producer side and
> > there
> > > > is
> > > > > >> not
> > > > > >> > > much
> > > > > >> > > > > > > Polaris
> > > > > >> > > > > > > > > can
> > > > > >> > > > > > > > > > > do.
> > > > > >> > > > > > > > > > > > >> Robert suggested a Polaris-side default
> > is
> > > > > doable
> > > > > >> > via
> > > > > >> > > > > > Vert.x.
> > > > > >> > > > > > > > > Dmitri
> > > > > >> > > > > > > > > > > > >> agreed
> > > > > >> > > > > > > > > > > > >> the direction is worth exploring. The
> > meeting
> > > > > >> > > converged
> > > > > >> > > > > on a
> > > > > >> > > > > > > > > > > > >> battery-included default (likely
> > > > Vert.x-backed)
> > > > > >> with
> > > > > >> > > an
> > > > > >> > > > > SPI
> > > > > >> > > > > > > > shape
> > > > > >> > > > > > > > > > that
> > > > > >> > > > > > > > > > > > >> lets
> > > > > >> > > > > > > > > > > > >> power users route to a more scalable
> > backend
> > > > > >> > > (k8s-hosted
> > > > > >> > > > > > > queue,
> > > > > >> > > > > > > > > AWS
> > > > > >> > > > > > > > > > > SQS,
> > > > > >> > > > > > > > > > > > >> etc.).
> > > > > >> > > > > > > > > > > > >> •   Pluggable destinations: combining
> > Yufei's
> > > > > >> > > dashboard
> > > > > >> > > > > use
> > > > > >> > > > > > > case
> > > > > >> > > > > > > > > > with
> > > > > >> > > > > > > > > > > > >> Robert's JDBC-focused call-out, the
> > meeting
> > > > > >> agreed
> > > > > >> > the
> > > > > >> > > > SPI
> > > > > >> > > > > > > > should
> > > > > >> > > > > > > > > be
> > > > > >> > > > > > > > > > > > >> structured for multiple sinks so
> > integrations
> > > > > >> become
> > > > > >> > > > impl
> > > > > >> > > > > > > > choices
> > > > > >> > > > > > > > > > > rather
> > > > > >> > > > > > > > > > > > >> than architectural changes.
> > > > > >> > > > > > > > > > > > >>
> > > > > >> > > > > > > > > > > > >> The battery-included default is most
> > likely
> > > > to
> > > > > >> use
> > > > > >> > the
> > > > > >> > > > > > > existing
> > > > > >> > > > > > > > > > > > >> JDBC-backed
> > > > > >> > > > > > > > > > > > >> approach.
> > > > > >> > > > > > > > > > > > >>
> > > > > >> > > > > > > > > > > > >> *Direction (idea-level alignment)*
> > > > > >> > > > > > > > > > > > >> •   Single metrics_report table
> > consolidating
> > > > > >> > per-type
> > > > > >> > > > > > > metrics,
> > > > > >> > > > > > > > > > > > replacing
> > > > > >> > > > > > > > > > > > >> scan_metrics_report and
> > commit_metrics_report
> > > > > >> > > > > > > > > > > > >> •   Iceberg-focused naming and schema
> for
> > > > now,
> > > > > >> > revisit
> > > > > >> > > > if
> > > > > >> > > > > > > > > > > generic-table
> > > > > >> > > > > > > > > > > > or
> > > > > >> > > > > > > > > > > > >> operational metrics arrive
> > > > > >> > > > > > > > > > > > >> •   Metrics persistence as a separated
> > SPI,
> > > > not
> > > > > >> on
> > > > > >> > > > > > > > BasePersistence
> > > > > >> > > > > > > > > > > > >> •   Bootstrap path separated for
> metrics,
> > > > > >> > independent
> > > > > >> > > of
> > > > > >> > > > > > > > metastore
> > > > > >> > > > > > > > > > > > >> bootstrap
> > > > > >> > > > > > > > > > > > >> •   "Battery-included plus pluggable"
> as
> > the
> > > > > SPI
> > > > > >> > > design
> > > > > >> > > > > > > > philosophy
> > > > > >> > > > > > > > > > > > >>
> > > > > >> > > > > > > > > > > > >> *Open items*
> > > > > >> > > > > > > > > > > > >> •   Schema details:
> > metric_schema_version,
> > > > > >> payload
> > > > > >> > > > format,
> > > > > >> > > > > > IRC
> > > > > >> > > > > > > > v2
> > > > > >> > > > > > > > > > > > >> forward-compat shape
> > > > > >> > > > > > > > > > > > >> •   SPI design details — full review
> > either
> > > > in
> > > > > >> the
> > > > > >> > > next
> > > > > >> > > > > sync
> > > > > >> > > > > > > or
> > > > > >> > > > > > > > in
> > > > > >> > > > > > > > > > the
> > > > > >> > > > > > > > > > > > >> corresponding PR
> > > > > >> > > > > > > > > > > > >> •   Schema refactor PR ownership
> > > > > >> > > > > > > > > > > > >>
> > > > > >> > > > > > > > > > > > >> *Action items*
> > > > > >> > > > > > > > > > > > >> •   EJ to take a first stab at the SPI
> > design
> > > > > and
> > > > > >> > > > > > potentially
> > > > > >> > > > > > > > > > partner
> > > > > >> > > > > > > > > > > > with
> > > > > >> > > > > > > > > > > > >> Anand to incorporate the lessons
> learned
> > from
> > > > > the
> > > > > >> > > > existing
> > > > > >> > > > > > > > > reporter
> > > > > >> > > > > > > > > > > and
> > > > > >> > > > > > > > > > > > >> persistence work.
> > > > > >> > > > > > > > > > > > >> •   Schema refactor PR ownership is not
> > yet
> > > > > >> decided.
> > > > > >> > > If
> > > > > >> > > > > > anyone
> > > > > >> > > > > > > > is
> > > > > >> > > > > > > > > > > > >> interested in driving it, reply on this
> > > > thread.
> > > > > >> > > > > > > > > > > > >> •   JB to schedule the next sync,
> > tentatively
> > > > > in
> > > > > >> two
> > > > > >> > > > > weeks.
> > > > > >> > > > > > > > > > > > >>
> > > > > >> > > > > > > > > > > > >> -ej
> > > > > >> > > > > > > > > > > > >>
> > > > > >> > > > > > > > > > > > >> On Mon, Apr 27, 2026 at 3:07 PM EJ
> Wang <
> > > > > >> > > > > > > > > > > ej.wang.dev.interest@gmail.com
> > > > > >> > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > >> wrote:
> > > > > >> > > > > > > > > > > > >>
> > > > > >> > > > > > > > > > > > >> > Thanks Yufei for the +1.
> > > > > >> > > > > > > > > > > > >> >
> > > > > >> > > > > > > > > > > > >> > JB, could you help add a biweekly
> > metrics
> > > > > >> > > architecture
> > > > > >> > > > > > sync
> > > > > >> > > > > > > to
> > > > > >> > > > > > > > > the
> > > > > >> > > > > > > > > > > > >> Polaris
> > > > > >> > > > > > > > > > > > >> > community calendar? I'm thinking
> > Thursdays
> > > > at
> > > > > >> > 9-10am
> > > > > >> > > > PT,
> > > > > >> > > > > > on
> > > > > >> > > > > > > > the
> > > > > >> > > > > > > > > > > > >> off-weeks
> > > > > >> > > > > > > > > > > > >> > from the community meeting (starting
> > May
> > > > 7),
> > > > > 60
> > > > > >> > > > minutes.
> > > > > >> > > > > > > > > > > > >> >
> > > > > >> > > > > > > > > > > > >> > Here's a rough agenda to work through
> > over
> > > > > the
> > > > > >> > first
> > > > > >> > > > few
> > > > > >> > > > > > > > > sessions,
> > > > > >> > > > > > > > > > > > >> grouped
> > > > > >> > > > > > > > > > > > >> > by priority:
> > > > > >> > > > > > > > > > > > >> >
> > > > > >> > > > > > > > > > > > >> > *First: foundational direction*
> > > > > >> > > > > > > > > > > > >> >
> > > > > >> > > > > > > > > > > > >> > 1.  MetricsPersistence: public SPI or
> > > > > internal
> > > > > >> > > > > > > implementation
> > > > > >> > > > > > > > > > > detail?
> > > > > >> > > > > > > > > > > > >> >    •   Marked @Beta, javadoc calls
> it a
> > > > > >> "Service
> > > > > >> > > > > Provider
> > > > > >> > > > > > > > > > > Interface",
> > > > > >> > > > > > > > > > > > >> but
> > > > > >> > > > > > > > > > > > >> > only one consumer
> > > > (JdbcBasePersistenceImpl),
> > > > > >> lives
> > > > > >> > > on
> > > > > >> > > > > > > > > > > BasePersistence.
> > > > > >> > > > > > > > > > > > >> If
> > > > > >> > > > > > > > > > > > >> > demoted to a private helper inside a
> > > > > persisting
> > > > > >> > > > reporter
> > > > > >> > > > > > > impl,
> > > > > >> > > > > > > > > > most
> > > > > >> > > > > > > > > > > > >> > downstream design decisions become
> > > > > >> implementation
> > > > > >> > > > > details
> > > > > >> > > > > > > > rather
> > > > > >> > > > > > > > > > > than
> > > > > >> > > > > > > > > > > > >> > contract questions.
> > > > > >> > > > > > > > > > > > >> >
> > > > > >> > > > > > > > > > > > >> > 2.  Persistence schema redesign
> > > > > >> > > > > > > > > > > > >> >    •   Current two-table layout
> > > > > >> > > (scan_metrics_report,
> > > > > >> > > > > > > > > > > > >> > commit_metrics_report) with ~25
> > flattened
> > > > > >> columns
> > > > > >> > > > each.
> > > > > >> > > > > > > Every
> > > > > >> > > > > > > > > new
> > > > > >> > > > > > > > > > > > metric
> > > > > >> > > > > > > > > > > > >> > type requires a new table, SPI
> method,
> > > > record
> > > > > >> > class,
> > > > > >> > > > > > model,
> > > > > >> > > > > > > > > > > converter,
> > > > > >> > > > > > > > > > > > >> and
> > > > > >> > > > > > > > > > > > >> > schema migration. Direction to
> explore:
> > > > > single
> > > > > >> > table
> > > > > >> > > > > with
> > > > > >> > > > > > > > > > > metric_type
> > > > > >> > > > > > > > > > > > >> enum,
> > > > > >> > > > > > > > > > > > >> > schema_version, and JSON payload
> > column.
> > > > > >> > > > > > > > > > > > >> >
> > > > > >> > > > > > > > > > > > >> > *Second: design details once
> direction
> > is
> > > > > set*
> > > > > >> > > > > > > > > > > > >> >
> > > > > >> > > > > > > > > > > > >> > 3.  Partition key strategy
> > > > > >> > > > > > > > > > > > >> >    •   Single-table design means scan
> > > > metrics
> > > > > >> at
> > > > > >> > > scale
> > > > > >> > > > > > will
> > > > > >> > > > > > > > have
> > > > > >> > > > > > > > > > > high
> > > > > >> > > > > > > > > > > > >> > write concurrency per table. Schema
> > needs
> > > > to
> > > > > >> > expose
> > > > > >> > > > > enough
> > > > > >> > > > > > > > > > structure
> > > > > >> > > > > > > > > > > > for
> > > > > >> > > > > > > > > > > > >> > backends to shard by entity or time
> > range.
> > > > > >> > > > > > > > > > > > >> >
> > > > > >> > > > > > > > > > > > >> > 4.  Read/write path consistency
> > > > > >> > > > > > > > > > > > >> >    •   Writes go through
> > > > > PolarisMetricsManager
> > > > > >> on
> > > > > >> > > > > > > > > > MetaStoreManager.
> > > > > >> > > > > > > > > > > > >> Reads
> > > > > >> > > > > > > > > > > > >> > bypass MetaStoreManager and go
> > straight to
> > > > > >> > > > > > BasePersistence,
> > > > > >> > > > > > > > > > > excluding
> > > > > >> > > > > > > > > > > > >> > non-JDBC backends from the read API.
> > > > > >> > > > > > > > > > > > >> >
> > > > > >> > > > > > > > > > > > >> > *Third: cleanup and alignment*
> > > > > >> > > > > > > > > > > > >> >
> > > > > >> > > > > > > > > > > > >> > 5.  PolarisMetricsReporter naming
> > > > > >> > > > > > > > > > > > >> >    •   Only handles IRC
> > > > > >> (ScanReport/CommitReport),
> > > > > >> > > > > doesn't
> > > > > >> > > > > > > > cover
> > > > > >> > > > > > > > > > > > generic
> > > > > >> > > > > > > > > > > > >> > tables or operational metrics. Name
> is
> > > > > broader
> > > > > >> > than
> > > > > >> > > > > scope.
> > > > > >> > > > > > > > > > > > >> >
> > > > > >> > > > > > > > > > > > >> > 6.  PolarisMetricsManager facade
> > > > passthrough
> > > > > >> > > > > > > > > > > > >> >    •   Entire default method is
> > > > > >> > > > > > > > > > > > >>
> callCtx.getMetaStore().writeScanReport().
> > > > > >> > > > > > > > > > > > >> > Zero logic, passes Level 1 straight
> > through
> > > > > to
> > > > > >> > Level
> > > > > >> > > > 3.
> > > > > >> > > > > > Same
> > > > > >> > > > > > > > > > > > >> anti-pattern
> > > > > >> > > > > > > > > > > > >> > as PolarisEventManager.
> > > > > >> > > > > > > > > > > > >> >
> > > > > >> > > > > > > > > > > > >> > 7.  Iceberg community alignment
> > > > > >> > > > > > > > > > > > >> >    •   Payload-type extension needs
> > > > > discussion
> > > > > >> on
> > > > > >> > > > > > > dev@iceberg.
> > > > > >> > > > > > > > > > > > >> obelix74's
> > > > > >> > > > > > > > > > > > >> > Feb thread got zero replies. Needs a
> > > > > committer
> > > > > >> > > voice.
> > > > > >> > > > > > > > > > > > >> >
> > > > > >> > > > > > > > > > > > >> > Lets confirm prioritization in the
> > first
> > > > > >> session.
> > > > > >> > > > > > > > > > > > >> >
> > > > > >> > > > > > > > > > > > >> > -ej
> > > > > >> > > > > > > > > > > > >> >
> > > > > >> > > > > > > > > > > > >> > On Tue, Apr 21, 2026 at 3:18 PM Yufei
> > Gu <
> > > > > >> > > > > > > > flyrain000@gmail.com>
> > > > > >> > > > > > > > > > > > wrote:
> > > > > >> > > > > > > > > > > > >> >
> > > > > >> > > > > > > > > > > > >> >> Thanks everyone for continuing to
> > drive
> > > > this
> > > > > >> > > > forward. I
> > > > > >> > > > > > > agree
> > > > > >> > > > > > > > > > that
> > > > > >> > > > > > > > > > > > the
> > > > > >> > > > > > > > > > > > >> >> problem is getting complex enough
> > that a
> > > > > more
> > > > > >> > > > > structured
> > > > > >> > > > > > > > > > discussion
> > > > > >> > > > > > > > > > > > >> would
> > > > > >> > > > > > > > > > > > >> >> help.
> > > > > >> > > > > > > > > > > > >> >>
> > > > > >> > > > > > > > > > > > >> >> +1 on setting up a biweekly sync for
> > the
> > > > > >> metrics
> > > > > >> > > > > > > > architecture.
> > > > > >> > > > > > > > > > I’m
> > > > > >> > > > > > > > > > > > >> happy
> > > > > >> > > > > > > > > > > > >> >> to
> > > > > >> > > > > > > > > > > > >> >> join.
> > > > > >> > > > > > > > > > > > >> >>
> > > > > >> > > > > > > > > > > > >> >> Yufei
> > > > > >> > > > > > > > > > > > >> >>
> > > > > >> > > > > > > > > > > > >> >>
> > > > > >> > > > > > > > > > > > >> >> On Tue, Apr 21, 2026 at 2:34 PM EJ
> > Wang <
> > > > > >> > > > > > > > > > > > >> ej.wang.dev.interest@gmail.com>
> > > > > >> > > > > > > > > > > > >> >> wrote:
> > > > > >> > > > > > > > > > > > >> >>
> > > > > >> > > > > > > > > > > > >> >> > Also, I've been looking more
> > closely at
> > > > > the
> > > > > >> > > > > > *persistence
> > > > > >> > > > > > > > > schema
> > > > > >> > > > > > > > > > > in
> > > > > >> > > > > > > > > > > > >> the
> > > > > >> > > > > > > > > > > > >> >> > current metrics work*, and I think
> > > > > there's a
> > > > > >> > > > > structural
> > > > > >> > > > > > > > > > rigidity
> > > > > >> > > > > > > > > > > > >> problem
> > > > > >> > > > > > > > > > > > >> >> > worth raising before the shape
> gets
> > > > locked
> > > > > >> in.
> > > > > >> > > > > > > > > > > > >> >> >
> > > > > >> > > > > > > > > > > > >> >> > Right now we have two separate
> > tables
> > > > > >> > > > > > > (scan_metrics_report
> > > > > >> > > > > > > > > and
> > > > > >> > > > > > > > > > > > >> >> > commit_metrics_report), each with
> > ~25
> > > > > >> flattened
> > > > > >> > > > > columns
> > > > > >> > > > > > > > that
> > > > > >> > > > > > > > > > > > directly
> > > > > >> > > > > > > > > > > > >> >> > mirror the Iceberg report fields.
> > The
> > > > SPI
> > > > > >> > follows
> > > > > >> > > > the
> > > > > >> > > > > > > same
> > > > > >> > > > > > > > > > split:
> > > > > >> > > > > > > > > > > > >> >> > writeScanReport and
> > writeCommitReport as
> > > > > >> > separate
> > > > > >> > > > > > > methods,
> > > > > >> > > > > > > > > with
> > > > > >> > > > > > > > > > > > >> per-type
> > > > > >> > > > > > > > > > > > >> >> > record classes, converters, and
> > model
> > > > > >> objects.
> > > > > >> > > *The
> > > > > >> > > > > > > > practical
> > > > > >> > > > > > > > > > > cost:
> > > > > >> > > > > > > > > > > > >> >> > adding a new metric type
> > (operational
> > > > > >> metrics,
> > > > > >> > > for
> > > > > >> > > > > > > example)
> > > > > >> > > > > > > > > > > > requires
> > > > > >> > > > > > > > > > > > >> a
> > > > > >> > > > > > > > > > > > >> >> new
> > > > > >> > > > > > > > > > > > >> >> > table, a new SPI method, a new
> > record
> > > > > >> class, a
> > > > > >> > > new
> > > > > >> > > > > > model
> > > > > >> > > > > > > > > > class, a
> > > > > >> > > > > > > > > > > > new
> > > > > >> > > > > > > > > > > > >> >> > converter branch, and a schema
> > > > migration*.
> > > > > >> > > That's a
> > > > > >> > > > > lot
> > > > > >> > > > > > > of
> > > > > >> > > > > > > > > > > surface
> > > > > >> > > > > > > > > > > > >> area
> > > > > >> > > > > > > > > > > > >> >> > for what should be "one more kind
> of
> > > > > >> metric."
> > > > > >> > > > > > > > > > > > >> >> >
> > > > > >> > > > > > > > > > > > >> >> > *My bias* would be toward a single
> > > > metrics
> > > > > >> > table
> > > > > >> > > > with
> > > > > >> > > > > > *a
> > > > > >> > > > > > > > > typed
> > > > > >> > > > > > > > > > > JSON
> > > > > >> > > > > > > > > > > > >> >> > payload*. Something like:
> > metric_type
> > > > > >> (enum),
> > > > > >> > > > > > entity_id,
> > > > > >> > > > > > > > > > > > >> >> > table_identifier, snapshot_id
> > > > (nullable),
> > > > > >> > > > > received_ts,
> > > > > >> > > > > > > > > > > > >> schema_version,
> > > > > >> > > > > > > > > > > > >> >> and
> > > > > >> > > > > > > > > > > > >> >> > a payload column for the
> > metric-specific
> > > > > >> data.
> > > > > >> > > The
> > > > > >> > > > > > > > > metric_type
> > > > > >> > > > > > > > > > +
> > > > > >> > > > > > > > > > > > >> >> > schema_version pair gives us a
> > > > > >> > forward-compatible
> > > > > >> > > > > > > contract
> > > > > >> > > > > > > > > for
> > > > > >> > > > > > > > > > > the
> > > > > >> > > > > > > > > > > > >> >> payload
> > > > > >> > > > > > > > > > > > >> >> > shape. Adding a new metric type
> > becomes
> > > > an
> > > > > >> enum
> > > > > >> > > > value
> > > > > >> > > > > > > and a
> > > > > >> > > > > > > > > > > payload
> > > > > >> > > > > > > > > > > > >> >> schema,
> > > > > >> > > > > > > > > > > > >> >> > not a schema migration. One thing
> I
> > > > think
> > > > > we
> > > > > >> > need
> > > > > >> > > > to
> > > > > >> > > > > be
> > > > > >> > > > > > > > > > > deliberate
> > > > > >> > > > > > > > > > > > >> >> about is
> > > > > >> > > > > > > > > > > > >> >> > the partition key design. If all
> > metric
> > > > > >> types
> > > > > >> > > land
> > > > > >> > > > in
> > > > > >> > > > > > one
> > > > > >> > > > > > > > > > table,
> > > > > >> > > > > > > > > > > > scan
> > > > > >> > > > > > > > > > > > >> >> > metrics at scale (high
> concurrency,
> > high
> > > > > >> > > frequency
> > > > > >> > > > > > across
> > > > > >> > > > > > > > > many
> > > > > >> > > > > > > > > > > > >> tables)
> > > > > >> > > > > > > > > > > > >> >> > could easily create hot
> partitions.
> > We'd
> > > > > >> want
> > > > > >> > the
> > > > > >> > > > > > > > persistence
> > > > > >> > > > > > > > > > > layer
> > > > > >> > > > > > > > > > > > >> to
> > > > > >> > > > > > > > > > > > >> >> be
> > > > > >> > > > > > > > > > > > >> >> > able to shard by entity or time
> > range,
> > > > and
> > > > > >> that
> > > > > >> > > > means
> > > > > >> > > > > > the
> > > > > >> > > > > > > > > > logical
> > > > > >> > > > > > > > > > > > >> schema
> > > > > >> > > > > > > > > > > > >> >> > needs to expose enough structure
> for
> > > > > >> backends
> > > > > >> > to
> > > > > >> > > > > > > partition
> > > > > >> > > > > > > > > on.
> > > > > >> > > > > > > > > > I
> > > > > >> > > > > > > > > > > > >> don't
> > > > > >> > > > > > > > > > > > >> >> > think the current flattened layout
> > gives
> > > > > us
> > > > > >> > that.
> > > > > >> > > > > > > > > > > > >> >> >
> > > > > >> > > > > > > > > > > > >> >> > This is getting complex enough
> that
> > I
> > > > > don't
> > > > > >> > think
> > > > > >> > > > > > ad-hoc
> > > > > >> > > > > > > > > PR/ML
> > > > > >> > > > > > > > > > > > >> threads
> > > > > >> > > > > > > > > > > > >> >> > will converge well. *Would people
> be
> > > > open
> > > > > >> to a
> > > > > >> > > > > biweekly
> > > > > >> > > > > > > > sync
> > > > > >> > > > > > > > > > for
> > > > > >> > > > > > > > > > > > >> metrics
> > > > > >> > > > > > > > > > > > >> >> > architecture?* I think 30 minutes
> > every
> > > > > two
> > > > > >> > weeks
> > > > > >> > > > > with
> > > > > >> > > > > > > > > > interested
> > > > > >> > > > > > > > > > > > >> >> parties
> > > > > >> > > > > > > > > > > > >> >> > would be enough to work through
> the
> > > > > schema,
> > > > > >> SPI
> > > > > >> > > > > shape,
> > > > > >> > > > > > > and
> > > > > >> > > > > > > > > read
> > > > > >> > > > > > > > > > > API
> > > > > >> > > > > > > > > > > > >> >> design
> > > > > >> > > > > > > > > > > > >> >> > together. Happy to help set that
> up.
> > > > > >> > > > > > > > > > > > >> >> >
> > > > > >> > > > > > > > > > > > >> >> > -ej
> > > > > >> > > > > > > > > > > > >> >> >
> > > > > >> > > > > > > > > > > > >> >> > On Mon, Apr 20, 2026 at 2:19 PM EJ
> > Wang
> > > > <
> > > > > >> > > > > > > > > > > > >> ej.wang.dev.interest@gmail.com
> > > > > >> > > > > > > > > > > > >> >> >
> > > > > >> > > > > > > > > > > > >> >> > wrote:
> > > > > >> > > > > > > > > > > > >> >> >
> > > > > >> > > > > > > > > > > > >> >> >> Reviewed #4115, left a comment on
> > the
> > > > > code
> > > > > >> > > > > > organization
> > > > > >> > > > > > > > > side.
> > > > > >> > > > > > > > > > > > >> >> >>
> > > > > >> > > > > > > > > > > > >> >> >> One thing stood out: the metrics
> > write
> > > > > path
> > > > > >> > > enters
> > > > > >> > > > > > > through
> > > > > >> > > > > > > > > > > > >> >> >> PolarisMetricsManager on
> > > > > MetaStoreManager,
> > > > > >> but
> > > > > >> > > the
> > > > > >> > > > > new
> > > > > >> > > > > > > > read
> > > > > >> > > > > > > > > > path
> > > > > >> > > > > > > > > > > > >> >> bypasses
> > > > > >> > > > > > > > > > > > >> >> >> MetaStoreManager entirely and
> goes
> > > > > >> straight to
> > > > > >> > > > > > > > > BasePersistence
> > > > > >> > > > > > > > > > > via
> > > > > >> > > > > > > > > > > > >> >> >> callContext.getMetaStore(). That
> > means
> > > > > the
> > > > > >> > read
> > > > > >> > > > API
> > > > > >> > > > > > only
> > > > > >> > > > > > > > > works
> > > > > >> > > > > > > > > > > for
> > > > > >> > > > > > > > > > > > >> >> backends
> > > > > >> > > > > > > > > > > > >> >> >> that implement BasePersistence.
> > NoSQL
> > > > and
> > > > > >> > remote
> > > > > >> > > > > > > backends
> > > > > >> > > > > > > > > > can't
> > > > > >> > > > > > > > > > > > >> >> participate.
> > > > > >> > > > > > > > > > > > >> >> >>
> > > > > >> > > > > > > > > > > > >> >> >> Stepping back, I think the
> metrics
> > > > > >> subsystem
> > > > > >> > is
> > > > > >> > > > > > growing
> > > > > >> > > > > > > > into
> > > > > >> > > > > > > > > > > > >> something
> > > > > >> > > > > > > > > > > > >> >> >> real (write + read + REST API +
> > AuthZ +
> > > > > >> > > > pagination)
> > > > > >> > > > > > *but
> > > > > >> > > > > > > > the
> > > > > >> > > > > > > > > > > > >> >> persistence
> > > > > >> > > > > > > > > > > > >> >> >> side is split across two layers
> in
> > a
> > > > way
> > > > > >> > that's
> > > > > >> > > > hard
> > > > > >> > > > > > to
> > > > > >> > > > > > > > > > > extend*. I
> > > > > >> > > > > > > > > > > > >> put
> > > > > >> > > > > > > > > > > > >> >> >> together two diagrams to show
> what
> > I
> > > > mean
> > > > > >> (my
> > > > > >> > > best
> > > > > >> > > > > > > > effort).
> > > > > >> > > > > > > > > > > > >> >> >>
> > > > > >> > > > > > > > > > > > >> >> >> *Current state* (Diagram 1):
> three
> > > > > >> interfaces
> > > > > >> > at
> > > > > >> > > > > three
> > > > > >> > > > > > > > > > different
> > > > > >> > > > > > > > > > > > >> >> levels.
> > > > > >> > > > > > > > > > > > >> >> >> The engine-facing SPI
> > > > > >> (PolarisMetricsReporter)
> > > > > >> > > is
> > > > > >> > > > > > clean.
> > > > > >> > > > > > > > But
> > > > > >> > > > > > > > > > > > >> >> >> PolarisMetricsManager on
> > > > MetaStoreManager
> > > > > >> is a
> > > > > >> > > > > > > passthrough
> > > > > >> > > > > > > > > to
> > > > > >> > > > > > > > > > > > >> >> >> MetricsPersistence on
> > BasePersistence.
> > > > > The
> > > > > >> > @Beta
> > > > > >> > > > > > > > annotation
> > > > > >> > > > > > > > > > and
> > > > > >> > > > > > > > > > > > SPI
> > > > > >> > > > > > > > > > > > >> >> javadoc
> > > > > >> > > > > > > > > > > > >> >> >> are on the BasePersistence layer,
> > while
> > > > > the
> > > > > >> > > actual
> > > > > >> > > > > > > > extension
> > > > > >> > > > > > > > > > > > points
> > > > > >> > > > > > > > > > > > >> >> >> (PolarisMetricsReporter,
> > > > > >> > PolarisMetricsManager)
> > > > > >> > > > > carry
> > > > > >> > > > > > no
> > > > > >> > > > > > > > > > > stability
> > > > > >> > > > > > > > > > > > >> >> >> annotation. The write path goes
> > through
> > > > > the
> > > > > >> > > > > > > > MetaStoreManager
> > > > > >> > > > > > > > > > > > layer,
> > > > > >> > > > > > > > > > > > >> the
> > > > > >> > > > > > > > > > > > >> >> >> read path doesn't.
> > > > > >> > > > > > > > > > > > >> >> >>
> > > > > >> > > > > > > > > > > > >> >> >> *What I envision* (Diagram 2):
> two
> > SPIs
> > > > > at
> > > > > >> two
> > > > > >> > > > > levels.
> > > > > >> > > > > > > > > > > > >> >> >> PolarisMetricsReporter stays as
> the
> > > > > >> > > engine-facing
> > > > > >> > > > > SPI.
> > > > > >> > > > > > > > > > > > >> >> >> PolarisMetricsManager becomes the
> > > > > >> > backend-facing
> > > > > >> > > > SPI
> > > > > >> > > > > > > with
> > > > > >> > > > > > > > > both
> > > > > >> > > > > > > > > > > > write
> > > > > >> > > > > > > > > > > > >> >> and
> > > > > >> > > > > > > > > > > > >> >> >> read methods at the
> > MetaStoreManager
> > > > > level,
> > > > > >> > > where
> > > > > >> > > > > any
> > > > > >> > > > > > > > > backend
> > > > > >> > > > > > > > > > > > (JDBC,
> > > > > >> > > > > > > > > > > > >> >> NoSQL,
> > > > > >> > > > > > > > > > > > >> >> >> remote) can implement them.
> > > > > >> MetricsPersistence
> > > > > >> > > on
> > > > > >> > > > > > > > > > > BasePersistence
> > > > > >> > > > > > > > > > > > >> goes
> > > > > >> > > > > > > > > > > > >> >> >> away. Where metrics actually land
> > is an
> > > > > >> > > > > implementation
> > > > > >> > > > > > > > > detail,
> > > > > >> > > > > > > > > > > > not a
> > > > > >> > > > > > > > > > > > >> >> core
> > > > > >> > > > > > > > > > > > >> >> >> interface.
> > > > > >> > > > > > > > > > > > >> >> >>
> > > > > >> > > > > > > > > > > > >> >> >> *Minor naming thing*:
> > > > > >> PolarisMetricsReporter
> > > > > >> > is
> > > > > >> > > > > > broader
> > > > > >> > > > > > > > than
> > > > > >> > > > > > > > > > > what
> > > > > >> > > > > > > > > > > > it
> > > > > >> > > > > > > > > > > > >> >> >> actually handles. It only accepts
> > > > Iceberg
> > > > > >> REST
> > > > > >> > > > > Catalog
> > > > > >> > > > > > > > > metrics
> > > > > >> > > > > > > > > > > > >> >> (ScanReport,
> > > > > >> > > > > > > > > > > > >> >> >> CommitReport via MetricsReport).
> > > > Generic
> > > > > >> table
> > > > > >> > > > > metrics
> > > > > >> > > > > > > or
> > > > > >> > > > > > > > > > > > >> operational
> > > > > >> > > > > > > > > > > > >> >> >> metrics aren't in scope. Not
> > blocking,
> > > > > but
> > > > > >> > worth
> > > > > >> > > > > > noting
> > > > > >> > > > > > > if
> > > > > >> > > > > > > > > the
> > > > > >> > > > > > > > > > > > >> metrics
> > > > > >> > > > > > > > > > > > >> >> >> surface expands.
> > > > > >> > > > > > > > > > > > >> >> >>
> > > > > >> > > > > > > > > > > > >> >> >> *Rough sketch of how to get
> there*:
> > > > > >> > > > > > > > > > > > >> >> >>  1.  Add read methods to
> > > > > >> PolarisMetricsManager
> > > > > >> > > > > > > > > > (listScanReports,
> > > > > >> > > > > > > > > > > > >> >> >> listCommitReports) with default
> > no-op,
> > > > > >> same as
> > > > > >> > > the
> > > > > >> > > > > > > > existing
> > > > > >> > > > > > > > > > > write
> > > > > >> > > > > > > > > > > > >> >> methods.
> > > > > >> > > > > > > > > > > > >> >> >> (Probably make
> > PolarisMetricsManager
> > > > more
> > > > > >> > > explicit
> > > > > >> > > > > on
> > > > > >> > > > > > > > being
> > > > > >> > > > > > > > > > > > Iceberg
> > > > > >> > > > > > > > > > > > >> >> >> specific like package name or
> class
> > > > name
> > > > > >> etc.)
> > > > > >> > > > > > > > > > > > >> >> >>  2.  Wire MetricsReportsService
> > through
> > > > > >> > > > > > MetaStoreManager
> > > > > >> > > > > > > > > > instead
> > > > > >> > > > > > > > > > > > of
> > > > > >> > > > > > > > > > > > >> >> >> callContext.getMetaStore().
> > > > > >> > > > > > > > > > > > >> >> >>  3.  Extract metrics persistence
> > from
> > > > > >> > > > > > > > > JdbcBasePersistenceImpl
> > > > > >> > > > > > > > > > > into
> > > > > >> > > > > > > > > > > > >> its
> > > > > >> > > > > > > > > > > > >> >> >> own class. That file carries ~7
> > > > > >> > > responsibilities,
> > > > > >> > > > > > > metrics
> > > > > >> > > > > > > > > > being
> > > > > >> > > > > > > > > > > > one
> > > > > >> > > > > > > > > > > > >> of
> > > > > >> > > > > > > > > > > > >> >> them.
> > > > > >> > > > > > > > > > > > >> >> >>  4.  Remove MetricsPersistence
> from
> > > > > >> > > > BasePersistence.
> > > > > >> > > > > > > > > > > > >> >> >>
> > > > > >> > > > > > > > > > > > >> >> >> *None of this needs to happen in
> > #4115.
> > > > > >> But if
> > > > > >> > > the
> > > > > >> > > > > > > > direction
> > > > > >> > > > > > > > > > > makes
> > > > > >> > > > > > > > > > > > >> >> sense,
> > > > > >> > > > > > > > > > > > >> >> >> it would be good to align before
> > the
> > > > > >> metrics
> > > > > >> > > > surface
> > > > > >> > > > > > > grows
> > > > > >> > > > > > > > > > > > further.
> > > > > >> > > > > > > > > > > > >> >> Curious
> > > > > >> > > > > > > > > > > > >> >> >> what others think.*
> > > > > >> > > > > > > > > > > > >> >> >>
> > > > > >> > > > > > > > > > > > >> >> >> *My mental model note*: Level 1
> > > > > >> > > MetaStoreManager;
> > > > > >> > > > > > level
> > > > > >> > > > > > > 2
> > > > > >> > > > > > > > > > > > >> transactional
> > > > > >> > > > > > > > > > > > >> >> >> persistence; level 3 base
> > persistence
> > > > > >> > > > > > > > > > > > >> >> >>
> > > > > >> > > > > > > > > > > > >> >> >> Diagram 1
> > > > > >> > > > > > > > > > > > >> >> >> <
> > > > > >> > > > > > > > > > > > >> >>
> > > > > >> > > > > > > > > > > > >>
> > > > > >> > > > > > > > > > > >
> > > > > >> > > > > > > > > > >
> > > > > >> > > > > > > > > >
> > > > > >> > > > > > > > >
> > > > > >> > > > > > > >
> > > > > >> > > > > > >
> > > > > >> > > > > >
> > > > > >> > > > >
> > > > > >> > > >
> > > > > >> > >
> > > > > >> >
> > > > > >>
> > > > >
> > > >
> >
> https://urldefense.com/v3/__https://www.plantuml.com/plantuml/uml/bLHDR-Cs4BthLmpIYupw0zbkKQ1r3M-S7Bp8xhhM7WCOb3IM65EaGD9EX2RzxHrHb4CxRelwa4YSDu_lpOVcnZ9jzvM8BBS2uGjQpJC3dtHMSekPtMk44IpsMgEqa5XcCOhCZikQQLP1pR8TAp2n3ILhmZDP20m0fcIvUkAoW2qJXd9z1bpToO9BX3WXu0ucy5rpgGPNm0nW5_epUWtm2Ue3pn3kMOFQmKntGZW0BYtgBSi8k5A2QMwybJNMIbFiGSR9QZc4nUqIvikStF0jHprua5C-amge42aNt3R0f5JaaoivdV2Pkqbx4hee4ymOkBh5BTiB-_uIeGeo8zL8rPsPl4DktdEiK1jkB1NdZCRbrSTecDe_mlHbF0wvBmCkaOH5_S8a_TTTKI6-nmCAkEw4LpxsZ-LbYLKQFKMNOgf_wuM7_bV9gOer5SYMMksBSWXFcbi49KNZXNLicwfe3TETC7gPdPqI7uBcHMb1RSzYq34c6PDUM9mn8HRsUTZEiDBve3NjVZumBj0U7SS37mGO7vcwtiK-_pU7U7L_f-digo9YbhSwIfMRwIITKGXbxdIUTCGF1SeCJxloKsU-3k9ddRbX1eDq1q_fx1JbBGT0glVyXimDuP4TQ5qpCAmnGEj2s_6n5mtn1z-97-63itFQZLPO1Ev2tu_WF7Ju-VPc0Skg5bYXxBhkY1xpD7EM_7fyflSpIsqMgVth5xhVr4eQxWQ8enaSAJQSG16yFSDuJ798rrcXr_3n-lfdk7icQjEBmFujL7AodiP_Y4Z7-YxvtZNs4zMgpNTl6tF8sglyPsmqchrjvQ-m-aP94r-TwCA2Ka8upPJZwtvSpoYCXkYMZU2NXvRMBfq9P3i3Le4VAZUAlUZ_oPKsxPgY0Q_BSKLkyr9bhQhQrJjo_x3TPlIB0DPjnMfcIoYP0QaYw1a0fTKDr8fB6ntNuvmoL1ZGkXa69Njh43zf9GiGxHQrA_jDYWRSzF5--WmTVrN97_Sm8LbLUy_lGBmLanJjFkDlGkRqjA_4tm00__;!!Iz9xO38YGHZK!4rnlVIqBQQQ6xpGe1PMV58edOS_cGrKoqdfhw4qSp_JQCavvEjaxXk0OwGPnxyEcO60AzKQK_XDtp7H3DdkFXcrXuTBISHg$
> > > > > >> > > > > > > > > > > > >> >> >
> > > > > >> > > > > > > > > > > > >> >> >> :
> > > > > >> > > > > > > > > > > > >> >> >>
> > > > > >> > > > > > > > > > > > >> >> >> [image: image.png]
> > > > > >> > > > > > > > > > > > >> >> >>
> > > > > >> > > > > > > > > > > > >> >> >> Diagram 2
> > > > > >> > > > > > > > > > > > >> >> >> <
> > > > > >> > > > > > > > > > > > >> >>
> > > > > >> > > > > > > > > > > > >>
> > > > > >> > > > > > > > > > > >
> > > > > >> > > > > > > > > > >
> > > > > >> > > > > > > > > >
> > > > > >> > > > > > > > >
> > > > > >> > > > > > > >
> > > > > >> > > > > > >
> > > > > >> > > > > >
> > > > > >> > > > >
> > > > > >> > > >
> > > > > >> > >
> > > > > >> >
> > > > > >>
> > > > >
> > > >
> >
> https://urldefense.com/v3/__https://www.plantuml.com/plantuml/uml/VLLDR-8m4BtdLupO2sWBLVU8AaGB7AXAbssGzb896SS9RXqxjKqBwkv_tt7iV43fdaZYDpFlpRmnOsE9jhjSH9PRmM31hERKm8scMsuPjJlDe0yheZDc8RR4iYWoBrmMH9CS2a9VICPYUy1OZN0YCy5Q0BCbYNhdCeEK28En8G8wCvbnoQ0R8_05Bc6bkLIz3X03p1zzH7zR-9ZfDquPt9C3qoNCX2yV4G2NbkcKu5jdgGJHt0GbZwnG6i-UP3TUpk5gM6Ldqke350eZUqzoCft3U9xWHvxoa5-7K4nF1J46EbEMafsmdrCBbQ44gVggy18IZrn_ph5asd1ZiIKdQSgueZvjXrQFSFrdC3YN-nXmBacxbGiYyLVxLaBtdhqn0LSzdBDhqQtQoOJeGyad3z0lUqnYgpGB6Ns8oVyta00Dy_WnX0tIOZ8v6SYxHll1TrH6aejAik-mh-AphVFCwSUQqFypElag5QRGFDjQKEd96K1P8QP41c9TzA_IIQyvdAWyv_RSiS3skb0_EzDDkK2v5xWF6MiGFlvhpFLcD2Dq2pml14gaF67eQkmd8gulDoC4kSOu6KVpkvlUJg1RTbWISU40RdBUUS_9XfRZ2dwxm_SW8LYFISgm_MnlDQ6M9P1gbKEc4X-2pH_FvJCkCqm9pbVjD6LrwdLeOrDWfOaqc8Wh9BE85oNKxkNQ6o4yGRy_Eae0G_G8tZv81d3bHDB23WOdisohVr3nh_j6lbSjbNaLRTc8UgtPbAU1J_tygOfZX9DWEJeHDvYx-qmSi5FgNLPZwHrHcUsncGQ5-skhUclpE5fo4ounpFauYrUbkU6ccfnxMvitwag4IyerhTxj8In_Oj1bDO4pQru674loYrGlULHLEGCjwJJ8gDoVZR8MxO4BT3IzRvIcAQKezC6xpziGnTyImrfEGyJI_OcKfgtxIvnTqFEMS17L9Z-jsARN5FmTheP7HtSdtOMT0B4GY2FYHXxgQmMtj2bRqiLFGapiVe1_QVKDrkqXcm83aFEXnMYCZ-xlyHy__;!!Iz9xO38YGHZK!4rnlVIqBQQQ6xpGe1PMV58edOS_cGrKoqdfhw4qSp_JQCavvEjaxXk0OwGPnxyEcO60AzKQK_XDtp7H3DdkFXcrXSpmqcNk$
> > > > > >> > > > > > > > > > > > >> >> >
> > > > > >> > > > > > > > > > > > >> >> >> :
> > > > > >> > > > > > > > > > > > >> >> >> [image: image.png]
> > > > > >> > > > > > > > > > > > >> >> >>
> > > > > >> > > > > > > > > > > > >> >> >>  -ej
> > > > > >> > > > > > > > > > > > >> >> >>
> > > > > >> > > > > > > > > > > > >> >> >> On Wed, Apr 15, 2026 at 8:22 AM
> > Dmitri
> > > > > >> > > > Bourlatchkov
> > > > > >> > > > > <
> > > > > >> > > > > > > > > > > > >> dimas@apache.org>
> > > > > >> > > > > > > > > > > > >> >> >> wrote:
> > > > > >> > > > > > > > > > > > >> >> >>
> > > > > >> > > > > > > > > > > > >> >> >>> Hi All,
> > > > > >> > > > > > > > > > > > >> >> >>>
> > > > > >> > > > > > > > > > > > >> >> >>> Heads up: The current state of
> PR
> > > > [4115]
> > > > > >> > looks
> > > > > >> > > > > pretty
> > > > > >> > > > > > > > solid
> > > > > >> > > > > > > > > > to
> > > > > >> > > > > > > > > > > > me.
> > > > > >> > > > > > > > > > > > >> I
> > > > > >> > > > > > > > > > > > >> >> >>> believe this PR is approaching a
> > > > > mergeable
> > > > > >> > > > > condition.
> > > > > >> > > > > > > > > > > > >> >> >>>
> > > > > >> > > > > > > > > > > > >> >> >>> Please post your reviews if you
> > have
> > > > any
> > > > > >> > > > comments.
> > > > > >> > > > > > > > > > > > >> >> >>>
> > > > > >> > > > > > > > > > > > >> >> >>> [4115]
> > > > > >> > > >
> https://urldefense.com/v3/__https://github.com/apache/polaris/pull/4115__;!!Iz9xO38YGHZK!4rnlVIqBQQQ6xpGe1PMV58edOS_cGrKoqdfhw4qSp_JQCavvEjaxXk0OwGPnxyEcO60AzKQK_XDtp7H3DdkFXcrXtFRscKs$
> > > > > >> > > > > > > > > > > > >> >> >>>
> > > > > >> > > > > > > > > > > > >> >> >>> Thanks,
> > > > > >> > > > > > > > > > > > >> >> >>> Dmitri.
> > > > > >> > > > > > > > > > > > >> >> >>>
> > > > > >> > > > > > > > > > > > >> >> >>> On Tue, Mar 3, 2026 at 3:29 PM
> > Anand
> > > > > Kumar
> > > > > >> > > > Sankaran
> > > > > >> > > > > > via
> > > > > >> > > > > > > > > dev <
> > > > > >> > > > > > > > > > > > >> >> >>> dev@polaris.apache.org> wrote:
> > > > > >> > > > > > > > > > > > >> >> >>>
> > > > > >> > > > > > > > > > > > >> >> >>> > Hi Yufei and Dmitri,
> > > > > >> > > > > > > > > > > > >> >> >>> >
> > > > > >> > > > > > > > > > > > >> >> >>> > Here is a proposal for the
> REST
> > > > > >> endpoints
> > > > > >> > for
> > > > > >> > > > > > metrics
> > > > > >> > > > > > > > and
> > > > > >> > > > > > > > > > > > events.
> > > > > >> > > > > > > > > > > > >> >> >>> >
> > > > > >> > > > > > > > > > > > >> >> >>> >
> > > > > >> > > > > >
> https://urldefense.com/v3/__https://github.com/apache/polaris/pull/3924/changes__;!!Iz9xO38YGHZK!4rnlVIqBQQQ6xpGe1PMV58edOS_cGrKoqdfhw4qSp_JQCavvEjaxXk0OwGPnxyEcO60AzKQK_XDtp7H3DdkFXcrXcjoPhSs$
> > > > > >> > > > > > > > > > > > >> >> >>> >
> > > > > >> > > > > > > > > > > > >> >> >>> > I did not see any precursors
> for
> > > > > >> raising a
> > > > > >> > PR
> > > > > >> > > > for
> > > > > >> > > > > > > > > > proposals,
> > > > > >> > > > > > > > > > > so
> > > > > >> > > > > > > > > > > > >> >> trying
> > > > > >> > > > > > > > > > > > >> >> >>> > this.  Please let me know what
> > you
> > > > > >> think.
> > > > > >> > > > > > > > > > > > >> >> >>> >
> > > > > >> > > > > > > > > > > > >> >> >>> > -
> > > > > >> > > > > > > > > > > > >> >> >>> > Anand
> > > > > >> > > > > > > > > > > > >> >> >>> >
> > > > > >> > > > > > > > > > > > >> >> >>> > From: Anand Kumar Sankaran <
> > > > > >> > > > > > > anand.sankaran@workday.com
> > > > > >> > > > > > > > >
> > > > > >> > > > > > > > > > > > >> >> >>> > Date: Monday, March 2, 2026 at
> > > > > 10:25 AM
> > > > > >> > > > > > > > > > > > >> >> >>> > To: dev@polaris.apache.org <
> > > > > >> > > > > dev@polaris.apache.org
> > > > > >> > > > > > >
> > > > > >> > > > > > > > > > > > >> >> >>> > Subject: Re: Polaris Telemetry
> > and
> > > > > Audit
> > > > > >> > > Trail
> > > > > >> > > > > > > > > > > > >> >> >>> >
> > > > > >> > > > > > > > > > > > >> >> >>> > About the REST API, based on
> my
> > use
> > > > > >> cases:
> > > > > >> > > > > > > > > > > > >> >> >>> >
> > > > > >> > > > > > > > > > > > >> >> >>> >
> > > > > >> > > > > > > > > > > > >> >> >>> >   1.
> > > > > >> > > > > > > > > > > > >> >> >>> > I want to be able to query
> > commit
> > > > > >> metrics
> > > > > >> > to
> > > > > >> > > > > track
> > > > > >> > > > > > > > files
> > > > > >> > > > > > > > > > > added
> > > > > >> > > > > > > > > > > > /
> > > > > >> > > > > > > > > > > > >> >> >>> removed
> > > > > >> > > > > > > > > > > > >> >> >>> > per commit, along with record
> > > > counts.
> > > > > >> The
> > > > > >> > > > > ingestion
> > > > > >> > > > > > > > > > pipeline
> > > > > >> > > > > > > > > > > > that
> > > > > >> > > > > > > > > > > > >> >> >>> writes
> > > > > >> > > > > > > > > > > > >> >> >>> > this data is owned by us and
> we
> > are
> > > > > >> > > guaranteed
> > > > > >> > > > to
> > > > > >> > > > > > > write
> > > > > >> > > > > > > > > > this
> > > > > >> > > > > > > > > > > > >> >> >>> information
> > > > > >> > > > > > > > > > > > >> >> >>> > for each write.
> > > > > >> > > > > > > > > > > > >> >> >>> >   2.
> > > > > >> > > > > > > > > > > > >> >> >>> > I want to be able to query
> scan
> > > > > metrics
> > > > > >> for
> > > > > >> > > > > read. I
> > > > > >> > > > > > > > > > > understand
> > > > > >> > > > > > > > > > > > >> >> clients
> > > > > >> > > > > > > > > > > > >> >> >>> do
> > > > > >> > > > > > > > > > > > >> >> >>> > not fulfill this requirement.
> > > > > >> > > > > > > > > > > > >> >> >>> >   3.
> > > > > >> > > > > > > > > > > > >> >> >>> > I want to be able to query the
> > > > events
> > > > > >> table
> > > > > >> > > > > (events
> > > > > >> > > > > > > are
> > > > > >> > > > > > > > > > > > >> persisted) -
> > > > > >> > > > > > > > > > > > >> >> >>> this
> > > > > >> > > > > > > > > > > > >> >> >>> > may supersede #2, I am not
> sure
> > yet.
> > > > > >> > > > > > > > > > > > >> >> >>> >
> > > > > >> > > > > > > > > > > > >> >> >>> > All this information is in the
> > JDBC
> > > > > >> based
> > > > > >> > > > > > persistence
> > > > > >> > > > > > > > > model
> > > > > >> > > > > > > > > > > and
> > > > > >> > > > > > > > > > > > >> is
> > > > > >> > > > > > > > > > > > >> >> >>> > persisted in the metastore. I
> > > > > currently
> > > > > >> > don’t
> > > > > >> > > > > have
> > > > > >> > > > > > a
> > > > > >> > > > > > > > need
> > > > > >> > > > > > > > > > to
> > > > > >> > > > > > > > > > > > >> query
> > > > > >> > > > > > > > > > > > >> >> >>> > prometheus or open telemetry.
> I
> > do
> > > > > >> publish
> > > > > >> > > some
> > > > > >> > > > > > > events
> > > > > >> > > > > > > > to
> > > > > >> > > > > > > > > > > > >> Prometheus
> > > > > >> > > > > > > > > > > > >> >> >>> and
> > > > > >> > > > > > > > > > > > >> >> >>> > they are forwarded to our
> > dashboards
> > > > > >> > > elsewhere.
> > > > > >> > > > > > > > > > > > >> >> >>> >
> > > > > >> > > > > > > > > > > > >> >> >>> > About the CLI utilities, I
> > meant the
> > > > > >> admin
> > > > > >> > > user
> > > > > >> > > > > > > > > utilities.
> > > > > >> > > > > > > > > > In
> > > > > >> > > > > > > > > > > > >> one of
> > > > > >> > > > > > > > > > > > >> >> >>> the
> > > > > >> > > > > > > > > > > > >> >> >>> > earliest drafts of my
> proposal,
> > > > > Prashant
> > > > > >> > > > > mentioned
> > > > > >> > > > > > > that
> > > > > >> > > > > > > > > the
> > > > > >> > > > > > > > > > > > >> metrics
> > > > > >> > > > > > > > > > > > >> >> >>> tables
> > > > > >> > > > > > > > > > > > >> >> >>> > can grow indefinitely and
> that a
> > > > > similar
> > > > > >> > > > problem
> > > > > >> > > > > > > exists
> > > > > >> > > > > > > > > > with
> > > > > >> > > > > > > > > > > > the
> > > > > >> > > > > > > > > > > > >> >> events
> > > > > >> > > > > > > > > > > > >> >> >>> > table as well. We discussed
> that
> > > > > >> cleaning
> > > > > >> > up
> > > > > >> > > of
> > > > > >> > > > > old
> > > > > >> > > > > > > > > records
> > > > > >> > > > > > > > > > > > from
> > > > > >> > > > > > > > > > > > >> >> both
> > > > > >> > > > > > > > > > > > >> >> >>> > metrics tables and events
> > tables can
> > > > > be
> > > > > >> > done
> > > > > >> > > > via
> > > > > >> > > > > a
> > > > > >> > > > > > > CLI
> > > > > >> > > > > > > > > > > utility.
> > > > > >> > > > > > > > > > > > >> >> >>> >
> > > > > >> > > > > > > > > > > > >> >> >>> > I see that Yufei has covered
> the
> > > > > >> discussion
> > > > > >> > > > about
> > > > > >> > > > > > > > > > > datasources.
> > > > > >> > > > > > > > > > > > >> >> >>> >
> > > > > >> > > > > > > > > > > > >> >> >>> > -
> > > > > >> > > > > > > > > > > > >> >> >>> > Anand
> > > > > >> > > > > > > > > > > > >> >> >>> >
> > > > > >> > > > > > > > > > > > >> >> >>> >
> > > > > >> > > > > > > > > > > > >> >> >>> >
> > > > > >> > > > > > > > > > > > >> >> >>> > From: Yufei Gu <
> > > > flyrain000@gmail.com>
> > > > > >> > > > > > > > > > > > >> >> >>> > Date: Friday, February 27,
> 2026
> > at
> > > > > >> 9:54 PM
> > > > > >> > > > > > > > > > > > >> >> >>> > To: dev@polaris.apache.org <
> > > > > >> > > > > dev@polaris.apache.org
> > > > > >> > > > > > >
> > > > > >> > > > > > > > > > > > >> >> >>> > Subject: Re: Polaris Telemetry
> > and
> > > > > Audit
> > > > > >> > > Trail
> > > > > >> > > > > > > > > > > > >> >> >>> >
> > > > > >> > > > > > > > > > > > >> >> >>> > This Message Is From an
> External
> > > > > Sender
> > > > > >> > > > > > > > > > > > >> >> >>> > This message came from outside
> > your
> > > > > >> > > > organization.
> > > > > >> > > > > > > > > > > > >> >> >>> > Report Suspicious<
> > > > > >> > > > > > > > > > > > >> >> >>> >
> > > > > >> > > > > > > > > > > > >> >> >>>
> > > > > >> > > > > > > > > > > > >> >>
> > > > > >> > > > > > > > > > > > >>
> > > > > >> > > > > > > > > > > >
> > > > > >> > > > > > > > > > >
> > > > > >> > > > > > > > > >
> > > > > >> > > > > > > > >
> > > > > >> > > > > > > >
> > > > > >> > > > > > >
> > > > > >> > > > > >
> > > > > >> > > > >
> > > > > >> > > >
> > > > > >> > >
> > > > > >> >
> > > > > >>
> > > > >
> > > >
> >
> https://us-phishalarm-ewt.proofpoint.com/EWT/v1/Iz9xO38YGHZK!YhNDZABkHi1B699ote2uMwpOZw8i0QMCGO2Szc-HshuABGhGvwPJcymE6G2oUUxtS8xDkSrtGTPm_I3QnVDHoLMk50m9v8z_nZKTkd-bnVUbreF1u0WnfV_X5eYevZl_$
> > > > > >> > > > > > > > > > > > >> >> >>> > >
> > > > > >> > > > > > > > > > > > >> >> >>> >
> > > > > >> > > > > > > > > > > > >> >> >>> >
> > > > > >> > > > > > > > > > > > >> >> >>> > As I mentioned in
> > > > > >> > > > > > > > > > > > >> >> >>> >
> > > > > >> > > > > > > > > > > > >> >> >>>
> > > > > >> > > > > > > > > > > > >> >>
> > > > > >> > > > > > > > > > > > >>
> > > > > >> > > > > > > > > > > >
> > > > > >> > > > > > > > > > >
> > > > > >> > > > > > > > > >
> > > > > >> > > > > > > > >
> > > > > >> > > > > > > >
> > > > > >> > > > > > >
> > > > > >> > > > > >
> > > > > >> > > > >
> > > > > >> > > >
> > > > > >> > >
> > > > > >> >
> > > > > >>
> > > > >
> > > >
> >
> https://urldefense.com/v3/__https://github.com/apache/polaris/issues/3890__;!!Iz9xO38YGHZK!5EuyFFkk3vhRWVIRvQAWBSQfpJkTMA9HxugzDwXmN0LPPqhEFxYkFRGVhtb8AqUwXtDh2OplcMnbMDHKOxrvDU0$
> >> > > >> > > > > > > > > > > > >> >> >>> ,
> > > > > >> > > > > > > > > > > > >> >> >>> > supporting
> > > > > >> > > > > > > > > > > > >> >> >>> > multiple data sources is not a
> > > > trivial
> > > > > >> > > change.
> > > > > >> > > > I
> > > > > >> > > > > > > would
> > > > > >> > > > > > > > > > > strongly
> > > > > >> > > > > > > > > > > > >> >> >>> recommend
> > > > > >> > > > > > > > > > > > >> >> >>> > starting with a design
> document
> > to
> > > > > >> > carefully
> > > > > >> > > > > > evaluate
> > > > > >> > > > > > > > the
> > > > > >> > > > > > > > > > > > >> >> architectural
> > > > > >> > > > > > > > > > > > >> >> >>> > implications and long term
> > impact.
> > > > > >> > > > > > > > > > > > >> >> >>> >
> > > > > >> > > > > > > > > > > > >> >> >>> > A REST endpoint to query
> metrics
> > > > seems
> > > > > >> > > > reasonable
> > > > > >> > > > > > > given
> > > > > >> > > > > > > > > the
> > > > > >> > > > > > > > > > > > >> current
> > > > > >> > > > > > > > > > > > >> >> >>> JDBC
> > > > > >> > > > > > > > > > > > >> >> >>> > based persistence model. That
> > said,
> > > > we
> > > > > >> may
> > > > > >> > > also
> > > > > >> > > > > > > > consider
> > > > > >> > > > > > > > > > > > >> alternative
> > > > > >> > > > > > > > > > > > >> >> >>> > storage models. For example,
> if
> > we
> > > > > later
> > > > > >> > > adopt
> > > > > >> > > > a
> > > > > >> > > > > > time
> > > > > >> > > > > > > > > > series
> > > > > >> > > > > > > > > > > > >> system
> > > > > >> > > > > > > > > > > > >> >> >>> such as
> > > > > >> > > > > > > > > > > > >> >> >>> > Prometheus to store metrics,
> the
> > > > query
> > > > > >> > model
> > > > > >> > > > and
> > > > > >> > > > > > > access
> > > > > >> > > > > > > > > > > > patterns
> > > > > >> > > > > > > > > > > > >> >> would
> > > > > >> > > > > > > > > > > > >> >> >>> be
> > > > > >> > > > > > > > > > > > >> >> >>> > fundamentally different.
> > Designing
> > > > the
> > > > > >> REST
> > > > > >> > > API
> > > > > >> > > > > > > without
> > > > > >> > > > > > > > > > > > >> considering
> > > > > >> > > > > > > > > > > > >> >> >>> these
> > > > > >> > > > > > > > > > > > >> >> >>> > potential evolutions may limit
> > > > > >> flexibility.
> > > > > >> > > I'd
> > > > > >> > > > > > > suggest
> > > > > >> > > > > > > > > to
> > > > > >> > > > > > > > > > > > start
> > > > > >> > > > > > > > > > > > >> >> with
> > > > > >> > > > > > > > > > > > >> >> >>> the
> > > > > >> > > > > > > > > > > > >> >> >>> > use case.
> > > > > >> > > > > > > > > > > > >> >> >>> >
> > > > > >> > > > > > > > > > > > >> >> >>> > Yufei
> > > > > >> > > > > > > > > > > > >> >> >>> >
> > > > > >> > > > > > > > > > > > >> >> >>> >
> > > > > >> > > > > > > > > > > > >> >> >>> > On Fri, Feb 27, 2026 at
> 3:42 PM
> > > > Dmitri
> > > > > >> > > > > > Bourlatchkov <
> > > > > >> > > > > > > > > > > > >> >> dimas@apache.org>
> > > > > >> > > > > > > > > > > > >> >> >>> > wrote:
> > > > > >> > > > > > > > > > > > >> >> >>> >
> > > > > >> > > > > > > > > > > > >> >> >>> > > Hi Anand,
> > > > > >> > > > > > > > > > > > >> >> >>> > >
> > > > > >> > > > > > > > > > > > >> >> >>> > > Sharing my view... subject
> to
> > > > > >> discussion:
> > > > > >> > > > > > > > > > > > >> >> >>> > >
> > > > > >> > > > > > > > > > > > >> >> >>> > > 1. Adding non-IRC REST API
> to
> > > > > Polaris
> > > > > >> is
> > > > > >> > > > > > perfectly
> > > > > >> > > > > > > > > fine.
> > > > > >> > > > > > > > > > > > >> >> >>> > >
> > > > > >> > > > > > > > > > > > >> >> >>> > > Figuring out specific
> endpoint
> > > > URIs
> > > > > >> and
> > > > > >> > > > > payloads
> > > > > >> > > > > > > > might
> > > > > >> > > > > > > > > > > > require
> > > > > >> > > > > > > > > > > > >> a
> > > > > >> > > > > > > > > > > > >> >> few
> > > > > >> > > > > > > > > > > > >> >> >>> > > roundtrips, so opening a
> > separate
> > > > > >> thread
> > > > > >> > > for
> > > > > >> > > > > that
> > > > > >> > > > > > > > might
> > > > > >> > > > > > > > > > be
> > > > > >> > > > > > > > > > > > >> best.
> > > > > >> > > > > > > > > > > > >> >> >>> > > Contributors commonly create
> > > > Google
> > > > > >> Docs
> > > > > >> > > for
> > > > > >> > > > > new
> > > > > >> > > > > > > API
> > > > > >> > > > > > > > > > > > proposals
> > > > > >> > > > > > > > > > > > >> too
> > > > > >> > > > > > > > > > > > >> >> >>> (they
> > > > > >> > > > > > > > > > > > >> >> >>> > > fairly easy to update as the
> > email
> > > > > >> > > discussion
> > > > > >> > > > > > > > > > progresses).
> > > > > >> > > > > > > > > > > > >> >> >>> > >
> > > > > >> > > > > > > > > > > > >> >> >>> > > There was a suggestion to
> try
> > > > > Markdown
> > > > > >> > > (with
> > > > > >> > > > > PRs)
> > > > > >> > > > > > > for
> > > > > >> > > > > > > > > > > > proposals
> > > > > >> > > > > > > > > > > > >> >> [1]
> > > > > >> > > > > > > > > > > > >> >> >>> ...
> > > > > >> > > > > > > > > > > > >> >> >>> > > feel free to give it a try
> if
> > you
> > > > > are
> > > > > >> > > > > comfortable
> > > > > >> > > > > > > > with
> > > > > >> > > > > > > > > > > that.
> > > > > >> > > > > > > > > > > > >> >> >>> > >
> > > > > >> > > > > > > > > > > > >> >> >>> > > 2. Could you clarify whether
> > you
> > > > > mean
> > > > > >> end
> > > > > >> > > > user
> > > > > >> > > > > > > > > utilities
> > > > > >> > > > > > > > > > or
> > > > > >> > > > > > > > > > > > >> admin
> > > > > >> > > > > > > > > > > > >> >> >>> user
> > > > > >> > > > > > > > > > > > >> >> >>> > > utilities? In the latter
> case
> > > > those
> > > > > >> might
> > > > > >> > > be
> > > > > >> > > > > more
> > > > > >> > > > > > > > > > suitable
> > > > > >> > > > > > > > > > > > for
> > > > > >> > > > > > > > > > > > >> the
> > > > > >> > > > > > > > > > > > >> >> >>> Admin
> > > > > >> > > > > > > > > > > > >> >> >>> > > CLI (java) not the Python
> CLI,
> > > > IMHO.
> > > > > >> > > > > > > > > > > > >> >> >>> > >
> > > > > >> > > > > > > > > > > > >> >> >>> > > Why would these utilities be
> > > > common
> > > > > >> with
> > > > > >> > > > > events?
> > > > > >> > > > > > > > IMHO,
> > > > > >> > > > > > > > > > > event
> > > > > >> > > > > > > > > > > > >> use
> > > > > >> > > > > > > > > > > > >> >> >>> cases
> > > > > >> > > > > > > > > > > > >> >> >>> > are
> > > > > >> > > > > > > > > > > > >> >> >>> > > distinct from scan/commit
> > metrics.
> > > > > >> > > > > > > > > > > > >> >> >>> > >
> > > > > >> > > > > > > > > > > > >> >> >>> > > 3. I'd prefer separating
> > metrics
> > > > > >> > > persistence
> > > > > >> > > > > from
> > > > > >> > > > > > > > > > MetaStore
> > > > > >> > > > > > > > > > > > >> >> >>> persistence
> > > > > >> > > > > > > > > > > > >> >> >>> > at
> > > > > >> > > > > > > > > > > > >> >> >>> > > the code level, so that they
> > could
> > > > > be
> > > > > >> > mixed
> > > > > >> > > > and
> > > > > >> > > > > > > > matched
> > > > > >> > > > > > > > > > > > >> >> >>> independently.
> > > > > >> > > > > > > > > > > > >> >> >>> > The
> > > > > >> > > > > > > > > > > > >> >> >>> > > separate datasource question
> > will
> > > > > >> become
> > > > > >> > a
> > > > > >> > > > > > > non-issue
> > > > > >> > > > > > > > > with
> > > > > >> > > > > > > > > > > > that
> > > > > >> > > > > > > > > > > > >> >> >>> approach,
> > > > > >> > > > > > > > > > > > >> >> >>> > I
> > > > > >> > > > > > > > > > > > >> >> >>> > > guess.
> > > > > >> > > > > > > > > > > > >> >> >>> > >
> > > > > >> > > > > > > > > > > > >> >> >>> > > The rationale for separating
> > scan
> > > > > >> metrics
> > > > > >> > > and
> > > > > >> > > > > > > > metastore
> > > > > >> > > > > > > > > > > > >> >> persistence
> > > > > >> > > > > > > > > > > > >> >> >>> is
> > > > > >> > > > > > > > > > > > >> >> >>> > that
> > > > > >> > > > > > > > > > > > >> >> >>> > > "cascading deletes" between
> > them
> > > > are
> > > > > >> > hardly
> > > > > >> > > > > ever
> > > > > >> > > > > > > > > > required.
> > > > > >> > > > > > > > > > > > >> >> >>> Furthermore,
> > > > > >> > > > > > > > > > > > >> >> >>> > the
> > > > > >> > > > > > > > > > > > >> >> >>> > > data and query patterns are
> > very
> > > > > >> > different
> > > > > >> > > so
> > > > > >> > > > > > > > different
> > > > > >> > > > > > > > > > > > >> >> technologies
> > > > > >> > > > > > > > > > > > >> >> >>> > might
> > > > > >> > > > > > > > > > > > >> >> >>> > > be beneficial in each case.
> > > > > >> > > > > > > > > > > > >> >> >>> > >
> > > > > >> > > > > > > > > > > > >> >> >>> > > [1]
> > > > > >> > > > > > > > > > > > >> >> >>> >
> > > > > >> > > > > > > > > > > > >> >> >>>
> > > > > >> > > > > > > > > > > > >> >>
> > > > > >> > > > > > > > > > > > >>
> > > > > >> > > > > > > > > > > >
> > > > > >> > > > > > > > > > >
> > > > > >> > > > > > > > > >
> > > > > >> > > > > > > > >
> > > > > >> > > > > > > >
> > > > > >> > > > > > >
> > > > > >> > > > > >
> > > > > >> > > > >
> > > > > >> > > >
> > > > > >> > >
> > > > > >> >
> > > > > >>
> > > > >
> > > >
> >
> https://urldefense.com/v3/__https://lists.apache.org/thread/yto2wp982t43h1mqjwnslswhws5z47cy__;!!Iz9xO38YGHZK!5EuyFFkk3vhRWVIRvQAWBSQfpJkTMA9HxugzDwXmN0LPPqhEFxYkFRGVhtb8AqUwXtDh2OplcMnbMDHKxYDakNU$
> >> > > >> > > > > > > > > > > > >> >> >>> > >
> > > > > >> > > > > > > > > > > > >> >> >>> > > Cheers,
> > > > > >> > > > > > > > > > > > >> >> >>> > > Dmitri.
> > > > > >> > > > > > > > > > > > >> >> >>> > >
> > > > > >> > > > > > > > > > > > >> >> >>> > > On Fri, Feb 27, 2026 at
> > 6:19 PM
> > > > > Anand
> > > > > >> > Kumar
> > > > > >> > > > > > > Sankaran
> > > > > >> > > > > > > > > via
> > > > > >> > > > > > > > > > > dev
> > > > > >> > > > > > > > > > > > <
> > > > > >> > > > > > > > > > > > >> >> >>> > > dev@polaris.apache.org>
> > wrote:
> > > > > >> > > > > > > > > > > > >> >> >>> > >
> > > > > >> > > > > > > > > > > > >> >> >>> > > > Thanks all. This PR is
> > merged
> > > > now.
> > > > > >> > > > > > > > > > > > >> >> >>> > > >
> > > > > >> > > > > > > > > > > > >> >> >>> > > > Here are the follow-up
> > features
> > > > /
> > > > > >> work
> > > > > >> > > > > needed.
> > > > > >> > > > > > > > These
> > > > > >> > > > > > > > > > > were
> > > > > >> > > > > > > > > > > > >> all
> > > > > >> > > > > > > > > > > > >> >> >>> part of
> > > > > >> > > > > > > > > > > > >> >> >>> > > the
> > > > > >> > > > > > > > > > > > >> >> >>> > > > merged PR at some point in
> > time
> > > > > and
> > > > > >> > were
> > > > > >> > > > > > removed
> > > > > >> > > > > > > to
> > > > > >> > > > > > > > > > > reduce
> > > > > >> > > > > > > > > > > > >> >> scope.
> > > > > >> > > > > > > > > > > > >> >> >>> > > >
> > > > > >> > > > > > > > > > > > >> >> >>> > > > Please let me know what
> you
> > > > think.
> > > > > >> > > > > > > > > > > > >> >> >>> > > >
> > > > > >> > > > > > > > > > > > >> >> >>> > > >
> > > > > >> > > > > > > > > > > > >> >> >>> > > >   1.  A REST API to
> paginate
> > > > > through
> > > > > >> > > table
> > > > > >> > > > > > > metrics.
> > > > > >> > > > > > > > > > This
> > > > > >> > > > > > > > > > > > >> will be
> > > > > >> > > > > > > > > > > > >> >> >>> > non-IRC
> > > > > >> > > > > > > > > > > > >> >> >>> > > > standard addition.
> > > > > >> > > > > > > > > > > > >> >> >>> > > >   2.  Utilities for
> > managing old
> > > > > >> > records,
> > > > > >> > > > > > should
> > > > > >> > > > > > > be
> > > > > >> > > > > > > > > > > common
> > > > > >> > > > > > > > > > > > >> with
> > > > > >> > > > > > > > > > > > >> >> >>> events.
> > > > > >> > > > > > > > > > > > >> >> >>> > > > There was some discussion
> > that
> > > > it
> > > > > >> > belongs
> > > > > >> > > > to
> > > > > >> > > > > > the
> > > > > >> > > > > > > > CLI.
> > > > > >> > > > > > > > > > > > >> >> >>> > > >   3.  Separate datasource
> > > > > (metrics,
> > > > > >> > > events,
> > > > > >> > > > > > even
> > > > > >> > > > > > > > > other
> > > > > >> > > > > > > > > > > > >> tables?).
> > > > > >> > > > > > > > > > > > >> >> >>> > > >
> > > > > >> > > > > > > > > > > > >> >> >>> > > >
> > > > > >> > > > > > > > > > > > >> >> >>> > > > Anything else?
> > > > > >> > > > > > > > > > > > >> >> >>> > > >
> > > > > >> > > > > > > > > > > > >> >> >>> > > > -
> > > > > >> > > > > > > > > > > > >> >> >>> > > > Anand
> > > > > >> > > > > > > > > > > > >> >> >>> > > >
> > > > > >> > > > > > > > > > > > >> >> >>> > > >
> > > > > >> > > > > > > > > > > > >> >> >>> > >
> > > > > >> > > > > > > > > > > > >> >> >>> >
> > > > > >> > > > > > > > > > > > >> >> >>> >
> > > > > >> > > > > > > > > > > > >> >> >>>
> > > > > >> > > > > > > > > > > > >> >> >>
> > > > > >> > > > > > > > > > > > >> >>
> > > > > >> > > > > > > > > > > > >> >
> > > > > >> > > > > > > > > > > > >>
> > > > > >> > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > >
> > > > > >> > > > > > > > > > >
> > > > > >> > > > > > > > > >
> > > > > >> > > > > > > > >
> > > > > >> > > > > > > >
> > > > > >> > > > > > >
> > > > > >> > > > > >
> > > > > >> > > > >
> > > > > >> > > >
> > > > > >> > >
> > > > > >> >
> > > > > >>
> > > > > >
> > > > >
> > > >
> >
>
>

Re: Proposal for REST endpoints for table metrics and events

[Anand Kumar Sankaran via dev <de...@polaris.apache.org>]

Thanks EJ.  My plan is to introduce PR0 in a new branch / PR and repurpose #4115 and #4756 for PR1 and PR2 to maintain all the valuable discussions that happened there.

From: EJ Wang <ej...@gmail.com>
Date: Wednesday, July 15, 2026 at 10:41 AM
To: dev@polaris.apache.org <de...@polaris.apache.org>
Subject: Re: Proposal for REST endpoints for table metrics and events

Hi folks, Thanks again for the discussion today. I updated the sync doc with notes from today's metrics sync: https: //urldefense. com/v3/__https: //docs. google. com/document/d/100h7c4damrUzVuquYbBHM0EvA4LSWuW2IT2dN_7nYVA/edit?pli=1&tab=t. 2ahd7wze5f__;!!Iz9xO38YGHZK!4rnlVIqBQQQ6xpGe1PMV58edOS_cGrKoqdfhw4qSp_JQCavvEjaxXk0OwGPnxyEcO60AzKQK_XDtp7H3DdkFXcrXJwAz1g0$


Hi folks,

Thanks again for the discussion today. I updated the sync doc with notes
from today's metrics sync:
https://urldefense.com/v3/__https://docs.google.com/document/d/100h7c4damrUzVuquYbBHM0EvA4LSWuW2IT2dN_7nYVA/edit?pli=1&tab=t.2ahd7wze5f__;!!Iz9xO38YGHZK!4rnlVIqBQQQ6xpGe1PMV58edOS_cGrKoqdfhw4qSp_JQCavvEjaxXk0OwGPnxyEcO60AzKQK_XDtp7H3DdkFXcrXJwAz1g0$

See highlights below:

In previous meetings, EJ proposed a two prs framing to push forward the
metrics query work in polaris. Dmitri pointed out a subtle point that pr2
is currently also trying to do the persistence related code placement
refactor. For the sake of straightforward review process, EJ, Dmitri and
Anand (the author) agreed to separate the persistence refactoring into pr0,
and reframing the pr scopes as follow:

   1.

   Pr0: refactor irc metrics spi
   1.

      SPI should stay in core/ with metric type enveloped
      2.

      default spi impl, no-op, in extensions/ - default selection at
      runtime/
      3.

      move metrics persistence and correspondings to extensions/ as an
      alternative SPI impl
      2.

   Pr1: rest api for query
   https://urldefense.com/v3/__https://github.com/apache/polaris/pull/4115/changes__;!!Iz9xO38YGHZK!4rnlVIqBQQQ6xpGe1PMV58edOS_cGrKoqdfhw4qSp_JQCavvEjaxXk0OwGPnxyEcO60AzKQK_XDtp7H3DdkFXcrXFCbEbRk$
   1.

      rest api spec in extensions/ as optional spec
      2.

      SPI in extensions/ as optional spec spi
      3.

      default spi impl in extensions/ no-op
      4.

      handler related in runtime/ - thin adaptor from http shape to spi
      shape
      3.

   Pr2: jdbc impl for query https://urldefense.com/v3/__https://github.com/apache/polaris/pull/4756__;!!Iz9xO38YGHZK!4rnlVIqBQQQ6xpGe1PMV58edOS_cGrKoqdfhw4qSp_JQCavvEjaxXk0OwGPnxyEcO60AzKQK_XDtp7H3DdkFXcrXRVMBzqE$
   1.

      alternative spi impl in extensions/ -> query with jdbc


Thanks,
-ej

On Wed, Jul 1, 2026 at 8:01 AM Alexandre Dutra <ad...@apache.org> wrote:

> Hi all,
>
> > While the current implementation stores metrics in the metastore, other
> deployments
> may choose a KV store, a dedicated metrics warehouse, or another backend
> entirely. In those cases, exposing a Polaris REST API over a specific
> schema feels like coupling the API to an implementation detail.
>
> I wouldn't say to an implementation detail, but rather to a particular
> setup / configuration. I think that's fine. If you don't store
> metrics, you don't read metrics. Similar situations will certainly
> happen in the future, e.g. if we ever introduce an API to read events,
> that would only work if events are being persisted in the first place.
> Again I think that's a fair trade-off.
>
> Thanks,
> Alex
>
> On Wed, Jul 1, 2026 at 7:23 AM Dmitri Bourlatchkov <di...@apache.org>
> wrote:
> >
> > Hi Yufei,
> >
> > From my POV, it is fine for Polaris to have multiple REST API surfaces
> with
> > different features and different degrees of "readiness". Each API can
> > evolve fairly independently. With proper modularization (which Anand
> did),
> > changes to the main Polaris code and these extra modules should normally
> be
> > non-intrusive to each other.
> >
> > I assume Anand is going to be the first user of this API. However, if we
> do
> > not merge it we'll never know whether other users are interested. I do
> not
> > think many end users read the dev ML, but they certainly read the docs,
> > code and release notes.
> >
> > There is no risk for Polaris as a project in exposing this experimental
> > API. If usage dwindles over time, we can remove it. If it gains a
> > considerable user / contributor base, we will promote it to a
> "first-class"
> > (non-beta) API.
> >
> > Cheers,
> > Dmitri.
> >
> > On Tue, Jun 30, 2026 at 7:53 PM Yufei Gu <fl...@gmail.com> wrote:
> >
> > > Hi all,
> > >
> > > Thanks for driving this work forward. Building this as a POC makes
> perfect
> > > sense if the goal is to validate the idea and gather feedback. Should
> we
> > > merge a POC into the main branch before agreeing on the overall
> direction?
> > > To me, that feels like a separate discussion from whether the
> > > implementation itself looks reasonable.
> > >
> > > One concern that I don't think has been fully addressed is whether we
> need
> > > to expose metrics consumption through a Polaris REST endpoint at all.
> > >
> > > My concern is less about the API itself and more about the boundary
> Polaris
> > > should own. The proposed REST API is tied to a particular persistence
> > > schema, but that schema is just one possible implementation. While the
> > > current implementation stores metrics in the metastore, other
> deployments
> > > may choose a KV store, a dedicated metrics warehouse, or another
> backend
> > > entirely. In those cases, exposing a Polaris REST API over a specific
> > > schema feels like coupling the API to an implementation detail.
> > >
> > > Polaris isn't intended to become a full metrics system. I'd expect it
> to
> > > focus on producing metrics and let deployments choose the most
> appropriate
> > > storage and consumption model for their environment.
> > >
> > > I'm not opposed to exploring this direction, but we should first align
> on
> > > whether exposing a REST API through Polaris is the right long term
> approach
> > > before merging it into the main branch.
> > >
> > > Thanks,
> > > Yufei
> > >
> > > Yufei
> > >
> > >
> > > On Mon, Jun 29, 2026 at 9:27 AM Dmitri Bourlatchkov <di...@apache.org>
> > > wrote:
> > >
> > > > Hi All,
> > > >
> > > > It looks like Anand addressed all action items in [4115].
> > > >
> > > > Please review the latest state of the code. The PR LGTM and I
> approved in
> > > > GH from my side.
> > > >
> > > > [4115] https://urldefense.com/v3/__https://github.com/apache/polaris/pull/4115__;!!Iz9xO38YGHZK!4rnlVIqBQQQ6xpGe1PMV58edOS_cGrKoqdfhw4qSp_JQCavvEjaxXk0OwGPnxyEcO60AzKQK_XDtp7H3DdkFXcrXtFRscKs$
> > > >
> > > > Thanks,
> > > > Dmitri.
> > > >
> > > > On Thu, Jun 25, 2026 at 1:56 PM Dmitri Bourlatchkov <
> dimas@apache.org>
> > > > wrote:
> > > >
> > > > > Hi All,
> > > > >
> > > > > Recap of the discussion in today's Community Sync:
> > > > >
> > > > > * The modular approach for Metrics SPI(s) and implementation seems
> to
> > > > have
> > > > > community consensus (please comment if you still have concerns)
> > > > >
> > > > > * What modules / feature flags are enabled in Apache Polaris
> images by
> > > > > default will be discussed later (when the feature is close to
> > > > completion).
> > > > >
> > > > > In any case, users will be able to turn the full feature on/off at
> > > their
> > > > > discretion, regardless of the default flag values.
> > > > >
> > > > > * It is generally preferable to have smaller PRs and introduce new
> > > > > features gradually to simplify reviews.
> > > > >
> > > > > What this means for the Metrics API, IMHO is:
> > > > >
> > > > > a) The scope of PR 4115 is probably ok as it stands right now. If
> > > someone
> > > > > wants to propose splitting it further, please comment.
> > > > >
> > > > > b) After 4115 is merged, Metrics Persistence PRs will follow, with
> > > > smaller
> > > > > scope per PR as appropriate for the nature of the code changes.
> > > > >
> > > > > * The major concern was about exposing the new Metrics REST API to
> end
> > > > > users. The nature of this concern was about avoiding the (false)
> > > > impression
> > > > > that the new API is a firm and adopted Polaris specification.
> > > > >
> > > > > I think this concern is reasonable.
> > > > >
> > > > > I propose that Anand update the documentation sections of PR 4115
> to
> > > > > emphasise that the API is currently an experimental / POC API at
> this
> > > > time
> > > > > and is subject to change (including breaking changes). I believe it
> > > > already
> > > > > has the "beta" label. So this request is mostly for doc changes to
> > > > clarify
> > > > > user expectations, since the "beta" label alone may not be
> interpreted
> > > > the
> > > > > same way by all users.
> > > > >
> > > > > I hope this update will resolve all remaining concerns with 4115
> and
> > > > allow
> > > > > for it to be merged so that the community will be able to proceed
> to
> > > > > follow-up PRs (as noted above).
> > > > >
> > > > > Please add/correct me if I missed anything relevant.
> > > > >
> > > > > Thanks,
> > > > > Dmitri.
> > > > >
> > > > > On Wed, Jun 24, 2026 at 8:11 PM Yufei Gu <fl...@gmail.com>
> wrote:
> > > > >
> > > > >> I do not think we have agreement on the query REST APIs yet.
> > > > >>
> > > > >> In particular, the current query API shape seems highly coupled
> to the
> > > > >> *example[1]* metrics in the Iceberg REST spec. I do not think
> that is
> > > a
> > > > >> good way to design a REST API contract. For example, fields like
> > > > >> resultDataFiles, resultDeleteFiles, totalFileSizeBytes,
> > > > >> totalDataManifests,
> > > > >> totalDeleteManifests, and scannedDataManifests are very specific
> to
> > > one
> > > > >> style of scan metric. That makes the API look more like a direct
> > > > >> projection
> > > > >> of the current example payload than a general query model.
> > > > >>
> > > > >> My preference is that Polaris should not define or strongly opine
> on
> > > the
> > > > >> Iceberg scan or commit metrics query REST API at this point.
> Polaris
> > > can
> > > > >> resolve and authorize the table, accept the metrics report, and
> > > delegate
> > > > >> it
> > > > >> to the configured reporting path. Durable storage, filtering,
> > > > dashboards,
> > > > >> and query APIs should remain downstream impl. choices unless and
> until
> > > > the
> > > > >> Iceberg community standardizes a query contract, which I doubt
> will
> > > > >> happen.
> > > > >>
> > > > >> So I do not think the intake path has to be blocked, but I would
> > > prefer
> > > > >> not
> > > > >> to merge a Polaris specific query API that may be hard to evolve
> > > later.
> > > > >>
> > > > >> 1.
> > > > >>
> > > > >>
> > > >
> > >
> https://urldefense.com/v3/__https://github.com/apache/iceberg/blob/2a6c556c0c883c04c6d3fbd68e6aeda36b91e0aa/open-api/rest-catalog-open-api.yaml*L4152__;Iw!!Iz9xO38YGHZK!4rnlVIqBQQQ6xpGe1PMV58edOS_cGrKoqdfhw4qSp_JQCavvEjaxXk0OwGPnxyEcO60AzKQK_XDtp7H3DdkFXcrXCDsqYGo$
> > > > >>
> > > > >> Yufei
> > > > >>
> > > > >>
> > > > >> On Tue, Jun 23, 2026 at 9:30 AM Dmitri Bourlatchkov <
> dimas@apache.org
> > > >
> > > > >> wrote:
> > > > >>
> > > > >> > Hi All,
> > > > >> >
> > > > >> > So, what are the blockers for merging [4115], if any?
> > > > >> >
> > > > >> > [4115] https://urldefense.com/v3/__https://github.com/apache/polaris/pull/4115__;!!Iz9xO38YGHZK!4rnlVIqBQQQ6xpGe1PMV58edOS_cGrKoqdfhw4qSp_JQCavvEjaxXk0OwGPnxyEcO60AzKQK_XDtp7H3DdkFXcrXtFRscKs$
> > > > >> >
> > > > >> > Thanks,
> > > > >> > Dmitri.
> > > > >> >
> > > > >> > On Wed, Jun 17, 2026 at 5:28 PM EJ Wang <
> > > > ej.wang.dev.interest@gmail.com
> > > > >> >
> > > > >> > wrote:
> > > > >> >
> > > > >> > > Hi folks,
> > > > >> > >
> > > > >> > > Thanks again for the discussion today. I updated the sync doc
> with
> > > > >> notes
> > > > >> > > from today's metrics sync:
> > > > >> > >
> > > > >> > >
> > > > >> > >
> > > > >> >
> > > > >>
> > > >
> > >
> https://urldefense.com/v3/__https://docs.google.com/document/d/100h7c4damrUzVuquYbBHM0EvA4LSWuW2IT2dN_7nYVA/edit?tab=t.ezk0rgdx0c6m__;!!Iz9xO38YGHZK!4rnlVIqBQQQ6xpGe1PMV58edOS_cGrKoqdfhw4qSp_JQCavvEjaxXk0OwGPnxyEcO60AzKQK_XDtp7H3DdkFXcrX83IZkm0$
> > > > >> > >
> > > > >> > > A few highlights:
> > > > >> > >
> > > > >> > >    - No objection to keeping the default metrics behavior
> small:
> > > > >> > >    no-op/log-only is enough as the built-in default.
> > > > >> > >    - Durable storage, event forwarding, external queues,
> > > dashboards,
> > > > >> and
> > > > >> > >    custom filtering should be implementation choices behind
> the
> > > > >> metrics
> > > > >> > >    reporting path.
> > > > >> > >    - For PR #4115, the REST layer should resolve/authz/accept
> the
> > > > >> report,
> > > > >> > >    then delegate to the selected metrics implementation. It
> should
> > > > not
> > > > >> > own
> > > > >> > >    durable storage semantics.
> > > > >> > >    - The rough boundary we discussed is: spec owns the wire
> > > > contract,
> > > > >> > >    runtime owns framework wiring, API/contract modules own
> > > > >> > provider-facing
> > > > >> > >    contracts, and extensions own replaceable implementations.
> > > > >> > >    - OpenLineage has a similar path-naming question. We should
> > > avoid
> > > > >> > >    occupying a generic lineage path too early if we want room
> for
> > > > >> other
> > > > >> > >    lineage systems later.
> > > > >> > >
> > > > >> > > I believe the boundary rules previewed in sync meeting are
> useful
> > > > for
> > > > >> > > reviewing PR #4115, but it is not
> > > > >> > > a final module-layout proposal.
> > > > >> > >
> > > > >> > > For metrics specifically, I think the target remains:
> > > > >> > >    runtime
> > > > >> > >      -> metrics reporting/emitting contract
> > > > >> > >      -> default implementation
> > > > >> > >           -> lower implementation details, if needed
> > > > >> > >
> > > > >> > > That keeps the default simple, while allowing durable JDBC,
> > > > >> event-backed,
> > > > >> > > or external-queue implementations to be added separately.
> > > > >> > >
> > > > >> > > We also touched on the split between Iceberg metrics emitting
> and
> > > > >> > > Polaris-owned metrics querying. The querying side can keep
> > > evolving,
> > > > >> > > especially for dashboard or generic-table use cases, but I do
> not
> > > > >> think
> > > > >> > > that needs to block the intake path in this PR.
> > > > >> > >
> > > > >> > > Thanks,
> > > > >> > > -ej
> > > > >> > >
> > > > >> > > On Tue, Jun 16, 2026 at 2:17 PM Yufei Gu <
> flyrain000@gmail.com>
> > > > >> wrote:
> > > > >> > >
> > > > >> > > > Thanks for chiming in, EJ. Agreed that the default battery
> > > should
> > > > >> stay
> > > > >> > > > small. I'm leaning toward not including metrics persistence
> in
> > > the
> > > > >> > > default
> > > > >> > > > battery.
> > > > >> > > >
> > > > >> > > > Yufei
> > > > >> > > >
> > > > >> > > >
> > > > >> > > > On Thu, Jun 11, 2026 at 8:30 PM EJ Wang <
> > > > >> > ej.wang.dev.interest@gmail.com>
> > > > >> > > > wrote:
> > > > >> > > >
> > > > >> > > > > Hi Yufei,
> > > > >> > > > >
> > > > >> > > > > Thanks for connecting this back to the REST endpoint
> > > proposal. I
> > > > >> > > replied
> > > > >> > > > > with the fuller version on the event-forwarding thread,
> but
> > > > >> wanted to
> > > > >> > > add
> > > > >> > > > > the shorter version here too.
> > > > >> > > > >
> > > > >> > > > > I agree that metrics reporting/emitting is the right
> > > conceptual
> > > > >> > > boundary,
> > > > >> > > > > and I also think the event/listener path is a good
> > > > implementation
> > > > >> > > > direction
> > > > >> > > > > to explore. The distinction I want to keep clear is
> default
> > > > >> battery
> > > > >> > vs
> > > > >> > > > > extension implementation.
> > > > >> > > > >
> > > > >> > > > > For the REST/API side, I would keep the semantics narrow:
> > > > Polaris
> > > > >> > > > resolves
> > > > >> > > > > and authorizes the table, accepts the Iceberg scan/commit
> > > > metrics
> > > > >> > > report
> > > > >> > > > > into the configured reporting/emitting path, and returns
> 204.
> > > > That
> > > > >> > 204
> > > > >> > > > > should mean Polaris accepted the report into the ingestion
> > > path;
> > > > >> it
> > > > >> > > > should
> > > > >> > > > > not imply durable storage.
> > > > >> > > > >
> > > > >> > > > > Durable storage, event forwarding, external telemetry
> routing,
> > > > >> > > filtering,
> > > > >> > > > > and retention should sit behind that reporting/emitting
> > > boundary
> > > > >> as
> > > > >> > > > > implementation-layer behavior. A durable JDBC path can be
> one
> > > > >> named
> > > > >> > > > > extension implementation. An event/listener forwarding
> path
> > > can
> > > > be
> > > > >> > > > another
> > > > >> > > > > named extension implementation.
> > > > >> > > > >
> > > > >> > > > > So I am +1 on the event-forwarding idea as a non-default
> > > > extension
> > > > >> > > > > implementation. I just would not make it the default
> battery
> > > or
> > > > >> the
> > > > >> > > core
> > > > >> > > > > API shape. The default battery can stay small and safe,
> while
> > > > >> > > deployments
> > > > >> > > > > that need forwarding, dashboards, or durable storage can
> opt
> > > > into
> > > > >> the
> > > > >> > > > > implementation that matches their operational model.
> > > > >> > > > >
> > > > >> > > > > Thanks,
> > > > >> > > > > -ej
> > > > >> > > > >
> > > > >> > > > > On Wed, Jun 10, 2026 at 11:43 AM Yufei Gu <
> > > flyrain000@gmail.com
> > > > >
> > > > >> > > wrote:
> > > > >> > > > >
> > > > >> > > > > > Thanks EJ. I agree that the reporting/emitting boundary
> > > feels
> > > > >> like
> > > > >> > > the
> > > > >> > > > > > right SPI boundary, but I wonder if we can simplify this
> > > even
> > > > >> > > further.
> > > > >> > > > > >
> > > > >> > > > > > Iceberg scan and commit metrics look very similar to
> events
> > > to
> > > > >> me.
> > > > >> > > They
> > > > >> > > > > are
> > > > >> > > > > > append only, asynchronously consumed, often forwarded to
> > > > >> external
> > > > >> > > > > systems,
> > > > >> > > > > > and have similar retention and cleanup requirements.
> More
> > > > >> details
> > > > >> > can
> > > > >> > > > be
> > > > >> > > > > > found in this thread [1]. In that case, Polaris may only
> > > need
> > > > to
> > > > >> > > accept
> > > > >> > > > > the
> > > > >> > > > > > report and emit an event through the existing event
> > > framework.
> > > > >> > > > Filtering,
> > > > >> > > > > > async delivery, custom sinks, and retention mechanisms
> could
> > > > >> then
> > > > >> > be
> > > > >> > > > > shared
> > > > >> > > > > > instead of introducing a separate metrics specific
> extension
> > > > >> layer
> > > > >> > > and
> > > > >> > > > > SPI.
> > > > >> > > > > >
> > > > >> > > > > > The main requirement I still see is querying metrics
> through
> > > > >> > Polaris.
> > > > >> > > > If
> > > > >> > > > > we
> > > > >> > > > > > want that battery included experience, we could work on
> a
> > > > >> > persistence
> > > > >> > > > > > listener implementation that selectively saves metrics
> and
> > > > >> expose
> > > > >> > it
> > > > >> > > > via
> > > > >> > > > > > the IRC event endpoint which seems pretty close in
> Iceberg
> > > > >> > > > > > community. Another option I like more is to allow users
> to
> > > > >> develop
> > > > >> > > > their
> > > > >> > > > > > own listener, so they can persist scan/commit metrics
> to any
> > > > >> > systems
> > > > >> > > > they
> > > > >> > > > > > prefer. In general, that feels like an implementation
> choice
> > > > on
> > > > >> top
> > > > >> > > of
> > > > >> > > > > the
> > > > >> > > > > > event framework rather than something that needs to
> shape
> > > the
> > > > >> core
> > > > >> > > > > > architecture.
> > > > >> > > > > >
> > > > >> > > > > > 1.
> > > > >> >
> https://urldefense.com/v3/__https://lists.apache.org/thread/x9j8nscvy8hq61tyn01mj8yp6n9of0kp__;!!Iz9xO38YGHZK!4rnlVIqBQQQ6xpGe1PMV58edOS_cGrKoqdfhw4qSp_JQCavvEjaxXk0OwGPnxyEcO60AzKQK_XDtp7H3DdkFXcrXGcGDgks$
> > > > >> > > > > >
> > > > >> > > > > > Thanks,
> > > > >> > > > > > Yufei
> > > > >> > > > > >
> > > > >> > > > > >
> > > > >> > > > > > On Thu, Jun 4, 2026 at 5:14 PM EJ Wang <
> > > > >> > > ej.wang.dev.interest@gmail.com
> > > > >> > > > >
> > > > >> > > > > > wrote:
> > > > >> > > > > >
> > > > >> > > > > > > Good point. I agree the existing
> PolarisMetricsReporter is
> > > > >> > already
> > > > >> > > > very
> > > > >> > > > > > > close to the right conceptual boundary. I am not
> > > proposing a
> > > > >> > second
> > > > >> > > > > > > parallel reporter concept. The distinction I am
> trying to
> > > > >> make is
> > > > >> > > > > > narrower:
> > > > >> > > > > > >
> > > > >> > > > > > > Current state:
> > > > >> > > > > > >
> > > > >> > > > > > >    - PolarisMetricsReporter is the existing metrics
> > > > reporting
> > > > >> > hook.
> > > > >> > > > > > >    - But it currently lives under runtime/service.
> > > > >> > > > > > >    - Its contract is documented around Quarkus/CDI
> > > discovery
> > > > >> and
> > > > >> > > > > > > @Identifier
> > > > >> > > > > > >    selection.
> > > > >> > > > > > >    - The default implementation is log-only.
> > > > >> > > > > > >    - The durable implementation writes through
> > > > >> > > PolarisMetricsManager
> > > > >> > > > > and
> > > > >> > > > > > >    MetricsPersistence.
> > > > >> > > > > > >    - MetricsPersistence is currently inherited by
> > > > >> > BasePersistence.
> > > > >> > > > > > >
> > > > >> > > > > > > My proposal:
> > > > >> > > > > > >
> > > > >> > > > > > >    - Keep the reporting/emitting boundary as the SPI.
> > > > >> > > > > > >    - Revise that contract in a framework-agnostic
> optional
> > > > >> > Iceberg
> > > > >> > > > > > *metrics
> > > > >> > > > > > >    extension API layer*.
> > > > >> > > > > > >    - Keep runtime/service responsible for REST
> ingestion,
> > > > >> > > table/authz
> > > > >> > > > > > >    validation, and runtime wiring.
> > > > >> > > > > > >    - Keep the battery implementation log-only in the
> > > metrics
> > > > >> > > > extension
> > > > >> > > > > > API
> > > > >> > > > > > >    layer, *not under runtime/service*.
> > > > >> > > > > > >    - Treat durable JDBC metrics as one implementation
> of
> > > the
> > > > >> > > > reporting
> > > > >> > > > > > SPI.
> > > > >> > > > > > >    - Decompose the MetricsPersistence and
> BasePersistence
> > > > >> > coupling
> > > > >> > > as
> > > > >> > > > > > >    durable implementation detail.
> > > > >> > > > > > >
> > > > >> > > > > > > So the difference is not “new reporter concept vs old
> > > > reporter
> > > > >> > > > > concept.”
> > > > >> > > > > > > but:
> > > > >> > > > > > >
> > > > >> > > > > > >
> > > > >> > > > > > >    - PolarisMetricsReporter today = runtime/service
> > > > CDI-shaped
> > > > >> > > hook.
> > > > >> > > > > > >    - Target reporting SPI = same conceptual boundary,
> but
> > > > >> placed
> > > > >> > in
> > > > >> > > > the
> > > > >> > > > > > >    right extension API layer and kept
> framework-agnostic.
> > > > >> > > > > > >
> > > > >> > > > > > > For the current metrics PR, I think the minimal
> > > > architectural
> > > > >> > > cleanup
> > > > >> > > > > > > before merge is to *put the reporting SPI contract
> and the
> > > > >> > battery
> > > > >> > > > > > default
> > > > >> > > > > > > in the right place*. The durable JDBC implementation,
> > > > >> > event-backed
> > > > >> > > > > async
> > > > >> > > > > > > implementation, external queue support, and deeper
> > > > persistence
> > > > >> > > > cleanup
> > > > >> > > > > > can
> > > > >> > > > > > > continue as follow-ups, as long as we do not lock the
> SPI
> > > > >> > boundary
> > > > >> > > to
> > > > >> > > > > > > BasePersistence or to runtime/service wiring.
> > > > >> > > > > > >
> > > > >> > > > > > > -ej
> > > > >> > > > > > >
> > > > >> > > > > > > On Wed, Jun 3, 2026 at 3:00 PM Dmitri Bourlatchkov <
> > > > >> > > dimas@apache.org
> > > > >> > > > >
> > > > >> > > > > > > wrote:
> > > > >> > > > > > >
> > > > >> > > > > > > > Hi EJ,
> > > > >> > > > > > > >
> > > > >> > > > > > > > Thanks for the recap / summary!
> > > > >> > > > > > > >
> > > > >> > > > > > > > > Do folks agree that the stable SPI boundary
> should be
> > > > >> > > > > > > > metrics reporting/emitting, not metrics persistence?
> > > > >> > > > > > > >
> > > > >> > > > > > > > SGTM.
> > > > >> > > > > > > >
> > > > >> > > > > > > > > Does an optional Iceberg metrics extension API
> layer
> > > > sound
> > > > >> > like
> > > > >> > > > the
> > > > >> > > > > > > right
> > > > >> > > > > > > > home for this SPI?
> > > > >> > > > > > > >
> > > > >> > > > > > > > How is that different from current
> > > PolarisMetricsReporter?
> > > > >> > > > > > > >
> > > > >> > > > > > > > > Should the current durable metrics work be
> reframed
> > > as a
> > > > >> > > durable
> > > > >> > > > > > > > JDBC reference implementation of that SPI?
> > > > >> > > > > > > >
> > > > >> > > > > > > > SGTM.
> > > > >> > > > > > > >
> > > > >> > > > > > > > > What is the smallest PR sequence to get there
> without
> > > > >> > blocking
> > > > >> > > > > > > > the current metrics work unnecessarily?
> > > > >> > > > > > > >
> > > > >> > > > > > > > Let's get
> https://urldefense.com/v3/__https://github.com/apache/polaris/pull/4397__;!!Iz9xO38YGHZK!4rnlVIqBQQQ6xpGe1PMV58edOS_cGrKoqdfhw4qSp_JQCavvEjaxXk0OwGPnxyEcO60AzKQK_XDtp7H3DdkFXcrXuu5jefU$
> > > > >> merged
> > > > >> > > > first.
> > > > >> > > > > > > >
> > > > >> > > > > > > > Then, I'd propose isolating the Metrics schema from
> the
> > > > >> > MetaStore
> > > > >> > > > > > schema.
> > > > >> > > > > > > > This will probably have some ripple effect into the
> > > > >> bootstrap
> > > > >> > > > > > workflows,
> > > > >> > > > > > > so
> > > > >> > > > > > > > it's not a trivial change.
> > > > >> > > > > > > >
> > > > >> > > > > > > > Then, let's reassess.
> > > > >> > > > > > > >
> > > > >> > > > > > > > Cheers,
> > > > >> > > > > > > > Dmitri.
> > > > >> > > > > > > >
> > > > >> > > > > > > > On Wed, Jun 3, 2026 at 5:40 PM EJ Wang <
> > > > >> > > > > ej.wang.dev.interest@gmail.com
> > > > >> > > > > > >
> > > > >> > > > > > > > wrote:
> > > > >> > > > > > > >
> > > > >> > > > > > > > > Hi folks,
> > > > >> > > > > > > > >
> > > > >> > > > > > > > > Thanks again for the discussion today. I updated
> the
> > > > sync
> > > > >> doc
> > > > >> > > > with
> > > > >> > > > > > > notes
> > > > >> > > > > > > > > from the third metrics architecture sync:
> > > > >> > > > > > > > >
> > > > >> > > > > > > > >
> > > > >> > > > > > > > >
> > > > >> > > > > > > >
> > > > >> > > > > > >
> > > > >> > > > > >
> > > > >> > > > >
> > > > >> > > >
> > > > >> > >
> > > > >> >
> > > > >>
> > > >
> > >
> https://urldefense.com/v3/__https://docs.google.com/document/d/100h7c4damrUzVuquYbBHM0EvA4LSWuW2IT2dN_7nYVA/edit?tab=t.k96s2xyqr5u1__;!!Iz9xO38YGHZK!4rnlVIqBQQQ6xpGe1PMV58edOS_cGrKoqdfhw4qSp_JQCavvEjaxXk0OwGPnxyEcO60AzKQK_XDtp7H3DdkFXcrXrqCL-QE$
> > > > >> > > > > > > > >
> > > > >> > > > > > > > > A few highlights from today’s discussion:
> > > > >> > > > > > > > >
> > > > >> > > > > > > > >    - We clarified that Iceberg metrics reporting
> can
> > > be
> > > > >> > > > interpreted
> > > > >> > > > > > > > either
> > > > >> > > > > > > > >    as sync or async handling from Polaris’s
> > > perspective,
> > > > >> so
> > > > >> > > > Polaris
> > > > >> > > > > > > > should
> > > > >> > > > > > > > >    stay flexible as a platform instead of baking
> one
> > > > >> handling
> > > > >> > > > model
> > > > >> > > > > > > into
> > > > >> > > > > > > > > the
> > > > >> > > > > > > > >    REST API behavior.
> > > > >> > > > > > > > >    - We aligned that the built-in (battery)
> behavior
> > > for
> > > > >> > > metrics
> > > > >> > > > > > > emitting
> > > > >> > > > > > > > >    can stay simple: no-op/log-only is enough as
> the
> > > > >> default.
> > > > >> > > > > > > > >    - Durable metrics persistence should be
> treated as
> > > an
> > > > >> > > > > > implementation
> > > > >> > > > > > > > of
> > > > >> > > > > > > > >    the metrics reporting path, not as the core SPI
> > > > >> boundary.
> > > > >> > > > > > > > >    - The existing durable metrics work can be
> reviewed
> > > > as
> > > > >> a
> > > > >> > > > > reference
> > > > >> > > > > > > > >    implementation of the reporting SPI, with
> > > > >> > > persistence-related
> > > > >> > > > > > logic
> > > > >> > > > > > > > kept
> > > > >> > > > > > > > >    self-contained under a metrics durable
> > > implementation
> > > > >> > module
> > > > >> > > > > > rather
> > > > >> > > > > > > > than
> > > > >> > > > > > > > >    scattered through core entity persistence.
> > > > >> > > > > > > > >    - Dashboard/insights remains a real use case,
> but
> > > we
> > > > >> > agreed
> > > > >> > > to
> > > > >> > > > > > keep
> > > > >> > > > > > > it
> > > > >> > > > > > > > >    separate from the core metrics intake
> discussion
> > > for
> > > > >> now.
> > > > >> > > > > > > > >
> > > > >> > > > > > > > > I also did a quick source check after the meeting
> to
> > > > make
> > > > >> > sure
> > > > >> > > we
> > > > >> > > > > are
> > > > >> > > > > > > > > describing the current state accurately.
> > > > >> > > > > > > > >
> > > > >> > > > > > > > > Current state:
> > > > >> > > > > > > > >
> > > > >> > > > > > > > >    - Polaris already has a metrics reporting hook:
> > > > >> > > > > > > > PolarisMetricsReporter.
> > > > >> > > > > > > > >    - The default implementation is
> > > > DefaultMetricsReporter,
> > > > >> > > > selected
> > > > >> > > > > > by
> > > > >> > > > > > > > >
> polaris.iceberg-metrics.reporting.type=default. It
> > > is
> > > > >> > > > log-only,
> > > > >> > > > > > > > >    effectively quiet unless metrics logging is
> > > enabled.
> > > > >> > > > > > > > >    - There is also a PersistingMetricsReporter,
> > > selected
> > > > >> by
> > > > >> > > > > > > > >
> polaris.iceberg-metrics.reporting.type=persisting,
> > > > >> which
> > > > >> > > > > converts
> > > > >> > > > > > > > >    Iceberg scan/commit reports into Polaris
> metrics
> > > > >> records
> > > > >> > and
> > > > >> > > > > > writes
> > > > >> > > > > > > > > through
> > > > >> > > > > > > > >    PolarisMetricsManager -> MetricsPersistence.
> > > > >> > > > > > > > >    - MetricsPersistence currently exists in the
> > > > >> persistence
> > > > >> > > layer
> > > > >> > > > > and
> > > > >> > > > > > > is
> > > > >> > > > > > > > >    inherited by BasePersistence.
> > > > >> > > > > > > > >
> > > > >> > > > > > > > > My read from the discussion is that the target
> > > boundary
> > > > >> > should
> > > > >> > > > not
> > > > >> > > > > be
> > > > >> > > > > > > > > MetricsPersistence as inherited by
> BasePersistence.
> > > That
> > > > >> path
> > > > >> > > > > should
> > > > >> > > > > > be
> > > > >> > > > > > > > > decomposed as durable implementation detail (taken
> > > care
> > > > >> of by
> > > > >> > > > > > > > > https://urldefense.com/v3/__https://github.com/apache/polaris/pull/4397__;!!Iz9xO38YGHZK!4rnlVIqBQQQ6xpGe1PMV58edOS_cGrKoqdfhw4qSp_JQCavvEjaxXk0OwGPnxyEcO60AzKQK_XDtp7H3DdkFXcrXuu5jefU$). The
> > > > stable
> > > > >> > > > extension
> > > > >> > > > > > > point
> > > > >> > > > > > > > > should instead be the metrics reporting/emitting
> > > > boundary.
> > > > >> > > > > > > > >
> > > > >> > > > > > > > > Concretely, I think the next proposal should be
> shaped
> > > > >> like
> > > > >> > > this:
> > > > >> > > > > > > > >
> > > > >> > > > > > > > >    1.
> > > > >> > > > > > > > >
> > > > >> > > > > > > > >    Define the proper metrics reporting SPI at an
> > > > optional
> > > > >> > > Iceberg
> > > > >> > > > > > > metrics
> > > > >> > > > > > > > >    extension API layer.
> > > > >> > > > > > > > >
> > > > >> > > > > > > > >    Example direction:
> > > > >> > > > > > > > >
> > > > >> > > > > > > > >    public interface IcebergMetricsReporter {
> > > > >> > > > > > > > >      void reportMetric(IcebergMetricsReportContext
> > > > >> context,
> > > > >> > > > > > > > > MetricsReport report);
> > > > >> > > > > > > > >    }
> > > > >> > > > > > > > >
> > > > >> > > > > > > > >    The context would carry the small
> Polaris-resolved
> > > > >> > envelope:
> > > > >> > > > > > > > >    catalog/table identity, report type, received
> > > > >> timestamp,
> > > > >> > and
> > > > >> > > > > > > > > request/trace
> > > > >> > > > > > > > >    context if available. The raw Iceberg
> MetricsReport
> > > > >> > remains
> > > > >> > > > the
> > > > >> > > > > > > > payload.
> > > > >> > > > > > > > >    2.
> > > > >> > > > > > > > >
> > > > >> > > > > > > > >    Keep runtime/service as the REST ingestion and
> > > wiring
> > > > >> > layer.
> > > > >> > > > > > > > >
> > > > >> > > > > > > > >    The REST handler still resolves the table and
> > > > performs
> > > > >> > authz
> > > > >> > > > > > before
> > > > >> > > > > > > > >    accepting the report. After that, it calls the
> > > > selected
> > > > >> > > > > reporting
> > > > >> > > > > > > > >    implementation.
> > > > >> > > > > > > > >    3.
> > > > >> > > > > > > > >
> > > > >> > > > > > > > >    Keep the battery default no-op/log-only.
> > > > >> > > > > > > > >
> > > > >> > > > > > > > >    This preserves an out-of-box safe default and
> > > avoids
> > > > >> > > > requiring a
> > > > >> > > > > > > > durable
> > > > >> > > > > > > > >    metrics store for every Polaris deployment.
> > > > >> > > > > > > > >    4.
> > > > >> > > > > > > > >
> > > > >> > > > > > > > >    Move durable JDBC metrics into a self-contained
> > > > >> > > implementation
> > > > >> > > > > of
> > > > >> > > > > > > the
> > > > >> > > > > > > > >    reporting SPI.
> > > > >> > > > > > > > >
> > > > >> > > > > > > > >    That implementation can own its schema,
> bootstrap,
> > > > >> > > retention,
> > > > >> > > > > and
> > > > >> > > > > > > read
> > > > >> > > > > > > > >    API support. It should not define the core
> > > reporting
> > > > >> SPI
> > > > >> > > > > boundary,
> > > > >> > > > > > > and
> > > > >> > > > > > > > > it
> > > > >> > > > > > > > >    should not require metrics persistence to
> remain
> > > > >> inherited
> > > > >> > > > from
> > > > >> > > > > > > > >    BasePersistence.
> > > > >> > > > > > > > >    5.
> > > > >> > > > > > > > >
> > > > >> > > > > > > > >    Treat async/event-backed handling as another
> > > > >> > implementation
> > > > >> > > of
> > > > >> > > > > the
> > > > >> > > > > > > > same
> > > > >> > > > > > > > >    reporting SPI.
> > > > >> > > > > > > > >
> > > > >> > > > > > > > >    For example, an event-backed reporter could
> > > enqueue a
> > > > >> > > metrics
> > > > >> > > > > > event
> > > > >> > > > > > > > and
> > > > >> > > > > > > > >    let listeners handle durable storage or other
> > > sinks.
> > > > >> If we
> > > > >> > > > later
> > > > >> > > > > > > need
> > > > >> > > > > > > > a
> > > > >> > > > > > > > >    replaceable queue engine, that seems like a
> shared
> > > > >> > > > event/metrics
> > > > >> > > > > > > > > substrate
> > > > >> > > > > > > > >    topic rather than a metrics-only requirement.
> > > > >> > > > > > > > >
> > > > >> > > > > > > > > This framing lets us keep the REST metrics
> endpoint
> > > > >> simple,
> > > > >> > > > > preserve
> > > > >> > > > > > > the
> > > > >> > > > > > > > > current default behavior, support durable metrics
> > > users,
> > > > >> and
> > > > >> > > > still
> > > > >> > > > > > > leave
> > > > >> > > > > > > > > room for async/event-backed or
> external-queue-based
> > > > >> > > > > implementations.
> > > > >> > > > > > > > >
> > > > >> > > > > > > > > I think the main follow-up questions are:
> > > > >> > > > > > > > >
> > > > >> > > > > > > > >    - Do folks agree that the stable SPI boundary
> > > should
> > > > be
> > > > >> > > > metrics
> > > > >> > > > > > > > >    reporting/emitting, not metrics persistence?
> > > > >> > > > > > > > >    - Does an optional Iceberg metrics extension
> API
> > > > layer
> > > > >> > sound
> > > > >> > > > > like
> > > > >> > > > > > > the
> > > > >> > > > > > > > >    right home for this SPI?
> > > > >> > > > > > > > >    - Should the current durable metrics work be
> > > reframed
> > > > >> as a
> > > > >> > > > > durable
> > > > >> > > > > > > > JDBC
> > > > >> > > > > > > > >    reference implementation of that SPI?
> > > > >> > > > > > > > >    - What is the smallest PR sequence to get there
> > > > without
> > > > >> > > > blocking
> > > > >> > > > > > the
> > > > >> > > > > > > > >    current metrics work unnecessarily?
> > > > >> > > > > > > > >
> > > > >> > > > > > > > > Thanks,
> > > > >> > > > > > > > > -ej
> > > > >> > > > > > > > >
> > > > >> > > > > > > > > On Fri, May 15, 2026 at 5:16 PM Dmitri
> Bourlatchkov <
> > > > >> > > > > > dimas@apache.org>
> > > > >> > > > > > > > > wrote:
> > > > >> > > > > > > > >
> > > > >> > > > > > > > > > Hi JB,
> > > > >> > > > > > > > > >
> > > > >> > > > > > > > > > Could you set up another meeting, please? Same
> time
> > > on
> > > > >> > > > Wednesday
> > > > >> > > > > as
> > > > >> > > > > > > > last
> > > > >> > > > > > > > > > time... I hope it works for everyone.
> > > > >> > > > > > > > > >
> > > > >> > > > > > > > > > Cheers,
> > > > >> > > > > > > > > > Dmitri.
> > > > >> > > > > > > > > >
> > > > >> > > > > > > > > > On Fri, May 15, 2026 at 8:06 PM Yufei Gu <
> > > > >> > > flyrain000@gmail.com
> > > > >> > > > >
> > > > >> > > > > > > wrote:
> > > > >> > > > > > > > > >
> > > > >> > > > > > > > > > > +1 on another sync call next week.
> > > > >> > > > > > > > > > >
> > > > >> > > > > > > > > > > Yufei
> > > > >> > > > > > > > > > >
> > > > >> > > > > > > > > > >
> > > > >> > > > > > > > > > > On Fri, May 15, 2026 at 4:52 PM Dmitri
> > > Bourlatchkov
> > > > <
> > > > >> > > > > > > > dimas@apache.org>
> > > > >> > > > > > > > > > > wrote:
> > > > >> > > > > > > > > > >
> > > > >> > > > > > > > > > > > Hi All,
> > > > >> > > > > > > > > > > >
> > > > >> > > > > > > > > > > > WDYT about another sync call next week?
> > > > >> > > > > > > > > > > >
> > > > >> > > > > > > > > > > > Thanks,
> > > > >> > > > > > > > > > > > Dmitri.
> > > > >> > > > > > > > > > > >
> > > > >> > > > > > > > > > > > On Wed, May 6, 2026 at 5:29 PM Dmitri
> > > > Bourlatchkov <
> > > > >> > > > > > > > dimas@apache.org
> > > > >> > > > > > > > > >
> > > > >> > > > > > > > > > > > wrote:
> > > > >> > > > > > > > > > > >
> > > > >> > > > > > > > > > > > > Hi EJ,
> > > > >> > > > > > > > > > > > >
> > > > >> > > > > > > > > > > > > Thanks for the summary! It covers what we
> > > > >> discussed
> > > > >> > in
> > > > >> > > > the
> > > > >> > > > > > > > meeting
> > > > >> > > > > > > > > > very
> > > > >> > > > > > > > > > > > > well, IMHO.
> > > > >> > > > > > > > > > > > >
> > > > >> > > > > > > > > > > > > Looking forward to concrete PRs :)
> > > > >> > > > > > > > > > > > >
> > > > >> > > > > > > > > > > > > Cheers,
> > > > >> > > > > > > > > > > > > Dmitri.
> > > > >> > > > > > > > > > > > >
> > > > >> > > > > > > > > > > > > On Wed, May 6, 2026 at 5:08 PM EJ Wang <
> > > > >> > > > > > > > > > ej.wang.dev.interest@gmail.com
> > > > >> > > > > > > > > > > >
> > > > >> > > > > > > > > > > > > wrote:
> > > > >> > > > > > > > > > > > >
> > > > >> > > > > > > > > > > > >> Hi folks,
> > > > >> > > > > > > > > > > > >>
> > > > >> > > > > > > > > > > > >> We had a community sync earlier, thanks
> JB
> > > for
> > > > >> > > > scheduling
> > > > >> > > > > > it.
> > > > >> > > > > > > > > Notes
> > > > >> > > > > > > > > > > from
> > > > >> > > > > > > > > > > > >> the first metrics architecture sync (May
> 6,
> > > > >> 10-11am
> > > > >> > > PT).
> > > > >> > > > > > > > > Discussion
> > > > >> > > > > > > > > > > doc
> > > > >> > > > > > > > > > > > >> with per-section status:
> > > > >> > > > > > > > > > > > >>
> > > > >> > > > > > > > > > > > >>
> > > > >> > > > > > > > > > > >
> > > > >> > > > > > > > > > >
> > > > >> > > > > > > > > >
> > > > >> > > > > > > > >
> > > > >> > > > > > > >
> > > > >> > > > > > >
> > > > >> > > > > >
> > > > >> > > > >
> > > > >> > > >
> > > > >> > >
> > > > >> >
> > > > >>
> > > >
> > >
> https://urldefense.com/v3/__https://docs.google.com/document/d/100h7c4damrUzVuquYbBHM0EvA4LSWuW2IT2dN_7nYVA/edit?tab=t.0__;!!Iz9xO38YGHZK!4rnlVIqBQQQ6xpGe1PMV58edOS_cGrKoqdfhw4qSp_JQCavvEjaxXk0OwGPnxyEcO60AzKQK_XDtp7H3DdkFXcrXgIddeUs$
> > > > >> > > > > > > > > > > > >>
> > > > >> > > > > > > > > > > > >> *The meeting covered both topics from the
> > > doc.
> > > > >> > > > > > Direction-level
> > > > >> > > > > > > > > > > alignment
> > > > >> > > > > > > > > > > > >> was reached on the headline pieces;
> details
> > > > >> remain
> > > > >> > for
> > > > >> > > > PR
> > > > >> > > > > > > review
> > > > >> > > > > > > > > or
> > > > >> > > > > > > > > > > > >> follow-up sessions.*
> > > > >> > > > > > > > > > > > >>
> > > > >> > > > > > > > > > > > >> *Topic 1 — Persistence schema redesign*
> > > > >> > > > > > > > > > > > >> Idea-level alignment on consolidating
> > > per-type
> > > > >> > tables
> > > > >> > > > > > > > > > > > >> (scan_metrics_report,
> > > > >> > > > > > > > > > > > >> commit_metrics_report) into a single
> > > > >> metrics_report
> > > > >> > > > table.
> > > > >> > > > > > The
> > > > >> > > > > > > > > > > > motivating
> > > > >> > > > > > > > > > > > >> cost is the surface area added by every
> new
> > > > >> metric
> > > > >> > > type
> > > > >> > > > > > today:
> > > > >> > > > > > > > new
> > > > >> > > > > > > > > > > > table,
> > > > >> > > > > > > > > > > > >> SPI method, record class, model,
> converter,
> > > > >> schema
> > > > >> > > > > > migration.
> > > > >> > > > > > > > > > > > >>
> > > > >> > > > > > > > > > > > >> Most schema details are deferred to the
> > > schema
> > > > >> PR. A
> > > > >> > > few
> > > > >> > > > > > > > specific
> > > > >> > > > > > > > > > > points
> > > > >> > > > > > > > > > > > >> came up:
> > > > >> > > > > > > > > > > > >> •   metric_schema_version: Yufei prefers
> > > > dropping
> > > > >> > it,
> > > > >> > > > > since
> > > > >> > > > > > > > there
> > > > >> > > > > > > > > is
> > > > >> > > > > > > > > > > no
> > > > >> > > > > > > > > > > > >> spec-level concept of metrics versioning
> > > today
> > > > >> and
> > > > >> > it
> > > > >> > > is
> > > > >> > > > > > hard
> > > > >> > > > > > > to
> > > > >> > > > > > > > > > > define
> > > > >> > > > > > > > > > > > >> unilaterally. Robert prefers keeping it,
> > > given
> > > > >> IRC
> > > > >> > v2
> > > > >> > > is
> > > > >> > > > > > > coming
> > > > >> > > > > > > > > and
> > > > >> > > > > > > > > > > the
> > > > >> > > > > > > > > > > > >> schema should be considered against its
> > > likely
> > > > >> > shape;
> > > > >> > > > > Robert
> > > > >> > > > > > > > also
> > > > >> > > > > > > > > > > raised
> > > > >> > > > > > > > > > > > >> how to differentiate various payload
> formats
> > > if
> > > > >> any.
> > > > >> > > > EJ's
> > > > >> > > > > > read
> > > > >> > > > > > > > is
> > > > >> > > > > > > > > > that
> > > > >> > > > > > > > > > > > >> this
> > > > >> > > > > > > > > > > > >> is a two-way-door decision. We can start
> > > > without
> > > > >> the
> > > > >> > > > > field,
> > > > >> > > > > > > and
> > > > >> > > > > > > > if
> > > > >> > > > > > > > > > IRC
> > > > >> > > > > > > > > > > > v2
> > > > >> > > > > > > > > > > > >> changes the shape we would likely roll a
> > > > >> > corresponding
> > > > >> > > > new
> > > > >> > > > > > > > schema
> > > > >> > > > > > > > > > > > anyway,
> > > > >> > > > > > > > > > > > >> which is not particularly costly.
> > > > >> > > > > > > > > > > > >> •   Payload format: Robert pointed out
> that
> > > > >> future
> > > > >> > > > formats
> > > > >> > > > > > > > beyond
> > > > >> > > > > > > > > > JSON
> > > > >> > > > > > > > > > > > may
> > > > >> > > > > > > > > > > > >> be worth supporting. The exact shape is
> > > > deferred
> > > > >> to
> > > > >> > > the
> > > > >> > > > > > schema
> > > > >> > > > > > > > > > > > discussion.
> > > > >> > > > > > > > > > > > >> •   Partition strategy: Anand suggested
> > > monthly
> > > > >> > > > > partitioning
> > > > >> > > > > > > > based
> > > > >> > > > > > > > > > on
> > > > >> > > > > > > > > > > > his
> > > > >> > > > > > > > > > > > >> experience as potentially helpful at
> scale.
> > > > >> > > > > > > > > > > > >>
> > > > >> > > > > > > > > > > > >> *Topic 2 — Where metrics ingestion and
> > > storage
> > > > >> > belong*
> > > > >> > > > > > > > > > > > >> Idea-level alignment that metrics should
> be a
> > > > >> > > separated
> > > > >> > > > > SPI
> > > > >> > > > > > > from
> > > > >> > > > > > > > > the
> > > > >> > > > > > > > > > > > >> entity
> > > > >> > > > > > > > > > > > >> persistence stack. Two reasons surfaced:
> (a)
> > > > >> > workloads
> > > > >> > > > and
> > > > >> > > > > > > > > > capability
> > > > >> > > > > > > > > > > > >> requirements diverge enough that coupling
> > > them
> > > > >> > creates
> > > > >> > > > > > > > artificial
> > > > >> > > > > > > > > > > > >> constraints, and (b) admin experience
> > > improves
> > > > >> when
> > > > >> > > > > metrics
> > > > >> > > > > > > has
> > > > >> > > > > > > > > its
> > > > >> > > > > > > > > > > own
> > > > >> > > > > > > > > > > > >> bootstrap, retention, and lifecycle.
> Dmitri
> > > > noted
> > > > >> > > > Polaris
> > > > >> > > > > > > being
> > > > >> > > > > > > > a
> > > > >> > > > > > > > > > > > platform
> > > > >> > > > > > > > > > > > >> should have the flexibility to support
> > > > different
> > > > >> > > > > persistence
> > > > >> > > > > > > > > > backends
> > > > >> > > > > > > > > > > > per
> > > > >> > > > > > > > > > > > >> concern, and pointed to a concrete next
> step
> > > of
> > > > >> > > > separating
> > > > >> > > > > > the
> > > > >> > > > > > > > > JDBC
> > > > >> > > > > > > > > > > > >> bootstrap for metrics from the metastore
> > > > >> bootstrap.
> > > > >> > > > Robert
> > > > >> > > > > > > > > proposed
> > > > >> > > > > > > > > > an
> > > > >> > > > > > > > > > > > >> additional UX extension: detect an
> > > > unbootstrapped
> > > > >> > > > metrics
> > > > >> > > > > > > store
> > > > >> > > > > > > > on
> > > > >> > > > > > > > > > > first
> > > > >> > > > > > > > > > > > >> use and auto-bootstrap rather than
> requiring
> > > an
> > > > >> > > explicit
> > > > >> > > > > > > manual
> > > > >> > > > > > > > > > > > bootstrap
> > > > >> > > > > > > > > > > > >> step.
> > > > >> > > > > > > > > > > > >> The meeting also confirmed that Polaris
> > > metrics
> > > > >> can
> > > > >> > > > start
> > > > >> > > > > > > small
> > > > >> > > > > > > > > and
> > > > >> > > > > > > > > > > stay
> > > > >> > > > > > > > > > > > >> Iceberg-focused. Naming and persistence
> > > schema
> > > > >> can
> > > > >> > > lean
> > > > >> > > > > > > > > > > > Iceberg-specific.
> > > > >> > > > > > > > > > > > >> If a future expansion to generic-table
> > > metrics
> > > > or
> > > > >> > > > > > operational
> > > > >> > > > > > > > > > metrics
> > > > >> > > > > > > > > > > > >> arrives, an abstraction layer can be
> built on
> > > > >> top of
> > > > >> > > the
> > > > >> > > > > > > Iceberg
> > > > >> > > > > > > > > > > metrics
> > > > >> > > > > > > > > > > > >> reporter at that point. Robert remains
> on the
> > > > >> fence
> > > > >> > > and
> > > > >> > > > > > would
> > > > >> > > > > > > > > prefer
> > > > >> > > > > > > > > > > > >> something more generic but did not block
> the
> > > > >> > > direction;
> > > > >> > > > > > > Dmitri's
> > > > >> > > > > > > > > > read
> > > > >> > > > > > > > > > > > was
> > > > >> > > > > > > > > > > > >> that the proposed framework already has
> > > enough
> > > > >> > > > flexibility
> > > > >> > > > > > to
> > > > >> > > > > > > > > absorb
> > > > >> > > > > > > > > > > > >> future
> > > > >> > > > > > > > > > > > >> expansion.
> > > > >> > > > > > > > > > > > >>
> > > > >> > > > > > > > > > > > >> The Trade-offs and Proposed structure
> > > sections
> > > > in
> > > > >> > the
> > > > >> > > > doc
> > > > >> > > > > > were
> > > > >> > > > > > > > not
> > > > >> > > > > > > > > > > > >> reviewed
> > > > >> > > > > > > > > > > > >> in detail. They remain open for either
> the
> > > next
> > > > >> sync
> > > > >> > > or
> > > > >> > > > PR
> > > > >> > > > > > > > review.
> > > > >> > > > > > > > > > > > >>
> > > > >> > > > > > > > > > > > >> *Cross-cutting alignment —
> battery-included
> > > > plus
> > > > >> > > > > pluggable*
> > > > >> > > > > > > > > > > > >> A common philosophy emerged from the
> > > > discussion.
> > > > >> EJ
> > > > >> > > > > > summarized
> > > > >> > > > > > > > it
> > > > >> > > > > > > > > > as:
> > > > >> > > > > > > > > > > > >> Polaris should provide a
> battery-included UX
> > > > for
> > > > >> > > > beginners
> > > > >> > > > > > and
> > > > >> > > > > > > > the
> > > > >> > > > > > > > > > > > >> flexibility for advanced users to swap
> the
> > > > >> included
> > > > >> > > > > battery
> > > > >> > > > > > > for
> > > > >> > > > > > > > > > > > something
> > > > >> > > > > > > > > > > > >> more powerful or tailored to their use
> case.
> > > > The
> > > > >> SPI
> > > > >> > > > > design
> > > > >> > > > > > > > needs
> > > > >> > > > > > > > > to
> > > > >> > > > > > > > > > > > >> enable
> > > > >> > > > > > > > > > > > >> both.
> > > > >> > > > > > > > > > > > >>
> > > > >> > > > > > > > > > > > >> The inputs that shaped this framing:
> > > > >> > > > > > > > > > > > >> •   Anand described how his team uses the
> > > > current
> > > > >> > > > metrics
> > > > >> > > > > > > > > > persistence
> > > > >> > > > > > > > > > > > >> (three metrics consumers in v1.4).
> > > > >> > > > > > > > > > > > >> •   Yufei raised Grafana and dashboard
> > > > >> integrations
> > > > >> > > as a
> > > > >> > > > > > > > > destination
> > > > >> > > > > > > > > > > use
> > > > >> > > > > > > > > > > > >> case beyond the default.
> > > > >> > > > > > > > > > > > >> •   Robert called out that the current
> design
> > > > is
> > > > >> > more
> > > > >> > > > > > > > > JDBC-focused.
> > > > >> > > > > > > > > > > > >>
> > > > >> > > > > > > > > > > > >> Two concrete instances:
> > > > >> > > > > > > > > > > > >> •   Async metrics intake: Yufei's initial
> > > > >> position
> > > > >> > was
> > > > >> > > > > that
> > > > >> > > > > > > > async
> > > > >> > > > > > > > > > > should
> > > > >> > > > > > > > > > > > >> largely live on the producer side and
> there
> > > is
> > > > >> not
> > > > >> > > much
> > > > >> > > > > > > Polaris
> > > > >> > > > > > > > > can
> > > > >> > > > > > > > > > > do.
> > > > >> > > > > > > > > > > > >> Robert suggested a Polaris-side default
> is
> > > > doable
> > > > >> > via
> > > > >> > > > > > Vert.x.
> > > > >> > > > > > > > > Dmitri
> > > > >> > > > > > > > > > > > >> agreed
> > > > >> > > > > > > > > > > > >> the direction is worth exploring. The
> meeting
> > > > >> > > converged
> > > > >> > > > > on a
> > > > >> > > > > > > > > > > > >> battery-included default (likely
> > > Vert.x-backed)
> > > > >> with
> > > > >> > > an
> > > > >> > > > > SPI
> > > > >> > > > > > > > shape
> > > > >> > > > > > > > > > that
> > > > >> > > > > > > > > > > > >> lets
> > > > >> > > > > > > > > > > > >> power users route to a more scalable
> backend
> > > > >> > > (k8s-hosted
> > > > >> > > > > > > queue,
> > > > >> > > > > > > > > AWS
> > > > >> > > > > > > > > > > SQS,
> > > > >> > > > > > > > > > > > >> etc.).
> > > > >> > > > > > > > > > > > >> •   Pluggable destinations: combining
> Yufei's
> > > > >> > > dashboard
> > > > >> > > > > use
> > > > >> > > > > > > case
> > > > >> > > > > > > > > > with
> > > > >> > > > > > > > > > > > >> Robert's JDBC-focused call-out, the
> meeting
> > > > >> agreed
> > > > >> > the
> > > > >> > > > SPI
> > > > >> > > > > > > > should
> > > > >> > > > > > > > > be
> > > > >> > > > > > > > > > > > >> structured for multiple sinks so
> integrations
> > > > >> become
> > > > >> > > > impl
> > > > >> > > > > > > > choices
> > > > >> > > > > > > > > > > rather
> > > > >> > > > > > > > > > > > >> than architectural changes.
> > > > >> > > > > > > > > > > > >>
> > > > >> > > > > > > > > > > > >> The battery-included default is most
> likely
> > > to
> > > > >> use
> > > > >> > the
> > > > >> > > > > > > existing
> > > > >> > > > > > > > > > > > >> JDBC-backed
> > > > >> > > > > > > > > > > > >> approach.
> > > > >> > > > > > > > > > > > >>
> > > > >> > > > > > > > > > > > >> *Direction (idea-level alignment)*
> > > > >> > > > > > > > > > > > >> •   Single metrics_report table
> consolidating
> > > > >> > per-type
> > > > >> > > > > > > metrics,
> > > > >> > > > > > > > > > > > replacing
> > > > >> > > > > > > > > > > > >> scan_metrics_report and
> commit_metrics_report
> > > > >> > > > > > > > > > > > >> •   Iceberg-focused naming and schema for
> > > now,
> > > > >> > revisit
> > > > >> > > > if
> > > > >> > > > > > > > > > > generic-table
> > > > >> > > > > > > > > > > > or
> > > > >> > > > > > > > > > > > >> operational metrics arrive
> > > > >> > > > > > > > > > > > >> •   Metrics persistence as a separated
> SPI,
> > > not
> > > > >> on
> > > > >> > > > > > > > BasePersistence
> > > > >> > > > > > > > > > > > >> •   Bootstrap path separated for metrics,
> > > > >> > independent
> > > > >> > > of
> > > > >> > > > > > > > metastore
> > > > >> > > > > > > > > > > > >> bootstrap
> > > > >> > > > > > > > > > > > >> •   "Battery-included plus pluggable" as
> the
> > > > SPI
> > > > >> > > design
> > > > >> > > > > > > > philosophy
> > > > >> > > > > > > > > > > > >>
> > > > >> > > > > > > > > > > > >> *Open items*
> > > > >> > > > > > > > > > > > >> •   Schema details:
> metric_schema_version,
> > > > >> payload
> > > > >> > > > format,
> > > > >> > > > > > IRC
> > > > >> > > > > > > > v2
> > > > >> > > > > > > > > > > > >> forward-compat shape
> > > > >> > > > > > > > > > > > >> •   SPI design details — full review
> either
> > > in
> > > > >> the
> > > > >> > > next
> > > > >> > > > > sync
> > > > >> > > > > > > or
> > > > >> > > > > > > > in
> > > > >> > > > > > > > > > the
> > > > >> > > > > > > > > > > > >> corresponding PR
> > > > >> > > > > > > > > > > > >> •   Schema refactor PR ownership
> > > > >> > > > > > > > > > > > >>
> > > > >> > > > > > > > > > > > >> *Action items*
> > > > >> > > > > > > > > > > > >> •   EJ to take a first stab at the SPI
> design
> > > > and
> > > > >> > > > > > potentially
> > > > >> > > > > > > > > > partner
> > > > >> > > > > > > > > > > > with
> > > > >> > > > > > > > > > > > >> Anand to incorporate the lessons learned
> from
> > > > the
> > > > >> > > > existing
> > > > >> > > > > > > > > reporter
> > > > >> > > > > > > > > > > and
> > > > >> > > > > > > > > > > > >> persistence work.
> > > > >> > > > > > > > > > > > >> •   Schema refactor PR ownership is not
> yet
> > > > >> decided.
> > > > >> > > If
> > > > >> > > > > > anyone
> > > > >> > > > > > > > is
> > > > >> > > > > > > > > > > > >> interested in driving it, reply on this
> > > thread.
> > > > >> > > > > > > > > > > > >> •   JB to schedule the next sync,
> tentatively
> > > > in
> > > > >> two
> > > > >> > > > > weeks.
> > > > >> > > > > > > > > > > > >>
> > > > >> > > > > > > > > > > > >> -ej
> > > > >> > > > > > > > > > > > >>
> > > > >> > > > > > > > > > > > >> On Mon, Apr 27, 2026 at 3:07 PM EJ Wang <
> > > > >> > > > > > > > > > > ej.wang.dev.interest@gmail.com
> > > > >> > > > > > > > > > > > >
> > > > >> > > > > > > > > > > > >> wrote:
> > > > >> > > > > > > > > > > > >>
> > > > >> > > > > > > > > > > > >> > Thanks Yufei for the +1.
> > > > >> > > > > > > > > > > > >> >
> > > > >> > > > > > > > > > > > >> > JB, could you help add a biweekly
> metrics
> > > > >> > > architecture
> > > > >> > > > > > sync
> > > > >> > > > > > > to
> > > > >> > > > > > > > > the
> > > > >> > > > > > > > > > > > >> Polaris
> > > > >> > > > > > > > > > > > >> > community calendar? I'm thinking
> Thursdays
> > > at
> > > > >> > 9-10am
> > > > >> > > > PT,
> > > > >> > > > > > on
> > > > >> > > > > > > > the
> > > > >> > > > > > > > > > > > >> off-weeks
> > > > >> > > > > > > > > > > > >> > from the community meeting (starting
> May
> > > 7),
> > > > 60
> > > > >> > > > minutes.
> > > > >> > > > > > > > > > > > >> >
> > > > >> > > > > > > > > > > > >> > Here's a rough agenda to work through
> over
> > > > the
> > > > >> > first
> > > > >> > > > few
> > > > >> > > > > > > > > sessions,
> > > > >> > > > > > > > > > > > >> grouped
> > > > >> > > > > > > > > > > > >> > by priority:
> > > > >> > > > > > > > > > > > >> >
> > > > >> > > > > > > > > > > > >> > *First: foundational direction*
> > > > >> > > > > > > > > > > > >> >
> > > > >> > > > > > > > > > > > >> > 1.  MetricsPersistence: public SPI or
> > > > internal
> > > > >> > > > > > > implementation
> > > > >> > > > > > > > > > > detail?
> > > > >> > > > > > > > > > > > >> >    •   Marked @Beta, javadoc calls it a
> > > > >> "Service
> > > > >> > > > > Provider
> > > > >> > > > > > > > > > > Interface",
> > > > >> > > > > > > > > > > > >> but
> > > > >> > > > > > > > > > > > >> > only one consumer
> > > (JdbcBasePersistenceImpl),
> > > > >> lives
> > > > >> > > on
> > > > >> > > > > > > > > > > BasePersistence.
> > > > >> > > > > > > > > > > > >> If
> > > > >> > > > > > > > > > > > >> > demoted to a private helper inside a
> > > > persisting
> > > > >> > > > reporter
> > > > >> > > > > > > impl,
> > > > >> > > > > > > > > > most
> > > > >> > > > > > > > > > > > >> > downstream design decisions become
> > > > >> implementation
> > > > >> > > > > details
> > > > >> > > > > > > > rather
> > > > >> > > > > > > > > > > than
> > > > >> > > > > > > > > > > > >> > contract questions.
> > > > >> > > > > > > > > > > > >> >
> > > > >> > > > > > > > > > > > >> > 2.  Persistence schema redesign
> > > > >> > > > > > > > > > > > >> >    •   Current two-table layout
> > > > >> > > (scan_metrics_report,
> > > > >> > > > > > > > > > > > >> > commit_metrics_report) with ~25
> flattened
> > > > >> columns
> > > > >> > > > each.
> > > > >> > > > > > > Every
> > > > >> > > > > > > > > new
> > > > >> > > > > > > > > > > > metric
> > > > >> > > > > > > > > > > > >> > type requires a new table, SPI method,
> > > record
> > > > >> > class,
> > > > >> > > > > > model,
> > > > >> > > > > > > > > > > converter,
> > > > >> > > > > > > > > > > > >> and
> > > > >> > > > > > > > > > > > >> > schema migration. Direction to explore:
> > > > single
> > > > >> > table
> > > > >> > > > > with
> > > > >> > > > > > > > > > > metric_type
> > > > >> > > > > > > > > > > > >> enum,
> > > > >> > > > > > > > > > > > >> > schema_version, and JSON payload
> column.
> > > > >> > > > > > > > > > > > >> >
> > > > >> > > > > > > > > > > > >> > *Second: design details once direction
> is
> > > > set*
> > > > >> > > > > > > > > > > > >> >
> > > > >> > > > > > > > > > > > >> > 3.  Partition key strategy
> > > > >> > > > > > > > > > > > >> >    •   Single-table design means scan
> > > metrics
> > > > >> at
> > > > >> > > scale
> > > > >> > > > > > will
> > > > >> > > > > > > > have
> > > > >> > > > > > > > > > > high
> > > > >> > > > > > > > > > > > >> > write concurrency per table. Schema
> needs
> > > to
> > > > >> > expose
> > > > >> > > > > enough
> > > > >> > > > > > > > > > structure
> > > > >> > > > > > > > > > > > for
> > > > >> > > > > > > > > > > > >> > backends to shard by entity or time
> range.
> > > > >> > > > > > > > > > > > >> >
> > > > >> > > > > > > > > > > > >> > 4.  Read/write path consistency
> > > > >> > > > > > > > > > > > >> >    •   Writes go through
> > > > PolarisMetricsManager
> > > > >> on
> > > > >> > > > > > > > > > MetaStoreManager.
> > > > >> > > > > > > > > > > > >> Reads
> > > > >> > > > > > > > > > > > >> > bypass MetaStoreManager and go
> straight to
> > > > >> > > > > > BasePersistence,
> > > > >> > > > > > > > > > > excluding
> > > > >> > > > > > > > > > > > >> > non-JDBC backends from the read API.
> > > > >> > > > > > > > > > > > >> >
> > > > >> > > > > > > > > > > > >> > *Third: cleanup and alignment*
> > > > >> > > > > > > > > > > > >> >
> > > > >> > > > > > > > > > > > >> > 5.  PolarisMetricsReporter naming
> > > > >> > > > > > > > > > > > >> >    •   Only handles IRC
> > > > >> (ScanReport/CommitReport),
> > > > >> > > > > doesn't
> > > > >> > > > > > > > cover
> > > > >> > > > > > > > > > > > generic
> > > > >> > > > > > > > > > > > >> > tables or operational metrics. Name is
> > > > broader
> > > > >> > than
> > > > >> > > > > scope.
> > > > >> > > > > > > > > > > > >> >
> > > > >> > > > > > > > > > > > >> > 6.  PolarisMetricsManager facade
> > > passthrough
> > > > >> > > > > > > > > > > > >> >    •   Entire default method is
> > > > >> > > > > > > > > > > > >> callCtx.getMetaStore().writeScanReport().
> > > > >> > > > > > > > > > > > >> > Zero logic, passes Level 1 straight
> through
> > > > to
> > > > >> > Level
> > > > >> > > > 3.
> > > > >> > > > > > Same
> > > > >> > > > > > > > > > > > >> anti-pattern
> > > > >> > > > > > > > > > > > >> > as PolarisEventManager.
> > > > >> > > > > > > > > > > > >> >
> > > > >> > > > > > > > > > > > >> > 7.  Iceberg community alignment
> > > > >> > > > > > > > > > > > >> >    •   Payload-type extension needs
> > > > discussion
> > > > >> on
> > > > >> > > > > > > dev@iceberg.
> > > > >> > > > > > > > > > > > >> obelix74's
> > > > >> > > > > > > > > > > > >> > Feb thread got zero replies. Needs a
> > > > committer
> > > > >> > > voice.
> > > > >> > > > > > > > > > > > >> >
> > > > >> > > > > > > > > > > > >> > Lets confirm prioritization in the
> first
> > > > >> session.
> > > > >> > > > > > > > > > > > >> >
> > > > >> > > > > > > > > > > > >> > -ej
> > > > >> > > > > > > > > > > > >> >
> > > > >> > > > > > > > > > > > >> > On Tue, Apr 21, 2026 at 3:18 PM Yufei
> Gu <
> > > > >> > > > > > > > flyrain000@gmail.com>
> > > > >> > > > > > > > > > > > wrote:
> > > > >> > > > > > > > > > > > >> >
> > > > >> > > > > > > > > > > > >> >> Thanks everyone for continuing to
> drive
> > > this
> > > > >> > > > forward. I
> > > > >> > > > > > > agree
> > > > >> > > > > > > > > > that
> > > > >> > > > > > > > > > > > the
> > > > >> > > > > > > > > > > > >> >> problem is getting complex enough
> that a
> > > > more
> > > > >> > > > > structured
> > > > >> > > > > > > > > > discussion
> > > > >> > > > > > > > > > > > >> would
> > > > >> > > > > > > > > > > > >> >> help.
> > > > >> > > > > > > > > > > > >> >>
> > > > >> > > > > > > > > > > > >> >> +1 on setting up a biweekly sync for
> the
> > > > >> metrics
> > > > >> > > > > > > > architecture.
> > > > >> > > > > > > > > > I’m
> > > > >> > > > > > > > > > > > >> happy
> > > > >> > > > > > > > > > > > >> >> to
> > > > >> > > > > > > > > > > > >> >> join.
> > > > >> > > > > > > > > > > > >> >>
> > > > >> > > > > > > > > > > > >> >> Yufei
> > > > >> > > > > > > > > > > > >> >>
> > > > >> > > > > > > > > > > > >> >>
> > > > >> > > > > > > > > > > > >> >> On Tue, Apr 21, 2026 at 2:34 PM EJ
> Wang <
> > > > >> > > > > > > > > > > > >> ej.wang.dev.interest@gmail.com>
> > > > >> > > > > > > > > > > > >> >> wrote:
> > > > >> > > > > > > > > > > > >> >>
> > > > >> > > > > > > > > > > > >> >> > Also, I've been looking more
> closely at
> > > > the
> > > > >> > > > > > *persistence
> > > > >> > > > > > > > > schema
> > > > >> > > > > > > > > > > in
> > > > >> > > > > > > > > > > > >> the
> > > > >> > > > > > > > > > > > >> >> > current metrics work*, and I think
> > > > there's a
> > > > >> > > > > structural
> > > > >> > > > > > > > > > rigidity
> > > > >> > > > > > > > > > > > >> problem
> > > > >> > > > > > > > > > > > >> >> > worth raising before the shape gets
> > > locked
> > > > >> in.
> > > > >> > > > > > > > > > > > >> >> >
> > > > >> > > > > > > > > > > > >> >> > Right now we have two separate
> tables
> > > > >> > > > > > > (scan_metrics_report
> > > > >> > > > > > > > > and
> > > > >> > > > > > > > > > > > >> >> > commit_metrics_report), each with
> ~25
> > > > >> flattened
> > > > >> > > > > columns
> > > > >> > > > > > > > that
> > > > >> > > > > > > > > > > > directly
> > > > >> > > > > > > > > > > > >> >> > mirror the Iceberg report fields.
> The
> > > SPI
> > > > >> > follows
> > > > >> > > > the
> > > > >> > > > > > > same
> > > > >> > > > > > > > > > split:
> > > > >> > > > > > > > > > > > >> >> > writeScanReport and
> writeCommitReport as
> > > > >> > separate
> > > > >> > > > > > > methods,
> > > > >> > > > > > > > > with
> > > > >> > > > > > > > > > > > >> per-type
> > > > >> > > > > > > > > > > > >> >> > record classes, converters, and
> model
> > > > >> objects.
> > > > >> > > *The
> > > > >> > > > > > > > practical
> > > > >> > > > > > > > > > > cost:
> > > > >> > > > > > > > > > > > >> >> > adding a new metric type
> (operational
> > > > >> metrics,
> > > > >> > > for
> > > > >> > > > > > > example)
> > > > >> > > > > > > > > > > > requires
> > > > >> > > > > > > > > > > > >> a
> > > > >> > > > > > > > > > > > >> >> new
> > > > >> > > > > > > > > > > > >> >> > table, a new SPI method, a new
> record
> > > > >> class, a
> > > > >> > > new
> > > > >> > > > > > model
> > > > >> > > > > > > > > > class, a
> > > > >> > > > > > > > > > > > new
> > > > >> > > > > > > > > > > > >> >> > converter branch, and a schema
> > > migration*.
> > > > >> > > That's a
> > > > >> > > > > lot
> > > > >> > > > > > > of
> > > > >> > > > > > > > > > > surface
> > > > >> > > > > > > > > > > > >> area
> > > > >> > > > > > > > > > > > >> >> > for what should be "one more kind of
> > > > >> metric."
> > > > >> > > > > > > > > > > > >> >> >
> > > > >> > > > > > > > > > > > >> >> > *My bias* would be toward a single
> > > metrics
> > > > >> > table
> > > > >> > > > with
> > > > >> > > > > > *a
> > > > >> > > > > > > > > typed
> > > > >> > > > > > > > > > > JSON
> > > > >> > > > > > > > > > > > >> >> > payload*. Something like:
> metric_type
> > > > >> (enum),
> > > > >> > > > > > entity_id,
> > > > >> > > > > > > > > > > > >> >> > table_identifier, snapshot_id
> > > (nullable),
> > > > >> > > > > received_ts,
> > > > >> > > > > > > > > > > > >> schema_version,
> > > > >> > > > > > > > > > > > >> >> and
> > > > >> > > > > > > > > > > > >> >> > a payload column for the
> metric-specific
> > > > >> data.
> > > > >> > > The
> > > > >> > > > > > > > > metric_type
> > > > >> > > > > > > > > > +
> > > > >> > > > > > > > > > > > >> >> > schema_version pair gives us a
> > > > >> > forward-compatible
> > > > >> > > > > > > contract
> > > > >> > > > > > > > > for
> > > > >> > > > > > > > > > > the
> > > > >> > > > > > > > > > > > >> >> payload
> > > > >> > > > > > > > > > > > >> >> > shape. Adding a new metric type
> becomes
> > > an
> > > > >> enum
> > > > >> > > > value
> > > > >> > > > > > > and a
> > > > >> > > > > > > > > > > payload
> > > > >> > > > > > > > > > > > >> >> schema,
> > > > >> > > > > > > > > > > > >> >> > not a schema migration. One thing I
> > > think
> > > > we
> > > > >> > need
> > > > >> > > > to
> > > > >> > > > > be
> > > > >> > > > > > > > > > > deliberate
> > > > >> > > > > > > > > > > > >> >> about is
> > > > >> > > > > > > > > > > > >> >> > the partition key design. If all
> metric
> > > > >> types
> > > > >> > > land
> > > > >> > > > in
> > > > >> > > > > > one
> > > > >> > > > > > > > > > table,
> > > > >> > > > > > > > > > > > scan
> > > > >> > > > > > > > > > > > >> >> > metrics at scale (high concurrency,
> high
> > > > >> > > frequency
> > > > >> > > > > > across
> > > > >> > > > > > > > > many
> > > > >> > > > > > > > > > > > >> tables)
> > > > >> > > > > > > > > > > > >> >> > could easily create hot partitions.
> We'd
> > > > >> want
> > > > >> > the
> > > > >> > > > > > > > persistence
> > > > >> > > > > > > > > > > layer
> > > > >> > > > > > > > > > > > >> to
> > > > >> > > > > > > > > > > > >> >> be
> > > > >> > > > > > > > > > > > >> >> > able to shard by entity or time
> range,
> > > and
> > > > >> that
> > > > >> > > > means
> > > > >> > > > > > the
> > > > >> > > > > > > > > > logical
> > > > >> > > > > > > > > > > > >> schema
> > > > >> > > > > > > > > > > > >> >> > needs to expose enough structure for
> > > > >> backends
> > > > >> > to
> > > > >> > > > > > > partition
> > > > >> > > > > > > > > on.
> > > > >> > > > > > > > > > I
> > > > >> > > > > > > > > > > > >> don't
> > > > >> > > > > > > > > > > > >> >> > think the current flattened layout
> gives
> > > > us
> > > > >> > that.
> > > > >> > > > > > > > > > > > >> >> >
> > > > >> > > > > > > > > > > > >> >> > This is getting complex enough that
> I
> > > > don't
> > > > >> > think
> > > > >> > > > > > ad-hoc
> > > > >> > > > > > > > > PR/ML
> > > > >> > > > > > > > > > > > >> threads
> > > > >> > > > > > > > > > > > >> >> > will converge well. *Would people be
> > > open
> > > > >> to a
> > > > >> > > > > biweekly
> > > > >> > > > > > > > sync
> > > > >> > > > > > > > > > for
> > > > >> > > > > > > > > > > > >> metrics
> > > > >> > > > > > > > > > > > >> >> > architecture?* I think 30 minutes
> every
> > > > two
> > > > >> > weeks
> > > > >> > > > > with
> > > > >> > > > > > > > > > interested
> > > > >> > > > > > > > > > > > >> >> parties
> > > > >> > > > > > > > > > > > >> >> > would be enough to work through the
> > > > schema,
> > > > >> SPI
> > > > >> > > > > shape,
> > > > >> > > > > > > and
> > > > >> > > > > > > > > read
> > > > >> > > > > > > > > > > API
> > > > >> > > > > > > > > > > > >> >> design
> > > > >> > > > > > > > > > > > >> >> > together. Happy to help set that up.
> > > > >> > > > > > > > > > > > >> >> >
> > > > >> > > > > > > > > > > > >> >> > -ej
> > > > >> > > > > > > > > > > > >> >> >
> > > > >> > > > > > > > > > > > >> >> > On Mon, Apr 20, 2026 at 2:19 PM EJ
> Wang
> > > <
> > > > >> > > > > > > > > > > > >> ej.wang.dev.interest@gmail.com
> > > > >> > > > > > > > > > > > >> >> >
> > > > >> > > > > > > > > > > > >> >> > wrote:
> > > > >> > > > > > > > > > > > >> >> >
> > > > >> > > > > > > > > > > > >> >> >> Reviewed #4115, left a comment on
> the
> > > > code
> > > > >> > > > > > organization
> > > > >> > > > > > > > > side.
> > > > >> > > > > > > > > > > > >> >> >>
> > > > >> > > > > > > > > > > > >> >> >> One thing stood out: the metrics
> write
> > > > path
> > > > >> > > enters
> > > > >> > > > > > > through
> > > > >> > > > > > > > > > > > >> >> >> PolarisMetricsManager on
> > > > MetaStoreManager,
> > > > >> but
> > > > >> > > the
> > > > >> > > > > new
> > > > >> > > > > > > > read
> > > > >> > > > > > > > > > path
> > > > >> > > > > > > > > > > > >> >> bypasses
> > > > >> > > > > > > > > > > > >> >> >> MetaStoreManager entirely and goes
> > > > >> straight to
> > > > >> > > > > > > > > BasePersistence
> > > > >> > > > > > > > > > > via
> > > > >> > > > > > > > > > > > >> >> >> callContext.getMetaStore(). That
> means
> > > > the
> > > > >> > read
> > > > >> > > > API
> > > > >> > > > > > only
> > > > >> > > > > > > > > works
> > > > >> > > > > > > > > > > for
> > > > >> > > > > > > > > > > > >> >> backends
> > > > >> > > > > > > > > > > > >> >> >> that implement BasePersistence.
> NoSQL
> > > and
> > > > >> > remote
> > > > >> > > > > > > backends
> > > > >> > > > > > > > > > can't
> > > > >> > > > > > > > > > > > >> >> participate.
> > > > >> > > > > > > > > > > > >> >> >>
> > > > >> > > > > > > > > > > > >> >> >> Stepping back, I think the metrics
> > > > >> subsystem
> > > > >> > is
> > > > >> > > > > > growing
> > > > >> > > > > > > > into
> > > > >> > > > > > > > > > > > >> something
> > > > >> > > > > > > > > > > > >> >> >> real (write + read + REST API +
> AuthZ +
> > > > >> > > > pagination)
> > > > >> > > > > > *but
> > > > >> > > > > > > > the
> > > > >> > > > > > > > > > > > >> >> persistence
> > > > >> > > > > > > > > > > > >> >> >> side is split across two layers in
> a
> > > way
> > > > >> > that's
> > > > >> > > > hard
> > > > >> > > > > > to
> > > > >> > > > > > > > > > > extend*. I
> > > > >> > > > > > > > > > > > >> put
> > > > >> > > > > > > > > > > > >> >> >> together two diagrams to show what
> I
> > > mean
> > > > >> (my
> > > > >> > > best
> > > > >> > > > > > > > effort).
> > > > >> > > > > > > > > > > > >> >> >>
> > > > >> > > > > > > > > > > > >> >> >> *Current state* (Diagram 1): three
> > > > >> interfaces
> > > > >> > at
> > > > >> > > > > three
> > > > >> > > > > > > > > > different
> > > > >> > > > > > > > > > > > >> >> levels.
> > > > >> > > > > > > > > > > > >> >> >> The engine-facing SPI
> > > > >> (PolarisMetricsReporter)
> > > > >> > > is
> > > > >> > > > > > clean.
> > > > >> > > > > > > > But
> > > > >> > > > > > > > > > > > >> >> >> PolarisMetricsManager on
> > > MetaStoreManager
> > > > >> is a
> > > > >> > > > > > > passthrough
> > > > >> > > > > > > > > to
> > > > >> > > > > > > > > > > > >> >> >> MetricsPersistence on
> BasePersistence.
> > > > The
> > > > >> > @Beta
> > > > >> > > > > > > > annotation
> > > > >> > > > > > > > > > and
> > > > >> > > > > > > > > > > > SPI
> > > > >> > > > > > > > > > > > >> >> javadoc
> > > > >> > > > > > > > > > > > >> >> >> are on the BasePersistence layer,
> while
> > > > the
> > > > >> > > actual
> > > > >> > > > > > > > extension
> > > > >> > > > > > > > > > > > points
> > > > >> > > > > > > > > > > > >> >> >> (PolarisMetricsReporter,
> > > > >> > PolarisMetricsManager)
> > > > >> > > > > carry
> > > > >> > > > > > no
> > > > >> > > > > > > > > > > stability
> > > > >> > > > > > > > > > > > >> >> >> annotation. The write path goes
> through
> > > > the
> > > > >> > > > > > > > MetaStoreManager
> > > > >> > > > > > > > > > > > layer,
> > > > >> > > > > > > > > > > > >> the
> > > > >> > > > > > > > > > > > >> >> >> read path doesn't.
> > > > >> > > > > > > > > > > > >> >> >>
> > > > >> > > > > > > > > > > > >> >> >> *What I envision* (Diagram 2): two
> SPIs
> > > > at
> > > > >> two
> > > > >> > > > > levels.
> > > > >> > > > > > > > > > > > >> >> >> PolarisMetricsReporter stays as the
> > > > >> > > engine-facing
> > > > >> > > > > SPI.
> > > > >> > > > > > > > > > > > >> >> >> PolarisMetricsManager becomes the
> > > > >> > backend-facing
> > > > >> > > > SPI
> > > > >> > > > > > > with
> > > > >> > > > > > > > > both
> > > > >> > > > > > > > > > > > write
> > > > >> > > > > > > > > > > > >> >> and
> > > > >> > > > > > > > > > > > >> >> >> read methods at the
> MetaStoreManager
> > > > level,
> > > > >> > > where
> > > > >> > > > > any
> > > > >> > > > > > > > > backend
> > > > >> > > > > > > > > > > > (JDBC,
> > > > >> > > > > > > > > > > > >> >> NoSQL,
> > > > >> > > > > > > > > > > > >> >> >> remote) can implement them.
> > > > >> MetricsPersistence
> > > > >> > > on
> > > > >> > > > > > > > > > > BasePersistence
> > > > >> > > > > > > > > > > > >> goes
> > > > >> > > > > > > > > > > > >> >> >> away. Where metrics actually land
> is an
> > > > >> > > > > implementation
> > > > >> > > > > > > > > detail,
> > > > >> > > > > > > > > > > > not a
> > > > >> > > > > > > > > > > > >> >> core
> > > > >> > > > > > > > > > > > >> >> >> interface.
> > > > >> > > > > > > > > > > > >> >> >>
> > > > >> > > > > > > > > > > > >> >> >> *Minor naming thing*:
> > > > >> PolarisMetricsReporter
> > > > >> > is
> > > > >> > > > > > broader
> > > > >> > > > > > > > than
> > > > >> > > > > > > > > > > what
> > > > >> > > > > > > > > > > > it
> > > > >> > > > > > > > > > > > >> >> >> actually handles. It only accepts
> > > Iceberg
> > > > >> REST
> > > > >> > > > > Catalog
> > > > >> > > > > > > > > metrics
> > > > >> > > > > > > > > > > > >> >> (ScanReport,
> > > > >> > > > > > > > > > > > >> >> >> CommitReport via MetricsReport).
> > > Generic
> > > > >> table
> > > > >> > > > > metrics
> > > > >> > > > > > > or
> > > > >> > > > > > > > > > > > >> operational
> > > > >> > > > > > > > > > > > >> >> >> metrics aren't in scope. Not
> blocking,
> > > > but
> > > > >> > worth
> > > > >> > > > > > noting
> > > > >> > > > > > > if
> > > > >> > > > > > > > > the
> > > > >> > > > > > > > > > > > >> metrics
> > > > >> > > > > > > > > > > > >> >> >> surface expands.
> > > > >> > > > > > > > > > > > >> >> >>
> > > > >> > > > > > > > > > > > >> >> >> *Rough sketch of how to get there*:
> > > > >> > > > > > > > > > > > >> >> >>  1.  Add read methods to
> > > > >> PolarisMetricsManager
> > > > >> > > > > > > > > > (listScanReports,
> > > > >> > > > > > > > > > > > >> >> >> listCommitReports) with default
> no-op,
> > > > >> same as
> > > > >> > > the
> > > > >> > > > > > > > existing
> > > > >> > > > > > > > > > > write
> > > > >> > > > > > > > > > > > >> >> methods.
> > > > >> > > > > > > > > > > > >> >> >> (Probably make
> PolarisMetricsManager
> > > more
> > > > >> > > explicit
> > > > >> > > > > on
> > > > >> > > > > > > > being
> > > > >> > > > > > > > > > > > Iceberg
> > > > >> > > > > > > > > > > > >> >> >> specific like package name or class
> > > name
> > > > >> etc.)
> > > > >> > > > > > > > > > > > >> >> >>  2.  Wire MetricsReportsService
> through
> > > > >> > > > > > MetaStoreManager
> > > > >> > > > > > > > > > instead
> > > > >> > > > > > > > > > > > of
> > > > >> > > > > > > > > > > > >> >> >> callContext.getMetaStore().
> > > > >> > > > > > > > > > > > >> >> >>  3.  Extract metrics persistence
> from
> > > > >> > > > > > > > > JdbcBasePersistenceImpl
> > > > >> > > > > > > > > > > into
> > > > >> > > > > > > > > > > > >> its
> > > > >> > > > > > > > > > > > >> >> >> own class. That file carries ~7
> > > > >> > > responsibilities,
> > > > >> > > > > > > metrics
> > > > >> > > > > > > > > > being
> > > > >> > > > > > > > > > > > one
> > > > >> > > > > > > > > > > > >> of
> > > > >> > > > > > > > > > > > >> >> them.
> > > > >> > > > > > > > > > > > >> >> >>  4.  Remove MetricsPersistence from
> > > > >> > > > BasePersistence.
> > > > >> > > > > > > > > > > > >> >> >>
> > > > >> > > > > > > > > > > > >> >> >> *None of this needs to happen in
> #4115.
> > > > >> But if
> > > > >> > > the
> > > > >> > > > > > > > direction
> > > > >> > > > > > > > > > > makes
> > > > >> > > > > > > > > > > > >> >> sense,
> > > > >> > > > > > > > > > > > >> >> >> it would be good to align before
> the
> > > > >> metrics
> > > > >> > > > surface
> > > > >> > > > > > > grows
> > > > >> > > > > > > > > > > > further.
> > > > >> > > > > > > > > > > > >> >> Curious
> > > > >> > > > > > > > > > > > >> >> >> what others think.*
> > > > >> > > > > > > > > > > > >> >> >>
> > > > >> > > > > > > > > > > > >> >> >> *My mental model note*: Level 1
> > > > >> > > MetaStoreManager;
> > > > >> > > > > > level
> > > > >> > > > > > > 2
> > > > >> > > > > > > > > > > > >> transactional
> > > > >> > > > > > > > > > > > >> >> >> persistence; level 3 base
> persistence
> > > > >> > > > > > > > > > > > >> >> >>
> > > > >> > > > > > > > > > > > >> >> >> Diagram 1
> > > > >> > > > > > > > > > > > >> >> >> <
> > > > >> > > > > > > > > > > > >> >>
> > > > >> > > > > > > > > > > > >>
> > > > >> > > > > > > > > > > >
> > > > >> > > > > > > > > > >
> > > > >> > > > > > > > > >
> > > > >> > > > > > > > >
> > > > >> > > > > > > >
> > > > >> > > > > > >
> > > > >> > > > > >
> > > > >> > > > >
> > > > >> > > >
> > > > >> > >
> > > > >> >
> > > > >>
> > > >
> > >
> https://urldefense.com/v3/__https://www.plantuml.com/plantuml/uml/bLHDR-Cs4BthLmpIYupw0zbkKQ1r3M-S7Bp8xhhM7WCOb3IM65EaGD9EX2RzxHrHb4CxRelwa4YSDu_lpOVcnZ9jzvM8BBS2uGjQpJC3dtHMSekPtMk44IpsMgEqa5XcCOhCZikQQLP1pR8TAp2n3ILhmZDP20m0fcIvUkAoW2qJXd9z1bpToO9BX3WXu0ucy5rpgGPNm0nW5_epUWtm2Ue3pn3kMOFQmKntGZW0BYtgBSi8k5A2QMwybJNMIbFiGSR9QZc4nUqIvikStF0jHprua5C-amge42aNt3R0f5JaaoivdV2Pkqbx4hee4ymOkBh5BTiB-_uIeGeo8zL8rPsPl4DktdEiK1jkB1NdZCRbrSTecDe_mlHbF0wvBmCkaOH5_S8a_TTTKI6-nmCAkEw4LpxsZ-LbYLKQFKMNOgf_wuM7_bV9gOer5SYMMksBSWXFcbi49KNZXNLicwfe3TETC7gPdPqI7uBcHMb1RSzYq34c6PDUM9mn8HRsUTZEiDBve3NjVZumBj0U7SS37mGO7vcwtiK-_pU7U7L_f-digo9YbhSwIfMRwIITKGXbxdIUTCGF1SeCJxloKsU-3k9ddRbX1eDq1q_fx1JbBGT0glVyXimDuP4TQ5qpCAmnGEj2s_6n5mtn1z-97-63itFQZLPO1Ev2tu_WF7Ju-VPc0Skg5bYXxBhkY1xpD7EM_7fyflSpIsqMgVth5xhVr4eQxWQ8enaSAJQSG16yFSDuJ798rrcXr_3n-lfdk7icQjEBmFujL7AodiP_Y4Z7-YxvtZNs4zMgpNTl6tF8sglyPsmqchrjvQ-m-aP94r-TwCA2Ka8upPJZwtvSpoYCXkYMZU2NXvRMBfq9P3i3Le4VAZUAlUZ_oPKsxPgY0Q_BSKLkyr9bhQhQrJjo_x3TPlIB0DPjnMfcIoYP0QaYw1a0fTKDr8fB6ntNuvmoL1ZGkXa69Njh43zf9GiGxHQrA_jDYWRSzF5--WmTVrN97_Sm8LbLUy_lGBmLanJjFkDlGkRqjA_4tm00__;!!Iz9xO38YGHZK!4rnlVIqBQQQ6xpGe1PMV58edOS_cGrKoqdfhw4qSp_JQCavvEjaxXk0OwGPnxyEcO60AzKQK_XDtp7H3DdkFXcrXuTBISHg$
> > > > >> > > > > > > > > > > > >> >> >
> > > > >> > > > > > > > > > > > >> >> >> :
> > > > >> > > > > > > > > > > > >> >> >>
> > > > >> > > > > > > > > > > > >> >> >> [image: image.png]
> > > > >> > > > > > > > > > > > >> >> >>
> > > > >> > > > > > > > > > > > >> >> >> Diagram 2
> > > > >> > > > > > > > > > > > >> >> >> <
> > > > >> > > > > > > > > > > > >> >>
> > > > >> > > > > > > > > > > > >>
> > > > >> > > > > > > > > > > >
> > > > >> > > > > > > > > > >
> > > > >> > > > > > > > > >
> > > > >> > > > > > > > >
> > > > >> > > > > > > >
> > > > >> > > > > > >
> > > > >> > > > > >
> > > > >> > > > >
> > > > >> > > >
> > > > >> > >
> > > > >> >
> > > > >>
> > > >
> > >
> https://urldefense.com/v3/__https://www.plantuml.com/plantuml/uml/VLLDR-8m4BtdLupO2sWBLVU8AaGB7AXAbssGzb896SS9RXqxjKqBwkv_tt7iV43fdaZYDpFlpRmnOsE9jhjSH9PRmM31hERKm8scMsuPjJlDe0yheZDc8RR4iYWoBrmMH9CS2a9VICPYUy1OZN0YCy5Q0BCbYNhdCeEK28En8G8wCvbnoQ0R8_05Bc6bkLIz3X03p1zzH7zR-9ZfDquPt9C3qoNCX2yV4G2NbkcKu5jdgGJHt0GbZwnG6i-UP3TUpk5gM6Ldqke350eZUqzoCft3U9xWHvxoa5-7K4nF1J46EbEMafsmdrCBbQ44gVggy18IZrn_ph5asd1ZiIKdQSgueZvjXrQFSFrdC3YN-nXmBacxbGiYyLVxLaBtdhqn0LSzdBDhqQtQoOJeGyad3z0lUqnYgpGB6Ns8oVyta00Dy_WnX0tIOZ8v6SYxHll1TrH6aejAik-mh-AphVFCwSUQqFypElag5QRGFDjQKEd96K1P8QP41c9TzA_IIQyvdAWyv_RSiS3skb0_EzDDkK2v5xWF6MiGFlvhpFLcD2Dq2pml14gaF67eQkmd8gulDoC4kSOu6KVpkvlUJg1RTbWISU40RdBUUS_9XfRZ2dwxm_SW8LYFISgm_MnlDQ6M9P1gbKEc4X-2pH_FvJCkCqm9pbVjD6LrwdLeOrDWfOaqc8Wh9BE85oNKxkNQ6o4yGRy_Eae0G_G8tZv81d3bHDB23WOdisohVr3nh_j6lbSjbNaLRTc8UgtPbAU1J_tygOfZX9DWEJeHDvYx-qmSi5FgNLPZwHrHcUsncGQ5-skhUclpE5fo4ounpFauYrUbkU6ccfnxMvitwag4IyerhTxj8In_Oj1bDO4pQru674loYrGlULHLEGCjwJJ8gDoVZR8MxO4BT3IzRvIcAQKezC6xpziGnTyImrfEGyJI_OcKfgtxIvnTqFEMS17L9Z-jsARN5FmTheP7HtSdtOMT0B4GY2FYHXxgQmMtj2bRqiLFGapiVe1_QVKDrkqXcm83aFEXnMYCZ-xlyHy__;!!Iz9xO38YGHZK!4rnlVIqBQQQ6xpGe1PMV58edOS_cGrKoqdfhw4qSp_JQCavvEjaxXk0OwGPnxyEcO60AzKQK_XDtp7H3DdkFXcrXSpmqcNk$
> > > > >> > > > > > > > > > > > >> >> >
> > > > >> > > > > > > > > > > > >> >> >> :
> > > > >> > > > > > > > > > > > >> >> >> [image: image.png]
> > > > >> > > > > > > > > > > > >> >> >>
> > > > >> > > > > > > > > > > > >> >> >>  -ej
> > > > >> > > > > > > > > > > > >> >> >>
> > > > >> > > > > > > > > > > > >> >> >> On Wed, Apr 15, 2026 at 8:22 AM
> Dmitri
> > > > >> > > > Bourlatchkov
> > > > >> > > > > <
> > > > >> > > > > > > > > > > > >> dimas@apache.org>
> > > > >> > > > > > > > > > > > >> >> >> wrote:
> > > > >> > > > > > > > > > > > >> >> >>
> > > > >> > > > > > > > > > > > >> >> >>> Hi All,
> > > > >> > > > > > > > > > > > >> >> >>>
> > > > >> > > > > > > > > > > > >> >> >>> Heads up: The current state of PR
> > > [4115]
> > > > >> > looks
> > > > >> > > > > pretty
> > > > >> > > > > > > > solid
> > > > >> > > > > > > > > > to
> > > > >> > > > > > > > > > > > me.
> > > > >> > > > > > > > > > > > >> I
> > > > >> > > > > > > > > > > > >> >> >>> believe this PR is approaching a
> > > > mergeable
> > > > >> > > > > condition.
> > > > >> > > > > > > > > > > > >> >> >>>
> > > > >> > > > > > > > > > > > >> >> >>> Please post your reviews if you
> have
> > > any
> > > > >> > > > comments.
> > > > >> > > > > > > > > > > > >> >> >>>
> > > > >> > > > > > > > > > > > >> >> >>> [4115]
> > > > >> > > > https://urldefense.com/v3/__https://github.com/apache/polaris/pull/4115__;!!Iz9xO38YGHZK!4rnlVIqBQQQ6xpGe1PMV58edOS_cGrKoqdfhw4qSp_JQCavvEjaxXk0OwGPnxyEcO60AzKQK_XDtp7H3DdkFXcrXtFRscKs$
> > > > >> > > > > > > > > > > > >> >> >>>
> > > > >> > > > > > > > > > > > >> >> >>> Thanks,
> > > > >> > > > > > > > > > > > >> >> >>> Dmitri.
> > > > >> > > > > > > > > > > > >> >> >>>
> > > > >> > > > > > > > > > > > >> >> >>> On Tue, Mar 3, 2026 at 3:29 PM
> Anand
> > > > Kumar
> > > > >> > > > Sankaran
> > > > >> > > > > > via
> > > > >> > > > > > > > > dev <
> > > > >> > > > > > > > > > > > >> >> >>> dev@polaris.apache.org> wrote:
> > > > >> > > > > > > > > > > > >> >> >>>
> > > > >> > > > > > > > > > > > >> >> >>> > Hi Yufei and Dmitri,
> > > > >> > > > > > > > > > > > >> >> >>> >
> > > > >> > > > > > > > > > > > >> >> >>> > Here is a proposal for the REST
> > > > >> endpoints
> > > > >> > for
> > > > >> > > > > > metrics
> > > > >> > > > > > > > and
> > > > >> > > > > > > > > > > > events.
> > > > >> > > > > > > > > > > > >> >> >>> >
> > > > >> > > > > > > > > > > > >> >> >>> >
> > > > >> > > > > > https://urldefense.com/v3/__https://github.com/apache/polaris/pull/3924/changes__;!!Iz9xO38YGHZK!4rnlVIqBQQQ6xpGe1PMV58edOS_cGrKoqdfhw4qSp_JQCavvEjaxXk0OwGPnxyEcO60AzKQK_XDtp7H3DdkFXcrXcjoPhSs$
> > > > >> > > > > > > > > > > > >> >> >>> >
> > > > >> > > > > > > > > > > > >> >> >>> > I did not see any precursors for
> > > > >> raising a
> > > > >> > PR
> > > > >> > > > for
> > > > >> > > > > > > > > > proposals,
> > > > >> > > > > > > > > > > so
> > > > >> > > > > > > > > > > > >> >> trying
> > > > >> > > > > > > > > > > > >> >> >>> > this.  Please let me know what
> you
> > > > >> think.
> > > > >> > > > > > > > > > > > >> >> >>> >
> > > > >> > > > > > > > > > > > >> >> >>> > -
> > > > >> > > > > > > > > > > > >> >> >>> > Anand
> > > > >> > > > > > > > > > > > >> >> >>> >
> > > > >> > > > > > > > > > > > >> >> >>> > From: Anand Kumar Sankaran <
> > > > >> > > > > > > anand.sankaran@workday.com
> > > > >> > > > > > > > >
> > > > >> > > > > > > > > > > > >> >> >>> > Date: Monday, March 2, 2026 at
> > > > 10:25 AM
> > > > >> > > > > > > > > > > > >> >> >>> > To: dev@polaris.apache.org <
> > > > >> > > > > dev@polaris.apache.org
> > > > >> > > > > > >
> > > > >> > > > > > > > > > > > >> >> >>> > Subject: Re: Polaris Telemetry
> and
> > > > Audit
> > > > >> > > Trail
> > > > >> > > > > > > > > > > > >> >> >>> >
> > > > >> > > > > > > > > > > > >> >> >>> > About the REST API, based on my
> use
> > > > >> cases:
> > > > >> > > > > > > > > > > > >> >> >>> >
> > > > >> > > > > > > > > > > > >> >> >>> >
> > > > >> > > > > > > > > > > > >> >> >>> >   1.
> > > > >> > > > > > > > > > > > >> >> >>> > I want to be able to query
> commit
> > > > >> metrics
> > > > >> > to
> > > > >> > > > > track
> > > > >> > > > > > > > files
> > > > >> > > > > > > > > > > added
> > > > >> > > > > > > > > > > > /
> > > > >> > > > > > > > > > > > >> >> >>> removed
> > > > >> > > > > > > > > > > > >> >> >>> > per commit, along with record
> > > counts.
> > > > >> The
> > > > >> > > > > ingestion
> > > > >> > > > > > > > > > pipeline
> > > > >> > > > > > > > > > > > that
> > > > >> > > > > > > > > > > > >> >> >>> writes
> > > > >> > > > > > > > > > > > >> >> >>> > this data is owned by us and we
> are
> > > > >> > > guaranteed
> > > > >> > > > to
> > > > >> > > > > > > write
> > > > >> > > > > > > > > > this
> > > > >> > > > > > > > > > > > >> >> >>> information
> > > > >> > > > > > > > > > > > >> >> >>> > for each write.
> > > > >> > > > > > > > > > > > >> >> >>> >   2.
> > > > >> > > > > > > > > > > > >> >> >>> > I want to be able to query scan
> > > > metrics
> > > > >> for
> > > > >> > > > > read. I
> > > > >> > > > > > > > > > > understand
> > > > >> > > > > > > > > > > > >> >> clients
> > > > >> > > > > > > > > > > > >> >> >>> do
> > > > >> > > > > > > > > > > > >> >> >>> > not fulfill this requirement.
> > > > >> > > > > > > > > > > > >> >> >>> >   3.
> > > > >> > > > > > > > > > > > >> >> >>> > I want to be able to query the
> > > events
> > > > >> table
> > > > >> > > > > (events
> > > > >> > > > > > > are
> > > > >> > > > > > > > > > > > >> persisted) -
> > > > >> > > > > > > > > > > > >> >> >>> this
> > > > >> > > > > > > > > > > > >> >> >>> > may supersede #2, I am not sure
> yet.
> > > > >> > > > > > > > > > > > >> >> >>> >
> > > > >> > > > > > > > > > > > >> >> >>> > All this information is in the
> JDBC
> > > > >> based
> > > > >> > > > > > persistence
> > > > >> > > > > > > > > model
> > > > >> > > > > > > > > > > and
> > > > >> > > > > > > > > > > > >> is
> > > > >> > > > > > > > > > > > >> >> >>> > persisted in the metastore. I
> > > > currently
> > > > >> > don’t
> > > > >> > > > > have
> > > > >> > > > > > a
> > > > >> > > > > > > > need
> > > > >> > > > > > > > > > to
> > > > >> > > > > > > > > > > > >> query
> > > > >> > > > > > > > > > > > >> >> >>> > prometheus or open telemetry. I
> do
> > > > >> publish
> > > > >> > > some
> > > > >> > > > > > > events
> > > > >> > > > > > > > to
> > > > >> > > > > > > > > > > > >> Prometheus
> > > > >> > > > > > > > > > > > >> >> >>> and
> > > > >> > > > > > > > > > > > >> >> >>> > they are forwarded to our
> dashboards
> > > > >> > > elsewhere.
> > > > >> > > > > > > > > > > > >> >> >>> >
> > > > >> > > > > > > > > > > > >> >> >>> > About the CLI utilities, I
> meant the
> > > > >> admin
> > > > >> > > user
> > > > >> > > > > > > > > utilities.
> > > > >> > > > > > > > > > In
> > > > >> > > > > > > > > > > > >> one of
> > > > >> > > > > > > > > > > > >> >> >>> the
> > > > >> > > > > > > > > > > > >> >> >>> > earliest drafts of my proposal,
> > > > Prashant
> > > > >> > > > > mentioned
> > > > >> > > > > > > that
> > > > >> > > > > > > > > the
> > > > >> > > > > > > > > > > > >> metrics
> > > > >> > > > > > > > > > > > >> >> >>> tables
> > > > >> > > > > > > > > > > > >> >> >>> > can grow indefinitely and that a
> > > > similar
> > > > >> > > > problem
> > > > >> > > > > > > exists
> > > > >> > > > > > > > > > with
> > > > >> > > > > > > > > > > > the
> > > > >> > > > > > > > > > > > >> >> events
> > > > >> > > > > > > > > > > > >> >> >>> > table as well. We discussed that
> > > > >> cleaning
> > > > >> > up
> > > > >> > > of
> > > > >> > > > > old
> > > > >> > > > > > > > > records
> > > > >> > > > > > > > > > > > from
> > > > >> > > > > > > > > > > > >> >> both
> > > > >> > > > > > > > > > > > >> >> >>> > metrics tables and events
> tables can
> > > > be
> > > > >> > done
> > > > >> > > > via
> > > > >> > > > > a
> > > > >> > > > > > > CLI
> > > > >> > > > > > > > > > > utility.
> > > > >> > > > > > > > > > > > >> >> >>> >
> > > > >> > > > > > > > > > > > >> >> >>> > I see that Yufei has covered the
> > > > >> discussion
> > > > >> > > > about
> > > > >> > > > > > > > > > > datasources.
> > > > >> > > > > > > > > > > > >> >> >>> >
> > > > >> > > > > > > > > > > > >> >> >>> > -
> > > > >> > > > > > > > > > > > >> >> >>> > Anand
> > > > >> > > > > > > > > > > > >> >> >>> >
> > > > >> > > > > > > > > > > > >> >> >>> >
> > > > >> > > > > > > > > > > > >> >> >>> >
> > > > >> > > > > > > > > > > > >> >> >>> > From: Yufei Gu <
> > > flyrain000@gmail.com>
> > > > >> > > > > > > > > > > > >> >> >>> > Date: Friday, February 27, 2026
> at
> > > > >> 9:54 PM
> > > > >> > > > > > > > > > > > >> >> >>> > To: dev@polaris.apache.org <
> > > > >> > > > > dev@polaris.apache.org
> > > > >> > > > > > >
> > > > >> > > > > > > > > > > > >> >> >>> > Subject: Re: Polaris Telemetry
> and
> > > > Audit
> > > > >> > > Trail
> > > > >> > > > > > > > > > > > >> >> >>> >
> > > > >> > > > > > > > > > > > >> >> >>> > This Message Is From an External
> > > > Sender
> > > > >> > > > > > > > > > > > >> >> >>> > This message came from outside
> your
> > > > >> > > > organization.
> > > > >> > > > > > > > > > > > >> >> >>> > Report Suspicious<
> > > > >> > > > > > > > > > > > >> >> >>> >
> > > > >> > > > > > > > > > > > >> >> >>>
> > > > >> > > > > > > > > > > > >> >>
> > > > >> > > > > > > > > > > > >>
> > > > >> > > > > > > > > > > >
> > > > >> > > > > > > > > > >
> > > > >> > > > > > > > > >
> > > > >> > > > > > > > >
> > > > >> > > > > > > >
> > > > >> > > > > > >
> > > > >> > > > > >
> > > > >> > > > >
> > > > >> > > >
> > > > >> > >
> > > > >> >
> > > > >>
> > > >
> > >
> https://us-phishalarm-ewt.proofpoint.com/EWT/v1/Iz9xO38YGHZK!YhNDZABkHi1B699ote2uMwpOZw8i0QMCGO2Szc-HshuABGhGvwPJcymE6G2oUUxtS8xDkSrtGTPm_I3QnVDHoLMk50m9v8z_nZKTkd-bnVUbreF1u0WnfV_X5eYevZl_$
> > > > >> > > > > > > > > > > > >> >> >>> > >
> > > > >> > > > > > > > > > > > >> >> >>> >
> > > > >> > > > > > > > > > > > >> >> >>> >
> > > > >> > > > > > > > > > > > >> >> >>> > As I mentioned in
> > > > >> > > > > > > > > > > > >> >> >>> >
> > > > >> > > > > > > > > > > > >> >> >>>
> > > > >> > > > > > > > > > > > >> >>
> > > > >> > > > > > > > > > > > >>
> > > > >> > > > > > > > > > > >
> > > > >> > > > > > > > > > >
> > > > >> > > > > > > > > >
> > > > >> > > > > > > > >
> > > > >> > > > > > > >
> > > > >> > > > > > >
> > > > >> > > > > >
> > > > >> > > > >
> > > > >> > > >
> > > > >> > >
> > > > >> >
> > > > >>
> > > >
> > >
> https://urldefense.com/v3/__https://github.com/apache/polaris/issues/3890__;!!Iz9xO38YGHZK!5EuyFFkk3vhRWVIRvQAWBSQfpJkTMA9HxugzDwXmN0LPPqhEFxYkFRGVhtb8AqUwXtDh2OplcMnbMDHKOxrvDU0$
>> > > >> > > > > > > > > > > > >> >> >>> ,
> > > > >> > > > > > > > > > > > >> >> >>> > supporting
> > > > >> > > > > > > > > > > > >> >> >>> > multiple data sources is not a
> > > trivial
> > > > >> > > change.
> > > > >> > > > I
> > > > >> > > > > > > would
> > > > >> > > > > > > > > > > strongly
> > > > >> > > > > > > > > > > > >> >> >>> recommend
> > > > >> > > > > > > > > > > > >> >> >>> > starting with a design document
> to
> > > > >> > carefully
> > > > >> > > > > > evaluate
> > > > >> > > > > > > > the
> > > > >> > > > > > > > > > > > >> >> architectural
> > > > >> > > > > > > > > > > > >> >> >>> > implications and long term
> impact.
> > > > >> > > > > > > > > > > > >> >> >>> >
> > > > >> > > > > > > > > > > > >> >> >>> > A REST endpoint to query metrics
> > > seems
> > > > >> > > > reasonable
> > > > >> > > > > > > given
> > > > >> > > > > > > > > the
> > > > >> > > > > > > > > > > > >> current
> > > > >> > > > > > > > > > > > >> >> >>> JDBC
> > > > >> > > > > > > > > > > > >> >> >>> > based persistence model. That
> said,
> > > we
> > > > >> may
> > > > >> > > also
> > > > >> > > > > > > > consider
> > > > >> > > > > > > > > > > > >> alternative
> > > > >> > > > > > > > > > > > >> >> >>> > storage models. For example, if
> we
> > > > later
> > > > >> > > adopt
> > > > >> > > > a
> > > > >> > > > > > time
> > > > >> > > > > > > > > > series
> > > > >> > > > > > > > > > > > >> system
> > > > >> > > > > > > > > > > > >> >> >>> such as
> > > > >> > > > > > > > > > > > >> >> >>> > Prometheus to store metrics, the
> > > query
> > > > >> > model
> > > > >> > > > and
> > > > >> > > > > > > access
> > > > >> > > > > > > > > > > > patterns
> > > > >> > > > > > > > > > > > >> >> would
> > > > >> > > > > > > > > > > > >> >> >>> be
> > > > >> > > > > > > > > > > > >> >> >>> > fundamentally different.
> Designing
> > > the
> > > > >> REST
> > > > >> > > API
> > > > >> > > > > > > without
> > > > >> > > > > > > > > > > > >> considering
> > > > >> > > > > > > > > > > > >> >> >>> these
> > > > >> > > > > > > > > > > > >> >> >>> > potential evolutions may limit
> > > > >> flexibility.
> > > > >> > > I'd
> > > > >> > > > > > > suggest
> > > > >> > > > > > > > > to
> > > > >> > > > > > > > > > > > start
> > > > >> > > > > > > > > > > > >> >> with
> > > > >> > > > > > > > > > > > >> >> >>> the
> > > > >> > > > > > > > > > > > >> >> >>> > use case.
> > > > >> > > > > > > > > > > > >> >> >>> >
> > > > >> > > > > > > > > > > > >> >> >>> > Yufei
> > > > >> > > > > > > > > > > > >> >> >>> >
> > > > >> > > > > > > > > > > > >> >> >>> >
> > > > >> > > > > > > > > > > > >> >> >>> > On Fri, Feb 27, 2026 at 3:42 PM
> > > Dmitri
> > > > >> > > > > > Bourlatchkov <
> > > > >> > > > > > > > > > > > >> >> dimas@apache.org>
> > > > >> > > > > > > > > > > > >> >> >>> > wrote:
> > > > >> > > > > > > > > > > > >> >> >>> >
> > > > >> > > > > > > > > > > > >> >> >>> > > Hi Anand,
> > > > >> > > > > > > > > > > > >> >> >>> > >
> > > > >> > > > > > > > > > > > >> >> >>> > > Sharing my view... subject to
> > > > >> discussion:
> > > > >> > > > > > > > > > > > >> >> >>> > >
> > > > >> > > > > > > > > > > > >> >> >>> > > 1. Adding non-IRC REST API to
> > > > Polaris
> > > > >> is
> > > > >> > > > > > perfectly
> > > > >> > > > > > > > > fine.
> > > > >> > > > > > > > > > > > >> >> >>> > >
> > > > >> > > > > > > > > > > > >> >> >>> > > Figuring out specific endpoint
> > > URIs
> > > > >> and
> > > > >> > > > > payloads
> > > > >> > > > > > > > might
> > > > >> > > > > > > > > > > > require
> > > > >> > > > > > > > > > > > >> a
> > > > >> > > > > > > > > > > > >> >> few
> > > > >> > > > > > > > > > > > >> >> >>> > > roundtrips, so opening a
> separate
> > > > >> thread
> > > > >> > > for
> > > > >> > > > > that
> > > > >> > > > > > > > might
> > > > >> > > > > > > > > > be
> > > > >> > > > > > > > > > > > >> best.
> > > > >> > > > > > > > > > > > >> >> >>> > > Contributors commonly create
> > > Google
> > > > >> Docs
> > > > >> > > for
> > > > >> > > > > new
> > > > >> > > > > > > API
> > > > >> > > > > > > > > > > > proposals
> > > > >> > > > > > > > > > > > >> too
> > > > >> > > > > > > > > > > > >> >> >>> (they
> > > > >> > > > > > > > > > > > >> >> >>> > > fairly easy to update as the
> email
> > > > >> > > discussion
> > > > >> > > > > > > > > > progresses).
> > > > >> > > > > > > > > > > > >> >> >>> > >
> > > > >> > > > > > > > > > > > >> >> >>> > > There was a suggestion to try
> > > > Markdown
> > > > >> > > (with
> > > > >> > > > > PRs)
> > > > >> > > > > > > for
> > > > >> > > > > > > > > > > > proposals
> > > > >> > > > > > > > > > > > >> >> [1]
> > > > >> > > > > > > > > > > > >> >> >>> ...
> > > > >> > > > > > > > > > > > >> >> >>> > > feel free to give it a try if
> you
> > > > are
> > > > >> > > > > comfortable
> > > > >> > > > > > > > with
> > > > >> > > > > > > > > > > that.
> > > > >> > > > > > > > > > > > >> >> >>> > >
> > > > >> > > > > > > > > > > > >> >> >>> > > 2. Could you clarify whether
> you
> > > > mean
> > > > >> end
> > > > >> > > > user
> > > > >> > > > > > > > > utilities
> > > > >> > > > > > > > > > or
> > > > >> > > > > > > > > > > > >> admin
> > > > >> > > > > > > > > > > > >> >> >>> user
> > > > >> > > > > > > > > > > > >> >> >>> > > utilities? In the latter case
> > > those
> > > > >> might
> > > > >> > > be
> > > > >> > > > > more
> > > > >> > > > > > > > > > suitable
> > > > >> > > > > > > > > > > > for
> > > > >> > > > > > > > > > > > >> the
> > > > >> > > > > > > > > > > > >> >> >>> Admin
> > > > >> > > > > > > > > > > > >> >> >>> > > CLI (java) not the Python CLI,
> > > IMHO.
> > > > >> > > > > > > > > > > > >> >> >>> > >
> > > > >> > > > > > > > > > > > >> >> >>> > > Why would these utilities be
> > > common
> > > > >> with
> > > > >> > > > > events?
> > > > >> > > > > > > > IMHO,
> > > > >> > > > > > > > > > > event
> > > > >> > > > > > > > > > > > >> use
> > > > >> > > > > > > > > > > > >> >> >>> cases
> > > > >> > > > > > > > > > > > >> >> >>> > are
> > > > >> > > > > > > > > > > > >> >> >>> > > distinct from scan/commit
> metrics.
> > > > >> > > > > > > > > > > > >> >> >>> > >
> > > > >> > > > > > > > > > > > >> >> >>> > > 3. I'd prefer separating
> metrics
> > > > >> > > persistence
> > > > >> > > > > from
> > > > >> > > > > > > > > > MetaStore
> > > > >> > > > > > > > > > > > >> >> >>> persistence
> > > > >> > > > > > > > > > > > >> >> >>> > at
> > > > >> > > > > > > > > > > > >> >> >>> > > the code level, so that they
> could
> > > > be
> > > > >> > mixed
> > > > >> > > > and
> > > > >> > > > > > > > matched
> > > > >> > > > > > > > > > > > >> >> >>> independently.
> > > > >> > > > > > > > > > > > >> >> >>> > The
> > > > >> > > > > > > > > > > > >> >> >>> > > separate datasource question
> will
> > > > >> become
> > > > >> > a
> > > > >> > > > > > > non-issue
> > > > >> > > > > > > > > with
> > > > >> > > > > > > > > > > > that
> > > > >> > > > > > > > > > > > >> >> >>> approach,
> > > > >> > > > > > > > > > > > >> >> >>> > I
> > > > >> > > > > > > > > > > > >> >> >>> > > guess.
> > > > >> > > > > > > > > > > > >> >> >>> > >
> > > > >> > > > > > > > > > > > >> >> >>> > > The rationale for separating
> scan
> > > > >> metrics
> > > > >> > > and
> > > > >> > > > > > > > metastore
> > > > >> > > > > > > > > > > > >> >> persistence
> > > > >> > > > > > > > > > > > >> >> >>> is
> > > > >> > > > > > > > > > > > >> >> >>> > that
> > > > >> > > > > > > > > > > > >> >> >>> > > "cascading deletes" between
> them
> > > are
> > > > >> > hardly
> > > > >> > > > > ever
> > > > >> > > > > > > > > > required.
> > > > >> > > > > > > > > > > > >> >> >>> Furthermore,
> > > > >> > > > > > > > > > > > >> >> >>> > the
> > > > >> > > > > > > > > > > > >> >> >>> > > data and query patterns are
> very
> > > > >> > different
> > > > >> > > so
> > > > >> > > > > > > > different
> > > > >> > > > > > > > > > > > >> >> technologies
> > > > >> > > > > > > > > > > > >> >> >>> > might
> > > > >> > > > > > > > > > > > >> >> >>> > > be beneficial in each case.
> > > > >> > > > > > > > > > > > >> >> >>> > >
> > > > >> > > > > > > > > > > > >> >> >>> > > [1]
> > > > >> > > > > > > > > > > > >> >> >>> >
> > > > >> > > > > > > > > > > > >> >> >>>
> > > > >> > > > > > > > > > > > >> >>
> > > > >> > > > > > > > > > > > >>
> > > > >> > > > > > > > > > > >
> > > > >> > > > > > > > > > >
> > > > >> > > > > > > > > >
> > > > >> > > > > > > > >
> > > > >> > > > > > > >
> > > > >> > > > > > >
> > > > >> > > > > >
> > > > >> > > > >
> > > > >> > > >
> > > > >> > >
> > > > >> >
> > > > >>
> > > >
> > >
> https://urldefense.com/v3/__https://lists.apache.org/thread/yto2wp982t43h1mqjwnslswhws5z47cy__;!!Iz9xO38YGHZK!5EuyFFkk3vhRWVIRvQAWBSQfpJkTMA9HxugzDwXmN0LPPqhEFxYkFRGVhtb8AqUwXtDh2OplcMnbMDHKxYDakNU$
>> > > >> > > > > > > > > > > > >> >> >>> > >
> > > > >> > > > > > > > > > > > >> >> >>> > > Cheers,
> > > > >> > > > > > > > > > > > >> >> >>> > > Dmitri.
> > > > >> > > > > > > > > > > > >> >> >>> > >
> > > > >> > > > > > > > > > > > >> >> >>> > > On Fri, Feb 27, 2026 at
> 6:19 PM
> > > > Anand
> > > > >> > Kumar
> > > > >> > > > > > > Sankaran
> > > > >> > > > > > > > > via
> > > > >> > > > > > > > > > > dev
> > > > >> > > > > > > > > > > > <
> > > > >> > > > > > > > > > > > >> >> >>> > > dev@polaris.apache.org>
> wrote:
> > > > >> > > > > > > > > > > > >> >> >>> > >
> > > > >> > > > > > > > > > > > >> >> >>> > > > Thanks all. This PR is
> merged
> > > now.
> > > > >> > > > > > > > > > > > >> >> >>> > > >
> > > > >> > > > > > > > > > > > >> >> >>> > > > Here are the follow-up
> features
> > > /
> > > > >> work
> > > > >> > > > > needed.
> > > > >> > > > > > > > These
> > > > >> > > > > > > > > > > were
> > > > >> > > > > > > > > > > > >> all
> > > > >> > > > > > > > > > > > >> >> >>> part of
> > > > >> > > > > > > > > > > > >> >> >>> > > the
> > > > >> > > > > > > > > > > > >> >> >>> > > > merged PR at some point in
> time
> > > > and
> > > > >> > were
> > > > >> > > > > > removed
> > > > >> > > > > > > to
> > > > >> > > > > > > > > > > reduce
> > > > >> > > > > > > > > > > > >> >> scope.
> > > > >> > > > > > > > > > > > >> >> >>> > > >
> > > > >> > > > > > > > > > > > >> >> >>> > > > Please let me know what you
> > > think.
> > > > >> > > > > > > > > > > > >> >> >>> > > >
> > > > >> > > > > > > > > > > > >> >> >>> > > >
> > > > >> > > > > > > > > > > > >> >> >>> > > >   1.  A REST API to paginate
> > > > through
> > > > >> > > table
> > > > >> > > > > > > metrics.
> > > > >> > > > > > > > > > This
> > > > >> > > > > > > > > > > > >> will be
> > > > >> > > > > > > > > > > > >> >> >>> > non-IRC
> > > > >> > > > > > > > > > > > >> >> >>> > > > standard addition.
> > > > >> > > > > > > > > > > > >> >> >>> > > >   2.  Utilities for
> managing old
> > > > >> > records,
> > > > >> > > > > > should
> > > > >> > > > > > > be
> > > > >> > > > > > > > > > > common
> > > > >> > > > > > > > > > > > >> with
> > > > >> > > > > > > > > > > > >> >> >>> events.
> > > > >> > > > > > > > > > > > >> >> >>> > > > There was some discussion
> that
> > > it
> > > > >> > belongs
> > > > >> > > > to
> > > > >> > > > > > the
> > > > >> > > > > > > > CLI.
> > > > >> > > > > > > > > > > > >> >> >>> > > >   3.  Separate datasource
> > > > (metrics,
> > > > >> > > events,
> > > > >> > > > > > even
> > > > >> > > > > > > > > other
> > > > >> > > > > > > > > > > > >> tables?).
> > > > >> > > > > > > > > > > > >> >> >>> > > >
> > > > >> > > > > > > > > > > > >> >> >>> > > >
> > > > >> > > > > > > > > > > > >> >> >>> > > > Anything else?
> > > > >> > > > > > > > > > > > >> >> >>> > > >
> > > > >> > > > > > > > > > > > >> >> >>> > > > -
> > > > >> > > > > > > > > > > > >> >> >>> > > > Anand
> > > > >> > > > > > > > > > > > >> >> >>> > > >
> > > > >> > > > > > > > > > > > >> >> >>> > > >
> > > > >> > > > > > > > > > > > >> >> >>> > >
> > > > >> > > > > > > > > > > > >> >> >>> >
> > > > >> > > > > > > > > > > > >> >> >>> >
> > > > >> > > > > > > > > > > > >> >> >>>
> > > > >> > > > > > > > > > > > >> >> >>
> > > > >> > > > > > > > > > > > >> >>
> > > > >> > > > > > > > > > > > >> >
> > > > >> > > > > > > > > > > > >>
> > > > >> > > > > > > > > > > > >
> > > > >> > > > > > > > > > > >
> > > > >> > > > > > > > > > >
> > > > >> > > > > > > > > >
> > > > >> > > > > > > > >
> > > > >> > > > > > > >
> > > > >> > > > > > >
> > > > >> > > > > >
> > > > >> > > > >
> > > > >> > > >
> > > > >> > >
> > > > >> >
> > > > >>
> > > > >
> > > >
> > >
>

Re: Proposal for REST endpoints for table metrics and events

[EJ Wang <ej...@gmail.com>]

Hi folks,

Thanks again for the discussion today. I updated the sync doc with notes
from today's metrics sync:
https://docs.google.com/document/d/100h7c4damrUzVuquYbBHM0EvA4LSWuW2IT2dN_7nYVA/edit?pli=1&tab=t.2ahd7wze5f

See highlights below:

In previous meetings, EJ proposed a two prs framing to push forward the
metrics query work in polaris. Dmitri pointed out a subtle point that pr2
is currently also trying to do the persistence related code placement
refactor. For the sake of straightforward review process, EJ, Dmitri and
Anand (the author) agreed to separate the persistence refactoring into pr0,
and reframing the pr scopes as follow:

   1.

   Pr0: refactor irc metrics spi
   1.

      SPI should stay in core/ with metric type enveloped
      2.

      default spi impl, no-op, in extensions/ - default selection at
      runtime/
      3.

      move metrics persistence and correspondings to extensions/ as an
      alternative SPI impl
      2.

   Pr1: rest api for query
   https://github.com/apache/polaris/pull/4115/changes
   1.

      rest api spec in extensions/ as optional spec
      2.

      SPI in extensions/ as optional spec spi
      3.

      default spi impl in extensions/ no-op
      4.

      handler related in runtime/ - thin adaptor from http shape to spi
      shape
      3.

   Pr2: jdbc impl for query https://github.com/apache/polaris/pull/4756
   1.

      alternative spi impl in extensions/ -> query with jdbc


Thanks,
-ej

On Wed, Jul 1, 2026 at 8:01 AM Alexandre Dutra <ad...@apache.org> wrote:

> Hi all,
>
> > While the current implementation stores metrics in the metastore, other
> deployments
> may choose a KV store, a dedicated metrics warehouse, or another backend
> entirely. In those cases, exposing a Polaris REST API over a specific
> schema feels like coupling the API to an implementation detail.
>
> I wouldn't say to an implementation detail, but rather to a particular
> setup / configuration. I think that's fine. If you don't store
> metrics, you don't read metrics. Similar situations will certainly
> happen in the future, e.g. if we ever introduce an API to read events,
> that would only work if events are being persisted in the first place.
> Again I think that's a fair trade-off.
>
> Thanks,
> Alex
>
> On Wed, Jul 1, 2026 at 7:23 AM Dmitri Bourlatchkov <di...@apache.org>
> wrote:
> >
> > Hi Yufei,
> >
> > From my POV, it is fine for Polaris to have multiple REST API surfaces
> with
> > different features and different degrees of "readiness". Each API can
> > evolve fairly independently. With proper modularization (which Anand
> did),
> > changes to the main Polaris code and these extra modules should normally
> be
> > non-intrusive to each other.
> >
> > I assume Anand is going to be the first user of this API. However, if we
> do
> > not merge it we'll never know whether other users are interested. I do
> not
> > think many end users read the dev ML, but they certainly read the docs,
> > code and release notes.
> >
> > There is no risk for Polaris as a project in exposing this experimental
> > API. If usage dwindles over time, we can remove it. If it gains a
> > considerable user / contributor base, we will promote it to a
> "first-class"
> > (non-beta) API.
> >
> > Cheers,
> > Dmitri.
> >
> > On Tue, Jun 30, 2026 at 7:53 PM Yufei Gu <fl...@gmail.com> wrote:
> >
> > > Hi all,
> > >
> > > Thanks for driving this work forward. Building this as a POC makes
> perfect
> > > sense if the goal is to validate the idea and gather feedback. Should
> we
> > > merge a POC into the main branch before agreeing on the overall
> direction?
> > > To me, that feels like a separate discussion from whether the
> > > implementation itself looks reasonable.
> > >
> > > One concern that I don't think has been fully addressed is whether we
> need
> > > to expose metrics consumption through a Polaris REST endpoint at all.
> > >
> > > My concern is less about the API itself and more about the boundary
> Polaris
> > > should own. The proposed REST API is tied to a particular persistence
> > > schema, but that schema is just one possible implementation. While the
> > > current implementation stores metrics in the metastore, other
> deployments
> > > may choose a KV store, a dedicated metrics warehouse, or another
> backend
> > > entirely. In those cases, exposing a Polaris REST API over a specific
> > > schema feels like coupling the API to an implementation detail.
> > >
> > > Polaris isn't intended to become a full metrics system. I'd expect it
> to
> > > focus on producing metrics and let deployments choose the most
> appropriate
> > > storage and consumption model for their environment.
> > >
> > > I'm not opposed to exploring this direction, but we should first align
> on
> > > whether exposing a REST API through Polaris is the right long term
> approach
> > > before merging it into the main branch.
> > >
> > > Thanks,
> > > Yufei
> > >
> > > Yufei
> > >
> > >
> > > On Mon, Jun 29, 2026 at 9:27 AM Dmitri Bourlatchkov <di...@apache.org>
> > > wrote:
> > >
> > > > Hi All,
> > > >
> > > > It looks like Anand addressed all action items in [4115].
> > > >
> > > > Please review the latest state of the code. The PR LGTM and I
> approved in
> > > > GH from my side.
> > > >
> > > > [4115] https://github.com/apache/polaris/pull/4115
> > > >
> > > > Thanks,
> > > > Dmitri.
> > > >
> > > > On Thu, Jun 25, 2026 at 1:56 PM Dmitri Bourlatchkov <
> dimas@apache.org>
> > > > wrote:
> > > >
> > > > > Hi All,
> > > > >
> > > > > Recap of the discussion in today's Community Sync:
> > > > >
> > > > > * The modular approach for Metrics SPI(s) and implementation seems
> to
> > > > have
> > > > > community consensus (please comment if you still have concerns)
> > > > >
> > > > > * What modules / feature flags are enabled in Apache Polaris
> images by
> > > > > default will be discussed later (when the feature is close to
> > > > completion).
> > > > >
> > > > > In any case, users will be able to turn the full feature on/off at
> > > their
> > > > > discretion, regardless of the default flag values.
> > > > >
> > > > > * It is generally preferable to have smaller PRs and introduce new
> > > > > features gradually to simplify reviews.
> > > > >
> > > > > What this means for the Metrics API, IMHO is:
> > > > >
> > > > > a) The scope of PR 4115 is probably ok as it stands right now. If
> > > someone
> > > > > wants to propose splitting it further, please comment.
> > > > >
> > > > > b) After 4115 is merged, Metrics Persistence PRs will follow, with
> > > > smaller
> > > > > scope per PR as appropriate for the nature of the code changes.
> > > > >
> > > > > * The major concern was about exposing the new Metrics REST API to
> end
> > > > > users. The nature of this concern was about avoiding the (false)
> > > > impression
> > > > > that the new API is a firm and adopted Polaris specification.
> > > > >
> > > > > I think this concern is reasonable.
> > > > >
> > > > > I propose that Anand update the documentation sections of PR 4115
> to
> > > > > emphasise that the API is currently an experimental / POC API at
> this
> > > > time
> > > > > and is subject to change (including breaking changes). I believe it
> > > > already
> > > > > has the "beta" label. So this request is mostly for doc changes to
> > > > clarify
> > > > > user expectations, since the "beta" label alone may not be
> interpreted
> > > > the
> > > > > same way by all users.
> > > > >
> > > > > I hope this update will resolve all remaining concerns with 4115
> and
> > > > allow
> > > > > for it to be merged so that the community will be able to proceed
> to
> > > > > follow-up PRs (as noted above).
> > > > >
> > > > > Please add/correct me if I missed anything relevant.
> > > > >
> > > > > Thanks,
> > > > > Dmitri.
> > > > >
> > > > > On Wed, Jun 24, 2026 at 8:11 PM Yufei Gu <fl...@gmail.com>
> wrote:
> > > > >
> > > > >> I do not think we have agreement on the query REST APIs yet.
> > > > >>
> > > > >> In particular, the current query API shape seems highly coupled
> to the
> > > > >> *example[1]* metrics in the Iceberg REST spec. I do not think
> that is
> > > a
> > > > >> good way to design a REST API contract. For example, fields like
> > > > >> resultDataFiles, resultDeleteFiles, totalFileSizeBytes,
> > > > >> totalDataManifests,
> > > > >> totalDeleteManifests, and scannedDataManifests are very specific
> to
> > > one
> > > > >> style of scan metric. That makes the API look more like a direct
> > > > >> projection
> > > > >> of the current example payload than a general query model.
> > > > >>
> > > > >> My preference is that Polaris should not define or strongly opine
> on
> > > the
> > > > >> Iceberg scan or commit metrics query REST API at this point.
> Polaris
> > > can
> > > > >> resolve and authorize the table, accept the metrics report, and
> > > delegate
> > > > >> it
> > > > >> to the configured reporting path. Durable storage, filtering,
> > > > dashboards,
> > > > >> and query APIs should remain downstream impl. choices unless and
> until
> > > > the
> > > > >> Iceberg community standardizes a query contract, which I doubt
> will
> > > > >> happen.
> > > > >>
> > > > >> So I do not think the intake path has to be blocked, but I would
> > > prefer
> > > > >> not
> > > > >> to merge a Polaris specific query API that may be hard to evolve
> > > later.
> > > > >>
> > > > >> 1.
> > > > >>
> > > > >>
> > > >
> > >
> https://github.com/apache/iceberg/blob/2a6c556c0c883c04c6d3fbd68e6aeda36b91e0aa/open-api/rest-catalog-open-api.yaml#L4152
> > > > >>
> > > > >> Yufei
> > > > >>
> > > > >>
> > > > >> On Tue, Jun 23, 2026 at 9:30 AM Dmitri Bourlatchkov <
> dimas@apache.org
> > > >
> > > > >> wrote:
> > > > >>
> > > > >> > Hi All,
> > > > >> >
> > > > >> > So, what are the blockers for merging [4115], if any?
> > > > >> >
> > > > >> > [4115] https://github.com/apache/polaris/pull/4115
> > > > >> >
> > > > >> > Thanks,
> > > > >> > Dmitri.
> > > > >> >
> > > > >> > On Wed, Jun 17, 2026 at 5:28 PM EJ Wang <
> > > > ej.wang.dev.interest@gmail.com
> > > > >> >
> > > > >> > wrote:
> > > > >> >
> > > > >> > > Hi folks,
> > > > >> > >
> > > > >> > > Thanks again for the discussion today. I updated the sync doc
> with
> > > > >> notes
> > > > >> > > from today's metrics sync:
> > > > >> > >
> > > > >> > >
> > > > >> > >
> > > > >> >
> > > > >>
> > > >
> > >
> https://docs.google.com/document/d/100h7c4damrUzVuquYbBHM0EvA4LSWuW2IT2dN_7nYVA/edit?tab=t.ezk0rgdx0c6m
> > > > >> > >
> > > > >> > > A few highlights:
> > > > >> > >
> > > > >> > >    - No objection to keeping the default metrics behavior
> small:
> > > > >> > >    no-op/log-only is enough as the built-in default.
> > > > >> > >    - Durable storage, event forwarding, external queues,
> > > dashboards,
> > > > >> and
> > > > >> > >    custom filtering should be implementation choices behind
> the
> > > > >> metrics
> > > > >> > >    reporting path.
> > > > >> > >    - For PR #4115, the REST layer should resolve/authz/accept
> the
> > > > >> report,
> > > > >> > >    then delegate to the selected metrics implementation. It
> should
> > > > not
> > > > >> > own
> > > > >> > >    durable storage semantics.
> > > > >> > >    - The rough boundary we discussed is: spec owns the wire
> > > > contract,
> > > > >> > >    runtime owns framework wiring, API/contract modules own
> > > > >> > provider-facing
> > > > >> > >    contracts, and extensions own replaceable implementations.
> > > > >> > >    - OpenLineage has a similar path-naming question. We should
> > > avoid
> > > > >> > >    occupying a generic lineage path too early if we want room
> for
> > > > >> other
> > > > >> > >    lineage systems later.
> > > > >> > >
> > > > >> > > I believe the boundary rules previewed in sync meeting are
> useful
> > > > for
> > > > >> > > reviewing PR #4115, but it is not
> > > > >> > > a final module-layout proposal.
> > > > >> > >
> > > > >> > > For metrics specifically, I think the target remains:
> > > > >> > >    runtime
> > > > >> > >      -> metrics reporting/emitting contract
> > > > >> > >      -> default implementation
> > > > >> > >           -> lower implementation details, if needed
> > > > >> > >
> > > > >> > > That keeps the default simple, while allowing durable JDBC,
> > > > >> event-backed,
> > > > >> > > or external-queue implementations to be added separately.
> > > > >> > >
> > > > >> > > We also touched on the split between Iceberg metrics emitting
> and
> > > > >> > > Polaris-owned metrics querying. The querying side can keep
> > > evolving,
> > > > >> > > especially for dashboard or generic-table use cases, but I do
> not
> > > > >> think
> > > > >> > > that needs to block the intake path in this PR.
> > > > >> > >
> > > > >> > > Thanks,
> > > > >> > > -ej
> > > > >> > >
> > > > >> > > On Tue, Jun 16, 2026 at 2:17 PM Yufei Gu <
> flyrain000@gmail.com>
> > > > >> wrote:
> > > > >> > >
> > > > >> > > > Thanks for chiming in, EJ. Agreed that the default battery
> > > should
> > > > >> stay
> > > > >> > > > small. I'm leaning toward not including metrics persistence
> in
> > > the
> > > > >> > > default
> > > > >> > > > battery.
> > > > >> > > >
> > > > >> > > > Yufei
> > > > >> > > >
> > > > >> > > >
> > > > >> > > > On Thu, Jun 11, 2026 at 8:30 PM EJ Wang <
> > > > >> > ej.wang.dev.interest@gmail.com>
> > > > >> > > > wrote:
> > > > >> > > >
> > > > >> > > > > Hi Yufei,
> > > > >> > > > >
> > > > >> > > > > Thanks for connecting this back to the REST endpoint
> > > proposal. I
> > > > >> > > replied
> > > > >> > > > > with the fuller version on the event-forwarding thread,
> but
> > > > >> wanted to
> > > > >> > > add
> > > > >> > > > > the shorter version here too.
> > > > >> > > > >
> > > > >> > > > > I agree that metrics reporting/emitting is the right
> > > conceptual
> > > > >> > > boundary,
> > > > >> > > > > and I also think the event/listener path is a good
> > > > implementation
> > > > >> > > > direction
> > > > >> > > > > to explore. The distinction I want to keep clear is
> default
> > > > >> battery
> > > > >> > vs
> > > > >> > > > > extension implementation.
> > > > >> > > > >
> > > > >> > > > > For the REST/API side, I would keep the semantics narrow:
> > > > Polaris
> > > > >> > > > resolves
> > > > >> > > > > and authorizes the table, accepts the Iceberg scan/commit
> > > > metrics
> > > > >> > > report
> > > > >> > > > > into the configured reporting/emitting path, and returns
> 204.
> > > > That
> > > > >> > 204
> > > > >> > > > > should mean Polaris accepted the report into the ingestion
> > > path;
> > > > >> it
> > > > >> > > > should
> > > > >> > > > > not imply durable storage.
> > > > >> > > > >
> > > > >> > > > > Durable storage, event forwarding, external telemetry
> routing,
> > > > >> > > filtering,
> > > > >> > > > > and retention should sit behind that reporting/emitting
> > > boundary
> > > > >> as
> > > > >> > > > > implementation-layer behavior. A durable JDBC path can be
> one
> > > > >> named
> > > > >> > > > > extension implementation. An event/listener forwarding
> path
> > > can
> > > > be
> > > > >> > > > another
> > > > >> > > > > named extension implementation.
> > > > >> > > > >
> > > > >> > > > > So I am +1 on the event-forwarding idea as a non-default
> > > > extension
> > > > >> > > > > implementation. I just would not make it the default
> battery
> > > or
> > > > >> the
> > > > >> > > core
> > > > >> > > > > API shape. The default battery can stay small and safe,
> while
> > > > >> > > deployments
> > > > >> > > > > that need forwarding, dashboards, or durable storage can
> opt
> > > > into
> > > > >> the
> > > > >> > > > > implementation that matches their operational model.
> > > > >> > > > >
> > > > >> > > > > Thanks,
> > > > >> > > > > -ej
> > > > >> > > > >
> > > > >> > > > > On Wed, Jun 10, 2026 at 11:43 AM Yufei Gu <
> > > flyrain000@gmail.com
> > > > >
> > > > >> > > wrote:
> > > > >> > > > >
> > > > >> > > > > > Thanks EJ. I agree that the reporting/emitting boundary
> > > feels
> > > > >> like
> > > > >> > > the
> > > > >> > > > > > right SPI boundary, but I wonder if we can simplify this
> > > even
> > > > >> > > further.
> > > > >> > > > > >
> > > > >> > > > > > Iceberg scan and commit metrics look very similar to
> events
> > > to
> > > > >> me.
> > > > >> > > They
> > > > >> > > > > are
> > > > >> > > > > > append only, asynchronously consumed, often forwarded to
> > > > >> external
> > > > >> > > > > systems,
> > > > >> > > > > > and have similar retention and cleanup requirements.
> More
> > > > >> details
> > > > >> > can
> > > > >> > > > be
> > > > >> > > > > > found in this thread [1]. In that case, Polaris may only
> > > need
> > > > to
> > > > >> > > accept
> > > > >> > > > > the
> > > > >> > > > > > report and emit an event through the existing event
> > > framework.
> > > > >> > > > Filtering,
> > > > >> > > > > > async delivery, custom sinks, and retention mechanisms
> could
> > > > >> then
> > > > >> > be
> > > > >> > > > > shared
> > > > >> > > > > > instead of introducing a separate metrics specific
> extension
> > > > >> layer
> > > > >> > > and
> > > > >> > > > > SPI.
> > > > >> > > > > >
> > > > >> > > > > > The main requirement I still see is querying metrics
> through
> > > > >> > Polaris.
> > > > >> > > > If
> > > > >> > > > > we
> > > > >> > > > > > want that battery included experience, we could work on
> a
> > > > >> > persistence
> > > > >> > > > > > listener implementation that selectively saves metrics
> and
> > > > >> expose
> > > > >> > it
> > > > >> > > > via
> > > > >> > > > > > the IRC event endpoint which seems pretty close in
> Iceberg
> > > > >> > > > > > community. Another option I like more is to allow users
> to
> > > > >> develop
> > > > >> > > > their
> > > > >> > > > > > own listener, so they can persist scan/commit metrics
> to any
> > > > >> > systems
> > > > >> > > > they
> > > > >> > > > > > prefer. In general, that feels like an implementation
> choice
> > > > on
> > > > >> top
> > > > >> > > of
> > > > >> > > > > the
> > > > >> > > > > > event framework rather than something that needs to
> shape
> > > the
> > > > >> core
> > > > >> > > > > > architecture.
> > > > >> > > > > >
> > > > >> > > > > > 1.
> > > > >> >
> https://lists.apache.org/thread/x9j8nscvy8hq61tyn01mj8yp6n9of0kp
> > > > >> > > > > >
> > > > >> > > > > > Thanks,
> > > > >> > > > > > Yufei
> > > > >> > > > > >
> > > > >> > > > > >
> > > > >> > > > > > On Thu, Jun 4, 2026 at 5:14 PM EJ Wang <
> > > > >> > > ej.wang.dev.interest@gmail.com
> > > > >> > > > >
> > > > >> > > > > > wrote:
> > > > >> > > > > >
> > > > >> > > > > > > Good point. I agree the existing
> PolarisMetricsReporter is
> > > > >> > already
> > > > >> > > > very
> > > > >> > > > > > > close to the right conceptual boundary. I am not
> > > proposing a
> > > > >> > second
> > > > >> > > > > > > parallel reporter concept. The distinction I am
> trying to
> > > > >> make is
> > > > >> > > > > > narrower:
> > > > >> > > > > > >
> > > > >> > > > > > > Current state:
> > > > >> > > > > > >
> > > > >> > > > > > >    - PolarisMetricsReporter is the existing metrics
> > > > reporting
> > > > >> > hook.
> > > > >> > > > > > >    - But it currently lives under runtime/service.
> > > > >> > > > > > >    - Its contract is documented around Quarkus/CDI
> > > discovery
> > > > >> and
> > > > >> > > > > > > @Identifier
> > > > >> > > > > > >    selection.
> > > > >> > > > > > >    - The default implementation is log-only.
> > > > >> > > > > > >    - The durable implementation writes through
> > > > >> > > PolarisMetricsManager
> > > > >> > > > > and
> > > > >> > > > > > >    MetricsPersistence.
> > > > >> > > > > > >    - MetricsPersistence is currently inherited by
> > > > >> > BasePersistence.
> > > > >> > > > > > >
> > > > >> > > > > > > My proposal:
> > > > >> > > > > > >
> > > > >> > > > > > >    - Keep the reporting/emitting boundary as the SPI.
> > > > >> > > > > > >    - Revise that contract in a framework-agnostic
> optional
> > > > >> > Iceberg
> > > > >> > > > > > *metrics
> > > > >> > > > > > >    extension API layer*.
> > > > >> > > > > > >    - Keep runtime/service responsible for REST
> ingestion,
> > > > >> > > table/authz
> > > > >> > > > > > >    validation, and runtime wiring.
> > > > >> > > > > > >    - Keep the battery implementation log-only in the
> > > metrics
> > > > >> > > > extension
> > > > >> > > > > > API
> > > > >> > > > > > >    layer, *not under runtime/service*.
> > > > >> > > > > > >    - Treat durable JDBC metrics as one implementation
> of
> > > the
> > > > >> > > > reporting
> > > > >> > > > > > SPI.
> > > > >> > > > > > >    - Decompose the MetricsPersistence and
> BasePersistence
> > > > >> > coupling
> > > > >> > > as
> > > > >> > > > > > >    durable implementation detail.
> > > > >> > > > > > >
> > > > >> > > > > > > So the difference is not “new reporter concept vs old
> > > > reporter
> > > > >> > > > > concept.”
> > > > >> > > > > > > but:
> > > > >> > > > > > >
> > > > >> > > > > > >
> > > > >> > > > > > >    - PolarisMetricsReporter today = runtime/service
> > > > CDI-shaped
> > > > >> > > hook.
> > > > >> > > > > > >    - Target reporting SPI = same conceptual boundary,
> but
> > > > >> placed
> > > > >> > in
> > > > >> > > > the
> > > > >> > > > > > >    right extension API layer and kept
> framework-agnostic.
> > > > >> > > > > > >
> > > > >> > > > > > > For the current metrics PR, I think the minimal
> > > > architectural
> > > > >> > > cleanup
> > > > >> > > > > > > before merge is to *put the reporting SPI contract
> and the
> > > > >> > battery
> > > > >> > > > > > default
> > > > >> > > > > > > in the right place*. The durable JDBC implementation,
> > > > >> > event-backed
> > > > >> > > > > async
> > > > >> > > > > > > implementation, external queue support, and deeper
> > > > persistence
> > > > >> > > > cleanup
> > > > >> > > > > > can
> > > > >> > > > > > > continue as follow-ups, as long as we do not lock the
> SPI
> > > > >> > boundary
> > > > >> > > to
> > > > >> > > > > > > BasePersistence or to runtime/service wiring.
> > > > >> > > > > > >
> > > > >> > > > > > > -ej
> > > > >> > > > > > >
> > > > >> > > > > > > On Wed, Jun 3, 2026 at 3:00 PM Dmitri Bourlatchkov <
> > > > >> > > dimas@apache.org
> > > > >> > > > >
> > > > >> > > > > > > wrote:
> > > > >> > > > > > >
> > > > >> > > > > > > > Hi EJ,
> > > > >> > > > > > > >
> > > > >> > > > > > > > Thanks for the recap / summary!
> > > > >> > > > > > > >
> > > > >> > > > > > > > > Do folks agree that the stable SPI boundary
> should be
> > > > >> > > > > > > > metrics reporting/emitting, not metrics persistence?
> > > > >> > > > > > > >
> > > > >> > > > > > > > SGTM.
> > > > >> > > > > > > >
> > > > >> > > > > > > > > Does an optional Iceberg metrics extension API
> layer
> > > > sound
> > > > >> > like
> > > > >> > > > the
> > > > >> > > > > > > right
> > > > >> > > > > > > > home for this SPI?
> > > > >> > > > > > > >
> > > > >> > > > > > > > How is that different from current
> > > PolarisMetricsReporter?
> > > > >> > > > > > > >
> > > > >> > > > > > > > > Should the current durable metrics work be
> reframed
> > > as a
> > > > >> > > durable
> > > > >> > > > > > > > JDBC reference implementation of that SPI?
> > > > >> > > > > > > >
> > > > >> > > > > > > > SGTM.
> > > > >> > > > > > > >
> > > > >> > > > > > > > > What is the smallest PR sequence to get there
> without
> > > > >> > blocking
> > > > >> > > > > > > > the current metrics work unnecessarily?
> > > > >> > > > > > > >
> > > > >> > > > > > > > Let's get
> https://github.com/apache/polaris/pull/4397
> > > > >> merged
> > > > >> > > > first.
> > > > >> > > > > > > >
> > > > >> > > > > > > > Then, I'd propose isolating the Metrics schema from
> the
> > > > >> > MetaStore
> > > > >> > > > > > schema.
> > > > >> > > > > > > > This will probably have some ripple effect into the
> > > > >> bootstrap
> > > > >> > > > > > workflows,
> > > > >> > > > > > > so
> > > > >> > > > > > > > it's not a trivial change.
> > > > >> > > > > > > >
> > > > >> > > > > > > > Then, let's reassess.
> > > > >> > > > > > > >
> > > > >> > > > > > > > Cheers,
> > > > >> > > > > > > > Dmitri.
> > > > >> > > > > > > >
> > > > >> > > > > > > > On Wed, Jun 3, 2026 at 5:40 PM EJ Wang <
> > > > >> > > > > ej.wang.dev.interest@gmail.com
> > > > >> > > > > > >
> > > > >> > > > > > > > wrote:
> > > > >> > > > > > > >
> > > > >> > > > > > > > > Hi folks,
> > > > >> > > > > > > > >
> > > > >> > > > > > > > > Thanks again for the discussion today. I updated
> the
> > > > sync
> > > > >> doc
> > > > >> > > > with
> > > > >> > > > > > > notes
> > > > >> > > > > > > > > from the third metrics architecture sync:
> > > > >> > > > > > > > >
> > > > >> > > > > > > > >
> > > > >> > > > > > > > >
> > > > >> > > > > > > >
> > > > >> > > > > > >
> > > > >> > > > > >
> > > > >> > > > >
> > > > >> > > >
> > > > >> > >
> > > > >> >
> > > > >>
> > > >
> > >
> https://docs.google.com/document/d/100h7c4damrUzVuquYbBHM0EvA4LSWuW2IT2dN_7nYVA/edit?tab=t.k96s2xyqr5u1
> > > > >> > > > > > > > >
> > > > >> > > > > > > > > A few highlights from today’s discussion:
> > > > >> > > > > > > > >
> > > > >> > > > > > > > >    - We clarified that Iceberg metrics reporting
> can
> > > be
> > > > >> > > > interpreted
> > > > >> > > > > > > > either
> > > > >> > > > > > > > >    as sync or async handling from Polaris’s
> > > perspective,
> > > > >> so
> > > > >> > > > Polaris
> > > > >> > > > > > > > should
> > > > >> > > > > > > > >    stay flexible as a platform instead of baking
> one
> > > > >> handling
> > > > >> > > > model
> > > > >> > > > > > > into
> > > > >> > > > > > > > > the
> > > > >> > > > > > > > >    REST API behavior.
> > > > >> > > > > > > > >    - We aligned that the built-in (battery)
> behavior
> > > for
> > > > >> > > metrics
> > > > >> > > > > > > emitting
> > > > >> > > > > > > > >    can stay simple: no-op/log-only is enough as
> the
> > > > >> default.
> > > > >> > > > > > > > >    - Durable metrics persistence should be
> treated as
> > > an
> > > > >> > > > > > implementation
> > > > >> > > > > > > > of
> > > > >> > > > > > > > >    the metrics reporting path, not as the core SPI
> > > > >> boundary.
> > > > >> > > > > > > > >    - The existing durable metrics work can be
> reviewed
> > > > as
> > > > >> a
> > > > >> > > > > reference
> > > > >> > > > > > > > >    implementation of the reporting SPI, with
> > > > >> > > persistence-related
> > > > >> > > > > > logic
> > > > >> > > > > > > > kept
> > > > >> > > > > > > > >    self-contained under a metrics durable
> > > implementation
> > > > >> > module
> > > > >> > > > > > rather
> > > > >> > > > > > > > than
> > > > >> > > > > > > > >    scattered through core entity persistence.
> > > > >> > > > > > > > >    - Dashboard/insights remains a real use case,
> but
> > > we
> > > > >> > agreed
> > > > >> > > to
> > > > >> > > > > > keep
> > > > >> > > > > > > it
> > > > >> > > > > > > > >    separate from the core metrics intake
> discussion
> > > for
> > > > >> now.
> > > > >> > > > > > > > >
> > > > >> > > > > > > > > I also did a quick source check after the meeting
> to
> > > > make
> > > > >> > sure
> > > > >> > > we
> > > > >> > > > > are
> > > > >> > > > > > > > > describing the current state accurately.
> > > > >> > > > > > > > >
> > > > >> > > > > > > > > Current state:
> > > > >> > > > > > > > >
> > > > >> > > > > > > > >    - Polaris already has a metrics reporting hook:
> > > > >> > > > > > > > PolarisMetricsReporter.
> > > > >> > > > > > > > >    - The default implementation is
> > > > DefaultMetricsReporter,
> > > > >> > > > selected
> > > > >> > > > > > by
> > > > >> > > > > > > > >
> polaris.iceberg-metrics.reporting.type=default. It
> > > is
> > > > >> > > > log-only,
> > > > >> > > > > > > > >    effectively quiet unless metrics logging is
> > > enabled.
> > > > >> > > > > > > > >    - There is also a PersistingMetricsReporter,
> > > selected
> > > > >> by
> > > > >> > > > > > > > >
> polaris.iceberg-metrics.reporting.type=persisting,
> > > > >> which
> > > > >> > > > > converts
> > > > >> > > > > > > > >    Iceberg scan/commit reports into Polaris
> metrics
> > > > >> records
> > > > >> > and
> > > > >> > > > > > writes
> > > > >> > > > > > > > > through
> > > > >> > > > > > > > >    PolarisMetricsManager -> MetricsPersistence.
> > > > >> > > > > > > > >    - MetricsPersistence currently exists in the
> > > > >> persistence
> > > > >> > > layer
> > > > >> > > > > and
> > > > >> > > > > > > is
> > > > >> > > > > > > > >    inherited by BasePersistence.
> > > > >> > > > > > > > >
> > > > >> > > > > > > > > My read from the discussion is that the target
> > > boundary
> > > > >> > should
> > > > >> > > > not
> > > > >> > > > > be
> > > > >> > > > > > > > > MetricsPersistence as inherited by
> BasePersistence.
> > > That
> > > > >> path
> > > > >> > > > > should
> > > > >> > > > > > be
> > > > >> > > > > > > > > decomposed as durable implementation detail (taken
> > > care
> > > > >> of by
> > > > >> > > > > > > > > https://github.com/apache/polaris/pull/4397). The
> > > > stable
> > > > >> > > > extension
> > > > >> > > > > > > point
> > > > >> > > > > > > > > should instead be the metrics reporting/emitting
> > > > boundary.
> > > > >> > > > > > > > >
> > > > >> > > > > > > > > Concretely, I think the next proposal should be
> shaped
> > > > >> like
> > > > >> > > this:
> > > > >> > > > > > > > >
> > > > >> > > > > > > > >    1.
> > > > >> > > > > > > > >
> > > > >> > > > > > > > >    Define the proper metrics reporting SPI at an
> > > > optional
> > > > >> > > Iceberg
> > > > >> > > > > > > metrics
> > > > >> > > > > > > > >    extension API layer.
> > > > >> > > > > > > > >
> > > > >> > > > > > > > >    Example direction:
> > > > >> > > > > > > > >
> > > > >> > > > > > > > >    public interface IcebergMetricsReporter {
> > > > >> > > > > > > > >      void reportMetric(IcebergMetricsReportContext
> > > > >> context,
> > > > >> > > > > > > > > MetricsReport report);
> > > > >> > > > > > > > >    }
> > > > >> > > > > > > > >
> > > > >> > > > > > > > >    The context would carry the small
> Polaris-resolved
> > > > >> > envelope:
> > > > >> > > > > > > > >    catalog/table identity, report type, received
> > > > >> timestamp,
> > > > >> > and
> > > > >> > > > > > > > > request/trace
> > > > >> > > > > > > > >    context if available. The raw Iceberg
> MetricsReport
> > > > >> > remains
> > > > >> > > > the
> > > > >> > > > > > > > payload.
> > > > >> > > > > > > > >    2.
> > > > >> > > > > > > > >
> > > > >> > > > > > > > >    Keep runtime/service as the REST ingestion and
> > > wiring
> > > > >> > layer.
> > > > >> > > > > > > > >
> > > > >> > > > > > > > >    The REST handler still resolves the table and
> > > > performs
> > > > >> > authz
> > > > >> > > > > > before
> > > > >> > > > > > > > >    accepting the report. After that, it calls the
> > > > selected
> > > > >> > > > > reporting
> > > > >> > > > > > > > >    implementation.
> > > > >> > > > > > > > >    3.
> > > > >> > > > > > > > >
> > > > >> > > > > > > > >    Keep the battery default no-op/log-only.
> > > > >> > > > > > > > >
> > > > >> > > > > > > > >    This preserves an out-of-box safe default and
> > > avoids
> > > > >> > > > requiring a
> > > > >> > > > > > > > durable
> > > > >> > > > > > > > >    metrics store for every Polaris deployment.
> > > > >> > > > > > > > >    4.
> > > > >> > > > > > > > >
> > > > >> > > > > > > > >    Move durable JDBC metrics into a self-contained
> > > > >> > > implementation
> > > > >> > > > > of
> > > > >> > > > > > > the
> > > > >> > > > > > > > >    reporting SPI.
> > > > >> > > > > > > > >
> > > > >> > > > > > > > >    That implementation can own its schema,
> bootstrap,
> > > > >> > > retention,
> > > > >> > > > > and
> > > > >> > > > > > > read
> > > > >> > > > > > > > >    API support. It should not define the core
> > > reporting
> > > > >> SPI
> > > > >> > > > > boundary,
> > > > >> > > > > > > and
> > > > >> > > > > > > > > it
> > > > >> > > > > > > > >    should not require metrics persistence to
> remain
> > > > >> inherited
> > > > >> > > > from
> > > > >> > > > > > > > >    BasePersistence.
> > > > >> > > > > > > > >    5.
> > > > >> > > > > > > > >
> > > > >> > > > > > > > >    Treat async/event-backed handling as another
> > > > >> > implementation
> > > > >> > > of
> > > > >> > > > > the
> > > > >> > > > > > > > same
> > > > >> > > > > > > > >    reporting SPI.
> > > > >> > > > > > > > >
> > > > >> > > > > > > > >    For example, an event-backed reporter could
> > > enqueue a
> > > > >> > > metrics
> > > > >> > > > > > event
> > > > >> > > > > > > > and
> > > > >> > > > > > > > >    let listeners handle durable storage or other
> > > sinks.
> > > > >> If we
> > > > >> > > > later
> > > > >> > > > > > > need
> > > > >> > > > > > > > a
> > > > >> > > > > > > > >    replaceable queue engine, that seems like a
> shared
> > > > >> > > > event/metrics
> > > > >> > > > > > > > > substrate
> > > > >> > > > > > > > >    topic rather than a metrics-only requirement.
> > > > >> > > > > > > > >
> > > > >> > > > > > > > > This framing lets us keep the REST metrics
> endpoint
> > > > >> simple,
> > > > >> > > > > preserve
> > > > >> > > > > > > the
> > > > >> > > > > > > > > current default behavior, support durable metrics
> > > users,
> > > > >> and
> > > > >> > > > still
> > > > >> > > > > > > leave
> > > > >> > > > > > > > > room for async/event-backed or
> external-queue-based
> > > > >> > > > > implementations.
> > > > >> > > > > > > > >
> > > > >> > > > > > > > > I think the main follow-up questions are:
> > > > >> > > > > > > > >
> > > > >> > > > > > > > >    - Do folks agree that the stable SPI boundary
> > > should
> > > > be
> > > > >> > > > metrics
> > > > >> > > > > > > > >    reporting/emitting, not metrics persistence?
> > > > >> > > > > > > > >    - Does an optional Iceberg metrics extension
> API
> > > > layer
> > > > >> > sound
> > > > >> > > > > like
> > > > >> > > > > > > the
> > > > >> > > > > > > > >    right home for this SPI?
> > > > >> > > > > > > > >    - Should the current durable metrics work be
> > > reframed
> > > > >> as a
> > > > >> > > > > durable
> > > > >> > > > > > > > JDBC
> > > > >> > > > > > > > >    reference implementation of that SPI?
> > > > >> > > > > > > > >    - What is the smallest PR sequence to get there
> > > > without
> > > > >> > > > blocking
> > > > >> > > > > > the
> > > > >> > > > > > > > >    current metrics work unnecessarily?
> > > > >> > > > > > > > >
> > > > >> > > > > > > > > Thanks,
> > > > >> > > > > > > > > -ej
> > > > >> > > > > > > > >
> > > > >> > > > > > > > > On Fri, May 15, 2026 at 5:16 PM Dmitri
> Bourlatchkov <
> > > > >> > > > > > dimas@apache.org>
> > > > >> > > > > > > > > wrote:
> > > > >> > > > > > > > >
> > > > >> > > > > > > > > > Hi JB,
> > > > >> > > > > > > > > >
> > > > >> > > > > > > > > > Could you set up another meeting, please? Same
> time
> > > on
> > > > >> > > > Wednesday
> > > > >> > > > > as
> > > > >> > > > > > > > last
> > > > >> > > > > > > > > > time... I hope it works for everyone.
> > > > >> > > > > > > > > >
> > > > >> > > > > > > > > > Cheers,
> > > > >> > > > > > > > > > Dmitri.
> > > > >> > > > > > > > > >
> > > > >> > > > > > > > > > On Fri, May 15, 2026 at 8:06 PM Yufei Gu <
> > > > >> > > flyrain000@gmail.com
> > > > >> > > > >
> > > > >> > > > > > > wrote:
> > > > >> > > > > > > > > >
> > > > >> > > > > > > > > > > +1 on another sync call next week.
> > > > >> > > > > > > > > > >
> > > > >> > > > > > > > > > > Yufei
> > > > >> > > > > > > > > > >
> > > > >> > > > > > > > > > >
> > > > >> > > > > > > > > > > On Fri, May 15, 2026 at 4:52 PM Dmitri
> > > Bourlatchkov
> > > > <
> > > > >> > > > > > > > dimas@apache.org>
> > > > >> > > > > > > > > > > wrote:
> > > > >> > > > > > > > > > >
> > > > >> > > > > > > > > > > > Hi All,
> > > > >> > > > > > > > > > > >
> > > > >> > > > > > > > > > > > WDYT about another sync call next week?
> > > > >> > > > > > > > > > > >
> > > > >> > > > > > > > > > > > Thanks,
> > > > >> > > > > > > > > > > > Dmitri.
> > > > >> > > > > > > > > > > >
> > > > >> > > > > > > > > > > > On Wed, May 6, 2026 at 5:29 PM Dmitri
> > > > Bourlatchkov <
> > > > >> > > > > > > > dimas@apache.org
> > > > >> > > > > > > > > >
> > > > >> > > > > > > > > > > > wrote:
> > > > >> > > > > > > > > > > >
> > > > >> > > > > > > > > > > > > Hi EJ,
> > > > >> > > > > > > > > > > > >
> > > > >> > > > > > > > > > > > > Thanks for the summary! It covers what we
> > > > >> discussed
> > > > >> > in
> > > > >> > > > the
> > > > >> > > > > > > > meeting
> > > > >> > > > > > > > > > very
> > > > >> > > > > > > > > > > > > well, IMHO.
> > > > >> > > > > > > > > > > > >
> > > > >> > > > > > > > > > > > > Looking forward to concrete PRs :)
> > > > >> > > > > > > > > > > > >
> > > > >> > > > > > > > > > > > > Cheers,
> > > > >> > > > > > > > > > > > > Dmitri.
> > > > >> > > > > > > > > > > > >
> > > > >> > > > > > > > > > > > > On Wed, May 6, 2026 at 5:08 PM EJ Wang <
> > > > >> > > > > > > > > > ej.wang.dev.interest@gmail.com
> > > > >> > > > > > > > > > > >
> > > > >> > > > > > > > > > > > > wrote:
> > > > >> > > > > > > > > > > > >
> > > > >> > > > > > > > > > > > >> Hi folks,
> > > > >> > > > > > > > > > > > >>
> > > > >> > > > > > > > > > > > >> We had a community sync earlier, thanks
> JB
> > > for
> > > > >> > > > scheduling
> > > > >> > > > > > it.
> > > > >> > > > > > > > > Notes
> > > > >> > > > > > > > > > > from
> > > > >> > > > > > > > > > > > >> the first metrics architecture sync (May
> 6,
> > > > >> 10-11am
> > > > >> > > PT).
> > > > >> > > > > > > > > Discussion
> > > > >> > > > > > > > > > > doc
> > > > >> > > > > > > > > > > > >> with per-section status:
> > > > >> > > > > > > > > > > > >>
> > > > >> > > > > > > > > > > > >>
> > > > >> > > > > > > > > > > >
> > > > >> > > > > > > > > > >
> > > > >> > > > > > > > > >
> > > > >> > > > > > > > >
> > > > >> > > > > > > >
> > > > >> > > > > > >
> > > > >> > > > > >
> > > > >> > > > >
> > > > >> > > >
> > > > >> > >
> > > > >> >
> > > > >>
> > > >
> > >
> https://docs.google.com/document/d/100h7c4damrUzVuquYbBHM0EvA4LSWuW2IT2dN_7nYVA/edit?tab=t.0
> > > > >> > > > > > > > > > > > >>
> > > > >> > > > > > > > > > > > >> *The meeting covered both topics from the
> > > doc.
> > > > >> > > > > > Direction-level
> > > > >> > > > > > > > > > > alignment
> > > > >> > > > > > > > > > > > >> was reached on the headline pieces;
> details
> > > > >> remain
> > > > >> > for
> > > > >> > > > PR
> > > > >> > > > > > > review
> > > > >> > > > > > > > > or
> > > > >> > > > > > > > > > > > >> follow-up sessions.*
> > > > >> > > > > > > > > > > > >>
> > > > >> > > > > > > > > > > > >> *Topic 1 — Persistence schema redesign*
> > > > >> > > > > > > > > > > > >> Idea-level alignment on consolidating
> > > per-type
> > > > >> > tables
> > > > >> > > > > > > > > > > > >> (scan_metrics_report,
> > > > >> > > > > > > > > > > > >> commit_metrics_report) into a single
> > > > >> metrics_report
> > > > >> > > > table.
> > > > >> > > > > > The
> > > > >> > > > > > > > > > > > motivating
> > > > >> > > > > > > > > > > > >> cost is the surface area added by every
> new
> > > > >> metric
> > > > >> > > type
> > > > >> > > > > > today:
> > > > >> > > > > > > > new
> > > > >> > > > > > > > > > > > table,
> > > > >> > > > > > > > > > > > >> SPI method, record class, model,
> converter,
> > > > >> schema
> > > > >> > > > > > migration.
> > > > >> > > > > > > > > > > > >>
> > > > >> > > > > > > > > > > > >> Most schema details are deferred to the
> > > schema
> > > > >> PR. A
> > > > >> > > few
> > > > >> > > > > > > > specific
> > > > >> > > > > > > > > > > points
> > > > >> > > > > > > > > > > > >> came up:
> > > > >> > > > > > > > > > > > >> •   metric_schema_version: Yufei prefers
> > > > dropping
> > > > >> > it,
> > > > >> > > > > since
> > > > >> > > > > > > > there
> > > > >> > > > > > > > > is
> > > > >> > > > > > > > > > > no
> > > > >> > > > > > > > > > > > >> spec-level concept of metrics versioning
> > > today
> > > > >> and
> > > > >> > it
> > > > >> > > is
> > > > >> > > > > > hard
> > > > >> > > > > > > to
> > > > >> > > > > > > > > > > define
> > > > >> > > > > > > > > > > > >> unilaterally. Robert prefers keeping it,
> > > given
> > > > >> IRC
> > > > >> > v2
> > > > >> > > is
> > > > >> > > > > > > coming
> > > > >> > > > > > > > > and
> > > > >> > > > > > > > > > > the
> > > > >> > > > > > > > > > > > >> schema should be considered against its
> > > likely
> > > > >> > shape;
> > > > >> > > > > Robert
> > > > >> > > > > > > > also
> > > > >> > > > > > > > > > > raised
> > > > >> > > > > > > > > > > > >> how to differentiate various payload
> formats
> > > if
> > > > >> any.
> > > > >> > > > EJ's
> > > > >> > > > > > read
> > > > >> > > > > > > > is
> > > > >> > > > > > > > > > that
> > > > >> > > > > > > > > > > > >> this
> > > > >> > > > > > > > > > > > >> is a two-way-door decision. We can start
> > > > without
> > > > >> the
> > > > >> > > > > field,
> > > > >> > > > > > > and
> > > > >> > > > > > > > if
> > > > >> > > > > > > > > > IRC
> > > > >> > > > > > > > > > > > v2
> > > > >> > > > > > > > > > > > >> changes the shape we would likely roll a
> > > > >> > corresponding
> > > > >> > > > new
> > > > >> > > > > > > > schema
> > > > >> > > > > > > > > > > > anyway,
> > > > >> > > > > > > > > > > > >> which is not particularly costly.
> > > > >> > > > > > > > > > > > >> •   Payload format: Robert pointed out
> that
> > > > >> future
> > > > >> > > > formats
> > > > >> > > > > > > > beyond
> > > > >> > > > > > > > > > JSON
> > > > >> > > > > > > > > > > > may
> > > > >> > > > > > > > > > > > >> be worth supporting. The exact shape is
> > > > deferred
> > > > >> to
> > > > >> > > the
> > > > >> > > > > > schema
> > > > >> > > > > > > > > > > > discussion.
> > > > >> > > > > > > > > > > > >> •   Partition strategy: Anand suggested
> > > monthly
> > > > >> > > > > partitioning
> > > > >> > > > > > > > based
> > > > >> > > > > > > > > > on
> > > > >> > > > > > > > > > > > his
> > > > >> > > > > > > > > > > > >> experience as potentially helpful at
> scale.
> > > > >> > > > > > > > > > > > >>
> > > > >> > > > > > > > > > > > >> *Topic 2 — Where metrics ingestion and
> > > storage
> > > > >> > belong*
> > > > >> > > > > > > > > > > > >> Idea-level alignment that metrics should
> be a
> > > > >> > > separated
> > > > >> > > > > SPI
> > > > >> > > > > > > from
> > > > >> > > > > > > > > the
> > > > >> > > > > > > > > > > > >> entity
> > > > >> > > > > > > > > > > > >> persistence stack. Two reasons surfaced:
> (a)
> > > > >> > workloads
> > > > >> > > > and
> > > > >> > > > > > > > > > capability
> > > > >> > > > > > > > > > > > >> requirements diverge enough that coupling
> > > them
> > > > >> > creates
> > > > >> > > > > > > > artificial
> > > > >> > > > > > > > > > > > >> constraints, and (b) admin experience
> > > improves
> > > > >> when
> > > > >> > > > > metrics
> > > > >> > > > > > > has
> > > > >> > > > > > > > > its
> > > > >> > > > > > > > > > > own
> > > > >> > > > > > > > > > > > >> bootstrap, retention, and lifecycle.
> Dmitri
> > > > noted
> > > > >> > > > Polaris
> > > > >> > > > > > > being
> > > > >> > > > > > > > a
> > > > >> > > > > > > > > > > > platform
> > > > >> > > > > > > > > > > > >> should have the flexibility to support
> > > > different
> > > > >> > > > > persistence
> > > > >> > > > > > > > > > backends
> > > > >> > > > > > > > > > > > per
> > > > >> > > > > > > > > > > > >> concern, and pointed to a concrete next
> step
> > > of
> > > > >> > > > separating
> > > > >> > > > > > the
> > > > >> > > > > > > > > JDBC
> > > > >> > > > > > > > > > > > >> bootstrap for metrics from the metastore
> > > > >> bootstrap.
> > > > >> > > > Robert
> > > > >> > > > > > > > > proposed
> > > > >> > > > > > > > > > an
> > > > >> > > > > > > > > > > > >> additional UX extension: detect an
> > > > unbootstrapped
> > > > >> > > > metrics
> > > > >> > > > > > > store
> > > > >> > > > > > > > on
> > > > >> > > > > > > > > > > first
> > > > >> > > > > > > > > > > > >> use and auto-bootstrap rather than
> requiring
> > > an
> > > > >> > > explicit
> > > > >> > > > > > > manual
> > > > >> > > > > > > > > > > > bootstrap
> > > > >> > > > > > > > > > > > >> step.
> > > > >> > > > > > > > > > > > >> The meeting also confirmed that Polaris
> > > metrics
> > > > >> can
> > > > >> > > > start
> > > > >> > > > > > > small
> > > > >> > > > > > > > > and
> > > > >> > > > > > > > > > > stay
> > > > >> > > > > > > > > > > > >> Iceberg-focused. Naming and persistence
> > > schema
> > > > >> can
> > > > >> > > lean
> > > > >> > > > > > > > > > > > Iceberg-specific.
> > > > >> > > > > > > > > > > > >> If a future expansion to generic-table
> > > metrics
> > > > or
> > > > >> > > > > > operational
> > > > >> > > > > > > > > > metrics
> > > > >> > > > > > > > > > > > >> arrives, an abstraction layer can be
> built on
> > > > >> top of
> > > > >> > > the
> > > > >> > > > > > > Iceberg
> > > > >> > > > > > > > > > > metrics
> > > > >> > > > > > > > > > > > >> reporter at that point. Robert remains
> on the
> > > > >> fence
> > > > >> > > and
> > > > >> > > > > > would
> > > > >> > > > > > > > > prefer
> > > > >> > > > > > > > > > > > >> something more generic but did not block
> the
> > > > >> > > direction;
> > > > >> > > > > > > Dmitri's
> > > > >> > > > > > > > > > read
> > > > >> > > > > > > > > > > > was
> > > > >> > > > > > > > > > > > >> that the proposed framework already has
> > > enough
> > > > >> > > > flexibility
> > > > >> > > > > > to
> > > > >> > > > > > > > > absorb
> > > > >> > > > > > > > > > > > >> future
> > > > >> > > > > > > > > > > > >> expansion.
> > > > >> > > > > > > > > > > > >>
> > > > >> > > > > > > > > > > > >> The Trade-offs and Proposed structure
> > > sections
> > > > in
> > > > >> > the
> > > > >> > > > doc
> > > > >> > > > > > were
> > > > >> > > > > > > > not
> > > > >> > > > > > > > > > > > >> reviewed
> > > > >> > > > > > > > > > > > >> in detail. They remain open for either
> the
> > > next
> > > > >> sync
> > > > >> > > or
> > > > >> > > > PR
> > > > >> > > > > > > > review.
> > > > >> > > > > > > > > > > > >>
> > > > >> > > > > > > > > > > > >> *Cross-cutting alignment —
> battery-included
> > > > plus
> > > > >> > > > > pluggable*
> > > > >> > > > > > > > > > > > >> A common philosophy emerged from the
> > > > discussion.
> > > > >> EJ
> > > > >> > > > > > summarized
> > > > >> > > > > > > > it
> > > > >> > > > > > > > > > as:
> > > > >> > > > > > > > > > > > >> Polaris should provide a
> battery-included UX
> > > > for
> > > > >> > > > beginners
> > > > >> > > > > > and
> > > > >> > > > > > > > the
> > > > >> > > > > > > > > > > > >> flexibility for advanced users to swap
> the
> > > > >> included
> > > > >> > > > > battery
> > > > >> > > > > > > for
> > > > >> > > > > > > > > > > > something
> > > > >> > > > > > > > > > > > >> more powerful or tailored to their use
> case.
> > > > The
> > > > >> SPI
> > > > >> > > > > design
> > > > >> > > > > > > > needs
> > > > >> > > > > > > > > to
> > > > >> > > > > > > > > > > > >> enable
> > > > >> > > > > > > > > > > > >> both.
> > > > >> > > > > > > > > > > > >>
> > > > >> > > > > > > > > > > > >> The inputs that shaped this framing:
> > > > >> > > > > > > > > > > > >> •   Anand described how his team uses the
> > > > current
> > > > >> > > > metrics
> > > > >> > > > > > > > > > persistence
> > > > >> > > > > > > > > > > > >> (three metrics consumers in v1.4).
> > > > >> > > > > > > > > > > > >> •   Yufei raised Grafana and dashboard
> > > > >> integrations
> > > > >> > > as a
> > > > >> > > > > > > > > destination
> > > > >> > > > > > > > > > > use
> > > > >> > > > > > > > > > > > >> case beyond the default.
> > > > >> > > > > > > > > > > > >> •   Robert called out that the current
> design
> > > > is
> > > > >> > more
> > > > >> > > > > > > > > JDBC-focused.
> > > > >> > > > > > > > > > > > >>
> > > > >> > > > > > > > > > > > >> Two concrete instances:
> > > > >> > > > > > > > > > > > >> •   Async metrics intake: Yufei's initial
> > > > >> position
> > > > >> > was
> > > > >> > > > > that
> > > > >> > > > > > > > async
> > > > >> > > > > > > > > > > should
> > > > >> > > > > > > > > > > > >> largely live on the producer side and
> there
> > > is
> > > > >> not
> > > > >> > > much
> > > > >> > > > > > > Polaris
> > > > >> > > > > > > > > can
> > > > >> > > > > > > > > > > do.
> > > > >> > > > > > > > > > > > >> Robert suggested a Polaris-side default
> is
> > > > doable
> > > > >> > via
> > > > >> > > > > > Vert.x.
> > > > >> > > > > > > > > Dmitri
> > > > >> > > > > > > > > > > > >> agreed
> > > > >> > > > > > > > > > > > >> the direction is worth exploring. The
> meeting
> > > > >> > > converged
> > > > >> > > > > on a
> > > > >> > > > > > > > > > > > >> battery-included default (likely
> > > Vert.x-backed)
> > > > >> with
> > > > >> > > an
> > > > >> > > > > SPI
> > > > >> > > > > > > > shape
> > > > >> > > > > > > > > > that
> > > > >> > > > > > > > > > > > >> lets
> > > > >> > > > > > > > > > > > >> power users route to a more scalable
> backend
> > > > >> > > (k8s-hosted
> > > > >> > > > > > > queue,
> > > > >> > > > > > > > > AWS
> > > > >> > > > > > > > > > > SQS,
> > > > >> > > > > > > > > > > > >> etc.).
> > > > >> > > > > > > > > > > > >> •   Pluggable destinations: combining
> Yufei's
> > > > >> > > dashboard
> > > > >> > > > > use
> > > > >> > > > > > > case
> > > > >> > > > > > > > > > with
> > > > >> > > > > > > > > > > > >> Robert's JDBC-focused call-out, the
> meeting
> > > > >> agreed
> > > > >> > the
> > > > >> > > > SPI
> > > > >> > > > > > > > should
> > > > >> > > > > > > > > be
> > > > >> > > > > > > > > > > > >> structured for multiple sinks so
> integrations
> > > > >> become
> > > > >> > > > impl
> > > > >> > > > > > > > choices
> > > > >> > > > > > > > > > > rather
> > > > >> > > > > > > > > > > > >> than architectural changes.
> > > > >> > > > > > > > > > > > >>
> > > > >> > > > > > > > > > > > >> The battery-included default is most
> likely
> > > to
> > > > >> use
> > > > >> > the
> > > > >> > > > > > > existing
> > > > >> > > > > > > > > > > > >> JDBC-backed
> > > > >> > > > > > > > > > > > >> approach.
> > > > >> > > > > > > > > > > > >>
> > > > >> > > > > > > > > > > > >> *Direction (idea-level alignment)*
> > > > >> > > > > > > > > > > > >> •   Single metrics_report table
> consolidating
> > > > >> > per-type
> > > > >> > > > > > > metrics,
> > > > >> > > > > > > > > > > > replacing
> > > > >> > > > > > > > > > > > >> scan_metrics_report and
> commit_metrics_report
> > > > >> > > > > > > > > > > > >> •   Iceberg-focused naming and schema for
> > > now,
> > > > >> > revisit
> > > > >> > > > if
> > > > >> > > > > > > > > > > generic-table
> > > > >> > > > > > > > > > > > or
> > > > >> > > > > > > > > > > > >> operational metrics arrive
> > > > >> > > > > > > > > > > > >> •   Metrics persistence as a separated
> SPI,
> > > not
> > > > >> on
> > > > >> > > > > > > > BasePersistence
> > > > >> > > > > > > > > > > > >> •   Bootstrap path separated for metrics,
> > > > >> > independent
> > > > >> > > of
> > > > >> > > > > > > > metastore
> > > > >> > > > > > > > > > > > >> bootstrap
> > > > >> > > > > > > > > > > > >> •   "Battery-included plus pluggable" as
> the
> > > > SPI
> > > > >> > > design
> > > > >> > > > > > > > philosophy
> > > > >> > > > > > > > > > > > >>
> > > > >> > > > > > > > > > > > >> *Open items*
> > > > >> > > > > > > > > > > > >> •   Schema details:
> metric_schema_version,
> > > > >> payload
> > > > >> > > > format,
> > > > >> > > > > > IRC
> > > > >> > > > > > > > v2
> > > > >> > > > > > > > > > > > >> forward-compat shape
> > > > >> > > > > > > > > > > > >> •   SPI design details — full review
> either
> > > in
> > > > >> the
> > > > >> > > next
> > > > >> > > > > sync
> > > > >> > > > > > > or
> > > > >> > > > > > > > in
> > > > >> > > > > > > > > > the
> > > > >> > > > > > > > > > > > >> corresponding PR
> > > > >> > > > > > > > > > > > >> •   Schema refactor PR ownership
> > > > >> > > > > > > > > > > > >>
> > > > >> > > > > > > > > > > > >> *Action items*
> > > > >> > > > > > > > > > > > >> •   EJ to take a first stab at the SPI
> design
> > > > and
> > > > >> > > > > > potentially
> > > > >> > > > > > > > > > partner
> > > > >> > > > > > > > > > > > with
> > > > >> > > > > > > > > > > > >> Anand to incorporate the lessons learned
> from
> > > > the
> > > > >> > > > existing
> > > > >> > > > > > > > > reporter
> > > > >> > > > > > > > > > > and
> > > > >> > > > > > > > > > > > >> persistence work.
> > > > >> > > > > > > > > > > > >> •   Schema refactor PR ownership is not
> yet
> > > > >> decided.
> > > > >> > > If
> > > > >> > > > > > anyone
> > > > >> > > > > > > > is
> > > > >> > > > > > > > > > > > >> interested in driving it, reply on this
> > > thread.
> > > > >> > > > > > > > > > > > >> •   JB to schedule the next sync,
> tentatively
> > > > in
> > > > >> two
> > > > >> > > > > weeks.
> > > > >> > > > > > > > > > > > >>
> > > > >> > > > > > > > > > > > >> -ej
> > > > >> > > > > > > > > > > > >>
> > > > >> > > > > > > > > > > > >> On Mon, Apr 27, 2026 at 3:07 PM EJ Wang <
> > > > >> > > > > > > > > > > ej.wang.dev.interest@gmail.com
> > > > >> > > > > > > > > > > > >
> > > > >> > > > > > > > > > > > >> wrote:
> > > > >> > > > > > > > > > > > >>
> > > > >> > > > > > > > > > > > >> > Thanks Yufei for the +1.
> > > > >> > > > > > > > > > > > >> >
> > > > >> > > > > > > > > > > > >> > JB, could you help add a biweekly
> metrics
> > > > >> > > architecture
> > > > >> > > > > > sync
> > > > >> > > > > > > to
> > > > >> > > > > > > > > the
> > > > >> > > > > > > > > > > > >> Polaris
> > > > >> > > > > > > > > > > > >> > community calendar? I'm thinking
> Thursdays
> > > at
> > > > >> > 9-10am
> > > > >> > > > PT,
> > > > >> > > > > > on
> > > > >> > > > > > > > the
> > > > >> > > > > > > > > > > > >> off-weeks
> > > > >> > > > > > > > > > > > >> > from the community meeting (starting
> May
> > > 7),
> > > > 60
> > > > >> > > > minutes.
> > > > >> > > > > > > > > > > > >> >
> > > > >> > > > > > > > > > > > >> > Here's a rough agenda to work through
> over
> > > > the
> > > > >> > first
> > > > >> > > > few
> > > > >> > > > > > > > > sessions,
> > > > >> > > > > > > > > > > > >> grouped
> > > > >> > > > > > > > > > > > >> > by priority:
> > > > >> > > > > > > > > > > > >> >
> > > > >> > > > > > > > > > > > >> > *First: foundational direction*
> > > > >> > > > > > > > > > > > >> >
> > > > >> > > > > > > > > > > > >> > 1.  MetricsPersistence: public SPI or
> > > > internal
> > > > >> > > > > > > implementation
> > > > >> > > > > > > > > > > detail?
> > > > >> > > > > > > > > > > > >> >    •   Marked @Beta, javadoc calls it a
> > > > >> "Service
> > > > >> > > > > Provider
> > > > >> > > > > > > > > > > Interface",
> > > > >> > > > > > > > > > > > >> but
> > > > >> > > > > > > > > > > > >> > only one consumer
> > > (JdbcBasePersistenceImpl),
> > > > >> lives
> > > > >> > > on
> > > > >> > > > > > > > > > > BasePersistence.
> > > > >> > > > > > > > > > > > >> If
> > > > >> > > > > > > > > > > > >> > demoted to a private helper inside a
> > > > persisting
> > > > >> > > > reporter
> > > > >> > > > > > > impl,
> > > > >> > > > > > > > > > most
> > > > >> > > > > > > > > > > > >> > downstream design decisions become
> > > > >> implementation
> > > > >> > > > > details
> > > > >> > > > > > > > rather
> > > > >> > > > > > > > > > > than
> > > > >> > > > > > > > > > > > >> > contract questions.
> > > > >> > > > > > > > > > > > >> >
> > > > >> > > > > > > > > > > > >> > 2.  Persistence schema redesign
> > > > >> > > > > > > > > > > > >> >    •   Current two-table layout
> > > > >> > > (scan_metrics_report,
> > > > >> > > > > > > > > > > > >> > commit_metrics_report) with ~25
> flattened
> > > > >> columns
> > > > >> > > > each.
> > > > >> > > > > > > Every
> > > > >> > > > > > > > > new
> > > > >> > > > > > > > > > > > metric
> > > > >> > > > > > > > > > > > >> > type requires a new table, SPI method,
> > > record
> > > > >> > class,
> > > > >> > > > > > model,
> > > > >> > > > > > > > > > > converter,
> > > > >> > > > > > > > > > > > >> and
> > > > >> > > > > > > > > > > > >> > schema migration. Direction to explore:
> > > > single
> > > > >> > table
> > > > >> > > > > with
> > > > >> > > > > > > > > > > metric_type
> > > > >> > > > > > > > > > > > >> enum,
> > > > >> > > > > > > > > > > > >> > schema_version, and JSON payload
> column.
> > > > >> > > > > > > > > > > > >> >
> > > > >> > > > > > > > > > > > >> > *Second: design details once direction
> is
> > > > set*
> > > > >> > > > > > > > > > > > >> >
> > > > >> > > > > > > > > > > > >> > 3.  Partition key strategy
> > > > >> > > > > > > > > > > > >> >    •   Single-table design means scan
> > > metrics
> > > > >> at
> > > > >> > > scale
> > > > >> > > > > > will
> > > > >> > > > > > > > have
> > > > >> > > > > > > > > > > high
> > > > >> > > > > > > > > > > > >> > write concurrency per table. Schema
> needs
> > > to
> > > > >> > expose
> > > > >> > > > > enough
> > > > >> > > > > > > > > > structure
> > > > >> > > > > > > > > > > > for
> > > > >> > > > > > > > > > > > >> > backends to shard by entity or time
> range.
> > > > >> > > > > > > > > > > > >> >
> > > > >> > > > > > > > > > > > >> > 4.  Read/write path consistency
> > > > >> > > > > > > > > > > > >> >    •   Writes go through
> > > > PolarisMetricsManager
> > > > >> on
> > > > >> > > > > > > > > > MetaStoreManager.
> > > > >> > > > > > > > > > > > >> Reads
> > > > >> > > > > > > > > > > > >> > bypass MetaStoreManager and go
> straight to
> > > > >> > > > > > BasePersistence,
> > > > >> > > > > > > > > > > excluding
> > > > >> > > > > > > > > > > > >> > non-JDBC backends from the read API.
> > > > >> > > > > > > > > > > > >> >
> > > > >> > > > > > > > > > > > >> > *Third: cleanup and alignment*
> > > > >> > > > > > > > > > > > >> >
> > > > >> > > > > > > > > > > > >> > 5.  PolarisMetricsReporter naming
> > > > >> > > > > > > > > > > > >> >    •   Only handles IRC
> > > > >> (ScanReport/CommitReport),
> > > > >> > > > > doesn't
> > > > >> > > > > > > > cover
> > > > >> > > > > > > > > > > > generic
> > > > >> > > > > > > > > > > > >> > tables or operational metrics. Name is
> > > > broader
> > > > >> > than
> > > > >> > > > > scope.
> > > > >> > > > > > > > > > > > >> >
> > > > >> > > > > > > > > > > > >> > 6.  PolarisMetricsManager facade
> > > passthrough
> > > > >> > > > > > > > > > > > >> >    •   Entire default method is
> > > > >> > > > > > > > > > > > >> callCtx.getMetaStore().writeScanReport().
> > > > >> > > > > > > > > > > > >> > Zero logic, passes Level 1 straight
> through
> > > > to
> > > > >> > Level
> > > > >> > > > 3.
> > > > >> > > > > > Same
> > > > >> > > > > > > > > > > > >> anti-pattern
> > > > >> > > > > > > > > > > > >> > as PolarisEventManager.
> > > > >> > > > > > > > > > > > >> >
> > > > >> > > > > > > > > > > > >> > 7.  Iceberg community alignment
> > > > >> > > > > > > > > > > > >> >    •   Payload-type extension needs
> > > > discussion
> > > > >> on
> > > > >> > > > > > > dev@iceberg.
> > > > >> > > > > > > > > > > > >> obelix74's
> > > > >> > > > > > > > > > > > >> > Feb thread got zero replies. Needs a
> > > > committer
> > > > >> > > voice.
> > > > >> > > > > > > > > > > > >> >
> > > > >> > > > > > > > > > > > >> > Lets confirm prioritization in the
> first
> > > > >> session.
> > > > >> > > > > > > > > > > > >> >
> > > > >> > > > > > > > > > > > >> > -ej
> > > > >> > > > > > > > > > > > >> >
> > > > >> > > > > > > > > > > > >> > On Tue, Apr 21, 2026 at 3:18 PM Yufei
> Gu <
> > > > >> > > > > > > > flyrain000@gmail.com>
> > > > >> > > > > > > > > > > > wrote:
> > > > >> > > > > > > > > > > > >> >
> > > > >> > > > > > > > > > > > >> >> Thanks everyone for continuing to
> drive
> > > this
> > > > >> > > > forward. I
> > > > >> > > > > > > agree
> > > > >> > > > > > > > > > that
> > > > >> > > > > > > > > > > > the
> > > > >> > > > > > > > > > > > >> >> problem is getting complex enough
> that a
> > > > more
> > > > >> > > > > structured
> > > > >> > > > > > > > > > discussion
> > > > >> > > > > > > > > > > > >> would
> > > > >> > > > > > > > > > > > >> >> help.
> > > > >> > > > > > > > > > > > >> >>
> > > > >> > > > > > > > > > > > >> >> +1 on setting up a biweekly sync for
> the
> > > > >> metrics
> > > > >> > > > > > > > architecture.
> > > > >> > > > > > > > > > I’m
> > > > >> > > > > > > > > > > > >> happy
> > > > >> > > > > > > > > > > > >> >> to
> > > > >> > > > > > > > > > > > >> >> join.
> > > > >> > > > > > > > > > > > >> >>
> > > > >> > > > > > > > > > > > >> >> Yufei
> > > > >> > > > > > > > > > > > >> >>
> > > > >> > > > > > > > > > > > >> >>
> > > > >> > > > > > > > > > > > >> >> On Tue, Apr 21, 2026 at 2:34 PM EJ
> Wang <
> > > > >> > > > > > > > > > > > >> ej.wang.dev.interest@gmail.com>
> > > > >> > > > > > > > > > > > >> >> wrote:
> > > > >> > > > > > > > > > > > >> >>
> > > > >> > > > > > > > > > > > >> >> > Also, I've been looking more
> closely at
> > > > the
> > > > >> > > > > > *persistence
> > > > >> > > > > > > > > schema
> > > > >> > > > > > > > > > > in
> > > > >> > > > > > > > > > > > >> the
> > > > >> > > > > > > > > > > > >> >> > current metrics work*, and I think
> > > > there's a
> > > > >> > > > > structural
> > > > >> > > > > > > > > > rigidity
> > > > >> > > > > > > > > > > > >> problem
> > > > >> > > > > > > > > > > > >> >> > worth raising before the shape gets
> > > locked
> > > > >> in.
> > > > >> > > > > > > > > > > > >> >> >
> > > > >> > > > > > > > > > > > >> >> > Right now we have two separate
> tables
> > > > >> > > > > > > (scan_metrics_report
> > > > >> > > > > > > > > and
> > > > >> > > > > > > > > > > > >> >> > commit_metrics_report), each with
> ~25
> > > > >> flattened
> > > > >> > > > > columns
> > > > >> > > > > > > > that
> > > > >> > > > > > > > > > > > directly
> > > > >> > > > > > > > > > > > >> >> > mirror the Iceberg report fields.
> The
> > > SPI
> > > > >> > follows
> > > > >> > > > the
> > > > >> > > > > > > same
> > > > >> > > > > > > > > > split:
> > > > >> > > > > > > > > > > > >> >> > writeScanReport and
> writeCommitReport as
> > > > >> > separate
> > > > >> > > > > > > methods,
> > > > >> > > > > > > > > with
> > > > >> > > > > > > > > > > > >> per-type
> > > > >> > > > > > > > > > > > >> >> > record classes, converters, and
> model
> > > > >> objects.
> > > > >> > > *The
> > > > >> > > > > > > > practical
> > > > >> > > > > > > > > > > cost:
> > > > >> > > > > > > > > > > > >> >> > adding a new metric type
> (operational
> > > > >> metrics,
> > > > >> > > for
> > > > >> > > > > > > example)
> > > > >> > > > > > > > > > > > requires
> > > > >> > > > > > > > > > > > >> a
> > > > >> > > > > > > > > > > > >> >> new
> > > > >> > > > > > > > > > > > >> >> > table, a new SPI method, a new
> record
> > > > >> class, a
> > > > >> > > new
> > > > >> > > > > > model
> > > > >> > > > > > > > > > class, a
> > > > >> > > > > > > > > > > > new
> > > > >> > > > > > > > > > > > >> >> > converter branch, and a schema
> > > migration*.
> > > > >> > > That's a
> > > > >> > > > > lot
> > > > >> > > > > > > of
> > > > >> > > > > > > > > > > surface
> > > > >> > > > > > > > > > > > >> area
> > > > >> > > > > > > > > > > > >> >> > for what should be "one more kind of
> > > > >> metric."
> > > > >> > > > > > > > > > > > >> >> >
> > > > >> > > > > > > > > > > > >> >> > *My bias* would be toward a single
> > > metrics
> > > > >> > table
> > > > >> > > > with
> > > > >> > > > > > *a
> > > > >> > > > > > > > > typed
> > > > >> > > > > > > > > > > JSON
> > > > >> > > > > > > > > > > > >> >> > payload*. Something like:
> metric_type
> > > > >> (enum),
> > > > >> > > > > > entity_id,
> > > > >> > > > > > > > > > > > >> >> > table_identifier, snapshot_id
> > > (nullable),
> > > > >> > > > > received_ts,
> > > > >> > > > > > > > > > > > >> schema_version,
> > > > >> > > > > > > > > > > > >> >> and
> > > > >> > > > > > > > > > > > >> >> > a payload column for the
> metric-specific
> > > > >> data.
> > > > >> > > The
> > > > >> > > > > > > > > metric_type
> > > > >> > > > > > > > > > +
> > > > >> > > > > > > > > > > > >> >> > schema_version pair gives us a
> > > > >> > forward-compatible
> > > > >> > > > > > > contract
> > > > >> > > > > > > > > for
> > > > >> > > > > > > > > > > the
> > > > >> > > > > > > > > > > > >> >> payload
> > > > >> > > > > > > > > > > > >> >> > shape. Adding a new metric type
> becomes
> > > an
> > > > >> enum
> > > > >> > > > value
> > > > >> > > > > > > and a
> > > > >> > > > > > > > > > > payload
> > > > >> > > > > > > > > > > > >> >> schema,
> > > > >> > > > > > > > > > > > >> >> > not a schema migration. One thing I
> > > think
> > > > we
> > > > >> > need
> > > > >> > > > to
> > > > >> > > > > be
> > > > >> > > > > > > > > > > deliberate
> > > > >> > > > > > > > > > > > >> >> about is
> > > > >> > > > > > > > > > > > >> >> > the partition key design. If all
> metric
> > > > >> types
> > > > >> > > land
> > > > >> > > > in
> > > > >> > > > > > one
> > > > >> > > > > > > > > > table,
> > > > >> > > > > > > > > > > > scan
> > > > >> > > > > > > > > > > > >> >> > metrics at scale (high concurrency,
> high
> > > > >> > > frequency
> > > > >> > > > > > across
> > > > >> > > > > > > > > many
> > > > >> > > > > > > > > > > > >> tables)
> > > > >> > > > > > > > > > > > >> >> > could easily create hot partitions.
> We'd
> > > > >> want
> > > > >> > the
> > > > >> > > > > > > > persistence
> > > > >> > > > > > > > > > > layer
> > > > >> > > > > > > > > > > > >> to
> > > > >> > > > > > > > > > > > >> >> be
> > > > >> > > > > > > > > > > > >> >> > able to shard by entity or time
> range,
> > > and
> > > > >> that
> > > > >> > > > means
> > > > >> > > > > > the
> > > > >> > > > > > > > > > logical
> > > > >> > > > > > > > > > > > >> schema
> > > > >> > > > > > > > > > > > >> >> > needs to expose enough structure for
> > > > >> backends
> > > > >> > to
> > > > >> > > > > > > partition
> > > > >> > > > > > > > > on.
> > > > >> > > > > > > > > > I
> > > > >> > > > > > > > > > > > >> don't
> > > > >> > > > > > > > > > > > >> >> > think the current flattened layout
> gives
> > > > us
> > > > >> > that.
> > > > >> > > > > > > > > > > > >> >> >
> > > > >> > > > > > > > > > > > >> >> > This is getting complex enough that
> I
> > > > don't
> > > > >> > think
> > > > >> > > > > > ad-hoc
> > > > >> > > > > > > > > PR/ML
> > > > >> > > > > > > > > > > > >> threads
> > > > >> > > > > > > > > > > > >> >> > will converge well. *Would people be
> > > open
> > > > >> to a
> > > > >> > > > > biweekly
> > > > >> > > > > > > > sync
> > > > >> > > > > > > > > > for
> > > > >> > > > > > > > > > > > >> metrics
> > > > >> > > > > > > > > > > > >> >> > architecture?* I think 30 minutes
> every
> > > > two
> > > > >> > weeks
> > > > >> > > > > with
> > > > >> > > > > > > > > > interested
> > > > >> > > > > > > > > > > > >> >> parties
> > > > >> > > > > > > > > > > > >> >> > would be enough to work through the
> > > > schema,
> > > > >> SPI
> > > > >> > > > > shape,
> > > > >> > > > > > > and
> > > > >> > > > > > > > > read
> > > > >> > > > > > > > > > > API
> > > > >> > > > > > > > > > > > >> >> design
> > > > >> > > > > > > > > > > > >> >> > together. Happy to help set that up.
> > > > >> > > > > > > > > > > > >> >> >
> > > > >> > > > > > > > > > > > >> >> > -ej
> > > > >> > > > > > > > > > > > >> >> >
> > > > >> > > > > > > > > > > > >> >> > On Mon, Apr 20, 2026 at 2:19 PM EJ
> Wang
> > > <
> > > > >> > > > > > > > > > > > >> ej.wang.dev.interest@gmail.com
> > > > >> > > > > > > > > > > > >> >> >
> > > > >> > > > > > > > > > > > >> >> > wrote:
> > > > >> > > > > > > > > > > > >> >> >
> > > > >> > > > > > > > > > > > >> >> >> Reviewed #4115, left a comment on
> the
> > > > code
> > > > >> > > > > > organization
> > > > >> > > > > > > > > side.
> > > > >> > > > > > > > > > > > >> >> >>
> > > > >> > > > > > > > > > > > >> >> >> One thing stood out: the metrics
> write
> > > > path
> > > > >> > > enters
> > > > >> > > > > > > through
> > > > >> > > > > > > > > > > > >> >> >> PolarisMetricsManager on
> > > > MetaStoreManager,
> > > > >> but
> > > > >> > > the
> > > > >> > > > > new
> > > > >> > > > > > > > read
> > > > >> > > > > > > > > > path
> > > > >> > > > > > > > > > > > >> >> bypasses
> > > > >> > > > > > > > > > > > >> >> >> MetaStoreManager entirely and goes
> > > > >> straight to
> > > > >> > > > > > > > > BasePersistence
> > > > >> > > > > > > > > > > via
> > > > >> > > > > > > > > > > > >> >> >> callContext.getMetaStore(). That
> means
> > > > the
> > > > >> > read
> > > > >> > > > API
> > > > >> > > > > > only
> > > > >> > > > > > > > > works
> > > > >> > > > > > > > > > > for
> > > > >> > > > > > > > > > > > >> >> backends
> > > > >> > > > > > > > > > > > >> >> >> that implement BasePersistence.
> NoSQL
> > > and
> > > > >> > remote
> > > > >> > > > > > > backends
> > > > >> > > > > > > > > > can't
> > > > >> > > > > > > > > > > > >> >> participate.
> > > > >> > > > > > > > > > > > >> >> >>
> > > > >> > > > > > > > > > > > >> >> >> Stepping back, I think the metrics
> > > > >> subsystem
> > > > >> > is
> > > > >> > > > > > growing
> > > > >> > > > > > > > into
> > > > >> > > > > > > > > > > > >> something
> > > > >> > > > > > > > > > > > >> >> >> real (write + read + REST API +
> AuthZ +
> > > > >> > > > pagination)
> > > > >> > > > > > *but
> > > > >> > > > > > > > the
> > > > >> > > > > > > > > > > > >> >> persistence
> > > > >> > > > > > > > > > > > >> >> >> side is split across two layers in
> a
> > > way
> > > > >> > that's
> > > > >> > > > hard
> > > > >> > > > > > to
> > > > >> > > > > > > > > > > extend*. I
> > > > >> > > > > > > > > > > > >> put
> > > > >> > > > > > > > > > > > >> >> >> together two diagrams to show what
> I
> > > mean
> > > > >> (my
> > > > >> > > best
> > > > >> > > > > > > > effort).
> > > > >> > > > > > > > > > > > >> >> >>
> > > > >> > > > > > > > > > > > >> >> >> *Current state* (Diagram 1): three
> > > > >> interfaces
> > > > >> > at
> > > > >> > > > > three
> > > > >> > > > > > > > > > different
> > > > >> > > > > > > > > > > > >> >> levels.
> > > > >> > > > > > > > > > > > >> >> >> The engine-facing SPI
> > > > >> (PolarisMetricsReporter)
> > > > >> > > is
> > > > >> > > > > > clean.
> > > > >> > > > > > > > But
> > > > >> > > > > > > > > > > > >> >> >> PolarisMetricsManager on
> > > MetaStoreManager
> > > > >> is a
> > > > >> > > > > > > passthrough
> > > > >> > > > > > > > > to
> > > > >> > > > > > > > > > > > >> >> >> MetricsPersistence on
> BasePersistence.
> > > > The
> > > > >> > @Beta
> > > > >> > > > > > > > annotation
> > > > >> > > > > > > > > > and
> > > > >> > > > > > > > > > > > SPI
> > > > >> > > > > > > > > > > > >> >> javadoc
> > > > >> > > > > > > > > > > > >> >> >> are on the BasePersistence layer,
> while
> > > > the
> > > > >> > > actual
> > > > >> > > > > > > > extension
> > > > >> > > > > > > > > > > > points
> > > > >> > > > > > > > > > > > >> >> >> (PolarisMetricsReporter,
> > > > >> > PolarisMetricsManager)
> > > > >> > > > > carry
> > > > >> > > > > > no
> > > > >> > > > > > > > > > > stability
> > > > >> > > > > > > > > > > > >> >> >> annotation. The write path goes
> through
> > > > the
> > > > >> > > > > > > > MetaStoreManager
> > > > >> > > > > > > > > > > > layer,
> > > > >> > > > > > > > > > > > >> the
> > > > >> > > > > > > > > > > > >> >> >> read path doesn't.
> > > > >> > > > > > > > > > > > >> >> >>
> > > > >> > > > > > > > > > > > >> >> >> *What I envision* (Diagram 2): two
> SPIs
> > > > at
> > > > >> two
> > > > >> > > > > levels.
> > > > >> > > > > > > > > > > > >> >> >> PolarisMetricsReporter stays as the
> > > > >> > > engine-facing
> > > > >> > > > > SPI.
> > > > >> > > > > > > > > > > > >> >> >> PolarisMetricsManager becomes the
> > > > >> > backend-facing
> > > > >> > > > SPI
> > > > >> > > > > > > with
> > > > >> > > > > > > > > both
> > > > >> > > > > > > > > > > > write
> > > > >> > > > > > > > > > > > >> >> and
> > > > >> > > > > > > > > > > > >> >> >> read methods at the
> MetaStoreManager
> > > > level,
> > > > >> > > where
> > > > >> > > > > any
> > > > >> > > > > > > > > backend
> > > > >> > > > > > > > > > > > (JDBC,
> > > > >> > > > > > > > > > > > >> >> NoSQL,
> > > > >> > > > > > > > > > > > >> >> >> remote) can implement them.
> > > > >> MetricsPersistence
> > > > >> > > on
> > > > >> > > > > > > > > > > BasePersistence
> > > > >> > > > > > > > > > > > >> goes
> > > > >> > > > > > > > > > > > >> >> >> away. Where metrics actually land
> is an
> > > > >> > > > > implementation
> > > > >> > > > > > > > > detail,
> > > > >> > > > > > > > > > > > not a
> > > > >> > > > > > > > > > > > >> >> core
> > > > >> > > > > > > > > > > > >> >> >> interface.
> > > > >> > > > > > > > > > > > >> >> >>
> > > > >> > > > > > > > > > > > >> >> >> *Minor naming thing*:
> > > > >> PolarisMetricsReporter
> > > > >> > is
> > > > >> > > > > > broader
> > > > >> > > > > > > > than
> > > > >> > > > > > > > > > > what
> > > > >> > > > > > > > > > > > it
> > > > >> > > > > > > > > > > > >> >> >> actually handles. It only accepts
> > > Iceberg
> > > > >> REST
> > > > >> > > > > Catalog
> > > > >> > > > > > > > > metrics
> > > > >> > > > > > > > > > > > >> >> (ScanReport,
> > > > >> > > > > > > > > > > > >> >> >> CommitReport via MetricsReport).
> > > Generic
> > > > >> table
> > > > >> > > > > metrics
> > > > >> > > > > > > or
> > > > >> > > > > > > > > > > > >> operational
> > > > >> > > > > > > > > > > > >> >> >> metrics aren't in scope. Not
> blocking,
> > > > but
> > > > >> > worth
> > > > >> > > > > > noting
> > > > >> > > > > > > if
> > > > >> > > > > > > > > the
> > > > >> > > > > > > > > > > > >> metrics
> > > > >> > > > > > > > > > > > >> >> >> surface expands.
> > > > >> > > > > > > > > > > > >> >> >>
> > > > >> > > > > > > > > > > > >> >> >> *Rough sketch of how to get there*:
> > > > >> > > > > > > > > > > > >> >> >>  1.  Add read methods to
> > > > >> PolarisMetricsManager
> > > > >> > > > > > > > > > (listScanReports,
> > > > >> > > > > > > > > > > > >> >> >> listCommitReports) with default
> no-op,
> > > > >> same as
> > > > >> > > the
> > > > >> > > > > > > > existing
> > > > >> > > > > > > > > > > write
> > > > >> > > > > > > > > > > > >> >> methods.
> > > > >> > > > > > > > > > > > >> >> >> (Probably make
> PolarisMetricsManager
> > > more
> > > > >> > > explicit
> > > > >> > > > > on
> > > > >> > > > > > > > being
> > > > >> > > > > > > > > > > > Iceberg
> > > > >> > > > > > > > > > > > >> >> >> specific like package name or class
> > > name
> > > > >> etc.)
> > > > >> > > > > > > > > > > > >> >> >>  2.  Wire MetricsReportsService
> through
> > > > >> > > > > > MetaStoreManager
> > > > >> > > > > > > > > > instead
> > > > >> > > > > > > > > > > > of
> > > > >> > > > > > > > > > > > >> >> >> callContext.getMetaStore().
> > > > >> > > > > > > > > > > > >> >> >>  3.  Extract metrics persistence
> from
> > > > >> > > > > > > > > JdbcBasePersistenceImpl
> > > > >> > > > > > > > > > > into
> > > > >> > > > > > > > > > > > >> its
> > > > >> > > > > > > > > > > > >> >> >> own class. That file carries ~7
> > > > >> > > responsibilities,
> > > > >> > > > > > > metrics
> > > > >> > > > > > > > > > being
> > > > >> > > > > > > > > > > > one
> > > > >> > > > > > > > > > > > >> of
> > > > >> > > > > > > > > > > > >> >> them.
> > > > >> > > > > > > > > > > > >> >> >>  4.  Remove MetricsPersistence from
> > > > >> > > > BasePersistence.
> > > > >> > > > > > > > > > > > >> >> >>
> > > > >> > > > > > > > > > > > >> >> >> *None of this needs to happen in
> #4115.
> > > > >> But if
> > > > >> > > the
> > > > >> > > > > > > > direction
> > > > >> > > > > > > > > > > makes
> > > > >> > > > > > > > > > > > >> >> sense,
> > > > >> > > > > > > > > > > > >> >> >> it would be good to align before
> the
> > > > >> metrics
> > > > >> > > > surface
> > > > >> > > > > > > grows
> > > > >> > > > > > > > > > > > further.
> > > > >> > > > > > > > > > > > >> >> Curious
> > > > >> > > > > > > > > > > > >> >> >> what others think.*
> > > > >> > > > > > > > > > > > >> >> >>
> > > > >> > > > > > > > > > > > >> >> >> *My mental model note*: Level 1
> > > > >> > > MetaStoreManager;
> > > > >> > > > > > level
> > > > >> > > > > > > 2
> > > > >> > > > > > > > > > > > >> transactional
> > > > >> > > > > > > > > > > > >> >> >> persistence; level 3 base
> persistence
> > > > >> > > > > > > > > > > > >> >> >>
> > > > >> > > > > > > > > > > > >> >> >> Diagram 1
> > > > >> > > > > > > > > > > > >> >> >> <
> > > > >> > > > > > > > > > > > >> >>
> > > > >> > > > > > > > > > > > >>
> > > > >> > > > > > > > > > > >
> > > > >> > > > > > > > > > >
> > > > >> > > > > > > > > >
> > > > >> > > > > > > > >
> > > > >> > > > > > > >
> > > > >> > > > > > >
> > > > >> > > > > >
> > > > >> > > > >
> > > > >> > > >
> > > > >> > >
> > > > >> >
> > > > >>
> > > >
> > >
> https://www.plantuml.com/plantuml/uml/bLHDR-Cs4BthLmpIYupw0zbkKQ1r3M-S7Bp8xhhM7WCOb3IM65EaGD9EX2RzxHrHb4CxRelwa4YSDu_lpOVcnZ9jzvM8BBS2uGjQpJC3dtHMSekPtMk44IpsMgEqa5XcCOhCZikQQLP1pR8TAp2n3ILhmZDP20m0fcIvUkAoW2qJXd9z1bpToO9BX3WXu0ucy5rpgGPNm0nW5_epUWtm2Ue3pn3kMOFQmKntGZW0BYtgBSi8k5A2QMwybJNMIbFiGSR9QZc4nUqIvikStF0jHprua5C-amge42aNt3R0f5JaaoivdV2Pkqbx4hee4ymOkBh5BTiB-_uIeGeo8zL8rPsPl4DktdEiK1jkB1NdZCRbrSTecDe_mlHbF0wvBmCkaOH5_S8a_TTTKI6-nmCAkEw4LpxsZ-LbYLKQFKMNOgf_wuM7_bV9gOer5SYMMksBSWXFcbi49KNZXNLicwfe3TETC7gPdPqI7uBcHMb1RSzYq34c6PDUM9mn8HRsUTZEiDBve3NjVZumBj0U7SS37mGO7vcwtiK-_pU7U7L_f-digo9YbhSwIfMRwIITKGXbxdIUTCGF1SeCJxloKsU-3k9ddRbX1eDq1q_fx1JbBGT0glVyXimDuP4TQ5qpCAmnGEj2s_6n5mtn1z-97-63itFQZLPO1Ev2tu_WF7Ju-VPc0Skg5bYXxBhkY1xpD7EM_7fyflSpIsqMgVth5xhVr4eQxWQ8enaSAJQSG16yFSDuJ798rrcXr_3n-lfdk7icQjEBmFujL7AodiP_Y4Z7-YxvtZNs4zMgpNTl6tF8sglyPsmqchrjvQ-m-aP94r-TwCA2Ka8upPJZwtvSpoYCXkYMZU2NXvRMBfq9P3i3Le4VAZUAlUZ_oPKsxPgY0Q_BSKLkyr9bhQhQrJjo_x3TPlIB0DPjnMfcIoYP0QaYw1a0fTKDr8fB6ntNuvmoL1ZGkXa69Njh43zf9GiGxHQrA_jDYWRSzF5--WmTVrN97_Sm8LbLUy_lGBmLanJjFkDlGkRqjA_4tm00
> > > > >> > > > > > > > > > > > >> >> >
> > > > >> > > > > > > > > > > > >> >> >> :
> > > > >> > > > > > > > > > > > >> >> >>
> > > > >> > > > > > > > > > > > >> >> >> [image: image.png]
> > > > >> > > > > > > > > > > > >> >> >>
> > > > >> > > > > > > > > > > > >> >> >> Diagram 2
> > > > >> > > > > > > > > > > > >> >> >> <
> > > > >> > > > > > > > > > > > >> >>
> > > > >> > > > > > > > > > > > >>
> > > > >> > > > > > > > > > > >
> > > > >> > > > > > > > > > >
> > > > >> > > > > > > > > >
> > > > >> > > > > > > > >
> > > > >> > > > > > > >
> > > > >> > > > > > >
> > > > >> > > > > >
> > > > >> > > > >
> > > > >> > > >
> > > > >> > >
> > > > >> >
> > > > >>
> > > >
> > >
> https://www.plantuml.com/plantuml/uml/VLLDR-8m4BtdLupO2sWBLVU8AaGB7AXAbssGzb896SS9RXqxjKqBwkv_tt7iV43fdaZYDpFlpRmnOsE9jhjSH9PRmM31hERKm8scMsuPjJlDe0yheZDc8RR4iYWoBrmMH9CS2a9VICPYUy1OZN0YCy5Q0BCbYNhdCeEK28En8G8wCvbnoQ0R8_05Bc6bkLIz3X03p1zzH7zR-9ZfDquPt9C3qoNCX2yV4G2NbkcKu5jdgGJHt0GbZwnG6i-UP3TUpk5gM6Ldqke350eZUqzoCft3U9xWHvxoa5-7K4nF1J46EbEMafsmdrCBbQ44gVggy18IZrn_ph5asd1ZiIKdQSgueZvjXrQFSFrdC3YN-nXmBacxbGiYyLVxLaBtdhqn0LSzdBDhqQtQoOJeGyad3z0lUqnYgpGB6Ns8oVyta00Dy_WnX0tIOZ8v6SYxHll1TrH6aejAik-mh-AphVFCwSUQqFypElag5QRGFDjQKEd96K1P8QP41c9TzA_IIQyvdAWyv_RSiS3skb0_EzDDkK2v5xWF6MiGFlvhpFLcD2Dq2pml14gaF67eQkmd8gulDoC4kSOu6KVpkvlUJg1RTbWISU40RdBUUS_9XfRZ2dwxm_SW8LYFISgm_MnlDQ6M9P1gbKEc4X-2pH_FvJCkCqm9pbVjD6LrwdLeOrDWfOaqc8Wh9BE85oNKxkNQ6o4yGRy_Eae0G_G8tZv81d3bHDB23WOdisohVr3nh_j6lbSjbNaLRTc8UgtPbAU1J_tygOfZX9DWEJeHDvYx-qmSi5FgNLPZwHrHcUsncGQ5-skhUclpE5fo4ounpFauYrUbkU6ccfnxMvitwag4IyerhTxj8In_Oj1bDO4pQru674loYrGlULHLEGCjwJJ8gDoVZR8MxO4BT3IzRvIcAQKezC6xpziGnTyImrfEGyJI_OcKfgtxIvnTqFEMS17L9Z-jsARN5FmTheP7HtSdtOMT0B4GY2FYHXxgQmMtj2bRqiLFGapiVe1_QVKDrkqXcm83aFEXnMYCZ-xlyHy
> > > > >> > > > > > > > > > > > >> >> >
> > > > >> > > > > > > > > > > > >> >> >> :
> > > > >> > > > > > > > > > > > >> >> >> [image: image.png]
> > > > >> > > > > > > > > > > > >> >> >>
> > > > >> > > > > > > > > > > > >> >> >>  -ej
> > > > >> > > > > > > > > > > > >> >> >>
> > > > >> > > > > > > > > > > > >> >> >> On Wed, Apr 15, 2026 at 8:22 AM
> Dmitri
> > > > >> > > > Bourlatchkov
> > > > >> > > > > <
> > > > >> > > > > > > > > > > > >> dimas@apache.org>
> > > > >> > > > > > > > > > > > >> >> >> wrote:
> > > > >> > > > > > > > > > > > >> >> >>
> > > > >> > > > > > > > > > > > >> >> >>> Hi All,
> > > > >> > > > > > > > > > > > >> >> >>>
> > > > >> > > > > > > > > > > > >> >> >>> Heads up: The current state of PR
> > > [4115]
> > > > >> > looks
> > > > >> > > > > pretty
> > > > >> > > > > > > > solid
> > > > >> > > > > > > > > > to
> > > > >> > > > > > > > > > > > me.
> > > > >> > > > > > > > > > > > >> I
> > > > >> > > > > > > > > > > > >> >> >>> believe this PR is approaching a
> > > > mergeable
> > > > >> > > > > condition.
> > > > >> > > > > > > > > > > > >> >> >>>
> > > > >> > > > > > > > > > > > >> >> >>> Please post your reviews if you
> have
> > > any
> > > > >> > > > comments.
> > > > >> > > > > > > > > > > > >> >> >>>
> > > > >> > > > > > > > > > > > >> >> >>> [4115]
> > > > >> > > > https://github.com/apache/polaris/pull/4115
> > > > >> > > > > > > > > > > > >> >> >>>
> > > > >> > > > > > > > > > > > >> >> >>> Thanks,
> > > > >> > > > > > > > > > > > >> >> >>> Dmitri.
> > > > >> > > > > > > > > > > > >> >> >>>
> > > > >> > > > > > > > > > > > >> >> >>> On Tue, Mar 3, 2026 at 3:29 PM
> Anand
> > > > Kumar
> > > > >> > > > Sankaran
> > > > >> > > > > > via
> > > > >> > > > > > > > > dev <
> > > > >> > > > > > > > > > > > >> >> >>> dev@polaris.apache.org> wrote:
> > > > >> > > > > > > > > > > > >> >> >>>
> > > > >> > > > > > > > > > > > >> >> >>> > Hi Yufei and Dmitri,
> > > > >> > > > > > > > > > > > >> >> >>> >
> > > > >> > > > > > > > > > > > >> >> >>> > Here is a proposal for the REST
> > > > >> endpoints
> > > > >> > for
> > > > >> > > > > > metrics
> > > > >> > > > > > > > and
> > > > >> > > > > > > > > > > > events.
> > > > >> > > > > > > > > > > > >> >> >>> >
> > > > >> > > > > > > > > > > > >> >> >>> >
> > > > >> > > > > > https://github.com/apache/polaris/pull/3924/changes
> > > > >> > > > > > > > > > > > >> >> >>> >
> > > > >> > > > > > > > > > > > >> >> >>> > I did not see any precursors for
> > > > >> raising a
> > > > >> > PR
> > > > >> > > > for
> > > > >> > > > > > > > > > proposals,
> > > > >> > > > > > > > > > > so
> > > > >> > > > > > > > > > > > >> >> trying
> > > > >> > > > > > > > > > > > >> >> >>> > this.  Please let me know what
> you
> > > > >> think.
> > > > >> > > > > > > > > > > > >> >> >>> >
> > > > >> > > > > > > > > > > > >> >> >>> > -
> > > > >> > > > > > > > > > > > >> >> >>> > Anand
> > > > >> > > > > > > > > > > > >> >> >>> >
> > > > >> > > > > > > > > > > > >> >> >>> > From: Anand Kumar Sankaran <
> > > > >> > > > > > > anand.sankaran@workday.com
> > > > >> > > > > > > > >
> > > > >> > > > > > > > > > > > >> >> >>> > Date: Monday, March 2, 2026 at
> > > > 10:25 AM
> > > > >> > > > > > > > > > > > >> >> >>> > To: dev@polaris.apache.org <
> > > > >> > > > > dev@polaris.apache.org
> > > > >> > > > > > >
> > > > >> > > > > > > > > > > > >> >> >>> > Subject: Re: Polaris Telemetry
> and
> > > > Audit
> > > > >> > > Trail
> > > > >> > > > > > > > > > > > >> >> >>> >
> > > > >> > > > > > > > > > > > >> >> >>> > About the REST API, based on my
> use
> > > > >> cases:
> > > > >> > > > > > > > > > > > >> >> >>> >
> > > > >> > > > > > > > > > > > >> >> >>> >
> > > > >> > > > > > > > > > > > >> >> >>> >   1.
> > > > >> > > > > > > > > > > > >> >> >>> > I want to be able to query
> commit
> > > > >> metrics
> > > > >> > to
> > > > >> > > > > track
> > > > >> > > > > > > > files
> > > > >> > > > > > > > > > > added
> > > > >> > > > > > > > > > > > /
> > > > >> > > > > > > > > > > > >> >> >>> removed
> > > > >> > > > > > > > > > > > >> >> >>> > per commit, along with record
> > > counts.
> > > > >> The
> > > > >> > > > > ingestion
> > > > >> > > > > > > > > > pipeline
> > > > >> > > > > > > > > > > > that
> > > > >> > > > > > > > > > > > >> >> >>> writes
> > > > >> > > > > > > > > > > > >> >> >>> > this data is owned by us and we
> are
> > > > >> > > guaranteed
> > > > >> > > > to
> > > > >> > > > > > > write
> > > > >> > > > > > > > > > this
> > > > >> > > > > > > > > > > > >> >> >>> information
> > > > >> > > > > > > > > > > > >> >> >>> > for each write.
> > > > >> > > > > > > > > > > > >> >> >>> >   2.
> > > > >> > > > > > > > > > > > >> >> >>> > I want to be able to query scan
> > > > metrics
> > > > >> for
> > > > >> > > > > read. I
> > > > >> > > > > > > > > > > understand
> > > > >> > > > > > > > > > > > >> >> clients
> > > > >> > > > > > > > > > > > >> >> >>> do
> > > > >> > > > > > > > > > > > >> >> >>> > not fulfill this requirement.
> > > > >> > > > > > > > > > > > >> >> >>> >   3.
> > > > >> > > > > > > > > > > > >> >> >>> > I want to be able to query the
> > > events
> > > > >> table
> > > > >> > > > > (events
> > > > >> > > > > > > are
> > > > >> > > > > > > > > > > > >> persisted) -
> > > > >> > > > > > > > > > > > >> >> >>> this
> > > > >> > > > > > > > > > > > >> >> >>> > may supersede #2, I am not sure
> yet.
> > > > >> > > > > > > > > > > > >> >> >>> >
> > > > >> > > > > > > > > > > > >> >> >>> > All this information is in the
> JDBC
> > > > >> based
> > > > >> > > > > > persistence
> > > > >> > > > > > > > > model
> > > > >> > > > > > > > > > > and
> > > > >> > > > > > > > > > > > >> is
> > > > >> > > > > > > > > > > > >> >> >>> > persisted in the metastore. I
> > > > currently
> > > > >> > don’t
> > > > >> > > > > have
> > > > >> > > > > > a
> > > > >> > > > > > > > need
> > > > >> > > > > > > > > > to
> > > > >> > > > > > > > > > > > >> query
> > > > >> > > > > > > > > > > > >> >> >>> > prometheus or open telemetry. I
> do
> > > > >> publish
> > > > >> > > some
> > > > >> > > > > > > events
> > > > >> > > > > > > > to
> > > > >> > > > > > > > > > > > >> Prometheus
> > > > >> > > > > > > > > > > > >> >> >>> and
> > > > >> > > > > > > > > > > > >> >> >>> > they are forwarded to our
> dashboards
> > > > >> > > elsewhere.
> > > > >> > > > > > > > > > > > >> >> >>> >
> > > > >> > > > > > > > > > > > >> >> >>> > About the CLI utilities, I
> meant the
> > > > >> admin
> > > > >> > > user
> > > > >> > > > > > > > > utilities.
> > > > >> > > > > > > > > > In
> > > > >> > > > > > > > > > > > >> one of
> > > > >> > > > > > > > > > > > >> >> >>> the
> > > > >> > > > > > > > > > > > >> >> >>> > earliest drafts of my proposal,
> > > > Prashant
> > > > >> > > > > mentioned
> > > > >> > > > > > > that
> > > > >> > > > > > > > > the
> > > > >> > > > > > > > > > > > >> metrics
> > > > >> > > > > > > > > > > > >> >> >>> tables
> > > > >> > > > > > > > > > > > >> >> >>> > can grow indefinitely and that a
> > > > similar
> > > > >> > > > problem
> > > > >> > > > > > > exists
> > > > >> > > > > > > > > > with
> > > > >> > > > > > > > > > > > the
> > > > >> > > > > > > > > > > > >> >> events
> > > > >> > > > > > > > > > > > >> >> >>> > table as well. We discussed that
> > > > >> cleaning
> > > > >> > up
> > > > >> > > of
> > > > >> > > > > old
> > > > >> > > > > > > > > records
> > > > >> > > > > > > > > > > > from
> > > > >> > > > > > > > > > > > >> >> both
> > > > >> > > > > > > > > > > > >> >> >>> > metrics tables and events
> tables can
> > > > be
> > > > >> > done
> > > > >> > > > via
> > > > >> > > > > a
> > > > >> > > > > > > CLI
> > > > >> > > > > > > > > > > utility.
> > > > >> > > > > > > > > > > > >> >> >>> >
> > > > >> > > > > > > > > > > > >> >> >>> > I see that Yufei has covered the
> > > > >> discussion
> > > > >> > > > about
> > > > >> > > > > > > > > > > datasources.
> > > > >> > > > > > > > > > > > >> >> >>> >
> > > > >> > > > > > > > > > > > >> >> >>> > -
> > > > >> > > > > > > > > > > > >> >> >>> > Anand
> > > > >> > > > > > > > > > > > >> >> >>> >
> > > > >> > > > > > > > > > > > >> >> >>> >
> > > > >> > > > > > > > > > > > >> >> >>> >
> > > > >> > > > > > > > > > > > >> >> >>> > From: Yufei Gu <
> > > flyrain000@gmail.com>
> > > > >> > > > > > > > > > > > >> >> >>> > Date: Friday, February 27, 2026
> at
> > > > >> 9:54 PM
> > > > >> > > > > > > > > > > > >> >> >>> > To: dev@polaris.apache.org <
> > > > >> > > > > dev@polaris.apache.org
> > > > >> > > > > > >
> > > > >> > > > > > > > > > > > >> >> >>> > Subject: Re: Polaris Telemetry
> and
> > > > Audit
> > > > >> > > Trail
> > > > >> > > > > > > > > > > > >> >> >>> >
> > > > >> > > > > > > > > > > > >> >> >>> > This Message Is From an External
> > > > Sender
> > > > >> > > > > > > > > > > > >> >> >>> > This message came from outside
> your
> > > > >> > > > organization.
> > > > >> > > > > > > > > > > > >> >> >>> > Report Suspicious<
> > > > >> > > > > > > > > > > > >> >> >>> >
> > > > >> > > > > > > > > > > > >> >> >>>
> > > > >> > > > > > > > > > > > >> >>
> > > > >> > > > > > > > > > > > >>
> > > > >> > > > > > > > > > > >
> > > > >> > > > > > > > > > >
> > > > >> > > > > > > > > >
> > > > >> > > > > > > > >
> > > > >> > > > > > > >
> > > > >> > > > > > >
> > > > >> > > > > >
> > > > >> > > > >
> > > > >> > > >
> > > > >> > >
> > > > >> >
> > > > >>
> > > >
> > >
> https://us-phishalarm-ewt.proofpoint.com/EWT/v1/Iz9xO38YGHZK!YhNDZABkHi1B699ote2uMwpOZw8i0QMCGO2Szc-HshuABGhGvwPJcymE6G2oUUxtS8xDkSrtGTPm_I3QnVDHoLMk50m9v8z_nZKTkd-bnVUbreF1u0WnfV_X5eYevZl_$
> > > > >> > > > > > > > > > > > >> >> >>> > >
> > > > >> > > > > > > > > > > > >> >> >>> >
> > > > >> > > > > > > > > > > > >> >> >>> >
> > > > >> > > > > > > > > > > > >> >> >>> > As I mentioned in
> > > > >> > > > > > > > > > > > >> >> >>> >
> > > > >> > > > > > > > > > > > >> >> >>>
> > > > >> > > > > > > > > > > > >> >>
> > > > >> > > > > > > > > > > > >>
> > > > >> > > > > > > > > > > >
> > > > >> > > > > > > > > > >
> > > > >> > > > > > > > > >
> > > > >> > > > > > > > >
> > > > >> > > > > > > >
> > > > >> > > > > > >
> > > > >> > > > > >
> > > > >> > > > >
> > > > >> > > >
> > > > >> > >
> > > > >> >
> > > > >>
> > > >
> > >
> https://urldefense.com/v3/__https://github.com/apache/polaris/issues/3890__;!!Iz9xO38YGHZK!5EuyFFkk3vhRWVIRvQAWBSQfpJkTMA9HxugzDwXmN0LPPqhEFxYkFRGVhtb8AqUwXtDh2OplcMnbMDHKOxrvDU0$
> > > > >> > > > > > > > > > > > >> >> >>> ,
> > > > >> > > > > > > > > > > > >> >> >>> > supporting
> > > > >> > > > > > > > > > > > >> >> >>> > multiple data sources is not a
> > > trivial
> > > > >> > > change.
> > > > >> > > > I
> > > > >> > > > > > > would
> > > > >> > > > > > > > > > > strongly
> > > > >> > > > > > > > > > > > >> >> >>> recommend
> > > > >> > > > > > > > > > > > >> >> >>> > starting with a design document
> to
> > > > >> > carefully
> > > > >> > > > > > evaluate
> > > > >> > > > > > > > the
> > > > >> > > > > > > > > > > > >> >> architectural
> > > > >> > > > > > > > > > > > >> >> >>> > implications and long term
> impact.
> > > > >> > > > > > > > > > > > >> >> >>> >
> > > > >> > > > > > > > > > > > >> >> >>> > A REST endpoint to query metrics
> > > seems
> > > > >> > > > reasonable
> > > > >> > > > > > > given
> > > > >> > > > > > > > > the
> > > > >> > > > > > > > > > > > >> current
> > > > >> > > > > > > > > > > > >> >> >>> JDBC
> > > > >> > > > > > > > > > > > >> >> >>> > based persistence model. That
> said,
> > > we
> > > > >> may
> > > > >> > > also
> > > > >> > > > > > > > consider
> > > > >> > > > > > > > > > > > >> alternative
> > > > >> > > > > > > > > > > > >> >> >>> > storage models. For example, if
> we
> > > > later
> > > > >> > > adopt
> > > > >> > > > a
> > > > >> > > > > > time
> > > > >> > > > > > > > > > series
> > > > >> > > > > > > > > > > > >> system
> > > > >> > > > > > > > > > > > >> >> >>> such as
> > > > >> > > > > > > > > > > > >> >> >>> > Prometheus to store metrics, the
> > > query
> > > > >> > model
> > > > >> > > > and
> > > > >> > > > > > > access
> > > > >> > > > > > > > > > > > patterns
> > > > >> > > > > > > > > > > > >> >> would
> > > > >> > > > > > > > > > > > >> >> >>> be
> > > > >> > > > > > > > > > > > >> >> >>> > fundamentally different.
> Designing
> > > the
> > > > >> REST
> > > > >> > > API
> > > > >> > > > > > > without
> > > > >> > > > > > > > > > > > >> considering
> > > > >> > > > > > > > > > > > >> >> >>> these
> > > > >> > > > > > > > > > > > >> >> >>> > potential evolutions may limit
> > > > >> flexibility.
> > > > >> > > I'd
> > > > >> > > > > > > suggest
> > > > >> > > > > > > > > to
> > > > >> > > > > > > > > > > > start
> > > > >> > > > > > > > > > > > >> >> with
> > > > >> > > > > > > > > > > > >> >> >>> the
> > > > >> > > > > > > > > > > > >> >> >>> > use case.
> > > > >> > > > > > > > > > > > >> >> >>> >
> > > > >> > > > > > > > > > > > >> >> >>> > Yufei
> > > > >> > > > > > > > > > > > >> >> >>> >
> > > > >> > > > > > > > > > > > >> >> >>> >
> > > > >> > > > > > > > > > > > >> >> >>> > On Fri, Feb 27, 2026 at 3:42 PM
> > > Dmitri
> > > > >> > > > > > Bourlatchkov <
> > > > >> > > > > > > > > > > > >> >> dimas@apache.org>
> > > > >> > > > > > > > > > > > >> >> >>> > wrote:
> > > > >> > > > > > > > > > > > >> >> >>> >
> > > > >> > > > > > > > > > > > >> >> >>> > > Hi Anand,
> > > > >> > > > > > > > > > > > >> >> >>> > >
> > > > >> > > > > > > > > > > > >> >> >>> > > Sharing my view... subject to
> > > > >> discussion:
> > > > >> > > > > > > > > > > > >> >> >>> > >
> > > > >> > > > > > > > > > > > >> >> >>> > > 1. Adding non-IRC REST API to
> > > > Polaris
> > > > >> is
> > > > >> > > > > > perfectly
> > > > >> > > > > > > > > fine.
> > > > >> > > > > > > > > > > > >> >> >>> > >
> > > > >> > > > > > > > > > > > >> >> >>> > > Figuring out specific endpoint
> > > URIs
> > > > >> and
> > > > >> > > > > payloads
> > > > >> > > > > > > > might
> > > > >> > > > > > > > > > > > require
> > > > >> > > > > > > > > > > > >> a
> > > > >> > > > > > > > > > > > >> >> few
> > > > >> > > > > > > > > > > > >> >> >>> > > roundtrips, so opening a
> separate
> > > > >> thread
> > > > >> > > for
> > > > >> > > > > that
> > > > >> > > > > > > > might
> > > > >> > > > > > > > > > be
> > > > >> > > > > > > > > > > > >> best.
> > > > >> > > > > > > > > > > > >> >> >>> > > Contributors commonly create
> > > Google
> > > > >> Docs
> > > > >> > > for
> > > > >> > > > > new
> > > > >> > > > > > > API
> > > > >> > > > > > > > > > > > proposals
> > > > >> > > > > > > > > > > > >> too
> > > > >> > > > > > > > > > > > >> >> >>> (they
> > > > >> > > > > > > > > > > > >> >> >>> > > fairly easy to update as the
> email
> > > > >> > > discussion
> > > > >> > > > > > > > > > progresses).
> > > > >> > > > > > > > > > > > >> >> >>> > >
> > > > >> > > > > > > > > > > > >> >> >>> > > There was a suggestion to try
> > > > Markdown
> > > > >> > > (with
> > > > >> > > > > PRs)
> > > > >> > > > > > > for
> > > > >> > > > > > > > > > > > proposals
> > > > >> > > > > > > > > > > > >> >> [1]
> > > > >> > > > > > > > > > > > >> >> >>> ...
> > > > >> > > > > > > > > > > > >> >> >>> > > feel free to give it a try if
> you
> > > > are
> > > > >> > > > > comfortable
> > > > >> > > > > > > > with
> > > > >> > > > > > > > > > > that.
> > > > >> > > > > > > > > > > > >> >> >>> > >
> > > > >> > > > > > > > > > > > >> >> >>> > > 2. Could you clarify whether
> you
> > > > mean
> > > > >> end
> > > > >> > > > user
> > > > >> > > > > > > > > utilities
> > > > >> > > > > > > > > > or
> > > > >> > > > > > > > > > > > >> admin
> > > > >> > > > > > > > > > > > >> >> >>> user
> > > > >> > > > > > > > > > > > >> >> >>> > > utilities? In the latter case
> > > those
> > > > >> might
> > > > >> > > be
> > > > >> > > > > more
> > > > >> > > > > > > > > > suitable
> > > > >> > > > > > > > > > > > for
> > > > >> > > > > > > > > > > > >> the
> > > > >> > > > > > > > > > > > >> >> >>> Admin
> > > > >> > > > > > > > > > > > >> >> >>> > > CLI (java) not the Python CLI,
> > > IMHO.
> > > > >> > > > > > > > > > > > >> >> >>> > >
> > > > >> > > > > > > > > > > > >> >> >>> > > Why would these utilities be
> > > common
> > > > >> with
> > > > >> > > > > events?
> > > > >> > > > > > > > IMHO,
> > > > >> > > > > > > > > > > event
> > > > >> > > > > > > > > > > > >> use
> > > > >> > > > > > > > > > > > >> >> >>> cases
> > > > >> > > > > > > > > > > > >> >> >>> > are
> > > > >> > > > > > > > > > > > >> >> >>> > > distinct from scan/commit
> metrics.
> > > > >> > > > > > > > > > > > >> >> >>> > >
> > > > >> > > > > > > > > > > > >> >> >>> > > 3. I'd prefer separating
> metrics
> > > > >> > > persistence
> > > > >> > > > > from
> > > > >> > > > > > > > > > MetaStore
> > > > >> > > > > > > > > > > > >> >> >>> persistence
> > > > >> > > > > > > > > > > > >> >> >>> > at
> > > > >> > > > > > > > > > > > >> >> >>> > > the code level, so that they
> could
> > > > be
> > > > >> > mixed
> > > > >> > > > and
> > > > >> > > > > > > > matched
> > > > >> > > > > > > > > > > > >> >> >>> independently.
> > > > >> > > > > > > > > > > > >> >> >>> > The
> > > > >> > > > > > > > > > > > >> >> >>> > > separate datasource question
> will
> > > > >> become
> > > > >> > a
> > > > >> > > > > > > non-issue
> > > > >> > > > > > > > > with
> > > > >> > > > > > > > > > > > that
> > > > >> > > > > > > > > > > > >> >> >>> approach,
> > > > >> > > > > > > > > > > > >> >> >>> > I
> > > > >> > > > > > > > > > > > >> >> >>> > > guess.
> > > > >> > > > > > > > > > > > >> >> >>> > >
> > > > >> > > > > > > > > > > > >> >> >>> > > The rationale for separating
> scan
> > > > >> metrics
> > > > >> > > and
> > > > >> > > > > > > > metastore
> > > > >> > > > > > > > > > > > >> >> persistence
> > > > >> > > > > > > > > > > > >> >> >>> is
> > > > >> > > > > > > > > > > > >> >> >>> > that
> > > > >> > > > > > > > > > > > >> >> >>> > > "cascading deletes" between
> them
> > > are
> > > > >> > hardly
> > > > >> > > > > ever
> > > > >> > > > > > > > > > required.
> > > > >> > > > > > > > > > > > >> >> >>> Furthermore,
> > > > >> > > > > > > > > > > > >> >> >>> > the
> > > > >> > > > > > > > > > > > >> >> >>> > > data and query patterns are
> very
> > > > >> > different
> > > > >> > > so
> > > > >> > > > > > > > different
> > > > >> > > > > > > > > > > > >> >> technologies
> > > > >> > > > > > > > > > > > >> >> >>> > might
> > > > >> > > > > > > > > > > > >> >> >>> > > be beneficial in each case.
> > > > >> > > > > > > > > > > > >> >> >>> > >
> > > > >> > > > > > > > > > > > >> >> >>> > > [1]
> > > > >> > > > > > > > > > > > >> >> >>> >
> > > > >> > > > > > > > > > > > >> >> >>>
> > > > >> > > > > > > > > > > > >> >>
> > > > >> > > > > > > > > > > > >>
> > > > >> > > > > > > > > > > >
> > > > >> > > > > > > > > > >
> > > > >> > > > > > > > > >
> > > > >> > > > > > > > >
> > > > >> > > > > > > >
> > > > >> > > > > > >
> > > > >> > > > > >
> > > > >> > > > >
> > > > >> > > >
> > > > >> > >
> > > > >> >
> > > > >>
> > > >
> > >
> https://urldefense.com/v3/__https://lists.apache.org/thread/yto2wp982t43h1mqjwnslswhws5z47cy__;!!Iz9xO38YGHZK!5EuyFFkk3vhRWVIRvQAWBSQfpJkTMA9HxugzDwXmN0LPPqhEFxYkFRGVhtb8AqUwXtDh2OplcMnbMDHKxYDakNU$
> > > > >> > > > > > > > > > > > >> >> >>> > >
> > > > >> > > > > > > > > > > > >> >> >>> > > Cheers,
> > > > >> > > > > > > > > > > > >> >> >>> > > Dmitri.
> > > > >> > > > > > > > > > > > >> >> >>> > >
> > > > >> > > > > > > > > > > > >> >> >>> > > On Fri, Feb 27, 2026 at
> 6:19 PM
> > > > Anand
> > > > >> > Kumar
> > > > >> > > > > > > Sankaran
> > > > >> > > > > > > > > via
> > > > >> > > > > > > > > > > dev
> > > > >> > > > > > > > > > > > <
> > > > >> > > > > > > > > > > > >> >> >>> > > dev@polaris.apache.org>
> wrote:
> > > > >> > > > > > > > > > > > >> >> >>> > >
> > > > >> > > > > > > > > > > > >> >> >>> > > > Thanks all. This PR is
> merged
> > > now.
> > > > >> > > > > > > > > > > > >> >> >>> > > >
> > > > >> > > > > > > > > > > > >> >> >>> > > > Here are the follow-up
> features
> > > /
> > > > >> work
> > > > >> > > > > needed.
> > > > >> > > > > > > > These
> > > > >> > > > > > > > > > > were
> > > > >> > > > > > > > > > > > >> all
> > > > >> > > > > > > > > > > > >> >> >>> part of
> > > > >> > > > > > > > > > > > >> >> >>> > > the
> > > > >> > > > > > > > > > > > >> >> >>> > > > merged PR at some point in
> time
> > > > and
> > > > >> > were
> > > > >> > > > > > removed
> > > > >> > > > > > > to
> > > > >> > > > > > > > > > > reduce
> > > > >> > > > > > > > > > > > >> >> scope.
> > > > >> > > > > > > > > > > > >> >> >>> > > >
> > > > >> > > > > > > > > > > > >> >> >>> > > > Please let me know what you
> > > think.
> > > > >> > > > > > > > > > > > >> >> >>> > > >
> > > > >> > > > > > > > > > > > >> >> >>> > > >
> > > > >> > > > > > > > > > > > >> >> >>> > > >   1.  A REST API to paginate
> > > > through
> > > > >> > > table
> > > > >> > > > > > > metrics.
> > > > >> > > > > > > > > > This
> > > > >> > > > > > > > > > > > >> will be
> > > > >> > > > > > > > > > > > >> >> >>> > non-IRC
> > > > >> > > > > > > > > > > > >> >> >>> > > > standard addition.
> > > > >> > > > > > > > > > > > >> >> >>> > > >   2.  Utilities for
> managing old
> > > > >> > records,
> > > > >> > > > > > should
> > > > >> > > > > > > be
> > > > >> > > > > > > > > > > common
> > > > >> > > > > > > > > > > > >> with
> > > > >> > > > > > > > > > > > >> >> >>> events.
> > > > >> > > > > > > > > > > > >> >> >>> > > > There was some discussion
> that
> > > it
> > > > >> > belongs
> > > > >> > > > to
> > > > >> > > > > > the
> > > > >> > > > > > > > CLI.
> > > > >> > > > > > > > > > > > >> >> >>> > > >   3.  Separate datasource
> > > > (metrics,
> > > > >> > > events,
> > > > >> > > > > > even
> > > > >> > > > > > > > > other
> > > > >> > > > > > > > > > > > >> tables?).
> > > > >> > > > > > > > > > > > >> >> >>> > > >
> > > > >> > > > > > > > > > > > >> >> >>> > > >
> > > > >> > > > > > > > > > > > >> >> >>> > > > Anything else?
> > > > >> > > > > > > > > > > > >> >> >>> > > >
> > > > >> > > > > > > > > > > > >> >> >>> > > > -
> > > > >> > > > > > > > > > > > >> >> >>> > > > Anand
> > > > >> > > > > > > > > > > > >> >> >>> > > >
> > > > >> > > > > > > > > > > > >> >> >>> > > >
> > > > >> > > > > > > > > > > > >> >> >>> > >
> > > > >> > > > > > > > > > > > >> >> >>> >
> > > > >> > > > > > > > > > > > >> >> >>> >
> > > > >> > > > > > > > > > > > >> >> >>>
> > > > >> > > > > > > > > > > > >> >> >>
> > > > >> > > > > > > > > > > > >> >>
> > > > >> > > > > > > > > > > > >> >
> > > > >> > > > > > > > > > > > >>
> > > > >> > > > > > > > > > > > >
> > > > >> > > > > > > > > > > >
> > > > >> > > > > > > > > > >
> > > > >> > > > > > > > > >
> > > > >> > > > > > > > >
> > > > >> > > > > > > >
> > > > >> > > > > > >
> > > > >> > > > > >
> > > > >> > > > >
> > > > >> > > >
> > > > >> > >
> > > > >> >
> > > > >>
> > > > >
> > > >
> > >
>

Re: Proposal for REST endpoints for table metrics and events

[Alexandre Dutra <ad...@apache.org>]

Hi all,

> While the current implementation stores metrics in the metastore, other deployments
may choose a KV store, a dedicated metrics warehouse, or another backend
entirely. In those cases, exposing a Polaris REST API over a specific
schema feels like coupling the API to an implementation detail.

I wouldn't say to an implementation detail, but rather to a particular
setup / configuration. I think that's fine. If you don't store
metrics, you don't read metrics. Similar situations will certainly
happen in the future, e.g. if we ever introduce an API to read events,
that would only work if events are being persisted in the first place.
Again I think that's a fair trade-off.

Thanks,
Alex

On Wed, Jul 1, 2026 at 7:23 AM Dmitri Bourlatchkov <di...@apache.org> wrote:
>
> Hi Yufei,
>
> From my POV, it is fine for Polaris to have multiple REST API surfaces with
> different features and different degrees of "readiness". Each API can
> evolve fairly independently. With proper modularization (which Anand did),
> changes to the main Polaris code and these extra modules should normally be
> non-intrusive to each other.
>
> I assume Anand is going to be the first user of this API. However, if we do
> not merge it we'll never know whether other users are interested. I do not
> think many end users read the dev ML, but they certainly read the docs,
> code and release notes.
>
> There is no risk for Polaris as a project in exposing this experimental
> API. If usage dwindles over time, we can remove it. If it gains a
> considerable user / contributor base, we will promote it to a "first-class"
> (non-beta) API.
>
> Cheers,
> Dmitri.
>
> On Tue, Jun 30, 2026 at 7:53 PM Yufei Gu <fl...@gmail.com> wrote:
>
> > Hi all,
> >
> > Thanks for driving this work forward. Building this as a POC makes perfect
> > sense if the goal is to validate the idea and gather feedback. Should we
> > merge a POC into the main branch before agreeing on the overall direction?
> > To me, that feels like a separate discussion from whether the
> > implementation itself looks reasonable.
> >
> > One concern that I don't think has been fully addressed is whether we need
> > to expose metrics consumption through a Polaris REST endpoint at all.
> >
> > My concern is less about the API itself and more about the boundary Polaris
> > should own. The proposed REST API is tied to a particular persistence
> > schema, but that schema is just one possible implementation. While the
> > current implementation stores metrics in the metastore, other deployments
> > may choose a KV store, a dedicated metrics warehouse, or another backend
> > entirely. In those cases, exposing a Polaris REST API over a specific
> > schema feels like coupling the API to an implementation detail.
> >
> > Polaris isn't intended to become a full metrics system. I'd expect it to
> > focus on producing metrics and let deployments choose the most appropriate
> > storage and consumption model for their environment.
> >
> > I'm not opposed to exploring this direction, but we should first align on
> > whether exposing a REST API through Polaris is the right long term approach
> > before merging it into the main branch.
> >
> > Thanks,
> > Yufei
> >
> > Yufei
> >
> >
> > On Mon, Jun 29, 2026 at 9:27 AM Dmitri Bourlatchkov <di...@apache.org>
> > wrote:
> >
> > > Hi All,
> > >
> > > It looks like Anand addressed all action items in [4115].
> > >
> > > Please review the latest state of the code. The PR LGTM and I approved in
> > > GH from my side.
> > >
> > > [4115] https://github.com/apache/polaris/pull/4115
> > >
> > > Thanks,
> > > Dmitri.
> > >
> > > On Thu, Jun 25, 2026 at 1:56 PM Dmitri Bourlatchkov <di...@apache.org>
> > > wrote:
> > >
> > > > Hi All,
> > > >
> > > > Recap of the discussion in today's Community Sync:
> > > >
> > > > * The modular approach for Metrics SPI(s) and implementation seems to
> > > have
> > > > community consensus (please comment if you still have concerns)
> > > >
> > > > * What modules / feature flags are enabled in Apache Polaris images by
> > > > default will be discussed later (when the feature is close to
> > > completion).
> > > >
> > > > In any case, users will be able to turn the full feature on/off at
> > their
> > > > discretion, regardless of the default flag values.
> > > >
> > > > * It is generally preferable to have smaller PRs and introduce new
> > > > features gradually to simplify reviews.
> > > >
> > > > What this means for the Metrics API, IMHO is:
> > > >
> > > > a) The scope of PR 4115 is probably ok as it stands right now. If
> > someone
> > > > wants to propose splitting it further, please comment.
> > > >
> > > > b) After 4115 is merged, Metrics Persistence PRs will follow, with
> > > smaller
> > > > scope per PR as appropriate for the nature of the code changes.
> > > >
> > > > * The major concern was about exposing the new Metrics REST API to end
> > > > users. The nature of this concern was about avoiding the (false)
> > > impression
> > > > that the new API is a firm and adopted Polaris specification.
> > > >
> > > > I think this concern is reasonable.
> > > >
> > > > I propose that Anand update the documentation sections of PR 4115 to
> > > > emphasise that the API is currently an experimental / POC API at this
> > > time
> > > > and is subject to change (including breaking changes). I believe it
> > > already
> > > > has the "beta" label. So this request is mostly for doc changes to
> > > clarify
> > > > user expectations, since the "beta" label alone may not be interpreted
> > > the
> > > > same way by all users.
> > > >
> > > > I hope this update will resolve all remaining concerns with 4115 and
> > > allow
> > > > for it to be merged so that the community will be able to proceed to
> > > > follow-up PRs (as noted above).
> > > >
> > > > Please add/correct me if I missed anything relevant.
> > > >
> > > > Thanks,
> > > > Dmitri.
> > > >
> > > > On Wed, Jun 24, 2026 at 8:11 PM Yufei Gu <fl...@gmail.com> wrote:
> > > >
> > > >> I do not think we have agreement on the query REST APIs yet.
> > > >>
> > > >> In particular, the current query API shape seems highly coupled to the
> > > >> *example[1]* metrics in the Iceberg REST spec. I do not think that is
> > a
> > > >> good way to design a REST API contract. For example, fields like
> > > >> resultDataFiles, resultDeleteFiles, totalFileSizeBytes,
> > > >> totalDataManifests,
> > > >> totalDeleteManifests, and scannedDataManifests are very specific to
> > one
> > > >> style of scan metric. That makes the API look more like a direct
> > > >> projection
> > > >> of the current example payload than a general query model.
> > > >>
> > > >> My preference is that Polaris should not define or strongly opine on
> > the
> > > >> Iceberg scan or commit metrics query REST API at this point. Polaris
> > can
> > > >> resolve and authorize the table, accept the metrics report, and
> > delegate
> > > >> it
> > > >> to the configured reporting path. Durable storage, filtering,
> > > dashboards,
> > > >> and query APIs should remain downstream impl. choices unless and until
> > > the
> > > >> Iceberg community standardizes a query contract, which I doubt will
> > > >> happen.
> > > >>
> > > >> So I do not think the intake path has to be blocked, but I would
> > prefer
> > > >> not
> > > >> to merge a Polaris specific query API that may be hard to evolve
> > later.
> > > >>
> > > >> 1.
> > > >>
> > > >>
> > >
> > https://github.com/apache/iceberg/blob/2a6c556c0c883c04c6d3fbd68e6aeda36b91e0aa/open-api/rest-catalog-open-api.yaml#L4152
> > > >>
> > > >> Yufei
> > > >>
> > > >>
> > > >> On Tue, Jun 23, 2026 at 9:30 AM Dmitri Bourlatchkov <dimas@apache.org
> > >
> > > >> wrote:
> > > >>
> > > >> > Hi All,
> > > >> >
> > > >> > So, what are the blockers for merging [4115], if any?
> > > >> >
> > > >> > [4115] https://github.com/apache/polaris/pull/4115
> > > >> >
> > > >> > Thanks,
> > > >> > Dmitri.
> > > >> >
> > > >> > On Wed, Jun 17, 2026 at 5:28 PM EJ Wang <
> > > ej.wang.dev.interest@gmail.com
> > > >> >
> > > >> > wrote:
> > > >> >
> > > >> > > Hi folks,
> > > >> > >
> > > >> > > Thanks again for the discussion today. I updated the sync doc with
> > > >> notes
> > > >> > > from today's metrics sync:
> > > >> > >
> > > >> > >
> > > >> > >
> > > >> >
> > > >>
> > >
> > https://docs.google.com/document/d/100h7c4damrUzVuquYbBHM0EvA4LSWuW2IT2dN_7nYVA/edit?tab=t.ezk0rgdx0c6m
> > > >> > >
> > > >> > > A few highlights:
> > > >> > >
> > > >> > >    - No objection to keeping the default metrics behavior small:
> > > >> > >    no-op/log-only is enough as the built-in default.
> > > >> > >    - Durable storage, event forwarding, external queues,
> > dashboards,
> > > >> and
> > > >> > >    custom filtering should be implementation choices behind the
> > > >> metrics
> > > >> > >    reporting path.
> > > >> > >    - For PR #4115, the REST layer should resolve/authz/accept the
> > > >> report,
> > > >> > >    then delegate to the selected metrics implementation. It should
> > > not
> > > >> > own
> > > >> > >    durable storage semantics.
> > > >> > >    - The rough boundary we discussed is: spec owns the wire
> > > contract,
> > > >> > >    runtime owns framework wiring, API/contract modules own
> > > >> > provider-facing
> > > >> > >    contracts, and extensions own replaceable implementations.
> > > >> > >    - OpenLineage has a similar path-naming question. We should
> > avoid
> > > >> > >    occupying a generic lineage path too early if we want room for
> > > >> other
> > > >> > >    lineage systems later.
> > > >> > >
> > > >> > > I believe the boundary rules previewed in sync meeting are useful
> > > for
> > > >> > > reviewing PR #4115, but it is not
> > > >> > > a final module-layout proposal.
> > > >> > >
> > > >> > > For metrics specifically, I think the target remains:
> > > >> > >    runtime
> > > >> > >      -> metrics reporting/emitting contract
> > > >> > >      -> default implementation
> > > >> > >           -> lower implementation details, if needed
> > > >> > >
> > > >> > > That keeps the default simple, while allowing durable JDBC,
> > > >> event-backed,
> > > >> > > or external-queue implementations to be added separately.
> > > >> > >
> > > >> > > We also touched on the split between Iceberg metrics emitting and
> > > >> > > Polaris-owned metrics querying. The querying side can keep
> > evolving,
> > > >> > > especially for dashboard or generic-table use cases, but I do not
> > > >> think
> > > >> > > that needs to block the intake path in this PR.
> > > >> > >
> > > >> > > Thanks,
> > > >> > > -ej
> > > >> > >
> > > >> > > On Tue, Jun 16, 2026 at 2:17 PM Yufei Gu <fl...@gmail.com>
> > > >> wrote:
> > > >> > >
> > > >> > > > Thanks for chiming in, EJ. Agreed that the default battery
> > should
> > > >> stay
> > > >> > > > small. I'm leaning toward not including metrics persistence in
> > the
> > > >> > > default
> > > >> > > > battery.
> > > >> > > >
> > > >> > > > Yufei
> > > >> > > >
> > > >> > > >
> > > >> > > > On Thu, Jun 11, 2026 at 8:30 PM EJ Wang <
> > > >> > ej.wang.dev.interest@gmail.com>
> > > >> > > > wrote:
> > > >> > > >
> > > >> > > > > Hi Yufei,
> > > >> > > > >
> > > >> > > > > Thanks for connecting this back to the REST endpoint
> > proposal. I
> > > >> > > replied
> > > >> > > > > with the fuller version on the event-forwarding thread, but
> > > >> wanted to
> > > >> > > add
> > > >> > > > > the shorter version here too.
> > > >> > > > >
> > > >> > > > > I agree that metrics reporting/emitting is the right
> > conceptual
> > > >> > > boundary,
> > > >> > > > > and I also think the event/listener path is a good
> > > implementation
> > > >> > > > direction
> > > >> > > > > to explore. The distinction I want to keep clear is default
> > > >> battery
> > > >> > vs
> > > >> > > > > extension implementation.
> > > >> > > > >
> > > >> > > > > For the REST/API side, I would keep the semantics narrow:
> > > Polaris
> > > >> > > > resolves
> > > >> > > > > and authorizes the table, accepts the Iceberg scan/commit
> > > metrics
> > > >> > > report
> > > >> > > > > into the configured reporting/emitting path, and returns 204.
> > > That
> > > >> > 204
> > > >> > > > > should mean Polaris accepted the report into the ingestion
> > path;
> > > >> it
> > > >> > > > should
> > > >> > > > > not imply durable storage.
> > > >> > > > >
> > > >> > > > > Durable storage, event forwarding, external telemetry routing,
> > > >> > > filtering,
> > > >> > > > > and retention should sit behind that reporting/emitting
> > boundary
> > > >> as
> > > >> > > > > implementation-layer behavior. A durable JDBC path can be one
> > > >> named
> > > >> > > > > extension implementation. An event/listener forwarding path
> > can
> > > be
> > > >> > > > another
> > > >> > > > > named extension implementation.
> > > >> > > > >
> > > >> > > > > So I am +1 on the event-forwarding idea as a non-default
> > > extension
> > > >> > > > > implementation. I just would not make it the default battery
> > or
> > > >> the
> > > >> > > core
> > > >> > > > > API shape. The default battery can stay small and safe, while
> > > >> > > deployments
> > > >> > > > > that need forwarding, dashboards, or durable storage can opt
> > > into
> > > >> the
> > > >> > > > > implementation that matches their operational model.
> > > >> > > > >
> > > >> > > > > Thanks,
> > > >> > > > > -ej
> > > >> > > > >
> > > >> > > > > On Wed, Jun 10, 2026 at 11:43 AM Yufei Gu <
> > flyrain000@gmail.com
> > > >
> > > >> > > wrote:
> > > >> > > > >
> > > >> > > > > > Thanks EJ. I agree that the reporting/emitting boundary
> > feels
> > > >> like
> > > >> > > the
> > > >> > > > > > right SPI boundary, but I wonder if we can simplify this
> > even
> > > >> > > further.
> > > >> > > > > >
> > > >> > > > > > Iceberg scan and commit metrics look very similar to events
> > to
> > > >> me.
> > > >> > > They
> > > >> > > > > are
> > > >> > > > > > append only, asynchronously consumed, often forwarded to
> > > >> external
> > > >> > > > > systems,
> > > >> > > > > > and have similar retention and cleanup requirements. More
> > > >> details
> > > >> > can
> > > >> > > > be
> > > >> > > > > > found in this thread [1]. In that case, Polaris may only
> > need
> > > to
> > > >> > > accept
> > > >> > > > > the
> > > >> > > > > > report and emit an event through the existing event
> > framework.
> > > >> > > > Filtering,
> > > >> > > > > > async delivery, custom sinks, and retention mechanisms could
> > > >> then
> > > >> > be
> > > >> > > > > shared
> > > >> > > > > > instead of introducing a separate metrics specific extension
> > > >> layer
> > > >> > > and
> > > >> > > > > SPI.
> > > >> > > > > >
> > > >> > > > > > The main requirement I still see is querying metrics through
> > > >> > Polaris.
> > > >> > > > If
> > > >> > > > > we
> > > >> > > > > > want that battery included experience, we could work on a
> > > >> > persistence
> > > >> > > > > > listener implementation that selectively saves metrics and
> > > >> expose
> > > >> > it
> > > >> > > > via
> > > >> > > > > > the IRC event endpoint which seems pretty close in Iceberg
> > > >> > > > > > community. Another option I like more is to allow users to
> > > >> develop
> > > >> > > > their
> > > >> > > > > > own listener, so they can persist scan/commit metrics to any
> > > >> > systems
> > > >> > > > they
> > > >> > > > > > prefer. In general, that feels like an implementation choice
> > > on
> > > >> top
> > > >> > > of
> > > >> > > > > the
> > > >> > > > > > event framework rather than something that needs to shape
> > the
> > > >> core
> > > >> > > > > > architecture.
> > > >> > > > > >
> > > >> > > > > > 1.
> > > >> > https://lists.apache.org/thread/x9j8nscvy8hq61tyn01mj8yp6n9of0kp
> > > >> > > > > >
> > > >> > > > > > Thanks,
> > > >> > > > > > Yufei
> > > >> > > > > >
> > > >> > > > > >
> > > >> > > > > > On Thu, Jun 4, 2026 at 5:14 PM EJ Wang <
> > > >> > > ej.wang.dev.interest@gmail.com
> > > >> > > > >
> > > >> > > > > > wrote:
> > > >> > > > > >
> > > >> > > > > > > Good point. I agree the existing PolarisMetricsReporter is
> > > >> > already
> > > >> > > > very
> > > >> > > > > > > close to the right conceptual boundary. I am not
> > proposing a
> > > >> > second
> > > >> > > > > > > parallel reporter concept. The distinction I am trying to
> > > >> make is
> > > >> > > > > > narrower:
> > > >> > > > > > >
> > > >> > > > > > > Current state:
> > > >> > > > > > >
> > > >> > > > > > >    - PolarisMetricsReporter is the existing metrics
> > > reporting
> > > >> > hook.
> > > >> > > > > > >    - But it currently lives under runtime/service.
> > > >> > > > > > >    - Its contract is documented around Quarkus/CDI
> > discovery
> > > >> and
> > > >> > > > > > > @Identifier
> > > >> > > > > > >    selection.
> > > >> > > > > > >    - The default implementation is log-only.
> > > >> > > > > > >    - The durable implementation writes through
> > > >> > > PolarisMetricsManager
> > > >> > > > > and
> > > >> > > > > > >    MetricsPersistence.
> > > >> > > > > > >    - MetricsPersistence is currently inherited by
> > > >> > BasePersistence.
> > > >> > > > > > >
> > > >> > > > > > > My proposal:
> > > >> > > > > > >
> > > >> > > > > > >    - Keep the reporting/emitting boundary as the SPI.
> > > >> > > > > > >    - Revise that contract in a framework-agnostic optional
> > > >> > Iceberg
> > > >> > > > > > *metrics
> > > >> > > > > > >    extension API layer*.
> > > >> > > > > > >    - Keep runtime/service responsible for REST ingestion,
> > > >> > > table/authz
> > > >> > > > > > >    validation, and runtime wiring.
> > > >> > > > > > >    - Keep the battery implementation log-only in the
> > metrics
> > > >> > > > extension
> > > >> > > > > > API
> > > >> > > > > > >    layer, *not under runtime/service*.
> > > >> > > > > > >    - Treat durable JDBC metrics as one implementation of
> > the
> > > >> > > > reporting
> > > >> > > > > > SPI.
> > > >> > > > > > >    - Decompose the MetricsPersistence and BasePersistence
> > > >> > coupling
> > > >> > > as
> > > >> > > > > > >    durable implementation detail.
> > > >> > > > > > >
> > > >> > > > > > > So the difference is not “new reporter concept vs old
> > > reporter
> > > >> > > > > concept.”
> > > >> > > > > > > but:
> > > >> > > > > > >
> > > >> > > > > > >
> > > >> > > > > > >    - PolarisMetricsReporter today = runtime/service
> > > CDI-shaped
> > > >> > > hook.
> > > >> > > > > > >    - Target reporting SPI = same conceptual boundary, but
> > > >> placed
> > > >> > in
> > > >> > > > the
> > > >> > > > > > >    right extension API layer and kept framework-agnostic.
> > > >> > > > > > >
> > > >> > > > > > > For the current metrics PR, I think the minimal
> > > architectural
> > > >> > > cleanup
> > > >> > > > > > > before merge is to *put the reporting SPI contract and the
> > > >> > battery
> > > >> > > > > > default
> > > >> > > > > > > in the right place*. The durable JDBC implementation,
> > > >> > event-backed
> > > >> > > > > async
> > > >> > > > > > > implementation, external queue support, and deeper
> > > persistence
> > > >> > > > cleanup
> > > >> > > > > > can
> > > >> > > > > > > continue as follow-ups, as long as we do not lock the SPI
> > > >> > boundary
> > > >> > > to
> > > >> > > > > > > BasePersistence or to runtime/service wiring.
> > > >> > > > > > >
> > > >> > > > > > > -ej
> > > >> > > > > > >
> > > >> > > > > > > On Wed, Jun 3, 2026 at 3:00 PM Dmitri Bourlatchkov <
> > > >> > > dimas@apache.org
> > > >> > > > >
> > > >> > > > > > > wrote:
> > > >> > > > > > >
> > > >> > > > > > > > Hi EJ,
> > > >> > > > > > > >
> > > >> > > > > > > > Thanks for the recap / summary!
> > > >> > > > > > > >
> > > >> > > > > > > > > Do folks agree that the stable SPI boundary should be
> > > >> > > > > > > > metrics reporting/emitting, not metrics persistence?
> > > >> > > > > > > >
> > > >> > > > > > > > SGTM.
> > > >> > > > > > > >
> > > >> > > > > > > > > Does an optional Iceberg metrics extension API layer
> > > sound
> > > >> > like
> > > >> > > > the
> > > >> > > > > > > right
> > > >> > > > > > > > home for this SPI?
> > > >> > > > > > > >
> > > >> > > > > > > > How is that different from current
> > PolarisMetricsReporter?
> > > >> > > > > > > >
> > > >> > > > > > > > > Should the current durable metrics work be reframed
> > as a
> > > >> > > durable
> > > >> > > > > > > > JDBC reference implementation of that SPI?
> > > >> > > > > > > >
> > > >> > > > > > > > SGTM.
> > > >> > > > > > > >
> > > >> > > > > > > > > What is the smallest PR sequence to get there without
> > > >> > blocking
> > > >> > > > > > > > the current metrics work unnecessarily?
> > > >> > > > > > > >
> > > >> > > > > > > > Let's get https://github.com/apache/polaris/pull/4397
> > > >> merged
> > > >> > > > first.
> > > >> > > > > > > >
> > > >> > > > > > > > Then, I'd propose isolating the Metrics schema from the
> > > >> > MetaStore
> > > >> > > > > > schema.
> > > >> > > > > > > > This will probably have some ripple effect into the
> > > >> bootstrap
> > > >> > > > > > workflows,
> > > >> > > > > > > so
> > > >> > > > > > > > it's not a trivial change.
> > > >> > > > > > > >
> > > >> > > > > > > > Then, let's reassess.
> > > >> > > > > > > >
> > > >> > > > > > > > Cheers,
> > > >> > > > > > > > Dmitri.
> > > >> > > > > > > >
> > > >> > > > > > > > On Wed, Jun 3, 2026 at 5:40 PM EJ Wang <
> > > >> > > > > ej.wang.dev.interest@gmail.com
> > > >> > > > > > >
> > > >> > > > > > > > wrote:
> > > >> > > > > > > >
> > > >> > > > > > > > > Hi folks,
> > > >> > > > > > > > >
> > > >> > > > > > > > > Thanks again for the discussion today. I updated the
> > > sync
> > > >> doc
> > > >> > > > with
> > > >> > > > > > > notes
> > > >> > > > > > > > > from the third metrics architecture sync:
> > > >> > > > > > > > >
> > > >> > > > > > > > >
> > > >> > > > > > > > >
> > > >> > > > > > > >
> > > >> > > > > > >
> > > >> > > > > >
> > > >> > > > >
> > > >> > > >
> > > >> > >
> > > >> >
> > > >>
> > >
> > https://docs.google.com/document/d/100h7c4damrUzVuquYbBHM0EvA4LSWuW2IT2dN_7nYVA/edit?tab=t.k96s2xyqr5u1
> > > >> > > > > > > > >
> > > >> > > > > > > > > A few highlights from today’s discussion:
> > > >> > > > > > > > >
> > > >> > > > > > > > >    - We clarified that Iceberg metrics reporting can
> > be
> > > >> > > > interpreted
> > > >> > > > > > > > either
> > > >> > > > > > > > >    as sync or async handling from Polaris’s
> > perspective,
> > > >> so
> > > >> > > > Polaris
> > > >> > > > > > > > should
> > > >> > > > > > > > >    stay flexible as a platform instead of baking one
> > > >> handling
> > > >> > > > model
> > > >> > > > > > > into
> > > >> > > > > > > > > the
> > > >> > > > > > > > >    REST API behavior.
> > > >> > > > > > > > >    - We aligned that the built-in (battery) behavior
> > for
> > > >> > > metrics
> > > >> > > > > > > emitting
> > > >> > > > > > > > >    can stay simple: no-op/log-only is enough as the
> > > >> default.
> > > >> > > > > > > > >    - Durable metrics persistence should be treated as
> > an
> > > >> > > > > > implementation
> > > >> > > > > > > > of
> > > >> > > > > > > > >    the metrics reporting path, not as the core SPI
> > > >> boundary.
> > > >> > > > > > > > >    - The existing durable metrics work can be reviewed
> > > as
> > > >> a
> > > >> > > > > reference
> > > >> > > > > > > > >    implementation of the reporting SPI, with
> > > >> > > persistence-related
> > > >> > > > > > logic
> > > >> > > > > > > > kept
> > > >> > > > > > > > >    self-contained under a metrics durable
> > implementation
> > > >> > module
> > > >> > > > > > rather
> > > >> > > > > > > > than
> > > >> > > > > > > > >    scattered through core entity persistence.
> > > >> > > > > > > > >    - Dashboard/insights remains a real use case, but
> > we
> > > >> > agreed
> > > >> > > to
> > > >> > > > > > keep
> > > >> > > > > > > it
> > > >> > > > > > > > >    separate from the core metrics intake discussion
> > for
> > > >> now.
> > > >> > > > > > > > >
> > > >> > > > > > > > > I also did a quick source check after the meeting to
> > > make
> > > >> > sure
> > > >> > > we
> > > >> > > > > are
> > > >> > > > > > > > > describing the current state accurately.
> > > >> > > > > > > > >
> > > >> > > > > > > > > Current state:
> > > >> > > > > > > > >
> > > >> > > > > > > > >    - Polaris already has a metrics reporting hook:
> > > >> > > > > > > > PolarisMetricsReporter.
> > > >> > > > > > > > >    - The default implementation is
> > > DefaultMetricsReporter,
> > > >> > > > selected
> > > >> > > > > > by
> > > >> > > > > > > > >    polaris.iceberg-metrics.reporting.type=default. It
> > is
> > > >> > > > log-only,
> > > >> > > > > > > > >    effectively quiet unless metrics logging is
> > enabled.
> > > >> > > > > > > > >    - There is also a PersistingMetricsReporter,
> > selected
> > > >> by
> > > >> > > > > > > > >    polaris.iceberg-metrics.reporting.type=persisting,
> > > >> which
> > > >> > > > > converts
> > > >> > > > > > > > >    Iceberg scan/commit reports into Polaris metrics
> > > >> records
> > > >> > and
> > > >> > > > > > writes
> > > >> > > > > > > > > through
> > > >> > > > > > > > >    PolarisMetricsManager -> MetricsPersistence.
> > > >> > > > > > > > >    - MetricsPersistence currently exists in the
> > > >> persistence
> > > >> > > layer
> > > >> > > > > and
> > > >> > > > > > > is
> > > >> > > > > > > > >    inherited by BasePersistence.
> > > >> > > > > > > > >
> > > >> > > > > > > > > My read from the discussion is that the target
> > boundary
> > > >> > should
> > > >> > > > not
> > > >> > > > > be
> > > >> > > > > > > > > MetricsPersistence as inherited by BasePersistence.
> > That
> > > >> path
> > > >> > > > > should
> > > >> > > > > > be
> > > >> > > > > > > > > decomposed as durable implementation detail (taken
> > care
> > > >> of by
> > > >> > > > > > > > > https://github.com/apache/polaris/pull/4397). The
> > > stable
> > > >> > > > extension
> > > >> > > > > > > point
> > > >> > > > > > > > > should instead be the metrics reporting/emitting
> > > boundary.
> > > >> > > > > > > > >
> > > >> > > > > > > > > Concretely, I think the next proposal should be shaped
> > > >> like
> > > >> > > this:
> > > >> > > > > > > > >
> > > >> > > > > > > > >    1.
> > > >> > > > > > > > >
> > > >> > > > > > > > >    Define the proper metrics reporting SPI at an
> > > optional
> > > >> > > Iceberg
> > > >> > > > > > > metrics
> > > >> > > > > > > > >    extension API layer.
> > > >> > > > > > > > >
> > > >> > > > > > > > >    Example direction:
> > > >> > > > > > > > >
> > > >> > > > > > > > >    public interface IcebergMetricsReporter {
> > > >> > > > > > > > >      void reportMetric(IcebergMetricsReportContext
> > > >> context,
> > > >> > > > > > > > > MetricsReport report);
> > > >> > > > > > > > >    }
> > > >> > > > > > > > >
> > > >> > > > > > > > >    The context would carry the small Polaris-resolved
> > > >> > envelope:
> > > >> > > > > > > > >    catalog/table identity, report type, received
> > > >> timestamp,
> > > >> > and
> > > >> > > > > > > > > request/trace
> > > >> > > > > > > > >    context if available. The raw Iceberg MetricsReport
> > > >> > remains
> > > >> > > > the
> > > >> > > > > > > > payload.
> > > >> > > > > > > > >    2.
> > > >> > > > > > > > >
> > > >> > > > > > > > >    Keep runtime/service as the REST ingestion and
> > wiring
> > > >> > layer.
> > > >> > > > > > > > >
> > > >> > > > > > > > >    The REST handler still resolves the table and
> > > performs
> > > >> > authz
> > > >> > > > > > before
> > > >> > > > > > > > >    accepting the report. After that, it calls the
> > > selected
> > > >> > > > > reporting
> > > >> > > > > > > > >    implementation.
> > > >> > > > > > > > >    3.
> > > >> > > > > > > > >
> > > >> > > > > > > > >    Keep the battery default no-op/log-only.
> > > >> > > > > > > > >
> > > >> > > > > > > > >    This preserves an out-of-box safe default and
> > avoids
> > > >> > > > requiring a
> > > >> > > > > > > > durable
> > > >> > > > > > > > >    metrics store for every Polaris deployment.
> > > >> > > > > > > > >    4.
> > > >> > > > > > > > >
> > > >> > > > > > > > >    Move durable JDBC metrics into a self-contained
> > > >> > > implementation
> > > >> > > > > of
> > > >> > > > > > > the
> > > >> > > > > > > > >    reporting SPI.
> > > >> > > > > > > > >
> > > >> > > > > > > > >    That implementation can own its schema, bootstrap,
> > > >> > > retention,
> > > >> > > > > and
> > > >> > > > > > > read
> > > >> > > > > > > > >    API support. It should not define the core
> > reporting
> > > >> SPI
> > > >> > > > > boundary,
> > > >> > > > > > > and
> > > >> > > > > > > > > it
> > > >> > > > > > > > >    should not require metrics persistence to remain
> > > >> inherited
> > > >> > > > from
> > > >> > > > > > > > >    BasePersistence.
> > > >> > > > > > > > >    5.
> > > >> > > > > > > > >
> > > >> > > > > > > > >    Treat async/event-backed handling as another
> > > >> > implementation
> > > >> > > of
> > > >> > > > > the
> > > >> > > > > > > > same
> > > >> > > > > > > > >    reporting SPI.
> > > >> > > > > > > > >
> > > >> > > > > > > > >    For example, an event-backed reporter could
> > enqueue a
> > > >> > > metrics
> > > >> > > > > > event
> > > >> > > > > > > > and
> > > >> > > > > > > > >    let listeners handle durable storage or other
> > sinks.
> > > >> If we
> > > >> > > > later
> > > >> > > > > > > need
> > > >> > > > > > > > a
> > > >> > > > > > > > >    replaceable queue engine, that seems like a shared
> > > >> > > > event/metrics
> > > >> > > > > > > > > substrate
> > > >> > > > > > > > >    topic rather than a metrics-only requirement.
> > > >> > > > > > > > >
> > > >> > > > > > > > > This framing lets us keep the REST metrics endpoint
> > > >> simple,
> > > >> > > > > preserve
> > > >> > > > > > > the
> > > >> > > > > > > > > current default behavior, support durable metrics
> > users,
> > > >> and
> > > >> > > > still
> > > >> > > > > > > leave
> > > >> > > > > > > > > room for async/event-backed or external-queue-based
> > > >> > > > > implementations.
> > > >> > > > > > > > >
> > > >> > > > > > > > > I think the main follow-up questions are:
> > > >> > > > > > > > >
> > > >> > > > > > > > >    - Do folks agree that the stable SPI boundary
> > should
> > > be
> > > >> > > > metrics
> > > >> > > > > > > > >    reporting/emitting, not metrics persistence?
> > > >> > > > > > > > >    - Does an optional Iceberg metrics extension API
> > > layer
> > > >> > sound
> > > >> > > > > like
> > > >> > > > > > > the
> > > >> > > > > > > > >    right home for this SPI?
> > > >> > > > > > > > >    - Should the current durable metrics work be
> > reframed
> > > >> as a
> > > >> > > > > durable
> > > >> > > > > > > > JDBC
> > > >> > > > > > > > >    reference implementation of that SPI?
> > > >> > > > > > > > >    - What is the smallest PR sequence to get there
> > > without
> > > >> > > > blocking
> > > >> > > > > > the
> > > >> > > > > > > > >    current metrics work unnecessarily?
> > > >> > > > > > > > >
> > > >> > > > > > > > > Thanks,
> > > >> > > > > > > > > -ej
> > > >> > > > > > > > >
> > > >> > > > > > > > > On Fri, May 15, 2026 at 5:16 PM Dmitri Bourlatchkov <
> > > >> > > > > > dimas@apache.org>
> > > >> > > > > > > > > wrote:
> > > >> > > > > > > > >
> > > >> > > > > > > > > > Hi JB,
> > > >> > > > > > > > > >
> > > >> > > > > > > > > > Could you set up another meeting, please? Same time
> > on
> > > >> > > > Wednesday
> > > >> > > > > as
> > > >> > > > > > > > last
> > > >> > > > > > > > > > time... I hope it works for everyone.
> > > >> > > > > > > > > >
> > > >> > > > > > > > > > Cheers,
> > > >> > > > > > > > > > Dmitri.
> > > >> > > > > > > > > >
> > > >> > > > > > > > > > On Fri, May 15, 2026 at 8:06 PM Yufei Gu <
> > > >> > > flyrain000@gmail.com
> > > >> > > > >
> > > >> > > > > > > wrote:
> > > >> > > > > > > > > >
> > > >> > > > > > > > > > > +1 on another sync call next week.
> > > >> > > > > > > > > > >
> > > >> > > > > > > > > > > Yufei
> > > >> > > > > > > > > > >
> > > >> > > > > > > > > > >
> > > >> > > > > > > > > > > On Fri, May 15, 2026 at 4:52 PM Dmitri
> > Bourlatchkov
> > > <
> > > >> > > > > > > > dimas@apache.org>
> > > >> > > > > > > > > > > wrote:
> > > >> > > > > > > > > > >
> > > >> > > > > > > > > > > > Hi All,
> > > >> > > > > > > > > > > >
> > > >> > > > > > > > > > > > WDYT about another sync call next week?
> > > >> > > > > > > > > > > >
> > > >> > > > > > > > > > > > Thanks,
> > > >> > > > > > > > > > > > Dmitri.
> > > >> > > > > > > > > > > >
> > > >> > > > > > > > > > > > On Wed, May 6, 2026 at 5:29 PM Dmitri
> > > Bourlatchkov <
> > > >> > > > > > > > dimas@apache.org
> > > >> > > > > > > > > >
> > > >> > > > > > > > > > > > wrote:
> > > >> > > > > > > > > > > >
> > > >> > > > > > > > > > > > > Hi EJ,
> > > >> > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > Thanks for the summary! It covers what we
> > > >> discussed
> > > >> > in
> > > >> > > > the
> > > >> > > > > > > > meeting
> > > >> > > > > > > > > > very
> > > >> > > > > > > > > > > > > well, IMHO.
> > > >> > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > Looking forward to concrete PRs :)
> > > >> > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > Cheers,
> > > >> > > > > > > > > > > > > Dmitri.
> > > >> > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > On Wed, May 6, 2026 at 5:08 PM EJ Wang <
> > > >> > > > > > > > > > ej.wang.dev.interest@gmail.com
> > > >> > > > > > > > > > > >
> > > >> > > > > > > > > > > > > wrote:
> > > >> > > > > > > > > > > > >
> > > >> > > > > > > > > > > > >> Hi folks,
> > > >> > > > > > > > > > > > >>
> > > >> > > > > > > > > > > > >> We had a community sync earlier, thanks JB
> > for
> > > >> > > > scheduling
> > > >> > > > > > it.
> > > >> > > > > > > > > Notes
> > > >> > > > > > > > > > > from
> > > >> > > > > > > > > > > > >> the first metrics architecture sync (May 6,
> > > >> 10-11am
> > > >> > > PT).
> > > >> > > > > > > > > Discussion
> > > >> > > > > > > > > > > doc
> > > >> > > > > > > > > > > > >> with per-section status:
> > > >> > > > > > > > > > > > >>
> > > >> > > > > > > > > > > > >>
> > > >> > > > > > > > > > > >
> > > >> > > > > > > > > > >
> > > >> > > > > > > > > >
> > > >> > > > > > > > >
> > > >> > > > > > > >
> > > >> > > > > > >
> > > >> > > > > >
> > > >> > > > >
> > > >> > > >
> > > >> > >
> > > >> >
> > > >>
> > >
> > https://docs.google.com/document/d/100h7c4damrUzVuquYbBHM0EvA4LSWuW2IT2dN_7nYVA/edit?tab=t.0
> > > >> > > > > > > > > > > > >>
> > > >> > > > > > > > > > > > >> *The meeting covered both topics from the
> > doc.
> > > >> > > > > > Direction-level
> > > >> > > > > > > > > > > alignment
> > > >> > > > > > > > > > > > >> was reached on the headline pieces; details
> > > >> remain
> > > >> > for
> > > >> > > > PR
> > > >> > > > > > > review
> > > >> > > > > > > > > or
> > > >> > > > > > > > > > > > >> follow-up sessions.*
> > > >> > > > > > > > > > > > >>
> > > >> > > > > > > > > > > > >> *Topic 1 — Persistence schema redesign*
> > > >> > > > > > > > > > > > >> Idea-level alignment on consolidating
> > per-type
> > > >> > tables
> > > >> > > > > > > > > > > > >> (scan_metrics_report,
> > > >> > > > > > > > > > > > >> commit_metrics_report) into a single
> > > >> metrics_report
> > > >> > > > table.
> > > >> > > > > > The
> > > >> > > > > > > > > > > > motivating
> > > >> > > > > > > > > > > > >> cost is the surface area added by every new
> > > >> metric
> > > >> > > type
> > > >> > > > > > today:
> > > >> > > > > > > > new
> > > >> > > > > > > > > > > > table,
> > > >> > > > > > > > > > > > >> SPI method, record class, model, converter,
> > > >> schema
> > > >> > > > > > migration.
> > > >> > > > > > > > > > > > >>
> > > >> > > > > > > > > > > > >> Most schema details are deferred to the
> > schema
> > > >> PR. A
> > > >> > > few
> > > >> > > > > > > > specific
> > > >> > > > > > > > > > > points
> > > >> > > > > > > > > > > > >> came up:
> > > >> > > > > > > > > > > > >> •   metric_schema_version: Yufei prefers
> > > dropping
> > > >> > it,
> > > >> > > > > since
> > > >> > > > > > > > there
> > > >> > > > > > > > > is
> > > >> > > > > > > > > > > no
> > > >> > > > > > > > > > > > >> spec-level concept of metrics versioning
> > today
> > > >> and
> > > >> > it
> > > >> > > is
> > > >> > > > > > hard
> > > >> > > > > > > to
> > > >> > > > > > > > > > > define
> > > >> > > > > > > > > > > > >> unilaterally. Robert prefers keeping it,
> > given
> > > >> IRC
> > > >> > v2
> > > >> > > is
> > > >> > > > > > > coming
> > > >> > > > > > > > > and
> > > >> > > > > > > > > > > the
> > > >> > > > > > > > > > > > >> schema should be considered against its
> > likely
> > > >> > shape;
> > > >> > > > > Robert
> > > >> > > > > > > > also
> > > >> > > > > > > > > > > raised
> > > >> > > > > > > > > > > > >> how to differentiate various payload formats
> > if
> > > >> any.
> > > >> > > > EJ's
> > > >> > > > > > read
> > > >> > > > > > > > is
> > > >> > > > > > > > > > that
> > > >> > > > > > > > > > > > >> this
> > > >> > > > > > > > > > > > >> is a two-way-door decision. We can start
> > > without
> > > >> the
> > > >> > > > > field,
> > > >> > > > > > > and
> > > >> > > > > > > > if
> > > >> > > > > > > > > > IRC
> > > >> > > > > > > > > > > > v2
> > > >> > > > > > > > > > > > >> changes the shape we would likely roll a
> > > >> > corresponding
> > > >> > > > new
> > > >> > > > > > > > schema
> > > >> > > > > > > > > > > > anyway,
> > > >> > > > > > > > > > > > >> which is not particularly costly.
> > > >> > > > > > > > > > > > >> •   Payload format: Robert pointed out that
> > > >> future
> > > >> > > > formats
> > > >> > > > > > > > beyond
> > > >> > > > > > > > > > JSON
> > > >> > > > > > > > > > > > may
> > > >> > > > > > > > > > > > >> be worth supporting. The exact shape is
> > > deferred
> > > >> to
> > > >> > > the
> > > >> > > > > > schema
> > > >> > > > > > > > > > > > discussion.
> > > >> > > > > > > > > > > > >> •   Partition strategy: Anand suggested
> > monthly
> > > >> > > > > partitioning
> > > >> > > > > > > > based
> > > >> > > > > > > > > > on
> > > >> > > > > > > > > > > > his
> > > >> > > > > > > > > > > > >> experience as potentially helpful at scale.
> > > >> > > > > > > > > > > > >>
> > > >> > > > > > > > > > > > >> *Topic 2 — Where metrics ingestion and
> > storage
> > > >> > belong*
> > > >> > > > > > > > > > > > >> Idea-level alignment that metrics should be a
> > > >> > > separated
> > > >> > > > > SPI
> > > >> > > > > > > from
> > > >> > > > > > > > > the
> > > >> > > > > > > > > > > > >> entity
> > > >> > > > > > > > > > > > >> persistence stack. Two reasons surfaced: (a)
> > > >> > workloads
> > > >> > > > and
> > > >> > > > > > > > > > capability
> > > >> > > > > > > > > > > > >> requirements diverge enough that coupling
> > them
> > > >> > creates
> > > >> > > > > > > > artificial
> > > >> > > > > > > > > > > > >> constraints, and (b) admin experience
> > improves
> > > >> when
> > > >> > > > > metrics
> > > >> > > > > > > has
> > > >> > > > > > > > > its
> > > >> > > > > > > > > > > own
> > > >> > > > > > > > > > > > >> bootstrap, retention, and lifecycle. Dmitri
> > > noted
> > > >> > > > Polaris
> > > >> > > > > > > being
> > > >> > > > > > > > a
> > > >> > > > > > > > > > > > platform
> > > >> > > > > > > > > > > > >> should have the flexibility to support
> > > different
> > > >> > > > > persistence
> > > >> > > > > > > > > > backends
> > > >> > > > > > > > > > > > per
> > > >> > > > > > > > > > > > >> concern, and pointed to a concrete next step
> > of
> > > >> > > > separating
> > > >> > > > > > the
> > > >> > > > > > > > > JDBC
> > > >> > > > > > > > > > > > >> bootstrap for metrics from the metastore
> > > >> bootstrap.
> > > >> > > > Robert
> > > >> > > > > > > > > proposed
> > > >> > > > > > > > > > an
> > > >> > > > > > > > > > > > >> additional UX extension: detect an
> > > unbootstrapped
> > > >> > > > metrics
> > > >> > > > > > > store
> > > >> > > > > > > > on
> > > >> > > > > > > > > > > first
> > > >> > > > > > > > > > > > >> use and auto-bootstrap rather than requiring
> > an
> > > >> > > explicit
> > > >> > > > > > > manual
> > > >> > > > > > > > > > > > bootstrap
> > > >> > > > > > > > > > > > >> step.
> > > >> > > > > > > > > > > > >> The meeting also confirmed that Polaris
> > metrics
> > > >> can
> > > >> > > > start
> > > >> > > > > > > small
> > > >> > > > > > > > > and
> > > >> > > > > > > > > > > stay
> > > >> > > > > > > > > > > > >> Iceberg-focused. Naming and persistence
> > schema
> > > >> can
> > > >> > > lean
> > > >> > > > > > > > > > > > Iceberg-specific.
> > > >> > > > > > > > > > > > >> If a future expansion to generic-table
> > metrics
> > > or
> > > >> > > > > > operational
> > > >> > > > > > > > > > metrics
> > > >> > > > > > > > > > > > >> arrives, an abstraction layer can be built on
> > > >> top of
> > > >> > > the
> > > >> > > > > > > Iceberg
> > > >> > > > > > > > > > > metrics
> > > >> > > > > > > > > > > > >> reporter at that point. Robert remains on the
> > > >> fence
> > > >> > > and
> > > >> > > > > > would
> > > >> > > > > > > > > prefer
> > > >> > > > > > > > > > > > >> something more generic but did not block the
> > > >> > > direction;
> > > >> > > > > > > Dmitri's
> > > >> > > > > > > > > > read
> > > >> > > > > > > > > > > > was
> > > >> > > > > > > > > > > > >> that the proposed framework already has
> > enough
> > > >> > > > flexibility
> > > >> > > > > > to
> > > >> > > > > > > > > absorb
> > > >> > > > > > > > > > > > >> future
> > > >> > > > > > > > > > > > >> expansion.
> > > >> > > > > > > > > > > > >>
> > > >> > > > > > > > > > > > >> The Trade-offs and Proposed structure
> > sections
> > > in
> > > >> > the
> > > >> > > > doc
> > > >> > > > > > were
> > > >> > > > > > > > not
> > > >> > > > > > > > > > > > >> reviewed
> > > >> > > > > > > > > > > > >> in detail. They remain open for either the
> > next
> > > >> sync
> > > >> > > or
> > > >> > > > PR
> > > >> > > > > > > > review.
> > > >> > > > > > > > > > > > >>
> > > >> > > > > > > > > > > > >> *Cross-cutting alignment — battery-included
> > > plus
> > > >> > > > > pluggable*
> > > >> > > > > > > > > > > > >> A common philosophy emerged from the
> > > discussion.
> > > >> EJ
> > > >> > > > > > summarized
> > > >> > > > > > > > it
> > > >> > > > > > > > > > as:
> > > >> > > > > > > > > > > > >> Polaris should provide a battery-included UX
> > > for
> > > >> > > > beginners
> > > >> > > > > > and
> > > >> > > > > > > > the
> > > >> > > > > > > > > > > > >> flexibility for advanced users to swap the
> > > >> included
> > > >> > > > > battery
> > > >> > > > > > > for
> > > >> > > > > > > > > > > > something
> > > >> > > > > > > > > > > > >> more powerful or tailored to their use case.
> > > The
> > > >> SPI
> > > >> > > > > design
> > > >> > > > > > > > needs
> > > >> > > > > > > > > to
> > > >> > > > > > > > > > > > >> enable
> > > >> > > > > > > > > > > > >> both.
> > > >> > > > > > > > > > > > >>
> > > >> > > > > > > > > > > > >> The inputs that shaped this framing:
> > > >> > > > > > > > > > > > >> •   Anand described how his team uses the
> > > current
> > > >> > > > metrics
> > > >> > > > > > > > > > persistence
> > > >> > > > > > > > > > > > >> (three metrics consumers in v1.4).
> > > >> > > > > > > > > > > > >> •   Yufei raised Grafana and dashboard
> > > >> integrations
> > > >> > > as a
> > > >> > > > > > > > > destination
> > > >> > > > > > > > > > > use
> > > >> > > > > > > > > > > > >> case beyond the default.
> > > >> > > > > > > > > > > > >> •   Robert called out that the current design
> > > is
> > > >> > more
> > > >> > > > > > > > > JDBC-focused.
> > > >> > > > > > > > > > > > >>
> > > >> > > > > > > > > > > > >> Two concrete instances:
> > > >> > > > > > > > > > > > >> •   Async metrics intake: Yufei's initial
> > > >> position
> > > >> > was
> > > >> > > > > that
> > > >> > > > > > > > async
> > > >> > > > > > > > > > > should
> > > >> > > > > > > > > > > > >> largely live on the producer side and there
> > is
> > > >> not
> > > >> > > much
> > > >> > > > > > > Polaris
> > > >> > > > > > > > > can
> > > >> > > > > > > > > > > do.
> > > >> > > > > > > > > > > > >> Robert suggested a Polaris-side default is
> > > doable
> > > >> > via
> > > >> > > > > > Vert.x.
> > > >> > > > > > > > > Dmitri
> > > >> > > > > > > > > > > > >> agreed
> > > >> > > > > > > > > > > > >> the direction is worth exploring. The meeting
> > > >> > > converged
> > > >> > > > > on a
> > > >> > > > > > > > > > > > >> battery-included default (likely
> > Vert.x-backed)
> > > >> with
> > > >> > > an
> > > >> > > > > SPI
> > > >> > > > > > > > shape
> > > >> > > > > > > > > > that
> > > >> > > > > > > > > > > > >> lets
> > > >> > > > > > > > > > > > >> power users route to a more scalable backend
> > > >> > > (k8s-hosted
> > > >> > > > > > > queue,
> > > >> > > > > > > > > AWS
> > > >> > > > > > > > > > > SQS,
> > > >> > > > > > > > > > > > >> etc.).
> > > >> > > > > > > > > > > > >> •   Pluggable destinations: combining Yufei's
> > > >> > > dashboard
> > > >> > > > > use
> > > >> > > > > > > case
> > > >> > > > > > > > > > with
> > > >> > > > > > > > > > > > >> Robert's JDBC-focused call-out, the meeting
> > > >> agreed
> > > >> > the
> > > >> > > > SPI
> > > >> > > > > > > > should
> > > >> > > > > > > > > be
> > > >> > > > > > > > > > > > >> structured for multiple sinks so integrations
> > > >> become
> > > >> > > > impl
> > > >> > > > > > > > choices
> > > >> > > > > > > > > > > rather
> > > >> > > > > > > > > > > > >> than architectural changes.
> > > >> > > > > > > > > > > > >>
> > > >> > > > > > > > > > > > >> The battery-included default is most likely
> > to
> > > >> use
> > > >> > the
> > > >> > > > > > > existing
> > > >> > > > > > > > > > > > >> JDBC-backed
> > > >> > > > > > > > > > > > >> approach.
> > > >> > > > > > > > > > > > >>
> > > >> > > > > > > > > > > > >> *Direction (idea-level alignment)*
> > > >> > > > > > > > > > > > >> •   Single metrics_report table consolidating
> > > >> > per-type
> > > >> > > > > > > metrics,
> > > >> > > > > > > > > > > > replacing
> > > >> > > > > > > > > > > > >> scan_metrics_report and commit_metrics_report
> > > >> > > > > > > > > > > > >> •   Iceberg-focused naming and schema for
> > now,
> > > >> > revisit
> > > >> > > > if
> > > >> > > > > > > > > > > generic-table
> > > >> > > > > > > > > > > > or
> > > >> > > > > > > > > > > > >> operational metrics arrive
> > > >> > > > > > > > > > > > >> •   Metrics persistence as a separated SPI,
> > not
> > > >> on
> > > >> > > > > > > > BasePersistence
> > > >> > > > > > > > > > > > >> •   Bootstrap path separated for metrics,
> > > >> > independent
> > > >> > > of
> > > >> > > > > > > > metastore
> > > >> > > > > > > > > > > > >> bootstrap
> > > >> > > > > > > > > > > > >> •   "Battery-included plus pluggable" as the
> > > SPI
> > > >> > > design
> > > >> > > > > > > > philosophy
> > > >> > > > > > > > > > > > >>
> > > >> > > > > > > > > > > > >> *Open items*
> > > >> > > > > > > > > > > > >> •   Schema details: metric_schema_version,
> > > >> payload
> > > >> > > > format,
> > > >> > > > > > IRC
> > > >> > > > > > > > v2
> > > >> > > > > > > > > > > > >> forward-compat shape
> > > >> > > > > > > > > > > > >> •   SPI design details — full review either
> > in
> > > >> the
> > > >> > > next
> > > >> > > > > sync
> > > >> > > > > > > or
> > > >> > > > > > > > in
> > > >> > > > > > > > > > the
> > > >> > > > > > > > > > > > >> corresponding PR
> > > >> > > > > > > > > > > > >> •   Schema refactor PR ownership
> > > >> > > > > > > > > > > > >>
> > > >> > > > > > > > > > > > >> *Action items*
> > > >> > > > > > > > > > > > >> •   EJ to take a first stab at the SPI design
> > > and
> > > >> > > > > > potentially
> > > >> > > > > > > > > > partner
> > > >> > > > > > > > > > > > with
> > > >> > > > > > > > > > > > >> Anand to incorporate the lessons learned from
> > > the
> > > >> > > > existing
> > > >> > > > > > > > > reporter
> > > >> > > > > > > > > > > and
> > > >> > > > > > > > > > > > >> persistence work.
> > > >> > > > > > > > > > > > >> •   Schema refactor PR ownership is not yet
> > > >> decided.
> > > >> > > If
> > > >> > > > > > anyone
> > > >> > > > > > > > is
> > > >> > > > > > > > > > > > >> interested in driving it, reply on this
> > thread.
> > > >> > > > > > > > > > > > >> •   JB to schedule the next sync, tentatively
> > > in
> > > >> two
> > > >> > > > > weeks.
> > > >> > > > > > > > > > > > >>
> > > >> > > > > > > > > > > > >> -ej
> > > >> > > > > > > > > > > > >>
> > > >> > > > > > > > > > > > >> On Mon, Apr 27, 2026 at 3:07 PM EJ Wang <
> > > >> > > > > > > > > > > ej.wang.dev.interest@gmail.com
> > > >> > > > > > > > > > > > >
> > > >> > > > > > > > > > > > >> wrote:
> > > >> > > > > > > > > > > > >>
> > > >> > > > > > > > > > > > >> > Thanks Yufei for the +1.
> > > >> > > > > > > > > > > > >> >
> > > >> > > > > > > > > > > > >> > JB, could you help add a biweekly metrics
> > > >> > > architecture
> > > >> > > > > > sync
> > > >> > > > > > > to
> > > >> > > > > > > > > the
> > > >> > > > > > > > > > > > >> Polaris
> > > >> > > > > > > > > > > > >> > community calendar? I'm thinking Thursdays
> > at
> > > >> > 9-10am
> > > >> > > > PT,
> > > >> > > > > > on
> > > >> > > > > > > > the
> > > >> > > > > > > > > > > > >> off-weeks
> > > >> > > > > > > > > > > > >> > from the community meeting (starting May
> > 7),
> > > 60
> > > >> > > > minutes.
> > > >> > > > > > > > > > > > >> >
> > > >> > > > > > > > > > > > >> > Here's a rough agenda to work through over
> > > the
> > > >> > first
> > > >> > > > few
> > > >> > > > > > > > > sessions,
> > > >> > > > > > > > > > > > >> grouped
> > > >> > > > > > > > > > > > >> > by priority:
> > > >> > > > > > > > > > > > >> >
> > > >> > > > > > > > > > > > >> > *First: foundational direction*
> > > >> > > > > > > > > > > > >> >
> > > >> > > > > > > > > > > > >> > 1.  MetricsPersistence: public SPI or
> > > internal
> > > >> > > > > > > implementation
> > > >> > > > > > > > > > > detail?
> > > >> > > > > > > > > > > > >> >    •   Marked @Beta, javadoc calls it a
> > > >> "Service
> > > >> > > > > Provider
> > > >> > > > > > > > > > > Interface",
> > > >> > > > > > > > > > > > >> but
> > > >> > > > > > > > > > > > >> > only one consumer
> > (JdbcBasePersistenceImpl),
> > > >> lives
> > > >> > > on
> > > >> > > > > > > > > > > BasePersistence.
> > > >> > > > > > > > > > > > >> If
> > > >> > > > > > > > > > > > >> > demoted to a private helper inside a
> > > persisting
> > > >> > > > reporter
> > > >> > > > > > > impl,
> > > >> > > > > > > > > > most
> > > >> > > > > > > > > > > > >> > downstream design decisions become
> > > >> implementation
> > > >> > > > > details
> > > >> > > > > > > > rather
> > > >> > > > > > > > > > > than
> > > >> > > > > > > > > > > > >> > contract questions.
> > > >> > > > > > > > > > > > >> >
> > > >> > > > > > > > > > > > >> > 2.  Persistence schema redesign
> > > >> > > > > > > > > > > > >> >    •   Current two-table layout
> > > >> > > (scan_metrics_report,
> > > >> > > > > > > > > > > > >> > commit_metrics_report) with ~25 flattened
> > > >> columns
> > > >> > > > each.
> > > >> > > > > > > Every
> > > >> > > > > > > > > new
> > > >> > > > > > > > > > > > metric
> > > >> > > > > > > > > > > > >> > type requires a new table, SPI method,
> > record
> > > >> > class,
> > > >> > > > > > model,
> > > >> > > > > > > > > > > converter,
> > > >> > > > > > > > > > > > >> and
> > > >> > > > > > > > > > > > >> > schema migration. Direction to explore:
> > > single
> > > >> > table
> > > >> > > > > with
> > > >> > > > > > > > > > > metric_type
> > > >> > > > > > > > > > > > >> enum,
> > > >> > > > > > > > > > > > >> > schema_version, and JSON payload column.
> > > >> > > > > > > > > > > > >> >
> > > >> > > > > > > > > > > > >> > *Second: design details once direction is
> > > set*
> > > >> > > > > > > > > > > > >> >
> > > >> > > > > > > > > > > > >> > 3.  Partition key strategy
> > > >> > > > > > > > > > > > >> >    •   Single-table design means scan
> > metrics
> > > >> at
> > > >> > > scale
> > > >> > > > > > will
> > > >> > > > > > > > have
> > > >> > > > > > > > > > > high
> > > >> > > > > > > > > > > > >> > write concurrency per table. Schema needs
> > to
> > > >> > expose
> > > >> > > > > enough
> > > >> > > > > > > > > > structure
> > > >> > > > > > > > > > > > for
> > > >> > > > > > > > > > > > >> > backends to shard by entity or time range.
> > > >> > > > > > > > > > > > >> >
> > > >> > > > > > > > > > > > >> > 4.  Read/write path consistency
> > > >> > > > > > > > > > > > >> >    •   Writes go through
> > > PolarisMetricsManager
> > > >> on
> > > >> > > > > > > > > > MetaStoreManager.
> > > >> > > > > > > > > > > > >> Reads
> > > >> > > > > > > > > > > > >> > bypass MetaStoreManager and go straight to
> > > >> > > > > > BasePersistence,
> > > >> > > > > > > > > > > excluding
> > > >> > > > > > > > > > > > >> > non-JDBC backends from the read API.
> > > >> > > > > > > > > > > > >> >
> > > >> > > > > > > > > > > > >> > *Third: cleanup and alignment*
> > > >> > > > > > > > > > > > >> >
> > > >> > > > > > > > > > > > >> > 5.  PolarisMetricsReporter naming
> > > >> > > > > > > > > > > > >> >    •   Only handles IRC
> > > >> (ScanReport/CommitReport),
> > > >> > > > > doesn't
> > > >> > > > > > > > cover
> > > >> > > > > > > > > > > > generic
> > > >> > > > > > > > > > > > >> > tables or operational metrics. Name is
> > > broader
> > > >> > than
> > > >> > > > > scope.
> > > >> > > > > > > > > > > > >> >
> > > >> > > > > > > > > > > > >> > 6.  PolarisMetricsManager facade
> > passthrough
> > > >> > > > > > > > > > > > >> >    •   Entire default method is
> > > >> > > > > > > > > > > > >> callCtx.getMetaStore().writeScanReport().
> > > >> > > > > > > > > > > > >> > Zero logic, passes Level 1 straight through
> > > to
> > > >> > Level
> > > >> > > > 3.
> > > >> > > > > > Same
> > > >> > > > > > > > > > > > >> anti-pattern
> > > >> > > > > > > > > > > > >> > as PolarisEventManager.
> > > >> > > > > > > > > > > > >> >
> > > >> > > > > > > > > > > > >> > 7.  Iceberg community alignment
> > > >> > > > > > > > > > > > >> >    •   Payload-type extension needs
> > > discussion
> > > >> on
> > > >> > > > > > > dev@iceberg.
> > > >> > > > > > > > > > > > >> obelix74's
> > > >> > > > > > > > > > > > >> > Feb thread got zero replies. Needs a
> > > committer
> > > >> > > voice.
> > > >> > > > > > > > > > > > >> >
> > > >> > > > > > > > > > > > >> > Lets confirm prioritization in the first
> > > >> session.
> > > >> > > > > > > > > > > > >> >
> > > >> > > > > > > > > > > > >> > -ej
> > > >> > > > > > > > > > > > >> >
> > > >> > > > > > > > > > > > >> > On Tue, Apr 21, 2026 at 3:18 PM Yufei Gu <
> > > >> > > > > > > > flyrain000@gmail.com>
> > > >> > > > > > > > > > > > wrote:
> > > >> > > > > > > > > > > > >> >
> > > >> > > > > > > > > > > > >> >> Thanks everyone for continuing to drive
> > this
> > > >> > > > forward. I
> > > >> > > > > > > agree
> > > >> > > > > > > > > > that
> > > >> > > > > > > > > > > > the
> > > >> > > > > > > > > > > > >> >> problem is getting complex enough that a
> > > more
> > > >> > > > > structured
> > > >> > > > > > > > > > discussion
> > > >> > > > > > > > > > > > >> would
> > > >> > > > > > > > > > > > >> >> help.
> > > >> > > > > > > > > > > > >> >>
> > > >> > > > > > > > > > > > >> >> +1 on setting up a biweekly sync for the
> > > >> metrics
> > > >> > > > > > > > architecture.
> > > >> > > > > > > > > > I’m
> > > >> > > > > > > > > > > > >> happy
> > > >> > > > > > > > > > > > >> >> to
> > > >> > > > > > > > > > > > >> >> join.
> > > >> > > > > > > > > > > > >> >>
> > > >> > > > > > > > > > > > >> >> Yufei
> > > >> > > > > > > > > > > > >> >>
> > > >> > > > > > > > > > > > >> >>
> > > >> > > > > > > > > > > > >> >> On Tue, Apr 21, 2026 at 2:34 PM EJ Wang <
> > > >> > > > > > > > > > > > >> ej.wang.dev.interest@gmail.com>
> > > >> > > > > > > > > > > > >> >> wrote:
> > > >> > > > > > > > > > > > >> >>
> > > >> > > > > > > > > > > > >> >> > Also, I've been looking more closely at
> > > the
> > > >> > > > > > *persistence
> > > >> > > > > > > > > schema
> > > >> > > > > > > > > > > in
> > > >> > > > > > > > > > > > >> the
> > > >> > > > > > > > > > > > >> >> > current metrics work*, and I think
> > > there's a
> > > >> > > > > structural
> > > >> > > > > > > > > > rigidity
> > > >> > > > > > > > > > > > >> problem
> > > >> > > > > > > > > > > > >> >> > worth raising before the shape gets
> > locked
> > > >> in.
> > > >> > > > > > > > > > > > >> >> >
> > > >> > > > > > > > > > > > >> >> > Right now we have two separate tables
> > > >> > > > > > > (scan_metrics_report
> > > >> > > > > > > > > and
> > > >> > > > > > > > > > > > >> >> > commit_metrics_report), each with ~25
> > > >> flattened
> > > >> > > > > columns
> > > >> > > > > > > > that
> > > >> > > > > > > > > > > > directly
> > > >> > > > > > > > > > > > >> >> > mirror the Iceberg report fields. The
> > SPI
> > > >> > follows
> > > >> > > > the
> > > >> > > > > > > same
> > > >> > > > > > > > > > split:
> > > >> > > > > > > > > > > > >> >> > writeScanReport and writeCommitReport as
> > > >> > separate
> > > >> > > > > > > methods,
> > > >> > > > > > > > > with
> > > >> > > > > > > > > > > > >> per-type
> > > >> > > > > > > > > > > > >> >> > record classes, converters, and model
> > > >> objects.
> > > >> > > *The
> > > >> > > > > > > > practical
> > > >> > > > > > > > > > > cost:
> > > >> > > > > > > > > > > > >> >> > adding a new metric type (operational
> > > >> metrics,
> > > >> > > for
> > > >> > > > > > > example)
> > > >> > > > > > > > > > > > requires
> > > >> > > > > > > > > > > > >> a
> > > >> > > > > > > > > > > > >> >> new
> > > >> > > > > > > > > > > > >> >> > table, a new SPI method, a new record
> > > >> class, a
> > > >> > > new
> > > >> > > > > > model
> > > >> > > > > > > > > > class, a
> > > >> > > > > > > > > > > > new
> > > >> > > > > > > > > > > > >> >> > converter branch, and a schema
> > migration*.
> > > >> > > That's a
> > > >> > > > > lot
> > > >> > > > > > > of
> > > >> > > > > > > > > > > surface
> > > >> > > > > > > > > > > > >> area
> > > >> > > > > > > > > > > > >> >> > for what should be "one more kind of
> > > >> metric."
> > > >> > > > > > > > > > > > >> >> >
> > > >> > > > > > > > > > > > >> >> > *My bias* would be toward a single
> > metrics
> > > >> > table
> > > >> > > > with
> > > >> > > > > > *a
> > > >> > > > > > > > > typed
> > > >> > > > > > > > > > > JSON
> > > >> > > > > > > > > > > > >> >> > payload*. Something like: metric_type
> > > >> (enum),
> > > >> > > > > > entity_id,
> > > >> > > > > > > > > > > > >> >> > table_identifier, snapshot_id
> > (nullable),
> > > >> > > > > received_ts,
> > > >> > > > > > > > > > > > >> schema_version,
> > > >> > > > > > > > > > > > >> >> and
> > > >> > > > > > > > > > > > >> >> > a payload column for the metric-specific
> > > >> data.
> > > >> > > The
> > > >> > > > > > > > > metric_type
> > > >> > > > > > > > > > +
> > > >> > > > > > > > > > > > >> >> > schema_version pair gives us a
> > > >> > forward-compatible
> > > >> > > > > > > contract
> > > >> > > > > > > > > for
> > > >> > > > > > > > > > > the
> > > >> > > > > > > > > > > > >> >> payload
> > > >> > > > > > > > > > > > >> >> > shape. Adding a new metric type becomes
> > an
> > > >> enum
> > > >> > > > value
> > > >> > > > > > > and a
> > > >> > > > > > > > > > > payload
> > > >> > > > > > > > > > > > >> >> schema,
> > > >> > > > > > > > > > > > >> >> > not a schema migration. One thing I
> > think
> > > we
> > > >> > need
> > > >> > > > to
> > > >> > > > > be
> > > >> > > > > > > > > > > deliberate
> > > >> > > > > > > > > > > > >> >> about is
> > > >> > > > > > > > > > > > >> >> > the partition key design. If all metric
> > > >> types
> > > >> > > land
> > > >> > > > in
> > > >> > > > > > one
> > > >> > > > > > > > > > table,
> > > >> > > > > > > > > > > > scan
> > > >> > > > > > > > > > > > >> >> > metrics at scale (high concurrency, high
> > > >> > > frequency
> > > >> > > > > > across
> > > >> > > > > > > > > many
> > > >> > > > > > > > > > > > >> tables)
> > > >> > > > > > > > > > > > >> >> > could easily create hot partitions. We'd
> > > >> want
> > > >> > the
> > > >> > > > > > > > persistence
> > > >> > > > > > > > > > > layer
> > > >> > > > > > > > > > > > >> to
> > > >> > > > > > > > > > > > >> >> be
> > > >> > > > > > > > > > > > >> >> > able to shard by entity or time range,
> > and
> > > >> that
> > > >> > > > means
> > > >> > > > > > the
> > > >> > > > > > > > > > logical
> > > >> > > > > > > > > > > > >> schema
> > > >> > > > > > > > > > > > >> >> > needs to expose enough structure for
> > > >> backends
> > > >> > to
> > > >> > > > > > > partition
> > > >> > > > > > > > > on.
> > > >> > > > > > > > > > I
> > > >> > > > > > > > > > > > >> don't
> > > >> > > > > > > > > > > > >> >> > think the current flattened layout gives
> > > us
> > > >> > that.
> > > >> > > > > > > > > > > > >> >> >
> > > >> > > > > > > > > > > > >> >> > This is getting complex enough that I
> > > don't
> > > >> > think
> > > >> > > > > > ad-hoc
> > > >> > > > > > > > > PR/ML
> > > >> > > > > > > > > > > > >> threads
> > > >> > > > > > > > > > > > >> >> > will converge well. *Would people be
> > open
> > > >> to a
> > > >> > > > > biweekly
> > > >> > > > > > > > sync
> > > >> > > > > > > > > > for
> > > >> > > > > > > > > > > > >> metrics
> > > >> > > > > > > > > > > > >> >> > architecture?* I think 30 minutes every
> > > two
> > > >> > weeks
> > > >> > > > > with
> > > >> > > > > > > > > > interested
> > > >> > > > > > > > > > > > >> >> parties
> > > >> > > > > > > > > > > > >> >> > would be enough to work through the
> > > schema,
> > > >> SPI
> > > >> > > > > shape,
> > > >> > > > > > > and
> > > >> > > > > > > > > read
> > > >> > > > > > > > > > > API
> > > >> > > > > > > > > > > > >> >> design
> > > >> > > > > > > > > > > > >> >> > together. Happy to help set that up.
> > > >> > > > > > > > > > > > >> >> >
> > > >> > > > > > > > > > > > >> >> > -ej
> > > >> > > > > > > > > > > > >> >> >
> > > >> > > > > > > > > > > > >> >> > On Mon, Apr 20, 2026 at 2:19 PM EJ Wang
> > <
> > > >> > > > > > > > > > > > >> ej.wang.dev.interest@gmail.com
> > > >> > > > > > > > > > > > >> >> >
> > > >> > > > > > > > > > > > >> >> > wrote:
> > > >> > > > > > > > > > > > >> >> >
> > > >> > > > > > > > > > > > >> >> >> Reviewed #4115, left a comment on the
> > > code
> > > >> > > > > > organization
> > > >> > > > > > > > > side.
> > > >> > > > > > > > > > > > >> >> >>
> > > >> > > > > > > > > > > > >> >> >> One thing stood out: the metrics write
> > > path
> > > >> > > enters
> > > >> > > > > > > through
> > > >> > > > > > > > > > > > >> >> >> PolarisMetricsManager on
> > > MetaStoreManager,
> > > >> but
> > > >> > > the
> > > >> > > > > new
> > > >> > > > > > > > read
> > > >> > > > > > > > > > path
> > > >> > > > > > > > > > > > >> >> bypasses
> > > >> > > > > > > > > > > > >> >> >> MetaStoreManager entirely and goes
> > > >> straight to
> > > >> > > > > > > > > BasePersistence
> > > >> > > > > > > > > > > via
> > > >> > > > > > > > > > > > >> >> >> callContext.getMetaStore(). That means
> > > the
> > > >> > read
> > > >> > > > API
> > > >> > > > > > only
> > > >> > > > > > > > > works
> > > >> > > > > > > > > > > for
> > > >> > > > > > > > > > > > >> >> backends
> > > >> > > > > > > > > > > > >> >> >> that implement BasePersistence. NoSQL
> > and
> > > >> > remote
> > > >> > > > > > > backends
> > > >> > > > > > > > > > can't
> > > >> > > > > > > > > > > > >> >> participate.
> > > >> > > > > > > > > > > > >> >> >>
> > > >> > > > > > > > > > > > >> >> >> Stepping back, I think the metrics
> > > >> subsystem
> > > >> > is
> > > >> > > > > > growing
> > > >> > > > > > > > into
> > > >> > > > > > > > > > > > >> something
> > > >> > > > > > > > > > > > >> >> >> real (write + read + REST API + AuthZ +
> > > >> > > > pagination)
> > > >> > > > > > *but
> > > >> > > > > > > > the
> > > >> > > > > > > > > > > > >> >> persistence
> > > >> > > > > > > > > > > > >> >> >> side is split across two layers in a
> > way
> > > >> > that's
> > > >> > > > hard
> > > >> > > > > > to
> > > >> > > > > > > > > > > extend*. I
> > > >> > > > > > > > > > > > >> put
> > > >> > > > > > > > > > > > >> >> >> together two diagrams to show what I
> > mean
> > > >> (my
> > > >> > > best
> > > >> > > > > > > > effort).
> > > >> > > > > > > > > > > > >> >> >>
> > > >> > > > > > > > > > > > >> >> >> *Current state* (Diagram 1): three
> > > >> interfaces
> > > >> > at
> > > >> > > > > three
> > > >> > > > > > > > > > different
> > > >> > > > > > > > > > > > >> >> levels.
> > > >> > > > > > > > > > > > >> >> >> The engine-facing SPI
> > > >> (PolarisMetricsReporter)
> > > >> > > is
> > > >> > > > > > clean.
> > > >> > > > > > > > But
> > > >> > > > > > > > > > > > >> >> >> PolarisMetricsManager on
> > MetaStoreManager
> > > >> is a
> > > >> > > > > > > passthrough
> > > >> > > > > > > > > to
> > > >> > > > > > > > > > > > >> >> >> MetricsPersistence on BasePersistence.
> > > The
> > > >> > @Beta
> > > >> > > > > > > > annotation
> > > >> > > > > > > > > > and
> > > >> > > > > > > > > > > > SPI
> > > >> > > > > > > > > > > > >> >> javadoc
> > > >> > > > > > > > > > > > >> >> >> are on the BasePersistence layer, while
> > > the
> > > >> > > actual
> > > >> > > > > > > > extension
> > > >> > > > > > > > > > > > points
> > > >> > > > > > > > > > > > >> >> >> (PolarisMetricsReporter,
> > > >> > PolarisMetricsManager)
> > > >> > > > > carry
> > > >> > > > > > no
> > > >> > > > > > > > > > > stability
> > > >> > > > > > > > > > > > >> >> >> annotation. The write path goes through
> > > the
> > > >> > > > > > > > MetaStoreManager
> > > >> > > > > > > > > > > > layer,
> > > >> > > > > > > > > > > > >> the
> > > >> > > > > > > > > > > > >> >> >> read path doesn't.
> > > >> > > > > > > > > > > > >> >> >>
> > > >> > > > > > > > > > > > >> >> >> *What I envision* (Diagram 2): two SPIs
> > > at
> > > >> two
> > > >> > > > > levels.
> > > >> > > > > > > > > > > > >> >> >> PolarisMetricsReporter stays as the
> > > >> > > engine-facing
> > > >> > > > > SPI.
> > > >> > > > > > > > > > > > >> >> >> PolarisMetricsManager becomes the
> > > >> > backend-facing
> > > >> > > > SPI
> > > >> > > > > > > with
> > > >> > > > > > > > > both
> > > >> > > > > > > > > > > > write
> > > >> > > > > > > > > > > > >> >> and
> > > >> > > > > > > > > > > > >> >> >> read methods at the MetaStoreManager
> > > level,
> > > >> > > where
> > > >> > > > > any
> > > >> > > > > > > > > backend
> > > >> > > > > > > > > > > > (JDBC,
> > > >> > > > > > > > > > > > >> >> NoSQL,
> > > >> > > > > > > > > > > > >> >> >> remote) can implement them.
> > > >> MetricsPersistence
> > > >> > > on
> > > >> > > > > > > > > > > BasePersistence
> > > >> > > > > > > > > > > > >> goes
> > > >> > > > > > > > > > > > >> >> >> away. Where metrics actually land is an
> > > >> > > > > implementation
> > > >> > > > > > > > > detail,
> > > >> > > > > > > > > > > > not a
> > > >> > > > > > > > > > > > >> >> core
> > > >> > > > > > > > > > > > >> >> >> interface.
> > > >> > > > > > > > > > > > >> >> >>
> > > >> > > > > > > > > > > > >> >> >> *Minor naming thing*:
> > > >> PolarisMetricsReporter
> > > >> > is
> > > >> > > > > > broader
> > > >> > > > > > > > than
> > > >> > > > > > > > > > > what
> > > >> > > > > > > > > > > > it
> > > >> > > > > > > > > > > > >> >> >> actually handles. It only accepts
> > Iceberg
> > > >> REST
> > > >> > > > > Catalog
> > > >> > > > > > > > > metrics
> > > >> > > > > > > > > > > > >> >> (ScanReport,
> > > >> > > > > > > > > > > > >> >> >> CommitReport via MetricsReport).
> > Generic
> > > >> table
> > > >> > > > > metrics
> > > >> > > > > > > or
> > > >> > > > > > > > > > > > >> operational
> > > >> > > > > > > > > > > > >> >> >> metrics aren't in scope. Not blocking,
> > > but
> > > >> > worth
> > > >> > > > > > noting
> > > >> > > > > > > if
> > > >> > > > > > > > > the
> > > >> > > > > > > > > > > > >> metrics
> > > >> > > > > > > > > > > > >> >> >> surface expands.
> > > >> > > > > > > > > > > > >> >> >>
> > > >> > > > > > > > > > > > >> >> >> *Rough sketch of how to get there*:
> > > >> > > > > > > > > > > > >> >> >>  1.  Add read methods to
> > > >> PolarisMetricsManager
> > > >> > > > > > > > > > (listScanReports,
> > > >> > > > > > > > > > > > >> >> >> listCommitReports) with default no-op,
> > > >> same as
> > > >> > > the
> > > >> > > > > > > > existing
> > > >> > > > > > > > > > > write
> > > >> > > > > > > > > > > > >> >> methods.
> > > >> > > > > > > > > > > > >> >> >> (Probably make PolarisMetricsManager
> > more
> > > >> > > explicit
> > > >> > > > > on
> > > >> > > > > > > > being
> > > >> > > > > > > > > > > > Iceberg
> > > >> > > > > > > > > > > > >> >> >> specific like package name or class
> > name
> > > >> etc.)
> > > >> > > > > > > > > > > > >> >> >>  2.  Wire MetricsReportsService through
> > > >> > > > > > MetaStoreManager
> > > >> > > > > > > > > > instead
> > > >> > > > > > > > > > > > of
> > > >> > > > > > > > > > > > >> >> >> callContext.getMetaStore().
> > > >> > > > > > > > > > > > >> >> >>  3.  Extract metrics persistence from
> > > >> > > > > > > > > JdbcBasePersistenceImpl
> > > >> > > > > > > > > > > into
> > > >> > > > > > > > > > > > >> its
> > > >> > > > > > > > > > > > >> >> >> own class. That file carries ~7
> > > >> > > responsibilities,
> > > >> > > > > > > metrics
> > > >> > > > > > > > > > being
> > > >> > > > > > > > > > > > one
> > > >> > > > > > > > > > > > >> of
> > > >> > > > > > > > > > > > >> >> them.
> > > >> > > > > > > > > > > > >> >> >>  4.  Remove MetricsPersistence from
> > > >> > > > BasePersistence.
> > > >> > > > > > > > > > > > >> >> >>
> > > >> > > > > > > > > > > > >> >> >> *None of this needs to happen in #4115.
> > > >> But if
> > > >> > > the
> > > >> > > > > > > > direction
> > > >> > > > > > > > > > > makes
> > > >> > > > > > > > > > > > >> >> sense,
> > > >> > > > > > > > > > > > >> >> >> it would be good to align before the
> > > >> metrics
> > > >> > > > surface
> > > >> > > > > > > grows
> > > >> > > > > > > > > > > > further.
> > > >> > > > > > > > > > > > >> >> Curious
> > > >> > > > > > > > > > > > >> >> >> what others think.*
> > > >> > > > > > > > > > > > >> >> >>
> > > >> > > > > > > > > > > > >> >> >> *My mental model note*: Level 1
> > > >> > > MetaStoreManager;
> > > >> > > > > > level
> > > >> > > > > > > 2
> > > >> > > > > > > > > > > > >> transactional
> > > >> > > > > > > > > > > > >> >> >> persistence; level 3 base persistence
> > > >> > > > > > > > > > > > >> >> >>
> > > >> > > > > > > > > > > > >> >> >> Diagram 1
> > > >> > > > > > > > > > > > >> >> >> <
> > > >> > > > > > > > > > > > >> >>
> > > >> > > > > > > > > > > > >>
> > > >> > > > > > > > > > > >
> > > >> > > > > > > > > > >
> > > >> > > > > > > > > >
> > > >> > > > > > > > >
> > > >> > > > > > > >
> > > >> > > > > > >
> > > >> > > > > >
> > > >> > > > >
> > > >> > > >
> > > >> > >
> > > >> >
> > > >>
> > >
> > https://www.plantuml.com/plantuml/uml/bLHDR-Cs4BthLmpIYupw0zbkKQ1r3M-S7Bp8xhhM7WCOb3IM65EaGD9EX2RzxHrHb4CxRelwa4YSDu_lpOVcnZ9jzvM8BBS2uGjQpJC3dtHMSekPtMk44IpsMgEqa5XcCOhCZikQQLP1pR8TAp2n3ILhmZDP20m0fcIvUkAoW2qJXd9z1bpToO9BX3WXu0ucy5rpgGPNm0nW5_epUWtm2Ue3pn3kMOFQmKntGZW0BYtgBSi8k5A2QMwybJNMIbFiGSR9QZc4nUqIvikStF0jHprua5C-amge42aNt3R0f5JaaoivdV2Pkqbx4hee4ymOkBh5BTiB-_uIeGeo8zL8rPsPl4DktdEiK1jkB1NdZCRbrSTecDe_mlHbF0wvBmCkaOH5_S8a_TTTKI6-nmCAkEw4LpxsZ-LbYLKQFKMNOgf_wuM7_bV9gOer5SYMMksBSWXFcbi49KNZXNLicwfe3TETC7gPdPqI7uBcHMb1RSzYq34c6PDUM9mn8HRsUTZEiDBve3NjVZumBj0U7SS37mGO7vcwtiK-_pU7U7L_f-digo9YbhSwIfMRwIITKGXbxdIUTCGF1SeCJxloKsU-3k9ddRbX1eDq1q_fx1JbBGT0glVyXimDuP4TQ5qpCAmnGEj2s_6n5mtn1z-97-63itFQZLPO1Ev2tu_WF7Ju-VPc0Skg5bYXxBhkY1xpD7EM_7fyflSpIsqMgVth5xhVr4eQxWQ8enaSAJQSG16yFSDuJ798rrcXr_3n-lfdk7icQjEBmFujL7AodiP_Y4Z7-YxvtZNs4zMgpNTl6tF8sglyPsmqchrjvQ-m-aP94r-TwCA2Ka8upPJZwtvSpoYCXkYMZU2NXvRMBfq9P3i3Le4VAZUAlUZ_oPKsxPgY0Q_BSKLkyr9bhQhQrJjo_x3TPlIB0DPjnMfcIoYP0QaYw1a0fTKDr8fB6ntNuvmoL1ZGkXa69Njh43zf9GiGxHQrA_jDYWRSzF5--WmTVrN97_Sm8LbLUy_lGBmLanJjFkDlGkRqjA_4tm00
> > > >> > > > > > > > > > > > >> >> >
> > > >> > > > > > > > > > > > >> >> >> :
> > > >> > > > > > > > > > > > >> >> >>
> > > >> > > > > > > > > > > > >> >> >> [image: image.png]
> > > >> > > > > > > > > > > > >> >> >>
> > > >> > > > > > > > > > > > >> >> >> Diagram 2
> > > >> > > > > > > > > > > > >> >> >> <
> > > >> > > > > > > > > > > > >> >>
> > > >> > > > > > > > > > > > >>
> > > >> > > > > > > > > > > >
> > > >> > > > > > > > > > >
> > > >> > > > > > > > > >
> > > >> > > > > > > > >
> > > >> > > > > > > >
> > > >> > > > > > >
> > > >> > > > > >
> > > >> > > > >
> > > >> > > >
> > > >> > >
> > > >> >
> > > >>
> > >
> > https://www.plantuml.com/plantuml/uml/VLLDR-8m4BtdLupO2sWBLVU8AaGB7AXAbssGzb896SS9RXqxjKqBwkv_tt7iV43fdaZYDpFlpRmnOsE9jhjSH9PRmM31hERKm8scMsuPjJlDe0yheZDc8RR4iYWoBrmMH9CS2a9VICPYUy1OZN0YCy5Q0BCbYNhdCeEK28En8G8wCvbnoQ0R8_05Bc6bkLIz3X03p1zzH7zR-9ZfDquPt9C3qoNCX2yV4G2NbkcKu5jdgGJHt0GbZwnG6i-UP3TUpk5gM6Ldqke350eZUqzoCft3U9xWHvxoa5-7K4nF1J46EbEMafsmdrCBbQ44gVggy18IZrn_ph5asd1ZiIKdQSgueZvjXrQFSFrdC3YN-nXmBacxbGiYyLVxLaBtdhqn0LSzdBDhqQtQoOJeGyad3z0lUqnYgpGB6Ns8oVyta00Dy_WnX0tIOZ8v6SYxHll1TrH6aejAik-mh-AphVFCwSUQqFypElag5QRGFDjQKEd96K1P8QP41c9TzA_IIQyvdAWyv_RSiS3skb0_EzDDkK2v5xWF6MiGFlvhpFLcD2Dq2pml14gaF67eQkmd8gulDoC4kSOu6KVpkvlUJg1RTbWISU40RdBUUS_9XfRZ2dwxm_SW8LYFISgm_MnlDQ6M9P1gbKEc4X-2pH_FvJCkCqm9pbVjD6LrwdLeOrDWfOaqc8Wh9BE85oNKxkNQ6o4yGRy_Eae0G_G8tZv81d3bHDB23WOdisohVr3nh_j6lbSjbNaLRTc8UgtPbAU1J_tygOfZX9DWEJeHDvYx-qmSi5FgNLPZwHrHcUsncGQ5-skhUclpE5fo4ounpFauYrUbkU6ccfnxMvitwag4IyerhTxj8In_Oj1bDO4pQru674loYrGlULHLEGCjwJJ8gDoVZR8MxO4BT3IzRvIcAQKezC6xpziGnTyImrfEGyJI_OcKfgtxIvnTqFEMS17L9Z-jsARN5FmTheP7HtSdtOMT0B4GY2FYHXxgQmMtj2bRqiLFGapiVe1_QVKDrkqXcm83aFEXnMYCZ-xlyHy
> > > >> > > > > > > > > > > > >> >> >
> > > >> > > > > > > > > > > > >> >> >> :
> > > >> > > > > > > > > > > > >> >> >> [image: image.png]
> > > >> > > > > > > > > > > > >> >> >>
> > > >> > > > > > > > > > > > >> >> >>  -ej
> > > >> > > > > > > > > > > > >> >> >>
> > > >> > > > > > > > > > > > >> >> >> On Wed, Apr 15, 2026 at 8:22 AM Dmitri
> > > >> > > > Bourlatchkov
> > > >> > > > > <
> > > >> > > > > > > > > > > > >> dimas@apache.org>
> > > >> > > > > > > > > > > > >> >> >> wrote:
> > > >> > > > > > > > > > > > >> >> >>
> > > >> > > > > > > > > > > > >> >> >>> Hi All,
> > > >> > > > > > > > > > > > >> >> >>>
> > > >> > > > > > > > > > > > >> >> >>> Heads up: The current state of PR
> > [4115]
> > > >> > looks
> > > >> > > > > pretty
> > > >> > > > > > > > solid
> > > >> > > > > > > > > > to
> > > >> > > > > > > > > > > > me.
> > > >> > > > > > > > > > > > >> I
> > > >> > > > > > > > > > > > >> >> >>> believe this PR is approaching a
> > > mergeable
> > > >> > > > > condition.
> > > >> > > > > > > > > > > > >> >> >>>
> > > >> > > > > > > > > > > > >> >> >>> Please post your reviews if you have
> > any
> > > >> > > > comments.
> > > >> > > > > > > > > > > > >> >> >>>
> > > >> > > > > > > > > > > > >> >> >>> [4115]
> > > >> > > > https://github.com/apache/polaris/pull/4115
> > > >> > > > > > > > > > > > >> >> >>>
> > > >> > > > > > > > > > > > >> >> >>> Thanks,
> > > >> > > > > > > > > > > > >> >> >>> Dmitri.
> > > >> > > > > > > > > > > > >> >> >>>
> > > >> > > > > > > > > > > > >> >> >>> On Tue, Mar 3, 2026 at 3:29 PM Anand
> > > Kumar
> > > >> > > > Sankaran
> > > >> > > > > > via
> > > >> > > > > > > > > dev <
> > > >> > > > > > > > > > > > >> >> >>> dev@polaris.apache.org> wrote:
> > > >> > > > > > > > > > > > >> >> >>>
> > > >> > > > > > > > > > > > >> >> >>> > Hi Yufei and Dmitri,
> > > >> > > > > > > > > > > > >> >> >>> >
> > > >> > > > > > > > > > > > >> >> >>> > Here is a proposal for the REST
> > > >> endpoints
> > > >> > for
> > > >> > > > > > metrics
> > > >> > > > > > > > and
> > > >> > > > > > > > > > > > events.
> > > >> > > > > > > > > > > > >> >> >>> >
> > > >> > > > > > > > > > > > >> >> >>> >
> > > >> > > > > > https://github.com/apache/polaris/pull/3924/changes
> > > >> > > > > > > > > > > > >> >> >>> >
> > > >> > > > > > > > > > > > >> >> >>> > I did not see any precursors for
> > > >> raising a
> > > >> > PR
> > > >> > > > for
> > > >> > > > > > > > > > proposals,
> > > >> > > > > > > > > > > so
> > > >> > > > > > > > > > > > >> >> trying
> > > >> > > > > > > > > > > > >> >> >>> > this.  Please let me know what you
> > > >> think.
> > > >> > > > > > > > > > > > >> >> >>> >
> > > >> > > > > > > > > > > > >> >> >>> > -
> > > >> > > > > > > > > > > > >> >> >>> > Anand
> > > >> > > > > > > > > > > > >> >> >>> >
> > > >> > > > > > > > > > > > >> >> >>> > From: Anand Kumar Sankaran <
> > > >> > > > > > > anand.sankaran@workday.com
> > > >> > > > > > > > >
> > > >> > > > > > > > > > > > >> >> >>> > Date: Monday, March 2, 2026 at
> > > 10:25 AM
> > > >> > > > > > > > > > > > >> >> >>> > To: dev@polaris.apache.org <
> > > >> > > > > dev@polaris.apache.org
> > > >> > > > > > >
> > > >> > > > > > > > > > > > >> >> >>> > Subject: Re: Polaris Telemetry and
> > > Audit
> > > >> > > Trail
> > > >> > > > > > > > > > > > >> >> >>> >
> > > >> > > > > > > > > > > > >> >> >>> > About the REST API, based on my use
> > > >> cases:
> > > >> > > > > > > > > > > > >> >> >>> >
> > > >> > > > > > > > > > > > >> >> >>> >
> > > >> > > > > > > > > > > > >> >> >>> >   1.
> > > >> > > > > > > > > > > > >> >> >>> > I want to be able to query commit
> > > >> metrics
> > > >> > to
> > > >> > > > > track
> > > >> > > > > > > > files
> > > >> > > > > > > > > > > added
> > > >> > > > > > > > > > > > /
> > > >> > > > > > > > > > > > >> >> >>> removed
> > > >> > > > > > > > > > > > >> >> >>> > per commit, along with record
> > counts.
> > > >> The
> > > >> > > > > ingestion
> > > >> > > > > > > > > > pipeline
> > > >> > > > > > > > > > > > that
> > > >> > > > > > > > > > > > >> >> >>> writes
> > > >> > > > > > > > > > > > >> >> >>> > this data is owned by us and we are
> > > >> > > guaranteed
> > > >> > > > to
> > > >> > > > > > > write
> > > >> > > > > > > > > > this
> > > >> > > > > > > > > > > > >> >> >>> information
> > > >> > > > > > > > > > > > >> >> >>> > for each write.
> > > >> > > > > > > > > > > > >> >> >>> >   2.
> > > >> > > > > > > > > > > > >> >> >>> > I want to be able to query scan
> > > metrics
> > > >> for
> > > >> > > > > read. I
> > > >> > > > > > > > > > > understand
> > > >> > > > > > > > > > > > >> >> clients
> > > >> > > > > > > > > > > > >> >> >>> do
> > > >> > > > > > > > > > > > >> >> >>> > not fulfill this requirement.
> > > >> > > > > > > > > > > > >> >> >>> >   3.
> > > >> > > > > > > > > > > > >> >> >>> > I want to be able to query the
> > events
> > > >> table
> > > >> > > > > (events
> > > >> > > > > > > are
> > > >> > > > > > > > > > > > >> persisted) -
> > > >> > > > > > > > > > > > >> >> >>> this
> > > >> > > > > > > > > > > > >> >> >>> > may supersede #2, I am not sure yet.
> > > >> > > > > > > > > > > > >> >> >>> >
> > > >> > > > > > > > > > > > >> >> >>> > All this information is in the JDBC
> > > >> based
> > > >> > > > > > persistence
> > > >> > > > > > > > > model
> > > >> > > > > > > > > > > and
> > > >> > > > > > > > > > > > >> is
> > > >> > > > > > > > > > > > >> >> >>> > persisted in the metastore. I
> > > currently
> > > >> > don’t
> > > >> > > > > have
> > > >> > > > > > a
> > > >> > > > > > > > need
> > > >> > > > > > > > > > to
> > > >> > > > > > > > > > > > >> query
> > > >> > > > > > > > > > > > >> >> >>> > prometheus or open telemetry. I do
> > > >> publish
> > > >> > > some
> > > >> > > > > > > events
> > > >> > > > > > > > to
> > > >> > > > > > > > > > > > >> Prometheus
> > > >> > > > > > > > > > > > >> >> >>> and
> > > >> > > > > > > > > > > > >> >> >>> > they are forwarded to our dashboards
> > > >> > > elsewhere.
> > > >> > > > > > > > > > > > >> >> >>> >
> > > >> > > > > > > > > > > > >> >> >>> > About the CLI utilities, I meant the
> > > >> admin
> > > >> > > user
> > > >> > > > > > > > > utilities.
> > > >> > > > > > > > > > In
> > > >> > > > > > > > > > > > >> one of
> > > >> > > > > > > > > > > > >> >> >>> the
> > > >> > > > > > > > > > > > >> >> >>> > earliest drafts of my proposal,
> > > Prashant
> > > >> > > > > mentioned
> > > >> > > > > > > that
> > > >> > > > > > > > > the
> > > >> > > > > > > > > > > > >> metrics
> > > >> > > > > > > > > > > > >> >> >>> tables
> > > >> > > > > > > > > > > > >> >> >>> > can grow indefinitely and that a
> > > similar
> > > >> > > > problem
> > > >> > > > > > > exists
> > > >> > > > > > > > > > with
> > > >> > > > > > > > > > > > the
> > > >> > > > > > > > > > > > >> >> events
> > > >> > > > > > > > > > > > >> >> >>> > table as well. We discussed that
> > > >> cleaning
> > > >> > up
> > > >> > > of
> > > >> > > > > old
> > > >> > > > > > > > > records
> > > >> > > > > > > > > > > > from
> > > >> > > > > > > > > > > > >> >> both
> > > >> > > > > > > > > > > > >> >> >>> > metrics tables and events tables can
> > > be
> > > >> > done
> > > >> > > > via
> > > >> > > > > a
> > > >> > > > > > > CLI
> > > >> > > > > > > > > > > utility.
> > > >> > > > > > > > > > > > >> >> >>> >
> > > >> > > > > > > > > > > > >> >> >>> > I see that Yufei has covered the
> > > >> discussion
> > > >> > > > about
> > > >> > > > > > > > > > > datasources.
> > > >> > > > > > > > > > > > >> >> >>> >
> > > >> > > > > > > > > > > > >> >> >>> > -
> > > >> > > > > > > > > > > > >> >> >>> > Anand
> > > >> > > > > > > > > > > > >> >> >>> >
> > > >> > > > > > > > > > > > >> >> >>> >
> > > >> > > > > > > > > > > > >> >> >>> >
> > > >> > > > > > > > > > > > >> >> >>> > From: Yufei Gu <
> > flyrain000@gmail.com>
> > > >> > > > > > > > > > > > >> >> >>> > Date: Friday, February 27, 2026 at
> > > >> 9:54 PM
> > > >> > > > > > > > > > > > >> >> >>> > To: dev@polaris.apache.org <
> > > >> > > > > dev@polaris.apache.org
> > > >> > > > > > >
> > > >> > > > > > > > > > > > >> >> >>> > Subject: Re: Polaris Telemetry and
> > > Audit
> > > >> > > Trail
> > > >> > > > > > > > > > > > >> >> >>> >
> > > >> > > > > > > > > > > > >> >> >>> > This Message Is From an External
> > > Sender
> > > >> > > > > > > > > > > > >> >> >>> > This message came from outside your
> > > >> > > > organization.
> > > >> > > > > > > > > > > > >> >> >>> > Report Suspicious<
> > > >> > > > > > > > > > > > >> >> >>> >
> > > >> > > > > > > > > > > > >> >> >>>
> > > >> > > > > > > > > > > > >> >>
> > > >> > > > > > > > > > > > >>
> > > >> > > > > > > > > > > >
> > > >> > > > > > > > > > >
> > > >> > > > > > > > > >
> > > >> > > > > > > > >
> > > >> > > > > > > >
> > > >> > > > > > >
> > > >> > > > > >
> > > >> > > > >
> > > >> > > >
> > > >> > >
> > > >> >
> > > >>
> > >
> > https://us-phishalarm-ewt.proofpoint.com/EWT/v1/Iz9xO38YGHZK!YhNDZABkHi1B699ote2uMwpOZw8i0QMCGO2Szc-HshuABGhGvwPJcymE6G2oUUxtS8xDkSrtGTPm_I3QnVDHoLMk50m9v8z_nZKTkd-bnVUbreF1u0WnfV_X5eYevZl_$
> > > >> > > > > > > > > > > > >> >> >>> > >
> > > >> > > > > > > > > > > > >> >> >>> >
> > > >> > > > > > > > > > > > >> >> >>> >
> > > >> > > > > > > > > > > > >> >> >>> > As I mentioned in
> > > >> > > > > > > > > > > > >> >> >>> >
> > > >> > > > > > > > > > > > >> >> >>>
> > > >> > > > > > > > > > > > >> >>
> > > >> > > > > > > > > > > > >>
> > > >> > > > > > > > > > > >
> > > >> > > > > > > > > > >
> > > >> > > > > > > > > >
> > > >> > > > > > > > >
> > > >> > > > > > > >
> > > >> > > > > > >
> > > >> > > > > >
> > > >> > > > >
> > > >> > > >
> > > >> > >
> > > >> >
> > > >>
> > >
> > https://urldefense.com/v3/__https://github.com/apache/polaris/issues/3890__;!!Iz9xO38YGHZK!5EuyFFkk3vhRWVIRvQAWBSQfpJkTMA9HxugzDwXmN0LPPqhEFxYkFRGVhtb8AqUwXtDh2OplcMnbMDHKOxrvDU0$
> > > >> > > > > > > > > > > > >> >> >>> ,
> > > >> > > > > > > > > > > > >> >> >>> > supporting
> > > >> > > > > > > > > > > > >> >> >>> > multiple data sources is not a
> > trivial
> > > >> > > change.
> > > >> > > > I
> > > >> > > > > > > would
> > > >> > > > > > > > > > > strongly
> > > >> > > > > > > > > > > > >> >> >>> recommend
> > > >> > > > > > > > > > > > >> >> >>> > starting with a design document to
> > > >> > carefully
> > > >> > > > > > evaluate
> > > >> > > > > > > > the
> > > >> > > > > > > > > > > > >> >> architectural
> > > >> > > > > > > > > > > > >> >> >>> > implications and long term impact.
> > > >> > > > > > > > > > > > >> >> >>> >
> > > >> > > > > > > > > > > > >> >> >>> > A REST endpoint to query metrics
> > seems
> > > >> > > > reasonable
> > > >> > > > > > > given
> > > >> > > > > > > > > the
> > > >> > > > > > > > > > > > >> current
> > > >> > > > > > > > > > > > >> >> >>> JDBC
> > > >> > > > > > > > > > > > >> >> >>> > based persistence model. That said,
> > we
> > > >> may
> > > >> > > also
> > > >> > > > > > > > consider
> > > >> > > > > > > > > > > > >> alternative
> > > >> > > > > > > > > > > > >> >> >>> > storage models. For example, if we
> > > later
> > > >> > > adopt
> > > >> > > > a
> > > >> > > > > > time
> > > >> > > > > > > > > > series
> > > >> > > > > > > > > > > > >> system
> > > >> > > > > > > > > > > > >> >> >>> such as
> > > >> > > > > > > > > > > > >> >> >>> > Prometheus to store metrics, the
> > query
> > > >> > model
> > > >> > > > and
> > > >> > > > > > > access
> > > >> > > > > > > > > > > > patterns
> > > >> > > > > > > > > > > > >> >> would
> > > >> > > > > > > > > > > > >> >> >>> be
> > > >> > > > > > > > > > > > >> >> >>> > fundamentally different. Designing
> > the
> > > >> REST
> > > >> > > API
> > > >> > > > > > > without
> > > >> > > > > > > > > > > > >> considering
> > > >> > > > > > > > > > > > >> >> >>> these
> > > >> > > > > > > > > > > > >> >> >>> > potential evolutions may limit
> > > >> flexibility.
> > > >> > > I'd
> > > >> > > > > > > suggest
> > > >> > > > > > > > > to
> > > >> > > > > > > > > > > > start
> > > >> > > > > > > > > > > > >> >> with
> > > >> > > > > > > > > > > > >> >> >>> the
> > > >> > > > > > > > > > > > >> >> >>> > use case.
> > > >> > > > > > > > > > > > >> >> >>> >
> > > >> > > > > > > > > > > > >> >> >>> > Yufei
> > > >> > > > > > > > > > > > >> >> >>> >
> > > >> > > > > > > > > > > > >> >> >>> >
> > > >> > > > > > > > > > > > >> >> >>> > On Fri, Feb 27, 2026 at 3:42 PM
> > Dmitri
> > > >> > > > > > Bourlatchkov <
> > > >> > > > > > > > > > > > >> >> dimas@apache.org>
> > > >> > > > > > > > > > > > >> >> >>> > wrote:
> > > >> > > > > > > > > > > > >> >> >>> >
> > > >> > > > > > > > > > > > >> >> >>> > > Hi Anand,
> > > >> > > > > > > > > > > > >> >> >>> > >
> > > >> > > > > > > > > > > > >> >> >>> > > Sharing my view... subject to
> > > >> discussion:
> > > >> > > > > > > > > > > > >> >> >>> > >
> > > >> > > > > > > > > > > > >> >> >>> > > 1. Adding non-IRC REST API to
> > > Polaris
> > > >> is
> > > >> > > > > > perfectly
> > > >> > > > > > > > > fine.
> > > >> > > > > > > > > > > > >> >> >>> > >
> > > >> > > > > > > > > > > > >> >> >>> > > Figuring out specific endpoint
> > URIs
> > > >> and
> > > >> > > > > payloads
> > > >> > > > > > > > might
> > > >> > > > > > > > > > > > require
> > > >> > > > > > > > > > > > >> a
> > > >> > > > > > > > > > > > >> >> few
> > > >> > > > > > > > > > > > >> >> >>> > > roundtrips, so opening a separate
> > > >> thread
> > > >> > > for
> > > >> > > > > that
> > > >> > > > > > > > might
> > > >> > > > > > > > > > be
> > > >> > > > > > > > > > > > >> best.
> > > >> > > > > > > > > > > > >> >> >>> > > Contributors commonly create
> > Google
> > > >> Docs
> > > >> > > for
> > > >> > > > > new
> > > >> > > > > > > API
> > > >> > > > > > > > > > > > proposals
> > > >> > > > > > > > > > > > >> too
> > > >> > > > > > > > > > > > >> >> >>> (they
> > > >> > > > > > > > > > > > >> >> >>> > > fairly easy to update as the email
> > > >> > > discussion
> > > >> > > > > > > > > > progresses).
> > > >> > > > > > > > > > > > >> >> >>> > >
> > > >> > > > > > > > > > > > >> >> >>> > > There was a suggestion to try
> > > Markdown
> > > >> > > (with
> > > >> > > > > PRs)
> > > >> > > > > > > for
> > > >> > > > > > > > > > > > proposals
> > > >> > > > > > > > > > > > >> >> [1]
> > > >> > > > > > > > > > > > >> >> >>> ...
> > > >> > > > > > > > > > > > >> >> >>> > > feel free to give it a try if you
> > > are
> > > >> > > > > comfortable
> > > >> > > > > > > > with
> > > >> > > > > > > > > > > that.
> > > >> > > > > > > > > > > > >> >> >>> > >
> > > >> > > > > > > > > > > > >> >> >>> > > 2. Could you clarify whether you
> > > mean
> > > >> end
> > > >> > > > user
> > > >> > > > > > > > > utilities
> > > >> > > > > > > > > > or
> > > >> > > > > > > > > > > > >> admin
> > > >> > > > > > > > > > > > >> >> >>> user
> > > >> > > > > > > > > > > > >> >> >>> > > utilities? In the latter case
> > those
> > > >> might
> > > >> > > be
> > > >> > > > > more
> > > >> > > > > > > > > > suitable
> > > >> > > > > > > > > > > > for
> > > >> > > > > > > > > > > > >> the
> > > >> > > > > > > > > > > > >> >> >>> Admin
> > > >> > > > > > > > > > > > >> >> >>> > > CLI (java) not the Python CLI,
> > IMHO.
> > > >> > > > > > > > > > > > >> >> >>> > >
> > > >> > > > > > > > > > > > >> >> >>> > > Why would these utilities be
> > common
> > > >> with
> > > >> > > > > events?
> > > >> > > > > > > > IMHO,
> > > >> > > > > > > > > > > event
> > > >> > > > > > > > > > > > >> use
> > > >> > > > > > > > > > > > >> >> >>> cases
> > > >> > > > > > > > > > > > >> >> >>> > are
> > > >> > > > > > > > > > > > >> >> >>> > > distinct from scan/commit metrics.
> > > >> > > > > > > > > > > > >> >> >>> > >
> > > >> > > > > > > > > > > > >> >> >>> > > 3. I'd prefer separating metrics
> > > >> > > persistence
> > > >> > > > > from
> > > >> > > > > > > > > > MetaStore
> > > >> > > > > > > > > > > > >> >> >>> persistence
> > > >> > > > > > > > > > > > >> >> >>> > at
> > > >> > > > > > > > > > > > >> >> >>> > > the code level, so that they could
> > > be
> > > >> > mixed
> > > >> > > > and
> > > >> > > > > > > > matched
> > > >> > > > > > > > > > > > >> >> >>> independently.
> > > >> > > > > > > > > > > > >> >> >>> > The
> > > >> > > > > > > > > > > > >> >> >>> > > separate datasource question will
> > > >> become
> > > >> > a
> > > >> > > > > > > non-issue
> > > >> > > > > > > > > with
> > > >> > > > > > > > > > > > that
> > > >> > > > > > > > > > > > >> >> >>> approach,
> > > >> > > > > > > > > > > > >> >> >>> > I
> > > >> > > > > > > > > > > > >> >> >>> > > guess.
> > > >> > > > > > > > > > > > >> >> >>> > >
> > > >> > > > > > > > > > > > >> >> >>> > > The rationale for separating scan
> > > >> metrics
> > > >> > > and
> > > >> > > > > > > > metastore
> > > >> > > > > > > > > > > > >> >> persistence
> > > >> > > > > > > > > > > > >> >> >>> is
> > > >> > > > > > > > > > > > >> >> >>> > that
> > > >> > > > > > > > > > > > >> >> >>> > > "cascading deletes" between them
> > are
> > > >> > hardly
> > > >> > > > > ever
> > > >> > > > > > > > > > required.
> > > >> > > > > > > > > > > > >> >> >>> Furthermore,
> > > >> > > > > > > > > > > > >> >> >>> > the
> > > >> > > > > > > > > > > > >> >> >>> > > data and query patterns are very
> > > >> > different
> > > >> > > so
> > > >> > > > > > > > different
> > > >> > > > > > > > > > > > >> >> technologies
> > > >> > > > > > > > > > > > >> >> >>> > might
> > > >> > > > > > > > > > > > >> >> >>> > > be beneficial in each case.
> > > >> > > > > > > > > > > > >> >> >>> > >
> > > >> > > > > > > > > > > > >> >> >>> > > [1]
> > > >> > > > > > > > > > > > >> >> >>> >
> > > >> > > > > > > > > > > > >> >> >>>
> > > >> > > > > > > > > > > > >> >>
> > > >> > > > > > > > > > > > >>
> > > >> > > > > > > > > > > >
> > > >> > > > > > > > > > >
> > > >> > > > > > > > > >
> > > >> > > > > > > > >
> > > >> > > > > > > >
> > > >> > > > > > >
> > > >> > > > > >
> > > >> > > > >
> > > >> > > >
> > > >> > >
> > > >> >
> > > >>
> > >
> > https://urldefense.com/v3/__https://lists.apache.org/thread/yto2wp982t43h1mqjwnslswhws5z47cy__;!!Iz9xO38YGHZK!5EuyFFkk3vhRWVIRvQAWBSQfpJkTMA9HxugzDwXmN0LPPqhEFxYkFRGVhtb8AqUwXtDh2OplcMnbMDHKxYDakNU$
> > > >> > > > > > > > > > > > >> >> >>> > >
> > > >> > > > > > > > > > > > >> >> >>> > > Cheers,
> > > >> > > > > > > > > > > > >> >> >>> > > Dmitri.
> > > >> > > > > > > > > > > > >> >> >>> > >
> > > >> > > > > > > > > > > > >> >> >>> > > On Fri, Feb 27, 2026 at 6:19 PM
> > > Anand
> > > >> > Kumar
> > > >> > > > > > > Sankaran
> > > >> > > > > > > > > via
> > > >> > > > > > > > > > > dev
> > > >> > > > > > > > > > > > <
> > > >> > > > > > > > > > > > >> >> >>> > > dev@polaris.apache.org> wrote:
> > > >> > > > > > > > > > > > >> >> >>> > >
> > > >> > > > > > > > > > > > >> >> >>> > > > Thanks all. This PR is merged
> > now.
> > > >> > > > > > > > > > > > >> >> >>> > > >
> > > >> > > > > > > > > > > > >> >> >>> > > > Here are the follow-up features
> > /
> > > >> work
> > > >> > > > > needed.
> > > >> > > > > > > > These
> > > >> > > > > > > > > > > were
> > > >> > > > > > > > > > > > >> all
> > > >> > > > > > > > > > > > >> >> >>> part of
> > > >> > > > > > > > > > > > >> >> >>> > > the
> > > >> > > > > > > > > > > > >> >> >>> > > > merged PR at some point in time
> > > and
> > > >> > were
> > > >> > > > > > removed
> > > >> > > > > > > to
> > > >> > > > > > > > > > > reduce
> > > >> > > > > > > > > > > > >> >> scope.
> > > >> > > > > > > > > > > > >> >> >>> > > >
> > > >> > > > > > > > > > > > >> >> >>> > > > Please let me know what you
> > think.
> > > >> > > > > > > > > > > > >> >> >>> > > >
> > > >> > > > > > > > > > > > >> >> >>> > > >
> > > >> > > > > > > > > > > > >> >> >>> > > >   1.  A REST API to paginate
> > > through
> > > >> > > table
> > > >> > > > > > > metrics.
> > > >> > > > > > > > > > This
> > > >> > > > > > > > > > > > >> will be
> > > >> > > > > > > > > > > > >> >> >>> > non-IRC
> > > >> > > > > > > > > > > > >> >> >>> > > > standard addition.
> > > >> > > > > > > > > > > > >> >> >>> > > >   2.  Utilities for managing old
> > > >> > records,
> > > >> > > > > > should
> > > >> > > > > > > be
> > > >> > > > > > > > > > > common
> > > >> > > > > > > > > > > > >> with
> > > >> > > > > > > > > > > > >> >> >>> events.
> > > >> > > > > > > > > > > > >> >> >>> > > > There was some discussion that
> > it
> > > >> > belongs
> > > >> > > > to
> > > >> > > > > > the
> > > >> > > > > > > > CLI.
> > > >> > > > > > > > > > > > >> >> >>> > > >   3.  Separate datasource
> > > (metrics,
> > > >> > > events,
> > > >> > > > > > even
> > > >> > > > > > > > > other
> > > >> > > > > > > > > > > > >> tables?).
> > > >> > > > > > > > > > > > >> >> >>> > > >
> > > >> > > > > > > > > > > > >> >> >>> > > >
> > > >> > > > > > > > > > > > >> >> >>> > > > Anything else?
> > > >> > > > > > > > > > > > >> >> >>> > > >
> > > >> > > > > > > > > > > > >> >> >>> > > > -
> > > >> > > > > > > > > > > > >> >> >>> > > > Anand
> > > >> > > > > > > > > > > > >> >> >>> > > >
> > > >> > > > > > > > > > > > >> >> >>> > > >
> > > >> > > > > > > > > > > > >> >> >>> > >
> > > >> > > > > > > > > > > > >> >> >>> >
> > > >> > > > > > > > > > > > >> >> >>> >
> > > >> > > > > > > > > > > > >> >> >>>
> > > >> > > > > > > > > > > > >> >> >>
> > > >> > > > > > > > > > > > >> >>
> > > >> > > > > > > > > > > > >> >
> > > >> > > > > > > > > > > > >>
> > > >> > > > > > > > > > > > >
> > > >> > > > > > > > > > > >
> > > >> > > > > > > > > > >
> > > >> > > > > > > > > >
> > > >> > > > > > > > >
> > > >> > > > > > > >
> > > >> > > > > > >
> > > >> > > > > >
> > > >> > > > >
> > > >> > > >
> > > >> > >
> > > >> >
> > > >>
> > > >
> > >
> >

Re: Proposal for REST endpoints for table metrics and events

[Dmitri Bourlatchkov <di...@apache.org>]

Hi Yufei,

From my POV, it is fine for Polaris to have multiple REST API surfaces with
different features and different degrees of "readiness". Each API can
evolve fairly independently. With proper modularization (which Anand did),
changes to the main Polaris code and these extra modules should normally be
non-intrusive to each other.

I assume Anand is going to be the first user of this API. However, if we do
not merge it we'll never know whether other users are interested. I do not
think many end users read the dev ML, but they certainly read the docs,
code and release notes.

There is no risk for Polaris as a project in exposing this experimental
API. If usage dwindles over time, we can remove it. If it gains a
considerable user / contributor base, we will promote it to a "first-class"
(non-beta) API.

Cheers,
Dmitri.

On Tue, Jun 30, 2026 at 7:53 PM Yufei Gu <fl...@gmail.com> wrote:

> Hi all,
>
> Thanks for driving this work forward. Building this as a POC makes perfect
> sense if the goal is to validate the idea and gather feedback. Should we
> merge a POC into the main branch before agreeing on the overall direction?
> To me, that feels like a separate discussion from whether the
> implementation itself looks reasonable.
>
> One concern that I don't think has been fully addressed is whether we need
> to expose metrics consumption through a Polaris REST endpoint at all.
>
> My concern is less about the API itself and more about the boundary Polaris
> should own. The proposed REST API is tied to a particular persistence
> schema, but that schema is just one possible implementation. While the
> current implementation stores metrics in the metastore, other deployments
> may choose a KV store, a dedicated metrics warehouse, or another backend
> entirely. In those cases, exposing a Polaris REST API over a specific
> schema feels like coupling the API to an implementation detail.
>
> Polaris isn't intended to become a full metrics system. I'd expect it to
> focus on producing metrics and let deployments choose the most appropriate
> storage and consumption model for their environment.
>
> I'm not opposed to exploring this direction, but we should first align on
> whether exposing a REST API through Polaris is the right long term approach
> before merging it into the main branch.
>
> Thanks,
> Yufei
>
> Yufei
>
>
> On Mon, Jun 29, 2026 at 9:27 AM Dmitri Bourlatchkov <di...@apache.org>
> wrote:
>
> > Hi All,
> >
> > It looks like Anand addressed all action items in [4115].
> >
> > Please review the latest state of the code. The PR LGTM and I approved in
> > GH from my side.
> >
> > [4115] https://github.com/apache/polaris/pull/4115
> >
> > Thanks,
> > Dmitri.
> >
> > On Thu, Jun 25, 2026 at 1:56 PM Dmitri Bourlatchkov <di...@apache.org>
> > wrote:
> >
> > > Hi All,
> > >
> > > Recap of the discussion in today's Community Sync:
> > >
> > > * The modular approach for Metrics SPI(s) and implementation seems to
> > have
> > > community consensus (please comment if you still have concerns)
> > >
> > > * What modules / feature flags are enabled in Apache Polaris images by
> > > default will be discussed later (when the feature is close to
> > completion).
> > >
> > > In any case, users will be able to turn the full feature on/off at
> their
> > > discretion, regardless of the default flag values.
> > >
> > > * It is generally preferable to have smaller PRs and introduce new
> > > features gradually to simplify reviews.
> > >
> > > What this means for the Metrics API, IMHO is:
> > >
> > > a) The scope of PR 4115 is probably ok as it stands right now. If
> someone
> > > wants to propose splitting it further, please comment.
> > >
> > > b) After 4115 is merged, Metrics Persistence PRs will follow, with
> > smaller
> > > scope per PR as appropriate for the nature of the code changes.
> > >
> > > * The major concern was about exposing the new Metrics REST API to end
> > > users. The nature of this concern was about avoiding the (false)
> > impression
> > > that the new API is a firm and adopted Polaris specification.
> > >
> > > I think this concern is reasonable.
> > >
> > > I propose that Anand update the documentation sections of PR 4115 to
> > > emphasise that the API is currently an experimental / POC API at this
> > time
> > > and is subject to change (including breaking changes). I believe it
> > already
> > > has the "beta" label. So this request is mostly for doc changes to
> > clarify
> > > user expectations, since the "beta" label alone may not be interpreted
> > the
> > > same way by all users.
> > >
> > > I hope this update will resolve all remaining concerns with 4115 and
> > allow
> > > for it to be merged so that the community will be able to proceed to
> > > follow-up PRs (as noted above).
> > >
> > > Please add/correct me if I missed anything relevant.
> > >
> > > Thanks,
> > > Dmitri.
> > >
> > > On Wed, Jun 24, 2026 at 8:11 PM Yufei Gu <fl...@gmail.com> wrote:
> > >
> > >> I do not think we have agreement on the query REST APIs yet.
> > >>
> > >> In particular, the current query API shape seems highly coupled to the
> > >> *example[1]* metrics in the Iceberg REST spec. I do not think that is
> a
> > >> good way to design a REST API contract. For example, fields like
> > >> resultDataFiles, resultDeleteFiles, totalFileSizeBytes,
> > >> totalDataManifests,
> > >> totalDeleteManifests, and scannedDataManifests are very specific to
> one
> > >> style of scan metric. That makes the API look more like a direct
> > >> projection
> > >> of the current example payload than a general query model.
> > >>
> > >> My preference is that Polaris should not define or strongly opine on
> the
> > >> Iceberg scan or commit metrics query REST API at this point. Polaris
> can
> > >> resolve and authorize the table, accept the metrics report, and
> delegate
> > >> it
> > >> to the configured reporting path. Durable storage, filtering,
> > dashboards,
> > >> and query APIs should remain downstream impl. choices unless and until
> > the
> > >> Iceberg community standardizes a query contract, which I doubt will
> > >> happen.
> > >>
> > >> So I do not think the intake path has to be blocked, but I would
> prefer
> > >> not
> > >> to merge a Polaris specific query API that may be hard to evolve
> later.
> > >>
> > >> 1.
> > >>
> > >>
> >
> https://github.com/apache/iceberg/blob/2a6c556c0c883c04c6d3fbd68e6aeda36b91e0aa/open-api/rest-catalog-open-api.yaml#L4152
> > >>
> > >> Yufei
> > >>
> > >>
> > >> On Tue, Jun 23, 2026 at 9:30 AM Dmitri Bourlatchkov <dimas@apache.org
> >
> > >> wrote:
> > >>
> > >> > Hi All,
> > >> >
> > >> > So, what are the blockers for merging [4115], if any?
> > >> >
> > >> > [4115] https://github.com/apache/polaris/pull/4115
> > >> >
> > >> > Thanks,
> > >> > Dmitri.
> > >> >
> > >> > On Wed, Jun 17, 2026 at 5:28 PM EJ Wang <
> > ej.wang.dev.interest@gmail.com
> > >> >
> > >> > wrote:
> > >> >
> > >> > > Hi folks,
> > >> > >
> > >> > > Thanks again for the discussion today. I updated the sync doc with
> > >> notes
> > >> > > from today's metrics sync:
> > >> > >
> > >> > >
> > >> > >
> > >> >
> > >>
> >
> https://docs.google.com/document/d/100h7c4damrUzVuquYbBHM0EvA4LSWuW2IT2dN_7nYVA/edit?tab=t.ezk0rgdx0c6m
> > >> > >
> > >> > > A few highlights:
> > >> > >
> > >> > >    - No objection to keeping the default metrics behavior small:
> > >> > >    no-op/log-only is enough as the built-in default.
> > >> > >    - Durable storage, event forwarding, external queues,
> dashboards,
> > >> and
> > >> > >    custom filtering should be implementation choices behind the
> > >> metrics
> > >> > >    reporting path.
> > >> > >    - For PR #4115, the REST layer should resolve/authz/accept the
> > >> report,
> > >> > >    then delegate to the selected metrics implementation. It should
> > not
> > >> > own
> > >> > >    durable storage semantics.
> > >> > >    - The rough boundary we discussed is: spec owns the wire
> > contract,
> > >> > >    runtime owns framework wiring, API/contract modules own
> > >> > provider-facing
> > >> > >    contracts, and extensions own replaceable implementations.
> > >> > >    - OpenLineage has a similar path-naming question. We should
> avoid
> > >> > >    occupying a generic lineage path too early if we want room for
> > >> other
> > >> > >    lineage systems later.
> > >> > >
> > >> > > I believe the boundary rules previewed in sync meeting are useful
> > for
> > >> > > reviewing PR #4115, but it is not
> > >> > > a final module-layout proposal.
> > >> > >
> > >> > > For metrics specifically, I think the target remains:
> > >> > >    runtime
> > >> > >      -> metrics reporting/emitting contract
> > >> > >      -> default implementation
> > >> > >           -> lower implementation details, if needed
> > >> > >
> > >> > > That keeps the default simple, while allowing durable JDBC,
> > >> event-backed,
> > >> > > or external-queue implementations to be added separately.
> > >> > >
> > >> > > We also touched on the split between Iceberg metrics emitting and
> > >> > > Polaris-owned metrics querying. The querying side can keep
> evolving,
> > >> > > especially for dashboard or generic-table use cases, but I do not
> > >> think
> > >> > > that needs to block the intake path in this PR.
> > >> > >
> > >> > > Thanks,
> > >> > > -ej
> > >> > >
> > >> > > On Tue, Jun 16, 2026 at 2:17 PM Yufei Gu <fl...@gmail.com>
> > >> wrote:
> > >> > >
> > >> > > > Thanks for chiming in, EJ. Agreed that the default battery
> should
> > >> stay
> > >> > > > small. I'm leaning toward not including metrics persistence in
> the
> > >> > > default
> > >> > > > battery.
> > >> > > >
> > >> > > > Yufei
> > >> > > >
> > >> > > >
> > >> > > > On Thu, Jun 11, 2026 at 8:30 PM EJ Wang <
> > >> > ej.wang.dev.interest@gmail.com>
> > >> > > > wrote:
> > >> > > >
> > >> > > > > Hi Yufei,
> > >> > > > >
> > >> > > > > Thanks for connecting this back to the REST endpoint
> proposal. I
> > >> > > replied
> > >> > > > > with the fuller version on the event-forwarding thread, but
> > >> wanted to
> > >> > > add
> > >> > > > > the shorter version here too.
> > >> > > > >
> > >> > > > > I agree that metrics reporting/emitting is the right
> conceptual
> > >> > > boundary,
> > >> > > > > and I also think the event/listener path is a good
> > implementation
> > >> > > > direction
> > >> > > > > to explore. The distinction I want to keep clear is default
> > >> battery
> > >> > vs
> > >> > > > > extension implementation.
> > >> > > > >
> > >> > > > > For the REST/API side, I would keep the semantics narrow:
> > Polaris
> > >> > > > resolves
> > >> > > > > and authorizes the table, accepts the Iceberg scan/commit
> > metrics
> > >> > > report
> > >> > > > > into the configured reporting/emitting path, and returns 204.
> > That
> > >> > 204
> > >> > > > > should mean Polaris accepted the report into the ingestion
> path;
> > >> it
> > >> > > > should
> > >> > > > > not imply durable storage.
> > >> > > > >
> > >> > > > > Durable storage, event forwarding, external telemetry routing,
> > >> > > filtering,
> > >> > > > > and retention should sit behind that reporting/emitting
> boundary
> > >> as
> > >> > > > > implementation-layer behavior. A durable JDBC path can be one
> > >> named
> > >> > > > > extension implementation. An event/listener forwarding path
> can
> > be
> > >> > > > another
> > >> > > > > named extension implementation.
> > >> > > > >
> > >> > > > > So I am +1 on the event-forwarding idea as a non-default
> > extension
> > >> > > > > implementation. I just would not make it the default battery
> or
> > >> the
> > >> > > core
> > >> > > > > API shape. The default battery can stay small and safe, while
> > >> > > deployments
> > >> > > > > that need forwarding, dashboards, or durable storage can opt
> > into
> > >> the
> > >> > > > > implementation that matches their operational model.
> > >> > > > >
> > >> > > > > Thanks,
> > >> > > > > -ej
> > >> > > > >
> > >> > > > > On Wed, Jun 10, 2026 at 11:43 AM Yufei Gu <
> flyrain000@gmail.com
> > >
> > >> > > wrote:
> > >> > > > >
> > >> > > > > > Thanks EJ. I agree that the reporting/emitting boundary
> feels
> > >> like
> > >> > > the
> > >> > > > > > right SPI boundary, but I wonder if we can simplify this
> even
> > >> > > further.
> > >> > > > > >
> > >> > > > > > Iceberg scan and commit metrics look very similar to events
> to
> > >> me.
> > >> > > They
> > >> > > > > are
> > >> > > > > > append only, asynchronously consumed, often forwarded to
> > >> external
> > >> > > > > systems,
> > >> > > > > > and have similar retention and cleanup requirements. More
> > >> details
> > >> > can
> > >> > > > be
> > >> > > > > > found in this thread [1]. In that case, Polaris may only
> need
> > to
> > >> > > accept
> > >> > > > > the
> > >> > > > > > report and emit an event through the existing event
> framework.
> > >> > > > Filtering,
> > >> > > > > > async delivery, custom sinks, and retention mechanisms could
> > >> then
> > >> > be
> > >> > > > > shared
> > >> > > > > > instead of introducing a separate metrics specific extension
> > >> layer
> > >> > > and
> > >> > > > > SPI.
> > >> > > > > >
> > >> > > > > > The main requirement I still see is querying metrics through
> > >> > Polaris.
> > >> > > > If
> > >> > > > > we
> > >> > > > > > want that battery included experience, we could work on a
> > >> > persistence
> > >> > > > > > listener implementation that selectively saves metrics and
> > >> expose
> > >> > it
> > >> > > > via
> > >> > > > > > the IRC event endpoint which seems pretty close in Iceberg
> > >> > > > > > community. Another option I like more is to allow users to
> > >> develop
> > >> > > > their
> > >> > > > > > own listener, so they can persist scan/commit metrics to any
> > >> > systems
> > >> > > > they
> > >> > > > > > prefer. In general, that feels like an implementation choice
> > on
> > >> top
> > >> > > of
> > >> > > > > the
> > >> > > > > > event framework rather than something that needs to shape
> the
> > >> core
> > >> > > > > > architecture.
> > >> > > > > >
> > >> > > > > > 1.
> > >> > https://lists.apache.org/thread/x9j8nscvy8hq61tyn01mj8yp6n9of0kp
> > >> > > > > >
> > >> > > > > > Thanks,
> > >> > > > > > Yufei
> > >> > > > > >
> > >> > > > > >
> > >> > > > > > On Thu, Jun 4, 2026 at 5:14 PM EJ Wang <
> > >> > > ej.wang.dev.interest@gmail.com
> > >> > > > >
> > >> > > > > > wrote:
> > >> > > > > >
> > >> > > > > > > Good point. I agree the existing PolarisMetricsReporter is
> > >> > already
> > >> > > > very
> > >> > > > > > > close to the right conceptual boundary. I am not
> proposing a
> > >> > second
> > >> > > > > > > parallel reporter concept. The distinction I am trying to
> > >> make is
> > >> > > > > > narrower:
> > >> > > > > > >
> > >> > > > > > > Current state:
> > >> > > > > > >
> > >> > > > > > >    - PolarisMetricsReporter is the existing metrics
> > reporting
> > >> > hook.
> > >> > > > > > >    - But it currently lives under runtime/service.
> > >> > > > > > >    - Its contract is documented around Quarkus/CDI
> discovery
> > >> and
> > >> > > > > > > @Identifier
> > >> > > > > > >    selection.
> > >> > > > > > >    - The default implementation is log-only.
> > >> > > > > > >    - The durable implementation writes through
> > >> > > PolarisMetricsManager
> > >> > > > > and
> > >> > > > > > >    MetricsPersistence.
> > >> > > > > > >    - MetricsPersistence is currently inherited by
> > >> > BasePersistence.
> > >> > > > > > >
> > >> > > > > > > My proposal:
> > >> > > > > > >
> > >> > > > > > >    - Keep the reporting/emitting boundary as the SPI.
> > >> > > > > > >    - Revise that contract in a framework-agnostic optional
> > >> > Iceberg
> > >> > > > > > *metrics
> > >> > > > > > >    extension API layer*.
> > >> > > > > > >    - Keep runtime/service responsible for REST ingestion,
> > >> > > table/authz
> > >> > > > > > >    validation, and runtime wiring.
> > >> > > > > > >    - Keep the battery implementation log-only in the
> metrics
> > >> > > > extension
> > >> > > > > > API
> > >> > > > > > >    layer, *not under runtime/service*.
> > >> > > > > > >    - Treat durable JDBC metrics as one implementation of
> the
> > >> > > > reporting
> > >> > > > > > SPI.
> > >> > > > > > >    - Decompose the MetricsPersistence and BasePersistence
> > >> > coupling
> > >> > > as
> > >> > > > > > >    durable implementation detail.
> > >> > > > > > >
> > >> > > > > > > So the difference is not “new reporter concept vs old
> > reporter
> > >> > > > > concept.”
> > >> > > > > > > but:
> > >> > > > > > >
> > >> > > > > > >
> > >> > > > > > >    - PolarisMetricsReporter today = runtime/service
> > CDI-shaped
> > >> > > hook.
> > >> > > > > > >    - Target reporting SPI = same conceptual boundary, but
> > >> placed
> > >> > in
> > >> > > > the
> > >> > > > > > >    right extension API layer and kept framework-agnostic.
> > >> > > > > > >
> > >> > > > > > > For the current metrics PR, I think the minimal
> > architectural
> > >> > > cleanup
> > >> > > > > > > before merge is to *put the reporting SPI contract and the
> > >> > battery
> > >> > > > > > default
> > >> > > > > > > in the right place*. The durable JDBC implementation,
> > >> > event-backed
> > >> > > > > async
> > >> > > > > > > implementation, external queue support, and deeper
> > persistence
> > >> > > > cleanup
> > >> > > > > > can
> > >> > > > > > > continue as follow-ups, as long as we do not lock the SPI
> > >> > boundary
> > >> > > to
> > >> > > > > > > BasePersistence or to runtime/service wiring.
> > >> > > > > > >
> > >> > > > > > > -ej
> > >> > > > > > >
> > >> > > > > > > On Wed, Jun 3, 2026 at 3:00 PM Dmitri Bourlatchkov <
> > >> > > dimas@apache.org
> > >> > > > >
> > >> > > > > > > wrote:
> > >> > > > > > >
> > >> > > > > > > > Hi EJ,
> > >> > > > > > > >
> > >> > > > > > > > Thanks for the recap / summary!
> > >> > > > > > > >
> > >> > > > > > > > > Do folks agree that the stable SPI boundary should be
> > >> > > > > > > > metrics reporting/emitting, not metrics persistence?
> > >> > > > > > > >
> > >> > > > > > > > SGTM.
> > >> > > > > > > >
> > >> > > > > > > > > Does an optional Iceberg metrics extension API layer
> > sound
> > >> > like
> > >> > > > the
> > >> > > > > > > right
> > >> > > > > > > > home for this SPI?
> > >> > > > > > > >
> > >> > > > > > > > How is that different from current
> PolarisMetricsReporter?
> > >> > > > > > > >
> > >> > > > > > > > > Should the current durable metrics work be reframed
> as a
> > >> > > durable
> > >> > > > > > > > JDBC reference implementation of that SPI?
> > >> > > > > > > >
> > >> > > > > > > > SGTM.
> > >> > > > > > > >
> > >> > > > > > > > > What is the smallest PR sequence to get there without
> > >> > blocking
> > >> > > > > > > > the current metrics work unnecessarily?
> > >> > > > > > > >
> > >> > > > > > > > Let's get https://github.com/apache/polaris/pull/4397
> > >> merged
> > >> > > > first.
> > >> > > > > > > >
> > >> > > > > > > > Then, I'd propose isolating the Metrics schema from the
> > >> > MetaStore
> > >> > > > > > schema.
> > >> > > > > > > > This will probably have some ripple effect into the
> > >> bootstrap
> > >> > > > > > workflows,
> > >> > > > > > > so
> > >> > > > > > > > it's not a trivial change.
> > >> > > > > > > >
> > >> > > > > > > > Then, let's reassess.
> > >> > > > > > > >
> > >> > > > > > > > Cheers,
> > >> > > > > > > > Dmitri.
> > >> > > > > > > >
> > >> > > > > > > > On Wed, Jun 3, 2026 at 5:40 PM EJ Wang <
> > >> > > > > ej.wang.dev.interest@gmail.com
> > >> > > > > > >
> > >> > > > > > > > wrote:
> > >> > > > > > > >
> > >> > > > > > > > > Hi folks,
> > >> > > > > > > > >
> > >> > > > > > > > > Thanks again for the discussion today. I updated the
> > sync
> > >> doc
> > >> > > > with
> > >> > > > > > > notes
> > >> > > > > > > > > from the third metrics architecture sync:
> > >> > > > > > > > >
> > >> > > > > > > > >
> > >> > > > > > > > >
> > >> > > > > > > >
> > >> > > > > > >
> > >> > > > > >
> > >> > > > >
> > >> > > >
> > >> > >
> > >> >
> > >>
> >
> https://docs.google.com/document/d/100h7c4damrUzVuquYbBHM0EvA4LSWuW2IT2dN_7nYVA/edit?tab=t.k96s2xyqr5u1
> > >> > > > > > > > >
> > >> > > > > > > > > A few highlights from today’s discussion:
> > >> > > > > > > > >
> > >> > > > > > > > >    - We clarified that Iceberg metrics reporting can
> be
> > >> > > > interpreted
> > >> > > > > > > > either
> > >> > > > > > > > >    as sync or async handling from Polaris’s
> perspective,
> > >> so
> > >> > > > Polaris
> > >> > > > > > > > should
> > >> > > > > > > > >    stay flexible as a platform instead of baking one
> > >> handling
> > >> > > > model
> > >> > > > > > > into
> > >> > > > > > > > > the
> > >> > > > > > > > >    REST API behavior.
> > >> > > > > > > > >    - We aligned that the built-in (battery) behavior
> for
> > >> > > metrics
> > >> > > > > > > emitting
> > >> > > > > > > > >    can stay simple: no-op/log-only is enough as the
> > >> default.
> > >> > > > > > > > >    - Durable metrics persistence should be treated as
> an
> > >> > > > > > implementation
> > >> > > > > > > > of
> > >> > > > > > > > >    the metrics reporting path, not as the core SPI
> > >> boundary.
> > >> > > > > > > > >    - The existing durable metrics work can be reviewed
> > as
> > >> a
> > >> > > > > reference
> > >> > > > > > > > >    implementation of the reporting SPI, with
> > >> > > persistence-related
> > >> > > > > > logic
> > >> > > > > > > > kept
> > >> > > > > > > > >    self-contained under a metrics durable
> implementation
> > >> > module
> > >> > > > > > rather
> > >> > > > > > > > than
> > >> > > > > > > > >    scattered through core entity persistence.
> > >> > > > > > > > >    - Dashboard/insights remains a real use case, but
> we
> > >> > agreed
> > >> > > to
> > >> > > > > > keep
> > >> > > > > > > it
> > >> > > > > > > > >    separate from the core metrics intake discussion
> for
> > >> now.
> > >> > > > > > > > >
> > >> > > > > > > > > I also did a quick source check after the meeting to
> > make
> > >> > sure
> > >> > > we
> > >> > > > > are
> > >> > > > > > > > > describing the current state accurately.
> > >> > > > > > > > >
> > >> > > > > > > > > Current state:
> > >> > > > > > > > >
> > >> > > > > > > > >    - Polaris already has a metrics reporting hook:
> > >> > > > > > > > PolarisMetricsReporter.
> > >> > > > > > > > >    - The default implementation is
> > DefaultMetricsReporter,
> > >> > > > selected
> > >> > > > > > by
> > >> > > > > > > > >    polaris.iceberg-metrics.reporting.type=default. It
> is
> > >> > > > log-only,
> > >> > > > > > > > >    effectively quiet unless metrics logging is
> enabled.
> > >> > > > > > > > >    - There is also a PersistingMetricsReporter,
> selected
> > >> by
> > >> > > > > > > > >    polaris.iceberg-metrics.reporting.type=persisting,
> > >> which
> > >> > > > > converts
> > >> > > > > > > > >    Iceberg scan/commit reports into Polaris metrics
> > >> records
> > >> > and
> > >> > > > > > writes
> > >> > > > > > > > > through
> > >> > > > > > > > >    PolarisMetricsManager -> MetricsPersistence.
> > >> > > > > > > > >    - MetricsPersistence currently exists in the
> > >> persistence
> > >> > > layer
> > >> > > > > and
> > >> > > > > > > is
> > >> > > > > > > > >    inherited by BasePersistence.
> > >> > > > > > > > >
> > >> > > > > > > > > My read from the discussion is that the target
> boundary
> > >> > should
> > >> > > > not
> > >> > > > > be
> > >> > > > > > > > > MetricsPersistence as inherited by BasePersistence.
> That
> > >> path
> > >> > > > > should
> > >> > > > > > be
> > >> > > > > > > > > decomposed as durable implementation detail (taken
> care
> > >> of by
> > >> > > > > > > > > https://github.com/apache/polaris/pull/4397). The
> > stable
> > >> > > > extension
> > >> > > > > > > point
> > >> > > > > > > > > should instead be the metrics reporting/emitting
> > boundary.
> > >> > > > > > > > >
> > >> > > > > > > > > Concretely, I think the next proposal should be shaped
> > >> like
> > >> > > this:
> > >> > > > > > > > >
> > >> > > > > > > > >    1.
> > >> > > > > > > > >
> > >> > > > > > > > >    Define the proper metrics reporting SPI at an
> > optional
> > >> > > Iceberg
> > >> > > > > > > metrics
> > >> > > > > > > > >    extension API layer.
> > >> > > > > > > > >
> > >> > > > > > > > >    Example direction:
> > >> > > > > > > > >
> > >> > > > > > > > >    public interface IcebergMetricsReporter {
> > >> > > > > > > > >      void reportMetric(IcebergMetricsReportContext
> > >> context,
> > >> > > > > > > > > MetricsReport report);
> > >> > > > > > > > >    }
> > >> > > > > > > > >
> > >> > > > > > > > >    The context would carry the small Polaris-resolved
> > >> > envelope:
> > >> > > > > > > > >    catalog/table identity, report type, received
> > >> timestamp,
> > >> > and
> > >> > > > > > > > > request/trace
> > >> > > > > > > > >    context if available. The raw Iceberg MetricsReport
> > >> > remains
> > >> > > > the
> > >> > > > > > > > payload.
> > >> > > > > > > > >    2.
> > >> > > > > > > > >
> > >> > > > > > > > >    Keep runtime/service as the REST ingestion and
> wiring
> > >> > layer.
> > >> > > > > > > > >
> > >> > > > > > > > >    The REST handler still resolves the table and
> > performs
> > >> > authz
> > >> > > > > > before
> > >> > > > > > > > >    accepting the report. After that, it calls the
> > selected
> > >> > > > > reporting
> > >> > > > > > > > >    implementation.
> > >> > > > > > > > >    3.
> > >> > > > > > > > >
> > >> > > > > > > > >    Keep the battery default no-op/log-only.
> > >> > > > > > > > >
> > >> > > > > > > > >    This preserves an out-of-box safe default and
> avoids
> > >> > > > requiring a
> > >> > > > > > > > durable
> > >> > > > > > > > >    metrics store for every Polaris deployment.
> > >> > > > > > > > >    4.
> > >> > > > > > > > >
> > >> > > > > > > > >    Move durable JDBC metrics into a self-contained
> > >> > > implementation
> > >> > > > > of
> > >> > > > > > > the
> > >> > > > > > > > >    reporting SPI.
> > >> > > > > > > > >
> > >> > > > > > > > >    That implementation can own its schema, bootstrap,
> > >> > > retention,
> > >> > > > > and
> > >> > > > > > > read
> > >> > > > > > > > >    API support. It should not define the core
> reporting
> > >> SPI
> > >> > > > > boundary,
> > >> > > > > > > and
> > >> > > > > > > > > it
> > >> > > > > > > > >    should not require metrics persistence to remain
> > >> inherited
> > >> > > > from
> > >> > > > > > > > >    BasePersistence.
> > >> > > > > > > > >    5.
> > >> > > > > > > > >
> > >> > > > > > > > >    Treat async/event-backed handling as another
> > >> > implementation
> > >> > > of
> > >> > > > > the
> > >> > > > > > > > same
> > >> > > > > > > > >    reporting SPI.
> > >> > > > > > > > >
> > >> > > > > > > > >    For example, an event-backed reporter could
> enqueue a
> > >> > > metrics
> > >> > > > > > event
> > >> > > > > > > > and
> > >> > > > > > > > >    let listeners handle durable storage or other
> sinks.
> > >> If we
> > >> > > > later
> > >> > > > > > > need
> > >> > > > > > > > a
> > >> > > > > > > > >    replaceable queue engine, that seems like a shared
> > >> > > > event/metrics
> > >> > > > > > > > > substrate
> > >> > > > > > > > >    topic rather than a metrics-only requirement.
> > >> > > > > > > > >
> > >> > > > > > > > > This framing lets us keep the REST metrics endpoint
> > >> simple,
> > >> > > > > preserve
> > >> > > > > > > the
> > >> > > > > > > > > current default behavior, support durable metrics
> users,
> > >> and
> > >> > > > still
> > >> > > > > > > leave
> > >> > > > > > > > > room for async/event-backed or external-queue-based
> > >> > > > > implementations.
> > >> > > > > > > > >
> > >> > > > > > > > > I think the main follow-up questions are:
> > >> > > > > > > > >
> > >> > > > > > > > >    - Do folks agree that the stable SPI boundary
> should
> > be
> > >> > > > metrics
> > >> > > > > > > > >    reporting/emitting, not metrics persistence?
> > >> > > > > > > > >    - Does an optional Iceberg metrics extension API
> > layer
> > >> > sound
> > >> > > > > like
> > >> > > > > > > the
> > >> > > > > > > > >    right home for this SPI?
> > >> > > > > > > > >    - Should the current durable metrics work be
> reframed
> > >> as a
> > >> > > > > durable
> > >> > > > > > > > JDBC
> > >> > > > > > > > >    reference implementation of that SPI?
> > >> > > > > > > > >    - What is the smallest PR sequence to get there
> > without
> > >> > > > blocking
> > >> > > > > > the
> > >> > > > > > > > >    current metrics work unnecessarily?
> > >> > > > > > > > >
> > >> > > > > > > > > Thanks,
> > >> > > > > > > > > -ej
> > >> > > > > > > > >
> > >> > > > > > > > > On Fri, May 15, 2026 at 5:16 PM Dmitri Bourlatchkov <
> > >> > > > > > dimas@apache.org>
> > >> > > > > > > > > wrote:
> > >> > > > > > > > >
> > >> > > > > > > > > > Hi JB,
> > >> > > > > > > > > >
> > >> > > > > > > > > > Could you set up another meeting, please? Same time
> on
> > >> > > > Wednesday
> > >> > > > > as
> > >> > > > > > > > last
> > >> > > > > > > > > > time... I hope it works for everyone.
> > >> > > > > > > > > >
> > >> > > > > > > > > > Cheers,
> > >> > > > > > > > > > Dmitri.
> > >> > > > > > > > > >
> > >> > > > > > > > > > On Fri, May 15, 2026 at 8:06 PM Yufei Gu <
> > >> > > flyrain000@gmail.com
> > >> > > > >
> > >> > > > > > > wrote:
> > >> > > > > > > > > >
> > >> > > > > > > > > > > +1 on another sync call next week.
> > >> > > > > > > > > > >
> > >> > > > > > > > > > > Yufei
> > >> > > > > > > > > > >
> > >> > > > > > > > > > >
> > >> > > > > > > > > > > On Fri, May 15, 2026 at 4:52 PM Dmitri
> Bourlatchkov
> > <
> > >> > > > > > > > dimas@apache.org>
> > >> > > > > > > > > > > wrote:
> > >> > > > > > > > > > >
> > >> > > > > > > > > > > > Hi All,
> > >> > > > > > > > > > > >
> > >> > > > > > > > > > > > WDYT about another sync call next week?
> > >> > > > > > > > > > > >
> > >> > > > > > > > > > > > Thanks,
> > >> > > > > > > > > > > > Dmitri.
> > >> > > > > > > > > > > >
> > >> > > > > > > > > > > > On Wed, May 6, 2026 at 5:29 PM Dmitri
> > Bourlatchkov <
> > >> > > > > > > > dimas@apache.org
> > >> > > > > > > > > >
> > >> > > > > > > > > > > > wrote:
> > >> > > > > > > > > > > >
> > >> > > > > > > > > > > > > Hi EJ,
> > >> > > > > > > > > > > > >
> > >> > > > > > > > > > > > > Thanks for the summary! It covers what we
> > >> discussed
> > >> > in
> > >> > > > the
> > >> > > > > > > > meeting
> > >> > > > > > > > > > very
> > >> > > > > > > > > > > > > well, IMHO.
> > >> > > > > > > > > > > > >
> > >> > > > > > > > > > > > > Looking forward to concrete PRs :)
> > >> > > > > > > > > > > > >
> > >> > > > > > > > > > > > > Cheers,
> > >> > > > > > > > > > > > > Dmitri.
> > >> > > > > > > > > > > > >
> > >> > > > > > > > > > > > > On Wed, May 6, 2026 at 5:08 PM EJ Wang <
> > >> > > > > > > > > > ej.wang.dev.interest@gmail.com
> > >> > > > > > > > > > > >
> > >> > > > > > > > > > > > > wrote:
> > >> > > > > > > > > > > > >
> > >> > > > > > > > > > > > >> Hi folks,
> > >> > > > > > > > > > > > >>
> > >> > > > > > > > > > > > >> We had a community sync earlier, thanks JB
> for
> > >> > > > scheduling
> > >> > > > > > it.
> > >> > > > > > > > > Notes
> > >> > > > > > > > > > > from
> > >> > > > > > > > > > > > >> the first metrics architecture sync (May 6,
> > >> 10-11am
> > >> > > PT).
> > >> > > > > > > > > Discussion
> > >> > > > > > > > > > > doc
> > >> > > > > > > > > > > > >> with per-section status:
> > >> > > > > > > > > > > > >>
> > >> > > > > > > > > > > > >>
> > >> > > > > > > > > > > >
> > >> > > > > > > > > > >
> > >> > > > > > > > > >
> > >> > > > > > > > >
> > >> > > > > > > >
> > >> > > > > > >
> > >> > > > > >
> > >> > > > >
> > >> > > >
> > >> > >
> > >> >
> > >>
> >
> https://docs.google.com/document/d/100h7c4damrUzVuquYbBHM0EvA4LSWuW2IT2dN_7nYVA/edit?tab=t.0
> > >> > > > > > > > > > > > >>
> > >> > > > > > > > > > > > >> *The meeting covered both topics from the
> doc.
> > >> > > > > > Direction-level
> > >> > > > > > > > > > > alignment
> > >> > > > > > > > > > > > >> was reached on the headline pieces; details
> > >> remain
> > >> > for
> > >> > > > PR
> > >> > > > > > > review
> > >> > > > > > > > > or
> > >> > > > > > > > > > > > >> follow-up sessions.*
> > >> > > > > > > > > > > > >>
> > >> > > > > > > > > > > > >> *Topic 1 — Persistence schema redesign*
> > >> > > > > > > > > > > > >> Idea-level alignment on consolidating
> per-type
> > >> > tables
> > >> > > > > > > > > > > > >> (scan_metrics_report,
> > >> > > > > > > > > > > > >> commit_metrics_report) into a single
> > >> metrics_report
> > >> > > > table.
> > >> > > > > > The
> > >> > > > > > > > > > > > motivating
> > >> > > > > > > > > > > > >> cost is the surface area added by every new
> > >> metric
> > >> > > type
> > >> > > > > > today:
> > >> > > > > > > > new
> > >> > > > > > > > > > > > table,
> > >> > > > > > > > > > > > >> SPI method, record class, model, converter,
> > >> schema
> > >> > > > > > migration.
> > >> > > > > > > > > > > > >>
> > >> > > > > > > > > > > > >> Most schema details are deferred to the
> schema
> > >> PR. A
> > >> > > few
> > >> > > > > > > > specific
> > >> > > > > > > > > > > points
> > >> > > > > > > > > > > > >> came up:
> > >> > > > > > > > > > > > >> •   metric_schema_version: Yufei prefers
> > dropping
> > >> > it,
> > >> > > > > since
> > >> > > > > > > > there
> > >> > > > > > > > > is
> > >> > > > > > > > > > > no
> > >> > > > > > > > > > > > >> spec-level concept of metrics versioning
> today
> > >> and
> > >> > it
> > >> > > is
> > >> > > > > > hard
> > >> > > > > > > to
> > >> > > > > > > > > > > define
> > >> > > > > > > > > > > > >> unilaterally. Robert prefers keeping it,
> given
> > >> IRC
> > >> > v2
> > >> > > is
> > >> > > > > > > coming
> > >> > > > > > > > > and
> > >> > > > > > > > > > > the
> > >> > > > > > > > > > > > >> schema should be considered against its
> likely
> > >> > shape;
> > >> > > > > Robert
> > >> > > > > > > > also
> > >> > > > > > > > > > > raised
> > >> > > > > > > > > > > > >> how to differentiate various payload formats
> if
> > >> any.
> > >> > > > EJ's
> > >> > > > > > read
> > >> > > > > > > > is
> > >> > > > > > > > > > that
> > >> > > > > > > > > > > > >> this
> > >> > > > > > > > > > > > >> is a two-way-door decision. We can start
> > without
> > >> the
> > >> > > > > field,
> > >> > > > > > > and
> > >> > > > > > > > if
> > >> > > > > > > > > > IRC
> > >> > > > > > > > > > > > v2
> > >> > > > > > > > > > > > >> changes the shape we would likely roll a
> > >> > corresponding
> > >> > > > new
> > >> > > > > > > > schema
> > >> > > > > > > > > > > > anyway,
> > >> > > > > > > > > > > > >> which is not particularly costly.
> > >> > > > > > > > > > > > >> •   Payload format: Robert pointed out that
> > >> future
> > >> > > > formats
> > >> > > > > > > > beyond
> > >> > > > > > > > > > JSON
> > >> > > > > > > > > > > > may
> > >> > > > > > > > > > > > >> be worth supporting. The exact shape is
> > deferred
> > >> to
> > >> > > the
> > >> > > > > > schema
> > >> > > > > > > > > > > > discussion.
> > >> > > > > > > > > > > > >> •   Partition strategy: Anand suggested
> monthly
> > >> > > > > partitioning
> > >> > > > > > > > based
> > >> > > > > > > > > > on
> > >> > > > > > > > > > > > his
> > >> > > > > > > > > > > > >> experience as potentially helpful at scale.
> > >> > > > > > > > > > > > >>
> > >> > > > > > > > > > > > >> *Topic 2 — Where metrics ingestion and
> storage
> > >> > belong*
> > >> > > > > > > > > > > > >> Idea-level alignment that metrics should be a
> > >> > > separated
> > >> > > > > SPI
> > >> > > > > > > from
> > >> > > > > > > > > the
> > >> > > > > > > > > > > > >> entity
> > >> > > > > > > > > > > > >> persistence stack. Two reasons surfaced: (a)
> > >> > workloads
> > >> > > > and
> > >> > > > > > > > > > capability
> > >> > > > > > > > > > > > >> requirements diverge enough that coupling
> them
> > >> > creates
> > >> > > > > > > > artificial
> > >> > > > > > > > > > > > >> constraints, and (b) admin experience
> improves
> > >> when
> > >> > > > > metrics
> > >> > > > > > > has
> > >> > > > > > > > > its
> > >> > > > > > > > > > > own
> > >> > > > > > > > > > > > >> bootstrap, retention, and lifecycle. Dmitri
> > noted
> > >> > > > Polaris
> > >> > > > > > > being
> > >> > > > > > > > a
> > >> > > > > > > > > > > > platform
> > >> > > > > > > > > > > > >> should have the flexibility to support
> > different
> > >> > > > > persistence
> > >> > > > > > > > > > backends
> > >> > > > > > > > > > > > per
> > >> > > > > > > > > > > > >> concern, and pointed to a concrete next step
> of
> > >> > > > separating
> > >> > > > > > the
> > >> > > > > > > > > JDBC
> > >> > > > > > > > > > > > >> bootstrap for metrics from the metastore
> > >> bootstrap.
> > >> > > > Robert
> > >> > > > > > > > > proposed
> > >> > > > > > > > > > an
> > >> > > > > > > > > > > > >> additional UX extension: detect an
> > unbootstrapped
> > >> > > > metrics
> > >> > > > > > > store
> > >> > > > > > > > on
> > >> > > > > > > > > > > first
> > >> > > > > > > > > > > > >> use and auto-bootstrap rather than requiring
> an
> > >> > > explicit
> > >> > > > > > > manual
> > >> > > > > > > > > > > > bootstrap
> > >> > > > > > > > > > > > >> step.
> > >> > > > > > > > > > > > >> The meeting also confirmed that Polaris
> metrics
> > >> can
> > >> > > > start
> > >> > > > > > > small
> > >> > > > > > > > > and
> > >> > > > > > > > > > > stay
> > >> > > > > > > > > > > > >> Iceberg-focused. Naming and persistence
> schema
> > >> can
> > >> > > lean
> > >> > > > > > > > > > > > Iceberg-specific.
> > >> > > > > > > > > > > > >> If a future expansion to generic-table
> metrics
> > or
> > >> > > > > > operational
> > >> > > > > > > > > > metrics
> > >> > > > > > > > > > > > >> arrives, an abstraction layer can be built on
> > >> top of
> > >> > > the
> > >> > > > > > > Iceberg
> > >> > > > > > > > > > > metrics
> > >> > > > > > > > > > > > >> reporter at that point. Robert remains on the
> > >> fence
> > >> > > and
> > >> > > > > > would
> > >> > > > > > > > > prefer
> > >> > > > > > > > > > > > >> something more generic but did not block the
> > >> > > direction;
> > >> > > > > > > Dmitri's
> > >> > > > > > > > > > read
> > >> > > > > > > > > > > > was
> > >> > > > > > > > > > > > >> that the proposed framework already has
> enough
> > >> > > > flexibility
> > >> > > > > > to
> > >> > > > > > > > > absorb
> > >> > > > > > > > > > > > >> future
> > >> > > > > > > > > > > > >> expansion.
> > >> > > > > > > > > > > > >>
> > >> > > > > > > > > > > > >> The Trade-offs and Proposed structure
> sections
> > in
> > >> > the
> > >> > > > doc
> > >> > > > > > were
> > >> > > > > > > > not
> > >> > > > > > > > > > > > >> reviewed
> > >> > > > > > > > > > > > >> in detail. They remain open for either the
> next
> > >> sync
> > >> > > or
> > >> > > > PR
> > >> > > > > > > > review.
> > >> > > > > > > > > > > > >>
> > >> > > > > > > > > > > > >> *Cross-cutting alignment — battery-included
> > plus
> > >> > > > > pluggable*
> > >> > > > > > > > > > > > >> A common philosophy emerged from the
> > discussion.
> > >> EJ
> > >> > > > > > summarized
> > >> > > > > > > > it
> > >> > > > > > > > > > as:
> > >> > > > > > > > > > > > >> Polaris should provide a battery-included UX
> > for
> > >> > > > beginners
> > >> > > > > > and
> > >> > > > > > > > the
> > >> > > > > > > > > > > > >> flexibility for advanced users to swap the
> > >> included
> > >> > > > > battery
> > >> > > > > > > for
> > >> > > > > > > > > > > > something
> > >> > > > > > > > > > > > >> more powerful or tailored to their use case.
> > The
> > >> SPI
> > >> > > > > design
> > >> > > > > > > > needs
> > >> > > > > > > > > to
> > >> > > > > > > > > > > > >> enable
> > >> > > > > > > > > > > > >> both.
> > >> > > > > > > > > > > > >>
> > >> > > > > > > > > > > > >> The inputs that shaped this framing:
> > >> > > > > > > > > > > > >> •   Anand described how his team uses the
> > current
> > >> > > > metrics
> > >> > > > > > > > > > persistence
> > >> > > > > > > > > > > > >> (three metrics consumers in v1.4).
> > >> > > > > > > > > > > > >> •   Yufei raised Grafana and dashboard
> > >> integrations
> > >> > > as a
> > >> > > > > > > > > destination
> > >> > > > > > > > > > > use
> > >> > > > > > > > > > > > >> case beyond the default.
> > >> > > > > > > > > > > > >> •   Robert called out that the current design
> > is
> > >> > more
> > >> > > > > > > > > JDBC-focused.
> > >> > > > > > > > > > > > >>
> > >> > > > > > > > > > > > >> Two concrete instances:
> > >> > > > > > > > > > > > >> •   Async metrics intake: Yufei's initial
> > >> position
> > >> > was
> > >> > > > > that
> > >> > > > > > > > async
> > >> > > > > > > > > > > should
> > >> > > > > > > > > > > > >> largely live on the producer side and there
> is
> > >> not
> > >> > > much
> > >> > > > > > > Polaris
> > >> > > > > > > > > can
> > >> > > > > > > > > > > do.
> > >> > > > > > > > > > > > >> Robert suggested a Polaris-side default is
> > doable
> > >> > via
> > >> > > > > > Vert.x.
> > >> > > > > > > > > Dmitri
> > >> > > > > > > > > > > > >> agreed
> > >> > > > > > > > > > > > >> the direction is worth exploring. The meeting
> > >> > > converged
> > >> > > > > on a
> > >> > > > > > > > > > > > >> battery-included default (likely
> Vert.x-backed)
> > >> with
> > >> > > an
> > >> > > > > SPI
> > >> > > > > > > > shape
> > >> > > > > > > > > > that
> > >> > > > > > > > > > > > >> lets
> > >> > > > > > > > > > > > >> power users route to a more scalable backend
> > >> > > (k8s-hosted
> > >> > > > > > > queue,
> > >> > > > > > > > > AWS
> > >> > > > > > > > > > > SQS,
> > >> > > > > > > > > > > > >> etc.).
> > >> > > > > > > > > > > > >> •   Pluggable destinations: combining Yufei's
> > >> > > dashboard
> > >> > > > > use
> > >> > > > > > > case
> > >> > > > > > > > > > with
> > >> > > > > > > > > > > > >> Robert's JDBC-focused call-out, the meeting
> > >> agreed
> > >> > the
> > >> > > > SPI
> > >> > > > > > > > should
> > >> > > > > > > > > be
> > >> > > > > > > > > > > > >> structured for multiple sinks so integrations
> > >> become
> > >> > > > impl
> > >> > > > > > > > choices
> > >> > > > > > > > > > > rather
> > >> > > > > > > > > > > > >> than architectural changes.
> > >> > > > > > > > > > > > >>
> > >> > > > > > > > > > > > >> The battery-included default is most likely
> to
> > >> use
> > >> > the
> > >> > > > > > > existing
> > >> > > > > > > > > > > > >> JDBC-backed
> > >> > > > > > > > > > > > >> approach.
> > >> > > > > > > > > > > > >>
> > >> > > > > > > > > > > > >> *Direction (idea-level alignment)*
> > >> > > > > > > > > > > > >> •   Single metrics_report table consolidating
> > >> > per-type
> > >> > > > > > > metrics,
> > >> > > > > > > > > > > > replacing
> > >> > > > > > > > > > > > >> scan_metrics_report and commit_metrics_report
> > >> > > > > > > > > > > > >> •   Iceberg-focused naming and schema for
> now,
> > >> > revisit
> > >> > > > if
> > >> > > > > > > > > > > generic-table
> > >> > > > > > > > > > > > or
> > >> > > > > > > > > > > > >> operational metrics arrive
> > >> > > > > > > > > > > > >> •   Metrics persistence as a separated SPI,
> not
> > >> on
> > >> > > > > > > > BasePersistence
> > >> > > > > > > > > > > > >> •   Bootstrap path separated for metrics,
> > >> > independent
> > >> > > of
> > >> > > > > > > > metastore
> > >> > > > > > > > > > > > >> bootstrap
> > >> > > > > > > > > > > > >> •   "Battery-included plus pluggable" as the
> > SPI
> > >> > > design
> > >> > > > > > > > philosophy
> > >> > > > > > > > > > > > >>
> > >> > > > > > > > > > > > >> *Open items*
> > >> > > > > > > > > > > > >> •   Schema details: metric_schema_version,
> > >> payload
> > >> > > > format,
> > >> > > > > > IRC
> > >> > > > > > > > v2
> > >> > > > > > > > > > > > >> forward-compat shape
> > >> > > > > > > > > > > > >> •   SPI design details — full review either
> in
> > >> the
> > >> > > next
> > >> > > > > sync
> > >> > > > > > > or
> > >> > > > > > > > in
> > >> > > > > > > > > > the
> > >> > > > > > > > > > > > >> corresponding PR
> > >> > > > > > > > > > > > >> •   Schema refactor PR ownership
> > >> > > > > > > > > > > > >>
> > >> > > > > > > > > > > > >> *Action items*
> > >> > > > > > > > > > > > >> •   EJ to take a first stab at the SPI design
> > and
> > >> > > > > > potentially
> > >> > > > > > > > > > partner
> > >> > > > > > > > > > > > with
> > >> > > > > > > > > > > > >> Anand to incorporate the lessons learned from
> > the
> > >> > > > existing
> > >> > > > > > > > > reporter
> > >> > > > > > > > > > > and
> > >> > > > > > > > > > > > >> persistence work.
> > >> > > > > > > > > > > > >> •   Schema refactor PR ownership is not yet
> > >> decided.
> > >> > > If
> > >> > > > > > anyone
> > >> > > > > > > > is
> > >> > > > > > > > > > > > >> interested in driving it, reply on this
> thread.
> > >> > > > > > > > > > > > >> •   JB to schedule the next sync, tentatively
> > in
> > >> two
> > >> > > > > weeks.
> > >> > > > > > > > > > > > >>
> > >> > > > > > > > > > > > >> -ej
> > >> > > > > > > > > > > > >>
> > >> > > > > > > > > > > > >> On Mon, Apr 27, 2026 at 3:07 PM EJ Wang <
> > >> > > > > > > > > > > ej.wang.dev.interest@gmail.com
> > >> > > > > > > > > > > > >
> > >> > > > > > > > > > > > >> wrote:
> > >> > > > > > > > > > > > >>
> > >> > > > > > > > > > > > >> > Thanks Yufei for the +1.
> > >> > > > > > > > > > > > >> >
> > >> > > > > > > > > > > > >> > JB, could you help add a biweekly metrics
> > >> > > architecture
> > >> > > > > > sync
> > >> > > > > > > to
> > >> > > > > > > > > the
> > >> > > > > > > > > > > > >> Polaris
> > >> > > > > > > > > > > > >> > community calendar? I'm thinking Thursdays
> at
> > >> > 9-10am
> > >> > > > PT,
> > >> > > > > > on
> > >> > > > > > > > the
> > >> > > > > > > > > > > > >> off-weeks
> > >> > > > > > > > > > > > >> > from the community meeting (starting May
> 7),
> > 60
> > >> > > > minutes.
> > >> > > > > > > > > > > > >> >
> > >> > > > > > > > > > > > >> > Here's a rough agenda to work through over
> > the
> > >> > first
> > >> > > > few
> > >> > > > > > > > > sessions,
> > >> > > > > > > > > > > > >> grouped
> > >> > > > > > > > > > > > >> > by priority:
> > >> > > > > > > > > > > > >> >
> > >> > > > > > > > > > > > >> > *First: foundational direction*
> > >> > > > > > > > > > > > >> >
> > >> > > > > > > > > > > > >> > 1.  MetricsPersistence: public SPI or
> > internal
> > >> > > > > > > implementation
> > >> > > > > > > > > > > detail?
> > >> > > > > > > > > > > > >> >    •   Marked @Beta, javadoc calls it a
> > >> "Service
> > >> > > > > Provider
> > >> > > > > > > > > > > Interface",
> > >> > > > > > > > > > > > >> but
> > >> > > > > > > > > > > > >> > only one consumer
> (JdbcBasePersistenceImpl),
> > >> lives
> > >> > > on
> > >> > > > > > > > > > > BasePersistence.
> > >> > > > > > > > > > > > >> If
> > >> > > > > > > > > > > > >> > demoted to a private helper inside a
> > persisting
> > >> > > > reporter
> > >> > > > > > > impl,
> > >> > > > > > > > > > most
> > >> > > > > > > > > > > > >> > downstream design decisions become
> > >> implementation
> > >> > > > > details
> > >> > > > > > > > rather
> > >> > > > > > > > > > > than
> > >> > > > > > > > > > > > >> > contract questions.
> > >> > > > > > > > > > > > >> >
> > >> > > > > > > > > > > > >> > 2.  Persistence schema redesign
> > >> > > > > > > > > > > > >> >    •   Current two-table layout
> > >> > > (scan_metrics_report,
> > >> > > > > > > > > > > > >> > commit_metrics_report) with ~25 flattened
> > >> columns
> > >> > > > each.
> > >> > > > > > > Every
> > >> > > > > > > > > new
> > >> > > > > > > > > > > > metric
> > >> > > > > > > > > > > > >> > type requires a new table, SPI method,
> record
> > >> > class,
> > >> > > > > > model,
> > >> > > > > > > > > > > converter,
> > >> > > > > > > > > > > > >> and
> > >> > > > > > > > > > > > >> > schema migration. Direction to explore:
> > single
> > >> > table
> > >> > > > > with
> > >> > > > > > > > > > > metric_type
> > >> > > > > > > > > > > > >> enum,
> > >> > > > > > > > > > > > >> > schema_version, and JSON payload column.
> > >> > > > > > > > > > > > >> >
> > >> > > > > > > > > > > > >> > *Second: design details once direction is
> > set*
> > >> > > > > > > > > > > > >> >
> > >> > > > > > > > > > > > >> > 3.  Partition key strategy
> > >> > > > > > > > > > > > >> >    •   Single-table design means scan
> metrics
> > >> at
> > >> > > scale
> > >> > > > > > will
> > >> > > > > > > > have
> > >> > > > > > > > > > > high
> > >> > > > > > > > > > > > >> > write concurrency per table. Schema needs
> to
> > >> > expose
> > >> > > > > enough
> > >> > > > > > > > > > structure
> > >> > > > > > > > > > > > for
> > >> > > > > > > > > > > > >> > backends to shard by entity or time range.
> > >> > > > > > > > > > > > >> >
> > >> > > > > > > > > > > > >> > 4.  Read/write path consistency
> > >> > > > > > > > > > > > >> >    •   Writes go through
> > PolarisMetricsManager
> > >> on
> > >> > > > > > > > > > MetaStoreManager.
> > >> > > > > > > > > > > > >> Reads
> > >> > > > > > > > > > > > >> > bypass MetaStoreManager and go straight to
> > >> > > > > > BasePersistence,
> > >> > > > > > > > > > > excluding
> > >> > > > > > > > > > > > >> > non-JDBC backends from the read API.
> > >> > > > > > > > > > > > >> >
> > >> > > > > > > > > > > > >> > *Third: cleanup and alignment*
> > >> > > > > > > > > > > > >> >
> > >> > > > > > > > > > > > >> > 5.  PolarisMetricsReporter naming
> > >> > > > > > > > > > > > >> >    •   Only handles IRC
> > >> (ScanReport/CommitReport),
> > >> > > > > doesn't
> > >> > > > > > > > cover
> > >> > > > > > > > > > > > generic
> > >> > > > > > > > > > > > >> > tables or operational metrics. Name is
> > broader
> > >> > than
> > >> > > > > scope.
> > >> > > > > > > > > > > > >> >
> > >> > > > > > > > > > > > >> > 6.  PolarisMetricsManager facade
> passthrough
> > >> > > > > > > > > > > > >> >    •   Entire default method is
> > >> > > > > > > > > > > > >> callCtx.getMetaStore().writeScanReport().
> > >> > > > > > > > > > > > >> > Zero logic, passes Level 1 straight through
> > to
> > >> > Level
> > >> > > > 3.
> > >> > > > > > Same
> > >> > > > > > > > > > > > >> anti-pattern
> > >> > > > > > > > > > > > >> > as PolarisEventManager.
> > >> > > > > > > > > > > > >> >
> > >> > > > > > > > > > > > >> > 7.  Iceberg community alignment
> > >> > > > > > > > > > > > >> >    •   Payload-type extension needs
> > discussion
> > >> on
> > >> > > > > > > dev@iceberg.
> > >> > > > > > > > > > > > >> obelix74's
> > >> > > > > > > > > > > > >> > Feb thread got zero replies. Needs a
> > committer
> > >> > > voice.
> > >> > > > > > > > > > > > >> >
> > >> > > > > > > > > > > > >> > Lets confirm prioritization in the first
> > >> session.
> > >> > > > > > > > > > > > >> >
> > >> > > > > > > > > > > > >> > -ej
> > >> > > > > > > > > > > > >> >
> > >> > > > > > > > > > > > >> > On Tue, Apr 21, 2026 at 3:18 PM Yufei Gu <
> > >> > > > > > > > flyrain000@gmail.com>
> > >> > > > > > > > > > > > wrote:
> > >> > > > > > > > > > > > >> >
> > >> > > > > > > > > > > > >> >> Thanks everyone for continuing to drive
> this
> > >> > > > forward. I
> > >> > > > > > > agree
> > >> > > > > > > > > > that
> > >> > > > > > > > > > > > the
> > >> > > > > > > > > > > > >> >> problem is getting complex enough that a
> > more
> > >> > > > > structured
> > >> > > > > > > > > > discussion
> > >> > > > > > > > > > > > >> would
> > >> > > > > > > > > > > > >> >> help.
> > >> > > > > > > > > > > > >> >>
> > >> > > > > > > > > > > > >> >> +1 on setting up a biweekly sync for the
> > >> metrics
> > >> > > > > > > > architecture.
> > >> > > > > > > > > > I’m
> > >> > > > > > > > > > > > >> happy
> > >> > > > > > > > > > > > >> >> to
> > >> > > > > > > > > > > > >> >> join.
> > >> > > > > > > > > > > > >> >>
> > >> > > > > > > > > > > > >> >> Yufei
> > >> > > > > > > > > > > > >> >>
> > >> > > > > > > > > > > > >> >>
> > >> > > > > > > > > > > > >> >> On Tue, Apr 21, 2026 at 2:34 PM EJ Wang <
> > >> > > > > > > > > > > > >> ej.wang.dev.interest@gmail.com>
> > >> > > > > > > > > > > > >> >> wrote:
> > >> > > > > > > > > > > > >> >>
> > >> > > > > > > > > > > > >> >> > Also, I've been looking more closely at
> > the
> > >> > > > > > *persistence
> > >> > > > > > > > > schema
> > >> > > > > > > > > > > in
> > >> > > > > > > > > > > > >> the
> > >> > > > > > > > > > > > >> >> > current metrics work*, and I think
> > there's a
> > >> > > > > structural
> > >> > > > > > > > > > rigidity
> > >> > > > > > > > > > > > >> problem
> > >> > > > > > > > > > > > >> >> > worth raising before the shape gets
> locked
> > >> in.
> > >> > > > > > > > > > > > >> >> >
> > >> > > > > > > > > > > > >> >> > Right now we have two separate tables
> > >> > > > > > > (scan_metrics_report
> > >> > > > > > > > > and
> > >> > > > > > > > > > > > >> >> > commit_metrics_report), each with ~25
> > >> flattened
> > >> > > > > columns
> > >> > > > > > > > that
> > >> > > > > > > > > > > > directly
> > >> > > > > > > > > > > > >> >> > mirror the Iceberg report fields. The
> SPI
> > >> > follows
> > >> > > > the
> > >> > > > > > > same
> > >> > > > > > > > > > split:
> > >> > > > > > > > > > > > >> >> > writeScanReport and writeCommitReport as
> > >> > separate
> > >> > > > > > > methods,
> > >> > > > > > > > > with
> > >> > > > > > > > > > > > >> per-type
> > >> > > > > > > > > > > > >> >> > record classes, converters, and model
> > >> objects.
> > >> > > *The
> > >> > > > > > > > practical
> > >> > > > > > > > > > > cost:
> > >> > > > > > > > > > > > >> >> > adding a new metric type (operational
> > >> metrics,
> > >> > > for
> > >> > > > > > > example)
> > >> > > > > > > > > > > > requires
> > >> > > > > > > > > > > > >> a
> > >> > > > > > > > > > > > >> >> new
> > >> > > > > > > > > > > > >> >> > table, a new SPI method, a new record
> > >> class, a
> > >> > > new
> > >> > > > > > model
> > >> > > > > > > > > > class, a
> > >> > > > > > > > > > > > new
> > >> > > > > > > > > > > > >> >> > converter branch, and a schema
> migration*.
> > >> > > That's a
> > >> > > > > lot
> > >> > > > > > > of
> > >> > > > > > > > > > > surface
> > >> > > > > > > > > > > > >> area
> > >> > > > > > > > > > > > >> >> > for what should be "one more kind of
> > >> metric."
> > >> > > > > > > > > > > > >> >> >
> > >> > > > > > > > > > > > >> >> > *My bias* would be toward a single
> metrics
> > >> > table
> > >> > > > with
> > >> > > > > > *a
> > >> > > > > > > > > typed
> > >> > > > > > > > > > > JSON
> > >> > > > > > > > > > > > >> >> > payload*. Something like: metric_type
> > >> (enum),
> > >> > > > > > entity_id,
> > >> > > > > > > > > > > > >> >> > table_identifier, snapshot_id
> (nullable),
> > >> > > > > received_ts,
> > >> > > > > > > > > > > > >> schema_version,
> > >> > > > > > > > > > > > >> >> and
> > >> > > > > > > > > > > > >> >> > a payload column for the metric-specific
> > >> data.
> > >> > > The
> > >> > > > > > > > > metric_type
> > >> > > > > > > > > > +
> > >> > > > > > > > > > > > >> >> > schema_version pair gives us a
> > >> > forward-compatible
> > >> > > > > > > contract
> > >> > > > > > > > > for
> > >> > > > > > > > > > > the
> > >> > > > > > > > > > > > >> >> payload
> > >> > > > > > > > > > > > >> >> > shape. Adding a new metric type becomes
> an
> > >> enum
> > >> > > > value
> > >> > > > > > > and a
> > >> > > > > > > > > > > payload
> > >> > > > > > > > > > > > >> >> schema,
> > >> > > > > > > > > > > > >> >> > not a schema migration. One thing I
> think
> > we
> > >> > need
> > >> > > > to
> > >> > > > > be
> > >> > > > > > > > > > > deliberate
> > >> > > > > > > > > > > > >> >> about is
> > >> > > > > > > > > > > > >> >> > the partition key design. If all metric
> > >> types
> > >> > > land
> > >> > > > in
> > >> > > > > > one
> > >> > > > > > > > > > table,
> > >> > > > > > > > > > > > scan
> > >> > > > > > > > > > > > >> >> > metrics at scale (high concurrency, high
> > >> > > frequency
> > >> > > > > > across
> > >> > > > > > > > > many
> > >> > > > > > > > > > > > >> tables)
> > >> > > > > > > > > > > > >> >> > could easily create hot partitions. We'd
> > >> want
> > >> > the
> > >> > > > > > > > persistence
> > >> > > > > > > > > > > layer
> > >> > > > > > > > > > > > >> to
> > >> > > > > > > > > > > > >> >> be
> > >> > > > > > > > > > > > >> >> > able to shard by entity or time range,
> and
> > >> that
> > >> > > > means
> > >> > > > > > the
> > >> > > > > > > > > > logical
> > >> > > > > > > > > > > > >> schema
> > >> > > > > > > > > > > > >> >> > needs to expose enough structure for
> > >> backends
> > >> > to
> > >> > > > > > > partition
> > >> > > > > > > > > on.
> > >> > > > > > > > > > I
> > >> > > > > > > > > > > > >> don't
> > >> > > > > > > > > > > > >> >> > think the current flattened layout gives
> > us
> > >> > that.
> > >> > > > > > > > > > > > >> >> >
> > >> > > > > > > > > > > > >> >> > This is getting complex enough that I
> > don't
> > >> > think
> > >> > > > > > ad-hoc
> > >> > > > > > > > > PR/ML
> > >> > > > > > > > > > > > >> threads
> > >> > > > > > > > > > > > >> >> > will converge well. *Would people be
> open
> > >> to a
> > >> > > > > biweekly
> > >> > > > > > > > sync
> > >> > > > > > > > > > for
> > >> > > > > > > > > > > > >> metrics
> > >> > > > > > > > > > > > >> >> > architecture?* I think 30 minutes every
> > two
> > >> > weeks
> > >> > > > > with
> > >> > > > > > > > > > interested
> > >> > > > > > > > > > > > >> >> parties
> > >> > > > > > > > > > > > >> >> > would be enough to work through the
> > schema,
> > >> SPI
> > >> > > > > shape,
> > >> > > > > > > and
> > >> > > > > > > > > read
> > >> > > > > > > > > > > API
> > >> > > > > > > > > > > > >> >> design
> > >> > > > > > > > > > > > >> >> > together. Happy to help set that up.
> > >> > > > > > > > > > > > >> >> >
> > >> > > > > > > > > > > > >> >> > -ej
> > >> > > > > > > > > > > > >> >> >
> > >> > > > > > > > > > > > >> >> > On Mon, Apr 20, 2026 at 2:19 PM EJ Wang
> <
> > >> > > > > > > > > > > > >> ej.wang.dev.interest@gmail.com
> > >> > > > > > > > > > > > >> >> >
> > >> > > > > > > > > > > > >> >> > wrote:
> > >> > > > > > > > > > > > >> >> >
> > >> > > > > > > > > > > > >> >> >> Reviewed #4115, left a comment on the
> > code
> > >> > > > > > organization
> > >> > > > > > > > > side.
> > >> > > > > > > > > > > > >> >> >>
> > >> > > > > > > > > > > > >> >> >> One thing stood out: the metrics write
> > path
> > >> > > enters
> > >> > > > > > > through
> > >> > > > > > > > > > > > >> >> >> PolarisMetricsManager on
> > MetaStoreManager,
> > >> but
> > >> > > the
> > >> > > > > new
> > >> > > > > > > > read
> > >> > > > > > > > > > path
> > >> > > > > > > > > > > > >> >> bypasses
> > >> > > > > > > > > > > > >> >> >> MetaStoreManager entirely and goes
> > >> straight to
> > >> > > > > > > > > BasePersistence
> > >> > > > > > > > > > > via
> > >> > > > > > > > > > > > >> >> >> callContext.getMetaStore(). That means
> > the
> > >> > read
> > >> > > > API
> > >> > > > > > only
> > >> > > > > > > > > works
> > >> > > > > > > > > > > for
> > >> > > > > > > > > > > > >> >> backends
> > >> > > > > > > > > > > > >> >> >> that implement BasePersistence. NoSQL
> and
> > >> > remote
> > >> > > > > > > backends
> > >> > > > > > > > > > can't
> > >> > > > > > > > > > > > >> >> participate.
> > >> > > > > > > > > > > > >> >> >>
> > >> > > > > > > > > > > > >> >> >> Stepping back, I think the metrics
> > >> subsystem
> > >> > is
> > >> > > > > > growing
> > >> > > > > > > > into
> > >> > > > > > > > > > > > >> something
> > >> > > > > > > > > > > > >> >> >> real (write + read + REST API + AuthZ +
> > >> > > > pagination)
> > >> > > > > > *but
> > >> > > > > > > > the
> > >> > > > > > > > > > > > >> >> persistence
> > >> > > > > > > > > > > > >> >> >> side is split across two layers in a
> way
> > >> > that's
> > >> > > > hard
> > >> > > > > > to
> > >> > > > > > > > > > > extend*. I
> > >> > > > > > > > > > > > >> put
> > >> > > > > > > > > > > > >> >> >> together two diagrams to show what I
> mean
> > >> (my
> > >> > > best
> > >> > > > > > > > effort).
> > >> > > > > > > > > > > > >> >> >>
> > >> > > > > > > > > > > > >> >> >> *Current state* (Diagram 1): three
> > >> interfaces
> > >> > at
> > >> > > > > three
> > >> > > > > > > > > > different
> > >> > > > > > > > > > > > >> >> levels.
> > >> > > > > > > > > > > > >> >> >> The engine-facing SPI
> > >> (PolarisMetricsReporter)
> > >> > > is
> > >> > > > > > clean.
> > >> > > > > > > > But
> > >> > > > > > > > > > > > >> >> >> PolarisMetricsManager on
> MetaStoreManager
> > >> is a
> > >> > > > > > > passthrough
> > >> > > > > > > > > to
> > >> > > > > > > > > > > > >> >> >> MetricsPersistence on BasePersistence.
> > The
> > >> > @Beta
> > >> > > > > > > > annotation
> > >> > > > > > > > > > and
> > >> > > > > > > > > > > > SPI
> > >> > > > > > > > > > > > >> >> javadoc
> > >> > > > > > > > > > > > >> >> >> are on the BasePersistence layer, while
> > the
> > >> > > actual
> > >> > > > > > > > extension
> > >> > > > > > > > > > > > points
> > >> > > > > > > > > > > > >> >> >> (PolarisMetricsReporter,
> > >> > PolarisMetricsManager)
> > >> > > > > carry
> > >> > > > > > no
> > >> > > > > > > > > > > stability
> > >> > > > > > > > > > > > >> >> >> annotation. The write path goes through
> > the
> > >> > > > > > > > MetaStoreManager
> > >> > > > > > > > > > > > layer,
> > >> > > > > > > > > > > > >> the
> > >> > > > > > > > > > > > >> >> >> read path doesn't.
> > >> > > > > > > > > > > > >> >> >>
> > >> > > > > > > > > > > > >> >> >> *What I envision* (Diagram 2): two SPIs
> > at
> > >> two
> > >> > > > > levels.
> > >> > > > > > > > > > > > >> >> >> PolarisMetricsReporter stays as the
> > >> > > engine-facing
> > >> > > > > SPI.
> > >> > > > > > > > > > > > >> >> >> PolarisMetricsManager becomes the
> > >> > backend-facing
> > >> > > > SPI
> > >> > > > > > > with
> > >> > > > > > > > > both
> > >> > > > > > > > > > > > write
> > >> > > > > > > > > > > > >> >> and
> > >> > > > > > > > > > > > >> >> >> read methods at the MetaStoreManager
> > level,
> > >> > > where
> > >> > > > > any
> > >> > > > > > > > > backend
> > >> > > > > > > > > > > > (JDBC,
> > >> > > > > > > > > > > > >> >> NoSQL,
> > >> > > > > > > > > > > > >> >> >> remote) can implement them.
> > >> MetricsPersistence
> > >> > > on
> > >> > > > > > > > > > > BasePersistence
> > >> > > > > > > > > > > > >> goes
> > >> > > > > > > > > > > > >> >> >> away. Where metrics actually land is an
> > >> > > > > implementation
> > >> > > > > > > > > detail,
> > >> > > > > > > > > > > > not a
> > >> > > > > > > > > > > > >> >> core
> > >> > > > > > > > > > > > >> >> >> interface.
> > >> > > > > > > > > > > > >> >> >>
> > >> > > > > > > > > > > > >> >> >> *Minor naming thing*:
> > >> PolarisMetricsReporter
> > >> > is
> > >> > > > > > broader
> > >> > > > > > > > than
> > >> > > > > > > > > > > what
> > >> > > > > > > > > > > > it
> > >> > > > > > > > > > > > >> >> >> actually handles. It only accepts
> Iceberg
> > >> REST
> > >> > > > > Catalog
> > >> > > > > > > > > metrics
> > >> > > > > > > > > > > > >> >> (ScanReport,
> > >> > > > > > > > > > > > >> >> >> CommitReport via MetricsReport).
> Generic
> > >> table
> > >> > > > > metrics
> > >> > > > > > > or
> > >> > > > > > > > > > > > >> operational
> > >> > > > > > > > > > > > >> >> >> metrics aren't in scope. Not blocking,
> > but
> > >> > worth
> > >> > > > > > noting
> > >> > > > > > > if
> > >> > > > > > > > > the
> > >> > > > > > > > > > > > >> metrics
> > >> > > > > > > > > > > > >> >> >> surface expands.
> > >> > > > > > > > > > > > >> >> >>
> > >> > > > > > > > > > > > >> >> >> *Rough sketch of how to get there*:
> > >> > > > > > > > > > > > >> >> >>  1.  Add read methods to
> > >> PolarisMetricsManager
> > >> > > > > > > > > > (listScanReports,
> > >> > > > > > > > > > > > >> >> >> listCommitReports) with default no-op,
> > >> same as
> > >> > > the
> > >> > > > > > > > existing
> > >> > > > > > > > > > > write
> > >> > > > > > > > > > > > >> >> methods.
> > >> > > > > > > > > > > > >> >> >> (Probably make PolarisMetricsManager
> more
> > >> > > explicit
> > >> > > > > on
> > >> > > > > > > > being
> > >> > > > > > > > > > > > Iceberg
> > >> > > > > > > > > > > > >> >> >> specific like package name or class
> name
> > >> etc.)
> > >> > > > > > > > > > > > >> >> >>  2.  Wire MetricsReportsService through
> > >> > > > > > MetaStoreManager
> > >> > > > > > > > > > instead
> > >> > > > > > > > > > > > of
> > >> > > > > > > > > > > > >> >> >> callContext.getMetaStore().
> > >> > > > > > > > > > > > >> >> >>  3.  Extract metrics persistence from
> > >> > > > > > > > > JdbcBasePersistenceImpl
> > >> > > > > > > > > > > into
> > >> > > > > > > > > > > > >> its
> > >> > > > > > > > > > > > >> >> >> own class. That file carries ~7
> > >> > > responsibilities,
> > >> > > > > > > metrics
> > >> > > > > > > > > > being
> > >> > > > > > > > > > > > one
> > >> > > > > > > > > > > > >> of
> > >> > > > > > > > > > > > >> >> them.
> > >> > > > > > > > > > > > >> >> >>  4.  Remove MetricsPersistence from
> > >> > > > BasePersistence.
> > >> > > > > > > > > > > > >> >> >>
> > >> > > > > > > > > > > > >> >> >> *None of this needs to happen in #4115.
> > >> But if
> > >> > > the
> > >> > > > > > > > direction
> > >> > > > > > > > > > > makes
> > >> > > > > > > > > > > > >> >> sense,
> > >> > > > > > > > > > > > >> >> >> it would be good to align before the
> > >> metrics
> > >> > > > surface
> > >> > > > > > > grows
> > >> > > > > > > > > > > > further.
> > >> > > > > > > > > > > > >> >> Curious
> > >> > > > > > > > > > > > >> >> >> what others think.*
> > >> > > > > > > > > > > > >> >> >>
> > >> > > > > > > > > > > > >> >> >> *My mental model note*: Level 1
> > >> > > MetaStoreManager;
> > >> > > > > > level
> > >> > > > > > > 2
> > >> > > > > > > > > > > > >> transactional
> > >> > > > > > > > > > > > >> >> >> persistence; level 3 base persistence
> > >> > > > > > > > > > > > >> >> >>
> > >> > > > > > > > > > > > >> >> >> Diagram 1
> > >> > > > > > > > > > > > >> >> >> <
> > >> > > > > > > > > > > > >> >>
> > >> > > > > > > > > > > > >>
> > >> > > > > > > > > > > >
> > >> > > > > > > > > > >
> > >> > > > > > > > > >
> > >> > > > > > > > >
> > >> > > > > > > >
> > >> > > > > > >
> > >> > > > > >
> > >> > > > >
> > >> > > >
> > >> > >
> > >> >
> > >>
> >
> https://www.plantuml.com/plantuml/uml/bLHDR-Cs4BthLmpIYupw0zbkKQ1r3M-S7Bp8xhhM7WCOb3IM65EaGD9EX2RzxHrHb4CxRelwa4YSDu_lpOVcnZ9jzvM8BBS2uGjQpJC3dtHMSekPtMk44IpsMgEqa5XcCOhCZikQQLP1pR8TAp2n3ILhmZDP20m0fcIvUkAoW2qJXd9z1bpToO9BX3WXu0ucy5rpgGPNm0nW5_epUWtm2Ue3pn3kMOFQmKntGZW0BYtgBSi8k5A2QMwybJNMIbFiGSR9QZc4nUqIvikStF0jHprua5C-amge42aNt3R0f5JaaoivdV2Pkqbx4hee4ymOkBh5BTiB-_uIeGeo8zL8rPsPl4DktdEiK1jkB1NdZCRbrSTecDe_mlHbF0wvBmCkaOH5_S8a_TTTKI6-nmCAkEw4LpxsZ-LbYLKQFKMNOgf_wuM7_bV9gOer5SYMMksBSWXFcbi49KNZXNLicwfe3TETC7gPdPqI7uBcHMb1RSzYq34c6PDUM9mn8HRsUTZEiDBve3NjVZumBj0U7SS37mGO7vcwtiK-_pU7U7L_f-digo9YbhSwIfMRwIITKGXbxdIUTCGF1SeCJxloKsU-3k9ddRbX1eDq1q_fx1JbBGT0glVyXimDuP4TQ5qpCAmnGEj2s_6n5mtn1z-97-63itFQZLPO1Ev2tu_WF7Ju-VPc0Skg5bYXxBhkY1xpD7EM_7fyflSpIsqMgVth5xhVr4eQxWQ8enaSAJQSG16yFSDuJ798rrcXr_3n-lfdk7icQjEBmFujL7AodiP_Y4Z7-YxvtZNs4zMgpNTl6tF8sglyPsmqchrjvQ-m-aP94r-TwCA2Ka8upPJZwtvSpoYCXkYMZU2NXvRMBfq9P3i3Le4VAZUAlUZ_oPKsxPgY0Q_BSKLkyr9bhQhQrJjo_x3TPlIB0DPjnMfcIoYP0QaYw1a0fTKDr8fB6ntNuvmoL1ZGkXa69Njh43zf9GiGxHQrA_jDYWRSzF5--WmTVrN97_Sm8LbLUy_lGBmLanJjFkDlGkRqjA_4tm00
> > >> > > > > > > > > > > > >> >> >
> > >> > > > > > > > > > > > >> >> >> :
> > >> > > > > > > > > > > > >> >> >>
> > >> > > > > > > > > > > > >> >> >> [image: image.png]
> > >> > > > > > > > > > > > >> >> >>
> > >> > > > > > > > > > > > >> >> >> Diagram 2
> > >> > > > > > > > > > > > >> >> >> <
> > >> > > > > > > > > > > > >> >>
> > >> > > > > > > > > > > > >>
> > >> > > > > > > > > > > >
> > >> > > > > > > > > > >
> > >> > > > > > > > > >
> > >> > > > > > > > >
> > >> > > > > > > >
> > >> > > > > > >
> > >> > > > > >
> > >> > > > >
> > >> > > >
> > >> > >
> > >> >
> > >>
> >
> https://www.plantuml.com/plantuml/uml/VLLDR-8m4BtdLupO2sWBLVU8AaGB7AXAbssGzb896SS9RXqxjKqBwkv_tt7iV43fdaZYDpFlpRmnOsE9jhjSH9PRmM31hERKm8scMsuPjJlDe0yheZDc8RR4iYWoBrmMH9CS2a9VICPYUy1OZN0YCy5Q0BCbYNhdCeEK28En8G8wCvbnoQ0R8_05Bc6bkLIz3X03p1zzH7zR-9ZfDquPt9C3qoNCX2yV4G2NbkcKu5jdgGJHt0GbZwnG6i-UP3TUpk5gM6Ldqke350eZUqzoCft3U9xWHvxoa5-7K4nF1J46EbEMafsmdrCBbQ44gVggy18IZrn_ph5asd1ZiIKdQSgueZvjXrQFSFrdC3YN-nXmBacxbGiYyLVxLaBtdhqn0LSzdBDhqQtQoOJeGyad3z0lUqnYgpGB6Ns8oVyta00Dy_WnX0tIOZ8v6SYxHll1TrH6aejAik-mh-AphVFCwSUQqFypElag5QRGFDjQKEd96K1P8QP41c9TzA_IIQyvdAWyv_RSiS3skb0_EzDDkK2v5xWF6MiGFlvhpFLcD2Dq2pml14gaF67eQkmd8gulDoC4kSOu6KVpkvlUJg1RTbWISU40RdBUUS_9XfRZ2dwxm_SW8LYFISgm_MnlDQ6M9P1gbKEc4X-2pH_FvJCkCqm9pbVjD6LrwdLeOrDWfOaqc8Wh9BE85oNKxkNQ6o4yGRy_Eae0G_G8tZv81d3bHDB23WOdisohVr3nh_j6lbSjbNaLRTc8UgtPbAU1J_tygOfZX9DWEJeHDvYx-qmSi5FgNLPZwHrHcUsncGQ5-skhUclpE5fo4ounpFauYrUbkU6ccfnxMvitwag4IyerhTxj8In_Oj1bDO4pQru674loYrGlULHLEGCjwJJ8gDoVZR8MxO4BT3IzRvIcAQKezC6xpziGnTyImrfEGyJI_OcKfgtxIvnTqFEMS17L9Z-jsARN5FmTheP7HtSdtOMT0B4GY2FYHXxgQmMtj2bRqiLFGapiVe1_QVKDrkqXcm83aFEXnMYCZ-xlyHy
> > >> > > > > > > > > > > > >> >> >
> > >> > > > > > > > > > > > >> >> >> :
> > >> > > > > > > > > > > > >> >> >> [image: image.png]
> > >> > > > > > > > > > > > >> >> >>
> > >> > > > > > > > > > > > >> >> >>  -ej
> > >> > > > > > > > > > > > >> >> >>
> > >> > > > > > > > > > > > >> >> >> On Wed, Apr 15, 2026 at 8:22 AM Dmitri
> > >> > > > Bourlatchkov
> > >> > > > > <
> > >> > > > > > > > > > > > >> dimas@apache.org>
> > >> > > > > > > > > > > > >> >> >> wrote:
> > >> > > > > > > > > > > > >> >> >>
> > >> > > > > > > > > > > > >> >> >>> Hi All,
> > >> > > > > > > > > > > > >> >> >>>
> > >> > > > > > > > > > > > >> >> >>> Heads up: The current state of PR
> [4115]
> > >> > looks
> > >> > > > > pretty
> > >> > > > > > > > solid
> > >> > > > > > > > > > to
> > >> > > > > > > > > > > > me.
> > >> > > > > > > > > > > > >> I
> > >> > > > > > > > > > > > >> >> >>> believe this PR is approaching a
> > mergeable
> > >> > > > > condition.
> > >> > > > > > > > > > > > >> >> >>>
> > >> > > > > > > > > > > > >> >> >>> Please post your reviews if you have
> any
> > >> > > > comments.
> > >> > > > > > > > > > > > >> >> >>>
> > >> > > > > > > > > > > > >> >> >>> [4115]
> > >> > > > https://github.com/apache/polaris/pull/4115
> > >> > > > > > > > > > > > >> >> >>>
> > >> > > > > > > > > > > > >> >> >>> Thanks,
> > >> > > > > > > > > > > > >> >> >>> Dmitri.
> > >> > > > > > > > > > > > >> >> >>>
> > >> > > > > > > > > > > > >> >> >>> On Tue, Mar 3, 2026 at 3:29 PM Anand
> > Kumar
> > >> > > > Sankaran
> > >> > > > > > via
> > >> > > > > > > > > dev <
> > >> > > > > > > > > > > > >> >> >>> dev@polaris.apache.org> wrote:
> > >> > > > > > > > > > > > >> >> >>>
> > >> > > > > > > > > > > > >> >> >>> > Hi Yufei and Dmitri,
> > >> > > > > > > > > > > > >> >> >>> >
> > >> > > > > > > > > > > > >> >> >>> > Here is a proposal for the REST
> > >> endpoints
> > >> > for
> > >> > > > > > metrics
> > >> > > > > > > > and
> > >> > > > > > > > > > > > events.
> > >> > > > > > > > > > > > >> >> >>> >
> > >> > > > > > > > > > > > >> >> >>> >
> > >> > > > > > https://github.com/apache/polaris/pull/3924/changes
> > >> > > > > > > > > > > > >> >> >>> >
> > >> > > > > > > > > > > > >> >> >>> > I did not see any precursors for
> > >> raising a
> > >> > PR
> > >> > > > for
> > >> > > > > > > > > > proposals,
> > >> > > > > > > > > > > so
> > >> > > > > > > > > > > > >> >> trying
> > >> > > > > > > > > > > > >> >> >>> > this.  Please let me know what you
> > >> think.
> > >> > > > > > > > > > > > >> >> >>> >
> > >> > > > > > > > > > > > >> >> >>> > -
> > >> > > > > > > > > > > > >> >> >>> > Anand
> > >> > > > > > > > > > > > >> >> >>> >
> > >> > > > > > > > > > > > >> >> >>> > From: Anand Kumar Sankaran <
> > >> > > > > > > anand.sankaran@workday.com
> > >> > > > > > > > >
> > >> > > > > > > > > > > > >> >> >>> > Date: Monday, March 2, 2026 at
> > 10:25 AM
> > >> > > > > > > > > > > > >> >> >>> > To: dev@polaris.apache.org <
> > >> > > > > dev@polaris.apache.org
> > >> > > > > > >
> > >> > > > > > > > > > > > >> >> >>> > Subject: Re: Polaris Telemetry and
> > Audit
> > >> > > Trail
> > >> > > > > > > > > > > > >> >> >>> >
> > >> > > > > > > > > > > > >> >> >>> > About the REST API, based on my use
> > >> cases:
> > >> > > > > > > > > > > > >> >> >>> >
> > >> > > > > > > > > > > > >> >> >>> >
> > >> > > > > > > > > > > > >> >> >>> >   1.
> > >> > > > > > > > > > > > >> >> >>> > I want to be able to query commit
> > >> metrics
> > >> > to
> > >> > > > > track
> > >> > > > > > > > files
> > >> > > > > > > > > > > added
> > >> > > > > > > > > > > > /
> > >> > > > > > > > > > > > >> >> >>> removed
> > >> > > > > > > > > > > > >> >> >>> > per commit, along with record
> counts.
> > >> The
> > >> > > > > ingestion
> > >> > > > > > > > > > pipeline
> > >> > > > > > > > > > > > that
> > >> > > > > > > > > > > > >> >> >>> writes
> > >> > > > > > > > > > > > >> >> >>> > this data is owned by us and we are
> > >> > > guaranteed
> > >> > > > to
> > >> > > > > > > write
> > >> > > > > > > > > > this
> > >> > > > > > > > > > > > >> >> >>> information
> > >> > > > > > > > > > > > >> >> >>> > for each write.
> > >> > > > > > > > > > > > >> >> >>> >   2.
> > >> > > > > > > > > > > > >> >> >>> > I want to be able to query scan
> > metrics
> > >> for
> > >> > > > > read. I
> > >> > > > > > > > > > > understand
> > >> > > > > > > > > > > > >> >> clients
> > >> > > > > > > > > > > > >> >> >>> do
> > >> > > > > > > > > > > > >> >> >>> > not fulfill this requirement.
> > >> > > > > > > > > > > > >> >> >>> >   3.
> > >> > > > > > > > > > > > >> >> >>> > I want to be able to query the
> events
> > >> table
> > >> > > > > (events
> > >> > > > > > > are
> > >> > > > > > > > > > > > >> persisted) -
> > >> > > > > > > > > > > > >> >> >>> this
> > >> > > > > > > > > > > > >> >> >>> > may supersede #2, I am not sure yet.
> > >> > > > > > > > > > > > >> >> >>> >
> > >> > > > > > > > > > > > >> >> >>> > All this information is in the JDBC
> > >> based
> > >> > > > > > persistence
> > >> > > > > > > > > model
> > >> > > > > > > > > > > and
> > >> > > > > > > > > > > > >> is
> > >> > > > > > > > > > > > >> >> >>> > persisted in the metastore. I
> > currently
> > >> > don’t
> > >> > > > > have
> > >> > > > > > a
> > >> > > > > > > > need
> > >> > > > > > > > > > to
> > >> > > > > > > > > > > > >> query
> > >> > > > > > > > > > > > >> >> >>> > prometheus or open telemetry. I do
> > >> publish
> > >> > > some
> > >> > > > > > > events
> > >> > > > > > > > to
> > >> > > > > > > > > > > > >> Prometheus
> > >> > > > > > > > > > > > >> >> >>> and
> > >> > > > > > > > > > > > >> >> >>> > they are forwarded to our dashboards
> > >> > > elsewhere.
> > >> > > > > > > > > > > > >> >> >>> >
> > >> > > > > > > > > > > > >> >> >>> > About the CLI utilities, I meant the
> > >> admin
> > >> > > user
> > >> > > > > > > > > utilities.
> > >> > > > > > > > > > In
> > >> > > > > > > > > > > > >> one of
> > >> > > > > > > > > > > > >> >> >>> the
> > >> > > > > > > > > > > > >> >> >>> > earliest drafts of my proposal,
> > Prashant
> > >> > > > > mentioned
> > >> > > > > > > that
> > >> > > > > > > > > the
> > >> > > > > > > > > > > > >> metrics
> > >> > > > > > > > > > > > >> >> >>> tables
> > >> > > > > > > > > > > > >> >> >>> > can grow indefinitely and that a
> > similar
> > >> > > > problem
> > >> > > > > > > exists
> > >> > > > > > > > > > with
> > >> > > > > > > > > > > > the
> > >> > > > > > > > > > > > >> >> events
> > >> > > > > > > > > > > > >> >> >>> > table as well. We discussed that
> > >> cleaning
> > >> > up
> > >> > > of
> > >> > > > > old
> > >> > > > > > > > > records
> > >> > > > > > > > > > > > from
> > >> > > > > > > > > > > > >> >> both
> > >> > > > > > > > > > > > >> >> >>> > metrics tables and events tables can
> > be
> > >> > done
> > >> > > > via
> > >> > > > > a
> > >> > > > > > > CLI
> > >> > > > > > > > > > > utility.
> > >> > > > > > > > > > > > >> >> >>> >
> > >> > > > > > > > > > > > >> >> >>> > I see that Yufei has covered the
> > >> discussion
> > >> > > > about
> > >> > > > > > > > > > > datasources.
> > >> > > > > > > > > > > > >> >> >>> >
> > >> > > > > > > > > > > > >> >> >>> > -
> > >> > > > > > > > > > > > >> >> >>> > Anand
> > >> > > > > > > > > > > > >> >> >>> >
> > >> > > > > > > > > > > > >> >> >>> >
> > >> > > > > > > > > > > > >> >> >>> >
> > >> > > > > > > > > > > > >> >> >>> > From: Yufei Gu <
> flyrain000@gmail.com>
> > >> > > > > > > > > > > > >> >> >>> > Date: Friday, February 27, 2026 at
> > >> 9:54 PM
> > >> > > > > > > > > > > > >> >> >>> > To: dev@polaris.apache.org <
> > >> > > > > dev@polaris.apache.org
> > >> > > > > > >
> > >> > > > > > > > > > > > >> >> >>> > Subject: Re: Polaris Telemetry and
> > Audit
> > >> > > Trail
> > >> > > > > > > > > > > > >> >> >>> >
> > >> > > > > > > > > > > > >> >> >>> > This Message Is From an External
> > Sender
> > >> > > > > > > > > > > > >> >> >>> > This message came from outside your
> > >> > > > organization.
> > >> > > > > > > > > > > > >> >> >>> > Report Suspicious<
> > >> > > > > > > > > > > > >> >> >>> >
> > >> > > > > > > > > > > > >> >> >>>
> > >> > > > > > > > > > > > >> >>
> > >> > > > > > > > > > > > >>
> > >> > > > > > > > > > > >
> > >> > > > > > > > > > >
> > >> > > > > > > > > >
> > >> > > > > > > > >
> > >> > > > > > > >
> > >> > > > > > >
> > >> > > > > >
> > >> > > > >
> > >> > > >
> > >> > >
> > >> >
> > >>
> >
> https://us-phishalarm-ewt.proofpoint.com/EWT/v1/Iz9xO38YGHZK!YhNDZABkHi1B699ote2uMwpOZw8i0QMCGO2Szc-HshuABGhGvwPJcymE6G2oUUxtS8xDkSrtGTPm_I3QnVDHoLMk50m9v8z_nZKTkd-bnVUbreF1u0WnfV_X5eYevZl_$
> > >> > > > > > > > > > > > >> >> >>> > >
> > >> > > > > > > > > > > > >> >> >>> >
> > >> > > > > > > > > > > > >> >> >>> >
> > >> > > > > > > > > > > > >> >> >>> > As I mentioned in
> > >> > > > > > > > > > > > >> >> >>> >
> > >> > > > > > > > > > > > >> >> >>>
> > >> > > > > > > > > > > > >> >>
> > >> > > > > > > > > > > > >>
> > >> > > > > > > > > > > >
> > >> > > > > > > > > > >
> > >> > > > > > > > > >
> > >> > > > > > > > >
> > >> > > > > > > >
> > >> > > > > > >
> > >> > > > > >
> > >> > > > >
> > >> > > >
> > >> > >
> > >> >
> > >>
> >
> https://urldefense.com/v3/__https://github.com/apache/polaris/issues/3890__;!!Iz9xO38YGHZK!5EuyFFkk3vhRWVIRvQAWBSQfpJkTMA9HxugzDwXmN0LPPqhEFxYkFRGVhtb8AqUwXtDh2OplcMnbMDHKOxrvDU0$
> > >> > > > > > > > > > > > >> >> >>> ,
> > >> > > > > > > > > > > > >> >> >>> > supporting
> > >> > > > > > > > > > > > >> >> >>> > multiple data sources is not a
> trivial
> > >> > > change.
> > >> > > > I
> > >> > > > > > > would
> > >> > > > > > > > > > > strongly
> > >> > > > > > > > > > > > >> >> >>> recommend
> > >> > > > > > > > > > > > >> >> >>> > starting with a design document to
> > >> > carefully
> > >> > > > > > evaluate
> > >> > > > > > > > the
> > >> > > > > > > > > > > > >> >> architectural
> > >> > > > > > > > > > > > >> >> >>> > implications and long term impact.
> > >> > > > > > > > > > > > >> >> >>> >
> > >> > > > > > > > > > > > >> >> >>> > A REST endpoint to query metrics
> seems
> > >> > > > reasonable
> > >> > > > > > > given
> > >> > > > > > > > > the
> > >> > > > > > > > > > > > >> current
> > >> > > > > > > > > > > > >> >> >>> JDBC
> > >> > > > > > > > > > > > >> >> >>> > based persistence model. That said,
> we
> > >> may
> > >> > > also
> > >> > > > > > > > consider
> > >> > > > > > > > > > > > >> alternative
> > >> > > > > > > > > > > > >> >> >>> > storage models. For example, if we
> > later
> > >> > > adopt
> > >> > > > a
> > >> > > > > > time
> > >> > > > > > > > > > series
> > >> > > > > > > > > > > > >> system
> > >> > > > > > > > > > > > >> >> >>> such as
> > >> > > > > > > > > > > > >> >> >>> > Prometheus to store metrics, the
> query
> > >> > model
> > >> > > > and
> > >> > > > > > > access
> > >> > > > > > > > > > > > patterns
> > >> > > > > > > > > > > > >> >> would
> > >> > > > > > > > > > > > >> >> >>> be
> > >> > > > > > > > > > > > >> >> >>> > fundamentally different. Designing
> the
> > >> REST
> > >> > > API
> > >> > > > > > > without
> > >> > > > > > > > > > > > >> considering
> > >> > > > > > > > > > > > >> >> >>> these
> > >> > > > > > > > > > > > >> >> >>> > potential evolutions may limit
> > >> flexibility.
> > >> > > I'd
> > >> > > > > > > suggest
> > >> > > > > > > > > to
> > >> > > > > > > > > > > > start
> > >> > > > > > > > > > > > >> >> with
> > >> > > > > > > > > > > > >> >> >>> the
> > >> > > > > > > > > > > > >> >> >>> > use case.
> > >> > > > > > > > > > > > >> >> >>> >
> > >> > > > > > > > > > > > >> >> >>> > Yufei
> > >> > > > > > > > > > > > >> >> >>> >
> > >> > > > > > > > > > > > >> >> >>> >
> > >> > > > > > > > > > > > >> >> >>> > On Fri, Feb 27, 2026 at 3:42 PM
> Dmitri
> > >> > > > > > Bourlatchkov <
> > >> > > > > > > > > > > > >> >> dimas@apache.org>
> > >> > > > > > > > > > > > >> >> >>> > wrote:
> > >> > > > > > > > > > > > >> >> >>> >
> > >> > > > > > > > > > > > >> >> >>> > > Hi Anand,
> > >> > > > > > > > > > > > >> >> >>> > >
> > >> > > > > > > > > > > > >> >> >>> > > Sharing my view... subject to
> > >> discussion:
> > >> > > > > > > > > > > > >> >> >>> > >
> > >> > > > > > > > > > > > >> >> >>> > > 1. Adding non-IRC REST API to
> > Polaris
> > >> is
> > >> > > > > > perfectly
> > >> > > > > > > > > fine.
> > >> > > > > > > > > > > > >> >> >>> > >
> > >> > > > > > > > > > > > >> >> >>> > > Figuring out specific endpoint
> URIs
> > >> and
> > >> > > > > payloads
> > >> > > > > > > > might
> > >> > > > > > > > > > > > require
> > >> > > > > > > > > > > > >> a
> > >> > > > > > > > > > > > >> >> few
> > >> > > > > > > > > > > > >> >> >>> > > roundtrips, so opening a separate
> > >> thread
> > >> > > for
> > >> > > > > that
> > >> > > > > > > > might
> > >> > > > > > > > > > be
> > >> > > > > > > > > > > > >> best.
> > >> > > > > > > > > > > > >> >> >>> > > Contributors commonly create
> Google
> > >> Docs
> > >> > > for
> > >> > > > > new
> > >> > > > > > > API
> > >> > > > > > > > > > > > proposals
> > >> > > > > > > > > > > > >> too
> > >> > > > > > > > > > > > >> >> >>> (they
> > >> > > > > > > > > > > > >> >> >>> > > fairly easy to update as the email
> > >> > > discussion
> > >> > > > > > > > > > progresses).
> > >> > > > > > > > > > > > >> >> >>> > >
> > >> > > > > > > > > > > > >> >> >>> > > There was a suggestion to try
> > Markdown
> > >> > > (with
> > >> > > > > PRs)
> > >> > > > > > > for
> > >> > > > > > > > > > > > proposals
> > >> > > > > > > > > > > > >> >> [1]
> > >> > > > > > > > > > > > >> >> >>> ...
> > >> > > > > > > > > > > > >> >> >>> > > feel free to give it a try if you
> > are
> > >> > > > > comfortable
> > >> > > > > > > > with
> > >> > > > > > > > > > > that.
> > >> > > > > > > > > > > > >> >> >>> > >
> > >> > > > > > > > > > > > >> >> >>> > > 2. Could you clarify whether you
> > mean
> > >> end
> > >> > > > user
> > >> > > > > > > > > utilities
> > >> > > > > > > > > > or
> > >> > > > > > > > > > > > >> admin
> > >> > > > > > > > > > > > >> >> >>> user
> > >> > > > > > > > > > > > >> >> >>> > > utilities? In the latter case
> those
> > >> might
> > >> > > be
> > >> > > > > more
> > >> > > > > > > > > > suitable
> > >> > > > > > > > > > > > for
> > >> > > > > > > > > > > > >> the
> > >> > > > > > > > > > > > >> >> >>> Admin
> > >> > > > > > > > > > > > >> >> >>> > > CLI (java) not the Python CLI,
> IMHO.
> > >> > > > > > > > > > > > >> >> >>> > >
> > >> > > > > > > > > > > > >> >> >>> > > Why would these utilities be
> common
> > >> with
> > >> > > > > events?
> > >> > > > > > > > IMHO,
> > >> > > > > > > > > > > event
> > >> > > > > > > > > > > > >> use
> > >> > > > > > > > > > > > >> >> >>> cases
> > >> > > > > > > > > > > > >> >> >>> > are
> > >> > > > > > > > > > > > >> >> >>> > > distinct from scan/commit metrics.
> > >> > > > > > > > > > > > >> >> >>> > >
> > >> > > > > > > > > > > > >> >> >>> > > 3. I'd prefer separating metrics
> > >> > > persistence
> > >> > > > > from
> > >> > > > > > > > > > MetaStore
> > >> > > > > > > > > > > > >> >> >>> persistence
> > >> > > > > > > > > > > > >> >> >>> > at
> > >> > > > > > > > > > > > >> >> >>> > > the code level, so that they could
> > be
> > >> > mixed
> > >> > > > and
> > >> > > > > > > > matched
> > >> > > > > > > > > > > > >> >> >>> independently.
> > >> > > > > > > > > > > > >> >> >>> > The
> > >> > > > > > > > > > > > >> >> >>> > > separate datasource question will
> > >> become
> > >> > a
> > >> > > > > > > non-issue
> > >> > > > > > > > > with
> > >> > > > > > > > > > > > that
> > >> > > > > > > > > > > > >> >> >>> approach,
> > >> > > > > > > > > > > > >> >> >>> > I
> > >> > > > > > > > > > > > >> >> >>> > > guess.
> > >> > > > > > > > > > > > >> >> >>> > >
> > >> > > > > > > > > > > > >> >> >>> > > The rationale for separating scan
> > >> metrics
> > >> > > and
> > >> > > > > > > > metastore
> > >> > > > > > > > > > > > >> >> persistence
> > >> > > > > > > > > > > > >> >> >>> is
> > >> > > > > > > > > > > > >> >> >>> > that
> > >> > > > > > > > > > > > >> >> >>> > > "cascading deletes" between them
> are
> > >> > hardly
> > >> > > > > ever
> > >> > > > > > > > > > required.
> > >> > > > > > > > > > > > >> >> >>> Furthermore,
> > >> > > > > > > > > > > > >> >> >>> > the
> > >> > > > > > > > > > > > >> >> >>> > > data and query patterns are very
> > >> > different
> > >> > > so
> > >> > > > > > > > different
> > >> > > > > > > > > > > > >> >> technologies
> > >> > > > > > > > > > > > >> >> >>> > might
> > >> > > > > > > > > > > > >> >> >>> > > be beneficial in each case.
> > >> > > > > > > > > > > > >> >> >>> > >
> > >> > > > > > > > > > > > >> >> >>> > > [1]
> > >> > > > > > > > > > > > >> >> >>> >
> > >> > > > > > > > > > > > >> >> >>>
> > >> > > > > > > > > > > > >> >>
> > >> > > > > > > > > > > > >>
> > >> > > > > > > > > > > >
> > >> > > > > > > > > > >
> > >> > > > > > > > > >
> > >> > > > > > > > >
> > >> > > > > > > >
> > >> > > > > > >
> > >> > > > > >
> > >> > > > >
> > >> > > >
> > >> > >
> > >> >
> > >>
> >
> https://urldefense.com/v3/__https://lists.apache.org/thread/yto2wp982t43h1mqjwnslswhws5z47cy__;!!Iz9xO38YGHZK!5EuyFFkk3vhRWVIRvQAWBSQfpJkTMA9HxugzDwXmN0LPPqhEFxYkFRGVhtb8AqUwXtDh2OplcMnbMDHKxYDakNU$
> > >> > > > > > > > > > > > >> >> >>> > >
> > >> > > > > > > > > > > > >> >> >>> > > Cheers,
> > >> > > > > > > > > > > > >> >> >>> > > Dmitri.
> > >> > > > > > > > > > > > >> >> >>> > >
> > >> > > > > > > > > > > > >> >> >>> > > On Fri, Feb 27, 2026 at 6:19 PM
> > Anand
> > >> > Kumar
> > >> > > > > > > Sankaran
> > >> > > > > > > > > via
> > >> > > > > > > > > > > dev
> > >> > > > > > > > > > > > <
> > >> > > > > > > > > > > > >> >> >>> > > dev@polaris.apache.org> wrote:
> > >> > > > > > > > > > > > >> >> >>> > >
> > >> > > > > > > > > > > > >> >> >>> > > > Thanks all. This PR is merged
> now.
> > >> > > > > > > > > > > > >> >> >>> > > >
> > >> > > > > > > > > > > > >> >> >>> > > > Here are the follow-up features
> /
> > >> work
> > >> > > > > needed.
> > >> > > > > > > > These
> > >> > > > > > > > > > > were
> > >> > > > > > > > > > > > >> all
> > >> > > > > > > > > > > > >> >> >>> part of
> > >> > > > > > > > > > > > >> >> >>> > > the
> > >> > > > > > > > > > > > >> >> >>> > > > merged PR at some point in time
> > and
> > >> > were
> > >> > > > > > removed
> > >> > > > > > > to
> > >> > > > > > > > > > > reduce
> > >> > > > > > > > > > > > >> >> scope.
> > >> > > > > > > > > > > > >> >> >>> > > >
> > >> > > > > > > > > > > > >> >> >>> > > > Please let me know what you
> think.
> > >> > > > > > > > > > > > >> >> >>> > > >
> > >> > > > > > > > > > > > >> >> >>> > > >
> > >> > > > > > > > > > > > >> >> >>> > > >   1.  A REST API to paginate
> > through
> > >> > > table
> > >> > > > > > > metrics.
> > >> > > > > > > > > > This
> > >> > > > > > > > > > > > >> will be
> > >> > > > > > > > > > > > >> >> >>> > non-IRC
> > >> > > > > > > > > > > > >> >> >>> > > > standard addition.
> > >> > > > > > > > > > > > >> >> >>> > > >   2.  Utilities for managing old
> > >> > records,
> > >> > > > > > should
> > >> > > > > > > be
> > >> > > > > > > > > > > common
> > >> > > > > > > > > > > > >> with
> > >> > > > > > > > > > > > >> >> >>> events.
> > >> > > > > > > > > > > > >> >> >>> > > > There was some discussion that
> it
> > >> > belongs
> > >> > > > to
> > >> > > > > > the
> > >> > > > > > > > CLI.
> > >> > > > > > > > > > > > >> >> >>> > > >   3.  Separate datasource
> > (metrics,
> > >> > > events,
> > >> > > > > > even
> > >> > > > > > > > > other
> > >> > > > > > > > > > > > >> tables?).
> > >> > > > > > > > > > > > >> >> >>> > > >
> > >> > > > > > > > > > > > >> >> >>> > > >
> > >> > > > > > > > > > > > >> >> >>> > > > Anything else?
> > >> > > > > > > > > > > > >> >> >>> > > >
> > >> > > > > > > > > > > > >> >> >>> > > > -
> > >> > > > > > > > > > > > >> >> >>> > > > Anand
> > >> > > > > > > > > > > > >> >> >>> > > >
> > >> > > > > > > > > > > > >> >> >>> > > >
> > >> > > > > > > > > > > > >> >> >>> > >
> > >> > > > > > > > > > > > >> >> >>> >
> > >> > > > > > > > > > > > >> >> >>> >
> > >> > > > > > > > > > > > >> >> >>>
> > >> > > > > > > > > > > > >> >> >>
> > >> > > > > > > > > > > > >> >>
> > >> > > > > > > > > > > > >> >
> > >> > > > > > > > > > > > >>
> > >> > > > > > > > > > > > >
> > >> > > > > > > > > > > >
> > >> > > > > > > > > > >
> > >> > > > > > > > > >
> > >> > > > > > > > >
> > >> > > > > > > >
> > >> > > > > > >
> > >> > > > > >
> > >> > > > >
> > >> > > >
> > >> > >
> > >> >
> > >>
> > >
> >
>

Re: Proposal for REST endpoints for table metrics and events

[Yufei Gu <fl...@gmail.com>]

Hi all,

Thanks for driving this work forward. Building this as a POC makes perfect
sense if the goal is to validate the idea and gather feedback. Should we
merge a POC into the main branch before agreeing on the overall direction?
To me, that feels like a separate discussion from whether the
implementation itself looks reasonable.

One concern that I don't think has been fully addressed is whether we need
to expose metrics consumption through a Polaris REST endpoint at all.

My concern is less about the API itself and more about the boundary Polaris
should own. The proposed REST API is tied to a particular persistence
schema, but that schema is just one possible implementation. While the
current implementation stores metrics in the metastore, other deployments
may choose a KV store, a dedicated metrics warehouse, or another backend
entirely. In those cases, exposing a Polaris REST API over a specific
schema feels like coupling the API to an implementation detail.

Polaris isn't intended to become a full metrics system. I'd expect it to
focus on producing metrics and let deployments choose the most appropriate
storage and consumption model for their environment.

I'm not opposed to exploring this direction, but we should first align on
whether exposing a REST API through Polaris is the right long term approach
before merging it into the main branch.

Thanks,
Yufei

Yufei


On Mon, Jun 29, 2026 at 9:27 AM Dmitri Bourlatchkov <di...@apache.org>
wrote:

> Hi All,
>
> It looks like Anand addressed all action items in [4115].
>
> Please review the latest state of the code. The PR LGTM and I approved in
> GH from my side.
>
> [4115] https://github.com/apache/polaris/pull/4115
>
> Thanks,
> Dmitri.
>
> On Thu, Jun 25, 2026 at 1:56 PM Dmitri Bourlatchkov <di...@apache.org>
> wrote:
>
> > Hi All,
> >
> > Recap of the discussion in today's Community Sync:
> >
> > * The modular approach for Metrics SPI(s) and implementation seems to
> have
> > community consensus (please comment if you still have concerns)
> >
> > * What modules / feature flags are enabled in Apache Polaris images by
> > default will be discussed later (when the feature is close to
> completion).
> >
> > In any case, users will be able to turn the full feature on/off at their
> > discretion, regardless of the default flag values.
> >
> > * It is generally preferable to have smaller PRs and introduce new
> > features gradually to simplify reviews.
> >
> > What this means for the Metrics API, IMHO is:
> >
> > a) The scope of PR 4115 is probably ok as it stands right now. If someone
> > wants to propose splitting it further, please comment.
> >
> > b) After 4115 is merged, Metrics Persistence PRs will follow, with
> smaller
> > scope per PR as appropriate for the nature of the code changes.
> >
> > * The major concern was about exposing the new Metrics REST API to end
> > users. The nature of this concern was about avoiding the (false)
> impression
> > that the new API is a firm and adopted Polaris specification.
> >
> > I think this concern is reasonable.
> >
> > I propose that Anand update the documentation sections of PR 4115 to
> > emphasise that the API is currently an experimental / POC API at this
> time
> > and is subject to change (including breaking changes). I believe it
> already
> > has the "beta" label. So this request is mostly for doc changes to
> clarify
> > user expectations, since the "beta" label alone may not be interpreted
> the
> > same way by all users.
> >
> > I hope this update will resolve all remaining concerns with 4115 and
> allow
> > for it to be merged so that the community will be able to proceed to
> > follow-up PRs (as noted above).
> >
> > Please add/correct me if I missed anything relevant.
> >
> > Thanks,
> > Dmitri.
> >
> > On Wed, Jun 24, 2026 at 8:11 PM Yufei Gu <fl...@gmail.com> wrote:
> >
> >> I do not think we have agreement on the query REST APIs yet.
> >>
> >> In particular, the current query API shape seems highly coupled to the
> >> *example[1]* metrics in the Iceberg REST spec. I do not think that is a
> >> good way to design a REST API contract. For example, fields like
> >> resultDataFiles, resultDeleteFiles, totalFileSizeBytes,
> >> totalDataManifests,
> >> totalDeleteManifests, and scannedDataManifests are very specific to one
> >> style of scan metric. That makes the API look more like a direct
> >> projection
> >> of the current example payload than a general query model.
> >>
> >> My preference is that Polaris should not define or strongly opine on the
> >> Iceberg scan or commit metrics query REST API at this point. Polaris can
> >> resolve and authorize the table, accept the metrics report, and delegate
> >> it
> >> to the configured reporting path. Durable storage, filtering,
> dashboards,
> >> and query APIs should remain downstream impl. choices unless and until
> the
> >> Iceberg community standardizes a query contract, which I doubt will
> >> happen.
> >>
> >> So I do not think the intake path has to be blocked, but I would prefer
> >> not
> >> to merge a Polaris specific query API that may be hard to evolve later.
> >>
> >> 1.
> >>
> >>
> https://github.com/apache/iceberg/blob/2a6c556c0c883c04c6d3fbd68e6aeda36b91e0aa/open-api/rest-catalog-open-api.yaml#L4152
> >>
> >> Yufei
> >>
> >>
> >> On Tue, Jun 23, 2026 at 9:30 AM Dmitri Bourlatchkov <di...@apache.org>
> >> wrote:
> >>
> >> > Hi All,
> >> >
> >> > So, what are the blockers for merging [4115], if any?
> >> >
> >> > [4115] https://github.com/apache/polaris/pull/4115
> >> >
> >> > Thanks,
> >> > Dmitri.
> >> >
> >> > On Wed, Jun 17, 2026 at 5:28 PM EJ Wang <
> ej.wang.dev.interest@gmail.com
> >> >
> >> > wrote:
> >> >
> >> > > Hi folks,
> >> > >
> >> > > Thanks again for the discussion today. I updated the sync doc with
> >> notes
> >> > > from today's metrics sync:
> >> > >
> >> > >
> >> > >
> >> >
> >>
> https://docs.google.com/document/d/100h7c4damrUzVuquYbBHM0EvA4LSWuW2IT2dN_7nYVA/edit?tab=t.ezk0rgdx0c6m
> >> > >
> >> > > A few highlights:
> >> > >
> >> > >    - No objection to keeping the default metrics behavior small:
> >> > >    no-op/log-only is enough as the built-in default.
> >> > >    - Durable storage, event forwarding, external queues, dashboards,
> >> and
> >> > >    custom filtering should be implementation choices behind the
> >> metrics
> >> > >    reporting path.
> >> > >    - For PR #4115, the REST layer should resolve/authz/accept the
> >> report,
> >> > >    then delegate to the selected metrics implementation. It should
> not
> >> > own
> >> > >    durable storage semantics.
> >> > >    - The rough boundary we discussed is: spec owns the wire
> contract,
> >> > >    runtime owns framework wiring, API/contract modules own
> >> > provider-facing
> >> > >    contracts, and extensions own replaceable implementations.
> >> > >    - OpenLineage has a similar path-naming question. We should avoid
> >> > >    occupying a generic lineage path too early if we want room for
> >> other
> >> > >    lineage systems later.
> >> > >
> >> > > I believe the boundary rules previewed in sync meeting are useful
> for
> >> > > reviewing PR #4115, but it is not
> >> > > a final module-layout proposal.
> >> > >
> >> > > For metrics specifically, I think the target remains:
> >> > >    runtime
> >> > >      -> metrics reporting/emitting contract
> >> > >      -> default implementation
> >> > >           -> lower implementation details, if needed
> >> > >
> >> > > That keeps the default simple, while allowing durable JDBC,
> >> event-backed,
> >> > > or external-queue implementations to be added separately.
> >> > >
> >> > > We also touched on the split between Iceberg metrics emitting and
> >> > > Polaris-owned metrics querying. The querying side can keep evolving,
> >> > > especially for dashboard or generic-table use cases, but I do not
> >> think
> >> > > that needs to block the intake path in this PR.
> >> > >
> >> > > Thanks,
> >> > > -ej
> >> > >
> >> > > On Tue, Jun 16, 2026 at 2:17 PM Yufei Gu <fl...@gmail.com>
> >> wrote:
> >> > >
> >> > > > Thanks for chiming in, EJ. Agreed that the default battery should
> >> stay
> >> > > > small. I'm leaning toward not including metrics persistence in the
> >> > > default
> >> > > > battery.
> >> > > >
> >> > > > Yufei
> >> > > >
> >> > > >
> >> > > > On Thu, Jun 11, 2026 at 8:30 PM EJ Wang <
> >> > ej.wang.dev.interest@gmail.com>
> >> > > > wrote:
> >> > > >
> >> > > > > Hi Yufei,
> >> > > > >
> >> > > > > Thanks for connecting this back to the REST endpoint proposal. I
> >> > > replied
> >> > > > > with the fuller version on the event-forwarding thread, but
> >> wanted to
> >> > > add
> >> > > > > the shorter version here too.
> >> > > > >
> >> > > > > I agree that metrics reporting/emitting is the right conceptual
> >> > > boundary,
> >> > > > > and I also think the event/listener path is a good
> implementation
> >> > > > direction
> >> > > > > to explore. The distinction I want to keep clear is default
> >> battery
> >> > vs
> >> > > > > extension implementation.
> >> > > > >
> >> > > > > For the REST/API side, I would keep the semantics narrow:
> Polaris
> >> > > > resolves
> >> > > > > and authorizes the table, accepts the Iceberg scan/commit
> metrics
> >> > > report
> >> > > > > into the configured reporting/emitting path, and returns 204.
> That
> >> > 204
> >> > > > > should mean Polaris accepted the report into the ingestion path;
> >> it
> >> > > > should
> >> > > > > not imply durable storage.
> >> > > > >
> >> > > > > Durable storage, event forwarding, external telemetry routing,
> >> > > filtering,
> >> > > > > and retention should sit behind that reporting/emitting boundary
> >> as
> >> > > > > implementation-layer behavior. A durable JDBC path can be one
> >> named
> >> > > > > extension implementation. An event/listener forwarding path can
> be
> >> > > > another
> >> > > > > named extension implementation.
> >> > > > >
> >> > > > > So I am +1 on the event-forwarding idea as a non-default
> extension
> >> > > > > implementation. I just would not make it the default battery or
> >> the
> >> > > core
> >> > > > > API shape. The default battery can stay small and safe, while
> >> > > deployments
> >> > > > > that need forwarding, dashboards, or durable storage can opt
> into
> >> the
> >> > > > > implementation that matches their operational model.
> >> > > > >
> >> > > > > Thanks,
> >> > > > > -ej
> >> > > > >
> >> > > > > On Wed, Jun 10, 2026 at 11:43 AM Yufei Gu <flyrain000@gmail.com
> >
> >> > > wrote:
> >> > > > >
> >> > > > > > Thanks EJ. I agree that the reporting/emitting boundary feels
> >> like
> >> > > the
> >> > > > > > right SPI boundary, but I wonder if we can simplify this even
> >> > > further.
> >> > > > > >
> >> > > > > > Iceberg scan and commit metrics look very similar to events to
> >> me.
> >> > > They
> >> > > > > are
> >> > > > > > append only, asynchronously consumed, often forwarded to
> >> external
> >> > > > > systems,
> >> > > > > > and have similar retention and cleanup requirements. More
> >> details
> >> > can
> >> > > > be
> >> > > > > > found in this thread [1]. In that case, Polaris may only need
> to
> >> > > accept
> >> > > > > the
> >> > > > > > report and emit an event through the existing event framework.
> >> > > > Filtering,
> >> > > > > > async delivery, custom sinks, and retention mechanisms could
> >> then
> >> > be
> >> > > > > shared
> >> > > > > > instead of introducing a separate metrics specific extension
> >> layer
> >> > > and
> >> > > > > SPI.
> >> > > > > >
> >> > > > > > The main requirement I still see is querying metrics through
> >> > Polaris.
> >> > > > If
> >> > > > > we
> >> > > > > > want that battery included experience, we could work on a
> >> > persistence
> >> > > > > > listener implementation that selectively saves metrics and
> >> expose
> >> > it
> >> > > > via
> >> > > > > > the IRC event endpoint which seems pretty close in Iceberg
> >> > > > > > community. Another option I like more is to allow users to
> >> develop
> >> > > > their
> >> > > > > > own listener, so they can persist scan/commit metrics to any
> >> > systems
> >> > > > they
> >> > > > > > prefer. In general, that feels like an implementation choice
> on
> >> top
> >> > > of
> >> > > > > the
> >> > > > > > event framework rather than something that needs to shape the
> >> core
> >> > > > > > architecture.
> >> > > > > >
> >> > > > > > 1.
> >> > https://lists.apache.org/thread/x9j8nscvy8hq61tyn01mj8yp6n9of0kp
> >> > > > > >
> >> > > > > > Thanks,
> >> > > > > > Yufei
> >> > > > > >
> >> > > > > >
> >> > > > > > On Thu, Jun 4, 2026 at 5:14 PM EJ Wang <
> >> > > ej.wang.dev.interest@gmail.com
> >> > > > >
> >> > > > > > wrote:
> >> > > > > >
> >> > > > > > > Good point. I agree the existing PolarisMetricsReporter is
> >> > already
> >> > > > very
> >> > > > > > > close to the right conceptual boundary. I am not proposing a
> >> > second
> >> > > > > > > parallel reporter concept. The distinction I am trying to
> >> make is
> >> > > > > > narrower:
> >> > > > > > >
> >> > > > > > > Current state:
> >> > > > > > >
> >> > > > > > >    - PolarisMetricsReporter is the existing metrics
> reporting
> >> > hook.
> >> > > > > > >    - But it currently lives under runtime/service.
> >> > > > > > >    - Its contract is documented around Quarkus/CDI discovery
> >> and
> >> > > > > > > @Identifier
> >> > > > > > >    selection.
> >> > > > > > >    - The default implementation is log-only.
> >> > > > > > >    - The durable implementation writes through
> >> > > PolarisMetricsManager
> >> > > > > and
> >> > > > > > >    MetricsPersistence.
> >> > > > > > >    - MetricsPersistence is currently inherited by
> >> > BasePersistence.
> >> > > > > > >
> >> > > > > > > My proposal:
> >> > > > > > >
> >> > > > > > >    - Keep the reporting/emitting boundary as the SPI.
> >> > > > > > >    - Revise that contract in a framework-agnostic optional
> >> > Iceberg
> >> > > > > > *metrics
> >> > > > > > >    extension API layer*.
> >> > > > > > >    - Keep runtime/service responsible for REST ingestion,
> >> > > table/authz
> >> > > > > > >    validation, and runtime wiring.
> >> > > > > > >    - Keep the battery implementation log-only in the metrics
> >> > > > extension
> >> > > > > > API
> >> > > > > > >    layer, *not under runtime/service*.
> >> > > > > > >    - Treat durable JDBC metrics as one implementation of the
> >> > > > reporting
> >> > > > > > SPI.
> >> > > > > > >    - Decompose the MetricsPersistence and BasePersistence
> >> > coupling
> >> > > as
> >> > > > > > >    durable implementation detail.
> >> > > > > > >
> >> > > > > > > So the difference is not “new reporter concept vs old
> reporter
> >> > > > > concept.”
> >> > > > > > > but:
> >> > > > > > >
> >> > > > > > >
> >> > > > > > >    - PolarisMetricsReporter today = runtime/service
> CDI-shaped
> >> > > hook.
> >> > > > > > >    - Target reporting SPI = same conceptual boundary, but
> >> placed
> >> > in
> >> > > > the
> >> > > > > > >    right extension API layer and kept framework-agnostic.
> >> > > > > > >
> >> > > > > > > For the current metrics PR, I think the minimal
> architectural
> >> > > cleanup
> >> > > > > > > before merge is to *put the reporting SPI contract and the
> >> > battery
> >> > > > > > default
> >> > > > > > > in the right place*. The durable JDBC implementation,
> >> > event-backed
> >> > > > > async
> >> > > > > > > implementation, external queue support, and deeper
> persistence
> >> > > > cleanup
> >> > > > > > can
> >> > > > > > > continue as follow-ups, as long as we do not lock the SPI
> >> > boundary
> >> > > to
> >> > > > > > > BasePersistence or to runtime/service wiring.
> >> > > > > > >
> >> > > > > > > -ej
> >> > > > > > >
> >> > > > > > > On Wed, Jun 3, 2026 at 3:00 PM Dmitri Bourlatchkov <
> >> > > dimas@apache.org
> >> > > > >
> >> > > > > > > wrote:
> >> > > > > > >
> >> > > > > > > > Hi EJ,
> >> > > > > > > >
> >> > > > > > > > Thanks for the recap / summary!
> >> > > > > > > >
> >> > > > > > > > > Do folks agree that the stable SPI boundary should be
> >> > > > > > > > metrics reporting/emitting, not metrics persistence?
> >> > > > > > > >
> >> > > > > > > > SGTM.
> >> > > > > > > >
> >> > > > > > > > > Does an optional Iceberg metrics extension API layer
> sound
> >> > like
> >> > > > the
> >> > > > > > > right
> >> > > > > > > > home for this SPI?
> >> > > > > > > >
> >> > > > > > > > How is that different from current PolarisMetricsReporter?
> >> > > > > > > >
> >> > > > > > > > > Should the current durable metrics work be reframed as a
> >> > > durable
> >> > > > > > > > JDBC reference implementation of that SPI?
> >> > > > > > > >
> >> > > > > > > > SGTM.
> >> > > > > > > >
> >> > > > > > > > > What is the smallest PR sequence to get there without
> >> > blocking
> >> > > > > > > > the current metrics work unnecessarily?
> >> > > > > > > >
> >> > > > > > > > Let's get https://github.com/apache/polaris/pull/4397
> >> merged
> >> > > > first.
> >> > > > > > > >
> >> > > > > > > > Then, I'd propose isolating the Metrics schema from the
> >> > MetaStore
> >> > > > > > schema.
> >> > > > > > > > This will probably have some ripple effect into the
> >> bootstrap
> >> > > > > > workflows,
> >> > > > > > > so
> >> > > > > > > > it's not a trivial change.
> >> > > > > > > >
> >> > > > > > > > Then, let's reassess.
> >> > > > > > > >
> >> > > > > > > > Cheers,
> >> > > > > > > > Dmitri.
> >> > > > > > > >
> >> > > > > > > > On Wed, Jun 3, 2026 at 5:40 PM EJ Wang <
> >> > > > > ej.wang.dev.interest@gmail.com
> >> > > > > > >
> >> > > > > > > > wrote:
> >> > > > > > > >
> >> > > > > > > > > Hi folks,
> >> > > > > > > > >
> >> > > > > > > > > Thanks again for the discussion today. I updated the
> sync
> >> doc
> >> > > > with
> >> > > > > > > notes
> >> > > > > > > > > from the third metrics architecture sync:
> >> > > > > > > > >
> >> > > > > > > > >
> >> > > > > > > > >
> >> > > > > > > >
> >> > > > > > >
> >> > > > > >
> >> > > > >
> >> > > >
> >> > >
> >> >
> >>
> https://docs.google.com/document/d/100h7c4damrUzVuquYbBHM0EvA4LSWuW2IT2dN_7nYVA/edit?tab=t.k96s2xyqr5u1
> >> > > > > > > > >
> >> > > > > > > > > A few highlights from today’s discussion:
> >> > > > > > > > >
> >> > > > > > > > >    - We clarified that Iceberg metrics reporting can be
> >> > > > interpreted
> >> > > > > > > > either
> >> > > > > > > > >    as sync or async handling from Polaris’s perspective,
> >> so
> >> > > > Polaris
> >> > > > > > > > should
> >> > > > > > > > >    stay flexible as a platform instead of baking one
> >> handling
> >> > > > model
> >> > > > > > > into
> >> > > > > > > > > the
> >> > > > > > > > >    REST API behavior.
> >> > > > > > > > >    - We aligned that the built-in (battery) behavior for
> >> > > metrics
> >> > > > > > > emitting
> >> > > > > > > > >    can stay simple: no-op/log-only is enough as the
> >> default.
> >> > > > > > > > >    - Durable metrics persistence should be treated as an
> >> > > > > > implementation
> >> > > > > > > > of
> >> > > > > > > > >    the metrics reporting path, not as the core SPI
> >> boundary.
> >> > > > > > > > >    - The existing durable metrics work can be reviewed
> as
> >> a
> >> > > > > reference
> >> > > > > > > > >    implementation of the reporting SPI, with
> >> > > persistence-related
> >> > > > > > logic
> >> > > > > > > > kept
> >> > > > > > > > >    self-contained under a metrics durable implementation
> >> > module
> >> > > > > > rather
> >> > > > > > > > than
> >> > > > > > > > >    scattered through core entity persistence.
> >> > > > > > > > >    - Dashboard/insights remains a real use case, but we
> >> > agreed
> >> > > to
> >> > > > > > keep
> >> > > > > > > it
> >> > > > > > > > >    separate from the core metrics intake discussion for
> >> now.
> >> > > > > > > > >
> >> > > > > > > > > I also did a quick source check after the meeting to
> make
> >> > sure
> >> > > we
> >> > > > > are
> >> > > > > > > > > describing the current state accurately.
> >> > > > > > > > >
> >> > > > > > > > > Current state:
> >> > > > > > > > >
> >> > > > > > > > >    - Polaris already has a metrics reporting hook:
> >> > > > > > > > PolarisMetricsReporter.
> >> > > > > > > > >    - The default implementation is
> DefaultMetricsReporter,
> >> > > > selected
> >> > > > > > by
> >> > > > > > > > >    polaris.iceberg-metrics.reporting.type=default. It is
> >> > > > log-only,
> >> > > > > > > > >    effectively quiet unless metrics logging is enabled.
> >> > > > > > > > >    - There is also a PersistingMetricsReporter, selected
> >> by
> >> > > > > > > > >    polaris.iceberg-metrics.reporting.type=persisting,
> >> which
> >> > > > > converts
> >> > > > > > > > >    Iceberg scan/commit reports into Polaris metrics
> >> records
> >> > and
> >> > > > > > writes
> >> > > > > > > > > through
> >> > > > > > > > >    PolarisMetricsManager -> MetricsPersistence.
> >> > > > > > > > >    - MetricsPersistence currently exists in the
> >> persistence
> >> > > layer
> >> > > > > and
> >> > > > > > > is
> >> > > > > > > > >    inherited by BasePersistence.
> >> > > > > > > > >
> >> > > > > > > > > My read from the discussion is that the target boundary
> >> > should
> >> > > > not
> >> > > > > be
> >> > > > > > > > > MetricsPersistence as inherited by BasePersistence. That
> >> path
> >> > > > > should
> >> > > > > > be
> >> > > > > > > > > decomposed as durable implementation detail (taken care
> >> of by
> >> > > > > > > > > https://github.com/apache/polaris/pull/4397). The
> stable
> >> > > > extension
> >> > > > > > > point
> >> > > > > > > > > should instead be the metrics reporting/emitting
> boundary.
> >> > > > > > > > >
> >> > > > > > > > > Concretely, I think the next proposal should be shaped
> >> like
> >> > > this:
> >> > > > > > > > >
> >> > > > > > > > >    1.
> >> > > > > > > > >
> >> > > > > > > > >    Define the proper metrics reporting SPI at an
> optional
> >> > > Iceberg
> >> > > > > > > metrics
> >> > > > > > > > >    extension API layer.
> >> > > > > > > > >
> >> > > > > > > > >    Example direction:
> >> > > > > > > > >
> >> > > > > > > > >    public interface IcebergMetricsReporter {
> >> > > > > > > > >      void reportMetric(IcebergMetricsReportContext
> >> context,
> >> > > > > > > > > MetricsReport report);
> >> > > > > > > > >    }
> >> > > > > > > > >
> >> > > > > > > > >    The context would carry the small Polaris-resolved
> >> > envelope:
> >> > > > > > > > >    catalog/table identity, report type, received
> >> timestamp,
> >> > and
> >> > > > > > > > > request/trace
> >> > > > > > > > >    context if available. The raw Iceberg MetricsReport
> >> > remains
> >> > > > the
> >> > > > > > > > payload.
> >> > > > > > > > >    2.
> >> > > > > > > > >
> >> > > > > > > > >    Keep runtime/service as the REST ingestion and wiring
> >> > layer.
> >> > > > > > > > >
> >> > > > > > > > >    The REST handler still resolves the table and
> performs
> >> > authz
> >> > > > > > before
> >> > > > > > > > >    accepting the report. After that, it calls the
> selected
> >> > > > > reporting
> >> > > > > > > > >    implementation.
> >> > > > > > > > >    3.
> >> > > > > > > > >
> >> > > > > > > > >    Keep the battery default no-op/log-only.
> >> > > > > > > > >
> >> > > > > > > > >    This preserves an out-of-box safe default and avoids
> >> > > > requiring a
> >> > > > > > > > durable
> >> > > > > > > > >    metrics store for every Polaris deployment.
> >> > > > > > > > >    4.
> >> > > > > > > > >
> >> > > > > > > > >    Move durable JDBC metrics into a self-contained
> >> > > implementation
> >> > > > > of
> >> > > > > > > the
> >> > > > > > > > >    reporting SPI.
> >> > > > > > > > >
> >> > > > > > > > >    That implementation can own its schema, bootstrap,
> >> > > retention,
> >> > > > > and
> >> > > > > > > read
> >> > > > > > > > >    API support. It should not define the core reporting
> >> SPI
> >> > > > > boundary,
> >> > > > > > > and
> >> > > > > > > > > it
> >> > > > > > > > >    should not require metrics persistence to remain
> >> inherited
> >> > > > from
> >> > > > > > > > >    BasePersistence.
> >> > > > > > > > >    5.
> >> > > > > > > > >
> >> > > > > > > > >    Treat async/event-backed handling as another
> >> > implementation
> >> > > of
> >> > > > > the
> >> > > > > > > > same
> >> > > > > > > > >    reporting SPI.
> >> > > > > > > > >
> >> > > > > > > > >    For example, an event-backed reporter could enqueue a
> >> > > metrics
> >> > > > > > event
> >> > > > > > > > and
> >> > > > > > > > >    let listeners handle durable storage or other sinks.
> >> If we
> >> > > > later
> >> > > > > > > need
> >> > > > > > > > a
> >> > > > > > > > >    replaceable queue engine, that seems like a shared
> >> > > > event/metrics
> >> > > > > > > > > substrate
> >> > > > > > > > >    topic rather than a metrics-only requirement.
> >> > > > > > > > >
> >> > > > > > > > > This framing lets us keep the REST metrics endpoint
> >> simple,
> >> > > > > preserve
> >> > > > > > > the
> >> > > > > > > > > current default behavior, support durable metrics users,
> >> and
> >> > > > still
> >> > > > > > > leave
> >> > > > > > > > > room for async/event-backed or external-queue-based
> >> > > > > implementations.
> >> > > > > > > > >
> >> > > > > > > > > I think the main follow-up questions are:
> >> > > > > > > > >
> >> > > > > > > > >    - Do folks agree that the stable SPI boundary should
> be
> >> > > > metrics
> >> > > > > > > > >    reporting/emitting, not metrics persistence?
> >> > > > > > > > >    - Does an optional Iceberg metrics extension API
> layer
> >> > sound
> >> > > > > like
> >> > > > > > > the
> >> > > > > > > > >    right home for this SPI?
> >> > > > > > > > >    - Should the current durable metrics work be reframed
> >> as a
> >> > > > > durable
> >> > > > > > > > JDBC
> >> > > > > > > > >    reference implementation of that SPI?
> >> > > > > > > > >    - What is the smallest PR sequence to get there
> without
> >> > > > blocking
> >> > > > > > the
> >> > > > > > > > >    current metrics work unnecessarily?
> >> > > > > > > > >
> >> > > > > > > > > Thanks,
> >> > > > > > > > > -ej
> >> > > > > > > > >
> >> > > > > > > > > On Fri, May 15, 2026 at 5:16 PM Dmitri Bourlatchkov <
> >> > > > > > dimas@apache.org>
> >> > > > > > > > > wrote:
> >> > > > > > > > >
> >> > > > > > > > > > Hi JB,
> >> > > > > > > > > >
> >> > > > > > > > > > Could you set up another meeting, please? Same time on
> >> > > > Wednesday
> >> > > > > as
> >> > > > > > > > last
> >> > > > > > > > > > time... I hope it works for everyone.
> >> > > > > > > > > >
> >> > > > > > > > > > Cheers,
> >> > > > > > > > > > Dmitri.
> >> > > > > > > > > >
> >> > > > > > > > > > On Fri, May 15, 2026 at 8:06 PM Yufei Gu <
> >> > > flyrain000@gmail.com
> >> > > > >
> >> > > > > > > wrote:
> >> > > > > > > > > >
> >> > > > > > > > > > > +1 on another sync call next week.
> >> > > > > > > > > > >
> >> > > > > > > > > > > Yufei
> >> > > > > > > > > > >
> >> > > > > > > > > > >
> >> > > > > > > > > > > On Fri, May 15, 2026 at 4:52 PM Dmitri Bourlatchkov
> <
> >> > > > > > > > dimas@apache.org>
> >> > > > > > > > > > > wrote:
> >> > > > > > > > > > >
> >> > > > > > > > > > > > Hi All,
> >> > > > > > > > > > > >
> >> > > > > > > > > > > > WDYT about another sync call next week?
> >> > > > > > > > > > > >
> >> > > > > > > > > > > > Thanks,
> >> > > > > > > > > > > > Dmitri.
> >> > > > > > > > > > > >
> >> > > > > > > > > > > > On Wed, May 6, 2026 at 5:29 PM Dmitri
> Bourlatchkov <
> >> > > > > > > > dimas@apache.org
> >> > > > > > > > > >
> >> > > > > > > > > > > > wrote:
> >> > > > > > > > > > > >
> >> > > > > > > > > > > > > Hi EJ,
> >> > > > > > > > > > > > >
> >> > > > > > > > > > > > > Thanks for the summary! It covers what we
> >> discussed
> >> > in
> >> > > > the
> >> > > > > > > > meeting
> >> > > > > > > > > > very
> >> > > > > > > > > > > > > well, IMHO.
> >> > > > > > > > > > > > >
> >> > > > > > > > > > > > > Looking forward to concrete PRs :)
> >> > > > > > > > > > > > >
> >> > > > > > > > > > > > > Cheers,
> >> > > > > > > > > > > > > Dmitri.
> >> > > > > > > > > > > > >
> >> > > > > > > > > > > > > On Wed, May 6, 2026 at 5:08 PM EJ Wang <
> >> > > > > > > > > > ej.wang.dev.interest@gmail.com
> >> > > > > > > > > > > >
> >> > > > > > > > > > > > > wrote:
> >> > > > > > > > > > > > >
> >> > > > > > > > > > > > >> Hi folks,
> >> > > > > > > > > > > > >>
> >> > > > > > > > > > > > >> We had a community sync earlier, thanks JB for
> >> > > > scheduling
> >> > > > > > it.
> >> > > > > > > > > Notes
> >> > > > > > > > > > > from
> >> > > > > > > > > > > > >> the first metrics architecture sync (May 6,
> >> 10-11am
> >> > > PT).
> >> > > > > > > > > Discussion
> >> > > > > > > > > > > doc
> >> > > > > > > > > > > > >> with per-section status:
> >> > > > > > > > > > > > >>
> >> > > > > > > > > > > > >>
> >> > > > > > > > > > > >
> >> > > > > > > > > > >
> >> > > > > > > > > >
> >> > > > > > > > >
> >> > > > > > > >
> >> > > > > > >
> >> > > > > >
> >> > > > >
> >> > > >
> >> > >
> >> >
> >>
> https://docs.google.com/document/d/100h7c4damrUzVuquYbBHM0EvA4LSWuW2IT2dN_7nYVA/edit?tab=t.0
> >> > > > > > > > > > > > >>
> >> > > > > > > > > > > > >> *The meeting covered both topics from the doc.
> >> > > > > > Direction-level
> >> > > > > > > > > > > alignment
> >> > > > > > > > > > > > >> was reached on the headline pieces; details
> >> remain
> >> > for
> >> > > > PR
> >> > > > > > > review
> >> > > > > > > > > or
> >> > > > > > > > > > > > >> follow-up sessions.*
> >> > > > > > > > > > > > >>
> >> > > > > > > > > > > > >> *Topic 1 — Persistence schema redesign*
> >> > > > > > > > > > > > >> Idea-level alignment on consolidating per-type
> >> > tables
> >> > > > > > > > > > > > >> (scan_metrics_report,
> >> > > > > > > > > > > > >> commit_metrics_report) into a single
> >> metrics_report
> >> > > > table.
> >> > > > > > The
> >> > > > > > > > > > > > motivating
> >> > > > > > > > > > > > >> cost is the surface area added by every new
> >> metric
> >> > > type
> >> > > > > > today:
> >> > > > > > > > new
> >> > > > > > > > > > > > table,
> >> > > > > > > > > > > > >> SPI method, record class, model, converter,
> >> schema
> >> > > > > > migration.
> >> > > > > > > > > > > > >>
> >> > > > > > > > > > > > >> Most schema details are deferred to the schema
> >> PR. A
> >> > > few
> >> > > > > > > > specific
> >> > > > > > > > > > > points
> >> > > > > > > > > > > > >> came up:
> >> > > > > > > > > > > > >> •   metric_schema_version: Yufei prefers
> dropping
> >> > it,
> >> > > > > since
> >> > > > > > > > there
> >> > > > > > > > > is
> >> > > > > > > > > > > no
> >> > > > > > > > > > > > >> spec-level concept of metrics versioning today
> >> and
> >> > it
> >> > > is
> >> > > > > > hard
> >> > > > > > > to
> >> > > > > > > > > > > define
> >> > > > > > > > > > > > >> unilaterally. Robert prefers keeping it, given
> >> IRC
> >> > v2
> >> > > is
> >> > > > > > > coming
> >> > > > > > > > > and
> >> > > > > > > > > > > the
> >> > > > > > > > > > > > >> schema should be considered against its likely
> >> > shape;
> >> > > > > Robert
> >> > > > > > > > also
> >> > > > > > > > > > > raised
> >> > > > > > > > > > > > >> how to differentiate various payload formats if
> >> any.
> >> > > > EJ's
> >> > > > > > read
> >> > > > > > > > is
> >> > > > > > > > > > that
> >> > > > > > > > > > > > >> this
> >> > > > > > > > > > > > >> is a two-way-door decision. We can start
> without
> >> the
> >> > > > > field,
> >> > > > > > > and
> >> > > > > > > > if
> >> > > > > > > > > > IRC
> >> > > > > > > > > > > > v2
> >> > > > > > > > > > > > >> changes the shape we would likely roll a
> >> > corresponding
> >> > > > new
> >> > > > > > > > schema
> >> > > > > > > > > > > > anyway,
> >> > > > > > > > > > > > >> which is not particularly costly.
> >> > > > > > > > > > > > >> •   Payload format: Robert pointed out that
> >> future
> >> > > > formats
> >> > > > > > > > beyond
> >> > > > > > > > > > JSON
> >> > > > > > > > > > > > may
> >> > > > > > > > > > > > >> be worth supporting. The exact shape is
> deferred
> >> to
> >> > > the
> >> > > > > > schema
> >> > > > > > > > > > > > discussion.
> >> > > > > > > > > > > > >> •   Partition strategy: Anand suggested monthly
> >> > > > > partitioning
> >> > > > > > > > based
> >> > > > > > > > > > on
> >> > > > > > > > > > > > his
> >> > > > > > > > > > > > >> experience as potentially helpful at scale.
> >> > > > > > > > > > > > >>
> >> > > > > > > > > > > > >> *Topic 2 — Where metrics ingestion and storage
> >> > belong*
> >> > > > > > > > > > > > >> Idea-level alignment that metrics should be a
> >> > > separated
> >> > > > > SPI
> >> > > > > > > from
> >> > > > > > > > > the
> >> > > > > > > > > > > > >> entity
> >> > > > > > > > > > > > >> persistence stack. Two reasons surfaced: (a)
> >> > workloads
> >> > > > and
> >> > > > > > > > > > capability
> >> > > > > > > > > > > > >> requirements diverge enough that coupling them
> >> > creates
> >> > > > > > > > artificial
> >> > > > > > > > > > > > >> constraints, and (b) admin experience improves
> >> when
> >> > > > > metrics
> >> > > > > > > has
> >> > > > > > > > > its
> >> > > > > > > > > > > own
> >> > > > > > > > > > > > >> bootstrap, retention, and lifecycle. Dmitri
> noted
> >> > > > Polaris
> >> > > > > > > being
> >> > > > > > > > a
> >> > > > > > > > > > > > platform
> >> > > > > > > > > > > > >> should have the flexibility to support
> different
> >> > > > > persistence
> >> > > > > > > > > > backends
> >> > > > > > > > > > > > per
> >> > > > > > > > > > > > >> concern, and pointed to a concrete next step of
> >> > > > separating
> >> > > > > > the
> >> > > > > > > > > JDBC
> >> > > > > > > > > > > > >> bootstrap for metrics from the metastore
> >> bootstrap.
> >> > > > Robert
> >> > > > > > > > > proposed
> >> > > > > > > > > > an
> >> > > > > > > > > > > > >> additional UX extension: detect an
> unbootstrapped
> >> > > > metrics
> >> > > > > > > store
> >> > > > > > > > on
> >> > > > > > > > > > > first
> >> > > > > > > > > > > > >> use and auto-bootstrap rather than requiring an
> >> > > explicit
> >> > > > > > > manual
> >> > > > > > > > > > > > bootstrap
> >> > > > > > > > > > > > >> step.
> >> > > > > > > > > > > > >> The meeting also confirmed that Polaris metrics
> >> can
> >> > > > start
> >> > > > > > > small
> >> > > > > > > > > and
> >> > > > > > > > > > > stay
> >> > > > > > > > > > > > >> Iceberg-focused. Naming and persistence schema
> >> can
> >> > > lean
> >> > > > > > > > > > > > Iceberg-specific.
> >> > > > > > > > > > > > >> If a future expansion to generic-table metrics
> or
> >> > > > > > operational
> >> > > > > > > > > > metrics
> >> > > > > > > > > > > > >> arrives, an abstraction layer can be built on
> >> top of
> >> > > the
> >> > > > > > > Iceberg
> >> > > > > > > > > > > metrics
> >> > > > > > > > > > > > >> reporter at that point. Robert remains on the
> >> fence
> >> > > and
> >> > > > > > would
> >> > > > > > > > > prefer
> >> > > > > > > > > > > > >> something more generic but did not block the
> >> > > direction;
> >> > > > > > > Dmitri's
> >> > > > > > > > > > read
> >> > > > > > > > > > > > was
> >> > > > > > > > > > > > >> that the proposed framework already has enough
> >> > > > flexibility
> >> > > > > > to
> >> > > > > > > > > absorb
> >> > > > > > > > > > > > >> future
> >> > > > > > > > > > > > >> expansion.
> >> > > > > > > > > > > > >>
> >> > > > > > > > > > > > >> The Trade-offs and Proposed structure sections
> in
> >> > the
> >> > > > doc
> >> > > > > > were
> >> > > > > > > > not
> >> > > > > > > > > > > > >> reviewed
> >> > > > > > > > > > > > >> in detail. They remain open for either the next
> >> sync
> >> > > or
> >> > > > PR
> >> > > > > > > > review.
> >> > > > > > > > > > > > >>
> >> > > > > > > > > > > > >> *Cross-cutting alignment — battery-included
> plus
> >> > > > > pluggable*
> >> > > > > > > > > > > > >> A common philosophy emerged from the
> discussion.
> >> EJ
> >> > > > > > summarized
> >> > > > > > > > it
> >> > > > > > > > > > as:
> >> > > > > > > > > > > > >> Polaris should provide a battery-included UX
> for
> >> > > > beginners
> >> > > > > > and
> >> > > > > > > > the
> >> > > > > > > > > > > > >> flexibility for advanced users to swap the
> >> included
> >> > > > > battery
> >> > > > > > > for
> >> > > > > > > > > > > > something
> >> > > > > > > > > > > > >> more powerful or tailored to their use case.
> The
> >> SPI
> >> > > > > design
> >> > > > > > > > needs
> >> > > > > > > > > to
> >> > > > > > > > > > > > >> enable
> >> > > > > > > > > > > > >> both.
> >> > > > > > > > > > > > >>
> >> > > > > > > > > > > > >> The inputs that shaped this framing:
> >> > > > > > > > > > > > >> •   Anand described how his team uses the
> current
> >> > > > metrics
> >> > > > > > > > > > persistence
> >> > > > > > > > > > > > >> (three metrics consumers in v1.4).
> >> > > > > > > > > > > > >> •   Yufei raised Grafana and dashboard
> >> integrations
> >> > > as a
> >> > > > > > > > > destination
> >> > > > > > > > > > > use
> >> > > > > > > > > > > > >> case beyond the default.
> >> > > > > > > > > > > > >> •   Robert called out that the current design
> is
> >> > more
> >> > > > > > > > > JDBC-focused.
> >> > > > > > > > > > > > >>
> >> > > > > > > > > > > > >> Two concrete instances:
> >> > > > > > > > > > > > >> •   Async metrics intake: Yufei's initial
> >> position
> >> > was
> >> > > > > that
> >> > > > > > > > async
> >> > > > > > > > > > > should
> >> > > > > > > > > > > > >> largely live on the producer side and there is
> >> not
> >> > > much
> >> > > > > > > Polaris
> >> > > > > > > > > can
> >> > > > > > > > > > > do.
> >> > > > > > > > > > > > >> Robert suggested a Polaris-side default is
> doable
> >> > via
> >> > > > > > Vert.x.
> >> > > > > > > > > Dmitri
> >> > > > > > > > > > > > >> agreed
> >> > > > > > > > > > > > >> the direction is worth exploring. The meeting
> >> > > converged
> >> > > > > on a
> >> > > > > > > > > > > > >> battery-included default (likely Vert.x-backed)
> >> with
> >> > > an
> >> > > > > SPI
> >> > > > > > > > shape
> >> > > > > > > > > > that
> >> > > > > > > > > > > > >> lets
> >> > > > > > > > > > > > >> power users route to a more scalable backend
> >> > > (k8s-hosted
> >> > > > > > > queue,
> >> > > > > > > > > AWS
> >> > > > > > > > > > > SQS,
> >> > > > > > > > > > > > >> etc.).
> >> > > > > > > > > > > > >> •   Pluggable destinations: combining Yufei's
> >> > > dashboard
> >> > > > > use
> >> > > > > > > case
> >> > > > > > > > > > with
> >> > > > > > > > > > > > >> Robert's JDBC-focused call-out, the meeting
> >> agreed
> >> > the
> >> > > > SPI
> >> > > > > > > > should
> >> > > > > > > > > be
> >> > > > > > > > > > > > >> structured for multiple sinks so integrations
> >> become
> >> > > > impl
> >> > > > > > > > choices
> >> > > > > > > > > > > rather
> >> > > > > > > > > > > > >> than architectural changes.
> >> > > > > > > > > > > > >>
> >> > > > > > > > > > > > >> The battery-included default is most likely to
> >> use
> >> > the
> >> > > > > > > existing
> >> > > > > > > > > > > > >> JDBC-backed
> >> > > > > > > > > > > > >> approach.
> >> > > > > > > > > > > > >>
> >> > > > > > > > > > > > >> *Direction (idea-level alignment)*
> >> > > > > > > > > > > > >> •   Single metrics_report table consolidating
> >> > per-type
> >> > > > > > > metrics,
> >> > > > > > > > > > > > replacing
> >> > > > > > > > > > > > >> scan_metrics_report and commit_metrics_report
> >> > > > > > > > > > > > >> •   Iceberg-focused naming and schema for now,
> >> > revisit
> >> > > > if
> >> > > > > > > > > > > generic-table
> >> > > > > > > > > > > > or
> >> > > > > > > > > > > > >> operational metrics arrive
> >> > > > > > > > > > > > >> •   Metrics persistence as a separated SPI, not
> >> on
> >> > > > > > > > BasePersistence
> >> > > > > > > > > > > > >> •   Bootstrap path separated for metrics,
> >> > independent
> >> > > of
> >> > > > > > > > metastore
> >> > > > > > > > > > > > >> bootstrap
> >> > > > > > > > > > > > >> •   "Battery-included plus pluggable" as the
> SPI
> >> > > design
> >> > > > > > > > philosophy
> >> > > > > > > > > > > > >>
> >> > > > > > > > > > > > >> *Open items*
> >> > > > > > > > > > > > >> •   Schema details: metric_schema_version,
> >> payload
> >> > > > format,
> >> > > > > > IRC
> >> > > > > > > > v2
> >> > > > > > > > > > > > >> forward-compat shape
> >> > > > > > > > > > > > >> •   SPI design details — full review either in
> >> the
> >> > > next
> >> > > > > sync
> >> > > > > > > or
> >> > > > > > > > in
> >> > > > > > > > > > the
> >> > > > > > > > > > > > >> corresponding PR
> >> > > > > > > > > > > > >> •   Schema refactor PR ownership
> >> > > > > > > > > > > > >>
> >> > > > > > > > > > > > >> *Action items*
> >> > > > > > > > > > > > >> •   EJ to take a first stab at the SPI design
> and
> >> > > > > > potentially
> >> > > > > > > > > > partner
> >> > > > > > > > > > > > with
> >> > > > > > > > > > > > >> Anand to incorporate the lessons learned from
> the
> >> > > > existing
> >> > > > > > > > > reporter
> >> > > > > > > > > > > and
> >> > > > > > > > > > > > >> persistence work.
> >> > > > > > > > > > > > >> •   Schema refactor PR ownership is not yet
> >> decided.
> >> > > If
> >> > > > > > anyone
> >> > > > > > > > is
> >> > > > > > > > > > > > >> interested in driving it, reply on this thread.
> >> > > > > > > > > > > > >> •   JB to schedule the next sync, tentatively
> in
> >> two
> >> > > > > weeks.
> >> > > > > > > > > > > > >>
> >> > > > > > > > > > > > >> -ej
> >> > > > > > > > > > > > >>
> >> > > > > > > > > > > > >> On Mon, Apr 27, 2026 at 3:07 PM EJ Wang <
> >> > > > > > > > > > > ej.wang.dev.interest@gmail.com
> >> > > > > > > > > > > > >
> >> > > > > > > > > > > > >> wrote:
> >> > > > > > > > > > > > >>
> >> > > > > > > > > > > > >> > Thanks Yufei for the +1.
> >> > > > > > > > > > > > >> >
> >> > > > > > > > > > > > >> > JB, could you help add a biweekly metrics
> >> > > architecture
> >> > > > > > sync
> >> > > > > > > to
> >> > > > > > > > > the
> >> > > > > > > > > > > > >> Polaris
> >> > > > > > > > > > > > >> > community calendar? I'm thinking Thursdays at
> >> > 9-10am
> >> > > > PT,
> >> > > > > > on
> >> > > > > > > > the
> >> > > > > > > > > > > > >> off-weeks
> >> > > > > > > > > > > > >> > from the community meeting (starting May 7),
> 60
> >> > > > minutes.
> >> > > > > > > > > > > > >> >
> >> > > > > > > > > > > > >> > Here's a rough agenda to work through over
> the
> >> > first
> >> > > > few
> >> > > > > > > > > sessions,
> >> > > > > > > > > > > > >> grouped
> >> > > > > > > > > > > > >> > by priority:
> >> > > > > > > > > > > > >> >
> >> > > > > > > > > > > > >> > *First: foundational direction*
> >> > > > > > > > > > > > >> >
> >> > > > > > > > > > > > >> > 1.  MetricsPersistence: public SPI or
> internal
> >> > > > > > > implementation
> >> > > > > > > > > > > detail?
> >> > > > > > > > > > > > >> >    •   Marked @Beta, javadoc calls it a
> >> "Service
> >> > > > > Provider
> >> > > > > > > > > > > Interface",
> >> > > > > > > > > > > > >> but
> >> > > > > > > > > > > > >> > only one consumer (JdbcBasePersistenceImpl),
> >> lives
> >> > > on
> >> > > > > > > > > > > BasePersistence.
> >> > > > > > > > > > > > >> If
> >> > > > > > > > > > > > >> > demoted to a private helper inside a
> persisting
> >> > > > reporter
> >> > > > > > > impl,
> >> > > > > > > > > > most
> >> > > > > > > > > > > > >> > downstream design decisions become
> >> implementation
> >> > > > > details
> >> > > > > > > > rather
> >> > > > > > > > > > > than
> >> > > > > > > > > > > > >> > contract questions.
> >> > > > > > > > > > > > >> >
> >> > > > > > > > > > > > >> > 2.  Persistence schema redesign
> >> > > > > > > > > > > > >> >    •   Current two-table layout
> >> > > (scan_metrics_report,
> >> > > > > > > > > > > > >> > commit_metrics_report) with ~25 flattened
> >> columns
> >> > > > each.
> >> > > > > > > Every
> >> > > > > > > > > new
> >> > > > > > > > > > > > metric
> >> > > > > > > > > > > > >> > type requires a new table, SPI method, record
> >> > class,
> >> > > > > > model,
> >> > > > > > > > > > > converter,
> >> > > > > > > > > > > > >> and
> >> > > > > > > > > > > > >> > schema migration. Direction to explore:
> single
> >> > table
> >> > > > > with
> >> > > > > > > > > > > metric_type
> >> > > > > > > > > > > > >> enum,
> >> > > > > > > > > > > > >> > schema_version, and JSON payload column.
> >> > > > > > > > > > > > >> >
> >> > > > > > > > > > > > >> > *Second: design details once direction is
> set*
> >> > > > > > > > > > > > >> >
> >> > > > > > > > > > > > >> > 3.  Partition key strategy
> >> > > > > > > > > > > > >> >    •   Single-table design means scan metrics
> >> at
> >> > > scale
> >> > > > > > will
> >> > > > > > > > have
> >> > > > > > > > > > > high
> >> > > > > > > > > > > > >> > write concurrency per table. Schema needs to
> >> > expose
> >> > > > > enough
> >> > > > > > > > > > structure
> >> > > > > > > > > > > > for
> >> > > > > > > > > > > > >> > backends to shard by entity or time range.
> >> > > > > > > > > > > > >> >
> >> > > > > > > > > > > > >> > 4.  Read/write path consistency
> >> > > > > > > > > > > > >> >    •   Writes go through
> PolarisMetricsManager
> >> on
> >> > > > > > > > > > MetaStoreManager.
> >> > > > > > > > > > > > >> Reads
> >> > > > > > > > > > > > >> > bypass MetaStoreManager and go straight to
> >> > > > > > BasePersistence,
> >> > > > > > > > > > > excluding
> >> > > > > > > > > > > > >> > non-JDBC backends from the read API.
> >> > > > > > > > > > > > >> >
> >> > > > > > > > > > > > >> > *Third: cleanup and alignment*
> >> > > > > > > > > > > > >> >
> >> > > > > > > > > > > > >> > 5.  PolarisMetricsReporter naming
> >> > > > > > > > > > > > >> >    •   Only handles IRC
> >> (ScanReport/CommitReport),
> >> > > > > doesn't
> >> > > > > > > > cover
> >> > > > > > > > > > > > generic
> >> > > > > > > > > > > > >> > tables or operational metrics. Name is
> broader
> >> > than
> >> > > > > scope.
> >> > > > > > > > > > > > >> >
> >> > > > > > > > > > > > >> > 6.  PolarisMetricsManager facade passthrough
> >> > > > > > > > > > > > >> >    •   Entire default method is
> >> > > > > > > > > > > > >> callCtx.getMetaStore().writeScanReport().
> >> > > > > > > > > > > > >> > Zero logic, passes Level 1 straight through
> to
> >> > Level
> >> > > > 3.
> >> > > > > > Same
> >> > > > > > > > > > > > >> anti-pattern
> >> > > > > > > > > > > > >> > as PolarisEventManager.
> >> > > > > > > > > > > > >> >
> >> > > > > > > > > > > > >> > 7.  Iceberg community alignment
> >> > > > > > > > > > > > >> >    •   Payload-type extension needs
> discussion
> >> on
> >> > > > > > > dev@iceberg.
> >> > > > > > > > > > > > >> obelix74's
> >> > > > > > > > > > > > >> > Feb thread got zero replies. Needs a
> committer
> >> > > voice.
> >> > > > > > > > > > > > >> >
> >> > > > > > > > > > > > >> > Lets confirm prioritization in the first
> >> session.
> >> > > > > > > > > > > > >> >
> >> > > > > > > > > > > > >> > -ej
> >> > > > > > > > > > > > >> >
> >> > > > > > > > > > > > >> > On Tue, Apr 21, 2026 at 3:18 PM Yufei Gu <
> >> > > > > > > > flyrain000@gmail.com>
> >> > > > > > > > > > > > wrote:
> >> > > > > > > > > > > > >> >
> >> > > > > > > > > > > > >> >> Thanks everyone for continuing to drive this
> >> > > > forward. I
> >> > > > > > > agree
> >> > > > > > > > > > that
> >> > > > > > > > > > > > the
> >> > > > > > > > > > > > >> >> problem is getting complex enough that a
> more
> >> > > > > structured
> >> > > > > > > > > > discussion
> >> > > > > > > > > > > > >> would
> >> > > > > > > > > > > > >> >> help.
> >> > > > > > > > > > > > >> >>
> >> > > > > > > > > > > > >> >> +1 on setting up a biweekly sync for the
> >> metrics
> >> > > > > > > > architecture.
> >> > > > > > > > > > I’m
> >> > > > > > > > > > > > >> happy
> >> > > > > > > > > > > > >> >> to
> >> > > > > > > > > > > > >> >> join.
> >> > > > > > > > > > > > >> >>
> >> > > > > > > > > > > > >> >> Yufei
> >> > > > > > > > > > > > >> >>
> >> > > > > > > > > > > > >> >>
> >> > > > > > > > > > > > >> >> On Tue, Apr 21, 2026 at 2:34 PM EJ Wang <
> >> > > > > > > > > > > > >> ej.wang.dev.interest@gmail.com>
> >> > > > > > > > > > > > >> >> wrote:
> >> > > > > > > > > > > > >> >>
> >> > > > > > > > > > > > >> >> > Also, I've been looking more closely at
> the
> >> > > > > > *persistence
> >> > > > > > > > > schema
> >> > > > > > > > > > > in
> >> > > > > > > > > > > > >> the
> >> > > > > > > > > > > > >> >> > current metrics work*, and I think
> there's a
> >> > > > > structural
> >> > > > > > > > > > rigidity
> >> > > > > > > > > > > > >> problem
> >> > > > > > > > > > > > >> >> > worth raising before the shape gets locked
> >> in.
> >> > > > > > > > > > > > >> >> >
> >> > > > > > > > > > > > >> >> > Right now we have two separate tables
> >> > > > > > > (scan_metrics_report
> >> > > > > > > > > and
> >> > > > > > > > > > > > >> >> > commit_metrics_report), each with ~25
> >> flattened
> >> > > > > columns
> >> > > > > > > > that
> >> > > > > > > > > > > > directly
> >> > > > > > > > > > > > >> >> > mirror the Iceberg report fields. The SPI
> >> > follows
> >> > > > the
> >> > > > > > > same
> >> > > > > > > > > > split:
> >> > > > > > > > > > > > >> >> > writeScanReport and writeCommitReport as
> >> > separate
> >> > > > > > > methods,
> >> > > > > > > > > with
> >> > > > > > > > > > > > >> per-type
> >> > > > > > > > > > > > >> >> > record classes, converters, and model
> >> objects.
> >> > > *The
> >> > > > > > > > practical
> >> > > > > > > > > > > cost:
> >> > > > > > > > > > > > >> >> > adding a new metric type (operational
> >> metrics,
> >> > > for
> >> > > > > > > example)
> >> > > > > > > > > > > > requires
> >> > > > > > > > > > > > >> a
> >> > > > > > > > > > > > >> >> new
> >> > > > > > > > > > > > >> >> > table, a new SPI method, a new record
> >> class, a
> >> > > new
> >> > > > > > model
> >> > > > > > > > > > class, a
> >> > > > > > > > > > > > new
> >> > > > > > > > > > > > >> >> > converter branch, and a schema migration*.
> >> > > That's a
> >> > > > > lot
> >> > > > > > > of
> >> > > > > > > > > > > surface
> >> > > > > > > > > > > > >> area
> >> > > > > > > > > > > > >> >> > for what should be "one more kind of
> >> metric."
> >> > > > > > > > > > > > >> >> >
> >> > > > > > > > > > > > >> >> > *My bias* would be toward a single metrics
> >> > table
> >> > > > with
> >> > > > > > *a
> >> > > > > > > > > typed
> >> > > > > > > > > > > JSON
> >> > > > > > > > > > > > >> >> > payload*. Something like: metric_type
> >> (enum),
> >> > > > > > entity_id,
> >> > > > > > > > > > > > >> >> > table_identifier, snapshot_id (nullable),
> >> > > > > received_ts,
> >> > > > > > > > > > > > >> schema_version,
> >> > > > > > > > > > > > >> >> and
> >> > > > > > > > > > > > >> >> > a payload column for the metric-specific
> >> data.
> >> > > The
> >> > > > > > > > > metric_type
> >> > > > > > > > > > +
> >> > > > > > > > > > > > >> >> > schema_version pair gives us a
> >> > forward-compatible
> >> > > > > > > contract
> >> > > > > > > > > for
> >> > > > > > > > > > > the
> >> > > > > > > > > > > > >> >> payload
> >> > > > > > > > > > > > >> >> > shape. Adding a new metric type becomes an
> >> enum
> >> > > > value
> >> > > > > > > and a
> >> > > > > > > > > > > payload
> >> > > > > > > > > > > > >> >> schema,
> >> > > > > > > > > > > > >> >> > not a schema migration. One thing I think
> we
> >> > need
> >> > > > to
> >> > > > > be
> >> > > > > > > > > > > deliberate
> >> > > > > > > > > > > > >> >> about is
> >> > > > > > > > > > > > >> >> > the partition key design. If all metric
> >> types
> >> > > land
> >> > > > in
> >> > > > > > one
> >> > > > > > > > > > table,
> >> > > > > > > > > > > > scan
> >> > > > > > > > > > > > >> >> > metrics at scale (high concurrency, high
> >> > > frequency
> >> > > > > > across
> >> > > > > > > > > many
> >> > > > > > > > > > > > >> tables)
> >> > > > > > > > > > > > >> >> > could easily create hot partitions. We'd
> >> want
> >> > the
> >> > > > > > > > persistence
> >> > > > > > > > > > > layer
> >> > > > > > > > > > > > >> to
> >> > > > > > > > > > > > >> >> be
> >> > > > > > > > > > > > >> >> > able to shard by entity or time range, and
> >> that
> >> > > > means
> >> > > > > > the
> >> > > > > > > > > > logical
> >> > > > > > > > > > > > >> schema
> >> > > > > > > > > > > > >> >> > needs to expose enough structure for
> >> backends
> >> > to
> >> > > > > > > partition
> >> > > > > > > > > on.
> >> > > > > > > > > > I
> >> > > > > > > > > > > > >> don't
> >> > > > > > > > > > > > >> >> > think the current flattened layout gives
> us
> >> > that.
> >> > > > > > > > > > > > >> >> >
> >> > > > > > > > > > > > >> >> > This is getting complex enough that I
> don't
> >> > think
> >> > > > > > ad-hoc
> >> > > > > > > > > PR/ML
> >> > > > > > > > > > > > >> threads
> >> > > > > > > > > > > > >> >> > will converge well. *Would people be open
> >> to a
> >> > > > > biweekly
> >> > > > > > > > sync
> >> > > > > > > > > > for
> >> > > > > > > > > > > > >> metrics
> >> > > > > > > > > > > > >> >> > architecture?* I think 30 minutes every
> two
> >> > weeks
> >> > > > > with
> >> > > > > > > > > > interested
> >> > > > > > > > > > > > >> >> parties
> >> > > > > > > > > > > > >> >> > would be enough to work through the
> schema,
> >> SPI
> >> > > > > shape,
> >> > > > > > > and
> >> > > > > > > > > read
> >> > > > > > > > > > > API
> >> > > > > > > > > > > > >> >> design
> >> > > > > > > > > > > > >> >> > together. Happy to help set that up.
> >> > > > > > > > > > > > >> >> >
> >> > > > > > > > > > > > >> >> > -ej
> >> > > > > > > > > > > > >> >> >
> >> > > > > > > > > > > > >> >> > On Mon, Apr 20, 2026 at 2:19 PM EJ Wang <
> >> > > > > > > > > > > > >> ej.wang.dev.interest@gmail.com
> >> > > > > > > > > > > > >> >> >
> >> > > > > > > > > > > > >> >> > wrote:
> >> > > > > > > > > > > > >> >> >
> >> > > > > > > > > > > > >> >> >> Reviewed #4115, left a comment on the
> code
> >> > > > > > organization
> >> > > > > > > > > side.
> >> > > > > > > > > > > > >> >> >>
> >> > > > > > > > > > > > >> >> >> One thing stood out: the metrics write
> path
> >> > > enters
> >> > > > > > > through
> >> > > > > > > > > > > > >> >> >> PolarisMetricsManager on
> MetaStoreManager,
> >> but
> >> > > the
> >> > > > > new
> >> > > > > > > > read
> >> > > > > > > > > > path
> >> > > > > > > > > > > > >> >> bypasses
> >> > > > > > > > > > > > >> >> >> MetaStoreManager entirely and goes
> >> straight to
> >> > > > > > > > > BasePersistence
> >> > > > > > > > > > > via
> >> > > > > > > > > > > > >> >> >> callContext.getMetaStore(). That means
> the
> >> > read
> >> > > > API
> >> > > > > > only
> >> > > > > > > > > works
> >> > > > > > > > > > > for
> >> > > > > > > > > > > > >> >> backends
> >> > > > > > > > > > > > >> >> >> that implement BasePersistence. NoSQL and
> >> > remote
> >> > > > > > > backends
> >> > > > > > > > > > can't
> >> > > > > > > > > > > > >> >> participate.
> >> > > > > > > > > > > > >> >> >>
> >> > > > > > > > > > > > >> >> >> Stepping back, I think the metrics
> >> subsystem
> >> > is
> >> > > > > > growing
> >> > > > > > > > into
> >> > > > > > > > > > > > >> something
> >> > > > > > > > > > > > >> >> >> real (write + read + REST API + AuthZ +
> >> > > > pagination)
> >> > > > > > *but
> >> > > > > > > > the
> >> > > > > > > > > > > > >> >> persistence
> >> > > > > > > > > > > > >> >> >> side is split across two layers in a way
> >> > that's
> >> > > > hard
> >> > > > > > to
> >> > > > > > > > > > > extend*. I
> >> > > > > > > > > > > > >> put
> >> > > > > > > > > > > > >> >> >> together two diagrams to show what I mean
> >> (my
> >> > > best
> >> > > > > > > > effort).
> >> > > > > > > > > > > > >> >> >>
> >> > > > > > > > > > > > >> >> >> *Current state* (Diagram 1): three
> >> interfaces
> >> > at
> >> > > > > three
> >> > > > > > > > > > different
> >> > > > > > > > > > > > >> >> levels.
> >> > > > > > > > > > > > >> >> >> The engine-facing SPI
> >> (PolarisMetricsReporter)
> >> > > is
> >> > > > > > clean.
> >> > > > > > > > But
> >> > > > > > > > > > > > >> >> >> PolarisMetricsManager on MetaStoreManager
> >> is a
> >> > > > > > > passthrough
> >> > > > > > > > > to
> >> > > > > > > > > > > > >> >> >> MetricsPersistence on BasePersistence.
> The
> >> > @Beta
> >> > > > > > > > annotation
> >> > > > > > > > > > and
> >> > > > > > > > > > > > SPI
> >> > > > > > > > > > > > >> >> javadoc
> >> > > > > > > > > > > > >> >> >> are on the BasePersistence layer, while
> the
> >> > > actual
> >> > > > > > > > extension
> >> > > > > > > > > > > > points
> >> > > > > > > > > > > > >> >> >> (PolarisMetricsReporter,
> >> > PolarisMetricsManager)
> >> > > > > carry
> >> > > > > > no
> >> > > > > > > > > > > stability
> >> > > > > > > > > > > > >> >> >> annotation. The write path goes through
> the
> >> > > > > > > > MetaStoreManager
> >> > > > > > > > > > > > layer,
> >> > > > > > > > > > > > >> the
> >> > > > > > > > > > > > >> >> >> read path doesn't.
> >> > > > > > > > > > > > >> >> >>
> >> > > > > > > > > > > > >> >> >> *What I envision* (Diagram 2): two SPIs
> at
> >> two
> >> > > > > levels.
> >> > > > > > > > > > > > >> >> >> PolarisMetricsReporter stays as the
> >> > > engine-facing
> >> > > > > SPI.
> >> > > > > > > > > > > > >> >> >> PolarisMetricsManager becomes the
> >> > backend-facing
> >> > > > SPI
> >> > > > > > > with
> >> > > > > > > > > both
> >> > > > > > > > > > > > write
> >> > > > > > > > > > > > >> >> and
> >> > > > > > > > > > > > >> >> >> read methods at the MetaStoreManager
> level,
> >> > > where
> >> > > > > any
> >> > > > > > > > > backend
> >> > > > > > > > > > > > (JDBC,
> >> > > > > > > > > > > > >> >> NoSQL,
> >> > > > > > > > > > > > >> >> >> remote) can implement them.
> >> MetricsPersistence
> >> > > on
> >> > > > > > > > > > > BasePersistence
> >> > > > > > > > > > > > >> goes
> >> > > > > > > > > > > > >> >> >> away. Where metrics actually land is an
> >> > > > > implementation
> >> > > > > > > > > detail,
> >> > > > > > > > > > > > not a
> >> > > > > > > > > > > > >> >> core
> >> > > > > > > > > > > > >> >> >> interface.
> >> > > > > > > > > > > > >> >> >>
> >> > > > > > > > > > > > >> >> >> *Minor naming thing*:
> >> PolarisMetricsReporter
> >> > is
> >> > > > > > broader
> >> > > > > > > > than
> >> > > > > > > > > > > what
> >> > > > > > > > > > > > it
> >> > > > > > > > > > > > >> >> >> actually handles. It only accepts Iceberg
> >> REST
> >> > > > > Catalog
> >> > > > > > > > > metrics
> >> > > > > > > > > > > > >> >> (ScanReport,
> >> > > > > > > > > > > > >> >> >> CommitReport via MetricsReport). Generic
> >> table
> >> > > > > metrics
> >> > > > > > > or
> >> > > > > > > > > > > > >> operational
> >> > > > > > > > > > > > >> >> >> metrics aren't in scope. Not blocking,
> but
> >> > worth
> >> > > > > > noting
> >> > > > > > > if
> >> > > > > > > > > the
> >> > > > > > > > > > > > >> metrics
> >> > > > > > > > > > > > >> >> >> surface expands.
> >> > > > > > > > > > > > >> >> >>
> >> > > > > > > > > > > > >> >> >> *Rough sketch of how to get there*:
> >> > > > > > > > > > > > >> >> >>  1.  Add read methods to
> >> PolarisMetricsManager
> >> > > > > > > > > > (listScanReports,
> >> > > > > > > > > > > > >> >> >> listCommitReports) with default no-op,
> >> same as
> >> > > the
> >> > > > > > > > existing
> >> > > > > > > > > > > write
> >> > > > > > > > > > > > >> >> methods.
> >> > > > > > > > > > > > >> >> >> (Probably make PolarisMetricsManager more
> >> > > explicit
> >> > > > > on
> >> > > > > > > > being
> >> > > > > > > > > > > > Iceberg
> >> > > > > > > > > > > > >> >> >> specific like package name or class name
> >> etc.)
> >> > > > > > > > > > > > >> >> >>  2.  Wire MetricsReportsService through
> >> > > > > > MetaStoreManager
> >> > > > > > > > > > instead
> >> > > > > > > > > > > > of
> >> > > > > > > > > > > > >> >> >> callContext.getMetaStore().
> >> > > > > > > > > > > > >> >> >>  3.  Extract metrics persistence from
> >> > > > > > > > > JdbcBasePersistenceImpl
> >> > > > > > > > > > > into
> >> > > > > > > > > > > > >> its
> >> > > > > > > > > > > > >> >> >> own class. That file carries ~7
> >> > > responsibilities,
> >> > > > > > > metrics
> >> > > > > > > > > > being
> >> > > > > > > > > > > > one
> >> > > > > > > > > > > > >> of
> >> > > > > > > > > > > > >> >> them.
> >> > > > > > > > > > > > >> >> >>  4.  Remove MetricsPersistence from
> >> > > > BasePersistence.
> >> > > > > > > > > > > > >> >> >>
> >> > > > > > > > > > > > >> >> >> *None of this needs to happen in #4115.
> >> But if
> >> > > the
> >> > > > > > > > direction
> >> > > > > > > > > > > makes
> >> > > > > > > > > > > > >> >> sense,
> >> > > > > > > > > > > > >> >> >> it would be good to align before the
> >> metrics
> >> > > > surface
> >> > > > > > > grows
> >> > > > > > > > > > > > further.
> >> > > > > > > > > > > > >> >> Curious
> >> > > > > > > > > > > > >> >> >> what others think.*
> >> > > > > > > > > > > > >> >> >>
> >> > > > > > > > > > > > >> >> >> *My mental model note*: Level 1
> >> > > MetaStoreManager;
> >> > > > > > level
> >> > > > > > > 2
> >> > > > > > > > > > > > >> transactional
> >> > > > > > > > > > > > >> >> >> persistence; level 3 base persistence
> >> > > > > > > > > > > > >> >> >>
> >> > > > > > > > > > > > >> >> >> Diagram 1
> >> > > > > > > > > > > > >> >> >> <
> >> > > > > > > > > > > > >> >>
> >> > > > > > > > > > > > >>
> >> > > > > > > > > > > >
> >> > > > > > > > > > >
> >> > > > > > > > > >
> >> > > > > > > > >
> >> > > > > > > >
> >> > > > > > >
> >> > > > > >
> >> > > > >
> >> > > >
> >> > >
> >> >
> >>
> https://www.plantuml.com/plantuml/uml/bLHDR-Cs4BthLmpIYupw0zbkKQ1r3M-S7Bp8xhhM7WCOb3IM65EaGD9EX2RzxHrHb4CxRelwa4YSDu_lpOVcnZ9jzvM8BBS2uGjQpJC3dtHMSekPtMk44IpsMgEqa5XcCOhCZikQQLP1pR8TAp2n3ILhmZDP20m0fcIvUkAoW2qJXd9z1bpToO9BX3WXu0ucy5rpgGPNm0nW5_epUWtm2Ue3pn3kMOFQmKntGZW0BYtgBSi8k5A2QMwybJNMIbFiGSR9QZc4nUqIvikStF0jHprua5C-amge42aNt3R0f5JaaoivdV2Pkqbx4hee4ymOkBh5BTiB-_uIeGeo8zL8rPsPl4DktdEiK1jkB1NdZCRbrSTecDe_mlHbF0wvBmCkaOH5_S8a_TTTKI6-nmCAkEw4LpxsZ-LbYLKQFKMNOgf_wuM7_bV9gOer5SYMMksBSWXFcbi49KNZXNLicwfe3TETC7gPdPqI7uBcHMb1RSzYq34c6PDUM9mn8HRsUTZEiDBve3NjVZumBj0U7SS37mGO7vcwtiK-_pU7U7L_f-digo9YbhSwIfMRwIITKGXbxdIUTCGF1SeCJxloKsU-3k9ddRbX1eDq1q_fx1JbBGT0glVyXimDuP4TQ5qpCAmnGEj2s_6n5mtn1z-97-63itFQZLPO1Ev2tu_WF7Ju-VPc0Skg5bYXxBhkY1xpD7EM_7fyflSpIsqMgVth5xhVr4eQxWQ8enaSAJQSG16yFSDuJ798rrcXr_3n-lfdk7icQjEBmFujL7AodiP_Y4Z7-YxvtZNs4zMgpNTl6tF8sglyPsmqchrjvQ-m-aP94r-TwCA2Ka8upPJZwtvSpoYCXkYMZU2NXvRMBfq9P3i3Le4VAZUAlUZ_oPKsxPgY0Q_BSKLkyr9bhQhQrJjo_x3TPlIB0DPjnMfcIoYP0QaYw1a0fTKDr8fB6ntNuvmoL1ZGkXa69Njh43zf9GiGxHQrA_jDYWRSzF5--WmTVrN97_Sm8LbLUy_lGBmLanJjFkDlGkRqjA_4tm00
> >> > > > > > > > > > > > >> >> >
> >> > > > > > > > > > > > >> >> >> :
> >> > > > > > > > > > > > >> >> >>
> >> > > > > > > > > > > > >> >> >> [image: image.png]
> >> > > > > > > > > > > > >> >> >>
> >> > > > > > > > > > > > >> >> >> Diagram 2
> >> > > > > > > > > > > > >> >> >> <
> >> > > > > > > > > > > > >> >>
> >> > > > > > > > > > > > >>
> >> > > > > > > > > > > >
> >> > > > > > > > > > >
> >> > > > > > > > > >
> >> > > > > > > > >
> >> > > > > > > >
> >> > > > > > >
> >> > > > > >
> >> > > > >
> >> > > >
> >> > >
> >> >
> >>
> https://www.plantuml.com/plantuml/uml/VLLDR-8m4BtdLupO2sWBLVU8AaGB7AXAbssGzb896SS9RXqxjKqBwkv_tt7iV43fdaZYDpFlpRmnOsE9jhjSH9PRmM31hERKm8scMsuPjJlDe0yheZDc8RR4iYWoBrmMH9CS2a9VICPYUy1OZN0YCy5Q0BCbYNhdCeEK28En8G8wCvbnoQ0R8_05Bc6bkLIz3X03p1zzH7zR-9ZfDquPt9C3qoNCX2yV4G2NbkcKu5jdgGJHt0GbZwnG6i-UP3TUpk5gM6Ldqke350eZUqzoCft3U9xWHvxoa5-7K4nF1J46EbEMafsmdrCBbQ44gVggy18IZrn_ph5asd1ZiIKdQSgueZvjXrQFSFrdC3YN-nXmBacxbGiYyLVxLaBtdhqn0LSzdBDhqQtQoOJeGyad3z0lUqnYgpGB6Ns8oVyta00Dy_WnX0tIOZ8v6SYxHll1TrH6aejAik-mh-AphVFCwSUQqFypElag5QRGFDjQKEd96K1P8QP41c9TzA_IIQyvdAWyv_RSiS3skb0_EzDDkK2v5xWF6MiGFlvhpFLcD2Dq2pml14gaF67eQkmd8gulDoC4kSOu6KVpkvlUJg1RTbWISU40RdBUUS_9XfRZ2dwxm_SW8LYFISgm_MnlDQ6M9P1gbKEc4X-2pH_FvJCkCqm9pbVjD6LrwdLeOrDWfOaqc8Wh9BE85oNKxkNQ6o4yGRy_Eae0G_G8tZv81d3bHDB23WOdisohVr3nh_j6lbSjbNaLRTc8UgtPbAU1J_tygOfZX9DWEJeHDvYx-qmSi5FgNLPZwHrHcUsncGQ5-skhUclpE5fo4ounpFauYrUbkU6ccfnxMvitwag4IyerhTxj8In_Oj1bDO4pQru674loYrGlULHLEGCjwJJ8gDoVZR8MxO4BT3IzRvIcAQKezC6xpziGnTyImrfEGyJI_OcKfgtxIvnTqFEMS17L9Z-jsARN5FmTheP7HtSdtOMT0B4GY2FYHXxgQmMtj2bRqiLFGapiVe1_QVKDrkqXcm83aFEXnMYCZ-xlyHy
> >> > > > > > > > > > > > >> >> >
> >> > > > > > > > > > > > >> >> >> :
> >> > > > > > > > > > > > >> >> >> [image: image.png]
> >> > > > > > > > > > > > >> >> >>
> >> > > > > > > > > > > > >> >> >>  -ej
> >> > > > > > > > > > > > >> >> >>
> >> > > > > > > > > > > > >> >> >> On Wed, Apr 15, 2026 at 8:22 AM Dmitri
> >> > > > Bourlatchkov
> >> > > > > <
> >> > > > > > > > > > > > >> dimas@apache.org>
> >> > > > > > > > > > > > >> >> >> wrote:
> >> > > > > > > > > > > > >> >> >>
> >> > > > > > > > > > > > >> >> >>> Hi All,
> >> > > > > > > > > > > > >> >> >>>
> >> > > > > > > > > > > > >> >> >>> Heads up: The current state of PR [4115]
> >> > looks
> >> > > > > pretty
> >> > > > > > > > solid
> >> > > > > > > > > > to
> >> > > > > > > > > > > > me.
> >> > > > > > > > > > > > >> I
> >> > > > > > > > > > > > >> >> >>> believe this PR is approaching a
> mergeable
> >> > > > > condition.
> >> > > > > > > > > > > > >> >> >>>
> >> > > > > > > > > > > > >> >> >>> Please post your reviews if you have any
> >> > > > comments.
> >> > > > > > > > > > > > >> >> >>>
> >> > > > > > > > > > > > >> >> >>> [4115]
> >> > > > https://github.com/apache/polaris/pull/4115
> >> > > > > > > > > > > > >> >> >>>
> >> > > > > > > > > > > > >> >> >>> Thanks,
> >> > > > > > > > > > > > >> >> >>> Dmitri.
> >> > > > > > > > > > > > >> >> >>>
> >> > > > > > > > > > > > >> >> >>> On Tue, Mar 3, 2026 at 3:29 PM Anand
> Kumar
> >> > > > Sankaran
> >> > > > > > via
> >> > > > > > > > > dev <
> >> > > > > > > > > > > > >> >> >>> dev@polaris.apache.org> wrote:
> >> > > > > > > > > > > > >> >> >>>
> >> > > > > > > > > > > > >> >> >>> > Hi Yufei and Dmitri,
> >> > > > > > > > > > > > >> >> >>> >
> >> > > > > > > > > > > > >> >> >>> > Here is a proposal for the REST
> >> endpoints
> >> > for
> >> > > > > > metrics
> >> > > > > > > > and
> >> > > > > > > > > > > > events.
> >> > > > > > > > > > > > >> >> >>> >
> >> > > > > > > > > > > > >> >> >>> >
> >> > > > > > https://github.com/apache/polaris/pull/3924/changes
> >> > > > > > > > > > > > >> >> >>> >
> >> > > > > > > > > > > > >> >> >>> > I did not see any precursors for
> >> raising a
> >> > PR
> >> > > > for
> >> > > > > > > > > > proposals,
> >> > > > > > > > > > > so
> >> > > > > > > > > > > > >> >> trying
> >> > > > > > > > > > > > >> >> >>> > this.  Please let me know what you
> >> think.
> >> > > > > > > > > > > > >> >> >>> >
> >> > > > > > > > > > > > >> >> >>> > -
> >> > > > > > > > > > > > >> >> >>> > Anand
> >> > > > > > > > > > > > >> >> >>> >
> >> > > > > > > > > > > > >> >> >>> > From: Anand Kumar Sankaran <
> >> > > > > > > anand.sankaran@workday.com
> >> > > > > > > > >
> >> > > > > > > > > > > > >> >> >>> > Date: Monday, March 2, 2026 at
> 10:25 AM
> >> > > > > > > > > > > > >> >> >>> > To: dev@polaris.apache.org <
> >> > > > > dev@polaris.apache.org
> >> > > > > > >
> >> > > > > > > > > > > > >> >> >>> > Subject: Re: Polaris Telemetry and
> Audit
> >> > > Trail
> >> > > > > > > > > > > > >> >> >>> >
> >> > > > > > > > > > > > >> >> >>> > About the REST API, based on my use
> >> cases:
> >> > > > > > > > > > > > >> >> >>> >
> >> > > > > > > > > > > > >> >> >>> >
> >> > > > > > > > > > > > >> >> >>> >   1.
> >> > > > > > > > > > > > >> >> >>> > I want to be able to query commit
> >> metrics
> >> > to
> >> > > > > track
> >> > > > > > > > files
> >> > > > > > > > > > > added
> >> > > > > > > > > > > > /
> >> > > > > > > > > > > > >> >> >>> removed
> >> > > > > > > > > > > > >> >> >>> > per commit, along with record counts.
> >> The
> >> > > > > ingestion
> >> > > > > > > > > > pipeline
> >> > > > > > > > > > > > that
> >> > > > > > > > > > > > >> >> >>> writes
> >> > > > > > > > > > > > >> >> >>> > this data is owned by us and we are
> >> > > guaranteed
> >> > > > to
> >> > > > > > > write
> >> > > > > > > > > > this
> >> > > > > > > > > > > > >> >> >>> information
> >> > > > > > > > > > > > >> >> >>> > for each write.
> >> > > > > > > > > > > > >> >> >>> >   2.
> >> > > > > > > > > > > > >> >> >>> > I want to be able to query scan
> metrics
> >> for
> >> > > > > read. I
> >> > > > > > > > > > > understand
> >> > > > > > > > > > > > >> >> clients
> >> > > > > > > > > > > > >> >> >>> do
> >> > > > > > > > > > > > >> >> >>> > not fulfill this requirement.
> >> > > > > > > > > > > > >> >> >>> >   3.
> >> > > > > > > > > > > > >> >> >>> > I want to be able to query the events
> >> table
> >> > > > > (events
> >> > > > > > > are
> >> > > > > > > > > > > > >> persisted) -
> >> > > > > > > > > > > > >> >> >>> this
> >> > > > > > > > > > > > >> >> >>> > may supersede #2, I am not sure yet.
> >> > > > > > > > > > > > >> >> >>> >
> >> > > > > > > > > > > > >> >> >>> > All this information is in the JDBC
> >> based
> >> > > > > > persistence
> >> > > > > > > > > model
> >> > > > > > > > > > > and
> >> > > > > > > > > > > > >> is
> >> > > > > > > > > > > > >> >> >>> > persisted in the metastore. I
> currently
> >> > don’t
> >> > > > > have
> >> > > > > > a
> >> > > > > > > > need
> >> > > > > > > > > > to
> >> > > > > > > > > > > > >> query
> >> > > > > > > > > > > > >> >> >>> > prometheus or open telemetry. I do
> >> publish
> >> > > some
> >> > > > > > > events
> >> > > > > > > > to
> >> > > > > > > > > > > > >> Prometheus
> >> > > > > > > > > > > > >> >> >>> and
> >> > > > > > > > > > > > >> >> >>> > they are forwarded to our dashboards
> >> > > elsewhere.
> >> > > > > > > > > > > > >> >> >>> >
> >> > > > > > > > > > > > >> >> >>> > About the CLI utilities, I meant the
> >> admin
> >> > > user
> >> > > > > > > > > utilities.
> >> > > > > > > > > > In
> >> > > > > > > > > > > > >> one of
> >> > > > > > > > > > > > >> >> >>> the
> >> > > > > > > > > > > > >> >> >>> > earliest drafts of my proposal,
> Prashant
> >> > > > > mentioned
> >> > > > > > > that
> >> > > > > > > > > the
> >> > > > > > > > > > > > >> metrics
> >> > > > > > > > > > > > >> >> >>> tables
> >> > > > > > > > > > > > >> >> >>> > can grow indefinitely and that a
> similar
> >> > > > problem
> >> > > > > > > exists
> >> > > > > > > > > > with
> >> > > > > > > > > > > > the
> >> > > > > > > > > > > > >> >> events
> >> > > > > > > > > > > > >> >> >>> > table as well. We discussed that
> >> cleaning
> >> > up
> >> > > of
> >> > > > > old
> >> > > > > > > > > records
> >> > > > > > > > > > > > from
> >> > > > > > > > > > > > >> >> both
> >> > > > > > > > > > > > >> >> >>> > metrics tables and events tables can
> be
> >> > done
> >> > > > via
> >> > > > > a
> >> > > > > > > CLI
> >> > > > > > > > > > > utility.
> >> > > > > > > > > > > > >> >> >>> >
> >> > > > > > > > > > > > >> >> >>> > I see that Yufei has covered the
> >> discussion
> >> > > > about
> >> > > > > > > > > > > datasources.
> >> > > > > > > > > > > > >> >> >>> >
> >> > > > > > > > > > > > >> >> >>> > -
> >> > > > > > > > > > > > >> >> >>> > Anand
> >> > > > > > > > > > > > >> >> >>> >
> >> > > > > > > > > > > > >> >> >>> >
> >> > > > > > > > > > > > >> >> >>> >
> >> > > > > > > > > > > > >> >> >>> > From: Yufei Gu <fl...@gmail.com>
> >> > > > > > > > > > > > >> >> >>> > Date: Friday, February 27, 2026 at
> >> 9:54 PM
> >> > > > > > > > > > > > >> >> >>> > To: dev@polaris.apache.org <
> >> > > > > dev@polaris.apache.org
> >> > > > > > >
> >> > > > > > > > > > > > >> >> >>> > Subject: Re: Polaris Telemetry and
> Audit
> >> > > Trail
> >> > > > > > > > > > > > >> >> >>> >
> >> > > > > > > > > > > > >> >> >>> > This Message Is From an External
> Sender
> >> > > > > > > > > > > > >> >> >>> > This message came from outside your
> >> > > > organization.
> >> > > > > > > > > > > > >> >> >>> > Report Suspicious<
> >> > > > > > > > > > > > >> >> >>> >
> >> > > > > > > > > > > > >> >> >>>
> >> > > > > > > > > > > > >> >>
> >> > > > > > > > > > > > >>
> >> > > > > > > > > > > >
> >> > > > > > > > > > >
> >> > > > > > > > > >
> >> > > > > > > > >
> >> > > > > > > >
> >> > > > > > >
> >> > > > > >
> >> > > > >
> >> > > >
> >> > >
> >> >
> >>
> https://us-phishalarm-ewt.proofpoint.com/EWT/v1/Iz9xO38YGHZK!YhNDZABkHi1B699ote2uMwpOZw8i0QMCGO2Szc-HshuABGhGvwPJcymE6G2oUUxtS8xDkSrtGTPm_I3QnVDHoLMk50m9v8z_nZKTkd-bnVUbreF1u0WnfV_X5eYevZl_$
> >> > > > > > > > > > > > >> >> >>> > >
> >> > > > > > > > > > > > >> >> >>> >
> >> > > > > > > > > > > > >> >> >>> >
> >> > > > > > > > > > > > >> >> >>> > As I mentioned in
> >> > > > > > > > > > > > >> >> >>> >
> >> > > > > > > > > > > > >> >> >>>
> >> > > > > > > > > > > > >> >>
> >> > > > > > > > > > > > >>
> >> > > > > > > > > > > >
> >> > > > > > > > > > >
> >> > > > > > > > > >
> >> > > > > > > > >
> >> > > > > > > >
> >> > > > > > >
> >> > > > > >
> >> > > > >
> >> > > >
> >> > >
> >> >
> >>
> https://urldefense.com/v3/__https://github.com/apache/polaris/issues/3890__;!!Iz9xO38YGHZK!5EuyFFkk3vhRWVIRvQAWBSQfpJkTMA9HxugzDwXmN0LPPqhEFxYkFRGVhtb8AqUwXtDh2OplcMnbMDHKOxrvDU0$
> >> > > > > > > > > > > > >> >> >>> ,
> >> > > > > > > > > > > > >> >> >>> > supporting
> >> > > > > > > > > > > > >> >> >>> > multiple data sources is not a trivial
> >> > > change.
> >> > > > I
> >> > > > > > > would
> >> > > > > > > > > > > strongly
> >> > > > > > > > > > > > >> >> >>> recommend
> >> > > > > > > > > > > > >> >> >>> > starting with a design document to
> >> > carefully
> >> > > > > > evaluate
> >> > > > > > > > the
> >> > > > > > > > > > > > >> >> architectural
> >> > > > > > > > > > > > >> >> >>> > implications and long term impact.
> >> > > > > > > > > > > > >> >> >>> >
> >> > > > > > > > > > > > >> >> >>> > A REST endpoint to query metrics seems
> >> > > > reasonable
> >> > > > > > > given
> >> > > > > > > > > the
> >> > > > > > > > > > > > >> current
> >> > > > > > > > > > > > >> >> >>> JDBC
> >> > > > > > > > > > > > >> >> >>> > based persistence model. That said, we
> >> may
> >> > > also
> >> > > > > > > > consider
> >> > > > > > > > > > > > >> alternative
> >> > > > > > > > > > > > >> >> >>> > storage models. For example, if we
> later
> >> > > adopt
> >> > > > a
> >> > > > > > time
> >> > > > > > > > > > series
> >> > > > > > > > > > > > >> system
> >> > > > > > > > > > > > >> >> >>> such as
> >> > > > > > > > > > > > >> >> >>> > Prometheus to store metrics, the query
> >> > model
> >> > > > and
> >> > > > > > > access
> >> > > > > > > > > > > > patterns
> >> > > > > > > > > > > > >> >> would
> >> > > > > > > > > > > > >> >> >>> be
> >> > > > > > > > > > > > >> >> >>> > fundamentally different. Designing the
> >> REST
> >> > > API
> >> > > > > > > without
> >> > > > > > > > > > > > >> considering
> >> > > > > > > > > > > > >> >> >>> these
> >> > > > > > > > > > > > >> >> >>> > potential evolutions may limit
> >> flexibility.
> >> > > I'd
> >> > > > > > > suggest
> >> > > > > > > > > to
> >> > > > > > > > > > > > start
> >> > > > > > > > > > > > >> >> with
> >> > > > > > > > > > > > >> >> >>> the
> >> > > > > > > > > > > > >> >> >>> > use case.
> >> > > > > > > > > > > > >> >> >>> >
> >> > > > > > > > > > > > >> >> >>> > Yufei
> >> > > > > > > > > > > > >> >> >>> >
> >> > > > > > > > > > > > >> >> >>> >
> >> > > > > > > > > > > > >> >> >>> > On Fri, Feb 27, 2026 at 3:42 PM Dmitri
> >> > > > > > Bourlatchkov <
> >> > > > > > > > > > > > >> >> dimas@apache.org>
> >> > > > > > > > > > > > >> >> >>> > wrote:
> >> > > > > > > > > > > > >> >> >>> >
> >> > > > > > > > > > > > >> >> >>> > > Hi Anand,
> >> > > > > > > > > > > > >> >> >>> > >
> >> > > > > > > > > > > > >> >> >>> > > Sharing my view... subject to
> >> discussion:
> >> > > > > > > > > > > > >> >> >>> > >
> >> > > > > > > > > > > > >> >> >>> > > 1. Adding non-IRC REST API to
> Polaris
> >> is
> >> > > > > > perfectly
> >> > > > > > > > > fine.
> >> > > > > > > > > > > > >> >> >>> > >
> >> > > > > > > > > > > > >> >> >>> > > Figuring out specific endpoint URIs
> >> and
> >> > > > > payloads
> >> > > > > > > > might
> >> > > > > > > > > > > > require
> >> > > > > > > > > > > > >> a
> >> > > > > > > > > > > > >> >> few
> >> > > > > > > > > > > > >> >> >>> > > roundtrips, so opening a separate
> >> thread
> >> > > for
> >> > > > > that
> >> > > > > > > > might
> >> > > > > > > > > > be
> >> > > > > > > > > > > > >> best.
> >> > > > > > > > > > > > >> >> >>> > > Contributors commonly create Google
> >> Docs
> >> > > for
> >> > > > > new
> >> > > > > > > API
> >> > > > > > > > > > > > proposals
> >> > > > > > > > > > > > >> too
> >> > > > > > > > > > > > >> >> >>> (they
> >> > > > > > > > > > > > >> >> >>> > > fairly easy to update as the email
> >> > > discussion
> >> > > > > > > > > > progresses).
> >> > > > > > > > > > > > >> >> >>> > >
> >> > > > > > > > > > > > >> >> >>> > > There was a suggestion to try
> Markdown
> >> > > (with
> >> > > > > PRs)
> >> > > > > > > for
> >> > > > > > > > > > > > proposals
> >> > > > > > > > > > > > >> >> [1]
> >> > > > > > > > > > > > >> >> >>> ...
> >> > > > > > > > > > > > >> >> >>> > > feel free to give it a try if you
> are
> >> > > > > comfortable
> >> > > > > > > > with
> >> > > > > > > > > > > that.
> >> > > > > > > > > > > > >> >> >>> > >
> >> > > > > > > > > > > > >> >> >>> > > 2. Could you clarify whether you
> mean
> >> end
> >> > > > user
> >> > > > > > > > > utilities
> >> > > > > > > > > > or
> >> > > > > > > > > > > > >> admin
> >> > > > > > > > > > > > >> >> >>> user
> >> > > > > > > > > > > > >> >> >>> > > utilities? In the latter case those
> >> might
> >> > > be
> >> > > > > more
> >> > > > > > > > > > suitable
> >> > > > > > > > > > > > for
> >> > > > > > > > > > > > >> the
> >> > > > > > > > > > > > >> >> >>> Admin
> >> > > > > > > > > > > > >> >> >>> > > CLI (java) not the Python CLI, IMHO.
> >> > > > > > > > > > > > >> >> >>> > >
> >> > > > > > > > > > > > >> >> >>> > > Why would these utilities be common
> >> with
> >> > > > > events?
> >> > > > > > > > IMHO,
> >> > > > > > > > > > > event
> >> > > > > > > > > > > > >> use
> >> > > > > > > > > > > > >> >> >>> cases
> >> > > > > > > > > > > > >> >> >>> > are
> >> > > > > > > > > > > > >> >> >>> > > distinct from scan/commit metrics.
> >> > > > > > > > > > > > >> >> >>> > >
> >> > > > > > > > > > > > >> >> >>> > > 3. I'd prefer separating metrics
> >> > > persistence
> >> > > > > from
> >> > > > > > > > > > MetaStore
> >> > > > > > > > > > > > >> >> >>> persistence
> >> > > > > > > > > > > > >> >> >>> > at
> >> > > > > > > > > > > > >> >> >>> > > the code level, so that they could
> be
> >> > mixed
> >> > > > and
> >> > > > > > > > matched
> >> > > > > > > > > > > > >> >> >>> independently.
> >> > > > > > > > > > > > >> >> >>> > The
> >> > > > > > > > > > > > >> >> >>> > > separate datasource question will
> >> become
> >> > a
> >> > > > > > > non-issue
> >> > > > > > > > > with
> >> > > > > > > > > > > > that
> >> > > > > > > > > > > > >> >> >>> approach,
> >> > > > > > > > > > > > >> >> >>> > I
> >> > > > > > > > > > > > >> >> >>> > > guess.
> >> > > > > > > > > > > > >> >> >>> > >
> >> > > > > > > > > > > > >> >> >>> > > The rationale for separating scan
> >> metrics
> >> > > and
> >> > > > > > > > metastore
> >> > > > > > > > > > > > >> >> persistence
> >> > > > > > > > > > > > >> >> >>> is
> >> > > > > > > > > > > > >> >> >>> > that
> >> > > > > > > > > > > > >> >> >>> > > "cascading deletes" between them are
> >> > hardly
> >> > > > > ever
> >> > > > > > > > > > required.
> >> > > > > > > > > > > > >> >> >>> Furthermore,
> >> > > > > > > > > > > > >> >> >>> > the
> >> > > > > > > > > > > > >> >> >>> > > data and query patterns are very
> >> > different
> >> > > so
> >> > > > > > > > different
> >> > > > > > > > > > > > >> >> technologies
> >> > > > > > > > > > > > >> >> >>> > might
> >> > > > > > > > > > > > >> >> >>> > > be beneficial in each case.
> >> > > > > > > > > > > > >> >> >>> > >
> >> > > > > > > > > > > > >> >> >>> > > [1]
> >> > > > > > > > > > > > >> >> >>> >
> >> > > > > > > > > > > > >> >> >>>
> >> > > > > > > > > > > > >> >>
> >> > > > > > > > > > > > >>
> >> > > > > > > > > > > >
> >> > > > > > > > > > >
> >> > > > > > > > > >
> >> > > > > > > > >
> >> > > > > > > >
> >> > > > > > >
> >> > > > > >
> >> > > > >
> >> > > >
> >> > >
> >> >
> >>
> https://urldefense.com/v3/__https://lists.apache.org/thread/yto2wp982t43h1mqjwnslswhws5z47cy__;!!Iz9xO38YGHZK!5EuyFFkk3vhRWVIRvQAWBSQfpJkTMA9HxugzDwXmN0LPPqhEFxYkFRGVhtb8AqUwXtDh2OplcMnbMDHKxYDakNU$
> >> > > > > > > > > > > > >> >> >>> > >
> >> > > > > > > > > > > > >> >> >>> > > Cheers,
> >> > > > > > > > > > > > >> >> >>> > > Dmitri.
> >> > > > > > > > > > > > >> >> >>> > >
> >> > > > > > > > > > > > >> >> >>> > > On Fri, Feb 27, 2026 at 6:19 PM
> Anand
> >> > Kumar
> >> > > > > > > Sankaran
> >> > > > > > > > > via
> >> > > > > > > > > > > dev
> >> > > > > > > > > > > > <
> >> > > > > > > > > > > > >> >> >>> > > dev@polaris.apache.org> wrote:
> >> > > > > > > > > > > > >> >> >>> > >
> >> > > > > > > > > > > > >> >> >>> > > > Thanks all. This PR is merged now.
> >> > > > > > > > > > > > >> >> >>> > > >
> >> > > > > > > > > > > > >> >> >>> > > > Here are the follow-up features /
> >> work
> >> > > > > needed.
> >> > > > > > > > These
> >> > > > > > > > > > > were
> >> > > > > > > > > > > > >> all
> >> > > > > > > > > > > > >> >> >>> part of
> >> > > > > > > > > > > > >> >> >>> > > the
> >> > > > > > > > > > > > >> >> >>> > > > merged PR at some point in time
> and
> >> > were
> >> > > > > > removed
> >> > > > > > > to
> >> > > > > > > > > > > reduce
> >> > > > > > > > > > > > >> >> scope.
> >> > > > > > > > > > > > >> >> >>> > > >
> >> > > > > > > > > > > > >> >> >>> > > > Please let me know what you think.
> >> > > > > > > > > > > > >> >> >>> > > >
> >> > > > > > > > > > > > >> >> >>> > > >
> >> > > > > > > > > > > > >> >> >>> > > >   1.  A REST API to paginate
> through
> >> > > table
> >> > > > > > > metrics.
> >> > > > > > > > > > This
> >> > > > > > > > > > > > >> will be
> >> > > > > > > > > > > > >> >> >>> > non-IRC
> >> > > > > > > > > > > > >> >> >>> > > > standard addition.
> >> > > > > > > > > > > > >> >> >>> > > >   2.  Utilities for managing old
> >> > records,
> >> > > > > > should
> >> > > > > > > be
> >> > > > > > > > > > > common
> >> > > > > > > > > > > > >> with
> >> > > > > > > > > > > > >> >> >>> events.
> >> > > > > > > > > > > > >> >> >>> > > > There was some discussion that it
> >> > belongs
> >> > > > to
> >> > > > > > the
> >> > > > > > > > CLI.
> >> > > > > > > > > > > > >> >> >>> > > >   3.  Separate datasource
> (metrics,
> >> > > events,
> >> > > > > > even
> >> > > > > > > > > other
> >> > > > > > > > > > > > >> tables?).
> >> > > > > > > > > > > > >> >> >>> > > >
> >> > > > > > > > > > > > >> >> >>> > > >
> >> > > > > > > > > > > > >> >> >>> > > > Anything else?
> >> > > > > > > > > > > > >> >> >>> > > >
> >> > > > > > > > > > > > >> >> >>> > > > -
> >> > > > > > > > > > > > >> >> >>> > > > Anand
> >> > > > > > > > > > > > >> >> >>> > > >
> >> > > > > > > > > > > > >> >> >>> > > >
> >> > > > > > > > > > > > >> >> >>> > >
> >> > > > > > > > > > > > >> >> >>> >
> >> > > > > > > > > > > > >> >> >>> >
> >> > > > > > > > > > > > >> >> >>>
> >> > > > > > > > > > > > >> >> >>
> >> > > > > > > > > > > > >> >>
> >> > > > > > > > > > > > >> >
> >> > > > > > > > > > > > >>
> >> > > > > > > > > > > > >
> >> > > > > > > > > > > >
> >> > > > > > > > > > >
> >> > > > > > > > > >
> >> > > > > > > > >
> >> > > > > > > >
> >> > > > > > >
> >> > > > > >
> >> > > > >
> >> > > >
> >> > >
> >> >
> >>
> >
>

Re: Proposal for REST endpoints for table metrics and events

[Dmitri Bourlatchkov <di...@apache.org>]

Hi All,

It looks like Anand addressed all action items in [4115].

Please review the latest state of the code. The PR LGTM and I approved in
GH from my side.

[4115] https://github.com/apache/polaris/pull/4115

Thanks,
Dmitri.

On Thu, Jun 25, 2026 at 1:56 PM Dmitri Bourlatchkov <di...@apache.org>
wrote:

> Hi All,
>
> Recap of the discussion in today's Community Sync:
>
> * The modular approach for Metrics SPI(s) and implementation seems to have
> community consensus (please comment if you still have concerns)
>
> * What modules / feature flags are enabled in Apache Polaris images by
> default will be discussed later (when the feature is close to completion).
>
> In any case, users will be able to turn the full feature on/off at their
> discretion, regardless of the default flag values.
>
> * It is generally preferable to have smaller PRs and introduce new
> features gradually to simplify reviews.
>
> What this means for the Metrics API, IMHO is:
>
> a) The scope of PR 4115 is probably ok as it stands right now. If someone
> wants to propose splitting it further, please comment.
>
> b) After 4115 is merged, Metrics Persistence PRs will follow, with smaller
> scope per PR as appropriate for the nature of the code changes.
>
> * The major concern was about exposing the new Metrics REST API to end
> users. The nature of this concern was about avoiding the (false) impression
> that the new API is a firm and adopted Polaris specification.
>
> I think this concern is reasonable.
>
> I propose that Anand update the documentation sections of PR 4115 to
> emphasise that the API is currently an experimental / POC API at this time
> and is subject to change (including breaking changes). I believe it already
> has the "beta" label. So this request is mostly for doc changes to clarify
> user expectations, since the "beta" label alone may not be interpreted the
> same way by all users.
>
> I hope this update will resolve all remaining concerns with 4115 and allow
> for it to be merged so that the community will be able to proceed to
> follow-up PRs (as noted above).
>
> Please add/correct me if I missed anything relevant.
>
> Thanks,
> Dmitri.
>
> On Wed, Jun 24, 2026 at 8:11 PM Yufei Gu <fl...@gmail.com> wrote:
>
>> I do not think we have agreement on the query REST APIs yet.
>>
>> In particular, the current query API shape seems highly coupled to the
>> *example[1]* metrics in the Iceberg REST spec. I do not think that is a
>> good way to design a REST API contract. For example, fields like
>> resultDataFiles, resultDeleteFiles, totalFileSizeBytes,
>> totalDataManifests,
>> totalDeleteManifests, and scannedDataManifests are very specific to one
>> style of scan metric. That makes the API look more like a direct
>> projection
>> of the current example payload than a general query model.
>>
>> My preference is that Polaris should not define or strongly opine on the
>> Iceberg scan or commit metrics query REST API at this point. Polaris can
>> resolve and authorize the table, accept the metrics report, and delegate
>> it
>> to the configured reporting path. Durable storage, filtering, dashboards,
>> and query APIs should remain downstream impl. choices unless and until the
>> Iceberg community standardizes a query contract, which I doubt will
>> happen.
>>
>> So I do not think the intake path has to be blocked, but I would prefer
>> not
>> to merge a Polaris specific query API that may be hard to evolve later.
>>
>> 1.
>>
>> https://github.com/apache/iceberg/blob/2a6c556c0c883c04c6d3fbd68e6aeda36b91e0aa/open-api/rest-catalog-open-api.yaml#L4152
>>
>> Yufei
>>
>>
>> On Tue, Jun 23, 2026 at 9:30 AM Dmitri Bourlatchkov <di...@apache.org>
>> wrote:
>>
>> > Hi All,
>> >
>> > So, what are the blockers for merging [4115], if any?
>> >
>> > [4115] https://github.com/apache/polaris/pull/4115
>> >
>> > Thanks,
>> > Dmitri.
>> >
>> > On Wed, Jun 17, 2026 at 5:28 PM EJ Wang <ej.wang.dev.interest@gmail.com
>> >
>> > wrote:
>> >
>> > > Hi folks,
>> > >
>> > > Thanks again for the discussion today. I updated the sync doc with
>> notes
>> > > from today's metrics sync:
>> > >
>> > >
>> > >
>> >
>> https://docs.google.com/document/d/100h7c4damrUzVuquYbBHM0EvA4LSWuW2IT2dN_7nYVA/edit?tab=t.ezk0rgdx0c6m
>> > >
>> > > A few highlights:
>> > >
>> > >    - No objection to keeping the default metrics behavior small:
>> > >    no-op/log-only is enough as the built-in default.
>> > >    - Durable storage, event forwarding, external queues, dashboards,
>> and
>> > >    custom filtering should be implementation choices behind the
>> metrics
>> > >    reporting path.
>> > >    - For PR #4115, the REST layer should resolve/authz/accept the
>> report,
>> > >    then delegate to the selected metrics implementation. It should not
>> > own
>> > >    durable storage semantics.
>> > >    - The rough boundary we discussed is: spec owns the wire contract,
>> > >    runtime owns framework wiring, API/contract modules own
>> > provider-facing
>> > >    contracts, and extensions own replaceable implementations.
>> > >    - OpenLineage has a similar path-naming question. We should avoid
>> > >    occupying a generic lineage path too early if we want room for
>> other
>> > >    lineage systems later.
>> > >
>> > > I believe the boundary rules previewed in sync meeting are useful for
>> > > reviewing PR #4115, but it is not
>> > > a final module-layout proposal.
>> > >
>> > > For metrics specifically, I think the target remains:
>> > >    runtime
>> > >      -> metrics reporting/emitting contract
>> > >      -> default implementation
>> > >           -> lower implementation details, if needed
>> > >
>> > > That keeps the default simple, while allowing durable JDBC,
>> event-backed,
>> > > or external-queue implementations to be added separately.
>> > >
>> > > We also touched on the split between Iceberg metrics emitting and
>> > > Polaris-owned metrics querying. The querying side can keep evolving,
>> > > especially for dashboard or generic-table use cases, but I do not
>> think
>> > > that needs to block the intake path in this PR.
>> > >
>> > > Thanks,
>> > > -ej
>> > >
>> > > On Tue, Jun 16, 2026 at 2:17 PM Yufei Gu <fl...@gmail.com>
>> wrote:
>> > >
>> > > > Thanks for chiming in, EJ. Agreed that the default battery should
>> stay
>> > > > small. I'm leaning toward not including metrics persistence in the
>> > > default
>> > > > battery.
>> > > >
>> > > > Yufei
>> > > >
>> > > >
>> > > > On Thu, Jun 11, 2026 at 8:30 PM EJ Wang <
>> > ej.wang.dev.interest@gmail.com>
>> > > > wrote:
>> > > >
>> > > > > Hi Yufei,
>> > > > >
>> > > > > Thanks for connecting this back to the REST endpoint proposal. I
>> > > replied
>> > > > > with the fuller version on the event-forwarding thread, but
>> wanted to
>> > > add
>> > > > > the shorter version here too.
>> > > > >
>> > > > > I agree that metrics reporting/emitting is the right conceptual
>> > > boundary,
>> > > > > and I also think the event/listener path is a good implementation
>> > > > direction
>> > > > > to explore. The distinction I want to keep clear is default
>> battery
>> > vs
>> > > > > extension implementation.
>> > > > >
>> > > > > For the REST/API side, I would keep the semantics narrow: Polaris
>> > > > resolves
>> > > > > and authorizes the table, accepts the Iceberg scan/commit metrics
>> > > report
>> > > > > into the configured reporting/emitting path, and returns 204. That
>> > 204
>> > > > > should mean Polaris accepted the report into the ingestion path;
>> it
>> > > > should
>> > > > > not imply durable storage.
>> > > > >
>> > > > > Durable storage, event forwarding, external telemetry routing,
>> > > filtering,
>> > > > > and retention should sit behind that reporting/emitting boundary
>> as
>> > > > > implementation-layer behavior. A durable JDBC path can be one
>> named
>> > > > > extension implementation. An event/listener forwarding path can be
>> > > > another
>> > > > > named extension implementation.
>> > > > >
>> > > > > So I am +1 on the event-forwarding idea as a non-default extension
>> > > > > implementation. I just would not make it the default battery or
>> the
>> > > core
>> > > > > API shape. The default battery can stay small and safe, while
>> > > deployments
>> > > > > that need forwarding, dashboards, or durable storage can opt into
>> the
>> > > > > implementation that matches their operational model.
>> > > > >
>> > > > > Thanks,
>> > > > > -ej
>> > > > >
>> > > > > On Wed, Jun 10, 2026 at 11:43 AM Yufei Gu <fl...@gmail.com>
>> > > wrote:
>> > > > >
>> > > > > > Thanks EJ. I agree that the reporting/emitting boundary feels
>> like
>> > > the
>> > > > > > right SPI boundary, but I wonder if we can simplify this even
>> > > further.
>> > > > > >
>> > > > > > Iceberg scan and commit metrics look very similar to events to
>> me.
>> > > They
>> > > > > are
>> > > > > > append only, asynchronously consumed, often forwarded to
>> external
>> > > > > systems,
>> > > > > > and have similar retention and cleanup requirements. More
>> details
>> > can
>> > > > be
>> > > > > > found in this thread [1]. In that case, Polaris may only need to
>> > > accept
>> > > > > the
>> > > > > > report and emit an event through the existing event framework.
>> > > > Filtering,
>> > > > > > async delivery, custom sinks, and retention mechanisms could
>> then
>> > be
>> > > > > shared
>> > > > > > instead of introducing a separate metrics specific extension
>> layer
>> > > and
>> > > > > SPI.
>> > > > > >
>> > > > > > The main requirement I still see is querying metrics through
>> > Polaris.
>> > > > If
>> > > > > we
>> > > > > > want that battery included experience, we could work on a
>> > persistence
>> > > > > > listener implementation that selectively saves metrics and
>> expose
>> > it
>> > > > via
>> > > > > > the IRC event endpoint which seems pretty close in Iceberg
>> > > > > > community. Another option I like more is to allow users to
>> develop
>> > > > their
>> > > > > > own listener, so they can persist scan/commit metrics to any
>> > systems
>> > > > they
>> > > > > > prefer. In general, that feels like an implementation choice on
>> top
>> > > of
>> > > > > the
>> > > > > > event framework rather than something that needs to shape the
>> core
>> > > > > > architecture.
>> > > > > >
>> > > > > > 1.
>> > https://lists.apache.org/thread/x9j8nscvy8hq61tyn01mj8yp6n9of0kp
>> > > > > >
>> > > > > > Thanks,
>> > > > > > Yufei
>> > > > > >
>> > > > > >
>> > > > > > On Thu, Jun 4, 2026 at 5:14 PM EJ Wang <
>> > > ej.wang.dev.interest@gmail.com
>> > > > >
>> > > > > > wrote:
>> > > > > >
>> > > > > > > Good point. I agree the existing PolarisMetricsReporter is
>> > already
>> > > > very
>> > > > > > > close to the right conceptual boundary. I am not proposing a
>> > second
>> > > > > > > parallel reporter concept. The distinction I am trying to
>> make is
>> > > > > > narrower:
>> > > > > > >
>> > > > > > > Current state:
>> > > > > > >
>> > > > > > >    - PolarisMetricsReporter is the existing metrics reporting
>> > hook.
>> > > > > > >    - But it currently lives under runtime/service.
>> > > > > > >    - Its contract is documented around Quarkus/CDI discovery
>> and
>> > > > > > > @Identifier
>> > > > > > >    selection.
>> > > > > > >    - The default implementation is log-only.
>> > > > > > >    - The durable implementation writes through
>> > > PolarisMetricsManager
>> > > > > and
>> > > > > > >    MetricsPersistence.
>> > > > > > >    - MetricsPersistence is currently inherited by
>> > BasePersistence.
>> > > > > > >
>> > > > > > > My proposal:
>> > > > > > >
>> > > > > > >    - Keep the reporting/emitting boundary as the SPI.
>> > > > > > >    - Revise that contract in a framework-agnostic optional
>> > Iceberg
>> > > > > > *metrics
>> > > > > > >    extension API layer*.
>> > > > > > >    - Keep runtime/service responsible for REST ingestion,
>> > > table/authz
>> > > > > > >    validation, and runtime wiring.
>> > > > > > >    - Keep the battery implementation log-only in the metrics
>> > > > extension
>> > > > > > API
>> > > > > > >    layer, *not under runtime/service*.
>> > > > > > >    - Treat durable JDBC metrics as one implementation of the
>> > > > reporting
>> > > > > > SPI.
>> > > > > > >    - Decompose the MetricsPersistence and BasePersistence
>> > coupling
>> > > as
>> > > > > > >    durable implementation detail.
>> > > > > > >
>> > > > > > > So the difference is not “new reporter concept vs old reporter
>> > > > > concept.”
>> > > > > > > but:
>> > > > > > >
>> > > > > > >
>> > > > > > >    - PolarisMetricsReporter today = runtime/service CDI-shaped
>> > > hook.
>> > > > > > >    - Target reporting SPI = same conceptual boundary, but
>> placed
>> > in
>> > > > the
>> > > > > > >    right extension API layer and kept framework-agnostic.
>> > > > > > >
>> > > > > > > For the current metrics PR, I think the minimal architectural
>> > > cleanup
>> > > > > > > before merge is to *put the reporting SPI contract and the
>> > battery
>> > > > > > default
>> > > > > > > in the right place*. The durable JDBC implementation,
>> > event-backed
>> > > > > async
>> > > > > > > implementation, external queue support, and deeper persistence
>> > > > cleanup
>> > > > > > can
>> > > > > > > continue as follow-ups, as long as we do not lock the SPI
>> > boundary
>> > > to
>> > > > > > > BasePersistence or to runtime/service wiring.
>> > > > > > >
>> > > > > > > -ej
>> > > > > > >
>> > > > > > > On Wed, Jun 3, 2026 at 3:00 PM Dmitri Bourlatchkov <
>> > > dimas@apache.org
>> > > > >
>> > > > > > > wrote:
>> > > > > > >
>> > > > > > > > Hi EJ,
>> > > > > > > >
>> > > > > > > > Thanks for the recap / summary!
>> > > > > > > >
>> > > > > > > > > Do folks agree that the stable SPI boundary should be
>> > > > > > > > metrics reporting/emitting, not metrics persistence?
>> > > > > > > >
>> > > > > > > > SGTM.
>> > > > > > > >
>> > > > > > > > > Does an optional Iceberg metrics extension API layer sound
>> > like
>> > > > the
>> > > > > > > right
>> > > > > > > > home for this SPI?
>> > > > > > > >
>> > > > > > > > How is that different from current PolarisMetricsReporter?
>> > > > > > > >
>> > > > > > > > > Should the current durable metrics work be reframed as a
>> > > durable
>> > > > > > > > JDBC reference implementation of that SPI?
>> > > > > > > >
>> > > > > > > > SGTM.
>> > > > > > > >
>> > > > > > > > > What is the smallest PR sequence to get there without
>> > blocking
>> > > > > > > > the current metrics work unnecessarily?
>> > > > > > > >
>> > > > > > > > Let's get https://github.com/apache/polaris/pull/4397
>> merged
>> > > > first.
>> > > > > > > >
>> > > > > > > > Then, I'd propose isolating the Metrics schema from the
>> > MetaStore
>> > > > > > schema.
>> > > > > > > > This will probably have some ripple effect into the
>> bootstrap
>> > > > > > workflows,
>> > > > > > > so
>> > > > > > > > it's not a trivial change.
>> > > > > > > >
>> > > > > > > > Then, let's reassess.
>> > > > > > > >
>> > > > > > > > Cheers,
>> > > > > > > > Dmitri.
>> > > > > > > >
>> > > > > > > > On Wed, Jun 3, 2026 at 5:40 PM EJ Wang <
>> > > > > ej.wang.dev.interest@gmail.com
>> > > > > > >
>> > > > > > > > wrote:
>> > > > > > > >
>> > > > > > > > > Hi folks,
>> > > > > > > > >
>> > > > > > > > > Thanks again for the discussion today. I updated the sync
>> doc
>> > > > with
>> > > > > > > notes
>> > > > > > > > > from the third metrics architecture sync:
>> > > > > > > > >
>> > > > > > > > >
>> > > > > > > > >
>> > > > > > > >
>> > > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>> https://docs.google.com/document/d/100h7c4damrUzVuquYbBHM0EvA4LSWuW2IT2dN_7nYVA/edit?tab=t.k96s2xyqr5u1
>> > > > > > > > >
>> > > > > > > > > A few highlights from today’s discussion:
>> > > > > > > > >
>> > > > > > > > >    - We clarified that Iceberg metrics reporting can be
>> > > > interpreted
>> > > > > > > > either
>> > > > > > > > >    as sync or async handling from Polaris’s perspective,
>> so
>> > > > Polaris
>> > > > > > > > should
>> > > > > > > > >    stay flexible as a platform instead of baking one
>> handling
>> > > > model
>> > > > > > > into
>> > > > > > > > > the
>> > > > > > > > >    REST API behavior.
>> > > > > > > > >    - We aligned that the built-in (battery) behavior for
>> > > metrics
>> > > > > > > emitting
>> > > > > > > > >    can stay simple: no-op/log-only is enough as the
>> default.
>> > > > > > > > >    - Durable metrics persistence should be treated as an
>> > > > > > implementation
>> > > > > > > > of
>> > > > > > > > >    the metrics reporting path, not as the core SPI
>> boundary.
>> > > > > > > > >    - The existing durable metrics work can be reviewed as
>> a
>> > > > > reference
>> > > > > > > > >    implementation of the reporting SPI, with
>> > > persistence-related
>> > > > > > logic
>> > > > > > > > kept
>> > > > > > > > >    self-contained under a metrics durable implementation
>> > module
>> > > > > > rather
>> > > > > > > > than
>> > > > > > > > >    scattered through core entity persistence.
>> > > > > > > > >    - Dashboard/insights remains a real use case, but we
>> > agreed
>> > > to
>> > > > > > keep
>> > > > > > > it
>> > > > > > > > >    separate from the core metrics intake discussion for
>> now.
>> > > > > > > > >
>> > > > > > > > > I also did a quick source check after the meeting to make
>> > sure
>> > > we
>> > > > > are
>> > > > > > > > > describing the current state accurately.
>> > > > > > > > >
>> > > > > > > > > Current state:
>> > > > > > > > >
>> > > > > > > > >    - Polaris already has a metrics reporting hook:
>> > > > > > > > PolarisMetricsReporter.
>> > > > > > > > >    - The default implementation is DefaultMetricsReporter,
>> > > > selected
>> > > > > > by
>> > > > > > > > >    polaris.iceberg-metrics.reporting.type=default. It is
>> > > > log-only,
>> > > > > > > > >    effectively quiet unless metrics logging is enabled.
>> > > > > > > > >    - There is also a PersistingMetricsReporter, selected
>> by
>> > > > > > > > >    polaris.iceberg-metrics.reporting.type=persisting,
>> which
>> > > > > converts
>> > > > > > > > >    Iceberg scan/commit reports into Polaris metrics
>> records
>> > and
>> > > > > > writes
>> > > > > > > > > through
>> > > > > > > > >    PolarisMetricsManager -> MetricsPersistence.
>> > > > > > > > >    - MetricsPersistence currently exists in the
>> persistence
>> > > layer
>> > > > > and
>> > > > > > > is
>> > > > > > > > >    inherited by BasePersistence.
>> > > > > > > > >
>> > > > > > > > > My read from the discussion is that the target boundary
>> > should
>> > > > not
>> > > > > be
>> > > > > > > > > MetricsPersistence as inherited by BasePersistence. That
>> path
>> > > > > should
>> > > > > > be
>> > > > > > > > > decomposed as durable implementation detail (taken care
>> of by
>> > > > > > > > > https://github.com/apache/polaris/pull/4397). The stable
>> > > > extension
>> > > > > > > point
>> > > > > > > > > should instead be the metrics reporting/emitting boundary.
>> > > > > > > > >
>> > > > > > > > > Concretely, I think the next proposal should be shaped
>> like
>> > > this:
>> > > > > > > > >
>> > > > > > > > >    1.
>> > > > > > > > >
>> > > > > > > > >    Define the proper metrics reporting SPI at an optional
>> > > Iceberg
>> > > > > > > metrics
>> > > > > > > > >    extension API layer.
>> > > > > > > > >
>> > > > > > > > >    Example direction:
>> > > > > > > > >
>> > > > > > > > >    public interface IcebergMetricsReporter {
>> > > > > > > > >      void reportMetric(IcebergMetricsReportContext
>> context,
>> > > > > > > > > MetricsReport report);
>> > > > > > > > >    }
>> > > > > > > > >
>> > > > > > > > >    The context would carry the small Polaris-resolved
>> > envelope:
>> > > > > > > > >    catalog/table identity, report type, received
>> timestamp,
>> > and
>> > > > > > > > > request/trace
>> > > > > > > > >    context if available. The raw Iceberg MetricsReport
>> > remains
>> > > > the
>> > > > > > > > payload.
>> > > > > > > > >    2.
>> > > > > > > > >
>> > > > > > > > >    Keep runtime/service as the REST ingestion and wiring
>> > layer.
>> > > > > > > > >
>> > > > > > > > >    The REST handler still resolves the table and performs
>> > authz
>> > > > > > before
>> > > > > > > > >    accepting the report. After that, it calls the selected
>> > > > > reporting
>> > > > > > > > >    implementation.
>> > > > > > > > >    3.
>> > > > > > > > >
>> > > > > > > > >    Keep the battery default no-op/log-only.
>> > > > > > > > >
>> > > > > > > > >    This preserves an out-of-box safe default and avoids
>> > > > requiring a
>> > > > > > > > durable
>> > > > > > > > >    metrics store for every Polaris deployment.
>> > > > > > > > >    4.
>> > > > > > > > >
>> > > > > > > > >    Move durable JDBC metrics into a self-contained
>> > > implementation
>> > > > > of
>> > > > > > > the
>> > > > > > > > >    reporting SPI.
>> > > > > > > > >
>> > > > > > > > >    That implementation can own its schema, bootstrap,
>> > > retention,
>> > > > > and
>> > > > > > > read
>> > > > > > > > >    API support. It should not define the core reporting
>> SPI
>> > > > > boundary,
>> > > > > > > and
>> > > > > > > > > it
>> > > > > > > > >    should not require metrics persistence to remain
>> inherited
>> > > > from
>> > > > > > > > >    BasePersistence.
>> > > > > > > > >    5.
>> > > > > > > > >
>> > > > > > > > >    Treat async/event-backed handling as another
>> > implementation
>> > > of
>> > > > > the
>> > > > > > > > same
>> > > > > > > > >    reporting SPI.
>> > > > > > > > >
>> > > > > > > > >    For example, an event-backed reporter could enqueue a
>> > > metrics
>> > > > > > event
>> > > > > > > > and
>> > > > > > > > >    let listeners handle durable storage or other sinks.
>> If we
>> > > > later
>> > > > > > > need
>> > > > > > > > a
>> > > > > > > > >    replaceable queue engine, that seems like a shared
>> > > > event/metrics
>> > > > > > > > > substrate
>> > > > > > > > >    topic rather than a metrics-only requirement.
>> > > > > > > > >
>> > > > > > > > > This framing lets us keep the REST metrics endpoint
>> simple,
>> > > > > preserve
>> > > > > > > the
>> > > > > > > > > current default behavior, support durable metrics users,
>> and
>> > > > still
>> > > > > > > leave
>> > > > > > > > > room for async/event-backed or external-queue-based
>> > > > > implementations.
>> > > > > > > > >
>> > > > > > > > > I think the main follow-up questions are:
>> > > > > > > > >
>> > > > > > > > >    - Do folks agree that the stable SPI boundary should be
>> > > > metrics
>> > > > > > > > >    reporting/emitting, not metrics persistence?
>> > > > > > > > >    - Does an optional Iceberg metrics extension API layer
>> > sound
>> > > > > like
>> > > > > > > the
>> > > > > > > > >    right home for this SPI?
>> > > > > > > > >    - Should the current durable metrics work be reframed
>> as a
>> > > > > durable
>> > > > > > > > JDBC
>> > > > > > > > >    reference implementation of that SPI?
>> > > > > > > > >    - What is the smallest PR sequence to get there without
>> > > > blocking
>> > > > > > the
>> > > > > > > > >    current metrics work unnecessarily?
>> > > > > > > > >
>> > > > > > > > > Thanks,
>> > > > > > > > > -ej
>> > > > > > > > >
>> > > > > > > > > On Fri, May 15, 2026 at 5:16 PM Dmitri Bourlatchkov <
>> > > > > > dimas@apache.org>
>> > > > > > > > > wrote:
>> > > > > > > > >
>> > > > > > > > > > Hi JB,
>> > > > > > > > > >
>> > > > > > > > > > Could you set up another meeting, please? Same time on
>> > > > Wednesday
>> > > > > as
>> > > > > > > > last
>> > > > > > > > > > time... I hope it works for everyone.
>> > > > > > > > > >
>> > > > > > > > > > Cheers,
>> > > > > > > > > > Dmitri.
>> > > > > > > > > >
>> > > > > > > > > > On Fri, May 15, 2026 at 8:06 PM Yufei Gu <
>> > > flyrain000@gmail.com
>> > > > >
>> > > > > > > wrote:
>> > > > > > > > > >
>> > > > > > > > > > > +1 on another sync call next week.
>> > > > > > > > > > >
>> > > > > > > > > > > Yufei
>> > > > > > > > > > >
>> > > > > > > > > > >
>> > > > > > > > > > > On Fri, May 15, 2026 at 4:52 PM Dmitri Bourlatchkov <
>> > > > > > > > dimas@apache.org>
>> > > > > > > > > > > wrote:
>> > > > > > > > > > >
>> > > > > > > > > > > > Hi All,
>> > > > > > > > > > > >
>> > > > > > > > > > > > WDYT about another sync call next week?
>> > > > > > > > > > > >
>> > > > > > > > > > > > Thanks,
>> > > > > > > > > > > > Dmitri.
>> > > > > > > > > > > >
>> > > > > > > > > > > > On Wed, May 6, 2026 at 5:29 PM Dmitri Bourlatchkov <
>> > > > > > > > dimas@apache.org
>> > > > > > > > > >
>> > > > > > > > > > > > wrote:
>> > > > > > > > > > > >
>> > > > > > > > > > > > > Hi EJ,
>> > > > > > > > > > > > >
>> > > > > > > > > > > > > Thanks for the summary! It covers what we
>> discussed
>> > in
>> > > > the
>> > > > > > > > meeting
>> > > > > > > > > > very
>> > > > > > > > > > > > > well, IMHO.
>> > > > > > > > > > > > >
>> > > > > > > > > > > > > Looking forward to concrete PRs :)
>> > > > > > > > > > > > >
>> > > > > > > > > > > > > Cheers,
>> > > > > > > > > > > > > Dmitri.
>> > > > > > > > > > > > >
>> > > > > > > > > > > > > On Wed, May 6, 2026 at 5:08 PM EJ Wang <
>> > > > > > > > > > ej.wang.dev.interest@gmail.com
>> > > > > > > > > > > >
>> > > > > > > > > > > > > wrote:
>> > > > > > > > > > > > >
>> > > > > > > > > > > > >> Hi folks,
>> > > > > > > > > > > > >>
>> > > > > > > > > > > > >> We had a community sync earlier, thanks JB for
>> > > > scheduling
>> > > > > > it.
>> > > > > > > > > Notes
>> > > > > > > > > > > from
>> > > > > > > > > > > > >> the first metrics architecture sync (May 6,
>> 10-11am
>> > > PT).
>> > > > > > > > > Discussion
>> > > > > > > > > > > doc
>> > > > > > > > > > > > >> with per-section status:
>> > > > > > > > > > > > >>
>> > > > > > > > > > > > >>
>> > > > > > > > > > > >
>> > > > > > > > > > >
>> > > > > > > > > >
>> > > > > > > > >
>> > > > > > > >
>> > > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>> https://docs.google.com/document/d/100h7c4damrUzVuquYbBHM0EvA4LSWuW2IT2dN_7nYVA/edit?tab=t.0
>> > > > > > > > > > > > >>
>> > > > > > > > > > > > >> *The meeting covered both topics from the doc.
>> > > > > > Direction-level
>> > > > > > > > > > > alignment
>> > > > > > > > > > > > >> was reached on the headline pieces; details
>> remain
>> > for
>> > > > PR
>> > > > > > > review
>> > > > > > > > > or
>> > > > > > > > > > > > >> follow-up sessions.*
>> > > > > > > > > > > > >>
>> > > > > > > > > > > > >> *Topic 1 — Persistence schema redesign*
>> > > > > > > > > > > > >> Idea-level alignment on consolidating per-type
>> > tables
>> > > > > > > > > > > > >> (scan_metrics_report,
>> > > > > > > > > > > > >> commit_metrics_report) into a single
>> metrics_report
>> > > > table.
>> > > > > > The
>> > > > > > > > > > > > motivating
>> > > > > > > > > > > > >> cost is the surface area added by every new
>> metric
>> > > type
>> > > > > > today:
>> > > > > > > > new
>> > > > > > > > > > > > table,
>> > > > > > > > > > > > >> SPI method, record class, model, converter,
>> schema
>> > > > > > migration.
>> > > > > > > > > > > > >>
>> > > > > > > > > > > > >> Most schema details are deferred to the schema
>> PR. A
>> > > few
>> > > > > > > > specific
>> > > > > > > > > > > points
>> > > > > > > > > > > > >> came up:
>> > > > > > > > > > > > >> •   metric_schema_version: Yufei prefers dropping
>> > it,
>> > > > > since
>> > > > > > > > there
>> > > > > > > > > is
>> > > > > > > > > > > no
>> > > > > > > > > > > > >> spec-level concept of metrics versioning today
>> and
>> > it
>> > > is
>> > > > > > hard
>> > > > > > > to
>> > > > > > > > > > > define
>> > > > > > > > > > > > >> unilaterally. Robert prefers keeping it, given
>> IRC
>> > v2
>> > > is
>> > > > > > > coming
>> > > > > > > > > and
>> > > > > > > > > > > the
>> > > > > > > > > > > > >> schema should be considered against its likely
>> > shape;
>> > > > > Robert
>> > > > > > > > also
>> > > > > > > > > > > raised
>> > > > > > > > > > > > >> how to differentiate various payload formats if
>> any.
>> > > > EJ's
>> > > > > > read
>> > > > > > > > is
>> > > > > > > > > > that
>> > > > > > > > > > > > >> this
>> > > > > > > > > > > > >> is a two-way-door decision. We can start without
>> the
>> > > > > field,
>> > > > > > > and
>> > > > > > > > if
>> > > > > > > > > > IRC
>> > > > > > > > > > > > v2
>> > > > > > > > > > > > >> changes the shape we would likely roll a
>> > corresponding
>> > > > new
>> > > > > > > > schema
>> > > > > > > > > > > > anyway,
>> > > > > > > > > > > > >> which is not particularly costly.
>> > > > > > > > > > > > >> •   Payload format: Robert pointed out that
>> future
>> > > > formats
>> > > > > > > > beyond
>> > > > > > > > > > JSON
>> > > > > > > > > > > > may
>> > > > > > > > > > > > >> be worth supporting. The exact shape is deferred
>> to
>> > > the
>> > > > > > schema
>> > > > > > > > > > > > discussion.
>> > > > > > > > > > > > >> •   Partition strategy: Anand suggested monthly
>> > > > > partitioning
>> > > > > > > > based
>> > > > > > > > > > on
>> > > > > > > > > > > > his
>> > > > > > > > > > > > >> experience as potentially helpful at scale.
>> > > > > > > > > > > > >>
>> > > > > > > > > > > > >> *Topic 2 — Where metrics ingestion and storage
>> > belong*
>> > > > > > > > > > > > >> Idea-level alignment that metrics should be a
>> > > separated
>> > > > > SPI
>> > > > > > > from
>> > > > > > > > > the
>> > > > > > > > > > > > >> entity
>> > > > > > > > > > > > >> persistence stack. Two reasons surfaced: (a)
>> > workloads
>> > > > and
>> > > > > > > > > > capability
>> > > > > > > > > > > > >> requirements diverge enough that coupling them
>> > creates
>> > > > > > > > artificial
>> > > > > > > > > > > > >> constraints, and (b) admin experience improves
>> when
>> > > > > metrics
>> > > > > > > has
>> > > > > > > > > its
>> > > > > > > > > > > own
>> > > > > > > > > > > > >> bootstrap, retention, and lifecycle. Dmitri noted
>> > > > Polaris
>> > > > > > > being
>> > > > > > > > a
>> > > > > > > > > > > > platform
>> > > > > > > > > > > > >> should have the flexibility to support different
>> > > > > persistence
>> > > > > > > > > > backends
>> > > > > > > > > > > > per
>> > > > > > > > > > > > >> concern, and pointed to a concrete next step of
>> > > > separating
>> > > > > > the
>> > > > > > > > > JDBC
>> > > > > > > > > > > > >> bootstrap for metrics from the metastore
>> bootstrap.
>> > > > Robert
>> > > > > > > > > proposed
>> > > > > > > > > > an
>> > > > > > > > > > > > >> additional UX extension: detect an unbootstrapped
>> > > > metrics
>> > > > > > > store
>> > > > > > > > on
>> > > > > > > > > > > first
>> > > > > > > > > > > > >> use and auto-bootstrap rather than requiring an
>> > > explicit
>> > > > > > > manual
>> > > > > > > > > > > > bootstrap
>> > > > > > > > > > > > >> step.
>> > > > > > > > > > > > >> The meeting also confirmed that Polaris metrics
>> can
>> > > > start
>> > > > > > > small
>> > > > > > > > > and
>> > > > > > > > > > > stay
>> > > > > > > > > > > > >> Iceberg-focused. Naming and persistence schema
>> can
>> > > lean
>> > > > > > > > > > > > Iceberg-specific.
>> > > > > > > > > > > > >> If a future expansion to generic-table metrics or
>> > > > > > operational
>> > > > > > > > > > metrics
>> > > > > > > > > > > > >> arrives, an abstraction layer can be built on
>> top of
>> > > the
>> > > > > > > Iceberg
>> > > > > > > > > > > metrics
>> > > > > > > > > > > > >> reporter at that point. Robert remains on the
>> fence
>> > > and
>> > > > > > would
>> > > > > > > > > prefer
>> > > > > > > > > > > > >> something more generic but did not block the
>> > > direction;
>> > > > > > > Dmitri's
>> > > > > > > > > > read
>> > > > > > > > > > > > was
>> > > > > > > > > > > > >> that the proposed framework already has enough
>> > > > flexibility
>> > > > > > to
>> > > > > > > > > absorb
>> > > > > > > > > > > > >> future
>> > > > > > > > > > > > >> expansion.
>> > > > > > > > > > > > >>
>> > > > > > > > > > > > >> The Trade-offs and Proposed structure sections in
>> > the
>> > > > doc
>> > > > > > were
>> > > > > > > > not
>> > > > > > > > > > > > >> reviewed
>> > > > > > > > > > > > >> in detail. They remain open for either the next
>> sync
>> > > or
>> > > > PR
>> > > > > > > > review.
>> > > > > > > > > > > > >>
>> > > > > > > > > > > > >> *Cross-cutting alignment — battery-included plus
>> > > > > pluggable*
>> > > > > > > > > > > > >> A common philosophy emerged from the discussion.
>> EJ
>> > > > > > summarized
>> > > > > > > > it
>> > > > > > > > > > as:
>> > > > > > > > > > > > >> Polaris should provide a battery-included UX for
>> > > > beginners
>> > > > > > and
>> > > > > > > > the
>> > > > > > > > > > > > >> flexibility for advanced users to swap the
>> included
>> > > > > battery
>> > > > > > > for
>> > > > > > > > > > > > something
>> > > > > > > > > > > > >> more powerful or tailored to their use case. The
>> SPI
>> > > > > design
>> > > > > > > > needs
>> > > > > > > > > to
>> > > > > > > > > > > > >> enable
>> > > > > > > > > > > > >> both.
>> > > > > > > > > > > > >>
>> > > > > > > > > > > > >> The inputs that shaped this framing:
>> > > > > > > > > > > > >> •   Anand described how his team uses the current
>> > > > metrics
>> > > > > > > > > > persistence
>> > > > > > > > > > > > >> (three metrics consumers in v1.4).
>> > > > > > > > > > > > >> •   Yufei raised Grafana and dashboard
>> integrations
>> > > as a
>> > > > > > > > > destination
>> > > > > > > > > > > use
>> > > > > > > > > > > > >> case beyond the default.
>> > > > > > > > > > > > >> •   Robert called out that the current design is
>> > more
>> > > > > > > > > JDBC-focused.
>> > > > > > > > > > > > >>
>> > > > > > > > > > > > >> Two concrete instances:
>> > > > > > > > > > > > >> •   Async metrics intake: Yufei's initial
>> position
>> > was
>> > > > > that
>> > > > > > > > async
>> > > > > > > > > > > should
>> > > > > > > > > > > > >> largely live on the producer side and there is
>> not
>> > > much
>> > > > > > > Polaris
>> > > > > > > > > can
>> > > > > > > > > > > do.
>> > > > > > > > > > > > >> Robert suggested a Polaris-side default is doable
>> > via
>> > > > > > Vert.x.
>> > > > > > > > > Dmitri
>> > > > > > > > > > > > >> agreed
>> > > > > > > > > > > > >> the direction is worth exploring. The meeting
>> > > converged
>> > > > > on a
>> > > > > > > > > > > > >> battery-included default (likely Vert.x-backed)
>> with
>> > > an
>> > > > > SPI
>> > > > > > > > shape
>> > > > > > > > > > that
>> > > > > > > > > > > > >> lets
>> > > > > > > > > > > > >> power users route to a more scalable backend
>> > > (k8s-hosted
>> > > > > > > queue,
>> > > > > > > > > AWS
>> > > > > > > > > > > SQS,
>> > > > > > > > > > > > >> etc.).
>> > > > > > > > > > > > >> •   Pluggable destinations: combining Yufei's
>> > > dashboard
>> > > > > use
>> > > > > > > case
>> > > > > > > > > > with
>> > > > > > > > > > > > >> Robert's JDBC-focused call-out, the meeting
>> agreed
>> > the
>> > > > SPI
>> > > > > > > > should
>> > > > > > > > > be
>> > > > > > > > > > > > >> structured for multiple sinks so integrations
>> become
>> > > > impl
>> > > > > > > > choices
>> > > > > > > > > > > rather
>> > > > > > > > > > > > >> than architectural changes.
>> > > > > > > > > > > > >>
>> > > > > > > > > > > > >> The battery-included default is most likely to
>> use
>> > the
>> > > > > > > existing
>> > > > > > > > > > > > >> JDBC-backed
>> > > > > > > > > > > > >> approach.
>> > > > > > > > > > > > >>
>> > > > > > > > > > > > >> *Direction (idea-level alignment)*
>> > > > > > > > > > > > >> •   Single metrics_report table consolidating
>> > per-type
>> > > > > > > metrics,
>> > > > > > > > > > > > replacing
>> > > > > > > > > > > > >> scan_metrics_report and commit_metrics_report
>> > > > > > > > > > > > >> •   Iceberg-focused naming and schema for now,
>> > revisit
>> > > > if
>> > > > > > > > > > > generic-table
>> > > > > > > > > > > > or
>> > > > > > > > > > > > >> operational metrics arrive
>> > > > > > > > > > > > >> •   Metrics persistence as a separated SPI, not
>> on
>> > > > > > > > BasePersistence
>> > > > > > > > > > > > >> •   Bootstrap path separated for metrics,
>> > independent
>> > > of
>> > > > > > > > metastore
>> > > > > > > > > > > > >> bootstrap
>> > > > > > > > > > > > >> •   "Battery-included plus pluggable" as the SPI
>> > > design
>> > > > > > > > philosophy
>> > > > > > > > > > > > >>
>> > > > > > > > > > > > >> *Open items*
>> > > > > > > > > > > > >> •   Schema details: metric_schema_version,
>> payload
>> > > > format,
>> > > > > > IRC
>> > > > > > > > v2
>> > > > > > > > > > > > >> forward-compat shape
>> > > > > > > > > > > > >> •   SPI design details — full review either in
>> the
>> > > next
>> > > > > sync
>> > > > > > > or
>> > > > > > > > in
>> > > > > > > > > > the
>> > > > > > > > > > > > >> corresponding PR
>> > > > > > > > > > > > >> •   Schema refactor PR ownership
>> > > > > > > > > > > > >>
>> > > > > > > > > > > > >> *Action items*
>> > > > > > > > > > > > >> •   EJ to take a first stab at the SPI design and
>> > > > > > potentially
>> > > > > > > > > > partner
>> > > > > > > > > > > > with
>> > > > > > > > > > > > >> Anand to incorporate the lessons learned from the
>> > > > existing
>> > > > > > > > > reporter
>> > > > > > > > > > > and
>> > > > > > > > > > > > >> persistence work.
>> > > > > > > > > > > > >> •   Schema refactor PR ownership is not yet
>> decided.
>> > > If
>> > > > > > anyone
>> > > > > > > > is
>> > > > > > > > > > > > >> interested in driving it, reply on this thread.
>> > > > > > > > > > > > >> •   JB to schedule the next sync, tentatively in
>> two
>> > > > > weeks.
>> > > > > > > > > > > > >>
>> > > > > > > > > > > > >> -ej
>> > > > > > > > > > > > >>
>> > > > > > > > > > > > >> On Mon, Apr 27, 2026 at 3:07 PM EJ Wang <
>> > > > > > > > > > > ej.wang.dev.interest@gmail.com
>> > > > > > > > > > > > >
>> > > > > > > > > > > > >> wrote:
>> > > > > > > > > > > > >>
>> > > > > > > > > > > > >> > Thanks Yufei for the +1.
>> > > > > > > > > > > > >> >
>> > > > > > > > > > > > >> > JB, could you help add a biweekly metrics
>> > > architecture
>> > > > > > sync
>> > > > > > > to
>> > > > > > > > > the
>> > > > > > > > > > > > >> Polaris
>> > > > > > > > > > > > >> > community calendar? I'm thinking Thursdays at
>> > 9-10am
>> > > > PT,
>> > > > > > on
>> > > > > > > > the
>> > > > > > > > > > > > >> off-weeks
>> > > > > > > > > > > > >> > from the community meeting (starting May 7), 60
>> > > > minutes.
>> > > > > > > > > > > > >> >
>> > > > > > > > > > > > >> > Here's a rough agenda to work through over the
>> > first
>> > > > few
>> > > > > > > > > sessions,
>> > > > > > > > > > > > >> grouped
>> > > > > > > > > > > > >> > by priority:
>> > > > > > > > > > > > >> >
>> > > > > > > > > > > > >> > *First: foundational direction*
>> > > > > > > > > > > > >> >
>> > > > > > > > > > > > >> > 1.  MetricsPersistence: public SPI or internal
>> > > > > > > implementation
>> > > > > > > > > > > detail?
>> > > > > > > > > > > > >> >    •   Marked @Beta, javadoc calls it a
>> "Service
>> > > > > Provider
>> > > > > > > > > > > Interface",
>> > > > > > > > > > > > >> but
>> > > > > > > > > > > > >> > only one consumer (JdbcBasePersistenceImpl),
>> lives
>> > > on
>> > > > > > > > > > > BasePersistence.
>> > > > > > > > > > > > >> If
>> > > > > > > > > > > > >> > demoted to a private helper inside a persisting
>> > > > reporter
>> > > > > > > impl,
>> > > > > > > > > > most
>> > > > > > > > > > > > >> > downstream design decisions become
>> implementation
>> > > > > details
>> > > > > > > > rather
>> > > > > > > > > > > than
>> > > > > > > > > > > > >> > contract questions.
>> > > > > > > > > > > > >> >
>> > > > > > > > > > > > >> > 2.  Persistence schema redesign
>> > > > > > > > > > > > >> >    •   Current two-table layout
>> > > (scan_metrics_report,
>> > > > > > > > > > > > >> > commit_metrics_report) with ~25 flattened
>> columns
>> > > > each.
>> > > > > > > Every
>> > > > > > > > > new
>> > > > > > > > > > > > metric
>> > > > > > > > > > > > >> > type requires a new table, SPI method, record
>> > class,
>> > > > > > model,
>> > > > > > > > > > > converter,
>> > > > > > > > > > > > >> and
>> > > > > > > > > > > > >> > schema migration. Direction to explore: single
>> > table
>> > > > > with
>> > > > > > > > > > > metric_type
>> > > > > > > > > > > > >> enum,
>> > > > > > > > > > > > >> > schema_version, and JSON payload column.
>> > > > > > > > > > > > >> >
>> > > > > > > > > > > > >> > *Second: design details once direction is set*
>> > > > > > > > > > > > >> >
>> > > > > > > > > > > > >> > 3.  Partition key strategy
>> > > > > > > > > > > > >> >    •   Single-table design means scan metrics
>> at
>> > > scale
>> > > > > > will
>> > > > > > > > have
>> > > > > > > > > > > high
>> > > > > > > > > > > > >> > write concurrency per table. Schema needs to
>> > expose
>> > > > > enough
>> > > > > > > > > > structure
>> > > > > > > > > > > > for
>> > > > > > > > > > > > >> > backends to shard by entity or time range.
>> > > > > > > > > > > > >> >
>> > > > > > > > > > > > >> > 4.  Read/write path consistency
>> > > > > > > > > > > > >> >    •   Writes go through PolarisMetricsManager
>> on
>> > > > > > > > > > MetaStoreManager.
>> > > > > > > > > > > > >> Reads
>> > > > > > > > > > > > >> > bypass MetaStoreManager and go straight to
>> > > > > > BasePersistence,
>> > > > > > > > > > > excluding
>> > > > > > > > > > > > >> > non-JDBC backends from the read API.
>> > > > > > > > > > > > >> >
>> > > > > > > > > > > > >> > *Third: cleanup and alignment*
>> > > > > > > > > > > > >> >
>> > > > > > > > > > > > >> > 5.  PolarisMetricsReporter naming
>> > > > > > > > > > > > >> >    •   Only handles IRC
>> (ScanReport/CommitReport),
>> > > > > doesn't
>> > > > > > > > cover
>> > > > > > > > > > > > generic
>> > > > > > > > > > > > >> > tables or operational metrics. Name is broader
>> > than
>> > > > > scope.
>> > > > > > > > > > > > >> >
>> > > > > > > > > > > > >> > 6.  PolarisMetricsManager facade passthrough
>> > > > > > > > > > > > >> >    •   Entire default method is
>> > > > > > > > > > > > >> callCtx.getMetaStore().writeScanReport().
>> > > > > > > > > > > > >> > Zero logic, passes Level 1 straight through to
>> > Level
>> > > > 3.
>> > > > > > Same
>> > > > > > > > > > > > >> anti-pattern
>> > > > > > > > > > > > >> > as PolarisEventManager.
>> > > > > > > > > > > > >> >
>> > > > > > > > > > > > >> > 7.  Iceberg community alignment
>> > > > > > > > > > > > >> >    •   Payload-type extension needs discussion
>> on
>> > > > > > > dev@iceberg.
>> > > > > > > > > > > > >> obelix74's
>> > > > > > > > > > > > >> > Feb thread got zero replies. Needs a committer
>> > > voice.
>> > > > > > > > > > > > >> >
>> > > > > > > > > > > > >> > Lets confirm prioritization in the first
>> session.
>> > > > > > > > > > > > >> >
>> > > > > > > > > > > > >> > -ej
>> > > > > > > > > > > > >> >
>> > > > > > > > > > > > >> > On Tue, Apr 21, 2026 at 3:18 PM Yufei Gu <
>> > > > > > > > flyrain000@gmail.com>
>> > > > > > > > > > > > wrote:
>> > > > > > > > > > > > >> >
>> > > > > > > > > > > > >> >> Thanks everyone for continuing to drive this
>> > > > forward. I
>> > > > > > > agree
>> > > > > > > > > > that
>> > > > > > > > > > > > the
>> > > > > > > > > > > > >> >> problem is getting complex enough that a more
>> > > > > structured
>> > > > > > > > > > discussion
>> > > > > > > > > > > > >> would
>> > > > > > > > > > > > >> >> help.
>> > > > > > > > > > > > >> >>
>> > > > > > > > > > > > >> >> +1 on setting up a biweekly sync for the
>> metrics
>> > > > > > > > architecture.
>> > > > > > > > > > I’m
>> > > > > > > > > > > > >> happy
>> > > > > > > > > > > > >> >> to
>> > > > > > > > > > > > >> >> join.
>> > > > > > > > > > > > >> >>
>> > > > > > > > > > > > >> >> Yufei
>> > > > > > > > > > > > >> >>
>> > > > > > > > > > > > >> >>
>> > > > > > > > > > > > >> >> On Tue, Apr 21, 2026 at 2:34 PM EJ Wang <
>> > > > > > > > > > > > >> ej.wang.dev.interest@gmail.com>
>> > > > > > > > > > > > >> >> wrote:
>> > > > > > > > > > > > >> >>
>> > > > > > > > > > > > >> >> > Also, I've been looking more closely at the
>> > > > > > *persistence
>> > > > > > > > > schema
>> > > > > > > > > > > in
>> > > > > > > > > > > > >> the
>> > > > > > > > > > > > >> >> > current metrics work*, and I think there's a
>> > > > > structural
>> > > > > > > > > > rigidity
>> > > > > > > > > > > > >> problem
>> > > > > > > > > > > > >> >> > worth raising before the shape gets locked
>> in.
>> > > > > > > > > > > > >> >> >
>> > > > > > > > > > > > >> >> > Right now we have two separate tables
>> > > > > > > (scan_metrics_report
>> > > > > > > > > and
>> > > > > > > > > > > > >> >> > commit_metrics_report), each with ~25
>> flattened
>> > > > > columns
>> > > > > > > > that
>> > > > > > > > > > > > directly
>> > > > > > > > > > > > >> >> > mirror the Iceberg report fields. The SPI
>> > follows
>> > > > the
>> > > > > > > same
>> > > > > > > > > > split:
>> > > > > > > > > > > > >> >> > writeScanReport and writeCommitReport as
>> > separate
>> > > > > > > methods,
>> > > > > > > > > with
>> > > > > > > > > > > > >> per-type
>> > > > > > > > > > > > >> >> > record classes, converters, and model
>> objects.
>> > > *The
>> > > > > > > > practical
>> > > > > > > > > > > cost:
>> > > > > > > > > > > > >> >> > adding a new metric type (operational
>> metrics,
>> > > for
>> > > > > > > example)
>> > > > > > > > > > > > requires
>> > > > > > > > > > > > >> a
>> > > > > > > > > > > > >> >> new
>> > > > > > > > > > > > >> >> > table, a new SPI method, a new record
>> class, a
>> > > new
>> > > > > > model
>> > > > > > > > > > class, a
>> > > > > > > > > > > > new
>> > > > > > > > > > > > >> >> > converter branch, and a schema migration*.
>> > > That's a
>> > > > > lot
>> > > > > > > of
>> > > > > > > > > > > surface
>> > > > > > > > > > > > >> area
>> > > > > > > > > > > > >> >> > for what should be "one more kind of
>> metric."
>> > > > > > > > > > > > >> >> >
>> > > > > > > > > > > > >> >> > *My bias* would be toward a single metrics
>> > table
>> > > > with
>> > > > > > *a
>> > > > > > > > > typed
>> > > > > > > > > > > JSON
>> > > > > > > > > > > > >> >> > payload*. Something like: metric_type
>> (enum),
>> > > > > > entity_id,
>> > > > > > > > > > > > >> >> > table_identifier, snapshot_id (nullable),
>> > > > > received_ts,
>> > > > > > > > > > > > >> schema_version,
>> > > > > > > > > > > > >> >> and
>> > > > > > > > > > > > >> >> > a payload column for the metric-specific
>> data.
>> > > The
>> > > > > > > > > metric_type
>> > > > > > > > > > +
>> > > > > > > > > > > > >> >> > schema_version pair gives us a
>> > forward-compatible
>> > > > > > > contract
>> > > > > > > > > for
>> > > > > > > > > > > the
>> > > > > > > > > > > > >> >> payload
>> > > > > > > > > > > > >> >> > shape. Adding a new metric type becomes an
>> enum
>> > > > value
>> > > > > > > and a
>> > > > > > > > > > > payload
>> > > > > > > > > > > > >> >> schema,
>> > > > > > > > > > > > >> >> > not a schema migration. One thing I think we
>> > need
>> > > > to
>> > > > > be
>> > > > > > > > > > > deliberate
>> > > > > > > > > > > > >> >> about is
>> > > > > > > > > > > > >> >> > the partition key design. If all metric
>> types
>> > > land
>> > > > in
>> > > > > > one
>> > > > > > > > > > table,
>> > > > > > > > > > > > scan
>> > > > > > > > > > > > >> >> > metrics at scale (high concurrency, high
>> > > frequency
>> > > > > > across
>> > > > > > > > > many
>> > > > > > > > > > > > >> tables)
>> > > > > > > > > > > > >> >> > could easily create hot partitions. We'd
>> want
>> > the
>> > > > > > > > persistence
>> > > > > > > > > > > layer
>> > > > > > > > > > > > >> to
>> > > > > > > > > > > > >> >> be
>> > > > > > > > > > > > >> >> > able to shard by entity or time range, and
>> that
>> > > > means
>> > > > > > the
>> > > > > > > > > > logical
>> > > > > > > > > > > > >> schema
>> > > > > > > > > > > > >> >> > needs to expose enough structure for
>> backends
>> > to
>> > > > > > > partition
>> > > > > > > > > on.
>> > > > > > > > > > I
>> > > > > > > > > > > > >> don't
>> > > > > > > > > > > > >> >> > think the current flattened layout gives us
>> > that.
>> > > > > > > > > > > > >> >> >
>> > > > > > > > > > > > >> >> > This is getting complex enough that I don't
>> > think
>> > > > > > ad-hoc
>> > > > > > > > > PR/ML
>> > > > > > > > > > > > >> threads
>> > > > > > > > > > > > >> >> > will converge well. *Would people be open
>> to a
>> > > > > biweekly
>> > > > > > > > sync
>> > > > > > > > > > for
>> > > > > > > > > > > > >> metrics
>> > > > > > > > > > > > >> >> > architecture?* I think 30 minutes every two
>> > weeks
>> > > > > with
>> > > > > > > > > > interested
>> > > > > > > > > > > > >> >> parties
>> > > > > > > > > > > > >> >> > would be enough to work through the schema,
>> SPI
>> > > > > shape,
>> > > > > > > and
>> > > > > > > > > read
>> > > > > > > > > > > API
>> > > > > > > > > > > > >> >> design
>> > > > > > > > > > > > >> >> > together. Happy to help set that up.
>> > > > > > > > > > > > >> >> >
>> > > > > > > > > > > > >> >> > -ej
>> > > > > > > > > > > > >> >> >
>> > > > > > > > > > > > >> >> > On Mon, Apr 20, 2026 at 2:19 PM EJ Wang <
>> > > > > > > > > > > > >> ej.wang.dev.interest@gmail.com
>> > > > > > > > > > > > >> >> >
>> > > > > > > > > > > > >> >> > wrote:
>> > > > > > > > > > > > >> >> >
>> > > > > > > > > > > > >> >> >> Reviewed #4115, left a comment on the code
>> > > > > > organization
>> > > > > > > > > side.
>> > > > > > > > > > > > >> >> >>
>> > > > > > > > > > > > >> >> >> One thing stood out: the metrics write path
>> > > enters
>> > > > > > > through
>> > > > > > > > > > > > >> >> >> PolarisMetricsManager on MetaStoreManager,
>> but
>> > > the
>> > > > > new
>> > > > > > > > read
>> > > > > > > > > > path
>> > > > > > > > > > > > >> >> bypasses
>> > > > > > > > > > > > >> >> >> MetaStoreManager entirely and goes
>> straight to
>> > > > > > > > > BasePersistence
>> > > > > > > > > > > via
>> > > > > > > > > > > > >> >> >> callContext.getMetaStore(). That means the
>> > read
>> > > > API
>> > > > > > only
>> > > > > > > > > works
>> > > > > > > > > > > for
>> > > > > > > > > > > > >> >> backends
>> > > > > > > > > > > > >> >> >> that implement BasePersistence. NoSQL and
>> > remote
>> > > > > > > backends
>> > > > > > > > > > can't
>> > > > > > > > > > > > >> >> participate.
>> > > > > > > > > > > > >> >> >>
>> > > > > > > > > > > > >> >> >> Stepping back, I think the metrics
>> subsystem
>> > is
>> > > > > > growing
>> > > > > > > > into
>> > > > > > > > > > > > >> something
>> > > > > > > > > > > > >> >> >> real (write + read + REST API + AuthZ +
>> > > > pagination)
>> > > > > > *but
>> > > > > > > > the
>> > > > > > > > > > > > >> >> persistence
>> > > > > > > > > > > > >> >> >> side is split across two layers in a way
>> > that's
>> > > > hard
>> > > > > > to
>> > > > > > > > > > > extend*. I
>> > > > > > > > > > > > >> put
>> > > > > > > > > > > > >> >> >> together two diagrams to show what I mean
>> (my
>> > > best
>> > > > > > > > effort).
>> > > > > > > > > > > > >> >> >>
>> > > > > > > > > > > > >> >> >> *Current state* (Diagram 1): three
>> interfaces
>> > at
>> > > > > three
>> > > > > > > > > > different
>> > > > > > > > > > > > >> >> levels.
>> > > > > > > > > > > > >> >> >> The engine-facing SPI
>> (PolarisMetricsReporter)
>> > > is
>> > > > > > clean.
>> > > > > > > > But
>> > > > > > > > > > > > >> >> >> PolarisMetricsManager on MetaStoreManager
>> is a
>> > > > > > > passthrough
>> > > > > > > > > to
>> > > > > > > > > > > > >> >> >> MetricsPersistence on BasePersistence. The
>> > @Beta
>> > > > > > > > annotation
>> > > > > > > > > > and
>> > > > > > > > > > > > SPI
>> > > > > > > > > > > > >> >> javadoc
>> > > > > > > > > > > > >> >> >> are on the BasePersistence layer, while the
>> > > actual
>> > > > > > > > extension
>> > > > > > > > > > > > points
>> > > > > > > > > > > > >> >> >> (PolarisMetricsReporter,
>> > PolarisMetricsManager)
>> > > > > carry
>> > > > > > no
>> > > > > > > > > > > stability
>> > > > > > > > > > > > >> >> >> annotation. The write path goes through the
>> > > > > > > > MetaStoreManager
>> > > > > > > > > > > > layer,
>> > > > > > > > > > > > >> the
>> > > > > > > > > > > > >> >> >> read path doesn't.
>> > > > > > > > > > > > >> >> >>
>> > > > > > > > > > > > >> >> >> *What I envision* (Diagram 2): two SPIs at
>> two
>> > > > > levels.
>> > > > > > > > > > > > >> >> >> PolarisMetricsReporter stays as the
>> > > engine-facing
>> > > > > SPI.
>> > > > > > > > > > > > >> >> >> PolarisMetricsManager becomes the
>> > backend-facing
>> > > > SPI
>> > > > > > > with
>> > > > > > > > > both
>> > > > > > > > > > > > write
>> > > > > > > > > > > > >> >> and
>> > > > > > > > > > > > >> >> >> read methods at the MetaStoreManager level,
>> > > where
>> > > > > any
>> > > > > > > > > backend
>> > > > > > > > > > > > (JDBC,
>> > > > > > > > > > > > >> >> NoSQL,
>> > > > > > > > > > > > >> >> >> remote) can implement them.
>> MetricsPersistence
>> > > on
>> > > > > > > > > > > BasePersistence
>> > > > > > > > > > > > >> goes
>> > > > > > > > > > > > >> >> >> away. Where metrics actually land is an
>> > > > > implementation
>> > > > > > > > > detail,
>> > > > > > > > > > > > not a
>> > > > > > > > > > > > >> >> core
>> > > > > > > > > > > > >> >> >> interface.
>> > > > > > > > > > > > >> >> >>
>> > > > > > > > > > > > >> >> >> *Minor naming thing*:
>> PolarisMetricsReporter
>> > is
>> > > > > > broader
>> > > > > > > > than
>> > > > > > > > > > > what
>> > > > > > > > > > > > it
>> > > > > > > > > > > > >> >> >> actually handles. It only accepts Iceberg
>> REST
>> > > > > Catalog
>> > > > > > > > > metrics
>> > > > > > > > > > > > >> >> (ScanReport,
>> > > > > > > > > > > > >> >> >> CommitReport via MetricsReport). Generic
>> table
>> > > > > metrics
>> > > > > > > or
>> > > > > > > > > > > > >> operational
>> > > > > > > > > > > > >> >> >> metrics aren't in scope. Not blocking, but
>> > worth
>> > > > > > noting
>> > > > > > > if
>> > > > > > > > > the
>> > > > > > > > > > > > >> metrics
>> > > > > > > > > > > > >> >> >> surface expands.
>> > > > > > > > > > > > >> >> >>
>> > > > > > > > > > > > >> >> >> *Rough sketch of how to get there*:
>> > > > > > > > > > > > >> >> >>  1.  Add read methods to
>> PolarisMetricsManager
>> > > > > > > > > > (listScanReports,
>> > > > > > > > > > > > >> >> >> listCommitReports) with default no-op,
>> same as
>> > > the
>> > > > > > > > existing
>> > > > > > > > > > > write
>> > > > > > > > > > > > >> >> methods.
>> > > > > > > > > > > > >> >> >> (Probably make PolarisMetricsManager more
>> > > explicit
>> > > > > on
>> > > > > > > > being
>> > > > > > > > > > > > Iceberg
>> > > > > > > > > > > > >> >> >> specific like package name or class name
>> etc.)
>> > > > > > > > > > > > >> >> >>  2.  Wire MetricsReportsService through
>> > > > > > MetaStoreManager
>> > > > > > > > > > instead
>> > > > > > > > > > > > of
>> > > > > > > > > > > > >> >> >> callContext.getMetaStore().
>> > > > > > > > > > > > >> >> >>  3.  Extract metrics persistence from
>> > > > > > > > > JdbcBasePersistenceImpl
>> > > > > > > > > > > into
>> > > > > > > > > > > > >> its
>> > > > > > > > > > > > >> >> >> own class. That file carries ~7
>> > > responsibilities,
>> > > > > > > metrics
>> > > > > > > > > > being
>> > > > > > > > > > > > one
>> > > > > > > > > > > > >> of
>> > > > > > > > > > > > >> >> them.
>> > > > > > > > > > > > >> >> >>  4.  Remove MetricsPersistence from
>> > > > BasePersistence.
>> > > > > > > > > > > > >> >> >>
>> > > > > > > > > > > > >> >> >> *None of this needs to happen in #4115.
>> But if
>> > > the
>> > > > > > > > direction
>> > > > > > > > > > > makes
>> > > > > > > > > > > > >> >> sense,
>> > > > > > > > > > > > >> >> >> it would be good to align before the
>> metrics
>> > > > surface
>> > > > > > > grows
>> > > > > > > > > > > > further.
>> > > > > > > > > > > > >> >> Curious
>> > > > > > > > > > > > >> >> >> what others think.*
>> > > > > > > > > > > > >> >> >>
>> > > > > > > > > > > > >> >> >> *My mental model note*: Level 1
>> > > MetaStoreManager;
>> > > > > > level
>> > > > > > > 2
>> > > > > > > > > > > > >> transactional
>> > > > > > > > > > > > >> >> >> persistence; level 3 base persistence
>> > > > > > > > > > > > >> >> >>
>> > > > > > > > > > > > >> >> >> Diagram 1
>> > > > > > > > > > > > >> >> >> <
>> > > > > > > > > > > > >> >>
>> > > > > > > > > > > > >>
>> > > > > > > > > > > >
>> > > > > > > > > > >
>> > > > > > > > > >
>> > > > > > > > >
>> > > > > > > >
>> > > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>> https://www.plantuml.com/plantuml/uml/bLHDR-Cs4BthLmpIYupw0zbkKQ1r3M-S7Bp8xhhM7WCOb3IM65EaGD9EX2RzxHrHb4CxRelwa4YSDu_lpOVcnZ9jzvM8BBS2uGjQpJC3dtHMSekPtMk44IpsMgEqa5XcCOhCZikQQLP1pR8TAp2n3ILhmZDP20m0fcIvUkAoW2qJXd9z1bpToO9BX3WXu0ucy5rpgGPNm0nW5_epUWtm2Ue3pn3kMOFQmKntGZW0BYtgBSi8k5A2QMwybJNMIbFiGSR9QZc4nUqIvikStF0jHprua5C-amge42aNt3R0f5JaaoivdV2Pkqbx4hee4ymOkBh5BTiB-_uIeGeo8zL8rPsPl4DktdEiK1jkB1NdZCRbrSTecDe_mlHbF0wvBmCkaOH5_S8a_TTTKI6-nmCAkEw4LpxsZ-LbYLKQFKMNOgf_wuM7_bV9gOer5SYMMksBSWXFcbi49KNZXNLicwfe3TETC7gPdPqI7uBcHMb1RSzYq34c6PDUM9mn8HRsUTZEiDBve3NjVZumBj0U7SS37mGO7vcwtiK-_pU7U7L_f-digo9YbhSwIfMRwIITKGXbxdIUTCGF1SeCJxloKsU-3k9ddRbX1eDq1q_fx1JbBGT0glVyXimDuP4TQ5qpCAmnGEj2s_6n5mtn1z-97-63itFQZLPO1Ev2tu_WF7Ju-VPc0Skg5bYXxBhkY1xpD7EM_7fyflSpIsqMgVth5xhVr4eQxWQ8enaSAJQSG16yFSDuJ798rrcXr_3n-lfdk7icQjEBmFujL7AodiP_Y4Z7-YxvtZNs4zMgpNTl6tF8sglyPsmqchrjvQ-m-aP94r-TwCA2Ka8upPJZwtvSpoYCXkYMZU2NXvRMBfq9P3i3Le4VAZUAlUZ_oPKsxPgY0Q_BSKLkyr9bhQhQrJjo_x3TPlIB0DPjnMfcIoYP0QaYw1a0fTKDr8fB6ntNuvmoL1ZGkXa69Njh43zf9GiGxHQrA_jDYWRSzF5--WmTVrN97_Sm8LbLUy_lGBmLanJjFkDlGkRqjA_4tm00
>> > > > > > > > > > > > >> >> >
>> > > > > > > > > > > > >> >> >> :
>> > > > > > > > > > > > >> >> >>
>> > > > > > > > > > > > >> >> >> [image: image.png]
>> > > > > > > > > > > > >> >> >>
>> > > > > > > > > > > > >> >> >> Diagram 2
>> > > > > > > > > > > > >> >> >> <
>> > > > > > > > > > > > >> >>
>> > > > > > > > > > > > >>
>> > > > > > > > > > > >
>> > > > > > > > > > >
>> > > > > > > > > >
>> > > > > > > > >
>> > > > > > > >
>> > > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>> https://www.plantuml.com/plantuml/uml/VLLDR-8m4BtdLupO2sWBLVU8AaGB7AXAbssGzb896SS9RXqxjKqBwkv_tt7iV43fdaZYDpFlpRmnOsE9jhjSH9PRmM31hERKm8scMsuPjJlDe0yheZDc8RR4iYWoBrmMH9CS2a9VICPYUy1OZN0YCy5Q0BCbYNhdCeEK28En8G8wCvbnoQ0R8_05Bc6bkLIz3X03p1zzH7zR-9ZfDquPt9C3qoNCX2yV4G2NbkcKu5jdgGJHt0GbZwnG6i-UP3TUpk5gM6Ldqke350eZUqzoCft3U9xWHvxoa5-7K4nF1J46EbEMafsmdrCBbQ44gVggy18IZrn_ph5asd1ZiIKdQSgueZvjXrQFSFrdC3YN-nXmBacxbGiYyLVxLaBtdhqn0LSzdBDhqQtQoOJeGyad3z0lUqnYgpGB6Ns8oVyta00Dy_WnX0tIOZ8v6SYxHll1TrH6aejAik-mh-AphVFCwSUQqFypElag5QRGFDjQKEd96K1P8QP41c9TzA_IIQyvdAWyv_RSiS3skb0_EzDDkK2v5xWF6MiGFlvhpFLcD2Dq2pml14gaF67eQkmd8gulDoC4kSOu6KVpkvlUJg1RTbWISU40RdBUUS_9XfRZ2dwxm_SW8LYFISgm_MnlDQ6M9P1gbKEc4X-2pH_FvJCkCqm9pbVjD6LrwdLeOrDWfOaqc8Wh9BE85oNKxkNQ6o4yGRy_Eae0G_G8tZv81d3bHDB23WOdisohVr3nh_j6lbSjbNaLRTc8UgtPbAU1J_tygOfZX9DWEJeHDvYx-qmSi5FgNLPZwHrHcUsncGQ5-skhUclpE5fo4ounpFauYrUbkU6ccfnxMvitwag4IyerhTxj8In_Oj1bDO4pQru674loYrGlULHLEGCjwJJ8gDoVZR8MxO4BT3IzRvIcAQKezC6xpziGnTyImrfEGyJI_OcKfgtxIvnTqFEMS17L9Z-jsARN5FmTheP7HtSdtOMT0B4GY2FYHXxgQmMtj2bRqiLFGapiVe1_QVKDrkqXcm83aFEXnMYCZ-xlyHy
>> > > > > > > > > > > > >> >> >
>> > > > > > > > > > > > >> >> >> :
>> > > > > > > > > > > > >> >> >> [image: image.png]
>> > > > > > > > > > > > >> >> >>
>> > > > > > > > > > > > >> >> >>  -ej
>> > > > > > > > > > > > >> >> >>
>> > > > > > > > > > > > >> >> >> On Wed, Apr 15, 2026 at 8:22 AM Dmitri
>> > > > Bourlatchkov
>> > > > > <
>> > > > > > > > > > > > >> dimas@apache.org>
>> > > > > > > > > > > > >> >> >> wrote:
>> > > > > > > > > > > > >> >> >>
>> > > > > > > > > > > > >> >> >>> Hi All,
>> > > > > > > > > > > > >> >> >>>
>> > > > > > > > > > > > >> >> >>> Heads up: The current state of PR [4115]
>> > looks
>> > > > > pretty
>> > > > > > > > solid
>> > > > > > > > > > to
>> > > > > > > > > > > > me.
>> > > > > > > > > > > > >> I
>> > > > > > > > > > > > >> >> >>> believe this PR is approaching a mergeable
>> > > > > condition.
>> > > > > > > > > > > > >> >> >>>
>> > > > > > > > > > > > >> >> >>> Please post your reviews if you have any
>> > > > comments.
>> > > > > > > > > > > > >> >> >>>
>> > > > > > > > > > > > >> >> >>> [4115]
>> > > > https://github.com/apache/polaris/pull/4115
>> > > > > > > > > > > > >> >> >>>
>> > > > > > > > > > > > >> >> >>> Thanks,
>> > > > > > > > > > > > >> >> >>> Dmitri.
>> > > > > > > > > > > > >> >> >>>
>> > > > > > > > > > > > >> >> >>> On Tue, Mar 3, 2026 at 3:29 PM Anand Kumar
>> > > > Sankaran
>> > > > > > via
>> > > > > > > > > dev <
>> > > > > > > > > > > > >> >> >>> dev@polaris.apache.org> wrote:
>> > > > > > > > > > > > >> >> >>>
>> > > > > > > > > > > > >> >> >>> > Hi Yufei and Dmitri,
>> > > > > > > > > > > > >> >> >>> >
>> > > > > > > > > > > > >> >> >>> > Here is a proposal for the REST
>> endpoints
>> > for
>> > > > > > metrics
>> > > > > > > > and
>> > > > > > > > > > > > events.
>> > > > > > > > > > > > >> >> >>> >
>> > > > > > > > > > > > >> >> >>> >
>> > > > > > https://github.com/apache/polaris/pull/3924/changes
>> > > > > > > > > > > > >> >> >>> >
>> > > > > > > > > > > > >> >> >>> > I did not see any precursors for
>> raising a
>> > PR
>> > > > for
>> > > > > > > > > > proposals,
>> > > > > > > > > > > so
>> > > > > > > > > > > > >> >> trying
>> > > > > > > > > > > > >> >> >>> > this.  Please let me know what you
>> think.
>> > > > > > > > > > > > >> >> >>> >
>> > > > > > > > > > > > >> >> >>> > -
>> > > > > > > > > > > > >> >> >>> > Anand
>> > > > > > > > > > > > >> >> >>> >
>> > > > > > > > > > > > >> >> >>> > From: Anand Kumar Sankaran <
>> > > > > > > anand.sankaran@workday.com
>> > > > > > > > >
>> > > > > > > > > > > > >> >> >>> > Date: Monday, March 2, 2026 at 10:25 AM
>> > > > > > > > > > > > >> >> >>> > To: dev@polaris.apache.org <
>> > > > > dev@polaris.apache.org
>> > > > > > >
>> > > > > > > > > > > > >> >> >>> > Subject: Re: Polaris Telemetry and Audit
>> > > Trail
>> > > > > > > > > > > > >> >> >>> >
>> > > > > > > > > > > > >> >> >>> > About the REST API, based on my use
>> cases:
>> > > > > > > > > > > > >> >> >>> >
>> > > > > > > > > > > > >> >> >>> >
>> > > > > > > > > > > > >> >> >>> >   1.
>> > > > > > > > > > > > >> >> >>> > I want to be able to query commit
>> metrics
>> > to
>> > > > > track
>> > > > > > > > files
>> > > > > > > > > > > added
>> > > > > > > > > > > > /
>> > > > > > > > > > > > >> >> >>> removed
>> > > > > > > > > > > > >> >> >>> > per commit, along with record counts.
>> The
>> > > > > ingestion
>> > > > > > > > > > pipeline
>> > > > > > > > > > > > that
>> > > > > > > > > > > > >> >> >>> writes
>> > > > > > > > > > > > >> >> >>> > this data is owned by us and we are
>> > > guaranteed
>> > > > to
>> > > > > > > write
>> > > > > > > > > > this
>> > > > > > > > > > > > >> >> >>> information
>> > > > > > > > > > > > >> >> >>> > for each write.
>> > > > > > > > > > > > >> >> >>> >   2.
>> > > > > > > > > > > > >> >> >>> > I want to be able to query scan metrics
>> for
>> > > > > read. I
>> > > > > > > > > > > understand
>> > > > > > > > > > > > >> >> clients
>> > > > > > > > > > > > >> >> >>> do
>> > > > > > > > > > > > >> >> >>> > not fulfill this requirement.
>> > > > > > > > > > > > >> >> >>> >   3.
>> > > > > > > > > > > > >> >> >>> > I want to be able to query the events
>> table
>> > > > > (events
>> > > > > > > are
>> > > > > > > > > > > > >> persisted) -
>> > > > > > > > > > > > >> >> >>> this
>> > > > > > > > > > > > >> >> >>> > may supersede #2, I am not sure yet.
>> > > > > > > > > > > > >> >> >>> >
>> > > > > > > > > > > > >> >> >>> > All this information is in the JDBC
>> based
>> > > > > > persistence
>> > > > > > > > > model
>> > > > > > > > > > > and
>> > > > > > > > > > > > >> is
>> > > > > > > > > > > > >> >> >>> > persisted in the metastore. I currently
>> > don’t
>> > > > > have
>> > > > > > a
>> > > > > > > > need
>> > > > > > > > > > to
>> > > > > > > > > > > > >> query
>> > > > > > > > > > > > >> >> >>> > prometheus or open telemetry. I do
>> publish
>> > > some
>> > > > > > > events
>> > > > > > > > to
>> > > > > > > > > > > > >> Prometheus
>> > > > > > > > > > > > >> >> >>> and
>> > > > > > > > > > > > >> >> >>> > they are forwarded to our dashboards
>> > > elsewhere.
>> > > > > > > > > > > > >> >> >>> >
>> > > > > > > > > > > > >> >> >>> > About the CLI utilities, I meant the
>> admin
>> > > user
>> > > > > > > > > utilities.
>> > > > > > > > > > In
>> > > > > > > > > > > > >> one of
>> > > > > > > > > > > > >> >> >>> the
>> > > > > > > > > > > > >> >> >>> > earliest drafts of my proposal, Prashant
>> > > > > mentioned
>> > > > > > > that
>> > > > > > > > > the
>> > > > > > > > > > > > >> metrics
>> > > > > > > > > > > > >> >> >>> tables
>> > > > > > > > > > > > >> >> >>> > can grow indefinitely and that a similar
>> > > > problem
>> > > > > > > exists
>> > > > > > > > > > with
>> > > > > > > > > > > > the
>> > > > > > > > > > > > >> >> events
>> > > > > > > > > > > > >> >> >>> > table as well. We discussed that
>> cleaning
>> > up
>> > > of
>> > > > > old
>> > > > > > > > > records
>> > > > > > > > > > > > from
>> > > > > > > > > > > > >> >> both
>> > > > > > > > > > > > >> >> >>> > metrics tables and events tables can be
>> > done
>> > > > via
>> > > > > a
>> > > > > > > CLI
>> > > > > > > > > > > utility.
>> > > > > > > > > > > > >> >> >>> >
>> > > > > > > > > > > > >> >> >>> > I see that Yufei has covered the
>> discussion
>> > > > about
>> > > > > > > > > > > datasources.
>> > > > > > > > > > > > >> >> >>> >
>> > > > > > > > > > > > >> >> >>> > -
>> > > > > > > > > > > > >> >> >>> > Anand
>> > > > > > > > > > > > >> >> >>> >
>> > > > > > > > > > > > >> >> >>> >
>> > > > > > > > > > > > >> >> >>> >
>> > > > > > > > > > > > >> >> >>> > From: Yufei Gu <fl...@gmail.com>
>> > > > > > > > > > > > >> >> >>> > Date: Friday, February 27, 2026 at
>> 9:54 PM
>> > > > > > > > > > > > >> >> >>> > To: dev@polaris.apache.org <
>> > > > > dev@polaris.apache.org
>> > > > > > >
>> > > > > > > > > > > > >> >> >>> > Subject: Re: Polaris Telemetry and Audit
>> > > Trail
>> > > > > > > > > > > > >> >> >>> >
>> > > > > > > > > > > > >> >> >>> > This Message Is From an External Sender
>> > > > > > > > > > > > >> >> >>> > This message came from outside your
>> > > > organization.
>> > > > > > > > > > > > >> >> >>> > Report Suspicious<
>> > > > > > > > > > > > >> >> >>> >
>> > > > > > > > > > > > >> >> >>>
>> > > > > > > > > > > > >> >>
>> > > > > > > > > > > > >>
>> > > > > > > > > > > >
>> > > > > > > > > > >
>> > > > > > > > > >
>> > > > > > > > >
>> > > > > > > >
>> > > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>> https://us-phishalarm-ewt.proofpoint.com/EWT/v1/Iz9xO38YGHZK!YhNDZABkHi1B699ote2uMwpOZw8i0QMCGO2Szc-HshuABGhGvwPJcymE6G2oUUxtS8xDkSrtGTPm_I3QnVDHoLMk50m9v8z_nZKTkd-bnVUbreF1u0WnfV_X5eYevZl_$
>> > > > > > > > > > > > >> >> >>> > >
>> > > > > > > > > > > > >> >> >>> >
>> > > > > > > > > > > > >> >> >>> >
>> > > > > > > > > > > > >> >> >>> > As I mentioned in
>> > > > > > > > > > > > >> >> >>> >
>> > > > > > > > > > > > >> >> >>>
>> > > > > > > > > > > > >> >>
>> > > > > > > > > > > > >>
>> > > > > > > > > > > >
>> > > > > > > > > > >
>> > > > > > > > > >
>> > > > > > > > >
>> > > > > > > >
>> > > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>> https://urldefense.com/v3/__https://github.com/apache/polaris/issues/3890__;!!Iz9xO38YGHZK!5EuyFFkk3vhRWVIRvQAWBSQfpJkTMA9HxugzDwXmN0LPPqhEFxYkFRGVhtb8AqUwXtDh2OplcMnbMDHKOxrvDU0$
>> > > > > > > > > > > > >> >> >>> ,
>> > > > > > > > > > > > >> >> >>> > supporting
>> > > > > > > > > > > > >> >> >>> > multiple data sources is not a trivial
>> > > change.
>> > > > I
>> > > > > > > would
>> > > > > > > > > > > strongly
>> > > > > > > > > > > > >> >> >>> recommend
>> > > > > > > > > > > > >> >> >>> > starting with a design document to
>> > carefully
>> > > > > > evaluate
>> > > > > > > > the
>> > > > > > > > > > > > >> >> architectural
>> > > > > > > > > > > > >> >> >>> > implications and long term impact.
>> > > > > > > > > > > > >> >> >>> >
>> > > > > > > > > > > > >> >> >>> > A REST endpoint to query metrics seems
>> > > > reasonable
>> > > > > > > given
>> > > > > > > > > the
>> > > > > > > > > > > > >> current
>> > > > > > > > > > > > >> >> >>> JDBC
>> > > > > > > > > > > > >> >> >>> > based persistence model. That said, we
>> may
>> > > also
>> > > > > > > > consider
>> > > > > > > > > > > > >> alternative
>> > > > > > > > > > > > >> >> >>> > storage models. For example, if we later
>> > > adopt
>> > > > a
>> > > > > > time
>> > > > > > > > > > series
>> > > > > > > > > > > > >> system
>> > > > > > > > > > > > >> >> >>> such as
>> > > > > > > > > > > > >> >> >>> > Prometheus to store metrics, the query
>> > model
>> > > > and
>> > > > > > > access
>> > > > > > > > > > > > patterns
>> > > > > > > > > > > > >> >> would
>> > > > > > > > > > > > >> >> >>> be
>> > > > > > > > > > > > >> >> >>> > fundamentally different. Designing the
>> REST
>> > > API
>> > > > > > > without
>> > > > > > > > > > > > >> considering
>> > > > > > > > > > > > >> >> >>> these
>> > > > > > > > > > > > >> >> >>> > potential evolutions may limit
>> flexibility.
>> > > I'd
>> > > > > > > suggest
>> > > > > > > > > to
>> > > > > > > > > > > > start
>> > > > > > > > > > > > >> >> with
>> > > > > > > > > > > > >> >> >>> the
>> > > > > > > > > > > > >> >> >>> > use case.
>> > > > > > > > > > > > >> >> >>> >
>> > > > > > > > > > > > >> >> >>> > Yufei
>> > > > > > > > > > > > >> >> >>> >
>> > > > > > > > > > > > >> >> >>> >
>> > > > > > > > > > > > >> >> >>> > On Fri, Feb 27, 2026 at 3:42 PM Dmitri
>> > > > > > Bourlatchkov <
>> > > > > > > > > > > > >> >> dimas@apache.org>
>> > > > > > > > > > > > >> >> >>> > wrote:
>> > > > > > > > > > > > >> >> >>> >
>> > > > > > > > > > > > >> >> >>> > > Hi Anand,
>> > > > > > > > > > > > >> >> >>> > >
>> > > > > > > > > > > > >> >> >>> > > Sharing my view... subject to
>> discussion:
>> > > > > > > > > > > > >> >> >>> > >
>> > > > > > > > > > > > >> >> >>> > > 1. Adding non-IRC REST API to Polaris
>> is
>> > > > > > perfectly
>> > > > > > > > > fine.
>> > > > > > > > > > > > >> >> >>> > >
>> > > > > > > > > > > > >> >> >>> > > Figuring out specific endpoint URIs
>> and
>> > > > > payloads
>> > > > > > > > might
>> > > > > > > > > > > > require
>> > > > > > > > > > > > >> a
>> > > > > > > > > > > > >> >> few
>> > > > > > > > > > > > >> >> >>> > > roundtrips, so opening a separate
>> thread
>> > > for
>> > > > > that
>> > > > > > > > might
>> > > > > > > > > > be
>> > > > > > > > > > > > >> best.
>> > > > > > > > > > > > >> >> >>> > > Contributors commonly create Google
>> Docs
>> > > for
>> > > > > new
>> > > > > > > API
>> > > > > > > > > > > > proposals
>> > > > > > > > > > > > >> too
>> > > > > > > > > > > > >> >> >>> (they
>> > > > > > > > > > > > >> >> >>> > > fairly easy to update as the email
>> > > discussion
>> > > > > > > > > > progresses).
>> > > > > > > > > > > > >> >> >>> > >
>> > > > > > > > > > > > >> >> >>> > > There was a suggestion to try Markdown
>> > > (with
>> > > > > PRs)
>> > > > > > > for
>> > > > > > > > > > > > proposals
>> > > > > > > > > > > > >> >> [1]
>> > > > > > > > > > > > >> >> >>> ...
>> > > > > > > > > > > > >> >> >>> > > feel free to give it a try if you are
>> > > > > comfortable
>> > > > > > > > with
>> > > > > > > > > > > that.
>> > > > > > > > > > > > >> >> >>> > >
>> > > > > > > > > > > > >> >> >>> > > 2. Could you clarify whether you mean
>> end
>> > > > user
>> > > > > > > > > utilities
>> > > > > > > > > > or
>> > > > > > > > > > > > >> admin
>> > > > > > > > > > > > >> >> >>> user
>> > > > > > > > > > > > >> >> >>> > > utilities? In the latter case those
>> might
>> > > be
>> > > > > more
>> > > > > > > > > > suitable
>> > > > > > > > > > > > for
>> > > > > > > > > > > > >> the
>> > > > > > > > > > > > >> >> >>> Admin
>> > > > > > > > > > > > >> >> >>> > > CLI (java) not the Python CLI, IMHO.
>> > > > > > > > > > > > >> >> >>> > >
>> > > > > > > > > > > > >> >> >>> > > Why would these utilities be common
>> with
>> > > > > events?
>> > > > > > > > IMHO,
>> > > > > > > > > > > event
>> > > > > > > > > > > > >> use
>> > > > > > > > > > > > >> >> >>> cases
>> > > > > > > > > > > > >> >> >>> > are
>> > > > > > > > > > > > >> >> >>> > > distinct from scan/commit metrics.
>> > > > > > > > > > > > >> >> >>> > >
>> > > > > > > > > > > > >> >> >>> > > 3. I'd prefer separating metrics
>> > > persistence
>> > > > > from
>> > > > > > > > > > MetaStore
>> > > > > > > > > > > > >> >> >>> persistence
>> > > > > > > > > > > > >> >> >>> > at
>> > > > > > > > > > > > >> >> >>> > > the code level, so that they could be
>> > mixed
>> > > > and
>> > > > > > > > matched
>> > > > > > > > > > > > >> >> >>> independently.
>> > > > > > > > > > > > >> >> >>> > The
>> > > > > > > > > > > > >> >> >>> > > separate datasource question will
>> become
>> > a
>> > > > > > > non-issue
>> > > > > > > > > with
>> > > > > > > > > > > > that
>> > > > > > > > > > > > >> >> >>> approach,
>> > > > > > > > > > > > >> >> >>> > I
>> > > > > > > > > > > > >> >> >>> > > guess.
>> > > > > > > > > > > > >> >> >>> > >
>> > > > > > > > > > > > >> >> >>> > > The rationale for separating scan
>> metrics
>> > > and
>> > > > > > > > metastore
>> > > > > > > > > > > > >> >> persistence
>> > > > > > > > > > > > >> >> >>> is
>> > > > > > > > > > > > >> >> >>> > that
>> > > > > > > > > > > > >> >> >>> > > "cascading deletes" between them are
>> > hardly
>> > > > > ever
>> > > > > > > > > > required.
>> > > > > > > > > > > > >> >> >>> Furthermore,
>> > > > > > > > > > > > >> >> >>> > the
>> > > > > > > > > > > > >> >> >>> > > data and query patterns are very
>> > different
>> > > so
>> > > > > > > > different
>> > > > > > > > > > > > >> >> technologies
>> > > > > > > > > > > > >> >> >>> > might
>> > > > > > > > > > > > >> >> >>> > > be beneficial in each case.
>> > > > > > > > > > > > >> >> >>> > >
>> > > > > > > > > > > > >> >> >>> > > [1]
>> > > > > > > > > > > > >> >> >>> >
>> > > > > > > > > > > > >> >> >>>
>> > > > > > > > > > > > >> >>
>> > > > > > > > > > > > >>
>> > > > > > > > > > > >
>> > > > > > > > > > >
>> > > > > > > > > >
>> > > > > > > > >
>> > > > > > > >
>> > > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>> https://urldefense.com/v3/__https://lists.apache.org/thread/yto2wp982t43h1mqjwnslswhws5z47cy__;!!Iz9xO38YGHZK!5EuyFFkk3vhRWVIRvQAWBSQfpJkTMA9HxugzDwXmN0LPPqhEFxYkFRGVhtb8AqUwXtDh2OplcMnbMDHKxYDakNU$
>> > > > > > > > > > > > >> >> >>> > >
>> > > > > > > > > > > > >> >> >>> > > Cheers,
>> > > > > > > > > > > > >> >> >>> > > Dmitri.
>> > > > > > > > > > > > >> >> >>> > >
>> > > > > > > > > > > > >> >> >>> > > On Fri, Feb 27, 2026 at 6:19 PM Anand
>> > Kumar
>> > > > > > > Sankaran
>> > > > > > > > > via
>> > > > > > > > > > > dev
>> > > > > > > > > > > > <
>> > > > > > > > > > > > >> >> >>> > > dev@polaris.apache.org> wrote:
>> > > > > > > > > > > > >> >> >>> > >
>> > > > > > > > > > > > >> >> >>> > > > Thanks all. This PR is merged now.
>> > > > > > > > > > > > >> >> >>> > > >
>> > > > > > > > > > > > >> >> >>> > > > Here are the follow-up features /
>> work
>> > > > > needed.
>> > > > > > > > These
>> > > > > > > > > > > were
>> > > > > > > > > > > > >> all
>> > > > > > > > > > > > >> >> >>> part of
>> > > > > > > > > > > > >> >> >>> > > the
>> > > > > > > > > > > > >> >> >>> > > > merged PR at some point in time and
>> > were
>> > > > > > removed
>> > > > > > > to
>> > > > > > > > > > > reduce
>> > > > > > > > > > > > >> >> scope.
>> > > > > > > > > > > > >> >> >>> > > >
>> > > > > > > > > > > > >> >> >>> > > > Please let me know what you think.
>> > > > > > > > > > > > >> >> >>> > > >
>> > > > > > > > > > > > >> >> >>> > > >
>> > > > > > > > > > > > >> >> >>> > > >   1.  A REST API to paginate through
>> > > table
>> > > > > > > metrics.
>> > > > > > > > > > This
>> > > > > > > > > > > > >> will be
>> > > > > > > > > > > > >> >> >>> > non-IRC
>> > > > > > > > > > > > >> >> >>> > > > standard addition.
>> > > > > > > > > > > > >> >> >>> > > >   2.  Utilities for managing old
>> > records,
>> > > > > > should
>> > > > > > > be
>> > > > > > > > > > > common
>> > > > > > > > > > > > >> with
>> > > > > > > > > > > > >> >> >>> events.
>> > > > > > > > > > > > >> >> >>> > > > There was some discussion that it
>> > belongs
>> > > > to
>> > > > > > the
>> > > > > > > > CLI.
>> > > > > > > > > > > > >> >> >>> > > >   3.  Separate datasource (metrics,
>> > > events,
>> > > > > > even
>> > > > > > > > > other
>> > > > > > > > > > > > >> tables?).
>> > > > > > > > > > > > >> >> >>> > > >
>> > > > > > > > > > > > >> >> >>> > > >
>> > > > > > > > > > > > >> >> >>> > > > Anything else?
>> > > > > > > > > > > > >> >> >>> > > >
>> > > > > > > > > > > > >> >> >>> > > > -
>> > > > > > > > > > > > >> >> >>> > > > Anand
>> > > > > > > > > > > > >> >> >>> > > >
>> > > > > > > > > > > > >> >> >>> > > >
>> > > > > > > > > > > > >> >> >>> > >
>> > > > > > > > > > > > >> >> >>> >
>> > > > > > > > > > > > >> >> >>> >
>> > > > > > > > > > > > >> >> >>>
>> > > > > > > > > > > > >> >> >>
>> > > > > > > > > > > > >> >>
>> > > > > > > > > > > > >> >
>> > > > > > > > > > > > >>
>> > > > > > > > > > > > >
>> > > > > > > > > > > >
>> > > > > > > > > > >
>> > > > > > > > > >
>> > > > > > > > >
>> > > > > > > >
>> > > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>>
>

Re: Proposal for REST endpoints for table metrics and events

[Dmitri Bourlatchkov <di...@apache.org>]

Hi All,

Recap of the discussion in today's Community Sync:

* The modular approach for Metrics SPI(s) and implementation seems to have
community consensus (please comment if you still have concerns)

* What modules / feature flags are enabled in Apache Polaris images by
default will be discussed later (when the feature is close to completion).

In any case, users will be able to turn the full feature on/off at their
discretion, regardless of the default flag values.

* It is generally preferable to have smaller PRs and introduce new features
gradually to simplify reviews.

What this means for the Metrics API, IMHO is:

a) The scope of PR 4115 is probably ok as it stands right now. If someone
wants to propose splitting it further, please comment.

b) After 4115 is merged, Metrics Persistence PRs will follow, with smaller
scope per PR as appropriate for the nature of the code changes.

* The major concern was about exposing the new Metrics REST API to end
users. The nature of this concern was about avoiding the (false) impression
that the new API is a firm and adopted Polaris specification.

I think this concern is reasonable.

I propose that Anand update the documentation sections of PR 4115 to
emphasise that the API is currently an experimental / POC API at this time
and is subject to change (including breaking changes). I believe it already
has the "beta" label. So this request is mostly for doc changes to clarify
user expectations, since the "beta" label alone may not be interpreted the
same way by all users.

I hope this update will resolve all remaining concerns with 4115 and allow
for it to be merged so that the community will be able to proceed to
follow-up PRs (as noted above).

Please add/correct me if I missed anything relevant.

Thanks,
Dmitri.

On Wed, Jun 24, 2026 at 8:11 PM Yufei Gu <fl...@gmail.com> wrote:

> I do not think we have agreement on the query REST APIs yet.
>
> In particular, the current query API shape seems highly coupled to the
> *example[1]* metrics in the Iceberg REST spec. I do not think that is a
> good way to design a REST API contract. For example, fields like
> resultDataFiles, resultDeleteFiles, totalFileSizeBytes, totalDataManifests,
> totalDeleteManifests, and scannedDataManifests are very specific to one
> style of scan metric. That makes the API look more like a direct projection
> of the current example payload than a general query model.
>
> My preference is that Polaris should not define or strongly opine on the
> Iceberg scan or commit metrics query REST API at this point. Polaris can
> resolve and authorize the table, accept the metrics report, and delegate it
> to the configured reporting path. Durable storage, filtering, dashboards,
> and query APIs should remain downstream impl. choices unless and until the
> Iceberg community standardizes a query contract, which I doubt will happen.
>
> So I do not think the intake path has to be blocked, but I would prefer not
> to merge a Polaris specific query API that may be hard to evolve later.
>
> 1.
>
> https://github.com/apache/iceberg/blob/2a6c556c0c883c04c6d3fbd68e6aeda36b91e0aa/open-api/rest-catalog-open-api.yaml#L4152
>
> Yufei
>
>
> On Tue, Jun 23, 2026 at 9:30 AM Dmitri Bourlatchkov <di...@apache.org>
> wrote:
>
> > Hi All,
> >
> > So, what are the blockers for merging [4115], if any?
> >
> > [4115] https://github.com/apache/polaris/pull/4115
> >
> > Thanks,
> > Dmitri.
> >
> > On Wed, Jun 17, 2026 at 5:28 PM EJ Wang <ej...@gmail.com>
> > wrote:
> >
> > > Hi folks,
> > >
> > > Thanks again for the discussion today. I updated the sync doc with
> notes
> > > from today's metrics sync:
> > >
> > >
> > >
> >
> https://docs.google.com/document/d/100h7c4damrUzVuquYbBHM0EvA4LSWuW2IT2dN_7nYVA/edit?tab=t.ezk0rgdx0c6m
> > >
> > > A few highlights:
> > >
> > >    - No objection to keeping the default metrics behavior small:
> > >    no-op/log-only is enough as the built-in default.
> > >    - Durable storage, event forwarding, external queues, dashboards,
> and
> > >    custom filtering should be implementation choices behind the metrics
> > >    reporting path.
> > >    - For PR #4115, the REST layer should resolve/authz/accept the
> report,
> > >    then delegate to the selected metrics implementation. It should not
> > own
> > >    durable storage semantics.
> > >    - The rough boundary we discussed is: spec owns the wire contract,
> > >    runtime owns framework wiring, API/contract modules own
> > provider-facing
> > >    contracts, and extensions own replaceable implementations.
> > >    - OpenLineage has a similar path-naming question. We should avoid
> > >    occupying a generic lineage path too early if we want room for other
> > >    lineage systems later.
> > >
> > > I believe the boundary rules previewed in sync meeting are useful for
> > > reviewing PR #4115, but it is not
> > > a final module-layout proposal.
> > >
> > > For metrics specifically, I think the target remains:
> > >    runtime
> > >      -> metrics reporting/emitting contract
> > >      -> default implementation
> > >           -> lower implementation details, if needed
> > >
> > > That keeps the default simple, while allowing durable JDBC,
> event-backed,
> > > or external-queue implementations to be added separately.
> > >
> > > We also touched on the split between Iceberg metrics emitting and
> > > Polaris-owned metrics querying. The querying side can keep evolving,
> > > especially for dashboard or generic-table use cases, but I do not think
> > > that needs to block the intake path in this PR.
> > >
> > > Thanks,
> > > -ej
> > >
> > > On Tue, Jun 16, 2026 at 2:17 PM Yufei Gu <fl...@gmail.com> wrote:
> > >
> > > > Thanks for chiming in, EJ. Agreed that the default battery should
> stay
> > > > small. I'm leaning toward not including metrics persistence in the
> > > default
> > > > battery.
> > > >
> > > > Yufei
> > > >
> > > >
> > > > On Thu, Jun 11, 2026 at 8:30 PM EJ Wang <
> > ej.wang.dev.interest@gmail.com>
> > > > wrote:
> > > >
> > > > > Hi Yufei,
> > > > >
> > > > > Thanks for connecting this back to the REST endpoint proposal. I
> > > replied
> > > > > with the fuller version on the event-forwarding thread, but wanted
> to
> > > add
> > > > > the shorter version here too.
> > > > >
> > > > > I agree that metrics reporting/emitting is the right conceptual
> > > boundary,
> > > > > and I also think the event/listener path is a good implementation
> > > > direction
> > > > > to explore. The distinction I want to keep clear is default battery
> > vs
> > > > > extension implementation.
> > > > >
> > > > > For the REST/API side, I would keep the semantics narrow: Polaris
> > > > resolves
> > > > > and authorizes the table, accepts the Iceberg scan/commit metrics
> > > report
> > > > > into the configured reporting/emitting path, and returns 204. That
> > 204
> > > > > should mean Polaris accepted the report into the ingestion path; it
> > > > should
> > > > > not imply durable storage.
> > > > >
> > > > > Durable storage, event forwarding, external telemetry routing,
> > > filtering,
> > > > > and retention should sit behind that reporting/emitting boundary as
> > > > > implementation-layer behavior. A durable JDBC path can be one named
> > > > > extension implementation. An event/listener forwarding path can be
> > > > another
> > > > > named extension implementation.
> > > > >
> > > > > So I am +1 on the event-forwarding idea as a non-default extension
> > > > > implementation. I just would not make it the default battery or the
> > > core
> > > > > API shape. The default battery can stay small and safe, while
> > > deployments
> > > > > that need forwarding, dashboards, or durable storage can opt into
> the
> > > > > implementation that matches their operational model.
> > > > >
> > > > > Thanks,
> > > > > -ej
> > > > >
> > > > > On Wed, Jun 10, 2026 at 11:43 AM Yufei Gu <fl...@gmail.com>
> > > wrote:
> > > > >
> > > > > > Thanks EJ. I agree that the reporting/emitting boundary feels
> like
> > > the
> > > > > > right SPI boundary, but I wonder if we can simplify this even
> > > further.
> > > > > >
> > > > > > Iceberg scan and commit metrics look very similar to events to
> me.
> > > They
> > > > > are
> > > > > > append only, asynchronously consumed, often forwarded to external
> > > > > systems,
> > > > > > and have similar retention and cleanup requirements. More details
> > can
> > > > be
> > > > > > found in this thread [1]. In that case, Polaris may only need to
> > > accept
> > > > > the
> > > > > > report and emit an event through the existing event framework.
> > > > Filtering,
> > > > > > async delivery, custom sinks, and retention mechanisms could then
> > be
> > > > > shared
> > > > > > instead of introducing a separate metrics specific extension
> layer
> > > and
> > > > > SPI.
> > > > > >
> > > > > > The main requirement I still see is querying metrics through
> > Polaris.
> > > > If
> > > > > we
> > > > > > want that battery included experience, we could work on a
> > persistence
> > > > > > listener implementation that selectively saves metrics and expose
> > it
> > > > via
> > > > > > the IRC event endpoint which seems pretty close in Iceberg
> > > > > > community. Another option I like more is to allow users to
> develop
> > > > their
> > > > > > own listener, so they can persist scan/commit metrics to any
> > systems
> > > > they
> > > > > > prefer. In general, that feels like an implementation choice on
> top
> > > of
> > > > > the
> > > > > > event framework rather than something that needs to shape the
> core
> > > > > > architecture.
> > > > > >
> > > > > > 1.
> > https://lists.apache.org/thread/x9j8nscvy8hq61tyn01mj8yp6n9of0kp
> > > > > >
> > > > > > Thanks,
> > > > > > Yufei
> > > > > >
> > > > > >
> > > > > > On Thu, Jun 4, 2026 at 5:14 PM EJ Wang <
> > > ej.wang.dev.interest@gmail.com
> > > > >
> > > > > > wrote:
> > > > > >
> > > > > > > Good point. I agree the existing PolarisMetricsReporter is
> > already
> > > > very
> > > > > > > close to the right conceptual boundary. I am not proposing a
> > second
> > > > > > > parallel reporter concept. The distinction I am trying to make
> is
> > > > > > narrower:
> > > > > > >
> > > > > > > Current state:
> > > > > > >
> > > > > > >    - PolarisMetricsReporter is the existing metrics reporting
> > hook.
> > > > > > >    - But it currently lives under runtime/service.
> > > > > > >    - Its contract is documented around Quarkus/CDI discovery
> and
> > > > > > > @Identifier
> > > > > > >    selection.
> > > > > > >    - The default implementation is log-only.
> > > > > > >    - The durable implementation writes through
> > > PolarisMetricsManager
> > > > > and
> > > > > > >    MetricsPersistence.
> > > > > > >    - MetricsPersistence is currently inherited by
> > BasePersistence.
> > > > > > >
> > > > > > > My proposal:
> > > > > > >
> > > > > > >    - Keep the reporting/emitting boundary as the SPI.
> > > > > > >    - Revise that contract in a framework-agnostic optional
> > Iceberg
> > > > > > *metrics
> > > > > > >    extension API layer*.
> > > > > > >    - Keep runtime/service responsible for REST ingestion,
> > > table/authz
> > > > > > >    validation, and runtime wiring.
> > > > > > >    - Keep the battery implementation log-only in the metrics
> > > > extension
> > > > > > API
> > > > > > >    layer, *not under runtime/service*.
> > > > > > >    - Treat durable JDBC metrics as one implementation of the
> > > > reporting
> > > > > > SPI.
> > > > > > >    - Decompose the MetricsPersistence and BasePersistence
> > coupling
> > > as
> > > > > > >    durable implementation detail.
> > > > > > >
> > > > > > > So the difference is not “new reporter concept vs old reporter
> > > > > concept.”
> > > > > > > but:
> > > > > > >
> > > > > > >
> > > > > > >    - PolarisMetricsReporter today = runtime/service CDI-shaped
> > > hook.
> > > > > > >    - Target reporting SPI = same conceptual boundary, but
> placed
> > in
> > > > the
> > > > > > >    right extension API layer and kept framework-agnostic.
> > > > > > >
> > > > > > > For the current metrics PR, I think the minimal architectural
> > > cleanup
> > > > > > > before merge is to *put the reporting SPI contract and the
> > battery
> > > > > > default
> > > > > > > in the right place*. The durable JDBC implementation,
> > event-backed
> > > > > async
> > > > > > > implementation, external queue support, and deeper persistence
> > > > cleanup
> > > > > > can
> > > > > > > continue as follow-ups, as long as we do not lock the SPI
> > boundary
> > > to
> > > > > > > BasePersistence or to runtime/service wiring.
> > > > > > >
> > > > > > > -ej
> > > > > > >
> > > > > > > On Wed, Jun 3, 2026 at 3:00 PM Dmitri Bourlatchkov <
> > > dimas@apache.org
> > > > >
> > > > > > > wrote:
> > > > > > >
> > > > > > > > Hi EJ,
> > > > > > > >
> > > > > > > > Thanks for the recap / summary!
> > > > > > > >
> > > > > > > > > Do folks agree that the stable SPI boundary should be
> > > > > > > > metrics reporting/emitting, not metrics persistence?
> > > > > > > >
> > > > > > > > SGTM.
> > > > > > > >
> > > > > > > > > Does an optional Iceberg metrics extension API layer sound
> > like
> > > > the
> > > > > > > right
> > > > > > > > home for this SPI?
> > > > > > > >
> > > > > > > > How is that different from current PolarisMetricsReporter?
> > > > > > > >
> > > > > > > > > Should the current durable metrics work be reframed as a
> > > durable
> > > > > > > > JDBC reference implementation of that SPI?
> > > > > > > >
> > > > > > > > SGTM.
> > > > > > > >
> > > > > > > > > What is the smallest PR sequence to get there without
> > blocking
> > > > > > > > the current metrics work unnecessarily?
> > > > > > > >
> > > > > > > > Let's get https://github.com/apache/polaris/pull/4397 merged
> > > > first.
> > > > > > > >
> > > > > > > > Then, I'd propose isolating the Metrics schema from the
> > MetaStore
> > > > > > schema.
> > > > > > > > This will probably have some ripple effect into the bootstrap
> > > > > > workflows,
> > > > > > > so
> > > > > > > > it's not a trivial change.
> > > > > > > >
> > > > > > > > Then, let's reassess.
> > > > > > > >
> > > > > > > > Cheers,
> > > > > > > > Dmitri.
> > > > > > > >
> > > > > > > > On Wed, Jun 3, 2026 at 5:40 PM EJ Wang <
> > > > > ej.wang.dev.interest@gmail.com
> > > > > > >
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > > > Hi folks,
> > > > > > > > >
> > > > > > > > > Thanks again for the discussion today. I updated the sync
> doc
> > > > with
> > > > > > > notes
> > > > > > > > > from the third metrics architecture sync:
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://docs.google.com/document/d/100h7c4damrUzVuquYbBHM0EvA4LSWuW2IT2dN_7nYVA/edit?tab=t.k96s2xyqr5u1
> > > > > > > > >
> > > > > > > > > A few highlights from today’s discussion:
> > > > > > > > >
> > > > > > > > >    - We clarified that Iceberg metrics reporting can be
> > > > interpreted
> > > > > > > > either
> > > > > > > > >    as sync or async handling from Polaris’s perspective, so
> > > > Polaris
> > > > > > > > should
> > > > > > > > >    stay flexible as a platform instead of baking one
> handling
> > > > model
> > > > > > > into
> > > > > > > > > the
> > > > > > > > >    REST API behavior.
> > > > > > > > >    - We aligned that the built-in (battery) behavior for
> > > metrics
> > > > > > > emitting
> > > > > > > > >    can stay simple: no-op/log-only is enough as the
> default.
> > > > > > > > >    - Durable metrics persistence should be treated as an
> > > > > > implementation
> > > > > > > > of
> > > > > > > > >    the metrics reporting path, not as the core SPI
> boundary.
> > > > > > > > >    - The existing durable metrics work can be reviewed as a
> > > > > reference
> > > > > > > > >    implementation of the reporting SPI, with
> > > persistence-related
> > > > > > logic
> > > > > > > > kept
> > > > > > > > >    self-contained under a metrics durable implementation
> > module
> > > > > > rather
> > > > > > > > than
> > > > > > > > >    scattered through core entity persistence.
> > > > > > > > >    - Dashboard/insights remains a real use case, but we
> > agreed
> > > to
> > > > > > keep
> > > > > > > it
> > > > > > > > >    separate from the core metrics intake discussion for
> now.
> > > > > > > > >
> > > > > > > > > I also did a quick source check after the meeting to make
> > sure
> > > we
> > > > > are
> > > > > > > > > describing the current state accurately.
> > > > > > > > >
> > > > > > > > > Current state:
> > > > > > > > >
> > > > > > > > >    - Polaris already has a metrics reporting hook:
> > > > > > > > PolarisMetricsReporter.
> > > > > > > > >    - The default implementation is DefaultMetricsReporter,
> > > > selected
> > > > > > by
> > > > > > > > >    polaris.iceberg-metrics.reporting.type=default. It is
> > > > log-only,
> > > > > > > > >    effectively quiet unless metrics logging is enabled.
> > > > > > > > >    - There is also a PersistingMetricsReporter, selected by
> > > > > > > > >    polaris.iceberg-metrics.reporting.type=persisting, which
> > > > > converts
> > > > > > > > >    Iceberg scan/commit reports into Polaris metrics records
> > and
> > > > > > writes
> > > > > > > > > through
> > > > > > > > >    PolarisMetricsManager -> MetricsPersistence.
> > > > > > > > >    - MetricsPersistence currently exists in the persistence
> > > layer
> > > > > and
> > > > > > > is
> > > > > > > > >    inherited by BasePersistence.
> > > > > > > > >
> > > > > > > > > My read from the discussion is that the target boundary
> > should
> > > > not
> > > > > be
> > > > > > > > > MetricsPersistence as inherited by BasePersistence. That
> path
> > > > > should
> > > > > > be
> > > > > > > > > decomposed as durable implementation detail (taken care of
> by
> > > > > > > > > https://github.com/apache/polaris/pull/4397). The stable
> > > > extension
> > > > > > > point
> > > > > > > > > should instead be the metrics reporting/emitting boundary.
> > > > > > > > >
> > > > > > > > > Concretely, I think the next proposal should be shaped like
> > > this:
> > > > > > > > >
> > > > > > > > >    1.
> > > > > > > > >
> > > > > > > > >    Define the proper metrics reporting SPI at an optional
> > > Iceberg
> > > > > > > metrics
> > > > > > > > >    extension API layer.
> > > > > > > > >
> > > > > > > > >    Example direction:
> > > > > > > > >
> > > > > > > > >    public interface IcebergMetricsReporter {
> > > > > > > > >      void reportMetric(IcebergMetricsReportContext context,
> > > > > > > > > MetricsReport report);
> > > > > > > > >    }
> > > > > > > > >
> > > > > > > > >    The context would carry the small Polaris-resolved
> > envelope:
> > > > > > > > >    catalog/table identity, report type, received timestamp,
> > and
> > > > > > > > > request/trace
> > > > > > > > >    context if available. The raw Iceberg MetricsReport
> > remains
> > > > the
> > > > > > > > payload.
> > > > > > > > >    2.
> > > > > > > > >
> > > > > > > > >    Keep runtime/service as the REST ingestion and wiring
> > layer.
> > > > > > > > >
> > > > > > > > >    The REST handler still resolves the table and performs
> > authz
> > > > > > before
> > > > > > > > >    accepting the report. After that, it calls the selected
> > > > > reporting
> > > > > > > > >    implementation.
> > > > > > > > >    3.
> > > > > > > > >
> > > > > > > > >    Keep the battery default no-op/log-only.
> > > > > > > > >
> > > > > > > > >    This preserves an out-of-box safe default and avoids
> > > > requiring a
> > > > > > > > durable
> > > > > > > > >    metrics store for every Polaris deployment.
> > > > > > > > >    4.
> > > > > > > > >
> > > > > > > > >    Move durable JDBC metrics into a self-contained
> > > implementation
> > > > > of
> > > > > > > the
> > > > > > > > >    reporting SPI.
> > > > > > > > >
> > > > > > > > >    That implementation can own its schema, bootstrap,
> > > retention,
> > > > > and
> > > > > > > read
> > > > > > > > >    API support. It should not define the core reporting SPI
> > > > > boundary,
> > > > > > > and
> > > > > > > > > it
> > > > > > > > >    should not require metrics persistence to remain
> inherited
> > > > from
> > > > > > > > >    BasePersistence.
> > > > > > > > >    5.
> > > > > > > > >
> > > > > > > > >    Treat async/event-backed handling as another
> > implementation
> > > of
> > > > > the
> > > > > > > > same
> > > > > > > > >    reporting SPI.
> > > > > > > > >
> > > > > > > > >    For example, an event-backed reporter could enqueue a
> > > metrics
> > > > > > event
> > > > > > > > and
> > > > > > > > >    let listeners handle durable storage or other sinks. If
> we
> > > > later
> > > > > > > need
> > > > > > > > a
> > > > > > > > >    replaceable queue engine, that seems like a shared
> > > > event/metrics
> > > > > > > > > substrate
> > > > > > > > >    topic rather than a metrics-only requirement.
> > > > > > > > >
> > > > > > > > > This framing lets us keep the REST metrics endpoint simple,
> > > > > preserve
> > > > > > > the
> > > > > > > > > current default behavior, support durable metrics users,
> and
> > > > still
> > > > > > > leave
> > > > > > > > > room for async/event-backed or external-queue-based
> > > > > implementations.
> > > > > > > > >
> > > > > > > > > I think the main follow-up questions are:
> > > > > > > > >
> > > > > > > > >    - Do folks agree that the stable SPI boundary should be
> > > > metrics
> > > > > > > > >    reporting/emitting, not metrics persistence?
> > > > > > > > >    - Does an optional Iceberg metrics extension API layer
> > sound
> > > > > like
> > > > > > > the
> > > > > > > > >    right home for this SPI?
> > > > > > > > >    - Should the current durable metrics work be reframed
> as a
> > > > > durable
> > > > > > > > JDBC
> > > > > > > > >    reference implementation of that SPI?
> > > > > > > > >    - What is the smallest PR sequence to get there without
> > > > blocking
> > > > > > the
> > > > > > > > >    current metrics work unnecessarily?
> > > > > > > > >
> > > > > > > > > Thanks,
> > > > > > > > > -ej
> > > > > > > > >
> > > > > > > > > On Fri, May 15, 2026 at 5:16 PM Dmitri Bourlatchkov <
> > > > > > dimas@apache.org>
> > > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > Hi JB,
> > > > > > > > > >
> > > > > > > > > > Could you set up another meeting, please? Same time on
> > > > Wednesday
> > > > > as
> > > > > > > > last
> > > > > > > > > > time... I hope it works for everyone.
> > > > > > > > > >
> > > > > > > > > > Cheers,
> > > > > > > > > > Dmitri.
> > > > > > > > > >
> > > > > > > > > > On Fri, May 15, 2026 at 8:06 PM Yufei Gu <
> > > flyrain000@gmail.com
> > > > >
> > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > > +1 on another sync call next week.
> > > > > > > > > > >
> > > > > > > > > > > Yufei
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > On Fri, May 15, 2026 at 4:52 PM Dmitri Bourlatchkov <
> > > > > > > > dimas@apache.org>
> > > > > > > > > > > wrote:
> > > > > > > > > > >
> > > > > > > > > > > > Hi All,
> > > > > > > > > > > >
> > > > > > > > > > > > WDYT about another sync call next week?
> > > > > > > > > > > >
> > > > > > > > > > > > Thanks,
> > > > > > > > > > > > Dmitri.
> > > > > > > > > > > >
> > > > > > > > > > > > On Wed, May 6, 2026 at 5:29 PM Dmitri Bourlatchkov <
> > > > > > > > dimas@apache.org
> > > > > > > > > >
> > > > > > > > > > > > wrote:
> > > > > > > > > > > >
> > > > > > > > > > > > > Hi EJ,
> > > > > > > > > > > > >
> > > > > > > > > > > > > Thanks for the summary! It covers what we discussed
> > in
> > > > the
> > > > > > > > meeting
> > > > > > > > > > very
> > > > > > > > > > > > > well, IMHO.
> > > > > > > > > > > > >
> > > > > > > > > > > > > Looking forward to concrete PRs :)
> > > > > > > > > > > > >
> > > > > > > > > > > > > Cheers,
> > > > > > > > > > > > > Dmitri.
> > > > > > > > > > > > >
> > > > > > > > > > > > > On Wed, May 6, 2026 at 5:08 PM EJ Wang <
> > > > > > > > > > ej.wang.dev.interest@gmail.com
> > > > > > > > > > > >
> > > > > > > > > > > > > wrote:
> > > > > > > > > > > > >
> > > > > > > > > > > > >> Hi folks,
> > > > > > > > > > > > >>
> > > > > > > > > > > > >> We had a community sync earlier, thanks JB for
> > > > scheduling
> > > > > > it.
> > > > > > > > > Notes
> > > > > > > > > > > from
> > > > > > > > > > > > >> the first metrics architecture sync (May 6,
> 10-11am
> > > PT).
> > > > > > > > > Discussion
> > > > > > > > > > > doc
> > > > > > > > > > > > >> with per-section status:
> > > > > > > > > > > > >>
> > > > > > > > > > > > >>
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://docs.google.com/document/d/100h7c4damrUzVuquYbBHM0EvA4LSWuW2IT2dN_7nYVA/edit?tab=t.0
> > > > > > > > > > > > >>
> > > > > > > > > > > > >> *The meeting covered both topics from the doc.
> > > > > > Direction-level
> > > > > > > > > > > alignment
> > > > > > > > > > > > >> was reached on the headline pieces; details remain
> > for
> > > > PR
> > > > > > > review
> > > > > > > > > or
> > > > > > > > > > > > >> follow-up sessions.*
> > > > > > > > > > > > >>
> > > > > > > > > > > > >> *Topic 1 — Persistence schema redesign*
> > > > > > > > > > > > >> Idea-level alignment on consolidating per-type
> > tables
> > > > > > > > > > > > >> (scan_metrics_report,
> > > > > > > > > > > > >> commit_metrics_report) into a single
> metrics_report
> > > > table.
> > > > > > The
> > > > > > > > > > > > motivating
> > > > > > > > > > > > >> cost is the surface area added by every new metric
> > > type
> > > > > > today:
> > > > > > > > new
> > > > > > > > > > > > table,
> > > > > > > > > > > > >> SPI method, record class, model, converter, schema
> > > > > > migration.
> > > > > > > > > > > > >>
> > > > > > > > > > > > >> Most schema details are deferred to the schema
> PR. A
> > > few
> > > > > > > > specific
> > > > > > > > > > > points
> > > > > > > > > > > > >> came up:
> > > > > > > > > > > > >> •   metric_schema_version: Yufei prefers dropping
> > it,
> > > > > since
> > > > > > > > there
> > > > > > > > > is
> > > > > > > > > > > no
> > > > > > > > > > > > >> spec-level concept of metrics versioning today and
> > it
> > > is
> > > > > > hard
> > > > > > > to
> > > > > > > > > > > define
> > > > > > > > > > > > >> unilaterally. Robert prefers keeping it, given IRC
> > v2
> > > is
> > > > > > > coming
> > > > > > > > > and
> > > > > > > > > > > the
> > > > > > > > > > > > >> schema should be considered against its likely
> > shape;
> > > > > Robert
> > > > > > > > also
> > > > > > > > > > > raised
> > > > > > > > > > > > >> how to differentiate various payload formats if
> any.
> > > > EJ's
> > > > > > read
> > > > > > > > is
> > > > > > > > > > that
> > > > > > > > > > > > >> this
> > > > > > > > > > > > >> is a two-way-door decision. We can start without
> the
> > > > > field,
> > > > > > > and
> > > > > > > > if
> > > > > > > > > > IRC
> > > > > > > > > > > > v2
> > > > > > > > > > > > >> changes the shape we would likely roll a
> > corresponding
> > > > new
> > > > > > > > schema
> > > > > > > > > > > > anyway,
> > > > > > > > > > > > >> which is not particularly costly.
> > > > > > > > > > > > >> •   Payload format: Robert pointed out that future
> > > > formats
> > > > > > > > beyond
> > > > > > > > > > JSON
> > > > > > > > > > > > may
> > > > > > > > > > > > >> be worth supporting. The exact shape is deferred
> to
> > > the
> > > > > > schema
> > > > > > > > > > > > discussion.
> > > > > > > > > > > > >> •   Partition strategy: Anand suggested monthly
> > > > > partitioning
> > > > > > > > based
> > > > > > > > > > on
> > > > > > > > > > > > his
> > > > > > > > > > > > >> experience as potentially helpful at scale.
> > > > > > > > > > > > >>
> > > > > > > > > > > > >> *Topic 2 — Where metrics ingestion and storage
> > belong*
> > > > > > > > > > > > >> Idea-level alignment that metrics should be a
> > > separated
> > > > > SPI
> > > > > > > from
> > > > > > > > > the
> > > > > > > > > > > > >> entity
> > > > > > > > > > > > >> persistence stack. Two reasons surfaced: (a)
> > workloads
> > > > and
> > > > > > > > > > capability
> > > > > > > > > > > > >> requirements diverge enough that coupling them
> > creates
> > > > > > > > artificial
> > > > > > > > > > > > >> constraints, and (b) admin experience improves
> when
> > > > > metrics
> > > > > > > has
> > > > > > > > > its
> > > > > > > > > > > own
> > > > > > > > > > > > >> bootstrap, retention, and lifecycle. Dmitri noted
> > > > Polaris
> > > > > > > being
> > > > > > > > a
> > > > > > > > > > > > platform
> > > > > > > > > > > > >> should have the flexibility to support different
> > > > > persistence
> > > > > > > > > > backends
> > > > > > > > > > > > per
> > > > > > > > > > > > >> concern, and pointed to a concrete next step of
> > > > separating
> > > > > > the
> > > > > > > > > JDBC
> > > > > > > > > > > > >> bootstrap for metrics from the metastore
> bootstrap.
> > > > Robert
> > > > > > > > > proposed
> > > > > > > > > > an
> > > > > > > > > > > > >> additional UX extension: detect an unbootstrapped
> > > > metrics
> > > > > > > store
> > > > > > > > on
> > > > > > > > > > > first
> > > > > > > > > > > > >> use and auto-bootstrap rather than requiring an
> > > explicit
> > > > > > > manual
> > > > > > > > > > > > bootstrap
> > > > > > > > > > > > >> step.
> > > > > > > > > > > > >> The meeting also confirmed that Polaris metrics
> can
> > > > start
> > > > > > > small
> > > > > > > > > and
> > > > > > > > > > > stay
> > > > > > > > > > > > >> Iceberg-focused. Naming and persistence schema can
> > > lean
> > > > > > > > > > > > Iceberg-specific.
> > > > > > > > > > > > >> If a future expansion to generic-table metrics or
> > > > > > operational
> > > > > > > > > > metrics
> > > > > > > > > > > > >> arrives, an abstraction layer can be built on top
> of
> > > the
> > > > > > > Iceberg
> > > > > > > > > > > metrics
> > > > > > > > > > > > >> reporter at that point. Robert remains on the
> fence
> > > and
> > > > > > would
> > > > > > > > > prefer
> > > > > > > > > > > > >> something more generic but did not block the
> > > direction;
> > > > > > > Dmitri's
> > > > > > > > > > read
> > > > > > > > > > > > was
> > > > > > > > > > > > >> that the proposed framework already has enough
> > > > flexibility
> > > > > > to
> > > > > > > > > absorb
> > > > > > > > > > > > >> future
> > > > > > > > > > > > >> expansion.
> > > > > > > > > > > > >>
> > > > > > > > > > > > >> The Trade-offs and Proposed structure sections in
> > the
> > > > doc
> > > > > > were
> > > > > > > > not
> > > > > > > > > > > > >> reviewed
> > > > > > > > > > > > >> in detail. They remain open for either the next
> sync
> > > or
> > > > PR
> > > > > > > > review.
> > > > > > > > > > > > >>
> > > > > > > > > > > > >> *Cross-cutting alignment — battery-included plus
> > > > > pluggable*
> > > > > > > > > > > > >> A common philosophy emerged from the discussion.
> EJ
> > > > > > summarized
> > > > > > > > it
> > > > > > > > > > as:
> > > > > > > > > > > > >> Polaris should provide a battery-included UX for
> > > > beginners
> > > > > > and
> > > > > > > > the
> > > > > > > > > > > > >> flexibility for advanced users to swap the
> included
> > > > > battery
> > > > > > > for
> > > > > > > > > > > > something
> > > > > > > > > > > > >> more powerful or tailored to their use case. The
> SPI
> > > > > design
> > > > > > > > needs
> > > > > > > > > to
> > > > > > > > > > > > >> enable
> > > > > > > > > > > > >> both.
> > > > > > > > > > > > >>
> > > > > > > > > > > > >> The inputs that shaped this framing:
> > > > > > > > > > > > >> •   Anand described how his team uses the current
> > > > metrics
> > > > > > > > > > persistence
> > > > > > > > > > > > >> (three metrics consumers in v1.4).
> > > > > > > > > > > > >> •   Yufei raised Grafana and dashboard
> integrations
> > > as a
> > > > > > > > > destination
> > > > > > > > > > > use
> > > > > > > > > > > > >> case beyond the default.
> > > > > > > > > > > > >> •   Robert called out that the current design is
> > more
> > > > > > > > > JDBC-focused.
> > > > > > > > > > > > >>
> > > > > > > > > > > > >> Two concrete instances:
> > > > > > > > > > > > >> •   Async metrics intake: Yufei's initial position
> > was
> > > > > that
> > > > > > > > async
> > > > > > > > > > > should
> > > > > > > > > > > > >> largely live on the producer side and there is not
> > > much
> > > > > > > Polaris
> > > > > > > > > can
> > > > > > > > > > > do.
> > > > > > > > > > > > >> Robert suggested a Polaris-side default is doable
> > via
> > > > > > Vert.x.
> > > > > > > > > Dmitri
> > > > > > > > > > > > >> agreed
> > > > > > > > > > > > >> the direction is worth exploring. The meeting
> > > converged
> > > > > on a
> > > > > > > > > > > > >> battery-included default (likely Vert.x-backed)
> with
> > > an
> > > > > SPI
> > > > > > > > shape
> > > > > > > > > > that
> > > > > > > > > > > > >> lets
> > > > > > > > > > > > >> power users route to a more scalable backend
> > > (k8s-hosted
> > > > > > > queue,
> > > > > > > > > AWS
> > > > > > > > > > > SQS,
> > > > > > > > > > > > >> etc.).
> > > > > > > > > > > > >> •   Pluggable destinations: combining Yufei's
> > > dashboard
> > > > > use
> > > > > > > case
> > > > > > > > > > with
> > > > > > > > > > > > >> Robert's JDBC-focused call-out, the meeting agreed
> > the
> > > > SPI
> > > > > > > > should
> > > > > > > > > be
> > > > > > > > > > > > >> structured for multiple sinks so integrations
> become
> > > > impl
> > > > > > > > choices
> > > > > > > > > > > rather
> > > > > > > > > > > > >> than architectural changes.
> > > > > > > > > > > > >>
> > > > > > > > > > > > >> The battery-included default is most likely to use
> > the
> > > > > > > existing
> > > > > > > > > > > > >> JDBC-backed
> > > > > > > > > > > > >> approach.
> > > > > > > > > > > > >>
> > > > > > > > > > > > >> *Direction (idea-level alignment)*
> > > > > > > > > > > > >> •   Single metrics_report table consolidating
> > per-type
> > > > > > > metrics,
> > > > > > > > > > > > replacing
> > > > > > > > > > > > >> scan_metrics_report and commit_metrics_report
> > > > > > > > > > > > >> •   Iceberg-focused naming and schema for now,
> > revisit
> > > > if
> > > > > > > > > > > generic-table
> > > > > > > > > > > > or
> > > > > > > > > > > > >> operational metrics arrive
> > > > > > > > > > > > >> •   Metrics persistence as a separated SPI, not on
> > > > > > > > BasePersistence
> > > > > > > > > > > > >> •   Bootstrap path separated for metrics,
> > independent
> > > of
> > > > > > > > metastore
> > > > > > > > > > > > >> bootstrap
> > > > > > > > > > > > >> •   "Battery-included plus pluggable" as the SPI
> > > design
> > > > > > > > philosophy
> > > > > > > > > > > > >>
> > > > > > > > > > > > >> *Open items*
> > > > > > > > > > > > >> •   Schema details: metric_schema_version, payload
> > > > format,
> > > > > > IRC
> > > > > > > > v2
> > > > > > > > > > > > >> forward-compat shape
> > > > > > > > > > > > >> •   SPI design details — full review either in the
> > > next
> > > > > sync
> > > > > > > or
> > > > > > > > in
> > > > > > > > > > the
> > > > > > > > > > > > >> corresponding PR
> > > > > > > > > > > > >> •   Schema refactor PR ownership
> > > > > > > > > > > > >>
> > > > > > > > > > > > >> *Action items*
> > > > > > > > > > > > >> •   EJ to take a first stab at the SPI design and
> > > > > > potentially
> > > > > > > > > > partner
> > > > > > > > > > > > with
> > > > > > > > > > > > >> Anand to incorporate the lessons learned from the
> > > > existing
> > > > > > > > > reporter
> > > > > > > > > > > and
> > > > > > > > > > > > >> persistence work.
> > > > > > > > > > > > >> •   Schema refactor PR ownership is not yet
> decided.
> > > If
> > > > > > anyone
> > > > > > > > is
> > > > > > > > > > > > >> interested in driving it, reply on this thread.
> > > > > > > > > > > > >> •   JB to schedule the next sync, tentatively in
> two
> > > > > weeks.
> > > > > > > > > > > > >>
> > > > > > > > > > > > >> -ej
> > > > > > > > > > > > >>
> > > > > > > > > > > > >> On Mon, Apr 27, 2026 at 3:07 PM EJ Wang <
> > > > > > > > > > > ej.wang.dev.interest@gmail.com
> > > > > > > > > > > > >
> > > > > > > > > > > > >> wrote:
> > > > > > > > > > > > >>
> > > > > > > > > > > > >> > Thanks Yufei for the +1.
> > > > > > > > > > > > >> >
> > > > > > > > > > > > >> > JB, could you help add a biweekly metrics
> > > architecture
> > > > > > sync
> > > > > > > to
> > > > > > > > > the
> > > > > > > > > > > > >> Polaris
> > > > > > > > > > > > >> > community calendar? I'm thinking Thursdays at
> > 9-10am
> > > > PT,
> > > > > > on
> > > > > > > > the
> > > > > > > > > > > > >> off-weeks
> > > > > > > > > > > > >> > from the community meeting (starting May 7), 60
> > > > minutes.
> > > > > > > > > > > > >> >
> > > > > > > > > > > > >> > Here's a rough agenda to work through over the
> > first
> > > > few
> > > > > > > > > sessions,
> > > > > > > > > > > > >> grouped
> > > > > > > > > > > > >> > by priority:
> > > > > > > > > > > > >> >
> > > > > > > > > > > > >> > *First: foundational direction*
> > > > > > > > > > > > >> >
> > > > > > > > > > > > >> > 1.  MetricsPersistence: public SPI or internal
> > > > > > > implementation
> > > > > > > > > > > detail?
> > > > > > > > > > > > >> >    •   Marked @Beta, javadoc calls it a "Service
> > > > > Provider
> > > > > > > > > > > Interface",
> > > > > > > > > > > > >> but
> > > > > > > > > > > > >> > only one consumer (JdbcBasePersistenceImpl),
> lives
> > > on
> > > > > > > > > > > BasePersistence.
> > > > > > > > > > > > >> If
> > > > > > > > > > > > >> > demoted to a private helper inside a persisting
> > > > reporter
> > > > > > > impl,
> > > > > > > > > > most
> > > > > > > > > > > > >> > downstream design decisions become
> implementation
> > > > > details
> > > > > > > > rather
> > > > > > > > > > > than
> > > > > > > > > > > > >> > contract questions.
> > > > > > > > > > > > >> >
> > > > > > > > > > > > >> > 2.  Persistence schema redesign
> > > > > > > > > > > > >> >    •   Current two-table layout
> > > (scan_metrics_report,
> > > > > > > > > > > > >> > commit_metrics_report) with ~25 flattened
> columns
> > > > each.
> > > > > > > Every
> > > > > > > > > new
> > > > > > > > > > > > metric
> > > > > > > > > > > > >> > type requires a new table, SPI method, record
> > class,
> > > > > > model,
> > > > > > > > > > > converter,
> > > > > > > > > > > > >> and
> > > > > > > > > > > > >> > schema migration. Direction to explore: single
> > table
> > > > > with
> > > > > > > > > > > metric_type
> > > > > > > > > > > > >> enum,
> > > > > > > > > > > > >> > schema_version, and JSON payload column.
> > > > > > > > > > > > >> >
> > > > > > > > > > > > >> > *Second: design details once direction is set*
> > > > > > > > > > > > >> >
> > > > > > > > > > > > >> > 3.  Partition key strategy
> > > > > > > > > > > > >> >    •   Single-table design means scan metrics at
> > > scale
> > > > > > will
> > > > > > > > have
> > > > > > > > > > > high
> > > > > > > > > > > > >> > write concurrency per table. Schema needs to
> > expose
> > > > > enough
> > > > > > > > > > structure
> > > > > > > > > > > > for
> > > > > > > > > > > > >> > backends to shard by entity or time range.
> > > > > > > > > > > > >> >
> > > > > > > > > > > > >> > 4.  Read/write path consistency
> > > > > > > > > > > > >> >    •   Writes go through PolarisMetricsManager
> on
> > > > > > > > > > MetaStoreManager.
> > > > > > > > > > > > >> Reads
> > > > > > > > > > > > >> > bypass MetaStoreManager and go straight to
> > > > > > BasePersistence,
> > > > > > > > > > > excluding
> > > > > > > > > > > > >> > non-JDBC backends from the read API.
> > > > > > > > > > > > >> >
> > > > > > > > > > > > >> > *Third: cleanup and alignment*
> > > > > > > > > > > > >> >
> > > > > > > > > > > > >> > 5.  PolarisMetricsReporter naming
> > > > > > > > > > > > >> >    •   Only handles IRC
> (ScanReport/CommitReport),
> > > > > doesn't
> > > > > > > > cover
> > > > > > > > > > > > generic
> > > > > > > > > > > > >> > tables or operational metrics. Name is broader
> > than
> > > > > scope.
> > > > > > > > > > > > >> >
> > > > > > > > > > > > >> > 6.  PolarisMetricsManager facade passthrough
> > > > > > > > > > > > >> >    •   Entire default method is
> > > > > > > > > > > > >> callCtx.getMetaStore().writeScanReport().
> > > > > > > > > > > > >> > Zero logic, passes Level 1 straight through to
> > Level
> > > > 3.
> > > > > > Same
> > > > > > > > > > > > >> anti-pattern
> > > > > > > > > > > > >> > as PolarisEventManager.
> > > > > > > > > > > > >> >
> > > > > > > > > > > > >> > 7.  Iceberg community alignment
> > > > > > > > > > > > >> >    •   Payload-type extension needs discussion
> on
> > > > > > > dev@iceberg.
> > > > > > > > > > > > >> obelix74's
> > > > > > > > > > > > >> > Feb thread got zero replies. Needs a committer
> > > voice.
> > > > > > > > > > > > >> >
> > > > > > > > > > > > >> > Lets confirm prioritization in the first
> session.
> > > > > > > > > > > > >> >
> > > > > > > > > > > > >> > -ej
> > > > > > > > > > > > >> >
> > > > > > > > > > > > >> > On Tue, Apr 21, 2026 at 3:18 PM Yufei Gu <
> > > > > > > > flyrain000@gmail.com>
> > > > > > > > > > > > wrote:
> > > > > > > > > > > > >> >
> > > > > > > > > > > > >> >> Thanks everyone for continuing to drive this
> > > > forward. I
> > > > > > > agree
> > > > > > > > > > that
> > > > > > > > > > > > the
> > > > > > > > > > > > >> >> problem is getting complex enough that a more
> > > > > structured
> > > > > > > > > > discussion
> > > > > > > > > > > > >> would
> > > > > > > > > > > > >> >> help.
> > > > > > > > > > > > >> >>
> > > > > > > > > > > > >> >> +1 on setting up a biweekly sync for the
> metrics
> > > > > > > > architecture.
> > > > > > > > > > I’m
> > > > > > > > > > > > >> happy
> > > > > > > > > > > > >> >> to
> > > > > > > > > > > > >> >> join.
> > > > > > > > > > > > >> >>
> > > > > > > > > > > > >> >> Yufei
> > > > > > > > > > > > >> >>
> > > > > > > > > > > > >> >>
> > > > > > > > > > > > >> >> On Tue, Apr 21, 2026 at 2:34 PM EJ Wang <
> > > > > > > > > > > > >> ej.wang.dev.interest@gmail.com>
> > > > > > > > > > > > >> >> wrote:
> > > > > > > > > > > > >> >>
> > > > > > > > > > > > >> >> > Also, I've been looking more closely at the
> > > > > > *persistence
> > > > > > > > > schema
> > > > > > > > > > > in
> > > > > > > > > > > > >> the
> > > > > > > > > > > > >> >> > current metrics work*, and I think there's a
> > > > > structural
> > > > > > > > > > rigidity
> > > > > > > > > > > > >> problem
> > > > > > > > > > > > >> >> > worth raising before the shape gets locked
> in.
> > > > > > > > > > > > >> >> >
> > > > > > > > > > > > >> >> > Right now we have two separate tables
> > > > > > > (scan_metrics_report
> > > > > > > > > and
> > > > > > > > > > > > >> >> > commit_metrics_report), each with ~25
> flattened
> > > > > columns
> > > > > > > > that
> > > > > > > > > > > > directly
> > > > > > > > > > > > >> >> > mirror the Iceberg report fields. The SPI
> > follows
> > > > the
> > > > > > > same
> > > > > > > > > > split:
> > > > > > > > > > > > >> >> > writeScanReport and writeCommitReport as
> > separate
> > > > > > > methods,
> > > > > > > > > with
> > > > > > > > > > > > >> per-type
> > > > > > > > > > > > >> >> > record classes, converters, and model
> objects.
> > > *The
> > > > > > > > practical
> > > > > > > > > > > cost:
> > > > > > > > > > > > >> >> > adding a new metric type (operational
> metrics,
> > > for
> > > > > > > example)
> > > > > > > > > > > > requires
> > > > > > > > > > > > >> a
> > > > > > > > > > > > >> >> new
> > > > > > > > > > > > >> >> > table, a new SPI method, a new record class,
> a
> > > new
> > > > > > model
> > > > > > > > > > class, a
> > > > > > > > > > > > new
> > > > > > > > > > > > >> >> > converter branch, and a schema migration*.
> > > That's a
> > > > > lot
> > > > > > > of
> > > > > > > > > > > surface
> > > > > > > > > > > > >> area
> > > > > > > > > > > > >> >> > for what should be "one more kind of metric."
> > > > > > > > > > > > >> >> >
> > > > > > > > > > > > >> >> > *My bias* would be toward a single metrics
> > table
> > > > with
> > > > > > *a
> > > > > > > > > typed
> > > > > > > > > > > JSON
> > > > > > > > > > > > >> >> > payload*. Something like: metric_type (enum),
> > > > > > entity_id,
> > > > > > > > > > > > >> >> > table_identifier, snapshot_id (nullable),
> > > > > received_ts,
> > > > > > > > > > > > >> schema_version,
> > > > > > > > > > > > >> >> and
> > > > > > > > > > > > >> >> > a payload column for the metric-specific
> data.
> > > The
> > > > > > > > > metric_type
> > > > > > > > > > +
> > > > > > > > > > > > >> >> > schema_version pair gives us a
> > forward-compatible
> > > > > > > contract
> > > > > > > > > for
> > > > > > > > > > > the
> > > > > > > > > > > > >> >> payload
> > > > > > > > > > > > >> >> > shape. Adding a new metric type becomes an
> enum
> > > > value
> > > > > > > and a
> > > > > > > > > > > payload
> > > > > > > > > > > > >> >> schema,
> > > > > > > > > > > > >> >> > not a schema migration. One thing I think we
> > need
> > > > to
> > > > > be
> > > > > > > > > > > deliberate
> > > > > > > > > > > > >> >> about is
> > > > > > > > > > > > >> >> > the partition key design. If all metric types
> > > land
> > > > in
> > > > > > one
> > > > > > > > > > table,
> > > > > > > > > > > > scan
> > > > > > > > > > > > >> >> > metrics at scale (high concurrency, high
> > > frequency
> > > > > > across
> > > > > > > > > many
> > > > > > > > > > > > >> tables)
> > > > > > > > > > > > >> >> > could easily create hot partitions. We'd want
> > the
> > > > > > > > persistence
> > > > > > > > > > > layer
> > > > > > > > > > > > >> to
> > > > > > > > > > > > >> >> be
> > > > > > > > > > > > >> >> > able to shard by entity or time range, and
> that
> > > > means
> > > > > > the
> > > > > > > > > > logical
> > > > > > > > > > > > >> schema
> > > > > > > > > > > > >> >> > needs to expose enough structure for backends
> > to
> > > > > > > partition
> > > > > > > > > on.
> > > > > > > > > > I
> > > > > > > > > > > > >> don't
> > > > > > > > > > > > >> >> > think the current flattened layout gives us
> > that.
> > > > > > > > > > > > >> >> >
> > > > > > > > > > > > >> >> > This is getting complex enough that I don't
> > think
> > > > > > ad-hoc
> > > > > > > > > PR/ML
> > > > > > > > > > > > >> threads
> > > > > > > > > > > > >> >> > will converge well. *Would people be open to
> a
> > > > > biweekly
> > > > > > > > sync
> > > > > > > > > > for
> > > > > > > > > > > > >> metrics
> > > > > > > > > > > > >> >> > architecture?* I think 30 minutes every two
> > weeks
> > > > > with
> > > > > > > > > > interested
> > > > > > > > > > > > >> >> parties
> > > > > > > > > > > > >> >> > would be enough to work through the schema,
> SPI
> > > > > shape,
> > > > > > > and
> > > > > > > > > read
> > > > > > > > > > > API
> > > > > > > > > > > > >> >> design
> > > > > > > > > > > > >> >> > together. Happy to help set that up.
> > > > > > > > > > > > >> >> >
> > > > > > > > > > > > >> >> > -ej
> > > > > > > > > > > > >> >> >
> > > > > > > > > > > > >> >> > On Mon, Apr 20, 2026 at 2:19 PM EJ Wang <
> > > > > > > > > > > > >> ej.wang.dev.interest@gmail.com
> > > > > > > > > > > > >> >> >
> > > > > > > > > > > > >> >> > wrote:
> > > > > > > > > > > > >> >> >
> > > > > > > > > > > > >> >> >> Reviewed #4115, left a comment on the code
> > > > > > organization
> > > > > > > > > side.
> > > > > > > > > > > > >> >> >>
> > > > > > > > > > > > >> >> >> One thing stood out: the metrics write path
> > > enters
> > > > > > > through
> > > > > > > > > > > > >> >> >> PolarisMetricsManager on MetaStoreManager,
> but
> > > the
> > > > > new
> > > > > > > > read
> > > > > > > > > > path
> > > > > > > > > > > > >> >> bypasses
> > > > > > > > > > > > >> >> >> MetaStoreManager entirely and goes straight
> to
> > > > > > > > > BasePersistence
> > > > > > > > > > > via
> > > > > > > > > > > > >> >> >> callContext.getMetaStore(). That means the
> > read
> > > > API
> > > > > > only
> > > > > > > > > works
> > > > > > > > > > > for
> > > > > > > > > > > > >> >> backends
> > > > > > > > > > > > >> >> >> that implement BasePersistence. NoSQL and
> > remote
> > > > > > > backends
> > > > > > > > > > can't
> > > > > > > > > > > > >> >> participate.
> > > > > > > > > > > > >> >> >>
> > > > > > > > > > > > >> >> >> Stepping back, I think the metrics subsystem
> > is
> > > > > > growing
> > > > > > > > into
> > > > > > > > > > > > >> something
> > > > > > > > > > > > >> >> >> real (write + read + REST API + AuthZ +
> > > > pagination)
> > > > > > *but
> > > > > > > > the
> > > > > > > > > > > > >> >> persistence
> > > > > > > > > > > > >> >> >> side is split across two layers in a way
> > that's
> > > > hard
> > > > > > to
> > > > > > > > > > > extend*. I
> > > > > > > > > > > > >> put
> > > > > > > > > > > > >> >> >> together two diagrams to show what I mean
> (my
> > > best
> > > > > > > > effort).
> > > > > > > > > > > > >> >> >>
> > > > > > > > > > > > >> >> >> *Current state* (Diagram 1): three
> interfaces
> > at
> > > > > three
> > > > > > > > > > different
> > > > > > > > > > > > >> >> levels.
> > > > > > > > > > > > >> >> >> The engine-facing SPI
> (PolarisMetricsReporter)
> > > is
> > > > > > clean.
> > > > > > > > But
> > > > > > > > > > > > >> >> >> PolarisMetricsManager on MetaStoreManager
> is a
> > > > > > > passthrough
> > > > > > > > > to
> > > > > > > > > > > > >> >> >> MetricsPersistence on BasePersistence. The
> > @Beta
> > > > > > > > annotation
> > > > > > > > > > and
> > > > > > > > > > > > SPI
> > > > > > > > > > > > >> >> javadoc
> > > > > > > > > > > > >> >> >> are on the BasePersistence layer, while the
> > > actual
> > > > > > > > extension
> > > > > > > > > > > > points
> > > > > > > > > > > > >> >> >> (PolarisMetricsReporter,
> > PolarisMetricsManager)
> > > > > carry
> > > > > > no
> > > > > > > > > > > stability
> > > > > > > > > > > > >> >> >> annotation. The write path goes through the
> > > > > > > > MetaStoreManager
> > > > > > > > > > > > layer,
> > > > > > > > > > > > >> the
> > > > > > > > > > > > >> >> >> read path doesn't.
> > > > > > > > > > > > >> >> >>
> > > > > > > > > > > > >> >> >> *What I envision* (Diagram 2): two SPIs at
> two
> > > > > levels.
> > > > > > > > > > > > >> >> >> PolarisMetricsReporter stays as the
> > > engine-facing
> > > > > SPI.
> > > > > > > > > > > > >> >> >> PolarisMetricsManager becomes the
> > backend-facing
> > > > SPI
> > > > > > > with
> > > > > > > > > both
> > > > > > > > > > > > write
> > > > > > > > > > > > >> >> and
> > > > > > > > > > > > >> >> >> read methods at the MetaStoreManager level,
> > > where
> > > > > any
> > > > > > > > > backend
> > > > > > > > > > > > (JDBC,
> > > > > > > > > > > > >> >> NoSQL,
> > > > > > > > > > > > >> >> >> remote) can implement them.
> MetricsPersistence
> > > on
> > > > > > > > > > > BasePersistence
> > > > > > > > > > > > >> goes
> > > > > > > > > > > > >> >> >> away. Where metrics actually land is an
> > > > > implementation
> > > > > > > > > detail,
> > > > > > > > > > > > not a
> > > > > > > > > > > > >> >> core
> > > > > > > > > > > > >> >> >> interface.
> > > > > > > > > > > > >> >> >>
> > > > > > > > > > > > >> >> >> *Minor naming thing*: PolarisMetricsReporter
> > is
> > > > > > broader
> > > > > > > > than
> > > > > > > > > > > what
> > > > > > > > > > > > it
> > > > > > > > > > > > >> >> >> actually handles. It only accepts Iceberg
> REST
> > > > > Catalog
> > > > > > > > > metrics
> > > > > > > > > > > > >> >> (ScanReport,
> > > > > > > > > > > > >> >> >> CommitReport via MetricsReport). Generic
> table
> > > > > metrics
> > > > > > > or
> > > > > > > > > > > > >> operational
> > > > > > > > > > > > >> >> >> metrics aren't in scope. Not blocking, but
> > worth
> > > > > > noting
> > > > > > > if
> > > > > > > > > the
> > > > > > > > > > > > >> metrics
> > > > > > > > > > > > >> >> >> surface expands.
> > > > > > > > > > > > >> >> >>
> > > > > > > > > > > > >> >> >> *Rough sketch of how to get there*:
> > > > > > > > > > > > >> >> >>  1.  Add read methods to
> PolarisMetricsManager
> > > > > > > > > > (listScanReports,
> > > > > > > > > > > > >> >> >> listCommitReports) with default no-op, same
> as
> > > the
> > > > > > > > existing
> > > > > > > > > > > write
> > > > > > > > > > > > >> >> methods.
> > > > > > > > > > > > >> >> >> (Probably make PolarisMetricsManager more
> > > explicit
> > > > > on
> > > > > > > > being
> > > > > > > > > > > > Iceberg
> > > > > > > > > > > > >> >> >> specific like package name or class name
> etc.)
> > > > > > > > > > > > >> >> >>  2.  Wire MetricsReportsService through
> > > > > > MetaStoreManager
> > > > > > > > > > instead
> > > > > > > > > > > > of
> > > > > > > > > > > > >> >> >> callContext.getMetaStore().
> > > > > > > > > > > > >> >> >>  3.  Extract metrics persistence from
> > > > > > > > > JdbcBasePersistenceImpl
> > > > > > > > > > > into
> > > > > > > > > > > > >> its
> > > > > > > > > > > > >> >> >> own class. That file carries ~7
> > > responsibilities,
> > > > > > > metrics
> > > > > > > > > > being
> > > > > > > > > > > > one
> > > > > > > > > > > > >> of
> > > > > > > > > > > > >> >> them.
> > > > > > > > > > > > >> >> >>  4.  Remove MetricsPersistence from
> > > > BasePersistence.
> > > > > > > > > > > > >> >> >>
> > > > > > > > > > > > >> >> >> *None of this needs to happen in #4115. But
> if
> > > the
> > > > > > > > direction
> > > > > > > > > > > makes
> > > > > > > > > > > > >> >> sense,
> > > > > > > > > > > > >> >> >> it would be good to align before the metrics
> > > > surface
> > > > > > > grows
> > > > > > > > > > > > further.
> > > > > > > > > > > > >> >> Curious
> > > > > > > > > > > > >> >> >> what others think.*
> > > > > > > > > > > > >> >> >>
> > > > > > > > > > > > >> >> >> *My mental model note*: Level 1
> > > MetaStoreManager;
> > > > > > level
> > > > > > > 2
> > > > > > > > > > > > >> transactional
> > > > > > > > > > > > >> >> >> persistence; level 3 base persistence
> > > > > > > > > > > > >> >> >>
> > > > > > > > > > > > >> >> >> Diagram 1
> > > > > > > > > > > > >> >> >> <
> > > > > > > > > > > > >> >>
> > > > > > > > > > > > >>
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://www.plantuml.com/plantuml/uml/bLHDR-Cs4BthLmpIYupw0zbkKQ1r3M-S7Bp8xhhM7WCOb3IM65EaGD9EX2RzxHrHb4CxRelwa4YSDu_lpOVcnZ9jzvM8BBS2uGjQpJC3dtHMSekPtMk44IpsMgEqa5XcCOhCZikQQLP1pR8TAp2n3ILhmZDP20m0fcIvUkAoW2qJXd9z1bpToO9BX3WXu0ucy5rpgGPNm0nW5_epUWtm2Ue3pn3kMOFQmKntGZW0BYtgBSi8k5A2QMwybJNMIbFiGSR9QZc4nUqIvikStF0jHprua5C-amge42aNt3R0f5JaaoivdV2Pkqbx4hee4ymOkBh5BTiB-_uIeGeo8zL8rPsPl4DktdEiK1jkB1NdZCRbrSTecDe_mlHbF0wvBmCkaOH5_S8a_TTTKI6-nmCAkEw4LpxsZ-LbYLKQFKMNOgf_wuM7_bV9gOer5SYMMksBSWXFcbi49KNZXNLicwfe3TETC7gPdPqI7uBcHMb1RSzYq34c6PDUM9mn8HRsUTZEiDBve3NjVZumBj0U7SS37mGO7vcwtiK-_pU7U7L_f-digo9YbhSwIfMRwIITKGXbxdIUTCGF1SeCJxloKsU-3k9ddRbX1eDq1q_fx1JbBGT0glVyXimDuP4TQ5qpCAmnGEj2s_6n5mtn1z-97-63itFQZLPO1Ev2tu_WF7Ju-VPc0Skg5bYXxBhkY1xpD7EM_7fyflSpIsqMgVth5xhVr4eQxWQ8enaSAJQSG16yFSDuJ798rrcXr_3n-lfdk7icQjEBmFujL7AodiP_Y4Z7-YxvtZNs4zMgpNTl6tF8sglyPsmqchrjvQ-m-aP94r-TwCA2Ka8upPJZwtvSpoYCXkYMZU2NXvRMBfq9P3i3Le4VAZUAlUZ_oPKsxPgY0Q_BSKLkyr9bhQhQrJjo_x3TPlIB0DPjnMfcIoYP0QaYw1a0fTKDr8fB6ntNuvmoL1ZGkXa69Njh43zf9GiGxHQrA_jDYWRSzF5--WmTVrN97_Sm8LbLUy_lGBmLanJjFkDlGkRqjA_4tm00
> > > > > > > > > > > > >> >> >
> > > > > > > > > > > > >> >> >> :
> > > > > > > > > > > > >> >> >>
> > > > > > > > > > > > >> >> >> [image: image.png]
> > > > > > > > > > > > >> >> >>
> > > > > > > > > > > > >> >> >> Diagram 2
> > > > > > > > > > > > >> >> >> <
> > > > > > > > > > > > >> >>
> > > > > > > > > > > > >>
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://www.plantuml.com/plantuml/uml/VLLDR-8m4BtdLupO2sWBLVU8AaGB7AXAbssGzb896SS9RXqxjKqBwkv_tt7iV43fdaZYDpFlpRmnOsE9jhjSH9PRmM31hERKm8scMsuPjJlDe0yheZDc8RR4iYWoBrmMH9CS2a9VICPYUy1OZN0YCy5Q0BCbYNhdCeEK28En8G8wCvbnoQ0R8_05Bc6bkLIz3X03p1zzH7zR-9ZfDquPt9C3qoNCX2yV4G2NbkcKu5jdgGJHt0GbZwnG6i-UP3TUpk5gM6Ldqke350eZUqzoCft3U9xWHvxoa5-7K4nF1J46EbEMafsmdrCBbQ44gVggy18IZrn_ph5asd1ZiIKdQSgueZvjXrQFSFrdC3YN-nXmBacxbGiYyLVxLaBtdhqn0LSzdBDhqQtQoOJeGyad3z0lUqnYgpGB6Ns8oVyta00Dy_WnX0tIOZ8v6SYxHll1TrH6aejAik-mh-AphVFCwSUQqFypElag5QRGFDjQKEd96K1P8QP41c9TzA_IIQyvdAWyv_RSiS3skb0_EzDDkK2v5xWF6MiGFlvhpFLcD2Dq2pml14gaF67eQkmd8gulDoC4kSOu6KVpkvlUJg1RTbWISU40RdBUUS_9XfRZ2dwxm_SW8LYFISgm_MnlDQ6M9P1gbKEc4X-2pH_FvJCkCqm9pbVjD6LrwdLeOrDWfOaqc8Wh9BE85oNKxkNQ6o4yGRy_Eae0G_G8tZv81d3bHDB23WOdisohVr3nh_j6lbSjbNaLRTc8UgtPbAU1J_tygOfZX9DWEJeHDvYx-qmSi5FgNLPZwHrHcUsncGQ5-skhUclpE5fo4ounpFauYrUbkU6ccfnxMvitwag4IyerhTxj8In_Oj1bDO4pQru674loYrGlULHLEGCjwJJ8gDoVZR8MxO4BT3IzRvIcAQKezC6xpziGnTyImrfEGyJI_OcKfgtxIvnTqFEMS17L9Z-jsARN5FmTheP7HtSdtOMT0B4GY2FYHXxgQmMtj2bRqiLFGapiVe1_QVKDrkqXcm83aFEXnMYCZ-xlyHy
> > > > > > > > > > > > >> >> >
> > > > > > > > > > > > >> >> >> :
> > > > > > > > > > > > >> >> >> [image: image.png]
> > > > > > > > > > > > >> >> >>
> > > > > > > > > > > > >> >> >>  -ej
> > > > > > > > > > > > >> >> >>
> > > > > > > > > > > > >> >> >> On Wed, Apr 15, 2026 at 8:22 AM Dmitri
> > > > Bourlatchkov
> > > > > <
> > > > > > > > > > > > >> dimas@apache.org>
> > > > > > > > > > > > >> >> >> wrote:
> > > > > > > > > > > > >> >> >>
> > > > > > > > > > > > >> >> >>> Hi All,
> > > > > > > > > > > > >> >> >>>
> > > > > > > > > > > > >> >> >>> Heads up: The current state of PR [4115]
> > looks
> > > > > pretty
> > > > > > > > solid
> > > > > > > > > > to
> > > > > > > > > > > > me.
> > > > > > > > > > > > >> I
> > > > > > > > > > > > >> >> >>> believe this PR is approaching a mergeable
> > > > > condition.
> > > > > > > > > > > > >> >> >>>
> > > > > > > > > > > > >> >> >>> Please post your reviews if you have any
> > > > comments.
> > > > > > > > > > > > >> >> >>>
> > > > > > > > > > > > >> >> >>> [4115]
> > > > https://github.com/apache/polaris/pull/4115
> > > > > > > > > > > > >> >> >>>
> > > > > > > > > > > > >> >> >>> Thanks,
> > > > > > > > > > > > >> >> >>> Dmitri.
> > > > > > > > > > > > >> >> >>>
> > > > > > > > > > > > >> >> >>> On Tue, Mar 3, 2026 at 3:29 PM Anand Kumar
> > > > Sankaran
> > > > > > via
> > > > > > > > > dev <
> > > > > > > > > > > > >> >> >>> dev@polaris.apache.org> wrote:
> > > > > > > > > > > > >> >> >>>
> > > > > > > > > > > > >> >> >>> > Hi Yufei and Dmitri,
> > > > > > > > > > > > >> >> >>> >
> > > > > > > > > > > > >> >> >>> > Here is a proposal for the REST endpoints
> > for
> > > > > > metrics
> > > > > > > > and
> > > > > > > > > > > > events.
> > > > > > > > > > > > >> >> >>> >
> > > > > > > > > > > > >> >> >>> >
> > > > > > https://github.com/apache/polaris/pull/3924/changes
> > > > > > > > > > > > >> >> >>> >
> > > > > > > > > > > > >> >> >>> > I did not see any precursors for raising
> a
> > PR
> > > > for
> > > > > > > > > > proposals,
> > > > > > > > > > > so
> > > > > > > > > > > > >> >> trying
> > > > > > > > > > > > >> >> >>> > this.  Please let me know what you think.
> > > > > > > > > > > > >> >> >>> >
> > > > > > > > > > > > >> >> >>> > -
> > > > > > > > > > > > >> >> >>> > Anand
> > > > > > > > > > > > >> >> >>> >
> > > > > > > > > > > > >> >> >>> > From: Anand Kumar Sankaran <
> > > > > > > anand.sankaran@workday.com
> > > > > > > > >
> > > > > > > > > > > > >> >> >>> > Date: Monday, March 2, 2026 at 10:25 AM
> > > > > > > > > > > > >> >> >>> > To: dev@polaris.apache.org <
> > > > > dev@polaris.apache.org
> > > > > > >
> > > > > > > > > > > > >> >> >>> > Subject: Re: Polaris Telemetry and Audit
> > > Trail
> > > > > > > > > > > > >> >> >>> >
> > > > > > > > > > > > >> >> >>> > About the REST API, based on my use
> cases:
> > > > > > > > > > > > >> >> >>> >
> > > > > > > > > > > > >> >> >>> >
> > > > > > > > > > > > >> >> >>> >   1.
> > > > > > > > > > > > >> >> >>> > I want to be able to query commit metrics
> > to
> > > > > track
> > > > > > > > files
> > > > > > > > > > > added
> > > > > > > > > > > > /
> > > > > > > > > > > > >> >> >>> removed
> > > > > > > > > > > > >> >> >>> > per commit, along with record counts. The
> > > > > ingestion
> > > > > > > > > > pipeline
> > > > > > > > > > > > that
> > > > > > > > > > > > >> >> >>> writes
> > > > > > > > > > > > >> >> >>> > this data is owned by us and we are
> > > guaranteed
> > > > to
> > > > > > > write
> > > > > > > > > > this
> > > > > > > > > > > > >> >> >>> information
> > > > > > > > > > > > >> >> >>> > for each write.
> > > > > > > > > > > > >> >> >>> >   2.
> > > > > > > > > > > > >> >> >>> > I want to be able to query scan metrics
> for
> > > > > read. I
> > > > > > > > > > > understand
> > > > > > > > > > > > >> >> clients
> > > > > > > > > > > > >> >> >>> do
> > > > > > > > > > > > >> >> >>> > not fulfill this requirement.
> > > > > > > > > > > > >> >> >>> >   3.
> > > > > > > > > > > > >> >> >>> > I want to be able to query the events
> table
> > > > > (events
> > > > > > > are
> > > > > > > > > > > > >> persisted) -
> > > > > > > > > > > > >> >> >>> this
> > > > > > > > > > > > >> >> >>> > may supersede #2, I am not sure yet.
> > > > > > > > > > > > >> >> >>> >
> > > > > > > > > > > > >> >> >>> > All this information is in the JDBC based
> > > > > > persistence
> > > > > > > > > model
> > > > > > > > > > > and
> > > > > > > > > > > > >> is
> > > > > > > > > > > > >> >> >>> > persisted in the metastore. I currently
> > don’t
> > > > > have
> > > > > > a
> > > > > > > > need
> > > > > > > > > > to
> > > > > > > > > > > > >> query
> > > > > > > > > > > > >> >> >>> > prometheus or open telemetry. I do
> publish
> > > some
> > > > > > > events
> > > > > > > > to
> > > > > > > > > > > > >> Prometheus
> > > > > > > > > > > > >> >> >>> and
> > > > > > > > > > > > >> >> >>> > they are forwarded to our dashboards
> > > elsewhere.
> > > > > > > > > > > > >> >> >>> >
> > > > > > > > > > > > >> >> >>> > About the CLI utilities, I meant the
> admin
> > > user
> > > > > > > > > utilities.
> > > > > > > > > > In
> > > > > > > > > > > > >> one of
> > > > > > > > > > > > >> >> >>> the
> > > > > > > > > > > > >> >> >>> > earliest drafts of my proposal, Prashant
> > > > > mentioned
> > > > > > > that
> > > > > > > > > the
> > > > > > > > > > > > >> metrics
> > > > > > > > > > > > >> >> >>> tables
> > > > > > > > > > > > >> >> >>> > can grow indefinitely and that a similar
> > > > problem
> > > > > > > exists
> > > > > > > > > > with
> > > > > > > > > > > > the
> > > > > > > > > > > > >> >> events
> > > > > > > > > > > > >> >> >>> > table as well. We discussed that cleaning
> > up
> > > of
> > > > > old
> > > > > > > > > records
> > > > > > > > > > > > from
> > > > > > > > > > > > >> >> both
> > > > > > > > > > > > >> >> >>> > metrics tables and events tables can be
> > done
> > > > via
> > > > > a
> > > > > > > CLI
> > > > > > > > > > > utility.
> > > > > > > > > > > > >> >> >>> >
> > > > > > > > > > > > >> >> >>> > I see that Yufei has covered the
> discussion
> > > > about
> > > > > > > > > > > datasources.
> > > > > > > > > > > > >> >> >>> >
> > > > > > > > > > > > >> >> >>> > -
> > > > > > > > > > > > >> >> >>> > Anand
> > > > > > > > > > > > >> >> >>> >
> > > > > > > > > > > > >> >> >>> >
> > > > > > > > > > > > >> >> >>> >
> > > > > > > > > > > > >> >> >>> > From: Yufei Gu <fl...@gmail.com>
> > > > > > > > > > > > >> >> >>> > Date: Friday, February 27, 2026 at
> 9:54 PM
> > > > > > > > > > > > >> >> >>> > To: dev@polaris.apache.org <
> > > > > dev@polaris.apache.org
> > > > > > >
> > > > > > > > > > > > >> >> >>> > Subject: Re: Polaris Telemetry and Audit
> > > Trail
> > > > > > > > > > > > >> >> >>> >
> > > > > > > > > > > > >> >> >>> > This Message Is From an External Sender
> > > > > > > > > > > > >> >> >>> > This message came from outside your
> > > > organization.
> > > > > > > > > > > > >> >> >>> > Report Suspicious<
> > > > > > > > > > > > >> >> >>> >
> > > > > > > > > > > > >> >> >>>
> > > > > > > > > > > > >> >>
> > > > > > > > > > > > >>
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://us-phishalarm-ewt.proofpoint.com/EWT/v1/Iz9xO38YGHZK!YhNDZABkHi1B699ote2uMwpOZw8i0QMCGO2Szc-HshuABGhGvwPJcymE6G2oUUxtS8xDkSrtGTPm_I3QnVDHoLMk50m9v8z_nZKTkd-bnVUbreF1u0WnfV_X5eYevZl_$
> > > > > > > > > > > > >> >> >>> > >
> > > > > > > > > > > > >> >> >>> >
> > > > > > > > > > > > >> >> >>> >
> > > > > > > > > > > > >> >> >>> > As I mentioned in
> > > > > > > > > > > > >> >> >>> >
> > > > > > > > > > > > >> >> >>>
> > > > > > > > > > > > >> >>
> > > > > > > > > > > > >>
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://urldefense.com/v3/__https://github.com/apache/polaris/issues/3890__;!!Iz9xO38YGHZK!5EuyFFkk3vhRWVIRvQAWBSQfpJkTMA9HxugzDwXmN0LPPqhEFxYkFRGVhtb8AqUwXtDh2OplcMnbMDHKOxrvDU0$
> > > > > > > > > > > > >> >> >>> ,
> > > > > > > > > > > > >> >> >>> > supporting
> > > > > > > > > > > > >> >> >>> > multiple data sources is not a trivial
> > > change.
> > > > I
> > > > > > > would
> > > > > > > > > > > strongly
> > > > > > > > > > > > >> >> >>> recommend
> > > > > > > > > > > > >> >> >>> > starting with a design document to
> > carefully
> > > > > > evaluate
> > > > > > > > the
> > > > > > > > > > > > >> >> architectural
> > > > > > > > > > > > >> >> >>> > implications and long term impact.
> > > > > > > > > > > > >> >> >>> >
> > > > > > > > > > > > >> >> >>> > A REST endpoint to query metrics seems
> > > > reasonable
> > > > > > > given
> > > > > > > > > the
> > > > > > > > > > > > >> current
> > > > > > > > > > > > >> >> >>> JDBC
> > > > > > > > > > > > >> >> >>> > based persistence model. That said, we
> may
> > > also
> > > > > > > > consider
> > > > > > > > > > > > >> alternative
> > > > > > > > > > > > >> >> >>> > storage models. For example, if we later
> > > adopt
> > > > a
> > > > > > time
> > > > > > > > > > series
> > > > > > > > > > > > >> system
> > > > > > > > > > > > >> >> >>> such as
> > > > > > > > > > > > >> >> >>> > Prometheus to store metrics, the query
> > model
> > > > and
> > > > > > > access
> > > > > > > > > > > > patterns
> > > > > > > > > > > > >> >> would
> > > > > > > > > > > > >> >> >>> be
> > > > > > > > > > > > >> >> >>> > fundamentally different. Designing the
> REST
> > > API
> > > > > > > without
> > > > > > > > > > > > >> considering
> > > > > > > > > > > > >> >> >>> these
> > > > > > > > > > > > >> >> >>> > potential evolutions may limit
> flexibility.
> > > I'd
> > > > > > > suggest
> > > > > > > > > to
> > > > > > > > > > > > start
> > > > > > > > > > > > >> >> with
> > > > > > > > > > > > >> >> >>> the
> > > > > > > > > > > > >> >> >>> > use case.
> > > > > > > > > > > > >> >> >>> >
> > > > > > > > > > > > >> >> >>> > Yufei
> > > > > > > > > > > > >> >> >>> >
> > > > > > > > > > > > >> >> >>> >
> > > > > > > > > > > > >> >> >>> > On Fri, Feb 27, 2026 at 3:42 PM Dmitri
> > > > > > Bourlatchkov <
> > > > > > > > > > > > >> >> dimas@apache.org>
> > > > > > > > > > > > >> >> >>> > wrote:
> > > > > > > > > > > > >> >> >>> >
> > > > > > > > > > > > >> >> >>> > > Hi Anand,
> > > > > > > > > > > > >> >> >>> > >
> > > > > > > > > > > > >> >> >>> > > Sharing my view... subject to
> discussion:
> > > > > > > > > > > > >> >> >>> > >
> > > > > > > > > > > > >> >> >>> > > 1. Adding non-IRC REST API to Polaris
> is
> > > > > > perfectly
> > > > > > > > > fine.
> > > > > > > > > > > > >> >> >>> > >
> > > > > > > > > > > > >> >> >>> > > Figuring out specific endpoint URIs and
> > > > > payloads
> > > > > > > > might
> > > > > > > > > > > > require
> > > > > > > > > > > > >> a
> > > > > > > > > > > > >> >> few
> > > > > > > > > > > > >> >> >>> > > roundtrips, so opening a separate
> thread
> > > for
> > > > > that
> > > > > > > > might
> > > > > > > > > > be
> > > > > > > > > > > > >> best.
> > > > > > > > > > > > >> >> >>> > > Contributors commonly create Google
> Docs
> > > for
> > > > > new
> > > > > > > API
> > > > > > > > > > > > proposals
> > > > > > > > > > > > >> too
> > > > > > > > > > > > >> >> >>> (they
> > > > > > > > > > > > >> >> >>> > > fairly easy to update as the email
> > > discussion
> > > > > > > > > > progresses).
> > > > > > > > > > > > >> >> >>> > >
> > > > > > > > > > > > >> >> >>> > > There was a suggestion to try Markdown
> > > (with
> > > > > PRs)
> > > > > > > for
> > > > > > > > > > > > proposals
> > > > > > > > > > > > >> >> [1]
> > > > > > > > > > > > >> >> >>> ...
> > > > > > > > > > > > >> >> >>> > > feel free to give it a try if you are
> > > > > comfortable
> > > > > > > > with
> > > > > > > > > > > that.
> > > > > > > > > > > > >> >> >>> > >
> > > > > > > > > > > > >> >> >>> > > 2. Could you clarify whether you mean
> end
> > > > user
> > > > > > > > > utilities
> > > > > > > > > > or
> > > > > > > > > > > > >> admin
> > > > > > > > > > > > >> >> >>> user
> > > > > > > > > > > > >> >> >>> > > utilities? In the latter case those
> might
> > > be
> > > > > more
> > > > > > > > > > suitable
> > > > > > > > > > > > for
> > > > > > > > > > > > >> the
> > > > > > > > > > > > >> >> >>> Admin
> > > > > > > > > > > > >> >> >>> > > CLI (java) not the Python CLI, IMHO.
> > > > > > > > > > > > >> >> >>> > >
> > > > > > > > > > > > >> >> >>> > > Why would these utilities be common
> with
> > > > > events?
> > > > > > > > IMHO,
> > > > > > > > > > > event
> > > > > > > > > > > > >> use
> > > > > > > > > > > > >> >> >>> cases
> > > > > > > > > > > > >> >> >>> > are
> > > > > > > > > > > > >> >> >>> > > distinct from scan/commit metrics.
> > > > > > > > > > > > >> >> >>> > >
> > > > > > > > > > > > >> >> >>> > > 3. I'd prefer separating metrics
> > > persistence
> > > > > from
> > > > > > > > > > MetaStore
> > > > > > > > > > > > >> >> >>> persistence
> > > > > > > > > > > > >> >> >>> > at
> > > > > > > > > > > > >> >> >>> > > the code level, so that they could be
> > mixed
> > > > and
> > > > > > > > matched
> > > > > > > > > > > > >> >> >>> independently.
> > > > > > > > > > > > >> >> >>> > The
> > > > > > > > > > > > >> >> >>> > > separate datasource question will
> become
> > a
> > > > > > > non-issue
> > > > > > > > > with
> > > > > > > > > > > > that
> > > > > > > > > > > > >> >> >>> approach,
> > > > > > > > > > > > >> >> >>> > I
> > > > > > > > > > > > >> >> >>> > > guess.
> > > > > > > > > > > > >> >> >>> > >
> > > > > > > > > > > > >> >> >>> > > The rationale for separating scan
> metrics
> > > and
> > > > > > > > metastore
> > > > > > > > > > > > >> >> persistence
> > > > > > > > > > > > >> >> >>> is
> > > > > > > > > > > > >> >> >>> > that
> > > > > > > > > > > > >> >> >>> > > "cascading deletes" between them are
> > hardly
> > > > > ever
> > > > > > > > > > required.
> > > > > > > > > > > > >> >> >>> Furthermore,
> > > > > > > > > > > > >> >> >>> > the
> > > > > > > > > > > > >> >> >>> > > data and query patterns are very
> > different
> > > so
> > > > > > > > different
> > > > > > > > > > > > >> >> technologies
> > > > > > > > > > > > >> >> >>> > might
> > > > > > > > > > > > >> >> >>> > > be beneficial in each case.
> > > > > > > > > > > > >> >> >>> > >
> > > > > > > > > > > > >> >> >>> > > [1]
> > > > > > > > > > > > >> >> >>> >
> > > > > > > > > > > > >> >> >>>
> > > > > > > > > > > > >> >>
> > > > > > > > > > > > >>
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://urldefense.com/v3/__https://lists.apache.org/thread/yto2wp982t43h1mqjwnslswhws5z47cy__;!!Iz9xO38YGHZK!5EuyFFkk3vhRWVIRvQAWBSQfpJkTMA9HxugzDwXmN0LPPqhEFxYkFRGVhtb8AqUwXtDh2OplcMnbMDHKxYDakNU$
> > > > > > > > > > > > >> >> >>> > >
> > > > > > > > > > > > >> >> >>> > > Cheers,
> > > > > > > > > > > > >> >> >>> > > Dmitri.
> > > > > > > > > > > > >> >> >>> > >
> > > > > > > > > > > > >> >> >>> > > On Fri, Feb 27, 2026 at 6:19 PM Anand
> > Kumar
> > > > > > > Sankaran
> > > > > > > > > via
> > > > > > > > > > > dev
> > > > > > > > > > > > <
> > > > > > > > > > > > >> >> >>> > > dev@polaris.apache.org> wrote:
> > > > > > > > > > > > >> >> >>> > >
> > > > > > > > > > > > >> >> >>> > > > Thanks all. This PR is merged now.
> > > > > > > > > > > > >> >> >>> > > >
> > > > > > > > > > > > >> >> >>> > > > Here are the follow-up features /
> work
> > > > > needed.
> > > > > > > > These
> > > > > > > > > > > were
> > > > > > > > > > > > >> all
> > > > > > > > > > > > >> >> >>> part of
> > > > > > > > > > > > >> >> >>> > > the
> > > > > > > > > > > > >> >> >>> > > > merged PR at some point in time and
> > were
> > > > > > removed
> > > > > > > to
> > > > > > > > > > > reduce
> > > > > > > > > > > > >> >> scope.
> > > > > > > > > > > > >> >> >>> > > >
> > > > > > > > > > > > >> >> >>> > > > Please let me know what you think.
> > > > > > > > > > > > >> >> >>> > > >
> > > > > > > > > > > > >> >> >>> > > >
> > > > > > > > > > > > >> >> >>> > > >   1.  A REST API to paginate through
> > > table
> > > > > > > metrics.
> > > > > > > > > > This
> > > > > > > > > > > > >> will be
> > > > > > > > > > > > >> >> >>> > non-IRC
> > > > > > > > > > > > >> >> >>> > > > standard addition.
> > > > > > > > > > > > >> >> >>> > > >   2.  Utilities for managing old
> > records,
> > > > > > should
> > > > > > > be
> > > > > > > > > > > common
> > > > > > > > > > > > >> with
> > > > > > > > > > > > >> >> >>> events.
> > > > > > > > > > > > >> >> >>> > > > There was some discussion that it
> > belongs
> > > > to
> > > > > > the
> > > > > > > > CLI.
> > > > > > > > > > > > >> >> >>> > > >   3.  Separate datasource (metrics,
> > > events,
> > > > > > even
> > > > > > > > > other
> > > > > > > > > > > > >> tables?).
> > > > > > > > > > > > >> >> >>> > > >
> > > > > > > > > > > > >> >> >>> > > >
> > > > > > > > > > > > >> >> >>> > > > Anything else?
> > > > > > > > > > > > >> >> >>> > > >
> > > > > > > > > > > > >> >> >>> > > > -
> > > > > > > > > > > > >> >> >>> > > > Anand
> > > > > > > > > > > > >> >> >>> > > >
> > > > > > > > > > > > >> >> >>> > > >
> > > > > > > > > > > > >> >> >>> > >
> > > > > > > > > > > > >> >> >>> >
> > > > > > > > > > > > >> >> >>> >
> > > > > > > > > > > > >> >> >>>
> > > > > > > > > > > > >> >> >>
> > > > > > > > > > > > >> >>
> > > > > > > > > > > > >> >
> > > > > > > > > > > > >>
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>

Re: Proposal for REST endpoints for table metrics and events

[Yufei Gu <fl...@gmail.com>]

I do not think we have agreement on the query REST APIs yet.

In particular, the current query API shape seems highly coupled to the
*example[1]* metrics in the Iceberg REST spec. I do not think that is a
good way to design a REST API contract. For example, fields like
resultDataFiles, resultDeleteFiles, totalFileSizeBytes, totalDataManifests,
totalDeleteManifests, and scannedDataManifests are very specific to one
style of scan metric. That makes the API look more like a direct projection
of the current example payload than a general query model.

My preference is that Polaris should not define or strongly opine on the
Iceberg scan or commit metrics query REST API at this point. Polaris can
resolve and authorize the table, accept the metrics report, and delegate it
to the configured reporting path. Durable storage, filtering, dashboards,
and query APIs should remain downstream impl. choices unless and until the
Iceberg community standardizes a query contract, which I doubt will happen.

So I do not think the intake path has to be blocked, but I would prefer not
to merge a Polaris specific query API that may be hard to evolve later.

1.
https://github.com/apache/iceberg/blob/2a6c556c0c883c04c6d3fbd68e6aeda36b91e0aa/open-api/rest-catalog-open-api.yaml#L4152

Yufei


On Tue, Jun 23, 2026 at 9:30 AM Dmitri Bourlatchkov <di...@apache.org>
wrote:

> Hi All,
>
> So, what are the blockers for merging [4115], if any?
>
> [4115] https://github.com/apache/polaris/pull/4115
>
> Thanks,
> Dmitri.
>
> On Wed, Jun 17, 2026 at 5:28 PM EJ Wang <ej...@gmail.com>
> wrote:
>
> > Hi folks,
> >
> > Thanks again for the discussion today. I updated the sync doc with notes
> > from today's metrics sync:
> >
> >
> >
> https://docs.google.com/document/d/100h7c4damrUzVuquYbBHM0EvA4LSWuW2IT2dN_7nYVA/edit?tab=t.ezk0rgdx0c6m
> >
> > A few highlights:
> >
> >    - No objection to keeping the default metrics behavior small:
> >    no-op/log-only is enough as the built-in default.
> >    - Durable storage, event forwarding, external queues, dashboards, and
> >    custom filtering should be implementation choices behind the metrics
> >    reporting path.
> >    - For PR #4115, the REST layer should resolve/authz/accept the report,
> >    then delegate to the selected metrics implementation. It should not
> own
> >    durable storage semantics.
> >    - The rough boundary we discussed is: spec owns the wire contract,
> >    runtime owns framework wiring, API/contract modules own
> provider-facing
> >    contracts, and extensions own replaceable implementations.
> >    - OpenLineage has a similar path-naming question. We should avoid
> >    occupying a generic lineage path too early if we want room for other
> >    lineage systems later.
> >
> > I believe the boundary rules previewed in sync meeting are useful for
> > reviewing PR #4115, but it is not
> > a final module-layout proposal.
> >
> > For metrics specifically, I think the target remains:
> >    runtime
> >      -> metrics reporting/emitting contract
> >      -> default implementation
> >           -> lower implementation details, if needed
> >
> > That keeps the default simple, while allowing durable JDBC, event-backed,
> > or external-queue implementations to be added separately.
> >
> > We also touched on the split between Iceberg metrics emitting and
> > Polaris-owned metrics querying. The querying side can keep evolving,
> > especially for dashboard or generic-table use cases, but I do not think
> > that needs to block the intake path in this PR.
> >
> > Thanks,
> > -ej
> >
> > On Tue, Jun 16, 2026 at 2:17 PM Yufei Gu <fl...@gmail.com> wrote:
> >
> > > Thanks for chiming in, EJ. Agreed that the default battery should stay
> > > small. I'm leaning toward not including metrics persistence in the
> > default
> > > battery.
> > >
> > > Yufei
> > >
> > >
> > > On Thu, Jun 11, 2026 at 8:30 PM EJ Wang <
> ej.wang.dev.interest@gmail.com>
> > > wrote:
> > >
> > > > Hi Yufei,
> > > >
> > > > Thanks for connecting this back to the REST endpoint proposal. I
> > replied
> > > > with the fuller version on the event-forwarding thread, but wanted to
> > add
> > > > the shorter version here too.
> > > >
> > > > I agree that metrics reporting/emitting is the right conceptual
> > boundary,
> > > > and I also think the event/listener path is a good implementation
> > > direction
> > > > to explore. The distinction I want to keep clear is default battery
> vs
> > > > extension implementation.
> > > >
> > > > For the REST/API side, I would keep the semantics narrow: Polaris
> > > resolves
> > > > and authorizes the table, accepts the Iceberg scan/commit metrics
> > report
> > > > into the configured reporting/emitting path, and returns 204. That
> 204
> > > > should mean Polaris accepted the report into the ingestion path; it
> > > should
> > > > not imply durable storage.
> > > >
> > > > Durable storage, event forwarding, external telemetry routing,
> > filtering,
> > > > and retention should sit behind that reporting/emitting boundary as
> > > > implementation-layer behavior. A durable JDBC path can be one named
> > > > extension implementation. An event/listener forwarding path can be
> > > another
> > > > named extension implementation.
> > > >
> > > > So I am +1 on the event-forwarding idea as a non-default extension
> > > > implementation. I just would not make it the default battery or the
> > core
> > > > API shape. The default battery can stay small and safe, while
> > deployments
> > > > that need forwarding, dashboards, or durable storage can opt into the
> > > > implementation that matches their operational model.
> > > >
> > > > Thanks,
> > > > -ej
> > > >
> > > > On Wed, Jun 10, 2026 at 11:43 AM Yufei Gu <fl...@gmail.com>
> > wrote:
> > > >
> > > > > Thanks EJ. I agree that the reporting/emitting boundary feels like
> > the
> > > > > right SPI boundary, but I wonder if we can simplify this even
> > further.
> > > > >
> > > > > Iceberg scan and commit metrics look very similar to events to me.
> > They
> > > > are
> > > > > append only, asynchronously consumed, often forwarded to external
> > > > systems,
> > > > > and have similar retention and cleanup requirements. More details
> can
> > > be
> > > > > found in this thread [1]. In that case, Polaris may only need to
> > accept
> > > > the
> > > > > report and emit an event through the existing event framework.
> > > Filtering,
> > > > > async delivery, custom sinks, and retention mechanisms could then
> be
> > > > shared
> > > > > instead of introducing a separate metrics specific extension layer
> > and
> > > > SPI.
> > > > >
> > > > > The main requirement I still see is querying metrics through
> Polaris.
> > > If
> > > > we
> > > > > want that battery included experience, we could work on a
> persistence
> > > > > listener implementation that selectively saves metrics and expose
> it
> > > via
> > > > > the IRC event endpoint which seems pretty close in Iceberg
> > > > > community. Another option I like more is to allow users to develop
> > > their
> > > > > own listener, so they can persist scan/commit metrics to any
> systems
> > > they
> > > > > prefer. In general, that feels like an implementation choice on top
> > of
> > > > the
> > > > > event framework rather than something that needs to shape the core
> > > > > architecture.
> > > > >
> > > > > 1.
> https://lists.apache.org/thread/x9j8nscvy8hq61tyn01mj8yp6n9of0kp
> > > > >
> > > > > Thanks,
> > > > > Yufei
> > > > >
> > > > >
> > > > > On Thu, Jun 4, 2026 at 5:14 PM EJ Wang <
> > ej.wang.dev.interest@gmail.com
> > > >
> > > > > wrote:
> > > > >
> > > > > > Good point. I agree the existing PolarisMetricsReporter is
> already
> > > very
> > > > > > close to the right conceptual boundary. I am not proposing a
> second
> > > > > > parallel reporter concept. The distinction I am trying to make is
> > > > > narrower:
> > > > > >
> > > > > > Current state:
> > > > > >
> > > > > >    - PolarisMetricsReporter is the existing metrics reporting
> hook.
> > > > > >    - But it currently lives under runtime/service.
> > > > > >    - Its contract is documented around Quarkus/CDI discovery and
> > > > > > @Identifier
> > > > > >    selection.
> > > > > >    - The default implementation is log-only.
> > > > > >    - The durable implementation writes through
> > PolarisMetricsManager
> > > > and
> > > > > >    MetricsPersistence.
> > > > > >    - MetricsPersistence is currently inherited by
> BasePersistence.
> > > > > >
> > > > > > My proposal:
> > > > > >
> > > > > >    - Keep the reporting/emitting boundary as the SPI.
> > > > > >    - Revise that contract in a framework-agnostic optional
> Iceberg
> > > > > *metrics
> > > > > >    extension API layer*.
> > > > > >    - Keep runtime/service responsible for REST ingestion,
> > table/authz
> > > > > >    validation, and runtime wiring.
> > > > > >    - Keep the battery implementation log-only in the metrics
> > > extension
> > > > > API
> > > > > >    layer, *not under runtime/service*.
> > > > > >    - Treat durable JDBC metrics as one implementation of the
> > > reporting
> > > > > SPI.
> > > > > >    - Decompose the MetricsPersistence and BasePersistence
> coupling
> > as
> > > > > >    durable implementation detail.
> > > > > >
> > > > > > So the difference is not “new reporter concept vs old reporter
> > > > concept.”
> > > > > > but:
> > > > > >
> > > > > >
> > > > > >    - PolarisMetricsReporter today = runtime/service CDI-shaped
> > hook.
> > > > > >    - Target reporting SPI = same conceptual boundary, but placed
> in
> > > the
> > > > > >    right extension API layer and kept framework-agnostic.
> > > > > >
> > > > > > For the current metrics PR, I think the minimal architectural
> > cleanup
> > > > > > before merge is to *put the reporting SPI contract and the
> battery
> > > > > default
> > > > > > in the right place*. The durable JDBC implementation,
> event-backed
> > > > async
> > > > > > implementation, external queue support, and deeper persistence
> > > cleanup
> > > > > can
> > > > > > continue as follow-ups, as long as we do not lock the SPI
> boundary
> > to
> > > > > > BasePersistence or to runtime/service wiring.
> > > > > >
> > > > > > -ej
> > > > > >
> > > > > > On Wed, Jun 3, 2026 at 3:00 PM Dmitri Bourlatchkov <
> > dimas@apache.org
> > > >
> > > > > > wrote:
> > > > > >
> > > > > > > Hi EJ,
> > > > > > >
> > > > > > > Thanks for the recap / summary!
> > > > > > >
> > > > > > > > Do folks agree that the stable SPI boundary should be
> > > > > > > metrics reporting/emitting, not metrics persistence?
> > > > > > >
> > > > > > > SGTM.
> > > > > > >
> > > > > > > > Does an optional Iceberg metrics extension API layer sound
> like
> > > the
> > > > > > right
> > > > > > > home for this SPI?
> > > > > > >
> > > > > > > How is that different from current PolarisMetricsReporter?
> > > > > > >
> > > > > > > > Should the current durable metrics work be reframed as a
> > durable
> > > > > > > JDBC reference implementation of that SPI?
> > > > > > >
> > > > > > > SGTM.
> > > > > > >
> > > > > > > > What is the smallest PR sequence to get there without
> blocking
> > > > > > > the current metrics work unnecessarily?
> > > > > > >
> > > > > > > Let's get https://github.com/apache/polaris/pull/4397 merged
> > > first.
> > > > > > >
> > > > > > > Then, I'd propose isolating the Metrics schema from the
> MetaStore
> > > > > schema.
> > > > > > > This will probably have some ripple effect into the bootstrap
> > > > > workflows,
> > > > > > so
> > > > > > > it's not a trivial change.
> > > > > > >
> > > > > > > Then, let's reassess.
> > > > > > >
> > > > > > > Cheers,
> > > > > > > Dmitri.
> > > > > > >
> > > > > > > On Wed, Jun 3, 2026 at 5:40 PM EJ Wang <
> > > > ej.wang.dev.interest@gmail.com
> > > > > >
> > > > > > > wrote:
> > > > > > >
> > > > > > > > Hi folks,
> > > > > > > >
> > > > > > > > Thanks again for the discussion today. I updated the sync doc
> > > with
> > > > > > notes
> > > > > > > > from the third metrics architecture sync:
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://docs.google.com/document/d/100h7c4damrUzVuquYbBHM0EvA4LSWuW2IT2dN_7nYVA/edit?tab=t.k96s2xyqr5u1
> > > > > > > >
> > > > > > > > A few highlights from today’s discussion:
> > > > > > > >
> > > > > > > >    - We clarified that Iceberg metrics reporting can be
> > > interpreted
> > > > > > > either
> > > > > > > >    as sync or async handling from Polaris’s perspective, so
> > > Polaris
> > > > > > > should
> > > > > > > >    stay flexible as a platform instead of baking one handling
> > > model
> > > > > > into
> > > > > > > > the
> > > > > > > >    REST API behavior.
> > > > > > > >    - We aligned that the built-in (battery) behavior for
> > metrics
> > > > > > emitting
> > > > > > > >    can stay simple: no-op/log-only is enough as the default.
> > > > > > > >    - Durable metrics persistence should be treated as an
> > > > > implementation
> > > > > > > of
> > > > > > > >    the metrics reporting path, not as the core SPI boundary.
> > > > > > > >    - The existing durable metrics work can be reviewed as a
> > > > reference
> > > > > > > >    implementation of the reporting SPI, with
> > persistence-related
> > > > > logic
> > > > > > > kept
> > > > > > > >    self-contained under a metrics durable implementation
> module
> > > > > rather
> > > > > > > than
> > > > > > > >    scattered through core entity persistence.
> > > > > > > >    - Dashboard/insights remains a real use case, but we
> agreed
> > to
> > > > > keep
> > > > > > it
> > > > > > > >    separate from the core metrics intake discussion for now.
> > > > > > > >
> > > > > > > > I also did a quick source check after the meeting to make
> sure
> > we
> > > > are
> > > > > > > > describing the current state accurately.
> > > > > > > >
> > > > > > > > Current state:
> > > > > > > >
> > > > > > > >    - Polaris already has a metrics reporting hook:
> > > > > > > PolarisMetricsReporter.
> > > > > > > >    - The default implementation is DefaultMetricsReporter,
> > > selected
> > > > > by
> > > > > > > >    polaris.iceberg-metrics.reporting.type=default. It is
> > > log-only,
> > > > > > > >    effectively quiet unless metrics logging is enabled.
> > > > > > > >    - There is also a PersistingMetricsReporter, selected by
> > > > > > > >    polaris.iceberg-metrics.reporting.type=persisting, which
> > > > converts
> > > > > > > >    Iceberg scan/commit reports into Polaris metrics records
> and
> > > > > writes
> > > > > > > > through
> > > > > > > >    PolarisMetricsManager -> MetricsPersistence.
> > > > > > > >    - MetricsPersistence currently exists in the persistence
> > layer
> > > > and
> > > > > > is
> > > > > > > >    inherited by BasePersistence.
> > > > > > > >
> > > > > > > > My read from the discussion is that the target boundary
> should
> > > not
> > > > be
> > > > > > > > MetricsPersistence as inherited by BasePersistence. That path
> > > > should
> > > > > be
> > > > > > > > decomposed as durable implementation detail (taken care of by
> > > > > > > > https://github.com/apache/polaris/pull/4397). The stable
> > > extension
> > > > > > point
> > > > > > > > should instead be the metrics reporting/emitting boundary.
> > > > > > > >
> > > > > > > > Concretely, I think the next proposal should be shaped like
> > this:
> > > > > > > >
> > > > > > > >    1.
> > > > > > > >
> > > > > > > >    Define the proper metrics reporting SPI at an optional
> > Iceberg
> > > > > > metrics
> > > > > > > >    extension API layer.
> > > > > > > >
> > > > > > > >    Example direction:
> > > > > > > >
> > > > > > > >    public interface IcebergMetricsReporter {
> > > > > > > >      void reportMetric(IcebergMetricsReportContext context,
> > > > > > > > MetricsReport report);
> > > > > > > >    }
> > > > > > > >
> > > > > > > >    The context would carry the small Polaris-resolved
> envelope:
> > > > > > > >    catalog/table identity, report type, received timestamp,
> and
> > > > > > > > request/trace
> > > > > > > >    context if available. The raw Iceberg MetricsReport
> remains
> > > the
> > > > > > > payload.
> > > > > > > >    2.
> > > > > > > >
> > > > > > > >    Keep runtime/service as the REST ingestion and wiring
> layer.
> > > > > > > >
> > > > > > > >    The REST handler still resolves the table and performs
> authz
> > > > > before
> > > > > > > >    accepting the report. After that, it calls the selected
> > > > reporting
> > > > > > > >    implementation.
> > > > > > > >    3.
> > > > > > > >
> > > > > > > >    Keep the battery default no-op/log-only.
> > > > > > > >
> > > > > > > >    This preserves an out-of-box safe default and avoids
> > > requiring a
> > > > > > > durable
> > > > > > > >    metrics store for every Polaris deployment.
> > > > > > > >    4.
> > > > > > > >
> > > > > > > >    Move durable JDBC metrics into a self-contained
> > implementation
> > > > of
> > > > > > the
> > > > > > > >    reporting SPI.
> > > > > > > >
> > > > > > > >    That implementation can own its schema, bootstrap,
> > retention,
> > > > and
> > > > > > read
> > > > > > > >    API support. It should not define the core reporting SPI
> > > > boundary,
> > > > > > and
> > > > > > > > it
> > > > > > > >    should not require metrics persistence to remain inherited
> > > from
> > > > > > > >    BasePersistence.
> > > > > > > >    5.
> > > > > > > >
> > > > > > > >    Treat async/event-backed handling as another
> implementation
> > of
> > > > the
> > > > > > > same
> > > > > > > >    reporting SPI.
> > > > > > > >
> > > > > > > >    For example, an event-backed reporter could enqueue a
> > metrics
> > > > > event
> > > > > > > and
> > > > > > > >    let listeners handle durable storage or other sinks. If we
> > > later
> > > > > > need
> > > > > > > a
> > > > > > > >    replaceable queue engine, that seems like a shared
> > > event/metrics
> > > > > > > > substrate
> > > > > > > >    topic rather than a metrics-only requirement.
> > > > > > > >
> > > > > > > > This framing lets us keep the REST metrics endpoint simple,
> > > > preserve
> > > > > > the
> > > > > > > > current default behavior, support durable metrics users, and
> > > still
> > > > > > leave
> > > > > > > > room for async/event-backed or external-queue-based
> > > > implementations.
> > > > > > > >
> > > > > > > > I think the main follow-up questions are:
> > > > > > > >
> > > > > > > >    - Do folks agree that the stable SPI boundary should be
> > > metrics
> > > > > > > >    reporting/emitting, not metrics persistence?
> > > > > > > >    - Does an optional Iceberg metrics extension API layer
> sound
> > > > like
> > > > > > the
> > > > > > > >    right home for this SPI?
> > > > > > > >    - Should the current durable metrics work be reframed as a
> > > > durable
> > > > > > > JDBC
> > > > > > > >    reference implementation of that SPI?
> > > > > > > >    - What is the smallest PR sequence to get there without
> > > blocking
> > > > > the
> > > > > > > >    current metrics work unnecessarily?
> > > > > > > >
> > > > > > > > Thanks,
> > > > > > > > -ej
> > > > > > > >
> > > > > > > > On Fri, May 15, 2026 at 5:16 PM Dmitri Bourlatchkov <
> > > > > dimas@apache.org>
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > > > Hi JB,
> > > > > > > > >
> > > > > > > > > Could you set up another meeting, please? Same time on
> > > Wednesday
> > > > as
> > > > > > > last
> > > > > > > > > time... I hope it works for everyone.
> > > > > > > > >
> > > > > > > > > Cheers,
> > > > > > > > > Dmitri.
> > > > > > > > >
> > > > > > > > > On Fri, May 15, 2026 at 8:06 PM Yufei Gu <
> > flyrain000@gmail.com
> > > >
> > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > +1 on another sync call next week.
> > > > > > > > > >
> > > > > > > > > > Yufei
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > On Fri, May 15, 2026 at 4:52 PM Dmitri Bourlatchkov <
> > > > > > > dimas@apache.org>
> > > > > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > > Hi All,
> > > > > > > > > > >
> > > > > > > > > > > WDYT about another sync call next week?
> > > > > > > > > > >
> > > > > > > > > > > Thanks,
> > > > > > > > > > > Dmitri.
> > > > > > > > > > >
> > > > > > > > > > > On Wed, May 6, 2026 at 5:29 PM Dmitri Bourlatchkov <
> > > > > > > dimas@apache.org
> > > > > > > > >
> > > > > > > > > > > wrote:
> > > > > > > > > > >
> > > > > > > > > > > > Hi EJ,
> > > > > > > > > > > >
> > > > > > > > > > > > Thanks for the summary! It covers what we discussed
> in
> > > the
> > > > > > > meeting
> > > > > > > > > very
> > > > > > > > > > > > well, IMHO.
> > > > > > > > > > > >
> > > > > > > > > > > > Looking forward to concrete PRs :)
> > > > > > > > > > > >
> > > > > > > > > > > > Cheers,
> > > > > > > > > > > > Dmitri.
> > > > > > > > > > > >
> > > > > > > > > > > > On Wed, May 6, 2026 at 5:08 PM EJ Wang <
> > > > > > > > > ej.wang.dev.interest@gmail.com
> > > > > > > > > > >
> > > > > > > > > > > > wrote:
> > > > > > > > > > > >
> > > > > > > > > > > >> Hi folks,
> > > > > > > > > > > >>
> > > > > > > > > > > >> We had a community sync earlier, thanks JB for
> > > scheduling
> > > > > it.
> > > > > > > > Notes
> > > > > > > > > > from
> > > > > > > > > > > >> the first metrics architecture sync (May 6, 10-11am
> > PT).
> > > > > > > > Discussion
> > > > > > > > > > doc
> > > > > > > > > > > >> with per-section status:
> > > > > > > > > > > >>
> > > > > > > > > > > >>
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://docs.google.com/document/d/100h7c4damrUzVuquYbBHM0EvA4LSWuW2IT2dN_7nYVA/edit?tab=t.0
> > > > > > > > > > > >>
> > > > > > > > > > > >> *The meeting covered both topics from the doc.
> > > > > Direction-level
> > > > > > > > > > alignment
> > > > > > > > > > > >> was reached on the headline pieces; details remain
> for
> > > PR
> > > > > > review
> > > > > > > > or
> > > > > > > > > > > >> follow-up sessions.*
> > > > > > > > > > > >>
> > > > > > > > > > > >> *Topic 1 — Persistence schema redesign*
> > > > > > > > > > > >> Idea-level alignment on consolidating per-type
> tables
> > > > > > > > > > > >> (scan_metrics_report,
> > > > > > > > > > > >> commit_metrics_report) into a single metrics_report
> > > table.
> > > > > The
> > > > > > > > > > > motivating
> > > > > > > > > > > >> cost is the surface area added by every new metric
> > type
> > > > > today:
> > > > > > > new
> > > > > > > > > > > table,
> > > > > > > > > > > >> SPI method, record class, model, converter, schema
> > > > > migration.
> > > > > > > > > > > >>
> > > > > > > > > > > >> Most schema details are deferred to the schema PR. A
> > few
> > > > > > > specific
> > > > > > > > > > points
> > > > > > > > > > > >> came up:
> > > > > > > > > > > >> •   metric_schema_version: Yufei prefers dropping
> it,
> > > > since
> > > > > > > there
> > > > > > > > is
> > > > > > > > > > no
> > > > > > > > > > > >> spec-level concept of metrics versioning today and
> it
> > is
> > > > > hard
> > > > > > to
> > > > > > > > > > define
> > > > > > > > > > > >> unilaterally. Robert prefers keeping it, given IRC
> v2
> > is
> > > > > > coming
> > > > > > > > and
> > > > > > > > > > the
> > > > > > > > > > > >> schema should be considered against its likely
> shape;
> > > > Robert
> > > > > > > also
> > > > > > > > > > raised
> > > > > > > > > > > >> how to differentiate various payload formats if any.
> > > EJ's
> > > > > read
> > > > > > > is
> > > > > > > > > that
> > > > > > > > > > > >> this
> > > > > > > > > > > >> is a two-way-door decision. We can start without the
> > > > field,
> > > > > > and
> > > > > > > if
> > > > > > > > > IRC
> > > > > > > > > > > v2
> > > > > > > > > > > >> changes the shape we would likely roll a
> corresponding
> > > new
> > > > > > > schema
> > > > > > > > > > > anyway,
> > > > > > > > > > > >> which is not particularly costly.
> > > > > > > > > > > >> •   Payload format: Robert pointed out that future
> > > formats
> > > > > > > beyond
> > > > > > > > > JSON
> > > > > > > > > > > may
> > > > > > > > > > > >> be worth supporting. The exact shape is deferred to
> > the
> > > > > schema
> > > > > > > > > > > discussion.
> > > > > > > > > > > >> •   Partition strategy: Anand suggested monthly
> > > > partitioning
> > > > > > > based
> > > > > > > > > on
> > > > > > > > > > > his
> > > > > > > > > > > >> experience as potentially helpful at scale.
> > > > > > > > > > > >>
> > > > > > > > > > > >> *Topic 2 — Where metrics ingestion and storage
> belong*
> > > > > > > > > > > >> Idea-level alignment that metrics should be a
> > separated
> > > > SPI
> > > > > > from
> > > > > > > > the
> > > > > > > > > > > >> entity
> > > > > > > > > > > >> persistence stack. Two reasons surfaced: (a)
> workloads
> > > and
> > > > > > > > > capability
> > > > > > > > > > > >> requirements diverge enough that coupling them
> creates
> > > > > > > artificial
> > > > > > > > > > > >> constraints, and (b) admin experience improves when
> > > > metrics
> > > > > > has
> > > > > > > > its
> > > > > > > > > > own
> > > > > > > > > > > >> bootstrap, retention, and lifecycle. Dmitri noted
> > > Polaris
> > > > > > being
> > > > > > > a
> > > > > > > > > > > platform
> > > > > > > > > > > >> should have the flexibility to support different
> > > > persistence
> > > > > > > > > backends
> > > > > > > > > > > per
> > > > > > > > > > > >> concern, and pointed to a concrete next step of
> > > separating
> > > > > the
> > > > > > > > JDBC
> > > > > > > > > > > >> bootstrap for metrics from the metastore bootstrap.
> > > Robert
> > > > > > > > proposed
> > > > > > > > > an
> > > > > > > > > > > >> additional UX extension: detect an unbootstrapped
> > > metrics
> > > > > > store
> > > > > > > on
> > > > > > > > > > first
> > > > > > > > > > > >> use and auto-bootstrap rather than requiring an
> > explicit
> > > > > > manual
> > > > > > > > > > > bootstrap
> > > > > > > > > > > >> step.
> > > > > > > > > > > >> The meeting also confirmed that Polaris metrics can
> > > start
> > > > > > small
> > > > > > > > and
> > > > > > > > > > stay
> > > > > > > > > > > >> Iceberg-focused. Naming and persistence schema can
> > lean
> > > > > > > > > > > Iceberg-specific.
> > > > > > > > > > > >> If a future expansion to generic-table metrics or
> > > > > operational
> > > > > > > > > metrics
> > > > > > > > > > > >> arrives, an abstraction layer can be built on top of
> > the
> > > > > > Iceberg
> > > > > > > > > > metrics
> > > > > > > > > > > >> reporter at that point. Robert remains on the fence
> > and
> > > > > would
> > > > > > > > prefer
> > > > > > > > > > > >> something more generic but did not block the
> > direction;
> > > > > > Dmitri's
> > > > > > > > > read
> > > > > > > > > > > was
> > > > > > > > > > > >> that the proposed framework already has enough
> > > flexibility
> > > > > to
> > > > > > > > absorb
> > > > > > > > > > > >> future
> > > > > > > > > > > >> expansion.
> > > > > > > > > > > >>
> > > > > > > > > > > >> The Trade-offs and Proposed structure sections in
> the
> > > doc
> > > > > were
> > > > > > > not
> > > > > > > > > > > >> reviewed
> > > > > > > > > > > >> in detail. They remain open for either the next sync
> > or
> > > PR
> > > > > > > review.
> > > > > > > > > > > >>
> > > > > > > > > > > >> *Cross-cutting alignment — battery-included plus
> > > > pluggable*
> > > > > > > > > > > >> A common philosophy emerged from the discussion. EJ
> > > > > summarized
> > > > > > > it
> > > > > > > > > as:
> > > > > > > > > > > >> Polaris should provide a battery-included UX for
> > > beginners
> > > > > and
> > > > > > > the
> > > > > > > > > > > >> flexibility for advanced users to swap the included
> > > > battery
> > > > > > for
> > > > > > > > > > > something
> > > > > > > > > > > >> more powerful or tailored to their use case. The SPI
> > > > design
> > > > > > > needs
> > > > > > > > to
> > > > > > > > > > > >> enable
> > > > > > > > > > > >> both.
> > > > > > > > > > > >>
> > > > > > > > > > > >> The inputs that shaped this framing:
> > > > > > > > > > > >> •   Anand described how his team uses the current
> > > metrics
> > > > > > > > > persistence
> > > > > > > > > > > >> (three metrics consumers in v1.4).
> > > > > > > > > > > >> •   Yufei raised Grafana and dashboard integrations
> > as a
> > > > > > > > destination
> > > > > > > > > > use
> > > > > > > > > > > >> case beyond the default.
> > > > > > > > > > > >> •   Robert called out that the current design is
> more
> > > > > > > > JDBC-focused.
> > > > > > > > > > > >>
> > > > > > > > > > > >> Two concrete instances:
> > > > > > > > > > > >> •   Async metrics intake: Yufei's initial position
> was
> > > > that
> > > > > > > async
> > > > > > > > > > should
> > > > > > > > > > > >> largely live on the producer side and there is not
> > much
> > > > > > Polaris
> > > > > > > > can
> > > > > > > > > > do.
> > > > > > > > > > > >> Robert suggested a Polaris-side default is doable
> via
> > > > > Vert.x.
> > > > > > > > Dmitri
> > > > > > > > > > > >> agreed
> > > > > > > > > > > >> the direction is worth exploring. The meeting
> > converged
> > > > on a
> > > > > > > > > > > >> battery-included default (likely Vert.x-backed) with
> > an
> > > > SPI
> > > > > > > shape
> > > > > > > > > that
> > > > > > > > > > > >> lets
> > > > > > > > > > > >> power users route to a more scalable backend
> > (k8s-hosted
> > > > > > queue,
> > > > > > > > AWS
> > > > > > > > > > SQS,
> > > > > > > > > > > >> etc.).
> > > > > > > > > > > >> •   Pluggable destinations: combining Yufei's
> > dashboard
> > > > use
> > > > > > case
> > > > > > > > > with
> > > > > > > > > > > >> Robert's JDBC-focused call-out, the meeting agreed
> the
> > > SPI
> > > > > > > should
> > > > > > > > be
> > > > > > > > > > > >> structured for multiple sinks so integrations become
> > > impl
> > > > > > > choices
> > > > > > > > > > rather
> > > > > > > > > > > >> than architectural changes.
> > > > > > > > > > > >>
> > > > > > > > > > > >> The battery-included default is most likely to use
> the
> > > > > > existing
> > > > > > > > > > > >> JDBC-backed
> > > > > > > > > > > >> approach.
> > > > > > > > > > > >>
> > > > > > > > > > > >> *Direction (idea-level alignment)*
> > > > > > > > > > > >> •   Single metrics_report table consolidating
> per-type
> > > > > > metrics,
> > > > > > > > > > > replacing
> > > > > > > > > > > >> scan_metrics_report and commit_metrics_report
> > > > > > > > > > > >> •   Iceberg-focused naming and schema for now,
> revisit
> > > if
> > > > > > > > > > generic-table
> > > > > > > > > > > or
> > > > > > > > > > > >> operational metrics arrive
> > > > > > > > > > > >> •   Metrics persistence as a separated SPI, not on
> > > > > > > BasePersistence
> > > > > > > > > > > >> •   Bootstrap path separated for metrics,
> independent
> > of
> > > > > > > metastore
> > > > > > > > > > > >> bootstrap
> > > > > > > > > > > >> •   "Battery-included plus pluggable" as the SPI
> > design
> > > > > > > philosophy
> > > > > > > > > > > >>
> > > > > > > > > > > >> *Open items*
> > > > > > > > > > > >> •   Schema details: metric_schema_version, payload
> > > format,
> > > > > IRC
> > > > > > > v2
> > > > > > > > > > > >> forward-compat shape
> > > > > > > > > > > >> •   SPI design details — full review either in the
> > next
> > > > sync
> > > > > > or
> > > > > > > in
> > > > > > > > > the
> > > > > > > > > > > >> corresponding PR
> > > > > > > > > > > >> •   Schema refactor PR ownership
> > > > > > > > > > > >>
> > > > > > > > > > > >> *Action items*
> > > > > > > > > > > >> •   EJ to take a first stab at the SPI design and
> > > > > potentially
> > > > > > > > > partner
> > > > > > > > > > > with
> > > > > > > > > > > >> Anand to incorporate the lessons learned from the
> > > existing
> > > > > > > > reporter
> > > > > > > > > > and
> > > > > > > > > > > >> persistence work.
> > > > > > > > > > > >> •   Schema refactor PR ownership is not yet decided.
> > If
> > > > > anyone
> > > > > > > is
> > > > > > > > > > > >> interested in driving it, reply on this thread.
> > > > > > > > > > > >> •   JB to schedule the next sync, tentatively in two
> > > > weeks.
> > > > > > > > > > > >>
> > > > > > > > > > > >> -ej
> > > > > > > > > > > >>
> > > > > > > > > > > >> On Mon, Apr 27, 2026 at 3:07 PM EJ Wang <
> > > > > > > > > > ej.wang.dev.interest@gmail.com
> > > > > > > > > > > >
> > > > > > > > > > > >> wrote:
> > > > > > > > > > > >>
> > > > > > > > > > > >> > Thanks Yufei for the +1.
> > > > > > > > > > > >> >
> > > > > > > > > > > >> > JB, could you help add a biweekly metrics
> > architecture
> > > > > sync
> > > > > > to
> > > > > > > > the
> > > > > > > > > > > >> Polaris
> > > > > > > > > > > >> > community calendar? I'm thinking Thursdays at
> 9-10am
> > > PT,
> > > > > on
> > > > > > > the
> > > > > > > > > > > >> off-weeks
> > > > > > > > > > > >> > from the community meeting (starting May 7), 60
> > > minutes.
> > > > > > > > > > > >> >
> > > > > > > > > > > >> > Here's a rough agenda to work through over the
> first
> > > few
> > > > > > > > sessions,
> > > > > > > > > > > >> grouped
> > > > > > > > > > > >> > by priority:
> > > > > > > > > > > >> >
> > > > > > > > > > > >> > *First: foundational direction*
> > > > > > > > > > > >> >
> > > > > > > > > > > >> > 1.  MetricsPersistence: public SPI or internal
> > > > > > implementation
> > > > > > > > > > detail?
> > > > > > > > > > > >> >    •   Marked @Beta, javadoc calls it a "Service
> > > > Provider
> > > > > > > > > > Interface",
> > > > > > > > > > > >> but
> > > > > > > > > > > >> > only one consumer (JdbcBasePersistenceImpl), lives
> > on
> > > > > > > > > > BasePersistence.
> > > > > > > > > > > >> If
> > > > > > > > > > > >> > demoted to a private helper inside a persisting
> > > reporter
> > > > > > impl,
> > > > > > > > > most
> > > > > > > > > > > >> > downstream design decisions become implementation
> > > > details
> > > > > > > rather
> > > > > > > > > > than
> > > > > > > > > > > >> > contract questions.
> > > > > > > > > > > >> >
> > > > > > > > > > > >> > 2.  Persistence schema redesign
> > > > > > > > > > > >> >    •   Current two-table layout
> > (scan_metrics_report,
> > > > > > > > > > > >> > commit_metrics_report) with ~25 flattened columns
> > > each.
> > > > > > Every
> > > > > > > > new
> > > > > > > > > > > metric
> > > > > > > > > > > >> > type requires a new table, SPI method, record
> class,
> > > > > model,
> > > > > > > > > > converter,
> > > > > > > > > > > >> and
> > > > > > > > > > > >> > schema migration. Direction to explore: single
> table
> > > > with
> > > > > > > > > > metric_type
> > > > > > > > > > > >> enum,
> > > > > > > > > > > >> > schema_version, and JSON payload column.
> > > > > > > > > > > >> >
> > > > > > > > > > > >> > *Second: design details once direction is set*
> > > > > > > > > > > >> >
> > > > > > > > > > > >> > 3.  Partition key strategy
> > > > > > > > > > > >> >    •   Single-table design means scan metrics at
> > scale
> > > > > will
> > > > > > > have
> > > > > > > > > > high
> > > > > > > > > > > >> > write concurrency per table. Schema needs to
> expose
> > > > enough
> > > > > > > > > structure
> > > > > > > > > > > for
> > > > > > > > > > > >> > backends to shard by entity or time range.
> > > > > > > > > > > >> >
> > > > > > > > > > > >> > 4.  Read/write path consistency
> > > > > > > > > > > >> >    •   Writes go through PolarisMetricsManager on
> > > > > > > > > MetaStoreManager.
> > > > > > > > > > > >> Reads
> > > > > > > > > > > >> > bypass MetaStoreManager and go straight to
> > > > > BasePersistence,
> > > > > > > > > > excluding
> > > > > > > > > > > >> > non-JDBC backends from the read API.
> > > > > > > > > > > >> >
> > > > > > > > > > > >> > *Third: cleanup and alignment*
> > > > > > > > > > > >> >
> > > > > > > > > > > >> > 5.  PolarisMetricsReporter naming
> > > > > > > > > > > >> >    •   Only handles IRC (ScanReport/CommitReport),
> > > > doesn't
> > > > > > > cover
> > > > > > > > > > > generic
> > > > > > > > > > > >> > tables or operational metrics. Name is broader
> than
> > > > scope.
> > > > > > > > > > > >> >
> > > > > > > > > > > >> > 6.  PolarisMetricsManager facade passthrough
> > > > > > > > > > > >> >    •   Entire default method is
> > > > > > > > > > > >> callCtx.getMetaStore().writeScanReport().
> > > > > > > > > > > >> > Zero logic, passes Level 1 straight through to
> Level
> > > 3.
> > > > > Same
> > > > > > > > > > > >> anti-pattern
> > > > > > > > > > > >> > as PolarisEventManager.
> > > > > > > > > > > >> >
> > > > > > > > > > > >> > 7.  Iceberg community alignment
> > > > > > > > > > > >> >    •   Payload-type extension needs discussion on
> > > > > > dev@iceberg.
> > > > > > > > > > > >> obelix74's
> > > > > > > > > > > >> > Feb thread got zero replies. Needs a committer
> > voice.
> > > > > > > > > > > >> >
> > > > > > > > > > > >> > Lets confirm prioritization in the first session.
> > > > > > > > > > > >> >
> > > > > > > > > > > >> > -ej
> > > > > > > > > > > >> >
> > > > > > > > > > > >> > On Tue, Apr 21, 2026 at 3:18 PM Yufei Gu <
> > > > > > > flyrain000@gmail.com>
> > > > > > > > > > > wrote:
> > > > > > > > > > > >> >
> > > > > > > > > > > >> >> Thanks everyone for continuing to drive this
> > > forward. I
> > > > > > agree
> > > > > > > > > that
> > > > > > > > > > > the
> > > > > > > > > > > >> >> problem is getting complex enough that a more
> > > > structured
> > > > > > > > > discussion
> > > > > > > > > > > >> would
> > > > > > > > > > > >> >> help.
> > > > > > > > > > > >> >>
> > > > > > > > > > > >> >> +1 on setting up a biweekly sync for the metrics
> > > > > > > architecture.
> > > > > > > > > I’m
> > > > > > > > > > > >> happy
> > > > > > > > > > > >> >> to
> > > > > > > > > > > >> >> join.
> > > > > > > > > > > >> >>
> > > > > > > > > > > >> >> Yufei
> > > > > > > > > > > >> >>
> > > > > > > > > > > >> >>
> > > > > > > > > > > >> >> On Tue, Apr 21, 2026 at 2:34 PM EJ Wang <
> > > > > > > > > > > >> ej.wang.dev.interest@gmail.com>
> > > > > > > > > > > >> >> wrote:
> > > > > > > > > > > >> >>
> > > > > > > > > > > >> >> > Also, I've been looking more closely at the
> > > > > *persistence
> > > > > > > > schema
> > > > > > > > > > in
> > > > > > > > > > > >> the
> > > > > > > > > > > >> >> > current metrics work*, and I think there's a
> > > > structural
> > > > > > > > > rigidity
> > > > > > > > > > > >> problem
> > > > > > > > > > > >> >> > worth raising before the shape gets locked in.
> > > > > > > > > > > >> >> >
> > > > > > > > > > > >> >> > Right now we have two separate tables
> > > > > > (scan_metrics_report
> > > > > > > > and
> > > > > > > > > > > >> >> > commit_metrics_report), each with ~25 flattened
> > > > columns
> > > > > > > that
> > > > > > > > > > > directly
> > > > > > > > > > > >> >> > mirror the Iceberg report fields. The SPI
> follows
> > > the
> > > > > > same
> > > > > > > > > split:
> > > > > > > > > > > >> >> > writeScanReport and writeCommitReport as
> separate
> > > > > > methods,
> > > > > > > > with
> > > > > > > > > > > >> per-type
> > > > > > > > > > > >> >> > record classes, converters, and model objects.
> > *The
> > > > > > > practical
> > > > > > > > > > cost:
> > > > > > > > > > > >> >> > adding a new metric type (operational metrics,
> > for
> > > > > > example)
> > > > > > > > > > > requires
> > > > > > > > > > > >> a
> > > > > > > > > > > >> >> new
> > > > > > > > > > > >> >> > table, a new SPI method, a new record class, a
> > new
> > > > > model
> > > > > > > > > class, a
> > > > > > > > > > > new
> > > > > > > > > > > >> >> > converter branch, and a schema migration*.
> > That's a
> > > > lot
> > > > > > of
> > > > > > > > > > surface
> > > > > > > > > > > >> area
> > > > > > > > > > > >> >> > for what should be "one more kind of metric."
> > > > > > > > > > > >> >> >
> > > > > > > > > > > >> >> > *My bias* would be toward a single metrics
> table
> > > with
> > > > > *a
> > > > > > > > typed
> > > > > > > > > > JSON
> > > > > > > > > > > >> >> > payload*. Something like: metric_type (enum),
> > > > > entity_id,
> > > > > > > > > > > >> >> > table_identifier, snapshot_id (nullable),
> > > > received_ts,
> > > > > > > > > > > >> schema_version,
> > > > > > > > > > > >> >> and
> > > > > > > > > > > >> >> > a payload column for the metric-specific data.
> > The
> > > > > > > > metric_type
> > > > > > > > > +
> > > > > > > > > > > >> >> > schema_version pair gives us a
> forward-compatible
> > > > > > contract
> > > > > > > > for
> > > > > > > > > > the
> > > > > > > > > > > >> >> payload
> > > > > > > > > > > >> >> > shape. Adding a new metric type becomes an enum
> > > value
> > > > > > and a
> > > > > > > > > > payload
> > > > > > > > > > > >> >> schema,
> > > > > > > > > > > >> >> > not a schema migration. One thing I think we
> need
> > > to
> > > > be
> > > > > > > > > > deliberate
> > > > > > > > > > > >> >> about is
> > > > > > > > > > > >> >> > the partition key design. If all metric types
> > land
> > > in
> > > > > one
> > > > > > > > > table,
> > > > > > > > > > > scan
> > > > > > > > > > > >> >> > metrics at scale (high concurrency, high
> > frequency
> > > > > across
> > > > > > > > many
> > > > > > > > > > > >> tables)
> > > > > > > > > > > >> >> > could easily create hot partitions. We'd want
> the
> > > > > > > persistence
> > > > > > > > > > layer
> > > > > > > > > > > >> to
> > > > > > > > > > > >> >> be
> > > > > > > > > > > >> >> > able to shard by entity or time range, and that
> > > means
> > > > > the
> > > > > > > > > logical
> > > > > > > > > > > >> schema
> > > > > > > > > > > >> >> > needs to expose enough structure for backends
> to
> > > > > > partition
> > > > > > > > on.
> > > > > > > > > I
> > > > > > > > > > > >> don't
> > > > > > > > > > > >> >> > think the current flattened layout gives us
> that.
> > > > > > > > > > > >> >> >
> > > > > > > > > > > >> >> > This is getting complex enough that I don't
> think
> > > > > ad-hoc
> > > > > > > > PR/ML
> > > > > > > > > > > >> threads
> > > > > > > > > > > >> >> > will converge well. *Would people be open to a
> > > > biweekly
> > > > > > > sync
> > > > > > > > > for
> > > > > > > > > > > >> metrics
> > > > > > > > > > > >> >> > architecture?* I think 30 minutes every two
> weeks
> > > > with
> > > > > > > > > interested
> > > > > > > > > > > >> >> parties
> > > > > > > > > > > >> >> > would be enough to work through the schema, SPI
> > > > shape,
> > > > > > and
> > > > > > > > read
> > > > > > > > > > API
> > > > > > > > > > > >> >> design
> > > > > > > > > > > >> >> > together. Happy to help set that up.
> > > > > > > > > > > >> >> >
> > > > > > > > > > > >> >> > -ej
> > > > > > > > > > > >> >> >
> > > > > > > > > > > >> >> > On Mon, Apr 20, 2026 at 2:19 PM EJ Wang <
> > > > > > > > > > > >> ej.wang.dev.interest@gmail.com
> > > > > > > > > > > >> >> >
> > > > > > > > > > > >> >> > wrote:
> > > > > > > > > > > >> >> >
> > > > > > > > > > > >> >> >> Reviewed #4115, left a comment on the code
> > > > > organization
> > > > > > > > side.
> > > > > > > > > > > >> >> >>
> > > > > > > > > > > >> >> >> One thing stood out: the metrics write path
> > enters
> > > > > > through
> > > > > > > > > > > >> >> >> PolarisMetricsManager on MetaStoreManager, but
> > the
> > > > new
> > > > > > > read
> > > > > > > > > path
> > > > > > > > > > > >> >> bypasses
> > > > > > > > > > > >> >> >> MetaStoreManager entirely and goes straight to
> > > > > > > > BasePersistence
> > > > > > > > > > via
> > > > > > > > > > > >> >> >> callContext.getMetaStore(). That means the
> read
> > > API
> > > > > only
> > > > > > > > works
> > > > > > > > > > for
> > > > > > > > > > > >> >> backends
> > > > > > > > > > > >> >> >> that implement BasePersistence. NoSQL and
> remote
> > > > > > backends
> > > > > > > > > can't
> > > > > > > > > > > >> >> participate.
> > > > > > > > > > > >> >> >>
> > > > > > > > > > > >> >> >> Stepping back, I think the metrics subsystem
> is
> > > > > growing
> > > > > > > into
> > > > > > > > > > > >> something
> > > > > > > > > > > >> >> >> real (write + read + REST API + AuthZ +
> > > pagination)
> > > > > *but
> > > > > > > the
> > > > > > > > > > > >> >> persistence
> > > > > > > > > > > >> >> >> side is split across two layers in a way
> that's
> > > hard
> > > > > to
> > > > > > > > > > extend*. I
> > > > > > > > > > > >> put
> > > > > > > > > > > >> >> >> together two diagrams to show what I mean (my
> > best
> > > > > > > effort).
> > > > > > > > > > > >> >> >>
> > > > > > > > > > > >> >> >> *Current state* (Diagram 1): three interfaces
> at
> > > > three
> > > > > > > > > different
> > > > > > > > > > > >> >> levels.
> > > > > > > > > > > >> >> >> The engine-facing SPI (PolarisMetricsReporter)
> > is
> > > > > clean.
> > > > > > > But
> > > > > > > > > > > >> >> >> PolarisMetricsManager on MetaStoreManager is a
> > > > > > passthrough
> > > > > > > > to
> > > > > > > > > > > >> >> >> MetricsPersistence on BasePersistence. The
> @Beta
> > > > > > > annotation
> > > > > > > > > and
> > > > > > > > > > > SPI
> > > > > > > > > > > >> >> javadoc
> > > > > > > > > > > >> >> >> are on the BasePersistence layer, while the
> > actual
> > > > > > > extension
> > > > > > > > > > > points
> > > > > > > > > > > >> >> >> (PolarisMetricsReporter,
> PolarisMetricsManager)
> > > > carry
> > > > > no
> > > > > > > > > > stability
> > > > > > > > > > > >> >> >> annotation. The write path goes through the
> > > > > > > MetaStoreManager
> > > > > > > > > > > layer,
> > > > > > > > > > > >> the
> > > > > > > > > > > >> >> >> read path doesn't.
> > > > > > > > > > > >> >> >>
> > > > > > > > > > > >> >> >> *What I envision* (Diagram 2): two SPIs at two
> > > > levels.
> > > > > > > > > > > >> >> >> PolarisMetricsReporter stays as the
> > engine-facing
> > > > SPI.
> > > > > > > > > > > >> >> >> PolarisMetricsManager becomes the
> backend-facing
> > > SPI
> > > > > > with
> > > > > > > > both
> > > > > > > > > > > write
> > > > > > > > > > > >> >> and
> > > > > > > > > > > >> >> >> read methods at the MetaStoreManager level,
> > where
> > > > any
> > > > > > > > backend
> > > > > > > > > > > (JDBC,
> > > > > > > > > > > >> >> NoSQL,
> > > > > > > > > > > >> >> >> remote) can implement them. MetricsPersistence
> > on
> > > > > > > > > > BasePersistence
> > > > > > > > > > > >> goes
> > > > > > > > > > > >> >> >> away. Where metrics actually land is an
> > > > implementation
> > > > > > > > detail,
> > > > > > > > > > > not a
> > > > > > > > > > > >> >> core
> > > > > > > > > > > >> >> >> interface.
> > > > > > > > > > > >> >> >>
> > > > > > > > > > > >> >> >> *Minor naming thing*: PolarisMetricsReporter
> is
> > > > > broader
> > > > > > > than
> > > > > > > > > > what
> > > > > > > > > > > it
> > > > > > > > > > > >> >> >> actually handles. It only accepts Iceberg REST
> > > > Catalog
> > > > > > > > metrics
> > > > > > > > > > > >> >> (ScanReport,
> > > > > > > > > > > >> >> >> CommitReport via MetricsReport). Generic table
> > > > metrics
> > > > > > or
> > > > > > > > > > > >> operational
> > > > > > > > > > > >> >> >> metrics aren't in scope. Not blocking, but
> worth
> > > > > noting
> > > > > > if
> > > > > > > > the
> > > > > > > > > > > >> metrics
> > > > > > > > > > > >> >> >> surface expands.
> > > > > > > > > > > >> >> >>
> > > > > > > > > > > >> >> >> *Rough sketch of how to get there*:
> > > > > > > > > > > >> >> >>  1.  Add read methods to PolarisMetricsManager
> > > > > > > > > (listScanReports,
> > > > > > > > > > > >> >> >> listCommitReports) with default no-op, same as
> > the
> > > > > > > existing
> > > > > > > > > > write
> > > > > > > > > > > >> >> methods.
> > > > > > > > > > > >> >> >> (Probably make PolarisMetricsManager more
> > explicit
> > > > on
> > > > > > > being
> > > > > > > > > > > Iceberg
> > > > > > > > > > > >> >> >> specific like package name or class name etc.)
> > > > > > > > > > > >> >> >>  2.  Wire MetricsReportsService through
> > > > > MetaStoreManager
> > > > > > > > > instead
> > > > > > > > > > > of
> > > > > > > > > > > >> >> >> callContext.getMetaStore().
> > > > > > > > > > > >> >> >>  3.  Extract metrics persistence from
> > > > > > > > JdbcBasePersistenceImpl
> > > > > > > > > > into
> > > > > > > > > > > >> its
> > > > > > > > > > > >> >> >> own class. That file carries ~7
> > responsibilities,
> > > > > > metrics
> > > > > > > > > being
> > > > > > > > > > > one
> > > > > > > > > > > >> of
> > > > > > > > > > > >> >> them.
> > > > > > > > > > > >> >> >>  4.  Remove MetricsPersistence from
> > > BasePersistence.
> > > > > > > > > > > >> >> >>
> > > > > > > > > > > >> >> >> *None of this needs to happen in #4115. But if
> > the
> > > > > > > direction
> > > > > > > > > > makes
> > > > > > > > > > > >> >> sense,
> > > > > > > > > > > >> >> >> it would be good to align before the metrics
> > > surface
> > > > > > grows
> > > > > > > > > > > further.
> > > > > > > > > > > >> >> Curious
> > > > > > > > > > > >> >> >> what others think.*
> > > > > > > > > > > >> >> >>
> > > > > > > > > > > >> >> >> *My mental model note*: Level 1
> > MetaStoreManager;
> > > > > level
> > > > > > 2
> > > > > > > > > > > >> transactional
> > > > > > > > > > > >> >> >> persistence; level 3 base persistence
> > > > > > > > > > > >> >> >>
> > > > > > > > > > > >> >> >> Diagram 1
> > > > > > > > > > > >> >> >> <
> > > > > > > > > > > >> >>
> > > > > > > > > > > >>
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://www.plantuml.com/plantuml/uml/bLHDR-Cs4BthLmpIYupw0zbkKQ1r3M-S7Bp8xhhM7WCOb3IM65EaGD9EX2RzxHrHb4CxRelwa4YSDu_lpOVcnZ9jzvM8BBS2uGjQpJC3dtHMSekPtMk44IpsMgEqa5XcCOhCZikQQLP1pR8TAp2n3ILhmZDP20m0fcIvUkAoW2qJXd9z1bpToO9BX3WXu0ucy5rpgGPNm0nW5_epUWtm2Ue3pn3kMOFQmKntGZW0BYtgBSi8k5A2QMwybJNMIbFiGSR9QZc4nUqIvikStF0jHprua5C-amge42aNt3R0f5JaaoivdV2Pkqbx4hee4ymOkBh5BTiB-_uIeGeo8zL8rPsPl4DktdEiK1jkB1NdZCRbrSTecDe_mlHbF0wvBmCkaOH5_S8a_TTTKI6-nmCAkEw4LpxsZ-LbYLKQFKMNOgf_wuM7_bV9gOer5SYMMksBSWXFcbi49KNZXNLicwfe3TETC7gPdPqI7uBcHMb1RSzYq34c6PDUM9mn8HRsUTZEiDBve3NjVZumBj0U7SS37mGO7vcwtiK-_pU7U7L_f-digo9YbhSwIfMRwIITKGXbxdIUTCGF1SeCJxloKsU-3k9ddRbX1eDq1q_fx1JbBGT0glVyXimDuP4TQ5qpCAmnGEj2s_6n5mtn1z-97-63itFQZLPO1Ev2tu_WF7Ju-VPc0Skg5bYXxBhkY1xpD7EM_7fyflSpIsqMgVth5xhVr4eQxWQ8enaSAJQSG16yFSDuJ798rrcXr_3n-lfdk7icQjEBmFujL7AodiP_Y4Z7-YxvtZNs4zMgpNTl6tF8sglyPsmqchrjvQ-m-aP94r-TwCA2Ka8upPJZwtvSpoYCXkYMZU2NXvRMBfq9P3i3Le4VAZUAlUZ_oPKsxPgY0Q_BSKLkyr9bhQhQrJjo_x3TPlIB0DPjnMfcIoYP0QaYw1a0fTKDr8fB6ntNuvmoL1ZGkXa69Njh43zf9GiGxHQrA_jDYWRSzF5--WmTVrN97_Sm8LbLUy_lGBmLanJjFkDlGkRqjA_4tm00
> > > > > > > > > > > >> >> >
> > > > > > > > > > > >> >> >> :
> > > > > > > > > > > >> >> >>
> > > > > > > > > > > >> >> >> [image: image.png]
> > > > > > > > > > > >> >> >>
> > > > > > > > > > > >> >> >> Diagram 2
> > > > > > > > > > > >> >> >> <
> > > > > > > > > > > >> >>
> > > > > > > > > > > >>
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://www.plantuml.com/plantuml/uml/VLLDR-8m4BtdLupO2sWBLVU8AaGB7AXAbssGzb896SS9RXqxjKqBwkv_tt7iV43fdaZYDpFlpRmnOsE9jhjSH9PRmM31hERKm8scMsuPjJlDe0yheZDc8RR4iYWoBrmMH9CS2a9VICPYUy1OZN0YCy5Q0BCbYNhdCeEK28En8G8wCvbnoQ0R8_05Bc6bkLIz3X03p1zzH7zR-9ZfDquPt9C3qoNCX2yV4G2NbkcKu5jdgGJHt0GbZwnG6i-UP3TUpk5gM6Ldqke350eZUqzoCft3U9xWHvxoa5-7K4nF1J46EbEMafsmdrCBbQ44gVggy18IZrn_ph5asd1ZiIKdQSgueZvjXrQFSFrdC3YN-nXmBacxbGiYyLVxLaBtdhqn0LSzdBDhqQtQoOJeGyad3z0lUqnYgpGB6Ns8oVyta00Dy_WnX0tIOZ8v6SYxHll1TrH6aejAik-mh-AphVFCwSUQqFypElag5QRGFDjQKEd96K1P8QP41c9TzA_IIQyvdAWyv_RSiS3skb0_EzDDkK2v5xWF6MiGFlvhpFLcD2Dq2pml14gaF67eQkmd8gulDoC4kSOu6KVpkvlUJg1RTbWISU40RdBUUS_9XfRZ2dwxm_SW8LYFISgm_MnlDQ6M9P1gbKEc4X-2pH_FvJCkCqm9pbVjD6LrwdLeOrDWfOaqc8Wh9BE85oNKxkNQ6o4yGRy_Eae0G_G8tZv81d3bHDB23WOdisohVr3nh_j6lbSjbNaLRTc8UgtPbAU1J_tygOfZX9DWEJeHDvYx-qmSi5FgNLPZwHrHcUsncGQ5-skhUclpE5fo4ounpFauYrUbkU6ccfnxMvitwag4IyerhTxj8In_Oj1bDO4pQru674loYrGlULHLEGCjwJJ8gDoVZR8MxO4BT3IzRvIcAQKezC6xpziGnTyImrfEGyJI_OcKfgtxIvnTqFEMS17L9Z-jsARN5FmTheP7HtSdtOMT0B4GY2FYHXxgQmMtj2bRqiLFGapiVe1_QVKDrkqXcm83aFEXnMYCZ-xlyHy
> > > > > > > > > > > >> >> >
> > > > > > > > > > > >> >> >> :
> > > > > > > > > > > >> >> >> [image: image.png]
> > > > > > > > > > > >> >> >>
> > > > > > > > > > > >> >> >>  -ej
> > > > > > > > > > > >> >> >>
> > > > > > > > > > > >> >> >> On Wed, Apr 15, 2026 at 8:22 AM Dmitri
> > > Bourlatchkov
> > > > <
> > > > > > > > > > > >> dimas@apache.org>
> > > > > > > > > > > >> >> >> wrote:
> > > > > > > > > > > >> >> >>
> > > > > > > > > > > >> >> >>> Hi All,
> > > > > > > > > > > >> >> >>>
> > > > > > > > > > > >> >> >>> Heads up: The current state of PR [4115]
> looks
> > > > pretty
> > > > > > > solid
> > > > > > > > > to
> > > > > > > > > > > me.
> > > > > > > > > > > >> I
> > > > > > > > > > > >> >> >>> believe this PR is approaching a mergeable
> > > > condition.
> > > > > > > > > > > >> >> >>>
> > > > > > > > > > > >> >> >>> Please post your reviews if you have any
> > > comments.
> > > > > > > > > > > >> >> >>>
> > > > > > > > > > > >> >> >>> [4115]
> > > https://github.com/apache/polaris/pull/4115
> > > > > > > > > > > >> >> >>>
> > > > > > > > > > > >> >> >>> Thanks,
> > > > > > > > > > > >> >> >>> Dmitri.
> > > > > > > > > > > >> >> >>>
> > > > > > > > > > > >> >> >>> On Tue, Mar 3, 2026 at 3:29 PM Anand Kumar
> > > Sankaran
> > > > > via
> > > > > > > > dev <
> > > > > > > > > > > >> >> >>> dev@polaris.apache.org> wrote:
> > > > > > > > > > > >> >> >>>
> > > > > > > > > > > >> >> >>> > Hi Yufei and Dmitri,
> > > > > > > > > > > >> >> >>> >
> > > > > > > > > > > >> >> >>> > Here is a proposal for the REST endpoints
> for
> > > > > metrics
> > > > > > > and
> > > > > > > > > > > events.
> > > > > > > > > > > >> >> >>> >
> > > > > > > > > > > >> >> >>> >
> > > > > https://github.com/apache/polaris/pull/3924/changes
> > > > > > > > > > > >> >> >>> >
> > > > > > > > > > > >> >> >>> > I did not see any precursors for raising a
> PR
> > > for
> > > > > > > > > proposals,
> > > > > > > > > > so
> > > > > > > > > > > >> >> trying
> > > > > > > > > > > >> >> >>> > this.  Please let me know what you think.
> > > > > > > > > > > >> >> >>> >
> > > > > > > > > > > >> >> >>> > -
> > > > > > > > > > > >> >> >>> > Anand
> > > > > > > > > > > >> >> >>> >
> > > > > > > > > > > >> >> >>> > From: Anand Kumar Sankaran <
> > > > > > anand.sankaran@workday.com
> > > > > > > >
> > > > > > > > > > > >> >> >>> > Date: Monday, March 2, 2026 at 10:25 AM
> > > > > > > > > > > >> >> >>> > To: dev@polaris.apache.org <
> > > > dev@polaris.apache.org
> > > > > >
> > > > > > > > > > > >> >> >>> > Subject: Re: Polaris Telemetry and Audit
> > Trail
> > > > > > > > > > > >> >> >>> >
> > > > > > > > > > > >> >> >>> > About the REST API, based on my use cases:
> > > > > > > > > > > >> >> >>> >
> > > > > > > > > > > >> >> >>> >
> > > > > > > > > > > >> >> >>> >   1.
> > > > > > > > > > > >> >> >>> > I want to be able to query commit metrics
> to
> > > > track
> > > > > > > files
> > > > > > > > > > added
> > > > > > > > > > > /
> > > > > > > > > > > >> >> >>> removed
> > > > > > > > > > > >> >> >>> > per commit, along with record counts. The
> > > > ingestion
> > > > > > > > > pipeline
> > > > > > > > > > > that
> > > > > > > > > > > >> >> >>> writes
> > > > > > > > > > > >> >> >>> > this data is owned by us and we are
> > guaranteed
> > > to
> > > > > > write
> > > > > > > > > this
> > > > > > > > > > > >> >> >>> information
> > > > > > > > > > > >> >> >>> > for each write.
> > > > > > > > > > > >> >> >>> >   2.
> > > > > > > > > > > >> >> >>> > I want to be able to query scan metrics for
> > > > read. I
> > > > > > > > > > understand
> > > > > > > > > > > >> >> clients
> > > > > > > > > > > >> >> >>> do
> > > > > > > > > > > >> >> >>> > not fulfill this requirement.
> > > > > > > > > > > >> >> >>> >   3.
> > > > > > > > > > > >> >> >>> > I want to be able to query the events table
> > > > (events
> > > > > > are
> > > > > > > > > > > >> persisted) -
> > > > > > > > > > > >> >> >>> this
> > > > > > > > > > > >> >> >>> > may supersede #2, I am not sure yet.
> > > > > > > > > > > >> >> >>> >
> > > > > > > > > > > >> >> >>> > All this information is in the JDBC based
> > > > > persistence
> > > > > > > > model
> > > > > > > > > > and
> > > > > > > > > > > >> is
> > > > > > > > > > > >> >> >>> > persisted in the metastore. I currently
> don’t
> > > > have
> > > > > a
> > > > > > > need
> > > > > > > > > to
> > > > > > > > > > > >> query
> > > > > > > > > > > >> >> >>> > prometheus or open telemetry. I do publish
> > some
> > > > > > events
> > > > > > > to
> > > > > > > > > > > >> Prometheus
> > > > > > > > > > > >> >> >>> and
> > > > > > > > > > > >> >> >>> > they are forwarded to our dashboards
> > elsewhere.
> > > > > > > > > > > >> >> >>> >
> > > > > > > > > > > >> >> >>> > About the CLI utilities, I meant the admin
> > user
> > > > > > > > utilities.
> > > > > > > > > In
> > > > > > > > > > > >> one of
> > > > > > > > > > > >> >> >>> the
> > > > > > > > > > > >> >> >>> > earliest drafts of my proposal, Prashant
> > > > mentioned
> > > > > > that
> > > > > > > > the
> > > > > > > > > > > >> metrics
> > > > > > > > > > > >> >> >>> tables
> > > > > > > > > > > >> >> >>> > can grow indefinitely and that a similar
> > > problem
> > > > > > exists
> > > > > > > > > with
> > > > > > > > > > > the
> > > > > > > > > > > >> >> events
> > > > > > > > > > > >> >> >>> > table as well. We discussed that cleaning
> up
> > of
> > > > old
> > > > > > > > records
> > > > > > > > > > > from
> > > > > > > > > > > >> >> both
> > > > > > > > > > > >> >> >>> > metrics tables and events tables can be
> done
> > > via
> > > > a
> > > > > > CLI
> > > > > > > > > > utility.
> > > > > > > > > > > >> >> >>> >
> > > > > > > > > > > >> >> >>> > I see that Yufei has covered the discussion
> > > about
> > > > > > > > > > datasources.
> > > > > > > > > > > >> >> >>> >
> > > > > > > > > > > >> >> >>> > -
> > > > > > > > > > > >> >> >>> > Anand
> > > > > > > > > > > >> >> >>> >
> > > > > > > > > > > >> >> >>> >
> > > > > > > > > > > >> >> >>> >
> > > > > > > > > > > >> >> >>> > From: Yufei Gu <fl...@gmail.com>
> > > > > > > > > > > >> >> >>> > Date: Friday, February 27, 2026 at 9:54 PM
> > > > > > > > > > > >> >> >>> > To: dev@polaris.apache.org <
> > > > dev@polaris.apache.org
> > > > > >
> > > > > > > > > > > >> >> >>> > Subject: Re: Polaris Telemetry and Audit
> > Trail
> > > > > > > > > > > >> >> >>> >
> > > > > > > > > > > >> >> >>> > This Message Is From an External Sender
> > > > > > > > > > > >> >> >>> > This message came from outside your
> > > organization.
> > > > > > > > > > > >> >> >>> > Report Suspicious<
> > > > > > > > > > > >> >> >>> >
> > > > > > > > > > > >> >> >>>
> > > > > > > > > > > >> >>
> > > > > > > > > > > >>
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://us-phishalarm-ewt.proofpoint.com/EWT/v1/Iz9xO38YGHZK!YhNDZABkHi1B699ote2uMwpOZw8i0QMCGO2Szc-HshuABGhGvwPJcymE6G2oUUxtS8xDkSrtGTPm_I3QnVDHoLMk50m9v8z_nZKTkd-bnVUbreF1u0WnfV_X5eYevZl_$
> > > > > > > > > > > >> >> >>> > >
> > > > > > > > > > > >> >> >>> >
> > > > > > > > > > > >> >> >>> >
> > > > > > > > > > > >> >> >>> > As I mentioned in
> > > > > > > > > > > >> >> >>> >
> > > > > > > > > > > >> >> >>>
> > > > > > > > > > > >> >>
> > > > > > > > > > > >>
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://urldefense.com/v3/__https://github.com/apache/polaris/issues/3890__;!!Iz9xO38YGHZK!5EuyFFkk3vhRWVIRvQAWBSQfpJkTMA9HxugzDwXmN0LPPqhEFxYkFRGVhtb8AqUwXtDh2OplcMnbMDHKOxrvDU0$
> > > > > > > > > > > >> >> >>> ,
> > > > > > > > > > > >> >> >>> > supporting
> > > > > > > > > > > >> >> >>> > multiple data sources is not a trivial
> > change.
> > > I
> > > > > > would
> > > > > > > > > > strongly
> > > > > > > > > > > >> >> >>> recommend
> > > > > > > > > > > >> >> >>> > starting with a design document to
> carefully
> > > > > evaluate
> > > > > > > the
> > > > > > > > > > > >> >> architectural
> > > > > > > > > > > >> >> >>> > implications and long term impact.
> > > > > > > > > > > >> >> >>> >
> > > > > > > > > > > >> >> >>> > A REST endpoint to query metrics seems
> > > reasonable
> > > > > > given
> > > > > > > > the
> > > > > > > > > > > >> current
> > > > > > > > > > > >> >> >>> JDBC
> > > > > > > > > > > >> >> >>> > based persistence model. That said, we may
> > also
> > > > > > > consider
> > > > > > > > > > > >> alternative
> > > > > > > > > > > >> >> >>> > storage models. For example, if we later
> > adopt
> > > a
> > > > > time
> > > > > > > > > series
> > > > > > > > > > > >> system
> > > > > > > > > > > >> >> >>> such as
> > > > > > > > > > > >> >> >>> > Prometheus to store metrics, the query
> model
> > > and
> > > > > > access
> > > > > > > > > > > patterns
> > > > > > > > > > > >> >> would
> > > > > > > > > > > >> >> >>> be
> > > > > > > > > > > >> >> >>> > fundamentally different. Designing the REST
> > API
> > > > > > without
> > > > > > > > > > > >> considering
> > > > > > > > > > > >> >> >>> these
> > > > > > > > > > > >> >> >>> > potential evolutions may limit flexibility.
> > I'd
> > > > > > suggest
> > > > > > > > to
> > > > > > > > > > > start
> > > > > > > > > > > >> >> with
> > > > > > > > > > > >> >> >>> the
> > > > > > > > > > > >> >> >>> > use case.
> > > > > > > > > > > >> >> >>> >
> > > > > > > > > > > >> >> >>> > Yufei
> > > > > > > > > > > >> >> >>> >
> > > > > > > > > > > >> >> >>> >
> > > > > > > > > > > >> >> >>> > On Fri, Feb 27, 2026 at 3:42 PM Dmitri
> > > > > Bourlatchkov <
> > > > > > > > > > > >> >> dimas@apache.org>
> > > > > > > > > > > >> >> >>> > wrote:
> > > > > > > > > > > >> >> >>> >
> > > > > > > > > > > >> >> >>> > > Hi Anand,
> > > > > > > > > > > >> >> >>> > >
> > > > > > > > > > > >> >> >>> > > Sharing my view... subject to discussion:
> > > > > > > > > > > >> >> >>> > >
> > > > > > > > > > > >> >> >>> > > 1. Adding non-IRC REST API to Polaris is
> > > > > perfectly
> > > > > > > > fine.
> > > > > > > > > > > >> >> >>> > >
> > > > > > > > > > > >> >> >>> > > Figuring out specific endpoint URIs and
> > > > payloads
> > > > > > > might
> > > > > > > > > > > require
> > > > > > > > > > > >> a
> > > > > > > > > > > >> >> few
> > > > > > > > > > > >> >> >>> > > roundtrips, so opening a separate thread
> > for
> > > > that
> > > > > > > might
> > > > > > > > > be
> > > > > > > > > > > >> best.
> > > > > > > > > > > >> >> >>> > > Contributors commonly create Google Docs
> > for
> > > > new
> > > > > > API
> > > > > > > > > > > proposals
> > > > > > > > > > > >> too
> > > > > > > > > > > >> >> >>> (they
> > > > > > > > > > > >> >> >>> > > fairly easy to update as the email
> > discussion
> > > > > > > > > progresses).
> > > > > > > > > > > >> >> >>> > >
> > > > > > > > > > > >> >> >>> > > There was a suggestion to try Markdown
> > (with
> > > > PRs)
> > > > > > for
> > > > > > > > > > > proposals
> > > > > > > > > > > >> >> [1]
> > > > > > > > > > > >> >> >>> ...
> > > > > > > > > > > >> >> >>> > > feel free to give it a try if you are
> > > > comfortable
> > > > > > > with
> > > > > > > > > > that.
> > > > > > > > > > > >> >> >>> > >
> > > > > > > > > > > >> >> >>> > > 2. Could you clarify whether you mean end
> > > user
> > > > > > > > utilities
> > > > > > > > > or
> > > > > > > > > > > >> admin
> > > > > > > > > > > >> >> >>> user
> > > > > > > > > > > >> >> >>> > > utilities? In the latter case those might
> > be
> > > > more
> > > > > > > > > suitable
> > > > > > > > > > > for
> > > > > > > > > > > >> the
> > > > > > > > > > > >> >> >>> Admin
> > > > > > > > > > > >> >> >>> > > CLI (java) not the Python CLI, IMHO.
> > > > > > > > > > > >> >> >>> > >
> > > > > > > > > > > >> >> >>> > > Why would these utilities be common with
> > > > events?
> > > > > > > IMHO,
> > > > > > > > > > event
> > > > > > > > > > > >> use
> > > > > > > > > > > >> >> >>> cases
> > > > > > > > > > > >> >> >>> > are
> > > > > > > > > > > >> >> >>> > > distinct from scan/commit metrics.
> > > > > > > > > > > >> >> >>> > >
> > > > > > > > > > > >> >> >>> > > 3. I'd prefer separating metrics
> > persistence
> > > > from
> > > > > > > > > MetaStore
> > > > > > > > > > > >> >> >>> persistence
> > > > > > > > > > > >> >> >>> > at
> > > > > > > > > > > >> >> >>> > > the code level, so that they could be
> mixed
> > > and
> > > > > > > matched
> > > > > > > > > > > >> >> >>> independently.
> > > > > > > > > > > >> >> >>> > The
> > > > > > > > > > > >> >> >>> > > separate datasource question will become
> a
> > > > > > non-issue
> > > > > > > > with
> > > > > > > > > > > that
> > > > > > > > > > > >> >> >>> approach,
> > > > > > > > > > > >> >> >>> > I
> > > > > > > > > > > >> >> >>> > > guess.
> > > > > > > > > > > >> >> >>> > >
> > > > > > > > > > > >> >> >>> > > The rationale for separating scan metrics
> > and
> > > > > > > metastore
> > > > > > > > > > > >> >> persistence
> > > > > > > > > > > >> >> >>> is
> > > > > > > > > > > >> >> >>> > that
> > > > > > > > > > > >> >> >>> > > "cascading deletes" between them are
> hardly
> > > > ever
> > > > > > > > > required.
> > > > > > > > > > > >> >> >>> Furthermore,
> > > > > > > > > > > >> >> >>> > the
> > > > > > > > > > > >> >> >>> > > data and query patterns are very
> different
> > so
> > > > > > > different
> > > > > > > > > > > >> >> technologies
> > > > > > > > > > > >> >> >>> > might
> > > > > > > > > > > >> >> >>> > > be beneficial in each case.
> > > > > > > > > > > >> >> >>> > >
> > > > > > > > > > > >> >> >>> > > [1]
> > > > > > > > > > > >> >> >>> >
> > > > > > > > > > > >> >> >>>
> > > > > > > > > > > >> >>
> > > > > > > > > > > >>
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://urldefense.com/v3/__https://lists.apache.org/thread/yto2wp982t43h1mqjwnslswhws5z47cy__;!!Iz9xO38YGHZK!5EuyFFkk3vhRWVIRvQAWBSQfpJkTMA9HxugzDwXmN0LPPqhEFxYkFRGVhtb8AqUwXtDh2OplcMnbMDHKxYDakNU$
> > > > > > > > > > > >> >> >>> > >
> > > > > > > > > > > >> >> >>> > > Cheers,
> > > > > > > > > > > >> >> >>> > > Dmitri.
> > > > > > > > > > > >> >> >>> > >
> > > > > > > > > > > >> >> >>> > > On Fri, Feb 27, 2026 at 6:19 PM Anand
> Kumar
> > > > > > Sankaran
> > > > > > > > via
> > > > > > > > > > dev
> > > > > > > > > > > <
> > > > > > > > > > > >> >> >>> > > dev@polaris.apache.org> wrote:
> > > > > > > > > > > >> >> >>> > >
> > > > > > > > > > > >> >> >>> > > > Thanks all. This PR is merged now.
> > > > > > > > > > > >> >> >>> > > >
> > > > > > > > > > > >> >> >>> > > > Here are the follow-up features / work
> > > > needed.
> > > > > > > These
> > > > > > > > > > were
> > > > > > > > > > > >> all
> > > > > > > > > > > >> >> >>> part of
> > > > > > > > > > > >> >> >>> > > the
> > > > > > > > > > > >> >> >>> > > > merged PR at some point in time and
> were
> > > > > removed
> > > > > > to
> > > > > > > > > > reduce
> > > > > > > > > > > >> >> scope.
> > > > > > > > > > > >> >> >>> > > >
> > > > > > > > > > > >> >> >>> > > > Please let me know what you think.
> > > > > > > > > > > >> >> >>> > > >
> > > > > > > > > > > >> >> >>> > > >
> > > > > > > > > > > >> >> >>> > > >   1.  A REST API to paginate through
> > table
> > > > > > metrics.
> > > > > > > > > This
> > > > > > > > > > > >> will be
> > > > > > > > > > > >> >> >>> > non-IRC
> > > > > > > > > > > >> >> >>> > > > standard addition.
> > > > > > > > > > > >> >> >>> > > >   2.  Utilities for managing old
> records,
> > > > > should
> > > > > > be
> > > > > > > > > > common
> > > > > > > > > > > >> with
> > > > > > > > > > > >> >> >>> events.
> > > > > > > > > > > >> >> >>> > > > There was some discussion that it
> belongs
> > > to
> > > > > the
> > > > > > > CLI.
> > > > > > > > > > > >> >> >>> > > >   3.  Separate datasource (metrics,
> > events,
> > > > > even
> > > > > > > > other
> > > > > > > > > > > >> tables?).
> > > > > > > > > > > >> >> >>> > > >
> > > > > > > > > > > >> >> >>> > > >
> > > > > > > > > > > >> >> >>> > > > Anything else?
> > > > > > > > > > > >> >> >>> > > >
> > > > > > > > > > > >> >> >>> > > > -
> > > > > > > > > > > >> >> >>> > > > Anand
> > > > > > > > > > > >> >> >>> > > >
> > > > > > > > > > > >> >> >>> > > >
> > > > > > > > > > > >> >> >>> > >
> > > > > > > > > > > >> >> >>> >
> > > > > > > > > > > >> >> >>> >
> > > > > > > > > > > >> >> >>>
> > > > > > > > > > > >> >> >>
> > > > > > > > > > > >> >>
> > > > > > > > > > > >> >
> > > > > > > > > > > >>
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>

Re: Proposal for REST endpoints for table metrics and events

[Dmitri Bourlatchkov <di...@apache.org>]

Hi All,

So, what are the blockers for merging [4115], if any?

[4115] https://github.com/apache/polaris/pull/4115

Thanks,
Dmitri.

On Wed, Jun 17, 2026 at 5:28 PM EJ Wang <ej...@gmail.com>
wrote:

> Hi folks,
>
> Thanks again for the discussion today. I updated the sync doc with notes
> from today's metrics sync:
>
>
> https://docs.google.com/document/d/100h7c4damrUzVuquYbBHM0EvA4LSWuW2IT2dN_7nYVA/edit?tab=t.ezk0rgdx0c6m
>
> A few highlights:
>
>    - No objection to keeping the default metrics behavior small:
>    no-op/log-only is enough as the built-in default.
>    - Durable storage, event forwarding, external queues, dashboards, and
>    custom filtering should be implementation choices behind the metrics
>    reporting path.
>    - For PR #4115, the REST layer should resolve/authz/accept the report,
>    then delegate to the selected metrics implementation. It should not own
>    durable storage semantics.
>    - The rough boundary we discussed is: spec owns the wire contract,
>    runtime owns framework wiring, API/contract modules own provider-facing
>    contracts, and extensions own replaceable implementations.
>    - OpenLineage has a similar path-naming question. We should avoid
>    occupying a generic lineage path too early if we want room for other
>    lineage systems later.
>
> I believe the boundary rules previewed in sync meeting are useful for
> reviewing PR #4115, but it is not
> a final module-layout proposal.
>
> For metrics specifically, I think the target remains:
>    runtime
>      -> metrics reporting/emitting contract
>      -> default implementation
>           -> lower implementation details, if needed
>
> That keeps the default simple, while allowing durable JDBC, event-backed,
> or external-queue implementations to be added separately.
>
> We also touched on the split between Iceberg metrics emitting and
> Polaris-owned metrics querying. The querying side can keep evolving,
> especially for dashboard or generic-table use cases, but I do not think
> that needs to block the intake path in this PR.
>
> Thanks,
> -ej
>
> On Tue, Jun 16, 2026 at 2:17 PM Yufei Gu <fl...@gmail.com> wrote:
>
> > Thanks for chiming in, EJ. Agreed that the default battery should stay
> > small. I'm leaning toward not including metrics persistence in the
> default
> > battery.
> >
> > Yufei
> >
> >
> > On Thu, Jun 11, 2026 at 8:30 PM EJ Wang <ej...@gmail.com>
> > wrote:
> >
> > > Hi Yufei,
> > >
> > > Thanks for connecting this back to the REST endpoint proposal. I
> replied
> > > with the fuller version on the event-forwarding thread, but wanted to
> add
> > > the shorter version here too.
> > >
> > > I agree that metrics reporting/emitting is the right conceptual
> boundary,
> > > and I also think the event/listener path is a good implementation
> > direction
> > > to explore. The distinction I want to keep clear is default battery vs
> > > extension implementation.
> > >
> > > For the REST/API side, I would keep the semantics narrow: Polaris
> > resolves
> > > and authorizes the table, accepts the Iceberg scan/commit metrics
> report
> > > into the configured reporting/emitting path, and returns 204. That 204
> > > should mean Polaris accepted the report into the ingestion path; it
> > should
> > > not imply durable storage.
> > >
> > > Durable storage, event forwarding, external telemetry routing,
> filtering,
> > > and retention should sit behind that reporting/emitting boundary as
> > > implementation-layer behavior. A durable JDBC path can be one named
> > > extension implementation. An event/listener forwarding path can be
> > another
> > > named extension implementation.
> > >
> > > So I am +1 on the event-forwarding idea as a non-default extension
> > > implementation. I just would not make it the default battery or the
> core
> > > API shape. The default battery can stay small and safe, while
> deployments
> > > that need forwarding, dashboards, or durable storage can opt into the
> > > implementation that matches their operational model.
> > >
> > > Thanks,
> > > -ej
> > >
> > > On Wed, Jun 10, 2026 at 11:43 AM Yufei Gu <fl...@gmail.com>
> wrote:
> > >
> > > > Thanks EJ. I agree that the reporting/emitting boundary feels like
> the
> > > > right SPI boundary, but I wonder if we can simplify this even
> further.
> > > >
> > > > Iceberg scan and commit metrics look very similar to events to me.
> They
> > > are
> > > > append only, asynchronously consumed, often forwarded to external
> > > systems,
> > > > and have similar retention and cleanup requirements. More details can
> > be
> > > > found in this thread [1]. In that case, Polaris may only need to
> accept
> > > the
> > > > report and emit an event through the existing event framework.
> > Filtering,
> > > > async delivery, custom sinks, and retention mechanisms could then be
> > > shared
> > > > instead of introducing a separate metrics specific extension layer
> and
> > > SPI.
> > > >
> > > > The main requirement I still see is querying metrics through Polaris.
> > If
> > > we
> > > > want that battery included experience, we could work on a persistence
> > > > listener implementation that selectively saves metrics and expose it
> > via
> > > > the IRC event endpoint which seems pretty close in Iceberg
> > > > community. Another option I like more is to allow users to develop
> > their
> > > > own listener, so they can persist scan/commit metrics to any systems
> > they
> > > > prefer. In general, that feels like an implementation choice on top
> of
> > > the
> > > > event framework rather than something that needs to shape the core
> > > > architecture.
> > > >
> > > > 1. https://lists.apache.org/thread/x9j8nscvy8hq61tyn01mj8yp6n9of0kp
> > > >
> > > > Thanks,
> > > > Yufei
> > > >
> > > >
> > > > On Thu, Jun 4, 2026 at 5:14 PM EJ Wang <
> ej.wang.dev.interest@gmail.com
> > >
> > > > wrote:
> > > >
> > > > > Good point. I agree the existing PolarisMetricsReporter is already
> > very
> > > > > close to the right conceptual boundary. I am not proposing a second
> > > > > parallel reporter concept. The distinction I am trying to make is
> > > > narrower:
> > > > >
> > > > > Current state:
> > > > >
> > > > >    - PolarisMetricsReporter is the existing metrics reporting hook.
> > > > >    - But it currently lives under runtime/service.
> > > > >    - Its contract is documented around Quarkus/CDI discovery and
> > > > > @Identifier
> > > > >    selection.
> > > > >    - The default implementation is log-only.
> > > > >    - The durable implementation writes through
> PolarisMetricsManager
> > > and
> > > > >    MetricsPersistence.
> > > > >    - MetricsPersistence is currently inherited by BasePersistence.
> > > > >
> > > > > My proposal:
> > > > >
> > > > >    - Keep the reporting/emitting boundary as the SPI.
> > > > >    - Revise that contract in a framework-agnostic optional Iceberg
> > > > *metrics
> > > > >    extension API layer*.
> > > > >    - Keep runtime/service responsible for REST ingestion,
> table/authz
> > > > >    validation, and runtime wiring.
> > > > >    - Keep the battery implementation log-only in the metrics
> > extension
> > > > API
> > > > >    layer, *not under runtime/service*.
> > > > >    - Treat durable JDBC metrics as one implementation of the
> > reporting
> > > > SPI.
> > > > >    - Decompose the MetricsPersistence and BasePersistence coupling
> as
> > > > >    durable implementation detail.
> > > > >
> > > > > So the difference is not “new reporter concept vs old reporter
> > > concept.”
> > > > > but:
> > > > >
> > > > >
> > > > >    - PolarisMetricsReporter today = runtime/service CDI-shaped
> hook.
> > > > >    - Target reporting SPI = same conceptual boundary, but placed in
> > the
> > > > >    right extension API layer and kept framework-agnostic.
> > > > >
> > > > > For the current metrics PR, I think the minimal architectural
> cleanup
> > > > > before merge is to *put the reporting SPI contract and the battery
> > > > default
> > > > > in the right place*. The durable JDBC implementation, event-backed
> > > async
> > > > > implementation, external queue support, and deeper persistence
> > cleanup
> > > > can
> > > > > continue as follow-ups, as long as we do not lock the SPI boundary
> to
> > > > > BasePersistence or to runtime/service wiring.
> > > > >
> > > > > -ej
> > > > >
> > > > > On Wed, Jun 3, 2026 at 3:00 PM Dmitri Bourlatchkov <
> dimas@apache.org
> > >
> > > > > wrote:
> > > > >
> > > > > > Hi EJ,
> > > > > >
> > > > > > Thanks for the recap / summary!
> > > > > >
> > > > > > > Do folks agree that the stable SPI boundary should be
> > > > > > metrics reporting/emitting, not metrics persistence?
> > > > > >
> > > > > > SGTM.
> > > > > >
> > > > > > > Does an optional Iceberg metrics extension API layer sound like
> > the
> > > > > right
> > > > > > home for this SPI?
> > > > > >
> > > > > > How is that different from current PolarisMetricsReporter?
> > > > > >
> > > > > > > Should the current durable metrics work be reframed as a
> durable
> > > > > > JDBC reference implementation of that SPI?
> > > > > >
> > > > > > SGTM.
> > > > > >
> > > > > > > What is the smallest PR sequence to get there without blocking
> > > > > > the current metrics work unnecessarily?
> > > > > >
> > > > > > Let's get https://github.com/apache/polaris/pull/4397 merged
> > first.
> > > > > >
> > > > > > Then, I'd propose isolating the Metrics schema from the MetaStore
> > > > schema.
> > > > > > This will probably have some ripple effect into the bootstrap
> > > > workflows,
> > > > > so
> > > > > > it's not a trivial change.
> > > > > >
> > > > > > Then, let's reassess.
> > > > > >
> > > > > > Cheers,
> > > > > > Dmitri.
> > > > > >
> > > > > > On Wed, Jun 3, 2026 at 5:40 PM EJ Wang <
> > > ej.wang.dev.interest@gmail.com
> > > > >
> > > > > > wrote:
> > > > > >
> > > > > > > Hi folks,
> > > > > > >
> > > > > > > Thanks again for the discussion today. I updated the sync doc
> > with
> > > > > notes
> > > > > > > from the third metrics architecture sync:
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://docs.google.com/document/d/100h7c4damrUzVuquYbBHM0EvA4LSWuW2IT2dN_7nYVA/edit?tab=t.k96s2xyqr5u1
> > > > > > >
> > > > > > > A few highlights from today’s discussion:
> > > > > > >
> > > > > > >    - We clarified that Iceberg metrics reporting can be
> > interpreted
> > > > > > either
> > > > > > >    as sync or async handling from Polaris’s perspective, so
> > Polaris
> > > > > > should
> > > > > > >    stay flexible as a platform instead of baking one handling
> > model
> > > > > into
> > > > > > > the
> > > > > > >    REST API behavior.
> > > > > > >    - We aligned that the built-in (battery) behavior for
> metrics
> > > > > emitting
> > > > > > >    can stay simple: no-op/log-only is enough as the default.
> > > > > > >    - Durable metrics persistence should be treated as an
> > > > implementation
> > > > > > of
> > > > > > >    the metrics reporting path, not as the core SPI boundary.
> > > > > > >    - The existing durable metrics work can be reviewed as a
> > > reference
> > > > > > >    implementation of the reporting SPI, with
> persistence-related
> > > > logic
> > > > > > kept
> > > > > > >    self-contained under a metrics durable implementation module
> > > > rather
> > > > > > than
> > > > > > >    scattered through core entity persistence.
> > > > > > >    - Dashboard/insights remains a real use case, but we agreed
> to
> > > > keep
> > > > > it
> > > > > > >    separate from the core metrics intake discussion for now.
> > > > > > >
> > > > > > > I also did a quick source check after the meeting to make sure
> we
> > > are
> > > > > > > describing the current state accurately.
> > > > > > >
> > > > > > > Current state:
> > > > > > >
> > > > > > >    - Polaris already has a metrics reporting hook:
> > > > > > PolarisMetricsReporter.
> > > > > > >    - The default implementation is DefaultMetricsReporter,
> > selected
> > > > by
> > > > > > >    polaris.iceberg-metrics.reporting.type=default. It is
> > log-only,
> > > > > > >    effectively quiet unless metrics logging is enabled.
> > > > > > >    - There is also a PersistingMetricsReporter, selected by
> > > > > > >    polaris.iceberg-metrics.reporting.type=persisting, which
> > > converts
> > > > > > >    Iceberg scan/commit reports into Polaris metrics records and
> > > > writes
> > > > > > > through
> > > > > > >    PolarisMetricsManager -> MetricsPersistence.
> > > > > > >    - MetricsPersistence currently exists in the persistence
> layer
> > > and
> > > > > is
> > > > > > >    inherited by BasePersistence.
> > > > > > >
> > > > > > > My read from the discussion is that the target boundary should
> > not
> > > be
> > > > > > > MetricsPersistence as inherited by BasePersistence. That path
> > > should
> > > > be
> > > > > > > decomposed as durable implementation detail (taken care of by
> > > > > > > https://github.com/apache/polaris/pull/4397). The stable
> > extension
> > > > > point
> > > > > > > should instead be the metrics reporting/emitting boundary.
> > > > > > >
> > > > > > > Concretely, I think the next proposal should be shaped like
> this:
> > > > > > >
> > > > > > >    1.
> > > > > > >
> > > > > > >    Define the proper metrics reporting SPI at an optional
> Iceberg
> > > > > metrics
> > > > > > >    extension API layer.
> > > > > > >
> > > > > > >    Example direction:
> > > > > > >
> > > > > > >    public interface IcebergMetricsReporter {
> > > > > > >      void reportMetric(IcebergMetricsReportContext context,
> > > > > > > MetricsReport report);
> > > > > > >    }
> > > > > > >
> > > > > > >    The context would carry the small Polaris-resolved envelope:
> > > > > > >    catalog/table identity, report type, received timestamp, and
> > > > > > > request/trace
> > > > > > >    context if available. The raw Iceberg MetricsReport remains
> > the
> > > > > > payload.
> > > > > > >    2.
> > > > > > >
> > > > > > >    Keep runtime/service as the REST ingestion and wiring layer.
> > > > > > >
> > > > > > >    The REST handler still resolves the table and performs authz
> > > > before
> > > > > > >    accepting the report. After that, it calls the selected
> > > reporting
> > > > > > >    implementation.
> > > > > > >    3.
> > > > > > >
> > > > > > >    Keep the battery default no-op/log-only.
> > > > > > >
> > > > > > >    This preserves an out-of-box safe default and avoids
> > requiring a
> > > > > > durable
> > > > > > >    metrics store for every Polaris deployment.
> > > > > > >    4.
> > > > > > >
> > > > > > >    Move durable JDBC metrics into a self-contained
> implementation
> > > of
> > > > > the
> > > > > > >    reporting SPI.
> > > > > > >
> > > > > > >    That implementation can own its schema, bootstrap,
> retention,
> > > and
> > > > > read
> > > > > > >    API support. It should not define the core reporting SPI
> > > boundary,
> > > > > and
> > > > > > > it
> > > > > > >    should not require metrics persistence to remain inherited
> > from
> > > > > > >    BasePersistence.
> > > > > > >    5.
> > > > > > >
> > > > > > >    Treat async/event-backed handling as another implementation
> of
> > > the
> > > > > > same
> > > > > > >    reporting SPI.
> > > > > > >
> > > > > > >    For example, an event-backed reporter could enqueue a
> metrics
> > > > event
> > > > > > and
> > > > > > >    let listeners handle durable storage or other sinks. If we
> > later
> > > > > need
> > > > > > a
> > > > > > >    replaceable queue engine, that seems like a shared
> > event/metrics
> > > > > > > substrate
> > > > > > >    topic rather than a metrics-only requirement.
> > > > > > >
> > > > > > > This framing lets us keep the REST metrics endpoint simple,
> > > preserve
> > > > > the
> > > > > > > current default behavior, support durable metrics users, and
> > still
> > > > > leave
> > > > > > > room for async/event-backed or external-queue-based
> > > implementations.
> > > > > > >
> > > > > > > I think the main follow-up questions are:
> > > > > > >
> > > > > > >    - Do folks agree that the stable SPI boundary should be
> > metrics
> > > > > > >    reporting/emitting, not metrics persistence?
> > > > > > >    - Does an optional Iceberg metrics extension API layer sound
> > > like
> > > > > the
> > > > > > >    right home for this SPI?
> > > > > > >    - Should the current durable metrics work be reframed as a
> > > durable
> > > > > > JDBC
> > > > > > >    reference implementation of that SPI?
> > > > > > >    - What is the smallest PR sequence to get there without
> > blocking
> > > > the
> > > > > > >    current metrics work unnecessarily?
> > > > > > >
> > > > > > > Thanks,
> > > > > > > -ej
> > > > > > >
> > > > > > > On Fri, May 15, 2026 at 5:16 PM Dmitri Bourlatchkov <
> > > > dimas@apache.org>
> > > > > > > wrote:
> > > > > > >
> > > > > > > > Hi JB,
> > > > > > > >
> > > > > > > > Could you set up another meeting, please? Same time on
> > Wednesday
> > > as
> > > > > > last
> > > > > > > > time... I hope it works for everyone.
> > > > > > > >
> > > > > > > > Cheers,
> > > > > > > > Dmitri.
> > > > > > > >
> > > > > > > > On Fri, May 15, 2026 at 8:06 PM Yufei Gu <
> flyrain000@gmail.com
> > >
> > > > > wrote:
> > > > > > > >
> > > > > > > > > +1 on another sync call next week.
> > > > > > > > >
> > > > > > > > > Yufei
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > On Fri, May 15, 2026 at 4:52 PM Dmitri Bourlatchkov <
> > > > > > dimas@apache.org>
> > > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > Hi All,
> > > > > > > > > >
> > > > > > > > > > WDYT about another sync call next week?
> > > > > > > > > >
> > > > > > > > > > Thanks,
> > > > > > > > > > Dmitri.
> > > > > > > > > >
> > > > > > > > > > On Wed, May 6, 2026 at 5:29 PM Dmitri Bourlatchkov <
> > > > > > dimas@apache.org
> > > > > > > >
> > > > > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > > Hi EJ,
> > > > > > > > > > >
> > > > > > > > > > > Thanks for the summary! It covers what we discussed in
> > the
> > > > > > meeting
> > > > > > > > very
> > > > > > > > > > > well, IMHO.
> > > > > > > > > > >
> > > > > > > > > > > Looking forward to concrete PRs :)
> > > > > > > > > > >
> > > > > > > > > > > Cheers,
> > > > > > > > > > > Dmitri.
> > > > > > > > > > >
> > > > > > > > > > > On Wed, May 6, 2026 at 5:08 PM EJ Wang <
> > > > > > > > ej.wang.dev.interest@gmail.com
> > > > > > > > > >
> > > > > > > > > > > wrote:
> > > > > > > > > > >
> > > > > > > > > > >> Hi folks,
> > > > > > > > > > >>
> > > > > > > > > > >> We had a community sync earlier, thanks JB for
> > scheduling
> > > > it.
> > > > > > > Notes
> > > > > > > > > from
> > > > > > > > > > >> the first metrics architecture sync (May 6, 10-11am
> PT).
> > > > > > > Discussion
> > > > > > > > > doc
> > > > > > > > > > >> with per-section status:
> > > > > > > > > > >>
> > > > > > > > > > >>
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://docs.google.com/document/d/100h7c4damrUzVuquYbBHM0EvA4LSWuW2IT2dN_7nYVA/edit?tab=t.0
> > > > > > > > > > >>
> > > > > > > > > > >> *The meeting covered both topics from the doc.
> > > > Direction-level
> > > > > > > > > alignment
> > > > > > > > > > >> was reached on the headline pieces; details remain for
> > PR
> > > > > review
> > > > > > > or
> > > > > > > > > > >> follow-up sessions.*
> > > > > > > > > > >>
> > > > > > > > > > >> *Topic 1 — Persistence schema redesign*
> > > > > > > > > > >> Idea-level alignment on consolidating per-type tables
> > > > > > > > > > >> (scan_metrics_report,
> > > > > > > > > > >> commit_metrics_report) into a single metrics_report
> > table.
> > > > The
> > > > > > > > > > motivating
> > > > > > > > > > >> cost is the surface area added by every new metric
> type
> > > > today:
> > > > > > new
> > > > > > > > > > table,
> > > > > > > > > > >> SPI method, record class, model, converter, schema
> > > > migration.
> > > > > > > > > > >>
> > > > > > > > > > >> Most schema details are deferred to the schema PR. A
> few
> > > > > > specific
> > > > > > > > > points
> > > > > > > > > > >> came up:
> > > > > > > > > > >> •   metric_schema_version: Yufei prefers dropping it,
> > > since
> > > > > > there
> > > > > > > is
> > > > > > > > > no
> > > > > > > > > > >> spec-level concept of metrics versioning today and it
> is
> > > > hard
> > > > > to
> > > > > > > > > define
> > > > > > > > > > >> unilaterally. Robert prefers keeping it, given IRC v2
> is
> > > > > coming
> > > > > > > and
> > > > > > > > > the
> > > > > > > > > > >> schema should be considered against its likely shape;
> > > Robert
> > > > > > also
> > > > > > > > > raised
> > > > > > > > > > >> how to differentiate various payload formats if any.
> > EJ's
> > > > read
> > > > > > is
> > > > > > > > that
> > > > > > > > > > >> this
> > > > > > > > > > >> is a two-way-door decision. We can start without the
> > > field,
> > > > > and
> > > > > > if
> > > > > > > > IRC
> > > > > > > > > > v2
> > > > > > > > > > >> changes the shape we would likely roll a corresponding
> > new
> > > > > > schema
> > > > > > > > > > anyway,
> > > > > > > > > > >> which is not particularly costly.
> > > > > > > > > > >> •   Payload format: Robert pointed out that future
> > formats
> > > > > > beyond
> > > > > > > > JSON
> > > > > > > > > > may
> > > > > > > > > > >> be worth supporting. The exact shape is deferred to
> the
> > > > schema
> > > > > > > > > > discussion.
> > > > > > > > > > >> •   Partition strategy: Anand suggested monthly
> > > partitioning
> > > > > > based
> > > > > > > > on
> > > > > > > > > > his
> > > > > > > > > > >> experience as potentially helpful at scale.
> > > > > > > > > > >>
> > > > > > > > > > >> *Topic 2 — Where metrics ingestion and storage belong*
> > > > > > > > > > >> Idea-level alignment that metrics should be a
> separated
> > > SPI
> > > > > from
> > > > > > > the
> > > > > > > > > > >> entity
> > > > > > > > > > >> persistence stack. Two reasons surfaced: (a) workloads
> > and
> > > > > > > > capability
> > > > > > > > > > >> requirements diverge enough that coupling them creates
> > > > > > artificial
> > > > > > > > > > >> constraints, and (b) admin experience improves when
> > > metrics
> > > > > has
> > > > > > > its
> > > > > > > > > own
> > > > > > > > > > >> bootstrap, retention, and lifecycle. Dmitri noted
> > Polaris
> > > > > being
> > > > > > a
> > > > > > > > > > platform
> > > > > > > > > > >> should have the flexibility to support different
> > > persistence
> > > > > > > > backends
> > > > > > > > > > per
> > > > > > > > > > >> concern, and pointed to a concrete next step of
> > separating
> > > > the
> > > > > > > JDBC
> > > > > > > > > > >> bootstrap for metrics from the metastore bootstrap.
> > Robert
> > > > > > > proposed
> > > > > > > > an
> > > > > > > > > > >> additional UX extension: detect an unbootstrapped
> > metrics
> > > > > store
> > > > > > on
> > > > > > > > > first
> > > > > > > > > > >> use and auto-bootstrap rather than requiring an
> explicit
> > > > > manual
> > > > > > > > > > bootstrap
> > > > > > > > > > >> step.
> > > > > > > > > > >> The meeting also confirmed that Polaris metrics can
> > start
> > > > > small
> > > > > > > and
> > > > > > > > > stay
> > > > > > > > > > >> Iceberg-focused. Naming and persistence schema can
> lean
> > > > > > > > > > Iceberg-specific.
> > > > > > > > > > >> If a future expansion to generic-table metrics or
> > > > operational
> > > > > > > > metrics
> > > > > > > > > > >> arrives, an abstraction layer can be built on top of
> the
> > > > > Iceberg
> > > > > > > > > metrics
> > > > > > > > > > >> reporter at that point. Robert remains on the fence
> and
> > > > would
> > > > > > > prefer
> > > > > > > > > > >> something more generic but did not block the
> direction;
> > > > > Dmitri's
> > > > > > > > read
> > > > > > > > > > was
> > > > > > > > > > >> that the proposed framework already has enough
> > flexibility
> > > > to
> > > > > > > absorb
> > > > > > > > > > >> future
> > > > > > > > > > >> expansion.
> > > > > > > > > > >>
> > > > > > > > > > >> The Trade-offs and Proposed structure sections in the
> > doc
> > > > were
> > > > > > not
> > > > > > > > > > >> reviewed
> > > > > > > > > > >> in detail. They remain open for either the next sync
> or
> > PR
> > > > > > review.
> > > > > > > > > > >>
> > > > > > > > > > >> *Cross-cutting alignment — battery-included plus
> > > pluggable*
> > > > > > > > > > >> A common philosophy emerged from the discussion. EJ
> > > > summarized
> > > > > > it
> > > > > > > > as:
> > > > > > > > > > >> Polaris should provide a battery-included UX for
> > beginners
> > > > and
> > > > > > the
> > > > > > > > > > >> flexibility for advanced users to swap the included
> > > battery
> > > > > for
> > > > > > > > > > something
> > > > > > > > > > >> more powerful or tailored to their use case. The SPI
> > > design
> > > > > > needs
> > > > > > > to
> > > > > > > > > > >> enable
> > > > > > > > > > >> both.
> > > > > > > > > > >>
> > > > > > > > > > >> The inputs that shaped this framing:
> > > > > > > > > > >> •   Anand described how his team uses the current
> > metrics
> > > > > > > > persistence
> > > > > > > > > > >> (three metrics consumers in v1.4).
> > > > > > > > > > >> •   Yufei raised Grafana and dashboard integrations
> as a
> > > > > > > destination
> > > > > > > > > use
> > > > > > > > > > >> case beyond the default.
> > > > > > > > > > >> •   Robert called out that the current design is more
> > > > > > > JDBC-focused.
> > > > > > > > > > >>
> > > > > > > > > > >> Two concrete instances:
> > > > > > > > > > >> •   Async metrics intake: Yufei's initial position was
> > > that
> > > > > > async
> > > > > > > > > should
> > > > > > > > > > >> largely live on the producer side and there is not
> much
> > > > > Polaris
> > > > > > > can
> > > > > > > > > do.
> > > > > > > > > > >> Robert suggested a Polaris-side default is doable via
> > > > Vert.x.
> > > > > > > Dmitri
> > > > > > > > > > >> agreed
> > > > > > > > > > >> the direction is worth exploring. The meeting
> converged
> > > on a
> > > > > > > > > > >> battery-included default (likely Vert.x-backed) with
> an
> > > SPI
> > > > > > shape
> > > > > > > > that
> > > > > > > > > > >> lets
> > > > > > > > > > >> power users route to a more scalable backend
> (k8s-hosted
> > > > > queue,
> > > > > > > AWS
> > > > > > > > > SQS,
> > > > > > > > > > >> etc.).
> > > > > > > > > > >> •   Pluggable destinations: combining Yufei's
> dashboard
> > > use
> > > > > case
> > > > > > > > with
> > > > > > > > > > >> Robert's JDBC-focused call-out, the meeting agreed the
> > SPI
> > > > > > should
> > > > > > > be
> > > > > > > > > > >> structured for multiple sinks so integrations become
> > impl
> > > > > > choices
> > > > > > > > > rather
> > > > > > > > > > >> than architectural changes.
> > > > > > > > > > >>
> > > > > > > > > > >> The battery-included default is most likely to use the
> > > > > existing
> > > > > > > > > > >> JDBC-backed
> > > > > > > > > > >> approach.
> > > > > > > > > > >>
> > > > > > > > > > >> *Direction (idea-level alignment)*
> > > > > > > > > > >> •   Single metrics_report table consolidating per-type
> > > > > metrics,
> > > > > > > > > > replacing
> > > > > > > > > > >> scan_metrics_report and commit_metrics_report
> > > > > > > > > > >> •   Iceberg-focused naming and schema for now, revisit
> > if
> > > > > > > > > generic-table
> > > > > > > > > > or
> > > > > > > > > > >> operational metrics arrive
> > > > > > > > > > >> •   Metrics persistence as a separated SPI, not on
> > > > > > BasePersistence
> > > > > > > > > > >> •   Bootstrap path separated for metrics, independent
> of
> > > > > > metastore
> > > > > > > > > > >> bootstrap
> > > > > > > > > > >> •   "Battery-included plus pluggable" as the SPI
> design
> > > > > > philosophy
> > > > > > > > > > >>
> > > > > > > > > > >> *Open items*
> > > > > > > > > > >> •   Schema details: metric_schema_version, payload
> > format,
> > > > IRC
> > > > > > v2
> > > > > > > > > > >> forward-compat shape
> > > > > > > > > > >> •   SPI design details — full review either in the
> next
> > > sync
> > > > > or
> > > > > > in
> > > > > > > > the
> > > > > > > > > > >> corresponding PR
> > > > > > > > > > >> •   Schema refactor PR ownership
> > > > > > > > > > >>
> > > > > > > > > > >> *Action items*
> > > > > > > > > > >> •   EJ to take a first stab at the SPI design and
> > > > potentially
> > > > > > > > partner
> > > > > > > > > > with
> > > > > > > > > > >> Anand to incorporate the lessons learned from the
> > existing
> > > > > > > reporter
> > > > > > > > > and
> > > > > > > > > > >> persistence work.
> > > > > > > > > > >> •   Schema refactor PR ownership is not yet decided.
> If
> > > > anyone
> > > > > > is
> > > > > > > > > > >> interested in driving it, reply on this thread.
> > > > > > > > > > >> •   JB to schedule the next sync, tentatively in two
> > > weeks.
> > > > > > > > > > >>
> > > > > > > > > > >> -ej
> > > > > > > > > > >>
> > > > > > > > > > >> On Mon, Apr 27, 2026 at 3:07 PM EJ Wang <
> > > > > > > > > ej.wang.dev.interest@gmail.com
> > > > > > > > > > >
> > > > > > > > > > >> wrote:
> > > > > > > > > > >>
> > > > > > > > > > >> > Thanks Yufei for the +1.
> > > > > > > > > > >> >
> > > > > > > > > > >> > JB, could you help add a biweekly metrics
> architecture
> > > > sync
> > > > > to
> > > > > > > the
> > > > > > > > > > >> Polaris
> > > > > > > > > > >> > community calendar? I'm thinking Thursdays at 9-10am
> > PT,
> > > > on
> > > > > > the
> > > > > > > > > > >> off-weeks
> > > > > > > > > > >> > from the community meeting (starting May 7), 60
> > minutes.
> > > > > > > > > > >> >
> > > > > > > > > > >> > Here's a rough agenda to work through over the first
> > few
> > > > > > > sessions,
> > > > > > > > > > >> grouped
> > > > > > > > > > >> > by priority:
> > > > > > > > > > >> >
> > > > > > > > > > >> > *First: foundational direction*
> > > > > > > > > > >> >
> > > > > > > > > > >> > 1.  MetricsPersistence: public SPI or internal
> > > > > implementation
> > > > > > > > > detail?
> > > > > > > > > > >> >    •   Marked @Beta, javadoc calls it a "Service
> > > Provider
> > > > > > > > > Interface",
> > > > > > > > > > >> but
> > > > > > > > > > >> > only one consumer (JdbcBasePersistenceImpl), lives
> on
> > > > > > > > > BasePersistence.
> > > > > > > > > > >> If
> > > > > > > > > > >> > demoted to a private helper inside a persisting
> > reporter
> > > > > impl,
> > > > > > > > most
> > > > > > > > > > >> > downstream design decisions become implementation
> > > details
> > > > > > rather
> > > > > > > > > than
> > > > > > > > > > >> > contract questions.
> > > > > > > > > > >> >
> > > > > > > > > > >> > 2.  Persistence schema redesign
> > > > > > > > > > >> >    •   Current two-table layout
> (scan_metrics_report,
> > > > > > > > > > >> > commit_metrics_report) with ~25 flattened columns
> > each.
> > > > > Every
> > > > > > > new
> > > > > > > > > > metric
> > > > > > > > > > >> > type requires a new table, SPI method, record class,
> > > > model,
> > > > > > > > > converter,
> > > > > > > > > > >> and
> > > > > > > > > > >> > schema migration. Direction to explore: single table
> > > with
> > > > > > > > > metric_type
> > > > > > > > > > >> enum,
> > > > > > > > > > >> > schema_version, and JSON payload column.
> > > > > > > > > > >> >
> > > > > > > > > > >> > *Second: design details once direction is set*
> > > > > > > > > > >> >
> > > > > > > > > > >> > 3.  Partition key strategy
> > > > > > > > > > >> >    •   Single-table design means scan metrics at
> scale
> > > > will
> > > > > > have
> > > > > > > > > high
> > > > > > > > > > >> > write concurrency per table. Schema needs to expose
> > > enough
> > > > > > > > structure
> > > > > > > > > > for
> > > > > > > > > > >> > backends to shard by entity or time range.
> > > > > > > > > > >> >
> > > > > > > > > > >> > 4.  Read/write path consistency
> > > > > > > > > > >> >    •   Writes go through PolarisMetricsManager on
> > > > > > > > MetaStoreManager.
> > > > > > > > > > >> Reads
> > > > > > > > > > >> > bypass MetaStoreManager and go straight to
> > > > BasePersistence,
> > > > > > > > > excluding
> > > > > > > > > > >> > non-JDBC backends from the read API.
> > > > > > > > > > >> >
> > > > > > > > > > >> > *Third: cleanup and alignment*
> > > > > > > > > > >> >
> > > > > > > > > > >> > 5.  PolarisMetricsReporter naming
> > > > > > > > > > >> >    •   Only handles IRC (ScanReport/CommitReport),
> > > doesn't
> > > > > > cover
> > > > > > > > > > generic
> > > > > > > > > > >> > tables or operational metrics. Name is broader than
> > > scope.
> > > > > > > > > > >> >
> > > > > > > > > > >> > 6.  PolarisMetricsManager facade passthrough
> > > > > > > > > > >> >    •   Entire default method is
> > > > > > > > > > >> callCtx.getMetaStore().writeScanReport().
> > > > > > > > > > >> > Zero logic, passes Level 1 straight through to Level
> > 3.
> > > > Same
> > > > > > > > > > >> anti-pattern
> > > > > > > > > > >> > as PolarisEventManager.
> > > > > > > > > > >> >
> > > > > > > > > > >> > 7.  Iceberg community alignment
> > > > > > > > > > >> >    •   Payload-type extension needs discussion on
> > > > > dev@iceberg.
> > > > > > > > > > >> obelix74's
> > > > > > > > > > >> > Feb thread got zero replies. Needs a committer
> voice.
> > > > > > > > > > >> >
> > > > > > > > > > >> > Lets confirm prioritization in the first session.
> > > > > > > > > > >> >
> > > > > > > > > > >> > -ej
> > > > > > > > > > >> >
> > > > > > > > > > >> > On Tue, Apr 21, 2026 at 3:18 PM Yufei Gu <
> > > > > > flyrain000@gmail.com>
> > > > > > > > > > wrote:
> > > > > > > > > > >> >
> > > > > > > > > > >> >> Thanks everyone for continuing to drive this
> > forward. I
> > > > > agree
> > > > > > > > that
> > > > > > > > > > the
> > > > > > > > > > >> >> problem is getting complex enough that a more
> > > structured
> > > > > > > > discussion
> > > > > > > > > > >> would
> > > > > > > > > > >> >> help.
> > > > > > > > > > >> >>
> > > > > > > > > > >> >> +1 on setting up a biweekly sync for the metrics
> > > > > > architecture.
> > > > > > > > I’m
> > > > > > > > > > >> happy
> > > > > > > > > > >> >> to
> > > > > > > > > > >> >> join.
> > > > > > > > > > >> >>
> > > > > > > > > > >> >> Yufei
> > > > > > > > > > >> >>
> > > > > > > > > > >> >>
> > > > > > > > > > >> >> On Tue, Apr 21, 2026 at 2:34 PM EJ Wang <
> > > > > > > > > > >> ej.wang.dev.interest@gmail.com>
> > > > > > > > > > >> >> wrote:
> > > > > > > > > > >> >>
> > > > > > > > > > >> >> > Also, I've been looking more closely at the
> > > > *persistence
> > > > > > > schema
> > > > > > > > > in
> > > > > > > > > > >> the
> > > > > > > > > > >> >> > current metrics work*, and I think there's a
> > > structural
> > > > > > > > rigidity
> > > > > > > > > > >> problem
> > > > > > > > > > >> >> > worth raising before the shape gets locked in.
> > > > > > > > > > >> >> >
> > > > > > > > > > >> >> > Right now we have two separate tables
> > > > > (scan_metrics_report
> > > > > > > and
> > > > > > > > > > >> >> > commit_metrics_report), each with ~25 flattened
> > > columns
> > > > > > that
> > > > > > > > > > directly
> > > > > > > > > > >> >> > mirror the Iceberg report fields. The SPI follows
> > the
> > > > > same
> > > > > > > > split:
> > > > > > > > > > >> >> > writeScanReport and writeCommitReport as separate
> > > > > methods,
> > > > > > > with
> > > > > > > > > > >> per-type
> > > > > > > > > > >> >> > record classes, converters, and model objects.
> *The
> > > > > > practical
> > > > > > > > > cost:
> > > > > > > > > > >> >> > adding a new metric type (operational metrics,
> for
> > > > > example)
> > > > > > > > > > requires
> > > > > > > > > > >> a
> > > > > > > > > > >> >> new
> > > > > > > > > > >> >> > table, a new SPI method, a new record class, a
> new
> > > > model
> > > > > > > > class, a
> > > > > > > > > > new
> > > > > > > > > > >> >> > converter branch, and a schema migration*.
> That's a
> > > lot
> > > > > of
> > > > > > > > > surface
> > > > > > > > > > >> area
> > > > > > > > > > >> >> > for what should be "one more kind of metric."
> > > > > > > > > > >> >> >
> > > > > > > > > > >> >> > *My bias* would be toward a single metrics table
> > with
> > > > *a
> > > > > > > typed
> > > > > > > > > JSON
> > > > > > > > > > >> >> > payload*. Something like: metric_type (enum),
> > > > entity_id,
> > > > > > > > > > >> >> > table_identifier, snapshot_id (nullable),
> > > received_ts,
> > > > > > > > > > >> schema_version,
> > > > > > > > > > >> >> and
> > > > > > > > > > >> >> > a payload column for the metric-specific data.
> The
> > > > > > > metric_type
> > > > > > > > +
> > > > > > > > > > >> >> > schema_version pair gives us a forward-compatible
> > > > > contract
> > > > > > > for
> > > > > > > > > the
> > > > > > > > > > >> >> payload
> > > > > > > > > > >> >> > shape. Adding a new metric type becomes an enum
> > value
> > > > > and a
> > > > > > > > > payload
> > > > > > > > > > >> >> schema,
> > > > > > > > > > >> >> > not a schema migration. One thing I think we need
> > to
> > > be
> > > > > > > > > deliberate
> > > > > > > > > > >> >> about is
> > > > > > > > > > >> >> > the partition key design. If all metric types
> land
> > in
> > > > one
> > > > > > > > table,
> > > > > > > > > > scan
> > > > > > > > > > >> >> > metrics at scale (high concurrency, high
> frequency
> > > > across
> > > > > > > many
> > > > > > > > > > >> tables)
> > > > > > > > > > >> >> > could easily create hot partitions. We'd want the
> > > > > > persistence
> > > > > > > > > layer
> > > > > > > > > > >> to
> > > > > > > > > > >> >> be
> > > > > > > > > > >> >> > able to shard by entity or time range, and that
> > means
> > > > the
> > > > > > > > logical
> > > > > > > > > > >> schema
> > > > > > > > > > >> >> > needs to expose enough structure for backends to
> > > > > partition
> > > > > > > on.
> > > > > > > > I
> > > > > > > > > > >> don't
> > > > > > > > > > >> >> > think the current flattened layout gives us that.
> > > > > > > > > > >> >> >
> > > > > > > > > > >> >> > This is getting complex enough that I don't think
> > > > ad-hoc
> > > > > > > PR/ML
> > > > > > > > > > >> threads
> > > > > > > > > > >> >> > will converge well. *Would people be open to a
> > > biweekly
> > > > > > sync
> > > > > > > > for
> > > > > > > > > > >> metrics
> > > > > > > > > > >> >> > architecture?* I think 30 minutes every two weeks
> > > with
> > > > > > > > interested
> > > > > > > > > > >> >> parties
> > > > > > > > > > >> >> > would be enough to work through the schema, SPI
> > > shape,
> > > > > and
> > > > > > > read
> > > > > > > > > API
> > > > > > > > > > >> >> design
> > > > > > > > > > >> >> > together. Happy to help set that up.
> > > > > > > > > > >> >> >
> > > > > > > > > > >> >> > -ej
> > > > > > > > > > >> >> >
> > > > > > > > > > >> >> > On Mon, Apr 20, 2026 at 2:19 PM EJ Wang <
> > > > > > > > > > >> ej.wang.dev.interest@gmail.com
> > > > > > > > > > >> >> >
> > > > > > > > > > >> >> > wrote:
> > > > > > > > > > >> >> >
> > > > > > > > > > >> >> >> Reviewed #4115, left a comment on the code
> > > > organization
> > > > > > > side.
> > > > > > > > > > >> >> >>
> > > > > > > > > > >> >> >> One thing stood out: the metrics write path
> enters
> > > > > through
> > > > > > > > > > >> >> >> PolarisMetricsManager on MetaStoreManager, but
> the
> > > new
> > > > > > read
> > > > > > > > path
> > > > > > > > > > >> >> bypasses
> > > > > > > > > > >> >> >> MetaStoreManager entirely and goes straight to
> > > > > > > BasePersistence
> > > > > > > > > via
> > > > > > > > > > >> >> >> callContext.getMetaStore(). That means the read
> > API
> > > > only
> > > > > > > works
> > > > > > > > > for
> > > > > > > > > > >> >> backends
> > > > > > > > > > >> >> >> that implement BasePersistence. NoSQL and remote
> > > > > backends
> > > > > > > > can't
> > > > > > > > > > >> >> participate.
> > > > > > > > > > >> >> >>
> > > > > > > > > > >> >> >> Stepping back, I think the metrics subsystem is
> > > > growing
> > > > > > into
> > > > > > > > > > >> something
> > > > > > > > > > >> >> >> real (write + read + REST API + AuthZ +
> > pagination)
> > > > *but
> > > > > > the
> > > > > > > > > > >> >> persistence
> > > > > > > > > > >> >> >> side is split across two layers in a way that's
> > hard
> > > > to
> > > > > > > > > extend*. I
> > > > > > > > > > >> put
> > > > > > > > > > >> >> >> together two diagrams to show what I mean (my
> best
> > > > > > effort).
> > > > > > > > > > >> >> >>
> > > > > > > > > > >> >> >> *Current state* (Diagram 1): three interfaces at
> > > three
> > > > > > > > different
> > > > > > > > > > >> >> levels.
> > > > > > > > > > >> >> >> The engine-facing SPI (PolarisMetricsReporter)
> is
> > > > clean.
> > > > > > But
> > > > > > > > > > >> >> >> PolarisMetricsManager on MetaStoreManager is a
> > > > > passthrough
> > > > > > > to
> > > > > > > > > > >> >> >> MetricsPersistence on BasePersistence. The @Beta
> > > > > > annotation
> > > > > > > > and
> > > > > > > > > > SPI
> > > > > > > > > > >> >> javadoc
> > > > > > > > > > >> >> >> are on the BasePersistence layer, while the
> actual
> > > > > > extension
> > > > > > > > > > points
> > > > > > > > > > >> >> >> (PolarisMetricsReporter, PolarisMetricsManager)
> > > carry
> > > > no
> > > > > > > > > stability
> > > > > > > > > > >> >> >> annotation. The write path goes through the
> > > > > > MetaStoreManager
> > > > > > > > > > layer,
> > > > > > > > > > >> the
> > > > > > > > > > >> >> >> read path doesn't.
> > > > > > > > > > >> >> >>
> > > > > > > > > > >> >> >> *What I envision* (Diagram 2): two SPIs at two
> > > levels.
> > > > > > > > > > >> >> >> PolarisMetricsReporter stays as the
> engine-facing
> > > SPI.
> > > > > > > > > > >> >> >> PolarisMetricsManager becomes the backend-facing
> > SPI
> > > > > with
> > > > > > > both
> > > > > > > > > > write
> > > > > > > > > > >> >> and
> > > > > > > > > > >> >> >> read methods at the MetaStoreManager level,
> where
> > > any
> > > > > > > backend
> > > > > > > > > > (JDBC,
> > > > > > > > > > >> >> NoSQL,
> > > > > > > > > > >> >> >> remote) can implement them. MetricsPersistence
> on
> > > > > > > > > BasePersistence
> > > > > > > > > > >> goes
> > > > > > > > > > >> >> >> away. Where metrics actually land is an
> > > implementation
> > > > > > > detail,
> > > > > > > > > > not a
> > > > > > > > > > >> >> core
> > > > > > > > > > >> >> >> interface.
> > > > > > > > > > >> >> >>
> > > > > > > > > > >> >> >> *Minor naming thing*: PolarisMetricsReporter is
> > > > broader
> > > > > > than
> > > > > > > > > what
> > > > > > > > > > it
> > > > > > > > > > >> >> >> actually handles. It only accepts Iceberg REST
> > > Catalog
> > > > > > > metrics
> > > > > > > > > > >> >> (ScanReport,
> > > > > > > > > > >> >> >> CommitReport via MetricsReport). Generic table
> > > metrics
> > > > > or
> > > > > > > > > > >> operational
> > > > > > > > > > >> >> >> metrics aren't in scope. Not blocking, but worth
> > > > noting
> > > > > if
> > > > > > > the
> > > > > > > > > > >> metrics
> > > > > > > > > > >> >> >> surface expands.
> > > > > > > > > > >> >> >>
> > > > > > > > > > >> >> >> *Rough sketch of how to get there*:
> > > > > > > > > > >> >> >>  1.  Add read methods to PolarisMetricsManager
> > > > > > > > (listScanReports,
> > > > > > > > > > >> >> >> listCommitReports) with default no-op, same as
> the
> > > > > > existing
> > > > > > > > > write
> > > > > > > > > > >> >> methods.
> > > > > > > > > > >> >> >> (Probably make PolarisMetricsManager more
> explicit
> > > on
> > > > > > being
> > > > > > > > > > Iceberg
> > > > > > > > > > >> >> >> specific like package name or class name etc.)
> > > > > > > > > > >> >> >>  2.  Wire MetricsReportsService through
> > > > MetaStoreManager
> > > > > > > > instead
> > > > > > > > > > of
> > > > > > > > > > >> >> >> callContext.getMetaStore().
> > > > > > > > > > >> >> >>  3.  Extract metrics persistence from
> > > > > > > JdbcBasePersistenceImpl
> > > > > > > > > into
> > > > > > > > > > >> its
> > > > > > > > > > >> >> >> own class. That file carries ~7
> responsibilities,
> > > > > metrics
> > > > > > > > being
> > > > > > > > > > one
> > > > > > > > > > >> of
> > > > > > > > > > >> >> them.
> > > > > > > > > > >> >> >>  4.  Remove MetricsPersistence from
> > BasePersistence.
> > > > > > > > > > >> >> >>
> > > > > > > > > > >> >> >> *None of this needs to happen in #4115. But if
> the
> > > > > > direction
> > > > > > > > > makes
> > > > > > > > > > >> >> sense,
> > > > > > > > > > >> >> >> it would be good to align before the metrics
> > surface
> > > > > grows
> > > > > > > > > > further.
> > > > > > > > > > >> >> Curious
> > > > > > > > > > >> >> >> what others think.*
> > > > > > > > > > >> >> >>
> > > > > > > > > > >> >> >> *My mental model note*: Level 1
> MetaStoreManager;
> > > > level
> > > > > 2
> > > > > > > > > > >> transactional
> > > > > > > > > > >> >> >> persistence; level 3 base persistence
> > > > > > > > > > >> >> >>
> > > > > > > > > > >> >> >> Diagram 1
> > > > > > > > > > >> >> >> <
> > > > > > > > > > >> >>
> > > > > > > > > > >>
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://www.plantuml.com/plantuml/uml/bLHDR-Cs4BthLmpIYupw0zbkKQ1r3M-S7Bp8xhhM7WCOb3IM65EaGD9EX2RzxHrHb4CxRelwa4YSDu_lpOVcnZ9jzvM8BBS2uGjQpJC3dtHMSekPtMk44IpsMgEqa5XcCOhCZikQQLP1pR8TAp2n3ILhmZDP20m0fcIvUkAoW2qJXd9z1bpToO9BX3WXu0ucy5rpgGPNm0nW5_epUWtm2Ue3pn3kMOFQmKntGZW0BYtgBSi8k5A2QMwybJNMIbFiGSR9QZc4nUqIvikStF0jHprua5C-amge42aNt3R0f5JaaoivdV2Pkqbx4hee4ymOkBh5BTiB-_uIeGeo8zL8rPsPl4DktdEiK1jkB1NdZCRbrSTecDe_mlHbF0wvBmCkaOH5_S8a_TTTKI6-nmCAkEw4LpxsZ-LbYLKQFKMNOgf_wuM7_bV9gOer5SYMMksBSWXFcbi49KNZXNLicwfe3TETC7gPdPqI7uBcHMb1RSzYq34c6PDUM9mn8HRsUTZEiDBve3NjVZumBj0U7SS37mGO7vcwtiK-_pU7U7L_f-digo9YbhSwIfMRwIITKGXbxdIUTCGF1SeCJxloKsU-3k9ddRbX1eDq1q_fx1JbBGT0glVyXimDuP4TQ5qpCAmnGEj2s_6n5mtn1z-97-63itFQZLPO1Ev2tu_WF7Ju-VPc0Skg5bYXxBhkY1xpD7EM_7fyflSpIsqMgVth5xhVr4eQxWQ8enaSAJQSG16yFSDuJ798rrcXr_3n-lfdk7icQjEBmFujL7AodiP_Y4Z7-YxvtZNs4zMgpNTl6tF8sglyPsmqchrjvQ-m-aP94r-TwCA2Ka8upPJZwtvSpoYCXkYMZU2NXvRMBfq9P3i3Le4VAZUAlUZ_oPKsxPgY0Q_BSKLkyr9bhQhQrJjo_x3TPlIB0DPjnMfcIoYP0QaYw1a0fTKDr8fB6ntNuvmoL1ZGkXa69Njh43zf9GiGxHQrA_jDYWRSzF5--WmTVrN97_Sm8LbLUy_lGBmLanJjFkDlGkRqjA_4tm00
> > > > > > > > > > >> >> >
> > > > > > > > > > >> >> >> :
> > > > > > > > > > >> >> >>
> > > > > > > > > > >> >> >> [image: image.png]
> > > > > > > > > > >> >> >>
> > > > > > > > > > >> >> >> Diagram 2
> > > > > > > > > > >> >> >> <
> > > > > > > > > > >> >>
> > > > > > > > > > >>
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://www.plantuml.com/plantuml/uml/VLLDR-8m4BtdLupO2sWBLVU8AaGB7AXAbssGzb896SS9RXqxjKqBwkv_tt7iV43fdaZYDpFlpRmnOsE9jhjSH9PRmM31hERKm8scMsuPjJlDe0yheZDc8RR4iYWoBrmMH9CS2a9VICPYUy1OZN0YCy5Q0BCbYNhdCeEK28En8G8wCvbnoQ0R8_05Bc6bkLIz3X03p1zzH7zR-9ZfDquPt9C3qoNCX2yV4G2NbkcKu5jdgGJHt0GbZwnG6i-UP3TUpk5gM6Ldqke350eZUqzoCft3U9xWHvxoa5-7K4nF1J46EbEMafsmdrCBbQ44gVggy18IZrn_ph5asd1ZiIKdQSgueZvjXrQFSFrdC3YN-nXmBacxbGiYyLVxLaBtdhqn0LSzdBDhqQtQoOJeGyad3z0lUqnYgpGB6Ns8oVyta00Dy_WnX0tIOZ8v6SYxHll1TrH6aejAik-mh-AphVFCwSUQqFypElag5QRGFDjQKEd96K1P8QP41c9TzA_IIQyvdAWyv_RSiS3skb0_EzDDkK2v5xWF6MiGFlvhpFLcD2Dq2pml14gaF67eQkmd8gulDoC4kSOu6KVpkvlUJg1RTbWISU40RdBUUS_9XfRZ2dwxm_SW8LYFISgm_MnlDQ6M9P1gbKEc4X-2pH_FvJCkCqm9pbVjD6LrwdLeOrDWfOaqc8Wh9BE85oNKxkNQ6o4yGRy_Eae0G_G8tZv81d3bHDB23WOdisohVr3nh_j6lbSjbNaLRTc8UgtPbAU1J_tygOfZX9DWEJeHDvYx-qmSi5FgNLPZwHrHcUsncGQ5-skhUclpE5fo4ounpFauYrUbkU6ccfnxMvitwag4IyerhTxj8In_Oj1bDO4pQru674loYrGlULHLEGCjwJJ8gDoVZR8MxO4BT3IzRvIcAQKezC6xpziGnTyImrfEGyJI_OcKfgtxIvnTqFEMS17L9Z-jsARN5FmTheP7HtSdtOMT0B4GY2FYHXxgQmMtj2bRqiLFGapiVe1_QVKDrkqXcm83aFEXnMYCZ-xlyHy
> > > > > > > > > > >> >> >
> > > > > > > > > > >> >> >> :
> > > > > > > > > > >> >> >> [image: image.png]
> > > > > > > > > > >> >> >>
> > > > > > > > > > >> >> >>  -ej
> > > > > > > > > > >> >> >>
> > > > > > > > > > >> >> >> On Wed, Apr 15, 2026 at 8:22 AM Dmitri
> > Bourlatchkov
> > > <
> > > > > > > > > > >> dimas@apache.org>
> > > > > > > > > > >> >> >> wrote:
> > > > > > > > > > >> >> >>
> > > > > > > > > > >> >> >>> Hi All,
> > > > > > > > > > >> >> >>>
> > > > > > > > > > >> >> >>> Heads up: The current state of PR [4115] looks
> > > pretty
> > > > > > solid
> > > > > > > > to
> > > > > > > > > > me.
> > > > > > > > > > >> I
> > > > > > > > > > >> >> >>> believe this PR is approaching a mergeable
> > > condition.
> > > > > > > > > > >> >> >>>
> > > > > > > > > > >> >> >>> Please post your reviews if you have any
> > comments.
> > > > > > > > > > >> >> >>>
> > > > > > > > > > >> >> >>> [4115]
> > https://github.com/apache/polaris/pull/4115
> > > > > > > > > > >> >> >>>
> > > > > > > > > > >> >> >>> Thanks,
> > > > > > > > > > >> >> >>> Dmitri.
> > > > > > > > > > >> >> >>>
> > > > > > > > > > >> >> >>> On Tue, Mar 3, 2026 at 3:29 PM Anand Kumar
> > Sankaran
> > > > via
> > > > > > > dev <
> > > > > > > > > > >> >> >>> dev@polaris.apache.org> wrote:
> > > > > > > > > > >> >> >>>
> > > > > > > > > > >> >> >>> > Hi Yufei and Dmitri,
> > > > > > > > > > >> >> >>> >
> > > > > > > > > > >> >> >>> > Here is a proposal for the REST endpoints for
> > > > metrics
> > > > > > and
> > > > > > > > > > events.
> > > > > > > > > > >> >> >>> >
> > > > > > > > > > >> >> >>> >
> > > > https://github.com/apache/polaris/pull/3924/changes
> > > > > > > > > > >> >> >>> >
> > > > > > > > > > >> >> >>> > I did not see any precursors for raising a PR
> > for
> > > > > > > > proposals,
> > > > > > > > > so
> > > > > > > > > > >> >> trying
> > > > > > > > > > >> >> >>> > this.  Please let me know what you think.
> > > > > > > > > > >> >> >>> >
> > > > > > > > > > >> >> >>> > -
> > > > > > > > > > >> >> >>> > Anand
> > > > > > > > > > >> >> >>> >
> > > > > > > > > > >> >> >>> > From: Anand Kumar Sankaran <
> > > > > anand.sankaran@workday.com
> > > > > > >
> > > > > > > > > > >> >> >>> > Date: Monday, March 2, 2026 at 10:25 AM
> > > > > > > > > > >> >> >>> > To: dev@polaris.apache.org <
> > > dev@polaris.apache.org
> > > > >
> > > > > > > > > > >> >> >>> > Subject: Re: Polaris Telemetry and Audit
> Trail
> > > > > > > > > > >> >> >>> >
> > > > > > > > > > >> >> >>> > About the REST API, based on my use cases:
> > > > > > > > > > >> >> >>> >
> > > > > > > > > > >> >> >>> >
> > > > > > > > > > >> >> >>> >   1.
> > > > > > > > > > >> >> >>> > I want to be able to query commit metrics to
> > > track
> > > > > > files
> > > > > > > > > added
> > > > > > > > > > /
> > > > > > > > > > >> >> >>> removed
> > > > > > > > > > >> >> >>> > per commit, along with record counts. The
> > > ingestion
> > > > > > > > pipeline
> > > > > > > > > > that
> > > > > > > > > > >> >> >>> writes
> > > > > > > > > > >> >> >>> > this data is owned by us and we are
> guaranteed
> > to
> > > > > write
> > > > > > > > this
> > > > > > > > > > >> >> >>> information
> > > > > > > > > > >> >> >>> > for each write.
> > > > > > > > > > >> >> >>> >   2.
> > > > > > > > > > >> >> >>> > I want to be able to query scan metrics for
> > > read. I
> > > > > > > > > understand
> > > > > > > > > > >> >> clients
> > > > > > > > > > >> >> >>> do
> > > > > > > > > > >> >> >>> > not fulfill this requirement.
> > > > > > > > > > >> >> >>> >   3.
> > > > > > > > > > >> >> >>> > I want to be able to query the events table
> > > (events
> > > > > are
> > > > > > > > > > >> persisted) -
> > > > > > > > > > >> >> >>> this
> > > > > > > > > > >> >> >>> > may supersede #2, I am not sure yet.
> > > > > > > > > > >> >> >>> >
> > > > > > > > > > >> >> >>> > All this information is in the JDBC based
> > > > persistence
> > > > > > > model
> > > > > > > > > and
> > > > > > > > > > >> is
> > > > > > > > > > >> >> >>> > persisted in the metastore. I currently don’t
> > > have
> > > > a
> > > > > > need
> > > > > > > > to
> > > > > > > > > > >> query
> > > > > > > > > > >> >> >>> > prometheus or open telemetry. I do publish
> some
> > > > > events
> > > > > > to
> > > > > > > > > > >> Prometheus
> > > > > > > > > > >> >> >>> and
> > > > > > > > > > >> >> >>> > they are forwarded to our dashboards
> elsewhere.
> > > > > > > > > > >> >> >>> >
> > > > > > > > > > >> >> >>> > About the CLI utilities, I meant the admin
> user
> > > > > > > utilities.
> > > > > > > > In
> > > > > > > > > > >> one of
> > > > > > > > > > >> >> >>> the
> > > > > > > > > > >> >> >>> > earliest drafts of my proposal, Prashant
> > > mentioned
> > > > > that
> > > > > > > the
> > > > > > > > > > >> metrics
> > > > > > > > > > >> >> >>> tables
> > > > > > > > > > >> >> >>> > can grow indefinitely and that a similar
> > problem
> > > > > exists
> > > > > > > > with
> > > > > > > > > > the
> > > > > > > > > > >> >> events
> > > > > > > > > > >> >> >>> > table as well. We discussed that cleaning up
> of
> > > old
> > > > > > > records
> > > > > > > > > > from
> > > > > > > > > > >> >> both
> > > > > > > > > > >> >> >>> > metrics tables and events tables can be done
> > via
> > > a
> > > > > CLI
> > > > > > > > > utility.
> > > > > > > > > > >> >> >>> >
> > > > > > > > > > >> >> >>> > I see that Yufei has covered the discussion
> > about
> > > > > > > > > datasources.
> > > > > > > > > > >> >> >>> >
> > > > > > > > > > >> >> >>> > -
> > > > > > > > > > >> >> >>> > Anand
> > > > > > > > > > >> >> >>> >
> > > > > > > > > > >> >> >>> >
> > > > > > > > > > >> >> >>> >
> > > > > > > > > > >> >> >>> > From: Yufei Gu <fl...@gmail.com>
> > > > > > > > > > >> >> >>> > Date: Friday, February 27, 2026 at 9:54 PM
> > > > > > > > > > >> >> >>> > To: dev@polaris.apache.org <
> > > dev@polaris.apache.org
> > > > >
> > > > > > > > > > >> >> >>> > Subject: Re: Polaris Telemetry and Audit
> Trail
> > > > > > > > > > >> >> >>> >
> > > > > > > > > > >> >> >>> > This Message Is From an External Sender
> > > > > > > > > > >> >> >>> > This message came from outside your
> > organization.
> > > > > > > > > > >> >> >>> > Report Suspicious<
> > > > > > > > > > >> >> >>> >
> > > > > > > > > > >> >> >>>
> > > > > > > > > > >> >>
> > > > > > > > > > >>
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://us-phishalarm-ewt.proofpoint.com/EWT/v1/Iz9xO38YGHZK!YhNDZABkHi1B699ote2uMwpOZw8i0QMCGO2Szc-HshuABGhGvwPJcymE6G2oUUxtS8xDkSrtGTPm_I3QnVDHoLMk50m9v8z_nZKTkd-bnVUbreF1u0WnfV_X5eYevZl_$
> > > > > > > > > > >> >> >>> > >
> > > > > > > > > > >> >> >>> >
> > > > > > > > > > >> >> >>> >
> > > > > > > > > > >> >> >>> > As I mentioned in
> > > > > > > > > > >> >> >>> >
> > > > > > > > > > >> >> >>>
> > > > > > > > > > >> >>
> > > > > > > > > > >>
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://urldefense.com/v3/__https://github.com/apache/polaris/issues/3890__;!!Iz9xO38YGHZK!5EuyFFkk3vhRWVIRvQAWBSQfpJkTMA9HxugzDwXmN0LPPqhEFxYkFRGVhtb8AqUwXtDh2OplcMnbMDHKOxrvDU0$
> > > > > > > > > > >> >> >>> ,
> > > > > > > > > > >> >> >>> > supporting
> > > > > > > > > > >> >> >>> > multiple data sources is not a trivial
> change.
> > I
> > > > > would
> > > > > > > > > strongly
> > > > > > > > > > >> >> >>> recommend
> > > > > > > > > > >> >> >>> > starting with a design document to carefully
> > > > evaluate
> > > > > > the
> > > > > > > > > > >> >> architectural
> > > > > > > > > > >> >> >>> > implications and long term impact.
> > > > > > > > > > >> >> >>> >
> > > > > > > > > > >> >> >>> > A REST endpoint to query metrics seems
> > reasonable
> > > > > given
> > > > > > > the
> > > > > > > > > > >> current
> > > > > > > > > > >> >> >>> JDBC
> > > > > > > > > > >> >> >>> > based persistence model. That said, we may
> also
> > > > > > consider
> > > > > > > > > > >> alternative
> > > > > > > > > > >> >> >>> > storage models. For example, if we later
> adopt
> > a
> > > > time
> > > > > > > > series
> > > > > > > > > > >> system
> > > > > > > > > > >> >> >>> such as
> > > > > > > > > > >> >> >>> > Prometheus to store metrics, the query model
> > and
> > > > > access
> > > > > > > > > > patterns
> > > > > > > > > > >> >> would
> > > > > > > > > > >> >> >>> be
> > > > > > > > > > >> >> >>> > fundamentally different. Designing the REST
> API
> > > > > without
> > > > > > > > > > >> considering
> > > > > > > > > > >> >> >>> these
> > > > > > > > > > >> >> >>> > potential evolutions may limit flexibility.
> I'd
> > > > > suggest
> > > > > > > to
> > > > > > > > > > start
> > > > > > > > > > >> >> with
> > > > > > > > > > >> >> >>> the
> > > > > > > > > > >> >> >>> > use case.
> > > > > > > > > > >> >> >>> >
> > > > > > > > > > >> >> >>> > Yufei
> > > > > > > > > > >> >> >>> >
> > > > > > > > > > >> >> >>> >
> > > > > > > > > > >> >> >>> > On Fri, Feb 27, 2026 at 3:42 PM Dmitri
> > > > Bourlatchkov <
> > > > > > > > > > >> >> dimas@apache.org>
> > > > > > > > > > >> >> >>> > wrote:
> > > > > > > > > > >> >> >>> >
> > > > > > > > > > >> >> >>> > > Hi Anand,
> > > > > > > > > > >> >> >>> > >
> > > > > > > > > > >> >> >>> > > Sharing my view... subject to discussion:
> > > > > > > > > > >> >> >>> > >
> > > > > > > > > > >> >> >>> > > 1. Adding non-IRC REST API to Polaris is
> > > > perfectly
> > > > > > > fine.
> > > > > > > > > > >> >> >>> > >
> > > > > > > > > > >> >> >>> > > Figuring out specific endpoint URIs and
> > > payloads
> > > > > > might
> > > > > > > > > > require
> > > > > > > > > > >> a
> > > > > > > > > > >> >> few
> > > > > > > > > > >> >> >>> > > roundtrips, so opening a separate thread
> for
> > > that
> > > > > > might
> > > > > > > > be
> > > > > > > > > > >> best.
> > > > > > > > > > >> >> >>> > > Contributors commonly create Google Docs
> for
> > > new
> > > > > API
> > > > > > > > > > proposals
> > > > > > > > > > >> too
> > > > > > > > > > >> >> >>> (they
> > > > > > > > > > >> >> >>> > > fairly easy to update as the email
> discussion
> > > > > > > > progresses).
> > > > > > > > > > >> >> >>> > >
> > > > > > > > > > >> >> >>> > > There was a suggestion to try Markdown
> (with
> > > PRs)
> > > > > for
> > > > > > > > > > proposals
> > > > > > > > > > >> >> [1]
> > > > > > > > > > >> >> >>> ...
> > > > > > > > > > >> >> >>> > > feel free to give it a try if you are
> > > comfortable
> > > > > > with
> > > > > > > > > that.
> > > > > > > > > > >> >> >>> > >
> > > > > > > > > > >> >> >>> > > 2. Could you clarify whether you mean end
> > user
> > > > > > > utilities
> > > > > > > > or
> > > > > > > > > > >> admin
> > > > > > > > > > >> >> >>> user
> > > > > > > > > > >> >> >>> > > utilities? In the latter case those might
> be
> > > more
> > > > > > > > suitable
> > > > > > > > > > for
> > > > > > > > > > >> the
> > > > > > > > > > >> >> >>> Admin
> > > > > > > > > > >> >> >>> > > CLI (java) not the Python CLI, IMHO.
> > > > > > > > > > >> >> >>> > >
> > > > > > > > > > >> >> >>> > > Why would these utilities be common with
> > > events?
> > > > > > IMHO,
> > > > > > > > > event
> > > > > > > > > > >> use
> > > > > > > > > > >> >> >>> cases
> > > > > > > > > > >> >> >>> > are
> > > > > > > > > > >> >> >>> > > distinct from scan/commit metrics.
> > > > > > > > > > >> >> >>> > >
> > > > > > > > > > >> >> >>> > > 3. I'd prefer separating metrics
> persistence
> > > from
> > > > > > > > MetaStore
> > > > > > > > > > >> >> >>> persistence
> > > > > > > > > > >> >> >>> > at
> > > > > > > > > > >> >> >>> > > the code level, so that they could be mixed
> > and
> > > > > > matched
> > > > > > > > > > >> >> >>> independently.
> > > > > > > > > > >> >> >>> > The
> > > > > > > > > > >> >> >>> > > separate datasource question will become a
> > > > > non-issue
> > > > > > > with
> > > > > > > > > > that
> > > > > > > > > > >> >> >>> approach,
> > > > > > > > > > >> >> >>> > I
> > > > > > > > > > >> >> >>> > > guess.
> > > > > > > > > > >> >> >>> > >
> > > > > > > > > > >> >> >>> > > The rationale for separating scan metrics
> and
> > > > > > metastore
> > > > > > > > > > >> >> persistence
> > > > > > > > > > >> >> >>> is
> > > > > > > > > > >> >> >>> > that
> > > > > > > > > > >> >> >>> > > "cascading deletes" between them are hardly
> > > ever
> > > > > > > > required.
> > > > > > > > > > >> >> >>> Furthermore,
> > > > > > > > > > >> >> >>> > the
> > > > > > > > > > >> >> >>> > > data and query patterns are very different
> so
> > > > > > different
> > > > > > > > > > >> >> technologies
> > > > > > > > > > >> >> >>> > might
> > > > > > > > > > >> >> >>> > > be beneficial in each case.
> > > > > > > > > > >> >> >>> > >
> > > > > > > > > > >> >> >>> > > [1]
> > > > > > > > > > >> >> >>> >
> > > > > > > > > > >> >> >>>
> > > > > > > > > > >> >>
> > > > > > > > > > >>
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://urldefense.com/v3/__https://lists.apache.org/thread/yto2wp982t43h1mqjwnslswhws5z47cy__;!!Iz9xO38YGHZK!5EuyFFkk3vhRWVIRvQAWBSQfpJkTMA9HxugzDwXmN0LPPqhEFxYkFRGVhtb8AqUwXtDh2OplcMnbMDHKxYDakNU$
> > > > > > > > > > >> >> >>> > >
> > > > > > > > > > >> >> >>> > > Cheers,
> > > > > > > > > > >> >> >>> > > Dmitri.
> > > > > > > > > > >> >> >>> > >
> > > > > > > > > > >> >> >>> > > On Fri, Feb 27, 2026 at 6:19 PM Anand Kumar
> > > > > Sankaran
> > > > > > > via
> > > > > > > > > dev
> > > > > > > > > > <
> > > > > > > > > > >> >> >>> > > dev@polaris.apache.org> wrote:
> > > > > > > > > > >> >> >>> > >
> > > > > > > > > > >> >> >>> > > > Thanks all. This PR is merged now.
> > > > > > > > > > >> >> >>> > > >
> > > > > > > > > > >> >> >>> > > > Here are the follow-up features / work
> > > needed.
> > > > > > These
> > > > > > > > > were
> > > > > > > > > > >> all
> > > > > > > > > > >> >> >>> part of
> > > > > > > > > > >> >> >>> > > the
> > > > > > > > > > >> >> >>> > > > merged PR at some point in time and were
> > > > removed
> > > > > to
> > > > > > > > > reduce
> > > > > > > > > > >> >> scope.
> > > > > > > > > > >> >> >>> > > >
> > > > > > > > > > >> >> >>> > > > Please let me know what you think.
> > > > > > > > > > >> >> >>> > > >
> > > > > > > > > > >> >> >>> > > >
> > > > > > > > > > >> >> >>> > > >   1.  A REST API to paginate through
> table
> > > > > metrics.
> > > > > > > > This
> > > > > > > > > > >> will be
> > > > > > > > > > >> >> >>> > non-IRC
> > > > > > > > > > >> >> >>> > > > standard addition.
> > > > > > > > > > >> >> >>> > > >   2.  Utilities for managing old records,
> > > > should
> > > > > be
> > > > > > > > > common
> > > > > > > > > > >> with
> > > > > > > > > > >> >> >>> events.
> > > > > > > > > > >> >> >>> > > > There was some discussion that it belongs
> > to
> > > > the
> > > > > > CLI.
> > > > > > > > > > >> >> >>> > > >   3.  Separate datasource (metrics,
> events,
> > > > even
> > > > > > > other
> > > > > > > > > > >> tables?).
> > > > > > > > > > >> >> >>> > > >
> > > > > > > > > > >> >> >>> > > >
> > > > > > > > > > >> >> >>> > > > Anything else?
> > > > > > > > > > >> >> >>> > > >
> > > > > > > > > > >> >> >>> > > > -
> > > > > > > > > > >> >> >>> > > > Anand
> > > > > > > > > > >> >> >>> > > >
> > > > > > > > > > >> >> >>> > > >
> > > > > > > > > > >> >> >>> > >
> > > > > > > > > > >> >> >>> >
> > > > > > > > > > >> >> >>> >
> > > > > > > > > > >> >> >>>
> > > > > > > > > > >> >> >>
> > > > > > > > > > >> >>
> > > > > > > > > > >> >
> > > > > > > > > > >>
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>

Re: Proposal for REST endpoints for table metrics and events

[EJ Wang <ej...@gmail.com>]

Hi folks,

Thanks again for the discussion today. I updated the sync doc with notes
from today's metrics sync:

https://docs.google.com/document/d/100h7c4damrUzVuquYbBHM0EvA4LSWuW2IT2dN_7nYVA/edit?tab=t.ezk0rgdx0c6m

A few highlights:

   - No objection to keeping the default metrics behavior small:
   no-op/log-only is enough as the built-in default.
   - Durable storage, event forwarding, external queues, dashboards, and
   custom filtering should be implementation choices behind the metrics
   reporting path.
   - For PR #4115, the REST layer should resolve/authz/accept the report,
   then delegate to the selected metrics implementation. It should not own
   durable storage semantics.
   - The rough boundary we discussed is: spec owns the wire contract,
   runtime owns framework wiring, API/contract modules own provider-facing
   contracts, and extensions own replaceable implementations.
   - OpenLineage has a similar path-naming question. We should avoid
   occupying a generic lineage path too early if we want room for other
   lineage systems later.

I believe the boundary rules previewed in sync meeting are useful for
reviewing PR #4115, but it is not
a final module-layout proposal.

For metrics specifically, I think the target remains:
   runtime
     -> metrics reporting/emitting contract
     -> default implementation
          -> lower implementation details, if needed

That keeps the default simple, while allowing durable JDBC, event-backed,
or external-queue implementations to be added separately.

We also touched on the split between Iceberg metrics emitting and
Polaris-owned metrics querying. The querying side can keep evolving,
especially for dashboard or generic-table use cases, but I do not think
that needs to block the intake path in this PR.

Thanks,
-ej

On Tue, Jun 16, 2026 at 2:17 PM Yufei Gu <fl...@gmail.com> wrote:

> Thanks for chiming in, EJ. Agreed that the default battery should stay
> small. I'm leaning toward not including metrics persistence in the default
> battery.
>
> Yufei
>
>
> On Thu, Jun 11, 2026 at 8:30 PM EJ Wang <ej...@gmail.com>
> wrote:
>
> > Hi Yufei,
> >
> > Thanks for connecting this back to the REST endpoint proposal. I replied
> > with the fuller version on the event-forwarding thread, but wanted to add
> > the shorter version here too.
> >
> > I agree that metrics reporting/emitting is the right conceptual boundary,
> > and I also think the event/listener path is a good implementation
> direction
> > to explore. The distinction I want to keep clear is default battery vs
> > extension implementation.
> >
> > For the REST/API side, I would keep the semantics narrow: Polaris
> resolves
> > and authorizes the table, accepts the Iceberg scan/commit metrics report
> > into the configured reporting/emitting path, and returns 204. That 204
> > should mean Polaris accepted the report into the ingestion path; it
> should
> > not imply durable storage.
> >
> > Durable storage, event forwarding, external telemetry routing, filtering,
> > and retention should sit behind that reporting/emitting boundary as
> > implementation-layer behavior. A durable JDBC path can be one named
> > extension implementation. An event/listener forwarding path can be
> another
> > named extension implementation.
> >
> > So I am +1 on the event-forwarding idea as a non-default extension
> > implementation. I just would not make it the default battery or the core
> > API shape. The default battery can stay small and safe, while deployments
> > that need forwarding, dashboards, or durable storage can opt into the
> > implementation that matches their operational model.
> >
> > Thanks,
> > -ej
> >
> > On Wed, Jun 10, 2026 at 11:43 AM Yufei Gu <fl...@gmail.com> wrote:
> >
> > > Thanks EJ. I agree that the reporting/emitting boundary feels like the
> > > right SPI boundary, but I wonder if we can simplify this even further.
> > >
> > > Iceberg scan and commit metrics look very similar to events to me. They
> > are
> > > append only, asynchronously consumed, often forwarded to external
> > systems,
> > > and have similar retention and cleanup requirements. More details can
> be
> > > found in this thread [1]. In that case, Polaris may only need to accept
> > the
> > > report and emit an event through the existing event framework.
> Filtering,
> > > async delivery, custom sinks, and retention mechanisms could then be
> > shared
> > > instead of introducing a separate metrics specific extension layer and
> > SPI.
> > >
> > > The main requirement I still see is querying metrics through Polaris.
> If
> > we
> > > want that battery included experience, we could work on a persistence
> > > listener implementation that selectively saves metrics and expose it
> via
> > > the IRC event endpoint which seems pretty close in Iceberg
> > > community. Another option I like more is to allow users to develop
> their
> > > own listener, so they can persist scan/commit metrics to any systems
> they
> > > prefer. In general, that feels like an implementation choice on top of
> > the
> > > event framework rather than something that needs to shape the core
> > > architecture.
> > >
> > > 1. https://lists.apache.org/thread/x9j8nscvy8hq61tyn01mj8yp6n9of0kp
> > >
> > > Thanks,
> > > Yufei
> > >
> > >
> > > On Thu, Jun 4, 2026 at 5:14 PM EJ Wang <ej.wang.dev.interest@gmail.com
> >
> > > wrote:
> > >
> > > > Good point. I agree the existing PolarisMetricsReporter is already
> very
> > > > close to the right conceptual boundary. I am not proposing a second
> > > > parallel reporter concept. The distinction I am trying to make is
> > > narrower:
> > > >
> > > > Current state:
> > > >
> > > >    - PolarisMetricsReporter is the existing metrics reporting hook.
> > > >    - But it currently lives under runtime/service.
> > > >    - Its contract is documented around Quarkus/CDI discovery and
> > > > @Identifier
> > > >    selection.
> > > >    - The default implementation is log-only.
> > > >    - The durable implementation writes through PolarisMetricsManager
> > and
> > > >    MetricsPersistence.
> > > >    - MetricsPersistence is currently inherited by BasePersistence.
> > > >
> > > > My proposal:
> > > >
> > > >    - Keep the reporting/emitting boundary as the SPI.
> > > >    - Revise that contract in a framework-agnostic optional Iceberg
> > > *metrics
> > > >    extension API layer*.
> > > >    - Keep runtime/service responsible for REST ingestion, table/authz
> > > >    validation, and runtime wiring.
> > > >    - Keep the battery implementation log-only in the metrics
> extension
> > > API
> > > >    layer, *not under runtime/service*.
> > > >    - Treat durable JDBC metrics as one implementation of the
> reporting
> > > SPI.
> > > >    - Decompose the MetricsPersistence and BasePersistence coupling as
> > > >    durable implementation detail.
> > > >
> > > > So the difference is not “new reporter concept vs old reporter
> > concept.”
> > > > but:
> > > >
> > > >
> > > >    - PolarisMetricsReporter today = runtime/service CDI-shaped hook.
> > > >    - Target reporting SPI = same conceptual boundary, but placed in
> the
> > > >    right extension API layer and kept framework-agnostic.
> > > >
> > > > For the current metrics PR, I think the minimal architectural cleanup
> > > > before merge is to *put the reporting SPI contract and the battery
> > > default
> > > > in the right place*. The durable JDBC implementation, event-backed
> > async
> > > > implementation, external queue support, and deeper persistence
> cleanup
> > > can
> > > > continue as follow-ups, as long as we do not lock the SPI boundary to
> > > > BasePersistence or to runtime/service wiring.
> > > >
> > > > -ej
> > > >
> > > > On Wed, Jun 3, 2026 at 3:00 PM Dmitri Bourlatchkov <dimas@apache.org
> >
> > > > wrote:
> > > >
> > > > > Hi EJ,
> > > > >
> > > > > Thanks for the recap / summary!
> > > > >
> > > > > > Do folks agree that the stable SPI boundary should be
> > > > > metrics reporting/emitting, not metrics persistence?
> > > > >
> > > > > SGTM.
> > > > >
> > > > > > Does an optional Iceberg metrics extension API layer sound like
> the
> > > > right
> > > > > home for this SPI?
> > > > >
> > > > > How is that different from current PolarisMetricsReporter?
> > > > >
> > > > > > Should the current durable metrics work be reframed as a durable
> > > > > JDBC reference implementation of that SPI?
> > > > >
> > > > > SGTM.
> > > > >
> > > > > > What is the smallest PR sequence to get there without blocking
> > > > > the current metrics work unnecessarily?
> > > > >
> > > > > Let's get https://github.com/apache/polaris/pull/4397 merged
> first.
> > > > >
> > > > > Then, I'd propose isolating the Metrics schema from the MetaStore
> > > schema.
> > > > > This will probably have some ripple effect into the bootstrap
> > > workflows,
> > > > so
> > > > > it's not a trivial change.
> > > > >
> > > > > Then, let's reassess.
> > > > >
> > > > > Cheers,
> > > > > Dmitri.
> > > > >
> > > > > On Wed, Jun 3, 2026 at 5:40 PM EJ Wang <
> > ej.wang.dev.interest@gmail.com
> > > >
> > > > > wrote:
> > > > >
> > > > > > Hi folks,
> > > > > >
> > > > > > Thanks again for the discussion today. I updated the sync doc
> with
> > > > notes
> > > > > > from the third metrics architecture sync:
> > > > > >
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://docs.google.com/document/d/100h7c4damrUzVuquYbBHM0EvA4LSWuW2IT2dN_7nYVA/edit?tab=t.k96s2xyqr5u1
> > > > > >
> > > > > > A few highlights from today’s discussion:
> > > > > >
> > > > > >    - We clarified that Iceberg metrics reporting can be
> interpreted
> > > > > either
> > > > > >    as sync or async handling from Polaris’s perspective, so
> Polaris
> > > > > should
> > > > > >    stay flexible as a platform instead of baking one handling
> model
> > > > into
> > > > > > the
> > > > > >    REST API behavior.
> > > > > >    - We aligned that the built-in (battery) behavior for metrics
> > > > emitting
> > > > > >    can stay simple: no-op/log-only is enough as the default.
> > > > > >    - Durable metrics persistence should be treated as an
> > > implementation
> > > > > of
> > > > > >    the metrics reporting path, not as the core SPI boundary.
> > > > > >    - The existing durable metrics work can be reviewed as a
> > reference
> > > > > >    implementation of the reporting SPI, with persistence-related
> > > logic
> > > > > kept
> > > > > >    self-contained under a metrics durable implementation module
> > > rather
> > > > > than
> > > > > >    scattered through core entity persistence.
> > > > > >    - Dashboard/insights remains a real use case, but we agreed to
> > > keep
> > > > it
> > > > > >    separate from the core metrics intake discussion for now.
> > > > > >
> > > > > > I also did a quick source check after the meeting to make sure we
> > are
> > > > > > describing the current state accurately.
> > > > > >
> > > > > > Current state:
> > > > > >
> > > > > >    - Polaris already has a metrics reporting hook:
> > > > > PolarisMetricsReporter.
> > > > > >    - The default implementation is DefaultMetricsReporter,
> selected
> > > by
> > > > > >    polaris.iceberg-metrics.reporting.type=default. It is
> log-only,
> > > > > >    effectively quiet unless metrics logging is enabled.
> > > > > >    - There is also a PersistingMetricsReporter, selected by
> > > > > >    polaris.iceberg-metrics.reporting.type=persisting, which
> > converts
> > > > > >    Iceberg scan/commit reports into Polaris metrics records and
> > > writes
> > > > > > through
> > > > > >    PolarisMetricsManager -> MetricsPersistence.
> > > > > >    - MetricsPersistence currently exists in the persistence layer
> > and
> > > > is
> > > > > >    inherited by BasePersistence.
> > > > > >
> > > > > > My read from the discussion is that the target boundary should
> not
> > be
> > > > > > MetricsPersistence as inherited by BasePersistence. That path
> > should
> > > be
> > > > > > decomposed as durable implementation detail (taken care of by
> > > > > > https://github.com/apache/polaris/pull/4397). The stable
> extension
> > > > point
> > > > > > should instead be the metrics reporting/emitting boundary.
> > > > > >
> > > > > > Concretely, I think the next proposal should be shaped like this:
> > > > > >
> > > > > >    1.
> > > > > >
> > > > > >    Define the proper metrics reporting SPI at an optional Iceberg
> > > > metrics
> > > > > >    extension API layer.
> > > > > >
> > > > > >    Example direction:
> > > > > >
> > > > > >    public interface IcebergMetricsReporter {
> > > > > >      void reportMetric(IcebergMetricsReportContext context,
> > > > > > MetricsReport report);
> > > > > >    }
> > > > > >
> > > > > >    The context would carry the small Polaris-resolved envelope:
> > > > > >    catalog/table identity, report type, received timestamp, and
> > > > > > request/trace
> > > > > >    context if available. The raw Iceberg MetricsReport remains
> the
> > > > > payload.
> > > > > >    2.
> > > > > >
> > > > > >    Keep runtime/service as the REST ingestion and wiring layer.
> > > > > >
> > > > > >    The REST handler still resolves the table and performs authz
> > > before
> > > > > >    accepting the report. After that, it calls the selected
> > reporting
> > > > > >    implementation.
> > > > > >    3.
> > > > > >
> > > > > >    Keep the battery default no-op/log-only.
> > > > > >
> > > > > >    This preserves an out-of-box safe default and avoids
> requiring a
> > > > > durable
> > > > > >    metrics store for every Polaris deployment.
> > > > > >    4.
> > > > > >
> > > > > >    Move durable JDBC metrics into a self-contained implementation
> > of
> > > > the
> > > > > >    reporting SPI.
> > > > > >
> > > > > >    That implementation can own its schema, bootstrap, retention,
> > and
> > > > read
> > > > > >    API support. It should not define the core reporting SPI
> > boundary,
> > > > and
> > > > > > it
> > > > > >    should not require metrics persistence to remain inherited
> from
> > > > > >    BasePersistence.
> > > > > >    5.
> > > > > >
> > > > > >    Treat async/event-backed handling as another implementation of
> > the
> > > > > same
> > > > > >    reporting SPI.
> > > > > >
> > > > > >    For example, an event-backed reporter could enqueue a metrics
> > > event
> > > > > and
> > > > > >    let listeners handle durable storage or other sinks. If we
> later
> > > > need
> > > > > a
> > > > > >    replaceable queue engine, that seems like a shared
> event/metrics
> > > > > > substrate
> > > > > >    topic rather than a metrics-only requirement.
> > > > > >
> > > > > > This framing lets us keep the REST metrics endpoint simple,
> > preserve
> > > > the
> > > > > > current default behavior, support durable metrics users, and
> still
> > > > leave
> > > > > > room for async/event-backed or external-queue-based
> > implementations.
> > > > > >
> > > > > > I think the main follow-up questions are:
> > > > > >
> > > > > >    - Do folks agree that the stable SPI boundary should be
> metrics
> > > > > >    reporting/emitting, not metrics persistence?
> > > > > >    - Does an optional Iceberg metrics extension API layer sound
> > like
> > > > the
> > > > > >    right home for this SPI?
> > > > > >    - Should the current durable metrics work be reframed as a
> > durable
> > > > > JDBC
> > > > > >    reference implementation of that SPI?
> > > > > >    - What is the smallest PR sequence to get there without
> blocking
> > > the
> > > > > >    current metrics work unnecessarily?
> > > > > >
> > > > > > Thanks,
> > > > > > -ej
> > > > > >
> > > > > > On Fri, May 15, 2026 at 5:16 PM Dmitri Bourlatchkov <
> > > dimas@apache.org>
> > > > > > wrote:
> > > > > >
> > > > > > > Hi JB,
> > > > > > >
> > > > > > > Could you set up another meeting, please? Same time on
> Wednesday
> > as
> > > > > last
> > > > > > > time... I hope it works for everyone.
> > > > > > >
> > > > > > > Cheers,
> > > > > > > Dmitri.
> > > > > > >
> > > > > > > On Fri, May 15, 2026 at 8:06 PM Yufei Gu <flyrain000@gmail.com
> >
> > > > wrote:
> > > > > > >
> > > > > > > > +1 on another sync call next week.
> > > > > > > >
> > > > > > > > Yufei
> > > > > > > >
> > > > > > > >
> > > > > > > > On Fri, May 15, 2026 at 4:52 PM Dmitri Bourlatchkov <
> > > > > dimas@apache.org>
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > > > Hi All,
> > > > > > > > >
> > > > > > > > > WDYT about another sync call next week?
> > > > > > > > >
> > > > > > > > > Thanks,
> > > > > > > > > Dmitri.
> > > > > > > > >
> > > > > > > > > On Wed, May 6, 2026 at 5:29 PM Dmitri Bourlatchkov <
> > > > > dimas@apache.org
> > > > > > >
> > > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > Hi EJ,
> > > > > > > > > >
> > > > > > > > > > Thanks for the summary! It covers what we discussed in
> the
> > > > > meeting
> > > > > > > very
> > > > > > > > > > well, IMHO.
> > > > > > > > > >
> > > > > > > > > > Looking forward to concrete PRs :)
> > > > > > > > > >
> > > > > > > > > > Cheers,
> > > > > > > > > > Dmitri.
> > > > > > > > > >
> > > > > > > > > > On Wed, May 6, 2026 at 5:08 PM EJ Wang <
> > > > > > > ej.wang.dev.interest@gmail.com
> > > > > > > > >
> > > > > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > >> Hi folks,
> > > > > > > > > >>
> > > > > > > > > >> We had a community sync earlier, thanks JB for
> scheduling
> > > it.
> > > > > > Notes
> > > > > > > > from
> > > > > > > > > >> the first metrics architecture sync (May 6, 10-11am PT).
> > > > > > Discussion
> > > > > > > > doc
> > > > > > > > > >> with per-section status:
> > > > > > > > > >>
> > > > > > > > > >>
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://docs.google.com/document/d/100h7c4damrUzVuquYbBHM0EvA4LSWuW2IT2dN_7nYVA/edit?tab=t.0
> > > > > > > > > >>
> > > > > > > > > >> *The meeting covered both topics from the doc.
> > > Direction-level
> > > > > > > > alignment
> > > > > > > > > >> was reached on the headline pieces; details remain for
> PR
> > > > review
> > > > > > or
> > > > > > > > > >> follow-up sessions.*
> > > > > > > > > >>
> > > > > > > > > >> *Topic 1 — Persistence schema redesign*
> > > > > > > > > >> Idea-level alignment on consolidating per-type tables
> > > > > > > > > >> (scan_metrics_report,
> > > > > > > > > >> commit_metrics_report) into a single metrics_report
> table.
> > > The
> > > > > > > > > motivating
> > > > > > > > > >> cost is the surface area added by every new metric type
> > > today:
> > > > > new
> > > > > > > > > table,
> > > > > > > > > >> SPI method, record class, model, converter, schema
> > > migration.
> > > > > > > > > >>
> > > > > > > > > >> Most schema details are deferred to the schema PR. A few
> > > > > specific
> > > > > > > > points
> > > > > > > > > >> came up:
> > > > > > > > > >> •   metric_schema_version: Yufei prefers dropping it,
> > since
> > > > > there
> > > > > > is
> > > > > > > > no
> > > > > > > > > >> spec-level concept of metrics versioning today and it is
> > > hard
> > > > to
> > > > > > > > define
> > > > > > > > > >> unilaterally. Robert prefers keeping it, given IRC v2 is
> > > > coming
> > > > > > and
> > > > > > > > the
> > > > > > > > > >> schema should be considered against its likely shape;
> > Robert
> > > > > also
> > > > > > > > raised
> > > > > > > > > >> how to differentiate various payload formats if any.
> EJ's
> > > read
> > > > > is
> > > > > > > that
> > > > > > > > > >> this
> > > > > > > > > >> is a two-way-door decision. We can start without the
> > field,
> > > > and
> > > > > if
> > > > > > > IRC
> > > > > > > > > v2
> > > > > > > > > >> changes the shape we would likely roll a corresponding
> new
> > > > > schema
> > > > > > > > > anyway,
> > > > > > > > > >> which is not particularly costly.
> > > > > > > > > >> •   Payload format: Robert pointed out that future
> formats
> > > > > beyond
> > > > > > > JSON
> > > > > > > > > may
> > > > > > > > > >> be worth supporting. The exact shape is deferred to the
> > > schema
> > > > > > > > > discussion.
> > > > > > > > > >> •   Partition strategy: Anand suggested monthly
> > partitioning
> > > > > based
> > > > > > > on
> > > > > > > > > his
> > > > > > > > > >> experience as potentially helpful at scale.
> > > > > > > > > >>
> > > > > > > > > >> *Topic 2 — Where metrics ingestion and storage belong*
> > > > > > > > > >> Idea-level alignment that metrics should be a separated
> > SPI
> > > > from
> > > > > > the
> > > > > > > > > >> entity
> > > > > > > > > >> persistence stack. Two reasons surfaced: (a) workloads
> and
> > > > > > > capability
> > > > > > > > > >> requirements diverge enough that coupling them creates
> > > > > artificial
> > > > > > > > > >> constraints, and (b) admin experience improves when
> > metrics
> > > > has
> > > > > > its
> > > > > > > > own
> > > > > > > > > >> bootstrap, retention, and lifecycle. Dmitri noted
> Polaris
> > > > being
> > > > > a
> > > > > > > > > platform
> > > > > > > > > >> should have the flexibility to support different
> > persistence
> > > > > > > backends
> > > > > > > > > per
> > > > > > > > > >> concern, and pointed to a concrete next step of
> separating
> > > the
> > > > > > JDBC
> > > > > > > > > >> bootstrap for metrics from the metastore bootstrap.
> Robert
> > > > > > proposed
> > > > > > > an
> > > > > > > > > >> additional UX extension: detect an unbootstrapped
> metrics
> > > > store
> > > > > on
> > > > > > > > first
> > > > > > > > > >> use and auto-bootstrap rather than requiring an explicit
> > > > manual
> > > > > > > > > bootstrap
> > > > > > > > > >> step.
> > > > > > > > > >> The meeting also confirmed that Polaris metrics can
> start
> > > > small
> > > > > > and
> > > > > > > > stay
> > > > > > > > > >> Iceberg-focused. Naming and persistence schema can lean
> > > > > > > > > Iceberg-specific.
> > > > > > > > > >> If a future expansion to generic-table metrics or
> > > operational
> > > > > > > metrics
> > > > > > > > > >> arrives, an abstraction layer can be built on top of the
> > > > Iceberg
> > > > > > > > metrics
> > > > > > > > > >> reporter at that point. Robert remains on the fence and
> > > would
> > > > > > prefer
> > > > > > > > > >> something more generic but did not block the direction;
> > > > Dmitri's
> > > > > > > read
> > > > > > > > > was
> > > > > > > > > >> that the proposed framework already has enough
> flexibility
> > > to
> > > > > > absorb
> > > > > > > > > >> future
> > > > > > > > > >> expansion.
> > > > > > > > > >>
> > > > > > > > > >> The Trade-offs and Proposed structure sections in the
> doc
> > > were
> > > > > not
> > > > > > > > > >> reviewed
> > > > > > > > > >> in detail. They remain open for either the next sync or
> PR
> > > > > review.
> > > > > > > > > >>
> > > > > > > > > >> *Cross-cutting alignment — battery-included plus
> > pluggable*
> > > > > > > > > >> A common philosophy emerged from the discussion. EJ
> > > summarized
> > > > > it
> > > > > > > as:
> > > > > > > > > >> Polaris should provide a battery-included UX for
> beginners
> > > and
> > > > > the
> > > > > > > > > >> flexibility for advanced users to swap the included
> > battery
> > > > for
> > > > > > > > > something
> > > > > > > > > >> more powerful or tailored to their use case. The SPI
> > design
> > > > > needs
> > > > > > to
> > > > > > > > > >> enable
> > > > > > > > > >> both.
> > > > > > > > > >>
> > > > > > > > > >> The inputs that shaped this framing:
> > > > > > > > > >> •   Anand described how his team uses the current
> metrics
> > > > > > > persistence
> > > > > > > > > >> (three metrics consumers in v1.4).
> > > > > > > > > >> •   Yufei raised Grafana and dashboard integrations as a
> > > > > > destination
> > > > > > > > use
> > > > > > > > > >> case beyond the default.
> > > > > > > > > >> •   Robert called out that the current design is more
> > > > > > JDBC-focused.
> > > > > > > > > >>
> > > > > > > > > >> Two concrete instances:
> > > > > > > > > >> •   Async metrics intake: Yufei's initial position was
> > that
> > > > > async
> > > > > > > > should
> > > > > > > > > >> largely live on the producer side and there is not much
> > > > Polaris
> > > > > > can
> > > > > > > > do.
> > > > > > > > > >> Robert suggested a Polaris-side default is doable via
> > > Vert.x.
> > > > > > Dmitri
> > > > > > > > > >> agreed
> > > > > > > > > >> the direction is worth exploring. The meeting converged
> > on a
> > > > > > > > > >> battery-included default (likely Vert.x-backed) with an
> > SPI
> > > > > shape
> > > > > > > that
> > > > > > > > > >> lets
> > > > > > > > > >> power users route to a more scalable backend (k8s-hosted
> > > > queue,
> > > > > > AWS
> > > > > > > > SQS,
> > > > > > > > > >> etc.).
> > > > > > > > > >> •   Pluggable destinations: combining Yufei's dashboard
> > use
> > > > case
> > > > > > > with
> > > > > > > > > >> Robert's JDBC-focused call-out, the meeting agreed the
> SPI
> > > > > should
> > > > > > be
> > > > > > > > > >> structured for multiple sinks so integrations become
> impl
> > > > > choices
> > > > > > > > rather
> > > > > > > > > >> than architectural changes.
> > > > > > > > > >>
> > > > > > > > > >> The battery-included default is most likely to use the
> > > > existing
> > > > > > > > > >> JDBC-backed
> > > > > > > > > >> approach.
> > > > > > > > > >>
> > > > > > > > > >> *Direction (idea-level alignment)*
> > > > > > > > > >> •   Single metrics_report table consolidating per-type
> > > > metrics,
> > > > > > > > > replacing
> > > > > > > > > >> scan_metrics_report and commit_metrics_report
> > > > > > > > > >> •   Iceberg-focused naming and schema for now, revisit
> if
> > > > > > > > generic-table
> > > > > > > > > or
> > > > > > > > > >> operational metrics arrive
> > > > > > > > > >> •   Metrics persistence as a separated SPI, not on
> > > > > BasePersistence
> > > > > > > > > >> •   Bootstrap path separated for metrics, independent of
> > > > > metastore
> > > > > > > > > >> bootstrap
> > > > > > > > > >> •   "Battery-included plus pluggable" as the SPI design
> > > > > philosophy
> > > > > > > > > >>
> > > > > > > > > >> *Open items*
> > > > > > > > > >> •   Schema details: metric_schema_version, payload
> format,
> > > IRC
> > > > > v2
> > > > > > > > > >> forward-compat shape
> > > > > > > > > >> •   SPI design details — full review either in the next
> > sync
> > > > or
> > > > > in
> > > > > > > the
> > > > > > > > > >> corresponding PR
> > > > > > > > > >> •   Schema refactor PR ownership
> > > > > > > > > >>
> > > > > > > > > >> *Action items*
> > > > > > > > > >> •   EJ to take a first stab at the SPI design and
> > > potentially
> > > > > > > partner
> > > > > > > > > with
> > > > > > > > > >> Anand to incorporate the lessons learned from the
> existing
> > > > > > reporter
> > > > > > > > and
> > > > > > > > > >> persistence work.
> > > > > > > > > >> •   Schema refactor PR ownership is not yet decided. If
> > > anyone
> > > > > is
> > > > > > > > > >> interested in driving it, reply on this thread.
> > > > > > > > > >> •   JB to schedule the next sync, tentatively in two
> > weeks.
> > > > > > > > > >>
> > > > > > > > > >> -ej
> > > > > > > > > >>
> > > > > > > > > >> On Mon, Apr 27, 2026 at 3:07 PM EJ Wang <
> > > > > > > > ej.wang.dev.interest@gmail.com
> > > > > > > > > >
> > > > > > > > > >> wrote:
> > > > > > > > > >>
> > > > > > > > > >> > Thanks Yufei for the +1.
> > > > > > > > > >> >
> > > > > > > > > >> > JB, could you help add a biweekly metrics architecture
> > > sync
> > > > to
> > > > > > the
> > > > > > > > > >> Polaris
> > > > > > > > > >> > community calendar? I'm thinking Thursdays at 9-10am
> PT,
> > > on
> > > > > the
> > > > > > > > > >> off-weeks
> > > > > > > > > >> > from the community meeting (starting May 7), 60
> minutes.
> > > > > > > > > >> >
> > > > > > > > > >> > Here's a rough agenda to work through over the first
> few
> > > > > > sessions,
> > > > > > > > > >> grouped
> > > > > > > > > >> > by priority:
> > > > > > > > > >> >
> > > > > > > > > >> > *First: foundational direction*
> > > > > > > > > >> >
> > > > > > > > > >> > 1.  MetricsPersistence: public SPI or internal
> > > > implementation
> > > > > > > > detail?
> > > > > > > > > >> >    •   Marked @Beta, javadoc calls it a "Service
> > Provider
> > > > > > > > Interface",
> > > > > > > > > >> but
> > > > > > > > > >> > only one consumer (JdbcBasePersistenceImpl), lives on
> > > > > > > > BasePersistence.
> > > > > > > > > >> If
> > > > > > > > > >> > demoted to a private helper inside a persisting
> reporter
> > > > impl,
> > > > > > > most
> > > > > > > > > >> > downstream design decisions become implementation
> > details
> > > > > rather
> > > > > > > > than
> > > > > > > > > >> > contract questions.
> > > > > > > > > >> >
> > > > > > > > > >> > 2.  Persistence schema redesign
> > > > > > > > > >> >    •   Current two-table layout (scan_metrics_report,
> > > > > > > > > >> > commit_metrics_report) with ~25 flattened columns
> each.
> > > > Every
> > > > > > new
> > > > > > > > > metric
> > > > > > > > > >> > type requires a new table, SPI method, record class,
> > > model,
> > > > > > > > converter,
> > > > > > > > > >> and
> > > > > > > > > >> > schema migration. Direction to explore: single table
> > with
> > > > > > > > metric_type
> > > > > > > > > >> enum,
> > > > > > > > > >> > schema_version, and JSON payload column.
> > > > > > > > > >> >
> > > > > > > > > >> > *Second: design details once direction is set*
> > > > > > > > > >> >
> > > > > > > > > >> > 3.  Partition key strategy
> > > > > > > > > >> >    •   Single-table design means scan metrics at scale
> > > will
> > > > > have
> > > > > > > > high
> > > > > > > > > >> > write concurrency per table. Schema needs to expose
> > enough
> > > > > > > structure
> > > > > > > > > for
> > > > > > > > > >> > backends to shard by entity or time range.
> > > > > > > > > >> >
> > > > > > > > > >> > 4.  Read/write path consistency
> > > > > > > > > >> >    •   Writes go through PolarisMetricsManager on
> > > > > > > MetaStoreManager.
> > > > > > > > > >> Reads
> > > > > > > > > >> > bypass MetaStoreManager and go straight to
> > > BasePersistence,
> > > > > > > > excluding
> > > > > > > > > >> > non-JDBC backends from the read API.
> > > > > > > > > >> >
> > > > > > > > > >> > *Third: cleanup and alignment*
> > > > > > > > > >> >
> > > > > > > > > >> > 5.  PolarisMetricsReporter naming
> > > > > > > > > >> >    •   Only handles IRC (ScanReport/CommitReport),
> > doesn't
> > > > > cover
> > > > > > > > > generic
> > > > > > > > > >> > tables or operational metrics. Name is broader than
> > scope.
> > > > > > > > > >> >
> > > > > > > > > >> > 6.  PolarisMetricsManager facade passthrough
> > > > > > > > > >> >    •   Entire default method is
> > > > > > > > > >> callCtx.getMetaStore().writeScanReport().
> > > > > > > > > >> > Zero logic, passes Level 1 straight through to Level
> 3.
> > > Same
> > > > > > > > > >> anti-pattern
> > > > > > > > > >> > as PolarisEventManager.
> > > > > > > > > >> >
> > > > > > > > > >> > 7.  Iceberg community alignment
> > > > > > > > > >> >    •   Payload-type extension needs discussion on
> > > > dev@iceberg.
> > > > > > > > > >> obelix74's
> > > > > > > > > >> > Feb thread got zero replies. Needs a committer voice.
> > > > > > > > > >> >
> > > > > > > > > >> > Lets confirm prioritization in the first session.
> > > > > > > > > >> >
> > > > > > > > > >> > -ej
> > > > > > > > > >> >
> > > > > > > > > >> > On Tue, Apr 21, 2026 at 3:18 PM Yufei Gu <
> > > > > flyrain000@gmail.com>
> > > > > > > > > wrote:
> > > > > > > > > >> >
> > > > > > > > > >> >> Thanks everyone for continuing to drive this
> forward. I
> > > > agree
> > > > > > > that
> > > > > > > > > the
> > > > > > > > > >> >> problem is getting complex enough that a more
> > structured
> > > > > > > discussion
> > > > > > > > > >> would
> > > > > > > > > >> >> help.
> > > > > > > > > >> >>
> > > > > > > > > >> >> +1 on setting up a biweekly sync for the metrics
> > > > > architecture.
> > > > > > > I’m
> > > > > > > > > >> happy
> > > > > > > > > >> >> to
> > > > > > > > > >> >> join.
> > > > > > > > > >> >>
> > > > > > > > > >> >> Yufei
> > > > > > > > > >> >>
> > > > > > > > > >> >>
> > > > > > > > > >> >> On Tue, Apr 21, 2026 at 2:34 PM EJ Wang <
> > > > > > > > > >> ej.wang.dev.interest@gmail.com>
> > > > > > > > > >> >> wrote:
> > > > > > > > > >> >>
> > > > > > > > > >> >> > Also, I've been looking more closely at the
> > > *persistence
> > > > > > schema
> > > > > > > > in
> > > > > > > > > >> the
> > > > > > > > > >> >> > current metrics work*, and I think there's a
> > structural
> > > > > > > rigidity
> > > > > > > > > >> problem
> > > > > > > > > >> >> > worth raising before the shape gets locked in.
> > > > > > > > > >> >> >
> > > > > > > > > >> >> > Right now we have two separate tables
> > > > (scan_metrics_report
> > > > > > and
> > > > > > > > > >> >> > commit_metrics_report), each with ~25 flattened
> > columns
> > > > > that
> > > > > > > > > directly
> > > > > > > > > >> >> > mirror the Iceberg report fields. The SPI follows
> the
> > > > same
> > > > > > > split:
> > > > > > > > > >> >> > writeScanReport and writeCommitReport as separate
> > > > methods,
> > > > > > with
> > > > > > > > > >> per-type
> > > > > > > > > >> >> > record classes, converters, and model objects. *The
> > > > > practical
> > > > > > > > cost:
> > > > > > > > > >> >> > adding a new metric type (operational metrics, for
> > > > example)
> > > > > > > > > requires
> > > > > > > > > >> a
> > > > > > > > > >> >> new
> > > > > > > > > >> >> > table, a new SPI method, a new record class, a new
> > > model
> > > > > > > class, a
> > > > > > > > > new
> > > > > > > > > >> >> > converter branch, and a schema migration*. That's a
> > lot
> > > > of
> > > > > > > > surface
> > > > > > > > > >> area
> > > > > > > > > >> >> > for what should be "one more kind of metric."
> > > > > > > > > >> >> >
> > > > > > > > > >> >> > *My bias* would be toward a single metrics table
> with
> > > *a
> > > > > > typed
> > > > > > > > JSON
> > > > > > > > > >> >> > payload*. Something like: metric_type (enum),
> > > entity_id,
> > > > > > > > > >> >> > table_identifier, snapshot_id (nullable),
> > received_ts,
> > > > > > > > > >> schema_version,
> > > > > > > > > >> >> and
> > > > > > > > > >> >> > a payload column for the metric-specific data. The
> > > > > > metric_type
> > > > > > > +
> > > > > > > > > >> >> > schema_version pair gives us a forward-compatible
> > > > contract
> > > > > > for
> > > > > > > > the
> > > > > > > > > >> >> payload
> > > > > > > > > >> >> > shape. Adding a new metric type becomes an enum
> value
> > > > and a
> > > > > > > > payload
> > > > > > > > > >> >> schema,
> > > > > > > > > >> >> > not a schema migration. One thing I think we need
> to
> > be
> > > > > > > > deliberate
> > > > > > > > > >> >> about is
> > > > > > > > > >> >> > the partition key design. If all metric types land
> in
> > > one
> > > > > > > table,
> > > > > > > > > scan
> > > > > > > > > >> >> > metrics at scale (high concurrency, high frequency
> > > across
> > > > > > many
> > > > > > > > > >> tables)
> > > > > > > > > >> >> > could easily create hot partitions. We'd want the
> > > > > persistence
> > > > > > > > layer
> > > > > > > > > >> to
> > > > > > > > > >> >> be
> > > > > > > > > >> >> > able to shard by entity or time range, and that
> means
> > > the
> > > > > > > logical
> > > > > > > > > >> schema
> > > > > > > > > >> >> > needs to expose enough structure for backends to
> > > > partition
> > > > > > on.
> > > > > > > I
> > > > > > > > > >> don't
> > > > > > > > > >> >> > think the current flattened layout gives us that.
> > > > > > > > > >> >> >
> > > > > > > > > >> >> > This is getting complex enough that I don't think
> > > ad-hoc
> > > > > > PR/ML
> > > > > > > > > >> threads
> > > > > > > > > >> >> > will converge well. *Would people be open to a
> > biweekly
> > > > > sync
> > > > > > > for
> > > > > > > > > >> metrics
> > > > > > > > > >> >> > architecture?* I think 30 minutes every two weeks
> > with
> > > > > > > interested
> > > > > > > > > >> >> parties
> > > > > > > > > >> >> > would be enough to work through the schema, SPI
> > shape,
> > > > and
> > > > > > read
> > > > > > > > API
> > > > > > > > > >> >> design
> > > > > > > > > >> >> > together. Happy to help set that up.
> > > > > > > > > >> >> >
> > > > > > > > > >> >> > -ej
> > > > > > > > > >> >> >
> > > > > > > > > >> >> > On Mon, Apr 20, 2026 at 2:19 PM EJ Wang <
> > > > > > > > > >> ej.wang.dev.interest@gmail.com
> > > > > > > > > >> >> >
> > > > > > > > > >> >> > wrote:
> > > > > > > > > >> >> >
> > > > > > > > > >> >> >> Reviewed #4115, left a comment on the code
> > > organization
> > > > > > side.
> > > > > > > > > >> >> >>
> > > > > > > > > >> >> >> One thing stood out: the metrics write path enters
> > > > through
> > > > > > > > > >> >> >> PolarisMetricsManager on MetaStoreManager, but the
> > new
> > > > > read
> > > > > > > path
> > > > > > > > > >> >> bypasses
> > > > > > > > > >> >> >> MetaStoreManager entirely and goes straight to
> > > > > > BasePersistence
> > > > > > > > via
> > > > > > > > > >> >> >> callContext.getMetaStore(). That means the read
> API
> > > only
> > > > > > works
> > > > > > > > for
> > > > > > > > > >> >> backends
> > > > > > > > > >> >> >> that implement BasePersistence. NoSQL and remote
> > > > backends
> > > > > > > can't
> > > > > > > > > >> >> participate.
> > > > > > > > > >> >> >>
> > > > > > > > > >> >> >> Stepping back, I think the metrics subsystem is
> > > growing
> > > > > into
> > > > > > > > > >> something
> > > > > > > > > >> >> >> real (write + read + REST API + AuthZ +
> pagination)
> > > *but
> > > > > the
> > > > > > > > > >> >> persistence
> > > > > > > > > >> >> >> side is split across two layers in a way that's
> hard
> > > to
> > > > > > > > extend*. I
> > > > > > > > > >> put
> > > > > > > > > >> >> >> together two diagrams to show what I mean (my best
> > > > > effort).
> > > > > > > > > >> >> >>
> > > > > > > > > >> >> >> *Current state* (Diagram 1): three interfaces at
> > three
> > > > > > > different
> > > > > > > > > >> >> levels.
> > > > > > > > > >> >> >> The engine-facing SPI (PolarisMetricsReporter) is
> > > clean.
> > > > > But
> > > > > > > > > >> >> >> PolarisMetricsManager on MetaStoreManager is a
> > > > passthrough
> > > > > > to
> > > > > > > > > >> >> >> MetricsPersistence on BasePersistence. The @Beta
> > > > > annotation
> > > > > > > and
> > > > > > > > > SPI
> > > > > > > > > >> >> javadoc
> > > > > > > > > >> >> >> are on the BasePersistence layer, while the actual
> > > > > extension
> > > > > > > > > points
> > > > > > > > > >> >> >> (PolarisMetricsReporter, PolarisMetricsManager)
> > carry
> > > no
> > > > > > > > stability
> > > > > > > > > >> >> >> annotation. The write path goes through the
> > > > > MetaStoreManager
> > > > > > > > > layer,
> > > > > > > > > >> the
> > > > > > > > > >> >> >> read path doesn't.
> > > > > > > > > >> >> >>
> > > > > > > > > >> >> >> *What I envision* (Diagram 2): two SPIs at two
> > levels.
> > > > > > > > > >> >> >> PolarisMetricsReporter stays as the engine-facing
> > SPI.
> > > > > > > > > >> >> >> PolarisMetricsManager becomes the backend-facing
> SPI
> > > > with
> > > > > > both
> > > > > > > > > write
> > > > > > > > > >> >> and
> > > > > > > > > >> >> >> read methods at the MetaStoreManager level, where
> > any
> > > > > > backend
> > > > > > > > > (JDBC,
> > > > > > > > > >> >> NoSQL,
> > > > > > > > > >> >> >> remote) can implement them. MetricsPersistence on
> > > > > > > > BasePersistence
> > > > > > > > > >> goes
> > > > > > > > > >> >> >> away. Where metrics actually land is an
> > implementation
> > > > > > detail,
> > > > > > > > > not a
> > > > > > > > > >> >> core
> > > > > > > > > >> >> >> interface.
> > > > > > > > > >> >> >>
> > > > > > > > > >> >> >> *Minor naming thing*: PolarisMetricsReporter is
> > > broader
> > > > > than
> > > > > > > > what
> > > > > > > > > it
> > > > > > > > > >> >> >> actually handles. It only accepts Iceberg REST
> > Catalog
> > > > > > metrics
> > > > > > > > > >> >> (ScanReport,
> > > > > > > > > >> >> >> CommitReport via MetricsReport). Generic table
> > metrics
> > > > or
> > > > > > > > > >> operational
> > > > > > > > > >> >> >> metrics aren't in scope. Not blocking, but worth
> > > noting
> > > > if
> > > > > > the
> > > > > > > > > >> metrics
> > > > > > > > > >> >> >> surface expands.
> > > > > > > > > >> >> >>
> > > > > > > > > >> >> >> *Rough sketch of how to get there*:
> > > > > > > > > >> >> >>  1.  Add read methods to PolarisMetricsManager
> > > > > > > (listScanReports,
> > > > > > > > > >> >> >> listCommitReports) with default no-op, same as the
> > > > > existing
> > > > > > > > write
> > > > > > > > > >> >> methods.
> > > > > > > > > >> >> >> (Probably make PolarisMetricsManager more explicit
> > on
> > > > > being
> > > > > > > > > Iceberg
> > > > > > > > > >> >> >> specific like package name or class name etc.)
> > > > > > > > > >> >> >>  2.  Wire MetricsReportsService through
> > > MetaStoreManager
> > > > > > > instead
> > > > > > > > > of
> > > > > > > > > >> >> >> callContext.getMetaStore().
> > > > > > > > > >> >> >>  3.  Extract metrics persistence from
> > > > > > JdbcBasePersistenceImpl
> > > > > > > > into
> > > > > > > > > >> its
> > > > > > > > > >> >> >> own class. That file carries ~7 responsibilities,
> > > > metrics
> > > > > > > being
> > > > > > > > > one
> > > > > > > > > >> of
> > > > > > > > > >> >> them.
> > > > > > > > > >> >> >>  4.  Remove MetricsPersistence from
> BasePersistence.
> > > > > > > > > >> >> >>
> > > > > > > > > >> >> >> *None of this needs to happen in #4115. But if the
> > > > > direction
> > > > > > > > makes
> > > > > > > > > >> >> sense,
> > > > > > > > > >> >> >> it would be good to align before the metrics
> surface
> > > > grows
> > > > > > > > > further.
> > > > > > > > > >> >> Curious
> > > > > > > > > >> >> >> what others think.*
> > > > > > > > > >> >> >>
> > > > > > > > > >> >> >> *My mental model note*: Level 1 MetaStoreManager;
> > > level
> > > > 2
> > > > > > > > > >> transactional
> > > > > > > > > >> >> >> persistence; level 3 base persistence
> > > > > > > > > >> >> >>
> > > > > > > > > >> >> >> Diagram 1
> > > > > > > > > >> >> >> <
> > > > > > > > > >> >>
> > > > > > > > > >>
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://www.plantuml.com/plantuml/uml/bLHDR-Cs4BthLmpIYupw0zbkKQ1r3M-S7Bp8xhhM7WCOb3IM65EaGD9EX2RzxHrHb4CxRelwa4YSDu_lpOVcnZ9jzvM8BBS2uGjQpJC3dtHMSekPtMk44IpsMgEqa5XcCOhCZikQQLP1pR8TAp2n3ILhmZDP20m0fcIvUkAoW2qJXd9z1bpToO9BX3WXu0ucy5rpgGPNm0nW5_epUWtm2Ue3pn3kMOFQmKntGZW0BYtgBSi8k5A2QMwybJNMIbFiGSR9QZc4nUqIvikStF0jHprua5C-amge42aNt3R0f5JaaoivdV2Pkqbx4hee4ymOkBh5BTiB-_uIeGeo8zL8rPsPl4DktdEiK1jkB1NdZCRbrSTecDe_mlHbF0wvBmCkaOH5_S8a_TTTKI6-nmCAkEw4LpxsZ-LbYLKQFKMNOgf_wuM7_bV9gOer5SYMMksBSWXFcbi49KNZXNLicwfe3TETC7gPdPqI7uBcHMb1RSzYq34c6PDUM9mn8HRsUTZEiDBve3NjVZumBj0U7SS37mGO7vcwtiK-_pU7U7L_f-digo9YbhSwIfMRwIITKGXbxdIUTCGF1SeCJxloKsU-3k9ddRbX1eDq1q_fx1JbBGT0glVyXimDuP4TQ5qpCAmnGEj2s_6n5mtn1z-97-63itFQZLPO1Ev2tu_WF7Ju-VPc0Skg5bYXxBhkY1xpD7EM_7fyflSpIsqMgVth5xhVr4eQxWQ8enaSAJQSG16yFSDuJ798rrcXr_3n-lfdk7icQjEBmFujL7AodiP_Y4Z7-YxvtZNs4zMgpNTl6tF8sglyPsmqchrjvQ-m-aP94r-TwCA2Ka8upPJZwtvSpoYCXkYMZU2NXvRMBfq9P3i3Le4VAZUAlUZ_oPKsxPgY0Q_BSKLkyr9bhQhQrJjo_x3TPlIB0DPjnMfcIoYP0QaYw1a0fTKDr8fB6ntNuvmoL1ZGkXa69Njh43zf9GiGxHQrA_jDYWRSzF5--WmTVrN97_Sm8LbLUy_lGBmLanJjFkDlGkRqjA_4tm00
> > > > > > > > > >> >> >
> > > > > > > > > >> >> >> :
> > > > > > > > > >> >> >>
> > > > > > > > > >> >> >> [image: image.png]
> > > > > > > > > >> >> >>
> > > > > > > > > >> >> >> Diagram 2
> > > > > > > > > >> >> >> <
> > > > > > > > > >> >>
> > > > > > > > > >>
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://www.plantuml.com/plantuml/uml/VLLDR-8m4BtdLupO2sWBLVU8AaGB7AXAbssGzb896SS9RXqxjKqBwkv_tt7iV43fdaZYDpFlpRmnOsE9jhjSH9PRmM31hERKm8scMsuPjJlDe0yheZDc8RR4iYWoBrmMH9CS2a9VICPYUy1OZN0YCy5Q0BCbYNhdCeEK28En8G8wCvbnoQ0R8_05Bc6bkLIz3X03p1zzH7zR-9ZfDquPt9C3qoNCX2yV4G2NbkcKu5jdgGJHt0GbZwnG6i-UP3TUpk5gM6Ldqke350eZUqzoCft3U9xWHvxoa5-7K4nF1J46EbEMafsmdrCBbQ44gVggy18IZrn_ph5asd1ZiIKdQSgueZvjXrQFSFrdC3YN-nXmBacxbGiYyLVxLaBtdhqn0LSzdBDhqQtQoOJeGyad3z0lUqnYgpGB6Ns8oVyta00Dy_WnX0tIOZ8v6SYxHll1TrH6aejAik-mh-AphVFCwSUQqFypElag5QRGFDjQKEd96K1P8QP41c9TzA_IIQyvdAWyv_RSiS3skb0_EzDDkK2v5xWF6MiGFlvhpFLcD2Dq2pml14gaF67eQkmd8gulDoC4kSOu6KVpkvlUJg1RTbWISU40RdBUUS_9XfRZ2dwxm_SW8LYFISgm_MnlDQ6M9P1gbKEc4X-2pH_FvJCkCqm9pbVjD6LrwdLeOrDWfOaqc8Wh9BE85oNKxkNQ6o4yGRy_Eae0G_G8tZv81d3bHDB23WOdisohVr3nh_j6lbSjbNaLRTc8UgtPbAU1J_tygOfZX9DWEJeHDvYx-qmSi5FgNLPZwHrHcUsncGQ5-skhUclpE5fo4ounpFauYrUbkU6ccfnxMvitwag4IyerhTxj8In_Oj1bDO4pQru674loYrGlULHLEGCjwJJ8gDoVZR8MxO4BT3IzRvIcAQKezC6xpziGnTyImrfEGyJI_OcKfgtxIvnTqFEMS17L9Z-jsARN5FmTheP7HtSdtOMT0B4GY2FYHXxgQmMtj2bRqiLFGapiVe1_QVKDrkqXcm83aFEXnMYCZ-xlyHy
> > > > > > > > > >> >> >
> > > > > > > > > >> >> >> :
> > > > > > > > > >> >> >> [image: image.png]
> > > > > > > > > >> >> >>
> > > > > > > > > >> >> >>  -ej
> > > > > > > > > >> >> >>
> > > > > > > > > >> >> >> On Wed, Apr 15, 2026 at 8:22 AM Dmitri
> Bourlatchkov
> > <
> > > > > > > > > >> dimas@apache.org>
> > > > > > > > > >> >> >> wrote:
> > > > > > > > > >> >> >>
> > > > > > > > > >> >> >>> Hi All,
> > > > > > > > > >> >> >>>
> > > > > > > > > >> >> >>> Heads up: The current state of PR [4115] looks
> > pretty
> > > > > solid
> > > > > > > to
> > > > > > > > > me.
> > > > > > > > > >> I
> > > > > > > > > >> >> >>> believe this PR is approaching a mergeable
> > condition.
> > > > > > > > > >> >> >>>
> > > > > > > > > >> >> >>> Please post your reviews if you have any
> comments.
> > > > > > > > > >> >> >>>
> > > > > > > > > >> >> >>> [4115]
> https://github.com/apache/polaris/pull/4115
> > > > > > > > > >> >> >>>
> > > > > > > > > >> >> >>> Thanks,
> > > > > > > > > >> >> >>> Dmitri.
> > > > > > > > > >> >> >>>
> > > > > > > > > >> >> >>> On Tue, Mar 3, 2026 at 3:29 PM Anand Kumar
> Sankaran
> > > via
> > > > > > dev <
> > > > > > > > > >> >> >>> dev@polaris.apache.org> wrote:
> > > > > > > > > >> >> >>>
> > > > > > > > > >> >> >>> > Hi Yufei and Dmitri,
> > > > > > > > > >> >> >>> >
> > > > > > > > > >> >> >>> > Here is a proposal for the REST endpoints for
> > > metrics
> > > > > and
> > > > > > > > > events.
> > > > > > > > > >> >> >>> >
> > > > > > > > > >> >> >>> >
> > > https://github.com/apache/polaris/pull/3924/changes
> > > > > > > > > >> >> >>> >
> > > > > > > > > >> >> >>> > I did not see any precursors for raising a PR
> for
> > > > > > > proposals,
> > > > > > > > so
> > > > > > > > > >> >> trying
> > > > > > > > > >> >> >>> > this.  Please let me know what you think.
> > > > > > > > > >> >> >>> >
> > > > > > > > > >> >> >>> > -
> > > > > > > > > >> >> >>> > Anand
> > > > > > > > > >> >> >>> >
> > > > > > > > > >> >> >>> > From: Anand Kumar Sankaran <
> > > > anand.sankaran@workday.com
> > > > > >
> > > > > > > > > >> >> >>> > Date: Monday, March 2, 2026 at 10:25 AM
> > > > > > > > > >> >> >>> > To: dev@polaris.apache.org <
> > dev@polaris.apache.org
> > > >
> > > > > > > > > >> >> >>> > Subject: Re: Polaris Telemetry and Audit Trail
> > > > > > > > > >> >> >>> >
> > > > > > > > > >> >> >>> > About the REST API, based on my use cases:
> > > > > > > > > >> >> >>> >
> > > > > > > > > >> >> >>> >
> > > > > > > > > >> >> >>> >   1.
> > > > > > > > > >> >> >>> > I want to be able to query commit metrics to
> > track
> > > > > files
> > > > > > > > added
> > > > > > > > > /
> > > > > > > > > >> >> >>> removed
> > > > > > > > > >> >> >>> > per commit, along with record counts. The
> > ingestion
> > > > > > > pipeline
> > > > > > > > > that
> > > > > > > > > >> >> >>> writes
> > > > > > > > > >> >> >>> > this data is owned by us and we are guaranteed
> to
> > > > write
> > > > > > > this
> > > > > > > > > >> >> >>> information
> > > > > > > > > >> >> >>> > for each write.
> > > > > > > > > >> >> >>> >   2.
> > > > > > > > > >> >> >>> > I want to be able to query scan metrics for
> > read. I
> > > > > > > > understand
> > > > > > > > > >> >> clients
> > > > > > > > > >> >> >>> do
> > > > > > > > > >> >> >>> > not fulfill this requirement.
> > > > > > > > > >> >> >>> >   3.
> > > > > > > > > >> >> >>> > I want to be able to query the events table
> > (events
> > > > are
> > > > > > > > > >> persisted) -
> > > > > > > > > >> >> >>> this
> > > > > > > > > >> >> >>> > may supersede #2, I am not sure yet.
> > > > > > > > > >> >> >>> >
> > > > > > > > > >> >> >>> > All this information is in the JDBC based
> > > persistence
> > > > > > model
> > > > > > > > and
> > > > > > > > > >> is
> > > > > > > > > >> >> >>> > persisted in the metastore. I currently don’t
> > have
> > > a
> > > > > need
> > > > > > > to
> > > > > > > > > >> query
> > > > > > > > > >> >> >>> > prometheus or open telemetry. I do publish some
> > > > events
> > > > > to
> > > > > > > > > >> Prometheus
> > > > > > > > > >> >> >>> and
> > > > > > > > > >> >> >>> > they are forwarded to our dashboards elsewhere.
> > > > > > > > > >> >> >>> >
> > > > > > > > > >> >> >>> > About the CLI utilities, I meant the admin user
> > > > > > utilities.
> > > > > > > In
> > > > > > > > > >> one of
> > > > > > > > > >> >> >>> the
> > > > > > > > > >> >> >>> > earliest drafts of my proposal, Prashant
> > mentioned
> > > > that
> > > > > > the
> > > > > > > > > >> metrics
> > > > > > > > > >> >> >>> tables
> > > > > > > > > >> >> >>> > can grow indefinitely and that a similar
> problem
> > > > exists
> > > > > > > with
> > > > > > > > > the
> > > > > > > > > >> >> events
> > > > > > > > > >> >> >>> > table as well. We discussed that cleaning up of
> > old
> > > > > > records
> > > > > > > > > from
> > > > > > > > > >> >> both
> > > > > > > > > >> >> >>> > metrics tables and events tables can be done
> via
> > a
> > > > CLI
> > > > > > > > utility.
> > > > > > > > > >> >> >>> >
> > > > > > > > > >> >> >>> > I see that Yufei has covered the discussion
> about
> > > > > > > > datasources.
> > > > > > > > > >> >> >>> >
> > > > > > > > > >> >> >>> > -
> > > > > > > > > >> >> >>> > Anand
> > > > > > > > > >> >> >>> >
> > > > > > > > > >> >> >>> >
> > > > > > > > > >> >> >>> >
> > > > > > > > > >> >> >>> > From: Yufei Gu <fl...@gmail.com>
> > > > > > > > > >> >> >>> > Date: Friday, February 27, 2026 at 9:54 PM
> > > > > > > > > >> >> >>> > To: dev@polaris.apache.org <
> > dev@polaris.apache.org
> > > >
> > > > > > > > > >> >> >>> > Subject: Re: Polaris Telemetry and Audit Trail
> > > > > > > > > >> >> >>> >
> > > > > > > > > >> >> >>> > This Message Is From an External Sender
> > > > > > > > > >> >> >>> > This message came from outside your
> organization.
> > > > > > > > > >> >> >>> > Report Suspicious<
> > > > > > > > > >> >> >>> >
> > > > > > > > > >> >> >>>
> > > > > > > > > >> >>
> > > > > > > > > >>
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://us-phishalarm-ewt.proofpoint.com/EWT/v1/Iz9xO38YGHZK!YhNDZABkHi1B699ote2uMwpOZw8i0QMCGO2Szc-HshuABGhGvwPJcymE6G2oUUxtS8xDkSrtGTPm_I3QnVDHoLMk50m9v8z_nZKTkd-bnVUbreF1u0WnfV_X5eYevZl_$
> > > > > > > > > >> >> >>> > >
> > > > > > > > > >> >> >>> >
> > > > > > > > > >> >> >>> >
> > > > > > > > > >> >> >>> > As I mentioned in
> > > > > > > > > >> >> >>> >
> > > > > > > > > >> >> >>>
> > > > > > > > > >> >>
> > > > > > > > > >>
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://urldefense.com/v3/__https://github.com/apache/polaris/issues/3890__;!!Iz9xO38YGHZK!5EuyFFkk3vhRWVIRvQAWBSQfpJkTMA9HxugzDwXmN0LPPqhEFxYkFRGVhtb8AqUwXtDh2OplcMnbMDHKOxrvDU0$
> > > > > > > > > >> >> >>> ,
> > > > > > > > > >> >> >>> > supporting
> > > > > > > > > >> >> >>> > multiple data sources is not a trivial change.
> I
> > > > would
> > > > > > > > strongly
> > > > > > > > > >> >> >>> recommend
> > > > > > > > > >> >> >>> > starting with a design document to carefully
> > > evaluate
> > > > > the
> > > > > > > > > >> >> architectural
> > > > > > > > > >> >> >>> > implications and long term impact.
> > > > > > > > > >> >> >>> >
> > > > > > > > > >> >> >>> > A REST endpoint to query metrics seems
> reasonable
> > > > given
> > > > > > the
> > > > > > > > > >> current
> > > > > > > > > >> >> >>> JDBC
> > > > > > > > > >> >> >>> > based persistence model. That said, we may also
> > > > > consider
> > > > > > > > > >> alternative
> > > > > > > > > >> >> >>> > storage models. For example, if we later adopt
> a
> > > time
> > > > > > > series
> > > > > > > > > >> system
> > > > > > > > > >> >> >>> such as
> > > > > > > > > >> >> >>> > Prometheus to store metrics, the query model
> and
> > > > access
> > > > > > > > > patterns
> > > > > > > > > >> >> would
> > > > > > > > > >> >> >>> be
> > > > > > > > > >> >> >>> > fundamentally different. Designing the REST API
> > > > without
> > > > > > > > > >> considering
> > > > > > > > > >> >> >>> these
> > > > > > > > > >> >> >>> > potential evolutions may limit flexibility. I'd
> > > > suggest
> > > > > > to
> > > > > > > > > start
> > > > > > > > > >> >> with
> > > > > > > > > >> >> >>> the
> > > > > > > > > >> >> >>> > use case.
> > > > > > > > > >> >> >>> >
> > > > > > > > > >> >> >>> > Yufei
> > > > > > > > > >> >> >>> >
> > > > > > > > > >> >> >>> >
> > > > > > > > > >> >> >>> > On Fri, Feb 27, 2026 at 3:42 PM Dmitri
> > > Bourlatchkov <
> > > > > > > > > >> >> dimas@apache.org>
> > > > > > > > > >> >> >>> > wrote:
> > > > > > > > > >> >> >>> >
> > > > > > > > > >> >> >>> > > Hi Anand,
> > > > > > > > > >> >> >>> > >
> > > > > > > > > >> >> >>> > > Sharing my view... subject to discussion:
> > > > > > > > > >> >> >>> > >
> > > > > > > > > >> >> >>> > > 1. Adding non-IRC REST API to Polaris is
> > > perfectly
> > > > > > fine.
> > > > > > > > > >> >> >>> > >
> > > > > > > > > >> >> >>> > > Figuring out specific endpoint URIs and
> > payloads
> > > > > might
> > > > > > > > > require
> > > > > > > > > >> a
> > > > > > > > > >> >> few
> > > > > > > > > >> >> >>> > > roundtrips, so opening a separate thread for
> > that
> > > > > might
> > > > > > > be
> > > > > > > > > >> best.
> > > > > > > > > >> >> >>> > > Contributors commonly create Google Docs for
> > new
> > > > API
> > > > > > > > > proposals
> > > > > > > > > >> too
> > > > > > > > > >> >> >>> (they
> > > > > > > > > >> >> >>> > > fairly easy to update as the email discussion
> > > > > > > progresses).
> > > > > > > > > >> >> >>> > >
> > > > > > > > > >> >> >>> > > There was a suggestion to try Markdown (with
> > PRs)
> > > > for
> > > > > > > > > proposals
> > > > > > > > > >> >> [1]
> > > > > > > > > >> >> >>> ...
> > > > > > > > > >> >> >>> > > feel free to give it a try if you are
> > comfortable
> > > > > with
> > > > > > > > that.
> > > > > > > > > >> >> >>> > >
> > > > > > > > > >> >> >>> > > 2. Could you clarify whether you mean end
> user
> > > > > > utilities
> > > > > > > or
> > > > > > > > > >> admin
> > > > > > > > > >> >> >>> user
> > > > > > > > > >> >> >>> > > utilities? In the latter case those might be
> > more
> > > > > > > suitable
> > > > > > > > > for
> > > > > > > > > >> the
> > > > > > > > > >> >> >>> Admin
> > > > > > > > > >> >> >>> > > CLI (java) not the Python CLI, IMHO.
> > > > > > > > > >> >> >>> > >
> > > > > > > > > >> >> >>> > > Why would these utilities be common with
> > events?
> > > > > IMHO,
> > > > > > > > event
> > > > > > > > > >> use
> > > > > > > > > >> >> >>> cases
> > > > > > > > > >> >> >>> > are
> > > > > > > > > >> >> >>> > > distinct from scan/commit metrics.
> > > > > > > > > >> >> >>> > >
> > > > > > > > > >> >> >>> > > 3. I'd prefer separating metrics persistence
> > from
> > > > > > > MetaStore
> > > > > > > > > >> >> >>> persistence
> > > > > > > > > >> >> >>> > at
> > > > > > > > > >> >> >>> > > the code level, so that they could be mixed
> and
> > > > > matched
> > > > > > > > > >> >> >>> independently.
> > > > > > > > > >> >> >>> > The
> > > > > > > > > >> >> >>> > > separate datasource question will become a
> > > > non-issue
> > > > > > with
> > > > > > > > > that
> > > > > > > > > >> >> >>> approach,
> > > > > > > > > >> >> >>> > I
> > > > > > > > > >> >> >>> > > guess.
> > > > > > > > > >> >> >>> > >
> > > > > > > > > >> >> >>> > > The rationale for separating scan metrics and
> > > > > metastore
> > > > > > > > > >> >> persistence
> > > > > > > > > >> >> >>> is
> > > > > > > > > >> >> >>> > that
> > > > > > > > > >> >> >>> > > "cascading deletes" between them are hardly
> > ever
> > > > > > > required.
> > > > > > > > > >> >> >>> Furthermore,
> > > > > > > > > >> >> >>> > the
> > > > > > > > > >> >> >>> > > data and query patterns are very different so
> > > > > different
> > > > > > > > > >> >> technologies
> > > > > > > > > >> >> >>> > might
> > > > > > > > > >> >> >>> > > be beneficial in each case.
> > > > > > > > > >> >> >>> > >
> > > > > > > > > >> >> >>> > > [1]
> > > > > > > > > >> >> >>> >
> > > > > > > > > >> >> >>>
> > > > > > > > > >> >>
> > > > > > > > > >>
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://urldefense.com/v3/__https://lists.apache.org/thread/yto2wp982t43h1mqjwnslswhws5z47cy__;!!Iz9xO38YGHZK!5EuyFFkk3vhRWVIRvQAWBSQfpJkTMA9HxugzDwXmN0LPPqhEFxYkFRGVhtb8AqUwXtDh2OplcMnbMDHKxYDakNU$
> > > > > > > > > >> >> >>> > >
> > > > > > > > > >> >> >>> > > Cheers,
> > > > > > > > > >> >> >>> > > Dmitri.
> > > > > > > > > >> >> >>> > >
> > > > > > > > > >> >> >>> > > On Fri, Feb 27, 2026 at 6:19 PM Anand Kumar
> > > > Sankaran
> > > > > > via
> > > > > > > > dev
> > > > > > > > > <
> > > > > > > > > >> >> >>> > > dev@polaris.apache.org> wrote:
> > > > > > > > > >> >> >>> > >
> > > > > > > > > >> >> >>> > > > Thanks all. This PR is merged now.
> > > > > > > > > >> >> >>> > > >
> > > > > > > > > >> >> >>> > > > Here are the follow-up features / work
> > needed.
> > > > > These
> > > > > > > > were
> > > > > > > > > >> all
> > > > > > > > > >> >> >>> part of
> > > > > > > > > >> >> >>> > > the
> > > > > > > > > >> >> >>> > > > merged PR at some point in time and were
> > > removed
> > > > to
> > > > > > > > reduce
> > > > > > > > > >> >> scope.
> > > > > > > > > >> >> >>> > > >
> > > > > > > > > >> >> >>> > > > Please let me know what you think.
> > > > > > > > > >> >> >>> > > >
> > > > > > > > > >> >> >>> > > >
> > > > > > > > > >> >> >>> > > >   1.  A REST API to paginate through table
> > > > metrics.
> > > > > > > This
> > > > > > > > > >> will be
> > > > > > > > > >> >> >>> > non-IRC
> > > > > > > > > >> >> >>> > > > standard addition.
> > > > > > > > > >> >> >>> > > >   2.  Utilities for managing old records,
> > > should
> > > > be
> > > > > > > > common
> > > > > > > > > >> with
> > > > > > > > > >> >> >>> events.
> > > > > > > > > >> >> >>> > > > There was some discussion that it belongs
> to
> > > the
> > > > > CLI.
> > > > > > > > > >> >> >>> > > >   3.  Separate datasource (metrics, events,
> > > even
> > > > > > other
> > > > > > > > > >> tables?).
> > > > > > > > > >> >> >>> > > >
> > > > > > > > > >> >> >>> > > >
> > > > > > > > > >> >> >>> > > > Anything else?
> > > > > > > > > >> >> >>> > > >
> > > > > > > > > >> >> >>> > > > -
> > > > > > > > > >> >> >>> > > > Anand
> > > > > > > > > >> >> >>> > > >
> > > > > > > > > >> >> >>> > > >
> > > > > > > > > >> >> >>> > >
> > > > > > > > > >> >> >>> >
> > > > > > > > > >> >> >>> >
> > > > > > > > > >> >> >>>
> > > > > > > > > >> >> >>
> > > > > > > > > >> >>
> > > > > > > > > >> >
> > > > > > > > > >>
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>

Re: Proposal for REST endpoints for table metrics and events

[Yufei Gu <fl...@gmail.com>]

Thanks for chiming in, EJ. Agreed that the default battery should stay
small. I'm leaning toward not including metrics persistence in the default
battery.

Yufei


On Thu, Jun 11, 2026 at 8:30 PM EJ Wang <ej...@gmail.com>
wrote:

> Hi Yufei,
>
> Thanks for connecting this back to the REST endpoint proposal. I replied
> with the fuller version on the event-forwarding thread, but wanted to add
> the shorter version here too.
>
> I agree that metrics reporting/emitting is the right conceptual boundary,
> and I also think the event/listener path is a good implementation direction
> to explore. The distinction I want to keep clear is default battery vs
> extension implementation.
>
> For the REST/API side, I would keep the semantics narrow: Polaris resolves
> and authorizes the table, accepts the Iceberg scan/commit metrics report
> into the configured reporting/emitting path, and returns 204. That 204
> should mean Polaris accepted the report into the ingestion path; it should
> not imply durable storage.
>
> Durable storage, event forwarding, external telemetry routing, filtering,
> and retention should sit behind that reporting/emitting boundary as
> implementation-layer behavior. A durable JDBC path can be one named
> extension implementation. An event/listener forwarding path can be another
> named extension implementation.
>
> So I am +1 on the event-forwarding idea as a non-default extension
> implementation. I just would not make it the default battery or the core
> API shape. The default battery can stay small and safe, while deployments
> that need forwarding, dashboards, or durable storage can opt into the
> implementation that matches their operational model.
>
> Thanks,
> -ej
>
> On Wed, Jun 10, 2026 at 11:43 AM Yufei Gu <fl...@gmail.com> wrote:
>
> > Thanks EJ. I agree that the reporting/emitting boundary feels like the
> > right SPI boundary, but I wonder if we can simplify this even further.
> >
> > Iceberg scan and commit metrics look very similar to events to me. They
> are
> > append only, asynchronously consumed, often forwarded to external
> systems,
> > and have similar retention and cleanup requirements. More details can be
> > found in this thread [1]. In that case, Polaris may only need to accept
> the
> > report and emit an event through the existing event framework. Filtering,
> > async delivery, custom sinks, and retention mechanisms could then be
> shared
> > instead of introducing a separate metrics specific extension layer and
> SPI.
> >
> > The main requirement I still see is querying metrics through Polaris. If
> we
> > want that battery included experience, we could work on a persistence
> > listener implementation that selectively saves metrics and expose it via
> > the IRC event endpoint which seems pretty close in Iceberg
> > community. Another option I like more is to allow users to develop their
> > own listener, so they can persist scan/commit metrics to any systems they
> > prefer. In general, that feels like an implementation choice on top of
> the
> > event framework rather than something that needs to shape the core
> > architecture.
> >
> > 1. https://lists.apache.org/thread/x9j8nscvy8hq61tyn01mj8yp6n9of0kp
> >
> > Thanks,
> > Yufei
> >
> >
> > On Thu, Jun 4, 2026 at 5:14 PM EJ Wang <ej...@gmail.com>
> > wrote:
> >
> > > Good point. I agree the existing PolarisMetricsReporter is already very
> > > close to the right conceptual boundary. I am not proposing a second
> > > parallel reporter concept. The distinction I am trying to make is
> > narrower:
> > >
> > > Current state:
> > >
> > >    - PolarisMetricsReporter is the existing metrics reporting hook.
> > >    - But it currently lives under runtime/service.
> > >    - Its contract is documented around Quarkus/CDI discovery and
> > > @Identifier
> > >    selection.
> > >    - The default implementation is log-only.
> > >    - The durable implementation writes through PolarisMetricsManager
> and
> > >    MetricsPersistence.
> > >    - MetricsPersistence is currently inherited by BasePersistence.
> > >
> > > My proposal:
> > >
> > >    - Keep the reporting/emitting boundary as the SPI.
> > >    - Revise that contract in a framework-agnostic optional Iceberg
> > *metrics
> > >    extension API layer*.
> > >    - Keep runtime/service responsible for REST ingestion, table/authz
> > >    validation, and runtime wiring.
> > >    - Keep the battery implementation log-only in the metrics extension
> > API
> > >    layer, *not under runtime/service*.
> > >    - Treat durable JDBC metrics as one implementation of the reporting
> > SPI.
> > >    - Decompose the MetricsPersistence and BasePersistence coupling as
> > >    durable implementation detail.
> > >
> > > So the difference is not “new reporter concept vs old reporter
> concept.”
> > > but:
> > >
> > >
> > >    - PolarisMetricsReporter today = runtime/service CDI-shaped hook.
> > >    - Target reporting SPI = same conceptual boundary, but placed in the
> > >    right extension API layer and kept framework-agnostic.
> > >
> > > For the current metrics PR, I think the minimal architectural cleanup
> > > before merge is to *put the reporting SPI contract and the battery
> > default
> > > in the right place*. The durable JDBC implementation, event-backed
> async
> > > implementation, external queue support, and deeper persistence cleanup
> > can
> > > continue as follow-ups, as long as we do not lock the SPI boundary to
> > > BasePersistence or to runtime/service wiring.
> > >
> > > -ej
> > >
> > > On Wed, Jun 3, 2026 at 3:00 PM Dmitri Bourlatchkov <di...@apache.org>
> > > wrote:
> > >
> > > > Hi EJ,
> > > >
> > > > Thanks for the recap / summary!
> > > >
> > > > > Do folks agree that the stable SPI boundary should be
> > > > metrics reporting/emitting, not metrics persistence?
> > > >
> > > > SGTM.
> > > >
> > > > > Does an optional Iceberg metrics extension API layer sound like the
> > > right
> > > > home for this SPI?
> > > >
> > > > How is that different from current PolarisMetricsReporter?
> > > >
> > > > > Should the current durable metrics work be reframed as a durable
> > > > JDBC reference implementation of that SPI?
> > > >
> > > > SGTM.
> > > >
> > > > > What is the smallest PR sequence to get there without blocking
> > > > the current metrics work unnecessarily?
> > > >
> > > > Let's get https://github.com/apache/polaris/pull/4397 merged first.
> > > >
> > > > Then, I'd propose isolating the Metrics schema from the MetaStore
> > schema.
> > > > This will probably have some ripple effect into the bootstrap
> > workflows,
> > > so
> > > > it's not a trivial change.
> > > >
> > > > Then, let's reassess.
> > > >
> > > > Cheers,
> > > > Dmitri.
> > > >
> > > > On Wed, Jun 3, 2026 at 5:40 PM EJ Wang <
> ej.wang.dev.interest@gmail.com
> > >
> > > > wrote:
> > > >
> > > > > Hi folks,
> > > > >
> > > > > Thanks again for the discussion today. I updated the sync doc with
> > > notes
> > > > > from the third metrics architecture sync:
> > > > >
> > > > >
> > > > >
> > > >
> > >
> >
> https://docs.google.com/document/d/100h7c4damrUzVuquYbBHM0EvA4LSWuW2IT2dN_7nYVA/edit?tab=t.k96s2xyqr5u1
> > > > >
> > > > > A few highlights from today’s discussion:
> > > > >
> > > > >    - We clarified that Iceberg metrics reporting can be interpreted
> > > > either
> > > > >    as sync or async handling from Polaris’s perspective, so Polaris
> > > > should
> > > > >    stay flexible as a platform instead of baking one handling model
> > > into
> > > > > the
> > > > >    REST API behavior.
> > > > >    - We aligned that the built-in (battery) behavior for metrics
> > > emitting
> > > > >    can stay simple: no-op/log-only is enough as the default.
> > > > >    - Durable metrics persistence should be treated as an
> > implementation
> > > > of
> > > > >    the metrics reporting path, not as the core SPI boundary.
> > > > >    - The existing durable metrics work can be reviewed as a
> reference
> > > > >    implementation of the reporting SPI, with persistence-related
> > logic
> > > > kept
> > > > >    self-contained under a metrics durable implementation module
> > rather
> > > > than
> > > > >    scattered through core entity persistence.
> > > > >    - Dashboard/insights remains a real use case, but we agreed to
> > keep
> > > it
> > > > >    separate from the core metrics intake discussion for now.
> > > > >
> > > > > I also did a quick source check after the meeting to make sure we
> are
> > > > > describing the current state accurately.
> > > > >
> > > > > Current state:
> > > > >
> > > > >    - Polaris already has a metrics reporting hook:
> > > > PolarisMetricsReporter.
> > > > >    - The default implementation is DefaultMetricsReporter, selected
> > by
> > > > >    polaris.iceberg-metrics.reporting.type=default. It is log-only,
> > > > >    effectively quiet unless metrics logging is enabled.
> > > > >    - There is also a PersistingMetricsReporter, selected by
> > > > >    polaris.iceberg-metrics.reporting.type=persisting, which
> converts
> > > > >    Iceberg scan/commit reports into Polaris metrics records and
> > writes
> > > > > through
> > > > >    PolarisMetricsManager -> MetricsPersistence.
> > > > >    - MetricsPersistence currently exists in the persistence layer
> and
> > > is
> > > > >    inherited by BasePersistence.
> > > > >
> > > > > My read from the discussion is that the target boundary should not
> be
> > > > > MetricsPersistence as inherited by BasePersistence. That path
> should
> > be
> > > > > decomposed as durable implementation detail (taken care of by
> > > > > https://github.com/apache/polaris/pull/4397). The stable extension
> > > point
> > > > > should instead be the metrics reporting/emitting boundary.
> > > > >
> > > > > Concretely, I think the next proposal should be shaped like this:
> > > > >
> > > > >    1.
> > > > >
> > > > >    Define the proper metrics reporting SPI at an optional Iceberg
> > > metrics
> > > > >    extension API layer.
> > > > >
> > > > >    Example direction:
> > > > >
> > > > >    public interface IcebergMetricsReporter {
> > > > >      void reportMetric(IcebergMetricsReportContext context,
> > > > > MetricsReport report);
> > > > >    }
> > > > >
> > > > >    The context would carry the small Polaris-resolved envelope:
> > > > >    catalog/table identity, report type, received timestamp, and
> > > > > request/trace
> > > > >    context if available. The raw Iceberg MetricsReport remains the
> > > > payload.
> > > > >    2.
> > > > >
> > > > >    Keep runtime/service as the REST ingestion and wiring layer.
> > > > >
> > > > >    The REST handler still resolves the table and performs authz
> > before
> > > > >    accepting the report. After that, it calls the selected
> reporting
> > > > >    implementation.
> > > > >    3.
> > > > >
> > > > >    Keep the battery default no-op/log-only.
> > > > >
> > > > >    This preserves an out-of-box safe default and avoids requiring a
> > > > durable
> > > > >    metrics store for every Polaris deployment.
> > > > >    4.
> > > > >
> > > > >    Move durable JDBC metrics into a self-contained implementation
> of
> > > the
> > > > >    reporting SPI.
> > > > >
> > > > >    That implementation can own its schema, bootstrap, retention,
> and
> > > read
> > > > >    API support. It should not define the core reporting SPI
> boundary,
> > > and
> > > > > it
> > > > >    should not require metrics persistence to remain inherited from
> > > > >    BasePersistence.
> > > > >    5.
> > > > >
> > > > >    Treat async/event-backed handling as another implementation of
> the
> > > > same
> > > > >    reporting SPI.
> > > > >
> > > > >    For example, an event-backed reporter could enqueue a metrics
> > event
> > > > and
> > > > >    let listeners handle durable storage or other sinks. If we later
> > > need
> > > > a
> > > > >    replaceable queue engine, that seems like a shared event/metrics
> > > > > substrate
> > > > >    topic rather than a metrics-only requirement.
> > > > >
> > > > > This framing lets us keep the REST metrics endpoint simple,
> preserve
> > > the
> > > > > current default behavior, support durable metrics users, and still
> > > leave
> > > > > room for async/event-backed or external-queue-based
> implementations.
> > > > >
> > > > > I think the main follow-up questions are:
> > > > >
> > > > >    - Do folks agree that the stable SPI boundary should be metrics
> > > > >    reporting/emitting, not metrics persistence?
> > > > >    - Does an optional Iceberg metrics extension API layer sound
> like
> > > the
> > > > >    right home for this SPI?
> > > > >    - Should the current durable metrics work be reframed as a
> durable
> > > > JDBC
> > > > >    reference implementation of that SPI?
> > > > >    - What is the smallest PR sequence to get there without blocking
> > the
> > > > >    current metrics work unnecessarily?
> > > > >
> > > > > Thanks,
> > > > > -ej
> > > > >
> > > > > On Fri, May 15, 2026 at 5:16 PM Dmitri Bourlatchkov <
> > dimas@apache.org>
> > > > > wrote:
> > > > >
> > > > > > Hi JB,
> > > > > >
> > > > > > Could you set up another meeting, please? Same time on Wednesday
> as
> > > > last
> > > > > > time... I hope it works for everyone.
> > > > > >
> > > > > > Cheers,
> > > > > > Dmitri.
> > > > > >
> > > > > > On Fri, May 15, 2026 at 8:06 PM Yufei Gu <fl...@gmail.com>
> > > wrote:
> > > > > >
> > > > > > > +1 on another sync call next week.
> > > > > > >
> > > > > > > Yufei
> > > > > > >
> > > > > > >
> > > > > > > On Fri, May 15, 2026 at 4:52 PM Dmitri Bourlatchkov <
> > > > dimas@apache.org>
> > > > > > > wrote:
> > > > > > >
> > > > > > > > Hi All,
> > > > > > > >
> > > > > > > > WDYT about another sync call next week?
> > > > > > > >
> > > > > > > > Thanks,
> > > > > > > > Dmitri.
> > > > > > > >
> > > > > > > > On Wed, May 6, 2026 at 5:29 PM Dmitri Bourlatchkov <
> > > > dimas@apache.org
> > > > > >
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > > > Hi EJ,
> > > > > > > > >
> > > > > > > > > Thanks for the summary! It covers what we discussed in the
> > > > meeting
> > > > > > very
> > > > > > > > > well, IMHO.
> > > > > > > > >
> > > > > > > > > Looking forward to concrete PRs :)
> > > > > > > > >
> > > > > > > > > Cheers,
> > > > > > > > > Dmitri.
> > > > > > > > >
> > > > > > > > > On Wed, May 6, 2026 at 5:08 PM EJ Wang <
> > > > > > ej.wang.dev.interest@gmail.com
> > > > > > > >
> > > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > >> Hi folks,
> > > > > > > > >>
> > > > > > > > >> We had a community sync earlier, thanks JB for scheduling
> > it.
> > > > > Notes
> > > > > > > from
> > > > > > > > >> the first metrics architecture sync (May 6, 10-11am PT).
> > > > > Discussion
> > > > > > > doc
> > > > > > > > >> with per-section status:
> > > > > > > > >>
> > > > > > > > >>
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://docs.google.com/document/d/100h7c4damrUzVuquYbBHM0EvA4LSWuW2IT2dN_7nYVA/edit?tab=t.0
> > > > > > > > >>
> > > > > > > > >> *The meeting covered both topics from the doc.
> > Direction-level
> > > > > > > alignment
> > > > > > > > >> was reached on the headline pieces; details remain for PR
> > > review
> > > > > or
> > > > > > > > >> follow-up sessions.*
> > > > > > > > >>
> > > > > > > > >> *Topic 1 — Persistence schema redesign*
> > > > > > > > >> Idea-level alignment on consolidating per-type tables
> > > > > > > > >> (scan_metrics_report,
> > > > > > > > >> commit_metrics_report) into a single metrics_report table.
> > The
> > > > > > > > motivating
> > > > > > > > >> cost is the surface area added by every new metric type
> > today:
> > > > new
> > > > > > > > table,
> > > > > > > > >> SPI method, record class, model, converter, schema
> > migration.
> > > > > > > > >>
> > > > > > > > >> Most schema details are deferred to the schema PR. A few
> > > > specific
> > > > > > > points
> > > > > > > > >> came up:
> > > > > > > > >> •   metric_schema_version: Yufei prefers dropping it,
> since
> > > > there
> > > > > is
> > > > > > > no
> > > > > > > > >> spec-level concept of metrics versioning today and it is
> > hard
> > > to
> > > > > > > define
> > > > > > > > >> unilaterally. Robert prefers keeping it, given IRC v2 is
> > > coming
> > > > > and
> > > > > > > the
> > > > > > > > >> schema should be considered against its likely shape;
> Robert
> > > > also
> > > > > > > raised
> > > > > > > > >> how to differentiate various payload formats if any. EJ's
> > read
> > > > is
> > > > > > that
> > > > > > > > >> this
> > > > > > > > >> is a two-way-door decision. We can start without the
> field,
> > > and
> > > > if
> > > > > > IRC
> > > > > > > > v2
> > > > > > > > >> changes the shape we would likely roll a corresponding new
> > > > schema
> > > > > > > > anyway,
> > > > > > > > >> which is not particularly costly.
> > > > > > > > >> •   Payload format: Robert pointed out that future formats
> > > > beyond
> > > > > > JSON
> > > > > > > > may
> > > > > > > > >> be worth supporting. The exact shape is deferred to the
> > schema
> > > > > > > > discussion.
> > > > > > > > >> •   Partition strategy: Anand suggested monthly
> partitioning
> > > > based
> > > > > > on
> > > > > > > > his
> > > > > > > > >> experience as potentially helpful at scale.
> > > > > > > > >>
> > > > > > > > >> *Topic 2 — Where metrics ingestion and storage belong*
> > > > > > > > >> Idea-level alignment that metrics should be a separated
> SPI
> > > from
> > > > > the
> > > > > > > > >> entity
> > > > > > > > >> persistence stack. Two reasons surfaced: (a) workloads and
> > > > > > capability
> > > > > > > > >> requirements diverge enough that coupling them creates
> > > > artificial
> > > > > > > > >> constraints, and (b) admin experience improves when
> metrics
> > > has
> > > > > its
> > > > > > > own
> > > > > > > > >> bootstrap, retention, and lifecycle. Dmitri noted Polaris
> > > being
> > > > a
> > > > > > > > platform
> > > > > > > > >> should have the flexibility to support different
> persistence
> > > > > > backends
> > > > > > > > per
> > > > > > > > >> concern, and pointed to a concrete next step of separating
> > the
> > > > > JDBC
> > > > > > > > >> bootstrap for metrics from the metastore bootstrap. Robert
> > > > > proposed
> > > > > > an
> > > > > > > > >> additional UX extension: detect an unbootstrapped metrics
> > > store
> > > > on
> > > > > > > first
> > > > > > > > >> use and auto-bootstrap rather than requiring an explicit
> > > manual
> > > > > > > > bootstrap
> > > > > > > > >> step.
> > > > > > > > >> The meeting also confirmed that Polaris metrics can start
> > > small
> > > > > and
> > > > > > > stay
> > > > > > > > >> Iceberg-focused. Naming and persistence schema can lean
> > > > > > > > Iceberg-specific.
> > > > > > > > >> If a future expansion to generic-table metrics or
> > operational
> > > > > > metrics
> > > > > > > > >> arrives, an abstraction layer can be built on top of the
> > > Iceberg
> > > > > > > metrics
> > > > > > > > >> reporter at that point. Robert remains on the fence and
> > would
> > > > > prefer
> > > > > > > > >> something more generic but did not block the direction;
> > > Dmitri's
> > > > > > read
> > > > > > > > was
> > > > > > > > >> that the proposed framework already has enough flexibility
> > to
> > > > > absorb
> > > > > > > > >> future
> > > > > > > > >> expansion.
> > > > > > > > >>
> > > > > > > > >> The Trade-offs and Proposed structure sections in the doc
> > were
> > > > not
> > > > > > > > >> reviewed
> > > > > > > > >> in detail. They remain open for either the next sync or PR
> > > > review.
> > > > > > > > >>
> > > > > > > > >> *Cross-cutting alignment — battery-included plus
> pluggable*
> > > > > > > > >> A common philosophy emerged from the discussion. EJ
> > summarized
> > > > it
> > > > > > as:
> > > > > > > > >> Polaris should provide a battery-included UX for beginners
> > and
> > > > the
> > > > > > > > >> flexibility for advanced users to swap the included
> battery
> > > for
> > > > > > > > something
> > > > > > > > >> more powerful or tailored to their use case. The SPI
> design
> > > > needs
> > > > > to
> > > > > > > > >> enable
> > > > > > > > >> both.
> > > > > > > > >>
> > > > > > > > >> The inputs that shaped this framing:
> > > > > > > > >> •   Anand described how his team uses the current metrics
> > > > > > persistence
> > > > > > > > >> (three metrics consumers in v1.4).
> > > > > > > > >> •   Yufei raised Grafana and dashboard integrations as a
> > > > > destination
> > > > > > > use
> > > > > > > > >> case beyond the default.
> > > > > > > > >> •   Robert called out that the current design is more
> > > > > JDBC-focused.
> > > > > > > > >>
> > > > > > > > >> Two concrete instances:
> > > > > > > > >> •   Async metrics intake: Yufei's initial position was
> that
> > > > async
> > > > > > > should
> > > > > > > > >> largely live on the producer side and there is not much
> > > Polaris
> > > > > can
> > > > > > > do.
> > > > > > > > >> Robert suggested a Polaris-side default is doable via
> > Vert.x.
> > > > > Dmitri
> > > > > > > > >> agreed
> > > > > > > > >> the direction is worth exploring. The meeting converged
> on a
> > > > > > > > >> battery-included default (likely Vert.x-backed) with an
> SPI
> > > > shape
> > > > > > that
> > > > > > > > >> lets
> > > > > > > > >> power users route to a more scalable backend (k8s-hosted
> > > queue,
> > > > > AWS
> > > > > > > SQS,
> > > > > > > > >> etc.).
> > > > > > > > >> •   Pluggable destinations: combining Yufei's dashboard
> use
> > > case
> > > > > > with
> > > > > > > > >> Robert's JDBC-focused call-out, the meeting agreed the SPI
> > > > should
> > > > > be
> > > > > > > > >> structured for multiple sinks so integrations become impl
> > > > choices
> > > > > > > rather
> > > > > > > > >> than architectural changes.
> > > > > > > > >>
> > > > > > > > >> The battery-included default is most likely to use the
> > > existing
> > > > > > > > >> JDBC-backed
> > > > > > > > >> approach.
> > > > > > > > >>
> > > > > > > > >> *Direction (idea-level alignment)*
> > > > > > > > >> •   Single metrics_report table consolidating per-type
> > > metrics,
> > > > > > > > replacing
> > > > > > > > >> scan_metrics_report and commit_metrics_report
> > > > > > > > >> •   Iceberg-focused naming and schema for now, revisit if
> > > > > > > generic-table
> > > > > > > > or
> > > > > > > > >> operational metrics arrive
> > > > > > > > >> •   Metrics persistence as a separated SPI, not on
> > > > BasePersistence
> > > > > > > > >> •   Bootstrap path separated for metrics, independent of
> > > > metastore
> > > > > > > > >> bootstrap
> > > > > > > > >> •   "Battery-included plus pluggable" as the SPI design
> > > > philosophy
> > > > > > > > >>
> > > > > > > > >> *Open items*
> > > > > > > > >> •   Schema details: metric_schema_version, payload format,
> > IRC
> > > > v2
> > > > > > > > >> forward-compat shape
> > > > > > > > >> •   SPI design details — full review either in the next
> sync
> > > or
> > > > in
> > > > > > the
> > > > > > > > >> corresponding PR
> > > > > > > > >> •   Schema refactor PR ownership
> > > > > > > > >>
> > > > > > > > >> *Action items*
> > > > > > > > >> •   EJ to take a first stab at the SPI design and
> > potentially
> > > > > > partner
> > > > > > > > with
> > > > > > > > >> Anand to incorporate the lessons learned from the existing
> > > > > reporter
> > > > > > > and
> > > > > > > > >> persistence work.
> > > > > > > > >> •   Schema refactor PR ownership is not yet decided. If
> > anyone
> > > > is
> > > > > > > > >> interested in driving it, reply on this thread.
> > > > > > > > >> •   JB to schedule the next sync, tentatively in two
> weeks.
> > > > > > > > >>
> > > > > > > > >> -ej
> > > > > > > > >>
> > > > > > > > >> On Mon, Apr 27, 2026 at 3:07 PM EJ Wang <
> > > > > > > ej.wang.dev.interest@gmail.com
> > > > > > > > >
> > > > > > > > >> wrote:
> > > > > > > > >>
> > > > > > > > >> > Thanks Yufei for the +1.
> > > > > > > > >> >
> > > > > > > > >> > JB, could you help add a biweekly metrics architecture
> > sync
> > > to
> > > > > the
> > > > > > > > >> Polaris
> > > > > > > > >> > community calendar? I'm thinking Thursdays at 9-10am PT,
> > on
> > > > the
> > > > > > > > >> off-weeks
> > > > > > > > >> > from the community meeting (starting May 7), 60 minutes.
> > > > > > > > >> >
> > > > > > > > >> > Here's a rough agenda to work through over the first few
> > > > > sessions,
> > > > > > > > >> grouped
> > > > > > > > >> > by priority:
> > > > > > > > >> >
> > > > > > > > >> > *First: foundational direction*
> > > > > > > > >> >
> > > > > > > > >> > 1.  MetricsPersistence: public SPI or internal
> > > implementation
> > > > > > > detail?
> > > > > > > > >> >    •   Marked @Beta, javadoc calls it a "Service
> Provider
> > > > > > > Interface",
> > > > > > > > >> but
> > > > > > > > >> > only one consumer (JdbcBasePersistenceImpl), lives on
> > > > > > > BasePersistence.
> > > > > > > > >> If
> > > > > > > > >> > demoted to a private helper inside a persisting reporter
> > > impl,
> > > > > > most
> > > > > > > > >> > downstream design decisions become implementation
> details
> > > > rather
> > > > > > > than
> > > > > > > > >> > contract questions.
> > > > > > > > >> >
> > > > > > > > >> > 2.  Persistence schema redesign
> > > > > > > > >> >    •   Current two-table layout (scan_metrics_report,
> > > > > > > > >> > commit_metrics_report) with ~25 flattened columns each.
> > > Every
> > > > > new
> > > > > > > > metric
> > > > > > > > >> > type requires a new table, SPI method, record class,
> > model,
> > > > > > > converter,
> > > > > > > > >> and
> > > > > > > > >> > schema migration. Direction to explore: single table
> with
> > > > > > > metric_type
> > > > > > > > >> enum,
> > > > > > > > >> > schema_version, and JSON payload column.
> > > > > > > > >> >
> > > > > > > > >> > *Second: design details once direction is set*
> > > > > > > > >> >
> > > > > > > > >> > 3.  Partition key strategy
> > > > > > > > >> >    •   Single-table design means scan metrics at scale
> > will
> > > > have
> > > > > > > high
> > > > > > > > >> > write concurrency per table. Schema needs to expose
> enough
> > > > > > structure
> > > > > > > > for
> > > > > > > > >> > backends to shard by entity or time range.
> > > > > > > > >> >
> > > > > > > > >> > 4.  Read/write path consistency
> > > > > > > > >> >    •   Writes go through PolarisMetricsManager on
> > > > > > MetaStoreManager.
> > > > > > > > >> Reads
> > > > > > > > >> > bypass MetaStoreManager and go straight to
> > BasePersistence,
> > > > > > > excluding
> > > > > > > > >> > non-JDBC backends from the read API.
> > > > > > > > >> >
> > > > > > > > >> > *Third: cleanup and alignment*
> > > > > > > > >> >
> > > > > > > > >> > 5.  PolarisMetricsReporter naming
> > > > > > > > >> >    •   Only handles IRC (ScanReport/CommitReport),
> doesn't
> > > > cover
> > > > > > > > generic
> > > > > > > > >> > tables or operational metrics. Name is broader than
> scope.
> > > > > > > > >> >
> > > > > > > > >> > 6.  PolarisMetricsManager facade passthrough
> > > > > > > > >> >    •   Entire default method is
> > > > > > > > >> callCtx.getMetaStore().writeScanReport().
> > > > > > > > >> > Zero logic, passes Level 1 straight through to Level 3.
> > Same
> > > > > > > > >> anti-pattern
> > > > > > > > >> > as PolarisEventManager.
> > > > > > > > >> >
> > > > > > > > >> > 7.  Iceberg community alignment
> > > > > > > > >> >    •   Payload-type extension needs discussion on
> > > dev@iceberg.
> > > > > > > > >> obelix74's
> > > > > > > > >> > Feb thread got zero replies. Needs a committer voice.
> > > > > > > > >> >
> > > > > > > > >> > Lets confirm prioritization in the first session.
> > > > > > > > >> >
> > > > > > > > >> > -ej
> > > > > > > > >> >
> > > > > > > > >> > On Tue, Apr 21, 2026 at 3:18 PM Yufei Gu <
> > > > flyrain000@gmail.com>
> > > > > > > > wrote:
> > > > > > > > >> >
> > > > > > > > >> >> Thanks everyone for continuing to drive this forward. I
> > > agree
> > > > > > that
> > > > > > > > the
> > > > > > > > >> >> problem is getting complex enough that a more
> structured
> > > > > > discussion
> > > > > > > > >> would
> > > > > > > > >> >> help.
> > > > > > > > >> >>
> > > > > > > > >> >> +1 on setting up a biweekly sync for the metrics
> > > > architecture.
> > > > > > I’m
> > > > > > > > >> happy
> > > > > > > > >> >> to
> > > > > > > > >> >> join.
> > > > > > > > >> >>
> > > > > > > > >> >> Yufei
> > > > > > > > >> >>
> > > > > > > > >> >>
> > > > > > > > >> >> On Tue, Apr 21, 2026 at 2:34 PM EJ Wang <
> > > > > > > > >> ej.wang.dev.interest@gmail.com>
> > > > > > > > >> >> wrote:
> > > > > > > > >> >>
> > > > > > > > >> >> > Also, I've been looking more closely at the
> > *persistence
> > > > > schema
> > > > > > > in
> > > > > > > > >> the
> > > > > > > > >> >> > current metrics work*, and I think there's a
> structural
> > > > > > rigidity
> > > > > > > > >> problem
> > > > > > > > >> >> > worth raising before the shape gets locked in.
> > > > > > > > >> >> >
> > > > > > > > >> >> > Right now we have two separate tables
> > > (scan_metrics_report
> > > > > and
> > > > > > > > >> >> > commit_metrics_report), each with ~25 flattened
> columns
> > > > that
> > > > > > > > directly
> > > > > > > > >> >> > mirror the Iceberg report fields. The SPI follows the
> > > same
> > > > > > split:
> > > > > > > > >> >> > writeScanReport and writeCommitReport as separate
> > > methods,
> > > > > with
> > > > > > > > >> per-type
> > > > > > > > >> >> > record classes, converters, and model objects. *The
> > > > practical
> > > > > > > cost:
> > > > > > > > >> >> > adding a new metric type (operational metrics, for
> > > example)
> > > > > > > > requires
> > > > > > > > >> a
> > > > > > > > >> >> new
> > > > > > > > >> >> > table, a new SPI method, a new record class, a new
> > model
> > > > > > class, a
> > > > > > > > new
> > > > > > > > >> >> > converter branch, and a schema migration*. That's a
> lot
> > > of
> > > > > > > surface
> > > > > > > > >> area
> > > > > > > > >> >> > for what should be "one more kind of metric."
> > > > > > > > >> >> >
> > > > > > > > >> >> > *My bias* would be toward a single metrics table with
> > *a
> > > > > typed
> > > > > > > JSON
> > > > > > > > >> >> > payload*. Something like: metric_type (enum),
> > entity_id,
> > > > > > > > >> >> > table_identifier, snapshot_id (nullable),
> received_ts,
> > > > > > > > >> schema_version,
> > > > > > > > >> >> and
> > > > > > > > >> >> > a payload column for the metric-specific data. The
> > > > > metric_type
> > > > > > +
> > > > > > > > >> >> > schema_version pair gives us a forward-compatible
> > > contract
> > > > > for
> > > > > > > the
> > > > > > > > >> >> payload
> > > > > > > > >> >> > shape. Adding a new metric type becomes an enum value
> > > and a
> > > > > > > payload
> > > > > > > > >> >> schema,
> > > > > > > > >> >> > not a schema migration. One thing I think we need to
> be
> > > > > > > deliberate
> > > > > > > > >> >> about is
> > > > > > > > >> >> > the partition key design. If all metric types land in
> > one
> > > > > > table,
> > > > > > > > scan
> > > > > > > > >> >> > metrics at scale (high concurrency, high frequency
> > across
> > > > > many
> > > > > > > > >> tables)
> > > > > > > > >> >> > could easily create hot partitions. We'd want the
> > > > persistence
> > > > > > > layer
> > > > > > > > >> to
> > > > > > > > >> >> be
> > > > > > > > >> >> > able to shard by entity or time range, and that means
> > the
> > > > > > logical
> > > > > > > > >> schema
> > > > > > > > >> >> > needs to expose enough structure for backends to
> > > partition
> > > > > on.
> > > > > > I
> > > > > > > > >> don't
> > > > > > > > >> >> > think the current flattened layout gives us that.
> > > > > > > > >> >> >
> > > > > > > > >> >> > This is getting complex enough that I don't think
> > ad-hoc
> > > > > PR/ML
> > > > > > > > >> threads
> > > > > > > > >> >> > will converge well. *Would people be open to a
> biweekly
> > > > sync
> > > > > > for
> > > > > > > > >> metrics
> > > > > > > > >> >> > architecture?* I think 30 minutes every two weeks
> with
> > > > > > interested
> > > > > > > > >> >> parties
> > > > > > > > >> >> > would be enough to work through the schema, SPI
> shape,
> > > and
> > > > > read
> > > > > > > API
> > > > > > > > >> >> design
> > > > > > > > >> >> > together. Happy to help set that up.
> > > > > > > > >> >> >
> > > > > > > > >> >> > -ej
> > > > > > > > >> >> >
> > > > > > > > >> >> > On Mon, Apr 20, 2026 at 2:19 PM EJ Wang <
> > > > > > > > >> ej.wang.dev.interest@gmail.com
> > > > > > > > >> >> >
> > > > > > > > >> >> > wrote:
> > > > > > > > >> >> >
> > > > > > > > >> >> >> Reviewed #4115, left a comment on the code
> > organization
> > > > > side.
> > > > > > > > >> >> >>
> > > > > > > > >> >> >> One thing stood out: the metrics write path enters
> > > through
> > > > > > > > >> >> >> PolarisMetricsManager on MetaStoreManager, but the
> new
> > > > read
> > > > > > path
> > > > > > > > >> >> bypasses
> > > > > > > > >> >> >> MetaStoreManager entirely and goes straight to
> > > > > BasePersistence
> > > > > > > via
> > > > > > > > >> >> >> callContext.getMetaStore(). That means the read API
> > only
> > > > > works
> > > > > > > for
> > > > > > > > >> >> backends
> > > > > > > > >> >> >> that implement BasePersistence. NoSQL and remote
> > > backends
> > > > > > can't
> > > > > > > > >> >> participate.
> > > > > > > > >> >> >>
> > > > > > > > >> >> >> Stepping back, I think the metrics subsystem is
> > growing
> > > > into
> > > > > > > > >> something
> > > > > > > > >> >> >> real (write + read + REST API + AuthZ + pagination)
> > *but
> > > > the
> > > > > > > > >> >> persistence
> > > > > > > > >> >> >> side is split across two layers in a way that's hard
> > to
> > > > > > > extend*. I
> > > > > > > > >> put
> > > > > > > > >> >> >> together two diagrams to show what I mean (my best
> > > > effort).
> > > > > > > > >> >> >>
> > > > > > > > >> >> >> *Current state* (Diagram 1): three interfaces at
> three
> > > > > > different
> > > > > > > > >> >> levels.
> > > > > > > > >> >> >> The engine-facing SPI (PolarisMetricsReporter) is
> > clean.
> > > > But
> > > > > > > > >> >> >> PolarisMetricsManager on MetaStoreManager is a
> > > passthrough
> > > > > to
> > > > > > > > >> >> >> MetricsPersistence on BasePersistence. The @Beta
> > > > annotation
> > > > > > and
> > > > > > > > SPI
> > > > > > > > >> >> javadoc
> > > > > > > > >> >> >> are on the BasePersistence layer, while the actual
> > > > extension
> > > > > > > > points
> > > > > > > > >> >> >> (PolarisMetricsReporter, PolarisMetricsManager)
> carry
> > no
> > > > > > > stability
> > > > > > > > >> >> >> annotation. The write path goes through the
> > > > MetaStoreManager
> > > > > > > > layer,
> > > > > > > > >> the
> > > > > > > > >> >> >> read path doesn't.
> > > > > > > > >> >> >>
> > > > > > > > >> >> >> *What I envision* (Diagram 2): two SPIs at two
> levels.
> > > > > > > > >> >> >> PolarisMetricsReporter stays as the engine-facing
> SPI.
> > > > > > > > >> >> >> PolarisMetricsManager becomes the backend-facing SPI
> > > with
> > > > > both
> > > > > > > > write
> > > > > > > > >> >> and
> > > > > > > > >> >> >> read methods at the MetaStoreManager level, where
> any
> > > > > backend
> > > > > > > > (JDBC,
> > > > > > > > >> >> NoSQL,
> > > > > > > > >> >> >> remote) can implement them. MetricsPersistence on
> > > > > > > BasePersistence
> > > > > > > > >> goes
> > > > > > > > >> >> >> away. Where metrics actually land is an
> implementation
> > > > > detail,
> > > > > > > > not a
> > > > > > > > >> >> core
> > > > > > > > >> >> >> interface.
> > > > > > > > >> >> >>
> > > > > > > > >> >> >> *Minor naming thing*: PolarisMetricsReporter is
> > broader
> > > > than
> > > > > > > what
> > > > > > > > it
> > > > > > > > >> >> >> actually handles. It only accepts Iceberg REST
> Catalog
> > > > > metrics
> > > > > > > > >> >> (ScanReport,
> > > > > > > > >> >> >> CommitReport via MetricsReport). Generic table
> metrics
> > > or
> > > > > > > > >> operational
> > > > > > > > >> >> >> metrics aren't in scope. Not blocking, but worth
> > noting
> > > if
> > > > > the
> > > > > > > > >> metrics
> > > > > > > > >> >> >> surface expands.
> > > > > > > > >> >> >>
> > > > > > > > >> >> >> *Rough sketch of how to get there*:
> > > > > > > > >> >> >>  1.  Add read methods to PolarisMetricsManager
> > > > > > (listScanReports,
> > > > > > > > >> >> >> listCommitReports) with default no-op, same as the
> > > > existing
> > > > > > > write
> > > > > > > > >> >> methods.
> > > > > > > > >> >> >> (Probably make PolarisMetricsManager more explicit
> on
> > > > being
> > > > > > > > Iceberg
> > > > > > > > >> >> >> specific like package name or class name etc.)
> > > > > > > > >> >> >>  2.  Wire MetricsReportsService through
> > MetaStoreManager
> > > > > > instead
> > > > > > > > of
> > > > > > > > >> >> >> callContext.getMetaStore().
> > > > > > > > >> >> >>  3.  Extract metrics persistence from
> > > > > JdbcBasePersistenceImpl
> > > > > > > into
> > > > > > > > >> its
> > > > > > > > >> >> >> own class. That file carries ~7 responsibilities,
> > > metrics
> > > > > > being
> > > > > > > > one
> > > > > > > > >> of
> > > > > > > > >> >> them.
> > > > > > > > >> >> >>  4.  Remove MetricsPersistence from BasePersistence.
> > > > > > > > >> >> >>
> > > > > > > > >> >> >> *None of this needs to happen in #4115. But if the
> > > > direction
> > > > > > > makes
> > > > > > > > >> >> sense,
> > > > > > > > >> >> >> it would be good to align before the metrics surface
> > > grows
> > > > > > > > further.
> > > > > > > > >> >> Curious
> > > > > > > > >> >> >> what others think.*
> > > > > > > > >> >> >>
> > > > > > > > >> >> >> *My mental model note*: Level 1 MetaStoreManager;
> > level
> > > 2
> > > > > > > > >> transactional
> > > > > > > > >> >> >> persistence; level 3 base persistence
> > > > > > > > >> >> >>
> > > > > > > > >> >> >> Diagram 1
> > > > > > > > >> >> >> <
> > > > > > > > >> >>
> > > > > > > > >>
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://www.plantuml.com/plantuml/uml/bLHDR-Cs4BthLmpIYupw0zbkKQ1r3M-S7Bp8xhhM7WCOb3IM65EaGD9EX2RzxHrHb4CxRelwa4YSDu_lpOVcnZ9jzvM8BBS2uGjQpJC3dtHMSekPtMk44IpsMgEqa5XcCOhCZikQQLP1pR8TAp2n3ILhmZDP20m0fcIvUkAoW2qJXd9z1bpToO9BX3WXu0ucy5rpgGPNm0nW5_epUWtm2Ue3pn3kMOFQmKntGZW0BYtgBSi8k5A2QMwybJNMIbFiGSR9QZc4nUqIvikStF0jHprua5C-amge42aNt3R0f5JaaoivdV2Pkqbx4hee4ymOkBh5BTiB-_uIeGeo8zL8rPsPl4DktdEiK1jkB1NdZCRbrSTecDe_mlHbF0wvBmCkaOH5_S8a_TTTKI6-nmCAkEw4LpxsZ-LbYLKQFKMNOgf_wuM7_bV9gOer5SYMMksBSWXFcbi49KNZXNLicwfe3TETC7gPdPqI7uBcHMb1RSzYq34c6PDUM9mn8HRsUTZEiDBve3NjVZumBj0U7SS37mGO7vcwtiK-_pU7U7L_f-digo9YbhSwIfMRwIITKGXbxdIUTCGF1SeCJxloKsU-3k9ddRbX1eDq1q_fx1JbBGT0glVyXimDuP4TQ5qpCAmnGEj2s_6n5mtn1z-97-63itFQZLPO1Ev2tu_WF7Ju-VPc0Skg5bYXxBhkY1xpD7EM_7fyflSpIsqMgVth5xhVr4eQxWQ8enaSAJQSG16yFSDuJ798rrcXr_3n-lfdk7icQjEBmFujL7AodiP_Y4Z7-YxvtZNs4zMgpNTl6tF8sglyPsmqchrjvQ-m-aP94r-TwCA2Ka8upPJZwtvSpoYCXkYMZU2NXvRMBfq9P3i3Le4VAZUAlUZ_oPKsxPgY0Q_BSKLkyr9bhQhQrJjo_x3TPlIB0DPjnMfcIoYP0QaYw1a0fTKDr8fB6ntNuvmoL1ZGkXa69Njh43zf9GiGxHQrA_jDYWRSzF5--WmTVrN97_Sm8LbLUy_lGBmLanJjFkDlGkRqjA_4tm00
> > > > > > > > >> >> >
> > > > > > > > >> >> >> :
> > > > > > > > >> >> >>
> > > > > > > > >> >> >> [image: image.png]
> > > > > > > > >> >> >>
> > > > > > > > >> >> >> Diagram 2
> > > > > > > > >> >> >> <
> > > > > > > > >> >>
> > > > > > > > >>
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://www.plantuml.com/plantuml/uml/VLLDR-8m4BtdLupO2sWBLVU8AaGB7AXAbssGzb896SS9RXqxjKqBwkv_tt7iV43fdaZYDpFlpRmnOsE9jhjSH9PRmM31hERKm8scMsuPjJlDe0yheZDc8RR4iYWoBrmMH9CS2a9VICPYUy1OZN0YCy5Q0BCbYNhdCeEK28En8G8wCvbnoQ0R8_05Bc6bkLIz3X03p1zzH7zR-9ZfDquPt9C3qoNCX2yV4G2NbkcKu5jdgGJHt0GbZwnG6i-UP3TUpk5gM6Ldqke350eZUqzoCft3U9xWHvxoa5-7K4nF1J46EbEMafsmdrCBbQ44gVggy18IZrn_ph5asd1ZiIKdQSgueZvjXrQFSFrdC3YN-nXmBacxbGiYyLVxLaBtdhqn0LSzdBDhqQtQoOJeGyad3z0lUqnYgpGB6Ns8oVyta00Dy_WnX0tIOZ8v6SYxHll1TrH6aejAik-mh-AphVFCwSUQqFypElag5QRGFDjQKEd96K1P8QP41c9TzA_IIQyvdAWyv_RSiS3skb0_EzDDkK2v5xWF6MiGFlvhpFLcD2Dq2pml14gaF67eQkmd8gulDoC4kSOu6KVpkvlUJg1RTbWISU40RdBUUS_9XfRZ2dwxm_SW8LYFISgm_MnlDQ6M9P1gbKEc4X-2pH_FvJCkCqm9pbVjD6LrwdLeOrDWfOaqc8Wh9BE85oNKxkNQ6o4yGRy_Eae0G_G8tZv81d3bHDB23WOdisohVr3nh_j6lbSjbNaLRTc8UgtPbAU1J_tygOfZX9DWEJeHDvYx-qmSi5FgNLPZwHrHcUsncGQ5-skhUclpE5fo4ounpFauYrUbkU6ccfnxMvitwag4IyerhTxj8In_Oj1bDO4pQru674loYrGlULHLEGCjwJJ8gDoVZR8MxO4BT3IzRvIcAQKezC6xpziGnTyImrfEGyJI_OcKfgtxIvnTqFEMS17L9Z-jsARN5FmTheP7HtSdtOMT0B4GY2FYHXxgQmMtj2bRqiLFGapiVe1_QVKDrkqXcm83aFEXnMYCZ-xlyHy
> > > > > > > > >> >> >
> > > > > > > > >> >> >> :
> > > > > > > > >> >> >> [image: image.png]
> > > > > > > > >> >> >>
> > > > > > > > >> >> >>  -ej
> > > > > > > > >> >> >>
> > > > > > > > >> >> >> On Wed, Apr 15, 2026 at 8:22 AM Dmitri Bourlatchkov
> <
> > > > > > > > >> dimas@apache.org>
> > > > > > > > >> >> >> wrote:
> > > > > > > > >> >> >>
> > > > > > > > >> >> >>> Hi All,
> > > > > > > > >> >> >>>
> > > > > > > > >> >> >>> Heads up: The current state of PR [4115] looks
> pretty
> > > > solid
> > > > > > to
> > > > > > > > me.
> > > > > > > > >> I
> > > > > > > > >> >> >>> believe this PR is approaching a mergeable
> condition.
> > > > > > > > >> >> >>>
> > > > > > > > >> >> >>> Please post your reviews if you have any comments.
> > > > > > > > >> >> >>>
> > > > > > > > >> >> >>> [4115] https://github.com/apache/polaris/pull/4115
> > > > > > > > >> >> >>>
> > > > > > > > >> >> >>> Thanks,
> > > > > > > > >> >> >>> Dmitri.
> > > > > > > > >> >> >>>
> > > > > > > > >> >> >>> On Tue, Mar 3, 2026 at 3:29 PM Anand Kumar Sankaran
> > via
> > > > > dev <
> > > > > > > > >> >> >>> dev@polaris.apache.org> wrote:
> > > > > > > > >> >> >>>
> > > > > > > > >> >> >>> > Hi Yufei and Dmitri,
> > > > > > > > >> >> >>> >
> > > > > > > > >> >> >>> > Here is a proposal for the REST endpoints for
> > metrics
> > > > and
> > > > > > > > events.
> > > > > > > > >> >> >>> >
> > > > > > > > >> >> >>> >
> > https://github.com/apache/polaris/pull/3924/changes
> > > > > > > > >> >> >>> >
> > > > > > > > >> >> >>> > I did not see any precursors for raising a PR for
> > > > > > proposals,
> > > > > > > so
> > > > > > > > >> >> trying
> > > > > > > > >> >> >>> > this.  Please let me know what you think.
> > > > > > > > >> >> >>> >
> > > > > > > > >> >> >>> > -
> > > > > > > > >> >> >>> > Anand
> > > > > > > > >> >> >>> >
> > > > > > > > >> >> >>> > From: Anand Kumar Sankaran <
> > > anand.sankaran@workday.com
> > > > >
> > > > > > > > >> >> >>> > Date: Monday, March 2, 2026 at 10:25 AM
> > > > > > > > >> >> >>> > To: dev@polaris.apache.org <
> dev@polaris.apache.org
> > >
> > > > > > > > >> >> >>> > Subject: Re: Polaris Telemetry and Audit Trail
> > > > > > > > >> >> >>> >
> > > > > > > > >> >> >>> > About the REST API, based on my use cases:
> > > > > > > > >> >> >>> >
> > > > > > > > >> >> >>> >
> > > > > > > > >> >> >>> >   1.
> > > > > > > > >> >> >>> > I want to be able to query commit metrics to
> track
> > > > files
> > > > > > > added
> > > > > > > > /
> > > > > > > > >> >> >>> removed
> > > > > > > > >> >> >>> > per commit, along with record counts. The
> ingestion
> > > > > > pipeline
> > > > > > > > that
> > > > > > > > >> >> >>> writes
> > > > > > > > >> >> >>> > this data is owned by us and we are guaranteed to
> > > write
> > > > > > this
> > > > > > > > >> >> >>> information
> > > > > > > > >> >> >>> > for each write.
> > > > > > > > >> >> >>> >   2.
> > > > > > > > >> >> >>> > I want to be able to query scan metrics for
> read. I
> > > > > > > understand
> > > > > > > > >> >> clients
> > > > > > > > >> >> >>> do
> > > > > > > > >> >> >>> > not fulfill this requirement.
> > > > > > > > >> >> >>> >   3.
> > > > > > > > >> >> >>> > I want to be able to query the events table
> (events
> > > are
> > > > > > > > >> persisted) -
> > > > > > > > >> >> >>> this
> > > > > > > > >> >> >>> > may supersede #2, I am not sure yet.
> > > > > > > > >> >> >>> >
> > > > > > > > >> >> >>> > All this information is in the JDBC based
> > persistence
> > > > > model
> > > > > > > and
> > > > > > > > >> is
> > > > > > > > >> >> >>> > persisted in the metastore. I currently don’t
> have
> > a
> > > > need
> > > > > > to
> > > > > > > > >> query
> > > > > > > > >> >> >>> > prometheus or open telemetry. I do publish some
> > > events
> > > > to
> > > > > > > > >> Prometheus
> > > > > > > > >> >> >>> and
> > > > > > > > >> >> >>> > they are forwarded to our dashboards elsewhere.
> > > > > > > > >> >> >>> >
> > > > > > > > >> >> >>> > About the CLI utilities, I meant the admin user
> > > > > utilities.
> > > > > > In
> > > > > > > > >> one of
> > > > > > > > >> >> >>> the
> > > > > > > > >> >> >>> > earliest drafts of my proposal, Prashant
> mentioned
> > > that
> > > > > the
> > > > > > > > >> metrics
> > > > > > > > >> >> >>> tables
> > > > > > > > >> >> >>> > can grow indefinitely and that a similar problem
> > > exists
> > > > > > with
> > > > > > > > the
> > > > > > > > >> >> events
> > > > > > > > >> >> >>> > table as well. We discussed that cleaning up of
> old
> > > > > records
> > > > > > > > from
> > > > > > > > >> >> both
> > > > > > > > >> >> >>> > metrics tables and events tables can be done via
> a
> > > CLI
> > > > > > > utility.
> > > > > > > > >> >> >>> >
> > > > > > > > >> >> >>> > I see that Yufei has covered the discussion about
> > > > > > > datasources.
> > > > > > > > >> >> >>> >
> > > > > > > > >> >> >>> > -
> > > > > > > > >> >> >>> > Anand
> > > > > > > > >> >> >>> >
> > > > > > > > >> >> >>> >
> > > > > > > > >> >> >>> >
> > > > > > > > >> >> >>> > From: Yufei Gu <fl...@gmail.com>
> > > > > > > > >> >> >>> > Date: Friday, February 27, 2026 at 9:54 PM
> > > > > > > > >> >> >>> > To: dev@polaris.apache.org <
> dev@polaris.apache.org
> > >
> > > > > > > > >> >> >>> > Subject: Re: Polaris Telemetry and Audit Trail
> > > > > > > > >> >> >>> >
> > > > > > > > >> >> >>> > This Message Is From an External Sender
> > > > > > > > >> >> >>> > This message came from outside your organization.
> > > > > > > > >> >> >>> > Report Suspicious<
> > > > > > > > >> >> >>> >
> > > > > > > > >> >> >>>
> > > > > > > > >> >>
> > > > > > > > >>
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://us-phishalarm-ewt.proofpoint.com/EWT/v1/Iz9xO38YGHZK!YhNDZABkHi1B699ote2uMwpOZw8i0QMCGO2Szc-HshuABGhGvwPJcymE6G2oUUxtS8xDkSrtGTPm_I3QnVDHoLMk50m9v8z_nZKTkd-bnVUbreF1u0WnfV_X5eYevZl_$
> > > > > > > > >> >> >>> > >
> > > > > > > > >> >> >>> >
> > > > > > > > >> >> >>> >
> > > > > > > > >> >> >>> > As I mentioned in
> > > > > > > > >> >> >>> >
> > > > > > > > >> >> >>>
> > > > > > > > >> >>
> > > > > > > > >>
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://urldefense.com/v3/__https://github.com/apache/polaris/issues/3890__;!!Iz9xO38YGHZK!5EuyFFkk3vhRWVIRvQAWBSQfpJkTMA9HxugzDwXmN0LPPqhEFxYkFRGVhtb8AqUwXtDh2OplcMnbMDHKOxrvDU0$
> > > > > > > > >> >> >>> ,
> > > > > > > > >> >> >>> > supporting
> > > > > > > > >> >> >>> > multiple data sources is not a trivial change. I
> > > would
> > > > > > > strongly
> > > > > > > > >> >> >>> recommend
> > > > > > > > >> >> >>> > starting with a design document to carefully
> > evaluate
> > > > the
> > > > > > > > >> >> architectural
> > > > > > > > >> >> >>> > implications and long term impact.
> > > > > > > > >> >> >>> >
> > > > > > > > >> >> >>> > A REST endpoint to query metrics seems reasonable
> > > given
> > > > > the
> > > > > > > > >> current
> > > > > > > > >> >> >>> JDBC
> > > > > > > > >> >> >>> > based persistence model. That said, we may also
> > > > consider
> > > > > > > > >> alternative
> > > > > > > > >> >> >>> > storage models. For example, if we later adopt a
> > time
> > > > > > series
> > > > > > > > >> system
> > > > > > > > >> >> >>> such as
> > > > > > > > >> >> >>> > Prometheus to store metrics, the query model and
> > > access
> > > > > > > > patterns
> > > > > > > > >> >> would
> > > > > > > > >> >> >>> be
> > > > > > > > >> >> >>> > fundamentally different. Designing the REST API
> > > without
> > > > > > > > >> considering
> > > > > > > > >> >> >>> these
> > > > > > > > >> >> >>> > potential evolutions may limit flexibility. I'd
> > > suggest
> > > > > to
> > > > > > > > start
> > > > > > > > >> >> with
> > > > > > > > >> >> >>> the
> > > > > > > > >> >> >>> > use case.
> > > > > > > > >> >> >>> >
> > > > > > > > >> >> >>> > Yufei
> > > > > > > > >> >> >>> >
> > > > > > > > >> >> >>> >
> > > > > > > > >> >> >>> > On Fri, Feb 27, 2026 at 3:42 PM Dmitri
> > Bourlatchkov <
> > > > > > > > >> >> dimas@apache.org>
> > > > > > > > >> >> >>> > wrote:
> > > > > > > > >> >> >>> >
> > > > > > > > >> >> >>> > > Hi Anand,
> > > > > > > > >> >> >>> > >
> > > > > > > > >> >> >>> > > Sharing my view... subject to discussion:
> > > > > > > > >> >> >>> > >
> > > > > > > > >> >> >>> > > 1. Adding non-IRC REST API to Polaris is
> > perfectly
> > > > > fine.
> > > > > > > > >> >> >>> > >
> > > > > > > > >> >> >>> > > Figuring out specific endpoint URIs and
> payloads
> > > > might
> > > > > > > > require
> > > > > > > > >> a
> > > > > > > > >> >> few
> > > > > > > > >> >> >>> > > roundtrips, so opening a separate thread for
> that
> > > > might
> > > > > > be
> > > > > > > > >> best.
> > > > > > > > >> >> >>> > > Contributors commonly create Google Docs for
> new
> > > API
> > > > > > > > proposals
> > > > > > > > >> too
> > > > > > > > >> >> >>> (they
> > > > > > > > >> >> >>> > > fairly easy to update as the email discussion
> > > > > > progresses).
> > > > > > > > >> >> >>> > >
> > > > > > > > >> >> >>> > > There was a suggestion to try Markdown (with
> PRs)
> > > for
> > > > > > > > proposals
> > > > > > > > >> >> [1]
> > > > > > > > >> >> >>> ...
> > > > > > > > >> >> >>> > > feel free to give it a try if you are
> comfortable
> > > > with
> > > > > > > that.
> > > > > > > > >> >> >>> > >
> > > > > > > > >> >> >>> > > 2. Could you clarify whether you mean end user
> > > > > utilities
> > > > > > or
> > > > > > > > >> admin
> > > > > > > > >> >> >>> user
> > > > > > > > >> >> >>> > > utilities? In the latter case those might be
> more
> > > > > > suitable
> > > > > > > > for
> > > > > > > > >> the
> > > > > > > > >> >> >>> Admin
> > > > > > > > >> >> >>> > > CLI (java) not the Python CLI, IMHO.
> > > > > > > > >> >> >>> > >
> > > > > > > > >> >> >>> > > Why would these utilities be common with
> events?
> > > > IMHO,
> > > > > > > event
> > > > > > > > >> use
> > > > > > > > >> >> >>> cases
> > > > > > > > >> >> >>> > are
> > > > > > > > >> >> >>> > > distinct from scan/commit metrics.
> > > > > > > > >> >> >>> > >
> > > > > > > > >> >> >>> > > 3. I'd prefer separating metrics persistence
> from
> > > > > > MetaStore
> > > > > > > > >> >> >>> persistence
> > > > > > > > >> >> >>> > at
> > > > > > > > >> >> >>> > > the code level, so that they could be mixed and
> > > > matched
> > > > > > > > >> >> >>> independently.
> > > > > > > > >> >> >>> > The
> > > > > > > > >> >> >>> > > separate datasource question will become a
> > > non-issue
> > > > > with
> > > > > > > > that
> > > > > > > > >> >> >>> approach,
> > > > > > > > >> >> >>> > I
> > > > > > > > >> >> >>> > > guess.
> > > > > > > > >> >> >>> > >
> > > > > > > > >> >> >>> > > The rationale for separating scan metrics and
> > > > metastore
> > > > > > > > >> >> persistence
> > > > > > > > >> >> >>> is
> > > > > > > > >> >> >>> > that
> > > > > > > > >> >> >>> > > "cascading deletes" between them are hardly
> ever
> > > > > > required.
> > > > > > > > >> >> >>> Furthermore,
> > > > > > > > >> >> >>> > the
> > > > > > > > >> >> >>> > > data and query patterns are very different so
> > > > different
> > > > > > > > >> >> technologies
> > > > > > > > >> >> >>> > might
> > > > > > > > >> >> >>> > > be beneficial in each case.
> > > > > > > > >> >> >>> > >
> > > > > > > > >> >> >>> > > [1]
> > > > > > > > >> >> >>> >
> > > > > > > > >> >> >>>
> > > > > > > > >> >>
> > > > > > > > >>
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://urldefense.com/v3/__https://lists.apache.org/thread/yto2wp982t43h1mqjwnslswhws5z47cy__;!!Iz9xO38YGHZK!5EuyFFkk3vhRWVIRvQAWBSQfpJkTMA9HxugzDwXmN0LPPqhEFxYkFRGVhtb8AqUwXtDh2OplcMnbMDHKxYDakNU$
> > > > > > > > >> >> >>> > >
> > > > > > > > >> >> >>> > > Cheers,
> > > > > > > > >> >> >>> > > Dmitri.
> > > > > > > > >> >> >>> > >
> > > > > > > > >> >> >>> > > On Fri, Feb 27, 2026 at 6:19 PM Anand Kumar
> > > Sankaran
> > > > > via
> > > > > > > dev
> > > > > > > > <
> > > > > > > > >> >> >>> > > dev@polaris.apache.org> wrote:
> > > > > > > > >> >> >>> > >
> > > > > > > > >> >> >>> > > > Thanks all. This PR is merged now.
> > > > > > > > >> >> >>> > > >
> > > > > > > > >> >> >>> > > > Here are the follow-up features / work
> needed.
> > > > These
> > > > > > > were
> > > > > > > > >> all
> > > > > > > > >> >> >>> part of
> > > > > > > > >> >> >>> > > the
> > > > > > > > >> >> >>> > > > merged PR at some point in time and were
> > removed
> > > to
> > > > > > > reduce
> > > > > > > > >> >> scope.
> > > > > > > > >> >> >>> > > >
> > > > > > > > >> >> >>> > > > Please let me know what you think.
> > > > > > > > >> >> >>> > > >
> > > > > > > > >> >> >>> > > >
> > > > > > > > >> >> >>> > > >   1.  A REST API to paginate through table
> > > metrics.
> > > > > > This
> > > > > > > > >> will be
> > > > > > > > >> >> >>> > non-IRC
> > > > > > > > >> >> >>> > > > standard addition.
> > > > > > > > >> >> >>> > > >   2.  Utilities for managing old records,
> > should
> > > be
> > > > > > > common
> > > > > > > > >> with
> > > > > > > > >> >> >>> events.
> > > > > > > > >> >> >>> > > > There was some discussion that it belongs to
> > the
> > > > CLI.
> > > > > > > > >> >> >>> > > >   3.  Separate datasource (metrics, events,
> > even
> > > > > other
> > > > > > > > >> tables?).
> > > > > > > > >> >> >>> > > >
> > > > > > > > >> >> >>> > > >
> > > > > > > > >> >> >>> > > > Anything else?
> > > > > > > > >> >> >>> > > >
> > > > > > > > >> >> >>> > > > -
> > > > > > > > >> >> >>> > > > Anand
> > > > > > > > >> >> >>> > > >
> > > > > > > > >> >> >>> > > >
> > > > > > > > >> >> >>> > >
> > > > > > > > >> >> >>> >
> > > > > > > > >> >> >>> >
> > > > > > > > >> >> >>>
> > > > > > > > >> >> >>
> > > > > > > > >> >>
> > > > > > > > >> >
> > > > > > > > >>
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>

Re: Proposal for REST endpoints for table metrics and events

[EJ Wang <ej...@gmail.com>]

Hi Yufei,

Thanks for connecting this back to the REST endpoint proposal. I replied
with the fuller version on the event-forwarding thread, but wanted to add
the shorter version here too.

I agree that metrics reporting/emitting is the right conceptual boundary,
and I also think the event/listener path is a good implementation direction
to explore. The distinction I want to keep clear is default battery vs
extension implementation.

For the REST/API side, I would keep the semantics narrow: Polaris resolves
and authorizes the table, accepts the Iceberg scan/commit metrics report
into the configured reporting/emitting path, and returns 204. That 204
should mean Polaris accepted the report into the ingestion path; it should
not imply durable storage.

Durable storage, event forwarding, external telemetry routing, filtering,
and retention should sit behind that reporting/emitting boundary as
implementation-layer behavior. A durable JDBC path can be one named
extension implementation. An event/listener forwarding path can be another
named extension implementation.

So I am +1 on the event-forwarding idea as a non-default extension
implementation. I just would not make it the default battery or the core
API shape. The default battery can stay small and safe, while deployments
that need forwarding, dashboards, or durable storage can opt into the
implementation that matches their operational model.

Thanks,
-ej

On Wed, Jun 10, 2026 at 11:43 AM Yufei Gu <fl...@gmail.com> wrote:

> Thanks EJ. I agree that the reporting/emitting boundary feels like the
> right SPI boundary, but I wonder if we can simplify this even further.
>
> Iceberg scan and commit metrics look very similar to events to me. They are
> append only, asynchronously consumed, often forwarded to external systems,
> and have similar retention and cleanup requirements. More details can be
> found in this thread [1]. In that case, Polaris may only need to accept the
> report and emit an event through the existing event framework. Filtering,
> async delivery, custom sinks, and retention mechanisms could then be shared
> instead of introducing a separate metrics specific extension layer and SPI.
>
> The main requirement I still see is querying metrics through Polaris. If we
> want that battery included experience, we could work on a persistence
> listener implementation that selectively saves metrics and expose it via
> the IRC event endpoint which seems pretty close in Iceberg
> community. Another option I like more is to allow users to develop their
> own listener, so they can persist scan/commit metrics to any systems they
> prefer. In general, that feels like an implementation choice on top of the
> event framework rather than something that needs to shape the core
> architecture.
>
> 1. https://lists.apache.org/thread/x9j8nscvy8hq61tyn01mj8yp6n9of0kp
>
> Thanks,
> Yufei
>
>
> On Thu, Jun 4, 2026 at 5:14 PM EJ Wang <ej...@gmail.com>
> wrote:
>
> > Good point. I agree the existing PolarisMetricsReporter is already very
> > close to the right conceptual boundary. I am not proposing a second
> > parallel reporter concept. The distinction I am trying to make is
> narrower:
> >
> > Current state:
> >
> >    - PolarisMetricsReporter is the existing metrics reporting hook.
> >    - But it currently lives under runtime/service.
> >    - Its contract is documented around Quarkus/CDI discovery and
> > @Identifier
> >    selection.
> >    - The default implementation is log-only.
> >    - The durable implementation writes through PolarisMetricsManager and
> >    MetricsPersistence.
> >    - MetricsPersistence is currently inherited by BasePersistence.
> >
> > My proposal:
> >
> >    - Keep the reporting/emitting boundary as the SPI.
> >    - Revise that contract in a framework-agnostic optional Iceberg
> *metrics
> >    extension API layer*.
> >    - Keep runtime/service responsible for REST ingestion, table/authz
> >    validation, and runtime wiring.
> >    - Keep the battery implementation log-only in the metrics extension
> API
> >    layer, *not under runtime/service*.
> >    - Treat durable JDBC metrics as one implementation of the reporting
> SPI.
> >    - Decompose the MetricsPersistence and BasePersistence coupling as
> >    durable implementation detail.
> >
> > So the difference is not “new reporter concept vs old reporter concept.”
> > but:
> >
> >
> >    - PolarisMetricsReporter today = runtime/service CDI-shaped hook.
> >    - Target reporting SPI = same conceptual boundary, but placed in the
> >    right extension API layer and kept framework-agnostic.
> >
> > For the current metrics PR, I think the minimal architectural cleanup
> > before merge is to *put the reporting SPI contract and the battery
> default
> > in the right place*. The durable JDBC implementation, event-backed async
> > implementation, external queue support, and deeper persistence cleanup
> can
> > continue as follow-ups, as long as we do not lock the SPI boundary to
> > BasePersistence or to runtime/service wiring.
> >
> > -ej
> >
> > On Wed, Jun 3, 2026 at 3:00 PM Dmitri Bourlatchkov <di...@apache.org>
> > wrote:
> >
> > > Hi EJ,
> > >
> > > Thanks for the recap / summary!
> > >
> > > > Do folks agree that the stable SPI boundary should be
> > > metrics reporting/emitting, not metrics persistence?
> > >
> > > SGTM.
> > >
> > > > Does an optional Iceberg metrics extension API layer sound like the
> > right
> > > home for this SPI?
> > >
> > > How is that different from current PolarisMetricsReporter?
> > >
> > > > Should the current durable metrics work be reframed as a durable
> > > JDBC reference implementation of that SPI?
> > >
> > > SGTM.
> > >
> > > > What is the smallest PR sequence to get there without blocking
> > > the current metrics work unnecessarily?
> > >
> > > Let's get https://github.com/apache/polaris/pull/4397 merged first.
> > >
> > > Then, I'd propose isolating the Metrics schema from the MetaStore
> schema.
> > > This will probably have some ripple effect into the bootstrap
> workflows,
> > so
> > > it's not a trivial change.
> > >
> > > Then, let's reassess.
> > >
> > > Cheers,
> > > Dmitri.
> > >
> > > On Wed, Jun 3, 2026 at 5:40 PM EJ Wang <ej.wang.dev.interest@gmail.com
> >
> > > wrote:
> > >
> > > > Hi folks,
> > > >
> > > > Thanks again for the discussion today. I updated the sync doc with
> > notes
> > > > from the third metrics architecture sync:
> > > >
> > > >
> > > >
> > >
> >
> https://docs.google.com/document/d/100h7c4damrUzVuquYbBHM0EvA4LSWuW2IT2dN_7nYVA/edit?tab=t.k96s2xyqr5u1
> > > >
> > > > A few highlights from today’s discussion:
> > > >
> > > >    - We clarified that Iceberg metrics reporting can be interpreted
> > > either
> > > >    as sync or async handling from Polaris’s perspective, so Polaris
> > > should
> > > >    stay flexible as a platform instead of baking one handling model
> > into
> > > > the
> > > >    REST API behavior.
> > > >    - We aligned that the built-in (battery) behavior for metrics
> > emitting
> > > >    can stay simple: no-op/log-only is enough as the default.
> > > >    - Durable metrics persistence should be treated as an
> implementation
> > > of
> > > >    the metrics reporting path, not as the core SPI boundary.
> > > >    - The existing durable metrics work can be reviewed as a reference
> > > >    implementation of the reporting SPI, with persistence-related
> logic
> > > kept
> > > >    self-contained under a metrics durable implementation module
> rather
> > > than
> > > >    scattered through core entity persistence.
> > > >    - Dashboard/insights remains a real use case, but we agreed to
> keep
> > it
> > > >    separate from the core metrics intake discussion for now.
> > > >
> > > > I also did a quick source check after the meeting to make sure we are
> > > > describing the current state accurately.
> > > >
> > > > Current state:
> > > >
> > > >    - Polaris already has a metrics reporting hook:
> > > PolarisMetricsReporter.
> > > >    - The default implementation is DefaultMetricsReporter, selected
> by
> > > >    polaris.iceberg-metrics.reporting.type=default. It is log-only,
> > > >    effectively quiet unless metrics logging is enabled.
> > > >    - There is also a PersistingMetricsReporter, selected by
> > > >    polaris.iceberg-metrics.reporting.type=persisting, which converts
> > > >    Iceberg scan/commit reports into Polaris metrics records and
> writes
> > > > through
> > > >    PolarisMetricsManager -> MetricsPersistence.
> > > >    - MetricsPersistence currently exists in the persistence layer and
> > is
> > > >    inherited by BasePersistence.
> > > >
> > > > My read from the discussion is that the target boundary should not be
> > > > MetricsPersistence as inherited by BasePersistence. That path should
> be
> > > > decomposed as durable implementation detail (taken care of by
> > > > https://github.com/apache/polaris/pull/4397). The stable extension
> > point
> > > > should instead be the metrics reporting/emitting boundary.
> > > >
> > > > Concretely, I think the next proposal should be shaped like this:
> > > >
> > > >    1.
> > > >
> > > >    Define the proper metrics reporting SPI at an optional Iceberg
> > metrics
> > > >    extension API layer.
> > > >
> > > >    Example direction:
> > > >
> > > >    public interface IcebergMetricsReporter {
> > > >      void reportMetric(IcebergMetricsReportContext context,
> > > > MetricsReport report);
> > > >    }
> > > >
> > > >    The context would carry the small Polaris-resolved envelope:
> > > >    catalog/table identity, report type, received timestamp, and
> > > > request/trace
> > > >    context if available. The raw Iceberg MetricsReport remains the
> > > payload.
> > > >    2.
> > > >
> > > >    Keep runtime/service as the REST ingestion and wiring layer.
> > > >
> > > >    The REST handler still resolves the table and performs authz
> before
> > > >    accepting the report. After that, it calls the selected reporting
> > > >    implementation.
> > > >    3.
> > > >
> > > >    Keep the battery default no-op/log-only.
> > > >
> > > >    This preserves an out-of-box safe default and avoids requiring a
> > > durable
> > > >    metrics store for every Polaris deployment.
> > > >    4.
> > > >
> > > >    Move durable JDBC metrics into a self-contained implementation of
> > the
> > > >    reporting SPI.
> > > >
> > > >    That implementation can own its schema, bootstrap, retention, and
> > read
> > > >    API support. It should not define the core reporting SPI boundary,
> > and
> > > > it
> > > >    should not require metrics persistence to remain inherited from
> > > >    BasePersistence.
> > > >    5.
> > > >
> > > >    Treat async/event-backed handling as another implementation of the
> > > same
> > > >    reporting SPI.
> > > >
> > > >    For example, an event-backed reporter could enqueue a metrics
> event
> > > and
> > > >    let listeners handle durable storage or other sinks. If we later
> > need
> > > a
> > > >    replaceable queue engine, that seems like a shared event/metrics
> > > > substrate
> > > >    topic rather than a metrics-only requirement.
> > > >
> > > > This framing lets us keep the REST metrics endpoint simple, preserve
> > the
> > > > current default behavior, support durable metrics users, and still
> > leave
> > > > room for async/event-backed or external-queue-based implementations.
> > > >
> > > > I think the main follow-up questions are:
> > > >
> > > >    - Do folks agree that the stable SPI boundary should be metrics
> > > >    reporting/emitting, not metrics persistence?
> > > >    - Does an optional Iceberg metrics extension API layer sound like
> > the
> > > >    right home for this SPI?
> > > >    - Should the current durable metrics work be reframed as a durable
> > > JDBC
> > > >    reference implementation of that SPI?
> > > >    - What is the smallest PR sequence to get there without blocking
> the
> > > >    current metrics work unnecessarily?
> > > >
> > > > Thanks,
> > > > -ej
> > > >
> > > > On Fri, May 15, 2026 at 5:16 PM Dmitri Bourlatchkov <
> dimas@apache.org>
> > > > wrote:
> > > >
> > > > > Hi JB,
> > > > >
> > > > > Could you set up another meeting, please? Same time on Wednesday as
> > > last
> > > > > time... I hope it works for everyone.
> > > > >
> > > > > Cheers,
> > > > > Dmitri.
> > > > >
> > > > > On Fri, May 15, 2026 at 8:06 PM Yufei Gu <fl...@gmail.com>
> > wrote:
> > > > >
> > > > > > +1 on another sync call next week.
> > > > > >
> > > > > > Yufei
> > > > > >
> > > > > >
> > > > > > On Fri, May 15, 2026 at 4:52 PM Dmitri Bourlatchkov <
> > > dimas@apache.org>
> > > > > > wrote:
> > > > > >
> > > > > > > Hi All,
> > > > > > >
> > > > > > > WDYT about another sync call next week?
> > > > > > >
> > > > > > > Thanks,
> > > > > > > Dmitri.
> > > > > > >
> > > > > > > On Wed, May 6, 2026 at 5:29 PM Dmitri Bourlatchkov <
> > > dimas@apache.org
> > > > >
> > > > > > > wrote:
> > > > > > >
> > > > > > > > Hi EJ,
> > > > > > > >
> > > > > > > > Thanks for the summary! It covers what we discussed in the
> > > meeting
> > > > > very
> > > > > > > > well, IMHO.
> > > > > > > >
> > > > > > > > Looking forward to concrete PRs :)
> > > > > > > >
> > > > > > > > Cheers,
> > > > > > > > Dmitri.
> > > > > > > >
> > > > > > > > On Wed, May 6, 2026 at 5:08 PM EJ Wang <
> > > > > ej.wang.dev.interest@gmail.com
> > > > > > >
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > >> Hi folks,
> > > > > > > >>
> > > > > > > >> We had a community sync earlier, thanks JB for scheduling
> it.
> > > > Notes
> > > > > > from
> > > > > > > >> the first metrics architecture sync (May 6, 10-11am PT).
> > > > Discussion
> > > > > > doc
> > > > > > > >> with per-section status:
> > > > > > > >>
> > > > > > > >>
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://docs.google.com/document/d/100h7c4damrUzVuquYbBHM0EvA4LSWuW2IT2dN_7nYVA/edit?tab=t.0
> > > > > > > >>
> > > > > > > >> *The meeting covered both topics from the doc.
> Direction-level
> > > > > > alignment
> > > > > > > >> was reached on the headline pieces; details remain for PR
> > review
> > > > or
> > > > > > > >> follow-up sessions.*
> > > > > > > >>
> > > > > > > >> *Topic 1 — Persistence schema redesign*
> > > > > > > >> Idea-level alignment on consolidating per-type tables
> > > > > > > >> (scan_metrics_report,
> > > > > > > >> commit_metrics_report) into a single metrics_report table.
> The
> > > > > > > motivating
> > > > > > > >> cost is the surface area added by every new metric type
> today:
> > > new
> > > > > > > table,
> > > > > > > >> SPI method, record class, model, converter, schema
> migration.
> > > > > > > >>
> > > > > > > >> Most schema details are deferred to the schema PR. A few
> > > specific
> > > > > > points
> > > > > > > >> came up:
> > > > > > > >> •   metric_schema_version: Yufei prefers dropping it, since
> > > there
> > > > is
> > > > > > no
> > > > > > > >> spec-level concept of metrics versioning today and it is
> hard
> > to
> > > > > > define
> > > > > > > >> unilaterally. Robert prefers keeping it, given IRC v2 is
> > coming
> > > > and
> > > > > > the
> > > > > > > >> schema should be considered against its likely shape; Robert
> > > also
> > > > > > raised
> > > > > > > >> how to differentiate various payload formats if any. EJ's
> read
> > > is
> > > > > that
> > > > > > > >> this
> > > > > > > >> is a two-way-door decision. We can start without the field,
> > and
> > > if
> > > > > IRC
> > > > > > > v2
> > > > > > > >> changes the shape we would likely roll a corresponding new
> > > schema
> > > > > > > anyway,
> > > > > > > >> which is not particularly costly.
> > > > > > > >> •   Payload format: Robert pointed out that future formats
> > > beyond
> > > > > JSON
> > > > > > > may
> > > > > > > >> be worth supporting. The exact shape is deferred to the
> schema
> > > > > > > discussion.
> > > > > > > >> •   Partition strategy: Anand suggested monthly partitioning
> > > based
> > > > > on
> > > > > > > his
> > > > > > > >> experience as potentially helpful at scale.
> > > > > > > >>
> > > > > > > >> *Topic 2 — Where metrics ingestion and storage belong*
> > > > > > > >> Idea-level alignment that metrics should be a separated SPI
> > from
> > > > the
> > > > > > > >> entity
> > > > > > > >> persistence stack. Two reasons surfaced: (a) workloads and
> > > > > capability
> > > > > > > >> requirements diverge enough that coupling them creates
> > > artificial
> > > > > > > >> constraints, and (b) admin experience improves when metrics
> > has
> > > > its
> > > > > > own
> > > > > > > >> bootstrap, retention, and lifecycle. Dmitri noted Polaris
> > being
> > > a
> > > > > > > platform
> > > > > > > >> should have the flexibility to support different persistence
> > > > > backends
> > > > > > > per
> > > > > > > >> concern, and pointed to a concrete next step of separating
> the
> > > > JDBC
> > > > > > > >> bootstrap for metrics from the metastore bootstrap. Robert
> > > > proposed
> > > > > an
> > > > > > > >> additional UX extension: detect an unbootstrapped metrics
> > store
> > > on
> > > > > > first
> > > > > > > >> use and auto-bootstrap rather than requiring an explicit
> > manual
> > > > > > > bootstrap
> > > > > > > >> step.
> > > > > > > >> The meeting also confirmed that Polaris metrics can start
> > small
> > > > and
> > > > > > stay
> > > > > > > >> Iceberg-focused. Naming and persistence schema can lean
> > > > > > > Iceberg-specific.
> > > > > > > >> If a future expansion to generic-table metrics or
> operational
> > > > > metrics
> > > > > > > >> arrives, an abstraction layer can be built on top of the
> > Iceberg
> > > > > > metrics
> > > > > > > >> reporter at that point. Robert remains on the fence and
> would
> > > > prefer
> > > > > > > >> something more generic but did not block the direction;
> > Dmitri's
> > > > > read
> > > > > > > was
> > > > > > > >> that the proposed framework already has enough flexibility
> to
> > > > absorb
> > > > > > > >> future
> > > > > > > >> expansion.
> > > > > > > >>
> > > > > > > >> The Trade-offs and Proposed structure sections in the doc
> were
> > > not
> > > > > > > >> reviewed
> > > > > > > >> in detail. They remain open for either the next sync or PR
> > > review.
> > > > > > > >>
> > > > > > > >> *Cross-cutting alignment — battery-included plus pluggable*
> > > > > > > >> A common philosophy emerged from the discussion. EJ
> summarized
> > > it
> > > > > as:
> > > > > > > >> Polaris should provide a battery-included UX for beginners
> and
> > > the
> > > > > > > >> flexibility for advanced users to swap the included battery
> > for
> > > > > > > something
> > > > > > > >> more powerful or tailored to their use case. The SPI design
> > > needs
> > > > to
> > > > > > > >> enable
> > > > > > > >> both.
> > > > > > > >>
> > > > > > > >> The inputs that shaped this framing:
> > > > > > > >> •   Anand described how his team uses the current metrics
> > > > > persistence
> > > > > > > >> (three metrics consumers in v1.4).
> > > > > > > >> •   Yufei raised Grafana and dashboard integrations as a
> > > > destination
> > > > > > use
> > > > > > > >> case beyond the default.
> > > > > > > >> •   Robert called out that the current design is more
> > > > JDBC-focused.
> > > > > > > >>
> > > > > > > >> Two concrete instances:
> > > > > > > >> •   Async metrics intake: Yufei's initial position was that
> > > async
> > > > > > should
> > > > > > > >> largely live on the producer side and there is not much
> > Polaris
> > > > can
> > > > > > do.
> > > > > > > >> Robert suggested a Polaris-side default is doable via
> Vert.x.
> > > > Dmitri
> > > > > > > >> agreed
> > > > > > > >> the direction is worth exploring. The meeting converged on a
> > > > > > > >> battery-included default (likely Vert.x-backed) with an SPI
> > > shape
> > > > > that
> > > > > > > >> lets
> > > > > > > >> power users route to a more scalable backend (k8s-hosted
> > queue,
> > > > AWS
> > > > > > SQS,
> > > > > > > >> etc.).
> > > > > > > >> •   Pluggable destinations: combining Yufei's dashboard use
> > case
> > > > > with
> > > > > > > >> Robert's JDBC-focused call-out, the meeting agreed the SPI
> > > should
> > > > be
> > > > > > > >> structured for multiple sinks so integrations become impl
> > > choices
> > > > > > rather
> > > > > > > >> than architectural changes.
> > > > > > > >>
> > > > > > > >> The battery-included default is most likely to use the
> > existing
> > > > > > > >> JDBC-backed
> > > > > > > >> approach.
> > > > > > > >>
> > > > > > > >> *Direction (idea-level alignment)*
> > > > > > > >> •   Single metrics_report table consolidating per-type
> > metrics,
> > > > > > > replacing
> > > > > > > >> scan_metrics_report and commit_metrics_report
> > > > > > > >> •   Iceberg-focused naming and schema for now, revisit if
> > > > > > generic-table
> > > > > > > or
> > > > > > > >> operational metrics arrive
> > > > > > > >> •   Metrics persistence as a separated SPI, not on
> > > BasePersistence
> > > > > > > >> •   Bootstrap path separated for metrics, independent of
> > > metastore
> > > > > > > >> bootstrap
> > > > > > > >> •   "Battery-included plus pluggable" as the SPI design
> > > philosophy
> > > > > > > >>
> > > > > > > >> *Open items*
> > > > > > > >> •   Schema details: metric_schema_version, payload format,
> IRC
> > > v2
> > > > > > > >> forward-compat shape
> > > > > > > >> •   SPI design details — full review either in the next sync
> > or
> > > in
> > > > > the
> > > > > > > >> corresponding PR
> > > > > > > >> •   Schema refactor PR ownership
> > > > > > > >>
> > > > > > > >> *Action items*
> > > > > > > >> •   EJ to take a first stab at the SPI design and
> potentially
> > > > > partner
> > > > > > > with
> > > > > > > >> Anand to incorporate the lessons learned from the existing
> > > > reporter
> > > > > > and
> > > > > > > >> persistence work.
> > > > > > > >> •   Schema refactor PR ownership is not yet decided. If
> anyone
> > > is
> > > > > > > >> interested in driving it, reply on this thread.
> > > > > > > >> •   JB to schedule the next sync, tentatively in two weeks.
> > > > > > > >>
> > > > > > > >> -ej
> > > > > > > >>
> > > > > > > >> On Mon, Apr 27, 2026 at 3:07 PM EJ Wang <
> > > > > > ej.wang.dev.interest@gmail.com
> > > > > > > >
> > > > > > > >> wrote:
> > > > > > > >>
> > > > > > > >> > Thanks Yufei for the +1.
> > > > > > > >> >
> > > > > > > >> > JB, could you help add a biweekly metrics architecture
> sync
> > to
> > > > the
> > > > > > > >> Polaris
> > > > > > > >> > community calendar? I'm thinking Thursdays at 9-10am PT,
> on
> > > the
> > > > > > > >> off-weeks
> > > > > > > >> > from the community meeting (starting May 7), 60 minutes.
> > > > > > > >> >
> > > > > > > >> > Here's a rough agenda to work through over the first few
> > > > sessions,
> > > > > > > >> grouped
> > > > > > > >> > by priority:
> > > > > > > >> >
> > > > > > > >> > *First: foundational direction*
> > > > > > > >> >
> > > > > > > >> > 1.  MetricsPersistence: public SPI or internal
> > implementation
> > > > > > detail?
> > > > > > > >> >    •   Marked @Beta, javadoc calls it a "Service Provider
> > > > > > Interface",
> > > > > > > >> but
> > > > > > > >> > only one consumer (JdbcBasePersistenceImpl), lives on
> > > > > > BasePersistence.
> > > > > > > >> If
> > > > > > > >> > demoted to a private helper inside a persisting reporter
> > impl,
> > > > > most
> > > > > > > >> > downstream design decisions become implementation details
> > > rather
> > > > > > than
> > > > > > > >> > contract questions.
> > > > > > > >> >
> > > > > > > >> > 2.  Persistence schema redesign
> > > > > > > >> >    •   Current two-table layout (scan_metrics_report,
> > > > > > > >> > commit_metrics_report) with ~25 flattened columns each.
> > Every
> > > > new
> > > > > > > metric
> > > > > > > >> > type requires a new table, SPI method, record class,
> model,
> > > > > > converter,
> > > > > > > >> and
> > > > > > > >> > schema migration. Direction to explore: single table with
> > > > > > metric_type
> > > > > > > >> enum,
> > > > > > > >> > schema_version, and JSON payload column.
> > > > > > > >> >
> > > > > > > >> > *Second: design details once direction is set*
> > > > > > > >> >
> > > > > > > >> > 3.  Partition key strategy
> > > > > > > >> >    •   Single-table design means scan metrics at scale
> will
> > > have
> > > > > > high
> > > > > > > >> > write concurrency per table. Schema needs to expose enough
> > > > > structure
> > > > > > > for
> > > > > > > >> > backends to shard by entity or time range.
> > > > > > > >> >
> > > > > > > >> > 4.  Read/write path consistency
> > > > > > > >> >    •   Writes go through PolarisMetricsManager on
> > > > > MetaStoreManager.
> > > > > > > >> Reads
> > > > > > > >> > bypass MetaStoreManager and go straight to
> BasePersistence,
> > > > > > excluding
> > > > > > > >> > non-JDBC backends from the read API.
> > > > > > > >> >
> > > > > > > >> > *Third: cleanup and alignment*
> > > > > > > >> >
> > > > > > > >> > 5.  PolarisMetricsReporter naming
> > > > > > > >> >    •   Only handles IRC (ScanReport/CommitReport), doesn't
> > > cover
> > > > > > > generic
> > > > > > > >> > tables or operational metrics. Name is broader than scope.
> > > > > > > >> >
> > > > > > > >> > 6.  PolarisMetricsManager facade passthrough
> > > > > > > >> >    •   Entire default method is
> > > > > > > >> callCtx.getMetaStore().writeScanReport().
> > > > > > > >> > Zero logic, passes Level 1 straight through to Level 3.
> Same
> > > > > > > >> anti-pattern
> > > > > > > >> > as PolarisEventManager.
> > > > > > > >> >
> > > > > > > >> > 7.  Iceberg community alignment
> > > > > > > >> >    •   Payload-type extension needs discussion on
> > dev@iceberg.
> > > > > > > >> obelix74's
> > > > > > > >> > Feb thread got zero replies. Needs a committer voice.
> > > > > > > >> >
> > > > > > > >> > Lets confirm prioritization in the first session.
> > > > > > > >> >
> > > > > > > >> > -ej
> > > > > > > >> >
> > > > > > > >> > On Tue, Apr 21, 2026 at 3:18 PM Yufei Gu <
> > > flyrain000@gmail.com>
> > > > > > > wrote:
> > > > > > > >> >
> > > > > > > >> >> Thanks everyone for continuing to drive this forward. I
> > agree
> > > > > that
> > > > > > > the
> > > > > > > >> >> problem is getting complex enough that a more structured
> > > > > discussion
> > > > > > > >> would
> > > > > > > >> >> help.
> > > > > > > >> >>
> > > > > > > >> >> +1 on setting up a biweekly sync for the metrics
> > > architecture.
> > > > > I’m
> > > > > > > >> happy
> > > > > > > >> >> to
> > > > > > > >> >> join.
> > > > > > > >> >>
> > > > > > > >> >> Yufei
> > > > > > > >> >>
> > > > > > > >> >>
> > > > > > > >> >> On Tue, Apr 21, 2026 at 2:34 PM EJ Wang <
> > > > > > > >> ej.wang.dev.interest@gmail.com>
> > > > > > > >> >> wrote:
> > > > > > > >> >>
> > > > > > > >> >> > Also, I've been looking more closely at the
> *persistence
> > > > schema
> > > > > > in
> > > > > > > >> the
> > > > > > > >> >> > current metrics work*, and I think there's a structural
> > > > > rigidity
> > > > > > > >> problem
> > > > > > > >> >> > worth raising before the shape gets locked in.
> > > > > > > >> >> >
> > > > > > > >> >> > Right now we have two separate tables
> > (scan_metrics_report
> > > > and
> > > > > > > >> >> > commit_metrics_report), each with ~25 flattened columns
> > > that
> > > > > > > directly
> > > > > > > >> >> > mirror the Iceberg report fields. The SPI follows the
> > same
> > > > > split:
> > > > > > > >> >> > writeScanReport and writeCommitReport as separate
> > methods,
> > > > with
> > > > > > > >> per-type
> > > > > > > >> >> > record classes, converters, and model objects. *The
> > > practical
> > > > > > cost:
> > > > > > > >> >> > adding a new metric type (operational metrics, for
> > example)
> > > > > > > requires
> > > > > > > >> a
> > > > > > > >> >> new
> > > > > > > >> >> > table, a new SPI method, a new record class, a new
> model
> > > > > class, a
> > > > > > > new
> > > > > > > >> >> > converter branch, and a schema migration*. That's a lot
> > of
> > > > > > surface
> > > > > > > >> area
> > > > > > > >> >> > for what should be "one more kind of metric."
> > > > > > > >> >> >
> > > > > > > >> >> > *My bias* would be toward a single metrics table with
> *a
> > > > typed
> > > > > > JSON
> > > > > > > >> >> > payload*. Something like: metric_type (enum),
> entity_id,
> > > > > > > >> >> > table_identifier, snapshot_id (nullable), received_ts,
> > > > > > > >> schema_version,
> > > > > > > >> >> and
> > > > > > > >> >> > a payload column for the metric-specific data. The
> > > > metric_type
> > > > > +
> > > > > > > >> >> > schema_version pair gives us a forward-compatible
> > contract
> > > > for
> > > > > > the
> > > > > > > >> >> payload
> > > > > > > >> >> > shape. Adding a new metric type becomes an enum value
> > and a
> > > > > > payload
> > > > > > > >> >> schema,
> > > > > > > >> >> > not a schema migration. One thing I think we need to be
> > > > > > deliberate
> > > > > > > >> >> about is
> > > > > > > >> >> > the partition key design. If all metric types land in
> one
> > > > > table,
> > > > > > > scan
> > > > > > > >> >> > metrics at scale (high concurrency, high frequency
> across
> > > > many
> > > > > > > >> tables)
> > > > > > > >> >> > could easily create hot partitions. We'd want the
> > > persistence
> > > > > > layer
> > > > > > > >> to
> > > > > > > >> >> be
> > > > > > > >> >> > able to shard by entity or time range, and that means
> the
> > > > > logical
> > > > > > > >> schema
> > > > > > > >> >> > needs to expose enough structure for backends to
> > partition
> > > > on.
> > > > > I
> > > > > > > >> don't
> > > > > > > >> >> > think the current flattened layout gives us that.
> > > > > > > >> >> >
> > > > > > > >> >> > This is getting complex enough that I don't think
> ad-hoc
> > > > PR/ML
> > > > > > > >> threads
> > > > > > > >> >> > will converge well. *Would people be open to a biweekly
> > > sync
> > > > > for
> > > > > > > >> metrics
> > > > > > > >> >> > architecture?* I think 30 minutes every two weeks with
> > > > > interested
> > > > > > > >> >> parties
> > > > > > > >> >> > would be enough to work through the schema, SPI shape,
> > and
> > > > read
> > > > > > API
> > > > > > > >> >> design
> > > > > > > >> >> > together. Happy to help set that up.
> > > > > > > >> >> >
> > > > > > > >> >> > -ej
> > > > > > > >> >> >
> > > > > > > >> >> > On Mon, Apr 20, 2026 at 2:19 PM EJ Wang <
> > > > > > > >> ej.wang.dev.interest@gmail.com
> > > > > > > >> >> >
> > > > > > > >> >> > wrote:
> > > > > > > >> >> >
> > > > > > > >> >> >> Reviewed #4115, left a comment on the code
> organization
> > > > side.
> > > > > > > >> >> >>
> > > > > > > >> >> >> One thing stood out: the metrics write path enters
> > through
> > > > > > > >> >> >> PolarisMetricsManager on MetaStoreManager, but the new
> > > read
> > > > > path
> > > > > > > >> >> bypasses
> > > > > > > >> >> >> MetaStoreManager entirely and goes straight to
> > > > BasePersistence
> > > > > > via
> > > > > > > >> >> >> callContext.getMetaStore(). That means the read API
> only
> > > > works
> > > > > > for
> > > > > > > >> >> backends
> > > > > > > >> >> >> that implement BasePersistence. NoSQL and remote
> > backends
> > > > > can't
> > > > > > > >> >> participate.
> > > > > > > >> >> >>
> > > > > > > >> >> >> Stepping back, I think the metrics subsystem is
> growing
> > > into
> > > > > > > >> something
> > > > > > > >> >> >> real (write + read + REST API + AuthZ + pagination)
> *but
> > > the
> > > > > > > >> >> persistence
> > > > > > > >> >> >> side is split across two layers in a way that's hard
> to
> > > > > > extend*. I
> > > > > > > >> put
> > > > > > > >> >> >> together two diagrams to show what I mean (my best
> > > effort).
> > > > > > > >> >> >>
> > > > > > > >> >> >> *Current state* (Diagram 1): three interfaces at three
> > > > > different
> > > > > > > >> >> levels.
> > > > > > > >> >> >> The engine-facing SPI (PolarisMetricsReporter) is
> clean.
> > > But
> > > > > > > >> >> >> PolarisMetricsManager on MetaStoreManager is a
> > passthrough
> > > > to
> > > > > > > >> >> >> MetricsPersistence on BasePersistence. The @Beta
> > > annotation
> > > > > and
> > > > > > > SPI
> > > > > > > >> >> javadoc
> > > > > > > >> >> >> are on the BasePersistence layer, while the actual
> > > extension
> > > > > > > points
> > > > > > > >> >> >> (PolarisMetricsReporter, PolarisMetricsManager) carry
> no
> > > > > > stability
> > > > > > > >> >> >> annotation. The write path goes through the
> > > MetaStoreManager
> > > > > > > layer,
> > > > > > > >> the
> > > > > > > >> >> >> read path doesn't.
> > > > > > > >> >> >>
> > > > > > > >> >> >> *What I envision* (Diagram 2): two SPIs at two levels.
> > > > > > > >> >> >> PolarisMetricsReporter stays as the engine-facing SPI.
> > > > > > > >> >> >> PolarisMetricsManager becomes the backend-facing SPI
> > with
> > > > both
> > > > > > > write
> > > > > > > >> >> and
> > > > > > > >> >> >> read methods at the MetaStoreManager level, where any
> > > > backend
> > > > > > > (JDBC,
> > > > > > > >> >> NoSQL,
> > > > > > > >> >> >> remote) can implement them. MetricsPersistence on
> > > > > > BasePersistence
> > > > > > > >> goes
> > > > > > > >> >> >> away. Where metrics actually land is an implementation
> > > > detail,
> > > > > > > not a
> > > > > > > >> >> core
> > > > > > > >> >> >> interface.
> > > > > > > >> >> >>
> > > > > > > >> >> >> *Minor naming thing*: PolarisMetricsReporter is
> broader
> > > than
> > > > > > what
> > > > > > > it
> > > > > > > >> >> >> actually handles. It only accepts Iceberg REST Catalog
> > > > metrics
> > > > > > > >> >> (ScanReport,
> > > > > > > >> >> >> CommitReport via MetricsReport). Generic table metrics
> > or
> > > > > > > >> operational
> > > > > > > >> >> >> metrics aren't in scope. Not blocking, but worth
> noting
> > if
> > > > the
> > > > > > > >> metrics
> > > > > > > >> >> >> surface expands.
> > > > > > > >> >> >>
> > > > > > > >> >> >> *Rough sketch of how to get there*:
> > > > > > > >> >> >>  1.  Add read methods to PolarisMetricsManager
> > > > > (listScanReports,
> > > > > > > >> >> >> listCommitReports) with default no-op, same as the
> > > existing
> > > > > > write
> > > > > > > >> >> methods.
> > > > > > > >> >> >> (Probably make PolarisMetricsManager more explicit on
> > > being
> > > > > > > Iceberg
> > > > > > > >> >> >> specific like package name or class name etc.)
> > > > > > > >> >> >>  2.  Wire MetricsReportsService through
> MetaStoreManager
> > > > > instead
> > > > > > > of
> > > > > > > >> >> >> callContext.getMetaStore().
> > > > > > > >> >> >>  3.  Extract metrics persistence from
> > > > JdbcBasePersistenceImpl
> > > > > > into
> > > > > > > >> its
> > > > > > > >> >> >> own class. That file carries ~7 responsibilities,
> > metrics
> > > > > being
> > > > > > > one
> > > > > > > >> of
> > > > > > > >> >> them.
> > > > > > > >> >> >>  4.  Remove MetricsPersistence from BasePersistence.
> > > > > > > >> >> >>
> > > > > > > >> >> >> *None of this needs to happen in #4115. But if the
> > > direction
> > > > > > makes
> > > > > > > >> >> sense,
> > > > > > > >> >> >> it would be good to align before the metrics surface
> > grows
> > > > > > > further.
> > > > > > > >> >> Curious
> > > > > > > >> >> >> what others think.*
> > > > > > > >> >> >>
> > > > > > > >> >> >> *My mental model note*: Level 1 MetaStoreManager;
> level
> > 2
> > > > > > > >> transactional
> > > > > > > >> >> >> persistence; level 3 base persistence
> > > > > > > >> >> >>
> > > > > > > >> >> >> Diagram 1
> > > > > > > >> >> >> <
> > > > > > > >> >>
> > > > > > > >>
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://www.plantuml.com/plantuml/uml/bLHDR-Cs4BthLmpIYupw0zbkKQ1r3M-S7Bp8xhhM7WCOb3IM65EaGD9EX2RzxHrHb4CxRelwa4YSDu_lpOVcnZ9jzvM8BBS2uGjQpJC3dtHMSekPtMk44IpsMgEqa5XcCOhCZikQQLP1pR8TAp2n3ILhmZDP20m0fcIvUkAoW2qJXd9z1bpToO9BX3WXu0ucy5rpgGPNm0nW5_epUWtm2Ue3pn3kMOFQmKntGZW0BYtgBSi8k5A2QMwybJNMIbFiGSR9QZc4nUqIvikStF0jHprua5C-amge42aNt3R0f5JaaoivdV2Pkqbx4hee4ymOkBh5BTiB-_uIeGeo8zL8rPsPl4DktdEiK1jkB1NdZCRbrSTecDe_mlHbF0wvBmCkaOH5_S8a_TTTKI6-nmCAkEw4LpxsZ-LbYLKQFKMNOgf_wuM7_bV9gOer5SYMMksBSWXFcbi49KNZXNLicwfe3TETC7gPdPqI7uBcHMb1RSzYq34c6PDUM9mn8HRsUTZEiDBve3NjVZumBj0U7SS37mGO7vcwtiK-_pU7U7L_f-digo9YbhSwIfMRwIITKGXbxdIUTCGF1SeCJxloKsU-3k9ddRbX1eDq1q_fx1JbBGT0glVyXimDuP4TQ5qpCAmnGEj2s_6n5mtn1z-97-63itFQZLPO1Ev2tu_WF7Ju-VPc0Skg5bYXxBhkY1xpD7EM_7fyflSpIsqMgVth5xhVr4eQxWQ8enaSAJQSG16yFSDuJ798rrcXr_3n-lfdk7icQjEBmFujL7AodiP_Y4Z7-YxvtZNs4zMgpNTl6tF8sglyPsmqchrjvQ-m-aP94r-TwCA2Ka8upPJZwtvSpoYCXkYMZU2NXvRMBfq9P3i3Le4VAZUAlUZ_oPKsxPgY0Q_BSKLkyr9bhQhQrJjo_x3TPlIB0DPjnMfcIoYP0QaYw1a0fTKDr8fB6ntNuvmoL1ZGkXa69Njh43zf9GiGxHQrA_jDYWRSzF5--WmTVrN97_Sm8LbLUy_lGBmLanJjFkDlGkRqjA_4tm00
> > > > > > > >> >> >
> > > > > > > >> >> >> :
> > > > > > > >> >> >>
> > > > > > > >> >> >> [image: image.png]
> > > > > > > >> >> >>
> > > > > > > >> >> >> Diagram 2
> > > > > > > >> >> >> <
> > > > > > > >> >>
> > > > > > > >>
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://www.plantuml.com/plantuml/uml/VLLDR-8m4BtdLupO2sWBLVU8AaGB7AXAbssGzb896SS9RXqxjKqBwkv_tt7iV43fdaZYDpFlpRmnOsE9jhjSH9PRmM31hERKm8scMsuPjJlDe0yheZDc8RR4iYWoBrmMH9CS2a9VICPYUy1OZN0YCy5Q0BCbYNhdCeEK28En8G8wCvbnoQ0R8_05Bc6bkLIz3X03p1zzH7zR-9ZfDquPt9C3qoNCX2yV4G2NbkcKu5jdgGJHt0GbZwnG6i-UP3TUpk5gM6Ldqke350eZUqzoCft3U9xWHvxoa5-7K4nF1J46EbEMafsmdrCBbQ44gVggy18IZrn_ph5asd1ZiIKdQSgueZvjXrQFSFrdC3YN-nXmBacxbGiYyLVxLaBtdhqn0LSzdBDhqQtQoOJeGyad3z0lUqnYgpGB6Ns8oVyta00Dy_WnX0tIOZ8v6SYxHll1TrH6aejAik-mh-AphVFCwSUQqFypElag5QRGFDjQKEd96K1P8QP41c9TzA_IIQyvdAWyv_RSiS3skb0_EzDDkK2v5xWF6MiGFlvhpFLcD2Dq2pml14gaF67eQkmd8gulDoC4kSOu6KVpkvlUJg1RTbWISU40RdBUUS_9XfRZ2dwxm_SW8LYFISgm_MnlDQ6M9P1gbKEc4X-2pH_FvJCkCqm9pbVjD6LrwdLeOrDWfOaqc8Wh9BE85oNKxkNQ6o4yGRy_Eae0G_G8tZv81d3bHDB23WOdisohVr3nh_j6lbSjbNaLRTc8UgtPbAU1J_tygOfZX9DWEJeHDvYx-qmSi5FgNLPZwHrHcUsncGQ5-skhUclpE5fo4ounpFauYrUbkU6ccfnxMvitwag4IyerhTxj8In_Oj1bDO4pQru674loYrGlULHLEGCjwJJ8gDoVZR8MxO4BT3IzRvIcAQKezC6xpziGnTyImrfEGyJI_OcKfgtxIvnTqFEMS17L9Z-jsARN5FmTheP7HtSdtOMT0B4GY2FYHXxgQmMtj2bRqiLFGapiVe1_QVKDrkqXcm83aFEXnMYCZ-xlyHy
> > > > > > > >> >> >
> > > > > > > >> >> >> :
> > > > > > > >> >> >> [image: image.png]
> > > > > > > >> >> >>
> > > > > > > >> >> >>  -ej
> > > > > > > >> >> >>
> > > > > > > >> >> >> On Wed, Apr 15, 2026 at 8:22 AM Dmitri Bourlatchkov <
> > > > > > > >> dimas@apache.org>
> > > > > > > >> >> >> wrote:
> > > > > > > >> >> >>
> > > > > > > >> >> >>> Hi All,
> > > > > > > >> >> >>>
> > > > > > > >> >> >>> Heads up: The current state of PR [4115] looks pretty
> > > solid
> > > > > to
> > > > > > > me.
> > > > > > > >> I
> > > > > > > >> >> >>> believe this PR is approaching a mergeable condition.
> > > > > > > >> >> >>>
> > > > > > > >> >> >>> Please post your reviews if you have any comments.
> > > > > > > >> >> >>>
> > > > > > > >> >> >>> [4115] https://github.com/apache/polaris/pull/4115
> > > > > > > >> >> >>>
> > > > > > > >> >> >>> Thanks,
> > > > > > > >> >> >>> Dmitri.
> > > > > > > >> >> >>>
> > > > > > > >> >> >>> On Tue, Mar 3, 2026 at 3:29 PM Anand Kumar Sankaran
> via
> > > > dev <
> > > > > > > >> >> >>> dev@polaris.apache.org> wrote:
> > > > > > > >> >> >>>
> > > > > > > >> >> >>> > Hi Yufei and Dmitri,
> > > > > > > >> >> >>> >
> > > > > > > >> >> >>> > Here is a proposal for the REST endpoints for
> metrics
> > > and
> > > > > > > events.
> > > > > > > >> >> >>> >
> > > > > > > >> >> >>> >
> https://github.com/apache/polaris/pull/3924/changes
> > > > > > > >> >> >>> >
> > > > > > > >> >> >>> > I did not see any precursors for raising a PR for
> > > > > proposals,
> > > > > > so
> > > > > > > >> >> trying
> > > > > > > >> >> >>> > this.  Please let me know what you think.
> > > > > > > >> >> >>> >
> > > > > > > >> >> >>> > -
> > > > > > > >> >> >>> > Anand
> > > > > > > >> >> >>> >
> > > > > > > >> >> >>> > From: Anand Kumar Sankaran <
> > anand.sankaran@workday.com
> > > >
> > > > > > > >> >> >>> > Date: Monday, March 2, 2026 at 10:25 AM
> > > > > > > >> >> >>> > To: dev@polaris.apache.org <dev@polaris.apache.org
> >
> > > > > > > >> >> >>> > Subject: Re: Polaris Telemetry and Audit Trail
> > > > > > > >> >> >>> >
> > > > > > > >> >> >>> > About the REST API, based on my use cases:
> > > > > > > >> >> >>> >
> > > > > > > >> >> >>> >
> > > > > > > >> >> >>> >   1.
> > > > > > > >> >> >>> > I want to be able to query commit metrics to track
> > > files
> > > > > > added
> > > > > > > /
> > > > > > > >> >> >>> removed
> > > > > > > >> >> >>> > per commit, along with record counts. The ingestion
> > > > > pipeline
> > > > > > > that
> > > > > > > >> >> >>> writes
> > > > > > > >> >> >>> > this data is owned by us and we are guaranteed to
> > write
> > > > > this
> > > > > > > >> >> >>> information
> > > > > > > >> >> >>> > for each write.
> > > > > > > >> >> >>> >   2.
> > > > > > > >> >> >>> > I want to be able to query scan metrics for read. I
> > > > > > understand
> > > > > > > >> >> clients
> > > > > > > >> >> >>> do
> > > > > > > >> >> >>> > not fulfill this requirement.
> > > > > > > >> >> >>> >   3.
> > > > > > > >> >> >>> > I want to be able to query the events table (events
> > are
> > > > > > > >> persisted) -
> > > > > > > >> >> >>> this
> > > > > > > >> >> >>> > may supersede #2, I am not sure yet.
> > > > > > > >> >> >>> >
> > > > > > > >> >> >>> > All this information is in the JDBC based
> persistence
> > > > model
> > > > > > and
> > > > > > > >> is
> > > > > > > >> >> >>> > persisted in the metastore. I currently don’t have
> a
> > > need
> > > > > to
> > > > > > > >> query
> > > > > > > >> >> >>> > prometheus or open telemetry. I do publish some
> > events
> > > to
> > > > > > > >> Prometheus
> > > > > > > >> >> >>> and
> > > > > > > >> >> >>> > they are forwarded to our dashboards elsewhere.
> > > > > > > >> >> >>> >
> > > > > > > >> >> >>> > About the CLI utilities, I meant the admin user
> > > > utilities.
> > > > > In
> > > > > > > >> one of
> > > > > > > >> >> >>> the
> > > > > > > >> >> >>> > earliest drafts of my proposal, Prashant mentioned
> > that
> > > > the
> > > > > > > >> metrics
> > > > > > > >> >> >>> tables
> > > > > > > >> >> >>> > can grow indefinitely and that a similar problem
> > exists
> > > > > with
> > > > > > > the
> > > > > > > >> >> events
> > > > > > > >> >> >>> > table as well. We discussed that cleaning up of old
> > > > records
> > > > > > > from
> > > > > > > >> >> both
> > > > > > > >> >> >>> > metrics tables and events tables can be done via a
> > CLI
> > > > > > utility.
> > > > > > > >> >> >>> >
> > > > > > > >> >> >>> > I see that Yufei has covered the discussion about
> > > > > > datasources.
> > > > > > > >> >> >>> >
> > > > > > > >> >> >>> > -
> > > > > > > >> >> >>> > Anand
> > > > > > > >> >> >>> >
> > > > > > > >> >> >>> >
> > > > > > > >> >> >>> >
> > > > > > > >> >> >>> > From: Yufei Gu <fl...@gmail.com>
> > > > > > > >> >> >>> > Date: Friday, February 27, 2026 at 9:54 PM
> > > > > > > >> >> >>> > To: dev@polaris.apache.org <dev@polaris.apache.org
> >
> > > > > > > >> >> >>> > Subject: Re: Polaris Telemetry and Audit Trail
> > > > > > > >> >> >>> >
> > > > > > > >> >> >>> > This Message Is From an External Sender
> > > > > > > >> >> >>> > This message came from outside your organization.
> > > > > > > >> >> >>> > Report Suspicious<
> > > > > > > >> >> >>> >
> > > > > > > >> >> >>>
> > > > > > > >> >>
> > > > > > > >>
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://us-phishalarm-ewt.proofpoint.com/EWT/v1/Iz9xO38YGHZK!YhNDZABkHi1B699ote2uMwpOZw8i0QMCGO2Szc-HshuABGhGvwPJcymE6G2oUUxtS8xDkSrtGTPm_I3QnVDHoLMk50m9v8z_nZKTkd-bnVUbreF1u0WnfV_X5eYevZl_$
> > > > > > > >> >> >>> > >
> > > > > > > >> >> >>> >
> > > > > > > >> >> >>> >
> > > > > > > >> >> >>> > As I mentioned in
> > > > > > > >> >> >>> >
> > > > > > > >> >> >>>
> > > > > > > >> >>
> > > > > > > >>
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://urldefense.com/v3/__https://github.com/apache/polaris/issues/3890__;!!Iz9xO38YGHZK!5EuyFFkk3vhRWVIRvQAWBSQfpJkTMA9HxugzDwXmN0LPPqhEFxYkFRGVhtb8AqUwXtDh2OplcMnbMDHKOxrvDU0$
> > > > > > > >> >> >>> ,
> > > > > > > >> >> >>> > supporting
> > > > > > > >> >> >>> > multiple data sources is not a trivial change. I
> > would
> > > > > > strongly
> > > > > > > >> >> >>> recommend
> > > > > > > >> >> >>> > starting with a design document to carefully
> evaluate
> > > the
> > > > > > > >> >> architectural
> > > > > > > >> >> >>> > implications and long term impact.
> > > > > > > >> >> >>> >
> > > > > > > >> >> >>> > A REST endpoint to query metrics seems reasonable
> > given
> > > > the
> > > > > > > >> current
> > > > > > > >> >> >>> JDBC
> > > > > > > >> >> >>> > based persistence model. That said, we may also
> > > consider
> > > > > > > >> alternative
> > > > > > > >> >> >>> > storage models. For example, if we later adopt a
> time
> > > > > series
> > > > > > > >> system
> > > > > > > >> >> >>> such as
> > > > > > > >> >> >>> > Prometheus to store metrics, the query model and
> > access
> > > > > > > patterns
> > > > > > > >> >> would
> > > > > > > >> >> >>> be
> > > > > > > >> >> >>> > fundamentally different. Designing the REST API
> > without
> > > > > > > >> considering
> > > > > > > >> >> >>> these
> > > > > > > >> >> >>> > potential evolutions may limit flexibility. I'd
> > suggest
> > > > to
> > > > > > > start
> > > > > > > >> >> with
> > > > > > > >> >> >>> the
> > > > > > > >> >> >>> > use case.
> > > > > > > >> >> >>> >
> > > > > > > >> >> >>> > Yufei
> > > > > > > >> >> >>> >
> > > > > > > >> >> >>> >
> > > > > > > >> >> >>> > On Fri, Feb 27, 2026 at 3:42 PM Dmitri
> Bourlatchkov <
> > > > > > > >> >> dimas@apache.org>
> > > > > > > >> >> >>> > wrote:
> > > > > > > >> >> >>> >
> > > > > > > >> >> >>> > > Hi Anand,
> > > > > > > >> >> >>> > >
> > > > > > > >> >> >>> > > Sharing my view... subject to discussion:
> > > > > > > >> >> >>> > >
> > > > > > > >> >> >>> > > 1. Adding non-IRC REST API to Polaris is
> perfectly
> > > > fine.
> > > > > > > >> >> >>> > >
> > > > > > > >> >> >>> > > Figuring out specific endpoint URIs and payloads
> > > might
> > > > > > > require
> > > > > > > >> a
> > > > > > > >> >> few
> > > > > > > >> >> >>> > > roundtrips, so opening a separate thread for that
> > > might
> > > > > be
> > > > > > > >> best.
> > > > > > > >> >> >>> > > Contributors commonly create Google Docs for new
> > API
> > > > > > > proposals
> > > > > > > >> too
> > > > > > > >> >> >>> (they
> > > > > > > >> >> >>> > > fairly easy to update as the email discussion
> > > > > progresses).
> > > > > > > >> >> >>> > >
> > > > > > > >> >> >>> > > There was a suggestion to try Markdown (with PRs)
> > for
> > > > > > > proposals
> > > > > > > >> >> [1]
> > > > > > > >> >> >>> ...
> > > > > > > >> >> >>> > > feel free to give it a try if you are comfortable
> > > with
> > > > > > that.
> > > > > > > >> >> >>> > >
> > > > > > > >> >> >>> > > 2. Could you clarify whether you mean end user
> > > > utilities
> > > > > or
> > > > > > > >> admin
> > > > > > > >> >> >>> user
> > > > > > > >> >> >>> > > utilities? In the latter case those might be more
> > > > > suitable
> > > > > > > for
> > > > > > > >> the
> > > > > > > >> >> >>> Admin
> > > > > > > >> >> >>> > > CLI (java) not the Python CLI, IMHO.
> > > > > > > >> >> >>> > >
> > > > > > > >> >> >>> > > Why would these utilities be common with events?
> > > IMHO,
> > > > > > event
> > > > > > > >> use
> > > > > > > >> >> >>> cases
> > > > > > > >> >> >>> > are
> > > > > > > >> >> >>> > > distinct from scan/commit metrics.
> > > > > > > >> >> >>> > >
> > > > > > > >> >> >>> > > 3. I'd prefer separating metrics persistence from
> > > > > MetaStore
> > > > > > > >> >> >>> persistence
> > > > > > > >> >> >>> > at
> > > > > > > >> >> >>> > > the code level, so that they could be mixed and
> > > matched
> > > > > > > >> >> >>> independently.
> > > > > > > >> >> >>> > The
> > > > > > > >> >> >>> > > separate datasource question will become a
> > non-issue
> > > > with
> > > > > > > that
> > > > > > > >> >> >>> approach,
> > > > > > > >> >> >>> > I
> > > > > > > >> >> >>> > > guess.
> > > > > > > >> >> >>> > >
> > > > > > > >> >> >>> > > The rationale for separating scan metrics and
> > > metastore
> > > > > > > >> >> persistence
> > > > > > > >> >> >>> is
> > > > > > > >> >> >>> > that
> > > > > > > >> >> >>> > > "cascading deletes" between them are hardly ever
> > > > > required.
> > > > > > > >> >> >>> Furthermore,
> > > > > > > >> >> >>> > the
> > > > > > > >> >> >>> > > data and query patterns are very different so
> > > different
> > > > > > > >> >> technologies
> > > > > > > >> >> >>> > might
> > > > > > > >> >> >>> > > be beneficial in each case.
> > > > > > > >> >> >>> > >
> > > > > > > >> >> >>> > > [1]
> > > > > > > >> >> >>> >
> > > > > > > >> >> >>>
> > > > > > > >> >>
> > > > > > > >>
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://urldefense.com/v3/__https://lists.apache.org/thread/yto2wp982t43h1mqjwnslswhws5z47cy__;!!Iz9xO38YGHZK!5EuyFFkk3vhRWVIRvQAWBSQfpJkTMA9HxugzDwXmN0LPPqhEFxYkFRGVhtb8AqUwXtDh2OplcMnbMDHKxYDakNU$
> > > > > > > >> >> >>> > >
> > > > > > > >> >> >>> > > Cheers,
> > > > > > > >> >> >>> > > Dmitri.
> > > > > > > >> >> >>> > >
> > > > > > > >> >> >>> > > On Fri, Feb 27, 2026 at 6:19 PM Anand Kumar
> > Sankaran
> > > > via
> > > > > > dev
> > > > > > > <
> > > > > > > >> >> >>> > > dev@polaris.apache.org> wrote:
> > > > > > > >> >> >>> > >
> > > > > > > >> >> >>> > > > Thanks all. This PR is merged now.
> > > > > > > >> >> >>> > > >
> > > > > > > >> >> >>> > > > Here are the follow-up features / work needed.
> > > These
> > > > > > were
> > > > > > > >> all
> > > > > > > >> >> >>> part of
> > > > > > > >> >> >>> > > the
> > > > > > > >> >> >>> > > > merged PR at some point in time and were
> removed
> > to
> > > > > > reduce
> > > > > > > >> >> scope.
> > > > > > > >> >> >>> > > >
> > > > > > > >> >> >>> > > > Please let me know what you think.
> > > > > > > >> >> >>> > > >
> > > > > > > >> >> >>> > > >
> > > > > > > >> >> >>> > > >   1.  A REST API to paginate through table
> > metrics.
> > > > > This
> > > > > > > >> will be
> > > > > > > >> >> >>> > non-IRC
> > > > > > > >> >> >>> > > > standard addition.
> > > > > > > >> >> >>> > > >   2.  Utilities for managing old records,
> should
> > be
> > > > > > common
> > > > > > > >> with
> > > > > > > >> >> >>> events.
> > > > > > > >> >> >>> > > > There was some discussion that it belongs to
> the
> > > CLI.
> > > > > > > >> >> >>> > > >   3.  Separate datasource (metrics, events,
> even
> > > > other
> > > > > > > >> tables?).
> > > > > > > >> >> >>> > > >
> > > > > > > >> >> >>> > > >
> > > > > > > >> >> >>> > > > Anything else?
> > > > > > > >> >> >>> > > >
> > > > > > > >> >> >>> > > > -
> > > > > > > >> >> >>> > > > Anand
> > > > > > > >> >> >>> > > >
> > > > > > > >> >> >>> > > >
> > > > > > > >> >> >>> > >
> > > > > > > >> >> >>> >
> > > > > > > >> >> >>> >
> > > > > > > >> >> >>>
> > > > > > > >> >> >>
> > > > > > > >> >>
> > > > > > > >> >
> > > > > > > >>
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>

Re: Proposal for REST endpoints for table metrics and events

[Yufei Gu <fl...@gmail.com>]

Thanks EJ. I agree that the reporting/emitting boundary feels like the
right SPI boundary, but I wonder if we can simplify this even further.

Iceberg scan and commit metrics look very similar to events to me. They are
append only, asynchronously consumed, often forwarded to external systems,
and have similar retention and cleanup requirements. More details can be
found in this thread [1]. In that case, Polaris may only need to accept the
report and emit an event through the existing event framework. Filtering,
async delivery, custom sinks, and retention mechanisms could then be shared
instead of introducing a separate metrics specific extension layer and SPI.

The main requirement I still see is querying metrics through Polaris. If we
want that battery included experience, we could work on a persistence
listener implementation that selectively saves metrics and expose it via
the IRC event endpoint which seems pretty close in Iceberg
community. Another option I like more is to allow users to develop their
own listener, so they can persist scan/commit metrics to any systems they
prefer. In general, that feels like an implementation choice on top of the
event framework rather than something that needs to shape the core
architecture.

1. https://lists.apache.org/thread/x9j8nscvy8hq61tyn01mj8yp6n9of0kp

Thanks,
Yufei


On Thu, Jun 4, 2026 at 5:14 PM EJ Wang <ej...@gmail.com>
wrote:

> Good point. I agree the existing PolarisMetricsReporter is already very
> close to the right conceptual boundary. I am not proposing a second
> parallel reporter concept. The distinction I am trying to make is narrower:
>
> Current state:
>
>    - PolarisMetricsReporter is the existing metrics reporting hook.
>    - But it currently lives under runtime/service.
>    - Its contract is documented around Quarkus/CDI discovery and
> @Identifier
>    selection.
>    - The default implementation is log-only.
>    - The durable implementation writes through PolarisMetricsManager and
>    MetricsPersistence.
>    - MetricsPersistence is currently inherited by BasePersistence.
>
> My proposal:
>
>    - Keep the reporting/emitting boundary as the SPI.
>    - Revise that contract in a framework-agnostic optional Iceberg *metrics
>    extension API layer*.
>    - Keep runtime/service responsible for REST ingestion, table/authz
>    validation, and runtime wiring.
>    - Keep the battery implementation log-only in the metrics extension API
>    layer, *not under runtime/service*.
>    - Treat durable JDBC metrics as one implementation of the reporting SPI.
>    - Decompose the MetricsPersistence and BasePersistence coupling as
>    durable implementation detail.
>
> So the difference is not “new reporter concept vs old reporter concept.”
> but:
>
>
>    - PolarisMetricsReporter today = runtime/service CDI-shaped hook.
>    - Target reporting SPI = same conceptual boundary, but placed in the
>    right extension API layer and kept framework-agnostic.
>
> For the current metrics PR, I think the minimal architectural cleanup
> before merge is to *put the reporting SPI contract and the battery default
> in the right place*. The durable JDBC implementation, event-backed async
> implementation, external queue support, and deeper persistence cleanup can
> continue as follow-ups, as long as we do not lock the SPI boundary to
> BasePersistence or to runtime/service wiring.
>
> -ej
>
> On Wed, Jun 3, 2026 at 3:00 PM Dmitri Bourlatchkov <di...@apache.org>
> wrote:
>
> > Hi EJ,
> >
> > Thanks for the recap / summary!
> >
> > > Do folks agree that the stable SPI boundary should be
> > metrics reporting/emitting, not metrics persistence?
> >
> > SGTM.
> >
> > > Does an optional Iceberg metrics extension API layer sound like the
> right
> > home for this SPI?
> >
> > How is that different from current PolarisMetricsReporter?
> >
> > > Should the current durable metrics work be reframed as a durable
> > JDBC reference implementation of that SPI?
> >
> > SGTM.
> >
> > > What is the smallest PR sequence to get there without blocking
> > the current metrics work unnecessarily?
> >
> > Let's get https://github.com/apache/polaris/pull/4397 merged first.
> >
> > Then, I'd propose isolating the Metrics schema from the MetaStore schema.
> > This will probably have some ripple effect into the bootstrap workflows,
> so
> > it's not a trivial change.
> >
> > Then, let's reassess.
> >
> > Cheers,
> > Dmitri.
> >
> > On Wed, Jun 3, 2026 at 5:40 PM EJ Wang <ej...@gmail.com>
> > wrote:
> >
> > > Hi folks,
> > >
> > > Thanks again for the discussion today. I updated the sync doc with
> notes
> > > from the third metrics architecture sync:
> > >
> > >
> > >
> >
> https://docs.google.com/document/d/100h7c4damrUzVuquYbBHM0EvA4LSWuW2IT2dN_7nYVA/edit?tab=t.k96s2xyqr5u1
> > >
> > > A few highlights from today’s discussion:
> > >
> > >    - We clarified that Iceberg metrics reporting can be interpreted
> > either
> > >    as sync or async handling from Polaris’s perspective, so Polaris
> > should
> > >    stay flexible as a platform instead of baking one handling model
> into
> > > the
> > >    REST API behavior.
> > >    - We aligned that the built-in (battery) behavior for metrics
> emitting
> > >    can stay simple: no-op/log-only is enough as the default.
> > >    - Durable metrics persistence should be treated as an implementation
> > of
> > >    the metrics reporting path, not as the core SPI boundary.
> > >    - The existing durable metrics work can be reviewed as a reference
> > >    implementation of the reporting SPI, with persistence-related logic
> > kept
> > >    self-contained under a metrics durable implementation module rather
> > than
> > >    scattered through core entity persistence.
> > >    - Dashboard/insights remains a real use case, but we agreed to keep
> it
> > >    separate from the core metrics intake discussion for now.
> > >
> > > I also did a quick source check after the meeting to make sure we are
> > > describing the current state accurately.
> > >
> > > Current state:
> > >
> > >    - Polaris already has a metrics reporting hook:
> > PolarisMetricsReporter.
> > >    - The default implementation is DefaultMetricsReporter, selected by
> > >    polaris.iceberg-metrics.reporting.type=default. It is log-only,
> > >    effectively quiet unless metrics logging is enabled.
> > >    - There is also a PersistingMetricsReporter, selected by
> > >    polaris.iceberg-metrics.reporting.type=persisting, which converts
> > >    Iceberg scan/commit reports into Polaris metrics records and writes
> > > through
> > >    PolarisMetricsManager -> MetricsPersistence.
> > >    - MetricsPersistence currently exists in the persistence layer and
> is
> > >    inherited by BasePersistence.
> > >
> > > My read from the discussion is that the target boundary should not be
> > > MetricsPersistence as inherited by BasePersistence. That path should be
> > > decomposed as durable implementation detail (taken care of by
> > > https://github.com/apache/polaris/pull/4397). The stable extension
> point
> > > should instead be the metrics reporting/emitting boundary.
> > >
> > > Concretely, I think the next proposal should be shaped like this:
> > >
> > >    1.
> > >
> > >    Define the proper metrics reporting SPI at an optional Iceberg
> metrics
> > >    extension API layer.
> > >
> > >    Example direction:
> > >
> > >    public interface IcebergMetricsReporter {
> > >      void reportMetric(IcebergMetricsReportContext context,
> > > MetricsReport report);
> > >    }
> > >
> > >    The context would carry the small Polaris-resolved envelope:
> > >    catalog/table identity, report type, received timestamp, and
> > > request/trace
> > >    context if available. The raw Iceberg MetricsReport remains the
> > payload.
> > >    2.
> > >
> > >    Keep runtime/service as the REST ingestion and wiring layer.
> > >
> > >    The REST handler still resolves the table and performs authz before
> > >    accepting the report. After that, it calls the selected reporting
> > >    implementation.
> > >    3.
> > >
> > >    Keep the battery default no-op/log-only.
> > >
> > >    This preserves an out-of-box safe default and avoids requiring a
> > durable
> > >    metrics store for every Polaris deployment.
> > >    4.
> > >
> > >    Move durable JDBC metrics into a self-contained implementation of
> the
> > >    reporting SPI.
> > >
> > >    That implementation can own its schema, bootstrap, retention, and
> read
> > >    API support. It should not define the core reporting SPI boundary,
> and
> > > it
> > >    should not require metrics persistence to remain inherited from
> > >    BasePersistence.
> > >    5.
> > >
> > >    Treat async/event-backed handling as another implementation of the
> > same
> > >    reporting SPI.
> > >
> > >    For example, an event-backed reporter could enqueue a metrics event
> > and
> > >    let listeners handle durable storage or other sinks. If we later
> need
> > a
> > >    replaceable queue engine, that seems like a shared event/metrics
> > > substrate
> > >    topic rather than a metrics-only requirement.
> > >
> > > This framing lets us keep the REST metrics endpoint simple, preserve
> the
> > > current default behavior, support durable metrics users, and still
> leave
> > > room for async/event-backed or external-queue-based implementations.
> > >
> > > I think the main follow-up questions are:
> > >
> > >    - Do folks agree that the stable SPI boundary should be metrics
> > >    reporting/emitting, not metrics persistence?
> > >    - Does an optional Iceberg metrics extension API layer sound like
> the
> > >    right home for this SPI?
> > >    - Should the current durable metrics work be reframed as a durable
> > JDBC
> > >    reference implementation of that SPI?
> > >    - What is the smallest PR sequence to get there without blocking the
> > >    current metrics work unnecessarily?
> > >
> > > Thanks,
> > > -ej
> > >
> > > On Fri, May 15, 2026 at 5:16 PM Dmitri Bourlatchkov <di...@apache.org>
> > > wrote:
> > >
> > > > Hi JB,
> > > >
> > > > Could you set up another meeting, please? Same time on Wednesday as
> > last
> > > > time... I hope it works for everyone.
> > > >
> > > > Cheers,
> > > > Dmitri.
> > > >
> > > > On Fri, May 15, 2026 at 8:06 PM Yufei Gu <fl...@gmail.com>
> wrote:
> > > >
> > > > > +1 on another sync call next week.
> > > > >
> > > > > Yufei
> > > > >
> > > > >
> > > > > On Fri, May 15, 2026 at 4:52 PM Dmitri Bourlatchkov <
> > dimas@apache.org>
> > > > > wrote:
> > > > >
> > > > > > Hi All,
> > > > > >
> > > > > > WDYT about another sync call next week?
> > > > > >
> > > > > > Thanks,
> > > > > > Dmitri.
> > > > > >
> > > > > > On Wed, May 6, 2026 at 5:29 PM Dmitri Bourlatchkov <
> > dimas@apache.org
> > > >
> > > > > > wrote:
> > > > > >
> > > > > > > Hi EJ,
> > > > > > >
> > > > > > > Thanks for the summary! It covers what we discussed in the
> > meeting
> > > > very
> > > > > > > well, IMHO.
> > > > > > >
> > > > > > > Looking forward to concrete PRs :)
> > > > > > >
> > > > > > > Cheers,
> > > > > > > Dmitri.
> > > > > > >
> > > > > > > On Wed, May 6, 2026 at 5:08 PM EJ Wang <
> > > > ej.wang.dev.interest@gmail.com
> > > > > >
> > > > > > > wrote:
> > > > > > >
> > > > > > >> Hi folks,
> > > > > > >>
> > > > > > >> We had a community sync earlier, thanks JB for scheduling it.
> > > Notes
> > > > > from
> > > > > > >> the first metrics architecture sync (May 6, 10-11am PT).
> > > Discussion
> > > > > doc
> > > > > > >> with per-section status:
> > > > > > >>
> > > > > > >>
> > > > > >
> > > > >
> > > >
> > >
> >
> https://docs.google.com/document/d/100h7c4damrUzVuquYbBHM0EvA4LSWuW2IT2dN_7nYVA/edit?tab=t.0
> > > > > > >>
> > > > > > >> *The meeting covered both topics from the doc. Direction-level
> > > > > alignment
> > > > > > >> was reached on the headline pieces; details remain for PR
> review
> > > or
> > > > > > >> follow-up sessions.*
> > > > > > >>
> > > > > > >> *Topic 1 — Persistence schema redesign*
> > > > > > >> Idea-level alignment on consolidating per-type tables
> > > > > > >> (scan_metrics_report,
> > > > > > >> commit_metrics_report) into a single metrics_report table. The
> > > > > > motivating
> > > > > > >> cost is the surface area added by every new metric type today:
> > new
> > > > > > table,
> > > > > > >> SPI method, record class, model, converter, schema migration.
> > > > > > >>
> > > > > > >> Most schema details are deferred to the schema PR. A few
> > specific
> > > > > points
> > > > > > >> came up:
> > > > > > >> •   metric_schema_version: Yufei prefers dropping it, since
> > there
> > > is
> > > > > no
> > > > > > >> spec-level concept of metrics versioning today and it is hard
> to
> > > > > define
> > > > > > >> unilaterally. Robert prefers keeping it, given IRC v2 is
> coming
> > > and
> > > > > the
> > > > > > >> schema should be considered against its likely shape; Robert
> > also
> > > > > raised
> > > > > > >> how to differentiate various payload formats if any. EJ's read
> > is
> > > > that
> > > > > > >> this
> > > > > > >> is a two-way-door decision. We can start without the field,
> and
> > if
> > > > IRC
> > > > > > v2
> > > > > > >> changes the shape we would likely roll a corresponding new
> > schema
> > > > > > anyway,
> > > > > > >> which is not particularly costly.
> > > > > > >> •   Payload format: Robert pointed out that future formats
> > beyond
> > > > JSON
> > > > > > may
> > > > > > >> be worth supporting. The exact shape is deferred to the schema
> > > > > > discussion.
> > > > > > >> •   Partition strategy: Anand suggested monthly partitioning
> > based
> > > > on
> > > > > > his
> > > > > > >> experience as potentially helpful at scale.
> > > > > > >>
> > > > > > >> *Topic 2 — Where metrics ingestion and storage belong*
> > > > > > >> Idea-level alignment that metrics should be a separated SPI
> from
> > > the
> > > > > > >> entity
> > > > > > >> persistence stack. Two reasons surfaced: (a) workloads and
> > > > capability
> > > > > > >> requirements diverge enough that coupling them creates
> > artificial
> > > > > > >> constraints, and (b) admin experience improves when metrics
> has
> > > its
> > > > > own
> > > > > > >> bootstrap, retention, and lifecycle. Dmitri noted Polaris
> being
> > a
> > > > > > platform
> > > > > > >> should have the flexibility to support different persistence
> > > > backends
> > > > > > per
> > > > > > >> concern, and pointed to a concrete next step of separating the
> > > JDBC
> > > > > > >> bootstrap for metrics from the metastore bootstrap. Robert
> > > proposed
> > > > an
> > > > > > >> additional UX extension: detect an unbootstrapped metrics
> store
> > on
> > > > > first
> > > > > > >> use and auto-bootstrap rather than requiring an explicit
> manual
> > > > > > bootstrap
> > > > > > >> step.
> > > > > > >> The meeting also confirmed that Polaris metrics can start
> small
> > > and
> > > > > stay
> > > > > > >> Iceberg-focused. Naming and persistence schema can lean
> > > > > > Iceberg-specific.
> > > > > > >> If a future expansion to generic-table metrics or operational
> > > > metrics
> > > > > > >> arrives, an abstraction layer can be built on top of the
> Iceberg
> > > > > metrics
> > > > > > >> reporter at that point. Robert remains on the fence and would
> > > prefer
> > > > > > >> something more generic but did not block the direction;
> Dmitri's
> > > > read
> > > > > > was
> > > > > > >> that the proposed framework already has enough flexibility to
> > > absorb
> > > > > > >> future
> > > > > > >> expansion.
> > > > > > >>
> > > > > > >> The Trade-offs and Proposed structure sections in the doc were
> > not
> > > > > > >> reviewed
> > > > > > >> in detail. They remain open for either the next sync or PR
> > review.
> > > > > > >>
> > > > > > >> *Cross-cutting alignment — battery-included plus pluggable*
> > > > > > >> A common philosophy emerged from the discussion. EJ summarized
> > it
> > > > as:
> > > > > > >> Polaris should provide a battery-included UX for beginners and
> > the
> > > > > > >> flexibility for advanced users to swap the included battery
> for
> > > > > > something
> > > > > > >> more powerful or tailored to their use case. The SPI design
> > needs
> > > to
> > > > > > >> enable
> > > > > > >> both.
> > > > > > >>
> > > > > > >> The inputs that shaped this framing:
> > > > > > >> •   Anand described how his team uses the current metrics
> > > > persistence
> > > > > > >> (three metrics consumers in v1.4).
> > > > > > >> •   Yufei raised Grafana and dashboard integrations as a
> > > destination
> > > > > use
> > > > > > >> case beyond the default.
> > > > > > >> •   Robert called out that the current design is more
> > > JDBC-focused.
> > > > > > >>
> > > > > > >> Two concrete instances:
> > > > > > >> •   Async metrics intake: Yufei's initial position was that
> > async
> > > > > should
> > > > > > >> largely live on the producer side and there is not much
> Polaris
> > > can
> > > > > do.
> > > > > > >> Robert suggested a Polaris-side default is doable via Vert.x.
> > > Dmitri
> > > > > > >> agreed
> > > > > > >> the direction is worth exploring. The meeting converged on a
> > > > > > >> battery-included default (likely Vert.x-backed) with an SPI
> > shape
> > > > that
> > > > > > >> lets
> > > > > > >> power users route to a more scalable backend (k8s-hosted
> queue,
> > > AWS
> > > > > SQS,
> > > > > > >> etc.).
> > > > > > >> •   Pluggable destinations: combining Yufei's dashboard use
> case
> > > > with
> > > > > > >> Robert's JDBC-focused call-out, the meeting agreed the SPI
> > should
> > > be
> > > > > > >> structured for multiple sinks so integrations become impl
> > choices
> > > > > rather
> > > > > > >> than architectural changes.
> > > > > > >>
> > > > > > >> The battery-included default is most likely to use the
> existing
> > > > > > >> JDBC-backed
> > > > > > >> approach.
> > > > > > >>
> > > > > > >> *Direction (idea-level alignment)*
> > > > > > >> •   Single metrics_report table consolidating per-type
> metrics,
> > > > > > replacing
> > > > > > >> scan_metrics_report and commit_metrics_report
> > > > > > >> •   Iceberg-focused naming and schema for now, revisit if
> > > > > generic-table
> > > > > > or
> > > > > > >> operational metrics arrive
> > > > > > >> •   Metrics persistence as a separated SPI, not on
> > BasePersistence
> > > > > > >> •   Bootstrap path separated for metrics, independent of
> > metastore
> > > > > > >> bootstrap
> > > > > > >> •   "Battery-included plus pluggable" as the SPI design
> > philosophy
> > > > > > >>
> > > > > > >> *Open items*
> > > > > > >> •   Schema details: metric_schema_version, payload format, IRC
> > v2
> > > > > > >> forward-compat shape
> > > > > > >> •   SPI design details — full review either in the next sync
> or
> > in
> > > > the
> > > > > > >> corresponding PR
> > > > > > >> •   Schema refactor PR ownership
> > > > > > >>
> > > > > > >> *Action items*
> > > > > > >> •   EJ to take a first stab at the SPI design and potentially
> > > > partner
> > > > > > with
> > > > > > >> Anand to incorporate the lessons learned from the existing
> > > reporter
> > > > > and
> > > > > > >> persistence work.
> > > > > > >> •   Schema refactor PR ownership is not yet decided. If anyone
> > is
> > > > > > >> interested in driving it, reply on this thread.
> > > > > > >> •   JB to schedule the next sync, tentatively in two weeks.
> > > > > > >>
> > > > > > >> -ej
> > > > > > >>
> > > > > > >> On Mon, Apr 27, 2026 at 3:07 PM EJ Wang <
> > > > > ej.wang.dev.interest@gmail.com
> > > > > > >
> > > > > > >> wrote:
> > > > > > >>
> > > > > > >> > Thanks Yufei for the +1.
> > > > > > >> >
> > > > > > >> > JB, could you help add a biweekly metrics architecture sync
> to
> > > the
> > > > > > >> Polaris
> > > > > > >> > community calendar? I'm thinking Thursdays at 9-10am PT, on
> > the
> > > > > > >> off-weeks
> > > > > > >> > from the community meeting (starting May 7), 60 minutes.
> > > > > > >> >
> > > > > > >> > Here's a rough agenda to work through over the first few
> > > sessions,
> > > > > > >> grouped
> > > > > > >> > by priority:
> > > > > > >> >
> > > > > > >> > *First: foundational direction*
> > > > > > >> >
> > > > > > >> > 1.  MetricsPersistence: public SPI or internal
> implementation
> > > > > detail?
> > > > > > >> >    •   Marked @Beta, javadoc calls it a "Service Provider
> > > > > Interface",
> > > > > > >> but
> > > > > > >> > only one consumer (JdbcBasePersistenceImpl), lives on
> > > > > BasePersistence.
> > > > > > >> If
> > > > > > >> > demoted to a private helper inside a persisting reporter
> impl,
> > > > most
> > > > > > >> > downstream design decisions become implementation details
> > rather
> > > > > than
> > > > > > >> > contract questions.
> > > > > > >> >
> > > > > > >> > 2.  Persistence schema redesign
> > > > > > >> >    •   Current two-table layout (scan_metrics_report,
> > > > > > >> > commit_metrics_report) with ~25 flattened columns each.
> Every
> > > new
> > > > > > metric
> > > > > > >> > type requires a new table, SPI method, record class, model,
> > > > > converter,
> > > > > > >> and
> > > > > > >> > schema migration. Direction to explore: single table with
> > > > > metric_type
> > > > > > >> enum,
> > > > > > >> > schema_version, and JSON payload column.
> > > > > > >> >
> > > > > > >> > *Second: design details once direction is set*
> > > > > > >> >
> > > > > > >> > 3.  Partition key strategy
> > > > > > >> >    •   Single-table design means scan metrics at scale will
> > have
> > > > > high
> > > > > > >> > write concurrency per table. Schema needs to expose enough
> > > > structure
> > > > > > for
> > > > > > >> > backends to shard by entity or time range.
> > > > > > >> >
> > > > > > >> > 4.  Read/write path consistency
> > > > > > >> >    •   Writes go through PolarisMetricsManager on
> > > > MetaStoreManager.
> > > > > > >> Reads
> > > > > > >> > bypass MetaStoreManager and go straight to BasePersistence,
> > > > > excluding
> > > > > > >> > non-JDBC backends from the read API.
> > > > > > >> >
> > > > > > >> > *Third: cleanup and alignment*
> > > > > > >> >
> > > > > > >> > 5.  PolarisMetricsReporter naming
> > > > > > >> >    •   Only handles IRC (ScanReport/CommitReport), doesn't
> > cover
> > > > > > generic
> > > > > > >> > tables or operational metrics. Name is broader than scope.
> > > > > > >> >
> > > > > > >> > 6.  PolarisMetricsManager facade passthrough
> > > > > > >> >    •   Entire default method is
> > > > > > >> callCtx.getMetaStore().writeScanReport().
> > > > > > >> > Zero logic, passes Level 1 straight through to Level 3. Same
> > > > > > >> anti-pattern
> > > > > > >> > as PolarisEventManager.
> > > > > > >> >
> > > > > > >> > 7.  Iceberg community alignment
> > > > > > >> >    •   Payload-type extension needs discussion on
> dev@iceberg.
> > > > > > >> obelix74's
> > > > > > >> > Feb thread got zero replies. Needs a committer voice.
> > > > > > >> >
> > > > > > >> > Lets confirm prioritization in the first session.
> > > > > > >> >
> > > > > > >> > -ej
> > > > > > >> >
> > > > > > >> > On Tue, Apr 21, 2026 at 3:18 PM Yufei Gu <
> > flyrain000@gmail.com>
> > > > > > wrote:
> > > > > > >> >
> > > > > > >> >> Thanks everyone for continuing to drive this forward. I
> agree
> > > > that
> > > > > > the
> > > > > > >> >> problem is getting complex enough that a more structured
> > > > discussion
> > > > > > >> would
> > > > > > >> >> help.
> > > > > > >> >>
> > > > > > >> >> +1 on setting up a biweekly sync for the metrics
> > architecture.
> > > > I’m
> > > > > > >> happy
> > > > > > >> >> to
> > > > > > >> >> join.
> > > > > > >> >>
> > > > > > >> >> Yufei
> > > > > > >> >>
> > > > > > >> >>
> > > > > > >> >> On Tue, Apr 21, 2026 at 2:34 PM EJ Wang <
> > > > > > >> ej.wang.dev.interest@gmail.com>
> > > > > > >> >> wrote:
> > > > > > >> >>
> > > > > > >> >> > Also, I've been looking more closely at the *persistence
> > > schema
> > > > > in
> > > > > > >> the
> > > > > > >> >> > current metrics work*, and I think there's a structural
> > > > rigidity
> > > > > > >> problem
> > > > > > >> >> > worth raising before the shape gets locked in.
> > > > > > >> >> >
> > > > > > >> >> > Right now we have two separate tables
> (scan_metrics_report
> > > and
> > > > > > >> >> > commit_metrics_report), each with ~25 flattened columns
> > that
> > > > > > directly
> > > > > > >> >> > mirror the Iceberg report fields. The SPI follows the
> same
> > > > split:
> > > > > > >> >> > writeScanReport and writeCommitReport as separate
> methods,
> > > with
> > > > > > >> per-type
> > > > > > >> >> > record classes, converters, and model objects. *The
> > practical
> > > > > cost:
> > > > > > >> >> > adding a new metric type (operational metrics, for
> example)
> > > > > > requires
> > > > > > >> a
> > > > > > >> >> new
> > > > > > >> >> > table, a new SPI method, a new record class, a new model
> > > > class, a
> > > > > > new
> > > > > > >> >> > converter branch, and a schema migration*. That's a lot
> of
> > > > > surface
> > > > > > >> area
> > > > > > >> >> > for what should be "one more kind of metric."
> > > > > > >> >> >
> > > > > > >> >> > *My bias* would be toward a single metrics table with *a
> > > typed
> > > > > JSON
> > > > > > >> >> > payload*. Something like: metric_type (enum), entity_id,
> > > > > > >> >> > table_identifier, snapshot_id (nullable), received_ts,
> > > > > > >> schema_version,
> > > > > > >> >> and
> > > > > > >> >> > a payload column for the metric-specific data. The
> > > metric_type
> > > > +
> > > > > > >> >> > schema_version pair gives us a forward-compatible
> contract
> > > for
> > > > > the
> > > > > > >> >> payload
> > > > > > >> >> > shape. Adding a new metric type becomes an enum value
> and a
> > > > > payload
> > > > > > >> >> schema,
> > > > > > >> >> > not a schema migration. One thing I think we need to be
> > > > > deliberate
> > > > > > >> >> about is
> > > > > > >> >> > the partition key design. If all metric types land in one
> > > > table,
> > > > > > scan
> > > > > > >> >> > metrics at scale (high concurrency, high frequency across
> > > many
> > > > > > >> tables)
> > > > > > >> >> > could easily create hot partitions. We'd want the
> > persistence
> > > > > layer
> > > > > > >> to
> > > > > > >> >> be
> > > > > > >> >> > able to shard by entity or time range, and that means the
> > > > logical
> > > > > > >> schema
> > > > > > >> >> > needs to expose enough structure for backends to
> partition
> > > on.
> > > > I
> > > > > > >> don't
> > > > > > >> >> > think the current flattened layout gives us that.
> > > > > > >> >> >
> > > > > > >> >> > This is getting complex enough that I don't think ad-hoc
> > > PR/ML
> > > > > > >> threads
> > > > > > >> >> > will converge well. *Would people be open to a biweekly
> > sync
> > > > for
> > > > > > >> metrics
> > > > > > >> >> > architecture?* I think 30 minutes every two weeks with
> > > > interested
> > > > > > >> >> parties
> > > > > > >> >> > would be enough to work through the schema, SPI shape,
> and
> > > read
> > > > > API
> > > > > > >> >> design
> > > > > > >> >> > together. Happy to help set that up.
> > > > > > >> >> >
> > > > > > >> >> > -ej
> > > > > > >> >> >
> > > > > > >> >> > On Mon, Apr 20, 2026 at 2:19 PM EJ Wang <
> > > > > > >> ej.wang.dev.interest@gmail.com
> > > > > > >> >> >
> > > > > > >> >> > wrote:
> > > > > > >> >> >
> > > > > > >> >> >> Reviewed #4115, left a comment on the code organization
> > > side.
> > > > > > >> >> >>
> > > > > > >> >> >> One thing stood out: the metrics write path enters
> through
> > > > > > >> >> >> PolarisMetricsManager on MetaStoreManager, but the new
> > read
> > > > path
> > > > > > >> >> bypasses
> > > > > > >> >> >> MetaStoreManager entirely and goes straight to
> > > BasePersistence
> > > > > via
> > > > > > >> >> >> callContext.getMetaStore(). That means the read API only
> > > works
> > > > > for
> > > > > > >> >> backends
> > > > > > >> >> >> that implement BasePersistence. NoSQL and remote
> backends
> > > > can't
> > > > > > >> >> participate.
> > > > > > >> >> >>
> > > > > > >> >> >> Stepping back, I think the metrics subsystem is growing
> > into
> > > > > > >> something
> > > > > > >> >> >> real (write + read + REST API + AuthZ + pagination) *but
> > the
> > > > > > >> >> persistence
> > > > > > >> >> >> side is split across two layers in a way that's hard to
> > > > > extend*. I
> > > > > > >> put
> > > > > > >> >> >> together two diagrams to show what I mean (my best
> > effort).
> > > > > > >> >> >>
> > > > > > >> >> >> *Current state* (Diagram 1): three interfaces at three
> > > > different
> > > > > > >> >> levels.
> > > > > > >> >> >> The engine-facing SPI (PolarisMetricsReporter) is clean.
> > But
> > > > > > >> >> >> PolarisMetricsManager on MetaStoreManager is a
> passthrough
> > > to
> > > > > > >> >> >> MetricsPersistence on BasePersistence. The @Beta
> > annotation
> > > > and
> > > > > > SPI
> > > > > > >> >> javadoc
> > > > > > >> >> >> are on the BasePersistence layer, while the actual
> > extension
> > > > > > points
> > > > > > >> >> >> (PolarisMetricsReporter, PolarisMetricsManager) carry no
> > > > > stability
> > > > > > >> >> >> annotation. The write path goes through the
> > MetaStoreManager
> > > > > > layer,
> > > > > > >> the
> > > > > > >> >> >> read path doesn't.
> > > > > > >> >> >>
> > > > > > >> >> >> *What I envision* (Diagram 2): two SPIs at two levels.
> > > > > > >> >> >> PolarisMetricsReporter stays as the engine-facing SPI.
> > > > > > >> >> >> PolarisMetricsManager becomes the backend-facing SPI
> with
> > > both
> > > > > > write
> > > > > > >> >> and
> > > > > > >> >> >> read methods at the MetaStoreManager level, where any
> > > backend
> > > > > > (JDBC,
> > > > > > >> >> NoSQL,
> > > > > > >> >> >> remote) can implement them. MetricsPersistence on
> > > > > BasePersistence
> > > > > > >> goes
> > > > > > >> >> >> away. Where metrics actually land is an implementation
> > > detail,
> > > > > > not a
> > > > > > >> >> core
> > > > > > >> >> >> interface.
> > > > > > >> >> >>
> > > > > > >> >> >> *Minor naming thing*: PolarisMetricsReporter is broader
> > than
> > > > > what
> > > > > > it
> > > > > > >> >> >> actually handles. It only accepts Iceberg REST Catalog
> > > metrics
> > > > > > >> >> (ScanReport,
> > > > > > >> >> >> CommitReport via MetricsReport). Generic table metrics
> or
> > > > > > >> operational
> > > > > > >> >> >> metrics aren't in scope. Not blocking, but worth noting
> if
> > > the
> > > > > > >> metrics
> > > > > > >> >> >> surface expands.
> > > > > > >> >> >>
> > > > > > >> >> >> *Rough sketch of how to get there*:
> > > > > > >> >> >>  1.  Add read methods to PolarisMetricsManager
> > > > (listScanReports,
> > > > > > >> >> >> listCommitReports) with default no-op, same as the
> > existing
> > > > > write
> > > > > > >> >> methods.
> > > > > > >> >> >> (Probably make PolarisMetricsManager more explicit on
> > being
> > > > > > Iceberg
> > > > > > >> >> >> specific like package name or class name etc.)
> > > > > > >> >> >>  2.  Wire MetricsReportsService through MetaStoreManager
> > > > instead
> > > > > > of
> > > > > > >> >> >> callContext.getMetaStore().
> > > > > > >> >> >>  3.  Extract metrics persistence from
> > > JdbcBasePersistenceImpl
> > > > > into
> > > > > > >> its
> > > > > > >> >> >> own class. That file carries ~7 responsibilities,
> metrics
> > > > being
> > > > > > one
> > > > > > >> of
> > > > > > >> >> them.
> > > > > > >> >> >>  4.  Remove MetricsPersistence from BasePersistence.
> > > > > > >> >> >>
> > > > > > >> >> >> *None of this needs to happen in #4115. But if the
> > direction
> > > > > makes
> > > > > > >> >> sense,
> > > > > > >> >> >> it would be good to align before the metrics surface
> grows
> > > > > > further.
> > > > > > >> >> Curious
> > > > > > >> >> >> what others think.*
> > > > > > >> >> >>
> > > > > > >> >> >> *My mental model note*: Level 1 MetaStoreManager; level
> 2
> > > > > > >> transactional
> > > > > > >> >> >> persistence; level 3 base persistence
> > > > > > >> >> >>
> > > > > > >> >> >> Diagram 1
> > > > > > >> >> >> <
> > > > > > >> >>
> > > > > > >>
> > > > > >
> > > > >
> > > >
> > >
> >
> https://www.plantuml.com/plantuml/uml/bLHDR-Cs4BthLmpIYupw0zbkKQ1r3M-S7Bp8xhhM7WCOb3IM65EaGD9EX2RzxHrHb4CxRelwa4YSDu_lpOVcnZ9jzvM8BBS2uGjQpJC3dtHMSekPtMk44IpsMgEqa5XcCOhCZikQQLP1pR8TAp2n3ILhmZDP20m0fcIvUkAoW2qJXd9z1bpToO9BX3WXu0ucy5rpgGPNm0nW5_epUWtm2Ue3pn3kMOFQmKntGZW0BYtgBSi8k5A2QMwybJNMIbFiGSR9QZc4nUqIvikStF0jHprua5C-amge42aNt3R0f5JaaoivdV2Pkqbx4hee4ymOkBh5BTiB-_uIeGeo8zL8rPsPl4DktdEiK1jkB1NdZCRbrSTecDe_mlHbF0wvBmCkaOH5_S8a_TTTKI6-nmCAkEw4LpxsZ-LbYLKQFKMNOgf_wuM7_bV9gOer5SYMMksBSWXFcbi49KNZXNLicwfe3TETC7gPdPqI7uBcHMb1RSzYq34c6PDUM9mn8HRsUTZEiDBve3NjVZumBj0U7SS37mGO7vcwtiK-_pU7U7L_f-digo9YbhSwIfMRwIITKGXbxdIUTCGF1SeCJxloKsU-3k9ddRbX1eDq1q_fx1JbBGT0glVyXimDuP4TQ5qpCAmnGEj2s_6n5mtn1z-97-63itFQZLPO1Ev2tu_WF7Ju-VPc0Skg5bYXxBhkY1xpD7EM_7fyflSpIsqMgVth5xhVr4eQxWQ8enaSAJQSG16yFSDuJ798rrcXr_3n-lfdk7icQjEBmFujL7AodiP_Y4Z7-YxvtZNs4zMgpNTl6tF8sglyPsmqchrjvQ-m-aP94r-TwCA2Ka8upPJZwtvSpoYCXkYMZU2NXvRMBfq9P3i3Le4VAZUAlUZ_oPKsxPgY0Q_BSKLkyr9bhQhQrJjo_x3TPlIB0DPjnMfcIoYP0QaYw1a0fTKDr8fB6ntNuvmoL1ZGkXa69Njh43zf9GiGxHQrA_jDYWRSzF5--WmTVrN97_Sm8LbLUy_lGBmLanJjFkDlGkRqjA_4tm00
> > > > > > >> >> >
> > > > > > >> >> >> :
> > > > > > >> >> >>
> > > > > > >> >> >> [image: image.png]
> > > > > > >> >> >>
> > > > > > >> >> >> Diagram 2
> > > > > > >> >> >> <
> > > > > > >> >>
> > > > > > >>
> > > > > >
> > > > >
> > > >
> > >
> >
> https://www.plantuml.com/plantuml/uml/VLLDR-8m4BtdLupO2sWBLVU8AaGB7AXAbssGzb896SS9RXqxjKqBwkv_tt7iV43fdaZYDpFlpRmnOsE9jhjSH9PRmM31hERKm8scMsuPjJlDe0yheZDc8RR4iYWoBrmMH9CS2a9VICPYUy1OZN0YCy5Q0BCbYNhdCeEK28En8G8wCvbnoQ0R8_05Bc6bkLIz3X03p1zzH7zR-9ZfDquPt9C3qoNCX2yV4G2NbkcKu5jdgGJHt0GbZwnG6i-UP3TUpk5gM6Ldqke350eZUqzoCft3U9xWHvxoa5-7K4nF1J46EbEMafsmdrCBbQ44gVggy18IZrn_ph5asd1ZiIKdQSgueZvjXrQFSFrdC3YN-nXmBacxbGiYyLVxLaBtdhqn0LSzdBDhqQtQoOJeGyad3z0lUqnYgpGB6Ns8oVyta00Dy_WnX0tIOZ8v6SYxHll1TrH6aejAik-mh-AphVFCwSUQqFypElag5QRGFDjQKEd96K1P8QP41c9TzA_IIQyvdAWyv_RSiS3skb0_EzDDkK2v5xWF6MiGFlvhpFLcD2Dq2pml14gaF67eQkmd8gulDoC4kSOu6KVpkvlUJg1RTbWISU40RdBUUS_9XfRZ2dwxm_SW8LYFISgm_MnlDQ6M9P1gbKEc4X-2pH_FvJCkCqm9pbVjD6LrwdLeOrDWfOaqc8Wh9BE85oNKxkNQ6o4yGRy_Eae0G_G8tZv81d3bHDB23WOdisohVr3nh_j6lbSjbNaLRTc8UgtPbAU1J_tygOfZX9DWEJeHDvYx-qmSi5FgNLPZwHrHcUsncGQ5-skhUclpE5fo4ounpFauYrUbkU6ccfnxMvitwag4IyerhTxj8In_Oj1bDO4pQru674loYrGlULHLEGCjwJJ8gDoVZR8MxO4BT3IzRvIcAQKezC6xpziGnTyImrfEGyJI_OcKfgtxIvnTqFEMS17L9Z-jsARN5FmTheP7HtSdtOMT0B4GY2FYHXxgQmMtj2bRqiLFGapiVe1_QVKDrkqXcm83aFEXnMYCZ-xlyHy
> > > > > > >> >> >
> > > > > > >> >> >> :
> > > > > > >> >> >> [image: image.png]
> > > > > > >> >> >>
> > > > > > >> >> >>  -ej
> > > > > > >> >> >>
> > > > > > >> >> >> On Wed, Apr 15, 2026 at 8:22 AM Dmitri Bourlatchkov <
> > > > > > >> dimas@apache.org>
> > > > > > >> >> >> wrote:
> > > > > > >> >> >>
> > > > > > >> >> >>> Hi All,
> > > > > > >> >> >>>
> > > > > > >> >> >>> Heads up: The current state of PR [4115] looks pretty
> > solid
> > > > to
> > > > > > me.
> > > > > > >> I
> > > > > > >> >> >>> believe this PR is approaching a mergeable condition.
> > > > > > >> >> >>>
> > > > > > >> >> >>> Please post your reviews if you have any comments.
> > > > > > >> >> >>>
> > > > > > >> >> >>> [4115] https://github.com/apache/polaris/pull/4115
> > > > > > >> >> >>>
> > > > > > >> >> >>> Thanks,
> > > > > > >> >> >>> Dmitri.
> > > > > > >> >> >>>
> > > > > > >> >> >>> On Tue, Mar 3, 2026 at 3:29 PM Anand Kumar Sankaran via
> > > dev <
> > > > > > >> >> >>> dev@polaris.apache.org> wrote:
> > > > > > >> >> >>>
> > > > > > >> >> >>> > Hi Yufei and Dmitri,
> > > > > > >> >> >>> >
> > > > > > >> >> >>> > Here is a proposal for the REST endpoints for metrics
> > and
> > > > > > events.
> > > > > > >> >> >>> >
> > > > > > >> >> >>> > https://github.com/apache/polaris/pull/3924/changes
> > > > > > >> >> >>> >
> > > > > > >> >> >>> > I did not see any precursors for raising a PR for
> > > > proposals,
> > > > > so
> > > > > > >> >> trying
> > > > > > >> >> >>> > this.  Please let me know what you think.
> > > > > > >> >> >>> >
> > > > > > >> >> >>> > -
> > > > > > >> >> >>> > Anand
> > > > > > >> >> >>> >
> > > > > > >> >> >>> > From: Anand Kumar Sankaran <
> anand.sankaran@workday.com
> > >
> > > > > > >> >> >>> > Date: Monday, March 2, 2026 at 10:25 AM
> > > > > > >> >> >>> > To: dev@polaris.apache.org <de...@polaris.apache.org>
> > > > > > >> >> >>> > Subject: Re: Polaris Telemetry and Audit Trail
> > > > > > >> >> >>> >
> > > > > > >> >> >>> > About the REST API, based on my use cases:
> > > > > > >> >> >>> >
> > > > > > >> >> >>> >
> > > > > > >> >> >>> >   1.
> > > > > > >> >> >>> > I want to be able to query commit metrics to track
> > files
> > > > > added
> > > > > > /
> > > > > > >> >> >>> removed
> > > > > > >> >> >>> > per commit, along with record counts. The ingestion
> > > > pipeline
> > > > > > that
> > > > > > >> >> >>> writes
> > > > > > >> >> >>> > this data is owned by us and we are guaranteed to
> write
> > > > this
> > > > > > >> >> >>> information
> > > > > > >> >> >>> > for each write.
> > > > > > >> >> >>> >   2.
> > > > > > >> >> >>> > I want to be able to query scan metrics for read. I
> > > > > understand
> > > > > > >> >> clients
> > > > > > >> >> >>> do
> > > > > > >> >> >>> > not fulfill this requirement.
> > > > > > >> >> >>> >   3.
> > > > > > >> >> >>> > I want to be able to query the events table (events
> are
> > > > > > >> persisted) -
> > > > > > >> >> >>> this
> > > > > > >> >> >>> > may supersede #2, I am not sure yet.
> > > > > > >> >> >>> >
> > > > > > >> >> >>> > All this information is in the JDBC based persistence
> > > model
> > > > > and
> > > > > > >> is
> > > > > > >> >> >>> > persisted in the metastore. I currently don’t have a
> > need
> > > > to
> > > > > > >> query
> > > > > > >> >> >>> > prometheus or open telemetry. I do publish some
> events
> > to
> > > > > > >> Prometheus
> > > > > > >> >> >>> and
> > > > > > >> >> >>> > they are forwarded to our dashboards elsewhere.
> > > > > > >> >> >>> >
> > > > > > >> >> >>> > About the CLI utilities, I meant the admin user
> > > utilities.
> > > > In
> > > > > > >> one of
> > > > > > >> >> >>> the
> > > > > > >> >> >>> > earliest drafts of my proposal, Prashant mentioned
> that
> > > the
> > > > > > >> metrics
> > > > > > >> >> >>> tables
> > > > > > >> >> >>> > can grow indefinitely and that a similar problem
> exists
> > > > with
> > > > > > the
> > > > > > >> >> events
> > > > > > >> >> >>> > table as well. We discussed that cleaning up of old
> > > records
> > > > > > from
> > > > > > >> >> both
> > > > > > >> >> >>> > metrics tables and events tables can be done via a
> CLI
> > > > > utility.
> > > > > > >> >> >>> >
> > > > > > >> >> >>> > I see that Yufei has covered the discussion about
> > > > > datasources.
> > > > > > >> >> >>> >
> > > > > > >> >> >>> > -
> > > > > > >> >> >>> > Anand
> > > > > > >> >> >>> >
> > > > > > >> >> >>> >
> > > > > > >> >> >>> >
> > > > > > >> >> >>> > From: Yufei Gu <fl...@gmail.com>
> > > > > > >> >> >>> > Date: Friday, February 27, 2026 at 9:54 PM
> > > > > > >> >> >>> > To: dev@polaris.apache.org <de...@polaris.apache.org>
> > > > > > >> >> >>> > Subject: Re: Polaris Telemetry and Audit Trail
> > > > > > >> >> >>> >
> > > > > > >> >> >>> > This Message Is From an External Sender
> > > > > > >> >> >>> > This message came from outside your organization.
> > > > > > >> >> >>> > Report Suspicious<
> > > > > > >> >> >>> >
> > > > > > >> >> >>>
> > > > > > >> >>
> > > > > > >>
> > > > > >
> > > > >
> > > >
> > >
> >
> https://us-phishalarm-ewt.proofpoint.com/EWT/v1/Iz9xO38YGHZK!YhNDZABkHi1B699ote2uMwpOZw8i0QMCGO2Szc-HshuABGhGvwPJcymE6G2oUUxtS8xDkSrtGTPm_I3QnVDHoLMk50m9v8z_nZKTkd-bnVUbreF1u0WnfV_X5eYevZl_$
> > > > > > >> >> >>> > >
> > > > > > >> >> >>> >
> > > > > > >> >> >>> >
> > > > > > >> >> >>> > As I mentioned in
> > > > > > >> >> >>> >
> > > > > > >> >> >>>
> > > > > > >> >>
> > > > > > >>
> > > > > >
> > > > >
> > > >
> > >
> >
> https://urldefense.com/v3/__https://github.com/apache/polaris/issues/3890__;!!Iz9xO38YGHZK!5EuyFFkk3vhRWVIRvQAWBSQfpJkTMA9HxugzDwXmN0LPPqhEFxYkFRGVhtb8AqUwXtDh2OplcMnbMDHKOxrvDU0$
> > > > > > >> >> >>> ,
> > > > > > >> >> >>> > supporting
> > > > > > >> >> >>> > multiple data sources is not a trivial change. I
> would
> > > > > strongly
> > > > > > >> >> >>> recommend
> > > > > > >> >> >>> > starting with a design document to carefully evaluate
> > the
> > > > > > >> >> architectural
> > > > > > >> >> >>> > implications and long term impact.
> > > > > > >> >> >>> >
> > > > > > >> >> >>> > A REST endpoint to query metrics seems reasonable
> given
> > > the
> > > > > > >> current
> > > > > > >> >> >>> JDBC
> > > > > > >> >> >>> > based persistence model. That said, we may also
> > consider
> > > > > > >> alternative
> > > > > > >> >> >>> > storage models. For example, if we later adopt a time
> > > > series
> > > > > > >> system
> > > > > > >> >> >>> such as
> > > > > > >> >> >>> > Prometheus to store metrics, the query model and
> access
> > > > > > patterns
> > > > > > >> >> would
> > > > > > >> >> >>> be
> > > > > > >> >> >>> > fundamentally different. Designing the REST API
> without
> > > > > > >> considering
> > > > > > >> >> >>> these
> > > > > > >> >> >>> > potential evolutions may limit flexibility. I'd
> suggest
> > > to
> > > > > > start
> > > > > > >> >> with
> > > > > > >> >> >>> the
> > > > > > >> >> >>> > use case.
> > > > > > >> >> >>> >
> > > > > > >> >> >>> > Yufei
> > > > > > >> >> >>> >
> > > > > > >> >> >>> >
> > > > > > >> >> >>> > On Fri, Feb 27, 2026 at 3:42 PM Dmitri Bourlatchkov <
> > > > > > >> >> dimas@apache.org>
> > > > > > >> >> >>> > wrote:
> > > > > > >> >> >>> >
> > > > > > >> >> >>> > > Hi Anand,
> > > > > > >> >> >>> > >
> > > > > > >> >> >>> > > Sharing my view... subject to discussion:
> > > > > > >> >> >>> > >
> > > > > > >> >> >>> > > 1. Adding non-IRC REST API to Polaris is perfectly
> > > fine.
> > > > > > >> >> >>> > >
> > > > > > >> >> >>> > > Figuring out specific endpoint URIs and payloads
> > might
> > > > > > require
> > > > > > >> a
> > > > > > >> >> few
> > > > > > >> >> >>> > > roundtrips, so opening a separate thread for that
> > might
> > > > be
> > > > > > >> best.
> > > > > > >> >> >>> > > Contributors commonly create Google Docs for new
> API
> > > > > > proposals
> > > > > > >> too
> > > > > > >> >> >>> (they
> > > > > > >> >> >>> > > fairly easy to update as the email discussion
> > > > progresses).
> > > > > > >> >> >>> > >
> > > > > > >> >> >>> > > There was a suggestion to try Markdown (with PRs)
> for
> > > > > > proposals
> > > > > > >> >> [1]
> > > > > > >> >> >>> ...
> > > > > > >> >> >>> > > feel free to give it a try if you are comfortable
> > with
> > > > > that.
> > > > > > >> >> >>> > >
> > > > > > >> >> >>> > > 2. Could you clarify whether you mean end user
> > > utilities
> > > > or
> > > > > > >> admin
> > > > > > >> >> >>> user
> > > > > > >> >> >>> > > utilities? In the latter case those might be more
> > > > suitable
> > > > > > for
> > > > > > >> the
> > > > > > >> >> >>> Admin
> > > > > > >> >> >>> > > CLI (java) not the Python CLI, IMHO.
> > > > > > >> >> >>> > >
> > > > > > >> >> >>> > > Why would these utilities be common with events?
> > IMHO,
> > > > > event
> > > > > > >> use
> > > > > > >> >> >>> cases
> > > > > > >> >> >>> > are
> > > > > > >> >> >>> > > distinct from scan/commit metrics.
> > > > > > >> >> >>> > >
> > > > > > >> >> >>> > > 3. I'd prefer separating metrics persistence from
> > > > MetaStore
> > > > > > >> >> >>> persistence
> > > > > > >> >> >>> > at
> > > > > > >> >> >>> > > the code level, so that they could be mixed and
> > matched
> > > > > > >> >> >>> independently.
> > > > > > >> >> >>> > The
> > > > > > >> >> >>> > > separate datasource question will become a
> non-issue
> > > with
> > > > > > that
> > > > > > >> >> >>> approach,
> > > > > > >> >> >>> > I
> > > > > > >> >> >>> > > guess.
> > > > > > >> >> >>> > >
> > > > > > >> >> >>> > > The rationale for separating scan metrics and
> > metastore
> > > > > > >> >> persistence
> > > > > > >> >> >>> is
> > > > > > >> >> >>> > that
> > > > > > >> >> >>> > > "cascading deletes" between them are hardly ever
> > > > required.
> > > > > > >> >> >>> Furthermore,
> > > > > > >> >> >>> > the
> > > > > > >> >> >>> > > data and query patterns are very different so
> > different
> > > > > > >> >> technologies
> > > > > > >> >> >>> > might
> > > > > > >> >> >>> > > be beneficial in each case.
> > > > > > >> >> >>> > >
> > > > > > >> >> >>> > > [1]
> > > > > > >> >> >>> >
> > > > > > >> >> >>>
> > > > > > >> >>
> > > > > > >>
> > > > > >
> > > > >
> > > >
> > >
> >
> https://urldefense.com/v3/__https://lists.apache.org/thread/yto2wp982t43h1mqjwnslswhws5z47cy__;!!Iz9xO38YGHZK!5EuyFFkk3vhRWVIRvQAWBSQfpJkTMA9HxugzDwXmN0LPPqhEFxYkFRGVhtb8AqUwXtDh2OplcMnbMDHKxYDakNU$
> > > > > > >> >> >>> > >
> > > > > > >> >> >>> > > Cheers,
> > > > > > >> >> >>> > > Dmitri.
> > > > > > >> >> >>> > >
> > > > > > >> >> >>> > > On Fri, Feb 27, 2026 at 6:19 PM Anand Kumar
> Sankaran
> > > via
> > > > > dev
> > > > > > <
> > > > > > >> >> >>> > > dev@polaris.apache.org> wrote:
> > > > > > >> >> >>> > >
> > > > > > >> >> >>> > > > Thanks all. This PR is merged now.
> > > > > > >> >> >>> > > >
> > > > > > >> >> >>> > > > Here are the follow-up features / work needed.
> > These
> > > > > were
> > > > > > >> all
> > > > > > >> >> >>> part of
> > > > > > >> >> >>> > > the
> > > > > > >> >> >>> > > > merged PR at some point in time and were removed
> to
> > > > > reduce
> > > > > > >> >> scope.
> > > > > > >> >> >>> > > >
> > > > > > >> >> >>> > > > Please let me know what you think.
> > > > > > >> >> >>> > > >
> > > > > > >> >> >>> > > >
> > > > > > >> >> >>> > > >   1.  A REST API to paginate through table
> metrics.
> > > > This
> > > > > > >> will be
> > > > > > >> >> >>> > non-IRC
> > > > > > >> >> >>> > > > standard addition.
> > > > > > >> >> >>> > > >   2.  Utilities for managing old records, should
> be
> > > > > common
> > > > > > >> with
> > > > > > >> >> >>> events.
> > > > > > >> >> >>> > > > There was some discussion that it belongs to the
> > CLI.
> > > > > > >> >> >>> > > >   3.  Separate datasource (metrics, events, even
> > > other
> > > > > > >> tables?).
> > > > > > >> >> >>> > > >
> > > > > > >> >> >>> > > >
> > > > > > >> >> >>> > > > Anything else?
> > > > > > >> >> >>> > > >
> > > > > > >> >> >>> > > > -
> > > > > > >> >> >>> > > > Anand
> > > > > > >> >> >>> > > >
> > > > > > >> >> >>> > > >
> > > > > > >> >> >>> > >
> > > > > > >> >> >>> >
> > > > > > >> >> >>> >
> > > > > > >> >> >>>
> > > > > > >> >> >>
> > > > > > >> >>
> > > > > > >> >
> > > > > > >>
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>

Re: Proposal for REST endpoints for table metrics and events

[EJ Wang <ej...@gmail.com>]

Good point. I agree the existing PolarisMetricsReporter is already very
close to the right conceptual boundary. I am not proposing a second
parallel reporter concept. The distinction I am trying to make is narrower:

Current state:

   - PolarisMetricsReporter is the existing metrics reporting hook.
   - But it currently lives under runtime/service.
   - Its contract is documented around Quarkus/CDI discovery and @Identifier
   selection.
   - The default implementation is log-only.
   - The durable implementation writes through PolarisMetricsManager and
   MetricsPersistence.
   - MetricsPersistence is currently inherited by BasePersistence.

My proposal:

   - Keep the reporting/emitting boundary as the SPI.
   - Revise that contract in a framework-agnostic optional Iceberg *metrics
   extension API layer*.
   - Keep runtime/service responsible for REST ingestion, table/authz
   validation, and runtime wiring.
   - Keep the battery implementation log-only in the metrics extension API
   layer, *not under runtime/service*.
   - Treat durable JDBC metrics as one implementation of the reporting SPI.
   - Decompose the MetricsPersistence and BasePersistence coupling as
   durable implementation detail.

So the difference is not “new reporter concept vs old reporter concept.”
but:


   - PolarisMetricsReporter today = runtime/service CDI-shaped hook.
   - Target reporting SPI = same conceptual boundary, but placed in the
   right extension API layer and kept framework-agnostic.

For the current metrics PR, I think the minimal architectural cleanup
before merge is to *put the reporting SPI contract and the battery default
in the right place*. The durable JDBC implementation, event-backed async
implementation, external queue support, and deeper persistence cleanup can
continue as follow-ups, as long as we do not lock the SPI boundary to
BasePersistence or to runtime/service wiring.

-ej

On Wed, Jun 3, 2026 at 3:00 PM Dmitri Bourlatchkov <di...@apache.org> wrote:

> Hi EJ,
>
> Thanks for the recap / summary!
>
> > Do folks agree that the stable SPI boundary should be
> metrics reporting/emitting, not metrics persistence?
>
> SGTM.
>
> > Does an optional Iceberg metrics extension API layer sound like the right
> home for this SPI?
>
> How is that different from current PolarisMetricsReporter?
>
> > Should the current durable metrics work be reframed as a durable
> JDBC reference implementation of that SPI?
>
> SGTM.
>
> > What is the smallest PR sequence to get there without blocking
> the current metrics work unnecessarily?
>
> Let's get https://github.com/apache/polaris/pull/4397 merged first.
>
> Then, I'd propose isolating the Metrics schema from the MetaStore schema.
> This will probably have some ripple effect into the bootstrap workflows, so
> it's not a trivial change.
>
> Then, let's reassess.
>
> Cheers,
> Dmitri.
>
> On Wed, Jun 3, 2026 at 5:40 PM EJ Wang <ej...@gmail.com>
> wrote:
>
> > Hi folks,
> >
> > Thanks again for the discussion today. I updated the sync doc with notes
> > from the third metrics architecture sync:
> >
> >
> >
> https://docs.google.com/document/d/100h7c4damrUzVuquYbBHM0EvA4LSWuW2IT2dN_7nYVA/edit?tab=t.k96s2xyqr5u1
> >
> > A few highlights from today’s discussion:
> >
> >    - We clarified that Iceberg metrics reporting can be interpreted
> either
> >    as sync or async handling from Polaris’s perspective, so Polaris
> should
> >    stay flexible as a platform instead of baking one handling model into
> > the
> >    REST API behavior.
> >    - We aligned that the built-in (battery) behavior for metrics emitting
> >    can stay simple: no-op/log-only is enough as the default.
> >    - Durable metrics persistence should be treated as an implementation
> of
> >    the metrics reporting path, not as the core SPI boundary.
> >    - The existing durable metrics work can be reviewed as a reference
> >    implementation of the reporting SPI, with persistence-related logic
> kept
> >    self-contained under a metrics durable implementation module rather
> than
> >    scattered through core entity persistence.
> >    - Dashboard/insights remains a real use case, but we agreed to keep it
> >    separate from the core metrics intake discussion for now.
> >
> > I also did a quick source check after the meeting to make sure we are
> > describing the current state accurately.
> >
> > Current state:
> >
> >    - Polaris already has a metrics reporting hook:
> PolarisMetricsReporter.
> >    - The default implementation is DefaultMetricsReporter, selected by
> >    polaris.iceberg-metrics.reporting.type=default. It is log-only,
> >    effectively quiet unless metrics logging is enabled.
> >    - There is also a PersistingMetricsReporter, selected by
> >    polaris.iceberg-metrics.reporting.type=persisting, which converts
> >    Iceberg scan/commit reports into Polaris metrics records and writes
> > through
> >    PolarisMetricsManager -> MetricsPersistence.
> >    - MetricsPersistence currently exists in the persistence layer and is
> >    inherited by BasePersistence.
> >
> > My read from the discussion is that the target boundary should not be
> > MetricsPersistence as inherited by BasePersistence. That path should be
> > decomposed as durable implementation detail (taken care of by
> > https://github.com/apache/polaris/pull/4397). The stable extension point
> > should instead be the metrics reporting/emitting boundary.
> >
> > Concretely, I think the next proposal should be shaped like this:
> >
> >    1.
> >
> >    Define the proper metrics reporting SPI at an optional Iceberg metrics
> >    extension API layer.
> >
> >    Example direction:
> >
> >    public interface IcebergMetricsReporter {
> >      void reportMetric(IcebergMetricsReportContext context,
> > MetricsReport report);
> >    }
> >
> >    The context would carry the small Polaris-resolved envelope:
> >    catalog/table identity, report type, received timestamp, and
> > request/trace
> >    context if available. The raw Iceberg MetricsReport remains the
> payload.
> >    2.
> >
> >    Keep runtime/service as the REST ingestion and wiring layer.
> >
> >    The REST handler still resolves the table and performs authz before
> >    accepting the report. After that, it calls the selected reporting
> >    implementation.
> >    3.
> >
> >    Keep the battery default no-op/log-only.
> >
> >    This preserves an out-of-box safe default and avoids requiring a
> durable
> >    metrics store for every Polaris deployment.
> >    4.
> >
> >    Move durable JDBC metrics into a self-contained implementation of the
> >    reporting SPI.
> >
> >    That implementation can own its schema, bootstrap, retention, and read
> >    API support. It should not define the core reporting SPI boundary, and
> > it
> >    should not require metrics persistence to remain inherited from
> >    BasePersistence.
> >    5.
> >
> >    Treat async/event-backed handling as another implementation of the
> same
> >    reporting SPI.
> >
> >    For example, an event-backed reporter could enqueue a metrics event
> and
> >    let listeners handle durable storage or other sinks. If we later need
> a
> >    replaceable queue engine, that seems like a shared event/metrics
> > substrate
> >    topic rather than a metrics-only requirement.
> >
> > This framing lets us keep the REST metrics endpoint simple, preserve the
> > current default behavior, support durable metrics users, and still leave
> > room for async/event-backed or external-queue-based implementations.
> >
> > I think the main follow-up questions are:
> >
> >    - Do folks agree that the stable SPI boundary should be metrics
> >    reporting/emitting, not metrics persistence?
> >    - Does an optional Iceberg metrics extension API layer sound like the
> >    right home for this SPI?
> >    - Should the current durable metrics work be reframed as a durable
> JDBC
> >    reference implementation of that SPI?
> >    - What is the smallest PR sequence to get there without blocking the
> >    current metrics work unnecessarily?
> >
> > Thanks,
> > -ej
> >
> > On Fri, May 15, 2026 at 5:16 PM Dmitri Bourlatchkov <di...@apache.org>
> > wrote:
> >
> > > Hi JB,
> > >
> > > Could you set up another meeting, please? Same time on Wednesday as
> last
> > > time... I hope it works for everyone.
> > >
> > > Cheers,
> > > Dmitri.
> > >
> > > On Fri, May 15, 2026 at 8:06 PM Yufei Gu <fl...@gmail.com> wrote:
> > >
> > > > +1 on another sync call next week.
> > > >
> > > > Yufei
> > > >
> > > >
> > > > On Fri, May 15, 2026 at 4:52 PM Dmitri Bourlatchkov <
> dimas@apache.org>
> > > > wrote:
> > > >
> > > > > Hi All,
> > > > >
> > > > > WDYT about another sync call next week?
> > > > >
> > > > > Thanks,
> > > > > Dmitri.
> > > > >
> > > > > On Wed, May 6, 2026 at 5:29 PM Dmitri Bourlatchkov <
> dimas@apache.org
> > >
> > > > > wrote:
> > > > >
> > > > > > Hi EJ,
> > > > > >
> > > > > > Thanks for the summary! It covers what we discussed in the
> meeting
> > > very
> > > > > > well, IMHO.
> > > > > >
> > > > > > Looking forward to concrete PRs :)
> > > > > >
> > > > > > Cheers,
> > > > > > Dmitri.
> > > > > >
> > > > > > On Wed, May 6, 2026 at 5:08 PM EJ Wang <
> > > ej.wang.dev.interest@gmail.com
> > > > >
> > > > > > wrote:
> > > > > >
> > > > > >> Hi folks,
> > > > > >>
> > > > > >> We had a community sync earlier, thanks JB for scheduling it.
> > Notes
> > > > from
> > > > > >> the first metrics architecture sync (May 6, 10-11am PT).
> > Discussion
> > > > doc
> > > > > >> with per-section status:
> > > > > >>
> > > > > >>
> > > > >
> > > >
> > >
> >
> https://docs.google.com/document/d/100h7c4damrUzVuquYbBHM0EvA4LSWuW2IT2dN_7nYVA/edit?tab=t.0
> > > > > >>
> > > > > >> *The meeting covered both topics from the doc. Direction-level
> > > > alignment
> > > > > >> was reached on the headline pieces; details remain for PR review
> > or
> > > > > >> follow-up sessions.*
> > > > > >>
> > > > > >> *Topic 1 — Persistence schema redesign*
> > > > > >> Idea-level alignment on consolidating per-type tables
> > > > > >> (scan_metrics_report,
> > > > > >> commit_metrics_report) into a single metrics_report table. The
> > > > > motivating
> > > > > >> cost is the surface area added by every new metric type today:
> new
> > > > > table,
> > > > > >> SPI method, record class, model, converter, schema migration.
> > > > > >>
> > > > > >> Most schema details are deferred to the schema PR. A few
> specific
> > > > points
> > > > > >> came up:
> > > > > >> •   metric_schema_version: Yufei prefers dropping it, since
> there
> > is
> > > > no
> > > > > >> spec-level concept of metrics versioning today and it is hard to
> > > > define
> > > > > >> unilaterally. Robert prefers keeping it, given IRC v2 is coming
> > and
> > > > the
> > > > > >> schema should be considered against its likely shape; Robert
> also
> > > > raised
> > > > > >> how to differentiate various payload formats if any. EJ's read
> is
> > > that
> > > > > >> this
> > > > > >> is a two-way-door decision. We can start without the field, and
> if
> > > IRC
> > > > > v2
> > > > > >> changes the shape we would likely roll a corresponding new
> schema
> > > > > anyway,
> > > > > >> which is not particularly costly.
> > > > > >> •   Payload format: Robert pointed out that future formats
> beyond
> > > JSON
> > > > > may
> > > > > >> be worth supporting. The exact shape is deferred to the schema
> > > > > discussion.
> > > > > >> •   Partition strategy: Anand suggested monthly partitioning
> based
> > > on
> > > > > his
> > > > > >> experience as potentially helpful at scale.
> > > > > >>
> > > > > >> *Topic 2 — Where metrics ingestion and storage belong*
> > > > > >> Idea-level alignment that metrics should be a separated SPI from
> > the
> > > > > >> entity
> > > > > >> persistence stack. Two reasons surfaced: (a) workloads and
> > > capability
> > > > > >> requirements diverge enough that coupling them creates
> artificial
> > > > > >> constraints, and (b) admin experience improves when metrics has
> > its
> > > > own
> > > > > >> bootstrap, retention, and lifecycle. Dmitri noted Polaris being
> a
> > > > > platform
> > > > > >> should have the flexibility to support different persistence
> > > backends
> > > > > per
> > > > > >> concern, and pointed to a concrete next step of separating the
> > JDBC
> > > > > >> bootstrap for metrics from the metastore bootstrap. Robert
> > proposed
> > > an
> > > > > >> additional UX extension: detect an unbootstrapped metrics store
> on
> > > > first
> > > > > >> use and auto-bootstrap rather than requiring an explicit manual
> > > > > bootstrap
> > > > > >> step.
> > > > > >> The meeting also confirmed that Polaris metrics can start small
> > and
> > > > stay
> > > > > >> Iceberg-focused. Naming and persistence schema can lean
> > > > > Iceberg-specific.
> > > > > >> If a future expansion to generic-table metrics or operational
> > > metrics
> > > > > >> arrives, an abstraction layer can be built on top of the Iceberg
> > > > metrics
> > > > > >> reporter at that point. Robert remains on the fence and would
> > prefer
> > > > > >> something more generic but did not block the direction; Dmitri's
> > > read
> > > > > was
> > > > > >> that the proposed framework already has enough flexibility to
> > absorb
> > > > > >> future
> > > > > >> expansion.
> > > > > >>
> > > > > >> The Trade-offs and Proposed structure sections in the doc were
> not
> > > > > >> reviewed
> > > > > >> in detail. They remain open for either the next sync or PR
> review.
> > > > > >>
> > > > > >> *Cross-cutting alignment — battery-included plus pluggable*
> > > > > >> A common philosophy emerged from the discussion. EJ summarized
> it
> > > as:
> > > > > >> Polaris should provide a battery-included UX for beginners and
> the
> > > > > >> flexibility for advanced users to swap the included battery for
> > > > > something
> > > > > >> more powerful or tailored to their use case. The SPI design
> needs
> > to
> > > > > >> enable
> > > > > >> both.
> > > > > >>
> > > > > >> The inputs that shaped this framing:
> > > > > >> •   Anand described how his team uses the current metrics
> > > persistence
> > > > > >> (three metrics consumers in v1.4).
> > > > > >> •   Yufei raised Grafana and dashboard integrations as a
> > destination
> > > > use
> > > > > >> case beyond the default.
> > > > > >> •   Robert called out that the current design is more
> > JDBC-focused.
> > > > > >>
> > > > > >> Two concrete instances:
> > > > > >> •   Async metrics intake: Yufei's initial position was that
> async
> > > > should
> > > > > >> largely live on the producer side and there is not much Polaris
> > can
> > > > do.
> > > > > >> Robert suggested a Polaris-side default is doable via Vert.x.
> > Dmitri
> > > > > >> agreed
> > > > > >> the direction is worth exploring. The meeting converged on a
> > > > > >> battery-included default (likely Vert.x-backed) with an SPI
> shape
> > > that
> > > > > >> lets
> > > > > >> power users route to a more scalable backend (k8s-hosted queue,
> > AWS
> > > > SQS,
> > > > > >> etc.).
> > > > > >> •   Pluggable destinations: combining Yufei's dashboard use case
> > > with
> > > > > >> Robert's JDBC-focused call-out, the meeting agreed the SPI
> should
> > be
> > > > > >> structured for multiple sinks so integrations become impl
> choices
> > > > rather
> > > > > >> than architectural changes.
> > > > > >>
> > > > > >> The battery-included default is most likely to use the existing
> > > > > >> JDBC-backed
> > > > > >> approach.
> > > > > >>
> > > > > >> *Direction (idea-level alignment)*
> > > > > >> •   Single metrics_report table consolidating per-type metrics,
> > > > > replacing
> > > > > >> scan_metrics_report and commit_metrics_report
> > > > > >> •   Iceberg-focused naming and schema for now, revisit if
> > > > generic-table
> > > > > or
> > > > > >> operational metrics arrive
> > > > > >> •   Metrics persistence as a separated SPI, not on
> BasePersistence
> > > > > >> •   Bootstrap path separated for metrics, independent of
> metastore
> > > > > >> bootstrap
> > > > > >> •   "Battery-included plus pluggable" as the SPI design
> philosophy
> > > > > >>
> > > > > >> *Open items*
> > > > > >> •   Schema details: metric_schema_version, payload format, IRC
> v2
> > > > > >> forward-compat shape
> > > > > >> •   SPI design details — full review either in the next sync or
> in
> > > the
> > > > > >> corresponding PR
> > > > > >> •   Schema refactor PR ownership
> > > > > >>
> > > > > >> *Action items*
> > > > > >> •   EJ to take a first stab at the SPI design and potentially
> > > partner
> > > > > with
> > > > > >> Anand to incorporate the lessons learned from the existing
> > reporter
> > > > and
> > > > > >> persistence work.
> > > > > >> •   Schema refactor PR ownership is not yet decided. If anyone
> is
> > > > > >> interested in driving it, reply on this thread.
> > > > > >> •   JB to schedule the next sync, tentatively in two weeks.
> > > > > >>
> > > > > >> -ej
> > > > > >>
> > > > > >> On Mon, Apr 27, 2026 at 3:07 PM EJ Wang <
> > > > ej.wang.dev.interest@gmail.com
> > > > > >
> > > > > >> wrote:
> > > > > >>
> > > > > >> > Thanks Yufei for the +1.
> > > > > >> >
> > > > > >> > JB, could you help add a biweekly metrics architecture sync to
> > the
> > > > > >> Polaris
> > > > > >> > community calendar? I'm thinking Thursdays at 9-10am PT, on
> the
> > > > > >> off-weeks
> > > > > >> > from the community meeting (starting May 7), 60 minutes.
> > > > > >> >
> > > > > >> > Here's a rough agenda to work through over the first few
> > sessions,
> > > > > >> grouped
> > > > > >> > by priority:
> > > > > >> >
> > > > > >> > *First: foundational direction*
> > > > > >> >
> > > > > >> > 1.  MetricsPersistence: public SPI or internal implementation
> > > > detail?
> > > > > >> >    •   Marked @Beta, javadoc calls it a "Service Provider
> > > > Interface",
> > > > > >> but
> > > > > >> > only one consumer (JdbcBasePersistenceImpl), lives on
> > > > BasePersistence.
> > > > > >> If
> > > > > >> > demoted to a private helper inside a persisting reporter impl,
> > > most
> > > > > >> > downstream design decisions become implementation details
> rather
> > > > than
> > > > > >> > contract questions.
> > > > > >> >
> > > > > >> > 2.  Persistence schema redesign
> > > > > >> >    •   Current two-table layout (scan_metrics_report,
> > > > > >> > commit_metrics_report) with ~25 flattened columns each. Every
> > new
> > > > > metric
> > > > > >> > type requires a new table, SPI method, record class, model,
> > > > converter,
> > > > > >> and
> > > > > >> > schema migration. Direction to explore: single table with
> > > > metric_type
> > > > > >> enum,
> > > > > >> > schema_version, and JSON payload column.
> > > > > >> >
> > > > > >> > *Second: design details once direction is set*
> > > > > >> >
> > > > > >> > 3.  Partition key strategy
> > > > > >> >    •   Single-table design means scan metrics at scale will
> have
> > > > high
> > > > > >> > write concurrency per table. Schema needs to expose enough
> > > structure
> > > > > for
> > > > > >> > backends to shard by entity or time range.
> > > > > >> >
> > > > > >> > 4.  Read/write path consistency
> > > > > >> >    •   Writes go through PolarisMetricsManager on
> > > MetaStoreManager.
> > > > > >> Reads
> > > > > >> > bypass MetaStoreManager and go straight to BasePersistence,
> > > > excluding
> > > > > >> > non-JDBC backends from the read API.
> > > > > >> >
> > > > > >> > *Third: cleanup and alignment*
> > > > > >> >
> > > > > >> > 5.  PolarisMetricsReporter naming
> > > > > >> >    •   Only handles IRC (ScanReport/CommitReport), doesn't
> cover
> > > > > generic
> > > > > >> > tables or operational metrics. Name is broader than scope.
> > > > > >> >
> > > > > >> > 6.  PolarisMetricsManager facade passthrough
> > > > > >> >    •   Entire default method is
> > > > > >> callCtx.getMetaStore().writeScanReport().
> > > > > >> > Zero logic, passes Level 1 straight through to Level 3. Same
> > > > > >> anti-pattern
> > > > > >> > as PolarisEventManager.
> > > > > >> >
> > > > > >> > 7.  Iceberg community alignment
> > > > > >> >    •   Payload-type extension needs discussion on dev@iceberg.
> > > > > >> obelix74's
> > > > > >> > Feb thread got zero replies. Needs a committer voice.
> > > > > >> >
> > > > > >> > Lets confirm prioritization in the first session.
> > > > > >> >
> > > > > >> > -ej
> > > > > >> >
> > > > > >> > On Tue, Apr 21, 2026 at 3:18 PM Yufei Gu <
> flyrain000@gmail.com>
> > > > > wrote:
> > > > > >> >
> > > > > >> >> Thanks everyone for continuing to drive this forward. I agree
> > > that
> > > > > the
> > > > > >> >> problem is getting complex enough that a more structured
> > > discussion
> > > > > >> would
> > > > > >> >> help.
> > > > > >> >>
> > > > > >> >> +1 on setting up a biweekly sync for the metrics
> architecture.
> > > I’m
> > > > > >> happy
> > > > > >> >> to
> > > > > >> >> join.
> > > > > >> >>
> > > > > >> >> Yufei
> > > > > >> >>
> > > > > >> >>
> > > > > >> >> On Tue, Apr 21, 2026 at 2:34 PM EJ Wang <
> > > > > >> ej.wang.dev.interest@gmail.com>
> > > > > >> >> wrote:
> > > > > >> >>
> > > > > >> >> > Also, I've been looking more closely at the *persistence
> > schema
> > > > in
> > > > > >> the
> > > > > >> >> > current metrics work*, and I think there's a structural
> > > rigidity
> > > > > >> problem
> > > > > >> >> > worth raising before the shape gets locked in.
> > > > > >> >> >
> > > > > >> >> > Right now we have two separate tables (scan_metrics_report
> > and
> > > > > >> >> > commit_metrics_report), each with ~25 flattened columns
> that
> > > > > directly
> > > > > >> >> > mirror the Iceberg report fields. The SPI follows the same
> > > split:
> > > > > >> >> > writeScanReport and writeCommitReport as separate methods,
> > with
> > > > > >> per-type
> > > > > >> >> > record classes, converters, and model objects. *The
> practical
> > > > cost:
> > > > > >> >> > adding a new metric type (operational metrics, for example)
> > > > > requires
> > > > > >> a
> > > > > >> >> new
> > > > > >> >> > table, a new SPI method, a new record class, a new model
> > > class, a
> > > > > new
> > > > > >> >> > converter branch, and a schema migration*. That's a lot of
> > > > surface
> > > > > >> area
> > > > > >> >> > for what should be "one more kind of metric."
> > > > > >> >> >
> > > > > >> >> > *My bias* would be toward a single metrics table with *a
> > typed
> > > > JSON
> > > > > >> >> > payload*. Something like: metric_type (enum), entity_id,
> > > > > >> >> > table_identifier, snapshot_id (nullable), received_ts,
> > > > > >> schema_version,
> > > > > >> >> and
> > > > > >> >> > a payload column for the metric-specific data. The
> > metric_type
> > > +
> > > > > >> >> > schema_version pair gives us a forward-compatible contract
> > for
> > > > the
> > > > > >> >> payload
> > > > > >> >> > shape. Adding a new metric type becomes an enum value and a
> > > > payload
> > > > > >> >> schema,
> > > > > >> >> > not a schema migration. One thing I think we need to be
> > > > deliberate
> > > > > >> >> about is
> > > > > >> >> > the partition key design. If all metric types land in one
> > > table,
> > > > > scan
> > > > > >> >> > metrics at scale (high concurrency, high frequency across
> > many
> > > > > >> tables)
> > > > > >> >> > could easily create hot partitions. We'd want the
> persistence
> > > > layer
> > > > > >> to
> > > > > >> >> be
> > > > > >> >> > able to shard by entity or time range, and that means the
> > > logical
> > > > > >> schema
> > > > > >> >> > needs to expose enough structure for backends to partition
> > on.
> > > I
> > > > > >> don't
> > > > > >> >> > think the current flattened layout gives us that.
> > > > > >> >> >
> > > > > >> >> > This is getting complex enough that I don't think ad-hoc
> > PR/ML
> > > > > >> threads
> > > > > >> >> > will converge well. *Would people be open to a biweekly
> sync
> > > for
> > > > > >> metrics
> > > > > >> >> > architecture?* I think 30 minutes every two weeks with
> > > interested
> > > > > >> >> parties
> > > > > >> >> > would be enough to work through the schema, SPI shape, and
> > read
> > > > API
> > > > > >> >> design
> > > > > >> >> > together. Happy to help set that up.
> > > > > >> >> >
> > > > > >> >> > -ej
> > > > > >> >> >
> > > > > >> >> > On Mon, Apr 20, 2026 at 2:19 PM EJ Wang <
> > > > > >> ej.wang.dev.interest@gmail.com
> > > > > >> >> >
> > > > > >> >> > wrote:
> > > > > >> >> >
> > > > > >> >> >> Reviewed #4115, left a comment on the code organization
> > side.
> > > > > >> >> >>
> > > > > >> >> >> One thing stood out: the metrics write path enters through
> > > > > >> >> >> PolarisMetricsManager on MetaStoreManager, but the new
> read
> > > path
> > > > > >> >> bypasses
> > > > > >> >> >> MetaStoreManager entirely and goes straight to
> > BasePersistence
> > > > via
> > > > > >> >> >> callContext.getMetaStore(). That means the read API only
> > works
> > > > for
> > > > > >> >> backends
> > > > > >> >> >> that implement BasePersistence. NoSQL and remote backends
> > > can't
> > > > > >> >> participate.
> > > > > >> >> >>
> > > > > >> >> >> Stepping back, I think the metrics subsystem is growing
> into
> > > > > >> something
> > > > > >> >> >> real (write + read + REST API + AuthZ + pagination) *but
> the
> > > > > >> >> persistence
> > > > > >> >> >> side is split across two layers in a way that's hard to
> > > > extend*. I
> > > > > >> put
> > > > > >> >> >> together two diagrams to show what I mean (my best
> effort).
> > > > > >> >> >>
> > > > > >> >> >> *Current state* (Diagram 1): three interfaces at three
> > > different
> > > > > >> >> levels.
> > > > > >> >> >> The engine-facing SPI (PolarisMetricsReporter) is clean.
> But
> > > > > >> >> >> PolarisMetricsManager on MetaStoreManager is a passthrough
> > to
> > > > > >> >> >> MetricsPersistence on BasePersistence. The @Beta
> annotation
> > > and
> > > > > SPI
> > > > > >> >> javadoc
> > > > > >> >> >> are on the BasePersistence layer, while the actual
> extension
> > > > > points
> > > > > >> >> >> (PolarisMetricsReporter, PolarisMetricsManager) carry no
> > > > stability
> > > > > >> >> >> annotation. The write path goes through the
> MetaStoreManager
> > > > > layer,
> > > > > >> the
> > > > > >> >> >> read path doesn't.
> > > > > >> >> >>
> > > > > >> >> >> *What I envision* (Diagram 2): two SPIs at two levels.
> > > > > >> >> >> PolarisMetricsReporter stays as the engine-facing SPI.
> > > > > >> >> >> PolarisMetricsManager becomes the backend-facing SPI with
> > both
> > > > > write
> > > > > >> >> and
> > > > > >> >> >> read methods at the MetaStoreManager level, where any
> > backend
> > > > > (JDBC,
> > > > > >> >> NoSQL,
> > > > > >> >> >> remote) can implement them. MetricsPersistence on
> > > > BasePersistence
> > > > > >> goes
> > > > > >> >> >> away. Where metrics actually land is an implementation
> > detail,
> > > > > not a
> > > > > >> >> core
> > > > > >> >> >> interface.
> > > > > >> >> >>
> > > > > >> >> >> *Minor naming thing*: PolarisMetricsReporter is broader
> than
> > > > what
> > > > > it
> > > > > >> >> >> actually handles. It only accepts Iceberg REST Catalog
> > metrics
> > > > > >> >> (ScanReport,
> > > > > >> >> >> CommitReport via MetricsReport). Generic table metrics or
> > > > > >> operational
> > > > > >> >> >> metrics aren't in scope. Not blocking, but worth noting if
> > the
> > > > > >> metrics
> > > > > >> >> >> surface expands.
> > > > > >> >> >>
> > > > > >> >> >> *Rough sketch of how to get there*:
> > > > > >> >> >>  1.  Add read methods to PolarisMetricsManager
> > > (listScanReports,
> > > > > >> >> >> listCommitReports) with default no-op, same as the
> existing
> > > > write
> > > > > >> >> methods.
> > > > > >> >> >> (Probably make PolarisMetricsManager more explicit on
> being
> > > > > Iceberg
> > > > > >> >> >> specific like package name or class name etc.)
> > > > > >> >> >>  2.  Wire MetricsReportsService through MetaStoreManager
> > > instead
> > > > > of
> > > > > >> >> >> callContext.getMetaStore().
> > > > > >> >> >>  3.  Extract metrics persistence from
> > JdbcBasePersistenceImpl
> > > > into
> > > > > >> its
> > > > > >> >> >> own class. That file carries ~7 responsibilities, metrics
> > > being
> > > > > one
> > > > > >> of
> > > > > >> >> them.
> > > > > >> >> >>  4.  Remove MetricsPersistence from BasePersistence.
> > > > > >> >> >>
> > > > > >> >> >> *None of this needs to happen in #4115. But if the
> direction
> > > > makes
> > > > > >> >> sense,
> > > > > >> >> >> it would be good to align before the metrics surface grows
> > > > > further.
> > > > > >> >> Curious
> > > > > >> >> >> what others think.*
> > > > > >> >> >>
> > > > > >> >> >> *My mental model note*: Level 1 MetaStoreManager; level 2
> > > > > >> transactional
> > > > > >> >> >> persistence; level 3 base persistence
> > > > > >> >> >>
> > > > > >> >> >> Diagram 1
> > > > > >> >> >> <
> > > > > >> >>
> > > > > >>
> > > > >
> > > >
> > >
> >
> https://www.plantuml.com/plantuml/uml/bLHDR-Cs4BthLmpIYupw0zbkKQ1r3M-S7Bp8xhhM7WCOb3IM65EaGD9EX2RzxHrHb4CxRelwa4YSDu_lpOVcnZ9jzvM8BBS2uGjQpJC3dtHMSekPtMk44IpsMgEqa5XcCOhCZikQQLP1pR8TAp2n3ILhmZDP20m0fcIvUkAoW2qJXd9z1bpToO9BX3WXu0ucy5rpgGPNm0nW5_epUWtm2Ue3pn3kMOFQmKntGZW0BYtgBSi8k5A2QMwybJNMIbFiGSR9QZc4nUqIvikStF0jHprua5C-amge42aNt3R0f5JaaoivdV2Pkqbx4hee4ymOkBh5BTiB-_uIeGeo8zL8rPsPl4DktdEiK1jkB1NdZCRbrSTecDe_mlHbF0wvBmCkaOH5_S8a_TTTKI6-nmCAkEw4LpxsZ-LbYLKQFKMNOgf_wuM7_bV9gOer5SYMMksBSWXFcbi49KNZXNLicwfe3TETC7gPdPqI7uBcHMb1RSzYq34c6PDUM9mn8HRsUTZEiDBve3NjVZumBj0U7SS37mGO7vcwtiK-_pU7U7L_f-digo9YbhSwIfMRwIITKGXbxdIUTCGF1SeCJxloKsU-3k9ddRbX1eDq1q_fx1JbBGT0glVyXimDuP4TQ5qpCAmnGEj2s_6n5mtn1z-97-63itFQZLPO1Ev2tu_WF7Ju-VPc0Skg5bYXxBhkY1xpD7EM_7fyflSpIsqMgVth5xhVr4eQxWQ8enaSAJQSG16yFSDuJ798rrcXr_3n-lfdk7icQjEBmFujL7AodiP_Y4Z7-YxvtZNs4zMgpNTl6tF8sglyPsmqchrjvQ-m-aP94r-TwCA2Ka8upPJZwtvSpoYCXkYMZU2NXvRMBfq9P3i3Le4VAZUAlUZ_oPKsxPgY0Q_BSKLkyr9bhQhQrJjo_x3TPlIB0DPjnMfcIoYP0QaYw1a0fTKDr8fB6ntNuvmoL1ZGkXa69Njh43zf9GiGxHQrA_jDYWRSzF5--WmTVrN97_Sm8LbLUy_lGBmLanJjFkDlGkRqjA_4tm00
> > > > > >> >> >
> > > > > >> >> >> :
> > > > > >> >> >>
> > > > > >> >> >> [image: image.png]
> > > > > >> >> >>
> > > > > >> >> >> Diagram 2
> > > > > >> >> >> <
> > > > > >> >>
> > > > > >>
> > > > >
> > > >
> > >
> >
> https://www.plantuml.com/plantuml/uml/VLLDR-8m4BtdLupO2sWBLVU8AaGB7AXAbssGzb896SS9RXqxjKqBwkv_tt7iV43fdaZYDpFlpRmnOsE9jhjSH9PRmM31hERKm8scMsuPjJlDe0yheZDc8RR4iYWoBrmMH9CS2a9VICPYUy1OZN0YCy5Q0BCbYNhdCeEK28En8G8wCvbnoQ0R8_05Bc6bkLIz3X03p1zzH7zR-9ZfDquPt9C3qoNCX2yV4G2NbkcKu5jdgGJHt0GbZwnG6i-UP3TUpk5gM6Ldqke350eZUqzoCft3U9xWHvxoa5-7K4nF1J46EbEMafsmdrCBbQ44gVggy18IZrn_ph5asd1ZiIKdQSgueZvjXrQFSFrdC3YN-nXmBacxbGiYyLVxLaBtdhqn0LSzdBDhqQtQoOJeGyad3z0lUqnYgpGB6Ns8oVyta00Dy_WnX0tIOZ8v6SYxHll1TrH6aejAik-mh-AphVFCwSUQqFypElag5QRGFDjQKEd96K1P8QP41c9TzA_IIQyvdAWyv_RSiS3skb0_EzDDkK2v5xWF6MiGFlvhpFLcD2Dq2pml14gaF67eQkmd8gulDoC4kSOu6KVpkvlUJg1RTbWISU40RdBUUS_9XfRZ2dwxm_SW8LYFISgm_MnlDQ6M9P1gbKEc4X-2pH_FvJCkCqm9pbVjD6LrwdLeOrDWfOaqc8Wh9BE85oNKxkNQ6o4yGRy_Eae0G_G8tZv81d3bHDB23WOdisohVr3nh_j6lbSjbNaLRTc8UgtPbAU1J_tygOfZX9DWEJeHDvYx-qmSi5FgNLPZwHrHcUsncGQ5-skhUclpE5fo4ounpFauYrUbkU6ccfnxMvitwag4IyerhTxj8In_Oj1bDO4pQru674loYrGlULHLEGCjwJJ8gDoVZR8MxO4BT3IzRvIcAQKezC6xpziGnTyImrfEGyJI_OcKfgtxIvnTqFEMS17L9Z-jsARN5FmTheP7HtSdtOMT0B4GY2FYHXxgQmMtj2bRqiLFGapiVe1_QVKDrkqXcm83aFEXnMYCZ-xlyHy
> > > > > >> >> >
> > > > > >> >> >> :
> > > > > >> >> >> [image: image.png]
> > > > > >> >> >>
> > > > > >> >> >>  -ej
> > > > > >> >> >>
> > > > > >> >> >> On Wed, Apr 15, 2026 at 8:22 AM Dmitri Bourlatchkov <
> > > > > >> dimas@apache.org>
> > > > > >> >> >> wrote:
> > > > > >> >> >>
> > > > > >> >> >>> Hi All,
> > > > > >> >> >>>
> > > > > >> >> >>> Heads up: The current state of PR [4115] looks pretty
> solid
> > > to
> > > > > me.
> > > > > >> I
> > > > > >> >> >>> believe this PR is approaching a mergeable condition.
> > > > > >> >> >>>
> > > > > >> >> >>> Please post your reviews if you have any comments.
> > > > > >> >> >>>
> > > > > >> >> >>> [4115] https://github.com/apache/polaris/pull/4115
> > > > > >> >> >>>
> > > > > >> >> >>> Thanks,
> > > > > >> >> >>> Dmitri.
> > > > > >> >> >>>
> > > > > >> >> >>> On Tue, Mar 3, 2026 at 3:29 PM Anand Kumar Sankaran via
> > dev <
> > > > > >> >> >>> dev@polaris.apache.org> wrote:
> > > > > >> >> >>>
> > > > > >> >> >>> > Hi Yufei and Dmitri,
> > > > > >> >> >>> >
> > > > > >> >> >>> > Here is a proposal for the REST endpoints for metrics
> and
> > > > > events.
> > > > > >> >> >>> >
> > > > > >> >> >>> > https://github.com/apache/polaris/pull/3924/changes
> > > > > >> >> >>> >
> > > > > >> >> >>> > I did not see any precursors for raising a PR for
> > > proposals,
> > > > so
> > > > > >> >> trying
> > > > > >> >> >>> > this.  Please let me know what you think.
> > > > > >> >> >>> >
> > > > > >> >> >>> > -
> > > > > >> >> >>> > Anand
> > > > > >> >> >>> >
> > > > > >> >> >>> > From: Anand Kumar Sankaran <anand.sankaran@workday.com
> >
> > > > > >> >> >>> > Date: Monday, March 2, 2026 at 10:25 AM
> > > > > >> >> >>> > To: dev@polaris.apache.org <de...@polaris.apache.org>
> > > > > >> >> >>> > Subject: Re: Polaris Telemetry and Audit Trail
> > > > > >> >> >>> >
> > > > > >> >> >>> > About the REST API, based on my use cases:
> > > > > >> >> >>> >
> > > > > >> >> >>> >
> > > > > >> >> >>> >   1.
> > > > > >> >> >>> > I want to be able to query commit metrics to track
> files
> > > > added
> > > > > /
> > > > > >> >> >>> removed
> > > > > >> >> >>> > per commit, along with record counts. The ingestion
> > > pipeline
> > > > > that
> > > > > >> >> >>> writes
> > > > > >> >> >>> > this data is owned by us and we are guaranteed to write
> > > this
> > > > > >> >> >>> information
> > > > > >> >> >>> > for each write.
> > > > > >> >> >>> >   2.
> > > > > >> >> >>> > I want to be able to query scan metrics for read. I
> > > > understand
> > > > > >> >> clients
> > > > > >> >> >>> do
> > > > > >> >> >>> > not fulfill this requirement.
> > > > > >> >> >>> >   3.
> > > > > >> >> >>> > I want to be able to query the events table (events are
> > > > > >> persisted) -
> > > > > >> >> >>> this
> > > > > >> >> >>> > may supersede #2, I am not sure yet.
> > > > > >> >> >>> >
> > > > > >> >> >>> > All this information is in the JDBC based persistence
> > model
> > > > and
> > > > > >> is
> > > > > >> >> >>> > persisted in the metastore. I currently don’t have a
> need
> > > to
> > > > > >> query
> > > > > >> >> >>> > prometheus or open telemetry. I do publish some events
> to
> > > > > >> Prometheus
> > > > > >> >> >>> and
> > > > > >> >> >>> > they are forwarded to our dashboards elsewhere.
> > > > > >> >> >>> >
> > > > > >> >> >>> > About the CLI utilities, I meant the admin user
> > utilities.
> > > In
> > > > > >> one of
> > > > > >> >> >>> the
> > > > > >> >> >>> > earliest drafts of my proposal, Prashant mentioned that
> > the
> > > > > >> metrics
> > > > > >> >> >>> tables
> > > > > >> >> >>> > can grow indefinitely and that a similar problem exists
> > > with
> > > > > the
> > > > > >> >> events
> > > > > >> >> >>> > table as well. We discussed that cleaning up of old
> > records
> > > > > from
> > > > > >> >> both
> > > > > >> >> >>> > metrics tables and events tables can be done via a CLI
> > > > utility.
> > > > > >> >> >>> >
> > > > > >> >> >>> > I see that Yufei has covered the discussion about
> > > > datasources.
> > > > > >> >> >>> >
> > > > > >> >> >>> > -
> > > > > >> >> >>> > Anand
> > > > > >> >> >>> >
> > > > > >> >> >>> >
> > > > > >> >> >>> >
> > > > > >> >> >>> > From: Yufei Gu <fl...@gmail.com>
> > > > > >> >> >>> > Date: Friday, February 27, 2026 at 9:54 PM
> > > > > >> >> >>> > To: dev@polaris.apache.org <de...@polaris.apache.org>
> > > > > >> >> >>> > Subject: Re: Polaris Telemetry and Audit Trail
> > > > > >> >> >>> >
> > > > > >> >> >>> > This Message Is From an External Sender
> > > > > >> >> >>> > This message came from outside your organization.
> > > > > >> >> >>> > Report Suspicious<
> > > > > >> >> >>> >
> > > > > >> >> >>>
> > > > > >> >>
> > > > > >>
> > > > >
> > > >
> > >
> >
> https://us-phishalarm-ewt.proofpoint.com/EWT/v1/Iz9xO38YGHZK!YhNDZABkHi1B699ote2uMwpOZw8i0QMCGO2Szc-HshuABGhGvwPJcymE6G2oUUxtS8xDkSrtGTPm_I3QnVDHoLMk50m9v8z_nZKTkd-bnVUbreF1u0WnfV_X5eYevZl_$
> > > > > >> >> >>> > >
> > > > > >> >> >>> >
> > > > > >> >> >>> >
> > > > > >> >> >>> > As I mentioned in
> > > > > >> >> >>> >
> > > > > >> >> >>>
> > > > > >> >>
> > > > > >>
> > > > >
> > > >
> > >
> >
> https://urldefense.com/v3/__https://github.com/apache/polaris/issues/3890__;!!Iz9xO38YGHZK!5EuyFFkk3vhRWVIRvQAWBSQfpJkTMA9HxugzDwXmN0LPPqhEFxYkFRGVhtb8AqUwXtDh2OplcMnbMDHKOxrvDU0$
> > > > > >> >> >>> ,
> > > > > >> >> >>> > supporting
> > > > > >> >> >>> > multiple data sources is not a trivial change. I would
> > > > strongly
> > > > > >> >> >>> recommend
> > > > > >> >> >>> > starting with a design document to carefully evaluate
> the
> > > > > >> >> architectural
> > > > > >> >> >>> > implications and long term impact.
> > > > > >> >> >>> >
> > > > > >> >> >>> > A REST endpoint to query metrics seems reasonable given
> > the
> > > > > >> current
> > > > > >> >> >>> JDBC
> > > > > >> >> >>> > based persistence model. That said, we may also
> consider
> > > > > >> alternative
> > > > > >> >> >>> > storage models. For example, if we later adopt a time
> > > series
> > > > > >> system
> > > > > >> >> >>> such as
> > > > > >> >> >>> > Prometheus to store metrics, the query model and access
> > > > > patterns
> > > > > >> >> would
> > > > > >> >> >>> be
> > > > > >> >> >>> > fundamentally different. Designing the REST API without
> > > > > >> considering
> > > > > >> >> >>> these
> > > > > >> >> >>> > potential evolutions may limit flexibility. I'd suggest
> > to
> > > > > start
> > > > > >> >> with
> > > > > >> >> >>> the
> > > > > >> >> >>> > use case.
> > > > > >> >> >>> >
> > > > > >> >> >>> > Yufei
> > > > > >> >> >>> >
> > > > > >> >> >>> >
> > > > > >> >> >>> > On Fri, Feb 27, 2026 at 3:42 PM Dmitri Bourlatchkov <
> > > > > >> >> dimas@apache.org>
> > > > > >> >> >>> > wrote:
> > > > > >> >> >>> >
> > > > > >> >> >>> > > Hi Anand,
> > > > > >> >> >>> > >
> > > > > >> >> >>> > > Sharing my view... subject to discussion:
> > > > > >> >> >>> > >
> > > > > >> >> >>> > > 1. Adding non-IRC REST API to Polaris is perfectly
> > fine.
> > > > > >> >> >>> > >
> > > > > >> >> >>> > > Figuring out specific endpoint URIs and payloads
> might
> > > > > require
> > > > > >> a
> > > > > >> >> few
> > > > > >> >> >>> > > roundtrips, so opening a separate thread for that
> might
> > > be
> > > > > >> best.
> > > > > >> >> >>> > > Contributors commonly create Google Docs for new API
> > > > > proposals
> > > > > >> too
> > > > > >> >> >>> (they
> > > > > >> >> >>> > > fairly easy to update as the email discussion
> > > progresses).
> > > > > >> >> >>> > >
> > > > > >> >> >>> > > There was a suggestion to try Markdown (with PRs) for
> > > > > proposals
> > > > > >> >> [1]
> > > > > >> >> >>> ...
> > > > > >> >> >>> > > feel free to give it a try if you are comfortable
> with
> > > > that.
> > > > > >> >> >>> > >
> > > > > >> >> >>> > > 2. Could you clarify whether you mean end user
> > utilities
> > > or
> > > > > >> admin
> > > > > >> >> >>> user
> > > > > >> >> >>> > > utilities? In the latter case those might be more
> > > suitable
> > > > > for
> > > > > >> the
> > > > > >> >> >>> Admin
> > > > > >> >> >>> > > CLI (java) not the Python CLI, IMHO.
> > > > > >> >> >>> > >
> > > > > >> >> >>> > > Why would these utilities be common with events?
> IMHO,
> > > > event
> > > > > >> use
> > > > > >> >> >>> cases
> > > > > >> >> >>> > are
> > > > > >> >> >>> > > distinct from scan/commit metrics.
> > > > > >> >> >>> > >
> > > > > >> >> >>> > > 3. I'd prefer separating metrics persistence from
> > > MetaStore
> > > > > >> >> >>> persistence
> > > > > >> >> >>> > at
> > > > > >> >> >>> > > the code level, so that they could be mixed and
> matched
> > > > > >> >> >>> independently.
> > > > > >> >> >>> > The
> > > > > >> >> >>> > > separate datasource question will become a non-issue
> > with
> > > > > that
> > > > > >> >> >>> approach,
> > > > > >> >> >>> > I
> > > > > >> >> >>> > > guess.
> > > > > >> >> >>> > >
> > > > > >> >> >>> > > The rationale for separating scan metrics and
> metastore
> > > > > >> >> persistence
> > > > > >> >> >>> is
> > > > > >> >> >>> > that
> > > > > >> >> >>> > > "cascading deletes" between them are hardly ever
> > > required.
> > > > > >> >> >>> Furthermore,
> > > > > >> >> >>> > the
> > > > > >> >> >>> > > data and query patterns are very different so
> different
> > > > > >> >> technologies
> > > > > >> >> >>> > might
> > > > > >> >> >>> > > be beneficial in each case.
> > > > > >> >> >>> > >
> > > > > >> >> >>> > > [1]
> > > > > >> >> >>> >
> > > > > >> >> >>>
> > > > > >> >>
> > > > > >>
> > > > >
> > > >
> > >
> >
> https://urldefense.com/v3/__https://lists.apache.org/thread/yto2wp982t43h1mqjwnslswhws5z47cy__;!!Iz9xO38YGHZK!5EuyFFkk3vhRWVIRvQAWBSQfpJkTMA9HxugzDwXmN0LPPqhEFxYkFRGVhtb8AqUwXtDh2OplcMnbMDHKxYDakNU$
> > > > > >> >> >>> > >
> > > > > >> >> >>> > > Cheers,
> > > > > >> >> >>> > > Dmitri.
> > > > > >> >> >>> > >
> > > > > >> >> >>> > > On Fri, Feb 27, 2026 at 6:19 PM Anand Kumar Sankaran
> > via
> > > > dev
> > > > > <
> > > > > >> >> >>> > > dev@polaris.apache.org> wrote:
> > > > > >> >> >>> > >
> > > > > >> >> >>> > > > Thanks all. This PR is merged now.
> > > > > >> >> >>> > > >
> > > > > >> >> >>> > > > Here are the follow-up features / work needed.
> These
> > > > were
> > > > > >> all
> > > > > >> >> >>> part of
> > > > > >> >> >>> > > the
> > > > > >> >> >>> > > > merged PR at some point in time and were removed to
> > > > reduce
> > > > > >> >> scope.
> > > > > >> >> >>> > > >
> > > > > >> >> >>> > > > Please let me know what you think.
> > > > > >> >> >>> > > >
> > > > > >> >> >>> > > >
> > > > > >> >> >>> > > >   1.  A REST API to paginate through table metrics.
> > > This
> > > > > >> will be
> > > > > >> >> >>> > non-IRC
> > > > > >> >> >>> > > > standard addition.
> > > > > >> >> >>> > > >   2.  Utilities for managing old records, should be
> > > > common
> > > > > >> with
> > > > > >> >> >>> events.
> > > > > >> >> >>> > > > There was some discussion that it belongs to the
> CLI.
> > > > > >> >> >>> > > >   3.  Separate datasource (metrics, events, even
> > other
> > > > > >> tables?).
> > > > > >> >> >>> > > >
> > > > > >> >> >>> > > >
> > > > > >> >> >>> > > > Anything else?
> > > > > >> >> >>> > > >
> > > > > >> >> >>> > > > -
> > > > > >> >> >>> > > > Anand
> > > > > >> >> >>> > > >
> > > > > >> >> >>> > > >
> > > > > >> >> >>> > >
> > > > > >> >> >>> >
> > > > > >> >> >>> >
> > > > > >> >> >>>
> > > > > >> >> >>
> > > > > >> >>
> > > > > >> >
> > > > > >>
> > > > > >
> > > > >
> > > >
> > >
> >
>

Re: Proposal for REST endpoints for table metrics and events

[Dmitri Bourlatchkov <di...@apache.org>]

Hi EJ,

Thanks for the recap / summary!

> Do folks agree that the stable SPI boundary should be
metrics reporting/emitting, not metrics persistence?

SGTM.

> Does an optional Iceberg metrics extension API layer sound like the right
home for this SPI?

How is that different from current PolarisMetricsReporter?

> Should the current durable metrics work be reframed as a durable
JDBC reference implementation of that SPI?

SGTM.

> What is the smallest PR sequence to get there without blocking
the current metrics work unnecessarily?

Let's get https://github.com/apache/polaris/pull/4397 merged first.

Then, I'd propose isolating the Metrics schema from the MetaStore schema.
This will probably have some ripple effect into the bootstrap workflows, so
it's not a trivial change.

Then, let's reassess.

Cheers,
Dmitri.

On Wed, Jun 3, 2026 at 5:40 PM EJ Wang <ej...@gmail.com>
wrote:

> Hi folks,
>
> Thanks again for the discussion today. I updated the sync doc with notes
> from the third metrics architecture sync:
>
>
> https://docs.google.com/document/d/100h7c4damrUzVuquYbBHM0EvA4LSWuW2IT2dN_7nYVA/edit?tab=t.k96s2xyqr5u1
>
> A few highlights from today’s discussion:
>
>    - We clarified that Iceberg metrics reporting can be interpreted either
>    as sync or async handling from Polaris’s perspective, so Polaris should
>    stay flexible as a platform instead of baking one handling model into
> the
>    REST API behavior.
>    - We aligned that the built-in (battery) behavior for metrics emitting
>    can stay simple: no-op/log-only is enough as the default.
>    - Durable metrics persistence should be treated as an implementation of
>    the metrics reporting path, not as the core SPI boundary.
>    - The existing durable metrics work can be reviewed as a reference
>    implementation of the reporting SPI, with persistence-related logic kept
>    self-contained under a metrics durable implementation module rather than
>    scattered through core entity persistence.
>    - Dashboard/insights remains a real use case, but we agreed to keep it
>    separate from the core metrics intake discussion for now.
>
> I also did a quick source check after the meeting to make sure we are
> describing the current state accurately.
>
> Current state:
>
>    - Polaris already has a metrics reporting hook: PolarisMetricsReporter.
>    - The default implementation is DefaultMetricsReporter, selected by
>    polaris.iceberg-metrics.reporting.type=default. It is log-only,
>    effectively quiet unless metrics logging is enabled.
>    - There is also a PersistingMetricsReporter, selected by
>    polaris.iceberg-metrics.reporting.type=persisting, which converts
>    Iceberg scan/commit reports into Polaris metrics records and writes
> through
>    PolarisMetricsManager -> MetricsPersistence.
>    - MetricsPersistence currently exists in the persistence layer and is
>    inherited by BasePersistence.
>
> My read from the discussion is that the target boundary should not be
> MetricsPersistence as inherited by BasePersistence. That path should be
> decomposed as durable implementation detail (taken care of by
> https://github.com/apache/polaris/pull/4397). The stable extension point
> should instead be the metrics reporting/emitting boundary.
>
> Concretely, I think the next proposal should be shaped like this:
>
>    1.
>
>    Define the proper metrics reporting SPI at an optional Iceberg metrics
>    extension API layer.
>
>    Example direction:
>
>    public interface IcebergMetricsReporter {
>      void reportMetric(IcebergMetricsReportContext context,
> MetricsReport report);
>    }
>
>    The context would carry the small Polaris-resolved envelope:
>    catalog/table identity, report type, received timestamp, and
> request/trace
>    context if available. The raw Iceberg MetricsReport remains the payload.
>    2.
>
>    Keep runtime/service as the REST ingestion and wiring layer.
>
>    The REST handler still resolves the table and performs authz before
>    accepting the report. After that, it calls the selected reporting
>    implementation.
>    3.
>
>    Keep the battery default no-op/log-only.
>
>    This preserves an out-of-box safe default and avoids requiring a durable
>    metrics store for every Polaris deployment.
>    4.
>
>    Move durable JDBC metrics into a self-contained implementation of the
>    reporting SPI.
>
>    That implementation can own its schema, bootstrap, retention, and read
>    API support. It should not define the core reporting SPI boundary, and
> it
>    should not require metrics persistence to remain inherited from
>    BasePersistence.
>    5.
>
>    Treat async/event-backed handling as another implementation of the same
>    reporting SPI.
>
>    For example, an event-backed reporter could enqueue a metrics event and
>    let listeners handle durable storage or other sinks. If we later need a
>    replaceable queue engine, that seems like a shared event/metrics
> substrate
>    topic rather than a metrics-only requirement.
>
> This framing lets us keep the REST metrics endpoint simple, preserve the
> current default behavior, support durable metrics users, and still leave
> room for async/event-backed or external-queue-based implementations.
>
> I think the main follow-up questions are:
>
>    - Do folks agree that the stable SPI boundary should be metrics
>    reporting/emitting, not metrics persistence?
>    - Does an optional Iceberg metrics extension API layer sound like the
>    right home for this SPI?
>    - Should the current durable metrics work be reframed as a durable JDBC
>    reference implementation of that SPI?
>    - What is the smallest PR sequence to get there without blocking the
>    current metrics work unnecessarily?
>
> Thanks,
> -ej
>
> On Fri, May 15, 2026 at 5:16 PM Dmitri Bourlatchkov <di...@apache.org>
> wrote:
>
> > Hi JB,
> >
> > Could you set up another meeting, please? Same time on Wednesday as last
> > time... I hope it works for everyone.
> >
> > Cheers,
> > Dmitri.
> >
> > On Fri, May 15, 2026 at 8:06 PM Yufei Gu <fl...@gmail.com> wrote:
> >
> > > +1 on another sync call next week.
> > >
> > > Yufei
> > >
> > >
> > > On Fri, May 15, 2026 at 4:52 PM Dmitri Bourlatchkov <di...@apache.org>
> > > wrote:
> > >
> > > > Hi All,
> > > >
> > > > WDYT about another sync call next week?
> > > >
> > > > Thanks,
> > > > Dmitri.
> > > >
> > > > On Wed, May 6, 2026 at 5:29 PM Dmitri Bourlatchkov <dimas@apache.org
> >
> > > > wrote:
> > > >
> > > > > Hi EJ,
> > > > >
> > > > > Thanks for the summary! It covers what we discussed in the meeting
> > very
> > > > > well, IMHO.
> > > > >
> > > > > Looking forward to concrete PRs :)
> > > > >
> > > > > Cheers,
> > > > > Dmitri.
> > > > >
> > > > > On Wed, May 6, 2026 at 5:08 PM EJ Wang <
> > ej.wang.dev.interest@gmail.com
> > > >
> > > > > wrote:
> > > > >
> > > > >> Hi folks,
> > > > >>
> > > > >> We had a community sync earlier, thanks JB for scheduling it.
> Notes
> > > from
> > > > >> the first metrics architecture sync (May 6, 10-11am PT).
> Discussion
> > > doc
> > > > >> with per-section status:
> > > > >>
> > > > >>
> > > >
> > >
> >
> https://docs.google.com/document/d/100h7c4damrUzVuquYbBHM0EvA4LSWuW2IT2dN_7nYVA/edit?tab=t.0
> > > > >>
> > > > >> *The meeting covered both topics from the doc. Direction-level
> > > alignment
> > > > >> was reached on the headline pieces; details remain for PR review
> or
> > > > >> follow-up sessions.*
> > > > >>
> > > > >> *Topic 1 — Persistence schema redesign*
> > > > >> Idea-level alignment on consolidating per-type tables
> > > > >> (scan_metrics_report,
> > > > >> commit_metrics_report) into a single metrics_report table. The
> > > > motivating
> > > > >> cost is the surface area added by every new metric type today: new
> > > > table,
> > > > >> SPI method, record class, model, converter, schema migration.
> > > > >>
> > > > >> Most schema details are deferred to the schema PR. A few specific
> > > points
> > > > >> came up:
> > > > >> •   metric_schema_version: Yufei prefers dropping it, since there
> is
> > > no
> > > > >> spec-level concept of metrics versioning today and it is hard to
> > > define
> > > > >> unilaterally. Robert prefers keeping it, given IRC v2 is coming
> and
> > > the
> > > > >> schema should be considered against its likely shape; Robert also
> > > raised
> > > > >> how to differentiate various payload formats if any. EJ's read is
> > that
> > > > >> this
> > > > >> is a two-way-door decision. We can start without the field, and if
> > IRC
> > > > v2
> > > > >> changes the shape we would likely roll a corresponding new schema
> > > > anyway,
> > > > >> which is not particularly costly.
> > > > >> •   Payload format: Robert pointed out that future formats beyond
> > JSON
> > > > may
> > > > >> be worth supporting. The exact shape is deferred to the schema
> > > > discussion.
> > > > >> •   Partition strategy: Anand suggested monthly partitioning based
> > on
> > > > his
> > > > >> experience as potentially helpful at scale.
> > > > >>
> > > > >> *Topic 2 — Where metrics ingestion and storage belong*
> > > > >> Idea-level alignment that metrics should be a separated SPI from
> the
> > > > >> entity
> > > > >> persistence stack. Two reasons surfaced: (a) workloads and
> > capability
> > > > >> requirements diverge enough that coupling them creates artificial
> > > > >> constraints, and (b) admin experience improves when metrics has
> its
> > > own
> > > > >> bootstrap, retention, and lifecycle. Dmitri noted Polaris being a
> > > > platform
> > > > >> should have the flexibility to support different persistence
> > backends
> > > > per
> > > > >> concern, and pointed to a concrete next step of separating the
> JDBC
> > > > >> bootstrap for metrics from the metastore bootstrap. Robert
> proposed
> > an
> > > > >> additional UX extension: detect an unbootstrapped metrics store on
> > > first
> > > > >> use and auto-bootstrap rather than requiring an explicit manual
> > > > bootstrap
> > > > >> step.
> > > > >> The meeting also confirmed that Polaris metrics can start small
> and
> > > stay
> > > > >> Iceberg-focused. Naming and persistence schema can lean
> > > > Iceberg-specific.
> > > > >> If a future expansion to generic-table metrics or operational
> > metrics
> > > > >> arrives, an abstraction layer can be built on top of the Iceberg
> > > metrics
> > > > >> reporter at that point. Robert remains on the fence and would
> prefer
> > > > >> something more generic but did not block the direction; Dmitri's
> > read
> > > > was
> > > > >> that the proposed framework already has enough flexibility to
> absorb
> > > > >> future
> > > > >> expansion.
> > > > >>
> > > > >> The Trade-offs and Proposed structure sections in the doc were not
> > > > >> reviewed
> > > > >> in detail. They remain open for either the next sync or PR review.
> > > > >>
> > > > >> *Cross-cutting alignment — battery-included plus pluggable*
> > > > >> A common philosophy emerged from the discussion. EJ summarized it
> > as:
> > > > >> Polaris should provide a battery-included UX for beginners and the
> > > > >> flexibility for advanced users to swap the included battery for
> > > > something
> > > > >> more powerful or tailored to their use case. The SPI design needs
> to
> > > > >> enable
> > > > >> both.
> > > > >>
> > > > >> The inputs that shaped this framing:
> > > > >> •   Anand described how his team uses the current metrics
> > persistence
> > > > >> (three metrics consumers in v1.4).
> > > > >> •   Yufei raised Grafana and dashboard integrations as a
> destination
> > > use
> > > > >> case beyond the default.
> > > > >> •   Robert called out that the current design is more
> JDBC-focused.
> > > > >>
> > > > >> Two concrete instances:
> > > > >> •   Async metrics intake: Yufei's initial position was that async
> > > should
> > > > >> largely live on the producer side and there is not much Polaris
> can
> > > do.
> > > > >> Robert suggested a Polaris-side default is doable via Vert.x.
> Dmitri
> > > > >> agreed
> > > > >> the direction is worth exploring. The meeting converged on a
> > > > >> battery-included default (likely Vert.x-backed) with an SPI shape
> > that
> > > > >> lets
> > > > >> power users route to a more scalable backend (k8s-hosted queue,
> AWS
> > > SQS,
> > > > >> etc.).
> > > > >> •   Pluggable destinations: combining Yufei's dashboard use case
> > with
> > > > >> Robert's JDBC-focused call-out, the meeting agreed the SPI should
> be
> > > > >> structured for multiple sinks so integrations become impl choices
> > > rather
> > > > >> than architectural changes.
> > > > >>
> > > > >> The battery-included default is most likely to use the existing
> > > > >> JDBC-backed
> > > > >> approach.
> > > > >>
> > > > >> *Direction (idea-level alignment)*
> > > > >> •   Single metrics_report table consolidating per-type metrics,
> > > > replacing
> > > > >> scan_metrics_report and commit_metrics_report
> > > > >> •   Iceberg-focused naming and schema for now, revisit if
> > > generic-table
> > > > or
> > > > >> operational metrics arrive
> > > > >> •   Metrics persistence as a separated SPI, not on BasePersistence
> > > > >> •   Bootstrap path separated for metrics, independent of metastore
> > > > >> bootstrap
> > > > >> •   "Battery-included plus pluggable" as the SPI design philosophy
> > > > >>
> > > > >> *Open items*
> > > > >> •   Schema details: metric_schema_version, payload format, IRC v2
> > > > >> forward-compat shape
> > > > >> •   SPI design details — full review either in the next sync or in
> > the
> > > > >> corresponding PR
> > > > >> •   Schema refactor PR ownership
> > > > >>
> > > > >> *Action items*
> > > > >> •   EJ to take a first stab at the SPI design and potentially
> > partner
> > > > with
> > > > >> Anand to incorporate the lessons learned from the existing
> reporter
> > > and
> > > > >> persistence work.
> > > > >> •   Schema refactor PR ownership is not yet decided. If anyone is
> > > > >> interested in driving it, reply on this thread.
> > > > >> •   JB to schedule the next sync, tentatively in two weeks.
> > > > >>
> > > > >> -ej
> > > > >>
> > > > >> On Mon, Apr 27, 2026 at 3:07 PM EJ Wang <
> > > ej.wang.dev.interest@gmail.com
> > > > >
> > > > >> wrote:
> > > > >>
> > > > >> > Thanks Yufei for the +1.
> > > > >> >
> > > > >> > JB, could you help add a biweekly metrics architecture sync to
> the
> > > > >> Polaris
> > > > >> > community calendar? I'm thinking Thursdays at 9-10am PT, on the
> > > > >> off-weeks
> > > > >> > from the community meeting (starting May 7), 60 minutes.
> > > > >> >
> > > > >> > Here's a rough agenda to work through over the first few
> sessions,
> > > > >> grouped
> > > > >> > by priority:
> > > > >> >
> > > > >> > *First: foundational direction*
> > > > >> >
> > > > >> > 1.  MetricsPersistence: public SPI or internal implementation
> > > detail?
> > > > >> >    •   Marked @Beta, javadoc calls it a "Service Provider
> > > Interface",
> > > > >> but
> > > > >> > only one consumer (JdbcBasePersistenceImpl), lives on
> > > BasePersistence.
> > > > >> If
> > > > >> > demoted to a private helper inside a persisting reporter impl,
> > most
> > > > >> > downstream design decisions become implementation details rather
> > > than
> > > > >> > contract questions.
> > > > >> >
> > > > >> > 2.  Persistence schema redesign
> > > > >> >    •   Current two-table layout (scan_metrics_report,
> > > > >> > commit_metrics_report) with ~25 flattened columns each. Every
> new
> > > > metric
> > > > >> > type requires a new table, SPI method, record class, model,
> > > converter,
> > > > >> and
> > > > >> > schema migration. Direction to explore: single table with
> > > metric_type
> > > > >> enum,
> > > > >> > schema_version, and JSON payload column.
> > > > >> >
> > > > >> > *Second: design details once direction is set*
> > > > >> >
> > > > >> > 3.  Partition key strategy
> > > > >> >    •   Single-table design means scan metrics at scale will have
> > > high
> > > > >> > write concurrency per table. Schema needs to expose enough
> > structure
> > > > for
> > > > >> > backends to shard by entity or time range.
> > > > >> >
> > > > >> > 4.  Read/write path consistency
> > > > >> >    •   Writes go through PolarisMetricsManager on
> > MetaStoreManager.
> > > > >> Reads
> > > > >> > bypass MetaStoreManager and go straight to BasePersistence,
> > > excluding
> > > > >> > non-JDBC backends from the read API.
> > > > >> >
> > > > >> > *Third: cleanup and alignment*
> > > > >> >
> > > > >> > 5.  PolarisMetricsReporter naming
> > > > >> >    •   Only handles IRC (ScanReport/CommitReport), doesn't cover
> > > > generic
> > > > >> > tables or operational metrics. Name is broader than scope.
> > > > >> >
> > > > >> > 6.  PolarisMetricsManager facade passthrough
> > > > >> >    •   Entire default method is
> > > > >> callCtx.getMetaStore().writeScanReport().
> > > > >> > Zero logic, passes Level 1 straight through to Level 3. Same
> > > > >> anti-pattern
> > > > >> > as PolarisEventManager.
> > > > >> >
> > > > >> > 7.  Iceberg community alignment
> > > > >> >    •   Payload-type extension needs discussion on dev@iceberg.
> > > > >> obelix74's
> > > > >> > Feb thread got zero replies. Needs a committer voice.
> > > > >> >
> > > > >> > Lets confirm prioritization in the first session.
> > > > >> >
> > > > >> > -ej
> > > > >> >
> > > > >> > On Tue, Apr 21, 2026 at 3:18 PM Yufei Gu <fl...@gmail.com>
> > > > wrote:
> > > > >> >
> > > > >> >> Thanks everyone for continuing to drive this forward. I agree
> > that
> > > > the
> > > > >> >> problem is getting complex enough that a more structured
> > discussion
> > > > >> would
> > > > >> >> help.
> > > > >> >>
> > > > >> >> +1 on setting up a biweekly sync for the metrics architecture.
> > I’m
> > > > >> happy
> > > > >> >> to
> > > > >> >> join.
> > > > >> >>
> > > > >> >> Yufei
> > > > >> >>
> > > > >> >>
> > > > >> >> On Tue, Apr 21, 2026 at 2:34 PM EJ Wang <
> > > > >> ej.wang.dev.interest@gmail.com>
> > > > >> >> wrote:
> > > > >> >>
> > > > >> >> > Also, I've been looking more closely at the *persistence
> schema
> > > in
> > > > >> the
> > > > >> >> > current metrics work*, and I think there's a structural
> > rigidity
> > > > >> problem
> > > > >> >> > worth raising before the shape gets locked in.
> > > > >> >> >
> > > > >> >> > Right now we have two separate tables (scan_metrics_report
> and
> > > > >> >> > commit_metrics_report), each with ~25 flattened columns that
> > > > directly
> > > > >> >> > mirror the Iceberg report fields. The SPI follows the same
> > split:
> > > > >> >> > writeScanReport and writeCommitReport as separate methods,
> with
> > > > >> per-type
> > > > >> >> > record classes, converters, and model objects. *The practical
> > > cost:
> > > > >> >> > adding a new metric type (operational metrics, for example)
> > > > requires
> > > > >> a
> > > > >> >> new
> > > > >> >> > table, a new SPI method, a new record class, a new model
> > class, a
> > > > new
> > > > >> >> > converter branch, and a schema migration*. That's a lot of
> > > surface
> > > > >> area
> > > > >> >> > for what should be "one more kind of metric."
> > > > >> >> >
> > > > >> >> > *My bias* would be toward a single metrics table with *a
> typed
> > > JSON
> > > > >> >> > payload*. Something like: metric_type (enum), entity_id,
> > > > >> >> > table_identifier, snapshot_id (nullable), received_ts,
> > > > >> schema_version,
> > > > >> >> and
> > > > >> >> > a payload column for the metric-specific data. The
> metric_type
> > +
> > > > >> >> > schema_version pair gives us a forward-compatible contract
> for
> > > the
> > > > >> >> payload
> > > > >> >> > shape. Adding a new metric type becomes an enum value and a
> > > payload
> > > > >> >> schema,
> > > > >> >> > not a schema migration. One thing I think we need to be
> > > deliberate
> > > > >> >> about is
> > > > >> >> > the partition key design. If all metric types land in one
> > table,
> > > > scan
> > > > >> >> > metrics at scale (high concurrency, high frequency across
> many
> > > > >> tables)
> > > > >> >> > could easily create hot partitions. We'd want the persistence
> > > layer
> > > > >> to
> > > > >> >> be
> > > > >> >> > able to shard by entity or time range, and that means the
> > logical
> > > > >> schema
> > > > >> >> > needs to expose enough structure for backends to partition
> on.
> > I
> > > > >> don't
> > > > >> >> > think the current flattened layout gives us that.
> > > > >> >> >
> > > > >> >> > This is getting complex enough that I don't think ad-hoc
> PR/ML
> > > > >> threads
> > > > >> >> > will converge well. *Would people be open to a biweekly sync
> > for
> > > > >> metrics
> > > > >> >> > architecture?* I think 30 minutes every two weeks with
> > interested
> > > > >> >> parties
> > > > >> >> > would be enough to work through the schema, SPI shape, and
> read
> > > API
> > > > >> >> design
> > > > >> >> > together. Happy to help set that up.
> > > > >> >> >
> > > > >> >> > -ej
> > > > >> >> >
> > > > >> >> > On Mon, Apr 20, 2026 at 2:19 PM EJ Wang <
> > > > >> ej.wang.dev.interest@gmail.com
> > > > >> >> >
> > > > >> >> > wrote:
> > > > >> >> >
> > > > >> >> >> Reviewed #4115, left a comment on the code organization
> side.
> > > > >> >> >>
> > > > >> >> >> One thing stood out: the metrics write path enters through
> > > > >> >> >> PolarisMetricsManager on MetaStoreManager, but the new read
> > path
> > > > >> >> bypasses
> > > > >> >> >> MetaStoreManager entirely and goes straight to
> BasePersistence
> > > via
> > > > >> >> >> callContext.getMetaStore(). That means the read API only
> works
> > > for
> > > > >> >> backends
> > > > >> >> >> that implement BasePersistence. NoSQL and remote backends
> > can't
> > > > >> >> participate.
> > > > >> >> >>
> > > > >> >> >> Stepping back, I think the metrics subsystem is growing into
> > > > >> something
> > > > >> >> >> real (write + read + REST API + AuthZ + pagination) *but the
> > > > >> >> persistence
> > > > >> >> >> side is split across two layers in a way that's hard to
> > > extend*. I
> > > > >> put
> > > > >> >> >> together two diagrams to show what I mean (my best effort).
> > > > >> >> >>
> > > > >> >> >> *Current state* (Diagram 1): three interfaces at three
> > different
> > > > >> >> levels.
> > > > >> >> >> The engine-facing SPI (PolarisMetricsReporter) is clean. But
> > > > >> >> >> PolarisMetricsManager on MetaStoreManager is a passthrough
> to
> > > > >> >> >> MetricsPersistence on BasePersistence. The @Beta annotation
> > and
> > > > SPI
> > > > >> >> javadoc
> > > > >> >> >> are on the BasePersistence layer, while the actual extension
> > > > points
> > > > >> >> >> (PolarisMetricsReporter, PolarisMetricsManager) carry no
> > > stability
> > > > >> >> >> annotation. The write path goes through the MetaStoreManager
> > > > layer,
> > > > >> the
> > > > >> >> >> read path doesn't.
> > > > >> >> >>
> > > > >> >> >> *What I envision* (Diagram 2): two SPIs at two levels.
> > > > >> >> >> PolarisMetricsReporter stays as the engine-facing SPI.
> > > > >> >> >> PolarisMetricsManager becomes the backend-facing SPI with
> both
> > > > write
> > > > >> >> and
> > > > >> >> >> read methods at the MetaStoreManager level, where any
> backend
> > > > (JDBC,
> > > > >> >> NoSQL,
> > > > >> >> >> remote) can implement them. MetricsPersistence on
> > > BasePersistence
> > > > >> goes
> > > > >> >> >> away. Where metrics actually land is an implementation
> detail,
> > > > not a
> > > > >> >> core
> > > > >> >> >> interface.
> > > > >> >> >>
> > > > >> >> >> *Minor naming thing*: PolarisMetricsReporter is broader than
> > > what
> > > > it
> > > > >> >> >> actually handles. It only accepts Iceberg REST Catalog
> metrics
> > > > >> >> (ScanReport,
> > > > >> >> >> CommitReport via MetricsReport). Generic table metrics or
> > > > >> operational
> > > > >> >> >> metrics aren't in scope. Not blocking, but worth noting if
> the
> > > > >> metrics
> > > > >> >> >> surface expands.
> > > > >> >> >>
> > > > >> >> >> *Rough sketch of how to get there*:
> > > > >> >> >>  1.  Add read methods to PolarisMetricsManager
> > (listScanReports,
> > > > >> >> >> listCommitReports) with default no-op, same as the existing
> > > write
> > > > >> >> methods.
> > > > >> >> >> (Probably make PolarisMetricsManager more explicit on being
> > > > Iceberg
> > > > >> >> >> specific like package name or class name etc.)
> > > > >> >> >>  2.  Wire MetricsReportsService through MetaStoreManager
> > instead
> > > > of
> > > > >> >> >> callContext.getMetaStore().
> > > > >> >> >>  3.  Extract metrics persistence from
> JdbcBasePersistenceImpl
> > > into
> > > > >> its
> > > > >> >> >> own class. That file carries ~7 responsibilities, metrics
> > being
> > > > one
> > > > >> of
> > > > >> >> them.
> > > > >> >> >>  4.  Remove MetricsPersistence from BasePersistence.
> > > > >> >> >>
> > > > >> >> >> *None of this needs to happen in #4115. But if the direction
> > > makes
> > > > >> >> sense,
> > > > >> >> >> it would be good to align before the metrics surface grows
> > > > further.
> > > > >> >> Curious
> > > > >> >> >> what others think.*
> > > > >> >> >>
> > > > >> >> >> *My mental model note*: Level 1 MetaStoreManager; level 2
> > > > >> transactional
> > > > >> >> >> persistence; level 3 base persistence
> > > > >> >> >>
> > > > >> >> >> Diagram 1
> > > > >> >> >> <
> > > > >> >>
> > > > >>
> > > >
> > >
> >
> https://www.plantuml.com/plantuml/uml/bLHDR-Cs4BthLmpIYupw0zbkKQ1r3M-S7Bp8xhhM7WCOb3IM65EaGD9EX2RzxHrHb4CxRelwa4YSDu_lpOVcnZ9jzvM8BBS2uGjQpJC3dtHMSekPtMk44IpsMgEqa5XcCOhCZikQQLP1pR8TAp2n3ILhmZDP20m0fcIvUkAoW2qJXd9z1bpToO9BX3WXu0ucy5rpgGPNm0nW5_epUWtm2Ue3pn3kMOFQmKntGZW0BYtgBSi8k5A2QMwybJNMIbFiGSR9QZc4nUqIvikStF0jHprua5C-amge42aNt3R0f5JaaoivdV2Pkqbx4hee4ymOkBh5BTiB-_uIeGeo8zL8rPsPl4DktdEiK1jkB1NdZCRbrSTecDe_mlHbF0wvBmCkaOH5_S8a_TTTKI6-nmCAkEw4LpxsZ-LbYLKQFKMNOgf_wuM7_bV9gOer5SYMMksBSWXFcbi49KNZXNLicwfe3TETC7gPdPqI7uBcHMb1RSzYq34c6PDUM9mn8HRsUTZEiDBve3NjVZumBj0U7SS37mGO7vcwtiK-_pU7U7L_f-digo9YbhSwIfMRwIITKGXbxdIUTCGF1SeCJxloKsU-3k9ddRbX1eDq1q_fx1JbBGT0glVyXimDuP4TQ5qpCAmnGEj2s_6n5mtn1z-97-63itFQZLPO1Ev2tu_WF7Ju-VPc0Skg5bYXxBhkY1xpD7EM_7fyflSpIsqMgVth5xhVr4eQxWQ8enaSAJQSG16yFSDuJ798rrcXr_3n-lfdk7icQjEBmFujL7AodiP_Y4Z7-YxvtZNs4zMgpNTl6tF8sglyPsmqchrjvQ-m-aP94r-TwCA2Ka8upPJZwtvSpoYCXkYMZU2NXvRMBfq9P3i3Le4VAZUAlUZ_oPKsxPgY0Q_BSKLkyr9bhQhQrJjo_x3TPlIB0DPjnMfcIoYP0QaYw1a0fTKDr8fB6ntNuvmoL1ZGkXa69Njh43zf9GiGxHQrA_jDYWRSzF5--WmTVrN97_Sm8LbLUy_lGBmLanJjFkDlGkRqjA_4tm00
> > > > >> >> >
> > > > >> >> >> :
> > > > >> >> >>
> > > > >> >> >> [image: image.png]
> > > > >> >> >>
> > > > >> >> >> Diagram 2
> > > > >> >> >> <
> > > > >> >>
> > > > >>
> > > >
> > >
> >
> https://www.plantuml.com/plantuml/uml/VLLDR-8m4BtdLupO2sWBLVU8AaGB7AXAbssGzb896SS9RXqxjKqBwkv_tt7iV43fdaZYDpFlpRmnOsE9jhjSH9PRmM31hERKm8scMsuPjJlDe0yheZDc8RR4iYWoBrmMH9CS2a9VICPYUy1OZN0YCy5Q0BCbYNhdCeEK28En8G8wCvbnoQ0R8_05Bc6bkLIz3X03p1zzH7zR-9ZfDquPt9C3qoNCX2yV4G2NbkcKu5jdgGJHt0GbZwnG6i-UP3TUpk5gM6Ldqke350eZUqzoCft3U9xWHvxoa5-7K4nF1J46EbEMafsmdrCBbQ44gVggy18IZrn_ph5asd1ZiIKdQSgueZvjXrQFSFrdC3YN-nXmBacxbGiYyLVxLaBtdhqn0LSzdBDhqQtQoOJeGyad3z0lUqnYgpGB6Ns8oVyta00Dy_WnX0tIOZ8v6SYxHll1TrH6aejAik-mh-AphVFCwSUQqFypElag5QRGFDjQKEd96K1P8QP41c9TzA_IIQyvdAWyv_RSiS3skb0_EzDDkK2v5xWF6MiGFlvhpFLcD2Dq2pml14gaF67eQkmd8gulDoC4kSOu6KVpkvlUJg1RTbWISU40RdBUUS_9XfRZ2dwxm_SW8LYFISgm_MnlDQ6M9P1gbKEc4X-2pH_FvJCkCqm9pbVjD6LrwdLeOrDWfOaqc8Wh9BE85oNKxkNQ6o4yGRy_Eae0G_G8tZv81d3bHDB23WOdisohVr3nh_j6lbSjbNaLRTc8UgtPbAU1J_tygOfZX9DWEJeHDvYx-qmSi5FgNLPZwHrHcUsncGQ5-skhUclpE5fo4ounpFauYrUbkU6ccfnxMvitwag4IyerhTxj8In_Oj1bDO4pQru674loYrGlULHLEGCjwJJ8gDoVZR8MxO4BT3IzRvIcAQKezC6xpziGnTyImrfEGyJI_OcKfgtxIvnTqFEMS17L9Z-jsARN5FmTheP7HtSdtOMT0B4GY2FYHXxgQmMtj2bRqiLFGapiVe1_QVKDrkqXcm83aFEXnMYCZ-xlyHy
> > > > >> >> >
> > > > >> >> >> :
> > > > >> >> >> [image: image.png]
> > > > >> >> >>
> > > > >> >> >>  -ej
> > > > >> >> >>
> > > > >> >> >> On Wed, Apr 15, 2026 at 8:22 AM Dmitri Bourlatchkov <
> > > > >> dimas@apache.org>
> > > > >> >> >> wrote:
> > > > >> >> >>
> > > > >> >> >>> Hi All,
> > > > >> >> >>>
> > > > >> >> >>> Heads up: The current state of PR [4115] looks pretty solid
> > to
> > > > me.
> > > > >> I
> > > > >> >> >>> believe this PR is approaching a mergeable condition.
> > > > >> >> >>>
> > > > >> >> >>> Please post your reviews if you have any comments.
> > > > >> >> >>>
> > > > >> >> >>> [4115] https://github.com/apache/polaris/pull/4115
> > > > >> >> >>>
> > > > >> >> >>> Thanks,
> > > > >> >> >>> Dmitri.
> > > > >> >> >>>
> > > > >> >> >>> On Tue, Mar 3, 2026 at 3:29 PM Anand Kumar Sankaran via
> dev <
> > > > >> >> >>> dev@polaris.apache.org> wrote:
> > > > >> >> >>>
> > > > >> >> >>> > Hi Yufei and Dmitri,
> > > > >> >> >>> >
> > > > >> >> >>> > Here is a proposal for the REST endpoints for metrics and
> > > > events.
> > > > >> >> >>> >
> > > > >> >> >>> > https://github.com/apache/polaris/pull/3924/changes
> > > > >> >> >>> >
> > > > >> >> >>> > I did not see any precursors for raising a PR for
> > proposals,
> > > so
> > > > >> >> trying
> > > > >> >> >>> > this.  Please let me know what you think.
> > > > >> >> >>> >
> > > > >> >> >>> > -
> > > > >> >> >>> > Anand
> > > > >> >> >>> >
> > > > >> >> >>> > From: Anand Kumar Sankaran <an...@workday.com>
> > > > >> >> >>> > Date: Monday, March 2, 2026 at 10:25 AM
> > > > >> >> >>> > To: dev@polaris.apache.org <de...@polaris.apache.org>
> > > > >> >> >>> > Subject: Re: Polaris Telemetry and Audit Trail
> > > > >> >> >>> >
> > > > >> >> >>> > About the REST API, based on my use cases:
> > > > >> >> >>> >
> > > > >> >> >>> >
> > > > >> >> >>> >   1.
> > > > >> >> >>> > I want to be able to query commit metrics to track files
> > > added
> > > > /
> > > > >> >> >>> removed
> > > > >> >> >>> > per commit, along with record counts. The ingestion
> > pipeline
> > > > that
> > > > >> >> >>> writes
> > > > >> >> >>> > this data is owned by us and we are guaranteed to write
> > this
> > > > >> >> >>> information
> > > > >> >> >>> > for each write.
> > > > >> >> >>> >   2.
> > > > >> >> >>> > I want to be able to query scan metrics for read. I
> > > understand
> > > > >> >> clients
> > > > >> >> >>> do
> > > > >> >> >>> > not fulfill this requirement.
> > > > >> >> >>> >   3.
> > > > >> >> >>> > I want to be able to query the events table (events are
> > > > >> persisted) -
> > > > >> >> >>> this
> > > > >> >> >>> > may supersede #2, I am not sure yet.
> > > > >> >> >>> >
> > > > >> >> >>> > All this information is in the JDBC based persistence
> model
> > > and
> > > > >> is
> > > > >> >> >>> > persisted in the metastore. I currently don’t have a need
> > to
> > > > >> query
> > > > >> >> >>> > prometheus or open telemetry. I do publish some events to
> > > > >> Prometheus
> > > > >> >> >>> and
> > > > >> >> >>> > they are forwarded to our dashboards elsewhere.
> > > > >> >> >>> >
> > > > >> >> >>> > About the CLI utilities, I meant the admin user
> utilities.
> > In
> > > > >> one of
> > > > >> >> >>> the
> > > > >> >> >>> > earliest drafts of my proposal, Prashant mentioned that
> the
> > > > >> metrics
> > > > >> >> >>> tables
> > > > >> >> >>> > can grow indefinitely and that a similar problem exists
> > with
> > > > the
> > > > >> >> events
> > > > >> >> >>> > table as well. We discussed that cleaning up of old
> records
> > > > from
> > > > >> >> both
> > > > >> >> >>> > metrics tables and events tables can be done via a CLI
> > > utility.
> > > > >> >> >>> >
> > > > >> >> >>> > I see that Yufei has covered the discussion about
> > > datasources.
> > > > >> >> >>> >
> > > > >> >> >>> > -
> > > > >> >> >>> > Anand
> > > > >> >> >>> >
> > > > >> >> >>> >
> > > > >> >> >>> >
> > > > >> >> >>> > From: Yufei Gu <fl...@gmail.com>
> > > > >> >> >>> > Date: Friday, February 27, 2026 at 9:54 PM
> > > > >> >> >>> > To: dev@polaris.apache.org <de...@polaris.apache.org>
> > > > >> >> >>> > Subject: Re: Polaris Telemetry and Audit Trail
> > > > >> >> >>> >
> > > > >> >> >>> > This Message Is From an External Sender
> > > > >> >> >>> > This message came from outside your organization.
> > > > >> >> >>> > Report Suspicious<
> > > > >> >> >>> >
> > > > >> >> >>>
> > > > >> >>
> > > > >>
> > > >
> > >
> >
> https://us-phishalarm-ewt.proofpoint.com/EWT/v1/Iz9xO38YGHZK!YhNDZABkHi1B699ote2uMwpOZw8i0QMCGO2Szc-HshuABGhGvwPJcymE6G2oUUxtS8xDkSrtGTPm_I3QnVDHoLMk50m9v8z_nZKTkd-bnVUbreF1u0WnfV_X5eYevZl_$
> > > > >> >> >>> > >
> > > > >> >> >>> >
> > > > >> >> >>> >
> > > > >> >> >>> > As I mentioned in
> > > > >> >> >>> >
> > > > >> >> >>>
> > > > >> >>
> > > > >>
> > > >
> > >
> >
> https://urldefense.com/v3/__https://github.com/apache/polaris/issues/3890__;!!Iz9xO38YGHZK!5EuyFFkk3vhRWVIRvQAWBSQfpJkTMA9HxugzDwXmN0LPPqhEFxYkFRGVhtb8AqUwXtDh2OplcMnbMDHKOxrvDU0$
> > > > >> >> >>> ,
> > > > >> >> >>> > supporting
> > > > >> >> >>> > multiple data sources is not a trivial change. I would
> > > strongly
> > > > >> >> >>> recommend
> > > > >> >> >>> > starting with a design document to carefully evaluate the
> > > > >> >> architectural
> > > > >> >> >>> > implications and long term impact.
> > > > >> >> >>> >
> > > > >> >> >>> > A REST endpoint to query metrics seems reasonable given
> the
> > > > >> current
> > > > >> >> >>> JDBC
> > > > >> >> >>> > based persistence model. That said, we may also consider
> > > > >> alternative
> > > > >> >> >>> > storage models. For example, if we later adopt a time
> > series
> > > > >> system
> > > > >> >> >>> such as
> > > > >> >> >>> > Prometheus to store metrics, the query model and access
> > > > patterns
> > > > >> >> would
> > > > >> >> >>> be
> > > > >> >> >>> > fundamentally different. Designing the REST API without
> > > > >> considering
> > > > >> >> >>> these
> > > > >> >> >>> > potential evolutions may limit flexibility. I'd suggest
> to
> > > > start
> > > > >> >> with
> > > > >> >> >>> the
> > > > >> >> >>> > use case.
> > > > >> >> >>> >
> > > > >> >> >>> > Yufei
> > > > >> >> >>> >
> > > > >> >> >>> >
> > > > >> >> >>> > On Fri, Feb 27, 2026 at 3:42 PM Dmitri Bourlatchkov <
> > > > >> >> dimas@apache.org>
> > > > >> >> >>> > wrote:
> > > > >> >> >>> >
> > > > >> >> >>> > > Hi Anand,
> > > > >> >> >>> > >
> > > > >> >> >>> > > Sharing my view... subject to discussion:
> > > > >> >> >>> > >
> > > > >> >> >>> > > 1. Adding non-IRC REST API to Polaris is perfectly
> fine.
> > > > >> >> >>> > >
> > > > >> >> >>> > > Figuring out specific endpoint URIs and payloads might
> > > > require
> > > > >> a
> > > > >> >> few
> > > > >> >> >>> > > roundtrips, so opening a separate thread for that might
> > be
> > > > >> best.
> > > > >> >> >>> > > Contributors commonly create Google Docs for new API
> > > > proposals
> > > > >> too
> > > > >> >> >>> (they
> > > > >> >> >>> > > fairly easy to update as the email discussion
> > progresses).
> > > > >> >> >>> > >
> > > > >> >> >>> > > There was a suggestion to try Markdown (with PRs) for
> > > > proposals
> > > > >> >> [1]
> > > > >> >> >>> ...
> > > > >> >> >>> > > feel free to give it a try if you are comfortable with
> > > that.
> > > > >> >> >>> > >
> > > > >> >> >>> > > 2. Could you clarify whether you mean end user
> utilities
> > or
> > > > >> admin
> > > > >> >> >>> user
> > > > >> >> >>> > > utilities? In the latter case those might be more
> > suitable
> > > > for
> > > > >> the
> > > > >> >> >>> Admin
> > > > >> >> >>> > > CLI (java) not the Python CLI, IMHO.
> > > > >> >> >>> > >
> > > > >> >> >>> > > Why would these utilities be common with events? IMHO,
> > > event
> > > > >> use
> > > > >> >> >>> cases
> > > > >> >> >>> > are
> > > > >> >> >>> > > distinct from scan/commit metrics.
> > > > >> >> >>> > >
> > > > >> >> >>> > > 3. I'd prefer separating metrics persistence from
> > MetaStore
> > > > >> >> >>> persistence
> > > > >> >> >>> > at
> > > > >> >> >>> > > the code level, so that they could be mixed and matched
> > > > >> >> >>> independently.
> > > > >> >> >>> > The
> > > > >> >> >>> > > separate datasource question will become a non-issue
> with
> > > > that
> > > > >> >> >>> approach,
> > > > >> >> >>> > I
> > > > >> >> >>> > > guess.
> > > > >> >> >>> > >
> > > > >> >> >>> > > The rationale for separating scan metrics and metastore
> > > > >> >> persistence
> > > > >> >> >>> is
> > > > >> >> >>> > that
> > > > >> >> >>> > > "cascading deletes" between them are hardly ever
> > required.
> > > > >> >> >>> Furthermore,
> > > > >> >> >>> > the
> > > > >> >> >>> > > data and query patterns are very different so different
> > > > >> >> technologies
> > > > >> >> >>> > might
> > > > >> >> >>> > > be beneficial in each case.
> > > > >> >> >>> > >
> > > > >> >> >>> > > [1]
> > > > >> >> >>> >
> > > > >> >> >>>
> > > > >> >>
> > > > >>
> > > >
> > >
> >
> https://urldefense.com/v3/__https://lists.apache.org/thread/yto2wp982t43h1mqjwnslswhws5z47cy__;!!Iz9xO38YGHZK!5EuyFFkk3vhRWVIRvQAWBSQfpJkTMA9HxugzDwXmN0LPPqhEFxYkFRGVhtb8AqUwXtDh2OplcMnbMDHKxYDakNU$
> > > > >> >> >>> > >
> > > > >> >> >>> > > Cheers,
> > > > >> >> >>> > > Dmitri.
> > > > >> >> >>> > >
> > > > >> >> >>> > > On Fri, Feb 27, 2026 at 6:19 PM Anand Kumar Sankaran
> via
> > > dev
> > > > <
> > > > >> >> >>> > > dev@polaris.apache.org> wrote:
> > > > >> >> >>> > >
> > > > >> >> >>> > > > Thanks all. This PR is merged now.
> > > > >> >> >>> > > >
> > > > >> >> >>> > > > Here are the follow-up features / work needed.  These
> > > were
> > > > >> all
> > > > >> >> >>> part of
> > > > >> >> >>> > > the
> > > > >> >> >>> > > > merged PR at some point in time and were removed to
> > > reduce
> > > > >> >> scope.
> > > > >> >> >>> > > >
> > > > >> >> >>> > > > Please let me know what you think.
> > > > >> >> >>> > > >
> > > > >> >> >>> > > >
> > > > >> >> >>> > > >   1.  A REST API to paginate through table metrics.
> > This
> > > > >> will be
> > > > >> >> >>> > non-IRC
> > > > >> >> >>> > > > standard addition.
> > > > >> >> >>> > > >   2.  Utilities for managing old records, should be
> > > common
> > > > >> with
> > > > >> >> >>> events.
> > > > >> >> >>> > > > There was some discussion that it belongs to the CLI.
> > > > >> >> >>> > > >   3.  Separate datasource (metrics, events, even
> other
> > > > >> tables?).
> > > > >> >> >>> > > >
> > > > >> >> >>> > > >
> > > > >> >> >>> > > > Anything else?
> > > > >> >> >>> > > >
> > > > >> >> >>> > > > -
> > > > >> >> >>> > > > Anand
> > > > >> >> >>> > > >
> > > > >> >> >>> > > >
> > > > >> >> >>> > >
> > > > >> >> >>> >
> > > > >> >> >>> >
> > > > >> >> >>>
> > > > >> >> >>
> > > > >> >>
> > > > >> >
> > > > >>
> > > > >
> > > >
> > >
> >
>

Re: Proposal for REST endpoints for table metrics and events

[EJ Wang <ej...@gmail.com>]

Hi folks,

Thanks again for the discussion today. I updated the sync doc with notes
from the third metrics architecture sync:

https://docs.google.com/document/d/100h7c4damrUzVuquYbBHM0EvA4LSWuW2IT2dN_7nYVA/edit?tab=t.k96s2xyqr5u1

A few highlights from today’s discussion:

   - We clarified that Iceberg metrics reporting can be interpreted either
   as sync or async handling from Polaris’s perspective, so Polaris should
   stay flexible as a platform instead of baking one handling model into the
   REST API behavior.
   - We aligned that the built-in (battery) behavior for metrics emitting
   can stay simple: no-op/log-only is enough as the default.
   - Durable metrics persistence should be treated as an implementation of
   the metrics reporting path, not as the core SPI boundary.
   - The existing durable metrics work can be reviewed as a reference
   implementation of the reporting SPI, with persistence-related logic kept
   self-contained under a metrics durable implementation module rather than
   scattered through core entity persistence.
   - Dashboard/insights remains a real use case, but we agreed to keep it
   separate from the core metrics intake discussion for now.

I also did a quick source check after the meeting to make sure we are
describing the current state accurately.

Current state:

   - Polaris already has a metrics reporting hook: PolarisMetricsReporter.
   - The default implementation is DefaultMetricsReporter, selected by
   polaris.iceberg-metrics.reporting.type=default. It is log-only,
   effectively quiet unless metrics logging is enabled.
   - There is also a PersistingMetricsReporter, selected by
   polaris.iceberg-metrics.reporting.type=persisting, which converts
   Iceberg scan/commit reports into Polaris metrics records and writes through
   PolarisMetricsManager -> MetricsPersistence.
   - MetricsPersistence currently exists in the persistence layer and is
   inherited by BasePersistence.

My read from the discussion is that the target boundary should not be
MetricsPersistence as inherited by BasePersistence. That path should be
decomposed as durable implementation detail (taken care of by
https://github.com/apache/polaris/pull/4397). The stable extension point
should instead be the metrics reporting/emitting boundary.

Concretely, I think the next proposal should be shaped like this:

   1.

   Define the proper metrics reporting SPI at an optional Iceberg metrics
   extension API layer.

   Example direction:

   public interface IcebergMetricsReporter {
     void reportMetric(IcebergMetricsReportContext context,
MetricsReport report);
   }

   The context would carry the small Polaris-resolved envelope:
   catalog/table identity, report type, received timestamp, and request/trace
   context if available. The raw Iceberg MetricsReport remains the payload.
   2.

   Keep runtime/service as the REST ingestion and wiring layer.

   The REST handler still resolves the table and performs authz before
   accepting the report. After that, it calls the selected reporting
   implementation.
   3.

   Keep the battery default no-op/log-only.

   This preserves an out-of-box safe default and avoids requiring a durable
   metrics store for every Polaris deployment.
   4.

   Move durable JDBC metrics into a self-contained implementation of the
   reporting SPI.

   That implementation can own its schema, bootstrap, retention, and read
   API support. It should not define the core reporting SPI boundary, and it
   should not require metrics persistence to remain inherited from
   BasePersistence.
   5.

   Treat async/event-backed handling as another implementation of the same
   reporting SPI.

   For example, an event-backed reporter could enqueue a metrics event and
   let listeners handle durable storage or other sinks. If we later need a
   replaceable queue engine, that seems like a shared event/metrics substrate
   topic rather than a metrics-only requirement.

This framing lets us keep the REST metrics endpoint simple, preserve the
current default behavior, support durable metrics users, and still leave
room for async/event-backed or external-queue-based implementations.

I think the main follow-up questions are:

   - Do folks agree that the stable SPI boundary should be metrics
   reporting/emitting, not metrics persistence?
   - Does an optional Iceberg metrics extension API layer sound like the
   right home for this SPI?
   - Should the current durable metrics work be reframed as a durable JDBC
   reference implementation of that SPI?
   - What is the smallest PR sequence to get there without blocking the
   current metrics work unnecessarily?

Thanks,
-ej

On Fri, May 15, 2026 at 5:16 PM Dmitri Bourlatchkov <di...@apache.org>
wrote:

> Hi JB,
>
> Could you set up another meeting, please? Same time on Wednesday as last
> time... I hope it works for everyone.
>
> Cheers,
> Dmitri.
>
> On Fri, May 15, 2026 at 8:06 PM Yufei Gu <fl...@gmail.com> wrote:
>
> > +1 on another sync call next week.
> >
> > Yufei
> >
> >
> > On Fri, May 15, 2026 at 4:52 PM Dmitri Bourlatchkov <di...@apache.org>
> > wrote:
> >
> > > Hi All,
> > >
> > > WDYT about another sync call next week?
> > >
> > > Thanks,
> > > Dmitri.
> > >
> > > On Wed, May 6, 2026 at 5:29 PM Dmitri Bourlatchkov <di...@apache.org>
> > > wrote:
> > >
> > > > Hi EJ,
> > > >
> > > > Thanks for the summary! It covers what we discussed in the meeting
> very
> > > > well, IMHO.
> > > >
> > > > Looking forward to concrete PRs :)
> > > >
> > > > Cheers,
> > > > Dmitri.
> > > >
> > > > On Wed, May 6, 2026 at 5:08 PM EJ Wang <
> ej.wang.dev.interest@gmail.com
> > >
> > > > wrote:
> > > >
> > > >> Hi folks,
> > > >>
> > > >> We had a community sync earlier, thanks JB for scheduling it. Notes
> > from
> > > >> the first metrics architecture sync (May 6, 10-11am PT). Discussion
> > doc
> > > >> with per-section status:
> > > >>
> > > >>
> > >
> >
> https://docs.google.com/document/d/100h7c4damrUzVuquYbBHM0EvA4LSWuW2IT2dN_7nYVA/edit?tab=t.0
> > > >>
> > > >> *The meeting covered both topics from the doc. Direction-level
> > alignment
> > > >> was reached on the headline pieces; details remain for PR review or
> > > >> follow-up sessions.*
> > > >>
> > > >> *Topic 1 — Persistence schema redesign*
> > > >> Idea-level alignment on consolidating per-type tables
> > > >> (scan_metrics_report,
> > > >> commit_metrics_report) into a single metrics_report table. The
> > > motivating
> > > >> cost is the surface area added by every new metric type today: new
> > > table,
> > > >> SPI method, record class, model, converter, schema migration.
> > > >>
> > > >> Most schema details are deferred to the schema PR. A few specific
> > points
> > > >> came up:
> > > >> •   metric_schema_version: Yufei prefers dropping it, since there is
> > no
> > > >> spec-level concept of metrics versioning today and it is hard to
> > define
> > > >> unilaterally. Robert prefers keeping it, given IRC v2 is coming and
> > the
> > > >> schema should be considered against its likely shape; Robert also
> > raised
> > > >> how to differentiate various payload formats if any. EJ's read is
> that
> > > >> this
> > > >> is a two-way-door decision. We can start without the field, and if
> IRC
> > > v2
> > > >> changes the shape we would likely roll a corresponding new schema
> > > anyway,
> > > >> which is not particularly costly.
> > > >> •   Payload format: Robert pointed out that future formats beyond
> JSON
> > > may
> > > >> be worth supporting. The exact shape is deferred to the schema
> > > discussion.
> > > >> •   Partition strategy: Anand suggested monthly partitioning based
> on
> > > his
> > > >> experience as potentially helpful at scale.
> > > >>
> > > >> *Topic 2 — Where metrics ingestion and storage belong*
> > > >> Idea-level alignment that metrics should be a separated SPI from the
> > > >> entity
> > > >> persistence stack. Two reasons surfaced: (a) workloads and
> capability
> > > >> requirements diverge enough that coupling them creates artificial
> > > >> constraints, and (b) admin experience improves when metrics has its
> > own
> > > >> bootstrap, retention, and lifecycle. Dmitri noted Polaris being a
> > > platform
> > > >> should have the flexibility to support different persistence
> backends
> > > per
> > > >> concern, and pointed to a concrete next step of separating the JDBC
> > > >> bootstrap for metrics from the metastore bootstrap. Robert proposed
> an
> > > >> additional UX extension: detect an unbootstrapped metrics store on
> > first
> > > >> use and auto-bootstrap rather than requiring an explicit manual
> > > bootstrap
> > > >> step.
> > > >> The meeting also confirmed that Polaris metrics can start small and
> > stay
> > > >> Iceberg-focused. Naming and persistence schema can lean
> > > Iceberg-specific.
> > > >> If a future expansion to generic-table metrics or operational
> metrics
> > > >> arrives, an abstraction layer can be built on top of the Iceberg
> > metrics
> > > >> reporter at that point. Robert remains on the fence and would prefer
> > > >> something more generic but did not block the direction; Dmitri's
> read
> > > was
> > > >> that the proposed framework already has enough flexibility to absorb
> > > >> future
> > > >> expansion.
> > > >>
> > > >> The Trade-offs and Proposed structure sections in the doc were not
> > > >> reviewed
> > > >> in detail. They remain open for either the next sync or PR review.
> > > >>
> > > >> *Cross-cutting alignment — battery-included plus pluggable*
> > > >> A common philosophy emerged from the discussion. EJ summarized it
> as:
> > > >> Polaris should provide a battery-included UX for beginners and the
> > > >> flexibility for advanced users to swap the included battery for
> > > something
> > > >> more powerful or tailored to their use case. The SPI design needs to
> > > >> enable
> > > >> both.
> > > >>
> > > >> The inputs that shaped this framing:
> > > >> •   Anand described how his team uses the current metrics
> persistence
> > > >> (three metrics consumers in v1.4).
> > > >> •   Yufei raised Grafana and dashboard integrations as a destination
> > use
> > > >> case beyond the default.
> > > >> •   Robert called out that the current design is more JDBC-focused.
> > > >>
> > > >> Two concrete instances:
> > > >> •   Async metrics intake: Yufei's initial position was that async
> > should
> > > >> largely live on the producer side and there is not much Polaris can
> > do.
> > > >> Robert suggested a Polaris-side default is doable via Vert.x. Dmitri
> > > >> agreed
> > > >> the direction is worth exploring. The meeting converged on a
> > > >> battery-included default (likely Vert.x-backed) with an SPI shape
> that
> > > >> lets
> > > >> power users route to a more scalable backend (k8s-hosted queue, AWS
> > SQS,
> > > >> etc.).
> > > >> •   Pluggable destinations: combining Yufei's dashboard use case
> with
> > > >> Robert's JDBC-focused call-out, the meeting agreed the SPI should be
> > > >> structured for multiple sinks so integrations become impl choices
> > rather
> > > >> than architectural changes.
> > > >>
> > > >> The battery-included default is most likely to use the existing
> > > >> JDBC-backed
> > > >> approach.
> > > >>
> > > >> *Direction (idea-level alignment)*
> > > >> •   Single metrics_report table consolidating per-type metrics,
> > > replacing
> > > >> scan_metrics_report and commit_metrics_report
> > > >> •   Iceberg-focused naming and schema for now, revisit if
> > generic-table
> > > or
> > > >> operational metrics arrive
> > > >> •   Metrics persistence as a separated SPI, not on BasePersistence
> > > >> •   Bootstrap path separated for metrics, independent of metastore
> > > >> bootstrap
> > > >> •   "Battery-included plus pluggable" as the SPI design philosophy
> > > >>
> > > >> *Open items*
> > > >> •   Schema details: metric_schema_version, payload format, IRC v2
> > > >> forward-compat shape
> > > >> •   SPI design details — full review either in the next sync or in
> the
> > > >> corresponding PR
> > > >> •   Schema refactor PR ownership
> > > >>
> > > >> *Action items*
> > > >> •   EJ to take a first stab at the SPI design and potentially
> partner
> > > with
> > > >> Anand to incorporate the lessons learned from the existing reporter
> > and
> > > >> persistence work.
> > > >> •   Schema refactor PR ownership is not yet decided. If anyone is
> > > >> interested in driving it, reply on this thread.
> > > >> •   JB to schedule the next sync, tentatively in two weeks.
> > > >>
> > > >> -ej
> > > >>
> > > >> On Mon, Apr 27, 2026 at 3:07 PM EJ Wang <
> > ej.wang.dev.interest@gmail.com
> > > >
> > > >> wrote:
> > > >>
> > > >> > Thanks Yufei for the +1.
> > > >> >
> > > >> > JB, could you help add a biweekly metrics architecture sync to the
> > > >> Polaris
> > > >> > community calendar? I'm thinking Thursdays at 9-10am PT, on the
> > > >> off-weeks
> > > >> > from the community meeting (starting May 7), 60 minutes.
> > > >> >
> > > >> > Here's a rough agenda to work through over the first few sessions,
> > > >> grouped
> > > >> > by priority:
> > > >> >
> > > >> > *First: foundational direction*
> > > >> >
> > > >> > 1.  MetricsPersistence: public SPI or internal implementation
> > detail?
> > > >> >    •   Marked @Beta, javadoc calls it a "Service Provider
> > Interface",
> > > >> but
> > > >> > only one consumer (JdbcBasePersistenceImpl), lives on
> > BasePersistence.
> > > >> If
> > > >> > demoted to a private helper inside a persisting reporter impl,
> most
> > > >> > downstream design decisions become implementation details rather
> > than
> > > >> > contract questions.
> > > >> >
> > > >> > 2.  Persistence schema redesign
> > > >> >    •   Current two-table layout (scan_metrics_report,
> > > >> > commit_metrics_report) with ~25 flattened columns each. Every new
> > > metric
> > > >> > type requires a new table, SPI method, record class, model,
> > converter,
> > > >> and
> > > >> > schema migration. Direction to explore: single table with
> > metric_type
> > > >> enum,
> > > >> > schema_version, and JSON payload column.
> > > >> >
> > > >> > *Second: design details once direction is set*
> > > >> >
> > > >> > 3.  Partition key strategy
> > > >> >    •   Single-table design means scan metrics at scale will have
> > high
> > > >> > write concurrency per table. Schema needs to expose enough
> structure
> > > for
> > > >> > backends to shard by entity or time range.
> > > >> >
> > > >> > 4.  Read/write path consistency
> > > >> >    •   Writes go through PolarisMetricsManager on
> MetaStoreManager.
> > > >> Reads
> > > >> > bypass MetaStoreManager and go straight to BasePersistence,
> > excluding
> > > >> > non-JDBC backends from the read API.
> > > >> >
> > > >> > *Third: cleanup and alignment*
> > > >> >
> > > >> > 5.  PolarisMetricsReporter naming
> > > >> >    •   Only handles IRC (ScanReport/CommitReport), doesn't cover
> > > generic
> > > >> > tables or operational metrics. Name is broader than scope.
> > > >> >
> > > >> > 6.  PolarisMetricsManager facade passthrough
> > > >> >    •   Entire default method is
> > > >> callCtx.getMetaStore().writeScanReport().
> > > >> > Zero logic, passes Level 1 straight through to Level 3. Same
> > > >> anti-pattern
> > > >> > as PolarisEventManager.
> > > >> >
> > > >> > 7.  Iceberg community alignment
> > > >> >    •   Payload-type extension needs discussion on dev@iceberg.
> > > >> obelix74's
> > > >> > Feb thread got zero replies. Needs a committer voice.
> > > >> >
> > > >> > Lets confirm prioritization in the first session.
> > > >> >
> > > >> > -ej
> > > >> >
> > > >> > On Tue, Apr 21, 2026 at 3:18 PM Yufei Gu <fl...@gmail.com>
> > > wrote:
> > > >> >
> > > >> >> Thanks everyone for continuing to drive this forward. I agree
> that
> > > the
> > > >> >> problem is getting complex enough that a more structured
> discussion
> > > >> would
> > > >> >> help.
> > > >> >>
> > > >> >> +1 on setting up a biweekly sync for the metrics architecture.
> I’m
> > > >> happy
> > > >> >> to
> > > >> >> join.
> > > >> >>
> > > >> >> Yufei
> > > >> >>
> > > >> >>
> > > >> >> On Tue, Apr 21, 2026 at 2:34 PM EJ Wang <
> > > >> ej.wang.dev.interest@gmail.com>
> > > >> >> wrote:
> > > >> >>
> > > >> >> > Also, I've been looking more closely at the *persistence schema
> > in
> > > >> the
> > > >> >> > current metrics work*, and I think there's a structural
> rigidity
> > > >> problem
> > > >> >> > worth raising before the shape gets locked in.
> > > >> >> >
> > > >> >> > Right now we have two separate tables (scan_metrics_report and
> > > >> >> > commit_metrics_report), each with ~25 flattened columns that
> > > directly
> > > >> >> > mirror the Iceberg report fields. The SPI follows the same
> split:
> > > >> >> > writeScanReport and writeCommitReport as separate methods, with
> > > >> per-type
> > > >> >> > record classes, converters, and model objects. *The practical
> > cost:
> > > >> >> > adding a new metric type (operational metrics, for example)
> > > requires
> > > >> a
> > > >> >> new
> > > >> >> > table, a new SPI method, a new record class, a new model
> class, a
> > > new
> > > >> >> > converter branch, and a schema migration*. That's a lot of
> > surface
> > > >> area
> > > >> >> > for what should be "one more kind of metric."
> > > >> >> >
> > > >> >> > *My bias* would be toward a single metrics table with *a typed
> > JSON
> > > >> >> > payload*. Something like: metric_type (enum), entity_id,
> > > >> >> > table_identifier, snapshot_id (nullable), received_ts,
> > > >> schema_version,
> > > >> >> and
> > > >> >> > a payload column for the metric-specific data. The metric_type
> +
> > > >> >> > schema_version pair gives us a forward-compatible contract for
> > the
> > > >> >> payload
> > > >> >> > shape. Adding a new metric type becomes an enum value and a
> > payload
> > > >> >> schema,
> > > >> >> > not a schema migration. One thing I think we need to be
> > deliberate
> > > >> >> about is
> > > >> >> > the partition key design. If all metric types land in one
> table,
> > > scan
> > > >> >> > metrics at scale (high concurrency, high frequency across many
> > > >> tables)
> > > >> >> > could easily create hot partitions. We'd want the persistence
> > layer
> > > >> to
> > > >> >> be
> > > >> >> > able to shard by entity or time range, and that means the
> logical
> > > >> schema
> > > >> >> > needs to expose enough structure for backends to partition on.
> I
> > > >> don't
> > > >> >> > think the current flattened layout gives us that.
> > > >> >> >
> > > >> >> > This is getting complex enough that I don't think ad-hoc PR/ML
> > > >> threads
> > > >> >> > will converge well. *Would people be open to a biweekly sync
> for
> > > >> metrics
> > > >> >> > architecture?* I think 30 minutes every two weeks with
> interested
> > > >> >> parties
> > > >> >> > would be enough to work through the schema, SPI shape, and read
> > API
> > > >> >> design
> > > >> >> > together. Happy to help set that up.
> > > >> >> >
> > > >> >> > -ej
> > > >> >> >
> > > >> >> > On Mon, Apr 20, 2026 at 2:19 PM EJ Wang <
> > > >> ej.wang.dev.interest@gmail.com
> > > >> >> >
> > > >> >> > wrote:
> > > >> >> >
> > > >> >> >> Reviewed #4115, left a comment on the code organization side.
> > > >> >> >>
> > > >> >> >> One thing stood out: the metrics write path enters through
> > > >> >> >> PolarisMetricsManager on MetaStoreManager, but the new read
> path
> > > >> >> bypasses
> > > >> >> >> MetaStoreManager entirely and goes straight to BasePersistence
> > via
> > > >> >> >> callContext.getMetaStore(). That means the read API only works
> > for
> > > >> >> backends
> > > >> >> >> that implement BasePersistence. NoSQL and remote backends
> can't
> > > >> >> participate.
> > > >> >> >>
> > > >> >> >> Stepping back, I think the metrics subsystem is growing into
> > > >> something
> > > >> >> >> real (write + read + REST API + AuthZ + pagination) *but the
> > > >> >> persistence
> > > >> >> >> side is split across two layers in a way that's hard to
> > extend*. I
> > > >> put
> > > >> >> >> together two diagrams to show what I mean (my best effort).
> > > >> >> >>
> > > >> >> >> *Current state* (Diagram 1): three interfaces at three
> different
> > > >> >> levels.
> > > >> >> >> The engine-facing SPI (PolarisMetricsReporter) is clean. But
> > > >> >> >> PolarisMetricsManager on MetaStoreManager is a passthrough to
> > > >> >> >> MetricsPersistence on BasePersistence. The @Beta annotation
> and
> > > SPI
> > > >> >> javadoc
> > > >> >> >> are on the BasePersistence layer, while the actual extension
> > > points
> > > >> >> >> (PolarisMetricsReporter, PolarisMetricsManager) carry no
> > stability
> > > >> >> >> annotation. The write path goes through the MetaStoreManager
> > > layer,
> > > >> the
> > > >> >> >> read path doesn't.
> > > >> >> >>
> > > >> >> >> *What I envision* (Diagram 2): two SPIs at two levels.
> > > >> >> >> PolarisMetricsReporter stays as the engine-facing SPI.
> > > >> >> >> PolarisMetricsManager becomes the backend-facing SPI with both
> > > write
> > > >> >> and
> > > >> >> >> read methods at the MetaStoreManager level, where any backend
> > > (JDBC,
> > > >> >> NoSQL,
> > > >> >> >> remote) can implement them. MetricsPersistence on
> > BasePersistence
> > > >> goes
> > > >> >> >> away. Where metrics actually land is an implementation detail,
> > > not a
> > > >> >> core
> > > >> >> >> interface.
> > > >> >> >>
> > > >> >> >> *Minor naming thing*: PolarisMetricsReporter is broader than
> > what
> > > it
> > > >> >> >> actually handles. It only accepts Iceberg REST Catalog metrics
> > > >> >> (ScanReport,
> > > >> >> >> CommitReport via MetricsReport). Generic table metrics or
> > > >> operational
> > > >> >> >> metrics aren't in scope. Not blocking, but worth noting if the
> > > >> metrics
> > > >> >> >> surface expands.
> > > >> >> >>
> > > >> >> >> *Rough sketch of how to get there*:
> > > >> >> >>  1.  Add read methods to PolarisMetricsManager
> (listScanReports,
> > > >> >> >> listCommitReports) with default no-op, same as the existing
> > write
> > > >> >> methods.
> > > >> >> >> (Probably make PolarisMetricsManager more explicit on being
> > > Iceberg
> > > >> >> >> specific like package name or class name etc.)
> > > >> >> >>  2.  Wire MetricsReportsService through MetaStoreManager
> instead
> > > of
> > > >> >> >> callContext.getMetaStore().
> > > >> >> >>  3.  Extract metrics persistence from JdbcBasePersistenceImpl
> > into
> > > >> its
> > > >> >> >> own class. That file carries ~7 responsibilities, metrics
> being
> > > one
> > > >> of
> > > >> >> them.
> > > >> >> >>  4.  Remove MetricsPersistence from BasePersistence.
> > > >> >> >>
> > > >> >> >> *None of this needs to happen in #4115. But if the direction
> > makes
> > > >> >> sense,
> > > >> >> >> it would be good to align before the metrics surface grows
> > > further.
> > > >> >> Curious
> > > >> >> >> what others think.*
> > > >> >> >>
> > > >> >> >> *My mental model note*: Level 1 MetaStoreManager; level 2
> > > >> transactional
> > > >> >> >> persistence; level 3 base persistence
> > > >> >> >>
> > > >> >> >> Diagram 1
> > > >> >> >> <
> > > >> >>
> > > >>
> > >
> >
> https://www.plantuml.com/plantuml/uml/bLHDR-Cs4BthLmpIYupw0zbkKQ1r3M-S7Bp8xhhM7WCOb3IM65EaGD9EX2RzxHrHb4CxRelwa4YSDu_lpOVcnZ9jzvM8BBS2uGjQpJC3dtHMSekPtMk44IpsMgEqa5XcCOhCZikQQLP1pR8TAp2n3ILhmZDP20m0fcIvUkAoW2qJXd9z1bpToO9BX3WXu0ucy5rpgGPNm0nW5_epUWtm2Ue3pn3kMOFQmKntGZW0BYtgBSi8k5A2QMwybJNMIbFiGSR9QZc4nUqIvikStF0jHprua5C-amge42aNt3R0f5JaaoivdV2Pkqbx4hee4ymOkBh5BTiB-_uIeGeo8zL8rPsPl4DktdEiK1jkB1NdZCRbrSTecDe_mlHbF0wvBmCkaOH5_S8a_TTTKI6-nmCAkEw4LpxsZ-LbYLKQFKMNOgf_wuM7_bV9gOer5SYMMksBSWXFcbi49KNZXNLicwfe3TETC7gPdPqI7uBcHMb1RSzYq34c6PDUM9mn8HRsUTZEiDBve3NjVZumBj0U7SS37mGO7vcwtiK-_pU7U7L_f-digo9YbhSwIfMRwIITKGXbxdIUTCGF1SeCJxloKsU-3k9ddRbX1eDq1q_fx1JbBGT0glVyXimDuP4TQ5qpCAmnGEj2s_6n5mtn1z-97-63itFQZLPO1Ev2tu_WF7Ju-VPc0Skg5bYXxBhkY1xpD7EM_7fyflSpIsqMgVth5xhVr4eQxWQ8enaSAJQSG16yFSDuJ798rrcXr_3n-lfdk7icQjEBmFujL7AodiP_Y4Z7-YxvtZNs4zMgpNTl6tF8sglyPsmqchrjvQ-m-aP94r-TwCA2Ka8upPJZwtvSpoYCXkYMZU2NXvRMBfq9P3i3Le4VAZUAlUZ_oPKsxPgY0Q_BSKLkyr9bhQhQrJjo_x3TPlIB0DPjnMfcIoYP0QaYw1a0fTKDr8fB6ntNuvmoL1ZGkXa69Njh43zf9GiGxHQrA_jDYWRSzF5--WmTVrN97_Sm8LbLUy_lGBmLanJjFkDlGkRqjA_4tm00
> > > >> >> >
> > > >> >> >> :
> > > >> >> >>
> > > >> >> >> [image: image.png]
> > > >> >> >>
> > > >> >> >> Diagram 2
> > > >> >> >> <
> > > >> >>
> > > >>
> > >
> >
> https://www.plantuml.com/plantuml/uml/VLLDR-8m4BtdLupO2sWBLVU8AaGB7AXAbssGzb896SS9RXqxjKqBwkv_tt7iV43fdaZYDpFlpRmnOsE9jhjSH9PRmM31hERKm8scMsuPjJlDe0yheZDc8RR4iYWoBrmMH9CS2a9VICPYUy1OZN0YCy5Q0BCbYNhdCeEK28En8G8wCvbnoQ0R8_05Bc6bkLIz3X03p1zzH7zR-9ZfDquPt9C3qoNCX2yV4G2NbkcKu5jdgGJHt0GbZwnG6i-UP3TUpk5gM6Ldqke350eZUqzoCft3U9xWHvxoa5-7K4nF1J46EbEMafsmdrCBbQ44gVggy18IZrn_ph5asd1ZiIKdQSgueZvjXrQFSFrdC3YN-nXmBacxbGiYyLVxLaBtdhqn0LSzdBDhqQtQoOJeGyad3z0lUqnYgpGB6Ns8oVyta00Dy_WnX0tIOZ8v6SYxHll1TrH6aejAik-mh-AphVFCwSUQqFypElag5QRGFDjQKEd96K1P8QP41c9TzA_IIQyvdAWyv_RSiS3skb0_EzDDkK2v5xWF6MiGFlvhpFLcD2Dq2pml14gaF67eQkmd8gulDoC4kSOu6KVpkvlUJg1RTbWISU40RdBUUS_9XfRZ2dwxm_SW8LYFISgm_MnlDQ6M9P1gbKEc4X-2pH_FvJCkCqm9pbVjD6LrwdLeOrDWfOaqc8Wh9BE85oNKxkNQ6o4yGRy_Eae0G_G8tZv81d3bHDB23WOdisohVr3nh_j6lbSjbNaLRTc8UgtPbAU1J_tygOfZX9DWEJeHDvYx-qmSi5FgNLPZwHrHcUsncGQ5-skhUclpE5fo4ounpFauYrUbkU6ccfnxMvitwag4IyerhTxj8In_Oj1bDO4pQru674loYrGlULHLEGCjwJJ8gDoVZR8MxO4BT3IzRvIcAQKezC6xpziGnTyImrfEGyJI_OcKfgtxIvnTqFEMS17L9Z-jsARN5FmTheP7HtSdtOMT0B4GY2FYHXxgQmMtj2bRqiLFGapiVe1_QVKDrkqXcm83aFEXnMYCZ-xlyHy
> > > >> >> >
> > > >> >> >> :
> > > >> >> >> [image: image.png]
> > > >> >> >>
> > > >> >> >>  -ej
> > > >> >> >>
> > > >> >> >> On Wed, Apr 15, 2026 at 8:22 AM Dmitri Bourlatchkov <
> > > >> dimas@apache.org>
> > > >> >> >> wrote:
> > > >> >> >>
> > > >> >> >>> Hi All,
> > > >> >> >>>
> > > >> >> >>> Heads up: The current state of PR [4115] looks pretty solid
> to
> > > me.
> > > >> I
> > > >> >> >>> believe this PR is approaching a mergeable condition.
> > > >> >> >>>
> > > >> >> >>> Please post your reviews if you have any comments.
> > > >> >> >>>
> > > >> >> >>> [4115] https://github.com/apache/polaris/pull/4115
> > > >> >> >>>
> > > >> >> >>> Thanks,
> > > >> >> >>> Dmitri.
> > > >> >> >>>
> > > >> >> >>> On Tue, Mar 3, 2026 at 3:29 PM Anand Kumar Sankaran via dev <
> > > >> >> >>> dev@polaris.apache.org> wrote:
> > > >> >> >>>
> > > >> >> >>> > Hi Yufei and Dmitri,
> > > >> >> >>> >
> > > >> >> >>> > Here is a proposal for the REST endpoints for metrics and
> > > events.
> > > >> >> >>> >
> > > >> >> >>> > https://github.com/apache/polaris/pull/3924/changes
> > > >> >> >>> >
> > > >> >> >>> > I did not see any precursors for raising a PR for
> proposals,
> > so
> > > >> >> trying
> > > >> >> >>> > this.  Please let me know what you think.
> > > >> >> >>> >
> > > >> >> >>> > -
> > > >> >> >>> > Anand
> > > >> >> >>> >
> > > >> >> >>> > From: Anand Kumar Sankaran <an...@workday.com>
> > > >> >> >>> > Date: Monday, March 2, 2026 at 10:25 AM
> > > >> >> >>> > To: dev@polaris.apache.org <de...@polaris.apache.org>
> > > >> >> >>> > Subject: Re: Polaris Telemetry and Audit Trail
> > > >> >> >>> >
> > > >> >> >>> > About the REST API, based on my use cases:
> > > >> >> >>> >
> > > >> >> >>> >
> > > >> >> >>> >   1.
> > > >> >> >>> > I want to be able to query commit metrics to track files
> > added
> > > /
> > > >> >> >>> removed
> > > >> >> >>> > per commit, along with record counts. The ingestion
> pipeline
> > > that
> > > >> >> >>> writes
> > > >> >> >>> > this data is owned by us and we are guaranteed to write
> this
> > > >> >> >>> information
> > > >> >> >>> > for each write.
> > > >> >> >>> >   2.
> > > >> >> >>> > I want to be able to query scan metrics for read. I
> > understand
> > > >> >> clients
> > > >> >> >>> do
> > > >> >> >>> > not fulfill this requirement.
> > > >> >> >>> >   3.
> > > >> >> >>> > I want to be able to query the events table (events are
> > > >> persisted) -
> > > >> >> >>> this
> > > >> >> >>> > may supersede #2, I am not sure yet.
> > > >> >> >>> >
> > > >> >> >>> > All this information is in the JDBC based persistence model
> > and
> > > >> is
> > > >> >> >>> > persisted in the metastore. I currently don’t have a need
> to
> > > >> query
> > > >> >> >>> > prometheus or open telemetry. I do publish some events to
> > > >> Prometheus
> > > >> >> >>> and
> > > >> >> >>> > they are forwarded to our dashboards elsewhere.
> > > >> >> >>> >
> > > >> >> >>> > About the CLI utilities, I meant the admin user utilities.
> In
> > > >> one of
> > > >> >> >>> the
> > > >> >> >>> > earliest drafts of my proposal, Prashant mentioned that the
> > > >> metrics
> > > >> >> >>> tables
> > > >> >> >>> > can grow indefinitely and that a similar problem exists
> with
> > > the
> > > >> >> events
> > > >> >> >>> > table as well. We discussed that cleaning up of old records
> > > from
> > > >> >> both
> > > >> >> >>> > metrics tables and events tables can be done via a CLI
> > utility.
> > > >> >> >>> >
> > > >> >> >>> > I see that Yufei has covered the discussion about
> > datasources.
> > > >> >> >>> >
> > > >> >> >>> > -
> > > >> >> >>> > Anand
> > > >> >> >>> >
> > > >> >> >>> >
> > > >> >> >>> >
> > > >> >> >>> > From: Yufei Gu <fl...@gmail.com>
> > > >> >> >>> > Date: Friday, February 27, 2026 at 9:54 PM
> > > >> >> >>> > To: dev@polaris.apache.org <de...@polaris.apache.org>
> > > >> >> >>> > Subject: Re: Polaris Telemetry and Audit Trail
> > > >> >> >>> >
> > > >> >> >>> > This Message Is From an External Sender
> > > >> >> >>> > This message came from outside your organization.
> > > >> >> >>> > Report Suspicious<
> > > >> >> >>> >
> > > >> >> >>>
> > > >> >>
> > > >>
> > >
> >
> https://us-phishalarm-ewt.proofpoint.com/EWT/v1/Iz9xO38YGHZK!YhNDZABkHi1B699ote2uMwpOZw8i0QMCGO2Szc-HshuABGhGvwPJcymE6G2oUUxtS8xDkSrtGTPm_I3QnVDHoLMk50m9v8z_nZKTkd-bnVUbreF1u0WnfV_X5eYevZl_$
> > > >> >> >>> > >
> > > >> >> >>> >
> > > >> >> >>> >
> > > >> >> >>> > As I mentioned in
> > > >> >> >>> >
> > > >> >> >>>
> > > >> >>
> > > >>
> > >
> >
> https://urldefense.com/v3/__https://github.com/apache/polaris/issues/3890__;!!Iz9xO38YGHZK!5EuyFFkk3vhRWVIRvQAWBSQfpJkTMA9HxugzDwXmN0LPPqhEFxYkFRGVhtb8AqUwXtDh2OplcMnbMDHKOxrvDU0$
> > > >> >> >>> ,
> > > >> >> >>> > supporting
> > > >> >> >>> > multiple data sources is not a trivial change. I would
> > strongly
> > > >> >> >>> recommend
> > > >> >> >>> > starting with a design document to carefully evaluate the
> > > >> >> architectural
> > > >> >> >>> > implications and long term impact.
> > > >> >> >>> >
> > > >> >> >>> > A REST endpoint to query metrics seems reasonable given the
> > > >> current
> > > >> >> >>> JDBC
> > > >> >> >>> > based persistence model. That said, we may also consider
> > > >> alternative
> > > >> >> >>> > storage models. For example, if we later adopt a time
> series
> > > >> system
> > > >> >> >>> such as
> > > >> >> >>> > Prometheus to store metrics, the query model and access
> > > patterns
> > > >> >> would
> > > >> >> >>> be
> > > >> >> >>> > fundamentally different. Designing the REST API without
> > > >> considering
> > > >> >> >>> these
> > > >> >> >>> > potential evolutions may limit flexibility. I'd suggest to
> > > start
> > > >> >> with
> > > >> >> >>> the
> > > >> >> >>> > use case.
> > > >> >> >>> >
> > > >> >> >>> > Yufei
> > > >> >> >>> >
> > > >> >> >>> >
> > > >> >> >>> > On Fri, Feb 27, 2026 at 3:42 PM Dmitri Bourlatchkov <
> > > >> >> dimas@apache.org>
> > > >> >> >>> > wrote:
> > > >> >> >>> >
> > > >> >> >>> > > Hi Anand,
> > > >> >> >>> > >
> > > >> >> >>> > > Sharing my view... subject to discussion:
> > > >> >> >>> > >
> > > >> >> >>> > > 1. Adding non-IRC REST API to Polaris is perfectly fine.
> > > >> >> >>> > >
> > > >> >> >>> > > Figuring out specific endpoint URIs and payloads might
> > > require
> > > >> a
> > > >> >> few
> > > >> >> >>> > > roundtrips, so opening a separate thread for that might
> be
> > > >> best.
> > > >> >> >>> > > Contributors commonly create Google Docs for new API
> > > proposals
> > > >> too
> > > >> >> >>> (they
> > > >> >> >>> > > fairly easy to update as the email discussion
> progresses).
> > > >> >> >>> > >
> > > >> >> >>> > > There was a suggestion to try Markdown (with PRs) for
> > > proposals
> > > >> >> [1]
> > > >> >> >>> ...
> > > >> >> >>> > > feel free to give it a try if you are comfortable with
> > that.
> > > >> >> >>> > >
> > > >> >> >>> > > 2. Could you clarify whether you mean end user utilities
> or
> > > >> admin
> > > >> >> >>> user
> > > >> >> >>> > > utilities? In the latter case those might be more
> suitable
> > > for
> > > >> the
> > > >> >> >>> Admin
> > > >> >> >>> > > CLI (java) not the Python CLI, IMHO.
> > > >> >> >>> > >
> > > >> >> >>> > > Why would these utilities be common with events? IMHO,
> > event
> > > >> use
> > > >> >> >>> cases
> > > >> >> >>> > are
> > > >> >> >>> > > distinct from scan/commit metrics.
> > > >> >> >>> > >
> > > >> >> >>> > > 3. I'd prefer separating metrics persistence from
> MetaStore
> > > >> >> >>> persistence
> > > >> >> >>> > at
> > > >> >> >>> > > the code level, so that they could be mixed and matched
> > > >> >> >>> independently.
> > > >> >> >>> > The
> > > >> >> >>> > > separate datasource question will become a non-issue with
> > > that
> > > >> >> >>> approach,
> > > >> >> >>> > I
> > > >> >> >>> > > guess.
> > > >> >> >>> > >
> > > >> >> >>> > > The rationale for separating scan metrics and metastore
> > > >> >> persistence
> > > >> >> >>> is
> > > >> >> >>> > that
> > > >> >> >>> > > "cascading deletes" between them are hardly ever
> required.
> > > >> >> >>> Furthermore,
> > > >> >> >>> > the
> > > >> >> >>> > > data and query patterns are very different so different
> > > >> >> technologies
> > > >> >> >>> > might
> > > >> >> >>> > > be beneficial in each case.
> > > >> >> >>> > >
> > > >> >> >>> > > [1]
> > > >> >> >>> >
> > > >> >> >>>
> > > >> >>
> > > >>
> > >
> >
> https://urldefense.com/v3/__https://lists.apache.org/thread/yto2wp982t43h1mqjwnslswhws5z47cy__;!!Iz9xO38YGHZK!5EuyFFkk3vhRWVIRvQAWBSQfpJkTMA9HxugzDwXmN0LPPqhEFxYkFRGVhtb8AqUwXtDh2OplcMnbMDHKxYDakNU$
> > > >> >> >>> > >
> > > >> >> >>> > > Cheers,
> > > >> >> >>> > > Dmitri.
> > > >> >> >>> > >
> > > >> >> >>> > > On Fri, Feb 27, 2026 at 6:19 PM Anand Kumar Sankaran via
> > dev
> > > <
> > > >> >> >>> > > dev@polaris.apache.org> wrote:
> > > >> >> >>> > >
> > > >> >> >>> > > > Thanks all. This PR is merged now.
> > > >> >> >>> > > >
> > > >> >> >>> > > > Here are the follow-up features / work needed.  These
> > were
> > > >> all
> > > >> >> >>> part of
> > > >> >> >>> > > the
> > > >> >> >>> > > > merged PR at some point in time and were removed to
> > reduce
> > > >> >> scope.
> > > >> >> >>> > > >
> > > >> >> >>> > > > Please let me know what you think.
> > > >> >> >>> > > >
> > > >> >> >>> > > >
> > > >> >> >>> > > >   1.  A REST API to paginate through table metrics.
> This
> > > >> will be
> > > >> >> >>> > non-IRC
> > > >> >> >>> > > > standard addition.
> > > >> >> >>> > > >   2.  Utilities for managing old records, should be
> > common
> > > >> with
> > > >> >> >>> events.
> > > >> >> >>> > > > There was some discussion that it belongs to the CLI.
> > > >> >> >>> > > >   3.  Separate datasource (metrics, events, even other
> > > >> tables?).
> > > >> >> >>> > > >
> > > >> >> >>> > > >
> > > >> >> >>> > > > Anything else?
> > > >> >> >>> > > >
> > > >> >> >>> > > > -
> > > >> >> >>> > > > Anand
> > > >> >> >>> > > >
> > > >> >> >>> > > >
> > > >> >> >>> > >
> > > >> >> >>> >
> > > >> >> >>> >
> > > >> >> >>>
> > > >> >> >>
> > > >> >>
> > > >> >
> > > >>
> > > >
> > >
> >
>

Re: Proposal for REST endpoints for table metrics and events

[Dmitri Bourlatchkov <di...@apache.org>]

Hi JB,

Could you set up another meeting, please? Same time on Wednesday as last
time... I hope it works for everyone.

Cheers,
Dmitri.

On Fri, May 15, 2026 at 8:06 PM Yufei Gu <fl...@gmail.com> wrote:

> +1 on another sync call next week.
>
> Yufei
>
>
> On Fri, May 15, 2026 at 4:52 PM Dmitri Bourlatchkov <di...@apache.org>
> wrote:
>
> > Hi All,
> >
> > WDYT about another sync call next week?
> >
> > Thanks,
> > Dmitri.
> >
> > On Wed, May 6, 2026 at 5:29 PM Dmitri Bourlatchkov <di...@apache.org>
> > wrote:
> >
> > > Hi EJ,
> > >
> > > Thanks for the summary! It covers what we discussed in the meeting very
> > > well, IMHO.
> > >
> > > Looking forward to concrete PRs :)
> > >
> > > Cheers,
> > > Dmitri.
> > >
> > > On Wed, May 6, 2026 at 5:08 PM EJ Wang <ej.wang.dev.interest@gmail.com
> >
> > > wrote:
> > >
> > >> Hi folks,
> > >>
> > >> We had a community sync earlier, thanks JB for scheduling it. Notes
> from
> > >> the first metrics architecture sync (May 6, 10-11am PT). Discussion
> doc
> > >> with per-section status:
> > >>
> > >>
> >
> https://docs.google.com/document/d/100h7c4damrUzVuquYbBHM0EvA4LSWuW2IT2dN_7nYVA/edit?tab=t.0
> > >>
> > >> *The meeting covered both topics from the doc. Direction-level
> alignment
> > >> was reached on the headline pieces; details remain for PR review or
> > >> follow-up sessions.*
> > >>
> > >> *Topic 1 — Persistence schema redesign*
> > >> Idea-level alignment on consolidating per-type tables
> > >> (scan_metrics_report,
> > >> commit_metrics_report) into a single metrics_report table. The
> > motivating
> > >> cost is the surface area added by every new metric type today: new
> > table,
> > >> SPI method, record class, model, converter, schema migration.
> > >>
> > >> Most schema details are deferred to the schema PR. A few specific
> points
> > >> came up:
> > >> •   metric_schema_version: Yufei prefers dropping it, since there is
> no
> > >> spec-level concept of metrics versioning today and it is hard to
> define
> > >> unilaterally. Robert prefers keeping it, given IRC v2 is coming and
> the
> > >> schema should be considered against its likely shape; Robert also
> raised
> > >> how to differentiate various payload formats if any. EJ's read is that
> > >> this
> > >> is a two-way-door decision. We can start without the field, and if IRC
> > v2
> > >> changes the shape we would likely roll a corresponding new schema
> > anyway,
> > >> which is not particularly costly.
> > >> •   Payload format: Robert pointed out that future formats beyond JSON
> > may
> > >> be worth supporting. The exact shape is deferred to the schema
> > discussion.
> > >> •   Partition strategy: Anand suggested monthly partitioning based on
> > his
> > >> experience as potentially helpful at scale.
> > >>
> > >> *Topic 2 — Where metrics ingestion and storage belong*
> > >> Idea-level alignment that metrics should be a separated SPI from the
> > >> entity
> > >> persistence stack. Two reasons surfaced: (a) workloads and capability
> > >> requirements diverge enough that coupling them creates artificial
> > >> constraints, and (b) admin experience improves when metrics has its
> own
> > >> bootstrap, retention, and lifecycle. Dmitri noted Polaris being a
> > platform
> > >> should have the flexibility to support different persistence backends
> > per
> > >> concern, and pointed to a concrete next step of separating the JDBC
> > >> bootstrap for metrics from the metastore bootstrap. Robert proposed an
> > >> additional UX extension: detect an unbootstrapped metrics store on
> first
> > >> use and auto-bootstrap rather than requiring an explicit manual
> > bootstrap
> > >> step.
> > >> The meeting also confirmed that Polaris metrics can start small and
> stay
> > >> Iceberg-focused. Naming and persistence schema can lean
> > Iceberg-specific.
> > >> If a future expansion to generic-table metrics or operational metrics
> > >> arrives, an abstraction layer can be built on top of the Iceberg
> metrics
> > >> reporter at that point. Robert remains on the fence and would prefer
> > >> something more generic but did not block the direction; Dmitri's read
> > was
> > >> that the proposed framework already has enough flexibility to absorb
> > >> future
> > >> expansion.
> > >>
> > >> The Trade-offs and Proposed structure sections in the doc were not
> > >> reviewed
> > >> in detail. They remain open for either the next sync or PR review.
> > >>
> > >> *Cross-cutting alignment — battery-included plus pluggable*
> > >> A common philosophy emerged from the discussion. EJ summarized it as:
> > >> Polaris should provide a battery-included UX for beginners and the
> > >> flexibility for advanced users to swap the included battery for
> > something
> > >> more powerful or tailored to their use case. The SPI design needs to
> > >> enable
> > >> both.
> > >>
> > >> The inputs that shaped this framing:
> > >> •   Anand described how his team uses the current metrics persistence
> > >> (three metrics consumers in v1.4).
> > >> •   Yufei raised Grafana and dashboard integrations as a destination
> use
> > >> case beyond the default.
> > >> •   Robert called out that the current design is more JDBC-focused.
> > >>
> > >> Two concrete instances:
> > >> •   Async metrics intake: Yufei's initial position was that async
> should
> > >> largely live on the producer side and there is not much Polaris can
> do.
> > >> Robert suggested a Polaris-side default is doable via Vert.x. Dmitri
> > >> agreed
> > >> the direction is worth exploring. The meeting converged on a
> > >> battery-included default (likely Vert.x-backed) with an SPI shape that
> > >> lets
> > >> power users route to a more scalable backend (k8s-hosted queue, AWS
> SQS,
> > >> etc.).
> > >> •   Pluggable destinations: combining Yufei's dashboard use case with
> > >> Robert's JDBC-focused call-out, the meeting agreed the SPI should be
> > >> structured for multiple sinks so integrations become impl choices
> rather
> > >> than architectural changes.
> > >>
> > >> The battery-included default is most likely to use the existing
> > >> JDBC-backed
> > >> approach.
> > >>
> > >> *Direction (idea-level alignment)*
> > >> •   Single metrics_report table consolidating per-type metrics,
> > replacing
> > >> scan_metrics_report and commit_metrics_report
> > >> •   Iceberg-focused naming and schema for now, revisit if
> generic-table
> > or
> > >> operational metrics arrive
> > >> •   Metrics persistence as a separated SPI, not on BasePersistence
> > >> •   Bootstrap path separated for metrics, independent of metastore
> > >> bootstrap
> > >> •   "Battery-included plus pluggable" as the SPI design philosophy
> > >>
> > >> *Open items*
> > >> •   Schema details: metric_schema_version, payload format, IRC v2
> > >> forward-compat shape
> > >> •   SPI design details — full review either in the next sync or in the
> > >> corresponding PR
> > >> •   Schema refactor PR ownership
> > >>
> > >> *Action items*
> > >> •   EJ to take a first stab at the SPI design and potentially partner
> > with
> > >> Anand to incorporate the lessons learned from the existing reporter
> and
> > >> persistence work.
> > >> •   Schema refactor PR ownership is not yet decided. If anyone is
> > >> interested in driving it, reply on this thread.
> > >> •   JB to schedule the next sync, tentatively in two weeks.
> > >>
> > >> -ej
> > >>
> > >> On Mon, Apr 27, 2026 at 3:07 PM EJ Wang <
> ej.wang.dev.interest@gmail.com
> > >
> > >> wrote:
> > >>
> > >> > Thanks Yufei for the +1.
> > >> >
> > >> > JB, could you help add a biweekly metrics architecture sync to the
> > >> Polaris
> > >> > community calendar? I'm thinking Thursdays at 9-10am PT, on the
> > >> off-weeks
> > >> > from the community meeting (starting May 7), 60 minutes.
> > >> >
> > >> > Here's a rough agenda to work through over the first few sessions,
> > >> grouped
> > >> > by priority:
> > >> >
> > >> > *First: foundational direction*
> > >> >
> > >> > 1.  MetricsPersistence: public SPI or internal implementation
> detail?
> > >> >    •   Marked @Beta, javadoc calls it a "Service Provider
> Interface",
> > >> but
> > >> > only one consumer (JdbcBasePersistenceImpl), lives on
> BasePersistence.
> > >> If
> > >> > demoted to a private helper inside a persisting reporter impl, most
> > >> > downstream design decisions become implementation details rather
> than
> > >> > contract questions.
> > >> >
> > >> > 2.  Persistence schema redesign
> > >> >    •   Current two-table layout (scan_metrics_report,
> > >> > commit_metrics_report) with ~25 flattened columns each. Every new
> > metric
> > >> > type requires a new table, SPI method, record class, model,
> converter,
> > >> and
> > >> > schema migration. Direction to explore: single table with
> metric_type
> > >> enum,
> > >> > schema_version, and JSON payload column.
> > >> >
> > >> > *Second: design details once direction is set*
> > >> >
> > >> > 3.  Partition key strategy
> > >> >    •   Single-table design means scan metrics at scale will have
> high
> > >> > write concurrency per table. Schema needs to expose enough structure
> > for
> > >> > backends to shard by entity or time range.
> > >> >
> > >> > 4.  Read/write path consistency
> > >> >    •   Writes go through PolarisMetricsManager on MetaStoreManager.
> > >> Reads
> > >> > bypass MetaStoreManager and go straight to BasePersistence,
> excluding
> > >> > non-JDBC backends from the read API.
> > >> >
> > >> > *Third: cleanup and alignment*
> > >> >
> > >> > 5.  PolarisMetricsReporter naming
> > >> >    •   Only handles IRC (ScanReport/CommitReport), doesn't cover
> > generic
> > >> > tables or operational metrics. Name is broader than scope.
> > >> >
> > >> > 6.  PolarisMetricsManager facade passthrough
> > >> >    •   Entire default method is
> > >> callCtx.getMetaStore().writeScanReport().
> > >> > Zero logic, passes Level 1 straight through to Level 3. Same
> > >> anti-pattern
> > >> > as PolarisEventManager.
> > >> >
> > >> > 7.  Iceberg community alignment
> > >> >    •   Payload-type extension needs discussion on dev@iceberg.
> > >> obelix74's
> > >> > Feb thread got zero replies. Needs a committer voice.
> > >> >
> > >> > Lets confirm prioritization in the first session.
> > >> >
> > >> > -ej
> > >> >
> > >> > On Tue, Apr 21, 2026 at 3:18 PM Yufei Gu <fl...@gmail.com>
> > wrote:
> > >> >
> > >> >> Thanks everyone for continuing to drive this forward. I agree that
> > the
> > >> >> problem is getting complex enough that a more structured discussion
> > >> would
> > >> >> help.
> > >> >>
> > >> >> +1 on setting up a biweekly sync for the metrics architecture. I’m
> > >> happy
> > >> >> to
> > >> >> join.
> > >> >>
> > >> >> Yufei
> > >> >>
> > >> >>
> > >> >> On Tue, Apr 21, 2026 at 2:34 PM EJ Wang <
> > >> ej.wang.dev.interest@gmail.com>
> > >> >> wrote:
> > >> >>
> > >> >> > Also, I've been looking more closely at the *persistence schema
> in
> > >> the
> > >> >> > current metrics work*, and I think there's a structural rigidity
> > >> problem
> > >> >> > worth raising before the shape gets locked in.
> > >> >> >
> > >> >> > Right now we have two separate tables (scan_metrics_report and
> > >> >> > commit_metrics_report), each with ~25 flattened columns that
> > directly
> > >> >> > mirror the Iceberg report fields. The SPI follows the same split:
> > >> >> > writeScanReport and writeCommitReport as separate methods, with
> > >> per-type
> > >> >> > record classes, converters, and model objects. *The practical
> cost:
> > >> >> > adding a new metric type (operational metrics, for example)
> > requires
> > >> a
> > >> >> new
> > >> >> > table, a new SPI method, a new record class, a new model class, a
> > new
> > >> >> > converter branch, and a schema migration*. That's a lot of
> surface
> > >> area
> > >> >> > for what should be "one more kind of metric."
> > >> >> >
> > >> >> > *My bias* would be toward a single metrics table with *a typed
> JSON
> > >> >> > payload*. Something like: metric_type (enum), entity_id,
> > >> >> > table_identifier, snapshot_id (nullable), received_ts,
> > >> schema_version,
> > >> >> and
> > >> >> > a payload column for the metric-specific data. The metric_type +
> > >> >> > schema_version pair gives us a forward-compatible contract for
> the
> > >> >> payload
> > >> >> > shape. Adding a new metric type becomes an enum value and a
> payload
> > >> >> schema,
> > >> >> > not a schema migration. One thing I think we need to be
> deliberate
> > >> >> about is
> > >> >> > the partition key design. If all metric types land in one table,
> > scan
> > >> >> > metrics at scale (high concurrency, high frequency across many
> > >> tables)
> > >> >> > could easily create hot partitions. We'd want the persistence
> layer
> > >> to
> > >> >> be
> > >> >> > able to shard by entity or time range, and that means the logical
> > >> schema
> > >> >> > needs to expose enough structure for backends to partition on. I
> > >> don't
> > >> >> > think the current flattened layout gives us that.
> > >> >> >
> > >> >> > This is getting complex enough that I don't think ad-hoc PR/ML
> > >> threads
> > >> >> > will converge well. *Would people be open to a biweekly sync for
> > >> metrics
> > >> >> > architecture?* I think 30 minutes every two weeks with interested
> > >> >> parties
> > >> >> > would be enough to work through the schema, SPI shape, and read
> API
> > >> >> design
> > >> >> > together. Happy to help set that up.
> > >> >> >
> > >> >> > -ej
> > >> >> >
> > >> >> > On Mon, Apr 20, 2026 at 2:19 PM EJ Wang <
> > >> ej.wang.dev.interest@gmail.com
> > >> >> >
> > >> >> > wrote:
> > >> >> >
> > >> >> >> Reviewed #4115, left a comment on the code organization side.
> > >> >> >>
> > >> >> >> One thing stood out: the metrics write path enters through
> > >> >> >> PolarisMetricsManager on MetaStoreManager, but the new read path
> > >> >> bypasses
> > >> >> >> MetaStoreManager entirely and goes straight to BasePersistence
> via
> > >> >> >> callContext.getMetaStore(). That means the read API only works
> for
> > >> >> backends
> > >> >> >> that implement BasePersistence. NoSQL and remote backends can't
> > >> >> participate.
> > >> >> >>
> > >> >> >> Stepping back, I think the metrics subsystem is growing into
> > >> something
> > >> >> >> real (write + read + REST API + AuthZ + pagination) *but the
> > >> >> persistence
> > >> >> >> side is split across two layers in a way that's hard to
> extend*. I
> > >> put
> > >> >> >> together two diagrams to show what I mean (my best effort).
> > >> >> >>
> > >> >> >> *Current state* (Diagram 1): three interfaces at three different
> > >> >> levels.
> > >> >> >> The engine-facing SPI (PolarisMetricsReporter) is clean. But
> > >> >> >> PolarisMetricsManager on MetaStoreManager is a passthrough to
> > >> >> >> MetricsPersistence on BasePersistence. The @Beta annotation and
> > SPI
> > >> >> javadoc
> > >> >> >> are on the BasePersistence layer, while the actual extension
> > points
> > >> >> >> (PolarisMetricsReporter, PolarisMetricsManager) carry no
> stability
> > >> >> >> annotation. The write path goes through the MetaStoreManager
> > layer,
> > >> the
> > >> >> >> read path doesn't.
> > >> >> >>
> > >> >> >> *What I envision* (Diagram 2): two SPIs at two levels.
> > >> >> >> PolarisMetricsReporter stays as the engine-facing SPI.
> > >> >> >> PolarisMetricsManager becomes the backend-facing SPI with both
> > write
> > >> >> and
> > >> >> >> read methods at the MetaStoreManager level, where any backend
> > (JDBC,
> > >> >> NoSQL,
> > >> >> >> remote) can implement them. MetricsPersistence on
> BasePersistence
> > >> goes
> > >> >> >> away. Where metrics actually land is an implementation detail,
> > not a
> > >> >> core
> > >> >> >> interface.
> > >> >> >>
> > >> >> >> *Minor naming thing*: PolarisMetricsReporter is broader than
> what
> > it
> > >> >> >> actually handles. It only accepts Iceberg REST Catalog metrics
> > >> >> (ScanReport,
> > >> >> >> CommitReport via MetricsReport). Generic table metrics or
> > >> operational
> > >> >> >> metrics aren't in scope. Not blocking, but worth noting if the
> > >> metrics
> > >> >> >> surface expands.
> > >> >> >>
> > >> >> >> *Rough sketch of how to get there*:
> > >> >> >>  1.  Add read methods to PolarisMetricsManager (listScanReports,
> > >> >> >> listCommitReports) with default no-op, same as the existing
> write
> > >> >> methods.
> > >> >> >> (Probably make PolarisMetricsManager more explicit on being
> > Iceberg
> > >> >> >> specific like package name or class name etc.)
> > >> >> >>  2.  Wire MetricsReportsService through MetaStoreManager instead
> > of
> > >> >> >> callContext.getMetaStore().
> > >> >> >>  3.  Extract metrics persistence from JdbcBasePersistenceImpl
> into
> > >> its
> > >> >> >> own class. That file carries ~7 responsibilities, metrics being
> > one
> > >> of
> > >> >> them.
> > >> >> >>  4.  Remove MetricsPersistence from BasePersistence.
> > >> >> >>
> > >> >> >> *None of this needs to happen in #4115. But if the direction
> makes
> > >> >> sense,
> > >> >> >> it would be good to align before the metrics surface grows
> > further.
> > >> >> Curious
> > >> >> >> what others think.*
> > >> >> >>
> > >> >> >> *My mental model note*: Level 1 MetaStoreManager; level 2
> > >> transactional
> > >> >> >> persistence; level 3 base persistence
> > >> >> >>
> > >> >> >> Diagram 1
> > >> >> >> <
> > >> >>
> > >>
> >
> https://www.plantuml.com/plantuml/uml/bLHDR-Cs4BthLmpIYupw0zbkKQ1r3M-S7Bp8xhhM7WCOb3IM65EaGD9EX2RzxHrHb4CxRelwa4YSDu_lpOVcnZ9jzvM8BBS2uGjQpJC3dtHMSekPtMk44IpsMgEqa5XcCOhCZikQQLP1pR8TAp2n3ILhmZDP20m0fcIvUkAoW2qJXd9z1bpToO9BX3WXu0ucy5rpgGPNm0nW5_epUWtm2Ue3pn3kMOFQmKntGZW0BYtgBSi8k5A2QMwybJNMIbFiGSR9QZc4nUqIvikStF0jHprua5C-amge42aNt3R0f5JaaoivdV2Pkqbx4hee4ymOkBh5BTiB-_uIeGeo8zL8rPsPl4DktdEiK1jkB1NdZCRbrSTecDe_mlHbF0wvBmCkaOH5_S8a_TTTKI6-nmCAkEw4LpxsZ-LbYLKQFKMNOgf_wuM7_bV9gOer5SYMMksBSWXFcbi49KNZXNLicwfe3TETC7gPdPqI7uBcHMb1RSzYq34c6PDUM9mn8HRsUTZEiDBve3NjVZumBj0U7SS37mGO7vcwtiK-_pU7U7L_f-digo9YbhSwIfMRwIITKGXbxdIUTCGF1SeCJxloKsU-3k9ddRbX1eDq1q_fx1JbBGT0glVyXimDuP4TQ5qpCAmnGEj2s_6n5mtn1z-97-63itFQZLPO1Ev2tu_WF7Ju-VPc0Skg5bYXxBhkY1xpD7EM_7fyflSpIsqMgVth5xhVr4eQxWQ8enaSAJQSG16yFSDuJ798rrcXr_3n-lfdk7icQjEBmFujL7AodiP_Y4Z7-YxvtZNs4zMgpNTl6tF8sglyPsmqchrjvQ-m-aP94r-TwCA2Ka8upPJZwtvSpoYCXkYMZU2NXvRMBfq9P3i3Le4VAZUAlUZ_oPKsxPgY0Q_BSKLkyr9bhQhQrJjo_x3TPlIB0DPjnMfcIoYP0QaYw1a0fTKDr8fB6ntNuvmoL1ZGkXa69Njh43zf9GiGxHQrA_jDYWRSzF5--WmTVrN97_Sm8LbLUy_lGBmLanJjFkDlGkRqjA_4tm00
> > >> >> >
> > >> >> >> :
> > >> >> >>
> > >> >> >> [image: image.png]
> > >> >> >>
> > >> >> >> Diagram 2
> > >> >> >> <
> > >> >>
> > >>
> >
> https://www.plantuml.com/plantuml/uml/VLLDR-8m4BtdLupO2sWBLVU8AaGB7AXAbssGzb896SS9RXqxjKqBwkv_tt7iV43fdaZYDpFlpRmnOsE9jhjSH9PRmM31hERKm8scMsuPjJlDe0yheZDc8RR4iYWoBrmMH9CS2a9VICPYUy1OZN0YCy5Q0BCbYNhdCeEK28En8G8wCvbnoQ0R8_05Bc6bkLIz3X03p1zzH7zR-9ZfDquPt9C3qoNCX2yV4G2NbkcKu5jdgGJHt0GbZwnG6i-UP3TUpk5gM6Ldqke350eZUqzoCft3U9xWHvxoa5-7K4nF1J46EbEMafsmdrCBbQ44gVggy18IZrn_ph5asd1ZiIKdQSgueZvjXrQFSFrdC3YN-nXmBacxbGiYyLVxLaBtdhqn0LSzdBDhqQtQoOJeGyad3z0lUqnYgpGB6Ns8oVyta00Dy_WnX0tIOZ8v6SYxHll1TrH6aejAik-mh-AphVFCwSUQqFypElag5QRGFDjQKEd96K1P8QP41c9TzA_IIQyvdAWyv_RSiS3skb0_EzDDkK2v5xWF6MiGFlvhpFLcD2Dq2pml14gaF67eQkmd8gulDoC4kSOu6KVpkvlUJg1RTbWISU40RdBUUS_9XfRZ2dwxm_SW8LYFISgm_MnlDQ6M9P1gbKEc4X-2pH_FvJCkCqm9pbVjD6LrwdLeOrDWfOaqc8Wh9BE85oNKxkNQ6o4yGRy_Eae0G_G8tZv81d3bHDB23WOdisohVr3nh_j6lbSjbNaLRTc8UgtPbAU1J_tygOfZX9DWEJeHDvYx-qmSi5FgNLPZwHrHcUsncGQ5-skhUclpE5fo4ounpFauYrUbkU6ccfnxMvitwag4IyerhTxj8In_Oj1bDO4pQru674loYrGlULHLEGCjwJJ8gDoVZR8MxO4BT3IzRvIcAQKezC6xpziGnTyImrfEGyJI_OcKfgtxIvnTqFEMS17L9Z-jsARN5FmTheP7HtSdtOMT0B4GY2FYHXxgQmMtj2bRqiLFGapiVe1_QVKDrkqXcm83aFEXnMYCZ-xlyHy
> > >> >> >
> > >> >> >> :
> > >> >> >> [image: image.png]
> > >> >> >>
> > >> >> >>  -ej
> > >> >> >>
> > >> >> >> On Wed, Apr 15, 2026 at 8:22 AM Dmitri Bourlatchkov <
> > >> dimas@apache.org>
> > >> >> >> wrote:
> > >> >> >>
> > >> >> >>> Hi All,
> > >> >> >>>
> > >> >> >>> Heads up: The current state of PR [4115] looks pretty solid to
> > me.
> > >> I
> > >> >> >>> believe this PR is approaching a mergeable condition.
> > >> >> >>>
> > >> >> >>> Please post your reviews if you have any comments.
> > >> >> >>>
> > >> >> >>> [4115] https://github.com/apache/polaris/pull/4115
> > >> >> >>>
> > >> >> >>> Thanks,
> > >> >> >>> Dmitri.
> > >> >> >>>
> > >> >> >>> On Tue, Mar 3, 2026 at 3:29 PM Anand Kumar Sankaran via dev <
> > >> >> >>> dev@polaris.apache.org> wrote:
> > >> >> >>>
> > >> >> >>> > Hi Yufei and Dmitri,
> > >> >> >>> >
> > >> >> >>> > Here is a proposal for the REST endpoints for metrics and
> > events.
> > >> >> >>> >
> > >> >> >>> > https://github.com/apache/polaris/pull/3924/changes
> > >> >> >>> >
> > >> >> >>> > I did not see any precursors for raising a PR for proposals,
> so
> > >> >> trying
> > >> >> >>> > this.  Please let me know what you think.
> > >> >> >>> >
> > >> >> >>> > -
> > >> >> >>> > Anand
> > >> >> >>> >
> > >> >> >>> > From: Anand Kumar Sankaran <an...@workday.com>
> > >> >> >>> > Date: Monday, March 2, 2026 at 10:25 AM
> > >> >> >>> > To: dev@polaris.apache.org <de...@polaris.apache.org>
> > >> >> >>> > Subject: Re: Polaris Telemetry and Audit Trail
> > >> >> >>> >
> > >> >> >>> > About the REST API, based on my use cases:
> > >> >> >>> >
> > >> >> >>> >
> > >> >> >>> >   1.
> > >> >> >>> > I want to be able to query commit metrics to track files
> added
> > /
> > >> >> >>> removed
> > >> >> >>> > per commit, along with record counts. The ingestion pipeline
> > that
> > >> >> >>> writes
> > >> >> >>> > this data is owned by us and we are guaranteed to write this
> > >> >> >>> information
> > >> >> >>> > for each write.
> > >> >> >>> >   2.
> > >> >> >>> > I want to be able to query scan metrics for read. I
> understand
> > >> >> clients
> > >> >> >>> do
> > >> >> >>> > not fulfill this requirement.
> > >> >> >>> >   3.
> > >> >> >>> > I want to be able to query the events table (events are
> > >> persisted) -
> > >> >> >>> this
> > >> >> >>> > may supersede #2, I am not sure yet.
> > >> >> >>> >
> > >> >> >>> > All this information is in the JDBC based persistence model
> and
> > >> is
> > >> >> >>> > persisted in the metastore. I currently don’t have a need to
> > >> query
> > >> >> >>> > prometheus or open telemetry. I do publish some events to
> > >> Prometheus
> > >> >> >>> and
> > >> >> >>> > they are forwarded to our dashboards elsewhere.
> > >> >> >>> >
> > >> >> >>> > About the CLI utilities, I meant the admin user utilities. In
> > >> one of
> > >> >> >>> the
> > >> >> >>> > earliest drafts of my proposal, Prashant mentioned that the
> > >> metrics
> > >> >> >>> tables
> > >> >> >>> > can grow indefinitely and that a similar problem exists with
> > the
> > >> >> events
> > >> >> >>> > table as well. We discussed that cleaning up of old records
> > from
> > >> >> both
> > >> >> >>> > metrics tables and events tables can be done via a CLI
> utility.
> > >> >> >>> >
> > >> >> >>> > I see that Yufei has covered the discussion about
> datasources.
> > >> >> >>> >
> > >> >> >>> > -
> > >> >> >>> > Anand
> > >> >> >>> >
> > >> >> >>> >
> > >> >> >>> >
> > >> >> >>> > From: Yufei Gu <fl...@gmail.com>
> > >> >> >>> > Date: Friday, February 27, 2026 at 9:54 PM
> > >> >> >>> > To: dev@polaris.apache.org <de...@polaris.apache.org>
> > >> >> >>> > Subject: Re: Polaris Telemetry and Audit Trail
> > >> >> >>> >
> > >> >> >>> > This Message Is From an External Sender
> > >> >> >>> > This message came from outside your organization.
> > >> >> >>> > Report Suspicious<
> > >> >> >>> >
> > >> >> >>>
> > >> >>
> > >>
> >
> https://us-phishalarm-ewt.proofpoint.com/EWT/v1/Iz9xO38YGHZK!YhNDZABkHi1B699ote2uMwpOZw8i0QMCGO2Szc-HshuABGhGvwPJcymE6G2oUUxtS8xDkSrtGTPm_I3QnVDHoLMk50m9v8z_nZKTkd-bnVUbreF1u0WnfV_X5eYevZl_$
> > >> >> >>> > >
> > >> >> >>> >
> > >> >> >>> >
> > >> >> >>> > As I mentioned in
> > >> >> >>> >
> > >> >> >>>
> > >> >>
> > >>
> >
> https://urldefense.com/v3/__https://github.com/apache/polaris/issues/3890__;!!Iz9xO38YGHZK!5EuyFFkk3vhRWVIRvQAWBSQfpJkTMA9HxugzDwXmN0LPPqhEFxYkFRGVhtb8AqUwXtDh2OplcMnbMDHKOxrvDU0$
> > >> >> >>> ,
> > >> >> >>> > supporting
> > >> >> >>> > multiple data sources is not a trivial change. I would
> strongly
> > >> >> >>> recommend
> > >> >> >>> > starting with a design document to carefully evaluate the
> > >> >> architectural
> > >> >> >>> > implications and long term impact.
> > >> >> >>> >
> > >> >> >>> > A REST endpoint to query metrics seems reasonable given the
> > >> current
> > >> >> >>> JDBC
> > >> >> >>> > based persistence model. That said, we may also consider
> > >> alternative
> > >> >> >>> > storage models. For example, if we later adopt a time series
> > >> system
> > >> >> >>> such as
> > >> >> >>> > Prometheus to store metrics, the query model and access
> > patterns
> > >> >> would
> > >> >> >>> be
> > >> >> >>> > fundamentally different. Designing the REST API without
> > >> considering
> > >> >> >>> these
> > >> >> >>> > potential evolutions may limit flexibility. I'd suggest to
> > start
> > >> >> with
> > >> >> >>> the
> > >> >> >>> > use case.
> > >> >> >>> >
> > >> >> >>> > Yufei
> > >> >> >>> >
> > >> >> >>> >
> > >> >> >>> > On Fri, Feb 27, 2026 at 3:42 PM Dmitri Bourlatchkov <
> > >> >> dimas@apache.org>
> > >> >> >>> > wrote:
> > >> >> >>> >
> > >> >> >>> > > Hi Anand,
> > >> >> >>> > >
> > >> >> >>> > > Sharing my view... subject to discussion:
> > >> >> >>> > >
> > >> >> >>> > > 1. Adding non-IRC REST API to Polaris is perfectly fine.
> > >> >> >>> > >
> > >> >> >>> > > Figuring out specific endpoint URIs and payloads might
> > require
> > >> a
> > >> >> few
> > >> >> >>> > > roundtrips, so opening a separate thread for that might be
> > >> best.
> > >> >> >>> > > Contributors commonly create Google Docs for new API
> > proposals
> > >> too
> > >> >> >>> (they
> > >> >> >>> > > fairly easy to update as the email discussion progresses).
> > >> >> >>> > >
> > >> >> >>> > > There was a suggestion to try Markdown (with PRs) for
> > proposals
> > >> >> [1]
> > >> >> >>> ...
> > >> >> >>> > > feel free to give it a try if you are comfortable with
> that.
> > >> >> >>> > >
> > >> >> >>> > > 2. Could you clarify whether you mean end user utilities or
> > >> admin
> > >> >> >>> user
> > >> >> >>> > > utilities? In the latter case those might be more suitable
> > for
> > >> the
> > >> >> >>> Admin
> > >> >> >>> > > CLI (java) not the Python CLI, IMHO.
> > >> >> >>> > >
> > >> >> >>> > > Why would these utilities be common with events? IMHO,
> event
> > >> use
> > >> >> >>> cases
> > >> >> >>> > are
> > >> >> >>> > > distinct from scan/commit metrics.
> > >> >> >>> > >
> > >> >> >>> > > 3. I'd prefer separating metrics persistence from MetaStore
> > >> >> >>> persistence
> > >> >> >>> > at
> > >> >> >>> > > the code level, so that they could be mixed and matched
> > >> >> >>> independently.
> > >> >> >>> > The
> > >> >> >>> > > separate datasource question will become a non-issue with
> > that
> > >> >> >>> approach,
> > >> >> >>> > I
> > >> >> >>> > > guess.
> > >> >> >>> > >
> > >> >> >>> > > The rationale for separating scan metrics and metastore
> > >> >> persistence
> > >> >> >>> is
> > >> >> >>> > that
> > >> >> >>> > > "cascading deletes" between them are hardly ever required.
> > >> >> >>> Furthermore,
> > >> >> >>> > the
> > >> >> >>> > > data and query patterns are very different so different
> > >> >> technologies
> > >> >> >>> > might
> > >> >> >>> > > be beneficial in each case.
> > >> >> >>> > >
> > >> >> >>> > > [1]
> > >> >> >>> >
> > >> >> >>>
> > >> >>
> > >>
> >
> https://urldefense.com/v3/__https://lists.apache.org/thread/yto2wp982t43h1mqjwnslswhws5z47cy__;!!Iz9xO38YGHZK!5EuyFFkk3vhRWVIRvQAWBSQfpJkTMA9HxugzDwXmN0LPPqhEFxYkFRGVhtb8AqUwXtDh2OplcMnbMDHKxYDakNU$
> > >> >> >>> > >
> > >> >> >>> > > Cheers,
> > >> >> >>> > > Dmitri.
> > >> >> >>> > >
> > >> >> >>> > > On Fri, Feb 27, 2026 at 6:19 PM Anand Kumar Sankaran via
> dev
> > <
> > >> >> >>> > > dev@polaris.apache.org> wrote:
> > >> >> >>> > >
> > >> >> >>> > > > Thanks all. This PR is merged now.
> > >> >> >>> > > >
> > >> >> >>> > > > Here are the follow-up features / work needed.  These
> were
> > >> all
> > >> >> >>> part of
> > >> >> >>> > > the
> > >> >> >>> > > > merged PR at some point in time and were removed to
> reduce
> > >> >> scope.
> > >> >> >>> > > >
> > >> >> >>> > > > Please let me know what you think.
> > >> >> >>> > > >
> > >> >> >>> > > >
> > >> >> >>> > > >   1.  A REST API to paginate through table metrics. This
> > >> will be
> > >> >> >>> > non-IRC
> > >> >> >>> > > > standard addition.
> > >> >> >>> > > >   2.  Utilities for managing old records, should be
> common
> > >> with
> > >> >> >>> events.
> > >> >> >>> > > > There was some discussion that it belongs to the CLI.
> > >> >> >>> > > >   3.  Separate datasource (metrics, events, even other
> > >> tables?).
> > >> >> >>> > > >
> > >> >> >>> > > >
> > >> >> >>> > > > Anything else?
> > >> >> >>> > > >
> > >> >> >>> > > > -
> > >> >> >>> > > > Anand
> > >> >> >>> > > >
> > >> >> >>> > > >
> > >> >> >>> > >
> > >> >> >>> >
> > >> >> >>> >
> > >> >> >>>
> > >> >> >>
> > >> >>
> > >> >
> > >>
> > >
> >
>

Re: Proposal for REST endpoints for table metrics and events

[Yufei Gu <fl...@gmail.com>]

+1 on another sync call next week.

Yufei


On Fri, May 15, 2026 at 4:52 PM Dmitri Bourlatchkov <di...@apache.org>
wrote:

> Hi All,
>
> WDYT about another sync call next week?
>
> Thanks,
> Dmitri.
>
> On Wed, May 6, 2026 at 5:29 PM Dmitri Bourlatchkov <di...@apache.org>
> wrote:
>
> > Hi EJ,
> >
> > Thanks for the summary! It covers what we discussed in the meeting very
> > well, IMHO.
> >
> > Looking forward to concrete PRs :)
> >
> > Cheers,
> > Dmitri.
> >
> > On Wed, May 6, 2026 at 5:08 PM EJ Wang <ej...@gmail.com>
> > wrote:
> >
> >> Hi folks,
> >>
> >> We had a community sync earlier, thanks JB for scheduling it. Notes from
> >> the first metrics architecture sync (May 6, 10-11am PT). Discussion doc
> >> with per-section status:
> >>
> >>
> https://docs.google.com/document/d/100h7c4damrUzVuquYbBHM0EvA4LSWuW2IT2dN_7nYVA/edit?tab=t.0
> >>
> >> *The meeting covered both topics from the doc. Direction-level alignment
> >> was reached on the headline pieces; details remain for PR review or
> >> follow-up sessions.*
> >>
> >> *Topic 1 — Persistence schema redesign*
> >> Idea-level alignment on consolidating per-type tables
> >> (scan_metrics_report,
> >> commit_metrics_report) into a single metrics_report table. The
> motivating
> >> cost is the surface area added by every new metric type today: new
> table,
> >> SPI method, record class, model, converter, schema migration.
> >>
> >> Most schema details are deferred to the schema PR. A few specific points
> >> came up:
> >> •   metric_schema_version: Yufei prefers dropping it, since there is no
> >> spec-level concept of metrics versioning today and it is hard to define
> >> unilaterally. Robert prefers keeping it, given IRC v2 is coming and the
> >> schema should be considered against its likely shape; Robert also raised
> >> how to differentiate various payload formats if any. EJ's read is that
> >> this
> >> is a two-way-door decision. We can start without the field, and if IRC
> v2
> >> changes the shape we would likely roll a corresponding new schema
> anyway,
> >> which is not particularly costly.
> >> •   Payload format: Robert pointed out that future formats beyond JSON
> may
> >> be worth supporting. The exact shape is deferred to the schema
> discussion.
> >> •   Partition strategy: Anand suggested monthly partitioning based on
> his
> >> experience as potentially helpful at scale.
> >>
> >> *Topic 2 — Where metrics ingestion and storage belong*
> >> Idea-level alignment that metrics should be a separated SPI from the
> >> entity
> >> persistence stack. Two reasons surfaced: (a) workloads and capability
> >> requirements diverge enough that coupling them creates artificial
> >> constraints, and (b) admin experience improves when metrics has its own
> >> bootstrap, retention, and lifecycle. Dmitri noted Polaris being a
> platform
> >> should have the flexibility to support different persistence backends
> per
> >> concern, and pointed to a concrete next step of separating the JDBC
> >> bootstrap for metrics from the metastore bootstrap. Robert proposed an
> >> additional UX extension: detect an unbootstrapped metrics store on first
> >> use and auto-bootstrap rather than requiring an explicit manual
> bootstrap
> >> step.
> >> The meeting also confirmed that Polaris metrics can start small and stay
> >> Iceberg-focused. Naming and persistence schema can lean
> Iceberg-specific.
> >> If a future expansion to generic-table metrics or operational metrics
> >> arrives, an abstraction layer can be built on top of the Iceberg metrics
> >> reporter at that point. Robert remains on the fence and would prefer
> >> something more generic but did not block the direction; Dmitri's read
> was
> >> that the proposed framework already has enough flexibility to absorb
> >> future
> >> expansion.
> >>
> >> The Trade-offs and Proposed structure sections in the doc were not
> >> reviewed
> >> in detail. They remain open for either the next sync or PR review.
> >>
> >> *Cross-cutting alignment — battery-included plus pluggable*
> >> A common philosophy emerged from the discussion. EJ summarized it as:
> >> Polaris should provide a battery-included UX for beginners and the
> >> flexibility for advanced users to swap the included battery for
> something
> >> more powerful or tailored to their use case. The SPI design needs to
> >> enable
> >> both.
> >>
> >> The inputs that shaped this framing:
> >> •   Anand described how his team uses the current metrics persistence
> >> (three metrics consumers in v1.4).
> >> •   Yufei raised Grafana and dashboard integrations as a destination use
> >> case beyond the default.
> >> •   Robert called out that the current design is more JDBC-focused.
> >>
> >> Two concrete instances:
> >> •   Async metrics intake: Yufei's initial position was that async should
> >> largely live on the producer side and there is not much Polaris can do.
> >> Robert suggested a Polaris-side default is doable via Vert.x. Dmitri
> >> agreed
> >> the direction is worth exploring. The meeting converged on a
> >> battery-included default (likely Vert.x-backed) with an SPI shape that
> >> lets
> >> power users route to a more scalable backend (k8s-hosted queue, AWS SQS,
> >> etc.).
> >> •   Pluggable destinations: combining Yufei's dashboard use case with
> >> Robert's JDBC-focused call-out, the meeting agreed the SPI should be
> >> structured for multiple sinks so integrations become impl choices rather
> >> than architectural changes.
> >>
> >> The battery-included default is most likely to use the existing
> >> JDBC-backed
> >> approach.
> >>
> >> *Direction (idea-level alignment)*
> >> •   Single metrics_report table consolidating per-type metrics,
> replacing
> >> scan_metrics_report and commit_metrics_report
> >> •   Iceberg-focused naming and schema for now, revisit if generic-table
> or
> >> operational metrics arrive
> >> •   Metrics persistence as a separated SPI, not on BasePersistence
> >> •   Bootstrap path separated for metrics, independent of metastore
> >> bootstrap
> >> •   "Battery-included plus pluggable" as the SPI design philosophy
> >>
> >> *Open items*
> >> •   Schema details: metric_schema_version, payload format, IRC v2
> >> forward-compat shape
> >> •   SPI design details — full review either in the next sync or in the
> >> corresponding PR
> >> •   Schema refactor PR ownership
> >>
> >> *Action items*
> >> •   EJ to take a first stab at the SPI design and potentially partner
> with
> >> Anand to incorporate the lessons learned from the existing reporter and
> >> persistence work.
> >> •   Schema refactor PR ownership is not yet decided. If anyone is
> >> interested in driving it, reply on this thread.
> >> •   JB to schedule the next sync, tentatively in two weeks.
> >>
> >> -ej
> >>
> >> On Mon, Apr 27, 2026 at 3:07 PM EJ Wang <ej.wang.dev.interest@gmail.com
> >
> >> wrote:
> >>
> >> > Thanks Yufei for the +1.
> >> >
> >> > JB, could you help add a biweekly metrics architecture sync to the
> >> Polaris
> >> > community calendar? I'm thinking Thursdays at 9-10am PT, on the
> >> off-weeks
> >> > from the community meeting (starting May 7), 60 minutes.
> >> >
> >> > Here's a rough agenda to work through over the first few sessions,
> >> grouped
> >> > by priority:
> >> >
> >> > *First: foundational direction*
> >> >
> >> > 1.  MetricsPersistence: public SPI or internal implementation detail?
> >> >    •   Marked @Beta, javadoc calls it a "Service Provider Interface",
> >> but
> >> > only one consumer (JdbcBasePersistenceImpl), lives on BasePersistence.
> >> If
> >> > demoted to a private helper inside a persisting reporter impl, most
> >> > downstream design decisions become implementation details rather than
> >> > contract questions.
> >> >
> >> > 2.  Persistence schema redesign
> >> >    •   Current two-table layout (scan_metrics_report,
> >> > commit_metrics_report) with ~25 flattened columns each. Every new
> metric
> >> > type requires a new table, SPI method, record class, model, converter,
> >> and
> >> > schema migration. Direction to explore: single table with metric_type
> >> enum,
> >> > schema_version, and JSON payload column.
> >> >
> >> > *Second: design details once direction is set*
> >> >
> >> > 3.  Partition key strategy
> >> >    •   Single-table design means scan metrics at scale will have high
> >> > write concurrency per table. Schema needs to expose enough structure
> for
> >> > backends to shard by entity or time range.
> >> >
> >> > 4.  Read/write path consistency
> >> >    •   Writes go through PolarisMetricsManager on MetaStoreManager.
> >> Reads
> >> > bypass MetaStoreManager and go straight to BasePersistence, excluding
> >> > non-JDBC backends from the read API.
> >> >
> >> > *Third: cleanup and alignment*
> >> >
> >> > 5.  PolarisMetricsReporter naming
> >> >    •   Only handles IRC (ScanReport/CommitReport), doesn't cover
> generic
> >> > tables or operational metrics. Name is broader than scope.
> >> >
> >> > 6.  PolarisMetricsManager facade passthrough
> >> >    •   Entire default method is
> >> callCtx.getMetaStore().writeScanReport().
> >> > Zero logic, passes Level 1 straight through to Level 3. Same
> >> anti-pattern
> >> > as PolarisEventManager.
> >> >
> >> > 7.  Iceberg community alignment
> >> >    •   Payload-type extension needs discussion on dev@iceberg.
> >> obelix74's
> >> > Feb thread got zero replies. Needs a committer voice.
> >> >
> >> > Lets confirm prioritization in the first session.
> >> >
> >> > -ej
> >> >
> >> > On Tue, Apr 21, 2026 at 3:18 PM Yufei Gu <fl...@gmail.com>
> wrote:
> >> >
> >> >> Thanks everyone for continuing to drive this forward. I agree that
> the
> >> >> problem is getting complex enough that a more structured discussion
> >> would
> >> >> help.
> >> >>
> >> >> +1 on setting up a biweekly sync for the metrics architecture. I’m
> >> happy
> >> >> to
> >> >> join.
> >> >>
> >> >> Yufei
> >> >>
> >> >>
> >> >> On Tue, Apr 21, 2026 at 2:34 PM EJ Wang <
> >> ej.wang.dev.interest@gmail.com>
> >> >> wrote:
> >> >>
> >> >> > Also, I've been looking more closely at the *persistence schema in
> >> the
> >> >> > current metrics work*, and I think there's a structural rigidity
> >> problem
> >> >> > worth raising before the shape gets locked in.
> >> >> >
> >> >> > Right now we have two separate tables (scan_metrics_report and
> >> >> > commit_metrics_report), each with ~25 flattened columns that
> directly
> >> >> > mirror the Iceberg report fields. The SPI follows the same split:
> >> >> > writeScanReport and writeCommitReport as separate methods, with
> >> per-type
> >> >> > record classes, converters, and model objects. *The practical cost:
> >> >> > adding a new metric type (operational metrics, for example)
> requires
> >> a
> >> >> new
> >> >> > table, a new SPI method, a new record class, a new model class, a
> new
> >> >> > converter branch, and a schema migration*. That's a lot of surface
> >> area
> >> >> > for what should be "one more kind of metric."
> >> >> >
> >> >> > *My bias* would be toward a single metrics table with *a typed JSON
> >> >> > payload*. Something like: metric_type (enum), entity_id,
> >> >> > table_identifier, snapshot_id (nullable), received_ts,
> >> schema_version,
> >> >> and
> >> >> > a payload column for the metric-specific data. The metric_type +
> >> >> > schema_version pair gives us a forward-compatible contract for the
> >> >> payload
> >> >> > shape. Adding a new metric type becomes an enum value and a payload
> >> >> schema,
> >> >> > not a schema migration. One thing I think we need to be deliberate
> >> >> about is
> >> >> > the partition key design. If all metric types land in one table,
> scan
> >> >> > metrics at scale (high concurrency, high frequency across many
> >> tables)
> >> >> > could easily create hot partitions. We'd want the persistence layer
> >> to
> >> >> be
> >> >> > able to shard by entity or time range, and that means the logical
> >> schema
> >> >> > needs to expose enough structure for backends to partition on. I
> >> don't
> >> >> > think the current flattened layout gives us that.
> >> >> >
> >> >> > This is getting complex enough that I don't think ad-hoc PR/ML
> >> threads
> >> >> > will converge well. *Would people be open to a biweekly sync for
> >> metrics
> >> >> > architecture?* I think 30 minutes every two weeks with interested
> >> >> parties
> >> >> > would be enough to work through the schema, SPI shape, and read API
> >> >> design
> >> >> > together. Happy to help set that up.
> >> >> >
> >> >> > -ej
> >> >> >
> >> >> > On Mon, Apr 20, 2026 at 2:19 PM EJ Wang <
> >> ej.wang.dev.interest@gmail.com
> >> >> >
> >> >> > wrote:
> >> >> >
> >> >> >> Reviewed #4115, left a comment on the code organization side.
> >> >> >>
> >> >> >> One thing stood out: the metrics write path enters through
> >> >> >> PolarisMetricsManager on MetaStoreManager, but the new read path
> >> >> bypasses
> >> >> >> MetaStoreManager entirely and goes straight to BasePersistence via
> >> >> >> callContext.getMetaStore(). That means the read API only works for
> >> >> backends
> >> >> >> that implement BasePersistence. NoSQL and remote backends can't
> >> >> participate.
> >> >> >>
> >> >> >> Stepping back, I think the metrics subsystem is growing into
> >> something
> >> >> >> real (write + read + REST API + AuthZ + pagination) *but the
> >> >> persistence
> >> >> >> side is split across two layers in a way that's hard to extend*. I
> >> put
> >> >> >> together two diagrams to show what I mean (my best effort).
> >> >> >>
> >> >> >> *Current state* (Diagram 1): three interfaces at three different
> >> >> levels.
> >> >> >> The engine-facing SPI (PolarisMetricsReporter) is clean. But
> >> >> >> PolarisMetricsManager on MetaStoreManager is a passthrough to
> >> >> >> MetricsPersistence on BasePersistence. The @Beta annotation and
> SPI
> >> >> javadoc
> >> >> >> are on the BasePersistence layer, while the actual extension
> points
> >> >> >> (PolarisMetricsReporter, PolarisMetricsManager) carry no stability
> >> >> >> annotation. The write path goes through the MetaStoreManager
> layer,
> >> the
> >> >> >> read path doesn't.
> >> >> >>
> >> >> >> *What I envision* (Diagram 2): two SPIs at two levels.
> >> >> >> PolarisMetricsReporter stays as the engine-facing SPI.
> >> >> >> PolarisMetricsManager becomes the backend-facing SPI with both
> write
> >> >> and
> >> >> >> read methods at the MetaStoreManager level, where any backend
> (JDBC,
> >> >> NoSQL,
> >> >> >> remote) can implement them. MetricsPersistence on BasePersistence
> >> goes
> >> >> >> away. Where metrics actually land is an implementation detail,
> not a
> >> >> core
> >> >> >> interface.
> >> >> >>
> >> >> >> *Minor naming thing*: PolarisMetricsReporter is broader than what
> it
> >> >> >> actually handles. It only accepts Iceberg REST Catalog metrics
> >> >> (ScanReport,
> >> >> >> CommitReport via MetricsReport). Generic table metrics or
> >> operational
> >> >> >> metrics aren't in scope. Not blocking, but worth noting if the
> >> metrics
> >> >> >> surface expands.
> >> >> >>
> >> >> >> *Rough sketch of how to get there*:
> >> >> >>  1.  Add read methods to PolarisMetricsManager (listScanReports,
> >> >> >> listCommitReports) with default no-op, same as the existing write
> >> >> methods.
> >> >> >> (Probably make PolarisMetricsManager more explicit on being
> Iceberg
> >> >> >> specific like package name or class name etc.)
> >> >> >>  2.  Wire MetricsReportsService through MetaStoreManager instead
> of
> >> >> >> callContext.getMetaStore().
> >> >> >>  3.  Extract metrics persistence from JdbcBasePersistenceImpl into
> >> its
> >> >> >> own class. That file carries ~7 responsibilities, metrics being
> one
> >> of
> >> >> them.
> >> >> >>  4.  Remove MetricsPersistence from BasePersistence.
> >> >> >>
> >> >> >> *None of this needs to happen in #4115. But if the direction makes
> >> >> sense,
> >> >> >> it would be good to align before the metrics surface grows
> further.
> >> >> Curious
> >> >> >> what others think.*
> >> >> >>
> >> >> >> *My mental model note*: Level 1 MetaStoreManager; level 2
> >> transactional
> >> >> >> persistence; level 3 base persistence
> >> >> >>
> >> >> >> Diagram 1
> >> >> >> <
> >> >>
> >>
> https://www.plantuml.com/plantuml/uml/bLHDR-Cs4BthLmpIYupw0zbkKQ1r3M-S7Bp8xhhM7WCOb3IM65EaGD9EX2RzxHrHb4CxRelwa4YSDu_lpOVcnZ9jzvM8BBS2uGjQpJC3dtHMSekPtMk44IpsMgEqa5XcCOhCZikQQLP1pR8TAp2n3ILhmZDP20m0fcIvUkAoW2qJXd9z1bpToO9BX3WXu0ucy5rpgGPNm0nW5_epUWtm2Ue3pn3kMOFQmKntGZW0BYtgBSi8k5A2QMwybJNMIbFiGSR9QZc4nUqIvikStF0jHprua5C-amge42aNt3R0f5JaaoivdV2Pkqbx4hee4ymOkBh5BTiB-_uIeGeo8zL8rPsPl4DktdEiK1jkB1NdZCRbrSTecDe_mlHbF0wvBmCkaOH5_S8a_TTTKI6-nmCAkEw4LpxsZ-LbYLKQFKMNOgf_wuM7_bV9gOer5SYMMksBSWXFcbi49KNZXNLicwfe3TETC7gPdPqI7uBcHMb1RSzYq34c6PDUM9mn8HRsUTZEiDBve3NjVZumBj0U7SS37mGO7vcwtiK-_pU7U7L_f-digo9YbhSwIfMRwIITKGXbxdIUTCGF1SeCJxloKsU-3k9ddRbX1eDq1q_fx1JbBGT0glVyXimDuP4TQ5qpCAmnGEj2s_6n5mtn1z-97-63itFQZLPO1Ev2tu_WF7Ju-VPc0Skg5bYXxBhkY1xpD7EM_7fyflSpIsqMgVth5xhVr4eQxWQ8enaSAJQSG16yFSDuJ798rrcXr_3n-lfdk7icQjEBmFujL7AodiP_Y4Z7-YxvtZNs4zMgpNTl6tF8sglyPsmqchrjvQ-m-aP94r-TwCA2Ka8upPJZwtvSpoYCXkYMZU2NXvRMBfq9P3i3Le4VAZUAlUZ_oPKsxPgY0Q_BSKLkyr9bhQhQrJjo_x3TPlIB0DPjnMfcIoYP0QaYw1a0fTKDr8fB6ntNuvmoL1ZGkXa69Njh43zf9GiGxHQrA_jDYWRSzF5--WmTVrN97_Sm8LbLUy_lGBmLanJjFkDlGkRqjA_4tm00
> >> >> >
> >> >> >> :
> >> >> >>
> >> >> >> [image: image.png]
> >> >> >>
> >> >> >> Diagram 2
> >> >> >> <
> >> >>
> >>
> https://www.plantuml.com/plantuml/uml/VLLDR-8m4BtdLupO2sWBLVU8AaGB7AXAbssGzb896SS9RXqxjKqBwkv_tt7iV43fdaZYDpFlpRmnOsE9jhjSH9PRmM31hERKm8scMsuPjJlDe0yheZDc8RR4iYWoBrmMH9CS2a9VICPYUy1OZN0YCy5Q0BCbYNhdCeEK28En8G8wCvbnoQ0R8_05Bc6bkLIz3X03p1zzH7zR-9ZfDquPt9C3qoNCX2yV4G2NbkcKu5jdgGJHt0GbZwnG6i-UP3TUpk5gM6Ldqke350eZUqzoCft3U9xWHvxoa5-7K4nF1J46EbEMafsmdrCBbQ44gVggy18IZrn_ph5asd1ZiIKdQSgueZvjXrQFSFrdC3YN-nXmBacxbGiYyLVxLaBtdhqn0LSzdBDhqQtQoOJeGyad3z0lUqnYgpGB6Ns8oVyta00Dy_WnX0tIOZ8v6SYxHll1TrH6aejAik-mh-AphVFCwSUQqFypElag5QRGFDjQKEd96K1P8QP41c9TzA_IIQyvdAWyv_RSiS3skb0_EzDDkK2v5xWF6MiGFlvhpFLcD2Dq2pml14gaF67eQkmd8gulDoC4kSOu6KVpkvlUJg1RTbWISU40RdBUUS_9XfRZ2dwxm_SW8LYFISgm_MnlDQ6M9P1gbKEc4X-2pH_FvJCkCqm9pbVjD6LrwdLeOrDWfOaqc8Wh9BE85oNKxkNQ6o4yGRy_Eae0G_G8tZv81d3bHDB23WOdisohVr3nh_j6lbSjbNaLRTc8UgtPbAU1J_tygOfZX9DWEJeHDvYx-qmSi5FgNLPZwHrHcUsncGQ5-skhUclpE5fo4ounpFauYrUbkU6ccfnxMvitwag4IyerhTxj8In_Oj1bDO4pQru674loYrGlULHLEGCjwJJ8gDoVZR8MxO4BT3IzRvIcAQKezC6xpziGnTyImrfEGyJI_OcKfgtxIvnTqFEMS17L9Z-jsARN5FmTheP7HtSdtOMT0B4GY2FYHXxgQmMtj2bRqiLFGapiVe1_QVKDrkqXcm83aFEXnMYCZ-xlyHy
> >> >> >
> >> >> >> :
> >> >> >> [image: image.png]
> >> >> >>
> >> >> >>  -ej
> >> >> >>
> >> >> >> On Wed, Apr 15, 2026 at 8:22 AM Dmitri Bourlatchkov <
> >> dimas@apache.org>
> >> >> >> wrote:
> >> >> >>
> >> >> >>> Hi All,
> >> >> >>>
> >> >> >>> Heads up: The current state of PR [4115] looks pretty solid to
> me.
> >> I
> >> >> >>> believe this PR is approaching a mergeable condition.
> >> >> >>>
> >> >> >>> Please post your reviews if you have any comments.
> >> >> >>>
> >> >> >>> [4115] https://github.com/apache/polaris/pull/4115
> >> >> >>>
> >> >> >>> Thanks,
> >> >> >>> Dmitri.
> >> >> >>>
> >> >> >>> On Tue, Mar 3, 2026 at 3:29 PM Anand Kumar Sankaran via dev <
> >> >> >>> dev@polaris.apache.org> wrote:
> >> >> >>>
> >> >> >>> > Hi Yufei and Dmitri,
> >> >> >>> >
> >> >> >>> > Here is a proposal for the REST endpoints for metrics and
> events.
> >> >> >>> >
> >> >> >>> > https://github.com/apache/polaris/pull/3924/changes
> >> >> >>> >
> >> >> >>> > I did not see any precursors for raising a PR for proposals, so
> >> >> trying
> >> >> >>> > this.  Please let me know what you think.
> >> >> >>> >
> >> >> >>> > -
> >> >> >>> > Anand
> >> >> >>> >
> >> >> >>> > From: Anand Kumar Sankaran <an...@workday.com>
> >> >> >>> > Date: Monday, March 2, 2026 at 10:25 AM
> >> >> >>> > To: dev@polaris.apache.org <de...@polaris.apache.org>
> >> >> >>> > Subject: Re: Polaris Telemetry and Audit Trail
> >> >> >>> >
> >> >> >>> > About the REST API, based on my use cases:
> >> >> >>> >
> >> >> >>> >
> >> >> >>> >   1.
> >> >> >>> > I want to be able to query commit metrics to track files added
> /
> >> >> >>> removed
> >> >> >>> > per commit, along with record counts. The ingestion pipeline
> that
> >> >> >>> writes
> >> >> >>> > this data is owned by us and we are guaranteed to write this
> >> >> >>> information
> >> >> >>> > for each write.
> >> >> >>> >   2.
> >> >> >>> > I want to be able to query scan metrics for read. I understand
> >> >> clients
> >> >> >>> do
> >> >> >>> > not fulfill this requirement.
> >> >> >>> >   3.
> >> >> >>> > I want to be able to query the events table (events are
> >> persisted) -
> >> >> >>> this
> >> >> >>> > may supersede #2, I am not sure yet.
> >> >> >>> >
> >> >> >>> > All this information is in the JDBC based persistence model and
> >> is
> >> >> >>> > persisted in the metastore. I currently don’t have a need to
> >> query
> >> >> >>> > prometheus or open telemetry. I do publish some events to
> >> Prometheus
> >> >> >>> and
> >> >> >>> > they are forwarded to our dashboards elsewhere.
> >> >> >>> >
> >> >> >>> > About the CLI utilities, I meant the admin user utilities. In
> >> one of
> >> >> >>> the
> >> >> >>> > earliest drafts of my proposal, Prashant mentioned that the
> >> metrics
> >> >> >>> tables
> >> >> >>> > can grow indefinitely and that a similar problem exists with
> the
> >> >> events
> >> >> >>> > table as well. We discussed that cleaning up of old records
> from
> >> >> both
> >> >> >>> > metrics tables and events tables can be done via a CLI utility.
> >> >> >>> >
> >> >> >>> > I see that Yufei has covered the discussion about datasources.
> >> >> >>> >
> >> >> >>> > -
> >> >> >>> > Anand
> >> >> >>> >
> >> >> >>> >
> >> >> >>> >
> >> >> >>> > From: Yufei Gu <fl...@gmail.com>
> >> >> >>> > Date: Friday, February 27, 2026 at 9:54 PM
> >> >> >>> > To: dev@polaris.apache.org <de...@polaris.apache.org>
> >> >> >>> > Subject: Re: Polaris Telemetry and Audit Trail
> >> >> >>> >
> >> >> >>> > This Message Is From an External Sender
> >> >> >>> > This message came from outside your organization.
> >> >> >>> > Report Suspicious<
> >> >> >>> >
> >> >> >>>
> >> >>
> >>
> https://us-phishalarm-ewt.proofpoint.com/EWT/v1/Iz9xO38YGHZK!YhNDZABkHi1B699ote2uMwpOZw8i0QMCGO2Szc-HshuABGhGvwPJcymE6G2oUUxtS8xDkSrtGTPm_I3QnVDHoLMk50m9v8z_nZKTkd-bnVUbreF1u0WnfV_X5eYevZl_$
> >> >> >>> > >
> >> >> >>> >
> >> >> >>> >
> >> >> >>> > As I mentioned in
> >> >> >>> >
> >> >> >>>
> >> >>
> >>
> https://urldefense.com/v3/__https://github.com/apache/polaris/issues/3890__;!!Iz9xO38YGHZK!5EuyFFkk3vhRWVIRvQAWBSQfpJkTMA9HxugzDwXmN0LPPqhEFxYkFRGVhtb8AqUwXtDh2OplcMnbMDHKOxrvDU0$
> >> >> >>> ,
> >> >> >>> > supporting
> >> >> >>> > multiple data sources is not a trivial change. I would strongly
> >> >> >>> recommend
> >> >> >>> > starting with a design document to carefully evaluate the
> >> >> architectural
> >> >> >>> > implications and long term impact.
> >> >> >>> >
> >> >> >>> > A REST endpoint to query metrics seems reasonable given the
> >> current
> >> >> >>> JDBC
> >> >> >>> > based persistence model. That said, we may also consider
> >> alternative
> >> >> >>> > storage models. For example, if we later adopt a time series
> >> system
> >> >> >>> such as
> >> >> >>> > Prometheus to store metrics, the query model and access
> patterns
> >> >> would
> >> >> >>> be
> >> >> >>> > fundamentally different. Designing the REST API without
> >> considering
> >> >> >>> these
> >> >> >>> > potential evolutions may limit flexibility. I'd suggest to
> start
> >> >> with
> >> >> >>> the
> >> >> >>> > use case.
> >> >> >>> >
> >> >> >>> > Yufei
> >> >> >>> >
> >> >> >>> >
> >> >> >>> > On Fri, Feb 27, 2026 at 3:42 PM Dmitri Bourlatchkov <
> >> >> dimas@apache.org>
> >> >> >>> > wrote:
> >> >> >>> >
> >> >> >>> > > Hi Anand,
> >> >> >>> > >
> >> >> >>> > > Sharing my view... subject to discussion:
> >> >> >>> > >
> >> >> >>> > > 1. Adding non-IRC REST API to Polaris is perfectly fine.
> >> >> >>> > >
> >> >> >>> > > Figuring out specific endpoint URIs and payloads might
> require
> >> a
> >> >> few
> >> >> >>> > > roundtrips, so opening a separate thread for that might be
> >> best.
> >> >> >>> > > Contributors commonly create Google Docs for new API
> proposals
> >> too
> >> >> >>> (they
> >> >> >>> > > fairly easy to update as the email discussion progresses).
> >> >> >>> > >
> >> >> >>> > > There was a suggestion to try Markdown (with PRs) for
> proposals
> >> >> [1]
> >> >> >>> ...
> >> >> >>> > > feel free to give it a try if you are comfortable with that.
> >> >> >>> > >
> >> >> >>> > > 2. Could you clarify whether you mean end user utilities or
> >> admin
> >> >> >>> user
> >> >> >>> > > utilities? In the latter case those might be more suitable
> for
> >> the
> >> >> >>> Admin
> >> >> >>> > > CLI (java) not the Python CLI, IMHO.
> >> >> >>> > >
> >> >> >>> > > Why would these utilities be common with events? IMHO, event
> >> use
> >> >> >>> cases
> >> >> >>> > are
> >> >> >>> > > distinct from scan/commit metrics.
> >> >> >>> > >
> >> >> >>> > > 3. I'd prefer separating metrics persistence from MetaStore
> >> >> >>> persistence
> >> >> >>> > at
> >> >> >>> > > the code level, so that they could be mixed and matched
> >> >> >>> independently.
> >> >> >>> > The
> >> >> >>> > > separate datasource question will become a non-issue with
> that
> >> >> >>> approach,
> >> >> >>> > I
> >> >> >>> > > guess.
> >> >> >>> > >
> >> >> >>> > > The rationale for separating scan metrics and metastore
> >> >> persistence
> >> >> >>> is
> >> >> >>> > that
> >> >> >>> > > "cascading deletes" between them are hardly ever required.
> >> >> >>> Furthermore,
> >> >> >>> > the
> >> >> >>> > > data and query patterns are very different so different
> >> >> technologies
> >> >> >>> > might
> >> >> >>> > > be beneficial in each case.
> >> >> >>> > >
> >> >> >>> > > [1]
> >> >> >>> >
> >> >> >>>
> >> >>
> >>
> https://urldefense.com/v3/__https://lists.apache.org/thread/yto2wp982t43h1mqjwnslswhws5z47cy__;!!Iz9xO38YGHZK!5EuyFFkk3vhRWVIRvQAWBSQfpJkTMA9HxugzDwXmN0LPPqhEFxYkFRGVhtb8AqUwXtDh2OplcMnbMDHKxYDakNU$
> >> >> >>> > >
> >> >> >>> > > Cheers,
> >> >> >>> > > Dmitri.
> >> >> >>> > >
> >> >> >>> > > On Fri, Feb 27, 2026 at 6:19 PM Anand Kumar Sankaran via dev
> <
> >> >> >>> > > dev@polaris.apache.org> wrote:
> >> >> >>> > >
> >> >> >>> > > > Thanks all. This PR is merged now.
> >> >> >>> > > >
> >> >> >>> > > > Here are the follow-up features / work needed.  These were
> >> all
> >> >> >>> part of
> >> >> >>> > > the
> >> >> >>> > > > merged PR at some point in time and were removed to reduce
> >> >> scope.
> >> >> >>> > > >
> >> >> >>> > > > Please let me know what you think.
> >> >> >>> > > >
> >> >> >>> > > >
> >> >> >>> > > >   1.  A REST API to paginate through table metrics. This
> >> will be
> >> >> >>> > non-IRC
> >> >> >>> > > > standard addition.
> >> >> >>> > > >   2.  Utilities for managing old records, should be common
> >> with
> >> >> >>> events.
> >> >> >>> > > > There was some discussion that it belongs to the CLI.
> >> >> >>> > > >   3.  Separate datasource (metrics, events, even other
> >> tables?).
> >> >> >>> > > >
> >> >> >>> > > >
> >> >> >>> > > > Anything else?
> >> >> >>> > > >
> >> >> >>> > > > -
> >> >> >>> > > > Anand
> >> >> >>> > > >
> >> >> >>> > > >
> >> >> >>> > >
> >> >> >>> >
> >> >> >>> >
> >> >> >>>
> >> >> >>
> >> >>
> >> >
> >>
> >
>

Re: Proposal for REST endpoints for table metrics and events

[Dmitri Bourlatchkov <di...@apache.org>]

Hi All,

WDYT about another sync call next week?

Thanks,
Dmitri.

On Wed, May 6, 2026 at 5:29 PM Dmitri Bourlatchkov <di...@apache.org> wrote:

> Hi EJ,
>
> Thanks for the summary! It covers what we discussed in the meeting very
> well, IMHO.
>
> Looking forward to concrete PRs :)
>
> Cheers,
> Dmitri.
>
> On Wed, May 6, 2026 at 5:08 PM EJ Wang <ej...@gmail.com>
> wrote:
>
>> Hi folks,
>>
>> We had a community sync earlier, thanks JB for scheduling it. Notes from
>> the first metrics architecture sync (May 6, 10-11am PT). Discussion doc
>> with per-section status:
>>
>> https://docs.google.com/document/d/100h7c4damrUzVuquYbBHM0EvA4LSWuW2IT2dN_7nYVA/edit?tab=t.0
>>
>> *The meeting covered both topics from the doc. Direction-level alignment
>> was reached on the headline pieces; details remain for PR review or
>> follow-up sessions.*
>>
>> *Topic 1 — Persistence schema redesign*
>> Idea-level alignment on consolidating per-type tables
>> (scan_metrics_report,
>> commit_metrics_report) into a single metrics_report table. The motivating
>> cost is the surface area added by every new metric type today: new table,
>> SPI method, record class, model, converter, schema migration.
>>
>> Most schema details are deferred to the schema PR. A few specific points
>> came up:
>> •   metric_schema_version: Yufei prefers dropping it, since there is no
>> spec-level concept of metrics versioning today and it is hard to define
>> unilaterally. Robert prefers keeping it, given IRC v2 is coming and the
>> schema should be considered against its likely shape; Robert also raised
>> how to differentiate various payload formats if any. EJ's read is that
>> this
>> is a two-way-door decision. We can start without the field, and if IRC v2
>> changes the shape we would likely roll a corresponding new schema anyway,
>> which is not particularly costly.
>> •   Payload format: Robert pointed out that future formats beyond JSON may
>> be worth supporting. The exact shape is deferred to the schema discussion.
>> •   Partition strategy: Anand suggested monthly partitioning based on his
>> experience as potentially helpful at scale.
>>
>> *Topic 2 — Where metrics ingestion and storage belong*
>> Idea-level alignment that metrics should be a separated SPI from the
>> entity
>> persistence stack. Two reasons surfaced: (a) workloads and capability
>> requirements diverge enough that coupling them creates artificial
>> constraints, and (b) admin experience improves when metrics has its own
>> bootstrap, retention, and lifecycle. Dmitri noted Polaris being a platform
>> should have the flexibility to support different persistence backends per
>> concern, and pointed to a concrete next step of separating the JDBC
>> bootstrap for metrics from the metastore bootstrap. Robert proposed an
>> additional UX extension: detect an unbootstrapped metrics store on first
>> use and auto-bootstrap rather than requiring an explicit manual bootstrap
>> step.
>> The meeting also confirmed that Polaris metrics can start small and stay
>> Iceberg-focused. Naming and persistence schema can lean Iceberg-specific.
>> If a future expansion to generic-table metrics or operational metrics
>> arrives, an abstraction layer can be built on top of the Iceberg metrics
>> reporter at that point. Robert remains on the fence and would prefer
>> something more generic but did not block the direction; Dmitri's read was
>> that the proposed framework already has enough flexibility to absorb
>> future
>> expansion.
>>
>> The Trade-offs and Proposed structure sections in the doc were not
>> reviewed
>> in detail. They remain open for either the next sync or PR review.
>>
>> *Cross-cutting alignment — battery-included plus pluggable*
>> A common philosophy emerged from the discussion. EJ summarized it as:
>> Polaris should provide a battery-included UX for beginners and the
>> flexibility for advanced users to swap the included battery for something
>> more powerful or tailored to their use case. The SPI design needs to
>> enable
>> both.
>>
>> The inputs that shaped this framing:
>> •   Anand described how his team uses the current metrics persistence
>> (three metrics consumers in v1.4).
>> •   Yufei raised Grafana and dashboard integrations as a destination use
>> case beyond the default.
>> •   Robert called out that the current design is more JDBC-focused.
>>
>> Two concrete instances:
>> •   Async metrics intake: Yufei's initial position was that async should
>> largely live on the producer side and there is not much Polaris can do.
>> Robert suggested a Polaris-side default is doable via Vert.x. Dmitri
>> agreed
>> the direction is worth exploring. The meeting converged on a
>> battery-included default (likely Vert.x-backed) with an SPI shape that
>> lets
>> power users route to a more scalable backend (k8s-hosted queue, AWS SQS,
>> etc.).
>> •   Pluggable destinations: combining Yufei's dashboard use case with
>> Robert's JDBC-focused call-out, the meeting agreed the SPI should be
>> structured for multiple sinks so integrations become impl choices rather
>> than architectural changes.
>>
>> The battery-included default is most likely to use the existing
>> JDBC-backed
>> approach.
>>
>> *Direction (idea-level alignment)*
>> •   Single metrics_report table consolidating per-type metrics, replacing
>> scan_metrics_report and commit_metrics_report
>> •   Iceberg-focused naming and schema for now, revisit if generic-table or
>> operational metrics arrive
>> •   Metrics persistence as a separated SPI, not on BasePersistence
>> •   Bootstrap path separated for metrics, independent of metastore
>> bootstrap
>> •   "Battery-included plus pluggable" as the SPI design philosophy
>>
>> *Open items*
>> •   Schema details: metric_schema_version, payload format, IRC v2
>> forward-compat shape
>> •   SPI design details — full review either in the next sync or in the
>> corresponding PR
>> •   Schema refactor PR ownership
>>
>> *Action items*
>> •   EJ to take a first stab at the SPI design and potentially partner with
>> Anand to incorporate the lessons learned from the existing reporter and
>> persistence work.
>> •   Schema refactor PR ownership is not yet decided. If anyone is
>> interested in driving it, reply on this thread.
>> •   JB to schedule the next sync, tentatively in two weeks.
>>
>> -ej
>>
>> On Mon, Apr 27, 2026 at 3:07 PM EJ Wang <ej...@gmail.com>
>> wrote:
>>
>> > Thanks Yufei for the +1.
>> >
>> > JB, could you help add a biweekly metrics architecture sync to the
>> Polaris
>> > community calendar? I'm thinking Thursdays at 9-10am PT, on the
>> off-weeks
>> > from the community meeting (starting May 7), 60 minutes.
>> >
>> > Here's a rough agenda to work through over the first few sessions,
>> grouped
>> > by priority:
>> >
>> > *First: foundational direction*
>> >
>> > 1.  MetricsPersistence: public SPI or internal implementation detail?
>> >    •   Marked @Beta, javadoc calls it a "Service Provider Interface",
>> but
>> > only one consumer (JdbcBasePersistenceImpl), lives on BasePersistence.
>> If
>> > demoted to a private helper inside a persisting reporter impl, most
>> > downstream design decisions become implementation details rather than
>> > contract questions.
>> >
>> > 2.  Persistence schema redesign
>> >    •   Current two-table layout (scan_metrics_report,
>> > commit_metrics_report) with ~25 flattened columns each. Every new metric
>> > type requires a new table, SPI method, record class, model, converter,
>> and
>> > schema migration. Direction to explore: single table with metric_type
>> enum,
>> > schema_version, and JSON payload column.
>> >
>> > *Second: design details once direction is set*
>> >
>> > 3.  Partition key strategy
>> >    •   Single-table design means scan metrics at scale will have high
>> > write concurrency per table. Schema needs to expose enough structure for
>> > backends to shard by entity or time range.
>> >
>> > 4.  Read/write path consistency
>> >    •   Writes go through PolarisMetricsManager on MetaStoreManager.
>> Reads
>> > bypass MetaStoreManager and go straight to BasePersistence, excluding
>> > non-JDBC backends from the read API.
>> >
>> > *Third: cleanup and alignment*
>> >
>> > 5.  PolarisMetricsReporter naming
>> >    •   Only handles IRC (ScanReport/CommitReport), doesn't cover generic
>> > tables or operational metrics. Name is broader than scope.
>> >
>> > 6.  PolarisMetricsManager facade passthrough
>> >    •   Entire default method is
>> callCtx.getMetaStore().writeScanReport().
>> > Zero logic, passes Level 1 straight through to Level 3. Same
>> anti-pattern
>> > as PolarisEventManager.
>> >
>> > 7.  Iceberg community alignment
>> >    •   Payload-type extension needs discussion on dev@iceberg.
>> obelix74's
>> > Feb thread got zero replies. Needs a committer voice.
>> >
>> > Lets confirm prioritization in the first session.
>> >
>> > -ej
>> >
>> > On Tue, Apr 21, 2026 at 3:18 PM Yufei Gu <fl...@gmail.com> wrote:
>> >
>> >> Thanks everyone for continuing to drive this forward. I agree that the
>> >> problem is getting complex enough that a more structured discussion
>> would
>> >> help.
>> >>
>> >> +1 on setting up a biweekly sync for the metrics architecture. I’m
>> happy
>> >> to
>> >> join.
>> >>
>> >> Yufei
>> >>
>> >>
>> >> On Tue, Apr 21, 2026 at 2:34 PM EJ Wang <
>> ej.wang.dev.interest@gmail.com>
>> >> wrote:
>> >>
>> >> > Also, I've been looking more closely at the *persistence schema in
>> the
>> >> > current metrics work*, and I think there's a structural rigidity
>> problem
>> >> > worth raising before the shape gets locked in.
>> >> >
>> >> > Right now we have two separate tables (scan_metrics_report and
>> >> > commit_metrics_report), each with ~25 flattened columns that directly
>> >> > mirror the Iceberg report fields. The SPI follows the same split:
>> >> > writeScanReport and writeCommitReport as separate methods, with
>> per-type
>> >> > record classes, converters, and model objects. *The practical cost:
>> >> > adding a new metric type (operational metrics, for example) requires
>> a
>> >> new
>> >> > table, a new SPI method, a new record class, a new model class, a new
>> >> > converter branch, and a schema migration*. That's a lot of surface
>> area
>> >> > for what should be "one more kind of metric."
>> >> >
>> >> > *My bias* would be toward a single metrics table with *a typed JSON
>> >> > payload*. Something like: metric_type (enum), entity_id,
>> >> > table_identifier, snapshot_id (nullable), received_ts,
>> schema_version,
>> >> and
>> >> > a payload column for the metric-specific data. The metric_type +
>> >> > schema_version pair gives us a forward-compatible contract for the
>> >> payload
>> >> > shape. Adding a new metric type becomes an enum value and a payload
>> >> schema,
>> >> > not a schema migration. One thing I think we need to be deliberate
>> >> about is
>> >> > the partition key design. If all metric types land in one table, scan
>> >> > metrics at scale (high concurrency, high frequency across many
>> tables)
>> >> > could easily create hot partitions. We'd want the persistence layer
>> to
>> >> be
>> >> > able to shard by entity or time range, and that means the logical
>> schema
>> >> > needs to expose enough structure for backends to partition on. I
>> don't
>> >> > think the current flattened layout gives us that.
>> >> >
>> >> > This is getting complex enough that I don't think ad-hoc PR/ML
>> threads
>> >> > will converge well. *Would people be open to a biweekly sync for
>> metrics
>> >> > architecture?* I think 30 minutes every two weeks with interested
>> >> parties
>> >> > would be enough to work through the schema, SPI shape, and read API
>> >> design
>> >> > together. Happy to help set that up.
>> >> >
>> >> > -ej
>> >> >
>> >> > On Mon, Apr 20, 2026 at 2:19 PM EJ Wang <
>> ej.wang.dev.interest@gmail.com
>> >> >
>> >> > wrote:
>> >> >
>> >> >> Reviewed #4115, left a comment on the code organization side.
>> >> >>
>> >> >> One thing stood out: the metrics write path enters through
>> >> >> PolarisMetricsManager on MetaStoreManager, but the new read path
>> >> bypasses
>> >> >> MetaStoreManager entirely and goes straight to BasePersistence via
>> >> >> callContext.getMetaStore(). That means the read API only works for
>> >> backends
>> >> >> that implement BasePersistence. NoSQL and remote backends can't
>> >> participate.
>> >> >>
>> >> >> Stepping back, I think the metrics subsystem is growing into
>> something
>> >> >> real (write + read + REST API + AuthZ + pagination) *but the
>> >> persistence
>> >> >> side is split across two layers in a way that's hard to extend*. I
>> put
>> >> >> together two diagrams to show what I mean (my best effort).
>> >> >>
>> >> >> *Current state* (Diagram 1): three interfaces at three different
>> >> levels.
>> >> >> The engine-facing SPI (PolarisMetricsReporter) is clean. But
>> >> >> PolarisMetricsManager on MetaStoreManager is a passthrough to
>> >> >> MetricsPersistence on BasePersistence. The @Beta annotation and SPI
>> >> javadoc
>> >> >> are on the BasePersistence layer, while the actual extension points
>> >> >> (PolarisMetricsReporter, PolarisMetricsManager) carry no stability
>> >> >> annotation. The write path goes through the MetaStoreManager layer,
>> the
>> >> >> read path doesn't.
>> >> >>
>> >> >> *What I envision* (Diagram 2): two SPIs at two levels.
>> >> >> PolarisMetricsReporter stays as the engine-facing SPI.
>> >> >> PolarisMetricsManager becomes the backend-facing SPI with both write
>> >> and
>> >> >> read methods at the MetaStoreManager level, where any backend (JDBC,
>> >> NoSQL,
>> >> >> remote) can implement them. MetricsPersistence on BasePersistence
>> goes
>> >> >> away. Where metrics actually land is an implementation detail, not a
>> >> core
>> >> >> interface.
>> >> >>
>> >> >> *Minor naming thing*: PolarisMetricsReporter is broader than what it
>> >> >> actually handles. It only accepts Iceberg REST Catalog metrics
>> >> (ScanReport,
>> >> >> CommitReport via MetricsReport). Generic table metrics or
>> operational
>> >> >> metrics aren't in scope. Not blocking, but worth noting if the
>> metrics
>> >> >> surface expands.
>> >> >>
>> >> >> *Rough sketch of how to get there*:
>> >> >>  1.  Add read methods to PolarisMetricsManager (listScanReports,
>> >> >> listCommitReports) with default no-op, same as the existing write
>> >> methods.
>> >> >> (Probably make PolarisMetricsManager more explicit on being Iceberg
>> >> >> specific like package name or class name etc.)
>> >> >>  2.  Wire MetricsReportsService through MetaStoreManager instead of
>> >> >> callContext.getMetaStore().
>> >> >>  3.  Extract metrics persistence from JdbcBasePersistenceImpl into
>> its
>> >> >> own class. That file carries ~7 responsibilities, metrics being one
>> of
>> >> them.
>> >> >>  4.  Remove MetricsPersistence from BasePersistence.
>> >> >>
>> >> >> *None of this needs to happen in #4115. But if the direction makes
>> >> sense,
>> >> >> it would be good to align before the metrics surface grows further.
>> >> Curious
>> >> >> what others think.*
>> >> >>
>> >> >> *My mental model note*: Level 1 MetaStoreManager; level 2
>> transactional
>> >> >> persistence; level 3 base persistence
>> >> >>
>> >> >> Diagram 1
>> >> >> <
>> >>
>> https://www.plantuml.com/plantuml/uml/bLHDR-Cs4BthLmpIYupw0zbkKQ1r3M-S7Bp8xhhM7WCOb3IM65EaGD9EX2RzxHrHb4CxRelwa4YSDu_lpOVcnZ9jzvM8BBS2uGjQpJC3dtHMSekPtMk44IpsMgEqa5XcCOhCZikQQLP1pR8TAp2n3ILhmZDP20m0fcIvUkAoW2qJXd9z1bpToO9BX3WXu0ucy5rpgGPNm0nW5_epUWtm2Ue3pn3kMOFQmKntGZW0BYtgBSi8k5A2QMwybJNMIbFiGSR9QZc4nUqIvikStF0jHprua5C-amge42aNt3R0f5JaaoivdV2Pkqbx4hee4ymOkBh5BTiB-_uIeGeo8zL8rPsPl4DktdEiK1jkB1NdZCRbrSTecDe_mlHbF0wvBmCkaOH5_S8a_TTTKI6-nmCAkEw4LpxsZ-LbYLKQFKMNOgf_wuM7_bV9gOer5SYMMksBSWXFcbi49KNZXNLicwfe3TETC7gPdPqI7uBcHMb1RSzYq34c6PDUM9mn8HRsUTZEiDBve3NjVZumBj0U7SS37mGO7vcwtiK-_pU7U7L_f-digo9YbhSwIfMRwIITKGXbxdIUTCGF1SeCJxloKsU-3k9ddRbX1eDq1q_fx1JbBGT0glVyXimDuP4TQ5qpCAmnGEj2s_6n5mtn1z-97-63itFQZLPO1Ev2tu_WF7Ju-VPc0Skg5bYXxBhkY1xpD7EM_7fyflSpIsqMgVth5xhVr4eQxWQ8enaSAJQSG16yFSDuJ798rrcXr_3n-lfdk7icQjEBmFujL7AodiP_Y4Z7-YxvtZNs4zMgpNTl6tF8sglyPsmqchrjvQ-m-aP94r-TwCA2Ka8upPJZwtvSpoYCXkYMZU2NXvRMBfq9P3i3Le4VAZUAlUZ_oPKsxPgY0Q_BSKLkyr9bhQhQrJjo_x3TPlIB0DPjnMfcIoYP0QaYw1a0fTKDr8fB6ntNuvmoL1ZGkXa69Njh43zf9GiGxHQrA_jDYWRSzF5--WmTVrN97_Sm8LbLUy_lGBmLanJjFkDlGkRqjA_4tm00
>> >> >
>> >> >> :
>> >> >>
>> >> >> [image: image.png]
>> >> >>
>> >> >> Diagram 2
>> >> >> <
>> >>
>> https://www.plantuml.com/plantuml/uml/VLLDR-8m4BtdLupO2sWBLVU8AaGB7AXAbssGzb896SS9RXqxjKqBwkv_tt7iV43fdaZYDpFlpRmnOsE9jhjSH9PRmM31hERKm8scMsuPjJlDe0yheZDc8RR4iYWoBrmMH9CS2a9VICPYUy1OZN0YCy5Q0BCbYNhdCeEK28En8G8wCvbnoQ0R8_05Bc6bkLIz3X03p1zzH7zR-9ZfDquPt9C3qoNCX2yV4G2NbkcKu5jdgGJHt0GbZwnG6i-UP3TUpk5gM6Ldqke350eZUqzoCft3U9xWHvxoa5-7K4nF1J46EbEMafsmdrCBbQ44gVggy18IZrn_ph5asd1ZiIKdQSgueZvjXrQFSFrdC3YN-nXmBacxbGiYyLVxLaBtdhqn0LSzdBDhqQtQoOJeGyad3z0lUqnYgpGB6Ns8oVyta00Dy_WnX0tIOZ8v6SYxHll1TrH6aejAik-mh-AphVFCwSUQqFypElag5QRGFDjQKEd96K1P8QP41c9TzA_IIQyvdAWyv_RSiS3skb0_EzDDkK2v5xWF6MiGFlvhpFLcD2Dq2pml14gaF67eQkmd8gulDoC4kSOu6KVpkvlUJg1RTbWISU40RdBUUS_9XfRZ2dwxm_SW8LYFISgm_MnlDQ6M9P1gbKEc4X-2pH_FvJCkCqm9pbVjD6LrwdLeOrDWfOaqc8Wh9BE85oNKxkNQ6o4yGRy_Eae0G_G8tZv81d3bHDB23WOdisohVr3nh_j6lbSjbNaLRTc8UgtPbAU1J_tygOfZX9DWEJeHDvYx-qmSi5FgNLPZwHrHcUsncGQ5-skhUclpE5fo4ounpFauYrUbkU6ccfnxMvitwag4IyerhTxj8In_Oj1bDO4pQru674loYrGlULHLEGCjwJJ8gDoVZR8MxO4BT3IzRvIcAQKezC6xpziGnTyImrfEGyJI_OcKfgtxIvnTqFEMS17L9Z-jsARN5FmTheP7HtSdtOMT0B4GY2FYHXxgQmMtj2bRqiLFGapiVe1_QVKDrkqXcm83aFEXnMYCZ-xlyHy
>> >> >
>> >> >> :
>> >> >> [image: image.png]
>> >> >>
>> >> >>  -ej
>> >> >>
>> >> >> On Wed, Apr 15, 2026 at 8:22 AM Dmitri Bourlatchkov <
>> dimas@apache.org>
>> >> >> wrote:
>> >> >>
>> >> >>> Hi All,
>> >> >>>
>> >> >>> Heads up: The current state of PR [4115] looks pretty solid to me.
>> I
>> >> >>> believe this PR is approaching a mergeable condition.
>> >> >>>
>> >> >>> Please post your reviews if you have any comments.
>> >> >>>
>> >> >>> [4115] https://github.com/apache/polaris/pull/4115
>> >> >>>
>> >> >>> Thanks,
>> >> >>> Dmitri.
>> >> >>>
>> >> >>> On Tue, Mar 3, 2026 at 3:29 PM Anand Kumar Sankaran via dev <
>> >> >>> dev@polaris.apache.org> wrote:
>> >> >>>
>> >> >>> > Hi Yufei and Dmitri,
>> >> >>> >
>> >> >>> > Here is a proposal for the REST endpoints for metrics and events.
>> >> >>> >
>> >> >>> > https://github.com/apache/polaris/pull/3924/changes
>> >> >>> >
>> >> >>> > I did not see any precursors for raising a PR for proposals, so
>> >> trying
>> >> >>> > this.  Please let me know what you think.
>> >> >>> >
>> >> >>> > -
>> >> >>> > Anand
>> >> >>> >
>> >> >>> > From: Anand Kumar Sankaran <an...@workday.com>
>> >> >>> > Date: Monday, March 2, 2026 at 10:25 AM
>> >> >>> > To: dev@polaris.apache.org <de...@polaris.apache.org>
>> >> >>> > Subject: Re: Polaris Telemetry and Audit Trail
>> >> >>> >
>> >> >>> > About the REST API, based on my use cases:
>> >> >>> >
>> >> >>> >
>> >> >>> >   1.
>> >> >>> > I want to be able to query commit metrics to track files added /
>> >> >>> removed
>> >> >>> > per commit, along with record counts. The ingestion pipeline that
>> >> >>> writes
>> >> >>> > this data is owned by us and we are guaranteed to write this
>> >> >>> information
>> >> >>> > for each write.
>> >> >>> >   2.
>> >> >>> > I want to be able to query scan metrics for read. I understand
>> >> clients
>> >> >>> do
>> >> >>> > not fulfill this requirement.
>> >> >>> >   3.
>> >> >>> > I want to be able to query the events table (events are
>> persisted) -
>> >> >>> this
>> >> >>> > may supersede #2, I am not sure yet.
>> >> >>> >
>> >> >>> > All this information is in the JDBC based persistence model and
>> is
>> >> >>> > persisted in the metastore. I currently don’t have a need to
>> query
>> >> >>> > prometheus or open telemetry. I do publish some events to
>> Prometheus
>> >> >>> and
>> >> >>> > they are forwarded to our dashboards elsewhere.
>> >> >>> >
>> >> >>> > About the CLI utilities, I meant the admin user utilities. In
>> one of
>> >> >>> the
>> >> >>> > earliest drafts of my proposal, Prashant mentioned that the
>> metrics
>> >> >>> tables
>> >> >>> > can grow indefinitely and that a similar problem exists with the
>> >> events
>> >> >>> > table as well. We discussed that cleaning up of old records from
>> >> both
>> >> >>> > metrics tables and events tables can be done via a CLI utility.
>> >> >>> >
>> >> >>> > I see that Yufei has covered the discussion about datasources.
>> >> >>> >
>> >> >>> > -
>> >> >>> > Anand
>> >> >>> >
>> >> >>> >
>> >> >>> >
>> >> >>> > From: Yufei Gu <fl...@gmail.com>
>> >> >>> > Date: Friday, February 27, 2026 at 9:54 PM
>> >> >>> > To: dev@polaris.apache.org <de...@polaris.apache.org>
>> >> >>> > Subject: Re: Polaris Telemetry and Audit Trail
>> >> >>> >
>> >> >>> > This Message Is From an External Sender
>> >> >>> > This message came from outside your organization.
>> >> >>> > Report Suspicious<
>> >> >>> >
>> >> >>>
>> >>
>> https://us-phishalarm-ewt.proofpoint.com/EWT/v1/Iz9xO38YGHZK!YhNDZABkHi1B699ote2uMwpOZw8i0QMCGO2Szc-HshuABGhGvwPJcymE6G2oUUxtS8xDkSrtGTPm_I3QnVDHoLMk50m9v8z_nZKTkd-bnVUbreF1u0WnfV_X5eYevZl_$
>> >> >>> > >
>> >> >>> >
>> >> >>> >
>> >> >>> > As I mentioned in
>> >> >>> >
>> >> >>>
>> >>
>> https://urldefense.com/v3/__https://github.com/apache/polaris/issues/3890__;!!Iz9xO38YGHZK!5EuyFFkk3vhRWVIRvQAWBSQfpJkTMA9HxugzDwXmN0LPPqhEFxYkFRGVhtb8AqUwXtDh2OplcMnbMDHKOxrvDU0$
>> >> >>> ,
>> >> >>> > supporting
>> >> >>> > multiple data sources is not a trivial change. I would strongly
>> >> >>> recommend
>> >> >>> > starting with a design document to carefully evaluate the
>> >> architectural
>> >> >>> > implications and long term impact.
>> >> >>> >
>> >> >>> > A REST endpoint to query metrics seems reasonable given the
>> current
>> >> >>> JDBC
>> >> >>> > based persistence model. That said, we may also consider
>> alternative
>> >> >>> > storage models. For example, if we later adopt a time series
>> system
>> >> >>> such as
>> >> >>> > Prometheus to store metrics, the query model and access patterns
>> >> would
>> >> >>> be
>> >> >>> > fundamentally different. Designing the REST API without
>> considering
>> >> >>> these
>> >> >>> > potential evolutions may limit flexibility. I'd suggest to start
>> >> with
>> >> >>> the
>> >> >>> > use case.
>> >> >>> >
>> >> >>> > Yufei
>> >> >>> >
>> >> >>> >
>> >> >>> > On Fri, Feb 27, 2026 at 3:42 PM Dmitri Bourlatchkov <
>> >> dimas@apache.org>
>> >> >>> > wrote:
>> >> >>> >
>> >> >>> > > Hi Anand,
>> >> >>> > >
>> >> >>> > > Sharing my view... subject to discussion:
>> >> >>> > >
>> >> >>> > > 1. Adding non-IRC REST API to Polaris is perfectly fine.
>> >> >>> > >
>> >> >>> > > Figuring out specific endpoint URIs and payloads might require
>> a
>> >> few
>> >> >>> > > roundtrips, so opening a separate thread for that might be
>> best.
>> >> >>> > > Contributors commonly create Google Docs for new API proposals
>> too
>> >> >>> (they
>> >> >>> > > fairly easy to update as the email discussion progresses).
>> >> >>> > >
>> >> >>> > > There was a suggestion to try Markdown (with PRs) for proposals
>> >> [1]
>> >> >>> ...
>> >> >>> > > feel free to give it a try if you are comfortable with that.
>> >> >>> > >
>> >> >>> > > 2. Could you clarify whether you mean end user utilities or
>> admin
>> >> >>> user
>> >> >>> > > utilities? In the latter case those might be more suitable for
>> the
>> >> >>> Admin
>> >> >>> > > CLI (java) not the Python CLI, IMHO.
>> >> >>> > >
>> >> >>> > > Why would these utilities be common with events? IMHO, event
>> use
>> >> >>> cases
>> >> >>> > are
>> >> >>> > > distinct from scan/commit metrics.
>> >> >>> > >
>> >> >>> > > 3. I'd prefer separating metrics persistence from MetaStore
>> >> >>> persistence
>> >> >>> > at
>> >> >>> > > the code level, so that they could be mixed and matched
>> >> >>> independently.
>> >> >>> > The
>> >> >>> > > separate datasource question will become a non-issue with that
>> >> >>> approach,
>> >> >>> > I
>> >> >>> > > guess.
>> >> >>> > >
>> >> >>> > > The rationale for separating scan metrics and metastore
>> >> persistence
>> >> >>> is
>> >> >>> > that
>> >> >>> > > "cascading deletes" between them are hardly ever required.
>> >> >>> Furthermore,
>> >> >>> > the
>> >> >>> > > data and query patterns are very different so different
>> >> technologies
>> >> >>> > might
>> >> >>> > > be beneficial in each case.
>> >> >>> > >
>> >> >>> > > [1]
>> >> >>> >
>> >> >>>
>> >>
>> https://urldefense.com/v3/__https://lists.apache.org/thread/yto2wp982t43h1mqjwnslswhws5z47cy__;!!Iz9xO38YGHZK!5EuyFFkk3vhRWVIRvQAWBSQfpJkTMA9HxugzDwXmN0LPPqhEFxYkFRGVhtb8AqUwXtDh2OplcMnbMDHKxYDakNU$
>> >> >>> > >
>> >> >>> > > Cheers,
>> >> >>> > > Dmitri.
>> >> >>> > >
>> >> >>> > > On Fri, Feb 27, 2026 at 6:19 PM Anand Kumar Sankaran via dev <
>> >> >>> > > dev@polaris.apache.org> wrote:
>> >> >>> > >
>> >> >>> > > > Thanks all. This PR is merged now.
>> >> >>> > > >
>> >> >>> > > > Here are the follow-up features / work needed.  These were
>> all
>> >> >>> part of
>> >> >>> > > the
>> >> >>> > > > merged PR at some point in time and were removed to reduce
>> >> scope.
>> >> >>> > > >
>> >> >>> > > > Please let me know what you think.
>> >> >>> > > >
>> >> >>> > > >
>> >> >>> > > >   1.  A REST API to paginate through table metrics. This
>> will be
>> >> >>> > non-IRC
>> >> >>> > > > standard addition.
>> >> >>> > > >   2.  Utilities for managing old records, should be common
>> with
>> >> >>> events.
>> >> >>> > > > There was some discussion that it belongs to the CLI.
>> >> >>> > > >   3.  Separate datasource (metrics, events, even other
>> tables?).
>> >> >>> > > >
>> >> >>> > > >
>> >> >>> > > > Anything else?
>> >> >>> > > >
>> >> >>> > > > -
>> >> >>> > > > Anand
>> >> >>> > > >
>> >> >>> > > >
>> >> >>> > >
>> >> >>> >
>> >> >>> >
>> >> >>>
>> >> >>
>> >>
>> >
>>
>

Re: Proposal for REST endpoints for table metrics and events

[Dmitri Bourlatchkov <di...@apache.org>]

Hi EJ,

Thanks for the summary! It covers what we discussed in the meeting very
well, IMHO.

Looking forward to concrete PRs :)

Cheers,
Dmitri.

On Wed, May 6, 2026 at 5:08 PM EJ Wang <ej...@gmail.com>
wrote:

> Hi folks,
>
> We had a community sync earlier, thanks JB for scheduling it. Notes from
> the first metrics architecture sync (May 6, 10-11am PT). Discussion doc
> with per-section status:
>
> https://docs.google.com/document/d/100h7c4damrUzVuquYbBHM0EvA4LSWuW2IT2dN_7nYVA/edit?tab=t.0
>
> *The meeting covered both topics from the doc. Direction-level alignment
> was reached on the headline pieces; details remain for PR review or
> follow-up sessions.*
>
> *Topic 1 — Persistence schema redesign*
> Idea-level alignment on consolidating per-type tables (scan_metrics_report,
> commit_metrics_report) into a single metrics_report table. The motivating
> cost is the surface area added by every new metric type today: new table,
> SPI method, record class, model, converter, schema migration.
>
> Most schema details are deferred to the schema PR. A few specific points
> came up:
> •   metric_schema_version: Yufei prefers dropping it, since there is no
> spec-level concept of metrics versioning today and it is hard to define
> unilaterally. Robert prefers keeping it, given IRC v2 is coming and the
> schema should be considered against its likely shape; Robert also raised
> how to differentiate various payload formats if any. EJ's read is that this
> is a two-way-door decision. We can start without the field, and if IRC v2
> changes the shape we would likely roll a corresponding new schema anyway,
> which is not particularly costly.
> •   Payload format: Robert pointed out that future formats beyond JSON may
> be worth supporting. The exact shape is deferred to the schema discussion.
> •   Partition strategy: Anand suggested monthly partitioning based on his
> experience as potentially helpful at scale.
>
> *Topic 2 — Where metrics ingestion and storage belong*
> Idea-level alignment that metrics should be a separated SPI from the entity
> persistence stack. Two reasons surfaced: (a) workloads and capability
> requirements diverge enough that coupling them creates artificial
> constraints, and (b) admin experience improves when metrics has its own
> bootstrap, retention, and lifecycle. Dmitri noted Polaris being a platform
> should have the flexibility to support different persistence backends per
> concern, and pointed to a concrete next step of separating the JDBC
> bootstrap for metrics from the metastore bootstrap. Robert proposed an
> additional UX extension: detect an unbootstrapped metrics store on first
> use and auto-bootstrap rather than requiring an explicit manual bootstrap
> step.
> The meeting also confirmed that Polaris metrics can start small and stay
> Iceberg-focused. Naming and persistence schema can lean Iceberg-specific.
> If a future expansion to generic-table metrics or operational metrics
> arrives, an abstraction layer can be built on top of the Iceberg metrics
> reporter at that point. Robert remains on the fence and would prefer
> something more generic but did not block the direction; Dmitri's read was
> that the proposed framework already has enough flexibility to absorb future
> expansion.
>
> The Trade-offs and Proposed structure sections in the doc were not reviewed
> in detail. They remain open for either the next sync or PR review.
>
> *Cross-cutting alignment — battery-included plus pluggable*
> A common philosophy emerged from the discussion. EJ summarized it as:
> Polaris should provide a battery-included UX for beginners and the
> flexibility for advanced users to swap the included battery for something
> more powerful or tailored to their use case. The SPI design needs to enable
> both.
>
> The inputs that shaped this framing:
> •   Anand described how his team uses the current metrics persistence
> (three metrics consumers in v1.4).
> •   Yufei raised Grafana and dashboard integrations as a destination use
> case beyond the default.
> •   Robert called out that the current design is more JDBC-focused.
>
> Two concrete instances:
> •   Async metrics intake: Yufei's initial position was that async should
> largely live on the producer side and there is not much Polaris can do.
> Robert suggested a Polaris-side default is doable via Vert.x. Dmitri agreed
> the direction is worth exploring. The meeting converged on a
> battery-included default (likely Vert.x-backed) with an SPI shape that lets
> power users route to a more scalable backend (k8s-hosted queue, AWS SQS,
> etc.).
> •   Pluggable destinations: combining Yufei's dashboard use case with
> Robert's JDBC-focused call-out, the meeting agreed the SPI should be
> structured for multiple sinks so integrations become impl choices rather
> than architectural changes.
>
> The battery-included default is most likely to use the existing JDBC-backed
> approach.
>
> *Direction (idea-level alignment)*
> •   Single metrics_report table consolidating per-type metrics, replacing
> scan_metrics_report and commit_metrics_report
> •   Iceberg-focused naming and schema for now, revisit if generic-table or
> operational metrics arrive
> •   Metrics persistence as a separated SPI, not on BasePersistence
> •   Bootstrap path separated for metrics, independent of metastore
> bootstrap
> •   "Battery-included plus pluggable" as the SPI design philosophy
>
> *Open items*
> •   Schema details: metric_schema_version, payload format, IRC v2
> forward-compat shape
> •   SPI design details — full review either in the next sync or in the
> corresponding PR
> •   Schema refactor PR ownership
>
> *Action items*
> •   EJ to take a first stab at the SPI design and potentially partner with
> Anand to incorporate the lessons learned from the existing reporter and
> persistence work.
> •   Schema refactor PR ownership is not yet decided. If anyone is
> interested in driving it, reply on this thread.
> •   JB to schedule the next sync, tentatively in two weeks.
>
> -ej
>
> On Mon, Apr 27, 2026 at 3:07 PM EJ Wang <ej...@gmail.com>
> wrote:
>
> > Thanks Yufei for the +1.
> >
> > JB, could you help add a biweekly metrics architecture sync to the
> Polaris
> > community calendar? I'm thinking Thursdays at 9-10am PT, on the off-weeks
> > from the community meeting (starting May 7), 60 minutes.
> >
> > Here's a rough agenda to work through over the first few sessions,
> grouped
> > by priority:
> >
> > *First: foundational direction*
> >
> > 1.  MetricsPersistence: public SPI or internal implementation detail?
> >    •   Marked @Beta, javadoc calls it a "Service Provider Interface", but
> > only one consumer (JdbcBasePersistenceImpl), lives on BasePersistence. If
> > demoted to a private helper inside a persisting reporter impl, most
> > downstream design decisions become implementation details rather than
> > contract questions.
> >
> > 2.  Persistence schema redesign
> >    •   Current two-table layout (scan_metrics_report,
> > commit_metrics_report) with ~25 flattened columns each. Every new metric
> > type requires a new table, SPI method, record class, model, converter,
> and
> > schema migration. Direction to explore: single table with metric_type
> enum,
> > schema_version, and JSON payload column.
> >
> > *Second: design details once direction is set*
> >
> > 3.  Partition key strategy
> >    •   Single-table design means scan metrics at scale will have high
> > write concurrency per table. Schema needs to expose enough structure for
> > backends to shard by entity or time range.
> >
> > 4.  Read/write path consistency
> >    •   Writes go through PolarisMetricsManager on MetaStoreManager. Reads
> > bypass MetaStoreManager and go straight to BasePersistence, excluding
> > non-JDBC backends from the read API.
> >
> > *Third: cleanup and alignment*
> >
> > 5.  PolarisMetricsReporter naming
> >    •   Only handles IRC (ScanReport/CommitReport), doesn't cover generic
> > tables or operational metrics. Name is broader than scope.
> >
> > 6.  PolarisMetricsManager facade passthrough
> >    •   Entire default method is callCtx.getMetaStore().writeScanReport().
> > Zero logic, passes Level 1 straight through to Level 3. Same anti-pattern
> > as PolarisEventManager.
> >
> > 7.  Iceberg community alignment
> >    •   Payload-type extension needs discussion on dev@iceberg.
> obelix74's
> > Feb thread got zero replies. Needs a committer voice.
> >
> > Lets confirm prioritization in the first session.
> >
> > -ej
> >
> > On Tue, Apr 21, 2026 at 3:18 PM Yufei Gu <fl...@gmail.com> wrote:
> >
> >> Thanks everyone for continuing to drive this forward. I agree that the
> >> problem is getting complex enough that a more structured discussion
> would
> >> help.
> >>
> >> +1 on setting up a biweekly sync for the metrics architecture. I’m happy
> >> to
> >> join.
> >>
> >> Yufei
> >>
> >>
> >> On Tue, Apr 21, 2026 at 2:34 PM EJ Wang <ej.wang.dev.interest@gmail.com
> >
> >> wrote:
> >>
> >> > Also, I've been looking more closely at the *persistence schema in the
> >> > current metrics work*, and I think there's a structural rigidity
> problem
> >> > worth raising before the shape gets locked in.
> >> >
> >> > Right now we have two separate tables (scan_metrics_report and
> >> > commit_metrics_report), each with ~25 flattened columns that directly
> >> > mirror the Iceberg report fields. The SPI follows the same split:
> >> > writeScanReport and writeCommitReport as separate methods, with
> per-type
> >> > record classes, converters, and model objects. *The practical cost:
> >> > adding a new metric type (operational metrics, for example) requires a
> >> new
> >> > table, a new SPI method, a new record class, a new model class, a new
> >> > converter branch, and a schema migration*. That's a lot of surface
> area
> >> > for what should be "one more kind of metric."
> >> >
> >> > *My bias* would be toward a single metrics table with *a typed JSON
> >> > payload*. Something like: metric_type (enum), entity_id,
> >> > table_identifier, snapshot_id (nullable), received_ts, schema_version,
> >> and
> >> > a payload column for the metric-specific data. The metric_type +
> >> > schema_version pair gives us a forward-compatible contract for the
> >> payload
> >> > shape. Adding a new metric type becomes an enum value and a payload
> >> schema,
> >> > not a schema migration. One thing I think we need to be deliberate
> >> about is
> >> > the partition key design. If all metric types land in one table, scan
> >> > metrics at scale (high concurrency, high frequency across many tables)
> >> > could easily create hot partitions. We'd want the persistence layer to
> >> be
> >> > able to shard by entity or time range, and that means the logical
> schema
> >> > needs to expose enough structure for backends to partition on. I don't
> >> > think the current flattened layout gives us that.
> >> >
> >> > This is getting complex enough that I don't think ad-hoc PR/ML threads
> >> > will converge well. *Would people be open to a biweekly sync for
> metrics
> >> > architecture?* I think 30 minutes every two weeks with interested
> >> parties
> >> > would be enough to work through the schema, SPI shape, and read API
> >> design
> >> > together. Happy to help set that up.
> >> >
> >> > -ej
> >> >
> >> > On Mon, Apr 20, 2026 at 2:19 PM EJ Wang <
> ej.wang.dev.interest@gmail.com
> >> >
> >> > wrote:
> >> >
> >> >> Reviewed #4115, left a comment on the code organization side.
> >> >>
> >> >> One thing stood out: the metrics write path enters through
> >> >> PolarisMetricsManager on MetaStoreManager, but the new read path
> >> bypasses
> >> >> MetaStoreManager entirely and goes straight to BasePersistence via
> >> >> callContext.getMetaStore(). That means the read API only works for
> >> backends
> >> >> that implement BasePersistence. NoSQL and remote backends can't
> >> participate.
> >> >>
> >> >> Stepping back, I think the metrics subsystem is growing into
> something
> >> >> real (write + read + REST API + AuthZ + pagination) *but the
> >> persistence
> >> >> side is split across two layers in a way that's hard to extend*. I
> put
> >> >> together two diagrams to show what I mean (my best effort).
> >> >>
> >> >> *Current state* (Diagram 1): three interfaces at three different
> >> levels.
> >> >> The engine-facing SPI (PolarisMetricsReporter) is clean. But
> >> >> PolarisMetricsManager on MetaStoreManager is a passthrough to
> >> >> MetricsPersistence on BasePersistence. The @Beta annotation and SPI
> >> javadoc
> >> >> are on the BasePersistence layer, while the actual extension points
> >> >> (PolarisMetricsReporter, PolarisMetricsManager) carry no stability
> >> >> annotation. The write path goes through the MetaStoreManager layer,
> the
> >> >> read path doesn't.
> >> >>
> >> >> *What I envision* (Diagram 2): two SPIs at two levels.
> >> >> PolarisMetricsReporter stays as the engine-facing SPI.
> >> >> PolarisMetricsManager becomes the backend-facing SPI with both write
> >> and
> >> >> read methods at the MetaStoreManager level, where any backend (JDBC,
> >> NoSQL,
> >> >> remote) can implement them. MetricsPersistence on BasePersistence
> goes
> >> >> away. Where metrics actually land is an implementation detail, not a
> >> core
> >> >> interface.
> >> >>
> >> >> *Minor naming thing*: PolarisMetricsReporter is broader than what it
> >> >> actually handles. It only accepts Iceberg REST Catalog metrics
> >> (ScanReport,
> >> >> CommitReport via MetricsReport). Generic table metrics or operational
> >> >> metrics aren't in scope. Not blocking, but worth noting if the
> metrics
> >> >> surface expands.
> >> >>
> >> >> *Rough sketch of how to get there*:
> >> >>  1.  Add read methods to PolarisMetricsManager (listScanReports,
> >> >> listCommitReports) with default no-op, same as the existing write
> >> methods.
> >> >> (Probably make PolarisMetricsManager more explicit on being Iceberg
> >> >> specific like package name or class name etc.)
> >> >>  2.  Wire MetricsReportsService through MetaStoreManager instead of
> >> >> callContext.getMetaStore().
> >> >>  3.  Extract metrics persistence from JdbcBasePersistenceImpl into
> its
> >> >> own class. That file carries ~7 responsibilities, metrics being one
> of
> >> them.
> >> >>  4.  Remove MetricsPersistence from BasePersistence.
> >> >>
> >> >> *None of this needs to happen in #4115. But if the direction makes
> >> sense,
> >> >> it would be good to align before the metrics surface grows further.
> >> Curious
> >> >> what others think.*
> >> >>
> >> >> *My mental model note*: Level 1 MetaStoreManager; level 2
> transactional
> >> >> persistence; level 3 base persistence
> >> >>
> >> >> Diagram 1
> >> >> <
> >>
> https://www.plantuml.com/plantuml/uml/bLHDR-Cs4BthLmpIYupw0zbkKQ1r3M-S7Bp8xhhM7WCOb3IM65EaGD9EX2RzxHrHb4CxRelwa4YSDu_lpOVcnZ9jzvM8BBS2uGjQpJC3dtHMSekPtMk44IpsMgEqa5XcCOhCZikQQLP1pR8TAp2n3ILhmZDP20m0fcIvUkAoW2qJXd9z1bpToO9BX3WXu0ucy5rpgGPNm0nW5_epUWtm2Ue3pn3kMOFQmKntGZW0BYtgBSi8k5A2QMwybJNMIbFiGSR9QZc4nUqIvikStF0jHprua5C-amge42aNt3R0f5JaaoivdV2Pkqbx4hee4ymOkBh5BTiB-_uIeGeo8zL8rPsPl4DktdEiK1jkB1NdZCRbrSTecDe_mlHbF0wvBmCkaOH5_S8a_TTTKI6-nmCAkEw4LpxsZ-LbYLKQFKMNOgf_wuM7_bV9gOer5SYMMksBSWXFcbi49KNZXNLicwfe3TETC7gPdPqI7uBcHMb1RSzYq34c6PDUM9mn8HRsUTZEiDBve3NjVZumBj0U7SS37mGO7vcwtiK-_pU7U7L_f-digo9YbhSwIfMRwIITKGXbxdIUTCGF1SeCJxloKsU-3k9ddRbX1eDq1q_fx1JbBGT0glVyXimDuP4TQ5qpCAmnGEj2s_6n5mtn1z-97-63itFQZLPO1Ev2tu_WF7Ju-VPc0Skg5bYXxBhkY1xpD7EM_7fyflSpIsqMgVth5xhVr4eQxWQ8enaSAJQSG16yFSDuJ798rrcXr_3n-lfdk7icQjEBmFujL7AodiP_Y4Z7-YxvtZNs4zMgpNTl6tF8sglyPsmqchrjvQ-m-aP94r-TwCA2Ka8upPJZwtvSpoYCXkYMZU2NXvRMBfq9P3i3Le4VAZUAlUZ_oPKsxPgY0Q_BSKLkyr9bhQhQrJjo_x3TPlIB0DPjnMfcIoYP0QaYw1a0fTKDr8fB6ntNuvmoL1ZGkXa69Njh43zf9GiGxHQrA_jDYWRSzF5--WmTVrN97_Sm8LbLUy_lGBmLanJjFkDlGkRqjA_4tm00
> >> >
> >> >> :
> >> >>
> >> >> [image: image.png]
> >> >>
> >> >> Diagram 2
> >> >> <
> >>
> https://www.plantuml.com/plantuml/uml/VLLDR-8m4BtdLupO2sWBLVU8AaGB7AXAbssGzb896SS9RXqxjKqBwkv_tt7iV43fdaZYDpFlpRmnOsE9jhjSH9PRmM31hERKm8scMsuPjJlDe0yheZDc8RR4iYWoBrmMH9CS2a9VICPYUy1OZN0YCy5Q0BCbYNhdCeEK28En8G8wCvbnoQ0R8_05Bc6bkLIz3X03p1zzH7zR-9ZfDquPt9C3qoNCX2yV4G2NbkcKu5jdgGJHt0GbZwnG6i-UP3TUpk5gM6Ldqke350eZUqzoCft3U9xWHvxoa5-7K4nF1J46EbEMafsmdrCBbQ44gVggy18IZrn_ph5asd1ZiIKdQSgueZvjXrQFSFrdC3YN-nXmBacxbGiYyLVxLaBtdhqn0LSzdBDhqQtQoOJeGyad3z0lUqnYgpGB6Ns8oVyta00Dy_WnX0tIOZ8v6SYxHll1TrH6aejAik-mh-AphVFCwSUQqFypElag5QRGFDjQKEd96K1P8QP41c9TzA_IIQyvdAWyv_RSiS3skb0_EzDDkK2v5xWF6MiGFlvhpFLcD2Dq2pml14gaF67eQkmd8gulDoC4kSOu6KVpkvlUJg1RTbWISU40RdBUUS_9XfRZ2dwxm_SW8LYFISgm_MnlDQ6M9P1gbKEc4X-2pH_FvJCkCqm9pbVjD6LrwdLeOrDWfOaqc8Wh9BE85oNKxkNQ6o4yGRy_Eae0G_G8tZv81d3bHDB23WOdisohVr3nh_j6lbSjbNaLRTc8UgtPbAU1J_tygOfZX9DWEJeHDvYx-qmSi5FgNLPZwHrHcUsncGQ5-skhUclpE5fo4ounpFauYrUbkU6ccfnxMvitwag4IyerhTxj8In_Oj1bDO4pQru674loYrGlULHLEGCjwJJ8gDoVZR8MxO4BT3IzRvIcAQKezC6xpziGnTyImrfEGyJI_OcKfgtxIvnTqFEMS17L9Z-jsARN5FmTheP7HtSdtOMT0B4GY2FYHXxgQmMtj2bRqiLFGapiVe1_QVKDrkqXcm83aFEXnMYCZ-xlyHy
> >> >
> >> >> :
> >> >> [image: image.png]
> >> >>
> >> >>  -ej
> >> >>
> >> >> On Wed, Apr 15, 2026 at 8:22 AM Dmitri Bourlatchkov <
> dimas@apache.org>
> >> >> wrote:
> >> >>
> >> >>> Hi All,
> >> >>>
> >> >>> Heads up: The current state of PR [4115] looks pretty solid to me. I
> >> >>> believe this PR is approaching a mergeable condition.
> >> >>>
> >> >>> Please post your reviews if you have any comments.
> >> >>>
> >> >>> [4115] https://github.com/apache/polaris/pull/4115
> >> >>>
> >> >>> Thanks,
> >> >>> Dmitri.
> >> >>>
> >> >>> On Tue, Mar 3, 2026 at 3:29 PM Anand Kumar Sankaran via dev <
> >> >>> dev@polaris.apache.org> wrote:
> >> >>>
> >> >>> > Hi Yufei and Dmitri,
> >> >>> >
> >> >>> > Here is a proposal for the REST endpoints for metrics and events.
> >> >>> >
> >> >>> > https://github.com/apache/polaris/pull/3924/changes
> >> >>> >
> >> >>> > I did not see any precursors for raising a PR for proposals, so
> >> trying
> >> >>> > this.  Please let me know what you think.
> >> >>> >
> >> >>> > -
> >> >>> > Anand
> >> >>> >
> >> >>> > From: Anand Kumar Sankaran <an...@workday.com>
> >> >>> > Date: Monday, March 2, 2026 at 10:25 AM
> >> >>> > To: dev@polaris.apache.org <de...@polaris.apache.org>
> >> >>> > Subject: Re: Polaris Telemetry and Audit Trail
> >> >>> >
> >> >>> > About the REST API, based on my use cases:
> >> >>> >
> >> >>> >
> >> >>> >   1.
> >> >>> > I want to be able to query commit metrics to track files added /
> >> >>> removed
> >> >>> > per commit, along with record counts. The ingestion pipeline that
> >> >>> writes
> >> >>> > this data is owned by us and we are guaranteed to write this
> >> >>> information
> >> >>> > for each write.
> >> >>> >   2.
> >> >>> > I want to be able to query scan metrics for read. I understand
> >> clients
> >> >>> do
> >> >>> > not fulfill this requirement.
> >> >>> >   3.
> >> >>> > I want to be able to query the events table (events are
> persisted) -
> >> >>> this
> >> >>> > may supersede #2, I am not sure yet.
> >> >>> >
> >> >>> > All this information is in the JDBC based persistence model and is
> >> >>> > persisted in the metastore. I currently don’t have a need to query
> >> >>> > prometheus or open telemetry. I do publish some events to
> Prometheus
> >> >>> and
> >> >>> > they are forwarded to our dashboards elsewhere.
> >> >>> >
> >> >>> > About the CLI utilities, I meant the admin user utilities. In one
> of
> >> >>> the
> >> >>> > earliest drafts of my proposal, Prashant mentioned that the
> metrics
> >> >>> tables
> >> >>> > can grow indefinitely and that a similar problem exists with the
> >> events
> >> >>> > table as well. We discussed that cleaning up of old records from
> >> both
> >> >>> > metrics tables and events tables can be done via a CLI utility.
> >> >>> >
> >> >>> > I see that Yufei has covered the discussion about datasources.
> >> >>> >
> >> >>> > -
> >> >>> > Anand
> >> >>> >
> >> >>> >
> >> >>> >
> >> >>> > From: Yufei Gu <fl...@gmail.com>
> >> >>> > Date: Friday, February 27, 2026 at 9:54 PM
> >> >>> > To: dev@polaris.apache.org <de...@polaris.apache.org>
> >> >>> > Subject: Re: Polaris Telemetry and Audit Trail
> >> >>> >
> >> >>> > This Message Is From an External Sender
> >> >>> > This message came from outside your organization.
> >> >>> > Report Suspicious<
> >> >>> >
> >> >>>
> >>
> https://us-phishalarm-ewt.proofpoint.com/EWT/v1/Iz9xO38YGHZK!YhNDZABkHi1B699ote2uMwpOZw8i0QMCGO2Szc-HshuABGhGvwPJcymE6G2oUUxtS8xDkSrtGTPm_I3QnVDHoLMk50m9v8z_nZKTkd-bnVUbreF1u0WnfV_X5eYevZl_$
> >> >>> > >
> >> >>> >
> >> >>> >
> >> >>> > As I mentioned in
> >> >>> >
> >> >>>
> >>
> https://urldefense.com/v3/__https://github.com/apache/polaris/issues/3890__;!!Iz9xO38YGHZK!5EuyFFkk3vhRWVIRvQAWBSQfpJkTMA9HxugzDwXmN0LPPqhEFxYkFRGVhtb8AqUwXtDh2OplcMnbMDHKOxrvDU0$
> >> >>> ,
> >> >>> > supporting
> >> >>> > multiple data sources is not a trivial change. I would strongly
> >> >>> recommend
> >> >>> > starting with a design document to carefully evaluate the
> >> architectural
> >> >>> > implications and long term impact.
> >> >>> >
> >> >>> > A REST endpoint to query metrics seems reasonable given the
> current
> >> >>> JDBC
> >> >>> > based persistence model. That said, we may also consider
> alternative
> >> >>> > storage models. For example, if we later adopt a time series
> system
> >> >>> such as
> >> >>> > Prometheus to store metrics, the query model and access patterns
> >> would
> >> >>> be
> >> >>> > fundamentally different. Designing the REST API without
> considering
> >> >>> these
> >> >>> > potential evolutions may limit flexibility. I'd suggest to start
> >> with
> >> >>> the
> >> >>> > use case.
> >> >>> >
> >> >>> > Yufei
> >> >>> >
> >> >>> >
> >> >>> > On Fri, Feb 27, 2026 at 3:42 PM Dmitri Bourlatchkov <
> >> dimas@apache.org>
> >> >>> > wrote:
> >> >>> >
> >> >>> > > Hi Anand,
> >> >>> > >
> >> >>> > > Sharing my view... subject to discussion:
> >> >>> > >
> >> >>> > > 1. Adding non-IRC REST API to Polaris is perfectly fine.
> >> >>> > >
> >> >>> > > Figuring out specific endpoint URIs and payloads might require a
> >> few
> >> >>> > > roundtrips, so opening a separate thread for that might be best.
> >> >>> > > Contributors commonly create Google Docs for new API proposals
> too
> >> >>> (they
> >> >>> > > fairly easy to update as the email discussion progresses).
> >> >>> > >
> >> >>> > > There was a suggestion to try Markdown (with PRs) for proposals
> >> [1]
> >> >>> ...
> >> >>> > > feel free to give it a try if you are comfortable with that.
> >> >>> > >
> >> >>> > > 2. Could you clarify whether you mean end user utilities or
> admin
> >> >>> user
> >> >>> > > utilities? In the latter case those might be more suitable for
> the
> >> >>> Admin
> >> >>> > > CLI (java) not the Python CLI, IMHO.
> >> >>> > >
> >> >>> > > Why would these utilities be common with events? IMHO, event use
> >> >>> cases
> >> >>> > are
> >> >>> > > distinct from scan/commit metrics.
> >> >>> > >
> >> >>> > > 3. I'd prefer separating metrics persistence from MetaStore
> >> >>> persistence
> >> >>> > at
> >> >>> > > the code level, so that they could be mixed and matched
> >> >>> independently.
> >> >>> > The
> >> >>> > > separate datasource question will become a non-issue with that
> >> >>> approach,
> >> >>> > I
> >> >>> > > guess.
> >> >>> > >
> >> >>> > > The rationale for separating scan metrics and metastore
> >> persistence
> >> >>> is
> >> >>> > that
> >> >>> > > "cascading deletes" between them are hardly ever required.
> >> >>> Furthermore,
> >> >>> > the
> >> >>> > > data and query patterns are very different so different
> >> technologies
> >> >>> > might
> >> >>> > > be beneficial in each case.
> >> >>> > >
> >> >>> > > [1]
> >> >>> >
> >> >>>
> >>
> https://urldefense.com/v3/__https://lists.apache.org/thread/yto2wp982t43h1mqjwnslswhws5z47cy__;!!Iz9xO38YGHZK!5EuyFFkk3vhRWVIRvQAWBSQfpJkTMA9HxugzDwXmN0LPPqhEFxYkFRGVhtb8AqUwXtDh2OplcMnbMDHKxYDakNU$
> >> >>> > >
> >> >>> > > Cheers,
> >> >>> > > Dmitri.
> >> >>> > >
> >> >>> > > On Fri, Feb 27, 2026 at 6:19 PM Anand Kumar Sankaran via dev <
> >> >>> > > dev@polaris.apache.org> wrote:
> >> >>> > >
> >> >>> > > > Thanks all. This PR is merged now.
> >> >>> > > >
> >> >>> > > > Here are the follow-up features / work needed.  These were all
> >> >>> part of
> >> >>> > > the
> >> >>> > > > merged PR at some point in time and were removed to reduce
> >> scope.
> >> >>> > > >
> >> >>> > > > Please let me know what you think.
> >> >>> > > >
> >> >>> > > >
> >> >>> > > >   1.  A REST API to paginate through table metrics. This will
> be
> >> >>> > non-IRC
> >> >>> > > > standard addition.
> >> >>> > > >   2.  Utilities for managing old records, should be common
> with
> >> >>> events.
> >> >>> > > > There was some discussion that it belongs to the CLI.
> >> >>> > > >   3.  Separate datasource (metrics, events, even other
> tables?).
> >> >>> > > >
> >> >>> > > >
> >> >>> > > > Anything else?
> >> >>> > > >
> >> >>> > > > -
> >> >>> > > > Anand
> >> >>> > > >
> >> >>> > > >
> >> >>> > >
> >> >>> >
> >> >>> >
> >> >>>
> >> >>
> >>
> >
>

Re: Proposal for REST endpoints for table metrics and events

[Dmitri Bourlatchkov <di...@apache.org>]

Hi All,

I approved RR [4115] in GH. It looks pretty reasonable to me and generally
matches the shape of other recent feature PRs. I tend to think it should be
fine to merge and do follow-up work on main.

From my POV, the only technical issue requiring immediate resolution is the
merge order with [4397].

WDYT?

[4115] https://github.com/apache/polaris/pull/4115

[4397] https://github.com/apache/polaris/pull/4397

Thanks,
Dmitri.

On Wed, May 6, 2026 at 5:08 PM EJ Wang <ej...@gmail.com>
wrote:

> Hi folks,
>
> We had a community sync earlier, thanks JB for scheduling it. Notes from
> the first metrics architecture sync (May 6, 10-11am PT). Discussion doc
> with per-section status:
>
> https://docs.google.com/document/d/100h7c4damrUzVuquYbBHM0EvA4LSWuW2IT2dN_7nYVA/edit?tab=t.0
>
> *The meeting covered both topics from the doc. Direction-level alignment
> was reached on the headline pieces; details remain for PR review or
> follow-up sessions.*
>
> *Topic 1 — Persistence schema redesign*
> Idea-level alignment on consolidating per-type tables (scan_metrics_report,
> commit_metrics_report) into a single metrics_report table. The motivating
> cost is the surface area added by every new metric type today: new table,
> SPI method, record class, model, converter, schema migration.
>
> Most schema details are deferred to the schema PR. A few specific points
> came up:
> •   metric_schema_version: Yufei prefers dropping it, since there is no
> spec-level concept of metrics versioning today and it is hard to define
> unilaterally. Robert prefers keeping it, given IRC v2 is coming and the
> schema should be considered against its likely shape; Robert also raised
> how to differentiate various payload formats if any. EJ's read is that this
> is a two-way-door decision. We can start without the field, and if IRC v2
> changes the shape we would likely roll a corresponding new schema anyway,
> which is not particularly costly.
> •   Payload format: Robert pointed out that future formats beyond JSON may
> be worth supporting. The exact shape is deferred to the schema discussion.
> •   Partition strategy: Anand suggested monthly partitioning based on his
> experience as potentially helpful at scale.
>
> *Topic 2 — Where metrics ingestion and storage belong*
> Idea-level alignment that metrics should be a separated SPI from the entity
> persistence stack. Two reasons surfaced: (a) workloads and capability
> requirements diverge enough that coupling them creates artificial
> constraints, and (b) admin experience improves when metrics has its own
> bootstrap, retention, and lifecycle. Dmitri noted Polaris being a platform
> should have the flexibility to support different persistence backends per
> concern, and pointed to a concrete next step of separating the JDBC
> bootstrap for metrics from the metastore bootstrap. Robert proposed an
> additional UX extension: detect an unbootstrapped metrics store on first
> use and auto-bootstrap rather than requiring an explicit manual bootstrap
> step.
> The meeting also confirmed that Polaris metrics can start small and stay
> Iceberg-focused. Naming and persistence schema can lean Iceberg-specific.
> If a future expansion to generic-table metrics or operational metrics
> arrives, an abstraction layer can be built on top of the Iceberg metrics
> reporter at that point. Robert remains on the fence and would prefer
> something more generic but did not block the direction; Dmitri's read was
> that the proposed framework already has enough flexibility to absorb future
> expansion.
>
> The Trade-offs and Proposed structure sections in the doc were not reviewed
> in detail. They remain open for either the next sync or PR review.
>
> *Cross-cutting alignment — battery-included plus pluggable*
> A common philosophy emerged from the discussion. EJ summarized it as:
> Polaris should provide a battery-included UX for beginners and the
> flexibility for advanced users to swap the included battery for something
> more powerful or tailored to their use case. The SPI design needs to enable
> both.
>
> The inputs that shaped this framing:
> •   Anand described how his team uses the current metrics persistence
> (three metrics consumers in v1.4).
> •   Yufei raised Grafana and dashboard integrations as a destination use
> case beyond the default.
> •   Robert called out that the current design is more JDBC-focused.
>
> Two concrete instances:
> •   Async metrics intake: Yufei's initial position was that async should
> largely live on the producer side and there is not much Polaris can do.
> Robert suggested a Polaris-side default is doable via Vert.x. Dmitri agreed
> the direction is worth exploring. The meeting converged on a
> battery-included default (likely Vert.x-backed) with an SPI shape that lets
> power users route to a more scalable backend (k8s-hosted queue, AWS SQS,
> etc.).
> •   Pluggable destinations: combining Yufei's dashboard use case with
> Robert's JDBC-focused call-out, the meeting agreed the SPI should be
> structured for multiple sinks so integrations become impl choices rather
> than architectural changes.
>
> The battery-included default is most likely to use the existing JDBC-backed
> approach.
>
> *Direction (idea-level alignment)*
> •   Single metrics_report table consolidating per-type metrics, replacing
> scan_metrics_report and commit_metrics_report
> •   Iceberg-focused naming and schema for now, revisit if generic-table or
> operational metrics arrive
> •   Metrics persistence as a separated SPI, not on BasePersistence
> •   Bootstrap path separated for metrics, independent of metastore
> bootstrap
> •   "Battery-included plus pluggable" as the SPI design philosophy
>
> *Open items*
> •   Schema details: metric_schema_version, payload format, IRC v2
> forward-compat shape
> •   SPI design details — full review either in the next sync or in the
> corresponding PR
> •   Schema refactor PR ownership
>
> *Action items*
> •   EJ to take a first stab at the SPI design and potentially partner with
> Anand to incorporate the lessons learned from the existing reporter and
> persistence work.
> •   Schema refactor PR ownership is not yet decided. If anyone is
> interested in driving it, reply on this thread.
> •   JB to schedule the next sync, tentatively in two weeks.
>
> -ej
>
> On Mon, Apr 27, 2026 at 3:07 PM EJ Wang <ej...@gmail.com>
> wrote:
>
> > Thanks Yufei for the +1.
> >
> > JB, could you help add a biweekly metrics architecture sync to the
> Polaris
> > community calendar? I'm thinking Thursdays at 9-10am PT, on the off-weeks
> > from the community meeting (starting May 7), 60 minutes.
> >
> > Here's a rough agenda to work through over the first few sessions,
> grouped
> > by priority:
> >
> > *First: foundational direction*
> >
> > 1.  MetricsPersistence: public SPI or internal implementation detail?
> >    •   Marked @Beta, javadoc calls it a "Service Provider Interface", but
> > only one consumer (JdbcBasePersistenceImpl), lives on BasePersistence. If
> > demoted to a private helper inside a persisting reporter impl, most
> > downstream design decisions become implementation details rather than
> > contract questions.
> >
> > 2.  Persistence schema redesign
> >    •   Current two-table layout (scan_metrics_report,
> > commit_metrics_report) with ~25 flattened columns each. Every new metric
> > type requires a new table, SPI method, record class, model, converter,
> and
> > schema migration. Direction to explore: single table with metric_type
> enum,
> > schema_version, and JSON payload column.
> >
> > *Second: design details once direction is set*
> >
> > 3.  Partition key strategy
> >    •   Single-table design means scan metrics at scale will have high
> > write concurrency per table. Schema needs to expose enough structure for
> > backends to shard by entity or time range.
> >
> > 4.  Read/write path consistency
> >    •   Writes go through PolarisMetricsManager on MetaStoreManager. Reads
> > bypass MetaStoreManager and go straight to BasePersistence, excluding
> > non-JDBC backends from the read API.
> >
> > *Third: cleanup and alignment*
> >
> > 5.  PolarisMetricsReporter naming
> >    •   Only handles IRC (ScanReport/CommitReport), doesn't cover generic
> > tables or operational metrics. Name is broader than scope.
> >
> > 6.  PolarisMetricsManager facade passthrough
> >    •   Entire default method is callCtx.getMetaStore().writeScanReport().
> > Zero logic, passes Level 1 straight through to Level 3. Same anti-pattern
> > as PolarisEventManager.
> >
> > 7.  Iceberg community alignment
> >    •   Payload-type extension needs discussion on dev@iceberg.
> obelix74's
> > Feb thread got zero replies. Needs a committer voice.
> >
> > Lets confirm prioritization in the first session.
> >
> > -ej
> >
> > On Tue, Apr 21, 2026 at 3:18 PM Yufei Gu <fl...@gmail.com> wrote:
> >
> >> Thanks everyone for continuing to drive this forward. I agree that the
> >> problem is getting complex enough that a more structured discussion
> would
> >> help.
> >>
> >> +1 on setting up a biweekly sync for the metrics architecture. I’m happy
> >> to
> >> join.
> >>
> >> Yufei
> >>
> >>
> >> On Tue, Apr 21, 2026 at 2:34 PM EJ Wang <ej.wang.dev.interest@gmail.com
> >
> >> wrote:
> >>
> >> > Also, I've been looking more closely at the *persistence schema in the
> >> > current metrics work*, and I think there's a structural rigidity
> problem
> >> > worth raising before the shape gets locked in.
> >> >
> >> > Right now we have two separate tables (scan_metrics_report and
> >> > commit_metrics_report), each with ~25 flattened columns that directly
> >> > mirror the Iceberg report fields. The SPI follows the same split:
> >> > writeScanReport and writeCommitReport as separate methods, with
> per-type
> >> > record classes, converters, and model objects. *The practical cost:
> >> > adding a new metric type (operational metrics, for example) requires a
> >> new
> >> > table, a new SPI method, a new record class, a new model class, a new
> >> > converter branch, and a schema migration*. That's a lot of surface
> area
> >> > for what should be "one more kind of metric."
> >> >
> >> > *My bias* would be toward a single metrics table with *a typed JSON
> >> > payload*. Something like: metric_type (enum), entity_id,
> >> > table_identifier, snapshot_id (nullable), received_ts, schema_version,
> >> and
> >> > a payload column for the metric-specific data. The metric_type +
> >> > schema_version pair gives us a forward-compatible contract for the
> >> payload
> >> > shape. Adding a new metric type becomes an enum value and a payload
> >> schema,
> >> > not a schema migration. One thing I think we need to be deliberate
> >> about is
> >> > the partition key design. If all metric types land in one table, scan
> >> > metrics at scale (high concurrency, high frequency across many tables)
> >> > could easily create hot partitions. We'd want the persistence layer to
> >> be
> >> > able to shard by entity or time range, and that means the logical
> schema
> >> > needs to expose enough structure for backends to partition on. I don't
> >> > think the current flattened layout gives us that.
> >> >
> >> > This is getting complex enough that I don't think ad-hoc PR/ML threads
> >> > will converge well. *Would people be open to a biweekly sync for
> metrics
> >> > architecture?* I think 30 minutes every two weeks with interested
> >> parties
> >> > would be enough to work through the schema, SPI shape, and read API
> >> design
> >> > together. Happy to help set that up.
> >> >
> >> > -ej
> >> >
> >> > On Mon, Apr 20, 2026 at 2:19 PM EJ Wang <
> ej.wang.dev.interest@gmail.com
> >> >
> >> > wrote:
> >> >
> >> >> Reviewed #4115, left a comment on the code organization side.
> >> >>
> >> >> One thing stood out: the metrics write path enters through
> >> >> PolarisMetricsManager on MetaStoreManager, but the new read path
> >> bypasses
> >> >> MetaStoreManager entirely and goes straight to BasePersistence via
> >> >> callContext.getMetaStore(). That means the read API only works for
> >> backends
> >> >> that implement BasePersistence. NoSQL and remote backends can't
> >> participate.
> >> >>
> >> >> Stepping back, I think the metrics subsystem is growing into
> something
> >> >> real (write + read + REST API + AuthZ + pagination) *but the
> >> persistence
> >> >> side is split across two layers in a way that's hard to extend*. I
> put
> >> >> together two diagrams to show what I mean (my best effort).
> >> >>
> >> >> *Current state* (Diagram 1): three interfaces at three different
> >> levels.
> >> >> The engine-facing SPI (PolarisMetricsReporter) is clean. But
> >> >> PolarisMetricsManager on MetaStoreManager is a passthrough to
> >> >> MetricsPersistence on BasePersistence. The @Beta annotation and SPI
> >> javadoc
> >> >> are on the BasePersistence layer, while the actual extension points
> >> >> (PolarisMetricsReporter, PolarisMetricsManager) carry no stability
> >> >> annotation. The write path goes through the MetaStoreManager layer,
> the
> >> >> read path doesn't.
> >> >>
> >> >> *What I envision* (Diagram 2): two SPIs at two levels.
> >> >> PolarisMetricsReporter stays as the engine-facing SPI.
> >> >> PolarisMetricsManager becomes the backend-facing SPI with both write
> >> and
> >> >> read methods at the MetaStoreManager level, where any backend (JDBC,
> >> NoSQL,
> >> >> remote) can implement them. MetricsPersistence on BasePersistence
> goes
> >> >> away. Where metrics actually land is an implementation detail, not a
> >> core
> >> >> interface.
> >> >>
> >> >> *Minor naming thing*: PolarisMetricsReporter is broader than what it
> >> >> actually handles. It only accepts Iceberg REST Catalog metrics
> >> (ScanReport,
> >> >> CommitReport via MetricsReport). Generic table metrics or operational
> >> >> metrics aren't in scope. Not blocking, but worth noting if the
> metrics
> >> >> surface expands.
> >> >>
> >> >> *Rough sketch of how to get there*:
> >> >>  1.  Add read methods to PolarisMetricsManager (listScanReports,
> >> >> listCommitReports) with default no-op, same as the existing write
> >> methods.
> >> >> (Probably make PolarisMetricsManager more explicit on being Iceberg
> >> >> specific like package name or class name etc.)
> >> >>  2.  Wire MetricsReportsService through MetaStoreManager instead of
> >> >> callContext.getMetaStore().
> >> >>  3.  Extract metrics persistence from JdbcBasePersistenceImpl into
> its
> >> >> own class. That file carries ~7 responsibilities, metrics being one
> of
> >> them.
> >> >>  4.  Remove MetricsPersistence from BasePersistence.
> >> >>
> >> >> *None of this needs to happen in #4115. But if the direction makes
> >> sense,
> >> >> it would be good to align before the metrics surface grows further.
> >> Curious
> >> >> what others think.*
> >> >>
> >> >> *My mental model note*: Level 1 MetaStoreManager; level 2
> transactional
> >> >> persistence; level 3 base persistence
> >> >>
> >> >> Diagram 1
> >> >> <
> >>
> https://www.plantuml.com/plantuml/uml/bLHDR-Cs4BthLmpIYupw0zbkKQ1r3M-S7Bp8xhhM7WCOb3IM65EaGD9EX2RzxHrHb4CxRelwa4YSDu_lpOVcnZ9jzvM8BBS2uGjQpJC3dtHMSekPtMk44IpsMgEqa5XcCOhCZikQQLP1pR8TAp2n3ILhmZDP20m0fcIvUkAoW2qJXd9z1bpToO9BX3WXu0ucy5rpgGPNm0nW5_epUWtm2Ue3pn3kMOFQmKntGZW0BYtgBSi8k5A2QMwybJNMIbFiGSR9QZc4nUqIvikStF0jHprua5C-amge42aNt3R0f5JaaoivdV2Pkqbx4hee4ymOkBh5BTiB-_uIeGeo8zL8rPsPl4DktdEiK1jkB1NdZCRbrSTecDe_mlHbF0wvBmCkaOH5_S8a_TTTKI6-nmCAkEw4LpxsZ-LbYLKQFKMNOgf_wuM7_bV9gOer5SYMMksBSWXFcbi49KNZXNLicwfe3TETC7gPdPqI7uBcHMb1RSzYq34c6PDUM9mn8HRsUTZEiDBve3NjVZumBj0U7SS37mGO7vcwtiK-_pU7U7L_f-digo9YbhSwIfMRwIITKGXbxdIUTCGF1SeCJxloKsU-3k9ddRbX1eDq1q_fx1JbBGT0glVyXimDuP4TQ5qpCAmnGEj2s_6n5mtn1z-97-63itFQZLPO1Ev2tu_WF7Ju-VPc0Skg5bYXxBhkY1xpD7EM_7fyflSpIsqMgVth5xhVr4eQxWQ8enaSAJQSG16yFSDuJ798rrcXr_3n-lfdk7icQjEBmFujL7AodiP_Y4Z7-YxvtZNs4zMgpNTl6tF8sglyPsmqchrjvQ-m-aP94r-TwCA2Ka8upPJZwtvSpoYCXkYMZU2NXvRMBfq9P3i3Le4VAZUAlUZ_oPKsxPgY0Q_BSKLkyr9bhQhQrJjo_x3TPlIB0DPjnMfcIoYP0QaYw1a0fTKDr8fB6ntNuvmoL1ZGkXa69Njh43zf9GiGxHQrA_jDYWRSzF5--WmTVrN97_Sm8LbLUy_lGBmLanJjFkDlGkRqjA_4tm00
> >> >
> >> >> :
> >> >>
> >> >> [image: image.png]
> >> >>
> >> >> Diagram 2
> >> >> <
> >>
> https://www.plantuml.com/plantuml/uml/VLLDR-8m4BtdLupO2sWBLVU8AaGB7AXAbssGzb896SS9RXqxjKqBwkv_tt7iV43fdaZYDpFlpRmnOsE9jhjSH9PRmM31hERKm8scMsuPjJlDe0yheZDc8RR4iYWoBrmMH9CS2a9VICPYUy1OZN0YCy5Q0BCbYNhdCeEK28En8G8wCvbnoQ0R8_05Bc6bkLIz3X03p1zzH7zR-9ZfDquPt9C3qoNCX2yV4G2NbkcKu5jdgGJHt0GbZwnG6i-UP3TUpk5gM6Ldqke350eZUqzoCft3U9xWHvxoa5-7K4nF1J46EbEMafsmdrCBbQ44gVggy18IZrn_ph5asd1ZiIKdQSgueZvjXrQFSFrdC3YN-nXmBacxbGiYyLVxLaBtdhqn0LSzdBDhqQtQoOJeGyad3z0lUqnYgpGB6Ns8oVyta00Dy_WnX0tIOZ8v6SYxHll1TrH6aejAik-mh-AphVFCwSUQqFypElag5QRGFDjQKEd96K1P8QP41c9TzA_IIQyvdAWyv_RSiS3skb0_EzDDkK2v5xWF6MiGFlvhpFLcD2Dq2pml14gaF67eQkmd8gulDoC4kSOu6KVpkvlUJg1RTbWISU40RdBUUS_9XfRZ2dwxm_SW8LYFISgm_MnlDQ6M9P1gbKEc4X-2pH_FvJCkCqm9pbVjD6LrwdLeOrDWfOaqc8Wh9BE85oNKxkNQ6o4yGRy_Eae0G_G8tZv81d3bHDB23WOdisohVr3nh_j6lbSjbNaLRTc8UgtPbAU1J_tygOfZX9DWEJeHDvYx-qmSi5FgNLPZwHrHcUsncGQ5-skhUclpE5fo4ounpFauYrUbkU6ccfnxMvitwag4IyerhTxj8In_Oj1bDO4pQru674loYrGlULHLEGCjwJJ8gDoVZR8MxO4BT3IzRvIcAQKezC6xpziGnTyImrfEGyJI_OcKfgtxIvnTqFEMS17L9Z-jsARN5FmTheP7HtSdtOMT0B4GY2FYHXxgQmMtj2bRqiLFGapiVe1_QVKDrkqXcm83aFEXnMYCZ-xlyHy
> >> >
> >> >> :
> >> >> [image: image.png]
> >> >>
> >> >>  -ej
> >> >>
> >> >> On Wed, Apr 15, 2026 at 8:22 AM Dmitri Bourlatchkov <
> dimas@apache.org>
> >> >> wrote:
> >> >>
> >> >>> Hi All,
> >> >>>
> >> >>> Heads up: The current state of PR [4115] looks pretty solid to me. I
> >> >>> believe this PR is approaching a mergeable condition.
> >> >>>
> >> >>> Please post your reviews if you have any comments.
> >> >>>
> >> >>> [4115] https://github.com/apache/polaris/pull/4115
> >> >>>
> >> >>> Thanks,
> >> >>> Dmitri.
> >> >>>
> >> >>> On Tue, Mar 3, 2026 at 3:29 PM Anand Kumar Sankaran via dev <
> >> >>> dev@polaris.apache.org> wrote:
> >> >>>
> >> >>> > Hi Yufei and Dmitri,
> >> >>> >
> >> >>> > Here is a proposal for the REST endpoints for metrics and events.
> >> >>> >
> >> >>> > https://github.com/apache/polaris/pull/3924/changes
> >> >>> >
> >> >>> > I did not see any precursors for raising a PR for proposals, so
> >> trying
> >> >>> > this.  Please let me know what you think.
> >> >>> >
> >> >>> > -
> >> >>> > Anand
> >> >>> >
> >> >>> > From: Anand Kumar Sankaran <an...@workday.com>
> >> >>> > Date: Monday, March 2, 2026 at 10:25 AM
> >> >>> > To: dev@polaris.apache.org <de...@polaris.apache.org>
> >> >>> > Subject: Re: Polaris Telemetry and Audit Trail
> >> >>> >
> >> >>> > About the REST API, based on my use cases:
> >> >>> >
> >> >>> >
> >> >>> >   1.
> >> >>> > I want to be able to query commit metrics to track files added /
> >> >>> removed
> >> >>> > per commit, along with record counts. The ingestion pipeline that
> >> >>> writes
> >> >>> > this data is owned by us and we are guaranteed to write this
> >> >>> information
> >> >>> > for each write.
> >> >>> >   2.
> >> >>> > I want to be able to query scan metrics for read. I understand
> >> clients
> >> >>> do
> >> >>> > not fulfill this requirement.
> >> >>> >   3.
> >> >>> > I want to be able to query the events table (events are
> persisted) -
> >> >>> this
> >> >>> > may supersede #2, I am not sure yet.
> >> >>> >
> >> >>> > All this information is in the JDBC based persistence model and is
> >> >>> > persisted in the metastore. I currently don’t have a need to query
> >> >>> > prometheus or open telemetry. I do publish some events to
> Prometheus
> >> >>> and
> >> >>> > they are forwarded to our dashboards elsewhere.
> >> >>> >
> >> >>> > About the CLI utilities, I meant the admin user utilities. In one
> of
> >> >>> the
> >> >>> > earliest drafts of my proposal, Prashant mentioned that the
> metrics
> >> >>> tables
> >> >>> > can grow indefinitely and that a similar problem exists with the
> >> events
> >> >>> > table as well. We discussed that cleaning up of old records from
> >> both
> >> >>> > metrics tables and events tables can be done via a CLI utility.
> >> >>> >
> >> >>> > I see that Yufei has covered the discussion about datasources.
> >> >>> >
> >> >>> > -
> >> >>> > Anand
> >> >>> >
> >> >>> >
> >> >>> >
> >> >>> > From: Yufei Gu <fl...@gmail.com>
> >> >>> > Date: Friday, February 27, 2026 at 9:54 PM
> >> >>> > To: dev@polaris.apache.org <de...@polaris.apache.org>
> >> >>> > Subject: Re: Polaris Telemetry and Audit Trail
> >> >>> >
> >> >>> > This Message Is From an External Sender
> >> >>> > This message came from outside your organization.
> >> >>> > Report Suspicious<
> >> >>> >
> >> >>>
> >>
> https://us-phishalarm-ewt.proofpoint.com/EWT/v1/Iz9xO38YGHZK!YhNDZABkHi1B699ote2uMwpOZw8i0QMCGO2Szc-HshuABGhGvwPJcymE6G2oUUxtS8xDkSrtGTPm_I3QnVDHoLMk50m9v8z_nZKTkd-bnVUbreF1u0WnfV_X5eYevZl_$
> >> >>> > >
> >> >>> >
> >> >>> >
> >> >>> > As I mentioned in
> >> >>> >
> >> >>>
> >>
> https://urldefense.com/v3/__https://github.com/apache/polaris/issues/3890__;!!Iz9xO38YGHZK!5EuyFFkk3vhRWVIRvQAWBSQfpJkTMA9HxugzDwXmN0LPPqhEFxYkFRGVhtb8AqUwXtDh2OplcMnbMDHKOxrvDU0$
> >> >>> ,
> >> >>> > supporting
> >> >>> > multiple data sources is not a trivial change. I would strongly
> >> >>> recommend
> >> >>> > starting with a design document to carefully evaluate the
> >> architectural
> >> >>> > implications and long term impact.
> >> >>> >
> >> >>> > A REST endpoint to query metrics seems reasonable given the
> current
> >> >>> JDBC
> >> >>> > based persistence model. That said, we may also consider
> alternative
> >> >>> > storage models. For example, if we later adopt a time series
> system
> >> >>> such as
> >> >>> > Prometheus to store metrics, the query model and access patterns
> >> would
> >> >>> be
> >> >>> > fundamentally different. Designing the REST API without
> considering
> >> >>> these
> >> >>> > potential evolutions may limit flexibility. I'd suggest to start
> >> with
> >> >>> the
> >> >>> > use case.
> >> >>> >
> >> >>> > Yufei
> >> >>> >
> >> >>> >
> >> >>> > On Fri, Feb 27, 2026 at 3:42 PM Dmitri Bourlatchkov <
> >> dimas@apache.org>
> >> >>> > wrote:
> >> >>> >
> >> >>> > > Hi Anand,
> >> >>> > >
> >> >>> > > Sharing my view... subject to discussion:
> >> >>> > >
> >> >>> > > 1. Adding non-IRC REST API to Polaris is perfectly fine.
> >> >>> > >
> >> >>> > > Figuring out specific endpoint URIs and payloads might require a
> >> few
> >> >>> > > roundtrips, so opening a separate thread for that might be best.
> >> >>> > > Contributors commonly create Google Docs for new API proposals
> too
> >> >>> (they
> >> >>> > > fairly easy to update as the email discussion progresses).
> >> >>> > >
> >> >>> > > There was a suggestion to try Markdown (with PRs) for proposals
> >> [1]
> >> >>> ...
> >> >>> > > feel free to give it a try if you are comfortable with that.
> >> >>> > >
> >> >>> > > 2. Could you clarify whether you mean end user utilities or
> admin
> >> >>> user
> >> >>> > > utilities? In the latter case those might be more suitable for
> the
> >> >>> Admin
> >> >>> > > CLI (java) not the Python CLI, IMHO.
> >> >>> > >
> >> >>> > > Why would these utilities be common with events? IMHO, event use
> >> >>> cases
> >> >>> > are
> >> >>> > > distinct from scan/commit metrics.
> >> >>> > >
> >> >>> > > 3. I'd prefer separating metrics persistence from MetaStore
> >> >>> persistence
> >> >>> > at
> >> >>> > > the code level, so that they could be mixed and matched
> >> >>> independently.
> >> >>> > The
> >> >>> > > separate datasource question will become a non-issue with that
> >> >>> approach,
> >> >>> > I
> >> >>> > > guess.
> >> >>> > >
> >> >>> > > The rationale for separating scan metrics and metastore
> >> persistence
> >> >>> is
> >> >>> > that
> >> >>> > > "cascading deletes" between them are hardly ever required.
> >> >>> Furthermore,
> >> >>> > the
> >> >>> > > data and query patterns are very different so different
> >> technologies
> >> >>> > might
> >> >>> > > be beneficial in each case.
> >> >>> > >
> >> >>> > > [1]
> >> >>> >
> >> >>>
> >>
> https://urldefense.com/v3/__https://lists.apache.org/thread/yto2wp982t43h1mqjwnslswhws5z47cy__;!!Iz9xO38YGHZK!5EuyFFkk3vhRWVIRvQAWBSQfpJkTMA9HxugzDwXmN0LPPqhEFxYkFRGVhtb8AqUwXtDh2OplcMnbMDHKxYDakNU$
> >> >>> > >
> >> >>> > > Cheers,
> >> >>> > > Dmitri.
> >> >>> > >
> >> >>> > > On Fri, Feb 27, 2026 at 6:19 PM Anand Kumar Sankaran via dev <
> >> >>> > > dev@polaris.apache.org> wrote:
> >> >>> > >
> >> >>> > > > Thanks all. This PR is merged now.
> >> >>> > > >
> >> >>> > > > Here are the follow-up features / work needed.  These were all
> >> >>> part of
> >> >>> > > the
> >> >>> > > > merged PR at some point in time and were removed to reduce
> >> scope.
> >> >>> > > >
> >> >>> > > > Please let me know what you think.
> >> >>> > > >
> >> >>> > > >
> >> >>> > > >   1.  A REST API to paginate through table metrics. This will
> be
> >> >>> > non-IRC
> >> >>> > > > standard addition.
> >> >>> > > >   2.  Utilities for managing old records, should be common
> with
> >> >>> events.
> >> >>> > > > There was some discussion that it belongs to the CLI.
> >> >>> > > >   3.  Separate datasource (metrics, events, even other
> tables?).
> >> >>> > > >
> >> >>> > > >
> >> >>> > > > Anything else?
> >> >>> > > >
> >> >>> > > > -
> >> >>> > > > Anand
> >> >>> > > >
> >> >>> > > >
> >> >>> > >
> >> >>> >
> >> >>> >
> >> >>>
> >> >>
> >>
> >
>

Re: Proposal for REST endpoints for table metrics and events

[EJ Wang <ej...@gmail.com>]

Hi folks,

We had a community sync earlier, thanks JB for scheduling it. Notes from
the first metrics architecture sync (May 6, 10-11am PT). Discussion doc
with per-section status:
https://docs.google.com/document/d/100h7c4damrUzVuquYbBHM0EvA4LSWuW2IT2dN_7nYVA/edit?tab=t.0

*The meeting covered both topics from the doc. Direction-level alignment
was reached on the headline pieces; details remain for PR review or
follow-up sessions.*

*Topic 1 — Persistence schema redesign*
Idea-level alignment on consolidating per-type tables (scan_metrics_report,
commit_metrics_report) into a single metrics_report table. The motivating
cost is the surface area added by every new metric type today: new table,
SPI method, record class, model, converter, schema migration.

Most schema details are deferred to the schema PR. A few specific points
came up:
•   metric_schema_version: Yufei prefers dropping it, since there is no
spec-level concept of metrics versioning today and it is hard to define
unilaterally. Robert prefers keeping it, given IRC v2 is coming and the
schema should be considered against its likely shape; Robert also raised
how to differentiate various payload formats if any. EJ's read is that this
is a two-way-door decision. We can start without the field, and if IRC v2
changes the shape we would likely roll a corresponding new schema anyway,
which is not particularly costly.
•   Payload format: Robert pointed out that future formats beyond JSON may
be worth supporting. The exact shape is deferred to the schema discussion.
•   Partition strategy: Anand suggested monthly partitioning based on his
experience as potentially helpful at scale.

*Topic 2 — Where metrics ingestion and storage belong*
Idea-level alignment that metrics should be a separated SPI from the entity
persistence stack. Two reasons surfaced: (a) workloads and capability
requirements diverge enough that coupling them creates artificial
constraints, and (b) admin experience improves when metrics has its own
bootstrap, retention, and lifecycle. Dmitri noted Polaris being a platform
should have the flexibility to support different persistence backends per
concern, and pointed to a concrete next step of separating the JDBC
bootstrap for metrics from the metastore bootstrap. Robert proposed an
additional UX extension: detect an unbootstrapped metrics store on first
use and auto-bootstrap rather than requiring an explicit manual bootstrap
step.
The meeting also confirmed that Polaris metrics can start small and stay
Iceberg-focused. Naming and persistence schema can lean Iceberg-specific.
If a future expansion to generic-table metrics or operational metrics
arrives, an abstraction layer can be built on top of the Iceberg metrics
reporter at that point. Robert remains on the fence and would prefer
something more generic but did not block the direction; Dmitri's read was
that the proposed framework already has enough flexibility to absorb future
expansion.

The Trade-offs and Proposed structure sections in the doc were not reviewed
in detail. They remain open for either the next sync or PR review.

*Cross-cutting alignment — battery-included plus pluggable*
A common philosophy emerged from the discussion. EJ summarized it as:
Polaris should provide a battery-included UX for beginners and the
flexibility for advanced users to swap the included battery for something
more powerful or tailored to their use case. The SPI design needs to enable
both.

The inputs that shaped this framing:
•   Anand described how his team uses the current metrics persistence
(three metrics consumers in v1.4).
•   Yufei raised Grafana and dashboard integrations as a destination use
case beyond the default.
•   Robert called out that the current design is more JDBC-focused.

Two concrete instances:
•   Async metrics intake: Yufei's initial position was that async should
largely live on the producer side and there is not much Polaris can do.
Robert suggested a Polaris-side default is doable via Vert.x. Dmitri agreed
the direction is worth exploring. The meeting converged on a
battery-included default (likely Vert.x-backed) with an SPI shape that lets
power users route to a more scalable backend (k8s-hosted queue, AWS SQS,
etc.).
•   Pluggable destinations: combining Yufei's dashboard use case with
Robert's JDBC-focused call-out, the meeting agreed the SPI should be
structured for multiple sinks so integrations become impl choices rather
than architectural changes.

The battery-included default is most likely to use the existing JDBC-backed
approach.

*Direction (idea-level alignment)*
•   Single metrics_report table consolidating per-type metrics, replacing
scan_metrics_report and commit_metrics_report
•   Iceberg-focused naming and schema for now, revisit if generic-table or
operational metrics arrive
•   Metrics persistence as a separated SPI, not on BasePersistence
•   Bootstrap path separated for metrics, independent of metastore bootstrap
•   "Battery-included plus pluggable" as the SPI design philosophy

*Open items*
•   Schema details: metric_schema_version, payload format, IRC v2
forward-compat shape
•   SPI design details — full review either in the next sync or in the
corresponding PR
•   Schema refactor PR ownership

*Action items*
•   EJ to take a first stab at the SPI design and potentially partner with
Anand to incorporate the lessons learned from the existing reporter and
persistence work.
•   Schema refactor PR ownership is not yet decided. If anyone is
interested in driving it, reply on this thread.
•   JB to schedule the next sync, tentatively in two weeks.

-ej

On Mon, Apr 27, 2026 at 3:07 PM EJ Wang <ej...@gmail.com>
wrote:

> Thanks Yufei for the +1.
>
> JB, could you help add a biweekly metrics architecture sync to the Polaris
> community calendar? I'm thinking Thursdays at 9-10am PT, on the off-weeks
> from the community meeting (starting May 7), 60 minutes.
>
> Here's a rough agenda to work through over the first few sessions, grouped
> by priority:
>
> *First: foundational direction*
>
> 1.  MetricsPersistence: public SPI or internal implementation detail?
>    •   Marked @Beta, javadoc calls it a "Service Provider Interface", but
> only one consumer (JdbcBasePersistenceImpl), lives on BasePersistence. If
> demoted to a private helper inside a persisting reporter impl, most
> downstream design decisions become implementation details rather than
> contract questions.
>
> 2.  Persistence schema redesign
>    •   Current two-table layout (scan_metrics_report,
> commit_metrics_report) with ~25 flattened columns each. Every new metric
> type requires a new table, SPI method, record class, model, converter, and
> schema migration. Direction to explore: single table with metric_type enum,
> schema_version, and JSON payload column.
>
> *Second: design details once direction is set*
>
> 3.  Partition key strategy
>    •   Single-table design means scan metrics at scale will have high
> write concurrency per table. Schema needs to expose enough structure for
> backends to shard by entity or time range.
>
> 4.  Read/write path consistency
>    •   Writes go through PolarisMetricsManager on MetaStoreManager. Reads
> bypass MetaStoreManager and go straight to BasePersistence, excluding
> non-JDBC backends from the read API.
>
> *Third: cleanup and alignment*
>
> 5.  PolarisMetricsReporter naming
>    •   Only handles IRC (ScanReport/CommitReport), doesn't cover generic
> tables or operational metrics. Name is broader than scope.
>
> 6.  PolarisMetricsManager facade passthrough
>    •   Entire default method is callCtx.getMetaStore().writeScanReport().
> Zero logic, passes Level 1 straight through to Level 3. Same anti-pattern
> as PolarisEventManager.
>
> 7.  Iceberg community alignment
>    •   Payload-type extension needs discussion on dev@iceberg. obelix74's
> Feb thread got zero replies. Needs a committer voice.
>
> Lets confirm prioritization in the first session.
>
> -ej
>
> On Tue, Apr 21, 2026 at 3:18 PM Yufei Gu <fl...@gmail.com> wrote:
>
>> Thanks everyone for continuing to drive this forward. I agree that the
>> problem is getting complex enough that a more structured discussion would
>> help.
>>
>> +1 on setting up a biweekly sync for the metrics architecture. I’m happy
>> to
>> join.
>>
>> Yufei
>>
>>
>> On Tue, Apr 21, 2026 at 2:34 PM EJ Wang <ej...@gmail.com>
>> wrote:
>>
>> > Also, I've been looking more closely at the *persistence schema in the
>> > current metrics work*, and I think there's a structural rigidity problem
>> > worth raising before the shape gets locked in.
>> >
>> > Right now we have two separate tables (scan_metrics_report and
>> > commit_metrics_report), each with ~25 flattened columns that directly
>> > mirror the Iceberg report fields. The SPI follows the same split:
>> > writeScanReport and writeCommitReport as separate methods, with per-type
>> > record classes, converters, and model objects. *The practical cost:
>> > adding a new metric type (operational metrics, for example) requires a
>> new
>> > table, a new SPI method, a new record class, a new model class, a new
>> > converter branch, and a schema migration*. That's a lot of surface area
>> > for what should be "one more kind of metric."
>> >
>> > *My bias* would be toward a single metrics table with *a typed JSON
>> > payload*. Something like: metric_type (enum), entity_id,
>> > table_identifier, snapshot_id (nullable), received_ts, schema_version,
>> and
>> > a payload column for the metric-specific data. The metric_type +
>> > schema_version pair gives us a forward-compatible contract for the
>> payload
>> > shape. Adding a new metric type becomes an enum value and a payload
>> schema,
>> > not a schema migration. One thing I think we need to be deliberate
>> about is
>> > the partition key design. If all metric types land in one table, scan
>> > metrics at scale (high concurrency, high frequency across many tables)
>> > could easily create hot partitions. We'd want the persistence layer to
>> be
>> > able to shard by entity or time range, and that means the logical schema
>> > needs to expose enough structure for backends to partition on. I don't
>> > think the current flattened layout gives us that.
>> >
>> > This is getting complex enough that I don't think ad-hoc PR/ML threads
>> > will converge well. *Would people be open to a biweekly sync for metrics
>> > architecture?* I think 30 minutes every two weeks with interested
>> parties
>> > would be enough to work through the schema, SPI shape, and read API
>> design
>> > together. Happy to help set that up.
>> >
>> > -ej
>> >
>> > On Mon, Apr 20, 2026 at 2:19 PM EJ Wang <ej.wang.dev.interest@gmail.com
>> >
>> > wrote:
>> >
>> >> Reviewed #4115, left a comment on the code organization side.
>> >>
>> >> One thing stood out: the metrics write path enters through
>> >> PolarisMetricsManager on MetaStoreManager, but the new read path
>> bypasses
>> >> MetaStoreManager entirely and goes straight to BasePersistence via
>> >> callContext.getMetaStore(). That means the read API only works for
>> backends
>> >> that implement BasePersistence. NoSQL and remote backends can't
>> participate.
>> >>
>> >> Stepping back, I think the metrics subsystem is growing into something
>> >> real (write + read + REST API + AuthZ + pagination) *but the
>> persistence
>> >> side is split across two layers in a way that's hard to extend*. I put
>> >> together two diagrams to show what I mean (my best effort).
>> >>
>> >> *Current state* (Diagram 1): three interfaces at three different
>> levels.
>> >> The engine-facing SPI (PolarisMetricsReporter) is clean. But
>> >> PolarisMetricsManager on MetaStoreManager is a passthrough to
>> >> MetricsPersistence on BasePersistence. The @Beta annotation and SPI
>> javadoc
>> >> are on the BasePersistence layer, while the actual extension points
>> >> (PolarisMetricsReporter, PolarisMetricsManager) carry no stability
>> >> annotation. The write path goes through the MetaStoreManager layer, the
>> >> read path doesn't.
>> >>
>> >> *What I envision* (Diagram 2): two SPIs at two levels.
>> >> PolarisMetricsReporter stays as the engine-facing SPI.
>> >> PolarisMetricsManager becomes the backend-facing SPI with both write
>> and
>> >> read methods at the MetaStoreManager level, where any backend (JDBC,
>> NoSQL,
>> >> remote) can implement them. MetricsPersistence on BasePersistence goes
>> >> away. Where metrics actually land is an implementation detail, not a
>> core
>> >> interface.
>> >>
>> >> *Minor naming thing*: PolarisMetricsReporter is broader than what it
>> >> actually handles. It only accepts Iceberg REST Catalog metrics
>> (ScanReport,
>> >> CommitReport via MetricsReport). Generic table metrics or operational
>> >> metrics aren't in scope. Not blocking, but worth noting if the metrics
>> >> surface expands.
>> >>
>> >> *Rough sketch of how to get there*:
>> >>  1.  Add read methods to PolarisMetricsManager (listScanReports,
>> >> listCommitReports) with default no-op, same as the existing write
>> methods.
>> >> (Probably make PolarisMetricsManager more explicit on being Iceberg
>> >> specific like package name or class name etc.)
>> >>  2.  Wire MetricsReportsService through MetaStoreManager instead of
>> >> callContext.getMetaStore().
>> >>  3.  Extract metrics persistence from JdbcBasePersistenceImpl into its
>> >> own class. That file carries ~7 responsibilities, metrics being one of
>> them.
>> >>  4.  Remove MetricsPersistence from BasePersistence.
>> >>
>> >> *None of this needs to happen in #4115. But if the direction makes
>> sense,
>> >> it would be good to align before the metrics surface grows further.
>> Curious
>> >> what others think.*
>> >>
>> >> *My mental model note*: Level 1 MetaStoreManager; level 2 transactional
>> >> persistence; level 3 base persistence
>> >>
>> >> Diagram 1
>> >> <
>> https://www.plantuml.com/plantuml/uml/bLHDR-Cs4BthLmpIYupw0zbkKQ1r3M-S7Bp8xhhM7WCOb3IM65EaGD9EX2RzxHrHb4CxRelwa4YSDu_lpOVcnZ9jzvM8BBS2uGjQpJC3dtHMSekPtMk44IpsMgEqa5XcCOhCZikQQLP1pR8TAp2n3ILhmZDP20m0fcIvUkAoW2qJXd9z1bpToO9BX3WXu0ucy5rpgGPNm0nW5_epUWtm2Ue3pn3kMOFQmKntGZW0BYtgBSi8k5A2QMwybJNMIbFiGSR9QZc4nUqIvikStF0jHprua5C-amge42aNt3R0f5JaaoivdV2Pkqbx4hee4ymOkBh5BTiB-_uIeGeo8zL8rPsPl4DktdEiK1jkB1NdZCRbrSTecDe_mlHbF0wvBmCkaOH5_S8a_TTTKI6-nmCAkEw4LpxsZ-LbYLKQFKMNOgf_wuM7_bV9gOer5SYMMksBSWXFcbi49KNZXNLicwfe3TETC7gPdPqI7uBcHMb1RSzYq34c6PDUM9mn8HRsUTZEiDBve3NjVZumBj0U7SS37mGO7vcwtiK-_pU7U7L_f-digo9YbhSwIfMRwIITKGXbxdIUTCGF1SeCJxloKsU-3k9ddRbX1eDq1q_fx1JbBGT0glVyXimDuP4TQ5qpCAmnGEj2s_6n5mtn1z-97-63itFQZLPO1Ev2tu_WF7Ju-VPc0Skg5bYXxBhkY1xpD7EM_7fyflSpIsqMgVth5xhVr4eQxWQ8enaSAJQSG16yFSDuJ798rrcXr_3n-lfdk7icQjEBmFujL7AodiP_Y4Z7-YxvtZNs4zMgpNTl6tF8sglyPsmqchrjvQ-m-aP94r-TwCA2Ka8upPJZwtvSpoYCXkYMZU2NXvRMBfq9P3i3Le4VAZUAlUZ_oPKsxPgY0Q_BSKLkyr9bhQhQrJjo_x3TPlIB0DPjnMfcIoYP0QaYw1a0fTKDr8fB6ntNuvmoL1ZGkXa69Njh43zf9GiGxHQrA_jDYWRSzF5--WmTVrN97_Sm8LbLUy_lGBmLanJjFkDlGkRqjA_4tm00
>> >
>> >> :
>> >>
>> >> [image: image.png]
>> >>
>> >> Diagram 2
>> >> <
>> https://www.plantuml.com/plantuml/uml/VLLDR-8m4BtdLupO2sWBLVU8AaGB7AXAbssGzb896SS9RXqxjKqBwkv_tt7iV43fdaZYDpFlpRmnOsE9jhjSH9PRmM31hERKm8scMsuPjJlDe0yheZDc8RR4iYWoBrmMH9CS2a9VICPYUy1OZN0YCy5Q0BCbYNhdCeEK28En8G8wCvbnoQ0R8_05Bc6bkLIz3X03p1zzH7zR-9ZfDquPt9C3qoNCX2yV4G2NbkcKu5jdgGJHt0GbZwnG6i-UP3TUpk5gM6Ldqke350eZUqzoCft3U9xWHvxoa5-7K4nF1J46EbEMafsmdrCBbQ44gVggy18IZrn_ph5asd1ZiIKdQSgueZvjXrQFSFrdC3YN-nXmBacxbGiYyLVxLaBtdhqn0LSzdBDhqQtQoOJeGyad3z0lUqnYgpGB6Ns8oVyta00Dy_WnX0tIOZ8v6SYxHll1TrH6aejAik-mh-AphVFCwSUQqFypElag5QRGFDjQKEd96K1P8QP41c9TzA_IIQyvdAWyv_RSiS3skb0_EzDDkK2v5xWF6MiGFlvhpFLcD2Dq2pml14gaF67eQkmd8gulDoC4kSOu6KVpkvlUJg1RTbWISU40RdBUUS_9XfRZ2dwxm_SW8LYFISgm_MnlDQ6M9P1gbKEc4X-2pH_FvJCkCqm9pbVjD6LrwdLeOrDWfOaqc8Wh9BE85oNKxkNQ6o4yGRy_Eae0G_G8tZv81d3bHDB23WOdisohVr3nh_j6lbSjbNaLRTc8UgtPbAU1J_tygOfZX9DWEJeHDvYx-qmSi5FgNLPZwHrHcUsncGQ5-skhUclpE5fo4ounpFauYrUbkU6ccfnxMvitwag4IyerhTxj8In_Oj1bDO4pQru674loYrGlULHLEGCjwJJ8gDoVZR8MxO4BT3IzRvIcAQKezC6xpziGnTyImrfEGyJI_OcKfgtxIvnTqFEMS17L9Z-jsARN5FmTheP7HtSdtOMT0B4GY2FYHXxgQmMtj2bRqiLFGapiVe1_QVKDrkqXcm83aFEXnMYCZ-xlyHy
>> >
>> >> :
>> >> [image: image.png]
>> >>
>> >>  -ej
>> >>
>> >> On Wed, Apr 15, 2026 at 8:22 AM Dmitri Bourlatchkov <di...@apache.org>
>> >> wrote:
>> >>
>> >>> Hi All,
>> >>>
>> >>> Heads up: The current state of PR [4115] looks pretty solid to me. I
>> >>> believe this PR is approaching a mergeable condition.
>> >>>
>> >>> Please post your reviews if you have any comments.
>> >>>
>> >>> [4115] https://github.com/apache/polaris/pull/4115
>> >>>
>> >>> Thanks,
>> >>> Dmitri.
>> >>>
>> >>> On Tue, Mar 3, 2026 at 3:29 PM Anand Kumar Sankaran via dev <
>> >>> dev@polaris.apache.org> wrote:
>> >>>
>> >>> > Hi Yufei and Dmitri,
>> >>> >
>> >>> > Here is a proposal for the REST endpoints for metrics and events.
>> >>> >
>> >>> > https://github.com/apache/polaris/pull/3924/changes
>> >>> >
>> >>> > I did not see any precursors for raising a PR for proposals, so
>> trying
>> >>> > this.  Please let me know what you think.
>> >>> >
>> >>> > -
>> >>> > Anand
>> >>> >
>> >>> > From: Anand Kumar Sankaran <an...@workday.com>
>> >>> > Date: Monday, March 2, 2026 at 10:25 AM
>> >>> > To: dev@polaris.apache.org <de...@polaris.apache.org>
>> >>> > Subject: Re: Polaris Telemetry and Audit Trail
>> >>> >
>> >>> > About the REST API, based on my use cases:
>> >>> >
>> >>> >
>> >>> >   1.
>> >>> > I want to be able to query commit metrics to track files added /
>> >>> removed
>> >>> > per commit, along with record counts. The ingestion pipeline that
>> >>> writes
>> >>> > this data is owned by us and we are guaranteed to write this
>> >>> information
>> >>> > for each write.
>> >>> >   2.
>> >>> > I want to be able to query scan metrics for read. I understand
>> clients
>> >>> do
>> >>> > not fulfill this requirement.
>> >>> >   3.
>> >>> > I want to be able to query the events table (events are persisted) -
>> >>> this
>> >>> > may supersede #2, I am not sure yet.
>> >>> >
>> >>> > All this information is in the JDBC based persistence model and is
>> >>> > persisted in the metastore. I currently don’t have a need to query
>> >>> > prometheus or open telemetry. I do publish some events to Prometheus
>> >>> and
>> >>> > they are forwarded to our dashboards elsewhere.
>> >>> >
>> >>> > About the CLI utilities, I meant the admin user utilities. In one of
>> >>> the
>> >>> > earliest drafts of my proposal, Prashant mentioned that the metrics
>> >>> tables
>> >>> > can grow indefinitely and that a similar problem exists with the
>> events
>> >>> > table as well. We discussed that cleaning up of old records from
>> both
>> >>> > metrics tables and events tables can be done via a CLI utility.
>> >>> >
>> >>> > I see that Yufei has covered the discussion about datasources.
>> >>> >
>> >>> > -
>> >>> > Anand
>> >>> >
>> >>> >
>> >>> >
>> >>> > From: Yufei Gu <fl...@gmail.com>
>> >>> > Date: Friday, February 27, 2026 at 9:54 PM
>> >>> > To: dev@polaris.apache.org <de...@polaris.apache.org>
>> >>> > Subject: Re: Polaris Telemetry and Audit Trail
>> >>> >
>> >>> > This Message Is From an External Sender
>> >>> > This message came from outside your organization.
>> >>> > Report Suspicious<
>> >>> >
>> >>>
>> https://us-phishalarm-ewt.proofpoint.com/EWT/v1/Iz9xO38YGHZK!YhNDZABkHi1B699ote2uMwpOZw8i0QMCGO2Szc-HshuABGhGvwPJcymE6G2oUUxtS8xDkSrtGTPm_I3QnVDHoLMk50m9v8z_nZKTkd-bnVUbreF1u0WnfV_X5eYevZl_$
>> >>> > >
>> >>> >
>> >>> >
>> >>> > As I mentioned in
>> >>> >
>> >>>
>> https://urldefense.com/v3/__https://github.com/apache/polaris/issues/3890__;!!Iz9xO38YGHZK!5EuyFFkk3vhRWVIRvQAWBSQfpJkTMA9HxugzDwXmN0LPPqhEFxYkFRGVhtb8AqUwXtDh2OplcMnbMDHKOxrvDU0$
>> >>> ,
>> >>> > supporting
>> >>> > multiple data sources is not a trivial change. I would strongly
>> >>> recommend
>> >>> > starting with a design document to carefully evaluate the
>> architectural
>> >>> > implications and long term impact.
>> >>> >
>> >>> > A REST endpoint to query metrics seems reasonable given the current
>> >>> JDBC
>> >>> > based persistence model. That said, we may also consider alternative
>> >>> > storage models. For example, if we later adopt a time series system
>> >>> such as
>> >>> > Prometheus to store metrics, the query model and access patterns
>> would
>> >>> be
>> >>> > fundamentally different. Designing the REST API without considering
>> >>> these
>> >>> > potential evolutions may limit flexibility. I'd suggest to start
>> with
>> >>> the
>> >>> > use case.
>> >>> >
>> >>> > Yufei
>> >>> >
>> >>> >
>> >>> > On Fri, Feb 27, 2026 at 3:42 PM Dmitri Bourlatchkov <
>> dimas@apache.org>
>> >>> > wrote:
>> >>> >
>> >>> > > Hi Anand,
>> >>> > >
>> >>> > > Sharing my view... subject to discussion:
>> >>> > >
>> >>> > > 1. Adding non-IRC REST API to Polaris is perfectly fine.
>> >>> > >
>> >>> > > Figuring out specific endpoint URIs and payloads might require a
>> few
>> >>> > > roundtrips, so opening a separate thread for that might be best.
>> >>> > > Contributors commonly create Google Docs for new API proposals too
>> >>> (they
>> >>> > > fairly easy to update as the email discussion progresses).
>> >>> > >
>> >>> > > There was a suggestion to try Markdown (with PRs) for proposals
>> [1]
>> >>> ...
>> >>> > > feel free to give it a try if you are comfortable with that.
>> >>> > >
>> >>> > > 2. Could you clarify whether you mean end user utilities or admin
>> >>> user
>> >>> > > utilities? In the latter case those might be more suitable for the
>> >>> Admin
>> >>> > > CLI (java) not the Python CLI, IMHO.
>> >>> > >
>> >>> > > Why would these utilities be common with events? IMHO, event use
>> >>> cases
>> >>> > are
>> >>> > > distinct from scan/commit metrics.
>> >>> > >
>> >>> > > 3. I'd prefer separating metrics persistence from MetaStore
>> >>> persistence
>> >>> > at
>> >>> > > the code level, so that they could be mixed and matched
>> >>> independently.
>> >>> > The
>> >>> > > separate datasource question will become a non-issue with that
>> >>> approach,
>> >>> > I
>> >>> > > guess.
>> >>> > >
>> >>> > > The rationale for separating scan metrics and metastore
>> persistence
>> >>> is
>> >>> > that
>> >>> > > "cascading deletes" between them are hardly ever required.
>> >>> Furthermore,
>> >>> > the
>> >>> > > data and query patterns are very different so different
>> technologies
>> >>> > might
>> >>> > > be beneficial in each case.
>> >>> > >
>> >>> > > [1]
>> >>> >
>> >>>
>> https://urldefense.com/v3/__https://lists.apache.org/thread/yto2wp982t43h1mqjwnslswhws5z47cy__;!!Iz9xO38YGHZK!5EuyFFkk3vhRWVIRvQAWBSQfpJkTMA9HxugzDwXmN0LPPqhEFxYkFRGVhtb8AqUwXtDh2OplcMnbMDHKxYDakNU$
>> >>> > >
>> >>> > > Cheers,
>> >>> > > Dmitri.
>> >>> > >
>> >>> > > On Fri, Feb 27, 2026 at 6:19 PM Anand Kumar Sankaran via dev <
>> >>> > > dev@polaris.apache.org> wrote:
>> >>> > >
>> >>> > > > Thanks all. This PR is merged now.
>> >>> > > >
>> >>> > > > Here are the follow-up features / work needed.  These were all
>> >>> part of
>> >>> > > the
>> >>> > > > merged PR at some point in time and were removed to reduce
>> scope.
>> >>> > > >
>> >>> > > > Please let me know what you think.
>> >>> > > >
>> >>> > > >
>> >>> > > >   1.  A REST API to paginate through table metrics. This will be
>> >>> > non-IRC
>> >>> > > > standard addition.
>> >>> > > >   2.  Utilities for managing old records, should be common with
>> >>> events.
>> >>> > > > There was some discussion that it belongs to the CLI.
>> >>> > > >   3.  Separate datasource (metrics, events, even other tables?).
>> >>> > > >
>> >>> > > >
>> >>> > > > Anything else?
>> >>> > > >
>> >>> > > > -
>> >>> > > > Anand
>> >>> > > >
>> >>> > > >
>> >>> > >
>> >>> >
>> >>> >
>> >>>
>> >>
>>
>

Re: Proposal for REST endpoints for table metrics and events

[EJ Wang <ej...@gmail.com>]

Thanks Yufei for the +1.

JB, could you help add a biweekly metrics architecture sync to the Polaris
community calendar? I'm thinking Thursdays at 9-10am PT, on the off-weeks
from the community meeting (starting May 7), 60 minutes.

Here's a rough agenda to work through over the first few sessions, grouped
by priority:

*First: foundational direction*

1.  MetricsPersistence: public SPI or internal implementation detail?
   •   Marked @Beta, javadoc calls it a "Service Provider Interface", but
only one consumer (JdbcBasePersistenceImpl), lives on BasePersistence. If
demoted to a private helper inside a persisting reporter impl, most
downstream design decisions become implementation details rather than
contract questions.

2.  Persistence schema redesign
   •   Current two-table layout (scan_metrics_report,
commit_metrics_report) with ~25 flattened columns each. Every new metric
type requires a new table, SPI method, record class, model, converter, and
schema migration. Direction to explore: single table with metric_type enum,
schema_version, and JSON payload column.

*Second: design details once direction is set*

3.  Partition key strategy
   •   Single-table design means scan metrics at scale will have high write
concurrency per table. Schema needs to expose enough structure for backends
to shard by entity or time range.

4.  Read/write path consistency
   •   Writes go through PolarisMetricsManager on MetaStoreManager. Reads
bypass MetaStoreManager and go straight to BasePersistence, excluding
non-JDBC backends from the read API.

*Third: cleanup and alignment*

5.  PolarisMetricsReporter naming
   •   Only handles IRC (ScanReport/CommitReport), doesn't cover generic
tables or operational metrics. Name is broader than scope.

6.  PolarisMetricsManager facade passthrough
   •   Entire default method is callCtx.getMetaStore().writeScanReport().
Zero logic, passes Level 1 straight through to Level 3. Same anti-pattern
as PolarisEventManager.

7.  Iceberg community alignment
   •   Payload-type extension needs discussion on dev@iceberg. obelix74's
Feb thread got zero replies. Needs a committer voice.

Lets confirm prioritization in the first session.

-ej

On Tue, Apr 21, 2026 at 3:18 PM Yufei Gu <fl...@gmail.com> wrote:

> Thanks everyone for continuing to drive this forward. I agree that the
> problem is getting complex enough that a more structured discussion would
> help.
>
> +1 on setting up a biweekly sync for the metrics architecture. I’m happy to
> join.
>
> Yufei
>
>
> On Tue, Apr 21, 2026 at 2:34 PM EJ Wang <ej...@gmail.com>
> wrote:
>
> > Also, I've been looking more closely at the *persistence schema in the
> > current metrics work*, and I think there's a structural rigidity problem
> > worth raising before the shape gets locked in.
> >
> > Right now we have two separate tables (scan_metrics_report and
> > commit_metrics_report), each with ~25 flattened columns that directly
> > mirror the Iceberg report fields. The SPI follows the same split:
> > writeScanReport and writeCommitReport as separate methods, with per-type
> > record classes, converters, and model objects. *The practical cost:
> > adding a new metric type (operational metrics, for example) requires a
> new
> > table, a new SPI method, a new record class, a new model class, a new
> > converter branch, and a schema migration*. That's a lot of surface area
> > for what should be "one more kind of metric."
> >
> > *My bias* would be toward a single metrics table with *a typed JSON
> > payload*. Something like: metric_type (enum), entity_id,
> > table_identifier, snapshot_id (nullable), received_ts, schema_version,
> and
> > a payload column for the metric-specific data. The metric_type +
> > schema_version pair gives us a forward-compatible contract for the
> payload
> > shape. Adding a new metric type becomes an enum value and a payload
> schema,
> > not a schema migration. One thing I think we need to be deliberate about
> is
> > the partition key design. If all metric types land in one table, scan
> > metrics at scale (high concurrency, high frequency across many tables)
> > could easily create hot partitions. We'd want the persistence layer to be
> > able to shard by entity or time range, and that means the logical schema
> > needs to expose enough structure for backends to partition on. I don't
> > think the current flattened layout gives us that.
> >
> > This is getting complex enough that I don't think ad-hoc PR/ML threads
> > will converge well. *Would people be open to a biweekly sync for metrics
> > architecture?* I think 30 minutes every two weeks with interested parties
> > would be enough to work through the schema, SPI shape, and read API
> design
> > together. Happy to help set that up.
> >
> > -ej
> >
> > On Mon, Apr 20, 2026 at 2:19 PM EJ Wang <ej...@gmail.com>
> > wrote:
> >
> >> Reviewed #4115, left a comment on the code organization side.
> >>
> >> One thing stood out: the metrics write path enters through
> >> PolarisMetricsManager on MetaStoreManager, but the new read path
> bypasses
> >> MetaStoreManager entirely and goes straight to BasePersistence via
> >> callContext.getMetaStore(). That means the read API only works for
> backends
> >> that implement BasePersistence. NoSQL and remote backends can't
> participate.
> >>
> >> Stepping back, I think the metrics subsystem is growing into something
> >> real (write + read + REST API + AuthZ + pagination) *but the persistence
> >> side is split across two layers in a way that's hard to extend*. I put
> >> together two diagrams to show what I mean (my best effort).
> >>
> >> *Current state* (Diagram 1): three interfaces at three different levels.
> >> The engine-facing SPI (PolarisMetricsReporter) is clean. But
> >> PolarisMetricsManager on MetaStoreManager is a passthrough to
> >> MetricsPersistence on BasePersistence. The @Beta annotation and SPI
> javadoc
> >> are on the BasePersistence layer, while the actual extension points
> >> (PolarisMetricsReporter, PolarisMetricsManager) carry no stability
> >> annotation. The write path goes through the MetaStoreManager layer, the
> >> read path doesn't.
> >>
> >> *What I envision* (Diagram 2): two SPIs at two levels.
> >> PolarisMetricsReporter stays as the engine-facing SPI.
> >> PolarisMetricsManager becomes the backend-facing SPI with both write and
> >> read methods at the MetaStoreManager level, where any backend (JDBC,
> NoSQL,
> >> remote) can implement them. MetricsPersistence on BasePersistence goes
> >> away. Where metrics actually land is an implementation detail, not a
> core
> >> interface.
> >>
> >> *Minor naming thing*: PolarisMetricsReporter is broader than what it
> >> actually handles. It only accepts Iceberg REST Catalog metrics
> (ScanReport,
> >> CommitReport via MetricsReport). Generic table metrics or operational
> >> metrics aren't in scope. Not blocking, but worth noting if the metrics
> >> surface expands.
> >>
> >> *Rough sketch of how to get there*:
> >>  1.  Add read methods to PolarisMetricsManager (listScanReports,
> >> listCommitReports) with default no-op, same as the existing write
> methods.
> >> (Probably make PolarisMetricsManager more explicit on being Iceberg
> >> specific like package name or class name etc.)
> >>  2.  Wire MetricsReportsService through MetaStoreManager instead of
> >> callContext.getMetaStore().
> >>  3.  Extract metrics persistence from JdbcBasePersistenceImpl into its
> >> own class. That file carries ~7 responsibilities, metrics being one of
> them.
> >>  4.  Remove MetricsPersistence from BasePersistence.
> >>
> >> *None of this needs to happen in #4115. But if the direction makes
> sense,
> >> it would be good to align before the metrics surface grows further.
> Curious
> >> what others think.*
> >>
> >> *My mental model note*: Level 1 MetaStoreManager; level 2 transactional
> >> persistence; level 3 base persistence
> >>
> >> Diagram 1
> >> <
> https://www.plantuml.com/plantuml/uml/bLHDR-Cs4BthLmpIYupw0zbkKQ1r3M-S7Bp8xhhM7WCOb3IM65EaGD9EX2RzxHrHb4CxRelwa4YSDu_lpOVcnZ9jzvM8BBS2uGjQpJC3dtHMSekPtMk44IpsMgEqa5XcCOhCZikQQLP1pR8TAp2n3ILhmZDP20m0fcIvUkAoW2qJXd9z1bpToO9BX3WXu0ucy5rpgGPNm0nW5_epUWtm2Ue3pn3kMOFQmKntGZW0BYtgBSi8k5A2QMwybJNMIbFiGSR9QZc4nUqIvikStF0jHprua5C-amge42aNt3R0f5JaaoivdV2Pkqbx4hee4ymOkBh5BTiB-_uIeGeo8zL8rPsPl4DktdEiK1jkB1NdZCRbrSTecDe_mlHbF0wvBmCkaOH5_S8a_TTTKI6-nmCAkEw4LpxsZ-LbYLKQFKMNOgf_wuM7_bV9gOer5SYMMksBSWXFcbi49KNZXNLicwfe3TETC7gPdPqI7uBcHMb1RSzYq34c6PDUM9mn8HRsUTZEiDBve3NjVZumBj0U7SS37mGO7vcwtiK-_pU7U7L_f-digo9YbhSwIfMRwIITKGXbxdIUTCGF1SeCJxloKsU-3k9ddRbX1eDq1q_fx1JbBGT0glVyXimDuP4TQ5qpCAmnGEj2s_6n5mtn1z-97-63itFQZLPO1Ev2tu_WF7Ju-VPc0Skg5bYXxBhkY1xpD7EM_7fyflSpIsqMgVth5xhVr4eQxWQ8enaSAJQSG16yFSDuJ798rrcXr_3n-lfdk7icQjEBmFujL7AodiP_Y4Z7-YxvtZNs4zMgpNTl6tF8sglyPsmqchrjvQ-m-aP94r-TwCA2Ka8upPJZwtvSpoYCXkYMZU2NXvRMBfq9P3i3Le4VAZUAlUZ_oPKsxPgY0Q_BSKLkyr9bhQhQrJjo_x3TPlIB0DPjnMfcIoYP0QaYw1a0fTKDr8fB6ntNuvmoL1ZGkXa69Njh43zf9GiGxHQrA_jDYWRSzF5--WmTVrN97_Sm8LbLUy_lGBmLanJjFkDlGkRqjA_4tm00
> >
> >> :
> >>
> >> [image: image.png]
> >>
> >> Diagram 2
> >> <
> https://www.plantuml.com/plantuml/uml/VLLDR-8m4BtdLupO2sWBLVU8AaGB7AXAbssGzb896SS9RXqxjKqBwkv_tt7iV43fdaZYDpFlpRmnOsE9jhjSH9PRmM31hERKm8scMsuPjJlDe0yheZDc8RR4iYWoBrmMH9CS2a9VICPYUy1OZN0YCy5Q0BCbYNhdCeEK28En8G8wCvbnoQ0R8_05Bc6bkLIz3X03p1zzH7zR-9ZfDquPt9C3qoNCX2yV4G2NbkcKu5jdgGJHt0GbZwnG6i-UP3TUpk5gM6Ldqke350eZUqzoCft3U9xWHvxoa5-7K4nF1J46EbEMafsmdrCBbQ44gVggy18IZrn_ph5asd1ZiIKdQSgueZvjXrQFSFrdC3YN-nXmBacxbGiYyLVxLaBtdhqn0LSzdBDhqQtQoOJeGyad3z0lUqnYgpGB6Ns8oVyta00Dy_WnX0tIOZ8v6SYxHll1TrH6aejAik-mh-AphVFCwSUQqFypElag5QRGFDjQKEd96K1P8QP41c9TzA_IIQyvdAWyv_RSiS3skb0_EzDDkK2v5xWF6MiGFlvhpFLcD2Dq2pml14gaF67eQkmd8gulDoC4kSOu6KVpkvlUJg1RTbWISU40RdBUUS_9XfRZ2dwxm_SW8LYFISgm_MnlDQ6M9P1gbKEc4X-2pH_FvJCkCqm9pbVjD6LrwdLeOrDWfOaqc8Wh9BE85oNKxkNQ6o4yGRy_Eae0G_G8tZv81d3bHDB23WOdisohVr3nh_j6lbSjbNaLRTc8UgtPbAU1J_tygOfZX9DWEJeHDvYx-qmSi5FgNLPZwHrHcUsncGQ5-skhUclpE5fo4ounpFauYrUbkU6ccfnxMvitwag4IyerhTxj8In_Oj1bDO4pQru674loYrGlULHLEGCjwJJ8gDoVZR8MxO4BT3IzRvIcAQKezC6xpziGnTyImrfEGyJI_OcKfgtxIvnTqFEMS17L9Z-jsARN5FmTheP7HtSdtOMT0B4GY2FYHXxgQmMtj2bRqiLFGapiVe1_QVKDrkqXcm83aFEXnMYCZ-xlyHy
> >
> >> :
> >> [image: image.png]
> >>
> >>  -ej
> >>
> >> On Wed, Apr 15, 2026 at 8:22 AM Dmitri Bourlatchkov <di...@apache.org>
> >> wrote:
> >>
> >>> Hi All,
> >>>
> >>> Heads up: The current state of PR [4115] looks pretty solid to me. I
> >>> believe this PR is approaching a mergeable condition.
> >>>
> >>> Please post your reviews if you have any comments.
> >>>
> >>> [4115] https://github.com/apache/polaris/pull/4115
> >>>
> >>> Thanks,
> >>> Dmitri.
> >>>
> >>> On Tue, Mar 3, 2026 at 3:29 PM Anand Kumar Sankaran via dev <
> >>> dev@polaris.apache.org> wrote:
> >>>
> >>> > Hi Yufei and Dmitri,
> >>> >
> >>> > Here is a proposal for the REST endpoints for metrics and events.
> >>> >
> >>> > https://github.com/apache/polaris/pull/3924/changes
> >>> >
> >>> > I did not see any precursors for raising a PR for proposals, so
> trying
> >>> > this.  Please let me know what you think.
> >>> >
> >>> > -
> >>> > Anand
> >>> >
> >>> > From: Anand Kumar Sankaran <an...@workday.com>
> >>> > Date: Monday, March 2, 2026 at 10:25 AM
> >>> > To: dev@polaris.apache.org <de...@polaris.apache.org>
> >>> > Subject: Re: Polaris Telemetry and Audit Trail
> >>> >
> >>> > About the REST API, based on my use cases:
> >>> >
> >>> >
> >>> >   1.
> >>> > I want to be able to query commit metrics to track files added /
> >>> removed
> >>> > per commit, along with record counts. The ingestion pipeline that
> >>> writes
> >>> > this data is owned by us and we are guaranteed to write this
> >>> information
> >>> > for each write.
> >>> >   2.
> >>> > I want to be able to query scan metrics for read. I understand
> clients
> >>> do
> >>> > not fulfill this requirement.
> >>> >   3.
> >>> > I want to be able to query the events table (events are persisted) -
> >>> this
> >>> > may supersede #2, I am not sure yet.
> >>> >
> >>> > All this information is in the JDBC based persistence model and is
> >>> > persisted in the metastore. I currently don’t have a need to query
> >>> > prometheus or open telemetry. I do publish some events to Prometheus
> >>> and
> >>> > they are forwarded to our dashboards elsewhere.
> >>> >
> >>> > About the CLI utilities, I meant the admin user utilities. In one of
> >>> the
> >>> > earliest drafts of my proposal, Prashant mentioned that the metrics
> >>> tables
> >>> > can grow indefinitely and that a similar problem exists with the
> events
> >>> > table as well. We discussed that cleaning up of old records from both
> >>> > metrics tables and events tables can be done via a CLI utility.
> >>> >
> >>> > I see that Yufei has covered the discussion about datasources.
> >>> >
> >>> > -
> >>> > Anand
> >>> >
> >>> >
> >>> >
> >>> > From: Yufei Gu <fl...@gmail.com>
> >>> > Date: Friday, February 27, 2026 at 9:54 PM
> >>> > To: dev@polaris.apache.org <de...@polaris.apache.org>
> >>> > Subject: Re: Polaris Telemetry and Audit Trail
> >>> >
> >>> > This Message Is From an External Sender
> >>> > This message came from outside your organization.
> >>> > Report Suspicious<
> >>> >
> >>>
> https://us-phishalarm-ewt.proofpoint.com/EWT/v1/Iz9xO38YGHZK!YhNDZABkHi1B699ote2uMwpOZw8i0QMCGO2Szc-HshuABGhGvwPJcymE6G2oUUxtS8xDkSrtGTPm_I3QnVDHoLMk50m9v8z_nZKTkd-bnVUbreF1u0WnfV_X5eYevZl_$
> >>> > >
> >>> >
> >>> >
> >>> > As I mentioned in
> >>> >
> >>>
> https://urldefense.com/v3/__https://github.com/apache/polaris/issues/3890__;!!Iz9xO38YGHZK!5EuyFFkk3vhRWVIRvQAWBSQfpJkTMA9HxugzDwXmN0LPPqhEFxYkFRGVhtb8AqUwXtDh2OplcMnbMDHKOxrvDU0$
> >>> ,
> >>> > supporting
> >>> > multiple data sources is not a trivial change. I would strongly
> >>> recommend
> >>> > starting with a design document to carefully evaluate the
> architectural
> >>> > implications and long term impact.
> >>> >
> >>> > A REST endpoint to query metrics seems reasonable given the current
> >>> JDBC
> >>> > based persistence model. That said, we may also consider alternative
> >>> > storage models. For example, if we later adopt a time series system
> >>> such as
> >>> > Prometheus to store metrics, the query model and access patterns
> would
> >>> be
> >>> > fundamentally different. Designing the REST API without considering
> >>> these
> >>> > potential evolutions may limit flexibility. I'd suggest to start with
> >>> the
> >>> > use case.
> >>> >
> >>> > Yufei
> >>> >
> >>> >
> >>> > On Fri, Feb 27, 2026 at 3:42 PM Dmitri Bourlatchkov <
> dimas@apache.org>
> >>> > wrote:
> >>> >
> >>> > > Hi Anand,
> >>> > >
> >>> > > Sharing my view... subject to discussion:
> >>> > >
> >>> > > 1. Adding non-IRC REST API to Polaris is perfectly fine.
> >>> > >
> >>> > > Figuring out specific endpoint URIs and payloads might require a
> few
> >>> > > roundtrips, so opening a separate thread for that might be best.
> >>> > > Contributors commonly create Google Docs for new API proposals too
> >>> (they
> >>> > > fairly easy to update as the email discussion progresses).
> >>> > >
> >>> > > There was a suggestion to try Markdown (with PRs) for proposals [1]
> >>> ...
> >>> > > feel free to give it a try if you are comfortable with that.
> >>> > >
> >>> > > 2. Could you clarify whether you mean end user utilities or admin
> >>> user
> >>> > > utilities? In the latter case those might be more suitable for the
> >>> Admin
> >>> > > CLI (java) not the Python CLI, IMHO.
> >>> > >
> >>> > > Why would these utilities be common with events? IMHO, event use
> >>> cases
> >>> > are
> >>> > > distinct from scan/commit metrics.
> >>> > >
> >>> > > 3. I'd prefer separating metrics persistence from MetaStore
> >>> persistence
> >>> > at
> >>> > > the code level, so that they could be mixed and matched
> >>> independently.
> >>> > The
> >>> > > separate datasource question will become a non-issue with that
> >>> approach,
> >>> > I
> >>> > > guess.
> >>> > >
> >>> > > The rationale for separating scan metrics and metastore persistence
> >>> is
> >>> > that
> >>> > > "cascading deletes" between them are hardly ever required.
> >>> Furthermore,
> >>> > the
> >>> > > data and query patterns are very different so different
> technologies
> >>> > might
> >>> > > be beneficial in each case.
> >>> > >
> >>> > > [1]
> >>> >
> >>>
> https://urldefense.com/v3/__https://lists.apache.org/thread/yto2wp982t43h1mqjwnslswhws5z47cy__;!!Iz9xO38YGHZK!5EuyFFkk3vhRWVIRvQAWBSQfpJkTMA9HxugzDwXmN0LPPqhEFxYkFRGVhtb8AqUwXtDh2OplcMnbMDHKxYDakNU$
> >>> > >
> >>> > > Cheers,
> >>> > > Dmitri.
> >>> > >
> >>> > > On Fri, Feb 27, 2026 at 6:19 PM Anand Kumar Sankaran via dev <
> >>> > > dev@polaris.apache.org> wrote:
> >>> > >
> >>> > > > Thanks all. This PR is merged now.
> >>> > > >
> >>> > > > Here are the follow-up features / work needed.  These were all
> >>> part of
> >>> > > the
> >>> > > > merged PR at some point in time and were removed to reduce scope.
> >>> > > >
> >>> > > > Please let me know what you think.
> >>> > > >
> >>> > > >
> >>> > > >   1.  A REST API to paginate through table metrics. This will be
> >>> > non-IRC
> >>> > > > standard addition.
> >>> > > >   2.  Utilities for managing old records, should be common with
> >>> events.
> >>> > > > There was some discussion that it belongs to the CLI.
> >>> > > >   3.  Separate datasource (metrics, events, even other tables?).
> >>> > > >
> >>> > > >
> >>> > > > Anything else?
> >>> > > >
> >>> > > > -
> >>> > > > Anand
> >>> > > >
> >>> > > >
> >>> > >
> >>> >
> >>> >
> >>>
> >>
>

Re: Proposal for REST endpoints for table metrics and events

[Yufei Gu <fl...@gmail.com>]

Thanks everyone for continuing to drive this forward. I agree that the
problem is getting complex enough that a more structured discussion would
help.

+1 on setting up a biweekly sync for the metrics architecture. I’m happy to
join.

Yufei


On Tue, Apr 21, 2026 at 2:34 PM EJ Wang <ej...@gmail.com>
wrote:

> Also, I've been looking more closely at the *persistence schema in the
> current metrics work*, and I think there's a structural rigidity problem
> worth raising before the shape gets locked in.
>
> Right now we have two separate tables (scan_metrics_report and
> commit_metrics_report), each with ~25 flattened columns that directly
> mirror the Iceberg report fields. The SPI follows the same split:
> writeScanReport and writeCommitReport as separate methods, with per-type
> record classes, converters, and model objects. *The practical cost:
> adding a new metric type (operational metrics, for example) requires a new
> table, a new SPI method, a new record class, a new model class, a new
> converter branch, and a schema migration*. That's a lot of surface area
> for what should be "one more kind of metric."
>
> *My bias* would be toward a single metrics table with *a typed JSON
> payload*. Something like: metric_type (enum), entity_id,
> table_identifier, snapshot_id (nullable), received_ts, schema_version, and
> a payload column for the metric-specific data. The metric_type +
> schema_version pair gives us a forward-compatible contract for the payload
> shape. Adding a new metric type becomes an enum value and a payload schema,
> not a schema migration. One thing I think we need to be deliberate about is
> the partition key design. If all metric types land in one table, scan
> metrics at scale (high concurrency, high frequency across many tables)
> could easily create hot partitions. We'd want the persistence layer to be
> able to shard by entity or time range, and that means the logical schema
> needs to expose enough structure for backends to partition on. I don't
> think the current flattened layout gives us that.
>
> This is getting complex enough that I don't think ad-hoc PR/ML threads
> will converge well. *Would people be open to a biweekly sync for metrics
> architecture?* I think 30 minutes every two weeks with interested parties
> would be enough to work through the schema, SPI shape, and read API design
> together. Happy to help set that up.
>
> -ej
>
> On Mon, Apr 20, 2026 at 2:19 PM EJ Wang <ej...@gmail.com>
> wrote:
>
>> Reviewed #4115, left a comment on the code organization side.
>>
>> One thing stood out: the metrics write path enters through
>> PolarisMetricsManager on MetaStoreManager, but the new read path bypasses
>> MetaStoreManager entirely and goes straight to BasePersistence via
>> callContext.getMetaStore(). That means the read API only works for backends
>> that implement BasePersistence. NoSQL and remote backends can't participate.
>>
>> Stepping back, I think the metrics subsystem is growing into something
>> real (write + read + REST API + AuthZ + pagination) *but the persistence
>> side is split across two layers in a way that's hard to extend*. I put
>> together two diagrams to show what I mean (my best effort).
>>
>> *Current state* (Diagram 1): three interfaces at three different levels.
>> The engine-facing SPI (PolarisMetricsReporter) is clean. But
>> PolarisMetricsManager on MetaStoreManager is a passthrough to
>> MetricsPersistence on BasePersistence. The @Beta annotation and SPI javadoc
>> are on the BasePersistence layer, while the actual extension points
>> (PolarisMetricsReporter, PolarisMetricsManager) carry no stability
>> annotation. The write path goes through the MetaStoreManager layer, the
>> read path doesn't.
>>
>> *What I envision* (Diagram 2): two SPIs at two levels.
>> PolarisMetricsReporter stays as the engine-facing SPI.
>> PolarisMetricsManager becomes the backend-facing SPI with both write and
>> read methods at the MetaStoreManager level, where any backend (JDBC, NoSQL,
>> remote) can implement them. MetricsPersistence on BasePersistence goes
>> away. Where metrics actually land is an implementation detail, not a core
>> interface.
>>
>> *Minor naming thing*: PolarisMetricsReporter is broader than what it
>> actually handles. It only accepts Iceberg REST Catalog metrics (ScanReport,
>> CommitReport via MetricsReport). Generic table metrics or operational
>> metrics aren't in scope. Not blocking, but worth noting if the metrics
>> surface expands.
>>
>> *Rough sketch of how to get there*:
>>  1.  Add read methods to PolarisMetricsManager (listScanReports,
>> listCommitReports) with default no-op, same as the existing write methods.
>> (Probably make PolarisMetricsManager more explicit on being Iceberg
>> specific like package name or class name etc.)
>>  2.  Wire MetricsReportsService through MetaStoreManager instead of
>> callContext.getMetaStore().
>>  3.  Extract metrics persistence from JdbcBasePersistenceImpl into its
>> own class. That file carries ~7 responsibilities, metrics being one of them.
>>  4.  Remove MetricsPersistence from BasePersistence.
>>
>> *None of this needs to happen in #4115. But if the direction makes sense,
>> it would be good to align before the metrics surface grows further. Curious
>> what others think.*
>>
>> *My mental model note*: Level 1 MetaStoreManager; level 2 transactional
>> persistence; level 3 base persistence
>>
>> Diagram 1
>> <https://www.plantuml.com/plantuml/uml/bLHDR-Cs4BthLmpIYupw0zbkKQ1r3M-S7Bp8xhhM7WCOb3IM65EaGD9EX2RzxHrHb4CxRelwa4YSDu_lpOVcnZ9jzvM8BBS2uGjQpJC3dtHMSekPtMk44IpsMgEqa5XcCOhCZikQQLP1pR8TAp2n3ILhmZDP20m0fcIvUkAoW2qJXd9z1bpToO9BX3WXu0ucy5rpgGPNm0nW5_epUWtm2Ue3pn3kMOFQmKntGZW0BYtgBSi8k5A2QMwybJNMIbFiGSR9QZc4nUqIvikStF0jHprua5C-amge42aNt3R0f5JaaoivdV2Pkqbx4hee4ymOkBh5BTiB-_uIeGeo8zL8rPsPl4DktdEiK1jkB1NdZCRbrSTecDe_mlHbF0wvBmCkaOH5_S8a_TTTKI6-nmCAkEw4LpxsZ-LbYLKQFKMNOgf_wuM7_bV9gOer5SYMMksBSWXFcbi49KNZXNLicwfe3TETC7gPdPqI7uBcHMb1RSzYq34c6PDUM9mn8HRsUTZEiDBve3NjVZumBj0U7SS37mGO7vcwtiK-_pU7U7L_f-digo9YbhSwIfMRwIITKGXbxdIUTCGF1SeCJxloKsU-3k9ddRbX1eDq1q_fx1JbBGT0glVyXimDuP4TQ5qpCAmnGEj2s_6n5mtn1z-97-63itFQZLPO1Ev2tu_WF7Ju-VPc0Skg5bYXxBhkY1xpD7EM_7fyflSpIsqMgVth5xhVr4eQxWQ8enaSAJQSG16yFSDuJ798rrcXr_3n-lfdk7icQjEBmFujL7AodiP_Y4Z7-YxvtZNs4zMgpNTl6tF8sglyPsmqchrjvQ-m-aP94r-TwCA2Ka8upPJZwtvSpoYCXkYMZU2NXvRMBfq9P3i3Le4VAZUAlUZ_oPKsxPgY0Q_BSKLkyr9bhQhQrJjo_x3TPlIB0DPjnMfcIoYP0QaYw1a0fTKDr8fB6ntNuvmoL1ZGkXa69Njh43zf9GiGxHQrA_jDYWRSzF5--WmTVrN97_Sm8LbLUy_lGBmLanJjFkDlGkRqjA_4tm00>
>> :
>>
>> [image: image.png]
>>
>> Diagram 2
>> <https://www.plantuml.com/plantuml/uml/VLLDR-8m4BtdLupO2sWBLVU8AaGB7AXAbssGzb896SS9RXqxjKqBwkv_tt7iV43fdaZYDpFlpRmnOsE9jhjSH9PRmM31hERKm8scMsuPjJlDe0yheZDc8RR4iYWoBrmMH9CS2a9VICPYUy1OZN0YCy5Q0BCbYNhdCeEK28En8G8wCvbnoQ0R8_05Bc6bkLIz3X03p1zzH7zR-9ZfDquPt9C3qoNCX2yV4G2NbkcKu5jdgGJHt0GbZwnG6i-UP3TUpk5gM6Ldqke350eZUqzoCft3U9xWHvxoa5-7K4nF1J46EbEMafsmdrCBbQ44gVggy18IZrn_ph5asd1ZiIKdQSgueZvjXrQFSFrdC3YN-nXmBacxbGiYyLVxLaBtdhqn0LSzdBDhqQtQoOJeGyad3z0lUqnYgpGB6Ns8oVyta00Dy_WnX0tIOZ8v6SYxHll1TrH6aejAik-mh-AphVFCwSUQqFypElag5QRGFDjQKEd96K1P8QP41c9TzA_IIQyvdAWyv_RSiS3skb0_EzDDkK2v5xWF6MiGFlvhpFLcD2Dq2pml14gaF67eQkmd8gulDoC4kSOu6KVpkvlUJg1RTbWISU40RdBUUS_9XfRZ2dwxm_SW8LYFISgm_MnlDQ6M9P1gbKEc4X-2pH_FvJCkCqm9pbVjD6LrwdLeOrDWfOaqc8Wh9BE85oNKxkNQ6o4yGRy_Eae0G_G8tZv81d3bHDB23WOdisohVr3nh_j6lbSjbNaLRTc8UgtPbAU1J_tygOfZX9DWEJeHDvYx-qmSi5FgNLPZwHrHcUsncGQ5-skhUclpE5fo4ounpFauYrUbkU6ccfnxMvitwag4IyerhTxj8In_Oj1bDO4pQru674loYrGlULHLEGCjwJJ8gDoVZR8MxO4BT3IzRvIcAQKezC6xpziGnTyImrfEGyJI_OcKfgtxIvnTqFEMS17L9Z-jsARN5FmTheP7HtSdtOMT0B4GY2FYHXxgQmMtj2bRqiLFGapiVe1_QVKDrkqXcm83aFEXnMYCZ-xlyHy>
>> :
>> [image: image.png]
>>
>>  -ej
>>
>> On Wed, Apr 15, 2026 at 8:22 AM Dmitri Bourlatchkov <di...@apache.org>
>> wrote:
>>
>>> Hi All,
>>>
>>> Heads up: The current state of PR [4115] looks pretty solid to me. I
>>> believe this PR is approaching a mergeable condition.
>>>
>>> Please post your reviews if you have any comments.
>>>
>>> [4115] https://github.com/apache/polaris/pull/4115
>>>
>>> Thanks,
>>> Dmitri.
>>>
>>> On Tue, Mar 3, 2026 at 3:29 PM Anand Kumar Sankaran via dev <
>>> dev@polaris.apache.org> wrote:
>>>
>>> > Hi Yufei and Dmitri,
>>> >
>>> > Here is a proposal for the REST endpoints for metrics and events.
>>> >
>>> > https://github.com/apache/polaris/pull/3924/changes
>>> >
>>> > I did not see any precursors for raising a PR for proposals, so trying
>>> > this.  Please let me know what you think.
>>> >
>>> > -
>>> > Anand
>>> >
>>> > From: Anand Kumar Sankaran <an...@workday.com>
>>> > Date: Monday, March 2, 2026 at 10:25 AM
>>> > To: dev@polaris.apache.org <de...@polaris.apache.org>
>>> > Subject: Re: Polaris Telemetry and Audit Trail
>>> >
>>> > About the REST API, based on my use cases:
>>> >
>>> >
>>> >   1.
>>> > I want to be able to query commit metrics to track files added /
>>> removed
>>> > per commit, along with record counts. The ingestion pipeline that
>>> writes
>>> > this data is owned by us and we are guaranteed to write this
>>> information
>>> > for each write.
>>> >   2.
>>> > I want to be able to query scan metrics for read. I understand clients
>>> do
>>> > not fulfill this requirement.
>>> >   3.
>>> > I want to be able to query the events table (events are persisted) -
>>> this
>>> > may supersede #2, I am not sure yet.
>>> >
>>> > All this information is in the JDBC based persistence model and is
>>> > persisted in the metastore. I currently don’t have a need to query
>>> > prometheus or open telemetry. I do publish some events to Prometheus
>>> and
>>> > they are forwarded to our dashboards elsewhere.
>>> >
>>> > About the CLI utilities, I meant the admin user utilities. In one of
>>> the
>>> > earliest drafts of my proposal, Prashant mentioned that the metrics
>>> tables
>>> > can grow indefinitely and that a similar problem exists with the events
>>> > table as well. We discussed that cleaning up of old records from both
>>> > metrics tables and events tables can be done via a CLI utility.
>>> >
>>> > I see that Yufei has covered the discussion about datasources.
>>> >
>>> > -
>>> > Anand
>>> >
>>> >
>>> >
>>> > From: Yufei Gu <fl...@gmail.com>
>>> > Date: Friday, February 27, 2026 at 9:54 PM
>>> > To: dev@polaris.apache.org <de...@polaris.apache.org>
>>> > Subject: Re: Polaris Telemetry and Audit Trail
>>> >
>>> > This Message Is From an External Sender
>>> > This message came from outside your organization.
>>> > Report Suspicious<
>>> >
>>> https://us-phishalarm-ewt.proofpoint.com/EWT/v1/Iz9xO38YGHZK!YhNDZABkHi1B699ote2uMwpOZw8i0QMCGO2Szc-HshuABGhGvwPJcymE6G2oUUxtS8xDkSrtGTPm_I3QnVDHoLMk50m9v8z_nZKTkd-bnVUbreF1u0WnfV_X5eYevZl_$
>>> > >
>>> >
>>> >
>>> > As I mentioned in
>>> >
>>> https://urldefense.com/v3/__https://github.com/apache/polaris/issues/3890__;!!Iz9xO38YGHZK!5EuyFFkk3vhRWVIRvQAWBSQfpJkTMA9HxugzDwXmN0LPPqhEFxYkFRGVhtb8AqUwXtDh2OplcMnbMDHKOxrvDU0$
>>> ,
>>> > supporting
>>> > multiple data sources is not a trivial change. I would strongly
>>> recommend
>>> > starting with a design document to carefully evaluate the architectural
>>> > implications and long term impact.
>>> >
>>> > A REST endpoint to query metrics seems reasonable given the current
>>> JDBC
>>> > based persistence model. That said, we may also consider alternative
>>> > storage models. For example, if we later adopt a time series system
>>> such as
>>> > Prometheus to store metrics, the query model and access patterns would
>>> be
>>> > fundamentally different. Designing the REST API without considering
>>> these
>>> > potential evolutions may limit flexibility. I'd suggest to start with
>>> the
>>> > use case.
>>> >
>>> > Yufei
>>> >
>>> >
>>> > On Fri, Feb 27, 2026 at 3:42 PM Dmitri Bourlatchkov <di...@apache.org>
>>> > wrote:
>>> >
>>> > > Hi Anand,
>>> > >
>>> > > Sharing my view... subject to discussion:
>>> > >
>>> > > 1. Adding non-IRC REST API to Polaris is perfectly fine.
>>> > >
>>> > > Figuring out specific endpoint URIs and payloads might require a few
>>> > > roundtrips, so opening a separate thread for that might be best.
>>> > > Contributors commonly create Google Docs for new API proposals too
>>> (they
>>> > > fairly easy to update as the email discussion progresses).
>>> > >
>>> > > There was a suggestion to try Markdown (with PRs) for proposals [1]
>>> ...
>>> > > feel free to give it a try if you are comfortable with that.
>>> > >
>>> > > 2. Could you clarify whether you mean end user utilities or admin
>>> user
>>> > > utilities? In the latter case those might be more suitable for the
>>> Admin
>>> > > CLI (java) not the Python CLI, IMHO.
>>> > >
>>> > > Why would these utilities be common with events? IMHO, event use
>>> cases
>>> > are
>>> > > distinct from scan/commit metrics.
>>> > >
>>> > > 3. I'd prefer separating metrics persistence from MetaStore
>>> persistence
>>> > at
>>> > > the code level, so that they could be mixed and matched
>>> independently.
>>> > The
>>> > > separate datasource question will become a non-issue with that
>>> approach,
>>> > I
>>> > > guess.
>>> > >
>>> > > The rationale for separating scan metrics and metastore persistence
>>> is
>>> > that
>>> > > "cascading deletes" between them are hardly ever required.
>>> Furthermore,
>>> > the
>>> > > data and query patterns are very different so different technologies
>>> > might
>>> > > be beneficial in each case.
>>> > >
>>> > > [1]
>>> >
>>> https://urldefense.com/v3/__https://lists.apache.org/thread/yto2wp982t43h1mqjwnslswhws5z47cy__;!!Iz9xO38YGHZK!5EuyFFkk3vhRWVIRvQAWBSQfpJkTMA9HxugzDwXmN0LPPqhEFxYkFRGVhtb8AqUwXtDh2OplcMnbMDHKxYDakNU$
>>> > >
>>> > > Cheers,
>>> > > Dmitri.
>>> > >
>>> > > On Fri, Feb 27, 2026 at 6:19 PM Anand Kumar Sankaran via dev <
>>> > > dev@polaris.apache.org> wrote:
>>> > >
>>> > > > Thanks all. This PR is merged now.
>>> > > >
>>> > > > Here are the follow-up features / work needed.  These were all
>>> part of
>>> > > the
>>> > > > merged PR at some point in time and were removed to reduce scope.
>>> > > >
>>> > > > Please let me know what you think.
>>> > > >
>>> > > >
>>> > > >   1.  A REST API to paginate through table metrics. This will be
>>> > non-IRC
>>> > > > standard addition.
>>> > > >   2.  Utilities for managing old records, should be common with
>>> events.
>>> > > > There was some discussion that it belongs to the CLI.
>>> > > >   3.  Separate datasource (metrics, events, even other tables?).
>>> > > >
>>> > > >
>>> > > > Anything else?
>>> > > >
>>> > > > -
>>> > > > Anand
>>> > > >
>>> > > >
>>> > >
>>> >
>>> >
>>>
>>

Re: Proposal for REST endpoints for table metrics and events

[EJ Wang <ej...@gmail.com>]

Also, I've been looking more closely at the *persistence schema in the
current metrics work*, and I think there's a structural rigidity problem
worth raising before the shape gets locked in.

Right now we have two separate tables (scan_metrics_report and
commit_metrics_report), each with ~25 flattened columns that directly
mirror the Iceberg report fields. The SPI follows the same split:
writeScanReport and writeCommitReport as separate methods, with per-type
record classes, converters, and model objects. *The practical cost: adding
a new metric type (operational metrics, for example) requires a new table,
a new SPI method, a new record class, a new model class, a new converter
branch, and a schema migration*. That's a lot of surface area for what
should be "one more kind of metric."

*My bias* would be toward a single metrics table with *a typed JSON payload*.
Something like: metric_type (enum), entity_id, table_identifier,
snapshot_id (nullable), received_ts, schema_version, and a payload column
for the metric-specific data. The metric_type + schema_version pair gives
us a forward-compatible contract for the payload shape. Adding a new metric
type becomes an enum value and a payload schema, not a schema migration.
One thing I think we need to be deliberate about is the partition key
design. If all metric types land in one table, scan metrics at scale (high
concurrency, high frequency across many tables) could easily create hot
partitions. We'd want the persistence layer to be able to shard by entity
or time range, and that means the logical schema needs to expose enough
structure for backends to partition on. I don't think the current flattened
layout gives us that.

This is getting complex enough that I don't think ad-hoc PR/ML threads will
converge well. *Would people be open to a biweekly sync for metrics
architecture?* I think 30 minutes every two weeks with interested parties
would be enough to work through the schema, SPI shape, and read API design
together. Happy to help set that up.

-ej

On Mon, Apr 20, 2026 at 2:19 PM EJ Wang <ej...@gmail.com>
wrote:

> Reviewed #4115, left a comment on the code organization side.
>
> One thing stood out: the metrics write path enters through
> PolarisMetricsManager on MetaStoreManager, but the new read path bypasses
> MetaStoreManager entirely and goes straight to BasePersistence via
> callContext.getMetaStore(). That means the read API only works for backends
> that implement BasePersistence. NoSQL and remote backends can't participate.
>
> Stepping back, I think the metrics subsystem is growing into something
> real (write + read + REST API + AuthZ + pagination) *but the persistence
> side is split across two layers in a way that's hard to extend*. I put
> together two diagrams to show what I mean (my best effort).
>
> *Current state* (Diagram 1): three interfaces at three different levels.
> The engine-facing SPI (PolarisMetricsReporter) is clean. But
> PolarisMetricsManager on MetaStoreManager is a passthrough to
> MetricsPersistence on BasePersistence. The @Beta annotation and SPI javadoc
> are on the BasePersistence layer, while the actual extension points
> (PolarisMetricsReporter, PolarisMetricsManager) carry no stability
> annotation. The write path goes through the MetaStoreManager layer, the
> read path doesn't.
>
> *What I envision* (Diagram 2): two SPIs at two levels.
> PolarisMetricsReporter stays as the engine-facing SPI.
> PolarisMetricsManager becomes the backend-facing SPI with both write and
> read methods at the MetaStoreManager level, where any backend (JDBC, NoSQL,
> remote) can implement them. MetricsPersistence on BasePersistence goes
> away. Where metrics actually land is an implementation detail, not a core
> interface.
>
> *Minor naming thing*: PolarisMetricsReporter is broader than what it
> actually handles. It only accepts Iceberg REST Catalog metrics (ScanReport,
> CommitReport via MetricsReport). Generic table metrics or operational
> metrics aren't in scope. Not blocking, but worth noting if the metrics
> surface expands.
>
> *Rough sketch of how to get there*:
>  1.  Add read methods to PolarisMetricsManager (listScanReports,
> listCommitReports) with default no-op, same as the existing write methods.
> (Probably make PolarisMetricsManager more explicit on being Iceberg
> specific like package name or class name etc.)
>  2.  Wire MetricsReportsService through MetaStoreManager instead of
> callContext.getMetaStore().
>  3.  Extract metrics persistence from JdbcBasePersistenceImpl into its own
> class. That file carries ~7 responsibilities, metrics being one of them.
>  4.  Remove MetricsPersistence from BasePersistence.
>
> *None of this needs to happen in #4115. But if the direction makes sense,
> it would be good to align before the metrics surface grows further. Curious
> what others think.*
>
> *My mental model note*: Level 1 MetaStoreManager; level 2 transactional
> persistence; level 3 base persistence
>
> Diagram 1
> <https://www.plantuml.com/plantuml/uml/bLHDR-Cs4BthLmpIYupw0zbkKQ1r3M-S7Bp8xhhM7WCOb3IM65EaGD9EX2RzxHrHb4CxRelwa4YSDu_lpOVcnZ9jzvM8BBS2uGjQpJC3dtHMSekPtMk44IpsMgEqa5XcCOhCZikQQLP1pR8TAp2n3ILhmZDP20m0fcIvUkAoW2qJXd9z1bpToO9BX3WXu0ucy5rpgGPNm0nW5_epUWtm2Ue3pn3kMOFQmKntGZW0BYtgBSi8k5A2QMwybJNMIbFiGSR9QZc4nUqIvikStF0jHprua5C-amge42aNt3R0f5JaaoivdV2Pkqbx4hee4ymOkBh5BTiB-_uIeGeo8zL8rPsPl4DktdEiK1jkB1NdZCRbrSTecDe_mlHbF0wvBmCkaOH5_S8a_TTTKI6-nmCAkEw4LpxsZ-LbYLKQFKMNOgf_wuM7_bV9gOer5SYMMksBSWXFcbi49KNZXNLicwfe3TETC7gPdPqI7uBcHMb1RSzYq34c6PDUM9mn8HRsUTZEiDBve3NjVZumBj0U7SS37mGO7vcwtiK-_pU7U7L_f-digo9YbhSwIfMRwIITKGXbxdIUTCGF1SeCJxloKsU-3k9ddRbX1eDq1q_fx1JbBGT0glVyXimDuP4TQ5qpCAmnGEj2s_6n5mtn1z-97-63itFQZLPO1Ev2tu_WF7Ju-VPc0Skg5bYXxBhkY1xpD7EM_7fyflSpIsqMgVth5xhVr4eQxWQ8enaSAJQSG16yFSDuJ798rrcXr_3n-lfdk7icQjEBmFujL7AodiP_Y4Z7-YxvtZNs4zMgpNTl6tF8sglyPsmqchrjvQ-m-aP94r-TwCA2Ka8upPJZwtvSpoYCXkYMZU2NXvRMBfq9P3i3Le4VAZUAlUZ_oPKsxPgY0Q_BSKLkyr9bhQhQrJjo_x3TPlIB0DPjnMfcIoYP0QaYw1a0fTKDr8fB6ntNuvmoL1ZGkXa69Njh43zf9GiGxHQrA_jDYWRSzF5--WmTVrN97_Sm8LbLUy_lGBmLanJjFkDlGkRqjA_4tm00>
> :
>
> [image: image.png]
>
> Diagram 2
> <https://www.plantuml.com/plantuml/uml/VLLDR-8m4BtdLupO2sWBLVU8AaGB7AXAbssGzb896SS9RXqxjKqBwkv_tt7iV43fdaZYDpFlpRmnOsE9jhjSH9PRmM31hERKm8scMsuPjJlDe0yheZDc8RR4iYWoBrmMH9CS2a9VICPYUy1OZN0YCy5Q0BCbYNhdCeEK28En8G8wCvbnoQ0R8_05Bc6bkLIz3X03p1zzH7zR-9ZfDquPt9C3qoNCX2yV4G2NbkcKu5jdgGJHt0GbZwnG6i-UP3TUpk5gM6Ldqke350eZUqzoCft3U9xWHvxoa5-7K4nF1J46EbEMafsmdrCBbQ44gVggy18IZrn_ph5asd1ZiIKdQSgueZvjXrQFSFrdC3YN-nXmBacxbGiYyLVxLaBtdhqn0LSzdBDhqQtQoOJeGyad3z0lUqnYgpGB6Ns8oVyta00Dy_WnX0tIOZ8v6SYxHll1TrH6aejAik-mh-AphVFCwSUQqFypElag5QRGFDjQKEd96K1P8QP41c9TzA_IIQyvdAWyv_RSiS3skb0_EzDDkK2v5xWF6MiGFlvhpFLcD2Dq2pml14gaF67eQkmd8gulDoC4kSOu6KVpkvlUJg1RTbWISU40RdBUUS_9XfRZ2dwxm_SW8LYFISgm_MnlDQ6M9P1gbKEc4X-2pH_FvJCkCqm9pbVjD6LrwdLeOrDWfOaqc8Wh9BE85oNKxkNQ6o4yGRy_Eae0G_G8tZv81d3bHDB23WOdisohVr3nh_j6lbSjbNaLRTc8UgtPbAU1J_tygOfZX9DWEJeHDvYx-qmSi5FgNLPZwHrHcUsncGQ5-skhUclpE5fo4ounpFauYrUbkU6ccfnxMvitwag4IyerhTxj8In_Oj1bDO4pQru674loYrGlULHLEGCjwJJ8gDoVZR8MxO4BT3IzRvIcAQKezC6xpziGnTyImrfEGyJI_OcKfgtxIvnTqFEMS17L9Z-jsARN5FmTheP7HtSdtOMT0B4GY2FYHXxgQmMtj2bRqiLFGapiVe1_QVKDrkqXcm83aFEXnMYCZ-xlyHy>
> :
> [image: image.png]
>
>  -ej
>
> On Wed, Apr 15, 2026 at 8:22 AM Dmitri Bourlatchkov <di...@apache.org>
> wrote:
>
>> Hi All,
>>
>> Heads up: The current state of PR [4115] looks pretty solid to me. I
>> believe this PR is approaching a mergeable condition.
>>
>> Please post your reviews if you have any comments.
>>
>> [4115] https://github.com/apache/polaris/pull/4115
>>
>> Thanks,
>> Dmitri.
>>
>> On Tue, Mar 3, 2026 at 3:29 PM Anand Kumar Sankaran via dev <
>> dev@polaris.apache.org> wrote:
>>
>> > Hi Yufei and Dmitri,
>> >
>> > Here is a proposal for the REST endpoints for metrics and events.
>> >
>> > https://github.com/apache/polaris/pull/3924/changes
>> >
>> > I did not see any precursors for raising a PR for proposals, so trying
>> > this.  Please let me know what you think.
>> >
>> > -
>> > Anand
>> >
>> > From: Anand Kumar Sankaran <an...@workday.com>
>> > Date: Monday, March 2, 2026 at 10:25 AM
>> > To: dev@polaris.apache.org <de...@polaris.apache.org>
>> > Subject: Re: Polaris Telemetry and Audit Trail
>> >
>> > About the REST API, based on my use cases:
>> >
>> >
>> >   1.
>> > I want to be able to query commit metrics to track files added / removed
>> > per commit, along with record counts. The ingestion pipeline that writes
>> > this data is owned by us and we are guaranteed to write this information
>> > for each write.
>> >   2.
>> > I want to be able to query scan metrics for read. I understand clients
>> do
>> > not fulfill this requirement.
>> >   3.
>> > I want to be able to query the events table (events are persisted) -
>> this
>> > may supersede #2, I am not sure yet.
>> >
>> > All this information is in the JDBC based persistence model and is
>> > persisted in the metastore. I currently don’t have a need to query
>> > prometheus or open telemetry. I do publish some events to Prometheus and
>> > they are forwarded to our dashboards elsewhere.
>> >
>> > About the CLI utilities, I meant the admin user utilities. In one of the
>> > earliest drafts of my proposal, Prashant mentioned that the metrics
>> tables
>> > can grow indefinitely and that a similar problem exists with the events
>> > table as well. We discussed that cleaning up of old records from both
>> > metrics tables and events tables can be done via a CLI utility.
>> >
>> > I see that Yufei has covered the discussion about datasources.
>> >
>> > -
>> > Anand
>> >
>> >
>> >
>> > From: Yufei Gu <fl...@gmail.com>
>> > Date: Friday, February 27, 2026 at 9:54 PM
>> > To: dev@polaris.apache.org <de...@polaris.apache.org>
>> > Subject: Re: Polaris Telemetry and Audit Trail
>> >
>> > This Message Is From an External Sender
>> > This message came from outside your organization.
>> > Report Suspicious<
>> >
>> https://us-phishalarm-ewt.proofpoint.com/EWT/v1/Iz9xO38YGHZK!YhNDZABkHi1B699ote2uMwpOZw8i0QMCGO2Szc-HshuABGhGvwPJcymE6G2oUUxtS8xDkSrtGTPm_I3QnVDHoLMk50m9v8z_nZKTkd-bnVUbreF1u0WnfV_X5eYevZl_$
>> > >
>> >
>> >
>> > As I mentioned in
>> >
>> https://urldefense.com/v3/__https://github.com/apache/polaris/issues/3890__;!!Iz9xO38YGHZK!5EuyFFkk3vhRWVIRvQAWBSQfpJkTMA9HxugzDwXmN0LPPqhEFxYkFRGVhtb8AqUwXtDh2OplcMnbMDHKOxrvDU0$
>> ,
>> > supporting
>> > multiple data sources is not a trivial change. I would strongly
>> recommend
>> > starting with a design document to carefully evaluate the architectural
>> > implications and long term impact.
>> >
>> > A REST endpoint to query metrics seems reasonable given the current JDBC
>> > based persistence model. That said, we may also consider alternative
>> > storage models. For example, if we later adopt a time series system
>> such as
>> > Prometheus to store metrics, the query model and access patterns would
>> be
>> > fundamentally different. Designing the REST API without considering
>> these
>> > potential evolutions may limit flexibility. I'd suggest to start with
>> the
>> > use case.
>> >
>> > Yufei
>> >
>> >
>> > On Fri, Feb 27, 2026 at 3:42 PM Dmitri Bourlatchkov <di...@apache.org>
>> > wrote:
>> >
>> > > Hi Anand,
>> > >
>> > > Sharing my view... subject to discussion:
>> > >
>> > > 1. Adding non-IRC REST API to Polaris is perfectly fine.
>> > >
>> > > Figuring out specific endpoint URIs and payloads might require a few
>> > > roundtrips, so opening a separate thread for that might be best.
>> > > Contributors commonly create Google Docs for new API proposals too
>> (they
>> > > fairly easy to update as the email discussion progresses).
>> > >
>> > > There was a suggestion to try Markdown (with PRs) for proposals [1]
>> ...
>> > > feel free to give it a try if you are comfortable with that.
>> > >
>> > > 2. Could you clarify whether you mean end user utilities or admin user
>> > > utilities? In the latter case those might be more suitable for the
>> Admin
>> > > CLI (java) not the Python CLI, IMHO.
>> > >
>> > > Why would these utilities be common with events? IMHO, event use cases
>> > are
>> > > distinct from scan/commit metrics.
>> > >
>> > > 3. I'd prefer separating metrics persistence from MetaStore
>> persistence
>> > at
>> > > the code level, so that they could be mixed and matched independently.
>> > The
>> > > separate datasource question will become a non-issue with that
>> approach,
>> > I
>> > > guess.
>> > >
>> > > The rationale for separating scan metrics and metastore persistence is
>> > that
>> > > "cascading deletes" between them are hardly ever required.
>> Furthermore,
>> > the
>> > > data and query patterns are very different so different technologies
>> > might
>> > > be beneficial in each case.
>> > >
>> > > [1]
>> >
>> https://urldefense.com/v3/__https://lists.apache.org/thread/yto2wp982t43h1mqjwnslswhws5z47cy__;!!Iz9xO38YGHZK!5EuyFFkk3vhRWVIRvQAWBSQfpJkTMA9HxugzDwXmN0LPPqhEFxYkFRGVhtb8AqUwXtDh2OplcMnbMDHKxYDakNU$
>> > >
>> > > Cheers,
>> > > Dmitri.
>> > >
>> > > On Fri, Feb 27, 2026 at 6:19 PM Anand Kumar Sankaran via dev <
>> > > dev@polaris.apache.org> wrote:
>> > >
>> > > > Thanks all. This PR is merged now.
>> > > >
>> > > > Here are the follow-up features / work needed.  These were all part
>> of
>> > > the
>> > > > merged PR at some point in time and were removed to reduce scope.
>> > > >
>> > > > Please let me know what you think.
>> > > >
>> > > >
>> > > >   1.  A REST API to paginate through table metrics. This will be
>> > non-IRC
>> > > > standard addition.
>> > > >   2.  Utilities for managing old records, should be common with
>> events.
>> > > > There was some discussion that it belongs to the CLI.
>> > > >   3.  Separate datasource (metrics, events, even other tables?).
>> > > >
>> > > >
>> > > > Anything else?
>> > > >
>> > > > -
>> > > > Anand
>> > > >
>> > > >
>> > >
>> >
>> >
>>
>

Re: Proposal for REST endpoints for table metrics and events

[EJ Wang <ej...@gmail.com>]

Reviewed #4115, left a comment on the code organization side.

One thing stood out: the metrics write path enters through
PolarisMetricsManager on MetaStoreManager, but the new read path bypasses
MetaStoreManager entirely and goes straight to BasePersistence via
callContext.getMetaStore(). That means the read API only works for backends
that implement BasePersistence. NoSQL and remote backends can't participate.

Stepping back, I think the metrics subsystem is growing into something real
(write + read + REST API + AuthZ + pagination) *but the persistence side is
split across two layers in a way that's hard to extend*. I put together two
diagrams to show what I mean (my best effort).

*Current state* (Diagram 1): three interfaces at three different levels.
The engine-facing SPI (PolarisMetricsReporter) is clean. But
PolarisMetricsManager on MetaStoreManager is a passthrough to
MetricsPersistence on BasePersistence. The @Beta annotation and SPI javadoc
are on the BasePersistence layer, while the actual extension points
(PolarisMetricsReporter, PolarisMetricsManager) carry no stability
annotation. The write path goes through the MetaStoreManager layer, the
read path doesn't.

*What I envision* (Diagram 2): two SPIs at two levels.
PolarisMetricsReporter stays as the engine-facing SPI.
PolarisMetricsManager becomes the backend-facing SPI with both write and
read methods at the MetaStoreManager level, where any backend (JDBC, NoSQL,
remote) can implement them. MetricsPersistence on BasePersistence goes
away. Where metrics actually land is an implementation detail, not a core
interface.

*Minor naming thing*: PolarisMetricsReporter is broader than what it
actually handles. It only accepts Iceberg REST Catalog metrics (ScanReport,
CommitReport via MetricsReport). Generic table metrics or operational
metrics aren't in scope. Not blocking, but worth noting if the metrics
surface expands.

*Rough sketch of how to get there*:
 1.  Add read methods to PolarisMetricsManager (listScanReports,
listCommitReports) with default no-op, same as the existing write methods.
(Probably make PolarisMetricsManager more explicit on being Iceberg
specific like package name or class name etc.)
 2.  Wire MetricsReportsService through MetaStoreManager instead of
callContext.getMetaStore().
 3.  Extract metrics persistence from JdbcBasePersistenceImpl into its own
class. That file carries ~7 responsibilities, metrics being one of them.
 4.  Remove MetricsPersistence from BasePersistence.

*None of this needs to happen in #4115. But if the direction makes sense,
it would be good to align before the metrics surface grows further. Curious
what others think.*

*My mental model note*: Level 1 MetaStoreManager; level 2 transactional
persistence; level 3 base persistence

Diagram 1
<https://www.plantuml.com/plantuml/uml/bLHDR-Cs4BthLmpIYupw0zbkKQ1r3M-S7Bp8xhhM7WCOb3IM65EaGD9EX2RzxHrHb4CxRelwa4YSDu_lpOVcnZ9jzvM8BBS2uGjQpJC3dtHMSekPtMk44IpsMgEqa5XcCOhCZikQQLP1pR8TAp2n3ILhmZDP20m0fcIvUkAoW2qJXd9z1bpToO9BX3WXu0ucy5rpgGPNm0nW5_epUWtm2Ue3pn3kMOFQmKntGZW0BYtgBSi8k5A2QMwybJNMIbFiGSR9QZc4nUqIvikStF0jHprua5C-amge42aNt3R0f5JaaoivdV2Pkqbx4hee4ymOkBh5BTiB-_uIeGeo8zL8rPsPl4DktdEiK1jkB1NdZCRbrSTecDe_mlHbF0wvBmCkaOH5_S8a_TTTKI6-nmCAkEw4LpxsZ-LbYLKQFKMNOgf_wuM7_bV9gOer5SYMMksBSWXFcbi49KNZXNLicwfe3TETC7gPdPqI7uBcHMb1RSzYq34c6PDUM9mn8HRsUTZEiDBve3NjVZumBj0U7SS37mGO7vcwtiK-_pU7U7L_f-digo9YbhSwIfMRwIITKGXbxdIUTCGF1SeCJxloKsU-3k9ddRbX1eDq1q_fx1JbBGT0glVyXimDuP4TQ5qpCAmnGEj2s_6n5mtn1z-97-63itFQZLPO1Ev2tu_WF7Ju-VPc0Skg5bYXxBhkY1xpD7EM_7fyflSpIsqMgVth5xhVr4eQxWQ8enaSAJQSG16yFSDuJ798rrcXr_3n-lfdk7icQjEBmFujL7AodiP_Y4Z7-YxvtZNs4zMgpNTl6tF8sglyPsmqchrjvQ-m-aP94r-TwCA2Ka8upPJZwtvSpoYCXkYMZU2NXvRMBfq9P3i3Le4VAZUAlUZ_oPKsxPgY0Q_BSKLkyr9bhQhQrJjo_x3TPlIB0DPjnMfcIoYP0QaYw1a0fTKDr8fB6ntNuvmoL1ZGkXa69Njh43zf9GiGxHQrA_jDYWRSzF5--WmTVrN97_Sm8LbLUy_lGBmLanJjFkDlGkRqjA_4tm00>
:

[image: image.png]

Diagram 2
<https://www.plantuml.com/plantuml/uml/VLLDR-8m4BtdLupO2sWBLVU8AaGB7AXAbssGzb896SS9RXqxjKqBwkv_tt7iV43fdaZYDpFlpRmnOsE9jhjSH9PRmM31hERKm8scMsuPjJlDe0yheZDc8RR4iYWoBrmMH9CS2a9VICPYUy1OZN0YCy5Q0BCbYNhdCeEK28En8G8wCvbnoQ0R8_05Bc6bkLIz3X03p1zzH7zR-9ZfDquPt9C3qoNCX2yV4G2NbkcKu5jdgGJHt0GbZwnG6i-UP3TUpk5gM6Ldqke350eZUqzoCft3U9xWHvxoa5-7K4nF1J46EbEMafsmdrCBbQ44gVggy18IZrn_ph5asd1ZiIKdQSgueZvjXrQFSFrdC3YN-nXmBacxbGiYyLVxLaBtdhqn0LSzdBDhqQtQoOJeGyad3z0lUqnYgpGB6Ns8oVyta00Dy_WnX0tIOZ8v6SYxHll1TrH6aejAik-mh-AphVFCwSUQqFypElag5QRGFDjQKEd96K1P8QP41c9TzA_IIQyvdAWyv_RSiS3skb0_EzDDkK2v5xWF6MiGFlvhpFLcD2Dq2pml14gaF67eQkmd8gulDoC4kSOu6KVpkvlUJg1RTbWISU40RdBUUS_9XfRZ2dwxm_SW8LYFISgm_MnlDQ6M9P1gbKEc4X-2pH_FvJCkCqm9pbVjD6LrwdLeOrDWfOaqc8Wh9BE85oNKxkNQ6o4yGRy_Eae0G_G8tZv81d3bHDB23WOdisohVr3nh_j6lbSjbNaLRTc8UgtPbAU1J_tygOfZX9DWEJeHDvYx-qmSi5FgNLPZwHrHcUsncGQ5-skhUclpE5fo4ounpFauYrUbkU6ccfnxMvitwag4IyerhTxj8In_Oj1bDO4pQru674loYrGlULHLEGCjwJJ8gDoVZR8MxO4BT3IzRvIcAQKezC6xpziGnTyImrfEGyJI_OcKfgtxIvnTqFEMS17L9Z-jsARN5FmTheP7HtSdtOMT0B4GY2FYHXxgQmMtj2bRqiLFGapiVe1_QVKDrkqXcm83aFEXnMYCZ-xlyHy>
:
[image: image.png]

 -ej

On Wed, Apr 15, 2026 at 8:22 AM Dmitri Bourlatchkov <di...@apache.org>
wrote:

> Hi All,
>
> Heads up: The current state of PR [4115] looks pretty solid to me. I
> believe this PR is approaching a mergeable condition.
>
> Please post your reviews if you have any comments.
>
> [4115] https://github.com/apache/polaris/pull/4115
>
> Thanks,
> Dmitri.
>
> On Tue, Mar 3, 2026 at 3:29 PM Anand Kumar Sankaran via dev <
> dev@polaris.apache.org> wrote:
>
> > Hi Yufei and Dmitri,
> >
> > Here is a proposal for the REST endpoints for metrics and events.
> >
> > https://github.com/apache/polaris/pull/3924/changes
> >
> > I did not see any precursors for raising a PR for proposals, so trying
> > this.  Please let me know what you think.
> >
> > -
> > Anand
> >
> > From: Anand Kumar Sankaran <an...@workday.com>
> > Date: Monday, March 2, 2026 at 10:25 AM
> > To: dev@polaris.apache.org <de...@polaris.apache.org>
> > Subject: Re: Polaris Telemetry and Audit Trail
> >
> > About the REST API, based on my use cases:
> >
> >
> >   1.
> > I want to be able to query commit metrics to track files added / removed
> > per commit, along with record counts. The ingestion pipeline that writes
> > this data is owned by us and we are guaranteed to write this information
> > for each write.
> >   2.
> > I want to be able to query scan metrics for read. I understand clients do
> > not fulfill this requirement.
> >   3.
> > I want to be able to query the events table (events are persisted) - this
> > may supersede #2, I am not sure yet.
> >
> > All this information is in the JDBC based persistence model and is
> > persisted in the metastore. I currently don’t have a need to query
> > prometheus or open telemetry. I do publish some events to Prometheus and
> > they are forwarded to our dashboards elsewhere.
> >
> > About the CLI utilities, I meant the admin user utilities. In one of the
> > earliest drafts of my proposal, Prashant mentioned that the metrics
> tables
> > can grow indefinitely and that a similar problem exists with the events
> > table as well. We discussed that cleaning up of old records from both
> > metrics tables and events tables can be done via a CLI utility.
> >
> > I see that Yufei has covered the discussion about datasources.
> >
> > -
> > Anand
> >
> >
> >
> > From: Yufei Gu <fl...@gmail.com>
> > Date: Friday, February 27, 2026 at 9:54 PM
> > To: dev@polaris.apache.org <de...@polaris.apache.org>
> > Subject: Re: Polaris Telemetry and Audit Trail
> >
> > This Message Is From an External Sender
> > This message came from outside your organization.
> > Report Suspicious<
> >
> https://us-phishalarm-ewt.proofpoint.com/EWT/v1/Iz9xO38YGHZK!YhNDZABkHi1B699ote2uMwpOZw8i0QMCGO2Szc-HshuABGhGvwPJcymE6G2oUUxtS8xDkSrtGTPm_I3QnVDHoLMk50m9v8z_nZKTkd-bnVUbreF1u0WnfV_X5eYevZl_$
> > >
> >
> >
> > As I mentioned in
> >
> https://urldefense.com/v3/__https://github.com/apache/polaris/issues/3890__;!!Iz9xO38YGHZK!5EuyFFkk3vhRWVIRvQAWBSQfpJkTMA9HxugzDwXmN0LPPqhEFxYkFRGVhtb8AqUwXtDh2OplcMnbMDHKOxrvDU0$
> ,
> > supporting
> > multiple data sources is not a trivial change. I would strongly recommend
> > starting with a design document to carefully evaluate the architectural
> > implications and long term impact.
> >
> > A REST endpoint to query metrics seems reasonable given the current JDBC
> > based persistence model. That said, we may also consider alternative
> > storage models. For example, if we later adopt a time series system such
> as
> > Prometheus to store metrics, the query model and access patterns would be
> > fundamentally different. Designing the REST API without considering these
> > potential evolutions may limit flexibility. I'd suggest to start with the
> > use case.
> >
> > Yufei
> >
> >
> > On Fri, Feb 27, 2026 at 3:42 PM Dmitri Bourlatchkov <di...@apache.org>
> > wrote:
> >
> > > Hi Anand,
> > >
> > > Sharing my view... subject to discussion:
> > >
> > > 1. Adding non-IRC REST API to Polaris is perfectly fine.
> > >
> > > Figuring out specific endpoint URIs and payloads might require a few
> > > roundtrips, so opening a separate thread for that might be best.
> > > Contributors commonly create Google Docs for new API proposals too
> (they
> > > fairly easy to update as the email discussion progresses).
> > >
> > > There was a suggestion to try Markdown (with PRs) for proposals [1] ...
> > > feel free to give it a try if you are comfortable with that.
> > >
> > > 2. Could you clarify whether you mean end user utilities or admin user
> > > utilities? In the latter case those might be more suitable for the
> Admin
> > > CLI (java) not the Python CLI, IMHO.
> > >
> > > Why would these utilities be common with events? IMHO, event use cases
> > are
> > > distinct from scan/commit metrics.
> > >
> > > 3. I'd prefer separating metrics persistence from MetaStore persistence
> > at
> > > the code level, so that they could be mixed and matched independently.
> > The
> > > separate datasource question will become a non-issue with that
> approach,
> > I
> > > guess.
> > >
> > > The rationale for separating scan metrics and metastore persistence is
> > that
> > > "cascading deletes" between them are hardly ever required. Furthermore,
> > the
> > > data and query patterns are very different so different technologies
> > might
> > > be beneficial in each case.
> > >
> > > [1]
> >
> https://urldefense.com/v3/__https://lists.apache.org/thread/yto2wp982t43h1mqjwnslswhws5z47cy__;!!Iz9xO38YGHZK!5EuyFFkk3vhRWVIRvQAWBSQfpJkTMA9HxugzDwXmN0LPPqhEFxYkFRGVhtb8AqUwXtDh2OplcMnbMDHKxYDakNU$
> > >
> > > Cheers,
> > > Dmitri.
> > >
> > > On Fri, Feb 27, 2026 at 6:19 PM Anand Kumar Sankaran via dev <
> > > dev@polaris.apache.org> wrote:
> > >
> > > > Thanks all. This PR is merged now.
> > > >
> > > > Here are the follow-up features / work needed.  These were all part
> of
> > > the
> > > > merged PR at some point in time and were removed to reduce scope.
> > > >
> > > > Please let me know what you think.
> > > >
> > > >
> > > >   1.  A REST API to paginate through table metrics. This will be
> > non-IRC
> > > > standard addition.
> > > >   2.  Utilities for managing old records, should be common with
> events.
> > > > There was some discussion that it belongs to the CLI.
> > > >   3.  Separate datasource (metrics, events, even other tables?).
> > > >
> > > >
> > > > Anything else?
> > > >
> > > > -
> > > > Anand
> > > >
> > > >
> > >
> >
> >
>

Re: Proposal for REST endpoints for table metrics and events

[Dmitri Bourlatchkov <di...@apache.org>]

Hi All,

Heads up: The current state of PR [4115] looks pretty solid to me. I
believe this PR is approaching a mergeable condition.

Please post your reviews if you have any comments.

[4115] https://github.com/apache/polaris/pull/4115

Thanks,
Dmitri.

On Tue, Mar 3, 2026 at 3:29 PM Anand Kumar Sankaran via dev <
dev@polaris.apache.org> wrote:

> Hi Yufei and Dmitri,
>
> Here is a proposal for the REST endpoints for metrics and events.
>
> https://github.com/apache/polaris/pull/3924/changes
>
> I did not see any precursors for raising a PR for proposals, so trying
> this.  Please let me know what you think.
>
> -
> Anand
>
> From: Anand Kumar Sankaran <an...@workday.com>
> Date: Monday, March 2, 2026 at 10:25 AM
> To: dev@polaris.apache.org <de...@polaris.apache.org>
> Subject: Re: Polaris Telemetry and Audit Trail
>
> About the REST API, based on my use cases:
>
>
>   1.
> I want to be able to query commit metrics to track files added / removed
> per commit, along with record counts. The ingestion pipeline that writes
> this data is owned by us and we are guaranteed to write this information
> for each write.
>   2.
> I want to be able to query scan metrics for read. I understand clients do
> not fulfill this requirement.
>   3.
> I want to be able to query the events table (events are persisted) - this
> may supersede #2, I am not sure yet.
>
> All this information is in the JDBC based persistence model and is
> persisted in the metastore. I currently don’t have a need to query
> prometheus or open telemetry. I do publish some events to Prometheus and
> they are forwarded to our dashboards elsewhere.
>
> About the CLI utilities, I meant the admin user utilities. In one of the
> earliest drafts of my proposal, Prashant mentioned that the metrics tables
> can grow indefinitely and that a similar problem exists with the events
> table as well. We discussed that cleaning up of old records from both
> metrics tables and events tables can be done via a CLI utility.
>
> I see that Yufei has covered the discussion about datasources.
>
> -
> Anand
>
>
>
> From: Yufei Gu <fl...@gmail.com>
> Date: Friday, February 27, 2026 at 9:54 PM
> To: dev@polaris.apache.org <de...@polaris.apache.org>
> Subject: Re: Polaris Telemetry and Audit Trail
>
> This Message Is From an External Sender
> This message came from outside your organization.
> Report Suspicious<
> https://us-phishalarm-ewt.proofpoint.com/EWT/v1/Iz9xO38YGHZK!YhNDZABkHi1B699ote2uMwpOZw8i0QMCGO2Szc-HshuABGhGvwPJcymE6G2oUUxtS8xDkSrtGTPm_I3QnVDHoLMk50m9v8z_nZKTkd-bnVUbreF1u0WnfV_X5eYevZl_$
> >
>
>
> As I mentioned in
> https://urldefense.com/v3/__https://github.com/apache/polaris/issues/3890__;!!Iz9xO38YGHZK!5EuyFFkk3vhRWVIRvQAWBSQfpJkTMA9HxugzDwXmN0LPPqhEFxYkFRGVhtb8AqUwXtDh2OplcMnbMDHKOxrvDU0$,
> supporting
> multiple data sources is not a trivial change. I would strongly recommend
> starting with a design document to carefully evaluate the architectural
> implications and long term impact.
>
> A REST endpoint to query metrics seems reasonable given the current JDBC
> based persistence model. That said, we may also consider alternative
> storage models. For example, if we later adopt a time series system such as
> Prometheus to store metrics, the query model and access patterns would be
> fundamentally different. Designing the REST API without considering these
> potential evolutions may limit flexibility. I'd suggest to start with the
> use case.
>
> Yufei
>
>
> On Fri, Feb 27, 2026 at 3:42 PM Dmitri Bourlatchkov <di...@apache.org>
> wrote:
>
> > Hi Anand,
> >
> > Sharing my view... subject to discussion:
> >
> > 1. Adding non-IRC REST API to Polaris is perfectly fine.
> >
> > Figuring out specific endpoint URIs and payloads might require a few
> > roundtrips, so opening a separate thread for that might be best.
> > Contributors commonly create Google Docs for new API proposals too (they
> > fairly easy to update as the email discussion progresses).
> >
> > There was a suggestion to try Markdown (with PRs) for proposals [1] ...
> > feel free to give it a try if you are comfortable with that.
> >
> > 2. Could you clarify whether you mean end user utilities or admin user
> > utilities? In the latter case those might be more suitable for the Admin
> > CLI (java) not the Python CLI, IMHO.
> >
> > Why would these utilities be common with events? IMHO, event use cases
> are
> > distinct from scan/commit metrics.
> >
> > 3. I'd prefer separating metrics persistence from MetaStore persistence
> at
> > the code level, so that they could be mixed and matched independently.
> The
> > separate datasource question will become a non-issue with that approach,
> I
> > guess.
> >
> > The rationale for separating scan metrics and metastore persistence is
> that
> > "cascading deletes" between them are hardly ever required. Furthermore,
> the
> > data and query patterns are very different so different technologies
> might
> > be beneficial in each case.
> >
> > [1]
> https://urldefense.com/v3/__https://lists.apache.org/thread/yto2wp982t43h1mqjwnslswhws5z47cy__;!!Iz9xO38YGHZK!5EuyFFkk3vhRWVIRvQAWBSQfpJkTMA9HxugzDwXmN0LPPqhEFxYkFRGVhtb8AqUwXtDh2OplcMnbMDHKxYDakNU$
> >
> > Cheers,
> > Dmitri.
> >
> > On Fri, Feb 27, 2026 at 6:19 PM Anand Kumar Sankaran via dev <
> > dev@polaris.apache.org> wrote:
> >
> > > Thanks all. This PR is merged now.
> > >
> > > Here are the follow-up features / work needed.  These were all part of
> > the
> > > merged PR at some point in time and were removed to reduce scope.
> > >
> > > Please let me know what you think.
> > >
> > >
> > >   1.  A REST API to paginate through table metrics. This will be
> non-IRC
> > > standard addition.
> > >   2.  Utilities for managing old records, should be common with events.
> > > There was some discussion that it belongs to the CLI.
> > >   3.  Separate datasource (metrics, events, even other tables?).
> > >
> > >
> > > Anything else?
> > >
> > > -
> > > Anand
> > >
> > >
> >
>
>

Proposal for REST endpoints for table metrics and events

[Anand Kumar Sankaran via dev <de...@polaris.apache.org>]

Hi Yufei and Dmitri,

Here is a proposal for the REST endpoints for metrics and events.

https://github.com/apache/polaris/pull/3924/changes

I did not see any precursors for raising a PR for proposals, so trying this.  Please let me know what you think.

-
Anand

From: Anand Kumar Sankaran <an...@workday.com>
Date: Monday, March 2, 2026 at 10:25 AM
To: dev@polaris.apache.org <de...@polaris.apache.org>
Subject: Re: Polaris Telemetry and Audit Trail

About the REST API, based on my use cases:


  1.
I want to be able to query commit metrics to track files added / removed per commit, along with record counts. The ingestion pipeline that writes this data is owned by us and we are guaranteed to write this information for each write.
  2.
I want to be able to query scan metrics for read. I understand clients do not fulfill this requirement.
  3.
I want to be able to query the events table (events are persisted) - this may supersede #2, I am not sure yet.

All this information is in the JDBC based persistence model and is persisted in the metastore. I currently don’t have a need to query prometheus or open telemetry. I do publish some events to Prometheus and they are forwarded to our dashboards elsewhere.

About the CLI utilities, I meant the admin user utilities. In one of the earliest drafts of my proposal, Prashant mentioned that the metrics tables can grow indefinitely and that a similar problem exists with the events table as well. We discussed that cleaning up of old records from both metrics tables and events tables can be done via a CLI utility.

I see that Yufei has covered the discussion about datasources.

-
Anand



From: Yufei Gu <fl...@gmail.com>
Date: Friday, February 27, 2026 at 9:54 PM
To: dev@polaris.apache.org <de...@polaris.apache.org>
Subject: Re: Polaris Telemetry and Audit Trail

This Message Is From an External Sender
This message came from outside your organization.
Report Suspicious<https://us-phishalarm-ewt.proofpoint.com/EWT/v1/Iz9xO38YGHZK!YhNDZABkHi1B699ote2uMwpOZw8i0QMCGO2Szc-HshuABGhGvwPJcymE6G2oUUxtS8xDkSrtGTPm_I3QnVDHoLMk50m9v8z_nZKTkd-bnVUbreF1u0WnfV_X5eYevZl_$>


As I mentioned in https://urldefense.com/v3/__https://github.com/apache/polaris/issues/3890__;!!Iz9xO38YGHZK!5EuyFFkk3vhRWVIRvQAWBSQfpJkTMA9HxugzDwXmN0LPPqhEFxYkFRGVhtb8AqUwXtDh2OplcMnbMDHKOxrvDU0$, supporting
multiple data sources is not a trivial change. I would strongly recommend
starting with a design document to carefully evaluate the architectural
implications and long term impact.

A REST endpoint to query metrics seems reasonable given the current JDBC
based persistence model. That said, we may also consider alternative
storage models. For example, if we later adopt a time series system such as
Prometheus to store metrics, the query model and access patterns would be
fundamentally different. Designing the REST API without considering these
potential evolutions may limit flexibility. I'd suggest to start with the
use case.

Yufei


On Fri, Feb 27, 2026 at 3:42 PM Dmitri Bourlatchkov <di...@apache.org>
wrote:

> Hi Anand,
>
> Sharing my view... subject to discussion:
>
> 1. Adding non-IRC REST API to Polaris is perfectly fine.
>
> Figuring out specific endpoint URIs and payloads might require a few
> roundtrips, so opening a separate thread for that might be best.
> Contributors commonly create Google Docs for new API proposals too (they
> fairly easy to update as the email discussion progresses).
>
> There was a suggestion to try Markdown (with PRs) for proposals [1] ...
> feel free to give it a try if you are comfortable with that.
>
> 2. Could you clarify whether you mean end user utilities or admin user
> utilities? In the latter case those might be more suitable for the Admin
> CLI (java) not the Python CLI, IMHO.
>
> Why would these utilities be common with events? IMHO, event use cases are
> distinct from scan/commit metrics.
>
> 3. I'd prefer separating metrics persistence from MetaStore persistence at
> the code level, so that they could be mixed and matched independently. The
> separate datasource question will become a non-issue with that approach, I
> guess.
>
> The rationale for separating scan metrics and metastore persistence is that
> "cascading deletes" between them are hardly ever required. Furthermore, the
> data and query patterns are very different so different technologies might
> be beneficial in each case.
>
> [1] https://urldefense.com/v3/__https://lists.apache.org/thread/yto2wp982t43h1mqjwnslswhws5z47cy__;!!Iz9xO38YGHZK!5EuyFFkk3vhRWVIRvQAWBSQfpJkTMA9HxugzDwXmN0LPPqhEFxYkFRGVhtb8AqUwXtDh2OplcMnbMDHKxYDakNU$
>
> Cheers,
> Dmitri.
>
> On Fri, Feb 27, 2026 at 6:19 PM Anand Kumar Sankaran via dev <
> dev@polaris.apache.org> wrote:
>
> > Thanks all. This PR is merged now.
> >
> > Here are the follow-up features / work needed.  These were all part of
> the
> > merged PR at some point in time and were removed to reduce scope.
> >
> > Please let me know what you think.
> >
> >
> >   1.  A REST API to paginate through table metrics. This will be non-IRC
> > standard addition.
> >   2.  Utilities for managing old records, should be common with events.
> > There was some discussion that it belongs to the CLI.
> >   3.  Separate datasource (metrics, events, even other tables?).
> >
> >
> > Anything else?
> >
> > -
> > Anand
> >
> >
>

Re: Polaris Telemetry and Audit Trail

[Yufei Gu <fl...@gmail.com>]

Persisting only the update event is likely insufficient. What we really
care about here is the snapshot summary, which could appear as part of the
update event payload. However, it isn't clear whether event persistence
should always include the full payload ATM. If the payload becomes large,
it could cause the event table to grow very quickly.
Snapshot summaries and events also serve different purposes. Snapshot
summaries behave more like table metrics. For example, when users want to
understand trends such as table total storage size or number of rows over
time, they typically care about the metrics rather than the events
themselves. With that in mind, one approach is to persist the snapshot
summary as a type of metric, while allowing events to decide independently
whether to persist their payload. This would likely require making the
metrics persistence model more general so it can support different kinds of
metrics.

Yufei


On Sat, Mar 7, 2026 at 6:49 PM Anand Kumar Sankaran via dev <
dev@polaris.apache.org> wrote:

> Hi Yufei,
>
> If I understand you correctly, persisting the table_update event a
> querying it is sufficient?  I need the REST API for querying events at
> least.
>
> From: Yufei Gu <fl...@gmail.com>
> Date: Tuesday, March 3, 2026 at 11:58 AM
> To: dev@polaris.apache.org <de...@polaris.apache.org>
> Subject: Re: Polaris Telemetry and Audit Trail
>
> This Message Is From an External Sender
> This message came from outside your organization.
> Report Suspicious<
> https://us-phishalarm-ewt.proofpoint.com/EWT/v1/Iz9xO38YGHZK!YhNDZABkHi1B6h3NWydMnvDefAVhkBMCV4BSvMNI3goQW1uLJNEoJnS5coa77cngCWnC3sb-Hb42HlfhGHZ-uRj5S4JsOlj7IssK7yvqxN_n-k9CTPapLc7vIm8QxiJX$
> >
>
>
> > I want to be able to query commit metrics to track files added / removed
> per commit, along with record counts. The ingestion pipeline that writes
> this data is owned by us and we are guaranteed to write this information
> for each write.
>
> I think the snapshot summary would be more reliable for tracking them. The
> problem with the commit report metrics is that they are not necessarily
> committed. A commit failure can easily happen after the commit metrics
> report is sent to Polaris. The snapshot summary doesn't have this issue. A
> valid snapshot is a successful commit. To get them, users only need to load
> the table. One minor problem with snapshot summaries is that the snapshot
> itself could expire over time. To mitigate it, either users can persist
> them by leveraging table update event or Polaris can provide a solution to
> host the historical snapshot summary.
>
> Yufei
>
>
> On Tue, Mar 3, 2026 at 9:47 AM Dmitri Bourlatchkov <di...@apache.org>
> wrote:
>
> > Hi Anand,
> >
> > Your proposal for the new REST API to query scan/commit metrics makes
> sense
> > to me.
> >
> > My only request is to start a dedicated "[DISCUSS]" thread for it, just
> for
> > the sake of clarity since this thread is already pretty long, and all the
> > related PRs previously mentioned here have been merged. WDYT?
> >
> > Whether to create an accompanying Google Doc for the new REST API is up
> to
> > you.
> >
> > From my personal POV, a new email thread plus a GH PR with the API spec
> > would be a good start.
> >
> > Thanks,
> > Dmitri.
> >
> > On Mon, Mar 2, 2026 at 1:26 PM Anand Kumar Sankaran via dev <
> > dev@polaris.apache.org> wrote:
> >
> > > About the REST API, based on my use cases:
> > >
> > >
> > >   1.
> > > I want to be able to query commit metrics to track files added /
> removed
> > > per commit, along with record counts. The ingestion pipeline that
> writes
> > > this data is owned by us and we are guaranteed to write this
> information
> > > for each write.
> > >   2.
> > > I want to be able to query scan metrics for read. I understand clients
> do
> > > not fulfill this requirement.
> > >   3.
> > > I want to be able to query the events table (events are persisted) -
> this
> > > may supersede #2, I am not sure yet.
> > >
> > > All this information is in the JDBC based persistence model and is
> > > persisted in the metastore. I currently don’t have a need to query
> > > prometheus or open telemetry. I do publish some events to Prometheus
> and
> > > they are forwarded to our dashboards elsewhere.
> > >
> > > About the CLI utilities, I meant the admin user utilities. In one of
> the
> > > earliest drafts of my proposal, Prashant mentioned that the metrics
> > tables
> > > can grow indefinitely and that a similar problem exists with the events
> > > table as well. We discussed that cleaning up of old records from both
> > > metrics tables and events tables can be done via a CLI utility.
> > >
> > > I see that Yufei has covered the discussion about datasources.
> > >
> > > -
> > > Anand
> > >
> > >
> > >
> > > From: Yufei Gu <fl...@gmail.com>
> > > Date: Friday, February 27, 2026 at 9:54 PM
> > > To: dev@polaris.apache.org <de...@polaris.apache.org>
> > > Subject: Re: Polaris Telemetry and Audit Trail
> > >
> > > This Message Is From an External Sender
> > > This message came from outside your organization.
> > > Report Suspicious<
> > >
> >
> https://us-phishalarm-ewt.proofpoint.com/EWT/v1/Iz9xO38YGHZK!YhNDZABkHi1B699ote2uMwpOZw8i0QMCGO2Szc-HshuABGhGvwPJcymE6G2oUUxtS8xDkSrtGTPm_I3QnVDHoLMk50m9v8z_nZKTkd-bnVUbreF1u0WnfV_X5eYevZl_$
> > > >
> > >
> > >
> > > As I mentioned in
> > >
> >
> https://urldefense.com/v3/__https://github.com/apache/polaris/issues/3890__;!!Iz9xO38YGHZK!5EuyFFkk3vhRWVIRvQAWBSQfpJkTMA9HxugzDwXmN0LPPqhEFxYkFRGVhtb8AqUwXtDh2OplcMnbMDHKOxrvDU0$
> >,
> > > supporting
> > > multiple data sources is not a trivial change. I would strongly
> recommend
> > > starting with a design document to carefully evaluate the architectural
> > > implications and long term impact.
> > >
> > > A REST endpoint to query metrics seems reasonable given the current
> JDBC
> > > based persistence model. That said, we may also consider alternative
> > > storage models. For example, if we later adopt a time series system
> such
> > as
> > > Prometheus to store metrics, the query model and access patterns would
> be
> > > fundamentally different. Designing the REST API without considering
> these
> > > potential evolutions may limit flexibility. I'd suggest to start with
> the
> > > use case.
> > >
> > > Yufei
> > >
> > >
> > > On Fri, Feb 27, 2026 at 3:42 PM Dmitri Bourlatchkov <di...@apache.org>
> > > wrote:
> > >
> > > > Hi Anand,
> > > >
> > > > Sharing my view... subject to discussion:
> > > >
> > > > 1. Adding non-IRC REST API to Polaris is perfectly fine.
> > > >
> > > > Figuring out specific endpoint URIs and payloads might require a few
> > > > roundtrips, so opening a separate thread for that might be best.
> > > > Contributors commonly create Google Docs for new API proposals too
> > (they
> > > > fairly easy to update as the email discussion progresses).
> > > >
> > > > There was a suggestion to try Markdown (with PRs) for proposals [1]
> ...
> > > > feel free to give it a try if you are comfortable with that.
> > > >
> > > > 2. Could you clarify whether you mean end user utilities or admin
> user
> > > > utilities? In the latter case those might be more suitable for the
> > Admin
> > > > CLI (java) not the Python CLI, IMHO.
> > > >
> > > > Why would these utilities be common with events? IMHO, event use
> cases
> > > are
> > > > distinct from scan/commit metrics.
> > > >
> > > > 3. I'd prefer separating metrics persistence from MetaStore
> persistence
> > > at
> > > > the code level, so that they could be mixed and matched
> independently.
> > > The
> > > > separate datasource question will become a non-issue with that
> > approach,
> > > I
> > > > guess.
> > > >
> > > > The rationale for separating scan metrics and metastore persistence
> is
> > > that
> > > > "cascading deletes" between them are hardly ever required.
> Furthermore,
> > > the
> > > > data and query patterns are very different so different technologies
> > > might
> > > > be beneficial in each case.
> > > >
> > > > [1]
> > >
> >
> https://urldefense.com/v3/__https://lists.apache.org/thread/yto2wp982t43h1mqjwnslswhws5z47cy__;!!Iz9xO38YGHZK!5EuyFFkk3vhRWVIRvQAWBSQfpJkTMA9HxugzDwXmN0LPPqhEFxYkFRGVhtb8AqUwXtDh2OplcMnbMDHKxYDakNU$
> >> >
> > > > Cheers,
> > > > Dmitri.
> > > >
> > > > On Fri, Feb 27, 2026 at 6:19 PM Anand Kumar Sankaran via dev <
> > > > dev@polaris.apache.org> wrote:
> > > >
> > > > > Thanks all. This PR is merged now.
> > > > >
> > > > > Here are the follow-up features / work needed.  These were all part
> > of
> > > > the
> > > > > merged PR at some point in time and were removed to reduce scope.
> > > > >
> > > > > Please let me know what you think.
> > > > >
> > > > >
> > > > >   1.  A REST API to paginate through table metrics. This will be
> > > non-IRC
> > > > > standard addition.
> > > > >   2.  Utilities for managing old records, should be common with
> > events.
> > > > > There was some discussion that it belongs to the CLI.
> > > > >   3.  Separate datasource (metrics, events, even other tables?).
> > > > >
> > > > >
> > > > > Anything else?
> > > > >
> > > > > -
> > > > > Anand
> > > > >
> > > > >
> > > >
> > >
> > >
> >
>
>

Re: Polaris Telemetry and Audit Trail

[Anand Kumar Sankaran via dev <de...@polaris.apache.org>]

Hi Yufei,

If I understand you correctly, persisting the table_update event a querying it is sufficient?  I need the REST API for querying events at least.

From: Yufei Gu <fl...@gmail.com>
Date: Tuesday, March 3, 2026 at 11:58 AM
To: dev@polaris.apache.org <de...@polaris.apache.org>
Subject: Re: Polaris Telemetry and Audit Trail

This Message Is From an External Sender
This message came from outside your organization.
Report Suspicious<https://us-phishalarm-ewt.proofpoint.com/EWT/v1/Iz9xO38YGHZK!YhNDZABkHi1B6h3NWydMnvDefAVhkBMCV4BSvMNI3goQW1uLJNEoJnS5coa77cngCWnC3sb-Hb42HlfhGHZ-uRj5S4JsOlj7IssK7yvqxN_n-k9CTPapLc7vIm8QxiJX$>


> I want to be able to query commit metrics to track files added / removed
per commit, along with record counts. The ingestion pipeline that writes
this data is owned by us and we are guaranteed to write this information
for each write.

I think the snapshot summary would be more reliable for tracking them. The
problem with the commit report metrics is that they are not necessarily
committed. A commit failure can easily happen after the commit metrics
report is sent to Polaris. The snapshot summary doesn't have this issue. A
valid snapshot is a successful commit. To get them, users only need to load
the table. One minor problem with snapshot summaries is that the snapshot
itself could expire over time. To mitigate it, either users can persist
them by leveraging table update event or Polaris can provide a solution to
host the historical snapshot summary.

Yufei


On Tue, Mar 3, 2026 at 9:47 AM Dmitri Bourlatchkov <di...@apache.org> wrote:

> Hi Anand,
>
> Your proposal for the new REST API to query scan/commit metrics makes sense
> to me.
>
> My only request is to start a dedicated "[DISCUSS]" thread for it, just for
> the sake of clarity since this thread is already pretty long, and all the
> related PRs previously mentioned here have been merged. WDYT?
>
> Whether to create an accompanying Google Doc for the new REST API is up to
> you.
>
> From my personal POV, a new email thread plus a GH PR with the API spec
> would be a good start.
>
> Thanks,
> Dmitri.
>
> On Mon, Mar 2, 2026 at 1:26 PM Anand Kumar Sankaran via dev <
> dev@polaris.apache.org> wrote:
>
> > About the REST API, based on my use cases:
> >
> >
> >   1.
> > I want to be able to query commit metrics to track files added / removed
> > per commit, along with record counts. The ingestion pipeline that writes
> > this data is owned by us and we are guaranteed to write this information
> > for each write.
> >   2.
> > I want to be able to query scan metrics for read. I understand clients do
> > not fulfill this requirement.
> >   3.
> > I want to be able to query the events table (events are persisted) - this
> > may supersede #2, I am not sure yet.
> >
> > All this information is in the JDBC based persistence model and is
> > persisted in the metastore. I currently don’t have a need to query
> > prometheus or open telemetry. I do publish some events to Prometheus and
> > they are forwarded to our dashboards elsewhere.
> >
> > About the CLI utilities, I meant the admin user utilities. In one of the
> > earliest drafts of my proposal, Prashant mentioned that the metrics
> tables
> > can grow indefinitely and that a similar problem exists with the events
> > table as well. We discussed that cleaning up of old records from both
> > metrics tables and events tables can be done via a CLI utility.
> >
> > I see that Yufei has covered the discussion about datasources.
> >
> > -
> > Anand
> >
> >
> >
> > From: Yufei Gu <fl...@gmail.com>
> > Date: Friday, February 27, 2026 at 9:54 PM
> > To: dev@polaris.apache.org <de...@polaris.apache.org>
> > Subject: Re: Polaris Telemetry and Audit Trail
> >
> > This Message Is From an External Sender
> > This message came from outside your organization.
> > Report Suspicious<
> >
> https://us-phishalarm-ewt.proofpoint.com/EWT/v1/Iz9xO38YGHZK!YhNDZABkHi1B699ote2uMwpOZw8i0QMCGO2Szc-HshuABGhGvwPJcymE6G2oUUxtS8xDkSrtGTPm_I3QnVDHoLMk50m9v8z_nZKTkd-bnVUbreF1u0WnfV_X5eYevZl_$
> > >
> >
> >
> > As I mentioned in
> >
> https://urldefense.com/v3/__https://github.com/apache/polaris/issues/3890__;!!Iz9xO38YGHZK!5EuyFFkk3vhRWVIRvQAWBSQfpJkTMA9HxugzDwXmN0LPPqhEFxYkFRGVhtb8AqUwXtDh2OplcMnbMDHKOxrvDU0$
>,
> > supporting
> > multiple data sources is not a trivial change. I would strongly recommend
> > starting with a design document to carefully evaluate the architectural
> > implications and long term impact.
> >
> > A REST endpoint to query metrics seems reasonable given the current JDBC
> > based persistence model. That said, we may also consider alternative
> > storage models. For example, if we later adopt a time series system such
> as
> > Prometheus to store metrics, the query model and access patterns would be
> > fundamentally different. Designing the REST API without considering these
> > potential evolutions may limit flexibility. I'd suggest to start with the
> > use case.
> >
> > Yufei
> >
> >
> > On Fri, Feb 27, 2026 at 3:42 PM Dmitri Bourlatchkov <di...@apache.org>
> > wrote:
> >
> > > Hi Anand,
> > >
> > > Sharing my view... subject to discussion:
> > >
> > > 1. Adding non-IRC REST API to Polaris is perfectly fine.
> > >
> > > Figuring out specific endpoint URIs and payloads might require a few
> > > roundtrips, so opening a separate thread for that might be best.
> > > Contributors commonly create Google Docs for new API proposals too
> (they
> > > fairly easy to update as the email discussion progresses).
> > >
> > > There was a suggestion to try Markdown (with PRs) for proposals [1] ...
> > > feel free to give it a try if you are comfortable with that.
> > >
> > > 2. Could you clarify whether you mean end user utilities or admin user
> > > utilities? In the latter case those might be more suitable for the
> Admin
> > > CLI (java) not the Python CLI, IMHO.
> > >
> > > Why would these utilities be common with events? IMHO, event use cases
> > are
> > > distinct from scan/commit metrics.
> > >
> > > 3. I'd prefer separating metrics persistence from MetaStore persistence
> > at
> > > the code level, so that they could be mixed and matched independently.
> > The
> > > separate datasource question will become a non-issue with that
> approach,
> > I
> > > guess.
> > >
> > > The rationale for separating scan metrics and metastore persistence is
> > that
> > > "cascading deletes" between them are hardly ever required. Furthermore,
> > the
> > > data and query patterns are very different so different technologies
> > might
> > > be beneficial in each case.
> > >
> > > [1]
> >
> https://urldefense.com/v3/__https://lists.apache.org/thread/yto2wp982t43h1mqjwnslswhws5z47cy__;!!Iz9xO38YGHZK!5EuyFFkk3vhRWVIRvQAWBSQfpJkTMA9HxugzDwXmN0LPPqhEFxYkFRGVhtb8AqUwXtDh2OplcMnbMDHKxYDakNU$
>> >
> > > Cheers,
> > > Dmitri.
> > >
> > > On Fri, Feb 27, 2026 at 6:19 PM Anand Kumar Sankaran via dev <
> > > dev@polaris.apache.org> wrote:
> > >
> > > > Thanks all. This PR is merged now.
> > > >
> > > > Here are the follow-up features / work needed.  These were all part
> of
> > > the
> > > > merged PR at some point in time and were removed to reduce scope.
> > > >
> > > > Please let me know what you think.
> > > >
> > > >
> > > >   1.  A REST API to paginate through table metrics. This will be
> > non-IRC
> > > > standard addition.
> > > >   2.  Utilities for managing old records, should be common with
> events.
> > > > There was some discussion that it belongs to the CLI.
> > > >   3.  Separate datasource (metrics, events, even other tables?).
> > > >
> > > >
> > > > Anything else?
> > > >
> > > > -
> > > > Anand
> > > >
> > > >
> > >
> >
> >
>

Re: Polaris Telemetry and Audit Trail

[Yufei Gu <fl...@gmail.com>]

> I want to be able to query commit metrics to track files added / removed
per commit, along with record counts. The ingestion pipeline that writes
this data is owned by us and we are guaranteed to write this information
for each write.

I think the snapshot summary would be more reliable for tracking them. The
problem with the commit report metrics is that they are not necessarily
committed. A commit failure can easily happen after the commit metrics
report is sent to Polaris. The snapshot summary doesn't have this issue. A
valid snapshot is a successful commit. To get them, users only need to load
the table. One minor problem with snapshot summaries is that the snapshot
itself could expire over time. To mitigate it, either users can persist
them by leveraging table update event or Polaris can provide a solution to
host the historical snapshot summary.

Yufei


On Tue, Mar 3, 2026 at 9:47 AM Dmitri Bourlatchkov <di...@apache.org> wrote:

> Hi Anand,
>
> Your proposal for the new REST API to query scan/commit metrics makes sense
> to me.
>
> My only request is to start a dedicated "[DISCUSS]" thread for it, just for
> the sake of clarity since this thread is already pretty long, and all the
> related PRs previously mentioned here have been merged. WDYT?
>
> Whether to create an accompanying Google Doc for the new REST API is up to
> you.
>
> From my personal POV, a new email thread plus a GH PR with the API spec
> would be a good start.
>
> Thanks,
> Dmitri.
>
> On Mon, Mar 2, 2026 at 1:26 PM Anand Kumar Sankaran via dev <
> dev@polaris.apache.org> wrote:
>
> > About the REST API, based on my use cases:
> >
> >
> >   1.
> > I want to be able to query commit metrics to track files added / removed
> > per commit, along with record counts. The ingestion pipeline that writes
> > this data is owned by us and we are guaranteed to write this information
> > for each write.
> >   2.
> > I want to be able to query scan metrics for read. I understand clients do
> > not fulfill this requirement.
> >   3.
> > I want to be able to query the events table (events are persisted) - this
> > may supersede #2, I am not sure yet.
> >
> > All this information is in the JDBC based persistence model and is
> > persisted in the metastore. I currently don’t have a need to query
> > prometheus or open telemetry. I do publish some events to Prometheus and
> > they are forwarded to our dashboards elsewhere.
> >
> > About the CLI utilities, I meant the admin user utilities. In one of the
> > earliest drafts of my proposal, Prashant mentioned that the metrics
> tables
> > can grow indefinitely and that a similar problem exists with the events
> > table as well. We discussed that cleaning up of old records from both
> > metrics tables and events tables can be done via a CLI utility.
> >
> > I see that Yufei has covered the discussion about datasources.
> >
> > -
> > Anand
> >
> >
> >
> > From: Yufei Gu <fl...@gmail.com>
> > Date: Friday, February 27, 2026 at 9:54 PM
> > To: dev@polaris.apache.org <de...@polaris.apache.org>
> > Subject: Re: Polaris Telemetry and Audit Trail
> >
> > This Message Is From an External Sender
> > This message came from outside your organization.
> > Report Suspicious<
> >
> https://us-phishalarm-ewt.proofpoint.com/EWT/v1/Iz9xO38YGHZK!YhNDZABkHi1B699ote2uMwpOZw8i0QMCGO2Szc-HshuABGhGvwPJcymE6G2oUUxtS8xDkSrtGTPm_I3QnVDHoLMk50m9v8z_nZKTkd-bnVUbreF1u0WnfV_X5eYevZl_$
> > >
> >
> >
> > As I mentioned in
> >
> https://urldefense.com/v3/__https://github.com/apache/polaris/issues/3890__;!!Iz9xO38YGHZK!5EuyFFkk3vhRWVIRvQAWBSQfpJkTMA9HxugzDwXmN0LPPqhEFxYkFRGVhtb8AqUwXtDh2OplcMnbMDHKOxrvDU0$
> ,
> > supporting
> > multiple data sources is not a trivial change. I would strongly recommend
> > starting with a design document to carefully evaluate the architectural
> > implications and long term impact.
> >
> > A REST endpoint to query metrics seems reasonable given the current JDBC
> > based persistence model. That said, we may also consider alternative
> > storage models. For example, if we later adopt a time series system such
> as
> > Prometheus to store metrics, the query model and access patterns would be
> > fundamentally different. Designing the REST API without considering these
> > potential evolutions may limit flexibility. I'd suggest to start with the
> > use case.
> >
> > Yufei
> >
> >
> > On Fri, Feb 27, 2026 at 3:42 PM Dmitri Bourlatchkov <di...@apache.org>
> > wrote:
> >
> > > Hi Anand,
> > >
> > > Sharing my view... subject to discussion:
> > >
> > > 1. Adding non-IRC REST API to Polaris is perfectly fine.
> > >
> > > Figuring out specific endpoint URIs and payloads might require a few
> > > roundtrips, so opening a separate thread for that might be best.
> > > Contributors commonly create Google Docs for new API proposals too
> (they
> > > fairly easy to update as the email discussion progresses).
> > >
> > > There was a suggestion to try Markdown (with PRs) for proposals [1] ...
> > > feel free to give it a try if you are comfortable with that.
> > >
> > > 2. Could you clarify whether you mean end user utilities or admin user
> > > utilities? In the latter case those might be more suitable for the
> Admin
> > > CLI (java) not the Python CLI, IMHO.
> > >
> > > Why would these utilities be common with events? IMHO, event use cases
> > are
> > > distinct from scan/commit metrics.
> > >
> > > 3. I'd prefer separating metrics persistence from MetaStore persistence
> > at
> > > the code level, so that they could be mixed and matched independently.
> > The
> > > separate datasource question will become a non-issue with that
> approach,
> > I
> > > guess.
> > >
> > > The rationale for separating scan metrics and metastore persistence is
> > that
> > > "cascading deletes" between them are hardly ever required. Furthermore,
> > the
> > > data and query patterns are very different so different technologies
> > might
> > > be beneficial in each case.
> > >
> > > [1]
> >
> https://urldefense.com/v3/__https://lists.apache.org/thread/yto2wp982t43h1mqjwnslswhws5z47cy__;!!Iz9xO38YGHZK!5EuyFFkk3vhRWVIRvQAWBSQfpJkTMA9HxugzDwXmN0LPPqhEFxYkFRGVhtb8AqUwXtDh2OplcMnbMDHKxYDakNU$
> > >
> > > Cheers,
> > > Dmitri.
> > >
> > > On Fri, Feb 27, 2026 at 6:19 PM Anand Kumar Sankaran via dev <
> > > dev@polaris.apache.org> wrote:
> > >
> > > > Thanks all. This PR is merged now.
> > > >
> > > > Here are the follow-up features / work needed.  These were all part
> of
> > > the
> > > > merged PR at some point in time and were removed to reduce scope.
> > > >
> > > > Please let me know what you think.
> > > >
> > > >
> > > >   1.  A REST API to paginate through table metrics. This will be
> > non-IRC
> > > > standard addition.
> > > >   2.  Utilities for managing old records, should be common with
> events.
> > > > There was some discussion that it belongs to the CLI.
> > > >   3.  Separate datasource (metrics, events, even other tables?).
> > > >
> > > >
> > > > Anything else?
> > > >
> > > > -
> > > > Anand
> > > >
> > > >
> > >
> >
> >
>

Re: Polaris Telemetry and Audit Trail

[Dmitri Bourlatchkov <di...@apache.org>]

Hi Anand,

Your proposal for the new REST API to query scan/commit metrics makes sense
to me.

My only request is to start a dedicated "[DISCUSS]" thread for it, just for
the sake of clarity since this thread is already pretty long, and all the
related PRs previously mentioned here have been merged. WDYT?

Whether to create an accompanying Google Doc for the new REST API is up to
you.

From my personal POV, a new email thread plus a GH PR with the API spec
would be a good start.

Thanks,
Dmitri.

On Mon, Mar 2, 2026 at 1:26 PM Anand Kumar Sankaran via dev <
dev@polaris.apache.org> wrote:

> About the REST API, based on my use cases:
>
>
>   1.
> I want to be able to query commit metrics to track files added / removed
> per commit, along with record counts. The ingestion pipeline that writes
> this data is owned by us and we are guaranteed to write this information
> for each write.
>   2.
> I want to be able to query scan metrics for read. I understand clients do
> not fulfill this requirement.
>   3.
> I want to be able to query the events table (events are persisted) - this
> may supersede #2, I am not sure yet.
>
> All this information is in the JDBC based persistence model and is
> persisted in the metastore. I currently don’t have a need to query
> prometheus or open telemetry. I do publish some events to Prometheus and
> they are forwarded to our dashboards elsewhere.
>
> About the CLI utilities, I meant the admin user utilities. In one of the
> earliest drafts of my proposal, Prashant mentioned that the metrics tables
> can grow indefinitely and that a similar problem exists with the events
> table as well. We discussed that cleaning up of old records from both
> metrics tables and events tables can be done via a CLI utility.
>
> I see that Yufei has covered the discussion about datasources.
>
> -
> Anand
>
>
>
> From: Yufei Gu <fl...@gmail.com>
> Date: Friday, February 27, 2026 at 9:54 PM
> To: dev@polaris.apache.org <de...@polaris.apache.org>
> Subject: Re: Polaris Telemetry and Audit Trail
>
> This Message Is From an External Sender
> This message came from outside your organization.
> Report Suspicious<
> https://us-phishalarm-ewt.proofpoint.com/EWT/v1/Iz9xO38YGHZK!YhNDZABkHi1B699ote2uMwpOZw8i0QMCGO2Szc-HshuABGhGvwPJcymE6G2oUUxtS8xDkSrtGTPm_I3QnVDHoLMk50m9v8z_nZKTkd-bnVUbreF1u0WnfV_X5eYevZl_$
> >
>
>
> As I mentioned in
> https://urldefense.com/v3/__https://github.com/apache/polaris/issues/3890__;!!Iz9xO38YGHZK!5EuyFFkk3vhRWVIRvQAWBSQfpJkTMA9HxugzDwXmN0LPPqhEFxYkFRGVhtb8AqUwXtDh2OplcMnbMDHKOxrvDU0$,
> supporting
> multiple data sources is not a trivial change. I would strongly recommend
> starting with a design document to carefully evaluate the architectural
> implications and long term impact.
>
> A REST endpoint to query metrics seems reasonable given the current JDBC
> based persistence model. That said, we may also consider alternative
> storage models. For example, if we later adopt a time series system such as
> Prometheus to store metrics, the query model and access patterns would be
> fundamentally different. Designing the REST API without considering these
> potential evolutions may limit flexibility. I'd suggest to start with the
> use case.
>
> Yufei
>
>
> On Fri, Feb 27, 2026 at 3:42 PM Dmitri Bourlatchkov <di...@apache.org>
> wrote:
>
> > Hi Anand,
> >
> > Sharing my view... subject to discussion:
> >
> > 1. Adding non-IRC REST API to Polaris is perfectly fine.
> >
> > Figuring out specific endpoint URIs and payloads might require a few
> > roundtrips, so opening a separate thread for that might be best.
> > Contributors commonly create Google Docs for new API proposals too (they
> > fairly easy to update as the email discussion progresses).
> >
> > There was a suggestion to try Markdown (with PRs) for proposals [1] ...
> > feel free to give it a try if you are comfortable with that.
> >
> > 2. Could you clarify whether you mean end user utilities or admin user
> > utilities? In the latter case those might be more suitable for the Admin
> > CLI (java) not the Python CLI, IMHO.
> >
> > Why would these utilities be common with events? IMHO, event use cases
> are
> > distinct from scan/commit metrics.
> >
> > 3. I'd prefer separating metrics persistence from MetaStore persistence
> at
> > the code level, so that they could be mixed and matched independently.
> The
> > separate datasource question will become a non-issue with that approach,
> I
> > guess.
> >
> > The rationale for separating scan metrics and metastore persistence is
> that
> > "cascading deletes" between them are hardly ever required. Furthermore,
> the
> > data and query patterns are very different so different technologies
> might
> > be beneficial in each case.
> >
> > [1]
> https://urldefense.com/v3/__https://lists.apache.org/thread/yto2wp982t43h1mqjwnslswhws5z47cy__;!!Iz9xO38YGHZK!5EuyFFkk3vhRWVIRvQAWBSQfpJkTMA9HxugzDwXmN0LPPqhEFxYkFRGVhtb8AqUwXtDh2OplcMnbMDHKxYDakNU$
> >
> > Cheers,
> > Dmitri.
> >
> > On Fri, Feb 27, 2026 at 6:19 PM Anand Kumar Sankaran via dev <
> > dev@polaris.apache.org> wrote:
> >
> > > Thanks all. This PR is merged now.
> > >
> > > Here are the follow-up features / work needed.  These were all part of
> > the
> > > merged PR at some point in time and were removed to reduce scope.
> > >
> > > Please let me know what you think.
> > >
> > >
> > >   1.  A REST API to paginate through table metrics. This will be
> non-IRC
> > > standard addition.
> > >   2.  Utilities for managing old records, should be common with events.
> > > There was some discussion that it belongs to the CLI.
> > >   3.  Separate datasource (metrics, events, even other tables?).
> > >
> > >
> > > Anything else?
> > >
> > > -
> > > Anand
> > >
> > >
> >
>
>

Re: Polaris Telemetry and Audit Trail

[Anand Kumar Sankaran via dev <de...@polaris.apache.org>]

About the REST API, based on my use cases:


  1.
I want to be able to query commit metrics to track files added / removed per commit, along with record counts. The ingestion pipeline that writes this data is owned by us and we are guaranteed to write this information for each write.
  2.
I want to be able to query scan metrics for read. I understand clients do not fulfill this requirement.
  3.
I want to be able to query the events table (events are persisted) - this may supersede #2, I am not sure yet.

All this information is in the JDBC based persistence model and is persisted in the metastore. I currently don’t have a need to query prometheus or open telemetry. I do publish some events to Prometheus and they are forwarded to our dashboards elsewhere.

About the CLI utilities, I meant the admin user utilities. In one of the earliest drafts of my proposal, Prashant mentioned that the metrics tables can grow indefinitely and that a similar problem exists with the events table as well. We discussed that cleaning up of old records from both metrics tables and events tables can be done via a CLI utility.

I see that Yufei has covered the discussion about datasources.

-
Anand



From: Yufei Gu <fl...@gmail.com>
Date: Friday, February 27, 2026 at 9:54 PM
To: dev@polaris.apache.org <de...@polaris.apache.org>
Subject: Re: Polaris Telemetry and Audit Trail

This Message Is From an External Sender
This message came from outside your organization.
Report Suspicious<https://us-phishalarm-ewt.proofpoint.com/EWT/v1/Iz9xO38YGHZK!YhNDZABkHi1B699ote2uMwpOZw8i0QMCGO2Szc-HshuABGhGvwPJcymE6G2oUUxtS8xDkSrtGTPm_I3QnVDHoLMk50m9v8z_nZKTkd-bnVUbreF1u0WnfV_X5eYevZl_$>


As I mentioned in https://urldefense.com/v3/__https://github.com/apache/polaris/issues/3890__;!!Iz9xO38YGHZK!5EuyFFkk3vhRWVIRvQAWBSQfpJkTMA9HxugzDwXmN0LPPqhEFxYkFRGVhtb8AqUwXtDh2OplcMnbMDHKOxrvDU0$, supporting
multiple data sources is not a trivial change. I would strongly recommend
starting with a design document to carefully evaluate the architectural
implications and long term impact.

A REST endpoint to query metrics seems reasonable given the current JDBC
based persistence model. That said, we may also consider alternative
storage models. For example, if we later adopt a time series system such as
Prometheus to store metrics, the query model and access patterns would be
fundamentally different. Designing the REST API without considering these
potential evolutions may limit flexibility. I'd suggest to start with the
use case.

Yufei


On Fri, Feb 27, 2026 at 3:42 PM Dmitri Bourlatchkov <di...@apache.org>
wrote:

> Hi Anand,
>
> Sharing my view... subject to discussion:
>
> 1. Adding non-IRC REST API to Polaris is perfectly fine.
>
> Figuring out specific endpoint URIs and payloads might require a few
> roundtrips, so opening a separate thread for that might be best.
> Contributors commonly create Google Docs for new API proposals too (they
> fairly easy to update as the email discussion progresses).
>
> There was a suggestion to try Markdown (with PRs) for proposals [1] ...
> feel free to give it a try if you are comfortable with that.
>
> 2. Could you clarify whether you mean end user utilities or admin user
> utilities? In the latter case those might be more suitable for the Admin
> CLI (java) not the Python CLI, IMHO.
>
> Why would these utilities be common with events? IMHO, event use cases are
> distinct from scan/commit metrics.
>
> 3. I'd prefer separating metrics persistence from MetaStore persistence at
> the code level, so that they could be mixed and matched independently. The
> separate datasource question will become a non-issue with that approach, I
> guess.
>
> The rationale for separating scan metrics and metastore persistence is that
> "cascading deletes" between them are hardly ever required. Furthermore, the
> data and query patterns are very different so different technologies might
> be beneficial in each case.
>
> [1] https://urldefense.com/v3/__https://lists.apache.org/thread/yto2wp982t43h1mqjwnslswhws5z47cy__;!!Iz9xO38YGHZK!5EuyFFkk3vhRWVIRvQAWBSQfpJkTMA9HxugzDwXmN0LPPqhEFxYkFRGVhtb8AqUwXtDh2OplcMnbMDHKxYDakNU$
>
> Cheers,
> Dmitri.
>
> On Fri, Feb 27, 2026 at 6:19 PM Anand Kumar Sankaran via dev <
> dev@polaris.apache.org> wrote:
>
> > Thanks all. This PR is merged now.
> >
> > Here are the follow-up features / work needed.  These were all part of
> the
> > merged PR at some point in time and were removed to reduce scope.
> >
> > Please let me know what you think.
> >
> >
> >   1.  A REST API to paginate through table metrics. This will be non-IRC
> > standard addition.
> >   2.  Utilities for managing old records, should be common with events.
> > There was some discussion that it belongs to the CLI.
> >   3.  Separate datasource (metrics, events, even other tables?).
> >
> >
> > Anything else?
> >
> > -
> > Anand
> >
> >
>

Re: Polaris Telemetry and Audit Trail

[Yufei Gu <fl...@gmail.com>]

As I mentioned in https://github.com/apache/polaris/issues/3890, supporting
multiple data sources is not a trivial change. I would strongly recommend
starting with a design document to carefully evaluate the architectural
implications and long term impact.

A REST endpoint to query metrics seems reasonable given the current JDBC
based persistence model. That said, we may also consider alternative
storage models. For example, if we later adopt a time series system such as
Prometheus to store metrics, the query model and access patterns would be
fundamentally different. Designing the REST API without considering these
potential evolutions may limit flexibility. I'd suggest to start with the
use case.

Yufei


On Fri, Feb 27, 2026 at 3:42 PM Dmitri Bourlatchkov <di...@apache.org>
wrote:

> Hi Anand,
>
> Sharing my view... subject to discussion:
>
> 1. Adding non-IRC REST API to Polaris is perfectly fine.
>
> Figuring out specific endpoint URIs and payloads might require a few
> roundtrips, so opening a separate thread for that might be best.
> Contributors commonly create Google Docs for new API proposals too (they
> fairly easy to update as the email discussion progresses).
>
> There was a suggestion to try Markdown (with PRs) for proposals [1] ...
> feel free to give it a try if you are comfortable with that.
>
> 2. Could you clarify whether you mean end user utilities or admin user
> utilities? In the latter case those might be more suitable for the Admin
> CLI (java) not the Python CLI, IMHO.
>
> Why would these utilities be common with events? IMHO, event use cases are
> distinct from scan/commit metrics.
>
> 3. I'd prefer separating metrics persistence from MetaStore persistence at
> the code level, so that they could be mixed and matched independently. The
> separate datasource question will become a non-issue with that approach, I
> guess.
>
> The rationale for separating scan metrics and metastore persistence is that
> "cascading deletes" between them are hardly ever required. Furthermore, the
> data and query patterns are very different so different technologies might
> be beneficial in each case.
>
> [1] https://lists.apache.org/thread/yto2wp982t43h1mqjwnslswhws5z47cy
>
> Cheers,
> Dmitri.
>
> On Fri, Feb 27, 2026 at 6:19 PM Anand Kumar Sankaran via dev <
> dev@polaris.apache.org> wrote:
>
> > Thanks all. This PR is merged now.
> >
> > Here are the follow-up features / work needed.  These were all part of
> the
> > merged PR at some point in time and were removed to reduce scope.
> >
> > Please let me know what you think.
> >
> >
> >   1.  A REST API to paginate through table metrics. This will be non-IRC
> > standard addition.
> >   2.  Utilities for managing old records, should be common with events.
> > There was some discussion that it belongs to the CLI.
> >   3.  Separate datasource (metrics, events, even other tables?).
> >
> >
> > Anything else?
> >
> > -
> > Anand
> >
> >
>

Re: Polaris Telemetry and Audit Trail

[Dmitri Bourlatchkov <di...@apache.org>]

Hi Anand,

Sharing my view... subject to discussion:

1. Adding non-IRC REST API to Polaris is perfectly fine.

Figuring out specific endpoint URIs and payloads might require a few
roundtrips, so opening a separate thread for that might be best.
Contributors commonly create Google Docs for new API proposals too (they
fairly easy to update as the email discussion progresses).

There was a suggestion to try Markdown (with PRs) for proposals [1] ...
feel free to give it a try if you are comfortable with that.

2. Could you clarify whether you mean end user utilities or admin user
utilities? In the latter case those might be more suitable for the Admin
CLI (java) not the Python CLI, IMHO.

Why would these utilities be common with events? IMHO, event use cases are
distinct from scan/commit metrics.

3. I'd prefer separating metrics persistence from MetaStore persistence at
the code level, so that they could be mixed and matched independently. The
separate datasource question will become a non-issue with that approach, I
guess.

The rationale for separating scan metrics and metastore persistence is that
"cascading deletes" between them are hardly ever required. Furthermore, the
data and query patterns are very different so different technologies might
be beneficial in each case.

[1] https://lists.apache.org/thread/yto2wp982t43h1mqjwnslswhws5z47cy

Cheers,
Dmitri.

On Fri, Feb 27, 2026 at 6:19 PM Anand Kumar Sankaran via dev <
dev@polaris.apache.org> wrote:

> Thanks all. This PR is merged now.
>
> Here are the follow-up features / work needed.  These were all part of the
> merged PR at some point in time and were removed to reduce scope.
>
> Please let me know what you think.
>
>
>   1.  A REST API to paginate through table metrics. This will be non-IRC
> standard addition.
>   2.  Utilities for managing old records, should be common with events.
> There was some discussion that it belongs to the CLI.
>   3.  Separate datasource (metrics, events, even other tables?).
>
>
> Anything else?
>
> -
> Anand
>
>

Re: Polaris Telemetry and Audit Trail

[Anand Kumar Sankaran via dev <de...@polaris.apache.org>]

Thanks all. This PR is merged now.

Here are the follow-up features / work needed.  These were all part of the merged PR at some point in time and were removed to reduce scope.

Please let me know what you think.


  1.  A REST API to paginate through table metrics. This will be non-IRC standard addition.
  2.  Utilities for managing old records, should be common with events. There was some discussion that it belongs to the CLI.
  3.  Separate datasource (metrics, events, even other tables?).


Anything else?

-
Anand

Re: Polaris Telemetry and Audit Trail

[Dmitri Bourlatchkov <di...@apache.org>]

Hi Yufei,

Using one datasource for now sounds reasonable to me.

Cheers,
Dmitri.

On Wed, Feb 25, 2026 at 3:37 PM Yufei Gu <fl...@gmail.com> wrote:

> Thanks a lot Anand and Dmitri for working on this!
>
> Supporting multiple data sources introduces significant complexity as I
> recorded in this issue, https://github.com/apache/polaris/issues/3890. It
> requires a well-thought-out design before we commit to it. I'd propose to
> leave it now and move forward with the single data source as the initial
> step. WDYT?
>
> Yufei
>
>
> On Tue, Feb 24, 2026 at 12:17 PM Anand Kumar Sankaran via dev <
> dev@polaris.apache.org> wrote:
>
> > Hi Dmitri
> >
> > Thanks for all your support again.
> >
> > I talked to Yufei and I have addressed his review feedback in this
> commit.
> >
> https://github.com/apache/polaris/pull/3385/changes/9e3d4d0f409720ab4005db176190ebde29a9d2b6
> >
> > Yufei and Prashant - can you please review these changes?  Dmitri has
> > already reviewed the PR and approved.
> >
> > -
> > Anand
> >
> > From: Dmitri Bourlatchkov <di...@apache.org>
> > Date: Tuesday, February 24, 2026 at 10:38 AM
> > To: dev@polaris.apache.org <de...@polaris.apache.org>
> > Cc: Anand Kumar Sankaran <an...@workday.com>
> > Subject: Re: Polaris Telemetry and Audit Trail
> >
> > This Message Is From an External Sender
> > This message came from outside your organization.
> > Report Suspicious<
> >
> https://us-phishalarm-ewt.proofpoint.com/EWT/v1/Iz9xO38YGHZK!YhNDZAGr2cumY1xO1e2JFuKSh-PdIU2YhFBQOvhapv4xXZ0ucXUR7TPwnqpAVpb05t786Kda3YPcEe-qOyYlMkJM8hj02m4Yv48r-c47jSruOAIe3mbeFZ5dBQBrfOUk$
> > >
> >
> > Hi Yufei,
> >
> > I see that you propose a different approach to schema evolution for
> > metrics in GH [1]
> >
> > Please share your view of this in light of the previous discussion
> > regarding combining the metrics persistence with different metastore
> > persistence options. It would be good to sync up on this email thread
> since
> > GH comments tend to be hidden on long PRs and not all readers may be able
> > to easily find them.
> >
> > [1] https://github.com/apache/polaris/pull/3385#issuecomment-3953906091<
> >
> https://urldefense.com/v3/__https://github.com/apache/polaris/pull/3385*issuecomment-3953906091__;Iw!!Iz9xO38YGHZK!-KOwW_42EOhdoNvlXd2GvBD6Z7u_HteBuHkQAow0Z11eX9-Q5LAWMr02rP-8OE8VlcwlG4hSnLKT8_s2eg$
> > >
> >
> > Thanks,
> > Dmitri.
> >
> > On Thu, Feb 5, 2026 at 3:47 PM Anand Kumar Sankaran via dev <
> > dev@polaris.apache.org<ma...@polaris.apache.org>> wrote:
> > Hi Dmitri
> >
> > The SPI is designed to be pluggable via
> > MetaStoreManagerFactory.getOrCreateMetricsPersistence()<
> >
> https://github.com/apache/polaris/pull/3385/changes#diff-0e34667d083d30cace067936b87c736201e5a8cb001aa0b12e203ee52cddae99R51
> > <
> >
> https://urldefense.com/v3/__https://github.com/apache/polaris/pull/3385/changes*diff-0e34667d083d30cace067936b87c736201e5a8cb001aa0b12e203ee52cddae99R51__;Iw!!Iz9xO38YGHZK!-KOwW_42EOhdoNvlXd2GvBD6Z7u_HteBuHkQAow0Z11eX9-Q5LAWMr02rP-8OE8VlcwlG4hSnLKXRU26ZA$
> >>.
> > The default returns MetricsPersistence.NOOP, and only JDBC with schema
> v4+
> > returns a functional implementation.
> >
> >  Your proposal makes sense to me. To allow NoSQL Entity Persistence +
> JDBC
> > Metrics Persistence would require:
> >
> >      1. Separate metrics schema files (e.g., metrics-schema-v1.sql) that
> > can be bootstrapped independently
> >      2. Separate datasource configuration for metrics (e.g.,
> > polaris.metrics.jdbc.*)
> >      3. A dedicated `MetricsPersistenceFactory` interface (separate from
> > MetaStoreManagerFactory)
> >      4. Updates to the Admin Tool bootstrap workflow to handle both
> schemas
> >
> > Does that sound reasonable?  If we do this, all this work should go into
> > https://github.com/apache/polaris/pull/3523<
> >
> https://urldefense.com/v3/__https://github.com/apache/polaris/pull/3523__;!!Iz9xO38YGHZK!-KOwW_42EOhdoNvlXd2GvBD6Z7u_HteBuHkQAow0Z11eX9-Q5LAWMr02rP-8OE8VlcwlG4hSnLIg5ApDFA$
> >
> > as one atomic commit, correct?
> >
> > —
> > Anand
> >
> >
> >
> > From: Dmitri Bourlatchkov <di...@apache.org>>
> > Date: Thursday, February 5, 2026 at 8:28 AM
> > To: Anand Kumar Sankaran <anand.sankaran@workday.com<mailto:
> > anand.sankaran@workday.com>>
> > Cc: dev@polaris.apache.org<ma...@polaris.apache.org> <
> > dev@polaris.apache.org<ma...@polaris.apache.org>>
> > Subject: Re: Polaris Telemetry and Audit Trail
> >
> > This Message Is From an External Sender
> > This message came from outside your organization.
> > Report Suspicious<
> >
> https://us-phishalarm-ewt.proofpoint.com/EWT/v1/Iz9xO38YGHZK!YhNDZAGr2cumbRcPlIOttKwvGr4f38TF-J2-LmOw8H1KsFpWaM3W5g28Oq5qW2gnw2-WHz0rsWhm7F7hSGykA9l01LKYGuCcCkexkHp1sUssM_IegCtjk_id9s891orv$
> > >
> >
> >
> > Hi Anand,
> >
> > Thanks for driving the SPI definition for scan/commits metrics.
> >
> > I did not think from this angle before, but now I suppose it might be
> worth
> > introducing a separate SQL schema for metrics, detached from the main
> JDBC
> > Persistence schema.
> >
> > My rationale is that the SPI may be exercised independently of the JDBC
> > Entity Persistence. I can imagine people using NoSQL Entity Persistence
> > plus JDBC Scan Metrics Persistence in the same deployment.
> >
> > Using a separate schema will add further work on the Bootstrap workflow
> > (Admin Tool), but I hope the end result is going to be more convenient
> and
> > versatile for end users.
> >
> > WDYT?
> >
> > Thanks,
> > Dmitri.
> >
> > On Thu, Jan 29, 2026 at 1:49 PM Anand Kumar Sankaran <
> > anand.sankaran@workday.com<ma...@workday.com>> wrote:
> >
> > > Hi Dmitri,
> > >
> > > Thank you for taking the time to review. I had pulled out the schema in
> > > its own PR (based on comments) here:
> > >
> >
> https://urldefense.com/v3/__https://github.com/apache/polaris/pull/3523__;!!Iz9xO38YGHZK!65BVFQDe_GtKbDW8LkNdzVS-WYewzWsSns5zFHKclbMU3vUy2UaTXLB5G-wql1yyqoTFyDc9Bt3GRsh6AA$
> > >
> > > I will update the document with the SPI and carve out another PR that
> > > focuses on the SPI.
> > >
> > > —
> > > Anand
> > >
> > > *From: *Dmitri Bourlatchkov <dmitri.bourlatchkov@dremio.com<mailto:
> > dmitri.bourlatchkov@dremio.com>>
> > > *Date: *Thursday, January 29, 2026 at 10:43 AM
> > > *To: *dev@polaris.apache.org<ma...@polaris.apache.org> <
> > dev@polaris.apache.org<ma...@polaris.apache.org>>
> > > *Cc: *Anand Kumar Sankaran <anand.sankaran@workday.com<mailto:
> > anand.sankaran@workday.com>>
> > > *Subject: *Re: Polaris Telemetry and Audit Trail
> > >
> > > This Message Is From an External Sender
> > > This message came from outside your organization.
> > > Report Suspicious
> > > <
> >
> https://us-phishalarm-ewt.proofpoint.com/EWT/v1/Iz9xO38YGHZK!YhNDZAGr2cumY1cAlUeoPrU2eNUPz9Wxq2PaG1t99vEN3v60gucxC3Qsk-xK09RndcW46WtNjMMjQZbaQ495h1D1g293F_uqLhPM8gDMlA6xRmo5b5uqrSh2v7QKHrhQ$
> > >
> > >
> > > Hi Anand,
> > >
> > > Thanks for making a proposal doc and starting this thread. Apologies
> for
> > > late comments.
> > >
> > > As you know I already reviewed a couple of related PRs, which had a
> > > smaller impact on the codebase, but now, I believe, we're coming to the
> > > core of this feature and it might need a deeper discussion.
> > >
> > > For the sake of awareness of other project members, I'd like to
> > > highlight some aspects by email.
> > >
> > > * Storing scan metrics in RDBMS in general looks like a
> > > reasonable approach to me. However, Polaris Persistence is an
> extensible
> > > mechanism. Other backends are possible, including privately developed
> > > backends.
> > >
> > > From this POV, I believe it might be preferable to start not with an
> > RDBMS
> > > schema, but with a java SPI outlining expected read/write operations
> and
> > > the data model. This will make it easier to assess impact on other
> > > Persistence implementations. Naturally, Polaris service code will have
> to
> > > be able to work seamlessly regardless of the backend impl. (including
> the
> > > do-nothing impl.).
> > >
> > > I see that some connection between the SQL schema and java code is made
> > in
> > > PR [3385], but it does not appear to offer a clean SPI that could be
> > > implemented by different Persistence backends. Would it be ok from your
> > > perspective to make another PR with just SPI code?
> > >
> > > * Regarding ad-hoc SQL queries - it is certainly a valid use case for
> > > custom code to query the Polaris database directly. However, those
> ad-hoc
> > > queries may require certain indexes, which may not have a relevant use
> > case
> > > in OSS code. In that case, I believe it would be preferable to manage
> the
> > > extra indexes in custom code (not in Polaris). Whether this is the case
> > or
> > > not currently, I cannot say with certainty, so apologies if this is a
> > false
> > > alarm... Having the java SPI available would be a huge aid to reasoning
> > > about the proposed SQL schema, I think.
> > >
> > > That said, I do not mean to complicate the development of this feature,
> > > just trying to structure it in a way that is hopefully easy to
> understand
> > > (including myself), maintain and expand later. If I missed something,
> > > please feel free to point it out.
> > >
> > > [3385]
> >
> https://urldefense.com/v3/__https://github.com/apache/polaris/pull/3385__;!!Iz9xO38YGHZK!65BVFQDe_GtKbDW8LkNdzVS-WYewzWsSns5zFHKclbMU3vUy2UaTXLB5G-wql1yyqoTFyDc9Bt2W-eKl7A$
> > > <
> >
> https://urldefense.com/v3/__https://github.com/apache/polaris/pull/3385__;!!Iz9xO38YGHZK!_dbBfwEnZEuPp_8qIuBDoYfWRgCZKiPDMg9ouDltqsHhr9hqCHVlF9IedSkNVBDWupbneU7inC9YK-THL-lTjWkkyZhI0OE$
> > >
> > >
> > >Thanks,
> > > Dmitri.
> > >
> > > On Wed, Jan 14, 2026 at 10:05 PM Anand Kumar Sankaran via dev <
> > > dev@polaris.apache.org<ma...@polaris.apache.org>> wrote:
> > >
> > > Hi Yufei,
> > >
> > > Gave commenting privileges to all.
> > >
> > > Thanks.
> > >
> > > —
> > > Anand
> > >
> > > From: Yufei Gu <fl...@gmail.com>>
> > > Date: Wednesday, January 14, 2026 at 6:13 PM
> > > To: dev@polaris.apache.org<ma...@polaris.apache.org> <
> > dev@polaris.apache.org<ma...@polaris.apache.org>>
> > > Cc: Anand Kumar Sankaran <anand.sankaran@workday.com<mailto:
> > anand.sankaran@workday.com>>
> > > Subject: Re: Polaris Telemetry and Audit Trail
> > >
> > > This Message Is From an External Sender
> > > This message came from outside your organization.
> > > Report Suspicious<
> > >
> >
> https://us-phishalarm-ewt.proofpoint.com/EWT/v1/Iz9xO38YGHZK!YhNDZABkHi1B6hyPVYUK0T-K5j4Aoqdrlu07UjBpJcpjOsz7Ie7d_DhPl-ywi50dfQz4O6LL-AHLCOQTQ0X7Le8yD2cF0ogquGFB8Aw8uQLgB0RL6Ezpa-pLDPA4ziXA$
> > > >
> > >
> > > Hi Anand, thanks for the design doc and PR. Can you allow comments in
> the
> > > doc so that people can chime in? Thanks!
> > >
> > > Yufei
> > >
> > >
> > > On Sat, Jan 10, 2026 at 9:09 AM Anand Kumar Sankaran via dev <
> > > dev@polaris.apache.org<ma...@polaris.apache.org><mailto:
> > dev@polaris.apache.org<ma...@polaris.apache.org>>> wrote:
> > > Hi all
> > >
> > > My first PR for adding AWS STS Session Tags support for credential
> > vending
> > > was merged.
> >
> https://urldefense.com/v3/__https://github.com/apache/polaris/pull/3327__;!!Iz9xO38YGHZK!65BVFQDe_GtKbDW8LkNdzVS-WYewzWsSns5zFHKclbMU3vUy2UaTXLB5G-wql1yyqoTFyDc9Bt0YHY9sdA$
> > > <
> >
> https://urldefense.com/v3/__https://github.com/apache/polaris/pull/3327__;!!Iz9xO38YGHZK!_dbBfwEnZEuPp_8qIuBDoYfWRgCZKiPDMg9ouDltqsHhr9hqCHVlF9IedSkNVBDWupbneU7inC9YK-THL-lTjWkklJ4Xf9Y$
> > >
> > ><
> > >
> >
> https://urldefense.com/v3/__https://github.com/apache/polaris/pull/3327__;!!Iz9xO38YGHZK!8vlQ2_OqjmZNIpbrTTluFFNKFCEeM87nr95MP5pMpMu5fFWVSoBZ3eb6-C2xbAGCG-ry5eaCO-pFLxl3IZAJGqY$
> > >>
> > >
> > > I have been working with Prashant on an end-to-end telemetry and audit
> > > trail tracking for Polaris.  It is documented here.
> > >
> >
> https://urldefense.com/v3/__https://docs.google.com/document/d/1Ehzvi5RNPs4hChkBFI6VD23myEqm-7sWW3d2kjmuYj8/edit?tab=t.0__;!!Iz9xO38YGHZK!65BVFQDe_GtKbDW8LkNdzVS-WYewzWsSns5zFHKclbMU3vUy2UaTXLB5G-wql1yyqoTFyDc9Bt11Y1rbNQ$
> > > <
> >
> https://urldefense.com/v3/__https://docs.google.com/document/d/1Ehzvi5RNPs4hChkBFI6VD23myEqm-7sWW3d2kjmuYj8/edit?tab=t.0__;!!Iz9xO38YGHZK!_dbBfwEnZEuPp_8qIuBDoYfWRgCZKiPDMg9ouDltqsHhr9hqCHVlF9IedSkNVBDWupbneU7inC9YK-THL-lTjWkk8MVJPlY$
> > >
> > ><
> > >
> >
> https://urldefense.com/v3/__https://docs.google.com/document/d/1Ehzvi5RNPs4hChkBFI6VD23myEqm-7sWW3d2kjmuYj8/edit?tab=t.0__;!!Iz9xO38YGHZK!8vlQ2_OqjmZNIpbrTTluFFNKFCEeM87nr95MP5pMpMu5fFWVSoBZ3eb6-C2xbAGCG-ry5eaCO-pFLxl3hYqTlcg$
> > >>
> > >
> > > Based on Prashant’s initial feedback (parity with Apache Gravitino for
> > > metrics reports), I have an initial PR here.
> > >
> >
> https://urldefense.com/v3/__https://github.com/apache/polaris/pull/3385__;!!Iz9xO38YGHZK!65BVFQDe_GtKbDW8LkNdzVS-WYewzWsSns5zFHKclbMU3vUy2UaTXLB5G-wql1yyqoTFyDc9Bt2W-eKl7A$
> > > <
> >
> https://urldefense.com/v3/__https://github.com/apache/polaris/pull/3385__;!!Iz9xO38YGHZK!_dbBfwEnZEuPp_8qIuBDoYfWRgCZKiPDMg9ouDltqsHhr9hqCHVlF9IedSkNVBDWupbneU7inC9YK-THL-lTjWkkyZhI0OE$
> > >
> > ><
> > >
> >
> https://urldefense.com/v3/__https://github.com/apache/polaris/pull/3385__;!!Iz9xO38YGHZK!8vlQ2_OqjmZNIpbrTTluFFNKFCEeM87nr95MP5pMpMu5fFWVSoBZ3eb6-C2xbAGCG-ry5eaCO-pFLxl39opCznI$
> > >>
> > >
> > > I will be reaching out to various folks for review. I am open to
> breaking
> > > down the PR into smaller logical groups of PR if it helps.
> > >
> > > I request you to look at the Google doc and let me know what you think.
> > >
> > > —
> > > Anand
> > >
> > >
> > >
> > > --
> > > Dmitri Bourlatchkov
> > > Senior Staff Software Engineer, Dremio
> > > Dremio.com
> > > <
> >
> https://urldefense.com/v3/__https://www.dremio.com/?utm_medium=email&utm_source=signature&utm_term=na&utm_content=email-signature&utm_campaign=email-signature__;!!Iz9xO38YGHZK!_dbBfwEnZEuPp_8qIuBDoYfWRgCZKiPDMg9ouDltqsHhr9hqCHVlF9IedSkNVBDWupbneU7inC9YK-THL-lTjWkkiOwjJc0$
> > >/
> > > Follow Us on LinkedIn
> > > <
> >
> https://urldefense.com/v3/__https://www.linkedin.com/company/dremio__;!!Iz9xO38YGHZK!_dbBfwEnZEuPp_8qIuBDoYfWRgCZKiPDMg9ouDltqsHhr9hqCHVlF9IedSkNVBDWupbneU7inC9YK-THL-lTjWkkCLjBQ2M$
> > >/
> > > Get Started
> > > <
> >
> https://urldefense.com/v3/__https://www.dremio.com/get-started/__;!!Iz9xO38YGHZK!_dbBfwEnZEuPp_8qIuBDoYfWRgCZKiPDMg9ouDltqsHhr9hqCHVlF9IedSkNVBDWupbneU7inC9YK-THL-lTjWkkoKQUMII$
> > >
> > >
> > >
> > >The Agentic Lakehouse
> > > *The only lakehouse built for agents, managed by agents*
> > >
> > >
> > >
> >
> >
>

Re: Polaris Telemetry and Audit Trail

[Yufei Gu <fl...@gmail.com>]

Thanks a lot Anand and Dmitri for working on this!

Supporting multiple data sources introduces significant complexity as I
recorded in this issue, https://github.com/apache/polaris/issues/3890. It
requires a well-thought-out design before we commit to it. I'd propose to
leave it now and move forward with the single data source as the initial
step. WDYT?

Yufei


On Tue, Feb 24, 2026 at 12:17 PM Anand Kumar Sankaran via dev <
dev@polaris.apache.org> wrote:

> Hi Dmitri
>
> Thanks for all your support again.
>
> I talked to Yufei and I have addressed his review feedback in this commit.
> https://github.com/apache/polaris/pull/3385/changes/9e3d4d0f409720ab4005db176190ebde29a9d2b6
>
> Yufei and Prashant - can you please review these changes?  Dmitri has
> already reviewed the PR and approved.
>
> -
> Anand
>
> From: Dmitri Bourlatchkov <di...@apache.org>
> Date: Tuesday, February 24, 2026 at 10:38 AM
> To: dev@polaris.apache.org <de...@polaris.apache.org>
> Cc: Anand Kumar Sankaran <an...@workday.com>
> Subject: Re: Polaris Telemetry and Audit Trail
>
> This Message Is From an External Sender
> This message came from outside your organization.
> Report Suspicious<
> https://us-phishalarm-ewt.proofpoint.com/EWT/v1/Iz9xO38YGHZK!YhNDZAGr2cumY1xO1e2JFuKSh-PdIU2YhFBQOvhapv4xXZ0ucXUR7TPwnqpAVpb05t786Kda3YPcEe-qOyYlMkJM8hj02m4Yv48r-c47jSruOAIe3mbeFZ5dBQBrfOUk$
> >
>
> Hi Yufei,
>
> I see that you propose a different approach to schema evolution for
> metrics in GH [1]
>
> Please share your view of this in light of the previous discussion
> regarding combining the metrics persistence with different metastore
> persistence options. It would be good to sync up on this email thread since
> GH comments tend to be hidden on long PRs and not all readers may be able
> to easily find them.
>
> [1] https://github.com/apache/polaris/pull/3385#issuecomment-3953906091<
> https://urldefense.com/v3/__https://github.com/apache/polaris/pull/3385*issuecomment-3953906091__;Iw!!Iz9xO38YGHZK!-KOwW_42EOhdoNvlXd2GvBD6Z7u_HteBuHkQAow0Z11eX9-Q5LAWMr02rP-8OE8VlcwlG4hSnLKT8_s2eg$
> >
>
> Thanks,
> Dmitri.
>
> On Thu, Feb 5, 2026 at 3:47 PM Anand Kumar Sankaran via dev <
> dev@polaris.apache.org<ma...@polaris.apache.org>> wrote:
> Hi Dmitri
>
> The SPI is designed to be pluggable via
> MetaStoreManagerFactory.getOrCreateMetricsPersistence()<
> https://github.com/apache/polaris/pull/3385/changes#diff-0e34667d083d30cace067936b87c736201e5a8cb001aa0b12e203ee52cddae99R51
> <
> https://urldefense.com/v3/__https://github.com/apache/polaris/pull/3385/changes*diff-0e34667d083d30cace067936b87c736201e5a8cb001aa0b12e203ee52cddae99R51__;Iw!!Iz9xO38YGHZK!-KOwW_42EOhdoNvlXd2GvBD6Z7u_HteBuHkQAow0Z11eX9-Q5LAWMr02rP-8OE8VlcwlG4hSnLKXRU26ZA$>>.
> The default returns MetricsPersistence.NOOP, and only JDBC with schema v4+
> returns a functional implementation.
>
>  Your proposal makes sense to me. To allow NoSQL Entity Persistence + JDBC
> Metrics Persistence would require:
>
>      1. Separate metrics schema files (e.g., metrics-schema-v1.sql) that
> can be bootstrapped independently
>      2. Separate datasource configuration for metrics (e.g.,
> polaris.metrics.jdbc.*)
>      3. A dedicated `MetricsPersistenceFactory` interface (separate from
> MetaStoreManagerFactory)
>      4. Updates to the Admin Tool bootstrap workflow to handle both schemas
>
> Does that sound reasonable?  If we do this, all this work should go into
> https://github.com/apache/polaris/pull/3523<
> https://urldefense.com/v3/__https://github.com/apache/polaris/pull/3523__;!!Iz9xO38YGHZK!-KOwW_42EOhdoNvlXd2GvBD6Z7u_HteBuHkQAow0Z11eX9-Q5LAWMr02rP-8OE8VlcwlG4hSnLIg5ApDFA$>
> as one atomic commit, correct?
>
> —
> Anand
>
>
>
> From: Dmitri Bourlatchkov <di...@apache.org>>
> Date: Thursday, February 5, 2026 at 8:28 AM
> To: Anand Kumar Sankaran <anand.sankaran@workday.com<mailto:
> anand.sankaran@workday.com>>
> Cc: dev@polaris.apache.org<ma...@polaris.apache.org> <
> dev@polaris.apache.org<ma...@polaris.apache.org>>
> Subject: Re: Polaris Telemetry and Audit Trail
>
> This Message Is From an External Sender
> This message came from outside your organization.
> Report Suspicious<
> https://us-phishalarm-ewt.proofpoint.com/EWT/v1/Iz9xO38YGHZK!YhNDZAGr2cumbRcPlIOttKwvGr4f38TF-J2-LmOw8H1KsFpWaM3W5g28Oq5qW2gnw2-WHz0rsWhm7F7hSGykA9l01LKYGuCcCkexkHp1sUssM_IegCtjk_id9s891orv$
> >
>
>
> Hi Anand,
>
> Thanks for driving the SPI definition for scan/commits metrics.
>
> I did not think from this angle before, but now I suppose it might be worth
> introducing a separate SQL schema for metrics, detached from the main JDBC
> Persistence schema.
>
> My rationale is that the SPI may be exercised independently of the JDBC
> Entity Persistence. I can imagine people using NoSQL Entity Persistence
> plus JDBC Scan Metrics Persistence in the same deployment.
>
> Using a separate schema will add further work on the Bootstrap workflow
> (Admin Tool), but I hope the end result is going to be more convenient and
> versatile for end users.
>
> WDYT?
>
> Thanks,
> Dmitri.
>
> On Thu, Jan 29, 2026 at 1:49 PM Anand Kumar Sankaran <
> anand.sankaran@workday.com<ma...@workday.com>> wrote:
>
> > Hi Dmitri,
> >
> > Thank you for taking the time to review. I had pulled out the schema in
> > its own PR (based on comments) here:
> >
> https://urldefense.com/v3/__https://github.com/apache/polaris/pull/3523__;!!Iz9xO38YGHZK!65BVFQDe_GtKbDW8LkNdzVS-WYewzWsSns5zFHKclbMU3vUy2UaTXLB5G-wql1yyqoTFyDc9Bt3GRsh6AA$
> >
> > I will update the document with the SPI and carve out another PR that
> > focuses on the SPI.
> >
> > —
> > Anand
> >
> > *From: *Dmitri Bourlatchkov <dmitri.bourlatchkov@dremio.com<mailto:
> dmitri.bourlatchkov@dremio.com>>
> > *Date: *Thursday, January 29, 2026 at 10:43 AM
> > *To: *dev@polaris.apache.org<ma...@polaris.apache.org> <
> dev@polaris.apache.org<ma...@polaris.apache.org>>
> > *Cc: *Anand Kumar Sankaran <anand.sankaran@workday.com<mailto:
> anand.sankaran@workday.com>>
> > *Subject: *Re: Polaris Telemetry and Audit Trail
> >
> > This Message Is From an External Sender
> > This message came from outside your organization.
> > Report Suspicious
> > <
> https://us-phishalarm-ewt.proofpoint.com/EWT/v1/Iz9xO38YGHZK!YhNDZAGr2cumY1cAlUeoPrU2eNUPz9Wxq2PaG1t99vEN3v60gucxC3Qsk-xK09RndcW46WtNjMMjQZbaQ495h1D1g293F_uqLhPM8gDMlA6xRmo5b5uqrSh2v7QKHrhQ$
> >
> >
> > Hi Anand,
> >
> > Thanks for making a proposal doc and starting this thread. Apologies for
> > late comments.
> >
> > As you know I already reviewed a couple of related PRs, which had a
> > smaller impact on the codebase, but now, I believe, we're coming to the
> > core of this feature and it might need a deeper discussion.
> >
> > For the sake of awareness of other project members, I'd like to
> > highlight some aspects by email.
> >
> > * Storing scan metrics in RDBMS in general looks like a
> > reasonable approach to me. However, Polaris Persistence is an extensible
> > mechanism. Other backends are possible, including privately developed
> > backends.
> >
> > From this POV, I believe it might be preferable to start not with an
> RDBMS
> > schema, but with a java SPI outlining expected read/write operations and
> > the data model. This will make it easier to assess impact on other
> > Persistence implementations. Naturally, Polaris service code will have to
> > be able to work seamlessly regardless of the backend impl. (including the
> > do-nothing impl.).
> >
> > I see that some connection between the SQL schema and java code is made
> in
> > PR [3385], but it does not appear to offer a clean SPI that could be
> > implemented by different Persistence backends. Would it be ok from your
> > perspective to make another PR with just SPI code?
> >
> > * Regarding ad-hoc SQL queries - it is certainly a valid use case for
> > custom code to query the Polaris database directly. However, those ad-hoc
> > queries may require certain indexes, which may not have a relevant use
> case
> > in OSS code. In that case, I believe it would be preferable to manage the
> > extra indexes in custom code (not in Polaris). Whether this is the case
> or
> > not currently, I cannot say with certainty, so apologies if this is a
> false
> > alarm... Having the java SPI available would be a huge aid to reasoning
> > about the proposed SQL schema, I think.
> >
> > That said, I do not mean to complicate the development of this feature,
> > just trying to structure it in a way that is hopefully easy to understand
> > (including myself), maintain and expand later. If I missed something,
> > please feel free to point it out.
> >
> > [3385]
> https://urldefense.com/v3/__https://github.com/apache/polaris/pull/3385__;!!Iz9xO38YGHZK!65BVFQDe_GtKbDW8LkNdzVS-WYewzWsSns5zFHKclbMU3vUy2UaTXLB5G-wql1yyqoTFyDc9Bt2W-eKl7A$
> > <
> https://urldefense.com/v3/__https://github.com/apache/polaris/pull/3385__;!!Iz9xO38YGHZK!_dbBfwEnZEuPp_8qIuBDoYfWRgCZKiPDMg9ouDltqsHhr9hqCHVlF9IedSkNVBDWupbneU7inC9YK-THL-lTjWkkyZhI0OE$
> >
> >
> >Thanks,
> > Dmitri.
> >
> > On Wed, Jan 14, 2026 at 10:05 PM Anand Kumar Sankaran via dev <
> > dev@polaris.apache.org<ma...@polaris.apache.org>> wrote:
> >
> > Hi Yufei,
> >
> > Gave commenting privileges to all.
> >
> > Thanks.
> >
> > —
> > Anand
> >
> > From: Yufei Gu <fl...@gmail.com>>
> > Date: Wednesday, January 14, 2026 at 6:13 PM
> > To: dev@polaris.apache.org<ma...@polaris.apache.org> <
> dev@polaris.apache.org<ma...@polaris.apache.org>>
> > Cc: Anand Kumar Sankaran <anand.sankaran@workday.com<mailto:
> anand.sankaran@workday.com>>
> > Subject: Re: Polaris Telemetry and Audit Trail
> >
> > This Message Is From an External Sender
> > This message came from outside your organization.
> > Report Suspicious<
> >
> https://us-phishalarm-ewt.proofpoint.com/EWT/v1/Iz9xO38YGHZK!YhNDZABkHi1B6hyPVYUK0T-K5j4Aoqdrlu07UjBpJcpjOsz7Ie7d_DhPl-ywi50dfQz4O6LL-AHLCOQTQ0X7Le8yD2cF0ogquGFB8Aw8uQLgB0RL6Ezpa-pLDPA4ziXA$
> > >
> >
> > Hi Anand, thanks for the design doc and PR. Can you allow comments in the
> > doc so that people can chime in? Thanks!
> >
> > Yufei
> >
> >
> > On Sat, Jan 10, 2026 at 9:09 AM Anand Kumar Sankaran via dev <
> > dev@polaris.apache.org<ma...@polaris.apache.org><mailto:
> dev@polaris.apache.org<ma...@polaris.apache.org>>> wrote:
> > Hi all
> >
> > My first PR for adding AWS STS Session Tags support for credential
> vending
> > was merged.
> https://urldefense.com/v3/__https://github.com/apache/polaris/pull/3327__;!!Iz9xO38YGHZK!65BVFQDe_GtKbDW8LkNdzVS-WYewzWsSns5zFHKclbMU3vUy2UaTXLB5G-wql1yyqoTFyDc9Bt0YHY9sdA$
> > <
> https://urldefense.com/v3/__https://github.com/apache/polaris/pull/3327__;!!Iz9xO38YGHZK!_dbBfwEnZEuPp_8qIuBDoYfWRgCZKiPDMg9ouDltqsHhr9hqCHVlF9IedSkNVBDWupbneU7inC9YK-THL-lTjWkklJ4Xf9Y$
> >
> ><
> >
> https://urldefense.com/v3/__https://github.com/apache/polaris/pull/3327__;!!Iz9xO38YGHZK!8vlQ2_OqjmZNIpbrTTluFFNKFCEeM87nr95MP5pMpMu5fFWVSoBZ3eb6-C2xbAGCG-ry5eaCO-pFLxl3IZAJGqY$
> >>
> >
> > I have been working with Prashant on an end-to-end telemetry and audit
> > trail tracking for Polaris.  It is documented here.
> >
> https://urldefense.com/v3/__https://docs.google.com/document/d/1Ehzvi5RNPs4hChkBFI6VD23myEqm-7sWW3d2kjmuYj8/edit?tab=t.0__;!!Iz9xO38YGHZK!65BVFQDe_GtKbDW8LkNdzVS-WYewzWsSns5zFHKclbMU3vUy2UaTXLB5G-wql1yyqoTFyDc9Bt11Y1rbNQ$
> > <
> https://urldefense.com/v3/__https://docs.google.com/document/d/1Ehzvi5RNPs4hChkBFI6VD23myEqm-7sWW3d2kjmuYj8/edit?tab=t.0__;!!Iz9xO38YGHZK!_dbBfwEnZEuPp_8qIuBDoYfWRgCZKiPDMg9ouDltqsHhr9hqCHVlF9IedSkNVBDWupbneU7inC9YK-THL-lTjWkk8MVJPlY$
> >
> ><
> >
> https://urldefense.com/v3/__https://docs.google.com/document/d/1Ehzvi5RNPs4hChkBFI6VD23myEqm-7sWW3d2kjmuYj8/edit?tab=t.0__;!!Iz9xO38YGHZK!8vlQ2_OqjmZNIpbrTTluFFNKFCEeM87nr95MP5pMpMu5fFWVSoBZ3eb6-C2xbAGCG-ry5eaCO-pFLxl3hYqTlcg$
> >>
> >
> > Based on Prashant’s initial feedback (parity with Apache Gravitino for
> > metrics reports), I have an initial PR here.
> >
> https://urldefense.com/v3/__https://github.com/apache/polaris/pull/3385__;!!Iz9xO38YGHZK!65BVFQDe_GtKbDW8LkNdzVS-WYewzWsSns5zFHKclbMU3vUy2UaTXLB5G-wql1yyqoTFyDc9Bt2W-eKl7A$
> > <
> https://urldefense.com/v3/__https://github.com/apache/polaris/pull/3385__;!!Iz9xO38YGHZK!_dbBfwEnZEuPp_8qIuBDoYfWRgCZKiPDMg9ouDltqsHhr9hqCHVlF9IedSkNVBDWupbneU7inC9YK-THL-lTjWkkyZhI0OE$
> >
> ><
> >
> https://urldefense.com/v3/__https://github.com/apache/polaris/pull/3385__;!!Iz9xO38YGHZK!8vlQ2_OqjmZNIpbrTTluFFNKFCEeM87nr95MP5pMpMu5fFWVSoBZ3eb6-C2xbAGCG-ry5eaCO-pFLxl39opCznI$
> >>
> >
> > I will be reaching out to various folks for review. I am open to breaking
> > down the PR into smaller logical groups of PR if it helps.
> >
> > I request you to look at the Google doc and let me know what you think.
> >
> > —
> > Anand
> >
> >
> >
> > --
> > Dmitri Bourlatchkov
> > Senior Staff Software Engineer, Dremio
> > Dremio.com
> > <
> https://urldefense.com/v3/__https://www.dremio.com/?utm_medium=email&utm_source=signature&utm_term=na&utm_content=email-signature&utm_campaign=email-signature__;!!Iz9xO38YGHZK!_dbBfwEnZEuPp_8qIuBDoYfWRgCZKiPDMg9ouDltqsHhr9hqCHVlF9IedSkNVBDWupbneU7inC9YK-THL-lTjWkkiOwjJc0$
> >/
> > Follow Us on LinkedIn
> > <
> https://urldefense.com/v3/__https://www.linkedin.com/company/dremio__;!!Iz9xO38YGHZK!_dbBfwEnZEuPp_8qIuBDoYfWRgCZKiPDMg9ouDltqsHhr9hqCHVlF9IedSkNVBDWupbneU7inC9YK-THL-lTjWkkCLjBQ2M$
> >/
> > Get Started
> > <
> https://urldefense.com/v3/__https://www.dremio.com/get-started/__;!!Iz9xO38YGHZK!_dbBfwEnZEuPp_8qIuBDoYfWRgCZKiPDMg9ouDltqsHhr9hqCHVlF9IedSkNVBDWupbneU7inC9YK-THL-lTjWkkoKQUMII$
> >
> >
> >
> >The Agentic Lakehouse
> > *The only lakehouse built for agents, managed by agents*
> >
> >
> >
>
>

Re: Polaris Telemetry and Audit Trail

[Anand Kumar Sankaran via dev <de...@polaris.apache.org>]

Hi Dmitri

Thanks for all your support again.

I talked to Yufei and I have addressed his review feedback in this commit. https://github.com/apache/polaris/pull/3385/changes/9e3d4d0f409720ab4005db176190ebde29a9d2b6

Yufei and Prashant - can you please review these changes?  Dmitri has already reviewed the PR and approved.

-
Anand

From: Dmitri Bourlatchkov <di...@apache.org>
Date: Tuesday, February 24, 2026 at 10:38 AM
To: dev@polaris.apache.org <de...@polaris.apache.org>
Cc: Anand Kumar Sankaran <an...@workday.com>
Subject: Re: Polaris Telemetry and Audit Trail

This Message Is From an External Sender
This message came from outside your organization.
Report Suspicious<https://us-phishalarm-ewt.proofpoint.com/EWT/v1/Iz9xO38YGHZK!YhNDZAGr2cumY1xO1e2JFuKSh-PdIU2YhFBQOvhapv4xXZ0ucXUR7TPwnqpAVpb05t786Kda3YPcEe-qOyYlMkJM8hj02m4Yv48r-c47jSruOAIe3mbeFZ5dBQBrfOUk$>

Hi Yufei,

I see that you propose a different approach to schema evolution for metrics in GH [1]

Please share your view of this in light of the previous discussion regarding combining the metrics persistence with different metastore persistence options. It would be good to sync up on this email thread since GH comments tend to be hidden on long PRs and not all readers may be able to easily find them.

[1] https://github.com/apache/polaris/pull/3385#issuecomment-3953906091<https://urldefense.com/v3/__https://github.com/apache/polaris/pull/3385*issuecomment-3953906091__;Iw!!Iz9xO38YGHZK!-KOwW_42EOhdoNvlXd2GvBD6Z7u_HteBuHkQAow0Z11eX9-Q5LAWMr02rP-8OE8VlcwlG4hSnLKT8_s2eg$>

Thanks,
Dmitri.

On Thu, Feb 5, 2026 at 3:47 PM Anand Kumar Sankaran via dev <de...@polaris.apache.org>> wrote:
Hi Dmitri

The SPI is designed to be pluggable via MetaStoreManagerFactory.getOrCreateMetricsPersistence()<https://github.com/apache/polaris/pull/3385/changes#diff-0e34667d083d30cace067936b87c736201e5a8cb001aa0b12e203ee52cddae99R51<https://urldefense.com/v3/__https://github.com/apache/polaris/pull/3385/changes*diff-0e34667d083d30cace067936b87c736201e5a8cb001aa0b12e203ee52cddae99R51__;Iw!!Iz9xO38YGHZK!-KOwW_42EOhdoNvlXd2GvBD6Z7u_HteBuHkQAow0Z11eX9-Q5LAWMr02rP-8OE8VlcwlG4hSnLKXRU26ZA$>>. The default returns MetricsPersistence.NOOP, and only JDBC with schema v4+ returns a functional implementation.

 Your proposal makes sense to me. To allow NoSQL Entity Persistence + JDBC Metrics Persistence would require:

     1. Separate metrics schema files (e.g., metrics-schema-v1.sql) that can be bootstrapped independently
     2. Separate datasource configuration for metrics (e.g., polaris.metrics.jdbc.*)
     3. A dedicated `MetricsPersistenceFactory` interface (separate from MetaStoreManagerFactory)
     4. Updates to the Admin Tool bootstrap workflow to handle both schemas

Does that sound reasonable?  If we do this, all this work should go into https://github.com/apache/polaris/pull/3523<https://urldefense.com/v3/__https://github.com/apache/polaris/pull/3523__;!!Iz9xO38YGHZK!-KOwW_42EOhdoNvlXd2GvBD6Z7u_HteBuHkQAow0Z11eX9-Q5LAWMr02rP-8OE8VlcwlG4hSnLIg5ApDFA$> as one atomic commit, correct?

—
Anand



From: Dmitri Bourlatchkov <di...@apache.org>>
Date: Thursday, February 5, 2026 at 8:28 AM
To: Anand Kumar Sankaran <an...@workday.com>>
Cc: dev@polaris.apache.org<ma...@polaris.apache.org> <de...@polaris.apache.org>>
Subject: Re: Polaris Telemetry and Audit Trail

This Message Is From an External Sender
This message came from outside your organization.
Report Suspicious<https://us-phishalarm-ewt.proofpoint.com/EWT/v1/Iz9xO38YGHZK!YhNDZAGr2cumbRcPlIOttKwvGr4f38TF-J2-LmOw8H1KsFpWaM3W5g28Oq5qW2gnw2-WHz0rsWhm7F7hSGykA9l01LKYGuCcCkexkHp1sUssM_IegCtjk_id9s891orv$>


Hi Anand,

Thanks for driving the SPI definition for scan/commits metrics.

I did not think from this angle before, but now I suppose it might be worth
introducing a separate SQL schema for metrics, detached from the main JDBC
Persistence schema.

My rationale is that the SPI may be exercised independently of the JDBC
Entity Persistence. I can imagine people using NoSQL Entity Persistence
plus JDBC Scan Metrics Persistence in the same deployment.

Using a separate schema will add further work on the Bootstrap workflow
(Admin Tool), but I hope the end result is going to be more convenient and
versatile for end users.

WDYT?

Thanks,
Dmitri.

On Thu, Jan 29, 2026 at 1:49 PM Anand Kumar Sankaran <
anand.sankaran@workday.com<ma...@workday.com>> wrote:

> Hi Dmitri,
>
> Thank you for taking the time to review. I had pulled out the schema in
> its own PR (based on comments) here:
> https://urldefense.com/v3/__https://github.com/apache/polaris/pull/3523__;!!Iz9xO38YGHZK!65BVFQDe_GtKbDW8LkNdzVS-WYewzWsSns5zFHKclbMU3vUy2UaTXLB5G-wql1yyqoTFyDc9Bt3GRsh6AA$
>
> I will update the document with the SPI and carve out another PR that
> focuses on the SPI.
>
> —
> Anand
>
> *From: *Dmitri Bourlatchkov <dm...@dremio.com>>
> *Date: *Thursday, January 29, 2026 at 10:43 AM
> *To: *dev@polaris.apache.org<ma...@polaris.apache.org> <de...@polaris.apache.org>>
> *Cc: *Anand Kumar Sankaran <an...@workday.com>>
> *Subject: *Re: Polaris Telemetry and Audit Trail
>
> This Message Is From an External Sender
> This message came from outside your organization.
> Report Suspicious
> <https://us-phishalarm-ewt.proofpoint.com/EWT/v1/Iz9xO38YGHZK!YhNDZAGr2cumY1cAlUeoPrU2eNUPz9Wxq2PaG1t99vEN3v60gucxC3Qsk-xK09RndcW46WtNjMMjQZbaQ495h1D1g293F_uqLhPM8gDMlA6xRmo5b5uqrSh2v7QKHrhQ$ >
>
> Hi Anand,
>
> Thanks for making a proposal doc and starting this thread. Apologies for
> late comments.
>
> As you know I already reviewed a couple of related PRs, which had a
> smaller impact on the codebase, but now, I believe, we're coming to the
> core of this feature and it might need a deeper discussion.
>
> For the sake of awareness of other project members, I'd like to
> highlight some aspects by email.
>
> * Storing scan metrics in RDBMS in general looks like a
> reasonable approach to me. However, Polaris Persistence is an extensible
> mechanism. Other backends are possible, including privately developed
> backends.
>
> From this POV, I believe it might be preferable to start not with an RDBMS
> schema, but with a java SPI outlining expected read/write operations and
> the data model. This will make it easier to assess impact on other
> Persistence implementations. Naturally, Polaris service code will have to
> be able to work seamlessly regardless of the backend impl. (including the
> do-nothing impl.).
>
> I see that some connection between the SQL schema and java code is made in
> PR [3385], but it does not appear to offer a clean SPI that could be
> implemented by different Persistence backends. Would it be ok from your
> perspective to make another PR with just SPI code?
>
> * Regarding ad-hoc SQL queries - it is certainly a valid use case for
> custom code to query the Polaris database directly. However, those ad-hoc
> queries may require certain indexes, which may not have a relevant use case
> in OSS code. In that case, I believe it would be preferable to manage the
> extra indexes in custom code (not in Polaris). Whether this is the case or
> not currently, I cannot say with certainty, so apologies if this is a false
> alarm... Having the java SPI available would be a huge aid to reasoning
> about the proposed SQL schema, I think.
>
> That said, I do not mean to complicate the development of this feature,
> just trying to structure it in a way that is hopefully easy to understand
> (including myself), maintain and expand later. If I missed something,
> please feel free to point it out.
>
> [3385] https://urldefense.com/v3/__https://github.com/apache/polaris/pull/3385__;!!Iz9xO38YGHZK!65BVFQDe_GtKbDW8LkNdzVS-WYewzWsSns5zFHKclbMU3vUy2UaTXLB5G-wql1yyqoTFyDc9Bt2W-eKl7A$
> <https://urldefense.com/v3/__https://github.com/apache/polaris/pull/3385__;!!Iz9xO38YGHZK!_dbBfwEnZEuPp_8qIuBDoYfWRgCZKiPDMg9ouDltqsHhr9hqCHVlF9IedSkNVBDWupbneU7inC9YK-THL-lTjWkkyZhI0OE$>
>
>Thanks,
> Dmitri.
>
> On Wed, Jan 14, 2026 at 10:05 PM Anand Kumar Sankaran via dev <
> dev@polaris.apache.org<ma...@polaris.apache.org>> wrote:
>
> Hi Yufei,
>
> Gave commenting privileges to all.
>
> Thanks.
>
> —
> Anand
>
> From: Yufei Gu <fl...@gmail.com>>
> Date: Wednesday, January 14, 2026 at 6:13 PM
> To: dev@polaris.apache.org<ma...@polaris.apache.org> <de...@polaris.apache.org>>
> Cc: Anand Kumar Sankaran <an...@workday.com>>
> Subject: Re: Polaris Telemetry and Audit Trail
>
> This Message Is From an External Sender
> This message came from outside your organization.
> Report Suspicious<
> https://us-phishalarm-ewt.proofpoint.com/EWT/v1/Iz9xO38YGHZK!YhNDZABkHi1B6hyPVYUK0T-K5j4Aoqdrlu07UjBpJcpjOsz7Ie7d_DhPl-ywi50dfQz4O6LL-AHLCOQTQ0X7Le8yD2cF0ogquGFB8Aw8uQLgB0RL6Ezpa-pLDPA4ziXA$
> >
>
> Hi Anand, thanks for the design doc and PR. Can you allow comments in the
> doc so that people can chime in? Thanks!
>
> Yufei
>
>
> On Sat, Jan 10, 2026 at 9:09 AM Anand Kumar Sankaran via dev <
> dev@polaris.apache.org<ma...@polaris.apache.org>>> wrote:
> Hi all
>
> My first PR for adding AWS STS Session Tags support for credential vending
> was merged. https://urldefense.com/v3/__https://github.com/apache/polaris/pull/3327__;!!Iz9xO38YGHZK!65BVFQDe_GtKbDW8LkNdzVS-WYewzWsSns5zFHKclbMU3vUy2UaTXLB5G-wql1yyqoTFyDc9Bt0YHY9sdA$
> <https://urldefense.com/v3/__https://github.com/apache/polaris/pull/3327__;!!Iz9xO38YGHZK!_dbBfwEnZEuPp_8qIuBDoYfWRgCZKiPDMg9ouDltqsHhr9hqCHVlF9IedSkNVBDWupbneU7inC9YK-THL-lTjWkklJ4Xf9Y$>
><
> https://urldefense.com/v3/__https://github.com/apache/polaris/pull/3327__;!!Iz9xO38YGHZK!8vlQ2_OqjmZNIpbrTTluFFNKFCEeM87nr95MP5pMpMu5fFWVSoBZ3eb6-C2xbAGCG-ry5eaCO-pFLxl3IZAJGqY$
>>
>
> I have been working with Prashant on an end-to-end telemetry and audit
> trail tracking for Polaris.  It is documented here.
> https://urldefense.com/v3/__https://docs.google.com/document/d/1Ehzvi5RNPs4hChkBFI6VD23myEqm-7sWW3d2kjmuYj8/edit?tab=t.0__;!!Iz9xO38YGHZK!65BVFQDe_GtKbDW8LkNdzVS-WYewzWsSns5zFHKclbMU3vUy2UaTXLB5G-wql1yyqoTFyDc9Bt11Y1rbNQ$
> <https://urldefense.com/v3/__https://docs.google.com/document/d/1Ehzvi5RNPs4hChkBFI6VD23myEqm-7sWW3d2kjmuYj8/edit?tab=t.0__;!!Iz9xO38YGHZK!_dbBfwEnZEuPp_8qIuBDoYfWRgCZKiPDMg9ouDltqsHhr9hqCHVlF9IedSkNVBDWupbneU7inC9YK-THL-lTjWkk8MVJPlY$>
><
> https://urldefense.com/v3/__https://docs.google.com/document/d/1Ehzvi5RNPs4hChkBFI6VD23myEqm-7sWW3d2kjmuYj8/edit?tab=t.0__;!!Iz9xO38YGHZK!8vlQ2_OqjmZNIpbrTTluFFNKFCEeM87nr95MP5pMpMu5fFWVSoBZ3eb6-C2xbAGCG-ry5eaCO-pFLxl3hYqTlcg$
>>
>
> Based on Prashant’s initial feedback (parity with Apache Gravitino for
> metrics reports), I have an initial PR here.
> https://urldefense.com/v3/__https://github.com/apache/polaris/pull/3385__;!!Iz9xO38YGHZK!65BVFQDe_GtKbDW8LkNdzVS-WYewzWsSns5zFHKclbMU3vUy2UaTXLB5G-wql1yyqoTFyDc9Bt2W-eKl7A$
> <https://urldefense.com/v3/__https://github.com/apache/polaris/pull/3385__;!!Iz9xO38YGHZK!_dbBfwEnZEuPp_8qIuBDoYfWRgCZKiPDMg9ouDltqsHhr9hqCHVlF9IedSkNVBDWupbneU7inC9YK-THL-lTjWkkyZhI0OE$>
><
> https://urldefense.com/v3/__https://github.com/apache/polaris/pull/3385__;!!Iz9xO38YGHZK!8vlQ2_OqjmZNIpbrTTluFFNKFCEeM87nr95MP5pMpMu5fFWVSoBZ3eb6-C2xbAGCG-ry5eaCO-pFLxl39opCznI$
>>
>
> I will be reaching out to various folks for review. I am open to breaking
> down the PR into smaller logical groups of PR if it helps.
>
> I request you to look at the Google doc and let me know what you think.
>
> —
> Anand
>
>
>
> --
> Dmitri Bourlatchkov
> Senior Staff Software Engineer, Dremio
> Dremio.com
> <https://urldefense.com/v3/__https://www.dremio.com/?utm_medium=email&utm_source=signature&utm_term=na&utm_content=email-signature&utm_campaign=email-signature__;!!Iz9xO38YGHZK!_dbBfwEnZEuPp_8qIuBDoYfWRgCZKiPDMg9ouDltqsHhr9hqCHVlF9IedSkNVBDWupbneU7inC9YK-THL-lTjWkkiOwjJc0$>/
> Follow Us on LinkedIn
> <https://urldefense.com/v3/__https://www.linkedin.com/company/dremio__;!!Iz9xO38YGHZK!_dbBfwEnZEuPp_8qIuBDoYfWRgCZKiPDMg9ouDltqsHhr9hqCHVlF9IedSkNVBDWupbneU7inC9YK-THL-lTjWkkCLjBQ2M$>/
> Get Started
> <https://urldefense.com/v3/__https://www.dremio.com/get-started/__;!!Iz9xO38YGHZK!_dbBfwEnZEuPp_8qIuBDoYfWRgCZKiPDMg9ouDltqsHhr9hqCHVlF9IedSkNVBDWupbneU7inC9YK-THL-lTjWkkoKQUMII$>
>
>
>The Agentic Lakehouse
> *The only lakehouse built for agents, managed by agents*
>
>
>

Re: Polaris Telemetry and Audit Trail

[Dmitri Bourlatchkov <di...@apache.org>]

Hi Yufei,

I see that you propose a different approach to schema evolution for metrics
in GH [1]

Please share your view of this in light of the previous discussion
regarding combining the metrics persistence with different metastore
persistence options. It would be good to sync up on this email thread since
GH comments tend to be hidden on long PRs and not all readers may be able
to easily find them.

[1] https://github.com/apache/polaris/pull/3385#issuecomment-3953906091

Thanks,
Dmitri.

On Thu, Feb 5, 2026 at 3:47 PM Anand Kumar Sankaran via dev <
dev@polaris.apache.org> wrote:

> Hi Dmitri
>
> The SPI is designed to be pluggable via
> MetaStoreManagerFactory.getOrCreateMetricsPersistence()<
> https://github.com/apache/polaris/pull/3385/changes#diff-0e34667d083d30cace067936b87c736201e5a8cb001aa0b12e203ee52cddae99R51>.
> The default returns MetricsPersistence.NOOP, and only JDBC with schema v4+
> returns a functional implementation.
>
>  Your proposal makes sense to me. To allow NoSQL Entity Persistence + JDBC
> Metrics Persistence would require:
>
>      1. Separate metrics schema files (e.g., metrics-schema-v1.sql) that
> can be bootstrapped independently
>      2. Separate datasource configuration for metrics (e.g.,
> polaris.metrics.jdbc.*)
>      3. A dedicated `MetricsPersistenceFactory` interface (separate from
> MetaStoreManagerFactory)
>      4. Updates to the Admin Tool bootstrap workflow to handle both schemas
>
> Does that sound reasonable?  If we do this, all this work should go into
> https://github.com/apache/polaris/pull/3523 as one atomic commit, correct?
>
> —
> Anand
>
>
>
> From: Dmitri Bourlatchkov <di...@apache.org>
> Date: Thursday, February 5, 2026 at 8:28 AM
> To: Anand Kumar Sankaran <an...@workday.com>
> Cc: dev@polaris.apache.org <de...@polaris.apache.org>
> Subject: Re: Polaris Telemetry and Audit Trail
>
> This Message Is From an External Sender
> This message came from outside your organization.
> Report Suspicious<
> https://us-phishalarm-ewt.proofpoint.com/EWT/v1/Iz9xO38YGHZK!YhNDZAGr2cumbRcPlIOttKwvGr4f38TF-J2-LmOw8H1KsFpWaM3W5g28Oq5qW2gnw2-WHz0rsWhm7F7hSGykA9l01LKYGuCcCkexkHp1sUssM_IegCtjk_id9s891orv$
> >
>
>
> Hi Anand,
>
> Thanks for driving the SPI definition for scan/commits metrics.
>
> I did not think from this angle before, but now I suppose it might be worth
> introducing a separate SQL schema for metrics, detached from the main JDBC
> Persistence schema.
>
> My rationale is that the SPI may be exercised independently of the JDBC
> Entity Persistence. I can imagine people using NoSQL Entity Persistence
> plus JDBC Scan Metrics Persistence in the same deployment.
>
> Using a separate schema will add further work on the Bootstrap workflow
> (Admin Tool), but I hope the end result is going to be more convenient and
> versatile for end users.
>
> WDYT?
>
> Thanks,
> Dmitri.
>
> On Thu, Jan 29, 2026 at 1:49 PM Anand Kumar Sankaran <
> anand.sankaran@workday.com> wrote:
>
> > Hi Dmitri,
> >
> > Thank you for taking the time to review. I had pulled out the schema in
> > its own PR (based on comments) here:
> >
> https://urldefense.com/v3/__https://github.com/apache/polaris/pull/3523__;!!Iz9xO38YGHZK!65BVFQDe_GtKbDW8LkNdzVS-WYewzWsSns5zFHKclbMU3vUy2UaTXLB5G-wql1yyqoTFyDc9Bt3GRsh6AA$
> >
> > I will update the document with the SPI and carve out another PR that
> > focuses on the SPI.
> >
> > —
> > Anand
> >
> > *From: *Dmitri Bourlatchkov <dm...@dremio.com>
> > *Date: *Thursday, January 29, 2026 at 10:43 AM
> > *To: *dev@polaris.apache.org <de...@polaris.apache.org>
> > *Cc: *Anand Kumar Sankaran <an...@workday.com>
> > *Subject: *Re: Polaris Telemetry and Audit Trail
> >
> > This Message Is From an External Sender
> > This message came from outside your organization.
> > Report Suspicious
> > <
> https://us-phishalarm-ewt.proofpoint.com/EWT/v1/Iz9xO38YGHZK!YhNDZAGr2cumY1cAlUeoPrU2eNUPz9Wxq2PaG1t99vEN3v60gucxC3Qsk-xK09RndcW46WtNjMMjQZbaQ495h1D1g293F_uqLhPM8gDMlA6xRmo5b5uqrSh2v7QKHrhQ$
> >
> >
> > Hi Anand,
> >
> > Thanks for making a proposal doc and starting this thread. Apologies for
> > late comments.
> >
> > As you know I already reviewed a couple of related PRs, which had a
> > smaller impact on the codebase, but now, I believe, we're coming to the
> > core of this feature and it might need a deeper discussion.
> >
> > For the sake of awareness of other project members, I'd like to
> > highlight some aspects by email.
> >
> > * Storing scan metrics in RDBMS in general looks like a
> > reasonable approach to me. However, Polaris Persistence is an extensible
> > mechanism. Other backends are possible, including privately developed
> > backends.
> >
> > From this POV, I believe it might be preferable to start not with an
> RDBMS
> > schema, but with a java SPI outlining expected read/write operations and
> > the data model. This will make it easier to assess impact on other
> > Persistence implementations. Naturally, Polaris service code will have to
> > be able to work seamlessly regardless of the backend impl. (including the
> > do-nothing impl.).
> >
> > I see that some connection between the SQL schema and java code is made
> in
> > PR [3385], but it does not appear to offer a clean SPI that could be
> > implemented by different Persistence backends. Would it be ok from your
> > perspective to make another PR with just SPI code?
> >
> > * Regarding ad-hoc SQL queries - it is certainly a valid use case for
> > custom code to query the Polaris database directly. However, those ad-hoc
> > queries may require certain indexes, which may not have a relevant use
> case
> > in OSS code. In that case, I believe it would be preferable to manage the
> > extra indexes in custom code (not in Polaris). Whether this is the case
> or
> > not currently, I cannot say with certainty, so apologies if this is a
> false
> > alarm... Having the java SPI available would be a huge aid to reasoning
> > about the proposed SQL schema, I think.
> >
> > That said, I do not mean to complicate the development of this feature,
> > just trying to structure it in a way that is hopefully easy to understand
> > (including myself), maintain and expand later. If I missed something,
> > please feel free to point it out.
> >
> > [3385]
> https://urldefense.com/v3/__https://github.com/apache/polaris/pull/3385__;!!Iz9xO38YGHZK!65BVFQDe_GtKbDW8LkNdzVS-WYewzWsSns5zFHKclbMU3vUy2UaTXLB5G-wql1yyqoTFyDc9Bt2W-eKl7A$
> > <
> https://urldefense.com/v3/__https://github.com/apache/polaris/pull/3385__;!!Iz9xO38YGHZK!_dbBfwEnZEuPp_8qIuBDoYfWRgCZKiPDMg9ouDltqsHhr9hqCHVlF9IedSkNVBDWupbneU7inC9YK-THL-lTjWkkyZhI0OE$
> >
> >
> >Thanks,
> > Dmitri.
> >
> > On Wed, Jan 14, 2026 at 10:05 PM Anand Kumar Sankaran via dev <
> > dev@polaris.apache.org> wrote:
> >
> > Hi Yufei,
> >
> > Gave commenting privileges to all.
> >
> > Thanks.
> >
> > —
> > Anand
> >
> > From: Yufei Gu <fl...@gmail.com>
> > Date: Wednesday, January 14, 2026 at 6:13 PM
> > To: dev@polaris.apache.org <de...@polaris.apache.org>
> > Cc: Anand Kumar Sankaran <an...@workday.com>
> > Subject: Re: Polaris Telemetry and Audit Trail
> >
> > This Message Is From an External Sender
> > This message came from outside your organization.
> > Report Suspicious<
> >
> https://us-phishalarm-ewt.proofpoint.com/EWT/v1/Iz9xO38YGHZK!YhNDZABkHi1B6hyPVYUK0T-K5j4Aoqdrlu07UjBpJcpjOsz7Ie7d_DhPl-ywi50dfQz4O6LL-AHLCOQTQ0X7Le8yD2cF0ogquGFB8Aw8uQLgB0RL6Ezpa-pLDPA4ziXA$
> > >
> >
> > Hi Anand, thanks for the design doc and PR. Can you allow comments in the
> > doc so that people can chime in? Thanks!
> >
> > Yufei
> >
> >
> > On Sat, Jan 10, 2026 at 9:09 AM Anand Kumar Sankaran via dev <
> > dev@polaris.apache.org<ma...@polaris.apache.org>> wrote:
> > Hi all
> >
> > My first PR for adding AWS STS Session Tags support for credential
> vending
> > was merged.
> https://urldefense.com/v3/__https://github.com/apache/polaris/pull/3327__;!!Iz9xO38YGHZK!65BVFQDe_GtKbDW8LkNdzVS-WYewzWsSns5zFHKclbMU3vUy2UaTXLB5G-wql1yyqoTFyDc9Bt0YHY9sdA$
> > <
> https://urldefense.com/v3/__https://github.com/apache/polaris/pull/3327__;!!Iz9xO38YGHZK!_dbBfwEnZEuPp_8qIuBDoYfWRgCZKiPDMg9ouDltqsHhr9hqCHVlF9IedSkNVBDWupbneU7inC9YK-THL-lTjWkklJ4Xf9Y$
> >
> ><
> >
> https://urldefense.com/v3/__https://github.com/apache/polaris/pull/3327__;!!Iz9xO38YGHZK!8vlQ2_OqjmZNIpbrTTluFFNKFCEeM87nr95MP5pMpMu5fFWVSoBZ3eb6-C2xbAGCG-ry5eaCO-pFLxl3IZAJGqY$
> >>
> >
> > I have been working with Prashant on an end-to-end telemetry and audit
> > trail tracking for Polaris.  It is documented here.
> >
> https://urldefense.com/v3/__https://docs.google.com/document/d/1Ehzvi5RNPs4hChkBFI6VD23myEqm-7sWW3d2kjmuYj8/edit?tab=t.0__;!!Iz9xO38YGHZK!65BVFQDe_GtKbDW8LkNdzVS-WYewzWsSns5zFHKclbMU3vUy2UaTXLB5G-wql1yyqoTFyDc9Bt11Y1rbNQ$
> > <
> https://urldefense.com/v3/__https://docs.google.com/document/d/1Ehzvi5RNPs4hChkBFI6VD23myEqm-7sWW3d2kjmuYj8/edit?tab=t.0__;!!Iz9xO38YGHZK!_dbBfwEnZEuPp_8qIuBDoYfWRgCZKiPDMg9ouDltqsHhr9hqCHVlF9IedSkNVBDWupbneU7inC9YK-THL-lTjWkk8MVJPlY$
> >
> ><
> >
> https://urldefense.com/v3/__https://docs.google.com/document/d/1Ehzvi5RNPs4hChkBFI6VD23myEqm-7sWW3d2kjmuYj8/edit?tab=t.0__;!!Iz9xO38YGHZK!8vlQ2_OqjmZNIpbrTTluFFNKFCEeM87nr95MP5pMpMu5fFWVSoBZ3eb6-C2xbAGCG-ry5eaCO-pFLxl3hYqTlcg$
> >>
> >
> > Based on Prashant’s initial feedback (parity with Apache Gravitino for
> > metrics reports), I have an initial PR here.
> >
> https://urldefense.com/v3/__https://github.com/apache/polaris/pull/3385__;!!Iz9xO38YGHZK!65BVFQDe_GtKbDW8LkNdzVS-WYewzWsSns5zFHKclbMU3vUy2UaTXLB5G-wql1yyqoTFyDc9Bt2W-eKl7A$
> > <
> https://urldefense.com/v3/__https://github.com/apache/polaris/pull/3385__;!!Iz9xO38YGHZK!_dbBfwEnZEuPp_8qIuBDoYfWRgCZKiPDMg9ouDltqsHhr9hqCHVlF9IedSkNVBDWupbneU7inC9YK-THL-lTjWkkyZhI0OE$
> >
> ><
> >
> https://urldefense.com/v3/__https://github.com/apache/polaris/pull/3385__;!!Iz9xO38YGHZK!8vlQ2_OqjmZNIpbrTTluFFNKFCEeM87nr95MP5pMpMu5fFWVSoBZ3eb6-C2xbAGCG-ry5eaCO-pFLxl39opCznI$
> >>
> >
> > I will be reaching out to various folks for review. I am open to breaking
> > down the PR into smaller logical groups of PR if it helps.
> >
> > I request you to look at the Google doc and let me know what you think.
> >
> > —
> > Anand
> >
> >
> >
> > --
> > Dmitri Bourlatchkov
> > Senior Staff Software Engineer, Dremio
> > Dremio.com
> > <
> https://urldefense.com/v3/__https://www.dremio.com/?utm_medium=email&utm_source=signature&utm_term=na&utm_content=email-signature&utm_campaign=email-signature__;!!Iz9xO38YGHZK!_dbBfwEnZEuPp_8qIuBDoYfWRgCZKiPDMg9ouDltqsHhr9hqCHVlF9IedSkNVBDWupbneU7inC9YK-THL-lTjWkkiOwjJc0$
> >/
> > Follow Us on LinkedIn
> > <
> https://urldefense.com/v3/__https://www.linkedin.com/company/dremio__;!!Iz9xO38YGHZK!_dbBfwEnZEuPp_8qIuBDoYfWRgCZKiPDMg9ouDltqsHhr9hqCHVlF9IedSkNVBDWupbneU7inC9YK-THL-lTjWkkCLjBQ2M$
> >/
> > Get Started
> > <
> https://urldefense.com/v3/__https://www.dremio.com/get-started/__;!!Iz9xO38YGHZK!_dbBfwEnZEuPp_8qIuBDoYfWRgCZKiPDMg9ouDltqsHhr9hqCHVlF9IedSkNVBDWupbneU7inC9YK-THL-lTjWkkoKQUMII$
> >
> >
> >
> >The Agentic Lakehouse
> > *The only lakehouse built for agents, managed by agents*
> >
> >
> >
>
>

Re: Polaris Telemetry and Audit Trail

[Anand Kumar Sankaran via dev <de...@polaris.apache.org>]

Hi Prashant,

I have pushed a new comment to this PR: https://github.com/apache/polaris/pull/3523.

It prints a clear message that the table was not bootstrapped.

—
Anand

From: Dmitri Bourlatchkov <di...@apache.org>
Date: Thursday, February 12, 2026 at 9:25 AM
To: dev@polaris.apache.org <de...@polaris.apache.org>
Cc: Anand Kumar Sankaran <an...@workday.com>, Prashant Singh <pr...@snowflake.com>
Subject: Re: Polaris Telemetry and Audit Trail

This Message Is From an External Sender
This message came from outside your organization.
Report Suspicious<https://us-phishalarm-ewt.proofpoint.com/EWT/v1/Iz9xO38YGHZK!YhNDZAGr2cumYd4qX8MllZhZH1pIYw7PQoAy7f4z8UWkSJ6KS-qQyOAofn8Kn_IP1v3CjQhOcDGdRFn59tf5346j_NNnjG_5ym1oxPiBcv1isQ0zGX6O8onGwagEbKiW$>

Hi Prashant,

Re: separate (events / metrics) schema - in general I'd prefer to be able to evolve events tables separately from the "metastore" schema too.

My rationale is that metastore code does not depend on events (or metrics) persistence, therefore, their schemas could be evolved independently, which aids developing events/metrics enhancements without affecting the main metastore call paths.

Once does not have to bump the metastore schema in case events / metrics DDL changes, but the user elected not to persist them.

The metrics feature paves the way in that direction, but if the community agrees on this approach in general, I suppose we could refactor events to that effect later.

Re: conditional bootstrap - it would certainly make sense to make a ProductionReadinessCheck for verifying (on startup) that if metrics persistence is enabled, we actually have tables in place.

Cheers,
Dmitri.

On Thu, Feb 12, 2026 at 12:14 PM Prashant Singh via dev <de...@polaris.apache.org>> wrote:
Thank you for the Proposal Anand !
The only feedback I had was if we are doing a conditional bootstrap of
metrics table we need to make sure when someone enables the feature
get a clear message that hey table was not bootstrapped we should bootstrap
it first before enabling it ?
In events I am not sure we discussed this but the events table gets created
along with other required tables even when the config is off so we should
make sure we do that.

Regarding having a connection pool, i agree we should have named datasource
for this to not hog the connection pool which is used by polaris for
entities and ops

Let me know how it sounds !

Best,
Prashant Singh

On Mon, Feb 9, 2026 at 11:08 AM Anand Kumar Sankaran via dev <
dev@polaris.apache.org<ma...@polaris.apache.org>> wrote:

> Dmitri and others,
>
> Thanks for all the review feedback and being patient while we iterated. My
> initial proposal was to persist the table metrics as events (least amount
> of work 😊), we have come far from that proposal.
>
> To detach the SPI fully, there are a few things to do (loadSchemaVersion()
> and shared DatasourceOperations). We can add a separate datasource config
> to specify a different connection pool, perhaps. Can be done as a follow up
> PR.
>
> —
> Anand
>
> From: Dmitri Bourlatchkov <di...@apache.org>>
> Date: Monday, February 9, 2026 at 10:52 AM
> To: dev@polaris.apache.org<ma...@polaris.apache.org> <de...@polaris.apache.org>>
> Cc: Anand Kumar Sankaran <an...@workday.com>>
> Subject: Re: Polaris Telemetry and Audit Trail
>
> This Message Is From an External Sender
> This message came from outside your organization.
> Report Suspicious<
> https://us-phishalarm-ewt.proofpoint.com/EWT/v1/Iz9xO38YGHZK!YhNDZAGr2cumYduJ20ONMhMxljYPF8uZOe6R8BpEWyJ7tSG5GUUf8our05nYHWov0t1Vi7Rx_O8JuRNBBQQ57aBoctrR0wH7VSR9VxRuqYJPpz0q6I0JboPRfdLIwqal$
> >
>
> Hi Anand and all,
>
> Thanks for pushing this feature forward! From my POV the SPI [3616] and
> the schema [3523] PRs are good to merge as the initial implementation of
> this feature (annotated as "beta"). The way is open to the JDBC impl. PR
> [3385] (which will probably need a rebase after merging the SPI and schema
> code).
>
> I'm sending this email to highlight the direction taken with respect to
> the SQL schema in this feature, as it might have been visible enough to the
> broader community during the PR reviews.
>
> * The scan / commit metrics schema is managed / versioned separately from
> the JDBC MetaStore schema
>
> * The Metrics Persistence SPI is detached (mostly at this time, but
> supposedly fully detached later) from the JDBC MetaStore
>
> * The JDBC DataSource for metrics may be the same or different from the
> MetaStore datasource.
>
> * Metrics Persistence may have non-JDBC implementation in the future (not
> implemented ATM)
>
> * JDBC Metrics Persistence may co-exist with other MetaStore
> implementations.
>
> I believe this design allows great flexibility for end users to try this
> feature out and keeps options open for future enhancements in Scan / Commit
> Metrics in Polaris.
>
> [3616] https://github.com/apache/polaris/pull/3616<https://urldefense.com/v3/__https://github.com/apache/polaris/pull/3616__;!!Iz9xO38YGHZK!-GYRcRvIpJt2c_e6jWw_Bo86tm0XicvUFgJyQY6MDL0UJ9l8bsXRHk43x8zb8HOtZesyXzIflC9r2cpinQ$><
> https://urldefense.com/v3/__https://github.com/apache/polaris/pull/3616__;!!Iz9xO38YGHZK!7IwwfPXDN0ThhNZgol-n9Ww3FFLKcpZbSQ17lWEtcQnTHtJM9aVXnqGW99_6vmVkWACy7rYC8XEpN6Okdg$
> >
>
> [3523] https://github.com/apache/polaris/pull/3523<https://urldefense.com/v3/__https://github.com/apache/polaris/pull/3523__;!!Iz9xO38YGHZK!-GYRcRvIpJt2c_e6jWw_Bo86tm0XicvUFgJyQY6MDL0UJ9l8bsXRHk43x8zb8HOtZesyXzIflC90OoBMnA$><
> https://urldefense.com/v3/__https://github.com/apache/polaris/pull/3523__;!!Iz9xO38YGHZK!7IwwfPXDN0ThhNZgol-n9Ww3FFLKcpZbSQ17lWEtcQnTHtJM9aVXnqGW99_6vmVkWACy7rYC8XGOCHKhSg$
> >
>
> [3385] https://github.com/apache/polaris/pull/3385<https://urldefense.com/v3/__https://github.com/apache/polaris/pull/3385__;!!Iz9xO38YGHZK!-GYRcRvIpJt2c_e6jWw_Bo86tm0XicvUFgJyQY6MDL0UJ9l8bsXRHk43x8zb8HOtZesyXzIflC8KNDHjzA$><
> https://urldefense.com/v3/__https://github.com/apache/polaris/pull/3385__;!!Iz9xO38YGHZK!7IwwfPXDN0ThhNZgol-n9Ww3FFLKcpZbSQ17lWEtcQnTHtJM9aVXnqGW99_6vmVkWACy7rYC8XFlxLMhvQ$
> >
>
> Thoughts / comments?
>
> Cheers,
> Dmitri.
>
>
> On Thu, Feb 5, 2026 at 6:50 PM Anand Kumar Sankaran via dev <
> dev@polaris.apache.org<ma...@polaris.apache.org>>> wrote:
> Hi Dmitri,
>
> Following up on your feedback regarding the metrics schema separation
> approach. I've implemented the simplified design you suggested in
> https://github.com/apache/polaris/pull/3523<https://urldefense.com/v3/__https://github.com/apache/polaris/pull/3523__;!!Iz9xO38YGHZK!-GYRcRvIpJt2c_e6jWw_Bo86tm0XicvUFgJyQY6MDL0UJ9l8bsXRHk43x8zb8HOtZesyXzIflC90OoBMnA$><
> https://urldefense.com/v3/__https://github.com/apache/polaris/pull/3523__;!!Iz9xO38YGHZK!7IwwfPXDN0ThhNZgol-n9Ww3FFLKcpZbSQ17lWEtcQnTHtJM9aVXnqGW99_6vmVkWACy7rYC8XGOCHKhSg$
> >:
>
>
>  The metrics schema can now evolve independently from the entity schema.
> Let me know if you'd like any changes to this approach.
>
>  Thanks,
>   Anand
>
>
>
> From: Dmitri Bourlatchkov <di...@apache.org>>>
> Date: Thursday, February 5, 2026 at 2:57 PM
> To: dev@polaris.apache.org<ma...@polaris.apache.org>> <
> dev@polaris.apache.org<ma...@polaris.apache.org>>>
> Cc: Anand Kumar Sankaran <an...@workday.com><mailto:
> anand.sankaran@workday.com<ma...@workday.com>>>
> Subject: Re: Polaris Telemetry and Audit Trail
>
> This Message Is From an External Sender
> This message came from outside your organization.
> Report Suspicious<
> https://us-phishalarm-ewt.proofpoint.com/EWT/v1/Iz9xO38YGHZK!YhNDZAGr2cumY1iAus5GNbkIxSXoQaI552sXgYyv7Z9Wrvb7qeTskSy5WkXTVDJ_Bel2AbNh_fQVn-HS_UtOSdOtpsNfzhXDkK4fYWdTeoeBe70A6oQVGpAYT3SrbEvs$
> >
>
> Hi Anand,
>
> Re: MetricsPersistence Factory - TBH I do not think it is required. If we
> go with an MetricsPersistence impl. independent of the JDBC MetaStore, it
> could be injected into PersistingMetricsReporter directly by CDI.
>
> Re: DataSource - for a start we could probably reuse the same DataSource
> as the JDBS MetaStore uses. In principle, we could introduce a separate
> DataSource, but I do not think we need to complicate the initial
> implementation with that. It can be added later, if required.
>
> Re: schema / bootstrap - I'm thinking, indeed, of having a separate *.sql
> file for metrics DDL. Then we could add an option to the bootstrap command
> in the Admin Tool, or even a new command for creating / updating the
> metrics schema. This might require some refactoring, but I believe the
> bootstrap code needs some cleanup anyway.
>
> The general idea is to allow the metrics schema to evolve independently.
> It's a new and actively developed feature, so having flexibility is going
> to be good for making progress / improvements, I hope.
>
> Does that sound reasonable?
>
> Thanks,
> Dmitri.
>
> On Thu, Feb 5, 2026 at 3:47 PM Anand Kumar Sankaran via dev <
> dev@polaris.apache.org<ma...@polaris.apache.org>><mailto:
> dev@polaris.apache.org<ma...@polaris.apache.org>>>> wrote:
> Hi Dmitri
>
> The SPI is designed to be pluggable via
> MetaStoreManagerFactory.getOrCreateMetricsPersistence()<
> https://github.com/apache/polaris/pull/3385/changes#diff-0e34667d083d30cace067936b87c736201e5a8cb001aa0b12e203ee52cddae99R51<https://urldefense.com/v3/__https://github.com/apache/polaris/pull/3385/changes*diff-0e34667d083d30cace067936b87c736201e5a8cb001aa0b12e203ee52cddae99R51__;Iw!!Iz9xO38YGHZK!-GYRcRvIpJt2c_e6jWw_Bo86tm0XicvUFgJyQY6MDL0UJ9l8bsXRHk43x8zb8HOtZesyXzIflC9_kjOfGQ$>
> <
> https://urldefense.com/v3/__https://github.com/apache/polaris/pull/3385/changes*diff-0e34667d083d30cace067936b87c736201e5a8cb001aa0b12e203ee52cddae99R51__;Iw!!Iz9xO38YGHZK!7IwwfPXDN0ThhNZgol-n9Ww3FFLKcpZbSQ17lWEtcQnTHtJM9aVXnqGW99_6vmVkWACy7rYC8XFxRr4qRw$
> ><
> https://urldefense.com/v3/__https://github.com/apache/polaris/pull/3385/changes*diff-0e34667d083d30cace067936b87c736201e5a8cb001aa0b12e203ee52cddae99R51__;Iw!!Iz9xO38YGHZK!_K1mcH0_2x0nbU-mZlbBi3tk9QAGjB6VInPiqkQU21mpIP_ft2bXXyIiADNoAcotNr9AKqlphh65taUhcw$>>.
> The default returns MetricsPersistence.NOOP, and only JDBC with schema v4+
> returns a functional implementation.
>
>  Your proposal makes sense to me. To allow NoSQL Entity Persistence + JDBC
> Metrics Persistence would require:
>
>      1. Separate metrics schema files (e.g., metrics-schema-v1.sql) that
> can be bootstrapped independently
>      2. Separate datasource configuration for metrics (e.g.,
> polaris.metrics.jdbc.*)
>      3. A dedicated `MetricsPersistenceFactory` interface (separate from
> MetaStoreManagerFactory)
>      4. Updates to the Admin Tool bootstrap workflow to handle both schemas
>
> Does that sound reasonable?  If we do this, all this work should go into
> https://github.com/apache/polaris/pull/3523<https://urldefense.com/v3/__https://github.com/apache/polaris/pull/3523__;!!Iz9xO38YGHZK!-GYRcRvIpJt2c_e6jWw_Bo86tm0XicvUFgJyQY6MDL0UJ9l8bsXRHk43x8zb8HOtZesyXzIflC90OoBMnA$><
> https://urldefense.com/v3/__https://github.com/apache/polaris/pull/3523__;!!Iz9xO38YGHZK!7IwwfPXDN0ThhNZgol-n9Ww3FFLKcpZbSQ17lWEtcQnTHtJM9aVXnqGW99_6vmVkWACy7rYC8XGOCHKhSg$
> ><
> https://urldefense.com/v3/__https://github.com/apache/polaris/pull/3523__;!!Iz9xO38YGHZK!_K1mcH0_2x0nbU-mZlbBi3tk9QAGjB6VInPiqkQU21mpIP_ft2bXXyIiADNoAcotNr9AKqlphh4ZISZcDQ$>
> as one atomic commit, correct?
>
> —
> Anand
>
>
>
> From: Dmitri Bourlatchkov <di...@apache.org>
> ><ma...@apache.org>>>>
> Date: Thursday, February 5, 2026 at 8:28 AM
> To: Anand Kumar Sankaran <an...@workday.com><mailto:
> anand.sankaran@workday.com<ma...@workday.com><mailto:
> anand.sankaran@workday.com<ma...@workday.com>>>>
> Cc: dev@polaris.apache.org<ma...@polaris.apache.org>><mailto:
> dev@polaris.apache.org<ma...@polaris.apache.org>>> <
> dev@polaris.apache.org<ma...@polaris.apache.org>><mailto:
> dev@polaris.apache.org<ma...@polaris.apache.org>>>>
> Subject: Re: Polaris Telemetry and Audit Trail
>
> This Message Is From an External Sender
> This message came from outside your organization.
> Report Suspicious<
> https://us-phishalarm-ewt.proofpoint.com/EWT/v1/Iz9xO38YGHZK!YhNDZAGr2cumbRcPlIOttKwvGr4f38TF-J2-LmOw8H1KsFpWaM3W5g28Oq5qW2gnw2-WHz0rsWhm7F7hSGykA9l01LKYGuCcCkexkHp1sUssM_IegCtjk_id9s891orv$
> >
>
>
> Hi Anand,
>
> Thanks for driving the SPI definition for scan/commits metrics.
>
> I did not think from this angle before, but now I suppose it might be worth
> introducing a separate SQL schema for metrics, detached from the main JDBC
> Persistence schema.
>
> My rationale is that the SPI may be exercised independently of the JDBC
> Entity Persistence. I can imagine people using NoSQL Entity Persistence
> plus JDBC Scan Metrics Persistence in the same deployment.
>
> Using a separate schema will add further work on the Bootstrap workflow
> (Admin Tool), but I hope the end result is going to be more convenient and
> versatile for end users.
>
> WDYT?
>
> Thanks,
> Dmitri.
>
> On Thu, Jan 29, 2026 at 1:49 PM Anand Kumar Sankaran <
> anand.sankaran@workday.com<ma...@workday.com>><mailto:
> anand.sankaran@workday.com<ma...@workday.com>>>> wrote:
>
> > Hi Dmitri,
> >
> > Thank you for taking the time to review. I had pulled out the schema in
> > its own PR (based on comments) here:
> >
> https://urldefense.com/v3/__https://github.com/apache/polaris/pull/3523__;!!Iz9xO38YGHZK!65BVFQDe_GtKbDW8LkNdzVS-WYewzWsSns5zFHKclbMU3vUy2UaTXLB5G-wql1yyqoTFyDc9Bt3GRsh6AA$
> >
> > I will update the document with the SPI and carve out another PR that
> > focuses on the SPI.
> >
> > —
> > Anand
> >
> > *From: *Dmitri Bourlatchkov <dm...@dremio.com><mailto:
> dmitri.bourlatchkov@dremio.com<ma...@dremio.com>
> <ma...@dremio.com>>>>
> > *Date: *Thursday, January 29, 2026 at 10:43 AM
> > *To: *dev@polaris.apache.org<ma...@polaris.apache.org>><mailto:
> dev@polaris.apache.org<ma...@polaris.apache.org>>> <
> dev@polaris.apache.org<ma...@polaris.apache.org>><mailto:
> dev@polaris.apache.org<ma...@polaris.apache.org>>>>
> > *Cc: *Anand Kumar Sankaran <an...@workday.com><mailto:
> anand.sankaran@workday.com<ma...@workday.com><mailto:
> anand.sankaran@workday.com<ma...@workday.com>>>>
> > *Subject: *Re: Polaris Telemetry and Audit Trail
> >
> > This Message Is From an External Sender
> > This message came from outside your organization.
> > Report Suspicious
> > <
> https://us-phishalarm-ewt.proofpoint.com/EWT/v1/Iz9xO38YGHZK!YhNDZAGr2cumY1cAlUeoPrU2eNUPz9Wxq2PaG1t99vEN3v60gucxC3Qsk-xK09RndcW46WtNjMMjQZbaQ495h1D1g293F_uqLhPM8gDMlA6xRmo5b5uqrSh2v7QKHrhQ$
> >
> >
> > Hi Anand,
> >
> > Thanks for making a proposal doc and starting this thread. Apologies for
> > late comments.
> >
> > As you know I already reviewed a couple of related PRs, which had a
> > smaller impact on the codebase, but now, I believe, we're coming to the
> > core of this feature and it might need a deeper discussion.
> >
> > For the sake of awareness of other project members, I'd like to
> > highlight some aspects by email.
> >
> > * Storing scan metrics in RDBMS in general looks like a
> > reasonable approach to me. However, Polaris Persistence is an extensible
> > mechanism. Other backends are possible, including privately developed
> > backends.
> >
> > From this POV, I believe it might be preferable to start not with an
> RDBMS
> > schema, but with a java SPI outlining expected read/write operations and
> > the data model. This will make it easier to assess impact on other
> > Persistence implementations. Naturally, Polaris service code will have to
> > be able to work seamlessly regardless of the backend impl. (including the
> > do-nothing impl.).
> >
> > I see that some connection between the SQL schema and java code is made
> in
> > PR [3385], but it does not appear to offer a clean SPI that could be
> > implemented by different Persistence backends. Would it be ok from your
> > perspective to make another PR with just SPI code?
> >
> > * Regarding ad-hoc SQL queries - it is certainly a valid use case for
> > custom code to query the Polaris database directly. However, those ad-hoc
> > queries may require certain indexes, which may not have a relevant use
> case
> > in OSS code. In that case, I believe it would be preferable to manage the
> > extra indexes in custom code (not in Polaris). Whether this is the case
> or
> > not currently, I cannot say with certainty, so apologies if this is a
> false
> > alarm... Having the java SPI available would be a huge aid to reasoning
> > about the proposed SQL schema, I think.
> >
> > That said, I do not mean to complicate the development of this feature,
> > just trying to structure it in a way that is hopefully easy to understand
> > (including myself), maintain and expand later. If I missed something,
> > please feel free to point it out.
> >
> > [3385]
> https://urldefense.com/v3/__https://github.com/apache/polaris/pull/3385__;!!Iz9xO38YGHZK!65BVFQDe_GtKbDW8LkNdzVS-WYewzWsSns5zFHKclbMU3vUy2UaTXLB5G-wql1yyqoTFyDc9Bt2W-eKl7A$
> > <
> https://urldefense.com/v3/__https://github.com/apache/polaris/pull/3385__;!!Iz9xO38YGHZK!_dbBfwEnZEuPp_8qIuBDoYfWRgCZKiPDMg9ouDltqsHhr9hqCHVlF9IedSkNVBDWupbneU7inC9YK-THL-lTjWkkyZhI0OE$
> >
> >
> >Thanks,
> > Dmitri.
> >
> > On Wed, Jan 14, 2026 at 10:05 PM Anand Kumar Sankaran via dev <
> > dev@polaris.apache.org<ma...@polaris.apache.org>><mailto:
> dev@polaris.apache.org<ma...@polaris.apache.org>>>> wrote:
> >
> > Hi Yufei,
> >
> > Gave commenting privileges to all.
> >
> > Thanks.
> >
> > —
> > Anand
> >
> > From: Yufei Gu <fl...@gmail.com>
> ><ma...@gmail.com>>>>
> > Date: Wednesday, January 14, 2026 at 6:13 PM
> > To: dev@polaris.apache.org<ma...@polaris.apache.org>><mailto:
> dev@polaris.apache.org<ma...@polaris.apache.org>>> <
> dev@polaris.apache.org<ma...@polaris.apache.org>><mailto:
> dev@polaris.apache.org<ma...@polaris.apache.org>>>>
> > Cc: Anand Kumar Sankaran <an...@workday.com><mailto:
> anand.sankaran@workday.com<ma...@workday.com><mailto:
> anand.sankaran@workday.com<ma...@workday.com>>>>
> > Subject: Re: Polaris Telemetry and Audit Trail
> >
> > This Message Is From an External Sender
> > This message came from outside your organization.
> > Report Suspicious<
> >
> https://us-phishalarm-ewt.proofpoint.com/EWT/v1/Iz9xO38YGHZK!YhNDZABkHi1B6hyPVYUK0T-K5j4Aoqdrlu07UjBpJcpjOsz7Ie7d_DhPl-ywi50dfQz4O6LL-AHLCOQTQ0X7Le8yD2cF0ogquGFB8Aw8uQLgB0RL6Ezpa-pLDPA4ziXA$
> > >
> >
> > Hi Anand, thanks for the design doc and PR. Can you allow comments in the
> > doc so that people can chime in? Thanks!
> >
> > Yufei
> >
> >
> > On Sat, Jan 10, 2026 at 9:09 AM Anand Kumar Sankaran via dev <
> > dev@polaris.apache.org<ma...@polaris.apache.org>><mailto:
> dev@polaris.apache.org<ma...@polaris.apache.org>>><mailto:
> dev@polaris.apache.org<ma...@polaris.apache.org>><mailto:
> dev@polaris.apache.org<ma...@polaris.apache.org>>>>> wrote:
> > Hi all
> >
> > My first PR for adding AWS STS Session Tags support for credential
> vending
> > was merged.
> https://urldefense.com/v3/__https://github.com/apache/polaris/pull/3327__;!!Iz9xO38YGHZK!65BVFQDe_GtKbDW8LkNdzVS-WYewzWsSns5zFHKclbMU3vUy2UaTXLB5G-wql1yyqoTFyDc9Bt0YHY9sdA$
> > <
> https://urldefense.com/v3/__https://github.com/apache/polaris/pull/3327__;!!Iz9xO38YGHZK!_dbBfwEnZEuPp_8qIuBDoYfWRgCZKiPDMg9ouDltqsHhr9hqCHVlF9IedSkNVBDWupbneU7inC9YK-THL-lTjWkklJ4Xf9Y$
> >
> ><
> >
> https://urldefense.com/v3/__https://github.com/apache/polaris/pull/3327__;!!Iz9xO38YGHZK!8vlQ2_OqjmZNIpbrTTluFFNKFCEeM87nr95MP5pMpMu5fFWVSoBZ3eb6-C2xbAGCG-ry5eaCO-pFLxl3IZAJGqY$
> >>
> >
> > I have been working with Prashant on an end-to-end telemetry and audit
> > trail tracking for Polaris.  It is documented here.
> >
> https://urldefense.com/v3/__https://docs.google.com/document/d/1Ehzvi5RNPs4hChkBFI6VD23myEqm-7sWW3d2kjmuYj8/edit?tab=t.0__;!!Iz9xO38YGHZK!65BVFQDe_GtKbDW8LkNdzVS-WYewzWsSns5zFHKclbMU3vUy2UaTXLB5G-wql1yyqoTFyDc9Bt11Y1rbNQ$
> > <
> https://urldefense.com/v3/__https://docs.google.com/document/d/1Ehzvi5RNPs4hChkBFI6VD23myEqm-7sWW3d2kjmuYj8/edit?tab=t.0__;!!Iz9xO38YGHZK!_dbBfwEnZEuPp_8qIuBDoYfWRgCZKiPDMg9ouDltqsHhr9hqCHVlF9IedSkNVBDWupbneU7inC9YK-THL-lTjWkk8MVJPlY$
> >
> ><
> >
> https://urldefense.com/v3/__https://docs.google.com/document/d/1Ehzvi5RNPs4hChkBFI6VD23myEqm-7sWW3d2kjmuYj8/edit?tab=t.0__;!!Iz9xO38YGHZK!8vlQ2_OqjmZNIpbrTTluFFNKFCEeM87nr95MP5pMpMu5fFWVSoBZ3eb6-C2xbAGCG-ry5eaCO-pFLxl3hYqTlcg$
> >>
> >
> > Based on Prashant’s initial feedback (parity with Apache Gravitino for
> > metrics reports), I have an initial PR here.
> >
> https://urldefense.com/v3/__https://github.com/apache/polaris/pull/3385__;!!Iz9xO38YGHZK!65BVFQDe_GtKbDW8LkNdzVS-WYewzWsSns5zFHKclbMU3vUy2UaTXLB5G-wql1yyqoTFyDc9Bt2W-eKl7A$
> > <
> https://urldefense.com/v3/__https://github.com/apache/polaris/pull/3385__;!!Iz9xO38YGHZK!_dbBfwEnZEuPp_8qIuBDoYfWRgCZKiPDMg9ouDltqsHhr9hqCHVlF9IedSkNVBDWupbneU7inC9YK-THL-lTjWkkyZhI0OE$
> >
> ><
> >
> https://urldefense.com/v3/__https://github.com/apache/polaris/pull/3385__;!!Iz9xO38YGHZK!8vlQ2_OqjmZNIpbrTTluFFNKFCEeM87nr95MP5pMpMu5fFWVSoBZ3eb6-C2xbAGCG-ry5eaCO-pFLxl39opCznI$
> >>
> >
> > I will be reaching out to various folks for review. I am open to breaking
> > down the PR into smaller logical groups of PR if it helps.
> >
> > I request you to look at the Google doc and let me know what you think.
> >
> > —
> > Anand
> >
> >
> >
> > --
> > Dmitri Bourlatchkov
> > Senior Staff Software Engineer, Dremio
> > Dremio.com
> > <
> https://urldefense.com/v3/__https://www.dremio.com/?utm_medium=email&utm_source=signature&utm_term=na&utm_content=email-signature&utm_campaign=email-signature__;!!Iz9xO38YGHZK!_dbBfwEnZEuPp_8qIuBDoYfWRgCZKiPDMg9ouDltqsHhr9hqCHVlF9IedSkNVBDWupbneU7inC9YK-THL-lTjWkkiOwjJc0$
> >/
> > Follow Us on LinkedIn
> > <
> https://urldefense.com/v3/__https://www.linkedin.com/company/dremio__;!!Iz9xO38YGHZK!_dbBfwEnZEuPp_8qIuBDoYfWRgCZKiPDMg9ouDltqsHhr9hqCHVlF9IedSkNVBDWupbneU7inC9YK-THL-lTjWkkCLjBQ2M$
> >/
> > Get Started
> > <
> https://urldefense.com/v3/__https://www.dremio.com/get-started/__;!!Iz9xO38YGHZK!_dbBfwEnZEuPp_8qIuBDoYfWRgCZKiPDMg9ouDltqsHhr9hqCHVlF9IedSkNVBDWupbneU7inC9YK-THL-lTjWkkoKQUMII$
> >
> >
> >
> >The Agentic Lakehouse
> > *The only lakehouse built for agents, managed by agents*
> >
> >
> >
>
>

Re: Polaris Telemetry and Audit Trail

[Dmitri Bourlatchkov <di...@apache.org>]

Hi Prashant,

Re: separate (events / metrics) schema - in general I'd prefer to be able
to evolve events tables separately from the "metastore" schema too.

My rationale is that metastore code does not depend on events (or metrics)
persistence, therefore, their schemas could be evolved independently, which
aids developing events/metrics enhancements without affecting the main
metastore call paths.

Once does not have to bump the metastore schema in case events / metrics
DDL changes, but the user elected not to persist them.

The metrics feature paves the way in that direction, but if the community
agrees on this approach in general, I suppose we could refactor events to
that effect later.

Re: conditional bootstrap - it would certainly make sense to make
a ProductionReadinessCheck for verifying (on startup) that if metrics
persistence is enabled, we actually have tables in place.

Cheers,
Dmitri.

On Thu, Feb 12, 2026 at 12:14 PM Prashant Singh via dev <
dev@polaris.apache.org> wrote:

> Thank you for the Proposal Anand !
> The only feedback I had was if we are doing a conditional bootstrap of
> metrics table we need to make sure when someone enables the feature
> get a clear message that hey table was not bootstrapped we should bootstrap
> it first before enabling it ?
> In events I am not sure we discussed this but the events table gets created
> along with other required tables even when the config is off so we should
> make sure we do that.
>
> Regarding having a connection pool, i agree we should have named datasource
> for this to not hog the connection pool which is used by polaris for
> entities and ops
>
> Let me know how it sounds !
>
> Best,
> Prashant Singh
>
> On Mon, Feb 9, 2026 at 11:08 AM Anand Kumar Sankaran via dev <
> dev@polaris.apache.org> wrote:
>
> > Dmitri and others,
> >
> > Thanks for all the review feedback and being patient while we iterated.
> My
> > initial proposal was to persist the table metrics as events (least amount
> > of work 😊), we have come far from that proposal.
> >
> > To detach the SPI fully, there are a few things to do
> (loadSchemaVersion()
> > and shared DatasourceOperations). We can add a separate datasource config
> > to specify a different connection pool, perhaps. Can be done as a follow
> up
> > PR.
> >
> > —
> > Anand
> >
> > From: Dmitri Bourlatchkov <di...@apache.org>
> > Date: Monday, February 9, 2026 at 10:52 AM
> > To: dev@polaris.apache.org <de...@polaris.apache.org>
> > Cc: Anand Kumar Sankaran <an...@workday.com>
> > Subject: Re: Polaris Telemetry and Audit Trail
> >
> > This Message Is From an External Sender
> > This message came from outside your organization.
> > Report Suspicious<
> >
> https://us-phishalarm-ewt.proofpoint.com/EWT/v1/Iz9xO38YGHZK!YhNDZAGr2cumYduJ20ONMhMxljYPF8uZOe6R8BpEWyJ7tSG5GUUf8our05nYHWov0t1Vi7Rx_O8JuRNBBQQ57aBoctrR0wH7VSR9VxRuqYJPpz0q6I0JboPRfdLIwqal$
> > >
> >
> > Hi Anand and all,
> >
> > Thanks for pushing this feature forward! From my POV the SPI [3616] and
> > the schema [3523] PRs are good to merge as the initial implementation of
> > this feature (annotated as "beta"). The way is open to the JDBC impl. PR
> > [3385] (which will probably need a rebase after merging the SPI and
> schema
> > code).
> >
> > I'm sending this email to highlight the direction taken with respect to
> > the SQL schema in this feature, as it might have been visible enough to
> the
> > broader community during the PR reviews.
> >
> > * The scan / commit metrics schema is managed / versioned separately from
> > the JDBC MetaStore schema
> >
> > * The Metrics Persistence SPI is detached (mostly at this time, but
> > supposedly fully detached later) from the JDBC MetaStore
> >
> > * The JDBC DataSource for metrics may be the same or different from the
> > MetaStore datasource.
> >
> > * Metrics Persistence may have non-JDBC implementation in the future (not
> > implemented ATM)
> >
> > * JDBC Metrics Persistence may co-exist with other MetaStore
> > implementations.
> >
> > I believe this design allows great flexibility for end users to try this
> > feature out and keeps options open for future enhancements in Scan /
> Commit
> > Metrics in Polaris.
> >
> > [3616] https://github.com/apache/polaris/pull/3616<
> >
> https://urldefense.com/v3/__https://github.com/apache/polaris/pull/3616__;!!Iz9xO38YGHZK!7IwwfPXDN0ThhNZgol-n9Ww3FFLKcpZbSQ17lWEtcQnTHtJM9aVXnqGW99_6vmVkWACy7rYC8XEpN6Okdg$
> > >
> >
> > [3523] https://github.com/apache/polaris/pull/3523<
> >
> https://urldefense.com/v3/__https://github.com/apache/polaris/pull/3523__;!!Iz9xO38YGHZK!7IwwfPXDN0ThhNZgol-n9Ww3FFLKcpZbSQ17lWEtcQnTHtJM9aVXnqGW99_6vmVkWACy7rYC8XGOCHKhSg$
> > >
> >
> > [3385] https://github.com/apache/polaris/pull/3385<
> >
> https://urldefense.com/v3/__https://github.com/apache/polaris/pull/3385__;!!Iz9xO38YGHZK!7IwwfPXDN0ThhNZgol-n9Ww3FFLKcpZbSQ17lWEtcQnTHtJM9aVXnqGW99_6vmVkWACy7rYC8XFlxLMhvQ$
> > >
> >
> > Thoughts / comments?
> >
> > Cheers,
> > Dmitri.
> >
> >
> > On Thu, Feb 5, 2026 at 6:50 PM Anand Kumar Sankaran via dev <
> > dev@polaris.apache.org<ma...@polaris.apache.org>> wrote:
> > Hi Dmitri,
> >
> > Following up on your feedback regarding the metrics schema separation
> > approach. I've implemented the simplified design you suggested in
> > https://github.com/apache/polaris/pull/3523<
> >
> https://urldefense.com/v3/__https://github.com/apache/polaris/pull/3523__;!!Iz9xO38YGHZK!7IwwfPXDN0ThhNZgol-n9Ww3FFLKcpZbSQ17lWEtcQnTHtJM9aVXnqGW99_6vmVkWACy7rYC8XGOCHKhSg$
> > >:
> >
> >
> >  The metrics schema can now evolve independently from the entity schema.
> > Let me know if you'd like any changes to this approach.
> >
> >  Thanks,
> >   Anand
> >
> >
> >
> > From: Dmitri Bourlatchkov <di...@apache.org>>
> > Date: Thursday, February 5, 2026 at 2:57 PM
> > To: dev@polaris.apache.org<ma...@polaris.apache.org> <
> > dev@polaris.apache.org<ma...@polaris.apache.org>>
> > Cc: Anand Kumar Sankaran <anand.sankaran@workday.com<mailto:
> > anand.sankaran@workday.com>>
> > Subject: Re: Polaris Telemetry and Audit Trail
> >
> > This Message Is From an External Sender
> > This message came from outside your organization.
> > Report Suspicious<
> >
> https://us-phishalarm-ewt.proofpoint.com/EWT/v1/Iz9xO38YGHZK!YhNDZAGr2cumY1iAus5GNbkIxSXoQaI552sXgYyv7Z9Wrvb7qeTskSy5WkXTVDJ_Bel2AbNh_fQVn-HS_UtOSdOtpsNfzhXDkK4fYWdTeoeBe70A6oQVGpAYT3SrbEvs$
> > >
> >
> > Hi Anand,
> >
> > Re: MetricsPersistence Factory - TBH I do not think it is required. If we
> > go with an MetricsPersistence impl. independent of the JDBC MetaStore, it
> > could be injected into PersistingMetricsReporter directly by CDI.
> >
> > Re: DataSource - for a start we could probably reuse the same DataSource
> > as the JDBS MetaStore uses. In principle, we could introduce a separate
> > DataSource, but I do not think we need to complicate the initial
> > implementation with that. It can be added later, if required.
> >
> > Re: schema / bootstrap - I'm thinking, indeed, of having a separate *.sql
> > file for metrics DDL. Then we could add an option to the bootstrap
> command
> > in the Admin Tool, or even a new command for creating / updating the
> > metrics schema. This might require some refactoring, but I believe the
> > bootstrap code needs some cleanup anyway.
> >
> > The general idea is to allow the metrics schema to evolve independently.
> > It's a new and actively developed feature, so having flexibility is going
> > to be good for making progress / improvements, I hope.
> >
> > Does that sound reasonable?
> >
> > Thanks,
> > Dmitri.
> >
> > On Thu, Feb 5, 2026 at 3:47 PM Anand Kumar Sankaran via dev <
> > dev@polaris.apache.org<ma...@polaris.apache.org><mailto:
> > dev@polaris.apache.org<ma...@polaris.apache.org>>> wrote:
> > Hi Dmitri
> >
> > The SPI is designed to be pluggable via
> > MetaStoreManagerFactory.getOrCreateMetricsPersistence()<
> >
> https://github.com/apache/polaris/pull/3385/changes#diff-0e34667d083d30cace067936b87c736201e5a8cb001aa0b12e203ee52cddae99R51
> > <
> >
> https://urldefense.com/v3/__https://github.com/apache/polaris/pull/3385/changes*diff-0e34667d083d30cace067936b87c736201e5a8cb001aa0b12e203ee52cddae99R51__;Iw!!Iz9xO38YGHZK!7IwwfPXDN0ThhNZgol-n9Ww3FFLKcpZbSQ17lWEtcQnTHtJM9aVXnqGW99_6vmVkWACy7rYC8XFxRr4qRw$
> > ><
> >
> https://urldefense.com/v3/__https://github.com/apache/polaris/pull/3385/changes*diff-0e34667d083d30cace067936b87c736201e5a8cb001aa0b12e203ee52cddae99R51__;Iw!!Iz9xO38YGHZK!_K1mcH0_2x0nbU-mZlbBi3tk9QAGjB6VInPiqkQU21mpIP_ft2bXXyIiADNoAcotNr9AKqlphh65taUhcw$
> >>.
> > The default returns MetricsPersistence.NOOP, and only JDBC with schema
> v4+
> > returns a functional implementation.
> >
> >  Your proposal makes sense to me. To allow NoSQL Entity Persistence +
> JDBC
> > Metrics Persistence would require:
> >
> >      1. Separate metrics schema files (e.g., metrics-schema-v1.sql) that
> > can be bootstrapped independently
> >      2. Separate datasource configuration for metrics (e.g.,
> > polaris.metrics.jdbc.*)
> >      3. A dedicated `MetricsPersistenceFactory` interface (separate from
> > MetaStoreManagerFactory)
> >      4. Updates to the Admin Tool bootstrap workflow to handle both
> schemas
> >
> > Does that sound reasonable?  If we do this, all this work should go into
> > https://github.com/apache/polaris/pull/3523<
> >
> https://urldefense.com/v3/__https://github.com/apache/polaris/pull/3523__;!!Iz9xO38YGHZK!7IwwfPXDN0ThhNZgol-n9Ww3FFLKcpZbSQ17lWEtcQnTHtJM9aVXnqGW99_6vmVkWACy7rYC8XGOCHKhSg$
> > ><
> >
> https://urldefense.com/v3/__https://github.com/apache/polaris/pull/3523__;!!Iz9xO38YGHZK!_K1mcH0_2x0nbU-mZlbBi3tk9QAGjB6VInPiqkQU21mpIP_ft2bXXyIiADNoAcotNr9AKqlphh4ZISZcDQ$
> >
> > as one atomic commit, correct?
> >
> > —
> > Anand
> >
> >
> >
> > From: Dmitri Bourlatchkov <dimas@apache.org<mailto:dimas@apache.org
> > ><ma...@apache.org>>>
> > Date: Thursday, February 5, 2026 at 8:28 AM
> > To: Anand Kumar Sankaran <anand.sankaran@workday.com<mailto:
> > anand.sankaran@workday.com><mailto:anand.sankaran@workday.com<mailto:
> > anand.sankaran@workday.com>>>
> > Cc: dev@polaris.apache.org<ma...@polaris.apache.org><mailto:
> > dev@polaris.apache.org<ma...@polaris.apache.org>> <
> > dev@polaris.apache.org<ma...@polaris.apache.org><mailto:
> > dev@polaris.apache.org<ma...@polaris.apache.org>>>
> > Subject: Re: Polaris Telemetry and Audit Trail
> >
> > This Message Is From an External Sender
> > This message came from outside your organization.
> > Report Suspicious<
> >
> https://us-phishalarm-ewt.proofpoint.com/EWT/v1/Iz9xO38YGHZK!YhNDZAGr2cumbRcPlIOttKwvGr4f38TF-J2-LmOw8H1KsFpWaM3W5g28Oq5qW2gnw2-WHz0rsWhm7F7hSGykA9l01LKYGuCcCkexkHp1sUssM_IegCtjk_id9s891orv$
> > >
> >
> >
> > Hi Anand,
> >
> > Thanks for driving the SPI definition for scan/commits metrics.
> >
> > I did not think from this angle before, but now I suppose it might be
> worth
> > introducing a separate SQL schema for metrics, detached from the main
> JDBC
> > Persistence schema.
> >
> > My rationale is that the SPI may be exercised independently of the JDBC
> > Entity Persistence. I can imagine people using NoSQL Entity Persistence
> > plus JDBC Scan Metrics Persistence in the same deployment.
> >
> > Using a separate schema will add further work on the Bootstrap workflow
> > (Admin Tool), but I hope the end result is going to be more convenient
> and
> > versatile for end users.
> >
> > WDYT?
> >
> > Thanks,
> > Dmitri.
> >
> > On Thu, Jan 29, 2026 at 1:49 PM Anand Kumar Sankaran <
> > anand.sankaran@workday.com<ma...@workday.com><mailto:
> > anand.sankaran@workday.com<ma...@workday.com>>> wrote:
> >
> > > Hi Dmitri,
> > >
> > > Thank you for taking the time to review. I had pulled out the schema in
> > > its own PR (based on comments) here:
> > >
> >
> https://urldefense.com/v3/__https://github.com/apache/polaris/pull/3523__;!!Iz9xO38YGHZK!65BVFQDe_GtKbDW8LkNdzVS-WYewzWsSns5zFHKclbMU3vUy2UaTXLB5G-wql1yyqoTFyDc9Bt3GRsh6AA$
> > >
> > > I will update the document with the SPI and carve out another PR that
> > > focuses on the SPI.
> > >
> > > —
> > > Anand
> > >
> > > *From: *Dmitri Bourlatchkov <dmitri.bourlatchkov@dremio.com<mailto:
> > dmitri.bourlatchkov@dremio.com><mailto:dmitri.bourlatchkov@dremio.com
> > <ma...@dremio.com>>>
> > > *Date: *Thursday, January 29, 2026 at 10:43 AM
> > > *To: *dev@polaris.apache.org<ma...@polaris.apache.org><mailto:
> > dev@polaris.apache.org<ma...@polaris.apache.org>> <
> > dev@polaris.apache.org<ma...@polaris.apache.org><mailto:
> > dev@polaris.apache.org<ma...@polaris.apache.org>>>
> > > *Cc: *Anand Kumar Sankaran <anand.sankaran@workday.com<mailto:
> > anand.sankaran@workday.com><mailto:anand.sankaran@workday.com<mailto:
> > anand.sankaran@workday.com>>>
> > > *Subject: *Re: Polaris Telemetry and Audit Trail
> > >
> > > This Message Is From an External Sender
> > > This message came from outside your organization.
> > > Report Suspicious
> > > <
> >
> https://us-phishalarm-ewt.proofpoint.com/EWT/v1/Iz9xO38YGHZK!YhNDZAGr2cumY1cAlUeoPrU2eNUPz9Wxq2PaG1t99vEN3v60gucxC3Qsk-xK09RndcW46WtNjMMjQZbaQ495h1D1g293F_uqLhPM8gDMlA6xRmo5b5uqrSh2v7QKHrhQ$
> > >
> > >
> > > Hi Anand,
> > >
> > > Thanks for making a proposal doc and starting this thread. Apologies
> for
> > > late comments.
> > >
> > > As you know I already reviewed a couple of related PRs, which had a
> > > smaller impact on the codebase, but now, I believe, we're coming to the
> > > core of this feature and it might need a deeper discussion.
> > >
> > > For the sake of awareness of other project members, I'd like to
> > > highlight some aspects by email.
> > >
> > > * Storing scan metrics in RDBMS in general looks like a
> > > reasonable approach to me. However, Polaris Persistence is an
> extensible
> > > mechanism. Other backends are possible, including privately developed
> > > backends.
> > >
> > > From this POV, I believe it might be preferable to start not with an
> > RDBMS
> > > schema, but with a java SPI outlining expected read/write operations
> and
> > > the data model. This will make it easier to assess impact on other
> > > Persistence implementations. Naturally, Polaris service code will have
> to
> > > be able to work seamlessly regardless of the backend impl. (including
> the
> > > do-nothing impl.).
> > >
> > > I see that some connection between the SQL schema and java code is made
> > in
> > > PR [3385], but it does not appear to offer a clean SPI that could be
> > > implemented by different Persistence backends. Would it be ok from your
> > > perspective to make another PR with just SPI code?
> > >
> > > * Regarding ad-hoc SQL queries - it is certainly a valid use case for
> > > custom code to query the Polaris database directly. However, those
> ad-hoc
> > > queries may require certain indexes, which may not have a relevant use
> > case
> > > in OSS code. In that case, I believe it would be preferable to manage
> the
> > > extra indexes in custom code (not in Polaris). Whether this is the case
> > or
> > > not currently, I cannot say with certainty, so apologies if this is a
> > false
> > > alarm... Having the java SPI available would be a huge aid to reasoning
> > > about the proposed SQL schema, I think.
> > >
> > > That said, I do not mean to complicate the development of this feature,
> > > just trying to structure it in a way that is hopefully easy to
> understand
> > > (including myself), maintain and expand later. If I missed something,
> > > please feel free to point it out.
> > >
> > > [3385]
> >
> https://urldefense.com/v3/__https://github.com/apache/polaris/pull/3385__;!!Iz9xO38YGHZK!65BVFQDe_GtKbDW8LkNdzVS-WYewzWsSns5zFHKclbMU3vUy2UaTXLB5G-wql1yyqoTFyDc9Bt2W-eKl7A$
> > > <
> >
> https://urldefense.com/v3/__https://github.com/apache/polaris/pull/3385__;!!Iz9xO38YGHZK!_dbBfwEnZEuPp_8qIuBDoYfWRgCZKiPDMg9ouDltqsHhr9hqCHVlF9IedSkNVBDWupbneU7inC9YK-THL-lTjWkkyZhI0OE$
> > >
> > >
> > >Thanks,
> > > Dmitri.
> > >
> > > On Wed, Jan 14, 2026 at 10:05 PM Anand Kumar Sankaran via dev <
> > > dev@polaris.apache.org<ma...@polaris.apache.org><mailto:
> > dev@polaris.apache.org<ma...@polaris.apache.org>>> wrote:
> > >
> > > Hi Yufei,
> > >
> > > Gave commenting privileges to all.
> > >
> > > Thanks.
> > >
> > > —
> > > Anand
> > >
> > > From: Yufei Gu <flyrain000@gmail.com<mailto:flyrain000@gmail.com
> > ><ma...@gmail.com>>>
> > > Date: Wednesday, January 14, 2026 at 6:13 PM
> > > To: dev@polaris.apache.org<ma...@polaris.apache.org><mailto:
> > dev@polaris.apache.org<ma...@polaris.apache.org>> <
> > dev@polaris.apache.org<ma...@polaris.apache.org><mailto:
> > dev@polaris.apache.org<ma...@polaris.apache.org>>>
> > > Cc: Anand Kumar Sankaran <anand.sankaran@workday.com<mailto:
> > anand.sankaran@workday.com><mailto:anand.sankaran@workday.com<mailto:
> > anand.sankaran@workday.com>>>
> > > Subject: Re: Polaris Telemetry and Audit Trail
> > >
> > > This Message Is From an External Sender
> > > This message came from outside your organization.
> > > Report Suspicious<
> > >
> >
> https://us-phishalarm-ewt.proofpoint.com/EWT/v1/Iz9xO38YGHZK!YhNDZABkHi1B6hyPVYUK0T-K5j4Aoqdrlu07UjBpJcpjOsz7Ie7d_DhPl-ywi50dfQz4O6LL-AHLCOQTQ0X7Le8yD2cF0ogquGFB8Aw8uQLgB0RL6Ezpa-pLDPA4ziXA$
> > > >
> > >
> > > Hi Anand, thanks for the design doc and PR. Can you allow comments in
> the
> > > doc so that people can chime in? Thanks!
> > >
> > > Yufei
> > >
> > >
> > > On Sat, Jan 10, 2026 at 9:09 AM Anand Kumar Sankaran via dev <
> > > dev@polaris.apache.org<ma...@polaris.apache.org><mailto:
> > dev@polaris.apache.org<ma...@polaris.apache.org>><mailto:
> > dev@polaris.apache.org<ma...@polaris.apache.org><mailto:
> > dev@polaris.apache.org<ma...@polaris.apache.org>>>> wrote:
> > > Hi all
> > >
> > > My first PR for adding AWS STS Session Tags support for credential
> > vending
> > > was merged.
> >
> https://urldefense.com/v3/__https://github.com/apache/polaris/pull/3327__;!!Iz9xO38YGHZK!65BVFQDe_GtKbDW8LkNdzVS-WYewzWsSns5zFHKclbMU3vUy2UaTXLB5G-wql1yyqoTFyDc9Bt0YHY9sdA$
> > > <
> >
> https://urldefense.com/v3/__https://github.com/apache/polaris/pull/3327__;!!Iz9xO38YGHZK!_dbBfwEnZEuPp_8qIuBDoYfWRgCZKiPDMg9ouDltqsHhr9hqCHVlF9IedSkNVBDWupbneU7inC9YK-THL-lTjWkklJ4Xf9Y$
> > >
> > ><
> > >
> >
> https://urldefense.com/v3/__https://github.com/apache/polaris/pull/3327__;!!Iz9xO38YGHZK!8vlQ2_OqjmZNIpbrTTluFFNKFCEeM87nr95MP5pMpMu5fFWVSoBZ3eb6-C2xbAGCG-ry5eaCO-pFLxl3IZAJGqY$
> > >>
> > >
> > > I have been working with Prashant on an end-to-end telemetry and audit
> > > trail tracking for Polaris.  It is documented here.
> > >
> >
> https://urldefense.com/v3/__https://docs.google.com/document/d/1Ehzvi5RNPs4hChkBFI6VD23myEqm-7sWW3d2kjmuYj8/edit?tab=t.0__;!!Iz9xO38YGHZK!65BVFQDe_GtKbDW8LkNdzVS-WYewzWsSns5zFHKclbMU3vUy2UaTXLB5G-wql1yyqoTFyDc9Bt11Y1rbNQ$
> > > <
> >
> https://urldefense.com/v3/__https://docs.google.com/document/d/1Ehzvi5RNPs4hChkBFI6VD23myEqm-7sWW3d2kjmuYj8/edit?tab=t.0__;!!Iz9xO38YGHZK!_dbBfwEnZEuPp_8qIuBDoYfWRgCZKiPDMg9ouDltqsHhr9hqCHVlF9IedSkNVBDWupbneU7inC9YK-THL-lTjWkk8MVJPlY$
> > >
> > ><
> > >
> >
> https://urldefense.com/v3/__https://docs.google.com/document/d/1Ehzvi5RNPs4hChkBFI6VD23myEqm-7sWW3d2kjmuYj8/edit?tab=t.0__;!!Iz9xO38YGHZK!8vlQ2_OqjmZNIpbrTTluFFNKFCEeM87nr95MP5pMpMu5fFWVSoBZ3eb6-C2xbAGCG-ry5eaCO-pFLxl3hYqTlcg$
> > >>
> > >
> > > Based on Prashant’s initial feedback (parity with Apache Gravitino for
> > > metrics reports), I have an initial PR here.
> > >
> >
> https://urldefense.com/v3/__https://github.com/apache/polaris/pull/3385__;!!Iz9xO38YGHZK!65BVFQDe_GtKbDW8LkNdzVS-WYewzWsSns5zFHKclbMU3vUy2UaTXLB5G-wql1yyqoTFyDc9Bt2W-eKl7A$
> > > <
> >
> https://urldefense.com/v3/__https://github.com/apache/polaris/pull/3385__;!!Iz9xO38YGHZK!_dbBfwEnZEuPp_8qIuBDoYfWRgCZKiPDMg9ouDltqsHhr9hqCHVlF9IedSkNVBDWupbneU7inC9YK-THL-lTjWkkyZhI0OE$
> > >
> > ><
> > >
> >
> https://urldefense.com/v3/__https://github.com/apache/polaris/pull/3385__;!!Iz9xO38YGHZK!8vlQ2_OqjmZNIpbrTTluFFNKFCEeM87nr95MP5pMpMu5fFWVSoBZ3eb6-C2xbAGCG-ry5eaCO-pFLxl39opCznI$
> > >>
> > >
> > > I will be reaching out to various folks for review. I am open to
> breaking
> > > down the PR into smaller logical groups of PR if it helps.
> > >
> > > I request you to look at the Google doc and let me know what you think.
> > >
> > > —
> > > Anand
> > >
> > >
> > >
> > > --
> > > Dmitri Bourlatchkov
> > > Senior Staff Software Engineer, Dremio
> > > Dremio.com
> > > <
> >
> https://urldefense.com/v3/__https://www.dremio.com/?utm_medium=email&utm_source=signature&utm_term=na&utm_content=email-signature&utm_campaign=email-signature__;!!Iz9xO38YGHZK!_dbBfwEnZEuPp_8qIuBDoYfWRgCZKiPDMg9ouDltqsHhr9hqCHVlF9IedSkNVBDWupbneU7inC9YK-THL-lTjWkkiOwjJc0$
> > >/
> > > Follow Us on LinkedIn
> > > <
> >
> https://urldefense.com/v3/__https://www.linkedin.com/company/dremio__;!!Iz9xO38YGHZK!_dbBfwEnZEuPp_8qIuBDoYfWRgCZKiPDMg9ouDltqsHhr9hqCHVlF9IedSkNVBDWupbneU7inC9YK-THL-lTjWkkCLjBQ2M$
> > >/
> > > Get Started
> > > <
> >
> https://urldefense.com/v3/__https://www.dremio.com/get-started/__;!!Iz9xO38YGHZK!_dbBfwEnZEuPp_8qIuBDoYfWRgCZKiPDMg9ouDltqsHhr9hqCHVlF9IedSkNVBDWupbneU7inC9YK-THL-lTjWkkoKQUMII$
> > >
> > >
> > >
> > >The Agentic Lakehouse
> > > *The only lakehouse built for agents, managed by agents*
> > >
> > >
> > >
> >
> >
>

Re: Polaris Telemetry and Audit Trail

[Prashant Singh via dev <de...@polaris.apache.org>]

Thank you for the Proposal Anand !
The only feedback I had was if we are doing a conditional bootstrap of
metrics table we need to make sure when someone enables the feature
get a clear message that hey table was not bootstrapped we should bootstrap
it first before enabling it ?
In events I am not sure we discussed this but the events table gets created
along with other required tables even when the config is off so we should
make sure we do that.

Regarding having a connection pool, i agree we should have named datasource
for this to not hog the connection pool which is used by polaris for
entities and ops

Let me know how it sounds !

Best,
Prashant Singh

On Mon, Feb 9, 2026 at 11:08 AM Anand Kumar Sankaran via dev <
dev@polaris.apache.org> wrote:

> Dmitri and others,
>
> Thanks for all the review feedback and being patient while we iterated. My
> initial proposal was to persist the table metrics as events (least amount
> of work 😊), we have come far from that proposal.
>
> To detach the SPI fully, there are a few things to do (loadSchemaVersion()
> and shared DatasourceOperations). We can add a separate datasource config
> to specify a different connection pool, perhaps. Can be done as a follow up
> PR.
>
> —
> Anand
>
> From: Dmitri Bourlatchkov <di...@apache.org>
> Date: Monday, February 9, 2026 at 10:52 AM
> To: dev@polaris.apache.org <de...@polaris.apache.org>
> Cc: Anand Kumar Sankaran <an...@workday.com>
> Subject: Re: Polaris Telemetry and Audit Trail
>
> This Message Is From an External Sender
> This message came from outside your organization.
> Report Suspicious<
> https://us-phishalarm-ewt.proofpoint.com/EWT/v1/Iz9xO38YGHZK!YhNDZAGr2cumYduJ20ONMhMxljYPF8uZOe6R8BpEWyJ7tSG5GUUf8our05nYHWov0t1Vi7Rx_O8JuRNBBQQ57aBoctrR0wH7VSR9VxRuqYJPpz0q6I0JboPRfdLIwqal$
> >
>
> Hi Anand and all,
>
> Thanks for pushing this feature forward! From my POV the SPI [3616] and
> the schema [3523] PRs are good to merge as the initial implementation of
> this feature (annotated as "beta"). The way is open to the JDBC impl. PR
> [3385] (which will probably need a rebase after merging the SPI and schema
> code).
>
> I'm sending this email to highlight the direction taken with respect to
> the SQL schema in this feature, as it might have been visible enough to the
> broader community during the PR reviews.
>
> * The scan / commit metrics schema is managed / versioned separately from
> the JDBC MetaStore schema
>
> * The Metrics Persistence SPI is detached (mostly at this time, but
> supposedly fully detached later) from the JDBC MetaStore
>
> * The JDBC DataSource for metrics may be the same or different from the
> MetaStore datasource.
>
> * Metrics Persistence may have non-JDBC implementation in the future (not
> implemented ATM)
>
> * JDBC Metrics Persistence may co-exist with other MetaStore
> implementations.
>
> I believe this design allows great flexibility for end users to try this
> feature out and keeps options open for future enhancements in Scan / Commit
> Metrics in Polaris.
>
> [3616] https://github.com/apache/polaris/pull/3616<
> https://urldefense.com/v3/__https://github.com/apache/polaris/pull/3616__;!!Iz9xO38YGHZK!7IwwfPXDN0ThhNZgol-n9Ww3FFLKcpZbSQ17lWEtcQnTHtJM9aVXnqGW99_6vmVkWACy7rYC8XEpN6Okdg$
> >
>
> [3523] https://github.com/apache/polaris/pull/3523<
> https://urldefense.com/v3/__https://github.com/apache/polaris/pull/3523__;!!Iz9xO38YGHZK!7IwwfPXDN0ThhNZgol-n9Ww3FFLKcpZbSQ17lWEtcQnTHtJM9aVXnqGW99_6vmVkWACy7rYC8XGOCHKhSg$
> >
>
> [3385] https://github.com/apache/polaris/pull/3385<
> https://urldefense.com/v3/__https://github.com/apache/polaris/pull/3385__;!!Iz9xO38YGHZK!7IwwfPXDN0ThhNZgol-n9Ww3FFLKcpZbSQ17lWEtcQnTHtJM9aVXnqGW99_6vmVkWACy7rYC8XFlxLMhvQ$
> >
>
> Thoughts / comments?
>
> Cheers,
> Dmitri.
>
>
> On Thu, Feb 5, 2026 at 6:50 PM Anand Kumar Sankaran via dev <
> dev@polaris.apache.org<ma...@polaris.apache.org>> wrote:
> Hi Dmitri,
>
> Following up on your feedback regarding the metrics schema separation
> approach. I've implemented the simplified design you suggested in
> https://github.com/apache/polaris/pull/3523<
> https://urldefense.com/v3/__https://github.com/apache/polaris/pull/3523__;!!Iz9xO38YGHZK!7IwwfPXDN0ThhNZgol-n9Ww3FFLKcpZbSQ17lWEtcQnTHtJM9aVXnqGW99_6vmVkWACy7rYC8XGOCHKhSg$
> >:
>
>
>  The metrics schema can now evolve independently from the entity schema.
> Let me know if you'd like any changes to this approach.
>
>  Thanks,
>   Anand
>
>
>
> From: Dmitri Bourlatchkov <di...@apache.org>>
> Date: Thursday, February 5, 2026 at 2:57 PM
> To: dev@polaris.apache.org<ma...@polaris.apache.org> <
> dev@polaris.apache.org<ma...@polaris.apache.org>>
> Cc: Anand Kumar Sankaran <anand.sankaran@workday.com<mailto:
> anand.sankaran@workday.com>>
> Subject: Re: Polaris Telemetry and Audit Trail
>
> This Message Is From an External Sender
> This message came from outside your organization.
> Report Suspicious<
> https://us-phishalarm-ewt.proofpoint.com/EWT/v1/Iz9xO38YGHZK!YhNDZAGr2cumY1iAus5GNbkIxSXoQaI552sXgYyv7Z9Wrvb7qeTskSy5WkXTVDJ_Bel2AbNh_fQVn-HS_UtOSdOtpsNfzhXDkK4fYWdTeoeBe70A6oQVGpAYT3SrbEvs$
> >
>
> Hi Anand,
>
> Re: MetricsPersistence Factory - TBH I do not think it is required. If we
> go with an MetricsPersistence impl. independent of the JDBC MetaStore, it
> could be injected into PersistingMetricsReporter directly by CDI.
>
> Re: DataSource - for a start we could probably reuse the same DataSource
> as the JDBS MetaStore uses. In principle, we could introduce a separate
> DataSource, but I do not think we need to complicate the initial
> implementation with that. It can be added later, if required.
>
> Re: schema / bootstrap - I'm thinking, indeed, of having a separate *.sql
> file for metrics DDL. Then we could add an option to the bootstrap command
> in the Admin Tool, or even a new command for creating / updating the
> metrics schema. This might require some refactoring, but I believe the
> bootstrap code needs some cleanup anyway.
>
> The general idea is to allow the metrics schema to evolve independently.
> It's a new and actively developed feature, so having flexibility is going
> to be good for making progress / improvements, I hope.
>
> Does that sound reasonable?
>
> Thanks,
> Dmitri.
>
> On Thu, Feb 5, 2026 at 3:47 PM Anand Kumar Sankaran via dev <
> dev@polaris.apache.org<ma...@polaris.apache.org><mailto:
> dev@polaris.apache.org<ma...@polaris.apache.org>>> wrote:
> Hi Dmitri
>
> The SPI is designed to be pluggable via
> MetaStoreManagerFactory.getOrCreateMetricsPersistence()<
> https://github.com/apache/polaris/pull/3385/changes#diff-0e34667d083d30cace067936b87c736201e5a8cb001aa0b12e203ee52cddae99R51
> <
> https://urldefense.com/v3/__https://github.com/apache/polaris/pull/3385/changes*diff-0e34667d083d30cace067936b87c736201e5a8cb001aa0b12e203ee52cddae99R51__;Iw!!Iz9xO38YGHZK!7IwwfPXDN0ThhNZgol-n9Ww3FFLKcpZbSQ17lWEtcQnTHtJM9aVXnqGW99_6vmVkWACy7rYC8XFxRr4qRw$
> ><
> https://urldefense.com/v3/__https://github.com/apache/polaris/pull/3385/changes*diff-0e34667d083d30cace067936b87c736201e5a8cb001aa0b12e203ee52cddae99R51__;Iw!!Iz9xO38YGHZK!_K1mcH0_2x0nbU-mZlbBi3tk9QAGjB6VInPiqkQU21mpIP_ft2bXXyIiADNoAcotNr9AKqlphh65taUhcw$>>.
> The default returns MetricsPersistence.NOOP, and only JDBC with schema v4+
> returns a functional implementation.
>
>  Your proposal makes sense to me. To allow NoSQL Entity Persistence + JDBC
> Metrics Persistence would require:
>
>      1. Separate metrics schema files (e.g., metrics-schema-v1.sql) that
> can be bootstrapped independently
>      2. Separate datasource configuration for metrics (e.g.,
> polaris.metrics.jdbc.*)
>      3. A dedicated `MetricsPersistenceFactory` interface (separate from
> MetaStoreManagerFactory)
>      4. Updates to the Admin Tool bootstrap workflow to handle both schemas
>
> Does that sound reasonable?  If we do this, all this work should go into
> https://github.com/apache/polaris/pull/3523<
> https://urldefense.com/v3/__https://github.com/apache/polaris/pull/3523__;!!Iz9xO38YGHZK!7IwwfPXDN0ThhNZgol-n9Ww3FFLKcpZbSQ17lWEtcQnTHtJM9aVXnqGW99_6vmVkWACy7rYC8XGOCHKhSg$
> ><
> https://urldefense.com/v3/__https://github.com/apache/polaris/pull/3523__;!!Iz9xO38YGHZK!_K1mcH0_2x0nbU-mZlbBi3tk9QAGjB6VInPiqkQU21mpIP_ft2bXXyIiADNoAcotNr9AKqlphh4ZISZcDQ$>
> as one atomic commit, correct?
>
> —
> Anand
>
>
>
> From: Dmitri Bourlatchkov <dimas@apache.org<mailto:dimas@apache.org
> ><ma...@apache.org>>>
> Date: Thursday, February 5, 2026 at 8:28 AM
> To: Anand Kumar Sankaran <anand.sankaran@workday.com<mailto:
> anand.sankaran@workday.com><mailto:anand.sankaran@workday.com<mailto:
> anand.sankaran@workday.com>>>
> Cc: dev@polaris.apache.org<ma...@polaris.apache.org><mailto:
> dev@polaris.apache.org<ma...@polaris.apache.org>> <
> dev@polaris.apache.org<ma...@polaris.apache.org><mailto:
> dev@polaris.apache.org<ma...@polaris.apache.org>>>
> Subject: Re: Polaris Telemetry and Audit Trail
>
> This Message Is From an External Sender
> This message came from outside your organization.
> Report Suspicious<
> https://us-phishalarm-ewt.proofpoint.com/EWT/v1/Iz9xO38YGHZK!YhNDZAGr2cumbRcPlIOttKwvGr4f38TF-J2-LmOw8H1KsFpWaM3W5g28Oq5qW2gnw2-WHz0rsWhm7F7hSGykA9l01LKYGuCcCkexkHp1sUssM_IegCtjk_id9s891orv$
> >
>
>
> Hi Anand,
>
> Thanks for driving the SPI definition for scan/commits metrics.
>
> I did not think from this angle before, but now I suppose it might be worth
> introducing a separate SQL schema for metrics, detached from the main JDBC
> Persistence schema.
>
> My rationale is that the SPI may be exercised independently of the JDBC
> Entity Persistence. I can imagine people using NoSQL Entity Persistence
> plus JDBC Scan Metrics Persistence in the same deployment.
>
> Using a separate schema will add further work on the Bootstrap workflow
> (Admin Tool), but I hope the end result is going to be more convenient and
> versatile for end users.
>
> WDYT?
>
> Thanks,
> Dmitri.
>
> On Thu, Jan 29, 2026 at 1:49 PM Anand Kumar Sankaran <
> anand.sankaran@workday.com<ma...@workday.com><mailto:
> anand.sankaran@workday.com<ma...@workday.com>>> wrote:
>
> > Hi Dmitri,
> >
> > Thank you for taking the time to review. I had pulled out the schema in
> > its own PR (based on comments) here:
> >
> https://urldefense.com/v3/__https://github.com/apache/polaris/pull/3523__;!!Iz9xO38YGHZK!65BVFQDe_GtKbDW8LkNdzVS-WYewzWsSns5zFHKclbMU3vUy2UaTXLB5G-wql1yyqoTFyDc9Bt3GRsh6AA$
> >
> > I will update the document with the SPI and carve out another PR that
> > focuses on the SPI.
> >
> > —
> > Anand
> >
> > *From: *Dmitri Bourlatchkov <dmitri.bourlatchkov@dremio.com<mailto:
> dmitri.bourlatchkov@dremio.com><mailto:dmitri.bourlatchkov@dremio.com
> <ma...@dremio.com>>>
> > *Date: *Thursday, January 29, 2026 at 10:43 AM
> > *To: *dev@polaris.apache.org<ma...@polaris.apache.org><mailto:
> dev@polaris.apache.org<ma...@polaris.apache.org>> <
> dev@polaris.apache.org<ma...@polaris.apache.org><mailto:
> dev@polaris.apache.org<ma...@polaris.apache.org>>>
> > *Cc: *Anand Kumar Sankaran <anand.sankaran@workday.com<mailto:
> anand.sankaran@workday.com><mailto:anand.sankaran@workday.com<mailto:
> anand.sankaran@workday.com>>>
> > *Subject: *Re: Polaris Telemetry and Audit Trail
> >
> > This Message Is From an External Sender
> > This message came from outside your organization.
> > Report Suspicious
> > <
> https://us-phishalarm-ewt.proofpoint.com/EWT/v1/Iz9xO38YGHZK!YhNDZAGr2cumY1cAlUeoPrU2eNUPz9Wxq2PaG1t99vEN3v60gucxC3Qsk-xK09RndcW46WtNjMMjQZbaQ495h1D1g293F_uqLhPM8gDMlA6xRmo5b5uqrSh2v7QKHrhQ$
> >
> >
> > Hi Anand,
> >
> > Thanks for making a proposal doc and starting this thread. Apologies for
> > late comments.
> >
> > As you know I already reviewed a couple of related PRs, which had a
> > smaller impact on the codebase, but now, I believe, we're coming to the
> > core of this feature and it might need a deeper discussion.
> >
> > For the sake of awareness of other project members, I'd like to
> > highlight some aspects by email.
> >
> > * Storing scan metrics in RDBMS in general looks like a
> > reasonable approach to me. However, Polaris Persistence is an extensible
> > mechanism. Other backends are possible, including privately developed
> > backends.
> >
> > From this POV, I believe it might be preferable to start not with an
> RDBMS
> > schema, but with a java SPI outlining expected read/write operations and
> > the data model. This will make it easier to assess impact on other
> > Persistence implementations. Naturally, Polaris service code will have to
> > be able to work seamlessly regardless of the backend impl. (including the
> > do-nothing impl.).
> >
> > I see that some connection between the SQL schema and java code is made
> in
> > PR [3385], but it does not appear to offer a clean SPI that could be
> > implemented by different Persistence backends. Would it be ok from your
> > perspective to make another PR with just SPI code?
> >
> > * Regarding ad-hoc SQL queries - it is certainly a valid use case for
> > custom code to query the Polaris database directly. However, those ad-hoc
> > queries may require certain indexes, which may not have a relevant use
> case
> > in OSS code. In that case, I believe it would be preferable to manage the
> > extra indexes in custom code (not in Polaris). Whether this is the case
> or
> > not currently, I cannot say with certainty, so apologies if this is a
> false
> > alarm... Having the java SPI available would be a huge aid to reasoning
> > about the proposed SQL schema, I think.
> >
> > That said, I do not mean to complicate the development of this feature,
> > just trying to structure it in a way that is hopefully easy to understand
> > (including myself), maintain and expand later. If I missed something,
> > please feel free to point it out.
> >
> > [3385]
> https://urldefense.com/v3/__https://github.com/apache/polaris/pull/3385__;!!Iz9xO38YGHZK!65BVFQDe_GtKbDW8LkNdzVS-WYewzWsSns5zFHKclbMU3vUy2UaTXLB5G-wql1yyqoTFyDc9Bt2W-eKl7A$
> > <
> https://urldefense.com/v3/__https://github.com/apache/polaris/pull/3385__;!!Iz9xO38YGHZK!_dbBfwEnZEuPp_8qIuBDoYfWRgCZKiPDMg9ouDltqsHhr9hqCHVlF9IedSkNVBDWupbneU7inC9YK-THL-lTjWkkyZhI0OE$
> >
> >
> >Thanks,
> > Dmitri.
> >
> > On Wed, Jan 14, 2026 at 10:05 PM Anand Kumar Sankaran via dev <
> > dev@polaris.apache.org<ma...@polaris.apache.org><mailto:
> dev@polaris.apache.org<ma...@polaris.apache.org>>> wrote:
> >
> > Hi Yufei,
> >
> > Gave commenting privileges to all.
> >
> > Thanks.
> >
> > —
> > Anand
> >
> > From: Yufei Gu <flyrain000@gmail.com<mailto:flyrain000@gmail.com
> ><ma...@gmail.com>>>
> > Date: Wednesday, January 14, 2026 at 6:13 PM
> > To: dev@polaris.apache.org<ma...@polaris.apache.org><mailto:
> dev@polaris.apache.org<ma...@polaris.apache.org>> <
> dev@polaris.apache.org<ma...@polaris.apache.org><mailto:
> dev@polaris.apache.org<ma...@polaris.apache.org>>>
> > Cc: Anand Kumar Sankaran <anand.sankaran@workday.com<mailto:
> anand.sankaran@workday.com><mailto:anand.sankaran@workday.com<mailto:
> anand.sankaran@workday.com>>>
> > Subject: Re: Polaris Telemetry and Audit Trail
> >
> > This Message Is From an External Sender
> > This message came from outside your organization.
> > Report Suspicious<
> >
> https://us-phishalarm-ewt.proofpoint.com/EWT/v1/Iz9xO38YGHZK!YhNDZABkHi1B6hyPVYUK0T-K5j4Aoqdrlu07UjBpJcpjOsz7Ie7d_DhPl-ywi50dfQz4O6LL-AHLCOQTQ0X7Le8yD2cF0ogquGFB8Aw8uQLgB0RL6Ezpa-pLDPA4ziXA$
> > >
> >
> > Hi Anand, thanks for the design doc and PR. Can you allow comments in the
> > doc so that people can chime in? Thanks!
> >
> > Yufei
> >
> >
> > On Sat, Jan 10, 2026 at 9:09 AM Anand Kumar Sankaran via dev <
> > dev@polaris.apache.org<ma...@polaris.apache.org><mailto:
> dev@polaris.apache.org<ma...@polaris.apache.org>><mailto:
> dev@polaris.apache.org<ma...@polaris.apache.org><mailto:
> dev@polaris.apache.org<ma...@polaris.apache.org>>>> wrote:
> > Hi all
> >
> > My first PR for adding AWS STS Session Tags support for credential
> vending
> > was merged.
> https://urldefense.com/v3/__https://github.com/apache/polaris/pull/3327__;!!Iz9xO38YGHZK!65BVFQDe_GtKbDW8LkNdzVS-WYewzWsSns5zFHKclbMU3vUy2UaTXLB5G-wql1yyqoTFyDc9Bt0YHY9sdA$
> > <
> https://urldefense.com/v3/__https://github.com/apache/polaris/pull/3327__;!!Iz9xO38YGHZK!_dbBfwEnZEuPp_8qIuBDoYfWRgCZKiPDMg9ouDltqsHhr9hqCHVlF9IedSkNVBDWupbneU7inC9YK-THL-lTjWkklJ4Xf9Y$
> >
> ><
> >
> https://urldefense.com/v3/__https://github.com/apache/polaris/pull/3327__;!!Iz9xO38YGHZK!8vlQ2_OqjmZNIpbrTTluFFNKFCEeM87nr95MP5pMpMu5fFWVSoBZ3eb6-C2xbAGCG-ry5eaCO-pFLxl3IZAJGqY$
> >>
> >
> > I have been working with Prashant on an end-to-end telemetry and audit
> > trail tracking for Polaris.  It is documented here.
> >
> https://urldefense.com/v3/__https://docs.google.com/document/d/1Ehzvi5RNPs4hChkBFI6VD23myEqm-7sWW3d2kjmuYj8/edit?tab=t.0__;!!Iz9xO38YGHZK!65BVFQDe_GtKbDW8LkNdzVS-WYewzWsSns5zFHKclbMU3vUy2UaTXLB5G-wql1yyqoTFyDc9Bt11Y1rbNQ$
> > <
> https://urldefense.com/v3/__https://docs.google.com/document/d/1Ehzvi5RNPs4hChkBFI6VD23myEqm-7sWW3d2kjmuYj8/edit?tab=t.0__;!!Iz9xO38YGHZK!_dbBfwEnZEuPp_8qIuBDoYfWRgCZKiPDMg9ouDltqsHhr9hqCHVlF9IedSkNVBDWupbneU7inC9YK-THL-lTjWkk8MVJPlY$
> >
> ><
> >
> https://urldefense.com/v3/__https://docs.google.com/document/d/1Ehzvi5RNPs4hChkBFI6VD23myEqm-7sWW3d2kjmuYj8/edit?tab=t.0__;!!Iz9xO38YGHZK!8vlQ2_OqjmZNIpbrTTluFFNKFCEeM87nr95MP5pMpMu5fFWVSoBZ3eb6-C2xbAGCG-ry5eaCO-pFLxl3hYqTlcg$
> >>
> >
> > Based on Prashant’s initial feedback (parity with Apache Gravitino for
> > metrics reports), I have an initial PR here.
> >
> https://urldefense.com/v3/__https://github.com/apache/polaris/pull/3385__;!!Iz9xO38YGHZK!65BVFQDe_GtKbDW8LkNdzVS-WYewzWsSns5zFHKclbMU3vUy2UaTXLB5G-wql1yyqoTFyDc9Bt2W-eKl7A$
> > <
> https://urldefense.com/v3/__https://github.com/apache/polaris/pull/3385__;!!Iz9xO38YGHZK!_dbBfwEnZEuPp_8qIuBDoYfWRgCZKiPDMg9ouDltqsHhr9hqCHVlF9IedSkNVBDWupbneU7inC9YK-THL-lTjWkkyZhI0OE$
> >
> ><
> >
> https://urldefense.com/v3/__https://github.com/apache/polaris/pull/3385__;!!Iz9xO38YGHZK!8vlQ2_OqjmZNIpbrTTluFFNKFCEeM87nr95MP5pMpMu5fFWVSoBZ3eb6-C2xbAGCG-ry5eaCO-pFLxl39opCznI$
> >>
> >
> > I will be reaching out to various folks for review. I am open to breaking
> > down the PR into smaller logical groups of PR if it helps.
> >
> > I request you to look at the Google doc and let me know what you think.
> >
> > —
> > Anand
> >
> >
> >
> > --
> > Dmitri Bourlatchkov
> > Senior Staff Software Engineer, Dremio
> > Dremio.com
> > <
> https://urldefense.com/v3/__https://www.dremio.com/?utm_medium=email&utm_source=signature&utm_term=na&utm_content=email-signature&utm_campaign=email-signature__;!!Iz9xO38YGHZK!_dbBfwEnZEuPp_8qIuBDoYfWRgCZKiPDMg9ouDltqsHhr9hqCHVlF9IedSkNVBDWupbneU7inC9YK-THL-lTjWkkiOwjJc0$
> >/
> > Follow Us on LinkedIn
> > <
> https://urldefense.com/v3/__https://www.linkedin.com/company/dremio__;!!Iz9xO38YGHZK!_dbBfwEnZEuPp_8qIuBDoYfWRgCZKiPDMg9ouDltqsHhr9hqCHVlF9IedSkNVBDWupbneU7inC9YK-THL-lTjWkkCLjBQ2M$
> >/
> > Get Started
> > <
> https://urldefense.com/v3/__https://www.dremio.com/get-started/__;!!Iz9xO38YGHZK!_dbBfwEnZEuPp_8qIuBDoYfWRgCZKiPDMg9ouDltqsHhr9hqCHVlF9IedSkNVBDWupbneU7inC9YK-THL-lTjWkkoKQUMII$
> >
> >
> >
> >The Agentic Lakehouse
> > *The only lakehouse built for agents, managed by agents*
> >
> >
> >
>
>

Re: Polaris Telemetry and Audit Trail

[Dmitri Bourlatchkov <di...@apache.org>]

Hi All,

Anand's PR [3385] has come a long way. In its current state it provides an
implementation for Metrics Persistence according to this discussion thread
and all GH comments have been addressed, AFAIK.

If anyone has concerns, please reply to this email or post a fresh GH
review.

All in all, I believe this PR is non-intrusive to other Polaris
functionality and can be merged now, so that the feature could be put to
practice by interested users (including Anand, I'm sure).

The feature is marked as "beta" so follow-up can and will continue on main.

[3385] https://github.com/apache/polaris/pull/3385

Cheers,
Dmitri.

On Thu, Feb 19, 2026 at 1:25 PM Dmitri Bourlatchkov <di...@apache.org>
wrote:

> Hi All,
>
> Heads up: I have reviewed and approved [3385], which completes the
> implementation of Anand's proposal.
>
> If there are no fresh comments / concerns, I propose merging it before EOD
> Feb 20.
>
> [3385] https://github.com/apache/polaris/pull/3385
>
> Cheers,
> Dmitri.
>
> On Tue, Feb 17, 2026 at 4:44 PM Anand Kumar Sankaran via dev <
> dev@polaris.apache.org> wrote:
>
>> Hi Dmitri,
>>
>> Thanks again.  I have rebased [3385] against main.
>>
>> —
>> Anand
>>
>> From: Dmitri Bourlatchkov <di...@apache.org>
>> Date: Tuesday, February 17, 2026 at 12:11 PM
>> To: dev@polaris.apache.org <de...@polaris.apache.org>
>> Cc: Anand Kumar Sankaran <an...@workday.com>
>> Subject: Re: Polaris Telemetry and Audit Trail
>>
>> This Message Is From an External Sender
>> This message came from outside your organization.
>> Report Suspicious<
>> https://us-phishalarm-ewt.proofpoint.com/EWT/v1/Iz9xO38YGHZK!YhNDZAGr2cumYdvL-GeGN_sUtpf-vTAsw9ogTCTPGKCcS5ePHJr_UGiyUqgDQ5VNCl6xTwYj4qqUWwkvyuTh43tJ_AVdZJPgCLZDpZXbpB6L3jCaJL5OVoyRsNvKnAwL$
>> >
>>
>> Hi Anand,
>>
>> PR [3523] (SQL schema) has been merged.
>>
>> Please rebase [3385].
>>
>> [3523] https://github.com/apache/polaris/pull/3523<
>> https://urldefense.com/v3/__https://github.com/apache/polaris/pull/3523__;!!Iz9xO38YGHZK!4sLrfIU8HUaeJshSZbprgWTwyaWUNAT-9Yof_YpVYP9T1Har02LkclV0s0wrDVe2-1tjesf_EfwLg5VWow$
>> >
>>
>> [3385] https://github.com/apache/polaris/pull/3385<
>> https://urldefense.com/v3/__https://github.com/apache/polaris/pull/3385__;!!Iz9xO38YGHZK!4sLrfIU8HUaeJshSZbprgWTwyaWUNAT-9Yof_YpVYP9T1Har02LkclV0s0wrDVe2-1tjesf_Efwsf7PUHQ$
>> >
>>
>> Thanks,
>> Dmitri.
>>
>> On Fri, Feb 13, 2026 at 6:10 PM Dmitri Bourlatchkov <dimas@apache.org
>> <ma...@apache.org>> wrote:
>> Hi All,
>>
>> The SPI PR [3616] has been merged.
>>
>> I propose to merge the Schema PR [3523] on Mon if there are no objections
>> or fresh comments.
>>
>> [3616] https://github.com/apache/polaris/pull/3616<
>> https://urldefense.com/v3/__https://github.com/apache/polaris/pull/3616__;!!Iz9xO38YGHZK!4sLrfIU8HUaeJshSZbprgWTwyaWUNAT-9Yof_YpVYP9T1Har02LkclV0s0wrDVe2-1tjesf_EfzzvX9eZQ$
>> >
>>
>> [3523] https://github.com/apache/polaris/pull/3523<
>> https://urldefense.com/v3/__https://github.com/apache/polaris/pull/3523__;!!Iz9xO38YGHZK!4sLrfIU8HUaeJshSZbprgWTwyaWUNAT-9Yof_YpVYP9T1Har02LkclV0s0wrDVe2-1tjesf_EfwLg5VWow$
>> >
>>
>> Cheers,
>> Dmitri.
>>
>>
>> On Thu, Feb 12, 2026 at 6:00 PM Dmitri Bourlatchkov <dimas@apache.org
>> <ma...@apache.org>> wrote:
>> Hi All,
>>
>> The SPI PR [3616] for Scan/Commit metrics has been in review for 2 weeks
>> including 3 days in an approved state.
>>
>> Unless there are any last minute comments, I propose to merge it tomorrow
>> (Feb 13).
>>
>> [3616] https://github.com/apache/polaris/pull/3616<
>> https://urldefense.com/v3/__https://github.com/apache/polaris/pull/3616__;!!Iz9xO38YGHZK!4sLrfIU8HUaeJshSZbprgWTwyaWUNAT-9Yof_YpVYP9T1Har02LkclV0s0wrDVe2-1tjesf_EfzzvX9eZQ$
>> >
>>
>> Cheers,
>> Dmitri.
>>
>> On Mon, Feb 9, 2026 at 2:08 PM Anand Kumar Sankaran via dev <
>> dev@polaris.apache.org<ma...@polaris.apache.org>> wrote:
>> Dmitri and others,
>>
>> Thanks for all the review feedback and being patient while we iterated.
>> My initial proposal was to persist the table metrics as events (least
>> amount of work 😊), we have come far from that proposal.
>>
>> To detach the SPI fully, there are a few things to do
>> (loadSchemaVersion() and shared DatasourceOperations). We can add a
>> separate datasource config to specify a different connection pool, perhaps.
>> Can be done as a follow up PR.
>>
>> —
>> Anand
>>
>> From: Dmitri Bourlatchkov <di...@apache.org>>
>> Date: Monday, February 9, 2026 at 10:52 AM
>> To: dev@polaris.apache.org<ma...@polaris.apache.org> <
>> dev@polaris.apache.org<ma...@polaris.apache.org>>
>> Cc: Anand Kumar Sankaran <anand.sankaran@workday.com<mailto:
>> anand.sankaran@workday.com>>
>> Subject: Re: Polaris Telemetry and Audit Trail
>>
>> This Message Is From an External Sender
>> This message came from outside your organization.
>> Report Suspicious<
>> https://us-phishalarm-ewt.proofpoint.com/EWT/v1/Iz9xO38YGHZK!YhNDZAGr2cumYduJ20ONMhMxljYPF8uZOe6R8BpEWyJ7tSG5GUUf8our05nYHWov0t1Vi7Rx_O8JuRNBBQQ57aBoctrR0wH7VSR9VxRuqYJPpz0q6I0JboPRfdLIwqal$
>> >
>>
>> Hi Anand and all,
>>
>> Thanks for pushing this feature forward! From my POV the SPI [3616] and
>> the schema [3523] PRs are good to merge as the initial implementation of
>> this feature (annotated as "beta"). The way is open to the JDBC impl. PR
>> [3385] (which will probably need a rebase after merging the SPI and schema
>> code).
>>
>> I'm sending this email to highlight the direction taken with respect to
>> the SQL schema in this feature, as it might have been visible enough to the
>> broader community during the PR reviews.
>>
>> * The scan / commit metrics schema is managed / versioned separately from
>> the JDBC MetaStore schema
>>
>> * The Metrics Persistence SPI is detached (mostly at this time, but
>> supposedly fully detached later) from the JDBC MetaStore
>>
>> * The JDBC DataSource for metrics may be the same or different from the
>> MetaStore datasource.
>>
>> * Metrics Persistence may have non-JDBC implementation in the future (not
>> implemented ATM)
>>
>> * JDBC Metrics Persistence may co-exist with other MetaStore
>> implementations.
>>
>> I believe this design allows great flexibility for end users to try this
>> feature out and keeps options open for future enhancements in Scan / Commit
>> Metrics in Polaris.
>>
>> [3616] https://github.com/apache/polaris/pull/3616<
>> https://urldefense.com/v3/__https://github.com/apache/polaris/pull/3616__;!!Iz9xO38YGHZK!4sLrfIU8HUaeJshSZbprgWTwyaWUNAT-9Yof_YpVYP9T1Har02LkclV0s0wrDVe2-1tjesf_EfzzvX9eZQ$
>> ><
>> https://urldefense.com/v3/__https://github.com/apache/polaris/pull/3616__;!!Iz9xO38YGHZK!7IwwfPXDN0ThhNZgol-n9Ww3FFLKcpZbSQ17lWEtcQnTHtJM9aVXnqGW99_6vmVkWACy7rYC8XEpN6Okdg$
>> >
>>
>> [3523] https://github.com/apache/polaris/pull/3523<
>> https://urldefense.com/v3/__https://github.com/apache/polaris/pull/3523__;!!Iz9xO38YGHZK!4sLrfIU8HUaeJshSZbprgWTwyaWUNAT-9Yof_YpVYP9T1Har02LkclV0s0wrDVe2-1tjesf_EfwLg5VWow$
>> ><
>> https://urldefense.com/v3/__https://github.com/apache/polaris/pull/3523__;!!Iz9xO38YGHZK!7IwwfPXDN0ThhNZgol-n9Ww3FFLKcpZbSQ17lWEtcQnTHtJM9aVXnqGW99_6vmVkWACy7rYC8XGOCHKhSg$
>> >
>>
>> [3385] https://github.com/apache/polaris/pull/3385<
>> https://urldefense.com/v3/__https://github.com/apache/polaris/pull/3385__;!!Iz9xO38YGHZK!4sLrfIU8HUaeJshSZbprgWTwyaWUNAT-9Yof_YpVYP9T1Har02LkclV0s0wrDVe2-1tjesf_Efwsf7PUHQ$
>> ><
>> https://urldefense.com/v3/__https://github.com/apache/polaris/pull/3385__;!!Iz9xO38YGHZK!7IwwfPXDN0ThhNZgol-n9Ww3FFLKcpZbSQ17lWEtcQnTHtJM9aVXnqGW99_6vmVkWACy7rYC8XFlxLMhvQ$
>> >
>>
>> Thoughts / comments?
>>
>> Cheers,
>> Dmitri.
>>
>>
>> On Thu, Feb 5, 2026 at 6:50 PM Anand Kumar Sankaran via dev <
>> dev@polaris.apache.org<ma...@polaris.apache.org><mailto:
>> dev@polaris.apache.org<ma...@polaris.apache.org>>> wrote:
>> Hi Dmitri,
>>
>> Following up on your feedback regarding the metrics schema separation
>> approach. I've implemented the simplified design you suggested in
>> https://github.com/apache/polaris/pull/3523<
>> https://urldefense.com/v3/__https://github.com/apache/polaris/pull/3523__;!!Iz9xO38YGHZK!4sLrfIU8HUaeJshSZbprgWTwyaWUNAT-9Yof_YpVYP9T1Har02LkclV0s0wrDVe2-1tjesf_EfwLg5VWow$
>> ><
>> https://urldefense.com/v3/__https://github.com/apache/polaris/pull/3523__;!!Iz9xO38YGHZK!7IwwfPXDN0ThhNZgol-n9Ww3FFLKcpZbSQ17lWEtcQnTHtJM9aVXnqGW99_6vmVkWACy7rYC8XGOCHKhSg$
>> >:
>>
>>
>>  The metrics schema can now evolve independently from the entity schema.
>> Let me know if you'd like any changes to this approach.
>>
>>  Thanks,
>>   Anand
>>
>>
>>
>> From: Dmitri Bourlatchkov <dimas@apache.org<mailto:dimas@apache.org
>> ><ma...@apache.org>>>
>> Date: Thursday, February 5, 2026 at 2:57 PM
>> To: dev@polaris.apache.org<ma...@polaris.apache.org><mailto:
>> dev@polaris.apache.org<ma...@polaris.apache.org>> <
>> dev@polaris.apache.org<ma...@polaris.apache.org><mailto:
>> dev@polaris.apache.org<ma...@polaris.apache.org>>>
>> Cc: Anand Kumar Sankaran <anand.sankaran@workday.com<mailto:
>> anand.sankaran@workday.com><mailto:anand.sankaran@workday.com<mailto:
>> anand.sankaran@workday.com>>>
>> Subject: Re: Polaris Telemetry and Audit Trail
>>
>> This Message Is From an External Sender
>> This message came from outside your organization.
>> Report Suspicious<
>> https://us-phishalarm-ewt.proofpoint.com/EWT/v1/Iz9xO38YGHZK!YhNDZAGr2cumY1iAus5GNbkIxSXoQaI552sXgYyv7Z9Wrvb7qeTskSy5WkXTVDJ_Bel2AbNh_fQVn-HS_UtOSdOtpsNfzhXDkK4fYWdTeoeBe70A6oQVGpAYT3SrbEvs$
>> >
>>
>> Hi Anand,
>>
>> Re: MetricsPersistence Factory - TBH I do not think it is required. If we
>> go with an MetricsPersistence impl. independent of the JDBC MetaStore, it
>> could be injected into PersistingMetricsReporter directly by CDI.
>>
>> Re: DataSource - for a start we could probably reuse the same DataSource
>> as the JDBS MetaStore uses. In principle, we could introduce a separate
>> DataSource, but I do not think we need to complicate the initial
>> implementation with that. It can be added later, if required.
>>
>> Re: schema / bootstrap - I'm thinking, indeed, of having a separate *.sql
>> file for metrics DDL. Then we could add an option to the bootstrap command
>> in the Admin Tool, or even a new command for creating / updating the
>> metrics schema. This might require some refactoring, but I believe the
>> bootstrap code needs some cleanup anyway.
>>
>> The general idea is to allow the metrics schema to evolve independently.
>> It's a new and actively developed feature, so having flexibility is going
>> to be good for making progress / improvements, I hope.
>>
>> Does that sound reasonable?
>>
>> Thanks,
>> Dmitri.
>>
>> On Thu, Feb 5, 2026 at 3:47 PM Anand Kumar Sankaran via dev <
>> dev@polaris.apache.org<ma...@polaris.apache.org><mailto:
>> dev@polaris.apache.org<ma...@polaris.apache.org>><mailto:
>> dev@polaris.apache.org<ma...@polaris.apache.org><mailto:
>> dev@polaris.apache.org<ma...@polaris.apache.org>>>> wrote:
>> Hi Dmitri
>>
>> The SPI is designed to be pluggable via
>> MetaStoreManagerFactory.getOrCreateMetricsPersistence()<
>> https://github.com/apache/polaris/pull/3385/changes#diff-0e34667d083d30cace067936b87c736201e5a8cb001aa0b12e203ee52cddae99R51
>> <
>> https://urldefense.com/v3/__https://github.com/apache/polaris/pull/3385/changes*diff-0e34667d083d30cace067936b87c736201e5a8cb001aa0b12e203ee52cddae99R51__;Iw!!Iz9xO38YGHZK!4sLrfIU8HUaeJshSZbprgWTwyaWUNAT-9Yof_YpVYP9T1Har02LkclV0s0wrDVe2-1tjesf_Efx9xKkRig$
>> ><
>> https://urldefense.com/v3/__https://github.com/apache/polaris/pull/3385/changes*diff-0e34667d083d30cace067936b87c736201e5a8cb001aa0b12e203ee52cddae99R51__;Iw!!Iz9xO38YGHZK!7IwwfPXDN0ThhNZgol-n9Ww3FFLKcpZbSQ17lWEtcQnTHtJM9aVXnqGW99_6vmVkWACy7rYC8XFxRr4qRw$
>> ><
>> https://urldefense.com/v3/__https://github.com/apache/polaris/pull/3385/changes*diff-0e34667d083d30cace067936b87c736201e5a8cb001aa0b12e203ee52cddae99R51__;Iw!!Iz9xO38YGHZK!_K1mcH0_2x0nbU-mZlbBi3tk9QAGjB6VInPiqkQU21mpIP_ft2bXXyIiADNoAcotNr9AKqlphh65taUhcw$>>.
>> The default returns MetricsPersistence.NOOP, and only JDBC with schema v4+
>> returns a functional implementation.
>>
>>  Your proposal makes sense to me. To allow NoSQL Entity Persistence +
>> JDBC Metrics Persistence would require:
>>
>>      1. Separate metrics schema files (e.g., metrics-schema-v1.sql) that
>> can be bootstrapped independently
>>      2. Separate datasource configuration for metrics (e.g.,
>> polaris.metrics.jdbc.*)
>>      3. A dedicated `MetricsPersistenceFactory` interface (separate from
>> MetaStoreManagerFactory)
>>      4. Updates to the Admin Tool bootstrap workflow to handle both
>> schemas
>>
>> Does that sound reasonable?  If we do this, all this work should go into
>> https://github.com/apache/polaris/pull/3523<
>> https://urldefense.com/v3/__https://github.com/apache/polaris/pull/3523__;!!Iz9xO38YGHZK!4sLrfIU8HUaeJshSZbprgWTwyaWUNAT-9Yof_YpVYP9T1Har02LkclV0s0wrDVe2-1tjesf_EfwLg5VWow$
>> ><
>> https://urldefense.com/v3/__https://github.com/apache/polaris/pull/3523__;!!Iz9xO38YGHZK!7IwwfPXDN0ThhNZgol-n9Ww3FFLKcpZbSQ17lWEtcQnTHtJM9aVXnqGW99_6vmVkWACy7rYC8XGOCHKhSg$
>> ><
>> https://urldefense.com/v3/__https://github.com/apache/polaris/pull/3523__;!!Iz9xO38YGHZK!_K1mcH0_2x0nbU-mZlbBi3tk9QAGjB6VInPiqkQU21mpIP_ft2bXXyIiADNoAcotNr9AKqlphh4ZISZcDQ$>
>> as one atomic commit, correct?
>>
>> —
>> Anand
>>
>>
>>
>> From: Dmitri Bourlatchkov <dimas@apache.org<mailto:dimas@apache.org
>> ><ma...@apache.org>><mailto:
>> dimas@apache.org<ma...@apache.org><mailto:dimas@apache.org<mailto:
>> dimas@apache.org>>>>
>> Date: Thursday, February 5, 2026 at 8:28 AM
>> To: Anand Kumar Sankaran <anand.sankaran@workday.com<mailto:
>> anand.sankaran@workday.com><mailto:anand.sankaran@workday.com<mailto:
>> anand.sankaran@workday.com>><mailto:anand.sankaran@workday.com<mailto:
>> anand.sankaran@workday.com><mailto:anand.sankaran@workday.com<mailto:
>> anand.sankaran@workday.com>>>>
>> Cc: dev@polaris.apache.org<ma...@polaris.apache.org><mailto:
>> dev@polaris.apache.org<ma...@polaris.apache.org>><mailto:
>> dev@polaris.apache.org<ma...@polaris.apache.org><mailto:
>> dev@polaris.apache.org<ma...@polaris.apache.org>>> <
>> dev@polaris.apache.org<ma...@polaris.apache.org><mailto:
>> dev@polaris.apache.org<ma...@polaris.apache.org>><mailto:
>> dev@polaris.apache.org<ma...@polaris.apache.org><mailto:
>> dev@polaris.apache.org<ma...@polaris.apache.org>>>>
>> Subject: Re: Polaris Telemetry and Audit Trail
>>
>> This Message Is From an External Sender
>> This message came from outside your organization.
>> Report Suspicious<
>> https://us-phishalarm-ewt.proofpoint.com/EWT/v1/Iz9xO38YGHZK!YhNDZAGr2cumbRcPlIOttKwvGr4f38TF-J2-LmOw8H1KsFpWaM3W5g28Oq5qW2gnw2-WHz0rsWhm7F7hSGykA9l01LKYGuCcCkexkHp1sUssM_IegCtjk_id9s891orv$
>> >
>>
>>
>> Hi Anand,
>>
>> Thanks for driving the SPI definition for scan/commits metrics.
>>
>> I did not think from this angle before, but now I suppose it might be
>> worth
>> introducing a separate SQL schema for metrics, detached from the main JDBC
>> Persistence schema.
>>
>> My rationale is that the SPI may be exercised independently of the JDBC
>> Entity Persistence. I can imagine people using NoSQL Entity Persistence
>> plus JDBC Scan Metrics Persistence in the same deployment.
>>
>> Using a separate schema will add further work on the Bootstrap workflow
>> (Admin Tool), but I hope the end result is going to be more convenient and
>> versatile for end users.
>>
>> WDYT?
>>
>> Thanks,
>> Dmitri.
>>
>> On Thu, Jan 29, 2026 at 1:49 PM Anand Kumar Sankaran <
>> anand.sankaran@workday.com<ma...@workday.com><mailto:
>> anand.sankaran@workday.com<ma...@workday.com>><mailto:
>> anand.sankaran@workday.com<ma...@workday.com><mailto:
>> anand.sankaran@workday.com<ma...@workday.com>>>> wrote:
>>
>> > Hi Dmitri,
>> >
>> > Thank you for taking the time to review. I had pulled out the schema in
>> > its own PR (based on comments) here:
>> >
>> https://urldefense.com/v3/__https://github.com/apache/polaris/pull/3523__;!!Iz9xO38YGHZK!65BVFQDe_GtKbDW8LkNdzVS-WYewzWsSns5zFHKclbMU3vUy2UaTXLB5G-wql1yyqoTFyDc9Bt3GRsh6AA$
>> >
>> > I will update the document with the SPI and carve out another PR that
>> > focuses on the SPI.
>> >
>> > —
>> > Anand
>> >
>> > *From: *Dmitri Bourlatchkov <dmitri.bourlatchkov@dremio.com<mailto:
>> dmitri.bourlatchkov@dremio.com><mailto:dmitri.bourlatchkov@dremio.com
>> <ma...@dremio.com>><mailto:
>> dmitri.bourlatchkov@dremio.com<mailto:dmitri.bourlatchkov@dremio.com
>> ><mailto:dmitri.bourlatchkov@dremio.com<mailto:
>> dmitri.bourlatchkov@dremio.com>>>>
>> > *Date: *Thursday, January 29, 2026 at 10:43 AM
>> > *To: *dev@polaris.apache.org<ma...@polaris.apache.org><mailto:
>> dev@polaris.apache.org<ma...@polaris.apache.org>><mailto:
>> dev@polaris.apache.org<ma...@polaris.apache.org><mailto:
>> dev@polaris.apache.org<ma...@polaris.apache.org>>> <
>> dev@polaris.apache.org<ma...@polaris.apache.org><mailto:
>> dev@polaris.apache.org<ma...@polaris.apache.org>><mailto:
>> dev@polaris.apache.org<ma...@polaris.apache.org><mailto:
>> dev@polaris.apache.org<ma...@polaris.apache.org>>>>
>> > *Cc: *Anand Kumar Sankaran <anand.sankaran@workday.com<mailto:
>> anand.sankaran@workday.com><mailto:anand.sankaran@workday.com<mailto:
>> anand.sankaran@workday.com>><mailto:anand.sankaran@workday.com<mailto:
>> anand.sankaran@workday.com><mailto:anand.sankaran@workday.com<mailto:
>> anand.sankaran@workday.com>>>>
>> > *Subject: *Re: Polaris Telemetry and Audit Trail
>> >
>> > This Message Is From an External Sender
>> > This message came from outside your organization.
>> > Report Suspicious
>> > <
>> https://us-phishalarm-ewt.proofpoint.com/EWT/v1/Iz9xO38YGHZK!YhNDZAGr2cumY1cAlUeoPrU2eNUPz9Wxq2PaG1t99vEN3v60gucxC3Qsk-xK09RndcW46WtNjMMjQZbaQ495h1D1g293F_uqLhPM8gDMlA6xRmo5b5uqrSh2v7QKHrhQ$
>> >
>> >
>> > Hi Anand,
>> >
>> > Thanks for making a proposal doc and starting this thread. Apologies for
>> > late comments.
>> >
>> > As you know I already reviewed a couple of related PRs, which had a
>> > smaller impact on the codebase, but now, I believe, we're coming to the
>> > core of this feature and it might need a deeper discussion.
>> >
>> > For the sake of awareness of other project members, I'd like to
>> > highlight some aspects by email.
>> >
>> > * Storing scan metrics in RDBMS in general looks like a
>> > reasonable approach to me. However, Polaris Persistence is an extensible
>> > mechanism. Other backends are possible, including privately developed
>> > backends.
>> >
>> > From this POV, I believe it might be preferable to start not with an
>> RDBMS
>> > schema, but with a java SPI outlining expected read/write operations and
>> > the data model. This will make it easier to assess impact on other
>> > Persistence implementations. Naturally, Polaris service code will have
>> to
>> > be able to work seamlessly regardless of the backend impl. (including
>> the
>> > do-nothing impl.).
>> >
>> > I see that some connection between the SQL schema and java code is made
>> in
>> > PR [3385], but it does not appear to offer a clean SPI that could be
>> > implemented by different Persistence backends. Would it be ok from your
>> > perspective to make another PR with just SPI code?
>> >
>> > * Regarding ad-hoc SQL queries - it is certainly a valid use case for
>> > custom code to query the Polaris database directly. However, those
>> ad-hoc
>> > queries may require certain indexes, which may not have a relevant use
>> case
>> > in OSS code. In that case, I believe it would be preferable to manage
>> the
>> > extra indexes in custom code (not in Polaris). Whether this is the case
>> or
>> > not currently, I cannot say with certainty, so apologies if this is a
>> false
>> > alarm... Having the java SPI available would be a huge aid to reasoning
>> > about the proposed SQL schema, I think.
>> >
>> > That said, I do not mean to complicate the development of this feature,
>> > just trying to structure it in a way that is hopefully easy to
>> understand
>> > (including myself), maintain and expand later. If I missed something,
>> > please feel free to point it out.
>> >
>> > [3385]
>> https://urldefense.com/v3/__https://github.com/apache/polaris/pull/3385__;!!Iz9xO38YGHZK!65BVFQDe_GtKbDW8LkNdzVS-WYewzWsSns5zFHKclbMU3vUy2UaTXLB5G-wql1yyqoTFyDc9Bt2W-eKl7A$
>> > <
>> https://urldefense.com/v3/__https://github.com/apache/polaris/pull/3385__;!!Iz9xO38YGHZK!_dbBfwEnZEuPp_8qIuBDoYfWRgCZKiPDMg9ouDltqsHhr9hqCHVlF9IedSkNVBDWupbneU7inC9YK-THL-lTjWkkyZhI0OE$
>> >
>> >
>> >Thanks,
>> > Dmitri.
>> >
>> > On Wed, Jan 14, 2026 at 10:05 PM Anand Kumar Sankaran via dev <
>> > dev@polaris.apache.org<ma...@polaris.apache.org><mailto:
>> dev@polaris.apache.org<ma...@polaris.apache.org>><mailto:
>> dev@polaris.apache.org<ma...@polaris.apache.org><mailto:
>> dev@polaris.apache.org<ma...@polaris.apache.org>>>> wrote:
>> >
>> > Hi Yufei,
>> >
>> > Gave commenting privileges to all.
>> >
>> > Thanks.
>> >
>> > —
>> > Anand
>> >
>> > From: Yufei Gu <flyrain000@gmail.com<mailto:flyrain000@gmail.com
>> ><ma...@gmail.com>><mailto:
>> flyrain000@gmail.com<ma...@gmail.com><mailto:
>> flyrain000@gmail.com<ma...@gmail.com>>>>
>> > Date: Wednesday, January 14, 2026 at 6:13 PM
>> > To: dev@polaris.apache.org<ma...@polaris.apache.org><mailto:
>> dev@polaris.apache.org<ma...@polaris.apache.org>><mailto:
>> dev@polaris.apache.org<ma...@polaris.apache.org><mailto:
>> dev@polaris.apache.org<ma...@polaris.apache.org>>> <
>> dev@polaris.apache.org<ma...@polaris.apache.org><mailto:
>> dev@polaris.apache.org<ma...@polaris.apache.org>><mailto:
>> dev@polaris.apache.org<ma...@polaris.apache.org><mailto:
>> dev@polaris.apache.org<ma...@polaris.apache.org>>>>
>> > Cc: Anand Kumar Sankaran <anand.sankaran@workday.com<mailto:
>> anand.sankaran@workday.com><mailto:anand.sankaran@workday.com<mailto:
>> anand.sankaran@workday.com>><mailto:anand.sankaran@workday.com<mailto:
>> anand.sankaran@workday.com><mailto:anand.sankaran@workday.com<mailto:
>> anand.sankaran@workday.com>>>>
>> > Subject: Re: Polaris Telemetry and Audit Trail
>> >
>> > This Message Is From an External Sender
>> > This message came from outside your organization.
>> > Report Suspicious<
>> >
>> https://us-phishalarm-ewt.proofpoint.com/EWT/v1/Iz9xO38YGHZK!YhNDZABkHi1B6hyPVYUK0T-K5j4Aoqdrlu07UjBpJcpjOsz7Ie7d_DhPl-ywi50dfQz4O6LL-AHLCOQTQ0X7Le8yD2cF0ogquGFB8Aw8uQLgB0RL6Ezpa-pLDPA4ziXA$
>> > >
>> >
>> > Hi Anand, thanks for the design doc and PR. Can you allow comments in
>> the
>> > doc so that people can chime in? Thanks!
>> >
>> > Yufei
>> >
>> >
>> > On Sat, Jan 10, 2026 at 9:09 AM Anand Kumar Sankaran via dev <
>> > dev@polaris.apache.org<ma...@polaris.apache.org><mailto:
>> dev@polaris.apache.org<ma...@polaris.apache.org>><mailto:
>> dev@polaris.apache.org<ma...@polaris.apache.org><mailto:
>> dev@polaris.apache.org<ma...@polaris.apache.org>>><mailto:
>> dev@polaris.apache.org<ma...@polaris.apache.org><mailto:
>> dev@polaris.apache.org<ma...@polaris.apache.org>><mailto:
>> dev@polaris.apache.org<ma...@polaris.apache.org><mailto:
>> dev@polaris.apache.org<ma...@polaris.apache.org>>>>> wrote:
>> > Hi all
>> >
>> > My first PR for adding AWS STS Session Tags support for credential
>> vending
>> > was merged.
>> https://urldefense.com/v3/__https://github.com/apache/polaris/pull/3327__;!!Iz9xO38YGHZK!65BVFQDe_GtKbDW8LkNdzVS-WYewzWsSns5zFHKclbMU3vUy2UaTXLB5G-wql1yyqoTFyDc9Bt0YHY9sdA$
>> > <
>> https://urldefense.com/v3/__https://github.com/apache/polaris/pull/3327__;!!Iz9xO38YGHZK!_dbBfwEnZEuPp_8qIuBDoYfWRgCZKiPDMg9ouDltqsHhr9hqCHVlF9IedSkNVBDWupbneU7inC9YK-THL-lTjWkklJ4Xf9Y$
>> >
>> ><
>> >
>> https://urldefense.com/v3/__https://github.com/apache/polaris/pull/3327__;!!Iz9xO38YGHZK!8vlQ2_OqjmZNIpbrTTluFFNKFCEeM87nr95MP5pMpMu5fFWVSoBZ3eb6-C2xbAGCG-ry5eaCO-pFLxl3IZAJGqY$
>> >>
>> >
>> > I have been working with Prashant on an end-to-end telemetry and audit
>> > trail tracking for Polaris.  It is documented here.
>> >
>> https://urldefense.com/v3/__https://docs.google.com/document/d/1Ehzvi5RNPs4hChkBFI6VD23myEqm-7sWW3d2kjmuYj8/edit?tab=t.0__;!!Iz9xO38YGHZK!65BVFQDe_GtKbDW8LkNdzVS-WYewzWsSns5zFHKclbMU3vUy2UaTXLB5G-wql1yyqoTFyDc9Bt11Y1rbNQ$
>> > <
>> https://urldefense.com/v3/__https://docs.google.com/document/d/1Ehzvi5RNPs4hChkBFI6VD23myEqm-7sWW3d2kjmuYj8/edit?tab=t.0__;!!Iz9xO38YGHZK!_dbBfwEnZEuPp_8qIuBDoYfWRgCZKiPDMg9ouDltqsHhr9hqCHVlF9IedSkNVBDWupbneU7inC9YK-THL-lTjWkk8MVJPlY$
>> >
>> ><
>> >
>> https://urldefense.com/v3/__https://docs.google.com/document/d/1Ehzvi5RNPs4hChkBFI6VD23myEqm-7sWW3d2kjmuYj8/edit?tab=t.0__;!!Iz9xO38YGHZK!8vlQ2_OqjmZNIpbrTTluFFNKFCEeM87nr95MP5pMpMu5fFWVSoBZ3eb6-C2xbAGCG-ry5eaCO-pFLxl3hYqTlcg$
>> >>
>> >
>> > Based on Prashant’s initial feedback (parity with Apache Gravitino for
>> > metrics reports), I have an initial PR here.
>> >
>> https://urldefense.com/v3/__https://github.com/apache/polaris/pull/3385__;!!Iz9xO38YGHZK!65BVFQDe_GtKbDW8LkNdzVS-WYewzWsSns5zFHKclbMU3vUy2UaTXLB5G-wql1yyqoTFyDc9Bt2W-eKl7A$
>> > <
>> https://urldefense.com/v3/__https://github.com/apache/polaris/pull/3385__;!!Iz9xO38YGHZK!_dbBfwEnZEuPp_8qIuBDoYfWRgCZKiPDMg9ouDltqsHhr9hqCHVlF9IedSkNVBDWupbneU7inC9YK-THL-lTjWkkyZhI0OE$
>> >
>> ><
>> >
>> https://urldefense.com/v3/__https://github.com/apache/polaris/pull/3385__;!!Iz9xO38YGHZK!8vlQ2_OqjmZNIpbrTTluFFNKFCEeM87nr95MP5pMpMu5fFWVSoBZ3eb6-C2xbAGCG-ry5eaCO-pFLxl39opCznI$
>> >>
>> >
>> > I will be reaching out to various folks for review. I am open to
>> breaking
>> > down the PR into smaller logical groups of PR if it helps.
>> >
>> > I request you to look at the Google doc and let me know what you think.
>> >
>> > —
>> > Anand
>> >
>> >
>> >
>> > --
>> > Dmitri Bourlatchkov
>> > Senior Staff Software Engineer, Dremio
>> > Dremio.com
>> > <
>> https://urldefense.com/v3/__https://www.dremio.com/?utm_medium=email&utm_source=signature&utm_term=na&utm_content=email-signature&utm_campaign=email-signature__;!!Iz9xO38YGHZK!_dbBfwEnZEuPp_8qIuBDoYfWRgCZKiPDMg9ouDltqsHhr9hqCHVlF9IedSkNVBDWupbneU7inC9YK-THL-lTjWkkiOwjJc0$
>> >/
>> > Follow Us on LinkedIn
>> > <
>> https://urldefense.com/v3/__https://www.linkedin.com/company/dremio__;!!Iz9xO38YGHZK!_dbBfwEnZEuPp_8qIuBDoYfWRgCZKiPDMg9ouDltqsHhr9hqCHVlF9IedSkNVBDWupbneU7inC9YK-THL-lTjWkkCLjBQ2M$
>> >/
>> > Get Started
>> > <
>> https://urldefense.com/v3/__https://www.dremio.com/get-started/__;!!Iz9xO38YGHZK!_dbBfwEnZEuPp_8qIuBDoYfWRgCZKiPDMg9ouDltqsHhr9hqCHVlF9IedSkNVBDWupbneU7inC9YK-THL-lTjWkkoKQUMII$
>> >
>> >
>> >
>> >The Agentic Lakehouse
>> > *The only lakehouse built for agents, managed by agents*
>> >
>> >
>> >
>>
>>

Re: Polaris Telemetry and Audit Trail

[Dmitri Bourlatchkov <di...@apache.org>]

Hi All,

Heads up: I have reviewed and approved [3385], which completes the
implementation of Anand's proposal.

If there are no fresh comments / concerns, I propose merging it before EOD
Feb 20.

[3385] https://github.com/apache/polaris/pull/3385

Cheers,
Dmitri.

On Tue, Feb 17, 2026 at 4:44 PM Anand Kumar Sankaran via dev <
dev@polaris.apache.org> wrote:

> Hi Dmitri,
>
> Thanks again.  I have rebased [3385] against main.
>
> —
> Anand
>
> From: Dmitri Bourlatchkov <di...@apache.org>
> Date: Tuesday, February 17, 2026 at 12:11 PM
> To: dev@polaris.apache.org <de...@polaris.apache.org>
> Cc: Anand Kumar Sankaran <an...@workday.com>
> Subject: Re: Polaris Telemetry and Audit Trail
>
> This Message Is From an External Sender
> This message came from outside your organization.
> Report Suspicious<
> https://us-phishalarm-ewt.proofpoint.com/EWT/v1/Iz9xO38YGHZK!YhNDZAGr2cumYdvL-GeGN_sUtpf-vTAsw9ogTCTPGKCcS5ePHJr_UGiyUqgDQ5VNCl6xTwYj4qqUWwkvyuTh43tJ_AVdZJPgCLZDpZXbpB6L3jCaJL5OVoyRsNvKnAwL$
> >
>
> Hi Anand,
>
> PR [3523] (SQL schema) has been merged.
>
> Please rebase [3385].
>
> [3523] https://github.com/apache/polaris/pull/3523<
> https://urldefense.com/v3/__https://github.com/apache/polaris/pull/3523__;!!Iz9xO38YGHZK!4sLrfIU8HUaeJshSZbprgWTwyaWUNAT-9Yof_YpVYP9T1Har02LkclV0s0wrDVe2-1tjesf_EfwLg5VWow$
> >
>
> [3385] https://github.com/apache/polaris/pull/3385<
> https://urldefense.com/v3/__https://github.com/apache/polaris/pull/3385__;!!Iz9xO38YGHZK!4sLrfIU8HUaeJshSZbprgWTwyaWUNAT-9Yof_YpVYP9T1Har02LkclV0s0wrDVe2-1tjesf_Efwsf7PUHQ$
> >
>
> Thanks,
> Dmitri.
>
> On Fri, Feb 13, 2026 at 6:10 PM Dmitri Bourlatchkov <dimas@apache.org
> <ma...@apache.org>> wrote:
> Hi All,
>
> The SPI PR [3616] has been merged.
>
> I propose to merge the Schema PR [3523] on Mon if there are no objections
> or fresh comments.
>
> [3616] https://github.com/apache/polaris/pull/3616<
> https://urldefense.com/v3/__https://github.com/apache/polaris/pull/3616__;!!Iz9xO38YGHZK!4sLrfIU8HUaeJshSZbprgWTwyaWUNAT-9Yof_YpVYP9T1Har02LkclV0s0wrDVe2-1tjesf_EfzzvX9eZQ$
> >
>
> [3523] https://github.com/apache/polaris/pull/3523<
> https://urldefense.com/v3/__https://github.com/apache/polaris/pull/3523__;!!Iz9xO38YGHZK!4sLrfIU8HUaeJshSZbprgWTwyaWUNAT-9Yof_YpVYP9T1Har02LkclV0s0wrDVe2-1tjesf_EfwLg5VWow$
> >
>
> Cheers,
> Dmitri.
>
>
> On Thu, Feb 12, 2026 at 6:00 PM Dmitri Bourlatchkov <dimas@apache.org
> <ma...@apache.org>> wrote:
> Hi All,
>
> The SPI PR [3616] for Scan/Commit metrics has been in review for 2 weeks
> including 3 days in an approved state.
>
> Unless there are any last minute comments, I propose to merge it tomorrow
> (Feb 13).
>
> [3616] https://github.com/apache/polaris/pull/3616<
> https://urldefense.com/v3/__https://github.com/apache/polaris/pull/3616__;!!Iz9xO38YGHZK!4sLrfIU8HUaeJshSZbprgWTwyaWUNAT-9Yof_YpVYP9T1Har02LkclV0s0wrDVe2-1tjesf_EfzzvX9eZQ$
> >
>
> Cheers,
> Dmitri.
>
> On Mon, Feb 9, 2026 at 2:08 PM Anand Kumar Sankaran via dev <
> dev@polaris.apache.org<ma...@polaris.apache.org>> wrote:
> Dmitri and others,
>
> Thanks for all the review feedback and being patient while we iterated. My
> initial proposal was to persist the table metrics as events (least amount
> of work 😊), we have come far from that proposal.
>
> To detach the SPI fully, there are a few things to do (loadSchemaVersion()
> and shared DatasourceOperations). We can add a separate datasource config
> to specify a different connection pool, perhaps. Can be done as a follow up
> PR.
>
> —
> Anand
>
> From: Dmitri Bourlatchkov <di...@apache.org>>
> Date: Monday, February 9, 2026 at 10:52 AM
> To: dev@polaris.apache.org<ma...@polaris.apache.org> <
> dev@polaris.apache.org<ma...@polaris.apache.org>>
> Cc: Anand Kumar Sankaran <anand.sankaran@workday.com<mailto:
> anand.sankaran@workday.com>>
> Subject: Re: Polaris Telemetry and Audit Trail
>
> This Message Is From an External Sender
> This message came from outside your organization.
> Report Suspicious<
> https://us-phishalarm-ewt.proofpoint.com/EWT/v1/Iz9xO38YGHZK!YhNDZAGr2cumYduJ20ONMhMxljYPF8uZOe6R8BpEWyJ7tSG5GUUf8our05nYHWov0t1Vi7Rx_O8JuRNBBQQ57aBoctrR0wH7VSR9VxRuqYJPpz0q6I0JboPRfdLIwqal$
> >
>
> Hi Anand and all,
>
> Thanks for pushing this feature forward! From my POV the SPI [3616] and
> the schema [3523] PRs are good to merge as the initial implementation of
> this feature (annotated as "beta"). The way is open to the JDBC impl. PR
> [3385] (which will probably need a rebase after merging the SPI and schema
> code).
>
> I'm sending this email to highlight the direction taken with respect to
> the SQL schema in this feature, as it might have been visible enough to the
> broader community during the PR reviews.
>
> * The scan / commit metrics schema is managed / versioned separately from
> the JDBC MetaStore schema
>
> * The Metrics Persistence SPI is detached (mostly at this time, but
> supposedly fully detached later) from the JDBC MetaStore
>
> * The JDBC DataSource for metrics may be the same or different from the
> MetaStore datasource.
>
> * Metrics Persistence may have non-JDBC implementation in the future (not
> implemented ATM)
>
> * JDBC Metrics Persistence may co-exist with other MetaStore
> implementations.
>
> I believe this design allows great flexibility for end users to try this
> feature out and keeps options open for future enhancements in Scan / Commit
> Metrics in Polaris.
>
> [3616] https://github.com/apache/polaris/pull/3616<
> https://urldefense.com/v3/__https://github.com/apache/polaris/pull/3616__;!!Iz9xO38YGHZK!4sLrfIU8HUaeJshSZbprgWTwyaWUNAT-9Yof_YpVYP9T1Har02LkclV0s0wrDVe2-1tjesf_EfzzvX9eZQ$
> ><
> https://urldefense.com/v3/__https://github.com/apache/polaris/pull/3616__;!!Iz9xO38YGHZK!7IwwfPXDN0ThhNZgol-n9Ww3FFLKcpZbSQ17lWEtcQnTHtJM9aVXnqGW99_6vmVkWACy7rYC8XEpN6Okdg$
> >
>
> [3523] https://github.com/apache/polaris/pull/3523<
> https://urldefense.com/v3/__https://github.com/apache/polaris/pull/3523__;!!Iz9xO38YGHZK!4sLrfIU8HUaeJshSZbprgWTwyaWUNAT-9Yof_YpVYP9T1Har02LkclV0s0wrDVe2-1tjesf_EfwLg5VWow$
> ><
> https://urldefense.com/v3/__https://github.com/apache/polaris/pull/3523__;!!Iz9xO38YGHZK!7IwwfPXDN0ThhNZgol-n9Ww3FFLKcpZbSQ17lWEtcQnTHtJM9aVXnqGW99_6vmVkWACy7rYC8XGOCHKhSg$
> >
>
> [3385] https://github.com/apache/polaris/pull/3385<
> https://urldefense.com/v3/__https://github.com/apache/polaris/pull/3385__;!!Iz9xO38YGHZK!4sLrfIU8HUaeJshSZbprgWTwyaWUNAT-9Yof_YpVYP9T1Har02LkclV0s0wrDVe2-1tjesf_Efwsf7PUHQ$
> ><
> https://urldefense.com/v3/__https://github.com/apache/polaris/pull/3385__;!!Iz9xO38YGHZK!7IwwfPXDN0ThhNZgol-n9Ww3FFLKcpZbSQ17lWEtcQnTHtJM9aVXnqGW99_6vmVkWACy7rYC8XFlxLMhvQ$
> >
>
> Thoughts / comments?
>
> Cheers,
> Dmitri.
>
>
> On Thu, Feb 5, 2026 at 6:50 PM Anand Kumar Sankaran via dev <
> dev@polaris.apache.org<ma...@polaris.apache.org><mailto:
> dev@polaris.apache.org<ma...@polaris.apache.org>>> wrote:
> Hi Dmitri,
>
> Following up on your feedback regarding the metrics schema separation
> approach. I've implemented the simplified design you suggested in
> https://github.com/apache/polaris/pull/3523<
> https://urldefense.com/v3/__https://github.com/apache/polaris/pull/3523__;!!Iz9xO38YGHZK!4sLrfIU8HUaeJshSZbprgWTwyaWUNAT-9Yof_YpVYP9T1Har02LkclV0s0wrDVe2-1tjesf_EfwLg5VWow$
> ><
> https://urldefense.com/v3/__https://github.com/apache/polaris/pull/3523__;!!Iz9xO38YGHZK!7IwwfPXDN0ThhNZgol-n9Ww3FFLKcpZbSQ17lWEtcQnTHtJM9aVXnqGW99_6vmVkWACy7rYC8XGOCHKhSg$
> >:
>
>
>  The metrics schema can now evolve independently from the entity schema.
> Let me know if you'd like any changes to this approach.
>
>  Thanks,
>   Anand
>
>
>
> From: Dmitri Bourlatchkov <dimas@apache.org<mailto:dimas@apache.org
> ><ma...@apache.org>>>
> Date: Thursday, February 5, 2026 at 2:57 PM
> To: dev@polaris.apache.org<ma...@polaris.apache.org><mailto:
> dev@polaris.apache.org<ma...@polaris.apache.org>> <
> dev@polaris.apache.org<ma...@polaris.apache.org><mailto:
> dev@polaris.apache.org<ma...@polaris.apache.org>>>
> Cc: Anand Kumar Sankaran <anand.sankaran@workday.com<mailto:
> anand.sankaran@workday.com><mailto:anand.sankaran@workday.com<mailto:
> anand.sankaran@workday.com>>>
> Subject: Re: Polaris Telemetry and Audit Trail
>
> This Message Is From an External Sender
> This message came from outside your organization.
> Report Suspicious<
> https://us-phishalarm-ewt.proofpoint.com/EWT/v1/Iz9xO38YGHZK!YhNDZAGr2cumY1iAus5GNbkIxSXoQaI552sXgYyv7Z9Wrvb7qeTskSy5WkXTVDJ_Bel2AbNh_fQVn-HS_UtOSdOtpsNfzhXDkK4fYWdTeoeBe70A6oQVGpAYT3SrbEvs$
> >
>
> Hi Anand,
>
> Re: MetricsPersistence Factory - TBH I do not think it is required. If we
> go with an MetricsPersistence impl. independent of the JDBC MetaStore, it
> could be injected into PersistingMetricsReporter directly by CDI.
>
> Re: DataSource - for a start we could probably reuse the same DataSource
> as the JDBS MetaStore uses. In principle, we could introduce a separate
> DataSource, but I do not think we need to complicate the initial
> implementation with that. It can be added later, if required.
>
> Re: schema / bootstrap - I'm thinking, indeed, of having a separate *.sql
> file for metrics DDL. Then we could add an option to the bootstrap command
> in the Admin Tool, or even a new command for creating / updating the
> metrics schema. This might require some refactoring, but I believe the
> bootstrap code needs some cleanup anyway.
>
> The general idea is to allow the metrics schema to evolve independently.
> It's a new and actively developed feature, so having flexibility is going
> to be good for making progress / improvements, I hope.
>
> Does that sound reasonable?
>
> Thanks,
> Dmitri.
>
> On Thu, Feb 5, 2026 at 3:47 PM Anand Kumar Sankaran via dev <
> dev@polaris.apache.org<ma...@polaris.apache.org><mailto:
> dev@polaris.apache.org<ma...@polaris.apache.org>><mailto:
> dev@polaris.apache.org<ma...@polaris.apache.org><mailto:
> dev@polaris.apache.org<ma...@polaris.apache.org>>>> wrote:
> Hi Dmitri
>
> The SPI is designed to be pluggable via
> MetaStoreManagerFactory.getOrCreateMetricsPersistence()<
> https://github.com/apache/polaris/pull/3385/changes#diff-0e34667d083d30cace067936b87c736201e5a8cb001aa0b12e203ee52cddae99R51
> <
> https://urldefense.com/v3/__https://github.com/apache/polaris/pull/3385/changes*diff-0e34667d083d30cace067936b87c736201e5a8cb001aa0b12e203ee52cddae99R51__;Iw!!Iz9xO38YGHZK!4sLrfIU8HUaeJshSZbprgWTwyaWUNAT-9Yof_YpVYP9T1Har02LkclV0s0wrDVe2-1tjesf_Efx9xKkRig$
> ><
> https://urldefense.com/v3/__https://github.com/apache/polaris/pull/3385/changes*diff-0e34667d083d30cace067936b87c736201e5a8cb001aa0b12e203ee52cddae99R51__;Iw!!Iz9xO38YGHZK!7IwwfPXDN0ThhNZgol-n9Ww3FFLKcpZbSQ17lWEtcQnTHtJM9aVXnqGW99_6vmVkWACy7rYC8XFxRr4qRw$
> ><
> https://urldefense.com/v3/__https://github.com/apache/polaris/pull/3385/changes*diff-0e34667d083d30cace067936b87c736201e5a8cb001aa0b12e203ee52cddae99R51__;Iw!!Iz9xO38YGHZK!_K1mcH0_2x0nbU-mZlbBi3tk9QAGjB6VInPiqkQU21mpIP_ft2bXXyIiADNoAcotNr9AKqlphh65taUhcw$>>.
> The default returns MetricsPersistence.NOOP, and only JDBC with schema v4+
> returns a functional implementation.
>
>  Your proposal makes sense to me. To allow NoSQL Entity Persistence + JDBC
> Metrics Persistence would require:
>
>      1. Separate metrics schema files (e.g., metrics-schema-v1.sql) that
> can be bootstrapped independently
>      2. Separate datasource configuration for metrics (e.g.,
> polaris.metrics.jdbc.*)
>      3. A dedicated `MetricsPersistenceFactory` interface (separate from
> MetaStoreManagerFactory)
>      4. Updates to the Admin Tool bootstrap workflow to handle both schemas
>
> Does that sound reasonable?  If we do this, all this work should go into
> https://github.com/apache/polaris/pull/3523<
> https://urldefense.com/v3/__https://github.com/apache/polaris/pull/3523__;!!Iz9xO38YGHZK!4sLrfIU8HUaeJshSZbprgWTwyaWUNAT-9Yof_YpVYP9T1Har02LkclV0s0wrDVe2-1tjesf_EfwLg5VWow$
> ><
> https://urldefense.com/v3/__https://github.com/apache/polaris/pull/3523__;!!Iz9xO38YGHZK!7IwwfPXDN0ThhNZgol-n9Ww3FFLKcpZbSQ17lWEtcQnTHtJM9aVXnqGW99_6vmVkWACy7rYC8XGOCHKhSg$
> ><
> https://urldefense.com/v3/__https://github.com/apache/polaris/pull/3523__;!!Iz9xO38YGHZK!_K1mcH0_2x0nbU-mZlbBi3tk9QAGjB6VInPiqkQU21mpIP_ft2bXXyIiADNoAcotNr9AKqlphh4ZISZcDQ$>
> as one atomic commit, correct?
>
> —
> Anand
>
>
>
> From: Dmitri Bourlatchkov <dimas@apache.org<mailto:dimas@apache.org
> ><ma...@apache.org>><mailto:
> dimas@apache.org<ma...@apache.org><mailto:dimas@apache.org<mailto:
> dimas@apache.org>>>>
> Date: Thursday, February 5, 2026 at 8:28 AM
> To: Anand Kumar Sankaran <anand.sankaran@workday.com<mailto:
> anand.sankaran@workday.com><mailto:anand.sankaran@workday.com<mailto:
> anand.sankaran@workday.com>><mailto:anand.sankaran@workday.com<mailto:
> anand.sankaran@workday.com><mailto:anand.sankaran@workday.com<mailto:
> anand.sankaran@workday.com>>>>
> Cc: dev@polaris.apache.org<ma...@polaris.apache.org><mailto:
> dev@polaris.apache.org<ma...@polaris.apache.org>><mailto:
> dev@polaris.apache.org<ma...@polaris.apache.org><mailto:
> dev@polaris.apache.org<ma...@polaris.apache.org>>> <
> dev@polaris.apache.org<ma...@polaris.apache.org><mailto:
> dev@polaris.apache.org<ma...@polaris.apache.org>><mailto:
> dev@polaris.apache.org<ma...@polaris.apache.org><mailto:
> dev@polaris.apache.org<ma...@polaris.apache.org>>>>
> Subject: Re: Polaris Telemetry and Audit Trail
>
> This Message Is From an External Sender
> This message came from outside your organization.
> Report Suspicious<
> https://us-phishalarm-ewt.proofpoint.com/EWT/v1/Iz9xO38YGHZK!YhNDZAGr2cumbRcPlIOttKwvGr4f38TF-J2-LmOw8H1KsFpWaM3W5g28Oq5qW2gnw2-WHz0rsWhm7F7hSGykA9l01LKYGuCcCkexkHp1sUssM_IegCtjk_id9s891orv$
> >
>
>
> Hi Anand,
>
> Thanks for driving the SPI definition for scan/commits metrics.
>
> I did not think from this angle before, but now I suppose it might be worth
> introducing a separate SQL schema for metrics, detached from the main JDBC
> Persistence schema.
>
> My rationale is that the SPI may be exercised independently of the JDBC
> Entity Persistence. I can imagine people using NoSQL Entity Persistence
> plus JDBC Scan Metrics Persistence in the same deployment.
>
> Using a separate schema will add further work on the Bootstrap workflow
> (Admin Tool), but I hope the end result is going to be more convenient and
> versatile for end users.
>
> WDYT?
>
> Thanks,
> Dmitri.
>
> On Thu, Jan 29, 2026 at 1:49 PM Anand Kumar Sankaran <
> anand.sankaran@workday.com<ma...@workday.com><mailto:
> anand.sankaran@workday.com<ma...@workday.com>><mailto:
> anand.sankaran@workday.com<ma...@workday.com><mailto:
> anand.sankaran@workday.com<ma...@workday.com>>>> wrote:
>
> > Hi Dmitri,
> >
> > Thank you for taking the time to review. I had pulled out the schema in
> > its own PR (based on comments) here:
> >
> https://urldefense.com/v3/__https://github.com/apache/polaris/pull/3523__;!!Iz9xO38YGHZK!65BVFQDe_GtKbDW8LkNdzVS-WYewzWsSns5zFHKclbMU3vUy2UaTXLB5G-wql1yyqoTFyDc9Bt3GRsh6AA$
> >
> > I will update the document with the SPI and carve out another PR that
> > focuses on the SPI.
> >
> > —
> > Anand
> >
> > *From: *Dmitri Bourlatchkov <dmitri.bourlatchkov@dremio.com<mailto:
> dmitri.bourlatchkov@dremio.com><mailto:dmitri.bourlatchkov@dremio.com
> <ma...@dremio.com>><mailto:
> dmitri.bourlatchkov@dremio.com<mailto:dmitri.bourlatchkov@dremio.com
> ><mailto:dmitri.bourlatchkov@dremio.com<mailto:
> dmitri.bourlatchkov@dremio.com>>>>
> > *Date: *Thursday, January 29, 2026 at 10:43 AM
> > *To: *dev@polaris.apache.org<ma...@polaris.apache.org><mailto:
> dev@polaris.apache.org<ma...@polaris.apache.org>><mailto:
> dev@polaris.apache.org<ma...@polaris.apache.org><mailto:
> dev@polaris.apache.org<ma...@polaris.apache.org>>> <
> dev@polaris.apache.org<ma...@polaris.apache.org><mailto:
> dev@polaris.apache.org<ma...@polaris.apache.org>><mailto:
> dev@polaris.apache.org<ma...@polaris.apache.org><mailto:
> dev@polaris.apache.org<ma...@polaris.apache.org>>>>
> > *Cc: *Anand Kumar Sankaran <anand.sankaran@workday.com<mailto:
> anand.sankaran@workday.com><mailto:anand.sankaran@workday.com<mailto:
> anand.sankaran@workday.com>><mailto:anand.sankaran@workday.com<mailto:
> anand.sankaran@workday.com><mailto:anand.sankaran@workday.com<mailto:
> anand.sankaran@workday.com>>>>
> > *Subject: *Re: Polaris Telemetry and Audit Trail
> >
> > This Message Is From an External Sender
> > This message came from outside your organization.
> > Report Suspicious
> > <
> https://us-phishalarm-ewt.proofpoint.com/EWT/v1/Iz9xO38YGHZK!YhNDZAGr2cumY1cAlUeoPrU2eNUPz9Wxq2PaG1t99vEN3v60gucxC3Qsk-xK09RndcW46WtNjMMjQZbaQ495h1D1g293F_uqLhPM8gDMlA6xRmo5b5uqrSh2v7QKHrhQ$
> >
> >
> > Hi Anand,
> >
> > Thanks for making a proposal doc and starting this thread. Apologies for
> > late comments.
> >
> > As you know I already reviewed a couple of related PRs, which had a
> > smaller impact on the codebase, but now, I believe, we're coming to the
> > core of this feature and it might need a deeper discussion.
> >
> > For the sake of awareness of other project members, I'd like to
> > highlight some aspects by email.
> >
> > * Storing scan metrics in RDBMS in general looks like a
> > reasonable approach to me. However, Polaris Persistence is an extensible
> > mechanism. Other backends are possible, including privately developed
> > backends.
> >
> > From this POV, I believe it might be preferable to start not with an
> RDBMS
> > schema, but with a java SPI outlining expected read/write operations and
> > the data model. This will make it easier to assess impact on other
> > Persistence implementations. Naturally, Polaris service code will have to
> > be able to work seamlessly regardless of the backend impl. (including the
> > do-nothing impl.).
> >
> > I see that some connection between the SQL schema and java code is made
> in
> > PR [3385], but it does not appear to offer a clean SPI that could be
> > implemented by different Persistence backends. Would it be ok from your
> > perspective to make another PR with just SPI code?
> >
> > * Regarding ad-hoc SQL queries - it is certainly a valid use case for
> > custom code to query the Polaris database directly. However, those ad-hoc
> > queries may require certain indexes, which may not have a relevant use
> case
> > in OSS code. In that case, I believe it would be preferable to manage the
> > extra indexes in custom code (not in Polaris). Whether this is the case
> or
> > not currently, I cannot say with certainty, so apologies if this is a
> false
> > alarm... Having the java SPI available would be a huge aid to reasoning
> > about the proposed SQL schema, I think.
> >
> > That said, I do not mean to complicate the development of this feature,
> > just trying to structure it in a way that is hopefully easy to understand
> > (including myself), maintain and expand later. If I missed something,
> > please feel free to point it out.
> >
> > [3385]
> https://urldefense.com/v3/__https://github.com/apache/polaris/pull/3385__;!!Iz9xO38YGHZK!65BVFQDe_GtKbDW8LkNdzVS-WYewzWsSns5zFHKclbMU3vUy2UaTXLB5G-wql1yyqoTFyDc9Bt2W-eKl7A$
> > <
> https://urldefense.com/v3/__https://github.com/apache/polaris/pull/3385__;!!Iz9xO38YGHZK!_dbBfwEnZEuPp_8qIuBDoYfWRgCZKiPDMg9ouDltqsHhr9hqCHVlF9IedSkNVBDWupbneU7inC9YK-THL-lTjWkkyZhI0OE$
> >
> >
> >Thanks,
> > Dmitri.
> >
> > On Wed, Jan 14, 2026 at 10:05 PM Anand Kumar Sankaran via dev <
> > dev@polaris.apache.org<ma...@polaris.apache.org><mailto:
> dev@polaris.apache.org<ma...@polaris.apache.org>><mailto:
> dev@polaris.apache.org<ma...@polaris.apache.org><mailto:
> dev@polaris.apache.org<ma...@polaris.apache.org>>>> wrote:
> >
> > Hi Yufei,
> >
> > Gave commenting privileges to all.
> >
> > Thanks.
> >
> > —
> > Anand
> >
> > From: Yufei Gu <flyrain000@gmail.com<mailto:flyrain000@gmail.com
> ><ma...@gmail.com>><mailto:
> flyrain000@gmail.com<ma...@gmail.com><mailto:
> flyrain000@gmail.com<ma...@gmail.com>>>>
> > Date: Wednesday, January 14, 2026 at 6:13 PM
> > To: dev@polaris.apache.org<ma...@polaris.apache.org><mailto:
> dev@polaris.apache.org<ma...@polaris.apache.org>><mailto:
> dev@polaris.apache.org<ma...@polaris.apache.org><mailto:
> dev@polaris.apache.org<ma...@polaris.apache.org>>> <
> dev@polaris.apache.org<ma...@polaris.apache.org><mailto:
> dev@polaris.apache.org<ma...@polaris.apache.org>><mailto:
> dev@polaris.apache.org<ma...@polaris.apache.org><mailto:
> dev@polaris.apache.org<ma...@polaris.apache.org>>>>
> > Cc: Anand Kumar Sankaran <anand.sankaran@workday.com<mailto:
> anand.sankaran@workday.com><mailto:anand.sankaran@workday.com<mailto:
> anand.sankaran@workday.com>><mailto:anand.sankaran@workday.com<mailto:
> anand.sankaran@workday.com><mailto:anand.sankaran@workday.com<mailto:
> anand.sankaran@workday.com>>>>
> > Subject: Re: Polaris Telemetry and Audit Trail
> >
> > This Message Is From an External Sender
> > This message came from outside your organization.
> > Report Suspicious<
> >
> https://us-phishalarm-ewt.proofpoint.com/EWT/v1/Iz9xO38YGHZK!YhNDZABkHi1B6hyPVYUK0T-K5j4Aoqdrlu07UjBpJcpjOsz7Ie7d_DhPl-ywi50dfQz4O6LL-AHLCOQTQ0X7Le8yD2cF0ogquGFB8Aw8uQLgB0RL6Ezpa-pLDPA4ziXA$
> > >
> >
> > Hi Anand, thanks for the design doc and PR. Can you allow comments in the
> > doc so that people can chime in? Thanks!
> >
> > Yufei
> >
> >
> > On Sat, Jan 10, 2026 at 9:09 AM Anand Kumar Sankaran via dev <
> > dev@polaris.apache.org<ma...@polaris.apache.org><mailto:
> dev@polaris.apache.org<ma...@polaris.apache.org>><mailto:
> dev@polaris.apache.org<ma...@polaris.apache.org><mailto:
> dev@polaris.apache.org<ma...@polaris.apache.org>>><mailto:
> dev@polaris.apache.org<ma...@polaris.apache.org><mailto:
> dev@polaris.apache.org<ma...@polaris.apache.org>><mailto:
> dev@polaris.apache.org<ma...@polaris.apache.org><mailto:
> dev@polaris.apache.org<ma...@polaris.apache.org>>>>> wrote:
> > Hi all
> >
> > My first PR for adding AWS STS Session Tags support for credential
> vending
> > was merged.
> https://urldefense.com/v3/__https://github.com/apache/polaris/pull/3327__;!!Iz9xO38YGHZK!65BVFQDe_GtKbDW8LkNdzVS-WYewzWsSns5zFHKclbMU3vUy2UaTXLB5G-wql1yyqoTFyDc9Bt0YHY9sdA$
> > <
> https://urldefense.com/v3/__https://github.com/apache/polaris/pull/3327__;!!Iz9xO38YGHZK!_dbBfwEnZEuPp_8qIuBDoYfWRgCZKiPDMg9ouDltqsHhr9hqCHVlF9IedSkNVBDWupbneU7inC9YK-THL-lTjWkklJ4Xf9Y$
> >
> ><
> >
> https://urldefense.com/v3/__https://github.com/apache/polaris/pull/3327__;!!Iz9xO38YGHZK!8vlQ2_OqjmZNIpbrTTluFFNKFCEeM87nr95MP5pMpMu5fFWVSoBZ3eb6-C2xbAGCG-ry5eaCO-pFLxl3IZAJGqY$
> >>
> >
> > I have been working with Prashant on an end-to-end telemetry and audit
> > trail tracking for Polaris.  It is documented here.
> >
> https://urldefense.com/v3/__https://docs.google.com/document/d/1Ehzvi5RNPs4hChkBFI6VD23myEqm-7sWW3d2kjmuYj8/edit?tab=t.0__;!!Iz9xO38YGHZK!65BVFQDe_GtKbDW8LkNdzVS-WYewzWsSns5zFHKclbMU3vUy2UaTXLB5G-wql1yyqoTFyDc9Bt11Y1rbNQ$
> > <
> https://urldefense.com/v3/__https://docs.google.com/document/d/1Ehzvi5RNPs4hChkBFI6VD23myEqm-7sWW3d2kjmuYj8/edit?tab=t.0__;!!Iz9xO38YGHZK!_dbBfwEnZEuPp_8qIuBDoYfWRgCZKiPDMg9ouDltqsHhr9hqCHVlF9IedSkNVBDWupbneU7inC9YK-THL-lTjWkk8MVJPlY$
> >
> ><
> >
> https://urldefense.com/v3/__https://docs.google.com/document/d/1Ehzvi5RNPs4hChkBFI6VD23myEqm-7sWW3d2kjmuYj8/edit?tab=t.0__;!!Iz9xO38YGHZK!8vlQ2_OqjmZNIpbrTTluFFNKFCEeM87nr95MP5pMpMu5fFWVSoBZ3eb6-C2xbAGCG-ry5eaCO-pFLxl3hYqTlcg$
> >>
> >
> > Based on Prashant’s initial feedback (parity with Apache Gravitino for
> > metrics reports), I have an initial PR here.
> >
> https://urldefense.com/v3/__https://github.com/apache/polaris/pull/3385__;!!Iz9xO38YGHZK!65BVFQDe_GtKbDW8LkNdzVS-WYewzWsSns5zFHKclbMU3vUy2UaTXLB5G-wql1yyqoTFyDc9Bt2W-eKl7A$
> > <
> https://urldefense.com/v3/__https://github.com/apache/polaris/pull/3385__;!!Iz9xO38YGHZK!_dbBfwEnZEuPp_8qIuBDoYfWRgCZKiPDMg9ouDltqsHhr9hqCHVlF9IedSkNVBDWupbneU7inC9YK-THL-lTjWkkyZhI0OE$
> >
> ><
> >
> https://urldefense.com/v3/__https://github.com/apache/polaris/pull/3385__;!!Iz9xO38YGHZK!8vlQ2_OqjmZNIpbrTTluFFNKFCEeM87nr95MP5pMpMu5fFWVSoBZ3eb6-C2xbAGCG-ry5eaCO-pFLxl39opCznI$
> >>
> >
> > I will be reaching out to various folks for review. I am open to breaking
> > down the PR into smaller logical groups of PR if it helps.
> >
> > I request you to look at the Google doc and let me know what you think.
> >
> > —
> > Anand
> >
> >
> >
> > --
> > Dmitri Bourlatchkov
> > Senior Staff Software Engineer, Dremio
> > Dremio.com
> > <
> https://urldefense.com/v3/__https://www.dremio.com/?utm_medium=email&utm_source=signature&utm_term=na&utm_content=email-signature&utm_campaign=email-signature__;!!Iz9xO38YGHZK!_dbBfwEnZEuPp_8qIuBDoYfWRgCZKiPDMg9ouDltqsHhr9hqCHVlF9IedSkNVBDWupbneU7inC9YK-THL-lTjWkkiOwjJc0$
> >/
> > Follow Us on LinkedIn
> > <
> https://urldefense.com/v3/__https://www.linkedin.com/company/dremio__;!!Iz9xO38YGHZK!_dbBfwEnZEuPp_8qIuBDoYfWRgCZKiPDMg9ouDltqsHhr9hqCHVlF9IedSkNVBDWupbneU7inC9YK-THL-lTjWkkCLjBQ2M$
> >/
> > Get Started
> > <
> https://urldefense.com/v3/__https://www.dremio.com/get-started/__;!!Iz9xO38YGHZK!_dbBfwEnZEuPp_8qIuBDoYfWRgCZKiPDMg9ouDltqsHhr9hqCHVlF9IedSkNVBDWupbneU7inC9YK-THL-lTjWkkoKQUMII$
> >
> >
> >
> >The Agentic Lakehouse
> > *The only lakehouse built for agents, managed by agents*
> >
> >
> >
>
>

Re: Polaris Telemetry and Audit Trail

[Anand Kumar Sankaran via dev <de...@polaris.apache.org>]

Hi Dmitri,

Thanks again.  I have rebased [3385] against main.

—
Anand

From: Dmitri Bourlatchkov <di...@apache.org>
Date: Tuesday, February 17, 2026 at 12:11 PM
To: dev@polaris.apache.org <de...@polaris.apache.org>
Cc: Anand Kumar Sankaran <an...@workday.com>
Subject: Re: Polaris Telemetry and Audit Trail

This Message Is From an External Sender
This message came from outside your organization.
Report Suspicious<https://us-phishalarm-ewt.proofpoint.com/EWT/v1/Iz9xO38YGHZK!YhNDZAGr2cumYdvL-GeGN_sUtpf-vTAsw9ogTCTPGKCcS5ePHJr_UGiyUqgDQ5VNCl6xTwYj4qqUWwkvyuTh43tJ_AVdZJPgCLZDpZXbpB6L3jCaJL5OVoyRsNvKnAwL$>

Hi Anand,

PR [3523] (SQL schema) has been merged.

Please rebase [3385].

[3523] https://github.com/apache/polaris/pull/3523<https://urldefense.com/v3/__https://github.com/apache/polaris/pull/3523__;!!Iz9xO38YGHZK!4sLrfIU8HUaeJshSZbprgWTwyaWUNAT-9Yof_YpVYP9T1Har02LkclV0s0wrDVe2-1tjesf_EfwLg5VWow$>

[3385] https://github.com/apache/polaris/pull/3385<https://urldefense.com/v3/__https://github.com/apache/polaris/pull/3385__;!!Iz9xO38YGHZK!4sLrfIU8HUaeJshSZbprgWTwyaWUNAT-9Yof_YpVYP9T1Har02LkclV0s0wrDVe2-1tjesf_Efwsf7PUHQ$>

Thanks,
Dmitri.

On Fri, Feb 13, 2026 at 6:10 PM Dmitri Bourlatchkov <di...@apache.org>> wrote:
Hi All,

The SPI PR [3616] has been merged.

I propose to merge the Schema PR [3523] on Mon if there are no objections or fresh comments.

[3616] https://github.com/apache/polaris/pull/3616<https://urldefense.com/v3/__https://github.com/apache/polaris/pull/3616__;!!Iz9xO38YGHZK!4sLrfIU8HUaeJshSZbprgWTwyaWUNAT-9Yof_YpVYP9T1Har02LkclV0s0wrDVe2-1tjesf_EfzzvX9eZQ$>

[3523] https://github.com/apache/polaris/pull/3523<https://urldefense.com/v3/__https://github.com/apache/polaris/pull/3523__;!!Iz9xO38YGHZK!4sLrfIU8HUaeJshSZbprgWTwyaWUNAT-9Yof_YpVYP9T1Har02LkclV0s0wrDVe2-1tjesf_EfwLg5VWow$>

Cheers,
Dmitri.


On Thu, Feb 12, 2026 at 6:00 PM Dmitri Bourlatchkov <di...@apache.org>> wrote:
Hi All,

The SPI PR [3616] for Scan/Commit metrics has been in review for 2 weeks including 3 days in an approved state.

Unless there are any last minute comments, I propose to merge it tomorrow (Feb 13).

[3616] https://github.com/apache/polaris/pull/3616<https://urldefense.com/v3/__https://github.com/apache/polaris/pull/3616__;!!Iz9xO38YGHZK!4sLrfIU8HUaeJshSZbprgWTwyaWUNAT-9Yof_YpVYP9T1Har02LkclV0s0wrDVe2-1tjesf_EfzzvX9eZQ$>

Cheers,
Dmitri.

On Mon, Feb 9, 2026 at 2:08 PM Anand Kumar Sankaran via dev <de...@polaris.apache.org>> wrote:
Dmitri and others,

Thanks for all the review feedback and being patient while we iterated. My initial proposal was to persist the table metrics as events (least amount of work 😊), we have come far from that proposal.

To detach the SPI fully, there are a few things to do (loadSchemaVersion() and shared DatasourceOperations). We can add a separate datasource config to specify a different connection pool, perhaps. Can be done as a follow up PR.

—
Anand

From: Dmitri Bourlatchkov <di...@apache.org>>
Date: Monday, February 9, 2026 at 10:52 AM
To: dev@polaris.apache.org<ma...@polaris.apache.org> <de...@polaris.apache.org>>
Cc: Anand Kumar Sankaran <an...@workday.com>>
Subject: Re: Polaris Telemetry and Audit Trail

This Message Is From an External Sender
This message came from outside your organization.
Report Suspicious<https://us-phishalarm-ewt.proofpoint.com/EWT/v1/Iz9xO38YGHZK!YhNDZAGr2cumYduJ20ONMhMxljYPF8uZOe6R8BpEWyJ7tSG5GUUf8our05nYHWov0t1Vi7Rx_O8JuRNBBQQ57aBoctrR0wH7VSR9VxRuqYJPpz0q6I0JboPRfdLIwqal$>

Hi Anand and all,

Thanks for pushing this feature forward! From my POV the SPI [3616] and the schema [3523] PRs are good to merge as the initial implementation of this feature (annotated as "beta"). The way is open to the JDBC impl. PR [3385] (which will probably need a rebase after merging the SPI and schema code).

I'm sending this email to highlight the direction taken with respect to the SQL schema in this feature, as it might have been visible enough to the broader community during the PR reviews.

* The scan / commit metrics schema is managed / versioned separately from the JDBC MetaStore schema

* The Metrics Persistence SPI is detached (mostly at this time, but supposedly fully detached later) from the JDBC MetaStore

* The JDBC DataSource for metrics may be the same or different from the MetaStore datasource.

* Metrics Persistence may have non-JDBC implementation in the future (not implemented ATM)

* JDBC Metrics Persistence may co-exist with other MetaStore implementations.

I believe this design allows great flexibility for end users to try this feature out and keeps options open for future enhancements in Scan / Commit Metrics in Polaris.

[3616] https://github.com/apache/polaris/pull/3616<https://urldefense.com/v3/__https://github.com/apache/polaris/pull/3616__;!!Iz9xO38YGHZK!4sLrfIU8HUaeJshSZbprgWTwyaWUNAT-9Yof_YpVYP9T1Har02LkclV0s0wrDVe2-1tjesf_EfzzvX9eZQ$><https://urldefense.com/v3/__https://github.com/apache/polaris/pull/3616__;!!Iz9xO38YGHZK!7IwwfPXDN0ThhNZgol-n9Ww3FFLKcpZbSQ17lWEtcQnTHtJM9aVXnqGW99_6vmVkWACy7rYC8XEpN6Okdg$>

[3523] https://github.com/apache/polaris/pull/3523<https://urldefense.com/v3/__https://github.com/apache/polaris/pull/3523__;!!Iz9xO38YGHZK!4sLrfIU8HUaeJshSZbprgWTwyaWUNAT-9Yof_YpVYP9T1Har02LkclV0s0wrDVe2-1tjesf_EfwLg5VWow$><https://urldefense.com/v3/__https://github.com/apache/polaris/pull/3523__;!!Iz9xO38YGHZK!7IwwfPXDN0ThhNZgol-n9Ww3FFLKcpZbSQ17lWEtcQnTHtJM9aVXnqGW99_6vmVkWACy7rYC8XGOCHKhSg$>

[3385] https://github.com/apache/polaris/pull/3385<https://urldefense.com/v3/__https://github.com/apache/polaris/pull/3385__;!!Iz9xO38YGHZK!4sLrfIU8HUaeJshSZbprgWTwyaWUNAT-9Yof_YpVYP9T1Har02LkclV0s0wrDVe2-1tjesf_Efwsf7PUHQ$><https://urldefense.com/v3/__https://github.com/apache/polaris/pull/3385__;!!Iz9xO38YGHZK!7IwwfPXDN0ThhNZgol-n9Ww3FFLKcpZbSQ17lWEtcQnTHtJM9aVXnqGW99_6vmVkWACy7rYC8XFlxLMhvQ$>

Thoughts / comments?

Cheers,
Dmitri.


On Thu, Feb 5, 2026 at 6:50 PM Anand Kumar Sankaran via dev <de...@polaris.apache.org>>> wrote:
Hi Dmitri,

Following up on your feedback regarding the metrics schema separation approach. I've implemented the simplified design you suggested in https://github.com/apache/polaris/pull/3523<https://urldefense.com/v3/__https://github.com/apache/polaris/pull/3523__;!!Iz9xO38YGHZK!4sLrfIU8HUaeJshSZbprgWTwyaWUNAT-9Yof_YpVYP9T1Har02LkclV0s0wrDVe2-1tjesf_EfwLg5VWow$><https://urldefense.com/v3/__https://github.com/apache/polaris/pull/3523__;!!Iz9xO38YGHZK!7IwwfPXDN0ThhNZgol-n9Ww3FFLKcpZbSQ17lWEtcQnTHtJM9aVXnqGW99_6vmVkWACy7rYC8XGOCHKhSg$>:


 The metrics schema can now evolve independently from the entity schema. Let me know if you'd like any changes to this approach.

 Thanks,
  Anand



From: Dmitri Bourlatchkov <di...@apache.org>>>
Date: Thursday, February 5, 2026 at 2:57 PM
To: dev@polaris.apache.org<ma...@polaris.apache.org>> <de...@polaris.apache.org>>>
Cc: Anand Kumar Sankaran <an...@workday.com>>>
Subject: Re: Polaris Telemetry and Audit Trail

This Message Is From an External Sender
This message came from outside your organization.
Report Suspicious<https://us-phishalarm-ewt.proofpoint.com/EWT/v1/Iz9xO38YGHZK!YhNDZAGr2cumY1iAus5GNbkIxSXoQaI552sXgYyv7Z9Wrvb7qeTskSy5WkXTVDJ_Bel2AbNh_fQVn-HS_UtOSdOtpsNfzhXDkK4fYWdTeoeBe70A6oQVGpAYT3SrbEvs$>

Hi Anand,

Re: MetricsPersistence Factory - TBH I do not think it is required. If we go with an MetricsPersistence impl. independent of the JDBC MetaStore, it could be injected into PersistingMetricsReporter directly by CDI.

Re: DataSource - for a start we could probably reuse the same DataSource as the JDBS MetaStore uses. In principle, we could introduce a separate DataSource, but I do not think we need to complicate the initial implementation with that. It can be added later, if required.

Re: schema / bootstrap - I'm thinking, indeed, of having a separate *.sql file for metrics DDL. Then we could add an option to the bootstrap command in the Admin Tool, or even a new command for creating / updating the metrics schema. This might require some refactoring, but I believe the bootstrap code needs some cleanup anyway.

The general idea is to allow the metrics schema to evolve independently. It's a new and actively developed feature, so having flexibility is going to be good for making progress / improvements, I hope.

Does that sound reasonable?

Thanks,
Dmitri.

On Thu, Feb 5, 2026 at 3:47 PM Anand Kumar Sankaran via dev <de...@polaris.apache.org>>>> wrote:
Hi Dmitri

The SPI is designed to be pluggable via MetaStoreManagerFactory.getOrCreateMetricsPersistence()<https://github.com/apache/polaris/pull/3385/changes#diff-0e34667d083d30cace067936b87c736201e5a8cb001aa0b12e203ee52cddae99R51<https://urldefense.com/v3/__https://github.com/apache/polaris/pull/3385/changes*diff-0e34667d083d30cace067936b87c736201e5a8cb001aa0b12e203ee52cddae99R51__;Iw!!Iz9xO38YGHZK!4sLrfIU8HUaeJshSZbprgWTwyaWUNAT-9Yof_YpVYP9T1Har02LkclV0s0wrDVe2-1tjesf_Efx9xKkRig$><https://urldefense.com/v3/__https://github.com/apache/polaris/pull/3385/changes*diff-0e34667d083d30cace067936b87c736201e5a8cb001aa0b12e203ee52cddae99R51__;Iw!!Iz9xO38YGHZK!7IwwfPXDN0ThhNZgol-n9Ww3FFLKcpZbSQ17lWEtcQnTHtJM9aVXnqGW99_6vmVkWACy7rYC8XFxRr4qRw$><https://urldefense.com/v3/__https://github.com/apache/polaris/pull/3385/changes*diff-0e34667d083d30cace067936b87c736201e5a8cb001aa0b12e203ee52cddae99R51__;Iw!!Iz9xO38YGHZK!_K1mcH0_2x0nbU-mZlbBi3tk9QAGjB6VInPiqkQU21mpIP_ft2bXXyIiADNoAcotNr9AKqlphh65taUhcw$>>. The default returns MetricsPersistence.NOOP, and only JDBC with schema v4+ returns a functional implementation.

 Your proposal makes sense to me. To allow NoSQL Entity Persistence + JDBC Metrics Persistence would require:

     1. Separate metrics schema files (e.g., metrics-schema-v1.sql) that can be bootstrapped independently
     2. Separate datasource configuration for metrics (e.g., polaris.metrics.jdbc.*)
     3. A dedicated `MetricsPersistenceFactory` interface (separate from MetaStoreManagerFactory)
     4. Updates to the Admin Tool bootstrap workflow to handle both schemas

Does that sound reasonable?  If we do this, all this work should go into https://github.com/apache/polaris/pull/3523<https://urldefense.com/v3/__https://github.com/apache/polaris/pull/3523__;!!Iz9xO38YGHZK!4sLrfIU8HUaeJshSZbprgWTwyaWUNAT-9Yof_YpVYP9T1Har02LkclV0s0wrDVe2-1tjesf_EfwLg5VWow$><https://urldefense.com/v3/__https://github.com/apache/polaris/pull/3523__;!!Iz9xO38YGHZK!7IwwfPXDN0ThhNZgol-n9Ww3FFLKcpZbSQ17lWEtcQnTHtJM9aVXnqGW99_6vmVkWACy7rYC8XGOCHKhSg$><https://urldefense.com/v3/__https://github.com/apache/polaris/pull/3523__;!!Iz9xO38YGHZK!_K1mcH0_2x0nbU-mZlbBi3tk9QAGjB6VInPiqkQU21mpIP_ft2bXXyIiADNoAcotNr9AKqlphh4ZISZcDQ$> as one atomic commit, correct?

—
Anand



From: Dmitri Bourlatchkov <di...@apache.org>>>>
Date: Thursday, February 5, 2026 at 8:28 AM
To: Anand Kumar Sankaran <an...@workday.com>>>>
Cc: dev@polaris.apache.org<ma...@polaris.apache.org>>> <de...@polaris.apache.org>>>>
Subject: Re: Polaris Telemetry and Audit Trail

This Message Is From an External Sender
This message came from outside your organization.
Report Suspicious<https://us-phishalarm-ewt.proofpoint.com/EWT/v1/Iz9xO38YGHZK!YhNDZAGr2cumbRcPlIOttKwvGr4f38TF-J2-LmOw8H1KsFpWaM3W5g28Oq5qW2gnw2-WHz0rsWhm7F7hSGykA9l01LKYGuCcCkexkHp1sUssM_IegCtjk_id9s891orv$>


Hi Anand,

Thanks for driving the SPI definition for scan/commits metrics.

I did not think from this angle before, but now I suppose it might be worth
introducing a separate SQL schema for metrics, detached from the main JDBC
Persistence schema.

My rationale is that the SPI may be exercised independently of the JDBC
Entity Persistence. I can imagine people using NoSQL Entity Persistence
plus JDBC Scan Metrics Persistence in the same deployment.

Using a separate schema will add further work on the Bootstrap workflow
(Admin Tool), but I hope the end result is going to be more convenient and
versatile for end users.

WDYT?

Thanks,
Dmitri.

On Thu, Jan 29, 2026 at 1:49 PM Anand Kumar Sankaran <
anand.sankaran@workday.com<ma...@workday.com>>>> wrote:

> Hi Dmitri,
>
> Thank you for taking the time to review. I had pulled out the schema in
> its own PR (based on comments) here:
> https://urldefense.com/v3/__https://github.com/apache/polaris/pull/3523__;!!Iz9xO38YGHZK!65BVFQDe_GtKbDW8LkNdzVS-WYewzWsSns5zFHKclbMU3vUy2UaTXLB5G-wql1yyqoTFyDc9Bt3GRsh6AA$
>
> I will update the document with the SPI and carve out another PR that
> focuses on the SPI.
>
> —
> Anand
>
> *From: *Dmitri Bourlatchkov <dm...@dremio.com>>>>
> *Date: *Thursday, January 29, 2026 at 10:43 AM
> *To: *dev@polaris.apache.org<ma...@polaris.apache.org>>> <de...@polaris.apache.org>>>>
> *Cc: *Anand Kumar Sankaran <an...@workday.com>>>>
> *Subject: *Re: Polaris Telemetry and Audit Trail
>
> This Message Is From an External Sender
> This message came from outside your organization.
> Report Suspicious
> <https://us-phishalarm-ewt.proofpoint.com/EWT/v1/Iz9xO38YGHZK!YhNDZAGr2cumY1cAlUeoPrU2eNUPz9Wxq2PaG1t99vEN3v60gucxC3Qsk-xK09RndcW46WtNjMMjQZbaQ495h1D1g293F_uqLhPM8gDMlA6xRmo5b5uqrSh2v7QKHrhQ$ >
>
> Hi Anand,
>
> Thanks for making a proposal doc and starting this thread. Apologies for
> late comments.
>
> As you know I already reviewed a couple of related PRs, which had a
> smaller impact on the codebase, but now, I believe, we're coming to the
> core of this feature and it might need a deeper discussion.
>
> For the sake of awareness of other project members, I'd like to
> highlight some aspects by email.
>
> * Storing scan metrics in RDBMS in general looks like a
> reasonable approach to me. However, Polaris Persistence is an extensible
> mechanism. Other backends are possible, including privately developed
> backends.
>
> From this POV, I believe it might be preferable to start not with an RDBMS
> schema, but with a java SPI outlining expected read/write operations and
> the data model. This will make it easier to assess impact on other
> Persistence implementations. Naturally, Polaris service code will have to
> be able to work seamlessly regardless of the backend impl. (including the
> do-nothing impl.).
>
> I see that some connection between the SQL schema and java code is made in
> PR [3385], but it does not appear to offer a clean SPI that could be
> implemented by different Persistence backends. Would it be ok from your
> perspective to make another PR with just SPI code?
>
> * Regarding ad-hoc SQL queries - it is certainly a valid use case for
> custom code to query the Polaris database directly. However, those ad-hoc
> queries may require certain indexes, which may not have a relevant use case
> in OSS code. In that case, I believe it would be preferable to manage the
> extra indexes in custom code (not in Polaris). Whether this is the case or
> not currently, I cannot say with certainty, so apologies if this is a false
> alarm... Having the java SPI available would be a huge aid to reasoning
> about the proposed SQL schema, I think.
>
> That said, I do not mean to complicate the development of this feature,
> just trying to structure it in a way that is hopefully easy to understand
> (including myself), maintain and expand later. If I missed something,
> please feel free to point it out.
>
> [3385] https://urldefense.com/v3/__https://github.com/apache/polaris/pull/3385__;!!Iz9xO38YGHZK!65BVFQDe_GtKbDW8LkNdzVS-WYewzWsSns5zFHKclbMU3vUy2UaTXLB5G-wql1yyqoTFyDc9Bt2W-eKl7A$
> <https://urldefense.com/v3/__https://github.com/apache/polaris/pull/3385__;!!Iz9xO38YGHZK!_dbBfwEnZEuPp_8qIuBDoYfWRgCZKiPDMg9ouDltqsHhr9hqCHVlF9IedSkNVBDWupbneU7inC9YK-THL-lTjWkkyZhI0OE$>
>
>Thanks,
> Dmitri.
>
> On Wed, Jan 14, 2026 at 10:05 PM Anand Kumar Sankaran via dev <
> dev@polaris.apache.org<ma...@polaris.apache.org>>>> wrote:
>
> Hi Yufei,
>
> Gave commenting privileges to all.
>
> Thanks.
>
> —
> Anand
>
> From: Yufei Gu <fl...@gmail.com>>>>
> Date: Wednesday, January 14, 2026 at 6:13 PM
> To: dev@polaris.apache.org<ma...@polaris.apache.org>>> <de...@polaris.apache.org>>>>
> Cc: Anand Kumar Sankaran <an...@workday.com>>>>
> Subject: Re: Polaris Telemetry and Audit Trail
>
> This Message Is From an External Sender
> This message came from outside your organization.
> Report Suspicious<
> https://us-phishalarm-ewt.proofpoint.com/EWT/v1/Iz9xO38YGHZK!YhNDZABkHi1B6hyPVYUK0T-K5j4Aoqdrlu07UjBpJcpjOsz7Ie7d_DhPl-ywi50dfQz4O6LL-AHLCOQTQ0X7Le8yD2cF0ogquGFB8Aw8uQLgB0RL6Ezpa-pLDPA4ziXA$
> >
>
> Hi Anand, thanks for the design doc and PR. Can you allow comments in the
> doc so that people can chime in? Thanks!
>
> Yufei
>
>
> On Sat, Jan 10, 2026 at 9:09 AM Anand Kumar Sankaran via dev <
> dev@polaris.apache.org<ma...@polaris.apache.org>>>>> wrote:
> Hi all
>
> My first PR for adding AWS STS Session Tags support for credential vending
> was merged. https://urldefense.com/v3/__https://github.com/apache/polaris/pull/3327__;!!Iz9xO38YGHZK!65BVFQDe_GtKbDW8LkNdzVS-WYewzWsSns5zFHKclbMU3vUy2UaTXLB5G-wql1yyqoTFyDc9Bt0YHY9sdA$
> <https://urldefense.com/v3/__https://github.com/apache/polaris/pull/3327__;!!Iz9xO38YGHZK!_dbBfwEnZEuPp_8qIuBDoYfWRgCZKiPDMg9ouDltqsHhr9hqCHVlF9IedSkNVBDWupbneU7inC9YK-THL-lTjWkklJ4Xf9Y$>
><
> https://urldefense.com/v3/__https://github.com/apache/polaris/pull/3327__;!!Iz9xO38YGHZK!8vlQ2_OqjmZNIpbrTTluFFNKFCEeM87nr95MP5pMpMu5fFWVSoBZ3eb6-C2xbAGCG-ry5eaCO-pFLxl3IZAJGqY$
>>
>
> I have been working with Prashant on an end-to-end telemetry and audit
> trail tracking for Polaris.  It is documented here.
> https://urldefense.com/v3/__https://docs.google.com/document/d/1Ehzvi5RNPs4hChkBFI6VD23myEqm-7sWW3d2kjmuYj8/edit?tab=t.0__;!!Iz9xO38YGHZK!65BVFQDe_GtKbDW8LkNdzVS-WYewzWsSns5zFHKclbMU3vUy2UaTXLB5G-wql1yyqoTFyDc9Bt11Y1rbNQ$
> <https://urldefense.com/v3/__https://docs.google.com/document/d/1Ehzvi5RNPs4hChkBFI6VD23myEqm-7sWW3d2kjmuYj8/edit?tab=t.0__;!!Iz9xO38YGHZK!_dbBfwEnZEuPp_8qIuBDoYfWRgCZKiPDMg9ouDltqsHhr9hqCHVlF9IedSkNVBDWupbneU7inC9YK-THL-lTjWkk8MVJPlY$>
><
> https://urldefense.com/v3/__https://docs.google.com/document/d/1Ehzvi5RNPs4hChkBFI6VD23myEqm-7sWW3d2kjmuYj8/edit?tab=t.0__;!!Iz9xO38YGHZK!8vlQ2_OqjmZNIpbrTTluFFNKFCEeM87nr95MP5pMpMu5fFWVSoBZ3eb6-C2xbAGCG-ry5eaCO-pFLxl3hYqTlcg$
>>
>
> Based on Prashant’s initial feedback (parity with Apache Gravitino for
> metrics reports), I have an initial PR here.
> https://urldefense.com/v3/__https://github.com/apache/polaris/pull/3385__;!!Iz9xO38YGHZK!65BVFQDe_GtKbDW8LkNdzVS-WYewzWsSns5zFHKclbMU3vUy2UaTXLB5G-wql1yyqoTFyDc9Bt2W-eKl7A$
> <https://urldefense.com/v3/__https://github.com/apache/polaris/pull/3385__;!!Iz9xO38YGHZK!_dbBfwEnZEuPp_8qIuBDoYfWRgCZKiPDMg9ouDltqsHhr9hqCHVlF9IedSkNVBDWupbneU7inC9YK-THL-lTjWkkyZhI0OE$>
><
> https://urldefense.com/v3/__https://github.com/apache/polaris/pull/3385__;!!Iz9xO38YGHZK!8vlQ2_OqjmZNIpbrTTluFFNKFCEeM87nr95MP5pMpMu5fFWVSoBZ3eb6-C2xbAGCG-ry5eaCO-pFLxl39opCznI$
>>
>
> I will be reaching out to various folks for review. I am open to breaking
> down the PR into smaller logical groups of PR if it helps.
>
> I request you to look at the Google doc and let me know what you think.
>
> —
> Anand
>
>
>
> --
> Dmitri Bourlatchkov
> Senior Staff Software Engineer, Dremio
> Dremio.com
> <https://urldefense.com/v3/__https://www.dremio.com/?utm_medium=email&utm_source=signature&utm_term=na&utm_content=email-signature&utm_campaign=email-signature__;!!Iz9xO38YGHZK!_dbBfwEnZEuPp_8qIuBDoYfWRgCZKiPDMg9ouDltqsHhr9hqCHVlF9IedSkNVBDWupbneU7inC9YK-THL-lTjWkkiOwjJc0$>/
> Follow Us on LinkedIn
> <https://urldefense.com/v3/__https://www.linkedin.com/company/dremio__;!!Iz9xO38YGHZK!_dbBfwEnZEuPp_8qIuBDoYfWRgCZKiPDMg9ouDltqsHhr9hqCHVlF9IedSkNVBDWupbneU7inC9YK-THL-lTjWkkCLjBQ2M$>/
> Get Started
> <https://urldefense.com/v3/__https://www.dremio.com/get-started/__;!!Iz9xO38YGHZK!_dbBfwEnZEuPp_8qIuBDoYfWRgCZKiPDMg9ouDltqsHhr9hqCHVlF9IedSkNVBDWupbneU7inC9YK-THL-lTjWkkoKQUMII$>
>
>
>The Agentic Lakehouse
> *The only lakehouse built for agents, managed by agents*
>
>
>

Re: Polaris Telemetry and Audit Trail

[Dmitri Bourlatchkov <di...@apache.org>]

Hi Anand,

PR [3523] (SQL schema) has been merged.

Please rebase [3385].

[3523] https://github.com/apache/polaris/pull/3523

[3385] https://github.com/apache/polaris/pull/3385

Thanks,
Dmitri.

On Fri, Feb 13, 2026 at 6:10 PM Dmitri Bourlatchkov <di...@apache.org>
wrote:

> Hi All,
>
> The SPI PR [3616] has been merged.
>
> I propose to merge the Schema PR [3523] on Mon if there are no objections
> or fresh comments.
>
> [3616] https://github.com/apache/polaris/pull/3616
>
> [3523] https://github.com/apache/polaris/pull/3523
>
> Cheers,
> Dmitri.
>
>
> On Thu, Feb 12, 2026 at 6:00 PM Dmitri Bourlatchkov <di...@apache.org>
> wrote:
>
>> Hi All,
>>
>> The SPI PR [3616] for Scan/Commit metrics has been in review for 2 weeks
>> including 3 days in an approved state.
>>
>> Unless there are any last minute comments, I propose to merge it tomorrow
>> (Feb 13).
>>
>> [3616] https://github.com/apache/polaris/pull/3616
>>
>> Cheers,
>> Dmitri.
>>
>> On Mon, Feb 9, 2026 at 2:08 PM Anand Kumar Sankaran via dev <
>> dev@polaris.apache.org> wrote:
>>
>>> Dmitri and others,
>>>
>>> Thanks for all the review feedback and being patient while we iterated.
>>> My initial proposal was to persist the table metrics as events (least
>>> amount of work 😊), we have come far from that proposal.
>>>
>>> To detach the SPI fully, there are a few things to do
>>> (loadSchemaVersion() and shared DatasourceOperations). We can add a
>>> separate datasource config to specify a different connection pool, perhaps.
>>> Can be done as a follow up PR.
>>>
>>> —
>>> Anand
>>>
>>> From: Dmitri Bourlatchkov <di...@apache.org>
>>> Date: Monday, February 9, 2026 at 10:52 AM
>>> To: dev@polaris.apache.org <de...@polaris.apache.org>
>>> Cc: Anand Kumar Sankaran <an...@workday.com>
>>> Subject: Re: Polaris Telemetry and Audit Trail
>>>
>>> This Message Is From an External Sender
>>> This message came from outside your organization.
>>> Report Suspicious<
>>> https://us-phishalarm-ewt.proofpoint.com/EWT/v1/Iz9xO38YGHZK!YhNDZAGr2cumYduJ20ONMhMxljYPF8uZOe6R8BpEWyJ7tSG5GUUf8our05nYHWov0t1Vi7Rx_O8JuRNBBQQ57aBoctrR0wH7VSR9VxRuqYJPpz0q6I0JboPRfdLIwqal$
>>> >
>>>
>>> Hi Anand and all,
>>>
>>> Thanks for pushing this feature forward! From my POV the SPI [3616] and
>>> the schema [3523] PRs are good to merge as the initial implementation of
>>> this feature (annotated as "beta"). The way is open to the JDBC impl. PR
>>> [3385] (which will probably need a rebase after merging the SPI and schema
>>> code).
>>>
>>> I'm sending this email to highlight the direction taken with respect to
>>> the SQL schema in this feature, as it might have been visible enough to the
>>> broader community during the PR reviews.
>>>
>>> * The scan / commit metrics schema is managed / versioned separately
>>> from the JDBC MetaStore schema
>>>
>>> * The Metrics Persistence SPI is detached (mostly at this time, but
>>> supposedly fully detached later) from the JDBC MetaStore
>>>
>>> * The JDBC DataSource for metrics may be the same or different from the
>>> MetaStore datasource.
>>>
>>> * Metrics Persistence may have non-JDBC implementation in the future
>>> (not implemented ATM)
>>>
>>> * JDBC Metrics Persistence may co-exist with other MetaStore
>>> implementations.
>>>
>>> I believe this design allows great flexibility for end users to try this
>>> feature out and keeps options open for future enhancements in Scan / Commit
>>> Metrics in Polaris.
>>>
>>> [3616] https://github.com/apache/polaris/pull/3616<
>>> https://urldefense.com/v3/__https://github.com/apache/polaris/pull/3616__;!!Iz9xO38YGHZK!7IwwfPXDN0ThhNZgol-n9Ww3FFLKcpZbSQ17lWEtcQnTHtJM9aVXnqGW99_6vmVkWACy7rYC8XEpN6Okdg$
>>> >
>>>
>>> [3523] https://github.com/apache/polaris/pull/3523<
>>> https://urldefense.com/v3/__https://github.com/apache/polaris/pull/3523__;!!Iz9xO38YGHZK!7IwwfPXDN0ThhNZgol-n9Ww3FFLKcpZbSQ17lWEtcQnTHtJM9aVXnqGW99_6vmVkWACy7rYC8XGOCHKhSg$
>>> >
>>>
>>> [3385] https://github.com/apache/polaris/pull/3385<
>>> https://urldefense.com/v3/__https://github.com/apache/polaris/pull/3385__;!!Iz9xO38YGHZK!7IwwfPXDN0ThhNZgol-n9Ww3FFLKcpZbSQ17lWEtcQnTHtJM9aVXnqGW99_6vmVkWACy7rYC8XFlxLMhvQ$
>>> >
>>>
>>> Thoughts / comments?
>>>
>>> Cheers,
>>> Dmitri.
>>>
>>>
>>> On Thu, Feb 5, 2026 at 6:50 PM Anand Kumar Sankaran via dev <
>>> dev@polaris.apache.org<ma...@polaris.apache.org>> wrote:
>>> Hi Dmitri,
>>>
>>> Following up on your feedback regarding the metrics schema separation
>>> approach. I've implemented the simplified design you suggested in
>>> https://github.com/apache/polaris/pull/3523<
>>> https://urldefense.com/v3/__https://github.com/apache/polaris/pull/3523__;!!Iz9xO38YGHZK!7IwwfPXDN0ThhNZgol-n9Ww3FFLKcpZbSQ17lWEtcQnTHtJM9aVXnqGW99_6vmVkWACy7rYC8XGOCHKhSg$
>>> >:
>>>
>>>
>>>  The metrics schema can now evolve independently from the entity schema.
>>> Let me know if you'd like any changes to this approach.
>>>
>>>  Thanks,
>>>   Anand
>>>
>>>
>>>
>>> From: Dmitri Bourlatchkov <di...@apache.org>>
>>> Date: Thursday, February 5, 2026 at 2:57 PM
>>> To: dev@polaris.apache.org<ma...@polaris.apache.org> <
>>> dev@polaris.apache.org<ma...@polaris.apache.org>>
>>> Cc: Anand Kumar Sankaran <anand.sankaran@workday.com<mailto:
>>> anand.sankaran@workday.com>>
>>> Subject: Re: Polaris Telemetry and Audit Trail
>>>
>>> This Message Is From an External Sender
>>> This message came from outside your organization.
>>> Report Suspicious<
>>> https://us-phishalarm-ewt.proofpoint.com/EWT/v1/Iz9xO38YGHZK!YhNDZAGr2cumY1iAus5GNbkIxSXoQaI552sXgYyv7Z9Wrvb7qeTskSy5WkXTVDJ_Bel2AbNh_fQVn-HS_UtOSdOtpsNfzhXDkK4fYWdTeoeBe70A6oQVGpAYT3SrbEvs$
>>> >
>>>
>>> Hi Anand,
>>>
>>> Re: MetricsPersistence Factory - TBH I do not think it is required. If
>>> we go with an MetricsPersistence impl. independent of the JDBC MetaStore,
>>> it could be injected into PersistingMetricsReporter directly by CDI.
>>>
>>> Re: DataSource - for a start we could probably reuse the same DataSource
>>> as the JDBS MetaStore uses. In principle, we could introduce a separate
>>> DataSource, but I do not think we need to complicate the initial
>>> implementation with that. It can be added later, if required.
>>>
>>> Re: schema / bootstrap - I'm thinking, indeed, of having a separate
>>> *.sql file for metrics DDL. Then we could add an option to the bootstrap
>>> command in the Admin Tool, or even a new command for creating / updating
>>> the metrics schema. This might require some refactoring, but I believe the
>>> bootstrap code needs some cleanup anyway.
>>>
>>> The general idea is to allow the metrics schema to evolve independently.
>>> It's a new and actively developed feature, so having flexibility is going
>>> to be good for making progress / improvements, I hope.
>>>
>>> Does that sound reasonable?
>>>
>>> Thanks,
>>> Dmitri.
>>>
>>> On Thu, Feb 5, 2026 at 3:47 PM Anand Kumar Sankaran via dev <
>>> dev@polaris.apache.org<ma...@polaris.apache.org><mailto:
>>> dev@polaris.apache.org<ma...@polaris.apache.org>>> wrote:
>>> Hi Dmitri
>>>
>>> The SPI is designed to be pluggable via
>>> MetaStoreManagerFactory.getOrCreateMetricsPersistence()<
>>> https://github.com/apache/polaris/pull/3385/changes#diff-0e34667d083d30cace067936b87c736201e5a8cb001aa0b12e203ee52cddae99R51
>>> <
>>> https://urldefense.com/v3/__https://github.com/apache/polaris/pull/3385/changes*diff-0e34667d083d30cace067936b87c736201e5a8cb001aa0b12e203ee52cddae99R51__;Iw!!Iz9xO38YGHZK!7IwwfPXDN0ThhNZgol-n9Ww3FFLKcpZbSQ17lWEtcQnTHtJM9aVXnqGW99_6vmVkWACy7rYC8XFxRr4qRw$
>>> ><
>>> https://urldefense.com/v3/__https://github.com/apache/polaris/pull/3385/changes*diff-0e34667d083d30cace067936b87c736201e5a8cb001aa0b12e203ee52cddae99R51__;Iw!!Iz9xO38YGHZK!_K1mcH0_2x0nbU-mZlbBi3tk9QAGjB6VInPiqkQU21mpIP_ft2bXXyIiADNoAcotNr9AKqlphh65taUhcw$>>.
>>> The default returns MetricsPersistence.NOOP, and only JDBC with schema v4+
>>> returns a functional implementation.
>>>
>>>  Your proposal makes sense to me. To allow NoSQL Entity Persistence +
>>> JDBC Metrics Persistence would require:
>>>
>>>      1. Separate metrics schema files (e.g., metrics-schema-v1.sql) that
>>> can be bootstrapped independently
>>>      2. Separate datasource configuration for metrics (e.g.,
>>> polaris.metrics.jdbc.*)
>>>      3. A dedicated `MetricsPersistenceFactory` interface (separate from
>>> MetaStoreManagerFactory)
>>>      4. Updates to the Admin Tool bootstrap workflow to handle both
>>> schemas
>>>
>>> Does that sound reasonable?  If we do this, all this work should go into
>>> https://github.com/apache/polaris/pull/3523<
>>> https://urldefense.com/v3/__https://github.com/apache/polaris/pull/3523__;!!Iz9xO38YGHZK!7IwwfPXDN0ThhNZgol-n9Ww3FFLKcpZbSQ17lWEtcQnTHtJM9aVXnqGW99_6vmVkWACy7rYC8XGOCHKhSg$
>>> ><
>>> https://urldefense.com/v3/__https://github.com/apache/polaris/pull/3523__;!!Iz9xO38YGHZK!_K1mcH0_2x0nbU-mZlbBi3tk9QAGjB6VInPiqkQU21mpIP_ft2bXXyIiADNoAcotNr9AKqlphh4ZISZcDQ$>
>>> as one atomic commit, correct?
>>>
>>> —
>>> Anand
>>>
>>>
>>>
>>> From: Dmitri Bourlatchkov <dimas@apache.org<mailto:dimas@apache.org
>>> ><ma...@apache.org>>>
>>> Date: Thursday, February 5, 2026 at 8:28 AM
>>> To: Anand Kumar Sankaran <anand.sankaran@workday.com<mailto:
>>> anand.sankaran@workday.com><mailto:anand.sankaran@workday.com<mailto:
>>> anand.sankaran@workday.com>>>
>>> Cc: dev@polaris.apache.org<ma...@polaris.apache.org><mailto:
>>> dev@polaris.apache.org<ma...@polaris.apache.org>> <
>>> dev@polaris.apache.org<ma...@polaris.apache.org><mailto:
>>> dev@polaris.apache.org<ma...@polaris.apache.org>>>
>>> Subject: Re: Polaris Telemetry and Audit Trail
>>>
>>> This Message Is From an External Sender
>>> This message came from outside your organization.
>>> Report Suspicious<
>>> https://us-phishalarm-ewt.proofpoint.com/EWT/v1/Iz9xO38YGHZK!YhNDZAGr2cumbRcPlIOttKwvGr4f38TF-J2-LmOw8H1KsFpWaM3W5g28Oq5qW2gnw2-WHz0rsWhm7F7hSGykA9l01LKYGuCcCkexkHp1sUssM_IegCtjk_id9s891orv$
>>> >
>>>
>>>
>>> Hi Anand,
>>>
>>> Thanks for driving the SPI definition for scan/commits metrics.
>>>
>>> I did not think from this angle before, but now I suppose it might be
>>> worth
>>> introducing a separate SQL schema for metrics, detached from the main
>>> JDBC
>>> Persistence schema.
>>>
>>> My rationale is that the SPI may be exercised independently of the JDBC
>>> Entity Persistence. I can imagine people using NoSQL Entity Persistence
>>> plus JDBC Scan Metrics Persistence in the same deployment.
>>>
>>> Using a separate schema will add further work on the Bootstrap workflow
>>> (Admin Tool), but I hope the end result is going to be more convenient
>>> and
>>> versatile for end users.
>>>
>>> WDYT?
>>>
>>> Thanks,
>>> Dmitri.
>>>
>>> On Thu, Jan 29, 2026 at 1:49 PM Anand Kumar Sankaran <
>>> anand.sankaran@workday.com<ma...@workday.com><mailto:
>>> anand.sankaran@workday.com<ma...@workday.com>>> wrote:
>>>
>>> > Hi Dmitri,
>>> >
>>> > Thank you for taking the time to review. I had pulled out the schema in
>>> > its own PR (based on comments) here:
>>> >
>>> https://urldefense.com/v3/__https://github.com/apache/polaris/pull/3523__;!!Iz9xO38YGHZK!65BVFQDe_GtKbDW8LkNdzVS-WYewzWsSns5zFHKclbMU3vUy2UaTXLB5G-wql1yyqoTFyDc9Bt3GRsh6AA$
>>> >
>>> > I will update the document with the SPI and carve out another PR that
>>> > focuses on the SPI.
>>> >
>>> > —
>>> > Anand
>>> >
>>> > *From: *Dmitri Bourlatchkov <dmitri.bourlatchkov@dremio.com<mailto:
>>> dmitri.bourlatchkov@dremio.com><mailto:dmitri.bourlatchkov@dremio.com
>>> <ma...@dremio.com>>>
>>> > *Date: *Thursday, January 29, 2026 at 10:43 AM
>>> > *To: *dev@polaris.apache.org<ma...@polaris.apache.org><mailto:
>>> dev@polaris.apache.org<ma...@polaris.apache.org>> <
>>> dev@polaris.apache.org<ma...@polaris.apache.org><mailto:
>>> dev@polaris.apache.org<ma...@polaris.apache.org>>>
>>> > *Cc: *Anand Kumar Sankaran <anand.sankaran@workday.com<mailto:
>>> anand.sankaran@workday.com><mailto:anand.sankaran@workday.com<mailto:
>>> anand.sankaran@workday.com>>>
>>> > *Subject: *Re: Polaris Telemetry and Audit Trail
>>> >
>>> > This Message Is From an External Sender
>>> > This message came from outside your organization.
>>> > Report Suspicious
>>> > <
>>> https://us-phishalarm-ewt.proofpoint.com/EWT/v1/Iz9xO38YGHZK!YhNDZAGr2cumY1cAlUeoPrU2eNUPz9Wxq2PaG1t99vEN3v60gucxC3Qsk-xK09RndcW46WtNjMMjQZbaQ495h1D1g293F_uqLhPM8gDMlA6xRmo5b5uqrSh2v7QKHrhQ$
>>> >
>>> >
>>> > Hi Anand,
>>> >
>>> > Thanks for making a proposal doc and starting this thread. Apologies
>>> for
>>> > late comments.
>>> >
>>> > As you know I already reviewed a couple of related PRs, which had a
>>> > smaller impact on the codebase, but now, I believe, we're coming to the
>>> > core of this feature and it might need a deeper discussion.
>>> >
>>> > For the sake of awareness of other project members, I'd like to
>>> > highlight some aspects by email.
>>> >
>>> > * Storing scan metrics in RDBMS in general looks like a
>>> > reasonable approach to me. However, Polaris Persistence is an
>>> extensible
>>> > mechanism. Other backends are possible, including privately developed
>>> > backends.
>>> >
>>> > From this POV, I believe it might be preferable to start not with an
>>> RDBMS
>>> > schema, but with a java SPI outlining expected read/write operations
>>> and
>>> > the data model. This will make it easier to assess impact on other
>>> > Persistence implementations. Naturally, Polaris service code will have
>>> to
>>> > be able to work seamlessly regardless of the backend impl. (including
>>> the
>>> > do-nothing impl.).
>>> >
>>> > I see that some connection between the SQL schema and java code is
>>> made in
>>> > PR [3385], but it does not appear to offer a clean SPI that could be
>>> > implemented by different Persistence backends. Would it be ok from your
>>> > perspective to make another PR with just SPI code?
>>> >
>>> > * Regarding ad-hoc SQL queries - it is certainly a valid use case for
>>> > custom code to query the Polaris database directly. However, those
>>> ad-hoc
>>> > queries may require certain indexes, which may not have a relevant use
>>> case
>>> > in OSS code. In that case, I believe it would be preferable to manage
>>> the
>>> > extra indexes in custom code (not in Polaris). Whether this is the
>>> case or
>>> > not currently, I cannot say with certainty, so apologies if this is a
>>> false
>>> > alarm... Having the java SPI available would be a huge aid to reasoning
>>> > about the proposed SQL schema, I think.
>>> >
>>> > That said, I do not mean to complicate the development of this feature,
>>> > just trying to structure it in a way that is hopefully easy to
>>> understand
>>> > (including myself), maintain and expand later. If I missed something,
>>> > please feel free to point it out.
>>> >
>>> > [3385]
>>> https://urldefense.com/v3/__https://github.com/apache/polaris/pull/3385__;!!Iz9xO38YGHZK!65BVFQDe_GtKbDW8LkNdzVS-WYewzWsSns5zFHKclbMU3vUy2UaTXLB5G-wql1yyqoTFyDc9Bt2W-eKl7A$
>>> > <
>>> https://urldefense.com/v3/__https://github.com/apache/polaris/pull/3385__;!!Iz9xO38YGHZK!_dbBfwEnZEuPp_8qIuBDoYfWRgCZKiPDMg9ouDltqsHhr9hqCHVlF9IedSkNVBDWupbneU7inC9YK-THL-lTjWkkyZhI0OE$
>>> >
>>> >
>>> >Thanks,
>>> > Dmitri.
>>> >
>>> > On Wed, Jan 14, 2026 at 10:05 PM Anand Kumar Sankaran via dev <
>>> > dev@polaris.apache.org<ma...@polaris.apache.org><mailto:
>>> dev@polaris.apache.org<ma...@polaris.apache.org>>> wrote:
>>> >
>>> > Hi Yufei,
>>> >
>>> > Gave commenting privileges to all.
>>> >
>>> > Thanks.
>>> >
>>> > —
>>> > Anand
>>> >
>>> > From: Yufei Gu <flyrain000@gmail.com<mailto:flyrain000@gmail.com
>>> ><ma...@gmail.com>>>
>>> > Date: Wednesday, January 14, 2026 at 6:13 PM
>>> > To: dev@polaris.apache.org<ma...@polaris.apache.org><mailto:
>>> dev@polaris.apache.org<ma...@polaris.apache.org>> <
>>> dev@polaris.apache.org<ma...@polaris.apache.org><mailto:
>>> dev@polaris.apache.org<ma...@polaris.apache.org>>>
>>> > Cc: Anand Kumar Sankaran <anand.sankaran@workday.com<mailto:
>>> anand.sankaran@workday.com><mailto:anand.sankaran@workday.com<mailto:
>>> anand.sankaran@workday.com>>>
>>> > Subject: Re: Polaris Telemetry and Audit Trail
>>> >
>>> > This Message Is From an External Sender
>>> > This message came from outside your organization.
>>> > Report Suspicious<
>>> >
>>> https://us-phishalarm-ewt.proofpoint.com/EWT/v1/Iz9xO38YGHZK!YhNDZABkHi1B6hyPVYUK0T-K5j4Aoqdrlu07UjBpJcpjOsz7Ie7d_DhPl-ywi50dfQz4O6LL-AHLCOQTQ0X7Le8yD2cF0ogquGFB8Aw8uQLgB0RL6Ezpa-pLDPA4ziXA$
>>> > >
>>> >
>>> > Hi Anand, thanks for the design doc and PR. Can you allow comments in
>>> the
>>> > doc so that people can chime in? Thanks!
>>> >
>>> > Yufei
>>> >
>>> >
>>> > On Sat, Jan 10, 2026 at 9:09 AM Anand Kumar Sankaran via dev <
>>> > dev@polaris.apache.org<ma...@polaris.apache.org><mailto:
>>> dev@polaris.apache.org<ma...@polaris.apache.org>><mailto:
>>> dev@polaris.apache.org<ma...@polaris.apache.org><mailto:
>>> dev@polaris.apache.org<ma...@polaris.apache.org>>>> wrote:
>>> > Hi all
>>> >
>>> > My first PR for adding AWS STS Session Tags support for credential
>>> vending
>>> > was merged.
>>> https://urldefense.com/v3/__https://github.com/apache/polaris/pull/3327__;!!Iz9xO38YGHZK!65BVFQDe_GtKbDW8LkNdzVS-WYewzWsSns5zFHKclbMU3vUy2UaTXLB5G-wql1yyqoTFyDc9Bt0YHY9sdA$
>>> > <
>>> https://urldefense.com/v3/__https://github.com/apache/polaris/pull/3327__;!!Iz9xO38YGHZK!_dbBfwEnZEuPp_8qIuBDoYfWRgCZKiPDMg9ouDltqsHhr9hqCHVlF9IedSkNVBDWupbneU7inC9YK-THL-lTjWkklJ4Xf9Y$
>>> >
>>> ><
>>> >
>>> https://urldefense.com/v3/__https://github.com/apache/polaris/pull/3327__;!!Iz9xO38YGHZK!8vlQ2_OqjmZNIpbrTTluFFNKFCEeM87nr95MP5pMpMu5fFWVSoBZ3eb6-C2xbAGCG-ry5eaCO-pFLxl3IZAJGqY$
>>> >>
>>> >
>>> > I have been working with Prashant on an end-to-end telemetry and audit
>>> > trail tracking for Polaris.  It is documented here.
>>> >
>>> https://urldefense.com/v3/__https://docs.google.com/document/d/1Ehzvi5RNPs4hChkBFI6VD23myEqm-7sWW3d2kjmuYj8/edit?tab=t.0__;!!Iz9xO38YGHZK!65BVFQDe_GtKbDW8LkNdzVS-WYewzWsSns5zFHKclbMU3vUy2UaTXLB5G-wql1yyqoTFyDc9Bt11Y1rbNQ$
>>> > <
>>> https://urldefense.com/v3/__https://docs.google.com/document/d/1Ehzvi5RNPs4hChkBFI6VD23myEqm-7sWW3d2kjmuYj8/edit?tab=t.0__;!!Iz9xO38YGHZK!_dbBfwEnZEuPp_8qIuBDoYfWRgCZKiPDMg9ouDltqsHhr9hqCHVlF9IedSkNVBDWupbneU7inC9YK-THL-lTjWkk8MVJPlY$
>>> >
>>> ><
>>> >
>>> https://urldefense.com/v3/__https://docs.google.com/document/d/1Ehzvi5RNPs4hChkBFI6VD23myEqm-7sWW3d2kjmuYj8/edit?tab=t.0__;!!Iz9xO38YGHZK!8vlQ2_OqjmZNIpbrTTluFFNKFCEeM87nr95MP5pMpMu5fFWVSoBZ3eb6-C2xbAGCG-ry5eaCO-pFLxl3hYqTlcg$
>>> >>
>>> >
>>> > Based on Prashant’s initial feedback (parity with Apache Gravitino for
>>> > metrics reports), I have an initial PR here.
>>> >
>>> https://urldefense.com/v3/__https://github.com/apache/polaris/pull/3385__;!!Iz9xO38YGHZK!65BVFQDe_GtKbDW8LkNdzVS-WYewzWsSns5zFHKclbMU3vUy2UaTXLB5G-wql1yyqoTFyDc9Bt2W-eKl7A$
>>> > <
>>> https://urldefense.com/v3/__https://github.com/apache/polaris/pull/3385__;!!Iz9xO38YGHZK!_dbBfwEnZEuPp_8qIuBDoYfWRgCZKiPDMg9ouDltqsHhr9hqCHVlF9IedSkNVBDWupbneU7inC9YK-THL-lTjWkkyZhI0OE$
>>> >
>>> ><
>>> >
>>> https://urldefense.com/v3/__https://github.com/apache/polaris/pull/3385__;!!Iz9xO38YGHZK!8vlQ2_OqjmZNIpbrTTluFFNKFCEeM87nr95MP5pMpMu5fFWVSoBZ3eb6-C2xbAGCG-ry5eaCO-pFLxl39opCznI$
>>> >>
>>> >
>>> > I will be reaching out to various folks for review. I am open to
>>> breaking
>>> > down the PR into smaller logical groups of PR if it helps.
>>> >
>>> > I request you to look at the Google doc and let me know what you think.
>>> >
>>> > —
>>> > Anand
>>> >
>>> >
>>> >
>>> > --
>>> > Dmitri Bourlatchkov
>>> > Senior Staff Software Engineer, Dremio
>>> > Dremio.com
>>> > <
>>> https://urldefense.com/v3/__https://www.dremio.com/?utm_medium=email&utm_source=signature&utm_term=na&utm_content=email-signature&utm_campaign=email-signature__;!!Iz9xO38YGHZK!_dbBfwEnZEuPp_8qIuBDoYfWRgCZKiPDMg9ouDltqsHhr9hqCHVlF9IedSkNVBDWupbneU7inC9YK-THL-lTjWkkiOwjJc0$
>>> >/
>>> > Follow Us on LinkedIn
>>> > <
>>> https://urldefense.com/v3/__https://www.linkedin.com/company/dremio__;!!Iz9xO38YGHZK!_dbBfwEnZEuPp_8qIuBDoYfWRgCZKiPDMg9ouDltqsHhr9hqCHVlF9IedSkNVBDWupbneU7inC9YK-THL-lTjWkkCLjBQ2M$
>>> >/
>>> > Get Started
>>> > <
>>> https://urldefense.com/v3/__https://www.dremio.com/get-started/__;!!Iz9xO38YGHZK!_dbBfwEnZEuPp_8qIuBDoYfWRgCZKiPDMg9ouDltqsHhr9hqCHVlF9IedSkNVBDWupbneU7inC9YK-THL-lTjWkkoKQUMII$
>>> >
>>> >
>>> >
>>> >The Agentic Lakehouse
>>> > *The only lakehouse built for agents, managed by agents*
>>> >
>>> >
>>> >
>>>
>>>

Re: Polaris Telemetry and Audit Trail

[Dmitri Bourlatchkov <di...@apache.org>]

Hi All,

The SPI PR [3616] has been merged.

I propose to merge the Schema PR [3523] on Mon if there are no objections
or fresh comments.

[3616] https://github.com/apache/polaris/pull/3616

[3523] https://github.com/apache/polaris/pull/3523

Cheers,
Dmitri.


On Thu, Feb 12, 2026 at 6:00 PM Dmitri Bourlatchkov <di...@apache.org>
wrote:

> Hi All,
>
> The SPI PR [3616] for Scan/Commit metrics has been in review for 2 weeks
> including 3 days in an approved state.
>
> Unless there are any last minute comments, I propose to merge it tomorrow
> (Feb 13).
>
> [3616] https://github.com/apache/polaris/pull/3616
>
> Cheers,
> Dmitri.
>
> On Mon, Feb 9, 2026 at 2:08 PM Anand Kumar Sankaran via dev <
> dev@polaris.apache.org> wrote:
>
>> Dmitri and others,
>>
>> Thanks for all the review feedback and being patient while we iterated.
>> My initial proposal was to persist the table metrics as events (least
>> amount of work 😊), we have come far from that proposal.
>>
>> To detach the SPI fully, there are a few things to do
>> (loadSchemaVersion() and shared DatasourceOperations). We can add a
>> separate datasource config to specify a different connection pool, perhaps.
>> Can be done as a follow up PR.
>>
>> —
>> Anand
>>
>> From: Dmitri Bourlatchkov <di...@apache.org>
>> Date: Monday, February 9, 2026 at 10:52 AM
>> To: dev@polaris.apache.org <de...@polaris.apache.org>
>> Cc: Anand Kumar Sankaran <an...@workday.com>
>> Subject: Re: Polaris Telemetry and Audit Trail
>>
>> This Message Is From an External Sender
>> This message came from outside your organization.
>> Report Suspicious<
>> https://us-phishalarm-ewt.proofpoint.com/EWT/v1/Iz9xO38YGHZK!YhNDZAGr2cumYduJ20ONMhMxljYPF8uZOe6R8BpEWyJ7tSG5GUUf8our05nYHWov0t1Vi7Rx_O8JuRNBBQQ57aBoctrR0wH7VSR9VxRuqYJPpz0q6I0JboPRfdLIwqal$
>> >
>>
>> Hi Anand and all,
>>
>> Thanks for pushing this feature forward! From my POV the SPI [3616] and
>> the schema [3523] PRs are good to merge as the initial implementation of
>> this feature (annotated as "beta"). The way is open to the JDBC impl. PR
>> [3385] (which will probably need a rebase after merging the SPI and schema
>> code).
>>
>> I'm sending this email to highlight the direction taken with respect to
>> the SQL schema in this feature, as it might have been visible enough to the
>> broader community during the PR reviews.
>>
>> * The scan / commit metrics schema is managed / versioned separately from
>> the JDBC MetaStore schema
>>
>> * The Metrics Persistence SPI is detached (mostly at this time, but
>> supposedly fully detached later) from the JDBC MetaStore
>>
>> * The JDBC DataSource for metrics may be the same or different from the
>> MetaStore datasource.
>>
>> * Metrics Persistence may have non-JDBC implementation in the future (not
>> implemented ATM)
>>
>> * JDBC Metrics Persistence may co-exist with other MetaStore
>> implementations.
>>
>> I believe this design allows great flexibility for end users to try this
>> feature out and keeps options open for future enhancements in Scan / Commit
>> Metrics in Polaris.
>>
>> [3616] https://github.com/apache/polaris/pull/3616<
>> https://urldefense.com/v3/__https://github.com/apache/polaris/pull/3616__;!!Iz9xO38YGHZK!7IwwfPXDN0ThhNZgol-n9Ww3FFLKcpZbSQ17lWEtcQnTHtJM9aVXnqGW99_6vmVkWACy7rYC8XEpN6Okdg$
>> >
>>
>> [3523] https://github.com/apache/polaris/pull/3523<
>> https://urldefense.com/v3/__https://github.com/apache/polaris/pull/3523__;!!Iz9xO38YGHZK!7IwwfPXDN0ThhNZgol-n9Ww3FFLKcpZbSQ17lWEtcQnTHtJM9aVXnqGW99_6vmVkWACy7rYC8XGOCHKhSg$
>> >
>>
>> [3385] https://github.com/apache/polaris/pull/3385<
>> https://urldefense.com/v3/__https://github.com/apache/polaris/pull/3385__;!!Iz9xO38YGHZK!7IwwfPXDN0ThhNZgol-n9Ww3FFLKcpZbSQ17lWEtcQnTHtJM9aVXnqGW99_6vmVkWACy7rYC8XFlxLMhvQ$
>> >
>>
>> Thoughts / comments?
>>
>> Cheers,
>> Dmitri.
>>
>>
>> On Thu, Feb 5, 2026 at 6:50 PM Anand Kumar Sankaran via dev <
>> dev@polaris.apache.org<ma...@polaris.apache.org>> wrote:
>> Hi Dmitri,
>>
>> Following up on your feedback regarding the metrics schema separation
>> approach. I've implemented the simplified design you suggested in
>> https://github.com/apache/polaris/pull/3523<
>> https://urldefense.com/v3/__https://github.com/apache/polaris/pull/3523__;!!Iz9xO38YGHZK!7IwwfPXDN0ThhNZgol-n9Ww3FFLKcpZbSQ17lWEtcQnTHtJM9aVXnqGW99_6vmVkWACy7rYC8XGOCHKhSg$
>> >:
>>
>>
>>  The metrics schema can now evolve independently from the entity schema.
>> Let me know if you'd like any changes to this approach.
>>
>>  Thanks,
>>   Anand
>>
>>
>>
>> From: Dmitri Bourlatchkov <di...@apache.org>>
>> Date: Thursday, February 5, 2026 at 2:57 PM
>> To: dev@polaris.apache.org<ma...@polaris.apache.org> <
>> dev@polaris.apache.org<ma...@polaris.apache.org>>
>> Cc: Anand Kumar Sankaran <anand.sankaran@workday.com<mailto:
>> anand.sankaran@workday.com>>
>> Subject: Re: Polaris Telemetry and Audit Trail
>>
>> This Message Is From an External Sender
>> This message came from outside your organization.
>> Report Suspicious<
>> https://us-phishalarm-ewt.proofpoint.com/EWT/v1/Iz9xO38YGHZK!YhNDZAGr2cumY1iAus5GNbkIxSXoQaI552sXgYyv7Z9Wrvb7qeTskSy5WkXTVDJ_Bel2AbNh_fQVn-HS_UtOSdOtpsNfzhXDkK4fYWdTeoeBe70A6oQVGpAYT3SrbEvs$
>> >
>>
>> Hi Anand,
>>
>> Re: MetricsPersistence Factory - TBH I do not think it is required. If we
>> go with an MetricsPersistence impl. independent of the JDBC MetaStore, it
>> could be injected into PersistingMetricsReporter directly by CDI.
>>
>> Re: DataSource - for a start we could probably reuse the same DataSource
>> as the JDBS MetaStore uses. In principle, we could introduce a separate
>> DataSource, but I do not think we need to complicate the initial
>> implementation with that. It can be added later, if required.
>>
>> Re: schema / bootstrap - I'm thinking, indeed, of having a separate *.sql
>> file for metrics DDL. Then we could add an option to the bootstrap command
>> in the Admin Tool, or even a new command for creating / updating the
>> metrics schema. This might require some refactoring, but I believe the
>> bootstrap code needs some cleanup anyway.
>>
>> The general idea is to allow the metrics schema to evolve independently.
>> It's a new and actively developed feature, so having flexibility is going
>> to be good for making progress / improvements, I hope.
>>
>> Does that sound reasonable?
>>
>> Thanks,
>> Dmitri.
>>
>> On Thu, Feb 5, 2026 at 3:47 PM Anand Kumar Sankaran via dev <
>> dev@polaris.apache.org<ma...@polaris.apache.org><mailto:
>> dev@polaris.apache.org<ma...@polaris.apache.org>>> wrote:
>> Hi Dmitri
>>
>> The SPI is designed to be pluggable via
>> MetaStoreManagerFactory.getOrCreateMetricsPersistence()<
>> https://github.com/apache/polaris/pull/3385/changes#diff-0e34667d083d30cace067936b87c736201e5a8cb001aa0b12e203ee52cddae99R51
>> <
>> https://urldefense.com/v3/__https://github.com/apache/polaris/pull/3385/changes*diff-0e34667d083d30cace067936b87c736201e5a8cb001aa0b12e203ee52cddae99R51__;Iw!!Iz9xO38YGHZK!7IwwfPXDN0ThhNZgol-n9Ww3FFLKcpZbSQ17lWEtcQnTHtJM9aVXnqGW99_6vmVkWACy7rYC8XFxRr4qRw$
>> ><
>> https://urldefense.com/v3/__https://github.com/apache/polaris/pull/3385/changes*diff-0e34667d083d30cace067936b87c736201e5a8cb001aa0b12e203ee52cddae99R51__;Iw!!Iz9xO38YGHZK!_K1mcH0_2x0nbU-mZlbBi3tk9QAGjB6VInPiqkQU21mpIP_ft2bXXyIiADNoAcotNr9AKqlphh65taUhcw$>>.
>> The default returns MetricsPersistence.NOOP, and only JDBC with schema v4+
>> returns a functional implementation.
>>
>>  Your proposal makes sense to me. To allow NoSQL Entity Persistence +
>> JDBC Metrics Persistence would require:
>>
>>      1. Separate metrics schema files (e.g., metrics-schema-v1.sql) that
>> can be bootstrapped independently
>>      2. Separate datasource configuration for metrics (e.g.,
>> polaris.metrics.jdbc.*)
>>      3. A dedicated `MetricsPersistenceFactory` interface (separate from
>> MetaStoreManagerFactory)
>>      4. Updates to the Admin Tool bootstrap workflow to handle both
>> schemas
>>
>> Does that sound reasonable?  If we do this, all this work should go into
>> https://github.com/apache/polaris/pull/3523<
>> https://urldefense.com/v3/__https://github.com/apache/polaris/pull/3523__;!!Iz9xO38YGHZK!7IwwfPXDN0ThhNZgol-n9Ww3FFLKcpZbSQ17lWEtcQnTHtJM9aVXnqGW99_6vmVkWACy7rYC8XGOCHKhSg$
>> ><
>> https://urldefense.com/v3/__https://github.com/apache/polaris/pull/3523__;!!Iz9xO38YGHZK!_K1mcH0_2x0nbU-mZlbBi3tk9QAGjB6VInPiqkQU21mpIP_ft2bXXyIiADNoAcotNr9AKqlphh4ZISZcDQ$>
>> as one atomic commit, correct?
>>
>> —
>> Anand
>>
>>
>>
>> From: Dmitri Bourlatchkov <dimas@apache.org<mailto:dimas@apache.org
>> ><ma...@apache.org>>>
>> Date: Thursday, February 5, 2026 at 8:28 AM
>> To: Anand Kumar Sankaran <anand.sankaran@workday.com<mailto:
>> anand.sankaran@workday.com><mailto:anand.sankaran@workday.com<mailto:
>> anand.sankaran@workday.com>>>
>> Cc: dev@polaris.apache.org<ma...@polaris.apache.org><mailto:
>> dev@polaris.apache.org<ma...@polaris.apache.org>> <
>> dev@polaris.apache.org<ma...@polaris.apache.org><mailto:
>> dev@polaris.apache.org<ma...@polaris.apache.org>>>
>> Subject: Re: Polaris Telemetry and Audit Trail
>>
>> This Message Is From an External Sender
>> This message came from outside your organization.
>> Report Suspicious<
>> https://us-phishalarm-ewt.proofpoint.com/EWT/v1/Iz9xO38YGHZK!YhNDZAGr2cumbRcPlIOttKwvGr4f38TF-J2-LmOw8H1KsFpWaM3W5g28Oq5qW2gnw2-WHz0rsWhm7F7hSGykA9l01LKYGuCcCkexkHp1sUssM_IegCtjk_id9s891orv$
>> >
>>
>>
>> Hi Anand,
>>
>> Thanks for driving the SPI definition for scan/commits metrics.
>>
>> I did not think from this angle before, but now I suppose it might be
>> worth
>> introducing a separate SQL schema for metrics, detached from the main JDBC
>> Persistence schema.
>>
>> My rationale is that the SPI may be exercised independently of the JDBC
>> Entity Persistence. I can imagine people using NoSQL Entity Persistence
>> plus JDBC Scan Metrics Persistence in the same deployment.
>>
>> Using a separate schema will add further work on the Bootstrap workflow
>> (Admin Tool), but I hope the end result is going to be more convenient and
>> versatile for end users.
>>
>> WDYT?
>>
>> Thanks,
>> Dmitri.
>>
>> On Thu, Jan 29, 2026 at 1:49 PM Anand Kumar Sankaran <
>> anand.sankaran@workday.com<ma...@workday.com><mailto:
>> anand.sankaran@workday.com<ma...@workday.com>>> wrote:
>>
>> > Hi Dmitri,
>> >
>> > Thank you for taking the time to review. I had pulled out the schema in
>> > its own PR (based on comments) here:
>> >
>> https://urldefense.com/v3/__https://github.com/apache/polaris/pull/3523__;!!Iz9xO38YGHZK!65BVFQDe_GtKbDW8LkNdzVS-WYewzWsSns5zFHKclbMU3vUy2UaTXLB5G-wql1yyqoTFyDc9Bt3GRsh6AA$
>> >
>> > I will update the document with the SPI and carve out another PR that
>> > focuses on the SPI.
>> >
>> > —
>> > Anand
>> >
>> > *From: *Dmitri Bourlatchkov <dmitri.bourlatchkov@dremio.com<mailto:
>> dmitri.bourlatchkov@dremio.com><mailto:dmitri.bourlatchkov@dremio.com
>> <ma...@dremio.com>>>
>> > *Date: *Thursday, January 29, 2026 at 10:43 AM
>> > *To: *dev@polaris.apache.org<ma...@polaris.apache.org><mailto:
>> dev@polaris.apache.org<ma...@polaris.apache.org>> <
>> dev@polaris.apache.org<ma...@polaris.apache.org><mailto:
>> dev@polaris.apache.org<ma...@polaris.apache.org>>>
>> > *Cc: *Anand Kumar Sankaran <anand.sankaran@workday.com<mailto:
>> anand.sankaran@workday.com><mailto:anand.sankaran@workday.com<mailto:
>> anand.sankaran@workday.com>>>
>> > *Subject: *Re: Polaris Telemetry and Audit Trail
>> >
>> > This Message Is From an External Sender
>> > This message came from outside your organization.
>> > Report Suspicious
>> > <
>> https://us-phishalarm-ewt.proofpoint.com/EWT/v1/Iz9xO38YGHZK!YhNDZAGr2cumY1cAlUeoPrU2eNUPz9Wxq2PaG1t99vEN3v60gucxC3Qsk-xK09RndcW46WtNjMMjQZbaQ495h1D1g293F_uqLhPM8gDMlA6xRmo5b5uqrSh2v7QKHrhQ$
>> >
>> >
>> > Hi Anand,
>> >
>> > Thanks for making a proposal doc and starting this thread. Apologies for
>> > late comments.
>> >
>> > As you know I already reviewed a couple of related PRs, which had a
>> > smaller impact on the codebase, but now, I believe, we're coming to the
>> > core of this feature and it might need a deeper discussion.
>> >
>> > For the sake of awareness of other project members, I'd like to
>> > highlight some aspects by email.
>> >
>> > * Storing scan metrics in RDBMS in general looks like a
>> > reasonable approach to me. However, Polaris Persistence is an extensible
>> > mechanism. Other backends are possible, including privately developed
>> > backends.
>> >
>> > From this POV, I believe it might be preferable to start not with an
>> RDBMS
>> > schema, but with a java SPI outlining expected read/write operations and
>> > the data model. This will make it easier to assess impact on other
>> > Persistence implementations. Naturally, Polaris service code will have
>> to
>> > be able to work seamlessly regardless of the backend impl. (including
>> the
>> > do-nothing impl.).
>> >
>> > I see that some connection between the SQL schema and java code is made
>> in
>> > PR [3385], but it does not appear to offer a clean SPI that could be
>> > implemented by different Persistence backends. Would it be ok from your
>> > perspective to make another PR with just SPI code?
>> >
>> > * Regarding ad-hoc SQL queries - it is certainly a valid use case for
>> > custom code to query the Polaris database directly. However, those
>> ad-hoc
>> > queries may require certain indexes, which may not have a relevant use
>> case
>> > in OSS code. In that case, I believe it would be preferable to manage
>> the
>> > extra indexes in custom code (not in Polaris). Whether this is the case
>> or
>> > not currently, I cannot say with certainty, so apologies if this is a
>> false
>> > alarm... Having the java SPI available would be a huge aid to reasoning
>> > about the proposed SQL schema, I think.
>> >
>> > That said, I do not mean to complicate the development of this feature,
>> > just trying to structure it in a way that is hopefully easy to
>> understand
>> > (including myself), maintain and expand later. If I missed something,
>> > please feel free to point it out.
>> >
>> > [3385]
>> https://urldefense.com/v3/__https://github.com/apache/polaris/pull/3385__;!!Iz9xO38YGHZK!65BVFQDe_GtKbDW8LkNdzVS-WYewzWsSns5zFHKclbMU3vUy2UaTXLB5G-wql1yyqoTFyDc9Bt2W-eKl7A$
>> > <
>> https://urldefense.com/v3/__https://github.com/apache/polaris/pull/3385__;!!Iz9xO38YGHZK!_dbBfwEnZEuPp_8qIuBDoYfWRgCZKiPDMg9ouDltqsHhr9hqCHVlF9IedSkNVBDWupbneU7inC9YK-THL-lTjWkkyZhI0OE$
>> >
>> >
>> >Thanks,
>> > Dmitri.
>> >
>> > On Wed, Jan 14, 2026 at 10:05 PM Anand Kumar Sankaran via dev <
>> > dev@polaris.apache.org<ma...@polaris.apache.org><mailto:
>> dev@polaris.apache.org<ma...@polaris.apache.org>>> wrote:
>> >
>> > Hi Yufei,
>> >
>> > Gave commenting privileges to all.
>> >
>> > Thanks.
>> >
>> > —
>> > Anand
>> >
>> > From: Yufei Gu <flyrain000@gmail.com<mailto:flyrain000@gmail.com
>> ><ma...@gmail.com>>>
>> > Date: Wednesday, January 14, 2026 at 6:13 PM
>> > To: dev@polaris.apache.org<ma...@polaris.apache.org><mailto:
>> dev@polaris.apache.org<ma...@polaris.apache.org>> <
>> dev@polaris.apache.org<ma...@polaris.apache.org><mailto:
>> dev@polaris.apache.org<ma...@polaris.apache.org>>>
>> > Cc: Anand Kumar Sankaran <anand.sankaran@workday.com<mailto:
>> anand.sankaran@workday.com><mailto:anand.sankaran@workday.com<mailto:
>> anand.sankaran@workday.com>>>
>> > Subject: Re: Polaris Telemetry and Audit Trail
>> >
>> > This Message Is From an External Sender
>> > This message came from outside your organization.
>> > Report Suspicious<
>> >
>> https://us-phishalarm-ewt.proofpoint.com/EWT/v1/Iz9xO38YGHZK!YhNDZABkHi1B6hyPVYUK0T-K5j4Aoqdrlu07UjBpJcpjOsz7Ie7d_DhPl-ywi50dfQz4O6LL-AHLCOQTQ0X7Le8yD2cF0ogquGFB8Aw8uQLgB0RL6Ezpa-pLDPA4ziXA$
>> > >
>> >
>> > Hi Anand, thanks for the design doc and PR. Can you allow comments in
>> the
>> > doc so that people can chime in? Thanks!
>> >
>> > Yufei
>> >
>> >
>> > On Sat, Jan 10, 2026 at 9:09 AM Anand Kumar Sankaran via dev <
>> > dev@polaris.apache.org<ma...@polaris.apache.org><mailto:
>> dev@polaris.apache.org<ma...@polaris.apache.org>><mailto:
>> dev@polaris.apache.org<ma...@polaris.apache.org><mailto:
>> dev@polaris.apache.org<ma...@polaris.apache.org>>>> wrote:
>> > Hi all
>> >
>> > My first PR for adding AWS STS Session Tags support for credential
>> vending
>> > was merged.
>> https://urldefense.com/v3/__https://github.com/apache/polaris/pull/3327__;!!Iz9xO38YGHZK!65BVFQDe_GtKbDW8LkNdzVS-WYewzWsSns5zFHKclbMU3vUy2UaTXLB5G-wql1yyqoTFyDc9Bt0YHY9sdA$
>> > <
>> https://urldefense.com/v3/__https://github.com/apache/polaris/pull/3327__;!!Iz9xO38YGHZK!_dbBfwEnZEuPp_8qIuBDoYfWRgCZKiPDMg9ouDltqsHhr9hqCHVlF9IedSkNVBDWupbneU7inC9YK-THL-lTjWkklJ4Xf9Y$
>> >
>> ><
>> >
>> https://urldefense.com/v3/__https://github.com/apache/polaris/pull/3327__;!!Iz9xO38YGHZK!8vlQ2_OqjmZNIpbrTTluFFNKFCEeM87nr95MP5pMpMu5fFWVSoBZ3eb6-C2xbAGCG-ry5eaCO-pFLxl3IZAJGqY$
>> >>
>> >
>> > I have been working with Prashant on an end-to-end telemetry and audit
>> > trail tracking for Polaris.  It is documented here.
>> >
>> https://urldefense.com/v3/__https://docs.google.com/document/d/1Ehzvi5RNPs4hChkBFI6VD23myEqm-7sWW3d2kjmuYj8/edit?tab=t.0__;!!Iz9xO38YGHZK!65BVFQDe_GtKbDW8LkNdzVS-WYewzWsSns5zFHKclbMU3vUy2UaTXLB5G-wql1yyqoTFyDc9Bt11Y1rbNQ$
>> > <
>> https://urldefense.com/v3/__https://docs.google.com/document/d/1Ehzvi5RNPs4hChkBFI6VD23myEqm-7sWW3d2kjmuYj8/edit?tab=t.0__;!!Iz9xO38YGHZK!_dbBfwEnZEuPp_8qIuBDoYfWRgCZKiPDMg9ouDltqsHhr9hqCHVlF9IedSkNVBDWupbneU7inC9YK-THL-lTjWkk8MVJPlY$
>> >
>> ><
>> >
>> https://urldefense.com/v3/__https://docs.google.com/document/d/1Ehzvi5RNPs4hChkBFI6VD23myEqm-7sWW3d2kjmuYj8/edit?tab=t.0__;!!Iz9xO38YGHZK!8vlQ2_OqjmZNIpbrTTluFFNKFCEeM87nr95MP5pMpMu5fFWVSoBZ3eb6-C2xbAGCG-ry5eaCO-pFLxl3hYqTlcg$
>> >>
>> >
>> > Based on Prashant’s initial feedback (parity with Apache Gravitino for
>> > metrics reports), I have an initial PR here.
>> >
>> https://urldefense.com/v3/__https://github.com/apache/polaris/pull/3385__;!!Iz9xO38YGHZK!65BVFQDe_GtKbDW8LkNdzVS-WYewzWsSns5zFHKclbMU3vUy2UaTXLB5G-wql1yyqoTFyDc9Bt2W-eKl7A$
>> > <
>> https://urldefense.com/v3/__https://github.com/apache/polaris/pull/3385__;!!Iz9xO38YGHZK!_dbBfwEnZEuPp_8qIuBDoYfWRgCZKiPDMg9ouDltqsHhr9hqCHVlF9IedSkNVBDWupbneU7inC9YK-THL-lTjWkkyZhI0OE$
>> >
>> ><
>> >
>> https://urldefense.com/v3/__https://github.com/apache/polaris/pull/3385__;!!Iz9xO38YGHZK!8vlQ2_OqjmZNIpbrTTluFFNKFCEeM87nr95MP5pMpMu5fFWVSoBZ3eb6-C2xbAGCG-ry5eaCO-pFLxl39opCznI$
>> >>
>> >
>> > I will be reaching out to various folks for review. I am open to
>> breaking
>> > down the PR into smaller logical groups of PR if it helps.
>> >
>> > I request you to look at the Google doc and let me know what you think.
>> >
>> > —
>> > Anand
>> >
>> >
>> >
>> > --
>> > Dmitri Bourlatchkov
>> > Senior Staff Software Engineer, Dremio
>> > Dremio.com
>> > <
>> https://urldefense.com/v3/__https://www.dremio.com/?utm_medium=email&utm_source=signature&utm_term=na&utm_content=email-signature&utm_campaign=email-signature__;!!Iz9xO38YGHZK!_dbBfwEnZEuPp_8qIuBDoYfWRgCZKiPDMg9ouDltqsHhr9hqCHVlF9IedSkNVBDWupbneU7inC9YK-THL-lTjWkkiOwjJc0$
>> >/
>> > Follow Us on LinkedIn
>> > <
>> https://urldefense.com/v3/__https://www.linkedin.com/company/dremio__;!!Iz9xO38YGHZK!_dbBfwEnZEuPp_8qIuBDoYfWRgCZKiPDMg9ouDltqsHhr9hqCHVlF9IedSkNVBDWupbneU7inC9YK-THL-lTjWkkCLjBQ2M$
>> >/
>> > Get Started
>> > <
>> https://urldefense.com/v3/__https://www.dremio.com/get-started/__;!!Iz9xO38YGHZK!_dbBfwEnZEuPp_8qIuBDoYfWRgCZKiPDMg9ouDltqsHhr9hqCHVlF9IedSkNVBDWupbneU7inC9YK-THL-lTjWkkoKQUMII$
>> >
>> >
>> >
>> >The Agentic Lakehouse
>> > *The only lakehouse built for agents, managed by agents*
>> >
>> >
>> >
>>
>>

Re: Polaris Telemetry and Audit Trail

[Dmitri Bourlatchkov <di...@apache.org>]

Hi All,

The SPI PR [3616] for Scan/Commit metrics has been in review for 2 weeks
including 3 days in an approved state.

Unless there are any last minute comments, I propose to merge it tomorrow
(Feb 13).

[3616] https://github.com/apache/polaris/pull/3616

Cheers,
Dmitri.

On Mon, Feb 9, 2026 at 2:08 PM Anand Kumar Sankaran via dev <
dev@polaris.apache.org> wrote:

> Dmitri and others,
>
> Thanks for all the review feedback and being patient while we iterated. My
> initial proposal was to persist the table metrics as events (least amount
> of work 😊), we have come far from that proposal.
>
> To detach the SPI fully, there are a few things to do (loadSchemaVersion()
> and shared DatasourceOperations). We can add a separate datasource config
> to specify a different connection pool, perhaps. Can be done as a follow up
> PR.
>
> —
> Anand
>
> From: Dmitri Bourlatchkov <di...@apache.org>
> Date: Monday, February 9, 2026 at 10:52 AM
> To: dev@polaris.apache.org <de...@polaris.apache.org>
> Cc: Anand Kumar Sankaran <an...@workday.com>
> Subject: Re: Polaris Telemetry and Audit Trail
>
> This Message Is From an External Sender
> This message came from outside your organization.
> Report Suspicious<
> https://us-phishalarm-ewt.proofpoint.com/EWT/v1/Iz9xO38YGHZK!YhNDZAGr2cumYduJ20ONMhMxljYPF8uZOe6R8BpEWyJ7tSG5GUUf8our05nYHWov0t1Vi7Rx_O8JuRNBBQQ57aBoctrR0wH7VSR9VxRuqYJPpz0q6I0JboPRfdLIwqal$
> >
>
> Hi Anand and all,
>
> Thanks for pushing this feature forward! From my POV the SPI [3616] and
> the schema [3523] PRs are good to merge as the initial implementation of
> this feature (annotated as "beta"). The way is open to the JDBC impl. PR
> [3385] (which will probably need a rebase after merging the SPI and schema
> code).
>
> I'm sending this email to highlight the direction taken with respect to
> the SQL schema in this feature, as it might have been visible enough to the
> broader community during the PR reviews.
>
> * The scan / commit metrics schema is managed / versioned separately from
> the JDBC MetaStore schema
>
> * The Metrics Persistence SPI is detached (mostly at this time, but
> supposedly fully detached later) from the JDBC MetaStore
>
> * The JDBC DataSource for metrics may be the same or different from the
> MetaStore datasource.
>
> * Metrics Persistence may have non-JDBC implementation in the future (not
> implemented ATM)
>
> * JDBC Metrics Persistence may co-exist with other MetaStore
> implementations.
>
> I believe this design allows great flexibility for end users to try this
> feature out and keeps options open for future enhancements in Scan / Commit
> Metrics in Polaris.
>
> [3616] https://github.com/apache/polaris/pull/3616<
> https://urldefense.com/v3/__https://github.com/apache/polaris/pull/3616__;!!Iz9xO38YGHZK!7IwwfPXDN0ThhNZgol-n9Ww3FFLKcpZbSQ17lWEtcQnTHtJM9aVXnqGW99_6vmVkWACy7rYC8XEpN6Okdg$
> >
>
> [3523] https://github.com/apache/polaris/pull/3523<
> https://urldefense.com/v3/__https://github.com/apache/polaris/pull/3523__;!!Iz9xO38YGHZK!7IwwfPXDN0ThhNZgol-n9Ww3FFLKcpZbSQ17lWEtcQnTHtJM9aVXnqGW99_6vmVkWACy7rYC8XGOCHKhSg$
> >
>
> [3385] https://github.com/apache/polaris/pull/3385<
> https://urldefense.com/v3/__https://github.com/apache/polaris/pull/3385__;!!Iz9xO38YGHZK!7IwwfPXDN0ThhNZgol-n9Ww3FFLKcpZbSQ17lWEtcQnTHtJM9aVXnqGW99_6vmVkWACy7rYC8XFlxLMhvQ$
> >
>
> Thoughts / comments?
>
> Cheers,
> Dmitri.
>
>
> On Thu, Feb 5, 2026 at 6:50 PM Anand Kumar Sankaran via dev <
> dev@polaris.apache.org<ma...@polaris.apache.org>> wrote:
> Hi Dmitri,
>
> Following up on your feedback regarding the metrics schema separation
> approach. I've implemented the simplified design you suggested in
> https://github.com/apache/polaris/pull/3523<
> https://urldefense.com/v3/__https://github.com/apache/polaris/pull/3523__;!!Iz9xO38YGHZK!7IwwfPXDN0ThhNZgol-n9Ww3FFLKcpZbSQ17lWEtcQnTHtJM9aVXnqGW99_6vmVkWACy7rYC8XGOCHKhSg$
> >:
>
>
>  The metrics schema can now evolve independently from the entity schema.
> Let me know if you'd like any changes to this approach.
>
>  Thanks,
>   Anand
>
>
>
> From: Dmitri Bourlatchkov <di...@apache.org>>
> Date: Thursday, February 5, 2026 at 2:57 PM
> To: dev@polaris.apache.org<ma...@polaris.apache.org> <
> dev@polaris.apache.org<ma...@polaris.apache.org>>
> Cc: Anand Kumar Sankaran <anand.sankaran@workday.com<mailto:
> anand.sankaran@workday.com>>
> Subject: Re: Polaris Telemetry and Audit Trail
>
> This Message Is From an External Sender
> This message came from outside your organization.
> Report Suspicious<
> https://us-phishalarm-ewt.proofpoint.com/EWT/v1/Iz9xO38YGHZK!YhNDZAGr2cumY1iAus5GNbkIxSXoQaI552sXgYyv7Z9Wrvb7qeTskSy5WkXTVDJ_Bel2AbNh_fQVn-HS_UtOSdOtpsNfzhXDkK4fYWdTeoeBe70A6oQVGpAYT3SrbEvs$
> >
>
> Hi Anand,
>
> Re: MetricsPersistence Factory - TBH I do not think it is required. If we
> go with an MetricsPersistence impl. independent of the JDBC MetaStore, it
> could be injected into PersistingMetricsReporter directly by CDI.
>
> Re: DataSource - for a start we could probably reuse the same DataSource
> as the JDBS MetaStore uses. In principle, we could introduce a separate
> DataSource, but I do not think we need to complicate the initial
> implementation with that. It can be added later, if required.
>
> Re: schema / bootstrap - I'm thinking, indeed, of having a separate *.sql
> file for metrics DDL. Then we could add an option to the bootstrap command
> in the Admin Tool, or even a new command for creating / updating the
> metrics schema. This might require some refactoring, but I believe the
> bootstrap code needs some cleanup anyway.
>
> The general idea is to allow the metrics schema to evolve independently.
> It's a new and actively developed feature, so having flexibility is going
> to be good for making progress / improvements, I hope.
>
> Does that sound reasonable?
>
> Thanks,
> Dmitri.
>
> On Thu, Feb 5, 2026 at 3:47 PM Anand Kumar Sankaran via dev <
> dev@polaris.apache.org<ma...@polaris.apache.org><mailto:
> dev@polaris.apache.org<ma...@polaris.apache.org>>> wrote:
> Hi Dmitri
>
> The SPI is designed to be pluggable via
> MetaStoreManagerFactory.getOrCreateMetricsPersistence()<
> https://github.com/apache/polaris/pull/3385/changes#diff-0e34667d083d30cace067936b87c736201e5a8cb001aa0b12e203ee52cddae99R51
> <
> https://urldefense.com/v3/__https://github.com/apache/polaris/pull/3385/changes*diff-0e34667d083d30cace067936b87c736201e5a8cb001aa0b12e203ee52cddae99R51__;Iw!!Iz9xO38YGHZK!7IwwfPXDN0ThhNZgol-n9Ww3FFLKcpZbSQ17lWEtcQnTHtJM9aVXnqGW99_6vmVkWACy7rYC8XFxRr4qRw$
> ><
> https://urldefense.com/v3/__https://github.com/apache/polaris/pull/3385/changes*diff-0e34667d083d30cace067936b87c736201e5a8cb001aa0b12e203ee52cddae99R51__;Iw!!Iz9xO38YGHZK!_K1mcH0_2x0nbU-mZlbBi3tk9QAGjB6VInPiqkQU21mpIP_ft2bXXyIiADNoAcotNr9AKqlphh65taUhcw$>>.
> The default returns MetricsPersistence.NOOP, and only JDBC with schema v4+
> returns a functional implementation.
>
>  Your proposal makes sense to me. To allow NoSQL Entity Persistence + JDBC
> Metrics Persistence would require:
>
>      1. Separate metrics schema files (e.g., metrics-schema-v1.sql) that
> can be bootstrapped independently
>      2. Separate datasource configuration for metrics (e.g.,
> polaris.metrics.jdbc.*)
>      3. A dedicated `MetricsPersistenceFactory` interface (separate from
> MetaStoreManagerFactory)
>      4. Updates to the Admin Tool bootstrap workflow to handle both schemas
>
> Does that sound reasonable?  If we do this, all this work should go into
> https://github.com/apache/polaris/pull/3523<
> https://urldefense.com/v3/__https://github.com/apache/polaris/pull/3523__;!!Iz9xO38YGHZK!7IwwfPXDN0ThhNZgol-n9Ww3FFLKcpZbSQ17lWEtcQnTHtJM9aVXnqGW99_6vmVkWACy7rYC8XGOCHKhSg$
> ><
> https://urldefense.com/v3/__https://github.com/apache/polaris/pull/3523__;!!Iz9xO38YGHZK!_K1mcH0_2x0nbU-mZlbBi3tk9QAGjB6VInPiqkQU21mpIP_ft2bXXyIiADNoAcotNr9AKqlphh4ZISZcDQ$>
> as one atomic commit, correct?
>
> —
> Anand
>
>
>
> From: Dmitri Bourlatchkov <dimas@apache.org<mailto:dimas@apache.org
> ><ma...@apache.org>>>
> Date: Thursday, February 5, 2026 at 8:28 AM
> To: Anand Kumar Sankaran <anand.sankaran@workday.com<mailto:
> anand.sankaran@workday.com><mailto:anand.sankaran@workday.com<mailto:
> anand.sankaran@workday.com>>>
> Cc: dev@polaris.apache.org<ma...@polaris.apache.org><mailto:
> dev@polaris.apache.org<ma...@polaris.apache.org>> <
> dev@polaris.apache.org<ma...@polaris.apache.org><mailto:
> dev@polaris.apache.org<ma...@polaris.apache.org>>>
> Subject: Re: Polaris Telemetry and Audit Trail
>
> This Message Is From an External Sender
> This message came from outside your organization.
> Report Suspicious<
> https://us-phishalarm-ewt.proofpoint.com/EWT/v1/Iz9xO38YGHZK!YhNDZAGr2cumbRcPlIOttKwvGr4f38TF-J2-LmOw8H1KsFpWaM3W5g28Oq5qW2gnw2-WHz0rsWhm7F7hSGykA9l01LKYGuCcCkexkHp1sUssM_IegCtjk_id9s891orv$
> >
>
>
> Hi Anand,
>
> Thanks for driving the SPI definition for scan/commits metrics.
>
> I did not think from this angle before, but now I suppose it might be worth
> introducing a separate SQL schema for metrics, detached from the main JDBC
> Persistence schema.
>
> My rationale is that the SPI may be exercised independently of the JDBC
> Entity Persistence. I can imagine people using NoSQL Entity Persistence
> plus JDBC Scan Metrics Persistence in the same deployment.
>
> Using a separate schema will add further work on the Bootstrap workflow
> (Admin Tool), but I hope the end result is going to be more convenient and
> versatile for end users.
>
> WDYT?
>
> Thanks,
> Dmitri.
>
> On Thu, Jan 29, 2026 at 1:49 PM Anand Kumar Sankaran <
> anand.sankaran@workday.com<ma...@workday.com><mailto:
> anand.sankaran@workday.com<ma...@workday.com>>> wrote:
>
> > Hi Dmitri,
> >
> > Thank you for taking the time to review. I had pulled out the schema in
> > its own PR (based on comments) here:
> >
> https://urldefense.com/v3/__https://github.com/apache/polaris/pull/3523__;!!Iz9xO38YGHZK!65BVFQDe_GtKbDW8LkNdzVS-WYewzWsSns5zFHKclbMU3vUy2UaTXLB5G-wql1yyqoTFyDc9Bt3GRsh6AA$
> >
> > I will update the document with the SPI and carve out another PR that
> > focuses on the SPI.
> >
> > —
> > Anand
> >
> > *From: *Dmitri Bourlatchkov <dmitri.bourlatchkov@dremio.com<mailto:
> dmitri.bourlatchkov@dremio.com><mailto:dmitri.bourlatchkov@dremio.com
> <ma...@dremio.com>>>
> > *Date: *Thursday, January 29, 2026 at 10:43 AM
> > *To: *dev@polaris.apache.org<ma...@polaris.apache.org><mailto:
> dev@polaris.apache.org<ma...@polaris.apache.org>> <
> dev@polaris.apache.org<ma...@polaris.apache.org><mailto:
> dev@polaris.apache.org<ma...@polaris.apache.org>>>
> > *Cc: *Anand Kumar Sankaran <anand.sankaran@workday.com<mailto:
> anand.sankaran@workday.com><mailto:anand.sankaran@workday.com<mailto:
> anand.sankaran@workday.com>>>
> > *Subject: *Re: Polaris Telemetry and Audit Trail
> >
> > This Message Is From an External Sender
> > This message came from outside your organization.
> > Report Suspicious
> > <
> https://us-phishalarm-ewt.proofpoint.com/EWT/v1/Iz9xO38YGHZK!YhNDZAGr2cumY1cAlUeoPrU2eNUPz9Wxq2PaG1t99vEN3v60gucxC3Qsk-xK09RndcW46WtNjMMjQZbaQ495h1D1g293F_uqLhPM8gDMlA6xRmo5b5uqrSh2v7QKHrhQ$
> >
> >
> > Hi Anand,
> >
> > Thanks for making a proposal doc and starting this thread. Apologies for
> > late comments.
> >
> > As you know I already reviewed a couple of related PRs, which had a
> > smaller impact on the codebase, but now, I believe, we're coming to the
> > core of this feature and it might need a deeper discussion.
> >
> > For the sake of awareness of other project members, I'd like to
> > highlight some aspects by email.
> >
> > * Storing scan metrics in RDBMS in general looks like a
> > reasonable approach to me. However, Polaris Persistence is an extensible
> > mechanism. Other backends are possible, including privately developed
> > backends.
> >
> > From this POV, I believe it might be preferable to start not with an
> RDBMS
> > schema, but with a java SPI outlining expected read/write operations and
> > the data model. This will make it easier to assess impact on other
> > Persistence implementations. Naturally, Polaris service code will have to
> > be able to work seamlessly regardless of the backend impl. (including the
> > do-nothing impl.).
> >
> > I see that some connection between the SQL schema and java code is made
> in
> > PR [3385], but it does not appear to offer a clean SPI that could be
> > implemented by different Persistence backends. Would it be ok from your
> > perspective to make another PR with just SPI code?
> >
> > * Regarding ad-hoc SQL queries - it is certainly a valid use case for
> > custom code to query the Polaris database directly. However, those ad-hoc
> > queries may require certain indexes, which may not have a relevant use
> case
> > in OSS code. In that case, I believe it would be preferable to manage the
> > extra indexes in custom code (not in Polaris). Whether this is the case
> or
> > not currently, I cannot say with certainty, so apologies if this is a
> false
> > alarm... Having the java SPI available would be a huge aid to reasoning
> > about the proposed SQL schema, I think.
> >
> > That said, I do not mean to complicate the development of this feature,
> > just trying to structure it in a way that is hopefully easy to understand
> > (including myself), maintain and expand later. If I missed something,
> > please feel free to point it out.
> >
> > [3385]
> https://urldefense.com/v3/__https://github.com/apache/polaris/pull/3385__;!!Iz9xO38YGHZK!65BVFQDe_GtKbDW8LkNdzVS-WYewzWsSns5zFHKclbMU3vUy2UaTXLB5G-wql1yyqoTFyDc9Bt2W-eKl7A$
> > <
> https://urldefense.com/v3/__https://github.com/apache/polaris/pull/3385__;!!Iz9xO38YGHZK!_dbBfwEnZEuPp_8qIuBDoYfWRgCZKiPDMg9ouDltqsHhr9hqCHVlF9IedSkNVBDWupbneU7inC9YK-THL-lTjWkkyZhI0OE$
> >
> >
> >Thanks,
> > Dmitri.
> >
> > On Wed, Jan 14, 2026 at 10:05 PM Anand Kumar Sankaran via dev <
> > dev@polaris.apache.org<ma...@polaris.apache.org><mailto:
> dev@polaris.apache.org<ma...@polaris.apache.org>>> wrote:
> >
> > Hi Yufei,
> >
> > Gave commenting privileges to all.
> >
> > Thanks.
> >
> > —
> > Anand
> >
> > From: Yufei Gu <flyrain000@gmail.com<mailto:flyrain000@gmail.com
> ><ma...@gmail.com>>>
> > Date: Wednesday, January 14, 2026 at 6:13 PM
> > To: dev@polaris.apache.org<ma...@polaris.apache.org><mailto:
> dev@polaris.apache.org<ma...@polaris.apache.org>> <
> dev@polaris.apache.org<ma...@polaris.apache.org><mailto:
> dev@polaris.apache.org<ma...@polaris.apache.org>>>
> > Cc: Anand Kumar Sankaran <anand.sankaran@workday.com<mailto:
> anand.sankaran@workday.com><mailto:anand.sankaran@workday.com<mailto:
> anand.sankaran@workday.com>>>
> > Subject: Re: Polaris Telemetry and Audit Trail
> >
> > This Message Is From an External Sender
> > This message came from outside your organization.
> > Report Suspicious<
> >
> https://us-phishalarm-ewt.proofpoint.com/EWT/v1/Iz9xO38YGHZK!YhNDZABkHi1B6hyPVYUK0T-K5j4Aoqdrlu07UjBpJcpjOsz7Ie7d_DhPl-ywi50dfQz4O6LL-AHLCOQTQ0X7Le8yD2cF0ogquGFB8Aw8uQLgB0RL6Ezpa-pLDPA4ziXA$
> > >
> >
> > Hi Anand, thanks for the design doc and PR. Can you allow comments in the
> > doc so that people can chime in? Thanks!
> >
> > Yufei
> >
> >
> > On Sat, Jan 10, 2026 at 9:09 AM Anand Kumar Sankaran via dev <
> > dev@polaris.apache.org<ma...@polaris.apache.org><mailto:
> dev@polaris.apache.org<ma...@polaris.apache.org>><mailto:
> dev@polaris.apache.org<ma...@polaris.apache.org><mailto:
> dev@polaris.apache.org<ma...@polaris.apache.org>>>> wrote:
> > Hi all
> >
> > My first PR for adding AWS STS Session Tags support for credential
> vending
> > was merged.
> https://urldefense.com/v3/__https://github.com/apache/polaris/pull/3327__;!!Iz9xO38YGHZK!65BVFQDe_GtKbDW8LkNdzVS-WYewzWsSns5zFHKclbMU3vUy2UaTXLB5G-wql1yyqoTFyDc9Bt0YHY9sdA$
> > <
> https://urldefense.com/v3/__https://github.com/apache/polaris/pull/3327__;!!Iz9xO38YGHZK!_dbBfwEnZEuPp_8qIuBDoYfWRgCZKiPDMg9ouDltqsHhr9hqCHVlF9IedSkNVBDWupbneU7inC9YK-THL-lTjWkklJ4Xf9Y$
> >
> ><
> >
> https://urldefense.com/v3/__https://github.com/apache/polaris/pull/3327__;!!Iz9xO38YGHZK!8vlQ2_OqjmZNIpbrTTluFFNKFCEeM87nr95MP5pMpMu5fFWVSoBZ3eb6-C2xbAGCG-ry5eaCO-pFLxl3IZAJGqY$
> >>
> >
> > I have been working with Prashant on an end-to-end telemetry and audit
> > trail tracking for Polaris.  It is documented here.
> >
> https://urldefense.com/v3/__https://docs.google.com/document/d/1Ehzvi5RNPs4hChkBFI6VD23myEqm-7sWW3d2kjmuYj8/edit?tab=t.0__;!!Iz9xO38YGHZK!65BVFQDe_GtKbDW8LkNdzVS-WYewzWsSns5zFHKclbMU3vUy2UaTXLB5G-wql1yyqoTFyDc9Bt11Y1rbNQ$
> > <
> https://urldefense.com/v3/__https://docs.google.com/document/d/1Ehzvi5RNPs4hChkBFI6VD23myEqm-7sWW3d2kjmuYj8/edit?tab=t.0__;!!Iz9xO38YGHZK!_dbBfwEnZEuPp_8qIuBDoYfWRgCZKiPDMg9ouDltqsHhr9hqCHVlF9IedSkNVBDWupbneU7inC9YK-THL-lTjWkk8MVJPlY$
> >
> ><
> >
> https://urldefense.com/v3/__https://docs.google.com/document/d/1Ehzvi5RNPs4hChkBFI6VD23myEqm-7sWW3d2kjmuYj8/edit?tab=t.0__;!!Iz9xO38YGHZK!8vlQ2_OqjmZNIpbrTTluFFNKFCEeM87nr95MP5pMpMu5fFWVSoBZ3eb6-C2xbAGCG-ry5eaCO-pFLxl3hYqTlcg$
> >>
> >
> > Based on Prashant’s initial feedback (parity with Apache Gravitino for
> > metrics reports), I have an initial PR here.
> >
> https://urldefense.com/v3/__https://github.com/apache/polaris/pull/3385__;!!Iz9xO38YGHZK!65BVFQDe_GtKbDW8LkNdzVS-WYewzWsSns5zFHKclbMU3vUy2UaTXLB5G-wql1yyqoTFyDc9Bt2W-eKl7A$
> > <
> https://urldefense.com/v3/__https://github.com/apache/polaris/pull/3385__;!!Iz9xO38YGHZK!_dbBfwEnZEuPp_8qIuBDoYfWRgCZKiPDMg9ouDltqsHhr9hqCHVlF9IedSkNVBDWupbneU7inC9YK-THL-lTjWkkyZhI0OE$
> >
> ><
> >
> https://urldefense.com/v3/__https://github.com/apache/polaris/pull/3385__;!!Iz9xO38YGHZK!8vlQ2_OqjmZNIpbrTTluFFNKFCEeM87nr95MP5pMpMu5fFWVSoBZ3eb6-C2xbAGCG-ry5eaCO-pFLxl39opCznI$
> >>
> >
> > I will be reaching out to various folks for review. I am open to breaking
> > down the PR into smaller logical groups of PR if it helps.
> >
> > I request you to look at the Google doc and let me know what you think.
> >
> > —
> > Anand
> >
> >
> >
> > --
> > Dmitri Bourlatchkov
> > Senior Staff Software Engineer, Dremio
> > Dremio.com
> > <
> https://urldefense.com/v3/__https://www.dremio.com/?utm_medium=email&utm_source=signature&utm_term=na&utm_content=email-signature&utm_campaign=email-signature__;!!Iz9xO38YGHZK!_dbBfwEnZEuPp_8qIuBDoYfWRgCZKiPDMg9ouDltqsHhr9hqCHVlF9IedSkNVBDWupbneU7inC9YK-THL-lTjWkkiOwjJc0$
> >/
> > Follow Us on LinkedIn
> > <
> https://urldefense.com/v3/__https://www.linkedin.com/company/dremio__;!!Iz9xO38YGHZK!_dbBfwEnZEuPp_8qIuBDoYfWRgCZKiPDMg9ouDltqsHhr9hqCHVlF9IedSkNVBDWupbneU7inC9YK-THL-lTjWkkCLjBQ2M$
> >/
> > Get Started
> > <
> https://urldefense.com/v3/__https://www.dremio.com/get-started/__;!!Iz9xO38YGHZK!_dbBfwEnZEuPp_8qIuBDoYfWRgCZKiPDMg9ouDltqsHhr9hqCHVlF9IedSkNVBDWupbneU7inC9YK-THL-lTjWkkoKQUMII$
> >
> >
> >
> >The Agentic Lakehouse
> > *The only lakehouse built for agents, managed by agents*
> >
> >
> >
>
>

Re: Polaris Telemetry and Audit Trail

[Anand Kumar Sankaran via dev <de...@polaris.apache.org>]

Dmitri and others,

Thanks for all the review feedback and being patient while we iterated. My initial proposal was to persist the table metrics as events (least amount of work 😊), we have come far from that proposal.

To detach the SPI fully, there are a few things to do (loadSchemaVersion() and shared DatasourceOperations). We can add a separate datasource config to specify a different connection pool, perhaps. Can be done as a follow up PR.

—
Anand

From: Dmitri Bourlatchkov <di...@apache.org>
Date: Monday, February 9, 2026 at 10:52 AM
To: dev@polaris.apache.org <de...@polaris.apache.org>
Cc: Anand Kumar Sankaran <an...@workday.com>
Subject: Re: Polaris Telemetry and Audit Trail

This Message Is From an External Sender
This message came from outside your organization.
Report Suspicious<https://us-phishalarm-ewt.proofpoint.com/EWT/v1/Iz9xO38YGHZK!YhNDZAGr2cumYduJ20ONMhMxljYPF8uZOe6R8BpEWyJ7tSG5GUUf8our05nYHWov0t1Vi7Rx_O8JuRNBBQQ57aBoctrR0wH7VSR9VxRuqYJPpz0q6I0JboPRfdLIwqal$>

Hi Anand and all,

Thanks for pushing this feature forward! From my POV the SPI [3616] and the schema [3523] PRs are good to merge as the initial implementation of this feature (annotated as "beta"). The way is open to the JDBC impl. PR [3385] (which will probably need a rebase after merging the SPI and schema code).

I'm sending this email to highlight the direction taken with respect to the SQL schema in this feature, as it might have been visible enough to the broader community during the PR reviews.

* The scan / commit metrics schema is managed / versioned separately from the JDBC MetaStore schema

* The Metrics Persistence SPI is detached (mostly at this time, but supposedly fully detached later) from the JDBC MetaStore

* The JDBC DataSource for metrics may be the same or different from the MetaStore datasource.

* Metrics Persistence may have non-JDBC implementation in the future (not implemented ATM)

* JDBC Metrics Persistence may co-exist with other MetaStore implementations.

I believe this design allows great flexibility for end users to try this feature out and keeps options open for future enhancements in Scan / Commit Metrics in Polaris.

[3616] https://github.com/apache/polaris/pull/3616<https://urldefense.com/v3/__https://github.com/apache/polaris/pull/3616__;!!Iz9xO38YGHZK!7IwwfPXDN0ThhNZgol-n9Ww3FFLKcpZbSQ17lWEtcQnTHtJM9aVXnqGW99_6vmVkWACy7rYC8XEpN6Okdg$>

[3523] https://github.com/apache/polaris/pull/3523<https://urldefense.com/v3/__https://github.com/apache/polaris/pull/3523__;!!Iz9xO38YGHZK!7IwwfPXDN0ThhNZgol-n9Ww3FFLKcpZbSQ17lWEtcQnTHtJM9aVXnqGW99_6vmVkWACy7rYC8XGOCHKhSg$>

[3385] https://github.com/apache/polaris/pull/3385<https://urldefense.com/v3/__https://github.com/apache/polaris/pull/3385__;!!Iz9xO38YGHZK!7IwwfPXDN0ThhNZgol-n9Ww3FFLKcpZbSQ17lWEtcQnTHtJM9aVXnqGW99_6vmVkWACy7rYC8XFlxLMhvQ$>

Thoughts / comments?

Cheers,
Dmitri.


On Thu, Feb 5, 2026 at 6:50 PM Anand Kumar Sankaran via dev <de...@polaris.apache.org>> wrote:
Hi Dmitri,

Following up on your feedback regarding the metrics schema separation approach. I've implemented the simplified design you suggested in https://github.com/apache/polaris/pull/3523<https://urldefense.com/v3/__https://github.com/apache/polaris/pull/3523__;!!Iz9xO38YGHZK!7IwwfPXDN0ThhNZgol-n9Ww3FFLKcpZbSQ17lWEtcQnTHtJM9aVXnqGW99_6vmVkWACy7rYC8XGOCHKhSg$>:


 The metrics schema can now evolve independently from the entity schema. Let me know if you'd like any changes to this approach.

 Thanks,
  Anand



From: Dmitri Bourlatchkov <di...@apache.org>>
Date: Thursday, February 5, 2026 at 2:57 PM
To: dev@polaris.apache.org<ma...@polaris.apache.org> <de...@polaris.apache.org>>
Cc: Anand Kumar Sankaran <an...@workday.com>>
Subject: Re: Polaris Telemetry and Audit Trail

This Message Is From an External Sender
This message came from outside your organization.
Report Suspicious<https://us-phishalarm-ewt.proofpoint.com/EWT/v1/Iz9xO38YGHZK!YhNDZAGr2cumY1iAus5GNbkIxSXoQaI552sXgYyv7Z9Wrvb7qeTskSy5WkXTVDJ_Bel2AbNh_fQVn-HS_UtOSdOtpsNfzhXDkK4fYWdTeoeBe70A6oQVGpAYT3SrbEvs$>

Hi Anand,

Re: MetricsPersistence Factory - TBH I do not think it is required. If we go with an MetricsPersistence impl. independent of the JDBC MetaStore, it could be injected into PersistingMetricsReporter directly by CDI.

Re: DataSource - for a start we could probably reuse the same DataSource as the JDBS MetaStore uses. In principle, we could introduce a separate DataSource, but I do not think we need to complicate the initial implementation with that. It can be added later, if required.

Re: schema / bootstrap - I'm thinking, indeed, of having a separate *.sql file for metrics DDL. Then we could add an option to the bootstrap command in the Admin Tool, or even a new command for creating / updating the metrics schema. This might require some refactoring, but I believe the bootstrap code needs some cleanup anyway.

The general idea is to allow the metrics schema to evolve independently. It's a new and actively developed feature, so having flexibility is going to be good for making progress / improvements, I hope.

Does that sound reasonable?

Thanks,
Dmitri.

On Thu, Feb 5, 2026 at 3:47 PM Anand Kumar Sankaran via dev <de...@polaris.apache.org>>> wrote:
Hi Dmitri

The SPI is designed to be pluggable via MetaStoreManagerFactory.getOrCreateMetricsPersistence()<https://github.com/apache/polaris/pull/3385/changes#diff-0e34667d083d30cace067936b87c736201e5a8cb001aa0b12e203ee52cddae99R51<https://urldefense.com/v3/__https://github.com/apache/polaris/pull/3385/changes*diff-0e34667d083d30cace067936b87c736201e5a8cb001aa0b12e203ee52cddae99R51__;Iw!!Iz9xO38YGHZK!7IwwfPXDN0ThhNZgol-n9Ww3FFLKcpZbSQ17lWEtcQnTHtJM9aVXnqGW99_6vmVkWACy7rYC8XFxRr4qRw$><https://urldefense.com/v3/__https://github.com/apache/polaris/pull/3385/changes*diff-0e34667d083d30cace067936b87c736201e5a8cb001aa0b12e203ee52cddae99R51__;Iw!!Iz9xO38YGHZK!_K1mcH0_2x0nbU-mZlbBi3tk9QAGjB6VInPiqkQU21mpIP_ft2bXXyIiADNoAcotNr9AKqlphh65taUhcw$>>. The default returns MetricsPersistence.NOOP, and only JDBC with schema v4+ returns a functional implementation.

 Your proposal makes sense to me. To allow NoSQL Entity Persistence + JDBC Metrics Persistence would require:

     1. Separate metrics schema files (e.g., metrics-schema-v1.sql) that can be bootstrapped independently
     2. Separate datasource configuration for metrics (e.g., polaris.metrics.jdbc.*)
     3. A dedicated `MetricsPersistenceFactory` interface (separate from MetaStoreManagerFactory)
     4. Updates to the Admin Tool bootstrap workflow to handle both schemas

Does that sound reasonable?  If we do this, all this work should go into https://github.com/apache/polaris/pull/3523<https://urldefense.com/v3/__https://github.com/apache/polaris/pull/3523__;!!Iz9xO38YGHZK!7IwwfPXDN0ThhNZgol-n9Ww3FFLKcpZbSQ17lWEtcQnTHtJM9aVXnqGW99_6vmVkWACy7rYC8XGOCHKhSg$><https://urldefense.com/v3/__https://github.com/apache/polaris/pull/3523__;!!Iz9xO38YGHZK!_K1mcH0_2x0nbU-mZlbBi3tk9QAGjB6VInPiqkQU21mpIP_ft2bXXyIiADNoAcotNr9AKqlphh4ZISZcDQ$> as one atomic commit, correct?

—
Anand



From: Dmitri Bourlatchkov <di...@apache.org>>>
Date: Thursday, February 5, 2026 at 8:28 AM
To: Anand Kumar Sankaran <an...@workday.com>>>
Cc: dev@polaris.apache.org<ma...@polaris.apache.org>> <de...@polaris.apache.org>>>
Subject: Re: Polaris Telemetry and Audit Trail

This Message Is From an External Sender
This message came from outside your organization.
Report Suspicious<https://us-phishalarm-ewt.proofpoint.com/EWT/v1/Iz9xO38YGHZK!YhNDZAGr2cumbRcPlIOttKwvGr4f38TF-J2-LmOw8H1KsFpWaM3W5g28Oq5qW2gnw2-WHz0rsWhm7F7hSGykA9l01LKYGuCcCkexkHp1sUssM_IegCtjk_id9s891orv$>


Hi Anand,

Thanks for driving the SPI definition for scan/commits metrics.

I did not think from this angle before, but now I suppose it might be worth
introducing a separate SQL schema for metrics, detached from the main JDBC
Persistence schema.

My rationale is that the SPI may be exercised independently of the JDBC
Entity Persistence. I can imagine people using NoSQL Entity Persistence
plus JDBC Scan Metrics Persistence in the same deployment.

Using a separate schema will add further work on the Bootstrap workflow
(Admin Tool), but I hope the end result is going to be more convenient and
versatile for end users.

WDYT?

Thanks,
Dmitri.

On Thu, Jan 29, 2026 at 1:49 PM Anand Kumar Sankaran <
anand.sankaran@workday.com<ma...@workday.com>>> wrote:

> Hi Dmitri,
>
> Thank you for taking the time to review. I had pulled out the schema in
> its own PR (based on comments) here:
> https://urldefense.com/v3/__https://github.com/apache/polaris/pull/3523__;!!Iz9xO38YGHZK!65BVFQDe_GtKbDW8LkNdzVS-WYewzWsSns5zFHKclbMU3vUy2UaTXLB5G-wql1yyqoTFyDc9Bt3GRsh6AA$
>
> I will update the document with the SPI and carve out another PR that
> focuses on the SPI.
>
> —
> Anand
>
> *From: *Dmitri Bourlatchkov <dm...@dremio.com>>>
> *Date: *Thursday, January 29, 2026 at 10:43 AM
> *To: *dev@polaris.apache.org<ma...@polaris.apache.org>> <de...@polaris.apache.org>>>
> *Cc: *Anand Kumar Sankaran <an...@workday.com>>>
> *Subject: *Re: Polaris Telemetry and Audit Trail
>
> This Message Is From an External Sender
> This message came from outside your organization.
> Report Suspicious
> <https://us-phishalarm-ewt.proofpoint.com/EWT/v1/Iz9xO38YGHZK!YhNDZAGr2cumY1cAlUeoPrU2eNUPz9Wxq2PaG1t99vEN3v60gucxC3Qsk-xK09RndcW46WtNjMMjQZbaQ495h1D1g293F_uqLhPM8gDMlA6xRmo5b5uqrSh2v7QKHrhQ$ >
>
> Hi Anand,
>
> Thanks for making a proposal doc and starting this thread. Apologies for
> late comments.
>
> As you know I already reviewed a couple of related PRs, which had a
> smaller impact on the codebase, but now, I believe, we're coming to the
> core of this feature and it might need a deeper discussion.
>
> For the sake of awareness of other project members, I'd like to
> highlight some aspects by email.
>
> * Storing scan metrics in RDBMS in general looks like a
> reasonable approach to me. However, Polaris Persistence is an extensible
> mechanism. Other backends are possible, including privately developed
> backends.
>
> From this POV, I believe it might be preferable to start not with an RDBMS
> schema, but with a java SPI outlining expected read/write operations and
> the data model. This will make it easier to assess impact on other
> Persistence implementations. Naturally, Polaris service code will have to
> be able to work seamlessly regardless of the backend impl. (including the
> do-nothing impl.).
>
> I see that some connection between the SQL schema and java code is made in
> PR [3385], but it does not appear to offer a clean SPI that could be
> implemented by different Persistence backends. Would it be ok from your
> perspective to make another PR with just SPI code?
>
> * Regarding ad-hoc SQL queries - it is certainly a valid use case for
> custom code to query the Polaris database directly. However, those ad-hoc
> queries may require certain indexes, which may not have a relevant use case
> in OSS code. In that case, I believe it would be preferable to manage the
> extra indexes in custom code (not in Polaris). Whether this is the case or
> not currently, I cannot say with certainty, so apologies if this is a false
> alarm... Having the java SPI available would be a huge aid to reasoning
> about the proposed SQL schema, I think.
>
> That said, I do not mean to complicate the development of this feature,
> just trying to structure it in a way that is hopefully easy to understand
> (including myself), maintain and expand later. If I missed something,
> please feel free to point it out.
>
> [3385] https://urldefense.com/v3/__https://github.com/apache/polaris/pull/3385__;!!Iz9xO38YGHZK!65BVFQDe_GtKbDW8LkNdzVS-WYewzWsSns5zFHKclbMU3vUy2UaTXLB5G-wql1yyqoTFyDc9Bt2W-eKl7A$
> <https://urldefense.com/v3/__https://github.com/apache/polaris/pull/3385__;!!Iz9xO38YGHZK!_dbBfwEnZEuPp_8qIuBDoYfWRgCZKiPDMg9ouDltqsHhr9hqCHVlF9IedSkNVBDWupbneU7inC9YK-THL-lTjWkkyZhI0OE$>
>
>Thanks,
> Dmitri.
>
> On Wed, Jan 14, 2026 at 10:05 PM Anand Kumar Sankaran via dev <
> dev@polaris.apache.org<ma...@polaris.apache.org>>> wrote:
>
> Hi Yufei,
>
> Gave commenting privileges to all.
>
> Thanks.
>
> —
> Anand
>
> From: Yufei Gu <fl...@gmail.com>>>
> Date: Wednesday, January 14, 2026 at 6:13 PM
> To: dev@polaris.apache.org<ma...@polaris.apache.org>> <de...@polaris.apache.org>>>
> Cc: Anand Kumar Sankaran <an...@workday.com>>>
> Subject: Re: Polaris Telemetry and Audit Trail
>
> This Message Is From an External Sender
> This message came from outside your organization.
> Report Suspicious<
> https://us-phishalarm-ewt.proofpoint.com/EWT/v1/Iz9xO38YGHZK!YhNDZABkHi1B6hyPVYUK0T-K5j4Aoqdrlu07UjBpJcpjOsz7Ie7d_DhPl-ywi50dfQz4O6LL-AHLCOQTQ0X7Le8yD2cF0ogquGFB8Aw8uQLgB0RL6Ezpa-pLDPA4ziXA$
> >
>
> Hi Anand, thanks for the design doc and PR. Can you allow comments in the
> doc so that people can chime in? Thanks!
>
> Yufei
>
>
> On Sat, Jan 10, 2026 at 9:09 AM Anand Kumar Sankaran via dev <
> dev@polaris.apache.org<ma...@polaris.apache.org>>>> wrote:
> Hi all
>
> My first PR for adding AWS STS Session Tags support for credential vending
> was merged. https://urldefense.com/v3/__https://github.com/apache/polaris/pull/3327__;!!Iz9xO38YGHZK!65BVFQDe_GtKbDW8LkNdzVS-WYewzWsSns5zFHKclbMU3vUy2UaTXLB5G-wql1yyqoTFyDc9Bt0YHY9sdA$
> <https://urldefense.com/v3/__https://github.com/apache/polaris/pull/3327__;!!Iz9xO38YGHZK!_dbBfwEnZEuPp_8qIuBDoYfWRgCZKiPDMg9ouDltqsHhr9hqCHVlF9IedSkNVBDWupbneU7inC9YK-THL-lTjWkklJ4Xf9Y$>
><
> https://urldefense.com/v3/__https://github.com/apache/polaris/pull/3327__;!!Iz9xO38YGHZK!8vlQ2_OqjmZNIpbrTTluFFNKFCEeM87nr95MP5pMpMu5fFWVSoBZ3eb6-C2xbAGCG-ry5eaCO-pFLxl3IZAJGqY$
>>
>
> I have been working with Prashant on an end-to-end telemetry and audit
> trail tracking for Polaris.  It is documented here.
> https://urldefense.com/v3/__https://docs.google.com/document/d/1Ehzvi5RNPs4hChkBFI6VD23myEqm-7sWW3d2kjmuYj8/edit?tab=t.0__;!!Iz9xO38YGHZK!65BVFQDe_GtKbDW8LkNdzVS-WYewzWsSns5zFHKclbMU3vUy2UaTXLB5G-wql1yyqoTFyDc9Bt11Y1rbNQ$
> <https://urldefense.com/v3/__https://docs.google.com/document/d/1Ehzvi5RNPs4hChkBFI6VD23myEqm-7sWW3d2kjmuYj8/edit?tab=t.0__;!!Iz9xO38YGHZK!_dbBfwEnZEuPp_8qIuBDoYfWRgCZKiPDMg9ouDltqsHhr9hqCHVlF9IedSkNVBDWupbneU7inC9YK-THL-lTjWkk8MVJPlY$>
><
> https://urldefense.com/v3/__https://docs.google.com/document/d/1Ehzvi5RNPs4hChkBFI6VD23myEqm-7sWW3d2kjmuYj8/edit?tab=t.0__;!!Iz9xO38YGHZK!8vlQ2_OqjmZNIpbrTTluFFNKFCEeM87nr95MP5pMpMu5fFWVSoBZ3eb6-C2xbAGCG-ry5eaCO-pFLxl3hYqTlcg$
>>
>
> Based on Prashant’s initial feedback (parity with Apache Gravitino for
> metrics reports), I have an initial PR here.
> https://urldefense.com/v3/__https://github.com/apache/polaris/pull/3385__;!!Iz9xO38YGHZK!65BVFQDe_GtKbDW8LkNdzVS-WYewzWsSns5zFHKclbMU3vUy2UaTXLB5G-wql1yyqoTFyDc9Bt2W-eKl7A$
> <https://urldefense.com/v3/__https://github.com/apache/polaris/pull/3385__;!!Iz9xO38YGHZK!_dbBfwEnZEuPp_8qIuBDoYfWRgCZKiPDMg9ouDltqsHhr9hqCHVlF9IedSkNVBDWupbneU7inC9YK-THL-lTjWkkyZhI0OE$>
><
> https://urldefense.com/v3/__https://github.com/apache/polaris/pull/3385__;!!Iz9xO38YGHZK!8vlQ2_OqjmZNIpbrTTluFFNKFCEeM87nr95MP5pMpMu5fFWVSoBZ3eb6-C2xbAGCG-ry5eaCO-pFLxl39opCznI$
>>
>
> I will be reaching out to various folks for review. I am open to breaking
> down the PR into smaller logical groups of PR if it helps.
>
> I request you to look at the Google doc and let me know what you think.
>
> —
> Anand
>
>
>
> --
> Dmitri Bourlatchkov
> Senior Staff Software Engineer, Dremio
> Dremio.com
> <https://urldefense.com/v3/__https://www.dremio.com/?utm_medium=email&utm_source=signature&utm_term=na&utm_content=email-signature&utm_campaign=email-signature__;!!Iz9xO38YGHZK!_dbBfwEnZEuPp_8qIuBDoYfWRgCZKiPDMg9ouDltqsHhr9hqCHVlF9IedSkNVBDWupbneU7inC9YK-THL-lTjWkkiOwjJc0$>/
> Follow Us on LinkedIn
> <https://urldefense.com/v3/__https://www.linkedin.com/company/dremio__;!!Iz9xO38YGHZK!_dbBfwEnZEuPp_8qIuBDoYfWRgCZKiPDMg9ouDltqsHhr9hqCHVlF9IedSkNVBDWupbneU7inC9YK-THL-lTjWkkCLjBQ2M$>/
> Get Started
> <https://urldefense.com/v3/__https://www.dremio.com/get-started/__;!!Iz9xO38YGHZK!_dbBfwEnZEuPp_8qIuBDoYfWRgCZKiPDMg9ouDltqsHhr9hqCHVlF9IedSkNVBDWupbneU7inC9YK-THL-lTjWkkoKQUMII$>
>
>
>The Agentic Lakehouse
> *The only lakehouse built for agents, managed by agents*
>
>
>

Re: Polaris Telemetry and Audit Trail

[Dmitri Bourlatchkov <di...@apache.org>]

Hi Anand and all,

Thanks for pushing this feature forward! From my POV the SPI [3616] and the
schema [3523] PRs are good to merge as the initial implementation of this
feature (annotated as "beta"). The way is open to the JDBC impl. PR [3385]
(which will probably need a rebase after merging the SPI and schema code).

I'm sending this email to highlight the direction taken with respect to the
SQL schema in this feature, as it might have been visible enough to the
broader community during the PR reviews.

* The scan / commit metrics schema is managed / versioned separately from
the JDBC MetaStore schema

* The Metrics Persistence SPI is detached (mostly at this time, but
supposedly fully detached later) from the JDBC MetaStore

* The JDBC DataSource for metrics may be the same or different from the
MetaStore datasource.

* Metrics Persistence may have non-JDBC implementation in the future (not
implemented ATM)

* JDBC Metrics Persistence may co-exist with other MetaStore
implementations.

I believe this design allows great flexibility for end users to try this
feature out and keeps options open for future enhancements in Scan / Commit
Metrics in Polaris.

[3616] https://github.com/apache/polaris/pull/3616

[3523] https://github.com/apache/polaris/pull/3523

[3385] https://github.com/apache/polaris/pull/3385

Thoughts / comments?

Cheers,
Dmitri.


On Thu, Feb 5, 2026 at 6:50 PM Anand Kumar Sankaran via dev <
dev@polaris.apache.org> wrote:

> Hi Dmitri,
>
> Following up on your feedback regarding the metrics schema separation
> approach. I've implemented the simplified design you suggested in
> https://github.com/apache/polaris/pull/3523:
>
>
>  The metrics schema can now evolve independently from the entity schema.
> Let me know if you'd like any changes to this approach.
>
>  Thanks,
>   Anand
>
>
>
> From: Dmitri Bourlatchkov <di...@apache.org>
> Date: Thursday, February 5, 2026 at 2:57 PM
> To: dev@polaris.apache.org <de...@polaris.apache.org>
> Cc: Anand Kumar Sankaran <an...@workday.com>
> Subject: Re: Polaris Telemetry and Audit Trail
>
> This Message Is From an External Sender
> This message came from outside your organization.
> Report Suspicious<
> https://us-phishalarm-ewt.proofpoint.com/EWT/v1/Iz9xO38YGHZK!YhNDZAGr2cumY1iAus5GNbkIxSXoQaI552sXgYyv7Z9Wrvb7qeTskSy5WkXTVDJ_Bel2AbNh_fQVn-HS_UtOSdOtpsNfzhXDkK4fYWdTeoeBe70A6oQVGpAYT3SrbEvs$
> >
>
> Hi Anand,
>
> Re: MetricsPersistence Factory - TBH I do not think it is required. If we
> go with an MetricsPersistence impl. independent of the JDBC MetaStore, it
> could be injected into PersistingMetricsReporter directly by CDI.
>
> Re: DataSource - for a start we could probably reuse the same DataSource
> as the JDBS MetaStore uses. In principle, we could introduce a separate
> DataSource, but I do not think we need to complicate the initial
> implementation with that. It can be added later, if required.
>
> Re: schema / bootstrap - I'm thinking, indeed, of having a separate *.sql
> file for metrics DDL. Then we could add an option to the bootstrap command
> in the Admin Tool, or even a new command for creating / updating the
> metrics schema. This might require some refactoring, but I believe the
> bootstrap code needs some cleanup anyway.
>
> The general idea is to allow the metrics schema to evolve independently.
> It's a new and actively developed feature, so having flexibility is going
> to be good for making progress / improvements, I hope.
>
> Does that sound reasonable?
>
> Thanks,
> Dmitri.
>
> On Thu, Feb 5, 2026 at 3:47 PM Anand Kumar Sankaran via dev <
> dev@polaris.apache.org<ma...@polaris.apache.org>> wrote:
> Hi Dmitri
>
> The SPI is designed to be pluggable via
> MetaStoreManagerFactory.getOrCreateMetricsPersistence()<
> https://github.com/apache/polaris/pull/3385/changes#diff-0e34667d083d30cace067936b87c736201e5a8cb001aa0b12e203ee52cddae99R51
> <
> https://urldefense.com/v3/__https://github.com/apache/polaris/pull/3385/changes*diff-0e34667d083d30cace067936b87c736201e5a8cb001aa0b12e203ee52cddae99R51__;Iw!!Iz9xO38YGHZK!_K1mcH0_2x0nbU-mZlbBi3tk9QAGjB6VInPiqkQU21mpIP_ft2bXXyIiADNoAcotNr9AKqlphh65taUhcw$>>.
> The default returns MetricsPersistence.NOOP, and only JDBC with schema v4+
> returns a functional implementation.
>
>  Your proposal makes sense to me. To allow NoSQL Entity Persistence + JDBC
> Metrics Persistence would require:
>
>      1. Separate metrics schema files (e.g., metrics-schema-v1.sql) that
> can be bootstrapped independently
>      2. Separate datasource configuration for metrics (e.g.,
> polaris.metrics.jdbc.*)
>      3. A dedicated `MetricsPersistenceFactory` interface (separate from
> MetaStoreManagerFactory)
>      4. Updates to the Admin Tool bootstrap workflow to handle both schemas
>
> Does that sound reasonable?  If we do this, all this work should go into
> https://github.com/apache/polaris/pull/3523<
> https://urldefense.com/v3/__https://github.com/apache/polaris/pull/3523__;!!Iz9xO38YGHZK!_K1mcH0_2x0nbU-mZlbBi3tk9QAGjB6VInPiqkQU21mpIP_ft2bXXyIiADNoAcotNr9AKqlphh4ZISZcDQ$>
> as one atomic commit, correct?
>
> —
> Anand
>
>
>
> From: Dmitri Bourlatchkov <di...@apache.org>>
> Date: Thursday, February 5, 2026 at 8:28 AM
> To: Anand Kumar Sankaran <anand.sankaran@workday.com<mailto:
> anand.sankaran@workday.com>>
> Cc: dev@polaris.apache.org<ma...@polaris.apache.org> <
> dev@polaris.apache.org<ma...@polaris.apache.org>>
> Subject: Re: Polaris Telemetry and Audit Trail
>
> This Message Is From an External Sender
> This message came from outside your organization.
> Report Suspicious<
> https://us-phishalarm-ewt.proofpoint.com/EWT/v1/Iz9xO38YGHZK!YhNDZAGr2cumbRcPlIOttKwvGr4f38TF-J2-LmOw8H1KsFpWaM3W5g28Oq5qW2gnw2-WHz0rsWhm7F7hSGykA9l01LKYGuCcCkexkHp1sUssM_IegCtjk_id9s891orv$
> >
>
>
> Hi Anand,
>
> Thanks for driving the SPI definition for scan/commits metrics.
>
> I did not think from this angle before, but now I suppose it might be worth
> introducing a separate SQL schema for metrics, detached from the main JDBC
> Persistence schema.
>
> My rationale is that the SPI may be exercised independently of the JDBC
> Entity Persistence. I can imagine people using NoSQL Entity Persistence
> plus JDBC Scan Metrics Persistence in the same deployment.
>
> Using a separate schema will add further work on the Bootstrap workflow
> (Admin Tool), but I hope the end result is going to be more convenient and
> versatile for end users.
>
> WDYT?
>
> Thanks,
> Dmitri.
>
> On Thu, Jan 29, 2026 at 1:49 PM Anand Kumar Sankaran <
> anand.sankaran@workday.com<ma...@workday.com>> wrote:
>
> > Hi Dmitri,
> >
> > Thank you for taking the time to review. I had pulled out the schema in
> > its own PR (based on comments) here:
> >
> https://urldefense.com/v3/__https://github.com/apache/polaris/pull/3523__;!!Iz9xO38YGHZK!65BVFQDe_GtKbDW8LkNdzVS-WYewzWsSns5zFHKclbMU3vUy2UaTXLB5G-wql1yyqoTFyDc9Bt3GRsh6AA$
> >
> > I will update the document with the SPI and carve out another PR that
> > focuses on the SPI.
> >
> > —
> > Anand
> >
> > *From: *Dmitri Bourlatchkov <dmitri.bourlatchkov@dremio.com<mailto:
> dmitri.bourlatchkov@dremio.com>>
> > *Date: *Thursday, January 29, 2026 at 10:43 AM
> > *To: *dev@polaris.apache.org<ma...@polaris.apache.org> <
> dev@polaris.apache.org<ma...@polaris.apache.org>>
> > *Cc: *Anand Kumar Sankaran <anand.sankaran@workday.com<mailto:
> anand.sankaran@workday.com>>
> > *Subject: *Re: Polaris Telemetry and Audit Trail
> >
> > This Message Is From an External Sender
> > This message came from outside your organization.
> > Report Suspicious
> > <
> https://us-phishalarm-ewt.proofpoint.com/EWT/v1/Iz9xO38YGHZK!YhNDZAGr2cumY1cAlUeoPrU2eNUPz9Wxq2PaG1t99vEN3v60gucxC3Qsk-xK09RndcW46WtNjMMjQZbaQ495h1D1g293F_uqLhPM8gDMlA6xRmo5b5uqrSh2v7QKHrhQ$
> >
> >
> > Hi Anand,
> >
> > Thanks for making a proposal doc and starting this thread. Apologies for
> > late comments.
> >
> > As you know I already reviewed a couple of related PRs, which had a
> > smaller impact on the codebase, but now, I believe, we're coming to the
> > core of this feature and it might need a deeper discussion.
> >
> > For the sake of awareness of other project members, I'd like to
> > highlight some aspects by email.
> >
> > * Storing scan metrics in RDBMS in general looks like a
> > reasonable approach to me. However, Polaris Persistence is an extensible
> > mechanism. Other backends are possible, including privately developed
> > backends.
> >
> > From this POV, I believe it might be preferable to start not with an
> RDBMS
> > schema, but with a java SPI outlining expected read/write operations and
> > the data model. This will make it easier to assess impact on other
> > Persistence implementations. Naturally, Polaris service code will have to
> > be able to work seamlessly regardless of the backend impl. (including the
> > do-nothing impl.).
> >
> > I see that some connection between the SQL schema and java code is made
> in
> > PR [3385], but it does not appear to offer a clean SPI that could be
> > implemented by different Persistence backends. Would it be ok from your
> > perspective to make another PR with just SPI code?
> >
> > * Regarding ad-hoc SQL queries - it is certainly a valid use case for
> > custom code to query the Polaris database directly. However, those ad-hoc
> > queries may require certain indexes, which may not have a relevant use
> case
> > in OSS code. In that case, I believe it would be preferable to manage the
> > extra indexes in custom code (not in Polaris). Whether this is the case
> or
> > not currently, I cannot say with certainty, so apologies if this is a
> false
> > alarm... Having the java SPI available would be a huge aid to reasoning
> > about the proposed SQL schema, I think.
> >
> > That said, I do not mean to complicate the development of this feature,
> > just trying to structure it in a way that is hopefully easy to understand
> > (including myself), maintain and expand later. If I missed something,
> > please feel free to point it out.
> >
> > [3385]
> https://urldefense.com/v3/__https://github.com/apache/polaris/pull/3385__;!!Iz9xO38YGHZK!65BVFQDe_GtKbDW8LkNdzVS-WYewzWsSns5zFHKclbMU3vUy2UaTXLB5G-wql1yyqoTFyDc9Bt2W-eKl7A$
> > <
> https://urldefense.com/v3/__https://github.com/apache/polaris/pull/3385__;!!Iz9xO38YGHZK!_dbBfwEnZEuPp_8qIuBDoYfWRgCZKiPDMg9ouDltqsHhr9hqCHVlF9IedSkNVBDWupbneU7inC9YK-THL-lTjWkkyZhI0OE$
> >
> >
> >Thanks,
> > Dmitri.
> >
> > On Wed, Jan 14, 2026 at 10:05 PM Anand Kumar Sankaran via dev <
> > dev@polaris.apache.org<ma...@polaris.apache.org>> wrote:
> >
> > Hi Yufei,
> >
> > Gave commenting privileges to all.
> >
> > Thanks.
> >
> > —
> > Anand
> >
> > From: Yufei Gu <fl...@gmail.com>>
> > Date: Wednesday, January 14, 2026 at 6:13 PM
> > To: dev@polaris.apache.org<ma...@polaris.apache.org> <
> dev@polaris.apache.org<ma...@polaris.apache.org>>
> > Cc: Anand Kumar Sankaran <anand.sankaran@workday.com<mailto:
> anand.sankaran@workday.com>>
> > Subject: Re: Polaris Telemetry and Audit Trail
> >
> > This Message Is From an External Sender
> > This message came from outside your organization.
> > Report Suspicious<
> >
> https://us-phishalarm-ewt.proofpoint.com/EWT/v1/Iz9xO38YGHZK!YhNDZABkHi1B6hyPVYUK0T-K5j4Aoqdrlu07UjBpJcpjOsz7Ie7d_DhPl-ywi50dfQz4O6LL-AHLCOQTQ0X7Le8yD2cF0ogquGFB8Aw8uQLgB0RL6Ezpa-pLDPA4ziXA$
> > >
> >
> > Hi Anand, thanks for the design doc and PR. Can you allow comments in the
> > doc so that people can chime in? Thanks!
> >
> > Yufei
> >
> >
> > On Sat, Jan 10, 2026 at 9:09 AM Anand Kumar Sankaran via dev <
> > dev@polaris.apache.org<ma...@polaris.apache.org><mailto:
> dev@polaris.apache.org<ma...@polaris.apache.org>>> wrote:
> > Hi all
> >
> > My first PR for adding AWS STS Session Tags support for credential
> vending
> > was merged.
> https://urldefense.com/v3/__https://github.com/apache/polaris/pull/3327__;!!Iz9xO38YGHZK!65BVFQDe_GtKbDW8LkNdzVS-WYewzWsSns5zFHKclbMU3vUy2UaTXLB5G-wql1yyqoTFyDc9Bt0YHY9sdA$
> > <
> https://urldefense.com/v3/__https://github.com/apache/polaris/pull/3327__;!!Iz9xO38YGHZK!_dbBfwEnZEuPp_8qIuBDoYfWRgCZKiPDMg9ouDltqsHhr9hqCHVlF9IedSkNVBDWupbneU7inC9YK-THL-lTjWkklJ4Xf9Y$
> >
> ><
> >
> https://urldefense.com/v3/__https://github.com/apache/polaris/pull/3327__;!!Iz9xO38YGHZK!8vlQ2_OqjmZNIpbrTTluFFNKFCEeM87nr95MP5pMpMu5fFWVSoBZ3eb6-C2xbAGCG-ry5eaCO-pFLxl3IZAJGqY$
> >>
> >
> > I have been working with Prashant on an end-to-end telemetry and audit
> > trail tracking for Polaris.  It is documented here.
> >
> https://urldefense.com/v3/__https://docs.google.com/document/d/1Ehzvi5RNPs4hChkBFI6VD23myEqm-7sWW3d2kjmuYj8/edit?tab=t.0__;!!Iz9xO38YGHZK!65BVFQDe_GtKbDW8LkNdzVS-WYewzWsSns5zFHKclbMU3vUy2UaTXLB5G-wql1yyqoTFyDc9Bt11Y1rbNQ$
> > <
> https://urldefense.com/v3/__https://docs.google.com/document/d/1Ehzvi5RNPs4hChkBFI6VD23myEqm-7sWW3d2kjmuYj8/edit?tab=t.0__;!!Iz9xO38YGHZK!_dbBfwEnZEuPp_8qIuBDoYfWRgCZKiPDMg9ouDltqsHhr9hqCHVlF9IedSkNVBDWupbneU7inC9YK-THL-lTjWkk8MVJPlY$
> >
> ><
> >
> https://urldefense.com/v3/__https://docs.google.com/document/d/1Ehzvi5RNPs4hChkBFI6VD23myEqm-7sWW3d2kjmuYj8/edit?tab=t.0__;!!Iz9xO38YGHZK!8vlQ2_OqjmZNIpbrTTluFFNKFCEeM87nr95MP5pMpMu5fFWVSoBZ3eb6-C2xbAGCG-ry5eaCO-pFLxl3hYqTlcg$
> >>
> >
> > Based on Prashant’s initial feedback (parity with Apache Gravitino for
> > metrics reports), I have an initial PR here.
> >
> https://urldefense.com/v3/__https://github.com/apache/polaris/pull/3385__;!!Iz9xO38YGHZK!65BVFQDe_GtKbDW8LkNdzVS-WYewzWsSns5zFHKclbMU3vUy2UaTXLB5G-wql1yyqoTFyDc9Bt2W-eKl7A$
> > <
> https://urldefense.com/v3/__https://github.com/apache/polaris/pull/3385__;!!Iz9xO38YGHZK!_dbBfwEnZEuPp_8qIuBDoYfWRgCZKiPDMg9ouDltqsHhr9hqCHVlF9IedSkNVBDWupbneU7inC9YK-THL-lTjWkkyZhI0OE$
> >
> ><
> >
> https://urldefense.com/v3/__https://github.com/apache/polaris/pull/3385__;!!Iz9xO38YGHZK!8vlQ2_OqjmZNIpbrTTluFFNKFCEeM87nr95MP5pMpMu5fFWVSoBZ3eb6-C2xbAGCG-ry5eaCO-pFLxl39opCznI$
> >>
> >
> > I will be reaching out to various folks for review. I am open to breaking
> > down the PR into smaller logical groups of PR if it helps.
> >
> > I request you to look at the Google doc and let me know what you think.
> >
> > —
> > Anand
> >
> >
> >
> > --
> > Dmitri Bourlatchkov
> > Senior Staff Software Engineer, Dremio
> > Dremio.com
> > <
> https://urldefense.com/v3/__https://www.dremio.com/?utm_medium=email&utm_source=signature&utm_term=na&utm_content=email-signature&utm_campaign=email-signature__;!!Iz9xO38YGHZK!_dbBfwEnZEuPp_8qIuBDoYfWRgCZKiPDMg9ouDltqsHhr9hqCHVlF9IedSkNVBDWupbneU7inC9YK-THL-lTjWkkiOwjJc0$
> >/
> > Follow Us on LinkedIn
> > <
> https://urldefense.com/v3/__https://www.linkedin.com/company/dremio__;!!Iz9xO38YGHZK!_dbBfwEnZEuPp_8qIuBDoYfWRgCZKiPDMg9ouDltqsHhr9hqCHVlF9IedSkNVBDWupbneU7inC9YK-THL-lTjWkkCLjBQ2M$
> >/
> > Get Started
> > <
> https://urldefense.com/v3/__https://www.dremio.com/get-started/__;!!Iz9xO38YGHZK!_dbBfwEnZEuPp_8qIuBDoYfWRgCZKiPDMg9ouDltqsHhr9hqCHVlF9IedSkNVBDWupbneU7inC9YK-THL-lTjWkkoKQUMII$
> >
> >
> >
> >The Agentic Lakehouse
> > *The only lakehouse built for agents, managed by agents*
> >
> >
> >
>
>

Re: Polaris Telemetry and Audit Trail

[Anand Kumar Sankaran via dev <de...@polaris.apache.org>]

Hi Dmitri,

Following up on your feedback regarding the metrics schema separation approach. I've implemented the simplified design you suggested in https://github.com/apache/polaris/pull/3523:


 The metrics schema can now evolve independently from the entity schema. Let me know if you'd like any changes to this approach.

 Thanks,
  Anand



From: Dmitri Bourlatchkov <di...@apache.org>
Date: Thursday, February 5, 2026 at 2:57 PM
To: dev@polaris.apache.org <de...@polaris.apache.org>
Cc: Anand Kumar Sankaran <an...@workday.com>
Subject: Re: Polaris Telemetry and Audit Trail

This Message Is From an External Sender
This message came from outside your organization.
Report Suspicious<https://us-phishalarm-ewt.proofpoint.com/EWT/v1/Iz9xO38YGHZK!YhNDZAGr2cumY1iAus5GNbkIxSXoQaI552sXgYyv7Z9Wrvb7qeTskSy5WkXTVDJ_Bel2AbNh_fQVn-HS_UtOSdOtpsNfzhXDkK4fYWdTeoeBe70A6oQVGpAYT3SrbEvs$>

Hi Anand,

Re: MetricsPersistence Factory - TBH I do not think it is required. If we go with an MetricsPersistence impl. independent of the JDBC MetaStore, it could be injected into PersistingMetricsReporter directly by CDI.

Re: DataSource - for a start we could probably reuse the same DataSource as the JDBS MetaStore uses. In principle, we could introduce a separate DataSource, but I do not think we need to complicate the initial implementation with that. It can be added later, if required.

Re: schema / bootstrap - I'm thinking, indeed, of having a separate *.sql file for metrics DDL. Then we could add an option to the bootstrap command in the Admin Tool, or even a new command for creating / updating the metrics schema. This might require some refactoring, but I believe the bootstrap code needs some cleanup anyway.

The general idea is to allow the metrics schema to evolve independently. It's a new and actively developed feature, so having flexibility is going to be good for making progress / improvements, I hope.

Does that sound reasonable?

Thanks,
Dmitri.

On Thu, Feb 5, 2026 at 3:47 PM Anand Kumar Sankaran via dev <de...@polaris.apache.org>> wrote:
Hi Dmitri

The SPI is designed to be pluggable via MetaStoreManagerFactory.getOrCreateMetricsPersistence()<https://github.com/apache/polaris/pull/3385/changes#diff-0e34667d083d30cace067936b87c736201e5a8cb001aa0b12e203ee52cddae99R51<https://urldefense.com/v3/__https://github.com/apache/polaris/pull/3385/changes*diff-0e34667d083d30cace067936b87c736201e5a8cb001aa0b12e203ee52cddae99R51__;Iw!!Iz9xO38YGHZK!_K1mcH0_2x0nbU-mZlbBi3tk9QAGjB6VInPiqkQU21mpIP_ft2bXXyIiADNoAcotNr9AKqlphh65taUhcw$>>. The default returns MetricsPersistence.NOOP, and only JDBC with schema v4+ returns a functional implementation.

 Your proposal makes sense to me. To allow NoSQL Entity Persistence + JDBC Metrics Persistence would require:

     1. Separate metrics schema files (e.g., metrics-schema-v1.sql) that can be bootstrapped independently
     2. Separate datasource configuration for metrics (e.g., polaris.metrics.jdbc.*)
     3. A dedicated `MetricsPersistenceFactory` interface (separate from MetaStoreManagerFactory)
     4. Updates to the Admin Tool bootstrap workflow to handle both schemas

Does that sound reasonable?  If we do this, all this work should go into https://github.com/apache/polaris/pull/3523<https://urldefense.com/v3/__https://github.com/apache/polaris/pull/3523__;!!Iz9xO38YGHZK!_K1mcH0_2x0nbU-mZlbBi3tk9QAGjB6VInPiqkQU21mpIP_ft2bXXyIiADNoAcotNr9AKqlphh4ZISZcDQ$> as one atomic commit, correct?

—
Anand



From: Dmitri Bourlatchkov <di...@apache.org>>
Date: Thursday, February 5, 2026 at 8:28 AM
To: Anand Kumar Sankaran <an...@workday.com>>
Cc: dev@polaris.apache.org<ma...@polaris.apache.org> <de...@polaris.apache.org>>
Subject: Re: Polaris Telemetry and Audit Trail

This Message Is From an External Sender
This message came from outside your organization.
Report Suspicious<https://us-phishalarm-ewt.proofpoint.com/EWT/v1/Iz9xO38YGHZK!YhNDZAGr2cumbRcPlIOttKwvGr4f38TF-J2-LmOw8H1KsFpWaM3W5g28Oq5qW2gnw2-WHz0rsWhm7F7hSGykA9l01LKYGuCcCkexkHp1sUssM_IegCtjk_id9s891orv$>


Hi Anand,

Thanks for driving the SPI definition for scan/commits metrics.

I did not think from this angle before, but now I suppose it might be worth
introducing a separate SQL schema for metrics, detached from the main JDBC
Persistence schema.

My rationale is that the SPI may be exercised independently of the JDBC
Entity Persistence. I can imagine people using NoSQL Entity Persistence
plus JDBC Scan Metrics Persistence in the same deployment.

Using a separate schema will add further work on the Bootstrap workflow
(Admin Tool), but I hope the end result is going to be more convenient and
versatile for end users.

WDYT?

Thanks,
Dmitri.

On Thu, Jan 29, 2026 at 1:49 PM Anand Kumar Sankaran <
anand.sankaran@workday.com<ma...@workday.com>> wrote:

> Hi Dmitri,
>
> Thank you for taking the time to review. I had pulled out the schema in
> its own PR (based on comments) here:
> https://urldefense.com/v3/__https://github.com/apache/polaris/pull/3523__;!!Iz9xO38YGHZK!65BVFQDe_GtKbDW8LkNdzVS-WYewzWsSns5zFHKclbMU3vUy2UaTXLB5G-wql1yyqoTFyDc9Bt3GRsh6AA$
>
> I will update the document with the SPI and carve out another PR that
> focuses on the SPI.
>
> —
> Anand
>
> *From: *Dmitri Bourlatchkov <dm...@dremio.com>>
> *Date: *Thursday, January 29, 2026 at 10:43 AM
> *To: *dev@polaris.apache.org<ma...@polaris.apache.org> <de...@polaris.apache.org>>
> *Cc: *Anand Kumar Sankaran <an...@workday.com>>
> *Subject: *Re: Polaris Telemetry and Audit Trail
>
> This Message Is From an External Sender
> This message came from outside your organization.
> Report Suspicious
> <https://us-phishalarm-ewt.proofpoint.com/EWT/v1/Iz9xO38YGHZK!YhNDZAGr2cumY1cAlUeoPrU2eNUPz9Wxq2PaG1t99vEN3v60gucxC3Qsk-xK09RndcW46WtNjMMjQZbaQ495h1D1g293F_uqLhPM8gDMlA6xRmo5b5uqrSh2v7QKHrhQ$ >
>
> Hi Anand,
>
> Thanks for making a proposal doc and starting this thread. Apologies for
> late comments.
>
> As you know I already reviewed a couple of related PRs, which had a
> smaller impact on the codebase, but now, I believe, we're coming to the
> core of this feature and it might need a deeper discussion.
>
> For the sake of awareness of other project members, I'd like to
> highlight some aspects by email.
>
> * Storing scan metrics in RDBMS in general looks like a
> reasonable approach to me. However, Polaris Persistence is an extensible
> mechanism. Other backends are possible, including privately developed
> backends.
>
> From this POV, I believe it might be preferable to start not with an RDBMS
> schema, but with a java SPI outlining expected read/write operations and
> the data model. This will make it easier to assess impact on other
> Persistence implementations. Naturally, Polaris service code will have to
> be able to work seamlessly regardless of the backend impl. (including the
> do-nothing impl.).
>
> I see that some connection between the SQL schema and java code is made in
> PR [3385], but it does not appear to offer a clean SPI that could be
> implemented by different Persistence backends. Would it be ok from your
> perspective to make another PR with just SPI code?
>
> * Regarding ad-hoc SQL queries - it is certainly a valid use case for
> custom code to query the Polaris database directly. However, those ad-hoc
> queries may require certain indexes, which may not have a relevant use case
> in OSS code. In that case, I believe it would be preferable to manage the
> extra indexes in custom code (not in Polaris). Whether this is the case or
> not currently, I cannot say with certainty, so apologies if this is a false
> alarm... Having the java SPI available would be a huge aid to reasoning
> about the proposed SQL schema, I think.
>
> That said, I do not mean to complicate the development of this feature,
> just trying to structure it in a way that is hopefully easy to understand
> (including myself), maintain and expand later. If I missed something,
> please feel free to point it out.
>
> [3385] https://urldefense.com/v3/__https://github.com/apache/polaris/pull/3385__;!!Iz9xO38YGHZK!65BVFQDe_GtKbDW8LkNdzVS-WYewzWsSns5zFHKclbMU3vUy2UaTXLB5G-wql1yyqoTFyDc9Bt2W-eKl7A$
> <https://urldefense.com/v3/__https://github.com/apache/polaris/pull/3385__;!!Iz9xO38YGHZK!_dbBfwEnZEuPp_8qIuBDoYfWRgCZKiPDMg9ouDltqsHhr9hqCHVlF9IedSkNVBDWupbneU7inC9YK-THL-lTjWkkyZhI0OE$>
>
>Thanks,
> Dmitri.
>
> On Wed, Jan 14, 2026 at 10:05 PM Anand Kumar Sankaran via dev <
> dev@polaris.apache.org<ma...@polaris.apache.org>> wrote:
>
> Hi Yufei,
>
> Gave commenting privileges to all.
>
> Thanks.
>
> —
> Anand
>
> From: Yufei Gu <fl...@gmail.com>>
> Date: Wednesday, January 14, 2026 at 6:13 PM
> To: dev@polaris.apache.org<ma...@polaris.apache.org> <de...@polaris.apache.org>>
> Cc: Anand Kumar Sankaran <an...@workday.com>>
> Subject: Re: Polaris Telemetry and Audit Trail
>
> This Message Is From an External Sender
> This message came from outside your organization.
> Report Suspicious<
> https://us-phishalarm-ewt.proofpoint.com/EWT/v1/Iz9xO38YGHZK!YhNDZABkHi1B6hyPVYUK0T-K5j4Aoqdrlu07UjBpJcpjOsz7Ie7d_DhPl-ywi50dfQz4O6LL-AHLCOQTQ0X7Le8yD2cF0ogquGFB8Aw8uQLgB0RL6Ezpa-pLDPA4ziXA$
> >
>
> Hi Anand, thanks for the design doc and PR. Can you allow comments in the
> doc so that people can chime in? Thanks!
>
> Yufei
>
>
> On Sat, Jan 10, 2026 at 9:09 AM Anand Kumar Sankaran via dev <
> dev@polaris.apache.org<ma...@polaris.apache.org>>> wrote:
> Hi all
>
> My first PR for adding AWS STS Session Tags support for credential vending
> was merged. https://urldefense.com/v3/__https://github.com/apache/polaris/pull/3327__;!!Iz9xO38YGHZK!65BVFQDe_GtKbDW8LkNdzVS-WYewzWsSns5zFHKclbMU3vUy2UaTXLB5G-wql1yyqoTFyDc9Bt0YHY9sdA$
> <https://urldefense.com/v3/__https://github.com/apache/polaris/pull/3327__;!!Iz9xO38YGHZK!_dbBfwEnZEuPp_8qIuBDoYfWRgCZKiPDMg9ouDltqsHhr9hqCHVlF9IedSkNVBDWupbneU7inC9YK-THL-lTjWkklJ4Xf9Y$>
><
> https://urldefense.com/v3/__https://github.com/apache/polaris/pull/3327__;!!Iz9xO38YGHZK!8vlQ2_OqjmZNIpbrTTluFFNKFCEeM87nr95MP5pMpMu5fFWVSoBZ3eb6-C2xbAGCG-ry5eaCO-pFLxl3IZAJGqY$
>>
>
> I have been working with Prashant on an end-to-end telemetry and audit
> trail tracking for Polaris.  It is documented here.
> https://urldefense.com/v3/__https://docs.google.com/document/d/1Ehzvi5RNPs4hChkBFI6VD23myEqm-7sWW3d2kjmuYj8/edit?tab=t.0__;!!Iz9xO38YGHZK!65BVFQDe_GtKbDW8LkNdzVS-WYewzWsSns5zFHKclbMU3vUy2UaTXLB5G-wql1yyqoTFyDc9Bt11Y1rbNQ$
> <https://urldefense.com/v3/__https://docs.google.com/document/d/1Ehzvi5RNPs4hChkBFI6VD23myEqm-7sWW3d2kjmuYj8/edit?tab=t.0__;!!Iz9xO38YGHZK!_dbBfwEnZEuPp_8qIuBDoYfWRgCZKiPDMg9ouDltqsHhr9hqCHVlF9IedSkNVBDWupbneU7inC9YK-THL-lTjWkk8MVJPlY$>
><
> https://urldefense.com/v3/__https://docs.google.com/document/d/1Ehzvi5RNPs4hChkBFI6VD23myEqm-7sWW3d2kjmuYj8/edit?tab=t.0__;!!Iz9xO38YGHZK!8vlQ2_OqjmZNIpbrTTluFFNKFCEeM87nr95MP5pMpMu5fFWVSoBZ3eb6-C2xbAGCG-ry5eaCO-pFLxl3hYqTlcg$
>>
>
> Based on Prashant’s initial feedback (parity with Apache Gravitino for
> metrics reports), I have an initial PR here.
> https://urldefense.com/v3/__https://github.com/apache/polaris/pull/3385__;!!Iz9xO38YGHZK!65BVFQDe_GtKbDW8LkNdzVS-WYewzWsSns5zFHKclbMU3vUy2UaTXLB5G-wql1yyqoTFyDc9Bt2W-eKl7A$
> <https://urldefense.com/v3/__https://github.com/apache/polaris/pull/3385__;!!Iz9xO38YGHZK!_dbBfwEnZEuPp_8qIuBDoYfWRgCZKiPDMg9ouDltqsHhr9hqCHVlF9IedSkNVBDWupbneU7inC9YK-THL-lTjWkkyZhI0OE$>
><
> https://urldefense.com/v3/__https://github.com/apache/polaris/pull/3385__;!!Iz9xO38YGHZK!8vlQ2_OqjmZNIpbrTTluFFNKFCEeM87nr95MP5pMpMu5fFWVSoBZ3eb6-C2xbAGCG-ry5eaCO-pFLxl39opCznI$
>>
>
> I will be reaching out to various folks for review. I am open to breaking
> down the PR into smaller logical groups of PR if it helps.
>
> I request you to look at the Google doc and let me know what you think.
>
> —
> Anand
>
>
>
> --
> Dmitri Bourlatchkov
> Senior Staff Software Engineer, Dremio
> Dremio.com
> <https://urldefense.com/v3/__https://www.dremio.com/?utm_medium=email&utm_source=signature&utm_term=na&utm_content=email-signature&utm_campaign=email-signature__;!!Iz9xO38YGHZK!_dbBfwEnZEuPp_8qIuBDoYfWRgCZKiPDMg9ouDltqsHhr9hqCHVlF9IedSkNVBDWupbneU7inC9YK-THL-lTjWkkiOwjJc0$>/
> Follow Us on LinkedIn
> <https://urldefense.com/v3/__https://www.linkedin.com/company/dremio__;!!Iz9xO38YGHZK!_dbBfwEnZEuPp_8qIuBDoYfWRgCZKiPDMg9ouDltqsHhr9hqCHVlF9IedSkNVBDWupbneU7inC9YK-THL-lTjWkkCLjBQ2M$>/
> Get Started
> <https://urldefense.com/v3/__https://www.dremio.com/get-started/__;!!Iz9xO38YGHZK!_dbBfwEnZEuPp_8qIuBDoYfWRgCZKiPDMg9ouDltqsHhr9hqCHVlF9IedSkNVBDWupbneU7inC9YK-THL-lTjWkkoKQUMII$>
>
>
>The Agentic Lakehouse
> *The only lakehouse built for agents, managed by agents*
>
>
>

Re: Polaris Telemetry and Audit Trail

[Dmitri Bourlatchkov <di...@apache.org>]

Hi Anand,

Re: MetricsPersistence Factory - TBH I do not think it is required. If we
go with an MetricsPersistence impl. independent of the JDBC MetaStore, it
could be injected into PersistingMetricsReporter directly by CDI.

Re: DataSource - for a start we could probably reuse the same DataSource as
the JDBS MetaStore uses. In principle, we could introduce a separate
DataSource, but I do not think we need to complicate the initial
implementation with that. It can be added later, if required.

Re: schema / bootstrap - I'm thinking, indeed, of having a separate *.sql
file for metrics DDL. Then we could add an option to the bootstrap command
in the Admin Tool, or even a new command for creating / updating the
metrics schema. This might require some refactoring, but I believe the
bootstrap code needs some cleanup anyway.

The general idea is to allow the metrics schema to evolve independently.
It's a new and actively developed feature, so having flexibility is going
to be good for making progress / improvements, I hope.

Does that sound reasonable?

Thanks,
Dmitri.

On Thu, Feb 5, 2026 at 3:47 PM Anand Kumar Sankaran via dev <
dev@polaris.apache.org> wrote:

> Hi Dmitri
>
> The SPI is designed to be pluggable via
> MetaStoreManagerFactory.getOrCreateMetricsPersistence()<
> https://github.com/apache/polaris/pull/3385/changes#diff-0e34667d083d30cace067936b87c736201e5a8cb001aa0b12e203ee52cddae99R51>.
> The default returns MetricsPersistence.NOOP, and only JDBC with schema v4+
> returns a functional implementation.
>
>  Your proposal makes sense to me. To allow NoSQL Entity Persistence + JDBC
> Metrics Persistence would require:
>
>      1. Separate metrics schema files (e.g., metrics-schema-v1.sql) that
> can be bootstrapped independently
>      2. Separate datasource configuration for metrics (e.g.,
> polaris.metrics.jdbc.*)
>      3. A dedicated `MetricsPersistenceFactory` interface (separate from
> MetaStoreManagerFactory)
>      4. Updates to the Admin Tool bootstrap workflow to handle both schemas
>
> Does that sound reasonable?  If we do this, all this work should go into
> https://github.com/apache/polaris/pull/3523 as one atomic commit, correct?
>
> —
> Anand
>
>
>
> From: Dmitri Bourlatchkov <di...@apache.org>
> Date: Thursday, February 5, 2026 at 8:28 AM
> To: Anand Kumar Sankaran <an...@workday.com>
> Cc: dev@polaris.apache.org <de...@polaris.apache.org>
> Subject: Re: Polaris Telemetry and Audit Trail
>
> This Message Is From an External Sender
> This message came from outside your organization.
> Report Suspicious<
> https://us-phishalarm-ewt.proofpoint.com/EWT/v1/Iz9xO38YGHZK!YhNDZAGr2cumbRcPlIOttKwvGr4f38TF-J2-LmOw8H1KsFpWaM3W5g28Oq5qW2gnw2-WHz0rsWhm7F7hSGykA9l01LKYGuCcCkexkHp1sUssM_IegCtjk_id9s891orv$
> >
>
>
> Hi Anand,
>
> Thanks for driving the SPI definition for scan/commits metrics.
>
> I did not think from this angle before, but now I suppose it might be worth
> introducing a separate SQL schema for metrics, detached from the main JDBC
> Persistence schema.
>
> My rationale is that the SPI may be exercised independently of the JDBC
> Entity Persistence. I can imagine people using NoSQL Entity Persistence
> plus JDBC Scan Metrics Persistence in the same deployment.
>
> Using a separate schema will add further work on the Bootstrap workflow
> (Admin Tool), but I hope the end result is going to be more convenient and
> versatile for end users.
>
> WDYT?
>
> Thanks,
> Dmitri.
>
> On Thu, Jan 29, 2026 at 1:49 PM Anand Kumar Sankaran <
> anand.sankaran@workday.com> wrote:
>
> > Hi Dmitri,
> >
> > Thank you for taking the time to review. I had pulled out the schema in
> > its own PR (based on comments) here:
> >
> https://urldefense.com/v3/__https://github.com/apache/polaris/pull/3523__;!!Iz9xO38YGHZK!65BVFQDe_GtKbDW8LkNdzVS-WYewzWsSns5zFHKclbMU3vUy2UaTXLB5G-wql1yyqoTFyDc9Bt3GRsh6AA$
> >
> > I will update the document with the SPI and carve out another PR that
> > focuses on the SPI.
> >
> > —
> > Anand
> >
> > *From: *Dmitri Bourlatchkov <dm...@dremio.com>
> > *Date: *Thursday, January 29, 2026 at 10:43 AM
> > *To: *dev@polaris.apache.org <de...@polaris.apache.org>
> > *Cc: *Anand Kumar Sankaran <an...@workday.com>
> > *Subject: *Re: Polaris Telemetry and Audit Trail
> >
> > This Message Is From an External Sender
> > This message came from outside your organization.
> > Report Suspicious
> > <
> https://us-phishalarm-ewt.proofpoint.com/EWT/v1/Iz9xO38YGHZK!YhNDZAGr2cumY1cAlUeoPrU2eNUPz9Wxq2PaG1t99vEN3v60gucxC3Qsk-xK09RndcW46WtNjMMjQZbaQ495h1D1g293F_uqLhPM8gDMlA6xRmo5b5uqrSh2v7QKHrhQ$
> >
> >
> > Hi Anand,
> >
> > Thanks for making a proposal doc and starting this thread. Apologies for
> > late comments.
> >
> > As you know I already reviewed a couple of related PRs, which had a
> > smaller impact on the codebase, but now, I believe, we're coming to the
> > core of this feature and it might need a deeper discussion.
> >
> > For the sake of awareness of other project members, I'd like to
> > highlight some aspects by email.
> >
> > * Storing scan metrics in RDBMS in general looks like a
> > reasonable approach to me. However, Polaris Persistence is an extensible
> > mechanism. Other backends are possible, including privately developed
> > backends.
> >
> > From this POV, I believe it might be preferable to start not with an
> RDBMS
> > schema, but with a java SPI outlining expected read/write operations and
> > the data model. This will make it easier to assess impact on other
> > Persistence implementations. Naturally, Polaris service code will have to
> > be able to work seamlessly regardless of the backend impl. (including the
> > do-nothing impl.).
> >
> > I see that some connection between the SQL schema and java code is made
> in
> > PR [3385], but it does not appear to offer a clean SPI that could be
> > implemented by different Persistence backends. Would it be ok from your
> > perspective to make another PR with just SPI code?
> >
> > * Regarding ad-hoc SQL queries - it is certainly a valid use case for
> > custom code to query the Polaris database directly. However, those ad-hoc
> > queries may require certain indexes, which may not have a relevant use
> case
> > in OSS code. In that case, I believe it would be preferable to manage the
> > extra indexes in custom code (not in Polaris). Whether this is the case
> or
> > not currently, I cannot say with certainty, so apologies if this is a
> false
> > alarm... Having the java SPI available would be a huge aid to reasoning
> > about the proposed SQL schema, I think.
> >
> > That said, I do not mean to complicate the development of this feature,
> > just trying to structure it in a way that is hopefully easy to understand
> > (including myself), maintain and expand later. If I missed something,
> > please feel free to point it out.
> >
> > [3385]
> https://urldefense.com/v3/__https://github.com/apache/polaris/pull/3385__;!!Iz9xO38YGHZK!65BVFQDe_GtKbDW8LkNdzVS-WYewzWsSns5zFHKclbMU3vUy2UaTXLB5G-wql1yyqoTFyDc9Bt2W-eKl7A$
> > <
> https://urldefense.com/v3/__https://github.com/apache/polaris/pull/3385__;!!Iz9xO38YGHZK!_dbBfwEnZEuPp_8qIuBDoYfWRgCZKiPDMg9ouDltqsHhr9hqCHVlF9IedSkNVBDWupbneU7inC9YK-THL-lTjWkkyZhI0OE$
> >
> >
> >Thanks,
> > Dmitri.
> >
> > On Wed, Jan 14, 2026 at 10:05 PM Anand Kumar Sankaran via dev <
> > dev@polaris.apache.org> wrote:
> >
> > Hi Yufei,
> >
> > Gave commenting privileges to all.
> >
> > Thanks.
> >
> > —
> > Anand
> >
> > From: Yufei Gu <fl...@gmail.com>
> > Date: Wednesday, January 14, 2026 at 6:13 PM
> > To: dev@polaris.apache.org <de...@polaris.apache.org>
> > Cc: Anand Kumar Sankaran <an...@workday.com>
> > Subject: Re: Polaris Telemetry and Audit Trail
> >
> > This Message Is From an External Sender
> > This message came from outside your organization.
> > Report Suspicious<
> >
> https://us-phishalarm-ewt.proofpoint.com/EWT/v1/Iz9xO38YGHZK!YhNDZABkHi1B6hyPVYUK0T-K5j4Aoqdrlu07UjBpJcpjOsz7Ie7d_DhPl-ywi50dfQz4O6LL-AHLCOQTQ0X7Le8yD2cF0ogquGFB8Aw8uQLgB0RL6Ezpa-pLDPA4ziXA$
> > >
> >
> > Hi Anand, thanks for the design doc and PR. Can you allow comments in the
> > doc so that people can chime in? Thanks!
> >
> > Yufei
> >
> >
> > On Sat, Jan 10, 2026 at 9:09 AM Anand Kumar Sankaran via dev <
> > dev@polaris.apache.org<ma...@polaris.apache.org>> wrote:
> > Hi all
> >
> > My first PR for adding AWS STS Session Tags support for credential
> vending
> > was merged.
> https://urldefense.com/v3/__https://github.com/apache/polaris/pull/3327__;!!Iz9xO38YGHZK!65BVFQDe_GtKbDW8LkNdzVS-WYewzWsSns5zFHKclbMU3vUy2UaTXLB5G-wql1yyqoTFyDc9Bt0YHY9sdA$
> > <
> https://urldefense.com/v3/__https://github.com/apache/polaris/pull/3327__;!!Iz9xO38YGHZK!_dbBfwEnZEuPp_8qIuBDoYfWRgCZKiPDMg9ouDltqsHhr9hqCHVlF9IedSkNVBDWupbneU7inC9YK-THL-lTjWkklJ4Xf9Y$
> >
> ><
> >
> https://urldefense.com/v3/__https://github.com/apache/polaris/pull/3327__;!!Iz9xO38YGHZK!8vlQ2_OqjmZNIpbrTTluFFNKFCEeM87nr95MP5pMpMu5fFWVSoBZ3eb6-C2xbAGCG-ry5eaCO-pFLxl3IZAJGqY$
> >>
> >
> > I have been working with Prashant on an end-to-end telemetry and audit
> > trail tracking for Polaris.  It is documented here.
> >
> https://urldefense.com/v3/__https://docs.google.com/document/d/1Ehzvi5RNPs4hChkBFI6VD23myEqm-7sWW3d2kjmuYj8/edit?tab=t.0__;!!Iz9xO38YGHZK!65BVFQDe_GtKbDW8LkNdzVS-WYewzWsSns5zFHKclbMU3vUy2UaTXLB5G-wql1yyqoTFyDc9Bt11Y1rbNQ$
> > <
> https://urldefense.com/v3/__https://docs.google.com/document/d/1Ehzvi5RNPs4hChkBFI6VD23myEqm-7sWW3d2kjmuYj8/edit?tab=t.0__;!!Iz9xO38YGHZK!_dbBfwEnZEuPp_8qIuBDoYfWRgCZKiPDMg9ouDltqsHhr9hqCHVlF9IedSkNVBDWupbneU7inC9YK-THL-lTjWkk8MVJPlY$
> >
> ><
> >
> https://urldefense.com/v3/__https://docs.google.com/document/d/1Ehzvi5RNPs4hChkBFI6VD23myEqm-7sWW3d2kjmuYj8/edit?tab=t.0__;!!Iz9xO38YGHZK!8vlQ2_OqjmZNIpbrTTluFFNKFCEeM87nr95MP5pMpMu5fFWVSoBZ3eb6-C2xbAGCG-ry5eaCO-pFLxl3hYqTlcg$
> >>
> >
> > Based on Prashant’s initial feedback (parity with Apache Gravitino for
> > metrics reports), I have an initial PR here.
> >
> https://urldefense.com/v3/__https://github.com/apache/polaris/pull/3385__;!!Iz9xO38YGHZK!65BVFQDe_GtKbDW8LkNdzVS-WYewzWsSns5zFHKclbMU3vUy2UaTXLB5G-wql1yyqoTFyDc9Bt2W-eKl7A$
> > <
> https://urldefense.com/v3/__https://github.com/apache/polaris/pull/3385__;!!Iz9xO38YGHZK!_dbBfwEnZEuPp_8qIuBDoYfWRgCZKiPDMg9ouDltqsHhr9hqCHVlF9IedSkNVBDWupbneU7inC9YK-THL-lTjWkkyZhI0OE$
> >
> ><
> >
> https://urldefense.com/v3/__https://github.com/apache/polaris/pull/3385__;!!Iz9xO38YGHZK!8vlQ2_OqjmZNIpbrTTluFFNKFCEeM87nr95MP5pMpMu5fFWVSoBZ3eb6-C2xbAGCG-ry5eaCO-pFLxl39opCznI$
> >>
> >
> > I will be reaching out to various folks for review. I am open to breaking
> > down the PR into smaller logical groups of PR if it helps.
> >
> > I request you to look at the Google doc and let me know what you think.
> >
> > —
> > Anand
> >
> >
> >
> > --
> > Dmitri Bourlatchkov
> > Senior Staff Software Engineer, Dremio
> > Dremio.com
> > <
> https://urldefense.com/v3/__https://www.dremio.com/?utm_medium=email&utm_source=signature&utm_term=na&utm_content=email-signature&utm_campaign=email-signature__;!!Iz9xO38YGHZK!_dbBfwEnZEuPp_8qIuBDoYfWRgCZKiPDMg9ouDltqsHhr9hqCHVlF9IedSkNVBDWupbneU7inC9YK-THL-lTjWkkiOwjJc0$
> >/
> > Follow Us on LinkedIn
> > <
> https://urldefense.com/v3/__https://www.linkedin.com/company/dremio__;!!Iz9xO38YGHZK!_dbBfwEnZEuPp_8qIuBDoYfWRgCZKiPDMg9ouDltqsHhr9hqCHVlF9IedSkNVBDWupbneU7inC9YK-THL-lTjWkkCLjBQ2M$
> >/
> > Get Started
> > <
> https://urldefense.com/v3/__https://www.dremio.com/get-started/__;!!Iz9xO38YGHZK!_dbBfwEnZEuPp_8qIuBDoYfWRgCZKiPDMg9ouDltqsHhr9hqCHVlF9IedSkNVBDWupbneU7inC9YK-THL-lTjWkkoKQUMII$
> >
> >
> >
> >The Agentic Lakehouse
> > *The only lakehouse built for agents, managed by agents*
> >
> >
> >
>
>

Re: Polaris Telemetry and Audit Trail

[Anand Kumar Sankaran via dev <de...@polaris.apache.org>]

Hi Dmitri

The SPI is designed to be pluggable via MetaStoreManagerFactory.getOrCreateMetricsPersistence()<https://github.com/apache/polaris/pull/3385/changes#diff-0e34667d083d30cace067936b87c736201e5a8cb001aa0b12e203ee52cddae99R51>. The default returns MetricsPersistence.NOOP, and only JDBC with schema v4+ returns a functional implementation.

 Your proposal makes sense to me. To allow NoSQL Entity Persistence + JDBC Metrics Persistence would require:

     1. Separate metrics schema files (e.g., metrics-schema-v1.sql) that can be bootstrapped independently
     2. Separate datasource configuration for metrics (e.g., polaris.metrics.jdbc.*)
     3. A dedicated `MetricsPersistenceFactory` interface (separate from MetaStoreManagerFactory)
     4. Updates to the Admin Tool bootstrap workflow to handle both schemas

Does that sound reasonable?  If we do this, all this work should go into https://github.com/apache/polaris/pull/3523 as one atomic commit, correct?

—
Anand



From: Dmitri Bourlatchkov <di...@apache.org>
Date: Thursday, February 5, 2026 at 8:28 AM
To: Anand Kumar Sankaran <an...@workday.com>
Cc: dev@polaris.apache.org <de...@polaris.apache.org>
Subject: Re: Polaris Telemetry and Audit Trail

This Message Is From an External Sender
This message came from outside your organization.
Report Suspicious<https://us-phishalarm-ewt.proofpoint.com/EWT/v1/Iz9xO38YGHZK!YhNDZAGr2cumbRcPlIOttKwvGr4f38TF-J2-LmOw8H1KsFpWaM3W5g28Oq5qW2gnw2-WHz0rsWhm7F7hSGykA9l01LKYGuCcCkexkHp1sUssM_IegCtjk_id9s891orv$>


Hi Anand,

Thanks for driving the SPI definition for scan/commits metrics.

I did not think from this angle before, but now I suppose it might be worth
introducing a separate SQL schema for metrics, detached from the main JDBC
Persistence schema.

My rationale is that the SPI may be exercised independently of the JDBC
Entity Persistence. I can imagine people using NoSQL Entity Persistence
plus JDBC Scan Metrics Persistence in the same deployment.

Using a separate schema will add further work on the Bootstrap workflow
(Admin Tool), but I hope the end result is going to be more convenient and
versatile for end users.

WDYT?

Thanks,
Dmitri.

On Thu, Jan 29, 2026 at 1:49 PM Anand Kumar Sankaran <
anand.sankaran@workday.com> wrote:

> Hi Dmitri,
>
> Thank you for taking the time to review. I had pulled out the schema in
> its own PR (based on comments) here:
> https://urldefense.com/v3/__https://github.com/apache/polaris/pull/3523__;!!Iz9xO38YGHZK!65BVFQDe_GtKbDW8LkNdzVS-WYewzWsSns5zFHKclbMU3vUy2UaTXLB5G-wql1yyqoTFyDc9Bt3GRsh6AA$
>
> I will update the document with the SPI and carve out another PR that
> focuses on the SPI.
>
> —
> Anand
>
> *From: *Dmitri Bourlatchkov <dm...@dremio.com>
> *Date: *Thursday, January 29, 2026 at 10:43 AM
> *To: *dev@polaris.apache.org <de...@polaris.apache.org>
> *Cc: *Anand Kumar Sankaran <an...@workday.com>
> *Subject: *Re: Polaris Telemetry and Audit Trail
>
> This Message Is From an External Sender
> This message came from outside your organization.
> Report Suspicious
> <https://us-phishalarm-ewt.proofpoint.com/EWT/v1/Iz9xO38YGHZK!YhNDZAGr2cumY1cAlUeoPrU2eNUPz9Wxq2PaG1t99vEN3v60gucxC3Qsk-xK09RndcW46WtNjMMjQZbaQ495h1D1g293F_uqLhPM8gDMlA6xRmo5b5uqrSh2v7QKHrhQ$ >
>
> Hi Anand,
>
> Thanks for making a proposal doc and starting this thread. Apologies for
> late comments.
>
> As you know I already reviewed a couple of related PRs, which had a
> smaller impact on the codebase, but now, I believe, we're coming to the
> core of this feature and it might need a deeper discussion.
>
> For the sake of awareness of other project members, I'd like to
> highlight some aspects by email.
>
> * Storing scan metrics in RDBMS in general looks like a
> reasonable approach to me. However, Polaris Persistence is an extensible
> mechanism. Other backends are possible, including privately developed
> backends.
>
> From this POV, I believe it might be preferable to start not with an RDBMS
> schema, but with a java SPI outlining expected read/write operations and
> the data model. This will make it easier to assess impact on other
> Persistence implementations. Naturally, Polaris service code will have to
> be able to work seamlessly regardless of the backend impl. (including the
> do-nothing impl.).
>
> I see that some connection between the SQL schema and java code is made in
> PR [3385], but it does not appear to offer a clean SPI that could be
> implemented by different Persistence backends. Would it be ok from your
> perspective to make another PR with just SPI code?
>
> * Regarding ad-hoc SQL queries - it is certainly a valid use case for
> custom code to query the Polaris database directly. However, those ad-hoc
> queries may require certain indexes, which may not have a relevant use case
> in OSS code. In that case, I believe it would be preferable to manage the
> extra indexes in custom code (not in Polaris). Whether this is the case or
> not currently, I cannot say with certainty, so apologies if this is a false
> alarm... Having the java SPI available would be a huge aid to reasoning
> about the proposed SQL schema, I think.
>
> That said, I do not mean to complicate the development of this feature,
> just trying to structure it in a way that is hopefully easy to understand
> (including myself), maintain and expand later. If I missed something,
> please feel free to point it out.
>
> [3385] https://urldefense.com/v3/__https://github.com/apache/polaris/pull/3385__;!!Iz9xO38YGHZK!65BVFQDe_GtKbDW8LkNdzVS-WYewzWsSns5zFHKclbMU3vUy2UaTXLB5G-wql1yyqoTFyDc9Bt2W-eKl7A$
> <https://urldefense.com/v3/__https://github.com/apache/polaris/pull/3385__;!!Iz9xO38YGHZK!_dbBfwEnZEuPp_8qIuBDoYfWRgCZKiPDMg9ouDltqsHhr9hqCHVlF9IedSkNVBDWupbneU7inC9YK-THL-lTjWkkyZhI0OE$>
>
>Thanks,
> Dmitri.
>
> On Wed, Jan 14, 2026 at 10:05 PM Anand Kumar Sankaran via dev <
> dev@polaris.apache.org> wrote:
>
> Hi Yufei,
>
> Gave commenting privileges to all.
>
> Thanks.
>
> —
> Anand
>
> From: Yufei Gu <fl...@gmail.com>
> Date: Wednesday, January 14, 2026 at 6:13 PM
> To: dev@polaris.apache.org <de...@polaris.apache.org>
> Cc: Anand Kumar Sankaran <an...@workday.com>
> Subject: Re: Polaris Telemetry and Audit Trail
>
> This Message Is From an External Sender
> This message came from outside your organization.
> Report Suspicious<
> https://us-phishalarm-ewt.proofpoint.com/EWT/v1/Iz9xO38YGHZK!YhNDZABkHi1B6hyPVYUK0T-K5j4Aoqdrlu07UjBpJcpjOsz7Ie7d_DhPl-ywi50dfQz4O6LL-AHLCOQTQ0X7Le8yD2cF0ogquGFB8Aw8uQLgB0RL6Ezpa-pLDPA4ziXA$
> >
>
> Hi Anand, thanks for the design doc and PR. Can you allow comments in the
> doc so that people can chime in? Thanks!
>
> Yufei
>
>
> On Sat, Jan 10, 2026 at 9:09 AM Anand Kumar Sankaran via dev <
> dev@polaris.apache.org<ma...@polaris.apache.org>> wrote:
> Hi all
>
> My first PR for adding AWS STS Session Tags support for credential vending
> was merged. https://urldefense.com/v3/__https://github.com/apache/polaris/pull/3327__;!!Iz9xO38YGHZK!65BVFQDe_GtKbDW8LkNdzVS-WYewzWsSns5zFHKclbMU3vUy2UaTXLB5G-wql1yyqoTFyDc9Bt0YHY9sdA$
> <https://urldefense.com/v3/__https://github.com/apache/polaris/pull/3327__;!!Iz9xO38YGHZK!_dbBfwEnZEuPp_8qIuBDoYfWRgCZKiPDMg9ouDltqsHhr9hqCHVlF9IedSkNVBDWupbneU7inC9YK-THL-lTjWkklJ4Xf9Y$>
><
> https://urldefense.com/v3/__https://github.com/apache/polaris/pull/3327__;!!Iz9xO38YGHZK!8vlQ2_OqjmZNIpbrTTluFFNKFCEeM87nr95MP5pMpMu5fFWVSoBZ3eb6-C2xbAGCG-ry5eaCO-pFLxl3IZAJGqY$
>>
>
> I have been working with Prashant on an end-to-end telemetry and audit
> trail tracking for Polaris.  It is documented here.
> https://urldefense.com/v3/__https://docs.google.com/document/d/1Ehzvi5RNPs4hChkBFI6VD23myEqm-7sWW3d2kjmuYj8/edit?tab=t.0__;!!Iz9xO38YGHZK!65BVFQDe_GtKbDW8LkNdzVS-WYewzWsSns5zFHKclbMU3vUy2UaTXLB5G-wql1yyqoTFyDc9Bt11Y1rbNQ$
> <https://urldefense.com/v3/__https://docs.google.com/document/d/1Ehzvi5RNPs4hChkBFI6VD23myEqm-7sWW3d2kjmuYj8/edit?tab=t.0__;!!Iz9xO38YGHZK!_dbBfwEnZEuPp_8qIuBDoYfWRgCZKiPDMg9ouDltqsHhr9hqCHVlF9IedSkNVBDWupbneU7inC9YK-THL-lTjWkk8MVJPlY$>
><
> https://urldefense.com/v3/__https://docs.google.com/document/d/1Ehzvi5RNPs4hChkBFI6VD23myEqm-7sWW3d2kjmuYj8/edit?tab=t.0__;!!Iz9xO38YGHZK!8vlQ2_OqjmZNIpbrTTluFFNKFCEeM87nr95MP5pMpMu5fFWVSoBZ3eb6-C2xbAGCG-ry5eaCO-pFLxl3hYqTlcg$
>>
>
> Based on Prashant’s initial feedback (parity with Apache Gravitino for
> metrics reports), I have an initial PR here.
> https://urldefense.com/v3/__https://github.com/apache/polaris/pull/3385__;!!Iz9xO38YGHZK!65BVFQDe_GtKbDW8LkNdzVS-WYewzWsSns5zFHKclbMU3vUy2UaTXLB5G-wql1yyqoTFyDc9Bt2W-eKl7A$
> <https://urldefense.com/v3/__https://github.com/apache/polaris/pull/3385__;!!Iz9xO38YGHZK!_dbBfwEnZEuPp_8qIuBDoYfWRgCZKiPDMg9ouDltqsHhr9hqCHVlF9IedSkNVBDWupbneU7inC9YK-THL-lTjWkkyZhI0OE$>
><
> https://urldefense.com/v3/__https://github.com/apache/polaris/pull/3385__;!!Iz9xO38YGHZK!8vlQ2_OqjmZNIpbrTTluFFNKFCEeM87nr95MP5pMpMu5fFWVSoBZ3eb6-C2xbAGCG-ry5eaCO-pFLxl39opCznI$
>>
>
> I will be reaching out to various folks for review. I am open to breaking
> down the PR into smaller logical groups of PR if it helps.
>
> I request you to look at the Google doc and let me know what you think.
>
> —
> Anand
>
>
>
> --
> Dmitri Bourlatchkov
> Senior Staff Software Engineer, Dremio
> Dremio.com
> <https://urldefense.com/v3/__https://www.dremio.com/?utm_medium=email&utm_source=signature&utm_term=na&utm_content=email-signature&utm_campaign=email-signature__;!!Iz9xO38YGHZK!_dbBfwEnZEuPp_8qIuBDoYfWRgCZKiPDMg9ouDltqsHhr9hqCHVlF9IedSkNVBDWupbneU7inC9YK-THL-lTjWkkiOwjJc0$>/
> Follow Us on LinkedIn
> <https://urldefense.com/v3/__https://www.linkedin.com/company/dremio__;!!Iz9xO38YGHZK!_dbBfwEnZEuPp_8qIuBDoYfWRgCZKiPDMg9ouDltqsHhr9hqCHVlF9IedSkNVBDWupbneU7inC9YK-THL-lTjWkkCLjBQ2M$>/
> Get Started
> <https://urldefense.com/v3/__https://www.dremio.com/get-started/__;!!Iz9xO38YGHZK!_dbBfwEnZEuPp_8qIuBDoYfWRgCZKiPDMg9ouDltqsHhr9hqCHVlF9IedSkNVBDWupbneU7inC9YK-THL-lTjWkkoKQUMII$>
>
>
>The Agentic Lakehouse
> *The only lakehouse built for agents, managed by agents*
>
>
>

Re: Polaris Telemetry and Audit Trail

[Dmitri Bourlatchkov <di...@apache.org>]

Hi Anand,

Thanks for driving the SPI definition for scan/commits metrics.

I did not think from this angle before, but now I suppose it might be worth
introducing a separate SQL schema for metrics, detached from the main JDBC
Persistence schema.

My rationale is that the SPI may be exercised independently of the JDBC
Entity Persistence. I can imagine people using NoSQL Entity Persistence
plus JDBC Scan Metrics Persistence in the same deployment.

Using a separate schema will add further work on the Bootstrap workflow
(Admin Tool), but I hope the end result is going to be more convenient and
versatile for end users.

WDYT?

Thanks,
Dmitri.

On Thu, Jan 29, 2026 at 1:49 PM Anand Kumar Sankaran <
anand.sankaran@workday.com> wrote:

> Hi Dmitri,
>
> Thank you for taking the time to review. I had pulled out the schema in
> its own PR (based on comments) here:
> https://github.com/apache/polaris/pull/3523
>
> I will update the document with the SPI and carve out another PR that
> focuses on the SPI.
>
> —
> Anand
>
> *From: *Dmitri Bourlatchkov <dm...@dremio.com>
> *Date: *Thursday, January 29, 2026 at 10:43 AM
> *To: *dev@polaris.apache.org <de...@polaris.apache.org>
> *Cc: *Anand Kumar Sankaran <an...@workday.com>
> *Subject: *Re: Polaris Telemetry and Audit Trail
>
> This Message Is From an External Sender
> This message came from outside your organization.
> Report Suspicious
> <https://us-phishalarm-ewt.proofpoint.com/EWT/v1/Iz9xO38YGHZK!YhNDZAGr2cumY1cAlUeoPrU2eNUPz9Wxq2PaG1t99vEN3v60gucxC3Qsk-xK09RndcW46WtNjMMjQZbaQ495h1D1g293F_uqLhPM8gDMlA6xRmo5b5uqrSh2v7QKHrhQ$>
>
> Hi Anand,
>
> Thanks for making a proposal doc and starting this thread. Apologies for
> late comments.
>
> As you know I already reviewed a couple of related PRs, which had a
> smaller impact on the codebase, but now, I believe, we're coming to the
> core of this feature and it might need a deeper discussion.
>
> For the sake of awareness of other project members, I'd like to
> highlight some aspects by email.
>
> * Storing scan metrics in RDBMS in general looks like a
> reasonable approach to me. However, Polaris Persistence is an extensible
> mechanism. Other backends are possible, including privately developed
> backends.
>
> From this POV, I believe it might be preferable to start not with an RDBMS
> schema, but with a java SPI outlining expected read/write operations and
> the data model. This will make it easier to assess impact on other
> Persistence implementations. Naturally, Polaris service code will have to
> be able to work seamlessly regardless of the backend impl. (including the
> do-nothing impl.).
>
> I see that some connection between the SQL schema and java code is made in
> PR [3385], but it does not appear to offer a clean SPI that could be
> implemented by different Persistence backends. Would it be ok from your
> perspective to make another PR with just SPI code?
>
> * Regarding ad-hoc SQL queries - it is certainly a valid use case for
> custom code to query the Polaris database directly. However, those ad-hoc
> queries may require certain indexes, which may not have a relevant use case
> in OSS code. In that case, I believe it would be preferable to manage the
> extra indexes in custom code (not in Polaris). Whether this is the case or
> not currently, I cannot say with certainty, so apologies if this is a false
> alarm... Having the java SPI available would be a huge aid to reasoning
> about the proposed SQL schema, I think.
>
> That said, I do not mean to complicate the development of this feature,
> just trying to structure it in a way that is hopefully easy to understand
> (including myself), maintain and expand later. If I missed something,
> please feel free to point it out.
>
> [3385] https://github.com/apache/polaris/pull/3385
> <https://urldefense.com/v3/__https://github.com/apache/polaris/pull/3385__;!!Iz9xO38YGHZK!_dbBfwEnZEuPp_8qIuBDoYfWRgCZKiPDMg9ouDltqsHhr9hqCHVlF9IedSkNVBDWupbneU7inC9YK-THL-lTjWkkyZhI0OE$>
>
> Thanks,
> Dmitri.
>
> On Wed, Jan 14, 2026 at 10:05 PM Anand Kumar Sankaran via dev <
> dev@polaris.apache.org> wrote:
>
> Hi Yufei,
>
> Gave commenting privileges to all.
>
> Thanks.
>
> —
> Anand
>
> From: Yufei Gu <fl...@gmail.com>
> Date: Wednesday, January 14, 2026 at 6:13 PM
> To: dev@polaris.apache.org <de...@polaris.apache.org>
> Cc: Anand Kumar Sankaran <an...@workday.com>
> Subject: Re: Polaris Telemetry and Audit Trail
>
> This Message Is From an External Sender
> This message came from outside your organization.
> Report Suspicious<
> https://us-phishalarm-ewt.proofpoint.com/EWT/v1/Iz9xO38YGHZK!YhNDZABkHi1B6hyPVYUK0T-K5j4Aoqdrlu07UjBpJcpjOsz7Ie7d_DhPl-ywi50dfQz4O6LL-AHLCOQTQ0X7Le8yD2cF0ogquGFB8Aw8uQLgB0RL6Ezpa-pLDPA4ziXA$
> >
>
> Hi Anand, thanks for the design doc and PR. Can you allow comments in the
> doc so that people can chime in? Thanks!
>
> Yufei
>
>
> On Sat, Jan 10, 2026 at 9:09 AM Anand Kumar Sankaran via dev <
> dev@polaris.apache.org<ma...@polaris.apache.org>> wrote:
> Hi all
>
> My first PR for adding AWS STS Session Tags support for credential vending
> was merged. https://github.com/apache/polaris/pull/3327
> <https://urldefense.com/v3/__https://github.com/apache/polaris/pull/3327__;!!Iz9xO38YGHZK!_dbBfwEnZEuPp_8qIuBDoYfWRgCZKiPDMg9ouDltqsHhr9hqCHVlF9IedSkNVBDWupbneU7inC9YK-THL-lTjWkklJ4Xf9Y$>
> <
> https://urldefense.com/v3/__https://github.com/apache/polaris/pull/3327__;!!Iz9xO38YGHZK!8vlQ2_OqjmZNIpbrTTluFFNKFCEeM87nr95MP5pMpMu5fFWVSoBZ3eb6-C2xbAGCG-ry5eaCO-pFLxl3IZAJGqY$
> >
>
> I have been working with Prashant on an end-to-end telemetry and audit
> trail tracking for Polaris.  It is documented here.
> https://docs.google.com/document/d/1Ehzvi5RNPs4hChkBFI6VD23myEqm-7sWW3d2kjmuYj8/edit?tab=t.0
> <https://urldefense.com/v3/__https://docs.google.com/document/d/1Ehzvi5RNPs4hChkBFI6VD23myEqm-7sWW3d2kjmuYj8/edit?tab=t.0__;!!Iz9xO38YGHZK!_dbBfwEnZEuPp_8qIuBDoYfWRgCZKiPDMg9ouDltqsHhr9hqCHVlF9IedSkNVBDWupbneU7inC9YK-THL-lTjWkk8MVJPlY$>
> <
> https://urldefense.com/v3/__https://docs.google.com/document/d/1Ehzvi5RNPs4hChkBFI6VD23myEqm-7sWW3d2kjmuYj8/edit?tab=t.0__;!!Iz9xO38YGHZK!8vlQ2_OqjmZNIpbrTTluFFNKFCEeM87nr95MP5pMpMu5fFWVSoBZ3eb6-C2xbAGCG-ry5eaCO-pFLxl3hYqTlcg$
> >
>
> Based on Prashant’s initial feedback (parity with Apache Gravitino for
> metrics reports), I have an initial PR here.
> https://github.com/apache/polaris/pull/3385
> <https://urldefense.com/v3/__https://github.com/apache/polaris/pull/3385__;!!Iz9xO38YGHZK!_dbBfwEnZEuPp_8qIuBDoYfWRgCZKiPDMg9ouDltqsHhr9hqCHVlF9IedSkNVBDWupbneU7inC9YK-THL-lTjWkkyZhI0OE$>
> <
> https://urldefense.com/v3/__https://github.com/apache/polaris/pull/3385__;!!Iz9xO38YGHZK!8vlQ2_OqjmZNIpbrTTluFFNKFCEeM87nr95MP5pMpMu5fFWVSoBZ3eb6-C2xbAGCG-ry5eaCO-pFLxl39opCznI$
> >
>
> I will be reaching out to various folks for review. I am open to breaking
> down the PR into smaller logical groups of PR if it helps.
>
> I request you to look at the Google doc and let me know what you think.
>
> —
> Anand
>
>
>
> --
> Dmitri Bourlatchkov
> Senior Staff Software Engineer, Dremio
> Dremio.com
> <https://urldefense.com/v3/__https://www.dremio.com/?utm_medium=email&utm_source=signature&utm_term=na&utm_content=email-signature&utm_campaign=email-signature__;!!Iz9xO38YGHZK!_dbBfwEnZEuPp_8qIuBDoYfWRgCZKiPDMg9ouDltqsHhr9hqCHVlF9IedSkNVBDWupbneU7inC9YK-THL-lTjWkkiOwjJc0$> /
> Follow Us on LinkedIn
> <https://urldefense.com/v3/__https://www.linkedin.com/company/dremio__;!!Iz9xO38YGHZK!_dbBfwEnZEuPp_8qIuBDoYfWRgCZKiPDMg9ouDltqsHhr9hqCHVlF9IedSkNVBDWupbneU7inC9YK-THL-lTjWkkCLjBQ2M$> /
> Get Started
> <https://urldefense.com/v3/__https://www.dremio.com/get-started/__;!!Iz9xO38YGHZK!_dbBfwEnZEuPp_8qIuBDoYfWRgCZKiPDMg9ouDltqsHhr9hqCHVlF9IedSkNVBDWupbneU7inC9YK-THL-lTjWkkoKQUMII$>
>
>
> The Agentic Lakehouse
> *The only lakehouse built for agents, managed by agents*
>
>
>

Re: Polaris Telemetry and Audit Trail

[Anand Kumar Sankaran via dev <de...@polaris.apache.org>]

Hi Dmitri,

Thank you for taking the time to review. I had pulled out the schema in its own PR (based on comments) here: https://github.com/apache/polaris/pull/3523

I will update the document with the SPI and carve out another PR that focuses on the SPI.

—
Anand

From: Dmitri Bourlatchkov <dm...@dremio.com>
Date: Thursday, January 29, 2026 at 10:43 AM
To: dev@polaris.apache.org <de...@polaris.apache.org>
Cc: Anand Kumar Sankaran <an...@workday.com>
Subject: Re: Polaris Telemetry and Audit Trail

This Message Is From an External Sender
This message came from outside your organization.
Report Suspicious<https://us-phishalarm-ewt.proofpoint.com/EWT/v1/Iz9xO38YGHZK!YhNDZAGr2cumY1cAlUeoPrU2eNUPz9Wxq2PaG1t99vEN3v60gucxC3Qsk-xK09RndcW46WtNjMMjQZbaQ495h1D1g293F_uqLhPM8gDMlA6xRmo5b5uqrSh2v7QKHrhQ$>

Hi Anand,

Thanks for making a proposal doc and starting this thread. Apologies for late comments.

As you know I already reviewed a couple of related PRs, which had a smaller impact on the codebase, but now, I believe, we're coming to the core of this feature and it might need a deeper discussion.

For the sake of awareness of other project members, I'd like to highlight some aspects by email.

* Storing scan metrics in RDBMS in general looks like a reasonable approach to me. However, Polaris Persistence is an extensible mechanism. Other backends are possible, including privately developed backends.

From this POV, I believe it might be preferable to start not with an RDBMS schema, but with a java SPI outlining expected read/write operations and the data model. This will make it easier to assess impact on other Persistence implementations. Naturally, Polaris service code will have to be able to work seamlessly regardless of the backend impl. (including the do-nothing impl.).

I see that some connection between the SQL schema and java code is made in PR [3385], but it does not appear to offer a clean SPI that could be implemented by different Persistence backends. Would it be ok from your perspective to make another PR with just SPI code?

* Regarding ad-hoc SQL queries - it is certainly a valid use case for custom code to query the Polaris database directly. However, those ad-hoc queries may require certain indexes, which may not have a relevant use case in OSS code. In that case, I believe it would be preferable to manage the extra indexes in custom code (not in Polaris). Whether this is the case or not currently, I cannot say with certainty, so apologies if this is a false alarm... Having the java SPI available would be a huge aid to reasoning about the proposed SQL schema, I think.

That said, I do not mean to complicate the development of this feature, just trying to structure it in a way that is hopefully easy to understand (including myself), maintain and expand later. If I missed something, please feel free to point it out.

[3385] https://github.com/apache/polaris/pull/3385<https://urldefense.com/v3/__https://github.com/apache/polaris/pull/3385__;!!Iz9xO38YGHZK!_dbBfwEnZEuPp_8qIuBDoYfWRgCZKiPDMg9ouDltqsHhr9hqCHVlF9IedSkNVBDWupbneU7inC9YK-THL-lTjWkkyZhI0OE$>

Thanks,
Dmitri.

On Wed, Jan 14, 2026 at 10:05 PM Anand Kumar Sankaran via dev <de...@polaris.apache.org>> wrote:
Hi Yufei,

Gave commenting privileges to all.

Thanks.

—
Anand

From: Yufei Gu <fl...@gmail.com>>
Date: Wednesday, January 14, 2026 at 6:13 PM
To: dev@polaris.apache.org<ma...@polaris.apache.org> <de...@polaris.apache.org>>
Cc: Anand Kumar Sankaran <an...@workday.com>>
Subject: Re: Polaris Telemetry and Audit Trail

This Message Is From an External Sender
This message came from outside your organization.
Report Suspicious<https://us-phishalarm-ewt.proofpoint.com/EWT/v1/Iz9xO38YGHZK!YhNDZABkHi1B6hyPVYUK0T-K5j4Aoqdrlu07UjBpJcpjOsz7Ie7d_DhPl-ywi50dfQz4O6LL-AHLCOQTQ0X7Le8yD2cF0ogquGFB8Aw8uQLgB0RL6Ezpa-pLDPA4ziXA$>

Hi Anand, thanks for the design doc and PR. Can you allow comments in the doc so that people can chime in? Thanks!

Yufei


On Sat, Jan 10, 2026 at 9:09 AM Anand Kumar Sankaran via dev <de...@polaris.apache.org>>> wrote:
Hi all

My first PR for adding AWS STS Session Tags support for credential vending was merged. https://github.com/apache/polaris/pull/3327<https://urldefense.com/v3/__https://github.com/apache/polaris/pull/3327__;!!Iz9xO38YGHZK!_dbBfwEnZEuPp_8qIuBDoYfWRgCZKiPDMg9ouDltqsHhr9hqCHVlF9IedSkNVBDWupbneU7inC9YK-THL-lTjWkklJ4Xf9Y$><https://urldefense.com/v3/__https://github.com/apache/polaris/pull/3327__;!!Iz9xO38YGHZK!8vlQ2_OqjmZNIpbrTTluFFNKFCEeM87nr95MP5pMpMu5fFWVSoBZ3eb6-C2xbAGCG-ry5eaCO-pFLxl3IZAJGqY$>

I have been working with Prashant on an end-to-end telemetry and audit trail tracking for Polaris.  It is documented here. https://docs.google.com/document/d/1Ehzvi5RNPs4hChkBFI6VD23myEqm-7sWW3d2kjmuYj8/edit?tab=t.0<https://urldefense.com/v3/__https://docs.google.com/document/d/1Ehzvi5RNPs4hChkBFI6VD23myEqm-7sWW3d2kjmuYj8/edit?tab=t.0__;!!Iz9xO38YGHZK!_dbBfwEnZEuPp_8qIuBDoYfWRgCZKiPDMg9ouDltqsHhr9hqCHVlF9IedSkNVBDWupbneU7inC9YK-THL-lTjWkk8MVJPlY$><https://urldefense.com/v3/__https://docs.google.com/document/d/1Ehzvi5RNPs4hChkBFI6VD23myEqm-7sWW3d2kjmuYj8/edit?tab=t.0__;!!Iz9xO38YGHZK!8vlQ2_OqjmZNIpbrTTluFFNKFCEeM87nr95MP5pMpMu5fFWVSoBZ3eb6-C2xbAGCG-ry5eaCO-pFLxl3hYqTlcg$>

Based on Prashant’s initial feedback (parity with Apache Gravitino for metrics reports), I have an initial PR here. https://github.com/apache/polaris/pull/3385<https://urldefense.com/v3/__https://github.com/apache/polaris/pull/3385__;!!Iz9xO38YGHZK!_dbBfwEnZEuPp_8qIuBDoYfWRgCZKiPDMg9ouDltqsHhr9hqCHVlF9IedSkNVBDWupbneU7inC9YK-THL-lTjWkkyZhI0OE$><https://urldefense.com/v3/__https://github.com/apache/polaris/pull/3385__;!!Iz9xO38YGHZK!8vlQ2_OqjmZNIpbrTTluFFNKFCEeM87nr95MP5pMpMu5fFWVSoBZ3eb6-C2xbAGCG-ry5eaCO-pFLxl39opCznI$>

I will be reaching out to various folks for review. I am open to breaking down the PR into smaller logical groups of PR if it helps.

I request you to look at the Google doc and let me know what you think.

—
Anand


--
Dmitri Bourlatchkov
Senior Staff Software Engineer, Dremio
Dremio.com<https://urldefense.com/v3/__https://www.dremio.com/?utm_medium=email&utm_source=signature&utm_term=na&utm_content=email-signature&utm_campaign=email-signature__;!!Iz9xO38YGHZK!_dbBfwEnZEuPp_8qIuBDoYfWRgCZKiPDMg9ouDltqsHhr9hqCHVlF9IedSkNVBDWupbneU7inC9YK-THL-lTjWkkiOwjJc0$> / Follow Us on LinkedIn<https://urldefense.com/v3/__https://www.linkedin.com/company/dremio__;!!Iz9xO38YGHZK!_dbBfwEnZEuPp_8qIuBDoYfWRgCZKiPDMg9ouDltqsHhr9hqCHVlF9IedSkNVBDWupbneU7inC9YK-THL-lTjWkkCLjBQ2M$> / Get Started<https://urldefense.com/v3/__https://www.dremio.com/get-started/__;!!Iz9xO38YGHZK!_dbBfwEnZEuPp_8qIuBDoYfWRgCZKiPDMg9ouDltqsHhr9hqCHVlF9IedSkNVBDWupbneU7inC9YK-THL-lTjWkkoKQUMII$>

[https://lh3.googleusercontent.com/d/1Ta6MlHHnksAk0fuUEm2yxgiTJbWMh6YM]

The Agentic Lakehouse
The only lakehouse built for agents, managed by agents

Re: Polaris Telemetry and Audit Trail

[Dmitri Bourlatchkov <dm...@dremio.com>]

Hi Anand,

Thanks for making a proposal doc and starting this thread. Apologies for
late comments.

As you know I already reviewed a couple of related PRs, which had a smaller
impact on the codebase, but now, I believe, we're coming to the core of
this feature and it might need a deeper discussion.

For the sake of awareness of other project members, I'd like to
highlight some aspects by email.

* Storing scan metrics in RDBMS in general looks like a reasonable approach
to me. However, Polaris Persistence is an extensible mechanism. Other
backends are possible, including privately developed backends.

From this POV, I believe it might be preferable to start not with an RDBMS
schema, but with a java SPI outlining expected read/write operations and
the data model. This will make it easier to assess impact on other
Persistence implementations. Naturally, Polaris service code will have to
be able to work seamlessly regardless of the backend impl. (including the
do-nothing impl.).

I see that some connection between the SQL schema and java code is made in
PR [3385], but it does not appear to offer a clean SPI that could be
implemented by different Persistence backends. Would it be ok from your
perspective to make another PR with just SPI code?

* Regarding ad-hoc SQL queries - it is certainly a valid use case for
custom code to query the Polaris database directly. However, those ad-hoc
queries may require certain indexes, which may not have a relevant use case
in OSS code. In that case, I believe it would be preferable to manage the
extra indexes in custom code (not in Polaris). Whether this is the case or
not currently, I cannot say with certainty, so apologies if this is a false
alarm... Having the java SPI available would be a huge aid to reasoning
about the proposed SQL schema, I think.

That said, I do not mean to complicate the development of this feature,
just trying to structure it in a way that is hopefully easy to understand
(including myself), maintain and expand later. If I missed something,
please feel free to point it out.

[3385] https://github.com/apache/polaris/pull/3385

Thanks,
Dmitri.

On Wed, Jan 14, 2026 at 10:05 PM Anand Kumar Sankaran via dev <
dev@polaris.apache.org> wrote:

> Hi Yufei,
>
> Gave commenting privileges to all.
>
> Thanks.
>
> —
> Anand
>
> From: Yufei Gu <fl...@gmail.com>
> Date: Wednesday, January 14, 2026 at 6:13 PM
> To: dev@polaris.apache.org <de...@polaris.apache.org>
> Cc: Anand Kumar Sankaran <an...@workday.com>
> Subject: Re: Polaris Telemetry and Audit Trail
>
> This Message Is From an External Sender
> This message came from outside your organization.
> Report Suspicious<
> https://us-phishalarm-ewt.proofpoint.com/EWT/v1/Iz9xO38YGHZK!YhNDZABkHi1B6hyPVYUK0T-K5j4Aoqdrlu07UjBpJcpjOsz7Ie7d_DhPl-ywi50dfQz4O6LL-AHLCOQTQ0X7Le8yD2cF0ogquGFB8Aw8uQLgB0RL6Ezpa-pLDPA4ziXA$
> >
>
> Hi Anand, thanks for the design doc and PR. Can you allow comments in the
> doc so that people can chime in? Thanks!
>
> Yufei
>
>
> On Sat, Jan 10, 2026 at 9:09 AM Anand Kumar Sankaran via dev <
> dev@polaris.apache.org<ma...@polaris.apache.org>> wrote:
> Hi all
>
> My first PR for adding AWS STS Session Tags support for credential vending
> was merged. https://github.com/apache/polaris/pull/3327<
> https://urldefense.com/v3/__https://github.com/apache/polaris/pull/3327__;!!Iz9xO38YGHZK!8vlQ2_OqjmZNIpbrTTluFFNKFCEeM87nr95MP5pMpMu5fFWVSoBZ3eb6-C2xbAGCG-ry5eaCO-pFLxl3IZAJGqY$
> >
>
> I have been working with Prashant on an end-to-end telemetry and audit
> trail tracking for Polaris.  It is documented here.
> https://docs.google.com/document/d/1Ehzvi5RNPs4hChkBFI6VD23myEqm-7sWW3d2kjmuYj8/edit?tab=t.0
> <
> https://urldefense.com/v3/__https://docs.google.com/document/d/1Ehzvi5RNPs4hChkBFI6VD23myEqm-7sWW3d2kjmuYj8/edit?tab=t.0__;!!Iz9xO38YGHZK!8vlQ2_OqjmZNIpbrTTluFFNKFCEeM87nr95MP5pMpMu5fFWVSoBZ3eb6-C2xbAGCG-ry5eaCO-pFLxl3hYqTlcg$
> >
>
> Based on Prashant’s initial feedback (parity with Apache Gravitino for
> metrics reports), I have an initial PR here.
> https://github.com/apache/polaris/pull/3385<
> https://urldefense.com/v3/__https://github.com/apache/polaris/pull/3385__;!!Iz9xO38YGHZK!8vlQ2_OqjmZNIpbrTTluFFNKFCEeM87nr95MP5pMpMu5fFWVSoBZ3eb6-C2xbAGCG-ry5eaCO-pFLxl39opCznI$
> >
>
> I will be reaching out to various folks for review. I am open to breaking
> down the PR into smaller logical groups of PR if it helps.
>
> I request you to look at the Google doc and let me know what you think.
>
> —
> Anand
>


-- 
Dmitri Bourlatchkov
Senior Staff Software Engineer, Dremio
Dremio.com
<https://www.dremio.com/?utm_medium=email&utm_source=signature&utm_term=na&utm_content=email-signature&utm_campaign=email-signature>
/
Follow Us on LinkedIn <https://www.linkedin.com/company/dremio> / Get
Started <https://www.dremio.com/get-started/>


The Agentic Lakehouse
The only lakehouse built for agents, managed by agents

Re: Polaris Telemetry and Audit Trail

[Anand Kumar Sankaran via dev <de...@polaris.apache.org>]

Hi Yufei,

Gave commenting privileges to all.

Thanks.

—
Anand

From: Yufei Gu <fl...@gmail.com>
Date: Wednesday, January 14, 2026 at 6:13 PM
To: dev@polaris.apache.org <de...@polaris.apache.org>
Cc: Anand Kumar Sankaran <an...@workday.com>
Subject: Re: Polaris Telemetry and Audit Trail

This Message Is From an External Sender
This message came from outside your organization.
Report Suspicious<https://us-phishalarm-ewt.proofpoint.com/EWT/v1/Iz9xO38YGHZK!YhNDZABkHi1B6hyPVYUK0T-K5j4Aoqdrlu07UjBpJcpjOsz7Ie7d_DhPl-ywi50dfQz4O6LL-AHLCOQTQ0X7Le8yD2cF0ogquGFB8Aw8uQLgB0RL6Ezpa-pLDPA4ziXA$>

Hi Anand, thanks for the design doc and PR. Can you allow comments in the doc so that people can chime in? Thanks!

Yufei


On Sat, Jan 10, 2026 at 9:09 AM Anand Kumar Sankaran via dev <de...@polaris.apache.org>> wrote:
Hi all

My first PR for adding AWS STS Session Tags support for credential vending was merged. https://github.com/apache/polaris/pull/3327<https://urldefense.com/v3/__https://github.com/apache/polaris/pull/3327__;!!Iz9xO38YGHZK!8vlQ2_OqjmZNIpbrTTluFFNKFCEeM87nr95MP5pMpMu5fFWVSoBZ3eb6-C2xbAGCG-ry5eaCO-pFLxl3IZAJGqY$>

I have been working with Prashant on an end-to-end telemetry and audit trail tracking for Polaris.  It is documented here. https://docs.google.com/document/d/1Ehzvi5RNPs4hChkBFI6VD23myEqm-7sWW3d2kjmuYj8/edit?tab=t.0<https://urldefense.com/v3/__https://docs.google.com/document/d/1Ehzvi5RNPs4hChkBFI6VD23myEqm-7sWW3d2kjmuYj8/edit?tab=t.0__;!!Iz9xO38YGHZK!8vlQ2_OqjmZNIpbrTTluFFNKFCEeM87nr95MP5pMpMu5fFWVSoBZ3eb6-C2xbAGCG-ry5eaCO-pFLxl3hYqTlcg$>

Based on Prashant’s initial feedback (parity with Apache Gravitino for metrics reports), I have an initial PR here. https://github.com/apache/polaris/pull/3385<https://urldefense.com/v3/__https://github.com/apache/polaris/pull/3385__;!!Iz9xO38YGHZK!8vlQ2_OqjmZNIpbrTTluFFNKFCEeM87nr95MP5pMpMu5fFWVSoBZ3eb6-C2xbAGCG-ry5eaCO-pFLxl39opCznI$>

I will be reaching out to various folks for review. I am open to breaking down the PR into smaller logical groups of PR if it helps.

I request you to look at the Google doc and let me know what you think.

—
Anand

Re: Polaris Telemetry and Audit Trail

[Yufei Gu <fl...@gmail.com>]

Hi Anand, thanks for the design doc and PR. Can you allow comments in the
doc so that people can chime in? Thanks!

Yufei


On Sat, Jan 10, 2026 at 9:09 AM Anand Kumar Sankaran via dev <
dev@polaris.apache.org> wrote:

> Hi all
>
> My first PR for adding AWS STS Session Tags support for credential vending
> was merged. https://github.com/apache/polaris/pull/3327
>
> I have been working with Prashant on an end-to-end telemetry and audit
> trail tracking for Polaris.  It is documented here.
> https://docs.google.com/document/d/1Ehzvi5RNPs4hChkBFI6VD23myEqm-7sWW3d2kjmuYj8/edit?tab=t.0
>
> Based on Prashant’s initial feedback (parity with Apache Gravitino for
> metrics reports), I have an initial PR here.
> https://github.com/apache/polaris/pull/3385
>
> I will be reaching out to various folks for review. I am open to breaking
> down the PR into smaller logical groups of PR if it helps.
>
> I request you to look at the Google doc and let me know what you think.
>
> —
> Anand
>