You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@solr.apache.org by "Matthew Biscocho (BLOOMBERG/ 919 3RD A)" <mb...@bloomberg.net> on 2025/04/11 20:59:26 UTC
SIP proposal: Switch from DropWizard to OpenTelemetry
Hey everyone,
Making this thread because I was interested in writing up a SIP for SOLR-17458 https://issues.apache.org/jira/browse/SOLR-17458 and starting a discussion around this. The proposed change involves migrating Solr's metrics framework from DropWizard to OpenTelemetry (OTel). This will move Solr to an attribute based metric framework and will also help future-proofing by keeping it from vendor lock-in, given OpenTelemetry's stance on being vendor-neutral and working with many different pipelines. Since Solr already packages an Open Telemetry SDK as a module for exporting spans through OTLP, we can use that foundation to have it push metrics as well. There will be a significant change here which will probably break many components and no longer be backwards compatible so this will probably need to be a major release version change but we can potentially keep a few things backwards compatible if needed. Just some things off the top of my head:
* The Prometheus Exporter (Maybe should be deprecated?)
* GET /admin/metrics endpoint
* JVM metrics collected from OTel
* Metric reporters?
Anyone have thoughts?
Re: SIP proposal: Switch from DropWizard to OpenTelemetry
Posted by David Smiley <ds...@apache.org>.
To be clear, this is a blocker for Solr 10. Shipping without this would
mean waiting till Solr 11. It would *suck* to wait that long, and it'd
suck for everyone maintaining the main branch & branch_10x branch with this
significant difference. We'll get this transition done.
On Wed, Aug 13, 2025 at 9:37 PM Matthew Biscocho (BLOOMBERG/ 919 3RD A) <
mbiscocho@bloomberg.net> wrote:
> Hi everyone,
>
> I wanted to share an update on the progress of the OTEL (Open Telemetry)
> migration from Dropwizard. If you haven’t been following, most of the work
> is being tracked under SOLR-17458 and merged into the feature/SOLR-17458
> branch. The foundation for OTEL instrumentation in Solr is mostly in place
> with metrics having attributes/labels, along with some removals to slim
> down the project.
>
> Here are some highlights:
> */admin/metrics now exposes Prometheus-formatted metrics, with OpenMetrics
> format available as an optional parameter.
> *JMX, Graphite, and SLF4J reporters have been removed.
> *Prometheus Exporter removed (currently in PR review)
> *OTLP metric exporter to push metrics is included in the OpenTelemetry
> module (currently in PR review).
>
> The /admin/metrics endpoint with prometheus and OTLP metric exporter
> specifically opens Solr to a lot of new supported options of monitoring
> with different backends and pipelines.
>
> I have also migrated a good chunk of metrics over to OTEL but there are
> still a handful of components in progress. Still aiming for Solr 10 release
> and hopeful we will get there. If anyone is interested in contributing or
> helping out, I have a list of additional metrics still needing migration to
> OTEL. Also big thanks to David for reviewing almost all the code changes
> and helping with this project!
>
> - Matt
>
> From: dev@solr.apache.org At: 05/13/25 15:19:26 UTC-4:00To:
> dev@solr.apache.org
> Subject: Re: SIP proposal: Switch from DropWizard to OpenTelemetry
>
> Sorry for not following up in a timely manner; I was on vacation, plus
> personal stuff, and now/soon changed jobs. I mostly just want to connect
> to improve the SIP then identify next steps. Just DM me to connect at your
> next convenience. The outcome will be a better SIP and we share it again.
>
> On Mon, Apr 21, 2025 at 10:42 AM Matthew Biscocho (BLOOMBERG/ 919 3RD A) <
> mbiscocho@bloomberg.net> wrote:
>
> > Happy to schedule an open video for anyone who wants to join for more
> > discussion around the SIP and improve on it. The more the better. Also
> with
> > this over haul, might be worth adding additional metrics or events to
> > measure. For example, Solr's prometheus exporter gets live nodes and
> shard
> > state metrics by pulling the GET `/admin/collections CLUSTERSTATUS` API.
> So
> > it's something missing directly from the metrics api. Might be worth
> > measuring the event with a gauge instead of it having to call and pull
> from
> > a separate API.
> >
> > As for OTEL adoption, seems like this is becoming common. Spring Boot
> uses
> > Micrometer for it's metrics framework which itself has a
> OtlpMeterRegistry
> >
>
> https://github.com/micrometer-metrics/micrometer/blob/main/implementations/micro
>
> meter-registry-otlp/src/main/java/io/micrometer/registry/otlp/OtlpMeterRegistry
> <https://github.com/micrometer-metrics/micrometer/blob/main/implementations/micrometer-registry-otlp/src/main/java/io/micrometer/registry/otlp/OtlpMeterRegistry>
> .
> java.
> > Open Telemetry has a large instrumentation list of support libraries for
> > its java agent
> >
>
> https://github.com/open-telemetry/opentelemetry-java-instrumentation/blob/main/d
> ocs/supported-libraries.md#libraries--frameworks
> <https://github.com/open-telemetry/opentelemetry-java-instrumentation/blob/main/docs/supported-libraries.md#libraries--frameworks>
> > including a spring boot one that works.
> >
> > From: dev@solr.apache.org At: 04/18/25 18:07:39 UTC-4:00To:
> > dev@solr.apache.org
> > Subject: Re: SIP proposal: Switch from DropWizard to OpenTelemetry
> >
> > It's notable that Solr-core in main already includes OpenTelemetry
> > dependencies. I expect this will be increasingly common, like SLF4J has
> > become.
> >
> > Docker: definitely a separate topic altogether. See
> > https://issues.apache.org/jira/browse/SOLR-17653 which I'm likely to
> > pursue. I'm very familiar with running Solr in Docker in integration
> > tests.
> >
> > On Fri, Apr 18, 2025 at 10:47 AM Gus Heck <gu...@gmail.com> wrote:
> >
> > > This sounds good to me because Dropwizard metrics is used by the
> > Dropwizard
> > > framework. Recent experiences where Dropwizard is used as an
> intermediary
> > > layer and integration tests using our test framework have shown it's a
> > path
> > > to jar hell with Solr wanting one version and the Dropwizard based code
> > > wanting another... What I don't know is the extent to which OTel might
> be
> > > adopted by other similar frameworks (what does spring boot usually work
> > > with?) The issue of us having several hundred dependencies and folks
> > using
> > > our test framework is always going to have some clash but maybe this
> will
> > > help?
> > >
> > > Side topic: I'm aware of at least two places that have come to the
> > > conclusion that integrated tests should spin up a separate JVM for
> solr,
> > > and part of me wonders if there's a way to push our test framework to
> do
> > > that natively so that folks using solr don't need to do a lot of
> > > engineering to create integration tests. This might also benefit our
> > tests
> > > by having a separate log for the server and the client side of
> > > interactions. Currently, you can't just scroll through and look for the
> > > stack trace trivially because often you have one reported from the
> server
> > > and one reported from the test itself. We might even be able to set a
> "no
> > > stack trace in the logs" general standard for the server side of things
> > > this way and fail any tests that generate a stack trace to ensure that
> a
> > > healthy solr never logs stack traces. Such a thing might involve
> having a
> > > testing module (not loaded in production) that allows/simplifies
> > > configuration without security hurdles...
> > >
> > > The obvious question of course is speed.... But maybe a solidification
> of
> > > what's an integration test and what's a unit test helps there, and
> > > integration tests are intended for build infra, or final checks rather
> > than
> > > developer testing during development.
> > >
> > > -Gus
> > >
> > > On Thu, Apr 17, 2025 at 11:45 PM David Smiley <ds...@apache.org>
> > wrote:
> > >
> > > > That's a nice start! I'd like to work with you on making the SIP
> > better.
> > > > Maybe a 1:1 (others invited too!) video might be most productive.
> I'd
> > > love
> > > > to involve AB more here. I think the SIP template that you clearly
> > > started
> > > > from isn't great; I'm tempted to replace it with a list of
> > considerations
> > > > and not a proposed format. People should structure their SIP however
> > the
> > > > topic best presents itself.
> > > >
> > > > On Thu, Apr 17, 2025 at 2:34 PM Matthew Biscocho (BLOOMBERG/ 919 3RD
> > A) <
> > > > mbiscocho@bloomberg.net> wrote:
> > > >
> > > > > Based on the discussion from yesterdays meetup, created SIP-23 for
> > > moving
> > > > > to Open Telemetry
> > > > >
> > > >
> > >
> >
> >
>
> https://cwiki.apache.org/confluence/display/SOLR/SIP-23%3A+Switch+from+DropWizar
> > d+to+OpenTelemetry
> > > > >
> > > > > With Solr 10 around the corner, hoping this work can get done
> before
> > > > then.
> > > > >
> > > > > From: dev@solr.apache.org At: 04/14/25 01:04:13 UTC-4:00To:
> > > > > dev@solr.apache.org
> > > > > Subject: Re: SIP proposal: Switch from DropWizard to OpenTelemetry
> > > > >
> > > > > I'm excited about the change! Where I work, it would
> *significantly*
> > > > > simplify our metrics pipeline if Solr were to embrace OTel for
> > metrics,
> > > > as
> > > > > we could then use company-provided OTel plugins. The broad
> industry
> > > > > adoption of OTel points to this being the least friction. With
> > > > DropWizard,
> > > > > it appears we hacked attributes onto it, in a sense.
> > > > >
> > > > > The primary criteria/requirement that comes to mind is to have
> > > > > strong/sophisticated ways to filter the right metrics to publish.
> > Solr
> > > > has
> > > > > that today.
> > > > >
> > > > > On Fri, Apr 11, 2025 at 5:00 PM Matthew Biscocho (BLOOMBERG/ 919
> 3RD
> > > A) <
> > > > > mbiscocho@bloomberg.net> wrote:
> > > > >
> > > > > > Hey everyone,
> > > > > >
> > > > > > Making this thread because I was interested in writing up a SIP
> for
> > > > > > SOLR-17458 https://issues.apache.org/jira/browse/SOLR-17458 and
> > > > starting
> > > > > > a discussion around this. The proposed change involves migrating
> > > Solr's
> > > > > > metrics framework from DropWizard to OpenTelemetry (OTel). This
> > will
> > > > move
> > > > > > Solr to an attribute based metric framework and will also help
> > > > > > future-proofing by keeping it from vendor lock-in, given
> > > > OpenTelemetry's
> > > > > > stance on being vendor-neutral and working with many different
> > > > pipelines.
> > > > > > Since Solr already packages an Open Telemetry SDK as a module for
> > > > > exporting
> > > > > > spans through OTLP, we can use that foundation to have it push
> > > metrics
> > > > as
> > > > > > well. There will be a significant change here which will probably
> > > break
> > > > > > many components and no longer be backwards compatible so this
> will
> > > > > probably
> > > > > > need to be a major release version change but we can potentially
> > > keep a
> > > > > few
> > > > > > things backwards compatible if needed. Just some things off the
> top
> > > of
> > > > my
> > > > > > head:
> > > > > >
> > > > > > * The Prometheus Exporter (Maybe should be deprecated?)
> > > > > > * GET /admin/metrics endpoint
> > > > > > * JVM metrics collected from OTel
> > > > > > * Metric reporters?
> > > > > >
> > > > > > Anyone have thoughts?
> > > > >
> > > > >
> > > > >
> > > >
> > >
> > >
> > > --
> > > http://www.needhamsoftware.com (work)
> > > https://a.co/d/b2sZLD9 (my fantasy fiction book)
> > >
> >
> >
> >
>
>
>
Re: SIP proposal: Switch from DropWizard to OpenTelemetry
Posted by "Matthew Biscocho (BLOOMBERG/ 919 3RD A)" <mb...@bloomberg.net>.
Hi everyone,
I wanted to share an update on the progress of the OTEL (Open Telemetry) migration from Dropwizard. If you haven’t been following, most of the work is being tracked under SOLR-17458 and merged into the feature/SOLR-17458 branch. The foundation for OTEL instrumentation in Solr is mostly in place with metrics having attributes/labels, along with some removals to slim down the project.
Here are some highlights:
*/admin/metrics now exposes Prometheus-formatted metrics, with OpenMetrics format available as an optional parameter.
*JMX, Graphite, and SLF4J reporters have been removed.
*Prometheus Exporter removed (currently in PR review)
*OTLP metric exporter to push metrics is included in the OpenTelemetry module (currently in PR review).
The /admin/metrics endpoint with prometheus and OTLP metric exporter specifically opens Solr to a lot of new supported options of monitoring with different backends and pipelines.
I have also migrated a good chunk of metrics over to OTEL but there are still a handful of components in progress. Still aiming for Solr 10 release and hopeful we will get there. If anyone is interested in contributing or helping out, I have a list of additional metrics still needing migration to OTEL. Also big thanks to David for reviewing almost all the code changes and helping with this project!
- Matt
From: dev@solr.apache.org At: 05/13/25 15:19:26 UTC-4:00To: dev@solr.apache.org
Subject: Re: SIP proposal: Switch from DropWizard to OpenTelemetry
Sorry for not following up in a timely manner; I was on vacation, plus
personal stuff, and now/soon changed jobs. I mostly just want to connect
to improve the SIP then identify next steps. Just DM me to connect at your
next convenience. The outcome will be a better SIP and we share it again.
On Mon, Apr 21, 2025 at 10:42 AM Matthew Biscocho (BLOOMBERG/ 919 3RD A) <
mbiscocho@bloomberg.net> wrote:
> Happy to schedule an open video for anyone who wants to join for more
> discussion around the SIP and improve on it. The more the better. Also with
> this over haul, might be worth adding additional metrics or events to
> measure. For example, Solr's prometheus exporter gets live nodes and shard
> state metrics by pulling the GET `/admin/collections CLUSTERSTATUS` API. So
> it's something missing directly from the metrics api. Might be worth
> measuring the event with a gauge instead of it having to call and pull from
> a separate API.
>
> As for OTEL adoption, seems like this is becoming common. Spring Boot uses
> Micrometer for it's metrics framework which itself has a OtlpMeterRegistry
>
https://github.com/micrometer-metrics/micrometer/blob/main/implementations/micro
meter-registry-otlp/src/main/java/io/micrometer/registry/otlp/OtlpMeterRegistry.
java.
> Open Telemetry has a large instrumentation list of support libraries for
> its java agent
>
https://github.com/open-telemetry/opentelemetry-java-instrumentation/blob/main/d
ocs/supported-libraries.md#libraries--frameworks
> including a spring boot one that works.
>
> From: dev@solr.apache.org At: 04/18/25 18:07:39 UTC-4:00To:
> dev@solr.apache.org
> Subject: Re: SIP proposal: Switch from DropWizard to OpenTelemetry
>
> It's notable that Solr-core in main already includes OpenTelemetry
> dependencies. I expect this will be increasingly common, like SLF4J has
> become.
>
> Docker: definitely a separate topic altogether. See
> https://issues.apache.org/jira/browse/SOLR-17653 which I'm likely to
> pursue. I'm very familiar with running Solr in Docker in integration
> tests.
>
> On Fri, Apr 18, 2025 at 10:47 AM Gus Heck <gu...@gmail.com> wrote:
>
> > This sounds good to me because Dropwizard metrics is used by the
> Dropwizard
> > framework. Recent experiences where Dropwizard is used as an intermediary
> > layer and integration tests using our test framework have shown it's a
> path
> > to jar hell with Solr wanting one version and the Dropwizard based code
> > wanting another... What I don't know is the extent to which OTel might be
> > adopted by other similar frameworks (what does spring boot usually work
> > with?) The issue of us having several hundred dependencies and folks
> using
> > our test framework is always going to have some clash but maybe this will
> > help?
> >
> > Side topic: I'm aware of at least two places that have come to the
> > conclusion that integrated tests should spin up a separate JVM for solr,
> > and part of me wonders if there's a way to push our test framework to do
> > that natively so that folks using solr don't need to do a lot of
> > engineering to create integration tests. This might also benefit our
> tests
> > by having a separate log for the server and the client side of
> > interactions. Currently, you can't just scroll through and look for the
> > stack trace trivially because often you have one reported from the server
> > and one reported from the test itself. We might even be able to set a "no
> > stack trace in the logs" general standard for the server side of things
> > this way and fail any tests that generate a stack trace to ensure that a
> > healthy solr never logs stack traces. Such a thing might involve having a
> > testing module (not loaded in production) that allows/simplifies
> > configuration without security hurdles...
> >
> > The obvious question of course is speed.... But maybe a solidification of
> > what's an integration test and what's a unit test helps there, and
> > integration tests are intended for build infra, or final checks rather
> than
> > developer testing during development.
> >
> > -Gus
> >
> > On Thu, Apr 17, 2025 at 11:45 PM David Smiley <ds...@apache.org>
> wrote:
> >
> > > That's a nice start! I'd like to work with you on making the SIP
> better.
> > > Maybe a 1:1 (others invited too!) video might be most productive. I'd
> > love
> > > to involve AB more here. I think the SIP template that you clearly
> > started
> > > from isn't great; I'm tempted to replace it with a list of
> considerations
> > > and not a proposed format. People should structure their SIP however
> the
> > > topic best presents itself.
> > >
> > > On Thu, Apr 17, 2025 at 2:34 PM Matthew Biscocho (BLOOMBERG/ 919 3RD
> A) <
> > > mbiscocho@bloomberg.net> wrote:
> > >
> > > > Based on the discussion from yesterdays meetup, created SIP-23 for
> > moving
> > > > to Open Telemetry
> > > >
> > >
> >
>
>
https://cwiki.apache.org/confluence/display/SOLR/SIP-23%3A+Switch+from+DropWizar
> d+to+OpenTelemetry
> > > >
> > > > With Solr 10 around the corner, hoping this work can get done before
> > > then.
> > > >
> > > > From: dev@solr.apache.org At: 04/14/25 01:04:13 UTC-4:00To:
> > > > dev@solr.apache.org
> > > > Subject: Re: SIP proposal: Switch from DropWizard to OpenTelemetry
> > > >
> > > > I'm excited about the change! Where I work, it would *significantly*
> > > > simplify our metrics pipeline if Solr were to embrace OTel for
> metrics,
> > > as
> > > > we could then use company-provided OTel plugins. The broad industry
> > > > adoption of OTel points to this being the least friction. With
> > > DropWizard,
> > > > it appears we hacked attributes onto it, in a sense.
> > > >
> > > > The primary criteria/requirement that comes to mind is to have
> > > > strong/sophisticated ways to filter the right metrics to publish.
> Solr
> > > has
> > > > that today.
> > > >
> > > > On Fri, Apr 11, 2025 at 5:00 PM Matthew Biscocho (BLOOMBERG/ 919 3RD
> > A) <
> > > > mbiscocho@bloomberg.net> wrote:
> > > >
> > > > > Hey everyone,
> > > > >
> > > > > Making this thread because I was interested in writing up a SIP for
> > > > > SOLR-17458 https://issues.apache.org/jira/browse/SOLR-17458 and
> > > starting
> > > > > a discussion around this. The proposed change involves migrating
> > Solr's
> > > > > metrics framework from DropWizard to OpenTelemetry (OTel). This
> will
> > > move
> > > > > Solr to an attribute based metric framework and will also help
> > > > > future-proofing by keeping it from vendor lock-in, given
> > > OpenTelemetry's
> > > > > stance on being vendor-neutral and working with many different
> > > pipelines.
> > > > > Since Solr already packages an Open Telemetry SDK as a module for
> > > > exporting
> > > > > spans through OTLP, we can use that foundation to have it push
> > metrics
> > > as
> > > > > well. There will be a significant change here which will probably
> > break
> > > > > many components and no longer be backwards compatible so this will
> > > > probably
> > > > > need to be a major release version change but we can potentially
> > keep a
> > > > few
> > > > > things backwards compatible if needed. Just some things off the top
> > of
> > > my
> > > > > head:
> > > > >
> > > > > * The Prometheus Exporter (Maybe should be deprecated?)
> > > > > * GET /admin/metrics endpoint
> > > > > * JVM metrics collected from OTel
> > > > > * Metric reporters?
> > > > >
> > > > > Anyone have thoughts?
> > > >
> > > >
> > > >
> > >
> >
> >
> > --
> > http://www.needhamsoftware.com (work)
> > https://a.co/d/b2sZLD9 (my fantasy fiction book)
> >
>
>
>
Re: SIP proposal: Switch from DropWizard to OpenTelemetry
Posted by David Smiley <ds...@apache.org>.
Sorry for not following up in a timely manner; I was on vacation, plus
personal stuff, and now/soon changed jobs. I mostly just want to connect
to improve the SIP then identify next steps. Just DM me to connect at your
next convenience. The outcome will be a better SIP and we share it again.
On Mon, Apr 21, 2025 at 10:42 AM Matthew Biscocho (BLOOMBERG/ 919 3RD A) <
mbiscocho@bloomberg.net> wrote:
> Happy to schedule an open video for anyone who wants to join for more
> discussion around the SIP and improve on it. The more the better. Also with
> this over haul, might be worth adding additional metrics or events to
> measure. For example, Solr's prometheus exporter gets live nodes and shard
> state metrics by pulling the GET `/admin/collections CLUSTERSTATUS` API. So
> it's something missing directly from the metrics api. Might be worth
> measuring the event with a gauge instead of it having to call and pull from
> a separate API.
>
> As for OTEL adoption, seems like this is becoming common. Spring Boot uses
> Micrometer for it's metrics framework which itself has a OtlpMeterRegistry
> https://github.com/micrometer-metrics/micrometer/blob/main/implementations/micrometer-registry-otlp/src/main/java/io/micrometer/registry/otlp/OtlpMeterRegistry.java.
> Open Telemetry has a large instrumentation list of support libraries for
> its java agent
> https://github.com/open-telemetry/opentelemetry-java-instrumentation/blob/main/docs/supported-libraries.md#libraries--frameworks
> including a spring boot one that works.
>
> From: dev@solr.apache.org At: 04/18/25 18:07:39 UTC-4:00To:
> dev@solr.apache.org
> Subject: Re: SIP proposal: Switch from DropWizard to OpenTelemetry
>
> It's notable that Solr-core in main already includes OpenTelemetry
> dependencies. I expect this will be increasingly common, like SLF4J has
> become.
>
> Docker: definitely a separate topic altogether. See
> https://issues.apache.org/jira/browse/SOLR-17653 which I'm likely to
> pursue. I'm very familiar with running Solr in Docker in integration
> tests.
>
> On Fri, Apr 18, 2025 at 10:47 AM Gus Heck <gu...@gmail.com> wrote:
>
> > This sounds good to me because Dropwizard metrics is used by the
> Dropwizard
> > framework. Recent experiences where Dropwizard is used as an intermediary
> > layer and integration tests using our test framework have shown it's a
> path
> > to jar hell with Solr wanting one version and the Dropwizard based code
> > wanting another... What I don't know is the extent to which OTel might be
> > adopted by other similar frameworks (what does spring boot usually work
> > with?) The issue of us having several hundred dependencies and folks
> using
> > our test framework is always going to have some clash but maybe this will
> > help?
> >
> > Side topic: I'm aware of at least two places that have come to the
> > conclusion that integrated tests should spin up a separate JVM for solr,
> > and part of me wonders if there's a way to push our test framework to do
> > that natively so that folks using solr don't need to do a lot of
> > engineering to create integration tests. This might also benefit our
> tests
> > by having a separate log for the server and the client side of
> > interactions. Currently, you can't just scroll through and look for the
> > stack trace trivially because often you have one reported from the server
> > and one reported from the test itself. We might even be able to set a "no
> > stack trace in the logs" general standard for the server side of things
> > this way and fail any tests that generate a stack trace to ensure that a
> > healthy solr never logs stack traces. Such a thing might involve having a
> > testing module (not loaded in production) that allows/simplifies
> > configuration without security hurdles...
> >
> > The obvious question of course is speed.... But maybe a solidification of
> > what's an integration test and what's a unit test helps there, and
> > integration tests are intended for build infra, or final checks rather
> than
> > developer testing during development.
> >
> > -Gus
> >
> > On Thu, Apr 17, 2025 at 11:45 PM David Smiley <ds...@apache.org>
> wrote:
> >
> > > That's a nice start! I'd like to work with you on making the SIP
> better.
> > > Maybe a 1:1 (others invited too!) video might be most productive. I'd
> > love
> > > to involve AB more here. I think the SIP template that you clearly
> > started
> > > from isn't great; I'm tempted to replace it with a list of
> considerations
> > > and not a proposed format. People should structure their SIP however
> the
> > > topic best presents itself.
> > >
> > > On Thu, Apr 17, 2025 at 2:34 PM Matthew Biscocho (BLOOMBERG/ 919 3RD
> A) <
> > > mbiscocho@bloomberg.net> wrote:
> > >
> > > > Based on the discussion from yesterdays meetup, created SIP-23 for
> > moving
> > > > to Open Telemetry
> > > >
> > >
> >
>
> https://cwiki.apache.org/confluence/display/SOLR/SIP-23%3A+Switch+from+DropWizar
> d+to+OpenTelemetry
> > > >
> > > > With Solr 10 around the corner, hoping this work can get done before
> > > then.
> > > >
> > > > From: dev@solr.apache.org At: 04/14/25 01:04:13 UTC-4:00To:
> > > > dev@solr.apache.org
> > > > Subject: Re: SIP proposal: Switch from DropWizard to OpenTelemetry
> > > >
> > > > I'm excited about the change! Where I work, it would *significantly*
> > > > simplify our metrics pipeline if Solr were to embrace OTel for
> metrics,
> > > as
> > > > we could then use company-provided OTel plugins. The broad industry
> > > > adoption of OTel points to this being the least friction. With
> > > DropWizard,
> > > > it appears we hacked attributes onto it, in a sense.
> > > >
> > > > The primary criteria/requirement that comes to mind is to have
> > > > strong/sophisticated ways to filter the right metrics to publish.
> Solr
> > > has
> > > > that today.
> > > >
> > > > On Fri, Apr 11, 2025 at 5:00 PM Matthew Biscocho (BLOOMBERG/ 919 3RD
> > A) <
> > > > mbiscocho@bloomberg.net> wrote:
> > > >
> > > > > Hey everyone,
> > > > >
> > > > > Making this thread because I was interested in writing up a SIP for
> > > > > SOLR-17458 https://issues.apache.org/jira/browse/SOLR-17458 and
> > > starting
> > > > > a discussion around this. The proposed change involves migrating
> > Solr's
> > > > > metrics framework from DropWizard to OpenTelemetry (OTel). This
> will
> > > move
> > > > > Solr to an attribute based metric framework and will also help
> > > > > future-proofing by keeping it from vendor lock-in, given
> > > OpenTelemetry's
> > > > > stance on being vendor-neutral and working with many different
> > > pipelines.
> > > > > Since Solr already packages an Open Telemetry SDK as a module for
> > > > exporting
> > > > > spans through OTLP, we can use that foundation to have it push
> > metrics
> > > as
> > > > > well. There will be a significant change here which will probably
> > break
> > > > > many components and no longer be backwards compatible so this will
> > > > probably
> > > > > need to be a major release version change but we can potentially
> > keep a
> > > > few
> > > > > things backwards compatible if needed. Just some things off the top
> > of
> > > my
> > > > > head:
> > > > >
> > > > > * The Prometheus Exporter (Maybe should be deprecated?)
> > > > > * GET /admin/metrics endpoint
> > > > > * JVM metrics collected from OTel
> > > > > * Metric reporters?
> > > > >
> > > > > Anyone have thoughts?
> > > >
> > > >
> > > >
> > >
> >
> >
> > --
> > http://www.needhamsoftware.com (work)
> > https://a.co/d/b2sZLD9 (my fantasy fiction book)
> >
>
>
>
Re: SIP proposal: Switch from DropWizard to OpenTelemetry
Posted by "Matthew Biscocho (BLOOMBERG/ 919 3RD A)" <mb...@bloomberg.net>.
Happy to schedule an open video for anyone who wants to join for more discussion around the SIP and improve on it. The more the better. Also with this over haul, might be worth adding additional metrics or events to measure. For example, Solr's prometheus exporter gets live nodes and shard state metrics by pulling the GET `/admin/collections CLUSTERSTATUS` API. So it's something missing directly from the metrics api. Might be worth measuring the event with a gauge instead of it having to call and pull from a separate API.
As for OTEL adoption, seems like this is becoming common. Spring Boot uses Micrometer for it's metrics framework which itself has a OtlpMeterRegistry https://github.com/micrometer-metrics/micrometer/blob/main/implementations/micrometer-registry-otlp/src/main/java/io/micrometer/registry/otlp/OtlpMeterRegistry.java. Open Telemetry has a large instrumentation list of support libraries for its java agent https://github.com/open-telemetry/opentelemetry-java-instrumentation/blob/main/docs/supported-libraries.md#libraries--frameworks including a spring boot one that works.
From: dev@solr.apache.org At: 04/18/25 18:07:39 UTC-4:00To: dev@solr.apache.org
Subject: Re: SIP proposal: Switch from DropWizard to OpenTelemetry
It's notable that Solr-core in main already includes OpenTelemetry
dependencies. I expect this will be increasingly common, like SLF4J has
become.
Docker: definitely a separate topic altogether. See
https://issues.apache.org/jira/browse/SOLR-17653 which I'm likely to
pursue. I'm very familiar with running Solr in Docker in integration tests.
On Fri, Apr 18, 2025 at 10:47 AM Gus Heck <gu...@gmail.com> wrote:
> This sounds good to me because Dropwizard metrics is used by the Dropwizard
> framework. Recent experiences where Dropwizard is used as an intermediary
> layer and integration tests using our test framework have shown it's a path
> to jar hell with Solr wanting one version and the Dropwizard based code
> wanting another... What I don't know is the extent to which OTel might be
> adopted by other similar frameworks (what does spring boot usually work
> with?) The issue of us having several hundred dependencies and folks using
> our test framework is always going to have some clash but maybe this will
> help?
>
> Side topic: I'm aware of at least two places that have come to the
> conclusion that integrated tests should spin up a separate JVM for solr,
> and part of me wonders if there's a way to push our test framework to do
> that natively so that folks using solr don't need to do a lot of
> engineering to create integration tests. This might also benefit our tests
> by having a separate log for the server and the client side of
> interactions. Currently, you can't just scroll through and look for the
> stack trace trivially because often you have one reported from the server
> and one reported from the test itself. We might even be able to set a "no
> stack trace in the logs" general standard for the server side of things
> this way and fail any tests that generate a stack trace to ensure that a
> healthy solr never logs stack traces. Such a thing might involve having a
> testing module (not loaded in production) that allows/simplifies
> configuration without security hurdles...
>
> The obvious question of course is speed.... But maybe a solidification of
> what's an integration test and what's a unit test helps there, and
> integration tests are intended for build infra, or final checks rather than
> developer testing during development.
>
> -Gus
>
> On Thu, Apr 17, 2025 at 11:45 PM David Smiley <ds...@apache.org> wrote:
>
> > That's a nice start! I'd like to work with you on making the SIP better.
> > Maybe a 1:1 (others invited too!) video might be most productive. I'd
> love
> > to involve AB more here. I think the SIP template that you clearly
> started
> > from isn't great; I'm tempted to replace it with a list of considerations
> > and not a proposed format. People should structure their SIP however the
> > topic best presents itself.
> >
> > On Thu, Apr 17, 2025 at 2:34 PM Matthew Biscocho (BLOOMBERG/ 919 3RD A) <
> > mbiscocho@bloomberg.net> wrote:
> >
> > > Based on the discussion from yesterdays meetup, created SIP-23 for
> moving
> > > to Open Telemetry
> > >
> >
>
https://cwiki.apache.org/confluence/display/SOLR/SIP-23%3A+Switch+from+DropWizar
d+to+OpenTelemetry
> > >
> > > With Solr 10 around the corner, hoping this work can get done before
> > then.
> > >
> > > From: dev@solr.apache.org At: 04/14/25 01:04:13 UTC-4:00To:
> > > dev@solr.apache.org
> > > Subject: Re: SIP proposal: Switch from DropWizard to OpenTelemetry
> > >
> > > I'm excited about the change! Where I work, it would *significantly*
> > > simplify our metrics pipeline if Solr were to embrace OTel for metrics,
> > as
> > > we could then use company-provided OTel plugins. The broad industry
> > > adoption of OTel points to this being the least friction. With
> > DropWizard,
> > > it appears we hacked attributes onto it, in a sense.
> > >
> > > The primary criteria/requirement that comes to mind is to have
> > > strong/sophisticated ways to filter the right metrics to publish. Solr
> > has
> > > that today.
> > >
> > > On Fri, Apr 11, 2025 at 5:00 PM Matthew Biscocho (BLOOMBERG/ 919 3RD
> A) <
> > > mbiscocho@bloomberg.net> wrote:
> > >
> > > > Hey everyone,
> > > >
> > > > Making this thread because I was interested in writing up a SIP for
> > > > SOLR-17458 https://issues.apache.org/jira/browse/SOLR-17458 and
> > starting
> > > > a discussion around this. The proposed change involves migrating
> Solr's
> > > > metrics framework from DropWizard to OpenTelemetry (OTel). This will
> > move
> > > > Solr to an attribute based metric framework and will also help
> > > > future-proofing by keeping it from vendor lock-in, given
> > OpenTelemetry's
> > > > stance on being vendor-neutral and working with many different
> > pipelines.
> > > > Since Solr already packages an Open Telemetry SDK as a module for
> > > exporting
> > > > spans through OTLP, we can use that foundation to have it push
> metrics
> > as
> > > > well. There will be a significant change here which will probably
> break
> > > > many components and no longer be backwards compatible so this will
> > > probably
> > > > need to be a major release version change but we can potentially
> keep a
> > > few
> > > > things backwards compatible if needed. Just some things off the top
> of
> > my
> > > > head:
> > > >
> > > > * The Prometheus Exporter (Maybe should be deprecated?)
> > > > * GET /admin/metrics endpoint
> > > > * JVM metrics collected from OTel
> > > > * Metric reporters?
> > > >
> > > > Anyone have thoughts?
> > >
> > >
> > >
> >
>
>
> --
> http://www.needhamsoftware.com (work)
> https://a.co/d/b2sZLD9 (my fantasy fiction book)
>
Re: SIP proposal: Switch from DropWizard to OpenTelemetry
Posted by David Smiley <ds...@apache.org>.
It's notable that Solr-core in main already includes OpenTelemetry
dependencies. I expect this will be increasingly common, like SLF4J has
become.
Docker: definitely a separate topic altogether. See
https://issues.apache.org/jira/browse/SOLR-17653 which I'm likely to
pursue. I'm very familiar with running Solr in Docker in integration tests.
On Fri, Apr 18, 2025 at 10:47 AM Gus Heck <gu...@gmail.com> wrote:
> This sounds good to me because Dropwizard metrics is used by the Dropwizard
> framework. Recent experiences where Dropwizard is used as an intermediary
> layer and integration tests using our test framework have shown it's a path
> to jar hell with Solr wanting one version and the Dropwizard based code
> wanting another... What I don't know is the extent to which OTel might be
> adopted by other similar frameworks (what does spring boot usually work
> with?) The issue of us having several hundred dependencies and folks using
> our test framework is always going to have some clash but maybe this will
> help?
>
> Side topic: I'm aware of at least two places that have come to the
> conclusion that integrated tests should spin up a separate JVM for solr,
> and part of me wonders if there's a way to push our test framework to do
> that natively so that folks using solr don't need to do a lot of
> engineering to create integration tests. This might also benefit our tests
> by having a separate log for the server and the client side of
> interactions. Currently, you can't just scroll through and look for the
> stack trace trivially because often you have one reported from the server
> and one reported from the test itself. We might even be able to set a "no
> stack trace in the logs" general standard for the server side of things
> this way and fail any tests that generate a stack trace to ensure that a
> healthy solr never logs stack traces. Such a thing might involve having a
> testing module (not loaded in production) that allows/simplifies
> configuration without security hurdles...
>
> The obvious question of course is speed.... But maybe a solidification of
> what's an integration test and what's a unit test helps there, and
> integration tests are intended for build infra, or final checks rather than
> developer testing during development.
>
> -Gus
>
> On Thu, Apr 17, 2025 at 11:45 PM David Smiley <ds...@apache.org> wrote:
>
> > That's a nice start! I'd like to work with you on making the SIP better.
> > Maybe a 1:1 (others invited too!) video might be most productive. I'd
> love
> > to involve AB more here. I think the SIP template that you clearly
> started
> > from isn't great; I'm tempted to replace it with a list of considerations
> > and not a proposed format. People should structure their SIP however the
> > topic best presents itself.
> >
> > On Thu, Apr 17, 2025 at 2:34 PM Matthew Biscocho (BLOOMBERG/ 919 3RD A) <
> > mbiscocho@bloomberg.net> wrote:
> >
> > > Based on the discussion from yesterdays meetup, created SIP-23 for
> moving
> > > to Open Telemetry
> > >
> >
> https://cwiki.apache.org/confluence/display/SOLR/SIP-23%3A+Switch+from+DropWizard+to+OpenTelemetry
> > >
> > > With Solr 10 around the corner, hoping this work can get done before
> > then.
> > >
> > > From: dev@solr.apache.org At: 04/14/25 01:04:13 UTC-4:00To:
> > > dev@solr.apache.org
> > > Subject: Re: SIP proposal: Switch from DropWizard to OpenTelemetry
> > >
> > > I'm excited about the change! Where I work, it would *significantly*
> > > simplify our metrics pipeline if Solr were to embrace OTel for metrics,
> > as
> > > we could then use company-provided OTel plugins. The broad industry
> > > adoption of OTel points to this being the least friction. With
> > DropWizard,
> > > it appears we hacked attributes onto it, in a sense.
> > >
> > > The primary criteria/requirement that comes to mind is to have
> > > strong/sophisticated ways to filter the right metrics to publish. Solr
> > has
> > > that today.
> > >
> > > On Fri, Apr 11, 2025 at 5:00 PM Matthew Biscocho (BLOOMBERG/ 919 3RD
> A) <
> > > mbiscocho@bloomberg.net> wrote:
> > >
> > > > Hey everyone,
> > > >
> > > > Making this thread because I was interested in writing up a SIP for
> > > > SOLR-17458 https://issues.apache.org/jira/browse/SOLR-17458 and
> > starting
> > > > a discussion around this. The proposed change involves migrating
> Solr's
> > > > metrics framework from DropWizard to OpenTelemetry (OTel). This will
> > move
> > > > Solr to an attribute based metric framework and will also help
> > > > future-proofing by keeping it from vendor lock-in, given
> > OpenTelemetry's
> > > > stance on being vendor-neutral and working with many different
> > pipelines.
> > > > Since Solr already packages an Open Telemetry SDK as a module for
> > > exporting
> > > > spans through OTLP, we can use that foundation to have it push
> metrics
> > as
> > > > well. There will be a significant change here which will probably
> break
> > > > many components and no longer be backwards compatible so this will
> > > probably
> > > > need to be a major release version change but we can potentially
> keep a
> > > few
> > > > things backwards compatible if needed. Just some things off the top
> of
> > my
> > > > head:
> > > >
> > > > * The Prometheus Exporter (Maybe should be deprecated?)
> > > > * GET /admin/metrics endpoint
> > > > * JVM metrics collected from OTel
> > > > * Metric reporters?
> > > >
> > > > Anyone have thoughts?
> > >
> > >
> > >
> >
>
>
> --
> http://www.needhamsoftware.com (work)
> https://a.co/d/b2sZLD9 (my fantasy fiction book)
>
Re: SIP proposal: Switch from DropWizard to OpenTelemetry
Posted by Gus Heck <gu...@gmail.com>.
This sounds good to me because Dropwizard metrics is used by the Dropwizard
framework. Recent experiences where Dropwizard is used as an intermediary
layer and integration tests using our test framework have shown it's a path
to jar hell with Solr wanting one version and the Dropwizard based code
wanting another... What I don't know is the extent to which OTel might be
adopted by other similar frameworks (what does spring boot usually work
with?) The issue of us having several hundred dependencies and folks using
our test framework is always going to have some clash but maybe this will
help?
Side topic: I'm aware of at least two places that have come to the
conclusion that integrated tests should spin up a separate JVM for solr,
and part of me wonders if there's a way to push our test framework to do
that natively so that folks using solr don't need to do a lot of
engineering to create integration tests. This might also benefit our tests
by having a separate log for the server and the client side of
interactions. Currently, you can't just scroll through and look for the
stack trace trivially because often you have one reported from the server
and one reported from the test itself. We might even be able to set a "no
stack trace in the logs" general standard for the server side of things
this way and fail any tests that generate a stack trace to ensure that a
healthy solr never logs stack traces. Such a thing might involve having a
testing module (not loaded in production) that allows/simplifies
configuration without security hurdles...
The obvious question of course is speed.... But maybe a solidification of
what's an integration test and what's a unit test helps there, and
integration tests are intended for build infra, or final checks rather than
developer testing during development.
-Gus
On Thu, Apr 17, 2025 at 11:45 PM David Smiley <ds...@apache.org> wrote:
> That's a nice start! I'd like to work with you on making the SIP better.
> Maybe a 1:1 (others invited too!) video might be most productive. I'd love
> to involve AB more here. I think the SIP template that you clearly started
> from isn't great; I'm tempted to replace it with a list of considerations
> and not a proposed format. People should structure their SIP however the
> topic best presents itself.
>
> On Thu, Apr 17, 2025 at 2:34 PM Matthew Biscocho (BLOOMBERG/ 919 3RD A) <
> mbiscocho@bloomberg.net> wrote:
>
> > Based on the discussion from yesterdays meetup, created SIP-23 for moving
> > to Open Telemetry
> >
> https://cwiki.apache.org/confluence/display/SOLR/SIP-23%3A+Switch+from+DropWizard+to+OpenTelemetry
> >
> > With Solr 10 around the corner, hoping this work can get done before
> then.
> >
> > From: dev@solr.apache.org At: 04/14/25 01:04:13 UTC-4:00To:
> > dev@solr.apache.org
> > Subject: Re: SIP proposal: Switch from DropWizard to OpenTelemetry
> >
> > I'm excited about the change! Where I work, it would *significantly*
> > simplify our metrics pipeline if Solr were to embrace OTel for metrics,
> as
> > we could then use company-provided OTel plugins. The broad industry
> > adoption of OTel points to this being the least friction. With
> DropWizard,
> > it appears we hacked attributes onto it, in a sense.
> >
> > The primary criteria/requirement that comes to mind is to have
> > strong/sophisticated ways to filter the right metrics to publish. Solr
> has
> > that today.
> >
> > On Fri, Apr 11, 2025 at 5:00 PM Matthew Biscocho (BLOOMBERG/ 919 3RD A) <
> > mbiscocho@bloomberg.net> wrote:
> >
> > > Hey everyone,
> > >
> > > Making this thread because I was interested in writing up a SIP for
> > > SOLR-17458 https://issues.apache.org/jira/browse/SOLR-17458 and
> starting
> > > a discussion around this. The proposed change involves migrating Solr's
> > > metrics framework from DropWizard to OpenTelemetry (OTel). This will
> move
> > > Solr to an attribute based metric framework and will also help
> > > future-proofing by keeping it from vendor lock-in, given
> OpenTelemetry's
> > > stance on being vendor-neutral and working with many different
> pipelines.
> > > Since Solr already packages an Open Telemetry SDK as a module for
> > exporting
> > > spans through OTLP, we can use that foundation to have it push metrics
> as
> > > well. There will be a significant change here which will probably break
> > > many components and no longer be backwards compatible so this will
> > probably
> > > need to be a major release version change but we can potentially keep a
> > few
> > > things backwards compatible if needed. Just some things off the top of
> my
> > > head:
> > >
> > > * The Prometheus Exporter (Maybe should be deprecated?)
> > > * GET /admin/metrics endpoint
> > > * JVM metrics collected from OTel
> > > * Metric reporters?
> > >
> > > Anyone have thoughts?
> >
> >
> >
>
--
http://www.needhamsoftware.com (work)
https://a.co/d/b2sZLD9 (my fantasy fiction book)
Re: SIP proposal: Switch from DropWizard to OpenTelemetry
Posted by David Smiley <ds...@apache.org>.
That's a nice start! I'd like to work with you on making the SIP better.
Maybe a 1:1 (others invited too!) video might be most productive. I'd love
to involve AB more here. I think the SIP template that you clearly started
from isn't great; I'm tempted to replace it with a list of considerations
and not a proposed format. People should structure their SIP however the
topic best presents itself.
On Thu, Apr 17, 2025 at 2:34 PM Matthew Biscocho (BLOOMBERG/ 919 3RD A) <
mbiscocho@bloomberg.net> wrote:
> Based on the discussion from yesterdays meetup, created SIP-23 for moving
> to Open Telemetry
> https://cwiki.apache.org/confluence/display/SOLR/SIP-23%3A+Switch+from+DropWizard+to+OpenTelemetry
>
> With Solr 10 around the corner, hoping this work can get done before then.
>
> From: dev@solr.apache.org At: 04/14/25 01:04:13 UTC-4:00To:
> dev@solr.apache.org
> Subject: Re: SIP proposal: Switch from DropWizard to OpenTelemetry
>
> I'm excited about the change! Where I work, it would *significantly*
> simplify our metrics pipeline if Solr were to embrace OTel for metrics, as
> we could then use company-provided OTel plugins. The broad industry
> adoption of OTel points to this being the least friction. With DropWizard,
> it appears we hacked attributes onto it, in a sense.
>
> The primary criteria/requirement that comes to mind is to have
> strong/sophisticated ways to filter the right metrics to publish. Solr has
> that today.
>
> On Fri, Apr 11, 2025 at 5:00 PM Matthew Biscocho (BLOOMBERG/ 919 3RD A) <
> mbiscocho@bloomberg.net> wrote:
>
> > Hey everyone,
> >
> > Making this thread because I was interested in writing up a SIP for
> > SOLR-17458 https://issues.apache.org/jira/browse/SOLR-17458 and starting
> > a discussion around this. The proposed change involves migrating Solr's
> > metrics framework from DropWizard to OpenTelemetry (OTel). This will move
> > Solr to an attribute based metric framework and will also help
> > future-proofing by keeping it from vendor lock-in, given OpenTelemetry's
> > stance on being vendor-neutral and working with many different pipelines.
> > Since Solr already packages an Open Telemetry SDK as a module for
> exporting
> > spans through OTLP, we can use that foundation to have it push metrics as
> > well. There will be a significant change here which will probably break
> > many components and no longer be backwards compatible so this will
> probably
> > need to be a major release version change but we can potentially keep a
> few
> > things backwards compatible if needed. Just some things off the top of my
> > head:
> >
> > * The Prometheus Exporter (Maybe should be deprecated?)
> > * GET /admin/metrics endpoint
> > * JVM metrics collected from OTel
> > * Metric reporters?
> >
> > Anyone have thoughts?
>
>
>
Re: SIP proposal: Switch from DropWizard to OpenTelemetry
Posted by "Matthew Biscocho (BLOOMBERG/ 919 3RD A)" <mb...@bloomberg.net>.
Based on the discussion from yesterdays meetup, created SIP-23 for moving to Open Telemetry https://cwiki.apache.org/confluence/display/SOLR/SIP-23%3A+Switch+from+DropWizard+to+OpenTelemetry
With Solr 10 around the corner, hoping this work can get done before then.
From: dev@solr.apache.org At: 04/14/25 01:04:13 UTC-4:00To: dev@solr.apache.org
Subject: Re: SIP proposal: Switch from DropWizard to OpenTelemetry
I'm excited about the change! Where I work, it would *significantly*
simplify our metrics pipeline if Solr were to embrace OTel for metrics, as
we could then use company-provided OTel plugins. The broad industry
adoption of OTel points to this being the least friction. With DropWizard,
it appears we hacked attributes onto it, in a sense.
The primary criteria/requirement that comes to mind is to have
strong/sophisticated ways to filter the right metrics to publish. Solr has
that today.
On Fri, Apr 11, 2025 at 5:00 PM Matthew Biscocho (BLOOMBERG/ 919 3RD A) <
mbiscocho@bloomberg.net> wrote:
> Hey everyone,
>
> Making this thread because I was interested in writing up a SIP for
> SOLR-17458 https://issues.apache.org/jira/browse/SOLR-17458 and starting
> a discussion around this. The proposed change involves migrating Solr's
> metrics framework from DropWizard to OpenTelemetry (OTel). This will move
> Solr to an attribute based metric framework and will also help
> future-proofing by keeping it from vendor lock-in, given OpenTelemetry's
> stance on being vendor-neutral and working with many different pipelines.
> Since Solr already packages an Open Telemetry SDK as a module for exporting
> spans through OTLP, we can use that foundation to have it push metrics as
> well. There will be a significant change here which will probably break
> many components and no longer be backwards compatible so this will probably
> need to be a major release version change but we can potentially keep a few
> things backwards compatible if needed. Just some things off the top of my
> head:
>
> * The Prometheus Exporter (Maybe should be deprecated?)
> * GET /admin/metrics endpoint
> * JVM metrics collected from OTel
> * Metric reporters?
>
> Anyone have thoughts?
Re: SIP proposal: Switch from DropWizard to OpenTelemetry
Posted by David Smiley <ds...@apache.org>.
I'm excited about the change! Where I work, it would *significantly*
simplify our metrics pipeline if Solr were to embrace OTel for metrics, as
we could then use company-provided OTel plugins. The broad industry
adoption of OTel points to this being the least friction. With DropWizard,
it appears we hacked attributes onto it, in a sense.
The primary criteria/requirement that comes to mind is to have
strong/sophisticated ways to filter the right metrics to publish. Solr has
that today.
On Fri, Apr 11, 2025 at 5:00 PM Matthew Biscocho (BLOOMBERG/ 919 3RD A) <
mbiscocho@bloomberg.net> wrote:
> Hey everyone,
>
> Making this thread because I was interested in writing up a SIP for
> SOLR-17458 https://issues.apache.org/jira/browse/SOLR-17458 and starting
> a discussion around this. The proposed change involves migrating Solr's
> metrics framework from DropWizard to OpenTelemetry (OTel). This will move
> Solr to an attribute based metric framework and will also help
> future-proofing by keeping it from vendor lock-in, given OpenTelemetry's
> stance on being vendor-neutral and working with many different pipelines.
> Since Solr already packages an Open Telemetry SDK as a module for exporting
> spans through OTLP, we can use that foundation to have it push metrics as
> well. There will be a significant change here which will probably break
> many components and no longer be backwards compatible so this will probably
> need to be a major release version change but we can potentially keep a few
> things backwards compatible if needed. Just some things off the top of my
> head:
>
> * The Prometheus Exporter (Maybe should be deprecated?)
> * GET /admin/metrics endpoint
> * JVM metrics collected from OTel
> * Metric reporters?
>
> Anyone have thoughts?