You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@solr.apache.org by "stillalex (via GitHub)" <gi...@apache.org> on 2023/08/14 21:33:03 UTC
[GitHub] [solr] stillalex opened a new pull request, #1841: SOLR-16354 Migrate from Jaeger/OpenTracing to OTEL
stillalex opened a new pull request, #1841:
URL: https://github.com/apache/solr/pull/1841
https://issues.apache.org/jira/browse/SOLR-16354
<!--
_(If you are a project committer then you may remove some/all of the following template.)_
Before creating a pull request, please file an issue in the ASF Jira system for Solr:
* https://issues.apache.org/jira/projects/SOLR
For something minor (i.e. that wouldn't be worth putting in release notes), you can skip JIRA.
To create a Jira issue, you will need to create an account there first.
The title of the PR should reference the Jira issue number in the form:
* SOLR-####: <short description of problem or changes>
SOLR must be fully capitalized. A short description helps people scanning pull requests for items they can work on.
Properly referencing the issue in the title ensures that Jira is correctly updated with code review comments and commits. -->
# Description
Please provide a short description of the changes you're making with this pull request.
# Solution
Please provide a short description of the approach taken to implement your solution.
# Tests
Please describe the tests you've developed or run to confirm this patch implements the feature or solves the problem.
# Checklist
Please review the following and check all that apply:
- [ ] I have reviewed the guidelines for [How to Contribute](https://wiki.apache.org/solr/HowToContribute) and my code conforms to the standards described there to the best of my ability.
- [ ] I have created a Jira issue and added the issue ID to my pull request title.
- [ ] I have given Solr maintainers [access](https://help.github.com/en/articles/allowing-changes-to-a-pull-request-branch-created-from-a-fork) to contribute to my PR branch. (optional but recommended)
- [ ] I have developed this patch against the `main` branch.
- [ ] I have run `./gradlew check`.
- [ ] I have added tests for my changes.
- [ ] I have added documentation for the [Reference Guide](https://github.com/apache/solr/tree/main/solr/solr-ref-guide)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
[GitHub] [solr] stillalex commented on pull request #1841: SOLR-16536 Replace OpenTracing instrumentation with OpenTelemetry
Posted by "stillalex (via GitHub)" <gi...@apache.org>.
stillalex commented on PR #1841:
URL: https://github.com/apache/solr/pull/1841#issuecomment-1680832985
> Would you like to give the docs a shot @stillalex ? Also, please add a line to CHANGES.txt.
@janhoy updated, please take a look.
Just to run one idea by you: the otel library upgrade is causing a lot of conflicts, and in itself it would be good to have it on 9.x branch too. what do you think about splitting only the lib upgrade to a different PR which can be simple and merged to 9.x branch?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
[GitHub] [solr] stillalex commented on a diff in pull request #1841: SOLR-16536 Replace OpenTracing instrumentation with OpenTelemetry
Posted by "stillalex (via GitHub)" <gi...@apache.org>.
stillalex commented on code in PR #1841:
URL: https://github.com/apache/solr/pull/1841#discussion_r1296094876
##########
solr/modules/opentelemetry/src/test/org/apache/solr/opentelemetry/CustomTestOtelTracerConfigurator.java:
##########
@@ -0,0 +1,87 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.solr.opentelemetry;
+
+import io.opentelemetry.api.GlobalOpenTelemetry;
+import io.opentelemetry.api.trace.Tracer;
+import io.opentelemetry.sdk.OpenTelemetrySdk;
+import io.opentelemetry.sdk.autoconfigure.AutoConfiguredOpenTelemetrySdk;
+import io.opentelemetry.sdk.testing.exporter.InMemorySpanExporter;
+import io.opentelemetry.sdk.trace.export.SimpleSpanProcessor;
+import org.apache.solr.common.util.NamedList;
+
+public class CustomTestOtelTracerConfigurator extends OtelTracerConfigurator {
+
+ static {
+ if (System.getProperty("host") == null) {
+ System.setProperty("host", "localhost");
+ }
+ }
+
+ private static final InMemorySpanExporter exporter = InMemorySpanExporter.create();
Review Comment:
updated the test to not crash solr and reflect the underlying issue. it does not make it less flaky, the more I look at this the more I worry about the very bad design of GlobalOpenTelemetry. I will need to keep a close eye on the error rates of this test going forward.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
[GitHub] [solr] stillalex commented on a diff in pull request #1841: SOLR-16536 Replace OpenTracing instrumentation with OpenTelemetry
Posted by "stillalex (via GitHub)" <gi...@apache.org>.
stillalex commented on code in PR #1841:
URL: https://github.com/apache/solr/pull/1841#discussion_r1294549451
##########
solr/core/src/java/org/apache/solr/api/AnnotatedApi.java:
##########
@@ -215,7 +214,6 @@ public void call(SolrQueryRequest req, SolrQueryResponse rsp) {
}
for (CommandOperation cmd : cmds) {
- TraceUtils.ifNotNoop(req.getSpan(), (span) -> span.log("Command: " + cmd.name));
Review Comment:
there is no span.log method anymore, so I removed some of these calls. are these 'log' events useful? I can look into alternatives from otel side.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
[GitHub] [solr] janhoy commented on pull request #1841: SOLR-16354 Migrate from Jaeger/OpenTracing to OTEL
Posted by "janhoy (via GitHub)" <gi...@apache.org>.
janhoy commented on PR #1841:
URL: https://github.com/apache/solr/pull/1841#issuecomment-1678854385
> Dependency analysis found issues.
> usedUndeclaredArtifacts
> - io.opentelemetry:opentelemetry-api:1.28.0@jar
> - org.apache.solr:solrj:10.0.0-SNAPSHOT@
> unusedDeclaredArtifacts
> - io.opentelemetry:opentelemetry-api-events:1.28.0-alpha@jar
> - io.opentelemetry:opentelemetry-sdk-trace:1.28.0@jar
> - io.opentelemetry:opentelemetry-sdk:1.28.0@jar
> - io.opentelemetry:opentelemetry-semconv:1.28.0-alpha@jar
This hits at changes needed in gradle build. There are `permitUnusedDeclared` keywords that can be used in build.gradle to tell the build that they are on purpose.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
[GitHub] [solr] janhoy commented on a diff in pull request #1841: SOLR-16536 Replace OpenTracing instrumentation with OpenTelemetry
Posted by "janhoy (via GitHub)" <gi...@apache.org>.
janhoy commented on code in PR #1841:
URL: https://github.com/apache/solr/pull/1841#discussion_r1294974887
##########
solr/core/src/java/org/apache/solr/api/AnnotatedApi.java:
##########
@@ -215,7 +214,6 @@ public void call(SolrQueryRequest req, SolrQueryResponse rsp) {
}
for (CommandOperation cmd : cmds) {
- TraceUtils.ifNotNoop(req.getSpan(), (span) -> span.log("Command: " + cmd.name));
Review Comment:
From the [docs](https://opentracing.io/docs/overview/tags-logs-baggage/#logs):
> Logs are key:value pairs that are useful for capturing timed log messages and other debugging or informational output from the application itself. Logs may be useful for documenting a specific moment or event within the span (in contrast to tags which should apply to the span regardless of time).
The equivalent in OTEL seems to be "[event](https://opentelemetry.io/docs/concepts/signals/traces/#span-events)", i.e. `mySpan.addEvent("foo")`. Perhaps use that for this PR and then decide in followup PRs whether it is even the right tool for the job - does not seem like correct use of `log()` in this case imo.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
[GitHub] [solr] stillalex commented on a diff in pull request #1841: SOLR-16536 Replace OpenTracing instrumentation with OpenTelemetry
Posted by "stillalex (via GitHub)" <gi...@apache.org>.
stillalex commented on code in PR #1841:
URL: https://github.com/apache/solr/pull/1841#discussion_r1296026006
##########
solr/modules/opentelemetry/src/test/org/apache/solr/opentelemetry/CustomTestOtelTracerConfigurator.java:
##########
@@ -0,0 +1,87 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.solr.opentelemetry;
+
+import io.opentelemetry.api.GlobalOpenTelemetry;
+import io.opentelemetry.api.trace.Tracer;
+import io.opentelemetry.sdk.OpenTelemetrySdk;
+import io.opentelemetry.sdk.autoconfigure.AutoConfiguredOpenTelemetrySdk;
+import io.opentelemetry.sdk.testing.exporter.InMemorySpanExporter;
+import io.opentelemetry.sdk.trace.export.SimpleSpanProcessor;
+import org.apache.solr.common.util.NamedList;
+
+public class CustomTestOtelTracerConfigurator extends OtelTracerConfigurator {
+
+ static {
+ if (System.getProperty("host") == null) {
+ System.setProperty("host", "localhost");
+ }
+ }
+
+ private static final InMemorySpanExporter exporter = InMemorySpanExporter.create();
Review Comment:
yes the error is strange. I don't think the exported is the cause but I added shutdown to it, it was a good suggestion (I think this gets closed via the otel sdk clode method).
the issue seems to be some component calls GlobalOpenTelemetry.get() before the actual init happens (as an aside this is a very strange design decision on otel side: calling GlobalOpenTelemetry.get() will actually _set_ it to noop and will not allow anyone else to set it again hence the failure in the logs). I am also considering the test itself as a culprit (it does a call to .get() to verify otel is active). still massaging the test code to make it into a test failure instead of a crash-like situation.
```
Caused by: java.lang.IllegalStateException: GlobalOpenTelemetry.set has already been called. GlobalOpenTelemetry.set must be called only once before any calls to GlobalOpenTelemetry.get. If you are using the OpenTelemetrySdk, use OpenTelemetrySdkBuilder.buildAndRegisterGlobal instead. Previous invocation set to cause of this exception.
2> at io.opentelemetry.api.GlobalOpenTelemetry.set(GlobalOpenTelemetry.java:104) ~[opentelemetry-api-1.28.0.jar:1.28.0]
2> at io.opentelemetry.sdk.autoconfigure.AutoConfiguredOpenTelemetrySdkBuilder.build(AutoConfiguredOpenTelemetrySdkBuilder.java:401) ~[opentelemetry-sdk-extension-autoconfigure-1.28.0.jar:1.28.0]
2> ... 21 more
2> Caused by: java.lang.Throwable
2> at io.opentelemetry.api.GlobalOpenTelemetry.set(GlobalOpenTelemetry.java:112) ~[opentelemetry-api-1.28.0.jar:1.28.0]
2> at io.opentelemetry.api.GlobalOpenTelemetry.get(GlobalOpenTelemetry.java:82) ~[opentelemetry-api-1.28.0.jar:1.28.0]
2> at io.opentelemetry.api.GlobalOpenTelemetry.getTracer(GlobalOpenTelemetry.java:131) ~[opentelemetry-api-1.28.0.jar:1.28.0]
2> at org.apache.solr.opentelemetry.OtelTracerConfigurator.getTracer(OtelTracerConfigurator.java:53) ~[main/:?]
2> at org.apache.solr.opentelemetry.CustomTestOtelTracerConfigurator.getTracer(CustomTestOtelTracerConfigurator.java:51) ~[test/:?]
2> ... 18 more
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
[GitHub] [solr] janhoy commented on a diff in pull request #1841: SOLR-16354 Migrate from Jaeger/OpenTracing to OTEL
Posted by "janhoy (via GitHub)" <gi...@apache.org>.
janhoy commented on code in PR #1841:
URL: https://github.com/apache/solr/pull/1841#discussion_r1294526185
##########
solr/core/src/java/org/apache/solr/servlet/HttpSolrCall.java:
##########
@@ -977,15 +979,32 @@ protected void writeResponse(
}
if (Method.HEAD != reqMethod) {
- OutputStream out = response.getOutputStream();
- QueryResponseWriterUtil.writeQueryResponse(out, responseWriter, solrReq, solrRsp, ct);
+ writeQueryResponseWithTracing(responseWriter, solrRsp, ct);
Review Comment:
Sorry, did not read the PR description first. I see now that this is a new span added. Can we perhaps move this to a followup PR for a clearer separation? That would allow reviewers to compare old and new trace spans and expect them to contain more or less the same data.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
[GitHub] [solr] stillalex commented on a diff in pull request #1841: SOLR-16536 Replace OpenTracing instrumentation with OpenTelemetry
Posted by "stillalex (via GitHub)" <gi...@apache.org>.
stillalex commented on code in PR #1841:
URL: https://github.com/apache/solr/pull/1841#discussion_r1294552518
##########
solr/modules/opentelemetry/src/java/org/apache/solr/opentelemetry/OtelTracerConfigurator.java:
##########
@@ -16,38 +16,56 @@
*/
package org.apache.solr.opentelemetry;
-import io.opentelemetry.opentracingshim.OpenTracingShim;
-import io.opentelemetry.sdk.OpenTelemetrySdk;
+import io.opentelemetry.api.GlobalOpenTelemetry;
+import io.opentelemetry.api.trace.Tracer;
import io.opentelemetry.sdk.autoconfigure.AutoConfiguredOpenTelemetrySdk;
-import io.opentracing.Tracer;
import java.lang.invoke.MethodHandles;
import java.util.Arrays;
import java.util.HashMap;
import java.util.Locale;
import java.util.Map;
import java.util.Objects;
import java.util.stream.Collectors;
+import org.apache.solr.common.util.NamedList;
import org.apache.solr.core.TracerConfigurator;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
/**
- * OpenTracing TracerConfigurator implementation which exports spans to OpenTelemetry in OTLP
- * format. This impl re-uses the existing OpenTracing instrumentation through a shim, and takes care
- * of properly closing the backing Tracer when Solr shuts down.
+ * Tracing TracerConfigurator implementation which exports spans to OpenTelemetry in OTLP format.
*/
public class OtelTracerConfigurator extends TracerConfigurator {
private static final Logger log = LoggerFactory.getLogger(MethodHandles.lookup().lookupClass());
- // Copy of environment. Can be overridden by tests
- Map<String, String> currentEnv = System.getenv();
+
+ private final Map<String, String> currentEnv;
+
+ public OtelTracerConfigurator() {
+ this(System.getenv());
+ }
+
+ OtelTracerConfigurator(Map<String, String> currentEnv) {
+ this.currentEnv = currentEnv;
+ }
@Override
public Tracer getTracer() {
+ // TODO remove reliance on global
+ return GlobalOpenTelemetry.getTracer("solr");
+ }
+
+ @Override
+ public void init(NamedList<?> args) {
+ prepareConfiguration();
+ AutoConfiguredOpenTelemetrySdk.initialize();
+ }
+
+ void prepareConfiguration() {
setDefaultIfNotConfigured("OTEL_SERVICE_NAME", "solr");
setDefaultIfNotConfigured("OTEL_TRACES_EXPORTER", "otlp");
setDefaultIfNotConfigured("OTEL_EXPORTER_OTLP_PROTOCOL", "grpc");
setDefaultIfNotConfigured("OTEL_TRACES_SAMPLER", "parentbased_always_on");
- addOtelResourceAttributes(Map.of("host.name", System.getProperty("host")));
+ setDefaultIfNotConfigured("OTEL_PROPAGATORS", "tracecontext,baggage");
+ addOtelResourceAttributes(Map.of("host.name", System.getProperty("host", "localhost")));
Review Comment:
I think mainly testing was breaking. maybe moving this to test init code would fit better.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
[GitHub] [solr] stillalex commented on a diff in pull request #1841: SOLR-16536 Replace OpenTracing instrumentation with OpenTelemetry
Posted by "stillalex (via GitHub)" <gi...@apache.org>.
stillalex commented on code in PR #1841:
URL: https://github.com/apache/solr/pull/1841#discussion_r1294554607
##########
solr/core/src/java/org/apache/solr/core/TracerConfigurator.java:
##########
@@ -17,83 +17,88 @@
package org.apache.solr.core;
-import io.opentracing.Scope;
-import io.opentracing.Span;
-import io.opentracing.Tracer;
-import io.opentracing.util.GlobalTracer;
-import java.lang.invoke.MethodHandles;
-import java.util.concurrent.atomic.AtomicReference;
-import org.apache.solr.common.util.ExecutorUtil;
+import io.opentelemetry.api.trace.Tracer;
import org.apache.solr.util.plugin.NamedListInitializedPlugin;
-import org.slf4j.Logger;
-import org.slf4j.LoggerFactory;
+import org.apache.solr.util.tracing.TraceUtils;
-/** Produces an OpenTracing {@link Tracer} from configuration. */
+/** Produces a {@link Tracer} from configuration. */
public abstract class TracerConfigurator implements NamedListInitializedPlugin {
Review Comment:
I think I would prefer discussing this as a followup. I can create a ticket under the umbrella for this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
[GitHub] [solr] janhoy commented on pull request #1841: SOLR-16536 Replace OpenTracing instrumentation with OpenTelemetry
Posted by "janhoy (via GitHub)" <gi...@apache.org>.
janhoy commented on PR #1841:
URL: https://github.com/apache/solr/pull/1841#issuecomment-1680314430
Wrt documentation. As long as the user-facing config and env vars are the same, and the spans created are almost identical, I think the only Ref-guide documentation to add is in "Major changes" section of upgrade notes, we can mention the deprecation of `http.status_code` and friends. If we plan to keep sending both for the entire 10.x line then we're fine. But this is of course linked with whatever consumer of traces (Jaeger etc) the user has. If we stick to the ECS standard from Solr 11 on, users can always remain on solr 10 if they require the old tags...
Would you like to give the docs a shot @stillalex ? Also, please add a line to `CHANGES.txt` for version 10.0.
Thinking about it - if we this PR is purely getting rid of OpenTracing and we do not change configuration in any way, it could be canidate for backporting to 9.4 release?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
[GitHub] [solr] stillalex commented on a diff in pull request #1841: SOLR-16536 Replace OpenTracing instrumentation with OpenTelemetry
Posted by "stillalex (via GitHub)" <gi...@apache.org>.
stillalex commented on code in PR #1841:
URL: https://github.com/apache/solr/pull/1841#discussion_r1296183918
##########
solr/modules/opentelemetry/src/test/org/apache/solr/opentelemetry/CustomTestOtelTracerConfigurator.java:
##########
@@ -0,0 +1,87 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.solr.opentelemetry;
+
+import io.opentelemetry.api.GlobalOpenTelemetry;
+import io.opentelemetry.api.trace.Tracer;
+import io.opentelemetry.sdk.OpenTelemetrySdk;
+import io.opentelemetry.sdk.autoconfigure.AutoConfiguredOpenTelemetrySdk;
+import io.opentelemetry.sdk.testing.exporter.InMemorySpanExporter;
+import io.opentelemetry.sdk.trace.export.SimpleSpanProcessor;
+import org.apache.solr.common.util.NamedList;
+
+public class CustomTestOtelTracerConfigurator extends OtelTracerConfigurator {
+
+ static {
+ if (System.getProperty("host") == null) {
+ System.setProperty("host", "localhost");
+ }
+ }
+
+ private static final InMemorySpanExporter exporter = InMemorySpanExporter.create();
Review Comment:
ok. I think I found it. pushed a change to tests to force init tracer before the cluster init. this way there is no race between test side and cluster side. I am no longer seeing failures locally, running a long beast 50 times now.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
[GitHub] [solr] stillalex commented on a diff in pull request #1841: SOLR-16536 Replace OpenTracing instrumentation with OpenTelemetry
Posted by "stillalex (via GitHub)" <gi...@apache.org>.
stillalex commented on code in PR #1841:
URL: https://github.com/apache/solr/pull/1841#discussion_r1294580656
##########
solr/core/src/java/org/apache/solr/api/V2HttpCall.java:
##########
@@ -510,18 +507,21 @@ protected void populateTracingSpan(Span span) {
// if this api has commands ...
final Map<String, JsonSchemaValidator> validators = getValidators(); // should be cached
if (validators != null && validators.isEmpty() == false && solrReq != null) {
- boolean validateInput = true; // because getCommands caches it; and we want it validated later
+ // boolean validateInput = true; // because getCommands caches it; and we want it
+ // validated later
// does this request have one command?
- List<CommandOperation> cmds = solrReq.getCommands(validateInput);
- if (cmds.size() == 1) {
- verb = cmds.get(0).name;
- }
+ // this can clash with other calls to read commands via CommandOperation.readCommands
+ // see "org.apache.solr.security.jwt.JWTAuthPluginIntegrationTest.testMetrics"
+ // List<CommandOperation> cmds = solrReq.getCommands(validateInput);
+ // if (cmds.size() == 1) {
+ // verb = cmds.get(0).name;
+ // }
Review Comment:
sure. will get back to this part as soon as I get the PR to a stable state, I seem to have some trouble passing a successful build.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
[GitHub] [solr] stillalex commented on a diff in pull request #1841: SOLR-16536 Replace OpenTracing instrumentation with OpenTelemetry
Posted by "stillalex (via GitHub)" <gi...@apache.org>.
stillalex commented on code in PR #1841:
URL: https://github.com/apache/solr/pull/1841#discussion_r1295082536
##########
solr/core/src/java/org/apache/solr/servlet/SolrDispatchFilter.java:
##########
@@ -255,15 +247,12 @@ private void dispatch(
Action result = call.call();
switch (result) {
case PASSTHROUGH:
- getSpan(request).log("SolrDispatchFilter PASSTHROUGH");
Review Comment:
replaced with 'addEvent', as discussed above
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
[GitHub] [solr] stillalex commented on pull request #1841: SOLR-16536 Replace OpenTracing instrumentation with OpenTelemetry
Posted by "stillalex (via GitHub)" <gi...@apache.org>.
stillalex commented on PR #1841:
URL: https://github.com/apache/solr/pull/1841#issuecomment-1682180045
> PS: Did you do a broader review of tag name changes apart from the two we covered, or should we put it up as a new JIRA?
No review yet. I was thinking I could do one as part of SOLR-16935 because I wanted to add 2 more spans.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
[GitHub] [solr] stillalex commented on a diff in pull request #1841: SOLR-16536 Replace OpenTracing instrumentation with OpenTelemetry
Posted by "stillalex (via GitHub)" <gi...@apache.org>.
stillalex commented on code in PR #1841:
URL: https://github.com/apache/solr/pull/1841#discussion_r1295032461
##########
solr/modules/opentelemetry/src/java/org/apache/solr/opentelemetry/OtelTracerConfigurator.java:
##########
@@ -16,38 +16,56 @@
*/
package org.apache.solr.opentelemetry;
-import io.opentelemetry.opentracingshim.OpenTracingShim;
-import io.opentelemetry.sdk.OpenTelemetrySdk;
+import io.opentelemetry.api.GlobalOpenTelemetry;
+import io.opentelemetry.api.trace.Tracer;
import io.opentelemetry.sdk.autoconfigure.AutoConfiguredOpenTelemetrySdk;
-import io.opentracing.Tracer;
import java.lang.invoke.MethodHandles;
import java.util.Arrays;
import java.util.HashMap;
import java.util.Locale;
import java.util.Map;
import java.util.Objects;
import java.util.stream.Collectors;
+import org.apache.solr.common.util.NamedList;
import org.apache.solr.core.TracerConfigurator;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
/**
- * OpenTracing TracerConfigurator implementation which exports spans to OpenTelemetry in OTLP
- * format. This impl re-uses the existing OpenTracing instrumentation through a shim, and takes care
- * of properly closing the backing Tracer when Solr shuts down.
+ * Tracing TracerConfigurator implementation which exports spans to OpenTelemetry in OTLP format.
*/
public class OtelTracerConfigurator extends TracerConfigurator {
private static final Logger log = LoggerFactory.getLogger(MethodHandles.lookup().lookupClass());
- // Copy of environment. Can be overridden by tests
- Map<String, String> currentEnv = System.getenv();
+
+ private final Map<String, String> currentEnv;
+
+ public OtelTracerConfigurator() {
+ this(System.getenv());
+ }
+
+ OtelTracerConfigurator(Map<String, String> currentEnv) {
+ this.currentEnv = currentEnv;
+ }
@Override
public Tracer getTracer() {
+ // TODO remove reliance on global
+ return GlobalOpenTelemetry.getTracer("solr");
+ }
+
+ @Override
+ public void init(NamedList<?> args) {
+ prepareConfiguration();
+ AutoConfiguredOpenTelemetrySdk.initialize();
+ }
+
+ void prepareConfiguration() {
setDefaultIfNotConfigured("OTEL_SERVICE_NAME", "solr");
setDefaultIfNotConfigured("OTEL_TRACES_EXPORTER", "otlp");
setDefaultIfNotConfigured("OTEL_EXPORTER_OTLP_PROTOCOL", "grpc");
setDefaultIfNotConfigured("OTEL_TRACES_SAMPLER", "parentbased_always_on");
- addOtelResourceAttributes(Map.of("host.name", System.getProperty("host")));
+ setDefaultIfNotConfigured("OTEL_PROPAGATORS", "tracecontext,baggage");
+ addOtelResourceAttributes(Map.of("host.name", System.getProperty("host", "localhost")));
Review Comment:
I think we agree to agree. I removed it and running the tests now to update the failing tests that are missing this property
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
[GitHub] [solr] stillalex commented on a diff in pull request #1841: SOLR-16536 Replace OpenTracing instrumentation with OpenTelemetry
Posted by "stillalex (via GitHub)" <gi...@apache.org>.
stillalex commented on code in PR #1841:
URL: https://github.com/apache/solr/pull/1841#discussion_r1295031075
##########
solr/core/src/java/org/apache/solr/api/AnnotatedApi.java:
##########
@@ -215,7 +214,6 @@ public void call(SolrQueryRequest req, SolrQueryResponse rsp) {
}
for (CommandOperation cmd : cmds) {
- TraceUtils.ifNotNoop(req.getSpan(), (span) -> span.log("Command: " + cmd.name));
Review Comment:
sure. will add this 'addEvent' version in.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
[GitHub] [solr] janhoy commented on a diff in pull request #1841: SOLR-16536 Replace OpenTracing instrumentation with OpenTelemetry
Posted by "janhoy (via GitHub)" <gi...@apache.org>.
janhoy commented on code in PR #1841:
URL: https://github.com/apache/solr/pull/1841#discussion_r1295650436
##########
solr/modules/opentelemetry/src/test/org/apache/solr/opentelemetry/CustomTestOtelTracerConfigurator.java:
##########
@@ -0,0 +1,87 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.solr.opentelemetry;
+
+import io.opentelemetry.api.GlobalOpenTelemetry;
+import io.opentelemetry.api.trace.Tracer;
+import io.opentelemetry.sdk.OpenTelemetrySdk;
+import io.opentelemetry.sdk.autoconfigure.AutoConfiguredOpenTelemetrySdk;
+import io.opentelemetry.sdk.testing.exporter.InMemorySpanExporter;
+import io.opentelemetry.sdk.trace.export.SimpleSpanProcessor;
+import org.apache.solr.common.util.NamedList;
+
+public class CustomTestOtelTracerConfigurator extends OtelTracerConfigurator {
+
+ static {
+ if (System.getProperty("host") == null) {
+ System.setProperty("host", "localhost");
+ }
+ }
+
+ private static final InMemorySpanExporter exporter = InMemorySpanExporter.create();
Review Comment:
This exporter is never cleanly shut down. It supports a `shutdown()` method.
I think this should be instantiated in `initTracer` and `shutdown()` in `resetForTest()`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
[GitHub] [solr] janhoy commented on pull request #1841: SOLR-16536 Replace OpenTracing instrumentation with OpenTelemetry
Posted by "janhoy (via GitHub)" <gi...@apache.org>.
janhoy commented on PR #1841:
URL: https://github.com/apache/solr/pull/1841#issuecomment-1680170683
> if this is for 2 fields only (`http.status_code` => `http.response.status_code` and `http.method` => `http.request.method`) it feels like overkill to implement this mechanism. I would emit both attributes old and new for a while and then drop them. let me know if you feel differently.
I'm ok with duplicating tags. But we should probably (now or as a follow-up) review every tag name that we explicitly define in our code, to see how it aligns with the recommendations, to make sure we covered all of them.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
[GitHub] [solr] janhoy commented on pull request #1841: SOLR-16354 Migrate from Jaeger/OpenTracing to OTEL
Posted by "janhoy (via GitHub)" <gi...@apache.org>.
janhoy commented on PR #1841:
URL: https://github.com/apache/solr/pull/1841#issuecomment-1678733988
Hi, please connect this PR to [SOLR-16536](https://issues.apache.org/jira/browse/SOLR-16536) instead (and not the umbrella issue).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
[GitHub] [solr] janhoy commented on a diff in pull request #1841: SOLR-16536 Replace OpenTracing instrumentation with OpenTelemetry
Posted by "janhoy (via GitHub)" <gi...@apache.org>.
janhoy commented on code in PR #1841:
URL: https://github.com/apache/solr/pull/1841#discussion_r1295650436
##########
solr/modules/opentelemetry/src/test/org/apache/solr/opentelemetry/CustomTestOtelTracerConfigurator.java:
##########
@@ -0,0 +1,87 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.solr.opentelemetry;
+
+import io.opentelemetry.api.GlobalOpenTelemetry;
+import io.opentelemetry.api.trace.Tracer;
+import io.opentelemetry.sdk.OpenTelemetrySdk;
+import io.opentelemetry.sdk.autoconfigure.AutoConfiguredOpenTelemetrySdk;
+import io.opentelemetry.sdk.testing.exporter.InMemorySpanExporter;
+import io.opentelemetry.sdk.trace.export.SimpleSpanProcessor;
+import org.apache.solr.common.util.NamedList;
+
+public class CustomTestOtelTracerConfigurator extends OtelTracerConfigurator {
+
+ static {
+ if (System.getProperty("host") == null) {
+ System.setProperty("host", "localhost");
+ }
+ }
+
+ private static final InMemorySpanExporter exporter = InMemorySpanExporter.create();
Review Comment:
This exporter is never cleanly shut down. It supports a `shutdown()` method.
I think this should be instantiated in `initTracer` and `shutdown()` in `resetForTest()`.
Cannot say for sure that this is causing the intermittent thread leak, but who knows?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
[GitHub] [solr] stillalex commented on pull request #1841: SOLR-16354 Migrate from Jaeger/OpenTracing to OTEL
Posted by "stillalex (via GitHub)" <gi...@apache.org>.
stillalex commented on PR #1841:
URL: https://github.com/apache/solr/pull/1841#issuecomment-1678111678
posting a screenshot from jaeger ui to demo what this data looks like for a query.

--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
[GitHub] [solr] janhoy commented on pull request #1841: SOLR-16354 Migrate from Jaeger/OpenTracing to OTEL
Posted by "janhoy (via GitHub)" <gi...@apache.org>.
janhoy commented on PR #1841:
URL: https://github.com/apache/solr/pull/1841#issuecomment-1678855564
> WARNING: there were unreferenced files under license folder:
> - /home/runner/work/solr/solr/solr/licenses/opentracing-api-LICENSE-ASL.txt
> - /home/runner/work/solr/solr/solr/licenses/opentracing-api-NOTICE.txt
> - /home/runner/work/solr/solr/solr/licenses/opentracing-mock-LICENSE-ASL.txt
> - /home/runner/work/solr/solr/solr/licenses/opentracing-mock-NOTICE.txt
> - /home/runner/work/solr/solr/solr/licenses/opentracing-noop-LICENSE-ASL.txt
> - /home/runner/work/solr/solr/solr/licenses/opentracing-noop-NOTICE.txt
> - /home/runner/work/solr/solr/solr/licenses/opentracing-util-LICENSE-ASL.txt
> - /home/runner/work/solr/solr/solr/licenses/opentracing-util-NOTICE.txt
These files from `licenses/` folder can now be removed as we don't have opentracing as a dependency anymore.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
[GitHub] [solr] janhoy commented on pull request #1841: SOLR-16536 Replace OpenTracing instrumentation with OpenTelemetry
Posted by "janhoy (via GitHub)" <gi...@apache.org>.
janhoy commented on PR #1841:
URL: https://github.com/apache/solr/pull/1841#issuecomment-1679714260
> * 9.3 spans have `http.status_code` attrib, where PR has `http.status`
Please consult the [otel semantic conventions](https://github.com/open-telemetry/semantic-conventions/tree/main/docs/http) for what attribute names to use. The former correct tag was `http.status_code`, but last month the OTEL project [adopted ECS (Elastic Common Schema)](https://github.com/open-telemetry/semantic-conventions/blob/main/CHANGELOG.md?plain=1#L97-L103) as the new standard, so those conventions will change and stabilize, thus the correct tag will be `http.response.status_code`.
Perhaps we should emit both for some versions, or implement the suggested env var `OTEL_SEMCONV_STABILITY_OPT_IN`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
[GitHub] [solr] stillalex commented on pull request #1841: SOLR-16536 Replace OpenTracing instrumentation with OpenTelemetry
Posted by "stillalex (via GitHub)" <gi...@apache.org>.
stillalex commented on PR #1841:
URL: https://github.com/apache/solr/pull/1841#issuecomment-1678871227
> Hi, please connect this PR to [SOLR-16536](https://issues.apache.org/jira/browse/SOLR-16536) instead (and not the umbrella issue).
you are right. renamed the PR to reflect the correct issue
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
[GitHub] [solr] stillalex commented on pull request #1841: SOLR-16536 Replace OpenTracing instrumentation with OpenTelemetry
Posted by "stillalex (via GitHub)" <gi...@apache.org>.
stillalex commented on PR #1841:
URL: https://github.com/apache/solr/pull/1841#issuecomment-1678936745
Thank you @janhoy for the review!
I think the biggest question was the removal of span.log. it felt strange to have logs mixed with spans. maybe I misunderstood the use. happy to look into otel alternatives
I am seeing some backup related tests failing on my local and on the crave build. will dig into those a bit more, I want to reach a stable state before making more changes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
[GitHub] [solr] stillalex commented on a diff in pull request #1841: SOLR-16536 Replace OpenTracing instrumentation with OpenTelemetry
Posted by "stillalex (via GitHub)" <gi...@apache.org>.
stillalex commented on code in PR #1841:
URL: https://github.com/apache/solr/pull/1841#discussion_r1295080612
##########
solr/core/src/java/org/apache/solr/api/AnnotatedApi.java:
##########
@@ -215,7 +214,6 @@ public void call(SolrQueryRequest req, SolrQueryResponse rsp) {
}
for (CommandOperation cmd : cmds) {
- TraceUtils.ifNotNoop(req.getSpan(), (span) -> span.log("Command: " + cmd.name));
Review Comment:
addEvent added
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
[GitHub] [solr] janhoy commented on a diff in pull request #1841: SOLR-16536 Replace OpenTracing instrumentation with OpenTelemetry
Posted by "janhoy (via GitHub)" <gi...@apache.org>.
janhoy commented on code in PR #1841:
URL: https://github.com/apache/solr/pull/1841#discussion_r1295565981
##########
solr/core/src/java/org/apache/solr/core/TracerConfigurator.java:
##########
@@ -17,83 +17,88 @@
package org.apache.solr.core;
-import io.opentracing.Scope;
-import io.opentracing.Span;
-import io.opentracing.Tracer;
-import io.opentracing.util.GlobalTracer;
-import java.lang.invoke.MethodHandles;
-import java.util.concurrent.atomic.AtomicReference;
-import org.apache.solr.common.util.ExecutorUtil;
+import io.opentelemetry.api.trace.Tracer;
import org.apache.solr.util.plugin.NamedListInitializedPlugin;
-import org.slf4j.Logger;
-import org.slf4j.LoggerFactory;
+import org.apache.solr.util.tracing.TraceUtils;
-/** Produces an OpenTracing {@link Tracer} from configuration. */
+/** Produces a {@link Tracer} from configuration. */
public abstract class TracerConfigurator implements NamedListInitializedPlugin {
Review Comment:
Added https://issues.apache.org/jira/browse/SOLR-16938 for this
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
[GitHub] [solr] stillalex commented on pull request #1841: SOLR-16536 Replace OpenTracing instrumentation with OpenTelemetry
Posted by "stillalex (via GitHub)" <gi...@apache.org>.
stillalex commented on PR #1841:
URL: https://github.com/apache/solr/pull/1841#issuecomment-1681592749
@janhoy otel upgrade split, merged and rebase completed. running checks now to verify if I missed anything.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
[GitHub] [solr] stillalex commented on pull request #1841: SOLR-16536 Replace OpenTracing instrumentation with OpenTelemetry
Posted by "stillalex (via GitHub)" <gi...@apache.org>.
stillalex commented on PR #1841:
URL: https://github.com/apache/solr/pull/1841#issuecomment-1681573914
moved the simple otel update to https://github.com/apache/solr/pull/1846 will rebase once that is in
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
[GitHub] [solr] janhoy commented on a diff in pull request #1841: SOLR-16536 Replace OpenTracing instrumentation with OpenTelemetry
Posted by "janhoy (via GitHub)" <gi...@apache.org>.
janhoy commented on code in PR #1841:
URL: https://github.com/apache/solr/pull/1841#discussion_r1294974887
##########
solr/core/src/java/org/apache/solr/api/AnnotatedApi.java:
##########
@@ -215,7 +214,6 @@ public void call(SolrQueryRequest req, SolrQueryResponse rsp) {
}
for (CommandOperation cmd : cmds) {
- TraceUtils.ifNotNoop(req.getSpan(), (span) -> span.log("Command: " + cmd.name));
Review Comment:
From the [docs](https://opentracing.io/docs/overview/tags-logs-baggage/#logs):
> Logs are key:value pairs that are useful for capturing timed log messages and other debugging or informational output from the application itself. Logs may be useful for documenting a specific moment or event within the span (in contrast to tags which should apply to the span regardless of time).
The equivalent in OTEL seems to be "event", i.e. `mySpan.addEvent("foo")`. Perhaps use that for this PR and then decide in followup PRs whether it is even the right tool for the job - does not seem like correct use of `log()` in this case imo.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
[GitHub] [solr] stillalex commented on pull request #1841: SOLR-16536 Replace OpenTracing instrumentation with OpenTelemetry
Posted by "stillalex (via GitHub)" <gi...@apache.org>.
stillalex commented on PR #1841:
URL: https://github.com/apache/solr/pull/1841#issuecomment-1679664416
quick comparison of span data (exporting from jaeger on 9.3 vs this PR):
<details>
<summary>Query spans on this PR</summary>
```
{
"data": [
{
"traceID": "b3e378158f072905b2920df7714ddd37",
"spans": [
{
"traceID": "b3e378158f072905b2920df7714ddd37",
"spanID": "86487341790e7776",
"operationName": "post:/{core}/select",
"references": [
{
"refType": "CHILD_OF",
"traceID": "b3e378158f072905b2920df7714ddd37",
"spanID": "bcf1836208e5a7a4"
}
],
"startTime": 1692133668595665,
"duration": 30937,
"tags": [
{
"key": "otel.library.name",
"type": "string",
"value": "solr"
},
{
"key": "http.method",
"type": "string",
"value": "POST"
},
{
"key": "http.url",
"type": "string",
"value": "http://localhost:8982/solr/test_shard2_replica_n1/select"
},
{
"key": "db.instance",
"type": "string",
"value": "test_shard2_replica_n1"
},
{
"key": "db.type",
"type": "string",
"value": "solr"
},
{
"key": "http.status",
"type": "int64",
"value": 200
},
{
"key": "span.kind",
"type": "string",
"value": "server"
},
{
"key": "internal.span.format",
"type": "string",
"value": "proto"
}
],
"logs": [],
"processID": "p1",
"warnings": null
},
{
"traceID": "b3e378158f072905b2920df7714ddd37",
"spanID": "bcf1836208e5a7a4",
"operationName": "get:/{collection}/select",
"references": [],
"startTime": 1692133668566962,
"duration": 63467,
"tags": [
{
"key": "otel.library.name",
"type": "string",
"value": "solr"
},
{
"key": "http.method",
"type": "string",
"value": "GET"
},
{
"key": "http.url",
"type": "string",
"value": "http://localhost:8983/solr/test/select"
},
{
"key": "db.instance",
"type": "string",
"value": "test"
},
{
"key": "db.type",
"type": "string",
"value": "solr"
},
{
"key": "http.status",
"type": "int64",
"value": 200
},
{
"key": "http.params",
"type": "string",
"value": "_=1692133666339\u0026indent=true\u0026q=*:*\u0026q.op=OR\u0026useParams="
},
{
"key": "span.kind",
"type": "string",
"value": "server"
},
{
"key": "internal.span.format",
"type": "string",
"value": "proto"
}
],
"logs": [],
"processID": "p1",
"warnings": null
},
{
"traceID": "b3e378158f072905b2920df7714ddd37",
"spanID": "3a9142862158b626",
"operationName": "post:/{core}/select",
"references": [
{
"refType": "CHILD_OF",
"traceID": "b3e378158f072905b2920df7714ddd37",
"spanID": "bcf1836208e5a7a4"
}
],
"startTime": 1692133668595437,
"duration": 20848,
"tags": [
{
"key": "otel.library.name",
"type": "string",
"value": "solr"
},
{
"key": "http.method",
"type": "string",
"value": "POST"
},
{
"key": "http.url",
"type": "string",
"value": "http://localhost:8983/solr/test_shard1_replica_n2/select"
},
{
"key": "db.instance",
"type": "string",
"value": "test_shard1_replica_n2"
},
{
"key": "db.type",
"type": "string",
"value": "solr"
},
{
"key": "http.status",
"type": "int64",
"value": 200
},
{
"key": "span.kind",
"type": "string",
"value": "server"
},
{
"key": "internal.span.format",
"type": "string",
"value": "proto"
}
],
"logs": [],
"processID": "p1",
"warnings": null
}
],
"processes": {
"p1": {
"serviceName": "solr",
"tags": [
{
"key": "host.name",
"type": "string",
"value": "localhost"
},
{
"key": "telemetry.sdk.language",
"type": "string",
"value": "java"
},
{
"key": "telemetry.sdk.name",
"type": "string",
"value": "opentelemetry"
},
{
"key": "telemetry.sdk.version",
"type": "string",
"value": "1.28.0"
}
]
}
},
"warnings": null
}
],
"total": 0,
"limit": 0,
"offset": 0,
"errors": null
}
```
</details>
vs
<details>
<summary>Query spans on this 9.3</summary>
```
{
"data": [
{
"traceID": "36d81b003d238f7278c467132917ee3c",
"spans": [
{
"traceID": "36d81b003d238f7278c467132917ee3c",
"spanID": "0ea50951fdf8c5ed",
"operationName": "post:/{core}/select",
"references": [
{
"refType": "CHILD_OF",
"traceID": "36d81b003d238f7278c467132917ee3c",
"spanID": "9cf9b219437d234c"
},
{
"refType": "CHILD_OF",
"traceID": "36d81b003d238f7278c467132917ee3c",
"spanID": "9cf9b219437d234c"
}
],
"startTime": 1692134969573648,
"duration": 5985,
"tags": [
{
"key": "otel.library.name",
"type": "string",
"value": "opentracing-shim"
},
{
"key": "http.url",
"type": "string",
"value": "http://localhost:8982/solr/test_shard2_replica_n1/select"
},
{
"key": "http.status_code",
"type": "int64",
"value": 200
},
{
"key": "http.method",
"type": "string",
"value": "POST"
},
{
"key": "db.type",
"type": "string",
"value": "solr"
},
{
"key": "db.instance",
"type": "string",
"value": "test_shard2_replica_n1"
},
{
"key": "span.kind",
"type": "string",
"value": "server"
},
{
"key": "span.kind",
"type": "string",
"value": "internal"
},
{
"key": "internal.span.format",
"type": "string",
"value": "proto"
}
],
"logs": [],
"processID": "p1",
"warnings": null
},
{
"traceID": "36d81b003d238f7278c467132917ee3c",
"spanID": "7d539974b1cf00bf",
"operationName": "post:/{core}/select",
"references": [
{
"refType": "CHILD_OF",
"traceID": "36d81b003d238f7278c467132917ee3c",
"spanID": "9cf9b219437d234c"
},
{
"refType": "CHILD_OF",
"traceID": "36d81b003d238f7278c467132917ee3c",
"spanID": "9cf9b219437d234c"
}
],
"startTime": 1692134969542767,
"duration": 26760,
"tags": [
{
"key": "otel.library.name",
"type": "string",
"value": "opentracing-shim"
},
{
"key": "http.url",
"type": "string",
"value": "http://localhost:8982/solr/test_shard2_replica_n1/select"
},
{
"key": "http.status_code",
"type": "int64",
"value": 200
},
{
"key": "http.method",
"type": "string",
"value": "POST"
},
{
"key": "db.type",
"type": "string",
"value": "solr"
},
{
"key": "db.instance",
"type": "string",
"value": "test_shard2_replica_n1"
},
{
"key": "span.kind",
"type": "string",
"value": "server"
},
{
"key": "span.kind",
"type": "string",
"value": "internal"
},
{
"key": "internal.span.format",
"type": "string",
"value": "proto"
}
],
"logs": [],
"processID": "p1",
"warnings": null
},
{
"traceID": "36d81b003d238f7278c467132917ee3c",
"spanID": "9cf9b219437d234c",
"operationName": "get:/{collection}/select",
"references": [],
"startTime": 1692134969508568,
"duration": 72193,
"tags": [
{
"key": "otel.library.name",
"type": "string",
"value": "opentracing-shim"
},
{
"key": "http.url",
"type": "string",
"value": "http://localhost:8983/solr/test/select"
},
{
"key": "http.status_code",
"type": "int64",
"value": 200
},
{
"key": "http.method",
"type": "string",
"value": "GET"
},
{
"key": "http.params",
"type": "string",
"value": "_=1692134966416\u0026indent=true\u0026q=*:*\u0026q.op=OR\u0026useParams="
},
{
"key": "db.type",
"type": "string",
"value": "solr"
},
{
"key": "db.instance",
"type": "string",
"value": "test"
},
{
"key": "span.kind",
"type": "string",
"value": "server"
},
{
"key": "span.kind",
"type": "string",
"value": "internal"
},
{
"key": "internal.span.format",
"type": "string",
"value": "proto"
}
],
"logs": [],
"processID": "p1",
"warnings": null
},
{
"traceID": "36d81b003d238f7278c467132917ee3c",
"spanID": "4899cdb8a9e4200d",
"operationName": "post:/{core}/select",
"references": [
{
"refType": "CHILD_OF",
"traceID": "36d81b003d238f7278c467132917ee3c",
"spanID": "9cf9b219437d234c"
},
{
"refType": "CHILD_OF",
"traceID": "36d81b003d238f7278c467132917ee3c",
"spanID": "9cf9b219437d234c"
}
],
"startTime": 1692134969543674,
"duration": 14919,
"tags": [
{
"key": "otel.library.name",
"type": "string",
"value": "opentracing-shim"
},
{
"key": "http.url",
"type": "string",
"value": "http://localhost:8983/solr/test_shard1_replica_n2/select"
},
{
"key": "http.status_code",
"type": "int64",
"value": 200
},
{
"key": "http.method",
"type": "string",
"value": "POST"
},
{
"key": "db.type",
"type": "string",
"value": "solr"
},
{
"key": "db.instance",
"type": "string",
"value": "test_shard1_replica_n2"
},
{
"key": "span.kind",
"type": "string",
"value": "server"
},
{
"key": "span.kind",
"type": "string",
"value": "internal"
},
{
"key": "internal.span.format",
"type": "string",
"value": "proto"
}
],
"logs": [],
"processID": "p1",
"warnings": null
}
],
"processes": {
"p1": {
"serviceName": "solr",
"tags": [
{
"key": "host.name",
"type": "string",
"value": "localhost"
},
{
"key": "telemetry.sdk.language",
"type": "string",
"value": "java"
},
{
"key": "telemetry.sdk.name",
"type": "string",
"value": "opentelemetry"
},
{
"key": "telemetry.sdk.version",
"type": "string",
"value": "1.21.0"
}
]
}
},
"warnings": null
}
],
"total": 0,
"limit": 0,
"offset": 0,
"errors": null
}
```
<details>
Diff:
* 9.3 every child has 2 entries under `references` but they are the same, this looks like a duplication problem with 9.3 data
* 9.3 every span entry has 2 `span.kind` entries one `server` and one `internal` vs. PR with a single entry `server`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
[GitHub] [solr] stillalex commented on pull request #1841: SOLR-16536 Replace OpenTracing instrumentation with OpenTelemetry
Posted by "stillalex (via GitHub)" <gi...@apache.org>.
stillalex commented on PR #1841:
URL: https://github.com/apache/solr/pull/1841#issuecomment-1679674060
quick comparison of _Indexing_ span data (exporting from jaeger on 9.3 vs this PR):
<details>
<summary>Update spans on this PR</summary>
```
{
"data": [
{
"traceID": "d0cd6efa699d993637bfc1f313742f1f",
"spans": [
{
"traceID": "d0cd6efa699d993637bfc1f313742f1f",
"spanID": "ac76ee5c9f41ceb6",
"operationName": "post:/{core}/update",
"references": [
{
"refType": "CHILD_OF",
"traceID": "d0cd6efa699d993637bfc1f313742f1f",
"spanID": "f9d87bf73cd5a669"
}
],
"startTime": 1692134062932096,
"duration": 222281,
"tags": [
{
"key": "otel.library.name",
"type": "string",
"value": "solr"
},
{
"key": "http.method",
"type": "string",
"value": "POST"
},
{
"key": "http.url",
"type": "string",
"value": "http://localhost:8982/solr/test_shard2_replica_n1/update"
},
{
"key": "db.instance",
"type": "string",
"value": "test_shard2_replica_n1"
},
{
"key": "db.type",
"type": "string",
"value": "solr"
},
{
"key": "http.status",
"type": "int64",
"value": 200
},
{
"key": "http.params",
"type": "string",
"value": "update.distrib=TOLEADER\u0026distrib.from=http%3A%2F%2Flocalhost%3A8983%2Fsolr%2Ftest_shard1_replica_n2%2F\u0026wt=javabin\u0026version=2"
},
{
"key": "span.kind",
"type": "string",
"value": "server"
},
{
"key": "internal.span.format",
"type": "string",
"value": "proto"
}
],
"logs": [],
"processID": "p1",
"warnings": null
},
{
"traceID": "d0cd6efa699d993637bfc1f313742f1f",
"spanID": "f9d87bf73cd5a669",
"operationName": "post:/{collection}/update",
"references": [],
"startTime": 1692134062686908,
"duration": 468961,
"tags": [
{
"key": "otel.library.name",
"type": "string",
"value": "solr"
},
{
"key": "http.method",
"type": "string",
"value": "POST"
},
{
"key": "http.url",
"type": "string",
"value": "http://localhost:8983/solr/test/update"
},
{
"key": "db.instance",
"type": "string",
"value": "test"
},
{
"key": "db.type",
"type": "string",
"value": "solr"
},
{
"key": "http.status",
"type": "int64",
"value": 200
},
{
"key": "http.params",
"type": "string",
"value": "_=1692133972530\u0026commitWithin=10\u0026overwrite=true\u0026wt=json"
},
{
"key": "span.kind",
"type": "string",
"value": "server"
},
{
"key": "internal.span.format",
"type": "string",
"value": "proto"
}
],
"logs": [],
"processID": "p1",
"warnings": null
}
],
"processes": {
"p1": {
"serviceName": "solr",
"tags": [
{
"key": "host.name",
"type": "string",
"value": "localhost"
},
{
"key": "telemetry.sdk.language",
"type": "string",
"value": "java"
},
{
"key": "telemetry.sdk.name",
"type": "string",
"value": "opentelemetry"
},
{
"key": "telemetry.sdk.version",
"type": "string",
"value": "1.28.0"
}
]
}
},
"warnings": null
}
],
"total": 0,
"limit": 0,
"offset": 0,
"errors": null
}
```
</details>
<details>
<summary>Update spans on Solr 9.3 PR</summary>
```
{
"data": [
{
"traceID": "49503807dd79f2bf5e20fddfabbca6b0",
"spans": [
{
"traceID": "49503807dd79f2bf5e20fddfabbca6b0",
"spanID": "e79354cfcc9a1c8d",
"operationName": "post:/{core}/update",
"references": [
{
"refType": "CHILD_OF",
"traceID": "49503807dd79f2bf5e20fddfabbca6b0",
"spanID": "3b5a75c06c661620"
},
{
"refType": "CHILD_OF",
"traceID": "49503807dd79f2bf5e20fddfabbca6b0",
"spanID": "3b5a75c06c661620"
}
],
"startTime": 1692134739803056,
"duration": 216899,
"tags": [
{
"key": "otel.library.name",
"type": "string",
"value": "opentracing-shim"
},
{
"key": "http.url",
"type": "string",
"value": "http://localhost:8982/solr/test_shard2_replica_n1/update"
},
{
"key": "http.status_code",
"type": "int64",
"value": 200
},
{
"key": "http.method",
"type": "string",
"value": "POST"
},
{
"key": "http.params",
"type": "string",
"value": "update.distrib=TOLEADER\u0026distrib.from=http%3A%2F%2Flocalhost%3A8983%2Fsolr%2Ftest_shard1_replica_n2%2F\u0026wt=javabin\u0026version=2"
},
{
"key": "db.type",
"type": "string",
"value": "solr"
},
{
"key": "db.instance",
"type": "string",
"value": "test_shard2_replica_n1"
},
{
"key": "span.kind",
"type": "string",
"value": "server"
},
{
"key": "span.kind",
"type": "string",
"value": "internal"
},
{
"key": "internal.span.format",
"type": "string",
"value": "proto"
}
],
"logs": [],
"processID": "p1",
"warnings": null
},
{
"traceID": "49503807dd79f2bf5e20fddfabbca6b0",
"spanID": "3b5a75c06c661620",
"operationName": "post:/{collection}/update",
"references": [],
"startTime": 1692134739529315,
"duration": 491930,
"tags": [
{
"key": "otel.library.name",
"type": "string",
"value": "opentracing-shim"
},
{
"key": "http.url",
"type": "string",
"value": "http://localhost:8983/solr/test/update"
},
{
"key": "http.status_code",
"type": "int64",
"value": 200
},
{
"key": "http.method",
"type": "string",
"value": "POST"
},
{
"key": "http.params",
"type": "string",
"value": "_=1692133972530\u0026commitWithin=10\u0026overwrite=true\u0026wt=json"
},
{
"key": "db.type",
"type": "string",
"value": "solr"
},
{
"key": "db.instance",
"type": "string",
"value": "test"
},
{
"key": "span.kind",
"type": "string",
"value": "server"
},
{
"key": "span.kind",
"type": "string",
"value": "internal"
},
{
"key": "internal.span.format",
"type": "string",
"value": "proto"
}
],
"logs": [],
"processID": "p1",
"warnings": null
}
],
"processes": {
"p1": {
"serviceName": "solr",
"tags": [
{
"key": "host.name",
"type": "string",
"value": "localhost"
},
{
"key": "telemetry.sdk.language",
"type": "string",
"value": "java"
},
{
"key": "telemetry.sdk.name",
"type": "string",
"value": "opentelemetry"
},
{
"key": "telemetry.sdk.version",
"type": "string",
"value": "1.21.0"
}
]
}
},
"warnings": null
}
],
"total": 0,
"limit": 0,
"offset": 0,
"errors": null
}
```
</details>
Diff:
* 9.3 every child has 2 entries under `references` but they are the same, this looks like a duplication problem with 9.3 data (same as Query data)
* 9.3 every span entry has 2 `span.kind` entries one `server` and one `internal` vs. PR with a single entry `server` (same as Query data)
* 9.3 spans have `http.status_code` attrib, where PR has `http.status` (same as Query data)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
[GitHub] [solr] janhoy commented on a diff in pull request #1841: SOLR-16354 Migrate from Jaeger/OpenTracing to OTEL
Posted by "janhoy (via GitHub)" <gi...@apache.org>.
janhoy commented on code in PR #1841:
URL: https://github.com/apache/solr/pull/1841#discussion_r1294524997
##########
solr/core/src/java/org/apache/solr/core/TracerConfigurator.java:
##########
@@ -17,83 +17,88 @@
package org.apache.solr.core;
-import io.opentracing.Scope;
-import io.opentracing.Span;
-import io.opentracing.Tracer;
-import io.opentracing.util.GlobalTracer;
-import java.lang.invoke.MethodHandles;
-import java.util.concurrent.atomic.AtomicReference;
-import org.apache.solr.common.util.ExecutorUtil;
+import io.opentelemetry.api.trace.Tracer;
import org.apache.solr.util.plugin.NamedListInitializedPlugin;
-import org.slf4j.Logger;
-import org.slf4j.LoggerFactory;
+import org.apache.solr.util.tracing.TraceUtils;
-/** Produces an OpenTracing {@link Tracer} from configuration. */
+/** Produces a {@link Tracer} from configuration. */
public abstract class TracerConfigurator implements NamedListInitializedPlugin {
Review Comment:
We could alternatively do followup PRs for this change, since we have a 1:1 change of tracing library in this PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
[GitHub] [solr] stillalex commented on a diff in pull request #1841: SOLR-16536 Replace OpenTracing instrumentation with OpenTelemetry
Posted by "stillalex (via GitHub)" <gi...@apache.org>.
stillalex commented on code in PR #1841:
URL: https://github.com/apache/solr/pull/1841#discussion_r1294577381
##########
solr/modules/opentelemetry/src/java/org/apache/solr/opentelemetry/OtelTracerConfigurator.java:
##########
@@ -16,38 +16,56 @@
*/
package org.apache.solr.opentelemetry;
-import io.opentelemetry.opentracingshim.OpenTracingShim;
-import io.opentelemetry.sdk.OpenTelemetrySdk;
+import io.opentelemetry.api.GlobalOpenTelemetry;
+import io.opentelemetry.api.trace.Tracer;
import io.opentelemetry.sdk.autoconfigure.AutoConfiguredOpenTelemetrySdk;
-import io.opentracing.Tracer;
import java.lang.invoke.MethodHandles;
import java.util.Arrays;
import java.util.HashMap;
import java.util.Locale;
import java.util.Map;
import java.util.Objects;
import java.util.stream.Collectors;
+import org.apache.solr.common.util.NamedList;
import org.apache.solr.core.TracerConfigurator;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
/**
- * OpenTracing TracerConfigurator implementation which exports spans to OpenTelemetry in OTLP
- * format. This impl re-uses the existing OpenTracing instrumentation through a shim, and takes care
- * of properly closing the backing Tracer when Solr shuts down.
+ * Tracing TracerConfigurator implementation which exports spans to OpenTelemetry in OTLP format.
*/
public class OtelTracerConfigurator extends TracerConfigurator {
private static final Logger log = LoggerFactory.getLogger(MethodHandles.lookup().lookupClass());
- // Copy of environment. Can be overridden by tests
- Map<String, String> currentEnv = System.getenv();
+
+ private final Map<String, String> currentEnv;
+
+ public OtelTracerConfigurator() {
+ this(System.getenv());
+ }
+
+ OtelTracerConfigurator(Map<String, String> currentEnv) {
+ this.currentEnv = currentEnv;
+ }
@Override
public Tracer getTracer() {
+ // TODO remove reliance on global
+ return GlobalOpenTelemetry.getTracer("solr");
+ }
+
+ @Override
+ public void init(NamedList<?> args) {
+ prepareConfiguration();
+ AutoConfiguredOpenTelemetrySdk.initialize();
+ }
+
+ void prepareConfiguration() {
setDefaultIfNotConfigured("OTEL_SERVICE_NAME", "solr");
setDefaultIfNotConfigured("OTEL_TRACES_EXPORTER", "otlp");
setDefaultIfNotConfigured("OTEL_EXPORTER_OTLP_PROTOCOL", "grpc");
setDefaultIfNotConfigured("OTEL_TRACES_SAMPLER", "parentbased_always_on");
- addOtelResourceAttributes(Map.of("host.name", System.getProperty("host")));
+ setDefaultIfNotConfigured("OTEL_PROPAGATORS", "tracecontext,baggage");
+ addOtelResourceAttributes(Map.of("host.name", System.getProperty("host", "localhost")));
Review Comment:
also, the 'localhost' value is just the fallback. it only gets used when there is no value defined. the impact is zero for correctly setup solr instances
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
[GitHub] [solr] stillalex commented on a diff in pull request #1841: SOLR-16536 Replace OpenTracing instrumentation with OpenTelemetry
Posted by "stillalex (via GitHub)" <gi...@apache.org>.
stillalex commented on code in PR #1841:
URL: https://github.com/apache/solr/pull/1841#discussion_r1295202810
##########
solr/core/src/java/org/apache/solr/api/V2HttpCall.java:
##########
@@ -510,18 +507,21 @@ protected void populateTracingSpan(Span span) {
// if this api has commands ...
final Map<String, JsonSchemaValidator> validators = getValidators(); // should be cached
if (validators != null && validators.isEmpty() == false && solrReq != null) {
- boolean validateInput = true; // because getCommands caches it; and we want it validated later
+ // boolean validateInput = true; // because getCommands caches it; and we want it
+ // validated later
// does this request have one command?
- List<CommandOperation> cmds = solrReq.getCommands(validateInput);
- if (cmds.size() == 1) {
- verb = cmds.get(0).name;
- }
+ // this can clash with other calls to read commands via CommandOperation.readCommands
+ // see "org.apache.solr.security.jwt.JWTAuthPluginIntegrationTest.testMetrics"
+ // List<CommandOperation> cmds = solrReq.getCommands(validateInput);
+ // if (cmds.size() == 1) {
+ // verb = cmds.get(0).name;
+ // }
Review Comment:
could not repro this anymore, so reverted my changes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
[GitHub] [solr] stillalex commented on a diff in pull request #1841: SOLR-16536 Replace OpenTracing instrumentation with OpenTelemetry
Posted by "stillalex (via GitHub)" <gi...@apache.org>.
stillalex commented on code in PR #1841:
URL: https://github.com/apache/solr/pull/1841#discussion_r1296026006
##########
solr/modules/opentelemetry/src/test/org/apache/solr/opentelemetry/CustomTestOtelTracerConfigurator.java:
##########
@@ -0,0 +1,87 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.solr.opentelemetry;
+
+import io.opentelemetry.api.GlobalOpenTelemetry;
+import io.opentelemetry.api.trace.Tracer;
+import io.opentelemetry.sdk.OpenTelemetrySdk;
+import io.opentelemetry.sdk.autoconfigure.AutoConfiguredOpenTelemetrySdk;
+import io.opentelemetry.sdk.testing.exporter.InMemorySpanExporter;
+import io.opentelemetry.sdk.trace.export.SimpleSpanProcessor;
+import org.apache.solr.common.util.NamedList;
+
+public class CustomTestOtelTracerConfigurator extends OtelTracerConfigurator {
+
+ static {
+ if (System.getProperty("host") == null) {
+ System.setProperty("host", "localhost");
+ }
+ }
+
+ private static final InMemorySpanExporter exporter = InMemorySpanExporter.create();
Review Comment:
yes the error is strange. I don't think the exported is the cause but I added shutdown to it, it was a good suggestion (I think this gets closed via the otel sdk clode method).
the issue seems to be some component calls GlobalOpenTelemetry.get() before the actual init happens (as an aside this is a very strange design decision on otel side: calling GlobalOpenTelemetry.get() will actually _set_ it to noop and will not allow anyone else to set it again hence the failure in the logs). I am also considering the test itself as a culprit (it does a call to .get() to verify otel is active). still massaging the test code to make it into a test failure instead of a crash-like situation.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
[GitHub] [solr] stillalex commented on a diff in pull request #1841: SOLR-16536 Replace OpenTracing instrumentation with OpenTelemetry
Posted by "stillalex (via GitHub)" <gi...@apache.org>.
stillalex commented on code in PR #1841:
URL: https://github.com/apache/solr/pull/1841#discussion_r1294555321
##########
solr/modules/opentelemetry/src/java/org/apache/solr/opentelemetry/OtelTracerConfigurator.java:
##########
@@ -16,38 +16,56 @@
*/
package org.apache.solr.opentelemetry;
-import io.opentelemetry.opentracingshim.OpenTracingShim;
-import io.opentelemetry.sdk.OpenTelemetrySdk;
+import io.opentelemetry.api.GlobalOpenTelemetry;
+import io.opentelemetry.api.trace.Tracer;
import io.opentelemetry.sdk.autoconfigure.AutoConfiguredOpenTelemetrySdk;
-import io.opentracing.Tracer;
import java.lang.invoke.MethodHandles;
import java.util.Arrays;
import java.util.HashMap;
import java.util.Locale;
import java.util.Map;
import java.util.Objects;
import java.util.stream.Collectors;
+import org.apache.solr.common.util.NamedList;
import org.apache.solr.core.TracerConfigurator;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
/**
- * OpenTracing TracerConfigurator implementation which exports spans to OpenTelemetry in OTLP
- * format. This impl re-uses the existing OpenTracing instrumentation through a shim, and takes care
- * of properly closing the backing Tracer when Solr shuts down.
+ * Tracing TracerConfigurator implementation which exports spans to OpenTelemetry in OTLP format.
*/
public class OtelTracerConfigurator extends TracerConfigurator {
private static final Logger log = LoggerFactory.getLogger(MethodHandles.lookup().lookupClass());
- // Copy of environment. Can be overridden by tests
- Map<String, String> currentEnv = System.getenv();
+
+ private final Map<String, String> currentEnv;
+
+ public OtelTracerConfigurator() {
+ this(System.getenv());
+ }
+
+ OtelTracerConfigurator(Map<String, String> currentEnv) {
+ this.currentEnv = currentEnv;
+ }
@Override
public Tracer getTracer() {
+ // TODO remove reliance on global
+ return GlobalOpenTelemetry.getTracer("solr");
+ }
+
+ @Override
+ public void init(NamedList<?> args) {
Review Comment:
answered below. would prefer a different PR
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
[GitHub] [solr] stillalex commented on a diff in pull request #1841: SOLR-16536 Replace OpenTracing instrumentation with OpenTelemetry
Posted by "stillalex (via GitHub)" <gi...@apache.org>.
stillalex commented on code in PR #1841:
URL: https://github.com/apache/solr/pull/1841#discussion_r1294556596
##########
solr/core/src/java/org/apache/solr/servlet/HttpSolrCall.java:
##########
@@ -977,15 +979,32 @@ protected void writeResponse(
}
if (Method.HEAD != reqMethod) {
- OutputStream out = response.getOutputStream();
- QueryResponseWriterUtil.writeQueryResponse(out, responseWriter, solrReq, solrRsp, ct);
+ writeQueryResponseWithTracing(responseWriter, solrRsp, ct);
Review Comment:
makes sense. will try to remove this bit for a followup. there are 2 new spans I added, will remove both.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
[GitHub] [solr] janhoy commented on pull request #1841: SOLR-16536 Replace OpenTracing instrumentation with OpenTelemetry
Posted by "janhoy (via GitHub)" <gi...@apache.org>.
janhoy commented on PR #1841:
URL: https://github.com/apache/solr/pull/1841#issuecomment-1681360684
Yes, splitting out the otel version upgrade is a good idea!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
[GitHub] [solr] janhoy commented on a diff in pull request #1841: SOLR-16354 Migrate from Jaeger/OpenTracing to OTEL
Posted by "janhoy (via GitHub)" <gi...@apache.org>.
janhoy commented on code in PR #1841:
URL: https://github.com/apache/solr/pull/1841#discussion_r1294442984
##########
solr/modules/opentelemetry/src/java/org/apache/solr/opentelemetry/OtelTracerConfigurator.java:
##########
@@ -16,38 +16,56 @@
*/
package org.apache.solr.opentelemetry;
-import io.opentelemetry.opentracingshim.OpenTracingShim;
-import io.opentelemetry.sdk.OpenTelemetrySdk;
+import io.opentelemetry.api.GlobalOpenTelemetry;
+import io.opentelemetry.api.trace.Tracer;
import io.opentelemetry.sdk.autoconfigure.AutoConfiguredOpenTelemetrySdk;
-import io.opentracing.Tracer;
import java.lang.invoke.MethodHandles;
import java.util.Arrays;
import java.util.HashMap;
import java.util.Locale;
import java.util.Map;
import java.util.Objects;
import java.util.stream.Collectors;
+import org.apache.solr.common.util.NamedList;
import org.apache.solr.core.TracerConfigurator;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
/**
- * OpenTracing TracerConfigurator implementation which exports spans to OpenTelemetry in OTLP
- * format. This impl re-uses the existing OpenTracing instrumentation through a shim, and takes care
- * of properly closing the backing Tracer when Solr shuts down.
+ * Tracing TracerConfigurator implementation which exports spans to OpenTelemetry in OTLP format.
*/
public class OtelTracerConfigurator extends TracerConfigurator {
private static final Logger log = LoggerFactory.getLogger(MethodHandles.lookup().lookupClass());
- // Copy of environment. Can be overridden by tests
- Map<String, String> currentEnv = System.getenv();
+
+ private final Map<String, String> currentEnv;
+
+ public OtelTracerConfigurator() {
+ this(System.getenv());
+ }
+
+ OtelTracerConfigurator(Map<String, String> currentEnv) {
+ this.currentEnv = currentEnv;
+ }
@Override
public Tracer getTracer() {
+ // TODO remove reliance on global
+ return GlobalOpenTelemetry.getTracer("solr");
+ }
+
+ @Override
+ public void init(NamedList<?> args) {
+ prepareConfiguration();
+ AutoConfiguredOpenTelemetrySdk.initialize();
+ }
+
+ void prepareConfiguration() {
setDefaultIfNotConfigured("OTEL_SERVICE_NAME", "solr");
setDefaultIfNotConfigured("OTEL_TRACES_EXPORTER", "otlp");
setDefaultIfNotConfigured("OTEL_EXPORTER_OTLP_PROTOCOL", "grpc");
setDefaultIfNotConfigured("OTEL_TRACES_SAMPLER", "parentbased_always_on");
- addOtelResourceAttributes(Map.of("host.name", System.getProperty("host")));
+ setDefaultIfNotConfigured("OTEL_PROPAGATORS", "tracecontext,baggage");
+ addOtelResourceAttributes(Map.of("host.name", System.getProperty("host", "localhost")));
Review Comment:
Why localhost here? All validly started Solr instances will have the `host` sysprop set?
##########
solr/core/src/java/org/apache/solr/api/AnnotatedApi.java:
##########
@@ -215,7 +214,6 @@ public void call(SolrQueryRequest req, SolrQueryResponse rsp) {
}
for (CommandOperation cmd : cmds) {
- TraceUtils.ifNotNoop(req.getSpan(), (span) -> span.log("Command: " + cmd.name));
Review Comment:
Why is this removed?
##########
solr/modules/opentelemetry/src/java/org/apache/solr/opentelemetry/OtelTracerConfigurator.java:
##########
@@ -16,38 +16,56 @@
*/
package org.apache.solr.opentelemetry;
-import io.opentelemetry.opentracingshim.OpenTracingShim;
-import io.opentelemetry.sdk.OpenTelemetrySdk;
+import io.opentelemetry.api.GlobalOpenTelemetry;
+import io.opentelemetry.api.trace.Tracer;
import io.opentelemetry.sdk.autoconfigure.AutoConfiguredOpenTelemetrySdk;
-import io.opentracing.Tracer;
import java.lang.invoke.MethodHandles;
import java.util.Arrays;
import java.util.HashMap;
import java.util.Locale;
import java.util.Map;
import java.util.Objects;
import java.util.stream.Collectors;
+import org.apache.solr.common.util.NamedList;
import org.apache.solr.core.TracerConfigurator;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
/**
- * OpenTracing TracerConfigurator implementation which exports spans to OpenTelemetry in OTLP
- * format. This impl re-uses the existing OpenTracing instrumentation through a shim, and takes care
- * of properly closing the backing Tracer when Solr shuts down.
+ * Tracing TracerConfigurator implementation which exports spans to OpenTelemetry in OTLP format.
*/
public class OtelTracerConfigurator extends TracerConfigurator {
private static final Logger log = LoggerFactory.getLogger(MethodHandles.lookup().lookupClass());
- // Copy of environment. Can be overridden by tests
- Map<String, String> currentEnv = System.getenv();
+
+ private final Map<String, String> currentEnv;
+
+ public OtelTracerConfigurator() {
+ this(System.getenv());
+ }
+
+ OtelTracerConfigurator(Map<String, String> currentEnv) {
+ this.currentEnv = currentEnv;
+ }
@Override
public Tracer getTracer() {
+ // TODO remove reliance on global
+ return GlobalOpenTelemetry.getTracer("solr");
+ }
+
+ @Override
+ public void init(NamedList<?> args) {
Review Comment:
This is only called if there is a `<tracerConfig` tag in solrconfig.xml, right?
In https://issues.apache.org/jira/browse/SOLR-16536?focusedCommentId=17655736&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-17655736 I coin the idea of getting rid of TracerConfigurator altogether, and instead initialize the tracer in static context here in the `opentelemetry` module. WDYT?
##########
solr/core/src/java/org/apache/solr/servlet/HttpSolrCall.java:
##########
@@ -977,15 +979,32 @@ protected void writeResponse(
}
if (Method.HEAD != reqMethod) {
- OutputStream out = response.getOutputStream();
- QueryResponseWriterUtil.writeQueryResponse(out, responseWriter, solrReq, solrRsp, ct);
+ writeQueryResponseWithTracing(responseWriter, solrRsp, ct);
Review Comment:
Is this `writeQueryResponse` a new span that we did not record before?
##########
solr/core/src/java/org/apache/solr/api/V2HttpCall.java:
##########
@@ -510,18 +507,21 @@ protected void populateTracingSpan(Span span) {
// if this api has commands ...
final Map<String, JsonSchemaValidator> validators = getValidators(); // should be cached
if (validators != null && validators.isEmpty() == false && solrReq != null) {
- boolean validateInput = true; // because getCommands caches it; and we want it validated later
+ // boolean validateInput = true; // because getCommands caches it; and we want it
+ // validated later
// does this request have one command?
- List<CommandOperation> cmds = solrReq.getCommands(validateInput);
- if (cmds.size() == 1) {
- verb = cmds.get(0).name;
- }
+ // this can clash with other calls to read commands via CommandOperation.readCommands
+ // see "org.apache.solr.security.jwt.JWTAuthPluginIntegrationTest.testMetrics"
+ // List<CommandOperation> cmds = solrReq.getCommands(validateInput);
+ // if (cmds.size() == 1) {
+ // verb = cmds.get(0).name;
+ // }
Review Comment:
Can you explain the clash?
##########
solr/core/src/java/org/apache/solr/handler/RequestHandlerBase.java:
##########
@@ -241,6 +243,18 @@ public void handleRequest(SolrQueryRequest req, SolrQueryResponse rsp) {
}
}
+ private void handleRequestBodyWithTracing(SolrQueryRequest req, SolrQueryResponse rsp)
Review Comment:
Why is this handled differently form opentracing? How was the span started earlier?
##########
solr/core/src/java/org/apache/solr/servlet/SolrDispatchFilter.java:
##########
@@ -255,15 +247,12 @@ private void dispatch(
Action result = call.call();
switch (result) {
case PASSTHROUGH:
- getSpan(request).log("SolrDispatchFilter PASSTHROUGH");
Review Comment:
Not sure what this log call did, are we removing something from the trace here that we had before?
##########
solr/core/src/java/org/apache/solr/core/TracerConfigurator.java:
##########
@@ -17,83 +17,88 @@
package org.apache.solr.core;
-import io.opentracing.Scope;
-import io.opentracing.Span;
-import io.opentracing.Tracer;
-import io.opentracing.util.GlobalTracer;
-import java.lang.invoke.MethodHandles;
-import java.util.concurrent.atomic.AtomicReference;
-import org.apache.solr.common.util.ExecutorUtil;
+import io.opentelemetry.api.trace.Tracer;
import org.apache.solr.util.plugin.NamedListInitializedPlugin;
-import org.slf4j.Logger;
-import org.slf4j.LoggerFactory;
+import org.apache.solr.util.tracing.TraceUtils;
-/** Produces an OpenTracing {@link Tracer} from configuration. */
+/** Produces a {@link Tracer} from configuration. */
public abstract class TracerConfigurator implements NamedListInitializedPlugin {
Review Comment:
See comment elsewhere - can we get away with this entire plugin class, and remove the need for `<tracerConfig` in solr.xml to configure tracing? Since this will be a Solr 10 PR this breaking change is ok.
##########
solr/core/src/java/org/apache/solr/util/tracing/HttpServletRequestGetter.java:
##########
@@ -0,0 +1,42 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.solr.util.tracing;
+
+import io.opentelemetry.context.propagation.TextMapGetter;
+import java.util.Iterator;
+import javax.servlet.http.HttpServletRequest;
+
+public class HttpServletRequestGetter implements TextMapGetter<HttpServletRequest> {
Review Comment:
JavaDoc on class
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
[GitHub] [solr] janhoy commented on a diff in pull request #1841: SOLR-16536 Replace OpenTracing instrumentation with OpenTelemetry
Posted by "janhoy (via GitHub)" <gi...@apache.org>.
janhoy commented on code in PR #1841:
URL: https://github.com/apache/solr/pull/1841#discussion_r1294975785
##########
solr/modules/opentelemetry/src/java/org/apache/solr/opentelemetry/OtelTracerConfigurator.java:
##########
@@ -16,38 +16,56 @@
*/
package org.apache.solr.opentelemetry;
-import io.opentelemetry.opentracingshim.OpenTracingShim;
-import io.opentelemetry.sdk.OpenTelemetrySdk;
+import io.opentelemetry.api.GlobalOpenTelemetry;
+import io.opentelemetry.api.trace.Tracer;
import io.opentelemetry.sdk.autoconfigure.AutoConfiguredOpenTelemetrySdk;
-import io.opentracing.Tracer;
import java.lang.invoke.MethodHandles;
import java.util.Arrays;
import java.util.HashMap;
import java.util.Locale;
import java.util.Map;
import java.util.Objects;
import java.util.stream.Collectors;
+import org.apache.solr.common.util.NamedList;
import org.apache.solr.core.TracerConfigurator;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
/**
- * OpenTracing TracerConfigurator implementation which exports spans to OpenTelemetry in OTLP
- * format. This impl re-uses the existing OpenTracing instrumentation through a shim, and takes care
- * of properly closing the backing Tracer when Solr shuts down.
+ * Tracing TracerConfigurator implementation which exports spans to OpenTelemetry in OTLP format.
*/
public class OtelTracerConfigurator extends TracerConfigurator {
private static final Logger log = LoggerFactory.getLogger(MethodHandles.lookup().lookupClass());
- // Copy of environment. Can be overridden by tests
- Map<String, String> currentEnv = System.getenv();
+
+ private final Map<String, String> currentEnv;
+
+ public OtelTracerConfigurator() {
+ this(System.getenv());
+ }
+
+ OtelTracerConfigurator(Map<String, String> currentEnv) {
+ this.currentEnv = currentEnv;
+ }
@Override
public Tracer getTracer() {
+ // TODO remove reliance on global
+ return GlobalOpenTelemetry.getTracer("solr");
+ }
+
+ @Override
+ public void init(NamedList<?> args) {
+ prepareConfiguration();
+ AutoConfiguredOpenTelemetrySdk.initialize();
+ }
+
+ void prepareConfiguration() {
setDefaultIfNotConfigured("OTEL_SERVICE_NAME", "solr");
setDefaultIfNotConfigured("OTEL_TRACES_EXPORTER", "otlp");
setDefaultIfNotConfigured("OTEL_EXPORTER_OTLP_PROTOCOL", "grpc");
setDefaultIfNotConfigured("OTEL_TRACES_SAMPLER", "parentbased_always_on");
- addOtelResourceAttributes(Map.of("host.name", System.getProperty("host")));
+ setDefaultIfNotConfigured("OTEL_PROPAGATORS", "tracecontext,baggage");
+ addOtelResourceAttributes(Map.of("host.name", System.getProperty("host", "localhost")));
Review Comment:
Sure, it won't be any harm, although I prefer if production code is explicit and fails fast, i.e. better with `null` as host than faulty `localhost`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
[GitHub] [solr] stillalex commented on a diff in pull request #1841: SOLR-16536 Replace OpenTracing instrumentation with OpenTelemetry
Posted by "stillalex (via GitHub)" <gi...@apache.org>.
stillalex commented on code in PR #1841:
URL: https://github.com/apache/solr/pull/1841#discussion_r1296026006
##########
solr/modules/opentelemetry/src/test/org/apache/solr/opentelemetry/CustomTestOtelTracerConfigurator.java:
##########
@@ -0,0 +1,87 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.solr.opentelemetry;
+
+import io.opentelemetry.api.GlobalOpenTelemetry;
+import io.opentelemetry.api.trace.Tracer;
+import io.opentelemetry.sdk.OpenTelemetrySdk;
+import io.opentelemetry.sdk.autoconfigure.AutoConfiguredOpenTelemetrySdk;
+import io.opentelemetry.sdk.testing.exporter.InMemorySpanExporter;
+import io.opentelemetry.sdk.trace.export.SimpleSpanProcessor;
+import org.apache.solr.common.util.NamedList;
+
+public class CustomTestOtelTracerConfigurator extends OtelTracerConfigurator {
+
+ static {
+ if (System.getProperty("host") == null) {
+ System.setProperty("host", "localhost");
+ }
+ }
+
+ private static final InMemorySpanExporter exporter = InMemorySpanExporter.create();
Review Comment:
yes the error is strange. I don't think the exported is the cause but I added shutdown to it, it was a good suggestion (I think this gets closed via the otel sdk clode method).
the issue seems to be some component calls GlobalOpenTelemetry.get() before the actual init happens (as an aside this is a very strange design decision on otel side: calling GlobalOpenTelemetry.get() will actually _set_ it to noop and will not allow anyone else to set it again hence the failure in the logs). I am also considering the test itself as a culprit (it does a call to .get() to verify otel is active). still massaging the test code to make it into a test failure instead of a crash-like situation.
```
Caused by: java.lang.IllegalStateException: GlobalOpenTelemetry.set has already been called. GlobalOpenTelemetry.set must be called only once before any calls to GlobalOpenTelemetry.get. If you are using the OpenTelemetrySdk, use OpenTelemetrySdkBuilder.buildAndRegisterGlobal instead. Previous invocation set to cause of this exception.
2> at io.opentelemetry.api.GlobalOpenTelemetry.set(GlobalOpenTelemetry.java:104) ~[opentelemetry-api-1.28.0.jar:1.28.0]
2> at io.opentelemetry.sdk.autoconfigure.AutoConfiguredOpenTelemetrySdkBuilder.build(AutoConfiguredOpenTelemetrySdkBuilder.java:401) ~[opentelemetry-sdk-extension-autoconfigure-1.28.0.jar:1.28.0]
2> ... 21 more
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
[GitHub] [solr] stillalex commented on pull request #1841: SOLR-16536 Replace OpenTracing instrumentation with OpenTelemetry
Posted by "stillalex (via GitHub)" <gi...@apache.org>.
stillalex commented on PR #1841:
URL: https://github.com/apache/solr/pull/1841#issuecomment-1680999594
quick smoke test on benchmarking side, I tried running the existing `CloudIndexing` benchmark and numbers seem pretty close (I am sure there are more opportunities for improvement but I did not spend much time on this):
* 5 threads indexing over 60 seconds
```
./jmh.sh -wi 1 -i 1 -r 60 CloudIndexing -t 5 -p nodeCount=1 -p numShards=1 -p numReplicas=1 -p useStringUtf8Over=0 -p directBuffer=true
```
* PR
```
Benchmark (directBuffer) (nodeCount) (numReplicas) (numShards) (scale) (useStringUtf8Over) Mode Cnt Score Error Units
CloudIndexing.indexLargeDoc true 1 1 1 1 0 thrpt 1116.343 ops/s
CloudIndexing.indexSmallDoc true 1 1 1 1 0 thrpt 29966.343 ops/s
```
* MAIN
```
Benchmark (directBuffer) (nodeCount) (numReplicas) (numShards) (scale) (useStringUtf8Over) Mode Cnt Score Error Units
CloudIndexing.indexLargeDoc true 1 1 1 1 0 thrpt 1419.507 ops/s
CloudIndexing.indexSmallDoc true 1 1 1 1 0 thrpt 30111.511 ops/s
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
[GitHub] [solr] janhoy commented on pull request #1841: SOLR-16536 Replace OpenTracing instrumentation with OpenTelemetry
Posted by "janhoy (via GitHub)" <gi...@apache.org>.
janhoy commented on PR #1841:
URL: https://github.com/apache/solr/pull/1841#issuecomment-1680223334
Tests seem to fail with `SEVERE: 9 threads leaked from SUITE scope at org.apache.solr.opentelemetry.TestDistributedTracing:`. Cannot reproduce locally. Re-running crave tests.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
[GitHub] [solr] stillalex commented on a diff in pull request #1841: SOLR-16536 Replace OpenTracing instrumentation with OpenTelemetry
Posted by "stillalex (via GitHub)" <gi...@apache.org>.
stillalex commented on code in PR #1841:
URL: https://github.com/apache/solr/pull/1841#discussion_r1294592758
##########
solr/core/src/java/org/apache/solr/servlet/HttpSolrCall.java:
##########
@@ -977,15 +979,32 @@ protected void writeResponse(
}
if (Method.HEAD != reqMethod) {
- OutputStream out = response.getOutputStream();
- QueryResponseWriterUtil.writeQueryResponse(out, responseWriter, solrReq, solrRsp, ct);
+ writeQueryResponseWithTracing(responseWriter, solrRsp, ct);
Review Comment:
new methods removed. will keep them for followup
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
[GitHub] [solr] stillalex commented on a diff in pull request #1841: SOLR-16536 Replace OpenTracing instrumentation with OpenTelemetry
Posted by "stillalex (via GitHub)" <gi...@apache.org>.
stillalex commented on code in PR #1841:
URL: https://github.com/apache/solr/pull/1841#discussion_r1294592415
##########
solr/core/src/java/org/apache/solr/handler/RequestHandlerBase.java:
##########
@@ -241,6 +243,18 @@ public void handleRequest(SolrQueryRequest req, SolrQueryResponse rsp) {
}
}
+ private void handleRequestBodyWithTracing(SolrQueryRequest req, SolrQueryResponse rsp)
Review Comment:
new span method removed, as mentioned below
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
[GitHub] [solr] stillalex merged pull request #1841: SOLR-16536 Replace OpenTracing instrumentation with OpenTelemetry
Posted by "stillalex (via GitHub)" <gi...@apache.org>.
stillalex merged PR #1841:
URL: https://github.com/apache/solr/pull/1841
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
[GitHub] [solr] janhoy commented on pull request #1841: SOLR-16536 Replace OpenTracing instrumentation with OpenTelemetry
Posted by "janhoy (via GitHub)" <gi...@apache.org>.
janhoy commented on PR #1841:
URL: https://github.com/apache/solr/pull/1841#issuecomment-1680279442
I reproduced the thread leak locally, but it does not happen every time. Also, the new Crave test run was successful. Looks like there is a timing issue. @stillalex how is the tracer life cycle handled now that we don't explicitly shut down the tracer in `CoreContainer`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
[GitHub] [solr] stillalex commented on a diff in pull request #1841: SOLR-16536 Replace OpenTracing instrumentation with OpenTelemetry
Posted by "stillalex (via GitHub)" <gi...@apache.org>.
stillalex commented on code in PR #1841:
URL: https://github.com/apache/solr/pull/1841#discussion_r1294556596
##########
solr/core/src/java/org/apache/solr/servlet/HttpSolrCall.java:
##########
@@ -977,15 +979,32 @@ protected void writeResponse(
}
if (Method.HEAD != reqMethod) {
- OutputStream out = response.getOutputStream();
- QueryResponseWriterUtil.writeQueryResponse(out, responseWriter, solrReq, solrRsp, ct);
+ writeQueryResponseWithTracing(responseWriter, solrRsp, ct);
Review Comment:
makes sense. will try to remove this bit for a followup
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
[GitHub] [solr] stillalex commented on a diff in pull request #1841: SOLR-16536 Replace OpenTracing instrumentation with OpenTelemetry
Posted by "stillalex (via GitHub)" <gi...@apache.org>.
stillalex commented on code in PR #1841:
URL: https://github.com/apache/solr/pull/1841#discussion_r1294595678
##########
solr/core/src/java/org/apache/solr/util/tracing/HttpServletRequestGetter.java:
##########
@@ -0,0 +1,42 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.solr.util.tracing;
+
+import io.opentelemetry.context.propagation.TextMapGetter;
+import java.util.Iterator;
+import javax.servlet.http.HttpServletRequest;
+
+public class HttpServletRequestGetter implements TextMapGetter<HttpServletRequest> {
Review Comment:
javadocs added
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
[GitHub] [solr] stillalex commented on pull request #1841: SOLR-16536 Replace OpenTracing instrumentation with OpenTelemetry
Posted by "stillalex (via GitHub)" <gi...@apache.org>.
stillalex commented on PR #1841:
URL: https://github.com/apache/solr/pull/1841#issuecomment-1679730960
> Perhaps we should emit both for some versions, or implement the suggested env var OTEL_SEMCONV_STABILITY_OPT_IN?
if this is for 2 fields only (`http.status_code` => `http.response.status_code` and `http.method` => `http.request.method`) it feels like overkill to implement this mechanism. I would emit both attributes old and new for a while and then drop them. let me know if you feel differently.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
[GitHub] [solr] stillalex commented on pull request #1841: SOLR-16536 Replace OpenTracing instrumentation with OpenTelemetry
Posted by "stillalex (via GitHub)" <gi...@apache.org>.
stillalex commented on PR #1841:
URL: https://github.com/apache/solr/pull/1841#issuecomment-1679599489
sorry, had to do a rebase on top of main branch to fix some conflicts.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org