You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@druid.apache.org by "jtuglu1 (via GitHub)" <gi...@apache.org> on 2025/09/24 19:01:00 UTC
[PR] Support verbose SQL query status metrics (druid)
jtuglu1 opened a new pull request, #18569:
URL: https://github.com/apache/druid/pull/18569
Also simplifies/unifies the query count metric tracking architecture between the multiple query resource classes.
<!-- Thanks for trying to help us make Apache Druid be the best it can be! Please fill out as much of the following information as is possible (where relevant, and remove it when irrelevant) to help make the intention and scope of this PR clear in order to ease review. -->
<!-- Please read the doc for contribution (https://github.com/apache/druid/blob/master/CONTRIBUTING.md) before making this PR. Also, once you open a PR, please _avoid using force pushes and rebasing_ since these make it difficult for reviewers to see what you've changed in response to their reviews. See [the 'If your pull request shows conflicts with master' section](https://github.com/apache/druid/blob/master/CONTRIBUTING.md#if-your-pull-request-shows-conflicts-with-master) for more details. -->
Fixes #XXXX.
<!-- Replace XXXX with the id of the issue fixed in this PR. Remove this section if there is no corresponding issue. Don't reference the issue in the title of this pull-request. -->
<!-- If you are a committer, follow the PR action item checklist for committers:
https://github.com/apache/druid/blob/master/dev/committer-instructions.md#pr-and-issue-action-item-checklist-for-committers. -->
### Description
<!-- Describe the goal of this PR, what problem are you fixing. If there is a corresponding issue (referenced above), it's not necessary to repeat the description here, however, you may choose to keep one summary sentence. -->
<!-- Describe your patch: what did you change in code? How did you fix the problem? -->
<!-- If there are several relatively logically separate changes in this PR, create a mini-section for each of them. For example: -->
#### Fixed the bug ...
#### Renamed the class ...
#### Added a forbidden-apis entry ...
<!--
In each section, please describe design decisions made, including:
- Choice of algorithms
- Behavioral aspects. What configuration values are acceptable? How are corner cases and error conditions handled, such as when there are insufficient resources?
- Class organization and design (how the logic is split between classes, inheritance, composition, design patterns)
- Method organization and design (how the logic is split between methods, parameters and return types)
- Naming (class, method, API, configuration, HTTP endpoint, names of emitted metrics)
-->
<!-- It's good to describe an alternative design (or mention an alternative name) for every design (or naming) decision point and compare the alternatives with the designs that you've implemented (or the names you've chosen) to highlight the advantages of the chosen designs and names. -->
<!-- If there was a discussion of the design of the feature implemented in this PR elsewhere (e. g. a "Proposal" issue, any other issue, or a thread in the development mailing list), link to that discussion from this PR description and explain what have changed in your final design compared to your original proposal or the consensus version in the end of the discussion. If something hasn't changed since the original discussion, you can omit a detailed discussion of those aspects of the design here, perhaps apart from brief mentioning for the sake of readability of this PR description. -->
<!-- Some of the aspects mentioned above may be omitted for simple and small changes. -->
#### Release note
<!-- Give your best effort to summarize your changes in a couple of sentences aimed toward Druid users.
If your change doesn't have end user impact, you can skip this section.
For tips about how to write a good release note, see [Release notes](https://github.com/apache/druid/blob/master/CONTRIBUTING.md#release-notes).
-->
<hr>
##### Key changed/added classes in this PR
* `MyFoo`
* `OurBar`
* `TheirBaz`
<hr>
<!-- Check the items by putting "x" in the brackets for the done things. Not all of these items apply to every PR. Remove the items which are not done or not relevant to the PR. None of the items from the checklist below are strictly necessary, but it would be very helpful if you at least self-review the PR. -->
This PR has:
- [ ] been self-reviewed.
- [ ] using the [concurrency checklist](https://github.com/apache/druid/blob/master/dev/code-review/concurrency.md) (Remove this item if the PR doesn't have any relation to concurrency.)
- [ ] added documentation for new or modified features or behaviors.
- [ ] a release note entry in the PR description.
- [ ] added Javadocs for most classes and all non-trivial methods. Linked related entities via Javadoc links.
- [ ] added or updated version, license, or notice information in [licenses.yaml](https://github.com/apache/druid/blob/master/dev/license.md)
- [ ] added comments explaining the "why" and the intent of the code wherever would not be obvious for an unfamiliar reader.
- [ ] added unit tests or modified existing tests to cover new code paths, ensuring the threshold for [code coverage](https://github.com/apache/druid/blob/master/dev/code-review/code-coverage.md) is met.
- [ ] added integration tests.
- [ ] been tested in a test Druid cluster.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] Support verbose SQL query status metrics (druid)
Posted by "kfaraz (via GitHub)" <gi...@apache.org>.
kfaraz commented on code in PR #18569:
URL: https://github.com/apache/druid/pull/18569#discussion_r2377945356
##########
sql/src/main/java/org/apache/druid/sql/http/SqlResource.java:
##########
@@ -97,7 +98,8 @@ public SqlResource(
final SqlEngineRegistry sqlEngineRegistry,
final SqlResourceQueryResultPusherFactory resultPusherFactory,
final DefaultQueryConfig defaultQueryConfig,
- final ServerConfig serverConfig
+ final ServerConfig serverConfig,
+ final QueryCountStatsProvider counter
Review Comment:
Inject the annotated impl here to keep SQL and native query stats separate.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] Support verbose SQL query status metrics (druid)
Posted by "kfaraz (via GitHub)" <gi...@apache.org>.
kfaraz commented on code in PR #18569:
URL: https://github.com/apache/druid/pull/18569#discussion_r2378098904
##########
sql/src/main/java/org/apache/druid/sql/http/SqlHttpModule.java:
##########
@@ -35,6 +37,7 @@ public class SqlHttpModule implements Module
public void configure(Binder binder)
{
binder.bind(SqlResource.class).in(LazySingleton.class);
+ binder.bind(QueryCountStatsProvider.class).to(BaseQueryCountResource.class).in(LazySingleton.class);
Review Comment:
I see. That's probably because quidem test Guice setup doesn't use a CliBroker directly.
In that case, let's move this binding to the appropriate place in the quidem test setup itself.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] Support verbose SQL query status metrics (druid)
Posted by "kfaraz (via GitHub)" <gi...@apache.org>.
kfaraz commented on code in PR #18569:
URL: https://github.com/apache/druid/pull/18569#discussion_r2378618271
##########
sql/src/main/java/org/apache/druid/sql/http/SqlResource.java:
##########
@@ -97,7 +98,8 @@ public SqlResource(
final SqlEngineRegistry sqlEngineRegistry,
final SqlResourceQueryResultPusherFactory resultPusherFactory,
final DefaultQueryConfig defaultQueryConfig,
- final ServerConfig serverConfig
+ final ServerConfig serverConfig,
+ final QueryCountStatsProvider counter
Review Comment:
Why would it be confusing? You just need to use an annotated binding.
Druid code uses it in several places and this is a perfectly valid use case.
We keep several instances of `ServiceClient`, `DruidLeaderSelector`, `ServiceLocator` and some classes with MSQ as well.
Check out the annotations `@IndexingService`, `@Coordinator`, `@Broker` which are used to do this.
There are also several places which use the `@Named` annotation.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] Support verbose SQL query status metrics (druid)
Posted by "jtuglu1 (via GitHub)" <gi...@apache.org>.
jtuglu1 commented on code in PR #18569:
URL: https://github.com/apache/druid/pull/18569#discussion_r2377948561
##########
server/src/test/java/org/apache/druid/server/metrics/QueryCountStatsMonitorTest.java:
##########
@@ -55,37 +56,57 @@ public void setUp()
queryCountStatsProvider = new QueryCountStatsProvider()
{
- private long successEmitCount = 0;
- private long failedEmitCount = 0;
- private long interruptedEmitCount = 0;
- private long timedOutEmitCount = 0;
+ private final AtomicLong successEmitCount = new AtomicLong(0);
+ private final AtomicLong failedEmitCount = new AtomicLong(0);
+ private final AtomicLong interruptedEmitCount = new AtomicLong(0);
+ private final AtomicLong timedOutEmitCount = new AtomicLong(0);
@Override
public long getSuccessfulQueryCount()
{
- successEmitCount += 1;
- return successEmitCount;
+ return successEmitCount.get();
}
@Override
public long getFailedQueryCount()
{
- failedEmitCount += 2;
- return failedEmitCount;
+ return failedEmitCount.get();
}
@Override
public long getInterruptedQueryCount()
{
- interruptedEmitCount += 3;
- return interruptedEmitCount;
+ return interruptedEmitCount.get();
}
@Override
public long getTimedOutQueryCount()
{
- timedOutEmitCount += 4;
Review Comment:
No idea
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] Support verbose SQL query status metrics (druid)
Posted by "jtuglu1 (via GitHub)" <gi...@apache.org>.
jtuglu1 commented on code in PR #18569:
URL: https://github.com/apache/druid/pull/18569#discussion_r2377957868
##########
sql/src/main/java/org/apache/druid/sql/http/SqlHttpModule.java:
##########
@@ -35,6 +37,7 @@ public class SqlHttpModule implements Module
public void configure(Binder binder)
{
binder.bind(SqlResource.class).in(LazySingleton.class);
+ binder.bind(QueryCountStatsProvider.class).to(BaseQueryCountResource.class).in(LazySingleton.class);
Review Comment:
I needed to add this since quidem tests were failing.
##########
sql/src/main/java/org/apache/druid/sql/http/SqlHttpModule.java:
##########
@@ -35,6 +37,7 @@ public class SqlHttpModule implements Module
public void configure(Binder binder)
{
binder.bind(SqlResource.class).in(LazySingleton.class);
+ binder.bind(QueryCountStatsProvider.class).to(BaseQueryCountResource.class).in(LazySingleton.class);
Review Comment:
I needed to add this binding to `SqlHttpModule` since quidem tests were failing.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] Support verbose SQL query status metrics (druid)
Posted by "jtuglu1 (via GitHub)" <gi...@apache.org>.
jtuglu1 commented on code in PR #18569:
URL: https://github.com/apache/druid/pull/18569#discussion_r2378177246
##########
sql/src/main/java/org/apache/druid/sql/http/SqlResource.java:
##########
@@ -97,7 +98,8 @@ public SqlResource(
final SqlEngineRegistry sqlEngineRegistry,
final SqlResourceQueryResultPusherFactory resultPusherFactory,
final DefaultQueryConfig defaultQueryConfig,
- final ServerConfig serverConfig
+ final ServerConfig serverConfig,
+ final QueryCountStatsProvider counter
Review Comment:
That should be done with tags IMO and not differing metrics. What happened before this change is these 2 `*Resource` classes diverged to the point where there was gap in metrics between both. I'd prefer to keep one shared "counter." That success/failure can already be gauged from `sqlQuery/time`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] Support verbose SQL query status metrics (druid)
Posted by "jtuglu1 (via GitHub)" <gi...@apache.org>.
jtuglu1 commented on code in PR #18569:
URL: https://github.com/apache/druid/pull/18569#discussion_r2377244461
##########
server/src/main/java/org/apache/druid/server/metrics/BaseQueryCountResource.java:
##########
@@ -0,0 +1,85 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.server.metrics;
+
+import org.apache.druid.guice.LazySingleton;
+
+import java.util.concurrent.atomic.AtomicLong;
+
+/*
+ A thread-safe query count statistics tracker. Meant to be a shared between the various query resource handlers present
Review Comment:
will fix typo after review
##########
server/src/main/java/org/apache/druid/server/metrics/BaseQueryCountResource.java:
##########
@@ -0,0 +1,85 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.server.metrics;
+
+import org.apache.druid.guice.LazySingleton;
+
+import java.util.concurrent.atomic.AtomicLong;
+
+/*
+ A thread-safe query count statistics tracker. Meant to be a shared between the various query resource handlers present
Review Comment:
will fix typo after review to minimize CI runs
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] Support verbose SQL query status metrics (druid)
Posted by "kfaraz (via GitHub)" <gi...@apache.org>.
kfaraz commented on code in PR #18569:
URL: https://github.com/apache/druid/pull/18569#discussion_r2377866672
##########
server/src/test/java/org/apache/druid/server/metrics/QueryCountStatsMonitorTest.java:
##########
@@ -55,37 +56,57 @@ public void setUp()
queryCountStatsProvider = new QueryCountStatsProvider()
Review Comment:
Should we just use the new impl `BaseQueryCountResource` here?
##########
server/src/test/java/org/apache/druid/server/QueryResourceTest.java:
##########
@@ -769,7 +777,7 @@ public void testTruncatedResponseContextShouldFail() throws IOException
SIMPLE_TIMESERIES_QUERY.getBytes(StandardCharsets.UTF_8),
queryResource
);
- Assert.assertEquals(1, queryResource.getInterruptedQueryCount());
+ Assert.assertEquals(1, queryResource.counter.getInterruptedQueryCount());
Review Comment:
Let's not access the `counter` via the `QueryResource`
since it doesn't really belong to the resource and is a global service-level entity.
We should just hold a reference to the `counter` itself.
##########
server/src/test/java/org/apache/druid/server/QueryResourceTest.java:
##########
@@ -1379,8 +1391,8 @@ public void testTooManyQuery() throws InterruptedException, ExecutionException
for (Future<Boolean> theFuture : back2) {
Assert.assertTrue(theFuture.get());
}
- Assert.assertEquals(2, queryResource.getSuccessfulQueryCount());
- Assert.assertEquals(1, queryResource.getFailedQueryCount());
+ Assert.assertEquals(2, queryResource.counter.getSuccessfulQueryCount());
+ Assert.assertEquals(1, queryResource.counter.getFailedQueryCount());
Review Comment:
Same.
##########
sql/src/main/java/org/apache/druid/sql/http/SqlResourceQueryResultPusherFactory.java:
##########
@@ -52,7 +53,7 @@ public SqlResourceQueryResultPusherFactory(
this.selfNode = selfNode;
}
- public SqlResourceQueryResultPusher factorize(HttpServletRequest req, HttpStatement stmt, SqlQuery sqlQuery)
+ public SqlResourceQueryResultPusher factorize(final QueryCountStatsProvider counter, HttpServletRequest req, HttpStatement stmt, SqlQuery sqlQuery)
Review Comment:
Please put args on separate lines
##########
sql/src/main/java/org/apache/druid/sql/http/SqlHttpModule.java:
##########
@@ -35,6 +37,7 @@ public class SqlHttpModule implements Module
public void configure(Binder binder)
{
binder.bind(SqlResource.class).in(LazySingleton.class);
+ binder.bind(QueryCountStatsProvider.class).to(BaseQueryCountResource.class).in(LazySingleton.class);
Review Comment:
This is already bound on `CliBroker` which is the only service which uses `SqlHttpModule` (AFAICT).
Do we still need this additional binding?
##########
server/src/test/java/org/apache/druid/server/metrics/QueryCountStatsMonitorTest.java:
##########
@@ -126,6 +155,14 @@ public void testMonitor_emitPendingRequests()
new QueryCountStatsMonitor(queryCountStatsProvider, monitorsConfig, mergeBufferPool);
final StubServiceEmitter emitter = new StubServiceEmitter("service", "host");
monitor.doMonitor(emitter);
+
+ // Mock metrics emission
Review Comment:
```suggestion
// Simulate metrics emission
```
##########
server/src/test/java/org/apache/druid/server/QueryResultPusherTest.java:
##########
@@ -134,9 +133,33 @@ public ResultsWriter start()
assertTrue("recordFailure(e) should have been invoked!", recordFailureInvoked.get());
}
- static class NoopQueryMetricCounter implements QueryMetricCounter
+ static class NoopQueryMetricCounter implements QueryCountStatsProvider
Review Comment:
I suppose this might not be needed anymore either.
We may just use the new impl `BaseQueryCountResource`, unless something in the test asserts that some metric is zero (which we can update to reflect the true value).
##########
server/src/main/java/org/apache/druid/server/QueryResource.java:
##########
@@ -97,7 +97,7 @@ public class QueryResource implements QueryCountStatsProvider
private final AtomicLong failedQueryCount = new AtomicLong();
private final AtomicLong interruptedQueryCount = new AtomicLong();
private final AtomicLong timedOutQueryCount = new AtomicLong();
- private final QueryResourceQueryMetricCounter counter = new QueryResourceQueryMetricCounter();
+ final QueryCountStatsProvider counter;
Review Comment:
```suggestion
private final QueryCountStatsProvider counter;
```
##########
services/src/main/java/org/apache/druid/server/AsyncQueryForwardingServlet.java:
##########
@@ -159,7 +160,8 @@ public AsyncQueryForwardingServlet(
GenericQueryMetricsFactory queryMetricsFactory,
AuthenticatorMapper authenticatorMapper,
Properties properties,
- final ServerConfig serverConfig
+ final ServerConfig serverConfig,
+ QueryCountStatsProvider counter
Review Comment:
Nit: for homogeneity with the other fields
```suggestion
final QueryCountStatsProvider counter
```
##########
server/src/test/java/org/apache/druid/server/QueryResourceTest.java:
##########
@@ -1079,7 +1089,7 @@ public <T> QueryRunner<T> getQueryRunnerForSegments(Query<T> query, Iterable<Seg
Assert.assertEquals("Query did not complete within configured timeout period. You can " +
"increase query timeout or tune the performance of query.", ex.getMessage());
Assert.assertEquals(QueryException.QUERY_TIMEOUT_ERROR_CODE, ex.getErrorCode());
- Assert.assertEquals(1, timeoutQueryResource.getTimedOutQueryCount());
+ Assert.assertEquals(1, timeoutQueryResource.counter.getTimedOutQueryCount());
Review Comment:
Same comment here regarding accessing the `counter` via the resource class.
##########
server/src/main/java/org/apache/druid/server/metrics/BaseQueryCountResource.java:
##########
@@ -0,0 +1,85 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.server.metrics;
+
+import org.apache.druid.guice.LazySingleton;
+
+import java.util.concurrent.atomic.AtomicLong;
+
+/*
+ A thread-safe query count statistics tracker. Meant to be a shared between the various query resource handlers present
+ on an instance.
+*/
+@LazySingleton
Review Comment:
No need to annotate since we are already binding it appropriately where applicable.
##########
sql/src/main/java/org/apache/druid/sql/http/SqlResource.java:
##########
@@ -80,14 +81,14 @@ public class SqlResource
public static final String SQL_HEADER_VALUE = "yes";
private static final Logger log = new Logger(SqlResource.class);
- public static final SqlResourceQueryMetricCounter QUERY_METRIC_COUNTER = new SqlResourceQueryMetricCounter();
private final AuthorizerMapper authorizerMapper;
private final SqlResourceQueryResultPusherFactory resultPusherFactory;
private final SqlLifecycleManager sqlLifecycleManager;
private final SqlEngineRegistry sqlEngineRegistry;
private final DefaultQueryConfig defaultQueryConfig;
private final ServerConfig serverConfig;
+ final QueryCountStatsProvider counter;
Review Comment:
```suggestion
private final QueryCountStatsProvider counter;
```
##########
sql/src/test/java/org/apache/druid/sql/http/SqlResourceTest.java:
##########
@@ -1869,7 +1876,9 @@ public void testTooManyRequestsAfterTotalLaning() throws Exception
}
}
Assert.assertEquals(2, success);
+ Assert.assertEquals(2, resource.counter.getSuccessfulQueryCount());
Assert.assertEquals(1, limited);
+ Assert.assertEquals(1, resource.counter.getFailedQueryCount());
Review Comment:
We shouldn't access `counter` via the resource class.
##########
server/src/test/java/org/apache/druid/server/metrics/QueryCountStatsMonitorTest.java:
##########
@@ -106,15 +127,23 @@ public void testMonitor()
new QueryCountStatsMonitor(queryCountStatsProvider, monitorsConfig, mergeBufferPool);
final StubServiceEmitter emitter = new StubServiceEmitter("service", "host");
monitor.doMonitor(emitter);
+
+ // Mock metrics emission
+ queryCountStatsProvider.incrementSuccess();
+ queryCountStatsProvider.incrementFailed();
+ queryCountStatsProvider.incrementFailed();
+ queryCountStatsProvider.incrementInterrupted();
+ queryCountStatsProvider.incrementTimedOut();
+
emitter.flush();
// Trigger metric emission
monitor.doMonitor(emitter);
Assert.assertEquals(5, emitter.getNumEmittedEvents());
emitter.verifyValue("query/success/count", 1L);
emitter.verifyValue("query/failed/count", 2L);
- emitter.verifyValue("query/interrupted/count", 3L);
- emitter.verifyValue("query/timeout/count", 4L);
- emitter.verifyValue("query/count", 10L);
Review Comment:
I assume these had to be changed due to the weird increments in the test `QueryCountStatsProvider` defined above?
##########
server/src/test/java/org/apache/druid/server/metrics/QueryCountStatsMonitorTest.java:
##########
@@ -55,37 +56,57 @@ public void setUp()
queryCountStatsProvider = new QueryCountStatsProvider()
{
- private long successEmitCount = 0;
- private long failedEmitCount = 0;
- private long interruptedEmitCount = 0;
- private long timedOutEmitCount = 0;
+ private final AtomicLong successEmitCount = new AtomicLong(0);
+ private final AtomicLong failedEmitCount = new AtomicLong(0);
+ private final AtomicLong interruptedEmitCount = new AtomicLong(0);
+ private final AtomicLong timedOutEmitCount = new AtomicLong(0);
@Override
public long getSuccessfulQueryCount()
{
- successEmitCount += 1;
- return successEmitCount;
+ return successEmitCount.get();
}
@Override
public long getFailedQueryCount()
{
- failedEmitCount += 2;
- return failedEmitCount;
+ return failedEmitCount.get();
}
@Override
public long getInterruptedQueryCount()
{
- interruptedEmitCount += 3;
- return interruptedEmitCount;
+ return interruptedEmitCount.get();
}
@Override
public long getTimedOutQueryCount()
{
- timedOutEmitCount += 4;
Review Comment:
Why was the increment +4? 😅
##########
sql/src/main/java/org/apache/druid/sql/http/SqlResource.java:
##########
@@ -80,14 +81,14 @@ public class SqlResource
public static final String SQL_HEADER_VALUE = "yes";
private static final Logger log = new Logger(SqlResource.class);
- public static final SqlResourceQueryMetricCounter QUERY_METRIC_COUNTER = new SqlResourceQueryMetricCounter();
Review Comment:
I suppose this was just a dummy instance that didn't even collect the stats?
##########
server/src/main/java/org/apache/druid/server/metrics/QueryCountStatsProvider.java:
##########
@@ -40,4 +40,24 @@ public interface QueryCountStatsProvider
* Returns the number of timed out queries during the emission period.
*/
long getTimedOutQueryCount();
+
+ /**
+ * Increments successful query count.
+ */
+ void incrementSuccess();
Review Comment:
```suggestion
void incrementSuccessful();
```
##########
server/src/main/java/org/apache/druid/server/metrics/BaseQueryCountResource.java:
##########
@@ -0,0 +1,85 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.server.metrics;
+
+import org.apache.druid.guice.LazySingleton;
+
+import java.util.concurrent.atomic.AtomicLong;
+
+/*
+ A thread-safe query count statistics tracker. Meant to be a shared between the various query resource handlers present
+ on an instance.
+*/
+@LazySingleton
+public class BaseQueryCountResource implements QueryCountStatsProvider
Review Comment:
Please rename this to something like `QueryCountStatsAccumulator` or `*Tracker` or `*ProviderImpl`.
(`Base*` implies that something extends this and `*Resource` is used for either REST resource classes or closeable resources).
##########
sql/src/main/java/org/apache/druid/sql/http/SqlHttpModule.java:
##########
@@ -35,6 +37,7 @@ public class SqlHttpModule implements Module
public void configure(Binder binder)
{
binder.bind(SqlResource.class).in(LazySingleton.class);
+ binder.bind(QueryCountStatsProvider.class).to(BaseQueryCountResource.class).in(LazySingleton.class);
Review Comment:
We could however make a `Names.named("sql")` annotation binding here and use the named binding
in `SqlResource` which would allow separate tracking of sql and native query counts.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] Support verbose SQL query status metrics (druid)
Posted by "jtuglu1 (via GitHub)" <gi...@apache.org>.
jtuglu1 commented on code in PR #18569:
URL: https://github.com/apache/druid/pull/18569#discussion_r2379757131
##########
sql/src/main/java/org/apache/druid/sql/http/SqlResource.java:
##########
@@ -97,7 +98,8 @@ public SqlResource(
final SqlEngineRegistry sqlEngineRegistry,
final SqlResourceQueryResultPusherFactory resultPusherFactory,
final DefaultQueryConfig defaultQueryConfig,
- final ServerConfig serverConfig
+ final ServerConfig serverConfig,
+ final QueryCountStatsProvider counter
Review Comment:
An alternative might be to keep a single class/instance, but introduce 1 layer of indirection to increment/fetch the metrics. This could support future combinations of (engine, queryFormat) where entries might look like (native, native) or (native, SQL) or (msq, SQL), etc. This way you don't get an explosion of unique metric names and/or annotated instances and that cardinality is instead pushed into the dimensions under a unified metric.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] Support verbose SQL query status metrics (druid)
Posted by "jtuglu1 (via GitHub)" <gi...@apache.org>.
jtuglu1 commented on code in PR #18569:
URL: https://github.com/apache/druid/pull/18569#discussion_r2380815896
##########
sql/src/main/java/org/apache/druid/sql/http/SqlResource.java:
##########
@@ -97,7 +98,8 @@ public SqlResource(
final SqlEngineRegistry sqlEngineRegistry,
final SqlResourceQueryResultPusherFactory resultPusherFactory,
final DefaultQueryConfig defaultQueryConfig,
- final ServerConfig serverConfig
+ final ServerConfig serverConfig,
+ final QueryCountStatsProvider counter
Review Comment:
100% agree. Given that the behavior before this was `QueryResource` emitting query count metrics only (SQL was just doing no-op, because these classes didn't share any common class), do you think we can scope this PR to refactoring both to share a single metrics counter instance? Then, a follow-up can be adding the query execution type tag?
I think this change should address the core problem the GH issue mentions: that SQL counts were not being tracked in `query/*/count` metrics. Then, adding other ways of aggregating over these metrics with an extra dimension can be added in a follow-up task.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] Support verbose SQL query status metrics (druid)
Posted by "kfaraz (via GitHub)" <gi...@apache.org>.
kfaraz commented on code in PR #18569:
URL: https://github.com/apache/druid/pull/18569#discussion_r2378618271
##########
sql/src/main/java/org/apache/druid/sql/http/SqlResource.java:
##########
@@ -97,7 +98,8 @@ public SqlResource(
final SqlEngineRegistry sqlEngineRegistry,
final SqlResourceQueryResultPusherFactory resultPusherFactory,
final DefaultQueryConfig defaultQueryConfig,
- final ServerConfig serverConfig
+ final ServerConfig serverConfig,
+ final QueryCountStatsProvider counter
Review Comment:
Why would it be confusing? You just need to use an annotated binding.
Druid code uses it in several places and this is a perfectly valid use case.
We keep several instances of `ServiceClient`, `DruidLeaderSelector`, `ServiceLocator` and some classes with MSQ as well.
Check out the annotations `@IndexingService`, `@Coordinator` which do this.
There are also several places which use the `@Named` annotation.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] Support verbose SQL query status metrics (druid)
Posted by "jtuglu1 (via GitHub)" <gi...@apache.org>.
jtuglu1 commented on code in PR #18569:
URL: https://github.com/apache/druid/pull/18569#discussion_r2377948561
##########
server/src/test/java/org/apache/druid/server/metrics/QueryCountStatsMonitorTest.java:
##########
@@ -55,37 +56,57 @@ public void setUp()
queryCountStatsProvider = new QueryCountStatsProvider()
{
- private long successEmitCount = 0;
- private long failedEmitCount = 0;
- private long interruptedEmitCount = 0;
- private long timedOutEmitCount = 0;
+ private final AtomicLong successEmitCount = new AtomicLong(0);
+ private final AtomicLong failedEmitCount = new AtomicLong(0);
+ private final AtomicLong interruptedEmitCount = new AtomicLong(0);
+ private final AtomicLong timedOutEmitCount = new AtomicLong(0);
@Override
public long getSuccessfulQueryCount()
{
- successEmitCount += 1;
- return successEmitCount;
+ return successEmitCount.get();
}
@Override
public long getFailedQueryCount()
{
- failedEmitCount += 2;
- return failedEmitCount;
+ return failedEmitCount.get();
}
@Override
public long getInterruptedQueryCount()
{
- interruptedEmitCount += 3;
- return interruptedEmitCount;
+ return interruptedEmitCount.get();
}
@Override
public long getTimedOutQueryCount()
{
- timedOutEmitCount += 4;
Review Comment:
No idea, I think to just add some variability so assertions were on some larger #s and so you don't need to do 4x calls.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] Support verbose SQL query status metrics (druid)
Posted by "kfaraz (via GitHub)" <gi...@apache.org>.
kfaraz commented on code in PR #18569:
URL: https://github.com/apache/druid/pull/18569#discussion_r2379802655
##########
sql/src/main/java/org/apache/druid/sql/http/SqlResource.java:
##########
@@ -97,7 +98,8 @@ public SqlResource(
final SqlEngineRegistry sqlEngineRegistry,
final SqlResourceQueryResultPusherFactory resultPusherFactory,
final DefaultQueryConfig defaultQueryConfig,
- final ServerConfig serverConfig
+ final ServerConfig serverConfig,
+ final QueryCountStatsProvider counter
Review Comment:
> An alternative might be to keep a single class/instance, but introduce 1 layer of indirection to increment/fetch the metrics. This could support future combinations of (engine, queryFormat) where entries might look like (native, native) or (native, SQL) or (msq, SQL), etc. This way you don't get an explosion of unique metric names and/or annotated instances and that cardinality is instead pushed into the dimensions under a unified metric.
Yes, it would indeed be nice to accept some kind of `QueryType` in all the methods of `QueryCountStatsProvider`.
e.g
```java
void incrementSuccessful(QueryType type);
int getSuccessfulQueryCount(QueryType type);
AND/OR
Map<QueryType, Integer> getSuccessfulQueryCountsByType();
enum QueryType {NATIVE_JSON, NATIVE_SQL, MSQ, DART}
```
The query type will be emitted as a dimension.
But this will require more work as you would need to change the signature of `QueryCountStatsProvider` methods
and pass the query type down to the relevant classes like `QueryResultPusher`, etc.
If you feel like making that change, that would work for me too.
Multiple bindings achieves the same result for now (since we are only emitting SQL and native query metrics) but with less code change.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] Support verbose SQL query status metrics (druid)
Posted by "kfaraz (via GitHub)" <gi...@apache.org>.
kfaraz commented on code in PR #18569:
URL: https://github.com/apache/druid/pull/18569#discussion_r2379796217
##########
sql/src/main/java/org/apache/druid/sql/http/SqlResource.java:
##########
@@ -97,7 +98,8 @@ public SqlResource(
final SqlEngineRegistry sqlEngineRegistry,
final SqlResourceQueryResultPusherFactory resultPusherFactory,
final DefaultQueryConfig defaultQueryConfig,
- final ServerConfig serverConfig
+ final ServerConfig serverConfig,
+ final QueryCountStatsProvider counter
Review Comment:
> This way you don't get an explosion of metrics and that cardinality is instead pushed into the dimensions under a unified metric.
Sure, as said earlier, same metric name with different dimension values works fine too.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] Support verbose SQL query status metrics (druid)
Posted by "jtuglu1 (via GitHub)" <gi...@apache.org>.
jtuglu1 commented on code in PR #18569:
URL: https://github.com/apache/druid/pull/18569#discussion_r2380815896
##########
sql/src/main/java/org/apache/druid/sql/http/SqlResource.java:
##########
@@ -97,7 +98,8 @@ public SqlResource(
final SqlEngineRegistry sqlEngineRegistry,
final SqlResourceQueryResultPusherFactory resultPusherFactory,
final DefaultQueryConfig defaultQueryConfig,
- final ServerConfig serverConfig
+ final ServerConfig serverConfig,
+ final QueryCountStatsProvider counter
Review Comment:
Given that the behavior before this was `QueryResource` emitting query count metrics only (SQL was no-op), do you think we can scope this PR to refactoring both to share a single `query/.*/count` instance? Then, a follow-up can be adding the query execution type tag?
I think this change should address the core issue that SQL counts were not being tracked, then adding other ways of aggregating over these metrics with an extra dimension can be added in a follow-up.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] Support verbose SQL query status metrics (druid)
Posted by "kfaraz (via GitHub)" <gi...@apache.org>.
kfaraz commented on code in PR #18569:
URL: https://github.com/apache/druid/pull/18569#discussion_r2379802655
##########
sql/src/main/java/org/apache/druid/sql/http/SqlResource.java:
##########
@@ -97,7 +98,8 @@ public SqlResource(
final SqlEngineRegistry sqlEngineRegistry,
final SqlResourceQueryResultPusherFactory resultPusherFactory,
final DefaultQueryConfig defaultQueryConfig,
- final ServerConfig serverConfig
+ final ServerConfig serverConfig,
+ final QueryCountStatsProvider counter
Review Comment:
> An alternative might be to keep a single class/instance, but introduce 1 layer of indirection to increment/fetch the metrics. This could support future combinations of (engine, queryFormat) where entries might look like (native, native) or (native, SQL) or (msq, SQL), etc. This way you don't get an explosion of unique metric names and/or annotated instances and that cardinality is instead pushed into the dimensions under a unified metric.
Yes, it would indeed be nice to accept some kind of `QueryType` in all the methods of `QueryCountStatsProvider`.
e.g
```java
void incrementSuccessful(QueryType type);
int getSuccessfulQueryCount(QueryType type);
enum QueryType {NATIVE_JSON, NATIVE_SQL, MSQ, DART}
```
The query type will be emitted as a dimension.
But this will require more work as you would need to change the signature of `QueryCountStatsProvider` methods
and pass the query type down to the relevant classes like `QueryResultPusher`, etc.
If you feel like making that change, that would work for me too.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] Support verbose SQL query status metrics (druid)
Posted by "kfaraz (via GitHub)" <gi...@apache.org>.
kfaraz commented on code in PR #18569:
URL: https://github.com/apache/druid/pull/18569#discussion_r2379796217
##########
sql/src/main/java/org/apache/druid/sql/http/SqlResource.java:
##########
@@ -97,7 +98,8 @@ public SqlResource(
final SqlEngineRegistry sqlEngineRegistry,
final SqlResourceQueryResultPusherFactory resultPusherFactory,
final DefaultQueryConfig defaultQueryConfig,
- final ServerConfig serverConfig
+ final ServerConfig serverConfig,
+ final QueryCountStatsProvider counter
Review Comment:
> This way you don't get an explosion of metrics and that cardinality is instead pushed into the dimensions under a unified metric.
Sure, as said earlier, same metric name with different dimensions works fine too.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] Support verbose SQL query status metrics (druid)
Posted by "jtuglu1 (via GitHub)" <gi...@apache.org>.
jtuglu1 closed pull request #18569: Support verbose SQL query status metrics
URL: https://github.com/apache/druid/pull/18569
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] Support verbose SQL query status metrics (druid)
Posted by "jtuglu1 (via GitHub)" <gi...@apache.org>.
jtuglu1 commented on code in PR #18569:
URL: https://github.com/apache/druid/pull/18569#discussion_r2377960958
##########
sql/src/main/java/org/apache/druid/sql/http/SqlResource.java:
##########
@@ -97,7 +98,8 @@ public SqlResource(
final SqlEngineRegistry sqlEngineRegistry,
final SqlResourceQueryResultPusherFactory resultPusherFactory,
final DefaultQueryConfig defaultQueryConfig,
- final ServerConfig serverConfig
+ final ServerConfig serverConfig,
+ final QueryCountStatsProvider counter
Review Comment:
I don't want to keep them separate (ideally). Why do you want to keep them separate?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] Support verbose SQL query status metrics (druid)
Posted by "jtuglu1 (via GitHub)" <gi...@apache.org>.
jtuglu1 commented on code in PR #18569:
URL: https://github.com/apache/druid/pull/18569#discussion_r2378367379
##########
sql/src/main/java/org/apache/druid/sql/http/SqlResource.java:
##########
@@ -97,7 +98,8 @@ public SqlResource(
final SqlEngineRegistry sqlEngineRegistry,
final SqlResourceQueryResultPusherFactory resultPusherFactory,
final DefaultQueryConfig defaultQueryConfig,
- final ServerConfig serverConfig
+ final ServerConfig serverConfig,
+ final QueryCountStatsProvider counter
Review Comment:
> Were SQL query count metrics ever emitted?
At this granularity (timeout, failure, interrupt, etc.) no.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] Support verbose SQL query status metrics (druid)
Posted by "jtuglu1 (via GitHub)" <gi...@apache.org>.
jtuglu1 commented on code in PR #18569:
URL: https://github.com/apache/druid/pull/18569#discussion_r2379757131
##########
sql/src/main/java/org/apache/druid/sql/http/SqlResource.java:
##########
@@ -97,7 +98,8 @@ public SqlResource(
final SqlEngineRegistry sqlEngineRegistry,
final SqlResourceQueryResultPusherFactory resultPusherFactory,
final DefaultQueryConfig defaultQueryConfig,
- final ServerConfig serverConfig
+ final ServerConfig serverConfig,
+ final QueryCountStatsProvider counter
Review Comment:
An alternative might be to keep a single class/instance, but introduce 1 layer of indirection to increment/fetch the metrics. This could support future combinations of (engine, queryFormat) where entries might look like (native, native) or (native, SQL) or (msq, SQL), etc. This way you don't get an explosion of unique metric names and/or annotated instances and that cardinality is instead pushed into the dimensions under a unified metric.
`QueryCountStatsMonitor` would know how to parse this map and create the relevant dimensions for the core set of metrics it emits.
I'd like to get to a point where people don't need to worry about a resource "picking up" a core metric because the code paths are separated. This might help with that.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] Support verbose SQL query status metrics (druid)
Posted by "jtuglu1 (via GitHub)" <gi...@apache.org>.
jtuglu1 commented on code in PR #18569:
URL: https://github.com/apache/druid/pull/18569#discussion_r2379757131
##########
sql/src/main/java/org/apache/druid/sql/http/SqlResource.java:
##########
@@ -97,7 +98,8 @@ public SqlResource(
final SqlEngineRegistry sqlEngineRegistry,
final SqlResourceQueryResultPusherFactory resultPusherFactory,
final DefaultQueryConfig defaultQueryConfig,
- final ServerConfig serverConfig
+ final ServerConfig serverConfig,
+ final QueryCountStatsProvider counter
Review Comment:
An alternative might be to keep a single class/instance, but introduce 1 layer of indirection to increment/fetch the metrics. This could support future combinations of (engine, queryFormat) where entries might look like (native, native) or (native, SQL) or (msq, SQL), etc. This way you don't get an explosion of metrics and that cardinality is instead pushed into the dimensions under a unified metric.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] Support verbose SQL query status metrics (druid)
Posted by "kfaraz (via GitHub)" <gi...@apache.org>.
kfaraz commented on code in PR #18569:
URL: https://github.com/apache/druid/pull/18569#discussion_r2379802655
##########
sql/src/main/java/org/apache/druid/sql/http/SqlResource.java:
##########
@@ -97,7 +98,8 @@ public SqlResource(
final SqlEngineRegistry sqlEngineRegistry,
final SqlResourceQueryResultPusherFactory resultPusherFactory,
final DefaultQueryConfig defaultQueryConfig,
- final ServerConfig serverConfig
+ final ServerConfig serverConfig,
+ final QueryCountStatsProvider counter
Review Comment:
> An alternative might be to keep a single class/instance, but introduce 1 layer of indirection to increment/fetch the metrics. This could support future combinations of (engine, queryFormat) where entries might look like (native, native) or (native, SQL) or (msq, SQL), etc. This way you don't get an explosion of unique metric names and/or annotated instances and that cardinality is instead pushed into the dimensions under a unified metric.
Yes, it would indeed be nice to accept some kind of `QueryType` in all the methods of `QueryCountStatsProvider`.
e.g
```java
void incrementSuccessful(QueryType type);
int getSuccessfulQueryCount(QueryType type);
OR
Map<QueryType, Integer> getSuccessfulQueryCountsByType();
enum QueryType {NATIVE_JSON, NATIVE_SQL, MSQ, DART}
```
The query type will be emitted as a dimension.
But this will require more work as you would need to change the signature of `QueryCountStatsProvider` methods
and pass the query type down to the relevant classes like `QueryResultPusher`, etc.
If you feel like making that change, that would work for me too.
Multiple bindings achieves the same result for now (since we are only emitting SQL and native query metrics) but with less code change.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] Support verbose SQL query status metrics (druid)
Posted by "jtuglu1 (via GitHub)" <gi...@apache.org>.
jtuglu1 commented on PR #18569:
URL: https://github.com/apache/druid/pull/18569#issuecomment-3414201286
Closing as not needed at the moment
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] Support verbose SQL query status metrics (druid)
Posted by "jtuglu1 (via GitHub)" <gi...@apache.org>.
jtuglu1 commented on code in PR #18569:
URL: https://github.com/apache/druid/pull/18569#discussion_r2379757131
##########
sql/src/main/java/org/apache/druid/sql/http/SqlResource.java:
##########
@@ -97,7 +98,8 @@ public SqlResource(
final SqlEngineRegistry sqlEngineRegistry,
final SqlResourceQueryResultPusherFactory resultPusherFactory,
final DefaultQueryConfig defaultQueryConfig,
- final ServerConfig serverConfig
+ final ServerConfig serverConfig,
+ final QueryCountStatsProvider counter
Review Comment:
An alternative might be to keep a single class/instance, but introduce 1 layer of indirection to increment/fetch the metrics. This could support future combinations of (engine, queryFormat) where entries might look like (native, native) or (native, SQL) or (msq, SQL), etc. This way you don't get an explosion of unique metric names and/or annotated instances and that cardinality is instead pushed into the dimensions under a unified metric.
`QueryCountStatsMonitor` would know how to parse this map and create the relevant dimensions for the core set of metrics it emits.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] Support verbose SQL query status metrics (druid)
Posted by "jtuglu1 (via GitHub)" <gi...@apache.org>.
jtuglu1 commented on code in PR #18569:
URL: https://github.com/apache/druid/pull/18569#discussion_r2380815896
##########
sql/src/main/java/org/apache/druid/sql/http/SqlResource.java:
##########
@@ -97,7 +98,8 @@ public SqlResource(
final SqlEngineRegistry sqlEngineRegistry,
final SqlResourceQueryResultPusherFactory resultPusherFactory,
final DefaultQueryConfig defaultQueryConfig,
- final ServerConfig serverConfig
+ final ServerConfig serverConfig,
+ final QueryCountStatsProvider counter
Review Comment:
Given that the behavior before this was `QueryResource` emitting query count metrics only (SQL was just doing no-op, because these classes didn't share any common class), do you think we can scope this PR to refactoring both to share a single metrics counter instance? Then, a follow-up can be adding the query execution type tag?
I think this change should address the core problem the GH issue mentions: that SQL counts were not being tracked in `query/*/count` metrics. Then, adding other ways of aggregating over these metrics with an extra dimension can be added in a follow-up task.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] Support verbose SQL query status metrics (druid)
Posted by "jtuglu1 (via GitHub)" <gi...@apache.org>.
jtuglu1 commented on code in PR #18569:
URL: https://github.com/apache/druid/pull/18569#discussion_r2379787670
##########
sql/src/main/java/org/apache/druid/sql/http/SqlResource.java:
##########
@@ -97,7 +98,8 @@ public SqlResource(
final SqlEngineRegistry sqlEngineRegistry,
final SqlResourceQueryResultPusherFactory resultPusherFactory,
final DefaultQueryConfig defaultQueryConfig,
- final ServerConfig serverConfig
+ final ServerConfig serverConfig,
+ final QueryCountStatsProvider counter
Review Comment:
@gianm any thoughts here?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] Support verbose SQL query status metrics (druid)
Posted by "kfaraz (via GitHub)" <gi...@apache.org>.
kfaraz commented on code in PR #18569:
URL: https://github.com/apache/druid/pull/18569#discussion_r2380828591
##########
sql/src/main/java/org/apache/druid/sql/http/SqlResource.java:
##########
@@ -97,7 +98,8 @@ public SqlResource(
final SqlEngineRegistry sqlEngineRegistry,
final SqlResourceQueryResultPusherFactory resultPusherFactory,
final DefaultQueryConfig defaultQueryConfig,
- final ServerConfig serverConfig
+ final ServerConfig serverConfig,
+ final QueryCountStatsProvider counter
Review Comment:
Agreed, we should address the core problem of publishing SQL metrics, but not at the cost of affecting existing metric values, with no way to tell whether a metric is originating from SQL or native.
The refactor can definitely be done in a follow up PR, but in that case, we should use the multiple bindings approach in this PR. The multiple bindings approach would have a very small footprint that can be easily removed later.
If we just a single instance with no dimensions in this PR,
I fear we might miss the Druid 35 code freeze (around Oct 7) and end up releasing a partial (possibly regressive) feature.
I would rather hold off on this PR until we have a complete solution, since we have never emitted SQL metrics anyway.
Let me know if you agree.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] Support verbose SQL query status metrics (druid)
Posted by "jtuglu1 (via GitHub)" <gi...@apache.org>.
jtuglu1 commented on code in PR #18569:
URL: https://github.com/apache/druid/pull/18569#discussion_r2378177246
##########
sql/src/main/java/org/apache/druid/sql/http/SqlResource.java:
##########
@@ -97,7 +98,8 @@ public SqlResource(
final SqlEngineRegistry sqlEngineRegistry,
final SqlResourceQueryResultPusherFactory resultPusherFactory,
final DefaultQueryConfig defaultQueryConfig,
- final ServerConfig serverConfig
+ final ServerConfig serverConfig,
+ final QueryCountStatsProvider counter
Review Comment:
That should be done with tags IMO and not differing metrics. What happened before this change is these 2 `*Resource` classes diverged to the point where there was gap in metrics between them. That success/failure can already be gauged from `sqlQuery/time`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] Support verbose SQL query status metrics (druid)
Posted by "kfaraz (via GitHub)" <gi...@apache.org>.
kfaraz commented on code in PR #18569:
URL: https://github.com/apache/druid/pull/18569#discussion_r2378265725
##########
sql/src/main/java/org/apache/druid/sql/http/SqlResource.java:
##########
@@ -97,7 +98,8 @@ public SqlResource(
final SqlEngineRegistry sqlEngineRegistry,
final SqlResourceQueryResultPusherFactory resultPusherFactory,
final DefaultQueryConfig defaultQueryConfig,
- final ServerConfig serverConfig
+ final ServerConfig serverConfig,
+ final QueryCountStatsProvider counter
Review Comment:
> That should be done with tags IMO and not differing metrics.
By tags, do you mean metric dimensions? That would be okay too [as suggested here](https://github.com/apache/druid/pull/18569#pullrequestreview-3265925315), as long as there is some way to distinguish between the two. But even different dimensions would still require you to collect the two stats separately via two different instances of the stats provider.
> What happened before this change is these 2 *Resource classes diverged to the point where there was gap in metrics between them.
Were SQL query count metrics ever emitted?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] Support verbose SQL query status metrics (druid)
Posted by "jtuglu1 (via GitHub)" <gi...@apache.org>.
jtuglu1 commented on code in PR #18569:
URL: https://github.com/apache/druid/pull/18569#discussion_r2378370739
##########
sql/src/main/java/org/apache/druid/sql/http/SqlResource.java:
##########
@@ -97,7 +98,8 @@ public SqlResource(
final SqlEngineRegistry sqlEngineRegistry,
final SqlResourceQueryResultPusherFactory resultPusherFactory,
final DefaultQueryConfig defaultQueryConfig,
- final ServerConfig serverConfig
+ final ServerConfig serverConfig,
+ final QueryCountStatsProvider counter
Review Comment:
I would prefer to just keep one shared instance. There hasn't been a use-case thus far where this was desired and adding more instances would make things more confusing IMO.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] Support verbose SQL query status metrics (druid)
Posted by "jtuglu1 (via GitHub)" <gi...@apache.org>.
jtuglu1 commented on code in PR #18569:
URL: https://github.com/apache/druid/pull/18569#discussion_r2378370739
##########
sql/src/main/java/org/apache/druid/sql/http/SqlResource.java:
##########
@@ -97,7 +98,8 @@ public SqlResource(
final SqlEngineRegistry sqlEngineRegistry,
final SqlResourceQueryResultPusherFactory resultPusherFactory,
final DefaultQueryConfig defaultQueryConfig,
- final ServerConfig serverConfig
+ final ServerConfig serverConfig,
+ final QueryCountStatsProvider counter
Review Comment:
I would prefer to just keep one big class. There hasn't been a use-case thus far where this was desired and adding more instances would make things more confusing IMO.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] Support verbose SQL query status metrics (druid)
Posted by "jtuglu1 (via GitHub)" <gi...@apache.org>.
jtuglu1 commented on code in PR #18569:
URL: https://github.com/apache/druid/pull/18569#discussion_r2378437407
##########
server/src/test/java/org/apache/druid/server/metrics/QueryCountStatsMonitorTest.java:
##########
@@ -55,37 +56,57 @@ public void setUp()
queryCountStatsProvider = new QueryCountStatsProvider()
Review Comment:
Sure, was trying to keep diff minimal.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] Support verbose SQL query status metrics (druid)
Posted by "kfaraz (via GitHub)" <gi...@apache.org>.
kfaraz commented on code in PR #18569:
URL: https://github.com/apache/druid/pull/18569#discussion_r2379802655
##########
sql/src/main/java/org/apache/druid/sql/http/SqlResource.java:
##########
@@ -97,7 +98,8 @@ public SqlResource(
final SqlEngineRegistry sqlEngineRegistry,
final SqlResourceQueryResultPusherFactory resultPusherFactory,
final DefaultQueryConfig defaultQueryConfig,
- final ServerConfig serverConfig
+ final ServerConfig serverConfig,
+ final QueryCountStatsProvider counter
Review Comment:
> An alternative might be to keep a single class/instance, but introduce 1 layer of indirection to increment/fetch the metrics. This could support future combinations of (engine, queryFormat) where entries might look like (native, native) or (native, SQL) or (msq, SQL), etc. This way you don't get an explosion of unique metric names and/or annotated instances and that cardinality is instead pushed into the dimensions under a unified metric.
Yes, it would indeed be nice to accept some kind of `QueryType` in all the methods of `QueryCountStatsProvider`.
e.g
```java
void incrementSuccessful(QueryType type);
int getSuccessfulQueryCount(QueryType type);
enum QueryType {NATIVE_JSON, NATIVE_SQL, MSQ, DART}
```
The query type will be emitted as a dimension.
But this will require more work as you would need to change the signature of `QueryCountStatsProvider` methods
and pass the query type down to the relevant classes like `QueryResultPusher`, etc.
If you feel like making that change, that would work for me too.
Multiple bindings achieves the same result for now (since we are only emitting SQL and native query metrics) but with less code change.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] Support verbose SQL query status metrics (druid)
Posted by "kfaraz (via GitHub)" <gi...@apache.org>.
kfaraz commented on code in PR #18569:
URL: https://github.com/apache/druid/pull/18569#discussion_r2378092214
##########
sql/src/main/java/org/apache/druid/sql/http/SqlResource.java:
##########
@@ -97,7 +98,8 @@ public SqlResource(
final SqlEngineRegistry sqlEngineRegistry,
final SqlResourceQueryResultPusherFactory resultPusherFactory,
final DefaultQueryConfig defaultQueryConfig,
- final ServerConfig serverConfig
+ final ServerConfig serverConfig,
+ final QueryCountStatsProvider counter
Review Comment:
There could be some existing assumptions regarding the query counts in downstream applications. Also, it might be useful for future use cases to be able to distinguish between the two.
Say an application might want to determine number of failing Sql queries vs failing native queries.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org