You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pinot.apache.org by "gortiz (via GitHub)" <gi...@apache.org> on 2025/04/22 06:44:31 UTC

[PR] Include stats on cancel [pinot]

gortiz opened a new pull request, #15609:
URL: https://github.com/apache/pinot/pull/15609

   This PR adds the ability to include stats on timeouts. Although we are ready to return stats on errors (since https://github.com/apache/pinot/pull/15245), the tricky part on timeouts is that different threads (on different jvms) use the same deadline calculated with `now() + timeout`. To get updated stats, we would need to do implement at least one of the following:
   
   1. Fail as close to the leaves as possible
   2. Ask for stats on failure
   3. Use an alternative mechanism
   
   In this PR we try to do the latest. What I propose (and I'm very open to discussing), is the following:
   - On servers, when an opchain is scheduled, we store the stage root operator in a Guava cache indexed by opchain id.
     - This cache has a maximum size that could be configured. The size is based on the number of operators.
     - When the opchain finishes:
       - If finished successfully, the entry is removed from the cache
       - In other cases, it is kept
     - When a query is cancelled, we look for the opchains in the cache and retrieve their statistics. These stats are now returned in the cancel response.
   - On brokers, when a query fails, a cancel signal is sent to all servers. This is what we did before, but now we wait a bit for the stats in the responses. These stats are sent to the user.
   
   This system has several flaws, race conditions, may return partial data (some opchains may have been removed from the cache) and requires to keep some extra memory (bounded). 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


Re: [PR] Include stats on cancel [pinot]

Posted by "gortiz (via GitHub)" <gi...@apache.org>.
gortiz commented on code in PR #15609:
URL: https://github.com/apache/pinot/pull/15609#discussion_r2060192055


##########
pinot-query-runtime/src/main/java/org/apache/pinot/query/service/dispatch/QueryDispatcher.java:
##########
@@ -146,18 +156,39 @@ public QueryResult submitAndReduce(RequestContext context, DispatchableSubPlan d
     Set<QueryServerInstance> servers = new HashSet<>();
     try {
       submit(requestId, dispatchableSubPlan, timeoutMs, servers, queryOptions);
-      try {
-        return runReducer(requestId, dispatchableSubPlan, timeoutMs, queryOptions, _mailboxService);
-      } finally {
-        if (isQueryCancellationEnabled()) {
-          _serversByQuery.remove(requestId);
-        }
-      }
+      return runReducer(requestId, dispatchableSubPlan, timeoutMs, queryOptions, _mailboxService);
+    } catch (Exception ex) {

Review Comment:
   Why? All throwables that are not exceptions (basically Errors) are going to be cancelled without trying to recover. Normally I wouldn't catch Throwable, but it was already caught in the original code.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


Re: [PR] Include stats on cancel [pinot]

Posted by "gortiz (via GitHub)" <gi...@apache.org>.
gortiz commented on code in PR #15609:
URL: https://github.com/apache/pinot/pull/15609#discussion_r2068519625


##########
pinot-query-runtime/src/main/java/org/apache/pinot/query/runtime/executor/OpChainSchedulerService.java:
##########
@@ -18,35 +18,65 @@
  */
 package org.apache.pinot.query.runtime.executor;
 
+import com.google.common.cache.Cache;
+import com.google.common.cache.CacheBuilder;
+import java.util.ArrayList;
 import java.util.Iterator;
 import java.util.Map;
 import java.util.concurrent.ConcurrentHashMap;
 import java.util.concurrent.ExecutorService;
 import java.util.concurrent.Future;
+import java.util.concurrent.TimeUnit;
+import java.util.stream.Collectors;
 import org.apache.pinot.core.util.trace.TraceRunnable;
 import org.apache.pinot.query.runtime.blocks.ErrorMseBlock;
 import org.apache.pinot.query.runtime.blocks.MseBlock;
+import org.apache.pinot.query.runtime.operator.MultiStageOperator;
 import org.apache.pinot.query.runtime.operator.OpChain;
 import org.apache.pinot.query.runtime.operator.OpChainId;
 import org.apache.pinot.query.runtime.plan.MultiStageQueryStats;
 import org.apache.pinot.spi.accounting.ThreadExecutionContext;
+import org.apache.pinot.spi.env.PinotConfiguration;
 import org.apache.pinot.spi.trace.Tracing;
+import org.apache.pinot.spi.utils.CommonConstants.MultiStageQueryRunner;
 import org.slf4j.Logger;
 import org.slf4j.LoggerFactory;
 
-
 public class OpChainSchedulerService {
   private static final Logger LOGGER = LoggerFactory.getLogger(OpChainSchedulerService.class);
 
   private final ExecutorService _executorService;
-  private final ConcurrentHashMap<OpChainId, Future<?>> _submittedOpChainMap;
+  private final ConcurrentHashMap<OpChainId, Future<?>> _submittedOpChainMap = new ConcurrentHashMap<>();
+  private final Cache<OpChainId, MultiStageOperator> _opChainCache;
+
+
+  public OpChainSchedulerService(ExecutorService executorService, PinotConfiguration config) {

Review Comment:
   Yes, nice catch



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


Re: [PR] Include stats on cancel [pinot]

Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #15609:
URL: https://github.com/apache/pinot/pull/15609#issuecomment-2820372082

   ## [Codecov](https://app.codecov.io/gh/apache/pinot/pull/15609?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
   Attention: Patch coverage is `0.69930%` with `142 lines` in your changes missing coverage. Please review.
   > Project coverage is 33.51%. Comparing base [(`1fe72ae`)](https://app.codecov.io/gh/apache/pinot/commit/1fe72ae1d3ecb0dfcd9df39edf26cd95abff58fd?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) to head [(`cba5e6c`)](https://app.codecov.io/gh/apache/pinot/commit/cba5e6cde72fb8cdb03e58086ed523cf778a9891?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
   > Report is 25 commits behind head on master.
   
   | [Files with missing lines](https://app.codecov.io/gh/apache/pinot/pull/15609?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Patch % | Lines |
   |---|---|---|
   | [.../pinot/query/service/dispatch/QueryDispatcher.java](https://app.codecov.io/gh/apache/pinot/pull/15609?src=pr&el=tree&filepath=pinot-query-runtime%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fpinot%2Fquery%2Fservice%2Fdispatch%2FQueryDispatcher.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-cGlub3QtcXVlcnktcnVudGltZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvcXVlcnkvc2VydmljZS9kaXNwYXRjaC9RdWVyeURpc3BhdGNoZXIuamF2YQ==) | 1.38% | [71 Missing :warning: ](https://app.codecov.io/gh/apache/pinot/pull/15609?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
   | [...uery/runtime/executor/OpChainSchedulerService.java](https://app.codecov.io/gh/apache/pinot/pull/15609?src=pr&el=tree&filepath=pinot-query-runtime%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fpinot%2Fquery%2Fruntime%2Fexecutor%2FOpChainSchedulerService.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-cGlub3QtcXVlcnktcnVudGltZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvcXVlcnkvcnVudGltZS9leGVjdXRvci9PcENoYWluU2NoZWR1bGVyU2VydmljZS5qYXZh) | 0.00% | [30 Missing :warning: ](https://app.codecov.io/gh/apache/pinot/pull/15609?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
   | [...requesthandler/MultiStageBrokerRequestHandler.java](https://app.codecov.io/gh/apache/pinot/pull/15609?src=pr&el=tree&filepath=pinot-broker%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fpinot%2Fbroker%2Frequesthandler%2FMultiStageBrokerRequestHandler.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-cGlub3QtYnJva2VyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9icm9rZXIvcmVxdWVzdGhhbmRsZXIvTXVsdGlTdGFnZUJyb2tlclJlcXVlc3RIYW5kbGVyLmphdmE=) | 0.00% | [14 Missing :warning: ](https://app.codecov.io/gh/apache/pinot/pull/15609?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
   | [...apache/pinot/query/service/server/QueryServer.java](https://app.codecov.io/gh/apache/pinot/pull/15609?src=pr&el=tree&filepath=pinot-query-runtime%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fpinot%2Fquery%2Fservice%2Fserver%2FQueryServer.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-cGlub3QtcXVlcnktcnVudGltZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvcXVlcnkvc2VydmljZS9zZXJ2ZXIvUXVlcnlTZXJ2ZXIuamF2YQ==) | 0.00% | [12 Missing :warning: ](https://app.codecov.io/gh/apache/pinot/pull/15609?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
   | [...pinot/query/runtime/plan/MultiStageQueryStats.java](https://app.codecov.io/gh/apache/pinot/pull/15609?src=pr&el=tree&filepath=pinot-query-runtime%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fpinot%2Fquery%2Fruntime%2Fplan%2FMultiStageQueryStats.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-cGlub3QtcXVlcnktcnVudGltZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvcXVlcnkvcnVudGltZS9wbGFuL011bHRpU3RhZ2VRdWVyeVN0YXRzLmphdmE=) | 0.00% | [11 Missing :warning: ](https://app.codecov.io/gh/apache/pinot/pull/15609?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
   | [...e/pinot/query/service/dispatch/DispatchClient.java](https://app.codecov.io/gh/apache/pinot/pull/15609?src=pr&el=tree&filepath=pinot-query-runtime%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fpinot%2Fquery%2Fservice%2Fdispatch%2FDispatchClient.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-cGlub3QtcXVlcnktcnVudGltZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvcXVlcnkvc2VydmljZS9kaXNwYXRjaC9EaXNwYXRjaENsaWVudC5qYXZh) | 0.00% | [2 Missing :warning: ](https://app.codecov.io/gh/apache/pinot/pull/15609?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
   | [...inot/query/runtime/MultiStageStatsTreeBuilder.java](https://app.codecov.io/gh/apache/pinot/pull/15609?src=pr&el=tree&filepath=pinot-query-runtime%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fpinot%2Fquery%2Fruntime%2FMultiStageStatsTreeBuilder.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-cGlub3QtcXVlcnktcnVudGltZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvcXVlcnkvcnVudGltZS9NdWx0aVN0YWdlU3RhdHNUcmVlQnVpbGRlci5qYXZh) | 0.00% | [1 Missing :warning: ](https://app.codecov.io/gh/apache/pinot/pull/15609?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
   | [...va/org/apache/pinot/query/runtime/QueryRunner.java](https://app.codecov.io/gh/apache/pinot/pull/15609?src=pr&el=tree&filepath=pinot-query-runtime%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fpinot%2Fquery%2Fruntime%2FQueryRunner.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-cGlub3QtcXVlcnktcnVudGltZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvcXVlcnkvcnVudGltZS9RdWVyeVJ1bm5lci5qYXZh) | 0.00% | [1 Missing :warning: ](https://app.codecov.io/gh/apache/pinot/pull/15609?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
   
   > :exclamation:  There is a different number of reports uploaded between BASE (1fe72ae) and HEAD (cba5e6c). Click for more details.
   > 
   > <details><summary>HEAD has 62 uploads less than BASE</summary>
   >
   >| Flag | BASE (1fe72ae) | HEAD (cba5e6c) |
   >|------|------|------|
   >|java-21|9|1|
   >|unittests|5|1|
   >|unittests1|3|0|
   >|skip-bytebuffers-true|3|1|
   >|temurin|14|1|
   >|unittests2|2|1|
   >|skip-bytebuffers-false|8|0|
   >|java-11|5|0|
   >|integration|9|0|
   >|integration2|3|0|
   >|integration1|3|0|
   >|custom-integration1|3|0|
   ></details>
   
   <details><summary>Additional details and impacted files</summary>
   
   
   ```diff
   @@              Coverage Diff              @@
   ##             master   #15609       +/-   ##
   =============================================
   - Coverage     62.85%   33.51%   -29.34%     
   + Complexity     1383      692      -691     
   =============================================
     Files          2863     2864        +1     
     Lines        162450   162820      +370     
     Branches      24881    24922       +41     
   =============================================
   - Hits         102101    54566    -47535     
   - Misses        52670   103892    +51222     
   + Partials       7679     4362     -3317     
   ```
   
   | [Flag](https://app.codecov.io/gh/apache/pinot/pull/15609/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
   |---|---|---|
   | [custom-integration1](https://app.codecov.io/gh/apache/pinot/pull/15609/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
   | [integration](https://app.codecov.io/gh/apache/pinot/pull/15609/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
   | [integration1](https://app.codecov.io/gh/apache/pinot/pull/15609/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
   | [integration2](https://app.codecov.io/gh/apache/pinot/pull/15609/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
   | [java-11](https://app.codecov.io/gh/apache/pinot/pull/15609/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
   | [java-21](https://app.codecov.io/gh/apache/pinot/pull/15609/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `33.51% <0.69%> (-29.32%)` | :arrow_down: |
   | [skip-bytebuffers-false](https://app.codecov.io/gh/apache/pinot/pull/15609/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
   | [skip-bytebuffers-true](https://app.codecov.io/gh/apache/pinot/pull/15609/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `33.51% <0.69%> (-22.26%)` | :arrow_down: |
   | [temurin](https://app.codecov.io/gh/apache/pinot/pull/15609/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `33.51% <0.69%> (-29.34%)` | :arrow_down: |
   | [unittests](https://app.codecov.io/gh/apache/pinot/pull/15609/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `33.51% <0.69%> (-29.34%)` | :arrow_down: |
   | [unittests1](https://app.codecov.io/gh/apache/pinot/pull/15609/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
   | [unittests2](https://app.codecov.io/gh/apache/pinot/pull/15609/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `33.51% <0.69%> (ø)` | |
   
   Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#carryforward-flags-in-the-pull-request-comment) to find out more.
   
   </details>
   
   [:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/apache/pinot/pull/15609?dropdown=coverage&src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).   
   :loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
   
   <details><summary> :rocket: New features to boost your workflow: </summary>
   
   - :package: [JS Bundle Analysis](https://docs.codecov.com/docs/javascript-bundle-analysis): Save yourself from yourself by tracking and limiting bundle sizes in JS merges.
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


Re: [PR] Include stats on cancel [pinot]

Posted by "gortiz (via GitHub)" <gi...@apache.org>.
gortiz commented on code in PR #15609:
URL: https://github.com/apache/pinot/pull/15609#discussion_r2055487926


##########
pinot-broker/src/main/java/org/apache/pinot/broker/requesthandler/MultiStageBrokerRequestHandler.java:
##########
@@ -443,11 +443,27 @@ private BrokerResponse query(QueryEnvironment.CompiledQuery query, long requestI
         Tracing.ThreadAccountantOps.clear();
         onQueryFinish(requestId);
       }
-      long executionEndTimeNs = System.nanoTime();
-      updatePhaseTimingForTables(tableNames, BrokerQueryPhase.QUERY_EXECUTION,
-          executionEndTimeNs - executionStartTimeNs);
 
       BrokerResponseNativeV2 brokerResponse = new BrokerResponseNativeV2();
+
+      QueryProcessingException processingException = queryResults.getProcessingException();
+      if (processingException != null) {
+        brokerResponse.addException(processingException);
+        QueryErrorCode errorCode = QueryErrorCode.fromErrorCode(processingException.getErrorCode());
+        if (errorCode == QueryErrorCode.EXECUTION_TIMEOUT) {
+          for (String table : tableNames) {
+            _brokerMetrics.addMeteredTableValue(table, BrokerMeter.BROKER_RESPONSES_WITH_TIMEOUTS, 1);
+          }
+          LOGGER.warn("Timed out executing request {}: {}", requestId, query);
+        }
+        requestContext.setErrorCode(errorCode);
+      } else {
+        brokerResponse.setResultTable(queryResults.getResultTable());
+        long executionEndTimeNs = System.nanoTime();
+        updatePhaseTimingForTables(tableNames, BrokerQueryPhase.QUERY_EXECUTION,

Review Comment:
   I'm not sure about that. 
   
   In the older code, `submitAndReduce` in IICU could either throw an error or finish successfully. In the first case, we catch the exception and return a BrokerResponseNative without calling `updaterPhaseTimingForTables`. So we only call `updatePhaseTimingForTables` when queries finish successfully



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


Re: [PR] Include stats on cancel [pinot]

Posted by "albertobastos (via GitHub)" <gi...@apache.org>.
albertobastos commented on code in PR #15609:
URL: https://github.com/apache/pinot/pull/15609#discussion_r2057947355


##########
pinot-query-runtime/src/main/java/org/apache/pinot/query/service/dispatch/QueryDispatcher.java:
##########
@@ -146,18 +156,39 @@ public QueryResult submitAndReduce(RequestContext context, DispatchableSubPlan d
     Set<QueryServerInstance> servers = new HashSet<>();
     try {
       submit(requestId, dispatchableSubPlan, timeoutMs, servers, queryOptions);
-      try {
-        return runReducer(requestId, dispatchableSubPlan, timeoutMs, queryOptions, _mailboxService);
-      } finally {
-        if (isQueryCancellationEnabled()) {
-          _serversByQuery.remove(requestId);
-        }
-      }
+      return runReducer(requestId, dispatchableSubPlan, timeoutMs, queryOptions, _mailboxService);
+    } catch (Exception ex) {

Review Comment:
   This `try-catch(Exception)-catch(Throwable)` chain feels a bit weird, I would at least add some comment explaining who is catching what.



##########
pinot-common/src/main/proto/worker.proto:
##########
@@ -39,6 +39,7 @@ message CancelRequest {
 
 message CancelResponse {
   // intentionally left empty

Review Comment:
   This comment no longer applies.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


Re: [PR] Include stats on cancel [pinot]

Posted by "gortiz (via GitHub)" <gi...@apache.org>.
gortiz commented on code in PR #15609:
URL: https://github.com/apache/pinot/pull/15609#discussion_r2068522578


##########
pinot-query-runtime/src/main/java/org/apache/pinot/query/runtime/executor/OpChainSchedulerService.java:
##########
@@ -18,35 +18,65 @@
  */
 package org.apache.pinot.query.runtime.executor;
 
+import com.google.common.cache.Cache;
+import com.google.common.cache.CacheBuilder;
+import java.util.ArrayList;
 import java.util.Iterator;
 import java.util.Map;
 import java.util.concurrent.ConcurrentHashMap;
 import java.util.concurrent.ExecutorService;
 import java.util.concurrent.Future;
+import java.util.concurrent.TimeUnit;
+import java.util.stream.Collectors;
 import org.apache.pinot.core.util.trace.TraceRunnable;
 import org.apache.pinot.query.runtime.blocks.ErrorMseBlock;
 import org.apache.pinot.query.runtime.blocks.MseBlock;
+import org.apache.pinot.query.runtime.operator.MultiStageOperator;
 import org.apache.pinot.query.runtime.operator.OpChain;
 import org.apache.pinot.query.runtime.operator.OpChainId;
 import org.apache.pinot.query.runtime.plan.MultiStageQueryStats;
 import org.apache.pinot.spi.accounting.ThreadExecutionContext;
+import org.apache.pinot.spi.env.PinotConfiguration;
 import org.apache.pinot.spi.trace.Tracing;
+import org.apache.pinot.spi.utils.CommonConstants.MultiStageQueryRunner;
 import org.slf4j.Logger;
 import org.slf4j.LoggerFactory;
 
-
 public class OpChainSchedulerService {
   private static final Logger LOGGER = LoggerFactory.getLogger(OpChainSchedulerService.class);
 
   private final ExecutorService _executorService;
-  private final ConcurrentHashMap<OpChainId, Future<?>> _submittedOpChainMap;
+  private final ConcurrentHashMap<OpChainId, Future<?>> _submittedOpChainMap = new ConcurrentHashMap<>();
+  private final Cache<OpChainId, MultiStageOperator> _opChainCache;
+
+
+  public OpChainSchedulerService(ExecutorService executorService, PinotConfiguration config) {
+    this(
+        executorService,
+        config.getProperty(MultiStageQueryRunner.KEY_OF_OP_STATS_CACHE_MAX,
+            MultiStageQueryRunner.DEFAULT_OF_OP_STATS_CACHE_MAX),
+        config.getProperty(MultiStageQueryRunner.KEY_OF_OP_STATS_CACHE_EXPIRE_MS,
+            MultiStageQueryRunner.DEFAULT_OF_OP_STATS_CACHE_EXPIRE_MS)
+    );
+  }
 
   public OpChainSchedulerService(ExecutorService executorService) {
+    this(executorService, MultiStageQueryRunner.DEFAULT_OF_OP_STATS_CACHE_MAX,
+        MultiStageQueryRunner.DEFAULT_OF_OP_STATS_CACHE_EXPIRE_MS);
+  }
+
+  public OpChainSchedulerService(ExecutorService executorService, int maxWeight, long expireAfterWriteMs) {
     _executorService = executorService;
-    _submittedOpChainMap = new ConcurrentHashMap<>();
+    _opChainCache = CacheBuilder.newBuilder()
+        .weigher((OpChainId key, MultiStageOperator value) -> countOperators(value))

Review Comment:
   The more operators in the stage, the more memory this cache keeps alive. Probably it is similar to using weigh = 1, because stages are not huge (AFAIK only UNION is not bound) but that may change in the future and it shouldn't be very expensive to count the number of opeartors



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


Re: [PR] Include stats on cancel [pinot]

Posted by "gortiz (via GitHub)" <gi...@apache.org>.
gortiz commented on code in PR #15609:
URL: https://github.com/apache/pinot/pull/15609#discussion_r2073370059


##########
pinot-spi/src/main/java/org/apache/pinot/spi/utils/CommonConstants.java:
##########
@@ -1577,6 +1576,18 @@ public static class PlanVersions {
     public static final String KEY_OF_MULTISTAGE_EXPLAIN_INCLUDE_SEGMENT_PLAN =
         "pinot.query.multistage.explain.include.segment.plan";
     public static final boolean DEFAULT_OF_MULTISTAGE_EXPLAIN_INCLUDE_SEGMENT_PLAN = false;
+
+    /// Max number of rows operators stored in the op stats cache.
+    /// Although the cache stores stages, each entry has a weight equal to the number of operators in the stage.
+    public static final String KEY_OF_OP_STATS_CACHE_MAX = "pinot.server.query.op.stats.cache.max";
+    public static final int DEFAULT_OF_OP_STATS_CACHE_MAX = 1000;
+
+    /// Max time to keep the op stats in the cache.
+    public static final String KEY_OF_OP_STATS_CACHE_EXPIRE_MS = "pinot.server.query.op.stats.cache.secs";
+    public static final int DEFAULT_OF_OP_STATS_CACHE_EXPIRE_MS = 10 * 60 * 1000;

Review Comment:
   Fixed



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


Re: [PR] Include stats on cancel [pinot]

Posted by "gortiz (via GitHub)" <gi...@apache.org>.
gortiz commented on code in PR #15609:
URL: https://github.com/apache/pinot/pull/15609#discussion_r2055477662


##########
pinot-broker/src/main/java/org/apache/pinot/broker/requesthandler/MultiStageBrokerRequestHandler.java:
##########
@@ -443,11 +443,27 @@ private BrokerResponse query(QueryEnvironment.CompiledQuery query, long requestI
         Tracing.ThreadAccountantOps.clear();
         onQueryFinish(requestId);
       }
-      long executionEndTimeNs = System.nanoTime();
-      updatePhaseTimingForTables(tableNames, BrokerQueryPhase.QUERY_EXECUTION,
-          executionEndTimeNs - executionStartTimeNs);
 
       BrokerResponseNativeV2 brokerResponse = new BrokerResponseNativeV2();
+
+      QueryProcessingException processingException = queryResults.getProcessingException();
+      if (processingException != null) {
+        brokerResponse.addException(processingException);
+        QueryErrorCode errorCode = QueryErrorCode.fromErrorCode(processingException.getErrorCode());
+        if (errorCode == QueryErrorCode.EXECUTION_TIMEOUT) {
+          for (String table : tableNames) {
+            _brokerMetrics.addMeteredTableValue(table, BrokerMeter.BROKER_RESPONSES_WITH_TIMEOUTS, 1);
+          }
+          LOGGER.warn("Timed out executing request {}: {}", requestId, query);

Review Comment:
   This replicates the logic already present in lines 428 and 433. Other error cases are either:
   
   1. Exceptions thrown in the same way as before, so they won't reach line 449.
   2. Errors caught internally and returned as an erroneous BrokerResponseNative, in which case we don't need to do anything else
   
   TL;DR: I've added these lines here to keep the same behavior when timeout is caught inside _queryDispatcher.submitAndReduce



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


Re: [PR] Include stats on cancel [pinot]

Posted by "gortiz (via GitHub)" <gi...@apache.org>.
gortiz commented on code in PR #15609:
URL: https://github.com/apache/pinot/pull/15609#discussion_r2068633803


##########
pinot-query-runtime/src/main/java/org/apache/pinot/query/service/dispatch/QueryDispatcher.java:
##########
@@ -146,18 +162,40 @@ public QueryResult submitAndReduce(RequestContext context, DispatchableSubPlan d
     Set<QueryServerInstance> servers = new HashSet<>();
     try {
       submit(requestId, dispatchableSubPlan, timeoutMs, servers, queryOptions);
-      try {
-        return runReducer(requestId, dispatchableSubPlan, timeoutMs, queryOptions, _mailboxService);
-      } finally {
-        if (isQueryCancellationEnabled()) {
-          _serversByQuery.remove(requestId);
-        }
-      }
+      return runReducer(requestId, dispatchableSubPlan, timeoutMs, queryOptions, _mailboxService);
+    } catch (Exception ex) {
+      return tryRecover(context.getRequestId(), servers, ex);
     } catch (Throwable e) {
       // TODO: Consider always cancel when it returns (early terminate)
-      cancel(requestId, servers);
+      cancel(requestId);
       throw e;
+    } finally {
+      if (isQueryCancellationEnabled()) {
+        _serversByQuery.remove(requestId);
+      }
+    }
+  }
+
+  private QueryResult tryRecover(long requestId, Set<QueryServerInstance> servers, Exception ex)
+      throws Exception {
+    if (servers.isEmpty()) {
+      throw ex;
+    }
+    QueryErrorCode errorCode;
+    if (ex instanceof TimeoutException) {
+      errorCode = QueryErrorCode.EXECUTION_TIMEOUT;
+    } else if (ex instanceof QueryException) {
+      errorCode = ((QueryException) ex).getErrorCode();

Review Comment:
   Not right now, but it may change in the future. I think it is a better design to be prepared for that (now allowed) case where runReducer or submit throw a QueryException. In fact, now the strange case would be to throw something that is not a QueryException



##########
pinot-query-runtime/src/main/java/org/apache/pinot/query/service/dispatch/QueryDispatcher.java:
##########
@@ -146,18 +162,40 @@ public QueryResult submitAndReduce(RequestContext context, DispatchableSubPlan d
     Set<QueryServerInstance> servers = new HashSet<>();
     try {
       submit(requestId, dispatchableSubPlan, timeoutMs, servers, queryOptions);
-      try {
-        return runReducer(requestId, dispatchableSubPlan, timeoutMs, queryOptions, _mailboxService);
-      } finally {
-        if (isQueryCancellationEnabled()) {
-          _serversByQuery.remove(requestId);
-        }
-      }
+      return runReducer(requestId, dispatchableSubPlan, timeoutMs, queryOptions, _mailboxService);
+    } catch (Exception ex) {
+      return tryRecover(context.getRequestId(), servers, ex);
     } catch (Throwable e) {
       // TODO: Consider always cancel when it returns (early terminate)
-      cancel(requestId, servers);
+      cancel(requestId);
       throw e;
+    } finally {
+      if (isQueryCancellationEnabled()) {
+        _serversByQuery.remove(requestId);
+      }
+    }
+  }
+
+  private QueryResult tryRecover(long requestId, Set<QueryServerInstance> servers, Exception ex)
+      throws Exception {
+    if (servers.isEmpty()) {
+      throw ex;
+    }
+    QueryErrorCode errorCode;
+    if (ex instanceof TimeoutException) {
+      errorCode = QueryErrorCode.EXECUTION_TIMEOUT;
+    } else if (ex instanceof QueryException) {
+      errorCode = ((QueryException) ex).getErrorCode();

Review Comment:
   Not right now, but it may change in the future. I think it is a better design to be prepared for that (now allowed) case where runReducer or submit throwd a QueryException. In fact, now the strange case would be to throw something that is not a QueryException



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


Re: [PR] Include stats on cancel [pinot]

Posted by "albertobastos (via GitHub)" <gi...@apache.org>.
albertobastos commented on code in PR #15609:
URL: https://github.com/apache/pinot/pull/15609#discussion_r2063716851


##########
pinot-query-runtime/src/main/java/org/apache/pinot/query/service/dispatch/QueryDispatcher.java:
##########
@@ -146,18 +156,39 @@ public QueryResult submitAndReduce(RequestContext context, DispatchableSubPlan d
     Set<QueryServerInstance> servers = new HashSet<>();
     try {
       submit(requestId, dispatchableSubPlan, timeoutMs, servers, queryOptions);
-      try {
-        return runReducer(requestId, dispatchableSubPlan, timeoutMs, queryOptions, _mailboxService);
-      } finally {
-        if (isQueryCancellationEnabled()) {
-          _serversByQuery.remove(requestId);
-        }
-      }
+      return runReducer(requestId, dispatchableSubPlan, timeoutMs, queryOptions, _mailboxService);
+    } catch (Exception ex) {

Review Comment:
   Forget it, I missed the PR diff line gap and didn't get that the potential exception raised by `tryRecover` was being rethrown.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


Re: [PR] Include stats on cancel [pinot]

Posted by "gortiz (via GitHub)" <gi...@apache.org>.
gortiz commented on code in PR #15609:
URL: https://github.com/apache/pinot/pull/15609#discussion_r2068639112


##########
pinot-query-runtime/src/main/java/org/apache/pinot/query/service/dispatch/QueryDispatcher.java:
##########
@@ -146,18 +162,40 @@ public QueryResult submitAndReduce(RequestContext context, DispatchableSubPlan d
     Set<QueryServerInstance> servers = new HashSet<>();
     try {
       submit(requestId, dispatchableSubPlan, timeoutMs, servers, queryOptions);
-      try {
-        return runReducer(requestId, dispatchableSubPlan, timeoutMs, queryOptions, _mailboxService);
-      } finally {
-        if (isQueryCancellationEnabled()) {
-          _serversByQuery.remove(requestId);
-        }
-      }
+      return runReducer(requestId, dispatchableSubPlan, timeoutMs, queryOptions, _mailboxService);
+    } catch (Exception ex) {
+      return tryRecover(context.getRequestId(), servers, ex);
     } catch (Throwable e) {
       // TODO: Consider always cancel when it returns (early terminate)
-      cancel(requestId, servers);
+      cancel(requestId);
       throw e;
+    } finally {
+      if (isQueryCancellationEnabled()) {
+        _serversByQuery.remove(requestId);
+      }
+    }
+  }
+
+  private QueryResult tryRecover(long requestId, Set<QueryServerInstance> servers, Exception ex)
+      throws Exception {
+    if (servers.isEmpty()) {
+      throw ex;
+    }
+    QueryErrorCode errorCode;
+    if (ex instanceof TimeoutException) {
+      errorCode = QueryErrorCode.EXECUTION_TIMEOUT;
+    } else if (ex instanceof QueryException) {
+      errorCode = ((QueryException) ex).getErrorCode();
+    } else {
+      cancel(requestId, servers);
+      throw ex;
+    }
+    MultiStageQueryStats stats = cancelWithStats(requestId, servers);

Review Comment:
   I think the correct point of view is the opposite. It is not that timeouts require special handling. We will also treat QueryExceptions in the same manner and that will be the standard behavior on standard failures.
   
   The special handling is in the other case: For unknown exceptions, we will cancel without statistics because we do not actually know what happened.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


Re: [PR] Include stats on cancel [pinot]

Posted by "gortiz (via GitHub)" <gi...@apache.org>.
gortiz commented on code in PR #15609:
URL: https://github.com/apache/pinot/pull/15609#discussion_r2068646994


##########
pinot-query-runtime/src/main/java/org/apache/pinot/query/service/dispatch/QueryDispatcher.java:
##########
@@ -146,18 +162,40 @@ public QueryResult submitAndReduce(RequestContext context, DispatchableSubPlan d
     Set<QueryServerInstance> servers = new HashSet<>();
     try {
       submit(requestId, dispatchableSubPlan, timeoutMs, servers, queryOptions);
-      try {
-        return runReducer(requestId, dispatchableSubPlan, timeoutMs, queryOptions, _mailboxService);
-      } finally {
-        if (isQueryCancellationEnabled()) {
-          _serversByQuery.remove(requestId);
-        }
-      }
+      return runReducer(requestId, dispatchableSubPlan, timeoutMs, queryOptions, _mailboxService);
+    } catch (Exception ex) {
+      return tryRecover(context.getRequestId(), servers, ex);
     } catch (Throwable e) {
       // TODO: Consider always cancel when it returns (early terminate)
-      cancel(requestId, servers);
+      cancel(requestId);
       throw e;
+    } finally {
+      if (isQueryCancellationEnabled()) {
+        _serversByQuery.remove(requestId);
+      }
+    }
+  }
+
+  private QueryResult tryRecover(long requestId, Set<QueryServerInstance> servers, Exception ex)
+      throws Exception {
+    if (servers.isEmpty()) {
+      throw ex;
+    }
+    QueryErrorCode errorCode;
+    if (ex instanceof TimeoutException) {
+      errorCode = QueryErrorCode.EXECUTION_TIMEOUT;
+    } else if (ex instanceof QueryException) {
+      errorCode = ((QueryException) ex).getErrorCode();
+    } else {
+      cancel(requestId, servers);
+      throw ex;
+    }
+    MultiStageQueryStats stats = cancelWithStats(requestId, servers);

Review Comment:
   I've added a comment anyway



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


Re: [PR] Include stats on cancel [pinot]

Posted by "gortiz (via GitHub)" <gi...@apache.org>.
gortiz commented on code in PR #15609:
URL: https://github.com/apache/pinot/pull/15609#discussion_r2073370059


##########
pinot-spi/src/main/java/org/apache/pinot/spi/utils/CommonConstants.java:
##########
@@ -1577,6 +1576,18 @@ public static class PlanVersions {
     public static final String KEY_OF_MULTISTAGE_EXPLAIN_INCLUDE_SEGMENT_PLAN =
         "pinot.query.multistage.explain.include.segment.plan";
     public static final boolean DEFAULT_OF_MULTISTAGE_EXPLAIN_INCLUDE_SEGMENT_PLAN = false;
+
+    /// Max number of rows operators stored in the op stats cache.
+    /// Although the cache stores stages, each entry has a weight equal to the number of operators in the stage.
+    public static final String KEY_OF_OP_STATS_CACHE_MAX = "pinot.server.query.op.stats.cache.max";
+    public static final int DEFAULT_OF_OP_STATS_CACHE_MAX = 1000;
+
+    /// Max time to keep the op stats in the cache.
+    public static final String KEY_OF_OP_STATS_CACHE_EXPIRE_MS = "pinot.server.query.op.stats.cache.secs";
+    public static final int DEFAULT_OF_OP_STATS_CACHE_EXPIRE_MS = 10 * 60 * 1000;

Review Comment:
   Fixed. AFAIU the main problem was the actual property name, but I'm also changing the default to 1 min, given that should be good enough (specially when we have a timeout of 1s for the cancellation mechanism)



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


Re: [PR] Include stats on cancel [pinot]

Posted by "Jackie-Jiang (via GitHub)" <gi...@apache.org>.
Jackie-Jiang commented on code in PR #15609:
URL: https://github.com/apache/pinot/pull/15609#discussion_r2054972689


##########
pinot-broker/src/main/java/org/apache/pinot/broker/requesthandler/MultiStageBrokerRequestHandler.java:
##########
@@ -443,11 +443,27 @@ private BrokerResponse query(QueryEnvironment.CompiledQuery query, long requestI
         Tracing.ThreadAccountantOps.clear();
         onQueryFinish(requestId);
       }
-      long executionEndTimeNs = System.nanoTime();
-      updatePhaseTimingForTables(tableNames, BrokerQueryPhase.QUERY_EXECUTION,
-          executionEndTimeNs - executionStartTimeNs);
 
       BrokerResponseNativeV2 brokerResponse = new BrokerResponseNativeV2();
+
+      QueryProcessingException processingException = queryResults.getProcessingException();
+      if (processingException != null) {
+        brokerResponse.addException(processingException);
+        QueryErrorCode errorCode = QueryErrorCode.fromErrorCode(processingException.getErrorCode());
+        if (errorCode == QueryErrorCode.EXECUTION_TIMEOUT) {
+          for (String table : tableNames) {
+            _brokerMetrics.addMeteredTableValue(table, BrokerMeter.BROKER_RESPONSES_WITH_TIMEOUTS, 1);
+          }
+          LOGGER.warn("Timed out executing request {}: {}", requestId, query);

Review Comment:
   Do we want to log for other exception types?



##########
pinot-broker/src/main/java/org/apache/pinot/broker/requesthandler/MultiStageBrokerRequestHandler.java:
##########
@@ -443,11 +443,27 @@ private BrokerResponse query(QueryEnvironment.CompiledQuery query, long requestI
         Tracing.ThreadAccountantOps.clear();
         onQueryFinish(requestId);
       }
-      long executionEndTimeNs = System.nanoTime();
-      updatePhaseTimingForTables(tableNames, BrokerQueryPhase.QUERY_EXECUTION,
-          executionEndTimeNs - executionStartTimeNs);
 
       BrokerResponseNativeV2 brokerResponse = new BrokerResponseNativeV2();
+
+      QueryProcessingException processingException = queryResults.getProcessingException();
+      if (processingException != null) {
+        brokerResponse.addException(processingException);
+        QueryErrorCode errorCode = QueryErrorCode.fromErrorCode(processingException.getErrorCode());
+        if (errorCode == QueryErrorCode.EXECUTION_TIMEOUT) {
+          for (String table : tableNames) {
+            _brokerMetrics.addMeteredTableValue(table, BrokerMeter.BROKER_RESPONSES_WITH_TIMEOUTS, 1);
+          }
+          LOGGER.warn("Timed out executing request {}: {}", requestId, query);
+        }
+        requestContext.setErrorCode(errorCode);
+      } else {
+        brokerResponse.setResultTable(queryResults.getResultTable());
+        long executionEndTimeNs = System.nanoTime();
+        updatePhaseTimingForTables(tableNames, BrokerQueryPhase.QUERY_EXECUTION,

Review Comment:
   This is a behavior change, where we only track time for success queries



##########
pinot-broker/src/main/java/org/apache/pinot/broker/requesthandler/MultiStageBrokerRequestHandler.java:
##########
@@ -443,11 +443,27 @@ private BrokerResponse query(QueryEnvironment.CompiledQuery query, long requestI
         Tracing.ThreadAccountantOps.clear();
         onQueryFinish(requestId);
       }
-      long executionEndTimeNs = System.nanoTime();
-      updatePhaseTimingForTables(tableNames, BrokerQueryPhase.QUERY_EXECUTION,
-          executionEndTimeNs - executionStartTimeNs);
 
       BrokerResponseNativeV2 brokerResponse = new BrokerResponseNativeV2();
+
+      QueryProcessingException processingException = queryResults.getProcessingException();
+      if (processingException != null) {
+        brokerResponse.addException(processingException);
+        QueryErrorCode errorCode = QueryErrorCode.fromErrorCode(processingException.getErrorCode());
+        if (errorCode == QueryErrorCode.EXECUTION_TIMEOUT) {
+          for (String table : tableNames) {
+            _brokerMetrics.addMeteredTableValue(table, BrokerMeter.BROKER_RESPONSES_WITH_TIMEOUTS, 1);
+          }
+          LOGGER.warn("Timed out executing request {}: {}", requestId, query);
+        }
+        requestContext.setErrorCode(errorCode);
+      } else {
+        brokerResponse.setResultTable(queryResults.getResultTable());

Review Comment:
   (minor) This is set on line 468



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


Re: [PR] Include stats on cancel [pinot]

Posted by "yashmayya (via GitHub)" <gi...@apache.org>.
yashmayya commented on code in PR #15609:
URL: https://github.com/apache/pinot/pull/15609#discussion_r2065941687


##########
pinot-spi/src/main/java/org/apache/pinot/spi/utils/CommonConstants.java:
##########
@@ -1570,6 +1569,18 @@ public static class PlanVersions {
     public static final String KEY_OF_MULTISTAGE_EXPLAIN_INCLUDE_SEGMENT_PLAN =
         "pinot.query.multistage.explain.include.segment.plan";
     public static final boolean DEFAULT_OF_MULTISTAGE_EXPLAIN_INCLUDE_SEGMENT_PLAN = false;
+
+    /// Max number of rows operators stored in the op stats cache.
+    /// Although the cache stores stages, each entry has a weight equal to the number of operators in the stage.
+    public static final String KEY_OF_OP_STATS_CACHE_MAX = "pinot.server.query.op.stats.cache.max";
+    public static final int DEFAULT_OF_OP_STATS_CACHE_MAX = 1000;
+
+    /// Max time to keep the op stats in the cache.
+    public static final String KEY_OF_OP_STATS_CACHE_EXPIRE_MS = "pinot.server.query.op.stats.cache.secs";
+    public static final int DEFAULT_OF_OP_STATS_CACHE_EXPIRE_MS = 10 * 60 * 1000;
+    /// Timeout of the cancel request, in milliseconds.
+    public static final String KEY_OF_CANCEL_TIMEOUT_MS = "pinot.server.query.cancel.timeout";

Review Comment:
   Is this intended to be applied to SSE query cancellation as well (in the future) or can we indicate that this is MSE specific in the config?



##########
pinot-spi/src/main/java/org/apache/pinot/spi/utils/CommonConstants.java:
##########
@@ -1570,6 +1569,18 @@ public static class PlanVersions {
     public static final String KEY_OF_MULTISTAGE_EXPLAIN_INCLUDE_SEGMENT_PLAN =
         "pinot.query.multistage.explain.include.segment.plan";
     public static final boolean DEFAULT_OF_MULTISTAGE_EXPLAIN_INCLUDE_SEGMENT_PLAN = false;
+
+    /// Max number of rows operators stored in the op stats cache.
+    /// Although the cache stores stages, each entry has a weight equal to the number of operators in the stage.
+    public static final String KEY_OF_OP_STATS_CACHE_MAX = "pinot.server.query.op.stats.cache.max";
+    public static final int DEFAULT_OF_OP_STATS_CACHE_MAX = 1000;
+
+    /// Max time to keep the op stats in the cache.
+    public static final String KEY_OF_OP_STATS_CACHE_EXPIRE_MS = "pinot.server.query.op.stats.cache.secs";
+    public static final int DEFAULT_OF_OP_STATS_CACHE_EXPIRE_MS = 10 * 60 * 1000;
+    /// Timeout of the cancel request, in milliseconds.
+    public static final String KEY_OF_CANCEL_TIMEOUT_MS = "pinot.server.query.cancel.timeout";

Review Comment:
   ```suggestion
       public static final String KEY_OF_CANCEL_TIMEOUT_MS = "pinot.server.query.cancel.timeout.ms";
   ```
   nit: other milliseconds based configs have this suffix



##########
pinot-query-runtime/src/main/java/org/apache/pinot/query/runtime/executor/OpChainSchedulerService.java:
##########
@@ -18,35 +18,65 @@
  */
 package org.apache.pinot.query.runtime.executor;
 
+import com.google.common.cache.Cache;
+import com.google.common.cache.CacheBuilder;
+import java.util.ArrayList;
 import java.util.Iterator;
 import java.util.Map;
 import java.util.concurrent.ConcurrentHashMap;
 import java.util.concurrent.ExecutorService;
 import java.util.concurrent.Future;
+import java.util.concurrent.TimeUnit;
+import java.util.stream.Collectors;
 import org.apache.pinot.core.util.trace.TraceRunnable;
 import org.apache.pinot.query.runtime.blocks.ErrorMseBlock;
 import org.apache.pinot.query.runtime.blocks.MseBlock;
+import org.apache.pinot.query.runtime.operator.MultiStageOperator;
 import org.apache.pinot.query.runtime.operator.OpChain;
 import org.apache.pinot.query.runtime.operator.OpChainId;
 import org.apache.pinot.query.runtime.plan.MultiStageQueryStats;
 import org.apache.pinot.spi.accounting.ThreadExecutionContext;
+import org.apache.pinot.spi.env.PinotConfiguration;
 import org.apache.pinot.spi.trace.Tracing;
+import org.apache.pinot.spi.utils.CommonConstants.MultiStageQueryRunner;
 import org.slf4j.Logger;
 import org.slf4j.LoggerFactory;
 
-
 public class OpChainSchedulerService {
   private static final Logger LOGGER = LoggerFactory.getLogger(OpChainSchedulerService.class);
 
   private final ExecutorService _executorService;
-  private final ConcurrentHashMap<OpChainId, Future<?>> _submittedOpChainMap;
+  private final ConcurrentHashMap<OpChainId, Future<?>> _submittedOpChainMap = new ConcurrentHashMap<>();
+  private final Cache<OpChainId, MultiStageOperator> _opChainCache;
+
+
+  public OpChainSchedulerService(ExecutorService executorService, PinotConfiguration config) {
+    this(
+        executorService,
+        config.getProperty(MultiStageQueryRunner.KEY_OF_OP_STATS_CACHE_MAX,
+            MultiStageQueryRunner.DEFAULT_OF_OP_STATS_CACHE_MAX),
+        config.getProperty(MultiStageQueryRunner.KEY_OF_OP_STATS_CACHE_EXPIRE_MS,
+            MultiStageQueryRunner.DEFAULT_OF_OP_STATS_CACHE_EXPIRE_MS)
+    );
+  }
 
   public OpChainSchedulerService(ExecutorService executorService) {
+    this(executorService, MultiStageQueryRunner.DEFAULT_OF_OP_STATS_CACHE_MAX,
+        MultiStageQueryRunner.DEFAULT_OF_OP_STATS_CACHE_EXPIRE_MS);
+  }
+
+  public OpChainSchedulerService(ExecutorService executorService, int maxWeight, long expireAfterWriteMs) {
     _executorService = executorService;
-    _submittedOpChainMap = new ConcurrentHashMap<>();
+    _opChainCache = CacheBuilder.newBuilder()
+        .weigher((OpChainId key, MultiStageOperator value) -> countOperators(value))

Review Comment:
   Why do we care about the weight / number of operators in the tree here? Is the concern about the total number of operator instances being kept alive in memory?



##########
pinot-query-runtime/src/main/java/org/apache/pinot/query/runtime/executor/OpChainSchedulerService.java:
##########
@@ -18,35 +18,65 @@
  */
 package org.apache.pinot.query.runtime.executor;
 
+import com.google.common.cache.Cache;
+import com.google.common.cache.CacheBuilder;
+import java.util.ArrayList;
 import java.util.Iterator;
 import java.util.Map;
 import java.util.concurrent.ConcurrentHashMap;
 import java.util.concurrent.ExecutorService;
 import java.util.concurrent.Future;
+import java.util.concurrent.TimeUnit;
+import java.util.stream.Collectors;
 import org.apache.pinot.core.util.trace.TraceRunnable;
 import org.apache.pinot.query.runtime.blocks.ErrorMseBlock;
 import org.apache.pinot.query.runtime.blocks.MseBlock;
+import org.apache.pinot.query.runtime.operator.MultiStageOperator;
 import org.apache.pinot.query.runtime.operator.OpChain;
 import org.apache.pinot.query.runtime.operator.OpChainId;
 import org.apache.pinot.query.runtime.plan.MultiStageQueryStats;
 import org.apache.pinot.spi.accounting.ThreadExecutionContext;
+import org.apache.pinot.spi.env.PinotConfiguration;
 import org.apache.pinot.spi.trace.Tracing;
+import org.apache.pinot.spi.utils.CommonConstants.MultiStageQueryRunner;
 import org.slf4j.Logger;
 import org.slf4j.LoggerFactory;
 
-
 public class OpChainSchedulerService {
   private static final Logger LOGGER = LoggerFactory.getLogger(OpChainSchedulerService.class);
 
   private final ExecutorService _executorService;
-  private final ConcurrentHashMap<OpChainId, Future<?>> _submittedOpChainMap;
+  private final ConcurrentHashMap<OpChainId, Future<?>> _submittedOpChainMap = new ConcurrentHashMap<>();
+  private final Cache<OpChainId, MultiStageOperator> _opChainCache;
+
+
+  public OpChainSchedulerService(ExecutorService executorService, PinotConfiguration config) {

Review Comment:
   I think this constructor should be called in `QueryRunner`?



##########
pinot-query-runtime/src/main/java/org/apache/pinot/query/service/dispatch/QueryDispatcher.java:
##########
@@ -146,18 +162,40 @@ public QueryResult submitAndReduce(RequestContext context, DispatchableSubPlan d
     Set<QueryServerInstance> servers = new HashSet<>();
     try {
       submit(requestId, dispatchableSubPlan, timeoutMs, servers, queryOptions);
-      try {
-        return runReducer(requestId, dispatchableSubPlan, timeoutMs, queryOptions, _mailboxService);
-      } finally {
-        if (isQueryCancellationEnabled()) {
-          _serversByQuery.remove(requestId);
-        }
-      }
+      return runReducer(requestId, dispatchableSubPlan, timeoutMs, queryOptions, _mailboxService);
+    } catch (Exception ex) {
+      return tryRecover(context.getRequestId(), servers, ex);
     } catch (Throwable e) {
       // TODO: Consider always cancel when it returns (early terminate)
-      cancel(requestId, servers);
+      cancel(requestId);
       throw e;
+    } finally {
+      if (isQueryCancellationEnabled()) {
+        _serversByQuery.remove(requestId);
+      }
+    }
+  }
+
+  private QueryResult tryRecover(long requestId, Set<QueryServerInstance> servers, Exception ex)
+      throws Exception {
+    if (servers.isEmpty()) {
+      throw ex;
+    }
+    QueryErrorCode errorCode;
+    if (ex instanceof TimeoutException) {
+      errorCode = QueryErrorCode.EXECUTION_TIMEOUT;
+    } else if (ex instanceof QueryException) {
+      errorCode = ((QueryException) ex).getErrorCode();

Review Comment:
   We don't need to try this recovery of stats for non timeout exceptions right? Shouldn't the error blocks we get from the broker's root stage mailbox in those cases already have the stats?



##########
pinot-query-runtime/src/main/java/org/apache/pinot/query/service/dispatch/QueryDispatcher.java:
##########
@@ -146,18 +162,40 @@ public QueryResult submitAndReduce(RequestContext context, DispatchableSubPlan d
     Set<QueryServerInstance> servers = new HashSet<>();
     try {
       submit(requestId, dispatchableSubPlan, timeoutMs, servers, queryOptions);
-      try {
-        return runReducer(requestId, dispatchableSubPlan, timeoutMs, queryOptions, _mailboxService);
-      } finally {
-        if (isQueryCancellationEnabled()) {
-          _serversByQuery.remove(requestId);
-        }
-      }
+      return runReducer(requestId, dispatchableSubPlan, timeoutMs, queryOptions, _mailboxService);
+    } catch (Exception ex) {
+      return tryRecover(context.getRequestId(), servers, ex);
     } catch (Throwable e) {
       // TODO: Consider always cancel when it returns (early terminate)
-      cancel(requestId, servers);
+      cancel(requestId);
       throw e;
+    } finally {
+      if (isQueryCancellationEnabled()) {
+        _serversByQuery.remove(requestId);
+      }
+    }
+  }
+
+  private QueryResult tryRecover(long requestId, Set<QueryServerInstance> servers, Exception ex)
+      throws Exception {
+    if (servers.isEmpty()) {
+      throw ex;
+    }
+    QueryErrorCode errorCode;
+    if (ex instanceof TimeoutException) {
+      errorCode = QueryErrorCode.EXECUTION_TIMEOUT;
+    } else if (ex instanceof QueryException) {
+      errorCode = ((QueryException) ex).getErrorCode();
+    } else {
+      cancel(requestId, servers);
+      throw ex;
+    }
+    MultiStageQueryStats stats = cancelWithStats(requestId, servers);

Review Comment:
   Might be useful to add a comment here briefly explaining why this special handling is required only for timeout errors.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


Re: [PR] Include stats on cancel [pinot]

Posted by "gortiz (via GitHub)" <gi...@apache.org>.
gortiz commented on code in PR #15609:
URL: https://github.com/apache/pinot/pull/15609#discussion_r2073370059


##########
pinot-spi/src/main/java/org/apache/pinot/spi/utils/CommonConstants.java:
##########
@@ -1577,6 +1576,18 @@ public static class PlanVersions {
     public static final String KEY_OF_MULTISTAGE_EXPLAIN_INCLUDE_SEGMENT_PLAN =
         "pinot.query.multistage.explain.include.segment.plan";
     public static final boolean DEFAULT_OF_MULTISTAGE_EXPLAIN_INCLUDE_SEGMENT_PLAN = false;
+
+    /// Max number of rows operators stored in the op stats cache.
+    /// Although the cache stores stages, each entry has a weight equal to the number of operators in the stage.
+    public static final String KEY_OF_OP_STATS_CACHE_MAX = "pinot.server.query.op.stats.cache.max";
+    public static final int DEFAULT_OF_OP_STATS_CACHE_MAX = 1000;
+
+    /// Max time to keep the op stats in the cache.
+    public static final String KEY_OF_OP_STATS_CACHE_EXPIRE_MS = "pinot.server.query.op.stats.cache.secs";
+    public static final int DEFAULT_OF_OP_STATS_CACHE_EXPIRE_MS = 10 * 60 * 1000;

Review Comment:
   Fixed. AFAIU the main problem was the actual property name



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


Re: [PR] Include stats on cancel [pinot]

Posted by "gortiz (via GitHub)" <gi...@apache.org>.
gortiz merged PR #15609:
URL: https://github.com/apache/pinot/pull/15609


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


Re: [PR] Include stats on cancel [pinot]

Posted by "gortiz (via GitHub)" <gi...@apache.org>.
gortiz commented on PR #15609:
URL: https://github.com/apache/pinot/pull/15609#issuecomment-2850946220

   I've applied some last minute changes, including a new option to configure whether opchains should be kept or not when they finish successfully. By default, we will not keep them. This keeps the cache as small as possible, but the side effect is that we may lose important information when part of the query finishes successfully early and then another part fails.
   
   When `pinot.server.query.op.stats.cache.invalidate.on.success` is false, these successfully ended opchains will be kept in the cache. The effect is that the cache will rotate more often. In high QPS environments, this may mean that queries that fail may be unable to find their stats.
   
   In future we can introduce a query option to disable successful invalidation on query bases, which could be useful when users write new queries.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


Re: [PR] Include stats on cancel [pinot]

Posted by "yashmayya (via GitHub)" <gi...@apache.org>.
yashmayya commented on code in PR #15609:
URL: https://github.com/apache/pinot/pull/15609#discussion_r2071256362


##########
pinot-query-runtime/src/main/java/org/apache/pinot/query/service/dispatch/QueryDispatcher.java:
##########
@@ -146,18 +162,40 @@ public QueryResult submitAndReduce(RequestContext context, DispatchableSubPlan d
     Set<QueryServerInstance> servers = new HashSet<>();
     try {
       submit(requestId, dispatchableSubPlan, timeoutMs, servers, queryOptions);
-      try {
-        return runReducer(requestId, dispatchableSubPlan, timeoutMs, queryOptions, _mailboxService);
-      } finally {
-        if (isQueryCancellationEnabled()) {
-          _serversByQuery.remove(requestId);
-        }
-      }
+      return runReducer(requestId, dispatchableSubPlan, timeoutMs, queryOptions, _mailboxService);
+    } catch (Exception ex) {
+      return tryRecover(context.getRequestId(), servers, ex);
     } catch (Throwable e) {
       // TODO: Consider always cancel when it returns (early terminate)
-      cancel(requestId, servers);
+      cancel(requestId);
       throw e;
+    } finally {
+      if (isQueryCancellationEnabled()) {
+        _serversByQuery.remove(requestId);
+      }
+    }
+  }
+
+  private QueryResult tryRecover(long requestId, Set<QueryServerInstance> servers, Exception ex)
+      throws Exception {
+    if (servers.isEmpty()) {
+      throw ex;
+    }
+    QueryErrorCode errorCode;
+    if (ex instanceof TimeoutException) {
+      errorCode = QueryErrorCode.EXECUTION_TIMEOUT;
+    } else if (ex instanceof QueryException) {
+      errorCode = ((QueryException) ex).getErrorCode();
+    } else {
+      cancel(requestId, servers);
+      throw ex;
+    }
+    MultiStageQueryStats stats = cancelWithStats(requestId, servers);

Review Comment:
   Oh, I missed that we're doing cancel with stats even for `QueryException`s in the same manner. I just saw your other comment as well, thanks for clarifying.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


Re: [PR] Include stats on cancel [pinot]

Posted by "yashmayya (via GitHub)" <gi...@apache.org>.
yashmayya commented on code in PR #15609:
URL: https://github.com/apache/pinot/pull/15609#discussion_r2071259158


##########
pinot-spi/src/main/java/org/apache/pinot/spi/utils/CommonConstants.java:
##########
@@ -1577,6 +1576,18 @@ public static class PlanVersions {
     public static final String KEY_OF_MULTISTAGE_EXPLAIN_INCLUDE_SEGMENT_PLAN =
         "pinot.query.multistage.explain.include.segment.plan";
     public static final boolean DEFAULT_OF_MULTISTAGE_EXPLAIN_INCLUDE_SEGMENT_PLAN = false;
+
+    /// Max number of rows operators stored in the op stats cache.
+    /// Although the cache stores stages, each entry has a weight equal to the number of operators in the stage.
+    public static final String KEY_OF_OP_STATS_CACHE_MAX = "pinot.server.query.op.stats.cache.max";
+    public static final int DEFAULT_OF_OP_STATS_CACHE_MAX = 1000;
+
+    /// Max time to keep the op stats in the cache.
+    public static final String KEY_OF_OP_STATS_CACHE_EXPIRE_MS = "pinot.server.query.op.stats.cache.secs";
+    public static final int DEFAULT_OF_OP_STATS_CACHE_EXPIRE_MS = 10 * 60 * 1000;

Review Comment:
   The variable name says MS and the default value seems to be 10 minutes but the config name says `secs` instead of `ms`?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org