You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pinot.apache.org by "Jackie-Jiang (via GitHub)" <gi...@apache.org> on 2023/01/24 01:15:10 UTC
[GitHub] [pinot] Jackie-Jiang opened a new pull request, #10169: Enhance early termination for combine operators
Jackie-Jiang opened a new pull request, #10169:
URL: https://github.com/apache/pinot/pull/10169
- For single block combine operator, skip all remaining operators when query is satisfied
- For streaming combine operator, allow early termination when enough results are streamed back
- Handle `EarlyTerminationException` in the `InstanceResponseOperator` so that we won't miss the interrupt caught in `CombineOperator.nextBlock()`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] siddharthteotia commented on a diff in pull request #10169: Enhance early termination for combine operators
Posted by "siddharthteotia (via GitHub)" <gi...@apache.org>.
siddharthteotia commented on code in PR #10169:
URL: https://github.com/apache/pinot/pull/10169#discussion_r1084755975
##########
pinot-core/src/main/java/org/apache/pinot/core/operator/InstanceResponseOperator.java:
##########
@@ -111,6 +116,10 @@ private BaseResultsBlock getCombinedResults() {
try {
prefetchAll();
return _combineOperator.nextBlock();
+ } catch (EarlyTerminationException e) {
+ Exception killedErrorMsg = Tracing.getThreadAccountant().getErrorStatus();
+ return new ExceptionResultsBlock(new QueryCancelledException(
+ "Cancelled while combining results" + (killedErrorMsg == null ? StringUtils.EMPTY : " " + killedErrorMsg)));
Review Comment:
```
`throw new QueryCancelledException(
"Cancelled while merging results blocks" + (killedErrorMsg == null ? StringUtils.EMPTY
: " " + killedErrorMsg), e);`
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] Jackie-Jiang merged pull request #10169: Enhance early termination for combine operators
Posted by "Jackie-Jiang (via GitHub)" <gi...@apache.org>.
Jackie-Jiang merged PR #10169:
URL: https://github.com/apache/pinot/pull/10169
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] walterddr commented on a diff in pull request #10169: Enhance early termination for combine operators
Posted by "walterddr (via GitHub)" <gi...@apache.org>.
walterddr commented on code in PR #10169:
URL: https://github.com/apache/pinot/pull/10169#discussion_r1085993792
##########
pinot-core/src/main/java/org/apache/pinot/core/operator/combine/BaseSingleBlockCombineOperator.java:
##########
@@ -71,11 +68,8 @@ protected BaseResultsBlock getNextBlock() {
try {
startProcess();
mergedBlock = mergeResults();
- } catch (InterruptedException | EarlyTerminationException e) {
- Exception killedErrorMsg = Tracing.getThreadAccountant().getErrorStatus();
- throw new QueryCancelledException(
- "Cancelled while merging results blocks" + (killedErrorMsg == null ? StringUtils.EMPTY
- : " " + killedErrorMsg), e);
+ } catch (InterruptedException e) {
+ throw new EarlyTerminationException();
Review Comment:
plz include the original exception ?
```suggestion
throw new EarlyTerminationException(e);
```
##########
pinot-core/src/main/java/org/apache/pinot/core/operator/combine/BaseSingleBlockCombineOperator.java:
##########
@@ -111,13 +105,11 @@ protected void processSegments() {
((AcquireReleaseColumnsSegmentOperator) operator).release();
}
}
-
+ _blockingQueue.offer(resultsBlock);
+ // When query is satisfied, skip processing the remaining segments
if (_resultsBlockMerger.isQuerySatisfied(resultsBlock)) {
- // Query is satisfied, skip processing the remaining segments
- _blockingQueue.offer(resultsBlock);
+ _nextOperatorId.set(_numOperators);
Review Comment:
hmm. i am a bit worry about this one.
basically, this will make numBlocksMerged to practically impossible to reach numOperators during mergeResultBlock method.
so we are solely relying on the early termination mechanism to also early terminate on the mergeResultBlock.
however the
`_resultsBlockMerger.isQuerySatisfied(resultsBlock)` is not the same as the one in merge method
`_resultsBlockMerger.isQuerySatisfied(mregedBlock)`
are we 100% sure these are equivalent for sure?
##########
pinot-core/src/main/java/org/apache/pinot/core/operator/streaming/BaseStreamingCombineOperator.java:
##########
@@ -50,49 +52,55 @@
// Use a BlockingQueue to store the intermediate results blocks
protected final BlockingQueue<BaseResultsBlock> _blockingQueue = new LinkedBlockingQueue<>();
+ protected final ResultsBlockMerger<T> _resultsBlockMerger;
protected int _numOperatorsFinished;
- protected BaseResultsBlock _exceptionBlock;
+ protected boolean _querySatisfied;
- public BaseStreamingCombineOperator(List<Operator> operators,
+ public BaseStreamingCombineOperator(ResultsBlockMerger<T> resultsBlockMerger, List<Operator> operators,
QueryContext queryContext, ExecutorService executorService) {
super(operators, queryContext, executorService);
+ _resultsBlockMerger = resultsBlockMerger;
}
+ /**
+ * {@inheritDoc}
+ *
+ * When all the results blocks are returned, returns a final metadata block. Caller shouldn't call this method after
+ * it returns the metadata block or exception block.
+ */
@Override
protected BaseResultsBlock getNextBlock() {
long endTimeMs = _queryContext.getEndTimeMs();
- // TODO: Early terminate when query is satisfied
- while (_exceptionBlock == null && _numOperatorsFinished < _numOperators) {
+ Object querySatisfiedTracker = createQuerySatisfiedTracker();
+ while (!_querySatisfied && _numOperatorsFinished < _numOperators) {
try {
BaseResultsBlock resultsBlock =
_blockingQueue.poll(endTimeMs - System.currentTimeMillis(), TimeUnit.MILLISECONDS);
-
if (resultsBlock == null) {
// Query times out, skip streaming the remaining results blocks
LOGGER.error("Timed out while polling results block (query: {})", _queryContext);
- _exceptionBlock = new ExceptionResultsBlock(
- QueryException.getException(QueryException.EXECUTION_TIMEOUT_ERROR,
- new TimeoutException("Timed out while polling results block")));
- return _exceptionBlock;
+ return new ExceptionResultsBlock(QueryException.getException(QueryException.EXECUTION_TIMEOUT_ERROR,
+ new TimeoutException("Timed out while polling results block")));
}
if (resultsBlock.getProcessingExceptions() != null) {
// Caught exception while processing segment, skip streaming the remaining results blocks and directly return
// the exception
- _exceptionBlock = resultsBlock;
- return _exceptionBlock;
+ return resultsBlock;
}
if (resultsBlock == LAST_RESULTS_BLOCK) {
// Caught LAST_RESULTS_BLOCK from a specific task, indicated it has finished.
// Skip returning this metadata block and continue to process the next from the _blockingQueue.
_numOperatorsFinished++;
continue;
}
+ _querySatisfied = isQuerySatisfied((T) resultsBlock, querySatisfiedTracker);
return resultsBlock;
+ } catch (InterruptedException e) {
+ throw new EarlyTerminationException();
Review Comment:
actually i have another suggestion, let's put `e` in it, which might include stack trace, or do a logger here.
##########
pinot-core/src/main/java/org/apache/pinot/core/operator/InstanceResponseOperator.java:
##########
@@ -111,6 +116,10 @@ private BaseResultsBlock getCombinedResults() {
try {
prefetchAll();
return _combineOperator.nextBlock();
+ } catch (EarlyTerminationException e) {
+ Exception killedErrorMsg = Tracing.getThreadAccountant().getErrorStatus();
+ return new ExceptionResultsBlock(new QueryCancelledException(
+ "Cancelled while combining results" + (killedErrorMsg == null ? StringUtils.EMPTY : " " + killedErrorMsg)));
Review Comment:
this is highly unlikely but i was able to fiddle around the underlying threading mechanism to reproduce it :-D.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] siddharthteotia commented on a diff in pull request #10169: Enhance early termination for combine operators
Posted by "siddharthteotia (via GitHub)" <gi...@apache.org>.
siddharthteotia commented on code in PR #10169:
URL: https://github.com/apache/pinot/pull/10169#discussion_r1084806337
##########
pinot-core/src/main/java/org/apache/pinot/core/operator/streaming/StreamingSelectionOnlyCombineOperator.java:
##########
@@ -47,13 +47,17 @@ public String toExplainString() {
}
@Override
- protected boolean isOperatorSatisfied(SelectionResultsBlock resultsBlock) {
- long numRowsCollected = _numRowsCollected.addAndGet(resultsBlock.getNumRows());
- return numRowsCollected >= _limit;
+ protected boolean isChildOperatorSingleBlock() {
+ return false;
Review Comment:
Shouldn't this return to false at `BaseStreamingCombineOperator` level itself assuming single block here means non-streaming combine operator that merge into a single block ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] siddharthteotia commented on a diff in pull request #10169: Enhance early termination for combine operators
Posted by "siddharthteotia (via GitHub)" <gi...@apache.org>.
siddharthteotia commented on code in PR #10169:
URL: https://github.com/apache/pinot/pull/10169#discussion_r1084762766
##########
pinot-core/src/main/java/org/apache/pinot/core/operator/streaming/BaseStreamingCombineOperator.java:
##########
@@ -50,49 +52,55 @@
// Use a BlockingQueue to store the intermediate results blocks
protected final BlockingQueue<BaseResultsBlock> _blockingQueue = new LinkedBlockingQueue<>();
+ protected final ResultsBlockMerger<T> _resultsBlockMerger;
protected int _numOperatorsFinished;
- protected BaseResultsBlock _exceptionBlock;
+ protected boolean _querySatisfied;
- public BaseStreamingCombineOperator(List<Operator> operators,
+ public BaseStreamingCombineOperator(ResultsBlockMerger<T> resultsBlockMerger, List<Operator> operators,
QueryContext queryContext, ExecutorService executorService) {
super(operators, queryContext, executorService);
+ _resultsBlockMerger = resultsBlockMerger;
}
+ /**
+ * {@inheritDoc}
+ *
+ * When all the results blocks are returned, returns a final metadata block. Caller shouldn't call this method after
+ * it returns the metadata block or exception block.
+ */
@Override
protected BaseResultsBlock getNextBlock() {
long endTimeMs = _queryContext.getEndTimeMs();
- // TODO: Early terminate when query is satisfied
- while (_exceptionBlock == null && _numOperatorsFinished < _numOperators) {
+ Object querySatisfiedTracker = createQuerySatisfiedTracker();
+ while (!_querySatisfied && _numOperatorsFinished < _numOperators) {
try {
BaseResultsBlock resultsBlock =
_blockingQueue.poll(endTimeMs - System.currentTimeMillis(), TimeUnit.MILLISECONDS);
-
if (resultsBlock == null) {
// Query times out, skip streaming the remaining results blocks
LOGGER.error("Timed out while polling results block (query: {})", _queryContext);
- _exceptionBlock = new ExceptionResultsBlock(
- QueryException.getException(QueryException.EXECUTION_TIMEOUT_ERROR,
- new TimeoutException("Timed out while polling results block")));
- return _exceptionBlock;
+ return new ExceptionResultsBlock(QueryException.getException(QueryException.EXECUTION_TIMEOUT_ERROR,
+ new TimeoutException("Timed out while polling results block")));
}
if (resultsBlock.getProcessingExceptions() != null) {
// Caught exception while processing segment, skip streaming the remaining results blocks and directly return
// the exception
- _exceptionBlock = resultsBlock;
- return _exceptionBlock;
+ return resultsBlock;
}
if (resultsBlock == LAST_RESULTS_BLOCK) {
// Caught LAST_RESULTS_BLOCK from a specific task, indicated it has finished.
// Skip returning this metadata block and continue to process the next from the _blockingQueue.
_numOperatorsFinished++;
continue;
}
+ _querySatisfied = isQuerySatisfied((T) resultsBlock, querySatisfiedTracker);
return resultsBlock;
+ } catch (InterruptedException e) {
+ throw new EarlyTerminationException();
Review Comment:
May be we should instantiate `EarlyTerminationException` with `e.getMessage()` ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] Jackie-Jiang commented on pull request #10169: Enhance early termination for combine operators
Posted by "Jackie-Jiang (via GitHub)" <gi...@apache.org>.
Jackie-Jiang commented on PR #10169:
URL: https://github.com/apache/pinot/pull/10169#issuecomment-1402792810
@siddharthteotia @walterddr Added message and cause to the EarlyTerminationException and returned it in the query response
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] Jackie-Jiang commented on a diff in pull request #10169: Enhance early termination for combine operators
Posted by "Jackie-Jiang (via GitHub)" <gi...@apache.org>.
Jackie-Jiang commented on code in PR #10169:
URL: https://github.com/apache/pinot/pull/10169#discussion_r1085975802
##########
pinot-core/src/main/java/org/apache/pinot/core/operator/InstanceResponseOperator.java:
##########
@@ -111,6 +116,10 @@ private BaseResultsBlock getCombinedResults() {
try {
prefetchAll();
return _combineOperator.nextBlock();
+ } catch (EarlyTerminationException e) {
+ Exception killedErrorMsg = Tracing.getThreadAccountant().getErrorStatus();
+ return new ExceptionResultsBlock(new QueryCancelledException(
+ "Cancelled while combining results" + (killedErrorMsg == null ? StringUtils.EMPTY : " " + killedErrorMsg)));
Review Comment:
In the original code, we expect all the exceptions being handled within the combine operator, which might not always be the case because interrupt might be handled in `nextBlock()` call instead of the `getNextBlock()`, which will throw `EarlyTerminationException`. In order to handle all of them, we should throw `EarlyTerminationException` out and let the `InstanceResponseOperator` to handle it
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] Jackie-Jiang commented on a diff in pull request #10169: Enhance early termination for combine operators
Posted by "Jackie-Jiang (via GitHub)" <gi...@apache.org>.
Jackie-Jiang commented on code in PR #10169:
URL: https://github.com/apache/pinot/pull/10169#discussion_r1086012211
##########
pinot-core/src/main/java/org/apache/pinot/core/operator/combine/BaseSingleBlockCombineOperator.java:
##########
@@ -111,13 +105,11 @@ protected void processSegments() {
((AcquireReleaseColumnsSegmentOperator) operator).release();
}
}
-
+ _blockingQueue.offer(resultsBlock);
+ // When query is satisfied, skip processing the remaining segments
if (_resultsBlockMerger.isQuerySatisfied(resultsBlock)) {
- // Query is satisfied, skip processing the remaining segments
- _blockingQueue.offer(resultsBlock);
+ _nextOperatorId.set(_numOperators);
Review Comment:
Yes. When any single block can satisfy the query, the merged block can always satisfy the query.
Before #9387 we actually pre-assign segments to the thread, so we never rely on all operators to be finished.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] Jackie-Jiang commented on a diff in pull request #10169: Enhance early termination for combine operators
Posted by "Jackie-Jiang (via GitHub)" <gi...@apache.org>.
Jackie-Jiang commented on code in PR #10169:
URL: https://github.com/apache/pinot/pull/10169#discussion_r1085979025
##########
pinot-core/src/main/java/org/apache/pinot/core/operator/streaming/BaseStreamingCombineOperator.java:
##########
@@ -50,49 +52,55 @@
// Use a BlockingQueue to store the intermediate results blocks
protected final BlockingQueue<BaseResultsBlock> _blockingQueue = new LinkedBlockingQueue<>();
+ protected final ResultsBlockMerger<T> _resultsBlockMerger;
protected int _numOperatorsFinished;
- protected BaseResultsBlock _exceptionBlock;
+ protected boolean _querySatisfied;
- public BaseStreamingCombineOperator(List<Operator> operators,
+ public BaseStreamingCombineOperator(ResultsBlockMerger<T> resultsBlockMerger, List<Operator> operators,
QueryContext queryContext, ExecutorService executorService) {
super(operators, queryContext, executorService);
+ _resultsBlockMerger = resultsBlockMerger;
}
+ /**
+ * {@inheritDoc}
+ *
+ * When all the results blocks are returned, returns a final metadata block. Caller shouldn't call this method after
+ * it returns the metadata block or exception block.
+ */
@Override
protected BaseResultsBlock getNextBlock() {
long endTimeMs = _queryContext.getEndTimeMs();
- // TODO: Early terminate when query is satisfied
- while (_exceptionBlock == null && _numOperatorsFinished < _numOperators) {
+ Object querySatisfiedTracker = createQuerySatisfiedTracker();
+ while (!_querySatisfied && _numOperatorsFinished < _numOperators) {
try {
BaseResultsBlock resultsBlock =
_blockingQueue.poll(endTimeMs - System.currentTimeMillis(), TimeUnit.MILLISECONDS);
-
if (resultsBlock == null) {
// Query times out, skip streaming the remaining results blocks
LOGGER.error("Timed out while polling results block (query: {})", _queryContext);
- _exceptionBlock = new ExceptionResultsBlock(
- QueryException.getException(QueryException.EXECUTION_TIMEOUT_ERROR,
- new TimeoutException("Timed out while polling results block")));
- return _exceptionBlock;
+ return new ExceptionResultsBlock(QueryException.getException(QueryException.EXECUTION_TIMEOUT_ERROR,
+ new TimeoutException("Timed out while polling results block")));
}
if (resultsBlock.getProcessingExceptions() != null) {
// Caught exception while processing segment, skip streaming the remaining results blocks and directly return
// the exception
- _exceptionBlock = resultsBlock;
- return _exceptionBlock;
+ return resultsBlock;
}
if (resultsBlock == LAST_RESULTS_BLOCK) {
// Caught LAST_RESULTS_BLOCK from a specific task, indicated it has finished.
// Skip returning this metadata block and continue to process the next from the _blockingQueue.
_numOperatorsFinished++;
continue;
}
+ _querySatisfied = isQuerySatisfied((T) resultsBlock, querySatisfiedTracker);
return resultsBlock;
+ } catch (InterruptedException e) {
+ throw new EarlyTerminationException();
Review Comment:
We use `EarlyTerminationException` to pass the interrupt back to the caller, which is always caused by the query cancellation. IMO the message itself is not very useful
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] jasperjiaguo commented on a diff in pull request #10169: Enhance early termination for combine operators
Posted by "jasperjiaguo (via GitHub)" <gi...@apache.org>.
jasperjiaguo commented on code in PR #10169:
URL: https://github.com/apache/pinot/pull/10169#discussion_r1084768322
##########
pinot-core/src/main/java/org/apache/pinot/core/operator/InstanceResponseOperator.java:
##########
@@ -111,6 +116,10 @@ private BaseResultsBlock getCombinedResults() {
try {
prefetchAll();
return _combineOperator.nextBlock();
+ } catch (EarlyTerminationException e) {
+ Exception killedErrorMsg = Tracing.getThreadAccountant().getErrorStatus();
+ return new ExceptionResultsBlock(new QueryCancelledException(
+ "Cancelled while combining results" + (killedErrorMsg == null ? StringUtils.EMPTY : " " + killedErrorMsg)));
Review Comment:
@Jackie-Jiang IIUC in the runner thread, `InstanceResponseOperator` calls `_combineOperator.nextBlock()`; where the `InterruptedException` is folded into `EarlyTerminationException`, right?
It should be fine this way @siddharthteotia as long as `Tracing.getThreadAccountant().getErrorStatus();` get called eventually. It would be able to collect the error status from the accountant.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] walterddr commented on a diff in pull request #10169: Enhance early termination for combine operators
Posted by "walterddr (via GitHub)" <gi...@apache.org>.
walterddr commented on code in PR #10169:
URL: https://github.com/apache/pinot/pull/10169#discussion_r1086022612
##########
pinot-core/src/main/java/org/apache/pinot/core/operator/combine/BaseSingleBlockCombineOperator.java:
##########
@@ -111,13 +105,11 @@ protected void processSegments() {
((AcquireReleaseColumnsSegmentOperator) operator).release();
}
}
-
+ _blockingQueue.offer(resultsBlock);
+ // When query is satisfied, skip processing the remaining segments
if (_resultsBlockMerger.isQuerySatisfied(resultsBlock)) {
- // Query is satisfied, skip processing the remaining segments
- _blockingQueue.offer(resultsBlock);
+ _nextOperatorId.set(_numOperators);
Review Comment:
sounds good. i think the ResultBlockerMerger javadoc also stated that.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] jasperjiaguo commented on a diff in pull request #10169: Enhance early termination for combine operators
Posted by "jasperjiaguo (via GitHub)" <gi...@apache.org>.
jasperjiaguo commented on code in PR #10169:
URL: https://github.com/apache/pinot/pull/10169#discussion_r1084768322
##########
pinot-core/src/main/java/org/apache/pinot/core/operator/InstanceResponseOperator.java:
##########
@@ -111,6 +116,10 @@ private BaseResultsBlock getCombinedResults() {
try {
prefetchAll();
return _combineOperator.nextBlock();
+ } catch (EarlyTerminationException e) {
+ Exception killedErrorMsg = Tracing.getThreadAccountant().getErrorStatus();
+ return new ExceptionResultsBlock(new QueryCancelledException(
+ "Cancelled while combining results" + (killedErrorMsg == null ? StringUtils.EMPTY : " " + killedErrorMsg)));
Review Comment:
@Jackie-Jiang IIUC in the runner thread, `InstanceResponseOperator` calls `_combineOperator.nextBlock()`; where the `InterruptedException` is folded into `EarlyTerminationException`, right?
It should be fine this way @siddharthteotia as long as `Tracing.getThreadAccountant().getErrorStatus();` get called eventually. It would be able to collect the error status from the accountant, as the query killing message is not carried in `InterruptedException` but in `Tracing.getThreadAccountant().getErrorStatus()`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] siddharthteotia commented on a diff in pull request #10169: Enhance early termination for combine operators
Posted by "siddharthteotia (via GitHub)" <gi...@apache.org>.
siddharthteotia commented on code in PR #10169:
URL: https://github.com/apache/pinot/pull/10169#discussion_r1084755879
##########
pinot-core/src/main/java/org/apache/pinot/core/operator/InstanceResponseOperator.java:
##########
@@ -111,6 +116,10 @@ private BaseResultsBlock getCombinedResults() {
try {
prefetchAll();
return _combineOperator.nextBlock();
+ } catch (EarlyTerminationException e) {
+ Exception killedErrorMsg = Tracing.getThreadAccountant().getErrorStatus();
+ return new ExceptionResultsBlock(new QueryCancelledException(
+ "Cancelled while combining results" + (killedErrorMsg == null ? StringUtils.EMPTY : " " + killedErrorMsg)));
Review Comment:
So one difference between this and the existing code is that this is not including the original exception which could have been `InterruptedException` or `EarlyTerminationExcetption`.
The original code was including that inside `QueryCancelledException`, whereas here we have folded both into `EarlyTerminationExcetption`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] codecov-commenter commented on pull request #10169: Enhance early termination for combine operators
Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #10169:
URL: https://github.com/apache/pinot/pull/10169#issuecomment-1401298809
# [Codecov](https://codecov.io/gh/apache/pinot/pull/10169?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#10169](https://codecov.io/gh/apache/pinot/pull/10169?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (b898468) into [master](https://codecov.io/gh/apache/pinot/commit/f53bdf8ce850cf1084ab2283d4c26e15f5db95ba?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (f53bdf8) will **decrease** coverage by `56.81%`.
> The diff coverage is `0.00%`.
```diff
@@ Coverage Diff @@
## master #10169 +/- ##
=============================================
- Coverage 70.42% 13.61% -56.81%
+ Complexity 5788 176 -5612
=============================================
Files 2006 1962 -44
Lines 108756 106468 -2288
Branches 16503 16241 -262
=============================================
- Hits 76593 14500 -62093
- Misses 26813 90827 +64014
+ Partials 5350 1141 -4209
```
| Flag | Coverage Δ | |
|---|---|---|
| integration1 | `?` | |
| integration2 | `?` | |
| unittests1 | `?` | |
| unittests2 | `13.61% <0.00%> (-0.02%)` | :arrow_down: |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more.
| [Impacted Files](https://codecov.io/gh/apache/pinot/pull/10169?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [.../pinot/core/operator/InstanceResponseOperator.java](https://codecov.io/gh/apache/pinot/pull/10169?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9vcGVyYXRvci9JbnN0YW5jZVJlc3BvbnNlT3BlcmF0b3IuamF2YQ==) | `0.00% <0.00%> (-87.18%)` | :arrow_down: |
| [...e/operator/combine/AggregationCombineOperator.java](https://codecov.io/gh/apache/pinot/pull/10169?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9vcGVyYXRvci9jb21iaW5lL0FnZ3JlZ2F0aW9uQ29tYmluZU9wZXJhdG9yLmphdmE=) | `0.00% <0.00%> (-83.34%)` | :arrow_down: |
| [...not/core/operator/combine/BaseCombineOperator.java](https://codecov.io/gh/apache/pinot/pull/10169?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9vcGVyYXRvci9jb21iaW5lL0Jhc2VDb21iaW5lT3BlcmF0b3IuamF2YQ==) | `0.00% <0.00%> (-85.72%)` | :arrow_down: |
| [...erator/combine/BaseSingleBlockCombineOperator.java](https://codecov.io/gh/apache/pinot/pull/10169?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9vcGVyYXRvci9jb21iaW5lL0Jhc2VTaW5nbGVCbG9ja0NvbWJpbmVPcGVyYXRvci5qYXZh) | `0.00% <0.00%> (ø)` | |
| [...core/operator/combine/DistinctCombineOperator.java](https://codecov.io/gh/apache/pinot/pull/10169?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9vcGVyYXRvci9jb21iaW5lL0Rpc3RpbmN0Q29tYmluZU9wZXJhdG9yLmphdmE=) | `0.00% <0.00%> (-90.00%)` | :arrow_down: |
| [.../core/operator/combine/GroupByCombineOperator.java](https://codecov.io/gh/apache/pinot/pull/10169?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9vcGVyYXRvci9jb21iaW5lL0dyb3VwQnlDb21iaW5lT3BlcmF0b3IuamF2YQ==) | `0.00% <0.00%> (-84.38%)` | :arrow_down: |
| [...nMaxValueBasedSelectionOrderByCombineOperator.java](https://codecov.io/gh/apache/pinot/pull/10169?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9vcGVyYXRvci9jb21iaW5lL01pbk1heFZhbHVlQmFzZWRTZWxlY3Rpb25PcmRlckJ5Q29tYmluZU9wZXJhdG9yLmphdmE=) | `0.00% <0.00%> (-75.18%)` | :arrow_down: |
| [...operator/combine/SelectionOnlyCombineOperator.java](https://codecov.io/gh/apache/pinot/pull/10169?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9vcGVyYXRvci9jb21iaW5lL1NlbGVjdGlvbk9ubHlDb21iaW5lT3BlcmF0b3IuamF2YQ==) | `0.00% <0.00%> (-62.97%)` | :arrow_down: |
| [...rator/combine/SelectionOrderByCombineOperator.java](https://codecov.io/gh/apache/pinot/pull/10169?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9vcGVyYXRvci9jb21iaW5lL1NlbGVjdGlvbk9yZGVyQnlDb21iaW5lT3BlcmF0b3IuamF2YQ==) | `0.00% <0.00%> (-54.55%)` | :arrow_down: |
| [.../combine/merger/AggregationResultsBlockMerger.java](https://codecov.io/gh/apache/pinot/pull/10169?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9vcGVyYXRvci9jb21iaW5lL21lcmdlci9BZ2dyZWdhdGlvblJlc3VsdHNCbG9ja01lcmdlci5qYXZh) | `0.00% <0.00%> (ø)` | |
| ... and [1603 more](https://codecov.io/gh/apache/pinot/pull/10169?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
:mega: We’re building smart automated test selection to slash your CI/CD build times. [Learn more](https://about.codecov.io/iterative-testing/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] jasperjiaguo commented on a diff in pull request #10169: Enhance early termination for combine operators
Posted by "jasperjiaguo (via GitHub)" <gi...@apache.org>.
jasperjiaguo commented on code in PR #10169:
URL: https://github.com/apache/pinot/pull/10169#discussion_r1084768322
##########
pinot-core/src/main/java/org/apache/pinot/core/operator/InstanceResponseOperator.java:
##########
@@ -111,6 +116,10 @@ private BaseResultsBlock getCombinedResults() {
try {
prefetchAll();
return _combineOperator.nextBlock();
+ } catch (EarlyTerminationException e) {
+ Exception killedErrorMsg = Tracing.getThreadAccountant().getErrorStatus();
+ return new ExceptionResultsBlock(new QueryCancelledException(
+ "Cancelled while combining results" + (killedErrorMsg == null ? StringUtils.EMPTY : " " + killedErrorMsg)));
Review Comment:
@Jackie-Jiang IIUC in the runner thread, `InstanceResponseOperator` basically calls `_combineOperator.nextBlock()`; where the `InterruptedException` is folded into `EarlyTerminationException`, right?
It should be fine this way @siddharthteotia as long as `Tracing.getThreadAccountant().getErrorStatus();` get called eventually. It would be able to collect the error status from the accountant.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] siddharthteotia commented on a diff in pull request #10169: Enhance early termination for combine operators
Posted by "siddharthteotia (via GitHub)" <gi...@apache.org>.
siddharthteotia commented on code in PR #10169:
URL: https://github.com/apache/pinot/pull/10169#discussion_r1084806337
##########
pinot-core/src/main/java/org/apache/pinot/core/operator/streaming/StreamingSelectionOnlyCombineOperator.java:
##########
@@ -47,13 +47,17 @@ public String toExplainString() {
}
@Override
- protected boolean isOperatorSatisfied(SelectionResultsBlock resultsBlock) {
- long numRowsCollected = _numRowsCollected.addAndGet(resultsBlock.getNumRows());
- return numRowsCollected >= _limit;
+ protected boolean isChildOperatorSingleBlock() {
+ return false;
Review Comment:
Shouldn't this return to false at `BaseStreamingCombineOperator` level itself assuming single block here non-streaming combine operator that merge into a single block ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] Jackie-Jiang commented on a diff in pull request #10169: Enhance early termination for combine operators
Posted by "Jackie-Jiang (via GitHub)" <gi...@apache.org>.
Jackie-Jiang commented on code in PR #10169:
URL: https://github.com/apache/pinot/pull/10169#discussion_r1085982212
##########
pinot-core/src/main/java/org/apache/pinot/core/operator/streaming/StreamingSelectionOnlyCombineOperator.java:
##########
@@ -47,13 +47,17 @@ public String toExplainString() {
}
@Override
- protected boolean isOperatorSatisfied(SelectionResultsBlock resultsBlock) {
- long numRowsCollected = _numRowsCollected.addAndGet(resultsBlock.getNumRows());
- return numRowsCollected >= _limit;
+ protected boolean isChildOperatorSingleBlock() {
+ return false;
Review Comment:
This is for the child operator (e.g. AggregationOperator) which is not the CombineOperator. Most of the operators (executor on a single segment) returns a single block, with the only exception of `StreamingSelectionOnlyOperator`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org