[GitHub] [pinot] Jackie-Jiang commented on a diff in pull request #11334: [multistage] Early terminate SortOperator if there is a limit

["Jackie-Jiang (via GitHub)" <gi...@apache.org>]

Jackie-Jiang commented on code in PR #11334:
URL: https://github.com/apache/pinot/pull/11334#discussion_r1294110002


##########
pinot-query-runtime/src/main/java/org/apache/pinot/query/mailbox/GrpcSendingMailbox.java:
##########
@@ -66,13 +67,19 @@ public void send(TransferableBlock block)
     }
     Preconditions.checkState(!_statusObserver.isFinished(), "Mailbox: %s is already closed", _id);
     _contentObserver.onNext(toMailboxContent(block));
+    _isEarlyTerminated = _statusObserver.isFinished();

Review Comment:
   This will almost always throw exception on line 68



##########
pinot-query-runtime/src/main/java/org/apache/pinot/query/runtime/operator/SortOperator.java:
##########
@@ -175,6 +175,14 @@ private void consumeInputBlocks() {
               _rows.addAll(container.subList(0, _numRowsToKeep - _rows.size()));
             }
           }
+          // '_fetch > 0' means this operator can early terminate, so we can stop consuming input blocks
+          if (_rows.size() >= _numRowsToKeep && _fetch > 0) {
+            _readyToConstruct = true;
+            _upstreamOperator.close();

Review Comment:
   We should not close operator here



##########
pinot-query-runtime/src/main/java/org/apache/pinot/query/mailbox/SendingMailbox.java:
##########
@@ -50,6 +50,11 @@ void send(TransferableBlock block)
    */
   void complete();
 
+  /**
+   * Returns true if the mailbox has been early terminated. This can happen if the receiver has cancelled the mailbox
+   */
+  boolean isEarlyTerminated();

Review Comment:
   I don't see this being used



##########
pinot-query-runtime/src/main/java/org/apache/pinot/query/runtime/operator/SortOperator.java:
##########
@@ -175,6 +175,14 @@ private void consumeInputBlocks() {
               _rows.addAll(container.subList(0, _numRowsToKeep - _rows.size()));
             }
           }
+          // '_fetch > 0' means this operator can early terminate, so we can stop consuming input blocks
+          if (_rows.size() >= _numRowsToKeep && _fetch > 0) {

Review Comment:
   We don't allow unlimited rows. Checking `if (_rows.size() == _numRowsToKeep)` is good enough



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org