You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pinot.apache.org by "Jackie-Jiang (via GitHub)" <gi...@apache.org> on 2023/08/23 07:10:21 UTC

[GitHub] [pinot] Jackie-Jiang opened a new pull request, #11411: Fix pipeline breaker error handling

Jackie-Jiang opened a new pull request, #11411:
URL: https://github.com/apache/pinot/pull/11411

   - Refactor pipeline breaker to explicitly return error block
   - When error occurs, propagate the error back without creating the whole op-chain
   - Fix the test to correctly verify the exception handling
   - Fix the test to correctly set the timeout into the request


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


[GitHub] [pinot] codecov-commenter commented on pull request #11411: Fix pipeline breaker error handling

Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #11411:
URL: https://github.com/apache/pinot/pull/11411#issuecomment-1689480672

   ## [Codecov](https://app.codecov.io/gh/apache/pinot/pull/11411?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
   > Merging [#11411](https://app.codecov.io/gh/apache/pinot/pull/11411?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) (1a16043) into [master](https://app.codecov.io/gh/apache/pinot/commit/e0d0aeb3d4d47e4e63589f7f96e194403f839075?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) (e0d0aeb) will **decrease** coverage by `48.39%`.
   > The diff coverage is `0.00%`.
   
   ```diff
   @@              Coverage Diff              @@
   ##             master   #11411       +/-   ##
   =============================================
   - Coverage     62.89%   14.51%   -48.39%     
   + Complexity     1091      207      -884     
   =============================================
     Files          2301     2301               
     Lines        123766   123770        +4     
     Branches      18833    18830        -3     
   =============================================
   - Hits          77845    17965    -59880     
   - Misses        40376   104281    +63905     
   + Partials       5545     1524     -4021     
   ```
   
   | [Flag](https://app.codecov.io/gh/apache/pinot/pull/11411/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
   |---|---|---|
   | [integration](https://app.codecov.io/gh/apache/pinot/pull/11411/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `<0.01% <0.00%> (ø)` | |
   | [integration1](https://app.codecov.io/gh/apache/pinot/pull/11411/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `<0.01% <0.00%> (ø)` | |
   | [integration2](https://app.codecov.io/gh/apache/pinot/pull/11411/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `0.00% <0.00%> (ø)` | |
   | [java-11](https://app.codecov.io/gh/apache/pinot/pull/11411/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `<0.01% <0.00%> (-62.87%)` | :arrow_down: |
   | [java-17](https://app.codecov.io/gh/apache/pinot/pull/11411/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `14.51% <0.00%> (-0.01%)` | :arrow_down: |
   | [java-20](https://app.codecov.io/gh/apache/pinot/pull/11411/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `0.00% <0.00%> (-62.77%)` | :arrow_down: |
   | [temurin](https://app.codecov.io/gh/apache/pinot/pull/11411/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `14.51% <0.00%> (-48.39%)` | :arrow_down: |
   | [unittests](https://app.codecov.io/gh/apache/pinot/pull/11411/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `14.51% <0.00%> (-48.39%)` | :arrow_down: |
   | [unittests1](https://app.codecov.io/gh/apache/pinot/pull/11411/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
   | [unittests2](https://app.codecov.io/gh/apache/pinot/pull/11411/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `14.51% <0.00%> (-0.01%)` | :arrow_down: |
   
   Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#carryforward-flags-in-the-pull-request-comment) to find out more.
   
   | [Files Changed](https://app.codecov.io/gh/apache/pinot/pull/11411?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
   |---|---|---|
   | [...org/apache/pinot/query/mailbox/MailboxService.java](https://app.codecov.io/gh/apache/pinot/pull/11411?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-cGlub3QtcXVlcnktcnVudGltZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvcXVlcnkvbWFpbGJveC9NYWlsYm94U2VydmljZS5qYXZh) | `0.00% <ø> (-76.48%)` | :arrow_down: |
   | [...va/org/apache/pinot/query/runtime/QueryRunner.java](https://app.codecov.io/gh/apache/pinot/pull/11411?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-cGlub3QtcXVlcnktcnVudGltZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvcXVlcnkvcnVudGltZS9RdWVyeVJ1bm5lci5qYXZh) | `0.00% <0.00%> (-69.00%)` | :arrow_down: |
   | [...runtime/plan/pipeline/PipelineBreakerExecutor.java](https://app.codecov.io/gh/apache/pinot/pull/11411?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-cGlub3QtcXVlcnktcnVudGltZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvcXVlcnkvcnVudGltZS9wbGFuL3BpcGVsaW5lL1BpcGVsaW5lQnJlYWtlckV4ZWN1dG9yLmphdmE=) | `0.00% <0.00%> (-92.86%)` | :arrow_down: |
   | [...runtime/plan/pipeline/PipelineBreakerOperator.java](https://app.codecov.io/gh/apache/pinot/pull/11411?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-cGlub3QtcXVlcnktcnVudGltZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvcXVlcnkvcnVudGltZS9wbGFuL3BpcGVsaW5lL1BpcGVsaW5lQnJlYWtlck9wZXJhdG9yLmphdmE=) | `0.00% <0.00%> (-71.12%)` | :arrow_down: |
   | [...y/runtime/plan/pipeline/PipelineBreakerResult.java](https://app.codecov.io/gh/apache/pinot/pull/11411?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-cGlub3QtcXVlcnktcnVudGltZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvcXVlcnkvcnVudGltZS9wbGFuL3BpcGVsaW5lL1BpcGVsaW5lQnJlYWtlclJlc3VsdC5qYXZh) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
   | [.../pinot/query/service/dispatch/QueryDispatcher.java](https://app.codecov.io/gh/apache/pinot/pull/11411?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-cGlub3QtcXVlcnktcnVudGltZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvcXVlcnkvc2VydmljZS9kaXNwYXRjaC9RdWVyeURpc3BhdGNoZXIuamF2YQ==) | `0.00% <0.00%> (-90.33%)` | :arrow_down: |
   
   ... and [1463 files with indirect coverage changes](https://app.codecov.io/gh/apache/pinot/pull/11411/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)
   
   :mega: We’re building smart automated test selection to slash your CI/CD build times. [Learn more](https://about.codecov.io/iterative-testing/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


[GitHub] [pinot] Jackie-Jiang commented on a diff in pull request #11411: Fix pipeline breaker error handling

Posted by "Jackie-Jiang (via GitHub)" <gi...@apache.org>.
Jackie-Jiang commented on code in PR #11411:
URL: https://github.com/apache/pinot/pull/11411#discussion_r1303409129


##########
pinot-query-runtime/src/main/java/org/apache/pinot/query/runtime/plan/pipeline/PipelineBreakerOperator.java:
##########
@@ -62,70 +55,55 @@ public Map<Integer, List<TransferableBlock>> getResultMap() {
   }
 
   @Nullable
+  public TransferableBlock getErrorBlock() {
+    return _errorBlock;
+  }
+
   @Override
   public String toExplainString() {
     return EXPLAIN_NAME;
   }
 
   @Override
   protected TransferableBlock getNextBlock() {
-    // Poll from every mailbox operator:
-    // - Return the first content block
-    // - If no content block found but there are mailboxes not finished, try again
-    // - If all content blocks are already returned, return end-of-stream block
-    while (!_workerEntries.isEmpty()) {
-      if (_finalBlock != null) {
-        return _finalBlock;
-      }
-      if (System.currentTimeMillis() > _context.getDeadlineMs()) {
-        _finalBlock = TransferableBlockUtils.getErrorTransferableBlock(QueryException.EXECUTION_TIMEOUT_ERROR);
-        constructErrorResponse(_finalBlock);
-        return _finalBlock;
-      }
-
-      Map.Entry<Integer, Operator<TransferableBlock>> worker = _workerEntries.getLast();
-      TransferableBlock block = worker.getValue().nextBlock();
-
-      if (block == null) {
-        if (LOGGER.isDebugEnabled()) {
-          LOGGER.debug("==[PB]== Null block on " + _context.getId() + " worker " + worker.getKey());
+    if (_errorBlock != null) {
+      return _errorBlock;
+    }
+    // NOTE: Put an empty list for each worker in case there is no data block returned from that worker
+    if (_workerMap.size() == 1) {
+      Map.Entry<Integer, Operator<TransferableBlock>> entry = _workerMap.entrySet().iterator().next();
+      List<TransferableBlock> dataBlocks = new ArrayList<>();
+      _resultMap = Collections.singletonMap(entry.getKey(), dataBlocks);
+      Operator<TransferableBlock> operator = entry.getValue();
+      TransferableBlock block = operator.nextBlock();
+      while (!block.isSuccessfulEndOfStreamBlock()) {
+        if (block.isErrorBlock()) {
+          _errorBlock = block;
+          return block;
         }
-        continue;
+        dataBlocks.add(block);
+        block = operator.nextBlock();
       }
-
-      // Release the mailbox worker when the block is end-of-stream
-      if (block.isSuccessfulEndOfStreamBlock()) {
-        _workerEntries.removeLast();
-        continue;
+    } else {
+      _resultMap = new HashMap<>();
+      for (int workerKey : _workerMap.keySet()) {
+        _resultMap.put(workerKey, new ArrayList<>());
       }
-
-      if (block.isErrorBlock()) {
-        _finalBlock = block;
-      }
-      if (LOGGER.isTraceEnabled()) {
-        LOGGER.trace("==[PB]== Returned block from : " + _context.getId() + " block: " + block);
-      }
-      _resultMap.get(worker.getKey()).add(block);
-      return block;
-    }
-    if (System.currentTimeMillis() > _context.getDeadlineMs()) {
-      _finalBlock = TransferableBlockUtils.getErrorTransferableBlock(QueryException.EXECUTION_TIMEOUT_ERROR);
-      return _finalBlock;
-    } else if (_finalBlock == null) {
-      if (LOGGER.isDebugEnabled()) {
-        LOGGER.debug("==[PB]== Finished : " + _context.getId());
+      // Keep polling from every operator in round-robin fashion
+      Queue<Map.Entry<Integer, Operator<TransferableBlock>>> entries = new ArrayDeque<>(_workerMap.entrySet());
+      while (!entries.isEmpty()) {
+        Map.Entry<Integer, Operator<TransferableBlock>> entry = entries.poll();
+        TransferableBlock block = entry.getValue().nextBlock();
+        if (block.isErrorBlock()) {
+          _errorBlock = block;
+          return block;
+        }
+        if (block.isDataBlock()) {
+          _resultMap.get(entry.getKey()).add(block);
+          entries.offer(entry);
+        }

Review Comment:
   It is tracked within the op-chain



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


[GitHub] [pinot] xiangfu0 commented on a diff in pull request #11411: Fix pipeline breaker error handling

Posted by "xiangfu0 (via GitHub)" <gi...@apache.org>.
xiangfu0 commented on code in PR #11411:
URL: https://github.com/apache/pinot/pull/11411#discussion_r1303392649


##########
pinot-query-runtime/src/main/java/org/apache/pinot/query/runtime/plan/pipeline/PipelineBreakerOperator.java:
##########
@@ -62,70 +55,55 @@ public Map<Integer, List<TransferableBlock>> getResultMap() {
   }
 
   @Nullable
+  public TransferableBlock getErrorBlock() {
+    return _errorBlock;
+  }
+
   @Override
   public String toExplainString() {
     return EXPLAIN_NAME;
   }
 
   @Override
   protected TransferableBlock getNextBlock() {
-    // Poll from every mailbox operator:
-    // - Return the first content block
-    // - If no content block found but there are mailboxes not finished, try again
-    // - If all content blocks are already returned, return end-of-stream block
-    while (!_workerEntries.isEmpty()) {
-      if (_finalBlock != null) {
-        return _finalBlock;
-      }
-      if (System.currentTimeMillis() > _context.getDeadlineMs()) {
-        _finalBlock = TransferableBlockUtils.getErrorTransferableBlock(QueryException.EXECUTION_TIMEOUT_ERROR);
-        constructErrorResponse(_finalBlock);
-        return _finalBlock;
-      }
-
-      Map.Entry<Integer, Operator<TransferableBlock>> worker = _workerEntries.getLast();
-      TransferableBlock block = worker.getValue().nextBlock();
-
-      if (block == null) {
-        if (LOGGER.isDebugEnabled()) {
-          LOGGER.debug("==[PB]== Null block on " + _context.getId() + " worker " + worker.getKey());
+    if (_errorBlock != null) {
+      return _errorBlock;
+    }
+    // NOTE: Put an empty list for each worker in case there is no data block returned from that worker
+    if (_workerMap.size() == 1) {
+      Map.Entry<Integer, Operator<TransferableBlock>> entry = _workerMap.entrySet().iterator().next();
+      List<TransferableBlock> dataBlocks = new ArrayList<>();
+      _resultMap = Collections.singletonMap(entry.getKey(), dataBlocks);
+      Operator<TransferableBlock> operator = entry.getValue();
+      TransferableBlock block = operator.nextBlock();
+      while (!block.isSuccessfulEndOfStreamBlock()) {
+        if (block.isErrorBlock()) {
+          _errorBlock = block;
+          return block;
         }
-        continue;
+        dataBlocks.add(block);
+        block = operator.nextBlock();
       }
-
-      // Release the mailbox worker when the block is end-of-stream
-      if (block.isSuccessfulEndOfStreamBlock()) {
-        _workerEntries.removeLast();
-        continue;
+    } else {
+      _resultMap = new HashMap<>();
+      for (int workerKey : _workerMap.keySet()) {
+        _resultMap.put(workerKey, new ArrayList<>());
       }
-
-      if (block.isErrorBlock()) {
-        _finalBlock = block;
-      }
-      if (LOGGER.isTraceEnabled()) {
-        LOGGER.trace("==[PB]== Returned block from : " + _context.getId() + " block: " + block);
-      }
-      _resultMap.get(worker.getKey()).add(block);
-      return block;
-    }
-    if (System.currentTimeMillis() > _context.getDeadlineMs()) {
-      _finalBlock = TransferableBlockUtils.getErrorTransferableBlock(QueryException.EXECUTION_TIMEOUT_ERROR);
-      return _finalBlock;
-    } else if (_finalBlock == null) {
-      if (LOGGER.isDebugEnabled()) {
-        LOGGER.debug("==[PB]== Finished : " + _context.getId());
+      // Keep polling from every operator in round-robin fashion
+      Queue<Map.Entry<Integer, Operator<TransferableBlock>>> entries = new ArrayDeque<>(_workerMap.entrySet());
+      while (!entries.isEmpty()) {
+        Map.Entry<Integer, Operator<TransferableBlock>> entry = entries.poll();
+        TransferableBlock block = entry.getValue().nextBlock();
+        if (block.isErrorBlock()) {
+          _errorBlock = block;
+          return block;
+        }
+        if (block.isDataBlock()) {
+          _resultMap.get(entry.getKey()).add(block);
+          entries.offer(entry);
+        }

Review Comment:
   Do we check for statsMap of endOfStream data block?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


[GitHub] [pinot] Jackie-Jiang merged pull request #11411: Fix pipeline breaker error handling

Posted by "Jackie-Jiang (via GitHub)" <gi...@apache.org>.
Jackie-Jiang merged PR #11411:
URL: https://github.com/apache/pinot/pull/11411


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org