You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pinot.apache.org by "61yao (via GitHub)" <gi...@apache.org> on 2023/01/27 09:05:59 UTC

[GitHub] [pinot] 61yao opened a new pull request, #10190: [multistage] [bugfix] Fix sending mailbox object leak

61yao opened a new pull request, #10190:
URL: https://github.com/apache/pinot/pull/10190

   Fix sending mailbox object leak by moving the sending mailbox object into exchange, where the ownership of the object should lie.
   
   This PR also changes the implementation of inMemoryMailbox a bit to be more aligned with GrpcMailbox.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


[GitHub] [pinot] 61yao commented on pull request #10190: [multistage] [bugfix] Fix sending mailbox object leak

Posted by "61yao (via GitHub)" <gi...@apache.org>.
61yao commented on PR #10190:
URL: https://github.com/apache/pinot/pull/10190#issuecomment-1412995050

   @xiangfu0 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


[GitHub] [pinot] walterddr commented on a diff in pull request #10190: [multistage] [bugfix] Fix sending mailbox object leak

Posted by "walterddr (via GitHub)" <gi...@apache.org>.
walterddr commented on code in PR #10190:
URL: https://github.com/apache/pinot/pull/10190#discussion_r1094865213


##########
pinot-query-runtime/src/main/java/org/apache/pinot/query/runtime/operator/exchange/BlockExchange.java:
##########
@@ -38,23 +39,25 @@ public abstract class BlockExchange {
   // TODO: Deduct this value via grpc config maximum byte size; and make it configurable with override.
   // TODO: Max block size is a soft limit. only counts fixedSize datatable byte buffer
   private static final int MAX_MAILBOX_CONTENT_SIZE_BYTES = 4 * 1024 * 1024;
-
-  private final MailboxService<TransferableBlock> _mailbox;
-  private final List<MailboxIdentifier> _destinations;
+  private final List<SendingMailbox> _sendingMailboxes;

Review Comment:
   for this and the rest of the generic raw reference. please fix them with
   ```suggestion
     private final List<SendingMailbox<TransferableBlock>> _sendingMailboxes;
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


[GitHub] [pinot] walterddr merged pull request #10190: [multistage] [bugfix] Fix sending mailbox object leak

Posted by "walterddr (via GitHub)" <gi...@apache.org>.
walterddr merged PR #10190:
URL: https://github.com/apache/pinot/pull/10190


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


[GitHub] [pinot] codecov-commenter commented on pull request #10190: [multistage] [bugfix] Fix sending mailbox object leak

Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #10190:
URL: https://github.com/apache/pinot/pull/10190#issuecomment-1412857172

   # [Codecov](https://codecov.io/gh/apache/pinot/pull/10190?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
   > Merging [#10190](https://codecov.io/gh/apache/pinot/pull/10190?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (0a2453c) into [master](https://codecov.io/gh/apache/pinot/commit/76a38bffab2530ca427e1ee281d50056a6b90633?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (76a38bf) will **decrease** coverage by `56.70%`.
   > The diff coverage is `0.00%`.
   
   ```diff
   @@              Coverage Diff              @@
   ##             master   #10190       +/-   ##
   =============================================
   - Coverage     70.40%   13.71%   -56.70%     
   + Complexity     5756      178     -5578     
   =============================================
     Files          2016     1961       -55     
     Lines        109100   106615     -2485     
     Branches      16562    16271      -291     
   =============================================
   - Hits          76815    14624    -62191     
   - Misses        26911    90841    +63930     
   + Partials       5374     1150     -4224     
   ```
   
   | Flag | Coverage Δ | |
   |---|---|---|
   | integration1 | `?` | |
   | integration2 | `?` | |
   | unittests1 | `?` | |
   | unittests2 | `13.71% <0.00%> (-0.01%)` | :arrow_down: |
   
   Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more.
   
   | [Impacted Files](https://codecov.io/gh/apache/pinot/pull/10190?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
   |---|---|---|
   | [...apache/pinot/query/mailbox/GrpcMailboxService.java](https://codecov.io/gh/apache/pinot/pull/10190?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtcXVlcnktcnVudGltZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvcXVlcnkvbWFpbGJveC9HcnBjTWFpbGJveFNlcnZpY2UuamF2YQ==) | `0.00% <0.00%> (-94.74%)` | :arrow_down: |
   | [...apache/pinot/query/mailbox/GrpcSendingMailbox.java](https://codecov.io/gh/apache/pinot/pull/10190?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtcXVlcnktcnVudGltZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvcXVlcnkvbWFpbGJveC9HcnBjU2VuZGluZ01haWxib3guamF2YQ==) | `0.00% <0.00%> (-82.36%)` | :arrow_down: |
   | [...he/pinot/query/mailbox/InMemoryMailboxService.java](https://codecov.io/gh/apache/pinot/pull/10190?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtcXVlcnktcnVudGltZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvcXVlcnkvbWFpbGJveC9Jbk1lbW9yeU1haWxib3hTZXJ2aWNlLmphdmE=) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
   | [...he/pinot/query/mailbox/InMemorySendingMailbox.java](https://codecov.io/gh/apache/pinot/pull/10190?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtcXVlcnktcnVudGltZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvcXVlcnkvbWFpbGJveC9Jbk1lbW9yeVNlbmRpbmdNYWlsYm94LmphdmE=) | `0.00% <0.00%> (-61.54%)` | :arrow_down: |
   | [...query/runtime/operator/exchange/BlockExchange.java](https://codecov.io/gh/apache/pinot/pull/10190?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtcXVlcnktcnVudGltZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvcXVlcnkvcnVudGltZS9vcGVyYXRvci9leGNoYW5nZS9CbG9ja0V4Y2hhbmdlLmphdmE=) | `0.00% <0.00%> (-94.12%)` | :arrow_down: |
   | [...y/runtime/operator/exchange/BroadcastExchange.java](https://codecov.io/gh/apache/pinot/pull/10190?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtcXVlcnktcnVudGltZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvcXVlcnkvcnVudGltZS9vcGVyYXRvci9leGNoYW5nZS9Ccm9hZGNhc3RFeGNoYW5nZS5qYXZh) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
   | [.../query/runtime/operator/exchange/HashExchange.java](https://codecov.io/gh/apache/pinot/pull/10190?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtcXVlcnktcnVudGltZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvcXVlcnkvcnVudGltZS9vcGVyYXRvci9leGNoYW5nZS9IYXNoRXhjaGFuZ2UuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
   | [...uery/runtime/operator/exchange/RandomExchange.java](https://codecov.io/gh/apache/pinot/pull/10190?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtcXVlcnktcnVudGltZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvcXVlcnkvcnVudGltZS9vcGVyYXRvci9leGNoYW5nZS9SYW5kb21FeGNoYW5nZS5qYXZh) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
   | [...y/runtime/operator/exchange/SingletonExchange.java](https://codecov.io/gh/apache/pinot/pull/10190?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtcXVlcnktcnVudGltZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvcXVlcnkvcnVudGltZS9vcGVyYXRvci9leGNoYW5nZS9TaW5nbGV0b25FeGNoYW5nZS5qYXZh) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
   | [...src/main/java/org/apache/pinot/sql/FilterKind.java](https://codecov.io/gh/apache/pinot/pull/10190?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9zcWwvRmlsdGVyS2luZC5qYXZh) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
   | ... and [1595 more](https://codecov.io/gh/apache/pinot/pull/10190?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
   
   :mega: We’re building smart automated test selection to slash your CI/CD build times. [Learn more](https://about.codecov.io/iterative-testing/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


[GitHub] [pinot] walterddr commented on a diff in pull request #10190: [multistage] [bugfix] Fix sending mailbox object leak

Posted by "walterddr (via GitHub)" <gi...@apache.org>.
walterddr commented on code in PR #10190:
URL: https://github.com/apache/pinot/pull/10190#discussion_r1094866345


##########
pinot-query-runtime/src/main/java/org/apache/pinot/query/runtime/operator/exchange/BlockExchange.java:
##########
@@ -63,52 +66,33 @@ public static BlockExchange getExchange(MailboxService<TransferableBlock> mailbo
     }
   }
 
-  protected BlockExchange(MailboxService<TransferableBlock> mailbox, List<MailboxIdentifier> destinations,
-      BlockSplitter splitter) {
-    _mailbox = mailbox;
-    _destinations = destinations;
+  protected BlockExchange(List<SendingMailbox> sendingMailboxes, BlockSplitter splitter) {
+    _sendingMailboxes = sendingMailboxes;
     _splitter = splitter;
   }
 
   public void send(TransferableBlock block) {
     if (block.isEndOfStreamBlock()) {
-      _destinations.forEach(destination -> sendBlock(destination, block));
+      _sendingMailboxes.forEach(destination -> sendBlock(destination, block));
       return;
     }
-
-    Iterator<RoutedBlock> routedBlocks = route(_destinations, block);
-    while (routedBlocks.hasNext()) {
-      RoutedBlock next = routedBlocks.next();
-      sendBlock(next._destination, next._block);
-    }
+    route(_sendingMailboxes, block);
   }
 
-  private void sendBlock(MailboxIdentifier mailboxId, TransferableBlock block) {
-    SendingMailbox<TransferableBlock> sendingMailbox = _mailbox.getSendingMailbox(mailboxId);
-
+  protected void sendBlock(SendingMailbox sendingMailbox, TransferableBlock block) {
     if (block.isEndOfStreamBlock()) {
       sendingMailbox.send(block);
       sendingMailbox.complete();
       return;
     }
 
     DataBlock.Type type = block.getType();
+    // TODO: Move splitter into mailbox service.

Review Comment:
   why do we need to move splitter into mailbox service? did you mean mailbox operator?



##########
pinot-query-runtime/src/main/java/org/apache/pinot/query/runtime/operator/exchange/BlockExchange.java:
##########
@@ -38,23 +39,25 @@ public abstract class BlockExchange {
   // TODO: Deduct this value via grpc config maximum byte size; and make it configurable with override.
   // TODO: Max block size is a soft limit. only counts fixedSize datatable byte buffer
   private static final int MAX_MAILBOX_CONTENT_SIZE_BYTES = 4 * 1024 * 1024;
-
-  private final MailboxService<TransferableBlock> _mailbox;
-  private final List<MailboxIdentifier> _destinations;
+  private final List<SendingMailbox> _sendingMailboxes;

Review Comment:
   ```suggestion
     private final List<SendingMailbox<TransferableBlock>> _sendingMailboxes;
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org