You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pinot.apache.org by "61yao (via GitHub)" <gi...@apache.org> on 2023/01/27 09:05:59 UTC
[GitHub] [pinot] 61yao opened a new pull request, #10190: [multistage] [bugfix] Fix sending mailbox object leak
61yao opened a new pull request, #10190:
URL: https://github.com/apache/pinot/pull/10190
Fix sending mailbox object leak by moving the sending mailbox object into exchange, where the ownership of the object should lie.
This PR also changes the implementation of inMemoryMailbox a bit to be more aligned with GrpcMailbox.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] 61yao commented on pull request #10190: [multistage] [bugfix] Fix sending mailbox object leak
Posted by "61yao (via GitHub)" <gi...@apache.org>.
61yao commented on PR #10190:
URL: https://github.com/apache/pinot/pull/10190#issuecomment-1412995050
@xiangfu0
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] walterddr commented on a diff in pull request #10190: [multistage] [bugfix] Fix sending mailbox object leak
Posted by "walterddr (via GitHub)" <gi...@apache.org>.
walterddr commented on code in PR #10190:
URL: https://github.com/apache/pinot/pull/10190#discussion_r1094865213
##########
pinot-query-runtime/src/main/java/org/apache/pinot/query/runtime/operator/exchange/BlockExchange.java:
##########
@@ -38,23 +39,25 @@ public abstract class BlockExchange {
// TODO: Deduct this value via grpc config maximum byte size; and make it configurable with override.
// TODO: Max block size is a soft limit. only counts fixedSize datatable byte buffer
private static final int MAX_MAILBOX_CONTENT_SIZE_BYTES = 4 * 1024 * 1024;
-
- private final MailboxService<TransferableBlock> _mailbox;
- private final List<MailboxIdentifier> _destinations;
+ private final List<SendingMailbox> _sendingMailboxes;
Review Comment:
for this and the rest of the generic raw reference. please fix them with
```suggestion
private final List<SendingMailbox<TransferableBlock>> _sendingMailboxes;
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] walterddr merged pull request #10190: [multistage] [bugfix] Fix sending mailbox object leak
Posted by "walterddr (via GitHub)" <gi...@apache.org>.
walterddr merged PR #10190:
URL: https://github.com/apache/pinot/pull/10190
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] codecov-commenter commented on pull request #10190: [multistage] [bugfix] Fix sending mailbox object leak
Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #10190:
URL: https://github.com/apache/pinot/pull/10190#issuecomment-1412857172
# [Codecov](https://codecov.io/gh/apache/pinot/pull/10190?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#10190](https://codecov.io/gh/apache/pinot/pull/10190?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (0a2453c) into [master](https://codecov.io/gh/apache/pinot/commit/76a38bffab2530ca427e1ee281d50056a6b90633?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (76a38bf) will **decrease** coverage by `56.70%`.
> The diff coverage is `0.00%`.
```diff
@@ Coverage Diff @@
## master #10190 +/- ##
=============================================
- Coverage 70.40% 13.71% -56.70%
+ Complexity 5756 178 -5578
=============================================
Files 2016 1961 -55
Lines 109100 106615 -2485
Branches 16562 16271 -291
=============================================
- Hits 76815 14624 -62191
- Misses 26911 90841 +63930
+ Partials 5374 1150 -4224
```
| Flag | Coverage Δ | |
|---|---|---|
| integration1 | `?` | |
| integration2 | `?` | |
| unittests1 | `?` | |
| unittests2 | `13.71% <0.00%> (-0.01%)` | :arrow_down: |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more.
| [Impacted Files](https://codecov.io/gh/apache/pinot/pull/10190?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [...apache/pinot/query/mailbox/GrpcMailboxService.java](https://codecov.io/gh/apache/pinot/pull/10190?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtcXVlcnktcnVudGltZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvcXVlcnkvbWFpbGJveC9HcnBjTWFpbGJveFNlcnZpY2UuamF2YQ==) | `0.00% <0.00%> (-94.74%)` | :arrow_down: |
| [...apache/pinot/query/mailbox/GrpcSendingMailbox.java](https://codecov.io/gh/apache/pinot/pull/10190?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtcXVlcnktcnVudGltZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvcXVlcnkvbWFpbGJveC9HcnBjU2VuZGluZ01haWxib3guamF2YQ==) | `0.00% <0.00%> (-82.36%)` | :arrow_down: |
| [...he/pinot/query/mailbox/InMemoryMailboxService.java](https://codecov.io/gh/apache/pinot/pull/10190?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtcXVlcnktcnVudGltZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvcXVlcnkvbWFpbGJveC9Jbk1lbW9yeU1haWxib3hTZXJ2aWNlLmphdmE=) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [...he/pinot/query/mailbox/InMemorySendingMailbox.java](https://codecov.io/gh/apache/pinot/pull/10190?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtcXVlcnktcnVudGltZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvcXVlcnkvbWFpbGJveC9Jbk1lbW9yeVNlbmRpbmdNYWlsYm94LmphdmE=) | `0.00% <0.00%> (-61.54%)` | :arrow_down: |
| [...query/runtime/operator/exchange/BlockExchange.java](https://codecov.io/gh/apache/pinot/pull/10190?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtcXVlcnktcnVudGltZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvcXVlcnkvcnVudGltZS9vcGVyYXRvci9leGNoYW5nZS9CbG9ja0V4Y2hhbmdlLmphdmE=) | `0.00% <0.00%> (-94.12%)` | :arrow_down: |
| [...y/runtime/operator/exchange/BroadcastExchange.java](https://codecov.io/gh/apache/pinot/pull/10190?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtcXVlcnktcnVudGltZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvcXVlcnkvcnVudGltZS9vcGVyYXRvci9leGNoYW5nZS9Ccm9hZGNhc3RFeGNoYW5nZS5qYXZh) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [.../query/runtime/operator/exchange/HashExchange.java](https://codecov.io/gh/apache/pinot/pull/10190?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtcXVlcnktcnVudGltZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvcXVlcnkvcnVudGltZS9vcGVyYXRvci9leGNoYW5nZS9IYXNoRXhjaGFuZ2UuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [...uery/runtime/operator/exchange/RandomExchange.java](https://codecov.io/gh/apache/pinot/pull/10190?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtcXVlcnktcnVudGltZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvcXVlcnkvcnVudGltZS9vcGVyYXRvci9leGNoYW5nZS9SYW5kb21FeGNoYW5nZS5qYXZh) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [...y/runtime/operator/exchange/SingletonExchange.java](https://codecov.io/gh/apache/pinot/pull/10190?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtcXVlcnktcnVudGltZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvcXVlcnkvcnVudGltZS9vcGVyYXRvci9leGNoYW5nZS9TaW5nbGV0b25FeGNoYW5nZS5qYXZh) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [...src/main/java/org/apache/pinot/sql/FilterKind.java](https://codecov.io/gh/apache/pinot/pull/10190?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9zcWwvRmlsdGVyS2luZC5qYXZh) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| ... and [1595 more](https://codecov.io/gh/apache/pinot/pull/10190?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
:mega: We’re building smart automated test selection to slash your CI/CD build times. [Learn more](https://about.codecov.io/iterative-testing/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] walterddr commented on a diff in pull request #10190: [multistage] [bugfix] Fix sending mailbox object leak
Posted by "walterddr (via GitHub)" <gi...@apache.org>.
walterddr commented on code in PR #10190:
URL: https://github.com/apache/pinot/pull/10190#discussion_r1094866345
##########
pinot-query-runtime/src/main/java/org/apache/pinot/query/runtime/operator/exchange/BlockExchange.java:
##########
@@ -63,52 +66,33 @@ public static BlockExchange getExchange(MailboxService<TransferableBlock> mailbo
}
}
- protected BlockExchange(MailboxService<TransferableBlock> mailbox, List<MailboxIdentifier> destinations,
- BlockSplitter splitter) {
- _mailbox = mailbox;
- _destinations = destinations;
+ protected BlockExchange(List<SendingMailbox> sendingMailboxes, BlockSplitter splitter) {
+ _sendingMailboxes = sendingMailboxes;
_splitter = splitter;
}
public void send(TransferableBlock block) {
if (block.isEndOfStreamBlock()) {
- _destinations.forEach(destination -> sendBlock(destination, block));
+ _sendingMailboxes.forEach(destination -> sendBlock(destination, block));
return;
}
-
- Iterator<RoutedBlock> routedBlocks = route(_destinations, block);
- while (routedBlocks.hasNext()) {
- RoutedBlock next = routedBlocks.next();
- sendBlock(next._destination, next._block);
- }
+ route(_sendingMailboxes, block);
}
- private void sendBlock(MailboxIdentifier mailboxId, TransferableBlock block) {
- SendingMailbox<TransferableBlock> sendingMailbox = _mailbox.getSendingMailbox(mailboxId);
-
+ protected void sendBlock(SendingMailbox sendingMailbox, TransferableBlock block) {
if (block.isEndOfStreamBlock()) {
sendingMailbox.send(block);
sendingMailbox.complete();
return;
}
DataBlock.Type type = block.getType();
+ // TODO: Move splitter into mailbox service.
Review Comment:
why do we need to move splitter into mailbox service? did you mean mailbox operator?
##########
pinot-query-runtime/src/main/java/org/apache/pinot/query/runtime/operator/exchange/BlockExchange.java:
##########
@@ -38,23 +39,25 @@ public abstract class BlockExchange {
// TODO: Deduct this value via grpc config maximum byte size; and make it configurable with override.
// TODO: Max block size is a soft limit. only counts fixedSize datatable byte buffer
private static final int MAX_MAILBOX_CONTENT_SIZE_BYTES = 4 * 1024 * 1024;
-
- private final MailboxService<TransferableBlock> _mailbox;
- private final List<MailboxIdentifier> _destinations;
+ private final List<SendingMailbox> _sendingMailboxes;
Review Comment:
```suggestion
private final List<SendingMailbox<TransferableBlock>> _sendingMailboxes;
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org