You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pinot.apache.org by "Jackie-Jiang (via GitHub)" <gi...@apache.org> on 2024/02/05 20:37:27 UTC
[PR] [Multi-stage] Optimize query plan serialization [pinot]
Jackie-Jiang opened a new pull request, #12370:
URL: https://github.com/apache/pinot/pull/12370
Serializing proto object to bytes is dominating the CPU profile when dispatching the query. This PR optimizes the plan serialization by pre-serializing the reusable proto objects to bytes to avoid the per server serialization.
## Backward Incompatible
This PR changes the proto object structure, which will cause backward incompatibility when broker and server are running different version. Since we are not maintaining backward compatibility for multi-stage engine as of now, user should upgrade both brokers and servers, then start querying.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] [Multi-stage] Optimize query plan serialization [pinot]
Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #12370:
URL: https://github.com/apache/pinot/pull/12370#issuecomment-1928124644
## [Codecov](https://app.codecov.io/gh/apache/pinot/pull/12370?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
Attention: `7 lines` in your changes are missing coverage. Please review.
> Comparison is base [(`0a5a134`)](https://app.codecov.io/gh/apache/pinot/commit/0a5a134fb57fa24450fb3d385cc4f3756a32b343?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) 61.73% compared to head [(`bb47fb1`)](https://app.codecov.io/gh/apache/pinot/pull/12370?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) 61.55%.
> Report is 3 commits behind head on master.
| [Files](https://app.codecov.io/gh/apache/pinot/pull/12370?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Patch % | Lines |
|---|---|---|
| [...apache/pinot/query/service/server/QueryServer.java](https://app.codecov.io/gh/apache/pinot/pull/12370?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-cGlub3QtcXVlcnktcnVudGltZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvcXVlcnkvc2VydmljZS9zZXJ2ZXIvUXVlcnlTZXJ2ZXIuamF2YQ==) | 22.22% | [7 Missing :warning: ](https://app.codecov.io/gh/apache/pinot/pull/12370?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
<details><summary>Additional details and impacted files</summary>
```diff
@@ Coverage Diff @@
## master #12370 +/- ##
============================================
- Coverage 61.73% 61.55% -0.19%
Complexity 207 207
============================================
Files 2426 2426
Lines 132675 132709 +34
Branches 20506 20519 +13
============================================
- Hits 81908 81683 -225
- Misses 44752 45020 +268
+ Partials 6015 6006 -9
```
| [Flag](https://app.codecov.io/gh/apache/pinot/pull/12370/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [custom-integration1](https://app.codecov.io/gh/apache/pinot/pull/12370/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `<0.01% <0.00%> (ø)` | |
| [integration](https://app.codecov.io/gh/apache/pinot/pull/12370/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `<0.01% <0.00%> (ø)` | |
| [integration1](https://app.codecov.io/gh/apache/pinot/pull/12370/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [integration2](https://app.codecov.io/gh/apache/pinot/pull/12370/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `0.00% <0.00%> (ø)` | |
| [java-11](https://app.codecov.io/gh/apache/pinot/pull/12370/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `27.67% <0.00%> (-33.97%)` | :arrow_down: |
| [java-21](https://app.codecov.io/gh/apache/pinot/pull/12370/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `61.54% <91.46%> (-0.08%)` | :arrow_down: |
| [skip-bytebuffers-false](https://app.codecov.io/gh/apache/pinot/pull/12370/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `61.53% <91.46%> (-0.18%)` | :arrow_down: |
| [skip-bytebuffers-true](https://app.codecov.io/gh/apache/pinot/pull/12370/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `34.83% <91.46%> (-26.73%)` | :arrow_down: |
| [temurin](https://app.codecov.io/gh/apache/pinot/pull/12370/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `61.55% <91.46%> (-0.19%)` | :arrow_down: |
| [unittests](https://app.codecov.io/gh/apache/pinot/pull/12370/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `61.54% <91.46%> (-0.19%)` | :arrow_down: |
| [unittests1](https://app.codecov.io/gh/apache/pinot/pull/12370/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `46.72% <91.46%> (-0.20%)` | :arrow_down: |
| [unittests2](https://app.codecov.io/gh/apache/pinot/pull/12370/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `27.68% <0.00%> (-0.05%)` | :arrow_down: |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#carryforward-flags-in-the-pull-request-comment) to find out more.
</details>
[:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/apache/pinot/pull/12370?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
:loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] [Multi-stage] Optimize query plan serialization [pinot]
Posted by "Jackie-Jiang (via GitHub)" <gi...@apache.org>.
Jackie-Jiang merged PR #12370:
URL: https://github.com/apache/pinot/pull/12370
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] [Multi-stage] Optimize query plan serialization [pinot]
Posted by "Jackie-Jiang (via GitHub)" <gi...@apache.org>.
Jackie-Jiang commented on code in PR #12370:
URL: https://github.com/apache/pinot/pull/12370#discussion_r1478917570
##########
pinot-common/src/main/proto/worker.proto:
##########
@@ -92,3 +72,7 @@ message MailboxMetadata {
repeated string virtualAddress = 2;
map<string, string> customProperty = 3;
}
+
+message Properties {
Review Comment:
This is used to ser/de the map
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] [Multi-stage] Optimize query plan serialization [pinot]
Posted by "Jackie-Jiang (via GitHub)" <gi...@apache.org>.
Jackie-Jiang commented on code in PR #12370:
URL: https://github.com/apache/pinot/pull/12370#discussion_r1480737883
##########
pinot-query-runtime/src/test/java/org/apache/pinot/query/runtime/operator/MailboxReceiveOperatorTest.java:
##########
@@ -64,40 +69,35 @@ public class MailboxReceiveOperatorTest {
private ReceivingMailbox _mailbox1;
@Mock
private ReceivingMailbox _mailbox2;
- private StageMetadata _stageMetadataBoth;
- private StageMetadata _stageMetadata1;
- @BeforeMethod
+ @BeforeClass
Review Comment:
Moving the per method setup to per class is test optimization
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] [Multi-stage] Optimize query plan serialization [pinot]
Posted by "Jackie-Jiang (via GitHub)" <gi...@apache.org>.
Jackie-Jiang commented on code in PR #12370:
URL: https://github.com/apache/pinot/pull/12370#discussion_r1480735711
##########
pinot-query-runtime/src/main/java/org/apache/pinot/query/service/dispatch/QueryDispatcher.java:
##########
@@ -204,6 +213,18 @@ void submit(long requestId, DispatchableSubPlan dispatchableSubPlan, long timeou
}
}
+ private static class StageInfo {
Review Comment:
This is a simple helper class to help organize the info for each stage.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] [Multi-stage] Optimize query plan serialization [pinot]
Posted by "walterddr (via GitHub)" <gi...@apache.org>.
walterddr commented on code in PR #12370:
URL: https://github.com/apache/pinot/pull/12370#discussion_r1478961612
##########
pinot-query-runtime/src/main/java/org/apache/pinot/query/runtime/plan/server/ServerPlanRequestUtils.java:
##########
@@ -92,8 +92,9 @@ public static OpChain compileLeafStage(OpChainExecutionContext executionContext,
DistributedStagePlan distributedStagePlan, HelixManager helixManager, ServerMetrics serverMetrics,
QueryExecutor leafQueryExecutor, ExecutorService executorService) {
long queryArrivalTimeMs = System.currentTimeMillis();
- ServerPlanRequestContext serverContext = new ServerPlanRequestContext(distributedStagePlan, leafQueryExecutor,
- executorService, executionContext.getPipelineBreakerResult());
+ ServerPlanRequestContext serverContext =
Review Comment:
seems like all auto-formatting? (skipping most of these kinds)
##########
pinot-common/src/main/proto/mailbox.proto:
##########
@@ -17,24 +17,6 @@
// under the License.
//
-/**
Review Comment:
lol we should do license check better
##########
pinot-query-runtime/src/test/java/org/apache/pinot/query/runtime/operator/MailboxReceiveOperatorTest.java:
##########
@@ -64,40 +69,35 @@ public class MailboxReceiveOperatorTest {
private ReceivingMailbox _mailbox1;
@Mock
private ReceivingMailbox _mailbox2;
- private StageMetadata _stageMetadataBoth;
- private StageMetadata _stageMetadata1;
- @BeforeMethod
+ @BeforeClass
Review Comment:
was this change related to Ser/De or just test optimization?
##########
pinot-query-runtime/src/main/java/org/apache/pinot/query/service/dispatch/QueryDispatcher.java:
##########
@@ -204,6 +213,18 @@ void submit(long requestId, DispatchableSubPlan dispatchableSubPlan, long timeou
}
}
+ private static class StageInfo {
Review Comment:
do we still need stage info . I saw it got added during #12358 then removed in #12363 ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] [Multi-stage] Optimize query plan serialization [pinot]
Posted by "xiangfu0 (via GitHub)" <gi...@apache.org>.
xiangfu0 commented on code in PR #12370:
URL: https://github.com/apache/pinot/pull/12370#discussion_r1478901209
##########
pinot-common/src/main/proto/worker.proto:
##########
@@ -92,3 +72,7 @@ message MailboxMetadata {
repeated string virtualAddress = 2;
map<string, string> customProperty = 3;
}
+
+message Properties {
Review Comment:
this is not used.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org