You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pinot.apache.org by "gortiz (via GitHub)" <gi...@apache.org> on 2024/11/19 15:40:40 UTC
[PR] [Spool] Introduce stage replacer and change send nodes to be able to send to more than one stage [pinot]
gortiz opened a new pull request, #14495:
URL: https://github.com/apache/pinot/pull/14495
This PR is a continuation of #14296 and the next step on #14196.
Here we define how to modify a query to replace stages that have been found equivalent. Not all equivalent stages can be substitute. Specifically, two stages that send to the same stage cannot be substituted by a spool. The reason for that is that the mailbox service assumes there the tuple (receiveStage, sendStage) is unique in the query. Probably this can be changed, but it doesn't seem to be worth it in this initial phase.
This PR also changes the equivalence function to consider not equivalent two senders with different distribution. The design algorithm includes some ways to also support this, but again, it adds a complexity that doesn't seem to be required in the first spool version.
Several tests have been added and also a new builder API to create spools in tests. Although the spool builder code is a bit ugly, it simplifies the tests that use it. There are some examples in `EquivalentStagesReplacerTest`.
The next PR may be the final one, where the worker protocol and operators are changed to support these new senders that can send everywhere.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] [Spool] Introduce stage replacer and change send nodes to be able to send to more than one stage [pinot]
Posted by "gortiz (via GitHub)" <gi...@apache.org>.
gortiz commented on code in PR #14495:
URL: https://github.com/apache/pinot/pull/14495#discussion_r1851542315
##########
pinot-query-planner/src/main/java/org/apache/pinot/query/planner/plannode/MailboxReceiveNode.java:
##########
@@ -93,6 +96,10 @@ public MailboxSendNode getSender() {
return _sender;
}
+ public void setSender(MailboxSendNode sender) {
+ _sender = sender;
+ }
Review Comment:
The main reason is to keep `_senderStageId`, but I have to say I already got fooled while debugging the code, so it may be a good idea.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] [Spool] Introduce stage replacer and change send nodes to be able to send to more than one stage [pinot]
Posted by "gortiz (via GitHub)" <gi...@apache.org>.
gortiz commented on code in PR #14495:
URL: https://github.com/apache/pinot/pull/14495#discussion_r1850238718
##########
pinot-query-planner/src/main/java/org/apache/pinot/query/planner/logical/PlanFragmenter.java:
##########
@@ -150,6 +150,7 @@ public PlanNode visitExchange(ExchangeNode node, Context context) {
// Split the ExchangeNode to a MailboxReceiveNode and a MailboxSendNode, where MailboxReceiveNode is the leave node
// of the current PlanFragment, and MailboxSendNode is the root node of the next PlanFragment.
int receiverPlanFragmentId = context._currentPlanFragmentId;
+ int receivedIdInPlan = -1; // TODO: Change this
Review Comment:
It isn't used. I'll remove it
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] [Spool] Introduce stage replacer and change send nodes to be able to send to more than one stage [pinot]
Posted by "yashmayya (via GitHub)" <gi...@apache.org>.
yashmayya commented on code in PR #14495:
URL: https://github.com/apache/pinot/pull/14495#discussion_r1849662897
##########
pinot-query-planner/src/main/java/org/apache/pinot/query/planner/plannode/MailboxSendNode.java:
##########
@@ -50,8 +55,50 @@ public MailboxSendNode(int stageId, DataSchema dataSchema, List<PlanNode> inputs
_sort = sort;
}
+ public MailboxSendNode(int stageId, DataSchema dataSchema, List<PlanNode> inputs,
+ int receiverStage, PinotRelExchangeType exchangeType,
+ RelDistribution.Type distributionType, @Nullable List<Integer> keys, boolean prePartitioned,
+ @Nullable List<RelFieldCollation> collations, boolean sort) {
+ this(stageId, dataSchema, inputs, toBitSet(receiverStage), exchangeType, distributionType, keys, prePartitioned,
+ collations, sort);
+ }
+
+ private static BitSet toBitSet(int receiverStage) {
+ BitSet bitSet = new BitSet(receiverStage + 1);
+ bitSet.set(receiverStage);
+ return bitSet;
+ }
+
+ public MailboxSendNode(int stageId, DataSchema dataSchema, List<PlanNode> inputs,
+ PinotRelExchangeType exchangeType, RelDistribution.Type distributionType, @Nullable List<Integer> keys,
+ boolean prePartitioned, @Nullable List<RelFieldCollation> collations, boolean sort) {
+ this(stageId, dataSchema, inputs, null, exchangeType, distributionType, keys, prePartitioned, collations, sort);
+ }
+
+ public BitSet getReceiverStages() {
+ Preconditions.checkState(_receiverStages != null && !_receiverStages.isEmpty(), "Receivers not set");
+ return _receiverStages;
+ }
+
+ @Deprecated
public int getReceiverStageId() {
- return _receiverStageId;
+ Preconditions.checkState(_receiverStages != null && !_receiverStages.isEmpty(), "Receivers not set");
Review Comment:
Looks like a ton of tests are failing here currently.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] [Spool] Introduce stage replacer and change send nodes to be able to send to more than one stage [pinot]
Posted by "yashmayya (via GitHub)" <gi...@apache.org>.
yashmayya commented on code in PR #14495:
URL: https://github.com/apache/pinot/pull/14495#discussion_r1850339957
##########
pinot-query-planner/src/main/java/org/apache/pinot/query/planner/plannode/MailboxSendNode.java:
##########
@@ -28,7 +30,7 @@
public class MailboxSendNode extends BasePlanNode {
- private final int _receiverStageId;
+ private final BitSet _receiverStages;
Review Comment:
> Originally I though about just changing the current field to add repeat, but it looks like that is problematic on numbers (see https://protobuf.dev/programming-guides/proto3/#updating).
Ah, that's unfortunate, that's exactly what I was curious about.
> Case 1. is impossible as long as we update servers before brokers (which AFAIK is the recommendation)
The recommended order is controller -> broker -> server (https://docs.pinot.apache.org/operators/operating-pinot/upgrading-pinot-cluster).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] [Spool] Introduce stage replacer and change send nodes to be able to send to more than one stage [pinot]
Posted by "vrajat (via GitHub)" <gi...@apache.org>.
vrajat commented on code in PR #14495:
URL: https://github.com/apache/pinot/pull/14495#discussion_r1852488777
##########
pinot-query-planner/src/main/java/org/apache/pinot/query/planner/plannode/MailboxReceiveNode.java:
##########
@@ -57,6 +57,8 @@ public MailboxReceiveNode(int stageId, DataSchema dataSchema, int senderStageId,
}
public int getSenderStageId() {
+ assert _sender == null || _sender.getStageId() == _senderStageId
Review Comment:
Oh - misread the statement. I was thinking `!=`. nvm
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] [Spool] Introduce stage replacer and change send nodes to be able to send to more than one stage [pinot]
Posted by "gortiz (via GitHub)" <gi...@apache.org>.
gortiz commented on code in PR #14495:
URL: https://github.com/apache/pinot/pull/14495#discussion_r1850256883
##########
pinot-query-planner/src/main/java/org/apache/pinot/query/planner/plannode/MailboxSendNode.java:
##########
@@ -28,7 +30,7 @@
public class MailboxSendNode extends BasePlanNode {
- private final int _receiverStageId;
+ private final BitSet _receiverStages;
Review Comment:
We can discuss about that in the next PR, but the idea is to keep the current proto field as it is and add a new one that will be a `repeat int32` with a new field id. Originally I though about just changing the current field to add `repeat`, but it looks like that is problematic on numbers (see https://protobuf.dev/programming-guides/proto3/#updating).
Therefore the old field will be kept as it is (storing just a single value) and the new one will be a list. There are two interesting cases:
1. An updated broker sends a multi sender to a not updated server.
2. A not updated broker sends a single sender to an updated server.
Case 1. is impossible as long as we update servers before brokers (which AFAIK is the recommendation). Case 2 can be solved in the code.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] [Spool] Introduce stage replacer and change send nodes to be able to send to more than one stage [pinot]
Posted by "gortiz (via GitHub)" <gi...@apache.org>.
gortiz commented on code in PR #14495:
URL: https://github.com/apache/pinot/pull/14495#discussion_r1850587411
##########
pinot-query-planner/src/main/java/org/apache/pinot/query/planner/plannode/MailboxSendNode.java:
##########
@@ -50,8 +53,42 @@ public MailboxSendNode(int stageId, DataSchema dataSchema, List<PlanNode> inputs
_sort = sort;
}
+ public MailboxSendNode(int stageId, DataSchema dataSchema, List<PlanNode> inputs,
+ int receiverStage, PinotRelExchangeType exchangeType,
+ RelDistribution.Type distributionType, @Nullable List<Integer> keys, boolean prePartitioned,
+ @Nullable List<RelFieldCollation> collations, boolean sort) {
+ this(stageId, dataSchema, inputs, toBitSet(receiverStage), exchangeType, distributionType, keys, prePartitioned,
+ collations, sort);
+ }
+
+ private static BitSet toBitSet(int receiverStage) {
+ BitSet bitSet = new BitSet(receiverStage + 1);
+ bitSet.set(receiverStage);
+ return bitSet;
+ }
+
+ public MailboxSendNode(int stageId, DataSchema dataSchema, List<PlanNode> inputs,
+ PinotRelExchangeType exchangeType, RelDistribution.Type distributionType, @Nullable List<Integer> keys,
+ boolean prePartitioned, @Nullable List<RelFieldCollation> collations, boolean sort) {
+ this(stageId, dataSchema, inputs, null, exchangeType, distributionType, keys, prePartitioned, collations, sort);
+ }
+
+ public BitSet getReceiverStages() {
+ Preconditions.checkState(!_receiverStages.isEmpty(), "Receivers not set");
+ return (BitSet) _receiverStages.clone();
+ }
Review Comment:
Oh! I misread you. I though the code wasn't returning a copy and you wanted to say that the returned BitSet should not be modified.
About your actual comment: I don't think that needs to be commented. Specifically, I think that should be the default. Even if we don't do defensive copies, by default we should never modify structures returned by objects. It is a bit like annotating with `@NotNull`. In the same way we assume that everything is not nullable by default, we should assume we should not change mutable properties by default.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] [Spool] Introduce stage replacer and change send nodes to be able to send to more than one stage [pinot]
Posted by "gortiz (via GitHub)" <gi...@apache.org>.
gortiz commented on PR #14495:
URL: https://github.com/apache/pinot/pull/14495#issuecomment-2486065520
cc @bziobrowski
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] [Spool] Introduce stage replacer and change send nodes to be able to send to more than one stage [pinot]
Posted by "gortiz (via GitHub)" <gi...@apache.org>.
gortiz commented on code in PR #14495:
URL: https://github.com/apache/pinot/pull/14495#discussion_r1850344673
##########
pinot-query-planner/src/main/java/org/apache/pinot/query/planner/plannode/MailboxSendNode.java:
##########
@@ -28,7 +30,7 @@
public class MailboxSendNode extends BasePlanNode {
- private final int _receiverStageId;
+ private final BitSet _receiverStages;
Review Comment:
I didn't know that! It sounds a bit strange to me, but anyway, as long as we do not activate spools by default during the migration, the result should be fine anyway
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] [Spool] Introduce stage replacer and change send nodes to be able to send to more than one stage [pinot]
Posted by "gortiz (via GitHub)" <gi...@apache.org>.
gortiz merged PR #14495:
URL: https://github.com/apache/pinot/pull/14495
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] [Spool] Introduce stage replacer and change send nodes to be able to send to more than one stage [pinot]
Posted by "gortiz (via GitHub)" <gi...@apache.org>.
gortiz commented on code in PR #14495:
URL: https://github.com/apache/pinot/pull/14495#discussion_r1851573201
##########
pinot-query-planner/src/test/java/org/apache/pinot/query/planner/logical/StagesTestBase.java:
##########
@@ -229,4 +260,127 @@ default P build(int stageId) {
return build(stageId, null, null);
}
}
+
+ public class Spool {
+ private final int _senderStageId;
+ private final Set<SpoolReceiverBuilder> _receiverBuilder = Collections.newSetFromMap(new IdentityHashMap<>());
+ private MailboxSendNode _sender;
+ private final SimpleChildBuilder<? extends PlanNode> _childBuilder;
+
+ public Spool(int senderStageId, SimpleChildBuilder<? extends PlanNode> childBuilder) {
+ _senderStageId = senderStageId;
+ _childBuilder = childBuilder;
Review Comment:
Yes, I'm adding javadoc to make this class easier to understand
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] [Spool] Introduce stage replacer and change send nodes to be able to send to more than one stage [pinot]
Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #14495:
URL: https://github.com/apache/pinot/pull/14495#issuecomment-2486180736
## [Codecov](https://app.codecov.io/gh/apache/pinot/pull/14495?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
Attention: Patch coverage is `0%` with `73 lines` in your changes missing coverage. Please review.
> Project coverage is 34.04%. Comparing base [(`59551e4`)](https://app.codecov.io/gh/apache/pinot/commit/59551e45224f1535c4863fd577622b37366ccc97?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) to head [(`05b894e`)](https://app.codecov.io/gh/apache/pinot/commit/05b894ec093e9548c689b624f05186d8eddaae75?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
> Report is 1358 commits behind head on master.
| [Files with missing lines](https://app.codecov.io/gh/apache/pinot/pull/14495?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Patch % | Lines |
|---|---|---|
| [.../pinot/query/planner/plannode/PlanNodeVisitor.java](https://app.codecov.io/gh/apache/pinot/pull/14495?src=pr&el=tree&filepath=pinot-query-planner%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fpinot%2Fquery%2Fplanner%2Fplannode%2FPlanNodeVisitor.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-cGlub3QtcXVlcnktcGxhbm5lci9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvcXVlcnkvcGxhbm5lci9wbGFubm9kZS9QbGFuTm9kZVZpc2l0b3IuamF2YQ==) | 0.00% | [27 Missing :warning: ](https://app.codecov.io/gh/apache/pinot/pull/14495?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [.../pinot/query/planner/plannode/MailboxSendNode.java](https://app.codecov.io/gh/apache/pinot/pull/14495?src=pr&el=tree&filepath=pinot-query-planner%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fpinot%2Fquery%2Fplanner%2Fplannode%2FMailboxSendNode.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-cGlub3QtcXVlcnktcGxhbm5lci9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvcXVlcnkvcGxhbm5lci9wbGFubm9kZS9NYWlsYm94U2VuZE5vZGUuamF2YQ==) | 0.00% | [24 Missing :warning: ](https://app.codecov.io/gh/apache/pinot/pull/14495?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...uery/planner/logical/EquivalentStagesReplacer.java](https://app.codecov.io/gh/apache/pinot/pull/14495?src=pr&el=tree&filepath=pinot-query-planner%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fpinot%2Fquery%2Fplanner%2Flogical%2FEquivalentStagesReplacer.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-cGlub3QtcXVlcnktcGxhbm5lci9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvcXVlcnkvcGxhbm5lci9sb2dpY2FsL0VxdWl2YWxlbnRTdGFnZXNSZXBsYWNlci5qYXZh) | 0.00% | [12 Missing :warning: ](https://app.codecov.io/gh/apache/pinot/pull/14495?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...not/query/planner/plannode/MailboxReceiveNode.java](https://app.codecov.io/gh/apache/pinot/pull/14495?src=pr&el=tree&filepath=pinot-query-planner%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fpinot%2Fquery%2Fplanner%2Fplannode%2FMailboxReceiveNode.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-cGlub3QtcXVlcnktcGxhbm5lci9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvcXVlcnkvcGxhbm5lci9wbGFubm9kZS9NYWlsYm94UmVjZWl2ZU5vZGUuamF2YQ==) | 0.00% | [5 Missing :warning: ](https://app.codecov.io/gh/apache/pinot/pull/14495?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [.../query/planner/logical/EquivalentStagesFinder.java](https://app.codecov.io/gh/apache/pinot/pull/14495?src=pr&el=tree&filepath=pinot-query-planner%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fpinot%2Fquery%2Fplanner%2Flogical%2FEquivalentStagesFinder.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-cGlub3QtcXVlcnktcGxhbm5lci9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvcXVlcnkvcGxhbm5lci9sb2dpY2FsL0VxdWl2YWxlbnRTdGFnZXNGaW5kZXIuamF2YQ==) | 0.00% | [2 Missing :warning: ](https://app.codecov.io/gh/apache/pinot/pull/14495?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...che/pinot/query/planner/logical/GroupedStages.java](https://app.codecov.io/gh/apache/pinot/pull/14495?src=pr&el=tree&filepath=pinot-query-planner%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fpinot%2Fquery%2Fplanner%2Flogical%2FGroupedStages.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-cGlub3QtcXVlcnktcGxhbm5lci9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvcXVlcnkvcGxhbm5lci9sb2dpY2FsL0dyb3VwZWRTdGFnZXMuamF2YQ==) | 0.00% | [1 Missing :warning: ](https://app.codecov.io/gh/apache/pinot/pull/14495?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...uery/planner/logical/PinotLogicalQueryPlanner.java](https://app.codecov.io/gh/apache/pinot/pull/14495?src=pr&el=tree&filepath=pinot-query-planner%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fpinot%2Fquery%2Fplanner%2Flogical%2FPinotLogicalQueryPlanner.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-cGlub3QtcXVlcnktcGxhbm5lci9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvcXVlcnkvcGxhbm5lci9sb2dpY2FsL1Bpbm90TG9naWNhbFF1ZXJ5UGxhbm5lci5qYXZh) | 0.00% | [1 Missing :warning: ](https://app.codecov.io/gh/apache/pinot/pull/14495?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...he/pinot/query/planner/logical/PlanFragmenter.java](https://app.codecov.io/gh/apache/pinot/pull/14495?src=pr&el=tree&filepath=pinot-query-planner%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fpinot%2Fquery%2Fplanner%2Flogical%2FPlanFragmenter.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-cGlub3QtcXVlcnktcGxhbm5lci9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvcXVlcnkvcGxhbm5lci9sb2dpY2FsL1BsYW5GcmFnbWVudGVyLmphdmE=) | 0.00% | [1 Missing :warning: ](https://app.codecov.io/gh/apache/pinot/pull/14495?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
> :exclamation: There is a different number of reports uploaded between BASE (59551e4) and HEAD (05b894e). Click for more details.
>
> <details><summary>HEAD has 53 uploads less than BASE</summary>
>
>| Flag | BASE (59551e4) | HEAD (05b894e) |
>|------|------|------|
>|integration|7|0|
>|integration2|3|0|
>|temurin|12|1|
>|java-21|7|1|
>|skip-bytebuffers-true|3|0|
>|skip-bytebuffers-false|7|1|
>|unittests|5|1|
>|unittests1|2|0|
>|java-11|5|0|
>|unittests2|3|1|
>|integration1|2|0|
>|custom-integration1|2|0|
></details>
<details><summary>Additional details and impacted files</summary>
```diff
@@ Coverage Diff @@
## master #14495 +/- ##
=============================================
- Coverage 61.75% 34.04% -27.71%
- Complexity 207 769 +562
=============================================
Files 2436 2668 +232
Lines 133233 146492 +13259
Branches 20636 22474 +1838
=============================================
- Hits 82274 49874 -32400
- Misses 44911 92577 +47666
+ Partials 6048 4041 -2007
```
| [Flag](https://app.codecov.io/gh/apache/pinot/pull/14495/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [custom-integration1](https://app.codecov.io/gh/apache/pinot/pull/14495/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [integration](https://app.codecov.io/gh/apache/pinot/pull/14495/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [integration1](https://app.codecov.io/gh/apache/pinot/pull/14495/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [integration2](https://app.codecov.io/gh/apache/pinot/pull/14495/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [java-11](https://app.codecov.io/gh/apache/pinot/pull/14495/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [java-21](https://app.codecov.io/gh/apache/pinot/pull/14495/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `34.04% <0.00%> (-27.58%)` | :arrow_down: |
| [skip-bytebuffers-false](https://app.codecov.io/gh/apache/pinot/pull/14495/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `34.04% <0.00%> (-27.71%)` | :arrow_down: |
| [skip-bytebuffers-true](https://app.codecov.io/gh/apache/pinot/pull/14495/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [temurin](https://app.codecov.io/gh/apache/pinot/pull/14495/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `34.04% <0.00%> (-27.71%)` | :arrow_down: |
| [unittests](https://app.codecov.io/gh/apache/pinot/pull/14495/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `34.04% <0.00%> (-27.71%)` | :arrow_down: |
| [unittests1](https://app.codecov.io/gh/apache/pinot/pull/14495/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [unittests2](https://app.codecov.io/gh/apache/pinot/pull/14495/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `34.04% <0.00%> (+6.31%)` | :arrow_up: |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#carryforward-flags-in-the-pull-request-comment) to find out more.
</details>
[:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/apache/pinot/pull/14495?dropdown=coverage&src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
:loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
----
🚨 Try these New Features:
- [Flaky Tests Detection](https://docs.codecov.com/docs/test-result-ingestion-beta) - Detect and resolve failed and flaky tests
- [JS Bundle Analysis](https://docs.codecov.com/docs/javascript-bundle-analysis) - Avoid shipping oversized bundles
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] [Spool] Introduce stage replacer and change send nodes to be able to send to more than one stage [pinot]
Posted by "gortiz (via GitHub)" <gi...@apache.org>.
gortiz commented on code in PR #14495:
URL: https://github.com/apache/pinot/pull/14495#discussion_r1850270296
##########
pinot-query-planner/src/main/java/org/apache/pinot/query/planner/plannode/MailboxSendNode.java:
##########
@@ -50,8 +53,42 @@ public MailboxSendNode(int stageId, DataSchema dataSchema, List<PlanNode> inputs
_sort = sort;
}
+ public MailboxSendNode(int stageId, DataSchema dataSchema, List<PlanNode> inputs,
+ int receiverStage, PinotRelExchangeType exchangeType,
+ RelDistribution.Type distributionType, @Nullable List<Integer> keys, boolean prePartitioned,
+ @Nullable List<RelFieldCollation> collations, boolean sort) {
+ this(stageId, dataSchema, inputs, toBitSet(receiverStage), exchangeType, distributionType, keys, prePartitioned,
+ collations, sort);
+ }
+
+ private static BitSet toBitSet(int receiverStage) {
+ BitSet bitSet = new BitSet(receiverStage + 1);
+ bitSet.set(receiverStage);
+ return bitSet;
+ }
+
+ public MailboxSendNode(int stageId, DataSchema dataSchema, List<PlanNode> inputs,
+ PinotRelExchangeType exchangeType, RelDistribution.Type distributionType, @Nullable List<Integer> keys,
+ boolean prePartitioned, @Nullable List<RelFieldCollation> collations, boolean sort) {
+ this(stageId, dataSchema, inputs, null, exchangeType, distributionType, keys, prePartitioned, collations, sort);
+ }
+
+ public BitSet getReceiverStages() {
+ Preconditions.checkState(!_receiverStages.isEmpty(), "Receivers not set");
+ return (BitSet) _receiverStages.clone();
+ }
+
+ @Deprecated
public int getReceiverStageId() {
- return _receiverStageId;
+ Preconditions.checkState(!_receiverStages.isEmpty(), "Receivers not set");
+ return _receiverStages.nextSetBit(0);
+ }
+
+ public void addReceiver(MailboxReceiveNode node) {
+ if (_receiverStages.get(node.getStageId())) {
+ throw new IllegalStateException("Receiver already added: " + node.getStageId());
+ }
+ _receiverStages.set(node.getStageId());
Review Comment:
Yes, I've tried to change that by adding a _receiverId_, but that makes things more complex. That is why I decided to do not support the _self join case_ in this PR. We can add it later if needed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] [Spool] Introduce stage replacer and change send nodes to be able to send to more than one stage [pinot]
Posted by "gortiz (via GitHub)" <gi...@apache.org>.
gortiz commented on code in PR #14495:
URL: https://github.com/apache/pinot/pull/14495#discussion_r1852342867
##########
pinot-query-planner/src/main/java/org/apache/pinot/query/planner/plannode/MailboxReceiveNode.java:
##########
@@ -57,6 +57,8 @@ public MailboxReceiveNode(int stageId, DataSchema dataSchema, int senderStageId,
}
public int getSenderStageId() {
+ assert _sender == null || _sender.getStageId() == _senderStageId
Review Comment:
I may be wrong, but in `setSender` there is now way this may fail (given we set both). This is just more a reminder of the invariant for readers.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] [Spool] Introduce stage replacer and change send nodes to be able to send to more than one stage [pinot]
Posted by "gortiz (via GitHub)" <gi...@apache.org>.
gortiz commented on code in PR #14495:
URL: https://github.com/apache/pinot/pull/14495#discussion_r1849804457
##########
pinot-query-planner/src/main/java/org/apache/pinot/query/planner/plannode/MailboxSendNode.java:
##########
@@ -50,8 +55,50 @@ public MailboxSendNode(int stageId, DataSchema dataSchema, List<PlanNode> inputs
_sort = sort;
}
+ public MailboxSendNode(int stageId, DataSchema dataSchema, List<PlanNode> inputs,
+ int receiverStage, PinotRelExchangeType exchangeType,
+ RelDistribution.Type distributionType, @Nullable List<Integer> keys, boolean prePartitioned,
+ @Nullable List<RelFieldCollation> collations, boolean sort) {
+ this(stageId, dataSchema, inputs, toBitSet(receiverStage), exchangeType, distributionType, keys, prePartitioned,
+ collations, sort);
+ }
+
+ private static BitSet toBitSet(int receiverStage) {
+ BitSet bitSet = new BitSet(receiverStage + 1);
+ bitSet.set(receiverStage);
+ return bitSet;
+ }
+
+ public MailboxSendNode(int stageId, DataSchema dataSchema, List<PlanNode> inputs,
+ PinotRelExchangeType exchangeType, RelDistribution.Type distributionType, @Nullable List<Integer> keys,
+ boolean prePartitioned, @Nullable List<RelFieldCollation> collations, boolean sort) {
+ this(stageId, dataSchema, inputs, null, exchangeType, distributionType, keys, prePartitioned, collations, sort);
+ }
+
+ public BitSet getReceiverStages() {
+ Preconditions.checkState(_receiverStages != null && !_receiverStages.isEmpty(), "Receivers not set");
+ return _receiverStages;
+ }
+
+ @Deprecated
public int getReceiverStageId() {
- return _receiverStageId;
+ Preconditions.checkState(_receiverStages != null && !_receiverStages.isEmpty(), "Receivers not set");
Review Comment:
Yep, there was naive error in PinotLogicalQueryPlanner. I've just fixed it in #1597f5b
##########
pinot-query-planner/src/main/java/org/apache/pinot/query/planner/plannode/MailboxSendNode.java:
##########
@@ -50,8 +55,50 @@ public MailboxSendNode(int stageId, DataSchema dataSchema, List<PlanNode> inputs
_sort = sort;
}
+ public MailboxSendNode(int stageId, DataSchema dataSchema, List<PlanNode> inputs,
+ int receiverStage, PinotRelExchangeType exchangeType,
+ RelDistribution.Type distributionType, @Nullable List<Integer> keys, boolean prePartitioned,
+ @Nullable List<RelFieldCollation> collations, boolean sort) {
+ this(stageId, dataSchema, inputs, toBitSet(receiverStage), exchangeType, distributionType, keys, prePartitioned,
+ collations, sort);
+ }
+
+ private static BitSet toBitSet(int receiverStage) {
+ BitSet bitSet = new BitSet(receiverStage + 1);
+ bitSet.set(receiverStage);
+ return bitSet;
+ }
+
+ public MailboxSendNode(int stageId, DataSchema dataSchema, List<PlanNode> inputs,
+ PinotRelExchangeType exchangeType, RelDistribution.Type distributionType, @Nullable List<Integer> keys,
+ boolean prePartitioned, @Nullable List<RelFieldCollation> collations, boolean sort) {
+ this(stageId, dataSchema, inputs, null, exchangeType, distributionType, keys, prePartitioned, collations, sort);
+ }
+
+ public BitSet getReceiverStages() {
+ Preconditions.checkState(_receiverStages != null && !_receiverStages.isEmpty(), "Receivers not set");
+ return _receiverStages;
+ }
+
+ @Deprecated
public int getReceiverStageId() {
- return _receiverStageId;
+ Preconditions.checkState(_receiverStages != null && !_receiverStages.isEmpty(), "Receivers not set");
Review Comment:
Yep, there was naive error in PinotLogicalQueryPlanner. I've just fixed it in 1597f5b
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] [Spool] Introduce stage replacer and change send nodes to be able to send to more than one stage [pinot]
Posted by "gortiz (via GitHub)" <gi...@apache.org>.
gortiz commented on code in PR #14495:
URL: https://github.com/apache/pinot/pull/14495#discussion_r1850241625
##########
pinot-query-planner/src/main/java/org/apache/pinot/query/planner/logical/EquivalentStagesReplacer.java:
##########
@@ -0,0 +1,79 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.query.planner.logical;
+
+import org.apache.pinot.query.planner.plannode.MailboxReceiveNode;
+import org.apache.pinot.query.planner.plannode.MailboxSendNode;
+import org.apache.pinot.query.planner.plannode.PlanNode;
+import org.apache.pinot.query.planner.plannode.PlanNodeVisitor;
+
+
+/**
+ * EquivalentStageReplacer is used to replace equivalent stages in the query plan.
+ *
+ * Given a {@link org.apache.pinot.query.planner.plannode.PlanNode} and a
+ * {@link GroupedStages}, modifies the plan node to replace equivalent stages.
+ *
+ * For each {@link MailboxReceiveNode} in the plan, if the sender is not the leader of the group,
+ * replaces the sender with the leader.
+ * The leader is also updated to include the receiver in its list of receivers.
+ */
+public class EquivalentStagesReplacer {
+ private EquivalentStagesReplacer() {
+ }
+
+ /**
+ * Replaces the equivalent stages in the query plan.
+ *
+ * @param root Root plan node
+ * @param equivalentStages Equivalent stages
+ */
+ public static void replaceEquivalentStages(PlanNode root, GroupedStages equivalentStages) {
+ root.visit(Replacer.INSTANCE, equivalentStages);
+ }
+
+ private static class Replacer extends PlanNodeVisitor.DepthFirstVisitor<Void, GroupedStages> {
+ private static final Replacer INSTANCE = new Replacer();
+
+ private Replacer() {
+ }
+
+ @Override
+ public Void visitMailboxReceive(MailboxReceiveNode node, GroupedStages equivalenceGroups) {
+ MailboxSendNode sender = node.getSender();
+ MailboxSendNode leader = equivalenceGroups.getGroup(sender).first();
+ if (canSubstitute(sender, leader)) {
+ // we don't want to visit the children of the node given it is going to be pruned
+ node.setSender(leader);
+ leader.addReceiver(node);
+ } else {
+ visitMailboxSend(leader, equivalenceGroups);
+ }
+ return null;
+ }
+
+ private boolean canSubstitute(MailboxSendNode actualSender, MailboxSendNode leader) {
+ return actualSender != leader // we don't need to replace the leader with itself
+ // the leader is already sending to this stage. Given we don't have the ability to send to multiple
+ // receivers in the same stage, we cannot optimize this case right now.
+ // If this case seems to be useful, it can be supported in the future.
+ && !leader.getReceiverStages().intersects(actualSender.getReceiverStages());
+ }
Review Comment:
Yes, but also recursive cases like:
- join <-- it also applies here
- left: join
- left: read T1
- right: read T2
- right: join
- left: read T1
- right: read T2
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] [Spool] Introduce stage replacer and change send nodes to be able to send to more than one stage [pinot]
Posted by "gortiz (via GitHub)" <gi...@apache.org>.
gortiz commented on code in PR #14495:
URL: https://github.com/apache/pinot/pull/14495#discussion_r1850244343
##########
pinot-query-planner/src/main/java/org/apache/pinot/query/planner/plannode/MailboxSendNode.java:
##########
@@ -50,8 +53,42 @@ public MailboxSendNode(int stageId, DataSchema dataSchema, List<PlanNode> inputs
_sort = sort;
}
+ public MailboxSendNode(int stageId, DataSchema dataSchema, List<PlanNode> inputs,
+ int receiverStage, PinotRelExchangeType exchangeType,
+ RelDistribution.Type distributionType, @Nullable List<Integer> keys, boolean prePartitioned,
+ @Nullable List<RelFieldCollation> collations, boolean sort) {
+ this(stageId, dataSchema, inputs, toBitSet(receiverStage), exchangeType, distributionType, keys, prePartitioned,
+ collations, sort);
+ }
+
+ private static BitSet toBitSet(int receiverStage) {
+ BitSet bitSet = new BitSet(receiverStage + 1);
+ bitSet.set(receiverStage);
+ return bitSet;
+ }
+
+ public MailboxSendNode(int stageId, DataSchema dataSchema, List<PlanNode> inputs,
+ PinotRelExchangeType exchangeType, RelDistribution.Type distributionType, @Nullable List<Integer> keys,
+ boolean prePartitioned, @Nullable List<RelFieldCollation> collations, boolean sort) {
+ this(stageId, dataSchema, inputs, null, exchangeType, distributionType, keys, prePartitioned, collations, sort);
+ }
+
+ public BitSet getReceiverStages() {
+ Preconditions.checkState(!_receiverStages.isEmpty(), "Receivers not set");
+ return (BitSet) _receiverStages.clone();
+ }
Review Comment:
I'm not a fan of leaking these constraints, even if we document it.
In the next PR (spools actually used in queries) I changed this code to return a `clone()`. I can apply this change in this PR as well.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] [Spool] Introduce stage replacer and change send nodes to be able to send to more than one stage [pinot]
Posted by "gortiz (via GitHub)" <gi...@apache.org>.
gortiz commented on code in PR #14495:
URL: https://github.com/apache/pinot/pull/14495#discussion_r1851545503
##########
pinot-query-planner/src/test/java/org/apache/pinot/query/planner/logical/EquivalentStagesFinderTest.java:
##########
@@ -109,7 +138,7 @@ public void differentHintsImplyNotEquivalent() {
}
@Test
- public void differentHintsOneNullImplyNotEquivalent() {
+ public void differentHintsOneNullBreakEquivalent() {
Review Comment:
```suggestion
public void differentHintsOneNullBreakEquivalence() {
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] [Spool] Introduce stage replacer and change send nodes to be able to send to more than one stage [pinot]
Posted by "gortiz (via GitHub)" <gi...@apache.org>.
gortiz commented on code in PR #14495:
URL: https://github.com/apache/pinot/pull/14495#discussion_r1850602515
##########
pinot-query-planner/src/main/java/org/apache/pinot/query/planner/plannode/MailboxSendNode.java:
##########
@@ -50,8 +53,42 @@ public MailboxSendNode(int stageId, DataSchema dataSchema, List<PlanNode> inputs
_sort = sort;
}
+ public MailboxSendNode(int stageId, DataSchema dataSchema, List<PlanNode> inputs,
+ int receiverStage, PinotRelExchangeType exchangeType,
+ RelDistribution.Type distributionType, @Nullable List<Integer> keys, boolean prePartitioned,
+ @Nullable List<RelFieldCollation> collations, boolean sort) {
+ this(stageId, dataSchema, inputs, toBitSet(receiverStage), exchangeType, distributionType, keys, prePartitioned,
+ collations, sort);
+ }
+
+ private static BitSet toBitSet(int receiverStage) {
+ BitSet bitSet = new BitSet(receiverStage + 1);
+ bitSet.set(receiverStage);
+ return bitSet;
+ }
+
+ public MailboxSendNode(int stageId, DataSchema dataSchema, List<PlanNode> inputs,
+ PinotRelExchangeType exchangeType, RelDistribution.Type distributionType, @Nullable List<Integer> keys,
+ boolean prePartitioned, @Nullable List<RelFieldCollation> collations, boolean sort) {
+ this(stageId, dataSchema, inputs, null, exchangeType, distributionType, keys, prePartitioned, collations, sort);
+ }
+
+ public BitSet getReceiverStages() {
+ Preconditions.checkState(!_receiverStages.isEmpty(), "Receivers not set");
+ return (BitSet) _receiverStages.clone();
+ }
Review Comment:
In my next push I'll remove this method and substitute it with a more specific one that will be easier to use and won't be affected by this undefined external mutability problem
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] [Spool] Introduce stage replacer and change send nodes to be able to send to more than one stage [pinot]
Posted by "gortiz (via GitHub)" <gi...@apache.org>.
gortiz commented on code in PR #14495:
URL: https://github.com/apache/pinot/pull/14495#discussion_r1850580654
##########
pinot-query-planner/src/main/java/org/apache/pinot/query/planner/logical/EquivalentStagesReplacer.java:
##########
@@ -0,0 +1,79 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.query.planner.logical;
+
+import org.apache.pinot.query.planner.plannode.MailboxReceiveNode;
+import org.apache.pinot.query.planner.plannode.MailboxSendNode;
+import org.apache.pinot.query.planner.plannode.PlanNode;
+import org.apache.pinot.query.planner.plannode.PlanNodeVisitor;
+
+
+/**
+ * EquivalentStageReplacer is used to replace equivalent stages in the query plan.
+ *
+ * Given a {@link org.apache.pinot.query.planner.plannode.PlanNode} and a
+ * {@link GroupedStages}, modifies the plan node to replace equivalent stages.
+ *
+ * For each {@link MailboxReceiveNode} in the plan, if the sender is not the leader of the group,
+ * replaces the sender with the leader.
+ * The leader is also updated to include the receiver in its list of receivers.
+ */
+public class EquivalentStagesReplacer {
+ private EquivalentStagesReplacer() {
+ }
+
+ /**
+ * Replaces the equivalent stages in the query plan.
+ *
+ * @param root Root plan node
+ * @param equivalentStages Equivalent stages
+ */
+ public static void replaceEquivalentStages(PlanNode root, GroupedStages equivalentStages) {
+ root.visit(Replacer.INSTANCE, equivalentStages);
+ }
+
+ private static class Replacer extends PlanNodeVisitor.DepthFirstVisitor<Void, GroupedStages> {
Review Comment:
The same happened to me. TBH the solution we were discussing initialy (the one that supports different distributions) wouldn't be that clean :laughing:
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] [Spool] Introduce stage replacer and change send nodes to be able to send to more than one stage [pinot]
Posted by "yashmayya (via GitHub)" <gi...@apache.org>.
yashmayya commented on code in PR #14495:
URL: https://github.com/apache/pinot/pull/14495#discussion_r1849912468
##########
pinot-query-planner/src/main/java/org/apache/pinot/query/planner/plannode/MailboxReceiveNode.java:
##########
@@ -126,7 +133,8 @@ public boolean equals(Object o) {
return false;
}
MailboxReceiveNode that = (MailboxReceiveNode) o;
- return _senderStageId == that._senderStageId && _sort == that._sort && _sortedOnSender == that._sortedOnSender
+ return getSenderStageId() == that.getSenderStageId() && Objects.equals(_sender, that._sender)
Review Comment:
Looks like this change is causing all the test cases in `PlanNodeSerDeTest` to fail because the sender is not serialized into the plan.
##########
pinot-query-planner/src/main/java/org/apache/pinot/query/planner/plannode/PlanNodeVisitor.java:
##########
@@ -117,96 +117,120 @@ protected boolean traverseStageBoundary() {
return true;
}
+ /**
+ * The method that is called by default to handle a node that does not have a specific visit method.
+ *
+ * This method can be overridden to provide a default behavior for all nodes.
+ *
+ * The returned value of this method is ignored by default
+ */
+ protected T preChildren(PlanNode node, C context) {
Review Comment:
Why not make this a `void` function then? Doesn't seem like there are any future planned use cases for the return value here either?
##########
pinot-query-planner/src/main/java/org/apache/pinot/query/planner/plannode/MailboxSendNode.java:
##########
@@ -28,7 +30,7 @@
public class MailboxSendNode extends BasePlanNode {
- private final int _receiverStageId;
+ private final BitSet _receiverStages;
Review Comment:
How are you planning to handle the corresponding plan serde changes in a compatible manner?
##########
pinot-query-planner/src/main/java/org/apache/pinot/query/planner/logical/EquivalentStagesReplacer.java:
##########
@@ -0,0 +1,79 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.query.planner.logical;
+
+import org.apache.pinot.query.planner.plannode.MailboxReceiveNode;
+import org.apache.pinot.query.planner.plannode.MailboxSendNode;
+import org.apache.pinot.query.planner.plannode.PlanNode;
+import org.apache.pinot.query.planner.plannode.PlanNodeVisitor;
+
+
+/**
+ * EquivalentStageReplacer is used to replace equivalent stages in the query plan.
+ *
+ * Given a {@link org.apache.pinot.query.planner.plannode.PlanNode} and a
+ * {@link GroupedStages}, modifies the plan node to replace equivalent stages.
+ *
+ * For each {@link MailboxReceiveNode} in the plan, if the sender is not the leader of the group,
+ * replaces the sender with the leader.
+ * The leader is also updated to include the receiver in its list of receivers.
+ */
+public class EquivalentStagesReplacer {
+ private EquivalentStagesReplacer() {
+ }
+
+ /**
+ * Replaces the equivalent stages in the query plan.
+ *
+ * @param root Root plan node
+ * @param equivalentStages Equivalent stages
+ */
+ public static void replaceEquivalentStages(PlanNode root, GroupedStages equivalentStages) {
+ root.visit(Replacer.INSTANCE, equivalentStages);
+ }
+
+ private static class Replacer extends PlanNodeVisitor.DepthFirstVisitor<Void, GroupedStages> {
Review Comment:
Nice, didn't expect the actual plan modification to be so simple and elegant!
##########
pinot-query-planner/src/main/java/org/apache/pinot/query/planner/plannode/MailboxSendNode.java:
##########
@@ -50,8 +53,42 @@ public MailboxSendNode(int stageId, DataSchema dataSchema, List<PlanNode> inputs
_sort = sort;
}
+ public MailboxSendNode(int stageId, DataSchema dataSchema, List<PlanNode> inputs,
+ int receiverStage, PinotRelExchangeType exchangeType,
+ RelDistribution.Type distributionType, @Nullable List<Integer> keys, boolean prePartitioned,
+ @Nullable List<RelFieldCollation> collations, boolean sort) {
+ this(stageId, dataSchema, inputs, toBitSet(receiverStage), exchangeType, distributionType, keys, prePartitioned,
+ collations, sort);
+ }
+
+ private static BitSet toBitSet(int receiverStage) {
+ BitSet bitSet = new BitSet(receiverStage + 1);
+ bitSet.set(receiverStage);
+ return bitSet;
+ }
+
+ public MailboxSendNode(int stageId, DataSchema dataSchema, List<PlanNode> inputs,
+ PinotRelExchangeType exchangeType, RelDistribution.Type distributionType, @Nullable List<Integer> keys,
+ boolean prePartitioned, @Nullable List<RelFieldCollation> collations, boolean sort) {
+ this(stageId, dataSchema, inputs, null, exchangeType, distributionType, keys, prePartitioned, collations, sort);
+ }
+
+ public BitSet getReceiverStages() {
+ Preconditions.checkState(!_receiverStages.isEmpty(), "Receivers not set");
+ return (BitSet) _receiverStages.clone();
+ }
+
+ @Deprecated
public int getReceiverStageId() {
- return _receiverStageId;
+ Preconditions.checkState(!_receiverStages.isEmpty(), "Receivers not set");
+ return _receiverStages.nextSetBit(0);
+ }
+
+ public void addReceiver(MailboxReceiveNode node) {
+ if (_receiverStages.get(node.getStageId())) {
+ throw new IllegalStateException("Receiver already added: " + node.getStageId());
+ }
+ _receiverStages.set(node.getStageId());
Review Comment:
This PR doesn't introduce the issue - but considering that a stage can have more than one mailbox receive node, doesn't it seem a little confusing to identify a receiver only by its stage ID?
##########
pinot-query-planner/src/main/java/org/apache/pinot/query/planner/logical/GroupedStages.java:
##########
@@ -77,6 +78,8 @@ public abstract SortedSet<MailboxSendNode> getGroup(MailboxSendNode stage)
*/
public abstract SortedSet<SortedSet<MailboxSendNode>> getGroups();
+ public abstract Set<MailboxSendNode> getStages();
Review Comment:
Small doc comment might be useful here to distinguish this method from `getLeaders` / `getGroups`.
##########
pinot-query-planner/src/main/java/org/apache/pinot/query/planner/logical/PlanFragmenter.java:
##########
@@ -150,6 +150,7 @@ public PlanNode visitExchange(ExchangeNode node, Context context) {
// Split the ExchangeNode to a MailboxReceiveNode and a MailboxSendNode, where MailboxReceiveNode is the leave node
// of the current PlanFragment, and MailboxSendNode is the root node of the next PlanFragment.
int receiverPlanFragmentId = context._currentPlanFragmentId;
+ int receivedIdInPlan = -1; // TODO: Change this
Review Comment:
What is this for?
##########
pinot-query-planner/src/main/java/org/apache/pinot/query/planner/logical/EquivalentStagesReplacer.java:
##########
@@ -0,0 +1,79 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.query.planner.logical;
+
+import org.apache.pinot.query.planner.plannode.MailboxReceiveNode;
+import org.apache.pinot.query.planner.plannode.MailboxSendNode;
+import org.apache.pinot.query.planner.plannode.PlanNode;
+import org.apache.pinot.query.planner.plannode.PlanNodeVisitor;
+
+
+/**
+ * EquivalentStageReplacer is used to replace equivalent stages in the query plan.
+ *
+ * Given a {@link org.apache.pinot.query.planner.plannode.PlanNode} and a
+ * {@link GroupedStages}, modifies the plan node to replace equivalent stages.
+ *
+ * For each {@link MailboxReceiveNode} in the plan, if the sender is not the leader of the group,
+ * replaces the sender with the leader.
+ * The leader is also updated to include the receiver in its list of receivers.
+ */
+public class EquivalentStagesReplacer {
+ private EquivalentStagesReplacer() {
+ }
+
+ /**
+ * Replaces the equivalent stages in the query plan.
+ *
+ * @param root Root plan node
+ * @param equivalentStages Equivalent stages
+ */
+ public static void replaceEquivalentStages(PlanNode root, GroupedStages equivalentStages) {
+ root.visit(Replacer.INSTANCE, equivalentStages);
+ }
+
+ private static class Replacer extends PlanNodeVisitor.DepthFirstVisitor<Void, GroupedStages> {
+ private static final Replacer INSTANCE = new Replacer();
+
+ private Replacer() {
+ }
+
+ @Override
+ public Void visitMailboxReceive(MailboxReceiveNode node, GroupedStages equivalenceGroups) {
+ MailboxSendNode sender = node.getSender();
+ MailboxSendNode leader = equivalenceGroups.getGroup(sender).first();
+ if (canSubstitute(sender, leader)) {
+ // we don't want to visit the children of the node given it is going to be pruned
+ node.setSender(leader);
+ leader.addReceiver(node);
+ } else {
+ visitMailboxSend(leader, equivalenceGroups);
+ }
+ return null;
+ }
+
+ private boolean canSubstitute(MailboxSendNode actualSender, MailboxSendNode leader) {
+ return actualSender != leader // we don't need to replace the leader with itself
+ // the leader is already sending to this stage. Given we don't have the ability to send to multiple
+ // receivers in the same stage, we cannot optimize this case right now.
+ // If this case seems to be useful, it can be supported in the future.
+ && !leader.getReceiverStages().intersects(actualSender.getReceiverStages());
+ }
Review Comment:
Is this for cases like self joins?
##########
pinot-query-planner/src/main/java/org/apache/pinot/query/planner/plannode/MailboxSendNode.java:
##########
@@ -50,8 +53,42 @@ public MailboxSendNode(int stageId, DataSchema dataSchema, List<PlanNode> inputs
_sort = sort;
}
+ public MailboxSendNode(int stageId, DataSchema dataSchema, List<PlanNode> inputs,
+ int receiverStage, PinotRelExchangeType exchangeType,
+ RelDistribution.Type distributionType, @Nullable List<Integer> keys, boolean prePartitioned,
+ @Nullable List<RelFieldCollation> collations, boolean sort) {
+ this(stageId, dataSchema, inputs, toBitSet(receiverStage), exchangeType, distributionType, keys, prePartitioned,
+ collations, sort);
+ }
+
+ private static BitSet toBitSet(int receiverStage) {
+ BitSet bitSet = new BitSet(receiverStage + 1);
+ bitSet.set(receiverStage);
+ return bitSet;
+ }
+
+ public MailboxSendNode(int stageId, DataSchema dataSchema, List<PlanNode> inputs,
+ PinotRelExchangeType exchangeType, RelDistribution.Type distributionType, @Nullable List<Integer> keys,
+ boolean prePartitioned, @Nullable List<RelFieldCollation> collations, boolean sort) {
+ this(stageId, dataSchema, inputs, null, exchangeType, distributionType, keys, prePartitioned, collations, sort);
+ }
+
+ public BitSet getReceiverStages() {
+ Preconditions.checkState(!_receiverStages.isEmpty(), "Receivers not set");
+ return (BitSet) _receiverStages.clone();
+ }
Review Comment:
nit: small doc comment clarifying that modifying the returned bitset doesn't affect this node's receiver stages might be useful.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] [Spool] Introduce stage replacer and change send nodes to be able to send to more than one stage [pinot]
Posted by "yashmayya (via GitHub)" <gi...@apache.org>.
yashmayya commented on code in PR #14495:
URL: https://github.com/apache/pinot/pull/14495#discussion_r1851411034
##########
pinot-query-planner/src/main/java/org/apache/pinot/query/planner/plannode/MailboxReceiveNode.java:
##########
@@ -93,6 +96,10 @@ public MailboxSendNode getSender() {
return _sender;
}
+ public void setSender(MailboxSendNode sender) {
+ _sender = sender;
+ }
Review Comment:
Why not also update `_senderStageId`? Seems like it'd be good to avoid mismatches between `_sender` and `_senderStageId`.
##########
pinot-query-planner/src/test/java/org/apache/pinot/query/planner/logical/EquivalentStagesFinderTest.java:
##########
@@ -68,6 +69,34 @@ public void sharedJoin() {
assertEquals(result.toString(), "[[0], [1, 2]]");
}
+ @Test
+ void sameBroadcastKeepEquivalence() {
+ when(
+ join(
+ exchange(1, tableScan("T1"))
+ .withDistributionType(RelDistribution.Type.RANDOM_DISTRIBUTED),
+ exchange(2, tableScan("T1"))
+ .withDistributionType(RelDistribution.Type.RANDOM_DISTRIBUTED)
+ )
+ );
+ GroupedStages groupedStages = EquivalentStagesFinder.findEquivalentStages(stage(0));
+ assertEquals(groupedStages.toString(), "[[0], [1, 2]]");
+ }
+
+ @Test
+ void differentBroadcastBreakEquivalence() {
Review Comment:
```suggestion
void differentDistributionBreakEquivalence() {
```
nit
##########
pinot-query-planner/src/test/java/org/apache/pinot/query/planner/logical/EquivalentStagesFinderTest.java:
##########
@@ -68,6 +69,34 @@ public void sharedJoin() {
assertEquals(result.toString(), "[[0], [1, 2]]");
}
+ @Test
+ void sameBroadcastKeepEquivalence() {
Review Comment:
```suggestion
void sameDistributionKeepEquivalence() {
```
nit
##########
pinot-query-planner/src/test/java/org/apache/pinot/query/planner/logical/EquivalentStagesReplacerTest.java:
##########
@@ -0,0 +1,146 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.query.planner.logical;
+
+import org.apache.pinot.query.planner.plannode.MailboxSendNode;
+import org.testng.annotations.Test;
+
+import static org.testng.Assert.*;
+
+
+public class EquivalentStagesReplacerTest extends StagesTestBase {
Review Comment:
The new code in `StagesTestBase` is pretty hard to read and understand but I think the complexity there is worth it considering how readable it makes these tests :)
##########
pinot-query-planner/src/test/java/org/apache/pinot/query/planner/logical/StagesTestBase.java:
##########
@@ -229,4 +260,127 @@ default P build(int stageId) {
return build(stageId, null, null);
}
}
+
+ public class Spool {
+ private final int _senderStageId;
+ private final Set<SpoolReceiverBuilder> _receiverBuilder = Collections.newSetFromMap(new IdentityHashMap<>());
+ private MailboxSendNode _sender;
+ private final SimpleChildBuilder<? extends PlanNode> _childBuilder;
+
+ public Spool(int senderStageId, SimpleChildBuilder<? extends PlanNode> childBuilder) {
+ _senderStageId = senderStageId;
+ _childBuilder = childBuilder;
Review Comment:
This is the builder for the input node to the sender node that is being spooled right?
##########
pinot-query-planner/src/test/java/org/apache/pinot/query/planner/logical/StagesTestBase.java:
##########
@@ -229,4 +260,127 @@ default P build(int stageId) {
return build(stageId, null, null);
}
}
+
+ public class Spool {
+ private final int _senderStageId;
+ private final Set<SpoolReceiverBuilder> _receiverBuilder = Collections.newSetFromMap(new IdentityHashMap<>());
+ private MailboxSendNode _sender;
+ private final SimpleChildBuilder<? extends PlanNode> _childBuilder;
+
+ public Spool(int senderStageId, SimpleChildBuilder<? extends PlanNode> childBuilder) {
+ _senderStageId = senderStageId;
+ _childBuilder = childBuilder;
+ }
+
+ public MailboxSendNode getSender() {
+ return _sender;
+ }
+
+ public SimpleChildBuilder<MailboxReceiveNode> newReceiver(
+ Function<SimpleChildBuilder<MailboxReceiveNode>, SimpleChildBuilder<MailboxReceiveNode>> customize) {
Review Comment:
What's the use case for this customization function?
##########
pinot-query-planner/src/test/java/org/apache/pinot/query/planner/logical/EquivalentStagesFinderTest.java:
##########
@@ -89,7 +118,7 @@ public void sameHintsDontBreakEquivalence() {
}
@Test
- public void differentHintsImplyNotEquivalent() {
+ public void differentHintsBreakEquivalent() {
Review Comment:
```suggestion
public void differentHintsBreakEquivalence() {
```
nit (similarly below)
##########
pinot-query-planner/src/test/java/org/apache/pinot/query/planner/logical/StagesTestBase.java:
##########
@@ -229,4 +260,127 @@ default P build(int stageId) {
return build(stageId, null, null);
}
}
+
+ public class Spool {
Review Comment:
nit: should be static?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] [Spool] Introduce stage replacer and change send nodes to be able to send to more than one stage [pinot]
Posted by "vrajat (via GitHub)" <gi...@apache.org>.
vrajat commented on code in PR #14495:
URL: https://github.com/apache/pinot/pull/14495#discussion_r1851950991
##########
pinot-query-planner/src/main/java/org/apache/pinot/query/planner/plannode/MailboxReceiveNode.java:
##########
@@ -57,6 +57,8 @@ public MailboxReceiveNode(int stageId, DataSchema dataSchema, int senderStageId,
}
public int getSenderStageId() {
+ assert _sender == null || _sender.getStageId() == _senderStageId
Review Comment:
Is it better to assert in `setSender` then there will be a trace of where it is set to null.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] [Spool] Introduce stage replacer and change send nodes to be able to send to more than one stage [pinot]
Posted by "gortiz (via GitHub)" <gi...@apache.org>.
gortiz commented on code in PR #14495:
URL: https://github.com/apache/pinot/pull/14495#discussion_r1851577117
##########
pinot-query-planner/src/test/java/org/apache/pinot/query/planner/logical/EquivalentStagesReplacerTest.java:
##########
@@ -0,0 +1,146 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.query.planner.logical;
+
+import org.apache.pinot.query.planner.plannode.MailboxSendNode;
+import org.testng.annotations.Test;
+
+import static org.testng.Assert.*;
+
+
+public class EquivalentStagesReplacerTest extends StagesTestBase {
Review Comment:
Yes, the spool class is a bit ugly and very state dependent, but after spending some time designing the DSL, all other solutions I found make tests more difficult to write and read.
I hope the documentation included in the next commits will make it easier to understand
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org