You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pinot.apache.org by "gortiz (via GitHub)" <gi...@apache.org> on 2024/08/01 15:12:09 UTC

[PR] Multi stage explain [pinot]

gortiz opened a new pull request, #13733:
URL: https://github.com/apache/pinot/pull/13733

   This PR introduces a new way to explain multi-stage queries in Pinot. 
   The main goal is to provide a more detailed explanation of the query execution plan, including information about
   the physical operators that are being used.
   
   By default, `explain plan for` will return the new plan. If you want to use the old plan you can use 
   `explain plan without implementation for`. This may be problematic, so we can discuss to introduce a new flag for this.
   The main reason to break the default behavior is that the new plan is more verbose actually what a user should expect
   when asking for _implementation_, at least following Calcite terminology.
   Alternatively we can change the syntax in the same way we already did with `explain physical plan for`.
   
   At architectural level, the new explain mode is closer to the one used in single stage.
   The broker parses and optimizes the query generating a logical plan, generating RelNodes.
   These nodes are transformed into PlanNodes as usual and sent to the servers.
   But instead of asking to execute the plan, the broker asks to explain it using a new protobuf endpoint.
   This new endpoint returns a list of PlanNodes.
   
   When the server receives the explain request, it analyzes the plan looking for leaf operators and creates single-stage
   operators as usual.
   There are two key differences with respect of the execution mode:
   1. The server tracks which PlanNodes have been converted into single-stage operators.
   2. The server does not execute the operator. Instead it calls a new introduced method `Operator.getOperatorInfo`, 
   which returns the same information returned by `Operator.explainPlan` but in POJOs.
   
   The server then convert these POJOs into PlanNodes and substitute the tracked PlanNodes with the new ones.
   Finally the new plan is sent back to the broker.
   
   In order to be able to introduce physical (aka index used, etc) information in the PlanNode, a new ExplainedPlanNode is
   created.
   These nodes are not meant to be translated into actual operators, but to be used to explain the query execution plan.
   When the broker receives the PlanNodes, it converts them back into a RelNode using a new class `PinotExplainedRelNode`.
   
   Then the broker substitutes the original logical RelNodes with the new ones returned by the servers.
   Finally, it explains the RelNode as expected in Calcite.
   
   The result can be see in the following pictures:
   
   Without implementation (similar to current explain)
   ![image](https://github.com/user-attachments/assets/761ee0d5-24d5-4f10-8d1c-d9c4a886ebef)
   
   
   With implementation:
   ![image](https://github.com/user-attachments/assets/13e96262-3ea4-4479-89a9-7c201dfe8993)
   
   
   The PR is still a work in progress, but it is already partially functional.
   
   - [ ] Combine different plans for segments (right now the first one is used)
       - Ideally we should _group by plan_ and count how many segments use that plan
   - [ ] Add a flag so that we can use to decide if we want to use the new plan or the old one by default
   - [ ] PlanNodeToRelConverted
     - [ ] Support Window
     - [ ] Support SetOp
     - [ ] Decide what to do with send and receive mailboxes
   - [ ] Support pipeline breaker
   - [ ] PinotTable.getRowType may not use TypeFactory
   - [ ] Try to simplify LeafStageTransferableBlockOperator.explain
   - [ ] Try to simplify QueryServer.explain (similar to what was done in QueryDispatcher)
   - [ ] Try to simplify QueryRunner.explain (similar to what was done in QueryDispatcher)
   - [ ] Resolve corner cases in MultiStageBrokerRequestHandler
   - [ ] Review ServerQueryExecutorV1Impl
   - [ ] Decide if we need to keep TransformationTraker
   - [ ] Decide if we need to keep ImplementationExplainUtils
   - [ ] Verify tests
   - [ ] Optional: Extend V1 to be able to use the new ExplainMode
   - [ ] Split PinotExplainedRelNode and PinotExplainedRelNode.Info


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


Re: [PR] Multi stage explain [pinot]

Posted by "gortiz (via GitHub)" <gi...@apache.org>.
gortiz commented on code in PR #13733:
URL: https://github.com/apache/pinot/pull/13733#discussion_r1738408737


##########
pinot-spi/src/main/java/org/apache/pinot/spi/utils/CommonConstants.java:
##########
@@ -1192,6 +1209,9 @@ public enum WindowOverFlowMode {
     public static class PlanVersions {
       public static final int V1 = 1;
     }
+
+    public static final String ASK_SERVERS_FOR_EXPLAIN_PLAN = "pinot.query.explain.ask.servers";
+    public static final boolean DEFAULT_ASK_SERVERS_FOR_EXPLAIN_PLAN = true;

Review Comment:
   Probably this should be changed to false in the initial version. I'm keeping it as true for a while to verify that all tests are running.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


Re: [PR] Multi stage explain [pinot]

Posted by "gortiz (via GitHub)" <gi...@apache.org>.
gortiz commented on code in PR #13733:
URL: https://github.com/apache/pinot/pull/13733#discussion_r1752102288


##########
pinot-core/src/main/java/org/apache/pinot/core/query/request/context/utils/QueryContextConverterUtils.java:
##########
@@ -155,11 +157,26 @@ public static QueryContext getQueryContext(PinotQuery pinotQuery) {
       }
     }
 
+    ExplainMode explainMode;
+    if (!pinotQuery.isExplain()) {
+      explainMode = ExplainMode.NONE;
+    } else if (isUsingV1(pinotQuery)) {
+      explainMode = ExplainMode.NODE;
+    } else {
+      explainMode = ExplainMode.DESCRIPTION;
+    }
+
     return new QueryContext.Builder().setTableName(tableName).setSubquery(subquery)
         .setSelectExpressions(selectExpressions).setDistinct(distinct).setAliasList(aliasList).setFilter(filter)
         .setGroupByExpressions(groupByExpressions).setOrderByExpressions(orderByExpressions)
         .setHavingFilter(havingFilter).setLimit(pinotQuery.getLimit()).setOffset(pinotQuery.getOffset())
         .setQueryOptions(pinotQuery.getQueryOptions()).setExpressionOverrideHints(expressionContextOverrideHints)
-        .setExplain(pinotQuery.isExplain()).build();
+        .setExplain(explainMode).build();
+  }
+
+  private static boolean isUsingV1(PinotQuery pinotQuery) {

Review Comment:
   Renamed and modified to delegate on QueryOptionUtils. I'm not directly using isUseMultistageEngine because I assume the map may be null, so we need to verify that first



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


Re: [PR] Multi stage explain [pinot]

Posted by "gortiz (via GitHub)" <gi...@apache.org>.
gortiz commented on code in PR #13733:
URL: https://github.com/apache/pinot/pull/13733#discussion_r1751902938


##########
pinot-query-runtime/src/main/java/org/apache/pinot/query/runtime/plan/PhysicalPlanVisitor.java:
##########
@@ -56,115 +58,151 @@
 
 
 /**
- * This visitor constructs a physical plan of operators from a {@link PlanNode} tree. Note that
- * this works only for the intermediate stage nodes, leaf stage nodes are expected to compile into
+ * A class used to transform PlanNodes (considered logical) into MultiStageOperator (considered physical).
+ *
+ * Note that this works only for the intermediate stage nodes, leaf stage nodes are expected to compile into
  * v1 operators at this point in time.
  *
- * <p>This class should be used statically via {@link #walkPlanNode(PlanNode, OpChainExecutionContext)}
+ * <p><b>Notice</b>: Here <em>physical</em> is used in the context of multi-stage engine, which means it transforms
+ * logical PlanNodes into MultiStageOperator.
+ * Probably another adjective should be used given physical means different things for Calcite and single-stage</p>
  */
-public class PhysicalPlanVisitor implements PlanNodeVisitor<MultiStageOperator, OpChainExecutionContext> {
+// TODO: rename as PhysicalPlaner or PlanNodeToOperator
+public class PhysicalPlanVisitor {
+
+  private PhysicalPlanVisitor() {
+  }
 
-  private static final PhysicalPlanVisitor INSTANCE = new PhysicalPlanVisitor();
+  public static OpChain planToOperators(PlanNode node, OpChainExecutionContext context) {
+    return planToOperators(node, context, (planNode, operator) -> {
+      // Do nothing
+    });
+  }
 
-  public static OpChain walkPlanNode(PlanNode node, OpChainExecutionContext context) {
-    MultiStageOperator root = node.visit(INSTANCE, context);
+  /**
+   * Like {@link #planToOperators(PlanNode, OpChainExecutionContext, BiConsumer)} but keeps tracking of the original
+   * PlanNode that created each MultiStageOperator
+   * @param tracker a consumer that will be called each time a MultiStageOperator is created.
+   * @return
+   */
+  public static OpChain planToOperators(PlanNode node, OpChainExecutionContext context,
+      BiConsumer<PlanNode, MultiStageOperator> tracker) {
+    MyVisitor visitor = new MyVisitor(tracker);
+    MultiStageOperator root = node.visit(visitor, context);
+    tracker.accept(node, root);
     return new OpChain(context, root);
   }
 
-  private <T extends PlanNode> MultiStageOperator visit(T node, OpChainExecutionContext context) {
-    if (context.getLeafStageContext() != null && context.getLeafStageContext().getLeafStageBoundaryNode() == node) {
-      ServerPlanRequestContext leafStageContext = context.getLeafStageContext();
-      return new LeafStageTransferableBlockOperator(context, leafStageContext.getServerQueryRequests(),
-          leafStageContext.getLeafStageBoundaryNode().getDataSchema(), leafStageContext.getLeafQueryExecutor(),
-          leafStageContext.getExecutorService());
-    } else {
-      return node.visit(this, context);
+  private static class MyVisitor implements PlanNodeVisitor<MultiStageOperator, OpChainExecutionContext> {

Review Comment:
   It wasn't necessary to modify `PhysicalPlanVisitor` so it doesn't implements `PlanNodeVisitor`, I'm just applying the [boy scouts rule](https://deviq.com/principles/boy-scout-rule) here, although I should also change the name of the class (as indicated in the TODO).
   
   Contrary to what its name suggested, `PhysicalPlanVisitor` is not a visitor. Or at least it doesn't have to be a visitor. Although in the older code `PhysicalPlanVisitor` indeed implemented  `PlanNodeVisitor`, it was an implementation detail. The original code didn't expect callers to call `node.visit(new PhysicalPlanVisitor(), null)`. Instead it was expected that `walkPlanNode` was called.
   
   That is why in this PR I'm moving the visitor part (again, an implementation detail) into a private class. Now consumers just see the `planToOperators` static method, making this class nicer to use.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


Re: [PR] Multi stage explain [pinot]

Posted by "gortiz (via GitHub)" <gi...@apache.org>.
gortiz commented on PR #13733:
URL: https://github.com/apache/pinot/pull/13733#issuecomment-2358554450

   > something I just realized is that we can lose the table related information in the new explain plan. For instance, this query on the basic quickstart:
   > Looks like you updated StreamingSelectionOnlyOperator to include the table name in its explain attributes; I guess we should do something similar for all other such operators that can be used in the leaf stage (for instance, StreamingInstanceResponseOperator)?
   
   Nice catch. Yes, I've found the same problem and tried to solve it that way, but it doesn't seem to be a scalable solution. Very easily we can end up having an operator that is not registering the table. Instead what I've done is to add the table in the `LeafStageTransferableBlockOperator`, which by definition knows the table


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


Re: [PR] Multi stage explain [pinot]

Posted by "gortiz (via GitHub)" <gi...@apache.org>.
gortiz commented on code in PR #13733:
URL: https://github.com/apache/pinot/pull/13733#discussion_r1749882690


##########
pinot-query-planner/src/main/java/org/apache/pinot/query/catalog/PinotTable.java:
##########
@@ -44,8 +43,13 @@ public PinotTable(Schema schema) {
 
   @Override
   public RelDataType getRowType(RelDataTypeFactory relDataTypeFactory) {
-    Preconditions.checkState(relDataTypeFactory instanceof TypeFactory);
-    TypeFactory typeFactory = (TypeFactory) relDataTypeFactory;
+    // TODO: Look for a better solution
+    TypeFactory typeFactory;
+    if (relDataTypeFactory instanceof TypeFactory) {
+      typeFactory = (TypeFactory) relDataTypeFactory;
+    } else {
+      typeFactory = TypeFactory.INSTANCE;
+    }

Review Comment:
   TBH I don't remember exactly, but it looks like this method can be called with a RelDataTypeFactory that may not be a TypeFactory, in which case it fails to generate the row types.
   
   I was trying to reproduce the issue and I wasn't able to do it, so maybe it was something I've got while I have the partial code? Given this explain plan will be disabled by default, we can remove this change and go back to it if is actually needed.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


Re: [PR] Multi stage explain [pinot]

Posted by "gortiz (via GitHub)" <gi...@apache.org>.
gortiz commented on code in PR #13733:
URL: https://github.com/apache/pinot/pull/13733#discussion_r1750317381


##########
pinot-common/src/main/proto/plan.proto:
##########
@@ -193,3 +194,29 @@ message WindowNode {
   int32 upperBound = 6;
   repeated Literal constants = 7;
 }
+
+message ExplainNode {
+  string type = 1;
+  map<string, AttributeValue> attributes = 2;
+
+  message AttributeValue {
+    MergeType mergeType = 1;
+
+    oneof value {
+      string string = 2;
+      int64 long = 3;
+      bool bool = 4;
+      string json = 5;
+    };
+
+    enum MergeType {
+      // Longs will be added, other types will behave like IDEMPOTENT.
+      DEFAULT = 0;
+      // When being merge, values must be the same. Otherwise cannot be merged.
+      IDEMPOTENT = 1;
+      // When being merge, if values are different, the value can be ignored.

Review Comment:
   Done



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


Re: [PR] Multi stage explain [pinot]

Posted by "gortiz (via GitHub)" <gi...@apache.org>.
gortiz commented on code in PR #13733:
URL: https://github.com/apache/pinot/pull/13733#discussion_r1752074374


##########
pinot-query-planner/src/main/java/org/apache/pinot/query/planner/logical/PlanNodeToRelConverter.java:
##########
@@ -0,0 +1,468 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.query.planner.logical;
+
+import com.google.common.base.Preconditions;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.List;
+import java.util.stream.Collectors;
+import org.apache.calcite.plan.RelOptCluster;
+import org.apache.calcite.plan.RelTraitSet;
+import org.apache.calcite.rel.RelCollation;
+import org.apache.calcite.rel.RelCollations;
+import org.apache.calcite.rel.RelDistribution;
+import org.apache.calcite.rel.RelDistributions;
+import org.apache.calcite.rel.RelNode;
+import org.apache.calcite.rel.core.SetOp;
+import org.apache.calcite.rel.core.Window;
+import org.apache.calcite.rel.logical.LogicalIntersect;
+import org.apache.calcite.rel.logical.LogicalMinus;
+import org.apache.calcite.rel.logical.LogicalSort;
+import org.apache.calcite.rel.logical.LogicalUnion;
+import org.apache.calcite.rel.logical.LogicalWindow;
+import org.apache.calcite.rel.type.RelDataType;
+import org.apache.calcite.rex.RexLiteral;
+import org.apache.calcite.rex.RexNode;
+import org.apache.calcite.rex.RexWindowBound;
+import org.apache.calcite.rex.RexWindowBounds;
+import org.apache.calcite.sql.SqlAggFunction;
+import org.apache.calcite.tools.RelBuilder;
+import org.apache.calcite.util.ImmutableBitSet;
+import org.apache.pinot.common.utils.DatabaseUtils;
+import org.apache.pinot.core.operator.ExplainAttributeBuilder;
+import org.apache.pinot.core.plan.PinotExplainedRelNode;
+import org.apache.pinot.query.planner.plannode.AggregateNode;
+import org.apache.pinot.query.planner.plannode.ExchangeNode;
+import org.apache.pinot.query.planner.plannode.ExplainedNode;
+import org.apache.pinot.query.planner.plannode.FilterNode;
+import org.apache.pinot.query.planner.plannode.JoinNode;
+import org.apache.pinot.query.planner.plannode.MailboxReceiveNode;
+import org.apache.pinot.query.planner.plannode.MailboxSendNode;
+import org.apache.pinot.query.planner.plannode.PlanNode;
+import org.apache.pinot.query.planner.plannode.PlanNodeVisitor;
+import org.apache.pinot.query.planner.plannode.ProjectNode;
+import org.apache.pinot.query.planner.plannode.SetOpNode;
+import org.apache.pinot.query.planner.plannode.SortNode;
+import org.apache.pinot.query.planner.plannode.TableScanNode;
+import org.apache.pinot.query.planner.plannode.ValueNode;
+import org.apache.pinot.query.planner.plannode.WindowNode;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+
+/**
+ * Converts a {@link PlanNode} into a {@link RelNode}.
+ *
+ * This class is used to convert serialized plan nodes into RelNodes so they can be used when explain with
+ * implementation is requested. Therefore some nodes may be transformed in a way that loses information that is
+ * required to create an actual executable plan but not necessary in order to describe the plan.
+ */
+public final class PlanNodeToRelConverter {
+  private static final Logger LOGGER = LoggerFactory.getLogger(PlanNodeToRelConverter.class);
+
+  private PlanNodeToRelConverter() {
+  }
+
+  public static RelNode convert(RelBuilder builder, PlanNode planNode) {
+    ConverterVisitor visitor = new ConverterVisitor(builder);
+    planNode.visit(visitor, null);
+
+    return visitor.build();
+  }
+
+  private static class ConverterVisitor implements PlanNodeVisitor<Void, Void> {
+    private final RelBuilder _builder;
+
+    public ConverterVisitor(RelBuilder builder) {
+      _builder = builder;
+    }
+
+    private void visitChildren(PlanNode node) {
+      node.getInputs().forEach(input -> input.visit(this, null));
+    }
+
+    @Override
+    public Void visitAggregate(AggregateNode node, Void context) {
+      visitChildren(node);
+
+      try {
+        int[] groupKeyArr = node.getGroupKeys().stream().mapToInt(Integer::intValue).toArray();
+        RelBuilder.GroupKey groupKey = _builder.groupKey(groupKeyArr);
+
+        List<RelBuilder.AggCall> aggCalls =
+            node.getAggCalls().stream().map(functionCall -> RexExpressionUtils.toAggCall(_builder, functionCall))
+                .collect(Collectors.toList());
+
+        _builder.aggregate(groupKey, aggCalls);
+      } catch (RuntimeException e) {
+        LOGGER.warn("Failed to convert aggregate node: {}", node, e);
+        _builder.push(new PinotExplainedRelNode(_builder.getCluster(), "UnknownAggregate", Collections.emptyMap(),
+            node.getDataSchema(), readAlreadyPushedChildren(node)));
+      }

Review Comment:
   Given the builder is quite complex, some subtle conditions may produce this error. Given the class javadoc explicitly that the generated tree should be used in explains because semantics may be lost, I think it is fair to return this synthetic node. In the end what we are trying to give users is something like:
   
   ```
   Execution Plan
   LogicalSort(offset=[0], fetch=[10])
     PinotLogicalSortExchange(distribution=[hash], collation=[[]], isSortOnSender=[false], isSortOnReceiver=[false])
       UnknownAggregate
         LogicalJoin(condition=[=($3, $5)], joinType=[inner])
           ...
   ```
   
   Instead of an error. Being optimistic, I think this will be more useful than just having a blind error.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


Re: [PR] Multi stage explain [pinot]

Posted by "gortiz (via GitHub)" <gi...@apache.org>.
gortiz commented on code in PR #13733:
URL: https://github.com/apache/pinot/pull/13733#discussion_r1750381217


##########
pinot-common/src/main/java/org/apache/pinot/common/utils/DatabaseUtils.java:
##########
@@ -32,6 +32,10 @@ public class DatabaseUtils {
   private DatabaseUtils() {
   }
 
+  public static String[] splitTableName(String tableName) {

Review Comment:
   Added the javadoc. The reason to add that is because Calcite RelBuilder supports qualified table names. In case we use them, this should honor them.
   
   Anyway I don't think PlanNodeToRelConverted.visitTableScan will ever be called given TableScans should always be transformed into other operators in the leaf stage.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


Re: [PR] Multi stage explain [pinot]

Posted by "yashmayya (via GitHub)" <gi...@apache.org>.
yashmayya commented on PR #13733:
URL: https://github.com/apache/pinot/pull/13733#issuecomment-2401368472

   https://github.com/apache/pinot/pull/14193


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


Re: [PR] Multi stage explain [pinot]

Posted by "gortiz (via GitHub)" <gi...@apache.org>.
gortiz commented on code in PR #13733:
URL: https://github.com/apache/pinot/pull/13733#discussion_r1750350125


##########
pinot-core/src/main/java/org/apache/pinot/core/operator/ProjectionOperator.java:
##########
@@ -100,6 +100,17 @@ public String toExplainString() {
     return stringBuilder.append(')').toString();
   }
 
+  @Override
+  protected String getExplainName() {
+    return EXPLAIN_NAME;
+  }
+
+  @Override
+  protected void explainAttributes(ExplainAttributeBuilder attributeBuilder) {
+    super.explainAttributes(attributeBuilder);

Review Comment:
   Ideally all explainAttributes should call its super just in case in future we add something common in the super class.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


Re: [PR] Multi stage explain [pinot]

Posted by "yashmayya (via GitHub)" <gi...@apache.org>.
yashmayya commented on code in PR #13733:
URL: https://github.com/apache/pinot/pull/13733#discussion_r1751736085


##########
pinot-core/src/main/java/org/apache/pinot/core/operator/query/SelectionPartiallyOrderedByAscOperator.java:
##########
@@ -77,6 +78,6 @@ public int getNumDocsScanned() {
 
   @Override
   protected String getExplainName() {
-    return EXPLAIN_NAME;
+    return CaseFormat.UPPER_UNDERSCORE.to(CaseFormat.UPPER_CAMEL, EXPLAIN_NAME);

Review Comment:
   Comment can be resolved, fixed in https://github.com/apache/pinot/pull/13733/commits/9339792aa9856d606afd3cec6513e53e55b4bdb3.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


Re: [PR] Multi stage explain [pinot]

Posted by "gortiz (via GitHub)" <gi...@apache.org>.
gortiz commented on code in PR #13733:
URL: https://github.com/apache/pinot/pull/13733#discussion_r1747177200


##########
pinot-spi/src/main/java/org/apache/pinot/spi/utils/CommonConstants.java:
##########
@@ -1192,6 +1209,9 @@ public enum WindowOverFlowMode {
     public static class PlanVersions {
       public static final int V1 = 1;
     }
+
+    public static final String ASK_SERVERS_FOR_EXPLAIN_PLAN = "pinot.query.explain.ask.servers";

Review Comment:
   I prefer to do not use _physical_ given we use that term to refer to different things (in Calcite, in distribution, here, etc). But I think it makes sense to make more clear that it is only used in multi-stage



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


Re: [PR] Multi stage explain [pinot]

Posted by "gortiz (via GitHub)" <gi...@apache.org>.
gortiz commented on code in PR #13733:
URL: https://github.com/apache/pinot/pull/13733#discussion_r1747183652


##########
pinot-query-runtime/src/main/java/org/apache/pinot/query/runtime/operator/MultiStageOperator.java:
##########
@@ -167,6 +173,26 @@ protected TransferableBlock updateEosBlock(TransferableBlock upstreamEos, StatMa
     return upstreamEos;
   }
 
+  @Override
+  public ExplainInfo getExplainInfo() {
+    return new ExplainInfo(getExplainName(), getExplainAttributes(), getChildrenExplainInfo());
+  }
+
+  protected List<ExplainInfo> getChildrenExplainInfo() {
+    return getChildOperators().stream()
+        .filter(Objects::nonNull)
+        .map(Operator::getExplainInfo)
+        .collect(Collectors.toList());
+  }
+
+  protected String getExplainName() {
+    return toExplainString();
+  }
+
+  protected Map<String, Plan.ExplainNode.AttributeValue> getExplainAttributes() {
+    return Collections.emptyMap();
+  }

Review Comment:
   That is what I tried on the first iteration but... these method should be public, which is something I don't like because that is not part of the `Operator` interface. We could create another abstract class that extended by both `BaseOperator` and `MultiStageOperator`. I'm open to add that change, but I think this is simpler for now.
   
   BTW, you can see that V1 explain faced the same problem. In that case developers decided to add methods as `default` in the interface. As a result, the `Operator` methods related to explain are pretty complex and callers don't actually know what they should call and what they should not. Should they call `toExplainString`? `explainPlan`? Should they call `prepareForExplainPlan` and `postExplainPlan` before/after calling the other two methods? 
   
   Another side effect is that like 1/3 of the methods in `Operator` are related to explain, when probably callers only care about 1 or 2. If we add these 3 protected methods in the interface the relation would be closer to `1/2`!



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


Re: [PR] Multi stage explain [pinot]

Posted by "gortiz (via GitHub)" <gi...@apache.org>.
gortiz commented on code in PR #13733:
URL: https://github.com/apache/pinot/pull/13733#discussion_r1749888617


##########
pinot-query-planner/src/main/java/org/apache/pinot/query/QueryEnvironment.java:
##########
@@ -172,7 +185,27 @@ public QueryPlannerResult explainQuery(String sqlQuery, SqlNodeAndOptions sqlNod
         SqlExplainLevel level =
             explain.getDetailLevel() == null ? SqlExplainLevel.DIGEST_ATTRIBUTES : explain.getDetailLevel();
         Set<String> tableNames = RelToPlanNodeConverter.getTableNamesFromRelRoot(relRoot.rel);
-        return new QueryPlannerResult(null, PlannerUtils.explainPlan(relRoot.rel, format, level), tableNames);
+        if (!explain.withImplementation() || !askServers) {

Review Comment:
   The `WITH IMPLEMENTATION` is part of Calcite. The idea is that `WITHOUT IMPLEMENTATION` should show the logical (LogicalFilter, etc) plan while by default or with `WITH IMPLEMENTATION` should return the physical plan (the actual operator we use, something closer to what we are showing in this PR).
   
   IMHO using `EXPLAIN IMPLEMENTATION PLAN FOR` was a mistake. We should have created something like `EXPLAIN PLAN WITH <our extension>` where `<our extension` is something like `TOPOLOGY` or `DISTRIBUTION` or something like that. We can try to use this PR as a base for a future extension where we can include distribution information in the normal plan (ie we could include, in mailbox send, the fanout as we do in execution stats)



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


Re: [PR] Multi stage explain [pinot]

Posted by "gortiz (via GitHub)" <gi...@apache.org>.
gortiz commented on code in PR #13733:
URL: https://github.com/apache/pinot/pull/13733#discussion_r1749891618


##########
pinot-integration-tests/src/test/java/org/apache/pinot/integration/tests/MultiStageEngineExplainIntegrationTest.java:
##########
@@ -0,0 +1,255 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.integration.tests;
+
+import com.fasterxml.jackson.databind.JsonNode;
+import java.io.File;
+import java.util.List;
+import org.apache.pinot.spi.config.table.TableConfig;
+import org.apache.pinot.spi.data.Schema;
+import org.apache.pinot.spi.env.PinotConfiguration;
+import org.apache.pinot.spi.utils.CommonConstants;
+import org.apache.pinot.util.TestUtils;
+import org.intellij.lang.annotations.Language;
+import org.testcontainers.shaded.org.apache.commons.io.FileUtils;
+import org.testng.Assert;
+import org.testng.annotations.AfterClass;
+import org.testng.annotations.BeforeClass;
+import org.testng.annotations.BeforeMethod;
+import org.testng.annotations.Test;
+
+
+public class MultiStageEngineExplainIntegrationTest extends BaseClusterIntegrationTest {
+  private static final String SCHEMA_FILE_NAME = "On_Time_On_Time_Performance_2014_100k_subset_nonulls.schema";
+  private static final String DEFAULT_DATABASE_NAME = CommonConstants.DEFAULT_DATABASE;
+  private static final String DATABASE_NAME = "db1";
+  private static final String TABLE_NAME_WITH_DATABASE = DATABASE_NAME + "." + DEFAULT_TABLE_NAME;
+  private String _tableName = DEFAULT_TABLE_NAME;
+
+  @Override
+  protected String getSchemaFileName() {
+    return SCHEMA_FILE_NAME;
+  }
+
+  @BeforeClass
+  public void setUp()
+      throws Exception {
+    TestUtils.ensureDirectoriesExistAndEmpty(_tempDir, _segmentDir, _tarDir);
+
+    // Start the Pinot cluster
+    startZk();
+    startController();
+    startBroker();
+    startServers(2);
+
+    // Create and upload the schema and table config
+    Schema schema = createSchema();
+    addSchema(schema);
+    TableConfig tableConfig = createOfflineTableConfig();
+    addTableConfig(tableConfig);
+
+    // Unpack the Avro files
+    List<File> avroFiles = unpackAvroData(_tempDir);
+
+    // Create and upload segments
+    ClusterIntegrationTestUtils.buildSegmentsFromAvro(avroFiles, tableConfig, schema, 0, _segmentDir, _tarDir);
+    uploadSegments(getTableName(), _tarDir);
+
+    // Wait for all documents loaded
+    waitForAllDocsLoaded(600_000L);
+  }
+
+  protected void overrideBrokerConf(PinotConfiguration brokerConf) {
+    brokerConf.setProperty(CommonConstants.MultiStageQueryRunner.ASK_SERVERS_FOR_EXPLAIN_PLAN, "true");
+  }
+
+  @BeforeMethod
+  public void resetMultiStage() {
+    setUseMultiStageQueryEngine(true);
+  }
+
+  @Test
+  public void simpleQuery() {
+    explain("SELECT 1 FROM mytable",
+        //@formatter:off
+        "Execution Plan\n"
+            + "PinotLogicalExchange(distribution=[broadcast])\n"

Review Comment:
   I thought about that and preferred to keep it like this just to clearly mark where the synthetic nodes start, but I'm open to change it.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


Re: [PR] Multi stage explain [pinot]

Posted by "gortiz (via GitHub)" <gi...@apache.org>.
gortiz commented on code in PR #13733:
URL: https://github.com/apache/pinot/pull/13733#discussion_r1750382813


##########
pinot-common/src/main/java/org/apache/pinot/common/utils/DataSchema.java:
##########
@@ -196,27 +201,135 @@ public int hashCode() {
     return EqualityUtils.hashCodeOf(Arrays.hashCode(_columnNames), Arrays.hashCode(_columnDataTypes));
   }
 
+  public RelDataType toRelDataType(RelDataTypeFactory typeFactory) {
+    List<RelDataType> columnTypes = new ArrayList<>(_columnDataTypes.length);
+    for (ColumnDataType columnDataType : _columnDataTypes) {
+      columnTypes.add(columnDataType.toType(typeFactory));
+    }
+    return typeFactory.createStructType(columnTypes, Arrays.asList(_columnNames));
+  }
+
   public enum ColumnDataType {
-    INT(NullValuePlaceHolder.INT),
-    LONG(NullValuePlaceHolder.LONG),
-    FLOAT(NullValuePlaceHolder.FLOAT),
-    DOUBLE(NullValuePlaceHolder.DOUBLE),
-    BIG_DECIMAL(NullValuePlaceHolder.BIG_DECIMAL),
-    BOOLEAN(INT, NullValuePlaceHolder.INT),
-    TIMESTAMP(LONG, NullValuePlaceHolder.LONG),
-    STRING(NullValuePlaceHolder.STRING),
-    JSON(STRING, NullValuePlaceHolder.STRING),
-    BYTES(NullValuePlaceHolder.INTERNAL_BYTES),
-    OBJECT(null),
-    INT_ARRAY(NullValuePlaceHolder.INT_ARRAY),
-    LONG_ARRAY(NullValuePlaceHolder.LONG_ARRAY),
-    FLOAT_ARRAY(NullValuePlaceHolder.FLOAT_ARRAY),
-    DOUBLE_ARRAY(NullValuePlaceHolder.DOUBLE_ARRAY),
-    BOOLEAN_ARRAY(INT_ARRAY, NullValuePlaceHolder.INT_ARRAY),
-    TIMESTAMP_ARRAY(LONG_ARRAY, NullValuePlaceHolder.LONG_ARRAY),
-    STRING_ARRAY(NullValuePlaceHolder.STRING_ARRAY),
-    BYTES_ARRAY(NullValuePlaceHolder.BYTES_ARRAY),
-    UNKNOWN(null);
+    INT(NullValuePlaceHolder.INT) {
+      @Override
+      public RelDataType toType(RelDataTypeFactory typeFactory) {
+        return typeFactory.createSqlType(SqlTypeName.INTEGER);
+      }
+    },
+    LONG(NullValuePlaceHolder.LONG) {
+      @Override
+      public RelDataType toType(RelDataTypeFactory typeFactory) {
+        return typeFactory.createSqlType(SqlTypeName.BIGINT);
+      }
+    },
+    FLOAT(NullValuePlaceHolder.FLOAT) {
+      @Override
+      public RelDataType toType(RelDataTypeFactory typeFactory) {
+        return typeFactory.createSqlType(SqlTypeName.FLOAT);

Review Comment:
   Changed. I'm not sure if we can unify these methods. It would be great, but I think we can do that in a future PR.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


Re: [PR] Multi stage explain [pinot]

Posted by "gortiz (via GitHub)" <gi...@apache.org>.
gortiz commented on PR #13733:
URL: https://github.com/apache/pinot/pull/13733#issuecomment-2341357190

   I've refactored the code a bit to reduce the usage of generic java.util.Function and add instead a AskingServerStageExplainer. I've also added some javadocs. I hope this will make it easier to understand this PR.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


Re: [PR] Multi stage explain [pinot]

Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #13733:
URL: https://github.com/apache/pinot/pull/13733#issuecomment-2268774392

   ## [Codecov](https://app.codecov.io/gh/apache/pinot/pull/13733?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
   Attention: Patch coverage is `0%` with `1011 lines` in your changes missing coverage. Please review.
   > Project coverage is 27.62%. Comparing base [(`59551e4`)](https://app.codecov.io/gh/apache/pinot/commit/59551e45224f1535c4863fd577622b37366ccc97?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) to head [(`a3b3874`)](https://app.codecov.io/gh/apache/pinot/commit/a3b387453bd16e0c5fb0df4e8550f4ef4452bf1e?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
   > Report is 820 commits behind head on master.
   
   | [Files](https://app.codecov.io/gh/apache/pinot/pull/13733?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Patch % | Lines |
   |---|---|---|
   | [.../query/planner/logical/PlanNodeToRelConverter.java](https://app.codecov.io/gh/apache/pinot/pull/13733?src=pr&el=tree&filepath=pinot-query-planner%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fpinot%2Fquery%2Fplanner%2Flogical%2FPlanNodeToRelConverter.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-cGlub3QtcXVlcnktcGxhbm5lci9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvcXVlcnkvcGxhbm5lci9sb2dpY2FsL1BsYW5Ob2RlVG9SZWxDb252ZXJ0ZXIuamF2YQ==) | 0.00% | [104 Missing :warning: ](https://app.codecov.io/gh/apache/pinot/pull/13733?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
   | [.../pinot/query/service/dispatch/QueryDispatcher.java](https://app.codecov.io/gh/apache/pinot/pull/13733?src=pr&el=tree&filepath=pinot-query-runtime%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fpinot%2Fquery%2Fservice%2Fdispatch%2FQueryDispatcher.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-cGlub3QtcXVlcnktcnVudGltZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvcXVlcnkvc2VydmljZS9kaXNwYXRjaC9RdWVyeURpc3BhdGNoZXIuamF2YQ==) | 0.00% | [104 Missing :warning: ](https://app.codecov.io/gh/apache/pinot/pull/13733?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
   | [...apache/pinot/query/service/server/QueryServer.java](https://app.codecov.io/gh/apache/pinot/pull/13733?src=pr&el=tree&filepath=pinot-query-runtime%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fpinot%2Fquery%2Fservice%2Fserver%2FQueryServer.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-cGlub3QtcXVlcnktcnVudGltZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvcXVlcnkvc2VydmljZS9zZXJ2ZXIvUXVlcnlTZXJ2ZXIuamF2YQ==) | 0.00% | [84 Missing :warning: ](https://app.codecov.io/gh/apache/pinot/pull/13733?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
   | [...inot/query/planner/logical/RexExpressionUtils.java](https://app.codecov.io/gh/apache/pinot/pull/13733?src=pr&el=tree&filepath=pinot-query-planner%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fpinot%2Fquery%2Fplanner%2Flogical%2FRexExpressionUtils.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-cGlub3QtcXVlcnktcGxhbm5lci9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvcXVlcnkvcGxhbm5lci9sb2dpY2FsL1JleEV4cHJlc3Npb25VdGlscy5qYXZh) | 0.00% | [72 Missing :warning: ](https://app.codecov.io/gh/apache/pinot/pull/13733?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
   | [...core/query/executor/ServerQueryExecutorV1Impl.java](https://app.codecov.io/gh/apache/pinot/pull/13733?src=pr&el=tree&filepath=pinot-core%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fpinot%2Fcore%2Fquery%2Fexecutor%2FServerQueryExecutorV1Impl.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9xdWVyeS9leGVjdXRvci9TZXJ2ZXJRdWVyeUV4ZWN1dG9yVjFJbXBsLmphdmE=) | 0.00% | [60 Missing :warning: ](https://app.codecov.io/gh/apache/pinot/pull/13733?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
   | [.../pinot/query/runtime/plan/PhysicalPlanVisitor.java](https://app.codecov.io/gh/apache/pinot/pull/13733?src=pr&el=tree&filepath=pinot-query-runtime%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fpinot%2Fquery%2Fruntime%2Fplan%2FPhysicalPlanVisitor.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-cGlub3QtcXVlcnktcnVudGltZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvcXVlcnkvcnVudGltZS9wbGFuL1BoeXNpY2FsUGxhblZpc2l0b3IuamF2YQ==) | 0.00% | [48 Missing :warning: ](https://app.codecov.io/gh/apache/pinot/pull/13733?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
   | [...java/org/apache/pinot/common/utils/DataSchema.java](https://app.codecov.io/gh/apache/pinot/pull/13733?src=pr&el=tree&filepath=pinot-common%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fpinot%2Fcommon%2Futils%2FDataSchema.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-cGlub3QtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9jb21tb24vdXRpbHMvRGF0YVNjaGVtYS5qYXZh) | 0.00% | [44 Missing :warning: ](https://app.codecov.io/gh/apache/pinot/pull/13733?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
   | [.../apache/pinot/core/plan/PinotExplainedRelNode.java](https://app.codecov.io/gh/apache/pinot/pull/13733?src=pr&el=tree&filepath=pinot-core%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fpinot%2Fcore%2Fplan%2FPinotExplainedRelNode.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9wbGFuL1Bpbm90RXhwbGFpbmVkUmVsTm9kZS5qYXZh) | 0.00% | [40 Missing :warning: ](https://app.codecov.io/gh/apache/pinot/pull/13733?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
   | [...va/org/apache/pinot/query/runtime/QueryRunner.java](https://app.codecov.io/gh/apache/pinot/pull/13733?src=pr&el=tree&filepath=pinot-query-runtime%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fpinot%2Fquery%2Fruntime%2FQueryRunner.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-cGlub3QtcXVlcnktcnVudGltZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvcXVlcnkvcnVudGltZS9RdWVyeVJ1bm5lci5qYXZh) | 0.00% | [38 Missing :warning: ](https://app.codecov.io/gh/apache/pinot/pull/13733?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
   | [...e/operator/LeafStageTransferableBlockOperator.java](https://app.codecov.io/gh/apache/pinot/pull/13733?src=pr&el=tree&filepath=pinot-query-runtime%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fpinot%2Fquery%2Fruntime%2Foperator%2FLeafStageTransferableBlockOperator.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-cGlub3QtcXVlcnktcnVudGltZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvcXVlcnkvcnVudGltZS9vcGVyYXRvci9MZWFmU3RhZ2VUcmFuc2ZlcmFibGVCbG9ja09wZXJhdG9yLmphdmE=) | 0.00% | [36 Missing :warning: ](https://app.codecov.io/gh/apache/pinot/pull/13733?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
   | ... and [70 more](https://app.codecov.io/gh/apache/pinot/pull/13733?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | |
   
   > :exclamation:  There is a different number of reports uploaded between BASE (59551e4) and HEAD (a3b3874). Click for more details.
   > 
   > <details><summary>HEAD has 53 uploads less than BASE</summary>
   >
   >| Flag | BASE (59551e4) | HEAD (a3b3874) |
   >|------|------|------|
   >|integration|7|0|
   >|integration2|3|0|
   >|temurin|12|1|
   >|java-21|7|1|
   >|skip-bytebuffers-true|3|0|
   >|skip-bytebuffers-false|7|1|
   >|unittests|5|1|
   >|unittests1|2|0|
   >|java-11|5|0|
   >|unittests2|3|1|
   >|integration1|2|0|
   >|custom-integration1|2|0|
   ></details>
   
   <details><summary>Additional details and impacted files</summary>
   
   
   ```diff
   @@              Coverage Diff              @@
   ##             master   #13733       +/-   ##
   =============================================
   - Coverage     61.75%   27.62%   -34.13%     
   + Complexity      207      192       -15     
   =============================================
     Files          2436     2565      +129     
     Lines        133233   141862     +8629     
     Branches      20636    22010     +1374     
   =============================================
   - Hits          82274    39192    -43082     
   - Misses        44911    99661    +54750     
   + Partials       6048     3009     -3039     
   ```
   
   | [Flag](https://app.codecov.io/gh/apache/pinot/pull/13733/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
   |---|---|---|
   | [custom-integration1](https://app.codecov.io/gh/apache/pinot/pull/13733/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
   | [integration](https://app.codecov.io/gh/apache/pinot/pull/13733/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
   | [integration1](https://app.codecov.io/gh/apache/pinot/pull/13733/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
   | [integration2](https://app.codecov.io/gh/apache/pinot/pull/13733/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
   | [java-11](https://app.codecov.io/gh/apache/pinot/pull/13733/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
   | [java-21](https://app.codecov.io/gh/apache/pinot/pull/13733/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `27.62% <0.00%> (-34.00%)` | :arrow_down: |
   | [skip-bytebuffers-false](https://app.codecov.io/gh/apache/pinot/pull/13733/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `27.62% <0.00%> (-34.13%)` | :arrow_down: |
   | [skip-bytebuffers-true](https://app.codecov.io/gh/apache/pinot/pull/13733/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
   | [temurin](https://app.codecov.io/gh/apache/pinot/pull/13733/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `27.62% <0.00%> (-34.13%)` | :arrow_down: |
   | [unittests](https://app.codecov.io/gh/apache/pinot/pull/13733/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `27.62% <0.00%> (-34.13%)` | :arrow_down: |
   | [unittests1](https://app.codecov.io/gh/apache/pinot/pull/13733/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
   | [unittests2](https://app.codecov.io/gh/apache/pinot/pull/13733/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `27.62% <0.00%> (-0.11%)` | :arrow_down: |
   
   Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#carryforward-flags-in-the-pull-request-comment) to find out more.
   
   </details>
   
   [:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/apache/pinot/pull/13733?dropdown=coverage&src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).   
   :loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


Re: [PR] Multi stage explain [pinot]

Posted by "gortiz (via GitHub)" <gi...@apache.org>.
gortiz commented on code in PR #13733:
URL: https://github.com/apache/pinot/pull/13733#discussion_r1771360606


##########
pinot-query-planner/src/main/java/org/apache/pinot/query/planner/explain/PlanNodeSorter.java:
##########
@@ -0,0 +1,229 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.query.planner.explain;
+
+import com.google.common.collect.Sets;
+import java.util.ArrayList;
+import java.util.Comparator;
+import java.util.List;
+import java.util.TreeSet;
+import org.apache.pinot.common.proto.Plan;
+import org.apache.pinot.query.planner.plannode.AggregateNode;
+import org.apache.pinot.query.planner.plannode.ExchangeNode;
+import org.apache.pinot.query.planner.plannode.ExplainedNode;
+import org.apache.pinot.query.planner.plannode.FilterNode;
+import org.apache.pinot.query.planner.plannode.JoinNode;
+import org.apache.pinot.query.planner.plannode.MailboxReceiveNode;
+import org.apache.pinot.query.planner.plannode.MailboxSendNode;
+import org.apache.pinot.query.planner.plannode.PlanNode;
+import org.apache.pinot.query.planner.plannode.PlanNodeVisitor;
+import org.apache.pinot.query.planner.plannode.ProjectNode;
+import org.apache.pinot.query.planner.plannode.SetOpNode;
+import org.apache.pinot.query.planner.plannode.SortNode;
+import org.apache.pinot.query.planner.plannode.TableScanNode;
+import org.apache.pinot.query.planner.plannode.ValueNode;
+import org.apache.pinot.query.planner.plannode.WindowNode;
+
+
+/**
+ * A utility class used to sort the plan nodes in a deterministic order.
+ *
+ * Any comparator can be passed to the sort method to sort the plan nodes, although the default comparator
+ * is used to sort the plan nodes based on the type and the attributes of the node.
+ *
+ * Only nodes that are simplifiable will be sorted. See {@link ExplainNodeSimplifier} for more information.
+ */
+public class PlanNodeSorter {
+
+  private PlanNodeSorter() {
+  }
+
+  /**
+   * Applies a default comparator to sort the plan nodes.
+   *
+   * The result may be the same as the input if the plan nodes are already sorted.
+   */
+  public static PlanNode sort(PlanNode planNode) {
+    return planNode.visit(new Sorter(), DefaultComparator.INSTANCE);
+  }
+
+  public static PlanNode sort(PlanNode planNode, Comparator<PlanNode> comparator) {
+    return planNode.visit(Sorter.INSTANCE, comparator);
+  }
+
+  private static class Sorter implements PlanNodeVisitor<PlanNode, Comparator<PlanNode>> {
+    private static final Sorter INSTANCE = new Sorter();
+
+    private PlanNode defaultNode(PlanNode node, Comparator<PlanNode> comparator) {
+      List<PlanNode> inputs = node.getInputs();
+      List<PlanNode> newInputs = applyToChildren(inputs, comparator);
+      return inputs != newInputs ? node.withInputs(newInputs) : node;
+    }
+
+    @Override
+    public PlanNode visitAggregate(AggregateNode node, Comparator<PlanNode> comparator) {
+      return defaultNode(node, comparator);
+    }
+
+    @Override
+    public PlanNode visitFilter(FilterNode node, Comparator<PlanNode> comparator) {
+      return defaultNode(node, comparator);
+    }
+
+    @Override
+    public PlanNode visitJoin(JoinNode node, Comparator<PlanNode> comparator) {
+      return defaultNode(node, comparator);
+    }
+
+    @Override
+    public PlanNode visitMailboxReceive(MailboxReceiveNode node, Comparator<PlanNode> comparator) {
+      return defaultNode(node, comparator);
+    }
+
+    @Override
+    public PlanNode visitMailboxSend(MailboxSendNode node, Comparator<PlanNode> comparator) {
+      return defaultNode(node, comparator);
+    }
+
+    @Override
+    public PlanNode visitProject(ProjectNode node, Comparator<PlanNode> comparator) {
+      return defaultNode(node, comparator);
+    }
+
+    @Override
+    public PlanNode visitSort(SortNode node, Comparator<PlanNode> comparator) {
+      return defaultNode(node, comparator);
+    }
+
+    @Override
+    public PlanNode visitTableScan(TableScanNode node, Comparator<PlanNode> comparator) {
+      return defaultNode(node, comparator);
+    }
+
+    @Override
+    public PlanNode visitValue(ValueNode node, Comparator<PlanNode> comparator) {
+      return defaultNode(node, comparator);
+    }
+
+    @Override
+    public PlanNode visitWindow(WindowNode node, Comparator<PlanNode> comparator) {
+      return defaultNode(node, comparator);
+    }
+
+    @Override
+    public PlanNode visitSetOp(SetOpNode node, Comparator<PlanNode> comparator) {
+      return defaultNode(node, comparator);
+    }
+
+    @Override
+    public PlanNode visitExchange(ExchangeNode node, Comparator<PlanNode> comparator) {
+      return defaultNode(node, comparator);
+    }
+
+    @Override
+    public PlanNode visitExplained(ExplainedNode node, Comparator<PlanNode> comparator) {
+      if (!node.getTitle().contains(ExplainNodeSimplifier.COMBINE) || node.getInputs().size() <= 1) {
+        return defaultNode(node, comparator);
+      }
+      List<PlanNode> simplifiedChildren = new ArrayList<>(applyToChildren(node.getInputs(), comparator));
+      simplifiedChildren.sort(comparator);
+      if (simplifiedChildren.equals(node.getInputs())) {
+        return node;
+      }
+      return node.withInputs(simplifiedChildren);
+    }
+
+    private List<PlanNode> applyToChildren(List<PlanNode> children, Comparator<PlanNode> comparator) {
+      List<PlanNode> simplifiedChildren = null;
+      for (int i = 0; i < children.size(); i++) {
+        PlanNode child = children.get(i);
+        PlanNode newChild = child.visit(this, comparator);
+        if (child != newChild) {
+          if (simplifiedChildren == null) {
+            simplifiedChildren = new ArrayList<>(children);
+          }
+          simplifiedChildren.set(i, newChild);
+        }
+      }
+      return simplifiedChildren != null ? simplifiedChildren : children;
+    }
+  }
+
+  public static class DefaultComparator implements Comparator<PlanNode> {
+    public static final DefaultComparator INSTANCE = new DefaultComparator();
+
+    private DefaultComparator() {
+    }
+
+    @Override
+    public int compare(PlanNode o1, PlanNode o2) {
+      if (!(o1 instanceof ExplainedNode) || !(o2 instanceof ExplainedNode)) {
+        return o1.getClass().getSimpleName().compareTo(o2.getClass().getSimpleName());
+      }
+      ExplainedNode node1 = (ExplainedNode) o1;
+      ExplainedNode node2 = (ExplainedNode) o2;
+
+      int cmp = node1.getTitle().compareTo(node2.getTitle());
+      if (cmp != 0) {
+        return cmp;
+      }
+      TreeSet<String> allKeys = new TreeSet<>(Sets.union(
+          node1.getAttributes().keySet(), node2.getAttributes().keySet()));
+      for (String key : allKeys) {
+        Plan.ExplainNode.AttributeValue value1 = node1.getAttributes().get(key);
+        Plan.ExplainNode.AttributeValue value2 = node2.getAttributes().get(key);
+        if (value1 == null) {
+          // there is an attribute in node2 with a _smaller_ key that is not in node1
+          return -1;
+        }
+        if (value2 == null) {
+          // there is an attribute in node1 with a _smaller_ key that is not in node2
+          return 1;
+        }
+        int cmp2 = value1.getValueCase().compareTo(value2.getValueCase());
+        if (cmp2 != 0) {
+          return cmp2;
+        }
+        int cmp3;
+        switch (value1.getValueCase()) {

Review Comment:
   Probably because either I missed it or because JSON (the previous type) is difficult to sort. Adding it now that it is just a list of strings



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


Re: [PR] Multi stage explain [pinot]

Posted by "gortiz (via GitHub)" <gi...@apache.org>.
gortiz commented on code in PR #13733:
URL: https://github.com/apache/pinot/pull/13733#discussion_r1771374971


##########
pinot-query-planner/src/main/java/org/apache/pinot/query/planner/explain/PlanNodeSorter.java:
##########
@@ -0,0 +1,229 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.query.planner.explain;
+
+import com.google.common.collect.Sets;
+import java.util.ArrayList;
+import java.util.Comparator;
+import java.util.List;
+import java.util.TreeSet;
+import org.apache.pinot.common.proto.Plan;
+import org.apache.pinot.query.planner.plannode.AggregateNode;
+import org.apache.pinot.query.planner.plannode.ExchangeNode;
+import org.apache.pinot.query.planner.plannode.ExplainedNode;
+import org.apache.pinot.query.planner.plannode.FilterNode;
+import org.apache.pinot.query.planner.plannode.JoinNode;
+import org.apache.pinot.query.planner.plannode.MailboxReceiveNode;
+import org.apache.pinot.query.planner.plannode.MailboxSendNode;
+import org.apache.pinot.query.planner.plannode.PlanNode;
+import org.apache.pinot.query.planner.plannode.PlanNodeVisitor;
+import org.apache.pinot.query.planner.plannode.ProjectNode;
+import org.apache.pinot.query.planner.plannode.SetOpNode;
+import org.apache.pinot.query.planner.plannode.SortNode;
+import org.apache.pinot.query.planner.plannode.TableScanNode;
+import org.apache.pinot.query.planner.plannode.ValueNode;
+import org.apache.pinot.query.planner.plannode.WindowNode;
+
+
+/**
+ * A utility class used to sort the plan nodes in a deterministic order.
+ *
+ * Any comparator can be passed to the sort method to sort the plan nodes, although the default comparator
+ * is used to sort the plan nodes based on the type and the attributes of the node.
+ *
+ * Only nodes that are simplifiable will be sorted. See {@link ExplainNodeSimplifier} for more information.
+ */
+public class PlanNodeSorter {
+
+  private PlanNodeSorter() {
+  }
+
+  /**
+   * Applies a default comparator to sort the plan nodes.
+   *
+   * The result may be the same as the input if the plan nodes are already sorted.
+   */
+  public static PlanNode sort(PlanNode planNode) {
+    return planNode.visit(new Sorter(), DefaultComparator.INSTANCE);
+  }
+
+  public static PlanNode sort(PlanNode planNode, Comparator<PlanNode> comparator) {
+    return planNode.visit(Sorter.INSTANCE, comparator);
+  }
+
+  private static class Sorter implements PlanNodeVisitor<PlanNode, Comparator<PlanNode>> {
+    private static final Sorter INSTANCE = new Sorter();
+
+    private PlanNode defaultNode(PlanNode node, Comparator<PlanNode> comparator) {
+      List<PlanNode> inputs = node.getInputs();
+      List<PlanNode> newInputs = applyToChildren(inputs, comparator);
+      return inputs != newInputs ? node.withInputs(newInputs) : node;
+    }
+
+    @Override
+    public PlanNode visitAggregate(AggregateNode node, Comparator<PlanNode> comparator) {
+      return defaultNode(node, comparator);
+    }
+
+    @Override
+    public PlanNode visitFilter(FilterNode node, Comparator<PlanNode> comparator) {
+      return defaultNode(node, comparator);
+    }
+
+    @Override
+    public PlanNode visitJoin(JoinNode node, Comparator<PlanNode> comparator) {
+      return defaultNode(node, comparator);
+    }
+
+    @Override
+    public PlanNode visitMailboxReceive(MailboxReceiveNode node, Comparator<PlanNode> comparator) {
+      return defaultNode(node, comparator);
+    }
+
+    @Override
+    public PlanNode visitMailboxSend(MailboxSendNode node, Comparator<PlanNode> comparator) {
+      return defaultNode(node, comparator);
+    }
+
+    @Override
+    public PlanNode visitProject(ProjectNode node, Comparator<PlanNode> comparator) {
+      return defaultNode(node, comparator);
+    }
+
+    @Override
+    public PlanNode visitSort(SortNode node, Comparator<PlanNode> comparator) {
+      return defaultNode(node, comparator);
+    }
+
+    @Override
+    public PlanNode visitTableScan(TableScanNode node, Comparator<PlanNode> comparator) {
+      return defaultNode(node, comparator);
+    }
+
+    @Override
+    public PlanNode visitValue(ValueNode node, Comparator<PlanNode> comparator) {
+      return defaultNode(node, comparator);
+    }
+
+    @Override
+    public PlanNode visitWindow(WindowNode node, Comparator<PlanNode> comparator) {
+      return defaultNode(node, comparator);
+    }
+
+    @Override
+    public PlanNode visitSetOp(SetOpNode node, Comparator<PlanNode> comparator) {
+      return defaultNode(node, comparator);
+    }
+
+    @Override
+    public PlanNode visitExchange(ExchangeNode node, Comparator<PlanNode> comparator) {
+      return defaultNode(node, comparator);
+    }
+
+    @Override
+    public PlanNode visitExplained(ExplainedNode node, Comparator<PlanNode> comparator) {
+      if (!node.getTitle().contains(ExplainNodeSimplifier.COMBINE) || node.getInputs().size() <= 1) {
+        return defaultNode(node, comparator);
+      }
+      List<PlanNode> simplifiedChildren = new ArrayList<>(applyToChildren(node.getInputs(), comparator));
+      simplifiedChildren.sort(comparator);
+      if (simplifiedChildren.equals(node.getInputs())) {
+        return node;
+      }
+      return node.withInputs(simplifiedChildren);
+    }
+
+    private List<PlanNode> applyToChildren(List<PlanNode> children, Comparator<PlanNode> comparator) {
+      List<PlanNode> simplifiedChildren = null;
+      for (int i = 0; i < children.size(); i++) {
+        PlanNode child = children.get(i);
+        PlanNode newChild = child.visit(this, comparator);
+        if (child != newChild) {
+          if (simplifiedChildren == null) {
+            simplifiedChildren = new ArrayList<>(children);
+          }
+          simplifiedChildren.set(i, newChild);
+        }
+      }
+      return simplifiedChildren != null ? simplifiedChildren : children;
+    }
+  }
+
+  public static class DefaultComparator implements Comparator<PlanNode> {
+    public static final DefaultComparator INSTANCE = new DefaultComparator();
+
+    private DefaultComparator() {
+    }
+
+    @Override
+    public int compare(PlanNode o1, PlanNode o2) {
+      if (!(o1 instanceof ExplainedNode) || !(o2 instanceof ExplainedNode)) {
+        return o1.getClass().getSimpleName().compareTo(o2.getClass().getSimpleName());
+      }
+      ExplainedNode node1 = (ExplainedNode) o1;
+      ExplainedNode node2 = (ExplainedNode) o2;
+
+      int cmp = node1.getTitle().compareTo(node2.getTitle());
+      if (cmp != 0) {
+        return cmp;
+      }
+      TreeSet<String> allKeys = new TreeSet<>(Sets.union(
+          node1.getAttributes().keySet(), node2.getAttributes().keySet()));
+      for (String key : allKeys) {
+        Plan.ExplainNode.AttributeValue value1 = node1.getAttributes().get(key);
+        Plan.ExplainNode.AttributeValue value2 = node2.getAttributes().get(key);
+        if (value1 == null) {
+          // there is an attribute in node2 with a _smaller_ key that is not in node1
+          return -1;
+        }
+        if (value2 == null) {
+          // there is an attribute in node1 with a _smaller_ key that is not in node2
+          return 1;
+        }
+        int cmp2 = value1.getValueCase().compareTo(value2.getValueCase());
+        if (cmp2 != 0) {
+          return cmp2;
+        }
+        int cmp3;
+        switch (value1.getValueCase()) {

Review Comment:
   Changed in  https://github.com/apache/pinot/pull/13733/commits/272263125a022bf5419ca85c71d766f6f976a83a



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


Re: [PR] Multi stage explain [pinot]

Posted by "gortiz (via GitHub)" <gi...@apache.org>.
gortiz commented on code in PR #13733:
URL: https://github.com/apache/pinot/pull/13733#discussion_r1750310455


##########
pinot-common/src/main/proto/plan.proto:
##########
@@ -193,3 +194,29 @@ message WindowNode {
   int32 upperBound = 6;
   repeated Literal constants = 7;
 }
+
+message ExplainNode {
+  string type = 1;
+  map<string, AttributeValue> attributes = 2;
+
+  message AttributeValue {
+    MergeType mergeType = 1;
+
+    oneof value {
+      string string = 2;
+      int64 long = 3;
+      bool bool = 4;
+      string json = 5;

Review Comment:
   Changed with StringList



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


Re: [PR] Multi stage explain [pinot]

Posted by "gortiz (via GitHub)" <gi...@apache.org>.
gortiz commented on code in PR #13733:
URL: https://github.com/apache/pinot/pull/13733#discussion_r1747193735


##########
pinot-query-planner/src/main/java/org/apache/pinot/query/planner/explain/PhysicalExplainPlanVisitor.java:
##########
@@ -248,6 +249,11 @@ public StringBuilder visitValue(ValueNode node, Context context) {
     return appendInfo(node, context);
   }
 
+  @Override
+  public StringBuilder visitExplained(ExplainedNode node, Context context) {
+    return appendInfo(node, context);

Review Comment:
   That is the case in this PR. If in the future we support asking servers in this mode, we will need to modify this code.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


Re: [PR] Multi stage explain [pinot]

Posted by "gortiz (via GitHub)" <gi...@apache.org>.
gortiz commented on code in PR #13733:
URL: https://github.com/apache/pinot/pull/13733#discussion_r1752061306


##########
pinot-query-planner/src/main/java/org/apache/pinot/query/planner/explain/PlanNodeSorter.java:
##########
@@ -0,0 +1,228 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.query.planner.explain;
+
+import com.google.common.collect.Sets;
+import java.util.ArrayList;
+import java.util.Comparator;
+import java.util.List;
+import java.util.TreeSet;
+import org.apache.pinot.common.proto.Plan;
+import org.apache.pinot.core.query.reduce.ExplainPlanDataTableReducer;
+import org.apache.pinot.query.planner.plannode.AggregateNode;
+import org.apache.pinot.query.planner.plannode.ExchangeNode;
+import org.apache.pinot.query.planner.plannode.ExplainedNode;
+import org.apache.pinot.query.planner.plannode.FilterNode;
+import org.apache.pinot.query.planner.plannode.JoinNode;
+import org.apache.pinot.query.planner.plannode.MailboxReceiveNode;
+import org.apache.pinot.query.planner.plannode.MailboxSendNode;
+import org.apache.pinot.query.planner.plannode.PlanNode;
+import org.apache.pinot.query.planner.plannode.PlanNodeVisitor;
+import org.apache.pinot.query.planner.plannode.ProjectNode;
+import org.apache.pinot.query.planner.plannode.SetOpNode;
+import org.apache.pinot.query.planner.plannode.SortNode;
+import org.apache.pinot.query.planner.plannode.TableScanNode;
+import org.apache.pinot.query.planner.plannode.ValueNode;
+import org.apache.pinot.query.planner.plannode.WindowNode;
+
+
+/**
+ * A utility class used to sort the plan nodes in a deterministic order.
+ *
+ * Any comparator can be passed to the sort method to sort the plan nodes, although the default comparator
+ * is used to sort the plan nodes based on the type and the attributes of the node.
+ */
+public class PlanNodeSorter {

Review Comment:
   We receive different plans from different servers and each server generate a different plan for each segment. The order inside each server is not guaranteed (they are calculated concurrently). Once we receive the server plans in the broker, we first sort each server plan.
   
   Once we have each server plan sorted, we merge them easily. Not only that, the result will always be shown in the same, which will help users. Otherwise two explains of the same query on the same data will return plans that are semantically equal but the order of some nodes will be different.
   
   Remember this sorting is only applied to `Combine` explain nodes. In other cases reordering inputs may break semantics.
   
   Two notes:
   1. This is something we also do in single-stage explain (see ExplainPlanDataTableReducer). In fact I copied the definition from there. As you can imagine, just sorting the nodes whose name contains `Combine` is not formally correct, but seems to be good enough to be used for a while.
   2. Your comment make me realize the code right now is looking for `COMBINE` in the text instead of the new upper camel case `Combine`. I'm changing that in a new commit.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


Re: [PR] Multi stage explain [pinot]

Posted by "gortiz (via GitHub)" <gi...@apache.org>.
gortiz commented on code in PR #13733:
URL: https://github.com/apache/pinot/pull/13733#discussion_r1754550758


##########
pinot-query-runtime/src/main/java/org/apache/pinot/query/service/dispatch/DispatchClient.java:
##########
@@ -50,12 +51,17 @@ public ManagedChannel getChannel() {
   }
 
   public void submit(Worker.QueryRequest request, QueryServerInstance virtualServer, Deadline deadline,
-      Consumer<AsyncQueryDispatchResponse> callback) {
-    _dispatchStub.withDeadline(deadline).submit(request, new DispatchObserver(virtualServer, callback));
+      Consumer<AsyncResponse<Worker.QueryResponse>> callback) {
+    _dispatchStub.withDeadline(deadline).submit(request, new LastValueDispatchObserver<>(virtualServer, callback));
   }
 
   public void cancel(long requestId) {
     Worker.CancelRequest cancelRequest = Worker.CancelRequest.newBuilder().setRequestId(requestId).build();
     _dispatchStub.cancel(cancelRequest, NO_OP_CANCEL_STREAM_OBSERVER);
   }
+
+  public void explain(Worker.QueryRequest request, QueryServerInstance virtualServer, Deadline deadline,
+      Consumer<AsyncResponse<List<Worker.ExplainResponse>>> callback) {
+    _dispatchStub.withDeadline(deadline).explain(request, new AllValuesDispatchObserver<>(virtualServer, callback));

Review Comment:
   I don't know why we only keep the last message in the submit case, but you can see the older implementation is using [DispatchObserver](https://github.com/apache/pinot/blob/master/pinot-query-runtime/src/main/java/org/apache/pinot/query/service/dispatch/DispatchObserver.java) and verify that it is keeping only the latest `QueryResponse` received in `onNext`, which is what `LastValueDispatcherObserver` does.
   
   TBH the old behavior is correct. You can see that `org.apache.pinot.query.service.server.QueryServer.submit` does only call `onNext` once, so it is ok to keep the last value. What I don't get is why the proto file is defined as:
   ```
     rpc Submit(ServerRequest) returns (stream ServerResponse);
   ```
   instead of
   ```
     rpc Submit(ServerRequest) returns (ServerResponse);
   ```
   
   Anyway, if we change the server implementation we will need to use the new `AllValuesDispatcherObserver` and change the code a bit to work with a list instead of a single element.
   
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


Re: [PR] Multi stage explain [pinot]

Posted by "gortiz (via GitHub)" <gi...@apache.org>.
gortiz commented on code in PR #13733:
URL: https://github.com/apache/pinot/pull/13733#discussion_r1747183652


##########
pinot-query-runtime/src/main/java/org/apache/pinot/query/runtime/operator/MultiStageOperator.java:
##########
@@ -167,6 +173,26 @@ protected TransferableBlock updateEosBlock(TransferableBlock upstreamEos, StatMa
     return upstreamEos;
   }
 
+  @Override
+  public ExplainInfo getExplainInfo() {
+    return new ExplainInfo(getExplainName(), getExplainAttributes(), getChildrenExplainInfo());
+  }
+
+  protected List<ExplainInfo> getChildrenExplainInfo() {
+    return getChildOperators().stream()
+        .filter(Objects::nonNull)
+        .map(Operator::getExplainInfo)
+        .collect(Collectors.toList());
+  }
+
+  protected String getExplainName() {
+    return toExplainString();
+  }
+
+  protected Map<String, Plan.ExplainNode.AttributeValue> getExplainAttributes() {
+    return Collections.emptyMap();
+  }

Review Comment:
   That is what I tried on the first iteration but... these method should be public, which is something I don't like because that is not part of the `Operator` interface. We could create another abstract class that extended by both `BaseOperator` and `MultiStageOperator`. I'm open to add that change, but I think this is simpler for now.
   
   BTW, you can see that V1 explain faced the same problem. In that case developers decided to add methods as `default` in the interface. As a result, the `Operator` methods related to explain are pretty complex and callers don't actually know what they should call and what they should not. Should they call `toExplainString`? `explainPlan`? Should they call `prepareForExplainPlan` and `postExplainPlan` before/after calling the other two methods?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


Re: [PR] Multi stage explain [pinot]

Posted by "gortiz (via GitHub)" <gi...@apache.org>.
gortiz commented on code in PR #13733:
URL: https://github.com/apache/pinot/pull/13733#discussion_r1749925095


##########
pinot-integration-tests/src/test/java/org/apache/pinot/integration/tests/MultiStageEngineExplainIntegrationTest.java:
##########
@@ -0,0 +1,255 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.integration.tests;
+
+import com.fasterxml.jackson.databind.JsonNode;
+import java.io.File;
+import java.util.List;
+import org.apache.pinot.spi.config.table.TableConfig;
+import org.apache.pinot.spi.data.Schema;
+import org.apache.pinot.spi.env.PinotConfiguration;
+import org.apache.pinot.spi.utils.CommonConstants;
+import org.apache.pinot.util.TestUtils;
+import org.intellij.lang.annotations.Language;
+import org.testcontainers.shaded.org.apache.commons.io.FileUtils;
+import org.testng.Assert;
+import org.testng.annotations.AfterClass;
+import org.testng.annotations.BeforeClass;
+import org.testng.annotations.BeforeMethod;
+import org.testng.annotations.Test;
+
+
+public class MultiStageEngineExplainIntegrationTest extends BaseClusterIntegrationTest {
+  private static final String SCHEMA_FILE_NAME = "On_Time_On_Time_Performance_2014_100k_subset_nonulls.schema";
+  private static final String DEFAULT_DATABASE_NAME = CommonConstants.DEFAULT_DATABASE;
+  private static final String DATABASE_NAME = "db1";
+  private static final String TABLE_NAME_WITH_DATABASE = DATABASE_NAME + "." + DEFAULT_TABLE_NAME;
+  private String _tableName = DEFAULT_TABLE_NAME;
+
+  @Override
+  protected String getSchemaFileName() {
+    return SCHEMA_FILE_NAME;
+  }
+
+  @BeforeClass
+  public void setUp()
+      throws Exception {
+    TestUtils.ensureDirectoriesExistAndEmpty(_tempDir, _segmentDir, _tarDir);
+
+    // Start the Pinot cluster
+    startZk();
+    startController();
+    startBroker();
+    startServers(2);
+
+    // Create and upload the schema and table config
+    Schema schema = createSchema();
+    addSchema(schema);
+    TableConfig tableConfig = createOfflineTableConfig();
+    addTableConfig(tableConfig);
+
+    // Unpack the Avro files
+    List<File> avroFiles = unpackAvroData(_tempDir);
+
+    // Create and upload segments
+    ClusterIntegrationTestUtils.buildSegmentsFromAvro(avroFiles, tableConfig, schema, 0, _segmentDir, _tarDir);
+    uploadSegments(getTableName(), _tarDir);
+
+    // Wait for all documents loaded
+    waitForAllDocsLoaded(600_000L);
+  }
+
+  protected void overrideBrokerConf(PinotConfiguration brokerConf) {
+    brokerConf.setProperty(CommonConstants.MultiStageQueryRunner.ASK_SERVERS_FOR_EXPLAIN_PLAN, "true");
+  }
+
+  @BeforeMethod
+  public void resetMultiStage() {
+    setUseMultiStageQueryEngine(true);
+  }
+
+  @Test
+  public void simpleQuery() {
+    explain("SELECT 1 FROM mytable",
+        //@formatter:off
+        "Execution Plan\n"
+            + "PinotLogicalExchange(distribution=[broadcast])\n"
+            + "  LEAF_STAGE_COMBINE_OPERATOR\n"
+            + "    STREAMING_INSTANCE_RESPONSE\n"
+            + "      STREAMING_COMBINE_SELECT(repeatedOnSegments=[12])\n"
+            + "        SELECT_STREAMING(totalDocs=[115545])\n"
+            + "          TRANSFORM(expressions=[[\"'1'\"]])\n"
+            + "            PROJECT(columns=[[]])\n"
+            + "              DOC_ID_SET(maxDocs=[120000])\n"
+            + "                FILTER_MATCH_ENTIRE_SEGMENT(numDocs=[115545])\n");
+        //@formatter:on
+  }
+
+  @Test
+  public void simpleQueryVerbose() {
+    explainVerbose("SELECT 1 FROM mytable",
+        //@formatter:off
+        "Execution Plan\n"
+            + "IntermediateCombine\n"
+            + "  Alternative(servers=[1])\n"
+            + "    PinotLogicalExchange(distribution=[broadcast])\n"
+            + "      LEAF_STAGE_COMBINE_OPERATOR\n"
+            + "        STREAMING_INSTANCE_RESPONSE\n"
+            + "          STREAMING_COMBINE_SELECT\n"
+            + "            SELECT_STREAMING(segment=[any], totalDocs=[any])\n"
+            + "              TRANSFORM(expressions=[[\"'1'\"]])\n"
+            + "                PROJECT(columns=[[]])\n"
+            + "                  DOC_ID_SET(maxDocs=[10000])\n"
+            + "                    FILTER_MATCH_ENTIRE_SEGMENT(numDocs=[any])\n"
+            + "            SELECT_STREAMING(segment=[any], totalDocs=[any])\n"
+            + "              TRANSFORM(expressions=[[\"'1'\"]])\n"
+            + "                PROJECT(columns=[[]])\n"
+            + "                  DOC_ID_SET(maxDocs=[10000])\n"
+            + "                    FILTER_MATCH_ENTIRE_SEGMENT(numDocs=[any])\n"
+            + "            SELECT_STREAMING(segment=[any], totalDocs=[any])\n"
+            + "              TRANSFORM(expressions=[[\"'1'\"]])\n"
+            + "                PROJECT(columns=[[]])\n"
+            + "                  DOC_ID_SET(maxDocs=[10000])\n"
+            + "                    FILTER_MATCH_ENTIRE_SEGMENT(numDocs=[any])\n"
+            + "            SELECT_STREAMING(segment=[any], totalDocs=[any])\n"
+            + "              TRANSFORM(expressions=[[\"'1'\"]])\n"
+            + "                PROJECT(columns=[[]])\n"
+            + "                  DOC_ID_SET(maxDocs=[10000])\n"
+            + "                    FILTER_MATCH_ENTIRE_SEGMENT(numDocs=[any])\n"
+            + "            SELECT_STREAMING(segment=[any], totalDocs=[any])\n"
+            + "              TRANSFORM(expressions=[[\"'1'\"]])\n"
+            + "                PROJECT(columns=[[]])\n"
+            + "                  DOC_ID_SET(maxDocs=[10000])\n"
+            + "                    FILTER_MATCH_ENTIRE_SEGMENT(numDocs=[any])\n"
+            + "            SELECT_STREAMING(segment=[any], totalDocs=[any])\n"
+            + "              TRANSFORM(expressions=[[\"'1'\"]])\n"
+            + "                PROJECT(columns=[[]])\n"
+            + "                  DOC_ID_SET(maxDocs=[10000])\n"
+            + "                    FILTER_MATCH_ENTIRE_SEGMENT(numDocs=[any])\n"
+            + "  Alternative(servers=[1])\n"
+            + "    PinotLogicalExchange(distribution=[broadcast])\n"
+            + "      LEAF_STAGE_COMBINE_OPERATOR\n"
+            + "        STREAMING_INSTANCE_RESPONSE\n"
+            + "          STREAMING_COMBINE_SELECT\n"
+            + "            SELECT_STREAMING(segment=[any], totalDocs=[any])\n"
+            + "              TRANSFORM(expressions=[[\"'1'\"]])\n"
+            + "                PROJECT(columns=[[]])\n"
+            + "                  DOC_ID_SET(maxDocs=[10000])\n"
+            + "                    FILTER_MATCH_ENTIRE_SEGMENT(numDocs=[any])\n"
+            + "            SELECT_STREAMING(segment=[any], totalDocs=[any])\n"
+            + "              TRANSFORM(expressions=[[\"'1'\"]])\n"
+            + "                PROJECT(columns=[[]])\n"
+            + "                  DOC_ID_SET(maxDocs=[10000])\n"
+            + "                    FILTER_MATCH_ENTIRE_SEGMENT(numDocs=[any])\n"
+            + "            SELECT_STREAMING(segment=[any], totalDocs=[any])\n"
+            + "              TRANSFORM(expressions=[[\"'1'\"]])\n"
+            + "                PROJECT(columns=[[]])\n"
+            + "                  DOC_ID_SET(maxDocs=[10000])\n"
+            + "                    FILTER_MATCH_ENTIRE_SEGMENT(numDocs=[any])\n"
+            + "            SELECT_STREAMING(segment=[any], totalDocs=[any])\n"
+            + "              TRANSFORM(expressions=[[\"'1'\"]])\n"
+            + "                PROJECT(columns=[[]])\n"
+            + "                  DOC_ID_SET(maxDocs=[10000])\n"
+            + "                    FILTER_MATCH_ENTIRE_SEGMENT(numDocs=[any])\n"
+            + "            SELECT_STREAMING(segment=[any], totalDocs=[any])\n"
+            + "              TRANSFORM(expressions=[[\"'1'\"]])\n"
+            + "                PROJECT(columns=[[]])\n"
+            + "                  DOC_ID_SET(maxDocs=[10000])\n"
+            + "                    FILTER_MATCH_ENTIRE_SEGMENT(numDocs=[any])\n"
+            + "            SELECT_STREAMING(segment=[any], totalDocs=[any])\n"
+            + "              TRANSFORM(expressions=[[\"'1'\"]])\n"
+            + "                PROJECT(columns=[[]])\n"
+            + "                  DOC_ID_SET(maxDocs=[10000])\n"
+            + "                    FILTER_MATCH_ENTIRE_SEGMENT(numDocs=[any])\n");
+    //@formatter:on
+  }
+
+  @Test
+  public void simpleQueryLogical() {
+    explainLogical("SELECT 1 FROM mytable",
+        //@formatter:off
+        "Execution Plan\n"
+            + "LogicalProject(EXPR$0=[1])\n"
+            + "  LogicalTableScan(table=[[default, mytable]])\n");
+    //@formatter:on
+  }
+
+  @Override
+  protected String getTableName() {
+    return _tableName;
+  }

Review Comment:
   I just copied it from MultiStageEngineIntegrationTest, which was actually changing the value :D. I'm going to remove it.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


Re: [PR] Multi stage explain [pinot]

Posted by "gortiz (via GitHub)" <gi...@apache.org>.
gortiz commented on code in PR #13733:
URL: https://github.com/apache/pinot/pull/13733#discussion_r1747205731


##########
pinot-core/src/main/java/org/apache/pinot/core/query/executor/ServerQueryExecutorV1Impl.java:
##########
@@ -691,4 +662,132 @@ private void addPrunerStats(InstanceResponseBlock instanceResponse, SegmentPrune
     instanceResponse.addMetadata(MetadataKey.NUM_SEGMENTS_PRUNED_BY_VALUE.getName(),
         String.valueOf(prunerStats.getValuePruned()));
   }
+
+  private List<IndexSegment> selectSegments(List<IndexSegment> indexSegments, QueryContext queryContext,
+      TimerContext timerContext, ExecutorService executorService, SegmentPrunerStatistics prunerStats) {
+    List<IndexSegment> selectedSegments;
+    if ((queryContext.getFilter() != null && queryContext.getFilter().isConstantFalse()) || (
+        queryContext.getHavingFilter() != null && queryContext.getHavingFilter().isConstantFalse())) {
+      selectedSegments = Collections.emptyList();
+    } else {
+      TimerContext.Timer segmentPruneTimer = timerContext.startNewPhaseTimer(ServerQueryPhase.SEGMENT_PRUNING);
+      selectedSegments = _segmentPrunerService.prune(indexSegments, queryContext, prunerStats, executorService);
+      segmentPruneTimer.stopAndRecord();
+    }
+    return selectedSegments;
+  }
+
+  private Plan planCombineQuery(QueryContext queryContext, TimerContext timerContext, ExecutorService executorService,
+      @Nullable ResultsBlockStreamer streamer, List<SegmentContext> selectedSegmentContexts) {
+    TimerContext.Timer planBuildTimer = timerContext.startNewPhaseTimer(ServerQueryPhase.BUILD_QUERY_PLAN);
+
+    Plan queryPlan;
+    if (streamer != null) {
+      queryPlan = _planMaker.makeStreamingInstancePlan(selectedSegmentContexts, queryContext, executorService,
+          streamer, _serverMetrics);
+    } else {
+      queryPlan = _planMaker.makeInstancePlan(selectedSegmentContexts, queryContext, executorService, _serverMetrics);
+    }
+    planBuildTimer.stopAndRecord();
+    return queryPlan;
+  }
+
+  private InstanceResponseBlock execute(TableDataManager tableDataManager,
+      List<IndexSegment> indexSegments, QueryContext queryContext, TimerContext timerContext,
+      ExecutorService executorService, ResultsBlockStreamer streamer, boolean enableStreaming,
+      List<IndexSegment> selectedSegments, List<SegmentContext> selectedSegmentContexts)
+      throws TimeoutException {
+    InstanceResponseBlock instanceResponse;
+    @Nullable
+    ResultsBlockStreamer actualStreamer = enableStreaming ? streamer : null;
+    switch (queryContext.getExplain()) {
+      case DESCRIPTION:
+        instanceResponse = executeDescribeExplain(indexSegments, queryContext, timerContext, executorService,
+            actualStreamer, selectedSegmentContexts);
+        break;
+      case NODE:
+        instanceResponse = executeNodeExplain(queryContext, timerContext, executorService, actualStreamer,
+            selectedSegmentContexts);
+        break;
+      case NONE:
+        instanceResponse = executeQuery(queryContext, timerContext, executorService, actualStreamer,
+            selectedSegmentContexts);
+        break;
+      default:
+        throw new IllegalStateException("Unsupported explain mode: " + queryContext.getExplain());
+    }
+    return instanceResponse;
+  }
+
+  private InstanceResponseBlock executeQuery(QueryContext queryContext, TimerContext timerContext,
+      ExecutorService executorService, @Nullable ResultsBlockStreamer streamer,
+      List<SegmentContext> selectedSegmentContexts)
+      throws TimeoutException {
+    if (selectedSegmentContexts.isEmpty()) {
+      return new InstanceResponseBlock(ResultsBlockUtils.buildEmptyQueryResults(queryContext));
+    }
+    Plan queryPlan = planCombineQuery(queryContext, timerContext, executorService, streamer, selectedSegmentContexts);
+
+    TimerContext.Timer planExecTimer = timerContext.startNewPhaseTimer(ServerQueryPhase.QUERY_PLAN_EXECUTION);
+    InstanceResponseBlock instanceResponse = queryPlan.execute();
+    planExecTimer.stopAndRecord();
+    return instanceResponse;
+  }
+
+  private InstanceResponseBlock executeNodeExplain(QueryContext queryContext, TimerContext timerContext,
+      ExecutorService executorService, @Nullable ResultsBlockStreamer streamer,
+      List<SegmentContext> selectedSegmentContexts) {
+
+    if (selectedSegmentContexts.isEmpty()) {
+      ExplainInfo emptyNode = new ExplainInfo(ExplainPlanRows.ALL_SEGMENTS_PRUNED_ON_SERVER);
+      ExplainV2ResultBlock explainResults = new ExplainV2ResultBlock(queryContext, emptyNode);
+
+      return new InstanceResponseBlock(explainResults);
+    }
+
+    Plan queryPlan = planCombineQuery(queryContext, timerContext, executorService, streamer,
+        selectedSegmentContexts);
+
+    TimerContext.Timer planExecTimer = timerContext.startNewPhaseTimer(ServerQueryPhase.QUERY_PLAN_EXECUTION);
+
+    InstanceResponseBlock instanceResponse;
+    InstanceResponseOperator responseOperator = (InstanceResponseOperator) queryPlan.getPlanNode().run();
+    try {
+      responseOperator.prefetchAll();
+
+      ExplainInfo explainInfo = responseOperator.getExplainInfo();
+      ExplainV2ResultBlock block = new ExplainV2ResultBlock(queryContext, explainInfo);
+
+      instanceResponse = new InstanceResponseBlock(block);
+      // TODO: Study if this metadata can/should be added

Review Comment:
   Do not forget about this TODO



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


Re: [PR] Multi stage explain [pinot]

Posted by "gortiz (via GitHub)" <gi...@apache.org>.
gortiz commented on code in PR #13733:
URL: https://github.com/apache/pinot/pull/13733#discussion_r1750345793


##########
pinot-core/src/main/java/org/apache/pinot/core/operator/query/AggregationOperator.java:
##########
@@ -100,4 +103,21 @@ public String toExplainString() {
 
     return stringBuilder.append(')').toString();
   }
+
+  @Override
+  protected String getExplainName() {
+    return EXPLAIN_NAME;
+  }
+
+  @Override
+  protected void explainAttributes(ExplainAttributeBuilder attributeBuilder) {
+    super.explainAttributes(attributeBuilder);
+    if (_aggregationFunctions.length == 0) {
+      return;
+    }
+    List<String> aggregations = Arrays.stream(_aggregationFunctions)
+        .map(AggregationFunction::toExplainString)
+        .collect(Collectors.toList());
+    attributeBuilder.putLongIdempotent("numAggregations", aggregations.size());

Review Comment:
   Solved. Now it includes the whole list



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


Re: [PR] Multi stage explain [pinot]

Posted by "yashmayya (via GitHub)" <gi...@apache.org>.
yashmayya commented on code in PR #13733:
URL: https://github.com/apache/pinot/pull/13733#discussion_r1750199715


##########
pinot-query-runtime/src/main/java/org/apache/pinot/query/runtime/plan/PhysicalPlanVisitor.java:
##########
@@ -56,115 +58,151 @@
 
 
 /**
- * This visitor constructs a physical plan of operators from a {@link PlanNode} tree. Note that
- * this works only for the intermediate stage nodes, leaf stage nodes are expected to compile into
+ * A class used to transform PlanNodes (considered logical) into MultiStageOperator (considered physical).
+ *
+ * Note that this works only for the intermediate stage nodes, leaf stage nodes are expected to compile into
  * v1 operators at this point in time.
  *
- * <p>This class should be used statically via {@link #walkPlanNode(PlanNode, OpChainExecutionContext)}
+ * <p><b>Notice</b>: Here <em>physical</em> is used in the context of multi-stage engine, which means it transforms
+ * logical PlanNodes into MultiStageOperator.
+ * Probably another adjective should be used given physical means different things for Calcite and single-stage</p>
  */
-public class PhysicalPlanVisitor implements PlanNodeVisitor<MultiStageOperator, OpChainExecutionContext> {
+// TODO: rename as PhysicalPlaner or PlanNodeToOperator
+public class PhysicalPlanVisitor {
+
+  private PhysicalPlanVisitor() {
+  }
 
-  private static final PhysicalPlanVisitor INSTANCE = new PhysicalPlanVisitor();
+  public static OpChain planToOperators(PlanNode node, OpChainExecutionContext context) {
+    return planToOperators(node, context, (planNode, operator) -> {
+      // Do nothing
+    });
+  }
 
-  public static OpChain walkPlanNode(PlanNode node, OpChainExecutionContext context) {
-    MultiStageOperator root = node.visit(INSTANCE, context);
+  /**
+   * Like {@link #planToOperators(PlanNode, OpChainExecutionContext, BiConsumer)} but keeps tracking of the original
+   * PlanNode that created each MultiStageOperator
+   * @param tracker a consumer that will be called each time a MultiStageOperator is created.
+   * @return
+   */
+  public static OpChain planToOperators(PlanNode node, OpChainExecutionContext context,
+      BiConsumer<PlanNode, MultiStageOperator> tracker) {
+    MyVisitor visitor = new MyVisitor(tracker);
+    MultiStageOperator root = node.visit(visitor, context);
+    tracker.accept(node, root);
     return new OpChain(context, root);
   }
 
-  private <T extends PlanNode> MultiStageOperator visit(T node, OpChainExecutionContext context) {
-    if (context.getLeafStageContext() != null && context.getLeafStageContext().getLeafStageBoundaryNode() == node) {
-      ServerPlanRequestContext leafStageContext = context.getLeafStageContext();
-      return new LeafStageTransferableBlockOperator(context, leafStageContext.getServerQueryRequests(),
-          leafStageContext.getLeafStageBoundaryNode().getDataSchema(), leafStageContext.getLeafQueryExecutor(),
-          leafStageContext.getExecutorService());
-    } else {
-      return node.visit(this, context);
+  private static class MyVisitor implements PlanNodeVisitor<MultiStageOperator, OpChainExecutionContext> {

Review Comment:
   Was this change done to accommodate the `PlanNode` <-> `MultiStageOperator` tracking mechanism? Couldn't we just use a static factory method instead? Even if not, `MyVisitor` doesn't really convey the purpose very well (although it's doing what `PhysicalPlanVisitor` itself was doing earlier and since it's now an inner class maybe we could just leave it at something like `Visitor`).



##########
pinot-core/src/main/java/org/apache/pinot/core/query/executor/ServerQueryExecutorV1Impl.java:
##########
@@ -691,4 +662,132 @@ private void addPrunerStats(InstanceResponseBlock instanceResponse, SegmentPrune
     instanceResponse.addMetadata(MetadataKey.NUM_SEGMENTS_PRUNED_BY_VALUE.getName(),
         String.valueOf(prunerStats.getValuePruned()));
   }
+
+  private List<IndexSegment> selectSegments(List<IndexSegment> indexSegments, QueryContext queryContext,
+      TimerContext timerContext, ExecutorService executorService, SegmentPrunerStatistics prunerStats) {
+    List<IndexSegment> selectedSegments;
+    if ((queryContext.getFilter() != null && queryContext.getFilter().isConstantFalse()) || (
+        queryContext.getHavingFilter() != null && queryContext.getHavingFilter().isConstantFalse())) {
+      selectedSegments = Collections.emptyList();
+    } else {
+      TimerContext.Timer segmentPruneTimer = timerContext.startNewPhaseTimer(ServerQueryPhase.SEGMENT_PRUNING);
+      selectedSegments = _segmentPrunerService.prune(indexSegments, queryContext, prunerStats, executorService);
+      segmentPruneTimer.stopAndRecord();
+    }
+    return selectedSegments;
+  }
+
+  private Plan planCombineQuery(QueryContext queryContext, TimerContext timerContext, ExecutorService executorService,
+      @Nullable ResultsBlockStreamer streamer, List<SegmentContext> selectedSegmentContexts) {
+    TimerContext.Timer planBuildTimer = timerContext.startNewPhaseTimer(ServerQueryPhase.BUILD_QUERY_PLAN);
+
+    Plan queryPlan;
+    if (streamer != null) {
+      queryPlan = _planMaker.makeStreamingInstancePlan(selectedSegmentContexts, queryContext, executorService,
+          streamer, _serverMetrics);
+    } else {
+      queryPlan = _planMaker.makeInstancePlan(selectedSegmentContexts, queryContext, executorService, _serverMetrics);
+    }
+    planBuildTimer.stopAndRecord();
+    return queryPlan;
+  }
+
+  private InstanceResponseBlock execute(TableDataManager tableDataManager,
+      List<IndexSegment> indexSegments, QueryContext queryContext, TimerContext timerContext,
+      ExecutorService executorService, ResultsBlockStreamer streamer, boolean enableStreaming,
+      List<IndexSegment> selectedSegments, List<SegmentContext> selectedSegmentContexts)

Review Comment:
   nit: couple of unused parameters here



##########
pinot-core/src/main/java/org/apache/pinot/core/operator/query/SelectionPartiallyOrderedByAscOperator.java:
##########
@@ -77,6 +78,6 @@ public int getNumDocsScanned() {
 
   @Override
   protected String getExplainName() {
-    return EXPLAIN_NAME;
+    return CaseFormat.UPPER_UNDERSCORE.to(CaseFormat.UPPER_CAMEL, EXPLAIN_NAME);

Review Comment:
   This will also change the v1 explain plan right? I guess we want to avoid that?



##########
pinot-query-planner/src/main/java/org/apache/pinot/query/planner/logical/PlanNodeToRelConverter.java:
##########
@@ -0,0 +1,468 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.query.planner.logical;
+
+import com.google.common.base.Preconditions;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.List;
+import java.util.stream.Collectors;
+import org.apache.calcite.plan.RelOptCluster;
+import org.apache.calcite.plan.RelTraitSet;
+import org.apache.calcite.rel.RelCollation;
+import org.apache.calcite.rel.RelCollations;
+import org.apache.calcite.rel.RelDistribution;
+import org.apache.calcite.rel.RelDistributions;
+import org.apache.calcite.rel.RelNode;
+import org.apache.calcite.rel.core.SetOp;
+import org.apache.calcite.rel.core.Window;
+import org.apache.calcite.rel.logical.LogicalIntersect;
+import org.apache.calcite.rel.logical.LogicalMinus;
+import org.apache.calcite.rel.logical.LogicalSort;
+import org.apache.calcite.rel.logical.LogicalUnion;
+import org.apache.calcite.rel.logical.LogicalWindow;
+import org.apache.calcite.rel.type.RelDataType;
+import org.apache.calcite.rex.RexLiteral;
+import org.apache.calcite.rex.RexNode;
+import org.apache.calcite.rex.RexWindowBound;
+import org.apache.calcite.rex.RexWindowBounds;
+import org.apache.calcite.sql.SqlAggFunction;
+import org.apache.calcite.tools.RelBuilder;
+import org.apache.calcite.util.ImmutableBitSet;
+import org.apache.pinot.common.utils.DatabaseUtils;
+import org.apache.pinot.core.operator.ExplainAttributeBuilder;
+import org.apache.pinot.core.plan.PinotExplainedRelNode;
+import org.apache.pinot.query.planner.plannode.AggregateNode;
+import org.apache.pinot.query.planner.plannode.ExchangeNode;
+import org.apache.pinot.query.planner.plannode.ExplainedNode;
+import org.apache.pinot.query.planner.plannode.FilterNode;
+import org.apache.pinot.query.planner.plannode.JoinNode;
+import org.apache.pinot.query.planner.plannode.MailboxReceiveNode;
+import org.apache.pinot.query.planner.plannode.MailboxSendNode;
+import org.apache.pinot.query.planner.plannode.PlanNode;
+import org.apache.pinot.query.planner.plannode.PlanNodeVisitor;
+import org.apache.pinot.query.planner.plannode.ProjectNode;
+import org.apache.pinot.query.planner.plannode.SetOpNode;
+import org.apache.pinot.query.planner.plannode.SortNode;
+import org.apache.pinot.query.planner.plannode.TableScanNode;
+import org.apache.pinot.query.planner.plannode.ValueNode;
+import org.apache.pinot.query.planner.plannode.WindowNode;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+
+/**
+ * Converts a {@link PlanNode} into a {@link RelNode}.
+ *
+ * This class is used to convert serialized plan nodes into RelNodes so they can be used when explain with
+ * implementation is requested. Therefore some nodes may be transformed in a way that loses information that is
+ * required to create an actual executable plan but not necessary in order to describe the plan.
+ */
+public final class PlanNodeToRelConverter {
+  private static final Logger LOGGER = LoggerFactory.getLogger(PlanNodeToRelConverter.class);
+
+  private PlanNodeToRelConverter() {
+  }
+
+  public static RelNode convert(RelBuilder builder, PlanNode planNode) {
+    ConverterVisitor visitor = new ConverterVisitor(builder);
+    planNode.visit(visitor, null);
+
+    return visitor.build();
+  }
+
+  private static class ConverterVisitor implements PlanNodeVisitor<Void, Void> {

Review Comment:
   Looks like the static inner class does all the heavy lifting here and the outer class doesn't really have any logic -- why not simply make `PlanNodeToRelConverter` itself implement the `PlanNodeVisitor` interface and get rid of this static inner class?



##########
pinot-core/src/main/java/org/apache/pinot/core/query/request/context/utils/QueryContextConverterUtils.java:
##########
@@ -155,11 +157,26 @@ public static QueryContext getQueryContext(PinotQuery pinotQuery) {
       }
     }
 
+    ExplainMode explainMode;
+    if (!pinotQuery.isExplain()) {
+      explainMode = ExplainMode.NONE;
+    } else if (isUsingV1(pinotQuery)) {
+      explainMode = ExplainMode.NODE;
+    } else {
+      explainMode = ExplainMode.DESCRIPTION;
+    }
+
     return new QueryContext.Builder().setTableName(tableName).setSubquery(subquery)
         .setSelectExpressions(selectExpressions).setDistinct(distinct).setAliasList(aliasList).setFilter(filter)
         .setGroupByExpressions(groupByExpressions).setOrderByExpressions(orderByExpressions)
         .setHavingFilter(havingFilter).setLimit(pinotQuery.getLimit()).setOffset(pinotQuery.getOffset())
         .setQueryOptions(pinotQuery.getQueryOptions()).setExpressionOverrideHints(expressionContextOverrideHints)
-        .setExplain(pinotQuery.isExplain()).build();
+        .setExplain(explainMode).build();
+  }
+
+  private static boolean isUsingV1(PinotQuery pinotQuery) {

Review Comment:
   Shouldn't this be `isUsingV2` or `isUsingMultistageEngine`? Also we could just use `QueryOptionUtils::isUseMultistageEngine` instead?



##########
pinot-query-runtime/src/main/java/org/apache/pinot/query/service/dispatch/DispatchClient.java:
##########
@@ -50,12 +51,17 @@ public ManagedChannel getChannel() {
   }
 
   public void submit(Worker.QueryRequest request, QueryServerInstance virtualServer, Deadline deadline,
-      Consumer<AsyncQueryDispatchResponse> callback) {
-    _dispatchStub.withDeadline(deadline).submit(request, new DispatchObserver(virtualServer, callback));
+      Consumer<AsyncResponse<Worker.QueryResponse>> callback) {
+    _dispatchStub.withDeadline(deadline).submit(request, new LastValueDispatchObserver<>(virtualServer, callback));
   }
 
   public void cancel(long requestId) {
     Worker.CancelRequest cancelRequest = Worker.CancelRequest.newBuilder().setRequestId(requestId).build();
     _dispatchStub.cancel(cancelRequest, NO_OP_CANCEL_STREAM_OBSERVER);
   }
+
+  public void explain(Worker.QueryRequest request, QueryServerInstance virtualServer, Deadline deadline,
+      Consumer<AsyncResponse<List<Worker.ExplainResponse>>> callback) {
+    _dispatchStub.withDeadline(deadline).explain(request, new AllValuesDispatchObserver<>(virtualServer, callback));

Review Comment:
   I didn't quite follow why we only care about the last value from the broker <-> server stream response in the case of a regular query request and all values from the stream in the case of an explain query request?



##########
pinot-core/src/main/java/org/apache/pinot/core/operator/transform/TransformOperator.java:
##########
@@ -86,7 +88,17 @@ public String toExplainString() {
   }
 
   protected String getExplainName() {
-    return EXPLAIN_NAME;
+    return CaseFormat.UPPER_UNDERSCORE.to(CaseFormat.UPPER_CAMEL, EXPLAIN_NAME);

Review Comment:
   Same concern as above, this will also change the v1 explain plan?



##########
pinot-query-runtime/src/main/java/org/apache/pinot/query/runtime/plan/PhysicalPlanVisitor.java:
##########
@@ -56,115 +58,151 @@
 
 
 /**
- * This visitor constructs a physical plan of operators from a {@link PlanNode} tree. Note that
- * this works only for the intermediate stage nodes, leaf stage nodes are expected to compile into
+ * A class used to transform PlanNodes (considered logical) into MultiStageOperator (considered physical).
+ *
+ * Note that this works only for the intermediate stage nodes, leaf stage nodes are expected to compile into
  * v1 operators at this point in time.
  *
- * <p>This class should be used statically via {@link #walkPlanNode(PlanNode, OpChainExecutionContext)}
+ * <p><b>Notice</b>: Here <em>physical</em> is used in the context of multi-stage engine, which means it transforms
+ * logical PlanNodes into MultiStageOperator.
+ * Probably another adjective should be used given physical means different things for Calcite and single-stage</p>
  */
-public class PhysicalPlanVisitor implements PlanNodeVisitor<MultiStageOperator, OpChainExecutionContext> {
+// TODO: rename as PhysicalPlaner or PlanNodeToOperator
+public class PhysicalPlanVisitor {
+
+  private PhysicalPlanVisitor() {
+  }
 
-  private static final PhysicalPlanVisitor INSTANCE = new PhysicalPlanVisitor();
+  public static OpChain planToOperators(PlanNode node, OpChainExecutionContext context) {
+    return planToOperators(node, context, (planNode, operator) -> {
+      // Do nothing
+    });
+  }
 
-  public static OpChain walkPlanNode(PlanNode node, OpChainExecutionContext context) {
-    MultiStageOperator root = node.visit(INSTANCE, context);
+  /**
+   * Like {@link #planToOperators(PlanNode, OpChainExecutionContext, BiConsumer)} but keeps tracking of the original
+   * PlanNode that created each MultiStageOperator
+   * @param tracker a consumer that will be called each time a MultiStageOperator is created.
+   * @return
+   */
+  public static OpChain planToOperators(PlanNode node, OpChainExecutionContext context,
+      BiConsumer<PlanNode, MultiStageOperator> tracker) {
+    MyVisitor visitor = new MyVisitor(tracker);
+    MultiStageOperator root = node.visit(visitor, context);
+    tracker.accept(node, root);
     return new OpChain(context, root);
   }
 
-  private <T extends PlanNode> MultiStageOperator visit(T node, OpChainExecutionContext context) {
-    if (context.getLeafStageContext() != null && context.getLeafStageContext().getLeafStageBoundaryNode() == node) {
-      ServerPlanRequestContext leafStageContext = context.getLeafStageContext();
-      return new LeafStageTransferableBlockOperator(context, leafStageContext.getServerQueryRequests(),
-          leafStageContext.getLeafStageBoundaryNode().getDataSchema(), leafStageContext.getLeafQueryExecutor(),
-          leafStageContext.getExecutorService());
-    } else {
-      return node.visit(this, context);
+  private static class MyVisitor implements PlanNodeVisitor<MultiStageOperator, OpChainExecutionContext> {
+    private final BiConsumer<PlanNode, MultiStageOperator> _tracker;
+
+    public MyVisitor(BiConsumer<PlanNode, MultiStageOperator> tracker) {
+      _tracker = tracker;
     }
-  }
 
-  @Override
-  public MultiStageOperator visitMailboxReceive(MailboxReceiveNode node, OpChainExecutionContext context) {
-    if (node.isSort()) {
-      return new SortedMailboxReceiveOperator(context, node);
-    } else {
-      return new MailboxReceiveOperator(context, node);
+    private <T extends PlanNode> MultiStageOperator visit(T node, OpChainExecutionContext context) {
+      MultiStageOperator result;
+      if (context.getLeafStageContext() != null && context.getLeafStageContext().getLeafStageBoundaryNode() == node) {
+        ServerPlanRequestContext leafStageContext = context.getLeafStageContext();
+        result = new LeafStageTransferableBlockOperator(context, leafStageContext.getServerQueryRequests(),
+            leafStageContext.getLeafStageBoundaryNode().getDataSchema(), leafStageContext.getLeafQueryExecutor(),
+            leafStageContext.getExecutorService());
+      } else {
+        result = node.visit(this, context);
+      }
+      _tracker.accept(node, result);
+      return result;
     }
-  }
 
-  @Override
-  public MultiStageOperator visitMailboxSend(MailboxSendNode node, OpChainExecutionContext context) {
-    return new MailboxSendOperator(context, visit(node.getInputs().get(0), context), node);
-  }
+    @Override
+    public MultiStageOperator visitMailboxReceive(MailboxReceiveNode node, OpChainExecutionContext context) {
+      if (node.isSort()) {
+        return new SortedMailboxReceiveOperator(context, node);
+      } else {
+        return new MailboxReceiveOperator(context, node);
+      }
+    }
 
-  @Override
-  public MultiStageOperator visitAggregate(AggregateNode node, OpChainExecutionContext context) {
-    return new AggregateOperator(context, visit(node.getInputs().get(0), context), node);
-  }
+    @Override
+    public MultiStageOperator visitMailboxSend(MailboxSendNode node, OpChainExecutionContext context) {
+      return new MailboxSendOperator(context, visit(node.getInputs().get(0), context), node);
+    }
 
-  @Override
-  public MultiStageOperator visitWindow(WindowNode node, OpChainExecutionContext context) {
-    PlanNode input = node.getInputs().get(0);
-    return new WindowAggregateOperator(context, visit(input, context), input.getDataSchema(), node);
-  }
+    @Override
+    public MultiStageOperator visitAggregate(AggregateNode node, OpChainExecutionContext context) {
+      return new AggregateOperator(context, visit(node.getInputs().get(0), context), node);
+    }
 
-  @Override
-  public MultiStageOperator visitSetOp(SetOpNode setOpNode, OpChainExecutionContext context) {
-    List<MultiStageOperator> inputOperators = new ArrayList<>(setOpNode.getInputs().size());
-    for (PlanNode input : setOpNode.getInputs()) {
-      inputOperators.add(visit(input, context));
+    @Override
+    public MultiStageOperator visitWindow(WindowNode node, OpChainExecutionContext context) {
+      PlanNode input = node.getInputs().get(0);
+      return new WindowAggregateOperator(context, visit(input, context), input.getDataSchema(), node);
     }
-    switch (setOpNode.getSetOpType()) {
-      case UNION:
-        return new UnionOperator(context, inputOperators, setOpNode.getInputs().get(0).getDataSchema());
-      case INTERSECT:
-        return setOpNode.isAll() ? new IntersectAllOperator(context, inputOperators,
-            setOpNode.getInputs().get(0).getDataSchema())
-            : new IntersectOperator(context, inputOperators, setOpNode.getInputs().get(0).getDataSchema());
-      case MINUS:
-        return setOpNode.isAll() ? new MinusAllOperator(context, inputOperators,
-            setOpNode.getInputs().get(0).getDataSchema())
-            : new MinusOperator(context, inputOperators, setOpNode.getInputs().get(0).getDataSchema());
-      default:
-        throw new IllegalStateException("Unsupported SetOpType: " + setOpNode.getSetOpType());
+
+    @Override
+    public MultiStageOperator visitSetOp(SetOpNode setOpNode, OpChainExecutionContext context) {
+      List<MultiStageOperator> inputOperators = new ArrayList<>(setOpNode.getInputs().size());
+      for (PlanNode input : setOpNode.getInputs()) {
+        inputOperators.add(visit(input, context));
+      }
+      switch (setOpNode.getSetOpType()) {
+        case UNION:
+          return new UnionOperator(context, inputOperators, setOpNode.getInputs().get(0).getDataSchema());
+        case INTERSECT:
+          return setOpNode.isAll() ? new IntersectAllOperator(context, inputOperators,
+              setOpNode.getInputs().get(0).getDataSchema())
+              : new IntersectOperator(context, inputOperators, setOpNode.getInputs().get(0).getDataSchema());
+        case MINUS:
+          return setOpNode.isAll() ? new MinusAllOperator(context, inputOperators,
+              setOpNode.getInputs().get(0).getDataSchema())
+              : new MinusOperator(context, inputOperators, setOpNode.getInputs().get(0).getDataSchema());
+        default:
+          throw new IllegalStateException("Unsupported SetOpType: " + setOpNode.getSetOpType());
+      }
     }
-  }
 
-  @Override
-  public MultiStageOperator visitExchange(ExchangeNode exchangeNode, OpChainExecutionContext context) {
-    throw new UnsupportedOperationException("ExchangeNode should not be visited");
-  }
+    @Override
+    public MultiStageOperator visitExchange(ExchangeNode exchangeNode, OpChainExecutionContext context) {
+      throw new UnsupportedOperationException("ExchangeNode should not be visited");
+    }
 
-  @Override
-  public MultiStageOperator visitFilter(FilterNode node, OpChainExecutionContext context) {
-    return new FilterOperator(context, visit(node.getInputs().get(0), context), node);
-  }
+    @Override
+    public MultiStageOperator visitFilter(FilterNode node, OpChainExecutionContext context) {
+      return new FilterOperator(context, visit(node.getInputs().get(0), context), node);
+    }
 
-  @Override
-  public MultiStageOperator visitJoin(JoinNode node, OpChainExecutionContext context) {
-    List<PlanNode> inputs = node.getInputs();
-    PlanNode left = inputs.get(0);
-    PlanNode right = inputs.get(1);
-    return new HashJoinOperator(context, visit(left, context), left.getDataSchema(), visit(right, context), node);
-  }
+    @Override
+    public MultiStageOperator visitJoin(JoinNode node, OpChainExecutionContext context) {
+      List<PlanNode> inputs = node.getInputs();
+      PlanNode left = inputs.get(0);
+      PlanNode right = inputs.get(1);
+      return new HashJoinOperator(context, visit(left, context), left.getDataSchema(), visit(right, context), node);
+    }
 
-  @Override
-  public MultiStageOperator visitProject(ProjectNode node, OpChainExecutionContext context) {
-    PlanNode input = node.getInputs().get(0);
-    return new TransformOperator(context, visit(input, context), input.getDataSchema(), node);
-  }
+    @Override
+    public MultiStageOperator visitProject(ProjectNode node, OpChainExecutionContext context) {
+      PlanNode input = node.getInputs().get(0);
+      return new TransformOperator(context, visit(input, context), input.getDataSchema(), node);
+    }
 
-  @Override
-  public MultiStageOperator visitSort(SortNode node, OpChainExecutionContext context) {
-    return new SortOperator(context, visit(node.getInputs().get(0), context), node);
-  }
+    @Override
+    public MultiStageOperator visitSort(SortNode node, OpChainExecutionContext context) {
+      return new SortOperator(context, visit(node.getInputs().get(0), context), node);
+    }
 
-  @Override
-  public MultiStageOperator visitTableScan(TableScanNode node, OpChainExecutionContext context) {
-    throw new UnsupportedOperationException("Stage node of type TableScanNode is not supported!");
-  }
+    @Override
+    public MultiStageOperator visitTableScan(TableScanNode node, OpChainExecutionContext context) {
+      throw new UnsupportedOperationException("Stage node of type TableScanNode is not supported!");
+    }
 
-  @Override
-  public MultiStageOperator visitValue(ValueNode node, OpChainExecutionContext context) {
-    return new LiteralValueOperator(context, node);
+    @Override
+    public MultiStageOperator visitValue(ValueNode node, OpChainExecutionContext context) {
+      return new LiteralValueOperator(context, node);
+    }
+
+    @Override
+    public MultiStageOperator visitExplained(ExplainedNode node, OpChainExecutionContext context) {
+      throw new UnsupportedOperationException("Stage node of type ExplainedNode is not supported!");

Review Comment:
   nit: I guess this is outdated since StageNode was renamed to PlanNode in https://github.com/apache/pinot/pull/10735?



##########
pinot-query-planner/src/main/java/org/apache/pinot/query/planner/logical/PlanNodeToRelConverter.java:
##########
@@ -0,0 +1,468 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.query.planner.logical;
+
+import com.google.common.base.Preconditions;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.List;
+import java.util.stream.Collectors;
+import org.apache.calcite.plan.RelOptCluster;
+import org.apache.calcite.plan.RelTraitSet;
+import org.apache.calcite.rel.RelCollation;
+import org.apache.calcite.rel.RelCollations;
+import org.apache.calcite.rel.RelDistribution;
+import org.apache.calcite.rel.RelDistributions;
+import org.apache.calcite.rel.RelNode;
+import org.apache.calcite.rel.core.SetOp;
+import org.apache.calcite.rel.core.Window;
+import org.apache.calcite.rel.logical.LogicalIntersect;
+import org.apache.calcite.rel.logical.LogicalMinus;
+import org.apache.calcite.rel.logical.LogicalSort;
+import org.apache.calcite.rel.logical.LogicalUnion;
+import org.apache.calcite.rel.logical.LogicalWindow;
+import org.apache.calcite.rel.type.RelDataType;
+import org.apache.calcite.rex.RexLiteral;
+import org.apache.calcite.rex.RexNode;
+import org.apache.calcite.rex.RexWindowBound;
+import org.apache.calcite.rex.RexWindowBounds;
+import org.apache.calcite.sql.SqlAggFunction;
+import org.apache.calcite.tools.RelBuilder;
+import org.apache.calcite.util.ImmutableBitSet;
+import org.apache.pinot.common.utils.DatabaseUtils;
+import org.apache.pinot.core.operator.ExplainAttributeBuilder;
+import org.apache.pinot.core.plan.PinotExplainedRelNode;
+import org.apache.pinot.query.planner.plannode.AggregateNode;
+import org.apache.pinot.query.planner.plannode.ExchangeNode;
+import org.apache.pinot.query.planner.plannode.ExplainedNode;
+import org.apache.pinot.query.planner.plannode.FilterNode;
+import org.apache.pinot.query.planner.plannode.JoinNode;
+import org.apache.pinot.query.planner.plannode.MailboxReceiveNode;
+import org.apache.pinot.query.planner.plannode.MailboxSendNode;
+import org.apache.pinot.query.planner.plannode.PlanNode;
+import org.apache.pinot.query.planner.plannode.PlanNodeVisitor;
+import org.apache.pinot.query.planner.plannode.ProjectNode;
+import org.apache.pinot.query.planner.plannode.SetOpNode;
+import org.apache.pinot.query.planner.plannode.SortNode;
+import org.apache.pinot.query.planner.plannode.TableScanNode;
+import org.apache.pinot.query.planner.plannode.ValueNode;
+import org.apache.pinot.query.planner.plannode.WindowNode;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+
+/**
+ * Converts a {@link PlanNode} into a {@link RelNode}.
+ *
+ * This class is used to convert serialized plan nodes into RelNodes so they can be used when explain with
+ * implementation is requested. Therefore some nodes may be transformed in a way that loses information that is
+ * required to create an actual executable plan but not necessary in order to describe the plan.
+ */
+public final class PlanNodeToRelConverter {
+  private static final Logger LOGGER = LoggerFactory.getLogger(PlanNodeToRelConverter.class);
+
+  private PlanNodeToRelConverter() {
+  }
+
+  public static RelNode convert(RelBuilder builder, PlanNode planNode) {
+    ConverterVisitor visitor = new ConverterVisitor(builder);
+    planNode.visit(visitor, null);
+
+    return visitor.build();
+  }
+
+  private static class ConverterVisitor implements PlanNodeVisitor<Void, Void> {
+    private final RelBuilder _builder;
+
+    public ConverterVisitor(RelBuilder builder) {
+      _builder = builder;
+    }
+
+    private void visitChildren(PlanNode node) {
+      node.getInputs().forEach(input -> input.visit(this, null));
+    }
+
+    @Override
+    public Void visitAggregate(AggregateNode node, Void context) {
+      visitChildren(node);
+
+      try {
+        int[] groupKeyArr = node.getGroupKeys().stream().mapToInt(Integer::intValue).toArray();
+        RelBuilder.GroupKey groupKey = _builder.groupKey(groupKeyArr);
+
+        List<RelBuilder.AggCall> aggCalls =
+            node.getAggCalls().stream().map(functionCall -> RexExpressionUtils.toAggCall(_builder, functionCall))
+                .collect(Collectors.toList());
+
+        _builder.aggregate(groupKey, aggCalls);
+      } catch (RuntimeException e) {
+        LOGGER.warn("Failed to convert aggregate node: {}", node, e);
+        _builder.push(new PinotExplainedRelNode(_builder.getCluster(), "UnknownAggregate", Collections.emptyMap(),
+            node.getDataSchema(), readAlreadyPushedChildren(node)));
+      }

Review Comment:
   Wouldn't it be better to fail in these unknown error cases?



##########
pinot-query-planner/src/main/java/org/apache/pinot/query/planner/logical/RexExpressionUtils.java:
##########
@@ -28,27 +28,174 @@
 import java.util.Set;
 import javax.annotation.Nullable;
 import org.apache.calcite.avatica.util.ByteString;
+import org.apache.calcite.plan.RelOptCluster;
 import org.apache.calcite.rel.core.AggregateCall;
 import org.apache.calcite.rel.core.Window;
+import org.apache.calcite.rex.RexBuilder;
 import org.apache.calcite.rex.RexCall;
 import org.apache.calcite.rex.RexInputRef;
 import org.apache.calcite.rex.RexLiteral;
 import org.apache.calcite.rex.RexNode;
+import org.apache.calcite.sql.SqlAggFunction;
+import org.apache.calcite.sql.SqlIdentifier;
 import org.apache.calcite.sql.SqlKind;
 import org.apache.calcite.sql.SqlOperator;
+import org.apache.calcite.sql.SqlSyntax;
+import org.apache.calcite.sql.parser.SqlParserPos;
 import org.apache.calcite.sql.type.SqlTypeName;
+import org.apache.calcite.sql.validate.SqlNameMatchers;
+import org.apache.calcite.tools.RelBuilder;
 import org.apache.calcite.util.NlsString;
 import org.apache.calcite.util.Sarg;
+import org.apache.calcite.util.TimestampString;
 import org.apache.pinot.common.utils.DataSchema.ColumnDataType;
 import org.apache.pinot.spi.utils.BooleanUtils;
 import org.apache.pinot.spi.utils.ByteArray;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
 
 
 @SuppressWarnings({"rawtypes", "unchecked"})
 public class RexExpressionUtils {
+  public static final Logger LOGGER = LoggerFactory.getLogger(RexExpressionUtils.class);
+
   private RexExpressionUtils() {
   }
 
+  public static RexNode toRexNode(RelBuilder builder, RexExpression rexExpression) {
+    if (rexExpression instanceof RexExpression.InputRef) {
+      return toRexInputRef(builder, (RexExpression.InputRef) rexExpression);
+    } else if (rexExpression instanceof RexExpression.Literal) {
+      return toRexLiteral(builder, (RexExpression.Literal) rexExpression);
+    } else if (rexExpression instanceof RexExpression.FunctionCall) {
+      return toRexCall(builder, (RexExpression.FunctionCall) rexExpression);
+    } else {
+      throw new IllegalArgumentException("Unsupported RexExpression type: " + rexExpression.getClass().getName());
+    }
+  }
+
+  private static RexNode toRexInputRef(RelBuilder builder, RexExpression.InputRef rexExpression) {
+    return builder.field(rexExpression.getIndex());
+  }
+
+  private static RexNode toRexCall(RelBuilder builder, RexExpression.FunctionCall rexExpression) {
+    List<RexExpression> functionOperands = rexExpression.getFunctionOperands();
+    List<RexNode> operands = new ArrayList<>(functionOperands.size());
+    for (RexExpression functionOperand : functionOperands) {
+      operands.add(toRexNode(builder, functionOperand));
+    }
+    // TODO: This needs to be improved.

Review Comment:
   What exactly needs to be improved / changed here? Also in `getAggFunction`.



##########
pinot-query-planner/src/main/java/org/apache/pinot/query/planner/explain/PlanNodeSorter.java:
##########
@@ -0,0 +1,228 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.query.planner.explain;
+
+import com.google.common.collect.Sets;
+import java.util.ArrayList;
+import java.util.Comparator;
+import java.util.List;
+import java.util.TreeSet;
+import org.apache.pinot.common.proto.Plan;
+import org.apache.pinot.core.query.reduce.ExplainPlanDataTableReducer;
+import org.apache.pinot.query.planner.plannode.AggregateNode;
+import org.apache.pinot.query.planner.plannode.ExchangeNode;
+import org.apache.pinot.query.planner.plannode.ExplainedNode;
+import org.apache.pinot.query.planner.plannode.FilterNode;
+import org.apache.pinot.query.planner.plannode.JoinNode;
+import org.apache.pinot.query.planner.plannode.MailboxReceiveNode;
+import org.apache.pinot.query.planner.plannode.MailboxSendNode;
+import org.apache.pinot.query.planner.plannode.PlanNode;
+import org.apache.pinot.query.planner.plannode.PlanNodeVisitor;
+import org.apache.pinot.query.planner.plannode.ProjectNode;
+import org.apache.pinot.query.planner.plannode.SetOpNode;
+import org.apache.pinot.query.planner.plannode.SortNode;
+import org.apache.pinot.query.planner.plannode.TableScanNode;
+import org.apache.pinot.query.planner.plannode.ValueNode;
+import org.apache.pinot.query.planner.plannode.WindowNode;
+
+
+/**
+ * A utility class used to sort the plan nodes in a deterministic order.
+ *
+ * Any comparator can be passed to the sort method to sort the plan nodes, although the default comparator
+ * is used to sort the plan nodes based on the type and the attributes of the node.
+ */
+public class PlanNodeSorter {

Review Comment:
   I didn't quite get the purpose of this plan node sorter utility. In which cases would we get a non-deterministic order of plan nodes (necessitating this plan node sorter)?



##########
pinot-query-runtime/src/main/java/org/apache/pinot/query/runtime/QueryRunner.java:
##########
@@ -256,4 +262,66 @@ private Map<String, String> consolidateMetadata(Map<String, String> customProper
   public void cancel(long requestId) {
     _opChainScheduler.cancel(requestId);
   }
+
+  public StagePlan explainQuery(
+      WorkerMetadata workerMetadata, StagePlan stagePlan, Map<String, String> requestMetadata) {
+
+    if (!workerMetadata.isLeafStageWorker()) {
+      LOGGER.debug("Explain query on intermediate stages is a NOOP");
+      return stagePlan;
+    }
+    long requestId = Long.parseLong(requestMetadata.get(CommonConstants.Query.Request.MetadataKeys.REQUEST_ID));
+    long timeoutMs = Long.parseLong(requestMetadata.get(CommonConstants.Broker.Request.QueryOptionKey.TIMEOUT_MS));
+    long deadlineMs = System.currentTimeMillis() + timeoutMs;
+
+    StageMetadata stageMetadata = stagePlan.getStageMetadata();
+    Map<String, String> opChainMetadata = consolidateMetadata(stageMetadata.getCustomProperties(), requestMetadata);
+
+    if (PipelineBreakerExecutor.hasPipelineBreakers(stagePlan)) {
+      // TODO: Support pipeline breakers before merging this feature.
+      LOGGER.error("Pipeline breaker is not supported in explain query");
+      return stagePlan;
+    }

Review Comment:
   I guess we're not going to support queries with pipeline breakers with this new explain plan initially right? What do we want to show in those cases?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


Re: [PR] Multi stage explain [pinot]

Posted by "gortiz (via GitHub)" <gi...@apache.org>.
gortiz commented on code in PR #13733:
URL: https://github.com/apache/pinot/pull/13733#discussion_r1751885807


##########
pinot-common/src/main/proto/plan.proto:
##########
@@ -193,3 +194,29 @@ message WindowNode {
   int32 upperBound = 6;
   repeated Literal constants = 7;
 }
+
+message ExplainNode {
+  string type = 1;

Review Comment:
   I think I'm going to rename it as _title_. WDYT?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


Re: [PR] Multi stage explain [pinot]

Posted by "gortiz (via GitHub)" <gi...@apache.org>.
gortiz commented on code in PR #13733:
URL: https://github.com/apache/pinot/pull/13733#discussion_r1765132285


##########
pinot-spi/src/main/java/org/apache/pinot/spi/utils/CommonConstants.java:
##########
@@ -1192,6 +1209,9 @@ public enum WindowOverFlowMode {
     public static class PlanVersions {
       public static final int V1 = 1;
     }
+
+    public static final String ASK_SERVERS_FOR_EXPLAIN_PLAN = "pinot.query.explain.ask.servers";
+    public static final boolean DEFAULT_ASK_SERVERS_FOR_EXPLAIN_PLAN = true;

Review Comment:
   Changed



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


Re: [PR] Multi stage explain [pinot]

Posted by "yashmayya (via GitHub)" <gi...@apache.org>.
yashmayya commented on code in PR #13733:
URL: https://github.com/apache/pinot/pull/13733#discussion_r1751897966


##########
pinot-common/src/main/proto/plan.proto:
##########
@@ -193,3 +194,29 @@ message WindowNode {
   int32 upperBound = 6;
   repeated Literal constants = 7;
 }
+
+message ExplainNode {
+  string type = 1;

Review Comment:
   Hm, I actually think `type` or maybe `operatorType` would be clearer, but as long as there's a clarifying comment, I think either is fine since this is only internally used for serde.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


Re: [PR] Multi stage explain [pinot]

Posted by "gortiz (via GitHub)" <gi...@apache.org>.
gortiz commented on code in PR #13733:
URL: https://github.com/apache/pinot/pull/13733#discussion_r1752031714


##########
pinot-query-runtime/src/main/java/org/apache/pinot/query/runtime/QueryRunner.java:
##########
@@ -256,4 +262,66 @@ private Map<String, String> consolidateMetadata(Map<String, String> customProper
   public void cancel(long requestId) {
     _opChainScheduler.cancel(requestId);
   }
+
+  public StagePlan explainQuery(
+      WorkerMetadata workerMetadata, StagePlan stagePlan, Map<String, String> requestMetadata) {
+
+    if (!workerMetadata.isLeafStageWorker()) {
+      LOGGER.debug("Explain query on intermediate stages is a NOOP");
+      return stagePlan;
+    }
+    long requestId = Long.parseLong(requestMetadata.get(CommonConstants.Query.Request.MetadataKeys.REQUEST_ID));
+    long timeoutMs = Long.parseLong(requestMetadata.get(CommonConstants.Broker.Request.QueryOptionKey.TIMEOUT_MS));
+    long deadlineMs = System.currentTimeMillis() + timeoutMs;
+
+    StageMetadata stageMetadata = stagePlan.getStageMetadata();
+    Map<String, String> opChainMetadata = consolidateMetadata(stageMetadata.getCustomProperties(), requestMetadata);
+
+    if (PipelineBreakerExecutor.hasPipelineBreakers(stagePlan)) {
+      // TODO: Support pipeline breakers before merging this feature.
+      LOGGER.error("Pipeline breaker is not supported in explain query");
+      return stagePlan;
+    }

Review Comment:
   The main problem is that in order to have the exact physical plan in this case we would need to actually execute the pipeline breaker part. For example, a query like:
   
   ```sql
   select whatever from table1 
   where col1 in (select something from table2 where col2 = cte)
   ```
   
   the actual physical plan will depend on the result of `(select something from table2 where col2 = cte)`. Assuming that subquery is evaluated to [100, 200, 300], the query would be:
   
   ```sql
   select whatever from table1 
   where col1 in (100, 200, 300)
   ```
   
   Which could, for example, use a inverted index. I had the happy idea of generating a random set of values on the `col1` type, but that could create incorrect plans. For example imagine we randomly generate the set of values `1,2`. The query we would be using to explain would be:
   
   ```sql
   select whatever from table1 
   where col1 in (1, 2)
   ```
   
   Now imagine that values of `col1` went from 100 to 200. That query would be explained as `ALL_SEGMENTS_PRUNED_ON_SERVER` when the actual plan would be probably different



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


Re: [PR] Multi stage explain [pinot]

Posted by "gortiz (via GitHub)" <gi...@apache.org>.
gortiz merged PR #13733:
URL: https://github.com/apache/pinot/pull/13733


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


Re: [PR] Multi stage explain [pinot]

Posted by "gortiz (via GitHub)" <gi...@apache.org>.
gortiz commented on code in PR #13733:
URL: https://github.com/apache/pinot/pull/13733#discussion_r1771372620


##########
pinot-query-runtime/src/main/java/org/apache/pinot/query/runtime/operator/LeafStageTransferableBlockOperator.java:
##########
@@ -270,6 +276,69 @@ private TransferableBlock constructMetadataBlock() {
     return TransferableBlockUtils.getEndOfStreamTransferableBlock(multiStageQueryStats);
   }
 
+  public ExplainedNode explain() {
+    Preconditions.checkState(_requests.stream()
+        .allMatch(request -> request.getQueryContext().getExplain() == ExplainMode.NODE),
+        "All requests must have explain mode set to ExplainMode.NODE");
+
+    if (_executionFuture == null) {
+      _executionFuture = startExecution();
+    }
+
+    List<PlanNode> childNodes = new ArrayList<>();
+    while (true) {
+      BaseResultsBlock resultsBlock;
+      try {
+        long timeout = _context.getDeadlineMs() - System.currentTimeMillis();
+        resultsBlock = _blockingQueue.poll(timeout, TimeUnit.MILLISECONDS);
+      } catch (InterruptedException e) {
+        Thread.currentThread().interrupt();
+        throw new RuntimeException("Interrupted while waiting for results block", e);
+      }
+      if (resultsBlock == null) {
+        throw new RuntimeException("Timed out waiting for results block");
+      }
+      // Terminate when receiving exception block
+      Map<Integer, String> exceptions = _exceptions;
+      if (exceptions != null) {
+        throw new RuntimeException("Received exception block: " + exceptions);
+      }
+      if (_isEarlyTerminated || resultsBlock == LAST_RESULTS_BLOCK) {
+        break;
+      } else if (!(resultsBlock instanceof ExplainV2ResultBlock)) {
+        throw new IllegalArgumentException("Expected ExplainV2ResultBlock, got: " + resultsBlock.getClass().getName());
+      } else {
+        ExplainV2ResultBlock block = (ExplainV2ResultBlock) resultsBlock;
+        for (ExplainInfo physicalPlan : block.getPhysicalPlans()) {
+          childNodes.add(asNode(physicalPlan));
+        }
+      }
+    }
+    String tableName = _context.getStageMetadata().getTableName();
+    Map<String, Plan.ExplainNode.AttributeValue> attributes;
+    if (tableName == null) { // this should never happen, but let's be paranoid to never fail
+      attributes = Collections.emptyMap();
+    } else {
+      attributes = Collections.singletonMap("table", Plan.ExplainNode.AttributeValue.newBuilder()
+          .setString(tableName)
+          .setMergeType(Plan.ExplainNode.AttributeValue.MergeType.IDEMPOTENT)

Review Comment:
   No, the default is _DEFAULT_ :laughing:. But you are right, semantically is the same for Strings and given how GRPC works, it would reduce a couple of bytes per operator



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


Re: [PR] Multi stage explain [pinot]

Posted by "gortiz (via GitHub)" <gi...@apache.org>.
gortiz commented on code in PR #13733:
URL: https://github.com/apache/pinot/pull/13733#discussion_r1750217340


##########
pinot-core/src/main/java/org/apache/pinot/core/query/request/context/QueryContext.java:
##########
@@ -85,7 +85,7 @@ public class QueryContext {
   private final int _offset;
   private final Map<String, String> _queryOptions;
   private final Map<ExpressionContext, ExpressionContext> _expressionOverrideHints;
-  private final boolean _explain;
+  private final ExplainMode _explain;

Review Comment:
   I guess the naming is not the best. I found difficult to find a better name. QueryMode is too generic and ends up meaning nothing.
   
   Anyway, I don't think this should be nullable. First, because it is error prone (as long as we don't use something like https://github.com/uber/NullAway we are not going to be able to enforce nullability). Second, because if we used `null` as _I don't want to explain_ it would be impossible to detect this case vs _the request is not specified_.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


Re: [PR] Multi stage explain [pinot]

Posted by "gortiz (via GitHub)" <gi...@apache.org>.
gortiz commented on code in PR #13733:
URL: https://github.com/apache/pinot/pull/13733#discussion_r1750212242


##########
pinot-core/src/main/java/org/apache/pinot/core/query/request/context/QueryContext.java:
##########
@@ -528,7 +538,16 @@ public Builder setExpressionOverrideHints(Map<ExpressionContext, ExpressionConte
       return this;
     }
 
+    /**
+     * @deprecated Use {@link #setExplain(ExplainMode)} instead.
+     */
+    @Deprecated

Review Comment:
   I just wanted to be more polite in case someone is using it. I don't think it hurts and we can remove it later.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


Re: [PR] Multi stage explain [pinot]

Posted by "gortiz (via GitHub)" <gi...@apache.org>.
gortiz commented on code in PR #13733:
URL: https://github.com/apache/pinot/pull/13733#discussion_r1750325812


##########
pinot-core/src/main/java/org/apache/pinot/core/operator/query/FilteredGroupByOperator.java:
##########
@@ -230,4 +232,26 @@ public String toExplainString() {
 
     return stringBuilder.append(')').toString();
   }
+
+  @Override
+  protected String getExplainName() {
+    return EXPLAIN_NAME;
+  }
+
+  @Override
+  protected void explainAttributes(ExplainAttributeBuilder attributeBuilder) {
+    super.explainAttributes(attributeBuilder);
+    if (_groupByExpressions.length > 0) {
+      List<String> groupKeys = Arrays.stream(_groupByExpressions)
+          .map(ExpressionContext::toString)
+          .collect(Collectors.toList());
+      attributeBuilder.putJson("groupKeys", groupKeys);
+    }
+    if (_aggregationFunctions.length > 0) {
+      List<String> aggregations = Arrays.stream(_aggregationFunctions)
+          .map(AggregationFunction::toExplainString)
+          .collect(Collectors.toList());
+      attributeBuilder.putJson("aggregations", aggregations);
+    }

Review Comment:
   Added  `_aggregationFunctions`



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


Re: [PR] Multi stage explain [pinot]

Posted by "yashmayya (via GitHub)" <gi...@apache.org>.
yashmayya commented on code in PR #13733:
URL: https://github.com/apache/pinot/pull/13733#discussion_r1771246208


##########
pinot-query-planner/src/main/java/org/apache/pinot/query/planner/explain/PlanNodeSorter.java:
##########
@@ -0,0 +1,229 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.query.planner.explain;
+
+import com.google.common.collect.Sets;
+import java.util.ArrayList;
+import java.util.Comparator;
+import java.util.List;
+import java.util.TreeSet;
+import org.apache.pinot.common.proto.Plan;
+import org.apache.pinot.query.planner.plannode.AggregateNode;
+import org.apache.pinot.query.planner.plannode.ExchangeNode;
+import org.apache.pinot.query.planner.plannode.ExplainedNode;
+import org.apache.pinot.query.planner.plannode.FilterNode;
+import org.apache.pinot.query.planner.plannode.JoinNode;
+import org.apache.pinot.query.planner.plannode.MailboxReceiveNode;
+import org.apache.pinot.query.planner.plannode.MailboxSendNode;
+import org.apache.pinot.query.planner.plannode.PlanNode;
+import org.apache.pinot.query.planner.plannode.PlanNodeVisitor;
+import org.apache.pinot.query.planner.plannode.ProjectNode;
+import org.apache.pinot.query.planner.plannode.SetOpNode;
+import org.apache.pinot.query.planner.plannode.SortNode;
+import org.apache.pinot.query.planner.plannode.TableScanNode;
+import org.apache.pinot.query.planner.plannode.ValueNode;
+import org.apache.pinot.query.planner.plannode.WindowNode;
+
+
+/**
+ * A utility class used to sort the plan nodes in a deterministic order.
+ *
+ * Any comparator can be passed to the sort method to sort the plan nodes, although the default comparator
+ * is used to sort the plan nodes based on the type and the attributes of the node.
+ *
+ * Only nodes that are simplifiable will be sorted. See {@link ExplainNodeSimplifier} for more information.
+ */
+public class PlanNodeSorter {
+
+  private PlanNodeSorter() {
+  }
+
+  /**
+   * Applies a default comparator to sort the plan nodes.
+   *
+   * The result may be the same as the input if the plan nodes are already sorted.
+   */
+  public static PlanNode sort(PlanNode planNode) {
+    return planNode.visit(new Sorter(), DefaultComparator.INSTANCE);
+  }
+
+  public static PlanNode sort(PlanNode planNode, Comparator<PlanNode> comparator) {
+    return planNode.visit(Sorter.INSTANCE, comparator);
+  }
+
+  private static class Sorter implements PlanNodeVisitor<PlanNode, Comparator<PlanNode>> {
+    private static final Sorter INSTANCE = new Sorter();
+
+    private PlanNode defaultNode(PlanNode node, Comparator<PlanNode> comparator) {
+      List<PlanNode> inputs = node.getInputs();
+      List<PlanNode> newInputs = applyToChildren(inputs, comparator);
+      return inputs != newInputs ? node.withInputs(newInputs) : node;
+    }
+
+    @Override
+    public PlanNode visitAggregate(AggregateNode node, Comparator<PlanNode> comparator) {
+      return defaultNode(node, comparator);
+    }
+
+    @Override
+    public PlanNode visitFilter(FilterNode node, Comparator<PlanNode> comparator) {
+      return defaultNode(node, comparator);
+    }
+
+    @Override
+    public PlanNode visitJoin(JoinNode node, Comparator<PlanNode> comparator) {
+      return defaultNode(node, comparator);
+    }
+
+    @Override
+    public PlanNode visitMailboxReceive(MailboxReceiveNode node, Comparator<PlanNode> comparator) {
+      return defaultNode(node, comparator);
+    }
+
+    @Override
+    public PlanNode visitMailboxSend(MailboxSendNode node, Comparator<PlanNode> comparator) {
+      return defaultNode(node, comparator);
+    }
+
+    @Override
+    public PlanNode visitProject(ProjectNode node, Comparator<PlanNode> comparator) {
+      return defaultNode(node, comparator);
+    }
+
+    @Override
+    public PlanNode visitSort(SortNode node, Comparator<PlanNode> comparator) {
+      return defaultNode(node, comparator);
+    }
+
+    @Override
+    public PlanNode visitTableScan(TableScanNode node, Comparator<PlanNode> comparator) {
+      return defaultNode(node, comparator);
+    }
+
+    @Override
+    public PlanNode visitValue(ValueNode node, Comparator<PlanNode> comparator) {
+      return defaultNode(node, comparator);
+    }
+
+    @Override
+    public PlanNode visitWindow(WindowNode node, Comparator<PlanNode> comparator) {
+      return defaultNode(node, comparator);
+    }
+
+    @Override
+    public PlanNode visitSetOp(SetOpNode node, Comparator<PlanNode> comparator) {
+      return defaultNode(node, comparator);
+    }
+
+    @Override
+    public PlanNode visitExchange(ExchangeNode node, Comparator<PlanNode> comparator) {
+      return defaultNode(node, comparator);
+    }
+
+    @Override
+    public PlanNode visitExplained(ExplainedNode node, Comparator<PlanNode> comparator) {
+      if (!node.getTitle().contains(ExplainNodeSimplifier.COMBINE) || node.getInputs().size() <= 1) {
+        return defaultNode(node, comparator);
+      }
+      List<PlanNode> simplifiedChildren = new ArrayList<>(applyToChildren(node.getInputs(), comparator));
+      simplifiedChildren.sort(comparator);
+      if (simplifiedChildren.equals(node.getInputs())) {
+        return node;
+      }
+      return node.withInputs(simplifiedChildren);
+    }
+
+    private List<PlanNode> applyToChildren(List<PlanNode> children, Comparator<PlanNode> comparator) {
+      List<PlanNode> simplifiedChildren = null;
+      for (int i = 0; i < children.size(); i++) {
+        PlanNode child = children.get(i);
+        PlanNode newChild = child.visit(this, comparator);
+        if (child != newChild) {
+          if (simplifiedChildren == null) {
+            simplifiedChildren = new ArrayList<>(children);
+          }
+          simplifiedChildren.set(i, newChild);
+        }
+      }
+      return simplifiedChildren != null ? simplifiedChildren : children;
+    }
+  }
+
+  public static class DefaultComparator implements Comparator<PlanNode> {
+    public static final DefaultComparator INSTANCE = new DefaultComparator();
+
+    private DefaultComparator() {
+    }
+
+    @Override
+    public int compare(PlanNode o1, PlanNode o2) {
+      if (!(o1 instanceof ExplainedNode) || !(o2 instanceof ExplainedNode)) {
+        return o1.getClass().getSimpleName().compareTo(o2.getClass().getSimpleName());
+      }
+      ExplainedNode node1 = (ExplainedNode) o1;
+      ExplainedNode node2 = (ExplainedNode) o2;
+
+      int cmp = node1.getTitle().compareTo(node2.getTitle());
+      if (cmp != 0) {
+        return cmp;
+      }
+      TreeSet<String> allKeys = new TreeSet<>(Sets.union(
+          node1.getAttributes().keySet(), node2.getAttributes().keySet()));
+      for (String key : allKeys) {
+        Plan.ExplainNode.AttributeValue value1 = node1.getAttributes().get(key);
+        Plan.ExplainNode.AttributeValue value2 = node2.getAttributes().get(key);
+        if (value1 == null) {
+          // there is an attribute in node2 with a _smaller_ key that is not in node1
+          return -1;
+        }
+        if (value2 == null) {
+          // there is an attribute in node1 with a _smaller_ key that is not in node2
+          return 1;
+        }
+        int cmp2 = value1.getValueCase().compareTo(value2.getValueCase());
+        if (cmp2 != 0) {
+          return cmp2;
+        }
+        int cmp3;
+        switch (value1.getValueCase()) {

Review Comment:
   Do we not need to handle the `STRINGLIST` case here?



##########
pinot-query-runtime/src/main/java/org/apache/pinot/query/runtime/operator/LeafStageTransferableBlockOperator.java:
##########
@@ -270,6 +276,69 @@ private TransferableBlock constructMetadataBlock() {
     return TransferableBlockUtils.getEndOfStreamTransferableBlock(multiStageQueryStats);
   }
 
+  public ExplainedNode explain() {
+    Preconditions.checkState(_requests.stream()
+        .allMatch(request -> request.getQueryContext().getExplain() == ExplainMode.NODE),
+        "All requests must have explain mode set to ExplainMode.NODE");
+
+    if (_executionFuture == null) {
+      _executionFuture = startExecution();
+    }
+
+    List<PlanNode> childNodes = new ArrayList<>();
+    while (true) {
+      BaseResultsBlock resultsBlock;
+      try {
+        long timeout = _context.getDeadlineMs() - System.currentTimeMillis();
+        resultsBlock = _blockingQueue.poll(timeout, TimeUnit.MILLISECONDS);
+      } catch (InterruptedException e) {
+        Thread.currentThread().interrupt();
+        throw new RuntimeException("Interrupted while waiting for results block", e);
+      }
+      if (resultsBlock == null) {
+        throw new RuntimeException("Timed out waiting for results block");
+      }
+      // Terminate when receiving exception block
+      Map<Integer, String> exceptions = _exceptions;
+      if (exceptions != null) {
+        throw new RuntimeException("Received exception block: " + exceptions);
+      }
+      if (_isEarlyTerminated || resultsBlock == LAST_RESULTS_BLOCK) {
+        break;
+      } else if (!(resultsBlock instanceof ExplainV2ResultBlock)) {
+        throw new IllegalArgumentException("Expected ExplainV2ResultBlock, got: " + resultsBlock.getClass().getName());
+      } else {
+        ExplainV2ResultBlock block = (ExplainV2ResultBlock) resultsBlock;
+        for (ExplainInfo physicalPlan : block.getPhysicalPlans()) {
+          childNodes.add(asNode(physicalPlan));
+        }
+      }
+    }
+    String tableName = _context.getStageMetadata().getTableName();
+    Map<String, Plan.ExplainNode.AttributeValue> attributes;
+    if (tableName == null) { // this should never happen, but let's be paranoid to never fail
+      attributes = Collections.emptyMap();
+    } else {
+      attributes = Collections.singletonMap("table", Plan.ExplainNode.AttributeValue.newBuilder()
+          .setString(tableName)
+          .setMergeType(Plan.ExplainNode.AttributeValue.MergeType.IDEMPOTENT)

Review Comment:
   This is the default merge type for string attribute values right?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


Re: [PR] Multi stage explain [pinot]

Posted by "gortiz (via GitHub)" <gi...@apache.org>.
gortiz commented on PR #13733:
URL: https://github.com/apache/pinot/pull/13733#issuecomment-2368149512

   The issue should be fixed. Anyway, my error merging this code shown that https://github.com/apache/pinot/pull/13999 is not testing the empty case and also to me it looks like the code that was added in `ServerQueryExecutorV1Impl` in that PR should have been added into `ResultsBlockUtils.buildEmptyQueryResults`. @ankitsultana can you take a look at that?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


Re: [PR] Multi stage explain [pinot]

Posted by "gortiz (via GitHub)" <gi...@apache.org>.
gortiz commented on code in PR #13733:
URL: https://github.com/apache/pinot/pull/13733#discussion_r1747210742


##########
pinot-core/src/main/java/org/apache/pinot/core/plan/PinotExplainedRelNode.java:
##########
@@ -0,0 +1,146 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.core.plan;
+
+import com.google.common.collect.Lists;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.List;
+import java.util.Map;
+import org.apache.calcite.plan.RelOptCluster;
+import org.apache.calcite.plan.RelTraitSet;
+import org.apache.calcite.rel.AbstractRelNode;
+import org.apache.calcite.rel.RelNode;
+import org.apache.calcite.rel.RelWriter;
+import org.apache.calcite.rel.type.RelDataType;
+import org.apache.calcite.rel.type.RelDataTypeFactory;
+import org.apache.calcite.rel.type.RelDataTypeField;
+import org.apache.calcite.rel.type.RelDataTypeFieldImpl;
+import org.apache.calcite.rel.type.RelRecordType;
+import org.apache.calcite.rel.type.StructKind;
+import org.apache.pinot.common.proto.Plan;
+import org.apache.pinot.common.utils.DataSchema;
+
+
+/**
+ * This class is a hackish way to adapt Pinot explain plan into Calcite explain plan.
+ *
+ * While in Calcite both logical and physical operators are represented by classes that extend RelNode, in multi-stage
+ * Pinot plans we first optimize {@link RelNode RelNodes} at logical level in the broker and then we transform send
+ * these RelNodes to the server (using {@code org.apache.pinot.query.planner.plannode.PlanNode} as intermediate format).
+ *
+ * Servers then extract leaf nodes from the stage, trying to execute as much as possible in a leaf operator. In that
+ * leaf operator is where nodes are compiled into single-stage physical operators
+ * (aka {@link org.apache.pinot.core.operator.BaseOperator}), which include important information like whether indexes
+ * are being used or not.
+ *
+ * {@link PinotExplainedRelNode} is a way to represent these single-stage operators in Calcite.
+ * <b>They are not meant to be executed</b>.
+ * Instead, when physical explain is required, the broker ask for the final plan to each server. They return
+ * a PlanNode and the broker then converts these PlanNodes into a PinotExplainedRelNode. The logical plan is then
+ * modified to substitute the nodes that were converted into single-stage physical plans with the corresponding
+ * PinotExplainedRelNode.
+ */
+public class PinotExplainedRelNode extends AbstractRelNode {
+
+  /**
+   * The name of this node, whose role is like a title in the explain plan.
+   */
+  private final String _type;
+  private final Map<String, Plan.ExplainNode.AttributeValue> _attributes;
+  private final List<RelNode> _inputs;
+  private final DataSchema _dataSchema;
+
+  public PinotExplainedRelNode(RelOptCluster cluster, RelTraitSet traitSet, String type,
+      Map<String, Plan.ExplainNode.AttributeValue> attributes, DataSchema dataSchema, RelNode input) {
+    this(cluster, traitSet, type, attributes, dataSchema, Lists.newArrayList(input));
+  }
+
+  public PinotExplainedRelNode(RelOptCluster cluster, RelTraitSet traitSet, String type,
+      Map<String, Plan.ExplainNode.AttributeValue> attributes, DataSchema dataSchema, List<? extends RelNode> inputs) {
+    super(cluster, traitSet);
+    _type = type;
+    _attributes = attributes;
+    _inputs = new ArrayList<>(inputs);
+    _dataSchema = dataSchema;
+  }
+
+  @Override
+  protected RelDataType deriveRowType() {
+    int size = _dataSchema.size();
+    List<RelDataTypeField> fields = new ArrayList<>(size);
+    RelDataTypeFactory typeFactory = getCluster().getTypeFactory();
+    for (int i = 0; i < size; i++) {
+      String columnName = _dataSchema.getColumnName(i);
+
+      DataSchema.ColumnDataType columnDataType = _dataSchema.getColumnDataType(i);
+      RelDataType type = columnDataType.toType(typeFactory);
+
+      fields.add(new RelDataTypeFieldImpl(columnName, i, type));
+    }
+    return new RelRecordType(StructKind.FULLY_QUALIFIED, fields, false);
+  }
+
+  @Override
+  public List<RelNode> getInputs() {
+    return Collections.unmodifiableList(_inputs);
+  }
+
+  @Override
+  public void replaceInput(int ordinalInParent, RelNode p) {
+    _inputs.set(ordinalInParent, p);
+  }
+
+  @Override
+  public String getRelTypeName() {
+    return _type;
+  }

Review Comment:
   We need to override this method to because the result is name that is presented in the explain plan. If we don't override it, the explain plan of all these nodes would be `PinotExplainedRelNode` for all cases.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


Re: [PR] Multi stage explain [pinot]

Posted by "gortiz (via GitHub)" <gi...@apache.org>.
gortiz commented on code in PR #13733:
URL: https://github.com/apache/pinot/pull/13733#discussion_r1747174202


##########
pinot-common/src/main/proto/plan.proto:
##########
@@ -193,3 +194,29 @@ message WindowNode {
   int32 upperBound = 6;
   repeated Literal constants = 7;
 }
+
+message ExplainNode {
+  string type = 1;

Review Comment:
   Yep, this is basically the _type_ of operator, which may be anything. Remember that these operators can even be extended with plugable indexes. Anyway we can add a comment to make it clearer for future readers :+1: 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


Re: [PR] Multi stage explain [pinot]

Posted by "yashmayya (via GitHub)" <gi...@apache.org>.
yashmayya commented on code in PR #13733:
URL: https://github.com/apache/pinot/pull/13733#discussion_r1741452894


##########
pinot-spi/src/main/java/org/apache/pinot/spi/utils/CommonConstants.java:
##########
@@ -1192,6 +1209,9 @@ public enum WindowOverFlowMode {
     public static class PlanVersions {
       public static final int V1 = 1;
     }
+
+    public static final String ASK_SERVERS_FOR_EXPLAIN_PLAN = "pinot.query.explain.ask.servers";

Review Comment:
   I think we should include an indication that this config is only for the multi-stage engine? Also I'm wondering if users will be able to make sense of "ask servers" without extensively consulting the documentation. What do you think about `include.physical` (or something similar) instead of `ask.servers`?



##########
pinot-common/src/main/proto/plan.proto:
##########
@@ -193,3 +194,29 @@ message WindowNode {
   int32 upperBound = 6;
   repeated Literal constants = 7;
 }
+
+message ExplainNode {
+  string type = 1;
+  map<string, AttributeValue> attributes = 2;
+
+  message AttributeValue {
+    MergeType mergeType = 1;
+
+    oneof value {
+      string string = 2;
+      int64 long = 3;
+      bool bool = 4;
+      string json = 5;

Review Comment:
   Why is this called `json`? It seems like we're using this field to store collections of strings serialized into a byte array via Jackson's serializer for some reason I didn't quite follow.



##########
pinot-common/src/main/java/org/apache/pinot/common/utils/DatabaseUtils.java:
##########
@@ -32,6 +32,10 @@ public class DatabaseUtils {
   private DatabaseUtils() {
   }
 
+  public static String[] splitTableName(String tableName) {

Review Comment:
   nit: can we have a small Javadoc demonstrating intended usage through an example? IIUC this is to convert `databaseName.tableName` to [`databaseName`, `tableName`] although I don't fully follow why this is needed or used in the `RelBuilder` call.



##########
pinot-common/src/main/proto/plan.proto:
##########
@@ -193,3 +194,29 @@ message WindowNode {
   int32 upperBound = 6;
   repeated Literal constants = 7;
 }
+
+message ExplainNode {
+  string type = 1;
+  map<string, AttributeValue> attributes = 2;
+
+  message AttributeValue {
+    MergeType mergeType = 1;
+
+    oneof value {
+      string string = 2;
+      int64 long = 3;
+      bool bool = 4;
+      string json = 5;
+    };
+
+    enum MergeType {
+      // Longs will be added, other types will behave like IDEMPOTENT.
+      DEFAULT = 0;
+      // When being merge, values must be the same. Otherwise cannot be merged.
+      IDEMPOTENT = 1;
+      // When being merge, if values are different, the value can be ignored.

Review Comment:
   Can we also include a comment on what "ignore" means in this context - i.e., what will be the result of merging two explain nodes that have an attribute value of merge type `IGNORABLE` with different values?



##########
pinot-core/src/main/java/org/apache/pinot/core/plan/PinotExplainedRelNode.java:
##########
@@ -0,0 +1,146 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.core.plan;
+
+import com.google.common.collect.Lists;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.List;
+import java.util.Map;
+import org.apache.calcite.plan.RelOptCluster;
+import org.apache.calcite.plan.RelTraitSet;
+import org.apache.calcite.rel.AbstractRelNode;
+import org.apache.calcite.rel.RelNode;
+import org.apache.calcite.rel.RelWriter;
+import org.apache.calcite.rel.type.RelDataType;
+import org.apache.calcite.rel.type.RelDataTypeFactory;
+import org.apache.calcite.rel.type.RelDataTypeField;
+import org.apache.calcite.rel.type.RelDataTypeFieldImpl;
+import org.apache.calcite.rel.type.RelRecordType;
+import org.apache.calcite.rel.type.StructKind;
+import org.apache.pinot.common.proto.Plan;
+import org.apache.pinot.common.utils.DataSchema;
+
+
+/**
+ * This class is a hackish way to adapt Pinot explain plan into Calcite explain plan.
+ *
+ * While in Calcite both logical and physical operators are represented by classes that extend RelNode, in multi-stage
+ * Pinot plans we first optimize {@link RelNode RelNodes} at logical level in the broker and then we transform send
+ * these RelNodes to the server (using {@code org.apache.pinot.query.planner.plannode.PlanNode} as intermediate format).
+ *
+ * Servers then extract leaf nodes from the stage, trying to execute as much as possible in a leaf operator. In that
+ * leaf operator is where nodes are compiled into single-stage physical operators
+ * (aka {@link org.apache.pinot.core.operator.BaseOperator}), which include important information like whether indexes
+ * are being used or not.
+ *
+ * {@link PinotExplainedRelNode} is a way to represent these single-stage operators in Calcite.
+ * <b>They are not meant to be executed</b>.
+ * Instead, when physical explain is required, the broker ask for the final plan to each server. They return
+ * a PlanNode and the broker then converts these PlanNodes into a PinotExplainedRelNode. The logical plan is then
+ * modified to substitute the nodes that were converted into single-stage physical plans with the corresponding
+ * PinotExplainedRelNode.
+ */
+public class PinotExplainedRelNode extends AbstractRelNode {
+
+  /**
+   * The name of this node, whose role is like a title in the explain plan.
+   */
+  private final String _type;
+  private final Map<String, Plan.ExplainNode.AttributeValue> _attributes;
+  private final List<RelNode> _inputs;
+  private final DataSchema _dataSchema;
+
+  public PinotExplainedRelNode(RelOptCluster cluster, RelTraitSet traitSet, String type,

Review Comment:
   Also looks like most (all) call-sites are passing in an empty trait set, maybe we can add an additional constructor that uses an empty trait set by default?



##########
pinot-core/src/main/java/org/apache/pinot/core/query/request/context/QueryContext.java:
##########
@@ -528,7 +538,16 @@ public Builder setExpressionOverrideHints(Map<ExpressionContext, ExpressionConte
       return this;
     }
 
+    /**
+     * @deprecated Use {@link #setExplain(ExplainMode)} instead.
+     */
+    @Deprecated

Review Comment:
   Isn't this an internal class / API? Can't we directly remove after migrating all usages instead of deprecating? Seems like unnecessary cruft to leave around unless I'm missing something regarding the usage of `QueryContext`.



##########
pinot-integration-tests/src/test/java/org/apache/pinot/integration/tests/MultiStageEngineExplainIntegrationTest.java:
##########
@@ -0,0 +1,255 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.integration.tests;
+
+import com.fasterxml.jackson.databind.JsonNode;
+import java.io.File;
+import java.util.List;
+import org.apache.pinot.spi.config.table.TableConfig;
+import org.apache.pinot.spi.data.Schema;
+import org.apache.pinot.spi.env.PinotConfiguration;
+import org.apache.pinot.spi.utils.CommonConstants;
+import org.apache.pinot.util.TestUtils;
+import org.intellij.lang.annotations.Language;
+import org.testcontainers.shaded.org.apache.commons.io.FileUtils;
+import org.testng.Assert;
+import org.testng.annotations.AfterClass;
+import org.testng.annotations.BeforeClass;
+import org.testng.annotations.BeforeMethod;
+import org.testng.annotations.Test;
+
+
+public class MultiStageEngineExplainIntegrationTest extends BaseClusterIntegrationTest {
+  private static final String SCHEMA_FILE_NAME = "On_Time_On_Time_Performance_2014_100k_subset_nonulls.schema";
+  private static final String DEFAULT_DATABASE_NAME = CommonConstants.DEFAULT_DATABASE;
+  private static final String DATABASE_NAME = "db1";
+  private static final String TABLE_NAME_WITH_DATABASE = DATABASE_NAME + "." + DEFAULT_TABLE_NAME;
+  private String _tableName = DEFAULT_TABLE_NAME;
+
+  @Override
+  protected String getSchemaFileName() {
+    return SCHEMA_FILE_NAME;
+  }

Review Comment:
   Isn't this just using the default schema?



##########
pinot-common/src/main/proto/plan.proto:
##########
@@ -193,3 +194,29 @@ message WindowNode {
   int32 upperBound = 6;
   repeated Literal constants = 7;
 }
+
+message ExplainNode {
+  string type = 1;

Review Comment:
   Answering my own question but after reviewing more parts of this PR, it doesn't seem to make sense to make it an enum based on the sheer number of different possible types.



##########
pinot-query-planner/src/main/java/org/apache/pinot/query/planner/explain/PhysicalExplainPlanVisitor.java:
##########
@@ -248,6 +249,11 @@ public StringBuilder visitValue(ValueNode node, Context context) {
     return appendInfo(node, context);
   }
 
+  @Override
+  public StringBuilder visitExplained(ExplainedNode node, Context context) {
+    return appendInfo(node, context);

Review Comment:
   We should never see an `ExplainedNode` in a "physical" explain plan if I understand correctly?



##########
pinot-common/src/main/java/org/apache/pinot/common/utils/DataSchema.java:
##########
@@ -196,27 +201,135 @@ public int hashCode() {
     return EqualityUtils.hashCodeOf(Arrays.hashCode(_columnNames), Arrays.hashCode(_columnDataTypes));
   }
 
+  public RelDataType toRelDataType(RelDataTypeFactory typeFactory) {
+    List<RelDataType> columnTypes = new ArrayList<>(_columnDataTypes.length);
+    for (ColumnDataType columnDataType : _columnDataTypes) {
+      columnTypes.add(columnDataType.toType(typeFactory));
+    }
+    return typeFactory.createStructType(columnTypes, Arrays.asList(_columnNames));
+  }
+
   public enum ColumnDataType {
-    INT(NullValuePlaceHolder.INT),
-    LONG(NullValuePlaceHolder.LONG),
-    FLOAT(NullValuePlaceHolder.FLOAT),
-    DOUBLE(NullValuePlaceHolder.DOUBLE),
-    BIG_DECIMAL(NullValuePlaceHolder.BIG_DECIMAL),
-    BOOLEAN(INT, NullValuePlaceHolder.INT),
-    TIMESTAMP(LONG, NullValuePlaceHolder.LONG),
-    STRING(NullValuePlaceHolder.STRING),
-    JSON(STRING, NullValuePlaceHolder.STRING),
-    BYTES(NullValuePlaceHolder.INTERNAL_BYTES),
-    OBJECT(null),
-    INT_ARRAY(NullValuePlaceHolder.INT_ARRAY),
-    LONG_ARRAY(NullValuePlaceHolder.LONG_ARRAY),
-    FLOAT_ARRAY(NullValuePlaceHolder.FLOAT_ARRAY),
-    DOUBLE_ARRAY(NullValuePlaceHolder.DOUBLE_ARRAY),
-    BOOLEAN_ARRAY(INT_ARRAY, NullValuePlaceHolder.INT_ARRAY),
-    TIMESTAMP_ARRAY(LONG_ARRAY, NullValuePlaceHolder.LONG_ARRAY),
-    STRING_ARRAY(NullValuePlaceHolder.STRING_ARRAY),
-    BYTES_ARRAY(NullValuePlaceHolder.BYTES_ARRAY),
-    UNKNOWN(null);
+    INT(NullValuePlaceHolder.INT) {
+      @Override
+      public RelDataType toType(RelDataTypeFactory typeFactory) {
+        return typeFactory.createSqlType(SqlTypeName.INTEGER);
+      }
+    },
+    LONG(NullValuePlaceHolder.LONG) {
+      @Override
+      public RelDataType toType(RelDataTypeFactory typeFactory) {
+        return typeFactory.createSqlType(SqlTypeName.BIGINT);
+      }
+    },
+    FLOAT(NullValuePlaceHolder.FLOAT) {
+      @Override
+      public RelDataType toType(RelDataTypeFactory typeFactory) {
+        return typeFactory.createSqlType(SqlTypeName.FLOAT);

Review Comment:
   Calcite `FLOAT` is 8 byte floating points - i.e., same as `DOUBLE`. Pinot's `FLOAT` type maps to Calcite's `REAL` type which is 4 byte floating points. See https://calcite.apache.org/docs/reference.html#scalar-types and https://github.com/apache/pinot/blob/3f324a4f0bfdecb1b14e459deec647a2ff2f734d/pinot-query-planner/src/main/java/org/apache/pinot/query/type/TypeFactory.java#L75-L76
   
   I think ideally we'd want to unify the `FieldSpec.DataType` -> `RelDataType` and `DataSchema.ColumnDataType` -> `RelDataType` conversions but I'm not sure there's an easy / clean way to do so with how things are currently structured.



##########
pinot-core/src/main/java/org/apache/pinot/core/operator/blocks/results/ExplainV2ResultBlock.java:
##########
@@ -0,0 +1,100 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.core.operator.blocks.results;
+
+import com.fasterxml.jackson.core.JsonProcessingException;
+import com.google.common.collect.Lists;
+import java.io.IOException;
+import java.io.UncheckedIOException;
+import java.util.ArrayList;
+import java.util.List;
+import javax.annotation.Nullable;
+import org.apache.pinot.common.datatable.DataTable;
+import org.apache.pinot.common.utils.DataSchema;
+import org.apache.pinot.core.common.datatable.DataTableBuilder;
+import org.apache.pinot.core.common.datatable.DataTableBuilderFactory;
+import org.apache.pinot.core.plan.ExplainInfo;
+import org.apache.pinot.core.query.request.context.QueryContext;
+import org.apache.pinot.spi.utils.JsonUtils;
+
+
+public class ExplainV2ResultBlock extends BaseResultsBlock {
+
+  private final QueryContext _queryContext;
+  private final List<ExplainInfo> _physicalPlan;
+
+  public static final DataSchema EXPLAIN_RESULT_SCHEMA =
+      new DataSchema(new String[]{"Plan"}, new DataSchema.ColumnDataType[]{DataSchema.ColumnDataType.STRING});
+
+  public ExplainV2ResultBlock(QueryContext queryContext, ExplainInfo physicalPlan) {
+    this(queryContext, Lists.newArrayList(physicalPlan));
+  }
+
+  public ExplainV2ResultBlock(QueryContext queryContext, List<ExplainInfo> physicalPlan) {
+    _queryContext = queryContext;
+    _physicalPlan = physicalPlan;
+  }
+
+  @Override
+  public int getNumRows() {
+    return _physicalPlan.size();
+  }
+
+  @Nullable
+  @Override
+  public QueryContext getQueryContext() {
+    return _queryContext;
+  }
+
+  @Nullable
+  @Override
+  public DataSchema getDataSchema() {
+    return DataSchema.EXPLAIN_RESULT_SCHEMA;
+  }
+
+  @Nullable
+  @Override
+  public List<Object[]> getRows() {
+    List<Object[]> rows = new ArrayList<>(_physicalPlan.size());
+    try {
+      for (ExplainInfo node : _physicalPlan) {
+        rows.add(new Object[]{JsonUtils.objectToString(node)});
+      }
+      return rows;
+    } catch (JsonProcessingException e) {
+      throw new UncheckedIOException(e);
+    }
+  }
+
+  @Override
+  public DataTable getDataTable()
+      throws IOException {
+    DataTableBuilder dataTableBuilder = DataTableBuilderFactory.getDataTableBuilder(DataSchema.EXPLAIN_RESULT_SCHEMA);

Review Comment:
   Same issue as above?



##########
pinot-core/src/main/java/org/apache/pinot/core/operator/BaseOperator.java:
##########
@@ -46,4 +50,25 @@ public final T nextBlock() {
 
   // Make it protected because we should always call nextBlock()
   protected abstract T getNextBlock();
+
+  @Override
+  public ExplainInfo getExplainInfo() {
+    ExplainAttributeBuilder attributeBuilder = new ExplainAttributeBuilder();
+    explainAttributes(attributeBuilder);
+    return new ExplainInfo(getExplainName(), attributeBuilder.build(), getChildrenExplainInfo());
+  }
+
+  protected List<ExplainInfo> getChildrenExplainInfo() {
+    return getChildOperators().stream()
+        .filter(Objects::nonNull)
+        .map(Operator::getExplainInfo)
+        .collect(Collectors.toList());
+  }
+
+  protected String getExplainName() {

Review Comment:
   I think it'd be useful to have a Javadoc here to elaborate how this is different from `toExplainString`.



##########
pinot-core/src/main/java/org/apache/pinot/core/common/Operator.java:
##########
@@ -37,23 +39,55 @@ public interface Operator<T extends Block> {
    * times, and will return non-empty block or null if no more documents available
    *
    * @throws EarlyTerminationException if the operator is early-terminated (interrupted) before processing the next
-   *         block of data. Operator can early terminated when the query times out, or is already satisfied.
+   *         block of data. Operator can early be terminated when the query times out, or is already satisfied.

Review Comment:
   ```suggestion
      *         block of data. Operator can be early terminated when the query times out, or is already satisfied.
   ```



##########
pinot-core/src/main/java/org/apache/pinot/core/operator/blocks/results/ExplainV2ResultBlock.java:
##########
@@ -0,0 +1,100 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.core.operator.blocks.results;
+
+import com.fasterxml.jackson.core.JsonProcessingException;
+import com.google.common.collect.Lists;
+import java.io.IOException;
+import java.io.UncheckedIOException;
+import java.util.ArrayList;
+import java.util.List;
+import javax.annotation.Nullable;
+import org.apache.pinot.common.datatable.DataTable;
+import org.apache.pinot.common.utils.DataSchema;
+import org.apache.pinot.core.common.datatable.DataTableBuilder;
+import org.apache.pinot.core.common.datatable.DataTableBuilderFactory;
+import org.apache.pinot.core.plan.ExplainInfo;
+import org.apache.pinot.core.query.request.context.QueryContext;
+import org.apache.pinot.spi.utils.JsonUtils;
+
+
+public class ExplainV2ResultBlock extends BaseResultsBlock {
+
+  private final QueryContext _queryContext;
+  private final List<ExplainInfo> _physicalPlan;
+
+  public static final DataSchema EXPLAIN_RESULT_SCHEMA =
+      new DataSchema(new String[]{"Plan"}, new DataSchema.ColumnDataType[]{DataSchema.ColumnDataType.STRING});
+
+  public ExplainV2ResultBlock(QueryContext queryContext, ExplainInfo physicalPlan) {
+    this(queryContext, Lists.newArrayList(physicalPlan));
+  }
+
+  public ExplainV2ResultBlock(QueryContext queryContext, List<ExplainInfo> physicalPlan) {
+    _queryContext = queryContext;
+    _physicalPlan = physicalPlan;
+  }
+
+  @Override
+  public int getNumRows() {
+    return _physicalPlan.size();
+  }
+
+  @Nullable
+  @Override
+  public QueryContext getQueryContext() {
+    return _queryContext;
+  }
+
+  @Nullable
+  @Override
+  public DataSchema getDataSchema() {
+    return DataSchema.EXPLAIN_RESULT_SCHEMA;

Review Comment:
   ```suggestion
       return EXPLAIN_RESULT_SCHEMA;
   ```
   I think this is accidentally using the wrong result schema?



##########
pinot-core/src/main/java/org/apache/pinot/core/operator/query/AggregationOperator.java:
##########
@@ -100,4 +103,21 @@ public String toExplainString() {
 
     return stringBuilder.append(')').toString();
   }
+
+  @Override
+  protected String getExplainName() {
+    return EXPLAIN_NAME;
+  }
+
+  @Override
+  protected void explainAttributes(ExplainAttributeBuilder attributeBuilder) {
+    super.explainAttributes(attributeBuilder);
+    if (_aggregationFunctions.length == 0) {
+      return;
+    }
+    List<String> aggregations = Arrays.stream(_aggregationFunctions)
+        .map(AggregationFunction::toExplainString)
+        .collect(Collectors.toList());
+    attributeBuilder.putLongIdempotent("numAggregations", aggregations.size());

Review Comment:
   This seems odd - we're collecting the explain strings of all the aggregation functions but then only including the number of aggregations?



##########
pinot-common/src/main/proto/plan.proto:
##########
@@ -193,3 +194,29 @@ message WindowNode {
   int32 upperBound = 6;
   repeated Literal constants = 7;
 }
+
+message ExplainNode {
+  string type = 1;

Review Comment:
   Is it not possible for this to be an enum?



##########
pinot-core/src/main/java/org/apache/pinot/core/plan/PinotExplainedRelNode.java:
##########
@@ -0,0 +1,146 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.core.plan;
+
+import com.google.common.collect.Lists;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.List;
+import java.util.Map;
+import org.apache.calcite.plan.RelOptCluster;
+import org.apache.calcite.plan.RelTraitSet;
+import org.apache.calcite.rel.AbstractRelNode;
+import org.apache.calcite.rel.RelNode;
+import org.apache.calcite.rel.RelWriter;
+import org.apache.calcite.rel.type.RelDataType;
+import org.apache.calcite.rel.type.RelDataTypeFactory;
+import org.apache.calcite.rel.type.RelDataTypeField;
+import org.apache.calcite.rel.type.RelDataTypeFieldImpl;
+import org.apache.calcite.rel.type.RelRecordType;
+import org.apache.calcite.rel.type.StructKind;
+import org.apache.pinot.common.proto.Plan;
+import org.apache.pinot.common.utils.DataSchema;
+
+
+/**
+ * This class is a hackish way to adapt Pinot explain plan into Calcite explain plan.
+ *
+ * While in Calcite both logical and physical operators are represented by classes that extend RelNode, in multi-stage
+ * Pinot plans we first optimize {@link RelNode RelNodes} at logical level in the broker and then we transform send
+ * these RelNodes to the server (using {@code org.apache.pinot.query.planner.plannode.PlanNode} as intermediate format).
+ *
+ * Servers then extract leaf nodes from the stage, trying to execute as much as possible in a leaf operator. In that
+ * leaf operator is where nodes are compiled into single-stage physical operators
+ * (aka {@link org.apache.pinot.core.operator.BaseOperator}), which include important information like whether indexes
+ * are being used or not.
+ *
+ * {@link PinotExplainedRelNode} is a way to represent these single-stage operators in Calcite.
+ * <b>They are not meant to be executed</b>.
+ * Instead, when physical explain is required, the broker ask for the final plan to each server. They return
+ * a PlanNode and the broker then converts these PlanNodes into a PinotExplainedRelNode. The logical plan is then
+ * modified to substitute the nodes that were converted into single-stage physical plans with the corresponding
+ * PinotExplainedRelNode.
+ */
+public class PinotExplainedRelNode extends AbstractRelNode {
+
+  /**
+   * The name of this node, whose role is like a title in the explain plan.
+   */
+  private final String _type;
+  private final Map<String, Plan.ExplainNode.AttributeValue> _attributes;
+  private final List<RelNode> _inputs;
+  private final DataSchema _dataSchema;
+
+  public PinotExplainedRelNode(RelOptCluster cluster, RelTraitSet traitSet, String type,
+      Map<String, Plan.ExplainNode.AttributeValue> attributes, DataSchema dataSchema, RelNode input) {
+    this(cluster, traitSet, type, attributes, dataSchema, Lists.newArrayList(input));
+  }
+
+  public PinotExplainedRelNode(RelOptCluster cluster, RelTraitSet traitSet, String type,
+      Map<String, Plan.ExplainNode.AttributeValue> attributes, DataSchema dataSchema, List<? extends RelNode> inputs) {
+    super(cluster, traitSet);
+    _type = type;
+    _attributes = attributes;
+    _inputs = new ArrayList<>(inputs);
+    _dataSchema = dataSchema;
+  }
+
+  @Override
+  protected RelDataType deriveRowType() {
+    int size = _dataSchema.size();
+    List<RelDataTypeField> fields = new ArrayList<>(size);
+    RelDataTypeFactory typeFactory = getCluster().getTypeFactory();
+    for (int i = 0; i < size; i++) {
+      String columnName = _dataSchema.getColumnName(i);
+
+      DataSchema.ColumnDataType columnDataType = _dataSchema.getColumnDataType(i);
+      RelDataType type = columnDataType.toType(typeFactory);
+
+      fields.add(new RelDataTypeFieldImpl(columnName, i, type));
+    }
+    return new RelRecordType(StructKind.FULLY_QUALIFIED, fields, false);

Review Comment:
   nit: looks like this is the default `StructKind` and nullability used in `RelRecordType` so we can simplify this?
   ```suggestion
       return new RelRecordType(fields);
   ```



##########
pinot-core/src/main/java/org/apache/pinot/core/operator/ProjectionOperator.java:
##########
@@ -100,6 +100,17 @@ public String toExplainString() {
     return stringBuilder.append(')').toString();
   }
 
+  @Override
+  protected String getExplainName() {
+    return EXPLAIN_NAME;
+  }
+
+  @Override
+  protected void explainAttributes(ExplainAttributeBuilder attributeBuilder) {
+    super.explainAttributes(attributeBuilder);

Review Comment:
   Unnecessary `super` call?



##########
pinot-core/src/main/java/org/apache/pinot/core/operator/query/FilteredGroupByOperator.java:
##########
@@ -230,4 +232,26 @@ public String toExplainString() {
 
     return stringBuilder.append(')').toString();
   }
+
+  @Override
+  protected String getExplainName() {
+    return EXPLAIN_NAME;
+  }
+
+  @Override
+  protected void explainAttributes(ExplainAttributeBuilder attributeBuilder) {
+    super.explainAttributes(attributeBuilder);
+    if (_groupByExpressions.length > 0) {
+      List<String> groupKeys = Arrays.stream(_groupByExpressions)
+          .map(ExpressionContext::toString)
+          .collect(Collectors.toList());
+      attributeBuilder.putJson("groupKeys", groupKeys);
+    }
+    if (_aggregationFunctions.length > 0) {
+      List<String> aggregations = Arrays.stream(_aggregationFunctions)
+          .map(AggregationFunction::toExplainString)
+          .collect(Collectors.toList());
+      attributeBuilder.putJson("aggregations", aggregations);
+    }

Review Comment:
   Can't comment on that file directly but we should add this to `FilteredAggregationOperator` as well.



##########
pinot-core/src/main/java/org/apache/pinot/core/plan/PinotExplainedRelNode.java:
##########
@@ -0,0 +1,146 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.core.plan;
+
+import com.google.common.collect.Lists;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.List;
+import java.util.Map;
+import org.apache.calcite.plan.RelOptCluster;
+import org.apache.calcite.plan.RelTraitSet;
+import org.apache.calcite.rel.AbstractRelNode;
+import org.apache.calcite.rel.RelNode;
+import org.apache.calcite.rel.RelWriter;
+import org.apache.calcite.rel.type.RelDataType;
+import org.apache.calcite.rel.type.RelDataTypeFactory;
+import org.apache.calcite.rel.type.RelDataTypeField;
+import org.apache.calcite.rel.type.RelDataTypeFieldImpl;
+import org.apache.calcite.rel.type.RelRecordType;
+import org.apache.calcite.rel.type.StructKind;
+import org.apache.pinot.common.proto.Plan;
+import org.apache.pinot.common.utils.DataSchema;
+
+
+/**
+ * This class is a hackish way to adapt Pinot explain plan into Calcite explain plan.
+ *
+ * While in Calcite both logical and physical operators are represented by classes that extend RelNode, in multi-stage
+ * Pinot plans we first optimize {@link RelNode RelNodes} at logical level in the broker and then we transform send
+ * these RelNodes to the server (using {@code org.apache.pinot.query.planner.plannode.PlanNode} as intermediate format).
+ *
+ * Servers then extract leaf nodes from the stage, trying to execute as much as possible in a leaf operator. In that
+ * leaf operator is where nodes are compiled into single-stage physical operators
+ * (aka {@link org.apache.pinot.core.operator.BaseOperator}), which include important information like whether indexes
+ * are being used or not.
+ *
+ * {@link PinotExplainedRelNode} is a way to represent these single-stage operators in Calcite.
+ * <b>They are not meant to be executed</b>.
+ * Instead, when physical explain is required, the broker ask for the final plan to each server. They return
+ * a PlanNode and the broker then converts these PlanNodes into a PinotExplainedRelNode. The logical plan is then
+ * modified to substitute the nodes that were converted into single-stage physical plans with the corresponding
+ * PinotExplainedRelNode.
+ */
+public class PinotExplainedRelNode extends AbstractRelNode {
+
+  /**
+   * The name of this node, whose role is like a title in the explain plan.
+   */
+  private final String _type;
+  private final Map<String, Plan.ExplainNode.AttributeValue> _attributes;
+  private final List<RelNode> _inputs;
+  private final DataSchema _dataSchema;
+
+  public PinotExplainedRelNode(RelOptCluster cluster, RelTraitSet traitSet, String type,
+      Map<String, Plan.ExplainNode.AttributeValue> attributes, DataSchema dataSchema, RelNode input) {
+    this(cluster, traitSet, type, attributes, dataSchema, Lists.newArrayList(input));
+  }
+
+  public PinotExplainedRelNode(RelOptCluster cluster, RelTraitSet traitSet, String type,
+      Map<String, Plan.ExplainNode.AttributeValue> attributes, DataSchema dataSchema, List<? extends RelNode> inputs) {
+    super(cluster, traitSet);
+    _type = type;
+    _attributes = attributes;
+    _inputs = new ArrayList<>(inputs);
+    _dataSchema = dataSchema;
+  }
+
+  @Override
+  protected RelDataType deriveRowType() {
+    int size = _dataSchema.size();
+    List<RelDataTypeField> fields = new ArrayList<>(size);
+    RelDataTypeFactory typeFactory = getCluster().getTypeFactory();
+    for (int i = 0; i < size; i++) {
+      String columnName = _dataSchema.getColumnName(i);
+
+      DataSchema.ColumnDataType columnDataType = _dataSchema.getColumnDataType(i);
+      RelDataType type = columnDataType.toType(typeFactory);
+
+      fields.add(new RelDataTypeFieldImpl(columnName, i, type));
+    }
+    return new RelRecordType(StructKind.FULLY_QUALIFIED, fields, false);
+  }
+
+  @Override
+  public List<RelNode> getInputs() {
+    return Collections.unmodifiableList(_inputs);
+  }
+
+  @Override
+  public void replaceInput(int ordinalInParent, RelNode p) {
+    _inputs.set(ordinalInParent, p);
+  }
+
+  @Override
+  public String getRelTypeName() {
+    return _type;
+  }
+
+  @Override
+  public RelWriter explainTerms(RelWriter pw) {
+    RelWriter relWriter = super.explainTerms(pw);
+    int inputSize = getInputs().size();
+    if (inputSize == 1) {
+      relWriter.input("input", getInputs().get(0));
+    } else if (inputSize > 1) {
+      for (int i = 0; i < inputSize; i++) {
+        relWriter.input("input#" + i, getInputs().get(i));
+      }
+    }

Review Comment:
   Where is this input term string being used? Does it matter if for a single input node we use `input#0` instead of `input`?



##########
pinot-core/src/main/java/org/apache/pinot/core/plan/PinotExplainedRelNode.java:
##########
@@ -0,0 +1,146 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.core.plan;
+
+import com.google.common.collect.Lists;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.List;
+import java.util.Map;
+import org.apache.calcite.plan.RelOptCluster;
+import org.apache.calcite.plan.RelTraitSet;
+import org.apache.calcite.rel.AbstractRelNode;
+import org.apache.calcite.rel.RelNode;
+import org.apache.calcite.rel.RelWriter;
+import org.apache.calcite.rel.type.RelDataType;
+import org.apache.calcite.rel.type.RelDataTypeFactory;
+import org.apache.calcite.rel.type.RelDataTypeField;
+import org.apache.calcite.rel.type.RelDataTypeFieldImpl;
+import org.apache.calcite.rel.type.RelRecordType;
+import org.apache.calcite.rel.type.StructKind;
+import org.apache.pinot.common.proto.Plan;
+import org.apache.pinot.common.utils.DataSchema;
+
+
+/**
+ * This class is a hackish way to adapt Pinot explain plan into Calcite explain plan.
+ *
+ * While in Calcite both logical and physical operators are represented by classes that extend RelNode, in multi-stage
+ * Pinot plans we first optimize {@link RelNode RelNodes} at logical level in the broker and then we transform send
+ * these RelNodes to the server (using {@code org.apache.pinot.query.planner.plannode.PlanNode} as intermediate format).
+ *
+ * Servers then extract leaf nodes from the stage, trying to execute as much as possible in a leaf operator. In that
+ * leaf operator is where nodes are compiled into single-stage physical operators
+ * (aka {@link org.apache.pinot.core.operator.BaseOperator}), which include important information like whether indexes
+ * are being used or not.
+ *
+ * {@link PinotExplainedRelNode} is a way to represent these single-stage operators in Calcite.
+ * <b>They are not meant to be executed</b>.
+ * Instead, when physical explain is required, the broker ask for the final plan to each server. They return
+ * a PlanNode and the broker then converts these PlanNodes into a PinotExplainedRelNode. The logical plan is then
+ * modified to substitute the nodes that were converted into single-stage physical plans with the corresponding
+ * PinotExplainedRelNode.
+ */
+public class PinotExplainedRelNode extends AbstractRelNode {
+
+  /**
+   * The name of this node, whose role is like a title in the explain plan.
+   */
+  private final String _type;
+  private final Map<String, Plan.ExplainNode.AttributeValue> _attributes;
+  private final List<RelNode> _inputs;
+  private final DataSchema _dataSchema;
+
+  public PinotExplainedRelNode(RelOptCluster cluster, RelTraitSet traitSet, String type,
+      Map<String, Plan.ExplainNode.AttributeValue> attributes, DataSchema dataSchema, RelNode input) {
+    this(cluster, traitSet, type, attributes, dataSchema, Lists.newArrayList(input));
+  }
+
+  public PinotExplainedRelNode(RelOptCluster cluster, RelTraitSet traitSet, String type,
+      Map<String, Plan.ExplainNode.AttributeValue> attributes, DataSchema dataSchema, List<? extends RelNode> inputs) {
+    super(cluster, traitSet);
+    _type = type;
+    _attributes = attributes;
+    _inputs = new ArrayList<>(inputs);
+    _dataSchema = dataSchema;
+  }
+
+  @Override
+  protected RelDataType deriveRowType() {
+    int size = _dataSchema.size();
+    List<RelDataTypeField> fields = new ArrayList<>(size);
+    RelDataTypeFactory typeFactory = getCluster().getTypeFactory();
+    for (int i = 0; i < size; i++) {
+      String columnName = _dataSchema.getColumnName(i);
+
+      DataSchema.ColumnDataType columnDataType = _dataSchema.getColumnDataType(i);
+      RelDataType type = columnDataType.toType(typeFactory);
+
+      fields.add(new RelDataTypeFieldImpl(columnName, i, type));
+    }
+    return new RelRecordType(StructKind.FULLY_QUALIFIED, fields, false);
+  }
+
+  @Override
+  public List<RelNode> getInputs() {
+    return Collections.unmodifiableList(_inputs);
+  }
+
+  @Override
+  public void replaceInput(int ordinalInParent, RelNode p) {
+    _inputs.set(ordinalInParent, p);
+  }
+
+  @Override
+  public String getRelTypeName() {
+    return _type;
+  }
+
+  @Override
+  public RelWriter explainTerms(RelWriter pw) {
+    RelWriter relWriter = super.explainTerms(pw);
+    int inputSize = getInputs().size();
+    if (inputSize == 1) {
+      relWriter.input("input", getInputs().get(0));
+    } else if (inputSize > 1) {
+      for (int i = 0; i < inputSize; i++) {
+        relWriter.input("input#" + i, getInputs().get(i));
+      }
+    }
+    for (Map.Entry<String, Plan.ExplainNode.AttributeValue> entry : _attributes.entrySet()) {
+      Plan.ExplainNode.AttributeValue value = entry.getValue();
+      if (value.hasString()) {
+        relWriter.item(entry.getKey(), value.getString());
+      } else if (value.hasLong()) {
+        relWriter.item(entry.getKey(), value.getLong());
+      } else if (value.hasBool()) {
+        relWriter.item(entry.getKey(), value.getBool());
+      } else if (value.hasJson()) {
+        relWriter.item(entry.getKey(), value.getJson());
+      } else {
+        relWriter.item(entry.getKey(), "unknown value");
+      }

Review Comment:
   nit: might be better to use `switch (value.getValueCase())` here instead.



##########
pinot-core/src/main/java/org/apache/pinot/core/plan/PinotExplainedRelNode.java:
##########
@@ -0,0 +1,146 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.core.plan;
+
+import com.google.common.collect.Lists;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.List;
+import java.util.Map;
+import org.apache.calcite.plan.RelOptCluster;
+import org.apache.calcite.plan.RelTraitSet;
+import org.apache.calcite.rel.AbstractRelNode;
+import org.apache.calcite.rel.RelNode;
+import org.apache.calcite.rel.RelWriter;
+import org.apache.calcite.rel.type.RelDataType;
+import org.apache.calcite.rel.type.RelDataTypeFactory;
+import org.apache.calcite.rel.type.RelDataTypeField;
+import org.apache.calcite.rel.type.RelDataTypeFieldImpl;
+import org.apache.calcite.rel.type.RelRecordType;
+import org.apache.calcite.rel.type.StructKind;
+import org.apache.pinot.common.proto.Plan;
+import org.apache.pinot.common.utils.DataSchema;
+
+
+/**
+ * This class is a hackish way to adapt Pinot explain plan into Calcite explain plan.
+ *
+ * While in Calcite both logical and physical operators are represented by classes that extend RelNode, in multi-stage
+ * Pinot plans we first optimize {@link RelNode RelNodes} at logical level in the broker and then we transform send
+ * these RelNodes to the server (using {@code org.apache.pinot.query.planner.plannode.PlanNode} as intermediate format).
+ *
+ * Servers then extract leaf nodes from the stage, trying to execute as much as possible in a leaf operator. In that
+ * leaf operator is where nodes are compiled into single-stage physical operators
+ * (aka {@link org.apache.pinot.core.operator.BaseOperator}), which include important information like whether indexes
+ * are being used or not.
+ *
+ * {@link PinotExplainedRelNode} is a way to represent these single-stage operators in Calcite.
+ * <b>They are not meant to be executed</b>.
+ * Instead, when physical explain is required, the broker ask for the final plan to each server. They return
+ * a PlanNode and the broker then converts these PlanNodes into a PinotExplainedRelNode. The logical plan is then
+ * modified to substitute the nodes that were converted into single-stage physical plans with the corresponding
+ * PinotExplainedRelNode.
+ */
+public class PinotExplainedRelNode extends AbstractRelNode {
+
+  /**
+   * The name of this node, whose role is like a title in the explain plan.
+   */
+  private final String _type;
+  private final Map<String, Plan.ExplainNode.AttributeValue> _attributes;
+  private final List<RelNode> _inputs;
+  private final DataSchema _dataSchema;
+
+  public PinotExplainedRelNode(RelOptCluster cluster, RelTraitSet traitSet, String type,

Review Comment:
   Unused constructor?



##########
pinot-core/src/main/java/org/apache/pinot/core/plan/PinotExplainedRelNode.java:
##########
@@ -0,0 +1,146 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.core.plan;
+
+import com.google.common.collect.Lists;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.List;
+import java.util.Map;
+import org.apache.calcite.plan.RelOptCluster;
+import org.apache.calcite.plan.RelTraitSet;
+import org.apache.calcite.rel.AbstractRelNode;
+import org.apache.calcite.rel.RelNode;
+import org.apache.calcite.rel.RelWriter;
+import org.apache.calcite.rel.type.RelDataType;
+import org.apache.calcite.rel.type.RelDataTypeFactory;
+import org.apache.calcite.rel.type.RelDataTypeField;
+import org.apache.calcite.rel.type.RelDataTypeFieldImpl;
+import org.apache.calcite.rel.type.RelRecordType;
+import org.apache.calcite.rel.type.StructKind;
+import org.apache.pinot.common.proto.Plan;
+import org.apache.pinot.common.utils.DataSchema;
+
+
+/**
+ * This class is a hackish way to adapt Pinot explain plan into Calcite explain plan.
+ *
+ * While in Calcite both logical and physical operators are represented by classes that extend RelNode, in multi-stage
+ * Pinot plans we first optimize {@link RelNode RelNodes} at logical level in the broker and then we transform send
+ * these RelNodes to the server (using {@code org.apache.pinot.query.planner.plannode.PlanNode} as intermediate format).
+ *
+ * Servers then extract leaf nodes from the stage, trying to execute as much as possible in a leaf operator. In that
+ * leaf operator is where nodes are compiled into single-stage physical operators
+ * (aka {@link org.apache.pinot.core.operator.BaseOperator}), which include important information like whether indexes
+ * are being used or not.
+ *
+ * {@link PinotExplainedRelNode} is a way to represent these single-stage operators in Calcite.
+ * <b>They are not meant to be executed</b>.
+ * Instead, when physical explain is required, the broker ask for the final plan to each server. They return
+ * a PlanNode and the broker then converts these PlanNodes into a PinotExplainedRelNode. The logical plan is then
+ * modified to substitute the nodes that were converted into single-stage physical plans with the corresponding
+ * PinotExplainedRelNode.
+ */
+public class PinotExplainedRelNode extends AbstractRelNode {
+
+  /**
+   * The name of this node, whose role is like a title in the explain plan.
+   */
+  private final String _type;
+  private final Map<String, Plan.ExplainNode.AttributeValue> _attributes;
+  private final List<RelNode> _inputs;
+  private final DataSchema _dataSchema;
+
+  public PinotExplainedRelNode(RelOptCluster cluster, RelTraitSet traitSet, String type,
+      Map<String, Plan.ExplainNode.AttributeValue> attributes, DataSchema dataSchema, RelNode input) {
+    this(cluster, traitSet, type, attributes, dataSchema, Lists.newArrayList(input));
+  }
+
+  public PinotExplainedRelNode(RelOptCluster cluster, RelTraitSet traitSet, String type,
+      Map<String, Plan.ExplainNode.AttributeValue> attributes, DataSchema dataSchema, List<? extends RelNode> inputs) {
+    super(cluster, traitSet);
+    _type = type;
+    _attributes = attributes;
+    _inputs = new ArrayList<>(inputs);
+    _dataSchema = dataSchema;
+  }
+
+  @Override
+  protected RelDataType deriveRowType() {
+    int size = _dataSchema.size();
+    List<RelDataTypeField> fields = new ArrayList<>(size);
+    RelDataTypeFactory typeFactory = getCluster().getTypeFactory();
+    for (int i = 0; i < size; i++) {
+      String columnName = _dataSchema.getColumnName(i);
+
+      DataSchema.ColumnDataType columnDataType = _dataSchema.getColumnDataType(i);
+      RelDataType type = columnDataType.toType(typeFactory);
+
+      fields.add(new RelDataTypeFieldImpl(columnName, i, type));
+    }
+    return new RelRecordType(StructKind.FULLY_QUALIFIED, fields, false);
+  }
+
+  @Override
+  public List<RelNode> getInputs() {
+    return Collections.unmodifiableList(_inputs);
+  }
+
+  @Override
+  public void replaceInput(int ordinalInParent, RelNode p) {
+    _inputs.set(ordinalInParent, p);
+  }
+
+  @Override
+  public String getRelTypeName() {
+    return _type;
+  }

Review Comment:
   There's a comment in `AbstractRelNode` that says that it is not recommended to override this method (default implementation uses the class name). Any particular reason we need this?



##########
pinot-core/src/main/java/org/apache/pinot/core/query/request/context/QueryContext.java:
##########
@@ -85,7 +85,7 @@ public class QueryContext {
   private final int _offset;
   private final Map<String, String> _queryOptions;
   private final Map<ExpressionContext, ExpressionContext> _expressionOverrideHints;
-  private final boolean _explain;
+  private final ExplainMode _explain;

Review Comment:
   Seems a little counter-intuitive to have an "explain mode" for actual queries as well - maybe we can use `null` here for `_explain` instead of having an enum value `NONE`?



##########
pinot-core/src/main/java/org/apache/pinot/core/query/request/context/utils/QueryContextConverterUtils.java:
##########
@@ -155,11 +157,23 @@ public static QueryContext getQueryContext(PinotQuery pinotQuery) {
       }
     }
 
+    ExplainMode explainMode;
+    if (!pinotQuery.isExplain()) {
+      explainMode = ExplainMode.NONE;
+    } else {
+      String useMultistageEngine = CommonConstants.Broker.Request.QueryOptionKey.USE_MULTISTAGE_ENGINE;
+      if (pinotQuery.getQueryOptions() != null && pinotQuery.getQueryOptions().containsKey(useMultistageEngine)) {

Review Comment:
   Shouldn't we check if the option value is `true` instead of just checking for the key's existence? We can use `QueryOptionsUtils::isUseMultistageEngine` here.



##########
pinot-integration-tests/src/test/java/org/apache/pinot/integration/tests/MultiStageEngineExplainIntegrationTest.java:
##########
@@ -0,0 +1,255 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.integration.tests;
+
+import com.fasterxml.jackson.databind.JsonNode;
+import java.io.File;
+import java.util.List;
+import org.apache.pinot.spi.config.table.TableConfig;
+import org.apache.pinot.spi.data.Schema;
+import org.apache.pinot.spi.env.PinotConfiguration;
+import org.apache.pinot.spi.utils.CommonConstants;
+import org.apache.pinot.util.TestUtils;
+import org.intellij.lang.annotations.Language;
+import org.testcontainers.shaded.org.apache.commons.io.FileUtils;
+import org.testng.Assert;
+import org.testng.annotations.AfterClass;
+import org.testng.annotations.BeforeClass;
+import org.testng.annotations.BeforeMethod;
+import org.testng.annotations.Test;
+
+
+public class MultiStageEngineExplainIntegrationTest extends BaseClusterIntegrationTest {
+  private static final String SCHEMA_FILE_NAME = "On_Time_On_Time_Performance_2014_100k_subset_nonulls.schema";
+  private static final String DEFAULT_DATABASE_NAME = CommonConstants.DEFAULT_DATABASE;
+  private static final String DATABASE_NAME = "db1";
+  private static final String TABLE_NAME_WITH_DATABASE = DATABASE_NAME + "." + DEFAULT_TABLE_NAME;
+  private String _tableName = DEFAULT_TABLE_NAME;
+
+  @Override
+  protected String getSchemaFileName() {
+    return SCHEMA_FILE_NAME;
+  }
+
+  @BeforeClass
+  public void setUp()
+      throws Exception {
+    TestUtils.ensureDirectoriesExistAndEmpty(_tempDir, _segmentDir, _tarDir);
+
+    // Start the Pinot cluster
+    startZk();
+    startController();
+    startBroker();
+    startServers(2);
+
+    // Create and upload the schema and table config
+    Schema schema = createSchema();
+    addSchema(schema);
+    TableConfig tableConfig = createOfflineTableConfig();
+    addTableConfig(tableConfig);
+
+    // Unpack the Avro files
+    List<File> avroFiles = unpackAvroData(_tempDir);
+
+    // Create and upload segments
+    ClusterIntegrationTestUtils.buildSegmentsFromAvro(avroFiles, tableConfig, schema, 0, _segmentDir, _tarDir);
+    uploadSegments(getTableName(), _tarDir);
+
+    // Wait for all documents loaded
+    waitForAllDocsLoaded(600_000L);
+  }
+
+  protected void overrideBrokerConf(PinotConfiguration brokerConf) {
+    brokerConf.setProperty(CommonConstants.MultiStageQueryRunner.ASK_SERVERS_FOR_EXPLAIN_PLAN, "true");
+  }
+
+  @BeforeMethod
+  public void resetMultiStage() {
+    setUseMultiStageQueryEngine(true);
+  }
+
+  @Test
+  public void simpleQuery() {
+    explain("SELECT 1 FROM mytable",
+        //@formatter:off
+        "Execution Plan\n"
+            + "PinotLogicalExchange(distribution=[broadcast])\n"

Review Comment:
   Right now, these plans look like two different types of plans mashed together (which they essentially are, but might be nice to hide that for users). Do you think it makes sense to unify the casing used by the leaf stage nodes' explain terms to match that of Calcite? I know that we're currently using the same format as what is used for single-stage engine explain queries and we might not want to change that so maybe we could just do a simple blind conversion from `SCREAMING_SNAKE_CASE` to `CamelCase`.



##########
pinot-query-runtime/src/main/java/org/apache/pinot/query/runtime/operator/MultiStageOperator.java:
##########
@@ -167,6 +173,26 @@ protected TransferableBlock updateEosBlock(TransferableBlock upstreamEos, StatMa
     return upstreamEos;
   }
 
+  @Override
+  public ExplainInfo getExplainInfo() {
+    return new ExplainInfo(getExplainName(), getExplainAttributes(), getChildrenExplainInfo());
+  }
+
+  protected List<ExplainInfo> getChildrenExplainInfo() {
+    return getChildOperators().stream()
+        .filter(Objects::nonNull)
+        .map(Operator::getExplainInfo)
+        .collect(Collectors.toList());
+  }
+
+  protected String getExplainName() {
+    return toExplainString();
+  }
+
+  protected Map<String, Plan.ExplainNode.AttributeValue> getExplainAttributes() {
+    return Collections.emptyMap();
+  }

Review Comment:
   These are added to `BaseOperator` as well, so why not put them in the parent `Operator` interface. FWICT, all implementations are either implementations of `BaseOperator` or `MultiStageOperator`?



##########
pinot-query-planner/src/main/java/org/apache/pinot/query/catalog/PinotTable.java:
##########
@@ -44,8 +43,13 @@ public PinotTable(Schema schema) {
 
   @Override
   public RelDataType getRowType(RelDataTypeFactory relDataTypeFactory) {
-    Preconditions.checkState(relDataTypeFactory instanceof TypeFactory);
-    TypeFactory typeFactory = (TypeFactory) relDataTypeFactory;
+    // TODO: Look for a better solution
+    TypeFactory typeFactory;
+    if (relDataTypeFactory instanceof TypeFactory) {
+      typeFactory = (TypeFactory) relDataTypeFactory;
+    } else {
+      typeFactory = TypeFactory.INSTANCE;
+    }

Review Comment:
   Why was this change needed?



##########
pinot-query-planner/src/main/java/org/apache/pinot/query/QueryEnvironment.java:
##########
@@ -172,7 +185,27 @@ public QueryPlannerResult explainQuery(String sqlQuery, SqlNodeAndOptions sqlNod
         SqlExplainLevel level =
             explain.getDetailLevel() == null ? SqlExplainLevel.DIGEST_ATTRIBUTES : explain.getDetailLevel();
         Set<String> tableNames = RelToPlanNodeConverter.getTableNamesFromRelRoot(relRoot.rel);
-        return new QueryPlannerResult(null, PlannerUtils.explainPlan(relRoot.rel, format, level), tableNames);
+        if (!explain.withImplementation() || !askServers) {

Review Comment:
   It seems a little strange that we have `EXPLAIN IMPLEMENTATION PLAN FOR...` which results in the `SqlPhysicalExplain` branch above but here we're checking for `SqlExplain::withImplementation` and the plans returned are different. Basically, I think it'll be super confusing to end users that `EXPLAIN IMPLEMENTATION PLAN FOR...` and `EXPLAIN PLAN WITH IMPLEMENTATION FOR...` return very different looking results. 
   
   I see that in the original issue for `EXPLAIN IMPLEMENTATION PLAN FOR...` (https://github.com/apache/pinot/issues/10901), the proposal originally was to use `EXPLAIN PHYSICAL PLAN FOR...` syntax, although I'm not sure we want to go that route either because the plan's semantics don't exactly match Calcite's physical plan semantics?



##########
pinot-integration-tests/src/test/java/org/apache/pinot/integration/tests/MultiStageEngineExplainIntegrationTest.java:
##########
@@ -0,0 +1,255 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.integration.tests;
+
+import com.fasterxml.jackson.databind.JsonNode;
+import java.io.File;
+import java.util.List;
+import org.apache.pinot.spi.config.table.TableConfig;
+import org.apache.pinot.spi.data.Schema;
+import org.apache.pinot.spi.env.PinotConfiguration;
+import org.apache.pinot.spi.utils.CommonConstants;
+import org.apache.pinot.util.TestUtils;
+import org.intellij.lang.annotations.Language;
+import org.testcontainers.shaded.org.apache.commons.io.FileUtils;
+import org.testng.Assert;
+import org.testng.annotations.AfterClass;
+import org.testng.annotations.BeforeClass;
+import org.testng.annotations.BeforeMethod;
+import org.testng.annotations.Test;
+
+
+public class MultiStageEngineExplainIntegrationTest extends BaseClusterIntegrationTest {
+  private static final String SCHEMA_FILE_NAME = "On_Time_On_Time_Performance_2014_100k_subset_nonulls.schema";
+  private static final String DEFAULT_DATABASE_NAME = CommonConstants.DEFAULT_DATABASE;
+  private static final String DATABASE_NAME = "db1";
+  private static final String TABLE_NAME_WITH_DATABASE = DATABASE_NAME + "." + DEFAULT_TABLE_NAME;
+  private String _tableName = DEFAULT_TABLE_NAME;
+
+  @Override
+  protected String getSchemaFileName() {
+    return SCHEMA_FILE_NAME;
+  }
+
+  @BeforeClass
+  public void setUp()
+      throws Exception {
+    TestUtils.ensureDirectoriesExistAndEmpty(_tempDir, _segmentDir, _tarDir);
+
+    // Start the Pinot cluster
+    startZk();
+    startController();
+    startBroker();
+    startServers(2);
+
+    // Create and upload the schema and table config
+    Schema schema = createSchema();
+    addSchema(schema);
+    TableConfig tableConfig = createOfflineTableConfig();
+    addTableConfig(tableConfig);
+
+    // Unpack the Avro files
+    List<File> avroFiles = unpackAvroData(_tempDir);
+
+    // Create and upload segments
+    ClusterIntegrationTestUtils.buildSegmentsFromAvro(avroFiles, tableConfig, schema, 0, _segmentDir, _tarDir);
+    uploadSegments(getTableName(), _tarDir);
+
+    // Wait for all documents loaded
+    waitForAllDocsLoaded(600_000L);
+  }
+
+  protected void overrideBrokerConf(PinotConfiguration brokerConf) {
+    brokerConf.setProperty(CommonConstants.MultiStageQueryRunner.ASK_SERVERS_FOR_EXPLAIN_PLAN, "true");
+  }
+
+  @BeforeMethod
+  public void resetMultiStage() {
+    setUseMultiStageQueryEngine(true);
+  }
+
+  @Test
+  public void simpleQuery() {
+    explain("SELECT 1 FROM mytable",
+        //@formatter:off
+        "Execution Plan\n"
+            + "PinotLogicalExchange(distribution=[broadcast])\n"
+            + "  LEAF_STAGE_COMBINE_OPERATOR\n"
+            + "    STREAMING_INSTANCE_RESPONSE\n"
+            + "      STREAMING_COMBINE_SELECT(repeatedOnSegments=[12])\n"
+            + "        SELECT_STREAMING(totalDocs=[115545])\n"
+            + "          TRANSFORM(expressions=[[\"'1'\"]])\n"
+            + "            PROJECT(columns=[[]])\n"
+            + "              DOC_ID_SET(maxDocs=[120000])\n"
+            + "                FILTER_MATCH_ENTIRE_SEGMENT(numDocs=[115545])\n");
+        //@formatter:on
+  }
+
+  @Test
+  public void simpleQueryVerbose() {
+    explainVerbose("SELECT 1 FROM mytable",
+        //@formatter:off
+        "Execution Plan\n"
+            + "IntermediateCombine\n"
+            + "  Alternative(servers=[1])\n"
+            + "    PinotLogicalExchange(distribution=[broadcast])\n"
+            + "      LEAF_STAGE_COMBINE_OPERATOR\n"
+            + "        STREAMING_INSTANCE_RESPONSE\n"
+            + "          STREAMING_COMBINE_SELECT\n"
+            + "            SELECT_STREAMING(segment=[any], totalDocs=[any])\n"
+            + "              TRANSFORM(expressions=[[\"'1'\"]])\n"
+            + "                PROJECT(columns=[[]])\n"
+            + "                  DOC_ID_SET(maxDocs=[10000])\n"
+            + "                    FILTER_MATCH_ENTIRE_SEGMENT(numDocs=[any])\n"
+            + "            SELECT_STREAMING(segment=[any], totalDocs=[any])\n"
+            + "              TRANSFORM(expressions=[[\"'1'\"]])\n"
+            + "                PROJECT(columns=[[]])\n"
+            + "                  DOC_ID_SET(maxDocs=[10000])\n"
+            + "                    FILTER_MATCH_ENTIRE_SEGMENT(numDocs=[any])\n"
+            + "            SELECT_STREAMING(segment=[any], totalDocs=[any])\n"
+            + "              TRANSFORM(expressions=[[\"'1'\"]])\n"
+            + "                PROJECT(columns=[[]])\n"
+            + "                  DOC_ID_SET(maxDocs=[10000])\n"
+            + "                    FILTER_MATCH_ENTIRE_SEGMENT(numDocs=[any])\n"
+            + "            SELECT_STREAMING(segment=[any], totalDocs=[any])\n"
+            + "              TRANSFORM(expressions=[[\"'1'\"]])\n"
+            + "                PROJECT(columns=[[]])\n"
+            + "                  DOC_ID_SET(maxDocs=[10000])\n"
+            + "                    FILTER_MATCH_ENTIRE_SEGMENT(numDocs=[any])\n"
+            + "            SELECT_STREAMING(segment=[any], totalDocs=[any])\n"
+            + "              TRANSFORM(expressions=[[\"'1'\"]])\n"
+            + "                PROJECT(columns=[[]])\n"
+            + "                  DOC_ID_SET(maxDocs=[10000])\n"
+            + "                    FILTER_MATCH_ENTIRE_SEGMENT(numDocs=[any])\n"
+            + "            SELECT_STREAMING(segment=[any], totalDocs=[any])\n"
+            + "              TRANSFORM(expressions=[[\"'1'\"]])\n"
+            + "                PROJECT(columns=[[]])\n"
+            + "                  DOC_ID_SET(maxDocs=[10000])\n"
+            + "                    FILTER_MATCH_ENTIRE_SEGMENT(numDocs=[any])\n"
+            + "  Alternative(servers=[1])\n"
+            + "    PinotLogicalExchange(distribution=[broadcast])\n"
+            + "      LEAF_STAGE_COMBINE_OPERATOR\n"
+            + "        STREAMING_INSTANCE_RESPONSE\n"
+            + "          STREAMING_COMBINE_SELECT\n"
+            + "            SELECT_STREAMING(segment=[any], totalDocs=[any])\n"
+            + "              TRANSFORM(expressions=[[\"'1'\"]])\n"
+            + "                PROJECT(columns=[[]])\n"
+            + "                  DOC_ID_SET(maxDocs=[10000])\n"
+            + "                    FILTER_MATCH_ENTIRE_SEGMENT(numDocs=[any])\n"
+            + "            SELECT_STREAMING(segment=[any], totalDocs=[any])\n"
+            + "              TRANSFORM(expressions=[[\"'1'\"]])\n"
+            + "                PROJECT(columns=[[]])\n"
+            + "                  DOC_ID_SET(maxDocs=[10000])\n"
+            + "                    FILTER_MATCH_ENTIRE_SEGMENT(numDocs=[any])\n"
+            + "            SELECT_STREAMING(segment=[any], totalDocs=[any])\n"
+            + "              TRANSFORM(expressions=[[\"'1'\"]])\n"
+            + "                PROJECT(columns=[[]])\n"
+            + "                  DOC_ID_SET(maxDocs=[10000])\n"
+            + "                    FILTER_MATCH_ENTIRE_SEGMENT(numDocs=[any])\n"
+            + "            SELECT_STREAMING(segment=[any], totalDocs=[any])\n"
+            + "              TRANSFORM(expressions=[[\"'1'\"]])\n"
+            + "                PROJECT(columns=[[]])\n"
+            + "                  DOC_ID_SET(maxDocs=[10000])\n"
+            + "                    FILTER_MATCH_ENTIRE_SEGMENT(numDocs=[any])\n"
+            + "            SELECT_STREAMING(segment=[any], totalDocs=[any])\n"
+            + "              TRANSFORM(expressions=[[\"'1'\"]])\n"
+            + "                PROJECT(columns=[[]])\n"
+            + "                  DOC_ID_SET(maxDocs=[10000])\n"
+            + "                    FILTER_MATCH_ENTIRE_SEGMENT(numDocs=[any])\n"
+            + "            SELECT_STREAMING(segment=[any], totalDocs=[any])\n"
+            + "              TRANSFORM(expressions=[[\"'1'\"]])\n"
+            + "                PROJECT(columns=[[]])\n"
+            + "                  DOC_ID_SET(maxDocs=[10000])\n"
+            + "                    FILTER_MATCH_ENTIRE_SEGMENT(numDocs=[any])\n");
+    //@formatter:on
+  }
+
+  @Test
+  public void simpleQueryLogical() {
+    explainLogical("SELECT 1 FROM mytable",
+        //@formatter:off
+        "Execution Plan\n"
+            + "LogicalProject(EXPR$0=[1])\n"
+            + "  LogicalTableScan(table=[[default, mytable]])\n");
+    //@formatter:on
+  }
+
+  @Override
+  protected String getTableName() {
+    return _tableName;
+  }

Review Comment:
   Same as above, isn't this just using the default table name?



##########
pinot-integration-tests/src/test/java/org/apache/pinot/integration/tests/MultiStageEngineExplainIntegrationTest.java:
##########
@@ -0,0 +1,255 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.integration.tests;
+
+import com.fasterxml.jackson.databind.JsonNode;
+import java.io.File;
+import java.util.List;
+import org.apache.pinot.spi.config.table.TableConfig;
+import org.apache.pinot.spi.data.Schema;
+import org.apache.pinot.spi.env.PinotConfiguration;
+import org.apache.pinot.spi.utils.CommonConstants;
+import org.apache.pinot.util.TestUtils;
+import org.intellij.lang.annotations.Language;
+import org.testcontainers.shaded.org.apache.commons.io.FileUtils;
+import org.testng.Assert;
+import org.testng.annotations.AfterClass;
+import org.testng.annotations.BeforeClass;
+import org.testng.annotations.BeforeMethod;
+import org.testng.annotations.Test;
+
+
+public class MultiStageEngineExplainIntegrationTest extends BaseClusterIntegrationTest {
+  private static final String SCHEMA_FILE_NAME = "On_Time_On_Time_Performance_2014_100k_subset_nonulls.schema";
+  private static final String DEFAULT_DATABASE_NAME = CommonConstants.DEFAULT_DATABASE;
+  private static final String DATABASE_NAME = "db1";
+  private static final String TABLE_NAME_WITH_DATABASE = DATABASE_NAME + "." + DEFAULT_TABLE_NAME;

Review Comment:
   Unused?



##########
pinot-query-planner/src/main/java/org/apache/pinot/query/planner/explain/ExplainNodeSimplifier.java:
##########
@@ -0,0 +1,169 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.query.planner.explain;
+
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import org.apache.pinot.common.proto.Plan;
+import org.apache.pinot.core.query.reduce.ExplainPlanDataTableReducer;
+import org.apache.pinot.query.planner.plannode.AggregateNode;
+import org.apache.pinot.query.planner.plannode.ExchangeNode;
+import org.apache.pinot.query.planner.plannode.ExplainedNode;
+import org.apache.pinot.query.planner.plannode.FilterNode;
+import org.apache.pinot.query.planner.plannode.JoinNode;
+import org.apache.pinot.query.planner.plannode.MailboxReceiveNode;
+import org.apache.pinot.query.planner.plannode.MailboxSendNode;
+import org.apache.pinot.query.planner.plannode.PlanNode;
+import org.apache.pinot.query.planner.plannode.PlanNodeVisitor;
+import org.apache.pinot.query.planner.plannode.ProjectNode;
+import org.apache.pinot.query.planner.plannode.SetOpNode;
+import org.apache.pinot.query.planner.plannode.SortNode;
+import org.apache.pinot.query.planner.plannode.TableScanNode;
+import org.apache.pinot.query.planner.plannode.ValueNode;
+import org.apache.pinot.query.planner.plannode.WindowNode;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+
+public class ExplainNodeSimplifier {
+  private static final Logger LOGGER = LoggerFactory.getLogger(ExplainNodeSimplifier.class);
+
+  private ExplainNodeSimplifier() {
+  }
+
+  public static PlanNode simplifyNode(PlanNode plan1) {
+    Visitor planNodeMerger = new Visitor();
+    return plan1.visit(planNodeMerger, null);
+  }
+
+  private static class Visitor implements PlanNodeVisitor<PlanNode, Void> {
+    private static final String REPEAT_ATTRIBUTE_KEY = "repeatedOnSegments";
+
+    private PlanNode defaultNode(PlanNode node) {
+      List<PlanNode> inputs = node.getInputs();
+      List<PlanNode> newInputs = simplifyChildren(inputs);
+      return inputs != newInputs ? node.withInputs(newInputs) : node;
+    }
+
+    @Override
+    public PlanNode visitAggregate(AggregateNode node, Void context) {
+      return defaultNode(node);
+    }
+
+    @Override
+    public PlanNode visitFilter(FilterNode node, Void context) {
+      return defaultNode(node);
+    }
+
+    @Override
+    public PlanNode visitJoin(JoinNode node, Void context) {
+      return defaultNode(node);
+    }
+
+    @Override
+    public PlanNode visitMailboxReceive(MailboxReceiveNode node, Void context) {
+      return defaultNode(node);
+    }
+
+    @Override
+    public PlanNode visitMailboxSend(MailboxSendNode node, Void context) {
+      return defaultNode(node);
+    }
+
+    @Override
+    public PlanNode visitProject(ProjectNode node, Void context) {
+      return defaultNode(node);
+    }
+
+    @Override
+    public PlanNode visitSort(SortNode node, Void context) {
+      return defaultNode(node);
+    }
+
+    @Override
+    public PlanNode visitTableScan(TableScanNode node, Void context) {
+      return defaultNode(node);
+    }
+
+    @Override
+    public PlanNode visitValue(ValueNode node, Void context) {
+      return defaultNode(node);
+    }
+
+    @Override
+    public PlanNode visitWindow(WindowNode node, Void context) {
+      return defaultNode(node);
+    }
+
+    @Override
+    public PlanNode visitSetOp(SetOpNode node, Void context) {
+      return defaultNode(node);
+    }
+
+    @Override
+    public PlanNode visitExchange(ExchangeNode node, Void context) {
+      return defaultNode(node);
+    }
+
+    @Override
+    public PlanNode visitExplained(ExplainedNode node, Void context) {
+      if (!node.getType().contains(ExplainPlanDataTableReducer.COMBINE) || node.getInputs().size() <= 1) {
+        return defaultNode(node);
+      }
+      List<PlanNode> simplifiedChildren = simplifyChildren(node.getInputs());
+      PlanNode child1 = simplifiedChildren.get(0);
+
+      for (int i = 1; i < simplifiedChildren.size(); i++) {
+        PlanNode child2 = simplifiedChildren.get(i);
+        PlanNode merged = PlanNodeMerger.mergePlans(child1, child2, false);
+        if (merged == null) {
+          LOGGER.info("Found unmergeable inputs on node of type {}: {} and {}", node, child1, child2);
+          assert false : "Unmergeable inputs found";

Review Comment:
   Why would we want to fail here if and only if assertions are enabled?



##########
pinot-core/src/main/java/org/apache/pinot/core/operator/BaseOperator.java:
##########
@@ -46,4 +50,25 @@ public final T nextBlock() {
 
   // Make it protected because we should always call nextBlock()
   protected abstract T getNextBlock();
+
+  @Override
+  public ExplainInfo getExplainInfo() {
+    ExplainAttributeBuilder attributeBuilder = new ExplainAttributeBuilder();
+    explainAttributes(attributeBuilder);
+    return new ExplainInfo(getExplainName(), attributeBuilder.build(), getChildrenExplainInfo());
+  }
+
+  protected List<ExplainInfo> getChildrenExplainInfo() {
+    return getChildOperators().stream()
+        .filter(Objects::nonNull)
+        .map(Operator::getExplainInfo)
+        .collect(Collectors.toList());
+  }
+
+  protected String getExplainName() {

Review Comment:
   Also, it looks like the vast majority of `toExplainString` are basically `getExplainName` + `explainAttributes` in string form. I think we might be able to clean up the interface to avoid this duplication in most operator implementations? Although this isn't super critical and can be done as a follow up.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


Re: [PR] Multi stage explain [pinot]

Posted by "gortiz (via GitHub)" <gi...@apache.org>.
gortiz commented on code in PR #13733:
URL: https://github.com/apache/pinot/pull/13733#discussion_r1754564993


##########
pinot-core/src/main/java/org/apache/pinot/core/operator/transform/TransformOperator.java:
##########
@@ -86,7 +88,17 @@ public String toExplainString() {
   }
 
   protected String getExplainName() {
-    return EXPLAIN_NAME;
+    return CaseFormat.UPPER_UNDERSCORE.to(CaseFormat.UPPER_CAMEL, EXPLAIN_NAME);

Review Comment:
   Yes. Good call. Luckily a test detected the same problem. The code is changed now and `toExplainString` (the one used in V1) does not call `getExplainName` (the one used in V2). Names are very confusing btw.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


Re: [PR] Multi stage explain [pinot]

Posted by "gortiz (via GitHub)" <gi...@apache.org>.
gortiz commented on code in PR #13733:
URL: https://github.com/apache/pinot/pull/13733#discussion_r1750210475


##########
pinot-core/src/main/java/org/apache/pinot/core/query/request/context/utils/QueryContextConverterUtils.java:
##########
@@ -155,11 +157,23 @@ public static QueryContext getQueryContext(PinotQuery pinotQuery) {
       }
     }
 
+    ExplainMode explainMode;
+    if (!pinotQuery.isExplain()) {
+      explainMode = ExplainMode.NONE;
+    } else {
+      String useMultistageEngine = CommonConstants.Broker.Request.QueryOptionKey.USE_MULTISTAGE_ENGINE;
+      if (pinotQuery.getQueryOptions() != null && pinotQuery.getQueryOptions().containsKey(useMultistageEngine)) {

Review Comment:
   Fixed in 34c6eca87595f9564d37ff6ee7dc4144e96a2168



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


Re: [PR] Multi stage explain [pinot]

Posted by "gortiz (via GitHub)" <gi...@apache.org>.
gortiz commented on code in PR #13733:
URL: https://github.com/apache/pinot/pull/13733#discussion_r1750308034


##########
pinot-core/src/main/java/org/apache/pinot/core/plan/PinotExplainedRelNode.java:
##########
@@ -0,0 +1,146 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.core.plan;
+
+import com.google.common.collect.Lists;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.List;
+import java.util.Map;
+import org.apache.calcite.plan.RelOptCluster;
+import org.apache.calcite.plan.RelTraitSet;
+import org.apache.calcite.rel.AbstractRelNode;
+import org.apache.calcite.rel.RelNode;
+import org.apache.calcite.rel.RelWriter;
+import org.apache.calcite.rel.type.RelDataType;
+import org.apache.calcite.rel.type.RelDataTypeFactory;
+import org.apache.calcite.rel.type.RelDataTypeField;
+import org.apache.calcite.rel.type.RelDataTypeFieldImpl;
+import org.apache.calcite.rel.type.RelRecordType;
+import org.apache.calcite.rel.type.StructKind;
+import org.apache.pinot.common.proto.Plan;
+import org.apache.pinot.common.utils.DataSchema;
+
+
+/**
+ * This class is a hackish way to adapt Pinot explain plan into Calcite explain plan.
+ *
+ * While in Calcite both logical and physical operators are represented by classes that extend RelNode, in multi-stage
+ * Pinot plans we first optimize {@link RelNode RelNodes} at logical level in the broker and then we transform send
+ * these RelNodes to the server (using {@code org.apache.pinot.query.planner.plannode.PlanNode} as intermediate format).
+ *
+ * Servers then extract leaf nodes from the stage, trying to execute as much as possible in a leaf operator. In that
+ * leaf operator is where nodes are compiled into single-stage physical operators
+ * (aka {@link org.apache.pinot.core.operator.BaseOperator}), which include important information like whether indexes
+ * are being used or not.
+ *
+ * {@link PinotExplainedRelNode} is a way to represent these single-stage operators in Calcite.
+ * <b>They are not meant to be executed</b>.
+ * Instead, when physical explain is required, the broker ask for the final plan to each server. They return
+ * a PlanNode and the broker then converts these PlanNodes into a PinotExplainedRelNode. The logical plan is then
+ * modified to substitute the nodes that were converted into single-stage physical plans with the corresponding
+ * PinotExplainedRelNode.
+ */
+public class PinotExplainedRelNode extends AbstractRelNode {
+
+  /**
+   * The name of this node, whose role is like a title in the explain plan.
+   */
+  private final String _type;
+  private final Map<String, Plan.ExplainNode.AttributeValue> _attributes;
+  private final List<RelNode> _inputs;
+  private final DataSchema _dataSchema;
+
+  public PinotExplainedRelNode(RelOptCluster cluster, RelTraitSet traitSet, String type,

Review Comment:
   Removed



##########
pinot-core/src/main/java/org/apache/pinot/core/plan/PinotExplainedRelNode.java:
##########
@@ -0,0 +1,146 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.core.plan;
+
+import com.google.common.collect.Lists;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.List;
+import java.util.Map;
+import org.apache.calcite.plan.RelOptCluster;
+import org.apache.calcite.plan.RelTraitSet;
+import org.apache.calcite.rel.AbstractRelNode;
+import org.apache.calcite.rel.RelNode;
+import org.apache.calcite.rel.RelWriter;
+import org.apache.calcite.rel.type.RelDataType;
+import org.apache.calcite.rel.type.RelDataTypeFactory;
+import org.apache.calcite.rel.type.RelDataTypeField;
+import org.apache.calcite.rel.type.RelDataTypeFieldImpl;
+import org.apache.calcite.rel.type.RelRecordType;
+import org.apache.calcite.rel.type.StructKind;
+import org.apache.pinot.common.proto.Plan;
+import org.apache.pinot.common.utils.DataSchema;
+
+
+/**
+ * This class is a hackish way to adapt Pinot explain plan into Calcite explain plan.
+ *
+ * While in Calcite both logical and physical operators are represented by classes that extend RelNode, in multi-stage
+ * Pinot plans we first optimize {@link RelNode RelNodes} at logical level in the broker and then we transform send
+ * these RelNodes to the server (using {@code org.apache.pinot.query.planner.plannode.PlanNode} as intermediate format).
+ *
+ * Servers then extract leaf nodes from the stage, trying to execute as much as possible in a leaf operator. In that
+ * leaf operator is where nodes are compiled into single-stage physical operators
+ * (aka {@link org.apache.pinot.core.operator.BaseOperator}), which include important information like whether indexes
+ * are being used or not.
+ *
+ * {@link PinotExplainedRelNode} is a way to represent these single-stage operators in Calcite.
+ * <b>They are not meant to be executed</b>.
+ * Instead, when physical explain is required, the broker ask for the final plan to each server. They return
+ * a PlanNode and the broker then converts these PlanNodes into a PinotExplainedRelNode. The logical plan is then
+ * modified to substitute the nodes that were converted into single-stage physical plans with the corresponding
+ * PinotExplainedRelNode.
+ */
+public class PinotExplainedRelNode extends AbstractRelNode {
+
+  /**
+   * The name of this node, whose role is like a title in the explain plan.
+   */
+  private final String _type;
+  private final Map<String, Plan.ExplainNode.AttributeValue> _attributes;
+  private final List<RelNode> _inputs;
+  private final DataSchema _dataSchema;
+
+  public PinotExplainedRelNode(RelOptCluster cluster, RelTraitSet traitSet, String type,

Review Comment:
   Added



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


Re: [PR] Multi stage explain [pinot]

Posted by "gortiz (via GitHub)" <gi...@apache.org>.
gortiz commented on code in PR #13733:
URL: https://github.com/apache/pinot/pull/13733#discussion_r1750354632


##########
pinot-core/src/main/java/org/apache/pinot/core/operator/ProjectionOperator.java:
##########
@@ -100,6 +100,17 @@ public String toExplainString() {
     return stringBuilder.append(')').toString();
   }
 
+  @Override
+  protected String getExplainName() {
+    return EXPLAIN_NAME;
+  }
+
+  @Override
+  protected void explainAttributes(ExplainAttributeBuilder attributeBuilder) {
+    super.explainAttributes(attributeBuilder);

Review Comment:
   In fact I've just added that in two operators that weren't doing it



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


Re: [PR] Multi stage explain [pinot]

Posted by "gortiz (via GitHub)" <gi...@apache.org>.
gortiz commented on code in PR #13733:
URL: https://github.com/apache/pinot/pull/13733#discussion_r1750300488


##########
pinot-core/src/main/java/org/apache/pinot/core/plan/PinotExplainedRelNode.java:
##########
@@ -0,0 +1,146 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.core.plan;
+
+import com.google.common.collect.Lists;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.List;
+import java.util.Map;
+import org.apache.calcite.plan.RelOptCluster;
+import org.apache.calcite.plan.RelTraitSet;
+import org.apache.calcite.rel.AbstractRelNode;
+import org.apache.calcite.rel.RelNode;
+import org.apache.calcite.rel.RelWriter;
+import org.apache.calcite.rel.type.RelDataType;
+import org.apache.calcite.rel.type.RelDataTypeFactory;
+import org.apache.calcite.rel.type.RelDataTypeField;
+import org.apache.calcite.rel.type.RelDataTypeFieldImpl;
+import org.apache.calcite.rel.type.RelRecordType;
+import org.apache.calcite.rel.type.StructKind;
+import org.apache.pinot.common.proto.Plan;
+import org.apache.pinot.common.utils.DataSchema;
+
+
+/**
+ * This class is a hackish way to adapt Pinot explain plan into Calcite explain plan.
+ *
+ * While in Calcite both logical and physical operators are represented by classes that extend RelNode, in multi-stage
+ * Pinot plans we first optimize {@link RelNode RelNodes} at logical level in the broker and then we transform send
+ * these RelNodes to the server (using {@code org.apache.pinot.query.planner.plannode.PlanNode} as intermediate format).
+ *
+ * Servers then extract leaf nodes from the stage, trying to execute as much as possible in a leaf operator. In that
+ * leaf operator is where nodes are compiled into single-stage physical operators
+ * (aka {@link org.apache.pinot.core.operator.BaseOperator}), which include important information like whether indexes
+ * are being used or not.
+ *
+ * {@link PinotExplainedRelNode} is a way to represent these single-stage operators in Calcite.
+ * <b>They are not meant to be executed</b>.
+ * Instead, when physical explain is required, the broker ask for the final plan to each server. They return
+ * a PlanNode and the broker then converts these PlanNodes into a PinotExplainedRelNode. The logical plan is then
+ * modified to substitute the nodes that were converted into single-stage physical plans with the corresponding
+ * PinotExplainedRelNode.
+ */
+public class PinotExplainedRelNode extends AbstractRelNode {
+
+  /**
+   * The name of this node, whose role is like a title in the explain plan.
+   */
+  private final String _type;
+  private final Map<String, Plan.ExplainNode.AttributeValue> _attributes;
+  private final List<RelNode> _inputs;
+  private final DataSchema _dataSchema;
+
+  public PinotExplainedRelNode(RelOptCluster cluster, RelTraitSet traitSet, String type,
+      Map<String, Plan.ExplainNode.AttributeValue> attributes, DataSchema dataSchema, RelNode input) {
+    this(cluster, traitSet, type, attributes, dataSchema, Lists.newArrayList(input));
+  }
+
+  public PinotExplainedRelNode(RelOptCluster cluster, RelTraitSet traitSet, String type,
+      Map<String, Plan.ExplainNode.AttributeValue> attributes, DataSchema dataSchema, List<? extends RelNode> inputs) {
+    super(cluster, traitSet);
+    _type = type;
+    _attributes = attributes;
+    _inputs = new ArrayList<>(inputs);
+    _dataSchema = dataSchema;
+  }
+
+  @Override
+  protected RelDataType deriveRowType() {
+    int size = _dataSchema.size();
+    List<RelDataTypeField> fields = new ArrayList<>(size);
+    RelDataTypeFactory typeFactory = getCluster().getTypeFactory();
+    for (int i = 0; i < size; i++) {
+      String columnName = _dataSchema.getColumnName(i);
+
+      DataSchema.ColumnDataType columnDataType = _dataSchema.getColumnDataType(i);
+      RelDataType type = columnDataType.toType(typeFactory);
+
+      fields.add(new RelDataTypeFieldImpl(columnName, i, type));
+    }
+    return new RelRecordType(StructKind.FULLY_QUALIFIED, fields, false);
+  }
+
+  @Override
+  public List<RelNode> getInputs() {
+    return Collections.unmodifiableList(_inputs);
+  }
+
+  @Override
+  public void replaceInput(int ordinalInParent, RelNode p) {
+    _inputs.set(ordinalInParent, p);
+  }
+
+  @Override
+  public String getRelTypeName() {
+    return _type;
+  }
+
+  @Override
+  public RelWriter explainTerms(RelWriter pw) {
+    RelWriter relWriter = super.explainTerms(pw);
+    int inputSize = getInputs().size();
+    if (inputSize == 1) {
+      relWriter.input("input", getInputs().get(0));
+    } else if (inputSize > 1) {
+      for (int i = 0; i < inputSize; i++) {
+        relWriter.input("input#" + i, getInputs().get(i));
+      }
+    }

Review Comment:
   > Where is this input term string being used?
   
   It is used internally in RelWriter (in Calcite).
   
   > Does it matter if for a single input node we use input#0 instead of input?
   
   Not really. As far as I can tell the only requirement is that inputs should have a unique name. So each input should be have a different `input#<id>`. How this name is used depends on the Calcite printer and as far as I can see neither of them actually print this name



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


Re: [PR] Multi stage explain [pinot]

Posted by "yashmayya (via GitHub)" <gi...@apache.org>.
yashmayya commented on code in PR #13733:
URL: https://github.com/apache/pinot/pull/13733#discussion_r1758559229


##########
pinot-spi/src/main/java/org/apache/pinot/spi/utils/CommonConstants.java:
##########
@@ -1192,6 +1209,9 @@ public enum WindowOverFlowMode {
     public static class PlanVersions {
       public static final int V1 = 1;
     }
+
+    public static final String ASK_SERVERS_FOR_EXPLAIN_PLAN = "pinot.query.explain.ask.servers";

Review Comment:
   Agreed on usage of `physical`, what about something like `pinot.multistage.explain.query.include.segment.level.plan`? It's fairly verbose but hopefully should be able to convey intent clearly to users.



##########
pinot-query-runtime/src/main/java/org/apache/pinot/query/service/dispatch/DispatchClient.java:
##########
@@ -50,12 +51,17 @@ public ManagedChannel getChannel() {
   }
 
   public void submit(Worker.QueryRequest request, QueryServerInstance virtualServer, Deadline deadline,
-      Consumer<AsyncQueryDispatchResponse> callback) {
-    _dispatchStub.withDeadline(deadline).submit(request, new DispatchObserver(virtualServer, callback));
+      Consumer<AsyncResponse<Worker.QueryResponse>> callback) {
+    _dispatchStub.withDeadline(deadline).submit(request, new LastValueDispatchObserver<>(virtualServer, callback));
   }
 
   public void cancel(long requestId) {
     Worker.CancelRequest cancelRequest = Worker.CancelRequest.newBuilder().setRequestId(requestId).build();
     _dispatchStub.cancel(cancelRequest, NO_OP_CANCEL_STREAM_OBSERVER);
   }
+
+  public void explain(Worker.QueryRequest request, QueryServerInstance virtualServer, Deadline deadline,
+      Consumer<AsyncResponse<List<Worker.ExplainResponse>>> callback) {
+    _dispatchStub.withDeadline(deadline).explain(request, new AllValuesDispatchObserver<>(virtualServer, callback));

Review Comment:
   > What I don't get is why the proto file is defined as:
   >
   >  ```rpc Submit(ServerRequest) returns (stream ServerResponse);```
   > instead of
   >
   > ```  rpc Submit(ServerRequest) returns (ServerResponse);```
   
   That's the proto definition for [GrpcQueryServer](https://github.com/apache/pinot/blob/de577bc457b580e89ecb4e82076ce09f209bca18/pinot-core/src/main/java/org/apache/pinot/core/transport/grpc/GrpcQueryServer.java#L65) right (which is for v1 streaming queries FWICT)? The proto definition for the multi-stage engine's [QueryServer](https://github.com/apache/pinot/blob/de577bc457b580e89ecb4e82076ce09f209bca18/pinot-query-runtime/src/main/java/org/apache/pinot/query/service/server/QueryServer.java#L50) `Submit` RPC is - https://github.com/apache/pinot/blob/de577bc457b580e89ecb4e82076ce09f209bca18/pinot-common/src/main/proto/worker.proto#L26
   
   So I guess this makes sense now since the new `Explain` RPC returns a `stream ExplainResponse` and the implementation does call `onNext` multiple times.



##########
pinot-query-runtime/src/main/java/org/apache/pinot/query/runtime/operator/MultiStageOperator.java:
##########
@@ -167,6 +173,26 @@ protected TransferableBlock updateEosBlock(TransferableBlock upstreamEos, StatMa
     return upstreamEos;
   }
 
+  @Override
+  public ExplainInfo getExplainInfo() {
+    return new ExplainInfo(getExplainName(), getExplainAttributes(), getChildrenExplainInfo());
+  }
+
+  protected List<ExplainInfo> getChildrenExplainInfo() {
+    return getChildOperators().stream()
+        .filter(Objects::nonNull)
+        .map(Operator::getExplainInfo)
+        .collect(Collectors.toList());
+  }
+
+  protected String getExplainName() {
+    return toExplainString();
+  }
+
+  protected Map<String, Plan.ExplainNode.AttributeValue> getExplainAttributes() {
+    return Collections.emptyMap();
+  }

Review Comment:
   Hm, agree on the current state of the `Operator` interface. Thanks for adding Javadocs to all the explain related methods - that should help out quite a bit. I think we can discuss potential refactoring of that interface separately, this looks good for now.



##########
pinot-query-planner/src/main/java/org/apache/pinot/query/planner/logical/PlanNodeToRelConverter.java:
##########
@@ -0,0 +1,468 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.query.planner.logical;
+
+import com.google.common.base.Preconditions;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.List;
+import java.util.stream.Collectors;
+import org.apache.calcite.plan.RelOptCluster;
+import org.apache.calcite.plan.RelTraitSet;
+import org.apache.calcite.rel.RelCollation;
+import org.apache.calcite.rel.RelCollations;
+import org.apache.calcite.rel.RelDistribution;
+import org.apache.calcite.rel.RelDistributions;
+import org.apache.calcite.rel.RelNode;
+import org.apache.calcite.rel.core.SetOp;
+import org.apache.calcite.rel.core.Window;
+import org.apache.calcite.rel.logical.LogicalIntersect;
+import org.apache.calcite.rel.logical.LogicalMinus;
+import org.apache.calcite.rel.logical.LogicalSort;
+import org.apache.calcite.rel.logical.LogicalUnion;
+import org.apache.calcite.rel.logical.LogicalWindow;
+import org.apache.calcite.rel.type.RelDataType;
+import org.apache.calcite.rex.RexLiteral;
+import org.apache.calcite.rex.RexNode;
+import org.apache.calcite.rex.RexWindowBound;
+import org.apache.calcite.rex.RexWindowBounds;
+import org.apache.calcite.sql.SqlAggFunction;
+import org.apache.calcite.tools.RelBuilder;
+import org.apache.calcite.util.ImmutableBitSet;
+import org.apache.pinot.common.utils.DatabaseUtils;
+import org.apache.pinot.core.operator.ExplainAttributeBuilder;
+import org.apache.pinot.core.plan.PinotExplainedRelNode;
+import org.apache.pinot.query.planner.plannode.AggregateNode;
+import org.apache.pinot.query.planner.plannode.ExchangeNode;
+import org.apache.pinot.query.planner.plannode.ExplainedNode;
+import org.apache.pinot.query.planner.plannode.FilterNode;
+import org.apache.pinot.query.planner.plannode.JoinNode;
+import org.apache.pinot.query.planner.plannode.MailboxReceiveNode;
+import org.apache.pinot.query.planner.plannode.MailboxSendNode;
+import org.apache.pinot.query.planner.plannode.PlanNode;
+import org.apache.pinot.query.planner.plannode.PlanNodeVisitor;
+import org.apache.pinot.query.planner.plannode.ProjectNode;
+import org.apache.pinot.query.planner.plannode.SetOpNode;
+import org.apache.pinot.query.planner.plannode.SortNode;
+import org.apache.pinot.query.planner.plannode.TableScanNode;
+import org.apache.pinot.query.planner.plannode.ValueNode;
+import org.apache.pinot.query.planner.plannode.WindowNode;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+
+/**
+ * Converts a {@link PlanNode} into a {@link RelNode}.
+ *
+ * This class is used to convert serialized plan nodes into RelNodes so they can be used when explain with
+ * implementation is requested. Therefore some nodes may be transformed in a way that loses information that is
+ * required to create an actual executable plan but not necessary in order to describe the plan.
+ */
+public final class PlanNodeToRelConverter {
+  private static final Logger LOGGER = LoggerFactory.getLogger(PlanNodeToRelConverter.class);
+
+  private PlanNodeToRelConverter() {
+  }
+
+  public static RelNode convert(RelBuilder builder, PlanNode planNode) {
+    ConverterVisitor visitor = new ConverterVisitor(builder);
+    planNode.visit(visitor, null);
+
+    return visitor.build();
+  }
+
+  private static class ConverterVisitor implements PlanNodeVisitor<Void, Void> {

Review Comment:
   Makes sense, thanks for elaborating! 😄 



##########
pinot-query-planner/src/main/java/org/apache/pinot/query/QueryEnvironment.java:
##########
@@ -172,7 +185,27 @@ public QueryPlannerResult explainQuery(String sqlQuery, SqlNodeAndOptions sqlNod
         SqlExplainLevel level =
             explain.getDetailLevel() == null ? SqlExplainLevel.DIGEST_ATTRIBUTES : explain.getDetailLevel();
         Set<String> tableNames = RelToPlanNodeConverter.getTableNamesFromRelRoot(relRoot.rel);
-        return new QueryPlannerResult(null, PlannerUtils.explainPlan(relRoot.rel, format, level), tableNames);
+        if (!explain.withImplementation() || !askServers) {

Review Comment:
   Sounds good, we can also plan to deprecate the `EXPLAIN IMPLEMENTATION PLAN FOR` syntax and change it to one of the `WITH <extension>` in the future.



##########
pinot-core/src/main/java/org/apache/pinot/core/query/request/context/QueryContext.java:
##########
@@ -85,7 +85,7 @@ public class QueryContext {
   private final int _offset;
   private final Map<String, String> _queryOptions;
   private final Map<ExpressionContext, ExpressionContext> _expressionOverrideHints;
-  private final boolean _explain;
+  private final ExplainMode _explain;

Review Comment:
   Fair enough, that makes sense. I can't think of better names either (`QueryMode` came to mind but agree with it being generic / potentially misleading); naming things is hard. 😄 



##########
pinot-query-runtime/src/main/java/org/apache/pinot/query/runtime/QueryRunner.java:
##########
@@ -256,4 +262,66 @@ private Map<String, String> consolidateMetadata(Map<String, String> customProper
   public void cancel(long requestId) {
     _opChainScheduler.cancel(requestId);
   }
+
+  public StagePlan explainQuery(
+      WorkerMetadata workerMetadata, StagePlan stagePlan, Map<String, String> requestMetadata) {
+
+    if (!workerMetadata.isLeafStageWorker()) {
+      LOGGER.debug("Explain query on intermediate stages is a NOOP");
+      return stagePlan;
+    }
+    long requestId = Long.parseLong(requestMetadata.get(CommonConstants.Query.Request.MetadataKeys.REQUEST_ID));
+    long timeoutMs = Long.parseLong(requestMetadata.get(CommonConstants.Broker.Request.QueryOptionKey.TIMEOUT_MS));
+    long deadlineMs = System.currentTimeMillis() + timeoutMs;
+
+    StageMetadata stageMetadata = stagePlan.getStageMetadata();
+    Map<String, String> opChainMetadata = consolidateMetadata(stageMetadata.getCustomProperties(), requestMetadata);
+
+    if (PipelineBreakerExecutor.hasPipelineBreakers(stagePlan)) {
+      // TODO: Support pipeline breakers before merging this feature.
+      LOGGER.error("Pipeline breaker is not supported in explain query");
+      return stagePlan;
+    }

Review Comment:
   Ah, I hadn't thought of that either, thanks for the explanation! I guess we can just update that TODO comment for now, it makes sense to defer this considering the current explain also doesn't properly support pipeline breaker.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


Re: [PR] Multi stage explain [pinot]

Posted by "gortiz (via GitHub)" <gi...@apache.org>.
gortiz commented on PR #13733:
URL: https://github.com/apache/pinot/pull/13733#issuecomment-2368118477

   > Thanks @gortiz, LGTM! There are some new tests added in https://github.com/apache/pinot/pull/13999 failing with an NPE here though (and looks related to the explain changes).
   
   Yes, I've fixed that, but now the same tests fail due to some assertion in the test I don't understand. That is why we need to merge this ASAP. The code in `ServerQueryExecutorV1Impl` is very sensible and it has been modified more often that we used due to the timeseries code


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


Re: [PR] Multi stage explain [pinot]

Posted by "gortiz (via GitHub)" <gi...@apache.org>.
gortiz commented on code in PR #13733:
URL: https://github.com/apache/pinot/pull/13733#discussion_r1765145450


##########
pinot-query-runtime/src/main/java/org/apache/pinot/query/service/dispatch/DispatchClient.java:
##########
@@ -50,12 +51,17 @@ public ManagedChannel getChannel() {
   }
 
   public void submit(Worker.QueryRequest request, QueryServerInstance virtualServer, Deadline deadline,
-      Consumer<AsyncQueryDispatchResponse> callback) {
-    _dispatchStub.withDeadline(deadline).submit(request, new DispatchObserver(virtualServer, callback));
+      Consumer<AsyncResponse<Worker.QueryResponse>> callback) {
+    _dispatchStub.withDeadline(deadline).submit(request, new LastValueDispatchObserver<>(virtualServer, callback));
   }
 
   public void cancel(long requestId) {
     Worker.CancelRequest cancelRequest = Worker.CancelRequest.newBuilder().setRequestId(requestId).build();
     _dispatchStub.cancel(cancelRequest, NO_OP_CANCEL_STREAM_OBSERVER);
   }
+
+  public void explain(Worker.QueryRequest request, QueryServerInstance virtualServer, Deadline deadline,
+      Consumer<AsyncResponse<List<Worker.ExplainResponse>>> callback) {
+    _dispatchStub.withDeadline(deadline).explain(request, new AllValuesDispatchObserver<>(virtualServer, callback));

Review Comment:
   You are right, the Submit endpoint is not defined as stream, so it makes sense to only keep the last (as only 1 is expected).
   
   BTW I don't think submit is used for V1. Remember we have two classes called QueryServer. `org.apache.pinot.query.service.server.QueryServer` is the one used in V2 and `org.apache.pinot.core.transport.QueryServer` is the one used in V1. Then we have ` org.apache.pinot.core.transport.grpc.GrpcQueryServer`. The later is used to execute V1 queries using GPRC and implements the protocol defined in `server.proto`.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


Re: [PR] Multi stage explain [pinot]

Posted by "gortiz (via GitHub)" <gi...@apache.org>.
gortiz commented on code in PR #13733:
URL: https://github.com/apache/pinot/pull/13733#discussion_r1765130608


##########
pinot-spi/src/main/java/org/apache/pinot/spi/utils/CommonConstants.java:
##########
@@ -1192,6 +1209,9 @@ public enum WindowOverFlowMode {
     public static class PlanVersions {
       public static final int V1 = 1;
     }
+
+    public static final String ASK_SERVERS_FOR_EXPLAIN_PLAN = "pinot.query.explain.ask.servers";

Review Comment:
   Changed to `pinot.query.multistage.explain.include.segment.plan`. I prefer to try to use the property name as a classifier. First `pinot`, then `query`, then `multistage` and `explain` 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


Re: [PR] Multi stage explain [pinot]

Posted by "gortiz (via GitHub)" <gi...@apache.org>.
gortiz commented on code in PR #13733:
URL: https://github.com/apache/pinot/pull/13733#discussion_r1752014193


##########
pinot-query-runtime/src/main/java/org/apache/pinot/query/runtime/QueryRunner.java:
##########
@@ -256,4 +262,66 @@ private Map<String, String> consolidateMetadata(Map<String, String> customProper
   public void cancel(long requestId) {
     _opChainScheduler.cancel(requestId);
   }
+
+  public StagePlan explainQuery(
+      WorkerMetadata workerMetadata, StagePlan stagePlan, Map<String, String> requestMetadata) {
+
+    if (!workerMetadata.isLeafStageWorker()) {
+      LOGGER.debug("Explain query on intermediate stages is a NOOP");
+      return stagePlan;
+    }
+    long requestId = Long.parseLong(requestMetadata.get(CommonConstants.Query.Request.MetadataKeys.REQUEST_ID));
+    long timeoutMs = Long.parseLong(requestMetadata.get(CommonConstants.Broker.Request.QueryOptionKey.TIMEOUT_MS));
+    long deadlineMs = System.currentTimeMillis() + timeoutMs;
+
+    StageMetadata stageMetadata = stagePlan.getStageMetadata();
+    Map<String, String> opChainMetadata = consolidateMetadata(stageMetadata.getCustomProperties(), requestMetadata);
+
+    if (PipelineBreakerExecutor.hasPipelineBreakers(stagePlan)) {
+      // TODO: Support pipeline breakers before merging this feature.
+      LOGGER.error("Pipeline breaker is not supported in explain query");
+      return stagePlan;
+    }

Review Comment:
   I modified the description yesterday to indicate the PipelineBreaker state. Basically we in the current explain we just prune these nodes. In the new explain we will fail with that error shown here.
   
   We can work on that feature in the future.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


Re: [PR] Multi stage explain [pinot]

Posted by "gortiz (via GitHub)" <gi...@apache.org>.
gortiz commented on code in PR #13733:
URL: https://github.com/apache/pinot/pull/13733#discussion_r1752011166


##########
pinot-common/src/main/proto/plan.proto:
##########
@@ -193,3 +194,29 @@ message WindowNode {
   int32 upperBound = 6;
   repeated Literal constants = 7;
 }
+
+message ExplainNode {
+  string type = 1;

Review Comment:
   I think title is a better name. Type seems to indicate that it is a unique thing that completely identifies the operator, when in fact it may not be. I've added some documentation which emphasizes that explain nodes should not be used to reconstruct the operators.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


Re: [PR] Multi stage explain [pinot]

Posted by "gortiz (via GitHub)" <gi...@apache.org>.
gortiz commented on code in PR #13733:
URL: https://github.com/apache/pinot/pull/13733#discussion_r1752092256


##########
pinot-query-planner/src/main/java/org/apache/pinot/query/planner/logical/PlanNodeToRelConverter.java:
##########
@@ -0,0 +1,468 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.query.planner.logical;
+
+import com.google.common.base.Preconditions;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.List;
+import java.util.stream.Collectors;
+import org.apache.calcite.plan.RelOptCluster;
+import org.apache.calcite.plan.RelTraitSet;
+import org.apache.calcite.rel.RelCollation;
+import org.apache.calcite.rel.RelCollations;
+import org.apache.calcite.rel.RelDistribution;
+import org.apache.calcite.rel.RelDistributions;
+import org.apache.calcite.rel.RelNode;
+import org.apache.calcite.rel.core.SetOp;
+import org.apache.calcite.rel.core.Window;
+import org.apache.calcite.rel.logical.LogicalIntersect;
+import org.apache.calcite.rel.logical.LogicalMinus;
+import org.apache.calcite.rel.logical.LogicalSort;
+import org.apache.calcite.rel.logical.LogicalUnion;
+import org.apache.calcite.rel.logical.LogicalWindow;
+import org.apache.calcite.rel.type.RelDataType;
+import org.apache.calcite.rex.RexLiteral;
+import org.apache.calcite.rex.RexNode;
+import org.apache.calcite.rex.RexWindowBound;
+import org.apache.calcite.rex.RexWindowBounds;
+import org.apache.calcite.sql.SqlAggFunction;
+import org.apache.calcite.tools.RelBuilder;
+import org.apache.calcite.util.ImmutableBitSet;
+import org.apache.pinot.common.utils.DatabaseUtils;
+import org.apache.pinot.core.operator.ExplainAttributeBuilder;
+import org.apache.pinot.core.plan.PinotExplainedRelNode;
+import org.apache.pinot.query.planner.plannode.AggregateNode;
+import org.apache.pinot.query.planner.plannode.ExchangeNode;
+import org.apache.pinot.query.planner.plannode.ExplainedNode;
+import org.apache.pinot.query.planner.plannode.FilterNode;
+import org.apache.pinot.query.planner.plannode.JoinNode;
+import org.apache.pinot.query.planner.plannode.MailboxReceiveNode;
+import org.apache.pinot.query.planner.plannode.MailboxSendNode;
+import org.apache.pinot.query.planner.plannode.PlanNode;
+import org.apache.pinot.query.planner.plannode.PlanNodeVisitor;
+import org.apache.pinot.query.planner.plannode.ProjectNode;
+import org.apache.pinot.query.planner.plannode.SetOpNode;
+import org.apache.pinot.query.planner.plannode.SortNode;
+import org.apache.pinot.query.planner.plannode.TableScanNode;
+import org.apache.pinot.query.planner.plannode.ValueNode;
+import org.apache.pinot.query.planner.plannode.WindowNode;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+
+/**
+ * Converts a {@link PlanNode} into a {@link RelNode}.
+ *
+ * This class is used to convert serialized plan nodes into RelNodes so they can be used when explain with
+ * implementation is requested. Therefore some nodes may be transformed in a way that loses information that is
+ * required to create an actual executable plan but not necessary in order to describe the plan.
+ */
+public final class PlanNodeToRelConverter {
+  private static final Logger LOGGER = LoggerFactory.getLogger(PlanNodeToRelConverter.class);
+
+  private PlanNodeToRelConverter() {
+  }
+
+  public static RelNode convert(RelBuilder builder, PlanNode planNode) {
+    ConverterVisitor visitor = new ConverterVisitor(builder);
+    planNode.visit(visitor, null);
+
+    return visitor.build();
+  }
+
+  private static class ConverterVisitor implements PlanNodeVisitor<Void, Void> {

Review Comment:
   It is similar to the other visitor I've converted. The fact that the implementation uses a visitor is an implementation detail we don't need to leak into the caller. Imagine the case you mention. Would you be able to know if you can reuse the same visitor in two different calls? Why would you need to call builder.build() in your code? What is easier to read?
   
   ```java
   Builder builder = ...;
   PlanNode planNode = ...;
   RelNode relNode = PlanNodeToRelConverter.convert(builder, planNode);
   ```
   
   or
   
   ```java
   Builder builder = ...;
   PlanNode planNode = ...;
   PlanNodeToRelConverter visitor = new PlanNodeToRelConverter.convert(builder, planNode);
   planNode.visit(visitor, null);
   RelNode relNode = builder.build();
   ```
   
   Also... the `convert` method has no javadoc but in case we have to add it, it is simpler to add it there.
   
   I love visitors (although I would love even more to be able to use sealed classes) but they are a poor man switch. Switches are great tools to use in your functions, but in Java we offer methods to be called, not switches.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


Re: [PR] Multi stage explain [pinot]

Posted by "gortiz (via GitHub)" <gi...@apache.org>.
gortiz commented on code in PR #13733:
URL: https://github.com/apache/pinot/pull/13733#discussion_r1747183652


##########
pinot-query-runtime/src/main/java/org/apache/pinot/query/runtime/operator/MultiStageOperator.java:
##########
@@ -167,6 +173,26 @@ protected TransferableBlock updateEosBlock(TransferableBlock upstreamEos, StatMa
     return upstreamEos;
   }
 
+  @Override
+  public ExplainInfo getExplainInfo() {
+    return new ExplainInfo(getExplainName(), getExplainAttributes(), getChildrenExplainInfo());
+  }
+
+  protected List<ExplainInfo> getChildrenExplainInfo() {
+    return getChildOperators().stream()
+        .filter(Objects::nonNull)
+        .map(Operator::getExplainInfo)
+        .collect(Collectors.toList());
+  }
+
+  protected String getExplainName() {
+    return toExplainString();
+  }
+
+  protected Map<String, Plan.ExplainNode.AttributeValue> getExplainAttributes() {
+    return Collections.emptyMap();
+  }

Review Comment:
   That is what I tried on the first iteration but... these method should be public, which is something I don't like because that is not part of the `Operator` interface. We could create another abstract class that extended by both `BaseOperator` and `MultiStageOperator`. I'm open to add that change, but I think this is simpler for now.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


Re: [PR] Multi stage explain [pinot]

Posted by "gortiz (via GitHub)" <gi...@apache.org>.
gortiz commented on code in PR #13733:
URL: https://github.com/apache/pinot/pull/13733#discussion_r1747199174


##########
pinot-query-planner/src/main/java/org/apache/pinot/query/planner/explain/ExplainNodeSimplifier.java:
##########
@@ -0,0 +1,169 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.query.planner.explain;
+
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import org.apache.pinot.common.proto.Plan;
+import org.apache.pinot.core.query.reduce.ExplainPlanDataTableReducer;
+import org.apache.pinot.query.planner.plannode.AggregateNode;
+import org.apache.pinot.query.planner.plannode.ExchangeNode;
+import org.apache.pinot.query.planner.plannode.ExplainedNode;
+import org.apache.pinot.query.planner.plannode.FilterNode;
+import org.apache.pinot.query.planner.plannode.JoinNode;
+import org.apache.pinot.query.planner.plannode.MailboxReceiveNode;
+import org.apache.pinot.query.planner.plannode.MailboxSendNode;
+import org.apache.pinot.query.planner.plannode.PlanNode;
+import org.apache.pinot.query.planner.plannode.PlanNodeVisitor;
+import org.apache.pinot.query.planner.plannode.ProjectNode;
+import org.apache.pinot.query.planner.plannode.SetOpNode;
+import org.apache.pinot.query.planner.plannode.SortNode;
+import org.apache.pinot.query.planner.plannode.TableScanNode;
+import org.apache.pinot.query.planner.plannode.ValueNode;
+import org.apache.pinot.query.planner.plannode.WindowNode;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+
+public class ExplainNodeSimplifier {
+  private static final Logger LOGGER = LoggerFactory.getLogger(ExplainNodeSimplifier.class);
+
+  private ExplainNodeSimplifier() {
+  }
+
+  public static PlanNode simplifyNode(PlanNode plan1) {
+    Visitor planNodeMerger = new Visitor();
+    return plan1.visit(planNodeMerger, null);
+  }
+
+  private static class Visitor implements PlanNodeVisitor<PlanNode, Void> {
+    private static final String REPEAT_ATTRIBUTE_KEY = "repeatedOnSegments";
+
+    private PlanNode defaultNode(PlanNode node) {
+      List<PlanNode> inputs = node.getInputs();
+      List<PlanNode> newInputs = simplifyChildren(inputs);
+      return inputs != newInputs ? node.withInputs(newInputs) : node;
+    }
+
+    @Override
+    public PlanNode visitAggregate(AggregateNode node, Void context) {
+      return defaultNode(node);
+    }
+
+    @Override
+    public PlanNode visitFilter(FilterNode node, Void context) {
+      return defaultNode(node);
+    }
+
+    @Override
+    public PlanNode visitJoin(JoinNode node, Void context) {
+      return defaultNode(node);
+    }
+
+    @Override
+    public PlanNode visitMailboxReceive(MailboxReceiveNode node, Void context) {
+      return defaultNode(node);
+    }
+
+    @Override
+    public PlanNode visitMailboxSend(MailboxSendNode node, Void context) {
+      return defaultNode(node);
+    }
+
+    @Override
+    public PlanNode visitProject(ProjectNode node, Void context) {
+      return defaultNode(node);
+    }
+
+    @Override
+    public PlanNode visitSort(SortNode node, Void context) {
+      return defaultNode(node);
+    }
+
+    @Override
+    public PlanNode visitTableScan(TableScanNode node, Void context) {
+      return defaultNode(node);
+    }
+
+    @Override
+    public PlanNode visitValue(ValueNode node, Void context) {
+      return defaultNode(node);
+    }
+
+    @Override
+    public PlanNode visitWindow(WindowNode node, Void context) {
+      return defaultNode(node);
+    }
+
+    @Override
+    public PlanNode visitSetOp(SetOpNode node, Void context) {
+      return defaultNode(node);
+    }
+
+    @Override
+    public PlanNode visitExchange(ExchangeNode node, Void context) {
+      return defaultNode(node);
+    }
+
+    @Override
+    public PlanNode visitExplained(ExplainedNode node, Void context) {
+      if (!node.getType().contains(ExplainPlanDataTableReducer.COMBINE) || node.getInputs().size() <= 1) {
+        return defaultNode(node);
+      }
+      List<PlanNode> simplifiedChildren = simplifyChildren(node.getInputs());
+      PlanNode child1 = simplifiedChildren.get(0);
+
+      for (int i = 1; i < simplifiedChildren.size(); i++) {
+        PlanNode child2 = simplifiedChildren.get(i);
+        PlanNode merged = PlanNodeMerger.mergePlans(child1, child2, false);
+        if (merged == null) {
+          LOGGER.info("Found unmergeable inputs on node of type {}: {} and {}", node, child1, child2);
+          assert false : "Unmergeable inputs found";

Review Comment:
   We shouldn't reach this code. All nodes at this point should be mergeable. In case we end up reaching this code in production, it would be better to not fail. But during tests we should to make it easier to find the problem.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org