[GitHub] [pinot] walterddr commented on a diff in pull request #9344: [Feature][multistage] Thread-safe query planning

[GitBox <gi...@apache.org>]

walterddr commented on code in PR #9344:
URL: https://github.com/apache/pinot/pull/9344#discussion_r966214894


##########
pinot-query-planner/src/test/java/org/apache/pinot/query/QueryCompilationTest.java:
##########
@@ -148,6 +149,36 @@ public void testQueryProjectFilterPushDownForJoin() {
     }
   }
 
+  // Test that plan query can be run as multi-thread.
+  @Test
+  public void testPlanQueryMultiThread()
+      throws Exception {
+    Runnable planQuery = () -> {
+      String query = "SELECT a.col1, a.ts, b.col2, b.col3 FROM a JOIN b ON a.col1 = b.col2 "
+          + "WHERE a.col3 >= 0 AND a.col2 IN  ('a', 'b') AND b.col3 < 0";
+      QueryPlan queryPlan = _queryEnvironment.planQuery(query);
+      List<StageNode> intermediateStageRoots =
+          queryPlan.getStageMetadataMap().entrySet().stream().filter(e -> e.getValue().getScannedTables().size() == 0)
+              .map(e -> queryPlan.getQueryStageMap().get(e.getKey())).collect(Collectors.toList());
+      // Assert that no project of filter node for any intermediate stage because all should've been pushed down.
+      for (StageNode roots : intermediateStageRoots) {
+        assertNodeTypeNotIn(roots, ImmutableList.of(ProjectNode.class, FilterNode.class));
+      }
+    };

Review Comment:
   no need to assert this. we should test something like this.
   - plan 2 type of SQL (say 
     - type a: SELECT * FROM tbl and 
     - type b: SELECT xxx JOIN)
   - run 4 planning (a1, b1, a2, b2) at the same time
   - assert that a1 plan == a2 plan, same for b type



##########
pinot-query-planner/src/main/java/org/apache/pinot/query/context/PlannerContext.java:
##########
@@ -19,15 +19,50 @@
 package org.apache.pinot.query.context;
 
 import java.util.Map;
-
+import org.apache.calcite.plan.Contexts;
+import org.apache.calcite.plan.RelOptPlanner;
+import org.apache.calcite.plan.hep.HepProgram;
+import org.apache.calcite.prepare.PlannerImpl;
+import org.apache.calcite.prepare.Prepare;
+import org.apache.calcite.rel.type.RelDataTypeFactory;
+import org.apache.calcite.sql.fun.SqlStdOperatorTable;
+import org.apache.calcite.sql.validate.SqlValidator;
+import org.apache.calcite.tools.FrameworkConfig;
+import org.apache.pinot.query.planner.logical.LogicalPlanner;
+import org.apache.pinot.query.validate.Validator;
 
 /**
  * PlannerContext is an object that holds all contextual information during planning phase.
  *
- * TODO: currently the planner context is not used since we don't support option or query rewrite. This construct is
- * here as a placeholder for the parsed out options.
+ * TODO: currently we don't support option or query rewrite.
+ * It is used to hold per query context for query planning, which cannot be shared across queries.
  */
 public class PlannerContext {

Review Comment:
   let's make this closeable and make sure the planner is closed 
   ```suggestion
   public class PlannerContext implement Closeable {
   ```



##########
pinot-query-planner/src/main/java/org/apache/pinot/query/context/PlannerContext.java:
##########
@@ -19,15 +19,50 @@
 package org.apache.pinot.query.context;
 
 import java.util.Map;
-
+import org.apache.calcite.plan.Contexts;
+import org.apache.calcite.plan.RelOptPlanner;
+import org.apache.calcite.plan.hep.HepProgram;
+import org.apache.calcite.prepare.PlannerImpl;
+import org.apache.calcite.prepare.Prepare;
+import org.apache.calcite.rel.type.RelDataTypeFactory;
+import org.apache.calcite.sql.fun.SqlStdOperatorTable;
+import org.apache.calcite.sql.validate.SqlValidator;
+import org.apache.calcite.tools.FrameworkConfig;
+import org.apache.pinot.query.planner.logical.LogicalPlanner;
+import org.apache.pinot.query.validate.Validator;
 
 /**
  * PlannerContext is an object that holds all contextual information during planning phase.
  *
- * TODO: currently the planner context is not used since we don't support option or query rewrite. This construct is
- * here as a placeholder for the parsed out options.
+ * TODO: currently we don't support option or query rewrite.
+ * It is used to hold per query context for query planning, which cannot be shared across queries.
  */
 public class PlannerContext {
+  public PlannerContext(FrameworkConfig config, Prepare.CatalogReader catalogReader, RelDataTypeFactory typeFactory,

Review Comment:
   great idea encoding these non-threadsafe objects into plannerContext!



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org