You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@iotdb.apache.org by "shizy818 (via GitHub)" <gi...@apache.org> on 2025/09/05 02:45:11 UTC

[PR] With clause enhance (Review Only Now!) [iotdb]

shizy818 opened a new pull request, #16353:
URL: https://github.com/apache/iotdb/pull/16353

   ## Description
   
   
   ### Content1 ...
   
   ### Content2 ...
   
   ### Content3 ...
   
   <!--
   In each section, please describe design decisions made, including:
    - Choice of algorithms
    - Behavioral aspects. What configuration values are acceptable? How are corner cases and error 
       conditions handled, such as when there are insufficient resources?
    - Class organization and design (how the logic is split between classes, inheritance, composition, 
       design patterns)
    - Method organization and design (how the logic is split between methods, parameters and return types)
    - Naming (class, method, API, configuration, HTTP endpoint, names of emitted metrics)
   -->
   
   
   <!-- It's good to describe an alternative design (or mention an alternative name) for every design 
   (or naming) decision point and compare the alternatives with the designs that you've implemented 
   (or the names you've chosen) to highlight the advantages of the chosen designs and names. -->
   
   <!-- If there was a discussion of the design of the feature implemented in this PR elsewhere 
   (e. g. a "Proposal" issue, any other issue, or a thread in the development mailing list), 
   link to that discussion from this PR description and explain what have changed in your final design 
   compared to your original proposal or the consensus version in the end of the discussion. 
   If something hasn't changed since the original discussion, you can omit a detailed discussion of 
   those aspects of the design here, perhaps apart from brief mentioning for the sake of readability 
   of this PR description. -->
   
   <!-- Some of the aspects mentioned above may be omitted for simple and small changes. -->
   
   <hr>
   
   This PR has:
   - [ ] been self-reviewed.
       - [ ] concurrent read
       - [ ] concurrent write
       - [ ] concurrent read and write 
   - [ ] added documentation for new or modified features or behaviors.
   - [ ] added Javadocs for most classes and all non-trivial methods. 
   - [ ] added or updated version, __license__, or notice information
   - [ ] added comments explaining the "why" and the intent of the code wherever would not be obvious 
     for an unfamiliar reader.
   - [ ] added unit tests or modified existing tests to cover new code paths, ensuring the threshold 
     for code coverage.
   - [ ] added integration tests.
   - [ ] been tested in a test IoTDB cluster.
   
   <!-- Check the items by putting "x" in the brackets for the done things. Not all of these items 
   apply to every PR. Remove the items which are not done or not relevant to the PR. None of the items 
   from the checklist above are strictly necessary, but it would be very helpful if you at least 
   self-review the PR. -->
   
   <hr>
   
   ##### Key changed/added classes (or packages if there are too many classes) in this PR
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] With clause enhance [iotdb]

Posted by "shizy818 (via GitHub)" <gi...@apache.org>.
shizy818 commented on code in PR #16353:
URL: https://github.com/apache/iotdb/pull/16353#discussion_r2645253717


##########
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/conf/IoTDBDescriptor.java:
##########
@@ -1057,6 +1057,24 @@ public void loadProperties(TrimProperties properties) throws BadNodeUrlException
     // The buffer for sort operator to calculate
     loadFixedSizeLimitForQuery(properties, "sort_buffer_size_in_bytes", conf::setSortBufferSize);
 
+    // The buffer for cte materialization.
+    long cteBufferSizeInBytes =
+        Long.parseLong(
+            properties.getProperty(
+                "cte_buffer_size_in_bytes", Long.toString(conf.getCteBufferSize())));
+    if (cteBufferSizeInBytes > 0) {
+      conf.setCteBufferSize(cteBufferSizeInBytes);
+    }
+
+    // max number of rows for cte materialization
+    int maxRowsInCteBuffer =
+        Integer.parseInt(
+            properties.getProperty(
+                "max_rows_in_cte_buffer", Integer.toString(conf.getMaxRowsInCteBuffer())));
+    if (maxRowsInCteBuffer > 0) {
+      conf.setMaxRowsInCteBuffer(maxRowsInCteBuffer);
+    }
+

Review Comment:
   done



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] With clause enhance [iotdb]

Posted by "shizy818 (via GitHub)" <gi...@apache.org>.
shizy818 commented on code in PR #16353:
URL: https://github.com/apache/iotdb/pull/16353#discussion_r2645257425


##########
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/queryengine/execution/operator/source/relational/CteScanOperator.java:
##########
@@ -0,0 +1,134 @@
+/*
+ *
+ *  * Licensed to the Apache Software Foundation (ASF) under one
+ *  * or more contributor license agreements.  See the NOTICE file
+ *  * distributed with this work for additional information
+ *  * regarding copyright ownership.  The ASF licenses this file
+ *  * to you under the Apache License, Version 2.0 (the
+ *  * "License"); you may not use this file except in compliance
+ *  * with the License.  You may obtain a copy of the License at
+ *  *
+ *  *     http://www.apache.org/licenses/LICENSE-2.0
+ *  *
+ *  * Unless required by applicable law or agreed to in writing,
+ *  * software distributed under the License is distributed on an
+ *  * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ *  * KIND, either express or implied.  See the License for the
+ *  * specific language governing permissions and limitations
+ *  * under the License.
+ *
+ */
+
+package org.apache.iotdb.db.queryengine.execution.operator.source.relational;
+
+import org.apache.iotdb.commons.utils.TestOnly;
+import org.apache.iotdb.db.queryengine.execution.MemoryEstimationHelper;
+import org.apache.iotdb.db.queryengine.execution.operator.OperatorContext;
+import org.apache.iotdb.db.queryengine.execution.operator.source.SourceOperator;
+import org.apache.iotdb.db.queryengine.plan.planner.plan.node.PlanNodeId;
+import org.apache.iotdb.db.utils.cte.CteDataReader;
+import org.apache.iotdb.db.utils.cte.CteDataStore;
+import org.apache.iotdb.db.utils.cte.MemoryReader;
+
+import org.apache.tsfile.read.common.block.TsBlock;
+import org.apache.tsfile.utils.RamUsageEstimator;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+public class CteScanOperator implements SourceOperator {
+  private static final Logger LOGGER = LoggerFactory.getLogger(CteScanOperator.class);
+  private static final long INSTANCE_SIZE =
+      RamUsageEstimator.shallowSizeOfInstance(CteScanOperator.class);
+
+  private final OperatorContext operatorContext;
+  private final PlanNodeId sourceId;
+
+  private final CteDataStore dataStore;
+  private final CteDataReader dataReader;
+  private final int dataStoreRefCount;
+
+  public CteScanOperator(
+      OperatorContext operatorContext, PlanNodeId sourceId, CteDataStore dataStore) {
+    this.operatorContext = operatorContext;
+    this.sourceId = sourceId;
+    this.dataStore = dataStore;
+    this.dataReader = new MemoryReader(dataStore.getCachedData());
+    this.dataStoreRefCount = dataStore.increaseRefCount();
+  }
+
+  @Override
+  public TsBlock next() throws Exception {
+    if (dataReader == null) {
+      return null;
+    }
+    return dataReader.next();
+  }
+
+  @Override
+  public boolean hasNext() throws Exception {
+    if (dataReader == null) {
+      return false;
+    }
+    return dataReader.hasNext();
+  }
+
+  @Override
+  public void close() throws Exception {
+    try {
+      if (dataReader != null) {
+        dataReader.close();
+      }
+    } catch (Exception e) {
+      LOGGER.error("Fail to close fileChannel", e);

Review Comment:
   done



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] With clause enhance [iotdb]

Posted by "shizy818 (via GitHub)" <gi...@apache.org>.
shizy818 commented on code in PR #16353:
URL: https://github.com/apache/iotdb/pull/16353#discussion_r2645087594


##########
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/queryengine/common/MPPQueryContext.java:
##########
@@ -103,6 +125,24 @@ public class MPPQueryContext implements IAuditEntity {
 
   private boolean userQuery = false;
 
+  private Map<NodeRef<Table>, Query> cteQueries = new HashMap<>();
+
+  // Stores the EXPLAIN/EXPLAIN ANALYZE results for Common Table Expressions (CTEs)
+  // Key: CTE table reference
+  // Value: Pair containing (max line length of the explain output, list of formatted explain lines)
+  // This ensures consistent formatting between the main query and its CTE sub-queries
+  private final Map<NodeRef<Table>, Pair<Integer, List<String>>> cteExplainResults =
+      new LinkedHashMap<>();
+  // Tracks the materialization time cost (in nanoseconds) for each CTE to help optimize query
+  // planning
+  private final Map<NodeRef<Table>, Long> cteMaterializationCosts = new HashMap<>();
+
+  // Never materialize CTE in a subquery.
+  private boolean subquery = false;

Review Comment:
   I just mean subquery. It is used in `createOutputPlan`, `createRelationPlan` and `createRelationPlan`.  As long as it's subquery, I think the logic is same.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] With clause enhance [iotdb]

Posted by "shizy818 (via GitHub)" <gi...@apache.org>.
shizy818 commented on code in PR #16353:
URL: https://github.com/apache/iotdb/pull/16353#discussion_r2647450753


##########
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/queryengine/plan/relational/planner/ir/PredicateWithUncorrelatedScalarSubqueryReconstructor.java:
##########
@@ -0,0 +1,247 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iotdb.db.queryengine.plan.relational.planner.ir;
+
+import org.apache.iotdb.commons.exception.IoTDBException;
+import org.apache.iotdb.commons.utils.TestOnly;
+import org.apache.iotdb.db.protocol.session.SessionManager;
+import org.apache.iotdb.db.queryengine.common.MPPQueryContext;
+import org.apache.iotdb.db.queryengine.common.MPPQueryContext.ExplainType;
+import org.apache.iotdb.db.queryengine.common.header.DatasetHeader;
+import org.apache.iotdb.db.queryengine.plan.Coordinator;
+import org.apache.iotdb.db.queryengine.plan.execution.ExecutionResult;
+import org.apache.iotdb.db.queryengine.plan.planner.LocalExecutionPlanner;
+import org.apache.iotdb.db.queryengine.plan.relational.analyzer.Analysis;
+import org.apache.iotdb.db.queryengine.plan.relational.sql.ast.BinaryLiteral;
+import org.apache.iotdb.db.queryengine.plan.relational.sql.ast.BooleanLiteral;
+import org.apache.iotdb.db.queryengine.plan.relational.sql.ast.ComparisonExpression;
+import org.apache.iotdb.db.queryengine.plan.relational.sql.ast.DoubleLiteral;
+import org.apache.iotdb.db.queryengine.plan.relational.sql.ast.Expression;
+import org.apache.iotdb.db.queryengine.plan.relational.sql.ast.FunctionCall;
+import org.apache.iotdb.db.queryengine.plan.relational.sql.ast.Identifier;
+import org.apache.iotdb.db.queryengine.plan.relational.sql.ast.Literal;
+import org.apache.iotdb.db.queryengine.plan.relational.sql.ast.LogicalExpression;
+import org.apache.iotdb.db.queryengine.plan.relational.sql.ast.LongLiteral;
+import org.apache.iotdb.db.queryengine.plan.relational.sql.ast.NotExpression;
+import org.apache.iotdb.db.queryengine.plan.relational.sql.ast.Query;
+import org.apache.iotdb.db.queryengine.plan.relational.sql.ast.StringLiteral;
+import org.apache.iotdb.db.queryengine.plan.relational.sql.ast.SubqueryExpression;
+import org.apache.iotdb.db.queryengine.plan.relational.sql.ast.With;
+import org.apache.iotdb.db.queryengine.plan.relational.sql.ast.WithQuery;
+import org.apache.iotdb.db.queryengine.plan.relational.sql.parser.SqlParser;
+import org.apache.iotdb.rpc.TSStatusCode;
+
+import org.apache.tsfile.block.column.Column;
+import org.apache.tsfile.enums.TSDataType;
+import org.apache.tsfile.read.common.block.TsBlock;
+
+import java.util.List;
+import java.util.Optional;
+import java.util.stream.Collectors;
+
+import static com.google.common.base.Preconditions.checkArgument;
+
+public class PredicateWithUncorrelatedScalarSubqueryReconstructor {

Review Comment:
   done



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] With clause enhance [iotdb]

Posted by "shizy818 (via GitHub)" <gi...@apache.org>.
shizy818 commented on code in PR #16353:
URL: https://github.com/apache/iotdb/pull/16353#discussion_r2646395850


##########
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/utils/cte/CteDataReader.java:
##########
@@ -0,0 +1,58 @@
+/*
+ *
+ *  * Licensed to the Apache Software Foundation (ASF) under one
+ *  * or more contributor license agreements.  See the NOTICE file
+ *  * distributed with this work for additional information
+ *  * regarding copyright ownership.  The ASF licenses this file
+ *  * to you under the Apache License, Version 2.0 (the
+ *  * "License"); you may not use this file except in compliance
+ *  * with the License.  You may obtain a copy of the License at
+ *  *
+ *  *     http://www.apache.org/licenses/LICENSE-2.0
+ *  *
+ *  * Unless required by applicable law or agreed to in writing,
+ *  * software distributed under the License is distributed on an
+ *  * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ *  * KIND, either express or implied.  See the License for the
+ *  * specific language governing permissions and limitations
+ *  * under the License.
+ *
+ */
+
+package org.apache.iotdb.db.utils.cte;
+
+import org.apache.iotdb.commons.exception.IoTDBException;
+
+import org.apache.tsfile.read.common.block.TsBlock;
+
+public interface CteDataReader {
+  /**
+   * Check if there is more data in CteDataReader. DiskSpillerReader may run out of current TsBlocks
+   * , then it needs to read from file and cache more data. This method should be called before
+   * next() to ensure that there is data to read.
+   *
+   * @throws IoTDBException the error occurs when reading data from fileChannel
+   */
+  boolean hasNext() throws IoTDBException;
+
+  /**
+   * output the cached data in CteDataReader, it needs to be called after hasNext() returns true.
+   *
+   * @return next TsBlock
+   */
+  TsBlock next() throws IoTDBException;
+
+  /**
+   * Close the CteDataReader and release resources.
+   *
+   * @throws IoTDBException the error occurs when closing fileChannel
+   */
+  void close() throws IoTDBException;
+
+  /**
+   * Get the bytes used by this CteDataReader.
+   *
+   * @return the bytes used by this CteDataReader
+   */
+  long bytesUsed();

Review Comment:
   done



##########
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/utils/cte/CteDataReader.java:
##########
@@ -0,0 +1,58 @@
+/*
+ *
+ *  * Licensed to the Apache Software Foundation (ASF) under one
+ *  * or more contributor license agreements.  See the NOTICE file
+ *  * distributed with this work for additional information
+ *  * regarding copyright ownership.  The ASF licenses this file
+ *  * to you under the Apache License, Version 2.0 (the
+ *  * "License"); you may not use this file except in compliance
+ *  * with the License.  You may obtain a copy of the License at
+ *  *
+ *  *     http://www.apache.org/licenses/LICENSE-2.0
+ *  *
+ *  * Unless required by applicable law or agreed to in writing,
+ *  * software distributed under the License is distributed on an
+ *  * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ *  * KIND, either express or implied.  See the License for the
+ *  * specific language governing permissions and limitations
+ *  * under the License.
+ *
+ */
+
+package org.apache.iotdb.db.utils.cte;
+
+import org.apache.iotdb.commons.exception.IoTDBException;
+
+import org.apache.tsfile.read.common.block.TsBlock;
+
+public interface CteDataReader {

Review Comment:
   done



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] With clause enhance [iotdb]

Posted by "shizy818 (via GitHub)" <gi...@apache.org>.
shizy818 commented on code in PR #16353:
URL: https://github.com/apache/iotdb/pull/16353#discussion_r2646391758


##########
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/queryengine/execution/fragment/FragmentInstanceExecution.java:
##########
@@ -141,17 +140,37 @@ public FragmentInstanceStateMachine getStateMachine() {
     return stateMachine;
   }
 
+  // Check if this fragment instance should be ignored for statistics
+  // (i.e., it contains ExplainAnalyzeOperator only)
+  private boolean shouldIgnoreForStatistics() {
+    if (drivers == null || drivers.isEmpty()) {
+      return false;
+    }
+    // Check if any driver contains ExplainAnalyzeOperator
+    return drivers.stream()
+        .anyMatch(
+            driver ->
+                driver.getDriverContext().getOperatorContexts().stream()
+                    .anyMatch(
+                        operatorContext ->
+                            ExplainAnalyzeOperator.class
+                                .getSimpleName()
+                                .equals(operatorContext.getOperatorType())));

Review Comment:
   done



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] With clause enhance [iotdb]

Posted by "shizy818 (via GitHub)" <gi...@apache.org>.
shizy818 commented on code in PR #16353:
URL: https://github.com/apache/iotdb/pull/16353#discussion_r2645099586


##########
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/queryengine/plan/relational/planner/ir/PredicateWithUncorrelatedScalarSubqueryReconstructor.java:
##########
@@ -0,0 +1,247 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iotdb.db.queryengine.plan.relational.planner.ir;
+
+import org.apache.iotdb.commons.exception.IoTDBException;
+import org.apache.iotdb.commons.utils.TestOnly;
+import org.apache.iotdb.db.protocol.session.SessionManager;
+import org.apache.iotdb.db.queryengine.common.MPPQueryContext;
+import org.apache.iotdb.db.queryengine.common.MPPQueryContext.ExplainType;
+import org.apache.iotdb.db.queryengine.common.header.DatasetHeader;
+import org.apache.iotdb.db.queryengine.plan.Coordinator;
+import org.apache.iotdb.db.queryengine.plan.execution.ExecutionResult;
+import org.apache.iotdb.db.queryengine.plan.planner.LocalExecutionPlanner;
+import org.apache.iotdb.db.queryengine.plan.relational.analyzer.Analysis;
+import org.apache.iotdb.db.queryengine.plan.relational.sql.ast.BinaryLiteral;
+import org.apache.iotdb.db.queryengine.plan.relational.sql.ast.BooleanLiteral;
+import org.apache.iotdb.db.queryengine.plan.relational.sql.ast.ComparisonExpression;
+import org.apache.iotdb.db.queryengine.plan.relational.sql.ast.DoubleLiteral;
+import org.apache.iotdb.db.queryengine.plan.relational.sql.ast.Expression;
+import org.apache.iotdb.db.queryengine.plan.relational.sql.ast.FunctionCall;
+import org.apache.iotdb.db.queryengine.plan.relational.sql.ast.Identifier;
+import org.apache.iotdb.db.queryengine.plan.relational.sql.ast.Literal;
+import org.apache.iotdb.db.queryengine.plan.relational.sql.ast.LogicalExpression;
+import org.apache.iotdb.db.queryengine.plan.relational.sql.ast.LongLiteral;
+import org.apache.iotdb.db.queryengine.plan.relational.sql.ast.NotExpression;
+import org.apache.iotdb.db.queryengine.plan.relational.sql.ast.Query;
+import org.apache.iotdb.db.queryengine.plan.relational.sql.ast.StringLiteral;
+import org.apache.iotdb.db.queryengine.plan.relational.sql.ast.SubqueryExpression;
+import org.apache.iotdb.db.queryengine.plan.relational.sql.ast.With;
+import org.apache.iotdb.db.queryengine.plan.relational.sql.ast.WithQuery;
+import org.apache.iotdb.db.queryengine.plan.relational.sql.parser.SqlParser;
+import org.apache.iotdb.rpc.TSStatusCode;
+
+import org.apache.tsfile.block.column.Column;
+import org.apache.tsfile.enums.TSDataType;
+import org.apache.tsfile.read.common.block.TsBlock;
+
+import java.util.List;
+import java.util.Optional;
+import java.util.stream.Collectors;
+
+import static com.google.common.base.Preconditions.checkArgument;
+
+public class PredicateWithUncorrelatedScalarSubqueryReconstructor {

Review Comment:
   It's for UT code coverage. With Singleton, I can mock at lower level. Then UncorrelatedSubqueryTest can test majority of uncorrelated fold code.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] With clause enhance [iotdb]

Posted by "shizy818 (via GitHub)" <gi...@apache.org>.
shizy818 commented on code in PR #16353:
URL: https://github.com/apache/iotdb/pull/16353#discussion_r2645237016


##########
iotdb-core/node-commons/src/assembly/resources/conf/iotdb-system.properties.template:
##########
@@ -1123,6 +1123,18 @@ batch_size=100000
 # Datatype: long
 sort_buffer_size_in_bytes=0
 
+# The buffer size for CTE materialization. If cte_buffer_size_in_bytes <= 0, a default value of 128 KB is used; otherwise the specified value
+# will be used.
+# effectiveMode: hot_reload
+# Datatype: long
+cte_buffer_size_in_bytes=0

Review Comment:
   done



##########
iotdb-core/node-commons/src/assembly/resources/conf/iotdb-system.properties.template:
##########
@@ -1123,6 +1123,18 @@ batch_size=100000
 # Datatype: long
 sort_buffer_size_in_bytes=0
 
+# The buffer size for CTE materialization. If cte_buffer_size_in_bytes <= 0, a default value of 128 KB is used; otherwise the specified value
+# will be used.
+# effectiveMode: hot_reload
+# Datatype: long
+cte_buffer_size_in_bytes=0
+
+
+# Max rows for CTE materialization
+# effectiveMode: hot_reload
+# Datatype: int

Review Comment:
   done



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] With clause enhance [iotdb]

Posted by "shizy818 (via GitHub)" <gi...@apache.org>.
shizy818 commented on code in PR #16353:
URL: https://github.com/apache/iotdb/pull/16353#discussion_r2646442981


##########
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/queryengine/plan/relational/analyzer/StatementAnalyzer.java:
##########
@@ -3581,8 +3630,15 @@ protected Scope visitJoin(Join node, Optional<Scope> scope) {
 
       joinConditionCheck(criteria);
 
-      Scope left = process(node.getLeft(), scope);
-      Scope right = process(node.getRight(), scope);
+      Optional<Scope> leftScope = scope.map(Scope::copy);
+      Scope left = process(node.getLeft(), leftScope);
+      Optional<Scope> rightScope = scope.map(Scope::copy);
+      Scope right = process(node.getRight(), rightScope);
+
+      if (scope.isPresent()) {
+        leftScope.ifPresent(l -> scope.get().addTables(l.getTables()));
+        rightScope.ifPresent(l -> scope.get().addTables(l.getTables()));
+      }

Review Comment:
   done



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] With clause enhance [iotdb]

Posted by "shizy818 (via GitHub)" <gi...@apache.org>.
shizy818 commented on code in PR #16353:
URL: https://github.com/apache/iotdb/pull/16353#discussion_r2645443187


##########
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/queryengine/plan/relational/analyzer/Analysis.java:
##########
@@ -123,6 +124,8 @@ public class Analysis implements IAnalysis {
 
   private final Map<NodeRef<Table>, Query> namedQueries = new LinkedHashMap<>();
 
+  private With with;

Review Comment:
   Done



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] With clause enhance [iotdb]

Posted by "shizy818 (via GitHub)" <gi...@apache.org>.
shizy818 commented on code in PR #16353:
URL: https://github.com/apache/iotdb/pull/16353#discussion_r2645071740


##########
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/queryengine/execution/operator/source/relational/CteScanOperator.java:
##########
@@ -0,0 +1,134 @@
+/*
+ *
+ *  * Licensed to the Apache Software Foundation (ASF) under one
+ *  * or more contributor license agreements.  See the NOTICE file
+ *  * distributed with this work for additional information
+ *  * regarding copyright ownership.  The ASF licenses this file
+ *  * to you under the Apache License, Version 2.0 (the
+ *  * "License"); you may not use this file except in compliance
+ *  * with the License.  You may obtain a copy of the License at
+ *  *
+ *  *     http://www.apache.org/licenses/LICENSE-2.0
+ *  *
+ *  * Unless required by applicable law or agreed to in writing,
+ *  * software distributed under the License is distributed on an
+ *  * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ *  * KIND, either express or implied.  See the License for the
+ *  * specific language governing permissions and limitations
+ *  * under the License.
+ *
+ */
+
+package org.apache.iotdb.db.queryengine.execution.operator.source.relational;
+
+import org.apache.iotdb.commons.utils.TestOnly;
+import org.apache.iotdb.db.queryengine.execution.MemoryEstimationHelper;
+import org.apache.iotdb.db.queryengine.execution.operator.OperatorContext;
+import org.apache.iotdb.db.queryengine.execution.operator.source.SourceOperator;
+import org.apache.iotdb.db.queryengine.plan.planner.plan.node.PlanNodeId;
+import org.apache.iotdb.db.utils.cte.CteDataReader;
+import org.apache.iotdb.db.utils.cte.CteDataStore;
+import org.apache.iotdb.db.utils.cte.MemoryReader;
+
+import org.apache.tsfile.read.common.block.TsBlock;
+import org.apache.tsfile.utils.RamUsageEstimator;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+public class CteScanOperator implements SourceOperator {
+  private static final Logger LOGGER = LoggerFactory.getLogger(CteScanOperator.class);
+  private static final long INSTANCE_SIZE =
+      RamUsageEstimator.shallowSizeOfInstance(CteScanOperator.class);
+
+  private final OperatorContext operatorContext;
+  private final PlanNodeId sourceId;
+
+  private final CteDataStore dataStore;
+  private final CteDataReader dataReader;
+  private final int dataStoreRefCount;
+
+  public CteScanOperator(
+      OperatorContext operatorContext, PlanNodeId sourceId, CteDataStore dataStore) {
+    this.operatorContext = operatorContext;
+    this.sourceId = sourceId;
+    this.dataStore = dataStore;
+    this.dataReader = new MemoryReader(dataStore.getCachedData());
+    this.dataStoreRefCount = dataStore.increaseRefCount();
+  }
+
+  @Override
+  public TsBlock next() throws Exception {
+    if (dataReader == null) {
+      return null;
+    }
+    return dataReader.next();
+  }
+
+  @Override
+  public boolean hasNext() throws Exception {
+    if (dataReader == null) {
+      return false;
+    }
+    return dataReader.hasNext();
+  }
+
+  @Override
+  public void close() throws Exception {
+    try {
+      if (dataReader != null) {
+        dataReader.close();
+      }
+    } catch (Exception e) {
+      LOGGER.error("Fail to close fileChannel", e);
+    }
+  }
+
+  @Override
+  public boolean isFinished() throws Exception {
+    return !hasNextWithTimer();
+  }
+
+  @Override
+  public long calculateMaxPeekMemory() {
+    return calculateRetainedSizeAfterCallingNext() + calculateMaxReturnSize();
+  }
+
+  @Override
+  public long calculateMaxReturnSize() {
+    // The returned object is a reference to TsBlock in CteDataReader
+    return RamUsageEstimator.NUM_BYTES_OBJECT_REF;
+  }
+
+  @Override
+  public long calculateRetainedSizeAfterCallingNext() {
+    return 0L;
+  }
+
+  @Override
+  public long ramBytesUsed() {
+    long bytes =
+        INSTANCE_SIZE
+            + MemoryEstimationHelper.getEstimatedSizeOfAccountableObject(operatorContext)
+            + dataReader.bytesUsed();
+    if (dataStoreRefCount == 1) {
+      bytes += dataStore.getCachedBytes();
+    }

Review Comment:
   > I think it's better to do memory alloc and free in CteDataReader.close() and init(). first init() call to alloc from LocalExecutionPlanner.reserveFromFreeMemoryForOperators and last free from LocalExecutionPlanner.releaseToFreeMemoryForOperators
   
   OK. I'll change this part



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] With clause enhance [iotdb]

Posted by "shizy818 (via GitHub)" <gi...@apache.org>.
shizy818 commented on code in PR #16353:
URL: https://github.com/apache/iotdb/pull/16353#discussion_r2645116827


##########
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/queryengine/plan/relational/planner/CteMaterializer.java:
##########
@@ -0,0 +1,354 @@
+/*
+ *
+ *  * Licensed to the Apache Software Foundation (ASF) under one
+ *  * or more contributor license agreements.  See the NOTICE file
+ *  * distributed with this work for additional information
+ *  * regarding copyright ownership.  The ASF licenses this file
+ *  * to you under the Apache License, Version 2.0 (the
+ *  * "License"); you may not use this file except in compliance
+ *  * with the License.  You may obtain a copy of the License at
+ *  *
+ *  *     http://www.apache.org/licenses/LICENSE-2.0
+ *  *
+ *  * Unless required by applicable law or agreed to in writing,
+ *  * software distributed under the License is distributed on an
+ *  * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ *  * KIND, either express or implied.  See the License for the
+ *  * specific language governing permissions and limitations
+ *  * under the License.
+ *
+ */
+
+package org.apache.iotdb.db.queryengine.plan.relational.planner;
+
+import org.apache.iotdb.common.rpc.thrift.TEndPoint;
+import org.apache.iotdb.commons.client.IClientManager;
+import org.apache.iotdb.commons.client.sync.SyncDataNodeInternalServiceClient;
+import org.apache.iotdb.commons.exception.IoTDBException;
+import org.apache.iotdb.commons.exception.IoTDBRuntimeException;
+import org.apache.iotdb.commons.schema.table.column.TsTableColumnCategory;
+import org.apache.iotdb.db.exception.mpp.FragmentInstanceFetchException;
+import org.apache.iotdb.db.protocol.session.SessionManager;
+import org.apache.iotdb.db.queryengine.common.FragmentInstanceId;
+import org.apache.iotdb.db.queryengine.common.MPPQueryContext;
+import org.apache.iotdb.db.queryengine.common.header.DatasetHeader;
+import org.apache.iotdb.db.queryengine.plan.Coordinator;
+import org.apache.iotdb.db.queryengine.plan.execution.ExecutionResult;
+import org.apache.iotdb.db.queryengine.plan.execution.QueryExecution;
+import org.apache.iotdb.db.queryengine.plan.planner.LocalExecutionPlanner;
+import org.apache.iotdb.db.queryengine.plan.planner.plan.DistributedQueryPlan;
+import org.apache.iotdb.db.queryengine.plan.planner.plan.FragmentInstance;
+import org.apache.iotdb.db.queryengine.plan.relational.analyzer.Analysis;
+import org.apache.iotdb.db.queryengine.plan.relational.metadata.ColumnSchema;
+import org.apache.iotdb.db.queryengine.plan.relational.metadata.TableSchema;
+import org.apache.iotdb.db.queryengine.plan.relational.sql.ast.Identifier;
+import org.apache.iotdb.db.queryengine.plan.relational.sql.ast.Query;
+import org.apache.iotdb.db.queryengine.plan.relational.sql.ast.Table;
+import org.apache.iotdb.db.queryengine.plan.relational.sql.ast.With;
+import org.apache.iotdb.db.queryengine.plan.relational.sql.ast.WithQuery;
+import org.apache.iotdb.db.queryengine.plan.relational.sql.parser.SqlParser;
+import org.apache.iotdb.db.queryengine.statistics.FragmentInstanceStatisticsDrawer;
+import org.apache.iotdb.db.queryengine.statistics.QueryStatisticsFetcher;
+import org.apache.iotdb.db.queryengine.statistics.StatisticLine;
+import org.apache.iotdb.db.utils.cte.CteDataStore;
+import org.apache.iotdb.mpp.rpc.thrift.TFetchFragmentInstanceStatisticsResp;
+import org.apache.iotdb.rpc.TSStatusCode;
+
+import com.google.common.collect.ImmutableList;
+import org.apache.tsfile.enums.TSDataType;
+import org.apache.tsfile.read.common.block.TsBlock;
+import org.apache.tsfile.read.common.type.TypeFactory;
+import org.apache.tsfile.utils.Pair;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.LinkedHashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.stream.Collectors;
+import java.util.stream.IntStream;
+
+public class CteMaterializer {
+  private static final Logger LOGGER = LoggerFactory.getLogger(CteMaterializer.class);
+
+  private static final Coordinator coordinator = Coordinator.getInstance();
+  private static final SessionManager sessionManager = SessionManager.getInstance();
+
+  public void materializeCTE(Analysis analysis, MPPQueryContext context) {
+    analysis
+        .getNamedQueries()
+        .forEach(
+            (tableRef, query) -> {
+              Table table = tableRef.getNode();
+              if (query.isMaterialized()) {
+                if (!query.isDone()) {

Review Comment:
   I thought even if it failed last time, maybe it's worthy try another time in case it's network problem.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] With clause enhance [iotdb]

Posted by "shizy818 (via GitHub)" <gi...@apache.org>.
shizy818 commented on PR #16353:
URL: https://github.com/apache/iotdb/pull/16353#issuecomment-3640310289

   Sonar Coverage Report
   
   Master branch: test coverage is 36.0%
   <img width="2280" height="164" alt="图片" src="https://github.com/user-attachments/assets/0a738dff-3b29-4758-85a2-32e8100e027a" />
   
   with_clause_enhance branch: test coverage for new code is 80%
   <img width="2280" height="164" alt="图片" src="https://github.com/user-attachments/assets/0a738dff-3b29-4758-85a2-32e8100e027a" />
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] With clause enhance [iotdb]

Posted by "shizy818 (via GitHub)" <gi...@apache.org>.
shizy818 commented on code in PR #16353:
URL: https://github.com/apache/iotdb/pull/16353#discussion_r2645458649


##########
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/queryengine/plan/relational/analyzer/Scope.java:
##########
@@ -50,6 +54,7 @@ public class Scope {
   private final RelationId relationId;
   private final RelationType relation;
   private final Map<String, WithQuery> namedQueries;
+  private final List<Identifier> tables;

Review Comment:
   done



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] With clause enhance [iotdb]

Posted by "JackieTien97 (via GitHub)" <gi...@apache.org>.
JackieTien97 commented on code in PR #16353:
URL: https://github.com/apache/iotdb/pull/16353#discussion_r2642961795


##########
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/queryengine/execution/fragment/FragmentInstanceExecution.java:
##########
@@ -141,17 +140,37 @@ public FragmentInstanceStateMachine getStateMachine() {
     return stateMachine;
   }
 
+  // Check if this fragment instance should be ignored for statistics
+  // (i.e., it contains ExplainAnalyzeOperator only)
+  private boolean shouldIgnoreForStatistics() {
+    if (drivers == null || drivers.isEmpty()) {
+      return false;
+    }
+    // Check if any driver contains ExplainAnalyzeOperator
+    return drivers.stream()
+        .anyMatch(
+            driver ->
+                driver.getDriverContext().getOperatorContexts().stream()
+                    .anyMatch(
+                        operatorContext ->
+                            ExplainAnalyzeOperator.class
+                                .getSimpleName()
+                                .equals(operatorContext.getOperatorType())));

Review Comment:
   it should be inited as a field while generating this FragmentInstanceExecution in LocalExecutionPlanner instead of doing this judgement each time.



##########
iotdb-core/node-commons/src/assembly/resources/conf/iotdb-system.properties.template:
##########
@@ -1123,6 +1123,18 @@ batch_size=100000
 # Datatype: long
 sort_buffer_size_in_bytes=0
 
+# The buffer size for CTE materialization. If cte_buffer_size_in_bytes <= 0, a default value of 128 KB is used; otherwise the specified value
+# will be used.
+# effectiveMode: hot_reload
+# Datatype: long
+cte_buffer_size_in_bytes=0

Review Comment:
   ```suggestion
   # Privilege: SYSTEM
   # unit: byte
   cte_buffer_size_in_bytes=131072
   ```



##########
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/conf/IoTDBDescriptor.java:
##########
@@ -1057,6 +1057,24 @@ public void loadProperties(TrimProperties properties) throws BadNodeUrlException
     // The buffer for sort operator to calculate
     loadFixedSizeLimitForQuery(properties, "sort_buffer_size_in_bytes", conf::setSortBufferSize);
 
+    // The buffer for cte materialization.
+    long cteBufferSizeInBytes =
+        Long.parseLong(
+            properties.getProperty(
+                "cte_buffer_size_in_bytes", Long.toString(conf.getCteBufferSize())));
+    if (cteBufferSizeInBytes > 0) {
+      conf.setCteBufferSize(cteBufferSizeInBytes);
+    }
+
+    // max number of rows for cte materialization
+    int maxRowsInCteBuffer =
+        Integer.parseInt(
+            properties.getProperty(
+                "max_rows_in_cte_buffer", Integer.toString(conf.getMaxRowsInCteBuffer())));
+    if (maxRowsInCteBuffer > 0) {
+      conf.setMaxRowsInCteBuffer(maxRowsInCteBuffer);
+    }
+

Review Comment:
   also add these in `loadHotModifiedProps` function, these two configurations should be hot loaded.



##########
iotdb-core/node-commons/src/assembly/resources/conf/iotdb-system.properties.template:
##########
@@ -1123,6 +1123,18 @@ batch_size=100000
 # Datatype: long
 sort_buffer_size_in_bytes=0
 
+# The buffer size for CTE materialization. If cte_buffer_size_in_bytes <= 0, a default value of 128 KB is used; otherwise the specified value
+# will be used.
+# effectiveMode: hot_reload
+# Datatype: long
+cte_buffer_size_in_bytes=0
+
+
+# Max rows for CTE materialization
+# effectiveMode: hot_reload
+# Datatype: int

Review Comment:
   ```suggestion
   # Datatype: int
   # Privilege: SYSTEM
   ```



##########
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/queryengine/execution/operator/source/relational/CteScanOperator.java:
##########
@@ -0,0 +1,134 @@
+/*
+ *
+ *  * Licensed to the Apache Software Foundation (ASF) under one
+ *  * or more contributor license agreements.  See the NOTICE file
+ *  * distributed with this work for additional information
+ *  * regarding copyright ownership.  The ASF licenses this file
+ *  * to you under the Apache License, Version 2.0 (the
+ *  * "License"); you may not use this file except in compliance
+ *  * with the License.  You may obtain a copy of the License at
+ *  *
+ *  *     http://www.apache.org/licenses/LICENSE-2.0
+ *  *
+ *  * Unless required by applicable law or agreed to in writing,
+ *  * software distributed under the License is distributed on an
+ *  * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ *  * KIND, either express or implied.  See the License for the
+ *  * specific language governing permissions and limitations
+ *  * under the License.
+ *
+ */
+
+package org.apache.iotdb.db.queryengine.execution.operator.source.relational;
+
+import org.apache.iotdb.commons.utils.TestOnly;
+import org.apache.iotdb.db.queryengine.execution.MemoryEstimationHelper;
+import org.apache.iotdb.db.queryengine.execution.operator.OperatorContext;
+import org.apache.iotdb.db.queryengine.execution.operator.source.SourceOperator;
+import org.apache.iotdb.db.queryengine.plan.planner.plan.node.PlanNodeId;
+import org.apache.iotdb.db.utils.cte.CteDataReader;
+import org.apache.iotdb.db.utils.cte.CteDataStore;
+import org.apache.iotdb.db.utils.cte.MemoryReader;
+
+import org.apache.tsfile.read.common.block.TsBlock;
+import org.apache.tsfile.utils.RamUsageEstimator;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+public class CteScanOperator implements SourceOperator {
+  private static final Logger LOGGER = LoggerFactory.getLogger(CteScanOperator.class);
+  private static final long INSTANCE_SIZE =
+      RamUsageEstimator.shallowSizeOfInstance(CteScanOperator.class);
+
+  private final OperatorContext operatorContext;
+  private final PlanNodeId sourceId;
+
+  private final CteDataStore dataStore;
+  private final CteDataReader dataReader;
+  private final int dataStoreRefCount;
+
+  public CteScanOperator(
+      OperatorContext operatorContext, PlanNodeId sourceId, CteDataStore dataStore) {
+    this.operatorContext = operatorContext;
+    this.sourceId = sourceId;
+    this.dataStore = dataStore;
+    this.dataReader = new MemoryReader(dataStore.getCachedData());
+    this.dataStoreRefCount = dataStore.increaseRefCount();
+  }
+
+  @Override
+  public TsBlock next() throws Exception {
+    if (dataReader == null) {
+      return null;
+    }
+    return dataReader.next();
+  }
+
+  @Override
+  public boolean hasNext() throws Exception {
+    if (dataReader == null) {
+      return false;
+    }
+    return dataReader.hasNext();
+  }
+
+  @Override
+  public void close() throws Exception {
+    try {
+      if (dataReader != null) {
+        dataReader.close();
+      }
+    } catch (Exception e) {
+      LOGGER.error("Fail to close fileChannel", e);

Review Comment:
   ```suggestion
         LOGGER.error("Fail to close CteDataReader", e);
   ```



##########
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/queryengine/execution/operator/source/relational/CteScanOperator.java:
##########
@@ -0,0 +1,134 @@
+/*
+ *
+ *  * Licensed to the Apache Software Foundation (ASF) under one
+ *  * or more contributor license agreements.  See the NOTICE file
+ *  * distributed with this work for additional information
+ *  * regarding copyright ownership.  The ASF licenses this file
+ *  * to you under the Apache License, Version 2.0 (the
+ *  * "License"); you may not use this file except in compliance
+ *  * with the License.  You may obtain a copy of the License at
+ *  *
+ *  *     http://www.apache.org/licenses/LICENSE-2.0
+ *  *
+ *  * Unless required by applicable law or agreed to in writing,
+ *  * software distributed under the License is distributed on an
+ *  * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ *  * KIND, either express or implied.  See the License for the
+ *  * specific language governing permissions and limitations
+ *  * under the License.
+ *
+ */
+
+package org.apache.iotdb.db.queryengine.execution.operator.source.relational;
+
+import org.apache.iotdb.commons.utils.TestOnly;
+import org.apache.iotdb.db.queryengine.execution.MemoryEstimationHelper;
+import org.apache.iotdb.db.queryengine.execution.operator.OperatorContext;
+import org.apache.iotdb.db.queryengine.execution.operator.source.SourceOperator;
+import org.apache.iotdb.db.queryengine.plan.planner.plan.node.PlanNodeId;
+import org.apache.iotdb.db.utils.cte.CteDataReader;
+import org.apache.iotdb.db.utils.cte.CteDataStore;
+import org.apache.iotdb.db.utils.cte.MemoryReader;
+
+import org.apache.tsfile.read.common.block.TsBlock;
+import org.apache.tsfile.utils.RamUsageEstimator;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+public class CteScanOperator implements SourceOperator {
+  private static final Logger LOGGER = LoggerFactory.getLogger(CteScanOperator.class);
+  private static final long INSTANCE_SIZE =
+      RamUsageEstimator.shallowSizeOfInstance(CteScanOperator.class);
+
+  private final OperatorContext operatorContext;
+  private final PlanNodeId sourceId;
+
+  private final CteDataStore dataStore;
+  private final CteDataReader dataReader;
+  private final int dataStoreRefCount;
+
+  public CteScanOperator(
+      OperatorContext operatorContext, PlanNodeId sourceId, CteDataStore dataStore) {
+    this.operatorContext = operatorContext;
+    this.sourceId = sourceId;
+    this.dataStore = dataStore;
+    this.dataReader = new MemoryReader(dataStore.getCachedData());
+    this.dataStoreRefCount = dataStore.increaseRefCount();
+  }
+
+  @Override
+  public TsBlock next() throws Exception {
+    if (dataReader == null) {
+      return null;
+    }
+    return dataReader.next();
+  }
+
+  @Override
+  public boolean hasNext() throws Exception {
+    if (dataReader == null) {
+      return false;
+    }
+    return dataReader.hasNext();
+  }
+
+  @Override
+  public void close() throws Exception {
+    try {
+      if (dataReader != null) {
+        dataReader.close();
+      }
+    } catch (Exception e) {
+      LOGGER.error("Fail to close fileChannel", e);
+    }
+  }
+
+  @Override
+  public boolean isFinished() throws Exception {
+    return !hasNextWithTimer();
+  }
+
+  @Override
+  public long calculateMaxPeekMemory() {
+    return calculateRetainedSizeAfterCallingNext() + calculateMaxReturnSize();
+  }
+
+  @Override
+  public long calculateMaxReturnSize() {
+    // The returned object is a reference to TsBlock in CteDataReader
+    return RamUsageEstimator.NUM_BYTES_OBJECT_REF;
+  }
+
+  @Override
+  public long calculateRetainedSizeAfterCallingNext() {
+    return 0L;
+  }
+
+  @Override
+  public long ramBytesUsed() {
+    long bytes =
+        INSTANCE_SIZE
+            + MemoryEstimationHelper.getEstimatedSizeOfAccountableObject(operatorContext)
+            + dataReader.bytesUsed();
+    if (dataStoreRefCount == 1) {
+      bytes += dataStore.getCachedBytes();
+    }

Review Comment:
   what if the FI which this CteScanOperator belongs to has finished, but other CteScanOperator with the same dataStore hasn't finished?



##########
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/queryengine/plan/relational/analyzer/Scope.java:
##########
@@ -50,6 +54,7 @@ public class Scope {
   private final RelationId relationId;
   private final RelationType relation;
   private final Map<String, WithQuery> namedQueries;
+  private final List<Identifier> tables;

Review Comment:
   what's this field used for? we need some comments to explain.



##########
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/queryengine/plan/relational/analyzer/Analysis.java:
##########
@@ -123,6 +124,8 @@ public class Analysis implements IAnalysis {
 
   private final Map<NodeRef<Table>, Query> namedQueries = new LinkedHashMap<>();
 
+  private With with;

Review Comment:
   add some comments about this important field, like what's this field used for? in which case



##########
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/queryengine/execution/operator/source/relational/CteScanOperator.java:
##########
@@ -0,0 +1,134 @@
+/*
+ *
+ *  * Licensed to the Apache Software Foundation (ASF) under one
+ *  * or more contributor license agreements.  See the NOTICE file
+ *  * distributed with this work for additional information
+ *  * regarding copyright ownership.  The ASF licenses this file
+ *  * to you under the Apache License, Version 2.0 (the
+ *  * "License"); you may not use this file except in compliance
+ *  * with the License.  You may obtain a copy of the License at
+ *  *
+ *  *     http://www.apache.org/licenses/LICENSE-2.0
+ *  *
+ *  * Unless required by applicable law or agreed to in writing,
+ *  * software distributed under the License is distributed on an
+ *  * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ *  * KIND, either express or implied.  See the License for the
+ *  * specific language governing permissions and limitations
+ *  * under the License.
+ *
+ */
+
+package org.apache.iotdb.db.queryengine.execution.operator.source.relational;
+
+import org.apache.iotdb.commons.utils.TestOnly;
+import org.apache.iotdb.db.queryengine.execution.MemoryEstimationHelper;
+import org.apache.iotdb.db.queryengine.execution.operator.OperatorContext;
+import org.apache.iotdb.db.queryengine.execution.operator.source.SourceOperator;
+import org.apache.iotdb.db.queryengine.plan.planner.plan.node.PlanNodeId;
+import org.apache.iotdb.db.utils.cte.CteDataReader;
+import org.apache.iotdb.db.utils.cte.CteDataStore;
+import org.apache.iotdb.db.utils.cte.MemoryReader;
+
+import org.apache.tsfile.read.common.block.TsBlock;
+import org.apache.tsfile.utils.RamUsageEstimator;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+public class CteScanOperator implements SourceOperator {
+  private static final Logger LOGGER = LoggerFactory.getLogger(CteScanOperator.class);
+  private static final long INSTANCE_SIZE =
+      RamUsageEstimator.shallowSizeOfInstance(CteScanOperator.class);
+
+  private final OperatorContext operatorContext;
+  private final PlanNodeId sourceId;
+
+  private final CteDataStore dataStore;
+  private final CteDataReader dataReader;
+  private final int dataStoreRefCount;
+
+  public CteScanOperator(
+      OperatorContext operatorContext, PlanNodeId sourceId, CteDataStore dataStore) {
+    this.operatorContext = operatorContext;
+    this.sourceId = sourceId;
+    this.dataStore = dataStore;
+    this.dataReader = new MemoryReader(dataStore.getCachedData());
+    this.dataStoreRefCount = dataStore.increaseRefCount();
+  }
+
+  @Override
+  public TsBlock next() throws Exception {
+    if (dataReader == null) {
+      return null;
+    }
+    return dataReader.next();
+  }
+
+  @Override
+  public boolean hasNext() throws Exception {
+    if (dataReader == null) {
+      return false;
+    }
+    return dataReader.hasNext();
+  }
+
+  @Override
+  public void close() throws Exception {
+    try {
+      if (dataReader != null) {
+        dataReader.close();
+      }
+    } catch (Exception e) {
+      LOGGER.error("Fail to close fileChannel", e);
+    }
+  }
+
+  @Override
+  public boolean isFinished() throws Exception {
+    return !hasNextWithTimer();
+  }
+
+  @Override
+  public long calculateMaxPeekMemory() {
+    return calculateRetainedSizeAfterCallingNext() + calculateMaxReturnSize();

Review Comment:
   same as `calculateMaxReturnSize`



##########
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/queryengine/plan/relational/planner/QueryPlanner.java:
##########
@@ -763,9 +764,13 @@ private PlanBuilder filter(PlanBuilder subPlan, Expression predicate, Node node)
     }
 
     subPlan = subqueryPlanner.handleSubqueries(subPlan, predicate, analysis.getSubqueries(node));
-    return subPlan.withNewRoot(
-        new FilterNode(
-            queryIdAllocator.genPlanNodeId(), subPlan.getRoot(), subPlan.rewrite(predicate)));
+    PlanBuilder planBuilder =
+        subPlan.withNewRoot(
+            new FilterNode(
+                queryIdAllocator.genPlanNodeId(), subPlan.getRoot(), subPlan.rewrite(predicate)));
+    PredicateWithUncorrelatedScalarSubqueryReconstructor.getInstance()
+        .clearShadowExpression(predicate);

Review Comment:
   We shouldn't clear shadow expression here, the predicate expression is still being used afterwards. We can only clear them while front end finished.



##########
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/queryengine/plan/relational/analyzer/StatementAnalyzer.java:
##########
@@ -3581,8 +3630,15 @@ protected Scope visitJoin(Join node, Optional<Scope> scope) {
 
       joinConditionCheck(criteria);
 
-      Scope left = process(node.getLeft(), scope);
-      Scope right = process(node.getRight(), scope);
+      Optional<Scope> leftScope = scope.map(Scope::copy);
+      Scope left = process(node.getLeft(), leftScope);
+      Optional<Scope> rightScope = scope.map(Scope::copy);
+      Scope right = process(node.getRight(), rightScope);
+
+      if (scope.isPresent()) {
+        leftScope.ifPresent(l -> scope.get().addTables(l.getTables()));
+        rightScope.ifPresent(l -> scope.get().addTables(l.getTables()));
+      }

Review Comment:
   why we need to do these changes? It seems that trino just directly use the scope.



##########
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/utils/cte/MemoryReader.java:
##########
@@ -0,0 +1,64 @@
+/*
+ *
+ *  * Licensed to the Apache Software Foundation (ASF) under one
+ *  * or more contributor license agreements.  See the NOTICE file
+ *  * distributed with this work for additional information
+ *  * regarding copyright ownership.  The ASF licenses this file
+ *  * to you under the Apache License, Version 2.0 (the
+ *  * "License"); you may not use this file except in compliance
+ *  * with the License.  You may obtain a copy of the License at
+ *  *
+ *  *     http://www.apache.org/licenses/LICENSE-2.0
+ *  *
+ *  * Unless required by applicable law or agreed to in writing,
+ *  * software distributed under the License is distributed on an
+ *  * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ *  * KIND, either express or implied.  See the License for the
+ *  * specific language governing permissions and limitations
+ *  * under the License.
+ *
+ */
+
+package org.apache.iotdb.db.utils.cte;
+
+import org.apache.iotdb.commons.exception.IoTDBException;
+
+import org.apache.tsfile.read.common.block.TsBlock;
+import org.apache.tsfile.utils.RamUsageEstimator;
+
+import java.util.List;
+
+public class MemoryReader implements CteDataReader {
+  private static final long INSTANCE_SIZE =
+      RamUsageEstimator.shallowSizeOfInstance(MemoryReader.class);
+
+  // all the data in MemoryReader lies in memory
+  private final List<TsBlock> cachedData;
+  private int tsBlockIndex;
+
+  public MemoryReader(List<TsBlock> cachedTsBlock) {
+    this.cachedData = cachedTsBlock;
+    this.tsBlockIndex = 0;
+  }
+
+  @Override
+  public boolean hasNext() throws IoTDBException {
+    return cachedData != null && tsBlockIndex < cachedData.size();
+  }
+
+  @Override
+  public TsBlock next() throws IoTDBException {
+    if (cachedData == null || tsBlockIndex >= cachedData.size()) {
+      return null;
+    }
+    return cachedData.get(tsBlockIndex++);
+  }
+
+  @Override
+  public void close() throws IoTDBException {}
+
+  @Override
+  public long bytesUsed() {
+    return INSTANCE_SIZE;

Review Comment:
   handle the memory footprint right.



##########
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/queryengine/plan/relational/planner/ir/PredicateWithUncorrelatedScalarSubqueryReconstructor.java:
##########
@@ -0,0 +1,247 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iotdb.db.queryengine.plan.relational.planner.ir;
+
+import org.apache.iotdb.commons.exception.IoTDBException;
+import org.apache.iotdb.commons.utils.TestOnly;
+import org.apache.iotdb.db.protocol.session.SessionManager;
+import org.apache.iotdb.db.queryengine.common.MPPQueryContext;
+import org.apache.iotdb.db.queryengine.common.MPPQueryContext.ExplainType;
+import org.apache.iotdb.db.queryengine.common.header.DatasetHeader;
+import org.apache.iotdb.db.queryengine.plan.Coordinator;
+import org.apache.iotdb.db.queryengine.plan.execution.ExecutionResult;
+import org.apache.iotdb.db.queryengine.plan.planner.LocalExecutionPlanner;
+import org.apache.iotdb.db.queryengine.plan.relational.analyzer.Analysis;
+import org.apache.iotdb.db.queryengine.plan.relational.sql.ast.BinaryLiteral;
+import org.apache.iotdb.db.queryengine.plan.relational.sql.ast.BooleanLiteral;
+import org.apache.iotdb.db.queryengine.plan.relational.sql.ast.ComparisonExpression;
+import org.apache.iotdb.db.queryengine.plan.relational.sql.ast.DoubleLiteral;
+import org.apache.iotdb.db.queryengine.plan.relational.sql.ast.Expression;
+import org.apache.iotdb.db.queryengine.plan.relational.sql.ast.FunctionCall;
+import org.apache.iotdb.db.queryengine.plan.relational.sql.ast.Identifier;
+import org.apache.iotdb.db.queryengine.plan.relational.sql.ast.Literal;
+import org.apache.iotdb.db.queryengine.plan.relational.sql.ast.LogicalExpression;
+import org.apache.iotdb.db.queryengine.plan.relational.sql.ast.LongLiteral;
+import org.apache.iotdb.db.queryengine.plan.relational.sql.ast.NotExpression;
+import org.apache.iotdb.db.queryengine.plan.relational.sql.ast.Query;
+import org.apache.iotdb.db.queryengine.plan.relational.sql.ast.StringLiteral;
+import org.apache.iotdb.db.queryengine.plan.relational.sql.ast.SubqueryExpression;
+import org.apache.iotdb.db.queryengine.plan.relational.sql.ast.With;
+import org.apache.iotdb.db.queryengine.plan.relational.sql.ast.WithQuery;
+import org.apache.iotdb.db.queryengine.plan.relational.sql.parser.SqlParser;
+import org.apache.iotdb.rpc.TSStatusCode;
+
+import org.apache.tsfile.block.column.Column;
+import org.apache.tsfile.enums.TSDataType;
+import org.apache.tsfile.read.common.block.TsBlock;
+
+import java.util.List;
+import java.util.Optional;
+import java.util.stream.Collectors;
+
+import static com.google.common.base.Preconditions.checkArgument;
+
+public class PredicateWithUncorrelatedScalarSubqueryReconstructor {

Review Comment:
   no need to be a singleton? if you just want to add UT, you should pass that mock one from the top level



##########
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/queryengine/common/MPPQueryContext.java:
##########
@@ -103,6 +125,24 @@ public class MPPQueryContext implements IAuditEntity {
 
   private boolean userQuery = false;
 
+  private Map<NodeRef<Table>, Query> cteQueries = new HashMap<>();
+
+  // Stores the EXPLAIN/EXPLAIN ANALYZE results for Common Table Expressions (CTEs)
+  // Key: CTE table reference
+  // Value: Pair containing (max line length of the explain output, list of formatted explain lines)
+  // This ensures consistent formatting between the main query and its CTE sub-queries
+  private final Map<NodeRef<Table>, Pair<Integer, List<String>>> cteExplainResults =
+      new LinkedHashMap<>();
+  // Tracks the materialization time cost (in nanoseconds) for each CTE to help optimize query
+  // planning
+  private final Map<NodeRef<Table>, Long> cteMaterializationCosts = new HashMap<>();
+
+  // Never materialize CTE in a subquery.
+  private boolean subquery = false;

Review Comment:
   not just a subQuery? I think the field only means materialized CTE query or folded uncorrelated scalar subquery?
   
   if there is no better names, I think you should write the meaning more clear in the java docs.



##########
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/queryengine/execution/operator/source/relational/CteScanOperator.java:
##########
@@ -0,0 +1,134 @@
+/*
+ *
+ *  * Licensed to the Apache Software Foundation (ASF) under one
+ *  * or more contributor license agreements.  See the NOTICE file
+ *  * distributed with this work for additional information
+ *  * regarding copyright ownership.  The ASF licenses this file
+ *  * to you under the Apache License, Version 2.0 (the
+ *  * "License"); you may not use this file except in compliance
+ *  * with the License.  You may obtain a copy of the License at
+ *  *
+ *  *     http://www.apache.org/licenses/LICENSE-2.0
+ *  *
+ *  * Unless required by applicable law or agreed to in writing,
+ *  * software distributed under the License is distributed on an
+ *  * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ *  * KIND, either express or implied.  See the License for the
+ *  * specific language governing permissions and limitations
+ *  * under the License.
+ *
+ */
+
+package org.apache.iotdb.db.queryengine.execution.operator.source.relational;
+
+import org.apache.iotdb.commons.utils.TestOnly;
+import org.apache.iotdb.db.queryengine.execution.MemoryEstimationHelper;
+import org.apache.iotdb.db.queryengine.execution.operator.OperatorContext;
+import org.apache.iotdb.db.queryengine.execution.operator.source.SourceOperator;
+import org.apache.iotdb.db.queryengine.plan.planner.plan.node.PlanNodeId;
+import org.apache.iotdb.db.utils.cte.CteDataReader;
+import org.apache.iotdb.db.utils.cte.CteDataStore;
+import org.apache.iotdb.db.utils.cte.MemoryReader;
+
+import org.apache.tsfile.read.common.block.TsBlock;
+import org.apache.tsfile.utils.RamUsageEstimator;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+public class CteScanOperator implements SourceOperator {
+  private static final Logger LOGGER = LoggerFactory.getLogger(CteScanOperator.class);
+  private static final long INSTANCE_SIZE =
+      RamUsageEstimator.shallowSizeOfInstance(CteScanOperator.class);
+
+  private final OperatorContext operatorContext;
+  private final PlanNodeId sourceId;
+
+  private final CteDataStore dataStore;
+  private final CteDataReader dataReader;
+  private final int dataStoreRefCount;
+
+  public CteScanOperator(
+      OperatorContext operatorContext, PlanNodeId sourceId, CteDataStore dataStore) {
+    this.operatorContext = operatorContext;
+    this.sourceId = sourceId;
+    this.dataStore = dataStore;
+    this.dataReader = new MemoryReader(dataStore.getCachedData());
+    this.dataStoreRefCount = dataStore.increaseRefCount();
+  }
+
+  @Override
+  public TsBlock next() throws Exception {
+    if (dataReader == null) {
+      return null;
+    }
+    return dataReader.next();
+  }
+
+  @Override
+  public boolean hasNext() throws Exception {
+    if (dataReader == null) {
+      return false;
+    }
+    return dataReader.hasNext();
+  }
+
+  @Override
+  public void close() throws Exception {
+    try {
+      if (dataReader != null) {
+        dataReader.close();
+      }
+    } catch (Exception e) {
+      LOGGER.error("Fail to close fileChannel", e);
+    }
+  }
+
+  @Override
+  public boolean isFinished() throws Exception {
+    return !hasNextWithTimer();
+  }
+
+  @Override
+  public long calculateMaxPeekMemory() {
+    return calculateRetainedSizeAfterCallingNext() + calculateMaxReturnSize();
+  }
+
+  @Override
+  public long calculateMaxReturnSize() {
+    // The returned object is a reference to TsBlock in CteDataReader
+    return RamUsageEstimator.NUM_BYTES_OBJECT_REF;

Review Comment:
   it should return the actual max return size among all the tsblocks in dataReader



##########
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/queryengine/plan/relational/analyzer/predicate/ConvertPredicateToTimeFilterVisitor.java:
##########
@@ -299,6 +300,13 @@ protected Filter visitBetweenPredicate(BetweenPredicate node, Void context) {
   }
 
   public static long getLongValue(Expression expression) {
-    return ((LongLiteral) expression).getParsedValue();
+    if (expression instanceof LongLiteral) {
+      return ((LongLiteral) expression).getParsedValue();
+    } else if (expression instanceof DoubleLiteral) {
+      return (long) ((DoubleLiteral) expression).getValue();
+    } else {
+      throw new IllegalArgumentException(
+          "Expression should be LongLiteral or DoubleLiteral, but got: " + expression.getClass());
+    }

Review Comment:
   You don't need to fix that, @alpass163 has already fixed that in https://github.com/apache/iotdb/pull/16917



##########
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/queryengine/plan/relational/planner/ir/PredicateWithUncorrelatedScalarSubqueryReconstructor.java:
##########
@@ -0,0 +1,247 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iotdb.db.queryengine.plan.relational.planner.ir;
+
+import org.apache.iotdb.commons.exception.IoTDBException;
+import org.apache.iotdb.commons.utils.TestOnly;
+import org.apache.iotdb.db.protocol.session.SessionManager;
+import org.apache.iotdb.db.queryengine.common.MPPQueryContext;
+import org.apache.iotdb.db.queryengine.common.MPPQueryContext.ExplainType;
+import org.apache.iotdb.db.queryengine.common.header.DatasetHeader;
+import org.apache.iotdb.db.queryengine.plan.Coordinator;
+import org.apache.iotdb.db.queryengine.plan.execution.ExecutionResult;
+import org.apache.iotdb.db.queryengine.plan.planner.LocalExecutionPlanner;
+import org.apache.iotdb.db.queryengine.plan.relational.analyzer.Analysis;
+import org.apache.iotdb.db.queryengine.plan.relational.sql.ast.BinaryLiteral;
+import org.apache.iotdb.db.queryengine.plan.relational.sql.ast.BooleanLiteral;
+import org.apache.iotdb.db.queryengine.plan.relational.sql.ast.ComparisonExpression;
+import org.apache.iotdb.db.queryengine.plan.relational.sql.ast.DoubleLiteral;
+import org.apache.iotdb.db.queryengine.plan.relational.sql.ast.Expression;
+import org.apache.iotdb.db.queryengine.plan.relational.sql.ast.FunctionCall;
+import org.apache.iotdb.db.queryengine.plan.relational.sql.ast.Identifier;
+import org.apache.iotdb.db.queryengine.plan.relational.sql.ast.Literal;
+import org.apache.iotdb.db.queryengine.plan.relational.sql.ast.LogicalExpression;
+import org.apache.iotdb.db.queryengine.plan.relational.sql.ast.LongLiteral;
+import org.apache.iotdb.db.queryengine.plan.relational.sql.ast.NotExpression;
+import org.apache.iotdb.db.queryengine.plan.relational.sql.ast.Query;
+import org.apache.iotdb.db.queryengine.plan.relational.sql.ast.StringLiteral;
+import org.apache.iotdb.db.queryengine.plan.relational.sql.ast.SubqueryExpression;
+import org.apache.iotdb.db.queryengine.plan.relational.sql.ast.With;
+import org.apache.iotdb.db.queryengine.plan.relational.sql.ast.WithQuery;
+import org.apache.iotdb.db.queryengine.plan.relational.sql.parser.SqlParser;
+import org.apache.iotdb.rpc.TSStatusCode;
+
+import org.apache.tsfile.block.column.Column;
+import org.apache.tsfile.enums.TSDataType;
+import org.apache.tsfile.read.common.block.TsBlock;
+
+import java.util.List;
+import java.util.Optional;
+import java.util.stream.Collectors;
+
+import static com.google.common.base.Preconditions.checkArgument;
+
+public class PredicateWithUncorrelatedScalarSubqueryReconstructor {
+
+  private static final SqlParser relationSqlParser = new SqlParser();

Review Comment:
   no need to new one, you can get it from `TableModelPlanner`



##########
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/queryengine/plan/relational/planner/CteMaterializer.java:
##########
@@ -0,0 +1,354 @@
+/*
+ *
+ *  * Licensed to the Apache Software Foundation (ASF) under one
+ *  * or more contributor license agreements.  See the NOTICE file
+ *  * distributed with this work for additional information
+ *  * regarding copyright ownership.  The ASF licenses this file
+ *  * to you under the Apache License, Version 2.0 (the
+ *  * "License"); you may not use this file except in compliance
+ *  * with the License.  You may obtain a copy of the License at
+ *  *
+ *  *     http://www.apache.org/licenses/LICENSE-2.0
+ *  *
+ *  * Unless required by applicable law or agreed to in writing,
+ *  * software distributed under the License is distributed on an
+ *  * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ *  * KIND, either express or implied.  See the License for the
+ *  * specific language governing permissions and limitations
+ *  * under the License.
+ *
+ */
+
+package org.apache.iotdb.db.queryengine.plan.relational.planner;
+
+import org.apache.iotdb.common.rpc.thrift.TEndPoint;
+import org.apache.iotdb.commons.client.IClientManager;
+import org.apache.iotdb.commons.client.sync.SyncDataNodeInternalServiceClient;
+import org.apache.iotdb.commons.exception.IoTDBException;
+import org.apache.iotdb.commons.exception.IoTDBRuntimeException;
+import org.apache.iotdb.commons.schema.table.column.TsTableColumnCategory;
+import org.apache.iotdb.db.exception.mpp.FragmentInstanceFetchException;
+import org.apache.iotdb.db.protocol.session.SessionManager;
+import org.apache.iotdb.db.queryengine.common.FragmentInstanceId;
+import org.apache.iotdb.db.queryengine.common.MPPQueryContext;
+import org.apache.iotdb.db.queryengine.common.header.DatasetHeader;
+import org.apache.iotdb.db.queryengine.plan.Coordinator;
+import org.apache.iotdb.db.queryengine.plan.execution.ExecutionResult;
+import org.apache.iotdb.db.queryengine.plan.execution.QueryExecution;
+import org.apache.iotdb.db.queryengine.plan.planner.LocalExecutionPlanner;
+import org.apache.iotdb.db.queryengine.plan.planner.plan.DistributedQueryPlan;
+import org.apache.iotdb.db.queryengine.plan.planner.plan.FragmentInstance;
+import org.apache.iotdb.db.queryengine.plan.relational.analyzer.Analysis;
+import org.apache.iotdb.db.queryengine.plan.relational.metadata.ColumnSchema;
+import org.apache.iotdb.db.queryengine.plan.relational.metadata.TableSchema;
+import org.apache.iotdb.db.queryengine.plan.relational.sql.ast.Identifier;
+import org.apache.iotdb.db.queryengine.plan.relational.sql.ast.Query;
+import org.apache.iotdb.db.queryengine.plan.relational.sql.ast.Table;
+import org.apache.iotdb.db.queryengine.plan.relational.sql.ast.With;
+import org.apache.iotdb.db.queryengine.plan.relational.sql.ast.WithQuery;
+import org.apache.iotdb.db.queryengine.plan.relational.sql.parser.SqlParser;
+import org.apache.iotdb.db.queryengine.statistics.FragmentInstanceStatisticsDrawer;
+import org.apache.iotdb.db.queryengine.statistics.QueryStatisticsFetcher;
+import org.apache.iotdb.db.queryengine.statistics.StatisticLine;
+import org.apache.iotdb.db.utils.cte.CteDataStore;
+import org.apache.iotdb.mpp.rpc.thrift.TFetchFragmentInstanceStatisticsResp;
+import org.apache.iotdb.rpc.TSStatusCode;
+
+import com.google.common.collect.ImmutableList;
+import org.apache.tsfile.enums.TSDataType;
+import org.apache.tsfile.read.common.block.TsBlock;
+import org.apache.tsfile.read.common.type.TypeFactory;
+import org.apache.tsfile.utils.Pair;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.LinkedHashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.stream.Collectors;
+import java.util.stream.IntStream;
+
+public class CteMaterializer {
+  private static final Logger LOGGER = LoggerFactory.getLogger(CteMaterializer.class);
+
+  private static final Coordinator coordinator = Coordinator.getInstance();
+  private static final SessionManager sessionManager = SessionManager.getInstance();
+
+  public void materializeCTE(Analysis analysis, MPPQueryContext context) {
+    analysis
+        .getNamedQueries()
+        .forEach(
+            (tableRef, query) -> {
+              Table table = tableRef.getNode();
+              if (query.isMaterialized()) {
+                if (!query.isDone()) {

Review Comment:
   we may need another if to check whether this cte has already failed last time? if so we can just return even if this cte is not done



##########
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/utils/cte/CteDataReader.java:
##########
@@ -0,0 +1,58 @@
+/*
+ *
+ *  * Licensed to the Apache Software Foundation (ASF) under one
+ *  * or more contributor license agreements.  See the NOTICE file
+ *  * distributed with this work for additional information
+ *  * regarding copyright ownership.  The ASF licenses this file
+ *  * to you under the Apache License, Version 2.0 (the
+ *  * "License"); you may not use this file except in compliance
+ *  * with the License.  You may obtain a copy of the License at
+ *  *
+ *  *     http://www.apache.org/licenses/LICENSE-2.0
+ *  *
+ *  * Unless required by applicable law or agreed to in writing,
+ *  * software distributed under the License is distributed on an
+ *  * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ *  * KIND, either express or implied.  See the License for the
+ *  * specific language governing permissions and limitations
+ *  * under the License.
+ *
+ */
+
+package org.apache.iotdb.db.utils.cte;
+
+import org.apache.iotdb.commons.exception.IoTDBException;
+
+import org.apache.tsfile.read.common.block.TsBlock;
+
+public interface CteDataReader {

Review Comment:
   ```suggestion
   public interface CteDataReader extends Accountable {
   ```



##########
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/queryengine/execution/operator/source/relational/CteScanOperator.java:
##########
@@ -0,0 +1,134 @@
+/*
+ *
+ *  * Licensed to the Apache Software Foundation (ASF) under one
+ *  * or more contributor license agreements.  See the NOTICE file
+ *  * distributed with this work for additional information
+ *  * regarding copyright ownership.  The ASF licenses this file
+ *  * to you under the Apache License, Version 2.0 (the
+ *  * "License"); you may not use this file except in compliance
+ *  * with the License.  You may obtain a copy of the License at
+ *  *
+ *  *     http://www.apache.org/licenses/LICENSE-2.0
+ *  *
+ *  * Unless required by applicable law or agreed to in writing,
+ *  * software distributed under the License is distributed on an
+ *  * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ *  * KIND, either express or implied.  See the License for the
+ *  * specific language governing permissions and limitations
+ *  * under the License.
+ *
+ */
+
+package org.apache.iotdb.db.queryengine.execution.operator.source.relational;
+
+import org.apache.iotdb.commons.utils.TestOnly;
+import org.apache.iotdb.db.queryengine.execution.MemoryEstimationHelper;
+import org.apache.iotdb.db.queryengine.execution.operator.OperatorContext;
+import org.apache.iotdb.db.queryengine.execution.operator.source.SourceOperator;
+import org.apache.iotdb.db.queryengine.plan.planner.plan.node.PlanNodeId;
+import org.apache.iotdb.db.utils.cte.CteDataReader;
+import org.apache.iotdb.db.utils.cte.CteDataStore;
+import org.apache.iotdb.db.utils.cte.MemoryReader;
+
+import org.apache.tsfile.read.common.block.TsBlock;
+import org.apache.tsfile.utils.RamUsageEstimator;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+public class CteScanOperator implements SourceOperator {
+  private static final Logger LOGGER = LoggerFactory.getLogger(CteScanOperator.class);
+  private static final long INSTANCE_SIZE =
+      RamUsageEstimator.shallowSizeOfInstance(CteScanOperator.class);
+
+  private final OperatorContext operatorContext;
+  private final PlanNodeId sourceId;
+
+  private final CteDataStore dataStore;
+  private final CteDataReader dataReader;
+  private final int dataStoreRefCount;
+
+  public CteScanOperator(
+      OperatorContext operatorContext, PlanNodeId sourceId, CteDataStore dataStore) {
+    this.operatorContext = operatorContext;
+    this.sourceId = sourceId;
+    this.dataStore = dataStore;
+    this.dataReader = new MemoryReader(dataStore.getCachedData());
+    this.dataStoreRefCount = dataStore.increaseRefCount();
+  }
+
+  @Override
+  public TsBlock next() throws Exception {
+    if (dataReader == null) {
+      return null;
+    }
+    return dataReader.next();
+  }
+
+  @Override
+  public boolean hasNext() throws Exception {
+    if (dataReader == null) {
+      return false;
+    }
+    return dataReader.hasNext();
+  }
+
+  @Override
+  public void close() throws Exception {
+    try {
+      if (dataReader != null) {
+        dataReader.close();
+      }
+    } catch (Exception e) {
+      LOGGER.error("Fail to close fileChannel", e);
+    }
+  }
+
+  @Override
+  public boolean isFinished() throws Exception {
+    return !hasNextWithTimer();
+  }
+
+  @Override
+  public long calculateMaxPeekMemory() {
+    return calculateRetainedSizeAfterCallingNext() + calculateMaxReturnSize();
+  }
+
+  @Override
+  public long calculateMaxReturnSize() {
+    // The returned object is a reference to TsBlock in CteDataReader
+    return RamUsageEstimator.NUM_BYTES_OBJECT_REF;
+  }
+
+  @Override
+  public long calculateRetainedSizeAfterCallingNext() {
+    return 0L;
+  }
+
+  @Override
+  public long ramBytesUsed() {
+    long bytes =
+        INSTANCE_SIZE
+            + MemoryEstimationHelper.getEstimatedSizeOfAccountableObject(operatorContext)
+            + dataReader.bytesUsed();
+    if (dataStoreRefCount == 1) {
+      bytes += dataStore.getCachedBytes();
+    }

Review Comment:
   I think it's better to do memory alloc and free in CteDataReader.close() and init(). first init() call to alloc from LocalExecutionPlanner.reserveFromFreeMemoryForOperators and last free from LocalExecutionPlanner.releaseToFreeMemoryForOperators



##########
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/utils/cte/CteDataReader.java:
##########
@@ -0,0 +1,58 @@
+/*
+ *
+ *  * Licensed to the Apache Software Foundation (ASF) under one
+ *  * or more contributor license agreements.  See the NOTICE file
+ *  * distributed with this work for additional information
+ *  * regarding copyright ownership.  The ASF licenses this file
+ *  * to you under the Apache License, Version 2.0 (the
+ *  * "License"); you may not use this file except in compliance
+ *  * with the License.  You may obtain a copy of the License at
+ *  *
+ *  *     http://www.apache.org/licenses/LICENSE-2.0
+ *  *
+ *  * Unless required by applicable law or agreed to in writing,
+ *  * software distributed under the License is distributed on an
+ *  * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ *  * KIND, either express or implied.  See the License for the
+ *  * specific language governing permissions and limitations
+ *  * under the License.
+ *
+ */
+
+package org.apache.iotdb.db.utils.cte;
+
+import org.apache.iotdb.commons.exception.IoTDBException;
+
+import org.apache.tsfile.read.common.block.TsBlock;
+
+public interface CteDataReader {
+  /**
+   * Check if there is more data in CteDataReader. DiskSpillerReader may run out of current TsBlocks
+   * , then it needs to read from file and cache more data. This method should be called before
+   * next() to ensure that there is data to read.
+   *
+   * @throws IoTDBException the error occurs when reading data from fileChannel
+   */
+  boolean hasNext() throws IoTDBException;
+
+  /**
+   * output the cached data in CteDataReader, it needs to be called after hasNext() returns true.
+   *
+   * @return next TsBlock
+   */
+  TsBlock next() throws IoTDBException;
+
+  /**
+   * Close the CteDataReader and release resources.
+   *
+   * @throws IoTDBException the error occurs when closing fileChannel
+   */
+  void close() throws IoTDBException;
+
+  /**
+   * Get the bytes used by this CteDataReader.
+   *
+   * @return the bytes used by this CteDataReader
+   */
+  long bytesUsed();

Review Comment:
   ```suggestion
     long ramBytesUsed();
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] With clause enhance [iotdb]

Posted by "shizy818 (via GitHub)" <gi...@apache.org>.
shizy818 commented on code in PR #16353:
URL: https://github.com/apache/iotdb/pull/16353#discussion_r2646386133


##########
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/queryengine/plan/relational/planner/ir/PredicateWithUncorrelatedScalarSubqueryReconstructor.java:
##########
@@ -0,0 +1,247 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iotdb.db.queryengine.plan.relational.planner.ir;
+
+import org.apache.iotdb.commons.exception.IoTDBException;
+import org.apache.iotdb.commons.utils.TestOnly;
+import org.apache.iotdb.db.protocol.session.SessionManager;
+import org.apache.iotdb.db.queryengine.common.MPPQueryContext;
+import org.apache.iotdb.db.queryengine.common.MPPQueryContext.ExplainType;
+import org.apache.iotdb.db.queryengine.common.header.DatasetHeader;
+import org.apache.iotdb.db.queryengine.plan.Coordinator;
+import org.apache.iotdb.db.queryengine.plan.execution.ExecutionResult;
+import org.apache.iotdb.db.queryengine.plan.planner.LocalExecutionPlanner;
+import org.apache.iotdb.db.queryengine.plan.relational.analyzer.Analysis;
+import org.apache.iotdb.db.queryengine.plan.relational.sql.ast.BinaryLiteral;
+import org.apache.iotdb.db.queryengine.plan.relational.sql.ast.BooleanLiteral;
+import org.apache.iotdb.db.queryengine.plan.relational.sql.ast.ComparisonExpression;
+import org.apache.iotdb.db.queryengine.plan.relational.sql.ast.DoubleLiteral;
+import org.apache.iotdb.db.queryengine.plan.relational.sql.ast.Expression;
+import org.apache.iotdb.db.queryengine.plan.relational.sql.ast.FunctionCall;
+import org.apache.iotdb.db.queryengine.plan.relational.sql.ast.Identifier;
+import org.apache.iotdb.db.queryengine.plan.relational.sql.ast.Literal;
+import org.apache.iotdb.db.queryengine.plan.relational.sql.ast.LogicalExpression;
+import org.apache.iotdb.db.queryengine.plan.relational.sql.ast.LongLiteral;
+import org.apache.iotdb.db.queryengine.plan.relational.sql.ast.NotExpression;
+import org.apache.iotdb.db.queryengine.plan.relational.sql.ast.Query;
+import org.apache.iotdb.db.queryengine.plan.relational.sql.ast.StringLiteral;
+import org.apache.iotdb.db.queryengine.plan.relational.sql.ast.SubqueryExpression;
+import org.apache.iotdb.db.queryengine.plan.relational.sql.ast.With;
+import org.apache.iotdb.db.queryengine.plan.relational.sql.ast.WithQuery;
+import org.apache.iotdb.db.queryengine.plan.relational.sql.parser.SqlParser;
+import org.apache.iotdb.rpc.TSStatusCode;
+
+import org.apache.tsfile.block.column.Column;
+import org.apache.tsfile.enums.TSDataType;
+import org.apache.tsfile.read.common.block.TsBlock;
+
+import java.util.List;
+import java.util.Optional;
+import java.util.stream.Collectors;
+
+import static com.google.common.base.Preconditions.checkArgument;
+
+public class PredicateWithUncorrelatedScalarSubqueryReconstructor {
+
+  private static final SqlParser relationSqlParser = new SqlParser();

Review Comment:
   done



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] With clause enhance [iotdb]

Posted by "JackieTien97 (via GitHub)" <gi...@apache.org>.
JackieTien97 merged PR #16353:
URL: https://github.com/apache/iotdb/pull/16353


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] With clause enhance [iotdb]

Posted by "shizy818 (via GitHub)" <gi...@apache.org>.
shizy818 commented on code in PR #16353:
URL: https://github.com/apache/iotdb/pull/16353#discussion_r2659245175


##########
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/queryengine/plan/relational/planner/CteMaterializer.java:
##########
@@ -0,0 +1,354 @@
+/*
+ *
+ *  * Licensed to the Apache Software Foundation (ASF) under one
+ *  * or more contributor license agreements.  See the NOTICE file
+ *  * distributed with this work for additional information
+ *  * regarding copyright ownership.  The ASF licenses this file
+ *  * to you under the Apache License, Version 2.0 (the
+ *  * "License"); you may not use this file except in compliance
+ *  * with the License.  You may obtain a copy of the License at
+ *  *
+ *  *     http://www.apache.org/licenses/LICENSE-2.0
+ *  *
+ *  * Unless required by applicable law or agreed to in writing,
+ *  * software distributed under the License is distributed on an
+ *  * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ *  * KIND, either express or implied.  See the License for the
+ *  * specific language governing permissions and limitations
+ *  * under the License.
+ *
+ */
+
+package org.apache.iotdb.db.queryengine.plan.relational.planner;
+
+import org.apache.iotdb.common.rpc.thrift.TEndPoint;
+import org.apache.iotdb.commons.client.IClientManager;
+import org.apache.iotdb.commons.client.sync.SyncDataNodeInternalServiceClient;
+import org.apache.iotdb.commons.exception.IoTDBException;
+import org.apache.iotdb.commons.exception.IoTDBRuntimeException;
+import org.apache.iotdb.commons.schema.table.column.TsTableColumnCategory;
+import org.apache.iotdb.db.exception.mpp.FragmentInstanceFetchException;
+import org.apache.iotdb.db.protocol.session.SessionManager;
+import org.apache.iotdb.db.queryengine.common.FragmentInstanceId;
+import org.apache.iotdb.db.queryengine.common.MPPQueryContext;
+import org.apache.iotdb.db.queryengine.common.header.DatasetHeader;
+import org.apache.iotdb.db.queryengine.plan.Coordinator;
+import org.apache.iotdb.db.queryengine.plan.execution.ExecutionResult;
+import org.apache.iotdb.db.queryengine.plan.execution.QueryExecution;
+import org.apache.iotdb.db.queryengine.plan.planner.LocalExecutionPlanner;
+import org.apache.iotdb.db.queryengine.plan.planner.plan.DistributedQueryPlan;
+import org.apache.iotdb.db.queryengine.plan.planner.plan.FragmentInstance;
+import org.apache.iotdb.db.queryengine.plan.relational.analyzer.Analysis;
+import org.apache.iotdb.db.queryengine.plan.relational.metadata.ColumnSchema;
+import org.apache.iotdb.db.queryengine.plan.relational.metadata.TableSchema;
+import org.apache.iotdb.db.queryengine.plan.relational.sql.ast.Identifier;
+import org.apache.iotdb.db.queryengine.plan.relational.sql.ast.Query;
+import org.apache.iotdb.db.queryengine.plan.relational.sql.ast.Table;
+import org.apache.iotdb.db.queryengine.plan.relational.sql.ast.With;
+import org.apache.iotdb.db.queryengine.plan.relational.sql.ast.WithQuery;
+import org.apache.iotdb.db.queryengine.plan.relational.sql.parser.SqlParser;
+import org.apache.iotdb.db.queryengine.statistics.FragmentInstanceStatisticsDrawer;
+import org.apache.iotdb.db.queryengine.statistics.QueryStatisticsFetcher;
+import org.apache.iotdb.db.queryengine.statistics.StatisticLine;
+import org.apache.iotdb.db.utils.cte.CteDataStore;
+import org.apache.iotdb.mpp.rpc.thrift.TFetchFragmentInstanceStatisticsResp;
+import org.apache.iotdb.rpc.TSStatusCode;
+
+import com.google.common.collect.ImmutableList;
+import org.apache.tsfile.enums.TSDataType;
+import org.apache.tsfile.read.common.block.TsBlock;
+import org.apache.tsfile.read.common.type.TypeFactory;
+import org.apache.tsfile.utils.Pair;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.LinkedHashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.stream.Collectors;
+import java.util.stream.IntStream;
+
+public class CteMaterializer {
+  private static final Logger LOGGER = LoggerFactory.getLogger(CteMaterializer.class);
+
+  private static final Coordinator coordinator = Coordinator.getInstance();
+  private static final SessionManager sessionManager = SessionManager.getInstance();
+
+  public void materializeCTE(Analysis analysis, MPPQueryContext context) {
+    analysis
+        .getNamedQueries()
+        .forEach(
+            (tableRef, query) -> {
+              Table table = tableRef.getNode();
+              if (query.isMaterialized()) {
+                if (!query.isDone()) {

Review Comment:
   done



##########
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/queryengine/plan/relational/planner/QueryPlanner.java:
##########
@@ -763,9 +764,13 @@ private PlanBuilder filter(PlanBuilder subPlan, Expression predicate, Node node)
     }
 
     subPlan = subqueryPlanner.handleSubqueries(subPlan, predicate, analysis.getSubqueries(node));
-    return subPlan.withNewRoot(
-        new FilterNode(
-            queryIdAllocator.genPlanNodeId(), subPlan.getRoot(), subPlan.rewrite(predicate)));
+    PlanBuilder planBuilder =
+        subPlan.withNewRoot(
+            new FilterNode(
+                queryIdAllocator.genPlanNodeId(), subPlan.getRoot(), subPlan.rewrite(predicate)));
+    PredicateWithUncorrelatedScalarSubqueryReconstructor.getInstance()
+        .clearShadowExpression(predicate);

Review Comment:
   done



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] With clause enhance [iotdb]

Posted by "shizy818 (via GitHub)" <gi...@apache.org>.
shizy818 commented on code in PR #16353:
URL: https://github.com/apache/iotdb/pull/16353#discussion_r2659245554


##########
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/queryengine/execution/operator/source/relational/CteScanOperator.java:
##########
@@ -0,0 +1,134 @@
+/*
+ *
+ *  * Licensed to the Apache Software Foundation (ASF) under one
+ *  * or more contributor license agreements.  See the NOTICE file
+ *  * distributed with this work for additional information
+ *  * regarding copyright ownership.  The ASF licenses this file
+ *  * to you under the Apache License, Version 2.0 (the
+ *  * "License"); you may not use this file except in compliance
+ *  * with the License.  You may obtain a copy of the License at
+ *  *
+ *  *     http://www.apache.org/licenses/LICENSE-2.0
+ *  *
+ *  * Unless required by applicable law or agreed to in writing,
+ *  * software distributed under the License is distributed on an
+ *  * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ *  * KIND, either express or implied.  See the License for the
+ *  * specific language governing permissions and limitations
+ *  * under the License.
+ *
+ */
+
+package org.apache.iotdb.db.queryengine.execution.operator.source.relational;
+
+import org.apache.iotdb.commons.utils.TestOnly;
+import org.apache.iotdb.db.queryengine.execution.MemoryEstimationHelper;
+import org.apache.iotdb.db.queryengine.execution.operator.OperatorContext;
+import org.apache.iotdb.db.queryengine.execution.operator.source.SourceOperator;
+import org.apache.iotdb.db.queryengine.plan.planner.plan.node.PlanNodeId;
+import org.apache.iotdb.db.utils.cte.CteDataReader;
+import org.apache.iotdb.db.utils.cte.CteDataStore;
+import org.apache.iotdb.db.utils.cte.MemoryReader;
+
+import org.apache.tsfile.read.common.block.TsBlock;
+import org.apache.tsfile.utils.RamUsageEstimator;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+public class CteScanOperator implements SourceOperator {
+  private static final Logger LOGGER = LoggerFactory.getLogger(CteScanOperator.class);
+  private static final long INSTANCE_SIZE =
+      RamUsageEstimator.shallowSizeOfInstance(CteScanOperator.class);
+
+  private final OperatorContext operatorContext;
+  private final PlanNodeId sourceId;
+
+  private final CteDataStore dataStore;
+  private final CteDataReader dataReader;
+  private final int dataStoreRefCount;
+
+  public CteScanOperator(
+      OperatorContext operatorContext, PlanNodeId sourceId, CteDataStore dataStore) {
+    this.operatorContext = operatorContext;
+    this.sourceId = sourceId;
+    this.dataStore = dataStore;
+    this.dataReader = new MemoryReader(dataStore.getCachedData());
+    this.dataStoreRefCount = dataStore.increaseRefCount();
+  }
+
+  @Override
+  public TsBlock next() throws Exception {
+    if (dataReader == null) {
+      return null;
+    }
+    return dataReader.next();
+  }
+
+  @Override
+  public boolean hasNext() throws Exception {
+    if (dataReader == null) {
+      return false;
+    }
+    return dataReader.hasNext();
+  }
+
+  @Override
+  public void close() throws Exception {
+    try {
+      if (dataReader != null) {
+        dataReader.close();
+      }
+    } catch (Exception e) {
+      LOGGER.error("Fail to close fileChannel", e);
+    }
+  }
+
+  @Override
+  public boolean isFinished() throws Exception {
+    return !hasNextWithTimer();
+  }
+
+  @Override
+  public long calculateMaxPeekMemory() {
+    return calculateRetainedSizeAfterCallingNext() + calculateMaxReturnSize();
+  }
+
+  @Override
+  public long calculateMaxReturnSize() {
+    // The returned object is a reference to TsBlock in CteDataReader
+    return RamUsageEstimator.NUM_BYTES_OBJECT_REF;
+  }
+
+  @Override
+  public long calculateRetainedSizeAfterCallingNext() {
+    return 0L;
+  }
+
+  @Override
+  public long ramBytesUsed() {
+    long bytes =
+        INSTANCE_SIZE
+            + MemoryEstimationHelper.getEstimatedSizeOfAccountableObject(operatorContext)
+            + dataReader.bytesUsed();
+    if (dataStoreRefCount == 1) {
+      bytes += dataStore.getCachedBytes();
+    }

Review Comment:
   done



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] With clause enhance [iotdb]

Posted by "shizy818 (via GitHub)" <gi...@apache.org>.
shizy818 commented on code in PR #16353:
URL: https://github.com/apache/iotdb/pull/16353#discussion_r2646446339


##########
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/queryengine/plan/relational/analyzer/predicate/ConvertPredicateToTimeFilterVisitor.java:
##########
@@ -299,6 +300,13 @@ protected Filter visitBetweenPredicate(BetweenPredicate node, Void context) {
   }
 
   public static long getLongValue(Expression expression) {
-    return ((LongLiteral) expression).getParsedValue();
+    if (expression instanceof LongLiteral) {
+      return ((LongLiteral) expression).getParsedValue();
+    } else if (expression instanceof DoubleLiteral) {
+      return (long) ((DoubleLiteral) expression).getValue();
+    } else {
+      throw new IllegalArgumentException(
+          "Expression should be LongLiteral or DoubleLiteral, but got: " + expression.getClass());
+    }

Review Comment:
   done



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] With clause enhance [iotdb]

Posted by "Copilot (via GitHub)" <gi...@apache.org>.
Copilot commented on code in PR #16353:
URL: https://github.com/apache/iotdb/pull/16353#discussion_r2596803309


##########
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/queryengine/plan/relational/analyzer/Scope.java:
##########
@@ -64,16 +68,30 @@ private Scope(
       boolean queryBoundary,
       RelationId relationId,
       RelationType relation,
-      Map<String, WithQuery> namedQueries) {
+      Map<String, WithQuery> namedQueries,
+      List<Identifier> tables) {
     this.parent = requireNonNull(parent, "parent is null");
     this.relationId = requireNonNull(relationId, "relationId is null");
     this.queryBoundary = queryBoundary;
     this.relation = requireNonNull(relation, "relation is null");
     this.namedQueries = ImmutableMap.copyOf(requireNonNull(namedQueries, "namedQueries is null"));
+    this.tables = new ArrayList<>(requireNonNull(tables, "tables is null"));
+  }
+
+  public void addTable(Table table) {
+    tables.add(new Identifier(table.getName().getSuffix()));
+  }
+
+  public Scope copy() {
+    return builder().like(this).build();
+  }
+
+  public List<Identifier> getTables() {

Review Comment:
   getTables exposes the internal representation stored in field tables. The value may be modified [after this call to getTables](1).
   getTables exposes the internal representation stored in field tables. The value may be modified [after this call to getTables](2).



##########
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/queryengine/common/MPPQueryContext.java:
##########
@@ -435,6 +477,50 @@ public void setUserQuery(boolean userQuery) {
     this.userQuery = userQuery;
   }
 
+  public boolean isSubquery() {
+    return subquery;
+  }
+
+  public void setSubquery(boolean subquery) {
+    this.subquery = subquery;
+  }
+
+  public void addCteMaterializationCost(Table table, long cost) {
+    cteMaterializationCosts.put(NodeRef.of(table), cost);
+  }
+
+  public Map<NodeRef<Table>, Long> getCteMaterializationCosts() {
+    return cteMaterializationCosts;
+  }
+
+  public void addCteDataStore(Table table, CteDataStore dataStore) {
+    cteDataStores.put(NodeRef.of(table), dataStore);
+  }
+
+  public Map<NodeRef<Table>, CteDataStore> getCteDataStores() {
+    return cteDataStores;

Review Comment:
   getCteDataStores exposes the internal representation stored in field cteDataStores. The value may be modified [after this call to getCteDataStores](1).
   ```suggestion
       return Collections.unmodifiableMap(cteDataStores);
   ```



##########
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/queryengine/common/MPPQueryContext.java:
##########
@@ -435,6 +477,50 @@ public void setUserQuery(boolean userQuery) {
     this.userQuery = userQuery;
   }
 
+  public boolean isSubquery() {
+    return subquery;
+  }
+
+  public void setSubquery(boolean subquery) {
+    this.subquery = subquery;
+  }
+
+  public void addCteMaterializationCost(Table table, long cost) {
+    cteMaterializationCosts.put(NodeRef.of(table), cost);
+  }
+
+  public Map<NodeRef<Table>, Long> getCteMaterializationCosts() {
+    return cteMaterializationCosts;
+  }
+
+  public void addCteDataStore(Table table, CteDataStore dataStore) {
+    cteDataStores.put(NodeRef.of(table), dataStore);
+  }
+
+  public Map<NodeRef<Table>, CteDataStore> getCteDataStores() {
+    return cteDataStores;
+  }
+
+  public CteDataStore getCteDataStore(Table table) {
+    return cteDataStores.get(NodeRef.of(table));
+  }
+
+  public void setCteDataStores(Map<NodeRef<Table>, CteDataStore> cteDataStores) {
+    this.cteDataStores = cteDataStores;
+  }
+
+  public Map<Query, List<Identifier>> getSubQueryTables() {
+    return subQueryTables;

Review Comment:
   getSubQueryTables exposes the internal representation stored in field subQueryTables. The value may be modified [after this call to getSubQueryTables](1).
   getSubQueryTables exposes the internal representation stored in field subQueryTables. The value may be modified [after this call to getSubQueryTables](2).
   ```suggestion
       return Collections.unmodifiableMap(subQueryTables);
   ```



##########
integration-test/src/test/java/org/apache/iotdb/relational/it/query/recent/IoTExplainAnalyzeIT.java:
##########
@@ -0,0 +1,260 @@
+/*
+ *
+ *  * Licensed to the Apache Software Foundation (ASF) under one
+ *  * or more contributor license agreements.  See the NOTICE file
+ *  * distributed with this work for additional information
+ *  * regarding copyright ownership.  The ASF licenses this file
+ *  * to you under the Apache License, Version 2.0 (the
+ *  * "License"); you may not use this file except in compliance
+ *  * with the License.  You may obtain a copy of the License at
+ *  *
+ *  *     http://www.apache.org/licenses/LICENSE-2.0
+ *  *
+ *  * Unless required by applicable law or agreed to in writing,
+ *  * software distributed under the License is distributed on an
+ *  * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ *  * KIND, either express or implied.  See the License for the
+ *  * specific language governing permissions and limitations
+ *  * under the License.
+ *
+ */
+
+package org.apache.iotdb.relational.it.query.recent;
+
+import org.apache.iotdb.it.env.EnvFactory;
+import org.apache.iotdb.it.framework.IoTDBTestRunner;
+import org.apache.iotdb.itbase.category.TableLocalStandaloneIT;
+import org.apache.iotdb.itbase.env.BaseEnv;
+
+import org.junit.After;
+import org.junit.AfterClass;
+import org.junit.Assert;
+import org.junit.Before;
+import org.junit.BeforeClass;
+import org.junit.Test;
+import org.junit.experimental.categories.Category;
+import org.junit.runner.RunWith;
+
+import java.sql.Connection;
+import java.sql.ResultSet;
+import java.sql.SQLException;
+import java.sql.Statement;
+import java.util.Arrays;
+import java.util.Locale;
+import java.util.function.ToLongFunction;
+import java.util.regex.Matcher;
+import java.util.regex.Pattern;
+
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.fail;
+
+@RunWith(IoTDBTestRunner.class)
+@Category({TableLocalStandaloneIT.class})
+public class IoTExplainAnalyzeIT {
+  private static final String DATABASE_NAME = "testdb";
+
+  private static final String[] creationSqls =
+      new String[] {
+        "CREATE DATABASE IF NOT EXISTS " + DATABASE_NAME,
+        "USE " + DATABASE_NAME,
+        "CREATE TABLE IF NOT EXISTS testtb(deviceid STRING TAG, voltage FLOAT FIELD)",
+        "INSERT INTO testtb VALUES(1000, 'd1', 100.0)",
+        "INSERT INTO testtb VALUES(2000, 'd1', 200.0)",
+        "INSERT INTO testtb VALUES(1000, 'd2', 300.0)",
+      };
+
+  private static final String dropDbSqls = "DROP DATABASE IF EXISTS " + DATABASE_NAME;
+
+  @BeforeClass
+  public static void setUpClass() {
+    Locale.setDefault(Locale.ENGLISH);
+
+    EnvFactory.getEnv()
+        .getConfig()
+        .getCommonConfig()
+        .setPartitionInterval(1000)
+        .setMemtableSizeThreshold(10000)
+        .setMaxRowsInCteBuffer(100);
+    EnvFactory.getEnv().initClusterEnvironment();
+  }
+
+  @AfterClass
+  public static void tearDownClass() {
+    EnvFactory.getEnv().cleanClusterEnvironment();
+  }
+
+  @Before
+  public void setUp() throws SQLException {
+    prepareData();
+  }
+
+  @After
+  public void tearDown() {
+    try (Connection connection = EnvFactory.getEnv().getConnection(BaseEnv.TABLE_SQL_DIALECT);
+        Statement statement = connection.createStatement()) {
+      statement.execute(dropDbSqls);
+    } catch (Exception e) {
+      fail(e.getMessage());
+    }
+  }
+
+  @Test
+  public void testEmptyCteQuery() {
+    String sql =
+        "explain analyze with cte1 as materialized (select * from testtb1) select * from testtb, cte1 where testtb.deviceid = cte1.deviceid";
+    try (Connection conn = EnvFactory.getEnv().getConnection(BaseEnv.TABLE_SQL_DIALECT);
+        Statement statement = conn.createStatement()) {
+      statement.execute("Use " + DATABASE_NAME);
+      statement.execute(
+          "CREATE TABLE IF NOT EXISTS testtb1(deviceid STRING TAG, voltage FLOAT FIELD)");
+      ResultSet resultSet = statement.executeQuery(sql);
+      StringBuilder sb = new StringBuilder();
+      while (resultSet.next()) {
+        System.out.println(resultSet.getString(1));
+        sb.append(resultSet.getString(1)).append(System.lineSeparator());
+      }
+      resultSet.close();
+
+      String result = sb.toString();
+      Assert.assertFalse(
+          "Explain Analyze should not contain ExplainAnalyze node.",
+          result.contains("ExplainAnalyzeNode"));
+
+      String[] lines = result.split(System.lineSeparator());
+      Assert.assertTrue(lines.length > 3);
+      Assert.assertEquals("CTE Query : 'cte1'", lines[0]);
+      Assert.assertEquals("", lines[1]);
+      Assert.assertEquals("Main Query", lines[2]);
+      statement.execute("DROP TABLE testtb1");
+
+    } catch (SQLException e) {
+      fail(e.getMessage());
+    }
+  }
+
+  @Test
+  public void testCteQueryExceedsThreshold() {
+    String sql =
+        "explain analyze with cte1 as materialized (select * from testtb2) select * from testtb where testtb.deviceid in (select deviceid from cte1)";
+    try (Connection conn = EnvFactory.getEnv().getConnection(BaseEnv.TABLE_SQL_DIALECT);
+        Statement statement = conn.createStatement()) {
+      statement.execute("Use " + DATABASE_NAME);
+      statement.execute(
+          "CREATE TABLE IF NOT EXISTS testtb2(deviceid STRING TAG, voltage FLOAT FIELD)");
+      for (int i = 0; i < 100; i++) {
+        statement.addBatch(
+            String.format("insert into testtb2(deviceid, voltage) values('d%d', %d)", i, i));
+      }
+      statement.executeBatch();
+      ResultSet resultSet = statement.executeQuery(sql);
+      StringBuilder sb = new StringBuilder();
+      while (resultSet.next()) {
+        sb.append(resultSet.getString(1)).append(System.lineSeparator());
+      }
+      resultSet.close();
+
+      String result = sb.toString();
+      Assert.assertFalse(
+          "Main Query should not contain CteScan node when the CTE query's result set exceeds threshold.",
+          result.contains("CteScanNode(CteScanOperator)"));
+      Assert.assertTrue(
+          "CTE Query should contain warning message when CTE query's result set exceeds threshold.",
+          result.contains("Failed to materialize CTE"));
+      Assert.assertFalse(
+          "Explain Analyze should not contain ExplainAnalyze node.",
+          result.contains("ExplainAnalyzeNode"));
+
+      String[] plans = result.split("Main Query");
+      for (String plan : plans) {
+        String[] lines = plan.split(System.lineSeparator());
+        long[] instanceCount =
+            Arrays.stream(lines)
+                .filter(line -> line.contains("Fragment Instances Count:"))
+                .mapToLong(extractNumber("Fragment Instances Count:\\s(\\d+)"))
+                .toArray();
+        assertEquals(instanceCount.length, 1);
+
+        long totalInstances =
+            Arrays.stream(lines).filter(line -> line.contains("FRAGMENT-INSTANCE")).count();
+        assertEquals(totalInstances, instanceCount[0]);
+      }
+
+      statement.execute("DROP TABLE testtb2");
+    } catch (SQLException e) {
+      fail(e.getMessage());
+    }
+  }
+
+  @Test
+  public void testCteQuerySuccess() {
+    String sql =
+        "explain analyze with cte1 as materialized (select voltage, deviceid from testtb3) select * from testtb where testtb.deviceid in (select deviceid from cte1)";
+    try (Connection conn = EnvFactory.getEnv().getConnection(BaseEnv.TABLE_SQL_DIALECT);
+        Statement statement = conn.createStatement()) {
+      statement.execute("Use " + DATABASE_NAME);
+      statement.execute(
+          "CREATE TABLE IF NOT EXISTS testtb3(deviceid STRING TAG, voltage FLOAT FIELD)");
+      for (int i = 0; i < 50; i++) {
+        statement.addBatch(
+            String.format("insert into testtb3(deviceid, voltage) values('d%d', %d)", i, i));
+      }
+      statement.executeBatch();
+      ResultSet resultSet = statement.executeQuery(sql);
+      StringBuilder sb = new StringBuilder();
+      while (resultSet.next()) {
+        sb.append(resultSet.getString(1)).append(System.lineSeparator());
+      }
+      resultSet.close();
+
+      String result = sb.toString();
+      Assert.assertTrue(
+          "Main Query should contain CteScan node when the CTE query's result set does not exceeds threshold.",
+          result.contains("CteScanNode(CteScanOperator)"));
+      Assert.assertFalse(
+          "Explain Analyze should not contain ExplainAnalyze node.",
+          result.contains("ExplainAnalyzeNode"));
+
+      String[] plans = result.split("Main Query");
+      for (String plan : plans) {
+        String[] lines = plan.split(System.lineSeparator());
+        long[] instanceCount =
+            Arrays.stream(lines)
+                .filter(line -> line.contains("Fragment Instances Count:"))
+                .mapToLong(extractNumber("Fragment Instances Count:\\s(\\d+)"))
+                .toArray();
+        assertEquals(instanceCount.length, 1);
+
+        long totalInstances =
+            Arrays.stream(lines).filter(line -> line.contains("FRAGMENT-INSTANCE")).count();
+        assertEquals(totalInstances, instanceCount[0]);
+      }
+
+      statement.execute("DROP TABLE testtb3");
+    } catch (SQLException e) {
+      fail(e.getMessage());
+    }
+  }
+
+  private static ToLongFunction<String> extractNumber(String regex) {
+    return line -> {
+      Pattern pattern = Pattern.compile(regex);
+      Matcher matcher = pattern.matcher(line);
+      if (matcher.find()) {
+        return Long.parseLong(matcher.group(1));

Review Comment:
   Potential uncaught 'java.lang.NumberFormatException'.
   ```suggestion
           try {
             return Long.parseLong(matcher.group(1));
           } catch (NumberFormatException e) {
             return 0;
           }
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] With clause enhance [iotdb]

Posted by "shizy818 (via GitHub)" <gi...@apache.org>.
shizy818 commented on code in PR #16353:
URL: https://github.com/apache/iotdb/pull/16353#discussion_r2645065828


##########
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/queryengine/execution/operator/source/relational/CteScanOperator.java:
##########
@@ -0,0 +1,134 @@
+/*
+ *
+ *  * Licensed to the Apache Software Foundation (ASF) under one
+ *  * or more contributor license agreements.  See the NOTICE file
+ *  * distributed with this work for additional information
+ *  * regarding copyright ownership.  The ASF licenses this file
+ *  * to you under the Apache License, Version 2.0 (the
+ *  * "License"); you may not use this file except in compliance
+ *  * with the License.  You may obtain a copy of the License at
+ *  *
+ *  *     http://www.apache.org/licenses/LICENSE-2.0
+ *  *
+ *  * Unless required by applicable law or agreed to in writing,
+ *  * software distributed under the License is distributed on an
+ *  * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ *  * KIND, either express or implied.  See the License for the
+ *  * specific language governing permissions and limitations
+ *  * under the License.
+ *
+ */
+
+package org.apache.iotdb.db.queryengine.execution.operator.source.relational;
+
+import org.apache.iotdb.commons.utils.TestOnly;
+import org.apache.iotdb.db.queryengine.execution.MemoryEstimationHelper;
+import org.apache.iotdb.db.queryengine.execution.operator.OperatorContext;
+import org.apache.iotdb.db.queryengine.execution.operator.source.SourceOperator;
+import org.apache.iotdb.db.queryengine.plan.planner.plan.node.PlanNodeId;
+import org.apache.iotdb.db.utils.cte.CteDataReader;
+import org.apache.iotdb.db.utils.cte.CteDataStore;
+import org.apache.iotdb.db.utils.cte.MemoryReader;
+
+import org.apache.tsfile.read.common.block.TsBlock;
+import org.apache.tsfile.utils.RamUsageEstimator;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+public class CteScanOperator implements SourceOperator {
+  private static final Logger LOGGER = LoggerFactory.getLogger(CteScanOperator.class);
+  private static final long INSTANCE_SIZE =
+      RamUsageEstimator.shallowSizeOfInstance(CteScanOperator.class);
+
+  private final OperatorContext operatorContext;
+  private final PlanNodeId sourceId;
+
+  private final CteDataStore dataStore;
+  private final CteDataReader dataReader;
+  private final int dataStoreRefCount;
+
+  public CteScanOperator(
+      OperatorContext operatorContext, PlanNodeId sourceId, CteDataStore dataStore) {
+    this.operatorContext = operatorContext;
+    this.sourceId = sourceId;
+    this.dataStore = dataStore;
+    this.dataReader = new MemoryReader(dataStore.getCachedData());
+    this.dataStoreRefCount = dataStore.increaseRefCount();
+  }
+
+  @Override
+  public TsBlock next() throws Exception {
+    if (dataReader == null) {
+      return null;
+    }
+    return dataReader.next();
+  }
+
+  @Override
+  public boolean hasNext() throws Exception {
+    if (dataReader == null) {
+      return false;
+    }
+    return dataReader.hasNext();
+  }
+
+  @Override
+  public void close() throws Exception {
+    try {
+      if (dataReader != null) {
+        dataReader.close();
+      }
+    } catch (Exception e) {
+      LOGGER.error("Fail to close fileChannel", e);
+    }
+  }
+
+  @Override
+  public boolean isFinished() throws Exception {
+    return !hasNextWithTimer();
+  }
+
+  @Override
+  public long calculateMaxPeekMemory() {
+    return calculateRetainedSizeAfterCallingNext() + calculateMaxReturnSize();
+  }
+
+  @Override
+  public long calculateMaxReturnSize() {
+    // The returned object is a reference to TsBlock in CteDataReader
+    return RamUsageEstimator.NUM_BYTES_OBJECT_REF;
+  }
+
+  @Override
+  public long calculateRetainedSizeAfterCallingNext() {
+    return 0L;
+  }
+
+  @Override
+  public long ramBytesUsed() {
+    long bytes =
+        INSTANCE_SIZE
+            + MemoryEstimationHelper.getEstimatedSizeOfAccountableObject(operatorContext)
+            + dataReader.bytesUsed();
+    if (dataStoreRefCount == 1) {
+      bytes += dataStore.getCachedBytes();
+    }

Review Comment:
   > what if the FI which this CteScanOperator belongs to has finished, but other CteScanOperator with the same dataStore hasn't finished?
   
   Yes currently it just ensures dataStore be counted once.  If the FI which this CteScanOperator belongs to has finished, the number of used memory decreases. But actually it is still used until other CteScanOperator with the same dataStore finish.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] With clause enhance [iotdb]

Posted by "shizy818 (via GitHub)" <gi...@apache.org>.
shizy818 commented on code in PR #16353:
URL: https://github.com/apache/iotdb/pull/16353#discussion_r2645089697


##########
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/queryengine/plan/relational/analyzer/predicate/ConvertPredicateToTimeFilterVisitor.java:
##########
@@ -299,6 +300,13 @@ protected Filter visitBetweenPredicate(BetweenPredicate node, Void context) {
   }
 
   public static long getLongValue(Expression expression) {
-    return ((LongLiteral) expression).getParsedValue();
+    if (expression instanceof LongLiteral) {
+      return ((LongLiteral) expression).getParsedValue();
+    } else if (expression instanceof DoubleLiteral) {
+      return (long) ((DoubleLiteral) expression).getValue();
+    } else {
+      throw new IllegalArgumentException(
+          "Expression should be LongLiteral or DoubleLiteral, but got: " + expression.getClass());
+    }

Review Comment:
   ok



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] With clause enhance [iotdb]

Posted by "shizy818 (via GitHub)" <gi...@apache.org>.
shizy818 commented on code in PR #16353:
URL: https://github.com/apache/iotdb/pull/16353#discussion_r2645262124


##########
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/queryengine/execution/operator/source/relational/CteScanOperator.java:
##########
@@ -0,0 +1,134 @@
+/*
+ *
+ *  * Licensed to the Apache Software Foundation (ASF) under one
+ *  * or more contributor license agreements.  See the NOTICE file
+ *  * distributed with this work for additional information
+ *  * regarding copyright ownership.  The ASF licenses this file
+ *  * to you under the Apache License, Version 2.0 (the
+ *  * "License"); you may not use this file except in compliance
+ *  * with the License.  You may obtain a copy of the License at
+ *  *
+ *  *     http://www.apache.org/licenses/LICENSE-2.0
+ *  *
+ *  * Unless required by applicable law or agreed to in writing,
+ *  * software distributed under the License is distributed on an
+ *  * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ *  * KIND, either express or implied.  See the License for the
+ *  * specific language governing permissions and limitations
+ *  * under the License.
+ *
+ */
+
+package org.apache.iotdb.db.queryengine.execution.operator.source.relational;
+
+import org.apache.iotdb.commons.utils.TestOnly;
+import org.apache.iotdb.db.queryengine.execution.MemoryEstimationHelper;
+import org.apache.iotdb.db.queryengine.execution.operator.OperatorContext;
+import org.apache.iotdb.db.queryengine.execution.operator.source.SourceOperator;
+import org.apache.iotdb.db.queryengine.plan.planner.plan.node.PlanNodeId;
+import org.apache.iotdb.db.utils.cte.CteDataReader;
+import org.apache.iotdb.db.utils.cte.CteDataStore;
+import org.apache.iotdb.db.utils.cte.MemoryReader;
+
+import org.apache.tsfile.read.common.block.TsBlock;
+import org.apache.tsfile.utils.RamUsageEstimator;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+public class CteScanOperator implements SourceOperator {
+  private static final Logger LOGGER = LoggerFactory.getLogger(CteScanOperator.class);
+  private static final long INSTANCE_SIZE =
+      RamUsageEstimator.shallowSizeOfInstance(CteScanOperator.class);
+
+  private final OperatorContext operatorContext;
+  private final PlanNodeId sourceId;
+
+  private final CteDataStore dataStore;
+  private final CteDataReader dataReader;
+  private final int dataStoreRefCount;
+
+  public CteScanOperator(
+      OperatorContext operatorContext, PlanNodeId sourceId, CteDataStore dataStore) {
+    this.operatorContext = operatorContext;
+    this.sourceId = sourceId;
+    this.dataStore = dataStore;
+    this.dataReader = new MemoryReader(dataStore.getCachedData());
+    this.dataStoreRefCount = dataStore.increaseRefCount();
+  }
+
+  @Override
+  public TsBlock next() throws Exception {
+    if (dataReader == null) {
+      return null;
+    }
+    return dataReader.next();
+  }
+
+  @Override
+  public boolean hasNext() throws Exception {
+    if (dataReader == null) {
+      return false;
+    }
+    return dataReader.hasNext();
+  }
+
+  @Override
+  public void close() throws Exception {
+    try {
+      if (dataReader != null) {
+        dataReader.close();
+      }
+    } catch (Exception e) {
+      LOGGER.error("Fail to close fileChannel", e);
+    }
+  }
+
+  @Override
+  public boolean isFinished() throws Exception {
+    return !hasNextWithTimer();
+  }
+
+  @Override
+  public long calculateMaxPeekMemory() {
+    return calculateRetainedSizeAfterCallingNext() + calculateMaxReturnSize();
+  }
+
+  @Override
+  public long calculateMaxReturnSize() {
+    // The returned object is a reference to TsBlock in CteDataReader
+    return RamUsageEstimator.NUM_BYTES_OBJECT_REF;

Review Comment:
   done



##########
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/queryengine/execution/operator/source/relational/CteScanOperator.java:
##########
@@ -0,0 +1,134 @@
+/*
+ *
+ *  * Licensed to the Apache Software Foundation (ASF) under one
+ *  * or more contributor license agreements.  See the NOTICE file
+ *  * distributed with this work for additional information
+ *  * regarding copyright ownership.  The ASF licenses this file
+ *  * to you under the Apache License, Version 2.0 (the
+ *  * "License"); you may not use this file except in compliance
+ *  * with the License.  You may obtain a copy of the License at
+ *  *
+ *  *     http://www.apache.org/licenses/LICENSE-2.0
+ *  *
+ *  * Unless required by applicable law or agreed to in writing,
+ *  * software distributed under the License is distributed on an
+ *  * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ *  * KIND, either express or implied.  See the License for the
+ *  * specific language governing permissions and limitations
+ *  * under the License.
+ *
+ */
+
+package org.apache.iotdb.db.queryengine.execution.operator.source.relational;
+
+import org.apache.iotdb.commons.utils.TestOnly;
+import org.apache.iotdb.db.queryengine.execution.MemoryEstimationHelper;
+import org.apache.iotdb.db.queryengine.execution.operator.OperatorContext;
+import org.apache.iotdb.db.queryengine.execution.operator.source.SourceOperator;
+import org.apache.iotdb.db.queryengine.plan.planner.plan.node.PlanNodeId;
+import org.apache.iotdb.db.utils.cte.CteDataReader;
+import org.apache.iotdb.db.utils.cte.CteDataStore;
+import org.apache.iotdb.db.utils.cte.MemoryReader;
+
+import org.apache.tsfile.read.common.block.TsBlock;
+import org.apache.tsfile.utils.RamUsageEstimator;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+public class CteScanOperator implements SourceOperator {
+  private static final Logger LOGGER = LoggerFactory.getLogger(CteScanOperator.class);
+  private static final long INSTANCE_SIZE =
+      RamUsageEstimator.shallowSizeOfInstance(CteScanOperator.class);
+
+  private final OperatorContext operatorContext;
+  private final PlanNodeId sourceId;
+
+  private final CteDataStore dataStore;
+  private final CteDataReader dataReader;
+  private final int dataStoreRefCount;
+
+  public CteScanOperator(
+      OperatorContext operatorContext, PlanNodeId sourceId, CteDataStore dataStore) {
+    this.operatorContext = operatorContext;
+    this.sourceId = sourceId;
+    this.dataStore = dataStore;
+    this.dataReader = new MemoryReader(dataStore.getCachedData());
+    this.dataStoreRefCount = dataStore.increaseRefCount();
+  }
+
+  @Override
+  public TsBlock next() throws Exception {
+    if (dataReader == null) {
+      return null;
+    }
+    return dataReader.next();
+  }
+
+  @Override
+  public boolean hasNext() throws Exception {
+    if (dataReader == null) {
+      return false;
+    }
+    return dataReader.hasNext();
+  }
+
+  @Override
+  public void close() throws Exception {
+    try {
+      if (dataReader != null) {
+        dataReader.close();
+      }
+    } catch (Exception e) {
+      LOGGER.error("Fail to close fileChannel", e);
+    }
+  }
+
+  @Override
+  public boolean isFinished() throws Exception {
+    return !hasNextWithTimer();
+  }
+
+  @Override
+  public long calculateMaxPeekMemory() {
+    return calculateRetainedSizeAfterCallingNext() + calculateMaxReturnSize();

Review Comment:
   done



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] With clause enhance [iotdb]

Posted by "shizy818 (via GitHub)" <gi...@apache.org>.
shizy818 commented on code in PR #16353:
URL: https://github.com/apache/iotdb/pull/16353#discussion_r2646711269


##########
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/queryengine/common/MPPQueryContext.java:
##########
@@ -103,6 +125,24 @@ public class MPPQueryContext implements IAuditEntity {
 
   private boolean userQuery = false;
 
+  private Map<NodeRef<Table>, Query> cteQueries = new HashMap<>();
+
+  // Stores the EXPLAIN/EXPLAIN ANALYZE results for Common Table Expressions (CTEs)
+  // Key: CTE table reference
+  // Value: Pair containing (max line length of the explain output, list of formatted explain lines)
+  // This ensures consistent formatting between the main query and its CTE sub-queries
+  private final Map<NodeRef<Table>, Pair<Integer, List<String>>> cteExplainResults =
+      new LinkedHashMap<>();
+  // Tracks the materialization time cost (in nanoseconds) for each CTE to help optimize query
+  // planning
+  private final Map<NodeRef<Table>, Long> cteMaterializationCosts = new HashMap<>();
+
+  // Never materialize CTE in a subquery.
+  private boolean subquery = false;

Review Comment:
   done



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org