You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@iotdb.apache.org by "shuwenwei (via GitHub)" <gi...@apache.org> on 2026/10/20 04:18:23 UTC

[PR] [Fix] Support unordered query results in COPY TO TsFile [iotdb]

shuwenwei opened a new pull request, #18928:
URL: https://github.com/apache/iotdb/pull/18928

   ## Description
   
   COPY TO TsFile currently fails when a SQL query returns timestamps out of order within a target device, for example with `ORDER BY time DESC`. Add an outer sort by the target TAG columns and TIME after planning the complete inner query, so existing optimizers can use streaming scans, eliminate redundant sorts, and merge ordered region results. The inner query's ORDER BY, LIMIT, and OFFSET semantics are preserved.
   
   Both table shorthand and SQL query exports use this path. Exports with generated timestamps retain their existing behavior, and duplicate timestamps for the same target device remain subject to TsFile writer validation. No SQL option is added.
   
   ## Validation
   
   - Add planner unit tests covering COPY TO entry points, streaming sort and distributed merge, TopK with OFFSET, aliases and target TAG order, and generated timestamps.
   - Add five integration tests covering descending timestamps, field-based ordering, LIMIT/OFFSET, aliased TIME with reordered TAGs, and exports without TAGs. Read back all exported rows and check increasing timestamps per device; use small TsBlocks to exercise multiple batches.
   - Set `MEMORY_THRESHOLD 1000000` for the new integration cases, matching existing COPY TO tests.
   - All five new integration tests passed according to the user's local validation. The agent did not rerun tests or compilation.
   
   This PR has:
   - [x] been self-reviewed.
   - [x] added unit tests.
   - [x] added integration tests.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Fix] Support unordered query results in COPY TO TsFile [iotdb]

Posted by "JackieTien97 (via GitHub)" <gi...@apache.org>.
JackieTien97 merged PR #18928:
URL: https://github.com/apache/iotdb/pull/18928


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Fix] Support unordered query results in COPY TO TsFile [iotdb]

Posted by "JackieTien97 (via GitHub)" <gi...@apache.org>.
JackieTien97 commented on code in PR #18928:
URL: https://github.com/apache/iotdb/pull/18928#discussion_r4323050487


##########
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/queryengine/plan/relational/planner/TableLogicalPlanner.java:
##########
@@ -656,17 +664,50 @@ private RelationPlan createRelationPlan(final CopyTo statement, final Analysis a
         .infer(analysis, innerQueryRelationPlan, innerQueryRespDatasetHeader.getColumnHeaders());
     statement.getOptions().check(innerQueryRespDatasetHeader.getColumnHeaders());
 
+    PlanNode copyToSource = innerQueryRelationPlan.getRoot();
+    if (statement.getOptions() instanceof CopyToTsFileOptions tsFileOptions
+        && !tsFileOptions.isGenerateNewTimeColumn()) {
+      List<ColumnHeader> columnHeaders = innerQueryRespDatasetHeader.getColumnHeaders();
+      List<Symbol> outputSymbols = outputNode.getOutputSymbols();
+      Map<String, Symbol> columnSymbols = new HashMap<>();
+      for (int i = 0; i < columnHeaders.size(); i++) {
+        columnSymbols.put(columnHeaders.get(i).getColumnName(), outputSymbols.get(i));
+      }
+      Map<Symbol, SortOrder> orderings = new LinkedHashMap<>();
+      for (String tagColumn : tsFileOptions.getTargetTagColumns()) {
+        orderings.put(columnSymbols.get(tagColumn), SortOrder.ASC_NULLS_FIRST);
+      }
+      orderings.put(
+          columnSymbols.get(tsFileOptions.getTargetTimeColumn()), SortOrder.ASC_NULLS_FIRST);
+      // Sort the complete inner query result, after its LIMIT and OFFSET.
+      copyToSource =
+          new SortNode(

Review Comment:
   [P2] Handle COPY sources without a table scan in the sort optimization
   
   Adding this sort makes previously valid no-FROM exports fail during logical optimization. For example:
   
   ```sql
   COPY (SELECT CAST(1 AS TIMESTAMP) AS time, 1 AS v)
   TO 'constant.tsfile' (MEMORY_THRESHOLD 1000000);
   ```
   
   This now returns `305 / INTERNAL_SERVER_ERROR` because `TransformSortToStreamSort.visitSort()` dereferences a null `DeviceTableScanNode`. I reproduced this on a live 1C1D instance; `EXPLAIN` and `EXPLAIN ANALYZE` wrappers fail as well, while a comparison using the baseline planner accepts the plain COPY query. Please make the sort optimization handle sources without a device scan, or safely eliminate unnecessary single-row sorts, and add regression coverage.



##########
iotdb-core/relational-grammar/src/main/antlr4/org/apache/iotdb/db/relational/grammar/sql/RelationalSql.g4:
##########
@@ -968,8 +968,8 @@ copyToStatementOption
 // ------------------------------------------- Query Statement ---------------------------------------------------------
 queryStatement
     : query                                                        #statementDefault
-    | EXPLAIN ('(' FORMAT identifier ')')? (query | executeStatement | executeImmediateStatement) #explain
-    | EXPLAIN ANALYZE VERBOSE? ('(' FORMAT identifier ')')? (query | executeStatement | executeImmediateStatement) #explainAnalyze
+    | EXPLAIN ('(' FORMAT identifier ')')? (query | copyToStatement | executeStatement | executeImmediateStatement) #explain
+    | EXPLAIN ANALYZE VERBOSE? ('(' FORMAT identifier ')')? (query | copyToStatement | executeStatement | executeImmediateStatement) #explainAnalyze

Review Comment:
   [P2] Update the existing EXPLAIN parser-error assertions
   
   Adding `COPY` to these alternatives changes the expected-token lists in parser errors, but `IoTDBInsertQueryIT.testExplain` still asserts the complete old messages at lines 464 and 476. `EXPLAIN INSERT ...` now reports `Expecting: '(', 'ANALYZE', 'COPY', 'EXECUTE', <query>`, so the first assertion fails; the `EXPLAIN ANALYZE INSERT ...` assertion has the same mismatch. This is already failing the [Windows (2) table-model CI job](https://github.com/apache/iotdb/actions/runs/39385979976/job/118505878007). Please update both assertions, or assert the stable parts of the parser error.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Fix] Support unordered query results in COPY TO TsFile [iotdb]

Posted by "codecov[bot] (via GitHub)" <gi...@apache.org>.
codecov[bot] commented on PR #18928:
URL: https://github.com/apache/iotdb/pull/18928#issuecomment-6286486059

   ## [Codecov](https://app.codecov.io/gh/apache/iotdb/pull/18928?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
   :x: Patch coverage is `47.61905%` with `44 lines` in your changes missing coverage. Please review.
   :white_check_mark: Project coverage is 46.85%. Comparing base ([`f028736`](https://app.codecov.io/gh/apache/iotdb/commit/f028736f105fc0c79e56d8e1ce13b6b268e3a077?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`f1ec16b`](https://app.codecov.io/gh/apache/iotdb/commit/f1ec16b8d95bf084c1731d26de7b5c86ffbf1d99?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)).
   :warning: Report is 11 commits behind head on master.
   
   | [Files with missing lines](https://app.codecov.io/gh/apache/iotdb/pull/18928?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Patch % | Lines |
   |---|---|---|
   | [...ngine/plan/planner/plan/node/PlanGraphPrinter.java](https://app.codecov.io/gh/apache/iotdb/pull/18928?src=pr&el=tree&filepath=iotdb-core%2Fdatanode%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fiotdb%2Fdb%2Fqueryengine%2Fplan%2Fplanner%2Fplan%2Fnode%2FPlanGraphPrinter.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aW90ZGItY29yZS9kYXRhbm9kZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvaW90ZGIvZGIvcXVlcnllbmdpbmUvcGxhbi9wbGFubmVyL3BsYW4vbm9kZS9QbGFuR3JhcGhQcmludGVyLmphdmE=) | 0.00% | [24 Missing :warning: ](https://app.codecov.io/gh/apache/iotdb/pull/18928?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
   | [...e/plan/planner/plan/node/PlanGraphJsonPrinter.java](https://app.codecov.io/gh/apache/iotdb/pull/18928?src=pr&el=tree&filepath=iotdb-core%2Fdatanode%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fiotdb%2Fdb%2Fqueryengine%2Fplan%2Fplanner%2Fplan%2Fnode%2FPlanGraphJsonPrinter.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aW90ZGItY29yZS9kYXRhbm9kZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvaW90ZGIvZGIvcXVlcnllbmdpbmUvcGxhbi9wbGFubmVyL3BsYW4vbm9kZS9QbGFuR3JhcGhKc29uUHJpbnRlci5qYXZh) | 12.50% | [7 Missing :warning: ](https://app.codecov.io/gh/apache/iotdb/pull/18928?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
   | [...n/operator/process/copyto/TableCopyToOperator.java](https://app.codecov.io/gh/apache/iotdb/pull/18928?src=pr&el=tree&filepath=iotdb-core%2Fdatanode%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fiotdb%2Fdb%2Fqueryengine%2Fexecution%2Foperator%2Fprocess%2Fcopyto%2FTableCopyToOperator.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aW90ZGItY29yZS9kYXRhbm9kZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvaW90ZGIvZGIvcXVlcnllbmdpbmUvZXhlY3V0aW9uL29wZXJhdG9yL3Byb2Nlc3MvY29weXRvL1RhYmxlQ29weVRvT3BlcmF0b3IuamF2YQ==) | 0.00% | [4 Missing :warning: ](https://app.codecov.io/gh/apache/iotdb/pull/18928?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
   | [...yengine/plan/relational/sql/parser/AstBuilder.java](https://app.codecov.io/gh/apache/iotdb/pull/18928?src=pr&el=tree&filepath=iotdb-core%2Fdatanode%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fiotdb%2Fdb%2Fqueryengine%2Fplan%2Frelational%2Fsql%2Fparser%2FAstBuilder.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aW90ZGItY29yZS9kYXRhbm9kZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvaW90ZGIvZGIvcXVlcnllbmdpbmUvcGxhbi9yZWxhdGlvbmFsL3NxbC9wYXJzZXIvQXN0QnVpbGRlci5qYXZh) | 0.00% | [4 Missing :warning: ](https://app.codecov.io/gh/apache/iotdb/pull/18928?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
   | [...ne/plan/relational/analyzer/StatementAnalyzer.java](https://app.codecov.io/gh/apache/iotdb/pull/18928?src=pr&el=tree&filepath=iotdb-core%2Fdatanode%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fiotdb%2Fdb%2Fqueryengine%2Fplan%2Frelational%2Fanalyzer%2FStatementAnalyzer.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aW90ZGItY29yZS9kYXRhbm9kZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvaW90ZGIvZGIvcXVlcnllbmdpbmUvcGxhbi9yZWxhdGlvbmFsL2FuYWx5emVyL1N0YXRlbWVudEFuYWx5emVyLmphdmE=) | 50.00% | [3 Missing :warning: ](https://app.codecov.io/gh/apache/iotdb/pull/18928?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
   | [...e/plan/relational/planner/TableLogicalPlanner.java](https://app.codecov.io/gh/apache/iotdb/pull/18928?src=pr&el=tree&filepath=iotdb-core%2Fdatanode%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fiotdb%2Fdb%2Fqueryengine%2Fplan%2Frelational%2Fplanner%2FTableLogicalPlanner.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aW90ZGItY29yZS9kYXRhbm9kZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvaW90ZGIvZGIvcXVlcnllbmdpbmUvcGxhbi9yZWxhdGlvbmFsL3BsYW5uZXIvVGFibGVMb2dpY2FsUGxhbm5lci5qYXZh) | 93.93% | [2 Missing :warning: ](https://app.codecov.io/gh/apache/iotdb/pull/18928?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
   
   <details><summary>Additional details and impacted files</summary>
   
   
   
   ```diff
   @@             Coverage Diff              @@
   ##             master   #18928      +/-   ##
   ============================================
   + Coverage     46.42%   46.85%   +0.43%     
     Complexity      713      713              
   ============================================
     Files          5537     5537              
     Lines        404332   404719     +387     
     Branches      52852    52915      +63     
   ============================================
   + Hits         187717   189647    +1930     
   + Misses       216615   215072    -1543     
   ```
   </details>
   
   [:umbrella: View full report in Codecov by Harness](https://app.codecov.io/gh/apache/iotdb/pull/18928?dropdown=coverage&src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).   
   :loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
   <details><summary> :rocket: New features to boost your workflow: </summary>
   
   - :snowflake: [Test Analytics](https://docs.codecov.com/docs/test-analytics): Detect flaky tests, report on failures, and find test suite problems.
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org