You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@iotdb.apache.org by "shuwenwei (via GitHub)" <gi...@apache.org> on 2026/10/20 04:18:23 UTC
[PR] [Fix] Support unordered query results in COPY TO TsFile [iotdb]
shuwenwei opened a new pull request, #18928:
URL: https://github.com/apache/iotdb/pull/18928
## Description
COPY TO TsFile currently fails when a SQL query returns timestamps out of order within a target device, for example with `ORDER BY time DESC`. Add an outer sort by the target TAG columns and TIME after planning the complete inner query, so existing optimizers can use streaming scans, eliminate redundant sorts, and merge ordered region results. The inner query's ORDER BY, LIMIT, and OFFSET semantics are preserved.
Both table shorthand and SQL query exports use this path. Exports with generated timestamps retain their existing behavior, and duplicate timestamps for the same target device remain subject to TsFile writer validation. No SQL option is added.
## Validation
- Add planner unit tests covering COPY TO entry points, streaming sort and distributed merge, TopK with OFFSET, aliases and target TAG order, and generated timestamps.
- Add five integration tests covering descending timestamps, field-based ordering, LIMIT/OFFSET, aliased TIME with reordered TAGs, and exports without TAGs. Read back all exported rows and check increasing timestamps per device; use small TsBlocks to exercise multiple batches.
- Set `MEMORY_THRESHOLD 1000000` for the new integration cases, matching existing COPY TO tests.
- All five new integration tests passed according to the user's local validation. The agent did not rerun tests or compilation.
This PR has:
- [x] been self-reviewed.
- [x] added unit tests.
- [x] added integration tests.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Fix] Support unordered query results in COPY TO TsFile [iotdb]
Posted by "JackieTien97 (via GitHub)" <gi...@apache.org>.
JackieTien97 merged PR #18928:
URL: https://github.com/apache/iotdb/pull/18928
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Fix] Support unordered query results in COPY TO TsFile [iotdb]
Posted by "JackieTien97 (via GitHub)" <gi...@apache.org>.
JackieTien97 commented on code in PR #18928:
URL: https://github.com/apache/iotdb/pull/18928#discussion_r4323050487
##########
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/queryengine/plan/relational/planner/TableLogicalPlanner.java:
##########
@@ -656,17 +664,50 @@ private RelationPlan createRelationPlan(final CopyTo statement, final Analysis a
.infer(analysis, innerQueryRelationPlan, innerQueryRespDatasetHeader.getColumnHeaders());
statement.getOptions().check(innerQueryRespDatasetHeader.getColumnHeaders());
+ PlanNode copyToSource = innerQueryRelationPlan.getRoot();
+ if (statement.getOptions() instanceof CopyToTsFileOptions tsFileOptions
+ && !tsFileOptions.isGenerateNewTimeColumn()) {
+ List<ColumnHeader> columnHeaders = innerQueryRespDatasetHeader.getColumnHeaders();
+ List<Symbol> outputSymbols = outputNode.getOutputSymbols();
+ Map<String, Symbol> columnSymbols = new HashMap<>();
+ for (int i = 0; i < columnHeaders.size(); i++) {
+ columnSymbols.put(columnHeaders.get(i).getColumnName(), outputSymbols.get(i));
+ }
+ Map<Symbol, SortOrder> orderings = new LinkedHashMap<>();
+ for (String tagColumn : tsFileOptions.getTargetTagColumns()) {
+ orderings.put(columnSymbols.get(tagColumn), SortOrder.ASC_NULLS_FIRST);
+ }
+ orderings.put(
+ columnSymbols.get(tsFileOptions.getTargetTimeColumn()), SortOrder.ASC_NULLS_FIRST);
+ // Sort the complete inner query result, after its LIMIT and OFFSET.
+ copyToSource =
+ new SortNode(
Review Comment:
[P2] Handle COPY sources without a table scan in the sort optimization
Adding this sort makes previously valid no-FROM exports fail during logical optimization. For example:
```sql
COPY (SELECT CAST(1 AS TIMESTAMP) AS time, 1 AS v)
TO 'constant.tsfile' (MEMORY_THRESHOLD 1000000);
```
This now returns `305 / INTERNAL_SERVER_ERROR` because `TransformSortToStreamSort.visitSort()` dereferences a null `DeviceTableScanNode`. I reproduced this on a live 1C1D instance; `EXPLAIN` and `EXPLAIN ANALYZE` wrappers fail as well, while a comparison using the baseline planner accepts the plain COPY query. Please make the sort optimization handle sources without a device scan, or safely eliminate unnecessary single-row sorts, and add regression coverage.
##########
iotdb-core/relational-grammar/src/main/antlr4/org/apache/iotdb/db/relational/grammar/sql/RelationalSql.g4:
##########
@@ -968,8 +968,8 @@ copyToStatementOption
// ------------------------------------------- Query Statement ---------------------------------------------------------
queryStatement
: query #statementDefault
- | EXPLAIN ('(' FORMAT identifier ')')? (query | executeStatement | executeImmediateStatement) #explain
- | EXPLAIN ANALYZE VERBOSE? ('(' FORMAT identifier ')')? (query | executeStatement | executeImmediateStatement) #explainAnalyze
+ | EXPLAIN ('(' FORMAT identifier ')')? (query | copyToStatement | executeStatement | executeImmediateStatement) #explain
+ | EXPLAIN ANALYZE VERBOSE? ('(' FORMAT identifier ')')? (query | copyToStatement | executeStatement | executeImmediateStatement) #explainAnalyze
Review Comment:
[P2] Update the existing EXPLAIN parser-error assertions
Adding `COPY` to these alternatives changes the expected-token lists in parser errors, but `IoTDBInsertQueryIT.testExplain` still asserts the complete old messages at lines 464 and 476. `EXPLAIN INSERT ...` now reports `Expecting: '(', 'ANALYZE', 'COPY', 'EXECUTE', <query>`, so the first assertion fails; the `EXPLAIN ANALYZE INSERT ...` assertion has the same mismatch. This is already failing the [Windows (2) table-model CI job](https://github.com/apache/iotdb/actions/runs/39385979976/job/118505878007). Please update both assertions, or assert the stable parts of the parser error.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Fix] Support unordered query results in COPY TO TsFile [iotdb]
Posted by "codecov[bot] (via GitHub)" <gi...@apache.org>.
codecov[bot] commented on PR #18928:
URL: https://github.com/apache/iotdb/pull/18928#issuecomment-6286486059
## [Codecov](https://app.codecov.io/gh/apache/iotdb/pull/18928?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
:x: Patch coverage is `47.61905%` with `44 lines` in your changes missing coverage. Please review.
:white_check_mark: Project coverage is 46.85%. Comparing base ([`f028736`](https://app.codecov.io/gh/apache/iotdb/commit/f028736f105fc0c79e56d8e1ce13b6b268e3a077?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`f1ec16b`](https://app.codecov.io/gh/apache/iotdb/commit/f1ec16b8d95bf084c1731d26de7b5c86ffbf1d99?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)).
:warning: Report is 11 commits behind head on master.
| [Files with missing lines](https://app.codecov.io/gh/apache/iotdb/pull/18928?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Patch % | Lines |
|---|---|---|
| [...ngine/plan/planner/plan/node/PlanGraphPrinter.java](https://app.codecov.io/gh/apache/iotdb/pull/18928?src=pr&el=tree&filepath=iotdb-core%2Fdatanode%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fiotdb%2Fdb%2Fqueryengine%2Fplan%2Fplanner%2Fplan%2Fnode%2FPlanGraphPrinter.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aW90ZGItY29yZS9kYXRhbm9kZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvaW90ZGIvZGIvcXVlcnllbmdpbmUvcGxhbi9wbGFubmVyL3BsYW4vbm9kZS9QbGFuR3JhcGhQcmludGVyLmphdmE=) | 0.00% | [24 Missing :warning: ](https://app.codecov.io/gh/apache/iotdb/pull/18928?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...e/plan/planner/plan/node/PlanGraphJsonPrinter.java](https://app.codecov.io/gh/apache/iotdb/pull/18928?src=pr&el=tree&filepath=iotdb-core%2Fdatanode%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fiotdb%2Fdb%2Fqueryengine%2Fplan%2Fplanner%2Fplan%2Fnode%2FPlanGraphJsonPrinter.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aW90ZGItY29yZS9kYXRhbm9kZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvaW90ZGIvZGIvcXVlcnllbmdpbmUvcGxhbi9wbGFubmVyL3BsYW4vbm9kZS9QbGFuR3JhcGhKc29uUHJpbnRlci5qYXZh) | 12.50% | [7 Missing :warning: ](https://app.codecov.io/gh/apache/iotdb/pull/18928?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...n/operator/process/copyto/TableCopyToOperator.java](https://app.codecov.io/gh/apache/iotdb/pull/18928?src=pr&el=tree&filepath=iotdb-core%2Fdatanode%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fiotdb%2Fdb%2Fqueryengine%2Fexecution%2Foperator%2Fprocess%2Fcopyto%2FTableCopyToOperator.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aW90ZGItY29yZS9kYXRhbm9kZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvaW90ZGIvZGIvcXVlcnllbmdpbmUvZXhlY3V0aW9uL29wZXJhdG9yL3Byb2Nlc3MvY29weXRvL1RhYmxlQ29weVRvT3BlcmF0b3IuamF2YQ==) | 0.00% | [4 Missing :warning: ](https://app.codecov.io/gh/apache/iotdb/pull/18928?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...yengine/plan/relational/sql/parser/AstBuilder.java](https://app.codecov.io/gh/apache/iotdb/pull/18928?src=pr&el=tree&filepath=iotdb-core%2Fdatanode%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fiotdb%2Fdb%2Fqueryengine%2Fplan%2Frelational%2Fsql%2Fparser%2FAstBuilder.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aW90ZGItY29yZS9kYXRhbm9kZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvaW90ZGIvZGIvcXVlcnllbmdpbmUvcGxhbi9yZWxhdGlvbmFsL3NxbC9wYXJzZXIvQXN0QnVpbGRlci5qYXZh) | 0.00% | [4 Missing :warning: ](https://app.codecov.io/gh/apache/iotdb/pull/18928?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...ne/plan/relational/analyzer/StatementAnalyzer.java](https://app.codecov.io/gh/apache/iotdb/pull/18928?src=pr&el=tree&filepath=iotdb-core%2Fdatanode%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fiotdb%2Fdb%2Fqueryengine%2Fplan%2Frelational%2Fanalyzer%2FStatementAnalyzer.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aW90ZGItY29yZS9kYXRhbm9kZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvaW90ZGIvZGIvcXVlcnllbmdpbmUvcGxhbi9yZWxhdGlvbmFsL2FuYWx5emVyL1N0YXRlbWVudEFuYWx5emVyLmphdmE=) | 50.00% | [3 Missing :warning: ](https://app.codecov.io/gh/apache/iotdb/pull/18928?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...e/plan/relational/planner/TableLogicalPlanner.java](https://app.codecov.io/gh/apache/iotdb/pull/18928?src=pr&el=tree&filepath=iotdb-core%2Fdatanode%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fiotdb%2Fdb%2Fqueryengine%2Fplan%2Frelational%2Fplanner%2FTableLogicalPlanner.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aW90ZGItY29yZS9kYXRhbm9kZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvaW90ZGIvZGIvcXVlcnllbmdpbmUvcGxhbi9yZWxhdGlvbmFsL3BsYW5uZXIvVGFibGVMb2dpY2FsUGxhbm5lci5qYXZh) | 93.93% | [2 Missing :warning: ](https://app.codecov.io/gh/apache/iotdb/pull/18928?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
<details><summary>Additional details and impacted files</summary>
```diff
@@ Coverage Diff @@
## master #18928 +/- ##
============================================
+ Coverage 46.42% 46.85% +0.43%
Complexity 713 713
============================================
Files 5537 5537
Lines 404332 404719 +387
Branches 52852 52915 +63
============================================
+ Hits 187717 189647 +1930
+ Misses 216615 215072 -1543
```
</details>
[:umbrella: View full report in Codecov by Harness](https://app.codecov.io/gh/apache/iotdb/pull/18928?dropdown=coverage&src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
:loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
<details><summary> :rocket: New features to boost your workflow: </summary>
- :snowflake: [Test Analytics](https://docs.codecov.com/docs/test-analytics): Detect flaky tests, report on failures, and find test suite problems.
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org