You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@iotdb.apache.org by "Caideyipi (via GitHub)" <gi...@apache.org> on 2026/07/02 02:46:59 UTC
[PR] Pipe: order historical TsFiles by flush time [iotdb]
Caideyipi opened a new pull request, #18088:
URL: https://github.com/apache/iotdb/pull/18088
## Description
This PR adds a new historical pipe source option:
- `extractor.history.tsfile.order-by-flush-time`
- `source.history.tsfile.order-by-flush-time`
- default: `true`
When enabled, historical TsFile-only extraction sends selected TsFiles in source-side file creation / flush-time order instead of progressIndex order. The goal is to preserve overwrite semantics for duplicated timestamps: older TsFiles are transferred first, and newer TsFiles are transferred later so the receiver can overwrite older values with newer values.
## Semantic Changes
For historical TsFile extraction where insertions are captured and deletions are not captured:
1. Historical working TsFile processors are closed synchronously before extraction, so the source has a stable set of TsFiles to order.
2. Selected TsFiles are sorted by:
- sequence files before unsequence files,
- file-name timestamp / flush time ascending,
- file version ascending,
- compaction version ascending,
- file path as a deterministic tie-breaker.
3. TsFiles filtered out by progress, time, path, delete status, pipe-generated status, or pin failure are removed before computing delayed progress. Their progressIndex is not reported by this historical flush-time path.
4. Because flush-time order is not guaranteed to be compatible with progressIndex topological order, per-TsFile commit progress reporting is disabled for reordered historical TsFile events.
5. Tablet events generated from those TsFile events inherit the no-report behavior, so decomposing a TsFile into tablets does not accidentally advance progress early.
6. After all selected reordered historical TsFiles are supplied, the source emits one `ProgressReportEvent` with the max progressIndex of the selected resources.
When historical deletions are captured together with insertions, the source keeps the previous progressIndex ordering. Deletion resources only carry progressIndex ordering information, so this avoids changing insertion/deletion ordering semantics.
The option can be set to `false` to keep the previous progressIndex-based ordering.
## Tests / Coverage
Added unit coverage for:
- the new option defaulting to `true`,
- sorting older flush-time TsFiles before newer flush-time TsFiles,
- explicitly disabling the new option and falling back to progressIndex order,
- delaying progress reporting until all reordered historical resources are consumed,
- excluding filtered-out TsFiles from the delayed max progressIndex,
- preserving the no-progress-report flag when a TsFile event is shallow-copied,
- making generated tablet events inherit the source TsFile event's no-progress-report behavior.
I did not add an integration test because ordinary end-to-end write/flush scenarios usually produce progressIndex order that matches flush-time order, so such an IT would not deterministically cover the regression. The deterministic behavior that matters here is the historical source's resource ordering and progress reporting contract, which is covered by unit tests with explicit TsFileResource ordering and progressIndex setup.
Local verification:
- `mvn spotless:apply -pl iotdb-core/datanode`
- `mvn spotless:apply -pl iotdb-core/node-commons`
- `mvn "-Ddevelocity.off=true" -o install -pl iotdb-core/node-commons "-DskipTests=true" "-Dcheckstyle.skip=true" "-Dspotless.check.skip=true"`
- `git diff --check origin/master..HEAD`
Attempted datanode targeted UT execution locally, but the module currently fails during main compilation in this checkout before reaching the test phase due to unrelated generated-source / thrift cache errors such as missing `TShowRepairDataPartitionTableProgressResp`, `IFill`, and `Accumulator` classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Pipe: order historical TsFiles by query priority [iotdb]
Posted by "Caideyipi (via GitHub)" <gi...@apache.org>.
Caideyipi commented on code in PR #18088:
URL: https://github.com/apache/iotdb/pull/18088#discussion_r3533498001
##########
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/pipe/source/dataregion/historical/PipeHistoricalDataRegionTsFileAndDeletionSource.java:
##########
@@ -881,27 +987,36 @@ public synchronized Event supply() {
return null;
}
- final PersistentResource resource = pendingQueue.peek();
- if (resource == null) {
- return supplyTerminateEvent();
- }
+ while (true) {
+ final PersistentResource resource = pendingQueue.peek();
+ if (resource == null) {
+ if (shouldReportMaxHistoricalProgressIndex) {
+ shouldReportMaxHistoricalProgressIndex = false;
+ return supplyProgressReportEvent(maxHistoricalProgressIndex);
+ }
+ return supplyTerminateEvent();
+ }
+
+ if (resource instanceof TsFileResource) {
+ final TsFileResource tsFileResource = (TsFileResource) resource;
+ if (consumeSkippedHistoricalTsFileEventIfNecessary(tsFileResource)) {
+ clearReplicateIndexForResource(tsFileResource);
+ pendingQueue.poll();
+ if (shouldUseHistoricalTsFileQueryPriorityOrder()) {
+ continue;
+ }
+ return supplyProgressReportEvent(tsFileResource.getMaxProgressIndex());
+ }
Review Comment:
Done, thanks for the suggestion.
I changed the query-priority path to precompute which TsFiles can safely report their own progress: a TsFile reports progress only when that progress index will not cover any later pending resource. For the `[1, 2, 4, 3, 5]` case, this reports `1`, `2`, `3`, and `5` incrementally, while still keeping the final max-progress report as a fallback after all historical TsFiles are supplied.
Added unit coverage for the incremental-safe-progress selection and for supplying the delayed `ProgressReportEvent` after a safe TsFile event.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Pipe: order historical TsFiles by query priority [iotdb]
Posted by "jt2594838 (via GitHub)" <gi...@apache.org>.
jt2594838 commented on code in PR #18088:
URL: https://github.com/apache/iotdb/pull/18088#discussion_r3526181047
##########
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/pipe/source/dataregion/historical/PipeHistoricalDataRegionTsFileAndDeletionSource.java:
##########
@@ -881,27 +987,36 @@ public synchronized Event supply() {
return null;
}
- final PersistentResource resource = pendingQueue.peek();
- if (resource == null) {
- return supplyTerminateEvent();
- }
+ while (true) {
+ final PersistentResource resource = pendingQueue.peek();
+ if (resource == null) {
+ if (shouldReportMaxHistoricalProgressIndex) {
+ shouldReportMaxHistoricalProgressIndex = false;
+ return supplyProgressReportEvent(maxHistoricalProgressIndex);
+ }
+ return supplyTerminateEvent();
+ }
+
+ if (resource instanceof TsFileResource) {
+ final TsFileResource tsFileResource = (TsFileResource) resource;
+ if (consumeSkippedHistoricalTsFileEventIfNecessary(tsFileResource)) {
+ clearReplicateIndexForResource(tsFileResource);
+ pendingQueue.poll();
+ if (shouldUseHistoricalTsFileQueryPriorityOrder()) {
+ continue;
+ }
+ return supplyProgressReportEvent(tsFileResource.getMaxProgressIndex());
+ }
Review Comment:
The historical task will only report progress once, which means if the pipe is restarted, all files will be retransmitted?
The cost is just too high. Maybe we can find some resources that can still be reported after processing.
For example, if we use a simple integer to represent the progressIndex of each TsFile, and we have 5 TsFiles ordered by the query order:
[1, 2, 4, 3, 5]
Then we can report progress on TsFile 1, 2, 3, 5, because for these files, we know that there is no file that is behind them and has a smaller index than them.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Pipe: order historical TsFiles by flush time [iotdb]
Posted by "codecov[bot] (via GitHub)" <gi...@apache.org>.
codecov[bot] commented on PR #18088:
URL: https://github.com/apache/iotdb/pull/18088#issuecomment-4861947288
## [Codecov](https://app.codecov.io/gh/apache/iotdb/pull/18088?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
:x: Patch coverage is `75.24752%` with `25 lines` in your changes missing coverage. Please review.
:white_check_mark: Project coverage is 41.66%. Comparing base ([`220e7a3`](https://app.codecov.io/gh/apache/iotdb/commit/220e7a319c283deeaa27a25bccdad82a2d2e18db?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`3a32f1e`](https://app.codecov.io/gh/apache/iotdb/commit/3a32f1ea9653af3b9a8b7c3383148d9759492af8?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)).
| [Files with missing lines](https://app.codecov.io/gh/apache/iotdb/pull/18088?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Patch % | Lines |
|---|---|---|
| [...peHistoricalDataRegionTsFileAndDeletionSource.java](https://app.codecov.io/gh/apache/iotdb/pull/18088?src=pr&el=tree&filepath=iotdb-core%2Fdatanode%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fiotdb%2Fdb%2Fpipe%2Fsource%2Fdataregion%2Fhistorical%2FPipeHistoricalDataRegionTsFileAndDeletionSource.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aW90ZGItY29yZS9kYXRhbm9kZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvaW90ZGIvZGIvcGlwZS9zb3VyY2UvZGF0YXJlZ2lvbi9oaXN0b3JpY2FsL1BpcGVIaXN0b3JpY2FsRGF0YVJlZ2lvblRzRmlsZUFuZERlbGV0aW9uU291cmNlLmphdmE=) | 70.51% | [23 Missing :warning: ](https://app.codecov.io/gh/apache/iotdb/pull/18088?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [.../event/common/tsfile/PipeTsFileInsertionEvent.java](https://app.codecov.io/gh/apache/iotdb/pull/18088?src=pr&el=tree&filepath=iotdb-core%2Fdatanode%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fiotdb%2Fdb%2Fpipe%2Fevent%2Fcommon%2Ftsfile%2FPipeTsFileInsertionEvent.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aW90ZGItY29yZS9kYXRhbm9kZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvaW90ZGIvZGIvcGlwZS9ldmVudC9jb21tb24vdHNmaWxlL1BpcGVUc0ZpbGVJbnNlcnRpb25FdmVudC5qYXZh) | 77.77% | [2 Missing :warning: ](https://app.codecov.io/gh/apache/iotdb/pull/18088?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
<details><summary>Additional details and impacted files</summary>
```diff
@@ Coverage Diff @@
## master #18088 +/- ##
=========================================
Coverage 41.65% 41.66%
Complexity 318 318
=========================================
Files 5296 5296
Lines 371663 371744 +81
Branches 48088 48103 +15
=========================================
+ Hits 154819 154889 +70
- Misses 216844 216855 +11
```
</details>
[:umbrella: View full report in Codecov by Harness](https://app.codecov.io/gh/apache/iotdb/pull/18088?dropdown=coverage&src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
:loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
<details><summary> :rocket: New features to boost your workflow: </summary>
- :snowflake: [Test Analytics](https://docs.codecov.com/docs/test-analytics): Detect flaky tests, report on failures, and find test suite problems.
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Pipe: order historical TsFiles by query priority [iotdb]
Posted by "sonarqubecloud[bot] (via GitHub)" <gi...@apache.org>.
sonarqubecloud[bot] commented on PR #18088:
URL: https://github.com/apache/iotdb/pull/18088#issuecomment-4899999665
## [](https://sonarcloud.io/dashboard?id=apache_iotdb&pullRequest=18088) **Quality Gate passed**
Issues
 [5 New issues](https://sonarcloud.io/project/issues?id=apache_iotdb&pullRequest=18088&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0 Accepted issues](https://sonarcloud.io/project/issues?id=apache_iotdb&pullRequest=18088&issueStatuses=ACCEPTED)
Measures
 [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_iotdb&pullRequest=18088&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0.0% Coverage on New Code](https://sonarcloud.io/component_measures?id=apache_iotdb&pullRequest=18088&metric=new_coverage&view=list)
 [0.0% Duplication on New Code](https://sonarcloud.io/component_measures?id=apache_iotdb&pullRequest=18088&metric=new_duplicated_lines_density&view=list)
<!-- sqra-placement-anchor -->
[See analysis details on SonarQube Cloud](https://sonarcloud.io/dashboard?id=apache_iotdb&pullRequest=18088)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Pipe: order historical TsFiles by flush time [iotdb]
Posted by "sonarqubecloud[bot] (via GitHub)" <gi...@apache.org>.
sonarqubecloud[bot] commented on PR #18088:
URL: https://github.com/apache/iotdb/pull/18088#issuecomment-4861757029
## [](https://sonarcloud.io/dashboard?id=apache_iotdb&pullRequest=18088) **Quality Gate passed**
Issues
 [5 New issues](https://sonarcloud.io/project/issues?id=apache_iotdb&pullRequest=18088&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0 Accepted issues](https://sonarcloud.io/project/issues?id=apache_iotdb&pullRequest=18088&issueStatuses=ACCEPTED)
Measures
 [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_iotdb&pullRequest=18088&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0.0% Coverage on New Code](https://sonarcloud.io/component_measures?id=apache_iotdb&pullRequest=18088&metric=new_coverage&view=list)
 [0.0% Duplication on New Code](https://sonarcloud.io/component_measures?id=apache_iotdb&pullRequest=18088&metric=new_duplicated_lines_density&view=list)
<!-- sqra-placement-anchor -->
[See analysis details on SonarQube Cloud](https://sonarcloud.io/dashboard?id=apache_iotdb&pullRequest=18088)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Pipe: order historical TsFiles by query priority [iotdb]
Posted by "sonarqubecloud[bot] (via GitHub)" <gi...@apache.org>.
sonarqubecloud[bot] commented on PR #18088:
URL: https://github.com/apache/iotdb/pull/18088#issuecomment-4903172738
## [](https://sonarcloud.io/dashboard?id=apache_iotdb&pullRequest=18088) **Quality Gate passed**
Issues
 [16 New issues](https://sonarcloud.io/project/issues?id=apache_iotdb&pullRequest=18088&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0 Accepted issues](https://sonarcloud.io/project/issues?id=apache_iotdb&pullRequest=18088&issueStatuses=ACCEPTED)
Measures
 [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_iotdb&pullRequest=18088&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0.0% Coverage on New Code](https://sonarcloud.io/component_measures?id=apache_iotdb&pullRequest=18088&metric=new_coverage&view=list)
 [0.0% Duplication on New Code](https://sonarcloud.io/component_measures?id=apache_iotdb&pullRequest=18088&metric=new_duplicated_lines_density&view=list)
<!-- sqra-placement-anchor -->
[See analysis details on SonarQube Cloud](https://sonarcloud.io/dashboard?id=apache_iotdb&pullRequest=18088)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Pipe: order historical TsFiles by query priority [iotdb]
Posted by "jt2594838 (via GitHub)" <gi...@apache.org>.
jt2594838 merged PR #18088:
URL: https://github.com/apache/iotdb/pull/18088
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org