You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@iotdb.apache.org by "shuwenwei (via GitHub)" <gi...@apache.org> on 2026/02/10 02:23:17 UTC
[PR] Fixed concurrency issues caused by write and flush sorting during query execution [iotdb]
shuwenwei opened a new pull request, #17193:
URL: https://github.com/apache/iotdb/pull/17193
## Description
Fixed concurrency issues caused by write and flush sorting during query execution
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Fixed concurrency issues caused by write and flush sorting during query execution [iotdb]
Posted by "sonarqubecloud[bot] (via GitHub)" <gi...@apache.org>.
sonarqubecloud[bot] commented on PR #17193:
URL: https://github.com/apache/iotdb/pull/17193#issuecomment-3875269205
## [](https://sonarcloud.io/dashboard?id=apache_iotdb&pullRequest=17193) **Quality Gate passed**
Issues
 [11 New issues](https://sonarcloud.io/project/issues?id=apache_iotdb&pullRequest=17193&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0 Accepted issues](https://sonarcloud.io/project/issues?id=apache_iotdb&pullRequest=17193&issueStatuses=ACCEPTED)
Measures
 [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_iotdb&pullRequest=17193&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0.0% Coverage on New Code](https://sonarcloud.io/component_measures?id=apache_iotdb&pullRequest=17193&metric=new_coverage&view=list)
 [0.0% Duplication on New Code](https://sonarcloud.io/component_measures?id=apache_iotdb&pullRequest=17193&metric=new_duplicated_lines_density&view=list)
[See analysis details on SonarQube Cloud](https://sonarcloud.io/dashboard?id=apache_iotdb&pullRequest=17193)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Fixed concurrency issues caused by write and flush sorting during query execution [iotdb]
Posted by "sonarqubecloud[bot] (via GitHub)" <gi...@apache.org>.
sonarqubecloud[bot] commented on PR #17193:
URL: https://github.com/apache/iotdb/pull/17193#issuecomment-3875020365
## [](https://sonarcloud.io/dashboard?id=apache_iotdb&pullRequest=17193) **Quality Gate passed**
Issues
 [11 New issues](https://sonarcloud.io/project/issues?id=apache_iotdb&pullRequest=17193&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0 Accepted issues](https://sonarcloud.io/project/issues?id=apache_iotdb&pullRequest=17193&issueStatuses=ACCEPTED)
Measures
 [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_iotdb&pullRequest=17193&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0.0% Coverage on New Code](https://sonarcloud.io/component_measures?id=apache_iotdb&pullRequest=17193&metric=new_coverage&view=list)
 [0.0% Duplication on New Code](https://sonarcloud.io/component_measures?id=apache_iotdb&pullRequest=17193&metric=new_duplicated_lines_density&view=list)
[See analysis details on SonarQube Cloud](https://sonarcloud.io/dashboard?id=apache_iotdb&pullRequest=17193)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Fixed concurrency issues caused by write and flush sorting during query execution [iotdb]
Posted by "sonarqubecloud[bot] (via GitHub)" <gi...@apache.org>.
sonarqubecloud[bot] commented on PR #17193:
URL: https://github.com/apache/iotdb/pull/17193#issuecomment-3882598172
## [](https://sonarcloud.io/dashboard?id=apache_iotdb&pullRequest=17193) **Quality Gate passed**
Issues
 [11 New issues](https://sonarcloud.io/project/issues?id=apache_iotdb&pullRequest=17193&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0 Accepted issues](https://sonarcloud.io/project/issues?id=apache_iotdb&pullRequest=17193&issueStatuses=ACCEPTED)
Measures
 [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_iotdb&pullRequest=17193&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0.0% Coverage on New Code](https://sonarcloud.io/component_measures?id=apache_iotdb&pullRequest=17193&metric=new_coverage&view=list)
 [0.0% Duplication on New Code](https://sonarcloud.io/component_measures?id=apache_iotdb&pullRequest=17193&metric=new_duplicated_lines_density&view=list)
[See analysis details on SonarQube Cloud](https://sonarcloud.io/dashboard?id=apache_iotdb&pullRequest=17193)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Fixed concurrency issues caused by write and flush sorting during query execution [iotdb]
Posted by "JackieTien97 (via GitHub)" <gi...@apache.org>.
JackieTien97 commented on code in PR #17193:
URL: https://github.com/apache/iotdb/pull/17193#discussion_r2791301496
##########
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/storageengine/dataregion/memtable/AlignedWritableMemChunk.java:
##########
@@ -557,6 +550,8 @@ public void encodeWorkingAlignedTVList(
long maxNumberOfPointsInChunk,
int maxNumberOfPointsInPage) {
BitMap allValueColDeletedMap;
+ AlignedTVList list = (AlignedTVList) listForFlushSort;
Review Comment:
Change all used places to listForFlushSort instead of hiding the `list` field.
##########
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/utils/datastructure/AlignedTVList.java:
##########
@@ -165,6 +165,18 @@ public TVList getTvListByColumnIndex(
return alignedTvList;
}
+ @Override
+ public AlignedTVList cloneForFlushSort() {
Review Comment:
```suggestion
public synchronized AlignedTVList cloneForFlushSort() {
```
##########
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/storageengine/dataregion/memtable/AbstractWritableMemChunk.java:
##########
@@ -198,7 +200,41 @@ public abstract void writeAlignedTablet(
public abstract IMeasurementSchema getSchema();
@Override
- public abstract void sortTvListForFlush();
+ public synchronized void sortTvListForFlush() {
Review Comment:
why we need `synchronized` here? Will sort for one memtable be parallel?
##########
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/storageengine/dataregion/memtable/AbstractWritableMemChunk.java:
##########
@@ -198,7 +200,41 @@ public abstract void writeAlignedTablet(
public abstract IMeasurementSchema getSchema();
@Override
- public abstract void sortTvListForFlush();
+ public synchronized void sortTvListForFlush() {
Review Comment:
including the encode method
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Fixed concurrency issues caused by write and flush sorting during query execution [iotdb]
Posted by "JackieTien97 (via GitHub)" <gi...@apache.org>.
JackieTien97 merged PR #17193:
URL: https://github.com/apache/iotdb/pull/17193
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Fixed concurrency issues caused by write and flush sorting during query execution [iotdb]
Posted by "codecov[bot] (via GitHub)" <gi...@apache.org>.
codecov[bot] commented on PR #17193:
URL: https://github.com/apache/iotdb/pull/17193#issuecomment-3875153320
## [Codecov](https://app.codecov.io/gh/apache/iotdb/pull/17193?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
:x: Patch coverage is `42.59259%` with `31 lines` in your changes missing coverage. Please review.
:white_check_mark: Project coverage is 39.50%. Comparing base ([`861d3ee`](https://app.codecov.io/gh/apache/iotdb/commit/861d3eec35b216b8d3cb97beef7e627d024c58e6?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`a0d71bd`](https://app.codecov.io/gh/apache/iotdb/commit/a0d71bdbdb21203970750e260dc917ae22224698?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)).
:warning: Report is 5 commits behind head on master.
| [Files with missing lines](https://app.codecov.io/gh/apache/iotdb/pull/17193?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Patch % | Lines |
|---|---|---|
| [...che/iotdb/db/utils/datastructure/BinaryTVList.java](https://app.codecov.io/gh/apache/iotdb/pull/17193?src=pr&el=tree&filepath=iotdb-core%2Fdatanode%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fiotdb%2Fdb%2Futils%2Fdatastructure%2FBinaryTVList.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aW90ZGItY29yZS9kYXRhbm9kZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvaW90ZGIvZGIvdXRpbHMvZGF0YXN0cnVjdHVyZS9CaW5hcnlUVkxpc3QuamF2YQ==) | 0.00% | [5 Missing :warning: ](https://app.codecov.io/gh/apache/iotdb/pull/17193?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...he/iotdb/db/utils/datastructure/BooleanTVList.java](https://app.codecov.io/gh/apache/iotdb/pull/17193?src=pr&el=tree&filepath=iotdb-core%2Fdatanode%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fiotdb%2Fdb%2Futils%2Fdatastructure%2FBooleanTVList.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aW90ZGItY29yZS9kYXRhbm9kZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvaW90ZGIvZGIvdXRpbHMvZGF0YXN0cnVjdHVyZS9Cb29sZWFuVFZMaXN0LmphdmE=) | 0.00% | [5 Missing :warning: ](https://app.codecov.io/gh/apache/iotdb/pull/17193?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...che/iotdb/db/utils/datastructure/DoubleTVList.java](https://app.codecov.io/gh/apache/iotdb/pull/17193?src=pr&el=tree&filepath=iotdb-core%2Fdatanode%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fiotdb%2Fdb%2Futils%2Fdatastructure%2FDoubleTVList.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aW90ZGItY29yZS9kYXRhbm9kZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvaW90ZGIvZGIvdXRpbHMvZGF0YXN0cnVjdHVyZS9Eb3VibGVUVkxpc3QuamF2YQ==) | 0.00% | [5 Missing :warning: ](https://app.codecov.io/gh/apache/iotdb/pull/17193?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...ache/iotdb/db/utils/datastructure/FloatTVList.java](https://app.codecov.io/gh/apache/iotdb/pull/17193?src=pr&el=tree&filepath=iotdb-core%2Fdatanode%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fiotdb%2Fdb%2Futils%2Fdatastructure%2FFloatTVList.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aW90ZGItY29yZS9kYXRhbm9kZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvaW90ZGIvZGIvdXRpbHMvZGF0YXN0cnVjdHVyZS9GbG9hdFRWTGlzdC5qYXZh) | 0.00% | [5 Missing :warning: ](https://app.codecov.io/gh/apache/iotdb/pull/17193?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...apache/iotdb/db/utils/datastructure/IntTVList.java](https://app.codecov.io/gh/apache/iotdb/pull/17193?src=pr&el=tree&filepath=iotdb-core%2Fdatanode%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fiotdb%2Fdb%2Futils%2Fdatastructure%2FIntTVList.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aW90ZGItY29yZS9kYXRhbm9kZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvaW90ZGIvZGIvdXRpbHMvZGF0YXN0cnVjdHVyZS9JbnRUVkxpc3QuamF2YQ==) | 0.00% | [5 Missing :warning: ](https://app.codecov.io/gh/apache/iotdb/pull/17193?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...pache/iotdb/db/utils/datastructure/LongTVList.java](https://app.codecov.io/gh/apache/iotdb/pull/17193?src=pr&el=tree&filepath=iotdb-core%2Fdatanode%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fiotdb%2Fdb%2Futils%2Fdatastructure%2FLongTVList.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aW90ZGItY29yZS9kYXRhbm9kZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvaW90ZGIvZGIvdXRpbHMvZGF0YXN0cnVjdHVyZS9Mb25nVFZMaXN0LmphdmE=) | 0.00% | [5 Missing :warning: ](https://app.codecov.io/gh/apache/iotdb/pull/17193?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...geengine/dataregion/memtable/WritableMemChunk.java](https://app.codecov.io/gh/apache/iotdb/pull/17193?src=pr&el=tree&filepath=iotdb-core%2Fdatanode%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fiotdb%2Fdb%2Fstorageengine%2Fdataregion%2Fmemtable%2FWritableMemChunk.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aW90ZGItY29yZS9kYXRhbm9kZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvaW90ZGIvZGIvc3RvcmFnZWVuZ2luZS9kYXRhcmVnaW9uL21lbXRhYmxlL1dyaXRhYmxlTWVtQ2h1bmsuamF2YQ==) | 50.00% | [1 Missing :warning: ](https://app.codecov.io/gh/apache/iotdb/pull/17193?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
<details><summary>Additional details and impacted files</summary>
```diff
@@ Coverage Diff @@
## master #17193 +/- ##
============================================
- Coverage 39.51% 39.50% -0.01%
Complexity 282 282
============================================
Files 5101 5101
Lines 341753 341796 +43
Branches 43543 43520 -23
============================================
- Hits 135048 135036 -12
- Misses 206705 206760 +55
```
</details>
[:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/apache/iotdb/pull/17193?dropdown=coverage&src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
:loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
<details><summary> :rocket: New features to boost your workflow: </summary>
- :snowflake: [Test Analytics](https://docs.codecov.com/docs/test-analytics): Detect flaky tests, report on failures, and find test suite problems.
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Fixed concurrency issues caused by write and flush sorting during query execution [iotdb]
Posted by "Copilot (via GitHub)" <gi...@apache.org>.
Copilot commented on code in PR #17193:
URL: https://github.com/apache/iotdb/pull/17193#discussion_r2791879095
##########
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/storageengine/dataregion/flush/MemTableFlushTask.java:
##########
@@ -272,6 +272,7 @@ public void run() {
times = new long[MAX_NUMBER_OF_POINTS_IN_PAGE];
}
writableMemChunk.encode(ioTaskQueue, encodeInfo, times);
+ writableMemChunk.releaseTemporaryTvListForFlush();
Review Comment:
`releaseTemporaryTvListForFlush()` is only called on the normal path. If `encode(...)` throws (e.g., unchecked exception during iterator/encoding), `workingListForFlush` will remain referenced on the mem chunk and may retain large cloned arrays longer than necessary. Wrap the `encode(...)` call in a `try/finally` so `releaseTemporaryTvListForFlush()` is always executed.
```suggestion
try {
writableMemChunk.encode(ioTaskQueue, encodeInfo, times);
} finally {
writableMemChunk.releaseTemporaryTvListForFlush();
}
```
##########
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/storageengine/dataregion/memtable/AbstractWritableMemChunk.java:
##########
@@ -198,7 +200,45 @@ public abstract void writeAlignedTablet(
public abstract IMeasurementSchema getSchema();
@Override
- public abstract void sortTvListForFlush();
+ public void sortTvListForFlush() {
+ TVList workingList = getWorkingTVList();
+ if (workingList.isSorted()) {
+ workingListForFlush = workingList;
+ return;
+ }
+
+ /*
+ * Concurrency background:
+ *
+ * A query may start earlier and record the current row count (rows) of the TVList as its visible range.
+ * After that, new unseq writes may arrive and immediately trigger a flush, which will sort the TVList.
+ *
+ * During sorting, the underlying indices array of the TVList may be reordered.
+ * If the query continues to use the previously recorded rows as its upper bound,
+ * it may convert a logical index to a physical index via the updated indices array.
+ *
+ * In this case, the converted physical index may exceed the previously visible
+ * rows range, leading to invalid access or unexpected behavior.
+ *
+ * To avoid this issue, when there are active queries on the working TVList, we must
+ * clone the times and indices before sorting, so that the flush sort does not mutate
+ * the data structures that concurrent queries rely on.
+ */
+ boolean needCloneTimesAndIndicesInWorkingTVList;
+ workingList.lockQueryList();
+ try {
+ needCloneTimesAndIndicesInWorkingTVList = !workingList.getQueryContextSet().isEmpty();
+ } finally {
+ workingList.unlockQueryList();
+ }
+ workingListForFlush =
+ needCloneTimesAndIndicesInWorkingTVList ? workingList.cloneForFlushSort() : workingList;
+ workingListForFlush.sort();
+ }
+
Review Comment:
This method overrides [IWritableMemChunk.releaseTemporaryTvListForFlush](1); it is advisable to add an Override annotation.
```suggestion
@Override
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Fixed concurrency issues caused by write and flush sorting during query execution [iotdb]
Posted by "sonarqubecloud[bot] (via GitHub)" <gi...@apache.org>.
sonarqubecloud[bot] commented on PR #17193:
URL: https://github.com/apache/iotdb/pull/17193#issuecomment-3882936629
## [](https://sonarcloud.io/dashboard?id=apache_iotdb&pullRequest=17193) **Quality Gate passed**
Issues
 [11 New issues](https://sonarcloud.io/project/issues?id=apache_iotdb&pullRequest=17193&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0 Accepted issues](https://sonarcloud.io/project/issues?id=apache_iotdb&pullRequest=17193&issueStatuses=ACCEPTED)
Measures
 [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_iotdb&pullRequest=17193&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0.0% Coverage on New Code](https://sonarcloud.io/component_measures?id=apache_iotdb&pullRequest=17193&metric=new_coverage&view=list)
 [0.0% Duplication on New Code](https://sonarcloud.io/component_measures?id=apache_iotdb&pullRequest=17193&metric=new_duplicated_lines_density&view=list)
[See analysis details on SonarQube Cloud](https://sonarcloud.io/dashboard?id=apache_iotdb&pullRequest=17193)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org