You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@iotdb.apache.org by "shuwenwei (via GitHub)" <gi...@apache.org> on 2026/05/29 02:02:33 UTC

[PR] Improve query modification loading memory control [iotdb]

shuwenwei opened a new pull request, #17788:
URL: https://github.com/apache/iotdb/pull/17788

   ## Summary
   - Add quota pre-checking before loading modification files into the query mods cache.
   - Reserve memory incrementally while building modification pattern trees.
   - Fall back to path-matched scanning when cache quota or memory reservation is insufficient.
   - Release partial tree quota and memory before returning fallback matched modifications.
   - Extend unit coverage for cache loading, fallback scanning, final reservation failure, and sort/merge reservation adjustment.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Improve query modification loading memory control [iotdb]

Posted by "sonarqubecloud[bot] (via GitHub)" <gi...@apache.org>.
sonarqubecloud[bot] commented on PR #17788:
URL: https://github.com/apache/iotdb/pull/17788#issuecomment-4569870911

   ## [![Quality Gate Passed](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/QualityGateBadge/qg-passed-20px.png 'Quality Gate Passed')](https://sonarcloud.io/dashboard?id=apache_iotdb&pullRequest=17788) **Quality Gate passed**  
   Issues  
   ![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/passed-16px.png '') [2 New issues](https://sonarcloud.io/project/issues?id=apache_iotdb&pullRequest=17788&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)  
   ![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/accepted-16px.png '') [0 Accepted issues](https://sonarcloud.io/project/issues?id=apache_iotdb&pullRequest=17788&issueStatuses=ACCEPTED)
   
   Measures  
   ![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/passed-16px.png '') [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_iotdb&pullRequest=17788&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)  
   ![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/passed-16px.png '') [0.0% Coverage on New Code](https://sonarcloud.io/component_measures?id=apache_iotdb&pullRequest=17788&metric=new_coverage&view=list)  
   ![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/passed-16px.png '') [0.0% Duplication on New Code](https://sonarcloud.io/component_measures?id=apache_iotdb&pullRequest=17788&metric=new_duplicated_lines_density&view=list)  
     
   <!-- sqra-placement-anchor -->
   [See analysis details on SonarQube Cloud](https://sonarcloud.io/dashboard?id=apache_iotdb&pullRequest=17788)
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Improve query modification loading memory control [iotdb]

Posted by "JackieTien97 (via GitHub)" <gi...@apache.org>.
JackieTien97 merged PR #17788:
URL: https://github.com/apache/iotdb/pull/17788


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Improve query modification loading memory control [iotdb]

Posted by "Caideyipi (via GitHub)" <gi...@apache.org>.
Caideyipi commented on PR #17788:
URL: https://github.com/apache/iotdb/pull/17788#issuecomment-4598589664

   Review notes:
   
   1. `QueryModificationLoader.java:115` uses `TsFileResource.getTotalModSizeInByte()` as the quota pre-check before trying to cache mods. However, `TsFileResource.getTotalModSizeInByte()` currently counts the full shared mods file length (`TsFileResource.java:453-455`), while the actual iterator only reads shared mods from `sharedModFileOffset` (`TsFileResource.java:1515-1516`). For TsFiles using shared mods, this can make the pre-check fail just because the shared file has grown globally, even when the unread portion for this TsFile is small. That means cache loading will be bypassed and the query will repeatedly fall back to scan. Please consider using the actual readable mods size here, e.g. `exclusiveModFileLength + max(0, sharedModFileLength - sharedModFileOffset)`.
   
   2. `FragmentInstanceContext.java:413` uses `modification -> modification.affects(deviceID)` as the fallback matcher for the device-only overload. This is broader than the cache path (`mods -> getPathModifications(mods, deviceID)`), which matches `deviceID + AlignedPath.VECTOR_PLACEHOLDER`. For example, a deletion on `root.sg.d1.s1` affects device `root.sg.d1`, but it does not match the time-column/vector-placeholder path. So cache success and fallback scan can return different results for the same API. Even if this overload is not on the hot path today, the fallback predicate should be equivalent to the pattern-tree matcher.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Improve query modification loading memory control [iotdb]

Posted by "codecov[bot] (via GitHub)" <gi...@apache.org>.
codecov[bot] commented on PR #17788:
URL: https://github.com/apache/iotdb/pull/17788#issuecomment-4570023499

   ## [Codecov](https://app.codecov.io/gh/apache/iotdb/pull/17788?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
   :x: Patch coverage is `78.45304%` with `39 lines` in your changes missing coverage. Please review.
   :white_check_mark: Project coverage is 40.80%. Comparing base ([`0c25e53`](https://app.codecov.io/gh/apache/iotdb/commit/0c25e53281f54c360bb33264b7acbc06fcf32bdc?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`55606d4`](https://app.codecov.io/gh/apache/iotdb/commit/55606d4ef4f5ccb740cbca6833aceaede4c60b27?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)).
   :warning: Report is 14 commits behind head on master.
   
   | [Files with missing lines](https://app.codecov.io/gh/apache/iotdb/pull/17788?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Patch % | Lines |
   |---|---|---|
   | [...ne/execution/fragment/FragmentInstanceContext.java](https://app.codecov.io/gh/apache/iotdb/pull/17788?src=pr&el=tree&filepath=iotdb-core%2Fdatanode%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fiotdb%2Fdb%2Fqueryengine%2Fexecution%2Ffragment%2FFragmentInstanceContext.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aW90ZGItY29yZS9kYXRhbm9kZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvaW90ZGIvZGIvcXVlcnllbmdpbmUvZXhlY3V0aW9uL2ZyYWdtZW50L0ZyYWdtZW50SW5zdGFuY2VDb250ZXh0LmphdmE=) | 18.18% | [18 Missing :warning: ](https://app.codecov.io/gh/apache/iotdb/pull/17788?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
   | [...ne/execution/fragment/QueryModificationLoader.java](https://app.codecov.io/gh/apache/iotdb/pull/17788?src=pr&el=tree&filepath=iotdb-core%2Fdatanode%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fiotdb%2Fdb%2Fqueryengine%2Fexecution%2Ffragment%2FQueryModificationLoader.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aW90ZGItY29yZS9kYXRhbm9kZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvaW90ZGIvZGIvcXVlcnllbmdpbmUvZXhlY3V0aW9uL2ZyYWdtZW50L1F1ZXJ5TW9kaWZpY2F0aW9uTG9hZGVyLmphdmE=) | 93.23% | [9 Missing :warning: ](https://app.codecov.io/gh/apache/iotdb/pull/17788?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
   | [.../memory/NotThreadSafeMemoryReservationManager.java](https://app.codecov.io/gh/apache/iotdb/pull/17788?src=pr&el=tree&filepath=iotdb-core%2Fdatanode%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fiotdb%2Fdb%2Fqueryengine%2Fplan%2Fplanner%2Fmemory%2FNotThreadSafeMemoryReservationManager.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aW90ZGItY29yZS9kYXRhbm9kZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvaW90ZGIvZGIvcXVlcnllbmdpbmUvcGxhbi9wbGFubmVyL21lbW9yeS9Ob3RUaHJlYWRTYWZlTWVtb3J5UmVzZXJ2YXRpb25NYW5hZ2VyLmphdmE=) | 0.00% | [5 Missing :warning: ](https://app.codecov.io/gh/apache/iotdb/pull/17788?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
   | [...b/queryengine/execution/fragment/QueryContext.java](https://app.codecov.io/gh/apache/iotdb/pull/17788?src=pr&el=tree&filepath=iotdb-core%2Fdatanode%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fiotdb%2Fdb%2Fqueryengine%2Fexecution%2Ffragment%2FQueryContext.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aW90ZGItY29yZS9kYXRhbm9kZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvaW90ZGIvZGIvcXVlcnllbmdpbmUvZXhlY3V0aW9uL2ZyYWdtZW50L1F1ZXJ5Q29udGV4dC5qYXZh) | 77.77% | [2 Missing :warning: ](https://app.codecov.io/gh/apache/iotdb/pull/17788?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
   | [...ner/memory/ThreadSafeMemoryReservationManager.java](https://app.codecov.io/gh/apache/iotdb/pull/17788?src=pr&el=tree&filepath=iotdb-core%2Fdatanode%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fiotdb%2Fdb%2Fqueryengine%2Fplan%2Fplanner%2Fmemory%2FThreadSafeMemoryReservationManager.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aW90ZGItY29yZS9kYXRhbm9kZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvaW90ZGIvZGIvcXVlcnllbmdpbmUvcGxhbi9wbGFubmVyL21lbW9yeS9UaHJlYWRTYWZlTWVtb3J5UmVzZXJ2YXRpb25NYW5hZ2VyLmphdmE=) | 0.00% | [2 Missing :warning: ](https://app.codecov.io/gh/apache/iotdb/pull/17788?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
   | [...torageengine/dataregion/tsfile/TsFileResource.java](https://app.codecov.io/gh/apache/iotdb/pull/17788?src=pr&el=tree&filepath=iotdb-core%2Fdatanode%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fiotdb%2Fdb%2Fstorageengine%2Fdataregion%2Ftsfile%2FTsFileResource.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aW90ZGItY29yZS9kYXRhbm9kZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvaW90ZGIvZGIvc3RvcmFnZWVuZ2luZS9kYXRhcmVnaW9uL3RzZmlsZS9Uc0ZpbGVSZXNvdXJjZS5qYXZh) | 77.77% | [2 Missing :warning: ](https://app.codecov.io/gh/apache/iotdb/pull/17788?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
   | [.../planner/memory/FakedMemoryReservationManager.java](https://app.codecov.io/gh/apache/iotdb/pull/17788?src=pr&el=tree&filepath=iotdb-core%2Fdatanode%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fiotdb%2Fdb%2Fqueryengine%2Fplan%2Fplanner%2Fmemory%2FFakedMemoryReservationManager.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aW90ZGItY29yZS9kYXRhbm9kZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvaW90ZGIvZGIvcXVlcnllbmdpbmUvcGxhbi9wbGFubmVyL21lbW9yeS9GYWtlZE1lbW9yeVJlc2VydmF0aW9uTWFuYWdlci5qYXZh) | 0.00% | [1 Missing :warning: ](https://app.codecov.io/gh/apache/iotdb/pull/17788?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
   
   <details><summary>Additional details and impacted files</summary>
   
   
   
   ```diff
   @@             Coverage Diff              @@
   ##             master   #17788      +/-   ##
   ============================================
   + Coverage     40.58%   40.80%   +0.21%     
   - Complexity     2575     2610      +35     
   ============================================
     Files          5181     5185       +4     
     Lines        350404   351013     +609     
     Branches      44801    44906     +105     
   ============================================
   + Hits         142225   143243    +1018     
   + Misses       208179   207770     -409     
   ```
   </details>
   
   [:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/apache/iotdb/pull/17788?dropdown=coverage&src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).   
   :loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
   <details><summary> :rocket: New features to boost your workflow: </summary>
   
   - :snowflake: [Test Analytics](https://docs.codecov.com/docs/test-analytics): Detect flaky tests, report on failures, and find test suite problems.
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Improve query modification loading memory control [iotdb]

Posted by "sonarqubecloud[bot] (via GitHub)" <gi...@apache.org>.
sonarqubecloud[bot] commented on PR #17788:
URL: https://github.com/apache/iotdb/pull/17788#issuecomment-4627708171

   ## [![Quality Gate Passed](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/QualityGateBadge/qg-passed-20px.png 'Quality Gate Passed')](https://sonarcloud.io/dashboard?id=apache_iotdb&pullRequest=17788) **Quality Gate passed**  
   Issues  
   ![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/passed-16px.png '') [4 New issues](https://sonarcloud.io/project/issues?id=apache_iotdb&pullRequest=17788&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)  
   ![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/accepted-16px.png '') [0 Accepted issues](https://sonarcloud.io/project/issues?id=apache_iotdb&pullRequest=17788&issueStatuses=ACCEPTED)
   
   Measures  
   ![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/passed-16px.png '') [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_iotdb&pullRequest=17788&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)  
   ![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/passed-16px.png '') [0.0% Coverage on New Code](https://sonarcloud.io/component_measures?id=apache_iotdb&pullRequest=17788&metric=new_coverage&view=list)  
   ![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/passed-16px.png '') [0.0% Duplication on New Code](https://sonarcloud.io/component_measures?id=apache_iotdb&pullRequest=17788&metric=new_duplicated_lines_density&view=list)  
     
   <!-- sqra-placement-anchor -->
   [See analysis details on SonarQube Cloud](https://sonarcloud.io/dashboard?id=apache_iotdb&pullRequest=17788)
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org