You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@iotdb.apache.org by "JackieTien97 (via GitHub)" <gi...@apache.org> on 2026/10/01 00:50:51 UTC

[PR] [Fix] Isolate FileReaderManager reader I/O with per-file locks [iotdb]

JackieTien97 opened a new pull request, #18788:
URL: https://github.com/apache/iotdb/pull/18788

   ## Description
   
   A slow reader open or close currently holds the FileReaderManager monitor, stalling reader lookup and reference operations for unrelated files. This change moves reader I/O and lifecycle transitions to stable per-file entries, so unrelated files can make progress while operations on the same file remain serialized.
   
   ### Locking and lifecycle
   
   - Keep separate closed/unclosed slots under each internal TsFileID entry and a separate path namespace for external files. Hold the registry lock only for admission, pinning, and reclamation; operation pins include threads waiting on an entry, preventing replacement of an entry while a waiter still references it.
   - Preserve resource-lock ordering and release the resource read lock when reference registration fails or a release encounters a close failure. Retain a failed-close handle for retry and do not serve it again; force close attempts both slots.
   - Serialize global test cleanup, block new admissions, and continue accepting releases. Drain release pins again before atomically reclaiming entries and reopening admission, restoring interruption only after cleanup. Callers must stop reader use before global cleanup.
   - Expose reader-map snapshots for tests and replace the metadata-cache test's direct map mutation with a test-only reader setter.
   
   ### Validation
   
   Validated locally with Amazon Corretto 17.0.5, Maven 3.9.9, and real cached TsFile `2.4.1-260915-SNAPSHOT` artifacts (`20260915.102750-1`). Maven ran offline with Develocity disabled in a fresh isolated worktree.
   
   - Initial clean dependency-reactor test: FileReaderManagerConcurrencyTest **15/15** and FileReaderManagerTest **1/1**, all passing.
   - Final clean regression run: **99 tests, 0 failures, 0 errors, 1 skipped** across 12 classes, including both FileReaderManager classes, metadata/chunk/bloom-filter caches, TsFileResource, scan operators and cancellation, resource management, device iteration, and inner compaction. The skip is the existing `TimeSeriesMetadataCacheTest#testCachePlaceHolder` annotated `@Ignore("Performance")`.
   - `mvn clean test-compile -DskipTests`: **53 modules successful**.
   - `mvn clean test-compile -P with-zh-locale -DskipTests`: **53 modules successful**.
   - `mvn spotless:apply -pl iotdb-core/datanode`, full-reactor `mvn spotless:check`, and `git diff --check`: passed.
   
   The 15 new regression tests cover slow open/close isolation, same-file serialization, namespaces, constructor/close failures, reference rollback, concurrent cleanup/releases, interruption, stable pinned entries, and concurrent churn. Cleanup admission tests verify the actual WAITING state, registry monitor, and pin/drain stack frame before releasing their gates.
   
   Final test selection:
   
   ```sh
   mvn clean test -pl iotdb-core/datanode -am \
     -Dtest=FileReaderManagerConcurrencyTest,FileReaderManagerTest,TimeSeriesMetadataCacheTest,ChunkCacheTest,BloomFilterCacheTest,TsFileResourceTest,SeriesScanUtilCancellationTest,SeriesScanOperatorTest,AlignedSeriesScanOperatorTest,MultiTsFileDeviceIteratorTest,ReadChunkInnerCompactionTest,ResourceManagerTest \
     -Dsurefire.failIfNoSpecifiedTests=false -DfailIfNoTests=false
   ```
   
   Distributed-cluster and object-storage end-to-end tests were not run.
   
   <hr>
   
   This PR has:
   - [x] been self-reviewed for concurrent reader and lifecycle operations.
   - [x] added comments explaining lock order and entry/pin lifetime.
   - [x] added regression tests and adapted the existing metadata-cache test.
   
   ##### Key changed/added classes
   
   `FileReaderManager`, `FileReaderManagerConcurrencyTest`, `TimeSeriesMetadataCacheTest`.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Fix] Isolate FileReaderManager reader I/O with per-file locks [iotdb]

Posted by "JackieTien97 (via GitHub)" <gi...@apache.org>.
JackieTien97 merged PR #18788:
URL: https://github.com/apache/iotdb/pull/18788


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org