You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@iotdb.apache.org by "Caideyipi (via GitHub)" <gi...@apache.org> on 2026/07/31 09:42:17 UTC

[PR] Pipe: Fix concurrent TsFile reference increases [iotdb]

Caideyipi opened a new pull request, #18376:
URL: https://github.com/apache/iotdb/pull/18376

   ## Description
   
   `PipeTsFileResourceManager.increaseFileReference` checked whether a resource existed before entering the creation critical section. Historical and realtime extractors could both observe a missing resource, then create and put separate `PipeTsFileResource` instances for the same hardlink. The later put overwrote the first reference count, so releasing one event could delete a hardlink that another event still needed and stop the pipe with `FileNotFoundException`.
   
   This change rechecks the resource map after acquiring the segment lock. If another caller has already created the resource, it increments and reuses that resource instead of overwriting it.
   
   A concurrent regression test starts 64 retain operations together and verifies that both the pipe-private and shared reference counts reach 64, then verifies complete cleanup after all releases. Before the fix, the test reproduced a lost count (`expected: 64, actual: 58`); with the fix, the full test class passes.
   
   Validation:
   - `mvn -pl iotdb-core/datanode -Ddevelocity.off=true -Dtest=PipeTsFileResourceManagerTest test`
   - `mvn -pl iotdb-core/datanode -Ddevelocity.off=true spotless:check`
   - `mvn -pl iotdb-core/datanode -Ddevelocity.off=true checkstyle:check`
   - `git diff --check`
   
   This PR has:
   - [x] been self-reviewed.
       - [x] concurrent write
   - [x] added comments explaining the why and the intent of the code wherever it would not be obvious.
   - [x] added unit tests to cover the concurrent code path.
   
   <hr>
   
   ##### Key changed/added classes
   
   - `PipeTsFileResourceManager`
   - `PipeTsFileResourceManagerTest`


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Pipe: Fix TsFile reference races and improve result set diffs [iotdb]

Posted by "Caideyipi (via GitHub)" <gi...@apache.org>.
Caideyipi commented on code in PR #18376:
URL: https://github.com/apache/iotdb/pull/18376#discussion_r3877899601


##########
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/pipe/resource/tsfile/PipeTsFileResourceManager.java:
##########
@@ -122,28 +122,39 @@ private File increaseFileReference(
 
     segmentLock.lock(hardlinkOrCopiedFile);
     try {
-      resultFile =
-          isTsFile
-              ? FileUtils.createHardLink(source, hardlinkOrCopiedFile)
-              : FileUtils.copyFile(source, hardlinkOrCopiedFile);
-
-      // If the file is not a hardlink or copied file, and there is no related hardlink or copied
-      // file in pipe dir, create a hardlink or copy it to pipe dir, maintain a reference count for
-      // the hardlink or copied file, and return the hardlink or copied file.
-      if (Objects.nonNull(pipeName)) {
-        pipeNameToPipeTsFileDirPathMap.putIfAbsent(
-            pipeName, hardlinkOrCopiedFile.getParentFile().getPath());
-        hardlinkOrCopiedFileToPipeTsFileResourceMap
-            .computeIfAbsent(pipeName, k -> new ConcurrentHashMap<>())
-            .put(resultFile.getPath(), new PipeTsFileResource(resultFile));
+      final PipeTsFileResource existingResource =
+          getResourceMap(pipeName).get(hardlinkOrCopiedFile.getPath());
+      if (existingResource != null) {
+        existingResource.increaseReferenceCount();
+        resultFile = existingResource.getFile();

Review Comment:
   Yes. `getResourceMap` accepts a nullable `pipeName`: a non-null name selects the per-pipe private map, while `null` deliberately selects `hardlinkOrCopiedFileToTsFilePublicResourceMap`. The `null` path is used for the shared public reference owned by the assigner. I added Javadoc in ed65e27af39 to make this contract explicit.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Pipe: Fix TsFile reference races and improve result set diffs [iotdb]

Posted by "jt2594838 (via GitHub)" <gi...@apache.org>.
jt2594838 merged PR #18376:
URL: https://github.com/apache/iotdb/pull/18376


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Pipe: Fix TsFile reference races and improve result set diffs [iotdb]

Posted by "jt2594838 (via GitHub)" <gi...@apache.org>.
jt2594838 commented on code in PR #18376:
URL: https://github.com/apache/iotdb/pull/18376#discussion_r3842177106


##########
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/pipe/resource/tsfile/PipeTsFileResourceManager.java:
##########
@@ -122,28 +122,39 @@ private File increaseFileReference(
 
     segmentLock.lock(hardlinkOrCopiedFile);
     try {
-      resultFile =
-          isTsFile
-              ? FileUtils.createHardLink(source, hardlinkOrCopiedFile)
-              : FileUtils.copyFile(source, hardlinkOrCopiedFile);
-
-      // If the file is not a hardlink or copied file, and there is no related hardlink or copied
-      // file in pipe dir, create a hardlink or copy it to pipe dir, maintain a reference count for
-      // the hardlink or copied file, and return the hardlink or copied file.
-      if (Objects.nonNull(pipeName)) {
-        pipeNameToPipeTsFileDirPathMap.putIfAbsent(
-            pipeName, hardlinkOrCopiedFile.getParentFile().getPath());
-        hardlinkOrCopiedFileToPipeTsFileResourceMap
-            .computeIfAbsent(pipeName, k -> new ConcurrentHashMap<>())
-            .put(resultFile.getPath(), new PipeTsFileResource(resultFile));
+      final PipeTsFileResource existingResource =
+          getResourceMap(pipeName).get(hardlinkOrCopiedFile.getPath());
+      if (existingResource != null) {
+        existingResource.increaseReferenceCount();
+        resultFile = existingResource.getFile();

Review Comment:
   The nullability of `pipeName` is not checked before calling ` getResourceMap(pipeName)`, is that ok?



##########
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/pipe/resource/tsfile/PipeTsFileResourceManager.java:
##########
@@ -122,28 +122,39 @@ private File increaseFileReference(
 
     segmentLock.lock(hardlinkOrCopiedFile);
     try {
-      resultFile =
-          isTsFile
-              ? FileUtils.createHardLink(source, hardlinkOrCopiedFile)
-              : FileUtils.copyFile(source, hardlinkOrCopiedFile);
-
-      // If the file is not a hardlink or copied file, and there is no related hardlink or copied
-      // file in pipe dir, create a hardlink or copy it to pipe dir, maintain a reference count for
-      // the hardlink or copied file, and return the hardlink or copied file.
-      if (Objects.nonNull(pipeName)) {
-        pipeNameToPipeTsFileDirPathMap.putIfAbsent(
-            pipeName, hardlinkOrCopiedFile.getParentFile().getPath());
-        hardlinkOrCopiedFileToPipeTsFileResourceMap
-            .computeIfAbsent(pipeName, k -> new ConcurrentHashMap<>())
-            .put(resultFile.getPath(), new PipeTsFileResource(resultFile));
+      final PipeTsFileResource existingResource =
+          getResourceMap(pipeName).get(hardlinkOrCopiedFile.getPath());
+      if (existingResource != null) {
+        existingResource.increaseReferenceCount();
+        resultFile = existingResource.getFile();

Review Comment:
   The nullability of `pipeName` is not checked before calling ` getResourceMap(pipeName)`. Is that ok?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Pipe: Fix TsFile reference races and improve result set diffs [iotdb]

Posted by "jt2594838 (via GitHub)" <gi...@apache.org>.
jt2594838 commented on code in PR #18376:
URL: https://github.com/apache/iotdb/pull/18376#discussion_r3842177106


##########
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/pipe/resource/tsfile/PipeTsFileResourceManager.java:
##########
@@ -122,28 +122,39 @@ private File increaseFileReference(
 
     segmentLock.lock(hardlinkOrCopiedFile);
     try {
-      resultFile =
-          isTsFile
-              ? FileUtils.createHardLink(source, hardlinkOrCopiedFile)
-              : FileUtils.copyFile(source, hardlinkOrCopiedFile);
-
-      // If the file is not a hardlink or copied file, and there is no related hardlink or copied
-      // file in pipe dir, create a hardlink or copy it to pipe dir, maintain a reference count for
-      // the hardlink or copied file, and return the hardlink or copied file.
-      if (Objects.nonNull(pipeName)) {
-        pipeNameToPipeTsFileDirPathMap.putIfAbsent(
-            pipeName, hardlinkOrCopiedFile.getParentFile().getPath());
-        hardlinkOrCopiedFileToPipeTsFileResourceMap
-            .computeIfAbsent(pipeName, k -> new ConcurrentHashMap<>())
-            .put(resultFile.getPath(), new PipeTsFileResource(resultFile));
+      final PipeTsFileResource existingResource =
+          getResourceMap(pipeName).get(hardlinkOrCopiedFile.getPath());
+      if (existingResource != null) {
+        existingResource.increaseReferenceCount();
+        resultFile = existingResource.getFile();

Review Comment:
   The nullability is not checked before calling ` getResourceMap(pipeName)`, is that ok?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org