[PR] [Pipe] Isolate sink subtasks and failures between pipes [iotdb]

["Caideyipi (via GitHub)" <gi...@apache.org>]

Caideyipi opened a new pull request, #18549:
URL: https://github.com/apache/iotdb/pull/18549

   ## Description
   
   ### Problem
   
   `PipeSinkSubtaskManager` reused sink resources according to sink attributes. As a result, different pipes with identical sink parameters could share a sink lifecycle, pending queue, executor, and connector. Stopping, deregistering, or reporting a critical sink failure for one pipe could therefore affect another pipe.
   
   ### Changes
   
   - Key sink subtasks and their queues, executors, connectors, task IDs, and metrics by pipe name, creation time, and sink attributes.
   - Scope DataNode and ConfigNode sink critical-exception propagation to the identified pipe instead of every pipe with equal sink parameters.
   - Keep explicit pipe name and creation time authoritative, including when task metadata is detached or concurrently removed.
   - Retain the legacy single-argument manager methods as deprecated compatibility APIs. They resolve only a unique matching pipe and fail with a localized `PipeException` when the match is ambiguous.
   - Avoid shutting down the shared IoTConsensus V2 executor when an individual pipe is deregistered.
   - Add pipe identity dimensions to sink metrics, including batch histograms.
   
   ### Compatibility and concurrency
   
   There is no SQL or configuration change. Existing callers with a unique legacy sink match continue to work. Concurrent task removal cannot redirect a critical exception to another pipe, and an explicitly identified pipe is still transitioned to `STOPPED` when its task map has already been removed.
   
   ### Verification
   
   Passed before PR creation:
   
   - Spotless for `iotdb-core/node-commons`, `iotdb-core/datanode`, and `iotdb-core/confignode`.
   - DataNode Checkstyle with zero violations.
   - `git diff --check`.
   - All 16 `PipeHeartbeatParserTest` tests.
   
   The focused DataNode tests were blocked before test execution by a pre-existing, unrelated compilation mismatch at `ConfigNodeClient.java:217`: `IConfigNodeRPCService.Iface.getProcedureResult(long)` is missing. This PR does not modify that code.
   
   <hr>
   
   This PR has:
   - [x] been self-reviewed.
       - [x] concurrent read
       - [x] concurrent write
       - [x] concurrent read and write
   - [x] added comments explaining the "why" and the intent of the code wherever it would not be obvious for an unfamiliar reader.
   - [x] added unit tests or modified existing tests to cover new code paths.
   
   <hr>
   
   ##### Key changed/added classes (or packages if there are too many classes) in this PR
   
   - `PipeSinkSubtaskManager`, `PipeSinkSubtask`, and `PipeTaskSinkStage`
   - `PipeTaskAgent` and `PipeDataNodeTaskAgent`
   - `PipeHeartbeatParser`
   - `PipeDataRegionSinkMetrics` and `PipeSchemaRegionSinkMetrics`
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] [Pipe] Isolate sink subtasks and failures between pipes [iotdb]

["jt2594838 (via GitHub)" <gi...@apache.org>]

jt2594838 merged PR #18549:
URL: https://github.com/apache/iotdb/pull/18549


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org