[PR] [Pipe] Restore sink progress under memory pressure [iotdb]

["Caideyipi (via GitHub)" <gi...@apache.org>]

Caideyipi opened a new pull request, #18640:
URL: https://github.com/apache/iotdb/pull/18640

   ## Description
   
   ### Restore sink-batch forward progress
   
   #18582 changed the memory reservation owned by `PipeTabletEventBatch` from a normal `PipeMemoryBlock` to a `PipeTabletMemoryBlock`. This made sink-batch growth subject to the tablet data-structure admission thresholds. Once retained tablet memory crossed that sub-quota, a sink batch could no longer grow enough to transfer and acknowledge events, so the retained/floating memory could not be released either.
   
   This PR restores the batch reservation to a normal pipe memory block. Tablet and TsFile allocations remain bounded by their data-structure thresholds, while sink batches can use the remaining non-floating memory to make downstream progress and release retained events.
   
   ### Back off memory-timeout retries
   
   `PipeAbstractSinkSubtask` previously retried `MEMORY_TIMEOUT` failures immediately while applying backoff to the other transient resource failures. The immediate retry loop could consume several CPU cores and flood logs while memory remained unavailable. Memory timeouts now use the existing non-report exception backoff after logging the transient condition.
   
   ### Regression coverage
   
   - Verify that tablet batch payload memory increases total pipe memory without increasing tablet-memory accounting.
   - Verify that a sink subtask waits before retrying a memory-timeout failure.
   - `PipeSleepIntervalTest`: 4 tests passed.
   - `PipeMemoryManagerResizeTest` and `PipeTransferBatchReqBuilderTest`: 8 tests passed after isolated compilation against the module classpath.
   - Spotless, Checkstyle, and `git diff --check` passed for the changed files.
   
   A full DataNode reactor compile was also attempted, but this Windows host exhausted native JVM memory while compiling the 2,983 DataNode sources. The focused DataNode classes and tests compile and pass independently.
   
   <hr>
   
   This PR has:
   - [x] been self-reviewed.
   - [x] added comments explaining the why and intent where the behavior is not obvious.
   - [x] added unit tests for the new code paths.
   
   <hr>
   
   ##### Key changed/added classes (or packages if there are too many classes) in this PR
   
   - `PipeTabletEventBatch`
   - `PipeAbstractSinkSubtask`
   - `PipeTransferBatchReqBuilderTest`
   - `PipeSleepIntervalTest`


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] [Pipe] Restore sink progress under memory pressure [iotdb]

["jt2594838 (via GitHub)" <gi...@apache.org>]

jt2594838 merged PR #18640:
URL: https://github.com/apache/iotdb/pull/18640


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org