[PR] [To dev/1.3] [Pipe] Restore sink progress under memory pressure (#18640) [iotdb]

["Caideyipi (via GitHub)" <gi...@apache.org>]

Caideyipi opened a new pull request, #18649:
URL: https://github.com/apache/iotdb/pull/18649

   ## Description
   
   Backport #18640 to `dev/1.3` by cherry-picking e2ac222343b93cd0388cae02e6f637bdd1522ddb.
   
   ### Restore sink-batch forward progress
   
   #18582 changed the memory reservation owned by `PipeTabletEventBatch` from a normal `PipeMemoryBlock` to a `PipeTabletMemoryBlock`. This made sink-batch growth subject to the tablet data-structure admission thresholds. Once retained tablet memory crossed that sub-quota, a sink batch could no longer grow enough to transfer and acknowledge events, so the retained/floating memory could not be released either.
   
   This change restores the batch reservation to a normal pipe memory block. Tablet and TsFile allocations remain bounded by their data-structure thresholds, while sink batches can use the remaining non-floating memory to make downstream progress and release retained events.
   
   ### Back off memory-timeout retries
   
   `PipeAbstractSinkSubtask` previously retried `MEMORY_TIMEOUT` failures immediately while applying backoff to the other transient resource failures. Memory timeouts now use the existing non-report exception backoff after logging the transient condition.
   
   The cherry-pick had one conflict in `PipeAbstractSinkSubtask`; the resolution preserves the `dev/1.3` logging code and applies the upstream backoff behavior.
   
   ### Regression coverage
   
   - `mvn spotless:apply -pl iotdb-core/node-commons,iotdb-core/datanode`: passed.
   - `mvn test -pl iotdb-core/node-commons -Dtest=PipeSleepIntervalTest`: passed (4 tests).
   - `git diff origin/dev/1.3..HEAD --check`: passed.
   - `PipeTransferBatchReqBuilderTest` was attempted, but the standalone DataNode build stopped before tests because the local `1.3.7-SNAPSHOT` dependency artifacts are older than the current `dev/1.3` sources (missing unrelated generated Thrift and cross-module APIs). The changed DataNode source itself produced no compiler error; CI will build it with reactor-consistent dependencies.
   
   <hr>
   
   This PR has:
   - [x] been self-reviewed.
   - [x] added unit tests for the changed code paths.
   
   <hr>
   
   ##### Key changed/added classes
   
   - `PipeTabletEventBatch`
   - `PipeAbstractSinkSubtask`
   - `PipeTransferBatchReqBuilderTest`
   - `PipeSleepIntervalTest`
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] [To dev/1.3] [Pipe] Restore sink progress under memory pressure (#18640) [iotdb]

["jt2594838 (via GitHub)" <gi...@apache.org>]

jt2594838 merged PR #18649:
URL: https://github.com/apache/iotdb/pull/18649


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org