You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ratis.apache.org by "spacemonkd (via GitHub)" <gi...@apache.org> on 2026/08/02 10:21:35 UTC

[PR] RATIS-2632. Apply backpressure on install-snapshot chunk loop [ratis]

spacemonkd opened a new pull request, #1540:
URL: https://github.com/apache/ratis/pull/1540

   ## What changes were proposed in this pull request?
   
   RATIS-2632. Apply backpressure on install-snapshot chunk loop
   
   When the leader streams a snapshot to a lagging follower, the chunk send loop in [GrpcLogAppender.installSnapshot()](https://github.com/apache/ratis/blob/a1bbf47f98a83d3bbddb4357c8a9b96bcfd7c727/ratis-grpc/src/main/java/org/apache/ratis/grpc/server/GrpcLogAppender.java#L775-L783) calls `snapshotRequestObserver.onNext(request)` for every chunk without ever checking `CallStreamObserver.isReady()`.
   
   The append-entries path already [checks isReady()](https://github.com/apache/ratis/blob/a1bbf47f98a83d3bbddb4357c8a9b96bcfd7c727/ratis-grpc/src/main/java/org/apache/ratis/grpc/server/GrpcLogAppender.java#L366), but the snapshot path does not, so it is possible that chunks are enqueued faster than the network can drain them.
   
   While on the default path `INSTALL_SNAPSHOT_REQUEST_ELEMENT_LIMIT_DEFAULT` is set to 8 so `StreamObserverWithTimeout` applies a request-response window, but since we allow setting the limit to 0, the semaphore can be `null` and cause:
   
   - Unbounded outbound buffering of snapshot chunks in the gRPC/Netty write queue when the follower is slow.
   - Leader OOM / direct-memory exhaustion during large-snapshot catch-up, which can crash the leader and impact the whole group.
   
   This is handled by having an explicit isReady() check, which causes minimal overhead on the default path (`element-limit=8`) but handles the case when it might be set to 0.
   
   ## What is the link to the Apache JIRA
   https://issues.apache.org/jira/browse/RATIS-2632
   
   ## How was this patch tested?
   Patch was tested via unit tests.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ratis.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] RATIS-2632. Apply backpressure on install-snapshot chunk loop [ratis]

Posted by "spacemonkd (via GitHub)" <gi...@apache.org>.
spacemonkd commented on PR #1540:
URL: https://github.com/apache/ratis/pull/1540#issuecomment-5159370716

   @szetszwo @ss77892 could you take a look at this PR?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ratis.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] RATIS-2632. Apply backpressure on install-snapshot chunk loop [ratis]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #1540:
URL: https://github.com/apache/ratis/pull/1540#issuecomment-5975338948

   This PR has been marked as stale due to 60 days of inactivity. Please comment or remove the stale label to keep it open. Otherwise, it will be automatically closed in ~30 days.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ratis.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] RATIS-2632. Apply backpressure on install-snapshot chunk loop [ratis]

Posted by "github-actions[bot] (via GitHub) via issues" <is...@ratis.apache.org>.
github-actions[bot] commented on PR #1540:
URL: https://github.com/apache/ratis/pull/1540#issuecomment-6715052400

   Thank you for your contribution. This PR is being closed due to inactivity. Please contact a maintainer if you would like to reopen it.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ratis.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] RATIS-2632. Apply backpressure on install-snapshot chunk loop [ratis]

Posted by "github-actions[bot] (via GitHub) via issues" <is...@ratis.apache.org>.
github-actions[bot] closed pull request #1540: RATIS-2632. Apply backpressure on install-snapshot chunk loop
URL: https://github.com/apache/ratis/pull/1540


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ratis.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org