You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ratis.apache.org by "Abhishek Pal (Jira)" <ji...@apache.org> on 2026/08/02 10:24:00 UTC
[jira] [Updated] (RATIS-2632) install-snapshot chunk loop send lacks isReady() backpressure
[ https://issues.apache.org/jira/browse/RATIS-2632?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Abhishek Pal updated RATIS-2632:
--------------------------------
Description:
When the leader streams a snapshot to a lagging follower, the chunk send loop in [GrpcLogAppender.installSnapshot()|https://github.com/apache/ratis/blob/a1bbf47f98a83d3bbddb4357c8a9b96bcfd7c727/ratis-grpc/src/main/java/org/apache/ratis/grpc/server/GrpcLogAppender.java#L775-L783] calls `*snapshotRequestObserver.onNext(request)*` for every chunk without ever checking `*CallStreamObserver.isReady()*`.
The append-entries path already [checks isReady()|https://github.com/apache/ratis/blob/a1bbf47f98a83d3bbddb4357c8a9b96bcfd7c727/ratis-grpc/src/main/java/org/apache/ratis/grpc/server/GrpcLogAppender.java#L366], but the snapshot path does not, so it is possible that chunks are enqueued faster than the network can drain them.
On the default path this is mitigated by the *INSTALL_SNAPSHOT_REQUEST_ELEMENT_LIMIT_DEFAULT* being 8, but since this can be set to 0, in the worst case the StreamObserverWithTimeout's *ResourceSemaphore* is null and cause:
- Unbounded outbound buffering of snapshot chunks in the gRPC/Netty write queue when the follower is slow.
- Leader OOM / direct-memory exhaustion during large-snapshot catch-up, which can crash the leader and impact the whole group.
was:
When the leader streams a snapshot to a lagging follower, the chunk send loop in [GrpcLogAppender.installSnapshot()|https://github.com/apache/ratis/blob/a1bbf47f98a83d3bbddb4357c8a9b96bcfd7c727/ratis-grpc/src/main/java/org/apache/ratis/grpc/server/GrpcLogAppender.java#L775-L783] calls `*snapshotRequestObserver.onNext(request)*` for every chunk without ever checking `*CallStreamObserver.isReady()*`.
The append-entries path already [checks isReady()|https://github.com/apache/ratis/blob/a1bbf47f98a83d3bbddb4357c8a9b96bcfd7c727/ratis-grpc/src/main/java/org/apache/ratis/grpc/server/GrpcLogAppender.java#L366], but the snapshot path does not, so it is possible that chunks are enqueued faster than the network can drain them.
This can cause:
- Unbounded outbound buffering of snapshot chunks in the gRPC/Netty write queue when the follower is slow.
- Leader OOM / direct-memory exhaustion during large-snapshot catch-up, which can crash the leader and impact the whole group.
> install-snapshot chunk loop send lacks isReady() backpressure
> -------------------------------------------------------------
>
> Key: RATIS-2632
> URL: https://issues.apache.org/jira/browse/RATIS-2632
> Project: Ratis
> Issue Type: Improvement
> Components: gRPC
> Reporter: Abhishek Pal
> Assignee: Abhishek Pal
> Priority: Major
> Time Spent: 10m
> Remaining Estimate: 0h
>
> When the leader streams a snapshot to a lagging follower, the chunk send loop in [GrpcLogAppender.installSnapshot()|https://github.com/apache/ratis/blob/a1bbf47f98a83d3bbddb4357c8a9b96bcfd7c727/ratis-grpc/src/main/java/org/apache/ratis/grpc/server/GrpcLogAppender.java#L775-L783] calls `*snapshotRequestObserver.onNext(request)*` for every chunk without ever checking `*CallStreamObserver.isReady()*`.
> The append-entries path already [checks isReady()|https://github.com/apache/ratis/blob/a1bbf47f98a83d3bbddb4357c8a9b96bcfd7c727/ratis-grpc/src/main/java/org/apache/ratis/grpc/server/GrpcLogAppender.java#L366], but the snapshot path does not, so it is possible that chunks are enqueued faster than the network can drain them.
> On the default path this is mitigated by the *INSTALL_SNAPSHOT_REQUEST_ELEMENT_LIMIT_DEFAULT* being 8, but since this can be set to 0, in the worst case the StreamObserverWithTimeout's *ResourceSemaphore* is null and cause:
> - Unbounded outbound buffering of snapshot chunks in the gRPC/Netty write queue when the follower is slow.
> - Leader OOM / direct-memory exhaustion during large-snapshot catch-up, which can crash the leader and impact the whole group.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)