You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ratis.apache.org by "Andrey Yarovoy (Jira)" <ji...@apache.org> on 2026/10/22 17:07:00 UTC

[jira] [Created] (RATIS-2775) Make the gRPC channel shutdown timeouts in GrpcUtil.shutdownManagedChannel configurable

Andrey Yarovoy created RATIS-2775:
-------------------------------------

             Summary: Make the gRPC channel shutdown timeouts in GrpcUtil.shutdownManagedChannel configurable
                 Key: RATIS-2775
                 URL: https://issues.apache.org/jira/browse/RATIS-2775
             Project: Ratis
          Issue Type: Improvement
          Components: util
            Reporter: Andrey Yarovoy


*Problem*

`GrpcUtil.shutdownManagedChannel(ManagedChannel)` shuts a channel down in two hard-coded steps:
1. `shutdown()`, then `awaitTermination(3, SECONDS)`, logging "Timed out gracefully shutting down connection";
2. if still running, `shutdownNow()`, then `awaitTermination(2, SECONDS)`.

`GrpcClientProtocolClient.close()` calls it for the client channel and then for the admin channel, and
`GrpcServerProtocolClient.close()` calls it as well. A channel terminates gracefully only once the server has ended
the client's request streams. So closing a `RaftClient` whose server doesn't end the stream in time costs the full
3 s, and no setting can shorten that.

That hurts processes that hold many `RaftClient`s and close them on exit. Apache Ozone keeps one `RaftClient` per
pipeline in each client process (HDDS-15255):
- MapReduce clients on a 12-datanode cluster spent up to 54 s exiting, exactly 3 s per channel that timed out.
- Closing the clients in parallel (the HDDS-15255 fix) brings that down to one grace period, but not below 3 s.
- A client that is shutting down has already given up on its pending replies. After the wait, `shutdownNow()`
cancels the streams anyway, so the wait buys it nothing.

The timeouts date from RATIS-921, which replaced a 5 s graceful-only wait to fix leaked channels. The values
themselves weren't discussed there.

*Proposed change*

- New client properties in `GrpcConfigKeys.Client`, defaulting to today's values so behaviour is unchanged:
- `raft.grpc.client.channel.shutdown.graceful.timeout` (`TimeDuration`, default `3s`);
- `raft.grpc.client.channel.shutdown.forceful.timeout` (`TimeDuration`, default `2s`).
- `GrpcUtil.shutdownManagedChannel(channel, gracefulTimeout, forcefulTimeout)`. The existing one-argument method
keeps today's values. A graceful timeout of 0 skips the graceful step and calls `shutdownNow()` straight away.
- `GrpcClientProtocolClient` reads the two values from its `RaftProperties` and uses them for both its channels.
- Optionally, the same pair for `GrpcServerProtocolClient` under `raft.grpc.server`, for leader-to-follower
channels.

*Compatibility*

The defaults keep the current behaviour. Ozone, or any client that closes on exit, could then set the graceful
timeout to 0 or a few hundred milliseconds.

*Tests*

A unit test that closes a client against a server which never completes its streams. It checks that the close takes
about the configured graceful plus forceful timeouts, and that with a 0 graceful timeout it returns immediately.



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)