You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@bookkeeper.apache.org by "wolfstudy (via GitHub)" <gi...@apache.org> on 2026/04/28 07:08:32 UTC

[PR] Support bookie server config tcp keep-alive [bookkeeper]

wolfstudy opened a new pull request, #4762:
URL: https://github.com/apache/bookkeeper/pull/4762

   ##  Motivation
   In PR #4683, we introduced support for configuring TCP keep-alive parameters (TCP_KEEPIDLE, TCP_KEEPINTVL, TCP_KEEPCNT) on the Bookie client side (PerChannelBookieClient). This allowed users to fine-tune keep-alive behavior per connection instead of relying solely on the kernel-level defaults defined in /proc/sys/net/ipv4/tcp_keepalive_*.
   
   However, as discussed in apache/pulsar#25580, a one-sided configuration is not enough. TCP keep-alive is a per-socket setting that works independently on each end of a connection. If only the client enables custom keep-alive parameters while the server still uses the OS-level defaults, operators are still forced to tune the kernel parameters of the host on which the Bookie runs in order to achieve a consistent end-to-end keep-alive behavior. This defeats the purpose of having fine-grained, application-level control and makes it cumbersome to deploy Bookies in environments where modifying OS-wide TCP settings is not permitted (e.g., shared or managed hosts).
   
   To close this gap, we need to provide the same configurability on the Bookie server side (BookieNettyServer), so that a complete, symmetric TCP keep-alive configuration can be achieved from BookKeeper itself, without requiring any changes to the underlying operating system.
   
   ## Changes
   - ServerConfiguration: added serverTcpKeepIdle / serverTcpKeepIntvl / serverTcpKeepCnt, all defaulting to -1 (meaning "use OS default").
   - BookieNettyServer: applies the new options via EpollChannelOption.TCP_KEEPIDLE / TCP_KEEPINTVL / TCP_KEEPCNT when the event loop is EpollEventLoopGroup, and only when the value is > 0. For Nio / IoUring / Default event loops, behavior falls back to the original SO_KEEPALIVE — fully backward compatible.
   ## Backward Compatibility
   All new options default to -1 and Epoll-specific channel options are only applied on EpollEventLoopGroup. Existing deployments see no behavior change after upgrading.
   ## References
   - Client-side support: apache/bookkeeper#4683
   - Motivation for symmetric server+client config: apache/pulsar#25580


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@bookkeeper.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Support bookie server config tcp keep-alive [bookkeeper]

Posted by "wolfstudy (via GitHub)" <gi...@apache.org>.
wolfstudy commented on PR #4762:
URL: https://github.com/apache/bookkeeper/pull/4762#issuecomment-4334041653

   @lhotari @wenbingshen PTAL again, thanks


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@bookkeeper.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Support bookie server config tcp keep-alive [bookkeeper]

Posted by "wolfstudy (via GitHub)" <gi...@apache.org>.
wolfstudy commented on PR #4762:
URL: https://github.com/apache/bookkeeper/pull/4762#issuecomment-4334589864

   
   rerun failure checks
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@bookkeeper.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Support bookie server config tcp keep-alive [bookkeeper]

Posted by "wenbingshen (via GitHub)" <gi...@apache.org>.
wenbingshen merged PR #4762:
URL: https://github.com/apache/bookkeeper/pull/4762


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@bookkeeper.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Support bookie server config tcp keep-alive [bookkeeper]

Posted by "wenbingshen (via GitHub)" <gi...@apache.org>.
wenbingshen commented on code in PR #4762:
URL: https://github.com/apache/bookkeeper/pull/4762#discussion_r3152721401


##########
bookkeeper-server/src/main/java/org/apache/bookkeeper/proto/BookieNettyServer.java:
##########
@@ -316,13 +318,37 @@ private void listenOn(InetSocketAddress address, BookieSocketAddress bookieAddre
             bootstrap.childOption(ChannelOption.ALLOCATOR, allocator);
             bootstrap.group(acceptorGroup, eventLoopGroup);
             bootstrap.childOption(ChannelOption.TCP_NODELAY, conf.getServerTcpNoDelay());
+            bootstrap.childOption(ChannelOption.SO_KEEPALIVE, conf.getServerSockKeepalive());
             bootstrap.childOption(ChannelOption.SO_LINGER, conf.getServerSockLinger());
             bootstrap.childOption(ChannelOption.RCVBUF_ALLOCATOR,
                     new AdaptiveRecvByteBufAllocator(conf.getRecvByteBufAllocatorSizeMin(),
                             conf.getRecvByteBufAllocatorSizeInitial(), conf.getRecvByteBufAllocatorSizeMax()));
             bootstrap.childOption(ChannelOption.WRITE_BUFFER_WATER_MARK, new WriteBufferWaterMark(
                     conf.getServerWriteBufferLowWaterMark(), conf.getServerWriteBufferHighWaterMark()));
 
+            // Set TCP keepalive parameters if configured
+            if (EventLoopUtil.isIoUringGroup(eventLoopGroup)) {
+                if (conf.getServerTcpKeepIdle() > 0) {
+                    bootstrap.option(IoUringChannelOption.TCP_KEEPIDLE, conf.getServerTcpKeepIdle());

Review Comment:
   `ServerBootstrap.option()` sets options on the listening `ServerSocketChannel`. For `TCP_KEEPIDLE/INTVL/CNT`, please use `childOption()` — it targets each accepted `SocketChannel` 
     directly, which is the Netty-idiomatic approach and avoids relying on OS-level socket option inheritance behavior. Same applies to `jvmBootstrap`.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@bookkeeper.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Support bookie server config tcp keep-alive [bookkeeper]

Posted by "wolfstudy (via GitHub)" <gi...@apache.org>.
wolfstudy commented on code in PR #4762:
URL: https://github.com/apache/bookkeeper/pull/4762#discussion_r3152979666


##########
bookkeeper-server/src/main/java/org/apache/bookkeeper/proto/BookieNettyServer.java:
##########


Review Comment:
   yes, will also be fixed together.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@bookkeeper.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Support bookie server config tcp keep-alive [bookkeeper]

Posted by "lhotari (via GitHub)" <gi...@apache.org>.
lhotari commented on code in PR #4762:
URL: https://github.com/apache/bookkeeper/pull/4762#discussion_r3152851325


##########
bookkeeper-server/src/main/java/org/apache/bookkeeper/proto/BookieNettyServer.java:
##########


Review Comment:
   these lines in the current code don't make sense for internal VM transport.



##########
bookkeeper-server/src/main/java/org/apache/bookkeeper/proto/BookieNettyServer.java:
##########
@@ -391,6 +417,29 @@ protected void initChannel(SocketChannel ch) throws Exception {
             jvmBootstrap.option(ChannelOption.WRITE_BUFFER_WATER_MARK, new WriteBufferWaterMark(
                     conf.getServerWriteBufferLowWaterMark(), conf.getServerWriteBufferHighWaterMark()));
 
+            // Set TCP keepalive parameters if configured
+            if (EventLoopUtil.isIoUringGroup(jvmEventLoopGroup)) {
+                if (conf.getServerTcpKeepIdle() > 0) {
+                    jvmBootstrap.option(IoUringChannelOption.TCP_KEEPIDLE, conf.getServerTcpKeepIdle());
+                }
+                if (conf.getServerTcpKeepIntvl() > 0) {
+                    jvmBootstrap.option(IoUringChannelOption.TCP_KEEPINTVL, conf.getServerTcpKeepIntvl());
+                }
+                if (conf.getServerTcpKeepCnt() > 0) {
+                    jvmBootstrap.option(IoUringChannelOption.TCP_KEEPCNT, conf.getServerTcpKeepCnt());
+                }
+            } else if (jvmEventLoopGroup instanceof EpollEventLoopGroup) {
+                if (conf.getServerTcpKeepIdle() > 0) {
+                    jvmBootstrap.option(EpollChannelOption.TCP_KEEPIDLE, conf.getServerTcpKeepIdle());
+                }
+                if (conf.getServerTcpKeepIntvl() > 0) {
+                    jvmBootstrap.option(EpollChannelOption.TCP_KEEPINTVL, conf.getServerTcpKeepIntvl());
+                }
+                if (conf.getServerTcpKeepCnt() > 0) {
+                    jvmBootstrap.option(EpollChannelOption.TCP_KEEPCNT, conf.getServerTcpKeepCnt());
+                }
+            }

Review Comment:
   `conf.isEnableLocalTransport` enables internal VM transport (https://issues.apache.org/jira/browse/BOOKKEEPER-896) where there's no network. The current code to use a different channel than [LocalServerChannel](https://netty.io/4.1/api/io/netty/channel/local/LocalServerChannel.html) doesn't make any sense and no network related settings like ChannelOption.TCP_NODELAY. ChannelOption.SO_KEEPALIVE or ChannelOption.SO_LINGER shouldn't be set for jvmBootStrap.



##########
bookkeeper-server/src/main/java/org/apache/bookkeeper/proto/BookieNettyServer.java:
##########


Review Comment:
   These options don't make sense for the internal VM transport.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@bookkeeper.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Support bookie server config tcp keep-alive [bookkeeper]

Posted by "wolfstudy (via GitHub)" <gi...@apache.org>.
wolfstudy commented on code in PR #4762:
URL: https://github.com/apache/bookkeeper/pull/4762#discussion_r3152944739


##########
bookkeeper-server/src/main/java/org/apache/bookkeeper/proto/BookieNettyServer.java:
##########
@@ -316,13 +318,37 @@ private void listenOn(InetSocketAddress address, BookieSocketAddress bookieAddre
             bootstrap.childOption(ChannelOption.ALLOCATOR, allocator);
             bootstrap.group(acceptorGroup, eventLoopGroup);
             bootstrap.childOption(ChannelOption.TCP_NODELAY, conf.getServerTcpNoDelay());
+            bootstrap.childOption(ChannelOption.SO_KEEPALIVE, conf.getServerSockKeepalive());
             bootstrap.childOption(ChannelOption.SO_LINGER, conf.getServerSockLinger());
             bootstrap.childOption(ChannelOption.RCVBUF_ALLOCATOR,
                     new AdaptiveRecvByteBufAllocator(conf.getRecvByteBufAllocatorSizeMin(),
                             conf.getRecvByteBufAllocatorSizeInitial(), conf.getRecvByteBufAllocatorSizeMax()));
             bootstrap.childOption(ChannelOption.WRITE_BUFFER_WATER_MARK, new WriteBufferWaterMark(
                     conf.getServerWriteBufferLowWaterMark(), conf.getServerWriteBufferHighWaterMark()));
 
+            // Set TCP keepalive parameters if configured
+            if (EventLoopUtil.isIoUringGroup(eventLoopGroup)) {
+                if (conf.getServerTcpKeepIdle() > 0) {
+                    bootstrap.option(IoUringChannelOption.TCP_KEEPIDLE, conf.getServerTcpKeepIdle());

Review Comment:
   yes, you are right, will fix



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@bookkeeper.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org