You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@bookkeeper.apache.org by GitBox <gi...@apache.org> on 2021/03/17 01:05:59 UTC
[GitHub] [bookkeeper] merlimat opened a new pull request #2646: Auto-throttle read operations
merlimat opened a new pull request #2646:
URL: https://github.com/apache/bookkeeper/pull/2646
### Motivation
When bookie is doing reads, it may use a considerable amount of memory if there are many simultaneous readers and especially when the network becomes the bottleneck.
The problem is that the read thread is scheduling the async operation to write on the socket and then it moves on for the next read operation. If the TCP window is full, these write requests on the socket are just piling up in memory, resulting in a spike of direct memory usage, which may exceed the max configured in the JVM.
Instead, if the socket write operation is blocking, we should block the BK read thread in order to auto-throttle and limit the amount of memory.
This change does not introduce any performance penalty on the read-path. The old behavior will still be available as a config option.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [bookkeeper] merlimat commented on a change in pull request #2646: Auto-throttle read operations
Posted by GitBox <gi...@apache.org>.
merlimat commented on a change in pull request #2646:
URL: https://github.com/apache/bookkeeper/pull/2646#discussion_r597049375
##########
File path: conf/bk_server.conf
##########
@@ -137,6 +137,11 @@ extraServerComponents=
# avoid the executor queue to grow indefinitely
# maxPendingAddRequestsPerThread=10000
+# Use auto-throttling of the read-worker threads. This is done
+# to ensure the bookie is not using unlimited amount of memory
+# to respond to read-requests.
+# readWorkerThreadsThrottle=true
Review comment:
Good point. Renamed to "ThrottlingEnabled"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [bookkeeper] merlimat merged pull request #2646: Auto-throttle read operations
Posted by GitBox <gi...@apache.org>.
merlimat merged pull request #2646:
URL: https://github.com/apache/bookkeeper/pull/2646
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [bookkeeper] codelipenghui commented on a change in pull request #2646: Auto-throttle read operations
Posted by GitBox <gi...@apache.org>.
codelipenghui commented on a change in pull request #2646:
URL: https://github.com/apache/bookkeeper/pull/2646#discussion_r596602460
##########
File path: conf/bk_server.conf
##########
@@ -137,6 +137,11 @@ extraServerComponents=
# avoid the executor queue to grow indefinitely
# maxPendingAddRequestsPerThread=10000
+# Use auto-throttling of the read-worker threads. This is done
+# to ensure the bookie is not using unlimited amount of memory
+# to respond to read-requests.
+# readWorkerThreadsThrottle=true
Review comment:
```suggestion
# readWorkerThreadsThrottleEnabled=true
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org