Re: [PR] [improve][pip] PIP-434: Expose Netty channel configuration WRITE_BUFFER_WATER_MARK to pulsar conf and pause receive requests when channel is unwritable [pulsar]

["poorbarcode (via GitHub)" <gi...@apache.org>]

poorbarcode commented on code in PR #24510:
URL: https://github.com/apache/pulsar/pull/24510#discussion_r2260605554


##########
pip/pip-434.md:
##########
@@ -0,0 +1,76 @@
+# PIP-434: Expose Netty channel configuration WRITE_BUFFER_WATER_MARK to pulsar conf and pause receive requests when channel is unwritable
+
+# Background knowledge & Motivation
+
+As we discussed along the discussion: https://lists.apache.org/thread/6jfs02ovt13mnhn441txqy5m6knw6rr8
+
+> Problem Statement:
+> We've encountered a critical issue in our Apache Pulsar clusters where brokers experience Out-Of-Memory (OOM) errors and continuous restarts under specific load patterns. This occurs when Netty channel write buffers become full, leading to a buildup of unacknowledged responses in the broker's memory.
+
+> Background:
+> Our clusters are configured with numerous namespaces, each containing approximately 8,000 to 10,000 topics. Our consumer applications are quite large, with each consumer using a regular expression (regex) pattern to subscribe to all topics within a namespace.
+
+> The problem manifests particularly during consumer application restarts. When a consumer restarts, it issues a getTopicsOfNamespace request. Due to the sheer number of topics, the response size is extremely large. This massive response overwhelms the socket output buffer, causing it to fill up rapidly. Consequently, the broker's responses get backlogged in memory, eventually leading to the broker's OOM and subsequent restart loop.

Review Comment:
   @lhotari 
   
   > I think this could be resolved by providing sensible defaults (finding out the values would require benchmarking) and providing an explanation.
   > The channel writability level settings could be useful as the last resort, but it will require setting the values a lot higher than the Netty defaults since Pulsar's performance requires pipelining. That's why it won't be possible to use this solution to solve the problems with topic listings. There will need to be another setting that is suitable as a broker level limit so that broker OOM can be avoided.
   
   Since we added a metric named `pulsar_server_channel_write_buf_memory_used_bytes` by this proposal, we can get a suitable value after real practice. Then we can start a new proposal to make the mechanisms better



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org