[PR] [pip] PIP-442: Add Flow Control for CommandGetTopicsOfNamespace and CommandWatchTopicList [pulsar]

["lhotari (via GitHub)" <gi...@apache.org>]

lhotari opened a new pull request, #24727:
URL: https://github.com/apache/pulsar/pull/24727

   ### Motivation
   
   **Problem**:
   The current topic listing implementation lacks flow control, creating
   unbounded memory allocation scenarios:
   - OutOfMemoryError when multiple clients request large topic lists simultaneously
   - Proxy cascading failures due to unbuffered response forwarding
   - Unpredictable resource usage making capacity planning difficult
   - Performance degradation from GC pressure affecting all broker operations
   
   A namespace with 10K topics can consume ~1MB per response. With 1K concurrent
   requests, this creates ~1GB memory pressure that can crash brokers.
   
   **Solution**:
   PIP-442 introduces MaxTopicListInFlightLimiter - a memory-aware semaphore system:
   - Dual memory tracking for heap (topic assembly) and direct memory (network buffers)
   - Asynchronous flow control with configurable timeouts and queue limits
   - Permit-based system ensuring memory is released after response transmission
   - Graceful degradation instead of broker crashes
   
   **Benefits**:
   - Prevents broker crashes from topic listing memory exhaustion
   - Predictable resource usage for capacity planning
   - Maintains full backward compatibility (no client changes required)
   - Comprehensive monitoring with detailed metrics
   - Fair resource sharing through queueing mechanisms
   
   The implementation adds flow control at two key points: after topic retrieval
   from metadata store (heap) and before response serialization (direct memory).
   
   ### Modifications
   
   See pip/pip-442.md in this PR
   
   ### Documentation
   
   <!-- DO NOT REMOVE THIS SECTION. CHECK THE PROPER BOX ONLY. -->
   
   - [x] `doc` <!-- Your PR contains doc changes. -->
   - [ ] `doc-required` <!-- Your PR changes impact docs and you will update later -->
   - [ ] `doc-not-needed` <!-- Your PR changes do not impact docs -->
   - [ ] `doc-complete` <!-- Docs have been already added -->


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] [feat][pip] PIP-442: Add memory limits for CommandGetTopicsOfNamespace and CommandWatchTopicList on Broker and Proxy [pulsar]

["lhotari (via GitHub)" <gi...@apache.org>]

lhotari merged PR #24727:
URL: https://github.com/apache/pulsar/pull/24727


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org