[I] [Bug] Potential OOM risk in NettyRemotingServer due to unbounded publicExecutor queue [rocketmq]

["QiuYucheng2003 (via GitHub)" <gi...@apache.org>]

QiuYucheng2003 opened a new issue, #9985:
URL: https://github.com/apache/rocketmq/issues/9985

   ### Before Creating the Bug Report
   
   - [x] I found a bug, not just asking a question, which should be created in [GitHub Discussions](https://github.com/apache/rocketmq/discussions).
   
   - [x] I have searched the [GitHub Issues](https://github.com/apache/rocketmq/issues) and [GitHub Discussions](https://github.com/apache/rocketmq/discussions)  of this repository and believe that this is not a duplicate.
   
   - [x] I have confirmed that this bug belongs to the current repository, not other repositories of RocketMQ.
   
   
   ### Runtime platform environment
   
   OS: Cross-platform (Issue found via static code analysis)
   
   ### RocketMQ version
   
   Branch: master
   
   
   ### JDK Version
   
   JDK 1.8
   
   ### Describe the Bug
   
   I conducted a static code analysis on 
   org.apache.rocketmq.remoting.netty.NettyRemotingServer. 
   I noticed that the publicExecutor is initialized using Executors.newFixedThreadPool.
   
   According to Java best practices (and explicitly mentioned in Alibaba Java Coding Guidelines), Executors.newFixedThreadPool uses an unbounded LinkedBlockingQueue (capacity is Integer.MAX_VALUE) by default.
   
   If the consumption rate of the thread pool is lower than the production rate (e.g., under high concurrency or network jitter), tasks will accumulate in the queue indefinitely. Since the queue is effectively unbounded, this creates a high risk of OutOfMemoryError (OOM).
   
   ### Steps to Reproduce
   
   1. Start the RocketMQ NameServer or Broker.
   
   2.Simulate a scenario where the publicExecutor handles a massive amount of requests, or the processing threads are blocked/slow.
   
   3.Observe the heap memory usage.
   
   4.The queue size of the executor will grow without limit, eventually leading to OOM.
   
   ### What Did You Expect to See?
   
   The publicExecutor should be initialized using ThreadPoolExecutor with a bounded queue 
   (e.g., LinkedBlockingQueue with a configurable capacity) and a proper RejectedExecutionHandler to prevent system crashes under load.
   
   ### What Did You See Instead?
   
   The publicExecutor uses an unbounded queue (Integer.MAX_VALUE), which offers no back-pressure mechanism or memory protection.
   
   ### Additional Context
   
   Code Location:
   https://github.com/apache/rocketmq/blob/master/remoting/src/main/java/org/apache/rocketmq/remoting/netty/NettyRemotingServer.java
   
   Snippet:
   this.publicExecutor = Executors.newFixedThreadPool(publicThreadNums, new ThreadFactory() {
       private AtomicInteger threadIndex = new AtomicInteger(0);
       @Override
       public Thread newThread(Runnable r) {
           return new Thread(r, "NettyServerPublicExecutor_" + this.threadIndex.incrementAndGet());
       }
   });
   
   I would be happy to submit a Pull Request to fix this (by replacing it with ThreadPoolExecutor) if the community confirms this issue.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@rocketmq.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] [Bug] Potential OOM risk in NettyRemotingServer due to unbounded publicExecutor queue [rocketmq]

["Kris20030907 (via GitHub)" <gi...@apache.org>]

Kris20030907 commented on issue #9985:
URL: https://github.com/apache/rocketmq/issues/9985#issuecomment-3741551935

   In fact, this issue has been raised before. https://github.com/apache/rocketmq/issues/579
   Perhaps you could implement a thread pool with a reasonable rejection policy.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@rocketmq.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org