[GitHub] Jaskey opened a new issue #2013: Need a limited Threadpool in consumer side

[GitBox <gi...@apache.org>]

Jaskey opened a new issue #2013: Need a limited Threadpool in consumer side
URL: https://github.com/apache/incubator-dubbo/issues/2013
 
 
   ### Environment
   
   * Dubbo version: 2.5.2
   * Java version: 1.7
   
   For now, the consumer used a cached thread pool which has no limited thread size, in some circumstances,  too many threads will be created  and thus process will suffer from OOM.
   
   Here is one case:
   
   consumer A call service from provider B in a very big tps, but provider B is not responding very quickly in some times due to some pause, and the calls timeout, but after a short time, provider B becomes quick so the large number of responses send back to consumer A nearly in the same, time, in this case, you will find many many consumer threads are created in consumer side due to the current design.
   
   
   You can easily reproduce this issue with the following provider sample 
   
   ```
   public class MockImpl implements Mock {
   
       private static final Logger logger = LoggerFactory.getLogger(MockImpl.class);
       public void sleep(long ms) {
           long span = computeNextMinuteSpan();
           logger.info("begin to sleep "+ms+" ms");
           try {
   
               Thread.sleep(ms);
   
           } catch (InterruptedException e) {
               e.printStackTrace();
           }
           logger.info("after sleep " + ms + " ms");
       }
   
       public void sleepToNextMinute() {
           long span = computeNextMinuteSpan();
           sleep(span);
       }
   
   
       public static long computeNextMinuteSpan() {
           long now = System.currentTimeMillis();
           Calendar cal = Calendar.getInstance();
           cal.setTimeInMillis(System.currentTimeMillis());
           cal.add(Calendar.MINUTE, 1);
           cal.set(Calendar.SECOND, 0);
           cal.set(Calendar.MILLISECOND, 0);
           return cal.getTimeInMillis() - now;
       }
   }
   ```
   
   
   1. export this provider with a short timeout say 10ms, with big thread count say 1000. And if you export many providers instead of only one, the problem will be more obvious. 
   2. in a for loop, continuously consume the service `sleepToNextMinute` in a single thread or in thread pool.
   3. Observe the thread count of DubboClient .
   
   Here is a sample for consumer 
   
   
   ```
   
   
           logger.info("sleeping till next minute ......");
           Thread.sleep(computeNextMinuteSpan());
           Executors.newScheduledThreadPool(1).scheduleAtFixedRate(new Runnable() {
               @Override
               public void run() {
                   int i = 0;
                   Set<Thread> threadSet = Thread.getAllStackTraces().keySet();
                   for (Thread t : threadSet) {
                       if (t.getName().startsWith("DubboClientHandler")) {
                           logger.info("dubbo thread: {}",t.getName());
                           i++;
                       }
                   }
                   logger.info("=================================Dubbo Thread  {}===================================",i);
   
               }
           },0,1000, TimeUnit.MILLISECONDS);
           logger.info("mocking...");
           for (int i =0;i<10000;i++) {
               try {
                   mock.sleepToNextMinute();
               } catch (Exception e) {
                   //logger.error(e.getMessage());
               }
           }
           logger.info("mocking ends");
   
   ```
   
   You can easily find that the consumer threads increase heavily in a very short time

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@dubbo.apache.org
For additional commands, e-mail: notifications-help@dubbo.apache.org