You are viewing a plain text version of this content. The canonical link for it is here.

Posted to dev@rocketmq.apache.org by GitBox <gi...@apache.org> on 2022/06/29 08:25:41 UTC

[GitHub] [rocketmq-spring] kahle23 opened a new issue, #467: Heap memory leak caused OOM

kahle23 opened a new issue, #467:
URL: https://github.com/apache/rocketmq-spring/issues/467

   **BUG REPORT**
   
   1. Please describe the issue you observed:
   
   Heap memory leak caused OOM.
   
   - What did you do (The steps to reproduce)?
   
   ```
   @Slf4j
   @Component
   @RocketMQMessageListener(topic = "${rocketmq.consumer.topic}"
           , consumerGroup ="${rocketmq.consumer.group}")
   public class EventListener implements RocketMQListener<Event>, RocketMQPushConsumerLifecycleListener {
   
       @Override
       public void prepareStart(DefaultMQPushConsumer consumer) {
           consumer.setMaxReconsumeTimes(5);
           consumer.setPullThresholdForTopic(200);
           consumer.setPullThresholdSizeForTopic(100);
       }
   
       @Override
       public void onMessage(Event message) {
   
           log.info("The event listener: {}", JSON.toJSONString(message));
       }
   
   }
   ```
   
   - What did you expect to see?
   
   Message consumption is normal.
   
   - What did you see instead?
   
   Messages are being consumed, but the old zone is also accumulating.
   
   3. Please tell us about your environment:
   
   JDK 1.8
   spring-boot  2.3.2.RELEASE
   rocketmq-spring-boot-starter  2.2.2
   
   5. Other information (e.g. detailed explanation, logs, related issues, suggestions how to fix, etc):
   
   ps_old_gen always unable to release, after holding for 1-2 days, OOM will occur.
   
   ![7cb9dde8227a7e2f8da406ed9759ae8](https://user-images.githubusercontent.com/16236755/176388669-2af8d227-0fe4-42b5-8d82-4f33817c0794.png)
   ![78cb0c26671ee0ee1628c9dfec7f6ec](https://user-images.githubusercontent.com/16236755/176388726-c9387d13-f574-4e37-bfa5-619bdd5a5f18.png)
   
   ![27d2c64a317ddfa49d408a1cc15ad2f](https://user-images.githubusercontent.com/16236755/176388289-85bca7d3-f72d-4e0f-b248-0a7bee8c2439.png)
   ![9930d024336a2496a084664380b49f3](https://user-images.githubusercontent.com/16236755/176388342-eb90ecc9-5b57-49cc-bfc9-c1bab0272eb1.png)
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@rocketmq.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [rocketmq-spring] panzhi33 commented on issue #467: Heap memory leak caused OOM

Posted by GitBox <gi...@apache.org>.

panzhi33 commented on issue #467:
URL: https://github.com/apache/rocketmq-spring/issues/467#issuecomment-1203614405

   Look at the stack information, use litePull, the above code uses push. litePull needs to take the initiative to pull it. If you don't take the 
    initiative to pull the cancellation fee, it will cache a large number of ConsumeRequest objects.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@rocketmq.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [rocketmq-spring] kahle23 commented on issue #467: Heap memory leak caused OOM

Posted by GitBox <gi...@apache.org>.

kahle23 commented on issue #467:
URL: https://github.com/apache/rocketmq-spring/issues/467#issuecomment-1201919900

   ![1659404269926](https://user-images.githubusercontent.com/16236755/182273793-53b996e0-4543-4625-b837-d4aa99e1720f.jpg)
   ![1659404519864](https://user-images.githubusercontent.com/16236755/182273810-10adc050-005a-47d2-a820-b1d853c8cc32.jpg)
   ![1659404587605](https://user-images.githubusercontent.com/16236755/182273821-11f71c41-1dfe-434c-be93-52ce363a9366.jpg)
   
   Is that it?
   
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@rocketmq.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [rocketmq-spring] panzhi33 commented on issue #467: Heap memory leak caused OOM

Posted by GitBox <gi...@apache.org>.

panzhi33 commented on issue #467:
URL: https://github.com/apache/rocketmq-spring/issues/467#issuecomment-1203417162

   Is litePull also used when using rocketmq-client ?
    DefaultLitePullConsumerImpl caches a large number of ConsumeRequest objects, and ConsumeRequest also holds message objects,the reason is still consume slowly.  can search rocketmq_client.log for 'The consume request count exceeds'


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@rocketmq.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [rocketmq-spring] kahle23 commented on issue #467: Heap memory leak caused OOM

Posted by GitBox <gi...@apache.org>.

kahle23 commented on issue #467:
URL: https://github.com/apache/rocketmq-spring/issues/467#issuecomment-1190950220

   However, I see MQ Dashboard messages are consumed and are manually GC, and the memory is still not release.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@rocketmq.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [rocketmq-spring] panzhi33 commented on issue #467: Heap memory leak caused OOM

Posted by GitBox <gi...@apache.org>.

panzhi33 commented on issue #467:
URL: https://github.com/apache/rocketmq-spring/issues/467#issuecomment-1190992309

   > So far I've switched to rocketmq-client 4.9.3 and everything works fine without changing any configuration. PullThresholdForTopic and PullThresholdSizeForTopic I tried a variety of, still can appear
   
   did not user rocketmq-spring-boot-starter 2.2.2。 use rocketmq-client 4.9.3 directly no problem?
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@rocketmq.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [rocketmq-spring] panzhi33 commented on issue #467: Heap memory leak caused OOM

Posted by GitBox <gi...@apache.org>.

panzhi33 commented on issue #467:
URL: https://github.com/apache/rocketmq-spring/issues/467#issuecomment-1172018193

   Maybe it is caused by too many cached messages. You can check rocketmq_client.log to see if there is a the cached message size exceeds the threshold . PullThresholdSizeForTopic can be set smaller.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@rocketmq.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [rocketmq-spring] kahle23 commented on issue #467: Heap memory leak caused OOM

Posted by GitBox <gi...@apache.org>.

kahle23 commented on issue #467:
URL: https://github.com/apache/rocketmq-spring/issues/467#issuecomment-1193450426

   > > So far I've switched to rocketmq-client 4.9.3 and everything works fine without changing any configuration. PullThresholdForTopic and PullThresholdSizeForTopic I tried a variety of, still can appear
   > 
   > did not user rocketmq-spring-boot-starter 2.2.2。 use rocketmq-client 4.9.3 directly no problem?
   
   Yes, i'm use rocketmq-client 4.9.3 is ok in the same MQ.  That's why I think it's rocketmq-spring-boot-starter 2.2.2 bug, because nothing else has changed


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@rocketmq.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [rocketmq-spring] kahle23 commented on issue #467: Heap memory leak caused OOM

Posted by GitBox <gi...@apache.org>.

kahle23 commented on issue #467:
URL: https://github.com/apache/rocketmq-spring/issues/467#issuecomment-1190956814

   So far I've switched to rocketmq-client 4.9.3 and everything works fine without changing any configuration.   PullThresholdForTopic and PullThresholdSizeForTopic I tried a variety of, still can appear


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@rocketmq.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [rocketmq-spring] kahle23 commented on issue #467: Heap memory leak caused OOM

Posted by GitBox <gi...@apache.org>.

kahle23 commented on issue #467:
URL: https://github.com/apache/rocketmq-spring/issues/467#issuecomment-1201005784

   ![image](https://user-images.githubusercontent.com/16236755/182127407-539ceca7-a3f7-4099-af45-ad6867759c6a.png)
   
   ![image](https://user-images.githubusercontent.com/16236755/182127518-665daaa6-ebcd-4575-99ea-08cd161b3d82.png)
   
   Is that it?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@rocketmq.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [rocketmq-spring] kahle23 commented on issue #467: Heap memory leak caused OOM

Posted by GitBox <gi...@apache.org>.

kahle23 commented on issue #467:
URL: https://github.com/apache/rocketmq-spring/issues/467#issuecomment-1193449839

   > > So far I've switched to rocketmq-client 4.9.3 and everything works fine without changing any configuration. PullThresholdForTopic and PullThresholdSizeForTopic I tried a variety of, still can appear
   > 
   > did not user rocketmq-spring-boot-starter 2.2.2。 use rocketmq-client 4.9.3 directly no problem?
   
   Yes, i'm use rocketmq-client 4.9.3 is ok in the same MQ.  That's why I think it's rocketmq-spring-boot-starter 2.2.2 bug, because nothing else has changed


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@rocketmq.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [rocketmq-spring] panzhi33 commented on issue #467: Heap memory leak caused OOM

Posted by GitBox <gi...@apache.org>.

panzhi33 commented on issue #467:
URL: https://github.com/apache/rocketmq-spring/issues/467#issuecomment-1193803710

   ![image](https://user-images.githubusercontent.com/29321745/180744264-1d1a1b77-a02c-47b1-a1d4-ec1e5441c04a.png)
   Look at the call chain here


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@rocketmq.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [rocketmq-spring] panzhi33 commented on issue #467: Heap memory leak caused OOM

Posted by GitBox <gi...@apache.org>.

panzhi33 commented on issue #467:
URL: https://github.com/apache/rocketmq-spring/issues/467#issuecomment-1201899472

   ![image](https://user-images.githubusercontent.com/29321745/182269653-6fa3941d-8ec3-4054-91ff-c5e7a587f21d.png)
   Here, want to know where this object is held for a long time


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@rocketmq.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org