You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pulsar.apache.org by GitBox <gi...@apache.org> on 2019/07/09 17:02:34 UTC
[GitHub] [pulsar] ArtemShaban opened a new issue #4697: Memory leak in
Pulsar 2.3.1
ArtemShaban opened a new issue #4697: Memory leak in Pulsar 2.3.1
URL: https://github.com/apache/pulsar/issues/4697
Not far ago we tried update our production to Pulsar 2.3.1 from 2.2.1 version. But after one-two days working we noticed big CPU and memory consumption.
We did memory dumps and found suspicious objects.
In all dumps of brokers there are huge count of org.apache.bookkeeper.mledger.impl.ManagedLedgerImpl and a lot of big objects of java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue
https://www.dropbox.com/s/jde4h7yesu4i0a5/Screenshot%202019-07-09%2019.45.25.png?dl=0
https://www.dropbox.com/s/pcewv24p4mx73dj/Screenshot%202019-07-09%2019.46.05.png?dl=0
and this 2 types of objects occupy most part of heap:
https://www.dropbox.com/s/kx213bfvwgbgjhk/Screenshot%202019-07-09%2019.49.12.png?dl=0
```Problem Suspect 1
29 instances of "java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue", loaded by "<system class loader>" occupy 3,834,885,672 (69.63%) bytes.
Biggest instances:
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue @ 0x5c19ba9c0 - 486,434,080 (8.83%) bytes.
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue @ 0x5c19bc680 - 480,259,160 (8.72%) bytes.
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue @ 0x5c19bcb40 - 479,897,832 (8.71%) bytes.
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue @ 0x5c19ba768 - 479,323,032 (8.70%) bytes.
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue @ 0x5c19bcd88 - 479,216,656 (8.70%) bytes.
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue @ 0x5c19ba2c8 - 478,129,544 (8.68%) bytes.
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue @ 0x5c19ba510 - 477,094,040 (8.66%) bytes.
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue @ 0x5c19bc8e8 - 474,507,096 (8.62%) bytes.
78,804 instances of "org.apache.bookkeeper.mledger.impl.ManagedLedgerImpl", loaded by "sun.misc.Launcher$AppClassLoader @ 0x5c1401cd0" occupy 1,139,905,784 (20.70%) bytes.
Keywords
org.apache.bookkeeper.mledger.impl.ManagedLedgerImpl
sun.misc.Launcher$AppClassLoader @ 0x5c14
```
Also I compared memory dumps of 2.3.1 and 2.2.1 - and 2.2.1 has no such problems.
I know and there is new updates of Pulsar, and I did't found any notions about such case.
We tried reproduce this case in test environment - unsuccessfully.
In our production case we create/delete a lot of topics (probably this note will be helpful).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services