You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pulsar.apache.org by GitBox <gi...@apache.org> on 2020/03/05 14:40:51 UTC
[GitHub] [pulsar] racorn opened a new issue #6482: Memory leak during
automatic topic compaction
racorn opened a new issue #6482: Memory leak during automatic topic compaction
URL: https://github.com/apache/pulsar/issues/6482
**Describe the bug**
When the automatic log compaction is triggered for a topic within a namespace with active compaction-threshold, there is severe leak of direct memory, eventually causing broker shutdown:
```
[bookkeeper-io-12-3] ERROR org.apache.pulsar.PulsarBrokerStarter - -- Shutting down - Received OOM exception: failed to allocate 16777216 byte(s) of direct memory (used: 6442450944, max: 6442450944)
io.netty.util.internal.OutOfDirectMemoryError: failed to allocate 16777216 byte(s) of direct memory (used: 6442450944, max: 6442450944)
at ...
```
Digging into this, it seems the phase two of compaction leaks most.
**To Reproduce**
Steps to reproduce the behavior:
1. start pulsar-broker with leak detection 'Paranoid'
`-Dpulsar.allocator.leak_detection=Paranoid`
2. Create namespace and set compaction-threshold
```
bin/pulsar-admin namespaces create -c pulsar-cluster default/compacted
bin/pulsar-admin namespaces set-compaction-threshold -t 4M default/compacted
```
3. Start producing messages on the topic, wait for compaction to be triggered.
4. grep for 'LEAK' in the pulsar broker log file
6. May take some time to wait for OOM error, but leaks should be logged at first compaction run
**Expected behavior**
Topic compaction should not leak memory
**Screenshots**
Showing typical direct memory usage before OOM
**Additional context**
Linux 4.15.x
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [pulsar] jiazhai commented on issue #6482: Memory leak during
automatic topic compaction
Posted by GitBox <gi...@apache.org>.
jiazhai commented on issue #6482: Memory leak during automatic topic compaction
URL: https://github.com/apache/pulsar/issues/6482#issuecomment-595568910
Thanks @racorn for your great help.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [pulsar] racorn commented on issue #6482: Memory leak during
automatic topic compaction
Posted by GitBox <gi...@apache.org>.
racorn commented on issue #6482: Memory leak during automatic topic compaction
URL: https://github.com/apache/pulsar/issues/6482#issuecomment-595268491
I will create a pull request to fix this.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [pulsar] sijie closed issue #6482: Memory leak during automatic
topic compaction
Posted by GitBox <gi...@apache.org>.
sijie closed issue #6482: Memory leak during automatic topic compaction
URL: https://github.com/apache/pulsar/issues/6482
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services