You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pulsar.apache.org by GitBox <gi...@apache.org> on 2020/03/05 19:57:00 UTC
[GitHub] [pulsar] racorn opened a new pull request #6485: Fix memory leak
when running topic compaction.
racorn opened a new pull request #6485: Fix memory leak when running topic compaction.
URL: https://github.com/apache/pulsar/pull/6485
- Redefine contract for `RawBacthConverter.rebatchMessage`
- Ensure message and rebatched message is released (closed)
during phase two loop (`TwoPhaseCompactor`)
- Ensure all ledger entries are closed when reading one entry
for caching (`CompactedTopicImpl`)
Modify RawReaderTest.testBatchingRebatch to show new contract.
Fixes #6482
### Motivation
Prevent topic compaction from leaking direct memory
### Modifications
Several leaks were discovered using Netty leak detection and code review.
* `CompactedTopicImpl.readOneMessageId` would get an `Enumeration` of `LedgerEntry`, but did not release the underlying buffers. Fix: iterate though the `Enumeration` and release underlying buffer. Instead of logging the case where the `Enumeration` did not contain any elements, complete the future exceptionally with the message (will be logged by Caffeine).
* Two main sources of leak in `TwoPhaseCompactor`. The `RawBacthConverter.rebatchMessage` method failed to close/release a `ByteBuf` (uncompressedPayload). Also, the return ByteBuf of `RawBacthConverter.rebatchMessage` was not closed. The first one was easy to fix (release buffer), to fix the second one and make the code easier to read, I decided to not let `RawBacthConverter.rebatchMessage` close the message read from the topic, instead the message read from the topic can be closed in a try/finally clause surrounding most of the method body handing a message from a topic (in phase two loop). Then if a new message was produced by `RawBacthConverter.rebatchMessage` we check that after we have added the message to the compact ledger and release the message.
### Verifying this change
Modified `RawReaderTest.testBatchingRebatch` to show new contract.
One can run the test described to reproduce the issue, to verify no leak is detected.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [pulsar] racorn commented on issue #6485: Fix memory leak when
running topic compaction.
Posted by GitBox <gi...@apache.org>.
racorn commented on issue #6485: Fix memory leak when running topic compaction.
URL: https://github.com/apache/pulsar/pull/6485#issuecomment-595484120
/pulsarbot run-failure-checks
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [pulsar] jiazhai commented on issue #6485: Fix memory leak when
running topic compaction.
Posted by GitBox <gi...@apache.org>.
jiazhai commented on issue #6485: Fix memory leak when running topic compaction.
URL: https://github.com/apache/pulsar/pull/6485#issuecomment-598574553
cherry-picked into branch-2.5
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [pulsar] sijie merged pull request #6485: Fix memory leak when
running topic compaction.
Posted by GitBox <gi...@apache.org>.
sijie merged pull request #6485: Fix memory leak when running topic compaction.
URL: https://github.com/apache/pulsar/pull/6485
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [pulsar] racorn commented on issue #6485: Fix memory leak when
running topic compaction.
Posted by GitBox <gi...@apache.org>.
racorn commented on issue #6485: Fix memory leak when running topic compaction.
URL: https://github.com/apache/pulsar/pull/6485#issuecomment-595492671
/pulsarbot run-failure-checks
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [pulsar] racorn commented on issue #6485: Fix memory leak when
running topic compaction.
Posted by GitBox <gi...@apache.org>.
racorn commented on issue #6485: Fix memory leak when running topic compaction.
URL: https://github.com/apache/pulsar/pull/6485#issuecomment-595475313
/pulsarbot run-failure-checks
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services