You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pulsar.apache.org by GitBox <gi...@apache.org> on 2022/01/25 11:05:40 UTC
[GitHub] [pulsar] gaozhangmin opened a new pull request #13939: Fix potential heap memory leak
gaozhangmin opened a new pull request #13939:
URL: https://github.com/apache/pulsar/pull/13939
### Motivation
Cursors in `org.apache.bookkeeper.mledger.impl.ManagedLedgerImpl#waitingCursors` only be removed by `notifyCursors` when there were no new entries added, but cursors were added every 10ms.
```
ledger.getScheduledExecutor()
.schedule(() -> checkForNewEntries(op, callback, ctx),
config.getNewEntriesCheckDelayInMillis(), TimeUnit.MILLISECONDS);
```
This will cause memory leak, If there are no new entry added.
### Modifications
1、Change ConcurrentLinkedQueue waitingCursors to concurrentHashMap waitingCursorsMap, don't add cursors deduplicated.
2、remove cursors in waitingCursors when `deactivateCursor`.
If this fix is properly, I will add unit test afterly.
### Documentation
Check the box below or label this PR directly (if you have committer privilege).
Need to update docs?
- [ ] `doc-required`
(If you need help on updating docs, create a doc issue)
- [ ] `no-need-doc`
(Please explain why)
- [ ] `doc`
(If this PR contains doc changes)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] gaozhangmin commented on pull request #13939: Fix potential heap memory leak
Posted by GitBox <gi...@apache.org>.
gaozhangmin commented on pull request #13939:
URL: https://github.com/apache/pulsar/pull/13939#issuecomment-1040098527
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] gaozhangmin commented on pull request #13939: Fix potential heap memory leak
Posted by GitBox <gi...@apache.org>.
gaozhangmin commented on pull request #13939:
URL: https://github.com/apache/pulsar/pull/13939#issuecomment-1041038628
/pulsarbot run-failure-checks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] eolivelli merged pull request #13939: [Broker] waitingCursors potential heap memory leak
Posted by GitBox <gi...@apache.org>.
eolivelli merged pull request #13939:
URL: https://github.com/apache/pulsar/pull/13939
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] gaozhangmin commented on pull request #13939: [Broker] waitingCursors potential heap memory leak
Posted by GitBox <gi...@apache.org>.
gaozhangmin commented on pull request #13939:
URL: https://github.com/apache/pulsar/pull/13939#issuecomment-1046817276
/pulsarbot run-failure-checks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] gaozhangmin commented on pull request #13939: Fix potential heap memory leak
Posted by GitBox <gi...@apache.org>.
gaozhangmin commented on pull request #13939:
URL: https://github.com/apache/pulsar/pull/13939#issuecomment-1040174398
/pulsarbot run-failure-checks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] gaozhangmin removed a comment on pull request #13939: Fix potential heap memory leak
Posted by GitBox <gi...@apache.org>.
gaozhangmin removed a comment on pull request #13939:
URL: https://github.com/apache/pulsar/pull/13939#issuecomment-1041038628
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] gaozhangmin commented on pull request #13939: Fix potential heap memory leak
Posted by GitBox <gi...@apache.org>.
gaozhangmin commented on pull request #13939:
URL: https://github.com/apache/pulsar/pull/13939#issuecomment-1038691199
/pulsarbot run-failure-checks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] gaozhangmin commented on pull request #13939: [Broker] waitingCursors potential heap memory leak
Posted by GitBox <gi...@apache.org>.
gaozhangmin commented on pull request #13939:
URL: https://github.com/apache/pulsar/pull/13939#issuecomment-1046765606
/pulsarbot run-failure-checks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] github-actions[bot] commented on pull request #13939: Fix potential heap memory leak
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #13939:
URL: https://github.com/apache/pulsar/pull/13939#issuecomment-1021117751
@gaozhangmin:Thanks for your contribution. For this PR, do we need to update docs?
(The [PR template contains info about doc](https://github.com/apache/pulsar/blob/master/.github/PULL_REQUEST_TEMPLATE.md#documentation), which helps others know more about the changes. Can you provide doc-related info in this and future PR descriptions? Thanks)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] gaozhangmin commented on pull request #13939: Fix potential heap memory leak
Posted by GitBox <gi...@apache.org>.
gaozhangmin commented on pull request #13939:
URL: https://github.com/apache/pulsar/pull/13939#issuecomment-1021800310
> @gaozhangmin I'm not sure I see the case where the cursor is added multiple times:
>
> ```
> ledger.getScheduledExecutor()
> .schedule(() -> checkForNewEntries(op, callback, ctx),
> config.getNewEntriesCheckDelayInMillis(), TimeUnit.MILLISECONDS);
> ```
>
> This is called only the first time, as a kind of optimization to avoid inserting in the queue for cases where consumer and producers are very close. After the first time is unsuccessful, then we insert in the queue and wait for the notification. There shouldn't be more than 1 entry per cursor in that queue.
@merlimat You can verify by this test
```
@Test
public void testSub() throws Exception {
final String topicName = "persistent://prop/ns-abc/stuckSubscriptionTopic";
for (int i=0;i<10000;i++) {
Consumer<String> consumer1 = pulsarClient.newConsumer(Schema.STRING).topic(topicName)
.subscriptionType(SubscriptionType.Failover).subscriptionName("test").subscribe();
consumer1.close();
}
PersistentTopic topicRef = (PersistentTopic) pulsar.getBrokerService().getTopicReference(topicName).get();
ManagedLedgerImpl ml = (ManagedLedgerImpl)(topicRef.ledger);
System.out.println(ml.getWaitingCursorsCount());
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] hangc0276 commented on pull request #13939: Fix potential heap memory leak
Posted by GitBox <gi...@apache.org>.
hangc0276 commented on pull request #13939:
URL: https://github.com/apache/pulsar/pull/13939#issuecomment-1038709830
@liudezhi2098 Please help review this Pr, thanks a lot.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] gaozhangmin commented on a change in pull request #13939: [Broker] waitingCursors potential heap memory leak
Posted by GitBox <gi...@apache.org>.
gaozhangmin commented on a change in pull request #13939:
URL: https://github.com/apache/pulsar/pull/13939#discussion_r811009951
##########
File path: managed-ledger/src/main/java/org/apache/bookkeeper/mledger/ManagedLedger.java
##########
@@ -292,6 +292,13 @@ ManagedCursor newNonDurableCursor(Position startPosition, String subscriptionNam
*/
void deleteCursor(String name) throws InterruptedException, ManagedLedgerException;
+ /**
+ * Remove a ManagedCursor from this ManagedLedger's waitingCursors.
+ *
+ * @param cursor the ManagedCursor
+ */
+ void removeWaitingCursor(ManagedCursor cursor);
Review comment:
@codelipenghui
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] gaozhangmin commented on a change in pull request #13939: [Broker] waitingCursors potential heap memory leak
Posted by GitBox <gi...@apache.org>.
gaozhangmin commented on a change in pull request #13939:
URL: https://github.com/apache/pulsar/pull/13939#discussion_r810990107
##########
File path: managed-ledger/src/main/java/org/apache/bookkeeper/mledger/ManagedLedger.java
##########
@@ -292,6 +292,13 @@ ManagedCursor newNonDurableCursor(Position startPosition, String subscriptionNam
*/
void deleteCursor(String name) throws InterruptedException, ManagedLedgerException;
+ /**
+ * Remove a ManagedCursor from this ManagedLedger's waitingCursors.
+ *
+ * @param cursor the ManagedCursor
+ */
+ void removeWaitingCursor(ManagedCursor cursor);
Review comment:
`deactivateCursor` is widely used not only by closeConsumer, We cannot change the behavior of `deactivateCursor`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] gaozhangmin commented on pull request #13939: Fix potential heap memory leak
Posted by GitBox <gi...@apache.org>.
gaozhangmin commented on pull request #13939:
URL: https://github.com/apache/pulsar/pull/13939#issuecomment-1038661280
@codelipenghui @merlimat @lhotari @Jason918 @hangc0276 pTAL
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] merlimat commented on pull request #13939: Fix potential heap memory leak
Posted by GitBox <gi...@apache.org>.
merlimat commented on pull request #13939:
URL: https://github.com/apache/pulsar/pull/13939#issuecomment-1021342125
@gaozhangmin I'm not sure I see the case where the cursor is added multiple times:
```
ledger.getScheduledExecutor()
.schedule(() -> checkForNewEntries(op, callback, ctx),
config.getNewEntriesCheckDelayInMillis(), TimeUnit.MILLISECONDS);
```
This is called only the first time, as a kind of optimization to avoid inserting in the queue for cases where consumer and producers are very close. After the first time is unsuccessful, then we insert in the queue and wait for the notification. There shouldn't be more than 1 entry per cursor in that queue.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] merlimat commented on pull request #13939: Fix potential heap memory leak
Posted by GitBox <gi...@apache.org>.
merlimat commented on pull request #13939:
URL: https://github.com/apache/pulsar/pull/13939#issuecomment-1021342125
@gaozhangmin I'm not sure I see the case where the cursor is added multiple times:
```
ledger.getScheduledExecutor()
.schedule(() -> checkForNewEntries(op, callback, ctx),
config.getNewEntriesCheckDelayInMillis(), TimeUnit.MILLISECONDS);
```
This is called only the first time, as a kind of optimization to avoid inserting in the queue for cases where consumer and producers are very close. After the first time is unsuccessful, then we insert in the queue and wait for the notification. There shouldn't be more than 1 entry per cursor in that queue.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] github-actions[bot] commented on pull request #13939: Fix potential heap memory leak
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #13939:
URL: https://github.com/apache/pulsar/pull/13939#issuecomment-1021117751
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] gaozhangmin commented on pull request #13939: Fix potential heap memory leak
Posted by GitBox <gi...@apache.org>.
gaozhangmin commented on pull request #13939:
URL: https://github.com/apache/pulsar/pull/13939#issuecomment-1041526767
@merlimat PTAL
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] codelipenghui commented on a change in pull request #13939: [Broker] waitingCursors potential heap memory leak
Posted by GitBox <gi...@apache.org>.
codelipenghui commented on a change in pull request #13939:
URL: https://github.com/apache/pulsar/pull/13939#discussion_r810987177
##########
File path: managed-ledger/src/main/java/org/apache/bookkeeper/mledger/ManagedLedger.java
##########
@@ -292,6 +292,13 @@ ManagedCursor newNonDurableCursor(Position startPosition, String subscriptionNam
*/
void deleteCursor(String name) throws InterruptedException, ManagedLedgerException;
+ /**
+ * Remove a ManagedCursor from this ManagedLedger's waitingCursors.
+ *
+ * @param cursor the ManagedCursor
+ */
+ void removeWaitingCursor(ManagedCursor cursor);
Review comment:
Is it better to improve the method `deactivateCursor(ManagedCursor cursor)` to remove the cursor from the `waitingCursors`? Looks like we have removed the cursor from the `activeCursors` but does not remove it from the `waitingCursors`, so that we don't need to introduce a new method for ManagedLedger.
##########
File path: pulsar-broker/src/test/java/org/apache/pulsar/broker/admin/CreateSubscriptionTest.java
##########
@@ -348,4 +352,18 @@ public void createSubscriptionBySpecifyingStringPosition() throws IOException, P
producer.close();
}
+
+ @Test
+ public void testWaitingCurosrCausedMemoryLeak() throws Exception {
+ String topic = "persistent://my-property/my-ns/my-topic";
+ for (int i=0; i<10; i++) {
Review comment:
```suggestion
for (int i = 0; i < 10; i ++) {
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] github-actions[bot] commented on pull request #13939: Fix potential heap memory leak
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #13939:
URL: https://github.com/apache/pulsar/pull/13939#issuecomment-1021126301
@gaozhangmin:Thanks for providing doc info!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] liudezhi2098 commented on pull request #13939: Fix potential heap memory leak
Posted by GitBox <gi...@apache.org>.
liudezhi2098 commented on pull request #13939:
URL: https://github.com/apache/pulsar/pull/13939#issuecomment-1021912776
This maybe will not cause memory leak, it's not a periodic task. constantly creating and shutting down consumers can lead to this problem.
```
ledger.getScheduledExecutor()
.schedule(() -> checkForNewEntries(op, callback, ctx),
config.getNewEntriesCheckDelayInMillis(), TimeUnit.MILLISECONDS);
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] gaozhangmin commented on pull request #13939: Fix potential heap memory leak
Posted by GitBox <gi...@apache.org>.
gaozhangmin commented on pull request #13939:
URL: https://github.com/apache/pulsar/pull/13939#issuecomment-1040098527
/pulsarbot run-failure-checks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org