You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pulsar.apache.org by GitBox <gi...@apache.org> on 2021/05/31 14:36:45 UTC
[GitHub] [pulsar] lhotari opened a new pull request #10755: [ML] Fix ByteBuf leaks when execution fails
lhotari opened a new pull request #10755:
URL: https://github.com/apache/pulsar/pull/10755
### Motivation
The repro case in #10738 reproduces some Netty ByteBuf leaks that are detected by the Netty Leak detector. There are a few exceptional execution paths where the data ByteBuf of OpAddEntry isn't released.
### Modifications
- release the data ByteBuf in a try-finally block
- release the data ByteBuf when execution is rejected in the invalid state case where the entry isn't the first in the queue.
This seems to stop the detected leaks in #10738 case, but it doesn't fix the issue where a lot of direct memory is consumed and the broker runs of out of direct memory.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] devinbost commented on a change in pull request #10755: [ML] Fix ByteBuf leaks when execution fails
Posted by GitBox <gi...@apache.org>.
devinbost commented on a change in pull request #10755:
URL: https://github.com/apache/pulsar/pull/10755#discussion_r642806337
##########
File path: managed-ledger/src/main/java/org/apache/bookkeeper/mledger/impl/OpAddEntry.java
##########
@@ -178,8 +178,21 @@ public void addComplete(int rc, final LedgerHandle lh, long entryId, Object ctx)
@Override
public void safeRun() {
// Remove this entry from the head of the pending queue
- OpAddEntry firstInQueue = ml.pendingAddEntries.poll();
- checkArgument(this == firstInQueue);
+ OpAddEntry firstInQueue = ml.pollPendingAddEntry();
+ if (this != firstInQueue) {
+ log.warn("Expected OpAddEntry wasn't first in queue. expected={} actual={}", this, firstInQueue);
+ ManagedLedgerException managedLedgerException =
+ new ManagedLedgerException("Expected OpAddEntry wasn't first in queue.");
+ // fail both entries
+ try {
+ if (firstInQueue != null) {
+ firstInQueue.failed(managedLedgerException);
+ }
+ } finally {
+ failed(managedLedgerException);
Review comment:
Regarding the comment from @codelipenghui , I don't remember seeing any logic to handle that case (of removing from pending entries upon failure), so should we just add that logic, or would we want to investigate the cause of why the `get` failed?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] lhotari closed pull request #10755: [ML] Fix ByteBuf leaks when execution fails
Posted by GitBox <gi...@apache.org>.
lhotari closed pull request #10755:
URL: https://github.com/apache/pulsar/pull/10755
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] eolivelli commented on a change in pull request #10755: [ML] Fix ByteBuf leaks when execution fails
Posted by GitBox <gi...@apache.org>.
eolivelli commented on a change in pull request #10755:
URL: https://github.com/apache/pulsar/pull/10755#discussion_r642649741
##########
File path: managed-ledger/src/main/java/org/apache/bookkeeper/mledger/impl/OpAddEntry.java
##########
@@ -201,42 +214,47 @@ public void safeRun() {
// `data` will be released in `closeComplete`
ledger.asyncClose(this, ctx);
} else {
- updateLatency();
- AddEntryCallback cb = callbackUpdater.getAndSet(this, null);
- if (cb != null) {
- cb.addComplete(lastEntry, data.asReadOnly(), ctx);
- ml.notifyCursors();
- ml.notifyWaitingEntryCallBacks();
- ReferenceCountUtil.release(data);
- this.recycle();
- } else {
+ ByteBuf data = this.data;
+ try {
+ updateLatency();
+ AddEntryCallback cb = callbackUpdater.getAndSet(this, null);
+ if (cb != null) {
+ cb.addComplete(lastEntry, data.asReadOnly(), ctx);
+ ml.notifyCursors();
+ ml.notifyWaitingEntryCallBacks();
+ this.recycle();
Review comment:
it looks like now we are calling `recycle` before `ReferenceCountUtil.release(data);`
may it be a problem ?
##########
File path: managed-ledger/src/main/java/org/apache/bookkeeper/mledger/impl/OpAddEntry.java
##########
@@ -201,42 +214,47 @@ public void safeRun() {
// `data` will be released in `closeComplete`
ledger.asyncClose(this, ctx);
} else {
- updateLatency();
- AddEntryCallback cb = callbackUpdater.getAndSet(this, null);
- if (cb != null) {
- cb.addComplete(lastEntry, data.asReadOnly(), ctx);
- ml.notifyCursors();
- ml.notifyWaitingEntryCallBacks();
- ReferenceCountUtil.release(data);
- this.recycle();
- } else {
+ ByteBuf data = this.data;
+ try {
+ updateLatency();
+ AddEntryCallback cb = callbackUpdater.getAndSet(this, null);
+ if (cb != null) {
+ cb.addComplete(lastEntry, data.asReadOnly(), ctx);
+ ml.notifyCursors();
+ ml.notifyWaitingEntryCallBacks();
+ this.recycle();
+ }
+ } finally {
ReferenceCountUtil.release(data);
}
}
}
@Override
public void closeComplete(int rc, LedgerHandle lh, Object ctx) {
- checkArgument(ledger.getId() == lh.getId(), "ledgerId %s doesn't match with acked ledgerId %s", ledger.getId(),
- lh.getId());
-
- if (rc == BKException.Code.OK) {
- log.debug("Successfully closed ledger {}", lh.getId());
- } else {
- log.warn("Error when closing ledger {}. Status={}", lh.getId(), BKException.getMessage(rc));
- }
+ ByteBuf data = this.data;
+ try {
+ checkArgument(ledger.getId() == lh.getId(), "ledgerId %s doesn't match with acked ledgerId %s",
+ ledger.getId(),
+ lh.getId());
+
+ if (rc == BKException.Code.OK) {
+ log.debug("Successfully closed ledger {}", lh.getId());
+ } else {
+ log.warn("Error when closing ledger {}. Status={}", lh.getId(), BKException.getMessage(rc));
+ }
- ml.ledgerClosed(lh);
- updateLatency();
+ ml.ledgerClosed(lh);
+ updateLatency();
- AddEntryCallback cb = callbackUpdater.getAndSet(this, null);
- if (cb != null) {
- cb.addComplete(PositionImpl.get(lh.getId(), entryId), data.asReadOnly(), ctx);
- ml.notifyCursors();
- ml.notifyWaitingEntryCallBacks();
- ReferenceCountUtil.release(data);
- this.recycle();
- } else {
+ AddEntryCallback cb = callbackUpdater.getAndSet(this, null);
+ if (cb != null) {
+ cb.addComplete(PositionImpl.get(lh.getId(), entryId), data.asReadOnly(), ctx);
+ ml.notifyCursors();
+ ml.notifyWaitingEntryCallBacks();
+ this.recycle();
Review comment:
the same here
##########
File path: managed-ledger/src/main/java/org/apache/bookkeeper/mledger/impl/ManagedLedgerImpl.java
##########
@@ -1536,6 +1538,11 @@ public synchronized void updateLedgersIdsComplete(Stat stat) {
}
}
+ synchronized OpAddEntry pollPendingAddEntry() {
+ return pendingAddEntries.poll();
Review comment:
should we make pendingAddEntries 'private' ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] lhotari commented on a change in pull request #10755: [ML] Fix ByteBuf leaks when execution fails
Posted by GitBox <gi...@apache.org>.
lhotari commented on a change in pull request #10755:
URL: https://github.com/apache/pulsar/pull/10755#discussion_r642782389
##########
File path: managed-ledger/src/main/java/org/apache/bookkeeper/mledger/impl/OpAddEntry.java
##########
@@ -178,8 +178,21 @@ public void addComplete(int rc, final LedgerHandle lh, long entryId, Object ctx)
@Override
public void safeRun() {
// Remove this entry from the head of the pending queue
- OpAddEntry firstInQueue = ml.pendingAddEntries.poll();
- checkArgument(this == firstInQueue);
+ OpAddEntry firstInQueue = ml.pollPendingAddEntry();
+ if (this != firstInQueue) {
+ log.warn("Expected OpAddEntry wasn't first in queue. expected={} actual={}", this, firstInQueue);
+ ManagedLedgerException managedLedgerException =
+ new ManagedLedgerException("Expected OpAddEntry wasn't first in queue.");
+ // fail both entries
+ try {
+ if (firstInQueue != null) {
+ firstInQueue.failed(managedLedgerException);
+ }
+ } finally {
+ failed(managedLedgerException);
Review comment:
that's true, but I guess that this PR is doing the right thing by calling `.failed` on the entries to discard them if the execution gets to this point? The previous behavior of using `checkArgument(this == firstInQueue)` would throw an `IllegalArgumentException` and leak ByteBufs on the discarded OpAddEntrys. Isn't this PR an improvement to that since it blocks the memory leak?
The repro case in #10738 was able to reproduce these errors in the original code amoung others.
@codelipenghui do you expect that some changes would be made to this PR?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] lhotari commented on a change in pull request #10755: [ML] Fix ByteBuf leaks when execution fails
Posted by GitBox <gi...@apache.org>.
lhotari commented on a change in pull request #10755:
URL: https://github.com/apache/pulsar/pull/10755#discussion_r642803038
##########
File path: managed-ledger/src/main/java/org/apache/bookkeeper/mledger/impl/OpAddEntry.java
##########
@@ -178,8 +178,21 @@ public void addComplete(int rc, final LedgerHandle lh, long entryId, Object ctx)
@Override
public void safeRun() {
// Remove this entry from the head of the pending queue
- OpAddEntry firstInQueue = ml.pendingAddEntries.poll();
- checkArgument(this == firstInQueue);
+ OpAddEntry firstInQueue = ml.pollPendingAddEntry();
+ if (this != firstInQueue) {
+ log.warn("Expected OpAddEntry wasn't first in queue. expected={} actual={}", this, firstInQueue);
+ ManagedLedgerException managedLedgerException =
+ new ManagedLedgerException("Expected OpAddEntry wasn't first in queue.");
+ // fail both entries
+ try {
+ if (firstInQueue != null) {
+ firstInQueue.failed(managedLedgerException);
+ }
+ } finally {
+ failed(managedLedgerException);
Review comment:
@devinbost if the arguments get rejected and one of the argument is a [Netty reference counted object](https://netty.io/wiki/reference-counted-objects.html), it would cause a memory leak if the called method is responsible for releasing the object.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] devinbost commented on a change in pull request #10755: [ML] Fix ByteBuf leaks when execution fails
Posted by GitBox <gi...@apache.org>.
devinbost commented on a change in pull request #10755:
URL: https://github.com/apache/pulsar/pull/10755#discussion_r642793393
##########
File path: managed-ledger/src/main/java/org/apache/bookkeeper/mledger/impl/ManagedLedgerImpl.java
##########
@@ -1536,6 +1538,11 @@ public synchronized void updateLedgersIdsComplete(Stat stat) {
}
}
+ synchronized OpAddEntry pollPendingAddEntry() {
+ return pendingAddEntries.poll();
Review comment:
You could mark it with `@VisibleForTesting`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] lhotari commented on a change in pull request #10755: [ML] Fix ByteBuf leaks when execution fails
Posted by GitBox <gi...@apache.org>.
lhotari commented on a change in pull request #10755:
URL: https://github.com/apache/pulsar/pull/10755#discussion_r642666631
##########
File path: managed-ledger/src/main/java/org/apache/bookkeeper/mledger/impl/OpAddEntry.java
##########
@@ -201,42 +214,47 @@ public void safeRun() {
// `data` will be released in `closeComplete`
ledger.asyncClose(this, ctx);
} else {
- updateLatency();
- AddEntryCallback cb = callbackUpdater.getAndSet(this, null);
- if (cb != null) {
- cb.addComplete(lastEntry, data.asReadOnly(), ctx);
- ml.notifyCursors();
- ml.notifyWaitingEntryCallBacks();
- ReferenceCountUtil.release(data);
- this.recycle();
- } else {
+ ByteBuf data = this.data;
+ try {
+ updateLatency();
+ AddEntryCallback cb = callbackUpdater.getAndSet(this, null);
+ if (cb != null) {
+ cb.addComplete(lastEntry, data.asReadOnly(), ctx);
+ ml.notifyCursors();
+ ml.notifyWaitingEntryCallBacks();
+ this.recycle();
+ }
+ } finally {
ReferenceCountUtil.release(data);
}
}
}
@Override
public void closeComplete(int rc, LedgerHandle lh, Object ctx) {
- checkArgument(ledger.getId() == lh.getId(), "ledgerId %s doesn't match with acked ledgerId %s", ledger.getId(),
- lh.getId());
-
- if (rc == BKException.Code.OK) {
- log.debug("Successfully closed ledger {}", lh.getId());
- } else {
- log.warn("Error when closing ledger {}. Status={}", lh.getId(), BKException.getMessage(rc));
- }
+ ByteBuf data = this.data;
+ try {
+ checkArgument(ledger.getId() == lh.getId(), "ledgerId %s doesn't match with acked ledgerId %s",
+ ledger.getId(),
+ lh.getId());
+
+ if (rc == BKException.Code.OK) {
+ log.debug("Successfully closed ledger {}", lh.getId());
+ } else {
+ log.warn("Error when closing ledger {}. Status={}", lh.getId(), BKException.getMessage(rc));
+ }
- ml.ledgerClosed(lh);
- updateLatency();
+ ml.ledgerClosed(lh);
+ updateLatency();
- AddEntryCallback cb = callbackUpdater.getAndSet(this, null);
- if (cb != null) {
- cb.addComplete(PositionImpl.get(lh.getId(), entryId), data.asReadOnly(), ctx);
- ml.notifyCursors();
- ml.notifyWaitingEntryCallBacks();
- ReferenceCountUtil.release(data);
- this.recycle();
- } else {
+ AddEntryCallback cb = callbackUpdater.getAndSet(this, null);
+ if (cb != null) {
+ cb.addComplete(PositionImpl.get(lh.getId(), entryId), data.asReadOnly(), ctx);
+ ml.notifyCursors();
+ ml.notifyWaitingEntryCallBacks();
+ this.recycle();
Review comment:
Not a problem.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] lhotari commented on a change in pull request #10755: [ML] Fix ByteBuf leaks when execution fails
Posted by GitBox <gi...@apache.org>.
lhotari commented on a change in pull request #10755:
URL: https://github.com/apache/pulsar/pull/10755#discussion_r642666555
##########
File path: managed-ledger/src/main/java/org/apache/bookkeeper/mledger/impl/OpAddEntry.java
##########
@@ -201,42 +214,47 @@ public void safeRun() {
// `data` will be released in `closeComplete`
ledger.asyncClose(this, ctx);
} else {
- updateLatency();
- AddEntryCallback cb = callbackUpdater.getAndSet(this, null);
- if (cb != null) {
- cb.addComplete(lastEntry, data.asReadOnly(), ctx);
- ml.notifyCursors();
- ml.notifyWaitingEntryCallBacks();
- ReferenceCountUtil.release(data);
- this.recycle();
- } else {
+ ByteBuf data = this.data;
+ try {
+ updateLatency();
+ AddEntryCallback cb = callbackUpdater.getAndSet(this, null);
+ if (cb != null) {
+ cb.addComplete(lastEntry, data.asReadOnly(), ctx);
+ ml.notifyCursors();
+ ml.notifyWaitingEntryCallBacks();
+ this.recycle();
Review comment:
No it's not a problem.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] devinbost commented on a change in pull request #10755: [ML] Fix ByteBuf leaks when execution fails
Posted by GitBox <gi...@apache.org>.
devinbost commented on a change in pull request #10755:
URL: https://github.com/apache/pulsar/pull/10755#discussion_r642796005
##########
File path: managed-ledger/src/main/java/org/apache/bookkeeper/mledger/impl/OpAddEntry.java
##########
@@ -178,8 +178,21 @@ public void addComplete(int rc, final LedgerHandle lh, long entryId, Object ctx)
@Override
public void safeRun() {
// Remove this entry from the head of the pending queue
- OpAddEntry firstInQueue = ml.pendingAddEntries.poll();
- checkArgument(this == firstInQueue);
+ OpAddEntry firstInQueue = ml.pollPendingAddEntry();
+ if (this != firstInQueue) {
+ log.warn("Expected OpAddEntry wasn't first in queue. expected={} actual={}", this, firstInQueue);
+ ManagedLedgerException managedLedgerException =
+ new ManagedLedgerException("Expected OpAddEntry wasn't first in queue.");
+ // fail both entries
+ try {
+ if (firstInQueue != null) {
+ firstInQueue.failed(managedLedgerException);
+ }
+ } finally {
+ failed(managedLedgerException);
Review comment:
@lhotari The `checkArgument(..)` call is used in a lot of places in Pulsar. Could it be an issue in other places as well?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] lhotari commented on pull request #10755: [ML] Fix ByteBuf leaks when execution fails
Posted by GitBox <gi...@apache.org>.
lhotari commented on pull request #10755:
URL: https://github.com/apache/pulsar/pull/10755#issuecomment-852270996
I discussed this PR with @merlimat . I'll split the PR into individual changes since this PR now contains multiple independent changes bundled in a single PR. I'll close this PR and open new PRs.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] devinbost commented on a change in pull request #10755: [ML] Fix ByteBuf leaks when execution fails
Posted by GitBox <gi...@apache.org>.
devinbost commented on a change in pull request #10755:
URL: https://github.com/apache/pulsar/pull/10755#discussion_r643400402
##########
File path: managed-ledger/src/main/java/org/apache/bookkeeper/mledger/impl/OpAddEntry.java
##########
@@ -178,8 +178,21 @@ public void addComplete(int rc, final LedgerHandle lh, long entryId, Object ctx)
@Override
public void safeRun() {
// Remove this entry from the head of the pending queue
- OpAddEntry firstInQueue = ml.pendingAddEntries.poll();
- checkArgument(this == firstInQueue);
+ OpAddEntry firstInQueue = ml.pollPendingAddEntry();
+ if (this != firstInQueue) {
+ log.warn("Expected OpAddEntry wasn't first in queue. expected={} actual={}", this, firstInQueue);
+ ManagedLedgerException managedLedgerException =
+ new ManagedLedgerException("Expected OpAddEntry wasn't first in queue.");
+ // fail both entries
+ try {
+ if (firstInQueue != null) {
+ firstInQueue.failed(managedLedgerException);
+ }
+ } finally {
+ failed(managedLedgerException);
Review comment:
Looks like it's only removed according to the state, but if the sanity check fails, it's not handled right.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] merlimat commented on a change in pull request #10755: [ML] Fix ByteBuf leaks when execution fails
Posted by GitBox <gi...@apache.org>.
merlimat commented on a change in pull request #10755:
URL: https://github.com/apache/pulsar/pull/10755#discussion_r643271542
##########
File path: managed-ledger/src/main/java/org/apache/bookkeeper/mledger/impl/OpAddEntry.java
##########
@@ -178,8 +178,21 @@ public void addComplete(int rc, final LedgerHandle lh, long entryId, Object ctx)
@Override
public void safeRun() {
// Remove this entry from the head of the pending queue
- OpAddEntry firstInQueue = ml.pendingAddEntries.poll();
- checkArgument(this == firstInQueue);
Review comment:
I'd leave this change out of this PR. This is a "logic" check. If this fails it means we really have a code bug issue and we cannot make many assumptions on how to recover from it.
##########
File path: managed-ledger/src/main/java/org/apache/bookkeeper/mledger/impl/OpAddEntry.java
##########
@@ -201,42 +214,47 @@ public void safeRun() {
// `data` will be released in `closeComplete`
ledger.asyncClose(this, ctx);
} else {
- updateLatency();
- AddEntryCallback cb = callbackUpdater.getAndSet(this, null);
- if (cb != null) {
- cb.addComplete(lastEntry, data.asReadOnly(), ctx);
- ml.notifyCursors();
- ml.notifyWaitingEntryCallBacks();
- ReferenceCountUtil.release(data);
- this.recycle();
- } else {
+ ByteBuf data = this.data;
+ try {
Review comment:
The change here LGTM, although I'd prefer to get it in a separate PR. Also, do you have a stack trace on why the exception would have been triggered here?
##########
File path: managed-ledger/src/main/java/org/apache/bookkeeper/mledger/impl/ManagedLedgerImpl.java
##########
@@ -1504,6 +1504,8 @@ public synchronized void updateLedgersIdsComplete(Stat stat) {
if (existsOp.ledger != null) {
existsOp.close();
existsOp = OpAddEntry.create(existsOp.ml, existsOp.data, existsOp.getNumberOfMessages(), existsOp.callback, existsOp.ctx);
+ // release the extra retain
+ ReferenceCountUtil.release(existsOp.data);
Review comment:
👍 I think this was introduced in #5942.
##########
File path: managed-ledger/src/main/java/org/apache/bookkeeper/mledger/impl/OpAddEntry.java
##########
@@ -201,42 +214,47 @@ public void safeRun() {
// `data` will be released in `closeComplete`
ledger.asyncClose(this, ctx);
} else {
- updateLatency();
- AddEntryCallback cb = callbackUpdater.getAndSet(this, null);
- if (cb != null) {
- cb.addComplete(lastEntry, data.asReadOnly(), ctx);
- ml.notifyCursors();
- ml.notifyWaitingEntryCallBacks();
- ReferenceCountUtil.release(data);
- this.recycle();
- } else {
+ ByteBuf data = this.data;
+ try {
+ updateLatency();
+ AddEntryCallback cb = callbackUpdater.getAndSet(this, null);
+ if (cb != null) {
+ cb.addComplete(lastEntry, data.asReadOnly(), ctx);
+ ml.notifyCursors();
+ ml.notifyWaitingEntryCallBacks();
+ this.recycle();
+ }
+ } finally {
ReferenceCountUtil.release(data);
}
}
}
@Override
public void closeComplete(int rc, LedgerHandle lh, Object ctx) {
- checkArgument(ledger.getId() == lh.getId(), "ledgerId %s doesn't match with acked ledgerId %s", ledger.getId(),
- lh.getId());
-
- if (rc == BKException.Code.OK) {
- log.debug("Successfully closed ledger {}", lh.getId());
- } else {
- log.warn("Error when closing ledger {}. Status={}", lh.getId(), BKException.getMessage(rc));
- }
+ ByteBuf data = this.data;
+ try {
+ checkArgument(ledger.getId() == lh.getId(), "ledgerId %s doesn't match with acked ledgerId %s",
Review comment:
Same comment for the other assertion above. This is really a code-sanity check. If this fails, it means things are **very** broken.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] codelipenghui commented on a change in pull request #10755: [ML] Fix ByteBuf leaks when execution fails
Posted by GitBox <gi...@apache.org>.
codelipenghui commented on a change in pull request #10755:
URL: https://github.com/apache/pulsar/pull/10755#discussion_r642736739
##########
File path: managed-ledger/src/main/java/org/apache/bookkeeper/mledger/impl/OpAddEntry.java
##########
@@ -178,8 +178,21 @@ public void addComplete(int rc, final LedgerHandle lh, long entryId, Object ctx)
@Override
public void safeRun() {
// Remove this entry from the head of the pending queue
- OpAddEntry firstInQueue = ml.pendingAddEntries.poll();
- checkArgument(this == firstInQueue);
+ OpAddEntry firstInQueue = ml.pollPendingAddEntry();
+ if (this != firstInQueue) {
+ log.warn("Expected OpAddEntry wasn't first in queue. expected={} actual={}", this, firstInQueue);
+ ManagedLedgerException managedLedgerException =
+ new ManagedLedgerException("Expected OpAddEntry wasn't first in queue.");
+ // fail both entries
+ try {
+ if (firstInQueue != null) {
+ firstInQueue.failed(managedLedgerException);
+ }
+ } finally {
+ failed(managedLedgerException);
Review comment:
Seems the OpAddEntry get failed but does not remove from the pending entries.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] lhotari commented on a change in pull request #10755: [ML] Fix ByteBuf leaks when execution fails
Posted by GitBox <gi...@apache.org>.
lhotari commented on a change in pull request #10755:
URL: https://github.com/apache/pulsar/pull/10755#discussion_r643256205
##########
File path: managed-ledger/src/main/java/org/apache/bookkeeper/mledger/impl/OpAddEntry.java
##########
@@ -178,8 +178,21 @@ public void addComplete(int rc, final LedgerHandle lh, long entryId, Object ctx)
@Override
public void safeRun() {
// Remove this entry from the head of the pending queue
- OpAddEntry firstInQueue = ml.pendingAddEntries.poll();
- checkArgument(this == firstInQueue);
+ OpAddEntry firstInQueue = ml.pollPendingAddEntry();
+ if (this != firstInQueue) {
+ log.warn("Expected OpAddEntry wasn't first in queue. expected={} actual={}", this, firstInQueue);
+ ManagedLedgerException managedLedgerException =
+ new ManagedLedgerException("Expected OpAddEntry wasn't first in queue.");
+ // fail both entries
+ try {
+ if (firstInQueue != null) {
+ firstInQueue.failed(managedLedgerException);
+ }
+ } finally {
+ failed(managedLedgerException);
Review comment:
@devinbost there is some logic for that. https://github.com/apache/pulsar/blob/6a17b3eed6e18e925b133720482ecdaf246e5ef5/managed-ledger/src/main/java/org/apache/bookkeeper/mledger/impl/ManagedLedgerImpl.java#L716-L731
It's a bit hard to understand why the `addOperation` gets added to the `pendingAddEntries` initially. Why doesn't this happen after the state is checked? However I consider handling this a separate issue.
In Netty, the contract of reference counted objects is that ["The general rule of thumb is that the party that accesses a reference-counted object last is also responsible for the destruction of that reference-counted object."](https://netty.io/wiki/reference-counted-objects.html#who-destroys-it). This PR attempts to follow this rule. Therefore I'd suggest that we first process this PR and file separate issues to fix other problems.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] lhotari commented on a change in pull request #10755: [ML] Fix ByteBuf leaks when execution fails
Posted by GitBox <gi...@apache.org>.
lhotari commented on a change in pull request #10755:
URL: https://github.com/apache/pulsar/pull/10755#discussion_r642666782
##########
File path: managed-ledger/src/main/java/org/apache/bookkeeper/mledger/impl/ManagedLedgerImpl.java
##########
@@ -1536,6 +1538,11 @@ public synchronized void updateLedgersIdsComplete(Stat stat) {
}
}
+ synchronized OpAddEntry pollPendingAddEntry() {
+ return pendingAddEntries.poll();
Review comment:
I was thinking of doing that but there is some test code that accesses the field directly.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] lhotari commented on pull request #10755: [ML] Fix ByteBuf leaks when execution fails
Posted by GitBox <gi...@apache.org>.
lhotari commented on pull request #10755:
URL: https://github.com/apache/pulsar/pull/10755#issuecomment-851640752
@merlimat Please review. This seems to fix the memory leak that should up in #10738. The last commit, f176c15631959456d5edfe9526766dadb199e46a, in this PR made the difference.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org