You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pulsar.apache.org by GitBox <gi...@apache.org> on 2022/03/14 05:27:18 UTC
[GitHub] [pulsar] mattisonchao opened a new pull request #14672: Fix ``rollCurrentLedgerIfFull`` to follow lazy creation of ledger
mattisonchao opened a new pull request #14672:
URL: https://github.com/apache/pulsar/pull/14672
### Motivation
The original ledger creation design was lazy, meaning that the ledger was only created when a new write operation was requested.
https://github.com/apache/pulsar/blob/ad2cc2d38280b7dd0f056ee981ec8d3b157e3526/managed-ledger/src/main/java/org/apache/bookkeeper/mledger/impl/ManagedLedgerImpl.java#L778-L786
However, the current ``rollCurrentLedgerIfFull`` logic is to create a ledger regardless of the condition (even if ``ManagedLedgerImpl#pendingAddEntries`` is empty)
https://github.com/apache/pulsar/blob/ad2cc2d38280b7dd0f056ee981ec8d3b157e3526/managed-ledger/src/main/java/org/apache/bookkeeper/mledger/impl/ManagedLedgerImpl.java#L1687-L1688
Therefore, we need to change to lazy creation according to the original design.
### Modifications
- Remove ``createLedgerAfterClosed`` invoke in the ``rollCurrentLedgerIfFull`` method.
### Verifying this change
- [x] Make sure that the change passes the CI checks.
### Documentation
- [x] `no-need-doc`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] mattisonchao commented on pull request #14672: Change ``rollCurrentLedgerIfFull`` logic to follow lazy creation of ledger
Posted by GitBox <gi...@apache.org>.
mattisonchao commented on pull request #14672:
URL: https://github.com/apache/pulsar/pull/14672#issuecomment-1067713515
@codelipenghui @BewareMyPower @eolivelli @lhotari @michaeljmarshall @Technoboy-
This change affects some tests, PTAL :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] mattisonchao edited a comment on pull request #14672: Change ``rollCurrentLedgerIfFull`` logic to follow lazy creation of ledger
Posted by GitBox <gi...@apache.org>.
mattisonchao edited a comment on pull request #14672:
URL: https://github.com/apache/pulsar/pull/14672#issuecomment-1066550620
@eolivelli
Since this PR changes the behaviour of ``rollCurrentLedgerIfFull``, the test ``ManagedCursorTest#testFindNewestMatchingAfterLedgerRollover`` needs some changes and I need your review.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] mattisonchao closed pull request #14672: Change ``rollCurrentLedgerIfFull`` logic to follow lazy creation of ledger
Posted by GitBox <gi...@apache.org>.
mattisonchao closed pull request #14672:
URL: https://github.com/apache/pulsar/pull/14672
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] mattisonchao commented on pull request #14672: Change ``rollCurrentLedgerIfFull`` logic to follow lazy creation of ledger
Posted by GitBox <gi...@apache.org>.
mattisonchao commented on pull request #14672:
URL: https://github.com/apache/pulsar/pull/14672#issuecomment-1066550620
@eolivelli
Since this PR changes the behaviour of "rollCurrentLedgerIfFull", the test "ManagedCursorTest#testFindNewestMatchingAfterLedgerRollover" needs some changes and I need your review.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] BewareMyPower commented on a change in pull request #14672: Change ``rollCurrentLedgerIfFull`` logic to follow lazy creation of ledger
Posted by GitBox <gi...@apache.org>.
BewareMyPower commented on a change in pull request #14672:
URL: https://github.com/apache/pulsar/pull/14672#discussion_r826779386
##########
File path: managed-ledger/src/test/java/org/apache/bookkeeper/mledger/impl/ManagedLedgerTest.java
##########
@@ -1972,15 +1975,25 @@ public void testDeletionAfterLedgerClosedAndRetention() throws Exception {
Field stateUpdater = ManagedLedgerImpl.class.getDeclaredField("state");
stateUpdater.setAccessible(true);
stateUpdater.set(ml, ManagedLedgerImpl.State.LedgerOpened);
+ long preLedgerId = ml.getLedgersInfoAsList().get(ml.getLedgersInfoAsList().size() -1).getLedgerId();
+ ml.pendingAddEntries.add(OpAddEntry.
+ createNoRetainBuffer(ml, ByteBufAllocator.DEFAULT.buffer(128).retain(), null, null));
ml.rollCurrentLedgerIfFull();
+ AtomicLong currentLedgerId = new AtomicLong(-1);
+ // create a new ledger
+ Awaitility.await().untilAsserted(()-> {
Review comment:
```suggestion
Awaitility.await().untilAsserted(() -> {
```
##########
File path: managed-ledger/src/test/java/org/apache/bookkeeper/mledger/impl/ManagedLedgerTest.java
##########
@@ -1972,15 +1975,25 @@ public void testDeletionAfterLedgerClosedAndRetention() throws Exception {
Field stateUpdater = ManagedLedgerImpl.class.getDeclaredField("state");
stateUpdater.setAccessible(true);
stateUpdater.set(ml, ManagedLedgerImpl.State.LedgerOpened);
Review comment:
I think we can remove the explicit state reset as well because now the `maxEntriesPerLedger` is 2 and the state is `LedgerOpened` not `ClosedLedger`.
##########
File path: pulsar-broker/src/test/java/org/apache/pulsar/broker/service/CurrentLedgerRolloverIfFullTest.java
##########
@@ -104,12 +110,51 @@ public void testCurrentLedgerRolloverIfFull() throws Exception {
stateUpdater.set(managedLedger, ManagedLedgerImpl.State.LedgerOpened);
managedLedger.rollCurrentLedgerIfFull();
- // the last ledger will be closed and removed and we have one ledger for empty
+ // If there are no pending write messages, the last ledger will be closed and still held.
Awaitility.await()
.pollInterval(Duration.ofMillis(1000L))
.untilAsserted(() -> {
Assert.assertEquals(managedLedger.getLedgersInfoAsList().size(), 1);
- Assert.assertEquals(managedLedger.getTotalSize(), 0);
+ Assert.assertNotEquals(managedLedger.getCurrentLedgerSize(), 0);
+ MLDataFormats.ManagedLedgerInfo.LedgerInfo lastLhAfterRollover =
+ managedLedger.getLedgersInfoAsList().get(managedLedger.getLedgersInfoAsList().size() - 1);
Review comment:
Maybe we can add a `getLastLedgerId` method to simplify the code.
```java
Supplier<Long> lastLedgerIdSupplier = () -> {
final List<LedgerInfo> ledgers = managedLedger.getLedgersInfoAsList();
Assert.assertFalse(ledgers.isEmpty());
return ledgers.get(ledgers.size() - 1).getLedgerId();
};
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] mattisonchao commented on a change in pull request #14672: Change ``rollCurrentLedgerIfFull`` logic to follow lazy creation of ledger
Posted by GitBox <gi...@apache.org>.
mattisonchao commented on a change in pull request #14672:
URL: https://github.com/apache/pulsar/pull/14672#discussion_r826845565
##########
File path: pulsar-broker/src/test/java/org/apache/pulsar/broker/service/CurrentLedgerRolloverIfFullTest.java
##########
@@ -104,12 +110,51 @@ public void testCurrentLedgerRolloverIfFull() throws Exception {
stateUpdater.set(managedLedger, ManagedLedgerImpl.State.LedgerOpened);
managedLedger.rollCurrentLedgerIfFull();
- // the last ledger will be closed and removed and we have one ledger for empty
+ // If there are no pending write messages, the last ledger will be closed and still held.
Awaitility.await()
.pollInterval(Duration.ofMillis(1000L))
.untilAsserted(() -> {
Assert.assertEquals(managedLedger.getLedgersInfoAsList().size(), 1);
- Assert.assertEquals(managedLedger.getTotalSize(), 0);
+ Assert.assertNotEquals(managedLedger.getCurrentLedgerSize(), 0);
+ MLDataFormats.ManagedLedgerInfo.LedgerInfo lastLhAfterRollover =
+ managedLedger.getLedgersInfoAsList().get(managedLedger.getLedgersInfoAsList().size() - 1);
Review comment:
Yes, I will change it ASAP.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] mattisonchao commented on a change in pull request #14672: Change ``rollCurrentLedgerIfFull`` logic to follow lazy creation of ledger
Posted by GitBox <gi...@apache.org>.
mattisonchao commented on a change in pull request #14672:
URL: https://github.com/apache/pulsar/pull/14672#discussion_r826845932
##########
File path: managed-ledger/src/test/java/org/apache/bookkeeper/mledger/impl/ManagedLedgerTest.java
##########
@@ -1972,15 +1975,25 @@ public void testDeletionAfterLedgerClosedAndRetention() throws Exception {
Field stateUpdater = ManagedLedgerImpl.class.getDeclaredField("state");
stateUpdater.setAccessible(true);
stateUpdater.set(ml, ManagedLedgerImpl.State.LedgerOpened);
Review comment:
Sure, i will fix it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] BewareMyPower merged pull request #14672: Change ``rollCurrentLedgerIfFull`` logic to follow lazy creation of ledger
Posted by GitBox <gi...@apache.org>.
BewareMyPower merged pull request #14672:
URL: https://github.com/apache/pulsar/pull/14672
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org