You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pulsar.apache.org by GitBox <gi...@apache.org> on 2021/09/22 19:08:27 UTC
[GitHub] [pulsar] dlg99 opened a new pull request #12141: fix: Various NPEs on terminated managed ledger
dlg99 opened a new pull request #12141:
URL: https://github.com/apache/pulsar/pull/12141
<!--
### Contribution Checklist
- Name the pull request in the form "[Issue XYZ][component] Title of the pull request", where *XYZ* should be replaced by the actual issue number.
Skip *Issue XYZ* if there is no associated github issue for this pull request.
Skip *component* if you are unsure about which is the best component. E.g. `[docs] Fix typo in produce method`.
- Fill out the template below to describe the changes contributed by the pull request. That will give reviewers the context they need to do the review.
- Each pull request should address only one issue, not mix up code from multiple issues.
- Each commit in the pull request has a meaningful commit message
- Once all items of the checklist are addressed, remove the above text and this checklist, leaving only the filled out template below.
**(The sections below can be removed for hotfixes of typos)**
-->
*(If this PR fixes a github issue, please add `Fixes #<xyz>`.)*
### Motivation
I encountered this while experimenting with repros for issue: #12070
The problem here is that various methods on terminated managed ledger throw NPEs after broker restart.
The rootcause is that `currentLedger` field is not expected to be set in case of terminated ledger hence stays null; the null check on that field is inconsistent.
### Modifications
Add null check everywhere the `currentLedger` is accessed, added unit test.
### Verifying this change
- [ ] Make sure that the change passes the CI checks.
*(Please pick either of the following options)*
This change added unit tests
### Does this pull request potentially affect one of the following parts:
*If `yes` was chosen, please highlight the changes*
No
### Documentation
Need to update docs?
- [X] no-need-doc
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] lhotari commented on pull request #12141: fix: Various NPEs on terminated managed ledger
Posted by GitBox <gi...@apache.org>.
lhotari commented on pull request #12141:
URL: https://github.com/apache/pulsar/pull/12141#issuecomment-1057218321
@dlg99 Please rebase this PR
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] sijie commented on pull request #12141: fix: Various NPEs on terminated managed ledger
Posted by GitBox <gi...@apache.org>.
sijie commented on pull request #12141:
URL: https://github.com/apache/pulsar/pull/12141#issuecomment-929689964
@codelipenghui @merlimat Can you take a look at this PR?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] sijie commented on pull request #12141: fix: Various NPEs on terminated managed ledger
Posted by GitBox <gi...@apache.org>.
sijie commented on pull request #12141:
URL: https://github.com/apache/pulsar/pull/12141#issuecomment-929689964
@codelipenghui @merlimat Can you take a look at this PR?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] eolivelli commented on a change in pull request #12141: fix: Various NPEs on terminated managed ledger
Posted by GitBox <gi...@apache.org>.
eolivelli commented on a change in pull request #12141:
URL: https://github.com/apache/pulsar/pull/12141#discussion_r714237977
##########
File path: pulsar-broker/src/test/java/org/apache/pulsar/broker/service/ConsumedLedgersTrimTest.java
##########
@@ -170,8 +171,89 @@ public void testConsumedLedgersTrimNoSubscriptions() throws Exception {
// lastMessageId should be available even in this case, but is must
// refer to -1
MessageId messageIdAfterTrim = pulsar.getAdminClient().topics().getLastMessageId(topicName);
- LOG.info("lastmessageid " + messageIdAfterTrim);
+ log.info("admin lastmessageid {}", messageIdAfterTrim);
+ assertEquals(messageIdAfterTrim, MessageId.earliest);
+
+ messageIdAfterTrim = persistentTopic.getLastMessageId().get();
+ log.info("topic lastmessageid {}", messageIdAfterTrim);
assertEquals(messageIdAfterTrim, MessageId.earliest);
+ }
+
+ @Test
+ public void testTerminateAndRestart() throws Exception {
+ conf.setRetentionCheckIntervalInSeconds(10000);
+ conf.setBrokerDeleteInactiveTopicsEnabled(false);
+ super.baseSetup();
+ final String topicName = "persistent://prop/ns-abc/testTerminateAndRestart";
+
+ // write some messages
+ @Cleanup
+ Producer<byte[]> producer = pulsarClient.newProducer()
+ .topic(topicName)
+ .producerName("producer-name")
+ .create();
+
+ // set retention parameters, the ledgers are to be deleted as soon as possible
+ // but the topic is not to be automatically deleted
+ PersistentTopic persistentTopic = (PersistentTopic) pulsar.getBrokerService().getOrCreateTopic(topicName).get();
+ ManagedLedgerConfig managedLedgerConfig = persistentTopic.getManagedLedger().getConfig();
+ managedLedgerConfig.setRetentionSizeInMB(-1);
+ managedLedgerConfig.setRetentionTime(100000, TimeUnit.SECONDS);
+ managedLedgerConfig.setMaxEntriesPerLedger(2);
+ managedLedgerConfig.setMinimumRolloverTime(1, TimeUnit.MILLISECONDS);
+ MessageId initialMessageId = persistentTopic.getLastMessageId().get();
+ log.info("lastmessageid " + initialMessageId);
Review comment:
Nit: use parameters
##########
File path: managed-ledger/src/main/java/org/apache/bookkeeper/mledger/impl/ManagedLedgerImpl.java
##########
@@ -770,13 +770,15 @@ private synchronized void internalAsyncAddEntry(OpAddEntry addOperation) {
currentLedgerSize += addOperation.data.readableBytes();
if (log.isDebugEnabled()) {
- log.debug("[{}] Write into current ledger lh={} entries={}", name, currentLedger.getId(),
+ log.debug("[{}] Write into current ledger lh={} entries={}", name,
+ currentLedger == null ? null : currentLedger.getId(),
Review comment:
For these logging cases we could add a Long getCurrentLedgerId() method that returns null is there is no currentLedger.
We will save some code duplication
##########
File path: managed-ledger/src/main/java/org/apache/bookkeeper/mledger/impl/ManagedLedgerImpl.java
##########
@@ -1602,7 +1605,7 @@ synchronized void createLedgerAfterClosed() {
@Override
public void rollCurrentLedgerIfFull() {
log.info("[{}] Start checking if current ledger is full", name);
- if (currentLedgerEntries > 0 && currentLedgerIsFull()) {
+ if (currentLedger != null && currentLedgerEntries > 0 && currentLedgerIsFull()) {
Review comment:
Will it make sense to exit early from this method if there is no 'currentLedger' ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] dlg99 commented on a change in pull request #12141: fix: Various NPEs on terminated managed ledger
Posted by GitBox <gi...@apache.org>.
dlg99 commented on a change in pull request #12141:
URL: https://github.com/apache/pulsar/pull/12141#discussion_r714282575
##########
File path: managed-ledger/src/main/java/org/apache/bookkeeper/mledger/impl/ManagedLedgerImpl.java
##########
@@ -1602,7 +1605,7 @@ synchronized void createLedgerAfterClosed() {
@Override
public void rollCurrentLedgerIfFull() {
log.info("[{}] Start checking if current ledger is full", name);
- if (currentLedgerEntries > 0 && currentLedgerIsFull()) {
+ if (currentLedger != null && currentLedgerEntries > 0 && currentLedgerIsFull()) {
Review comment:
essentially this is it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org