You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pulsar.apache.org by GitBox <gi...@apache.org> on 2022/11/11 01:17:25 UTC
[GitHub] [pulsar] coderzc opened a new pull request, #18420: [feat][broker][PIP-195] Implement delayed message bucket snapshot recover - part5
coderzc opened a new pull request, #18420:
URL: https://github.com/apache/pulsar/pull/18420
<!--
### Contribution Checklist
- PR title format should be *[type][component] summary*. For details, see *[Guideline - Pulsar PR Naming Convention](https://docs.google.com/document/d/1d8Pw6ZbWk-_pCKdOmdvx9rnhPiyuxwq60_TrD68d7BA/edit#heading=h.trs9rsex3xom)*.
Master Issue: #16763
### Motivation
#16763
### Modifications
Implement delayed message index bucket snapshot recover
### Verifying this change
- [x] Make sure that the change passes the CI checks.
*(Please pick either of the following options)*
This change is a trivial rework / code cleanup without any test coverage.
### Documentation
<!-- DO NOT REMOVE THIS SECTION. CHECK THE PROPER BOX ONLY. -->
- [ ] `doc` <!-- Your PR contains doc changes. Please attach the local preview screenshots (run `sh start.sh` at `pulsar/site2/website`) to your PR description, or else your PR might not get merged. -->
- [ ] `doc-required` <!-- Your PR changes impact docs and you will update later -->
- [x] `doc-not-needed` <!-- Your PR changes do not impact docs -->
- [ ] `doc-complete` <!-- Docs have been already added -->
### Matching PR in forked repository
PR in forked repository:
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] gaoran10 commented on a diff in pull request #18420: [feat][broker][PIP-195] Implement delayed message bucket snapshot recover - part5
Posted by GitBox <gi...@apache.org>.
gaoran10 commented on code in PR #18420:
URL: https://github.com/apache/pulsar/pull/18420#discussion_r1052796061
##########
pulsar-broker/src/main/java/org/apache/pulsar/broker/delayed/bucket/ImmutableBucket.java:
##########
@@ -52,11 +63,29 @@ CompletableFuture<List<DelayedIndex>> asyncLoadNextBucketSnapshotEntry(boolean i
return snapshotCreateFuture.thenCompose(__ -> {
final long bucketId = getAndUpdateBucketId();
- CompletableFuture<Integer> loadMetaDataFuture = new CompletableFuture<>();
+ final CompletableFuture<Integer> loadMetaDataFuture;
if (isRecover) {
- // TODO Recover bucket snapshot
+ final long cutoffTime = cutoffTimeSupplier.get();
+ // Load Metadata of bucket snapshot
+ loadMetaDataFuture = bucketSnapshotStorage.getBucketSnapshotMetadata(bucketId)
+ .thenApply(snapshotMetadata -> {
+ List<DelayedMessageIndexBucketSnapshotFormat.SnapshotSegmentMetadata> metadataList =
+ snapshotMetadata.getMetadataListList();
+
+ // Skip all already reach schedule time snapshot segments
Review Comment:
Does this mean entries before the cutoff time are all delivered?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] coderzc commented on a diff in pull request #18420: [feat][broker][PIP-195] Implement delayed message bucket snapshot recover - part5
Posted by GitBox <gi...@apache.org>.
coderzc commented on code in PR #18420:
URL: https://github.com/apache/pulsar/pull/18420#discussion_r1051882749
##########
pulsar-broker/src/main/java/org/apache/pulsar/broker/delayed/bucket/BucketDelayedDeliveryTracker.java:
##########
@@ -91,9 +98,66 @@ public BucketDelayedDeliveryTracker(PersistentDispatcherMultipleConsumers dispat
this.sharedBucketPriorityQueue = new TripleLongPriorityQueue();
this.immutableBuckets = TreeRangeMap.create();
this.snapshotSegmentLastIndexTable = HashBasedTable.create();
- this.numberDelayedMessages = 0L;
ManagedCursor cursor = dispatcher.getCursor();
this.lastMutableBucket = new MutableBucket(cursor, bucketSnapshotStorage);
+ this.numberDelayedMessages = recoverBucketSnapshot();
+ }
+
+ private long recoverBucketSnapshot() throws RuntimeException {
Review Comment:
It seems that we still need to block `initialize` method, when using `DelayedDeliveryTrackerFactory` to create `DelayedEdeliverytracker`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] mattisonchao commented on a diff in pull request #18420: [feat][broker][PIP-195] Implement delayed message bucket snapshot recover - part5
Posted by GitBox <gi...@apache.org>.
mattisonchao commented on code in PR #18420:
URL: https://github.com/apache/pulsar/pull/18420#discussion_r1052789262
##########
pulsar-broker/src/main/java/org/apache/pulsar/broker/delayed/bucket/Bucket.java:
##########
@@ -131,7 +131,8 @@ CompletableFuture<Long> asyncSaveBucketSnapshot(
bucketState.setBucketId(newBucketId);
String bucketKey = bucketState.bucketKey();
return putBucketKeyId(bucketKey, newBucketId).exceptionally(ex -> {
- log.warn("Failed to record bucketId to cursor property, bucketKey: {}", bucketKey);
+ log.warn("Failed to record bucketId to cursor property, bucketKey: {}, bucketId: {}",
Review Comment:
Is swallowing exception what we expect?
##########
pulsar-broker/src/main/java/org/apache/pulsar/broker/delayed/bucket/BucketDelayedDeliveryTracker.java:
##########
@@ -91,9 +98,66 @@ public BucketDelayedDeliveryTracker(PersistentDispatcherMultipleConsumers dispat
this.sharedBucketPriorityQueue = new TripleLongPriorityQueue();
this.immutableBuckets = TreeRangeMap.create();
this.snapshotSegmentLastIndexTable = HashBasedTable.create();
- this.numberDelayedMessages = 0L;
ManagedCursor cursor = dispatcher.getCursor();
this.lastMutableBucket = new MutableBucket(cursor, bucketSnapshotStorage);
+ this.numberDelayedMessages = recoverBucketSnapshot();
+ }
+
+ private long recoverBucketSnapshot() throws RuntimeException {
+ ManagedCursor cursor = this.lastMutableBucket.cursor;
+ cursor.getCursorProperties().keySet().forEach(key -> {
+ if (key.startsWith(DELAYED_BUCKET_KEY_PREFIX)) {
+ String[] keys = key.split(DELIMITER);
+ checkArgument(keys.length == 3);
+ ImmutableBucket immutableBucket =
+ new ImmutableBucket(cursor, this.lastMutableBucket.bucketSnapshotStorage,
+ Long.parseLong(keys[1]), Long.parseLong(keys[2]));
+ immutableBuckets.put(Range.closed(immutableBucket.startLedgerId, immutableBucket.endLedgerId),
+ immutableBucket);
+ }
+ });
+
+ if (immutableBuckets.asMapOfRanges().isEmpty()) {
+ return 0;
+ }
+
+ List<CompletableFuture<Void>> futures = new ArrayList<>(immutableBuckets.asMapOfRanges().size());
+ for (ImmutableBucket immutableBucket : immutableBuckets.asMapOfRanges().values()) {
+ CompletableFuture<Void> future =
+ immutableBucket.asyncRecoverBucketSnapshotEntry(this::getCutoffTime).thenAccept(indexList -> {
+ if (CollectionUtils.isEmpty(indexList)) {
+ return;
+ }
+ DelayedIndex lastDelayedIndex = indexList.get(indexList.size() - 1);
+ synchronized (this.snapshotSegmentLastIndexTable) {
Review Comment:
Is there a race condition for this method? if yes, we'd better make this method thread safe to avoid reading the same entries.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] coderzc commented on a diff in pull request #18420: [feat][broker][PIP-195] Implement delayed message bucket snapshot recover - part5
Posted by GitBox <gi...@apache.org>.
coderzc commented on code in PR #18420:
URL: https://github.com/apache/pulsar/pull/18420#discussion_r1052846840
##########
pulsar-broker/src/main/java/org/apache/pulsar/broker/delayed/bucket/Bucket.java:
##########
@@ -131,7 +131,8 @@ CompletableFuture<Long> asyncSaveBucketSnapshot(
bucketState.setBucketId(newBucketId);
String bucketKey = bucketState.bucketKey();
return putBucketKeyId(bucketKey, newBucketId).exceptionally(ex -> {
- log.warn("Failed to record bucketId to cursor property, bucketKey: {}", bucketKey);
+ log.warn("Failed to record bucketId to cursor property, bucketKey: {}, bucketId: {}",
Review Comment:
Yes, it will still work until the broker stop
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] coderzc commented on a diff in pull request #18420: [feat][broker][PIP-195] Implement delayed message bucket snapshot recover - part5
Posted by GitBox <gi...@apache.org>.
coderzc commented on code in PR #18420:
URL: https://github.com/apache/pulsar/pull/18420#discussion_r1051898668
##########
pulsar-broker/src/main/java/org/apache/pulsar/broker/delayed/bucket/ImmutableBucket.java:
##########
@@ -54,7 +65,28 @@ CompletableFuture<List<DelayedIndex>> asyncLoadNextBucketSnapshotEntry(boolean i
final long bucketId = getAndUpdateBucketId();
CompletableFuture<Integer> loadMetaDataFuture = new CompletableFuture<>();
if (isRecover) {
- // TODO Recover bucket snapshot
+ final long cutoffTime = cutoffTimeSupplier.get();
+ // Load Metadata of bucket snapshot
+ bucketSnapshotStorage.getBucketSnapshotMetadata(bucketId).thenAccept(snapshotMetadata -> {
+ List<DelayedMessageIndexBucketSnapshotFormat.SnapshotSegmentMetadata> metadataList =
+ snapshotMetadata.getMetadataListList();
+
+ // Skip all already reach schedule time snapshot segments
+ int nextSnapshotEntryIndex = 0;
+ while (nextSnapshotEntryIndex < metadataList.size()
+ && metadataList.get(nextSnapshotEntryIndex).getMaxScheduleTimestamp() <= cutoffTime) {
+ nextSnapshotEntryIndex++;
+ }
+
+ this.setLastSegmentEntryId(metadataList.size());
+ this.recoverDelayedIndexBitMapAndNumber(nextSnapshotEntryIndex, metadataList);
+
+ int nextSegmentEntryId = nextSnapshotEntryIndex + 1;
+ loadMetaDataFuture.complete(nextSegmentEntryId);
+ }).exceptionally(ex -> {
+ loadMetaDataFuture.completeExceptionally(ex);
Review Comment:
It seems that he still needs `loadMetaDataFuture`, but I improve the code, make the code easy to read
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] codecov-commenter commented on pull request #18420: [feat][broker][PIP-195] Implement delayed message bucket snapshot recover - part5
Posted by GitBox <gi...@apache.org>.
codecov-commenter commented on PR #18420:
URL: https://github.com/apache/pulsar/pull/18420#issuecomment-1357292952
# [Codecov](https://codecov.io/gh/apache/pulsar/pull/18420?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#18420](https://codecov.io/gh/apache/pulsar/pull/18420?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (9e0aed6) into [master](https://codecov.io/gh/apache/pulsar/commit/27186a1ee9298802a12f1d42f9bba03d8f3eea77?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (27186a1) will **decrease** coverage by `8.77%`.
> The diff coverage is `n/a`.
[](https://codecov.io/gh/apache/pulsar/pull/18420?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
```diff
@@ Coverage Diff @@
## master #18420 +/- ##
============================================
- Coverage 45.61% 36.84% -8.78%
+ Complexity 10728 1959 -8769
============================================
Files 752 209 -543
Lines 72521 14402 -58119
Branches 7791 1569 -6222
============================================
- Hits 33083 5306 -27777
+ Misses 35769 8521 -27248
+ Partials 3669 575 -3094
```
| Flag | Coverage Δ | |
|---|---|---|
| unittests | `36.84% <ø> (-8.78%)` | :arrow_down: |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more.
| [Impacted Files](https://codecov.io/gh/apache/pulsar/pull/18420?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [...apache/pulsar/client/impl/AutoClusterFailover.java](https://codecov.io/gh/apache/pulsar/pull/18420/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cHVsc2FyLWNsaWVudC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcHVsc2FyL2NsaWVudC9pbXBsL0F1dG9DbHVzdGVyRmFpbG92ZXIuamF2YQ==) | `70.00% <0.00%> (-6.12%)` | :arrow_down: |
| [.../apache/pulsar/client/impl/BatchMessageIdImpl.java](https://codecov.io/gh/apache/pulsar/pull/18420/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cHVsc2FyLWNsaWVudC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcHVsc2FyL2NsaWVudC9pbXBsL0JhdGNoTWVzc2FnZUlkSW1wbC5qYXZh) | `67.50% <0.00%> (-4.73%)` | :arrow_down: |
| [.../pulsar/client/impl/ControlledClusterFailover.java](https://codecov.io/gh/apache/pulsar/pull/18420/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cHVsc2FyLWNsaWVudC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcHVsc2FyL2NsaWVudC9pbXBsL0NvbnRyb2xsZWRDbHVzdGVyRmFpbG92ZXIuamF2YQ==) | `58.87% <0.00%> (-3.74%)` | :arrow_down: |
| [...va/org/apache/pulsar/client/impl/ProducerImpl.java](https://codecov.io/gh/apache/pulsar/pull/18420/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cHVsc2FyLWNsaWVudC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcHVsc2FyL2NsaWVudC9pbXBsL1Byb2R1Y2VySW1wbC5qYXZh) | `15.66% <0.00%> (-0.77%)` | :arrow_down: |
| [.../java/org/apache/pulsar/client/impl/ClientCnx.java](https://codecov.io/gh/apache/pulsar/pull/18420/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cHVsc2FyLWNsaWVudC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcHVsc2FyL2NsaWVudC9pbXBsL0NsaWVudENueC5qYXZh) | `29.97% <0.00%> (-0.20%)` | :arrow_down: |
| [...ar/client/impl/conf/ProducerConfigurationData.java](https://codecov.io/gh/apache/pulsar/pull/18420/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cHVsc2FyLWNsaWVudC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcHVsc2FyL2NsaWVudC9pbXBsL2NvbmYvUHJvZHVjZXJDb25maWd1cmF0aW9uRGF0YS5qYXZh) | `84.70% <0.00%> (-0.18%)` | :arrow_down: |
| [...he/pulsar/client/impl/MultiTopicsConsumerImpl.java](https://codecov.io/gh/apache/pulsar/pull/18420/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cHVsc2FyLWNsaWVudC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcHVsc2FyL2NsaWVudC9pbXBsL011bHRpVG9waWNzQ29uc3VtZXJJbXBsLmphdmE=) | `22.78% <0.00%> (-0.17%)` | :arrow_down: |
| [...va/org/apache/pulsar/client/impl/ConsumerBase.java](https://codecov.io/gh/apache/pulsar/pull/18420/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cHVsc2FyLWNsaWVudC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcHVsc2FyL2NsaWVudC9pbXBsL0NvbnN1bWVyQmFzZS5qYXZh) | `21.93% <0.00%> (-0.02%)` | :arrow_down: |
| [...a/org/apache/pulsar/client/impl/TableViewImpl.java](https://codecov.io/gh/apache/pulsar/pull/18420/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cHVsc2FyLWNsaWVudC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcHVsc2FyL2NsaWVudC9pbXBsL1RhYmxlVmlld0ltcGwuamF2YQ==) | `0.00% <0.00%> (ø)` | |
| [...pache/pulsar/client/impl/TableViewBuilderImpl.java](https://codecov.io/gh/apache/pulsar/pull/18420/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cHVsc2FyLWNsaWVudC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcHVsc2FyL2NsaWVudC9pbXBsL1RhYmxlVmlld0J1aWxkZXJJbXBsLmphdmE=) | `0.00% <0.00%> (ø)` | |
| ... and [560 more](https://codecov.io/gh/apache/pulsar/pull/18420/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] codelipenghui merged pull request #18420: [feat][broker][PIP-195] Implement delayed message bucket snapshot recover - part5
Posted by GitBox <gi...@apache.org>.
codelipenghui merged PR #18420:
URL: https://github.com/apache/pulsar/pull/18420
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] codelipenghui commented on a diff in pull request #18420: [feat][broker][PIP-195] Implement delayed message bucket snapshot recover - part5
Posted by GitBox <gi...@apache.org>.
codelipenghui commented on code in PR #18420:
URL: https://github.com/apache/pulsar/pull/18420#discussion_r1051782984
##########
pulsar-broker/src/main/java/org/apache/pulsar/broker/delayed/bucket/BucketDelayedDeliveryTracker.java:
##########
@@ -91,9 +98,66 @@ public BucketDelayedDeliveryTracker(PersistentDispatcherMultipleConsumers dispat
this.sharedBucketPriorityQueue = new TripleLongPriorityQueue();
this.immutableBuckets = TreeRangeMap.create();
this.snapshotSegmentLastIndexTable = HashBasedTable.create();
- this.numberDelayedMessages = 0L;
ManagedCursor cursor = dispatcher.getCursor();
this.lastMutableBucket = new MutableBucket(cursor, bucketSnapshotStorage);
+ this.numberDelayedMessages = recoverBucketSnapshot();
+ }
+
+ private long recoverBucketSnapshot() throws RuntimeException {
Review Comment:
Is it better to add a method `CompletableFuture<Void> initialize()` to the `DelayedDeliveryTracker` to avoid blocking calls when creating the `BucketDelayedDeliveryTracker` instance
##########
pulsar-broker/src/main/java/org/apache/pulsar/broker/delayed/bucket/ImmutableBucket.java:
##########
@@ -54,7 +65,28 @@ CompletableFuture<List<DelayedIndex>> asyncLoadNextBucketSnapshotEntry(boolean i
final long bucketId = getAndUpdateBucketId();
CompletableFuture<Integer> loadMetaDataFuture = new CompletableFuture<>();
if (isRecover) {
- // TODO Recover bucket snapshot
+ final long cutoffTime = cutoffTimeSupplier.get();
+ // Load Metadata of bucket snapshot
+ bucketSnapshotStorage.getBucketSnapshotMetadata(bucketId).thenAccept(snapshotMetadata -> {
+ List<DelayedMessageIndexBucketSnapshotFormat.SnapshotSegmentMetadata> metadataList =
+ snapshotMetadata.getMetadataListList();
+
+ // Skip all already reach schedule time snapshot segments
+ int nextSnapshotEntryIndex = 0;
+ while (nextSnapshotEntryIndex < metadataList.size()
+ && metadataList.get(nextSnapshotEntryIndex).getMaxScheduleTimestamp() <= cutoffTime) {
+ nextSnapshotEntryIndex++;
+ }
+
+ this.setLastSegmentEntryId(metadataList.size());
+ this.recoverDelayedIndexBitMapAndNumber(nextSnapshotEntryIndex, metadataList);
+
+ int nextSegmentEntryId = nextSnapshotEntryIndex + 1;
+ loadMetaDataFuture.complete(nextSegmentEntryId);
+ }).exceptionally(ex -> {
+ loadMetaDataFuture.completeExceptionally(ex);
Review Comment:
It looks like we can simplify the logic to avoid creating the `loadMetaDataFuture`.
It will make the code easy to read.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] gaoran10 commented on a diff in pull request #18420: [feat][broker][PIP-195] Implement delayed message bucket snapshot recover - part5
Posted by GitBox <gi...@apache.org>.
gaoran10 commented on code in PR #18420:
URL: https://github.com/apache/pulsar/pull/18420#discussion_r1053931530
##########
pulsar-broker/src/main/java/org/apache/pulsar/broker/delayed/bucket/ImmutableBucket.java:
##########
@@ -52,11 +63,29 @@ CompletableFuture<List<DelayedIndex>> asyncLoadNextBucketSnapshotEntry(boolean i
return snapshotCreateFuture.thenCompose(__ -> {
final long bucketId = getAndUpdateBucketId();
- CompletableFuture<Integer> loadMetaDataFuture = new CompletableFuture<>();
+ final CompletableFuture<Integer> loadMetaDataFuture;
if (isRecover) {
- // TODO Recover bucket snapshot
+ final long cutoffTime = cutoffTimeSupplier.get();
+ // Load Metadata of bucket snapshot
+ loadMetaDataFuture = bucketSnapshotStorage.getBucketSnapshotMetadata(bucketId)
+ .thenApply(snapshotMetadata -> {
+ List<DelayedMessageIndexBucketSnapshotFormat.SnapshotSegmentMetadata> metadataList =
+ snapshotMetadata.getMetadataListList();
+
+ // Skip all already reach schedule time snapshot segments
Review Comment:
Ok, thanks.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] coderzc commented on a diff in pull request #18420: [feat][broker][PIP-195] Implement delayed message bucket snapshot recover - part5
Posted by GitBox <gi...@apache.org>.
coderzc commented on code in PR #18420:
URL: https://github.com/apache/pulsar/pull/18420#discussion_r1052833502
##########
pulsar-broker/src/main/java/org/apache/pulsar/broker/delayed/bucket/ImmutableBucket.java:
##########
@@ -52,11 +63,29 @@ CompletableFuture<List<DelayedIndex>> asyncLoadNextBucketSnapshotEntry(boolean i
return snapshotCreateFuture.thenCompose(__ -> {
final long bucketId = getAndUpdateBucketId();
- CompletableFuture<Integer> loadMetaDataFuture = new CompletableFuture<>();
+ final CompletableFuture<Integer> loadMetaDataFuture;
if (isRecover) {
- // TODO Recover bucket snapshot
+ final long cutoffTime = cutoffTimeSupplier.get();
+ // Load Metadata of bucket snapshot
+ loadMetaDataFuture = bucketSnapshotStorage.getBucketSnapshotMetadata(bucketId)
+ .thenApply(snapshotMetadata -> {
+ List<DelayedMessageIndexBucketSnapshotFormat.SnapshotSegmentMetadata> metadataList =
+ snapshotMetadata.getMetadataListList();
+
+ // Skip all already reach schedule time snapshot segments
Review Comment:
All entries before the cutoff time will be sent to consumer directly
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] coderzc commented on a diff in pull request #18420: [feat][broker][PIP-195] Implement delayed message bucket snapshot recover - part5
Posted by GitBox <gi...@apache.org>.
coderzc commented on code in PR #18420:
URL: https://github.com/apache/pulsar/pull/18420#discussion_r1051882749
##########
pulsar-broker/src/main/java/org/apache/pulsar/broker/delayed/bucket/BucketDelayedDeliveryTracker.java:
##########
@@ -91,9 +98,66 @@ public BucketDelayedDeliveryTracker(PersistentDispatcherMultipleConsumers dispat
this.sharedBucketPriorityQueue = new TripleLongPriorityQueue();
this.immutableBuckets = TreeRangeMap.create();
this.snapshotSegmentLastIndexTable = HashBasedTable.create();
- this.numberDelayedMessages = 0L;
ManagedCursor cursor = dispatcher.getCursor();
this.lastMutableBucket = new MutableBucket(cursor, bucketSnapshotStorage);
+ this.numberDelayedMessages = recoverBucketSnapshot();
+ }
+
+ private long recoverBucketSnapshot() throws RuntimeException {
Review Comment:
It seems that he needs to block `initialize` method, when using `DelayedDeliveryTrackerFactory` to create `DelayedEdeliverytracker`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] codelipenghui commented on a diff in pull request #18420: [feat][broker][PIP-195] Implement delayed message bucket snapshot recover - part5
Posted by GitBox <gi...@apache.org>.
codelipenghui commented on code in PR #18420:
URL: https://github.com/apache/pulsar/pull/18420#discussion_r1051926752
##########
pulsar-broker/src/main/java/org/apache/pulsar/broker/delayed/bucket/BucketDelayedDeliveryTracker.java:
##########
@@ -91,9 +98,66 @@ public BucketDelayedDeliveryTracker(PersistentDispatcherMultipleConsumers dispat
this.sharedBucketPriorityQueue = new TripleLongPriorityQueue();
this.immutableBuckets = TreeRangeMap.create();
this.snapshotSegmentLastIndexTable = HashBasedTable.create();
- this.numberDelayedMessages = 0L;
ManagedCursor cursor = dispatcher.getCursor();
this.lastMutableBucket = new MutableBucket(cursor, bucketSnapshotStorage);
+ this.numberDelayedMessages = recoverBucketSnapshot();
+ }
+
+ private long recoverBucketSnapshot() throws RuntimeException {
Review Comment:
Ok, we can just keep it here for now.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] codelipenghui commented on pull request #18420: [feat][broker][PIP-195] Implement delayed message bucket snapshot recover - part5
Posted by GitBox <gi...@apache.org>.
codelipenghui commented on PR #18420:
URL: https://github.com/apache/pulsar/pull/18420#issuecomment-1357277398
/pulsarbot run-failure-checks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org