You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pulsar.apache.org by GitBox <gi...@apache.org> on 2021/01/24 03:58:36 UTC

[GitHub] [pulsar] rdhabalia opened a new pull request #9292: [pulsar-broker] Support large number of unack message store for cursor recovery

rdhabalia opened a new pull request #9292:
URL: https://github.com/apache/pulsar/pull/9292


   ### Motivation
   Right now, managed-cursor serializes individually deleted-message ranges and persist them in bookie ledger-entry for the recovery. managed-cursor can persist max 150K ranges due to limited bookie entry size (5MB). 150K ranges might not be enough for some of the use cases while recovering cursor. so, we need a mechanism to persist a couple of million ranges to support such usecases.
   
   ### Modification
   with #3818 and #3819 managed-cursor manages individually deleted messages in bitset with OpenRangeSet. Serializing OpenRangeSet can allow managed-cursor to store 10M ranges with 5MB data size. 
   
   ### Result
   Usecases require a large number of individually deleted-messages that can be supported with this change.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [pulsar] rdhabalia commented on a change in pull request #9292: [pulsar-broker] Support large number of unack message store for cursor recovery

Posted by GitBox <gi...@apache.org>.
rdhabalia commented on a change in pull request #9292:
URL: https://github.com/apache/pulsar/pull/9292#discussion_r564182454



##########
File path: managed-ledger/src/main/proto/MLDataFormats.proto
##########
@@ -71,6 +71,9 @@ message PositionInfo {
 
     // Store which index in the batch message has been deleted
     repeated BatchedEntryDeletionIndexInfo batchedEntryDeletionIndexInfo = 5;
+
+    // serialized individualDeletedMessages
+    optional bytes serializedIndividualDeletedMessages = 6;

Review comment:
       > That and that Java serialization is really bad in general.
   
   I agree. let me address it.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [pulsar] rdhabalia commented on a change in pull request #9292: [pulsar-broker] Support large number of unack message store for cursor recovery

Posted by GitBox <gi...@apache.org>.
rdhabalia commented on a change in pull request #9292:
URL: https://github.com/apache/pulsar/pull/9292#discussion_r563563466



##########
File path: managed-ledger/src/main/proto/MLDataFormats.proto
##########
@@ -71,6 +71,9 @@ message PositionInfo {
 
     // Store which index in the batch message has been deleted
     repeated BatchedEntryDeletionIndexInfo batchedEntryDeletionIndexInfo = 5;
+
+    // serialized individualDeletedMessages
+    optional bytes serializedIndividualDeletedMessages = 6;

Review comment:
       I tried to think about it but I don't see a clean way to get an internal structure of `BitSet` or internal reference which we can use to encode to protobuf. So, only way, I can see is to seralize BitSet to byte array and use the same to deserialize, It also gives a flexibility if we want to make range-set pluggable which can have a different implementation and that range set serialization doesn't have to depend on any specific protobuf format.
   but please let me know if you have any recommendation which can allow us to encode it directly in protobuf

##########
File path: managed-ledger/src/main/proto/MLDataFormats.proto
##########
@@ -71,6 +71,9 @@ message PositionInfo {
 
     // Store which index in the batch message has been deleted
     repeated BatchedEntryDeletionIndexInfo batchedEntryDeletionIndexInfo = 5;
+
+    // serialized individualDeletedMessages
+    optional bytes serializedIndividualDeletedMessages = 6;

Review comment:
       > That and that Java serialization is really bad in general.
   
   I agree. let me address it.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [pulsar] rdhabalia removed a comment on pull request #9292: [pulsar-broker] Support large number of unack message store for cursor recovery

Posted by GitBox <gi...@apache.org>.
rdhabalia removed a comment on pull request #9292:
URL: https://github.com/apache/pulsar/pull/9292#issuecomment-767396155


   /pulsarbot run-failure-checks


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [pulsar] rdhabalia commented on pull request #9292: [pulsar-broker] Support large number of unack message store for cursor recovery

Posted by GitBox <gi...@apache.org>.
rdhabalia commented on pull request #9292:
URL: https://github.com/apache/pulsar/pull/9292#issuecomment-769357085


   > what happens if I upgrade my broker to a new version and then I rollback to the previous version in case of problems ? IIUC we are going to write data that would not be understood by the old version. does it lead to some bad state of the system ? like having consumers that receive again the same messages ?
   
   no, it will not cause bad state in broker. broker understands both the format as it has separate buckets for each. broker recovers unack message ranges based on which ever bucket has ranges and later persist ranges in only new bucket. Broker rollback will not be able to recover unack messages from new bucket and in that case it will redeliver messages newer markDeleteMessageOffset.
   
   >  It is an improvement and doesn't solve all the problems.
   
   Yes, I shared my point of view in email regarding having unack message management. However, regardless abstraction changes, I think we can proceed with this PR as this can be used with existing default unack message implementation. 
   
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [pulsar] eolivelli commented on a change in pull request #9292: [pulsar-broker] Support large number of unack message store for cursor recovery

Posted by GitBox <gi...@apache.org>.
eolivelli commented on a change in pull request #9292:
URL: https://github.com/apache/pulsar/pull/9292#discussion_r566070165



##########
File path: managed-ledger/src/test/java/org/apache/bookkeeper/mledger/impl/ManagedCursorTest.java
##########
@@ -2817,7 +2817,9 @@ public void operationFailed(MetaStoreException e) {
                         LedgerEntry entry = seq.nextElement();
                         PositionInfo positionInfo;
                         positionInfo = PositionInfo.parseFrom(entry.getEntry());
-                        individualDeletedMessagesCount.set(positionInfo.getIndividualDeletedMessagesCount());
+                        c1.recoverIndividualDeletedMessages(positionInfo);
+                        individualDeletedMessagesCount.set(c1.getIndividuallyDeletedMessagesSet().asRanges().size());
+                        //individualDeletedMessagesCount.set(positionInfo.getIndividualDeletedMessagesCount());

Review comment:
       nit: drop commented line

##########
File path: pulsar-common/src/main/java/org/apache/pulsar/common/util/collections/LongPairRangeSet.java
##########
@@ -125,6 +128,26 @@
      */
     Range<T> lastRange();
 
+    /**
+     * Retrieve internal-ranges {@link LongPairRangeSet} to byte[].
+     * 
+     * @param maxRanges
+     *            to retrieve limited ranges.
+     * @return
+     */
+    default Map<Long, long[]> toRanges(int maxRanges) {
+        return Collections.emptyMap();

Review comment:
       isn't it better to throw UnsupportedOperationException ?
   
   also, if I retrieve only a subset of the data, how do I know that there would be more ranges ?




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [pulsar] merlimat commented on a change in pull request #9292: [pulsar-broker] Support large number of unack message store for cursor recovery

Posted by GitBox <gi...@apache.org>.
merlimat commented on a change in pull request #9292:
URL: https://github.com/apache/pulsar/pull/9292#discussion_r564069594



##########
File path: managed-ledger/src/main/proto/MLDataFormats.proto
##########
@@ -71,6 +71,9 @@ message PositionInfo {
 
     // Store which index in the batch message has been deleted
     repeated BatchedEntryDeletionIndexInfo batchedEntryDeletionIndexInfo = 5;
+
+    // serialized individualDeletedMessages
+    optional bytes serializedIndividualDeletedMessages = 6;

Review comment:
       My main concern here is that we're mixing 2 serialization systems together and therefore we need to pay attention when there are changes to that format. That and that Java serialization is really bad in general.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [pulsar] rdhabalia commented on pull request #9292: [pulsar-broker] Support large number of unack message store for cursor recovery

Posted by GitBox <gi...@apache.org>.
rdhabalia commented on pull request #9292:
URL: https://github.com/apache/pulsar/pull/9292#issuecomment-767396155


   /pulsarbot run-failure-checks


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [pulsar] rdhabalia removed a comment on pull request #9292: [pulsar-broker] Support large number of unack message store for cursor recovery

Posted by GitBox <gi...@apache.org>.
rdhabalia removed a comment on pull request #9292:
URL: https://github.com/apache/pulsar/pull/9292#issuecomment-766696902


   /pulsarbot run-failure-checks


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [pulsar] rdhabalia commented on pull request #9292: [pulsar-broker] Support large number of unack message store for cursor recovery

Posted by GitBox <gi...@apache.org>.
rdhabalia commented on pull request #9292:
URL: https://github.com/apache/pulsar/pull/9292#issuecomment-930391913


   @sijie @merlimat what's the conclusion on this PR. If PR will be blocked for no reason then it will not be merged. so, we can close in that case?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [pulsar] rdhabalia commented on a change in pull request #9292: [pulsar-broker] Support large number of unack message store for cursor recovery

Posted by GitBox <gi...@apache.org>.
rdhabalia commented on a change in pull request #9292:
URL: https://github.com/apache/pulsar/pull/9292#discussion_r563563466



##########
File path: managed-ledger/src/main/proto/MLDataFormats.proto
##########
@@ -71,6 +71,9 @@ message PositionInfo {
 
     // Store which index in the batch message has been deleted
     repeated BatchedEntryDeletionIndexInfo batchedEntryDeletionIndexInfo = 5;
+
+    // serialized individualDeletedMessages
+    optional bytes serializedIndividualDeletedMessages = 6;

Review comment:
       I tried to think about it but I don't see a clean way to get an internal structure of `BitSet` or internal reference which we can use to encode to protobuf. So, only way, I can see is to seralize BitSet to byte array and use the same to deserialize, It also gives a flexibility if we want to make range-set pluggable which can have a different implementation and that range set serialization doesn't have to depend on any specific protobuf format.
   but please let me know if you have any recommendation which can allow us to encode it directly in protobuf




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [pulsar] rdhabalia commented on pull request #9292: [pulsar-broker] Support large number of unack message store for cursor recovery

Posted by GitBox <gi...@apache.org>.
rdhabalia commented on pull request #9292:
URL: https://github.com/apache/pulsar/pull/9292#issuecomment-766696902


   /pulsarbot run-failure-checks


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [pulsar] rdhabalia commented on pull request #9292: [pulsar-broker] Support large number of unack message store for cursor recovery

Posted by GitBox <gi...@apache.org>.
rdhabalia commented on pull request #9292:
URL: https://github.com/apache/pulsar/pull/9292#issuecomment-930391913


   @sijie @merlimat what's the conclusion on this PR. If PR will be blocked for no reason then it will not be merged. so, we can close in that case?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [pulsar] merlimat commented on a change in pull request #9292: [pulsar-broker] Support large number of unack message store for cursor recovery

Posted by GitBox <gi...@apache.org>.
merlimat commented on a change in pull request #9292:
URL: https://github.com/apache/pulsar/pull/9292#discussion_r563328917



##########
File path: managed-ledger/src/main/proto/MLDataFormats.proto
##########
@@ -71,6 +71,9 @@ message PositionInfo {
 
     // Store which index in the batch message has been deleted
     repeated BatchedEntryDeletionIndexInfo batchedEntryDeletionIndexInfo = 5;
+
+    // serialized individualDeletedMessages
+    optional bytes serializedIndividualDeletedMessages = 6;

Review comment:
       Instead of having a byte array here, I think it would be better to have instead a way to encode the range set in protobuf directly

##########
File path: managed-ledger/src/main/proto/MLDataFormats.proto
##########
@@ -71,6 +71,9 @@ message PositionInfo {
 
     // Store which index in the batch message has been deleted
     repeated BatchedEntryDeletionIndexInfo batchedEntryDeletionIndexInfo = 5;
+
+    // serialized individualDeletedMessages
+    optional bytes serializedIndividualDeletedMessages = 6;

Review comment:
       My main concern here is that we're mixing 2 serialization systems together and therefore we need to pay attention when there are changes to that format. That and that Java serialization is really bad in general.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [pulsar] merlimat commented on a change in pull request #9292: [pulsar-broker] Support large number of unack message store for cursor recovery

Posted by GitBox <gi...@apache.org>.
merlimat commented on a change in pull request #9292:
URL: https://github.com/apache/pulsar/pull/9292#discussion_r563328917



##########
File path: managed-ledger/src/main/proto/MLDataFormats.proto
##########
@@ -71,6 +71,9 @@ message PositionInfo {
 
     // Store which index in the batch message has been deleted
     repeated BatchedEntryDeletionIndexInfo batchedEntryDeletionIndexInfo = 5;
+
+    // serialized individualDeletedMessages
+    optional bytes serializedIndividualDeletedMessages = 6;

Review comment:
       Instead of having a byte array here, I think it would be better to have instead a way to encode the range set in protobuf directly




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [pulsar] rdhabalia commented on pull request #9292: [pulsar-broker] Support large number of unack message store for cursor recovery

Posted by GitBox <gi...@apache.org>.
rdhabalia commented on pull request #9292:
URL: https://github.com/apache/pulsar/pull/9292#issuecomment-766696902






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [pulsar] rdhabalia commented on pull request #9292: [pulsar-broker] Support large number of unack message store for cursor recovery

Posted by GitBox <gi...@apache.org>.
rdhabalia commented on pull request #9292:
URL: https://github.com/apache/pulsar/pull/9292#issuecomment-767005589


   /pulsarbot run-failure-checks


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [pulsar] codelipenghui commented on pull request #9292: [pulsar-broker] Support large number of unack message store for cursor recovery

Posted by GitBox <gi...@apache.org>.
codelipenghui commented on pull request #9292:
URL: https://github.com/apache/pulsar/pull/9292#issuecomment-1058893946


   The pr had no activity for 30 days, mark with Stale label.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [pulsar] sijie commented on pull request #9292: [pulsar-broker] Support large number of unack message store for cursor recovery

Posted by GitBox <gi...@apache.org>.
sijie commented on pull request #9292:
URL: https://github.com/apache/pulsar/pull/9292#issuecomment-770528282


   > Yes, I shared my point of view in email regarding having unack message management. However, regardless abstraction changes, I think we can proceed with this PR as this can be used with existing default unack message implementation.
   
   I would like to have a consensus on providing an abstraction before merging this implementation. Otherwise, it becomes really hard to allow people to implement other algorithms. 


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [pulsar] sijie commented on pull request #9292: [pulsar-broker] Support large number of unack message store for cursor recovery

Posted by GitBox <gi...@apache.org>.
sijie commented on pull request #9292:
URL: https://github.com/apache/pulsar/pull/9292#issuecomment-770528479


   > no, it will not cause bad state in broker. broker understands both the format as it has separate buckets for each. 
   
   Can you add a backward compatibility integration test?


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [pulsar] github-actions[bot] commented on pull request #9292: [pulsar-broker] Support large number of unack message store for cursor recovery

Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #9292:
URL: https://github.com/apache/pulsar/pull/9292#issuecomment-1058897233


   @rdhabalia:Thanks for your contribution. For this PR, do we need to update docs?
   (The [PR template contains info about doc](https://github.com/apache/pulsar/blob/master/.github/PULL_REQUEST_TEMPLATE.md#documentation), which helps others know more about the changes. Can you provide doc-related info in this and future PR descriptions? Thanks)


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [pulsar] rdhabalia commented on a change in pull request #9292: [pulsar-broker] Support large number of unack message store for cursor recovery

Posted by GitBox <gi...@apache.org>.
rdhabalia commented on a change in pull request #9292:
URL: https://github.com/apache/pulsar/pull/9292#discussion_r564314167



##########
File path: managed-ledger/src/main/proto/MLDataFormats.proto
##########
@@ -71,6 +71,9 @@ message PositionInfo {
 
     // Store which index in the batch message has been deleted
     repeated BatchedEntryDeletionIndexInfo batchedEntryDeletionIndexInfo = 5;
+
+    // serialized individualDeletedMessages
+    optional bytes serializedIndividualDeletedMessages = 6;

Review comment:
       @merlimat removed the serialization and enocded to protobuf.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [pulsar] rdhabalia removed a comment on pull request #9292: [pulsar-broker] Support large number of unack message store for cursor recovery

Posted by GitBox <gi...@apache.org>.
rdhabalia removed a comment on pull request #9292:
URL: https://github.com/apache/pulsar/pull/9292#issuecomment-766696902


   /pulsarbot run-failure-checks


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [pulsar] rdhabalia commented on pull request #9292: [pulsar-broker] Support large number of unack message store for cursor recovery

Posted by GitBox <gi...@apache.org>.
rdhabalia commented on pull request #9292:
URL: https://github.com/apache/pulsar/pull/9292#issuecomment-768859816


   @codelipenghui 
   this improvement serializes long-array to protobuf directly which should not cost cpu, and managed-ledger has a cap on max entries per ledger so, that will make sure to have a limit on size of long array. encoded 150 long shouldn't be a problem in terms of size and cpu as broker persist into bookie into a separate non-io background thread. we can also control size with `managedLedgerMaxUnackedRangesToPersist` config.
   
   >  I think we should to add a flag for enabling or disabling this enhancement.
   
   Sure, let me add the flag.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [pulsar] rdhabalia commented on pull request #9292: [pulsar-broker] Support large number of unack message store for cursor recovery

Posted by GitBox <gi...@apache.org>.
rdhabalia commented on pull request #9292:
URL: https://github.com/apache/pulsar/pull/9292#issuecomment-768004250


   /pulsarbot run-failure-checks


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org