You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pulsar.apache.org by "shibd (via GitHub)" <gi...@apache.org> on 2025/02/26 04:05:45 UTC

[PR] Fix wrong result of hasNext after seeking by id or time [pulsar-client-go]

shibd opened a new pull request, #1340:
URL: https://github.com/apache/pulsar-client-go/pull/1340

   ### BackKnowage
   
   There are **three attributes** to handle seek/filter/subscribe for Reader.
   
   1. `startMessageID`: The start message ID, which can be specified by the user when creating the reader. It can be set to a specific position, or to the `earliest` or `latest`. When creating a consumer, this is [set in the request](https://github.com/apache/pulsar-client-go/blob/9aeeb4e7bbfcaa4039d66ab2857888be2d6fcaf3/pulsar/consumer_partition.go#L1979), and the broker uses it to set mark delete position.
   2. `lastDequeuedMsg`: The most recent message dispatched from the consumer's internal queue to the user. **If it's nil, it means no messages have been delivered to the user. If there is a value, it indicates the user has consumed up to this point.**
   3. `lastMessageInBroker`: The ID of the last message in the topic on the broker.
   
   #### Question-1: How to determine hasNext?
   
   [Refer to this code](https://github.com/apache/pulsar-client-go/blob/9aeeb4e7bbfcaa4039d66ab2857888be2d6fcaf3/pulsar/consumer_partition.go#L2224)
   
   In simple terms, to minimize interaction with the broker:
   
   1. If `lastDequeuedMsg` has a value, it clearly indicates consumption up to this position. Compare this `lastDequeuedMsg with lastMessageInBroker`.
   2. If there is no value, compare `startMessageID with lastMessageInBroker`.
   
   The final comparison result, **if true**, indicates there really are messages. **If false**, it doesn’t necessarily mean there are no messages, because `lastMessageInBroker` is a cache, and the topic may have more messages already written. Therefore, **lastMessageInBroker is updated from the broker again for reevaluation**
   
   ### Question-2: What happens when the consumer reconnects?
   
   When the consumer reconnects, it clears all messages in the `receive queue` and **updates startMessageId with the first message in the queue**. This indicates that the user has been consumed up to this position.
   
   
   
   #### !!! The above logic works well without a seek operation, but once the user calls a seek operation, the situation becomes special.
   
   #### Question-3: How does the seek operation work?
   
   **For the broker:**
   When the client sends a seek request to the broker, the broker will do the following:
   1. Rewind the cursor to the seek position (ID or time).
   2. Send a close request to the consumer.
   3. Send a respond that the seek operation is complete.
   
   **For the client:**
   1. Send a seek request. Wait for the broker to respond that the seek is complete
   2. Set `lastDequeuedMsg` to null because after seeking, this value will be invalid. `**(Tip: Currently missing, this PR fixes it.)**`
   3. Clear the queue to avoid resetting `startMessageId` to the first position of the queue upon reconnection. `**(Tip: Currently missing, this PR fixes it.)**`
   4. If seekByID:
       - If the passed ID is a specific position or earliest, set startMessageId to seek Id. `**(Tip: Currently missing, this PR fixes it.)**`
       - If the passed ID is latest, obtain the topic `lastMessageId` and set startMessageId to it. `**(Tip: Currently missing, a separate PR will fix this later as it [currently doesn’t support directly passing latest id](https://github.com/apache/pulsar-client-go/blob/9aeeb4e7bbfcaa4039d66ab2857888be2d6fcaf3/pulsar/reader_impl.go#L210-L214).)**`
   5. If seekByTime, the client doesn’t know the messageId corresponding to the time during the seek, it needs to call getLastMessageId and set it to startMessageId. `**(Tip: Currently missing, this PR fixes it.)**`
   6. Seek complete, then return to the user.
   7. BTW: Actually, this is just the completion of the seek operation. The real completion requires waiting for the consumer to successfully reconnect to the broker, but this happens in the background and the user doesn't need to be concerned about it. Also, **note that we don't need to wait for the consumer to reconnect to the broker before returning the seek completion to the client**, because when the broker responds that the seek is complete, **it has already reset the cursor.** After the client receives the response, it has cleared the queue, so no previous messages will come in. In extremely rare cases (**where data from the old connection's buffer re-enters the queue after clearing**), [we will filter these messages using startMessageId to filter message ID](https://github.com/apache/pulsar-client-go/blob/9aeeb4e7bbfcaa4039d66ab2857888be2d6fcaf3/pulsar/consumer_partition.go#L1475-L1486).
   
   
   ### Modifications
   
   You can run tests `TestReaderWithSeekByID` and `TestReaderWithSeekByTime` to understand the user-side issues fixed by this PR. Simply put, after seeking, calling `hasNext` again should meet expectations.
   
   
   
   ### Verifying this change
   1. Add TestReaderWithSeekByID and TestReaderWithSeekByTime to cover this change.
   2. I did not change any other test, other tests passed mean not introducing any break change.
   
   
   ### Does this pull request potentially affect one of the following parts:
   - No
   
   ### Documentation
   - No


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Fix wrong result of hasNext after seeking by id or time [pulsar-client-go]

Posted by "shibd (via GitHub)" <gi...@apache.org>.
shibd commented on code in PR #1340:
URL: https://github.com/apache/pulsar-client-go/pull/1340#discussion_r1973001587


##########
pulsar/consumer_test.go:
##########
@@ -1262,8 +1262,9 @@ func TestConsumerSeek(t *testing.T) {
 	defer producer.Close()
 
 	consumer, err := client.Subscribe(ConsumerOptions{
-		Topic:            topicName,
-		SubscriptionName: "sub-1",
+		Topic:                   topicName,
+		SubscriptionName:        "sub-1",
+		StartMessageIDInclusive: true,

Review Comment:
   This is necessary, Because this test seeks to N−50 and expects to receive this message next time.
   
   In fact, StartMessageIDInclusive should also apply to the seek operation.
   
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Fix wrong result of hasNext after seeking by id or time [pulsar-client-go]

Posted by "BewareMyPower (via GitHub)" <gi...@apache.org>.
BewareMyPower commented on code in PR #1340:
URL: https://github.com/apache/pulsar-client-go/pull/1340#discussion_r1978562318


##########
pulsar/consumer_test.go:
##########
@@ -1262,8 +1262,9 @@ func TestConsumerSeek(t *testing.T) {
 	defer producer.Close()
 
 	consumer, err := client.Subscribe(ConsumerOptions{
-		Topic:            topicName,
-		SubscriptionName: "sub-1",
+		Topic:                   topicName,
+		SubscriptionName:        "sub-1",
+		StartMessageIDInclusive: true,

Review Comment:
   Unlike `hasNext()`, if it returns true and there is no more message, the application will be stuck at `Next`.
   
   However, the existing behavior of `seek` won't cause such serious result. Indeed, some users might think it's a special behavior of Pulsar Go client and set the message id with the inclusive semantics.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Fix wrong result of reader.hasNext/Next after seeking by id or time [pulsar-client-go]

Posted by "geniusjoe (via GitHub)" <gi...@apache.org>.
geniusjoe commented on PR #1340:
URL: https://github.com/apache/pulsar-client-go/pull/1340#issuecomment-2706198300

   > If any user depends on the existing `seek` behavior (inclusive by default), he/she might treat it as a "message lost" issue.
   > 
   > This behavior change is beyond the scope of this PR. I also think it needs a discussion at the mail list. We also need to document the change and tell existing users to set `startMessageInclusive` with true to keep the old behavior. But it's still a breaking change because if `startMessageInclusive` is true, the behavior of subscribing also changes.
   > 
   > Therefore, my opinion is to keep the existing seek behavior not changed for now. The "fix" could be delayed to next release.
   
   I'm not very sure what do you mean "message lost"? Compared to current fixes, does previous behavior continue to consume messages regardless of `seek()` happened, or just continue to consume messages in client cache and then jump to consume the position which `seek()` defined?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Fix wrong result of reader.hasNext/Next after seeking by id or time [pulsar-client-go]

Posted by "nodece (via GitHub)" <gi...@apache.org>.
nodece commented on code in PR #1340:
URL: https://github.com/apache/pulsar-client-go/pull/1340#discussion_r1986593581


##########
pulsar/consumer_partition.go:
##########
@@ -2256,12 +2306,14 @@ func convertToMessageID(id *pb.MessageIdData) *trackingMessageID {
 
 	msgID := &trackingMessageID{
 		messageID: &messageID{
-			ledgerID: int64(*id.LedgerId),
-			entryID:  int64(*id.EntryId),
+			ledgerID:  int64(id.GetLedgerId()),
+			entryID:   int64(id.GetEntryId()),
+			batchIdx:  id.GetBatchIndex(),
+			batchSize: id.GetBatchSize(),
 		},
 	}
-	if id.BatchIndex != nil {
-		msgID.batchIdx = *id.BatchIndex
+	if msgID.batchIdx == -1 {
+		msgID.batchIdx = 0

Review Comment:
   > What are your main concerns? Currently, this method is only used for comparison in `hasNext`.
   
   If `3141:1:0` is a non-batch message, the correct message ID should be `3141:1:0:-1`, not `3141:1:0:0`.
   
   The batch index is greater than -1, which means this message is from the batch messages.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Fix wrong result of hasNext after seeking by id or time [pulsar-client-go]

Posted by "shibd (via GitHub)" <gi...@apache.org>.
shibd commented on code in PR #1340:
URL: https://github.com/apache/pulsar-client-go/pull/1340#discussion_r1973001494


##########
pulsar/consumer_zero_queue_test.go:
##########
@@ -474,6 +474,7 @@ func TestZeroQueueConsumer_Seek(t *testing.T) {
 		Topic:                   topicName,
 		EnableZeroQueueConsumer: true,
 		SubscriptionName:        "sub-1",
+		StartMessageIDInclusive: true,

Review Comment:
   This is necessary, Because this test seeks to N−5 and expects to receive this message next time. 
   
   In fact, StartMessageIDInclusive should also apply to the seek operation.



##########
pulsar/consumer_test.go:
##########
@@ -1262,8 +1262,9 @@ func TestConsumerSeek(t *testing.T) {
 	defer producer.Close()
 
 	consumer, err := client.Subscribe(ConsumerOptions{
-		Topic:            topicName,
-		SubscriptionName: "sub-1",
+		Topic:                   topicName,
+		SubscriptionName:        "sub-1",
+		StartMessageIDInclusive: true,

Review Comment:
   This is necessary, Because this test seeks to N−5 and expects to receive this message next time.
   
   In fact, StartMessageIDInclusive should also apply to the seek operation.
   
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Fix wrong result of hasNext after seeking by id or time [pulsar-client-go]

Posted by "shibd (via GitHub)" <gi...@apache.org>.
shibd commented on code in PR #1340:
URL: https://github.com/apache/pulsar-client-go/pull/1340#discussion_r1972997599


##########
pulsar/consumer_partition.go:
##########
@@ -2195,6 +2226,24 @@ func (pc *partitionConsumer) discardCorruptedMessage(msgID *pb.MessageIdData,
 }
 
 func (pc *partitionConsumer) hasNext() bool {
+
+	// If a seek by time has been performed, then the `startMessageId` becomes irrelevant.
+	// We need to compare `markDeletePosition` and `lastMessageId`,
+	// and then reset `startMessageID` to `markDeletePosition`.
+	if pc.hasSoughtByTime.CompareAndSwap(true, false) {

Review Comment:
   However, we need to add a new condition: when `&& lastDequeuedMsg == nil`. This is because the user might have received the message but hasn't called this method.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Fix wrong result of hasNext after seeking by id or time [pulsar-client-go]

Posted by "BewareMyPower (via GitHub)" <gi...@apache.org>.
BewareMyPower commented on code in PR #1340:
URL: https://github.com/apache/pulsar-client-go/pull/1340#discussion_r1978551390


##########
pulsar/consumer_partition.go:
##########
@@ -2195,6 +2226,24 @@ func (pc *partitionConsumer) discardCorruptedMessage(msgID *pb.MessageIdData,
 }
 
 func (pc *partitionConsumer) hasNext() bool {
+
+	// If a seek by time has been performed, then the `startMessageId` becomes irrelevant.
+	// We need to compare `markDeletePosition` and `lastMessageId`,
+	// and then reset `startMessageID` to `markDeletePosition`.
+	if pc.lastDequeuedMsg == nil && pc.hasSoughtByTime.CompareAndSwap(true, false) {
+		res, err := pc.getLastMessageIDAndMarkDeletePosition()
+		if err != nil {
+			pc.log.WithError(err).Error("Failed to get last message id")

Review Comment:
   In this case, `hasSoughtByTime` will still be `false`. Should we restore it with true here?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Fix wrong result of hasNext after seeking by id or time [pulsar-client-go]

Posted by "BewareMyPower (via GitHub)" <gi...@apache.org>.
BewareMyPower commented on code in PR #1340:
URL: https://github.com/apache/pulsar-client-go/pull/1340#discussion_r1972873268


##########
pulsar/consumer_test.go:
##########
@@ -1262,8 +1262,9 @@ func TestConsumerSeek(t *testing.T) {
 	defer producer.Close()
 
 	consumer, err := client.Subscribe(ConsumerOptions{
-		Topic:            topicName,
-		SubscriptionName: "sub-1",
+		Topic:                   topicName,
+		SubscriptionName:        "sub-1",
+		StartMessageIDInclusive: true,

Review Comment:
   This change seems irrelevant?



##########
pulsar/consumer_zero_queue_test.go:
##########
@@ -474,6 +474,7 @@ func TestZeroQueueConsumer_Seek(t *testing.T) {
 		Topic:                   topicName,
 		EnableZeroQueueConsumer: true,
 		SubscriptionName:        "sub-1",
+		StartMessageIDInclusive: true,

Review Comment:
   irrelevant change again



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Fix wrong result of hasNext/Receive after seeking by id or time [pulsar-client-go]

Posted by "shibd (via GitHub)" <gi...@apache.org>.
shibd commented on PR #1340:
URL: https://github.com/apache/pulsar-client-go/pull/1340#issuecomment-2703705672

   > This behavior change is beyond the scope of this PR.
   
   I changed the title. I think my intention was to clarify the issue with `hasNext` or `receive` after `seek`.
   
   I believe email discussions are pointless because this is a behavior fix. If we need to inform users, we should clearly explain it in the release notes for the next release.
   
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Fix wrong result of reader.hasNext/Next after seeking by id or time [pulsar-client-go]

Posted by "BewareMyPower (via GitHub)" <gi...@apache.org>.
BewareMyPower commented on PR #1340:
URL: https://github.com/apache/pulsar-client-go/pull/1340#issuecomment-2706634635

   @geniusjoe
   
   ```go
   func TestReaderSeek2(t *testing.T) {
           client, err := NewClient(ClientOptions{
                   URL: lookupURL,
           })
           assert.Nil(t, err)
           defer client.Close()
   
           topicName := newTopicName()
           ctx := context.Background()
           producer, err := client.CreateProducer(ProducerOptions{
                   Topic: topicName,
           })
           ids := make([]MessageID, 0)
           assert.Nil(t, err)
           for i := 0; i < 10; i++ {
                   id, err := producer.Send(ctx, &ProducerMessage{
                           Payload: []byte(fmt.Sprintf("msg-%d", i)),
                   })
                   assert.Nil(t, err)
                   ids = append(ids, id)
           }
           reader, err := client.CreateReader(ReaderOptions{
                   Topic:          topicName,
                   StartMessageID: EarliestMessageID(),
           })
           // Seek to id[3] so that reader can continue reading from "msg-3"
           reader.Seek(ids[3])
           msg, err := reader.Next(ctx)
           assert.Nil(t, err)
           t.Log(string(msg.Payload()))
   }
   ```
   
   Users seek to `ids[3]` (the position of "msg-3") to expect consuming from "msg-3". However, after the change, "msg-3" is unexpectedly skipped, the test code above will print "msg-4".


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Fix wrong result of hasNext after seeking by id or time [pulsar-client-go]

Posted by "BewareMyPower (via GitHub)" <gi...@apache.org>.
BewareMyPower commented on code in PR #1340:
URL: https://github.com/apache/pulsar-client-go/pull/1340#discussion_r1978554860


##########
pulsar/consumer_test.go:
##########
@@ -1262,8 +1262,9 @@ func TestConsumerSeek(t *testing.T) {
 	defer producer.Close()
 
 	consumer, err := client.Subscribe(ConsumerOptions{
-		Topic:            topicName,
-		SubscriptionName: "sub-1",
+		Topic:                   topicName,
+		SubscriptionName:        "sub-1",
+		StartMessageIDInclusive: true,

Review Comment:
   Anyway, this is a breaking change. Though the existing behavior might not be expected, users might already depend on this behavior.
   
   Unlike `hasNext()`, the existing behavior is that if a consumer seeks to the message id, it will still consume that message. While this PR's main purpose is to resolve the wrong result of `hasNext()`. I think you should use a different PR for changing the `seek` behavior.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Fix wrong result of hasNext after seeking by id or time [pulsar-client-go]

Posted by "BewareMyPower (via GitHub)" <gi...@apache.org>.
BewareMyPower commented on code in PR #1340:
URL: https://github.com/apache/pulsar-client-go/pull/1340#discussion_r1972923914


##########
pulsar/consumer_partition.go:
##########
@@ -2195,6 +2226,24 @@ func (pc *partitionConsumer) discardCorruptedMessage(msgID *pb.MessageIdData,
 }
 
 func (pc *partitionConsumer) hasNext() bool {
+
+	// If a seek by time has been performed, then the `startMessageId` becomes irrelevant.
+	// We need to compare `markDeletePosition` and `lastMessageId`,
+	// and then reset `startMessageID` to `markDeletePosition`.
+	if pc.hasSoughtByTime.CompareAndSwap(true, false) {

Review Comment:
   This CAS operation seems confusing.
   
   1. `hasSoughtByTime` is set with true in `internalSeekByTime`, so concurrent seek could happen. In Java client, it's prevented by `seekStatus.compareAndSet(SeekStatus.NOT_STARTED, SeekStatus.IN_PROGRESS)`.
   2. Here it's changed from true to false, which means next time `hasNext` is called, it will skip the code block here.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Fix wrong result of reader.hasNext/Next after seeking by id or time [pulsar-client-go]

Posted by "nodece (via GitHub)" <gi...@apache.org>.
nodece commented on code in PR #1340:
URL: https://github.com/apache/pulsar-client-go/pull/1340#discussion_r1985385864


##########
pulsar/consumer_partition.go:
##########
@@ -2256,12 +2306,14 @@ func convertToMessageID(id *pb.MessageIdData) *trackingMessageID {
 
 	msgID := &trackingMessageID{
 		messageID: &messageID{
-			ledgerID: int64(*id.LedgerId),
-			entryID:  int64(*id.EntryId),
+			ledgerID:  int64(id.GetLedgerId()),
+			entryID:   int64(id.GetEntryId()),
+			batchIdx:  id.GetBatchIndex(),
+			batchSize: id.GetBatchSize(),
 		},
 	}
-	if id.BatchIndex != nil {
-		msgID.batchIdx = *id.BatchIndex
+	if msgID.batchIdx == -1 {
+		msgID.batchIdx = 0

Review Comment:
   Why reset this value to 0? -1 is not the batch message, 0 is batch.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Fix wrong result of hasNext after seeking by id or time [pulsar-client-go]

Posted by "shibd (via GitHub)" <gi...@apache.org>.
shibd commented on code in PR #1340:
URL: https://github.com/apache/pulsar-client-go/pull/1340#discussion_r1972997599


##########
pulsar/consumer_partition.go:
##########
@@ -2195,6 +2226,24 @@ func (pc *partitionConsumer) discardCorruptedMessage(msgID *pb.MessageIdData,
 }
 
 func (pc *partitionConsumer) hasNext() bool {
+
+	// If a seek by time has been performed, then the `startMessageId` becomes irrelevant.
+	// We need to compare `markDeletePosition` and `lastMessageId`,
+	// and then reset `startMessageID` to `markDeletePosition`.
+	if pc.hasSoughtByTime.CompareAndSwap(true, false) {

Review Comment:
   However, we need to add a new condition: when `&& lastDequeuedMsg == nil`. This is because the user might have received the message but hasn't called this `HasNext`.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Fix wrong result of reader.hasNext/Next after seeking by id or time [pulsar-client-go]

Posted by "shibd (via GitHub)" <gi...@apache.org>.
shibd commented on code in PR #1340:
URL: https://github.com/apache/pulsar-client-go/pull/1340#discussion_r1986205947


##########
pulsar/consumer_partition.go:
##########
@@ -2256,12 +2306,14 @@ func convertToMessageID(id *pb.MessageIdData) *trackingMessageID {
 
 	msgID := &trackingMessageID{
 		messageID: &messageID{
-			ledgerID: int64(*id.LedgerId),
-			entryID:  int64(*id.EntryId),
+			ledgerID:  int64(id.GetLedgerId()),
+			entryID:   int64(id.GetEntryId()),
+			batchIdx:  id.GetBatchIndex(),
+			batchSize: id.GetBatchSize(),
 		},
 	}
-	if id.BatchIndex != nil {
-		msgID.batchIdx = *id.BatchIndex
+	if msgID.batchIdx == -1 {
+		msgID.batchIdx = 0

Review Comment:
   In the Go Client, it's not like this. You can run the following code to test:
   
   Need change msg id String method first.
   ``` go
   func (id *messageID) String() string {
   	return fmt.Sprintf("%d:%d:%d:%d", id.ledgerID, id.entryID, id.batchIdx, id.partitionIdx)
   }
   ```
   
   
   ``` go
   func TestMessageID_BatchIdx(t *testing.T) {
   	client, err := NewClient(ClientOptions{
   		URL: lookupURL,
   	})
   
   	assert.Nil(t, err)
   	defer client.Close()
   
   	topic := newTopicName()
   	ctx := context.Background()
   
   	// create producer
   	producer, err := client.CreateProducer(ProducerOptions{
   		Topic:           topic,
   		DisableBatching: true,
   	})
   	assert.Nil(t, err)
   	defer producer.Close()
   
   	// create consumer
   	consumer, err := client.Subscribe(ConsumerOptions{
   		Topic:             topic,
   		SubscriptionName:  "my-sub",
   		Type:              Exclusive,
   		ReceiverQueueSize: 10,
   	})
   	assert.Nil(t, err)
   	defer consumer.Close()
   
   	// send 10 messages with no batch
   	var lastMsgID MessageID
   	for i := 0; i < 10; i++ {
   		lastMsgID, err = producer.Send(ctx, &ProducerMessage{
   			Payload: []byte(fmt.Sprintf("hello-%d", i)),
   		})
   		t.Logf("Sended message msgId: %#v \n", lastMsgID.String())
   		assert.NoError(t, err)
   		assert.NotNil(t, lastMsgID)
   	}
   
   	// Receive 10 message with no batch
   	for i := 0; i < 10; i++ {
   		msg, err := consumer.Receive(ctx)
   		assert.Nil(t, err)
   		t.Logf("Received message msgId: %#v \n", msg.ID().String())
   		consumer.Ack(msg)
   	}
   }
   
   output:
      consumer_test.go:572: Sended message msgId: "3141:0:0:0" 
       consumer_test.go:572: Sended message msgId: "3141:1:0:0" 
       consumer_test.go:572: Sended message msgId: "3141:2:0:0" 
       consumer_test.go:572: Sended message msgId: "3141:3:0:0" 
       consumer_test.go:572: Sended message msgId: "3141:4:0:0" 
       consumer_test.go:572: Sended message msgId: "3141:5:0:0" 
       consumer_test.go:572: Sended message msgId: "3141:6:0:0" 
       consumer_test.go:572: Sended message msgId: "3141:7:0:0" 
       consumer_test.go:572: Sended message msgId: "3141:8:0:0" 
       consumer_test.go:572: Sended message msgId: "3141:9:0:0" 
       consumer_test.go:581: Received message msgId: "3141:0:0:0" 
       consumer_test.go:581: Received message msgId: "3141:1:0:0" 
       consumer_test.go:581: Received message msgId: "3141:2:0:0" 
       consumer_test.go:581: Received message msgId: "3141:3:0:0" 
       consumer_test.go:581: Received message msgId: "3141:4:0:0" 
       consumer_test.go:581: Received message msgId: "3141:5:0:0" 
       consumer_test.go:581: Received message msgId: "3141:6:0:0" 
       consumer_test.go:581: Received message msgId: "3141:7:0:0" 
       consumer_test.go:581: Received message msgId: "3141:8:0:0" 
       consumer_test.go:581: Received message msgId: "3141:9:0:0" 
   ```
   
   In Java, it will be a different implementation class, so there's no batchIndex field when disabled batch.
   
   If we want to keep it consistent, we can change it later.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Fix wrong result of reader.hasNext/Next after seeking by id or time [pulsar-client-go]

Posted by "nodece (via GitHub)" <gi...@apache.org>.
nodece commented on code in PR #1340:
URL: https://github.com/apache/pulsar-client-go/pull/1340#discussion_r1986356408


##########
pulsar/consumer_partition.go:
##########
@@ -2256,12 +2306,14 @@ func convertToMessageID(id *pb.MessageIdData) *trackingMessageID {
 
 	msgID := &trackingMessageID{
 		messageID: &messageID{
-			ledgerID: int64(*id.LedgerId),
-			entryID:  int64(*id.EntryId),
+			ledgerID:  int64(id.GetLedgerId()),
+			entryID:   int64(id.GetEntryId()),
+			batchIdx:  id.GetBatchIndex(),
+			batchSize: id.GetBatchSize(),
 		},
 	}
-	if id.BatchIndex != nil {
-		msgID.batchIdx = *id.BatchIndex
+	if msgID.batchIdx == -1 {
+		msgID.batchIdx = 0

Review Comment:
   I see, because the batchIndex type is `int32`, not point.
   
   I think you should remove this change. When the batch index is -1, do not change the value to 0.
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Fix wrong result of hasNext after seeking by id or time [pulsar-client-go]

Posted by "shibd (via GitHub)" <gi...@apache.org>.
shibd commented on code in PR #1340:
URL: https://github.com/apache/pulsar-client-go/pull/1340#discussion_r1983247669


##########
pulsar/consumer_partition.go:
##########
@@ -2195,6 +2226,24 @@ func (pc *partitionConsumer) discardCorruptedMessage(msgID *pb.MessageIdData,
 }
 
 func (pc *partitionConsumer) hasNext() bool {
+
+	// If a seek by time has been performed, then the `startMessageId` becomes irrelevant.
+	// We need to compare `markDeletePosition` and `lastMessageId`,
+	// and then reset `startMessageID` to `markDeletePosition`.
+	if pc.lastDequeuedMsg == nil && pc.hasSoughtByTime.CompareAndSwap(true, false) {
+		res, err := pc.getLastMessageIDAndMarkDeletePosition()
+		if err != nil {
+			pc.log.WithError(err).Error("Failed to get last message id")

Review Comment:
   Yes, it should actually return an error, but the API wasn't designed that way. Of course, I'll set it to true and let the next call continue with this logic. 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Fix wrong result of reader.hasNext/Next after seeking by id or time [pulsar-client-go]

Posted by "shibd (via GitHub)" <gi...@apache.org>.
shibd commented on code in PR #1340:
URL: https://github.com/apache/pulsar-client-go/pull/1340#discussion_r1986459004


##########
pulsar/consumer_partition.go:
##########
@@ -2256,12 +2306,14 @@ func convertToMessageID(id *pb.MessageIdData) *trackingMessageID {
 
 	msgID := &trackingMessageID{
 		messageID: &messageID{
-			ledgerID: int64(*id.LedgerId),
-			entryID:  int64(*id.EntryId),
+			ledgerID:  int64(id.GetLedgerId()),
+			entryID:   int64(id.GetEntryId()),
+			batchIdx:  id.GetBatchIndex(),
+			batchSize: id.GetBatchSize(),
 		},
 	}
-	if id.BatchIndex != nil {
-		msgID.batchIdx = *id.BatchIndex
+	if msgID.batchIdx == -1 {
+		msgID.batchIdx = 0

Review Comment:
   If remove this, when seek to a messageId(no batch), let say`(3141:1:0:0)`, When we compare, there will be unexpected behavior because the `lastBrokerMessageId` we get from the broker and convert to : `3141:1:-1:0`.
   
   What are your main concerns? Currently, this method is only used for comparison in `hasNext`.
   
   I think we need to maintain consistency because now the non-batch message ID returned by the producer and consumer is `batchIndex=0`.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Fix wrong result of hasNext after seeking by id or time [pulsar-client-go]

Posted by "BewareMyPower (via GitHub)" <gi...@apache.org>.
BewareMyPower commented on code in PR #1340:
URL: https://github.com/apache/pulsar-client-go/pull/1340#discussion_r1972862055


##########
pulsar/reader_test.go:
##########
@@ -1070,3 +1070,159 @@ func TestReaderNextReturnsOnClosedConsumer(t *testing.T) {
 	assert.ErrorAs(t, err, &e)
 	assert.Equal(t, ConsumerClosed, e.Result())
 }
+
+func testReaderSeekByIDWithHasNext(t *testing.T, startMessageID MessageID, startMessageIDInclusive bool) {
+	client, err := NewClient(ClientOptions{
+		URL: lookupURL,
+	})
+
+	assert.Nil(t, err)
+	defer client.Close()
+
+	topic := newTopicName()
+	ctx := context.Background()
+
+	// create producer
+	producer, err := client.CreateProducer(ProducerOptions{
+		Topic:           topic,
+		DisableBatching: true,
+	})
+	assert.Nil(t, err)
+	defer producer.Close()
+
+	// send 100 messages
+	var lastMsgID MessageID
+	for i := 0; i < 10; i++ {
+		lastMsgID, err = producer.Send(ctx, &ProducerMessage{
+			Payload: []byte(fmt.Sprintf("hello-%d", i)),
+		})
+		fmt.Println(lastMsgID.String())
+		assert.NoError(t, err)
+		assert.NotNil(t, lastMsgID)
+	}
+
+	reader, err := client.CreateReader(ReaderOptions{
+		Topic:                   topic,
+		StartMessageID:          startMessageID,
+		StartMessageIDInclusive: startMessageIDInclusive,
+	})
+	assert.Nil(t, err)
+	defer reader.Close()
+
+	// Seek to last message ID
+	err = reader.Seek(lastMsgID)
+	assert.NoError(t, err)
+
+	if startMessageIDInclusive {
+		assert.True(t, reader.HasNext())
+		ctx, cancel := context.WithTimeout(context.Background(), 1*time.Second)
+		msg, err := reader.Next(ctx)
+		assert.NoError(t, err)
+		assert.NotNil(t, msg)
+		assert.True(t, messageIDCompare(lastMsgID, msg.ID()) == 0)
+		cancel()
+	} else {
+		assert.False(t, reader.HasNext())
+		ctx, cancel := context.WithTimeout(context.Background(), 1*time.Second)
+		msg, err := reader.Next(ctx)
+		assert.Error(t, err)
+		assert.Nil(t, msg)
+		cancel()
+	}
+
+}
+
+func TestReaderWithSeekByID(t *testing.T) {
+	testReaderSeekByIDWithHasNext(t, EarliestMessageID(), false)
+	testReaderSeekByIDWithHasNext(t, EarliestMessageID(), true)
+	testReaderSeekByIDWithHasNext(t, LatestMessageID(), false)
+	testReaderSeekByIDWithHasNext(t, LatestMessageID(), true)
+}
+
+func testReaderSeekByTimeWithHasNext(t *testing.T, startMessageID MessageID) {
+	client, err := NewClient(ClientOptions{
+		URL: lookupURL,
+	})
+
+	assert.Nil(t, err)
+	defer client.Close()
+
+	topic := newTopicName()
+	ctx := context.Background()
+
+	// create producer
+	producer, err := client.CreateProducer(ProducerOptions{
+		Topic:           topic,
+		DisableBatching: true,
+	})
+	assert.Nil(t, err)
+	defer producer.Close()
+
+	// 1. send 10 messages
+	var lastMsgID MessageID
+	for i := 0; i < 10; i++ {
+		lastMsgID, err = producer.Send(ctx, &ProducerMessage{
+			Payload: []byte(fmt.Sprintf("hello-%d", i)),
+		})
+		fmt.Println(lastMsgID.String())
+		assert.NoError(t, err)
+
+		assert.NotNil(t, lastMsgID)
+	}
+
+	// 2. create reader
+	reader, err := client.CreateReader(ReaderOptions{
+		Topic:                   topic,
+		StartMessageID:          startMessageID,
+		StartMessageIDInclusive: false,
+	})
+	assert.Nil(t, err)
+	defer reader.Close()
+
+	// 3. Seek time to now
+	reader.SeekByTime(time.Now())
+
+	// 4. Should not receive msg
+	{
+		assert.False(t, reader.HasNext())
+		timeoutCtx, cancel := context.WithTimeout(context.Background(), 1*time.Second)
+		msg, err := reader.Next(timeoutCtx)
+		assert.Error(t, err)
+		assert.Nil(t, msg)
+		cancel()
+	}
+
+	// 5. send more 10 messages
+	for i := 0; i < 10; i++ {
+		lastMsgID, err = producer.Send(ctx, &ProducerMessage{
+			Payload: []byte(fmt.Sprintf("hello2-%d", i)),
+		})
+		fmt.Println(lastMsgID.String())
+		assert.NoError(t, err)
+		assert.NotNil(t, lastMsgID)
+	}
+
+	// 6. Assert these messages are received
+	for i := 0; i < 10; i++ {
+		assert.True(t, reader.HasNext())
+		msg, err := reader.Next(context.Background())
+		assert.NoError(t, err)
+		assert.Equal(t, fmt.Sprintf("hello2-%d", i), string(msg.Payload()))
+	}
+
+	// assert not more msg
+	{
+		assert.False(t, reader.HasNext())
+		timeoutCtx, cancel := context.WithTimeout(context.Background(), 1*time.Second)
+		msg, err := reader.Next(timeoutCtx)
+		assert.Error(t, err)
+		assert.Nil(t, msg)
+		cancel()
+	}
+}
+func TestReaderWithSeekByTime(t *testing.T) {
+	testReaderSeekByTimeWithHasNext(t, EarliestMessageID())
+	testReaderSeekByTimeWithHasNext(t, EarliestMessageID())
+	testReaderSeekByTimeWithHasNext(t, LatestMessageID())
+	testReaderSeekByTimeWithHasNext(t, LatestMessageID())

Review Comment:
   I don't get why did you run twice for each call? BTW, for parameterized tests, you can call `T.Run` so that each single sub-test can have its own test name and fail independently. See example here https://github.com/apache/pulsar-client-go/blob/020b55c1dce4475fe8781f4fb6f858de903844b8/pulsar/consumer_test.go#L4760



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Fix wrong result of reader.hasNext/Next after seeking by id or time [pulsar-client-go]

Posted by "nodece (via GitHub)" <gi...@apache.org>.
nodece commented on code in PR #1340:
URL: https://github.com/apache/pulsar-client-go/pull/1340#discussion_r1986694510


##########
pulsar/consumer_partition.go:
##########
@@ -2256,12 +2306,14 @@ func convertToMessageID(id *pb.MessageIdData) *trackingMessageID {
 
 	msgID := &trackingMessageID{
 		messageID: &messageID{
-			ledgerID: int64(*id.LedgerId),
-			entryID:  int64(*id.EntryId),
+			ledgerID:  int64(id.GetLedgerId()),
+			entryID:   int64(id.GetEntryId()),
+			batchIdx:  id.GetBatchIndex(),
+			batchSize: id.GetBatchSize(),
 		},
 	}
-	if id.BatchIndex != nil {
-		msgID.batchIdx = *id.BatchIndex
+	if msgID.batchIdx == -1 {
+		msgID.batchIdx = 0

Review Comment:
   The PR didn't change the original behavior.
   
   BTW, I think this is a worthy point of improvement.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Fix wrong result of reader.hasNext/Next after seeking by id or time [pulsar-client-go]

Posted by "nodece (via GitHub)" <gi...@apache.org>.
nodece merged PR #1340:
URL: https://github.com/apache/pulsar-client-go/pull/1340


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Fix wrong result of hasNext after seeking by id or time [pulsar-client-go]

Posted by "BewareMyPower (via GitHub)" <gi...@apache.org>.
BewareMyPower commented on code in PR #1340:
URL: https://github.com/apache/pulsar-client-go/pull/1340#discussion_r1972840290


##########
pulsar/reader_test.go:
##########
@@ -1070,3 +1070,159 @@ func TestReaderNextReturnsOnClosedConsumer(t *testing.T) {
 	assert.ErrorAs(t, err, &e)
 	assert.Equal(t, ConsumerClosed, e.Result())
 }
+
+func testReaderSeekByIDWithHasNext(t *testing.T, startMessageID MessageID, startMessageIDInclusive bool) {
+	client, err := NewClient(ClientOptions{
+		URL: lookupURL,
+	})
+
+	assert.Nil(t, err)
+	defer client.Close()
+
+	topic := newTopicName()
+	ctx := context.Background()
+
+	// create producer
+	producer, err := client.CreateProducer(ProducerOptions{
+		Topic:           topic,
+		DisableBatching: true,
+	})
+	assert.Nil(t, err)
+	defer producer.Close()
+
+	// send 100 messages
+	var lastMsgID MessageID
+	for i := 0; i < 10; i++ {
+		lastMsgID, err = producer.Send(ctx, &ProducerMessage{
+			Payload: []byte(fmt.Sprintf("hello-%d", i)),
+		})
+		fmt.Println(lastMsgID.String())

Review Comment:
   Don't use `fmt.Println` in tests



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Fix wrong result of hasNext after seeking by id or time [pulsar-client-go]

Posted by "shibd (via GitHub)" <gi...@apache.org>.
shibd commented on code in PR #1340:
URL: https://github.com/apache/pulsar-client-go/pull/1340#discussion_r1972995691


##########
pulsar/consumer_partition.go:
##########
@@ -2195,6 +2226,24 @@ func (pc *partitionConsumer) discardCorruptedMessage(msgID *pb.MessageIdData,
 }
 
 func (pc *partitionConsumer) hasNext() bool {
+
+	// If a seek by time has been performed, then the `startMessageId` becomes irrelevant.
+	// We need to compare `markDeletePosition` and `lastMessageId`,
+	// and then reset `startMessageID` to `markDeletePosition`.
+	if pc.hasSoughtByTime.CompareAndSwap(true, false) {

Review Comment:
   > Here it's changed from true to false, which means next time hasNext is called, it will skip the code block here.
   
   Yes, After executing `seekByTime`, we only need to enter this method once, as it updates startMessageID to markDeletePosition. Now, it clearly knows what the messageId was from the last execution of seekByTime



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Fix wrong result of reader.hasNext/Next after seeking by id or time [pulsar-client-go]

Posted by "shibd (via GitHub)" <gi...@apache.org>.
shibd commented on code in PR #1340:
URL: https://github.com/apache/pulsar-client-go/pull/1340#discussion_r1986459004


##########
pulsar/consumer_partition.go:
##########
@@ -2256,12 +2306,14 @@ func convertToMessageID(id *pb.MessageIdData) *trackingMessageID {
 
 	msgID := &trackingMessageID{
 		messageID: &messageID{
-			ledgerID: int64(*id.LedgerId),
-			entryID:  int64(*id.EntryId),
+			ledgerID:  int64(id.GetLedgerId()),
+			entryID:   int64(id.GetEntryId()),
+			batchIdx:  id.GetBatchIndex(),
+			batchSize: id.GetBatchSize(),
 		},
 	}
-	if id.BatchIndex != nil {
-		msgID.batchIdx = *id.BatchIndex
+	if msgID.batchIdx == -1 {
+		msgID.batchIdx = 0

Review Comment:
   If remove this, when seek to a messageId(no batch), let say`(3141:1:0:0)`, When we compare, there will be unexpected behavior because the `lastMessageId` we get from the broker is: `3141:1:-1:0`.
   
   What are your main concerns? Currently, this method is only used for comparison in `hasNext`.
   
   I think we need to maintain consistency because now the non-batch message ID returned by the producer and consumer is `batchIndex=0`.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Fix wrong result of reader.hasNext/Next after seeking by id or time [pulsar-client-go]

Posted by "BewareMyPower (via GitHub)" <gi...@apache.org>.
BewareMyPower commented on code in PR #1340:
URL: https://github.com/apache/pulsar-client-go/pull/1340#discussion_r1986098222


##########
pulsar/consumer_partition.go:
##########
@@ -2256,12 +2306,14 @@ func convertToMessageID(id *pb.MessageIdData) *trackingMessageID {
 
 	msgID := &trackingMessageID{
 		messageID: &messageID{
-			ledgerID: int64(*id.LedgerId),
-			entryID:  int64(*id.EntryId),
+			ledgerID:  int64(id.GetLedgerId()),
+			entryID:   int64(id.GetEntryId()),
+			batchIdx:  id.GetBatchIndex(),
+			batchSize: id.GetBatchSize(),
 		},
 	}
-	if id.BatchIndex != nil {
-		msgID.batchIdx = *id.BatchIndex
+	if msgID.batchIdx == -1 {
+		msgID.batchIdx = 0

Review Comment:
   Sounds like a concern. Could you provide a test case to show this change might not work?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Fix wrong result of hasNext after seeking by id or time [pulsar-client-go]

Posted by "shibd (via GitHub)" <gi...@apache.org>.
shibd commented on code in PR #1340:
URL: https://github.com/apache/pulsar-client-go/pull/1340#discussion_r1972993416


##########
pulsar/consumer_partition.go:
##########
@@ -997,7 +1016,15 @@ func (pc *partitionConsumer) requestSeek(msgID *messageID) error {
 	if err := pc.requestSeekWithoutClear(msgID); err != nil {
 		return err
 	}
-	pc.clearReceiverQueue()
+	// When the seek operation is successful, it indicates:
+	// 1. The broker has reset the cursor and sent a request to close the consumer on the client side.
+	//    Since this method is in the same goroutine as the reconnectToBroker,
+	//    we can safely clear the messages in the queue (at this point, it won't contain messages after the seek).
+	// 2. The startMessageID is reset to ensure accurate judgment when calling hasNext next time.
+	//    Since the messages in the queue are cleared here reconnection won't reset startMessageId.
+	pc.lastDequeuedMsg = nil

Review Comment:
   When executing seek, will clear `messageChan` first and then `pauseDispatchMessage` message until the seek operator is complete(reconnect success)
   
   Refer to this PR:
   
   https://github.com/apache/pulsar-client-go/pull/1265



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Fix wrong result of reader.hasNext/Next after seeking by id or time [pulsar-client-go]

Posted by "nodece (via GitHub)" <gi...@apache.org>.
nodece commented on code in PR #1340:
URL: https://github.com/apache/pulsar-client-go/pull/1340#discussion_r1986593581


##########
pulsar/consumer_partition.go:
##########
@@ -2256,12 +2306,14 @@ func convertToMessageID(id *pb.MessageIdData) *trackingMessageID {
 
 	msgID := &trackingMessageID{
 		messageID: &messageID{
-			ledgerID: int64(*id.LedgerId),
-			entryID:  int64(*id.EntryId),
+			ledgerID:  int64(id.GetLedgerId()),
+			entryID:   int64(id.GetEntryId()),
+			batchIdx:  id.GetBatchIndex(),
+			batchSize: id.GetBatchSize(),
 		},
 	}
-	if id.BatchIndex != nil {
-		msgID.batchIdx = *id.BatchIndex
+	if msgID.batchIdx == -1 {
+		msgID.batchIdx = 0

Review Comment:
   > What are your main concerns? Currently, this method is only used for comparison in `hasNext`.
   
   I need to consider the IDs of batch messages and non-batch messages.
   
   If `3141:1:0` is a non-batch message, the correct message ID should be `3141:1:0:-1`, not `3141:1:0:0`.
   
   The batch index is greater than -1, which means this message is from the batch messages.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Fix wrong result of hasNext after seeking by id or time [pulsar-client-go]

Posted by "shibd (via GitHub)" <gi...@apache.org>.
shibd commented on code in PR #1340:
URL: https://github.com/apache/pulsar-client-go/pull/1340#discussion_r1983243828


##########
pulsar/consumer_test.go:
##########
@@ -1262,8 +1262,9 @@ func TestConsumerSeek(t *testing.T) {
 	defer producer.Close()
 
 	consumer, err := client.Subscribe(ConsumerOptions{
-		Topic:            topicName,
-		SubscriptionName: "sub-1",
+		Topic:                   topicName,
+		SubscriptionName:        "sub-1",
+		StartMessageIDInclusive: true,

Review Comment:
   I do not agree that this is a break change; it is more like a behavior fix or bug fix.
   
   After this PR, it will keep the same behavior with the java client. Refer to(peek to id-5 and receive msg from id-6):
   https://github.com/apache/pulsar/blob/a1f3322ed358ab6841f0d3e43f2afcc54788b887/pulsar-broker/src/test/java/org/apache/pulsar/broker/service/SubscriptionSeekTest.java#L142-L165
   
   
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Fix wrong result of hasNext/Receive after seeking by id or time [pulsar-client-go]

Posted by "shibd (via GitHub)" <gi...@apache.org>.
shibd commented on PR #1340:
URL: https://github.com/apache/pulsar-client-go/pull/1340#issuecomment-2703982493

   Start discuss on thread: https://lists.apache.org/thread/knxsq637pyhltthn77859mpxb5wcxglj


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Fix wrong result of hasNext after seeking by id or time [pulsar-client-go]

Posted by "BewareMyPower (via GitHub)" <gi...@apache.org>.
BewareMyPower commented on code in PR #1340:
URL: https://github.com/apache/pulsar-client-go/pull/1340#discussion_r1972902281


##########
pulsar/consumer_partition.go:
##########
@@ -997,7 +1016,15 @@ func (pc *partitionConsumer) requestSeek(msgID *messageID) error {
 	if err := pc.requestSeekWithoutClear(msgID); err != nil {
 		return err
 	}
-	pc.clearReceiverQueue()
+	// When the seek operation is successful, it indicates:
+	// 1. The broker has reset the cursor and sent a request to close the consumer on the client side.
+	//    Since this method is in the same goroutine as the reconnectToBroker,
+	//    we can safely clear the messages in the queue (at this point, it won't contain messages after the seek).
+	// 2. The startMessageID is reset to ensure accurate judgment when calling hasNext next time.
+	//    Since the messages in the queue are cleared here reconnection won't reset startMessageId.
+	pc.lastDequeuedMsg = nil

Review Comment:
   I have a concern that if there is a race here. Before this change, `lastDequeuedMsg` is only updated by `setLastDequeuedMsg`, which is only called by the `Next` method called in user's goroutine. However, after that, `lastDequeuedMsg` could be reset with nil in the event loop goroutine (`runEventsLoop -> internalSeek -> requestSeek`)



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org