You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pulsar.apache.org by "Gilthoniel (via GitHub)" <gi...@apache.org> on 2024/07/26 06:54:44 UTC
[I] Incorrect sequence of messages due to a race between reconnect and close in producer [pulsar-client-go]
Gilthoniel opened a new issue, #1257:
URL: https://github.com/apache/pulsar-client-go/issues/1257
#### Expected behavior
Closing a producer should never produce an inconsistent sequence of messages according to Send / SendAsync order of calls.
#### Actual behavior
When a producer is starting a reconnect loop, and is requested to close at the same time, it can happen that one or more pending batches are dropped and following ones are published.
#### Steps to reproduce
I can't provide a consistent way of reproducing this because it is very random and rare as you need to get unlucky on the sequence of events.
Here is the list of logs that lead me to that discovery:
```
{"ts": "2024-07-23T20:50:42.023Z", "msg": "Closing producer", "producerID": 36}
{"ts": "2024-07-23T20:50:42.023Z", "msg": "Connected producer", "producerID": 36, "epoch": 10}
{"ts": "2024-07-23T20:50:42.038Z", "msg": "Failing 1 messages on closing producer", "producerID": 36}
```
This is demonstrating the possible race between:
```Go
func (p *partitionProducer) doClose(reason error)
```
And
```Go
func (p *partitionProducer) grabCnx(assignedBrokerURL string) error
```
#### System configuration
**Pulsar version**: v3.0.5
**Pulsar Go client**: v12.1
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Incorrect sequence of messages while reading dataChan after closing [pulsar-client-go]
Posted by "Gilthoniel (via GitHub)" <gi...@apache.org>.
Gilthoniel commented on issue #1257:
URL: https://github.com/apache/pulsar-client-go/issues/1257#issuecomment-2252780285
If I'm correct, that could be fixed by simply emptying the channel after closing it to ensure that it is not processing further.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Incorrect sequence of messages due to a race between reconnect and close in producer [pulsar-client-go]
Posted by "Gilthoniel (via GitHub)" <gi...@apache.org>.
Gilthoniel commented on issue #1257:
URL: https://github.com/apache/pulsar-client-go/issues/1257#issuecomment-2252727016
That could be fixed by introducing a `closed` channel that would be closed when `runEventsLoop` terminates then the `doClose` function can switch the state to Closing and close the channels then wait for the `closed` channel to be closed.
In that case the reconnect either succeed or fail and next iteration would return as the incoming channels are closed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Incorrect sequence of messages while reading dataChan after closing [pulsar-client-go]
Posted by "gunli (via GitHub)" <gi...@apache.org>.
gunli commented on issue #1257:
URL: https://github.com/apache/pulsar-client-go/issues/1257#issuecomment-2255699499
@Gilthoniel Good catch, could pls check if #1249 can fix this?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Incorrect sequence of messages while reading dataChan after closing [pulsar-client-go]
Posted by "Gilthoniel (via GitHub)" <gi...@apache.org>.
Gilthoniel commented on issue #1257:
URL: https://github.com/apache/pulsar-client-go/issues/1257#issuecomment-2268188109
@gunli Yes I think that's enough to avoid this situation. Please note that using a context like #1249 is not really what context is intended to and that should be handled via a channel.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Incorrect sequence of messages while reading dataChan after closing [pulsar-client-go]
Posted by "Gilthoniel (via GitHub)" <gi...@apache.org>.
Gilthoniel closed issue #1257: Incorrect sequence of messages while reading dataChan after closing
URL: https://github.com/apache/pulsar-client-go/issues/1257
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org