You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pulsar.apache.org by GitBox <gi...@apache.org> on 2022/03/04 09:47:02 UTC
[GitHub] [pulsar] wenbingshen opened a new issue #14559: Ignoring flow control message when resetCursorInProgress=TRUE
wenbingshen opened a new issue #14559:
URL: https://github.com/apache/pulsar/issues/14559
**Describe the bug**
1. Our broker is highly busy
2. Our business uses `Reader` for consumption, in which `seek` operations are often performed.
3. In the `seek` operation of pulsar, there will be a process of actively disconnecting the consumer and canceling the currently reading operation.
<img width="1437" alt="image" src="https://user-images.githubusercontent.com/35599757/156738988-ee032278-7c84-4346-9126-3ef43f665226.png">
4. Next, whether the `seek` operation is completed or not, the client has the right to subscribe the subscription.
<img width="1439" alt="image" src="https://user-images.githubusercontent.com/35599757/156739047-05bea0f0-7d06-436c-ab3a-f593aa339f94.png">
5. Because the server thread pool is busy, the `seek` operation is delayed. The client first subscribed a subscription and initiates a `FLOW` request, causing the broker to start reading data.
<img width="1439" alt="image" src="https://user-images.githubusercontent.com/35599757/156739353-10d9701f-3df4-4573-aa18-e4f4349b874c.png">
6. Next, the seek operation is completed, and the positions of `md position` and `rd position` are reset to the seek position.
<img width="1439" alt="image" src="https://user-images.githubusercontent.com/35599757/156739594-e36b57d2-1f6d-4bb2-b292-b035f594d293.png">
7. Then, the read operation is completed, after the data is sent to the client, the client executes ack, and md and rd are set to a certain position of the message read by the client.
<img width="1439" alt="image" src="https://user-images.githubusercontent.com/35599757/156739816-f7bad25a-ad94-40d5-83b5-cc4b5ff35cac.png">
8. Next, the seek operation is completed, and the positions of `md position` and `rd position` are reset to the seek position.
<img width="1439" alt="image" src="https://user-images.githubusercontent.com/35599757/156740044-fbb730cf-1070-4431-9add-fd9ec0efe5ec.png">
9. Then, the read operation is completed, after the data is sent to the client, the client executes ack, and md and rd are set to a certain position of the message read by the client.
<img width="1438" alt="image" src="https://user-images.githubusercontent.com/35599757/156739952-aa9dbf29-0a2e-487a-b5af-189d029cd0bf.png">
The entire execution flow is shown in the following figure:
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] wenbingshen commented on issue #14559: Ignoring flow control message when resetCursorInProgress=TRUE
Posted by GitBox <gi...@apache.org>.
wenbingshen commented on issue #14559:
URL: https://github.com/apache/pulsar/issues/14559#issuecomment-1061383201
> Looks like a similar problem that https://github.com/apache/pulsar/wiki/PIP-84-:-Pulsar-client:-Redeliver-command-add-epoch want to resolve.
Thanks for your reply. It looks a bit similar, but the two are slightly different. PIP-84 seems to only emphasize the synchronization consistency of data retransmission and data pull, but has no details about the seek operation. If you mean adding an epoch to the seek operation, assuming that the epoch of the seek operation is `1`, the broker will disconnect the client when the seek operation starts, and the epoch will be `2` when the client resubscribe the subscription again. The `FLOW` request at this time will cause the broker to start reading data and return it to the client. The client seems to have no way to filter out the data with epoch `2`, and then the broker completes the seek operation with epoch `1`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] wenbingshen commented on issue #14559: Ignoring flow control message when resetCursorInProgress=TRUE
Posted by GitBox <gi...@apache.org>.
wenbingshen commented on issue #14559:
URL: https://github.com/apache/pulsar/issues/14559#issuecomment-1060188329
> It will be better for others to understand this if you can emphasize which step is not expected and what final result you got.
Thanks for your replay. My expected result is: when my application finishes the seek operation, the first message read next must be the message where I seek.
But now unfortunately, in my cluster, before the seek operation is completed, the broker has started to read the data starting from the current position from the bookie, and after my seek operation is completed, this part of the data is returned to the client , but the client does not want to know whether the position of the data it reads matches where it seeks, because the client has only two steps: 1. Execute seek, 2. Read the data immediately after the seek is completed.
As shown in the figure below: The most important thing is to ensure the order of `5` and `6`. It should be executed first for `5`. The existing logic cannot guarantee this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] codelipenghui commented on issue #14559: Ignoring flow control message when resetCursorInProgress=TRUE
Posted by GitBox <gi...@apache.org>.
codelipenghui commented on issue #14559:
URL: https://github.com/apache/pulsar/issues/14559#issuecomment-1060785152
Looks like a similar problem that https://github.com/apache/pulsar/wiki/PIP-84-:-Pulsar-client:-Redeliver-command-add-epoch want to resolve.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] Jason918 commented on issue #14559: Ignoring flow control message when resetCursorInProgress=TRUE
Posted by GitBox <gi...@apache.org>.
Jason918 commented on issue #14559:
URL: https://github.com/apache/pulsar/issues/14559#issuecomment-1059887766
It will be better for others to understand this if you can emphasize which step is not expected and what final result you got.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org