You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pulsar.apache.org by GitBox <gi...@apache.org> on 2020/06/16 05:42:18 UTC
[GitHub] [pulsar] codelipenghui opened a new issue #6554: Dispatch messages to the new consumer after exist consumer acknowledged all dispatched messages in Key_Shared Subscription.
codelipenghui opened a new issue #6554:
URL: https://github.com/apache/pulsar/issues/6554
Master issue #4077
**Is your feature request related to a problem? Please describe.**
Dispatch messages to the new consumer after exist consumer acknowledged all dispatched messages in Key_Shared Subscription.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] codelipenghui commented on issue #6554: Dispatch messages to the new consumer after exist consumer acknowledged all dispatched messages in Key_Shared Subscription.
Posted by GitBox <gi...@apache.org>.
codelipenghui commented on issue #6554:
URL: https://github.com/apache/pulsar/issues/6554#issuecomment-618198792
@merlimat I like the approach you have mentioned.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] codelipenghui closed issue #6554: Dispatch messages to the new consumer after exist consumer acknowledged all dispatched messages in Key_Shared Subscription.
Posted by GitBox <gi...@apache.org>.
codelipenghui closed issue #6554:
URL: https://github.com/apache/pulsar/issues/6554
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] codelipenghui commented on issue #6554: Dispatch messages to the new consumer after exist consumer acknowledged all dispatched messages in Key_Shared Subscription.
Posted by GitBox <gi...@apache.org>.
codelipenghui commented on issue #6554:
URL: https://github.com/apache/pulsar/issues/6554#issuecomment-653274027
@rocketraman As discussed with @feeblefakie in slack channel, looks there are some problems with auto split mechanism. So I reopen this issue for tracking this problem. And @feeblefakie has verified consistent hash mechanism is work as expected. I will take a look at the auto split mechanism later.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] feeblefakie commented on issue #6554: Dispatch messages to the new consumer after exist consumer acknowledged all dispatched messages in Key_Shared Subscription.
Posted by GitBox <gi...@apache.org>.
feeblefakie commented on issue #6554:
URL: https://github.com/apache/pulsar/issues/6554#issuecomment-653959773
@codelipenghui #7455 PTAL.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] merlimat commented on issue #6554: Dispatch messages to the new consumer after exist consumer acknowledged all dispatched messages in Key_Shared Subscription.
Posted by GitBox <gi...@apache.org>.
merlimat commented on issue #6554:
URL: https://github.com/apache/pulsar/issues/6554#issuecomment-618123128
So, I think the problem here should be treated by having new consumers joining in a "paused" state.
For example, consider this sequence:
1. Subscriptions has `c1` and `c2` consumers
2. `c3` joins. Some of the keys are now supposed to go to `c3`.
3. Instead of starting delivering to `c3`. We mark the current `readPosition` (let's call it `rp0_c3`) of the cursor for `c3`.
4. Any message that now hashes to `c3` and that has `messageId >= rp0_c3` will be deferred for later re-delivery
5. Any message that might get re-delivered (eg: originally sent to `c1`, but `c1` has failed) to `c3` and that has `messageId < rp0_c3` will be sent to `c3`
6. When the `markDelete` position of the cursor will move past `rp0_c3` the restriction on `c3` will be lifted.
Essentially, `c3` joins but can only receive old messages, until everything that was read before joining gets acked.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] codelipenghui commented on issue #6554: Dispatch messages to the new consumer after exist consumer acknowledged all dispatched messages in Key_Shared Subscription.
Posted by GitBox <gi...@apache.org>.
codelipenghui commented on issue #6554:
URL: https://github.com/apache/pulsar/issues/6554#issuecomment-644543284
Reopen it via https://github.com/apache/pulsar/issues/5819#issuecomment-644510797.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] feeblefakie edited a comment on issue #6554: Dispatch messages to the new consumer after exist consumer acknowledged all dispatched messages in Key_Shared Subscription.
Posted by GitBox <gi...@apache.org>.
feeblefakie edited a comment on issue #6554:
URL: https://github.com/apache/pulsar/issues/6554#issuecomment-653279438
@codelipenghui Should I create an open issue for [this](https://github.com/apache/pulsar/issues/5819#issuecomment-644510797) ?
I realized that I just created how to reproduce doc in my repo and shared it in only the closed issue.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] codelipenghui commented on issue #6554: Dispatch messages to the new consumer after exist consumer acknowledged all dispatched messages in Key_Shared Subscription.
Posted by GitBox <gi...@apache.org>.
codelipenghui commented on issue #6554:
URL: https://github.com/apache/pulsar/issues/6554#issuecomment-630532887
@pouledodue @feeblefakie @merlimat I have pushed a PR #6799 for this issue. Please help review, thanks.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] feeblefakie commented on issue #6554: Dispatch messages to the new consumer after exist consumer acknowledged all dispatched messages in Key_Shared Subscription.
Posted by GitBox <gi...@apache.org>.
feeblefakie commented on issue #6554:
URL: https://github.com/apache/pulsar/issues/6554#issuecomment-618158748
@merlimat
I'm not fully sure about the solution but does it guarantee key ordering ?
For example, a message X with key "A" is first consumed in `c1`, and another message Y with key "A" can possibly go to `c3` ? (can we make sure message X is consumed and acked before message Y ?)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] feeblefakie commented on issue #6554: Dispatch messages to the new consumer after exist consumer acknowledged all dispatched messages in Key_Shared Subscription.
Posted by GitBox <gi...@apache.org>.
feeblefakie commented on issue #6554:
URL: https://github.com/apache/pulsar/issues/6554#issuecomment-653279438
@codelipenghui Should I create a open issue for [this](https://github.com/apache/pulsar/issues/5819#issuecomment-644510797) ?
I realized that I just created how to reproduce doc in my repo and shared it in only the closed issue.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] wolfstudy commented on issue #6554: Dispatch messages to the new consumer after exist consumer acknowledged all dispatched messages in Key_Shared Subscription.
Posted by GitBox <gi...@apache.org>.
wolfstudy commented on issue #6554:
URL: https://github.com/apache/pulsar/issues/6554#issuecomment-674175170
The issue will be tracked in #7455 so move the issue to release/2.6.2
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] codelipenghui commented on issue #6554: Dispatch messages to the new consumer after exist consumer acknowledged all dispatched messages in Key_Shared Subscription.
Posted by GitBox <gi...@apache.org>.
codelipenghui commented on issue #6554:
URL: https://github.com/apache/pulsar/issues/6554#issuecomment-653499406
@feeblefakie Yes, thanks.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] rocketraman commented on issue #6554: Dispatch messages to the new consumer after exist consumer acknowledged all dispatched messages in Key_Shared Subscription.
Posted by GitBox <gi...@apache.org>.
rocketraman commented on issue #6554:
URL: https://github.com/apache/pulsar/issues/6554#issuecomment-653226076
PR #7106 referred to by https://github.com/apache/pulsar/issues/5819#issuecomment-644543109 is marked as closed and merged, with milestone 2.6.0. Should this issue be closed with 2.6.0 milestone after all?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] codelipenghui edited a comment on issue #6554: Dispatch messages to the new consumer after exist consumer acknowledged all dispatched messages in Key_Shared Subscription.
Posted by GitBox <gi...@apache.org>.
codelipenghui edited a comment on issue #6554:
URL: https://github.com/apache/pulsar/issues/6554#issuecomment-630532887
@pouledodue @feeblefakie @merlimat I have pushed a PR #6977 for this issue. Please help review, thanks.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] merlimat commented on issue #6554: Dispatch messages to the new consumer after exist consumer acknowledged all dispatched messages in Key_Shared Subscription.
Posted by GitBox <gi...@apache.org>.
merlimat commented on issue #6554:
URL: https://github.com/apache/pulsar/issues/6554#issuecomment-618168505
> For example, a message X with key "A" is first consumed in c1, and another message Y with key "A" can possibly go to c3 ? (can we make sure message X is consumed and acked before message Y ?)
Yes, the above proposal takes care of that.
The important point is that X could be re-sent to `c3`, in case `c1` fails just after `c3` has joined.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org