You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pulsar.apache.org by GitBox <gi...@apache.org> on 2021/05/09 19:40:47 UTC
[GitHub] [pulsar] mrkingfoxx opened a new issue #10524: > I tested `pulsar 2.7.1 docker` on my local machine. Also java client version is `2.7.1`
mrkingfoxx opened a new issue #10524:
URL: https://github.com/apache/pulsar/issues/10524
> I tested `pulsar 2.7.1 docker` on my local machine. Also java client version is `2.7.1`
> `1` producer with `4` consumer in a single app container configured below.
> The producer produced messages orderly with message key ranged from `test-0` to `test-9`
>
> ```java
> private static final AtomicLong KEY_GENERATOR = new AtomicLong();
>
> private static String randomKey() {
> var idx = KEY_GENERATOR.getAndIncrement() % 10;
> return "test-" + idx;
> }
>
> @Bean
> public Producer<String> producer(PulsarClient client) throws PulsarClientException {
> return client.newProducer(Schema.STRING)
> .topic("topic-test")
> .sendTimeout(10, TimeUnit.SECONDS)
> .create();
> }
>
> @Bean
> public List<Consumer<String>> consumer(PulsarClient client) throws PulsarClientException {
> var consumers = new ArrayList<Consumer<String>>(4);
> for (var i = 0; i < 4; i++) {
> var consumer = client.newConsumer(Schema.STRING)
> .topic("topic-test")
> .subscriptionName("sub-test")
> .subscriptionType(SubscriptionType.Key_Shared)
> .subscribe();
> consumers.add(consumer);
> }
> return consumers;
> }
> ```
>
> When I first run only one app container. The consumer statistics output seems not evenly distributed.
>
> ```
> {
> "consumer-thread-2":[
> "test-2",
> "test-1",
> "test-0",
> "test-6",
> "test-4"
> ],
> "consumer-thread-3":[
> "test-9"
> ],
> "consumer-thread-0":[
> "test-3"
> ],
> "consumer-thread-1":[
> "test-8",
> "test-7",
> "test-5"
> ]
> }
> ```
>
> Then 30 seconds later, I started a new app container. Now there are two containers running at the same time.
>
> The previous contiainer consumer statistics output changed!
>
> ```
> {
> "consumer-thread-2":[
> "test-4"
> ],
> "consumer-thread-3":[
> "test-9"
> ],
> "consumer-thread-0":[
> "test-3"
> ],
> "consumer-thread-1":[
> "test-8",
> "test-7"
> ]
> }
> ```
>
> And the newer container consumer statistics output like this.
>
> ```
> {
> "consumer-thread-0":[
> "test-2",
> "test-1",
> "test-0",
> "test-6"
> ],
> "consumer-thread-1":[
> "test-5"
> ]
> }
> ```
>
> **Part of consumer in the newer container is even not distributed to consume any message.**
>
> Is there some best practice advices for Key_Shared subscription? I think this is a comman situation in production environment. The app containers may be deployed dynamically and caused subscription change.
_Originally posted by @mrkingfoxx in https://github.com/apache/pulsar/issues/10523#issuecomment-835871561_
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] codelipenghui commented on issue #10524: Messages aren't evenly distributed to multiple Key_Shared consumers
Posted by GitBox <gi...@apache.org>.
codelipenghui commented on issue #10524:
URL: https://github.com/apache/pulsar/issues/10524#issuecomment-1058890547
The issue had no activity for 30 days, mark with Stale label.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] mrkingfoxx commented on issue #10524: Messages aren't evenly distributed to multiple Key_Shared consumers
Posted by GitBox <gi...@apache.org>.
mrkingfoxx commented on issue #10524:
URL: https://github.com/apache/pulsar/issues/10524#issuecomment-843665992
I dropped the time drastically.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] yangou commented on issue #10524: > I tested `pulsar 2.7.1 docker` on my local machine. Also java client version is `2.7.1`
Posted by GitBox <gi...@apache.org>.
yangou commented on issue #10524:
URL: https://github.com/apache/pulsar/issues/10524#issuecomment-840077113
Have you tune the receiver queue size. When consumer polls the messages from pulsar, if the requested messages is greater than total number of messages, pulsar won't be able to evenly distribute the messages.
Pulsar uses poll model, not push model. If you want consumers to be evenly distributed, try setting the receiver queue size to 1.
https://pulsar.apache.org/docs/en/cookbooks-message-queue/#client-configuration-changes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] lhotari commented on issue #10524: > I tested `pulsar 2.7.1 docker` on my local machine. Also java client version is `2.7.1`
Posted by GitBox <gi...@apache.org>.
lhotari commented on issue #10524:
URL: https://github.com/apache/pulsar/issues/10524#issuecomment-841620558
Can you test if setting `.enableBatching(false)` on the producer makes a difference?
If so, the problem might be fixed by #7266 .
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] yangou edited a comment on issue #10524: > I tested `pulsar 2.7.1 docker` on my local machine. Also java client version is `2.7.1`
Posted by GitBox <gi...@apache.org>.
yangou edited a comment on issue #10524:
URL: https://github.com/apache/pulsar/issues/10524#issuecomment-840077113
Have you tune the receiver queue size. When consumer polls the messages from pulsar, if the requested messages is greater than total number of messages, pulsar won't be able to evenly distribute the messages.
Pulsar uses poll model, not push model. If you want consumers to be evenly distributed, try setting the receiver queue size to 1.
https://pulsar.apache.org/docs/en/cookbooks-message-queue/#client-configuration-changes
```If you'd like to have tight control over message dispatching across consumers, set the consumers' receiver queue size very low (potentially even to 0 if necessary). Each Pulsar consumer has a receiver queue that determines how many messages the consumer will attempt to fetch at a time. A receiver queue of 1000 (the default), for example, means that the consumer will attempt to process 1000 messages from the topic's backlog upon connection. Setting the receiver queue to zero essentially means ensuring that each consumer is only doing one thing at a time.```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] lhotari edited a comment on issue #10524: > I tested `pulsar 2.7.1 docker` on my local machine. Also java client version is `2.7.1`
Posted by GitBox <gi...@apache.org>.
lhotari edited a comment on issue #10524:
URL: https://github.com/apache/pulsar/issues/10524#issuecomment-841620558
Besides adjusting the receiver queue size, can you test if setting `.enableBatching(false)` on the producer makes a difference?
If so, the problem might be fixed by #7266 .
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org