You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pulsar.apache.org by GitBox <gi...@apache.org> on 2021/02/14 10:28:30 UTC
[GitHub] [pulsar] alexku7 opened a new issue #9585: The consumer doesn't reconnect to the multi partitioned topic after the topic deletion and recreating
alexku7 opened a new issue #9585:
URL: https://github.com/apache/pulsar/issues/9585
**Describe the bug**
When we delete the multi partitioned topic or the entire tenant and recreate it again the existing connected consumer can't re-subscribe and consume messages. We must kill the consumer and create again.
Note: The multi topic consumer with regex works as expected. The problem is with a regular consumer only.
Tested on the multi portioned topic only .
**To Reproduce**
Steps to reproduce the behavior:
1. Create multi portioned topic.
2. Create consumer to that topic
3. Delete the topic
4. Recreate it again.
**Expected behavior**
The consumer should discover again the topic and re-subscribe like multitopic consumer with regex.
**Screenshots**
If applicable, add screenshots to help explain your problem.
**Desktop (please complete the following information):**
- OS: [e.g. iOS]
**Additional context**
Add any other context about the problem here.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] mlyahmed commented on issue #9585: The consumer doesn't reconnect to the multi partitioned topic after the topic deletion and recreating
Posted by GitBox <gi...@apache.org>.
mlyahmed commented on issue #9585:
URL: https://github.com/apache/pulsar/issues/9585#issuecomment-788676129
thanks @codelipenghui for your feedback ! Sure for the tests (y)
regarding the #9173 I will get back to you soon with some comments to make sure I understand well the issue. I will happy to work on this issue if you'd like to assist me on it. Thanks
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] alexku7 commented on issue #9585: The consumer doesn't reconnect to the multi partitioned topic after the topic deletion and recreating
Posted by GitBox <gi...@apache.org>.
alexku7 commented on issue #9585:
URL: https://github.com/apache/pulsar/issues/9585#issuecomment-786162631
> > If the consumer is multiple topic consumer, only part of the topics been removed, the correct behavior is one part of the internal consumer been closed with an exception, others should continue works(such as you subscribe to topic-a topic-b topic-c), If topic-a removed, the consume should continue to consume messages from topic-b topic-c. But if topic-b topic-b been removed, the consumer should be closed.
>
> In the case of multi-topics, actually, if you delete the whole topic, hence all its partitions, the consumer does not close. **The partitions' consumers are closed but kept in the root consumer list.** Thus my PR #9660 updates this list automatically so the closed ones are dropped and new consumers are created when the topic is created again.
@codelipenghui In my case I observed the same behavior.
I delete multi partioned topic and create the same topic again after , let's say half minute. The client remained in kind of "hang" state and doesn't reconnect again. We should restart the client completely in order to reconnect, which is not always convenient .
We tried to catch this event in the code and re- create the client again in the code without restart the executable but we simply can't do it , because can"t get any exception or other indicator that could allow us to handle this situation inside the code without restart the client
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] sijie commented on issue #9585: The consumer doesn't reconnect to the multi partitioned topic after the topic deletion and recreating
Posted by GitBox <gi...@apache.org>.
sijie commented on issue #9585:
URL: https://github.com/apache/pulsar/issues/9585#issuecomment-781529685
@codelipenghui Can you take a look at this thread?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] mlyahmed edited a comment on issue #9585: The consumer doesn't reconnect to the multi partitioned topic after the topic deletion and recreating
Posted by GitBox <gi...@apache.org>.
mlyahmed edited a comment on issue #9585:
URL: https://github.com/apache/pulsar/issues/9585#issuecomment-788676129
thanks @codelipenghui for your feedback ! Sure for the tests (y)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] mlyahmed commented on issue #9585: The consumer doesn't reconnect to the multi partitioned topic after the topic deletion and recreating
Posted by GitBox <gi...@apache.org>.
mlyahmed commented on issue #9585:
URL: https://github.com/apache/pulsar/issues/9585#issuecomment-785750056
> If the consumer is multiple topic consumer, only part of the topics been removed, the correct behavior is one part of the internal consumer been closed with an exception, others should continue works(such as you subscribe to topic-a topic-b topic-c), If topic-a removed, the consume should continue to consume messages from topic-b topic-c. But if topic-b topic-b been removed, the consumer should be closed.
In the case of multi-topics, actually, if you delete the whole topic, hence all its partitions, the consumer does not close. **The partitions' consumers are closed but kept in the root consumer list.** Thus my PR #9660 updates this list automatically so the closed ones are dropped and new consumers are created when the topic is created again.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] mlyahmed commented on issue #9585: The consumer doesn't reconnect to the multi partitioned topic after the topic deletion and recreating
Posted by GitBox <gi...@apache.org>.
mlyahmed commented on issue #9585:
URL: https://github.com/apache/pulsar/issues/9585#issuecomment-782831234
Above a hack as a suggestion. I am not sure it is the best implementation. I think we can use a kinda of observer to remove the disconnected consumers on the fly. But first we need you @codelipenghui to qualify the current behavior whether it must be revisited.
I am available for a live chat if you want to discuss a suitable design.
Thanks
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] sijie closed issue #9585: The consumer doesn't reconnect to the multi partitioned topic after the topic deletion and recreating
Posted by GitBox <gi...@apache.org>.
sijie closed issue #9585:
URL: https://github.com/apache/pulsar/issues/9585
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] alexku7 commented on issue #9585: The consumer doesn't reconnect to the multi partitioned topic after the topic deletion and recreating
Posted by GitBox <gi...@apache.org>.
alexku7 commented on issue #9585:
URL: https://github.com/apache/pulsar/issues/9585#issuecomment-779126948
So seems the problem exists in 2.7.0 as well :(
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] mlyahmed commented on issue #9585: The consumer doesn't reconnect to the multi partitioned topic after the topic deletion and recreating
Posted by GitBox <gi...@apache.org>.
mlyahmed commented on issue #9585:
URL: https://github.com/apache/pulsar/issues/9585#issuecomment-778986735
Yes I confirm the topics created by the consumer are not partitions they are regular topics. And when I tried to list the partitioned topics it returns empty.
I tested it on the both versions 2.7.0 and 2.8.0-SNAPSHOT (CLI and Server)
Regarding the Go client I used the version v0.3.0 of github.com/apache/pulsar-client-go
OS : Ubuntu 20.04.2 LTS
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] alexku7 commented on issue #9585: The consumer doesn't reconnect to the multi partitioned topic after the topic deletion and recreating
Posted by GitBox <gi...@apache.org>.
alexku7 commented on issue #9585:
URL: https://github.com/apache/pulsar/issues/9585#issuecomment-779044870
Hi
In the first scenario everything is working for me as well. There is no problem with regex multitopic subscribers.
Regarding the second scenario - We have a policy that prevents automatic topic creation. We create them manually as needed.
So in our scenario I simply create the topic again as multi portioned, but as mentioned, the consumer can't re-subscribe and can't receive messages .
Are you sure that in your second scenario the consumer actually can consume messages ? May be the issue actually don't exist in 2.7.0 :)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] mlyahmed commented on issue #9585: The consumer doesn't reconnect to the multi partitioned topic after the topic deletion and recreating
Posted by GitBox <gi...@apache.org>.
mlyahmed commented on issue #9585:
URL: https://github.com/apache/pulsar/issues/9585#issuecomment-779389165
It seems that the way subscription to a topic is managed differently than to a pattern.
When the consumer points to a topic and it is partitioned then the subscription is managed by **MultiTopicsConsumerImpl** which handles only one case where there is new partition(s) and subscribes to the new one(s) only. So if you delete the topic and recreate it with the same partitions the consumer will do nothing. And if the number of the new partitions is less it logs an error and bypasses.
On the opposite, when the consumer points to a pattern, the subscription is managed by **PatternMultiTopicsConsumerImpl** which handles both cases of add and remove. By the way, it does not consider partitions in its treatment.
I think we need a senior contributor here to tell us what is the expected behavior in this case.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] codelipenghui commented on issue #9585: The consumer doesn't reconnect to the multi partitioned topic after the topic deletion and recreating
Posted by GitBox <gi...@apache.org>.
codelipenghui commented on issue #9585:
URL: https://github.com/apache/pulsar/issues/9585#issuecomment-781738542
@alexku7 @mlyahmed The problem is after you disabled the topic auto-creation and delete the topic forcefully, the regular consumer can't reconnect after recreating this topic? But the regex consumer works?
If I understand correctly, this is expected behavior. If the topic-auto creation is disabled, the consumer will stop when occurs TopicNotFoundException. The regex consumer has different behavior, the regex consumer will fetch topics that can match the regex, so it will reconnect to the recreated topic.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] mlyahmed commented on issue #9585: The consumer doesn't reconnect to the multi partitioned topic after the topic deletion and recreating
Posted by GitBox <gi...@apache.org>.
mlyahmed commented on issue #9585:
URL: https://github.com/apache/pulsar/issues/9585#issuecomment-781881135
Actually the first need is to qualify the current behavior if is it correct or must be reviewed.
When the consumer is subscribed to a pattern it is behavior is correct since is does not recreate the topics and when topics are created or deleted it changes its subscriptions automatically (discovery)
the "odd" behavior is noticed when it is subscribed to a partitioned topic:
- When the topic is deleted forcefully with auto-creation, it recreates regular topics with the same names as the deleted partitions and not as partitioned topic. Also it does not subscribe to these topics. Is it correct doing so ? Naively, I would suggest that it must recreate a partitioned topic as it was.
- When the topic is deleted forcefully with auto-creation disabled,
- it detects that and ignores it with error message (See MultiTopicsConsumerImpl.subscribeIncreasedTopicPartitions). It treats only the case of increasing partitions as the method name suggests.
- And when the topic is recreated manually:
- with the same partitions number ==> it does nothing since it compares the number registered in the consumer with the current number of the topics partitions.
- With a bigger partition number ==> it subscribes to the new partitions but does not resubscribe to the old ones.
==> is this correct as behavior? Naively, I would suggest that the consumer must resubscribe to the new topic automatically or exits, if there is no active subscription, to allow an auto-relaunch for example.
Thanks
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] mlyahmed commented on issue #9585: The consumer doesn't reconnect to the multi partitioned topic after the topic deletion and recreating
Posted by GitBox <gi...@apache.org>.
mlyahmed commented on issue #9585:
URL: https://github.com/apache/pulsar/issues/9585#issuecomment-778834699
Hi,
Could you please share more about the way you’ve proceeded? and the version you’ve used ?
Actually I created a partitioned topic through the cli let’s say :
`pulsar-admin topics create-partitioned-topic persistent://public/default/issue-9585 -p 3`
Created a consumer with the minimum options: topic + subscriptionName:
`pulsar-client consume -s ahmed -n 0 persistent://public/default/issue-9585`
When I deleted the topic without the force flag, -f, I got an error explaining that there is a current active procuder/subscription.
When I delete the topic with the force flag, -f, the topic is deleted. The consumer resubscribes to new topics with the same names as the previous partitions:
```
19:55:43.580 [pulsar-client-io-1-1] INFO org.apache.pulsar.client.impl.ConsumerImpl - [persistent://public/default/issue-9585-partition-1][ahmed] Subscribed to topic on localhost/127.0.0.1:6650 -- consumer: 1
19:55:43.589 [pulsar-client-io-1-1] INFO org.apache.pulsar.client.impl.ConsumerImpl - [persistent://public/default/issue-9585-partition-0][ahmed] Subscribed to topic on localhost/127.0.0.1:6650 -- consumer: 0
19:55:43.595 [pulsar-client-io-1-1] INFO org.apache.pulsar.client.impl.ConsumerImpl - [persistent://public/default/issue-9585-partition-2][ahmed] Subscribed to topic on localhost/127.0.0.1:6650 -- consumer: 2
```
And when I try to recreate the partitioned topic this error is thrown due the topics created by the consumer:
```
9:55:48.317 [AsyncHttpClient-7-1] WARN org.apache.pulsar.client.admin.internal.BaseResource - [http://localhost:8080/admin/v2/persistent/public/default/issue-9585/partitions] Failed to perform http put request: javax.ws.rs.ClientErrorException: HTTP 409 Conflict
This topic already exists
Reason: This topic already exists
```
I’ve also tested with the Go client and ended up with the same issue. And also, the Go consumer fall in error due to the number of partitions changed:
```
goroutine 85 [running]:
github.com/apache/pulsar-client-go/pulsar.(*consumer).internalTopicSubscribeToPartitions(0xc000282300, 0x0, 0x0)
/home/ahmed/tmp/pulsar/go-client/vendor/github.com/apache/pulsar-client-go/pulsar/consumer_impl.go:279 +0x890
github.com/apache/pulsar-client-go/pulsar.newInternalConsumer.func1(0xc000282300)
/home/ahmed/tmp/pulsar/go-client/vendor/github.com/apache/pulsar-client-go/pulsar/consumer_impl.go:238 +0xc8
created by github.com/apache/pulsar-client-go/pulsar.newInternalConsumer
/home/ahmed/tmp/pulsar/go-client/vendor/github.com/apache/pulsar-client-go/pulsar/consumer_impl.go:235 +0x2bc
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] mlyahmed commented on issue #9585: The consumer doesn't reconnect to the multi partitioned topic after the topic deletion and recreating
Posted by GitBox <gi...@apache.org>.
mlyahmed commented on issue #9585:
URL: https://github.com/apache/pulsar/issues/9585#issuecomment-779091249
I just retested with the topic auto-creation disabled and yes when recreate manually the partitioned topic and publish messages into it the consumer can't receive the messages.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] codelipenghui commented on issue #9585: The consumer doesn't reconnect to the multi partitioned topic after the topic deletion and recreating
Posted by GitBox <gi...@apache.org>.
codelipenghui commented on issue #9585:
URL: https://github.com/apache/pulsar/issues/9585#issuecomment-785668280
Hi @alexku7 If the topic auto-creation is disabled and we delete the existing topic, the clients should be rejected when they try to connect to a not exist topic. If the consumer is a single topic consumer and the topic been removed, the consumer should be close with an exception(Topic does not found), If the consumer is multiple topic consumer, only part of the topics been removed, the correct behavior is one part of the internal consumer been closed with an exception, others should continue works(such as you subscribe to topic-a topic-b topic-c), If topic-a removed, the consume should continue to consume messages from topic-b topic-c. But if topic-b topic-b been removed, the consumer should be closed.
> I think the only one thing is needed - some retry or rediscovery logic similar to the regex consumer with a pattern
It's different with the regex consumer here. If we allow retrying for a not existing topic, the consumer will try to reach the metadata service through the broker to check if the topic exists. This will introduce more work load on the metadata service.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] codelipenghui commented on issue #9585: The consumer doesn't reconnect to the multi partitioned topic after the topic deletion and recreating
Posted by GitBox <gi...@apache.org>.
codelipenghui commented on issue #9585:
URL: https://github.com/apache/pulsar/issues/9585#issuecomment-788529347
@alexku7 You can register a `ConsumerEventListener` to the consumer by `consumerBuilder.consumerEventListener()`, if a partition becomes inactive, you will receive a notification
@mlyahmed Thanks for the clarification, the change looks good, could you please help add some tests for it?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] mlyahmed commented on issue #9585: The consumer doesn't reconnect to the multi partitioned topic after the topic deletion and recreating
Posted by GitBox <gi...@apache.org>.
mlyahmed commented on issue #9585:
URL: https://github.com/apache/pulsar/issues/9585#issuecomment-782584241
I just test and yes, with the `allowAutoTopicCreationType` set to partitioned the consumer recreates the topic and its partitions and resubscribe to them.
But if `allowAutoTopicCreationType` is set to non-partitioned, the default, it creates the partitions as regular topics and does not subscribe to them.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] mlyahmed commented on issue #9585: The consumer doesn't reconnect to the multi partitioned topic after the topic deletion and recreating
Posted by GitBox <gi...@apache.org>.
mlyahmed commented on issue #9585:
URL: https://github.com/apache/pulsar/issues/9585#issuecomment-779033278
I just retested with the java client version 2.7.0
**Scenario: Subscription to pattern**
Given a partitioned topic
And a java consumer subscribed to a **pattern** matches that topic
When delete that topic **forcefully**
Then **No topic is recreated automatically**
And When recreate the partitioned topic manually
And publish new messages
Then the consumer receives the new messages
**Scenario: Subscription to the topic directly**
Given a partitioned topic
And a java consumer subscribed **directly** to that topic.
When delete that topic **forcefully**
Then **the topic partitions are recreated automatically as regular topic**
And **the partitioned topic is not created**.
And **error when try to recreate the partitioned topic**
Hence, what is the expected behavior for this last scenario ? The consumer must not recreate these topics and discover them when they are recreated as in the first scenario ?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] codelipenghui commented on issue #9585: The consumer doesn't reconnect to the multi partitioned topic after the topic deletion and recreating
Posted by GitBox <gi...@apache.org>.
codelipenghui commented on issue #9585:
URL: https://github.com/apache/pulsar/issues/9585#issuecomment-785670214
@mlyahmed I noticed your PR, seems related to a previous issue https://github.com/apache/pulsar/issues/9173. Currently, if disabled the topic auto-creation, it will avoid creating partitions of a partitioned topic even if the partitioned topic exist.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] alexku7 commented on issue #9585: The consumer doesn't reconnect to the multi partitioned topic after the topic deletion and recreating
Posted by GitBox <gi...@apache.org>.
alexku7 commented on issue #9585:
URL: https://github.com/apache/pulsar/issues/9585#issuecomment-782714744
Hi @codelipenghui
The problem that if we don't allow the auto topic creation at all ( the requirement in most secured environments) , we encounter with that issue.
Let's say that some legitimate topic was deleted for any reason, let's say by mistake. In this case, the consumer simple is being disconnected and "stuck". It doesn't have any retry or re-discovery logic like a regex consumer with pattern.
Moreover the client doesn't have any event or error in this case. So the only one solution is to determine the all clients and ask them to restart. That the problem :(
I think the only one thing is needed - some retry or rediscovery logic similar to the regex consumer with a pattern
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] alexku7 commented on issue #9585: The consumer doesn't reconnect to the multi partitioned topic after the topic deletion and recreating
Posted by GitBox <gi...@apache.org>.
alexku7 commented on issue #9585:
URL: https://github.com/apache/pulsar/issues/9585#issuecomment-778839073
Very Interesting .
I performed very similar tests with the same tools.
I used actually the 2.6.2 version (Client and server)
And the difference that I have auto topic create flag = false. So i must explicitly create the topic.
But it still interesting, because in your case the new auto created topic should be non-partitioned topic. I performed right now the same test. The pulsar-client consume , indeed ended up with the error that the partition decrease is not supported.
But I still got the same error with or without the topic auto creation flag.
The only regex mutli topic subscriber "survives" and re-subscribes to the recreated topic.
What is the version you used for the test?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] codelipenghui commented on issue #9585: The consumer doesn't reconnect to the multi partitioned topic after the topic deletion and recreating
Posted by GitBox <gi...@apache.org>.
codelipenghui commented on issue #9585:
URL: https://github.com/apache/pulsar/issues/9585#issuecomment-782479524
> When the topic is deleted forcefully with auto-creation, it recreates regular topics with the same names as the deleted partitions and not as partitioned topic. Also it does not subscribe to these topics. Is it correct doing so ? Naively, I would suggest that it must recreate a partitioned topic as it was.
Yes, the current behavior is recreating "partitions" since the consumer already subscribed to each partition. For the partitioned topic, the consumer fetches the partitioned metadata first then subscribes to each partition. Do you mean the partitions are created but the consumer does not subscribe to them? This is not expected behavior. If you want to recreate a partitioned topic you can change the topic auto-create mode to 'partitioned' in the broker.conf `allowAutoTopicCreationType`.
> it detects that and ignores it with error message (See MultiTopicsConsumerImpl.subscribeIncreasedTopicPartitions). It treats only the case of increasing partitions as the method name suggests.
This handles the partitions scaling. This will not affect the reconnect behavior. Essentially, the re-subscribe is processed in each consumer(under multi-topic consumer).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org