You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pulsar.apache.org by "KamenRiderKuuga (via GitHub)" <gi...@apache.org> on 2023/06/28 04:09:10 UTC
[GitHub] [pulsar] KamenRiderKuuga opened a new issue, #20667: [Bug] Unlimited append problem when subscribing with topics Pattern
KamenRiderKuuga opened a new issue, #20667:
URL: https://github.com/apache/pulsar/issues/20667
### Search before asking
- [X] I searched in the [issues](https://github.com/apache/pulsar/issues) and found nothing similar.
### Version
Due to consumers keep subscribing to their own retries and dead letter messages, Topic is infinitely appended (`allowAutoTopicCreation = true`)
### Minimal reproduce step
The code used to reproduce:
```java
public static void main(String[] args) throws IOException {
AuthenticationBasic auth = new AuthenticationBasic();
auth.configure("{\"userId\":\"superuser\",\"password\":\"admin\"}");
String serviceUrl = "pulsar://127.0.0.1:6650";
String topicPattern = "persistent://public/default/ubw.*";
PulsarClient pulsarClient = PulsarClient.builder().serviceUrl(serviceUrl).authentication(auth).build();
var consumerBuilder = pulsarClient.newConsumer(Schema.BYTES).topicsPattern(topicPattern).subscriptionName("ubw-test")
.subscriptionType(SubscriptionType.Shared)
.enableRetry(true)
.deadLetterPolicy(DeadLetterPolicy.builder()
.maxRedeliverCount(3)
.build())
.subscriptionInitialPosition(SubscriptionInitialPosition.Earliest)
.messageListener((consumer, msg) -> {
System.out.println("=======================================");
System.out.println("msg = " + new String(msg.getData()));
System.out.println("topic = " + msg.getTopicName());
System.out.println("key = " + msg.getKey());
System.out.println("properties = " + msg.getProperties());
try {
consumer.reconsumeLater(msg, 1, TimeUnit.SECONDS);
} catch (PulsarClientException e) {
throw new RuntimeException(e);
}
}).patternAutoDiscoveryPeriod(1, TimeUnit.SECONDS);
consumerBuilder.subscribe();
System.in.read();
}
```
### What did you expect to see?
This is a bit strange, I think it should automatically ack when the dead letter message is consumed?
This problem will eventually cause Pulsar to throw an exception:
Got exception org.apache.pulsar.shade.io.netty.handler.codec.TooLongFrameException: Adjusted frame length exceeds 5253120: 5263796 - discarded
### What did you see instead?
Topic is constantly appended, reaching a terrifying length
### Anything else?
_No response_
### Are you willing to submit a PR?
- [ ] I'm willing to submit a PR!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] lifepuzzlefun commented on issue #20667: [Bug] Topics appended unlimitedly when subscribing with topics pattern
Posted by "lifepuzzlefun (via GitHub)" <gi...@apache.org>.
lifepuzzlefun commented on issue #20667:
URL: https://github.com/apache/pulsar/issues/20667#issuecomment-1615563996
can you provide your topic in cluster (both autocreated topic and dlqTopic) ? and i'm not sure if this is related by deadletter topic auto create logic. can you set one dlq topic with already created one and test if this will happen ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug] Topics appended unlimitedly when subscribing with topics pattern [pulsar]
Posted by "GuillaumeCavelier (via GitHub)" <gi...@apache.org>.
GuillaumeCavelier commented on issue #20667:
URL: https://github.com/apache/pulsar/issues/20667#issuecomment-2895101420
We're having the same issue when registering a ".*" pattern.
In case of errors, it will create a first dead letter queue topic based on the initial subscription, register to this dead letter topic and then recreate a new dead letter topic for the first dead letter topic (and this indefinitely).
One solution can be to exclude the dead letter topics from the regex and use the retry topic feature instead as retry topics are supposed to be subscribed at the same time as the initial consumer (no explicit subscription is needed).
The problem is that, for topic patterns, we filter all the topics based on the regex, meaning that excluding the retry topics from the regex will make the consumer not subscribing to the retry topics (while the retry is indeed enabled).
And if we keep the retry topics in the regex, we face the same issue than with the dead letter ones.
So for now, it seems to me that the topic pattern feature is unusable with a dead letter policy. One solution can be to register retry topics even if they don't match the regex.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] github-actions[bot] commented on issue #20667: [Bug] Topics appended unlimitedly when subscribing with topics pattern
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #20667:
URL: https://github.com/apache/pulsar/issues/20667#issuecomment-1659448678
The issue had no activity for 30 days, mark with Stale label.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org