You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pulsar.apache.org by "thetumbled (via GitHub)" <gi...@apache.org> on 2023/02/07 08:58:48 UTC
[GitHub] [pulsar] thetumbled opened a new issue, #19449: [Bug] [Broker] restart broker and transactional consumers report errors continuously.
thetumbled opened a new issue, #19449:
URL: https://github.com/apache/pulsar/issues/19449
### Search before asking
- [X] I searched in the [issues](https://github.com/apache/pulsar/issues) and found nothing similar.
### Version
master branch, latest version.
### Minimal reproduce step
1. start transactional consumer.
```
nohup bin/pulsar-perf consume -r 2048000 -txn -nmt 500 persistent://test/tb5/testTxn10 > tmp.txt
```
2. restart broker.
### What did you expect to see?
transactional consumer will consumer messages without any errors after waiting for some time.
### What did you see instead?
transactional consumer will report many errors like below continuously.
```
2023-02-07T16:31:04,563+0800 [pulsar-client-io-1-1] ERROR org.apache.pulsar.testclient.PerformanceConsumer - Ack message org.apache.pulsar.client.impl.TopicMessageImpl@2932a084 failed with exception
java.util.concurrent.CompletionException: org.apache.pulsar.client.api.PulsarClientException$TransactionConflictException: {"errorMsg":"org.apache.pulsar.transaction.common.exception.TransactionConflictException: [persistent://test/tb5/testTxn10-partition-8][sub] Transaction:(7,465731) try to ack message:369901:110055:{1, 2} already acked before.","reqId":2312522801167108396, "remote":"cluster2-nn0.bigo.baina/172.24.25.41:6650", "local":"/172.24.25.42:47086"}
at java.util.concurrent.CompletableFuture.encodeRelay(CompletableFuture.java:326) ~[?:1.8.0_92]
at java.util.concurrent.CompletableFuture.completeRelay(CompletableFuture.java:338) ~[?:1.8.0_92]
at java.util.concurrent.CompletableFuture.uniRelay(CompletableFuture.java:911) ~[?:1.8.0_92]
at java.util.concurrent.CompletableFuture$UniRelay.tryFire(CompletableFuture.java:899) ~[?:1.8.0_92]
at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) [?:1.8.0_92]
at java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:1977) [?:1.8.0_92]
at org.apache.pulsar.client.impl.ClientCnx.handleAckResponse(ClientCnx.java:434) [org.apache.pulsar-pulsar-client-original-2.9.3.jar:2.9.3]
at org.apache.pulsar.common.protocol.PulsarDecoder.channelRead(PulsarDecoder.java:150) [org.apache.pulsar-pulsar-common-2.9.3.jar:2.9.3]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [io.netty-netty-transport-4.1.77.Final.jar:4.1.77.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [io.netty-netty-transport-4.1.77.Final.jar:4.1.77.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [io.netty-netty-transport-4.1.77.Final.jar:4.1.77.Final]
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:327) [io.netty-netty-codec-4.1.77.Final.jar:4.1.77.Final]
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:299) [io.netty-netty-codec-4.1.77.Final.jar:4.1.77.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [io.netty-netty-transport-4.1.77.Final.jar:4.1.77.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [io.netty-netty-transport-4.1.77.Final.jar:4.1.77.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [io.netty-netty-transport-4.1.77.Final.jar:4.1.77.Final]
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) [io.netty-netty-transport-4.1.77.Final.jar:4.1.77.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [io.netty-netty-transport-4.1.77.Final.jar:4.1.77.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [io.netty-netty-transport-4.1.77.Final.jar:4.1.77.Final]
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) [io.netty-netty-transport-4.1.77.Final.jar:4.1.77.Final]
at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:800) [io.netty-netty-transport-classes-epoll-4.1.77.Final.jar:4.1.77.Final
at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:487) [io.netty-netty-transport-classes-epoll-4.1.77.Final.jar:4.1.77.Final]
at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:385) [io.netty-netty-transport-classes-epoll-4.1.77.Final.jar:4.1.77.Final]
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:995) [io.netty-netty-common-4.1.77.Final.jar:4.1.77.Final]
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [io.netty-netty-common-4.1.77.Final.jar:4.1.77.Final]
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) [io.netty-netty-common-4.1.77.Final.jar:4.1.77.Final]
at java.lang.Thread.run(Thread.java:745) [?:1.8.0_92]
Caused by: org.apache.pulsar.client.api.PulsarClientException$TransactionConflictException: {"errorMsg":"org.apache.pulsar.transaction.common.exception.TransactionConflictException: [persistent://test/tb5/testTxn10-partition-8][sub] Transaction:(7,465731) try to ack message:369901:110055:{1, 2} already acked before.","reqId":2312522801167108396, "remote":"cluster2-nn0.bigo.baina/172.24.25.41:6650", "local":"/172.24.25.42:47086"}
at org.apache.pulsar.client.impl.ClientCnx.getPulsarClientException(ClientCnx.java:1147) ~[org.apache.pulsar-pulsar-client-original-2.9.3.jar:2.9.3]
at org.apache.pulsar.client.impl.ClientCnx.handleAckResponse(ClientCnx.java:435) ~[org.apache.pulsar-pulsar-client-original-2.9.3.jar:2.9.3]
... 20 more
```
and there are many transaction that cannot be commit successfully, the number of transactions aborted will increased.
<img width="418" alt="image" src="https://user-images.githubusercontent.com/52550727/217198553-08d32c81-e652-43e8-a56e-cccb609ffbd6.png">
<img width="408" alt="image" src="https://user-images.githubusercontent.com/52550727/217198610-b9479277-ce60-468f-91d4-07a486d23a11.png">
### Anything else?
_No response_
### Are you willing to submit a PR?
- [ ] I'm willing to submit a PR!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] aloyszhang commented on issue #19449: [Bug] [Broker] restart broker and transactional consumers report errors continuously.
Posted by "aloyszhang (via GitHub)" <gi...@apache.org>.
aloyszhang commented on issue #19449:
URL: https://github.com/apache/pulsar/issues/19449#issuecomment-1443072507
It seems the consumer received duplicated messages. #14327 and #19581 have fixed cases that may cause duplicated message delivery.
You can pick this two prs and test again.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug] [Broker] restart broker and transactional consumers report errors continuously. [pulsar]
Posted by "dao-jun (via GitHub)" <gi...@apache.org>.
dao-jun closed issue #19449: [Bug] [Broker] restart broker and transactional consumers report errors continuously.
URL: https://github.com/apache/pulsar/issues/19449
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] github-actions[bot] commented on issue #19449: [Bug] [Broker] restart broker and transactional consumers report errors continuously.
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #19449:
URL: https://github.com/apache/pulsar/issues/19449#issuecomment-1484356288
The issue had no activity for 30 days, mark with Stale label.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org