You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pulsar.apache.org by "codelipenghui (via GitHub)" <gi...@apache.org> on 2023/06/28 04:29:40 UTC

[GitHub] [pulsar] codelipenghui opened a new pull request, #20669: [fix][client]Fix deadlock issue of consumer while using multiple IO threads

codelipenghui opened a new pull request, #20669:
URL: https://github.com/apache/pulsar/pull/20669

   ### Motivation
   
   Fix a deadlock of the consumer when the consumer tries to subscribe to partitioned topics and non-partitioned topics if the Pulsar Client uses multiple IO threads (default is 1 thread).
   
   ```
   Found one Java-level deadlock:
   =============================
   "pulsar-client-io-1-2":
     waiting to lock monitor 0x00007f716c00ae80 (object 0x00000004312d0940, a java.util.concurrent.ConcurrentHashMap$Node),
     which is held by "pulsar-client-io-1-3"
   "pulsar-client-io-1-3":
     waiting to lock monitor 0x00007f6f8c009900 (object 0x000000042cf2ed28, a java.lang.Object),
     which is held by "pulsar-client-io-1-4"
   "pulsar-client-io-1-4":
     waiting to lock monitor 0x00007f716c00ae80 (object 0x00000004312d0940, a java.util.concurrent.ConcurrentHashMap$Node),
     which is held by "pulsar-client-io-1-3"
   
   Java stack information for the threads listed above:
   ===================================================
   "pulsar-client-io-1-2":
   	at java.util.concurrent.ConcurrentHashMap.compute(java.base@11.0.18/ConcurrentHashMap.java:1923)
   	- waiting to lock <0x00000004312d0940> (a java.util.concurrent.ConcurrentHashMap$Node)
   	at org.apache.pulsar.client.impl.MultiTopicsConsumerImpl.doSubscribeTopicPartitions(MultiTopicsConsumerImpl.java:1051)
   	at org.apache.pulsar.client.impl.MultiTopicsConsumerImpl.lambda$subscribeTopicPartitions$43(MultiTopicsConsumerImpl.java:993)
   	at org.apache.pulsar.client.impl.MultiTopicsConsumerImpl$$Lambda$4606/0x0000000841779840.accept(Unknown Source)
   	at java.util.concurrent.CompletableFuture.uniWhenComplete(java.base@11.0.18/CompletableFuture.java:859)
   	at java.util.concurrent.CompletableFuture.uniWhenCompleteStage(java.base@11.0.18/CompletableFuture.java:883)
   	at java.util.concurrent.CompletableFuture.whenComplete(java.base@11.0.18/CompletableFuture.java:2251)
   	at org.apache.pulsar.client.impl.MultiTopicsConsumerImpl.subscribeTopicPartitions(MultiTopicsConsumerImpl.java:991)
   	at org.apache.pulsar.client.impl.MultiTopicsConsumerImpl.lambda$subscribeAsync$38(MultiTopicsConsumerImpl.java:917)
   	at org.apache.pulsar.client.impl.MultiTopicsConsumerImpl$$Lambda$4600/0x0000000841777040.accept(Unknown Source)
   	at java.util.concurrent.CompletableFuture$UniAccept.tryFire(java.base@11.0.18/CompletableFuture.java:714)
   	at java.util.concurrent.CompletableFuture.postComplete(java.base@11.0.18/CompletableFuture.java:506)
   	at java.util.concurrent.CompletableFuture.complete(java.base@11.0.18/CompletableFuture.java:2073)
   	at org.apache.pulsar.client.impl.PulsarClientImpl$$Lambda$4598/0x0000000841775440.accept(Unknown Source)
   	at java.util.concurrent.CompletableFuture$UniAccept.tryFire(java.base@11.0.18/CompletableFuture.java:714)
   	at java.util.concurrent.CompletableFuture.postComplete(java.base@11.0.18/CompletableFuture.java:506)
   	at java.util.concurrent.CompletableFuture.complete(java.base@11.0.18/CompletableFuture.java:2073)
   	at org.apache.pulsar.client.impl.BinaryProtoLookupService.lambda$getPartitionedTopicMetadata$4(BinaryProtoLookupService.java:199)
   	at org.apache.pulsar.client.impl.BinaryProtoLookupService$$Lambda$4605/0x0000000841779040.accept(Unknown Source)
   	at java.util.concurrent.CompletableFuture.uniWhenComplete(java.base@11.0.18/CompletableFuture.java:859)
   	at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(java.base@11.0.18/CompletableFuture.java:837)
   	at java.util.concurrent.CompletableFuture.postComplete(java.base@11.0.18/CompletableFuture.java:506)
   	at java.util.concurrent.CompletableFuture.complete(java.base@11.0.18/CompletableFuture.java:2073)
   	at org.apache.pulsar.client.impl.ClientCnx.handlePartitionResponse(ClientCnx.java:618)
   	at org.apache.pulsar.common.protocol.PulsarDecoder.channelRead(PulsarDecoder.java:130)
   	at org.apache.pulsar.shade.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
   	at org.apache.pulsar.shade.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
   	at org.apache.pulsar.shade.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
   	at org.apache.pulsar.shade.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:327)
   	at org.apache.pulsar.shade.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:314)
   	at org.apache.pulsar.shade.io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:435)
   	at org.apache.pulsar.shade.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:279)
   	at org.apache.pulsar.shade.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
   	at org.apache.pulsar.shade.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
   	at org.apache.pulsar.shade.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
   	at org.apache.pulsar.shade.io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1372)
   	at org.apache.pulsar.shade.io.netty.handler.ssl.SslHandler.decodeNonJdkCompatible(SslHandler.java:1246)
   	at org.apache.pulsar.shade.io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1286)
   	at org.apache.pulsar.shade.io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:510)
   	at org.apache.pulsar.shade.io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:449)
   	at org.apache.pulsar.shade.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:279)
   	at org.apache.pulsar.shade.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
   	at org.apache.pulsar.shade.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
   	at org.apache.pulsar.shade.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
   	at org.apache.pulsar.shade.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
   	at org.apache.pulsar.shade.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
   	at org.apache.pulsar.shade.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
   	at org.apache.pulsar.shade.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
   	at org.apache.pulsar.shade.io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:800)
   	at org.apache.pulsar.shade.io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:487)
   	at org.apache.pulsar.shade.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:385)
   	at org.apache.pulsar.shade.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:995)
   	at org.apache.pulsar.shade.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
   	at org.apache.pulsar.shade.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
   	at java.lang.Thread.run(java.base@11.0.18/Thread.java:829)
   "pulsar-client-io-1-3":
   	at org.apache.pulsar.client.impl.MultiTopicsConsumerImpl.lambda$doSubscribeTopicPartitions$45(MultiTopicsConsumerImpl.java:1062)
   	- waiting to lock <0x000000042cf2ed28> (a java.lang.Object)
   	at org.apache.pulsar.client.impl.MultiTopicsConsumerImpl$$Lambda$4607/0x000000084177a840.apply(Unknown Source)
   	at java.util.concurrent.ConcurrentHashMap.compute(java.base@11.0.18/ConcurrentHashMap.java:1947)
   	- locked <0x00000004312d0940> (a java.util.concurrent.ConcurrentHashMap$Node)
   	at org.apache.pulsar.client.impl.MultiTopicsConsumerImpl.doSubscribeTopicPartitions(MultiTopicsConsumerImpl.java:1051)
   	at org.apache.pulsar.client.impl.MultiTopicsConsumerImpl.lambda$subscribeTopicPartitions$43(MultiTopicsConsumerImpl.java:993)
   	at org.apache.pulsar.client.impl.MultiTopicsConsumerImpl$$Lambda$4606/0x0000000841779840.accept(Unknown Source)
   	at java.util.concurrent.CompletableFuture.uniWhenComplete(java.base@11.0.18/CompletableFuture.java:859)
   	at java.util.concurrent.CompletableFuture.uniWhenCompleteStage(java.base@11.0.18/CompletableFuture.java:883)
   	at java.util.concurrent.CompletableFuture.whenComplete(java.base@11.0.18/CompletableFuture.java:2251)
   	at org.apache.pulsar.client.impl.MultiTopicsConsumerImpl.subscribeTopicPartitions(MultiTopicsConsumerImpl.java:991)
   	at org.apache.pulsar.client.impl.MultiTopicsConsumerImpl.lambda$subscribeAsync$38(MultiTopicsConsumerImpl.java:917)
   	at org.apache.pulsar.client.impl.MultiTopicsConsumerImpl$$Lambda$4600/0x0000000841777040.accept(Unknown Source)
   	at java.util.concurrent.CompletableFuture$UniAccept.tryFire(java.base@11.0.18/CompletableFuture.java:714)
   	at java.util.concurrent.CompletableFuture.postComplete(java.base@11.0.18/CompletableFuture.java:506)
   	at java.util.concurrent.CompletableFuture.complete(java.base@11.0.18/CompletableFuture.java:2073)
   	at org.apache.pulsar.client.impl.PulsarClientImpl$$Lambda$4598/0x0000000841775440.accept(Unknown Source)
   	at java.util.concurrent.CompletableFuture$UniAccept.tryFire(java.base@11.0.18/CompletableFuture.java:714)
   	at java.util.concurrent.CompletableFuture.postComplete(java.base@11.0.18/CompletableFuture.java:506)
   	at java.util.concurrent.CompletableFuture.complete(java.base@11.0.18/CompletableFuture.java:2073)
   	at org.apache.pulsar.client.impl.BinaryProtoLookupService.lambda$getPartitionedTopicMetadata$4(BinaryProtoLookupService.java:199)
   	at org.apache.pulsar.client.impl.BinaryProtoLookupService$$Lambda$4605/0x0000000841779040.accept(Unknown Source)
   	at java.util.concurrent.CompletableFuture.uniWhenComplete(java.base@11.0.18/CompletableFuture.java:859)
   	at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(java.base@11.0.18/CompletableFuture.java:837)
   	at java.util.concurrent.CompletableFuture.postComplete(java.base@11.0.18/CompletableFuture.java:506)
   	at java.util.concurrent.CompletableFuture.complete(java.base@11.0.18/CompletableFuture.java:2073)
   	at org.apache.pulsar.client.impl.ClientCnx.handlePartitionResponse(ClientCnx.java:618)
   	at org.apache.pulsar.common.protocol.PulsarDecoder.channelRead(PulsarDecoder.java:130)
   	at org.apache.pulsar.shade.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
   	at org.apache.pulsar.shade.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
   	at org.apache.pulsar.shade.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
   	at org.apache.pulsar.shade.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:327)
   	at org.apache.pulsar.shade.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:314)
   	at org.apache.pulsar.shade.io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:435)
   	at org.apache.pulsar.shade.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:279)
   	at org.apache.pulsar.shade.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
   	at org.apache.pulsar.shade.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
   	at org.apache.pulsar.shade.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
   	at org.apache.pulsar.shade.io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1372)
   	at org.apache.pulsar.shade.io.netty.handler.ssl.SslHandler.decodeNonJdkCompatible(SslHandler.java:1246)
   	at org.apache.pulsar.shade.io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1286)
   	at org.apache.pulsar.shade.io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:510)
   	at org.apache.pulsar.shade.io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:449)
   	at org.apache.pulsar.shade.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:279)
   	at org.apache.pulsar.shade.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
   	at org.apache.pulsar.shade.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
   	at org.apache.pulsar.shade.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
   	at org.apache.pulsar.shade.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
   	at org.apache.pulsar.shade.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
   	at org.apache.pulsar.shade.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
   	at org.apache.pulsar.shade.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
   	at org.apache.pulsar.shade.io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:800)
   	at org.apache.pulsar.shade.io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:487)
   	at org.apache.pulsar.shade.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:385)
   	at org.apache.pulsar.shade.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:995)
   	at org.apache.pulsar.shade.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
   	at org.apache.pulsar.shade.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
   	at java.lang.Thread.run(java.base@11.0.18/Thread.java:829)
   "pulsar-client-io-1-4":
   	at java.util.concurrent.ConcurrentHashMap.transfer(java.base@11.0.18/ConcurrentHashMap.java:2475)
   	- waiting to lock <0x00000004312d0940> (a java.util.concurrent.ConcurrentHashMap$Node)
   	at java.util.concurrent.ConcurrentHashMap.addCount(java.base@11.0.18/ConcurrentHashMap.java:2346)
   	at java.util.concurrent.ConcurrentHashMap.putVal(java.base@11.0.18/ConcurrentHashMap.java:1075)
   	at java.util.concurrent.ConcurrentHashMap.putIfAbsent(java.base@11.0.18/ConcurrentHashMap.java:1541)
   	at org.apache.pulsar.client.impl.MultiTopicsConsumerImpl.lambda$doSubscribeTopicPartitions$44(MultiTopicsConsumerImpl.java:1041)
   	- locked <0x000000042cf2ed28> (a java.lang.Object)
   	at org.apache.pulsar.client.impl.MultiTopicsConsumerImpl$$Lambda$7535/0x0000000841fba040.apply(Unknown Source)
   	at java.util.stream.IntPipeline$1$1.accept(java.base@11.0.18/IntPipeline.java:180)
   	at java.util.stream.Streams$RangeIntSpliterator.forEachRemaining(java.base@11.0.18/Streams.java:104)
   	at java.util.Spliterator$OfInt.forEachRemaining(java.base@11.0.18/Spliterator.java:699)
   	at java.util.stream.AbstractPipeline.copyInto(java.base@11.0.18/AbstractPipeline.java:484)
   	at java.util.stream.AbstractPipeline.wrapAndCopyInto(java.base@11.0.18/AbstractPipeline.java:474)
   	at java.util.stream.ReduceOps$ReduceOp.evaluateSequential(java.base@11.0.18/ReduceOps.java:913)
   	at java.util.stream.AbstractPipeline.evaluate(java.base@11.0.18/AbstractPipeline.java:234)
   	at java.util.stream.ReferencePipeline.collect(java.base@11.0.18/ReferencePipeline.java:578)
   	at org.apache.pulsar.client.impl.MultiTopicsConsumerImpl.doSubscribeTopicPartitions(MultiTopicsConsumerImpl.java:1045)
   	at org.apache.pulsar.client.impl.MultiTopicsConsumerImpl.lambda$subscribeTopicPartitions$43(MultiTopicsConsumerImpl.java:993)
   	at org.apache.pulsar.client.impl.MultiTopicsConsumerImpl$$Lambda$4606/0x0000000841779840.accept(Unknown Source)
   	at java.util.concurrent.CompletableFuture.uniWhenComplete(java.base@11.0.18/CompletableFuture.java:859)
   	at java.util.concurrent.CompletableFuture.uniWhenCompleteStage(java.base@11.0.18/CompletableFuture.java:883)
   	at java.util.concurrent.CompletableFuture.whenComplete(java.base@11.0.18/CompletableFuture.java:2251)
   	at org.apache.pulsar.client.impl.MultiTopicsConsumerImpl.subscribeTopicPartitions(MultiTopicsConsumerImpl.java:991)
   	at org.apache.pulsar.client.impl.MultiTopicsConsumerImpl.lambda$subscribeAsync$38(MultiTopicsConsumerImpl.java:917)
   	at org.apache.pulsar.client.impl.MultiTopicsConsumerImpl$$Lambda$4600/0x0000000841777040.accept(Unknown Source)
   	at java.util.concurrent.CompletableFuture$UniAccept.tryFire(java.base@11.0.18/CompletableFuture.java:714)
   	at java.util.concurrent.CompletableFuture.postComplete(java.base@11.0.18/CompletableFuture.java:506)
   	at java.util.concurrent.CompletableFuture.complete(java.base@11.0.18/CompletableFuture.java:2073)
   	at org.apache.pulsar.client.impl.PulsarClientImpl$$Lambda$4598/0x0000000841775440.accept(Unknown Source)
   	at java.util.concurrent.CompletableFuture$UniAccept.tryFire(java.base@11.0.18/CompletableFuture.java:714)
   	at java.util.concurrent.CompletableFuture.postComplete(java.base@11.0.18/CompletableFuture.java:506)
   	at java.util.concurrent.CompletableFuture.complete(java.base@11.0.18/CompletableFuture.java:2073)
   	at org.apache.pulsar.client.impl.BinaryProtoLookupService.lambda$getPartitionedTopicMetadata$4(BinaryProtoLookupService.java:199)
   	at org.apache.pulsar.client.impl.BinaryProtoLookupService$$Lambda$4605/0x0000000841779040.accept(Unknown Source)
   	at java.util.concurrent.CompletableFuture.uniWhenComplete(java.base@11.0.18/CompletableFuture.java:859)
   	at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(java.base@11.0.18/CompletableFuture.java:837)
   	at java.util.concurrent.CompletableFuture.postComplete(java.base@11.0.18/CompletableFuture.java:506)
   	at java.util.concurrent.CompletableFuture.complete(java.base@11.0.18/CompletableFuture.java:2073)
   	at org.apache.pulsar.client.impl.ClientCnx.handlePartitionResponse(ClientCnx.java:618)
   	at org.apache.pulsar.common.protocol.PulsarDecoder.channelRead(PulsarDecoder.java:130)
   	at org.apache.pulsar.shade.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
   	at org.apache.pulsar.shade.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
   	at org.apache.pulsar.shade.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
   	at org.apache.pulsar.shade.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:327)
   	at org.apache.pulsar.shade.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:314)
   	at org.apache.pulsar.shade.io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:435)
   	at org.apache.pulsar.shade.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:279)
   	at org.apache.pulsar.shade.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
   	at org.apache.pulsar.shade.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
   	at org.apache.pulsar.shade.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
   	at org.apache.pulsar.shade.io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1372)
   	at org.apache.pulsar.shade.io.netty.handler.ssl.SslHandler.decodeNonJdkCompatible(SslHandler.java:1246)
   	at org.apache.pulsar.shade.io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1286)
   	at org.apache.pulsar.shade.io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:510)
   	at org.apache.pulsar.shade.io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:449)
   	at org.apache.pulsar.shade.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:279)
   	at org.apache.pulsar.shade.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
   	at org.apache.pulsar.shade.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
   	at org.apache.pulsar.shade.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
   	at org.apache.pulsar.shade.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
   	at org.apache.pulsar.shade.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
   	at org.apache.pulsar.shade.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
   	at org.apache.pulsar.shade.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
   	at org.apache.pulsar.shade.io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:800)
   	at org.apache.pulsar.shade.io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:487)
   	at org.apache.pulsar.shade.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:385)
   	at org.apache.pulsar.shade.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:995)
   	at org.apache.pulsar.shade.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
   	at org.apache.pulsar.shade.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
   	at java.lang.Thread.run(java.base@11.0.18/Thread.java:829)
   
   Found 1 deadlock.
   ```
   
   ### Verifying this change
   
   New test is added
   
   ### Does this pull request potentially affect one of the following parts:
   
   <!-- DO NOT REMOVE THIS SECTION. CHECK THE PROPER BOX ONLY. -->
   
   *If the box was checked, please highlight the changes*
   
   - [ ] Dependencies (add or upgrade a dependency)
   - [ ] The public API
   - [ ] The schema
   - [ ] The default values of configurations
   - [ ] The threading model
   - [ ] The binary protocol
   - [ ] The REST endpoints
   - [ ] The admin CLI options
   - [ ] The metrics
   - [ ] Anything that affects deployment
   
   ### Documentation
   
   <!-- DO NOT REMOVE THIS SECTION. CHECK THE PROPER BOX ONLY. -->
   
   - [ ] `doc` <!-- Your PR contains doc changes. -->
   - [ ] `doc-required` <!-- Your PR changes impact docs and you will update later -->
   - [x] `doc-not-needed` <!-- Your PR changes do not impact docs -->
   - [ ] `doc-complete` <!-- Docs have been already added -->
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [pulsar] heesung-sn commented on a diff in pull request #20669: [fix][client]Fix deadlock issue of consumer while using multiple IO threads

Posted by "heesung-sn (via GitHub)" <gi...@apache.org>.
heesung-sn commented on code in PR #20669:
URL: https://github.com/apache/pulsar/pull/20669#discussion_r1245677876


##########
pulsar-client/src/main/java/org/apache/pulsar/client/impl/MultiTopicsConsumerImpl.java:
##########
@@ -1059,29 +1059,28 @@ private void doSubscribeTopicPartitions(Schema<T> schema,
 
             CompletableFuture<Consumer<T>> subFuture = new CompletableFuture<>();
 
-            consumers.compute(topicName, (key, existingValue) -> {
-                if (existingValue != null) {
-                    String errorMessage = String.format("[%s] Failed to subscribe for topic [%s] in topics consumer. "
-                            + "Topic is already being subscribed for in other thread.", topic, topicName);
-                    log.warn(errorMessage);
-                    subscribeResult.completeExceptionally(new PulsarClientException(errorMessage));
-                    return existingValue;
-                } else {
-                    internalConfig.setStartPaused(paused);
-                    ConsumerImpl<T> newConsumer = createInternalConsumer(internalConfig, topicName,
-                            -1, subFuture, createIfDoesNotExist, schema);
-
-                    synchronized (pauseMutex) {
+            synchronized (pauseMutex) {

Review Comment:
   Oh.. should we clean the code like this ?
   
   https://github.com/heesung-sn/pulsar/commit/0c415d7a6df9c12f1e4a78e640b2446230b96136



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [pulsar] mattisonchao commented on a diff in pull request #20669: [fix][client]Fix deadlock issue of consumer while using multiple IO threads

Posted by "mattisonchao (via GitHub)" <gi...@apache.org>.
mattisonchao commented on code in PR #20669:
URL: https://github.com/apache/pulsar/pull/20669#discussion_r1244717794


##########
pulsar-client/src/main/java/org/apache/pulsar/client/impl/MultiTopicsConsumerImpl.java:
##########
@@ -1059,29 +1059,28 @@ private void doSubscribeTopicPartitions(Schema<T> schema,
 
             CompletableFuture<Consumer<T>> subFuture = new CompletableFuture<>();
 
-            consumers.compute(topicName, (key, existingValue) -> {
-                if (existingValue != null) {
-                    String errorMessage = String.format("[%s] Failed to subscribe for topic [%s] in topics consumer. "
-                            + "Topic is already being subscribed for in other thread.", topic, topicName);
-                    log.warn(errorMessage);
-                    subscribeResult.completeExceptionally(new PulsarClientException(errorMessage));
-                    return existingValue;
-                } else {
-                    internalConfig.setStartPaused(paused);
-                    ConsumerImpl<T> newConsumer = createInternalConsumer(internalConfig, topicName,
-                            -1, subFuture, createIfDoesNotExist, schema);
-
-                    synchronized (pauseMutex) {
+            synchronized (pauseMutex) {

Review Comment:
   How about let `non-partitioned` and `partitioned` use the same lock logic?



##########
pulsar-client/src/main/java/org/apache/pulsar/client/impl/MultiTopicsConsumerImpl.java:
##########
@@ -1059,29 +1059,28 @@ private void doSubscribeTopicPartitions(Schema<T> schema,
 
             CompletableFuture<Consumer<T>> subFuture = new CompletableFuture<>();
 
-            consumers.compute(topicName, (key, existingValue) -> {
-                if (existingValue != null) {
-                    String errorMessage = String.format("[%s] Failed to subscribe for topic [%s] in topics consumer. "
-                            + "Topic is already being subscribed for in other thread.", topic, topicName);
-                    log.warn(errorMessage);
-                    subscribeResult.completeExceptionally(new PulsarClientException(errorMessage));
-                    return existingValue;
-                } else {
-                    internalConfig.setStartPaused(paused);
-                    ConsumerImpl<T> newConsumer = createInternalConsumer(internalConfig, topicName,
-                            -1, subFuture, createIfDoesNotExist, schema);
-
-                    synchronized (pauseMutex) {
+            synchronized (pauseMutex) {

Review Comment:
   How about letting `non-partitioned` and `partitioned` use the same lock logic?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [pulsar] coderzc commented on a diff in pull request #20669: [fix][client]Fix deadlock issue of consumer while using multiple IO threads

Posted by "coderzc (via GitHub)" <gi...@apache.org>.
coderzc commented on code in PR #20669:
URL: https://github.com/apache/pulsar/pull/20669#discussion_r1244777529


##########
pulsar-client/src/main/java/org/apache/pulsar/client/impl/MultiTopicsConsumerImpl.java:
##########
@@ -92,7 +92,7 @@ public class MultiTopicsConsumerImpl<T> extends ConsumerBase<T> {
     // sum of topicPartitions, simple topic has 1, partitioned topic equals to partition number.
     AtomicInteger allTopicPartitionsNumber;
 
-    private boolean paused = false;
+    private volatile boolean paused = false;

Review Comment:
   Why make `paused` volatile? It seems that all read and write are protected by `pauseMutex`.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [pulsar] lhotari commented on a diff in pull request #20669: [fix][client]Fix deadlock issue of consumer while using multiple IO threads

Posted by "lhotari (via GitHub)" <gi...@apache.org>.
lhotari commented on code in PR #20669:
URL: https://github.com/apache/pulsar/pull/20669#discussion_r1245635194


##########
pulsar-client/src/main/java/org/apache/pulsar/client/impl/MultiTopicsConsumerImpl.java:
##########
@@ -1059,29 +1059,28 @@ private void doSubscribeTopicPartitions(Schema<T> schema,
 
             CompletableFuture<Consumer<T>> subFuture = new CompletableFuture<>();
 
-            consumers.compute(topicName, (key, existingValue) -> {
-                if (existingValue != null) {
-                    String errorMessage = String.format("[%s] Failed to subscribe for topic [%s] in topics consumer. "
-                            + "Topic is already being subscribed for in other thread.", topic, topicName);
-                    log.warn(errorMessage);
-                    subscribeResult.completeExceptionally(new PulsarClientException(errorMessage));
-                    return existingValue;
-                } else {
-                    internalConfig.setStartPaused(paused);
-                    ConsumerImpl<T> newConsumer = createInternalConsumer(internalConfig, topicName,
-                            -1, subFuture, createIfDoesNotExist, schema);
-
-                    synchronized (pauseMutex) {
+            synchronized (pauseMutex) {

Review Comment:
   @michaeljmarshall  wouldn't there also be a need to cleanup `newConsumer` if it doesn't get used at all, in the `if (existingValue != null) {` branch?
   This seems to be a resource leak already in the existing code.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [pulsar] codelipenghui commented on pull request #20669: [fix][client]Fix deadlock issue of consumer while using multiple IO threads

Posted by "codelipenghui (via GitHub)" <gi...@apache.org>.
codelipenghui commented on PR #20669:
URL: https://github.com/apache/pulsar/pull/20669#issuecomment-1610701451

   /pulsarbot run-failure-checks


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [pulsar] heesung-sn commented on a diff in pull request #20669: [fix][client]Fix deadlock issue of consumer while using multiple IO threads

Posted by "heesung-sn (via GitHub)" <gi...@apache.org>.
heesung-sn commented on code in PR #20669:
URL: https://github.com/apache/pulsar/pull/20669#discussion_r1244689086


##########
pulsar-client/src/main/java/org/apache/pulsar/client/impl/MultiTopicsConsumerImpl.java:
##########
@@ -1038,12 +1038,12 @@ private void doSubscribeTopicPartitions(Schema<T> schema,
                 .range(0, numPartitions)
                 .mapToObj(
                     partitionIndex -> {
-                        String partitionName = TopicName.get(topicName).getPartition(partitionIndex).toString();
                         CompletableFuture<Consumer<T>> subFuture = new CompletableFuture<>();
-                        configurationData.setStartPaused(paused);
-                        ConsumerImpl<T> newConsumer = createInternalConsumer(configurationData, partitionName,
-                                partitionIndex, subFuture, createIfDoesNotExist, schema);
+                        String partitionName = TopicName.get(topicName).getPartition(partitionIndex).toString();
                         synchronized (pauseMutex) {
+                            configurationData.setStartPaused(paused);

Review Comment:
   Though the current version should synchronize logic, we may need to monitor the performance implication of putting this `createInternalConsumer` under this mutex. As paused is volatile now, I think we may need to consider moving these two lines back to the above `(synchronized (pauseMutex) {)` if the performance is signficantly degraded. I realized `createInternalConsumer`  is a quite expensive op.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [pulsar] mattisonchao commented on a diff in pull request #20669: [fix][client]Fix deadlock issue of consumer while using multiple IO threads

Posted by "mattisonchao (via GitHub)" <gi...@apache.org>.
mattisonchao commented on code in PR #20669:
URL: https://github.com/apache/pulsar/pull/20669#discussion_r1244717794


##########
pulsar-client/src/main/java/org/apache/pulsar/client/impl/MultiTopicsConsumerImpl.java:
##########
@@ -1059,29 +1059,28 @@ private void doSubscribeTopicPartitions(Schema<T> schema,
 
             CompletableFuture<Consumer<T>> subFuture = new CompletableFuture<>();
 
-            consumers.compute(topicName, (key, existingValue) -> {
-                if (existingValue != null) {
-                    String errorMessage = String.format("[%s] Failed to subscribe for topic [%s] in topics consumer. "
-                            + "Topic is already being subscribed for in other thread.", topic, topicName);
-                    log.warn(errorMessage);
-                    subscribeResult.completeExceptionally(new PulsarClientException(errorMessage));
-                    return existingValue;
-                } else {
-                    internalConfig.setStartPaused(paused);
-                    ConsumerImpl<T> newConsumer = createInternalConsumer(internalConfig, topicName,
-                            -1, subFuture, createIfDoesNotExist, schema);
-
-                    synchronized (pauseMutex) {
+            synchronized (pauseMutex) {

Review Comment:
   Let `non-partitioned` and `partitioned` use the same lock logic is better. It will avoid two locks to be deadlock. :) 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [pulsar] heesung-sn commented on a diff in pull request #20669: [fix][client]Fix deadlock issue of consumer while using multiple IO threads

Posted by "heesung-sn (via GitHub)" <gi...@apache.org>.
heesung-sn commented on code in PR #20669:
URL: https://github.com/apache/pulsar/pull/20669#discussion_r1245891073


##########
pulsar-client/src/main/java/org/apache/pulsar/client/impl/MultiTopicsConsumerImpl.java:
##########
@@ -1059,29 +1059,28 @@ private void doSubscribeTopicPartitions(Schema<T> schema,
 
             CompletableFuture<Consumer<T>> subFuture = new CompletableFuture<>();
 
-            consumers.compute(topicName, (key, existingValue) -> {
-                if (existingValue != null) {
-                    String errorMessage = String.format("[%s] Failed to subscribe for topic [%s] in topics consumer. "
-                            + "Topic is already being subscribed for in other thread.", topic, topicName);
-                    log.warn(errorMessage);
-                    subscribeResult.completeExceptionally(new PulsarClientException(errorMessage));
-                    return existingValue;
-                } else {
-                    internalConfig.setStartPaused(paused);
-                    ConsumerImpl<T> newConsumer = createInternalConsumer(internalConfig, topicName,
-                            -1, subFuture, createIfDoesNotExist, schema);
-
-                    synchronized (pauseMutex) {
+            synchronized (pauseMutex) {

Review Comment:
   IMO, though it creates many other objects, the most expensive op from consumer creation is `grabCnx();`,  but I believe this connection creation is an async operation. 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [pulsar] michaeljmarshall commented on a diff in pull request #20669: [fix][client]Fix deadlock issue of consumer while using multiple IO threads

Posted by "michaeljmarshall (via GitHub)" <gi...@apache.org>.
michaeljmarshall commented on code in PR #20669:
URL: https://github.com/apache/pulsar/pull/20669#discussion_r1245728090


##########
pulsar-client/src/main/java/org/apache/pulsar/client/impl/MultiTopicsConsumerImpl.java:
##########
@@ -1059,29 +1059,28 @@ private void doSubscribeTopicPartitions(Schema<T> schema,
 
             CompletableFuture<Consumer<T>> subFuture = new CompletableFuture<>();
 
-            consumers.compute(topicName, (key, existingValue) -> {
-                if (existingValue != null) {
-                    String errorMessage = String.format("[%s] Failed to subscribe for topic [%s] in topics consumer. "
-                            + "Topic is already being subscribed for in other thread.", topic, topicName);
-                    log.warn(errorMessage);
-                    subscribeResult.completeExceptionally(new PulsarClientException(errorMessage));
-                    return existingValue;
-                } else {
-                    internalConfig.setStartPaused(paused);
-                    ConsumerImpl<T> newConsumer = createInternalConsumer(internalConfig, topicName,
-                            -1, subFuture, createIfDoesNotExist, schema);
-
-                    synchronized (pauseMutex) {
+            synchronized (pauseMutex) {

Review Comment:
   > Oh.. should we clean the code like this ?
   > 
   > [heesung-sn@0c415d7](https://github.com/heesung-sn/pulsar/commit/0c415d7a6df9c12f1e4a78e640b2446230b96136)
   
   That approach would work. Do you know how expensive the consumer creation is? We could also use the class's internal executor to do it in a way that doesn't block the calling threads.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [pulsar] michaeljmarshall commented on a diff in pull request #20669: [fix][client]Fix deadlock issue of consumer while using multiple IO threads

Posted by "michaeljmarshall (via GitHub)" <gi...@apache.org>.
michaeljmarshall commented on code in PR #20669:
URL: https://github.com/apache/pulsar/pull/20669#discussion_r1245721658


##########
pulsar-client/src/main/java/org/apache/pulsar/client/impl/MultiTopicsConsumerImpl.java:
##########
@@ -1059,29 +1059,28 @@ private void doSubscribeTopicPartitions(Schema<T> schema,
 
             CompletableFuture<Consumer<T>> subFuture = new CompletableFuture<>();
 
-            consumers.compute(topicName, (key, existingValue) -> {
-                if (existingValue != null) {
-                    String errorMessage = String.format("[%s] Failed to subscribe for topic [%s] in topics consumer. "
-                            + "Topic is already being subscribed for in other thread.", topic, topicName);
-                    log.warn(errorMessage);
-                    subscribeResult.completeExceptionally(new PulsarClientException(errorMessage));
-                    return existingValue;
-                } else {
-                    internalConfig.setStartPaused(paused);
-                    ConsumerImpl<T> newConsumer = createInternalConsumer(internalConfig, topicName,
-                            -1, subFuture, createIfDoesNotExist, schema);
-
-                    synchronized (pauseMutex) {
+            synchronized (pauseMutex) {

Review Comment:
   @lhotari - see the second half of my comment. I do think we have a leak.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [pulsar] codelipenghui merged pull request #20669: [fix][client]Fix deadlock issue of consumer while using multiple IO threads

Posted by "codelipenghui (via GitHub)" <gi...@apache.org>.
codelipenghui merged PR #20669:
URL: https://github.com/apache/pulsar/pull/20669


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [pulsar] lhotari commented on pull request #20669: [fix][client]Fix deadlock issue of consumer while using multiple IO threads

Posted by "lhotari (via GitHub)" <gi...@apache.org>.
lhotari commented on PR #20669:
URL: https://github.com/apache/pulsar/pull/20669#issuecomment-1611440619

   > if the Pulsar Client uses multiple IO threads (default is 1 thread).
   
   the default is number of available processors since #19059 in 3.0.0 .


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [pulsar] lhotari commented on a diff in pull request #20669: [fix][client]Fix deadlock issue of consumer while using multiple IO threads

Posted by "lhotari (via GitHub)" <gi...@apache.org>.
lhotari commented on code in PR #20669:
URL: https://github.com/apache/pulsar/pull/20669#discussion_r1245635194


##########
pulsar-client/src/main/java/org/apache/pulsar/client/impl/MultiTopicsConsumerImpl.java:
##########
@@ -1059,29 +1059,28 @@ private void doSubscribeTopicPartitions(Schema<T> schema,
 
             CompletableFuture<Consumer<T>> subFuture = new CompletableFuture<>();
 
-            consumers.compute(topicName, (key, existingValue) -> {
-                if (existingValue != null) {
-                    String errorMessage = String.format("[%s] Failed to subscribe for topic [%s] in topics consumer. "
-                            + "Topic is already being subscribed for in other thread.", topic, topicName);
-                    log.warn(errorMessage);
-                    subscribeResult.completeExceptionally(new PulsarClientException(errorMessage));
-                    return existingValue;
-                } else {
-                    internalConfig.setStartPaused(paused);
-                    ConsumerImpl<T> newConsumer = createInternalConsumer(internalConfig, topicName,
-                            -1, subFuture, createIfDoesNotExist, schema);
-
-                    synchronized (pauseMutex) {
+            synchronized (pauseMutex) {

Review Comment:
   @michaeljmarshall  wouldn't there also be a need to cleanup `newConsumer` if it doesn't get used at all, in the `if (existingValue != null) {` branch?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [pulsar] mattisonchao commented on a diff in pull request #20669: [fix][client]Fix deadlock issue of consumer while using multiple IO threads

Posted by "mattisonchao (via GitHub)" <gi...@apache.org>.
mattisonchao commented on code in PR #20669:
URL: https://github.com/apache/pulsar/pull/20669#discussion_r1244717794


##########
pulsar-client/src/main/java/org/apache/pulsar/client/impl/MultiTopicsConsumerImpl.java:
##########
@@ -1059,29 +1059,28 @@ private void doSubscribeTopicPartitions(Schema<T> schema,
 
             CompletableFuture<Consumer<T>> subFuture = new CompletableFuture<>();
 
-            consumers.compute(topicName, (key, existingValue) -> {
-                if (existingValue != null) {
-                    String errorMessage = String.format("[%s] Failed to subscribe for topic [%s] in topics consumer. "
-                            + "Topic is already being subscribed for in other thread.", topic, topicName);
-                    log.warn(errorMessage);
-                    subscribeResult.completeExceptionally(new PulsarClientException(errorMessage));
-                    return existingValue;
-                } else {
-                    internalConfig.setStartPaused(paused);
-                    ConsumerImpl<T> newConsumer = createInternalConsumer(internalConfig, topicName,
-                            -1, subFuture, createIfDoesNotExist, schema);
-
-                    synchronized (pauseMutex) {
+            synchronized (pauseMutex) {

Review Comment:
   How about letting `non-partitioned` and `partitioned` use the same lock logic?
   
   lock node first then lock `pauseMutex`



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [pulsar] mattisonchao commented on a diff in pull request #20669: [fix][client]Fix deadlock issue of consumer while using multiple IO threads

Posted by "mattisonchao (via GitHub)" <gi...@apache.org>.
mattisonchao commented on code in PR #20669:
URL: https://github.com/apache/pulsar/pull/20669#discussion_r1244692104


##########
pulsar-client/src/main/java/org/apache/pulsar/client/impl/MultiTopicsConsumerImpl.java:
##########
@@ -1038,12 +1038,12 @@ private void doSubscribeTopicPartitions(Schema<T> schema,
                 .range(0, numPartitions)
                 .mapToObj(
                     partitionIndex -> {
-                        String partitionName = TopicName.get(topicName).getPartition(partitionIndex).toString();
                         CompletableFuture<Consumer<T>> subFuture = new CompletableFuture<>();
-                        configurationData.setStartPaused(paused);
-                        ConsumerImpl<T> newConsumer = createInternalConsumer(configurationData, partitionName,
-                                partitionIndex, subFuture, createIfDoesNotExist, schema);
+                        String partitionName = TopicName.get(topicName).getPartition(partitionIndex).toString();
                         synchronized (pauseMutex) {
+                            configurationData.setStartPaused(paused);

Review Comment:
   +1



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [pulsar] heesung-sn commented on a diff in pull request #20669: [fix][client]Fix deadlock issue of consumer while using multiple IO threads

Posted by "heesung-sn (via GitHub)" <gi...@apache.org>.
heesung-sn commented on code in PR #20669:
URL: https://github.com/apache/pulsar/pull/20669#discussion_r1244689086


##########
pulsar-client/src/main/java/org/apache/pulsar/client/impl/MultiTopicsConsumerImpl.java:
##########
@@ -1038,12 +1038,12 @@ private void doSubscribeTopicPartitions(Schema<T> schema,
                 .range(0, numPartitions)
                 .mapToObj(
                     partitionIndex -> {
-                        String partitionName = TopicName.get(topicName).getPartition(partitionIndex).toString();
                         CompletableFuture<Consumer<T>> subFuture = new CompletableFuture<>();
-                        configurationData.setStartPaused(paused);
-                        ConsumerImpl<T> newConsumer = createInternalConsumer(configurationData, partitionName,
-                                partitionIndex, subFuture, createIfDoesNotExist, schema);
+                        String partitionName = TopicName.get(topicName).getPartition(partitionIndex).toString();
                         synchronized (pauseMutex) {
+                            configurationData.setStartPaused(paused);

Review Comment:
   Though the current version should synchronize logic, we may need to monitor the performance implication of putting this `createInternalConsumer` under this mutex. As `paused` is volatile now, I think we may need to consider moving these two lines back to the above `(synchronized (pauseMutex) {)` if the performance is signficantly degraded. I realized `createInternalConsumer`  is a quite expensive op.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [pulsar] michaeljmarshall commented on a diff in pull request #20669: [fix][client]Fix deadlock issue of consumer while using multiple IO threads

Posted by "michaeljmarshall (via GitHub)" <gi...@apache.org>.
michaeljmarshall commented on code in PR #20669:
URL: https://github.com/apache/pulsar/pull/20669#discussion_r1245721658


##########
pulsar-client/src/main/java/org/apache/pulsar/client/impl/MultiTopicsConsumerImpl.java:
##########
@@ -1059,29 +1059,28 @@ private void doSubscribeTopicPartitions(Schema<T> schema,
 
             CompletableFuture<Consumer<T>> subFuture = new CompletableFuture<>();
 
-            consumers.compute(topicName, (key, existingValue) -> {
-                if (existingValue != null) {
-                    String errorMessage = String.format("[%s] Failed to subscribe for topic [%s] in topics consumer. "
-                            + "Topic is already being subscribed for in other thread.", topic, topicName);
-                    log.warn(errorMessage);
-                    subscribeResult.completeExceptionally(new PulsarClientException(errorMessage));
-                    return existingValue;
-                } else {
-                    internalConfig.setStartPaused(paused);
-                    ConsumerImpl<T> newConsumer = createInternalConsumer(internalConfig, topicName,
-                            -1, subFuture, createIfDoesNotExist, schema);
-
-                    synchronized (pauseMutex) {
+            synchronized (pauseMutex) {

Review Comment:
   @lhotari - see the second half of my comment.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [pulsar] mattisonchao commented on a diff in pull request #20669: [fix][client]Fix deadlock issue of consumer while using multiple IO threads

Posted by "mattisonchao (via GitHub)" <gi...@apache.org>.
mattisonchao commented on code in PR #20669:
URL: https://github.com/apache/pulsar/pull/20669#discussion_r1244717794


##########
pulsar-client/src/main/java/org/apache/pulsar/client/impl/MultiTopicsConsumerImpl.java:
##########
@@ -1059,29 +1059,28 @@ private void doSubscribeTopicPartitions(Schema<T> schema,
 
             CompletableFuture<Consumer<T>> subFuture = new CompletableFuture<>();
 
-            consumers.compute(topicName, (key, existingValue) -> {
-                if (existingValue != null) {
-                    String errorMessage = String.format("[%s] Failed to subscribe for topic [%s] in topics consumer. "
-                            + "Topic is already being subscribed for in other thread.", topic, topicName);
-                    log.warn(errorMessage);
-                    subscribeResult.completeExceptionally(new PulsarClientException(errorMessage));
-                    return existingValue;
-                } else {
-                    internalConfig.setStartPaused(paused);
-                    ConsumerImpl<T> newConsumer = createInternalConsumer(internalConfig, topicName,
-                            -1, subFuture, createIfDoesNotExist, schema);
-
-                    synchronized (pauseMutex) {
+            synchronized (pauseMutex) {

Review Comment:
   How about letting `non-partitioned` and `partitioned` use the same lock logic?
   
   lock node first then lock `pauseMutex`



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [pulsar] michaeljmarshall commented on a diff in pull request #20669: [fix][client]Fix deadlock issue of consumer while using multiple IO threads

Posted by "michaeljmarshall (via GitHub)" <gi...@apache.org>.
michaeljmarshall commented on code in PR #20669:
URL: https://github.com/apache/pulsar/pull/20669#discussion_r1245584114


##########
pulsar-client/src/main/java/org/apache/pulsar/client/impl/MultiTopicsConsumerImpl.java:
##########
@@ -1059,29 +1059,28 @@ private void doSubscribeTopicPartitions(Schema<T> schema,
 
             CompletableFuture<Consumer<T>> subFuture = new CompletableFuture<>();
 
-            consumers.compute(topicName, (key, existingValue) -> {
-                if (existingValue != null) {
-                    String errorMessage = String.format("[%s] Failed to subscribe for topic [%s] in topics consumer. "
-                            + "Topic is already being subscribed for in other thread.", topic, topicName);
-                    log.warn(errorMessage);
-                    subscribeResult.completeExceptionally(new PulsarClientException(errorMessage));
-                    return existingValue;
-                } else {
-                    internalConfig.setStartPaused(paused);
-                    ConsumerImpl<T> newConsumer = createInternalConsumer(internalConfig, topicName,
-                            -1, subFuture, createIfDoesNotExist, schema);
-
-                    synchronized (pauseMutex) {
+            synchronized (pauseMutex) {

Review Comment:
   An alternative solution could be something like
   
   ```java
               internalConfig.setStartPaused(paused);
               ConsumerImpl<T> newConsumer = createInternalConsumer(internalConfig, topicName,
                       -1, subFuture, createIfDoesNotExist, schema);
               synchronized (pauseMutex) {
                   if (paused) {
                       newConsumer.pause();
                   } else {
                       newConsumer.resume();
                   }
                   ConsumerImpl<T> existingValue = consumers.putIfAbsent(topicName, newConsumer);
                   if (existingValue != null) {
                       String errorMessage = String.format("[%s] Failed to subscribe for topic [%s] in topics consumer. "
                               + "Topic is already being subscribed for in other thread.", topic, topicName);
                       log.warn(errorMessage);
                       subscribeResult.completeExceptionally(new PulsarClientException(errorMessage));
                   }
               }
   ```
   This block is similar to others in the class. However, the primary issue is that we'd need to close the consumer if it is already created. Speaking of closing consumers, I actually think we might have a leak in this code:
   
   https://github.com/apache/pulsar/blob/e956db729f5098ff319237fd4220dbfb234c1b18/pulsar-client/src/main/java/org/apache/pulsar/client/impl/MultiTopicsConsumerImpl.java#L1044-L1053



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [pulsar] codecov-commenter commented on pull request #20669: [fix][client]Fix deadlock issue of consumer while using multiple IO threads

Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #20669:
URL: https://github.com/apache/pulsar/pull/20669#issuecomment-1612315467

   ## [Codecov](https://app.codecov.io/gh/apache/pulsar/pull/20669?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
   > Merging [#20669](https://app.codecov.io/gh/apache/pulsar/pull/20669?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) (ae2bd5e) into [master](https://app.codecov.io/gh/apache/pulsar/commit/2b01f83ea4f3ab7b22f4ea2b0290cc183e79bbb8?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) (2b01f83) will **increase** coverage by `39.51%`.
   > The diff coverage is `92.85%`.
   
   [![Impacted file tree graph](https://app.codecov.io/gh/apache/pulsar/pull/20669/graphs/tree.svg?width=650&height=150&src=pr&token=acYqCpsK9J&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)](https://app.codecov.io/gh/apache/pulsar/pull/20669?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)
   
   ```diff
   @@              Coverage Diff              @@
   ##             master   #20669       +/-   ##
   =============================================
   + Coverage     33.58%   73.09%   +39.51%     
   - Complexity    12127    31919    +19792     
   =============================================
     Files          1613     1867      +254     
     Lines        126241   138684    +12443     
     Branches      13770    15240     +1470     
   =============================================
   + Hits          42396   101370    +58974     
   + Misses        78331    29289    -49042     
   - Partials       5514     8025     +2511     
   ```
   
   | Flag | Coverage Δ | |
   |---|---|---|
   | inttests | `24.15% <50.00%> (-0.04%)` | :arrow_down: |
   | systests | `24.98% <50.00%> (?)` | |
   | unittests | `72.38% <92.85%> (+40.34%)` | :arrow_up: |
   
   Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#carryforward-flags-in-the-pull-request-comment) to find out more.
   
   | [Impacted Files](https://app.codecov.io/gh/apache/pulsar/pull/20669?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
   |---|---|---|
   | [...he/pulsar/client/impl/MultiTopicsConsumerImpl.java](https://app.codecov.io/gh/apache/pulsar/pull/20669?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-cHVsc2FyLWNsaWVudC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcHVsc2FyL2NsaWVudC9pbXBsL011bHRpVG9waWNzQ29uc3VtZXJJbXBsLmphdmE=) | `78.28% <92.85%> (+45.91%)` | :arrow_up: |
   
   ... and [1521 files with indirect coverage changes](https://app.codecov.io/gh/apache/pulsar/pull/20669/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [pulsar] coderzc commented on a diff in pull request #20669: [fix][client]Fix deadlock issue of consumer while using multiple IO threads

Posted by "coderzc (via GitHub)" <gi...@apache.org>.
coderzc commented on code in PR #20669:
URL: https://github.com/apache/pulsar/pull/20669#discussion_r1244777529


##########
pulsar-client/src/main/java/org/apache/pulsar/client/impl/MultiTopicsConsumerImpl.java:
##########
@@ -92,7 +92,7 @@ public class MultiTopicsConsumerImpl<T> extends ConsumerBase<T> {
     // sum of topicPartitions, simple topic has 1, partitioned topic equals to partition number.
     AtomicInteger allTopicPartitionsNumber;
 
-    private boolean paused = false;
+    private volatile boolean paused = false;

Review Comment:
   Why make `paused` volatile? It seems that all read and write are protected by `pauseMutex`.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [pulsar] lhotari commented on a diff in pull request #20669: [fix][client]Fix deadlock issue of consumer while using multiple IO threads

Posted by "lhotari (via GitHub)" <gi...@apache.org>.
lhotari commented on code in PR #20669:
URL: https://github.com/apache/pulsar/pull/20669#discussion_r1245635194


##########
pulsar-client/src/main/java/org/apache/pulsar/client/impl/MultiTopicsConsumerImpl.java:
##########
@@ -1059,29 +1059,28 @@ private void doSubscribeTopicPartitions(Schema<T> schema,
 
             CompletableFuture<Consumer<T>> subFuture = new CompletableFuture<>();
 
-            consumers.compute(topicName, (key, existingValue) -> {
-                if (existingValue != null) {
-                    String errorMessage = String.format("[%s] Failed to subscribe for topic [%s] in topics consumer. "
-                            + "Topic is already being subscribed for in other thread.", topic, topicName);
-                    log.warn(errorMessage);
-                    subscribeResult.completeExceptionally(new PulsarClientException(errorMessage));
-                    return existingValue;
-                } else {
-                    internalConfig.setStartPaused(paused);
-                    ConsumerImpl<T> newConsumer = createInternalConsumer(internalConfig, topicName,
-                            -1, subFuture, createIfDoesNotExist, schema);
-
-                    synchronized (pauseMutex) {
+            synchronized (pauseMutex) {

Review Comment:
   @michaeljmarshall  wouldn't there also be a need to cleanup `newConsumer` if it doesn't get used at all?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org