You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pulsar.apache.org by GitBox <gi...@apache.org> on 2022/01/07 14:37:08 UTC
[GitHub] [pulsar] tongsucn opened a new issue #13669: Unloading bundle with non-persistent topics may block the target broker.
tongsucn opened a new issue #13669:
URL: https://github.com/apache/pulsar/issues/13669
In our production environment, the bundles with non-persistent topics may result in the target broker getting stuck, when they are unloaded and moved to the target broker.
## Information from log
The producers on this bundle and all other bundles working on this broker, give `Timeout` error code, and the target broker's log shows information like
* `Error reading entries at AAA:BBB : Bookie operation timeout, Read Type Normal - Retrying to read in 15.0 seconds`
* `Failed to write entry (CCC, DDD): Bookie operation timeout`
When reading the broker log more carefully, I found that the broker got stuck at producer creating. It prints information like
`07:29:50.893 [pulsar-io-4-7] INFO org.apache.pulsar.broker.service.ServerCnx - [/IP:PORT][DOMAIN://TENANT/NS/TOPIC-partition-XX] Creating producer. producerId=XX`
but it doesn't print `Created new producer xxx`.
## Information from monitor
From the monitor, which is periodically read from `/admin/v2/broker-stats/topics` interface, I found that all publishing rate on this broker dropped to zero, and there were no bundle splitting before or during the unloading. (Actually I turned off the auto-bundle-split).
The problem only happens on the namespace with non-persistent topics, and since we enabled `ns-isolation-policy`, I didn't know if it affects other namespaces if different namespaces work together on the same broker.
## About the deployment and solution for now
The problem firstly appeared in Dec. 2021, when I just upgraded the broker from `2.7.1`to `2.8.1` and started using non-persistent topic. A few days ago I upgraded the broker to `2.8.2` , the problem persists. We are using the official docker image to deploy the broker.
This problem happened for more than 20 times since Dec. 2021. And for now, the only solution to bring the cluster back to normal is restarting the target broker.
## Information from `jstack`
I still didn't find the way the reproduce this problem, but I did find some clue via `jstack`. As you see, some producer creating and consumer subscribing are waiting for `<0x000000065ef5aea0>`, which is locked by `ForkJoinPool.commonPool-worker-13`, and it is waiting at `BrokerService.java:1742`:
```java
checkTopicNsOwnershipAsync(topic).join();
```
After reading the code, I think this problem is related to ownership cache, like #4190 and #5919 . But I'm not familiar with the metadata and ownership cache, so I didn't find any piece of code that could cause this stuck. Could you please help me to locate the problem?
```log
"pulsar-io-4-1" #83 prio=5 os_prio=0 cpu=221239.11ms elapsed=7428.85s allocated=761M defined_classes=55 tid=0x00007f7d9c0b9000 nid=0x106 waiting on condition [0x00007f7dba20e000]
java.lang.Thread.State: WAITING (parking)
at jdk.internal.misc.Unsafe.park(java.base@11.0.13/Native Method)
- parking to wait for <0x000000065ef5aea0> (a java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync)
at java.util.concurrent.locks.LockSupport.park(java.base@11.0.13/LockSupport.java:194)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(java.base@11.0.13/AbstractQueuedSynchronizer.java:885)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireShared(java.base@11.0.13/AbstractQueuedSynchronizer.java:1009)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireShared(java.base@11.0.13/AbstractQueuedSynchronizer.java:1324)
at java.util.concurrent.locks.ReentrantReadWriteLock$ReadLock.lock(java.base@11.0.13/ReentrantReadWriteLock.java:738)
at org.apache.pulsar.broker.service.nonpersistent.NonPersistentTopic.subscribe(NonPersistentTopic.java:261)
...
Locked ownable synchronizers:
- None
"pulsar-io-4-5" #89 prio=5 os_prio=0 cpu=227741.54ms elapsed=7428.85s allocated=768M defined_classes=67 tid=0x00007f7d9c0e1000 nid=0x10c waiting on condition [0x00007f7db9c08000]
java.lang.Thread.State: WAITING (parking)
at jdk.internal.misc.Unsafe.park(java.base@11.0.13/Native Method)
- parking to wait for <0x000000065c02def8> (a java.util.concurrent.CompletableFuture$Signaller)
at java.util.concurrent.locks.LockSupport.park(java.base@11.0.13/LockSupport.java:194)
at java.util.concurrent.CompletableFuture$Signaller.block(java.base@11.0.13/CompletableFuture.java:1796)
at java.util.concurrent.ForkJoinPool.managedBlock(java.base@11.0.13/ForkJoinPool.java:3128)
at java.util.concurrent.CompletableFuture.waitingGet(java.base@11.0.13/CompletableFuture.java:1823)
at java.util.concurrent.CompletableFuture.join(java.base@11.0.13/CompletableFuture.java:2043)
at org.apache.pulsar.broker.service.BrokerService.checkTopicNsOwnership(BrokerService.java:1742)
...
Locked ownable synchronizers:
- None
"pulsar-io-4-8" #95 prio=5 os_prio=0 cpu=216221.84ms elapsed=7428.84s allocated=784M defined_classes=44 tid=0x00007f7d9c0e7000 nid=0x112 waiting on condition [0x00007f7db9602000]
java.lang.Thread.State: WAITING (parking)
at jdk.internal.misc.Unsafe.park(java.base@11.0.13/Native Method)
- parking to wait for <0x000000065f1b83d8> (a java.util.concurrent.CompletableFuture$Signaller)
at java.util.concurrent.locks.LockSupport.park(java.base@11.0.13/LockSupport.java:194)
at java.util.concurrent.CompletableFuture$Signaller.block(java.base@11.0.13/CompletableFuture.java:1796)
at java.util.concurrent.ForkJoinPool.managedBlock(java.base@11.0.13/ForkJoinPool.java:3128)
at java.util.concurrent.CompletableFuture.waitingGet(java.base@11.0.13/CompletableFuture.java:1823)
at java.util.concurrent.CompletableFuture.join(java.base@11.0.13/CompletableFuture.java:2043)
at org.apache.pulsar.broker.service.BrokerService.checkTopicNsOwnership(BrokerService.java:1742)
...
Locked ownable synchronizers:
- None
"pulsar-io-4-10" #98 prio=5 os_prio=0 cpu=208215.64ms elapsed=7428.84s allocated=788M defined_classes=45 tid=0x00007f7d9c0eb000 nid=0x114 waiting on condition [0x00007f7db9400000]
java.lang.Thread.State: WAITING (parking)
at jdk.internal.misc.Unsafe.park(java.base@11.0.13/Native Method)
- parking to wait for <0x000000065ef5aea0> (a java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync)
at java.util.concurrent.locks.LockSupport.park(java.base@11.0.13/LockSupport.java:194)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(java.base@11.0.13/AbstractQueuedSynchronizer.java:885)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(java.base@11.0.13/AbstractQueuedSynchronizer.java:917)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(java.base@11.0.13/AbstractQueuedSynchronizer.java:1240)
at java.util.concurrent.locks.ReentrantReadWriteLock$WriteLock.lock(java.base@11.0.13/ReentrantReadWriteLock.java:959)
at org.apache.pulsar.broker.service.AbstractTopic.incrementTopicEpochIfNeeded(AbstractTopic.java:427)
...
Locked ownable synchronizers:
- None
"pulsar-io-4-12" #100 prio=5 os_prio=0 cpu=171300.86ms elapsed=7428.84s allocated=676M defined_classes=32 tid=0x00007f7d9c0ef000 nid=0x117 waiting on condition [0x00007f7db90fd000]
java.lang.Thread.State: WAITING (parking)
at jdk.internal.misc.Unsafe.park(java.base@11.0.13/Native Method)
- parking to wait for <0x00000006642f0790> (a java.util.concurrent.CompletableFuture$Signaller)
at java.util.concurrent.locks.LockSupport.park(java.base@11.0.13/LockSupport.java:194)
at java.util.concurrent.CompletableFuture$Signaller.block(java.base@11.0.13/CompletableFuture.java:1796)
at java.util.concurrent.ForkJoinPool.managedBlock(java.base@11.0.13/ForkJoinPool.java:3128)
at java.util.concurrent.CompletableFuture.waitingGet(java.base@11.0.13/CompletableFuture.java:1823)
at java.util.concurrent.CompletableFuture.join(java.base@11.0.13/CompletableFuture.java:2043)
at org.apache.pulsar.broker.service.BrokerService.checkTopicNsOwnership(BrokerService.java:1742)
...
Locked ownable synchronizers:
- None
"pulsar-io-4-13" #101 prio=5 os_prio=0 cpu=178036.59ms elapsed=7428.84s allocated=767M defined_classes=15 tid=0x00007f7d9c0f1000 nid=0x118 waiting on condition [0x00007f7db8ffc000]
java.lang.Thread.State: WAITING (parking)
at jdk.internal.misc.Unsafe.park(java.base@11.0.13/Native Method)
- parking to wait for <0x000000065d913920> (a java.util.concurrent.CompletableFuture$Signaller)
at java.util.concurrent.locks.LockSupport.park(java.base@11.0.13/LockSupport.java:194)
at java.util.concurrent.CompletableFuture$Signaller.block(java.base@11.0.13/CompletableFuture.java:1796)
at java.util.concurrent.ForkJoinPool.managedBlock(java.base@11.0.13/ForkJoinPool.java:3128)
at java.util.concurrent.CompletableFuture.waitingGet(java.base@11.0.13/CompletableFuture.java:1823)
at java.util.concurrent.CompletableFuture.join(java.base@11.0.13/CompletableFuture.java:2043)
at org.apache.pulsar.broker.service.BrokerService.checkTopicNsOwnership(BrokerService.java:1742)
at org.apache.pulsar.broker.service.nonpersistent.NonPersistentTopic.subscribe(NonPersistentTopic.java:236)
...
Locked ownable synchronizers:
- None
"pulsar-io-4-14" #103 prio=5 os_prio=0 cpu=168610.81ms elapsed=7428.83s allocated=716M defined_classes=57 tid=0x00007f7d9c0f3000 nid=0x11a waiting on condition [0x00007f7db8dfa000]
java.lang.Thread.State: WAITING (parking)
at jdk.internal.misc.Unsafe.park(java.base@11.0.13/Native Method)
- parking to wait for <0x000000066125abe0> (a java.util.concurrent.CompletableFuture$Signaller)
at java.util.concurrent.locks.LockSupport.park(java.base@11.0.13/LockSupport.java:194)
at java.util.concurrent.CompletableFuture$Signaller.block(java.base@11.0.13/CompletableFuture.java:1796)
at java.util.concurrent.ForkJoinPool.managedBlock(java.base@11.0.13/ForkJoinPool.java:3128)
at java.util.concurrent.CompletableFuture.waitingGet(java.base@11.0.13/CompletableFuture.java:1823)
at java.util.concurrent.CompletableFuture.join(java.base@11.0.13/CompletableFuture.java:2043)
at org.apache.pulsar.broker.service.BrokerService.checkTopicNsOwnership(BrokerService.java:1742)
...
Locked ownable synchronizers:
- None
"pulsar-web-41-13" #159 prio=5 os_prio=0 cpu=464.22ms elapsed=7424.46s allocated=40941K defined_classes=139 tid=0x00007f7c64006000 nid=0x153 waiting on condition [0x00007f7cdcfda000]
java.lang.Thread.State: WAITING (parking)
at jdk.internal.misc.Unsafe.park(java.base@11.0.13/Native Method)
- parking to wait for <0x000000065ef5aea0> (a java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync)
at java.util.concurrent.locks.LockSupport.park(java.base@11.0.13/LockSupport.java:194)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(java.base@11.0.13/AbstractQueuedSynchronizer.java:885)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(java.base@11.0.13/AbstractQueuedSynchronizer.java:917)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(java.base@11.0.13/AbstractQueuedSynchronizer.java:1240)
at java.util.concurrent.locks.ReentrantReadWriteLock$WriteLock.lock(java.base@11.0.13/ReentrantReadWriteLock.java:959)
at org.apache.pulsar.broker.service.nonpersistent.NonPersistentTopic.close(NonPersistentTopic.java:433)
...
Locked ownable synchronizers:
- <0x00000005fc81e108> (a java.util.concurrent.ThreadPoolExecutor$Worker)
"ForkJoinPool.commonPool-worker-15" #183 daemon prio=5 os_prio=0 cpu=30.26ms elapsed=4043.24s allocated=1854K defined_classes=0 tid=0x00007f7c1800d000 nid=0x1d4 waiting on condition [0x00007f7cdc4d3000]
java.lang.Thread.State: WAITING (parking)
at jdk.internal.misc.Unsafe.park(java.base@11.0.13/Native Method)
- parking to wait for <0x000000065ef5aea0> (a java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync)
at java.util.concurrent.locks.LockSupport.park(java.base@11.0.13/LockSupport.java:194)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(java.base@11.0.13/AbstractQueuedSynchronizer.java:885)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(java.base@11.0.13/AbstractQueuedSynchronizer.java:917)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(java.base@11.0.13/AbstractQueuedSynchronizer.java:1240)
at java.util.concurrent.locks.ReentrantReadWriteLock$WriteLock.lock(java.base@11.0.13/ReentrantReadWriteLock.java:959)
at org.apache.pulsar.broker.service.AbstractTopic.incrementTopicEpochIfNeeded(AbstractTopic.java:427)
...
Locked ownable synchronizers:
- None
"ForkJoinPool.commonPool-worker-13" #190 daemon prio=5 os_prio=0 cpu=15.71ms elapsed=2546.36s allocated=1173K defined_classes=6 tid=0x00007f7d3000a800 nid=0x20d waiting on condition [0x00007f7f813af000]
java.lang.Thread.State: WAITING (parking)
at jdk.internal.misc.Unsafe.park(java.base@11.0.13/Native Method)
- parking to wait for <0x00000007d5d313a8> (a java.util.concurrent.CompletableFuture$Signaller)
at java.util.concurrent.locks.LockSupport.park(java.base@11.0.13/LockSupport.java:194)
at java.util.concurrent.CompletableFuture$Signaller.block(java.base@11.0.13/CompletableFuture.java:1796)
at java.util.concurrent.ForkJoinPool.managedBlock(java.base@11.0.13/ForkJoinPool.java:3118)
at java.util.concurrent.CompletableFuture.waitingGet(java.base@11.0.13/CompletableFuture.java:1823)
at java.util.concurrent.CompletableFuture.join(java.base@11.0.13/CompletableFuture.java:2043)
at org.apache.pulsar.broker.service.BrokerService.checkTopicNsOwnership(BrokerService.java:1742)
at org.apache.pulsar.broker.service.AbstractTopic.lambda$addProducer$6(AbstractTopic.java:397)
at org.apache.pulsar.broker.service.AbstractTopic$$Lambda$845/0x0000000840639040.accept(Unknown Source)
at java.util.concurrent.CompletableFuture.uniAcceptNow(java.base@11.0.13/CompletableFuture.java:753)
at java.util.concurrent.CompletableFuture.uniAcceptStage(java.base@11.0.13/CompletableFuture.java:731)
at java.util.concurrent.CompletableFuture.thenAccept(java.base@11.0.13/CompletableFuture.java:2108)
at org.apache.pulsar.broker.service.AbstractTopic.addProducer(AbstractTopic.java:394)
at org.apache.pulsar.broker.service.ServerCnx.lambda$handleProducer$22(ServerCnx.java:1252)
at org.apache.pulsar.broker.service.ServerCnx$$Lambda$815/0x000000084061a440.accept(Unknown Source)
at java.util.concurrent.CompletableFuture$UniAccept.tryFire(java.base@11.0.13/CompletableFuture.java:714)
at java.util.concurrent.CompletableFuture.postComplete(java.base@11.0.13/CompletableFuture.java:506)
at java.util.concurrent.CompletableFuture.complete(java.base@11.0.13/CompletableFuture.java:2073)
at org.apache.pulsar.broker.service.schema.BookkeeperSchemaStorage.lambda$getSchema$7(BookkeeperSchemaStorage.java:224)
at org.apache.pulsar.broker.service.schema.BookkeeperSchemaStorage$$Lambda$804/0x000000084060c040.apply(Unknown Source)
at java.util.concurrent.CompletableFuture.uniHandle(java.base@11.0.13/CompletableFuture.java:930)
at java.util.concurrent.CompletableFuture$UniHandle.tryFire(java.base@11.0.13/CompletableFuture.java:907)
at java.util.concurrent.CompletableFuture$Completion.exec(java.base@11.0.13/CompletableFuture.java:479)
at java.util.concurrent.ForkJoinTask.doExec(java.base@11.0.13/ForkJoinTask.java:290)
at java.util.concurrent.ForkJoinPool$WorkQueue.topLevelExec(java.base@11.0.13/ForkJoinPool.java:1020)
at java.util.concurrent.ForkJoinPool.scan(java.base@11.0.13/ForkJoinPool.java:1656)
at java.util.concurrent.ForkJoinPool.runWorker(java.base@11.0.13/ForkJoinPool.java:1594)
at java.util.concurrent.ForkJoinWorkerThread.run(java.base@11.0.13/ForkJoinWorkerThread.java:183)
Locked ownable synchronizers:
- <0x000000065ef5aea0> (a java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync)
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] tongsucn commented on issue #13669: Unloading bundle with non-persistent topics may block the target broker.
Posted by GitBox <gi...@apache.org>.
tongsucn commented on issue #13669:
URL: https://github.com/apache/pulsar/issues/13669#issuecomment-1042549604
It seems to be fixed in 2.8.2.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] tongsucn closed issue #13669: Unloading bundle with non-persistent topics may block the target broker.
Posted by GitBox <gi...@apache.org>.
tongsucn closed issue #13669:
URL: https://github.com/apache/pulsar/issues/13669
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org