You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pulsar.apache.org by "mrtworo (via GitHub)" <gi...@apache.org> on 2025/01/03 11:27:20 UTC
[I] [Bug] "Not enough non-faulty bookies available error code: -6" persisting after outage while all bookies are reported as healthy [pulsar]
mrtworo opened a new issue, #23807:
URL: https://github.com/apache/pulsar/issues/23807
### Search before asking
- [X] I searched in the [issues](https://github.com/apache/pulsar/issues) and found nothing similar.
### Read release policy
- [X] I understand that unsupported versions don't get bug fixes. I will attempt to reproduce the issue on a supported version of Pulsar client and Pulsar broker.
### Version
Apache pulsar-3.8.0 chart with pulsar 4.0.1 deployed in EKS.
### Minimal reproduce step
Pulsar consisting of:
4 bookies
3 brokers
3 zookeepers
two persistent topics configured with E=3, Qw=3, Qa=2
Two bookies and zookeeper restart and are unavailable for minutes.
Both topics owned by the same brooker.
### What did you expect to see?
1. Both topics affected by 2/4 bookies unavailable.
2. Broker not being able to form new ensembles to handle writes.
3. Both recovered after bookies are up.
### What did you see instead?
Both were rightly complaining on "org.apache.bookkeeper.client.BKException$BKNotEnoughBookiesException: Not enough non-faulty bookies available", however when bookies were back, only first topic a7u_transactions was able to recover, second topic a7u_transactions_sanitized was stuck in endless loop of un-fencing and failed writes until broker was restarted an it opened normally on the new owner.
Before the restart in following morning, we've double checked the bookies status:
```
pulsar-toolset-0:/pulsar$ bin/bookkeeper shell listbookies -rw -h
...
2025-01-02T09:28:46,406+0000 [main] INFO org.apache.bookkeeper.tools.cli.commands.bookies.ListBookiesCommand - ReadWrite Bookies :
2025-01-02T09:28:46,422+0000 [main] INFO org.apache.bookkeeper.tools.cli.commands.bookies.ListBookiesCommand - BookieID:pulsar-bookie-2.pulsar-bookie.pulsar.svc.cluster.local:3181, IP:10.11.219.28, Port:3181, Hostname:pulsar-bookie-2.pulsar-bookie.pulsar.svc.cluster.local
2025-01-02T09:28:46,423+0000 [main] INFO org.apache.bookkeeper.tools.cli.commands.bookies.ListBookiesCommand - BookieID:pulsar-bookie-3.pulsar-bookie.pulsar.svc.cluster.local:3181, IP:10.11.219.36, Port:3181, Hostname:pulsar-bookie-3.pulsar-bookie.pulsar.svc.cluster.local
2025-01-02T09:28:46,424+0000 [main] INFO org.apache.bookkeeper.tools.cli.commands.bookies.ListBookiesCommand - BookieID:pulsar-bookie-0.pulsar-bookie.pulsar.svc.cluster.local:3181, IP:10.11.225.176, Port:3181, Hostname:pulsar-bookie-0.pulsar-bookie.pulsar.svc.cluster.local
2025-01-02T09:28:46,426+0000 [main] INFO org.apache.bookkeeper.tools.cli.commands.bookies.ListBookiesCommand - BookieID:pulsar-bookie-1.pulsar-bookie.pulsar.svc.cluster.local:3181, IP:10.11.232.43, Port:3181, Hostname:pulsar-bookie-1.pulsar-bookie.pulsar.svc.cluster.local
```
all were up, there were no other topics were reporting any issues with bookies.
We don't understand why a7u_transactions recovered while a7u_transactions_sanitized remained impacted for hours while all bookies were up. It seemed like it's an edge case where there was no attempt to form new ensemble for the affected topic.
Detailed logs attached.
### Anything else?
bookie-0 returns at:
```
2025-01-01T20:53:09,319+0000 [BookKeeperClientScheduler-OrderedScheduler-0-0] INFO org.apache.bookkeeper.net.NetworkTopologyImpl - Adding a new node: /default-rack/pulsar-bookie-0.pulsar-bookie.pulsar.svc.cluster.local:3181
...
2025-01-01T20:53:09,624+0000 [main] INFO org.apache.bookkeeper.common.component.ComponentStarter - Started component bookie-server.
```
bookie-1 returns at:
```
2025-01-01T20:53:34,979+0000 [BookKeeperClientScheduler-OrderedScheduler-0-0] INFO org.apache.bookkeeper.net.NetworkTopologyImpl - Adding a new node: /default-rack/pulsar-bookie-1.pulsar-bookie.pulsar.svc.cluster.local:3181
...
2025-01-01T20:53:35,305+0000 [main] INFO org.apache.bookkeeper.common.component.ComponentStarter - Started component bookie-server.
```
broker-1 (owner) logs when topics seemed to start behaving differently:
```
2025-01-01T20:53:09,314+0000 [bookkeeper-ml-scheduler-OrderedScheduler-0-0] ERROR org.apache.bookkeeper.proto.PerChannelBookieClient - Cannot connect to pulsar-bookie-0.pulsar-bookie.pulsar.svc.cluster.local:3181 as endpoint resolution failed (probably bookie is down) err org.apache.bookkeeper.proto.BookieAddressResolver$BookieIdNotResolvedException: Cannot resolve bookieId pulsar-bookie-0.pulsar-bookie.pulsar.svc.cluster.local:3181, bookie does not exist or it is not running
2025-01-01T20:53:09,314+0000 [bookkeeper-ml-scheduler-OrderedScheduler-0-0] WARN org.apache.bookkeeper.client.PendingAddOp - Failed to write entry (535058, 1) to bookie (0, pulsar-bookie-0.pulsar-bookie.pulsar.svc.cluster.local:3181): Bookie handle is not available
2025-01-01T20:53:09,315+0000 [bookkeeper-ml-scheduler-OrderedScheduler-0-0] INFO org.apache.bookkeeper.client.DefaultBookieAddressResolver - Cannot resolve pulsar-bookie-0.pulsar-bookie.pulsar.svc.cluster.local:3181, bookie is unknown org.apache.bookkeeper.client.BKException$BKBookieHandleNotAvailableException: Bookie handle is not available
2025-01-01T20:53:09,315+0000 [bookkeeper-ml-scheduler-OrderedScheduler-0-0] ERROR org.apache.bookkeeper.proto.PerChannelBookieClient - Cannot connect to pulsar-bookie-0.pulsar-bookie.pulsar.svc.cluster.local:3181 as endpoint resolution failed (probably bookie is down) err org.apache.bookkeeper.proto.BookieAddressResolver$BookieIdNotResolvedException: Cannot resolve bookieId pulsar-bookie-0.pulsar-bookie.pulsar.svc.cluster.local:3181, bookie does not exist or it is not running
2025-01-01T20:53:09,315+0000 [bookkeeper-ml-scheduler-OrderedScheduler-0-0] WARN org.apache.bookkeeper.client.PendingAddOp - Failed to write entry (535058, 2) to bookie (0, pulsar-bookie-0.pulsar-bookie.pulsar.svc.cluster.local:3181): Bookie handle is not available
2025-01-01T20:53:09,315+0000 [bookkeeper-ml-scheduler-OrderedScheduler-0-0] INFO org.apache.bookkeeper.client.DefaultBookieAddressResolver - Cannot resolve pulsar-bookie-0.pulsar-bookie.pulsar.svc.cluster.local:3181, bookie is unknown org.apache.bookkeeper.client.BKException$BKBookieHandleNotAvailableException: Bookie handle is not available
2025-01-01T20:53:09,315+0000 [bookkeeper-ml-scheduler-OrderedScheduler-0-0] ERROR org.apache.bookkeeper.proto.PerChannelBookieClient - Cannot connect to pulsar-bookie-0.pulsar-bookie.pulsar.svc.cluster.local:3181 as endpoint resolution failed (probably bookie is down) err org.apache.bookkeeper.proto.BookieAddressResolver$BookieIdNotResolvedException: Cannot resolve bookieId pulsar-bookie-0.pulsar-bookie.pulsar.svc.cluster.local:3181, bookie does not exist or it is not running
2025-01-01T20:53:09,315+0000 [bookkeeper-ml-scheduler-OrderedScheduler-0-0] WARN org.apache.bookkeeper.client.PendingAddOp - Failed to write entry (535058, 3) to bookie (0, pulsar-bookie-0.pulsar-bookie.pulsar.svc.cluster.local:3181): Bookie handle is not available
2025-01-01T20:53:09,314+0000 [BookKeeperClientWorker-OrderedExecutor-1-0] WARN org.apache.bookkeeper.client.LedgerHandle - [EnsembleChange(ledger:535058, change-id:0000000001)][attempt:1] Exception changing ensemble
org.apache.bookkeeper.client.BKException$BKNotEnoughBookiesException: Not enough non-faulty bookies available
at org.apache.bookkeeper.client.RackawareEnsemblePlacementPolicyImpl.selectRandomInternal(RackawareEnsemblePlacementPolicyImpl.java:850) ~[org.apache.bookkeeper-bookkeeper-server-4.17.1.jar:4.17.1]
at org.apache.bookkeeper.client.RackawareEnsemblePlacementPolicyImpl.selectRandom(RackawareEnsemblePlacementPolicyImpl.java:770) ~[org.apache.bookkeeper-bookkeeper-server-4.17.1.jar:4.17.1]
at org.apache.bookkeeper.client.RackawareEnsemblePlacementPolicyImpl.selectFromNetworkLocation(RackawareEnsemblePlacementPolicyImpl.java:659) ~[org.apache.bookkeeper-bookkeeper-server-4.17.1.jar:4.17.1]
at org.apache.bookkeeper.client.RackawareEnsemblePlacementPolicy.selectFromNetworkLocation(RackawareEnsemblePlacementPolicy.java:225) ~[org.apache.bookkeeper-bookkeeper-server-4.17.1.jar:4.17.1]
at org.apache.bookkeeper.client.RackawareEnsemblePlacementPolicyImpl.selectFromNetworkLocation(RackawareEnsemblePlacementPolicyImpl.java:619) ~[org.apache.bookkeeper-bookkeeper-server-4.17.1.jar:4.17.1]
at org.apache.bookkeeper.client.RackawareEnsemblePlacementPolicy.selectFromNetworkLocation(RackawareEnsemblePlacementPolicy.java:246) ~[org.apache.bookkeeper-bookkeeper-server-4.17.1.jar:4.17.1]
at org.apache.bookkeeper.client.RackawareEnsemblePlacementPolicyImpl.replaceBookie(RackawareEnsemblePlacementPolicyImpl.java:542) ~[org.apache.bookkeeper-bookkeeper-server-4.17.1.jar:4.17.1]
at org.apache.bookkeeper.client.RackawareEnsemblePlacementPolicy.replaceBookie(RackawareEnsemblePlacementPolicy.java:139) ~[org.apache.bookkeeper-bookkeeper-server-4.17.1.jar:4.17.1]
at org.apache.bookkeeper.client.BookieWatcherImpl.replaceBookie(BookieWatcherImpl.java:333) ~[org.apache.bookkeeper-bookkeeper-server-4.17.1.jar:4.17.1]
at org.apache.bookkeeper.client.EnsembleUtils.replaceBookiesInEnsemble(EnsembleUtils.java:69) ~[org.apache.bookkeeper-bookkeeper-server-4.17.1.jar:4.17.1]
at org.apache.bookkeeper.client.LedgerHandle.lambda$ensembleChangeLoop$11(LedgerHandle.java:2222) ~[org.apache.bookkeeper-bookkeeper-server-4.17.1.jar:4.17.1]
at org.apache.bookkeeper.client.MetadataUpdateLoop.writeLoop(MetadataUpdateLoop.java:133) ~[org.apache.bookkeeper-bookkeeper-server-4.17.1.jar:4.17.1]
at org.apache.bookkeeper.client.MetadataUpdateLoop.run(MetadataUpdateLoop.java:120) ~[org.apache.bookkeeper-bookkeeper-server-4.17.1.jar:4.17.1]
at org.apache.bookkeeper.client.LedgerHandle.ensembleChangeLoop(LedgerHandle.java:2241) ~[org.apache.bookkeeper-bookkeeper-server-4.17.1.jar:4.17.1]
at org.apache.bookkeeper.client.LedgerHandle.handleBookieFailure(LedgerHandle.java:2189) ~[org.apache.bookkeeper-bookkeeper-server-4.17.1.jar:4.17.1]
at org.apache.bookkeeper.client.PendingAddOp.writeComplete(PendingAddOp.java:353) ~[org.apache.bookkeeper-bookkeeper-server-4.17.1.jar:4.17.1]
at org.apache.bookkeeper.proto.BookieClientImpl.completeAdd(BookieClientImpl.java:287) ~[org.apache.bookkeeper-bookkeeper-server-4.17.1.jar:4.17.1]
at org.apache.bookkeeper.proto.BookieClientImpl.access$000(BookieClientImpl.java:79) ~[org.apache.bookkeeper-bookkeeper-server-4.17.1.jar:4.17.1]
at org.apache.bookkeeper.proto.BookieClientImpl$ChannelReadyForAddEntryCallback.operationComplete(BookieClientImpl.java:414) ~[org.apache.bookkeeper-bookkeeper-server-4.17.1.jar:4.17.1]
at org.apache.bookkeeper.proto.BookieClientImpl$ChannelReadyForAddEntryCallback.operationComplete(BookieClientImpl.java:373) ~[org.apache.bookkeeper-bookkeeper-server-4.17.1.jar:4.17.1]
at org.apache.bookkeeper.proto.PerChannelBookieClient$ConnectionFutureListener.operationComplete(PerChannelBookieClient.java:2687) ~[org.apache.bookkeeper-bookkeeper-server-4.17.1.jar:4.17.1]
at org.apache.bookkeeper.proto.PerChannelBookieClient$ConnectionFutureListener.operationComplete(PerChannelBookieClient.java:2592) ~[org.apache.bookkeeper-bookkeeper-server-4.17.1.jar:4.17.1]
at org.apache.bookkeeper.proto.PerChannelBookieClient.processBookieNotResolvedError(PerChannelBookieClient.java:336) ~[org.apache.bookkeeper-bookkeeper-server-4.17.1.jar:4.17.1]
at org.apache.bookkeeper.proto.PerChannelBookieClient.connect(PerChannelBookieClient.java:545) ~[org.apache.bookkeeper-bookkeeper-server-4.17.1.jar:4.17.1]
at org.apache.bookkeeper.proto.PerChannelBookieClient.connectIfNeededAndDoOp(PerChannelBookieClient.java:679) ~[org.apache.bookkeeper-bookkeeper-server-4.17.1.jar:4.17.1]
at org.apache.bookkeeper.proto.DefaultPerChannelBookieClientPool.obtain(DefaultPerChannelBookieClientPool.java:120) ~[org.apache.bookkeeper-bookkeeper-server-4.17.1.jar:4.17.1]
at org.apache.bookkeeper.proto.DefaultPerChannelBookieClientPool.obtain(DefaultPerChannelBookieClientPool.java:115) ~[org.apache.bookkeeper-bookkeeper-server-4.17.1.jar:4.17.1]
at org.apache.bookkeeper.proto.BookieClientImpl.addEntry(BookieClientImpl.java:312) ~[org.apache.bookkeeper-bookkeeper-server-4.17.1.jar:4.17.1]
at org.apache.bookkeeper.client.PendingAddOp.sendWriteRequest(PendingAddOp.java:150) ~[org.apache.bookkeeper-bookkeeper-server-4.17.1.jar:4.17.1]
at org.apache.bookkeeper.client.PendingAddOp.initiate(PendingAddOp.java:258) ~[org.apache.bookkeeper-bookkeeper-server-4.17.1.jar:4.17.1]
at org.apache.bookkeeper.client.LedgerHandle.doAsyncAddEntry(LedgerHandle.java:1640) ~[org.apache.bookkeeper-bookkeeper-server-4.17.1.jar:4.17.1]
at org.apache.bookkeeper.client.LedgerHandle.asyncAddEntry(LedgerHandle.java:1338) ~[org.apache.bookkeeper-bookkeeper-server-4.17.1.jar:4.17.1]
at org.apache.bookkeeper.mledger.impl.OpAddEntry.initiate(OpAddEntry.java:153) ~[org.apache.pulsar-managed-ledger-4.0.1.jar:4.0.1]
at org.apache.bookkeeper.mledger.impl.ManagedLedgerImpl.updateLedgersIdsComplete(ManagedLedgerImpl.java:1736) ~[org.apache.pulsar-managed-ledger-4.0.1.jar:4.0.1]
at org.apache.bookkeeper.mledger.impl.ManagedLedgerImpl$14.operationComplete(ManagedLedgerImpl.java:1606) ~[org.apache.pulsar-managed-ledger-4.0.1.jar:4.0.1]
at org.apache.bookkeeper.mledger.impl.ManagedLedgerImpl$14.operationComplete(ManagedLedgerImpl.java:1592) ~[org.apache.pulsar-managed-ledger-4.0.1.jar:4.0.1]
at org.apache.bookkeeper.mledger.impl.MetaStoreImpl.lambda$asyncUpdateLedgerIds$7(MetaStoreImpl.java:187) ~[org.apache.pulsar-managed-ledger-4.0.1.jar:4.0.1]
at java.base/java.util.concurrent.CompletableFuture$UniAccept.tryFire(Unknown Source) ~[?:?]
at java.base/java.util.concurrent.CompletableFuture$Completion.run(Unknown Source) ~[?:?]
at org.apache.bookkeeper.common.util.OrderedExecutor$TimedRunnable.run(OrderedExecutor.java:203) ~[org.apache.bookkeeper-bookkeeper-common-4.17.1.jar:4.17.1]
at org.apache.bookkeeper.common.util.SingleThreadSafeScheduledExecutorService$SafeRunnable.run(SingleThreadSafeScheduledExecutorService.java:46) ~[org.apache.bookkeeper-bookkeeper-common-4.17.1.jar:4.17.1]
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) ~[?:?]
at java.base/java.util.concurrent.FutureTask.run(Unknown Source) ~[?:?]
at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) ~[?:?]
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) ~[?:?]
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) ~[?:?]
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ~[io.netty-netty-common-4.1.115.Final.jar:4.1.115.Final]
at java.base/java.lang.Thread.run(Unknown Source) [?:?]
2025-01-01T20:53:09,315+0000 [BookKeeperClientWorker-OrderedExecutor-1-0] ERROR org.apache.bookkeeper.client.LedgerHandle - Closing ledger 535058 due to NotEnoughBookiesException: Not enough non-faulty bookies available
2025-01-01T20:53:09,315+0000 [BookKeeperClientWorker-OrderedExecutor-1-0] ERROR org.apache.bookkeeper.client.PendingAddOp - Write of ledger entry to quorum failed: L535058 E0
2025-01-01T20:53:09,315+0000 [BookKeeperClientWorker-OrderedExecutor-1-0] ERROR org.apache.bookkeeper.client.PendingAddOp - Write of ledger entry to quorum failed: L535058 E1
2025-01-01T20:53:09,315+0000 [BookKeeperClientWorker-OrderedExecutor-1-0] ERROR org.apache.bookkeeper.client.PendingAddOp - Write of ledger entry to quorum failed: L535058 E2
2025-01-01T20:53:09,315+0000 [BookKeeperClientWorker-OrderedExecutor-1-0] ERROR org.apache.bookkeeper.client.PendingAddOp - Write of ledger entry to quorum failed: L535058 E3
2025-01-01T20:53:09,315+0000 [metadata-store-12-1] INFO org.apache.pulsar.metadata.bookkeeper.PulsarRegistrationClient - Bookie pulsar-bookie-0.pulsar-bookie.pulsar.svc.cluster.local:3181 created. path: /ledgers/available/pulsar-bookie-0.pulsar-bookie.pulsar.svc.cluster.local:3181
2025-01-01T20:53:09,315+0000 [metadata-store-12-1] INFO org.apache.pulsar.metadata.bookkeeper.PulsarRegistrationClient - Bookie pulsar-bookie-0.pulsar-bookie.pulsar.svc.cluster.local:3181 created. path: /ledgers/available/pulsar-bookie-0.pulsar-bookie.pulsar.svc.cluster.local:3181
2025-01-01T20:53:09,315+0000 [BookKeeperClientWorker-OrderedExecutor-0-0] ERROR org.apache.pulsar.broker.transaction.buffer.impl.TopicTransactionBuffer - Failed to append buffer to txn (11,2578)
org.apache.bookkeeper.mledger.ManagedLedgerException: Not enough non-faulty bookies available error code: -6
2025-01-01T20:53:09,317+0000 [BookKeeperClientWorker-OrderedExecutor-0-0] INFO org.apache.pulsar.broker.service.Producer - Disconnecting producer: Producer{topic=PersistentTopic{topic=persistent://a7u/ddbs/a7u_transactions_sanitized}, client=[id: 0xd93fe9b7, L:/10.11.217.164:6650 - R:/10.11.219.27:58438] [SR:10.11.219.27, state:Connected], producerName=DynamoDBStreamProcessor-a7u_transactions-a7u_transactions_sanitized - 0d7bfcbc-724e-4674-aa90-8a48ec8d861a, producerId=0}, assignedBrokerLookupData: Optional.empty
2025-01-01T20:53:09,318+0000 [metadata-store-12-1] INFO org.apache.pulsar.metadata.impl.AbstractMetadataStore - Deleted path: /ledgers/00/0053/L5053 (v. Optional.empty)
2025-01-01T20:53:09,318+0000 [metadata-store-12-1] INFO org.apache.pulsar.metadata.impl.AbstractMetadataStore - Deleted path: /ledgers/00/0053/L5052 (v. Optional.empty)
2025-01-01T20:53:09,319+0000 [BookKeeperClientWorker-OrderedExecutor-0-0] WARN org.apache.pulsar.broker.service.persistent.PersistentTopic - [persistent://a7u/ddbs/a7u_transactions_sanitized] Failed to persist msg in store: Not enough non-faulty bookies available error code: -6
2025-01-01T20:53:09,319+0000 [BookKeeperClientWorker-OrderedExecutor-0-0] WARN org.apache.pulsar.broker.service.persistent.PersistentTopic - [persistent://a7u/ddbs/a7u_transactions_sanitized] Failed to persist msg in store: Not enough non-faulty bookies available error code: -6
2025-01-01T20:53:09,319+0000 [BookKeeperClientWorker-OrderedExecutor-0-0] WARN org.apache.pulsar.broker.service.persistent.PersistentTopic - [persistent://a7u/ddbs/a7u_transactions_sanitized] Failed to persist msg in store: Not enough non-faulty bookies available error code: -6
2025-01-01T20:53:09,319+0000 [BookKeeperClientWorker-OrderedExecutor-0-0] WARN org.apache.pulsar.broker.service.persistent.PersistentTopic - [persistent://a7u/ddbs/a7u_transactions_sanitized] Failed to persist msg in store: Not enough non-faulty bookies available error code: -6
2025-01-01T20:53:09,319+0000 [BookKeeperClientWorker-OrderedExecutor-0-0] WARN org.apache.pulsar.broker.service.persistent.PersistentTopic - [persistent://a7u/ddbs/a7u_transactions_sanitized] Failed to persist msg in store: Not enough non-faulty bookies available error code: -6
2025-01-01T20:53:09,320+0000 [BookKeeperClientWorker-OrderedExecutor-0-0] WARN org.apache.pulsar.broker.service.persistent.PersistentTopic - [persistent://a7u/ddbs/a7u_transactions_sanitized] Failed to persist msg in store: Not enough non-faulty bookies available error code: -6
2025-01-01T20:53:09,320+0000 [BookKeeperClientWorker-OrderedExecutor-0-0] WARN org.apache.pulsar.broker.service.persistent.PersistentTopic - [persistent://a7u/ddbs/a7u_transactions_sanitized] Failed to persist msg in store: Not enough non-faulty bookies available error code: -6
2025-01-01T20:53:09,320+0000 [BookKeeperClientWorker-OrderedExecutor-0-0] WARN org.apache.pulsar.broker.service.persistent.PersistentTopic - [persistent://a7u/ddbs/a7u_transactions_sanitized] Failed to persist msg in store: Not enough non-faulty bookies available error code: -6
2025-01-01T20:53:09,320+0000 [BookKeeperClientWorker-OrderedExecutor-0-0] WARN org.apache.pulsar.broker.service.persistent.PersistentTopic - [persistent://a7u/ddbs/a7u_transactions_sanitized] Failed to persist msg in store: Not enough non-faulty bookies available error code: -6
2025-01-01T20:53:09,320+0000 [BookKeeperClientWorker-OrderedExecutor-0-0] WARN org.apache.pulsar.broker.service.persistent.PersistentTopic - [persistent://a7u/ddbs/a7u_transactions_sanitized] Failed to persist msg in store: Not enough non-faulty bookies available error code: -6
2025-01-01T20:53:09,320+0000 [BookKeeperClientWorker-OrderedExecutor-0-0] WARN org.apache.pulsar.broker.service.persistent.PersistentTopic - [persistent://a7u/ddbs/a7u_transactions_sanitized] Failed to persist msg in store: Not enough non-faulty bookies available error code: -6
2025-01-01T20:53:09,320+0000 [BookKeeperClientWorker-OrderedExecutor-0-0] WARN org.apache.pulsar.broker.service.persistent.PersistentTopic - [persistent://a7u/ddbs/a7u_transactions_sanitized] Failed to persist msg in store: Not enough non-faulty bookies available error code: -6
2025-01-01T20:53:09,320+0000 [BookKeeperClientWorker-OrderedExecutor-0-0] WARN org.apache.pulsar.broker.service.persistent.PersistentTopic - [persistent://a7u/ddbs/a7u_transactions_sanitized] Failed to persist msg in store: Not enough non-faulty bookies available error code: -6
2025-01-01T20:53:09,320+0000 [BookKeeperClientWorker-OrderedExecutor-0-0] WARN org.apache.pulsar.broker.service.persistent.PersistentTopic - [persistent://a7u/ddbs/a7u_transactions_sanitized] Failed to persist msg in store: Not enough non-faulty bookies available error code: -6
2025-01-01T20:53:09,320+0000 [BookKeeperClientWorker-OrderedExecutor-0-0] WARN org.apache.pulsar.broker.service.persistent.PersistentTopic - [persistent://a7u/ddbs/a7u_transactions_sanitized] Failed to persist msg in store: Not enough non-faulty bookies available error code: -6
2025-01-01T20:53:09,320+0000 [BookKeeperClientWorker-OrderedExecutor-0-0] WARN org.apache.pulsar.broker.service.persistent.PersistentTopic - [persistent://a7u/ddbs/a7u_transactions_sanitized] Failed to persist msg in store: Not enough non-faulty bookies available error code: -6
2025-01-01T20:53:09,320+0000 [BookKeeperClientWorker-OrderedExecutor-0-0] WARN org.apache.pulsar.broker.service.persistent.PersistentTopic - [persistent://a7u/ddbs/a7u_transactions_sanitized] Failed to persist msg in store: Not enough non-faulty bookies available error code: -6
2025-01-01T20:53:09,320+0000 [BookKeeperClientWorker-OrderedExecutor-0-0] WARN org.apache.pulsar.broker.service.persistent.PersistentTopic - [persistent://a7u/ddbs/a7u_transactions_sanitized] Failed to persist msg in store: Not enough non-faulty bookies available error code: -6
2025-01-01T20:53:09,320+0000 [BookKeeperClientWorker-OrderedExecutor-0-0] WARN org.apache.pulsar.broker.service.persistent.PersistentTopic - [persistent://a7u/ddbs/a7u_transactions_sanitized] Failed to persist msg in store: Not enough non-faulty bookies available error code: -6
2025-01-01T20:53:09,320+0000 [BookKeeperClientWorker-OrderedExecutor-0-0] WARN org.apache.pulsar.broker.service.persistent.PersistentTopic - [persistent://a7u/ddbs/a7u_transactions_sanitized] Failed to persist msg in store: Not enough non-faulty bookies available error code: -6
2025-01-01T20:53:09,320+0000 [BookKeeperClientWorker-OrderedExecutor-0-0] WARN org.apache.pulsar.broker.service.persistent.PersistentTopic - [persistent://a7u/ddbs/a7u_transactions_sanitized] Failed to persist msg in store: Not enough non-faulty bookies available error code: -6
2025-01-01T20:53:09,320+0000 [BookKeeperClientWorker-OrderedExecutor-0-0] WARN org.apache.pulsar.broker.service.persistent.PersistentTopic - [persistent://a7u/ddbs/a7u_transactions_sanitized] Failed to persist msg in store: Not enough non-faulty bookies available error code: -6
2025-01-01T20:53:09,320+0000 [BookKeeperClientWorker-OrderedExecutor-0-0] WARN org.apache.pulsar.broker.service.persistent.PersistentTopic - [persistent://a7u/ddbs/a7u_transactions_sanitized] Failed to persist msg in store: Not enough non-faulty bookies available error code: -6
2025-01-01T20:53:09,320+0000 [BookKeeperClientWorker-OrderedExecutor-0-0] WARN org.apache.pulsar.broker.service.persistent.PersistentTopic - [persistent://a7u/ddbs/a7u_transactions_sanitized] Failed to persist msg in store: Not enough non-faulty bookies available error code: -6
2025-01-01T20:53:09,320+0000 [BookKeeperClientWorker-OrderedExecutor-0-0] WARN org.apache.pulsar.broker.service.persistent.PersistentTopic - [persistent://a7u/ddbs/a7u_transactions_sanitized] Failed to persist msg in store: Not enough non-faulty bookies available error code: -6
2025-01-01T20:53:09,321+0000 [BookKeeperClientWorker-OrderedExecutor-0-0] WARN org.apache.pulsar.broker.service.persistent.PersistentTopic - [persistent://a7u/ddbs/a7u_transactions_sanitized] Failed to persist msg in store: Not enough non-faulty bookies available error code: -6
2025-01-01T20:53:09,322+0000 [BookKeeperClientWorker-OrderedExecutor-0-0] WARN org.apache.pulsar.broker.service.persistent.PersistentTopic - [persistent://a7u/ddbs/a7u_transactions_sanitized] Failed to persist msg in store: Not enough non-faulty bookies available error code: -6
2025-01-01T20:53:09,322+0000 [BookKeeperClientWorker-OrderedExecutor-0-0] WARN org.apache.pulsar.broker.service.persistent.PersistentTopic - [persistent://a7u/ddbs/a7u_transactions_sanitized] Failed to persist msg in store: Not enough non-faulty bookies available error code: -6
2025-01-01T20:53:09,322+0000 [BookKeeperClientWorker-OrderedExecutor-0-0] INFO org.apache.pulsar.broker.service.persistent.PersistentTopic - [persistent://a7u/ddbs/a7u_transactions_sanitized] Un-fencing topic...
2025-01-01T20:53:09,322+0000 [BookKeeperClientWorker-OrderedExecutor-0-0] INFO org.apache.bookkeeper.mledger.impl.ManagedLedgerImpl - [a7u/ddbs/persistent/a7u_transactions_sanitized] Managed ledger is now ready to accept writes again
2025-01-01T20:53:09,322+0000 [BookKeeperClientWorker-OrderedExecutor-0-0] WARN org.apache.pulsar.broker.service.persistent.PersistentTopic - [persistent://a7u/ddbs/a7u_transactions_sanitized] Failed to persist msg in store: Not enough non-faulty bookies available error code: -6
2025-01-01T20:53:09,322+0000 [BookKeeperClientWorker-OrderedExecutor-0-0] INFO org.apache.bookkeeper.mledger.impl.ManagedLedgerImpl - [a7u/ddbs/persistent/a7u_transactions] Creating a new ledger after closed 535058
2025-01-01T20:53:09,322+0000 [BookKeeperClientWorker-OrderedExecutor-0-0] INFO org.apache.bookkeeper.mledger.impl.ManagedLedgerImpl - [a7u/ddbs/persistent/a7u_transactions_sanitized] Delete complete for empty ledger 535053. rc=0
2025-01-01T20:53:09,322+0000 [BookKeeperClientWorker-OrderedExecutor-0-0] INFO org.apache.bookkeeper.mledger.impl.ManagedLedgerImpl - [a7u/ddbs/persistent/a7u_transactions] Delete complete for empty ledger 535052. rc=0
2025-01-01T20:53:09,322+0000 [BookKeeperClientWorker-OrderedExecutor-0-0] WARN org.apache.bookkeeper.client.RackawareEnsemblePlacementPolicyImpl - Failed to find 1 bookies : excludeBookies [<Bookie:pulsar-bookie-2.pulsar-bookie.pulsar.svc.cluster.local:3181>, <Bookie:pulsar-bookie-3.pulsar-bookie.pulsar.svc.cluster.local:3181>], allBookies [<Bookie:pulsar-bookie-2.pulsar-bookie.pulsar.svc.cluster.local:3181>, <Bookie:pulsar-bookie-3.pulsar-bookie.pulsar.svc.cluster.local:3181>].
2025-01-01T20:53:09,322+0000 [BookKeeperClientWorker-OrderedExecutor-0-0] WARN org.apache.bookkeeper.client.RackawareEnsemblePlacementPolicyImpl - Failed to find 1 bookies : excludeBookies [<Bookie:pulsar-bookie-2.pulsar-bookie.pulsar.svc.cluster.local:3181>, <Bookie:pulsar-bookie-3.pulsar-bookie.pulsar.svc.cluster.local:3181>], allBookies [<Bookie:pulsar-bookie-2.pulsar-bookie.pulsar.svc.cluster.local:3181>, <Bookie:pulsar-bookie-3.pulsar-bookie.pulsar.svc.cluster.local:3181>].
2025-01-01T20:53:09,322+0000 [BookKeeperClientWorker-OrderedExecutor-0-0] ERROR org.apache.bookkeeper.client.LedgerCreateOp - Not enough bookies to create ledger with ensembleSize=3, writeQuorumSize=3 and ackQuorumSize=2
2025-01-01T20:53:09,322+0000 [BookKeeperClientWorker-OrderedExecutor-0-0] ERROR org.apache.bookkeeper.mledger.impl.ManagedLedgerImpl - [a7u/ddbs/persistent/a7u_transactions] Error creating ledger rc=-6 Not enough non-faulty bookies available
2025-01-01T20:53:09,322+0000 [BookKeeperClientWorker-OrderedExecutor-0-0] INFO org.apache.pulsar.broker.service.Producer - Disconnecting producer: Producer{topic=PersistentTopic{topic=persistent://a7u/ddbs/a7u_transactions}, client=[id: 0xe3c4b8be, L:/10.11.217.164:6650 - R:/10.11.233.64:37306] [SR:10.11.233.64, state:Connected], producerName=pulsar-313-191, producerId=0}, assignedBrokerLookupData: Optional.empty
2025-01-01T20:53:09,322+0000 [BookKeeperClientWorker-OrderedExecutor-0-0] INFO org.apache.pulsar.broker.service.Producer - Disconnecting producer: Producer{topic=PersistentTopic{topic=persistent://a7u/ddbs/a7u_transactions}, client=[id: 0x320c7da6, L:/10.11.217.164:6650 - R:/10.11.219.27:57008] [SR:10.11.219.27, state:Connected], producerName=pulsar-313-193, producerId=0}, assignedBrokerLookupData: Optional.empty
2025-01-01T20:53:09,322+0000 [BookKeeperClientWorker-OrderedExecutor-0-0] INFO org.apache.pulsar.broker.service.Producer - Disconnecting producer: Producer{topic=PersistentTopic{topic=persistent://a7u/ddbs/a7u_transactions}, client=[id: 0x96ac0dc3, L:/10.11.217.164:6650 - R:/10.11.233.64:37318] [SR:10.11.233.64, state:Connected], producerName=pulsar-313-192, producerId=0}, assignedBrokerLookupData: Optional.empty
2025-01-01T20:53:09,322+0000 [BookKeeperClientWorker-OrderedExecutor-0-0] INFO org.apache.pulsar.broker.service.Producer - Disconnecting producer: Producer{topic=PersistentTopic{topic=persistent://a7u/ddbs/a7u_transactions}, client=[id: 0x89f930ab, L:/10.11.217.164:6650 - R:/10.11.219.27:57010] [SR:10.11.219.27, state:Connected], producerName=pulsar-313-194, producerId=0}, assignedBrokerLookupData: Optional.empty
2025-01-01T20:53:09,322+0000 [BookKeeperClientWorker-OrderedExecutor-0-0] WARN org.apache.pulsar.broker.service.persistent.PersistentTopic - [persistent://a7u/ddbs/a7u_transactions] Failed to persist msg in store: Not enough non-faulty bookies available error code: -6
2025-01-01T20:53:09,322+0000 [BookKeeperClientWorker-OrderedExecutor-0-0] WARN org.apache.pulsar.broker.service.persistent.PersistentTopic - [persistent://a7u/ddbs/a7u_transactions] Failed to persist msg in store: Not enough non-faulty bookies available error code: -6
2025-01-01T20:53:09,322+0000 [BookKeeperClientWorker-OrderedExecutor-0-0] WARN org.apache.pulsar.broker.service.persistent.PersistentTopic - [persistent://a7u/ddbs/a7u_transactions] Failed to persist msg in store: Not enough non-faulty bookies available error code: -6
2025-01-01T20:53:09,322+0000 [BookKeeperClientWorker-OrderedExecutor-0-0] WARN org.apache.pulsar.broker.service.persistent.PersistentTopic - [persistent://a7u/ddbs/a7u_transactions] Failed to persist msg in store: Not enough non-faulty bookies available error code: -6
2025-01-01T20:53:09,323+0000 [pulsar-io-5-2] INFO org.apache.pulsar.broker.service.persistent.PersistentTopic - [persistent://a7u/ddbs/a7u_transactions] Un-fencing topic...
2025-01-01T20:53:09,323+0000 [pulsar-io-5-2] INFO org.apache.bookkeeper.mledger.impl.ManagedLedgerImpl - [a7u/ddbs/persistent/a7u_transactions] Managed ledger is now ready to accept writes again
2025-01-01T20:53:09,324+0000 [pulsar-io-5-1] INFO org.apache.pulsar.broker.service.ServerCnx - Closed connection from /10.11.233.64:37318
2025-01-01T20:53:09,325+0000 [pulsar-io-5-2] INFO org.apache.pulsar.broker.service.ServerCnx - Closed connection from /10.11.219.27:57008
2025-01-01T20:53:09,325+0000 [pulsar-io-5-3] INFO org.apache.pulsar.broker.service.ServerCnx - Closed connection from /10.11.219.27:57010
2025-01-01T20:53:09,325+0000 [metadata-store-12-1] INFO org.apache.pulsar.metadata.bookkeeper.PulsarRegistrationClient - Update BookieInfoCache (writable bookie) pulsar-bookie-0.pulsar-bookie.pulsar.svc.cluster.local:3181 -> BookieServiceInfo{properties={}, endpoints=[EndpointInfo{id=httpserver, port=8000, host=0.0.0.0, protocol=http, auth=[], extensions=[]}, EndpointInfo{id=bookie, port=3181, host=pulsar-bookie-0.pulsar-bookie.pulsar.svc.cluster.local, protocol=bookie-rpc, auth=[], extensions=[]}]}
2025-01-01T20:53:09,326+0000 [metadata-store-12-1] INFO org.apache.pulsar.metadata.bookkeeper.PulsarRegistrationClient - Update BookieInfoCache (writable bookie) pulsar-bookie-0.pulsar-bookie.pulsar.svc.cluster.local:3181 -> BookieServiceInfo{properties={}, endpoints=[EndpointInfo{id=httpserver, port=8000, host=0.0.0.0, protocol=http, auth=[], extensions=[]}, EndpointInfo{id=bookie, port=3181, host=pulsar-bookie-0.pulsar-bookie.pulsar.svc.cluster.local, protocol=bookie-rpc, auth=[], extensions=[]}]}
2025-01-01T20:53:09,326+0000 [pulsar-registration-client-38-1] WARN org.apache.bookkeeper.client.TopologyAwareEnsemblePlacementPolicy - Failed to resolve network location for pulsar-bookie-0.pulsar-bookie.pulsar.svc.cluster.local, using default rack for it : /default-rack.
2025-01-01T20:53:09,326+0000 [pulsar-registration-client-19-1] WARN org.apache.bookkeeper.client.TopologyAwareEnsemblePlacementPolicy - Failed to resolve network location for pulsar-bookie-0.pulsar-bookie.pulsar.svc.cluster.local, using default rack for it : /default-rack.
2025-01-01T20:53:09,326+0000 [pulsar-registration-client-38-1] INFO org.apache.bookkeeper.net.NetworkTopologyImpl - Adding a new node: /default-rack/pulsar-bookie-0.pulsar-bookie.pulsar.svc.cluster.local:3181
2025-01-01T20:53:09,326+0000 [pulsar-registration-client-19-1] INFO org.apache.bookkeeper.net.NetworkTopologyImpl - Adding a new node: /default-rack/pulsar-bookie-0.pulsar-bookie.pulsar.svc.cluster.local:3181
2025-01-01T20:53:09,328+0000 [pulsar-io-5-2] INFO org.apache.pulsar.broker.service.ServerCnx - Closed connection from /10.11.219.27:58438
2025-01-01T20:53:09,463+0000 [pulsar-io-5-3] INFO org.apache.pulsar.broker.service.ServerCnx - [/10.11.219.27:57018] connected role=system:serviceaccount:pulsar:pulsar-proxy and originalAuthRole=system:serviceaccount:a7u:ddbss-records-processor using authMethod=token, clientVersion=Pulsar-Java-v3.0.0, clientProtocolVersion=21, proxyVersion=4.0.1
2025-01-01T20:53:09,466+0000 [broker-topic-workers-OrderedExecutor-0-0] INFO org.apache.pulsar.broker.service.ServerCnx - [/10.11.219.27:57018] Created new producer: Producer{topic=PersistentTopic{topic=persistent://a7u/ddbs/a7u_transactions_sanitized}, client=[id: 0x3133e9f2, L:/10.11.217.164:6650 - R:/10.11.219.27:57018] [SR:10.11.219.27, state:Connected], producerName=DynamoDBStreamProcessor-a7u_transactions-a7u_transactions_sanitized - 0d7bfcbc-724e-4674-aa90-8a48ec8d861a, producerId=0}, role: system:serviceaccount:a7u:ddbss-records-processor
2025-01-01T20:53:09,469+0000 [pulsar-io-5-3] INFO org.apache.pulsar.broker.service.Producer - Disconnecting producer: Producer{topic=PersistentTopic{topic=persistent://a7u/ddbs/a7u_transactions_sanitized}, client=[id: 0x3133e9f2, L:/10.11.217.164:6650 - R:/10.11.219.27:57018] [SR:10.11.219.27, state:Connected], producerName=DynamoDBStreamProcessor-a7u_transactions-a7u_transactions_sanitized - 0d7bfcbc-724e-4674-aa90-8a48ec8d861a, producerId=0}, assignedBrokerLookupData: Optional.empty
2025-01-01T20:53:09,469+0000 [pulsar-io-5-3] WARN org.apache.pulsar.broker.service.persistent.PersistentTopic - [persistent://a7u/ddbs/a7u_transactions_sanitized] Failed to persist msg in store: Not enough non-faulty bookies available error code: -6
2025-01-01T20:53:09,469+0000 [pulsar-io-5-3] INFO org.apache.pulsar.broker.service.persistent.PersistentTopic - [persistent://a7u/ddbs/a7u_transactions_sanitized] Un-fencing topic...
2025-01-01T20:53:09,474+0000 [pulsar-io-5-3] INFO org.apache.pulsar.broker.service.ServerCnx - Closed connection from /10.11.219.27:57018
2025-01-01T20:53:09,482+0000 [pulsar-io-5-4] INFO org.apache.pulsar.broker.service.ServerCnx - [/10.11.230.148:60912] connected role=system:serviceaccount:pulsar:pulsar-proxy and originalAuthRole=system:serviceaccount:a7u:ddbss-records-processor using authMethod=token, clientVersion=Pulsar-Java-v3.0.0, clientProtocolVersion=21, proxyVersion=4.0.1
2025-01-01T20:53:09,550+0000 [pulsar-io-5-3] INFO org.apache.pulsar.broker.service.ServerCnx - [/10.11.219.27:57036] connected role=system:serviceaccount:pulsar:pulsar-proxy and originalAuthRole=a7u-ddbss using authMethod=token, clientVersion=Pulsar-CPP-v3.5.1, clientProtocolVersion=20, proxyVersion=4.0.1
2025-01-01T20:53:09,553+0000 [broker-topic-workers-OrderedExecutor-0-0] INFO org.apache.pulsar.broker.service.ServerCnx - [/10.11.219.27:57036] Created new producer: Producer{topic=PersistentTopic{topic=persistent://a7u/ddbs/a7u_transactions}, client=[id: 0x90f74dd6, L:/10.11.217.164:6650 - R:/10.11.219.27:57036] [SR:10.11.219.27, state:Connected], producerName=pulsar-313-193, producerId=0}, role: a7u-ddbss
2025-01-01T20:53:09,554+0000 [pulsar-io-5-1] INFO org.apache.pulsar.broker.service.ServerCnx - [/10.11.219.27:57030] connected role=system:serviceaccount:pulsar:pulsar-proxy and originalAuthRole=a7u-ddbss using authMethod=token, clientVersion=Pulsar-CPP-v3.5.1, clientProtocolVersion=20, proxyVersion=4.0.1
2025-01-01T20:53:09,555+0000 [pulsar-io-5-2] INFO org.apache.pulsar.broker.service.ServerCnx - [/10.11.233.64:37332] connected role=system:serviceaccount:pulsar:pulsar-proxy and originalAuthRole=a7u-ddbss using authMethod=token, clientVersion=Pulsar-CPP-v3.5.1, clientProtocolVersion=20, proxyVersion=4.0.1
2025-01-01T20:53:09,555+0000 [BookKeeperClientWorker-OrderedExecutor-0-0] INFO org.apache.bookkeeper.mledger.impl.ManagedLedgerImpl - [a7u/ddbs/persistent/a7u_transactions] Creating a new ledger
2025-01-01T20:53:09,555+0000 [BookKeeperClientWorker-OrderedExecutor-0-0] WARN org.apache.bookkeeper.client.BookieWatcherImpl - New ensemble: [pulsar-bookie-2.pulsar-bookie.pulsar.svc.cluster.local:3181, pulsar-bookie-3.pulsar-bookie.pulsar.svc.cluster.local:3181, pulsar-bookie-0.pulsar-bookie.pulsar.svc.cluster.local:3181] is not adhering to Placement Policy. quarantinedBookies: []
2025-01-01T20:53:09,557+0000 [broker-topic-workers-OrderedExecutor-0-0] INFO org.apache.pulsar.broker.service.ServerCnx - [/10.11.233.64:37332] Created new producer: Producer{topic=PersistentTopic{topic=persistent://a7u/ddbs/a7u_transactions}, client=[id: 0x563b128e, L:/10.11.217.164:6650 - R:/10.11.233.64:37332] [SR:10.11.233.64, state:Connected], producerName=pulsar-313-192, producerId=0}, role: a7u-ddbss
2025-01-01T20:53:09,566+0000 [broker-topic-workers-OrderedExecutor-0-0] INFO org.apache.pulsar.broker.service.ServerCnx - [/10.11.219.27:57030] Created new producer: Producer{topic=PersistentTopic{topic=persistent://a7u/ddbs/a7u_transactions}, client=[id: 0x7c18fd74, L:/10.11.217.164:6650 - R:/10.11.219.27:57030] [SR:10.11.219.27, state:Connected], producerName=pulsar-313-194, producerId=0}, role: a7u-ddbss
2025-01-01T20:53:09,571+0000 [metadata-store-12-1] INFO org.apache.pulsar.metadata.impl.AbstractMetadataStore - Deleting path: /ledgers/idgen/ID-0000535064 (v. Optional.empty)
2025-01-01T20:53:09,575+0000 [broker-topic-workers-OrderedExecutor-0-0] INFO org.apache.pulsar.broker.service.ServerCnx - [/10.11.230.148:60912] Created new producer: Producer{topic=PersistentTopic{topic=persistent://a7u/ddbs/a7u_transactions_sanitized}, client=[id: 0x6e4d4bd8, L:/10.11.217.164:6650 - R:/10.11.230.148:60912] [SR:10.11.230.148, state:Connected], producerName=DynamoDBStreamProcessor-a7u_transactions-a7u_transactions_sanitized - 0d7bfcbc-724e-4674-aa90-8a48ec8d861a, producerId=0}, role: system:serviceaccount:a7u:ddbss-records-processor
2025-01-01T20:53:09,578+0000 [metadata-store-12-1] INFO org.apache.pulsar.metadata.impl.AbstractMetadataStore - Deleted path: /ledgers/idgen/ID-0000535064 (v. Optional.empty)
2025-01-01T20:53:09,578+0000 [metadata-store-12-1] INFO org.apache.bookkeeper.client.LedgerCreateOp - Ensemble: [pulsar-bookie-2.pulsar-bookie.pulsar.svc.cluster.local:3181, pulsar-bookie-3.pulsar-bookie.pulsar.svc.cluster.local:3181, pulsar-bookie-0.pulsar-bookie.pulsar.svc.cluster.local:3181] for ledger: 535064
2025-01-01T20:53:09,578+0000 [BookKeeperClientWorker-OrderedExecutor-0-0] INFO org.apache.bookkeeper.mledger.impl.ManagedLedgerImpl - [a7u/ddbs/persistent/a7u_transactions] Created new ledger 535064
2025-01-01T20:53:09,580+0000 [pulsar-io-5-4] INFO org.apache.pulsar.broker.service.Producer - Disconnecting producer: Producer{topic=PersistentTopic{topic=persistent://a7u/ddbs/a7u_transactions_sanitized}, client=[id: 0x6e4d4bd8, L:/10.11.217.164:6650 - R:/10.11.230.148:60912] [SR:10.11.230.148, state:Connected], producerName=DynamoDBStreamProcessor-a7u_transactions-a7u_transactions_sanitized - 0d7bfcbc-724e-4674-aa90-8a48ec8d861a, producerId=0}, assignedBrokerLookupData: Optional.empty
2025-01-01T20:53:09,580+0000 [pulsar-io-5-4] WARN org.apache.pulsar.broker.service.persistent.PersistentTopic - [persistent://a7u/ddbs/a7u_transactions_sanitized] Failed to persist msg in store: Not enough non-faulty bookies available error code: -6
2025-01-01T20:53:09,580+0000 [pulsar-io-5-4] INFO org.apache.pulsar.broker.service.persistent.PersistentTopic - [persistent://a7u/ddbs/a7u_transactions_sanitized] Un-fencing topic...
2025-01-01T20:53:09,585+0000 [pulsar-io-5-4] INFO org.apache.pulsar.broker.service.ServerCnx - Closed connection from /10.11.230.148:60912
2025-01-01T20:53:09,589+0000 [bookkeeper-ml-scheduler-OrderedScheduler-0-0] INFO org.apache.pulsar.metadata.impl.AbstractMetadataStore - Deleting path: /ledgers/00/0053/L5058 (v. Optional.empty)
2025-01-01T20:53:09,602+0000 [pulsar-io-5-4] INFO org.apache.bookkeeper.proto.PerChannelBookieClient - Successfully connected to bookie: pulsar-bookie-0.pulsar-bookie.pulsar.svc.cluster.local:3181 [id: 0x3c1af6f7, L:/10.11.217.164:49698 - R:pulsar-bookie-0.pulsar-bookie.pulsar.svc.cluster.local/10.11.225.176:3181]
2025-01-01T20:53:09,603+0000 [pulsar-io-5-4] INFO org.apache.bookkeeper.proto.PerChannelBookieClient - connection [id: 0x3c1af6f7, L:/10.11.217.164:49698 - R:pulsar-bookie-0.pulsar-bookie.pulsar.svc.cluster.local/10.11.225.176:3181] authenticated as BookKeeperPrincipal{ANONYMOUS}
2025-01-01T20:53:09,609+0000 [metadata-store-12-1] INFO org.apache.pulsar.metadata.impl.AbstractMetadataStore - Deleted path: /ledgers/00/0053/L5058 (v. Optional.empty)
2025-01-01T20:53:09,609+0000 [BookKeeperClientWorker-OrderedExecutor-1-0] INFO org.apache.bookkeeper.mledger.impl.ManagedLedgerImpl - [a7u/ddbs/persistent/a7u_transactions] Delete complete for empty ledger 535058. rc=0
```
### Are you willing to submit a PR?
- [ ] I'm willing to submit a PR!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug] "Not enough non-faulty bookies available error code: -6" persisting after outage while all bookies are reported as healthy [pulsar]
Posted by "merlimat (via GitHub)" <gi...@apache.org>.
merlimat commented on issue #23807:
URL: https://github.com/apache/pulsar/issues/23807#issuecomment-6380962504
Thanks @mrtworo for the detailed report. The failure is in the BookKeeper client's ensemble replacement (`RackawareEnsemblePlacementPolicyImpl.replaceBookie`), and the logs show the bookie IDs could not be resolved right when the bookies came back. Master has moved to BookKeeper 4.18.2 since then, and @MichalKoziorowski-TomTom reported that a newer Helm chart made the problem go away. Could you check whether it still happens on a currently supported release (4.0.x, 4.2.x or 5.0.x) with the current Helm chart? If it does, please attach the broker logs for the stuck topic from when the bookies returned, plus the broker's `ensemblePlacementPolicy`/rack-awareness settings. @anosulchik, the same applies to your trace: please share the Pulsar version, Helm chart version and a fuller broker log. If there's no update, this issue will be closed in a few weeks.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug] "Not enough non-faulty bookies available error code: -6" persisting after outage while all bookies are reported as healthy [pulsar]
Posted by "mrtworo (via GitHub) via commits" <co...@pulsar.apache.org>.
mrtworo commented on issue #23807:
URL: https://github.com/apache/pulsar/issues/23807#issuecomment-6689578405
Hey @merlimat, thanks for looking into it. We’re closely following new Pulsar releases, I can’t find any logs showing that the problem has recurred in the last 60 days, so it seems safe to consider this case closed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug] "Not enough non-faulty bookies available error code: -6" persisting after outage while all bookies are reported as healthy [pulsar]
Posted by "anosulchik (via GitHub)" <gi...@apache.org>.
anosulchik commented on issue #23807:
URL: https://github.com/apache/pulsar/issues/23807#issuecomment-2887696320
I'm seeing the same problem with latest helm chart and pulsar version:
```
Caused by: [CIRCULAR REFERENCE: javax.ws.rs.InternalServerErrorException: HTTP 500 {"reason":"\n --- An unexpected error occurred in the server ---\n\nMessage: org.apache.bookkeeper.mledger.ManagedLedgerException: Not enough non-faulty bookies available error code: -6\n\nStacktrace:\n\norg.apache.pulsar.broker.service.BrokerServiceException$PersistenceException: org.apache.bookkeeper.mledger.ManagedLedgerException: Not enough non-faulty bookies available error code: -6\n\tat org.apache.pulsar.broker.service.BrokerService$2.openLedgerFailed(BrokerService.java:1899)\n\tat org.apache.bookkeeper.mledger.impl.ManagedLedgerFactoryImpl.lambda$asyncOpen$10(ManagedLedgerFactoryImpl.java:466)\n\tat java.base/java.util.concurrent.CompletableFuture.uniExceptionally(Unknown Source)\n\tat java.base/java.util.concurrent.CompletableFuture$UniExceptionally.tryFire(Unknown Source)\n\tat java.base/java.util.concurrent.CompletableFuture.postComplete(Unknown Source)\n\tat java.base/java.util.concurren
t.CompletableFuture.completeExceptionally(Unknown Source)\n\tat org.apache.bookkeeper.mledger.impl.ManagedLedgerFactoryImpl$2.initializeFailed(ManagedLedgerFactoryImpl.java:457)\n\tat org.apache.bookkeeper.mledger.impl.ManagedLedgerImpl.lambda$initializeBookKeeper$2(ManagedLedgerImpl.java:583)\n\tat org.apache.bookkeeper.common.util.SingleThreadExecutor.safeRunTask(SingleThreadExecutor.java:137)\n\tat org.apache.bookkeeper.common.util.SingleThreadExecutor.run(SingleThreadExecutor.java:107)\n\tat io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)\n\tat java.base/java.lang.Thread.run(Unknown Source)\nCaused by: org.apache.bookkeeper.mledger.ManagedLedgerException: Not enough non-faulty bookies available error code: -6\n"}]
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug] "Not enough non-faulty bookies available error code: -6" persisting after outage while all bookies are reported as healthy [pulsar]
Posted by "MichalKoziorowski-TomTom (via GitHub)" <gi...@apache.org>.
MichalKoziorowski-TomTom commented on issue #23807:
URL: https://github.com/apache/pulsar/issues/23807#issuecomment-2710465502
Hi @mrtworo
Have you found a fix for this issue?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org