You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pulsar.apache.org by GitBox <gi...@apache.org> on 2022/09/23 07:08:24 UTC
[GitHub] [pulsar] catsun26 opened a new issue, #17814: Failed to create producer: org.apache.bookkeeper.mledger.ManagedLedgerException: Not enough non-faulty bookies available
catsun26 opened a new issue, #17814:
URL: https://github.com/apache/pulsar/issues/17814
### Search before asking
- [X] I searched in the [issues](https://github.com/apache/pulsar/issues) and found nothing similar.
### Version
centos 7.9
apache-pulsar-2.8.0 one pulsar cluster
apache-pulsar-2.8.0 + broker-2.8.1 and two pulsar cluster
### Minimal reproduce step
1.Change the attribute in the broker.conf from 322 to 222, restart broker and stop one bookie's server
managedLedgerDefaultEnsembleSize=3
managedLedgerDefaultWriteQuorum=2
managedLedgerDefaultAckQuorum=2
Change to
managedLedgerDefaultEnsembleSize=2
managedLedgerDefaultWriteQuorum=2
managedLedgerDefaultAckQuorum=2
2.[root@pulsar1 apache-pulsar-2.8.0]# pulsar-admin brokers get-runtime-config |grep managedLedgerDefaultEnsembleSize
"managedLedgerDefaultEnsembleSize 2"
3. pulsar-client produce bsc/k8s/test -n 1 -m "hello"
14:29:47.029 [pulsar-client-io-1-1] INFO org.apache.pulsar.client.impl.ConnectionPool - [[id: 0x3d78fba3, L:/127.0.0.1:40608 - R:localhost/127.0.0.1:6660]] Connected to server
14:29:47.136 [pulsar-client-io-1-1] INFO org.apache.pulsar.client.impl.ProducerStatsRecorderImpl - Starting Pulsar producer perf with config: {
"topicName" : "bsc/k8s/test",
"producerName" : null,
"sendTimeoutMs" : 30000,
"blockIfQueueFull" : false,
"maxPendingMessages" : 1000,
"maxPendingMessagesAcrossPartitions" : 50000,
"messageRoutingMode" : "RoundRobinPartition",
"hashingScheme" : "JavaStringHash",
"cryptoFailureAction" : "FAIL",
"batchingMaxPublishDelayMicros" : 1000,
"batchingPartitionSwitchFrequencyByPublishDelay" : 10,
"batchingMaxMessages" : 1000,
"batchingMaxBytes" : 131072,
"batchingEnabled" : true,
"chunkingEnabled" : false,
"compressionType" : "NONE",
"initialSequenceId" : null,
"autoUpdatePartitions" : true,
"autoUpdatePartitionsIntervalSeconds" : 60,
"multiSchema" : true,
"accessMode" : "Shared",
"properties" : { }
}
14:29:47.146 [pulsar-client-io-1-1] INFO org.apache.pulsar.client.impl.ProducerStatsRecorderImpl - Pulsar client config: {
"serviceUrl" : "pulsar://localhost:6660/",
"authPluginClassName" : null,
"authParams" : null,
"authParamMap" : null,
"operationTimeoutMs" : 30000,
"statsIntervalSeconds" : 60,
"numIoThreads" : 1,
"numListenerThreads" : 1,
"connectionsPerBroker" : 1,
"useTcpNoDelay" : true,
"useTls" : false,
"tlsTrustCertsFilePath" : "",
"tlsAllowInsecureConnection" : false,
"tlsHostnameVerificationEnable" : false,
"concurrentLookupRequest" : 5000,
"maxLookupRequest" : 50000,
"maxLookupRedirects" : 20,
"maxNumberOfRejectedRequestPerConnection" : 50,
"keepAliveIntervalSeconds" : 30,
"connectionTimeoutMs" : 10000,
"requestTimeoutMs" : 60000,
"initialBackoffIntervalNanos" : 100000000,
"maxBackoffIntervalNanos" : 60000000000,
"enableBusyWait" : false,
"listenerName" : null,
"useKeyStoreTls" : false,
"sslProvider" : null,
"tlsTrustStoreType" : "JKS",
"tlsTrustStorePath" : "",
"tlsTrustStorePassword" : "",
"tlsCiphers" : [ ],
"tlsProtocols" : [ ],
"memoryLimitBytes" : 0,
"proxyServiceUrl" : null,
"proxyProtocol" : null,
"enableTransaction" : false
}
14:29:47.201 [pulsar-client-io-1-1] INFO org.apache.pulsar.client.impl.ConnectionPool - [[id: 0xd4f73691, L:/127.0.0.1:40614 - R:localhost/127.0.0.1:6660]] Connected to server
14:29:47.201 [pulsar-client-io-1-1] INFO org.apache.pulsar.client.impl.ClientCnx - [id: 0xd4f73691, L:/127.0.0.1:40614 - R:localhost/127.0.0.1:6660] Connected through proxy to target broker at pulsar3:6650
14:29:47.208 [pulsar-client-io-1-1] INFO org.apache.pulsar.client.impl.ProducerImpl - [bsc/k8s/test] [null] Creating producer on cnx [id: 0xd4f73691, L:/127.0.0.1:40614 - R:localhost/127.0.0.1:6660]
14:29:47.230 [pulsar-client-io-1-1] WARN org.apache.pulsar.client.impl.ClientCnx - [id: 0xd4f73691, L:/127.0.0.1:40614 - R:localhost/127.0.0.1:6660] Received error from server: org.apache.bookkeeper.mledger.ManagedLedgerException: Not enough non-faulty bookies available
14:29:47.231 [pulsar-client-io-1-1] ERROR org.apache.pulsar.client.impl.ProducerImpl - [bsc/k8s/test] [null] Failed to create producer: org.apache.bookkeeper.mledger.ManagedLedgerException: Not enough non-faulty bookies available
14:29:47.232 [pulsar-client-io-1-1] WARN org.apache.pulsar.client.impl.ConnectionHandler - [bsc/k8s/test] [null] Could not get connection to broker: org.apache.bookkeeper.mledger.ManagedLedgerException: Not enough non-faulty bookies available -- Will try again in 0.1 s
14:29:47.333 [pulsar-timer-5-1] INFO org.apache.pulsar.client.impl.ConnectionHandler - [bsc/k8s/test] [null] Reconnecting after connection was closed
4. broker's logs
.PerChannelBookieClient - Disconnected from bookie channel [id: 0xaff4064a, L:/192.168.209.83:54132 ! R:192.168.209.81/192.168.209.81:3181]
14:29:19.727 [pulsar-io-4-7] INFO org.apache.bookkeeper.proto.PerChannelBookieClient - Disconnected from bookie channel [id: 0xe474d3ad, L:/192.168.209.83:54119 ! R:192.168.209.81/192.168.209.81:3181]
14:29:19.727 [pulsar-io-4-2] WARN org.apache.bookkeeper.proto.PerChannelBookieClient - Exception caught on:[id: 0x74fdd840, L:/192.168.209.83:54112 - R:192.168.209.81/192.168.209.81:3181] cause: readAddress(..) failed: Connection reset by peer
14:29:19.727 [pulsar-io-4-2] INFO org.apache.bookkeeper.proto.PerChannelBookieClient - Disconnected from bookie channel [id: 0x74fdd840, L:/192.168.209.83:54112 ! R:192.168.209.81/192.168.209.81:3181]
14:29:19.727 [pulsar-io-4-1] WARN org.apache.bookkeeper.proto.PerChannelBookieClient - Exception caught on:[id: 0x320e056f, L:/192.168.209.83:54128 - R:192.168.209.81/192.168.209.81:3181] cause: readAddress(..) failed: Connection reset by peer
14:29:19.727 [pulsar-io-4-1] INFO org.apache.bookkeeper.proto.PerChannelBookieClient - Disconnected from bookie channel [id: 0x320e056f, L:/192.168.209.83:54128 ! R:192.168.209.81/192.168.209.81:3181]
14:29:19.779 [main-EventThread] INFO org.apache.bookkeeper.discover.ZKRegistrationClient - Invalidate cache for 192.168.209.81:3181
14:29:19.779 [main-EventThread] INFO org.apache.bookkeeper.discover.ZKRegistrationClient - Invalidate cache for 192.168.209.81:3181
14:29:19.783 [BookKeeperClientScheduler-OrderedScheduler-0-0] INFO org.apache.bookkeeper.net.NetworkTopologyImpl - Removing a node: /default-rack/192.168.209.81:3181
14:29:19.783 [BookKeeperClientScheduler-OrderedScheduler-0-0] INFO org.apache.bookkeeper.net.NetworkTopologyImpl - Removing a node: /default-rack/192.168.209.81:3181
14:29:20.897 [pulsar-web-40-3] INFO org.eclipse.jetty.server.RequestLog - 192.168.209.81 - - [23/九月/2022:14:29:20 +0800] "GET /metrics HTTP/1.1" 302 0 "-" "Prometheus/2.29.1" 0
14:29:20.900 [prometheus-stats-41-1] INFO org.eclipse.jetty.server.RequestLog - 192.168.209.81 - - [23/九月/2022:14:29:20 +0800] "GET /metrics/ HTTP/1.1" 200 28244 "http://192.168.209.83:8080/metrics" "Prometheus/2.29.1" 2
14:29:21.884 [pulsar-web-40-8] INFO org.eclipse.jetty.server.RequestLog - 192.168.209.81 - - [23/九月/2022:14:29:21 +0800] "GET /metrics HTTP/1.1" 302 0 "-" "Prometheus/2.29.1" 1
14:29:21.887 [prometheus-stats-41-1] INFO org.eclipse.jetty.server.RequestLog - 192.168.209.81 - - [23/九月/2022:14:29:21 +0800] "GET /metrics/ HTTP/1.1" 200 28244 "http://192.168.209.83:8080/metrics" "Prometheus/2.29.1" 3
14:29:35.897 [pulsar-web-40-1] INFO org.eclipse.jetty.server.RequestLog - 192.168.209.81 - - [23/九月/2022:14:29:35 +0800] "GET /metrics HTTP/1.1" 302 0 "-" "Prometheus/2.29.1" 0
14:29:35.900 [prometheus-stats-41-1] INFO org.eclipse.jetty.server.RequestLog - 192.168.209.81 - - [23/九月/2022:14:29:35 +0800] "GET /metrics/ HTTP/1.1" 200 28244 "http://192.168.209.83:8080/metrics" "Prometheus/2.29.1" 3
14:29:36.884 [pulsar-web-40-4] INFO org.eclipse.jetty.server.RequestLog - 192.168.209.81 - - [23/九月/2022:14:29:36 +0800] "GET /metrics HTTP/1.1" 302 0 "-" "Prometheus/2.29.1" 1
14:29:36.891 [prometheus-stats-41-1] INFO org.eclipse.jetty.server.RequestLog - 192.168.209.81 - - [23/九月/2022:14:29:36 +0800] "GET /metrics/ HTTP/1.1" 200 28245 "http://192.168.209.83:8080/metrics" "Prometheus/2.29.1" 6
14:29:47.058 [pulsar-io-4-5] INFO org.apache.pulsar.broker.service.ServerCnx - New connection from /192.168.209.81:33458
14:29:47.208 [pulsar-io-4-6] INFO org.apache.pulsar.broker.service.ServerCnx - New connection from /192.168.209.81:33464
14:29:47.215 [pulsar-io-4-6] INFO org.apache.pulsar.broker.service.ServerCnx - [/192.168.209.81:33464][persistent://bsc/k8s/test] Creating producer. producerId=0
14:29:47.216 [pulsar-ordered-OrderedExecutor-3-0] INFO org.apache.pulsar.broker.PulsarService - No ledger offloader configured, using NULL instance
14:29:47.216 [pulsar-ordered-OrderedExecutor-3-0] INFO org.apache.bookkeeper.mledger.impl.ManagedLedgerImpl - Opening managed ledger bsc/k8s/persistent/test
14:29:47.217 [bookkeeper-ml-scheduler-OrderedScheduler-4-0] INFO org.apache.bookkeeper.mledger.impl.MetaStoreImpl - Creating '/managed-ledgers/bsc/k8s/persistent/test'
14:29:47.220 [pulsar-ordered-OrderedExecutor-1-0-EventThread] INFO org.apache.pulsar.zookeeper.ZooKeeperCache - [State:CONNECTED Timeout:30000 sessionid:0x100004f5f9b000b local:/192.168.209.83:36584 remoteserver:pulsar1/192.168.209.81:2181 lastZxid:30064773910 xid:251 sent:251 recv:254 queuedpkts:0 pendingresp:0 queuedevents:1] Received ZooKeeper watch event: WatchedEvent state:SyncConnected type:NodeCreated path:/managed-ledgers/bsc/k8s/persistent/test
14:29:47.221 [metadata-store-6-1] INFO org.apache.bookkeeper.mledger.impl.ManagedLedgerImpl - [bsc/k8s/persistent/test] Creating ledger, metadata: {component=[109, 97, 110, 97, 103, 101, 100, 45, 108, 101, 100, 103, 101, 114], pulsar/managed-ledger=[98, 115, 99, 47, 107, 56, 115, 47, 112, 101, 114, 115, 105, 115, 116, 101, 110, 116, 47, 116, 101, 115, 116], application=[112, 117, 108, 115, 97, 114]} - metadata ops timeout : 60 seconds
14:29:47.221 [metadata-store-6-1] WARN org.apache.bookkeeper.client.RackawareEnsemblePlacementPolicyImpl - Failed to find 1 bookies : excludeBookies [<Bookie:192.168.209.82:3181>, <Bookie:192.168.209.83:3181>], allBookies [<Bookie:192.168.209.82:3181>, <Bookie:192.168.209.83:3181>].
14:29:47.221 [metadata-store-6-1] WARN org.apache.bookkeeper.client.RackawareEnsemblePlacementPolicyImpl - Failed to find 1 bookies : excludeBookies [<Bookie:192.168.209.82:3181>, <Bookie:192.168.209.83:3181>], allBookies [<Bookie:192.168.209.82:3181>, <Bookie:192.168.209.83:3181>].
14:29:47.221 [metadata-store-6-1] ERROR org.apache.bookkeeper.client.LedgerCreateOp - Not enough bookies to create ledger with ensembleSize=3, writeQuorumSize=2 and ackQuorumSize=2
14:29:47.221 [BookKeeperClientWorker-OrderedExecutor-0-0] ERROR org.apache.bookkeeper.mledger.impl.ManagedLedgerFactoryImpl - [bsc/k8s/persistent/test] Failed to initialize managed ledger: Not enough non-faulty bookies available
14:29:47.221 [pulsar-ordered-OrderedExecutor-1-0-EventThread] INFO org.apache.pulsar.zookeeper.ZooKeeperManagedLedgerCache - [State:CONNECTED Timeout:30000 sessionid:0x100004f5f9b000b local:/192.168.209.83:36584 remoteserver:pulsar1/192.168.209.81:2181 lastZxid:30064773910 xid:251 sent:251 recv:254 queuedpkts:0 pendingresp:0 queuedevents:0] Received ZooKeeper watch event: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/managed-ledgers/bsc/k8s/persistent
14:29:47.221 [pulsar-ordered-OrderedExecutor-1-0-EventThread] INFO org.apache.pulsar.zookeeper.ZooKeeperManagedLedgerCache - invalidate called in zookeeperChildrenCache for path /managed-ledgers/bsc/k8s/persistent
14:29:47.222 [BookKeeperClientWorker-OrderedExecutor-0-0] INFO org.apache.bookkeeper.mledger.impl.ManagedLedgerImpl - [bsc/k8s/persistent/test] Closing managed ledger
14:29:47.222 [BookKeeperClientWorker-OrderedExecutor-0-0] WARN org.apache.pulsar.broker.service.BrokerService - Failed to create topic persistent://bsc/k8s/test
org.apache.bookkeeper.mledger.ManagedLedgerException: Not enough non-faulty bookies available
14:29:47.350 [pulsar-io-4-6] INFO org.apache.pulsar.broker.service.ServerCnx - [/192.168.209.81:33464][persistent://bsc/k8s/test] Creating producer. producerId=0
14:29:47.351 [pulsar-ordered-OrderedExecutor-3-0] INFO org.apache.pulsar.broker.PulsarService - No ledger offloader configured, using NULL instance
14:29:47.351 [pulsar-ordered-OrderedExecutor-3-0] INFO org.apache.bookkeeper.mledger.impl.ManagedLedgerImpl - Opening managed ledger bsc/k8s/persistent/test
14:29:47.352 [bookkeeper-ml-scheduler-OrderedScheduler-4-0] INFO org.apache.bookkeeper.mledger.impl.ManagedLedgerImpl - [bsc/k8s/persistent/test] Creating ledger, metadata: {component=[109, 97, 110, 97, 103, 101, 100, 45, 108, 101, 100, 103, 101, 114], pulsar/managed-ledger=[98, 115, 99, 47, 107, 56, 115, 47, 112, 101, 114, 115, 105, 115, 116, 101, 110, 116, 47, 116, 101, 115, 116], application=[112, 117, 108, 115, 97, 114]} - metadata ops timeout : 60 seconds
14:29:47.353 [bookkeeper-ml-scheduler-OrderedScheduler-4-0] WARN org.apache.bookkeeper.client.RackawareEnsemblePlacementPolicyImpl - Failed to find 1 bookies : excludeBookies [<Bookie:192.168.209.82:3181>, <Bookie:192.168.209.83:3181>], allBookies [<Bookie:192.168.209.82:3181>, <Bookie:192.168.209.83:3181>].
14:29:47.353 [bookkeeper-ml-scheduler-OrderedScheduler-4-0] WARN org.apache.bookkeeper.client.RackawareEnsemblePlacementPolicyImpl - Failed to find 1 bookies : excludeBookies [<Bookie:192.168.209.82:3181>, <Bookie:192.168.209.83:3181>], allBookies [<Bookie:192.168.209.82:3181>, <Bookie:192.168.209.83:3181>].
14:29:47.353 [bookkeeper-ml-scheduler-OrderedScheduler-4-0] ERROR org.apache.bookkeeper.client.LedgerCreateOp - Not enough bookies to create ledger with ensembleSize=3, writeQuorumSize=2 and ackQuorumSize=2
14:29:47.353 [BookKeeperClientWorker-OrderedExecutor-0-0] ERROR org.apache.bookkeeper.mledger.impl.ManagedLedgerFactoryImpl - [bsc/k8s/persistent/test] Failed to initialize managed ledger: Not enough non-faulty bookies available
14:29:47.353 [BookKeeperClientWorker-OrderedExecutor-0-0] INFO org.apache.bookkeeper.mledger.impl.ManagedLedgerImpl - [bsc/k8s/persistent/test] Closing managed ledger
14:29:47.353 [BookKeeperClientWorker-OrderedExecutor-0-0] WARN org.apache.pulsar.broker.service.BrokerService - Failed to create topic persistent://bsc/k8s/test
org.apache.bookkeeper.mledger.ManagedLedgerException: Not enough non-faulty bookies available
### What did you expect to see?
The cluster can still produce and consume normally after stopping a bookie service
### What did you see instead?
The puslar cluster cannot normally produce and consume when a bookmark service is stopped
Change the attribute in the broker.conf from 322 to 222, restart broker and stop one bookie's server
managedLedgerDefaultEnsembleSize=3
managedLedgerDefaultWriteQuorum=2
managedLedgerDefaultAckQuorum=2
Change to
managedLedgerDefaultEnsembleSize=2
managedLedgerDefaultWriteQuorum=2
managedLedgerDefaultAckQuorum=2
### Anything else?
_No response_
### Are you willing to submit a PR?
- [ ] I'm willing to submit a PR!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] catsun26 commented on issue #17814: Failed to create producer: org.apache.bookkeeper.mledger.ManagedLedgerException: Not enough non-faulty bookies available
Posted by GitBox <gi...@apache.org>.
catsun26 commented on issue #17814:
URL: https://github.com/apache/pulsar/issues/17814#issuecomment-1256039625
>
This should be correct, because it has stopped（192.168.209.81:3181）
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] catsun26 commented on issue #17814: Failed to create producer: org.apache.bookkeeper.mledger.ManagedLedgerException: Not enough non-faulty bookies available
Posted by GitBox <gi...@apache.org>.
catsun26 commented on issue #17814:
URL: https://github.com/apache/pulsar/issues/17814#issuecomment-1256012688
why Change the attribute in the broker.conf from 322 to 222, restart broker and stop one bookie's server
managedLedgerDefaultEnsembleSize=3
managedLedgerDefaultWriteQuorum=2
managedLedgerDefaultAckQuorum=2
Change to
managedLedgerDefaultEnsembleSize=2
managedLedgerDefaultWriteQuorum=2
managedLedgerDefaultAckQuorum=2
**the broker's logs shown as follows**
ensembleSize=3, writeQuorumSize=2 and ackQuorumSize=2
14:29:47.353 [bookkeeper-ml-scheduler-OrderedScheduler-4-0] WARN org.apache.bookkeeper.client.RackawareEnsemblePlacementPolicyImpl - Failed to find 1 bookies : excludeBookies [Bookie:192.168.209.82:3181, Bookie:192.168.209.83:3181], allBookies [Bookie:192.168.209.82:3181, Bookie:192.168.209.83:3181].
14:29:47.353 [bookkeeper-ml-scheduler-OrderedScheduler-4-0] ERROR org.apache.bookkeeper.client.LedgerCreateOp - Not enough bookies to create ledger with ensembleSize=3, writeQuorumSize=2 and ackQuorumSize=2
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] ethqunzhong commented on issue #17814: Failed to create producer: org.apache.bookkeeper.mledger.ManagedLedgerException: Not enough non-faulty bookies available
Posted by GitBox <gi...@apache.org>.
ethqunzhong commented on issue #17814:
URL: https://github.com/apache/pulsar/issues/17814#issuecomment-1255953394
1. whether some bookie is unhealthy (read-only/...) ?
2. are you set `bookkeeperClientRegionawarePolicyEnabled=true` in broker.conf ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] catsun26 commented on issue #17814: Failed to create producer: org.apache.bookkeeper.mledger.ManagedLedgerException: Not enough non-faulty bookies available
Posted by GitBox <gi...@apache.org>.
catsun26 commented on issue #17814:
URL: https://github.com/apache/pulsar/issues/17814#issuecomment-1256056920
Thanks for your reply, this problem has been solved, because the policy was specified as 322 when the namespace was created
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] catsun26 commented on issue #17814: Failed to create producer: org.apache.bookkeeper.mledger.ManagedLedgerException: Not enough non-faulty bookies available
Posted by GitBox <gi...@apache.org>.
catsun26 commented on issue #17814:
URL: https://github.com/apache/pulsar/issues/17814#issuecomment-1256005073
>
1.bin/bookkeeper shell bookiesanity
17:45:19.971 [main] INFO org.apache.zookeeper.common.X509Util - Setting -D jdk.tls.rejectClientInitiatedRenegotiation=true to disable client-initiated TLS renegotiation
17:45:19.975 [main] INFO org.apache.zookeeper.ClientCnxnSocket - jute.maxbuffer value is 1048575 Bytes
17:45:19.980 [main] INFO org.apache.zookeeper.ClientCnxn - zookeeper.request.timeout value is 0. feature enabled=false
17:45:19.989 [main-SendThread(pulsar1:2181)] INFO org.apache.zookeeper.ClientCnxn - Opening socket connection to server pulsar1/192.168.209.81:2181.
17:45:19.989 [main-SendThread(pulsar1:2181)] INFO org.apache.zookeeper.ClientCnxn - SASL config status: Will not attempt to authenticate using SASL (unknown error)
17:45:19.992 [main-SendThread(pulsar1:2181)] INFO org.apache.zookeeper.ClientCnxn - Socket connection established, initiating session, client: /192.168.209.81:42762, server: pulsar1/192.168.209.81:2181
17:45:19.998 [main-SendThread(pulsar1:2181)] INFO org.apache.zookeeper.ClientCnxn - Session establishment complete on server pulsar1/192.168.209.81:2181, session id = 0x100004f5f9b0018, negotiated timeout = 30000
17:45:20.001 [main-EventThread] INFO org.apache.bookkeeper.zookeeper.ZooKeeperWatcherBase - ZooKeeper client is connected now.
17:45:20.466 [main] INFO org.apache.bookkeeper.client.BookKeeper - Weighted ledger placement is not enabled
17:45:20.529 [main-EventThread] INFO org.apache.bookkeeper.discover.ZKRegistrationClient - Update BookieInfoCache (writable bookie) 192.168.209.82:3181 -> BookieServiceInfo{properties={}, endpoints=[EndpointInfo{id=bookie, port=3181, host=192.168.209.82, protocol=bookie-rpc, auth=[], extensions=[]}]}
17:45:20.530 [main-EventThread] INFO org.apache.bookkeeper.discover.ZKRegistrationClient - Update BookieInfoCache (writable bookie) 192.168.209.83:3181 -> BookieServiceInfo{properties={}, endpoints=[EndpointInfo{id=bookie, port=3181, host=192.168.209.83, protocol=bookie-rpc, auth=[], extensions=[]}]}
17:45:20.596 [main-EventThread] INFO org.apache.bookkeeper.client.LedgerCreateOp - Ensemble: [192.168.209.81:3181] for ledger: 119
17:45:20.597 [main] INFO org.apache.bookkeeper.tools.cli.commands.bookie.SanityTestCommand - Create ledger 119
17:45:20.727 [BookKeeperClientWorker-OrderedExecutor-7-0] INFO org.apache.bookkeeper.client.DefaultBookieAddressResolver - Cannot resolve 192.168.209.81:3181, bookie is unknown org.apache.bookkeeper.client.BKException$BKBookieHandleNotAvailableException: Bookie handle is not available
17:45:20.727 [BookKeeperClientWorker-OrderedExecutor-7-0] ERROR org.apache.bookkeeper.proto.PerChannelBookieClient - Cannot connect to 192.168.209.81:3181 as endpoint resolution failed (probably bookie is down) err org.apache.bookkeeper.proto.BookieAddressResolver$BookieIdNotResolvedException: Cannot resolve bookieId 192.168.209.81:3181, bookie does not exist or it is not running
17:45:20.729 [BookKeeperClientWorker-OrderedExecutor-7-0] ERROR org.apache.bookkeeper.proto.PerChannelBookieClient - Could not connect to bookie: null/192.168.209.81:3181, current state CONNECTING :
org.apache.bookkeeper.proto.BookieAddressResolver$BookieIdNotResolvedException: Cannot resolve bookieId 192.168.209.81:3181, bookie does not exist or it is not running
at org.apache.bookkeeper.client.DefaultBookieAddressResolver.resolve(DefaultBookieAddressResolver.java:63) ~[org.apache.bookkeeper-bookkeeper-server-4.14.1.jar:4.14.1]
at org.apache.bookkeeper.proto.PerChannelBookieClient.connect(PerChannelBookieClient.java:531) [org.apache.bookkeeper-bookkeeper-server-4.14.1.jar:4.14.1]
at org.apache.bookkeeper.proto.PerChannelBookieClient.connectIfNeededAndDoOp(PerChannelBookieClient.java:657) [org.apache.bookkeeper-bookkeeper-server-4.14.1.jar:4.14.1]
at org.apache.bookkeeper.proto.DefaultPerChannelBookieClientPool.initialize(DefaultPerChannelBookieClientPool.java:92) [org.apache.bookkeeper-bookkeeper-server-4.14.1.jar:4.14.1]
at org.apache.bookkeeper.proto.BookieClientImpl.lookupClient(BookieClientImpl.java:217) [org.apache.bookkeeper-bookkeeper-server-4.14.1.jar:4.14.1]
at org.apache.bookkeeper.proto.BookieClientImpl.addEntry(BookieClientImpl.java:318) [org.apache.bookkeeper-bookkeeper-server-4.14.1.jar:4.14.1]
at org.apache.bookkeeper.client.PendingAddOp.sendWriteRequest(PendingAddOp.java:152) [org.apache.bookkeeper-bookkeeper-server-4.14.1.jar:4.14.1]
at org.apache.bookkeeper.client.PendingAddOp.safeRun(PendingAddOp.java:278) [org.apache.bookkeeper-bookkeeper-server-4.14.1.jar:4.14.1]
at org.apache.bookkeeper.common.util.SafeRunnable.run(SafeRunnable.java:36) [org.apache.bookkeeper-bookkeeper-common-4.14.1.jar:4.14.1]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_301]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_301]
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) [io.netty-netty-common-4.1.63.Final.jar:4.1.63.Final]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_301]
Caused by: org.apache.bookkeeper.client.BKException$BKBookieHandleNotAvailableException: Bookie handle is not available
at org.apache.bookkeeper.discover.ZKRegistrationClient.getBookieServiceInfo(ZKRegistrationClient.java:248) ~[org.apache.bookkeeper-bookkeeper-server-4.14.1.jar:4.14.1]
at org.apache.bookkeeper.client.DefaultBookieAddressResolver.resolve(DefaultBookieAddressResolver.java:43) ~[org.apache.bookkeeper-bookkeeper-server-4.14.1.jar:4.14.1]
... 12 more
17:45:20.737 [BookKeeperClientWorker-OrderedExecutor-7-0] INFO org.apache.bookkeeper.client.DefaultBookieAddressResolver - Cannot resolve 192.168.209.81:3181, bookie is unknown org.apache.bookkeeper.client.BKException$BKBookieHandleNotAvailableException: Bookie handle is not available
17:45:20.737 [BookKeeperClientWorker-OrderedExecutor-7-0] ERROR org.apache.bookkeeper.proto.PerChannelBookieClient - Cannot connect to 192.168.209.81:3181 as endpoint resolution failed (probably bookie is down) err org.apache.bookkeeper.proto.BookieAddressResolver$BookieIdNotResolvedException: Cannot resolve bookieId 192.168.209.81:3181, bookie does not exist or it is not running
17:45:20.737 [BookKeeperClientWorker-OrderedExecutor-7-0] WARN org.apache.bookkeeper.client.PendingAddOp - Failed to write entry (119, 0): Bookie handle is not available
17:45:20.743 [BookKeeperClientWorker-OrderedExecutor-7-0] ERROR org.apache.bookkeeper.client.MetadataUpdateLoop - UpdateLoop(ledgerId=119,loopId=0881972f) Exception updating
org.apache.bookkeeper.client.BKException$BKNotEnoughBookiesException: Not enough non-faulty bookies available
at org.apache.bookkeeper.bookie.LocalBookieEnsemblePlacementPolicy.replaceBookie(LocalBookieEnsemblePlacementPolicy.java:89) ~[org.apache.bookkeeper-bookkeeper-server-4.14.1.jar:4.14.1]
at org.apache.bookkeeper.client.BookieWatcherImpl.replaceBookie(BookieWatcherImpl.java:334) ~[org.apache.bookkeeper-bookkeeper-server-4.14.1.jar:4.14.1]
at org.apache.bookkeeper.client.EnsembleUtils.replaceBookiesInEnsemble(EnsembleUtils.java:71) ~[org.apache.bookkeeper-bookkeeper-server-4.14.1.jar:4.14.1]
at org.apache.bookkeeper.client.LedgerHandle.lambda$ensembleChangeLoop$2(LedgerHandle.java:1935) ~[org.apache.bookkeeper-bookkeeper-server-4.14.1.jar:4.14.1]
at org.apache.bookkeeper.client.MetadataUpdateLoop.writeLoop(MetadataUpdateLoop.java:134) [org.apache.bookkeeper-bookkeeper-server-4.14.1.jar:4.14.1]
at org.apache.bookkeeper.client.MetadataUpdateLoop.run(MetadataUpdateLoop.java:123) [org.apache.bookkeeper-bookkeeper-server-4.14.1.jar:4.14.1]
at org.apache.bookkeeper.client.LedgerHandle.ensembleChangeLoop(LedgerHandle.java:1954) [org.apache.bookkeeper-bookkeeper-server-4.14.1.jar:4.14.1]
at org.apache.bookkeeper.client.LedgerHandle.handleBookieFailure(LedgerHandle.java:1903) [org.apache.bookkeeper-bookkeeper-server-4.14.1.jar:4.14.1]
at org.apache.bookkeeper.client.PendingAddOp.writeComplete(PendingAddOp.java:378) [org.apache.bookkeeper-bookkeeper-server-4.14.1.jar:4.14.1]
at org.apache.bookkeeper.proto.BookieClientImpl$1.safeRun(BookieClientImpl.java:295) [org.apache.bookkeeper-bookkeeper-server-4.14.1.jar:4.14.1]
at org.apache.bookkeeper.common.util.SafeRunnable.run(SafeRunnable.java:36) [org.apache.bookkeeper-bookkeeper-common-4.14.1.jar:4.14.1]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_301]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_301]
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) [io.netty-netty-common-4.1.63.Final.jar:4.1.63.Final]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_301]
17:45:20.745 [BookKeeperClientWorker-OrderedExecutor-7-0] WARN org.apache.bookkeeper.client.LedgerHandle - [EnsembleChange(ledger:119, change-id:0000000001)][attempt:1] Exception changing ensemble
org.apache.bookkeeper.client.BKException$BKNotEnoughBookiesException: Not enough non-faulty bookies available
at org.apache.bookkeeper.bookie.LocalBookieEnsemblePlacementPolicy.replaceBookie(LocalBookieEnsemblePlacementPolicy.java:89) ~[org.apache.bookkeeper-bookkeeper-server-4.14.1.jar:4.14.1]
at org.apache.bookkeeper.client.BookieWatcherImpl.replaceBookie(BookieWatcherImpl.java:334) ~[org.apache.bookkeeper-bookkeeper-server-4.14.1.jar:4.14.1]
at org.apache.bookkeeper.client.EnsembleUtils.replaceBookiesInEnsemble(EnsembleUtils.java:71) ~[org.apache.bookkeeper-bookkeeper-server-4.14.1.jar:4.14.1]
at org.apache.bookkeeper.client.LedgerHandle.lambda$ensembleChangeLoop$2(LedgerHandle.java:1935) ~[org.apache.bookkeeper-bookkeeper-server-4.14.1.jar:4.14.1]
at org.apache.bookkeeper.client.MetadataUpdateLoop.writeLoop(MetadataUpdateLoop.java:134) ~[org.apache.bookkeeper-bookkeeper-server-4.14.1.jar:4.14.1]
at org.apache.bookkeeper.client.MetadataUpdateLoop.run(MetadataUpdateLoop.java:123) ~[org.apache.bookkeeper-bookkeeper-server-4.14.1.jar:4.14.1]
at org.apache.bookkeeper.client.LedgerHandle.ensembleChangeLoop(LedgerHandle.java:1954) ~[org.apache.bookkeeper-bookkeeper-server-4.14.1.jar:4.14.1]
at org.apache.bookkeeper.client.LedgerHandle.handleBookieFailure(LedgerHandle.java:1903) ~[org.apache.bookkeeper-bookkeeper-server-4.14.1.jar:4.14.1]
at org.apache.bookkeeper.client.PendingAddOp.writeComplete(PendingAddOp.java:378) ~[org.apache.bookkeeper-bookkeeper-server-4.14.1.jar:4.14.1]
at org.apache.bookkeeper.proto.BookieClientImpl$1.safeRun(BookieClientImpl.java:295) ~[org.apache.bookkeeper-bookkeeper-server-4.14.1.jar:4.14.1]
at org.apache.bookkeeper.common.util.SafeRunnable.run(SafeRunnable.java:36) ~[org.apache.bookkeeper-bookkeeper-common-4.14.1.jar:4.14.1]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_301]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_301]
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) [io.netty-netty-common-4.1.63.Final.jar:4.1.63.Final]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_301]
17:45:20.745 [BookKeeperClientWorker-OrderedExecutor-7-0] ERROR org.apache.bookkeeper.client.LedgerHandle - Closing ledger 119 due to NotEnoughBookiesException: Not enough non-faulty bookies available
17:45:20.747 [BookKeeperClientWorker-OrderedExecutor-7-0] ERROR org.apache.bookkeeper.client.PendingAddOp - Write of ledger entry to quorum failed: L119 E0
17:45:20.747 [main] WARN org.apache.bookkeeper.tools.cli.commands.bookie.SanityTestCommand - Error in bookie sanity test
org.apache.bookkeeper.client.BKException$BKNotEnoughBookiesException: Not enough non-faulty bookies available
at org.apache.bookkeeper.client.SyncCallbackUtils.finish(SyncCallbackUtils.java:83) ~[org.apache.bookkeeper-bookkeeper-server-4.14.1.jar:4.14.1]
at org.apache.bookkeeper.client.SyncCallbackUtils$SyncAddCallback.addComplete(SyncCallbackUtils.java:251) ~[org.apache.bookkeeper-bookkeeper-server-4.14.1.jar:4.14.1]
at org.apache.bookkeeper.client.AsyncCallback$AddCallback.addCompleteWithLatency(AsyncCallback.java:92) ~[org.apache.bookkeeper-bookkeeper-server-4.14.1.jar:4.14.1]
at org.apache.bookkeeper.client.PendingAddOp.submitCallback(PendingAddOp.java:431) ~[org.apache.bookkeeper-bookkeeper-server-4.14.1.jar:4.14.1]
at org.apache.bookkeeper.client.LedgerHandle.errorOutPendingAdds(LedgerHandle.java:1799) ~[org.apache.bookkeeper-bookkeeper-server-4.14.1.jar:4.14.1]
at org.apache.bookkeeper.client.LedgerHandle$5.safeRun(LedgerHandle.java:574) ~[org.apache.bookkeeper-bookkeeper-server-4.14.1.jar:4.14.1]
at org.apache.bookkeeper.common.util.SafeRunnable.run(SafeRunnable.java:36) ~[org.apache.bookkeeper-bookkeeper-common-4.14.1.jar:4.14.1]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[?:1.8.0_301]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[?:1.8.0_301]
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ~[io.netty-netty-common-4.1.63.Final.jar:4.1.63.Final]
at java.lang.Thread.run(Thread.java:748) ~[?:1.8.0_301]
17:45:20.763 [main] INFO org.apache.bookkeeper.tools.cli.commands.bookie.SanityTestCommand - Deleted ledger 119
17:45:20.764 [main] INFO org.apache.bookkeeper.proto.PerChannelBookieClient - Closing the per channel bookie client for 192.168.209.81:3181
17:45:20.768 [main-EventThread] WARN org.apache.bookkeeper.meta.AbstractZkLedgerManager - Ledger node does not exist in ZooKeeper: ledgerId=119
17:45:20.768 [main-EventThread] ERROR org.apache.bookkeeper.client.MetadataUpdateLoop - UpdateLoop(ledgerId=119,loopId=3e5ec475) Error writing metadata to store
org.apache.bookkeeper.client.BKException$BKNoSuchLedgerExistsOnMetadataServerException: No such ledger exists on Metadata Server
at org.apache.bookkeeper.meta.AbstractZkLedgerManager$4.processResult(AbstractZkLedgerManager.java:505) [org.apache.bookkeeper-bookkeeper-server-4.14.1.jar:4.14.1]
at org.apache.bookkeeper.zookeeper.ZooKeeperClient$22$1.processResult(ZooKeeperClient.java:1094) [org.apache.bookkeeper-bookkeeper-server-4.14.1.jar:4.14.1]
at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:638) [org.apache.zookeeper-zookeeper-3.6.3.jar:3.6.3]
at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:563) [org.apache.zookeeper-zookeeper-3.6.3.jar:3.6.3]
17:45:20.769 [main-EventThread] WARN org.apache.bookkeeper.client.LedgerHandle - Close failed: NoSuchLedgerExistsOnMetadataServerException: No such ledger exists on Metadata Server
17:45:20.882 [main] INFO org.apache.zookeeper.ZooKeeper - Session: 0x100004f5f9b0018 closed
17:45:20.882 [main-EventThread] INFO org.apache.zookeeper.ClientCnxn - EventThread shut down for session: 0x100004f5f9b0018
2.bin/bookkeeper shell simpletest --ensemble 2 -writeQuorum 2 --ackQuorum 2 --numEntries 2
17:45:39.995 [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:java.library.path=/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib
17:45:39.995 [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:java.io.tmpdir=/tmp
17:45:39.995 [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:java.compiler=<NA>
17:45:39.995 [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:os.name=Linux
17:45:39.995 [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:os.arch=amd64
17:45:39.995 [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:os.version=3.10.0-1160.el7.x86_64
17:45:39.995 [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:user.name=root
17:45:39.995 [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:user.home=/root
17:45:39.995 [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:user.dir=/data/apache-pulsar-2.8.0
17:45:39.995 [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:os.memory.free=1902MB
17:45:39.995 [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:os.memory.max=2048MB
17:45:39.995 [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:os.memory.total=2048MB
17:45:39.999 [main] INFO org.apache.zookeeper.ZooKeeper - Initiating client connection, connectString=pulsar1:2181,pulsar2:2181,pulsar3:2181 sessionTimeout=30000 watcher=org.apache.bookkeeper.zookeeper.ZooKeeperWatcherBase@18e36d14
17:45:40.003 [main] INFO org.apache.zookeeper.common.X509Util - Setting -D jdk.tls.rejectClientInitiatedRenegotiation=true to disable client-initiated TLS renegotiation
17:45:40.009 [main] INFO org.apache.zookeeper.ClientCnxnSocket - jute.maxbuffer value is 1048575 Bytes
17:45:40.015 [main] INFO org.apache.zookeeper.ClientCnxn - zookeeper.request.timeout value is 0. feature enabled=false
17:45:40.020 [main-SendThread(pulsar1:2181)] INFO org.apache.zookeeper.ClientCnxn - Opening socket connection to server pulsar1/192.168.209.81:2181.
17:45:40.020 [main-SendThread(pulsar1:2181)] INFO org.apache.zookeeper.ClientCnxn - SASL config status: Will not attempt to authenticate using SASL (unknown error)
17:45:40.026 [main-SendThread(pulsar1:2181)] INFO org.apache.zookeeper.ClientCnxn - Socket connection established, initiating session, client: /192.168.209.81:42782, server: pulsar1/192.168.209.81:2181
17:45:40.032 [main-SendThread(pulsar1:2181)] INFO org.apache.zookeeper.ClientCnxn - Session establishment complete on server pulsar1/192.168.209.81:2181, session id = 0x100004f5f9b0019, negotiated timeout = 30000
17:45:40.035 [main-EventThread] INFO org.apache.bookkeeper.zookeeper.ZooKeeperWatcherBase - ZooKeeper client is connected now.
17:45:40.419 [main] ERROR org.apache.bookkeeper.client.RackawareEnsemblePlacementPolicyImpl - Failed to initialize DNS Resolver org.apache.bookkeeper.net.ScriptBasedMapping, used default subnet resolver : java.lang.RuntimeException: No network topology script is found when using script based DNS resolver. No network topology script is found when using script based DNS resolver.
17:45:40.436 [main] INFO org.apache.bookkeeper.client.RackawareEnsemblePlacementPolicyImpl - Initialize rackaware ensemble placement policy @ <Bookie:192.168.209.81:0> @ /default-rack : org.apache.bookkeeper.client.TopologyAwareEnsemblePlacementPolicy$DefaultResolver.
17:45:40.437 [main] INFO org.apache.bookkeeper.client.RackawareEnsemblePlacementPolicyImpl - Not weighted
17:45:40.446 [main] INFO org.apache.bookkeeper.client.BookKeeper - Weighted ledger placement is not enabled
17:45:40.517 [main-EventThread] INFO org.apache.bookkeeper.discover.ZKRegistrationClient - Update BookieInfoCache (writable bookie) 192.168.209.82:3181 -> BookieServiceInfo{properties={}, endpoints=[EndpointInfo{id=bookie, port=3181, host=192.168.209.82, protocol=bookie-rpc, auth=[], extensions=[]}]}
17:45:40.517 [main-EventThread] INFO org.apache.bookkeeper.discover.ZKRegistrationClient - Update BookieInfoCache (writable bookie) 192.168.209.83:3181 -> BookieServiceInfo{properties={}, endpoints=[EndpointInfo{id=bookie, port=3181, host=192.168.209.83, protocol=bookie-rpc, auth=[], extensions=[]}]}
17:45:40.520 [BookKeeperClientScheduler-OrderedScheduler-0-0] INFO org.apache.bookkeeper.net.NetworkTopologyImpl - Adding a new node: /default-rack/192.168.209.82:3181
17:45:40.521 [BookKeeperClientScheduler-OrderedScheduler-0-0] INFO org.apache.bookkeeper.net.NetworkTopologyImpl - Adding a new node: /default-rack/192.168.209.83:3181
17:45:40.545 [main] WARN org.apache.bookkeeper.client.BookieWatcherImpl - New ensemble: [192.168.209.83:3181, 192.168.209.82:3181] is not adhering to Placement Policy. quarantinedBookies: []
17:45:40.600 [main-EventThread] INFO org.apache.bookkeeper.client.LedgerCreateOp - Ensemble: [192.168.209.83:3181, 192.168.209.82:3181] for ledger: 120
17:45:40.600 [main] INFO org.apache.bookkeeper.tools.cli.commands.client.SimpleTestCommand - Ledger ID: 120
17:45:40.619 [BookKeeperClientWorker-OrderedExecutor-0-0] INFO com.scurrilous.circe.checksum.Crc32cIntChecksum - SSE4.2 CRC32C provider initialized
17:45:40.706 [bookkeeper-io-3-1] INFO org.apache.bookkeeper.proto.PerChannelBookieClient - Successfully connected to bookie: 192.168.209.83:3181 [id: 0x787941fd, L:/192.168.209.81:49796 - R:192.168.209.83/192.168.209.83:3181]
17:45:40.706 [bookkeeper-io-3-2] INFO org.apache.bookkeeper.proto.PerChannelBookieClient - Successfully connected to bookie: 192.168.209.82:3181 [id: 0xb5657fd5, L:/192.168.209.81:36878 - R:192.168.209.82/192.168.209.82:3181]
17:45:40.743 [bookkeeper-io-3-2] INFO org.apache.bookkeeper.proto.PerChannelBookieClient - connection [id: 0xb5657fd5, L:/192.168.209.81:36878 - R:192.168.209.82/192.168.209.82:3181] authenticated as BookKeeperPrincipal{ANONYMOUS}
17:45:40.743 [bookkeeper-io-3-1] INFO org.apache.bookkeeper.proto.PerChannelBookieClient - connection [id: 0x787941fd, L:/192.168.209.81:49796 - R:192.168.209.83/192.168.209.83:3181] authenticated as BookKeeperPrincipal{ANONYMOUS}
17:45:40.783 [main] INFO org.apache.bookkeeper.tools.cli.commands.client.SimpleTestCommand - 2 entries written to ledger 120
17:45:40.800 [main] INFO org.apache.bookkeeper.proto.PerChannelBookieClient - Closing the per channel bookie client for 192.168.209.82:3181
17:45:40.805 [main] INFO org.apache.bookkeeper.proto.PerChannelBookieClient - Closing the per channel bookie client for 192.168.209.83:3181
17:45:40.806 [bookkeeper-io-3-1] INFO org.apache.bookkeeper.proto.PerChannelBookieClient - Disconnected from bookie channel [id: 0x787941fd, L:/192.168.209.81:49796 ! R:192.168.209.83/192.168.209.83:3181]
17:45:40.806 [bookkeeper-io-3-2] INFO org.apache.bookkeeper.proto.PerChannelBookieClient - Disconnected from bookie channel [id: 0xb5657fd5, L:/192.168.209.81:36878 ! R:192.168.209.82/192.168.209.82:3181]
17:45:40.925 [main-EventThread] INFO org.apache.zookeeper.ClientCnxn - EventThread shut down for session: 0x100004f5f9b0019
17:45:40.925 [main] INFO org.apache.zookeeper.ZooKeeper - Session: 0x100004f5f9b0019 closed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] ethqunzhong commented on issue #17814: Failed to create producer: org.apache.bookkeeper.mledger.ManagedLedgerException: Not enough non-faulty bookies available
Posted by GitBox <gi...@apache.org>.
ethqunzhong commented on issue #17814:
URL: https://github.com/apache/pulsar/issues/17814#issuecomment-1255996998
it seems all bookies not available
<img width="783" alt="image" src="https://user-images.githubusercontent.com/16517186/191932840-545468bb-dfc8-4f42-a21e-b2b5ac58ad6b.png">
you could have a test for bk cluster
1. `bin/bookkeeper shell bookiesanity`
2. `bin/bookkeeper shell simpletest --ensemble 2 -writeQuorum 2 --ackQuorum 2 --numEntries 2`
the key point is found out why these two bookies marked disabled-states.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] catsun26 commented on issue #17814: Failed to create producer: org.apache.bookkeeper.mledger.ManagedLedgerException: Not enough non-faulty bookies available
Posted by GitBox <gi...@apache.org>.
catsun26 commented on issue #17814:
URL: https://github.com/apache/pulsar/issues/17814#issuecomment-1255983529
> 1. whether some bookie is unhealthy (read-only/...) ?
>
> 2. are you set `bookkeeperClientRegionawarePolicyEnabled=true` in broker.conf ?
Thank you for your reply
1. stop one bookie's service,give the result as follows
[root@pulsar1 apache-pulsar-2.8.0]# pulsar-admin bookies list-bookies
{
"bookies" : [ {
"bookieId" : "192.168.209.82:3181"
}, {
"bookieId" : "192.168.209.81:3181"
}, {
"bookieId" : "192.168.209.83:3181"
} ]
}
ls /ledgers/available
[192.168.209.82:3181, 192.168.209.83:3181, readonly]
2.`bookkeeperClientRegionawarePolicyEnabled=false` in broker.conf
The broker configuration file has been updated and copied above
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] catsun26 closed issue #17814: Failed to create producer: org.apache.bookkeeper.mledger.ManagedLedgerException: Not enough non-faulty bookies available
Posted by GitBox <gi...@apache.org>.
catsun26 closed issue #17814: Failed to create producer: org.apache.bookkeeper.mledger.ManagedLedgerException: Not enough non-faulty bookies available
URL: https://github.com/apache/pulsar/issues/17814
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org