You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pulsar.apache.org by GitBox <gi...@apache.org> on 2019/11/12 09:29:24 UTC
[GitHub] [pulsar] vishalananth07 opened a new issue #5630: Error when trying
to run Producer-Consumer commands in a 2 node cluster
vishalananth07 opened a new issue #5630: Error when trying to run Producer-Consumer commands in a 2 node cluster
URL: https://github.com/apache/pulsar/issues/5630
**Describe the bug**
Producer command
```
$ bin/pulsar-client consume persistent://public/default/test -n 100 -s "consumer-test" -t "Exclusive"
```
and Consumer command
```
bin/pulsar-client consume persistent://public/default/test -n 100 -s "consumer-test" -t "Exclusive"
```
are giving the following error messages:
Producer error message:
```
bin/pulsar-client produce persistent://public/default/test -n 1 -m "hello pulsar"
14:28:32.171 [pulsar-client-io-1-1] INFO org.apache.pulsar.client.impl.ConnectionPool - [[id: 0x2c91e691, L:/192.168.1.13:35714 - R:192.168.1.5/192.168.1.5:6650]] Connected to server
14:28:32.707 [pulsar-client-io-1-1] INFO org.apache.pulsar.client.impl.ProducerStatsRecorderImpl - Starting Pulsar producer perf with config: {
"topicName" : "persistent://public/default/test",
"producerName" : null,
"sendTimeoutMs" : 30000,
"blockIfQueueFull" : false,
"maxPendingMessages" : 1000,
"maxPendingMessagesAcrossPartitions" : 50000,
"messageRoutingMode" : "RoundRobinPartition",
"hashingScheme" : "JavaStringHash",
"cryptoFailureAction" : "FAIL",
"batchingMaxPublishDelayMicros" : 1000,
"batchingMaxMessages" : 1000,
"batchingEnabled" : true,
"compressionType" : "NONE",
"initialSequenceId" : null,
"autoUpdatePartitions" : true,
"properties" : { }
}
14:28:32.710 [pulsar-client-io-1-1] INFO org.apache.pulsar.client.impl.ProducerStatsRecorderImpl - Pulsar client config: {
"serviceUrl" : "pulsar://192.168.1.13:6650,192.168.1.5:6650",
"authPluginClassName" : null,
"authParams" : null,
"operationTimeoutMs" : 30000,
"statsIntervalSeconds" : 60,
"numIoThreads" : 1,
"numListenerThreads" : 1,
"connectionsPerBroker" : 1,
"useTcpNoDelay" : true,
"useTls" : false,
"tlsTrustCertsFilePath" : "",
"tlsAllowInsecureConnection" : false,
"tlsHostnameVerificationEnable" : false,
"concurrentLookupRequest" : 5000,
"maxLookupRequest" : 50000,
"maxNumberOfRejectedRequestPerConnection" : 50,
"keepAliveIntervalSeconds" : 30,
"connectionTimeoutMs" : 10000,
"requestTimeoutMs" : 60000,
"defaultBackoffIntervalNanos" : 100000000,
"maxBackoffIntervalNanos" : 30000000000
}
14:28:32.714 [pulsar-client-io-1-1] INFO org.apache.pulsar.client.impl.ConnectionPool - [[id: 0x6e57db24, L:/192.168.1.13:43896 - R:192.168.1.13/192.168.1.13:6650]] Connected to server
14:28:32.734 [pulsar-client-io-1-1] INFO org.apache.pulsar.client.impl.ConnectionPool - [[id: 0x009288aa, L:/127.0.0.1:40736 - R:sundar-UX303UB/127.0.1.1:6650]] Connected to server
14:28:32.741 [pulsar-client-io-1-1] INFO org.apache.pulsar.client.impl.ProducerImpl - [persistent://public/default/test] [null] Creating producer on cnx [id: 0x009288aa, L:/127.0.0.1:40736 - R:sundar-UX303UB/127.0.1.1:6650]
14:28:32.864 [pulsar-client-io-1-1] INFO org.apache.pulsar.client.impl.ProducerImpl - [persistent://public/default/test] [test-cluster-1-0-3] Created producer on cnx [id: 0x009288aa, L:/127.0.0.1:40736 - R:sundar-UX303UB/127.0.1.1:6650]
14:28:32.951 [main] WARN com.scurrilous.circe.checksum.Crc32cIntChecksum - Failed to load Circe JNI library. Falling back to Java based CRC32c provider
14:29:02.954 [pulsar-timer-4-1] INFO org.apache.pulsar.client.impl.ProducerImpl - [persistent://public/default/test] [test-cluster-1-0-3] Message send timed out. Failing 1 messages
14:29:02.964 [main] ERROR org.apache.pulsar.client.cli.PulsarClientTool - Error while producing messages
14:29:02.965 [main] ERROR org.apache.pulsar.client.cli.PulsarClientTool - Could not send message to broker within given timeout
org.apache.pulsar.client.api.PulsarClientException$TimeoutException: Could not send message to broker within given timeout
at org.apache.pulsar.client.api.PulsarClientException.unwrap(PulsarClientException.java:247) ~[org.apache.pulsar-pulsar-client-api-2.4.1.jar:2.4.1]
at org.apache.pulsar.client.impl.ProducerBase.send(ProducerBase.java:86) ~[org.apache.pulsar-pulsar-client-original-2.4.1.jar:2.4.1]
at org.apache.pulsar.client.impl.TypedMessageBuilderImpl.send(TypedMessageBuilderImpl.java:61) ~[org.apache.pulsar-pulsar-client-original-2.4.1.jar:2.4.1]
at org.apache.pulsar.client.impl.ProducerBase.send(ProducerBase.java:51) ~[org.apache.pulsar-pulsar-client-original-2.4.1.jar:2.4.1]
at org.apache.pulsar.client.cli.CmdProduce.publish(CmdProduce.java:189) [org.apache.pulsar-pulsar-client-tools-2.4.1.jar:2.4.1]
at org.apache.pulsar.client.cli.CmdProduce.run(CmdProduce.java:169) [org.apache.pulsar-pulsar-client-tools-2.4.1.jar:2.4.1]
at org.apache.pulsar.client.cli.PulsarClientTool.run(PulsarClientTool.java:131) [org.apache.pulsar-pulsar-client-tools-2.4.1.jar:2.4.1]
at org.apache.pulsar.client.cli.PulsarClientTool.main(PulsarClientTool.java:165) [org.apache.pulsar-pulsar-client-tools-2.4.1.jar:2.4.1]
14:29:02.978 [main] INFO org.apache.pulsar.client.cli.PulsarClientTool - 0 messages successfully produced
```
Consumer error message:
```
bin/pulsar-client consume persistent://public/default/test -n 100 -s "consumer-test" -t "Exclusive"
14:29:02.442 [pulsar-client-io-1-1] INFO org.apache.pulsar.client.impl.ConnectionPool - [[id: 0xc40711ef, L:/192.168.1.5:46150 - R:192.168.1.13/192.168.1.13:6650]] Connected to server
14:29:02.653 [pulsar-client-io-1-1] INFO org.apache.pulsar.client.impl.ConsumerStatsRecorderImpl - Starting Pulsar consumer perf with config: {
"topicNames" : [ "persistent://public/default/test" ],
"topicsPattern" : null,
"subscriptionName" : "consumer-test",
"subscriptionType" : "Exclusive",
"receiverQueueSize" : 1000,
"acknowledgementsGroupTimeMicros" : 100000,
"negativeAckRedeliveryDelayMicros" : 60000000,
"maxTotalReceiverQueueSizeAcrossPartitions" : 50000,
"consumerName" : null,
"ackTimeoutMillis" : 0,
"tickDurationMillis" : 1000,
"priorityLevel" : 0,
"cryptoFailureAction" : "FAIL",
"properties" : { },
"readCompacted" : false,
"subscriptionInitialPosition" : "Latest",
"patternAutoDiscoveryPeriod" : 1,
"regexSubscriptionMode" : "PersistentOnly",
"deadLetterPolicy" : null,
"autoUpdatePartitions" : true,
"replicateSubscriptionState" : false,
"resetIncludeHead" : false
}
14:29:02.660 [pulsar-client-io-1-1] INFO org.apache.pulsar.client.impl.ConsumerStatsRecorderImpl - Pulsar client config: {
"serviceUrl" : "pulsar://192.168.1.13:6650,192.168.1.5:6650",
"authPluginClassName" : null,
"authParams" : null,
"operationTimeoutMs" : 30000,
"statsIntervalSeconds" : 60,
"numIoThreads" : 1,
"numListenerThreads" : 1,
"connectionsPerBroker" : 1,
"useTcpNoDelay" : true,
"useTls" : false,
"tlsTrustCertsFilePath" : "",
"tlsAllowInsecureConnection" : false,
"tlsHostnameVerificationEnable" : false,
"concurrentLookupRequest" : 5000,
"maxLookupRequest" : 50000,
"maxNumberOfRejectedRequestPerConnection" : 50,
"keepAliveIntervalSeconds" : 30,
"connectionTimeoutMs" : 10000,
"requestTimeoutMs" : 60000,
"defaultBackoffIntervalNanos" : 100000000,
"maxBackoffIntervalNanos" : 30000000000
}
14:29:02.679 [pulsar-client-io-1-1] INFO org.apache.pulsar.client.impl.ConnectionPool - [[id: 0xa7cf0295, L:/192.168.1.5:42154 - R:192.168.1.5/192.168.1.5:6650]] Connected to server
14:29:02.911 [pulsar-client-io-1-1] WARN org.apache.pulsar.client.impl.ConnectionPool - Failed to open connection to sundar-UX303UB:6650 : java.net.UnknownHostException: failed to resolve 'sundar-UX303UB' after 2 queries
14:29:02.913 [pulsar-client-io-1-1] WARN org.apache.pulsar.client.impl.ConnectionHandler - [persistent://public/default/test] [consumer-test] Error connecting to broker: org.apache.pulsar.client.api.PulsarClientException: java.util.concurrent.CompletionException: java.net.UnknownHostException: failed to resolve 'sundar-UX303UB' after 2 queries
14:29:02.913 [pulsar-client-io-1-1] WARN org.apache.pulsar.client.impl.ConnectionHandler - [persistent://public/default/test] [consumer-test] Could not get connection to broker: org.apache.pulsar.client.api.PulsarClientException: java.util.concurrent.CompletionException: java.net.UnknownHostException: failed to resolve 'sundar-UX303UB' after 2 queries -- Will try again in 0.1 s
```
**To Reproduce**
Steps to reproduce the behavior:
1. Start zookeeper, set cluster meta data, start bookie, start broker after modifying necessary configurations.
2. Bookiesanity is succeeding but simpletest fails with the following error, we are using 2 systems which run zookeeper,bookkeeper and broker
```
bin/bookkeeper shell simpletest --ensemble 2 --writeQuorum 2 --ackQuorum 2 --numEntries 2
JMX enabled by default
14:43:53.364 [main] INFO org.apache.bookkeeper.meta.MetadataDrivers - BookKeeper metadata driver manager initialized
14:43:53.366 [main] INFO org.apache.bookkeeper.meta.MetadataDrivers - BookKeeper metadata driver manager initialized
14:43:53.366 [main] INFO org.apache.bookkeeper.meta.MetadataDrivers - BookKeeper metadata driver manager initialized
14:43:53.373 [main] INFO org.apache.bookkeeper.meta.zk.ZKMetadataDriverBase - Initialize zookeeper metadata driver at metadata service uri zk+null://192.168.1.13:2181;192.168.1.5:2181/ledgers : zkServers = 192.168.1.13:2181,192.168.1.5:2181, ledgersRootPath = /ledgers.
14:43:53.381 [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:zookeeper.version=3.4.13-2d71af4dbe22557fda74f9a9b4309b15a7487f03, built on 06/29/2018 00:39 GMT
14:43:53.381 [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:host.name=sundar-UX303UB
14:43:53.382 [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:java.version=1.8.0_222
14:43:53.382 [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:java.vendor=Private Build
14:43:53.382 [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:java.home=/usr/lib/jvm/java-8-openjdk-amd64/jre
14:43:53.382 [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:java.class.path=/home/sundar/apache-pulsar-2.4.1/conf:::/home/sundar/apache-pulsar-2.4.1/lib/com.beust-jcommander-1.48.jar:/home/sundar/apache-pulsar-2.4.1/lib/com.carrotsearch-hppc-0.7.3.jar:/home/sundar/apache-pulsar-2.4.1/lib/com.fasterxml.jackson.core-jackson-annotations-2.9.9.jar:/home/sundar/apache-pulsar-2.4.1/lib/com.fasterxml.jackson.core-jackson-core-2.9.9.jar:/home/sundar/apache-pulsar-2.4.1/lib/com.fasterxml.jackson.core-jackson-databind-2.9.9.3.jar:/home/sundar/apache-pulsar-2.4.1/lib/com.fasterxml.jackson.dataformat-jackson-dataformat-yaml-2.9.9.jar:/home/sundar/apache-pulsar-2.4.1/lib/com.fasterxml.jackson.jaxrs-jackson-jaxrs-base-2.9.9.jar:/home/sundar/apache-pulsar-2.4.1/lib/com.fasterxml.jackson.jaxrs-jackson-jaxrs-json-provider-2.9.9.jar:/home/sundar/apache-pulsar-2.4.1/lib/com.fasterxml.jackson.module-jackson-module-jaxb-annotations-2.9.9.jar:/home/sundar/apache-pulsar-2.4.1/lib/com.fasterxml.jackson.module-jackson-module-jsonSchema-2.9.9.jar:/home/sundar/apache-pulsar-2.4.1/lib/com.github.ben-manes.caffeine-caffeine-2.6.2.jar:/home/sundar/apache-pulsar-2.4.1/lib/com.github.luben-zstd-jni-1.3.7-3.jar:/home/sundar/apache-pulsar-2.4.1/lib/com.github.zafarkhaja-java-semver-0.9.0.jar:/home/sundar/apache-pulsar-2.4.1/lib/com.google.api.grpc-proto-google-common-protos-1.12.0.jar:/home/sundar/apache-pulsar-2.4.1/lib/com.google.auth-google-auth-library-credentials-0.9.0.jar:/home/sundar/apache-pulsar-2.4.1/lib/com.google.code.findbugs-jsr305-3.0.2.jar:/home/sundar/apache-pulsar-2.4.1/lib/com.google.code.gson-gson-2.8.2.jar:/home/sundar/apache-pulsar-2.4.1/lib/com.google.errorprone-error_prone_annotations-2.2.0.jar:/home/sundar/apache-pulsar-2.4.1/lib/com.google.guava-guava-21.0.jar:/home/sundar/apache-pulsar-2.4.1/lib/com.google.protobuf.nano-protobuf-javanano-3.0.0-alpha-5.jar:/home/sundar/apache-pulsar-2.4.1/lib/com.google.protobuf-protobuf-java-3.5.1.jar:/home/sundar/apache-pulsar-2.4.1/lib/com.google.protobuf-protobuf-java-util-3.5.1.jar:/home/sundar/apache-pulsar-2.4.1/lib/commons-cli-commons-cli-1.2.jar:/home/sundar/apache-pulsar-2.4.1/lib/commons-codec-commons-codec-1.10.jar:/home/sundar/apache-pulsar-2.4.1/lib/commons-collections-commons-collections-3.2.2.jar:/home/sundar/apache-pulsar-2.4.1/lib/commons-configuration-commons-configuration-1.10.jar:/home/sundar/apache-pulsar-2.4.1/lib/commons-io-commons-io-2.5.jar:/home/sundar/apache-pulsar-2.4.1/lib/commons-lang-commons-lang-2.6.jar:/home/sundar/apache-pulsar-2.4.1/lib/commons-logging-commons-logging-1.1.1.jar:/home/sundar/apache-pulsar-2.4.1/lib/com.squareup.okhttp-logging-interceptor-2.7.5.jar:/home/sundar/apache-pulsar-2.4.1/lib/com.squareup.okhttp-okhttp-2.5.0.jar:/home/sundar/apache-pulsar-2.4.1/lib/com.squareup.okhttp-okhttp-ws-2.7.5.jar:/home/sundar/apache-pulsar-2.4.1/lib/com.squareup.okio-okio-1.13.0.jar:/home/sundar/apache-pulsar-2.4.1/lib/com.sun.activation-javax.activation-1.2.0.jar:/home/sundar/apache-pulsar-2.4.1/lib/com.thoughtworks.paranamer-paranamer-2.7.jar:/home/sundar/apache-pulsar-2.4.1/lib/com.typesafe.netty-netty-reactive-streams-2.0.0.jar:/home/sundar/apache-pulsar-2.4.1/lib/com.yahoo.datasketches-memory-0.8.3.jar:/home/sundar/apache-pulsar-2.4.1/lib/com.yahoo.datasketches-sketches-core-0.8.3.jar:/home/sundar/apache-pulsar-2.4.1/lib/io.dropwizard.metrics-metrics-core-3.1.0.jar:/home/sundar/apache-pulsar-2.4.1/lib/io.dropwizard.metrics-metrics-graphite-3.1.0.jar:/home/sundar/apache-pulsar-2.4.1/lib/io.dropwizard.metrics-metrics-jvm-3.1.0.jar:/home/sundar/apache-pulsar-2.4.1/lib/io.grpc-grpc-all-1.18.0.jar:/home/sundar/apache-pulsar-2.4.1/lib/io.grpc-grpc-auth-1.18.0.jar:/home/sundar/apache-pulsar-2.4.1/lib/io.grpc-grpc-context-1.18.0.jar:/home/sundar/apache-pulsar-2.4.1/lib/io.grpc-grpc-core-1.18.0.jar:/home/sundar/apache-pulsar-2.4.1/lib/io.grpc-grpc-netty-1.18.0.jar:/home/sundar/apache-pulsar-2.4.1/lib/io.grpc-grpc-okhttp-1.18.0.jar:/home/sundar/apache-pulsar-2.4.1/lib/io.grpc-grpc-protobuf-1.18.0.jar:/home/sundar/apache-pulsar-2.4.1/lib/io.grpc-grpc-protobuf-lite-1.18.0.jar:/home/sundar/apache-pulsar-2.4.1/lib/io.grpc-grpc-protobuf-nano-1.18.0.jar:/home/sundar/apache-pulsar-2.4.1/lib/io.grpc-grpc-stub-1.18.0.jar:/home/sundar/apache-pulsar-2.4.1/lib/io.grpc-grpc-testing-1.12.0.jar:/home/sundar/apache-pulsar-2.4.1/lib/io.jsonwebtoken-jjwt-api-0.10.5.jar:/home/sundar/apache-pulsar-2.4.1/lib/io.jsonwebtoken-jjwt-impl-0.10.5.jar:/home/sundar/apache-pulsar-2.4.1/lib/io.jsonwebtoken-jjwt-jackson-0.10.5.jar:/home/sundar/apache-pulsar-2.4.1/lib/io.kubernetes-client-java-2.0.0.jar:/home/sundar/apache-pulsar-2.4.1/lib/io.kubernetes-client-java-api-2.0.0.jar:/home/sundar/apache-pulsar-2.4.1/lib/io.kubernetes-client-java-proto-2.0.0.jar:/home/sundar/apache-pulsar-2.4.1/lib/io.netty-netty-3.10.1.Final.jar:/home/sundar/apache-pulsar-2.4.1/lib/io.netty-netty-all-4.1.32.Final.jar:/home/sundar/apache-pulsar-2.4.1/lib/io.netty-netty-tcnative-boringssl-static-2.0.20.Final.jar:/home/sundar/apache-pulsar-2.4.1/lib/io.opencensus-opencensus-api-0.18.0.jar:/home/sundar/apache-pulsar-2.4.1/lib/io.opencensus-opencensus-contrib-grpc-metrics-0.18.0.jar:/home/sundar/apache-pulsar-2.4.1/lib/io.prometheus-simpleclient-0.5.0.jar:/home/sundar/apache-pulsar-2.4.1/lib/io.prometheus-simpleclient_common-0.5.0.jar:/home/sundar/apache-pulsar-2.4.1/lib/io.prometheus-simpleclient_hotspot-0.5.0.jar:/home/sundar/apache-pulsar-2.4.1/lib/io.prometheus-simpleclient_httpserver-0.5.0.jar:/home/sundar/apache-pulsar-2.4.1/lib/io.prometheus-simpleclient_jetty-0.5.0.jar:/home/sundar/apache-pulsar-2.4.1/lib/io.prometheus-simpleclient_log4j2-0.5.0.jar:/home/sundar/apache-pulsar-2.4.1/lib/io.prometheus-simpleclient_servlet-0.5.0.jar:/home/sundar/apache-pulsar-2.4.1/lib/io.swagger-swagger-annotations-1.5.21.jar:/home/sundar/apache-pulsar-2.4.1/lib/io.swagger-swagger-core-1.5.21.jar:/home/sundar/apache-pulsar-2.4.1/lib/io.swagger-swagger-models-1.5.21.jar:/home/sundar/apache-pulsar-2.4.1/lib/io.vertx-vertx-auth-common-3.4.1.jar:/home/sundar/apache-pulsar-2.4.1/lib/io.vertx-vertx-core-3.4.1.jar:/home/sundar/apache-pulsar-2.4.1/lib/io.vertx-vertx-web-3.4.1.jar:/home/sundar/apache-pulsar-2.4.1/lib/javax.annotation-javax.annotation-api-1.2.jar:/home/sundar/apache-pulsar-2.4.1/lib/javax.inject-javax.inject-1.jar:/home/sundar/apache-pulsar-2.4.1/lib/javax.servlet-javax.servlet-api-3.1.0.jar:/home/sundar/apache-pulsar-2.4.1/lib/javax.validation-validation-api-1.1.0.Final.jar:/home/sundar/apache-pulsar-2.4.1/lib/javax.websocket-javax.websocket-client-api-1.0.jar:/home/sundar/apache-pulsar-2.4.1/lib/javax.ws.rs-javax.ws.rs-api-2.1.jar:/home/sundar/apache-pulsar-2.4.1/lib/javax.xml.bind-jaxb-api-2.3.1.jar:/home/sundar/apache-pulsar-2.4.1/lib/jline-jline-0.9.94.jar:/home/sundar/apache-pulsar-2.4.1/lib/joda-time-joda-time-2.10.1.jar:/home/sundar/apache-pulsar-2.4.1/lib/log4j-log4j-1.2.17.jar:/home/sundar/apache-pulsar-2.4.1/lib/net.java.dev.jna-jna-4.2.0.jar:/home/sundar/apache-pulsar-2.4.1/lib/net.jodah-typetools-0.5.0.jar:/home/sundar/apache-pulsar-2.4.1/lib/org.apache.avro-avro-1.8.2.jar:/home/sundar/apache-pulsar-2.4.1/lib/org.apache.avro-avro-protobuf-1.8.2.jar:/home/sundar/apache-pulsar-2.4.1/lib/org.apache.bookkeeper-bookkeeper-common-4.9.2.jar:/home/sundar/apache-pulsar-2.4.1/lib/org.apache.bookkeeper-bookkeeper-common-allocator-4.9.2.jar:/home/sundar/apache-pulsar-2.4.1/lib/org.apache.bookkeeper-bookkeeper-proto-4.9.2.jar:/home/sundar/apache-pulsar-2.4.1/lib/org.apache.bookkeeper-bookkeeper-server-4.9.2.jar:/home/sundar/apache-pulsar-2.4.1/lib/org.apache.bookkeeper-bookkeeper-tools-framework-4.9.2.jar:/home/sundar/apache-pulsar-2.4.1/lib/org.apache.bookkeeper-circe-checksum-4.9.2.jar:/home/sundar/apache-pulsar-2.4.1/lib/org.apache.bookkeeper-cpu-affinity-4.9.2.jar:/home/sundar/apache-pulsar-2.4.1/lib/org.apache.bookkeeper.http-http-server-4.9.2.jar:/home/sundar/apache-pulsar-2.4.1/lib/org.apache.bookkeeper.http-vertx-http-server-4.9.2.jar:/home/sundar/apache-pulsar-2.4.1/lib/org.apache.bookkeeper-statelib-4.9.2.jar:/home/sundar/apache-pulsar-2.4.1/lib/org.apache.bookkeeper.stats-bookkeeper-stats-api-4.9.2.jar:/home/sundar/apache-pulsar-2.4.1/lib/org.apache.bookkeeper.stats-codahale-metrics-provider-4.9.2.jar:/home/sundar/apache-pulsar-2.4.1/lib/org.apache.bookkeeper.stats-prometheus-metrics-provider-4.9.2.jar:/home/sundar/apache-pulsar-2.4.1/lib/org.apache.bookkeeper-stream-storage-api-4.9.2.jar:/home/sundar/apache-pulsar-2.4.1/lib/org.apache.bookkeeper-stream-storage-common-4.9.2.jar:/home/sundar/apache-pulsar-2.4.1/lib/org.apache.bookkeeper-stream-storage-java-client-4.9.2.jar:/home/sundar/apache-pulsar-2.4.1/lib/org.apache.bookkeeper-stream-storage-java-client-base-4.9.2.jar:/home/sundar/apache-pulsar-2.4.1/lib/org.apache.bookkeeper-stream-storage-proto-4.9.2.jar:/home/sundar/apache-pulsar-2.4.1/lib/org.apache.bookkeeper-stream-storage-server-4.9.2.jar:/home/sundar/apache-pulsar-2.4.1/lib/org.apache.bookkeeper-stream-storage-service-api-4.9.2.jar:/home/sundar/apache-pulsar-2.4.1/lib/org.apache.bookkeeper-stream-storage-service-impl-4.9.2.jar:/home/sundar/apache-pulsar-2.4.1/lib/org.apache.bookkeeper.tests-stream-storage-tests-common-4.9.2.jar:/home/sundar/apache-pulsar-2.4.1/lib/org.apache.commons-commons-collections4-4.1.jar:/home/sundar/apache-pulsar-2.4.1/lib/org.apache.commons-commons-compress-1.15.jar:/home/sundar/apache-pulsar-2.4.1/lib/org.apache.commons-commons-lang3-3.4.jar:/home/sundar/apache-pulsar-2.4.1/lib/org.apache.curator-curator-client-4.0.1.jar:/home/sundar/apache-pulsar-2.4.1/lib/org.apache.curator-curator-framework-4.0.1.jar:/home/sundar/apache-pulsar-2.4.1/lib/org.apache.curator-curator-recipes-4.0.1.jar:/home/sundar/apache-pulsar-2.4.1/lib/org.apache.distributedlog-distributedlog-common-4.9.2.jar:/home/sundar/apache-pulsar-2.4.1/lib/org.apache.distributedlog-distributedlog-core-4.9.2.jar:/home/sundar/apache-pulsar-2.4.1/lib/org.apache.distributedlog-distributedlog-core-4.9.2-tests.jar:/home/sundar/apache-pulsar-2.4.1/lib/org.apache.distributedlog-distributedlog-protocol-4.9.2.jar:/home/sundar/apache-pulsar-2.4.1/lib/org.apache.httpcomponents-httpclient-4.5.5.jar:/home/sundar/apache-pulsar-2.4.1/lib/org.apache.httpcomponents-httpcore-4.4.9.jar:/home/sundar/apache-pulsar-2.4.1/lib/org.apache.logging.log4j-log4j-api-2.10.0.jar:/home/sundar/apache-pulsar-2.4.1/lib/org.apache.logging.log4j-log4j-core-2.10.0.jar:/home/sundar/apache-pulsar-2.4.1/lib/org.apache.logging.log4j-log4j-slf4j-impl-2.10.0.jar:/home/sundar/apache-pulsar-2.4.1/lib/org.apache.logging.log4j-log4j-web-2.10.0.jar:/home/sundar/apache-pulsar-2.4.1/lib/org.apache.pulsar-managed-ledger-original-2.4.1.jar:/home/sundar/apache-pulsar-2.4.1/lib/org.apache.pulsar-protobuf-shaded-2.1.0-incubating.jar:/home/sundar/apache-pulsar-2.4.1/lib/org.apache.pulsar-pulsar-broker-2.4.1.jar:/home/sundar/apache-pulsar-2.4.1/lib/org.apache.pulsar-pulsar-broker-auth-sasl-2.4.1.jar:/home/sundar/apache-pulsar-2.4.1/lib/org.apache.pulsar-pulsar-broker-common-2.4.1.jar:/home/sundar/apache-pulsar-2.4.1/lib/org.apache.pulsar-pulsar-client-admin-original-2.4.1.jar:/home/sundar/apache-pulsar-2.4.1/lib/org.apache.pulsar-pulsar-client-api-2.4.1.jar:/home/sundar/apache-pulsar-2.4.1/lib/org.apache.pulsar-pulsar-client-auth-sasl-2.4.1.jar:/home/sundar/apache-pulsar-2.4.1/lib/org.apache.pulsar-pulsar-client-original-2.4.1.jar:/home/sundar/apache-pulsar-2.4.1/lib/org.apache.pulsar-pulsar-client-tools-2.4.1.jar:/home/sundar/apache-pulsar-2.4.1/lib/org.apache.pulsar-pulsar-common-2.4.1.jar:/home/sundar/apache-pulsar-2.4.1/lib/org.apache.pulsar-pulsar-discovery-service-2.4.1.jar:/home/sundar/apache-pulsar-2.4.1/lib/org.apache.pulsar-pulsar-functions-api-2.4.1.jar:/home/sundar/apache-pulsar-2.4.1/lib/org.apache.pulsar-pulsar-functions-api-examples-2.4.1.jar:/home/sundar/apache-pulsar-2.4.1/lib/org.apache.pulsar-pulsar-functions-instance-2.4.1.jar:/home/sundar/apache-pulsar-2.4.1/lib/org.apache.pulsar-pulsar-functions-local-runner-original-2.4.1.jar:/home/sundar/apache-pulsar-2.4.1/lib/org.apache.pulsar-pulsar-functions-proto-2.4.1.jar:/home/sundar/apache-pulsar-2.4.1/lib/org.apache.pulsar-pulsar-functions-runtime-2.4.1.jar:/home/sundar/apache-pulsar-2.4.1/lib/org.apache.pulsar-pulsar-functions-secrets-2.4.1.jar:/home/sundar/apache-pulsar-2.4.1/lib/org.apache.pulsar-pulsar-functions-utils-2.4.1.jar:/home/sundar/apache-pulsar-2.4.1/lib/org.apache.pulsar-pulsar-functions-worker-2.4.1.jar:/home/sundar/apache-pulsar-2.4.1/lib/org.apache.pulsar-pulsar-io-core-2.4.1.jar:/home/sundar/apache-pulsar-2.4.1/lib/org.apache.pulsar-pulsar-proxy-2.4.1.jar:/home/sundar/apache-pulsar-2.4.1/lib/org.apache.pulsar-pulsar-testclient-2.4.1.jar:/home/sundar/apache-pulsar-2.4.1/lib/org.apache.pulsar-pulsar-transaction-common-2.4.1.jar:/home/sundar/apache-pulsar-2.4.1/lib/org.apache.pulsar-pulsar-websocket-2.4.1.jar:/home/sundar/apache-pulsar-2.4.1/lib/org.apache.pulsar-pulsar-zookeeper-2.4.1.jar:/home/sundar/apache-pulsar-2.4.1/lib/org.apache.pulsar-pulsar-zookeeper-utils-2.4.1.jar:/home/sundar/apache-pulsar-2.4.1/lib/org.apache.yetus-audience-annotations-0.5.0.jar:/home/sundar/apache-pulsar-2.4.1/lib/org.aspectj-aspectjrt-1.9.2.jar:/home/sundar/apache-pulsar-2.4.1/lib/org.aspectj-aspectjweaver-1.9.2.jar:/home/sundar/apache-pulsar-2.4.1/lib/org.asynchttpclient-async-http-client-2.7.0.jar:/home/sundar/apache-pulsar-2.4.1/lib/org.asynchttpclient-async-http-client-netty-utils-2.7.0.jar:/home/sundar/apache-pulsar-2.4.1/lib/org.bouncycastle-bcpkix-jdk15on-1.60.jar:/home/sundar/apache-pulsar-2.4.1/lib/org.bouncycastle-bcprov-ext-jdk15on-1.60.jar:/home/sundar/apache-pulsar-2.4.1/lib/org.bouncycastle-bcprov-jdk15on-1.60.jar:/home/sundar/apache-pulsar-2.4.1/lib/org.checkerframework-checker-compat-qual-2.5.2.jar:/home/sundar/apache-pulsar-2.4.1/lib/org.codehaus.jackson-jackson-core-asl-1.9.13.jar:/home/sundar/apache-pulsar-2.4.1/lib/org.codehaus.jackson-jackson-mapper-asl-1.9.13.jar:/home/sundar/apache-pulsar-2.4.1/lib/org.codehaus.mojo-animal-sniffer-annotations-1.17.jar:/home/sundar/apache-pulsar-2.4.1/lib/org.eclipse.jetty-jetty-client-9.4.12.v20180830.jar:/home/sundar/apache-pulsar-2.4.1/lib/org.eclipse.jetty-jetty-continuation-9.4.12.v20180830.jar:/home/sundar/apache-pulsar-2.4.1/lib/org.eclipse.jetty-jetty-http-9.4.12.v20180830.jar:/home/sundar/apache-pulsar-2.4.1/lib/org.eclipse.jetty-jetty-io-9.4.12.v20180830.jar:/home/sundar/apache-pulsar-2.4.1/lib/org.eclipse.jetty-jetty-proxy-9.4.12.v20180830.jar:/home/sundar/apache-pulsar-2.4.1/lib/org.eclipse.jetty-jetty-security-9.4.12.v20180830.jar:/home/sundar/apache-pulsar-2.4.1/lib/org.eclipse.jetty-jetty-server-9.4.12.v20180830.jar:/home/sundar/apache-pulsar-2.4.1/lib/org.eclipse.jetty-jetty-servlet-9.4.12.v20180830.jar:/home/sundar/apache-pulsar-2.4.1/lib/org.eclipse.jetty-jetty-servlets-9.4.12.v20180830.jar:/home/sundar/apache-pulsar-2.4.1/lib/org.eclipse.jetty-jetty-util-9.4.12.v20180830.jar:/home/sundar/apache-pulsar-2.4.1/lib/org.eclipse.jetty-jetty-xml-9.4.12.v20180830.jar:/home/sundar/apache-pulsar-2.4.1/lib/org.eclipse.jetty.websocket-javax-websocket-client-impl-9.4.12.v20180830.jar:/home/sundar/apache-pulsar-2.4.1/lib/org.eclipse.jetty.websocket-websocket-api-9.4.12.v20180830.jar:/home/sundar/apache-pulsar-2.4.1/lib/org.eclipse.jetty.websocket-websocket-client-9.4.12.v20180830.jar:/home/sundar/apache-pulsar-2.4.1/lib/org.eclipse.jetty.websocket-websocket-common-9.4.12.v20180830.jar:/home/sundar/apache-pulsar-2.4.1/lib/org.eclipse.jetty.websocket-websocket-server-9.4.12.v20180830.jar:/home/sundar/apache-pulsar-2.4.1/lib/org.eclipse.jetty.websocket-websocket-servlet-9.4.12.v20180830.jar:/home/sundar/apache-pulsar-2.4.1/lib/org.glassfish.hk2.external-aopalliance-repackaged-2.5.0-b42.jar:/home/sundar/apache-pulsar-2.4.1/lib/org.glassfish.hk2.external-javax.inject-2.5.0-b42.jar:/home/sundar/apache-pulsar-2.4.1/lib/org.glassfish.hk2-hk2-api-2.5.0-b42.jar:/home/sundar/apache-pulsar-2.4.1/lib/org.glassfish.hk2-hk2-locator-2.5.0-b42.jar:/home/sundar/apache-pulsar-2.4.1/lib/org.glassfish.hk2-hk2-utils-2.5.0-b42.jar:/home/sundar/apache-pulsar-2.4.1/lib/org.glassfish.hk2-osgi-resource-locator-1.0.1.jar:/home/sundar/apache-pulsar-2.4.1/lib/org.glassfish.jersey.containers-jersey-container-servlet-2.27.jar:/home/sundar/apache-pulsar-2.4.1/lib/org.glassfish.jersey.containers-jersey-container-servlet-core-2.27.jar:/home/sundar/apache-pulsar-2.4.1/lib/org.glassfish.jersey.core-jersey-client-2.27.jar:/home/sundar/apache-pulsar-2.4.1/lib/org.glassfish.jersey.core-jersey-common-2.27.jar:/home/sundar/apache-pulsar-2.4.1/lib/org.glassfish.jersey.core-jersey-server-2.27.jar:/home/sundar/apache-pulsar-2.4.1/lib/org.glassfish.jersey.ext-jersey-entity-filtering-2.27.jar:/home/sundar/apache-pulsar-2.4.1/lib/org.glassfish.jersey.inject-jersey-hk2-2.27.jar:/home/sundar/apache-pulsar-2.4.1/lib/org.glassfish.jersey.media-jersey-media-jaxb-2.27.jar:/home/sundar/apache-pulsar-2.4.1/lib/org.glassfish.jersey.media-jersey-media-json-jackson-2.27.jar:/home/sundar/apache-pulsar-2.4.1/lib/org.glassfish.jersey.media-jersey-media-multipart-2.27.jar:/home/sundar/apache-pulsar-2.4.1/lib/org.hdrhistogram-HdrHistogram-2.1.9.jar:/home/sundar/apache-pulsar-2.4.1/lib/org.inferred-freebuilder-1.14.9.jar:/home/sundar/apache-pulsar-2.4.1/lib/org.javassist-javassist-3.25.0-GA.jar:/home/sundar/apache-pulsar-2.4.1/lib/org.jctools-jctools-core-2.1.2.jar:/home/sundar/apache-pulsar-2.4.1/lib/org.jvnet.mimepull-mimepull-1.9.6.jar:/home/sundar/apache-pulsar-2.4.1/lib/org.lz4-lz4-java-1.5.0.jar:/home/sundar/apache-pulsar-2.4.1/lib/org.objenesis-objenesis-2.6.jar:/home/sundar/apache-pulsar-2.4.1/lib/org.reactivestreams-reactive-streams-1.0.2.jar:/home/sundar/apache-pulsar-2.4.1/lib/org.rocksdb-rocksdbjni-5.13.3.jar:/home/sundar/apache-pulsar-2.4.1/lib/org.slf4j-jcl-over-slf4j-1.7.25.jar:/home/sundar/apache-pulsar-2.4.1/lib/org.slf4j-jul-to-slf4j-1.7.25.jar:/home/sundar/apache-pulsar-2.4.1/lib/org.slf4j-slf4j-api-1.7.25.jar:/home/sundar/apache-pulsar-2.4.1/lib/org.tukaani-xz-1.5.jar:/home/sundar/apache-pulsar-2.4.1/lib/org.xerial.snappy-snappy-java-1.1.1.3.jar:/home/sundar/apache-pulsar-2.4.1/lib/org.yaml-snakeyaml-1.23.jar:
14:43:53.384 [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:java.library.path=:/home/sundar/ns-allinone-2.35/otcl-1.14/:/home/sundar/ns-allinone-2.35/lib/:/usr/local/lib/:/usr/java/packages/lib/amd64:/usr/lib/x86_64-linux-gnu/jni:/lib/x86_64-linux-gnu:/usr/lib/x86_64-linux-gnu:/usr/lib/jni:/lib:/usr/lib
14:43:53.384 [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:java.io.tmpdir=/tmp
14:43:53.384 [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:java.compiler=<NA>
14:43:53.384 [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:os.name=Linux
14:43:53.384 [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:os.arch=amd64
14:43:53.384 [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:os.version=4.15.0-66-generic
14:43:53.384 [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:user.name=sundar
14:43:53.384 [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:user.home=/home/sundar
14:43:53.384 [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:user.dir=/home/sundar/apache-pulsar-2.4.1
14:43:53.389 [main] INFO org.apache.zookeeper.ZooKeeper - Initiating client connection, connectString=192.168.1.13:2181,192.168.1.5:2181 sessionTimeout=30000 watcher=org.apache.bookkeeper.zookeeper.ZooKeeperWatcherBase@50468873
14:43:53.510 [main-SendThread(192.168.1.5:2181)] INFO org.apache.zookeeper.ClientCnxn - Opening socket connection to server 192.168.1.5/192.168.1.5:2181. Will not attempt to authenticate using SASL (unknown error)
14:43:53.517 [main-SendThread(192.168.1.5:2181)] INFO org.apache.zookeeper.ClientCnxn - Socket connection established to 192.168.1.5/192.168.1.5:2181, initiating session
14:43:53.536 [main-SendThread(192.168.1.5:2181)] INFO org.apache.zookeeper.ClientCnxn - Session establishment complete on server 192.168.1.5/192.168.1.5:2181, sessionid = 0x2000535ffc2000e, negotiated timeout = 30000
14:43:53.541 [main-EventThread] INFO org.apache.bookkeeper.zookeeper.ZooKeeperWatcherBase - ZooKeeper client is connected now.
14:43:54.506 [main] INFO org.apache.bookkeeper.client.RackawareEnsemblePlacementPolicyImpl - Failed to initialize DNS Resolver org.apache.bookkeeper.net.ScriptBasedMapping, used default subnet resolver : java.lang.RuntimeException: No network topology script is found when using script based DNS resolver.
14:43:54.539 [main] INFO org.apache.bookkeeper.client.RackawareEnsemblePlacementPolicyImpl - Initialize rackaware ensemble placement policy @ <Bookie:127.0.1.1:0> @ /default-rack : org.apache.bookkeeper.client.RackawareEnsemblePlacementPolicyImpl$DefaultResolver.
14:43:54.540 [main] INFO org.apache.bookkeeper.client.RackawareEnsemblePlacementPolicyImpl - Not weighted
14:43:54.548 [main] INFO org.apache.bookkeeper.client.BookKeeper - Weighted ledger placement is not enabled
14:43:54.592 [BookKeeperClientScheduler-OrderedScheduler-0-0] INFO org.apache.bookkeeper.net.NetworkTopologyImpl - Adding a new node: /default-rack/sundar-UX303UB:3181
14:43:54.593 [BookKeeperClientScheduler-OrderedScheduler-0-0] INFO org.apache.bookkeeper.net.NetworkTopologyImpl - Adding a new node: /default-rack/vishal-Satellite-C55-B:3181
14:43:54.641 [main] WARN org.apache.bookkeeper.client.BookieWatcherImpl - New ensemble: [vishal-Satellite-C55-B:3181, sundar-UX303UB:3181] is not adhering to Placement Policy. quarantinedBookies: []
14:43:54.808 [main-EventThread] ERROR org.apache.bookkeeper.proto.checksum.CRC32CDigestManager - Sse42Crc32C is not supported, will use a slower CRC32C implementation.
14:43:54.818 [main-EventThread] INFO org.apache.bookkeeper.client.LedgerCreateOp - Ensemble: [vishal-Satellite-C55-B:3181, sundar-UX303UB:3181] for ledger: 1371
Ledger ID: 1371
14:43:54.888 [BookKeeperClientWorker-OrderedExecutor-3-0] WARN com.scurrilous.circe.checksum.Crc32cIntChecksum - Failed to load Circe JNI library. Falling back to Java based CRC32c provider
14:43:55.006 [bookkeeper-io-3-1] WARN org.apache.bookkeeper.proto.PerChannelBookieClient - Could not connect to bookie: [id: 0x6515e025]/vishal-Satellite-C55-B:3181, current state CONNECTING : vishal-Satellite-C55-B
14:43:55.008 [bookkeeper-io-3-2] INFO org.apache.bookkeeper.proto.PerChannelBookieClient - Successfully connected to bookie: [id: 0xdb053f5e, L:/127.0.0.1:41224 - R:sundar-UX303UB/127.0.1.1:3181]
14:43:55.019 [bookkeeper-io-3-2] INFO org.apache.bookkeeper.proto.PerChannelBookieClient - Successfully connected to bookie: sundar-UX303UB:3181
14:43:55.020 [BookKeeperClientWorker-OrderedExecutor-3-0] WARN org.apache.bookkeeper.client.PendingAddOp - Failed to write entry (1371, 0): Bookie handle is not available
14:43:55.037 [BookKeeperClientWorker-OrderedExecutor-3-0] WARN org.apache.bookkeeper.client.RackawareEnsemblePlacementPolicyImpl - Failed to find 1 bookies : excludeBookies [<Bookie:sundar-UX303UB:3181>, <Bookie:vishal-Satellite-C55-B:3181>], allBookies [<Bookie:sundar-UX303UB:3181>, <Bookie:vishal-Satellite-C55-B:3181>].
14:43:55.039 [BookKeeperClientWorker-OrderedExecutor-3-0] WARN org.apache.bookkeeper.client.RackawareEnsemblePlacementPolicyImpl - Failed to choose a bookie: excluded [<Bookie:sundar-UX303UB:3181>, <Bookie:vishal-Satellite-C55-B:3181>], fallback to choose bookie randomly from the cluster.
14:43:55.039 [BookKeeperClientWorker-OrderedExecutor-3-0] WARN org.apache.bookkeeper.client.RackawareEnsemblePlacementPolicyImpl - Failed to find 1 bookies : excludeBookies [<Bookie:sundar-UX303UB:3181>, <Bookie:vishal-Satellite-C55-B:3181>], allBookies [<Bookie:vishal-Satellite-C55-B:3181>, <Bookie:sundar-UX303UB:3181>].
14:43:55.040 [BookKeeperClientWorker-OrderedExecutor-3-0] WARN org.apache.bookkeeper.client.RackawareEnsemblePlacementPolicyImpl - Failed to find 1 bookies : excludeBookies [<Bookie:sundar-UX303UB:3181>, <Bookie:vishal-Satellite-C55-B:3181>], allBookies [<Bookie:sundar-UX303UB:3181>, <Bookie:vishal-Satellite-C55-B:3181>].
14:43:55.043 [BookKeeperClientWorker-OrderedExecutor-3-0] WARN org.apache.bookkeeper.client.RackawareEnsemblePlacementPolicyImpl - Failed to choose a bookie: excluded [<Bookie:sundar-UX303UB:3181>, <Bookie:vishal-Satellite-C55-B:3181>], fallback to choose bookie randomly from the cluster.
14:43:55.043 [BookKeeperClientWorker-OrderedExecutor-3-0] WARN org.apache.bookkeeper.client.RackawareEnsemblePlacementPolicyImpl - Failed to find 1 bookies : excludeBookies [<Bookie:sundar-UX303UB:3181>, <Bookie:vishal-Satellite-C55-B:3181>], allBookies [<Bookie:vishal-Satellite-C55-B:3181>, <Bookie:sundar-UX303UB:3181>].
14:43:55.043 [BookKeeperClientWorker-OrderedExecutor-3-0] ERROR org.apache.bookkeeper.client.MetadataUpdateLoop - UpdateLoop(ledgerId=1371,loopId=2d2b0a86) Exception updating
org.apache.bookkeeper.client.BKException$BKNotEnoughBookiesException: Not enough non-faulty bookies available
at org.apache.bookkeeper.client.RackawareEnsemblePlacementPolicyImpl.selectRandomInternal(RackawareEnsemblePlacementPolicyImpl.java:989) ~[org.apache.bookkeeper-bookkeeper-server-4.9.2.jar:4.9.2]
at org.apache.bookkeeper.client.RackawareEnsemblePlacementPolicyImpl.selectRandom(RackawareEnsemblePlacementPolicyImpl.java:907) ~[org.apache.bookkeeper-bookkeeper-server-4.9.2.jar:4.9.2]
at org.apache.bookkeeper.client.RackawareEnsemblePlacementPolicyImpl.selectFromNetworkLocation(RackawareEnsemblePlacementPolicyImpl.java:797) ~[org.apache.bookkeeper-bookkeeper-server-4.9.2.jar:4.9.2]
at org.apache.bookkeeper.client.RackawareEnsemblePlacementPolicy.selectFromNetworkLocation(RackawareEnsemblePlacementPolicy.java:200) ~[org.apache.bookkeeper-bookkeeper-server-4.9.2.jar:4.9.2]
at org.apache.bookkeeper.client.RackawareEnsemblePlacementPolicyImpl.selectFromNetworkLocation(RackawareEnsemblePlacementPolicyImpl.java:757) ~[org.apache.bookkeeper-bookkeeper-server-4.9.2.jar:4.9.2]
at org.apache.bookkeeper.client.RackawareEnsemblePlacementPolicy.selectFromNetworkLocation(RackawareEnsemblePlacementPolicy.java:221) ~[org.apache.bookkeeper-bookkeeper-server-4.9.2.jar:4.9.2]
at org.apache.bookkeeper.client.RackawareEnsemblePlacementPolicyImpl.replaceBookie(RackawareEnsemblePlacementPolicyImpl.java:659) ~[org.apache.bookkeeper-bookkeeper-server-4.9.2.jar:4.9.2]
at org.apache.bookkeeper.client.RackawareEnsemblePlacementPolicy.replaceBookie(RackawareEnsemblePlacementPolicy.java:114) ~[org.apache.bookkeeper-bookkeeper-server-4.9.2.jar:4.9.2]
at org.apache.bookkeeper.client.BookieWatcherImpl.replaceBookie(BookieWatcherImpl.java:295) ~[org.apache.bookkeeper-bookkeeper-server-4.9.2.jar:4.9.2]
at org.apache.bookkeeper.client.EnsembleUtils.replaceBookiesInEnsemble(EnsembleUtils.java:71) ~[org.apache.bookkeeper-bookkeeper-server-4.9.2.jar:4.9.2]
at org.apache.bookkeeper.client.LedgerHandle.lambda$ensembleChangeLoop$2(LedgerHandle.java:1908) ~[org.apache.bookkeeper-bookkeeper-server-4.9.2.jar:4.9.2]
at org.apache.bookkeeper.client.MetadataUpdateLoop.writeLoop(MetadataUpdateLoop.java:122) [org.apache.bookkeeper-bookkeeper-server-4.9.2.jar:4.9.2]
at org.apache.bookkeeper.client.MetadataUpdateLoop.run(MetadataUpdateLoop.java:111) [org.apache.bookkeeper-bookkeeper-server-4.9.2.jar:4.9.2]
at org.apache.bookkeeper.client.LedgerHandle.ensembleChangeLoop(LedgerHandle.java:1927) [org.apache.bookkeeper-bookkeeper-server-4.9.2.jar:4.9.2]
at org.apache.bookkeeper.client.LedgerHandle.handleBookieFailure(LedgerHandle.java:1876) [org.apache.bookkeeper-bookkeeper-server-4.9.2.jar:4.9.2]
at org.apache.bookkeeper.client.PendingAddOp.writeComplete(PendingAddOp.java:360) [org.apache.bookkeeper-bookkeeper-server-4.9.2.jar:4.9.2]
at org.apache.bookkeeper.proto.BookieClientImpl$1.safeRun(BookieClientImpl.java:280) [org.apache.bookkeeper-bookkeeper-server-4.9.2.jar:4.9.2]
at org.apache.bookkeeper.common.util.SafeRunnable.run(SafeRunnable.java:36) [org.apache.bookkeeper-bookkeeper-common-4.9.2.jar:4.9.2]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_222]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_222]
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) [io.netty-netty-all-4.1.32.Final.jar:4.1.32.Final]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_222]
14:43:55.052 [bookkeeper-io-3-2] INFO org.apache.bookkeeper.proto.PerChannelBookieClient - connection [id: 0xdb053f5e, L:/127.0.0.1:41224 - R:sundar-UX303UB/127.0.1.1:3181] authenticated as BookKeeperPrincipal{ANONYMOUS}
14:43:55.056 [BookKeeperClientWorker-OrderedExecutor-3-0] WARN org.apache.bookkeeper.client.LedgerHandle - [EnsembleChange(ledger:1371, change-id:0000000001)][attempt:1] Exception changing ensemble
org.apache.bookkeeper.client.BKException$BKNotEnoughBookiesException: Not enough non-faulty bookies available
at org.apache.bookkeeper.client.RackawareEnsemblePlacementPolicyImpl.selectRandomInternal(RackawareEnsemblePlacementPolicyImpl.java:989) ~[org.apache.bookkeeper-bookkeeper-server-4.9.2.jar:4.9.2]
at org.apache.bookkeeper.client.RackawareEnsemblePlacementPolicyImpl.selectRandom(RackawareEnsemblePlacementPolicyImpl.java:907) ~[org.apache.bookkeeper-bookkeeper-server-4.9.2.jar:4.9.2]
at org.apache.bookkeeper.client.RackawareEnsemblePlacementPolicyImpl.selectFromNetworkLocation(RackawareEnsemblePlacementPolicyImpl.java:797) ~[org.apache.bookkeeper-bookkeeper-server-4.9.2.jar:4.9.2]
at org.apache.bookkeeper.client.RackawareEnsemblePlacementPolicy.selectFromNetworkLocation(RackawareEnsemblePlacementPolicy.java:200) ~[org.apache.bookkeeper-bookkeeper-server-4.9.2.jar:4.9.2]
at org.apache.bookkeeper.client.RackawareEnsemblePlacementPolicyImpl.selectFromNetworkLocation(RackawareEnsemblePlacementPolicyImpl.java:757) ~[org.apache.bookkeeper-bookkeeper-server-4.9.2.jar:4.9.2]
at org.apache.bookkeeper.client.RackawareEnsemblePlacementPolicy.selectFromNetworkLocation(RackawareEnsemblePlacementPolicy.java:221) ~[org.apache.bookkeeper-bookkeeper-server-4.9.2.jar:4.9.2]
at org.apache.bookkeeper.client.RackawareEnsemblePlacementPolicyImpl.replaceBookie(RackawareEnsemblePlacementPolicyImpl.java:659) ~[org.apache.bookkeeper-bookkeeper-server-4.9.2.jar:4.9.2]
at org.apache.bookkeeper.client.RackawareEnsemblePlacementPolicy.replaceBookie(RackawareEnsemblePlacementPolicy.java:114) ~[org.apache.bookkeeper-bookkeeper-server-4.9.2.jar:4.9.2]
at org.apache.bookkeeper.client.BookieWatcherImpl.replaceBookie(BookieWatcherImpl.java:295) ~[org.apache.bookkeeper-bookkeeper-server-4.9.2.jar:4.9.2]
at org.apache.bookkeeper.client.EnsembleUtils.replaceBookiesInEnsemble(EnsembleUtils.java:71) ~[org.apache.bookkeeper-bookkeeper-server-4.9.2.jar:4.9.2]
at org.apache.bookkeeper.client.LedgerHandle.lambda$ensembleChangeLoop$2(LedgerHandle.java:1908) ~[org.apache.bookkeeper-bookkeeper-server-4.9.2.jar:4.9.2]
at org.apache.bookkeeper.client.MetadataUpdateLoop.writeLoop(MetadataUpdateLoop.java:122) ~[org.apache.bookkeeper-bookkeeper-server-4.9.2.jar:4.9.2]
at org.apache.bookkeeper.client.MetadataUpdateLoop.run(MetadataUpdateLoop.java:111) ~[org.apache.bookkeeper-bookkeeper-server-4.9.2.jar:4.9.2]
at org.apache.bookkeeper.client.LedgerHandle.ensembleChangeLoop(LedgerHandle.java:1927) ~[org.apache.bookkeeper-bookkeeper-server-4.9.2.jar:4.9.2]
at org.apache.bookkeeper.client.LedgerHandle.handleBookieFailure(LedgerHandle.java:1876) ~[org.apache.bookkeeper-bookkeeper-server-4.9.2.jar:4.9.2]
at org.apache.bookkeeper.client.PendingAddOp.writeComplete(PendingAddOp.java:360) ~[org.apache.bookkeeper-bookkeeper-server-4.9.2.jar:4.9.2]
at org.apache.bookkeeper.proto.BookieClientImpl$1.safeRun(BookieClientImpl.java:280) ~[org.apache.bookkeeper-bookkeeper-server-4.9.2.jar:4.9.2]
at org.apache.bookkeeper.common.util.SafeRunnable.run(SafeRunnable.java:36) ~[org.apache.bookkeeper-bookkeeper-common-4.9.2.jar:4.9.2]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_222]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_222]
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) [io.netty-netty-all-4.1.32.Final.jar:4.1.32.Final]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_222]
14:43:55.058 [BookKeeperClientWorker-OrderedExecutor-3-0] ERROR org.apache.bookkeeper.client.LedgerHandle - Closing ledger 1371 due to NotEnoughBookiesException: Not enough non-faulty bookies available
14:43:55.061 [BookKeeperClientWorker-OrderedExecutor-3-0] ERROR org.apache.bookkeeper.client.PendingAddOp - Write of ledger entry to quorum failed: L1371 E0
14:43:55.186 [main] INFO org.apache.bookkeeper.proto.PerChannelBookieClient - Closing the per channel bookie client for sundar-UX303UB:3181
14:43:55.190 [main] INFO org.apache.bookkeeper.proto.PerChannelBookieClient - Closing the per channel bookie client for vishal-Satellite-C55-B:3181
14:43:55.190 [bookkeeper-io-3-2] INFO org.apache.bookkeeper.proto.PerChannelBookieClient - Disconnected from bookie channel [id: 0xdb053f5e, L:/127.0.0.1:41224 ! R:sundar-UX303UB/127.0.1.1:3181]
14:43:55.210 [main] INFO org.apache.zookeeper.ZooKeeper - Session: 0x2000535ffc2000e closed
14:43:55.210 [main-EventThread] INFO org.apache.zookeeper.ClientCnxn - EventThread shut down for session: 0x2000535ffc2000e
14:43:55.211 [main] ERROR org.apache.bookkeeper.tools.cli.helpers.ClientCommand - Failed to process command 'simpletest'
org.apache.bookkeeper.client.BKException$BKNotEnoughBookiesException: Not enough non-faulty bookies available
at org.apache.bookkeeper.client.SyncCallbackUtils.finish(SyncCallbackUtils.java:83) ~[org.apache.bookkeeper-bookkeeper-server-4.9.2.jar:4.9.2]
at org.apache.bookkeeper.client.SyncCallbackUtils$SyncAddCallback.addComplete(SyncCallbackUtils.java:251) ~[org.apache.bookkeeper-bookkeeper-server-4.9.2.jar:4.9.2]
at org.apache.bookkeeper.client.AsyncCallback$AddCallback.addCompleteWithLatency(AsyncCallback.java:91) ~[org.apache.bookkeeper-bookkeeper-server-4.9.2.jar:4.9.2]
at org.apache.bookkeeper.client.PendingAddOp.submitCallback(PendingAddOp.java:390) ~[org.apache.bookkeeper-bookkeeper-server-4.9.2.jar:4.9.2]
at org.apache.bookkeeper.client.LedgerHandle.errorOutPendingAdds(LedgerHandle.java:1772) ~[org.apache.bookkeeper-bookkeeper-server-4.9.2.jar:4.9.2]
at org.apache.bookkeeper.client.LedgerHandle$5.safeRun(LedgerHandle.java:572) ~[org.apache.bookkeeper-bookkeeper-server-4.9.2.jar:4.9.2]
at org.apache.bookkeeper.common.util.SafeRunnable.run(SafeRunnable.java:36) ~[org.apache.bookkeeper-bookkeeper-common-4.9.2.jar:4.9.2]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[?:1.8.0_222]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[?:1.8.0_222]
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ~[io.netty-netty-all-4.1.32.Final.jar:4.1.32.Final]
at java.lang.Thread.run(Thread.java:748) ~[?:1.8.0_222]
```
3. Try running the producer and consumer commands on different broker systems after modifying the service urls in client.conf(we are using the following url webServiceUrl=http://192.168.1.13:8080,192.168.1.5:8080, brokerServiceUrl=pulsar://192.168.1.13:6650,192.168.1.5:6650)
**Expected behavior**
Producer should produce message and Consumer should consume it.
**Desktop (please complete the following information):**
- OS: Ubuntu 16.04, Ubuntu 18.04
**Additional context**
All of Zookeeper, Bookkeeper and Broker are starting, we are able to see all the processes running when we give jps command. But when we try to run producer or consumer command consumer broker crashes and error message is displayed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services