You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by A A <an...@hotmail.com> on 2014/03/11 03:17:31 UTC
Remote Zookeeper
Hi
Pretty new to Kafka. Have been successful in installing Kafka 0.8.0.
I am just wondering how should I make my kafka cluster (2 brokers) connect to a single remote zookeper server?
I am using $KAFKA/kafka-server-start.sh $KAFKA_CONFIG/server.properties on both the brokers to start them up
and I have a remote zookeeper running at 192.168.1.120
Can anyone help?
A
RE: Remote Zookeeper
Posted by A A <an...@hotmail.com>.
I have now added host.name property for Broker1 and Broker2 and it works.
Maybe someone can modify the wiki and add this for Step 6?
Also, I would like to mention, as previous replied to my thread; I didn't HAVE_TO add all the brokers to the brokers-list
$KAFKA_HOME/bin/kafka-console-producer.sh --broker-list 192.168.1.124:9092 --topic replicated-topic
$KAFKA_HOME/bin/kafka-console-producer.sh --broker-list 192.168.1.123:9092 --topic replicated-topic
$KAFKA_HOME/bin/kafka-console-producer.sh --broker-list 192.168.1.123:9092,192.168.1.124:9092 --topic replicated-topic
All of the above worked for me.
> From: andthereitgoes@hotmail.com
> To: users@kafka.apache.org
> Subject: RE: Remote Zookeeper
> Date: Tue, 11 Mar 2014 18:18:31 +0000
>
> Thanks. I already checked out the wiki and step 6 in particular.
> Just to clarify, Zk, Broker1 and Broker2 are on 3 different physical machines.
>
> > From: b@b3k.us
> > Date: Tue, 11 Mar 2014 10:58:50 -0700
> > Subject: Re: Remote Zookeeper
> > To: users@kafka.apache.org
> >
> > "I'd suggest deleting the zookeeper and Kafka logs and starting over using
> > the getting started tutorial from the wiki."
> >
> > |
> > v
> >
> > "Could this be an issue?
> > INFO Registered broker 2 at path /brokers/ids/2 with address localhost:9092.
> > INFO Registered broker 1 at path /brokers/ids/1 with address
> > localhost:9092."
> >
> > yes. see step 6: https://kafka.apache.org/documentation.html#quickstart
> >
> >
> >
> > On Tue, Mar 11, 2014 at 10:54 AM, A A <an...@hotmail.com> wrote:
> >
> > > Okay thanks. I removed zookeeper and kafka logs
> > >
> > > /tmp/kafka-logs
> > > /tmp/logs
> > > /var/zookeeper/version-2/
> > > /tmp/zookeeper*.log (on Zk)
> > >
> > > restarted zookeeper
> > >
> > > $KAFKA_HOME/bin/kafka-list-topic.sh --zookeeper 192.168.1.120:2181
> > > no topics exist!
> > >
> > > Started Broker 2
> > > $KAFKA/kafka-server-start.sh $KAFKA_CONFIG/server.properties
> > > [2014-03-11 12:05:32,651] INFO Verifying properties
> > > (kafka.utils.VerifiableProperties)
> > > [2014-03-11 12:05:32,681] INFO Property port is overridden to 9092
> > > (kafka.utils.VerifiableProperties)
> > > [2014-03-11 12:05:32,681] INFO Property socket.request.max.bytes is
> > > overridden to 104857600 (kafka.utils.VerifiableProperties)
> > > [2014-03-11 12:05:32,682] INFO Property num.io.threads is overridden to 2
> > > (kafka.utils.VerifiableProperties)
> > > [2014-03-11 12:05:32,682] INFO Property log.dirs is overridden to
> > > /tmp/kafka-logs (kafka.utils.VerifiableProperties)
> > > [2014-03-11 12:05:32,682] INFO Property log.cleanup.interval.mins is
> > > overridden to 1 (kafka.utils.VerifiableProperties)
> > > [2014-03-11 12:05:32,682] INFO Property socket.send.buffer.bytes is
> > > overridden to 1048576 (kafka.utils.VerifiableProperties)
> > > [2014-03-11 12:05:32,683] INFO Property log.flush.interval.ms is
> > > overridden to 1000 (kafka.utils.VerifiableProperties)
> > > [2014-03-11 12:05:32,683] INFO Property zookeeper.connect is overridden to
> > > 192.168.1.120:2181 (kafka.utils.VerifiableProperties)
> > > [2014-03-11 12:05:32,684] INFO Property broker.id is overridden to 2
> > > (kafka.utils.VerifiableProperties)
> > > [2014-03-11 12:05:32,684] INFO Property log.retention.hours is overridden
> > > to 168 (kafka.utils.VerifiableProperties)
> > > [2014-03-11 12:05:32,684] INFO Property num.network.threads is overridden
> > > to 2 (kafka.utils.VerifiableProperties)
> > > [2014-03-11 12:05:32,684] INFO Property socket.receive.buffer.bytes is
> > > overridden to 1048576 (kafka.utils.VerifiableProperties)
> > > [2014-03-11 12:05:32,684] INFO Property zookeeper.connection.timeout.msis overridden to 1000000 (kafka.utils.VerifiableProperties)
> > > [2014-03-11 12:05:32,685] INFO Property num.partitions is overridden to 2
> > > (kafka.utils.VerifiableProperties)
> > > [2014-03-11 12:05:32,685] INFO Property log.flush.interval.messages is
> > > overridden to 10000 (kafka.utils.VerifiableProperties)
> > > [2014-03-11 12:05:32,685] INFO Property log.segment.bytes is overridden to
> > > 536870912 (kafka.utils.VerifiableProperties)
> > > [2014-03-11 12:05:32,703] INFO [Kafka Server 2], Starting
> > > (kafka.server.KafkaServer)
> > > [2014-03-11 12:05:32,739] INFO [Log Manager on Broker 2] Starting log
> > > cleaner every 60000 ms (kafka.log.LogManager)
> > > [2014-03-11 12:05:32,753] INFO [Log Manager on Broker 2] Starting log
> > > flusher every 3000 ms with the following overrides Map()
> > > (kafka.log.LogManager)
> > > [2014-03-11 12:05:32,791] INFO Awaiting socket connections on 0.0.0.0:9092.
> > > (kafka.network.Acceptor)
> > > [2014-03-11 12:05:32,794] INFO [Socket Server on Broker 2], Started
> > > (kafka.network.SocketServer)
> > > [2014-03-11 12:05:32,801] INFO connecting to ZK: 192.168.1.120:2181(kafka.server.KafkaZooKeeper)
> > > [2014-03-11 12:05:32,815] INFO Starting ZkClient event thread.
> > > (org.I0Itec.zkclient.ZkEventThread)
> > > [2014-03-11 12:05:32,817] INFO Client
> > > environment:zookeeper.version=3.3.3-1203054, built on 11/17/2011 05:47 GMT
> > > (org.apache.zookeeper.ZooKeeper)
> > > [2014-03-11 12:05:32,817] INFO Client environment:host.name=localhost
> > > (org.apache.zookeeper.ZooKeeper)
> > > [2014-03-11 12:05:32,817] INFO Client environment:java.version=1.7.0_51
> > > (org.apache.zookeeper.ZooKeeper)
> > > [2014-03-11 12:05:32,822] INFO Client environment:java.vendor=Oracle
> > > Corporation (org.apache.zookeeper.ZooKeeper)
> > > [2014-03-11 12:05:32,822] INFO Client
> > > environment:java.home=/usr/lib/jvm/java-1.7.0-openjdk-1.7.0.51.x86_64/jre
> > > (org.apache.zookeeper.ZooKeeper)
> > > [2014-03-11 12:05:32,822] INFO Client
> > > environment:java.class.path=:/opt/kafka-0.8.0/bin/../core/target/scala-2.8.0/kafka_2.8.0-0.8.0.jar:/opt/kafka-0.8.0/bin/../core/target/scala-2.8.0/kafka-assembly-0.8.0-deps.jar:/opt/kafka-0.8.0/bin/../perf/target/scala-2.8.0/kafka-perf_2.8.0-0.8.0.jar:/opt/kafka-0.8.0/bin/../libs/*.jar:/opt/kafka-0.8.0/bin/../kafka*.jar
> > > (org.apache.zookeeper.ZooKeeper)
> > > [2014-03-11 12:05:32,822] INFO Client
> > > environment:java.library.path=/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib
> > > (org.apache.zookeeper.ZooKeeper)
> > > [2014-03-11 12:05:32,822] INFO Client environment:java.io.tmpdir=/tmp
> > > (org.apache.zookeeper.ZooKeeper)
> > > [2014-03-11 12:05:32,822] INFO Client environment:java.compiler=<NA>
> > > (org.apache.zookeeper.ZooKeeper)
> > > [2014-03-11 12:05:32,822] INFO Client environment:os.name=Linux
> > > (org.apache.zookeeper.ZooKeeper)
> > > [2014-03-11 12:05:32,822] INFO Client environment:os.arch=amd64
> > > (org.apache.zookeeper.ZooKeeper)
> > > [2014-03-11 12:05:32,822] INFO Client
> > > environment:os.version=2.6.32-279.el6.x86_64
> > > (org.apache.zookeeper.ZooKeeper)
> > > [2014-03-11 12:05:32,822] INFO Client environment:user.name=root
> > > (org.apache.zookeeper.ZooKeeper)
> > > [2014-03-11 12:05:32,822] INFO Client environment:user.home=/root
> > > (org.apache.zookeeper.ZooKeeper)
> > > [2014-03-11 12:05:32,822] INFO Client environment:user.dir=/opt
> > > (org.apache.zookeeper.ZooKeeper)
> > > [2014-03-11 12:05:32,825] INFO Initiating client connection, connectString=
> > > 192.168.1.120:2181 sessionTimeout=6000
> > > watcher=org.I0Itec.zkclient.ZkClient@721e34d8(org.apache.zookeeper.ZooKeeper)
> > > [2014-03-11 12:05:32,842] INFO Opening socket connection to server /
> > > 192.168.1.120:2181 (org.apache.zookeeper.ClientCnxn)
> > > [2014-03-11 12:05:32,846] INFO Socket connection established to
> > > 192.168.1.120/192.168.1.120:2181, initiating session
> > > (org.apache.zookeeper.ClientCnxn)
> > > [2014-03-11 12:05:32,854] INFO Session establishment complete on server
> > > 192.168.1.120/192.168.1.120:2181, sessionid = 0x144b104e0210004,
> > > negotiated timeout = 6000 (org.apache.zookeeper.ClientCnxn)
> > > [2014-03-11 12:05:32,856] INFO zookeeper state changed (SyncConnected)
> > > (org.I0Itec.zkclient.ZkClient)
> > > [2014-03-11 12:05:32,891] INFO Registered broker 2 at path /brokers/ids/2
> > > with address localhost:9092. (kafka.utils.ZkUtils$)
> > > [2014-03-11 12:05:32,892] INFO [Kafka Server 2], Connecting to ZK:
> > > 192.168.1.120:2181 (kafka.server.KafkaServer)
> > > [2014-03-11 12:05:33,031] INFO Will not load MX4J, mx4j-tools.jar is not
> > > in the classpath (kafka.utils.Mx4jLoader$)
> > > [2014-03-11 12:05:33,041] INFO 2 successfully elected as leader
> > > (kafka.server.ZookeeperLeaderElector)
> > > [2014-03-11 12:05:33,371] INFO [Kafka Server 2], Started
> > > (kafka.server.KafkaServer)
> > > [2014-03-11 12:05:33,377] INFO New leader is 2
> > > (kafka.server.ZookeeperLeaderElector$LeaderChangeListener)
> > >
> > > Then started Broker 1
> > >
> > > $KAFKA/kafka-server-start.sh $KAFKA_CONFIG/server.properties
> > > [2014-03-11 12:07:20,843] INFO Verifying properties
> > > (kafka.utils.VerifiableProperties)
> > > [2014-03-11 12:07:20,875] INFO Property port is overridden to 9092
> > > (kafka.utils.VerifiableProperties)
> > > [2014-03-11 12:07:20,876] INFO Property socket.request.max.bytes is
> > > overridden to 104857600 (kafka.utils.VerifiableProperties)
> > > [2014-03-11 12:07:20,876] INFO Property num.io.threads is overridden to 2
> > > (kafka.utils.VerifiableProperties)
> > > [2014-03-11 12:07:20,876] INFO Property log.dirs is overridden to
> > > /tmp/kafka-logs (kafka.utils.VerifiableProperties)
> > > [2014-03-11 12:07:20,876] INFO Property log.cleanup.interval.mins is
> > > overridden to 1 (kafka.utils.VerifiableProperties)
> > > [2014-03-11 12:07:20,877] INFO Property socket.send.buffer.bytes is
> > > overridden to 1048576 (kafka.utils.VerifiableProperties)
> > > [2014-03-11 12:07:20,877] INFO Property log.flush.interval.ms is
> > > overridden to 1000 (kafka.utils.VerifiableProperties)
> > > [2014-03-11 12:07:20,877] INFO Property zookeeper.connect is overridden to
> > > 192.168.1.120:2181 (kafka.utils.VerifiableProperties)
> > > [2014-03-11 12:07:20,877] INFO Property broker.id is overridden to 1
> > > (kafka.utils.VerifiableProperties)
> > > [2014-03-11 12:07:20,877] INFO Property log.retention.hours is overridden
> > > to 168 (kafka.utils.VerifiableProperties)
> > > [2014-03-11 12:07:20,878] INFO Property num.network.threads is overridden
> > > to 2 (kafka.utils.VerifiableProperties)
> > > [2014-03-11 12:07:20,878] INFO Property socket.receive.buffer.bytes is
> > > overridden to 1048576 (kafka.utils.VerifiableProperties)
> > > [2014-03-11 12:07:20,878] INFO Property zookeeper.connection.timeout.msis overridden to 1000000 (kafka.utils.VerifiableProperties)
> > > [2014-03-11 12:07:20,878] INFO Property num.partitions is overridden to 2
> > > (kafka.utils.VerifiableProperties)
> > > [2014-03-11 12:07:20,879] INFO Property log.flush.interval.messages is
> > > overridden to 10000 (kafka.utils.VerifiableProperties)
> > > [2014-03-11 12:07:20,879] INFO Property log.segment.bytes is overridden to
> > > 536870912 (kafka.utils.VerifiableProperties)
> > > [2014-03-11 12:07:20,897] INFO [Kafka Server 1], Starting
> > > (kafka.server.KafkaServer)
> > > [2014-03-11 12:07:20,922] INFO [Log Manager on Broker 1] Log directory
> > > '/tmp/kafka-logs' not found, creating it. (kafka.log.LogManager)
> > > [2014-03-11 12:07:20,938] INFO [Log Manager on Broker 1] Starting log
> > > cleaner every 60000 ms (kafka.log.LogManager)
> > > [2014-03-11 12:07:20,951] INFO [Log Manager on Broker 1] Starting log
> > > flusher every 3000 ms with the following overrides Map()
> > > (kafka.log.LogManager)
> > > [2014-03-11 12:07:20,991] INFO Awaiting socket connections on 0.0.0.0:9092.
> > > (kafka.network.Acceptor)
> > > [2014-03-11 12:07:20,993] INFO [Socket Server on Broker 1], Started
> > > (kafka.network.SocketServer)
> > > [2014-03-11 12:07:21,002] INFO connecting to ZK: 192.168.1.120:2181(kafka.server.KafkaZooKeeper)
> > > [2014-03-11 12:07:21,018] INFO Starting ZkClient event thread.
> > > (org.I0Itec.zkclient.ZkEventThread)
> > > [2014-03-11 12:07:21,029] INFO Client
> > > environment:zookeeper.version=3.3.3-1203054, built on 11/17/2011 05:47 GMT
> > > (org.apache.zookeeper.ZooKeeper)
> > > [2014-03-11 12:07:21,029] INFO Client environment:host.name=localhost
> > > (org.apache.zookeeper.ZooKeeper)
> > > [2014-03-11 12:07:21,029] INFO Client environment:java.version=1.7.0_51
> > > (org.apache.zookeeper.ZooKeeper)
> > > [2014-03-11 12:07:21,029] INFO Client environment:java.vendor=Oracle
> > > Corporation (org.apache.zookeeper.ZooKeeper)
> > > [2014-03-11 12:07:21,029] INFO Client
> > > environment:java.home=/usr/lib/jvm/java-1.7.0-openjdk-1.7.0.51.x86_64/jre
> > > (org.apache.zookeeper.ZooKeeper)
> > > [2014-03-11 12:07:21,029] INFO Client
> > > environment:java.class.path=:/opt/kafka-0.8.0/bin/../core/target/scala-2.8.0/kafka_2.8.0-0.8.0.jar:/opt/kafka-0.8.0/bin/../core/target/scala-2.8.0/kafka-assembly-0.8.0-deps.jar:/opt/kafka-0.8.0/bin/../perf/target/scala-2.8.0/kafka-perf_2.8.0-0.8.0.jar:/opt/kafka-0.8.0/bin/../libs/*.jar:/opt/kafka-0.8.0/bin/../kafka*.jar
> > > (org.apache.zookeeper.ZooKeeper)
> > > [2014-03-11 12:07:21,029] INFO Client
> > > environment:java.library.path=/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib
> > > (org.apache.zookeeper.ZooKeeper)
> > > [2014-03-11 12:07:21,029] INFO Client environment:java.io.tmpdir=/tmp
> > > (org.apache.zookeeper.ZooKeeper)
> > > [2014-03-11 12:07:21,030] INFO Client environment:java.compiler=<NA>
> > > (org.apache.zookeeper.ZooKeeper)
> > > [2014-03-11 12:07:21,030] INFO Client environment:os.name=Linux
> > > (org.apache.zookeeper.ZooKeeper)
> > > [2014-03-11 12:07:21,030] INFO Client environment:os.arch=amd64
> > > (org.apache.zookeeper.ZooKeeper)
> > > [2014-03-11 12:07:21,030] INFO Client
> > > environment:os.version=2.6.32-279.el6.x86_64
> > > (org.apache.zookeeper.ZooKeeper)
> > > [2014-03-11 12:07:21,030] INFO Client environment:user.name=root
> > > (org.apache.zookeeper.ZooKeeper)
> > > [2014-03-11 12:07:21,030] INFO Client environment:user.home=/root
> > > (org.apache.zookeeper.ZooKeeper)
> > > [2014-03-11 12:07:21,030] INFO Client environment:user.dir=/var/zookeeper
> > > (org.apache.zookeeper.ZooKeeper)
> > > [2014-03-11 12:07:21,031] INFO Initiating client connection, connectString=
> > > 192.168.1.120:2181 sessionTimeout=6000
> > > watcher=org.I0Itec.zkclient.ZkClient@45419cee(org.apache.zookeeper.ZooKeeper)
> > > [2014-03-11 12:07:21,043] INFO Opening socket connection to server /
> > > 192.168.1.120:2181 (org.apache.zookeeper.ClientCnxn)
> > > [2014-03-11 12:07:21,049] INFO Socket connection established to
> > > 192.168.1.120/192.168.1.120:2181, initiating session
> > > (org.apache.zookeeper.ClientCnxn)
> > > [2014-03-11 12:07:21,058] INFO Session establishment complete on server
> > > 192.168.1.120/192.168.1.120:2181, sessionid = 0x144b104e0210005,
> > > negotiated timeout = 6000 (org.apache.zookeeper.ClientCnxn)
> > > [2014-03-11 12:07:21,060] INFO zookeeper state changed (SyncConnected)
> > > (org.I0Itec.zkclient.ZkClient)
> > > [2014-03-11 12:07:21,095] INFO Registered broker 1 at path /brokers/ids/1
> > > with address localhost:9092. (kafka.utils.ZkUtils$)
> > > [2014-03-11 12:07:21,096] INFO [Kafka Server 1], Connecting to ZK:
> > > 192.168.1.120:2181 (kafka.server.KafkaServer)
> > > [2014-03-11 12:07:21,222] INFO Will not load MX4J, mx4j-tools.jar is not
> > > in the classpath (kafka.utils.Mx4jLoader$)
> > > [2014-03-11 12:07:21,249] INFO conflict in /controller data: {
> > > "brokerid":1, "timestamp":"1394539641229", "version":1 } stored data: {
> > > "brokerid":2, "timestamp":"1394539533037", "version":1 }
> > > (kafka.utils.ZkUtils$)
> > > [2014-03-11 12:07:21,469] INFO [Kafka Server 1], Started
> > > (kafka.server.KafkaServer)
> > >
> > > Could this be an issue?
> > > INFO Registered broker 2 at path /brokers/ids/2 with address
> > > localhost:9092.
> > > INFO Registered broker 1 at path /brokers/ids/1 with address
> > > localhost:9092.
> > >
> > >
> > >
> > > > Date: Tue, 11 Mar 2014 09:51:54 -0700
> > > > Subject: Re: Remote Zookeeper
> > > > From: b@b3k.us
> > > > To: users@kafka.apache.org
> > > >
> > > > Seems you my have put your cluster in a very confused state with random
> > > > addition and removal of brokers and topics. I'd suggest deleting the
> > > > zookeeper and Kafka logs and starting over using the getting started
> > > > tutorial from the wiki.
> > > >
> > > > On Tuesday, March 11, 2014, A A <an...@hotmail.com> wrote:
> > > >
> > > > > I noticed a problem with my topics. The previous two topics would only
> > > be
> > > > > present on 1 broker. So I created a new topic
> > > > >
> > > > > $KAFKA_HOME/bin/kafka-create-topic.sh --zookeeper 192.168.1.120:2181--replica
> > > 2 --partition 1 --topic test-rep
> > > > > $KAFKA_HOME/bin/kafka-list-topic.sh --zookeeper 192.168.1.120:2181
> > > > > topic: test partition: 0 leader: 1 replicas: 1 isr: 1
> > > > > topic: test-rep partition: 0 leader: 1 replicas: 1,2 isr: 1
> > > > > topic: test1 partition: 0 leader: 1 replicas: 1 isr: 1
> > > > >
> > > > > No there is no firewall. Even then I stopped the iptables. And then I
> > > > > tried the following
> > > > > $KAFKA_HOME/bin/kafka-console-producer.sh --broker-list
> > > > > 192.168.1.123:9092,192.168.1.124:9092 --topic test-rep
> > > > > replicated message
> > > > >
> > > > > [2014-03-11 10:59:25,844] WARN Error while fetching metadata
> > > > > [{TopicMetadata for topic test-rep ->
> > > > > No partition metadata for topic test-rep due to
> > > > > kafka.common.LeaderNotAvailableException}] for topic [test-rep]: class
> > > > > kafka.common.LeaderNotAvailableException
> > > > > (kafka.producer.BrokerPartitionInfo)
> > > > > [2014-03-11 10:59:25,867] WARN Error while fetching metadata
> > > > > [{TopicMetadata for topic test-rep ->
> > > > > No partition metadata for topic test-rep due to
> > > > > kafka.common.LeaderNotAvailableException}] for topic [test-rep]: class
> > > > > kafka.common.LeaderNotAvailableException
> > > > > (kafka.producer.BrokerPartitionInfo)
> > > > > [2014-03-11 10:59:25,868] ERROR Failed to collate messages by topic,
> > > > > partition due to: Failed to fetch topic metadata for topic: test-rep
> > > > > (kafka.producer.async.DefaultEventHandler)
> > > > > [2014-03-11 10:59:25,979] WARN Error while fetching metadata
> > > > > [{TopicMetadata for topic test-rep ->
> > > > > No partition metadata for topic test-rep due to
> > > > > kafka.common.LeaderNotAvailableException}] for topic [test-rep]: class
> > > > > kafka.common.LeaderNotAvailableException
> > > > > (kafka.producer.BrokerPartitionInfo)
> > > > > [2014-03-11 10:59:25,986] WARN Error while fetching metadata
> > > > > [{TopicMetadata for topic test-rep ->
> > > > > No partition metadata for topic test-rep due to
> > > > > kafka.common.LeaderNotAvailableException}] for topic [test-rep]: class
> > > > > kafka.common.LeaderNotAvailableException
> > > > > (kafka.producer.BrokerPartitionInfo)
> > > > > [2014-03-11 10:59:25,987] ERROR Failed to collate messages by topic,
> > > > > partition due to: Failed to fetch topic metadata for topic: test-rep
> > > > > (kafka.producer.async.DefaultEventHandler)
> > > > > [2014-03-11 10:59:26,098] WARN Error while fetching metadata
> > > > > [{TopicMetadata for topic test-rep ->
> > > > > No partition metadata for topic test-rep due to
> > > > > kafka.common.LeaderNotAvailableException}] for topic [test-rep]: class
> > > > > kafka.common.LeaderNotAvailableException
> > > > > (kafka.producer.BrokerPartitionInfo)
> > > > >
> > > > >
> > > > >
> > > > >
> > > > > > Date: Tue, 11 Mar 2014 09:05:54 -0700
> > > > > > Subject: Re: Remote Zookeeper
> > > > > > From: pradeepg26@gmail.com <javascript:;>
> > > > > > To: users@kafka.apache.org <javascript:;>
> > > > > >
> > > > > > Is there a firewall thats blocking connections on port 9092? Also,
> > > the
> > > > > > broker list should be comma separated.
> > > > > >
> > > > > >
> > > > > > On Tue, Mar 11, 2014 at 9:02 AM, A A <an...@hotmail.com>
> > > wrote:
> > > > > >
> > > > > > > Sorry one of the brokers for was down. Brought it back up. Tried
> > > the
> > > > > > > following
> > > > > > >
> > > > > > > $KAFKA_HOME/bin/kafka-console-producer.sh --broker-list
> > > localhost:9092
> > > > > > > 192.168.1.124:9092 --topic test
> > > > > > > hello brokers
> > > > > > >
> > > > > > > [2014-03-11 10:16:55,547] WARN Error while fetching metadata
> > > > > > > [{TopicMetadata for topic test ->
> > > > > > > No partition metadata for topic test due to
> > > > > > > kafka.common.LeaderNotAvailableException}] for topic [test]: class
> > > > > > > kafka.common.LeaderNotAvailableException
> > > > > > > (kafka.producer.BrokerPartitionInfo)
> > > > > > > [2014-03-11 10:16:55,576] WARN Error while fetching metadata
> > > > > > > [{TopicMetadata for topic test ->
> > > > > > > No partition metadata for topic test due to
> > > > > > > kafka.common.LeaderNotAvailableException}] for topic [test]: class
> > > > > > > kafka.common.LeaderNotAvailableException
> > > > > > > (kafka.producer.BrokerPartitionInfo)
> > > > > > > [2014-03-11 10:16:55,578] ERROR Failed to collate messages by
> > > topic,
> > > > > > > partition due to: Failed to fetch topic metadata for topic: test
> > > > > > > (kafka.producer.async.DefaultEventHandler)
> > > > > > > [2014-03-11 10:16:55,693] WARN Error while fetching metadata
> > > > > > > [{TopicMetadata for topic test ->
> > > > > > > No partition metadata for topic test due to
> > > > > > > kafka.common.LeaderNotAvailableException}] for topic [test]: class
> > > > > > > kafka.common.LeaderNotAvailableException
> > > > > > > (kafka.producer.BrokerPartitionInfo)
> > > > > > > [2014-03-11 10:16:55,706] WARN Error while fetching metadata
> > > > > > > [{TopicMetadata for topic test ->
> > > > > > > No partition metadata for topic test due to
> > > > > > > kafka.common.LeaderNotAvailableException}] for topic [test]: class
> > > > > > > kafka.common.LeaderNotAvailableException
> > > > > > > (kafka.producer.BrokerPartitionInfo)
> > > > > > > [2014-03-11 10:16:55,706] ERROR Failed to collate messages by
> > > topic,
> > > > > > > partition due to: Failed to fetch topic metadata for topic: test
> > > > > > > (kafka.producer.async.DefaultEventHandler)
> > > > > > > [2014-03-11 10:16:55,815] WARN Error while fetching metadata
> > > > > > > [{TopicMetadata for topic test ->
> > > > > > > No partition metadata for topic test due to
> > > > > > > kafka.common.LeaderNotAvailableException}] for topic [test]: class
> > > > > > > kafka.common.LeaderNotAvailableException
> > > > > > > (kafka.producer.BrokerPartitionInfo)
> > > > > > > [2014-03-11 10:16:55,823] WARN Error while fetching metadata
> > > > > > > [{TopicMetadata for topic test ->
> > > > > > > No partition metadata for topic test due to
> > > > > > > kafka.common.LeaderNotAvailableException}] for topic [test]: class
> > > > > > > kafka.common.LeaderNotAvailableException
> > > > > > > (kafka.producer.BrokerPartitionInfo)
> > > > > > > [2014-03-11 10:16:55,823] ERROR Failed to collate messages by
> > > topic,
> > > > > > > partition due to: Failed to fetch topic metadata for topic: test
> > > > > > > (kafka.producer.async.DefaultEventHandler)
> > > > > > > [2014-03-11 10:16:55,934] WARN Error while fetching metadata
> > > > > > > [{TopicMetadata for topic test ->
> > > > > > > No partition metadata for topic test due to
> > > > > > > kafka.common.LeaderNotAvailableException}] for topic [test]: class
> > > > > > > kafka.common.LeaderNotAvailableException
> > > > > > > (kafka.producer.BrokerPartitionInfo)
> > > > > > > [2014-03-11 10:16:55,949] WARN Error while fetching metadata
> > > > > > > [{TopicMetadata for topic test ->
> > > > > > > No partition metadata for topic test due to
> > > > > > > kafka.common.LeaderNotAvailableException}] for topic [test]: class
> > > > > > > kafka.common.LeaderNotAvailableException
> > > > > > > (kafka.producer.BrokerPartitionInfo)
> > > > > > > [2014-03-11 10:16:55,949] ERROR Failed to collate messages by
> > > topic,
> > > > > > > partition due to: Failed to fetch topic metadata for topic: test
> > > > > > > (kafka.producer.async.DefaultEventHandler)
> > > > > > > [2014-03-11 10:16:56,057] WARN Error while fetching metadata
> > > > > > > [{TopicMetadata for topic test ->
> > > > > > > No partition metadata for topic test due to<
> > >
> > >
>
RE: Remote Zookeeper
Posted by A A <an...@hotmail.com>.
Yes. True. Thanks for your help.
> Date: Tue, 11 Mar 2014 11:48:20 -0700
> Subject: Re: Remote Zookeeper
> From: b@b3k.us
> To: users@kafka.apache.org
>
> If they are on different physical machines then binding to localhost/using
> localhost as the host name is unlikely to be what you want.
>
> On Tuesday, March 11, 2014, A A <an...@hotmail.com> wrote:
>
> > Thanks. I already checked out the wiki and step 6 in particular.
> > Just to clarify, Zk, Broker1 and Broker2 are on 3 different physical
> > machines.
> >
> > > From: b@b3k.us <javascript:;>
> > > Date: Tue, 11 Mar 2014 10:58:50 -0700
> > > Subject: Re: Remote Zookeeper
> > > To: users@kafka.apache.org <javascript:;>
> > >
> > > "I'd suggest deleting the zookeeper and Kafka logs and starting over
> > using
> > > the getting started tutorial from the wiki."
> > >
> > > |
> > > v
> > >
> > > "Could this be an issue?
> > > INFO Registered broker 2 at path /brokers/ids/2 with address
> > localhost:9092.
> > > INFO Registered broker 1 at path /brokers/ids/1 with address
> > > localhost:9092."
> > >
> > > yes. see step 6: https://kafka.apache.org/documentation.html#quickstart
> > >
> > >
> > >
> > > On Tue, Mar 11, 2014 at 10:54 AM, A A <andthereitgoes@hotmail.com<javascript:;>>
> > wrote:
> > >
> > > > Okay thanks. I removed zookeeper and kafka logs
> > > >
> > > > /tmp/kafka-logs
> > > > /tmp/logs
> > > > /var/zookeeper/version-2/
> > > > /tmp/zookeeper*.log (on Zk)
> > > >
> > > > restarted zookeeper
> > > >
> > > > $KAFKA_HOME/bin/kafka-list-topic.sh --zookeeper 192.168.1.120:2181
> > > > no topics exist!
> > > >
> > > > Started Broker 2
> > > > $KAFKA/kafka-server-start.sh $KAFKA_CONFIG/server.properties
> > > > [2014-03-11 12:05:32,651] INFO Verifying properties
> > > > (kafka.utils.VerifiableProperties)
> > > > [2014-03-11 12:05:32,681] INFO Property port is overridden to 9092
> > > > (kafka.utils.VerifiableProperties)
> > > > [2014-03-11 12:05:32,681] INFO Property socket.request.max.bytes is
> > > > overridden to 104857600 (kafka.utils.VerifiableProperties)
> > > > [2014-03-11 12:05:32,682] INFO Property num.io.threads is overridden
> > to 2
> > > > (kafka.utils.VerifiableProperties)
> > > > [2014-03-11 12:05:32,682] INFO Property log.dirs is overridden to
> > > > /tmp/kafka-logs (kafka.utils.VerifiableProperties)
> > > > [2014-03-11 12:05:32,682] INFO Property log.cleanup.interval.mins is
> > > > overridden to 1 (kafka.utils.VerifiableProperties)
> > > > [2014-03-11 12:05:32,682] INFO Property socket.send.buffer.bytes is
> > > > overridden to 1048576 (kafka.utils.VerifiableProperties)
> > > > [2014-03-11 12:05:32,683] INFO Property log.flush.interval.ms is
> > > > overridden to 1000 (kafka.utils.VerifiableProperties)
> > > > [2014-03-11 12:05:32,683] INFO Property zookeeper.connect is
> > overridden to
> > > > 192.168.1.120:2181 (kafka.utils.VerifiableProperties)
> > > > [2014-03-11 12:05:32,684] INFO Property broker.id is overridden to 2
> > > > (kafka.utils.VerifiableProperties)
> > > > [2014-03-11 12:05:32,684] INFO Property log.retention.hours is
> > overridden
> > > > to 168 (kafka.utils.VerifiableProperties)
> > > > [2014-03-11 12:05:32,684] INFO Property num.network.threads is
> > overridden
> > > > to 2 (kafka.utils.VerifiableProperties)
> > > > [2014-03-11 12:05:32,684] INFO Property socket.receive.buffer.bytes is
> > > > overridden to 1048576 (kafka.utils.VerifiableProperties)
> > > > [2014-03-11 12:05:32,684] INFO Property
> > zookeeper.connection.timeout.msis overridden to 1000000
> > (kafka.utils.VerifiableProperties)
> > > > [2014-03-11 12:05:32,685] INFO Property num.partitions is overridden
> > to 2
> > > > (kafka.utils.VerifiableProperties)
> > > > [2014-03-11 12:05:32,685] INFO Property log.flush.interval.messages is
> > > > overridden to 10000 (kafka.utils.VerifiableProperties)
> > > > [2014-03-11 12:05:32,685] INFO Property log.segment.bytes is
> > overridden to
> > > > 536870912 (kafka.utils.VerifiableProperties)
> > > > [2014-03-11 12:05:32,703] INFO [Kafka Server 2], Starting
> > > > (kafka.server.KafkaServer)
> > > > [2014-03-11 12:05:32,739] INFO [Log Manager on Broker 2] Starting log
> > > > cleaner every 60000 ms (kafka.log.LogManager)
> > > > [2014-03-11 12:05:32,753] INFO [Log Manager on Broker 2] Starting log
> > > > flusher every 3000 ms with the following overrides Map()
> > > > (kafka.log.LogManager)
> > > > [2014-03-11 12:05:32,791] INFO Awaiting socket connections on
> > 0.0.0.0:9092.
> > > > (kafka.network.Acceptor)
> > > > [2014-03-11 12:05:32,794] INFO [Socket Server on Broker 2], Started
> > > > (kafka.network.SocketServer)
> > > > [2014-03-11 12:05:32,801] INFO connecting to ZK: 192.168.1.120:2181
> > (kafka.server.KafkaZooKeeper)
> > > > [2014-03-11 12:05:32,815] INFO Starting ZkClient event thread.
> > > > (org.I0Itec.zkclient.ZkEventThread)
> > > > [2014-03-11 12:05:32,817] INFO Client
> > > > environment:zookeeper.version=3.3.3-1203054, built on 11/17/2011 05:47
> > GMT
> > > > (org.apache.zookeeper.ZooKeeper)
> > > > [2014-03-11 12:05:32,817] INFO Client environment:host.name=localhost
> > > > (org.apache.zookeeper.ZooKeeper)
> > > > [2014-03-11 12:05:32,817] INFO Client environment:java.version=1.7.0_51
> > > > (org.apache.zookeeper.ZooKeeper)
> > > > [2014-03-11 12:05:32,822] INFO Client environment:java.vendor=Oracle
> > > > Corporation (org.apache.zookeeper.ZooKeeper)
> > > > [2014-03-11 12:05:32,822] INFO Client
> > > >
> > environment:java.home=/usr/lib/jvm/java-1.7.0-openjdk-1.7.0.51.x86_64/jre
> > > > (org.apache.zookeeper.ZooKeeper)
> > > > [2014-03-11 12:05:32,822] INFO Client
> > > >
> > environment:java.class.path=:/opt/kafka-0.8.0/bin/../core/target/scala-2.8.0/kafka_2.8.0-0.8.0.jar:/opt/kafka-0.8.0/bin/../core/target/scala-2.8.0/kafka-assembly-0.8.0-deps.jar:/opt/kafka-0.8.0/bin/../perf/target/scala-2.8.0/kafka-perf_2.8.0-0.8.0.jar:/opt/kafka-0.8.0/bin/../libs/*.jar:/opt/kafka-0.8.0/bin/../kafka*.jar
> > > > (org.apache.zookeeper.ZooKeeper)
> > > > [2014-03-11 12:05:32,822] INFO Client
> > > >
> > environment:java.library.path=/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib
> > > > (org.apache.zookeeper.ZooKeeper)
> > > > [2014-03-11 12:05:32,822] INFO Client environment:java.io.tmpdir=/tmp
> > > > (org.apache.zookeeper.ZooKeeper)
> > > > [2014-03-11 12:05:32,822] INFO Client environment:java.compiler=<NA>
> > > > (org.apache.zookeeper.ZooKeeper)
> > > > [2014-03-11 12:05:32,822] INFO Client environment:os.name=Linux
> > > > (org.apache.zookeeper.ZooKeeper)
> > > > [2014-03-11 12:05:32,822] INFO Client environment:os.arch=amd64
> > > > (org.apache.zookeeper.ZooKeeper)
> > > > [2014-03-11 12:05:32,822] INFO Client
> > > > environment:os.version=2.6.32-279.el6.x86_64
> > > > (org.apache.zookeeper.ZooKeeper)
> > > > [2014-03-11 12:05:32,822] INFO Client environment:user.name=root
> > > > (org.apache.zookeeper.ZooKeeper)
> > > > [2014-03-11 12:05:32,822] INFO Client environment:user.home=/root
> > > > (org.apache.zookeeper.ZooKeeper)
> > > > [2014-03-11 12:05:32,822] INFO Client environment:user.dir=/opt
> > > > (org.apache.zookeeper.ZooKeeper)
> > > > [2014-03-11 12:05:32,825] INFO Initiating client connection,
> > connectString=
> > > > 192.168.1.120:2181 sessionTimeout=6000
> > > > watcher=org.I0Itec.zkclient.ZkClient@721e34d8
> > (org.apache.zookeeper.ZooKeeper)
> > > > [2014-03-11 12:05:32,842] INFO Opening socket connection to server /
> > > > 192.168.1.120:2181 (org.apache.zookeeper.ClientCnxn)
> > > > [2014-03-11 12:05:32,846] INFO Socket connection established to
> > > > 192.168.1.120/192.168.1.120:2181, initiating session
> > > > (org.apache.zookeeper.ClientCnxn)
> > > > [2014-03-11 12:05:32,854] INFO Session establishment complete on server
> > > > 192.168.1.120/192.168.1.120:2181, sessionid = 0x144b104e0210004,
> > > > negotiated timeout = 6000 (org.apache.zookeeper.ClientCnxn)
> > > > [2014-03-11 12:05:32,856] INFO zookeeper state changed (SyncConnected)
> > > > (org.I0Itec.zkclient.ZkClient)
> > > > [2014-03-11 12:05:32,891] INFO Registered broker 2 at path
> > /brokers/ids/2
> > > > with address localhost:9092. (kafka.utils.ZkUtils$)
> > > > [2014-03-11 12:05:32,892] INFO [Kafka Server 2], Connecting to ZK:
> > > > 192.168.1.120:2181 (kafka.server.KafkaServer)
> > > > [2014-03-11 12:05:33,031] INFO Will not load MX4J, mx4j-tools.jar is
> > not
> > > > in the classpath (kafka.utils.Mx4jLoader$)
> > > > [2014-03-11 12:05:33,041] INFO 2 successfully elected as leader
> > > > (kafka.server.ZookeeperLeaderElector)
> > > > [2014-03-11 12:05:33,371] INFO [Kafka Server 2], Started
> > > > (kafka.server.KafkaServer)
> > > > [2014-03-11 12:05:33,377] INFO New leader is 2
> > > > (kafka.server.ZookeeperLeaderElector$LeaderChangeListener)
> > > >
> > > > Then started Broker 1
> > > >
> > > > $KAFKA/kafka-server-start.sh $KAFKA_CONFIG/server.properties
> > > > [2014-03-11 12:07:20,843] INFO Verifying properties
> > > > (kafka.utils.VerifiableProperties)
> > > > [2014-03-11 12:07:20,875] INFO Property port is overridden to 9092
> > > > (kafka.utils.VerifiableProperties)
> > > > [2014-03-11 12:07:20,876] INFO Property socket.request.max.bytes is
> > > > overridden to 104857600 (kafka.utils.VerifiableProperties)
> > > > [2014-03-11 12:07:20,876] INFO Property num.io.threads is overridden
> > to 2
> > > > (kafka.utils.VerifiableProperties)
> > > > [2014-03-11 12:07:20,876] INFO Property log.dirs is overridden to
> > > > /tmp/kafka-logs (kafka.utils.VerifiableProperties)
> > > > [2014-03-11 12:07:20,876] INFO Property log.cleanup.interval.mins is
> > > > overridden to 1 (kafka.utils.VerifiableProperties)
> > > > [2014-03-11 12:07:20,877] INFO Property socket.send.buffer.bytes is
> > > > overridden to 1048576 (kafka.utils.VerifiableProperties)
> > > > [2014-03-11 12:07:20,877] INFO Property log.flush.interval.ms is
> > > > overridden to 1000 (kafka.utils.VerifiableProperties)
> > > > [2014-03-11 12:07:20,877] INFO Property zookeeper.connect is
> > overridden to
> > > > 192.168.1.120:2181 (kafka.utils.VerifiableProperties)
> > > > [2014-03-11 12:07:20,877] INFO Property broker.id is overridden to 1
> > > > (kafka.utils.VerifiableProperties)
> > > > [2014-03-11 12:07:20,877] INFO Property log.retention.hours is
> > overridden
> > > > to 168 (kafka.utils.VerifiableProperties)
> > > > [2014-03-11 12:07:20,878] INFO Property num.network.threads is
> > overridden
> > > > to 2 (kafka.utils.VerifiableProperties)
> > > > [2014-03-11 12:07:20,878] INFO Property socket.receive.buffer.bytes is
> > > > overridden to 1048576 (kafka.utils.VerifiableProperties)
> > > > [2014-03-11 12:07:20,878] INFO Property
> > zookeeper.connection.timeout.msis overridden to 1000000
> > (kafka.utils.VerifiableProperties)
> > > > [2014-03-11 12:07:20,878] INFO Property num.partitions is overridden
> > to 2
> > > > (kafka.utils.VerifiableProperties)
> > > > [2014-03-11 12:07:20,879] INFO Property log.flush.interval.messages is
> > > > overridden to 10000 (kafka.utils.VerifiableProperties)
> > > > [2014-03-11 12:07:20,879] INFO Property log.segment.bytes is
> > overridden to
> > > > 536870912 (kafka.utils.VerifiableProperties)
> > > > [2014-03-11 12:07:20,897] INFO [Kafka Server 1], Starting
> > > > (kafka.server.KafkaServer)
> > > > [2014-03-11 12:07:20,922] INFO [Log Manager on Broker 1] Log directory
> > > > '/tmp/kafka-logs' not found, creating it. (kafka.log.LogManager)
> > > > [2014-03-11 12:07:20,938] INFO [Log Manager on Broker 1] Starting log
> > > > cleaner every 60000 ms (kafka.log.LogManager)
> > > > [2014-03-11 12:07:20,951] INFO [Log Manager on Broker 1] Starting log
> > > > flusher every 3000 ms with the following overrides Map()
> > > > (kafka.log.LogManager)
> > > > [2014-03-11 12:07:20,991] INFO Awaiting socket connections on
> > 0.0.0.0:9092.
> > > > (kafka.network.Acceptor)
> > > > [2014-03-11 12:07:20,993] INFO [Socket Server on Broker 1], Started
> > > > (kafka.network.SocketServer)
> > > > [2014-03-11 12:07:21,002] INFO connecting to ZK: 192.168.1.120:2181
> > (kafka.server.KafkaZooKeeper)
> > > > [2014-03-11 12:07:21,018] INFO Starting ZkClient event thread.
> > > > (org.I0Itec.zkclient.ZkEventThread)
> > > > [2014-03-11 12:07:21,029] INFO Client
> > > > environment:zookeeper.version=3.3.3-1203054, built on 11/17/2011 05:47
> > GMT
> > > > (org.apache.zookeeper.ZooKeeper)
> > > > [2014-03-11 12:07:21,029] INFO Client environment:host.name=localhost
> > > > (org.apache.zookeeper.ZooKeeper)
> > > > [2014-03-11 12:07:21,029] INFO Client environment:java.version=1.7.0_51
> > > > (org.apache.zookeeper.ZooKeeper)
> > > > [2014-03-11 12:07:21,029] INFO Client environment:java.vendor=Oracle
> > > > Corporation (org.apache.zookeeper.ZooKeeper)
> > > > [2014-03-11 12:07:21,029] INFO Client
> > > >
> > environment:java.home=/usr/lib/jvm/java-1.7.0-openjdk-1.7.0.51.x86_64/jre
> > > > (org.apache.zookeeper.ZooKeeper)
> > > > [2014-03-11 12:07:21,029] INFO Client
> > > >
> > environment:java.class.path=:/opt/kafka-0.8.0/bin/../core/target/scala-2.8.0/kafka_2.8.0-0.8.0.jar:/opt/kafka-0.8.0/bin/../core/target/scala-2.8.0/kafka-assembly-0.8.0-deps.jar:/opt/kafka-0.8.0/bin/../perf/target/scala-2.8.0/kafka-perf_2.8.0-0.8.0.jar:/opt/kafka-0.8.0/bin/../libs/*.jar:/opt/kafka-0.8.0/bin/../kafka*.jar
> > > > (org.apache.zookeeper.ZooKeeper)
> > > > [2014-03-11 12:07:21,029] INFO Client
> > > >
> > environment:java.library.path=/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib
> > > > (org.apache.zookeeper.ZooKeeper)
> > > > [2014-03-11 12:07:21,029] INFO Client environment:java.io.tmpdir=/tmp
> > > > (org.apache.zookeeper.ZooKeeper)
> > > > [2014-03-11 12:07:21,030] INFO Client environment:java.compiler=<NA>
> > > > (org.apache.zookeeper.ZooKeeper)
> > > > [2014-03-11 12:07:21,030] INFO Client environment:os.name=Linux
> > > > (org.apache.zookeeper.ZooKeeper)
> > > > [2014-03-11 12:07:21,030] INFO Client environment:os.arch=amd64
> > > > (org.apache.zookeeper.ZooKeeper)
> > > > [2014-03-11 12:07:21,030] INFO Client
> > > > environment:os.version=2.6.32-279.el6.x86_64
> > > > (org.apache.zookeeper.ZooKeeper)
> > > > [2014-03-11 12:07:21,030] INFO Client environment:user.name=root
> > > > (org.apache.zookeeper.ZooKeeper)
> > > > [2014-03-11 12:07:21,030] INFO Client environment:user.home=/root
> > > > (org.apache.zookeeper.ZooKeeper)
> > > > [2014-03-11 12:07:21,030] INFO Client
> > environment:user.dir=/var/zookeeper
> > > > (org.apache.zookeeper.ZooKeeper)
> > > > [2014-03-11 12:07:21,031] INFO Initiating client connection,
> > connectString=
> > > > 192.168.1.120:2181 sessionTimeout=6000
> > > > watcher=org.I0Itec.zkclient.ZkClient@45419cee
> > (org.apache.zookeeper.ZooKeeper)
> > > > [2014-03-11 12:07:21,043] INFO Opening socket connection to server /
> > > > 192.168.1.120:2181 (org.apache.zookeeper.ClientCnxn)
> > > > [2014-03-11 12:07:21,049] INFO Socket connection established to
> > > > 192.168.1.120/192.168.1.120:2181, initiating session
> > > > (org.apache.zookeeper.ClientCnxn)
> > > > [2014-03-11 12:07:21,058] INFO Session establishment complete on server
> > > > 192.168.1.120/192.168.1.120:2181, sessionid = 0x144b104e0210005,
> > > > negotiated timeout = 6000 (org.apache.zookeeper.ClientCnxn)
> > > > [2014-03-11 12:07:21,060] INFO zookeeper state changed (SyncConnected)
> > > > (org.I0Itec.zkclient.ZkClient)
> > > > [2014-03-11 12:07:21,095] INFO Registered broker 1 at path
> > /brokers/ids/1
> > > > with address localhost:9092. (kafka.utils.ZkUtils$)
> > > > [2014-03-11 12:07:21,096] INFO [Kafka Server 1], Connecting to ZK:
> > > > 192.168.1.120:2181 (kafka.server.KafkaServer)
> > > > [2014-03-11 12:07:21,222] INFO Will not load MX4J, mx4j-tools.jar is
> > not
> > > > in the classpath (kafka.utils.Mx4jLoader$)
> > > > [2014-03-11 12:07:21,249] INFO conflict in /controller data: {
> > > > "brokerid":1, "timestamp":"1394539641229", "version":1 } stored data: {
> > > > "brokerid":2, "timestamp":"1394539533037", "version":1 }
> > > > (kafka.utils.ZkUtils$)
> > > > [2014-03-11 12:07:21,469] INFO [Kafka Server 1], Started
> > > > (kafka.server.KafkaServer)
> > > >
> > > > Could this be an issue?
> > > > INFO Registered broker 2 at path /brokers/ids/2 with address
> > > > localhost:9092.
> > > > INFO Registered broker 1 at path /brokers/ids/1 wi
Re: Remote Zookeeper
Posted by Benjamin Black <b...@b3k.us>.
If they are on different physical machines then binding to localhost/using
localhost as the host name is unlikely to be what you want.
On Tuesday, March 11, 2014, A A <an...@hotmail.com> wrote:
> Thanks. I already checked out the wiki and step 6 in particular.
> Just to clarify, Zk, Broker1 and Broker2 are on 3 different physical
> machines.
>
> > From: b@b3k.us <javascript:;>
> > Date: Tue, 11 Mar 2014 10:58:50 -0700
> > Subject: Re: Remote Zookeeper
> > To: users@kafka.apache.org <javascript:;>
> >
> > "I'd suggest deleting the zookeeper and Kafka logs and starting over
> using
> > the getting started tutorial from the wiki."
> >
> > |
> > v
> >
> > "Could this be an issue?
> > INFO Registered broker 2 at path /brokers/ids/2 with address
> localhost:9092.
> > INFO Registered broker 1 at path /brokers/ids/1 with address
> > localhost:9092."
> >
> > yes. see step 6: https://kafka.apache.org/documentation.html#quickstart
> >
> >
> >
> > On Tue, Mar 11, 2014 at 10:54 AM, A A <andthereitgoes@hotmail.com<javascript:;>>
> wrote:
> >
> > > Okay thanks. I removed zookeeper and kafka logs
> > >
> > > /tmp/kafka-logs
> > > /tmp/logs
> > > /var/zookeeper/version-2/
> > > /tmp/zookeeper*.log (on Zk)
> > >
> > > restarted zookeeper
> > >
> > > $KAFKA_HOME/bin/kafka-list-topic.sh --zookeeper 192.168.1.120:2181
> > > no topics exist!
> > >
> > > Started Broker 2
> > > $KAFKA/kafka-server-start.sh $KAFKA_CONFIG/server.properties
> > > [2014-03-11 12:05:32,651] INFO Verifying properties
> > > (kafka.utils.VerifiableProperties)
> > > [2014-03-11 12:05:32,681] INFO Property port is overridden to 9092
> > > (kafka.utils.VerifiableProperties)
> > > [2014-03-11 12:05:32,681] INFO Property socket.request.max.bytes is
> > > overridden to 104857600 (kafka.utils.VerifiableProperties)
> > > [2014-03-11 12:05:32,682] INFO Property num.io.threads is overridden
> to 2
> > > (kafka.utils.VerifiableProperties)
> > > [2014-03-11 12:05:32,682] INFO Property log.dirs is overridden to
> > > /tmp/kafka-logs (kafka.utils.VerifiableProperties)
> > > [2014-03-11 12:05:32,682] INFO Property log.cleanup.interval.mins is
> > > overridden to 1 (kafka.utils.VerifiableProperties)
> > > [2014-03-11 12:05:32,682] INFO Property socket.send.buffer.bytes is
> > > overridden to 1048576 (kafka.utils.VerifiableProperties)
> > > [2014-03-11 12:05:32,683] INFO Property log.flush.interval.ms is
> > > overridden to 1000 (kafka.utils.VerifiableProperties)
> > > [2014-03-11 12:05:32,683] INFO Property zookeeper.connect is
> overridden to
> > > 192.168.1.120:2181 (kafka.utils.VerifiableProperties)
> > > [2014-03-11 12:05:32,684] INFO Property broker.id is overridden to 2
> > > (kafka.utils.VerifiableProperties)
> > > [2014-03-11 12:05:32,684] INFO Property log.retention.hours is
> overridden
> > > to 168 (kafka.utils.VerifiableProperties)
> > > [2014-03-11 12:05:32,684] INFO Property num.network.threads is
> overridden
> > > to 2 (kafka.utils.VerifiableProperties)
> > > [2014-03-11 12:05:32,684] INFO Property socket.receive.buffer.bytes is
> > > overridden to 1048576 (kafka.utils.VerifiableProperties)
> > > [2014-03-11 12:05:32,684] INFO Property
> zookeeper.connection.timeout.msis overridden to 1000000
> (kafka.utils.VerifiableProperties)
> > > [2014-03-11 12:05:32,685] INFO Property num.partitions is overridden
> to 2
> > > (kafka.utils.VerifiableProperties)
> > > [2014-03-11 12:05:32,685] INFO Property log.flush.interval.messages is
> > > overridden to 10000 (kafka.utils.VerifiableProperties)
> > > [2014-03-11 12:05:32,685] INFO Property log.segment.bytes is
> overridden to
> > > 536870912 (kafka.utils.VerifiableProperties)
> > > [2014-03-11 12:05:32,703] INFO [Kafka Server 2], Starting
> > > (kafka.server.KafkaServer)
> > > [2014-03-11 12:05:32,739] INFO [Log Manager on Broker 2] Starting log
> > > cleaner every 60000 ms (kafka.log.LogManager)
> > > [2014-03-11 12:05:32,753] INFO [Log Manager on Broker 2] Starting log
> > > flusher every 3000 ms with the following overrides Map()
> > > (kafka.log.LogManager)
> > > [2014-03-11 12:05:32,791] INFO Awaiting socket connections on
> 0.0.0.0:9092.
> > > (kafka.network.Acceptor)
> > > [2014-03-11 12:05:32,794] INFO [Socket Server on Broker 2], Started
> > > (kafka.network.SocketServer)
> > > [2014-03-11 12:05:32,801] INFO connecting to ZK: 192.168.1.120:2181
> (kafka.server.KafkaZooKeeper)
> > > [2014-03-11 12:05:32,815] INFO Starting ZkClient event thread.
> > > (org.I0Itec.zkclient.ZkEventThread)
> > > [2014-03-11 12:05:32,817] INFO Client
> > > environment:zookeeper.version=3.3.3-1203054, built on 11/17/2011 05:47
> GMT
> > > (org.apache.zookeeper.ZooKeeper)
> > > [2014-03-11 12:05:32,817] INFO Client environment:host.name=localhost
> > > (org.apache.zookeeper.ZooKeeper)
> > > [2014-03-11 12:05:32,817] INFO Client environment:java.version=1.7.0_51
> > > (org.apache.zookeeper.ZooKeeper)
> > > [2014-03-11 12:05:32,822] INFO Client environment:java.vendor=Oracle
> > > Corporation (org.apache.zookeeper.ZooKeeper)
> > > [2014-03-11 12:05:32,822] INFO Client
> > >
> environment:java.home=/usr/lib/jvm/java-1.7.0-openjdk-1.7.0.51.x86_64/jre
> > > (org.apache.zookeeper.ZooKeeper)
> > > [2014-03-11 12:05:32,822] INFO Client
> > >
> environment:java.class.path=:/opt/kafka-0.8.0/bin/../core/target/scala-2.8.0/kafka_2.8.0-0.8.0.jar:/opt/kafka-0.8.0/bin/../core/target/scala-2.8.0/kafka-assembly-0.8.0-deps.jar:/opt/kafka-0.8.0/bin/../perf/target/scala-2.8.0/kafka-perf_2.8.0-0.8.0.jar:/opt/kafka-0.8.0/bin/../libs/*.jar:/opt/kafka-0.8.0/bin/../kafka*.jar
> > > (org.apache.zookeeper.ZooKeeper)
> > > [2014-03-11 12:05:32,822] INFO Client
> > >
> environment:java.library.path=/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib
> > > (org.apache.zookeeper.ZooKeeper)
> > > [2014-03-11 12:05:32,822] INFO Client environment:java.io.tmpdir=/tmp
> > > (org.apache.zookeeper.ZooKeeper)
> > > [2014-03-11 12:05:32,822] INFO Client environment:java.compiler=<NA>
> > > (org.apache.zookeeper.ZooKeeper)
> > > [2014-03-11 12:05:32,822] INFO Client environment:os.name=Linux
> > > (org.apache.zookeeper.ZooKeeper)
> > > [2014-03-11 12:05:32,822] INFO Client environment:os.arch=amd64
> > > (org.apache.zookeeper.ZooKeeper)
> > > [2014-03-11 12:05:32,822] INFO Client
> > > environment:os.version=2.6.32-279.el6.x86_64
> > > (org.apache.zookeeper.ZooKeeper)
> > > [2014-03-11 12:05:32,822] INFO Client environment:user.name=root
> > > (org.apache.zookeeper.ZooKeeper)
> > > [2014-03-11 12:05:32,822] INFO Client environment:user.home=/root
> > > (org.apache.zookeeper.ZooKeeper)
> > > [2014-03-11 12:05:32,822] INFO Client environment:user.dir=/opt
> > > (org.apache.zookeeper.ZooKeeper)
> > > [2014-03-11 12:05:32,825] INFO Initiating client connection,
> connectString=
> > > 192.168.1.120:2181 sessionTimeout=6000
> > > watcher=org.I0Itec.zkclient.ZkClient@721e34d8
> (org.apache.zookeeper.ZooKeeper)
> > > [2014-03-11 12:05:32,842] INFO Opening socket connection to server /
> > > 192.168.1.120:2181 (org.apache.zookeeper.ClientCnxn)
> > > [2014-03-11 12:05:32,846] INFO Socket connection established to
> > > 192.168.1.120/192.168.1.120:2181, initiating session
> > > (org.apache.zookeeper.ClientCnxn)
> > > [2014-03-11 12:05:32,854] INFO Session establishment complete on server
> > > 192.168.1.120/192.168.1.120:2181, sessionid = 0x144b104e0210004,
> > > negotiated timeout = 6000 (org.apache.zookeeper.ClientCnxn)
> > > [2014-03-11 12:05:32,856] INFO zookeeper state changed (SyncConnected)
> > > (org.I0Itec.zkclient.ZkClient)
> > > [2014-03-11 12:05:32,891] INFO Registered broker 2 at path
> /brokers/ids/2
> > > with address localhost:9092. (kafka.utils.ZkUtils$)
> > > [2014-03-11 12:05:32,892] INFO [Kafka Server 2], Connecting to ZK:
> > > 192.168.1.120:2181 (kafka.server.KafkaServer)
> > > [2014-03-11 12:05:33,031] INFO Will not load MX4J, mx4j-tools.jar is
> not
> > > in the classpath (kafka.utils.Mx4jLoader$)
> > > [2014-03-11 12:05:33,041] INFO 2 successfully elected as leader
> > > (kafka.server.ZookeeperLeaderElector)
> > > [2014-03-11 12:05:33,371] INFO [Kafka Server 2], Started
> > > (kafka.server.KafkaServer)
> > > [2014-03-11 12:05:33,377] INFO New leader is 2
> > > (kafka.server.ZookeeperLeaderElector$LeaderChangeListener)
> > >
> > > Then started Broker 1
> > >
> > > $KAFKA/kafka-server-start.sh $KAFKA_CONFIG/server.properties
> > > [2014-03-11 12:07:20,843] INFO Verifying properties
> > > (kafka.utils.VerifiableProperties)
> > > [2014-03-11 12:07:20,875] INFO Property port is overridden to 9092
> > > (kafka.utils.VerifiableProperties)
> > > [2014-03-11 12:07:20,876] INFO Property socket.request.max.bytes is
> > > overridden to 104857600 (kafka.utils.VerifiableProperties)
> > > [2014-03-11 12:07:20,876] INFO Property num.io.threads is overridden
> to 2
> > > (kafka.utils.VerifiableProperties)
> > > [2014-03-11 12:07:20,876] INFO Property log.dirs is overridden to
> > > /tmp/kafka-logs (kafka.utils.VerifiableProperties)
> > > [2014-03-11 12:07:20,876] INFO Property log.cleanup.interval.mins is
> > > overridden to 1 (kafka.utils.VerifiableProperties)
> > > [2014-03-11 12:07:20,877] INFO Property socket.send.buffer.bytes is
> > > overridden to 1048576 (kafka.utils.VerifiableProperties)
> > > [2014-03-11 12:07:20,877] INFO Property log.flush.interval.ms is
> > > overridden to 1000 (kafka.utils.VerifiableProperties)
> > > [2014-03-11 12:07:20,877] INFO Property zookeeper.connect is
> overridden to
> > > 192.168.1.120:2181 (kafka.utils.VerifiableProperties)
> > > [2014-03-11 12:07:20,877] INFO Property broker.id is overridden to 1
> > > (kafka.utils.VerifiableProperties)
> > > [2014-03-11 12:07:20,877] INFO Property log.retention.hours is
> overridden
> > > to 168 (kafka.utils.VerifiableProperties)
> > > [2014-03-11 12:07:20,878] INFO Property num.network.threads is
> overridden
> > > to 2 (kafka.utils.VerifiableProperties)
> > > [2014-03-11 12:07:20,878] INFO Property socket.receive.buffer.bytes is
> > > overridden to 1048576 (kafka.utils.VerifiableProperties)
> > > [2014-03-11 12:07:20,878] INFO Property
> zookeeper.connection.timeout.msis overridden to 1000000
> (kafka.utils.VerifiableProperties)
> > > [2014-03-11 12:07:20,878] INFO Property num.partitions is overridden
> to 2
> > > (kafka.utils.VerifiableProperties)
> > > [2014-03-11 12:07:20,879] INFO Property log.flush.interval.messages is
> > > overridden to 10000 (kafka.utils.VerifiableProperties)
> > > [2014-03-11 12:07:20,879] INFO Property log.segment.bytes is
> overridden to
> > > 536870912 (kafka.utils.VerifiableProperties)
> > > [2014-03-11 12:07:20,897] INFO [Kafka Server 1], Starting
> > > (kafka.server.KafkaServer)
> > > [2014-03-11 12:07:20,922] INFO [Log Manager on Broker 1] Log directory
> > > '/tmp/kafka-logs' not found, creating it. (kafka.log.LogManager)
> > > [2014-03-11 12:07:20,938] INFO [Log Manager on Broker 1] Starting log
> > > cleaner every 60000 ms (kafka.log.LogManager)
> > > [2014-03-11 12:07:20,951] INFO [Log Manager on Broker 1] Starting log
> > > flusher every 3000 ms with the following overrides Map()
> > > (kafka.log.LogManager)
> > > [2014-03-11 12:07:20,991] INFO Awaiting socket connections on
> 0.0.0.0:9092.
> > > (kafka.network.Acceptor)
> > > [2014-03-11 12:07:20,993] INFO [Socket Server on Broker 1], Started
> > > (kafka.network.SocketServer)
> > > [2014-03-11 12:07:21,002] INFO connecting to ZK: 192.168.1.120:2181
> (kafka.server.KafkaZooKeeper)
> > > [2014-03-11 12:07:21,018] INFO Starting ZkClient event thread.
> > > (org.I0Itec.zkclient.ZkEventThread)
> > > [2014-03-11 12:07:21,029] INFO Client
> > > environment:zookeeper.version=3.3.3-1203054, built on 11/17/2011 05:47
> GMT
> > > (org.apache.zookeeper.ZooKeeper)
> > > [2014-03-11 12:07:21,029] INFO Client environment:host.name=localhost
> > > (org.apache.zookeeper.ZooKeeper)
> > > [2014-03-11 12:07:21,029] INFO Client environment:java.version=1.7.0_51
> > > (org.apache.zookeeper.ZooKeeper)
> > > [2014-03-11 12:07:21,029] INFO Client environment:java.vendor=Oracle
> > > Corporation (org.apache.zookeeper.ZooKeeper)
> > > [2014-03-11 12:07:21,029] INFO Client
> > >
> environment:java.home=/usr/lib/jvm/java-1.7.0-openjdk-1.7.0.51.x86_64/jre
> > > (org.apache.zookeeper.ZooKeeper)
> > > [2014-03-11 12:07:21,029] INFO Client
> > >
> environment:java.class.path=:/opt/kafka-0.8.0/bin/../core/target/scala-2.8.0/kafka_2.8.0-0.8.0.jar:/opt/kafka-0.8.0/bin/../core/target/scala-2.8.0/kafka-assembly-0.8.0-deps.jar:/opt/kafka-0.8.0/bin/../perf/target/scala-2.8.0/kafka-perf_2.8.0-0.8.0.jar:/opt/kafka-0.8.0/bin/../libs/*.jar:/opt/kafka-0.8.0/bin/../kafka*.jar
> > > (org.apache.zookeeper.ZooKeeper)
> > > [2014-03-11 12:07:21,029] INFO Client
> > >
> environment:java.library.path=/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib
> > > (org.apache.zookeeper.ZooKeeper)
> > > [2014-03-11 12:07:21,029] INFO Client environment:java.io.tmpdir=/tmp
> > > (org.apache.zookeeper.ZooKeeper)
> > > [2014-03-11 12:07:21,030] INFO Client environment:java.compiler=<NA>
> > > (org.apache.zookeeper.ZooKeeper)
> > > [2014-03-11 12:07:21,030] INFO Client environment:os.name=Linux
> > > (org.apache.zookeeper.ZooKeeper)
> > > [2014-03-11 12:07:21,030] INFO Client environment:os.arch=amd64
> > > (org.apache.zookeeper.ZooKeeper)
> > > [2014-03-11 12:07:21,030] INFO Client
> > > environment:os.version=2.6.32-279.el6.x86_64
> > > (org.apache.zookeeper.ZooKeeper)
> > > [2014-03-11 12:07:21,030] INFO Client environment:user.name=root
> > > (org.apache.zookeeper.ZooKeeper)
> > > [2014-03-11 12:07:21,030] INFO Client environment:user.home=/root
> > > (org.apache.zookeeper.ZooKeeper)
> > > [2014-03-11 12:07:21,030] INFO Client
> environment:user.dir=/var/zookeeper
> > > (org.apache.zookeeper.ZooKeeper)
> > > [2014-03-11 12:07:21,031] INFO Initiating client connection,
> connectString=
> > > 192.168.1.120:2181 sessionTimeout=6000
> > > watcher=org.I0Itec.zkclient.ZkClient@45419cee
> (org.apache.zookeeper.ZooKeeper)
> > > [2014-03-11 12:07:21,043] INFO Opening socket connection to server /
> > > 192.168.1.120:2181 (org.apache.zookeeper.ClientCnxn)
> > > [2014-03-11 12:07:21,049] INFO Socket connection established to
> > > 192.168.1.120/192.168.1.120:2181, initiating session
> > > (org.apache.zookeeper.ClientCnxn)
> > > [2014-03-11 12:07:21,058] INFO Session establishment complete on server
> > > 192.168.1.120/192.168.1.120:2181, sessionid = 0x144b104e0210005,
> > > negotiated timeout = 6000 (org.apache.zookeeper.ClientCnxn)
> > > [2014-03-11 12:07:21,060] INFO zookeeper state changed (SyncConnected)
> > > (org.I0Itec.zkclient.ZkClient)
> > > [2014-03-11 12:07:21,095] INFO Registered broker 1 at path
> /brokers/ids/1
> > > with address localhost:9092. (kafka.utils.ZkUtils$)
> > > [2014-03-11 12:07:21,096] INFO [Kafka Server 1], Connecting to ZK:
> > > 192.168.1.120:2181 (kafka.server.KafkaServer)
> > > [2014-03-11 12:07:21,222] INFO Will not load MX4J, mx4j-tools.jar is
> not
> > > in the classpath (kafka.utils.Mx4jLoader$)
> > > [2014-03-11 12:07:21,249] INFO conflict in /controller data: {
> > > "brokerid":1, "timestamp":"1394539641229", "version":1 } stored data: {
> > > "brokerid":2, "timestamp":"1394539533037", "version":1 }
> > > (kafka.utils.ZkUtils$)
> > > [2014-03-11 12:07:21,469] INFO [Kafka Server 1], Started
> > > (kafka.server.KafkaServer)
> > >
> > > Could this be an issue?
> > > INFO Registered broker 2 at path /brokers/ids/2 with address
> > > localhost:9092.
> > > INFO Registered broker 1 at path /brokers/ids/1 wi
RE: Remote Zookeeper
Posted by A A <an...@hotmail.com>.
Thanks. I already checked out the wiki and step 6 in particular.
Just to clarify, Zk, Broker1 and Broker2 are on 3 different physical machines.
> From: b@b3k.us
> Date: Tue, 11 Mar 2014 10:58:50 -0700
> Subject: Re: Remote Zookeeper
> To: users@kafka.apache.org
>
> "I'd suggest deleting the zookeeper and Kafka logs and starting over using
> the getting started tutorial from the wiki."
>
> |
> v
>
> "Could this be an issue?
> INFO Registered broker 2 at path /brokers/ids/2 with address localhost:9092.
> INFO Registered broker 1 at path /brokers/ids/1 with address
> localhost:9092."
>
> yes. see step 6: https://kafka.apache.org/documentation.html#quickstart
>
>
>
> On Tue, Mar 11, 2014 at 10:54 AM, A A <an...@hotmail.com> wrote:
>
> > Okay thanks. I removed zookeeper and kafka logs
> >
> > /tmp/kafka-logs
> > /tmp/logs
> > /var/zookeeper/version-2/
> > /tmp/zookeeper*.log (on Zk)
> >
> > restarted zookeeper
> >
> > $KAFKA_HOME/bin/kafka-list-topic.sh --zookeeper 192.168.1.120:2181
> > no topics exist!
> >
> > Started Broker 2
> > $KAFKA/kafka-server-start.sh $KAFKA_CONFIG/server.properties
> > [2014-03-11 12:05:32,651] INFO Verifying properties
> > (kafka.utils.VerifiableProperties)
> > [2014-03-11 12:05:32,681] INFO Property port is overridden to 9092
> > (kafka.utils.VerifiableProperties)
> > [2014-03-11 12:05:32,681] INFO Property socket.request.max.bytes is
> > overridden to 104857600 (kafka.utils.VerifiableProperties)
> > [2014-03-11 12:05:32,682] INFO Property num.io.threads is overridden to 2
> > (kafka.utils.VerifiableProperties)
> > [2014-03-11 12:05:32,682] INFO Property log.dirs is overridden to
> > /tmp/kafka-logs (kafka.utils.VerifiableProperties)
> > [2014-03-11 12:05:32,682] INFO Property log.cleanup.interval.mins is
> > overridden to 1 (kafka.utils.VerifiableProperties)
> > [2014-03-11 12:05:32,682] INFO Property socket.send.buffer.bytes is
> > overridden to 1048576 (kafka.utils.VerifiableProperties)
> > [2014-03-11 12:05:32,683] INFO Property log.flush.interval.ms is
> > overridden to 1000 (kafka.utils.VerifiableProperties)
> > [2014-03-11 12:05:32,683] INFO Property zookeeper.connect is overridden to
> > 192.168.1.120:2181 (kafka.utils.VerifiableProperties)
> > [2014-03-11 12:05:32,684] INFO Property broker.id is overridden to 2
> > (kafka.utils.VerifiableProperties)
> > [2014-03-11 12:05:32,684] INFO Property log.retention.hours is overridden
> > to 168 (kafka.utils.VerifiableProperties)
> > [2014-03-11 12:05:32,684] INFO Property num.network.threads is overridden
> > to 2 (kafka.utils.VerifiableProperties)
> > [2014-03-11 12:05:32,684] INFO Property socket.receive.buffer.bytes is
> > overridden to 1048576 (kafka.utils.VerifiableProperties)
> > [2014-03-11 12:05:32,684] INFO Property zookeeper.connection.timeout.msis overridden to 1000000 (kafka.utils.VerifiableProperties)
> > [2014-03-11 12:05:32,685] INFO Property num.partitions is overridden to 2
> > (kafka.utils.VerifiableProperties)
> > [2014-03-11 12:05:32,685] INFO Property log.flush.interval.messages is
> > overridden to 10000 (kafka.utils.VerifiableProperties)
> > [2014-03-11 12:05:32,685] INFO Property log.segment.bytes is overridden to
> > 536870912 (kafka.utils.VerifiableProperties)
> > [2014-03-11 12:05:32,703] INFO [Kafka Server 2], Starting
> > (kafka.server.KafkaServer)
> > [2014-03-11 12:05:32,739] INFO [Log Manager on Broker 2] Starting log
> > cleaner every 60000 ms (kafka.log.LogManager)
> > [2014-03-11 12:05:32,753] INFO [Log Manager on Broker 2] Starting log
> > flusher every 3000 ms with the following overrides Map()
> > (kafka.log.LogManager)
> > [2014-03-11 12:05:32,791] INFO Awaiting socket connections on 0.0.0.0:9092.
> > (kafka.network.Acceptor)
> > [2014-03-11 12:05:32,794] INFO [Socket Server on Broker 2], Started
> > (kafka.network.SocketServer)
> > [2014-03-11 12:05:32,801] INFO connecting to ZK: 192.168.1.120:2181(kafka.server.KafkaZooKeeper)
> > [2014-03-11 12:05:32,815] INFO Starting ZkClient event thread.
> > (org.I0Itec.zkclient.ZkEventThread)
> > [2014-03-11 12:05:32,817] INFO Client
> > environment:zookeeper.version=3.3.3-1203054, built on 11/17/2011 05:47 GMT
> > (org.apache.zookeeper.ZooKeeper)
> > [2014-03-11 12:05:32,817] INFO Client environment:host.name=localhost
> > (org.apache.zookeeper.ZooKeeper)
> > [2014-03-11 12:05:32,817] INFO Client environment:java.version=1.7.0_51
> > (org.apache.zookeeper.ZooKeeper)
> > [2014-03-11 12:05:32,822] INFO Client environment:java.vendor=Oracle
> > Corporation (org.apache.zookeeper.ZooKeeper)
> > [2014-03-11 12:05:32,822] INFO Client
> > environment:java.home=/usr/lib/jvm/java-1.7.0-openjdk-1.7.0.51.x86_64/jre
> > (org.apache.zookeeper.ZooKeeper)
> > [2014-03-11 12:05:32,822] INFO Client
> > environment:java.class.path=:/opt/kafka-0.8.0/bin/../core/target/scala-2.8.0/kafka_2.8.0-0.8.0.jar:/opt/kafka-0.8.0/bin/../core/target/scala-2.8.0/kafka-assembly-0.8.0-deps.jar:/opt/kafka-0.8.0/bin/../perf/target/scala-2.8.0/kafka-perf_2.8.0-0.8.0.jar:/opt/kafka-0.8.0/bin/../libs/*.jar:/opt/kafka-0.8.0/bin/../kafka*.jar
> > (org.apache.zookeeper.ZooKeeper)
> > [2014-03-11 12:05:32,822] INFO Client
> > environment:java.library.path=/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib
> > (org.apache.zookeeper.ZooKeeper)
> > [2014-03-11 12:05:32,822] INFO Client environment:java.io.tmpdir=/tmp
> > (org.apache.zookeeper.ZooKeeper)
> > [2014-03-11 12:05:32,822] INFO Client environment:java.compiler=<NA>
> > (org.apache.zookeeper.ZooKeeper)
> > [2014-03-11 12:05:32,822] INFO Client environment:os.name=Linux
> > (org.apache.zookeeper.ZooKeeper)
> > [2014-03-11 12:05:32,822] INFO Client environment:os.arch=amd64
> > (org.apache.zookeeper.ZooKeeper)
> > [2014-03-11 12:05:32,822] INFO Client
> > environment:os.version=2.6.32-279.el6.x86_64
> > (org.apache.zookeeper.ZooKeeper)
> > [2014-03-11 12:05:32,822] INFO Client environment:user.name=root
> > (org.apache.zookeeper.ZooKeeper)
> > [2014-03-11 12:05:32,822] INFO Client environment:user.home=/root
> > (org.apache.zookeeper.ZooKeeper)
> > [2014-03-11 12:05:32,822] INFO Client environment:user.dir=/opt
> > (org.apache.zookeeper.ZooKeeper)
> > [2014-03-11 12:05:32,825] INFO Initiating client connection, connectString=
> > 192.168.1.120:2181 sessionTimeout=6000
> > watcher=org.I0Itec.zkclient.ZkClient@721e34d8(org.apache.zookeeper.ZooKeeper)
> > [2014-03-11 12:05:32,842] INFO Opening socket connection to server /
> > 192.168.1.120:2181 (org.apache.zookeeper.ClientCnxn)
> > [2014-03-11 12:05:32,846] INFO Socket connection established to
> > 192.168.1.120/192.168.1.120:2181, initiating session
> > (org.apache.zookeeper.ClientCnxn)
> > [2014-03-11 12:05:32,854] INFO Session establishment complete on server
> > 192.168.1.120/192.168.1.120:2181, sessionid = 0x144b104e0210004,
> > negotiated timeout = 6000 (org.apache.zookeeper.ClientCnxn)
> > [2014-03-11 12:05:32,856] INFO zookeeper state changed (SyncConnected)
> > (org.I0Itec.zkclient.ZkClient)
> > [2014-03-11 12:05:32,891] INFO Registered broker 2 at path /brokers/ids/2
> > with address localhost:9092. (kafka.utils.ZkUtils$)
> > [2014-03-11 12:05:32,892] INFO [Kafka Server 2], Connecting to ZK:
> > 192.168.1.120:2181 (kafka.server.KafkaServer)
> > [2014-03-11 12:05:33,031] INFO Will not load MX4J, mx4j-tools.jar is not
> > in the classpath (kafka.utils.Mx4jLoader$)
> > [2014-03-11 12:05:33,041] INFO 2 successfully elected as leader
> > (kafka.server.ZookeeperLeaderElector)
> > [2014-03-11 12:05:33,371] INFO [Kafka Server 2], Started
> > (kafka.server.KafkaServer)
> > [2014-03-11 12:05:33,377] INFO New leader is 2
> > (kafka.server.ZookeeperLeaderElector$LeaderChangeListener)
> >
> > Then started Broker 1
> >
> > $KAFKA/kafka-server-start.sh $KAFKA_CONFIG/server.properties
> > [2014-03-11 12:07:20,843] INFO Verifying properties
> > (kafka.utils.VerifiableProperties)
> > [2014-03-11 12:07:20,875] INFO Property port is overridden to 9092
> > (kafka.utils.VerifiableProperties)
> > [2014-03-11 12:07:20,876] INFO Property socket.request.max.bytes is
> > overridden to 104857600 (kafka.utils.VerifiableProperties)
> > [2014-03-11 12:07:20,876] INFO Property num.io.threads is overridden to 2
> > (kafka.utils.VerifiableProperties)
> > [2014-03-11 12:07:20,876] INFO Property log.dirs is overridden to
> > /tmp/kafka-logs (kafka.utils.VerifiableProperties)
> > [2014-03-11 12:07:20,876] INFO Property log.cleanup.interval.mins is
> > overridden to 1 (kafka.utils.VerifiableProperties)
> > [2014-03-11 12:07:20,877] INFO Property socket.send.buffer.bytes is
> > overridden to 1048576 (kafka.utils.VerifiableProperties)
> > [2014-03-11 12:07:20,877] INFO Property log.flush.interval.ms is
> > overridden to 1000 (kafka.utils.VerifiableProperties)
> > [2014-03-11 12:07:20,877] INFO Property zookeeper.connect is overridden to
> > 192.168.1.120:2181 (kafka.utils.VerifiableProperties)
> > [2014-03-11 12:07:20,877] INFO Property broker.id is overridden to 1
> > (kafka.utils.VerifiableProperties)
> > [2014-03-11 12:07:20,877] INFO Property log.retention.hours is overridden
> > to 168 (kafka.utils.VerifiableProperties)
> > [2014-03-11 12:07:20,878] INFO Property num.network.threads is overridden
> > to 2 (kafka.utils.VerifiableProperties)
> > [2014-03-11 12:07:20,878] INFO Property socket.receive.buffer.bytes is
> > overridden to 1048576 (kafka.utils.VerifiableProperties)
> > [2014-03-11 12:07:20,878] INFO Property zookeeper.connection.timeout.msis overridden to 1000000 (kafka.utils.VerifiableProperties)
> > [2014-03-11 12:07:20,878] INFO Property num.partitions is overridden to 2
> > (kafka.utils.VerifiableProperties)
> > [2014-03-11 12:07:20,879] INFO Property log.flush.interval.messages is
> > overridden to 10000 (kafka.utils.VerifiableProperties)
> > [2014-03-11 12:07:20,879] INFO Property log.segment.bytes is overridden to
> > 536870912 (kafka.utils.VerifiableProperties)
> > [2014-03-11 12:07:20,897] INFO [Kafka Server 1], Starting
> > (kafka.server.KafkaServer)
> > [2014-03-11 12:07:20,922] INFO [Log Manager on Broker 1] Log directory
> > '/tmp/kafka-logs' not found, creating it. (kafka.log.LogManager)
> > [2014-03-11 12:07:20,938] INFO [Log Manager on Broker 1] Starting log
> > cleaner every 60000 ms (kafka.log.LogManager)
> > [2014-03-11 12:07:20,951] INFO [Log Manager on Broker 1] Starting log
> > flusher every 3000 ms with the following overrides Map()
> > (kafka.log.LogManager)
> > [2014-03-11 12:07:20,991] INFO Awaiting socket connections on 0.0.0.0:9092.
> > (kafka.network.Acceptor)
> > [2014-03-11 12:07:20,993] INFO [Socket Server on Broker 1], Started
> > (kafka.network.SocketServer)
> > [2014-03-11 12:07:21,002] INFO connecting to ZK: 192.168.1.120:2181(kafka.server.KafkaZooKeeper)
> > [2014-03-11 12:07:21,018] INFO Starting ZkClient event thread.
> > (org.I0Itec.zkclient.ZkEventThread)
> > [2014-03-11 12:07:21,029] INFO Client
> > environment:zookeeper.version=3.3.3-1203054, built on 11/17/2011 05:47 GMT
> > (org.apache.zookeeper.ZooKeeper)
> > [2014-03-11 12:07:21,029] INFO Client environment:host.name=localhost
> > (org.apache.zookeeper.ZooKeeper)
> > [2014-03-11 12:07:21,029] INFO Client environment:java.version=1.7.0_51
> > (org.apache.zookeeper.ZooKeeper)
> > [2014-03-11 12:07:21,029] INFO Client environment:java.vendor=Oracle
> > Corporation (org.apache.zookeeper.ZooKeeper)
> > [2014-03-11 12:07:21,029] INFO Client
> > environment:java.home=/usr/lib/jvm/java-1.7.0-openjdk-1.7.0.51.x86_64/jre
> > (org.apache.zookeeper.ZooKeeper)
> > [2014-03-11 12:07:21,029] INFO Client
> > environment:java.class.path=:/opt/kafka-0.8.0/bin/../core/target/scala-2.8.0/kafka_2.8.0-0.8.0.jar:/opt/kafka-0.8.0/bin/../core/target/scala-2.8.0/kafka-assembly-0.8.0-deps.jar:/opt/kafka-0.8.0/bin/../perf/target/scala-2.8.0/kafka-perf_2.8.0-0.8.0.jar:/opt/kafka-0.8.0/bin/../libs/*.jar:/opt/kafka-0.8.0/bin/../kafka*.jar
> > (org.apache.zookeeper.ZooKeeper)
> > [2014-03-11 12:07:21,029] INFO Client
> > environment:java.library.path=/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib
> > (org.apache.zookeeper.ZooKeeper)
> > [2014-03-11 12:07:21,029] INFO Client environment:java.io.tmpdir=/tmp
> > (org.apache.zookeeper.ZooKeeper)
> > [2014-03-11 12:07:21,030] INFO Client environment:java.compiler=<NA>
> > (org.apache.zookeeper.ZooKeeper)
> > [2014-03-11 12:07:21,030] INFO Client environment:os.name=Linux
> > (org.apache.zookeeper.ZooKeeper)
> > [2014-03-11 12:07:21,030] INFO Client environment:os.arch=amd64
> > (org.apache.zookeeper.ZooKeeper)
> > [2014-03-11 12:07:21,030] INFO Client
> > environment:os.version=2.6.32-279.el6.x86_64
> > (org.apache.zookeeper.ZooKeeper)
> > [2014-03-11 12:07:21,030] INFO Client environment:user.name=root
> > (org.apache.zookeeper.ZooKeeper)
> > [2014-03-11 12:07:21,030] INFO Client environment:user.home=/root
> > (org.apache.zookeeper.ZooKeeper)
> > [2014-03-11 12:07:21,030] INFO Client environment:user.dir=/var/zookeeper
> > (org.apache.zookeeper.ZooKeeper)
> > [2014-03-11 12:07:21,031] INFO Initiating client connection, connectString=
> > 192.168.1.120:2181 sessionTimeout=6000
> > watcher=org.I0Itec.zkclient.ZkClient@45419cee(org.apache.zookeeper.ZooKeeper)
> > [2014-03-11 12:07:21,043] INFO Opening socket connection to server /
> > 192.168.1.120:2181 (org.apache.zookeeper.ClientCnxn)
> > [2014-03-11 12:07:21,049] INFO Socket connection established to
> > 192.168.1.120/192.168.1.120:2181, initiating session
> > (org.apache.zookeeper.ClientCnxn)
> > [2014-03-11 12:07:21,058] INFO Session establishment complete on server
> > 192.168.1.120/192.168.1.120:2181, sessionid = 0x144b104e0210005,
> > negotiated timeout = 6000 (org.apache.zookeeper.ClientCnxn)
> > [2014-03-11 12:07:21,060] INFO zookeeper state changed (SyncConnected)
> > (org.I0Itec.zkclient.ZkClient)
> > [2014-03-11 12:07:21,095] INFO Registered broker 1 at path /brokers/ids/1
> > with address localhost:9092. (kafka.utils.ZkUtils$)
> > [2014-03-11 12:07:21,096] INFO [Kafka Server 1], Connecting to ZK:
> > 192.168.1.120:2181 (kafka.server.KafkaServer)
> > [2014-03-11 12:07:21,222] INFO Will not load MX4J, mx4j-tools.jar is not
> > in the classpath (kafka.utils.Mx4jLoader$)
> > [2014-03-11 12:07:21,249] INFO conflict in /controller data: {
> > "brokerid":1, "timestamp":"1394539641229", "version":1 } stored data: {
> > "brokerid":2, "timestamp":"1394539533037", "version":1 }
> > (kafka.utils.ZkUtils$)
> > [2014-03-11 12:07:21,469] INFO [Kafka Server 1], Started
> > (kafka.server.KafkaServer)
> >
> > Could this be an issue?
> > INFO Registered broker 2 at path /brokers/ids/2 with address
> > localhost:9092.
> > INFO Registered broker 1 at path /brokers/ids/1 with address
> > localhost:9092.
> >
> >
> >
> > > Date: Tue, 11 Mar 2014 09:51:54 -0700
> > > Subject: Re: Remote Zookeeper
> > > From: b@b3k.us
> > > To: users@kafka.apache.org
> > >
> > > Seems you my have put your cluster in a very confused state with random
> > > addition and removal of brokers and topics. I'd suggest deleting the
> > > zookeeper and Kafka logs and starting over using the getting started
> > > tutorial from the wiki.
> > >
> > > On Tuesday, March 11, 2014, A A <an...@hotmail.com> wrote:
> > >
> > > > I noticed a problem with my topics. The previous two topics would only
> > be
> > > > present on 1 broker. So I created a new topic
> > > >
> > > > $KAFKA_HOME/bin/kafka-create-topic.sh --zookeeper 192.168.1.120:2181--replica
> > 2 --partition 1 --topic test-rep
> > > > $KAFKA_HOME/bin/kafka-list-topic.sh --zookeeper 192.168.1.120:2181
> > > > topic: test partition: 0 leader: 1 replicas: 1 isr: 1
> > > > topic: test-rep partition: 0 leader: 1 replicas: 1,2 isr: 1
> > > > topic: test1 partition: 0 leader: 1 replicas: 1 isr: 1
> > > >
> > > > No there is no firewall. Even then I stopped the iptables. And then I
> > > > tried the following
> > > > $KAFKA_HOME/bin/kafka-console-producer.sh --broker-list
> > > > 192.168.1.123:9092,192.168.1.124:9092 --topic test-rep
> > > > replicated message
> > > >
> > > > [2014-03-11 10:59:25,844] WARN Error while fetching metadata
> > > > [{TopicMetadata for topic test-rep ->
> > > > No partition metadata for topic test-rep due to
> > > > kafka.common.LeaderNotAvailableException}] for topic [test-rep]: class
> > > > kafka.common.LeaderNotAvailableException
> > > > (kafka.producer.BrokerPartitionInfo)
> > > > [2014-03-11 10:59:25,867] WARN Error while fetching metadata
> > > > [{TopicMetadata for topic test-rep ->
> > > > No partition metadata for topic test-rep due to
> > > > kafka.common.LeaderNotAvailableException}] for topic [test-rep]: class
> > > > kafka.common.LeaderNotAvailableException
> > > > (kafka.producer.BrokerPartitionInfo)
> > > > [2014-03-11 10:59:25,868] ERROR Failed to collate messages by topic,
> > > > partition due to: Failed to fetch topic metadata for topic: test-rep
> > > > (kafka.producer.async.DefaultEventHandler)
> > > > [2014-03-11 10:59:25,979] WARN Error while fetching metadata
> > > > [{TopicMetadata for topic test-rep ->
> > > > No partition metadata for topic test-rep due to
> > > > kafka.common.LeaderNotAvailableException}] for topic [test-rep]: class
> > > > kafka.common.LeaderNotAvailableException
> > > > (kafka.producer.BrokerPartitionInfo)
> > > > [2014-03-11 10:59:25,986] WARN Error while fetching metadata
> > > > [{TopicMetadata for topic test-rep ->
> > > > No partition metadata for topic test-rep due to
> > > > kafka.common.LeaderNotAvailableException}] for topic [test-rep]: class
> > > > kafka.common.LeaderNotAvailableException
> > > > (kafka.producer.BrokerPartitionInfo)
> > > > [2014-03-11 10:59:25,987] ERROR Failed to collate messages by topic,
> > > > partition due to: Failed to fetch topic metadata for topic: test-rep
> > > > (kafka.producer.async.DefaultEventHandler)
> > > > [2014-03-11 10:59:26,098] WARN Error while fetching metadata
> > > > [{TopicMetadata for topic test-rep ->
> > > > No partition metadata for topic test-rep due to
> > > > kafka.common.LeaderNotAvailableException}] for topic [test-rep]: class
> > > > kafka.common.LeaderNotAvailableException
> > > > (kafka.producer.BrokerPartitionInfo)
> > > >
> > > >
> > > >
> > > >
> > > > > Date: Tue, 11 Mar 2014 09:05:54 -0700
> > > > > Subject: Re: Remote Zookeeper
> > > > > From: pradeepg26@gmail.com <javascript:;>
> > > > > To: users@kafka.apache.org <javascript:;>
> > > > >
> > > > > Is there a firewall thats blocking connections on port 9092? Also,
> > the
> > > > > broker list should be comma separated.
> > > > >
> > > > >
> > > > > On Tue, Mar 11, 2014 at 9:02 AM, A A <an...@hotmail.com>
> > wrote:
> > > > >
> > > > > > Sorry one of the brokers for was down. Brought it back up. Tried
> > the
> > > > > > following
> > > > > >
> > > > > > $KAFKA_HOME/bin/kafka-console-producer.sh --broker-list
> > localhost:9092
> > > > > > 192.168.1.124:9092 --topic test
> > > > > > hello brokers
> > > > > >
> > > > > > [2014-03-11 10:16:55,547] WARN Error while fetching metadata
> > > > > > [{TopicMetadata for topic test ->
> > > > > > No partition metadata for topic test due to
> > > > > > kafka.common.LeaderNotAvailableException}] for topic [test]: class
> > > > > > kafka.common.LeaderNotAvailableException
> > > > > > (kafka.producer.BrokerPartitionInfo)
> > > > > > [2014-03-11 10:16:55,576] WARN Error while fetching metadata
> > > > > > [{TopicMetadata for topic test ->
> > > > > > No partition metadata for topic test due to
> > > > > > kafka.common.LeaderNotAvailableException}] for topic [test]: class
> > > > > > kafka.common.LeaderNotAvailableException
> > > > > > (kafka.producer.BrokerPartitionInfo)
> > > > > > [2014-03-11 10:16:55,578] ERROR Failed to collate messages by
> > topic,
> > > > > > partition due to: Failed to fetch topic metadata for topic: test
> > > > > > (kafka.producer.async.DefaultEventHandler)
> > > > > > [2014-03-11 10:16:55,693] WARN Error while fetching metadata
> > > > > > [{TopicMetadata for topic test ->
> > > > > > No partition metadata for topic test due to
> > > > > > kafka.common.LeaderNotAvailableException}] for topic [test]: class
> > > > > > kafka.common.LeaderNotAvailableException
> > > > > > (kafka.producer.BrokerPartitionInfo)
> > > > > > [2014-03-11 10:16:55,706] WARN Error while fetching metadata
> > > > > > [{TopicMetadata for topic test ->
> > > > > > No partition metadata for topic test due to
> > > > > > kafka.common.LeaderNotAvailableException}] for topic [test]: class
> > > > > > kafka.common.LeaderNotAvailableException
> > > > > > (kafka.producer.BrokerPartitionInfo)
> > > > > > [2014-03-11 10:16:55,706] ERROR Failed to collate messages by
> > topic,
> > > > > > partition due to: Failed to fetch topic metadata for topic: test
> > > > > > (kafka.producer.async.DefaultEventHandler)
> > > > > > [2014-03-11 10:16:55,815] WARN Error while fetching metadata
> > > > > > [{TopicMetadata for topic test ->
> > > > > > No partition metadata for topic test due to
> > > > > > kafka.common.LeaderNotAvailableException}] for topic [test]: class
> > > > > > kafka.common.LeaderNotAvailableException
> > > > > > (kafka.producer.BrokerPartitionInfo)
> > > > > > [2014-03-11 10:16:55,823] WARN Error while fetching metadata
> > > > > > [{TopicMetadata for topic test ->
> > > > > > No partition metadata for topic test due to
> > > > > > kafka.common.LeaderNotAvailableException}] for topic [test]: class
> > > > > > kafka.common.LeaderNotAvailableException
> > > > > > (kafka.producer.BrokerPartitionInfo)
> > > > > > [2014-03-11 10:16:55,823] ERROR Failed to collate messages by
> > topic,
> > > > > > partition due to: Failed to fetch topic metadata for topic: test
> > > > > > (kafka.producer.async.DefaultEventHandler)
> > > > > > [2014-03-11 10:16:55,934] WARN Error while fetching metadata
> > > > > > [{TopicMetadata for topic test ->
> > > > > > No partition metadata for topic test due to
> > > > > > kafka.common.LeaderNotAvailableException}] for topic [test]: class
> > > > > > kafka.common.LeaderNotAvailableException
> > > > > > (kafka.producer.BrokerPartitionInfo)
> > > > > > [2014-03-11 10:16:55,949] WARN Error while fetching metadata
> > > > > > [{TopicMetadata for topic test ->
> > > > > > No partition metadata for topic test due to
> > > > > > kafka.common.LeaderNotAvailableException}] for topic [test]: class
> > > > > > kafka.common.LeaderNotAvailableException
> > > > > > (kafka.producer.BrokerPartitionInfo)
> > > > > > [2014-03-11 10:16:55,949] ERROR Failed to collate messages by
> > topic,
> > > > > > partition due to: Failed to fetch topic metadata for topic: test
> > > > > > (kafka.producer.async.DefaultEventHandler)
> > > > > > [2014-03-11 10:16:56,057] WARN Error while fetching metadata
> > > > > > [{TopicMetadata for topic test ->
> > > > > > No partition metadata for topic test due to<
> >
> >
Re: Remote Zookeeper
Posted by Benjamin Black <b...@b3k.us>.
"I'd suggest deleting the zookeeper and Kafka logs and starting over using
the getting started tutorial from the wiki."
|
v
"Could this be an issue?
INFO Registered broker 2 at path /brokers/ids/2 with address localhost:9092.
INFO Registered broker 1 at path /brokers/ids/1 with address
localhost:9092."
yes. see step 6: https://kafka.apache.org/documentation.html#quickstart
On Tue, Mar 11, 2014 at 10:54 AM, A A <an...@hotmail.com> wrote:
> Okay thanks. I removed zookeeper and kafka logs
>
> /tmp/kafka-logs
> /tmp/logs
> /var/zookeeper/version-2/
> /tmp/zookeeper*.log (on Zk)
>
> restarted zookeeper
>
> $KAFKA_HOME/bin/kafka-list-topic.sh --zookeeper 192.168.1.120:2181
> no topics exist!
>
> Started Broker 2
> $KAFKA/kafka-server-start.sh $KAFKA_CONFIG/server.properties
> [2014-03-11 12:05:32,651] INFO Verifying properties
> (kafka.utils.VerifiableProperties)
> [2014-03-11 12:05:32,681] INFO Property port is overridden to 9092
> (kafka.utils.VerifiableProperties)
> [2014-03-11 12:05:32,681] INFO Property socket.request.max.bytes is
> overridden to 104857600 (kafka.utils.VerifiableProperties)
> [2014-03-11 12:05:32,682] INFO Property num.io.threads is overridden to 2
> (kafka.utils.VerifiableProperties)
> [2014-03-11 12:05:32,682] INFO Property log.dirs is overridden to
> /tmp/kafka-logs (kafka.utils.VerifiableProperties)
> [2014-03-11 12:05:32,682] INFO Property log.cleanup.interval.mins is
> overridden to 1 (kafka.utils.VerifiableProperties)
> [2014-03-11 12:05:32,682] INFO Property socket.send.buffer.bytes is
> overridden to 1048576 (kafka.utils.VerifiableProperties)
> [2014-03-11 12:05:32,683] INFO Property log.flush.interval.ms is
> overridden to 1000 (kafka.utils.VerifiableProperties)
> [2014-03-11 12:05:32,683] INFO Property zookeeper.connect is overridden to
> 192.168.1.120:2181 (kafka.utils.VerifiableProperties)
> [2014-03-11 12:05:32,684] INFO Property broker.id is overridden to 2
> (kafka.utils.VerifiableProperties)
> [2014-03-11 12:05:32,684] INFO Property log.retention.hours is overridden
> to 168 (kafka.utils.VerifiableProperties)
> [2014-03-11 12:05:32,684] INFO Property num.network.threads is overridden
> to 2 (kafka.utils.VerifiableProperties)
> [2014-03-11 12:05:32,684] INFO Property socket.receive.buffer.bytes is
> overridden to 1048576 (kafka.utils.VerifiableProperties)
> [2014-03-11 12:05:32,684] INFO Property zookeeper.connection.timeout.msis overridden to 1000000 (kafka.utils.VerifiableProperties)
> [2014-03-11 12:05:32,685] INFO Property num.partitions is overridden to 2
> (kafka.utils.VerifiableProperties)
> [2014-03-11 12:05:32,685] INFO Property log.flush.interval.messages is
> overridden to 10000 (kafka.utils.VerifiableProperties)
> [2014-03-11 12:05:32,685] INFO Property log.segment.bytes is overridden to
> 536870912 (kafka.utils.VerifiableProperties)
> [2014-03-11 12:05:32,703] INFO [Kafka Server 2], Starting
> (kafka.server.KafkaServer)
> [2014-03-11 12:05:32,739] INFO [Log Manager on Broker 2] Starting log
> cleaner every 60000 ms (kafka.log.LogManager)
> [2014-03-11 12:05:32,753] INFO [Log Manager on Broker 2] Starting log
> flusher every 3000 ms with the following overrides Map()
> (kafka.log.LogManager)
> [2014-03-11 12:05:32,791] INFO Awaiting socket connections on 0.0.0.0:9092.
> (kafka.network.Acceptor)
> [2014-03-11 12:05:32,794] INFO [Socket Server on Broker 2], Started
> (kafka.network.SocketServer)
> [2014-03-11 12:05:32,801] INFO connecting to ZK: 192.168.1.120:2181(kafka.server.KafkaZooKeeper)
> [2014-03-11 12:05:32,815] INFO Starting ZkClient event thread.
> (org.I0Itec.zkclient.ZkEventThread)
> [2014-03-11 12:05:32,817] INFO Client
> environment:zookeeper.version=3.3.3-1203054, built on 11/17/2011 05:47 GMT
> (org.apache.zookeeper.ZooKeeper)
> [2014-03-11 12:05:32,817] INFO Client environment:host.name=localhost
> (org.apache.zookeeper.ZooKeeper)
> [2014-03-11 12:05:32,817] INFO Client environment:java.version=1.7.0_51
> (org.apache.zookeeper.ZooKeeper)
> [2014-03-11 12:05:32,822] INFO Client environment:java.vendor=Oracle
> Corporation (org.apache.zookeeper.ZooKeeper)
> [2014-03-11 12:05:32,822] INFO Client
> environment:java.home=/usr/lib/jvm/java-1.7.0-openjdk-1.7.0.51.x86_64/jre
> (org.apache.zookeeper.ZooKeeper)
> [2014-03-11 12:05:32,822] INFO Client
> environment:java.class.path=:/opt/kafka-0.8.0/bin/../core/target/scala-2.8.0/kafka_2.8.0-0.8.0.jar:/opt/kafka-0.8.0/bin/../core/target/scala-2.8.0/kafka-assembly-0.8.0-deps.jar:/opt/kafka-0.8.0/bin/../perf/target/scala-2.8.0/kafka-perf_2.8.0-0.8.0.jar:/opt/kafka-0.8.0/bin/../libs/*.jar:/opt/kafka-0.8.0/bin/../kafka*.jar
> (org.apache.zookeeper.ZooKeeper)
> [2014-03-11 12:05:32,822] INFO Client
> environment:java.library.path=/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib
> (org.apache.zookeeper.ZooKeeper)
> [2014-03-11 12:05:32,822] INFO Client environment:java.io.tmpdir=/tmp
> (org.apache.zookeeper.ZooKeeper)
> [2014-03-11 12:05:32,822] INFO Client environment:java.compiler=<NA>
> (org.apache.zookeeper.ZooKeeper)
> [2014-03-11 12:05:32,822] INFO Client environment:os.name=Linux
> (org.apache.zookeeper.ZooKeeper)
> [2014-03-11 12:05:32,822] INFO Client environment:os.arch=amd64
> (org.apache.zookeeper.ZooKeeper)
> [2014-03-11 12:05:32,822] INFO Client
> environment:os.version=2.6.32-279.el6.x86_64
> (org.apache.zookeeper.ZooKeeper)
> [2014-03-11 12:05:32,822] INFO Client environment:user.name=root
> (org.apache.zookeeper.ZooKeeper)
> [2014-03-11 12:05:32,822] INFO Client environment:user.home=/root
> (org.apache.zookeeper.ZooKeeper)
> [2014-03-11 12:05:32,822] INFO Client environment:user.dir=/opt
> (org.apache.zookeeper.ZooKeeper)
> [2014-03-11 12:05:32,825] INFO Initiating client connection, connectString=
> 192.168.1.120:2181 sessionTimeout=6000
> watcher=org.I0Itec.zkclient.ZkClient@721e34d8(org.apache.zookeeper.ZooKeeper)
> [2014-03-11 12:05:32,842] INFO Opening socket connection to server /
> 192.168.1.120:2181 (org.apache.zookeeper.ClientCnxn)
> [2014-03-11 12:05:32,846] INFO Socket connection established to
> 192.168.1.120/192.168.1.120:2181, initiating session
> (org.apache.zookeeper.ClientCnxn)
> [2014-03-11 12:05:32,854] INFO Session establishment complete on server
> 192.168.1.120/192.168.1.120:2181, sessionid = 0x144b104e0210004,
> negotiated timeout = 6000 (org.apache.zookeeper.ClientCnxn)
> [2014-03-11 12:05:32,856] INFO zookeeper state changed (SyncConnected)
> (org.I0Itec.zkclient.ZkClient)
> [2014-03-11 12:05:32,891] INFO Registered broker 2 at path /brokers/ids/2
> with address localhost:9092. (kafka.utils.ZkUtils$)
> [2014-03-11 12:05:32,892] INFO [Kafka Server 2], Connecting to ZK:
> 192.168.1.120:2181 (kafka.server.KafkaServer)
> [2014-03-11 12:05:33,031] INFO Will not load MX4J, mx4j-tools.jar is not
> in the classpath (kafka.utils.Mx4jLoader$)
> [2014-03-11 12:05:33,041] INFO 2 successfully elected as leader
> (kafka.server.ZookeeperLeaderElector)
> [2014-03-11 12:05:33,371] INFO [Kafka Server 2], Started
> (kafka.server.KafkaServer)
> [2014-03-11 12:05:33,377] INFO New leader is 2
> (kafka.server.ZookeeperLeaderElector$LeaderChangeListener)
>
> Then started Broker 1
>
> $KAFKA/kafka-server-start.sh $KAFKA_CONFIG/server.properties
> [2014-03-11 12:07:20,843] INFO Verifying properties
> (kafka.utils.VerifiableProperties)
> [2014-03-11 12:07:20,875] INFO Property port is overridden to 9092
> (kafka.utils.VerifiableProperties)
> [2014-03-11 12:07:20,876] INFO Property socket.request.max.bytes is
> overridden to 104857600 (kafka.utils.VerifiableProperties)
> [2014-03-11 12:07:20,876] INFO Property num.io.threads is overridden to 2
> (kafka.utils.VerifiableProperties)
> [2014-03-11 12:07:20,876] INFO Property log.dirs is overridden to
> /tmp/kafka-logs (kafka.utils.VerifiableProperties)
> [2014-03-11 12:07:20,876] INFO Property log.cleanup.interval.mins is
> overridden to 1 (kafka.utils.VerifiableProperties)
> [2014-03-11 12:07:20,877] INFO Property socket.send.buffer.bytes is
> overridden to 1048576 (kafka.utils.VerifiableProperties)
> [2014-03-11 12:07:20,877] INFO Property log.flush.interval.ms is
> overridden to 1000 (kafka.utils.VerifiableProperties)
> [2014-03-11 12:07:20,877] INFO Property zookeeper.connect is overridden to
> 192.168.1.120:2181 (kafka.utils.VerifiableProperties)
> [2014-03-11 12:07:20,877] INFO Property broker.id is overridden to 1
> (kafka.utils.VerifiableProperties)
> [2014-03-11 12:07:20,877] INFO Property log.retention.hours is overridden
> to 168 (kafka.utils.VerifiableProperties)
> [2014-03-11 12:07:20,878] INFO Property num.network.threads is overridden
> to 2 (kafka.utils.VerifiableProperties)
> [2014-03-11 12:07:20,878] INFO Property socket.receive.buffer.bytes is
> overridden to 1048576 (kafka.utils.VerifiableProperties)
> [2014-03-11 12:07:20,878] INFO Property zookeeper.connection.timeout.msis overridden to 1000000 (kafka.utils.VerifiableProperties)
> [2014-03-11 12:07:20,878] INFO Property num.partitions is overridden to 2
> (kafka.utils.VerifiableProperties)
> [2014-03-11 12:07:20,879] INFO Property log.flush.interval.messages is
> overridden to 10000 (kafka.utils.VerifiableProperties)
> [2014-03-11 12:07:20,879] INFO Property log.segment.bytes is overridden to
> 536870912 (kafka.utils.VerifiableProperties)
> [2014-03-11 12:07:20,897] INFO [Kafka Server 1], Starting
> (kafka.server.KafkaServer)
> [2014-03-11 12:07:20,922] INFO [Log Manager on Broker 1] Log directory
> '/tmp/kafka-logs' not found, creating it. (kafka.log.LogManager)
> [2014-03-11 12:07:20,938] INFO [Log Manager on Broker 1] Starting log
> cleaner every 60000 ms (kafka.log.LogManager)
> [2014-03-11 12:07:20,951] INFO [Log Manager on Broker 1] Starting log
> flusher every 3000 ms with the following overrides Map()
> (kafka.log.LogManager)
> [2014-03-11 12:07:20,991] INFO Awaiting socket connections on 0.0.0.0:9092.
> (kafka.network.Acceptor)
> [2014-03-11 12:07:20,993] INFO [Socket Server on Broker 1], Started
> (kafka.network.SocketServer)
> [2014-03-11 12:07:21,002] INFO connecting to ZK: 192.168.1.120:2181(kafka.server.KafkaZooKeeper)
> [2014-03-11 12:07:21,018] INFO Starting ZkClient event thread.
> (org.I0Itec.zkclient.ZkEventThread)
> [2014-03-11 12:07:21,029] INFO Client
> environment:zookeeper.version=3.3.3-1203054, built on 11/17/2011 05:47 GMT
> (org.apache.zookeeper.ZooKeeper)
> [2014-03-11 12:07:21,029] INFO Client environment:host.name=localhost
> (org.apache.zookeeper.ZooKeeper)
> [2014-03-11 12:07:21,029] INFO Client environment:java.version=1.7.0_51
> (org.apache.zookeeper.ZooKeeper)
> [2014-03-11 12:07:21,029] INFO Client environment:java.vendor=Oracle
> Corporation (org.apache.zookeeper.ZooKeeper)
> [2014-03-11 12:07:21,029] INFO Client
> environment:java.home=/usr/lib/jvm/java-1.7.0-openjdk-1.7.0.51.x86_64/jre
> (org.apache.zookeeper.ZooKeeper)
> [2014-03-11 12:07:21,029] INFO Client
> environment:java.class.path=:/opt/kafka-0.8.0/bin/../core/target/scala-2.8.0/kafka_2.8.0-0.8.0.jar:/opt/kafka-0.8.0/bin/../core/target/scala-2.8.0/kafka-assembly-0.8.0-deps.jar:/opt/kafka-0.8.0/bin/../perf/target/scala-2.8.0/kafka-perf_2.8.0-0.8.0.jar:/opt/kafka-0.8.0/bin/../libs/*.jar:/opt/kafka-0.8.0/bin/../kafka*.jar
> (org.apache.zookeeper.ZooKeeper)
> [2014-03-11 12:07:21,029] INFO Client
> environment:java.library.path=/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib
> (org.apache.zookeeper.ZooKeeper)
> [2014-03-11 12:07:21,029] INFO Client environment:java.io.tmpdir=/tmp
> (org.apache.zookeeper.ZooKeeper)
> [2014-03-11 12:07:21,030] INFO Client environment:java.compiler=<NA>
> (org.apache.zookeeper.ZooKeeper)
> [2014-03-11 12:07:21,030] INFO Client environment:os.name=Linux
> (org.apache.zookeeper.ZooKeeper)
> [2014-03-11 12:07:21,030] INFO Client environment:os.arch=amd64
> (org.apache.zookeeper.ZooKeeper)
> [2014-03-11 12:07:21,030] INFO Client
> environment:os.version=2.6.32-279.el6.x86_64
> (org.apache.zookeeper.ZooKeeper)
> [2014-03-11 12:07:21,030] INFO Client environment:user.name=root
> (org.apache.zookeeper.ZooKeeper)
> [2014-03-11 12:07:21,030] INFO Client environment:user.home=/root
> (org.apache.zookeeper.ZooKeeper)
> [2014-03-11 12:07:21,030] INFO Client environment:user.dir=/var/zookeeper
> (org.apache.zookeeper.ZooKeeper)
> [2014-03-11 12:07:21,031] INFO Initiating client connection, connectString=
> 192.168.1.120:2181 sessionTimeout=6000
> watcher=org.I0Itec.zkclient.ZkClient@45419cee(org.apache.zookeeper.ZooKeeper)
> [2014-03-11 12:07:21,043] INFO Opening socket connection to server /
> 192.168.1.120:2181 (org.apache.zookeeper.ClientCnxn)
> [2014-03-11 12:07:21,049] INFO Socket connection established to
> 192.168.1.120/192.168.1.120:2181, initiating session
> (org.apache.zookeeper.ClientCnxn)
> [2014-03-11 12:07:21,058] INFO Session establishment complete on server
> 192.168.1.120/192.168.1.120:2181, sessionid = 0x144b104e0210005,
> negotiated timeout = 6000 (org.apache.zookeeper.ClientCnxn)
> [2014-03-11 12:07:21,060] INFO zookeeper state changed (SyncConnected)
> (org.I0Itec.zkclient.ZkClient)
> [2014-03-11 12:07:21,095] INFO Registered broker 1 at path /brokers/ids/1
> with address localhost:9092. (kafka.utils.ZkUtils$)
> [2014-03-11 12:07:21,096] INFO [Kafka Server 1], Connecting to ZK:
> 192.168.1.120:2181 (kafka.server.KafkaServer)
> [2014-03-11 12:07:21,222] INFO Will not load MX4J, mx4j-tools.jar is not
> in the classpath (kafka.utils.Mx4jLoader$)
> [2014-03-11 12:07:21,249] INFO conflict in /controller data: {
> "brokerid":1, "timestamp":"1394539641229", "version":1 } stored data: {
> "brokerid":2, "timestamp":"1394539533037", "version":1 }
> (kafka.utils.ZkUtils$)
> [2014-03-11 12:07:21,469] INFO [Kafka Server 1], Started
> (kafka.server.KafkaServer)
>
> Could this be an issue?
> INFO Registered broker 2 at path /brokers/ids/2 with address
> localhost:9092.
> INFO Registered broker 1 at path /brokers/ids/1 with address
> localhost:9092.
>
>
>
> > Date: Tue, 11 Mar 2014 09:51:54 -0700
> > Subject: Re: Remote Zookeeper
> > From: b@b3k.us
> > To: users@kafka.apache.org
> >
> > Seems you my have put your cluster in a very confused state with random
> > addition and removal of brokers and topics. I'd suggest deleting the
> > zookeeper and Kafka logs and starting over using the getting started
> > tutorial from the wiki.
> >
> > On Tuesday, March 11, 2014, A A <an...@hotmail.com> wrote:
> >
> > > I noticed a problem with my topics. The previous two topics would only
> be
> > > present on 1 broker. So I created a new topic
> > >
> > > $KAFKA_HOME/bin/kafka-create-topic.sh --zookeeper 192.168.1.120:2181--replica
> 2 --partition 1 --topic test-rep
> > > $KAFKA_HOME/bin/kafka-list-topic.sh --zookeeper 192.168.1.120:2181
> > > topic: test partition: 0 leader: 1 replicas: 1 isr: 1
> > > topic: test-rep partition: 0 leader: 1 replicas: 1,2 isr: 1
> > > topic: test1 partition: 0 leader: 1 replicas: 1 isr: 1
> > >
> > > No there is no firewall. Even then I stopped the iptables. And then I
> > > tried the following
> > > $KAFKA_HOME/bin/kafka-console-producer.sh --broker-list
> > > 192.168.1.123:9092,192.168.1.124:9092 --topic test-rep
> > > replicated message
> > >
> > > [2014-03-11 10:59:25,844] WARN Error while fetching metadata
> > > [{TopicMetadata for topic test-rep ->
> > > No partition metadata for topic test-rep due to
> > > kafka.common.LeaderNotAvailableException}] for topic [test-rep]: class
> > > kafka.common.LeaderNotAvailableException
> > > (kafka.producer.BrokerPartitionInfo)
> > > [2014-03-11 10:59:25,867] WARN Error while fetching metadata
> > > [{TopicMetadata for topic test-rep ->
> > > No partition metadata for topic test-rep due to
> > > kafka.common.LeaderNotAvailableException}] for topic [test-rep]: class
> > > kafka.common.LeaderNotAvailableException
> > > (kafka.producer.BrokerPartitionInfo)
> > > [2014-03-11 10:59:25,868] ERROR Failed to collate messages by topic,
> > > partition due to: Failed to fetch topic metadata for topic: test-rep
> > > (kafka.producer.async.DefaultEventHandler)
> > > [2014-03-11 10:59:25,979] WARN Error while fetching metadata
> > > [{TopicMetadata for topic test-rep ->
> > > No partition metadata for topic test-rep due to
> > > kafka.common.LeaderNotAvailableException}] for topic [test-rep]: class
> > > kafka.common.LeaderNotAvailableException
> > > (kafka.producer.BrokerPartitionInfo)
> > > [2014-03-11 10:59:25,986] WARN Error while fetching metadata
> > > [{TopicMetadata for topic test-rep ->
> > > No partition metadata for topic test-rep due to
> > > kafka.common.LeaderNotAvailableException}] for topic [test-rep]: class
> > > kafka.common.LeaderNotAvailableException
> > > (kafka.producer.BrokerPartitionInfo)
> > > [2014-03-11 10:59:25,987] ERROR Failed to collate messages by topic,
> > > partition due to: Failed to fetch topic metadata for topic: test-rep
> > > (kafka.producer.async.DefaultEventHandler)
> > > [2014-03-11 10:59:26,098] WARN Error while fetching metadata
> > > [{TopicMetadata for topic test-rep ->
> > > No partition metadata for topic test-rep due to
> > > kafka.common.LeaderNotAvailableException}] for topic [test-rep]: class
> > > kafka.common.LeaderNotAvailableException
> > > (kafka.producer.BrokerPartitionInfo)
> > >
> > >
> > >
> > >
> > > > Date: Tue, 11 Mar 2014 09:05:54 -0700
> > > > Subject: Re: Remote Zookeeper
> > > > From: pradeepg26@gmail.com <javascript:;>
> > > > To: users@kafka.apache.org <javascript:;>
> > > >
> > > > Is there a firewall thats blocking connections on port 9092? Also,
> the
> > > > broker list should be comma separated.
> > > >
> > > >
> > > > On Tue, Mar 11, 2014 at 9:02 AM, A A <an...@hotmail.com>
> wrote:
> > > >
> > > > > Sorry one of the brokers for was down. Brought it back up. Tried
> the
> > > > > following
> > > > >
> > > > > $KAFKA_HOME/bin/kafka-console-producer.sh --broker-list
> localhost:9092
> > > > > 192.168.1.124:9092 --topic test
> > > > > hello brokers
> > > > >
> > > > > [2014-03-11 10:16:55,547] WARN Error while fetching metadata
> > > > > [{TopicMetadata for topic test ->
> > > > > No partition metadata for topic test due to
> > > > > kafka.common.LeaderNotAvailableException}] for topic [test]: class
> > > > > kafka.common.LeaderNotAvailableException
> > > > > (kafka.producer.BrokerPartitionInfo)
> > > > > [2014-03-11 10:16:55,576] WARN Error while fetching metadata
> > > > > [{TopicMetadata for topic test ->
> > > > > No partition metadata for topic test due to
> > > > > kafka.common.LeaderNotAvailableException}] for topic [test]: class
> > > > > kafka.common.LeaderNotAvailableException
> > > > > (kafka.producer.BrokerPartitionInfo)
> > > > > [2014-03-11 10:16:55,578] ERROR Failed to collate messages by
> topic,
> > > > > partition due to: Failed to fetch topic metadata for topic: test
> > > > > (kafka.producer.async.DefaultEventHandler)
> > > > > [2014-03-11 10:16:55,693] WARN Error while fetching metadata
> > > > > [{TopicMetadata for topic test ->
> > > > > No partition metadata for topic test due to
> > > > > kafka.common.LeaderNotAvailableException}] for topic [test]: class
> > > > > kafka.common.LeaderNotAvailableException
> > > > > (kafka.producer.BrokerPartitionInfo)
> > > > > [2014-03-11 10:16:55,706] WARN Error while fetching metadata
> > > > > [{TopicMetadata for topic test ->
> > > > > No partition metadata for topic test due to
> > > > > kafka.common.LeaderNotAvailableException}] for topic [test]: class
> > > > > kafka.common.LeaderNotAvailableException
> > > > > (kafka.producer.BrokerPartitionInfo)
> > > > > [2014-03-11 10:16:55,706] ERROR Failed to collate messages by
> topic,
> > > > > partition due to: Failed to fetch topic metadata for topic: test
> > > > > (kafka.producer.async.DefaultEventHandler)
> > > > > [2014-03-11 10:16:55,815] WARN Error while fetching metadata
> > > > > [{TopicMetadata for topic test ->
> > > > > No partition metadata for topic test due to
> > > > > kafka.common.LeaderNotAvailableException}] for topic [test]: class
> > > > > kafka.common.LeaderNotAvailableException
> > > > > (kafka.producer.BrokerPartitionInfo)
> > > > > [2014-03-11 10:16:55,823] WARN Error while fetching metadata
> > > > > [{TopicMetadata for topic test ->
> > > > > No partition metadata for topic test due to
> > > > > kafka.common.LeaderNotAvailableException}] for topic [test]: class
> > > > > kafka.common.LeaderNotAvailableException
> > > > > (kafka.producer.BrokerPartitionInfo)
> > > > > [2014-03-11 10:16:55,823] ERROR Failed to collate messages by
> topic,
> > > > > partition due to: Failed to fetch topic metadata for topic: test
> > > > > (kafka.producer.async.DefaultEventHandler)
> > > > > [2014-03-11 10:16:55,934] WARN Error while fetching metadata
> > > > > [{TopicMetadata for topic test ->
> > > > > No partition metadata for topic test due to
> > > > > kafka.common.LeaderNotAvailableException}] for topic [test]: class
> > > > > kafka.common.LeaderNotAvailableException
> > > > > (kafka.producer.BrokerPartitionInfo)
> > > > > [2014-03-11 10:16:55,949] WARN Error while fetching metadata
> > > > > [{TopicMetadata for topic test ->
> > > > > No partition metadata for topic test due to
> > > > > kafka.common.LeaderNotAvailableException}] for topic [test]: class
> > > > > kafka.common.LeaderNotAvailableException
> > > > > (kafka.producer.BrokerPartitionInfo)
> > > > > [2014-03-11 10:16:55,949] ERROR Failed to collate messages by
> topic,
> > > > > partition due to: Failed to fetch topic metadata for topic: test
> > > > > (kafka.producer.async.DefaultEventHandler)
> > > > > [2014-03-11 10:16:56,057] WARN Error while fetching metadata
> > > > > [{TopicMetadata for topic test ->
> > > > > No partition metadata for topic test due to<
>
>
RE: Remote Zookeeper
Posted by A A <an...@hotmail.com>.
Okay thanks. I removed zookeeper and kafka logs
/tmp/kafka-logs
/tmp/logs
/var/zookeeper/version-2/
/tmp/zookeeper*.log (on Zk)
restarted zookeeper
$KAFKA_HOME/bin/kafka-list-topic.sh --zookeeper 192.168.1.120:2181
no topics exist!
Started Broker 2
$KAFKA/kafka-server-start.sh $KAFKA_CONFIG/server.properties
[2014-03-11 12:05:32,651] INFO Verifying properties (kafka.utils.VerifiableProperties)
[2014-03-11 12:05:32,681] INFO Property port is overridden to 9092 (kafka.utils.VerifiableProperties)
[2014-03-11 12:05:32,681] INFO Property socket.request.max.bytes is overridden to 104857600 (kafka.utils.VerifiableProperties)
[2014-03-11 12:05:32,682] INFO Property num.io.threads is overridden to 2 (kafka.utils.VerifiableProperties)
[2014-03-11 12:05:32,682] INFO Property log.dirs is overridden to /tmp/kafka-logs (kafka.utils.VerifiableProperties)
[2014-03-11 12:05:32,682] INFO Property log.cleanup.interval.mins is overridden to 1 (kafka.utils.VerifiableProperties)
[2014-03-11 12:05:32,682] INFO Property socket.send.buffer.bytes is overridden to 1048576 (kafka.utils.VerifiableProperties)
[2014-03-11 12:05:32,683] INFO Property log.flush.interval.ms is overridden to 1000 (kafka.utils.VerifiableProperties)
[2014-03-11 12:05:32,683] INFO Property zookeeper.connect is overridden to 192.168.1.120:2181 (kafka.utils.VerifiableProperties)
[2014-03-11 12:05:32,684] INFO Property broker.id is overridden to 2 (kafka.utils.VerifiableProperties)
[2014-03-11 12:05:32,684] INFO Property log.retention.hours is overridden to 168 (kafka.utils.VerifiableProperties)
[2014-03-11 12:05:32,684] INFO Property num.network.threads is overridden to 2 (kafka.utils.VerifiableProperties)
[2014-03-11 12:05:32,684] INFO Property socket.receive.buffer.bytes is overridden to 1048576 (kafka.utils.VerifiableProperties)
[2014-03-11 12:05:32,684] INFO Property zookeeper.connection.timeout.ms is overridden to 1000000 (kafka.utils.VerifiableProperties)
[2014-03-11 12:05:32,685] INFO Property num.partitions is overridden to 2 (kafka.utils.VerifiableProperties)
[2014-03-11 12:05:32,685] INFO Property log.flush.interval.messages is overridden to 10000 (kafka.utils.VerifiableProperties)
[2014-03-11 12:05:32,685] INFO Property log.segment.bytes is overridden to 536870912 (kafka.utils.VerifiableProperties)
[2014-03-11 12:05:32,703] INFO [Kafka Server 2], Starting (kafka.server.KafkaServer)
[2014-03-11 12:05:32,739] INFO [Log Manager on Broker 2] Starting log cleaner every 60000 ms (kafka.log.LogManager)
[2014-03-11 12:05:32,753] INFO [Log Manager on Broker 2] Starting log flusher every 3000 ms with the following overrides Map() (kafka.log.LogManager)
[2014-03-11 12:05:32,791] INFO Awaiting socket connections on 0.0.0.0:9092. (kafka.network.Acceptor)
[2014-03-11 12:05:32,794] INFO [Socket Server on Broker 2], Started (kafka.network.SocketServer)
[2014-03-11 12:05:32,801] INFO connecting to ZK: 192.168.1.120:2181 (kafka.server.KafkaZooKeeper)
[2014-03-11 12:05:32,815] INFO Starting ZkClient event thread. (org.I0Itec.zkclient.ZkEventThread)
[2014-03-11 12:05:32,817] INFO Client environment:zookeeper.version=3.3.3-1203054, built on 11/17/2011 05:47 GMT (org.apache.zookeeper.ZooKeeper)
[2014-03-11 12:05:32,817] INFO Client environment:host.name=localhost (org.apache.zookeeper.ZooKeeper)
[2014-03-11 12:05:32,817] INFO Client environment:java.version=1.7.0_51 (org.apache.zookeeper.ZooKeeper)
[2014-03-11 12:05:32,822] INFO Client environment:java.vendor=Oracle Corporation (org.apache.zookeeper.ZooKeeper)
[2014-03-11 12:05:32,822] INFO Client environment:java.home=/usr/lib/jvm/java-1.7.0-openjdk-1.7.0.51.x86_64/jre (org.apache.zookeeper.ZooKeeper)
[2014-03-11 12:05:32,822] INFO Client environment:java.class.path=:/opt/kafka-0.8.0/bin/../core/target/scala-2.8.0/kafka_2.8.0-0.8.0.jar:/opt/kafka-0.8.0/bin/../core/target/scala-2.8.0/kafka-assembly-0.8.0-deps.jar:/opt/kafka-0.8.0/bin/../perf/target/scala-2.8.0/kafka-perf_2.8.0-0.8.0.jar:/opt/kafka-0.8.0/bin/../libs/*.jar:/opt/kafka-0.8.0/bin/../kafka*.jar (org.apache.zookeeper.ZooKeeper)
[2014-03-11 12:05:32,822] INFO Client environment:java.library.path=/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib (org.apache.zookeeper.ZooKeeper)
[2014-03-11 12:05:32,822] INFO Client environment:java.io.tmpdir=/tmp (org.apache.zookeeper.ZooKeeper)
[2014-03-11 12:05:32,822] INFO Client environment:java.compiler=<NA> (org.apache.zookeeper.ZooKeeper)
[2014-03-11 12:05:32,822] INFO Client environment:os.name=Linux (org.apache.zookeeper.ZooKeeper)
[2014-03-11 12:05:32,822] INFO Client environment:os.arch=amd64 (org.apache.zookeeper.ZooKeeper)
[2014-03-11 12:05:32,822] INFO Client environment:os.version=2.6.32-279.el6.x86_64 (org.apache.zookeeper.ZooKeeper)
[2014-03-11 12:05:32,822] INFO Client environment:user.name=root (org.apache.zookeeper.ZooKeeper)
[2014-03-11 12:05:32,822] INFO Client environment:user.home=/root (org.apache.zookeeper.ZooKeeper)
[2014-03-11 12:05:32,822] INFO Client environment:user.dir=/opt (org.apache.zookeeper.ZooKeeper)
[2014-03-11 12:05:32,825] INFO Initiating client connection, connectString=192.168.1.120:2181 sessionTimeout=6000 watcher=org.I0Itec.zkclient.ZkClient@721e34d8 (org.apache.zookeeper.ZooKeeper)
[2014-03-11 12:05:32,842] INFO Opening socket connection to server /192.168.1.120:2181 (org.apache.zookeeper.ClientCnxn)
[2014-03-11 12:05:32,846] INFO Socket connection established to 192.168.1.120/192.168.1.120:2181, initiating session (org.apache.zookeeper.ClientCnxn)
[2014-03-11 12:05:32,854] INFO Session establishment complete on server 192.168.1.120/192.168.1.120:2181, sessionid = 0x144b104e0210004, negotiated timeout = 6000 (org.apache.zookeeper.ClientCnxn)
[2014-03-11 12:05:32,856] INFO zookeeper state changed (SyncConnected) (org.I0Itec.zkclient.ZkClient)
[2014-03-11 12:05:32,891] INFO Registered broker 2 at path /brokers/ids/2 with address localhost:9092. (kafka.utils.ZkUtils$)
[2014-03-11 12:05:32,892] INFO [Kafka Server 2], Connecting to ZK: 192.168.1.120:2181 (kafka.server.KafkaServer)
[2014-03-11 12:05:33,031] INFO Will not load MX4J, mx4j-tools.jar is not in the classpath (kafka.utils.Mx4jLoader$)
[2014-03-11 12:05:33,041] INFO 2 successfully elected as leader (kafka.server.ZookeeperLeaderElector)
[2014-03-11 12:05:33,371] INFO [Kafka Server 2], Started (kafka.server.KafkaServer)
[2014-03-11 12:05:33,377] INFO New leader is 2 (kafka.server.ZookeeperLeaderElector$LeaderChangeListener)
Then started Broker 1
$KAFKA/kafka-server-start.sh $KAFKA_CONFIG/server.properties
[2014-03-11 12:07:20,843] INFO Verifying properties (kafka.utils.VerifiableProperties)
[2014-03-11 12:07:20,875] INFO Property port is overridden to 9092 (kafka.utils.VerifiableProperties)
[2014-03-11 12:07:20,876] INFO Property socket.request.max.bytes is overridden to 104857600 (kafka.utils.VerifiableProperties)
[2014-03-11 12:07:20,876] INFO Property num.io.threads is overridden to 2 (kafka.utils.VerifiableProperties)
[2014-03-11 12:07:20,876] INFO Property log.dirs is overridden to /tmp/kafka-logs (kafka.utils.VerifiableProperties)
[2014-03-11 12:07:20,876] INFO Property log.cleanup.interval.mins is overridden to 1 (kafka.utils.VerifiableProperties)
[2014-03-11 12:07:20,877] INFO Property socket.send.buffer.bytes is overridden to 1048576 (kafka.utils.VerifiableProperties)
[2014-03-11 12:07:20,877] INFO Property log.flush.interval.ms is overridden to 1000 (kafka.utils.VerifiableProperties)
[2014-03-11 12:07:20,877] INFO Property zookeeper.connect is overridden to 192.168.1.120:2181 (kafka.utils.VerifiableProperties)
[2014-03-11 12:07:20,877] INFO Property broker.id is overridden to 1 (kafka.utils.VerifiableProperties)
[2014-03-11 12:07:20,877] INFO Property log.retention.hours is overridden to 168 (kafka.utils.VerifiableProperties)
[2014-03-11 12:07:20,878] INFO Property num.network.threads is overridden to 2 (kafka.utils.VerifiableProperties)
[2014-03-11 12:07:20,878] INFO Property socket.receive.buffer.bytes is overridden to 1048576 (kafka.utils.VerifiableProperties)
[2014-03-11 12:07:20,878] INFO Property zookeeper.connection.timeout.ms is overridden to 1000000 (kafka.utils.VerifiableProperties)
[2014-03-11 12:07:20,878] INFO Property num.partitions is overridden to 2 (kafka.utils.VerifiableProperties)
[2014-03-11 12:07:20,879] INFO Property log.flush.interval.messages is overridden to 10000 (kafka.utils.VerifiableProperties)
[2014-03-11 12:07:20,879] INFO Property log.segment.bytes is overridden to 536870912 (kafka.utils.VerifiableProperties)
[2014-03-11 12:07:20,897] INFO [Kafka Server 1], Starting (kafka.server.KafkaServer)
[2014-03-11 12:07:20,922] INFO [Log Manager on Broker 1] Log directory '/tmp/kafka-logs' not found, creating it. (kafka.log.LogManager)
[2014-03-11 12:07:20,938] INFO [Log Manager on Broker 1] Starting log cleaner every 60000 ms (kafka.log.LogManager)
[2014-03-11 12:07:20,951] INFO [Log Manager on Broker 1] Starting log flusher every 3000 ms with the following overrides Map() (kafka.log.LogManager)
[2014-03-11 12:07:20,991] INFO Awaiting socket connections on 0.0.0.0:9092. (kafka.network.Acceptor)
[2014-03-11 12:07:20,993] INFO [Socket Server on Broker 1], Started (kafka.network.SocketServer)
[2014-03-11 12:07:21,002] INFO connecting to ZK: 192.168.1.120:2181 (kafka.server.KafkaZooKeeper)
[2014-03-11 12:07:21,018] INFO Starting ZkClient event thread. (org.I0Itec.zkclient.ZkEventThread)
[2014-03-11 12:07:21,029] INFO Client environment:zookeeper.version=3.3.3-1203054, built on 11/17/2011 05:47 GMT (org.apache.zookeeper.ZooKeeper)
[2014-03-11 12:07:21,029] INFO Client environment:host.name=localhost (org.apache.zookeeper.ZooKeeper)
[2014-03-11 12:07:21,029] INFO Client environment:java.version=1.7.0_51 (org.apache.zookeeper.ZooKeeper)
[2014-03-11 12:07:21,029] INFO Client environment:java.vendor=Oracle Corporation (org.apache.zookeeper.ZooKeeper)
[2014-03-11 12:07:21,029] INFO Client environment:java.home=/usr/lib/jvm/java-1.7.0-openjdk-1.7.0.51.x86_64/jre (org.apache.zookeeper.ZooKeeper)
[2014-03-11 12:07:21,029] INFO Client environment:java.class.path=:/opt/kafka-0.8.0/bin/../core/target/scala-2.8.0/kafka_2.8.0-0.8.0.jar:/opt/kafka-0.8.0/bin/../core/target/scala-2.8.0/kafka-assembly-0.8.0-deps.jar:/opt/kafka-0.8.0/bin/../perf/target/scala-2.8.0/kafka-perf_2.8.0-0.8.0.jar:/opt/kafka-0.8.0/bin/../libs/*.jar:/opt/kafka-0.8.0/bin/../kafka*.jar (org.apache.zookeeper.ZooKeeper)
[2014-03-11 12:07:21,029] INFO Client environment:java.library.path=/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib (org.apache.zookeeper.ZooKeeper)
[2014-03-11 12:07:21,029] INFO Client environment:java.io.tmpdir=/tmp (org.apache.zookeeper.ZooKeeper)
[2014-03-11 12:07:21,030] INFO Client environment:java.compiler=<NA> (org.apache.zookeeper.ZooKeeper)
[2014-03-11 12:07:21,030] INFO Client environment:os.name=Linux (org.apache.zookeeper.ZooKeeper)
[2014-03-11 12:07:21,030] INFO Client environment:os.arch=amd64 (org.apache.zookeeper.ZooKeeper)
[2014-03-11 12:07:21,030] INFO Client environment:os.version=2.6.32-279.el6.x86_64 (org.apache.zookeeper.ZooKeeper)
[2014-03-11 12:07:21,030] INFO Client environment:user.name=root (org.apache.zookeeper.ZooKeeper)
[2014-03-11 12:07:21,030] INFO Client environment:user.home=/root (org.apache.zookeeper.ZooKeeper)
[2014-03-11 12:07:21,030] INFO Client environment:user.dir=/var/zookeeper (org.apache.zookeeper.ZooKeeper)
[2014-03-11 12:07:21,031] INFO Initiating client connection, connectString=192.168.1.120:2181 sessionTimeout=6000 watcher=org.I0Itec.zkclient.ZkClient@45419cee (org.apache.zookeeper.ZooKeeper)
[2014-03-11 12:07:21,043] INFO Opening socket connection to server /192.168.1.120:2181 (org.apache.zookeeper.ClientCnxn)
[2014-03-11 12:07:21,049] INFO Socket connection established to 192.168.1.120/192.168.1.120:2181, initiating session (org.apache.zookeeper.ClientCnxn)
[2014-03-11 12:07:21,058] INFO Session establishment complete on server 192.168.1.120/192.168.1.120:2181, sessionid = 0x144b104e0210005, negotiated timeout = 6000 (org.apache.zookeeper.ClientCnxn)
[2014-03-11 12:07:21,060] INFO zookeeper state changed (SyncConnected) (org.I0Itec.zkclient.ZkClient)
[2014-03-11 12:07:21,095] INFO Registered broker 1 at path /brokers/ids/1 with address localhost:9092. (kafka.utils.ZkUtils$)
[2014-03-11 12:07:21,096] INFO [Kafka Server 1], Connecting to ZK: 192.168.1.120:2181 (kafka.server.KafkaServer)
[2014-03-11 12:07:21,222] INFO Will not load MX4J, mx4j-tools.jar is not in the classpath (kafka.utils.Mx4jLoader$)
[2014-03-11 12:07:21,249] INFO conflict in /controller data: { "brokerid":1, "timestamp":"1394539641229", "version":1 } stored data: { "brokerid":2, "timestamp":"1394539533037", "version":1 } (kafka.utils.ZkUtils$)
[2014-03-11 12:07:21,469] INFO [Kafka Server 1], Started (kafka.server.KafkaServer)
Could this be an issue?
INFO Registered broker 2 at path /brokers/ids/2 with address localhost:9092.
INFO Registered broker 1 at path /brokers/ids/1 with address localhost:9092.
> Date: Tue, 11 Mar 2014 09:51:54 -0700
> Subject: Re: Remote Zookeeper
> From: b@b3k.us
> To: users@kafka.apache.org
>
> Seems you my have put your cluster in a very confused state with random
> addition and removal of brokers and topics. I'd suggest deleting the
> zookeeper and Kafka logs and starting over using the getting started
> tutorial from the wiki.
>
> On Tuesday, March 11, 2014, A A <an...@hotmail.com> wrote:
>
> > I noticed a problem with my topics. The previous two topics would only be
> > present on 1 broker. So I created a new topic
> >
> > $KAFKA_HOME/bin/kafka-create-topic.sh --zookeeper 192.168.1.120:2181--replica 2 --partition 1 --topic test-rep
> > $KAFKA_HOME/bin/kafka-list-topic.sh --zookeeper 192.168.1.120:2181
> > topic: test partition: 0 leader: 1 replicas: 1 isr: 1
> > topic: test-rep partition: 0 leader: 1 replicas: 1,2 isr: 1
> > topic: test1 partition: 0 leader: 1 replicas: 1 isr: 1
> >
> > No there is no firewall. Even then I stopped the iptables. And then I
> > tried the following
> > $KAFKA_HOME/bin/kafka-console-producer.sh --broker-list
> > 192.168.1.123:9092,192.168.1.124:9092 --topic test-rep
> > replicated message
> >
> > [2014-03-11 10:59:25,844] WARN Error while fetching metadata
> > [{TopicMetadata for topic test-rep ->
> > No partition metadata for topic test-rep due to
> > kafka.common.LeaderNotAvailableException}] for topic [test-rep]: class
> > kafka.common.LeaderNotAvailableException
> > (kafka.producer.BrokerPartitionInfo)
> > [2014-03-11 10:59:25,867] WARN Error while fetching metadata
> > [{TopicMetadata for topic test-rep ->
> > No partition metadata for topic test-rep due to
> > kafka.common.LeaderNotAvailableException}] for topic [test-rep]: class
> > kafka.common.LeaderNotAvailableException
> > (kafka.producer.BrokerPartitionInfo)
> > [2014-03-11 10:59:25,868] ERROR Failed to collate messages by topic,
> > partition due to: Failed to fetch topic metadata for topic: test-rep
> > (kafka.producer.async.DefaultEventHandler)
> > [2014-03-11 10:59:25,979] WARN Error while fetching metadata
> > [{TopicMetadata for topic test-rep ->
> > No partition metadata for topic test-rep due to
> > kafka.common.LeaderNotAvailableException}] for topic [test-rep]: class
> > kafka.common.LeaderNotAvailableException
> > (kafka.producer.BrokerPartitionInfo)
> > [2014-03-11 10:59:25,986] WARN Error while fetching metadata
> > [{TopicMetadata for topic test-rep ->
> > No partition metadata for topic test-rep due to
> > kafka.common.LeaderNotAvailableException}] for topic [test-rep]: class
> > kafka.common.LeaderNotAvailableException
> > (kafka.producer.BrokerPartitionInfo)
> > [2014-03-11 10:59:25,987] ERROR Failed to collate messages by topic,
> > partition due to: Failed to fetch topic metadata for topic: test-rep
> > (kafka.producer.async.DefaultEventHandler)
> > [2014-03-11 10:59:26,098] WARN Error while fetching metadata
> > [{TopicMetadata for topic test-rep ->
> > No partition metadata for topic test-rep due to
> > kafka.common.LeaderNotAvailableException}] for topic [test-rep]: class
> > kafka.common.LeaderNotAvailableException
> > (kafka.producer.BrokerPartitionInfo)
> >
> >
> >
> >
> > > Date: Tue, 11 Mar 2014 09:05:54 -0700
> > > Subject: Re: Remote Zookeeper
> > > From: pradeepg26@gmail.com <javascript:;>
> > > To: users@kafka.apache.org <javascript:;>
> > >
> > > Is there a firewall thats blocking connections on port 9092? Also, the
> > > broker list should be comma separated.
> > >
> > >
> > > On Tue, Mar 11, 2014 at 9:02 AM, A A <an...@hotmail.com> wrote:
> > >
> > > > Sorry one of the brokers for was down. Brought it back up. Tried the
> > > > following
> > > >
> > > > $KAFKA_HOME/bin/kafka-console-producer.sh --broker-list localhost:9092
> > > > 192.168.1.124:9092 --topic test
> > > > hello brokers
> > > >
> > > > [2014-03-11 10:16:55,547] WARN Error while fetching metadata
> > > > [{TopicMetadata for topic test ->
> > > > No partition metadata for topic test due to
> > > > kafka.common.LeaderNotAvailableException}] for topic [test]: class
> > > > kafka.common.LeaderNotAvailableException
> > > > (kafka.producer.BrokerPartitionInfo)
> > > > [2014-03-11 10:16:55,576] WARN Error while fetching metadata
> > > > [{TopicMetadata for topic test ->
> > > > No partition metadata for topic test due to
> > > > kafka.common.LeaderNotAvailableException}] for topic [test]: class
> > > > kafka.common.LeaderNotAvailableException
> > > > (kafka.producer.BrokerPartitionInfo)
> > > > [2014-03-11 10:16:55,578] ERROR Failed to collate messages by topic,
> > > > partition due to: Failed to fetch topic metadata for topic: test
> > > > (kafka.producer.async.DefaultEventHandler)
> > > > [2014-03-11 10:16:55,693] WARN Error while fetching metadata
> > > > [{TopicMetadata for topic test ->
> > > > No partition metadata for topic test due to
> > > > kafka.common.LeaderNotAvailableException}] for topic [test]: class
> > > > kafka.common.LeaderNotAvailableException
> > > > (kafka.producer.BrokerPartitionInfo)
> > > > [2014-03-11 10:16:55,706] WARN Error while fetching metadata
> > > > [{TopicMetadata for topic test ->
> > > > No partition metadata for topic test due to
> > > > kafka.common.LeaderNotAvailableException}] for topic [test]: class
> > > > kafka.common.LeaderNotAvailableException
> > > > (kafka.producer.BrokerPartitionInfo)
> > > > [2014-03-11 10:16:55,706] ERROR Failed to collate messages by topic,
> > > > partition due to: Failed to fetch topic metadata for topic: test
> > > > (kafka.producer.async.DefaultEventHandler)
> > > > [2014-03-11 10:16:55,815] WARN Error while fetching metadata
> > > > [{TopicMetadata for topic test ->
> > > > No partition metadata for topic test due to
> > > > kafka.common.LeaderNotAvailableException}] for topic [test]: class
> > > > kafka.common.LeaderNotAvailableException
> > > > (kafka.producer.BrokerPartitionInfo)
> > > > [2014-03-11 10:16:55,823] WARN Error while fetching metadata
> > > > [{TopicMetadata for topic test ->
> > > > No partition metadata for topic test due to
> > > > kafka.common.LeaderNotAvailableException}] for topic [test]: class
> > > > kafka.common.LeaderNotAvailableException
> > > > (kafka.producer.BrokerPartitionInfo)
> > > > [2014-03-11 10:16:55,823] ERROR Failed to collate messages by topic,
> > > > partition due to: Failed to fetch topic metadata for topic: test
> > > > (kafka.producer.async.DefaultEventHandler)
> > > > [2014-03-11 10:16:55,934] WARN Error while fetching metadata
> > > > [{TopicMetadata for topic test ->
> > > > No partition metadata for topic test due to
> > > > kafka.common.LeaderNotAvailableException}] for topic [test]: class
> > > > kafka.common.LeaderNotAvailableException
> > > > (kafka.producer.BrokerPartitionInfo)
> > > > [2014-03-11 10:16:55,949] WARN Error while fetching metadata
> > > > [{TopicMetadata for topic test ->
> > > > No partition metadata for topic test due to
> > > > kafka.common.LeaderNotAvailableException}] for topic [test]: class
> > > > kafka.common.LeaderNotAvailableException
> > > > (kafka.producer.BrokerPartitionInfo)
> > > > [2014-03-11 10:16:55,949] ERROR Failed to collate messages by topic,
> > > > partition due to: Failed to fetch topic metadata for topic: test
> > > > (kafka.producer.async.DefaultEventHandler)
> > > > [2014-03-11 10:16:56,057] WARN Error while fetching metadata
> > > > [{TopicMetadata for topic test ->
> > > > No partition metadata for topic test due to<
Re: Remote Zookeeper
Posted by Benjamin Black <b...@b3k.us>.
Seems you my have put your cluster in a very confused state with random
addition and removal of brokers and topics. I'd suggest deleting the
zookeeper and Kafka logs and starting over using the getting started
tutorial from the wiki.
On Tuesday, March 11, 2014, A A <an...@hotmail.com> wrote:
> I noticed a problem with my topics. The previous two topics would only be
> present on 1 broker. So I created a new topic
>
> $KAFKA_HOME/bin/kafka-create-topic.sh --zookeeper 192.168.1.120:2181--replica 2 --partition 1 --topic test-rep
> $KAFKA_HOME/bin/kafka-list-topic.sh --zookeeper 192.168.1.120:2181
> topic: test partition: 0 leader: 1 replicas: 1 isr: 1
> topic: test-rep partition: 0 leader: 1 replicas: 1,2 isr: 1
> topic: test1 partition: 0 leader: 1 replicas: 1 isr: 1
>
> No there is no firewall. Even then I stopped the iptables. And then I
> tried the following
> $KAFKA_HOME/bin/kafka-console-producer.sh --broker-list
> 192.168.1.123:9092,192.168.1.124:9092 --topic test-rep
> replicated message
>
> [2014-03-11 10:59:25,844] WARN Error while fetching metadata
> [{TopicMetadata for topic test-rep ->
> No partition metadata for topic test-rep due to
> kafka.common.LeaderNotAvailableException}] for topic [test-rep]: class
> kafka.common.LeaderNotAvailableException
> (kafka.producer.BrokerPartitionInfo)
> [2014-03-11 10:59:25,867] WARN Error while fetching metadata
> [{TopicMetadata for topic test-rep ->
> No partition metadata for topic test-rep due to
> kafka.common.LeaderNotAvailableException}] for topic [test-rep]: class
> kafka.common.LeaderNotAvailableException
> (kafka.producer.BrokerPartitionInfo)
> [2014-03-11 10:59:25,868] ERROR Failed to collate messages by topic,
> partition due to: Failed to fetch topic metadata for topic: test-rep
> (kafka.producer.async.DefaultEventHandler)
> [2014-03-11 10:59:25,979] WARN Error while fetching metadata
> [{TopicMetadata for topic test-rep ->
> No partition metadata for topic test-rep due to
> kafka.common.LeaderNotAvailableException}] for topic [test-rep]: class
> kafka.common.LeaderNotAvailableException
> (kafka.producer.BrokerPartitionInfo)
> [2014-03-11 10:59:25,986] WARN Error while fetching metadata
> [{TopicMetadata for topic test-rep ->
> No partition metadata for topic test-rep due to
> kafka.common.LeaderNotAvailableException}] for topic [test-rep]: class
> kafka.common.LeaderNotAvailableException
> (kafka.producer.BrokerPartitionInfo)
> [2014-03-11 10:59:25,987] ERROR Failed to collate messages by topic,
> partition due to: Failed to fetch topic metadata for topic: test-rep
> (kafka.producer.async.DefaultEventHandler)
> [2014-03-11 10:59:26,098] WARN Error while fetching metadata
> [{TopicMetadata for topic test-rep ->
> No partition metadata for topic test-rep due to
> kafka.common.LeaderNotAvailableException}] for topic [test-rep]: class
> kafka.common.LeaderNotAvailableException
> (kafka.producer.BrokerPartitionInfo)
>
>
>
>
> > Date: Tue, 11 Mar 2014 09:05:54 -0700
> > Subject: Re: Remote Zookeeper
> > From: pradeepg26@gmail.com <javascript:;>
> > To: users@kafka.apache.org <javascript:;>
> >
> > Is there a firewall thats blocking connections on port 9092? Also, the
> > broker list should be comma separated.
> >
> >
> > On Tue, Mar 11, 2014 at 9:02 AM, A A <an...@hotmail.com> wrote:
> >
> > > Sorry one of the brokers for was down. Brought it back up. Tried the
> > > following
> > >
> > > $KAFKA_HOME/bin/kafka-console-producer.sh --broker-list localhost:9092
> > > 192.168.1.124:9092 --topic test
> > > hello brokers
> > >
> > > [2014-03-11 10:16:55,547] WARN Error while fetching metadata
> > > [{TopicMetadata for topic test ->
> > > No partition metadata for topic test due to
> > > kafka.common.LeaderNotAvailableException}] for topic [test]: class
> > > kafka.common.LeaderNotAvailableException
> > > (kafka.producer.BrokerPartitionInfo)
> > > [2014-03-11 10:16:55,576] WARN Error while fetching metadata
> > > [{TopicMetadata for topic test ->
> > > No partition metadata for topic test due to
> > > kafka.common.LeaderNotAvailableException}] for topic [test]: class
> > > kafka.common.LeaderNotAvailableException
> > > (kafka.producer.BrokerPartitionInfo)
> > > [2014-03-11 10:16:55,578] ERROR Failed to collate messages by topic,
> > > partition due to: Failed to fetch topic metadata for topic: test
> > > (kafka.producer.async.DefaultEventHandler)
> > > [2014-03-11 10:16:55,693] WARN Error while fetching metadata
> > > [{TopicMetadata for topic test ->
> > > No partition metadata for topic test due to
> > > kafka.common.LeaderNotAvailableException}] for topic [test]: class
> > > kafka.common.LeaderNotAvailableException
> > > (kafka.producer.BrokerPartitionInfo)
> > > [2014-03-11 10:16:55,706] WARN Error while fetching metadata
> > > [{TopicMetadata for topic test ->
> > > No partition metadata for topic test due to
> > > kafka.common.LeaderNotAvailableException}] for topic [test]: class
> > > kafka.common.LeaderNotAvailableException
> > > (kafka.producer.BrokerPartitionInfo)
> > > [2014-03-11 10:16:55,706] ERROR Failed to collate messages by topic,
> > > partition due to: Failed to fetch topic metadata for topic: test
> > > (kafka.producer.async.DefaultEventHandler)
> > > [2014-03-11 10:16:55,815] WARN Error while fetching metadata
> > > [{TopicMetadata for topic test ->
> > > No partition metadata for topic test due to
> > > kafka.common.LeaderNotAvailableException}] for topic [test]: class
> > > kafka.common.LeaderNotAvailableException
> > > (kafka.producer.BrokerPartitionInfo)
> > > [2014-03-11 10:16:55,823] WARN Error while fetching metadata
> > > [{TopicMetadata for topic test ->
> > > No partition metadata for topic test due to
> > > kafka.common.LeaderNotAvailableException}] for topic [test]: class
> > > kafka.common.LeaderNotAvailableException
> > > (kafka.producer.BrokerPartitionInfo)
> > > [2014-03-11 10:16:55,823] ERROR Failed to collate messages by topic,
> > > partition due to: Failed to fetch topic metadata for topic: test
> > > (kafka.producer.async.DefaultEventHandler)
> > > [2014-03-11 10:16:55,934] WARN Error while fetching metadata
> > > [{TopicMetadata for topic test ->
> > > No partition metadata for topic test due to
> > > kafka.common.LeaderNotAvailableException}] for topic [test]: class
> > > kafka.common.LeaderNotAvailableException
> > > (kafka.producer.BrokerPartitionInfo)
> > > [2014-03-11 10:16:55,949] WARN Error while fetching metadata
> > > [{TopicMetadata for topic test ->
> > > No partition metadata for topic test due to
> > > kafka.common.LeaderNotAvailableException}] for topic [test]: class
> > > kafka.common.LeaderNotAvailableException
> > > (kafka.producer.BrokerPartitionInfo)
> > > [2014-03-11 10:16:55,949] ERROR Failed to collate messages by topic,
> > > partition due to: Failed to fetch topic metadata for topic: test
> > > (kafka.producer.async.DefaultEventHandler)
> > > [2014-03-11 10:16:56,057] WARN Error while fetching metadata
> > > [{TopicMetadata for topic test ->
> > > No partition metadata for topic test due to<
RE: Remote Zookeeper
Posted by A A <an...@hotmail.com>.
I noticed a problem with my topics. The previous two topics would only be present on 1 broker. So I created a new topic
$KAFKA_HOME/bin/kafka-create-topic.sh --zookeeper 192.168.1.120:2181 --replica 2 --partition 1 --topic test-rep
$KAFKA_HOME/bin/kafka-list-topic.sh --zookeeper 192.168.1.120:2181
topic: test partition: 0 leader: 1 replicas: 1 isr: 1
topic: test-rep partition: 0 leader: 1 replicas: 1,2 isr: 1
topic: test1 partition: 0 leader: 1 replicas: 1 isr: 1
No there is no firewall. Even then I stopped the iptables. And then I tried the following
$KAFKA_HOME/bin/kafka-console-producer.sh --broker-list 192.168.1.123:9092,192.168.1.124:9092 --topic test-rep
replicated message
[2014-03-11 10:59:25,844] WARN Error while fetching metadata [{TopicMetadata for topic test-rep ->
No partition metadata for topic test-rep due to kafka.common.LeaderNotAvailableException}] for topic [test-rep]: class kafka.common.LeaderNotAvailableException (kafka.producer.BrokerPartitionInfo)
[2014-03-11 10:59:25,867] WARN Error while fetching metadata [{TopicMetadata for topic test-rep ->
No partition metadata for topic test-rep due to kafka.common.LeaderNotAvailableException}] for topic [test-rep]: class kafka.common.LeaderNotAvailableException (kafka.producer.BrokerPartitionInfo)
[2014-03-11 10:59:25,868] ERROR Failed to collate messages by topic, partition due to: Failed to fetch topic metadata for topic: test-rep (kafka.producer.async.DefaultEventHandler)
[2014-03-11 10:59:25,979] WARN Error while fetching metadata [{TopicMetadata for topic test-rep ->
No partition metadata for topic test-rep due to kafka.common.LeaderNotAvailableException}] for topic [test-rep]: class kafka.common.LeaderNotAvailableException (kafka.producer.BrokerPartitionInfo)
[2014-03-11 10:59:25,986] WARN Error while fetching metadata [{TopicMetadata for topic test-rep ->
No partition metadata for topic test-rep due to kafka.common.LeaderNotAvailableException}] for topic [test-rep]: class kafka.common.LeaderNotAvailableException (kafka.producer.BrokerPartitionInfo)
[2014-03-11 10:59:25,987] ERROR Failed to collate messages by topic, partition due to: Failed to fetch topic metadata for topic: test-rep (kafka.producer.async.DefaultEventHandler)
[2014-03-11 10:59:26,098] WARN Error while fetching metadata [{TopicMetadata for topic test-rep ->
No partition metadata for topic test-rep due to kafka.common.LeaderNotAvailableException}] for topic [test-rep]: class kafka.common.LeaderNotAvailableException (kafka.producer.BrokerPartitionInfo)
> Date: Tue, 11 Mar 2014 09:05:54 -0700
> Subject: Re: Remote Zookeeper
> From: pradeepg26@gmail.com
> To: users@kafka.apache.org
>
> Is there a firewall thats blocking connections on port 9092? Also, the
> broker list should be comma separated.
>
>
> On Tue, Mar 11, 2014 at 9:02 AM, A A <an...@hotmail.com> wrote:
>
> > Sorry one of the brokers for was down. Brought it back up. Tried the
> > following
> >
> > $KAFKA_HOME/bin/kafka-console-producer.sh --broker-list localhost:9092
> > 192.168.1.124:9092 --topic test
> > hello brokers
> >
> > [2014-03-11 10:16:55,547] WARN Error while fetching metadata
> > [{TopicMetadata for topic test ->
> > No partition metadata for topic test due to
> > kafka.common.LeaderNotAvailableException}] for topic [test]: class
> > kafka.common.LeaderNotAvailableException
> > (kafka.producer.BrokerPartitionInfo)
> > [2014-03-11 10:16:55,576] WARN Error while fetching metadata
> > [{TopicMetadata for topic test ->
> > No partition metadata for topic test due to
> > kafka.common.LeaderNotAvailableException}] for topic [test]: class
> > kafka.common.LeaderNotAvailableException
> > (kafka.producer.BrokerPartitionInfo)
> > [2014-03-11 10:16:55,578] ERROR Failed to collate messages by topic,
> > partition due to: Failed to fetch topic metadata for topic: test
> > (kafka.producer.async.DefaultEventHandler)
> > [2014-03-11 10:16:55,693] WARN Error while fetching metadata
> > [{TopicMetadata for topic test ->
> > No partition metadata for topic test due to
> > kafka.common.LeaderNotAvailableException}] for topic [test]: class
> > kafka.common.LeaderNotAvailableException
> > (kafka.producer.BrokerPartitionInfo)
> > [2014-03-11 10:16:55,706] WARN Error while fetching metadata
> > [{TopicMetadata for topic test ->
> > No partition metadata for topic test due to
> > kafka.common.LeaderNotAvailableException}] for topic [test]: class
> > kafka.common.LeaderNotAvailableException
> > (kafka.producer.BrokerPartitionInfo)
> > [2014-03-11 10:16:55,706] ERROR Failed to collate messages by topic,
> > partition due to: Failed to fetch topic metadata for topic: test
> > (kafka.producer.async.DefaultEventHandler)
> > [2014-03-11 10:16:55,815] WARN Error while fetching metadata
> > [{TopicMetadata for topic test ->
> > No partition metadata for topic test due to
> > kafka.common.LeaderNotAvailableException}] for topic [test]: class
> > kafka.common.LeaderNotAvailableException
> > (kafka.producer.BrokerPartitionInfo)
> > [2014-03-11 10:16:55,823] WARN Error while fetching metadata
> > [{TopicMetadata for topic test ->
> > No partition metadata for topic test due to
> > kafka.common.LeaderNotAvailableException}] for topic [test]: class
> > kafka.common.LeaderNotAvailableException
> > (kafka.producer.BrokerPartitionInfo)
> > [2014-03-11 10:16:55,823] ERROR Failed to collate messages by topic,
> > partition due to: Failed to fetch topic metadata for topic: test
> > (kafka.producer.async.DefaultEventHandler)
> > [2014-03-11 10:16:55,934] WARN Error while fetching metadata
> > [{TopicMetadata for topic test ->
> > No partition metadata for topic test due to
> > kafka.common.LeaderNotAvailableException}] for topic [test]: class
> > kafka.common.LeaderNotAvailableException
> > (kafka.producer.BrokerPartitionInfo)
> > [2014-03-11 10:16:55,949] WARN Error while fetching metadata
> > [{TopicMetadata for topic test ->
> > No partition metadata for topic test due to
> > kafka.common.LeaderNotAvailableException}] for topic [test]: class
> > kafka.common.LeaderNotAvailableException
> > (kafka.producer.BrokerPartitionInfo)
> > [2014-03-11 10:16:55,949] ERROR Failed to collate messages by topic,
> > partition due to: Failed to fetch topic metadata for topic: test
> > (kafka.producer.async.DefaultEventHandler)
> > [2014-03-11 10:16:56,057] WARN Error while fetching metadata
> > [{TopicMetadata for topic test ->
> > No partition metadata for topic test due to
> > kafka.common.LeaderNotAvailableException}] for topic [test]: class
> > kafka.common.LeaderNotAvailableException
> > (kafka.producer.BrokerPartitionInfo)
> > [2014-03-11 10:16:56,059] ERROR Failed to send requests for topics test
> > with correlation ids in [0,8] (kafka.producer.async.DefaultEventHandler)
> > [2014-03-11 10:16:56,061] ERROR Error in handling batch of 1 events
> > (kafka.producer.async.ProducerSendThread)
> > kafka.common.FailedToSendMessageException: Failed to send messages after 3
> > tries.
> > at
> > kafka.producer.async.DefaultEventHandler.handle(DefaultEventHandler.scala:90)
> > at
> > kafka.producer.async.ProducerSendThread.tryToHandle(ProducerSendThread.scala:104)
> > at
> > kafka.producer.async.ProducerSendThread$$anonfun$processEvents$3.apply(ProducerSendThread.scala:87)
> > at
> > kafka.producer.async.ProducerSendThread$$anonfun$processEvents$3.apply(ProducerSendThread.scala:67)
> > at scala.collection.immutable.Stream.foreach(Stream.scala:254)
> > at
> > kafka.producer.async.ProducerSendThread.processEvents(ProducerSendThread.scala:66)
> > at
> > kafka.producer.async.ProducerSendThread.run(ProducerSendThread.scala:44)
> >
> >
> >
> > From: andthereitgoes@hotmail.com
> > To: users@kafka.apache.org
> > Subject: RE: Remote Zookeeper
> > Date: Tue, 11 Mar 2014 15:59:44 +0000
> >
> >
> >
> >
> > Okay thanks. Just to verify my setup I tried the following on broker1 (by
> > publishing to localhost)
> >
> > $KAFKA_HOME/bin/kafka-console-producer.sh --broker-list localhost:9092
> > --topic test
> > test message
> > this is a test message
> > another test message
> >
> > $KAFKA_HOME/bin/kafka-console-consumer.sh --zookeeper 192.168.1.120:2181--topic test --from-beginning
> > test message
> > this is a test message
> > another test message
> > Consumed 3 messages
> >
> > This works fine. How do I provide the broker-list?
> > I tried the following on broker1 (192.168.1.123)
> > $KAFKA_HOME/bin/kafka-console-producer.sh --broker-list 192.168.1.124:9092
> > 192.168.1.123:9092 --topic test
> > $KAFKA_HOME/bin/kafka-console-producer.sh --broker-list 192.168.1.124:9092
> > ,192.168.1.123:9092 --topic test
> > $KAFKA_HOME/bin/kafka-console-producer.sh --broker-list localhost:9092
> > 192.168.1.123:9092 --topic test
> > $KAFKA_HOME/bin/kafka-console-producer.sh --broker-list localhost:9092,
> > 192.168.1.123:9092 --topic test
> >
> > Didn't work. I either get connection refused or no route to host
> > exceptions.
> >
> > > Date: Tue, 11 Mar 2014 08:38:22 -0700
> > > Subject: Re: Remote Zookeeper
> > > From: b@b3k.us
> > > To: users@kafka.apache.org
> > >
> > > That's not how Kafka works. You need to pass the full list of brokers.
> > >
> > > On Tuesday, March 11, 2014, A A <an...@hotmail.com> wrote:
> > >
> > > > Hi again.
> > > >
> > > > Got the setup working. I now have 2 brokers (broker 1 and broker 2)
> > with
> > > > one remote zk. I was also able to create some topics
> > > >
> > > > $KAFKA_HOME/bin/kafka-list-topic.sh --zookeeper 192.168.1.120:2181
> > > > topic: test partition: 0 leader: 1 replicas: 1 isr: 1
> > > > topic: test1 partition: 0 leader: 1 replicas: 1 isr: 1
> > > >
> > > > I then try to publish a message from broker1 to broker 1 (localhost)
> > while
> > > > I am consuming on broker2
> > > > $KAFKA_HOME/bin/kafka-console-producer.sh --broker-list localhost:9092
> > > > --topic test1
> > > > helloooo
> > > > [2014-03-11 07:36:35,206] WARN Error while fetching metadata
> > > > [{TopicMetadata for topic test1 ->
> > > > No partition metadata for topic test1 due to
> > > > kafka.common.LeaderNotAvailableException}] for topic [test1]: class
> > > > kafka.common.LeaderNotAvailableException (kafka.pr
> > > > oducer.BrokerPartitionInfo)
> > > > [2014-03-11 07:36:35,244] WARN Error while fetching metadata
> > > > [{TopicMetadata for topic test1 ->
> > > > No partition metadata for topic test1 due to
> > > > kafka.common.LeaderNotAvailableException}] for topic [test1]: class
> > > > kafka.common.LeaderNotAvailableException (kafka.pr
> > > > oducer.BrokerPartitionInfo)
> > > > [2014-03-11 07:36:35,248] ERROR Failed to collate messages by topic,
> > > > partition due to: Failed to fetch topic metadata for topic: test1
> > > > (kafka.producer.async.Default
> > > > EventHandler)
> > > > [2014-03-11 07:36:35,358] WARN Error while fetching metadata
> > > > [{TopicMetadata for topic test1 ->
> > > > No partition metadata for topic test1 due to
> > > > kafka.common.LeaderNotAvailableException}] for topic [test1]: class
> > > > kafka.common.LeaderNotAvailableException (kafka.pr
> > > > oducer.BrokerPartitionInfo)
> > > > [2014-03-11 07:36:35,368] WARN Error while fetching metadata
> > > > [{TopicMetadata for topic test1 ->
> > > > No partition metadata for topic test1 due to
> > > > kafka.common.LeaderNotAvailableException}] for topic [test1]: class
> > > > kafka.common.LeaderNotAvailableException (kafka.pr
> > > > oducer.BrokerPartitionInfo)
> > > > [2014-03-11 07:36:35,368] ERROR Failed to collate messages by topic,
> > > > partition due to: Failed to fetch topic metadata for topic: test1
> > > > (kafka.producer.async.Default
> > > > EventHandler)
> > > > [2014-03-11 07:36:35,476] WARN Error while fetching metadata
> > > > [{TopicMetadata for topic test1 ->
> > > > No partition metadata for topic test1 due to
> > > > kafka.common.LeaderNotAvailableException}] for topic [test1]: class
> > > > kafka.common.LeaderNotAvailableException (kafka.pr
> > > > oducer.BrokerPartitionInfo)
> > > > [2014-03-11 07:36:35,490] WARN Error while fetching metadata
> > > > [{TopicMetadata for topic test1 ->
> > > > No partition metadata for topic test1 due to
> > > > kafka.common.LeaderNotAvailableException}] for topic [test1]: class
> > > > kafka.common.LeaderNotAvailableException (kafka.pr
> > > > oducer.BrokerPartitionInfo)
> > > > [2014-03-11 07:36:35,490] ERROR Failed to collate messages by topic,
> > > > partition due to: Failed to fetch topic metadata for topic: test1
> > > > (kafka.producer.async.Default
> > > > EventHandler)
> > > > [2014-03-11 07:36:35,599] WARN Error while fetching metadata
> > > > [{TopicMetadata for topic test1 ->
> > > > No partition metadata for topic test1 due to
> > > > kafka.common.LeaderNotAvailableException}] for topic [test1]: class
> > > > kafka.common.LeaderNotAvailableException (kafka.pr
> > > > oducer.BrokerPartitionInfo)
> > > > [2014-03-11 07:36:35,609] WARN Error while fetching metadata
> > > > [{TopicMetadata for topic test1 ->
> > > > No partition metadata for topic test1 due to
> > > > kafka.common.LeaderNotAvailableException}] for topic [test1]: class
> > > > kafka.common.LeaderNotAvailableException (kafka.pr
> > > > oducer.BrokerPartitionInfo)
> > > > [2014-03-11 07:36:35,609] ERROR Failed to collate messages by topic,
> > > > partition due to: Failed to fetch topic metadata for topic: test1
> > > > (kafka.producer.async.Default
> > > > EventHandler)
> > > > [2014-03-11 07:36:35,721] WARN Error while fetching metadata
> > > > [{TopicMetadata for topic test1 ->
> > > > No partition metadata for topic test1 due to
> > > > kafka.common.LeaderNotAvailableException}] for topic [test1]: class
> > > > kafka.common.LeaderNotAvailableException (kafka.pr
> > > > oducer.BrokerPartitionInfo)
> > > > [2014-03-11 07:36:35,723] ERROR Failed to send requests for topics
> > test1
> > > > with correlation ids in [0,8]
> > (kafka.producer.async.DefaultEventHandler)
> > > > [2014-03-11 07:36:35,725] ERROR Error in handling batch of 1 events
> > > > (kafka.producer.async.ProducerSendThread)
> > > > kafka.common.FailedToSendMessageException: Failed to send messages
> > after 3
> > > > tries.
> > > > at
> > > >
> > kafka.producer.async.DefaultEventHandler.handle(DefaultEventHandler.scala:90)
> > > > at
> > > >
> > kafka.producer.async.ProducerSendThread.tryToHandle(ProducerSendThread.scala:104)
> > > > at
> > > >
> > kafka.producer.async.ProducerSendThread$$anonfun$processEvents$3.apply(ProducerSendThread.scala:87)
> > > > at
> > > >
> > kafka.producer.async.ProducerSendThread$$anonfun$processEvents$3.apply(ProducerSendThread.scala:67)
> > > > at scala.collection.immutable.Stream.foreach(Stream.scala:254)
> > > > at
> > > >
> > kafka.producer.async.ProducerSendThread.processEvents(ProducerSendThread.scala:66)
> > > > at
> > > >
> > kafka.producer.async.ProducerSendThread.run(ProducerSendThread.scala:44)
> > > >
> > > > What am I missing?
> > > >
> > > > > From: andthereitgoes@hotmail.com <javascript:;>
> > > > > To: users@kafka.apache.org <javascript:;>
> > > > > Subject: RE: Remote Zookeeper
> > > > > Date: Tue, 11 Mar 2014 02:29:34 +0000
> > > > >
> > > > > Thanks. Totally missed that.
> > > > >
> > > > > > From: b@b3k.us <javascript:;>
> > > > > > Date: Mon, 10 Mar 2014 19:18:50 -0700
> > > > > > Subject: Re: Remote Zookeeper
> > > > > > To: users@kafka.apache.org <javascript:;>
> > > > > >
> > > > > > zookeeper.connect
> > > > > >
> > > > > > https://kafka.apache.org/08/configuration.html
> > > > > >
> > > > > >
> > > > > > On Mon, Mar 10, 2014 at 7:17 PM, A A <andthereitgoes@hotmail.com
> > <javascript:;>>
> > > > wrote:
> > > > > >
> > > > > > > Hi
> > > > > > >
> > > > > > > Pretty new to Kafka. Have been successful in installing Kafka
> > 0.8.0.
> > > > > > > I am just wondering how should I make my kafka cluster (2
> > brokers)
> > > > connect
> > > > > > > to a single remote zookeper server?
> > > > > > >
> > > > > > > I am using $KAFKA/kafka-server-start.sh
> > > > $KAFKA_CONFIG/server.properties
> > > > > > > on both the brokers to start them up
> > > > > > > and I have a remote zookeeper running at 192.168.1.120
> > > > > > >
> > > > > > > Can anyone help?
> > > > > > >
> > > > > > > A
> > > > > > >
> > > > > > >
> > > > >
> > > >
> >
> >
> >
Re: Remote Zookeeper
Posted by Pradeep Gollakota <pr...@gmail.com>.
Is there a firewall thats blocking connections on port 9092? Also, the
broker list should be comma separated.
On Tue, Mar 11, 2014 at 9:02 AM, A A <an...@hotmail.com> wrote:
> Sorry one of the brokers for was down. Brought it back up. Tried the
> following
>
> $KAFKA_HOME/bin/kafka-console-producer.sh --broker-list localhost:9092
> 192.168.1.124:9092 --topic test
> hello brokers
>
> [2014-03-11 10:16:55,547] WARN Error while fetching metadata
> [{TopicMetadata for topic test ->
> No partition metadata for topic test due to
> kafka.common.LeaderNotAvailableException}] for topic [test]: class
> kafka.common.LeaderNotAvailableException
> (kafka.producer.BrokerPartitionInfo)
> [2014-03-11 10:16:55,576] WARN Error while fetching metadata
> [{TopicMetadata for topic test ->
> No partition metadata for topic test due to
> kafka.common.LeaderNotAvailableException}] for topic [test]: class
> kafka.common.LeaderNotAvailableException
> (kafka.producer.BrokerPartitionInfo)
> [2014-03-11 10:16:55,578] ERROR Failed to collate messages by topic,
> partition due to: Failed to fetch topic metadata for topic: test
> (kafka.producer.async.DefaultEventHandler)
> [2014-03-11 10:16:55,693] WARN Error while fetching metadata
> [{TopicMetadata for topic test ->
> No partition metadata for topic test due to
> kafka.common.LeaderNotAvailableException}] for topic [test]: class
> kafka.common.LeaderNotAvailableException
> (kafka.producer.BrokerPartitionInfo)
> [2014-03-11 10:16:55,706] WARN Error while fetching metadata
> [{TopicMetadata for topic test ->
> No partition metadata for topic test due to
> kafka.common.LeaderNotAvailableException}] for topic [test]: class
> kafka.common.LeaderNotAvailableException
> (kafka.producer.BrokerPartitionInfo)
> [2014-03-11 10:16:55,706] ERROR Failed to collate messages by topic,
> partition due to: Failed to fetch topic metadata for topic: test
> (kafka.producer.async.DefaultEventHandler)
> [2014-03-11 10:16:55,815] WARN Error while fetching metadata
> [{TopicMetadata for topic test ->
> No partition metadata for topic test due to
> kafka.common.LeaderNotAvailableException}] for topic [test]: class
> kafka.common.LeaderNotAvailableException
> (kafka.producer.BrokerPartitionInfo)
> [2014-03-11 10:16:55,823] WARN Error while fetching metadata
> [{TopicMetadata for topic test ->
> No partition metadata for topic test due to
> kafka.common.LeaderNotAvailableException}] for topic [test]: class
> kafka.common.LeaderNotAvailableException
> (kafka.producer.BrokerPartitionInfo)
> [2014-03-11 10:16:55,823] ERROR Failed to collate messages by topic,
> partition due to: Failed to fetch topic metadata for topic: test
> (kafka.producer.async.DefaultEventHandler)
> [2014-03-11 10:16:55,934] WARN Error while fetching metadata
> [{TopicMetadata for topic test ->
> No partition metadata for topic test due to
> kafka.common.LeaderNotAvailableException}] for topic [test]: class
> kafka.common.LeaderNotAvailableException
> (kafka.producer.BrokerPartitionInfo)
> [2014-03-11 10:16:55,949] WARN Error while fetching metadata
> [{TopicMetadata for topic test ->
> No partition metadata for topic test due to
> kafka.common.LeaderNotAvailableException}] for topic [test]: class
> kafka.common.LeaderNotAvailableException
> (kafka.producer.BrokerPartitionInfo)
> [2014-03-11 10:16:55,949] ERROR Failed to collate messages by topic,
> partition due to: Failed to fetch topic metadata for topic: test
> (kafka.producer.async.DefaultEventHandler)
> [2014-03-11 10:16:56,057] WARN Error while fetching metadata
> [{TopicMetadata for topic test ->
> No partition metadata for topic test due to
> kafka.common.LeaderNotAvailableException}] for topic [test]: class
> kafka.common.LeaderNotAvailableException
> (kafka.producer.BrokerPartitionInfo)
> [2014-03-11 10:16:56,059] ERROR Failed to send requests for topics test
> with correlation ids in [0,8] (kafka.producer.async.DefaultEventHandler)
> [2014-03-11 10:16:56,061] ERROR Error in handling batch of 1 events
> (kafka.producer.async.ProducerSendThread)
> kafka.common.FailedToSendMessageException: Failed to send messages after 3
> tries.
> at
> kafka.producer.async.DefaultEventHandler.handle(DefaultEventHandler.scala:90)
> at
> kafka.producer.async.ProducerSendThread.tryToHandle(ProducerSendThread.scala:104)
> at
> kafka.producer.async.ProducerSendThread$$anonfun$processEvents$3.apply(ProducerSendThread.scala:87)
> at
> kafka.producer.async.ProducerSendThread$$anonfun$processEvents$3.apply(ProducerSendThread.scala:67)
> at scala.collection.immutable.Stream.foreach(Stream.scala:254)
> at
> kafka.producer.async.ProducerSendThread.processEvents(ProducerSendThread.scala:66)
> at
> kafka.producer.async.ProducerSendThread.run(ProducerSendThread.scala:44)
>
>
>
> From: andthereitgoes@hotmail.com
> To: users@kafka.apache.org
> Subject: RE: Remote Zookeeper
> Date: Tue, 11 Mar 2014 15:59:44 +0000
>
>
>
>
> Okay thanks. Just to verify my setup I tried the following on broker1 (by
> publishing to localhost)
>
> $KAFKA_HOME/bin/kafka-console-producer.sh --broker-list localhost:9092
> --topic test
> test message
> this is a test message
> another test message
>
> $KAFKA_HOME/bin/kafka-console-consumer.sh --zookeeper 192.168.1.120:2181--topic test --from-beginning
> test message
> this is a test message
> another test message
> Consumed 3 messages
>
> This works fine. How do I provide the broker-list?
> I tried the following on broker1 (192.168.1.123)
> $KAFKA_HOME/bin/kafka-console-producer.sh --broker-list 192.168.1.124:9092
> 192.168.1.123:9092 --topic test
> $KAFKA_HOME/bin/kafka-console-producer.sh --broker-list 192.168.1.124:9092
> ,192.168.1.123:9092 --topic test
> $KAFKA_HOME/bin/kafka-console-producer.sh --broker-list localhost:9092
> 192.168.1.123:9092 --topic test
> $KAFKA_HOME/bin/kafka-console-producer.sh --broker-list localhost:9092,
> 192.168.1.123:9092 --topic test
>
> Didn't work. I either get connection refused or no route to host
> exceptions.
>
> > Date: Tue, 11 Mar 2014 08:38:22 -0700
> > Subject: Re: Remote Zookeeper
> > From: b@b3k.us
> > To: users@kafka.apache.org
> >
> > That's not how Kafka works. You need to pass the full list of brokers.
> >
> > On Tuesday, March 11, 2014, A A <an...@hotmail.com> wrote:
> >
> > > Hi again.
> > >
> > > Got the setup working. I now have 2 brokers (broker 1 and broker 2)
> with
> > > one remote zk. I was also able to create some topics
> > >
> > > $KAFKA_HOME/bin/kafka-list-topic.sh --zookeeper 192.168.1.120:2181
> > > topic: test partition: 0 leader: 1 replicas: 1 isr: 1
> > > topic: test1 partition: 0 leader: 1 replicas: 1 isr: 1
> > >
> > > I then try to publish a message from broker1 to broker 1 (localhost)
> while
> > > I am consuming on broker2
> > > $KAFKA_HOME/bin/kafka-console-producer.sh --broker-list localhost:9092
> > > --topic test1
> > > helloooo
> > > [2014-03-11 07:36:35,206] WARN Error while fetching metadata
> > > [{TopicMetadata for topic test1 ->
> > > No partition metadata for topic test1 due to
> > > kafka.common.LeaderNotAvailableException}] for topic [test1]: class
> > > kafka.common.LeaderNotAvailableException (kafka.pr
> > > oducer.BrokerPartitionInfo)
> > > [2014-03-11 07:36:35,244] WARN Error while fetching metadata
> > > [{TopicMetadata for topic test1 ->
> > > No partition metadata for topic test1 due to
> > > kafka.common.LeaderNotAvailableException}] for topic [test1]: class
> > > kafka.common.LeaderNotAvailableException (kafka.pr
> > > oducer.BrokerPartitionInfo)
> > > [2014-03-11 07:36:35,248] ERROR Failed to collate messages by topic,
> > > partition due to: Failed to fetch topic metadata for topic: test1
> > > (kafka.producer.async.Default
> > > EventHandler)
> > > [2014-03-11 07:36:35,358] WARN Error while fetching metadata
> > > [{TopicMetadata for topic test1 ->
> > > No partition metadata for topic test1 due to
> > > kafka.common.LeaderNotAvailableException}] for topic [test1]: class
> > > kafka.common.LeaderNotAvailableException (kafka.pr
> > > oducer.BrokerPartitionInfo)
> > > [2014-03-11 07:36:35,368] WARN Error while fetching metadata
> > > [{TopicMetadata for topic test1 ->
> > > No partition metadata for topic test1 due to
> > > kafka.common.LeaderNotAvailableException}] for topic [test1]: class
> > > kafka.common.LeaderNotAvailableException (kafka.pr
> > > oducer.BrokerPartitionInfo)
> > > [2014-03-11 07:36:35,368] ERROR Failed to collate messages by topic,
> > > partition due to: Failed to fetch topic metadata for topic: test1
> > > (kafka.producer.async.Default
> > > EventHandler)
> > > [2014-03-11 07:36:35,476] WARN Error while fetching metadata
> > > [{TopicMetadata for topic test1 ->
> > > No partition metadata for topic test1 due to
> > > kafka.common.LeaderNotAvailableException}] for topic [test1]: class
> > > kafka.common.LeaderNotAvailableException (kafka.pr
> > > oducer.BrokerPartitionInfo)
> > > [2014-03-11 07:36:35,490] WARN Error while fetching metadata
> > > [{TopicMetadata for topic test1 ->
> > > No partition metadata for topic test1 due to
> > > kafka.common.LeaderNotAvailableException}] for topic [test1]: class
> > > kafka.common.LeaderNotAvailableException (kafka.pr
> > > oducer.BrokerPartitionInfo)
> > > [2014-03-11 07:36:35,490] ERROR Failed to collate messages by topic,
> > > partition due to: Failed to fetch topic metadata for topic: test1
> > > (kafka.producer.async.Default
> > > EventHandler)
> > > [2014-03-11 07:36:35,599] WARN Error while fetching metadata
> > > [{TopicMetadata for topic test1 ->
> > > No partition metadata for topic test1 due to
> > > kafka.common.LeaderNotAvailableException}] for topic [test1]: class
> > > kafka.common.LeaderNotAvailableException (kafka.pr
> > > oducer.BrokerPartitionInfo)
> > > [2014-03-11 07:36:35,609] WARN Error while fetching metadata
> > > [{TopicMetadata for topic test1 ->
> > > No partition metadata for topic test1 due to
> > > kafka.common.LeaderNotAvailableException}] for topic [test1]: class
> > > kafka.common.LeaderNotAvailableException (kafka.pr
> > > oducer.BrokerPartitionInfo)
> > > [2014-03-11 07:36:35,609] ERROR Failed to collate messages by topic,
> > > partition due to: Failed to fetch topic metadata for topic: test1
> > > (kafka.producer.async.Default
> > > EventHandler)
> > > [2014-03-11 07:36:35,721] WARN Error while fetching metadata
> > > [{TopicMetadata for topic test1 ->
> > > No partition metadata for topic test1 due to
> > > kafka.common.LeaderNotAvailableException}] for topic [test1]: class
> > > kafka.common.LeaderNotAvailableException (kafka.pr
> > > oducer.BrokerPartitionInfo)
> > > [2014-03-11 07:36:35,723] ERROR Failed to send requests for topics
> test1
> > > with correlation ids in [0,8]
> (kafka.producer.async.DefaultEventHandler)
> > > [2014-03-11 07:36:35,725] ERROR Error in handling batch of 1 events
> > > (kafka.producer.async.ProducerSendThread)
> > > kafka.common.FailedToSendMessageException: Failed to send messages
> after 3
> > > tries.
> > > at
> > >
> kafka.producer.async.DefaultEventHandler.handle(DefaultEventHandler.scala:90)
> > > at
> > >
> kafka.producer.async.ProducerSendThread.tryToHandle(ProducerSendThread.scala:104)
> > > at
> > >
> kafka.producer.async.ProducerSendThread$$anonfun$processEvents$3.apply(ProducerSendThread.scala:87)
> > > at
> > >
> kafka.producer.async.ProducerSendThread$$anonfun$processEvents$3.apply(ProducerSendThread.scala:67)
> > > at scala.collection.immutable.Stream.foreach(Stream.scala:254)
> > > at
> > >
> kafka.producer.async.ProducerSendThread.processEvents(ProducerSendThread.scala:66)
> > > at
> > >
> kafka.producer.async.ProducerSendThread.run(ProducerSendThread.scala:44)
> > >
> > > What am I missing?
> > >
> > > > From: andthereitgoes@hotmail.com <javascript:;>
> > > > To: users@kafka.apache.org <javascript:;>
> > > > Subject: RE: Remote Zookeeper
> > > > Date: Tue, 11 Mar 2014 02:29:34 +0000
> > > >
> > > > Thanks. Totally missed that.
> > > >
> > > > > From: b@b3k.us <javascript:;>
> > > > > Date: Mon, 10 Mar 2014 19:18:50 -0700
> > > > > Subject: Re: Remote Zookeeper
> > > > > To: users@kafka.apache.org <javascript:;>
> > > > >
> > > > > zookeeper.connect
> > > > >
> > > > > https://kafka.apache.org/08/configuration.html
> > > > >
> > > > >
> > > > > On Mon, Mar 10, 2014 at 7:17 PM, A A <andthereitgoes@hotmail.com
> <javascript:;>>
> > > wrote:
> > > > >
> > > > > > Hi
> > > > > >
> > > > > > Pretty new to Kafka. Have been successful in installing Kafka
> 0.8.0.
> > > > > > I am just wondering how should I make my kafka cluster (2
> brokers)
> > > connect
> > > > > > to a single remote zookeper server?
> > > > > >
> > > > > > I am using $KAFKA/kafka-server-start.sh
> > > $KAFKA_CONFIG/server.properties
> > > > > > on both the brokers to start them up
> > > > > > and I have a remote zookeeper running at 192.168.1.120
> > > > > >
> > > > > > Can anyone help?
> > > > > >
> > > > > > A
> > > > > >
> > > > > >
> > > >
> > >
>
>
>
RE: Remote Zookeeper
Posted by A A <an...@hotmail.com>.
Sorry one of the brokers for was down. Brought it back up. Tried the following
$KAFKA_HOME/bin/kafka-console-producer.sh --broker-list localhost:9092 192.168.1.124:9092 --topic test
hello brokers
[2014-03-11 10:16:55,547] WARN Error while fetching metadata [{TopicMetadata for topic test ->
No partition metadata for topic test due to kafka.common.LeaderNotAvailableException}] for topic [test]: class kafka.common.LeaderNotAvailableException (kafka.producer.BrokerPartitionInfo)
[2014-03-11 10:16:55,576] WARN Error while fetching metadata [{TopicMetadata for topic test ->
No partition metadata for topic test due to kafka.common.LeaderNotAvailableException}] for topic [test]: class kafka.common.LeaderNotAvailableException (kafka.producer.BrokerPartitionInfo)
[2014-03-11 10:16:55,578] ERROR Failed to collate messages by topic, partition due to: Failed to fetch topic metadata for topic: test (kafka.producer.async.DefaultEventHandler)
[2014-03-11 10:16:55,693] WARN Error while fetching metadata [{TopicMetadata for topic test ->
No partition metadata for topic test due to kafka.common.LeaderNotAvailableException}] for topic [test]: class kafka.common.LeaderNotAvailableException (kafka.producer.BrokerPartitionInfo)
[2014-03-11 10:16:55,706] WARN Error while fetching metadata [{TopicMetadata for topic test ->
No partition metadata for topic test due to kafka.common.LeaderNotAvailableException}] for topic [test]: class kafka.common.LeaderNotAvailableException (kafka.producer.BrokerPartitionInfo)
[2014-03-11 10:16:55,706] ERROR Failed to collate messages by topic, partition due to: Failed to fetch topic metadata for topic: test (kafka.producer.async.DefaultEventHandler)
[2014-03-11 10:16:55,815] WARN Error while fetching metadata [{TopicMetadata for topic test ->
No partition metadata for topic test due to kafka.common.LeaderNotAvailableException}] for topic [test]: class kafka.common.LeaderNotAvailableException (kafka.producer.BrokerPartitionInfo)
[2014-03-11 10:16:55,823] WARN Error while fetching metadata [{TopicMetadata for topic test ->
No partition metadata for topic test due to kafka.common.LeaderNotAvailableException}] for topic [test]: class kafka.common.LeaderNotAvailableException (kafka.producer.BrokerPartitionInfo)
[2014-03-11 10:16:55,823] ERROR Failed to collate messages by topic, partition due to: Failed to fetch topic metadata for topic: test (kafka.producer.async.DefaultEventHandler)
[2014-03-11 10:16:55,934] WARN Error while fetching metadata [{TopicMetadata for topic test ->
No partition metadata for topic test due to kafka.common.LeaderNotAvailableException}] for topic [test]: class kafka.common.LeaderNotAvailableException (kafka.producer.BrokerPartitionInfo)
[2014-03-11 10:16:55,949] WARN Error while fetching metadata [{TopicMetadata for topic test ->
No partition metadata for topic test due to kafka.common.LeaderNotAvailableException}] for topic [test]: class kafka.common.LeaderNotAvailableException (kafka.producer.BrokerPartitionInfo)
[2014-03-11 10:16:55,949] ERROR Failed to collate messages by topic, partition due to: Failed to fetch topic metadata for topic: test (kafka.producer.async.DefaultEventHandler)
[2014-03-11 10:16:56,057] WARN Error while fetching metadata [{TopicMetadata for topic test ->
No partition metadata for topic test due to kafka.common.LeaderNotAvailableException}] for topic [test]: class kafka.common.LeaderNotAvailableException (kafka.producer.BrokerPartitionInfo)
[2014-03-11 10:16:56,059] ERROR Failed to send requests for topics test with correlation ids in [0,8] (kafka.producer.async.DefaultEventHandler)
[2014-03-11 10:16:56,061] ERROR Error in handling batch of 1 events (kafka.producer.async.ProducerSendThread)
kafka.common.FailedToSendMessageException: Failed to send messages after 3 tries.
at kafka.producer.async.DefaultEventHandler.handle(DefaultEventHandler.scala:90)
at kafka.producer.async.ProducerSendThread.tryToHandle(ProducerSendThread.scala:104)
at kafka.producer.async.ProducerSendThread$$anonfun$processEvents$3.apply(ProducerSendThread.scala:87)
at kafka.producer.async.ProducerSendThread$$anonfun$processEvents$3.apply(ProducerSendThread.scala:67)
at scala.collection.immutable.Stream.foreach(Stream.scala:254)
at kafka.producer.async.ProducerSendThread.processEvents(ProducerSendThread.scala:66)
at kafka.producer.async.ProducerSendThread.run(ProducerSendThread.scala:44)
From: andthereitgoes@hotmail.com
To: users@kafka.apache.org
Subject: RE: Remote Zookeeper
Date: Tue, 11 Mar 2014 15:59:44 +0000
Okay thanks. Just to verify my setup I tried the following on broker1 (by publishing to localhost)
$KAFKA_HOME/bin/kafka-console-producer.sh --broker-list localhost:9092 --topic test
test message
this is a test message
another test message
$KAFKA_HOME/bin/kafka-console-consumer.sh --zookeeper 192.168.1.120:2181 --topic test --from-beginning
test message
this is a test message
another test message
Consumed 3 messages
This works fine. How do I provide the broker-list?
I tried the following on broker1 (192.168.1.123)
$KAFKA_HOME/bin/kafka-console-producer.sh --broker-list 192.168.1.124:9092 192.168.1.123:9092 --topic test
$KAFKA_HOME/bin/kafka-console-producer.sh --broker-list 192.168.1.124:9092,192.168.1.123:9092 --topic test
$KAFKA_HOME/bin/kafka-console-producer.sh --broker-list localhost:9092 192.168.1.123:9092 --topic test
$KAFKA_HOME/bin/kafka-console-producer.sh --broker-list localhost:9092,192.168.1.123:9092 --topic test
Didn't work. I either get connection refused or no route to host exceptions.
> Date: Tue, 11 Mar 2014 08:38:22 -0700
> Subject: Re: Remote Zookeeper
> From: b@b3k.us
> To: users@kafka.apache.org
>
> That's not how Kafka works. You need to pass the full list of brokers.
>
> On Tuesday, March 11, 2014, A A <an...@hotmail.com> wrote:
>
> > Hi again.
> >
> > Got the setup working. I now have 2 brokers (broker 1 and broker 2) with
> > one remote zk. I was also able to create some topics
> >
> > $KAFKA_HOME/bin/kafka-list-topic.sh --zookeeper 192.168.1.120:2181
> > topic: test partition: 0 leader: 1 replicas: 1 isr: 1
> > topic: test1 partition: 0 leader: 1 replicas: 1 isr: 1
> >
> > I then try to publish a message from broker1 to broker 1 (localhost) while
> > I am consuming on broker2
> > $KAFKA_HOME/bin/kafka-console-producer.sh --broker-list localhost:9092
> > --topic test1
> > helloooo
> > [2014-03-11 07:36:35,206] WARN Error while fetching metadata
> > [{TopicMetadata for topic test1 ->
> > No partition metadata for topic test1 due to
> > kafka.common.LeaderNotAvailableException}] for topic [test1]: class
> > kafka.common.LeaderNotAvailableException (kafka.pr
> > oducer.BrokerPartitionInfo)
> > [2014-03-11 07:36:35,244] WARN Error while fetching metadata
> > [{TopicMetadata for topic test1 ->
> > No partition metadata for topic test1 due to
> > kafka.common.LeaderNotAvailableException}] for topic [test1]: class
> > kafka.common.LeaderNotAvailableException (kafka.pr
> > oducer.BrokerPartitionInfo)
> > [2014-03-11 07:36:35,248] ERROR Failed to collate messages by topic,
> > partition due to: Failed to fetch topic metadata for topic: test1
> > (kafka.producer.async.Default
> > EventHandler)
> > [2014-03-11 07:36:35,358] WARN Error while fetching metadata
> > [{TopicMetadata for topic test1 ->
> > No partition metadata for topic test1 due to
> > kafka.common.LeaderNotAvailableException}] for topic [test1]: class
> > kafka.common.LeaderNotAvailableException (kafka.pr
> > oducer.BrokerPartitionInfo)
> > [2014-03-11 07:36:35,368] WARN Error while fetching metadata
> > [{TopicMetadata for topic test1 ->
> > No partition metadata for topic test1 due to
> > kafka.common.LeaderNotAvailableException}] for topic [test1]: class
> > kafka.common.LeaderNotAvailableException (kafka.pr
> > oducer.BrokerPartitionInfo)
> > [2014-03-11 07:36:35,368] ERROR Failed to collate messages by topic,
> > partition due to: Failed to fetch topic metadata for topic: test1
> > (kafka.producer.async.Default
> > EventHandler)
> > [2014-03-11 07:36:35,476] WARN Error while fetching metadata
> > [{TopicMetadata for topic test1 ->
> > No partition metadata for topic test1 due to
> > kafka.common.LeaderNotAvailableException}] for topic [test1]: class
> > kafka.common.LeaderNotAvailableException (kafka.pr
> > oducer.BrokerPartitionInfo)
> > [2014-03-11 07:36:35,490] WARN Error while fetching metadata
> > [{TopicMetadata for topic test1 ->
> > No partition metadata for topic test1 due to
> > kafka.common.LeaderNotAvailableException}] for topic [test1]: class
> > kafka.common.LeaderNotAvailableException (kafka.pr
> > oducer.BrokerPartitionInfo)
> > [2014-03-11 07:36:35,490] ERROR Failed to collate messages by topic,
> > partition due to: Failed to fetch topic metadata for topic: test1
> > (kafka.producer.async.Default
> > EventHandler)
> > [2014-03-11 07:36:35,599] WARN Error while fetching metadata
> > [{TopicMetadata for topic test1 ->
> > No partition metadata for topic test1 due to
> > kafka.common.LeaderNotAvailableException}] for topic [test1]: class
> > kafka.common.LeaderNotAvailableException (kafka.pr
> > oducer.BrokerPartitionInfo)
> > [2014-03-11 07:36:35,609] WARN Error while fetching metadata
> > [{TopicMetadata for topic test1 ->
> > No partition metadata for topic test1 due to
> > kafka.common.LeaderNotAvailableException}] for topic [test1]: class
> > kafka.common.LeaderNotAvailableException (kafka.pr
> > oducer.BrokerPartitionInfo)
> > [2014-03-11 07:36:35,609] ERROR Failed to collate messages by topic,
> > partition due to: Failed to fetch topic metadata for topic: test1
> > (kafka.producer.async.Default
> > EventHandler)
> > [2014-03-11 07:36:35,721] WARN Error while fetching metadata
> > [{TopicMetadata for topic test1 ->
> > No partition metadata for topic test1 due to
> > kafka.common.LeaderNotAvailableException}] for topic [test1]: class
> > kafka.common.LeaderNotAvailableException (kafka.pr
> > oducer.BrokerPartitionInfo)
> > [2014-03-11 07:36:35,723] ERROR Failed to send requests for topics test1
> > with correlation ids in [0,8] (kafka.producer.async.DefaultEventHandler)
> > [2014-03-11 07:36:35,725] ERROR Error in handling batch of 1 events
> > (kafka.producer.async.ProducerSendThread)
> > kafka.common.FailedToSendMessageException: Failed to send messages after 3
> > tries.
> > at
> > kafka.producer.async.DefaultEventHandler.handle(DefaultEventHandler.scala:90)
> > at
> > kafka.producer.async.ProducerSendThread.tryToHandle(ProducerSendThread.scala:104)
> > at
> > kafka.producer.async.ProducerSendThread$$anonfun$processEvents$3.apply(ProducerSendThread.scala:87)
> > at
> > kafka.producer.async.ProducerSendThread$$anonfun$processEvents$3.apply(ProducerSendThread.scala:67)
> > at scala.collection.immutable.Stream.foreach(Stream.scala:254)
> > at
> > kafka.producer.async.ProducerSendThread.processEvents(ProducerSendThread.scala:66)
> > at
> > kafka.producer.async.ProducerSendThread.run(ProducerSendThread.scala:44)
> >
> > What am I missing?
> >
> > > From: andthereitgoes@hotmail.com <javascript:;>
> > > To: users@kafka.apache.org <javascript:;>
> > > Subject: RE: Remote Zookeeper
> > > Date: Tue, 11 Mar 2014 02:29:34 +0000
> > >
> > > Thanks. Totally missed that.
> > >
> > > > From: b@b3k.us <javascript:;>
> > > > Date: Mon, 10 Mar 2014 19:18:50 -0700
> > > > Subject: Re: Remote Zookeeper
> > > > To: users@kafka.apache.org <javascript:;>
> > > >
> > > > zookeeper.connect
> > > >
> > > > https://kafka.apache.org/08/configuration.html
> > > >
> > > >
> > > > On Mon, Mar 10, 2014 at 7:17 PM, A A <andthereitgoes@hotmail.com<javascript:;>>
> > wrote:
> > > >
> > > > > Hi
> > > > >
> > > > > Pretty new to Kafka. Have been successful in installing Kafka 0.8.0.
> > > > > I am just wondering how should I make my kafka cluster (2 brokers)
> > connect
> > > > > to a single remote zookeper server?
> > > > >
> > > > > I am using $KAFKA/kafka-server-start.sh
> > $KAFKA_CONFIG/server.properties
> > > > > on both the brokers to start them up
> > > > > and I have a remote zookeeper running at 192.168.1.120
> > > > >
> > > > > Can anyone help?
> > > > >
> > > > > A
> > > > >
> > > > >
> > >
> >
RE: Remote Zookeeper
Posted by A A <an...@hotmail.com>.
Okay thanks. Just to verify my setup I tried the following on broker1 (by publishing to localhost)
$KAFKA_HOME/bin/kafka-console-producer.sh --broker-list localhost:9092 --topic test
test message
this is a test message
another test message
$KAFKA_HOME/bin/kafka-console-consumer.sh --zookeeper 192.168.1.120:2181 --topic test --from-beginning
test message
this is a test message
another test message
Consumed 3 messages
This works fine. How do I provide the broker-list?
I tried the following on broker1 (192.168.1.123)
$KAFKA_HOME/bin/kafka-console-producer.sh --broker-list 192.168.1.124:9092 192.168.1.123:9092 --topic test
$KAFKA_HOME/bin/kafka-console-producer.sh --broker-list 192.168.1.124:9092,192.168.1.123:9092 --topic test
$KAFKA_HOME/bin/kafka-console-producer.sh --broker-list localhost:9092 192.168.1.123:9092 --topic test
$KAFKA_HOME/bin/kafka-console-producer.sh --broker-list localhost:9092,192.168.1.123:9092 --topic test
Didn't work. I either get connection refused or no route to host exceptions.
> Date: Tue, 11 Mar 2014 08:38:22 -0700
> Subject: Re: Remote Zookeeper
> From: b@b3k.us
> To: users@kafka.apache.org
>
> That's not how Kafka works. You need to pass the full list of brokers.
>
> On Tuesday, March 11, 2014, A A <an...@hotmail.com> wrote:
>
> > Hi again.
> >
> > Got the setup working. I now have 2 brokers (broker 1 and broker 2) with
> > one remote zk. I was also able to create some topics
> >
> > $KAFKA_HOME/bin/kafka-list-topic.sh --zookeeper 192.168.1.120:2181
> > topic: test partition: 0 leader: 1 replicas: 1 isr: 1
> > topic: test1 partition: 0 leader: 1 replicas: 1 isr: 1
> >
> > I then try to publish a message from broker1 to broker 1 (localhost) while
> > I am consuming on broker2
> > $KAFKA_HOME/bin/kafka-console-producer.sh --broker-list localhost:9092
> > --topic test1
> > helloooo
> > [2014-03-11 07:36:35,206] WARN Error while fetching metadata
> > [{TopicMetadata for topic test1 ->
> > No partition metadata for topic test1 due to
> > kafka.common.LeaderNotAvailableException}] for topic [test1]: class
> > kafka.common.LeaderNotAvailableException (kafka.pr
> > oducer.BrokerPartitionInfo)
> > [2014-03-11 07:36:35,244] WARN Error while fetching metadata
> > [{TopicMetadata for topic test1 ->
> > No partition metadata for topic test1 due to
> > kafka.common.LeaderNotAvailableException}] for topic [test1]: class
> > kafka.common.LeaderNotAvailableException (kafka.pr
> > oducer.BrokerPartitionInfo)
> > [2014-03-11 07:36:35,248] ERROR Failed to collate messages by topic,
> > partition due to: Failed to fetch topic metadata for topic: test1
> > (kafka.producer.async.Default
> > EventHandler)
> > [2014-03-11 07:36:35,358] WARN Error while fetching metadata
> > [{TopicMetadata for topic test1 ->
> > No partition metadata for topic test1 due to
> > kafka.common.LeaderNotAvailableException}] for topic [test1]: class
> > kafka.common.LeaderNotAvailableException (kafka.pr
> > oducer.BrokerPartitionInfo)
> > [2014-03-11 07:36:35,368] WARN Error while fetching metadata
> > [{TopicMetadata for topic test1 ->
> > No partition metadata for topic test1 due to
> > kafka.common.LeaderNotAvailableException}] for topic [test1]: class
> > kafka.common.LeaderNotAvailableException (kafka.pr
> > oducer.BrokerPartitionInfo)
> > [2014-03-11 07:36:35,368] ERROR Failed to collate messages by topic,
> > partition due to: Failed to fetch topic metadata for topic: test1
> > (kafka.producer.async.Default
> > EventHandler)
> > [2014-03-11 07:36:35,476] WARN Error while fetching metadata
> > [{TopicMetadata for topic test1 ->
> > No partition metadata for topic test1 due to
> > kafka.common.LeaderNotAvailableException}] for topic [test1]: class
> > kafka.common.LeaderNotAvailableException (kafka.pr
> > oducer.BrokerPartitionInfo)
> > [2014-03-11 07:36:35,490] WARN Error while fetching metadata
> > [{TopicMetadata for topic test1 ->
> > No partition metadata for topic test1 due to
> > kafka.common.LeaderNotAvailableException}] for topic [test1]: class
> > kafka.common.LeaderNotAvailableException (kafka.pr
> > oducer.BrokerPartitionInfo)
> > [2014-03-11 07:36:35,490] ERROR Failed to collate messages by topic,
> > partition due to: Failed to fetch topic metadata for topic: test1
> > (kafka.producer.async.Default
> > EventHandler)
> > [2014-03-11 07:36:35,599] WARN Error while fetching metadata
> > [{TopicMetadata for topic test1 ->
> > No partition metadata for topic test1 due to
> > kafka.common.LeaderNotAvailableException}] for topic [test1]: class
> > kafka.common.LeaderNotAvailableException (kafka.pr
> > oducer.BrokerPartitionInfo)
> > [2014-03-11 07:36:35,609] WARN Error while fetching metadata
> > [{TopicMetadata for topic test1 ->
> > No partition metadata for topic test1 due to
> > kafka.common.LeaderNotAvailableException}] for topic [test1]: class
> > kafka.common.LeaderNotAvailableException (kafka.pr
> > oducer.BrokerPartitionInfo)
> > [2014-03-11 07:36:35,609] ERROR Failed to collate messages by topic,
> > partition due to: Failed to fetch topic metadata for topic: test1
> > (kafka.producer.async.Default
> > EventHandler)
> > [2014-03-11 07:36:35,721] WARN Error while fetching metadata
> > [{TopicMetadata for topic test1 ->
> > No partition metadata for topic test1 due to
> > kafka.common.LeaderNotAvailableException}] for topic [test1]: class
> > kafka.common.LeaderNotAvailableException (kafka.pr
> > oducer.BrokerPartitionInfo)
> > [2014-03-11 07:36:35,723] ERROR Failed to send requests for topics test1
> > with correlation ids in [0,8] (kafka.producer.async.DefaultEventHandler)
> > [2014-03-11 07:36:35,725] ERROR Error in handling batch of 1 events
> > (kafka.producer.async.ProducerSendThread)
> > kafka.common.FailedToSendMessageException: Failed to send messages after 3
> > tries.
> > at
> > kafka.producer.async.DefaultEventHandler.handle(DefaultEventHandler.scala:90)
> > at
> > kafka.producer.async.ProducerSendThread.tryToHandle(ProducerSendThread.scala:104)
> > at
> > kafka.producer.async.ProducerSendThread$$anonfun$processEvents$3.apply(ProducerSendThread.scala:87)
> > at
> > kafka.producer.async.ProducerSendThread$$anonfun$processEvents$3.apply(ProducerSendThread.scala:67)
> > at scala.collection.immutable.Stream.foreach(Stream.scala:254)
> > at
> > kafka.producer.async.ProducerSendThread.processEvents(ProducerSendThread.scala:66)
> > at
> > kafka.producer.async.ProducerSendThread.run(ProducerSendThread.scala:44)
> >
> > What am I missing?
> >
> > > From: andthereitgoes@hotmail.com <javascript:;>
> > > To: users@kafka.apache.org <javascript:;>
> > > Subject: RE: Remote Zookeeper
> > > Date: Tue, 11 Mar 2014 02:29:34 +0000
> > >
> > > Thanks. Totally missed that.
> > >
> > > > From: b@b3k.us <javascript:;>
> > > > Date: Mon, 10 Mar 2014 19:18:50 -0700
> > > > Subject: Re: Remote Zookeeper
> > > > To: users@kafka.apache.org <javascript:;>
> > > >
> > > > zookeeper.connect
> > > >
> > > > https://kafka.apache.org/08/configuration.html
> > > >
> > > >
> > > > On Mon, Mar 10, 2014 at 7:17 PM, A A <andthereitgoes@hotmail.com<javascript:;>>
> > wrote:
> > > >
> > > > > Hi
> > > > >
> > > > > Pretty new to Kafka. Have been successful in installing Kafka 0.8.0.
> > > > > I am just wondering how should I make my kafka cluster (2 brokers)
> > connect
> > > > > to a single remote zookeper server?
> > > > >
> > > > > I am using $KAFKA/kafka-server-start.sh
> > $KAFKA_CONFIG/server.properties
> > > > > on both the brokers to start them up
> > > > > and I have a remote zookeeper running at 192.168.1.120
> > > > >
> > > > > Can anyone help?
> > > > >
> > > > > A
> > > > >
> > > > >
> > >
> >
Re: Remote Zookeeper
Posted by Benjamin Black <b...@b3k.us>.
That's not how Kafka works. You need to pass the full list of brokers.
On Tuesday, March 11, 2014, A A <an...@hotmail.com> wrote:
> Hi again.
>
> Got the setup working. I now have 2 brokers (broker 1 and broker 2) with
> one remote zk. I was also able to create some topics
>
> $KAFKA_HOME/bin/kafka-list-topic.sh --zookeeper 192.168.1.120:2181
> topic: test partition: 0 leader: 1 replicas: 1 isr: 1
> topic: test1 partition: 0 leader: 1 replicas: 1 isr: 1
>
> I then try to publish a message from broker1 to broker 1 (localhost) while
> I am consuming on broker2
> $KAFKA_HOME/bin/kafka-console-producer.sh --broker-list localhost:9092
> --topic test1
> helloooo
> [2014-03-11 07:36:35,206] WARN Error while fetching metadata
> [{TopicMetadata for topic test1 ->
> No partition metadata for topic test1 due to
> kafka.common.LeaderNotAvailableException}] for topic [test1]: class
> kafka.common.LeaderNotAvailableException (kafka.pr
> oducer.BrokerPartitionInfo)
> [2014-03-11 07:36:35,244] WARN Error while fetching metadata
> [{TopicMetadata for topic test1 ->
> No partition metadata for topic test1 due to
> kafka.common.LeaderNotAvailableException}] for topic [test1]: class
> kafka.common.LeaderNotAvailableException (kafka.pr
> oducer.BrokerPartitionInfo)
> [2014-03-11 07:36:35,248] ERROR Failed to collate messages by topic,
> partition due to: Failed to fetch topic metadata for topic: test1
> (kafka.producer.async.Default
> EventHandler)
> [2014-03-11 07:36:35,358] WARN Error while fetching metadata
> [{TopicMetadata for topic test1 ->
> No partition metadata for topic test1 due to
> kafka.common.LeaderNotAvailableException}] for topic [test1]: class
> kafka.common.LeaderNotAvailableException (kafka.pr
> oducer.BrokerPartitionInfo)
> [2014-03-11 07:36:35,368] WARN Error while fetching metadata
> [{TopicMetadata for topic test1 ->
> No partition metadata for topic test1 due to
> kafka.common.LeaderNotAvailableException}] for topic [test1]: class
> kafka.common.LeaderNotAvailableException (kafka.pr
> oducer.BrokerPartitionInfo)
> [2014-03-11 07:36:35,368] ERROR Failed to collate messages by topic,
> partition due to: Failed to fetch topic metadata for topic: test1
> (kafka.producer.async.Default
> EventHandler)
> [2014-03-11 07:36:35,476] WARN Error while fetching metadata
> [{TopicMetadata for topic test1 ->
> No partition metadata for topic test1 due to
> kafka.common.LeaderNotAvailableException}] for topic [test1]: class
> kafka.common.LeaderNotAvailableException (kafka.pr
> oducer.BrokerPartitionInfo)
> [2014-03-11 07:36:35,490] WARN Error while fetching metadata
> [{TopicMetadata for topic test1 ->
> No partition metadata for topic test1 due to
> kafka.common.LeaderNotAvailableException}] for topic [test1]: class
> kafka.common.LeaderNotAvailableException (kafka.pr
> oducer.BrokerPartitionInfo)
> [2014-03-11 07:36:35,490] ERROR Failed to collate messages by topic,
> partition due to: Failed to fetch topic metadata for topic: test1
> (kafka.producer.async.Default
> EventHandler)
> [2014-03-11 07:36:35,599] WARN Error while fetching metadata
> [{TopicMetadata for topic test1 ->
> No partition metadata for topic test1 due to
> kafka.common.LeaderNotAvailableException}] for topic [test1]: class
> kafka.common.LeaderNotAvailableException (kafka.pr
> oducer.BrokerPartitionInfo)
> [2014-03-11 07:36:35,609] WARN Error while fetching metadata
> [{TopicMetadata for topic test1 ->
> No partition metadata for topic test1 due to
> kafka.common.LeaderNotAvailableException}] for topic [test1]: class
> kafka.common.LeaderNotAvailableException (kafka.pr
> oducer.BrokerPartitionInfo)
> [2014-03-11 07:36:35,609] ERROR Failed to collate messages by topic,
> partition due to: Failed to fetch topic metadata for topic: test1
> (kafka.producer.async.Default
> EventHandler)
> [2014-03-11 07:36:35,721] WARN Error while fetching metadata
> [{TopicMetadata for topic test1 ->
> No partition metadata for topic test1 due to
> kafka.common.LeaderNotAvailableException}] for topic [test1]: class
> kafka.common.LeaderNotAvailableException (kafka.pr
> oducer.BrokerPartitionInfo)
> [2014-03-11 07:36:35,723] ERROR Failed to send requests for topics test1
> with correlation ids in [0,8] (kafka.producer.async.DefaultEventHandler)
> [2014-03-11 07:36:35,725] ERROR Error in handling batch of 1 events
> (kafka.producer.async.ProducerSendThread)
> kafka.common.FailedToSendMessageException: Failed to send messages after 3
> tries.
> at
> kafka.producer.async.DefaultEventHandler.handle(DefaultEventHandler.scala:90)
> at
> kafka.producer.async.ProducerSendThread.tryToHandle(ProducerSendThread.scala:104)
> at
> kafka.producer.async.ProducerSendThread$$anonfun$processEvents$3.apply(ProducerSendThread.scala:87)
> at
> kafka.producer.async.ProducerSendThread$$anonfun$processEvents$3.apply(ProducerSendThread.scala:67)
> at scala.collection.immutable.Stream.foreach(Stream.scala:254)
> at
> kafka.producer.async.ProducerSendThread.processEvents(ProducerSendThread.scala:66)
> at
> kafka.producer.async.ProducerSendThread.run(ProducerSendThread.scala:44)
>
> What am I missing?
>
> > From: andthereitgoes@hotmail.com <javascript:;>
> > To: users@kafka.apache.org <javascript:;>
> > Subject: RE: Remote Zookeeper
> > Date: Tue, 11 Mar 2014 02:29:34 +0000
> >
> > Thanks. Totally missed that.
> >
> > > From: b@b3k.us <javascript:;>
> > > Date: Mon, 10 Mar 2014 19:18:50 -0700
> > > Subject: Re: Remote Zookeeper
> > > To: users@kafka.apache.org <javascript:;>
> > >
> > > zookeeper.connect
> > >
> > > https://kafka.apache.org/08/configuration.html
> > >
> > >
> > > On Mon, Mar 10, 2014 at 7:17 PM, A A <andthereitgoes@hotmail.com<javascript:;>>
> wrote:
> > >
> > > > Hi
> > > >
> > > > Pretty new to Kafka. Have been successful in installing Kafka 0.8.0.
> > > > I am just wondering how should I make my kafka cluster (2 brokers)
> connect
> > > > to a single remote zookeper server?
> > > >
> > > > I am using $KAFKA/kafka-server-start.sh
> $KAFKA_CONFIG/server.properties
> > > > on both the brokers to start them up
> > > > and I have a remote zookeeper running at 192.168.1.120
> > > >
> > > > Can anyone help?
> > > >
> > > > A
> > > >
> > > >
> >
>
RE: Remote Zookeeper
Posted by A A <an...@hotmail.com>.
Hi again.
Got the setup working. I now have 2 brokers (broker 1 and broker 2) with one remote zk. I was also able to create some topics
$KAFKA_HOME/bin/kafka-list-topic.sh --zookeeper 192.168.1.120:2181
topic: test partition: 0 leader: 1 replicas: 1 isr: 1
topic: test1 partition: 0 leader: 1 replicas: 1 isr: 1
I then try to publish a message from broker1 to broker 1 (localhost) while I am consuming on broker2
$KAFKA_HOME/bin/kafka-console-producer.sh --broker-list localhost:9092 --topic test1
helloooo
[2014-03-11 07:36:35,206] WARN Error while fetching metadata [{TopicMetadata for topic test1 ->
No partition metadata for topic test1 due to kafka.common.LeaderNotAvailableException}] for topic [test1]: class kafka.common.LeaderNotAvailableException (kafka.pr oducer.BrokerPartitionInfo)
[2014-03-11 07:36:35,244] WARN Error while fetching metadata [{TopicMetadata for topic test1 ->
No partition metadata for topic test1 due to kafka.common.LeaderNotAvailableException}] for topic [test1]: class kafka.common.LeaderNotAvailableException (kafka.pr oducer.BrokerPartitionInfo)
[2014-03-11 07:36:35,248] ERROR Failed to collate messages by topic, partition due to: Failed to fetch topic metadata for topic: test1 (kafka.producer.async.Default EventHandler)
[2014-03-11 07:36:35,358] WARN Error while fetching metadata [{TopicMetadata for topic test1 ->
No partition metadata for topic test1 due to kafka.common.LeaderNotAvailableException}] for topic [test1]: class kafka.common.LeaderNotAvailableException (kafka.pr oducer.BrokerPartitionInfo)
[2014-03-11 07:36:35,368] WARN Error while fetching metadata [{TopicMetadata for topic test1 ->
No partition metadata for topic test1 due to kafka.common.LeaderNotAvailableException}] for topic [test1]: class kafka.common.LeaderNotAvailableException (kafka.pr oducer.BrokerPartitionInfo)
[2014-03-11 07:36:35,368] ERROR Failed to collate messages by topic, partition due to: Failed to fetch topic metadata for topic: test1 (kafka.producer.async.Default EventHandler)
[2014-03-11 07:36:35,476] WARN Error while fetching metadata [{TopicMetadata for topic test1 ->
No partition metadata for topic test1 due to kafka.common.LeaderNotAvailableException}] for topic [test1]: class kafka.common.LeaderNotAvailableException (kafka.pr oducer.BrokerPartitionInfo)
[2014-03-11 07:36:35,490] WARN Error while fetching metadata [{TopicMetadata for topic test1 ->
No partition metadata for topic test1 due to kafka.common.LeaderNotAvailableException}] for topic [test1]: class kafka.common.LeaderNotAvailableException (kafka.pr oducer.BrokerPartitionInfo)
[2014-03-11 07:36:35,490] ERROR Failed to collate messages by topic, partition due to: Failed to fetch topic metadata for topic: test1 (kafka.producer.async.Default EventHandler)
[2014-03-11 07:36:35,599] WARN Error while fetching metadata [{TopicMetadata for topic test1 ->
No partition metadata for topic test1 due to kafka.common.LeaderNotAvailableException}] for topic [test1]: class kafka.common.LeaderNotAvailableException (kafka.pr oducer.BrokerPartitionInfo)
[2014-03-11 07:36:35,609] WARN Error while fetching metadata [{TopicMetadata for topic test1 ->
No partition metadata for topic test1 due to kafka.common.LeaderNotAvailableException}] for topic [test1]: class kafka.common.LeaderNotAvailableException (kafka.pr oducer.BrokerPartitionInfo)
[2014-03-11 07:36:35,609] ERROR Failed to collate messages by topic, partition due to: Failed to fetch topic metadata for topic: test1 (kafka.producer.async.Default EventHandler)
[2014-03-11 07:36:35,721] WARN Error while fetching metadata [{TopicMetadata for topic test1 ->
No partition metadata for topic test1 due to kafka.common.LeaderNotAvailableException}] for topic [test1]: class kafka.common.LeaderNotAvailableException (kafka.pr oducer.BrokerPartitionInfo)
[2014-03-11 07:36:35,723] ERROR Failed to send requests for topics test1 with correlation ids in [0,8] (kafka.producer.async.DefaultEventHandler)
[2014-03-11 07:36:35,725] ERROR Error in handling batch of 1 events (kafka.producer.async.ProducerSendThread)
kafka.common.FailedToSendMessageException: Failed to send messages after 3 tries.
at kafka.producer.async.DefaultEventHandler.handle(DefaultEventHandler.scala:90)
at kafka.producer.async.ProducerSendThread.tryToHandle(ProducerSendThread.scala:104)
at kafka.producer.async.ProducerSendThread$$anonfun$processEvents$3.apply(ProducerSendThread.scala:87)
at kafka.producer.async.ProducerSendThread$$anonfun$processEvents$3.apply(ProducerSendThread.scala:67)
at scala.collection.immutable.Stream.foreach(Stream.scala:254)
at kafka.producer.async.ProducerSendThread.processEvents(ProducerSendThread.scala:66)
at kafka.producer.async.ProducerSendThread.run(ProducerSendThread.scala:44)
What am I missing?
> From: andthereitgoes@hotmail.com
> To: users@kafka.apache.org
> Subject: RE: Remote Zookeeper
> Date: Tue, 11 Mar 2014 02:29:34 +0000
>
> Thanks. Totally missed that.
>
> > From: b@b3k.us
> > Date: Mon, 10 Mar 2014 19:18:50 -0700
> > Subject: Re: Remote Zookeeper
> > To: users@kafka.apache.org
> >
> > zookeeper.connect
> >
> > https://kafka.apache.org/08/configuration.html
> >
> >
> > On Mon, Mar 10, 2014 at 7:17 PM, A A <an...@hotmail.com> wrote:
> >
> > > Hi
> > >
> > > Pretty new to Kafka. Have been successful in installing Kafka 0.8.0.
> > > I am just wondering how should I make my kafka cluster (2 brokers) connect
> > > to a single remote zookeper server?
> > >
> > > I am using $KAFKA/kafka-server-start.sh $KAFKA_CONFIG/server.properties
> > > on both the brokers to start them up
> > > and I have a remote zookeeper running at 192.168.1.120
> > >
> > > Can anyone help?
> > >
> > > A
> > >
> > >
>
RE: Remote Zookeeper
Posted by A A <an...@hotmail.com>.
Thanks. Totally missed that.
> From: b@b3k.us
> Date: Mon, 10 Mar 2014 19:18:50 -0700
> Subject: Re: Remote Zookeeper
> To: users@kafka.apache.org
>
> zookeeper.connect
>
> https://kafka.apache.org/08/configuration.html
>
>
> On Mon, Mar 10, 2014 at 7:17 PM, A A <an...@hotmail.com> wrote:
>
> > Hi
> >
> > Pretty new to Kafka. Have been successful in installing Kafka 0.8.0.
> > I am just wondering how should I make my kafka cluster (2 brokers) connect
> > to a single remote zookeper server?
> >
> > I am using $KAFKA/kafka-server-start.sh $KAFKA_CONFIG/server.properties
> > on both the brokers to start them up
> > and I have a remote zookeeper running at 192.168.1.120
> >
> > Can anyone help?
> >
> > A
> >
> >
Re: Remote Zookeeper
Posted by Benjamin Black <b...@b3k.us>.
zookeeper.connect
https://kafka.apache.org/08/configuration.html
On Mon, Mar 10, 2014 at 7:17 PM, A A <an...@hotmail.com> wrote:
> Hi
>
> Pretty new to Kafka. Have been successful in installing Kafka 0.8.0.
> I am just wondering how should I make my kafka cluster (2 brokers) connect
> to a single remote zookeper server?
>
> I am using $KAFKA/kafka-server-start.sh $KAFKA_CONFIG/server.properties
> on both the brokers to start them up
> and I have a remote zookeeper running at 192.168.1.120
>
> Can anyone help?
>
> A
>
>