You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by Mich Talebzadeh <mi...@gmail.com> on 2018/07/16 14:41:08 UTC
Zookeeper and Kafka cluster docker issue
Hi,
I have created a zookeeper and three brokers as dockers in a physical host
as shown below
[image: image.png]
The followings are used to create Zookeeper and Kafka dockers
docker run -d --name zookeeper -p 2181:2181 -p 2888:2888 -p 3888:3888
jplock/zookeeper
docker run -d --name kafka_broker0 -p 9092:9092 -e
KAFKA_ADVERTISED_HOST_NAME=50.140.197.220 -e ZOOKEEPER_IP=50.140.197.220 -e
KAFKA_BROKER_ID=0 -e KAFKA_BROKER_PORT=9092 ches/kafka
docker run -d --name kafka_broker1 -p 9093:9092 -e
KAFKA_ADVERTISED_HOST_NAME=50.140.197.220 -e ZOOKEEPER_IP=50.140.197.220 -e
KAFKA_BROKER_ID=1 -e KAFKA_BROKER_PORT=9092 ches/kafka
docker run -d --name kafka_broker2 -p 9094:9092 -e
KAFKA_ADVERTISED_HOST_NAME=50.140.197.220 -e ZOOKEEPER_IP=50.140.197.220 -e
KAFKA_BROKER_ID=2 -e KAFKA_BROKER_PORT=9092 ches/kafka
Note the mappings of ports to the port on the physical host.
I have created the following topic that works
${KAFKA_HOME}/bin/kafka-topics.sh --create --zookeeper rhes75:2181
--replication-factor 1 --partitions 1 --topic three
${KAFKA_HOME}/bin/kafka-topics.sh --describe -zookeeper rhes75:2181 --topic
three
Topic:three PartitionCount:1 ReplicationFactor:1 Configs:
Topic: three Partition: 0 Leader: 0 Replicas: 0 Isr:
0
*So there is only one partition and one replication factor*
The following producer works fine
cat ${IN_FILE} | ${KAFKA_HOME}/bin/kafka-console-producer.sh --broker-list
rhes75:9092, rhes75:9093, rhes75:9094 --topic three
However, when I define a topic as follows with --replication-factor 2
--partitions 2
hduser@rhes564: /data6/hduser/prices/avg_prices> rhes75:2181
--replication-factor 2 --partitions 2 --topic
newone <
Created topic "newone".
hduser@rhes564: /data6/hduser/prices/avg_prices>
${KAFKA_HOME}/bin/kafka-topics.sh --describe -zookeeper rhes75:2181
--topic newone
Topic:newone PartitionCount:2 ReplicationFactor:2 Configs:
Topic: newone Partition: 0 Leader: 2 Replicas: 2,0
Isr: 2,0
Topic: newone Partition: 1 Leader: 0 Replicas: 0,1
Isr: 0
It throws errors!
[2018-07-16 15:51:40,852] WARN [Producer clientId=console-producer] Got
error produce response with correlation id 12 on topic-partition newone-0,
retrying (1 attempts left). Error: NOT_LEADER_FOR_PARTITION
(org.apache.kafka.clients.producer.internals.Sender)
[2018-07-16 15:51:40,955] WARN [Producer clientId=console-producer] Got
error produce response with correlation id 14 on topic-partition newone-0,
retrying (0 attempts left). Error: NOT_LEADER_FOR_PARTITION
(org.apache.kafka.clients.producer.internals.Sender)
[2018-07-16 15:51:41,056] ERROR Error when sending message to topic newone
with key: null, value: 67 bytes with error:
(org.apache.kafka.clients.producer.internals.ErrorLoggingCallback)
org.apache.kafka.common.errors.NotLeaderForPartitionException: This server
is not the leader for that topic-partition.
[2018-07-16 15:51:41,059] ERROR Error when sending message to topic newone
with key: null, value: 67 bytes with error:
(org.apache.kafka.clients.producer.internals.ErrorLoggingCallback)
org.apache.kafka.common.errors.NotLeaderForPartitionException: This server
is not the leader for that topic-partition.
[2018-07-16 15:51:41,059] ERROR Error when sending message to topic newone
with key: null, value: 68 bytes with error:
(org.apache.kafka.clients.producer.internals.ErrorLoggingCallback)
org.apache.kafka.common.errors.NotLeaderForPartitionException: This server
is not the leader for that topic-partition.
[2018-07-16 15:51:41,060] ERROR Error when sending message to topic newone
with key: null, value: 67 bytes with error:
(org.apache.kafka.clients.producer.internals.ErrorLoggingCallback)
org.apache.kafka.common.errors.NotLeaderForPartitionException: This server
is not the leader for that topic-partition.
[2018-07-16 15:51:41,060] ERROR Error when sending message to topic newone
with key: null, value: 67 bytes with error:
(org.apache.kafka.clients.producer.internals.ErrorLoggingCallback)
org.apache.kafka.common.errors.NotLeaderForPartitionException: This server
is not the leader for that topic-partition.
I believe these Kafka brokers have problem talking to each other and the
message is lost!
Thanks
Dr Mich Talebzadeh
LinkedIn * https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
<https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>*
http://talebzadehmich.wordpress.com
*Disclaimer:* Use it at your own risk. Any and all responsibility for any
loss, damage or destruction of data or any other property which may arise
from relying on this email's technical content is explicitly disclaimed.
The author will in no case be liable for any monetary damages arising from
such loss, damage or destruction.
Re: Zookeeper and Kafka cluster docker issue
Posted by Mich Talebzadeh <mi...@gmail.com>.
Thanks Chris,
This is the way I gave defined Kafka brokers
docker run -d *--name kafka_broker0* -p *9092*:9092 -e
KAFKA_ADVERTISED_HOST_NAME=50.140.197.220 -e
ZOOKEEPER_IP=50.140.197.220 -e *KAFKA_BROKER_ID=0
*-e KAFKA_BROKER_PORT=9092 -e *KAFKA_ADVERTISED_PORT=9092* ches/kafka
docker run -d *--name kafka_broker1 * -p *9093*:9092 -e
KAFKA_ADVERTISED_HOST_NAME=50.140.197.220 -e
ZOOKEEPER_IP=50.140.197.220 -e *KAFKA_BROKER_ID=1
*-e KAFKA_BROKER_PORT=9092 -e *KAFKA_ADVERTISED_PORT=9092* ches/kafka
docker run -d *--name kafka_broker2* -p *9094*:9092 -e
KAFKA_ADVERTISED_HOST_NAME=50.140.197.220 -e ZOOKEEPER_IP=50.140.197.220 -e
*KAFKA_BROKER_ID=2* -e KAFKA_BROKER_PORT=9092 -e
*KAFKA_ADVERTISED_PORT=9092* ches/kafka
Now Broker ID 0 is the current leader
${KAFKA_HOME}/bin/kafka-topics.sh --describe -zookeeper rhes75:2181
--topic final
Topic:final PartitionCount:3 ReplicationFactor:3 Configs:
Topic: final Partition: 0 Leader: 0 Replicas: 1,2,0
Isr: 0
Topic: final Partition: 1 Leader: 0 Replicas: 2,0,1
Isr: 0
Topic: final Partition: 2 Leader: 0 Replicas: 0,1,2
Isr: 0
Is my assumption correct that the leader will remain the same until in this
case Broker 0 fails?
Dr Mich Talebzadeh
LinkedIn * https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
<https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>*
http://talebzadehmich.wordpress.com
*Disclaimer:* Use it at your own risk. Any and all responsibility for any
loss, damage or destruction of data or any other property which may arise
from relying on this email's technical content is explicitly disclaimed.
The author will in no case be liable for any monetary damages arising from
such loss, damage or destruction.
On Mon, 16 Jul 2018 at 21:44, Chris Richardson <ch...@chrisrichardson.net>
wrote:
> Could it be that you changed the KAFKA_ADVERTISED_PORT and restarted those
> brokers but didn't restart the rest (until now)?
> I wouldn't be surprised if the other brokers continued to use the incorrect
> advertised port.
>
> On Mon, Jul 16, 2018 at 1:40 PM, Mich Talebzadeh <
> mich.talebzadeh@gmail.com>
> wrote:
>
> > I restarted all Kafka dockers corresponding to broker 0-2 and now broker
> ID
> > 0 is selected as leader and is working
> >
> > ${KAFKA_HOME}/bin/kafka-topics.sh --describe -zookeeper rhes75:2181
> > --topic final
> > Topic:final PartitionCount:3 ReplicationFactor:3
> > Configs:
> > Topic: final Partition: 0 Leader: 0 Replicas: 1,2,0
> > Isr: 0
> > Topic: final Partition: 1 Leader: 0 Replicas: 2,0,1
> > Isr: 0
> > Topic: final Partition: 2 Leader: 0 Replicas: 0,1,2
> > Isr: 0
> >
> > That is a good one with Leader being 0 in all. But this leader selection
> > was not working
> >
> > ${KAFKA_HOME}/bin/kafka-topics.sh --describe -zookeeper rhes75:2181
> > --topic final
> > Topic:final PartitionCount:3 ReplicationFactor:3
> > Configs:
> > Topic: final Partition: 0 Leader: 1 Replicas: 1,2,0
> > Isr: 1,2,0
> > Topic: final Partition: 1 Leader: 2 Replicas: 2,0,1
> > Isr: 2,0,1
> > Topic: final Partition: 2 Leader: 0 Replicas: 0,1,2
> > Isr: 0
> >
> > Surely something is not working here although there reports that the
> > problem goes away when brokers are restarted!
> >
> > Thanks
> >
> > Dr Mich Talebzadeh
> >
> >
> >
> > LinkedIn * https://www.linkedin.com/profile/view?id=
> > AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
> > <https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCd
> > OABUrV8Pw>*
> >
> >
> >
> > http://talebzadehmich.wordpress.com
> >
> >
> > *Disclaimer:* Use it at your own risk. Any and all responsibility for any
> > loss, damage or destruction of data or any other property which may arise
> > from relying on this email's technical content is explicitly disclaimed.
> > The author will in no case be liable for any monetary damages arising
> from
> > such loss, damage or destruction.
> >
> >
> >
> >
> > On Mon, 16 Jul 2018 at 19:57, Mich Talebzadeh <mich.talebzadeh@gmail.com
> >
> > wrote:
> >
> > > Also I noticed that bar broker ID =0, the connection to broker ID 1
> )node
> > > 1) and broker ID 2 (node 2) could not be established
> > >
> > > [2018-07-16 18:41:10,419] WARN [Producer clientId=console-producer]
> > > Connection to node 1 could not be established. Broker may not be
> > available.
> > > (org.apache.kafka.clients.NetworkClient)
> > > [2018-07-16 18:41:10,420] WARN [Producer clientId=console-producer]
> > > Connection to node 2 could not be established. Broker may not be
> > available.
> > > (org.apache.kafka.clients.NetworkClient)
> > > [2018-07-16 18:41:10,464] WARN [Producer clientId=console-producer]
> > > Connection to node 1 could not be established. Broker may not be
> > available.
> > > (org.apache.kafka.clients.NetworkClient)
> > > [2018-07-16 18:41:10,470] WARN [Producer clientId=console-producer]
> > > Connection to node 2 could not be established. Broker may not be
> > available.
> > > (org.apache.kafka.clients.NetworkClient)
> > > [2018-07-16 18:41:10,561] WARN [Producer clientId=console-producer]
> > > Connection to node 2 could not be established. Broker may not be
> > available.
> > > (org.apache.kafka.clients.NetworkClient)
> > > [2018-07-16 18:41:10,563] WARN [Producer clientId=console-producer]
> > > Connection to node 1 could not be established. Broker may not be
> > available.
> > > (org.apache.kafka.clients.NetworkClient)
> > >
> > > Node 1 port 9092 is mapped to host port 9093 and node 2 port 9092 is
> > > mapped to host port 9094
> > >
> > >
> > >
> > > Dr Mich Talebzadeh
> > >
> > >
> > >
> > > LinkedIn * https://www.linkedin.com/profile/view?id=
> > AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
> > > <
> https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCd
> > OABUrV8Pw>*
> > >
> > >
> > >
> > > http://talebzadehmich.wordpress.com
> > >
> > >
> > > *Disclaimer:* Use it at your own risk. Any and all responsibility for
> any
> > > loss, damage or destruction of data or any other property which may
> arise
> > > from relying on this email's technical content is explicitly
> disclaimed.
> > > The author will in no case be liable for any monetary damages arising
> > from
> > > such loss, damage or destruction.
> > >
> > >
> > >
> > >
> > > On Mon, 16 Jul 2018 at 16:49, Mich Talebzadeh <
> mich.talebzadeh@gmail.com
> > >
> > > wrote:
> > >
> > >> Thanks Chris,
> > >>
> > >> I am afraid the issue is still there!
> > >>
> > >> docker run -d --name kafka_broker0 -p 9092:9092 -e
> > >> KAFKA_ADVERTISED_HOST_NAME=50.140.197.220 -e
> > ZOOKEEPER_IP=50.140.197.220 -e
> > >> KAFKA_BROKER_ID=0 -e KAFKA_BROKER_PORT=9092 -e
> > *KAFKA_ADVERTISED_PORT=9092
> > >> *ches/kafka
> > >>
> > >> ${KAFKA_HOME}/bin/kafka-topics.sh --create --zookeeper rhes75:2181
> > >> --replication-factor 3 --partitions 3 --topic r3p3
> > >>
> > >> Created topic "r3p3".
> > >>
> > >> ${KAFKA_HOME}/bin/kafka-topics.sh --describe -zookeeper rhes75:2181
> > --topic
> > >> r3p3
> > >>
> > >> Topic:r3p3 PartitionCount:3 ReplicationFactor:3
> Configs:
> > >>
> > >> Topic: r3p3 Partition: 0 Leader: 2 Replicas:
> 2,1,0
> > >> Isr: 2,1,0
> > >>
> > >> Topic: r3p3 Partition: 1 Leader: 0 Replicas:
> 0,2,1
> > >> Isr: 0
> > >>
> > >> Topic: r3p3 Partition: 2 Leader: 1 Replicas:
> 1,0,2
> > >> Isr: 1,0,2
> > >>
> > >> cat ${IN_FILE} | ${KAFKA_HOME}/bin/kafka-console-producer.sh
> > >> --broker-list rhes75:9092, rhes75:9093, rhes75:9094 --topic r3p3
> > >>
> > >> [2018-07-16 17:01:34,496] WARN [Producer clientId=console-producer]
> Got
> > >> error produce response with correlation id 10 on topic-partition
> r3p3-0,
> > >> retrying (2 attempts left). Error: NOT_LEADER_FOR_PARTITION
> > >> (org.apache.kafka.clients.producer.internals.Sender)
> > >> [2018-07-16 17:01:34,596] WARN [Producer clientId=console-producer]
> Got
> > >> error produce response with correlation id 13 on topic-partition
> r3p3-2,
> > >> retrying (1 attempts left). Error: NOT_LEADER_FOR_PARTITION
> > >> (org.apache.kafka.clients.producer.internals.Sender)
> > >> [2018-07-16 17:01:34,597] WARN [Producer clientId=console-producer]
> Got
> > >> error produce response with correlation id 14 on topic-partition
> r3p3-0,
> > >> retrying (1 attempts left). Error: NOT_LEADER_FOR_PARTITION
> > >> (org.apache.kafka.clients.producer.internals.Sender)
> > >> [2018-07-16 17:01:34,698] WARN [Producer clientId=console-producer]
> Got
> > >> error produce response with correlation id 16 on topic-partition
> r3p3-2,
> > >> retrying (0 attempts left). Error: NOT_LEADER_FOR_PARTITION
> > >> (org.apache.kafka.clients.producer.internals.Sender)
> > >> [2018-07-16 17:01:34,699] WARN [Producer clientId=console-producer]
> Got
> > >> error produce response with correlation id 17 on topic-partition
> r3p3-0,
> > >> retrying (0 attempts left). Error: NOT_LEADER_FOR_PARTITION
> > >> (org.apache.kafka.clients.producer.internals.Sender)
> > >> [2018-07-16 17:01:34,800] ERROR Error when sending message to topic
> r3p3
> > >> with key: null, value: 67 bytes with error:
> > >> (org.apache.kafka.clients.producer.internals.ErrorLoggingCallback)
> > >>
> > >> Dr Mich Talebzadeh
> > >>
> > >>
> > >>
> > >> LinkedIn * https://www.linkedin.com/profile/view?id=
> > AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
> > >> <https://www.linkedin.com/profile/view?id=
> > AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>*
> > >>
> > >>
> > >>
> > >> http://talebzadehmich.wordpress.com
> > >>
> > >>
> > >> *Disclaimer:* Use it at your own risk. Any and all responsibility for
> > >> any loss, damage or destruction of data or any other property which
> may
> > >> arise from relying on this email's technical content is explicitly
> > >> disclaimed. The author will in no case be liable for any monetary
> > damages
> > >> arising from such loss, damage or destruction.
> > >>
> > >>
> > >>
> > >>
> > >> On Mon, 16 Jul 2018 at 16:12, Chris Richardson <
> > chris@chrisrichardson.net>
> > >> wrote:
> > >>
> > >>> I believe you need to use -e KAFKA_ADVERTISED_PORT=909..
> > >>>
> > >>> On Mon, Jul 16, 2018 at 7:41 AM, Mich Talebzadeh <
> > >>> mich.talebzadeh@gmail.com>
> > >>> wrote:
> > >>>
> > >>> > Hi,
> > >>> >
> > >>> > I have created a zookeeper and three brokers as dockers in a
> physical
> > >>> host
> > >>> > as shown below
> > >>> >
> > >>> > [image: image.png]
> > >>> >
> > >>> > The followings are used to create Zookeeper and Kafka dockers
> > >>> >
> > >>> > docker run -d --name zookeeper -p 2181:2181 -p 2888:2888 -p
> 3888:3888
> > >>> > jplock/zookeeper
> > >>> >
> > >>> > docker run -d --name kafka_broker0 -p 9092:9092 -e
> > >>> > KAFKA_ADVERTISED_HOST_NAME=50.140.197.220 -e
> > >>> ZOOKEEPER_IP=50.140.197.220
> > >>> > -e KAFKA_BROKER_ID=0 -e KAFKA_BROKER_PORT=9092 ches/kafka
> > >>> >
> > >>> > docker run -d --name kafka_broker1 -p 9093:9092 -e
> > >>> > KAFKA_ADVERTISED_HOST_NAME=50.140.197.220 -e
> > >>> ZOOKEEPER_IP=50.140.197.220
> > >>> > -e KAFKA_BROKER_ID=1 -e KAFKA_BROKER_PORT=9092 ches/kafka
> > >>> >
> > >>> > docker run -d --name kafka_broker2 -p 9094:9092 -e
> > >>> > KAFKA_ADVERTISED_HOST_NAME=50.140.197.220 -e
> > >>> ZOOKEEPER_IP=50.140.197.220
> > >>> > -e KAFKA_BROKER_ID=2 -e KAFKA_BROKER_PORT=9092 ches/kafka
> > >>> >
> > >>> > Note the mappings of ports to the port on the physical host.
> > >>> >
> > >>> > I have created the following topic that works
> > >>> >
> > >>> > ${KAFKA_HOME}/bin/kafka-topics.sh --create --zookeeper rhes75:2181
> > >>> > --replication-factor 1 --partitions 1 --topic three
> > >>> >
> > >>> >
> > >>> > ${KAFKA_HOME}/bin/kafka-topics.sh --describe -zookeeper rhes75:2181
> > >>> --topic
> > >>> > three
> > >>> >
> > >>> > Topic:three PartitionCount:1 ReplicationFactor:1
> > >>> Configs:
> > >>> >
> > >>> > Topic: three Partition: 0 Leader: 0 Replicas: 0
> > >>> Isr:
> > >>> > 0
> > >>> >
> > >>> > *So there is only one partition and one replication factor*
> > >>> >
> > >>> > The following producer works fine
> > >>> >
> > >>> > cat ${IN_FILE} | ${KAFKA_HOME}/bin/kafka-console-producer.sh
> > >>> > --broker-list rhes75:9092, rhes75:9093, rhes75:9094 --topic three
> > >>> >
> > >>> > However, when I define a topic as follows with
> --replication-factor 2
> > >>> > --partitions 2
> > >>> >
> > >>> > hduser@rhes564: /data6/hduser/prices/avg_prices> rhes75:2181
> > >>> > --replication-factor 2 --partitions 2 --topic newone
> > >>> > <
> > >>> > Created topic "newone".
> > >>> > hduser@rhes564: /data6/hduser/prices/avg_prices>
> > >>> ${KAFKA_HOME}/bin/kafka-topics.sh
> > >>> > --describe -zookeeper rhes75:2181 --topic newone
> > >>> > Topic:newone PartitionCount:2 ReplicationFactor:2
> > >>> Configs:
> > >>> > Topic: newone Partition: 0 Leader: 2 Replicas:
> 2,0
> > >>> > Isr: 2,0
> > >>> > Topic: newone Partition: 1 Leader: 0 Replicas:
> 0,1
> > >>> > Isr: 0
> > >>> >
> > >>> > It throws errors!
> > >>> >
> > >>> > [2018-07-16 15:51:40,852] WARN [Producer clientId=console-producer]
> > Got
> > >>> > error produce response with correlation id 12 on topic-partition
> > >>> newone-0,
> > >>> > retrying (1 attempts left). Error: NOT_LEADER_FOR_PARTITION
> > >>> > (org.apache.kafka.clients.producer.internals.Sender)
> > >>> > [2018-07-16 15:51:40,955] WARN [Producer clientId=console-producer]
> > Got
> > >>> > error produce response with correlation id 14 on topic-partition
> > >>> newone-0,
> > >>> > retrying (0 attempts left). Error: NOT_LEADER_FOR_PARTITION
> > >>> > (org.apache.kafka.clients.producer.internals.Sender)
> > >>> > [2018-07-16 15:51:41,056] ERROR Error when sending message to topic
> > >>> newone
> > >>> > with key: null, value: 67 bytes with error:
> > (org.apache.kafka.clients.
> > >>> > producer.internals.ErrorLoggingCallback)
> > >>> > org.apache.kafka.common.errors.NotLeaderForPartitionException: This
> > >>> > server is not the leader for that topic-partition.
> > >>> > [2018-07-16 15:51:41,059] ERROR Error when sending message to topic
> > >>> newone
> > >>> > with key: null, value: 67 bytes with error:
> > (org.apache.kafka.clients.
> > >>> > producer.internals.ErrorLoggingCallback)
> > >>> > org.apache.kafka.common.errors.NotLeaderForPartitionException: This
> > >>> > server is not the leader for that topic-partition.
> > >>> > [2018-07-16 15:51:41,059] ERROR Error when sending message to topic
> > >>> newone
> > >>> > with key: null, value: 68 bytes with error:
> > (org.apache.kafka.clients.
> > >>> > producer.internals.ErrorLoggingCallback)
> > >>> > org.apache.kafka.common.errors.NotLeaderForPartitionException: This
> > >>> > server is not the leader for that topic-partition.
> > >>> > [2018-07-16 15:51:41,060] ERROR Error when sending message to topic
> > >>> newone
> > >>> > with key: null, value: 67 bytes with error:
> > (org.apache.kafka.clients.
> > >>> > producer.internals.ErrorLoggingCallback)
> > >>> > org.apache.kafka.common.errors.NotLeaderForPartitionException: This
> > >>> > server is not the leader for that topic-partition.
> > >>> > [2018-07-16 15:51:41,060] ERROR Error when sending message to topic
> > >>> newone
> > >>> > with key: null, value: 67 bytes with error:
> > (org.apache.kafka.clients.
> > >>> > producer.internals.ErrorLoggingCallback)
> > >>> > org.apache.kafka.common.errors.NotLeaderForPartitionException: This
> > >>> > server is not the leader for that topic-partition.
> > >>> >
> > >>> > I believe these Kafka brokers have problem talking to each other
> and
> > >>> the
> > >>> > message is lost!
> > >>> >
> > >>> > Thanks
> > >>> >
> > >>> > Dr Mich Talebzadeh
> > >>> >
> > >>> >
> > >>> >
> > >>> > LinkedIn *
> > >>> https://www.linkedin.com/profile/view?id=
> > AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
> > >>> > <
> > >>> https://www.linkedin.com/profile/view?id=
> > AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
> > >>> >*
> > >>> >
> > >>> >
> > >>> >
> > >>> > http://talebzadehmich.wordpress.com
> > >>> >
> > >>> >
> > >>> > *Disclaimer:* Use it at your own risk. Any and all responsibility
> for
> > >>> any
> > >>> > loss, damage or destruction of data or any other property which may
> > >>> arise
> > >>> > from relying on this email's technical content is explicitly
> > >>> disclaimed.
> > >>> > The author will in no case be liable for any monetary damages
> arising
> > >>> from
> > >>> > such loss, damage or destruction.
> > >>> >
> > >>> >
> > >>> >
> > >>>
> > >>>
> > >>>
> > >>> --
> > >>> Learn microservices - http://learn.microservices.io
> > >>> Microservices application platform http://eventuate.io
> > >>> Consulting and training http://chrisrichardson.net
> > >>>
> > >>
> >
>
>
>
> --
> Learn microservices - http://learn.microservices.io
> Microservices application platform http://eventuate.io
> Consulting and training http://chrisrichardson.net
>
Re: Zookeeper and Kafka cluster docker issue
Posted by Chris Richardson <ch...@chrisrichardson.net>.
Could it be that you changed the KAFKA_ADVERTISED_PORT and restarted those
brokers but didn't restart the rest (until now)?
I wouldn't be surprised if the other brokers continued to use the incorrect
advertised port.
On Mon, Jul 16, 2018 at 1:40 PM, Mich Talebzadeh <mi...@gmail.com>
wrote:
> I restarted all Kafka dockers corresponding to broker 0-2 and now broker ID
> 0 is selected as leader and is working
>
> ${KAFKA_HOME}/bin/kafka-topics.sh --describe -zookeeper rhes75:2181
> --topic final
> Topic:final PartitionCount:3 ReplicationFactor:3
> Configs:
> Topic: final Partition: 0 Leader: 0 Replicas: 1,2,0
> Isr: 0
> Topic: final Partition: 1 Leader: 0 Replicas: 2,0,1
> Isr: 0
> Topic: final Partition: 2 Leader: 0 Replicas: 0,1,2
> Isr: 0
>
> That is a good one with Leader being 0 in all. But this leader selection
> was not working
>
> ${KAFKA_HOME}/bin/kafka-topics.sh --describe -zookeeper rhes75:2181
> --topic final
> Topic:final PartitionCount:3 ReplicationFactor:3
> Configs:
> Topic: final Partition: 0 Leader: 1 Replicas: 1,2,0
> Isr: 1,2,0
> Topic: final Partition: 1 Leader: 2 Replicas: 2,0,1
> Isr: 2,0,1
> Topic: final Partition: 2 Leader: 0 Replicas: 0,1,2
> Isr: 0
>
> Surely something is not working here although there reports that the
> problem goes away when brokers are restarted!
>
> Thanks
>
> Dr Mich Talebzadeh
>
>
>
> LinkedIn * https://www.linkedin.com/profile/view?id=
> AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
> <https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCd
> OABUrV8Pw>*
>
>
>
> http://talebzadehmich.wordpress.com
>
>
> *Disclaimer:* Use it at your own risk. Any and all responsibility for any
> loss, damage or destruction of data or any other property which may arise
> from relying on this email's technical content is explicitly disclaimed.
> The author will in no case be liable for any monetary damages arising from
> such loss, damage or destruction.
>
>
>
>
> On Mon, 16 Jul 2018 at 19:57, Mich Talebzadeh <mi...@gmail.com>
> wrote:
>
> > Also I noticed that bar broker ID =0, the connection to broker ID 1 )node
> > 1) and broker ID 2 (node 2) could not be established
> >
> > [2018-07-16 18:41:10,419] WARN [Producer clientId=console-producer]
> > Connection to node 1 could not be established. Broker may not be
> available.
> > (org.apache.kafka.clients.NetworkClient)
> > [2018-07-16 18:41:10,420] WARN [Producer clientId=console-producer]
> > Connection to node 2 could not be established. Broker may not be
> available.
> > (org.apache.kafka.clients.NetworkClient)
> > [2018-07-16 18:41:10,464] WARN [Producer clientId=console-producer]
> > Connection to node 1 could not be established. Broker may not be
> available.
> > (org.apache.kafka.clients.NetworkClient)
> > [2018-07-16 18:41:10,470] WARN [Producer clientId=console-producer]
> > Connection to node 2 could not be established. Broker may not be
> available.
> > (org.apache.kafka.clients.NetworkClient)
> > [2018-07-16 18:41:10,561] WARN [Producer clientId=console-producer]
> > Connection to node 2 could not be established. Broker may not be
> available.
> > (org.apache.kafka.clients.NetworkClient)
> > [2018-07-16 18:41:10,563] WARN [Producer clientId=console-producer]
> > Connection to node 1 could not be established. Broker may not be
> available.
> > (org.apache.kafka.clients.NetworkClient)
> >
> > Node 1 port 9092 is mapped to host port 9093 and node 2 port 9092 is
> > mapped to host port 9094
> >
> >
> >
> > Dr Mich Talebzadeh
> >
> >
> >
> > LinkedIn * https://www.linkedin.com/profile/view?id=
> AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
> > <https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCd
> OABUrV8Pw>*
> >
> >
> >
> > http://talebzadehmich.wordpress.com
> >
> >
> > *Disclaimer:* Use it at your own risk. Any and all responsibility for any
> > loss, damage or destruction of data or any other property which may arise
> > from relying on this email's technical content is explicitly disclaimed.
> > The author will in no case be liable for any monetary damages arising
> from
> > such loss, damage or destruction.
> >
> >
> >
> >
> > On Mon, 16 Jul 2018 at 16:49, Mich Talebzadeh <mich.talebzadeh@gmail.com
> >
> > wrote:
> >
> >> Thanks Chris,
> >>
> >> I am afraid the issue is still there!
> >>
> >> docker run -d --name kafka_broker0 -p 9092:9092 -e
> >> KAFKA_ADVERTISED_HOST_NAME=50.140.197.220 -e
> ZOOKEEPER_IP=50.140.197.220 -e
> >> KAFKA_BROKER_ID=0 -e KAFKA_BROKER_PORT=9092 -e
> *KAFKA_ADVERTISED_PORT=9092
> >> *ches/kafka
> >>
> >> ${KAFKA_HOME}/bin/kafka-topics.sh --create --zookeeper rhes75:2181
> >> --replication-factor 3 --partitions 3 --topic r3p3
> >>
> >> Created topic "r3p3".
> >>
> >> ${KAFKA_HOME}/bin/kafka-topics.sh --describe -zookeeper rhes75:2181
> --topic
> >> r3p3
> >>
> >> Topic:r3p3 PartitionCount:3 ReplicationFactor:3 Configs:
> >>
> >> Topic: r3p3 Partition: 0 Leader: 2 Replicas: 2,1,0
> >> Isr: 2,1,0
> >>
> >> Topic: r3p3 Partition: 1 Leader: 0 Replicas: 0,2,1
> >> Isr: 0
> >>
> >> Topic: r3p3 Partition: 2 Leader: 1 Replicas: 1,0,2
> >> Isr: 1,0,2
> >>
> >> cat ${IN_FILE} | ${KAFKA_HOME}/bin/kafka-console-producer.sh
> >> --broker-list rhes75:9092, rhes75:9093, rhes75:9094 --topic r3p3
> >>
> >> [2018-07-16 17:01:34,496] WARN [Producer clientId=console-producer] Got
> >> error produce response with correlation id 10 on topic-partition r3p3-0,
> >> retrying (2 attempts left). Error: NOT_LEADER_FOR_PARTITION
> >> (org.apache.kafka.clients.producer.internals.Sender)
> >> [2018-07-16 17:01:34,596] WARN [Producer clientId=console-producer] Got
> >> error produce response with correlation id 13 on topic-partition r3p3-2,
> >> retrying (1 attempts left). Error: NOT_LEADER_FOR_PARTITION
> >> (org.apache.kafka.clients.producer.internals.Sender)
> >> [2018-07-16 17:01:34,597] WARN [Producer clientId=console-producer] Got
> >> error produce response with correlation id 14 on topic-partition r3p3-0,
> >> retrying (1 attempts left). Error: NOT_LEADER_FOR_PARTITION
> >> (org.apache.kafka.clients.producer.internals.Sender)
> >> [2018-07-16 17:01:34,698] WARN [Producer clientId=console-producer] Got
> >> error produce response with correlation id 16 on topic-partition r3p3-2,
> >> retrying (0 attempts left). Error: NOT_LEADER_FOR_PARTITION
> >> (org.apache.kafka.clients.producer.internals.Sender)
> >> [2018-07-16 17:01:34,699] WARN [Producer clientId=console-producer] Got
> >> error produce response with correlation id 17 on topic-partition r3p3-0,
> >> retrying (0 attempts left). Error: NOT_LEADER_FOR_PARTITION
> >> (org.apache.kafka.clients.producer.internals.Sender)
> >> [2018-07-16 17:01:34,800] ERROR Error when sending message to topic r3p3
> >> with key: null, value: 67 bytes with error:
> >> (org.apache.kafka.clients.producer.internals.ErrorLoggingCallback)
> >>
> >> Dr Mich Talebzadeh
> >>
> >>
> >>
> >> LinkedIn * https://www.linkedin.com/profile/view?id=
> AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
> >> <https://www.linkedin.com/profile/view?id=
> AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>*
> >>
> >>
> >>
> >> http://talebzadehmich.wordpress.com
> >>
> >>
> >> *Disclaimer:* Use it at your own risk. Any and all responsibility for
> >> any loss, damage or destruction of data or any other property which may
> >> arise from relying on this email's technical content is explicitly
> >> disclaimed. The author will in no case be liable for any monetary
> damages
> >> arising from such loss, damage or destruction.
> >>
> >>
> >>
> >>
> >> On Mon, 16 Jul 2018 at 16:12, Chris Richardson <
> chris@chrisrichardson.net>
> >> wrote:
> >>
> >>> I believe you need to use -e KAFKA_ADVERTISED_PORT=909..
> >>>
> >>> On Mon, Jul 16, 2018 at 7:41 AM, Mich Talebzadeh <
> >>> mich.talebzadeh@gmail.com>
> >>> wrote:
> >>>
> >>> > Hi,
> >>> >
> >>> > I have created a zookeeper and three brokers as dockers in a physical
> >>> host
> >>> > as shown below
> >>> >
> >>> > [image: image.png]
> >>> >
> >>> > The followings are used to create Zookeeper and Kafka dockers
> >>> >
> >>> > docker run -d --name zookeeper -p 2181:2181 -p 2888:2888 -p 3888:3888
> >>> > jplock/zookeeper
> >>> >
> >>> > docker run -d --name kafka_broker0 -p 9092:9092 -e
> >>> > KAFKA_ADVERTISED_HOST_NAME=50.140.197.220 -e
> >>> ZOOKEEPER_IP=50.140.197.220
> >>> > -e KAFKA_BROKER_ID=0 -e KAFKA_BROKER_PORT=9092 ches/kafka
> >>> >
> >>> > docker run -d --name kafka_broker1 -p 9093:9092 -e
> >>> > KAFKA_ADVERTISED_HOST_NAME=50.140.197.220 -e
> >>> ZOOKEEPER_IP=50.140.197.220
> >>> > -e KAFKA_BROKER_ID=1 -e KAFKA_BROKER_PORT=9092 ches/kafka
> >>> >
> >>> > docker run -d --name kafka_broker2 -p 9094:9092 -e
> >>> > KAFKA_ADVERTISED_HOST_NAME=50.140.197.220 -e
> >>> ZOOKEEPER_IP=50.140.197.220
> >>> > -e KAFKA_BROKER_ID=2 -e KAFKA_BROKER_PORT=9092 ches/kafka
> >>> >
> >>> > Note the mappings of ports to the port on the physical host.
> >>> >
> >>> > I have created the following topic that works
> >>> >
> >>> > ${KAFKA_HOME}/bin/kafka-topics.sh --create --zookeeper rhes75:2181
> >>> > --replication-factor 1 --partitions 1 --topic three
> >>> >
> >>> >
> >>> > ${KAFKA_HOME}/bin/kafka-topics.sh --describe -zookeeper rhes75:2181
> >>> --topic
> >>> > three
> >>> >
> >>> > Topic:three PartitionCount:1 ReplicationFactor:1
> >>> Configs:
> >>> >
> >>> > Topic: three Partition: 0 Leader: 0 Replicas: 0
> >>> Isr:
> >>> > 0
> >>> >
> >>> > *So there is only one partition and one replication factor*
> >>> >
> >>> > The following producer works fine
> >>> >
> >>> > cat ${IN_FILE} | ${KAFKA_HOME}/bin/kafka-console-producer.sh
> >>> > --broker-list rhes75:9092, rhes75:9093, rhes75:9094 --topic three
> >>> >
> >>> > However, when I define a topic as follows with --replication-factor 2
> >>> > --partitions 2
> >>> >
> >>> > hduser@rhes564: /data6/hduser/prices/avg_prices> rhes75:2181
> >>> > --replication-factor 2 --partitions 2 --topic newone
> >>> > <
> >>> > Created topic "newone".
> >>> > hduser@rhes564: /data6/hduser/prices/avg_prices>
> >>> ${KAFKA_HOME}/bin/kafka-topics.sh
> >>> > --describe -zookeeper rhes75:2181 --topic newone
> >>> > Topic:newone PartitionCount:2 ReplicationFactor:2
> >>> Configs:
> >>> > Topic: newone Partition: 0 Leader: 2 Replicas: 2,0
> >>> > Isr: 2,0
> >>> > Topic: newone Partition: 1 Leader: 0 Replicas: 0,1
> >>> > Isr: 0
> >>> >
> >>> > It throws errors!
> >>> >
> >>> > [2018-07-16 15:51:40,852] WARN [Producer clientId=console-producer]
> Got
> >>> > error produce response with correlation id 12 on topic-partition
> >>> newone-0,
> >>> > retrying (1 attempts left). Error: NOT_LEADER_FOR_PARTITION
> >>> > (org.apache.kafka.clients.producer.internals.Sender)
> >>> > [2018-07-16 15:51:40,955] WARN [Producer clientId=console-producer]
> Got
> >>> > error produce response with correlation id 14 on topic-partition
> >>> newone-0,
> >>> > retrying (0 attempts left). Error: NOT_LEADER_FOR_PARTITION
> >>> > (org.apache.kafka.clients.producer.internals.Sender)
> >>> > [2018-07-16 15:51:41,056] ERROR Error when sending message to topic
> >>> newone
> >>> > with key: null, value: 67 bytes with error:
> (org.apache.kafka.clients.
> >>> > producer.internals.ErrorLoggingCallback)
> >>> > org.apache.kafka.common.errors.NotLeaderForPartitionException: This
> >>> > server is not the leader for that topic-partition.
> >>> > [2018-07-16 15:51:41,059] ERROR Error when sending message to topic
> >>> newone
> >>> > with key: null, value: 67 bytes with error:
> (org.apache.kafka.clients.
> >>> > producer.internals.ErrorLoggingCallback)
> >>> > org.apache.kafka.common.errors.NotLeaderForPartitionException: This
> >>> > server is not the leader for that topic-partition.
> >>> > [2018-07-16 15:51:41,059] ERROR Error when sending message to topic
> >>> newone
> >>> > with key: null, value: 68 bytes with error:
> (org.apache.kafka.clients.
> >>> > producer.internals.ErrorLoggingCallback)
> >>> > org.apache.kafka.common.errors.NotLeaderForPartitionException: This
> >>> > server is not the leader for that topic-partition.
> >>> > [2018-07-16 15:51:41,060] ERROR Error when sending message to topic
> >>> newone
> >>> > with key: null, value: 67 bytes with error:
> (org.apache.kafka.clients.
> >>> > producer.internals.ErrorLoggingCallback)
> >>> > org.apache.kafka.common.errors.NotLeaderForPartitionException: This
> >>> > server is not the leader for that topic-partition.
> >>> > [2018-07-16 15:51:41,060] ERROR Error when sending message to topic
> >>> newone
> >>> > with key: null, value: 67 bytes with error:
> (org.apache.kafka.clients.
> >>> > producer.internals.ErrorLoggingCallback)
> >>> > org.apache.kafka.common.errors.NotLeaderForPartitionException: This
> >>> > server is not the leader for that topic-partition.
> >>> >
> >>> > I believe these Kafka brokers have problem talking to each other and
> >>> the
> >>> > message is lost!
> >>> >
> >>> > Thanks
> >>> >
> >>> > Dr Mich Talebzadeh
> >>> >
> >>> >
> >>> >
> >>> > LinkedIn *
> >>> https://www.linkedin.com/profile/view?id=
> AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
> >>> > <
> >>> https://www.linkedin.com/profile/view?id=
> AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
> >>> >*
> >>> >
> >>> >
> >>> >
> >>> > http://talebzadehmich.wordpress.com
> >>> >
> >>> >
> >>> > *Disclaimer:* Use it at your own risk. Any and all responsibility for
> >>> any
> >>> > loss, damage or destruction of data or any other property which may
> >>> arise
> >>> > from relying on this email's technical content is explicitly
> >>> disclaimed.
> >>> > The author will in no case be liable for any monetary damages arising
> >>> from
> >>> > such loss, damage or destruction.
> >>> >
> >>> >
> >>> >
> >>>
> >>>
> >>>
> >>> --
> >>> Learn microservices - http://learn.microservices.io
> >>> Microservices application platform http://eventuate.io
> >>> Consulting and training http://chrisrichardson.net
> >>>
> >>
>
--
Learn microservices - http://learn.microservices.io
Microservices application platform http://eventuate.io
Consulting and training http://chrisrichardson.net
Re: Zookeeper and Kafka cluster docker issue
Posted by Mich Talebzadeh <mi...@gmail.com>.
I restarted all Kafka dockers corresponding to broker 0-2 and now broker ID
0 is selected as leader and is working
${KAFKA_HOME}/bin/kafka-topics.sh --describe -zookeeper rhes75:2181
--topic final
Topic:final PartitionCount:3 ReplicationFactor:3
Configs:
Topic: final Partition: 0 Leader: 0 Replicas: 1,2,0
Isr: 0
Topic: final Partition: 1 Leader: 0 Replicas: 2,0,1
Isr: 0
Topic: final Partition: 2 Leader: 0 Replicas: 0,1,2
Isr: 0
That is a good one with Leader being 0 in all. But this leader selection
was not working
${KAFKA_HOME}/bin/kafka-topics.sh --describe -zookeeper rhes75:2181
--topic final
Topic:final PartitionCount:3 ReplicationFactor:3
Configs:
Topic: final Partition: 0 Leader: 1 Replicas: 1,2,0
Isr: 1,2,0
Topic: final Partition: 1 Leader: 2 Replicas: 2,0,1
Isr: 2,0,1
Topic: final Partition: 2 Leader: 0 Replicas: 0,1,2
Isr: 0
Surely something is not working here although there reports that the
problem goes away when brokers are restarted!
Thanks
Dr Mich Talebzadeh
LinkedIn * https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
<https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>*
http://talebzadehmich.wordpress.com
*Disclaimer:* Use it at your own risk. Any and all responsibility for any
loss, damage or destruction of data or any other property which may arise
from relying on this email's technical content is explicitly disclaimed.
The author will in no case be liable for any monetary damages arising from
such loss, damage or destruction.
On Mon, 16 Jul 2018 at 19:57, Mich Talebzadeh <mi...@gmail.com>
wrote:
> Also I noticed that bar broker ID =0, the connection to broker ID 1 )node
> 1) and broker ID 2 (node 2) could not be established
>
> [2018-07-16 18:41:10,419] WARN [Producer clientId=console-producer]
> Connection to node 1 could not be established. Broker may not be available.
> (org.apache.kafka.clients.NetworkClient)
> [2018-07-16 18:41:10,420] WARN [Producer clientId=console-producer]
> Connection to node 2 could not be established. Broker may not be available.
> (org.apache.kafka.clients.NetworkClient)
> [2018-07-16 18:41:10,464] WARN [Producer clientId=console-producer]
> Connection to node 1 could not be established. Broker may not be available.
> (org.apache.kafka.clients.NetworkClient)
> [2018-07-16 18:41:10,470] WARN [Producer clientId=console-producer]
> Connection to node 2 could not be established. Broker may not be available.
> (org.apache.kafka.clients.NetworkClient)
> [2018-07-16 18:41:10,561] WARN [Producer clientId=console-producer]
> Connection to node 2 could not be established. Broker may not be available.
> (org.apache.kafka.clients.NetworkClient)
> [2018-07-16 18:41:10,563] WARN [Producer clientId=console-producer]
> Connection to node 1 could not be established. Broker may not be available.
> (org.apache.kafka.clients.NetworkClient)
>
> Node 1 port 9092 is mapped to host port 9093 and node 2 port 9092 is
> mapped to host port 9094
>
>
>
> Dr Mich Talebzadeh
>
>
>
> LinkedIn * https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
> <https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>*
>
>
>
> http://talebzadehmich.wordpress.com
>
>
> *Disclaimer:* Use it at your own risk. Any and all responsibility for any
> loss, damage or destruction of data or any other property which may arise
> from relying on this email's technical content is explicitly disclaimed.
> The author will in no case be liable for any monetary damages arising from
> such loss, damage or destruction.
>
>
>
>
> On Mon, 16 Jul 2018 at 16:49, Mich Talebzadeh <mi...@gmail.com>
> wrote:
>
>> Thanks Chris,
>>
>> I am afraid the issue is still there!
>>
>> docker run -d --name kafka_broker0 -p 9092:9092 -e
>> KAFKA_ADVERTISED_HOST_NAME=50.140.197.220 -e ZOOKEEPER_IP=50.140.197.220 -e
>> KAFKA_BROKER_ID=0 -e KAFKA_BROKER_PORT=9092 -e *KAFKA_ADVERTISED_PORT=9092
>> *ches/kafka
>>
>> ${KAFKA_HOME}/bin/kafka-topics.sh --create --zookeeper rhes75:2181
>> --replication-factor 3 --partitions 3 --topic r3p3
>>
>> Created topic "r3p3".
>>
>> ${KAFKA_HOME}/bin/kafka-topics.sh --describe -zookeeper rhes75:2181 --topic
>> r3p3
>>
>> Topic:r3p3 PartitionCount:3 ReplicationFactor:3 Configs:
>>
>> Topic: r3p3 Partition: 0 Leader: 2 Replicas: 2,1,0
>> Isr: 2,1,0
>>
>> Topic: r3p3 Partition: 1 Leader: 0 Replicas: 0,2,1
>> Isr: 0
>>
>> Topic: r3p3 Partition: 2 Leader: 1 Replicas: 1,0,2
>> Isr: 1,0,2
>>
>> cat ${IN_FILE} | ${KAFKA_HOME}/bin/kafka-console-producer.sh
>> --broker-list rhes75:9092, rhes75:9093, rhes75:9094 --topic r3p3
>>
>> [2018-07-16 17:01:34,496] WARN [Producer clientId=console-producer] Got
>> error produce response with correlation id 10 on topic-partition r3p3-0,
>> retrying (2 attempts left). Error: NOT_LEADER_FOR_PARTITION
>> (org.apache.kafka.clients.producer.internals.Sender)
>> [2018-07-16 17:01:34,596] WARN [Producer clientId=console-producer] Got
>> error produce response with correlation id 13 on topic-partition r3p3-2,
>> retrying (1 attempts left). Error: NOT_LEADER_FOR_PARTITION
>> (org.apache.kafka.clients.producer.internals.Sender)
>> [2018-07-16 17:01:34,597] WARN [Producer clientId=console-producer] Got
>> error produce response with correlation id 14 on topic-partition r3p3-0,
>> retrying (1 attempts left). Error: NOT_LEADER_FOR_PARTITION
>> (org.apache.kafka.clients.producer.internals.Sender)
>> [2018-07-16 17:01:34,698] WARN [Producer clientId=console-producer] Got
>> error produce response with correlation id 16 on topic-partition r3p3-2,
>> retrying (0 attempts left). Error: NOT_LEADER_FOR_PARTITION
>> (org.apache.kafka.clients.producer.internals.Sender)
>> [2018-07-16 17:01:34,699] WARN [Producer clientId=console-producer] Got
>> error produce response with correlation id 17 on topic-partition r3p3-0,
>> retrying (0 attempts left). Error: NOT_LEADER_FOR_PARTITION
>> (org.apache.kafka.clients.producer.internals.Sender)
>> [2018-07-16 17:01:34,800] ERROR Error when sending message to topic r3p3
>> with key: null, value: 67 bytes with error:
>> (org.apache.kafka.clients.producer.internals.ErrorLoggingCallback)
>>
>> Dr Mich Talebzadeh
>>
>>
>>
>> LinkedIn * https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
>> <https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>*
>>
>>
>>
>> http://talebzadehmich.wordpress.com
>>
>>
>> *Disclaimer:* Use it at your own risk. Any and all responsibility for
>> any loss, damage or destruction of data or any other property which may
>> arise from relying on this email's technical content is explicitly
>> disclaimed. The author will in no case be liable for any monetary damages
>> arising from such loss, damage or destruction.
>>
>>
>>
>>
>> On Mon, 16 Jul 2018 at 16:12, Chris Richardson <ch...@chrisrichardson.net>
>> wrote:
>>
>>> I believe you need to use -e KAFKA_ADVERTISED_PORT=909..
>>>
>>> On Mon, Jul 16, 2018 at 7:41 AM, Mich Talebzadeh <
>>> mich.talebzadeh@gmail.com>
>>> wrote:
>>>
>>> > Hi,
>>> >
>>> > I have created a zookeeper and three brokers as dockers in a physical
>>> host
>>> > as shown below
>>> >
>>> > [image: image.png]
>>> >
>>> > The followings are used to create Zookeeper and Kafka dockers
>>> >
>>> > docker run -d --name zookeeper -p 2181:2181 -p 2888:2888 -p 3888:3888
>>> > jplock/zookeeper
>>> >
>>> > docker run -d --name kafka_broker0 -p 9092:9092 -e
>>> > KAFKA_ADVERTISED_HOST_NAME=50.140.197.220 -e
>>> ZOOKEEPER_IP=50.140.197.220
>>> > -e KAFKA_BROKER_ID=0 -e KAFKA_BROKER_PORT=9092 ches/kafka
>>> >
>>> > docker run -d --name kafka_broker1 -p 9093:9092 -e
>>> > KAFKA_ADVERTISED_HOST_NAME=50.140.197.220 -e
>>> ZOOKEEPER_IP=50.140.197.220
>>> > -e KAFKA_BROKER_ID=1 -e KAFKA_BROKER_PORT=9092 ches/kafka
>>> >
>>> > docker run -d --name kafka_broker2 -p 9094:9092 -e
>>> > KAFKA_ADVERTISED_HOST_NAME=50.140.197.220 -e
>>> ZOOKEEPER_IP=50.140.197.220
>>> > -e KAFKA_BROKER_ID=2 -e KAFKA_BROKER_PORT=9092 ches/kafka
>>> >
>>> > Note the mappings of ports to the port on the physical host.
>>> >
>>> > I have created the following topic that works
>>> >
>>> > ${KAFKA_HOME}/bin/kafka-topics.sh --create --zookeeper rhes75:2181
>>> > --replication-factor 1 --partitions 1 --topic three
>>> >
>>> >
>>> > ${KAFKA_HOME}/bin/kafka-topics.sh --describe -zookeeper rhes75:2181
>>> --topic
>>> > three
>>> >
>>> > Topic:three PartitionCount:1 ReplicationFactor:1
>>> Configs:
>>> >
>>> > Topic: three Partition: 0 Leader: 0 Replicas: 0
>>> Isr:
>>> > 0
>>> >
>>> > *So there is only one partition and one replication factor*
>>> >
>>> > The following producer works fine
>>> >
>>> > cat ${IN_FILE} | ${KAFKA_HOME}/bin/kafka-console-producer.sh
>>> > --broker-list rhes75:9092, rhes75:9093, rhes75:9094 --topic three
>>> >
>>> > However, when I define a topic as follows with --replication-factor 2
>>> > --partitions 2
>>> >
>>> > hduser@rhes564: /data6/hduser/prices/avg_prices> rhes75:2181
>>> > --replication-factor 2 --partitions 2 --topic newone
>>> > <
>>> > Created topic "newone".
>>> > hduser@rhes564: /data6/hduser/prices/avg_prices>
>>> ${KAFKA_HOME}/bin/kafka-topics.sh
>>> > --describe -zookeeper rhes75:2181 --topic newone
>>> > Topic:newone PartitionCount:2 ReplicationFactor:2
>>> Configs:
>>> > Topic: newone Partition: 0 Leader: 2 Replicas: 2,0
>>> > Isr: 2,0
>>> > Topic: newone Partition: 1 Leader: 0 Replicas: 0,1
>>> > Isr: 0
>>> >
>>> > It throws errors!
>>> >
>>> > [2018-07-16 15:51:40,852] WARN [Producer clientId=console-producer] Got
>>> > error produce response with correlation id 12 on topic-partition
>>> newone-0,
>>> > retrying (1 attempts left). Error: NOT_LEADER_FOR_PARTITION
>>> > (org.apache.kafka.clients.producer.internals.Sender)
>>> > [2018-07-16 15:51:40,955] WARN [Producer clientId=console-producer] Got
>>> > error produce response with correlation id 14 on topic-partition
>>> newone-0,
>>> > retrying (0 attempts left). Error: NOT_LEADER_FOR_PARTITION
>>> > (org.apache.kafka.clients.producer.internals.Sender)
>>> > [2018-07-16 15:51:41,056] ERROR Error when sending message to topic
>>> newone
>>> > with key: null, value: 67 bytes with error: (org.apache.kafka.clients.
>>> > producer.internals.ErrorLoggingCallback)
>>> > org.apache.kafka.common.errors.NotLeaderForPartitionException: This
>>> > server is not the leader for that topic-partition.
>>> > [2018-07-16 15:51:41,059] ERROR Error when sending message to topic
>>> newone
>>> > with key: null, value: 67 bytes with error: (org.apache.kafka.clients.
>>> > producer.internals.ErrorLoggingCallback)
>>> > org.apache.kafka.common.errors.NotLeaderForPartitionException: This
>>> > server is not the leader for that topic-partition.
>>> > [2018-07-16 15:51:41,059] ERROR Error when sending message to topic
>>> newone
>>> > with key: null, value: 68 bytes with error: (org.apache.kafka.clients.
>>> > producer.internals.ErrorLoggingCallback)
>>> > org.apache.kafka.common.errors.NotLeaderForPartitionException: This
>>> > server is not the leader for that topic-partition.
>>> > [2018-07-16 15:51:41,060] ERROR Error when sending message to topic
>>> newone
>>> > with key: null, value: 67 bytes with error: (org.apache.kafka.clients.
>>> > producer.internals.ErrorLoggingCallback)
>>> > org.apache.kafka.common.errors.NotLeaderForPartitionException: This
>>> > server is not the leader for that topic-partition.
>>> > [2018-07-16 15:51:41,060] ERROR Error when sending message to topic
>>> newone
>>> > with key: null, value: 67 bytes with error: (org.apache.kafka.clients.
>>> > producer.internals.ErrorLoggingCallback)
>>> > org.apache.kafka.common.errors.NotLeaderForPartitionException: This
>>> > server is not the leader for that topic-partition.
>>> >
>>> > I believe these Kafka brokers have problem talking to each other and
>>> the
>>> > message is lost!
>>> >
>>> > Thanks
>>> >
>>> > Dr Mich Talebzadeh
>>> >
>>> >
>>> >
>>> > LinkedIn *
>>> https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
>>> > <
>>> https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
>>> >*
>>> >
>>> >
>>> >
>>> > http://talebzadehmich.wordpress.com
>>> >
>>> >
>>> > *Disclaimer:* Use it at your own risk. Any and all responsibility for
>>> any
>>> > loss, damage or destruction of data or any other property which may
>>> arise
>>> > from relying on this email's technical content is explicitly
>>> disclaimed.
>>> > The author will in no case be liable for any monetary damages arising
>>> from
>>> > such loss, damage or destruction.
>>> >
>>> >
>>> >
>>>
>>>
>>>
>>> --
>>> Learn microservices - http://learn.microservices.io
>>> Microservices application platform http://eventuate.io
>>> Consulting and training http://chrisrichardson.net
>>>
>>
Re: Zookeeper and Kafka cluster docker issue
Posted by Mich Talebzadeh <mi...@gmail.com>.
Also I noticed that bar broker ID =0, the connection to broker ID 1 )node
1) and broker ID 2 (node 2) could not be established
[2018-07-16 18:41:10,419] WARN [Producer clientId=console-producer]
Connection to node 1 could not be established. Broker may not be available.
(org.apache.kafka.clients.NetworkClient)
[2018-07-16 18:41:10,420] WARN [Producer clientId=console-producer]
Connection to node 2 could not be established. Broker may not be available.
(org.apache.kafka.clients.NetworkClient)
[2018-07-16 18:41:10,464] WARN [Producer clientId=console-producer]
Connection to node 1 could not be established. Broker may not be available.
(org.apache.kafka.clients.NetworkClient)
[2018-07-16 18:41:10,470] WARN [Producer clientId=console-producer]
Connection to node 2 could not be established. Broker may not be available.
(org.apache.kafka.clients.NetworkClient)
[2018-07-16 18:41:10,561] WARN [Producer clientId=console-producer]
Connection to node 2 could not be established. Broker may not be available.
(org.apache.kafka.clients.NetworkClient)
[2018-07-16 18:41:10,563] WARN [Producer clientId=console-producer]
Connection to node 1 could not be established. Broker may not be available.
(org.apache.kafka.clients.NetworkClient)
Node 1 port 9092 is mapped to host port 9093 and node 2 port 9092 is mapped
to host port 9094
Dr Mich Talebzadeh
LinkedIn * https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
<https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>*
http://talebzadehmich.wordpress.com
*Disclaimer:* Use it at your own risk. Any and all responsibility for any
loss, damage or destruction of data or any other property which may arise
from relying on this email's technical content is explicitly disclaimed.
The author will in no case be liable for any monetary damages arising from
such loss, damage or destruction.
On Mon, 16 Jul 2018 at 16:49, Mich Talebzadeh <mi...@gmail.com>
wrote:
> Thanks Chris,
>
> I am afraid the issue is still there!
>
> docker run -d --name kafka_broker0 -p 9092:9092 -e
> KAFKA_ADVERTISED_HOST_NAME=50.140.197.220 -e ZOOKEEPER_IP=50.140.197.220 -e
> KAFKA_BROKER_ID=0 -e KAFKA_BROKER_PORT=9092 -e *KAFKA_ADVERTISED_PORT=9092
> *ches/kafka
>
> ${KAFKA_HOME}/bin/kafka-topics.sh --create --zookeeper rhes75:2181
> --replication-factor 3 --partitions 3 --topic r3p3
>
> Created topic "r3p3".
>
> ${KAFKA_HOME}/bin/kafka-topics.sh --describe -zookeeper rhes75:2181 --topic
> r3p3
>
> Topic:r3p3 PartitionCount:3 ReplicationFactor:3 Configs:
>
> Topic: r3p3 Partition: 0 Leader: 2 Replicas: 2,1,0
> Isr: 2,1,0
>
> Topic: r3p3 Partition: 1 Leader: 0 Replicas: 0,2,1
> Isr: 0
>
> Topic: r3p3 Partition: 2 Leader: 1 Replicas: 1,0,2
> Isr: 1,0,2
>
> cat ${IN_FILE} | ${KAFKA_HOME}/bin/kafka-console-producer.sh --broker-list
> rhes75:9092, rhes75:9093, rhes75:9094 --topic r3p3
>
> [2018-07-16 17:01:34,496] WARN [Producer clientId=console-producer] Got
> error produce response with correlation id 10 on topic-partition r3p3-0,
> retrying (2 attempts left). Error: NOT_LEADER_FOR_PARTITION
> (org.apache.kafka.clients.producer.internals.Sender)
> [2018-07-16 17:01:34,596] WARN [Producer clientId=console-producer] Got
> error produce response with correlation id 13 on topic-partition r3p3-2,
> retrying (1 attempts left). Error: NOT_LEADER_FOR_PARTITION
> (org.apache.kafka.clients.producer.internals.Sender)
> [2018-07-16 17:01:34,597] WARN [Producer clientId=console-producer] Got
> error produce response with correlation id 14 on topic-partition r3p3-0,
> retrying (1 attempts left). Error: NOT_LEADER_FOR_PARTITION
> (org.apache.kafka.clients.producer.internals.Sender)
> [2018-07-16 17:01:34,698] WARN [Producer clientId=console-producer] Got
> error produce response with correlation id 16 on topic-partition r3p3-2,
> retrying (0 attempts left). Error: NOT_LEADER_FOR_PARTITION
> (org.apache.kafka.clients.producer.internals.Sender)
> [2018-07-16 17:01:34,699] WARN [Producer clientId=console-producer] Got
> error produce response with correlation id 17 on topic-partition r3p3-0,
> retrying (0 attempts left). Error: NOT_LEADER_FOR_PARTITION
> (org.apache.kafka.clients.producer.internals.Sender)
> [2018-07-16 17:01:34,800] ERROR Error when sending message to topic r3p3
> with key: null, value: 67 bytes with error:
> (org.apache.kafka.clients.producer.internals.ErrorLoggingCallback)
>
> Dr Mich Talebzadeh
>
>
>
> LinkedIn * https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
> <https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>*
>
>
>
> http://talebzadehmich.wordpress.com
>
>
> *Disclaimer:* Use it at your own risk. Any and all responsibility for any
> loss, damage or destruction of data or any other property which may arise
> from relying on this email's technical content is explicitly disclaimed.
> The author will in no case be liable for any monetary damages arising from
> such loss, damage or destruction.
>
>
>
>
> On Mon, 16 Jul 2018 at 16:12, Chris Richardson <ch...@chrisrichardson.net>
> wrote:
>
>> I believe you need to use -e KAFKA_ADVERTISED_PORT=909..
>>
>> On Mon, Jul 16, 2018 at 7:41 AM, Mich Talebzadeh <
>> mich.talebzadeh@gmail.com>
>> wrote:
>>
>> > Hi,
>> >
>> > I have created a zookeeper and three brokers as dockers in a physical
>> host
>> > as shown below
>> >
>> > [image: image.png]
>> >
>> > The followings are used to create Zookeeper and Kafka dockers
>> >
>> > docker run -d --name zookeeper -p 2181:2181 -p 2888:2888 -p 3888:3888
>> > jplock/zookeeper
>> >
>> > docker run -d --name kafka_broker0 -p 9092:9092 -e
>> > KAFKA_ADVERTISED_HOST_NAME=50.140.197.220 -e ZOOKEEPER_IP=50.140.197.220
>> > -e KAFKA_BROKER_ID=0 -e KAFKA_BROKER_PORT=9092 ches/kafka
>> >
>> > docker run -d --name kafka_broker1 -p 9093:9092 -e
>> > KAFKA_ADVERTISED_HOST_NAME=50.140.197.220 -e ZOOKEEPER_IP=50.140.197.220
>> > -e KAFKA_BROKER_ID=1 -e KAFKA_BROKER_PORT=9092 ches/kafka
>> >
>> > docker run -d --name kafka_broker2 -p 9094:9092 -e
>> > KAFKA_ADVERTISED_HOST_NAME=50.140.197.220 -e ZOOKEEPER_IP=50.140.197.220
>> > -e KAFKA_BROKER_ID=2 -e KAFKA_BROKER_PORT=9092 ches/kafka
>> >
>> > Note the mappings of ports to the port on the physical host.
>> >
>> > I have created the following topic that works
>> >
>> > ${KAFKA_HOME}/bin/kafka-topics.sh --create --zookeeper rhes75:2181
>> > --replication-factor 1 --partitions 1 --topic three
>> >
>> >
>> > ${KAFKA_HOME}/bin/kafka-topics.sh --describe -zookeeper rhes75:2181
>> --topic
>> > three
>> >
>> > Topic:three PartitionCount:1 ReplicationFactor:1 Configs:
>> >
>> > Topic: three Partition: 0 Leader: 0 Replicas: 0
>> Isr:
>> > 0
>> >
>> > *So there is only one partition and one replication factor*
>> >
>> > The following producer works fine
>> >
>> > cat ${IN_FILE} | ${KAFKA_HOME}/bin/kafka-console-producer.sh
>> > --broker-list rhes75:9092, rhes75:9093, rhes75:9094 --topic three
>> >
>> > However, when I define a topic as follows with --replication-factor 2
>> > --partitions 2
>> >
>> > hduser@rhes564: /data6/hduser/prices/avg_prices> rhes75:2181
>> > --replication-factor 2 --partitions 2 --topic newone
>> > <
>> > Created topic "newone".
>> > hduser@rhes564: /data6/hduser/prices/avg_prices>
>> ${KAFKA_HOME}/bin/kafka-topics.sh
>> > --describe -zookeeper rhes75:2181 --topic newone
>> > Topic:newone PartitionCount:2 ReplicationFactor:2 Configs:
>> > Topic: newone Partition: 0 Leader: 2 Replicas: 2,0
>> > Isr: 2,0
>> > Topic: newone Partition: 1 Leader: 0 Replicas: 0,1
>> > Isr: 0
>> >
>> > It throws errors!
>> >
>> > [2018-07-16 15:51:40,852] WARN [Producer clientId=console-producer] Got
>> > error produce response with correlation id 12 on topic-partition
>> newone-0,
>> > retrying (1 attempts left). Error: NOT_LEADER_FOR_PARTITION
>> > (org.apache.kafka.clients.producer.internals.Sender)
>> > [2018-07-16 15:51:40,955] WARN [Producer clientId=console-producer] Got
>> > error produce response with correlation id 14 on topic-partition
>> newone-0,
>> > retrying (0 attempts left). Error: NOT_LEADER_FOR_PARTITION
>> > (org.apache.kafka.clients.producer.internals.Sender)
>> > [2018-07-16 15:51:41,056] ERROR Error when sending message to topic
>> newone
>> > with key: null, value: 67 bytes with error: (org.apache.kafka.clients.
>> > producer.internals.ErrorLoggingCallback)
>> > org.apache.kafka.common.errors.NotLeaderForPartitionException: This
>> > server is not the leader for that topic-partition.
>> > [2018-07-16 15:51:41,059] ERROR Error when sending message to topic
>> newone
>> > with key: null, value: 67 bytes with error: (org.apache.kafka.clients.
>> > producer.internals.ErrorLoggingCallback)
>> > org.apache.kafka.common.errors.NotLeaderForPartitionException: This
>> > server is not the leader for that topic-partition.
>> > [2018-07-16 15:51:41,059] ERROR Error when sending message to topic
>> newone
>> > with key: null, value: 68 bytes with error: (org.apache.kafka.clients.
>> > producer.internals.ErrorLoggingCallback)
>> > org.apache.kafka.common.errors.NotLeaderForPartitionException: This
>> > server is not the leader for that topic-partition.
>> > [2018-07-16 15:51:41,060] ERROR Error when sending message to topic
>> newone
>> > with key: null, value: 67 bytes with error: (org.apache.kafka.clients.
>> > producer.internals.ErrorLoggingCallback)
>> > org.apache.kafka.common.errors.NotLeaderForPartitionException: This
>> > server is not the leader for that topic-partition.
>> > [2018-07-16 15:51:41,060] ERROR Error when sending message to topic
>> newone
>> > with key: null, value: 67 bytes with error: (org.apache.kafka.clients.
>> > producer.internals.ErrorLoggingCallback)
>> > org.apache.kafka.common.errors.NotLeaderForPartitionException: This
>> > server is not the leader for that topic-partition.
>> >
>> > I believe these Kafka brokers have problem talking to each other and the
>> > message is lost!
>> >
>> > Thanks
>> >
>> > Dr Mich Talebzadeh
>> >
>> >
>> >
>> > LinkedIn *
>> https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
>> > <
>> https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
>> >*
>> >
>> >
>> >
>> > http://talebzadehmich.wordpress.com
>> >
>> >
>> > *Disclaimer:* Use it at your own risk. Any and all responsibility for
>> any
>> > loss, damage or destruction of data or any other property which may
>> arise
>> > from relying on this email's technical content is explicitly disclaimed.
>> > The author will in no case be liable for any monetary damages arising
>> from
>> > such loss, damage or destruction.
>> >
>> >
>> >
>>
>>
>>
>> --
>> Learn microservices - http://learn.microservices.io
>> Microservices application platform http://eventuate.io
>> Consulting and training http://chrisrichardson.net
>>
>
Re: Zookeeper and Kafka cluster docker issue
Posted by Mich Talebzadeh <mi...@gmail.com>.
Thanks Chris,
I am afraid the issue is still there!
docker run -d --name kafka_broker0 -p 9092:9092 -e
KAFKA_ADVERTISED_HOST_NAME=50.140.197.220 -e ZOOKEEPER_IP=50.140.197.220 -e
KAFKA_BROKER_ID=0 -e KAFKA_BROKER_PORT=9092 -e *KAFKA_ADVERTISED_PORT=9092
*ches/kafka
${KAFKA_HOME}/bin/kafka-topics.sh --create --zookeeper rhes75:2181
--replication-factor 3 --partitions 3 --topic r3p3
Created topic "r3p3".
${KAFKA_HOME}/bin/kafka-topics.sh --describe -zookeeper rhes75:2181 --topic
r3p3
Topic:r3p3 PartitionCount:3 ReplicationFactor:3 Configs:
Topic: r3p3 Partition: 0 Leader: 2 Replicas: 2,1,0
Isr: 2,1,0
Topic: r3p3 Partition: 1 Leader: 0 Replicas: 0,2,1
Isr: 0
Topic: r3p3 Partition: 2 Leader: 1 Replicas: 1,0,2
Isr: 1,0,2
cat ${IN_FILE} | ${KAFKA_HOME}/bin/kafka-console-producer.sh --broker-list
rhes75:9092, rhes75:9093, rhes75:9094 --topic r3p3
[2018-07-16 17:01:34,496] WARN [Producer clientId=console-producer] Got
error produce response with correlation id 10 on topic-partition r3p3-0,
retrying (2 attempts left). Error: NOT_LEADER_FOR_PARTITION
(org.apache.kafka.clients.producer.internals.Sender)
[2018-07-16 17:01:34,596] WARN [Producer clientId=console-producer] Got
error produce response with correlation id 13 on topic-partition r3p3-2,
retrying (1 attempts left). Error: NOT_LEADER_FOR_PARTITION
(org.apache.kafka.clients.producer.internals.Sender)
[2018-07-16 17:01:34,597] WARN [Producer clientId=console-producer] Got
error produce response with correlation id 14 on topic-partition r3p3-0,
retrying (1 attempts left). Error: NOT_LEADER_FOR_PARTITION
(org.apache.kafka.clients.producer.internals.Sender)
[2018-07-16 17:01:34,698] WARN [Producer clientId=console-producer] Got
error produce response with correlation id 16 on topic-partition r3p3-2,
retrying (0 attempts left). Error: NOT_LEADER_FOR_PARTITION
(org.apache.kafka.clients.producer.internals.Sender)
[2018-07-16 17:01:34,699] WARN [Producer clientId=console-producer] Got
error produce response with correlation id 17 on topic-partition r3p3-0,
retrying (0 attempts left). Error: NOT_LEADER_FOR_PARTITION
(org.apache.kafka.clients.producer.internals.Sender)
[2018-07-16 17:01:34,800] ERROR Error when sending message to topic r3p3
with key: null, value: 67 bytes with error:
(org.apache.kafka.clients.producer.internals.ErrorLoggingCallback)
Dr Mich Talebzadeh
LinkedIn * https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
<https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>*
http://talebzadehmich.wordpress.com
*Disclaimer:* Use it at your own risk. Any and all responsibility for any
loss, damage or destruction of data or any other property which may arise
from relying on this email's technical content is explicitly disclaimed.
The author will in no case be liable for any monetary damages arising from
such loss, damage or destruction.
On Mon, 16 Jul 2018 at 16:12, Chris Richardson <ch...@chrisrichardson.net>
wrote:
> I believe you need to use -e KAFKA_ADVERTISED_PORT=909..
>
> On Mon, Jul 16, 2018 at 7:41 AM, Mich Talebzadeh <
> mich.talebzadeh@gmail.com>
> wrote:
>
> > Hi,
> >
> > I have created a zookeeper and three brokers as dockers in a physical
> host
> > as shown below
> >
> > [image: image.png]
> >
> > The followings are used to create Zookeeper and Kafka dockers
> >
> > docker run -d --name zookeeper -p 2181:2181 -p 2888:2888 -p 3888:3888
> > jplock/zookeeper
> >
> > docker run -d --name kafka_broker0 -p 9092:9092 -e
> > KAFKA_ADVERTISED_HOST_NAME=50.140.197.220 -e ZOOKEEPER_IP=50.140.197.220
> > -e KAFKA_BROKER_ID=0 -e KAFKA_BROKER_PORT=9092 ches/kafka
> >
> > docker run -d --name kafka_broker1 -p 9093:9092 -e
> > KAFKA_ADVERTISED_HOST_NAME=50.140.197.220 -e ZOOKEEPER_IP=50.140.197.220
> > -e KAFKA_BROKER_ID=1 -e KAFKA_BROKER_PORT=9092 ches/kafka
> >
> > docker run -d --name kafka_broker2 -p 9094:9092 -e
> > KAFKA_ADVERTISED_HOST_NAME=50.140.197.220 -e ZOOKEEPER_IP=50.140.197.220
> > -e KAFKA_BROKER_ID=2 -e KAFKA_BROKER_PORT=9092 ches/kafka
> >
> > Note the mappings of ports to the port on the physical host.
> >
> > I have created the following topic that works
> >
> > ${KAFKA_HOME}/bin/kafka-topics.sh --create --zookeeper rhes75:2181
> > --replication-factor 1 --partitions 1 --topic three
> >
> >
> > ${KAFKA_HOME}/bin/kafka-topics.sh --describe -zookeeper rhes75:2181
> --topic
> > three
> >
> > Topic:three PartitionCount:1 ReplicationFactor:1 Configs:
> >
> > Topic: three Partition: 0 Leader: 0 Replicas: 0
> Isr:
> > 0
> >
> > *So there is only one partition and one replication factor*
> >
> > The following producer works fine
> >
> > cat ${IN_FILE} | ${KAFKA_HOME}/bin/kafka-console-producer.sh
> > --broker-list rhes75:9092, rhes75:9093, rhes75:9094 --topic three
> >
> > However, when I define a topic as follows with --replication-factor 2
> > --partitions 2
> >
> > hduser@rhes564: /data6/hduser/prices/avg_prices> rhes75:2181
> > --replication-factor 2 --partitions 2 --topic newone
> > <
> > Created topic "newone".
> > hduser@rhes564: /data6/hduser/prices/avg_prices>
> ${KAFKA_HOME}/bin/kafka-topics.sh
> > --describe -zookeeper rhes75:2181 --topic newone
> > Topic:newone PartitionCount:2 ReplicationFactor:2 Configs:
> > Topic: newone Partition: 0 Leader: 2 Replicas: 2,0
> > Isr: 2,0
> > Topic: newone Partition: 1 Leader: 0 Replicas: 0,1
> > Isr: 0
> >
> > It throws errors!
> >
> > [2018-07-16 15:51:40,852] WARN [Producer clientId=console-producer] Got
> > error produce response with correlation id 12 on topic-partition
> newone-0,
> > retrying (1 attempts left). Error: NOT_LEADER_FOR_PARTITION
> > (org.apache.kafka.clients.producer.internals.Sender)
> > [2018-07-16 15:51:40,955] WARN [Producer clientId=console-producer] Got
> > error produce response with correlation id 14 on topic-partition
> newone-0,
> > retrying (0 attempts left). Error: NOT_LEADER_FOR_PARTITION
> > (org.apache.kafka.clients.producer.internals.Sender)
> > [2018-07-16 15:51:41,056] ERROR Error when sending message to topic
> newone
> > with key: null, value: 67 bytes with error: (org.apache.kafka.clients.
> > producer.internals.ErrorLoggingCallback)
> > org.apache.kafka.common.errors.NotLeaderForPartitionException: This
> > server is not the leader for that topic-partition.
> > [2018-07-16 15:51:41,059] ERROR Error when sending message to topic
> newone
> > with key: null, value: 67 bytes with error: (org.apache.kafka.clients.
> > producer.internals.ErrorLoggingCallback)
> > org.apache.kafka.common.errors.NotLeaderForPartitionException: This
> > server is not the leader for that topic-partition.
> > [2018-07-16 15:51:41,059] ERROR Error when sending message to topic
> newone
> > with key: null, value: 68 bytes with error: (org.apache.kafka.clients.
> > producer.internals.ErrorLoggingCallback)
> > org.apache.kafka.common.errors.NotLeaderForPartitionException: This
> > server is not the leader for that topic-partition.
> > [2018-07-16 15:51:41,060] ERROR Error when sending message to topic
> newone
> > with key: null, value: 67 bytes with error: (org.apache.kafka.clients.
> > producer.internals.ErrorLoggingCallback)
> > org.apache.kafka.common.errors.NotLeaderForPartitionException: This
> > server is not the leader for that topic-partition.
> > [2018-07-16 15:51:41,060] ERROR Error when sending message to topic
> newone
> > with key: null, value: 67 bytes with error: (org.apache.kafka.clients.
> > producer.internals.ErrorLoggingCallback)
> > org.apache.kafka.common.errors.NotLeaderForPartitionException: This
> > server is not the leader for that topic-partition.
> >
> > I believe these Kafka brokers have problem talking to each other and the
> > message is lost!
> >
> > Thanks
> >
> > Dr Mich Talebzadeh
> >
> >
> >
> > LinkedIn *
> https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
> > <
> https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
> >*
> >
> >
> >
> > http://talebzadehmich.wordpress.com
> >
> >
> > *Disclaimer:* Use it at your own risk. Any and all responsibility for any
> > loss, damage or destruction of data or any other property which may arise
> > from relying on this email's technical content is explicitly disclaimed.
> > The author will in no case be liable for any monetary damages arising
> from
> > such loss, damage or destruction.
> >
> >
> >
>
>
>
> --
> Learn microservices - http://learn.microservices.io
> Microservices application platform http://eventuate.io
> Consulting and training http://chrisrichardson.net
>
Re: Zookeeper and Kafka cluster docker issue
Posted by Chris Richardson <ch...@chrisrichardson.net>.
I believe you need to use -e KAFKA_ADVERTISED_PORT=909..
On Mon, Jul 16, 2018 at 7:41 AM, Mich Talebzadeh <mi...@gmail.com>
wrote:
> Hi,
>
> I have created a zookeeper and three brokers as dockers in a physical host
> as shown below
>
> [image: image.png]
>
> The followings are used to create Zookeeper and Kafka dockers
>
> docker run -d --name zookeeper -p 2181:2181 -p 2888:2888 -p 3888:3888
> jplock/zookeeper
>
> docker run -d --name kafka_broker0 -p 9092:9092 -e
> KAFKA_ADVERTISED_HOST_NAME=50.140.197.220 -e ZOOKEEPER_IP=50.140.197.220
> -e KAFKA_BROKER_ID=0 -e KAFKA_BROKER_PORT=9092 ches/kafka
>
> docker run -d --name kafka_broker1 -p 9093:9092 -e
> KAFKA_ADVERTISED_HOST_NAME=50.140.197.220 -e ZOOKEEPER_IP=50.140.197.220
> -e KAFKA_BROKER_ID=1 -e KAFKA_BROKER_PORT=9092 ches/kafka
>
> docker run -d --name kafka_broker2 -p 9094:9092 -e
> KAFKA_ADVERTISED_HOST_NAME=50.140.197.220 -e ZOOKEEPER_IP=50.140.197.220
> -e KAFKA_BROKER_ID=2 -e KAFKA_BROKER_PORT=9092 ches/kafka
>
> Note the mappings of ports to the port on the physical host.
>
> I have created the following topic that works
>
> ${KAFKA_HOME}/bin/kafka-topics.sh --create --zookeeper rhes75:2181
> --replication-factor 1 --partitions 1 --topic three
>
>
> ${KAFKA_HOME}/bin/kafka-topics.sh --describe -zookeeper rhes75:2181 --topic
> three
>
> Topic:three PartitionCount:1 ReplicationFactor:1 Configs:
>
> Topic: three Partition: 0 Leader: 0 Replicas: 0 Isr:
> 0
>
> *So there is only one partition and one replication factor*
>
> The following producer works fine
>
> cat ${IN_FILE} | ${KAFKA_HOME}/bin/kafka-console-producer.sh
> --broker-list rhes75:9092, rhes75:9093, rhes75:9094 --topic three
>
> However, when I define a topic as follows with --replication-factor 2
> --partitions 2
>
> hduser@rhes564: /data6/hduser/prices/avg_prices> rhes75:2181
> --replication-factor 2 --partitions 2 --topic newone
> <
> Created topic "newone".
> hduser@rhes564: /data6/hduser/prices/avg_prices> ${KAFKA_HOME}/bin/kafka-topics.sh
> --describe -zookeeper rhes75:2181 --topic newone
> Topic:newone PartitionCount:2 ReplicationFactor:2 Configs:
> Topic: newone Partition: 0 Leader: 2 Replicas: 2,0
> Isr: 2,0
> Topic: newone Partition: 1 Leader: 0 Replicas: 0,1
> Isr: 0
>
> It throws errors!
>
> [2018-07-16 15:51:40,852] WARN [Producer clientId=console-producer] Got
> error produce response with correlation id 12 on topic-partition newone-0,
> retrying (1 attempts left). Error: NOT_LEADER_FOR_PARTITION
> (org.apache.kafka.clients.producer.internals.Sender)
> [2018-07-16 15:51:40,955] WARN [Producer clientId=console-producer] Got
> error produce response with correlation id 14 on topic-partition newone-0,
> retrying (0 attempts left). Error: NOT_LEADER_FOR_PARTITION
> (org.apache.kafka.clients.producer.internals.Sender)
> [2018-07-16 15:51:41,056] ERROR Error when sending message to topic newone
> with key: null, value: 67 bytes with error: (org.apache.kafka.clients.
> producer.internals.ErrorLoggingCallback)
> org.apache.kafka.common.errors.NotLeaderForPartitionException: This
> server is not the leader for that topic-partition.
> [2018-07-16 15:51:41,059] ERROR Error when sending message to topic newone
> with key: null, value: 67 bytes with error: (org.apache.kafka.clients.
> producer.internals.ErrorLoggingCallback)
> org.apache.kafka.common.errors.NotLeaderForPartitionException: This
> server is not the leader for that topic-partition.
> [2018-07-16 15:51:41,059] ERROR Error when sending message to topic newone
> with key: null, value: 68 bytes with error: (org.apache.kafka.clients.
> producer.internals.ErrorLoggingCallback)
> org.apache.kafka.common.errors.NotLeaderForPartitionException: This
> server is not the leader for that topic-partition.
> [2018-07-16 15:51:41,060] ERROR Error when sending message to topic newone
> with key: null, value: 67 bytes with error: (org.apache.kafka.clients.
> producer.internals.ErrorLoggingCallback)
> org.apache.kafka.common.errors.NotLeaderForPartitionException: This
> server is not the leader for that topic-partition.
> [2018-07-16 15:51:41,060] ERROR Error when sending message to topic newone
> with key: null, value: 67 bytes with error: (org.apache.kafka.clients.
> producer.internals.ErrorLoggingCallback)
> org.apache.kafka.common.errors.NotLeaderForPartitionException: This
> server is not the leader for that topic-partition.
>
> I believe these Kafka brokers have problem talking to each other and the
> message is lost!
>
> Thanks
>
> Dr Mich Talebzadeh
>
>
>
> LinkedIn * https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
> <https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>*
>
>
>
> http://talebzadehmich.wordpress.com
>
>
> *Disclaimer:* Use it at your own risk. Any and all responsibility for any
> loss, damage or destruction of data or any other property which may arise
> from relying on this email's technical content is explicitly disclaimed.
> The author will in no case be liable for any monetary damages arising from
> such loss, damage or destruction.
>
>
>
--
Learn microservices - http://learn.microservices.io
Microservices application platform http://eventuate.io
Consulting and training http://chrisrichardson.net