You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by Булат Юсупов <bi...@avtodoria.ru> on 2018/03/06 09:00:01 UTC
Kafka cluster loses messages after zookeeper restart
Hi,
I'm starting a cluster of kafka brokers using Docker (5 brokers for
example, one broker per container). Kafka version 2.12-0.11.0.0, Zookeeper
3.4.10.
*The scenario:*
- Starting 1st broker with config below
*zoo.cfg*
tickTime=2000
initLimit=10
syncLimit=5
dataDir=/opt/zookeeper/data
clientPort=2181
maxClientCnxns=10
minSessionTimeout=4000
maxSessionTimeout=1000000
server.1=0.0.0.0:2888:3888
*server.properties*
broker.id=1
listeners=PLAINTEXT://:9092
advertised.listeners=PLAINTEXT://broker1_IP:broker1_PORT
num.network.threads=3
num.io.threads=8
socket.send.buffer.bytes=102400
socket.receive.buffer.bytes=102400
socket.request.max.bytes=104857600
log.dirs=/tmp/kafka-logs
num.partitions=1
num.recovery.threads.per.data.dir=1
offsets.topic.replication.factor=1
transaction.state.log.replication.factor=1
transaction.state.log.min.isr=1
log.retention.hours=168
log.segment.bytes=1073741824log.retention.check.interval.ms=300000
zookeeper.connect=127.0.0.1:2181zookeeper.session.timeout.ms=6000zookeeper.connection.timeout.ms=1000000group.initial.rebalance.delay.ms=0
*producer.properties*
bootstrap.servers=localhost:9092
compression.type=none
*consumer.properties*
zookeeper.connect=127.0.0.1:2181zookeeper.session.timeout.ms=6000zookeeper.connection.timeout.ms=1000000group.id=test-consumer-group
-
Zookeeper is started in standalone mode, then starts kafka
-
Creating topic
/opt/kafka/bin/kafka-topics.sh --create --zookeeper localhost:2181
--replication-factor 1 --partitions 1 --topic my-test-topic1
- Sending message
echo "test_kafka1" | /opt/kafka/bin/kafka-console-producer.sh --broker-list
localhost:9092 --topic my-test-topic1
- Checking message
/opt/kafka/bin/kafka-console-consumer.sh --bootstrap-server localhost:9092
--from-beginning --topic my-test-topic1 --max-messages 1
*Message is recieved*
- Describe the topic
/opt/kafka/bin/kafka-topics.sh --describe --zookeeper localhost:2181
--topic my-test-topic1 Topic:my-test-topic1 PartitionCount:1
ReplicationFactor:1 Configs: Topic: my-test-topic1 Partition: 0 Leader: 1
Replicas: 1 Isr: 1
- Starting rest 4 brokers
*zoo.cfg* on each broker from 1st to 5th (only 0.0.0.0:2888:3888 position
differs)
tickTime=2000
initLimit=10
syncLimit=5
dataDir=/opt/zookeeper/data
clientPort=2181
maxClientCnxns=10
minSessionTimeout=4000
maxSessionTimeout=1000000
server.1=0.0.0.0:2888:3888
server.2=broker2_IP:broker2_2888:broker2_3888
server.3=broker3_IP:broker3_2888:broker3_3888
server.4=broker4_IP:broker4_2888:broker4_3888
server.5=broker5_IP:broker5_2888:broker5_3888
*server.properties* on each broker from 1st to 5th (broker.id are unique,
broker_IP:broker_PORT differs for ech broker)
broker.id=N
listeners=PLAINTEXT://:9092
advertised.listeners=PLAINTEXT://broker_IP:broker_PORT
num.network.threads=3
num.io.threads=8
socket.send.buffer.bytes=102400
socket.receive.buffer.bytes=102400
socket.request.max.bytes=104857600
log.dirs=/tmp/kafka-logs
num.partitions=1
num.recovery.threads.per.data.dir=1
offsets.topic.replication.factor=1
transaction.state.log.replication.factor=1
transaction.state.log.min.isr=1
log.retention.hours=168
log.segment.bytes=1073741824log.retention.check.interval.ms=300000
zookeeper.connect=127.0.0.1:2181zookeeper.session.timeout.ms=6000zookeeper.connection.timeout.ms=1000000group.initial.rebalance.delay.ms=0
*producer.properties* on each broker from 1st to 5th
bootstrap.servers=localhost:9092
compression.type=none
*consumer.properties* on each broker from 1st to 5th
zookeeper.connect=127.0.0.1:2181zookeeper.session.timeout.ms=6000zookeeper.connection.timeout.ms=1000000group.id=test-consumer-group
-
Restarting zookeeper on each broker to take effect of zoo.cfg
-
Zookeepers gather into cluster
-
Topic moved to broker 5
/opt/kafka/bin/kafka-topics.sh --describe --zookeeper localhost:2181
--topic my-test-topic1 Topic:my-test-topic1 PartitionCount:1
ReplicationFactor:1 Configs: Topic: my-test-topic1 Partition: 0 Leader: 5
Replicas: 5 Isr: 5
Is it normal behavior? Or should it stay on broker 1?
- Checking message on each broker
/opt/kafka/bin/kafka-console-consumer.sh --bootstrap-server localhost:9092
--from-beginning --topic my-test-topic1 --max-messages 1
*Message is lost* (message is not lost when topic stays on broker 1, so it
is floating situation)
Could you help me with this problem.
TIA
--
*Булат Юсупов */ Инженер-программист / Отдел разработки
biusupov@avtodoria.ru / +7 917 889 99 32
skype: usbulat
ООО "Автодория"
+7 843 524 74 12
Казань, Технопарк в сфере высоких технологий "ИТ-парк", Петербургская, 52,
офис 303
www.avtodoria.ru
*Инновации спасают жизни!*