You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by Jiamei Xie <Ji...@arm.com> on 2020/05/22 07:31:12 UTC
Is that a bug?
Hi
Kill all zookeeper and kafka process. Clear zookeeper and kafka data dir. Restart zookeeper and kafka. If there are any active client. Topic used by client will be auto-created.
How to reproduce?
1. Start zookeeper and kafka zookeeper and kafka config file.
nohup bin/zookeeper-server-start.sh config/zookeeper.properties &
nohup bin/kafka-server-start.sh config/server.properties &
1. Create topic test with 2 partitions
bin/kafka-topics.sh --create --zookeeper localhost:2181 --topic test --partitions 2 --replication-factor 1
1. Produce some data to topic test
bin/kafka-producer-perf-test.sh --topic test --num-records 50000000 --record-size 100 --throughput=-1 --producer-props bootstrap.servers=localhost:9092
1. Kill zookeeper and kafka. ProducerPerformance is still running.
jps
21072 QuorumPeerMain
21704 ProducerPerformance
21230 Kafka
21854 Jps
kill -9 21072 21230
1. Remove Zookeeper and Kafka data
rm -rf /tmp/zookeeper/
rm -rf /tmp/kafka-logs/
1. Start zookeeper and kafka
nohup bin/zookeeper-server-start.sh config/zookeeper.properties &
nohup bin/kafka-server-start.sh config/server.properties &
1. Check topic and you'll see there is topic named test with partition 1. And the ProducerPerformance process continues to run normally.
bin/kafka-topics.sh --describe --zookeeper localhost:2181
Topic: test PartitionCount: 1 ReplicationFactor: 1 Configs:
Topic: test Partition: 0 Leader: 0 Replicas: 0 Isr: 0
Some output of ProducerPerformance process.
1995632 records sent, 399126.4 records/sec (38.06 MB/sec), 378.6 ms avg latency, 435.0 ms max latency.
org.apache.kafka.common.errors.TimeoutException: Expiring 148 record(s) for test-1:120000 ms has passed since batch creation
.............................
org.apache.kafka.common.errors.TimeoutException: Expiring 148 record(s) for test-1:121774 ms has passed since batch creation
1711254 records sent, 342250.8 records/sec (32.64 MB/sec), 2324.5 ms avg latency, 123473.0 ms max latency.
Is that a bug?
Best Wishes,
Jiamei
IMPORTANT NOTICE: The contents of this email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please notify the sender immediately and do not disclose the contents to any other person, use it for any purpose, or store or copy the information in any medium. Thank you.
Re: Is that a bug?
Posted by Luke Chen <sh...@gmail.com>.
Yes, for more specific, it's the property: *allow.auto.create.topics, *which
default is set to true.
On Fri, May 22, 2020 at 5:07 PM M. Manna <ma...@gmail.com> wrote:
> Hey Xie,
>
> On Fri, 22 May 2020 at 08:31, Jiamei Xie <Ji...@arm.com> wrote:
>
> > Hi
> >
> > Kill all zookeeper and kafka process. Clear zookeeper and kafka data dir.
> > Restart zookeeper and kafka. If there are any active client. Topic used
> by
> > client will be auto-created.
> >
> > How to reproduce?
> >
> >
> > 1. Start zookeeper and kafka zookeeper and kafka config file.
> >
> > nohup bin/zookeeper-server-start.sh config/zookeeper.properties &
> >
> > nohup bin/kafka-server-start.sh config/server.properties &
> >
> >
> >
> > 1. Create topic test with 2 partitions
> >
> > bin/kafka-topics.sh --create --zookeeper localhost:2181 --topic test
> > --partitions 2 --replication-factor 1
> >
> >
> >
> > 1. Produce some data to topic test
> >
> > bin/kafka-producer-perf-test.sh --topic test --num-records 50000000
> > --record-size 100 --throughput=-1 --producer-props
> > bootstrap.servers=localhost:9092
> >
> >
> >
> > 1. Kill zookeeper and kafka. ProducerPerformance is still running.
> >
> > jps
> >
> > 21072 QuorumPeerMain
> >
> > 21704 ProducerPerformance
> >
> > 21230 Kafka
> >
> > 21854 Jps
> >
> > kill -9 21072 21230
> >
> >
> >
> > 1. Remove Zookeeper and Kafka data
> >
> > rm -rf /tmp/zookeeper/
> >
> > rm -rf /tmp/kafka-logs/
> >
> >
> >
> > 1. Start zookeeper and kafka
> >
> > nohup bin/zookeeper-server-start.sh config/zookeeper.properties &
> >
> > nohup bin/kafka-server-start.sh config/server.properties &
> >
> >
> >
> > 1. Check topic and you'll see there is topic named test with partition
> > 1. And the ProducerPerformance process continues to run normally.
> >
> > bin/kafka-topics.sh --describe --zookeeper localhost:2181
> >
> > Topic: test PartitionCount: 1 ReplicationFactor: 1 Configs:
> >
> > Topic: test Partition: 0 Leader: 0 Replicas: 0
> > Isr: 0
> >
> >
> > Some output of ProducerPerformance process.
> > 1995632 records sent, 399126.4 records/sec (38.06 MB/sec), 378.6 ms avg
> > latency, 435.0 ms max latency.
> > org.apache.kafka.common.errors.TimeoutException: Expiring 148 record(s)
> > for test-1:120000 ms has passed since batch creation
> > .............................
> > org.apache.kafka.common.errors.TimeoutException: Expiring 148 record(s)
> > for test-1:121774 ms has passed since batch creation
> > 1711254 records sent, 342250.8 records/sec (32.64 MB/sec), 2324.5 ms avg
> > latency, 123473.0 ms max latency.
> >
> >
> >
> > Is that a bug?
>
>
> How have you configured your server.properties and consumer.peoperties?
> Have you set auto creation ON?
>
> Regards,
>
> >
> >
> > Best Wishes,
> > Jiamei
> >
> > IMPORTANT NOTICE: The contents of this email and any attachments are
> > confidential and may also be privileged. If you are not the intended
> > recipient, please notify the sender immediately and do not disclose the
> > contents to any other person, use it for any purpose, or store or copy
> the
> > information in any medium. Thank you.
> >
>
Re: Is that a bug?
Posted by sunil chaudhari <su...@gmail.com>.
Simple..
Topic auto creation ON.
As soon as it encounters producer and consumers request for some topic then
it creates that topic automatically.
This is very common question and problem people come across, if they are
new. Same happened with me.😄
On Fri, 22 May 2020 at 3:28 PM, Jiamei Xie <Ji...@arm.com> wrote:
> Hi Manna, Luke
> I used the default configure files without any modification. The default
> value for allow.auto.create.topics is true according to
> https://kafka.apache.org/documentation/#brokerconfigs. Thanks for your
> reply.
>
> -----Original Message-----
> From: M. Manna <ma...@gmail.com>
> Sent: 2020年5月22日 16:50
> To: users@kafka.apache.org
> Subject: Re: Is that a bug?
>
> Hey Xie,
>
> On Fri, 22 May 2020 at 08:31, Jiamei Xie <Ji...@arm.com> wrote:
>
> > Hi
> >
> > Kill all zookeeper and kafka process. Clear zookeeper and kafka data dir.
> > Restart zookeeper and kafka. If there are any active client. Topic
> > used by client will be auto-created.
> >
> > How to reproduce?
> >
> >
> > 1. Start zookeeper and kafka zookeeper and kafka config file.
> >
> > nohup bin/zookeeper-server-start.sh config/zookeeper.properties &
> >
> > nohup bin/kafka-server-start.sh config/server.properties &
> >
> >
> >
> > 1. Create topic test with 2 partitions
> >
> > bin/kafka-topics.sh --create --zookeeper localhost:2181 --topic test
> > --partitions 2 --replication-factor 1
> >
> >
> >
> > 1. Produce some data to topic test
> >
> > bin/kafka-producer-perf-test.sh --topic test --num-records 50000000
> > --record-size 100 --throughput=-1 --producer-props
> > bootstrap.servers=localhost:9092
> >
> >
> >
> > 1. Kill zookeeper and kafka. ProducerPerformance is still running.
> >
> > jps
> >
> > 21072 QuorumPeerMain
> >
> > 21704 ProducerPerformance
> >
> > 21230 Kafka
> >
> > 21854 Jps
> >
> > kill -9 21072 21230
> >
> >
> >
> > 1. Remove Zookeeper and Kafka data
> >
> > rm -rf /tmp/zookeeper/
> >
> > rm -rf /tmp/kafka-logs/
> >
> >
> >
> > 1. Start zookeeper and kafka
> >
> > nohup bin/zookeeper-server-start.sh config/zookeeper.properties &
> >
> > nohup bin/kafka-server-start.sh config/server.properties &
> >
> >
> >
> > 1. Check topic and you'll see there is topic named test with
> > partition 1. And the ProducerPerformance process continues to run
> normally.
> >
> > bin/kafka-topics.sh --describe --zookeeper localhost:2181
> >
> > Topic: test PartitionCount: 1 ReplicationFactor: 1 Configs:
> >
> > Topic: test Partition: 0 Leader: 0 Replicas: 0
> > Isr: 0
> >
> >
> > Some output of ProducerPerformance process.
> > 1995632 records sent, 399126.4 records/sec (38.06 MB/sec), 378.6 ms
> > avg latency, 435.0 ms max latency.
> > org.apache.kafka.common.errors.TimeoutException: Expiring 148
> > record(s) for test-1:120000 ms has passed since batch creation
> > .............................
> > org.apache.kafka.common.errors.TimeoutException: Expiring 148
> > record(s) for test-1:121774 ms has passed since batch creation
> > 1711254 records sent, 342250.8 records/sec (32.64 MB/sec), 2324.5 ms
> > avg latency, 123473.0 ms max latency.
> >
> >
> >
> > Is that a bug?
>
>
> How have you configured your server.properties and consumer.peoperties?
> Have you set auto creation ON?
>
> Regards,
>
> >
> >
> > Best Wishes,
> > Jiamei
> >
> > IMPORTANT NOTICE: The contents of this email and any attachments are
> > confidential and may also be privileged. If you are not the intended
> > recipient, please notify the sender immediately and do not disclose
> > the contents to any other person, use it for any purpose, or store or
> > copy the information in any medium. Thank you.
> >
> IMPORTANT NOTICE: The contents of this email and any attachments are
> confidential and may also be privileged. If you are not the intended
> recipient, please notify the sender immediately and do not disclose the
> contents to any other person, use it for any purpose, or store or copy the
> information in any medium. Thank you.
>
RE: Is that a bug?
Posted by Jiamei Xie <Ji...@arm.com>.
Hi Manna, Luke
I used the default configure files without any modification. The default value for allow.auto.create.topics is true according to https://kafka.apache.org/documentation/#brokerconfigs. Thanks for your reply.
-----Original Message-----
From: M. Manna <ma...@gmail.com>
Sent: 2020年5月22日 16:50
To: users@kafka.apache.org
Subject: Re: Is that a bug?
Hey Xie,
On Fri, 22 May 2020 at 08:31, Jiamei Xie <Ji...@arm.com> wrote:
> Hi
>
> Kill all zookeeper and kafka process. Clear zookeeper and kafka data dir.
> Restart zookeeper and kafka. If there are any active client. Topic
> used by client will be auto-created.
>
> How to reproduce?
>
>
> 1. Start zookeeper and kafka zookeeper and kafka config file.
>
> nohup bin/zookeeper-server-start.sh config/zookeeper.properties &
>
> nohup bin/kafka-server-start.sh config/server.properties &
>
>
>
> 1. Create topic test with 2 partitions
>
> bin/kafka-topics.sh --create --zookeeper localhost:2181 --topic test
> --partitions 2 --replication-factor 1
>
>
>
> 1. Produce some data to topic test
>
> bin/kafka-producer-perf-test.sh --topic test --num-records 50000000
> --record-size 100 --throughput=-1 --producer-props
> bootstrap.servers=localhost:9092
>
>
>
> 1. Kill zookeeper and kafka. ProducerPerformance is still running.
>
> jps
>
> 21072 QuorumPeerMain
>
> 21704 ProducerPerformance
>
> 21230 Kafka
>
> 21854 Jps
>
> kill -9 21072 21230
>
>
>
> 1. Remove Zookeeper and Kafka data
>
> rm -rf /tmp/zookeeper/
>
> rm -rf /tmp/kafka-logs/
>
>
>
> 1. Start zookeeper and kafka
>
> nohup bin/zookeeper-server-start.sh config/zookeeper.properties &
>
> nohup bin/kafka-server-start.sh config/server.properties &
>
>
>
> 1. Check topic and you'll see there is topic named test with
> partition 1. And the ProducerPerformance process continues to run normally.
>
> bin/kafka-topics.sh --describe --zookeeper localhost:2181
>
> Topic: test PartitionCount: 1 ReplicationFactor: 1 Configs:
>
> Topic: test Partition: 0 Leader: 0 Replicas: 0
> Isr: 0
>
>
> Some output of ProducerPerformance process.
> 1995632 records sent, 399126.4 records/sec (38.06 MB/sec), 378.6 ms
> avg latency, 435.0 ms max latency.
> org.apache.kafka.common.errors.TimeoutException: Expiring 148
> record(s) for test-1:120000 ms has passed since batch creation
> .............................
> org.apache.kafka.common.errors.TimeoutException: Expiring 148
> record(s) for test-1:121774 ms has passed since batch creation
> 1711254 records sent, 342250.8 records/sec (32.64 MB/sec), 2324.5 ms
> avg latency, 123473.0 ms max latency.
>
>
>
> Is that a bug?
How have you configured your server.properties and consumer.peoperties?
Have you set auto creation ON?
Regards,
>
>
> Best Wishes,
> Jiamei
>
> IMPORTANT NOTICE: The contents of this email and any attachments are
> confidential and may also be privileged. If you are not the intended
> recipient, please notify the sender immediately and do not disclose
> the contents to any other person, use it for any purpose, or store or
> copy the information in any medium. Thank you.
>
IMPORTANT NOTICE: The contents of this email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please notify the sender immediately and do not disclose the contents to any other person, use it for any purpose, or store or copy the information in any medium. Thank you.
Re: Is that a bug?
Posted by "M. Manna" <ma...@gmail.com>.
Hey Xie,
On Fri, 22 May 2020 at 08:31, Jiamei Xie <Ji...@arm.com> wrote:
> Hi
>
> Kill all zookeeper and kafka process. Clear zookeeper and kafka data dir.
> Restart zookeeper and kafka. If there are any active client. Topic used by
> client will be auto-created.
>
> How to reproduce?
>
>
> 1. Start zookeeper and kafka zookeeper and kafka config file.
>
> nohup bin/zookeeper-server-start.sh config/zookeeper.properties &
>
> nohup bin/kafka-server-start.sh config/server.properties &
>
>
>
> 1. Create topic test with 2 partitions
>
> bin/kafka-topics.sh --create --zookeeper localhost:2181 --topic test
> --partitions 2 --replication-factor 1
>
>
>
> 1. Produce some data to topic test
>
> bin/kafka-producer-perf-test.sh --topic test --num-records 50000000
> --record-size 100 --throughput=-1 --producer-props
> bootstrap.servers=localhost:9092
>
>
>
> 1. Kill zookeeper and kafka. ProducerPerformance is still running.
>
> jps
>
> 21072 QuorumPeerMain
>
> 21704 ProducerPerformance
>
> 21230 Kafka
>
> 21854 Jps
>
> kill -9 21072 21230
>
>
>
> 1. Remove Zookeeper and Kafka data
>
> rm -rf /tmp/zookeeper/
>
> rm -rf /tmp/kafka-logs/
>
>
>
> 1. Start zookeeper and kafka
>
> nohup bin/zookeeper-server-start.sh config/zookeeper.properties &
>
> nohup bin/kafka-server-start.sh config/server.properties &
>
>
>
> 1. Check topic and you'll see there is topic named test with partition
> 1. And the ProducerPerformance process continues to run normally.
>
> bin/kafka-topics.sh --describe --zookeeper localhost:2181
>
> Topic: test PartitionCount: 1 ReplicationFactor: 1 Configs:
>
> Topic: test Partition: 0 Leader: 0 Replicas: 0
> Isr: 0
>
>
> Some output of ProducerPerformance process.
> 1995632 records sent, 399126.4 records/sec (38.06 MB/sec), 378.6 ms avg
> latency, 435.0 ms max latency.
> org.apache.kafka.common.errors.TimeoutException: Expiring 148 record(s)
> for test-1:120000 ms has passed since batch creation
> .............................
> org.apache.kafka.common.errors.TimeoutException: Expiring 148 record(s)
> for test-1:121774 ms has passed since batch creation
> 1711254 records sent, 342250.8 records/sec (32.64 MB/sec), 2324.5 ms avg
> latency, 123473.0 ms max latency.
>
>
>
> Is that a bug?
How have you configured your server.properties and consumer.peoperties?
Have you set auto creation ON?
Regards,
>
>
> Best Wishes,
> Jiamei
>
> IMPORTANT NOTICE: The contents of this email and any attachments are
> confidential and may also be privileged. If you are not the intended
> recipient, please notify the sender immediately and do not disclose the
> contents to any other person, use it for any purpose, or store or copy the
> information in any medium. Thank you.
>