You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by adrien ruffie <ad...@hotmail.fr> on 2018/02/04 19:14:09 UTC
Strange Topic ...
Hello all,
I'm a beginner in Kafka and this morning when I try some tests and when running this following cmd:
./bin kafka-topics.sh --zookeeper localhost:2181 --describe
I understand my 3 created topic like "customer-topic", "streams-plaintext-input", and "streams-wordcount-output"
But I already get this following output, why __consumer_offsets have 50 partitions ! I never created it ... do you know this beavior ?
Topic:__consumer_offsets PartitionCount:50 ReplicationFactor:1 Configs:segment.bytes=104857600,cleanup.policy=compact,compression.type=produ$
Topic: __consumer_offsets Partition: 0 Leader: 0 Replicas: 0 Isr: 0
Topic: __consumer_offsets Partition: 1 Leader: 0 Replicas: 0 Isr: 0
Topic: __consumer_offsets Partition: 2 Leader: 0 Replicas: 0 Isr: 0
Topic: __consumer_offsets Partition: 3 Leader: 0 Replicas: 0 Isr: 0
Topic: __consumer_offsets Partition: 4 Leader: 0 Replicas: 0 Isr: 0
Topic: __consumer_offsets Partition: 5 Leader: 0 Replicas: 0 Isr: 0
....
Topic: __consumer_offsets Partition: 49 Leader: 0 Replicas: 0 Isr: 0
Topic:customer-topic PartitionCount:1 ReplicationFactor:1 Configs:
Topic: customer-topic Partition: 0 Leader: 0 Replicas: 0 Isr: 0
Topic:streams-plaintext-input PartitionCount:1 ReplicationFactor:1 Configs:
Topic: streams-plaintext-input Partition: 0 Leader: 0 Replicas: 0 Isr: 0
Topic:streams-wordcount-output PartitionCount:1 ReplicationFactor:1 Configs:cleanup.policy=compact
Topic: streams-wordcount-output Partition: 0 Leader: 0 Replicas: 0 Isr: 0
Thank and bests regards,
Adrien
RE: Strange Topic ...
Posted by adrien ruffie <ad...@hotmail.fr>.
Hello Ted,
I use Kafka_2.11-1.0.0
And I log file I only have this:
[2018-01-27 23:12:09,545] INFO Shutting down the log cleaner. (kafka.log.LogCleaner)
[2018-01-27 23:12:09,545] INFO [kafka-log-cleaner-thread-0]: Shutting down (kafka.log.LogCleaner)
[2018-01-27 23:12:09,546] INFO [kafka-log-cleaner-thread-0]: Stopped (kafka.log.LogCleaner)
[2018-01-27 23:12:09,546] INFO [kafka-log-cleaner-thread-0]: Shutdown completed (kafka.log.LogCleaner)
And the topic appeared this morning, and log log-cleaner.log file, don't log anything after 01-27 ...
________________________________
De : Ted Yu <yu...@gmail.com>
Envoyé : dimanche 4 février 2018 20:37:59
À : users@kafka.apache.org
Objet : Re: Strange Topic ...
Which Kafka version are you using ?
Older versions of kafka (0.10 and prior) had some bugs in the log-cleaner
thread that might sometimes cause it to crash.
Please check the log-cleaner.log file to see if there was some clue.
Cheers
On Sun, Feb 4, 2018 at 11:14 AM, adrien ruffie <ad...@hotmail.fr>
wrote:
> Hello all,
>
>
> I'm a beginner in Kafka and this morning when I try some tests and when
> running this following cmd:
>
> ./bin kafka-topics.sh --zookeeper localhost:2181 --describe
>
>
> I understand my 3 created topic like "customer-topic",
> "streams-plaintext-input", and "streams-wordcount-output"
>
>
> But I already get this following output, why __consumer_offsets have 50
> partitions ! I never created it ... do you know this beavior ?
>
>
> Topic:__consumer_offsets PartitionCount:50
> ReplicationFactor:1 Configs:segment.bytes=104857600,cleanup.policy=
> compact,compression.type=produ$
> Topic: __consumer_offsets Partition: 0 Leader: 0
> Replicas: 0 Isr: 0
> Topic: __consumer_offsets Partition: 1 Leader: 0
> Replicas: 0 Isr: 0
> Topic: __consumer_offsets Partition: 2 Leader: 0
> Replicas: 0 Isr: 0
> Topic: __consumer_offsets Partition: 3 Leader: 0
> Replicas: 0 Isr: 0
> Topic: __consumer_offsets Partition: 4 Leader: 0
> Replicas: 0 Isr: 0
> Topic: __consumer_offsets Partition: 5 Leader: 0
> Replicas: 0 Isr: 0
> ....
> Topic: __consumer_offsets Partition: 49 Leader: 0
> Replicas: 0 Isr: 0
>
>
> Topic:customer-topic PartitionCount:1 ReplicationFactor:1
> Configs:
> Topic: customer-topic Partition: 0 Leader: 0 Replicas:
> 0 Isr: 0
> Topic:streams-plaintext-input PartitionCount:1
> ReplicationFactor:1 Configs:
> Topic: streams-plaintext-input Partition: 0 Leader: 0
> Replicas: 0 Isr: 0
> Topic:streams-wordcount-output PartitionCount:1
> ReplicationFactor:1 Configs:cleanup.policy=compact
> Topic: streams-wordcount-output Partition: 0 Leader: 0
> Replicas: 0 Isr: 0
>
>
> Thank and bests regards,
>
> Adrien
>
>
>
>
Re: Strange Topic ...
Posted by naresh Goud <na...@gmail.com>.
This is the topic used and created by Kafka internally to store consumer
offsets while use consumer programs running.
Thank you,
Naresh
On Sun, Feb 4, 2018 at 1:38 PM Ted Yu <yu...@gmail.com> wrote:
> Which Kafka version are you using ?
> Older versions of kafka (0.10 and prior) had some bugs in the log-cleaner
> thread that might sometimes cause it to crash.
>
> Please check the log-cleaner.log file to see if there was some clue.
>
> Cheers
>
> On Sun, Feb 4, 2018 at 11:14 AM, adrien ruffie <ad...@hotmail.fr>
> wrote:
>
> > Hello all,
> >
> >
> > I'm a beginner in Kafka and this morning when I try some tests and when
> > running this following cmd:
> >
> > ./bin kafka-topics.sh --zookeeper localhost:2181 --describe
> >
> >
> > I understand my 3 created topic like "customer-topic",
> > "streams-plaintext-input", and "streams-wordcount-output"
> >
> >
> > But I already get this following output, why __consumer_offsets have 50
> > partitions ! I never created it ... do you know this beavior ?
> >
> >
> > Topic:__consumer_offsets PartitionCount:50
> > ReplicationFactor:1 Configs:segment.bytes=104857600,cleanup.policy=
> > compact,compression.type=produ$
> > Topic: __consumer_offsets Partition: 0 Leader: 0
> > Replicas: 0 Isr: 0
> > Topic: __consumer_offsets Partition: 1 Leader: 0
> > Replicas: 0 Isr: 0
> > Topic: __consumer_offsets Partition: 2 Leader: 0
> > Replicas: 0 Isr: 0
> > Topic: __consumer_offsets Partition: 3 Leader: 0
> > Replicas: 0 Isr: 0
> > Topic: __consumer_offsets Partition: 4 Leader: 0
> > Replicas: 0 Isr: 0
> > Topic: __consumer_offsets Partition: 5 Leader: 0
> > Replicas: 0 Isr: 0
> > ....
> > Topic: __consumer_offsets Partition: 49 Leader: 0
> > Replicas: 0 Isr: 0
> >
> >
> > Topic:customer-topic PartitionCount:1 ReplicationFactor:1
> > Configs:
> > Topic: customer-topic Partition: 0 Leader: 0 Replicas:
> > 0 Isr: 0
> > Topic:streams-plaintext-input PartitionCount:1
> > ReplicationFactor:1 Configs:
> > Topic: streams-plaintext-input Partition: 0 Leader: 0
> > Replicas: 0 Isr: 0
> > Topic:streams-wordcount-output PartitionCount:1
> > ReplicationFactor:1 Configs:cleanup.policy=compact
> > Topic: streams-wordcount-output Partition: 0 Leader: 0
> > Replicas: 0 Isr: 0
> >
> >
> > Thank and bests regards,
> >
> > Adrien
> >
> >
> >
> >
>
Re: Strange Topic ...
Posted by Ted Yu <yu...@gmail.com>.
Which Kafka version are you using ?
Older versions of kafka (0.10 and prior) had some bugs in the log-cleaner
thread that might sometimes cause it to crash.
Please check the log-cleaner.log file to see if there was some clue.
Cheers
On Sun, Feb 4, 2018 at 11:14 AM, adrien ruffie <ad...@hotmail.fr>
wrote:
> Hello all,
>
>
> I'm a beginner in Kafka and this morning when I try some tests and when
> running this following cmd:
>
> ./bin kafka-topics.sh --zookeeper localhost:2181 --describe
>
>
> I understand my 3 created topic like "customer-topic",
> "streams-plaintext-input", and "streams-wordcount-output"
>
>
> But I already get this following output, why __consumer_offsets have 50
> partitions ! I never created it ... do you know this beavior ?
>
>
> Topic:__consumer_offsets PartitionCount:50
> ReplicationFactor:1 Configs:segment.bytes=104857600,cleanup.policy=
> compact,compression.type=produ$
> Topic: __consumer_offsets Partition: 0 Leader: 0
> Replicas: 0 Isr: 0
> Topic: __consumer_offsets Partition: 1 Leader: 0
> Replicas: 0 Isr: 0
> Topic: __consumer_offsets Partition: 2 Leader: 0
> Replicas: 0 Isr: 0
> Topic: __consumer_offsets Partition: 3 Leader: 0
> Replicas: 0 Isr: 0
> Topic: __consumer_offsets Partition: 4 Leader: 0
> Replicas: 0 Isr: 0
> Topic: __consumer_offsets Partition: 5 Leader: 0
> Replicas: 0 Isr: 0
> ....
> Topic: __consumer_offsets Partition: 49 Leader: 0
> Replicas: 0 Isr: 0
>
>
> Topic:customer-topic PartitionCount:1 ReplicationFactor:1
> Configs:
> Topic: customer-topic Partition: 0 Leader: 0 Replicas:
> 0 Isr: 0
> Topic:streams-plaintext-input PartitionCount:1
> ReplicationFactor:1 Configs:
> Topic: streams-plaintext-input Partition: 0 Leader: 0
> Replicas: 0 Isr: 0
> Topic:streams-wordcount-output PartitionCount:1
> ReplicationFactor:1 Configs:cleanup.policy=compact
> Topic: streams-wordcount-output Partition: 0 Leader: 0
> Replicas: 0 Isr: 0
>
>
> Thank and bests regards,
>
> Adrien
>
>
>
>