Re: Num of partitions != num.partitions in config

[Guozhang Wang <wa...@gmail.com>]

Hi Otis,

The partition segment files are only created when the first message of this
partition as arrived, so 1) if your producer send keyed messages which
specifies their partitions or 2) your producer has not sent enough data (in
0.8.x it stick to one partition for a while before going to the next one in
round robin) then later partitions may not be created. To verify that this
topic does have 24 partitions you can use bin/kafka-topics.sh tool to check
its partition distribution.

Guozhang


On Sat, May 31, 2014 at 12:46 PM, Otis Gospodnetic <
otis.gospodnetic@gmail.com> wrote:

> Hi,
>
> I have Kafka 0.8.1.1 running and configured to have 24 partitions:
> $ grep partition /usr/lib/kafka_2.8.0-0.8.1.1/config/server.properties
> # The default number of log partitions per topic. More partitions allow
> greater
> num.partitions=24
>
> But some of the topics don't seem to have 24 partitions, like this one:
>
> $ ls -la spm_af_alerts*
>
> drwxrwxr-x  2 ec2-user ec2-user   4096 May 30 07:43 spm_af_alerts_topic-0
> drwxrwxr-x  2 ec2-user ec2-user   4096 May 30 07:43 spm_af_alerts_topic-1
> drwxrwxr-x  2 ec2-user ec2-user   4096 May 30 07:43 spm_af_alerts_topic-10
> drwxrwxr-x  2 ec2-user ec2-user   4096 May 30 07:43 spm_af_alerts_topic-11
> drwxrwxr-x  2 ec2-user ec2-user   4096 May 30 07:43 spm_af_alerts_topic-2
> drwxrwxr-x  2 ec2-user ec2-user   4096 May 30 07:43 spm_af_alerts_topic-3
> drwxrwxr-x  2 ec2-user ec2-user   4096 May 30 07:43 spm_af_alerts_topic-4
> drwxrwxr-x  2 ec2-user ec2-user   4096 May 30 07:43 spm_af_alerts_topic-5
> drwxrwxr-x  2 ec2-user ec2-user   4096 May 30 07:43 spm_af_alerts_topic-6
> drwxrwxr-x  2 ec2-user ec2-user   4096 May 31 14:49 spm_af_alerts_topic-7
> drwxrwxr-x  2 ec2-user ec2-user   4096 May 30 07:43 spm_af_alerts_topic-8
> drwxrwxr-x  2 ec2-user ec2-user   4096 May 30 07:43 spm_af_alerts_topic-9
>
> How is this possible?  That -N at the end represents the partition number,
> right?
>
> Thanks,
> Otis
> --
> Performance Monitoring * Log Analytics * Search Analytics
> Solr & Elasticsearch Support * http://sematext.com/
>



-- 
-- Guozhang