Re: the impact of partition number

[Jun Rao <ju...@confluent.io>]

All writes to a single partition are synchronized on the broker. So, the
more partitions the more parallelism (and better throughput) you have.

Thanks,

Jun

On Mon, Dec 22, 2014 at 1:24 PM, Sa Li <sa...@gmail.com> wrote:

> Hi, All
>
> I've run bin/kafka-producer-perf-test.sh on our kafka-production cluster, I
> found the number of partitions really have huge impacts on the producer
> performance, see:
>
> start.time, end.time, compression, message.size, batch.size,
> total.data.sent.in.MB, MB.sec, total.data.sent.in.nMsg, nMsg.sec
> 2014-12-22 19:53:27:392, 2014-12-22 19:54:25:581, 1, 3000, 200, 2861.02,
> 49.1678, 1000000, 17185.3787
> 2014-12-22 19:55:27:048, 2014-12-22 19:56:23:318, 1, 3000, 200, 2861.02,
> 50.8446, 1000000, 17771.4590
> 2014-12-22 19:58:09:466, 2014-12-22 19:59:05:068, 1, 3000, 200, 2861.02,
> 51.4554, 1000000, 17984.9646
> 2014-12-22 19:59:40:389, 2014-12-22 20:00:28:646, 1, 3000, 200, 2861.02,
> 59.2872, 1000000, 20722.3822
> 2014-12-22 20:02:41:993, 2014-12-22 20:03:22:481, 1, 3000, 200, 2861.02,
> 70.6635, 1000000, 24698.6762
> 2014-12-22 20:03:47:594, 2014-12-22 20:04:26:238, 1, 3000, 200, 2861.02,
> 74.0354, 1000000, 25877.2384
> 2014-12-22 20:11:49:492, 2014-12-22 20:12:25:843, 1, 3000, 200, 2861.02,
> 78.7055, 1000000, 27509.5596
> 2014-12-22 20:12:53:290, 2014-12-22 20:13:29:746, 1, 3000, 200, 2861.02,
> 78.4788, 1000000, 27430.3270
> 2014-12-22 20:13:53:194, 2014-12-22 20:14:29:470, 1, 3000, 200, 2861.02,
> 78.8682, 1000000, 27566.4351
> 2014-12-22 20:14:51:491, 2014-12-22 20:15:25:451, 1, 3000, 200, 2861.02,
> 84.2468, 1000000, 29446.4075
> 2014-12-22 20:16:51:369, 2014-12-22 20:17:27:452, 1, 3000, 200, 2861.02,
> 79.2901, 1000000, 27713.8819
> 2014-12-22 20:17:57:882, 2014-12-22 20:18:33:957, 1, 3000, 200, 2861.02,
> 79.3076, 1000000, 27720.0277
>
>
> The number of partitions above are from 1 to 12, I wonder why it has such
> big difference?
>
> thanks
>
> --
>
> Alec Li
>