Limitation on kafka topic size

[Suchi Amalapurapu <su...@bloomreach.com>]

All
Are there any limitations on max topic size in Kafka. How large can a topic
get?
What are the implications on resources, latencies, performance and
feasability?
What are the pros and cons of using a single large topic vs multiple
smaller topics?
Suchi

Re: Limitation on kafka topic size

[Guozhang Wang <wa...@gmail.com>]

Hello Suchi,

Each topic will have its own partitions and each partition is represented
as a list of file segments. Hence having multiple smaller topics means you
will have # topics * avg. # partitions_per_topic file handlers.

Volume is usually not a problem for Kafka topics, however, if you have
consumers consuming not at relatively the tail (i.e., pretty caught up with
the produce rate) the it will hit the disk but not the file cache. This
will incur large latency not only for these consumers but also for others
since they are sharing the socket server processors.

So if all your producer/consumers of the topic have roughly the same
consuming behavior, you can go option 1; otherwise it is better to keep
groups of similar produce/consume behavioral cases separate topics.

Guozhang


On Mon, Oct 28, 2013 at 12:24 PM, Suchi Amalapurapu <su...@bloomreach.com>wrote:

> We are using kafka 0.7.2
> Suchi
>
>
> On Tue, Oct 29, 2013 at 12:53 AM, Suchi Amalapurapu <suchi@bloomreach.com
> >wrote:
>
> > All
> > Are there any limitations on max topic size in Kafka. How large can a
> > topic get?
> > What are the implications on resources, latencies, performance and
> > feasability?
> > What are the pros and cons of using a single large topic vs multiple
> > smaller topics?
> > Suchi
> >
>



-- 
-- Guozhang

Re: Limitation on kafka topic size

[Suchi Amalapurapu <su...@bloomreach.com>]

We are using kafka 0.7.2
Suchi


On Tue, Oct 29, 2013 at 12:53 AM, Suchi Amalapurapu <su...@bloomreach.com>wrote:

> All
> Are there any limitations on max topic size in Kafka. How large can a
> topic get?
> What are the implications on resources, latencies, performance and
> feasability?
> What are the pros and cons of using a single large topic vs multiple
> smaller topics?
> Suchi
>