Re: Kafka Topics

[Edward Capriolo <ed...@gmail.com>]

On Thursday, December 30, 2021, Suresh Chidambaram <ch...@gmail.com>
wrote:

> Hi Ola,
>
> I would suggest you can go with single Topic with multiple partitions. Once
> the data gets received from the Topic, you can do a DB update kind of a
> stuff to store the data , then use the data for analysing.
>
> Also, the below URL can be used to do the Topic sizing.
>
> eventsizer.io
>
>
> Thanks
> C Suresh
>
> On Thursday, December 30, 2021, Ola Bissani <ol...@easysoft.com.lb>
> wrote:
>
> > Dears,
> >
> > I'm looking for a way to get real-time updates using my service, I
> believe
> > kafka is the way to go but I still have an issue on how to use it.
> >
> > My system gets data from devices using GPRS, I then read this data and
> > analyze it to check what action I should do afterwards. I need the
> > analyzing step to be as fast as possible. I was thinking of two options:
> >
> > The first option is to gather all the data sent from all the devices into
> > one huge topic and then getting all the data from this topic and
> analyzing
> > it. The downside of this option is that the data analysis step is
> delaying
> > my work since I was to loop through the topic data, on the other hand the
> > advantage is that I have a manageable number of topics ( only 1 topic).
> >
> > The other option is to divide the data I'm gathering into several small
> > topics by allowing each device to have its own topic, take into
> > consideration that the number of devices is large, I'm talking about more
> > that 5000 devices. The downside of this option is that I have thousands
> of
> > topics, where the advantage is that each topic will have a manageable
> > amount of data allowing me to get my analysis done in much more
> reasonable
> > time.
> >
> > Can you advise on what option is better and whether there is a third
> > option that I'm not considering,
> >
> >
> >
> > Best Regards
> > Ola Bissani
> > Developer Manager
> > Easysoft
> > Mobile Lebanon   : +961       3 61 16 90
> > Office Lebanon      :+961       1 33 55 15/17
> > E mail:     ola.bissani@easysoft.com.lb
> > web site:www.easysoft.com.lb
> > "Tailored to Perfection"
> >
> >
> > The information transmitted is intended only for the person or entity to
> > which it is addressed and it may contain proprietary,
> > business-confidential, and/or legally privileged information. If you are
> > not the intended recipient of this email you are hereby notified that any
> > use, review, retransmission, dissemination, distribution, reproduction or
> > any other action taken in reliance upon this email is strictly
> prohibited.
> > If you have received this email in error, please contact the sender and
> > delete this email and its contents from any computer. Any views expressed
> > in this email are those of the individual sender and may not necessarily
> > reflect the views of the company.
> >
> >                Please consider the environmet before printing this email.
> >
> >
> > -----Original Message-----
> > From: Wes Peng <pe...@gmail.com>
> > Sent: Thursday, December 23, 2021 10:11 PM
> > To: users@kafka.apache.org
> > Subject: Re: Kafka Topics
> >
> > That depends on your resources such as ram, disk etc. General speaking
> > there is no problem.
> >
> > Regards
> >
> >
> > > Dears,
> > >
> > > I'm new to using Kafka, and I was wondering up to how many topics can
> > > Kafka Handle. I'm trying to use Kafka but using it I will be obliged
> > > to create thousands of topics to keep up with my data. Will Kafka on
> > > my server handle this kind of data?
> > >
> > > Thank you,
> > >
> > >
> > > Best Regards
> > >
> > > Ola Bissani
> > >
> > > Developer Manager
> > >
> > > Easysoft
> > >
> > > Mobile Lebanon   : +961       3 61 16 90
> > >
> > > Office Lebanon      :+961       1 33 55 15/17
> > >
> > > E mail:     ola.bissani@easysoft.com.lb
> > >
> > > web site:www.easysoft.com.lb
> > >
> > > "Tailored to Perfection"
> > >
> > >
> > >
> > >
> > > The information transmitted is intended only for the person or entity
> > > to which it is addressed and it may contain proprietary,
> > > business-confidential, and/or legally privileged information. If you
> > > are not the intended recipient of this email you are hereby notified
> > > that any use, review, retransmission, dissemination, distribution,
> > > reproduction or any other action taken in reliance upon this email is
> > strictly prohibited.
> > > If you have received this email in error, please contact the sender
> > > and delete this email and its contents from any computer. Any views
> > > expressed in this email are those of the individual sender and may not
> > > necessarily reflect the views of the company.
> > >
> > >                Please consider the environmet before printing this
> email.
> > >
> > >
> >
> >
>


Please dont listen to the folks that say "you can have a smany as you
want". You cant. Here is why.

Each topic is divided into partitions,  each partition is replicated , each
partition replica lives on a disk.

The higher your retention .. days weeks you have to keep more and more
files around.

And remember Kafka occasionally has to startup.. and when it starts it scan
though thousands of file which makes your startup take a long time.

So the short answer is not to many .especially if you intend on keeping the
data and you have fee brokers.

Of yes also zooleeper data has to fit in ram. And the way data is
structured their imposes other limits.



-- 
Sorry this was sent from mobile. Will do less grammar and spell check than
usual.