You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by Laxmi Narayan NIT DGP <ni...@gmail.com> on 2017/01/03 06:55:11 UTC
adding more partition
Hi,
Does adding more partitions to kafka will help in scaling ?
*Regards,*
*Laxmi Narayan Patel*
*MCA NIT Durgapur (2011-2014)*
*Mob:- 9741292048,8345847473 *
RE: adding more partition
Posted by "Tauzell, Dave" <Da...@surescripts.com>.
Kafka persists messages to disk (would be SSD if that is what you have). However, if you have a large enough memory and are pulling off messages quickly, then the receivers will likely get the messages directly from memory and the write to disk will happen asynchronously in the background. Because Kafka does not wait for the OS to sync to disk before acknowledging receipt you can get data loss which is why Kafka also has the concept of having backup partitions.
-Dave
-----Original Message-----
From: Laxmi Narayan NIT DGP [mailto:nit.dgp673@gmail.com]
Sent: Tuesday, January 3, 2017 1:26 AM
To: users@kafka.apache.org
Subject: Re: adding more partition
where does kafka runs RAM or SSD ?
My question how to delete message frm Kafka topics.
Is message expire is only way ?
*Regards,*
*Laxmi Narayan Patel*
*MCA NIT Durgapur (2011-2014)*
*Mob:- 9741292048,8345847473 *
On Tue, Jan 3, 2017 at 12:46 PM, Kunal Gupta <ku...@cube26.com> wrote:
> For your Question 1 No, Message will be read only once by anyone of
> the consumer in a consumer group
>
> For your Question 2 Yes, Because each consumer group will read a
> message exactly once. Kafka is meant for Guaranteed Message Processing
>
> There are three ways of Guaranteed Message Processing 1) Exactly once
> 2) At least Once 3) At most once you can get information well in
> google.
>
> If you set auto commit true then message will not be deleted, it all
> deals with consumer offset for reading a topic. At which offset
> consumer group has read the topic.
>
> *Thanks, Kunal*
> *+91-9958189589*
> *Data Analyst*
> *First Paper Publication : **http://dl.acm.org/citation.cfm?id=2790798
> <http://dl.acm.org/citation.cfm?id=2790798> * *Second Paper
> Publication : http://dl.acm.org/citation.cfm?id=2947432
> <http://dl.acm.org/citation.cfm?id=2947432>*
> *Blog:- **http://learnhardwithkunalgupta.blogspot.in
> <http://learnhardwithkunalgupta.blogspot.in> *
>
>
>
>
>
> On Tue, Jan 3, 2017 at 12:41 PM, Laxmi Narayan NIT DGP <
> nit.dgp673@gmail.com
> > wrote:
>
> > Hi ,
> >
> > If I have kafka cluster along with replication .
> >
> > Say cluster of 3 kafka-server and replication factor of message with 2.
> >
> > Que 1:
> >
> > I have 1-consumer group but 2 consumer in one group reading from
> > 1-Topic,
> >
> > If I enable auto commit true in consumer , will same message will be
> > read twice ?
> >
> > Que2:
> >
> > If I have 2 consumer group reading from 1-Topic
> >
> > If I enable auto commit true in consumer , will same message will be
> > read twice ?
> >
> >
> > and If I set auto-commit true will message will be deleted ?
> >
> >
> >
> >
> >
> >
> > *Regards,*
> > *Laxmi Narayan Patel*
> > *MCA NIT Durgapur (2011-2014)*
> > *Mob:- 9741292048,8345847473 *
> >
> > On Tue, Jan 3, 2017 at 12:34 PM, Kunal Gupta
> > <ku...@cube26.com>
> > wrote:
> >
> > > I don't think so because more partition can led to unavailability
> > > but
> yes
> > > can led to higher throughput. But it cause more problems like
> increasing
> > > end to end latency, requires more open file handler and require
> > > more
> > memory
> > > at client side.
> > >
> > > *Thanks, Kunal*
> > > *+91-9958189589*
> > > *Data Analyst*
> > > *First Paper Publication :
> > > **http://dl.acm.org/citation.cfm?id=2790798
> > > <http://dl.acm.org/citation.cfm?id=2790798> * *Second Paper
> > > Publication : http://dl.acm.org/citation.cfm?id=2947432
> > > <http://dl.acm.org/citation.cfm?id=2947432>*
> > > *Blog:- **http://learnhardwithkunalgupta.blogspot.in
> > > <http://learnhardwithkunalgupta.blogspot.in> *
> > >
> > >
> > >
> > >
> > >
> > > On Tue, Jan 3, 2017 at 12:25 PM, Laxmi Narayan NIT DGP <
> > > nit.dgp673@gmail.com
> > > > wrote:
> > >
> > > > Hi,
> > > >
> > > > Does adding more partitions to kafka will help in scaling ?
> > > >
> > > >
> > > > *Regards,*
> > > > *Laxmi Narayan Patel*
> > > > *MCA NIT Durgapur (2011-2014)*
> > > > *Mob:- 9741292048,8345847473 *
> > > >
> > >
> >
>
This e-mail and any files transmitted with it are confidential, may contain sensitive information, and are intended solely for the use of the individual or entity to whom they are addressed. If you have received this e-mail in error, please notify the sender by reply e-mail immediately and destroy all copies of the e-mail and any attachments.
Re: adding more partition
Posted by Laxmi Narayan NIT DGP <ni...@gmail.com>.
where does kafka runs RAM or SSD ?
My question how to delete message frm Kafka topics.
Is message expire is only way ?
*Regards,*
*Laxmi Narayan Patel*
*MCA NIT Durgapur (2011-2014)*
*Mob:- 9741292048,8345847473 *
On Tue, Jan 3, 2017 at 12:46 PM, Kunal Gupta <ku...@cube26.com> wrote:
> For your Question 1 No, Message will be read only once by anyone of the
> consumer in a consumer group
>
> For your Question 2 Yes, Because each consumer group will read a message
> exactly once. Kafka is meant for Guaranteed Message Processing
>
> There are three ways of Guaranteed Message Processing 1) Exactly once 2) At
> least Once 3) At most once
> you can get information well in google.
>
> If you set auto commit true then message will not be deleted, it all deals
> with consumer offset for reading a topic. At which offset consumer group
> has read the topic.
>
> *Thanks, Kunal*
> *+91-9958189589*
> *Data Analyst*
> *First Paper Publication : **http://dl.acm.org/citation.cfm?id=2790798
> <http://dl.acm.org/citation.cfm?id=2790798> *
> *Second Paper Publication : http://dl.acm.org/citation.cfm?id=2947432
> <http://dl.acm.org/citation.cfm?id=2947432>*
> *Blog:- **http://learnhardwithkunalgupta.blogspot.in
> <http://learnhardwithkunalgupta.blogspot.in> *
>
>
>
>
>
> On Tue, Jan 3, 2017 at 12:41 PM, Laxmi Narayan NIT DGP <
> nit.dgp673@gmail.com
> > wrote:
>
> > Hi ,
> >
> > If I have kafka cluster along with replication .
> >
> > Say cluster of 3 kafka-server and replication factor of message with 2.
> >
> > Que 1:
> >
> > I have 1-consumer group but 2 consumer in one group reading from 1-Topic,
> >
> > If I enable auto commit true in consumer , will same message will be read
> > twice ?
> >
> > Que2:
> >
> > If I have 2 consumer group reading from 1-Topic
> >
> > If I enable auto commit true in consumer , will same message will be read
> > twice ?
> >
> >
> > and If I set auto-commit true will message will be deleted ?
> >
> >
> >
> >
> >
> >
> > *Regards,*
> > *Laxmi Narayan Patel*
> > *MCA NIT Durgapur (2011-2014)*
> > *Mob:- 9741292048,8345847473 *
> >
> > On Tue, Jan 3, 2017 at 12:34 PM, Kunal Gupta <ku...@cube26.com>
> > wrote:
> >
> > > I don't think so because more partition can led to unavailability but
> yes
> > > can led to higher throughput. But it cause more problems like
> increasing
> > > end to end latency, requires more open file handler and require more
> > memory
> > > at client side.
> > >
> > > *Thanks, Kunal*
> > > *+91-9958189589*
> > > *Data Analyst*
> > > *First Paper Publication : **http://dl.acm.org/citation.cfm?id=2790798
> > > <http://dl.acm.org/citation.cfm?id=2790798> *
> > > *Second Paper Publication : http://dl.acm.org/citation.cfm?id=2947432
> > > <http://dl.acm.org/citation.cfm?id=2947432>*
> > > *Blog:- **http://learnhardwithkunalgupta.blogspot.in
> > > <http://learnhardwithkunalgupta.blogspot.in> *
> > >
> > >
> > >
> > >
> > >
> > > On Tue, Jan 3, 2017 at 12:25 PM, Laxmi Narayan NIT DGP <
> > > nit.dgp673@gmail.com
> > > > wrote:
> > >
> > > > Hi,
> > > >
> > > > Does adding more partitions to kafka will help in scaling ?
> > > >
> > > >
> > > > *Regards,*
> > > > *Laxmi Narayan Patel*
> > > > *MCA NIT Durgapur (2011-2014)*
> > > > *Mob:- 9741292048,8345847473 *
> > > >
> > >
> >
>
Re: adding more partition
Posted by Kunal Gupta <ku...@cube26.com>.
For your Question 1 No, Message will be read only once by anyone of the
consumer in a consumer group
For your Question 2 Yes, Because each consumer group will read a message
exactly once. Kafka is meant for Guaranteed Message Processing
There are three ways of Guaranteed Message Processing 1) Exactly once 2) At
least Once 3) At most once
you can get information well in google.
If you set auto commit true then message will not be deleted, it all deals
with consumer offset for reading a topic. At which offset consumer group
has read the topic.
*Thanks, Kunal*
*+91-9958189589*
*Data Analyst*
*First Paper Publication : **http://dl.acm.org/citation.cfm?id=2790798
<http://dl.acm.org/citation.cfm?id=2790798> *
*Second Paper Publication : http://dl.acm.org/citation.cfm?id=2947432
<http://dl.acm.org/citation.cfm?id=2947432>*
*Blog:- **http://learnhardwithkunalgupta.blogspot.in
<http://learnhardwithkunalgupta.blogspot.in> *
On Tue, Jan 3, 2017 at 12:41 PM, Laxmi Narayan NIT DGP <nit.dgp673@gmail.com
> wrote:
> Hi ,
>
> If I have kafka cluster along with replication .
>
> Say cluster of 3 kafka-server and replication factor of message with 2.
>
> Que 1:
>
> I have 1-consumer group but 2 consumer in one group reading from 1-Topic,
>
> If I enable auto commit true in consumer , will same message will be read
> twice ?
>
> Que2:
>
> If I have 2 consumer group reading from 1-Topic
>
> If I enable auto commit true in consumer , will same message will be read
> twice ?
>
>
> and If I set auto-commit true will message will be deleted ?
>
>
>
>
>
>
> *Regards,*
> *Laxmi Narayan Patel*
> *MCA NIT Durgapur (2011-2014)*
> *Mob:- 9741292048,8345847473 *
>
> On Tue, Jan 3, 2017 at 12:34 PM, Kunal Gupta <ku...@cube26.com>
> wrote:
>
> > I don't think so because more partition can led to unavailability but yes
> > can led to higher throughput. But it cause more problems like increasing
> > end to end latency, requires more open file handler and require more
> memory
> > at client side.
> >
> > *Thanks, Kunal*
> > *+91-9958189589*
> > *Data Analyst*
> > *First Paper Publication : **http://dl.acm.org/citation.cfm?id=2790798
> > <http://dl.acm.org/citation.cfm?id=2790798> *
> > *Second Paper Publication : http://dl.acm.org/citation.cfm?id=2947432
> > <http://dl.acm.org/citation.cfm?id=2947432>*
> > *Blog:- **http://learnhardwithkunalgupta.blogspot.in
> > <http://learnhardwithkunalgupta.blogspot.in> *
> >
> >
> >
> >
> >
> > On Tue, Jan 3, 2017 at 12:25 PM, Laxmi Narayan NIT DGP <
> > nit.dgp673@gmail.com
> > > wrote:
> >
> > > Hi,
> > >
> > > Does adding more partitions to kafka will help in scaling ?
> > >
> > >
> > > *Regards,*
> > > *Laxmi Narayan Patel*
> > > *MCA NIT Durgapur (2011-2014)*
> > > *Mob:- 9741292048,8345847473 *
> > >
> >
>
Re: adding more partition
Posted by Laxmi Narayan NIT DGP <ni...@gmail.com>.
Hi ,
If I have kafka cluster along with replication .
Say cluster of 3 kafka-server and replication factor of message with 2.
Que 1:
I have 1-consumer group but 2 consumer in one group reading from 1-Topic,
If I enable auto commit true in consumer , will same message will be read
twice ?
Que2:
If I have 2 consumer group reading from 1-Topic
If I enable auto commit true in consumer , will same message will be read
twice ?
and If I set auto-commit true will message will be deleted ?
*Regards,*
*Laxmi Narayan Patel*
*MCA NIT Durgapur (2011-2014)*
*Mob:- 9741292048,8345847473 *
On Tue, Jan 3, 2017 at 12:34 PM, Kunal Gupta <ku...@cube26.com> wrote:
> I don't think so because more partition can led to unavailability but yes
> can led to higher throughput. But it cause more problems like increasing
> end to end latency, requires more open file handler and require more memory
> at client side.
>
> *Thanks, Kunal*
> *+91-9958189589*
> *Data Analyst*
> *First Paper Publication : **http://dl.acm.org/citation.cfm?id=2790798
> <http://dl.acm.org/citation.cfm?id=2790798> *
> *Second Paper Publication : http://dl.acm.org/citation.cfm?id=2947432
> <http://dl.acm.org/citation.cfm?id=2947432>*
> *Blog:- **http://learnhardwithkunalgupta.blogspot.in
> <http://learnhardwithkunalgupta.blogspot.in> *
>
>
>
>
>
> On Tue, Jan 3, 2017 at 12:25 PM, Laxmi Narayan NIT DGP <
> nit.dgp673@gmail.com
> > wrote:
>
> > Hi,
> >
> > Does adding more partitions to kafka will help in scaling ?
> >
> >
> > *Regards,*
> > *Laxmi Narayan Patel*
> > *MCA NIT Durgapur (2011-2014)*
> > *Mob:- 9741292048,8345847473 *
> >
>
Re: adding more partition
Posted by Kunal Gupta <ku...@cube26.com>.
I don't think so because more partition can led to unavailability but yes
can led to higher throughput. But it cause more problems like increasing
end to end latency, requires more open file handler and require more memory
at client side.
*Thanks, Kunal*
*+91-9958189589*
*Data Analyst*
*First Paper Publication : **http://dl.acm.org/citation.cfm?id=2790798
<http://dl.acm.org/citation.cfm?id=2790798> *
*Second Paper Publication : http://dl.acm.org/citation.cfm?id=2947432
<http://dl.acm.org/citation.cfm?id=2947432>*
*Blog:- **http://learnhardwithkunalgupta.blogspot.in
<http://learnhardwithkunalgupta.blogspot.in> *
On Tue, Jan 3, 2017 at 12:25 PM, Laxmi Narayan NIT DGP <nit.dgp673@gmail.com
> wrote:
> Hi,
>
> Does adding more partitions to kafka will help in scaling ?
>
>
> *Regards,*
> *Laxmi Narayan Patel*
> *MCA NIT Durgapur (2011-2014)*
> *Mob:- 9741292048,8345847473 *
>