You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by Ankit Jain <an...@gmail.com> on 2013/04/01 21:27:38 UTC
Kafka log retentions
Hi All,
Once the message is consumed by consumer, we want it to delete from message
broker as well.
I was exploring the kafka configuration, but not sure which configuration
would full my need.
Guys, I need your help.
Thanks ..
--
Thanks,
Ankit Jain
Re: Kafka log retentions
Posted by Tom Brown <to...@gmail.com>.
I've often wondered about what it would take to be able to overwrite a
specific offset in a partition (it could be very useful for transaction
rollbacks, message deletions, etc). Unfortunately, I don't think that
feature currently exists.
--Tom
On Mon, Apr 1, 2013 at 11:14 PM, Pankaj Misra <pa...@impetus.co.in>wrote:
> Hi,
>
> Is it possible for consumer to trigger a message dequeue(message deletion)
> from the broker after consuming the message?
>
> Thanks & Regards
> Pankaj Misra
>
> -----Original Message-----
> From: Jason Rosenberg [mailto:jbr@squareup.com]
> Sent: Tuesday, April 02, 2013 1:30 AM
> To: users@kafka.apache.org
> Subject: Re: Kafka log retentions
>
> Essentially,
>
> There's a configuration property: log.retention.hours
>
> This determines the minimum time a message will remain available on the
> broker. The default is 7 days.
>
> The kafka broker doesn't keep track of whether the message has been
> consumed or not (or how many times it has been consumed).
>
> It's up to individual consumers to keep track of which message offset
> they've consumed (which typically gets saved in zookeeper).
>
> Jason
>
>
> On Mon, Apr 1, 2013 at 12:50 PM, Philip O'Toole <ph...@loggly.com> wrote:
>
> > On Mon, Apr 1, 2013 at 12:27 PM, Ankit Jain <an...@gmail.com>
> > wrote:
> >
> > > Hi All,
> > >
> > > Once the message is consumed by consumer, we want it to delete from
> > message
> > > broker as well.
> > >
> >
> > Kafka doesn't work this way. Read the design doc -- it's well written,
> > and should be read by anyone working with Kafka.
> >
> > http://kafka.apache.org/design.html
> >
> >
> >
> > >
> > > I was exploring the kafka configuration, but not sure which
> > > configuration would full my need.
> > >
> > > Guys, I need your help.
> > >
> > > Thanks ..
> > >
> > >
> > > --
> > > Thanks,
> > > Ankit Jain
> > >
> >
>
> ________________________________
>
>
>
>
>
>
> NOTE: This message may contain information that is confidential,
> proprietary, privileged or otherwise protected by law. The message is
> intended solely for the named addressee. If received in error, please
> destroy and notify the sender. Any use of this email is prohibited when
> received in error. Impetus does not represent, warrant and/or guarantee,
> that the integrity of this communication has been maintained nor that the
> communication is free of errors, virus, interception or interference.
>
RE: Kafka log retentions
Posted by Pankaj Misra <pa...@impetus.co.in>.
Hi,
Is it possible for consumer to trigger a message dequeue(message deletion) from the broker after consuming the message?
Thanks & Regards
Pankaj Misra
-----Original Message-----
From: Jason Rosenberg [mailto:jbr@squareup.com]
Sent: Tuesday, April 02, 2013 1:30 AM
To: users@kafka.apache.org
Subject: Re: Kafka log retentions
Essentially,
There's a configuration property: log.retention.hours
This determines the minimum time a message will remain available on the broker. The default is 7 days.
The kafka broker doesn't keep track of whether the message has been consumed or not (or how many times it has been consumed).
It's up to individual consumers to keep track of which message offset they've consumed (which typically gets saved in zookeeper).
Jason
On Mon, Apr 1, 2013 at 12:50 PM, Philip O'Toole <ph...@loggly.com> wrote:
> On Mon, Apr 1, 2013 at 12:27 PM, Ankit Jain <an...@gmail.com>
> wrote:
>
> > Hi All,
> >
> > Once the message is consumed by consumer, we want it to delete from
> message
> > broker as well.
> >
>
> Kafka doesn't work this way. Read the design doc -- it's well written,
> and should be read by anyone working with Kafka.
>
> http://kafka.apache.org/design.html
>
>
>
> >
> > I was exploring the kafka configuration, but not sure which
> > configuration would full my need.
> >
> > Guys, I need your help.
> >
> > Thanks ..
> >
> >
> > --
> > Thanks,
> > Ankit Jain
> >
>
________________________________
NOTE: This message may contain information that is confidential, proprietary, privileged or otherwise protected by law. The message is intended solely for the named addressee. If received in error, please destroy and notify the sender. Any use of this email is prohibited when received in error. Impetus does not represent, warrant and/or guarantee, that the integrity of this communication has been maintained nor that the communication is free of errors, virus, interception or interference.
Re: Kafka log retentions
Posted by Jun Rao <ju...@gmail.com>.
If you use the high-level consumer, when the consumer gets an
OffsetOutofRangeException, it will automatically reset to either the
smallest or the largest available offset, depend on the config
autooffset.reset.
Thanks,
Jun
On Mon, Apr 1, 2013 at 8:44 PM, Yonghui Zhao <zh...@gmail.com> wrote:
> A little question， when old message log is deleted， the start point offset
> is changed. If a simple consumer seek a offset less than start point
> offset, what will happen? Read all message from start?
>
>
> 2013/4/2 Jason Rosenberg <jb...@squareup.com>
>
> > Essentially,
> >
> > There's a configuration property: log.retention.hours
> >
> > This determines the minimum time a message will remain available on the
> > broker. The default is 7 days.
> >
> > The kafka broker doesn't keep track of whether the message has been
> > consumed or not (or how many times it has been consumed).
> >
> > It's up to individual consumers to keep track of which message offset
> > they've consumed (which typically gets saved in zookeeper).
> >
> > Jason
> >
> >
> > On Mon, Apr 1, 2013 at 12:50 PM, Philip O'Toole <ph...@loggly.com>
> wrote:
> >
> > > On Mon, Apr 1, 2013 at 12:27 PM, Ankit Jain <an...@gmail.com>
> > > wrote:
> > >
> > > > Hi All,
> > > >
> > > > Once the message is consumed by consumer, we want it to delete from
> > > message
> > > > broker as well.
> > > >
> > >
> > > Kafka doesn't work this way. Read the design doc -- it's well written,
> > and
> > > should be read by anyone working with Kafka.
> > >
> > > http://kafka.apache.org/design.html
> > >
> > >
> > >
> > > >
> > > > I was exploring the kafka configuration, but not sure which
> > configuration
> > > > would full my need.
> > > >
> > > > Guys, I need your help.
> > > >
> > > > Thanks ..
> > > >
> > > >
> > > > --
> > > > Thanks,
> > > > Ankit Jain
> > > >
> > >
> >
>
Re: Kafka log retentions
Posted by Yonghui Zhao <zh...@gmail.com>.
Got it，thanks
2013/4/2 Philip O'Toole <ph...@loggly.com>
> On Mon, Apr 1, 2013 at 8:44 PM, Yonghui Zhao <zh...@gmail.com>
> wrote:
>
> > A little question， when old message log is deleted， the start point
> offset
> > is changed. If a simple consumer seek a offset less than start point
> > offset, what will happen? Read all message from start?
> >
>
> Standard Kafka Consumer code will raise an OffsetOutOfRange exception. How
> that is handled depends on the the type of Consumer you use -- Simple or
> High-Level. I believe the latter moves you to the next available offset.
>
> Philip
>
>
> >
> > 2013/4/2 Jason Rosenberg <jb...@squareup.com>
> >
> > > Essentially,
> > >
> > > There's a configuration property: log.retention.hours
> > >
> > > This determines the minimum time a message will remain available on the
> > > broker. The default is 7 days.
> > >
> > > The kafka broker doesn't keep track of whether the message has been
> > > consumed or not (or how many times it has been consumed).
> > >
> > > It's up to individual consumers to keep track of which message offset
> > > they've consumed (which typically gets saved in zookeeper).
> > >
> > > Jason
> > >
> > >
> > > On Mon, Apr 1, 2013 at 12:50 PM, Philip O'Toole <ph...@loggly.com>
> > wrote:
> > >
> > > > On Mon, Apr 1, 2013 at 12:27 PM, Ankit Jain <ankitjaincs06@gmail.com
> >
> > > > wrote:
> > > >
> > > > > Hi All,
> > > > >
> > > > > Once the message is consumed by consumer, we want it to delete from
> > > > message
> > > > > broker as well.
> > > > >
> > > >
> > > > Kafka doesn't work this way. Read the design doc -- it's well
> written,
> > > and
> > > > should be read by anyone working with Kafka.
> > > >
> > > > http://kafka.apache.org/design.html
> > > >
> > > >
> > > >
> > > > >
> > > > > I was exploring the kafka configuration, but not sure which
> > > configuration
> > > > > would full my need.
> > > > >
> > > > > Guys, I need your help.
> > > > >
> > > > > Thanks ..
> > > > >
> > > > >
> > > > > --
> > > > > Thanks,
> > > > > Ankit Jain
> > > > >
> > > >
> > >
> >
>
Re: Kafka log retentions
Posted by Philip O'Toole <ph...@loggly.com>.
On Mon, Apr 1, 2013 at 8:44 PM, Yonghui Zhao <zh...@gmail.com> wrote:
> A little question， when old message log is deleted， the start point offset
> is changed. If a simple consumer seek a offset less than start point
> offset, what will happen? Read all message from start?
>
Standard Kafka Consumer code will raise an OffsetOutOfRange exception. How
that is handled depends on the the type of Consumer you use -- Simple or
High-Level. I believe the latter moves you to the next available offset.
Philip
>
> 2013/4/2 Jason Rosenberg <jb...@squareup.com>
>
> > Essentially,
> >
> > There's a configuration property: log.retention.hours
> >
> > This determines the minimum time a message will remain available on the
> > broker. The default is 7 days.
> >
> > The kafka broker doesn't keep track of whether the message has been
> > consumed or not (or how many times it has been consumed).
> >
> > It's up to individual consumers to keep track of which message offset
> > they've consumed (which typically gets saved in zookeeper).
> >
> > Jason
> >
> >
> > On Mon, Apr 1, 2013 at 12:50 PM, Philip O'Toole <ph...@loggly.com>
> wrote:
> >
> > > On Mon, Apr 1, 2013 at 12:27 PM, Ankit Jain <an...@gmail.com>
> > > wrote:
> > >
> > > > Hi All,
> > > >
> > > > Once the message is consumed by consumer, we want it to delete from
> > > message
> > > > broker as well.
> > > >
> > >
> > > Kafka doesn't work this way. Read the design doc -- it's well written,
> > and
> > > should be read by anyone working with Kafka.
> > >
> > > http://kafka.apache.org/design.html
> > >
> > >
> > >
> > > >
> > > > I was exploring the kafka configuration, but not sure which
> > configuration
> > > > would full my need.
> > > >
> > > > Guys, I need your help.
> > > >
> > > > Thanks ..
> > > >
> > > >
> > > > --
> > > > Thanks,
> > > > Ankit Jain
> > > >
> > >
> >
>
Re: Kafka log retentions
Posted by Yonghui Zhao <zh...@gmail.com>.
A little question， when old message log is deleted， the start point offset
is changed. If a simple consumer seek a offset less than start point
offset, what will happen? Read all message from start?
2013/4/2 Jason Rosenberg <jb...@squareup.com>
> Essentially,
>
> There's a configuration property: log.retention.hours
>
> This determines the minimum time a message will remain available on the
> broker. The default is 7 days.
>
> The kafka broker doesn't keep track of whether the message has been
> consumed or not (or how many times it has been consumed).
>
> It's up to individual consumers to keep track of which message offset
> they've consumed (which typically gets saved in zookeeper).
>
> Jason
>
>
> On Mon, Apr 1, 2013 at 12:50 PM, Philip O'Toole <ph...@loggly.com> wrote:
>
> > On Mon, Apr 1, 2013 at 12:27 PM, Ankit Jain <an...@gmail.com>
> > wrote:
> >
> > > Hi All,
> > >
> > > Once the message is consumed by consumer, we want it to delete from
> > message
> > > broker as well.
> > >
> >
> > Kafka doesn't work this way. Read the design doc -- it's well written,
> and
> > should be read by anyone working with Kafka.
> >
> > http://kafka.apache.org/design.html
> >
> >
> >
> > >
> > > I was exploring the kafka configuration, but not sure which
> configuration
> > > would full my need.
> > >
> > > Guys, I need your help.
> > >
> > > Thanks ..
> > >
> > >
> > > --
> > > Thanks,
> > > Ankit Jain
> > >
> >
>
Re: Kafka log retentions
Posted by Jason Rosenberg <jb...@squareup.com>.
Essentially,
There's a configuration property: log.retention.hours
This determines the minimum time a message will remain available on the
broker. The default is 7 days.
The kafka broker doesn't keep track of whether the message has been
consumed or not (or how many times it has been consumed).
It's up to individual consumers to keep track of which message offset
they've consumed (which typically gets saved in zookeeper).
Jason
On Mon, Apr 1, 2013 at 12:50 PM, Philip O'Toole <ph...@loggly.com> wrote:
> On Mon, Apr 1, 2013 at 12:27 PM, Ankit Jain <an...@gmail.com>
> wrote:
>
> > Hi All,
> >
> > Once the message is consumed by consumer, we want it to delete from
> message
> > broker as well.
> >
>
> Kafka doesn't work this way. Read the design doc -- it's well written, and
> should be read by anyone working with Kafka.
>
> http://kafka.apache.org/design.html
>
>
>
> >
> > I was exploring the kafka configuration, but not sure which configuration
> > would full my need.
> >
> > Guys, I need your help.
> >
> > Thanks ..
> >
> >
> > --
> > Thanks,
> > Ankit Jain
> >
>
Re: Kafka log retentions
Posted by Philip O'Toole <ph...@loggly.com>.
On Mon, Apr 1, 2013 at 12:27 PM, Ankit Jain <an...@gmail.com> wrote:
> Hi All,
>
> Once the message is consumed by consumer, we want it to delete from message
> broker as well.
>
Kafka doesn't work this way. Read the design doc -- it's well written, and
should be read by anyone working with Kafka.
http://kafka.apache.org/design.html
>
> I was exploring the kafka configuration, but not sure which configuration
> would full my need.
>
> Guys, I need your help.
>
> Thanks ..
>
>
> --
> Thanks,
> Ankit Jain
>