You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by Kashyap Mhaisekar <ka...@gmail.com> on 2014/05/05 07:11:12 UTC
Re: Log Retention in Kafka
Is there a way to do this at runtime using some available scripts in
kafka/bin? If so, any pointers on which script?
Regards,
Kashyap
On Tue, Apr 22, 2014 at 11:11 PM, Kashyap Mhaisekar <ka...@gmail.com>wrote:
> Thanks Joel. Am using version 2.8.0.
>
> Thanks,
> Kashyap
>
>
> On Tue, Apr 22, 2014 at 5:53 PM, Joel Koshy <jj...@gmail.com> wrote:
>
>> Which version of Kafka are you using?
>>
>> You can read up on the configuration options here:
>> http://kafka.apache.org/documentation.html#configuration
>>
>> You can specify time-based retention using log.retention.minutes which
>> will apply to all topics. You can override that on per-topic basis -
>> see further down in the above page under "topic-level configuration"
>>
>> On Tue, Apr 22, 2014 at 02:34:24PM -0500, Kashyap Mhaisekar wrote:
>> > Hi,
>> > I wanted to set the message expiry for a message on a kafka topic. Is
>> there
>> > anything like this in kafka?
>> > I came across a property - *log.retention.hours* and
>> > *topic.log.retention.hours*
>> > Had some queries around it.And it was mentioned that
>> > topic.log.retention.hours is per topic configuration.
>> > Had some queries around it -
>> > 1. Does it mean that I need to specific the
>> <topicname>.log.retention.hours
>> > in the kafka config?
>> > 2. Can this property be overriden anywhere?
>> > 3. Is it possible for the producer to set a message expiry so that the
>> > message expires after a configurable period of time?
>> >
>> > Regards,
>> > Kashyap
>>
>>
>
Re: Log Retention in Kafka
Posted by Jun Rao <ju...@gmail.com>.
Dynamically changing configs per topic is only available in 0.8.1. You can
try the latest release 0.8.1.1.
Thanks,
Jun
On Mon, May 5, 2014 at 12:28 PM, Kashyap Mhaisekar <ka...@gmail.com>wrote:
> Thanks Jun. I am using kafka_2.8.0-0.8.0-beta1.jar. I dont see this script
> kafka-topic.sh in the bin folder. Is there way to do this in the version
> mentioned?
>
> --Kashyap
>
>
> On Mon, May 5, 2014 at 11:18 AM, Jun Rao <ju...@gmail.com> wrote:
>
> > See http://kafka.apache.org/documentation.html#basic_ops_modify_topic
> >
> > Thanks,
> >
> > Jun
> >
> >
> > On Sun, May 4, 2014 at 10:11 PM, Kashyap Mhaisekar <kashyap.m@gmail.com
> > >wrote:
> >
> > > Is there a way to do this at runtime using some available scripts in
> > > kafka/bin? If so, any pointers on which script?
> > >
> > > Regards,
> > > Kashyap
> > >
> > >
> > > On Tue, Apr 22, 2014 at 11:11 PM, Kashyap Mhaisekar <
> kashyap.m@gmail.com
> > > >wrote:
> > >
> > > > Thanks Joel. Am using version 2.8.0.
> > > >
> > > > Thanks,
> > > > Kashyap
> > > >
> > > >
> > > > On Tue, Apr 22, 2014 at 5:53 PM, Joel Koshy <jj...@gmail.com>
> > wrote:
> > > >
> > > >> Which version of Kafka are you using?
> > > >>
> > > >> You can read up on the configuration options here:
> > > >> http://kafka.apache.org/documentation.html#configuration
> > > >>
> > > >> You can specify time-based retention using log.retention.minutes
> which
> > > >> will apply to all topics. You can override that on per-topic basis -
> > > >> see further down in the above page under "topic-level configuration"
> > > >>
> > > >> On Tue, Apr 22, 2014 at 02:34:24PM -0500, Kashyap Mhaisekar wrote:
> > > >> > Hi,
> > > >> > I wanted to set the message expiry for a message on a kafka topic.
> > Is
> > > >> there
> > > >> > anything like this in kafka?
> > > >> > I came across a property - *log.retention.hours* and
> > > >> > *topic.log.retention.hours*
> > > >> > Had some queries around it.And it was mentioned that
> > > >> > topic.log.retention.hours is per topic configuration.
> > > >> > Had some queries around it -
> > > >> > 1. Does it mean that I need to specific the
> > > >> <topicname>.log.retention.hours
> > > >> > in the kafka config?
> > > >> > 2. Can this property be overriden anywhere?
> > > >> > 3. Is it possible for the producer to set a message expiry so that
> > the
> > > >> > message expires after a configurable period of time?
> > > >> >
> > > >> > Regards,
> > > >> > Kashyap
> > > >>
> > > >>
> > > >
> > >
> >
>
Re: Log Retention in Kafka
Posted by Kashyap Mhaisekar <ka...@gmail.com>.
Thanks Jun. I am using kafka_2.8.0-0.8.0-beta1.jar. I dont see this script
kafka-topic.sh in the bin folder. Is there way to do this in the version
mentioned?
--Kashyap
On Mon, May 5, 2014 at 11:18 AM, Jun Rao <ju...@gmail.com> wrote:
> See http://kafka.apache.org/documentation.html#basic_ops_modify_topic
>
> Thanks,
>
> Jun
>
>
> On Sun, May 4, 2014 at 10:11 PM, Kashyap Mhaisekar <kashyap.m@gmail.com
> >wrote:
>
> > Is there a way to do this at runtime using some available scripts in
> > kafka/bin? If so, any pointers on which script?
> >
> > Regards,
> > Kashyap
> >
> >
> > On Tue, Apr 22, 2014 at 11:11 PM, Kashyap Mhaisekar <kashyap.m@gmail.com
> > >wrote:
> >
> > > Thanks Joel. Am using version 2.8.0.
> > >
> > > Thanks,
> > > Kashyap
> > >
> > >
> > > On Tue, Apr 22, 2014 at 5:53 PM, Joel Koshy <jj...@gmail.com>
> wrote:
> > >
> > >> Which version of Kafka are you using?
> > >>
> > >> You can read up on the configuration options here:
> > >> http://kafka.apache.org/documentation.html#configuration
> > >>
> > >> You can specify time-based retention using log.retention.minutes which
> > >> will apply to all topics. You can override that on per-topic basis -
> > >> see further down in the above page under "topic-level configuration"
> > >>
> > >> On Tue, Apr 22, 2014 at 02:34:24PM -0500, Kashyap Mhaisekar wrote:
> > >> > Hi,
> > >> > I wanted to set the message expiry for a message on a kafka topic.
> Is
> > >> there
> > >> > anything like this in kafka?
> > >> > I came across a property - *log.retention.hours* and
> > >> > *topic.log.retention.hours*
> > >> > Had some queries around it.And it was mentioned that
> > >> > topic.log.retention.hours is per topic configuration.
> > >> > Had some queries around it -
> > >> > 1. Does it mean that I need to specific the
> > >> <topicname>.log.retention.hours
> > >> > in the kafka config?
> > >> > 2. Can this property be overriden anywhere?
> > >> > 3. Is it possible for the producer to set a message expiry so that
> the
> > >> > message expires after a configurable period of time?
> > >> >
> > >> > Regards,
> > >> > Kashyap
> > >>
> > >>
> > >
> >
>
Re: Log Retention in Kafka
Posted by Jun Rao <ju...@gmail.com>.
See http://kafka.apache.org/documentation.html#basic_ops_modify_topic
Thanks,
Jun
On Sun, May 4, 2014 at 10:11 PM, Kashyap Mhaisekar <ka...@gmail.com>wrote:
> Is there a way to do this at runtime using some available scripts in
> kafka/bin? If so, any pointers on which script?
>
> Regards,
> Kashyap
>
>
> On Tue, Apr 22, 2014 at 11:11 PM, Kashyap Mhaisekar <kashyap.m@gmail.com
> >wrote:
>
> > Thanks Joel. Am using version 2.8.0.
> >
> > Thanks,
> > Kashyap
> >
> >
> > On Tue, Apr 22, 2014 at 5:53 PM, Joel Koshy <jj...@gmail.com> wrote:
> >
> >> Which version of Kafka are you using?
> >>
> >> You can read up on the configuration options here:
> >> http://kafka.apache.org/documentation.html#configuration
> >>
> >> You can specify time-based retention using log.retention.minutes which
> >> will apply to all topics. You can override that on per-topic basis -
> >> see further down in the above page under "topic-level configuration"
> >>
> >> On Tue, Apr 22, 2014 at 02:34:24PM -0500, Kashyap Mhaisekar wrote:
> >> > Hi,
> >> > I wanted to set the message expiry for a message on a kafka topic. Is
> >> there
> >> > anything like this in kafka?
> >> > I came across a property - *log.retention.hours* and
> >> > *topic.log.retention.hours*
> >> > Had some queries around it.And it was mentioned that
> >> > topic.log.retention.hours is per topic configuration.
> >> > Had some queries around it -
> >> > 1. Does it mean that I need to specific the
> >> <topicname>.log.retention.hours
> >> > in the kafka config?
> >> > 2. Can this property be overriden anywhere?
> >> > 3. Is it possible for the producer to set a message expiry so that the
> >> > message expires after a configurable period of time?
> >> >
> >> > Regards,
> >> > Kashyap
> >>
> >>
> >
>