You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@kafka.apache.org by Aman Rastogi <am...@gmail.com> on 2018/01/23 17:30:44 UTC
Regarding : Store stream for infinite time
Hi All,
We have a use case to store stream for infinite time (given we have enough
storage).
We are planning to solve this by Log Compaction. If each message key is
unique and Log compaction is enabled, it will store whole stream for
infinite time. Just wanted to check if my assumption is correct and this is
an appropriate way to solve this.
Thanks in advance.
Regards,
Aman
Re: Regarding : Store stream for infinite time
Posted by Aman Rastogi <am...@gmail.com>.
Thanks Matthias. This is really helpful.
Regards,
Aman
On Wed, Jan 24, 2018 at 2:47 AM, Matthias J. Sax <ma...@confluent.io>
wrote:
> It's simpler to set retention time to Long.MAX_VALUE instead of log
> compaction and generating unique keys. This effectively disable
> retention because the value is huge.
>
> The advantage is, that you don't need to worry about generating unique
> keys.
>
> -Matthias
>
> On 1/23/18 10:39 AM, Aman Rastogi wrote:
> > Thanks Svante.
> >
> > Regards,
> > Aman
> >
> > On Tue, Jan 23, 2018 at 11:38 PM, Svante Karlsson <
> svante.karlsson@csi.se>
> > wrote:
> >
> >> Yes, it will store the last value for each key
> >>
> >> 2018-01-23 18:30 GMT+01:00 Aman Rastogi <am...@gmail.com>:
> >>
> >>> Hi All,
> >>>
> >>> We have a use case to store stream for infinite time (given we have
> >> enough
> >>> storage).
> >>>
> >>> We are planning to solve this by Log Compaction. If each message key is
> >>> unique and Log compaction is enabled, it will store whole stream for
> >>> infinite time. Just wanted to check if my assumption is correct and
> this
> >> is
> >>> an appropriate way to solve this.
> >>>
> >>> Thanks in advance.
> >>>
> >>> Regards,
> >>> Aman
> >>>
> >>
> >
>
>
Re: Regarding : Store stream for infinite time
Posted by "Matthias J. Sax" <ma...@confluent.io>.
It's simpler to set retention time to Long.MAX_VALUE instead of log
compaction and generating unique keys. This effectively disable
retention because the value is huge.
The advantage is, that you don't need to worry about generating unique keys.
-Matthias
On 1/23/18 10:39 AM, Aman Rastogi wrote:
> Thanks Svante.
>
> Regards,
> Aman
>
> On Tue, Jan 23, 2018 at 11:38 PM, Svante Karlsson <sv...@csi.se>
> wrote:
>
>> Yes, it will store the last value for each key
>>
>> 2018-01-23 18:30 GMT+01:00 Aman Rastogi <am...@gmail.com>:
>>
>>> Hi All,
>>>
>>> We have a use case to store stream for infinite time (given we have
>> enough
>>> storage).
>>>
>>> We are planning to solve this by Log Compaction. If each message key is
>>> unique and Log compaction is enabled, it will store whole stream for
>>> infinite time. Just wanted to check if my assumption is correct and this
>> is
>>> an appropriate way to solve this.
>>>
>>> Thanks in advance.
>>>
>>> Regards,
>>> Aman
>>>
>>
>
Re: Regarding : Store stream for infinite time
Posted by Aman Rastogi <am...@gmail.com>.
Thanks Svante.
Regards,
Aman
On Tue, Jan 23, 2018 at 11:38 PM, Svante Karlsson <sv...@csi.se>
wrote:
> Yes, it will store the last value for each key
>
> 2018-01-23 18:30 GMT+01:00 Aman Rastogi <am...@gmail.com>:
>
> > Hi All,
> >
> > We have a use case to store stream for infinite time (given we have
> enough
> > storage).
> >
> > We are planning to solve this by Log Compaction. If each message key is
> > unique and Log compaction is enabled, it will store whole stream for
> > infinite time. Just wanted to check if my assumption is correct and this
> is
> > an appropriate way to solve this.
> >
> > Thanks in advance.
> >
> > Regards,
> > Aman
> >
>
Re: Regarding : Store stream for infinite time
Posted by Aman Rastogi <am...@gmail.com>.
Thanks Svante.
Regards,
Aman
On Tue, Jan 23, 2018 at 11:38 PM, Svante Karlsson <sv...@csi.se>
wrote:
> Yes, it will store the last value for each key
>
> 2018-01-23 18:30 GMT+01:00 Aman Rastogi <am...@gmail.com>:
>
> > Hi All,
> >
> > We have a use case to store stream for infinite time (given we have
> enough
> > storage).
> >
> > We are planning to solve this by Log Compaction. If each message key is
> > unique and Log compaction is enabled, it will store whole stream for
> > infinite time. Just wanted to check if my assumption is correct and this
> is
> > an appropriate way to solve this.
> >
> > Thanks in advance.
> >
> > Regards,
> > Aman
> >
>
Re: Regarding : Store stream for infinite time
Posted by Svante Karlsson <sv...@csi.se>.
Yes, it will store the last value for each key
2018-01-23 18:30 GMT+01:00 Aman Rastogi <am...@gmail.com>:
> Hi All,
>
> We have a use case to store stream for infinite time (given we have enough
> storage).
>
> We are planning to solve this by Log Compaction. If each message key is
> unique and Log compaction is enabled, it will store whole stream for
> infinite time. Just wanted to check if my assumption is correct and this is
> an appropriate way to solve this.
>
> Thanks in advance.
>
> Regards,
> Aman
>
Re: Regarding : Store stream for infinite time
Posted by Svante Karlsson <sv...@csi.se>.
Yes, it will store the last value for each key
2018-01-23 18:30 GMT+01:00 Aman Rastogi <am...@gmail.com>:
> Hi All,
>
> We have a use case to store stream for infinite time (given we have enough
> storage).
>
> We are planning to solve this by Log Compaction. If each message key is
> unique and Log compaction is enabled, it will store whole stream for
> infinite time. Just wanted to check if my assumption is correct and this is
> an appropriate way to solve this.
>
> Thanks in advance.
>
> Regards,
> Aman
>