You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by Frank Lyaruu <fl...@gmail.com> on 2017/03/21 19:13:09 UTC

Commits of slow moving topics in Kafka Streams time out

Hi Kafka people,

We have a Kafka Streams application that replicates a database, and
transforms it to a different data model. Some tables/topics move fast, with
many changes a second, some might be dormant for months. For those slow
moving topics, we have some trouble with the 'offsets.retention.minutes'.
After that timeout it's offset is effectively invalid. As long as it keeps
running, everything is ok, but when we restart the application it will fall
back to 'earliest' and redo the entire topic, which is obviously wrong.

How can we fix this or work around this? Our first idea was to enable a
very slow auto commit, so the offset.retention.minutes never times out, but
Kafka Streams seems to explicitly forbid that. The StreamConfig javadoc
says: ""Furthermore, it is not allowed to enable "enable.auto.commit" that
is disabled by Kafka Streams by default."

So how do we keep those commits alive? Why doesn't Kafka Streams allow a
slow auto commit to prevent this? Any other ideas on how to circumvent this
problem?

regards, Frank

We're running the trunk version of last week.

Re: Commits of slow moving topics in Kafka Streams time out

Posted by Frank Lyaruu <fl...@gmail.com>.
Ok, yes increasing the retention minutes to something silly is indeed what
we intend to do, and for now, that should work fine as we don't use many
short lived consumers.

For the long term, the change in semantics in the link you provide looks
good to us.

regards, Frank

On Tue, Mar 21, 2017 at 8:57 PM, Eno Thereska <en...@gmail.com>
wrote:

> Hi Frank,
>
> There is a similar discussion here https://www.mail-archive.com/
> users@kafka.apache.org/msg25089.html <https://www.mail-archive.com/
> users@kafka.apache.org/msg25089.html>, with a JIRA.
>
> Have you tried to increase the retention minutes to something large?
>
> Thanks
> Eno
> > On 21 Mar 2017, at 19:13, Frank Lyaruu <fl...@gmail.com> wrote:
> >
> > Hi Kafka people,
> >
> > We have a Kafka Streams application that replicates a database, and
> > transforms it to a different data model. Some tables/topics move fast,
> with
> > many changes a second, some might be dormant for months. For those slow
> > moving topics, we have some trouble with the 'offsets.retention.minutes'.
> > After that timeout it's offset is effectively invalid. As long as it
> keeps
> > running, everything is ok, but when we restart the application it will
> fall
> > back to 'earliest' and redo the entire topic, which is obviously wrong.
> >
> > How can we fix this or work around this? Our first idea was to enable a
> > very slow auto commit, so the offset.retention.minutes never times out,
> but
> > Kafka Streams seems to explicitly forbid that. The StreamConfig javadoc
> > says: ""Furthermore, it is not allowed to enable "enable.auto.commit"
> that
> > is disabled by Kafka Streams by default."
> >
> > So how do we keep those commits alive? Why doesn't Kafka Streams allow a
> > slow auto commit to prevent this? Any other ideas on how to circumvent
> this
> > problem?
> >
> > regards, Frank
> >
> > We're running the trunk version of last week.
>
>

Re: Commits of slow moving topics in Kafka Streams time out

Posted by Eno Thereska <en...@gmail.com>.
Hi Frank,

There is a similar discussion here https://www.mail-archive.com/users@kafka.apache.org/msg25089.html <https://www.mail-archive.com/users@kafka.apache.org/msg25089.html>, with a JIRA.

Have you tried to increase the retention minutes to something large?

Thanks
Eno
> On 21 Mar 2017, at 19:13, Frank Lyaruu <fl...@gmail.com> wrote:
> 
> Hi Kafka people,
> 
> We have a Kafka Streams application that replicates a database, and
> transforms it to a different data model. Some tables/topics move fast, with
> many changes a second, some might be dormant for months. For those slow
> moving topics, we have some trouble with the 'offsets.retention.minutes'.
> After that timeout it's offset is effectively invalid. As long as it keeps
> running, everything is ok, but when we restart the application it will fall
> back to 'earliest' and redo the entire topic, which is obviously wrong.
> 
> How can we fix this or work around this? Our first idea was to enable a
> very slow auto commit, so the offset.retention.minutes never times out, but
> Kafka Streams seems to explicitly forbid that. The StreamConfig javadoc
> says: ""Furthermore, it is not allowed to enable "enable.auto.commit" that
> is disabled by Kafka Streams by default."
> 
> So how do we keep those commits alive? Why doesn't Kafka Streams allow a
> slow auto commit to prevent this? Any other ideas on how to circumvent this
> problem?
> 
> regards, Frank
> 
> We're running the trunk version of last week.