Kafka streams application (v 0.10.0.1) stuck at close

[Balaprassanna Ilangovan <ba...@gmail.com>]

Hi,

I have the following three question regarding Apache Kafka streams.

1. I am in a position to use v 0.10.0.1 of Apache Kafka though so many bugs
related to streams are fixed in the later versions. My application consists
of low level processors that run for more than an hour for larger files
(video transcoding). So, we use a session timeout and request timeout of 2
hrs. streams.close() is stuck for a long time even the processors are idle.
Is there a reason? Is there a work around for this version?

2. Also, what does processorContext.commit() do exactly? Does it save the
position of application in a topology or commit consumed message offset in
the partition? Though commits are handled automatically by streams, should
context.commit() be called at the end of each processor in a topology?

3. In a topology, if two processor completes successfully and the
application goes down during third processor execution, does it start from
first when the application comes back?
-- 
With Regards,
Bala Prassanna I,
8124831208

Re: Kafka streams application (v 0.10.0.1) stuck at close

[Damian Guy <da...@gmail.com>]

Hi,

If you can then i'd recommend upgrading to a newer version. As you said
many bugs have been fixed since 0.10.0.1

On Wed, 23 Aug 2017 at 05:08 Balaprassanna Ilangovan <
balaprassanna1993@gmail.com> wrote:

> Hi,
>
> I have the following three question regarding Apache Kafka streams.
>
> 1. I am in a position to use v 0.10.0.1 of Apache Kafka though so many bugs
> related to streams are fixed in the later versions. My application consists
> of low level processors that run for more than an hour for larger files
> (video transcoding). So, we use a session timeout and request timeout of 2
> hrs. streams.close() is stuck for a long time even the processors are idle.
> Is there a reason? Is there a work around for this version?
>
>
There were some bugs to do with streams.close() in earlier versions that
may cause deadlocks. This may be the issue:
https://issues.apache.org/jira/browse/KAFKA-4366


> 2. Also, what does processorContext.commit() do exactly? Does it save the
> position of application in a topology or commit consumed message offset in
> the partition? Though commits are handled automatically by streams, should
> context.commit() be called at the end of each processor in a topology?
>
>
context.commit is just telling the Streams Library to commit that task the
next time it goes through the poll loop. You don't need to call this unless
you specifically want to commit after you have processed 1 or more records.
Otherwise this is handled automatically by the commit.interval.ms config


> 3. In a topology, if two processor completes successfully and the
> application goes down during third processor execution, does it start from
> first when the application comes back?
>

Each task will start from the last committed position. So if that was all
in a single thread, then it will start from the beginning again.


> --
> With Regards,
> Bala Prassanna I,
> 8124831208 <(812)%20483-1208>
>