Kafka stream moving from running to shutdown without any visible exception or error

[Samuel Azcona <sa...@itbit.com.INVALID>]

Hi guys,

I have a kotlin App that run 5 streams with different application.id, like:

my-app-stream1
my-app-stream2

some days a go I start experimenting, a very weird problem, all my streams
are in running state then in 3-5 minutes transition to shut down, I enable
the debug/trace logs, but I can't see any exception or error.

I have the setUncaughtExceptionHandler set with a log, but it's never
executed. So there are no exceptions inside the stream.

I already try to disable all my streams and just keep one not smart with a
foreach logging and the same happen, also I revert my code to a previous
state and the same happen.

I'm not able to reproduce in my local environment, this is happened to me
on production.

All the streams just change the state from running to pending shutdown

I only have some small logs like

State transition from RUNNING to PENDING_SHUTDOWN
.....
State transition from PENDING_SHUTDOWN to NOT_RUNNING
....
Waiting for the I/O thread to exit. Hard shutdown in 31536000000 ms.
...
Flushing all global globalStores registered in the state manager
...
Global thread has died. The streams application or client will now close to
ERROR
.....
Kafka consumer has been closed
....
Exiting AdminClientRunnable

any ideas? Thanks guys!!

Re: Kafka stream moving from running to shutdown without any visible exception or error

[Bruno Cadonna <ca...@apache.org>]

Hi Samuel,

from your few log messages I see that the global stream thread dies. 
That might be the cause of the shutdown. Look for what is causing your 
global stream thread/global store to die.


Best,
Bruno


On 13.09.22 22:55, Samuel Azcona wrote:
> Hi guys,
> 
> I have a kotlin App that run 5 streams with different application.id, like:
> 
> my-app-stream1
> my-app-stream2
> 
> some days a go I start experimenting, a very weird problem, all my streams
> are in running state then in 3-5 minutes transition to shut down, I enable
> the debug/trace logs, but I can't see any exception or error.
> 
> I have the setUncaughtExceptionHandler set with a log, but it's never
> executed. So there are no exceptions inside the stream.
> 
> I already try to disable all my streams and just keep one not smart with a
> foreach logging and the same happen, also I revert my code to a previous
> state and the same happen.
> 
> I'm not able to reproduce in my local environment, this is happened to me
> on production.
> 
> All the streams just change the state from running to pending shutdown
> 
> I only have some small logs like
> 
> State transition from RUNNING to PENDING_SHUTDOWN
> .....
> State transition from PENDING_SHUTDOWN to NOT_RUNNING
> ....
> Waiting for the I/O thread to exit. Hard shutdown in 31536000000 ms.
> ...
> Flushing all global globalStores registered in the state manager
> ...
> Global thread has died. The streams application or client will now close to
> ERROR
> .....
> Kafka consumer has been closed
> ....
> Exiting AdminClientRunnable
> 
> any ideas? Thanks guys!!
>