[jira] [Commented] (KAFKA-7214) Mystic FATAL error

["Seweryn Habdank-Wojewodzki (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/KAFKA-7214?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16605851#comment-16605851 ] 

Seweryn Habdank-Wojewodzki commented on KAFKA-7214:
---------------------------------------------------

Problem is that  KSTREAM-SOURCE-XXXXXXXXX is mostly  KSTREAM-SOURCE-000000000 independently of which process and how much processes are running (or trying to run).

How I reproduce error at my side. Let's assume I have low message flow < 100 Msg/sec Msg size ~ 1kB.

I am starting app using streaming API. This app reads from 30 topics and send messages to 1 topic.
Let's give this app 300MB JVM Heap. It is starting. Cool.

At second server I am starting. second instance. The same. It is starting.

The other case. Let's assume I have low message flow > 5000 Msg/sec Msg size ~ 1kB.

I am starting app using streaming API. This app reads from 30 topics and send messages to 1 topic.
Let's give this app 300 MB JVM Heap. It is not starting, even in memory spec stays that it is enough to calculate 30 sec of messages.
5000 Msg/s ~ 150 000 Mgs/30 sec ~ 150 MB.
I am giving to app 2GB Heap. Is starting. Everything between 300 MB and 2 GB leads at some point to yet another mystic crasches.

At second server I am starting. second instance. If I am starting it with 300 MB - I got immediately this error. Application tries to starrt, but then I got this error and all affected topics are goig to be dead. If I am giving 1GB, it is better application works some hours, but any minimal peak aroud 5000 Msg/s to e.g. 7000 Msg/s, causes the same. Finally - now - I am starting processes with 5GB. they could work continuously like 2-4 days.

I am sorry I have no better description.
Once I tried to start TRACE level logs in Kafka, but this is impossible with message flow at 5000 Msg/s.



> Mystic FATAL error
> ------------------
>
>                 Key: KAFKA-7214
>                 URL: https://issues.apache.org/jira/browse/KAFKA-7214
>             Project: Kafka
>          Issue Type: Bug
>          Components: streams
>    Affects Versions: 0.11.0.3, 1.1.1
>            Reporter: Seweryn Habdank-Wojewodzki
>            Priority: Critical
>
> Dears,
> Very often at startup of the streaming application I got exception:
> {code}
> Exception caught in process. taskId=0_1, processor=KSTREAM-SOURCE-0000000000, topic=my_instance_medium_topic, partition=1, offset=198900203; [org.apache.kafka.streams.processor.internals.StreamTask.process(StreamTask.java:212), org.apache.kafka.streams.processor.internals.AssignedTasks$2.apply(AssignedTasks.java:347), org.apache.kafka.streams.processor.internals.AssignedTasks.applyToRunningTasks(AssignedTasks.java:420), org.apache.kafka.streams.processor.internals.AssignedTasks.process(AssignedTasks.java:339), org.apache.kafka.streams.processor.internals.StreamThread.processAndPunctuate(StreamThread.java:648), org.apache.kafka.streams.processor.internals.StreamThread.runOnce(StreamThread.java:513), org.apache.kafka.streams.processor.internals.StreamThread.runLoop(StreamThread.java:482), org.apache.kafka.streams.processor.internals.StreamThread.run(StreamThread.java:459)] in thread my_application-my_instance-my_instance_medium-72ee1819-edeb-4d85-9d65-f67f7c321618-StreamThread-62
> {code}
> and then (without shutdown request from my side):
> {code}
> 2018-07-30 07:45:02 [ar313] [INFO ] StreamThread:912 - stream-thread [my_application-my_instance-my_instance-72ee1819-edeb-4d85-9d65-f67f7c321618-StreamThread-62] State transition from PENDING_SHUTDOWN to DEAD.
> {code}
> What is this?
> How to correctly handle it?
> Thanks in advance for help.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)