Intermittent KryoException

[Scott Sue <sc...@celer-tech.com>]

Hi,

When running our job, we’re seeing sporadic instances of when we have KryoExceptions.  I’m new to this area of Flink so I’m not exactly too sure what I could look out for.  From my understanding, Kryo is the default serializer for generic types, and whilst there is a potential performance penalty with using Kryo, it should be able to serialize / deserialize all objects without fail?

Another point is that our object is mutable through as it runs through the different operators, could periodic checkpointing be a cause of the below issues?

We are currently running Flink 1.7.1


11:02:43,075 INFO  org.apache.flink.runtime.taskmanager.Task                     - Window(ProcessingTimeSessionWindows(10000), ProcessingTimeTrigger, CoGroupWindowFunction) -> Flat Map -> Sink: Unnamed (1/1) (a83e88eaf06490de
c8326e4d9bd0ed26) switched from RUNNING to FAILED.
TimerException{com.esotericsoftware.kryo.KryoException: java.lang.ArrayIndexOutOfBoundsException: 1024
Serialization trace:
payload (com.celertech.analytics.bo.AnalyticsDataJsonMessage)}
        at org.apache.flink.streaming.runtime.tasks.SystemProcessingTimeService$TriggerTask.run(SystemProcessingTimeService.java:288)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
Caused by: com.esotericsoftware.kryo.KryoException: java.lang.ArrayIndexOutOfBoundsException: 1024
Serialization trace:
payload (com.celertech.analytics.bo.AnalyticsDataJsonMessage)
        at com.esotericsoftware.kryo.serializers.ObjectField.write(ObjectField.java:82)
        at com.esotericsoftware.kryo.serializers.FieldSerializer.write(FieldSerializer.java:495)
        at com.esotericsoftware.kryo.Kryo.writeObject(Kryo.java:505)
        at org.apache.flink.api.java.typeutils.runtime.kryo.KryoSerializer.copy(KryoSerializer.java:224)
        at org.apache.flink.streaming.api.datastream.CoGroupedStreams$UnionSerializer.copy(CoGroupedStreams.java:538)
        at org.apache.flink.streaming.api.datastream.CoGroupedStreams$UnionSerializer.copy(CoGroupedStreams.java:507)
        at org.apache.flink.api.common.typeutils.base.ListSerializer.copy(ListSerializer.java:99)
        at org.apache.flink.api.common.typeutils.base.ListSerializer.copy(ListSerializer.java:42)
        at org.apache.flink.runtime.state.heap.CopyOnWriteStateTable.get(CopyOnWriteStateTable.java:287)
        at org.apache.flink.runtime.state.heap.CopyOnWriteStateTable.get(CopyOnWriteStateTable.java:311)
        at org.apache.flink.runtime.state.heap.AbstractHeapAppendingState.getInternal(AbstractHeapAppendingState.java:57)
        at org.apache.flink.runtime.state.heap.HeapListState.get(HeapListState.java:85)
        at org.apache.flink.runtime.state.heap.HeapListState.get(HeapListState.java:43)
        at org.apache.flink.streaming.runtime.operators.windowing.WindowOperator.onProcessingTime(WindowOperator.java:498)
        at org.apache.flink.streaming.api.operators.InternalTimerServiceImpl.onProcessingTime(InternalTimerServiceImpl.java:235)
        at org.apache.flink.streaming.runtime.tasks.SystemProcessingTimeService$TriggerTask.run(SystemProcessingTimeService.java:285)
        ... 7 more
Caused by: java.lang.ArrayIndexOutOfBoundsException: 1024
        at com.esotericsoftware.kryo.util.IdentityObjectIntMap.getStash(IdentityObjectIntMap.java:256)
        at com.esotericsoftware.kryo.util.IdentityObjectIntMap.get(IdentityObjectIntMap.java:247)
        at com.esotericsoftware.kryo.util.MapReferenceResolver.getWrittenId(MapReferenceResolver.java:28)
        at com.esotericsoftware.kryo.Kryo.writeReferenceOrNull(Kryo.java:619)
        at com.esotericsoftware.kryo.Kryo.writeObjectOrNull(Kryo.java:564)
        at com.esotericsoftware.kryo.serializers.MapSerializer.write(MapSerializer.java:84)
        at com.esotericsoftware.kryo.serializers.MapSerializer.write(MapSerializer.java:21)
        at com.esotericsoftware.kryo.Kryo.writeObject(Kryo.java:523)
        at com.esotericsoftware.kryo.serializers.ObjectField.write(ObjectField.java:61)
        ... 22 more


Regards,
Scott






-- 








_This message, including any attachments, may include private, 
privileged and confidential information and is intended only for the 
personal and confidential use of the intended recipient(s). If the reader 
of this message is not an intended recipient, you are hereby notified that 
any review, use, dissemination, distribution, printing or copying of this 
message or its contents is strictly prohibited and may be unlawful. If you 
are not an intended recipient or have received this communication in error, 
please immediately notify the sender by telephone and/or a reply email and 
permanently delete the original message, including any attachments, without 
making a copy._