You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@storm.apache.org by "Julien Nioche (Jira)" <ji...@apache.org> on 2020/02/14 09:11:00 UTC
[jira] [Created] (STORM-3582) Kryo errors when using custom
serialization
Julien Nioche created STORM-3582:
------------------------------------
Summary: Kryo errors when using custom serialization
Key: STORM-3582
URL: https://issues.apache.org/jira/browse/STORM-3582
Project: Apache Storm
Issue Type: Bug
Components: storm-core
Affects Versions: 2.1.0
Reporter: Julien Nioche
(this has been reported on the user list in January and previously by a different user)
------------------------------------------------------------------------------------
My code works fine with Storm 1.x but the workers crash constantly with Storm 2.x
Some exceptions look like
{{_com.esotericsoftware.kryo.KryoException: Buffer underflow._}}
{{_at com.esotericsoftware.kryo.io.Input.require(Input.java:199) ~[kryo-3.0.3.jar:?]_}}
{{_at com.esotericsoftware.kryo.io.Input.readUtf8_slow(Input.java:575) ~[kryo-3.0.3.jar:?]_}}
{{_at com.esotericsoftware.kryo.io.Input.readUtf8(Input.java:553) ~[kryo-3.0.3.jar:?]_}}
{{_at com.esotericsoftware.kryo.io.Input.readString(Input.java:483) ~[kryo-3.0.3.jar:?]_}}
{{whereas others are}}
{{_2020-01-21 11:13:39.368 o.a.s.m.n.StormServerHandler Netty-server-localhost-6701-worker-1 [ERROR] server errors in handling the request_}}
{{_com.esotericsoftware.kryo.KryoException: Encountered unregistered class ID: *95*
at com.esotericsoftware.kryo.util.DefaultClassResolver.readClass(DefaultClassResolver.java:137) ~[kryo-3.0.3.jar:?]
at com.esotericsoftware.kryo.Kryo.readClass(Kryo.java:670) ~[kryo-3.0.3.jar:?]
at com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:781) ~[kryo-3.0.3.jar:?]
at com.esotericsoftware.kryo.serializers.MapSerializer.read(MapSerializer.java:153) ~[kryo-3.0.3.jar:?]
at com.esotericsoftware.kryo.serializers.MapSerializer.read(MapSerializer.java:39) ~[kryo-3.0.3.jar:?]
at com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:793) ~[kryo-3.0.3.jar:?]
at com.esotericsoftware.kryo.serializers.CollectionSerializer.read(CollectionSerializer.java:134) ~[kryo-3.0.3.jar:?]
at com.esotericsoftware.kryo.serializers.CollectionSerializer.read(CollectionSerializer.java:40) ~[kryo-3.0.3.jar:?]
at com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:689) ~[kryo-3.0.3.jar:?]
at org.apache.storm.serialization.KryoValuesDeserializer.deserializeFrom(KryoValuesDeserializer.java:31) ~[storm-client-2.1.0.jar:2.1.0]_}}
{{the class ID values are random integers. }}
{{}}
The tuples in my topologies contain mostly standard classes like String; the only exception is the following class
[https://github.com/DigitalPebble/storm-crawler/blob/2.x/core/src/main/java/com/digitalpebble/stormcrawler/Metadata.java#L41]
for which we specify a custom serialization for Kryo.
My configurations contain
| topology.kryo.register:|- com.digitalpebble.stormcrawler.Metadata|
to register the custom class.
The user who had reported the problem first also used custom serialization.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)