You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@storm.apache.org by Gergely Nagy <fo...@gmail.com> on 2015/12/14 13:54:01 UTC
java.io.NotSerializableException: redis.clients.jedis.BinaryJedis$SetFromList
Hello fellow storm troopers.
I am trying to build a storm topology in Storm 0.10.0. I am also trying to
use storm-redis to save some interim results during operation.
Now my problem is that when I run my topology locally everything works
fine, but when I run it remotely, storm sends me this nice stacktrace:
https://gist.github.com/Fogetti/23304178a534fa1f8f63
I don't get it. What's going on? Why is storm serializing BinaryJedis?
Shouldn't BinaryJedis be transient? And anyway, what is this cryptic thing
called 'consumeBatchToCursor()' doing?
The "offending" class can be seen below. It's just a simple bolt extending
AbstractRedisBolt.
https://github.com/Fogetti/fogetti-phish-storm/blob/master/src/main/java/fogetti/phish/storm/relatedness/GoogleSemBolt.java
Does anyone know what might be the root cause for this?
Regards,
Gergely
Re: java.io.NotSerializableException: redis.clients.jedis.BinaryJedis$SetFromList
Posted by Gergely Nagy <fo...@gmail.com>.
Thank you Jungtaek!
Yup. You are exactly right. I was too tired or something and I didn't
realize that the error is complaining about the inner class. Thank you for
taking your time to look into this.
Regards,
Gergely
2015-12-14 23:41 GMT+09:00 임정택 <ka...@gmail.com>:
> Hi Gergely,
>
> Thanks for using storm-redis, and report issue.
>
> The thing Storm was trying to serialize is BinaryJedis$SetFromList, not
> BinaryJedis.
>
>
> https://github.com/Fogetti/fogetti-phish-storm/blob/master/src/main/java/fogetti/phish/storm/relatedness/GoogleSemBolt.java#L74
>
> Jedis.smembers() returns SetFromList, which wraps List to provide Set
> interface.
> (It is just one of the optimizations in Jedis.)
>
> Unfortunately SetFromList doesn't implement Serializable for now, so you
> can encounter NotSerializableException when trying to emit the result of
> smembers directly.
>
> Two workarounds are here,
>
> 1. Implement custom Kyro serializer for SetFromList and register to
> topology.
> http://storm.apache.org/documentation/Serialization.html
>
> 2. Copy the result into HashSet and emit instead.
>
> I'll try to craft the patch to Jedis which makes SetFromList to implement
> Serializable.
> Before applying and releasing, I'll also try to make storm-redis to
> provide easy workaround. (Considering 1. for now.)
>
> Thanks!
> Jungtaek Lim (HeartSaVioR)
>
>
>
>
>
> 2015-12-14 21:54 GMT+09:00 Gergely Nagy <fo...@gmail.com>:
>
>> Hello fellow storm troopers.
>>
>> I am trying to build a storm topology in Storm 0.10.0. I am also trying
>> to use storm-redis to save some interim results during operation.
>>
>> Now my problem is that when I run my topology locally everything works
>> fine, but when I run it remotely, storm sends me this nice stacktrace:
>> https://gist.github.com/Fogetti/23304178a534fa1f8f63
>>
>> I don't get it. What's going on? Why is storm serializing BinaryJedis?
>> Shouldn't BinaryJedis be transient? And anyway, what is this cryptic thing
>> called 'consumeBatchToCursor()' doing?
>>
>> The "offending" class can be seen below. It's just a simple bolt
>> extending AbstractRedisBolt.
>>
>> https://github.com/Fogetti/fogetti-phish-storm/blob/master/src/main/java/fogetti/phish/storm/relatedness/GoogleSemBolt.java
>>
>> Does anyone know what might be the root cause for this?
>>
>> Regards,
>> Gergely
>>
>
>
>
> --
> Name : 임 정택
> Blog : http://www.heartsavior.net / http://dev.heartsavior.net
> Twitter : http://twitter.com/heartsavior
> LinkedIn : http://www.linkedin.com/in/heartsavior
>
Re: java.io.NotSerializableException: redis.clients.jedis.BinaryJedis$SetFromList
Posted by 임정택 <ka...@gmail.com>.
Hi Gergely,
Thanks for using storm-redis, and report issue.
The thing Storm was trying to serialize is BinaryJedis$SetFromList, not
BinaryJedis.
https://github.com/Fogetti/fogetti-phish-storm/blob/master/src/main/java/fogetti/phish/storm/relatedness/GoogleSemBolt.java#L74
Jedis.smembers() returns SetFromList, which wraps List to provide Set
interface.
(It is just one of the optimizations in Jedis.)
Unfortunately SetFromList doesn't implement Serializable for now, so you
can encounter NotSerializableException when trying to emit the result of
smembers directly.
Two workarounds are here,
1. Implement custom Kyro serializer for SetFromList and register to
topology.
http://storm.apache.org/documentation/Serialization.html
2. Copy the result into HashSet and emit instead.
I'll try to craft the patch to Jedis which makes SetFromList to implement
Serializable.
Before applying and releasing, I'll also try to make storm-redis to provide
easy workaround. (Considering 1. for now.)
Thanks!
Jungtaek Lim (HeartSaVioR)
2015-12-14 21:54 GMT+09:00 Gergely Nagy <fo...@gmail.com>:
> Hello fellow storm troopers.
>
> I am trying to build a storm topology in Storm 0.10.0. I am also trying to
> use storm-redis to save some interim results during operation.
>
> Now my problem is that when I run my topology locally everything works
> fine, but when I run it remotely, storm sends me this nice stacktrace:
> https://gist.github.com/Fogetti/23304178a534fa1f8f63
>
> I don't get it. What's going on? Why is storm serializing BinaryJedis?
> Shouldn't BinaryJedis be transient? And anyway, what is this cryptic thing
> called 'consumeBatchToCursor()' doing?
>
> The "offending" class can be seen below. It's just a simple bolt extending
> AbstractRedisBolt.
>
> https://github.com/Fogetti/fogetti-phish-storm/blob/master/src/main/java/fogetti/phish/storm/relatedness/GoogleSemBolt.java
>
> Does anyone know what might be the root cause for this?
>
> Regards,
> Gergely
>
--
Name : 임 정택
Blog : http://www.heartsavior.net / http://dev.heartsavior.net
Twitter : http://twitter.com/heartsavior
LinkedIn : http://www.linkedin.com/in/heartsavior