You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@storm.apache.org by anshu shukla <an...@gmail.com> on 2017/03/21 03:10:03 UTC

Converting storm tuple to bytearray

Hello,

*1- Can anyone explain how to convert storm tuple to byte array. I have
used the following code. *

private byte[] convertToBytes(Object object) throws IOException {
    try (ByteArrayOutputStream bos = new ByteArrayOutputStream();
ObjectOutput out = new ObjectOutputStream(bos)) {
     out.writeObject(object);
        return bos.toByteArray(); } }

*Getting the following error :*
java.io.NotSerializableException: org.apache.storm.tuple.TupleImpl
at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1183)
at java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:347)
at
storm.starter.OurStatefulBoltByteArrayTuple.convertToBytes(OurStatefulBoltByteArrayTuple.java:200)

*2- If the conversion can not be done like this then how does storm does
serialization of tuple over the network. Can anyone point me out that code
logic? *

Re: Converting storm tuple to bytearray

Posted by Arun Mahadevan <ar...@apache.org>.
Why don’t you rely on storm’s replay mechanisms to have the tuple replayed? Are you trying to handle the state once the number of tasks are increased? The replayed tuples can go to the new tasks as long as the state is migrated correctly.

 

I don’t fully get what you are trying to do. Maybe you can put your idea in some google doc and share it.

 

Thanks,

Arun

 

 

From: anshu shukla <an...@gmail.com>
Reply-To: "user@storm.apache.org" <us...@storm.apache.org>
Date: Tuesday, March 21, 2017 at 9:59 AM
To: "user@storm.apache.org" <us...@storm.apache.org>
Subject: Re: Converting storm tuple to bytearray

 

But suppose I want to replay value in the tuple to the older taskID or StreamID etc. all those details will be lost . (I am doing this for replaying tuples after state migration.)

 

On Tue, Mar 21, 2017 at 9:31 AM, Arun Mahadevan <ar...@apache.org> wrote:

Storm uses Kryo to serialize Tuples. Check this https://github.com/apache/storm/blob/master/storm-core/src/jvm/org/apache/storm/serialization/KryoTupleSerializer.java

 

Instead of serializing the entire tuple yourself may be you just want to serialize the relevant values within the tuple.

 

 

From: anshu shukla <an...@gmail.com>
Reply-To: "user@storm.apache.org" <us...@storm.apache.org>
Date: Tuesday, March 21, 2017 at 8:40 AM
To: "user@storm.apache.org" <us...@storm.apache.org>
Subject: Converting storm tuple to bytearray

 

Hello, 

 

1- Can anyone explain how to convert storm tuple to byte array. I have used the following code. 

 

private byte[] convertToBytes(Object object) throws IOException {

    try (ByteArrayOutputStream bos = new ByteArrayOutputStream();        ObjectOutput out = new ObjectOutputStream(bos)) {   

     out.writeObject(object);

        return bos.toByteArray(); } }

 

Getting the following error :

java.io.NotSerializableException: org.apache.storm.tuple.TupleImpl

at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1183)

at java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:347)

at storm.starter.OurStatefulBoltByteArrayTuple.convertToBytes(OurStatefulBoltByteArrayTuple.java:200)

 

2- If the conversion can not be done like this then how does storm does serialization of tuple over the network. Can anyone point me out that code logic? 



 

-- 

Thanks & Regards,
Anshu Shukla

Re: Converting storm tuple to bytearray

Posted by anshu shukla <an...@gmail.com>.
But suppose I want to replay value in the tuple to the older taskID or
StreamID etc. all those details will be lost . (I am doing this for
replaying tuples after state migration.)

On Tue, Mar 21, 2017 at 9:31 AM, Arun Mahadevan <ar...@apache.org> wrote:

> Storm uses Kryo to serialize Tuples. Check this https://github.com/apache/
> storm/blob/master/storm-core/src/jvm/org/apache/storm/serialization/
> KryoTupleSerializer.java
>
>
>
> Instead of serializing the entire tuple yourself may be you just want to
> serialize the relevant values within the tuple.
>
>
>
>
>
> *From: *anshu shukla <an...@gmail.com>
> *Reply-To: *"user@storm.apache.org" <us...@storm.apache.org>
> *Date: *Tuesday, March 21, 2017 at 8:40 AM
> *To: *"user@storm.apache.org" <us...@storm.apache.org>
> *Subject: *Converting storm tuple to bytearray
>
>
>
> Hello,
>
>
>
> *1- Can anyone explain how to convert storm tuple to byte array. I have
> used the following code. *
>
>
>
> *private byte*[] convertToBytes(Object object) *throws *IOException {
>
>     *try *(ByteArrayOutputStream bos = *new *ByteArrayOutputStream();
>     ObjectOutput out = *new *ObjectOutputStream(bos)) {
>
>      out.writeObject(object);
>
>         *return *bos.toByteArray(); } }
>
>
>
> *Getting the following error :*
>
> java.io.NotSerializableException: org.apache.storm.tuple.TupleImpl
>
> at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1183)
>
> at java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:347)
>
> at storm.starter.OurStatefulBoltByteArrayTuple.convertToBytes(
> OurStatefulBoltByteArrayTuple.java:200)
>
>
>
> *2- If the conversion can not be done like this then how does storm does
> serialization of tuple over the network. Can anyone point me out that code
> logic? *
>



-- 
Thanks & Regards,
Anshu Shukla

Re: Converting storm tuple to bytearray

Posted by Arun Mahadevan <ar...@apache.org>.
Storm uses Kryo to serialize Tuples. Check this https://github.com/apache/storm/blob/master/storm-core/src/jvm/org/apache/storm/serialization/KryoTupleSerializer.java

 

Instead of serializing the entire tuple yourself may be you just want to serialize the relevant values within the tuple.

 

 

From: anshu shukla <an...@gmail.com>
Reply-To: "user@storm.apache.org" <us...@storm.apache.org>
Date: Tuesday, March 21, 2017 at 8:40 AM
To: "user@storm.apache.org" <us...@storm.apache.org>
Subject: Converting storm tuple to bytearray

 

Hello, 

 

1- Can anyone explain how to convert storm tuple to byte array. I have used the following code. 

 

private byte[] convertToBytes(Object object) throws IOException {

    try (ByteArrayOutputStream bos = new ByteArrayOutputStream();        ObjectOutput out = new ObjectOutputStream(bos)) {   

     out.writeObject(object);

        return bos.toByteArray(); } }

 

Getting the following error :

java.io.NotSerializableException: org.apache.storm.tuple.TupleImpl

at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1183)

at java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:347)

at storm.starter.OurStatefulBoltByteArrayTuple.convertToBytes(OurStatefulBoltByteArrayTuple.java:200)

 

2- If the conversion can not be done like this then how does storm does serialization of tuple over the network. Can anyone point me out that code logic?