You are viewing a plain text version of this content. The canonical link for it is here.

Posted to user@storm.apache.org by Frank Bolder <fr...@gmail.com> on 2014/10/24 00:14:55 UTC

How to deserialize stormcode.ser

Hi to every Storm user,
congratulations for your precious work. I'm quite new to Storm and Java so
I'm sorry for my maybe trivial questions.
I'm trying to deserialize the file stormcode.ser that should contain the
StormTopology object in JSON, but I'm getting troubles. It's very hard for
me understanding the entire Storm code (Clojure + Java).

I ran the WordCount topology in Local Mode inside Eclipse.
Using FileInputStream, ObjectInputStream, readObject() and a StormTopology
object I managed to get a readable version of the topology. This is a frame
of the result:

StormTopology(spouts:{word-reader-SPOUT=SpoutSpec(spout_object:<ComponentObject
serialized_java:AC ED 00 05 73 72 00 1A 73 74 6F ... 64 65 72...>,
common:ComponentCommon(inputs:{},
streams:{default=StreamInfo(output_fields:[line], direct:false)},
json_conf:{"topology.tasks":1}))}, bolts:....

I'm not able to read/access this part:
<ComponentObject serialized_java:AC ED 00 05 73 72 00 1A 73 74 6F ... 64 65
72...>
It is a bytebuffer type, java.nio.heapbytebuffer that should contain
objects and data related to the WordReader.Java (class of the spout) I
wrote.

With get_spout_object() I obtained the spout ComponentObject.
With Utils.getSetComponentObject() I should have obtained the previous
object in a deserialized shape.
If I print it, it displays "mypackage.spouts.WordReader@53c015".
Then I also cast it to the class of the spout that I wrote
(WordReader.Java).
So it seems to be the right class object.

But the problem is that this object seems to be empty (collector=null,
completed = false, fileReader=null).
In particular I used fileReader so its data shoud be present.

I don't understand where is the problem.
Can you help me to find the mistakes and deserialize the following part?

<ComponentObject serialized_java:AC ED 00 05 73 72 00 1A 73 74 6F ... 64 65
72...>

Re: How to deserialize stormcode.ser

Posted by Frank Bolder <fr...@gmail.com>.

First of all thank you for your answer Taylor,
I know but it's very educational, I like to understand how things work ;)

I can reformulate my question:

1) Does stormcode.ser file contain the instantiated objects related to
spouts (eg: FileReader, Collector, .. , the objects inside NextTuple
methods and Open() methods..etc) and bolts that I have written in the Java
source code for spouts and bolt ? Or it contains just the structure of the
topology and the instantiated objects are put inside the topology later at
runtime?

Actually, yes indeed the topology code in Java is more readable but for a
moment let's suppose that another person wrote the topology code in Java
and run/executed it. So I would like to access the topology and its data
(objects defined inside it and hopefully the processing code written inside
the methods).

So the second question is:

2) Can you suggest me some ways to access the topology and its objects
before that the scheduler
definitively would execute the topology?



2014-10-24 22:25 GMT+02:00 P. Taylor Goetz <pt...@gmail.com>:

> What are you trying to do that led you down the path of trying to
> deserialize stormcode.ser? That file is just a serialized copy of the
> topology structure. The topology code that created it is going to be
> infinitely more readable. ;)
>
>
> -Taylor
>
> On Oct 23, 2014, at 6:14 PM, Frank Bolder <fr...@gmail.com> wrote:
>
> > Hi to every Storm user,
> > congratulations for your precious work. I'm quite new to Storm and Java
> so I'm sorry for my maybe trivial questions.
> > I'm trying to deserialize the file stormcode.ser that should contain the
> StormTopology object in JSON, but I'm getting troubles. It's very hard for
> me understanding the entire Storm code (Clojure + Java).
> >
> > I ran the WordCount topology in Local Mode inside Eclipse.
> > Using FileInputStream, ObjectInputStream, readObject() and a
> StormTopology object I managed to get a readable version of the topology.
> This is a frame of the result:
> >
> >
> StormTopology(spouts:{word-reader-SPOUT=SpoutSpec(spout_object:<ComponentObject
> serialized_java:AC ED 00 05 73 72 00 1A 73 74 6F ... 64 65 72...>,
> common:ComponentCommon(inputs:{},
> streams:{default=StreamInfo(output_fields:[line], direct:false)},
> json_conf:{"topology.tasks":1}))}, bolts:....
> >
> > I'm not able to read/access this part:
> > <ComponentObject serialized_java:AC ED 00 05 73 72 00 1A 73 74 6F ... 64
> 65 72...>
> > It is a bytebuffer type, java.nio.heapbytebuffer that should contain
> objects and data related to the WordReader.Java (class of the spout) I
> wrote.
> >
> > With get_spout_object() I obtained the spout ComponentObject.
> > With Utils.getSetComponentObject() I should have obtained the previous
> object in a deserialized shape.
> > If I print it, it displays "mypackage.spouts.WordReader@53c015".
> > Then I also cast it to the class of the spout that I wrote
> (WordReader.Java).
> > So it seems to be the right class object.
> >
> > But the problem is that this object seems to be empty (collector=null,
> completed = false, fileReader=null).
> > In particular I used fileReader so its data shoud be present.
> >
> > I don't understand where is the problem.
> > Can you help me to find the mistakes and deserialize the following part?
> >
> > <ComponentObject serialized_java:AC ED 00 05 73 72 00 1A 73 74 6F ... 64
> 65 72...>
> >
> >
> >
>
>

Re: How to deserialize stormcode.ser

Posted by "P. Taylor Goetz" <pt...@gmail.com>.

What are you trying to do that led you down the path of trying to deserialize stormcode.ser? That file is just a serialized copy of the topology structure. The topology code that created it is going to be infinitely more readable. ;)


-Taylor

On Oct 23, 2014, at 6:14 PM, Frank Bolder <fr...@gmail.com> wrote:

> Hi to every Storm user,
> congratulations for your precious work. I'm quite new to Storm and Java so I'm sorry for my maybe trivial questions.
> I'm trying to deserialize the file stormcode.ser that should contain the StormTopology object in JSON, but I'm getting troubles. It's very hard for me understanding the entire Storm code (Clojure + Java). 
> 
> I ran the WordCount topology in Local Mode inside Eclipse.
> Using FileInputStream, ObjectInputStream, readObject() and a StormTopology object I managed to get a readable version of the topology. This is a frame of the result:
> 
> StormTopology(spouts:{word-reader-SPOUT=SpoutSpec(spout_object:<ComponentObject serialized_java:AC ED 00 05 73 72 00 1A 73 74 6F ... 64 65 72...>, common:ComponentCommon(inputs:{}, streams:{default=StreamInfo(output_fields:[line], direct:false)}, json_conf:{"topology.tasks":1}))}, bolts:....
> 
> I'm not able to read/access this part:
> <ComponentObject serialized_java:AC ED 00 05 73 72 00 1A 73 74 6F ... 64 65 72...>
> It is a bytebuffer type, java.nio.heapbytebuffer that should contain objects and data related to the WordReader.Java (class of the spout) I wrote.
> 
> With get_spout_object() I obtained the spout ComponentObject.
> With Utils.getSetComponentObject() I should have obtained the previous object in a deserialized shape. 
> If I print it, it displays "mypackage.spouts.WordReader@53c015".
> Then I also cast it to the class of the spout that I wrote (WordReader.Java).
> So it seems to be the right class object.
> 
> But the problem is that this object seems to be empty (collector=null, completed = false, fileReader=null).
> In particular I used fileReader so its data shoud be present.
> 
> I don't understand where is the problem.
> Can you help me to find the mistakes and deserialize the following part?
> 
> <ComponentObject serialized_java:AC ED 00 05 73 72 00 1A 73 74 6F ... 64 65 72...>
> 
> 
>