Non-serializable objects as Bolt members

["Nick R. Katsipoulakis" <ni...@gmail.com>]

Hello,

I am currently working on a project in which I need to keep ZooKeeper
object members in the bolts of my topology. Am I allowed to do that? For
instance, can I have something like the following:

public class MyBolt extends BaseRichBolt {

    private ZooKeeper zk;

    public void prepare(Map conf, TopologyContext context, OutputCollector
collector) {
        zk = new ZooKeeper"127.0.0.1", 2181);
    }

    public void execute(Tuple tuple) {
        //Do something...
        zk.exists("/", false);
    }
    .....
}

The reason I am asking is because I know that a Bolt needs to be
Serializable and I have a feeling that a ZooKeeper object is not (I am
pretty confident about that, since it keeps a socket connection with the
zookeeper server). Can somebody give me more information on the topic?
Also, If I go ahead and use something like what I described earlier, what
might be the possible problems that I might run into? Apparently, in local
mode the above works.

Thank you,
Nick

Re: Non-serializable objects as Bolt members

[Parth Brahmbhatt <pb...@hortonworks.com>]

Transient just tells java serialization that serialization should ignore the field mark as transient and does not play any role in deciding GC eligibility.

On worker side the bolt object's prepare method is called once during initialization of that object. The bolt object is just like any other instance so as long as the executors/tasks are running and referring to your instance, it should not be GC eligible.
If the worker dies and some other worker picks up the work it will call prepare again and you will get a new zookeeper instance.

Thanks
Parth


From: "Nick R. Katsipoulakis" <ni...@gmail.com>>
Date: Wednesday, January 28, 2015 at 3:44 PM
To: Parth Brahmbhatt <pb...@hortonworks.com>>
Cc: "user@storm.apache.org<ma...@storm.apache.org>" <us...@storm.apache.org>>
Subject: Re: Non-serializable objects as Bolt members

I thought of transient, but the problem is that I want my ZooKeeper object to persist among consecutive invocations of the execute() function. Is the "transient" keyword going to guarantee the ZooKeeper object's persistence?

Re: Non-serializable objects as Bolt members

["Nick R. Katsipoulakis" <ni...@gmail.com>]

Hello Parth and thank you for your answer,

I thought of transient, but the problem is that I want my ZooKeeper object
to persist among consecutive invocations of the execute() function. Is the
"transient" keyword going to guarantee the ZooKeeper object's persistence?

Thanks
Nick

2015-01-28 18:41 GMT-05:00 Parth Brahmbhatt <pb...@hortonworks.com>:

>  You can mark zookeeper instance as transient.
>
>  Thanks
> Parth
>
>   From: "Nick R. Katsipoulakis" <ni...@gmail.com>
> Reply-To: "user@storm.apache.org" <us...@storm.apache.org>
> Date: Wednesday, January 28, 2015 at 3:38 PM
> To: "user@storm.apache.org" <us...@storm.apache.org>
> Subject: Non-serializable objects as Bolt members
>
>  The reason I am asking is because I know that a Bolt needs to be
> Serializable and I have a feeling that a ZooKeeper object is not (I am
> pretty confident about that, since it keeps a socket connection with the
> zookeeper server). Can somebody give me more information on the topic?
> Also, If I go ahead and use something like what I described earlier, what
> might be the possible problems that I might run into? Apparently, in local
> mode the above works.
>
>


-- 
Nikolaos Romanos Katsipoulakis,
University of Pittsburgh, PhD candidate

Re: Non-serializable objects as Bolt members

[Parth Brahmbhatt <pb...@hortonworks.com>]

You can mark zookeeper instance as transient.

Thanks
Parth

From: "Nick R. Katsipoulakis" <ni...@gmail.com>>
Reply-To: "user@storm.apache.org<ma...@storm.apache.org>" <us...@storm.apache.org>>
Date: Wednesday, January 28, 2015 at 3:38 PM
To: "user@storm.apache.org<ma...@storm.apache.org>" <us...@storm.apache.org>>
Subject: Non-serializable objects as Bolt members

The reason I am asking is because I know that a Bolt needs to be Serializable and I have a feeling that a ZooKeeper object is not (I am pretty confident about that, since it keeps a socket connection with the zookeeper server). Can somebody give me more information on the topic? Also, If I go ahead and use something like what I described earlier, what might be the possible problems that I might run into? Apparently, in local mode the above works.