You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@storm.apache.org by Kushan Maskey <ku...@mmillerassociates.com> on 2015/06/17 23:04:21 UTC
Singleton database connection
I am wondering how can I safely create a Singleton database connection and
use it through out the process.
Here is my scenario.
I got bunch of bolts that updates Cassandra database. I have reader and
writer classes where each needs a database session to execute queries. We
create session on each of the reader and writer classes. We also use
parallelism hints on the bolts which create that many instances of session
object.
I have tried creating session on topology and pass the instance through
stormConfig but didnt work with the error message below.
java.lang.IllegalArgumentException: Topology conf is not json-serializable
at backtype.storm.testing$submit_local_topology.invoke(testing.clj:251)
~[storm-core-0.9.2-incubating.jar:0.9.2-incubating]
at
backtype.storm.LocalCluster$_submitTopology.invoke(LocalCluster.clj:38)
~[storm-core-0.9.2-incubating.jar:0.9.2-incubating]
at backtype.storm.LocalCluster.submitTopology(Unknown Source)
~[storm-core-0.9.2-incubating.jar:0.9.2-incubating]
Any code sample will be helpful.
--
Kushan Maskey
817.403.7500
Precocity LLC <http://precocity.com>
M. Miller & Associates <http://mmillerassociates.com/>
kushan.maskey@mmillerassociates.com
Re: Singleton database connection
Posted by Javier Gonzalez <ja...@gmail.com>.
That I don't know - mind you that, if you're running more than one worker,
you're probably using a singleton _per jvm_, not a single instance across
the whole cluster. You could try assigning a UUID or something like that to
your singleton and printing it to the logs from every bolt, see if it's the
same one across all bolts.
Regards,
JG
On Thu, Jun 18, 2015 at 4:04 PM, Kushan Maskey <
kushan.maskey@mmillerassociates.com> wrote:
> Thanks Nathan and Javier.
>
> I ended up using lazy init singleton and things seem to be working. But
> now how do I know its using single instance of the session? Is there a way
> we can know from storm UI or and where else?
>
> --
> Kushan Maskey
> 817.403.7500
> Precocity LLC <http://precocity.com>
> M. Miller & Associates <http://mmillerassociates.com/>
> kushan.maskey@mmillerassociates.com
>
> On Thu, Jun 18, 2015 at 12:48 PM, Javier Gonzalez <ja...@gmail.com>
> wrote:
>
>> We had a similar issue (namely, you can't pass anything non-serializable
>> through the Configuration or create it in the constructor).
>>
>> What we did is pass in the Configuration or constructor a String with the
>> path to a properties file. From that configuration file, in the
>> bolt.prepare method you bring up whatever you need (such as a connection
>> pool) and off you go. We needed to use a library that was configured by
>> spring, so we passed in the path to the spring configuration file, created
>> a context with it in the prepare method, obtained the external beans, and
>> closed the context.
>>
>> On Wed, Jun 17, 2015 at 5:16 PM, Nathan Leung <nc...@gmail.com> wrote:
>>
>>> You cannot pass a connection through the builder, as you have noticed.
>>> You can use a lazy initialized singleton or some sort of connection pool.
>>> On Jun 17, 2015 5:05 PM, "Kushan Maskey" <
>>> kushan.maskey@mmillerassociates.com> wrote:
>>>
>>>> I am wondering how can I safely create a Singleton database connection
>>>> and use it through out the process.
>>>>
>>>> Here is my scenario.
>>>>
>>>> I got bunch of bolts that updates Cassandra database. I have reader and
>>>> writer classes where each needs a database session to execute queries. We
>>>> create session on each of the reader and writer classes. We also use
>>>> parallelism hints on the bolts which create that many instances of session
>>>> object.
>>>>
>>>> I have tried creating session on topology and pass the instance through
>>>> stormConfig but didnt work with the error message below.
>>>>
>>>> java.lang.IllegalArgumentException: Topology conf is not
>>>> json-serializable
>>>> at
>>>> backtype.storm.testing$submit_local_topology.invoke(testing.clj:251)
>>>> ~[storm-core-0.9.2-incubating.jar:0.9.2-incubating]
>>>> at
>>>> backtype.storm.LocalCluster$_submitTopology.invoke(LocalCluster.clj:38)
>>>> ~[storm-core-0.9.2-incubating.jar:0.9.2-incubating]
>>>> at backtype.storm.LocalCluster.submitTopology(Unknown Source)
>>>> ~[storm-core-0.9.2-incubating.jar:0.9.2-incubating]
>>>>
>>>> Any code sample will be helpful.
>>>>
>>>> --
>>>> Kushan Maskey
>>>> 817.403.7500
>>>> Precocity LLC <http://precocity.com>
>>>> M. Miller & Associates <http://mmillerassociates.com/>
>>>> kushan.maskey@mmillerassociates.com
>>>>
>>>
>>
>>
>> --
>> Javier González Nicolini
>>
>
>
--
Javier González Nicolini
Re: Singleton database connection
Posted by Kushan Maskey <ku...@mmillerassociates.com>.
Thanks Nathan and Javier.
I ended up using lazy init singleton and things seem to be working. But now
how do I know its using single instance of the session? Is there a way we
can know from storm UI or and where else?
--
Kushan Maskey
817.403.7500
Precocity LLC <http://precocity.com>
M. Miller & Associates <http://mmillerassociates.com/>
kushan.maskey@mmillerassociates.com
On Thu, Jun 18, 2015 at 12:48 PM, Javier Gonzalez <ja...@gmail.com>
wrote:
> We had a similar issue (namely, you can't pass anything non-serializable
> through the Configuration or create it in the constructor).
>
> What we did is pass in the Configuration or constructor a String with the
> path to a properties file. From that configuration file, in the
> bolt.prepare method you bring up whatever you need (such as a connection
> pool) and off you go. We needed to use a library that was configured by
> spring, so we passed in the path to the spring configuration file, created
> a context with it in the prepare method, obtained the external beans, and
> closed the context.
>
> On Wed, Jun 17, 2015 at 5:16 PM, Nathan Leung <nc...@gmail.com> wrote:
>
>> You cannot pass a connection through the builder, as you have noticed.
>> You can use a lazy initialized singleton or some sort of connection pool.
>> On Jun 17, 2015 5:05 PM, "Kushan Maskey" <
>> kushan.maskey@mmillerassociates.com> wrote:
>>
>>> I am wondering how can I safely create a Singleton database connection
>>> and use it through out the process.
>>>
>>> Here is my scenario.
>>>
>>> I got bunch of bolts that updates Cassandra database. I have reader and
>>> writer classes where each needs a database session to execute queries. We
>>> create session on each of the reader and writer classes. We also use
>>> parallelism hints on the bolts which create that many instances of session
>>> object.
>>>
>>> I have tried creating session on topology and pass the instance through
>>> stormConfig but didnt work with the error message below.
>>>
>>> java.lang.IllegalArgumentException: Topology conf is not
>>> json-serializable
>>> at
>>> backtype.storm.testing$submit_local_topology.invoke(testing.clj:251)
>>> ~[storm-core-0.9.2-incubating.jar:0.9.2-incubating]
>>> at
>>> backtype.storm.LocalCluster$_submitTopology.invoke(LocalCluster.clj:38)
>>> ~[storm-core-0.9.2-incubating.jar:0.9.2-incubating]
>>> at backtype.storm.LocalCluster.submitTopology(Unknown Source)
>>> ~[storm-core-0.9.2-incubating.jar:0.9.2-incubating]
>>>
>>> Any code sample will be helpful.
>>>
>>> --
>>> Kushan Maskey
>>> 817.403.7500
>>> Precocity LLC <http://precocity.com>
>>> M. Miller & Associates <http://mmillerassociates.com/>
>>> kushan.maskey@mmillerassociates.com
>>>
>>
>
>
> --
> Javier González Nicolini
>
Re: Singleton database connection
Posted by Javier Gonzalez <ja...@gmail.com>.
We had a similar issue (namely, you can't pass anything non-serializable
through the Configuration or create it in the constructor).
What we did is pass in the Configuration or constructor a String with the
path to a properties file. From that configuration file, in the
bolt.prepare method you bring up whatever you need (such as a connection
pool) and off you go. We needed to use a library that was configured by
spring, so we passed in the path to the spring configuration file, created
a context with it in the prepare method, obtained the external beans, and
closed the context.
On Wed, Jun 17, 2015 at 5:16 PM, Nathan Leung <nc...@gmail.com> wrote:
> You cannot pass a connection through the builder, as you have noticed. You
> can use a lazy initialized singleton or some sort of connection pool.
> On Jun 17, 2015 5:05 PM, "Kushan Maskey" <
> kushan.maskey@mmillerassociates.com> wrote:
>
>> I am wondering how can I safely create a Singleton database connection
>> and use it through out the process.
>>
>> Here is my scenario.
>>
>> I got bunch of bolts that updates Cassandra database. I have reader and
>> writer classes where each needs a database session to execute queries. We
>> create session on each of the reader and writer classes. We also use
>> parallelism hints on the bolts which create that many instances of session
>> object.
>>
>> I have tried creating session on topology and pass the instance through
>> stormConfig but didnt work with the error message below.
>>
>> java.lang.IllegalArgumentException: Topology conf is not json-serializable
>> at
>> backtype.storm.testing$submit_local_topology.invoke(testing.clj:251)
>> ~[storm-core-0.9.2-incubating.jar:0.9.2-incubating]
>> at
>> backtype.storm.LocalCluster$_submitTopology.invoke(LocalCluster.clj:38)
>> ~[storm-core-0.9.2-incubating.jar:0.9.2-incubating]
>> at backtype.storm.LocalCluster.submitTopology(Unknown Source)
>> ~[storm-core-0.9.2-incubating.jar:0.9.2-incubating]
>>
>> Any code sample will be helpful.
>>
>> --
>> Kushan Maskey
>> 817.403.7500
>> Precocity LLC <http://precocity.com>
>> M. Miller & Associates <http://mmillerassociates.com/>
>> kushan.maskey@mmillerassociates.com
>>
>
--
Javier González Nicolini
Re: Singleton database connection
Posted by Nathan Leung <nc...@gmail.com>.
You cannot pass a connection through the builder, as you have noticed. You
can use a lazy initialized singleton or some sort of connection pool.
On Jun 17, 2015 5:05 PM, "Kushan Maskey" <
kushan.maskey@mmillerassociates.com> wrote:
> I am wondering how can I safely create a Singleton database connection and
> use it through out the process.
>
> Here is my scenario.
>
> I got bunch of bolts that updates Cassandra database. I have reader and
> writer classes where each needs a database session to execute queries. We
> create session on each of the reader and writer classes. We also use
> parallelism hints on the bolts which create that many instances of session
> object.
>
> I have tried creating session on topology and pass the instance through
> stormConfig but didnt work with the error message below.
>
> java.lang.IllegalArgumentException: Topology conf is not json-serializable
> at
> backtype.storm.testing$submit_local_topology.invoke(testing.clj:251)
> ~[storm-core-0.9.2-incubating.jar:0.9.2-incubating]
> at
> backtype.storm.LocalCluster$_submitTopology.invoke(LocalCluster.clj:38)
> ~[storm-core-0.9.2-incubating.jar:0.9.2-incubating]
> at backtype.storm.LocalCluster.submitTopology(Unknown Source)
> ~[storm-core-0.9.2-incubating.jar:0.9.2-incubating]
>
> Any code sample will be helpful.
>
> --
> Kushan Maskey
> 817.403.7500
> Precocity LLC <http://precocity.com>
> M. Miller & Associates <http://mmillerassociates.com/>
> kushan.maskey@mmillerassociates.com
>