You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@storm.apache.org by Denis Sevosteenko <de...@gmail.com> on 2019/07/15 13:08:04 UTC
Storm and singletons
Hi all,
This may be an old question, but what is the correct way of using Singleton
pattern with Storm?
Currently we have a topology with around 200 bolts, of which around 50 read
and write data from/to Cassandra. Each of these 50 bolts creates a new
Cassandra session, which to my understanding of Cassandra is not great. We
have tried solving this issue with several implementations of singletons:
lazy static, eager static and enum. All of these implementations have
worked in dev environment running on single server, but have fallen apart
on QA environment which has two supervisor hosts. I'm new to Storm so I
have no idea if we are missing something crucial or obvious.
Thanks & Regards,
Denis Sevosteenko
Re: Storm and singletons
Posted by Roshan Naik <ro...@yahoo.com>.
Not sure about Cassandra.. but DB connection objects typically are not thread safe... ie you cannot have to two threads issue queries/write over the same connection at the same time.
On Wednesday, September 11, 2019, 05:00:44 AM PDT, Denis Sevosteenko <de...@gmail.com> wrote:
Hi all,
Just wanted to give an update (in case anyone is interested), since the issue has been solved now. Turns out maven shade plugin was configured incorrectly. We were excluding classes based on mask instead of using recommended provided scope for storm dependency. Switching to maven shade configuration provided as example in storm docs has helped.
Re: Storm and singletons
Posted by Denis Sevosteenko <de...@gmail.com>.
Hi all,
Just wanted to give an update (in case anyone is interested), since the
issue has been solved now. Turns out maven shade plugin was configured
incorrectly. We were excluding classes based on mask instead of using
recommended provided scope for storm dependency. Switching to maven shade
configuration provided as example in storm docs has helped.
Re: Storm and singletons
Posted by Petr Janeček <Ja...@seznam.cz>.
A NoClassDefFoundError always means classpath (or classloader, but
that's unlikely here unless you're on Storm 1.0.0 and less) issues. Take
a look at which class it cannot find and find out why it's not in the
jar you're sending to Storm.
A ExceptionInitializerError tells us nothing without its cause. Some
initializer failed, go take a look which one and why.
Long story short - there's noting special about Storm and singletons. As
always, all the instantiated classes in your topology (things that get
pushed into the TopologyBuilder) must be correctly serializable and
deserializable (did you forget to handle a transient field somewhere?).
Otherwise, without a minima example of the problem I'm afraid there
isn't much we can do for you.
PJ
Dne 15.07.2019 v 15:33 Denis Sevosteenko napsal(a):
> Just to clarify: I know this can possibly happen because of
> incorrectly packaged jar, but there are no such issues when we aren't
> using singletons. Nothing like this happens when I just add a new bolt
> with a new mongodb connection for example.
>
> Also regarding abstracting DB connection logic to a separate bolt: we
> are using a bolt to write data to Cassandra, but I have no idea how to
> do that with reads.
>
> On Mon, Jul 15, 2019 at 4:16 PM Denis Sevosteenko
> <denis.sevosteenko@gmail.com <ma...@gmail.com>> wrote:
>
> We got NoClassDefFoundError and ExceptionInitializerError in
> prepare() methods.
>
> On Mon, Jul 15, 2019 at 4:13 PM Petr Janeček
> <JanecekPetr@seznam.cz <ma...@seznam.cz>> wrote:
>
> Hello,
> you'll need to define "fallen apart". What did not work and
> why? Obviously a singleton is single in one Worker (one JVM).
> For DB connections, this seems to be a good thing - you'll
> want every JVM to have a single (or, rather, a limited pool
> of) DB connection encapsulated in a singleton resource.
>
> If you want a singleton to really be single in the topology,
> you'll need to abstract away the DB connection logic to a
> separate microservice, or a single bolt.
>
> PJ
>
> ---------- Původní e-mail ----------
> Od: Denis Sevosteenko <denis.sevosteenko@gmail.com
> <ma...@gmail.com>>
> Komu: user@storm.apache.org <ma...@storm.apache.org>
> Datum: 15. 7. 2019 15:08:26
> Předmět: Storm and singletons
>
> Hi all,
>
> This may be an old question, but what is the correct way
> of using Singleton pattern with Storm?
>
> Currently we have a topology with around 200 bolts, of
> which around 50 read and write data from/to Cassandra.
> Each of these 50 bolts creates a new Cassandra session,
> which to my understanding of Cassandra is not great. We
> have tried solving this issue with several implementations
> of singletons: lazy static, eager static and enum. All of
> these implementations have worked in dev environment
> running on single server, but have fallen apart on QA
> environment which has two supervisor hosts. I'm new to
> Storm so I have no idea if we are missing something
> crucial or obvious.
>
> Thanks & Regards,
> Denis Sevosteenko
>
Re: Storm and singletons
Posted by Denis Sevosteenko <de...@gmail.com>.
Just to clarify: I know this can possibly happen because of incorrectly
packaged jar, but there are no such issues when we aren't using singletons.
Nothing like this happens when I just add a new bolt with a new mongodb
connection for example.
Also regarding abstracting DB connection logic to a separate bolt: we are
using a bolt to write data to Cassandra, but I have no idea how to do that
with reads.
On Mon, Jul 15, 2019 at 4:16 PM Denis Sevosteenko <
denis.sevosteenko@gmail.com> wrote:
> We got NoClassDefFoundError and ExceptionInitializerError in prepare()
> methods.
>
> On Mon, Jul 15, 2019 at 4:13 PM Petr Janeček <Ja...@seznam.cz>
> wrote:
>
>> Hello,
>> you'll need to define "fallen apart". What did not work and why?
>> Obviously a singleton is single in one Worker (one JVM). For DB
>> connections, this seems to be a good thing - you'll want every JVM to have
>> a single (or, rather, a limited pool of) DB connection encapsulated in a
>> singleton resource.
>>
>> If you want a singleton to really be single in the topology, you'll need
>> to abstract away the DB connection logic to a separate microservice, or a
>> single bolt.
>>
>> PJ
>>
>> ---------- Původní e-mail ----------
>> Od: Denis Sevosteenko <de...@gmail.com>
>> Komu: user@storm.apache.org
>> Datum: 15. 7. 2019 15:08:26
>> Předmět: Storm and singletons
>>
>> Hi all,
>>
>> This may be an old question, but what is the correct way of using
>> Singleton pattern with Storm?
>>
>> Currently we have a topology with around 200 bolts, of which around 50
>> read and write data from/to Cassandra. Each of these 50 bolts creates a new
>> Cassandra session, which to my understanding of Cassandra is not great. We
>> have tried solving this issue with several implementations of singletons:
>> lazy static, eager static and enum. All of these implementations have
>> worked in dev environment running on single server, but have fallen apart
>> on QA environment which has two supervisor hosts. I'm new to Storm so I
>> have no idea if we are missing something crucial or obvious.
>>
>> Thanks & Regards,
>> Denis Sevosteenko
>>
>>
Re: Storm and singletons
Posted by Denis Sevosteenko <de...@gmail.com>.
We got NoClassDefFoundError and ExceptionInitializerError in prepare()
methods.
On Mon, Jul 15, 2019 at 4:13 PM Petr Janeček <Ja...@seznam.cz> wrote:
> Hello,
> you'll need to define "fallen apart". What did not work and why? Obviously
> a singleton is single in one Worker (one JVM). For DB connections, this
> seems to be a good thing - you'll want every JVM to have a single (or,
> rather, a limited pool of) DB connection encapsulated in a singleton
> resource.
>
> If you want a singleton to really be single in the topology, you'll need
> to abstract away the DB connection logic to a separate microservice, or a
> single bolt.
>
> PJ
>
> ---------- Původní e-mail ----------
> Od: Denis Sevosteenko <de...@gmail.com>
> Komu: user@storm.apache.org
> Datum: 15. 7. 2019 15:08:26
> Předmět: Storm and singletons
>
> Hi all,
>
> This may be an old question, but what is the correct way of using
> Singleton pattern with Storm?
>
> Currently we have a topology with around 200 bolts, of which around 50
> read and write data from/to Cassandra. Each of these 50 bolts creates a new
> Cassandra session, which to my understanding of Cassandra is not great. We
> have tried solving this issue with several implementations of singletons:
> lazy static, eager static and enum. All of these implementations have
> worked in dev environment running on single server, but have fallen apart
> on QA environment which has two supervisor hosts. I'm new to Storm so I
> have no idea if we are missing something crucial or obvious.
>
> Thanks & Regards,
> Denis Sevosteenko
>
>
Re: Storm and singletons
Posted by Petr Janeček <Ja...@seznam.cz>.
Hello,
you'll need to define "fallen apart". What did not work and why? Obviously a
singleton is single in one Worker (one JVM). For DB connections, this seems
to be a good thing - you'll want every JVM to have a single (or, rather, a
limited pool of) DB connection encapsulated in a singleton resource.
If you want a singleton to really be single in the topology, you'll need to
abstract away the DB connection logic to a separate microservice, or a
single bolt.
PJ
---------- Původní e-mail ----------
Od: Denis Sevosteenko <de...@gmail.com>
Komu: user@storm.apache.org
Datum: 15. 7. 2019 15:08:26
Předmět: Storm and singletons
"
Hi all,
This may be an old question, but what is the correct way of using Singleton
pattern with Storm?
Currently we have a topology with around 200 bolts, of which around 50 read
and write data from/to Cassandra. Each of these 50 bolts creates a new
Cassandra session, which to my understanding of Cassandra is not great. We
have tried solving this issue with several implementations of singletons:
lazy static, eager static and enum. All of these implementations have worked
in dev environment running on single server, but have fallen apart on QA
environment which has two supervisor hosts. I'm new to Storm so I have no
idea if we are missing something crucial or obvious.
Thanks & Regards,
Denis Sevosteenko
"