You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@storm.apache.org by Dima Dragan <di...@belleron.net> on 2015/05/21 18:02:12 UTC
Re: sharing data between tasks in the same bolt or spout
Hi,
for 1 worker you can create a class with your static thread-safe collection
and thread-safe initialization method (Double Checked Locking & volatile,
for example) which return collection reference . In bolt`s prepare() method
run init(). In this way all bolts will have the same ref to collection.
for case with more than 1 worker, previous option will not work, cause each
of worker has it`s own JVM process. You will have to persist your
collection in external database and implement static access to connection
pool. Each of worker will have it`s own pool.
On Thu, May 21, 2015 at 7:41 PM, applyhhj <ap...@163.com> wrote:
> Hi,
> Is it possible to share data between tasks of the same bolt or spout.
> For example, a string list is set as the shared data and referenced by all
> tasks of a bolt. Thank you very much!!
>
>
> 2015-05-21
> ------------------------------
>
--
Best regards,
Dmytro Dragan
Re: Re: Re: sharing data between tasks in the same bolt or spout
Posted by applyhhj <ap...@163.com>.
Thank you. I was thinking about that but I am not sure if it is fast enough. I will test and then decide to use zookeeper or other in memory database like redis. Thanks again!!
2015-05-21
发件人：Carlos Perelló Marín <ca...@serverdensity.com>
发送时间：2015-05-21 12:27
主题：Re: Re: sharing data between tasks in the same bolt or spout
收件人："user"<us...@storm.apache.org>
抄送：
If the data is not too big, I guess you may use Zookeeper, it's already there and it's supposed to allow exactly the use case you want to cover...
On 21 May 2015 at 19:13, applyhhj <ap...@163.com> wrote:
Very clear answer, I guest I need to find other ways. Thank you very much!!
2015-05-21
发件人：Dima Dragan <di...@belleron.net>
发送时间：2015-05-21 12:02
主题：Re: sharing data between tasks in the same bolt or spout
收件人："user"<us...@storm.apache.org>
抄送：
Hi,
for 1 worker you can create a class with your static thread-safe collection and thread-safe initialization method (Double Checked Locking & volatile, for example) which return collection reference . In bolt`s prepare() method run init(). In this way all bolts will have the same ref to collection.
for case with more than 1 worker, previous option will not work, cause each of worker has it`s own JVM process. You will have to persist your collection in external database and implement static access to connection pool. Each of worker will have it`s own pool.
On Thu, May 21, 2015 at 7:41 PM, applyhhj <ap...@163.com> wrote:
Hi,
Is it possible to share data between tasks of the same bolt or spout. For example, a string list is set as the shared data and referenced by all tasks of a bolt. Thank you very much!!
2015-05-21
--
Best regards,
Dmytro Dragan
--
Carlos Perelló Marín
https://www.serverdensity.com
Re: Re: sharing data between tasks in the same bolt or spout
Posted by Carlos Perelló Marín <ca...@serverdensity.com>.
If the data is not too big, I guess you may use Zookeeper, it's already
there and it's supposed to allow exactly the use case you want to cover...
On 21 May 2015 at 19:13, applyhhj <ap...@163.com> wrote:
> Very clear answer, I guest I need to find other ways. Thank you very
> much!!
>
> 2015-05-21
> ------------------------------
> ------------------------------
>
> *发件人：*Dima Dragan <di...@belleron.net>
> *发送时间：*2015-05-21 12:02
> *主题：*Re: sharing data between tasks in the same bolt or spout
> *收件人：*"user"<us...@storm.apache.org>
> *抄送：*
>
> Hi,
>
> for 1 worker you can create a class with your static thread-safe
> collection and thread-safe initialization method (Double Checked Locking
> & volatile, for example) which return collection reference . In bolt`s
> prepare() method run init(). In this way all bolts will have the same ref
> to collection.
>
> for case with more than 1 worker, previous option will not work, cause
> each of worker has it`s own JVM process. You will have to persist your
> collection in external database and implement static access to connection
> pool. Each of worker will have it`s own pool.
>
> On Thu, May 21, 2015 at 7:41 PM, applyhhj <ap...@163.com> wrote:
>
>> Hi,
>> Is it possible to share data between tasks of the same bolt or spout.
>> For example, a string list is set as the shared data and referenced by all
>> tasks of a bolt. Thank you very much!!
>>
>>
>> 2015-05-21
>> ------------------------------
>>
>
>
>
> --
> Best regards,
> Dmytro Dragan
>
>
>
--
Carlos Perelló Marínhttps://www.serverdensity.com
Re: Re: sharing data between tasks in the same bolt or spout
Posted by applyhhj <ap...@163.com>.
Very clear answer, I guest I need to find other ways. Thank you very much!!
2015-05-21
发件人：Dima Dragan <di...@belleron.net>
发送时间：2015-05-21 12:02
主题：Re: sharing data between tasks in the same bolt or spout
收件人："user"<us...@storm.apache.org>
抄送：
Hi,
for 1 worker you can create a class with your static thread-safe collection and thread-safe initialization method (Double Checked Locking & volatile, for example) which return collection reference . In bolt`s prepare() method run init(). In this way all bolts will have the same ref to collection.
for case with more than 1 worker, previous option will not work, cause each of worker has it`s own JVM process. You will have to persist your collection in external database and implement static access to connection pool. Each of worker will have it`s own pool.
On Thu, May 21, 2015 at 7:41 PM, applyhhj <ap...@163.com> wrote:
Hi,
Is it possible to share data between tasks of the same bolt or spout. For example, a string list is set as the shared data and referenced by all tasks of a bolt. Thank you very much!!
2015-05-21
--
Best regards,
Dmytro Dragan