You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@storm.apache.org by jimmy tekli <ji...@hotmail.com> on 2016/07/13 08:22:17 UTC
Storm Shared Memory
Hello ,I developed a java project based on the storm topology that process incoming tuples in a certain way.I tested it locally and it worked perfectly in the context of streaming tuples of course. My topology is formed of one spout and three bolts.In my code I used Data Structures (such as HashMaps,linkedHashmap and TreeMap) to store some information concerning the tuple's processing and I declared them static in the topology builder class so they could be shared by all the bolts because they need to accessed and varied by them frequently. The problem is that when i deployed it over a remote cluster these static attributes weren't visible (shared ) by the different bolts (I read about it online and I concluded because of multiple JVMs). My question is if there is a certain concept concerning "shared memory" in Apache Storm where we can store these DataStructures for them to be shared and accessed by all the bolts at any given time.thanks in advance for your help.
RE: Storm Shared Memory
Posted by jimmy tekli <ji...@hotmail.com>.
hey guys , okay i will check it! thanks for your reply!
From: data.xinwang@gmail.com
Date: Thu, 14 Jul 2016 10:08:07 +0800
Subject: Re: Storm Shared Memory
To: user@storm.apache.org
Now Storm no concept like 'shared memory' you should use external resources. As a reference, you can take a look at `storm-redis` https://github.com/apache/storm/tree/master/external/storm-redis.
2016-07-13 21:21 GMT+08:00 Stephen Powis <sp...@salesforce.com>:
You'd need to use some external data store (such as redis) to maintain state that exists across multiple JVMs
On Wed, Jul 13, 2016 at 4:22 AM, jimmy tekli <ji...@hotmail.com> wrote:
Hello ,I developed a java project based on the storm topology that process incoming tuples in a certain way.I tested it locally and it worked perfectly in the context of streaming tuples of course. My topology is formed of one spout and three bolts.In my code I used Data Structures (such as HashMaps,linkedHashmap and TreeMap) to store some information concerning the tuple's processing and I declared them static in the topology builder class so they could be shared by all the bolts because they need to accessed and varied by them frequently. The problem is that when i deployed it over a remote cluster these static attributes weren't visible (shared ) by the different bolts (I read about it online and I concluded because of multiple JVMs). My question is if there is a certain concept concerning "shared memory" in Apache Storm where we can store these DataStructures for them to be shared and accessed by all the bolts at any given time.thanks in advance for your help.
Re: Storm Shared Memory
Posted by Xin Wang <da...@gmail.com>.
Now Storm no concept like 'shared memory' you should use external
resources. As a reference, you can take a look at `storm-redis`
https://github.com/apache/storm/tree/master/external/storm-redis.
2016-07-13 21:21 GMT+08:00 Stephen Powis <sp...@salesforce.com>:
> You'd need to use some external data store (such as redis) to maintain
> state that exists across multiple JVMs
>
> On Wed, Jul 13, 2016 at 4:22 AM, jimmy tekli <ji...@hotmail.com>
> wrote:
>
>> Hello ,
>> I developed a java project based on the storm topology that process
>> incoming tuples in a certain way.I tested it locally and it worked
>> perfectly in the context of streaming tuples of course. My topology is
>> formed of one spout and three bolts.In my code I used Data Structures
>> (such as HashMaps,linkedHashmap and TreeMap) to store some information
>> concerning the tuple's processing and I declared them static in the
>> topology builder class so they could be shared by all the bolts because
>> they need to accessed and varied by them frequently. The problem is that
>> when i deployed it over a remote cluster these static attributes weren't
>> visible (shared ) by the different bolts (I read about it online and I
>> concluded because of multiple JVMs). My question is if there is a certain
>> concept concerning "shared memory" in Apache Storm where we can store these
>> DataStructures for them to be shared and accessed by all the bolts at any
>> given time.
>>
>> thanks in advance for your help.
>>
>
>
Re: Storm Shared Memory
Posted by Stephen Powis <sp...@salesforce.com>.
You'd need to use some external data store (such as redis) to maintain
state that exists across multiple JVMs
On Wed, Jul 13, 2016 at 4:22 AM, jimmy tekli <ji...@hotmail.com> wrote:
> Hello ,
> I developed a java project based on the storm topology that process
> incoming tuples in a certain way.I tested it locally and it worked
> perfectly in the context of streaming tuples of course. My topology is
> formed of one spout and three bolts.In my code I used Data Structures
> (such as HashMaps,linkedHashmap and TreeMap) to store some information
> concerning the tuple's processing and I declared them static in the
> topology builder class so they could be shared by all the bolts because
> they need to accessed and varied by them frequently. The problem is that
> when i deployed it over a remote cluster these static attributes weren't
> visible (shared ) by the different bolts (I read about it online and I
> concluded because of multiple JVMs). My question is if there is a certain
> concept concerning "shared memory" in Apache Storm where we can store these
> DataStructures for them to be shared and accessed by all the bolts at any
> given time.
>
> thanks in advance for your help.
>