Updating Changes after the cluster is submitted

[Ritesh Sinha <ku...@gmail.com>]

I need to write a bolt which processes the messages on the basis of IP.
Basically , i have a list of IP's stored in mongoDB.All the messages i get
will contain IP's, So , i need to check if that particular IP is present in
the list or not if it is present then i need to ignore the message else
process it.

The list of IP is dynamic .There can be addition or deletion of IPs after
the topology has been deployed to storm.

The problem is either I need to keep the IP is memory(but the changes can't
be reflected once the list is updated) or read it every time from mongoDB,
which makes the processing slow.

Is there any way to implement it efficiently ?
 Is it possible to trigger any event if addition or deletion of IP has
happened in mongoDB ?

Re: Updating Changes after the cluster is submitted

[Bobby Evans <ev...@yahoo-inc.com>]

The question is how consistent do you need this IP address filter to be?  If you need it to be 100% consistent, then you need to do it the slow way, but because you are using MongoDB, I assume that you can handle some inconsistency. The simplest way would be to use the keyed grouping and use a cache that has a timed expiry that is slightly less than the inconsistency that you can tolerate.  I am not a mongodb expert so I don't know if it can publish edits to an external pub/sub system, or send edit events back to clients.  But that would just be an efficiency improvement over the cache. 

Alternatively it looks like MongoDB does have an async API, so you could do the lookups asynchronously, if the return order of the events does not matter.  That would not change the overall latency of processing the events nor the load on MongoDB, but it would increase the throughput of your bolt/topology.
 - Bobby 


     On Thursday, August 6, 2015 3:28 AM, Ritesh Sinha <ku...@gmail.com> wrote:
   

 I need to write a bolt which processes the messages on the basis of IP. Basically , i have a list of IP's stored in mongoDB.All the messages i get will contain IP's, So , i need to check if that particular IP is present in the list or not if it is present then i need to ignore the message else process it.

The list of IP is dynamic .There can be addition or deletion of IPs after the topology has been deployed to storm.

The problem is either I need to keep the IP is memory(but the changes can't be reflected once the list is updated) or read it every time from mongoDB, which makes the processing slow.

Is there any way to implement it efficiently ?
 Is it possible to trigger any event if addition or deletion of IP has happened in mongoDB ?

  

Re: Updating Changes after the cluster is submitted

[Kishore Senji <ks...@gmail.com>]

Just to add few more to the list -

ZooKeeper based solution (see Storm-Signals project for inspiration)
tick tuples to do periodic refresh

On Thu, Aug 6, 2015 at 1:27 AM, Ritesh Sinha <
kumarriteshranjansinha@gmail.com> wrote:

> I need to write a bolt which processes the messages on the basis of IP.
> Basically , i have a list of IP's stored in mongoDB.All the messages i get
> will contain IP's, So , i need to check if that particular IP is present in
> the list or not if it is present then i need to ignore the message else
> process it.
>
> The list of IP is dynamic .There can be addition or deletion of IPs after
> the topology has been deployed to storm.
>
> The problem is either I need to keep the IP is memory(but the changes
> can't be reflected once the list is updated) or read it every time from
> mongoDB, which makes the processing slow.
>
> Is there any way to implement it efficiently ?
>  Is it possible to trigger any event if addition or deletion of IP has
> happened in mongoDB ?
>