Evaluating Apache Storm

[Ankur Garg <an...@gmail.com>]

Hi ,

We in our organization are building a system to send  real time
notification (Pub-Sub) to Mobile Users(Android and IOS) .

Broadly the use case is that  mobile users subscribe to some topics and
whenever there is/are events related to this topic , we need to send those
users a Notification for that message .

The estimate is we are going to have a large no of users over a period of
time (may be millions) to get these notifications .

Since Storm is used for real -time processing of data, I am planning to
explore this whether it fits our requirement .

Does storm fits the bill here ? Is it being used somewhere to do this ?

Looking for some pointers .

Thanks
Ankur

Re: Evaluating Apache Storm

[Ankur Garg <an...@gmail.com>]

Thanks Deepak for the reply .

Yes , here as well Storm will be acting as a subscriber for Incoming Event
and we plan to use this for aggregation (Map and Reduce) in real time. The
volume of events can be huge here .

 I can u a rough idea (sorry if i may sound opaque at some points as we are
just trying to connect dots at this moment).

So , the idea here is , we are trying to build a classic Pub/Sub model
 that we will have a message Bus (Kafka or RabbitMq) where we will get some
events .

To elaborate an event  , there will be various Topics (which will be some
String Value) and we will get events for topics which will nothing but some
metadata for that evebt . Now , we will Figure out  what all mobile users
have subscribed to this particular event and append it to event msg .

To give a very rough example (since we are also not sure what will an
actual event look like as at this point we are just finding ways to
define/solve this problem)  :

Say There is Topic say TomCruise and we get an event in json (containing
metadata for his upcoming movie release)  . This event , we will receive in
our Message Bus (say kafka or rabbit Mq)

Event -> {TomCruise , {EventBody} } .

We plan to  feed  this stream to Apache Storm Spout where in it calls 2
bolts (one for Map and One for Reduce)

In the Bolt1 ,we may be doing a look up as to which all mobile users have
subscribed to this Topic's Event  and their type  and then Change the above
to

{ {Event1},{MobileUser1} ,{iOS} } .
{ {Event2},{MobileUser2} ,{GCM} } .
{ {Event1} , {MobileUser3},{iOs}}
{ {Event2} , {MobileUser4}.{GCM}}

..and so on
iOs,GCM are delivery channels (GCM stands for Google Cloud Messaging) so
that we can push to Android devices.

And in Bolt2 , we do aggregation that which Event needs to be delivered to
which set of users

{
Messages {
    message { 'iOS', 'Event1', listOfMobileUsers {'MobileUser1',
'MobileUser3.....'} },
    message { 'GCM', 'Even2', listOfMobileUsers {'MobileUser2',
'MobileUser4.....'} },
 and So on
}

>From here we will again push this to a notification Message Bus and write
plugins which will be responsible for delivering them to end user .

Now,since we will be having potentially millions of subscribers . We want
to build a platform which can handle this scale reliably . Hence , thinking
of evaluating Storm for this .

Hope I could describe the use case here .

Thanks
Ankur



On Mon, Aug 31, 2015 at 11:04 PM, Deepak Sharma <de...@gmail.com>
wrote:

> Hi Ankur
> I have not seen this kind of setup earlier.But i would say that storm is
> basically the subscriber in majority of implementations that i have seen or
> worked upon.
> So here you can use storm to process the events/notifications and then
> push it from storm to different topics where the mobile users are
> subscribed to.
>
> HTH
> --Deepak
>
> On Mon, Aug 31, 2015 at 10:59 PM, Ankur Garg <an...@gmail.com> wrote:
>
>> Hi ,
>>
>> We in our organization are building a system to send  real time
>> notification (Pub-Sub) to Mobile Users(Android and IOS) .
>>
>> Broadly the use case is that  mobile users subscribe to some topics and
>> whenever there is/are events related to this topic , we need to send those
>> users a Notification for that message .
>>
>> The estimate is we are going to have a large no of users over a period of
>> time (may be millions) to get these notifications .
>>
>> Since Storm is used for real -time processing of data, I am planning to
>> explore this whether it fits our requirement .
>>
>> Does storm fits the bill here ? Is it being used somewhere to do this ?
>>
>> Looking for some pointers .
>>
>> Thanks
>> Ankur
>>
>
>
>
> --
> Thanks
> Deepak
> www.bigdatabig.com
> www.keosha.net
>

Re: Evaluating Apache Storm

[Deepak Sharma <de...@gmail.com>]

Hi Ankur
I have not seen this kind of setup earlier.But i would say that storm is
basically the subscriber in majority of implementations that i have seen or
worked upon.
So here you can use storm to process the events/notifications and then push
it from storm to different topics where the mobile users are subscribed to.

HTH
--Deepak

On Mon, Aug 31, 2015 at 10:59 PM, Ankur Garg <an...@gmail.com> wrote:

> Hi ,
>
> We in our organization are building a system to send  real time
> notification (Pub-Sub) to Mobile Users(Android and IOS) .
>
> Broadly the use case is that  mobile users subscribe to some topics and
> whenever there is/are events related to this topic , we need to send those
> users a Notification for that message .
>
> The estimate is we are going to have a large no of users over a period of
> time (may be millions) to get these notifications .
>
> Since Storm is used for real -time processing of data, I am planning to
> explore this whether it fits our requirement .
>
> Does storm fits the bill here ? Is it being used somewhere to do this ?
>
> Looking for some pointers .
>
> Thanks
> Ankur
>



-- 
Thanks
Deepak
www.bigdatabig.com
www.keosha.net