You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@storm.apache.org by Navin Ipe <na...@searchlighthealth.com> on 2016/04/26 11:59:18 UTC
Emit from a Java application and receive in a Bolt of another Java application?
Hi,
A colleague created a Bolt that writes data to a MongoDB application.
I have a Spout that reads that data from MongoDB.
My colleague's Bolt is in a separate Storm application he built. Mine is on
a separate storm application I built. Our applications may run on different
VM's.
My team lead wants my colleague's Bolt to emit data, and wants a Bolt in my
application to receive that data. So we basically avoid writing to MongoDB.
I suggested doing this via Java sockets, RMI or Redis. On hearing this, my
team lead tells me "You haven't understood Storm yet".
I've been through a lot of the documentation on Storm, and I haven't seen
any case where Storm can communicate. Except maybe using DRPC
<http://storm.apache.org/releases/0.10.0/Distributed-RPC.html>:
http://stackoverflow.com/questions/15690691/communication-between-several-storm-topologies
But there
But there are complaints of DRPC memory leaks and unexpected behaviour:
https://mithunsatheesh.wordpress.com/2014/01/04/storm-drpc-and-why-it-didnt-solve-the-case-for-us/
So is DRPC the way to go or does Storm have some other method of emitting
from one topology and receiving it in another topology?
--
Regards,
Navin
Re: Emit from a Java application and receive in a Bolt of another
Java application?
Posted by Navin Ipe <na...@searchlighthealth.com>.
Can't be DRPC. Kafka is another option I saw:
http://stackoverflow.com/questions/27936946/storm-topology-to-topology
On Tue, Apr 26, 2016 at 3:29 PM, Navin Ipe <na...@searchlighthealth.com>
wrote:
> Hi,
>
> A colleague created a Bolt that writes data to a MongoDB application.
> I have a Spout that reads that data from MongoDB.
> My colleague's Bolt is in a separate Storm application he built. Mine is
> on a separate storm application I built. Our applications may run on
> different VM's.
>
> My team lead wants my colleague's Bolt to emit data, and wants a Bolt in
> my application to receive that data. So we basically avoid writing to
> MongoDB.
>
> I suggested doing this via Java sockets, RMI or Redis. On hearing this, my
> team lead tells me "You haven't understood Storm yet".
>
> I've been through a lot of the documentation on Storm, and I haven't seen
> any case where Storm can communicate. Except maybe using DRPC
> <http://storm.apache.org/releases/0.10.0/Distributed-RPC.html>:
> http://stackoverflow.com/questions/15690691/communication-between-several-storm-topologies
> But there
> But there are complaints of DRPC memory leaks and unexpected behaviour:
> https://mithunsatheesh.wordpress.com/2014/01/04/storm-drpc-and-why-it-didnt-solve-the-case-for-us/
>
> So is DRPC the way to go or does Storm have some other method of emitting
> from one topology and receiving it in another topology?
>
> --
> Regards,
> Navin
>
--
Regards,
Navin
Re: Emit from a Java application and receive in a Bolt of another
Java application?
Posted by Navin Ipe <na...@searchlighthealth.com>.
Thanks Cody and Nathan. Kafka it is.
(ps: I know sockets aren't a good option because of not knowing which port
of which node to listen to and having to implement one's own queue etc..)
On Tue, Apr 26, 2016 at 4:06 PM, Nathan Leung <nc...@gmail.com> wrote:
> Kafka is probably better than redis, and definitely better than sockets.
> Other queues like rabbitmq can work too.
>
> Sockets are a terrible choice, but I will leave why as a mental exercise
> for now :). If you must know I can respond again.
> On Apr 26, 2016 6:27 AM, "Cody Lee" <co...@wellaware.us> wrote:
>
> 2 options come to mind without knowing what your code does: 1. join the
> topology code bases for one topology 2. use a distributed queue (if
> something else needs to use this data )
>
>
>
>
> -------- Original message --------
> From: Navin Ipe <na...@searchlighthealth.com>
> Date: 04/26/2016 4:59 AM (GMT-06:00)
> To: user@storm.apache.org
> Subject: Emit from a Java application and receive in a Bolt of another
> Java application?
>
> Hi,
>
> A colleague created a Bolt that writes data to a MongoDB application.
> I have a Spout that reads that data from MongoDB.
> My colleague's Bolt is in a separate Storm application he built. Mine is
> on a separate storm application I built. Our applications may run on
> different VM's.
>
> My team lead wants my colleague's Bolt to emit data, and wants a Bolt in
> my application to receive that data. So we basically avoid writing to
> MongoDB.
>
> I suggested doing this via Java sockets, RMI or Redis. On hearing this, my
> team lead tells me "You haven't understood Storm yet".
>
> I've been through a lot of the documentation on Storm, and I haven't seen
> any case where Storm can communicate. Except maybe using DRPC
> <http://storm.apache.org/releases/0.10.0/Distributed-RPC.html>:
> http://stackoverflow.com/questions/15690691/communication-between-several-storm-topologies
> But there
> But there are complaints of DRPC memory leaks and unexpected behaviour:
> https://mithunsatheesh.wordpress.com/2014/01/04/storm-drpc-and-why-it-didnt-solve-the-case-for-us/
>
> So is DRPC the way to go or does Storm have some other method of emitting
> from one topology and receiving it in another topology?
>
> --
> Regards,
> Navin
>
>
--
Regards,
Navin
RE: Emit from a Java application and receive in a Bolt of another
Java application?
Posted by Nathan Leung <nc...@gmail.com>.
Kafka is probably better than redis, and definitely better than sockets.
Other queues like rabbitmq can work too.
Sockets are a terrible choice, but I will leave why as a mental exercise
for now :). If you must know I can respond again.
On Apr 26, 2016 6:27 AM, "Cody Lee" <co...@wellaware.us> wrote:
2 options come to mind without knowing what your code does: 1. join the
topology code bases for one topology 2. use a distributed queue (if
something else needs to use this data )
-------- Original message --------
From: Navin Ipe <na...@searchlighthealth.com>
Date: 04/26/2016 4:59 AM (GMT-06:00)
To: user@storm.apache.org
Subject: Emit from a Java application and receive in a Bolt of another Java
application?
Hi,
A colleague created a Bolt that writes data to a MongoDB application.
I have a Spout that reads that data from MongoDB.
My colleague's Bolt is in a separate Storm application he built. Mine is on
a separate storm application I built. Our applications may run on different
VM's.
My team lead wants my colleague's Bolt to emit data, and wants a Bolt in my
application to receive that data. So we basically avoid writing to MongoDB.
I suggested doing this via Java sockets, RMI or Redis. On hearing this, my
team lead tells me "You haven't understood Storm yet".
I've been through a lot of the documentation on Storm, and I haven't seen
any case where Storm can communicate. Except maybe using DRPC
<http://storm.apache.org/releases/0.10.0/Distributed-RPC.html>:
http://stackoverflow.com/questions/15690691/communication-between-several-storm-topologies
But there
But there are complaints of DRPC memory leaks and unexpected behaviour:
https://mithunsatheesh.wordpress.com/2014/01/04/storm-drpc-and-why-it-didnt-solve-the-case-for-us/
So is DRPC the way to go or does Storm have some other method of emitting
from one topology and receiving it in another topology?
--
Regards,
Navin
RE: Emit from a Java application and receive in a Bolt of another
Java application?
Posted by Cody Lee <co...@wellaware.us>.
2 options come to mind without knowing what your code does: 1. join the topology code bases for one topology 2. use a distributed queue (if something else needs to use this data )
-------- Original message --------
From: Navin Ipe <na...@searchlighthealth.com>
Date: 04/26/2016 4:59 AM (GMT-06:00)
To: user@storm.apache.org
Subject: Emit from a Java application and receive in a Bolt of another Java application?
Hi,
A colleague created a Bolt that writes data to a MongoDB application.
I have a Spout that reads that data from MongoDB.
My colleague's Bolt is in a separate Storm application he built. Mine is on a separate storm application I built. Our applications may run on different VM's.
My team lead wants my colleague's Bolt to emit data, and wants a Bolt in my application to receive that data. So we basically avoid writing to MongoDB.
I suggested doing this via Java sockets, RMI or Redis. On hearing this, my team lead tells me "You haven't understood Storm yet".
I've been through a lot of the documentation on Storm, and I haven't seen any case where Storm can communicate. Except maybe using DRPC<http://storm.apache.org/releases/0.10.0/Distributed-RPC.html>: http://stackoverflow.com/questions/15690691/communication-between-several-storm-topologies
But there
But there are complaints of DRPC memory leaks and unexpected behaviour: https://mithunsatheesh.wordpress.com/2014/01/04/storm-drpc-and-why-it-didnt-solve-the-case-for-us/
So is DRPC the way to go or does Storm have some other method of emitting from one topology and receiving it in another topology?
--
Regards,
Navin