You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by Ajay <aj...@gmail.com> on 2014/12/23 09:03:50 UTC
RDD for Storm Streaming in Spark
Hi,
Can we use Storm Streaming as RDD in Spark? Or any way to get Spark work
with Storm?
Thanks
Ajay
Re: RDD for Storm Streaming in Spark
Posted by Ajay <aj...@gmail.com>.
Right. I contacted the SummingBird users as well. It doesn't support Spark
streaming currently.
We are heading towards Storm as it is mostly widely used. Is Spark
streaming production ready?
Thanks
Ajay
On Tue, Dec 23, 2014 at 3:47 PM, Gerard Maas <ge...@gmail.com> wrote:
> I'm not aware of a project trying to achieve this integration. At some
> point Summingbird had the intention of adding an Spark port, and that could
> potentially bridge Storm and Spark. Not sure if that evolved into something
> concrete.
>
> In any case, an attempt to bring Storm and Spark together will consist in
> Storm collecting the data to construct an RDD and provide that to Spark for
> processing, basically what Spark Streaming is doing.
>
> -kr, Gerard.
>
>
>
> On Tue, Dec 23, 2014 at 9:49 AM, Ajay <aj...@gmail.com> wrote:
>
>> Hi,
>>
>> The question is to do streaming in Spark with Storm (not using Spark
>> Streaming).
>>
>> The idea is to use Spark as a in-memory computation engine and static
>> data coming from Cassandra/Hbase and streaming data from Storm.
>>
>> Thanks
>> Ajay
>>
>>
>> On Tue, Dec 23, 2014 at 2:03 PM, Gerard Maas <ge...@gmail.com>
>> wrote:
>>
>>> Hi,
>>>
>>> I'm not sure what you are asking:
>>>
>>> Whether we can use spouts and bolts in Spark (=> no)
>>>
>>> or whether we can do streaming in Spark:
>>> http://spark.apache.org/docs/latest/streaming-programming-guide.html
>>>
>>> -kr, Gerard.
>>>
>>>
>>> On Tue, Dec 23, 2014 at 9:03 AM, Ajay <aj...@gmail.com> wrote:
>>>
>>>> Hi,
>>>>
>>>> Can we use Storm Streaming as RDD in Spark? Or any way to get Spark
>>>> work with Storm?
>>>>
>>>> Thanks
>>>> Ajay
>>>>
>>>
>>>
>>
>
Re: RDD for Storm Streaming in Spark
Posted by Gerard Maas <ge...@gmail.com>.
I'm not aware of a project trying to achieve this integration. At some
point Summingbird had the intention of adding an Spark port, and that could
potentially bridge Storm and Spark. Not sure if that evolved into something
concrete.
In any case, an attempt to bring Storm and Spark together will consist in
Storm collecting the data to construct an RDD and provide that to Spark for
processing, basically what Spark Streaming is doing.
-kr, Gerard.
On Tue, Dec 23, 2014 at 9:49 AM, Ajay <aj...@gmail.com> wrote:
> Hi,
>
> The question is to do streaming in Spark with Storm (not using Spark
> Streaming).
>
> The idea is to use Spark as a in-memory computation engine and static data
> coming from Cassandra/Hbase and streaming data from Storm.
>
> Thanks
> Ajay
>
>
> On Tue, Dec 23, 2014 at 2:03 PM, Gerard Maas <ge...@gmail.com>
> wrote:
>
>> Hi,
>>
>> I'm not sure what you are asking:
>>
>> Whether we can use spouts and bolts in Spark (=> no)
>>
>> or whether we can do streaming in Spark:
>> http://spark.apache.org/docs/latest/streaming-programming-guide.html
>>
>> -kr, Gerard.
>>
>>
>> On Tue, Dec 23, 2014 at 9:03 AM, Ajay <aj...@gmail.com> wrote:
>>
>>> Hi,
>>>
>>> Can we use Storm Streaming as RDD in Spark? Or any way to get Spark work
>>> with Storm?
>>>
>>> Thanks
>>> Ajay
>>>
>>
>>
>
Re: RDD for Storm Streaming in Spark
Posted by Ajay <aj...@gmail.com>.
Hi,
The question is to do streaming in Spark with Storm (not using Spark
Streaming).
The idea is to use Spark as a in-memory computation engine and static data
coming from Cassandra/Hbase and streaming data from Storm.
Thanks
Ajay
On Tue, Dec 23, 2014 at 2:03 PM, Gerard Maas <ge...@gmail.com> wrote:
> Hi,
>
> I'm not sure what you are asking:
>
> Whether we can use spouts and bolts in Spark (=> no)
>
> or whether we can do streaming in Spark:
> http://spark.apache.org/docs/latest/streaming-programming-guide.html
>
> -kr, Gerard.
>
>
> On Tue, Dec 23, 2014 at 9:03 AM, Ajay <aj...@gmail.com> wrote:
>
>> Hi,
>>
>> Can we use Storm Streaming as RDD in Spark? Or any way to get Spark work
>> with Storm?
>>
>> Thanks
>> Ajay
>>
>
>
Re: RDD for Storm Streaming in Spark
Posted by Gerard Maas <ge...@gmail.com>.
Hi,
I'm not sure what you are asking:
Whether we can use spouts and bolts in Spark (=> no)
or whether we can do streaming in Spark:
http://spark.apache.org/docs/latest/streaming-programming-guide.html
-kr, Gerard.
On Tue, Dec 23, 2014 at 9:03 AM, Ajay <aj...@gmail.com> wrote:
> Hi,
>
> Can we use Storm Streaming as RDD in Spark? Or any way to get Spark work
> with Storm?
>
> Thanks
> Ajay
>