You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by Benjamin Black <b...@b3k.us> on 2014/03/21 23:58:10 UTC
Spark streaming kafka _output_
Howdy, folks!
Anybody out there having a working kafka _output_ for Spark streaming?
Perhaps one that doesn't involve instantiating a new producer for every
batch?
Thanks!
b
Re: Spark streaming kafka _output_
Posted by Benjamin Black <b...@b3k.us>.
please no.
On Wed, Apr 2, 2014 at 9:47 PM, Tathagata Das
<ta...@gmail.com>wrote:
> If any body is interested is doing this, may I suggested taking a look at
> Twitter's Storehaus. It presents an abstract interface for pushing data to
> many different backends, including Kafka, mongodb, hbase, etc. Integrating
> DStream.foreachRDD with Storehaus maybe a very very useful thing to do.
>
> Someone brave enough to give it a try? ;)
>
> TD
>
>
> On Wed, Apr 2, 2014 at 8:53 PM, Soren Macbeth <so...@yieldbot.com> wrote:
>
>> Anybody? Seems like a reasonable thing to be able to do no?
>>
>>
>> On Fri, Mar 21, 2014 at 3:58 PM, Benjamin Black <b...@b3k.us> wrote:
>>
>>> Howdy, folks!
>>>
>>> Anybody out there having a working kafka _output_ for Spark streaming?
>>> Perhaps one that doesn't involve instantiating a new producer for every
>>> batch?
>>>
>>> Thanks!
>>>
>>> b
>>>
>>
>>
>
Re: Spark streaming kafka _output_
Posted by Tathagata Das <ta...@gmail.com>.
If any body is interested is doing this, may I suggested taking a look at
Twitter's Storehaus. It presents an abstract interface for pushing data to
many different backends, including Kafka, mongodb, hbase, etc. Integrating
DStream.foreachRDD with Storehaus maybe a very very useful thing to do.
Someone brave enough to give it a try? ;)
TD
On Wed, Apr 2, 2014 at 8:53 PM, Soren Macbeth <so...@yieldbot.com> wrote:
> Anybody? Seems like a reasonable thing to be able to do no?
>
>
> On Fri, Mar 21, 2014 at 3:58 PM, Benjamin Black <b...@b3k.us> wrote:
>
>> Howdy, folks!
>>
>> Anybody out there having a working kafka _output_ for Spark streaming?
>> Perhaps one that doesn't involve instantiating a new producer for every
>> batch?
>>
>> Thanks!
>>
>> b
>>
>
>
Re: Spark streaming kafka _output_
Posted by Soren Macbeth <so...@yieldbot.com>.
Anybody? Seems like a reasonable thing to be able to do no?
On Fri, Mar 21, 2014 at 3:58 PM, Benjamin Black <b...@b3k.us> wrote:
> Howdy, folks!
>
> Anybody out there having a working kafka _output_ for Spark streaming?
> Perhaps one that doesn't involve instantiating a new producer for every
> batch?
>
> Thanks!
>
> b
>