Any patterns for multiplexing the streaming data

[bdev <bu...@gmail.com>]

We are looking at consuming the kafka stream using Spark Streaming and
transform into various subsets like applying some transformation or
de-normalizing some fields, etc. and feed it back into Kafka as a different
topic for downstream consumers.

Wanted to know if there are any existing patterns for achieving this.

Thanks!



--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Any-patterns-for-multiplexing-the-streaming-data-tp18303.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.

---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org

Re: Any patterns for multiplexing the streaming data

[Anand Iyer <ai...@cloudera.com>]

Hi TD,

This is a common pattern that is emerging today. Kafka --> SS --> Kafka.

Spark Streaming comes with a built in consumer to read from Kafka. It will
be great to have an easy way for users to write back to Kafka without
having to code a customer producer using the Kafka Producert APIs.

Are there any plans to commit the code in the above github repo? If so, do
you have a rough estimate of when.

Thanks,

Anand

On Fri, Nov 7, 2014 at 1:25 PM, Tathagata Das <ta...@gmail.com>
wrote:

> I am not aware of any obvious existing pattern that does exactly this.
> Generally this sort of computation (subset, denormalization) things are so
> generic sounding terms but actually have very specific requirements that it
> hard to refer to a design pattern without more requirement info.
>
> If you want to feed back to kafka, you can take a look at this pull request
>
> https://github.com/apache/spark/pull/2994
>
> On Thu, Nov 6, 2014 at 4:15 PM, bdev <bu...@gmail.com> wrote:
>
>> We are looking at consuming the kafka stream using Spark Streaming and
>> transform into various subsets like applying some transformation or
>> de-normalizing some fields, etc. and feed it back into Kafka as a
>> different
>> topic for downstream consumers.
>>
>> Wanted to know if there are any existing patterns for achieving this.
>>
>> Thanks!
>>
>>
>>
>> --
>> View this message in context:
>> http://apache-spark-user-list.1001560.n3.nabble.com/Any-patterns-for-multiplexing-the-streaming-data-tp18303.html
>> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
>> For additional commands, e-mail: user-help@spark.apache.org
>>
>>
>

Re: Any patterns for multiplexing the streaming data

[Tathagata Das <ta...@gmail.com>]

I am not aware of any obvious existing pattern that does exactly this.
Generally this sort of computation (subset, denormalization) things are so
generic sounding terms but actually have very specific requirements that it
hard to refer to a design pattern without more requirement info.

If you want to feed back to kafka, you can take a look at this pull request

https://github.com/apache/spark/pull/2994

On Thu, Nov 6, 2014 at 4:15 PM, bdev <bu...@gmail.com> wrote:

> We are looking at consuming the kafka stream using Spark Streaming and
> transform into various subsets like applying some transformation or
> de-normalizing some fields, etc. and feed it back into Kafka as a different
> topic for downstream consumers.
>
> Wanted to know if there are any existing patterns for achieving this.
>
> Thanks!
>
>
>
> --
> View this message in context:
> http://apache-spark-user-list.1001560.n3.nabble.com/Any-patterns-for-multiplexing-the-streaming-data-tp18303.html
> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
> For additional commands, e-mail: user-help@spark.apache.org
>
>