You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by Pulkit Manchanda <pu...@gmail.com> on 2018/07/11 13:53:29 UTC

Http streaming using Kafka+Akka

Hi All,

I want to build a datapipeline with the following design. Can please anyone
advice me that is it feasible to do? Or there are better options.


HTTP Streams --> (HTTP stream consumer)(using AKKA HTTP Streaming) --> (kafka
Stream Producer)(using Kafka Streaming) --> (Kafka Stream Consumer)(using Spark
structured streaming)


DPContext --> (Kafka Stream Producer)(using Spark structured
streaming) --> (Kafka
Stream Consumer)(using AKKA Streaming) -->(HTTP stream Producer)(using AKKA
HTTP Streaming) --> HTTP Streams

Thanks
Pulkit

Re: Kafka stream or ksql design question

Posted by Guozhang Wang <wa...@gmail.com>.
Hello Will,

Your question is very high-level and hence I felt less guilty to give you a
general answer :)

You can read the web docs on on Kafka Streams achieve high throughput via
data parallelism here:

https://kafka.apache.org/11/documentation/streams/architecture


Regarding KSQL, you can look at the github issues for asked features and
tentative roadmap tasks:

https://github.com/confluentinc/ksql/issues


Guozhang


On Mon, Jul 16, 2018 at 5:06 AM, Will Du <wi...@gmail.com> wrote:

> Hi folks,
> As far as I know, Kafka Stream is a separate process by reading data from
> topic, transform, and writing to another topic if needed. In this case, how
> this process supports high throughout stream as well as load balance in
> terms of message traffic and computing resource for stream processing?
>
> Regarding to KSL, is there any query optimization in place or in roadmap?
>
> Thanks,
> Will
>
>


-- 
-- Guozhang

Kafka stream or ksql design question

Posted by Will Du <wi...@gmail.com>.
Hi folks,
As far as I know, Kafka Stream is a separate process by reading data from topic, transform, and writing to another topic if needed. In this case, how this process supports high throughout stream as well as load balance in terms of message traffic and computing resource for stream processing?

Regarding to KSL, is there any query optimization in place or in roadmap?

Thanks,
Will


Re: Http streaming using Kafka+Akka

Posted by Boris Lublinsky <bo...@lightbend.com>.
Yes, reactive Kafka

Boris Lublinsky
FDP Architect
boris.lublinsky@lightbend.com
https://www.lightbend.com/

> On Jul 11, 2018, at 9:46 AM, Pulkit Manchanda <pu...@gmail.com> wrote:
> 
> Thanks Boris.
> Are you using Alpakka for kafka -Akka integration?
> 
> On Wed, Jul 11, 2018 at 9:56 AM, Boris Lublinsky <
> boris.lublinsky@lightbend.com> wrote:
> 
>> This works fine, we (Lightbend) are using this approach all over the place
>> 
>> Boris Lublinsky
>> FDP Architect
>> boris.lublinsky@lightbend.com
>> https://www.lightbend.com/
>> 
>>> On Jul 11, 2018, at 8:53 AM, Pulkit Manchanda <pu...@gmail.com>
>> wrote:
>>> 
>>> Hi All,
>>> 
>>> I want to build a datapipeline with the following design. Can please
>> anyone
>>> advice me that is it feasible to do? Or there are better options.
>>> 
>>> 
>>> HTTP Streams --> (HTTP stream consumer)(using AKKA HTTP Streaming) -->
>> (kafka
>>> Stream Producer)(using Kafka Streaming) --> (Kafka Stream
>> Consumer)(using Spark
>>> structured streaming)
>>> 
>>> 
>>> DPContext --> (Kafka Stream Producer)(using Spark structured
>>> streaming) --> (Kafka
>>> Stream Consumer)(using AKKA Streaming) -->(HTTP stream Producer)(using
>> AKKA
>>> HTTP Streaming) --> HTTP Streams
>>> 
>>> Thanks
>>> Pulkit
>> 
>> 


Re: Http streaming using Kafka+Akka

Posted by Pulkit Manchanda <pu...@gmail.com>.
Thanks Boris.
Are you using Alpakka for kafka -Akka integration?

On Wed, Jul 11, 2018 at 9:56 AM, Boris Lublinsky <
boris.lublinsky@lightbend.com> wrote:

> This works fine, we (Lightbend) are using this approach all over the place
>
> Boris Lublinsky
> FDP Architect
> boris.lublinsky@lightbend.com
> https://www.lightbend.com/
>
> > On Jul 11, 2018, at 8:53 AM, Pulkit Manchanda <pu...@gmail.com>
> wrote:
> >
> > Hi All,
> >
> > I want to build a datapipeline with the following design. Can please
> anyone
> > advice me that is it feasible to do? Or there are better options.
> >
> >
> > HTTP Streams --> (HTTP stream consumer)(using AKKA HTTP Streaming) -->
> (kafka
> > Stream Producer)(using Kafka Streaming) --> (Kafka Stream
> Consumer)(using Spark
> > structured streaming)
> >
> >
> > DPContext --> (Kafka Stream Producer)(using Spark structured
> > streaming) --> (Kafka
> > Stream Consumer)(using AKKA Streaming) -->(HTTP stream Producer)(using
> AKKA
> > HTTP Streaming) --> HTTP Streams
> >
> > Thanks
> > Pulkit
>
>

Re: Http streaming using Kafka+Akka

Posted by Boris Lublinsky <bo...@lightbend.com>.
This works fine, we (Lightbend) are using this approach all over the place

Boris Lublinsky
FDP Architect
boris.lublinsky@lightbend.com
https://www.lightbend.com/

> On Jul 11, 2018, at 8:53 AM, Pulkit Manchanda <pu...@gmail.com> wrote:
> 
> Hi All,
> 
> I want to build a datapipeline with the following design. Can please anyone
> advice me that is it feasible to do? Or there are better options.
> 
> 
> HTTP Streams --> (HTTP stream consumer)(using AKKA HTTP Streaming) --> (kafka
> Stream Producer)(using Kafka Streaming) --> (Kafka Stream Consumer)(using Spark
> structured streaming)
> 
> 
> DPContext --> (Kafka Stream Producer)(using Spark structured
> streaming) --> (Kafka
> Stream Consumer)(using AKKA Streaming) -->(HTTP stream Producer)(using AKKA
> HTTP Streaming) --> HTTP Streams
> 
> Thanks
> Pulkit