Deployment of Kafka Stream app

[Mina Aslani <as...@gmail.com>]

I understand that for dev env creating containers might not be needed, and
as you said "start up an application and go"!
However, I would like to know to have HA in my env and make it scalable;
what is the proper setup I need to have.

- Would every Kafka streaming job/app require a new docker image and
deployment of the container/service? (e.g. 10 containers for 10 instances
of the same app)
- How should I structure things differently if I had more than one (e.g.
different) Kafka streaming apps/jobs?
- What are the advantages of using Kafka streaming over Spark streaming?
I'm asking b/c with Spark streaming I don't need to create and deploy a new
docker image every time I added a new app or changed the same app.

Best regards,
Mina

On Thu, Apr 27, 2017 at 12:03 PM, David Garcia <da...@spiceworks.com>
wrote:

> Unlike spark, you don’t need an entire framework to deploy your job.  With
> Kstreams, you just start up an application and go.  You don’t need docker
> either…although containerizing your stuff is probably a good strategy for
> the purposes of deployment management (something you get with Yarn or a
> spark Cluster)…but you’re not tied to any one framework (e.g. you can use
> kubernetes, mesos, Yarn, or anything else)
>
> On 4/27/17, 10:52 AM, "Mina Aslani" <as...@gmail.com> wrote:
>
>     Hi,
>
>     I created a kafka stream app and as I was informed I created a docker
> image
>     with the app and launched it as a container. However, I have couple of
>     questions:
>
>     - Would every Kafka streaming job require a new docker image and
> deployment
>     of the container/service?
>     - How should I structure things differently if I had more than one
> Kafka
>     streaming app/job?
>     - What are the advantages of using Kafka streaming over Spark
> streaming?
>     I'm asking b/c with Spark streaming I don't need to create and deploy
> a new
>     docker image every time I added or changed an/a app/job.
>
>     Best regards,
>     Mina
>
>
>

Re: Deployment of Kafka Stream app

["Matthias J. Sax" <ma...@confluent.io>]

If you use a containerized environment, if a container fails it will get
restarted -- so you get HA out of the box.

For scaling, you have a single docker image, but start the same image
multiple times. If you have multiple apps, you should have one image per
app and start the number of containers you need using the corresponding
image.

About Spark streaming (this might be a biased answer): it does micro
batching is thus has limited stream processing capabilities -- it also
seems not to be as stable/reliable as Spark core.

Furthermore, with Kafka Streams you can put your whole application into
your image and thus, run it as an holistic piece. With Spark Streaming,
you "stream processing" part would run on Spark, and the rest of you app
would run outside of Spark usually (consuming the result your streaming
job outputs). Thus, you have more moving parts using Spark Streaming.


-Matthias

On 4/27/17 12:20 PM, Mina Aslani wrote:
> I understand that for dev env creating containers might not be needed, and
> as you said "start up an application and go"!
> However, I would like to know to have HA in my env and make it scalable;
> what is the proper setup I need to have.
> 
> - Would every Kafka streaming job/app require a new docker image and
> deployment of the container/service? (e.g. 10 containers for 10 instances
> of the same app)
> - How should I structure things differently if I had more than one (e.g.
> different) Kafka streaming apps/jobs?
> - What are the advantages of using Kafka streaming over Spark streaming?
> I'm asking b/c with Spark streaming I don't need to create and deploy a new
> docker image every time I added a new app or changed the same app.
> 
> Best regards,
> Mina
> 
> On Thu, Apr 27, 2017 at 12:03 PM, David Garcia <da...@spiceworks.com>
> wrote:
> 
>> Unlike spark, you don’t need an entire framework to deploy your job.  With
>> Kstreams, you just start up an application and go.  You don’t need docker
>> either…although containerizing your stuff is probably a good strategy for
>> the purposes of deployment management (something you get with Yarn or a
>> spark Cluster)…but you’re not tied to any one framework (e.g. you can use
>> kubernetes, mesos, Yarn, or anything else)
>>
>> On 4/27/17, 10:52 AM, "Mina Aslani" <as...@gmail.com> wrote:
>>
>>     Hi,
>>
>>     I created a kafka stream app and as I was informed I created a docker
>> image
>>     with the app and launched it as a container. However, I have couple of
>>     questions:
>>
>>     - Would every Kafka streaming job require a new docker image and
>> deployment
>>     of the container/service?
>>     - How should I structure things differently if I had more than one
>> Kafka
>>     streaming app/job?
>>     - What are the advantages of using Kafka streaming over Spark
>> streaming?
>>     I'm asking b/c with Spark streaming I don't need to create and deploy
>> a new
>>     docker image every time I added or changed an/a app/job.
>>
>>     Best regards,
>>     Mina
>>
>>
>>
>