You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by Mina Aslani <as...@gmail.com> on 2017/03/21 20:08:26 UTC
Using Kafka Stream in the cluster of kafka on one or multiple docker-machine/s
Hi,
I am trying to understand how I can use a kafka stream app(jar file) in a
cluster of kafka containers.
Kafka does not have master/slave concept (unlike spark), how I should run
my app in the cluster of kafkas (e.g. on one or multiple docker-machine/s)?
I use below command line when having one VM/node with one kafka container
https://github.com/confluentinc/examples/blob/3.2.x/kafka-streams/src/main/
java/io/confluent/examples/streams/WordCountLambdaExample.java#L55-L62
Best regards,
Mina
Re: Using Kafka Stream in the cluster of kafka on one or multiple docker-machine/s
Posted by Mina Aslani <as...@gmail.com>.
Thank you Michael for the prompt response, really appreciate it!
Best regards,
Mina
On Thu, Mar 30, 2017 at 4:50 AM, Michael Noll <mi...@confluent.io> wrote:
> If you want to deploy a Kafka Streams application, then essentially you
> only need the (fat) jar of your application and a JRE in your container.
> In other words, it's the same setup you'd use to deploy *any* kind of Java
> application.
>
> You do not need to containerize "Kafka", which I assume you meant as
> containerizing "a Kafka cluster or broker".
>
> > And after I have the container running, in my container should I run Java
> -cp ... same as
> > https://github.com/confluentinc/examples/blob/3.
> 2.x/kafka-streams/src/main/java/io/confluent/examples/streams/
> WordCountLambdaExample.java#L55-L62?
>
> Yes.
>
>
> -Michael
>
>
>
> On Thu, Mar 30, 2017 at 4:23 AM, Mina Aslani <as...@gmail.com> wrote:
>
> > Hi,
> >
> > Do we have an example of a container with an instance of the jar file by
> > any chance? I am wondering if I should have a container of headless java
> or
> > should I have a container of Kafka?
> >
> > And after I have the container running, in my container should I run Java
> > -cp ... same as https://github.com/confluentinc/examples/blob/3.
> > 2.x/kafka-streams/src/main/java/io/confluent/examples/streams/
> > WordCountLambdaExample.java#L55-L62?
> >
> > Regards,
> > Mina
> >
> > On Tue, Mar 21, 2017 at 4:49 PM, Mina Aslani <as...@gmail.com>
> wrote:
> >
> > > Hi Michael,
> > >
> > > Thank you very much for the prompt response, really appreciate it!
> > >
> > > From https://github.com/confluentinc/examples/blob/3.2.x/
> > > kafka-streams/src/main/java/io/confluent/examples/streams/
> > > WordCountLambdaExample.java#L55-L62 and
> > > https://github.com/confluentinc/examples/tree/3.2.x/kafka-
> > > streams#packaging-and-running I missed the fact that the jar should be
> > > run in a separate container.
> > >
> > > Best regards,
> > > Mina
> > >
> > > On Tue, Mar 21, 2017 at 4:34 PM, Michael Noll <mi...@confluent.io>
> > > wrote:
> > >
> > >> Typically you'd containerize your app and then launch e.g. 10
> containers
> > >> if
> > >> you need to run 10 instances of your app.
> > >>
> > >> Also, what do you mean by "in a cluster of Kafka containers" and "in
> the
> > >> cluster of Kafkas"?
> > >>
> > >> On Tue, Mar 21, 2017 at 9:08 PM, Mina Aslani <as...@gmail.com>
> > >> wrote:
> > >>
> > >> > Hi,
> > >> >
> > >> > I am trying to understand how I can use a kafka stream app(jar file)
> > in
> > >> a
> > >> > cluster of kafka containers.
> > >> >
> > >> > Kafka does not have master/slave concept (unlike spark), how I
> should
> > >> run
> > >> > my app in the cluster of kafkas (e.g. on one or multiple
> > >> docker-machine/s)?
> > >> >
> > >> > I use below command line when having one VM/node with one kafka
> > >> container
> > >> > https://github.com/confluentinc/examples/blob/3.
> > >> > 2.x/kafka-streams/src/main/
> > >> > java/io/confluent/examples/streams/WordCountLambdaExample.java#
> > L55-L62
> > >> >
> > >> > Best regards,
> > >> > Mina
> > >> >
> > >>
> > >
> > >
> >
>
Re: Using Kafka Stream in the cluster of kafka on one or multiple docker-machine/s
Posted by Michael Noll <mi...@confluent.io>.
If you want to deploy a Kafka Streams application, then essentially you
only need the (fat) jar of your application and a JRE in your container.
In other words, it's the same setup you'd use to deploy *any* kind of Java
application.
You do not need to containerize "Kafka", which I assume you meant as
containerizing "a Kafka cluster or broker".
> And after I have the container running, in my container should I run Java
-cp ... same as
> https://github.com/confluentinc/examples/blob/3.
2.x/kafka-streams/src/main/java/io/confluent/examples/streams/WordCountLambdaExample.java#L55-L62?
Yes.
-Michael
On Thu, Mar 30, 2017 at 4:23 AM, Mina Aslani <as...@gmail.com> wrote:
> Hi,
>
> Do we have an example of a container with an instance of the jar file by
> any chance? I am wondering if I should have a container of headless java or
> should I have a container of Kafka?
>
> And after I have the container running, in my container should I run Java
> -cp ... same as https://github.com/confluentinc/examples/blob/3.
> 2.x/kafka-streams/src/main/java/io/confluent/examples/streams/
> WordCountLambdaExample.java#L55-L62?
>
> Regards,
> Mina
>
> On Tue, Mar 21, 2017 at 4:49 PM, Mina Aslani <as...@gmail.com> wrote:
>
> > Hi Michael,
> >
> > Thank you very much for the prompt response, really appreciate it!
> >
> > From https://github.com/confluentinc/examples/blob/3.2.x/
> > kafka-streams/src/main/java/io/confluent/examples/streams/
> > WordCountLambdaExample.java#L55-L62 and
> > https://github.com/confluentinc/examples/tree/3.2.x/kafka-
> > streams#packaging-and-running I missed the fact that the jar should be
> > run in a separate container.
> >
> > Best regards,
> > Mina
> >
> > On Tue, Mar 21, 2017 at 4:34 PM, Michael Noll <mi...@confluent.io>
> > wrote:
> >
> >> Typically you'd containerize your app and then launch e.g. 10 containers
> >> if
> >> you need to run 10 instances of your app.
> >>
> >> Also, what do you mean by "in a cluster of Kafka containers" and "in the
> >> cluster of Kafkas"?
> >>
> >> On Tue, Mar 21, 2017 at 9:08 PM, Mina Aslani <as...@gmail.com>
> >> wrote:
> >>
> >> > Hi,
> >> >
> >> > I am trying to understand how I can use a kafka stream app(jar file)
> in
> >> a
> >> > cluster of kafka containers.
> >> >
> >> > Kafka does not have master/slave concept (unlike spark), how I should
> >> run
> >> > my app in the cluster of kafkas (e.g. on one or multiple
> >> docker-machine/s)?
> >> >
> >> > I use below command line when having one VM/node with one kafka
> >> container
> >> > https://github.com/confluentinc/examples/blob/3.
> >> > 2.x/kafka-streams/src/main/
> >> > java/io/confluent/examples/streams/WordCountLambdaExample.java#
> L55-L62
> >> >
> >> > Best regards,
> >> > Mina
> >> >
> >>
> >
> >
>
Re: Using Kafka Stream in the cluster of kafka on one or multiple docker-machine/s
Posted by Mina Aslani <as...@gmail.com>.
Hi,
Do we have an example of a container with an instance of the jar file by
any chance? I am wondering if I should have a container of headless java or
should I have a container of Kafka?
And after I have the container running, in my container should I run Java
-cp ... same as https://github.com/confluentinc/examples/blob/3.
2.x/kafka-streams/src/main/java/io/confluent/examples/streams/
WordCountLambdaExample.java#L55-L62?
Regards,
Mina
On Tue, Mar 21, 2017 at 4:49 PM, Mina Aslani <as...@gmail.com> wrote:
> Hi Michael,
>
> Thank you very much for the prompt response, really appreciate it!
>
> From https://github.com/confluentinc/examples/blob/3.2.x/
> kafka-streams/src/main/java/io/confluent/examples/streams/
> WordCountLambdaExample.java#L55-L62 and
> https://github.com/confluentinc/examples/tree/3.2.x/kafka-
> streams#packaging-and-running I missed the fact that the jar should be
> run in a separate container.
>
> Best regards,
> Mina
>
> On Tue, Mar 21, 2017 at 4:34 PM, Michael Noll <mi...@confluent.io>
> wrote:
>
>> Typically you'd containerize your app and then launch e.g. 10 containers
>> if
>> you need to run 10 instances of your app.
>>
>> Also, what do you mean by "in a cluster of Kafka containers" and "in the
>> cluster of Kafkas"?
>>
>> On Tue, Mar 21, 2017 at 9:08 PM, Mina Aslani <as...@gmail.com>
>> wrote:
>>
>> > Hi,
>> >
>> > I am trying to understand how I can use a kafka stream app(jar file) in
>> a
>> > cluster of kafka containers.
>> >
>> > Kafka does not have master/slave concept (unlike spark), how I should
>> run
>> > my app in the cluster of kafkas (e.g. on one or multiple
>> docker-machine/s)?
>> >
>> > I use below command line when having one VM/node with one kafka
>> container
>> > https://github.com/confluentinc/examples/blob/3.
>> > 2.x/kafka-streams/src/main/
>> > java/io/confluent/examples/streams/WordCountLambdaExample.java#L55-L62
>> >
>> > Best regards,
>> > Mina
>> >
>>
>
>
Re: Using Kafka Stream in the cluster of kafka on one or multiple docker-machine/s
Posted by Mina Aslani <as...@gmail.com>.
Hi Michael,
Thank you very much for the prompt response, really appreciate it!
From https://github.com/confluentinc/examples/blob/3.2.x/kafka-
streams/src/main/java/io/confluent/examples/streams/Wor
dCountLambdaExample.java#L55-L62 and
https://github.com/confluentinc/examples/tree/3.2.x/kafka-streams#packaging-and-running
I
missed the fact that the jar should be run in a separate container.
Best regards,
Mina
On Tue, Mar 21, 2017 at 4:34 PM, Michael Noll <mi...@confluent.io> wrote:
> Typically you'd containerize your app and then launch e.g. 10 containers if
> you need to run 10 instances of your app.
>
> Also, what do you mean by "in a cluster of Kafka containers" and "in the
> cluster of Kafkas"?
>
> On Tue, Mar 21, 2017 at 9:08 PM, Mina Aslani <as...@gmail.com> wrote:
>
> > Hi,
> >
> > I am trying to understand how I can use a kafka stream app(jar file) in a
> > cluster of kafka containers.
> >
> > Kafka does not have master/slave concept (unlike spark), how I should run
> > my app in the cluster of kafkas (e.g. on one or multiple
> docker-machine/s)?
> >
> > I use below command line when having one VM/node with one kafka container
> > https://github.com/confluentinc/examples/blob/3.
> > 2.x/kafka-streams/src/main/
> > java/io/confluent/examples/streams/WordCountLambdaExample.java#L55-L62
> >
> > Best regards,
> > Mina
> >
>
Re: Using Kafka Stream in the cluster of kafka on one or multiple docker-machine/s
Posted by Michael Noll <mi...@confluent.io>.
Typically you'd containerize your app and then launch e.g. 10 containers if
you need to run 10 instances of your app.
Also, what do you mean by "in a cluster of Kafka containers" and "in the
cluster of Kafkas"?
On Tue, Mar 21, 2017 at 9:08 PM, Mina Aslani <as...@gmail.com> wrote:
> Hi,
>
> I am trying to understand how I can use a kafka stream app(jar file) in a
> cluster of kafka containers.
>
> Kafka does not have master/slave concept (unlike spark), how I should run
> my app in the cluster of kafkas (e.g. on one or multiple docker-machine/s)?
>
> I use below command line when having one VM/node with one kafka container
> https://github.com/confluentinc/examples/blob/3.
> 2.x/kafka-streams/src/main/
> java/io/confluent/examples/streams/WordCountLambdaExample.java#L55-L62
>
> Best regards,
> Mina
>