You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by Chandrashekhar Kotekar <sh...@gmail.com> on 2015/05/03 06:08:28 UTC
Kafka behind AWS ELB
Hi,
I am new to Apache Kafka. I have played with it on my laptop.
I want to use Kafka in AWS. Currently we have tomcat web servers based REST
API. We want to replace REST API with Apache Kafka, web servers are behind
ELB.
I would like to know if we can keep Kafka brokers behind ELB? Will it work?
Regards,
Chandrash3khar Kotekar
Mobile - +91 8600011455
Re: Kafka behind AWS ELB
Posted by Christian Csar <ch...@csar.us>.
Dillian,
On Mon, May 4, 2015 at 1:52 PM, Dillian Murphey <cr...@gmail.com>
wrote:
>
> I'm interested in this topic as well. If you put kafka brokers inside an
> autoscaling group, then AWS will automatically add brokers if demand
> increases, and the ELB will automatically round-robin across all of your
> kafka instances. So in your config files and code, you only need to
> provide a single DNS name (the load balancer). You don't need to specify
> all your kafka brokers inside your config file. If a broker dies, the ELB
> will only route to healthy nodes.
>
> So you get a lot of robustness, scalability, and fault-tolerance by using
> the AWS services. Kafka Brokers will automatically load balance, but the
> question is whether it is ok to put all your brokers behind an ELB and
> expect the system to work properly.
>
You should not expect it to work properly. Broker nodes are data bearing
which means that any operation to scale down would need to be aware of the
data distribution. The client connects to specific nodes to send them data
so even the Level 4 load balancing wouldn't work.
> What alternatives are there to dynamic/scalable broker clusters? I don't
> want to have to modify my config files or code if I add more brokers, and
I
> want to be able to handle a broker going down. So these are the reasons
AWS
> questions like this come up.
>
The clients already give you options for specifying only a subset of
brokers
https://cwiki.apache.org/confluence/display/KAFKA/0.8.0+Producer+Example
one of which must be alive to discover the rest of the cluster. The main
clients handle node failures (you'll still have some operational work).
Kafka and other data storage systems do not work the same as an HTTP driven
web application. While it can certainly be scaled, and automation could be
done to do so in response to load it's going to be more complicated. AWS's
off the shelf solution/low operations offering for some (definitely not
all) of Kafka's use cases is Kinesis, Azure's is EventHubs. Before using
Kakfa or any system in production you'll want to be sure you understand the
operational aspects of it.
Christian
>
> Thanks for any comments too. :)
>
>
>
>
> On Mon, May 4, 2015 at 9:03 AM, Mayuresh Gharat <
gharatmayuresh15@gmail.com>
> wrote:
>
> > Ok. You can deploy kafka in AWS. You can have brokers on AWS servers.
> > Kafka is not a push system. So you will need someone writing to kafka
and
> > consuming from kafka. It will work. My suggestion will be to try it out
on
> > a smaller instance in AWS and see the effects.
> >
> > As I do not know the actual use case about why you want to use kafka
for, I
> > cannot comment on whether it will work for you personalized use case.
> >
> > Thanks,
> >
> > Mayuresh
> >
> > On Mon, May 4, 2015 at 8:55 AM, Chandrashekhar Kotekar <
> > shekhar.kotekar@gmail.com> wrote:
> >
> > > I am sorry but I cannot reveal those details due to confidentiality
> > issues.
> > > I hope you understand.
> > >
> > >
> > > Regards,
> > > Chandrash3khar Kotekar
> > > Mobile - +91 8600011455
> > >
> > > On Mon, May 4, 2015 at 9:18 PM, Mayuresh Gharat <
> > > gharatmayuresh15@gmail.com>
> > > wrote:
> > >
> > > > Hi Chandrashekar,
> > > >
> > > > Can you please elaborate the use case for Kafka here, like how you
are
> > > > planning to use it.
> > > >
> > > >
> > > > Thanks,
> > > >
> > > > Mayuresh
> > > >
> > > > On Sat, May 2, 2015 at 9:08 PM, Chandrashekhar Kotekar <
> > > > shekhar.kotekar@gmail.com> wrote:
> > > >
> > > > > Hi,
> > > > >
> > > > > I am new to Apache Kafka. I have played with it on my laptop.
> > > > >
> > > > > I want to use Kafka in AWS. Currently we have tomcat web servers
> > based
> > > > REST
> > > > > API. We want to replace REST API with Apache Kafka, web servers
are
> > > > behind
> > > > > ELB.
> > > > >
> > > > > I would like to know if we can keep Kafka brokers behind ELB?
Will it
> > > > work?
> > > > >
> > > > > Regards,
> > > > > Chandrash3khar Kotekar
> > > > > Mobile - +91 8600011455
> > > > >
> > > >
> > > >
> > > >
> > > > --
> > > > -Regards,
> > > > Mayuresh R. Gharat
> > > > (862) 250-7125
> > > >
> > >
> >
> >
> >
> > --
> > -Regards,
> > Mayuresh R. Gharat
> > (862) 250-7125
> >
Re: Kafka behind AWS ELB
Posted by Jiangjie Qin <jq...@linkedin.com.INVALID>.
Hi Dillian,
Yeah, ELB + ASG will is pretty popular however might need further tricks
to use them for Kafka brokers.
As I replied to Chandrashekhar in another email. You can use ELB as
bootstrap.servers/metadata.broker.list to serve as client bootstrap use
case. But all the producing/consuming traffic still need to go to the
brokers directly. Another thing about ASG is that when new Kafka broker
comes up, it might need to copy a lot of data if its log is empty. That
might take a while. This might not be the case if you are using EBS. I
think Netflix has some practical experience of running Kafka on AWS with
dynamically changing number of brokers, but they had some tricks to manage
the metadata(broker id, etc) and log files.
Anyway, I would be surprised if ELB + ASG can be easily used for dynamic
provision of Kafka brokers. But I think it probably is a very helpful
improvement if we can do that.
Jiangjie (Becket) Qin
On 5/4/15, 1:52 PM, "Dillian Murphey" <cr...@gmail.com> wrote:
>I'm interested in this topic as well. If you put kafka brokers inside an
>autoscaling group, then AWS will automatically add brokers if demand
>increases, and the ELB will automatically round-robin across all of your
>kafka instances. So in your config files and code, you only need to
>provide a single DNS name (the load balancer). You don't need to specify
>all your kafka brokers inside your config file. If a broker dies, the ELB
>will only route to healthy nodes.
>
>So you get a lot of robustness, scalability, and fault-tolerance by using
>the AWS services. Kafka Brokers will automatically load balance, but the
>question is whether it is ok to put all your brokers behind an ELB and
>expect the system to work properly.
>
>What alternatives are there to dynamic/scalable broker clusters? I don't
>want to have to modify my config files or code if I add more brokers, and
>I
>want to be able to handle a broker going down. So these are the reasons
>AWS
>questions like this come up.
>
>Thanks for any comments too. :)
>
>
>
>
>On Mon, May 4, 2015 at 9:03 AM, Mayuresh Gharat
><gh...@gmail.com>
>wrote:
>
>> Ok. You can deploy kafka in AWS. You can have brokers on AWS servers.
>> Kafka is not a push system. So you will need someone writing to kafka
>>and
>> consuming from kafka. It will work. My suggestion will be to try it out
>>on
>> a smaller instance in AWS and see the effects.
>>
>> As I do not know the actual use case about why you want to use kafka
>>for, I
>> cannot comment on whether it will work for you personalized use case.
>>
>> Thanks,
>>
>> Mayuresh
>>
>> On Mon, May 4, 2015 at 8:55 AM, Chandrashekhar Kotekar <
>> shekhar.kotekar@gmail.com> wrote:
>>
>> > I am sorry but I cannot reveal those details due to confidentiality
>> issues.
>> > I hope you understand.
>> >
>> >
>> > Regards,
>> > Chandrash3khar Kotekar
>> > Mobile - +91 8600011455
>> >
>> > On Mon, May 4, 2015 at 9:18 PM, Mayuresh Gharat <
>> > gharatmayuresh15@gmail.com>
>> > wrote:
>> >
>> > > Hi Chandrashekar,
>> > >
>> > > Can you please elaborate the use case for Kafka here, like how you
>>are
>> > > planning to use it.
>> > >
>> > >
>> > > Thanks,
>> > >
>> > > Mayuresh
>> > >
>> > > On Sat, May 2, 2015 at 9:08 PM, Chandrashekhar Kotekar <
>> > > shekhar.kotekar@gmail.com> wrote:
>> > >
>> > > > Hi,
>> > > >
>> > > > I am new to Apache Kafka. I have played with it on my laptop.
>> > > >
>> > > > I want to use Kafka in AWS. Currently we have tomcat web servers
>> based
>> > > REST
>> > > > API. We want to replace REST API with Apache Kafka, web servers
>>are
>> > > behind
>> > > > ELB.
>> > > >
>> > > > I would like to know if we can keep Kafka brokers behind ELB?
>>Will it
>> > > work?
>> > > >
>> > > > Regards,
>> > > > Chandrash3khar Kotekar
>> > > > Mobile - +91 8600011455
>> > > >
>> > >
>> > >
>> > >
>> > > --
>> > > -Regards,
>> > > Mayuresh R. Gharat
>> > > (862) 250-7125
>> > >
>> >
>>
>>
>>
>> --
>> -Regards,
>> Mayuresh R. Gharat
>> (862) 250-7125
>>
Re: Kafka behind AWS ELB
Posted by Dillian Murphey <cr...@gmail.com>.
I'm interested in this topic as well. If you put kafka brokers inside an
autoscaling group, then AWS will automatically add brokers if demand
increases, and the ELB will automatically round-robin across all of your
kafka instances. So in your config files and code, you only need to
provide a single DNS name (the load balancer). You don't need to specify
all your kafka brokers inside your config file. If a broker dies, the ELB
will only route to healthy nodes.
So you get a lot of robustness, scalability, and fault-tolerance by using
the AWS services. Kafka Brokers will automatically load balance, but the
question is whether it is ok to put all your brokers behind an ELB and
expect the system to work properly.
What alternatives are there to dynamic/scalable broker clusters? I don't
want to have to modify my config files or code if I add more brokers, and I
want to be able to handle a broker going down. So these are the reasons AWS
questions like this come up.
Thanks for any comments too. :)
On Mon, May 4, 2015 at 9:03 AM, Mayuresh Gharat <gh...@gmail.com>
wrote:
> Ok. You can deploy kafka in AWS. You can have brokers on AWS servers.
> Kafka is not a push system. So you will need someone writing to kafka and
> consuming from kafka. It will work. My suggestion will be to try it out on
> a smaller instance in AWS and see the effects.
>
> As I do not know the actual use case about why you want to use kafka for, I
> cannot comment on whether it will work for you personalized use case.
>
> Thanks,
>
> Mayuresh
>
> On Mon, May 4, 2015 at 8:55 AM, Chandrashekhar Kotekar <
> shekhar.kotekar@gmail.com> wrote:
>
> > I am sorry but I cannot reveal those details due to confidentiality
> issues.
> > I hope you understand.
> >
> >
> > Regards,
> > Chandrash3khar Kotekar
> > Mobile - +91 8600011455
> >
> > On Mon, May 4, 2015 at 9:18 PM, Mayuresh Gharat <
> > gharatmayuresh15@gmail.com>
> > wrote:
> >
> > > Hi Chandrashekar,
> > >
> > > Can you please elaborate the use case for Kafka here, like how you are
> > > planning to use it.
> > >
> > >
> > > Thanks,
> > >
> > > Mayuresh
> > >
> > > On Sat, May 2, 2015 at 9:08 PM, Chandrashekhar Kotekar <
> > > shekhar.kotekar@gmail.com> wrote:
> > >
> > > > Hi,
> > > >
> > > > I am new to Apache Kafka. I have played with it on my laptop.
> > > >
> > > > I want to use Kafka in AWS. Currently we have tomcat web servers
> based
> > > REST
> > > > API. We want to replace REST API with Apache Kafka, web servers are
> > > behind
> > > > ELB.
> > > >
> > > > I would like to know if we can keep Kafka brokers behind ELB? Will it
> > > work?
> > > >
> > > > Regards,
> > > > Chandrash3khar Kotekar
> > > > Mobile - +91 8600011455
> > > >
> > >
> > >
> > >
> > > --
> > > -Regards,
> > > Mayuresh R. Gharat
> > > (862) 250-7125
> > >
> >
>
>
>
> --
> -Regards,
> Mayuresh R. Gharat
> (862) 250-7125
>
Re: Kafka behind AWS ELB
Posted by Mayuresh Gharat <gh...@gmail.com>.
Ok. You can deploy kafka in AWS. You can have brokers on AWS servers.
Kafka is not a push system. So you will need someone writing to kafka and
consuming from kafka. It will work. My suggestion will be to try it out on
a smaller instance in AWS and see the effects.
As I do not know the actual use case about why you want to use kafka for, I
cannot comment on whether it will work for you personalized use case.
Thanks,
Mayuresh
On Mon, May 4, 2015 at 8:55 AM, Chandrashekhar Kotekar <
shekhar.kotekar@gmail.com> wrote:
> I am sorry but I cannot reveal those details due to confidentiality issues.
> I hope you understand.
>
>
> Regards,
> Chandrash3khar Kotekar
> Mobile - +91 8600011455
>
> On Mon, May 4, 2015 at 9:18 PM, Mayuresh Gharat <
> gharatmayuresh15@gmail.com>
> wrote:
>
> > Hi Chandrashekar,
> >
> > Can you please elaborate the use case for Kafka here, like how you are
> > planning to use it.
> >
> >
> > Thanks,
> >
> > Mayuresh
> >
> > On Sat, May 2, 2015 at 9:08 PM, Chandrashekhar Kotekar <
> > shekhar.kotekar@gmail.com> wrote:
> >
> > > Hi,
> > >
> > > I am new to Apache Kafka. I have played with it on my laptop.
> > >
> > > I want to use Kafka in AWS. Currently we have tomcat web servers based
> > REST
> > > API. We want to replace REST API with Apache Kafka, web servers are
> > behind
> > > ELB.
> > >
> > > I would like to know if we can keep Kafka brokers behind ELB? Will it
> > work?
> > >
> > > Regards,
> > > Chandrash3khar Kotekar
> > > Mobile - +91 8600011455
> > >
> >
> >
> >
> > --
> > -Regards,
> > Mayuresh R. Gharat
> > (862) 250-7125
> >
>
--
-Regards,
Mayuresh R. Gharat
(862) 250-7125
Re: Kafka behind AWS ELB
Posted by Chandrashekhar Kotekar <sh...@gmail.com>.
I am sorry but I cannot reveal those details due to confidentiality issues.
I hope you understand.
Regards,
Chandrash3khar Kotekar
Mobile - +91 8600011455
On Mon, May 4, 2015 at 9:18 PM, Mayuresh Gharat <gh...@gmail.com>
wrote:
> Hi Chandrashekar,
>
> Can you please elaborate the use case for Kafka here, like how you are
> planning to use it.
>
>
> Thanks,
>
> Mayuresh
>
> On Sat, May 2, 2015 at 9:08 PM, Chandrashekhar Kotekar <
> shekhar.kotekar@gmail.com> wrote:
>
> > Hi,
> >
> > I am new to Apache Kafka. I have played with it on my laptop.
> >
> > I want to use Kafka in AWS. Currently we have tomcat web servers based
> REST
> > API. We want to replace REST API with Apache Kafka, web servers are
> behind
> > ELB.
> >
> > I would like to know if we can keep Kafka brokers behind ELB? Will it
> work?
> >
> > Regards,
> > Chandrash3khar Kotekar
> > Mobile - +91 8600011455
> >
>
>
>
> --
> -Regards,
> Mayuresh R. Gharat
> (862) 250-7125
>
Re: Kafka behind AWS ELB
Posted by Mayuresh Gharat <gh...@gmail.com>.
Hi Chandrashekar,
Can you please elaborate the use case for Kafka here, like how you are
planning to use it.
Thanks,
Mayuresh
On Sat, May 2, 2015 at 9:08 PM, Chandrashekhar Kotekar <
shekhar.kotekar@gmail.com> wrote:
> Hi,
>
> I am new to Apache Kafka. I have played with it on my laptop.
>
> I want to use Kafka in AWS. Currently we have tomcat web servers based REST
> API. We want to replace REST API with Apache Kafka, web servers are behind
> ELB.
>
> I would like to know if we can keep Kafka brokers behind ELB? Will it work?
>
> Regards,
> Chandrash3khar Kotekar
> Mobile - +91 8600011455
>
--
-Regards,
Mayuresh R. Gharat
(862) 250-7125