You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by Prabhjot Bharaj <pr...@gmail.com> on 2015/07/27 15:55:11 UTC
Best practices - Using kafka (with http server) as source-of-truth
Hi Folks,
I would like to understand the best practices when using kafka as the
source-of-truth, given the fact that I want to pump in data to Kafka using
http methods.
What are the current production configurations for such a use case:-
1. Kafka-http-client - is it scalable the way Nginx is ??
2. Using Kafka and Nginx together - If anybody has used this, please explain
3. Any other scalable method ?
Regards,
prabcs
Re: Best practices - Using kafka (with http server) as source-of-truth
Posted by Prabhjot Bharaj <pr...@gmail.com>.
Hi Ewen,
Thanks for your response. I'll experiment and benchmark it with the normal
proxy and NGinx as well and update the results.
Regards,
prabcs
On Mon, Jul 27, 2015 at 11:10 PM, Ewen Cheslack-Postava <ew...@confluent.io>
wrote:
> Hi Prabhjot,
>
> Confluent has a REST proxy with docs that may give some guidance:
> http://docs.confluent.io/1.0/kafka-rest/docs/intro.html The new producer
> that it uses is very efficient, so you should be able to get pretty good
> throughput. You take a bit of a hit due to the overhead of sending data
> through a proxy, but with appropriate batching you can get about 2/3 the
> performance as you would get using the Java producer directly.
>
> There are also a few other proxies you can find here:
> https://cwiki.apache.org/confluence/display/KAFKA/Clients#Clients-HTTPREST
>
> You can also put nginx (or HAProxy, or a variety of other solutions) in
> front of REST proxies for load balancing, HA, SSL termination, etc. This is
> yet another hop, so it might affect throughput and latency.
>
> -Ewen
>
> On Mon, Jul 27, 2015 at 6:55 AM, Prabhjot Bharaj <pr...@gmail.com>
> wrote:
>
> > Hi Folks,
> >
> > I would like to understand the best practices when using kafka as the
> > source-of-truth, given the fact that I want to pump in data to Kafka
> using
> > http methods.
> >
> > What are the current production configurations for such a use case:-
> >
> > 1. Kafka-http-client - is it scalable the way Nginx is ??
> > 2. Using Kafka and Nginx together - If anybody has used this, please
> > explain
> > 3. Any other scalable method ?
> >
> > Regards,
> > prabcs
> >
>
>
>
> --
> Thanks,
> Ewen
>
--
---------------------------------------------------------
"There are only 10 types of people in the world: Those who understand
binary, and those who don't"
Re: Best practices - Using kafka (with http server) as source-of-truth
Posted by Prabhjot Bharaj <pr...@gmail.com>.
Hi Ewen,
Thanks for your response. I'll experiment and benchmark it with the normal
proxy and NGinx as well and update the results.
Regards,
prabcs
On Mon, Jul 27, 2015 at 11:10 PM, Ewen Cheslack-Postava <ew...@confluent.io>
wrote:
> Hi Prabhjot,
>
> Confluent has a REST proxy with docs that may give some guidance:
> http://docs.confluent.io/1.0/kafka-rest/docs/intro.html The new producer
> that it uses is very efficient, so you should be able to get pretty good
> throughput. You take a bit of a hit due to the overhead of sending data
> through a proxy, but with appropriate batching you can get about 2/3 the
> performance as you would get using the Java producer directly.
>
> There are also a few other proxies you can find here:
> https://cwiki.apache.org/confluence/display/KAFKA/Clients#Clients-HTTPREST
>
> You can also put nginx (or HAProxy, or a variety of other solutions) in
> front of REST proxies for load balancing, HA, SSL termination, etc. This is
> yet another hop, so it might affect throughput and latency.
>
> -Ewen
>
> On Mon, Jul 27, 2015 at 6:55 AM, Prabhjot Bharaj <pr...@gmail.com>
> wrote:
>
> > Hi Folks,
> >
> > I would like to understand the best practices when using kafka as the
> > source-of-truth, given the fact that I want to pump in data to Kafka
> using
> > http methods.
> >
> > What are the current production configurations for such a use case:-
> >
> > 1. Kafka-http-client - is it scalable the way Nginx is ??
> > 2. Using Kafka and Nginx together - If anybody has used this, please
> > explain
> > 3. Any other scalable method ?
> >
> > Regards,
> > prabcs
> >
>
>
>
> --
> Thanks,
> Ewen
>
--
---------------------------------------------------------
"There are only 10 types of people in the world: Those who understand
binary, and those who don't"
Re: Best practices - Using kafka (with http server) as source-of-truth
Posted by Ewen Cheslack-Postava <ew...@confluent.io>.
Hi Prabhjot,
Confluent has a REST proxy with docs that may give some guidance:
http://docs.confluent.io/1.0/kafka-rest/docs/intro.html The new producer
that it uses is very efficient, so you should be able to get pretty good
throughput. You take a bit of a hit due to the overhead of sending data
through a proxy, but with appropriate batching you can get about 2/3 the
performance as you would get using the Java producer directly.
There are also a few other proxies you can find here:
https://cwiki.apache.org/confluence/display/KAFKA/Clients#Clients-HTTPREST
You can also put nginx (or HAProxy, or a variety of other solutions) in
front of REST proxies for load balancing, HA, SSL termination, etc. This is
yet another hop, so it might affect throughput and latency.
-Ewen
On Mon, Jul 27, 2015 at 6:55 AM, Prabhjot Bharaj <pr...@gmail.com>
wrote:
> Hi Folks,
>
> I would like to understand the best practices when using kafka as the
> source-of-truth, given the fact that I want to pump in data to Kafka using
> http methods.
>
> What are the current production configurations for such a use case:-
>
> 1. Kafka-http-client - is it scalable the way Nginx is ??
> 2. Using Kafka and Nginx together - If anybody has used this, please
> explain
> 3. Any other scalable method ?
>
> Regards,
> prabcs
>
--
Thanks,
Ewen
Re: Best practices - Using kafka (with http server) as source-of-truth
Posted by Ewen Cheslack-Postava <ew...@confluent.io>.
Hi Prabhjot,
Confluent has a REST proxy with docs that may give some guidance:
http://docs.confluent.io/1.0/kafka-rest/docs/intro.html The new producer
that it uses is very efficient, so you should be able to get pretty good
throughput. You take a bit of a hit due to the overhead of sending data
through a proxy, but with appropriate batching you can get about 2/3 the
performance as you would get using the Java producer directly.
There are also a few other proxies you can find here:
https://cwiki.apache.org/confluence/display/KAFKA/Clients#Clients-HTTPREST
You can also put nginx (or HAProxy, or a variety of other solutions) in
front of REST proxies for load balancing, HA, SSL termination, etc. This is
yet another hop, so it might affect throughput and latency.
-Ewen
On Mon, Jul 27, 2015 at 6:55 AM, Prabhjot Bharaj <pr...@gmail.com>
wrote:
> Hi Folks,
>
> I would like to understand the best practices when using kafka as the
> source-of-truth, given the fact that I want to pump in data to Kafka using
> http methods.
>
> What are the current production configurations for such a use case:-
>
> 1. Kafka-http-client - is it scalable the way Nginx is ??
> 2. Using Kafka and Nginx together - If anybody has used this, please
> explain
> 3. Any other scalable method ?
>
> Regards,
> prabcs
>
--
Thanks,
Ewen