You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by Chandrashekhar Kotekar <sh...@gmail.com> on 2015/07/16 15:24:05 UTC
Kafka brokers on HTTP port
Hi,
In my project Kafka producers won't be in the same network of Kafka brokers
and due to security reasons other ports are blocked.
I would like to know if it is possible to run Kafka brokers on HTTP port
(8080) so that Kafka producers will send Kafka messages over HTTP and
brokers can store them until consumers consume them.
I tried to search for this type of question in mailing list but couldn't
find exact question/answer. Sorry if this is duplicate question.
Thanks,
Chandrash3khar Kotekar
Mobile - +91 8600011455
Re: Kafka brokers on HTTP port
Posted by Ewen Cheslack-Postava <ew...@confluent.io>.
Yes, those are potential issues if you make your Kafka cluster publicly
accessible. There are some features currently being worked on that could
help address these problems (some security on connections and quotas), but
they are still in progress. You'll probably want a proxy layer to handle
this.
As for auto-scaling, you can do this but you should be aware that Kafka
doesn't currently do automatic balancing of data as brokers are added.
You'll need to manage that process yourself when new brokers are added.
-Ewen
On Thu, Jul 16, 2015 at 7:35 PM, Chandrashekhar Kotekar <
shekhar.kotekar@gmail.com> wrote:
> Thanks a lot Ewen and Edward for your valuable answers. According to new
> update from admin side, they can allow TCP only connections on Kafka
> brokers.
>
> Now another problem is that we want to keep Kafka brokers in AWS so that
> kafka brokers can be auto scaled in/out. As most of instances in AWS do not
> have public IP addresses, first we have to assign public IP to all Kafka
> brokers and keep them visible over internet.
>
> I would like to know if there will be any security issue like DoS attack or
> malicious user sending Kafka messages or something like that?
>
>
> Thanks,
> Chandrash3khar Kotekar
> Mobile - +91 8600011455
>
> On Fri, Jul 17, 2015 at 4:57 AM, Ewen Cheslack-Postava <ew...@confluent.io>
> wrote:
>
> > Chandrashekhar,
> >
> > If the firewall rules allow any TCP connection on those ports, you can
> just
> > use Kafka directly and change the default port. If they actually verify
> > that its HTTP traffic then you'd have to the REST Proxy Edward mentioned
> or
> > another HTTP-based proxy.
> >
> > -Ewen
> >
> > On Thu, Jul 16, 2015 at 9:23 AM, Edward Ribeiro <
> edward.ribeiro@gmail.com>
> > wrote:
> >
> > > Maybe what you are looking for is Kafka REST Proxy:
> > > http://docs.confluent.io/1.0/kafka-rest/docs/intro.html
> > >
> > > Edward
> > >
> > > On Thu, Jul 16, 2015 at 10:24 AM, Chandrashekhar Kotekar <
> > > shekhar.kotekar@gmail.com> wrote:
> > >
> > > > Hi,
> > > >
> > > > In my project Kafka producers won't be in the same network of Kafka
> > > brokers
> > > > and due to security reasons other ports are blocked.
> > > >
> > > > I would like to know if it is possible to run Kafka brokers on HTTP
> > port
> > > > (8080) so that Kafka producers will send Kafka messages over HTTP and
> > > > brokers can store them until consumers consume them.
> > > >
> > > > I tried to search for this type of question in mailing list but
> > couldn't
> > > > find exact question/answer. Sorry if this is duplicate question.
> > > >
> > > > Thanks,
> > > > Chandrash3khar Kotekar
> > > > Mobile - +91 8600011455
> > > >
> > >
> >
> >
> >
> > --
> > Thanks,
> > Ewen
> >
>
--
Thanks,
Ewen
Re: Kafka brokers on HTTP port
Posted by Chandrashekhar Kotekar <sh...@gmail.com>.
Thanks a lot Ewen and Edward for your valuable answers. According to new
update from admin side, they can allow TCP only connections on Kafka
brokers.
Now another problem is that we want to keep Kafka brokers in AWS so that
kafka brokers can be auto scaled in/out. As most of instances in AWS do not
have public IP addresses, first we have to assign public IP to all Kafka
brokers and keep them visible over internet.
I would like to know if there will be any security issue like DoS attack or
malicious user sending Kafka messages or something like that?
Thanks,
Chandrash3khar Kotekar
Mobile - +91 8600011455
On Fri, Jul 17, 2015 at 4:57 AM, Ewen Cheslack-Postava <ew...@confluent.io>
wrote:
> Chandrashekhar,
>
> If the firewall rules allow any TCP connection on those ports, you can just
> use Kafka directly and change the default port. If they actually verify
> that its HTTP traffic then you'd have to the REST Proxy Edward mentioned or
> another HTTP-based proxy.
>
> -Ewen
>
> On Thu, Jul 16, 2015 at 9:23 AM, Edward Ribeiro <ed...@gmail.com>
> wrote:
>
> > Maybe what you are looking for is Kafka REST Proxy:
> > http://docs.confluent.io/1.0/kafka-rest/docs/intro.html
> >
> > Edward
> >
> > On Thu, Jul 16, 2015 at 10:24 AM, Chandrashekhar Kotekar <
> > shekhar.kotekar@gmail.com> wrote:
> >
> > > Hi,
> > >
> > > In my project Kafka producers won't be in the same network of Kafka
> > brokers
> > > and due to security reasons other ports are blocked.
> > >
> > > I would like to know if it is possible to run Kafka brokers on HTTP
> port
> > > (8080) so that Kafka producers will send Kafka messages over HTTP and
> > > brokers can store them until consumers consume them.
> > >
> > > I tried to search for this type of question in mailing list but
> couldn't
> > > find exact question/answer. Sorry if this is duplicate question.
> > >
> > > Thanks,
> > > Chandrash3khar Kotekar
> > > Mobile - +91 8600011455
> > >
> >
>
>
>
> --
> Thanks,
> Ewen
>
Re: Kafka brokers on HTTP port
Posted by Ewen Cheslack-Postava <ew...@confluent.io>.
Chandrashekhar,
If the firewall rules allow any TCP connection on those ports, you can just
use Kafka directly and change the default port. If they actually verify
that its HTTP traffic then you'd have to the REST Proxy Edward mentioned or
another HTTP-based proxy.
-Ewen
On Thu, Jul 16, 2015 at 9:23 AM, Edward Ribeiro <ed...@gmail.com>
wrote:
> Maybe what you are looking for is Kafka REST Proxy:
> http://docs.confluent.io/1.0/kafka-rest/docs/intro.html
>
> Edward
>
> On Thu, Jul 16, 2015 at 10:24 AM, Chandrashekhar Kotekar <
> shekhar.kotekar@gmail.com> wrote:
>
> > Hi,
> >
> > In my project Kafka producers won't be in the same network of Kafka
> brokers
> > and due to security reasons other ports are blocked.
> >
> > I would like to know if it is possible to run Kafka brokers on HTTP port
> > (8080) so that Kafka producers will send Kafka messages over HTTP and
> > brokers can store them until consumers consume them.
> >
> > I tried to search for this type of question in mailing list but couldn't
> > find exact question/answer. Sorry if this is duplicate question.
> >
> > Thanks,
> > Chandrash3khar Kotekar
> > Mobile - +91 8600011455
> >
>
--
Thanks,
Ewen
Re: Kafka brokers on HTTP port
Posted by Edward Ribeiro <ed...@gmail.com>.
Maybe what you are looking for is Kafka REST Proxy:
http://docs.confluent.io/1.0/kafka-rest/docs/intro.html
Edward
On Thu, Jul 16, 2015 at 10:24 AM, Chandrashekhar Kotekar <
shekhar.kotekar@gmail.com> wrote:
> Hi,
>
> In my project Kafka producers won't be in the same network of Kafka brokers
> and due to security reasons other ports are blocked.
>
> I would like to know if it is possible to run Kafka brokers on HTTP port
> (8080) so that Kafka producers will send Kafka messages over HTTP and
> brokers can store them until consumers consume them.
>
> I tried to search for this type of question in mailing list but couldn't
> find exact question/answer. Sorry if this is duplicate question.
>
> Thanks,
> Chandrash3khar Kotekar
> Mobile - +91 8600011455
>