You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@kafka.apache.org by Akshay Das <ak...@fivetran.com.INVALID> on 2019/09/12 09:22:45 UTC

Kafka SSH Tunnel Connection without editing hostfile

Hi Team,

I'm trying to consume from a kafka cluster using java client, but the kafka
server can only be accessed via jumphost/ssh tunnel. But even after
creating ssh tunnel we are not able to read because once conusmer fetches
metadata it uses original hosts to connect to broker. Is it possible to
stop this behaviour?

Thanks,
Akshay Das

Re: Kafka SSH Tunnel Connection without editing hostfile

Posted by David Jacot <dj...@confluent.io>.
Hey,

I have done this before with this proxy:
https://github.com/grepplabs/kafka-proxy#connect-to-kafka-through-socks5-proxy-example

You can spin up a socks proxy when you ssh to your jumphost (-D argument if
not mistaken) and
configure the proxy as described in the readme.

It is good for dev and test purposes. I wouldn't do this in production.

Best,
David

On Thu, Sep 12, 2019 at 11:23 AM Akshay Das <ak...@fivetran.com.invalid>
wrote:

> Hi Team,
>
> I'm trying to consume from a kafka cluster using java client, but the kafka
> server can only be accessed via jumphost/ssh tunnel. But even after
> creating ssh tunnel we are not able to read because once conusmer fetches
> metadata it uses original hosts to connect to broker. Is it possible to
> stop this behaviour?
>
> Thanks,
> Akshay Das
>

Re: Kafka SSH Tunnel Connection without editing hostfile

Posted by Akshay Das <ak...@fivetran.com.INVALID>.
That is not the requirement. We want the communication via ssh tunnel.

On Fri, Sep 13, 2019 at 4:50 PM M. Manna <ma...@gmail.com> wrote:

> why not try using internal vs external traffic
>
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-103%3A+Separation+of+Internal+and+External+traffic
>
>
> if you set EXTERNAL enndpoints and map it to SSL - you clients should only
> receive EXTERNAL endpoints for comms. Does this sound okay for you?
>
> Thanks,
>
> On Fri, 13 Sep 2019 at 06:41, Akshay Das <ak...@fivetran.com> wrote:
>
>> We cannot use external endpoints because of security reasons.
>> Is there an option to tell zookeeper/broker not to send broker host
>> detail metadata to its clients?
>>
>> On Thu, Sep 12, 2019 at 3:05 PM M. Manna <ma...@gmail.com> wrote:
>>
>>> Have you tried using EXTERNAL endpoints for your Kafka broker to separate
>>> TLS from internal traffic? Also, have you checked zk admin whether the
>>> broker metadata is exposing your TLS endpoints to clients ?
>>>
>>>
>>> On Thu, 12 Sep 2019 at 10:23, Akshay Das <ak...@fivetran.com.invalid>
>>> wrote:
>>>
>>> > Hi Team,
>>> >
>>> > I'm trying to consume from a kafka cluster using java client, but the
>>> kafka
>>> > server can only be accessed via jumphost/ssh tunnel. But even after
>>> > creating ssh tunnel we are not able to read because once conusmer
>>> fetches
>>> > metadata it uses original hosts to connect to broker. Is it possible to
>>> > stop this behaviour?
>>> >
>>> > Thanks,
>>> > Akshay Das
>>> >
>>>
>>

Re: Kafka SSH Tunnel Connection without editing hostfile

Posted by "M. Manna" <ma...@gmail.com>.
why not try using internal vs external traffic

https://cwiki.apache.org/confluence/display/KAFKA/KIP-103%3A+Separation+of+Internal+and+External+traffic


if you set EXTERNAL enndpoints and map it to SSL - you clients should only
receive EXTERNAL endpoints for comms. Does this sound okay for you?

Thanks,

On Fri, 13 Sep 2019 at 06:41, Akshay Das <ak...@fivetran.com> wrote:

> We cannot use external endpoints because of security reasons.
> Is there an option to tell zookeeper/broker not to send broker host detail
> metadata to its clients?
>
> On Thu, Sep 12, 2019 at 3:05 PM M. Manna <ma...@gmail.com> wrote:
>
>> Have you tried using EXTERNAL endpoints for your Kafka broker to separate
>> TLS from internal traffic? Also, have you checked zk admin whether the
>> broker metadata is exposing your TLS endpoints to clients ?
>>
>>
>> On Thu, 12 Sep 2019 at 10:23, Akshay Das <ak...@fivetran.com.invalid>
>> wrote:
>>
>> > Hi Team,
>> >
>> > I'm trying to consume from a kafka cluster using java client, but the
>> kafka
>> > server can only be accessed via jumphost/ssh tunnel. But even after
>> > creating ssh tunnel we are not able to read because once conusmer
>> fetches
>> > metadata it uses original hosts to connect to broker. Is it possible to
>> > stop this behaviour?
>> >
>> > Thanks,
>> > Akshay Das
>> >
>>
>

Re: Kafka SSH Tunnel Connection without editing hostfile

Posted by Akshay Das <ak...@fivetran.com.INVALID>.
We cannot use external endpoints because of security reasons.
Is there an option to tell zookeeper/broker not to send broker host detail
metadata to its clients?

On Thu, Sep 12, 2019 at 3:05 PM M. Manna <ma...@gmail.com> wrote:

> Have you tried using EXTERNAL endpoints for your Kafka broker to separate
> TLS from internal traffic? Also, have you checked zk admin whether the
> broker metadata is exposing your TLS endpoints to clients ?
>
>
> On Thu, 12 Sep 2019 at 10:23, Akshay Das <ak...@fivetran.com.invalid>
> wrote:
>
> > Hi Team,
> >
> > I'm trying to consume from a kafka cluster using java client, but the
> kafka
> > server can only be accessed via jumphost/ssh tunnel. But even after
> > creating ssh tunnel we are not able to read because once conusmer fetches
> > metadata it uses original hosts to connect to broker. Is it possible to
> > stop this behaviour?
> >
> > Thanks,
> > Akshay Das
> >
>

Re: Kafka SSH Tunnel Connection without editing hostfile

Posted by "M. Manna" <ma...@gmail.com>.
Have you tried using EXTERNAL endpoints for your Kafka broker to separate
TLS from internal traffic? Also, have you checked zk admin whether the
broker metadata is exposing your TLS endpoints to clients ?


On Thu, 12 Sep 2019 at 10:23, Akshay Das <ak...@fivetran.com.invalid>
wrote:

> Hi Team,
>
> I'm trying to consume from a kafka cluster using java client, but the kafka
> server can only be accessed via jumphost/ssh tunnel. But even after
> creating ssh tunnel we are not able to read because once conusmer fetches
> metadata it uses original hosts to connect to broker. Is it possible to
> stop this behaviour?
>
> Thanks,
> Akshay Das
>