You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by Su She <su...@gmail.com> on 2015/07/07 20:09:20 UTC
Trying to access Kafka behind NAT, java.io.BrokenPipe error
Hello Everyone,
I am in a tricky situation.
I am running Kafka (downloaded via binary) on a private cloud behind NAT. I
am hoping to send messages to the broker from outside the cloud. Everything
else seems to be working fine, I can send messages from instances within
the private cloud. The instance has a private ip, but no public ip.
So public IP ip addr A:8081 is mapped through PFSense NAT Rules to a
private ip B:80
Here are some of the configurations I have tried:
server.properties:
port =80 //this works fine locally, no other PID is using htis port
advertised.host.name =public IP addrA //causes java.io.BrokenPipe Error,
have also tried the private ip
advertised.port =8081
I am not sure what else I can change in server.properties. Has anyone else
tried to do this? Is this possible? Or does the broker need to be attached
to its own public ip.
Thank you for the help!
-Su
Re: Trying to access Kafka behind NAT, java.io.BrokenPipe error
Posted by Hawin Jiang <ha...@gmail.com>.
Hi Su
Maybe you can try to use own public IP if you want to set host name to
public IP.
Or you can use private IP with load balancing configuration.
Thanks.
Best regards
Hawin
On Tue, Jul 7, 2015 at 11:09 AM, Su She <su...@gmail.com> wrote:
> Hello Everyone,
>
> I am in a tricky situation.
>
> I am running Kafka (downloaded via binary) on a private cloud behind NAT. I
> am hoping to send messages to the broker from outside the cloud. Everything
> else seems to be working fine, I can send messages from instances within
> the private cloud. The instance has a private ip, but no public ip.
>
> So public IP ip addr A:8081 is mapped through PFSense NAT Rules to a
> private ip B:80
>
> Here are some of the configurations I have tried:
>
> server.properties:
>
> port =80 //this works fine locally, no other PID is using htis port
>
> advertised.host.name =public IP addrA //causes java.io.BrokenPipe Error,
> have also tried the private ip
>
> advertised.port =8081
>
> I am not sure what else I can change in server.properties. Has anyone else
> tried to do this? Is this possible? Or does the broker need to be attached
> to its own public ip.
>
> Thank you for the help!
>
> -Su
>
Re: Trying to access Kafka behind NAT, java.io.BrokenPipe error
Posted by Mike Bridge <mi...@bridgecanada.com>.
If it's the case that zookeeper can't access the brokers at their
advertised addresses (and it's just a guess---you could test it by
telnetting from zookeeper to your broker's externalip+port), you might be
able to get away with adding a route so that the traffic avoids going out
to your firewall.
I set Kafka up on two separate NAT networks and that was the default
security setting for both....
It would be nice to have two separate settings---one that Zookeeper uses to
advertise the brokers, and another that it uses to contact the brokers.
On Tue, Jul 7, 2015 at 6:33 PM, Su She <su...@gmail.com> wrote:
> Thank you Hawin and Mike for the answers.
>
> Having a public IP would be the easiest solution, but unfortunately, this
> is a lengthy process for me due to company security rules.
>
> Since the zookeeper is behind the NAT, I guess it won't have access to the
> external IP...so it seems like there isn't a straight-forward solution.
>
> I guess I will see if I can produce my messages behind the NAT while I try
> to get an external IP.
>
> Thanks!
>
> -Su
>
> On Tue, Jul 7, 2015 at 3:13 PM, Mike Bridge <mi...@bridgecanada.com> wrote:
>
> > can zookeeper access the broker through the public IP?
> > advertised.host.name
> > is also what zookeeper uses to communicate to the broker, but if
> zookeeper
> > is inside the NAT it might not have a route via that external IP.
> > On Jul 7, 2015 2:10 PM, "Su She" <su...@gmail.com> wrote:
> >
> > > Hello Everyone,
> > >
> > > I am in a tricky situation.
> > >
> > > I am running Kafka (downloaded via binary) on a private cloud behind
> > NAT. I
> > > am hoping to send messages to the broker from outside the cloud.
> > Everything
> > > else seems to be working fine, I can send messages from instances
> within
> > > the private cloud. The instance has a private ip, but no public ip.
> > >
> > > So public IP ip addr A:8081 is mapped through PFSense NAT Rules to a
> > > private ip B:80
> > >
> > > Here are some of the configurations I have tried:
> > >
> > > server.properties:
> > >
> > > port =80 //this works fine locally, no other PID is using htis port
> > >
> > > advertised.host.name =public IP addrA //causes java.io.BrokenPipe
> Error,
> > > have also tried the private ip
> > >
> > > advertised.port =8081
> > >
> > > I am not sure what else I can change in server.properties. Has anyone
> > else
> > > tried to do this? Is this possible? Or does the broker need to be
> > attached
> > > to its own public ip.
> > >
> > > Thank you for the help!
> > >
> > > -Su
> > >
> >
>
Re: Trying to access Kafka behind NAT, java.io.BrokenPipe error
Posted by Su She <su...@gmail.com>.
Quick question,
so is there any way to set the advertised host name differently for the
zookeeper and clients?
Thanks,
Su
On Tue, Jul 7, 2015 at 5:33 PM, Su She <su...@gmail.com> wrote:
> Thank you Hawin and Mike for the answers.
>
> Having a public IP would be the easiest solution, but unfortunately, this
> is a lengthy process for me due to company security rules.
>
> Since the zookeeper is behind the NAT, I guess it won't have access to the
> external IP...so it seems like there isn't a straight-forward solution.
>
> I guess I will see if I can produce my messages behind the NAT while I
> try to get an external IP.
>
> Thanks!
>
> -Su
>
> On Tue, Jul 7, 2015 at 3:13 PM, Mike Bridge <mi...@bridgecanada.com> wrote:
>
>> can zookeeper access the broker through the public IP?
>> advertised.host.name
>> is also what zookeeper uses to communicate to the broker, but if zookeeper
>> is inside the NAT it might not have a route via that external IP.
>> On Jul 7, 2015 2:10 PM, "Su She" <su...@gmail.com> wrote:
>>
>> > Hello Everyone,
>> >
>> > I am in a tricky situation.
>> >
>> > I am running Kafka (downloaded via binary) on a private cloud behind
>> NAT. I
>> > am hoping to send messages to the broker from outside the cloud.
>> Everything
>> > else seems to be working fine, I can send messages from instances within
>> > the private cloud. The instance has a private ip, but no public ip.
>> >
>> > So public IP ip addr A:8081 is mapped through PFSense NAT Rules to a
>> > private ip B:80
>> >
>> > Here are some of the configurations I have tried:
>> >
>> > server.properties:
>> >
>> > port =80 //this works fine locally, no other PID is using htis port
>> >
>> > advertised.host.name =public IP addrA //causes java.io.BrokenPipe
>> Error,
>> > have also tried the private ip
>> >
>> > advertised.port =8081
>> >
>> > I am not sure what else I can change in server.properties. Has anyone
>> else
>> > tried to do this? Is this possible? Or does the broker need to be
>> attached
>> > to its own public ip.
>> >
>> > Thank you for the help!
>> >
>> > -Su
>> >
>>
>
>
Re: Trying to access Kafka behind NAT, java.io.BrokenPipe error
Posted by Su She <su...@gmail.com>.
Thank you Hawin and Mike for the answers.
Having a public IP would be the easiest solution, but unfortunately, this
is a lengthy process for me due to company security rules.
Since the zookeeper is behind the NAT, I guess it won't have access to the
external IP...so it seems like there isn't a straight-forward solution.
I guess I will see if I can produce my messages behind the NAT while I try
to get an external IP.
Thanks!
-Su
On Tue, Jul 7, 2015 at 3:13 PM, Mike Bridge <mi...@bridgecanada.com> wrote:
> can zookeeper access the broker through the public IP?
> advertised.host.name
> is also what zookeeper uses to communicate to the broker, but if zookeeper
> is inside the NAT it might not have a route via that external IP.
> On Jul 7, 2015 2:10 PM, "Su She" <su...@gmail.com> wrote:
>
> > Hello Everyone,
> >
> > I am in a tricky situation.
> >
> > I am running Kafka (downloaded via binary) on a private cloud behind
> NAT. I
> > am hoping to send messages to the broker from outside the cloud.
> Everything
> > else seems to be working fine, I can send messages from instances within
> > the private cloud. The instance has a private ip, but no public ip.
> >
> > So public IP ip addr A:8081 is mapped through PFSense NAT Rules to a
> > private ip B:80
> >
> > Here are some of the configurations I have tried:
> >
> > server.properties:
> >
> > port =80 //this works fine locally, no other PID is using htis port
> >
> > advertised.host.name =public IP addrA //causes java.io.BrokenPipe Error,
> > have also tried the private ip
> >
> > advertised.port =8081
> >
> > I am not sure what else I can change in server.properties. Has anyone
> else
> > tried to do this? Is this possible? Or does the broker need to be
> attached
> > to its own public ip.
> >
> > Thank you for the help!
> >
> > -Su
> >
>
Re: Trying to access Kafka behind NAT, java.io.BrokenPipe error
Posted by Mike Bridge <mi...@bridgecanada.com>.
can zookeeper access the broker through the public IP? advertised.host.name
is also what zookeeper uses to communicate to the broker, but if zookeeper
is inside the NAT it might not have a route via that external IP.
On Jul 7, 2015 2:10 PM, "Su She" <su...@gmail.com> wrote:
> Hello Everyone,
>
> I am in a tricky situation.
>
> I am running Kafka (downloaded via binary) on a private cloud behind NAT. I
> am hoping to send messages to the broker from outside the cloud. Everything
> else seems to be working fine, I can send messages from instances within
> the private cloud. The instance has a private ip, but no public ip.
>
> So public IP ip addr A:8081 is mapped through PFSense NAT Rules to a
> private ip B:80
>
> Here are some of the configurations I have tried:
>
> server.properties:
>
> port =80 //this works fine locally, no other PID is using htis port
>
> advertised.host.name =public IP addrA //causes java.io.BrokenPipe Error,
> have also tried the private ip
>
> advertised.port =8081
>
> I am not sure what else I can change in server.properties. Has anyone else
> tried to do this? Is this possible? Or does the broker need to be attached
> to its own public ip.
>
> Thank you for the help!
>
> -Su
>