You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by Roger Hoover <ro...@gmail.com> on 2013/10/18 00:13:02 UTC
Broker bind address versus published hostname in ZooKeeper
Hi all,
I'm getting started experimenting with Kafka and ran into a configuration
issue.
Currently, in server.properties, you can configure host.name which gets
used for two purposes: 1) to bind the socket 2) to publish the broker
details to ZK for clients to use.
There are times when these two settings need to be different. Here's an
example. I want to setup Kafka brokers on OpenStack virtual machines in a
private cloud but I need producers to connect from elsewhere on the
internal corporate network. With OpenStack, the virtual machines are only
exposed to DHCP addresses (typically RFC 1918 private addresses). You can
assign "floating ips" to a virtual machine but it's forwarded using Network
Address Translation and not exposed directly to the VM. Also, there's
typically no DNS to provide hostname lookup. Hosts have names like
"fubar.novalocal" that are not externally routable.
Here's what I want. I want the broker to bind to the VM's private network
IP but I want it to publish it's floating IP to ZooKeeper so that producers
can publish to it.
I propose a new optional parameter, "listen", which would allow you to
specify the socket address to listen on. If not set, the parameter would
default to host.name, which is the current behavior.
#Publish the externally routable IP in ZK
host.name = <floating ip>
#Accept connections from any interface the VM knows about
listen = *
I'm assuming others will eventually have the same requirement so I've added
a JIRA ticket.
https://issues.apache.org/jira/browse/KAFKA-1092
Thanks for your consideration.
Cheers,
Roger
Re: Broker bind address versus published hostname in ZooKeeper
Posted by Roger Hoover <ro...@gmail.com>.
OK. I've submitted a patch along with unit tests. This change is backward
compatible with two new optional parameters called advertise.host.name and
advertise.port. The reason I called them that instead of zkHost.name and
zkHost.port is that the zkHost.* names sound like they represent the host
and port of a ZooKeeper server rather than the host and port info of the
broker that gets published to ZK.
# Hostname the broker will advertise to producers and consumers. If not
set, it uses the
# value for "host.name" if configured. Otherwise, it will use the value
returned from
# java.net.InetAddress.getCanonicalHostName().
#advertise.host.name=<hostname routable by clients>
# The port to publish to ZooKeeper for clients to use. If this is not set,
# it will publish the same port that the broker binds to.
#advertise.port=<port accessible by clients>
https://issues.apache.org/jira/browse/KAFKA-1092
Please take a look and let me know if anything else is needed.
Cheers,
Roger
On Fri, Oct 25, 2013 at 4:05 PM, Roger Hoover <ro...@gmail.com>wrote:
> Ok. I'm working on it.
>
>
> On Thu, Oct 24, 2013 at 10:02 AM, Timothy Chen <tn...@gmail.com> wrote:
>
>> Hi Folks/Roger,
>>
>> Unfortunately I don't have legal clearance to contribute patches yet back
>> to Kafka for code done at work, so Roger it will be great if you can
>> provide this patch.
>>
>> Thanks!
>>
>> Tim
>>
>>
>>
>>
>> On Mon, Oct 21, 2013 at 11:17 AM, Roger Hoover <roger.hoover@gmail.com
>> >wrote:
>>
>> > Agreed. Tim, it would be very helpful is you could provide a patch.
>> > Otherwise, I may be willing to create one.
>> >
>> >
>> > On Thu, Oct 17, 2013 at 8:15 PM, Jun Rao <ju...@gmail.com> wrote:
>> >
>> > > Tim,
>> > >
>> > > This seems like a reasonable requirement. Would you be interested in
>> > > providing a patch to the jira?
>> > >
>> > > Thanks,
>> > >
>> > > Jun
>> > >
>> > >
>> > > On Thu, Oct 17, 2013 at 3:20 PM, Timothy Chen <tn...@gmail.com>
>> wrote:
>> > >
>> > > > Hi Roger,
>> > > >
>> > > > That's exactly what I need in my end, and actually internally
>> created a
>> > > new
>> > > > property called zkHost.name to publish a different host to zk. This
>> is
>> > > also
>> > > > needed for deploying Kafka into Azure.
>> > > >
>> > > > I also created zkHost.port since the internal and external ports
>> that's
>> > > > exposed might be different as well.
>> > > >
>> > > > Tim
>> > > >
>> > > >
>> > > > On Thu, Oct 17, 2013 at 3:13 PM, Roger Hoover <
>> roger.hoover@gmail.com
>> > > > >wrote:
>> > > >
>> > > > > Hi all,
>> > > > >
>> > > > > I'm getting started experimenting with Kafka and ran into a
>> > > configuration
>> > > > > issue.
>> > > > >
>> > > > > Currently, in server.properties, you can configure host.namewhich
>> > > gets
>> > > > > used for two purposes: 1) to bind the socket 2) to publish the
>> broker
>> > > > > details to ZK for clients to use.
>> > > > >
>> > > > > There are times when these two settings need to be different.
>> Here's
>> > an
>> > > > > example. I want to setup Kafka brokers on OpenStack virtual
>> machines
>> > > in a
>> > > > > private cloud but I need producers to connect from elsewhere on
>> the
>> > > > > internal corporate network. With OpenStack, the virtual machines
>> are
>> > > only
>> > > > > exposed to DHCP addresses (typically RFC 1918 private addresses).
>> You
>> > > can
>> > > > > assign "floating ips" to a virtual machine but it's forwarded
>> using
>> > > > Network
>> > > > > Address Translation and not exposed directly to the VM. Also,
>> there's
>> > > > > typically no DNS to provide hostname lookup. Hosts have names like
>> > > > > "fubar.novalocal" that are not externally routable.
>> > > > >
>> > > > > Here's what I want. I want the broker to bind to the VM's private
>> > > network
>> > > > > IP but I want it to publish it's floating IP to ZooKeeper so that
>> > > > producers
>> > > > > can publish to it.
>> > > > >
>> > > > > I propose a new optional parameter, "listen", which would allow
>> you
>> > to
>> > > > > specify the socket address to listen on. If not set, the parameter
>> > > would
>> > > > > default to host.name, which is the current behavior.
>> > > > >
>> > > > > #Publish the externally routable IP in ZK
>> > > > > host.name = <floating ip>
>> > > > > #Accept connections from any interface the VM knows about
>> > > > > listen = *
>> > > > >
>> > > > > I'm assuming others will eventually have the same requirement so
>> I've
>> > > > added
>> > > > > a JIRA ticket.
>> > > > >
>> > > > > https://issues.apache.org/jira/browse/KAFKA-1092
>> > > > >
>> > > > > Thanks for your consideration.
>> > > > >
>> > > > > Cheers,
>> > > > >
>> > > > > Roger
>> > > > >
>> > > >
>> > >
>> >
>>
>
>
Re: Broker bind address versus published hostname in ZooKeeper
Posted by Roger Hoover <ro...@gmail.com>.
Ok. I'm working on it.
On Thu, Oct 24, 2013 at 10:02 AM, Timothy Chen <tn...@gmail.com> wrote:
> Hi Folks/Roger,
>
> Unfortunately I don't have legal clearance to contribute patches yet back
> to Kafka for code done at work, so Roger it will be great if you can
> provide this patch.
>
> Thanks!
>
> Tim
>
>
>
>
> On Mon, Oct 21, 2013 at 11:17 AM, Roger Hoover <roger.hoover@gmail.com
> >wrote:
>
> > Agreed. Tim, it would be very helpful is you could provide a patch.
> > Otherwise, I may be willing to create one.
> >
> >
> > On Thu, Oct 17, 2013 at 8:15 PM, Jun Rao <ju...@gmail.com> wrote:
> >
> > > Tim,
> > >
> > > This seems like a reasonable requirement. Would you be interested in
> > > providing a patch to the jira?
> > >
> > > Thanks,
> > >
> > > Jun
> > >
> > >
> > > On Thu, Oct 17, 2013 at 3:20 PM, Timothy Chen <tn...@gmail.com>
> wrote:
> > >
> > > > Hi Roger,
> > > >
> > > > That's exactly what I need in my end, and actually internally
> created a
> > > new
> > > > property called zkHost.name to publish a different host to zk. This
> is
> > > also
> > > > needed for deploying Kafka into Azure.
> > > >
> > > > I also created zkHost.port since the internal and external ports
> that's
> > > > exposed might be different as well.
> > > >
> > > > Tim
> > > >
> > > >
> > > > On Thu, Oct 17, 2013 at 3:13 PM, Roger Hoover <
> roger.hoover@gmail.com
> > > > >wrote:
> > > >
> > > > > Hi all,
> > > > >
> > > > > I'm getting started experimenting with Kafka and ran into a
> > > configuration
> > > > > issue.
> > > > >
> > > > > Currently, in server.properties, you can configure host.name which
> > > gets
> > > > > used for two purposes: 1) to bind the socket 2) to publish the
> broker
> > > > > details to ZK for clients to use.
> > > > >
> > > > > There are times when these two settings need to be different.
> Here's
> > an
> > > > > example. I want to setup Kafka brokers on OpenStack virtual
> machines
> > > in a
> > > > > private cloud but I need producers to connect from elsewhere on the
> > > > > internal corporate network. With OpenStack, the virtual machines
> are
> > > only
> > > > > exposed to DHCP addresses (typically RFC 1918 private addresses).
> You
> > > can
> > > > > assign "floating ips" to a virtual machine but it's forwarded using
> > > > Network
> > > > > Address Translation and not exposed directly to the VM. Also,
> there's
> > > > > typically no DNS to provide hostname lookup. Hosts have names like
> > > > > "fubar.novalocal" that are not externally routable.
> > > > >
> > > > > Here's what I want. I want the broker to bind to the VM's private
> > > network
> > > > > IP but I want it to publish it's floating IP to ZooKeeper so that
> > > > producers
> > > > > can publish to it.
> > > > >
> > > > > I propose a new optional parameter, "listen", which would allow you
> > to
> > > > > specify the socket address to listen on. If not set, the parameter
> > > would
> > > > > default to host.name, which is the current behavior.
> > > > >
> > > > > #Publish the externally routable IP in ZK
> > > > > host.name = <floating ip>
> > > > > #Accept connections from any interface the VM knows about
> > > > > listen = *
> > > > >
> > > > > I'm assuming others will eventually have the same requirement so
> I've
> > > > added
> > > > > a JIRA ticket.
> > > > >
> > > > > https://issues.apache.org/jira/browse/KAFKA-1092
> > > > >
> > > > > Thanks for your consideration.
> > > > >
> > > > > Cheers,
> > > > >
> > > > > Roger
> > > > >
> > > >
> > >
> >
>
Re: Broker bind address versus published hostname in ZooKeeper
Posted by Timothy Chen <tn...@gmail.com>.
Hi Folks/Roger,
Unfortunately I don't have legal clearance to contribute patches yet back
to Kafka for code done at work, so Roger it will be great if you can
provide this patch.
Thanks!
Tim
On Mon, Oct 21, 2013 at 11:17 AM, Roger Hoover <ro...@gmail.com>wrote:
> Agreed. Tim, it would be very helpful is you could provide a patch.
> Otherwise, I may be willing to create one.
>
>
> On Thu, Oct 17, 2013 at 8:15 PM, Jun Rao <ju...@gmail.com> wrote:
>
> > Tim,
> >
> > This seems like a reasonable requirement. Would you be interested in
> > providing a patch to the jira?
> >
> > Thanks,
> >
> > Jun
> >
> >
> > On Thu, Oct 17, 2013 at 3:20 PM, Timothy Chen <tn...@gmail.com> wrote:
> >
> > > Hi Roger,
> > >
> > > That's exactly what I need in my end, and actually internally created a
> > new
> > > property called zkHost.name to publish a different host to zk. This is
> > also
> > > needed for deploying Kafka into Azure.
> > >
> > > I also created zkHost.port since the internal and external ports that's
> > > exposed might be different as well.
> > >
> > > Tim
> > >
> > >
> > > On Thu, Oct 17, 2013 at 3:13 PM, Roger Hoover <roger.hoover@gmail.com
> > > >wrote:
> > >
> > > > Hi all,
> > > >
> > > > I'm getting started experimenting with Kafka and ran into a
> > configuration
> > > > issue.
> > > >
> > > > Currently, in server.properties, you can configure host.name which
> > gets
> > > > used for two purposes: 1) to bind the socket 2) to publish the broker
> > > > details to ZK for clients to use.
> > > >
> > > > There are times when these two settings need to be different. Here's
> an
> > > > example. I want to setup Kafka brokers on OpenStack virtual machines
> > in a
> > > > private cloud but I need producers to connect from elsewhere on the
> > > > internal corporate network. With OpenStack, the virtual machines are
> > only
> > > > exposed to DHCP addresses (typically RFC 1918 private addresses). You
> > can
> > > > assign "floating ips" to a virtual machine but it's forwarded using
> > > Network
> > > > Address Translation and not exposed directly to the VM. Also, there's
> > > > typically no DNS to provide hostname lookup. Hosts have names like
> > > > "fubar.novalocal" that are not externally routable.
> > > >
> > > > Here's what I want. I want the broker to bind to the VM's private
> > network
> > > > IP but I want it to publish it's floating IP to ZooKeeper so that
> > > producers
> > > > can publish to it.
> > > >
> > > > I propose a new optional parameter, "listen", which would allow you
> to
> > > > specify the socket address to listen on. If not set, the parameter
> > would
> > > > default to host.name, which is the current behavior.
> > > >
> > > > #Publish the externally routable IP in ZK
> > > > host.name = <floating ip>
> > > > #Accept connections from any interface the VM knows about
> > > > listen = *
> > > >
> > > > I'm assuming others will eventually have the same requirement so I've
> > > added
> > > > a JIRA ticket.
> > > >
> > > > https://issues.apache.org/jira/browse/KAFKA-1092
> > > >
> > > > Thanks for your consideration.
> > > >
> > > > Cheers,
> > > >
> > > > Roger
> > > >
> > >
> >
>
Re: Broker bind address versus published hostname in ZooKeeper
Posted by Roger Hoover <ro...@gmail.com>.
Agreed. Tim, it would be very helpful is you could provide a patch.
Otherwise, I may be willing to create one.
On Thu, Oct 17, 2013 at 8:15 PM, Jun Rao <ju...@gmail.com> wrote:
> Tim,
>
> This seems like a reasonable requirement. Would you be interested in
> providing a patch to the jira?
>
> Thanks,
>
> Jun
>
>
> On Thu, Oct 17, 2013 at 3:20 PM, Timothy Chen <tn...@gmail.com> wrote:
>
> > Hi Roger,
> >
> > That's exactly what I need in my end, and actually internally created a
> new
> > property called zkHost.name to publish a different host to zk. This is
> also
> > needed for deploying Kafka into Azure.
> >
> > I also created zkHost.port since the internal and external ports that's
> > exposed might be different as well.
> >
> > Tim
> >
> >
> > On Thu, Oct 17, 2013 at 3:13 PM, Roger Hoover <roger.hoover@gmail.com
> > >wrote:
> >
> > > Hi all,
> > >
> > > I'm getting started experimenting with Kafka and ran into a
> configuration
> > > issue.
> > >
> > > Currently, in server.properties, you can configure host.name which
> gets
> > > used for two purposes: 1) to bind the socket 2) to publish the broker
> > > details to ZK for clients to use.
> > >
> > > There are times when these two settings need to be different. Here's an
> > > example. I want to setup Kafka brokers on OpenStack virtual machines
> in a
> > > private cloud but I need producers to connect from elsewhere on the
> > > internal corporate network. With OpenStack, the virtual machines are
> only
> > > exposed to DHCP addresses (typically RFC 1918 private addresses). You
> can
> > > assign "floating ips" to a virtual machine but it's forwarded using
> > Network
> > > Address Translation and not exposed directly to the VM. Also, there's
> > > typically no DNS to provide hostname lookup. Hosts have names like
> > > "fubar.novalocal" that are not externally routable.
> > >
> > > Here's what I want. I want the broker to bind to the VM's private
> network
> > > IP but I want it to publish it's floating IP to ZooKeeper so that
> > producers
> > > can publish to it.
> > >
> > > I propose a new optional parameter, "listen", which would allow you to
> > > specify the socket address to listen on. If not set, the parameter
> would
> > > default to host.name, which is the current behavior.
> > >
> > > #Publish the externally routable IP in ZK
> > > host.name = <floating ip>
> > > #Accept connections from any interface the VM knows about
> > > listen = *
> > >
> > > I'm assuming others will eventually have the same requirement so I've
> > added
> > > a JIRA ticket.
> > >
> > > https://issues.apache.org/jira/browse/KAFKA-1092
> > >
> > > Thanks for your consideration.
> > >
> > > Cheers,
> > >
> > > Roger
> > >
> >
>
Re: Broker bind address versus published hostname in ZooKeeper
Posted by Jun Rao <ju...@gmail.com>.
Tim,
This seems like a reasonable requirement. Would you be interested in
providing a patch to the jira?
Thanks,
Jun
On Thu, Oct 17, 2013 at 3:20 PM, Timothy Chen <tn...@gmail.com> wrote:
> Hi Roger,
>
> That's exactly what I need in my end, and actually internally created a new
> property called zkHost.name to publish a different host to zk. This is also
> needed for deploying Kafka into Azure.
>
> I also created zkHost.port since the internal and external ports that's
> exposed might be different as well.
>
> Tim
>
>
> On Thu, Oct 17, 2013 at 3:13 PM, Roger Hoover <roger.hoover@gmail.com
> >wrote:
>
> > Hi all,
> >
> > I'm getting started experimenting with Kafka and ran into a configuration
> > issue.
> >
> > Currently, in server.properties, you can configure host.name which gets
> > used for two purposes: 1) to bind the socket 2) to publish the broker
> > details to ZK for clients to use.
> >
> > There are times when these two settings need to be different. Here's an
> > example. I want to setup Kafka brokers on OpenStack virtual machines in a
> > private cloud but I need producers to connect from elsewhere on the
> > internal corporate network. With OpenStack, the virtual machines are only
> > exposed to DHCP addresses (typically RFC 1918 private addresses). You can
> > assign "floating ips" to a virtual machine but it's forwarded using
> Network
> > Address Translation and not exposed directly to the VM. Also, there's
> > typically no DNS to provide hostname lookup. Hosts have names like
> > "fubar.novalocal" that are not externally routable.
> >
> > Here's what I want. I want the broker to bind to the VM's private network
> > IP but I want it to publish it's floating IP to ZooKeeper so that
> producers
> > can publish to it.
> >
> > I propose a new optional parameter, "listen", which would allow you to
> > specify the socket address to listen on. If not set, the parameter would
> > default to host.name, which is the current behavior.
> >
> > #Publish the externally routable IP in ZK
> > host.name = <floating ip>
> > #Accept connections from any interface the VM knows about
> > listen = *
> >
> > I'm assuming others will eventually have the same requirement so I've
> added
> > a JIRA ticket.
> >
> > https://issues.apache.org/jira/browse/KAFKA-1092
> >
> > Thanks for your consideration.
> >
> > Cheers,
> >
> > Roger
> >
>
Re: Broker bind address versus published hostname in ZooKeeper
Posted by Timothy Chen <tn...@gmail.com>.
Hi Roger,
That's exactly what I need in my end, and actually internally created a new
property called zkHost.name to publish a different host to zk. This is also
needed for deploying Kafka into Azure.
I also created zkHost.port since the internal and external ports that's
exposed might be different as well.
Tim
On Thu, Oct 17, 2013 at 3:13 PM, Roger Hoover <ro...@gmail.com>wrote:
> Hi all,
>
> I'm getting started experimenting with Kafka and ran into a configuration
> issue.
>
> Currently, in server.properties, you can configure host.name which gets
> used for two purposes: 1) to bind the socket 2) to publish the broker
> details to ZK for clients to use.
>
> There are times when these two settings need to be different. Here's an
> example. I want to setup Kafka brokers on OpenStack virtual machines in a
> private cloud but I need producers to connect from elsewhere on the
> internal corporate network. With OpenStack, the virtual machines are only
> exposed to DHCP addresses (typically RFC 1918 private addresses). You can
> assign "floating ips" to a virtual machine but it's forwarded using Network
> Address Translation and not exposed directly to the VM. Also, there's
> typically no DNS to provide hostname lookup. Hosts have names like
> "fubar.novalocal" that are not externally routable.
>
> Here's what I want. I want the broker to bind to the VM's private network
> IP but I want it to publish it's floating IP to ZooKeeper so that producers
> can publish to it.
>
> I propose a new optional parameter, "listen", which would allow you to
> specify the socket address to listen on. If not set, the parameter would
> default to host.name, which is the current behavior.
>
> #Publish the externally routable IP in ZK
> host.name = <floating ip>
> #Accept connections from any interface the VM knows about
> listen = *
>
> I'm assuming others will eventually have the same requirement so I've added
> a JIRA ticket.
>
> https://issues.apache.org/jira/browse/KAFKA-1092
>
> Thanks for your consideration.
>
> Cheers,
>
> Roger
>