You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by Achanta Vamsi Subhash <ac...@flipkart.com> on 2015/05/22 13:20:45 UTC
Kafka broker - Ip-address instead of host naem
Hi,
Currently Kakfa brokers register the hostname in zookeeper.
[zk: localhost:2181(CONNECTED) 5] get /varadhi/kafka/brokers/ids/0
{"jmx_port":9999,"timestamp":"1427704934158","host":"currHostName","version":1,"port":9092}
Is there any config to make it use ip-address instead so that we don't
make a DNS lookup for the hostname?
--
Regards
Vamsi Subhash
Re: Kafka broker - Ip-address instead of host naem
Posted by Jason Rosenberg <jb...@squareup.com>.
Daniel,
Are you sure about that (it's not what I would have understood).
Generally, the way to do it is use a round-robin dns entry, which returns
successive nodes for successive requests. Kafka will retry a zookeeper
request on failure (and in the process get re-routed to a new zk node). If
a zk node fails, and a new one comes online, then it is necessary to update
the dns config of course.
If you don't have dns, then you can list your nodes explicitly, but you
won't be able to discover new nodes in the cluster if they are added (but
kafka will retry if it tries the bad one until it finds a good one).
Jason
On Mon, May 25, 2015 at 1:51 AM, Achanta Vamsi Subhash <
achanta.vamsi@flipkart.com> wrote:
> Oh ok. Got it. Thanks Gwen and Daniel.
>
> On Mon, May 25, 2015 at 5:15 AM, Daniel Compton <
> daniel.compton.lists@gmail.com> wrote:
>
> > Correct me if I'm wrong, but I thought that the zk connect config was
> only
> > needed explicitly at startup, and the zk cluster would update the active
> > zookeepers as they joined and exited. You only need to specify one zk to
> > join the cluster, and it will bootstrap the rest.
> >
> > So zk changes won't require a restart, but you'll want to make sure your
> > config is up to date when you do eventually come to do a restart.
> > On Mon, 25 May 2015 at 1:44 am Achanta Vamsi Subhash <
> > achanta.vamsi@flipkart.com> wrote:
> >
> > > Ok. I was talking about a scenario where there is no DNS/hostNames for
> > the
> > > zk nodes.
> > >
> > > If the connection string is given with all ip addresess of zk hosts
> and a
> > > new host is brought up in the zk cluster replacing a old node with a
> > > different ip address, we still need to re-load the zk connection
> string.
> > >
> > > If hostnames are used instead, the DNS mapping can point to the new IP
> > but
> > > in a scenario where there is no DNS, we need to again hard-code the IP
> > > address in the zk connection string and restart the broker.
> > >
> > > One way is to give the local mapping of the zk local hostname to ip in
> > > /etc/hosts file and change it to the new ip when the node changes. But
> > > would reload of the Kafka config with new zk nodes be a better option?
> > But
> > > as you said, if we cannot reload the server.properties file, what is
> the
> > > best way in case of no service discovery?
> > >
> > >
> > > On Sun, May 24, 2015 at 6:52 PM, Gwen Shapira <gs...@cloudera.com>
> > > wrote:
> > >
> > > > You can't dynamically re-load server properties.
> > > >
> > > > However, the norm in zookeeper is to configure the connection string
> > with
> > > > all the nodes in the zk cluster, so there will be no need to modify
> > > > properties when you replace zk nodes.
> > > >
> > > > On Sun, May 24, 2015 at 4:13 PM, Achanta Vamsi Subhash <
> > > > achanta.vamsi@flipkart.com> wrote:
> > > >
> > > > > Thanks Gwen.
> > > > >
> > > > > One more question:
> > > > > Is there a way we can dynamically re-load the contents of the Kafka
> > > > > server.properties contents without restarting the broker? Example
> > > > use-case
> > > > > is when a zk-node goes down and a new one is brought up, we need to
> > > > update
> > > > > the server.properties file to reflect this. Currently there is no
> way
> > > to
> > > > do
> > > > > this other than broker restart. Broker restart requires a lot of
> > things
> > > > to
> > > > > do before triggering it.
> > > > >
> > > > > This JIRA is already filed but un-resolved. We don't require all
> the
> > > > > configs to be reloaded but only variable external config changes
> > should
> > > > be
> > > > > allowed.
> > > > >
> > > > > https://issues.apache.org/jira/browse/KAFKA-1229
> > > > >
> > > > > On Sun, May 24, 2015 at 1:14 PM, Gwen Shapira <
> gshapira@cloudera.com
> > >
> > > > > wrote:
> > > > >
> > > > > > If you set advertised.hostname in server.properties to the ip
> > > address,
> > > > > the
> > > > > > IP will be registered in ZooKeeper.
> > > > > >
> > > > > >
> > > > > > On Fri, May 22, 2015 at 2:20 PM, Achanta Vamsi Subhash <
> > > > > > achanta.vamsi@flipkart.com> wrote:
> > > > > >
> > > > > > > Hi,
> > > > > > >
> > > > > > > Currently Kakfa brokers register the hostname in zookeeper.
> > > > > > >
> > > > > > > [zk: localhost:2181(CONNECTED) 5] get
> > /varadhi/kafka/brokers/ids/0
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> {"jmx_port":9999,"timestamp":"1427704934158","host":"currHostName","version":1,"port":9092}
> > > > > > >
> > > > > > > Is there any config to make it use ip-address instead so that
> we
> > > > don't
> > > > > > > make a DNS lookup for the hostname?
> > > > > > >
> > > > > > > --
> > > > > > > Regards
> > > > > > > Vamsi Subhash
> > > > > > >
> > > > > >
> > > > >
> > > > >
> > > > >
> > > > > --
> > > > > Regards
> > > > > Vamsi Subhash
> > > > >
> > > >
> > >
> > >
> > >
> > > --
> > > Regards
> > > Vamsi Subhash
> > >
> >
>
>
>
> --
> Regards
> Vamsi Subhash
>
Re: Kafka broker - Ip-address instead of host naem
Posted by Achanta Vamsi Subhash <ac...@flipkart.com>.
Oh ok. Got it. Thanks Gwen and Daniel.
On Mon, May 25, 2015 at 5:15 AM, Daniel Compton <
daniel.compton.lists@gmail.com> wrote:
> Correct me if I'm wrong, but I thought that the zk connect config was only
> needed explicitly at startup, and the zk cluster would update the active
> zookeepers as they joined and exited. You only need to specify one zk to
> join the cluster, and it will bootstrap the rest.
>
> So zk changes won't require a restart, but you'll want to make sure your
> config is up to date when you do eventually come to do a restart.
> On Mon, 25 May 2015 at 1:44 am Achanta Vamsi Subhash <
> achanta.vamsi@flipkart.com> wrote:
>
> > Ok. I was talking about a scenario where there is no DNS/hostNames for
> the
> > zk nodes.
> >
> > If the connection string is given with all ip addresess of zk hosts and a
> > new host is brought up in the zk cluster replacing a old node with a
> > different ip address, we still need to re-load the zk connection string.
> >
> > If hostnames are used instead, the DNS mapping can point to the new IP
> but
> > in a scenario where there is no DNS, we need to again hard-code the IP
> > address in the zk connection string and restart the broker.
> >
> > One way is to give the local mapping of the zk local hostname to ip in
> > /etc/hosts file and change it to the new ip when the node changes. But
> > would reload of the Kafka config with new zk nodes be a better option?
> But
> > as you said, if we cannot reload the server.properties file, what is the
> > best way in case of no service discovery?
> >
> >
> > On Sun, May 24, 2015 at 6:52 PM, Gwen Shapira <gs...@cloudera.com>
> > wrote:
> >
> > > You can't dynamically re-load server properties.
> > >
> > > However, the norm in zookeeper is to configure the connection string
> with
> > > all the nodes in the zk cluster, so there will be no need to modify
> > > properties when you replace zk nodes.
> > >
> > > On Sun, May 24, 2015 at 4:13 PM, Achanta Vamsi Subhash <
> > > achanta.vamsi@flipkart.com> wrote:
> > >
> > > > Thanks Gwen.
> > > >
> > > > One more question:
> > > > Is there a way we can dynamically re-load the contents of the Kafka
> > > > server.properties contents without restarting the broker? Example
> > > use-case
> > > > is when a zk-node goes down and a new one is brought up, we need to
> > > update
> > > > the server.properties file to reflect this. Currently there is no way
> > to
> > > do
> > > > this other than broker restart. Broker restart requires a lot of
> things
> > > to
> > > > do before triggering it.
> > > >
> > > > This JIRA is already filed but un-resolved. We don't require all the
> > > > configs to be reloaded but only variable external config changes
> should
> > > be
> > > > allowed.
> > > >
> > > > https://issues.apache.org/jira/browse/KAFKA-1229
> > > >
> > > > On Sun, May 24, 2015 at 1:14 PM, Gwen Shapira <gshapira@cloudera.com
> >
> > > > wrote:
> > > >
> > > > > If you set advertised.hostname in server.properties to the ip
> > address,
> > > > the
> > > > > IP will be registered in ZooKeeper.
> > > > >
> > > > >
> > > > > On Fri, May 22, 2015 at 2:20 PM, Achanta Vamsi Subhash <
> > > > > achanta.vamsi@flipkart.com> wrote:
> > > > >
> > > > > > Hi,
> > > > > >
> > > > > > Currently Kakfa brokers register the hostname in zookeeper.
> > > > > >
> > > > > > [zk: localhost:2181(CONNECTED) 5] get
> /varadhi/kafka/brokers/ids/0
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> {"jmx_port":9999,"timestamp":"1427704934158","host":"currHostName","version":1,"port":9092}
> > > > > >
> > > > > > Is there any config to make it use ip-address instead so that we
> > > don't
> > > > > > make a DNS lookup for the hostname?
> > > > > >
> > > > > > --
> > > > > > Regards
> > > > > > Vamsi Subhash
> > > > > >
> > > > >
> > > >
> > > >
> > > >
> > > > --
> > > > Regards
> > > > Vamsi Subhash
> > > >
> > >
> >
> >
> >
> > --
> > Regards
> > Vamsi Subhash
> >
>
--
Regards
Vamsi Subhash
Re: Kafka broker - Ip-address instead of host naem
Posted by Daniel Compton <da...@gmail.com>.
Correct me if I'm wrong, but I thought that the zk connect config was only
needed explicitly at startup, and the zk cluster would update the active
zookeepers as they joined and exited. You only need to specify one zk to
join the cluster, and it will bootstrap the rest.
So zk changes won't require a restart, but you'll want to make sure your
config is up to date when you do eventually come to do a restart.
On Mon, 25 May 2015 at 1:44 am Achanta Vamsi Subhash <
achanta.vamsi@flipkart.com> wrote:
> Ok. I was talking about a scenario where there is no DNS/hostNames for the
> zk nodes.
>
> If the connection string is given with all ip addresess of zk hosts and a
> new host is brought up in the zk cluster replacing a old node with a
> different ip address, we still need to re-load the zk connection string.
>
> If hostnames are used instead, the DNS mapping can point to the new IP but
> in a scenario where there is no DNS, we need to again hard-code the IP
> address in the zk connection string and restart the broker.
>
> One way is to give the local mapping of the zk local hostname to ip in
> /etc/hosts file and change it to the new ip when the node changes. But
> would reload of the Kafka config with new zk nodes be a better option? But
> as you said, if we cannot reload the server.properties file, what is the
> best way in case of no service discovery?
>
>
> On Sun, May 24, 2015 at 6:52 PM, Gwen Shapira <gs...@cloudera.com>
> wrote:
>
> > You can't dynamically re-load server properties.
> >
> > However, the norm in zookeeper is to configure the connection string with
> > all the nodes in the zk cluster, so there will be no need to modify
> > properties when you replace zk nodes.
> >
> > On Sun, May 24, 2015 at 4:13 PM, Achanta Vamsi Subhash <
> > achanta.vamsi@flipkart.com> wrote:
> >
> > > Thanks Gwen.
> > >
> > > One more question:
> > > Is there a way we can dynamically re-load the contents of the Kafka
> > > server.properties contents without restarting the broker? Example
> > use-case
> > > is when a zk-node goes down and a new one is brought up, we need to
> > update
> > > the server.properties file to reflect this. Currently there is no way
> to
> > do
> > > this other than broker restart. Broker restart requires a lot of things
> > to
> > > do before triggering it.
> > >
> > > This JIRA is already filed but un-resolved. We don't require all the
> > > configs to be reloaded but only variable external config changes should
> > be
> > > allowed.
> > >
> > > https://issues.apache.org/jira/browse/KAFKA-1229
> > >
> > > On Sun, May 24, 2015 at 1:14 PM, Gwen Shapira <gs...@cloudera.com>
> > > wrote:
> > >
> > > > If you set advertised.hostname in server.properties to the ip
> address,
> > > the
> > > > IP will be registered in ZooKeeper.
> > > >
> > > >
> > > > On Fri, May 22, 2015 at 2:20 PM, Achanta Vamsi Subhash <
> > > > achanta.vamsi@flipkart.com> wrote:
> > > >
> > > > > Hi,
> > > > >
> > > > > Currently Kakfa brokers register the hostname in zookeeper.
> > > > >
> > > > > [zk: localhost:2181(CONNECTED) 5] get /varadhi/kafka/brokers/ids/0
> > > > >
> > > > >
> > > >
> > >
> >
> {"jmx_port":9999,"timestamp":"1427704934158","host":"currHostName","version":1,"port":9092}
> > > > >
> > > > > Is there any config to make it use ip-address instead so that we
> > don't
> > > > > make a DNS lookup for the hostname?
> > > > >
> > > > > --
> > > > > Regards
> > > > > Vamsi Subhash
> > > > >
> > > >
> > >
> > >
> > >
> > > --
> > > Regards
> > > Vamsi Subhash
> > >
> >
>
>
>
> --
> Regards
> Vamsi Subhash
>
Re: Kafka broker - Ip-address instead of host naem
Posted by Achanta Vamsi Subhash <ac...@flipkart.com>.
Ok. I was talking about a scenario where there is no DNS/hostNames for the
zk nodes.
If the connection string is given with all ip addresess of zk hosts and a
new host is brought up in the zk cluster replacing a old node with a
different ip address, we still need to re-load the zk connection string.
If hostnames are used instead, the DNS mapping can point to the new IP but
in a scenario where there is no DNS, we need to again hard-code the IP
address in the zk connection string and restart the broker.
One way is to give the local mapping of the zk local hostname to ip in
/etc/hosts file and change it to the new ip when the node changes. But
would reload of the Kafka config with new zk nodes be a better option? But
as you said, if we cannot reload the server.properties file, what is the
best way in case of no service discovery?
On Sun, May 24, 2015 at 6:52 PM, Gwen Shapira <gs...@cloudera.com> wrote:
> You can't dynamically re-load server properties.
>
> However, the norm in zookeeper is to configure the connection string with
> all the nodes in the zk cluster, so there will be no need to modify
> properties when you replace zk nodes.
>
> On Sun, May 24, 2015 at 4:13 PM, Achanta Vamsi Subhash <
> achanta.vamsi@flipkart.com> wrote:
>
> > Thanks Gwen.
> >
> > One more question:
> > Is there a way we can dynamically re-load the contents of the Kafka
> > server.properties contents without restarting the broker? Example
> use-case
> > is when a zk-node goes down and a new one is brought up, we need to
> update
> > the server.properties file to reflect this. Currently there is no way to
> do
> > this other than broker restart. Broker restart requires a lot of things
> to
> > do before triggering it.
> >
> > This JIRA is already filed but un-resolved. We don't require all the
> > configs to be reloaded but only variable external config changes should
> be
> > allowed.
> >
> > https://issues.apache.org/jira/browse/KAFKA-1229
> >
> > On Sun, May 24, 2015 at 1:14 PM, Gwen Shapira <gs...@cloudera.com>
> > wrote:
> >
> > > If you set advertised.hostname in server.properties to the ip address,
> > the
> > > IP will be registered in ZooKeeper.
> > >
> > >
> > > On Fri, May 22, 2015 at 2:20 PM, Achanta Vamsi Subhash <
> > > achanta.vamsi@flipkart.com> wrote:
> > >
> > > > Hi,
> > > >
> > > > Currently Kakfa brokers register the hostname in zookeeper.
> > > >
> > > > [zk: localhost:2181(CONNECTED) 5] get /varadhi/kafka/brokers/ids/0
> > > >
> > > >
> > >
> >
> {"jmx_port":9999,"timestamp":"1427704934158","host":"currHostName","version":1,"port":9092}
> > > >
> > > > Is there any config to make it use ip-address instead so that we
> don't
> > > > make a DNS lookup for the hostname?
> > > >
> > > > --
> > > > Regards
> > > > Vamsi Subhash
> > > >
> > >
> >
> >
> >
> > --
> > Regards
> > Vamsi Subhash
> >
>
--
Regards
Vamsi Subhash
Re: Kafka broker - Ip-address instead of host naem
Posted by Gwen Shapira <gs...@cloudera.com>.
You can't dynamically re-load server properties.
However, the norm in zookeeper is to configure the connection string with
all the nodes in the zk cluster, so there will be no need to modify
properties when you replace zk nodes.
On Sun, May 24, 2015 at 4:13 PM, Achanta Vamsi Subhash <
achanta.vamsi@flipkart.com> wrote:
> Thanks Gwen.
>
> One more question:
> Is there a way we can dynamically re-load the contents of the Kafka
> server.properties contents without restarting the broker? Example use-case
> is when a zk-node goes down and a new one is brought up, we need to update
> the server.properties file to reflect this. Currently there is no way to do
> this other than broker restart. Broker restart requires a lot of things to
> do before triggering it.
>
> This JIRA is already filed but un-resolved. We don't require all the
> configs to be reloaded but only variable external config changes should be
> allowed.
>
> https://issues.apache.org/jira/browse/KAFKA-1229
>
> On Sun, May 24, 2015 at 1:14 PM, Gwen Shapira <gs...@cloudera.com>
> wrote:
>
> > If you set advertised.hostname in server.properties to the ip address,
> the
> > IP will be registered in ZooKeeper.
> >
> >
> > On Fri, May 22, 2015 at 2:20 PM, Achanta Vamsi Subhash <
> > achanta.vamsi@flipkart.com> wrote:
> >
> > > Hi,
> > >
> > > Currently Kakfa brokers register the hostname in zookeeper.
> > >
> > > [zk: localhost:2181(CONNECTED) 5] get /varadhi/kafka/brokers/ids/0
> > >
> > >
> >
> {"jmx_port":9999,"timestamp":"1427704934158","host":"currHostName","version":1,"port":9092}
> > >
> > > Is there any config to make it use ip-address instead so that we don't
> > > make a DNS lookup for the hostname?
> > >
> > > --
> > > Regards
> > > Vamsi Subhash
> > >
> >
>
>
>
> --
> Regards
> Vamsi Subhash
>
Re: Kafka broker - Ip-address instead of host naem
Posted by Achanta Vamsi Subhash <ac...@flipkart.com>.
Thanks Gwen.
One more question:
Is there a way we can dynamically re-load the contents of the Kafka
server.properties contents without restarting the broker? Example use-case
is when a zk-node goes down and a new one is brought up, we need to update
the server.properties file to reflect this. Currently there is no way to do
this other than broker restart. Broker restart requires a lot of things to
do before triggering it.
This JIRA is already filed but un-resolved. We don't require all the
configs to be reloaded but only variable external config changes should be
allowed.
https://issues.apache.org/jira/browse/KAFKA-1229
On Sun, May 24, 2015 at 1:14 PM, Gwen Shapira <gs...@cloudera.com> wrote:
> If you set advertised.hostname in server.properties to the ip address, the
> IP will be registered in ZooKeeper.
>
>
> On Fri, May 22, 2015 at 2:20 PM, Achanta Vamsi Subhash <
> achanta.vamsi@flipkart.com> wrote:
>
> > Hi,
> >
> > Currently Kakfa brokers register the hostname in zookeeper.
> >
> > [zk: localhost:2181(CONNECTED) 5] get /varadhi/kafka/brokers/ids/0
> >
> >
> {"jmx_port":9999,"timestamp":"1427704934158","host":"currHostName","version":1,"port":9092}
> >
> > Is there any config to make it use ip-address instead so that we don't
> > make a DNS lookup for the hostname?
> >
> > --
> > Regards
> > Vamsi Subhash
> >
>
--
Regards
Vamsi Subhash
Re: Kafka broker - Ip-address instead of host naem
Posted by Gwen Shapira <gs...@cloudera.com>.
If you set advertised.hostname in server.properties to the ip address, the
IP will be registered in ZooKeeper.
On Fri, May 22, 2015 at 2:20 PM, Achanta Vamsi Subhash <
achanta.vamsi@flipkart.com> wrote:
> Hi,
>
> Currently Kakfa brokers register the hostname in zookeeper.
>
> [zk: localhost:2181(CONNECTED) 5] get /varadhi/kafka/brokers/ids/0
>
> {"jmx_port":9999,"timestamp":"1427704934158","host":"currHostName","version":1,"port":9092}
>
> Is there any config to make it use ip-address instead so that we don't
> make a DNS lookup for the hostname?
>
> --
> Regards
> Vamsi Subhash
>