Is bootstrap.servers resolved only once?

[Michał Łowicki <ml...@gmail.com>]

Hey,

Trying to understand how *bootstrap.servers* is handled for KafkaProducer.
I see that it's processed during creating of producer (here
<https://github.com/apache/kafka/blob/trunk/clients/src/main/java/org/apache/kafka/clients/producer/KafkaProducer.java#L414>)
and later if client DNS lookup is set to "default" it's being resolved to
only one IP address (here
<https://github.com/apache/kafka/blob/trunk/clients/src/main/java/org/apache/kafka/clients/ClientUtils.java#L72>
).

1. Does it mean that if *bootstrap.servers* contains only one address which
resolves to multiple IP addresses then still only one IP will be taken into
account during bootstrap?
2. What happens during catastrophic scenario where all brokers die and
after restart they got different IPs? Will ever *bootstrap.servers* be
evaluated once again so the clients could re-connect to the cluster
assuming that bootstrap.servers will resolve to new IPs?

-- 
BR,
Michał Łowicki

Re: Is bootstrap.servers resolved only once?

[Shilin Wu <sh...@confluent.io.INVALID>]

Yes.

On initial bootstrapping dns resolves all ip as of 2.10, if you specified
client.dns.lookup to use all addresses.

But the dns is evaluated only once in the producer initialisation
(constructor) so it will fail if all broker ip changed at once.


On Mon, 17 May 2021 at 9:42 PM, Michał Łowicki <ml...@gmail.com> wrote:

> On Mon, May 17, 2021 at 2:25 PM Shilin Wu <sh...@confluent.io.invalid>
> wrote:
>
> > Bootstrap servers are just used for initial connection, clients will get
> > all server metadata from one of the bootstrap servers to discover the
> full
> > cluster membership (which may change dynamically), this list does not
> have
> > to contain the full set of servers (you may want more than one, though,
> in
> > case a server that used for bootstrapping is down).
> >
> > Clients (producers or consumers) make use of all servers irrespective of
> > which servers are specified in bootstrap.servers for bootstrapping.
> >
> > In case of cluster change (e.g. new ip addresses for new servers), the
> > clients will receive updates from the server group.
> >
> > To answer your question:
> > 1. Does it mean that if *bootstrap.servers* contains only one address
> which
> > resolves to multiple IP addresses then still only one IP will be taken
> into
> > account during bootstrap?
> > => It shouldn't matter. As long as it resolves at least one functioning
> > broker, it will work. Make sure the advertised listeners are configured
> > correctly and are externally accessible.
> > All actual server meta data are retrieved from the functioning broker and
> > updated accordingly.
> >
>
> AFAIU it can matter in one (probably minor) scenario where KafkaProducer
> will resolve DNS name into just one IP (first / random) and before
> connecting to it to get metadata about the cluster the node would die and
> no other IP would be tried out. If it would resolve to all then it could
> try others. Currently KafkaProducer just gives up. This is easy though to
> add logic to retry or so.
>
>
> >
> > 2. What happens during catastrophic scenario where all brokers die and
> > after restart they got different IPs? Will ever *bootstrap.servers* be
> > evaluated once again so the clients could re-connect to the cluster
> > assuming that bootstrap.servers will resolve to new IPs?
> > => If you set client.dns.lookup="use_all_dns_ips" in your client
> > configuration, it will use all of the IP addresses returned by DNS, not
> > just the first (or a random one).
> >
> > What if it is so all ip addresses are changed? and you still have same
> DNS,
> > but points to completely new host ips?
> >
> > From
> >
> >
> https://github.com/a0x8o/kafka/blob/master/clients/src/main/java/org/apache/kafka/clients/producer/KafkaProducer.java
> > ,
> > it seems that the address resolution for bootstrap server is done only
> once
> > in the constructor. So clients may have to be restarted in this case.
> > See line 413.
> >
> > However, if you maintain an active cluster in a controlled change process
> > (not due to a fault), and restart the servers and change ip addresses one
> > by one,
> > it should be possible to change the ip address gradually as the update of
> > metadata should happen in a timely manner.
> >
> > In short, don't kill them all and restart them at the same time with new
> > addresses.
> >
> > After all, you can always add an extra layer of retry loop to re-create
> the
> > producer with the same producer config, if Java dns cache TTL expires, it
> > may pick up new addresses eventually. This may take long though.
> >
>
> Yeah, we had scenario that because of the human error the cluster on K8s
> was wiped out, recreated fast but all IPs changed and consumers / producers
> couldn't reconnect without restarting. This still can be patched in the app
> by maybe checking for n consecutive send failures and recreate producer
> then.
>
>
> >
> >
> >
> >
> > [image: Confluent] <https://www.confluent.io>
> > Wu Shilin
> > Solution Architect
> > +6581007012
> > Follow us: [image: Blog]
> > <
> >
> https://www.confluent.io/blog?utm_source=footer&utm_medium=email&utm_campaign=ch.email-signature_type.community_content.blog
> > >[image:
> > Twitter] <https://twitter.com/ConfluentInc>[image: LinkedIn]
> > <https://www.linkedin.com/company/confluent/>[image: Slack]
> > <https://slackpass.io/confluentcommunity>[image: YouTube]
> > <https://youtube.com/confluent>
> > [image: Kafka Summit] <https://www.kafka-summit.org/>
> >
> >
> > On Mon, May 17, 2021 at 8:03 PM Michał Łowicki <ml...@gmail.com>
> wrote:
> >
> > > Hey,
> > >
> > > Trying to understand how *bootstrap.servers* is handled for
> > KafkaProducer.
> > > I see that it's processed during creating of producer (here
> > > <
> > >
> >
> https://github.com/apache/kafka/blob/trunk/clients/src/main/java/org/apache/kafka/clients/producer/KafkaProducer.java#L414
> > > >)
> > > and later if client DNS lookup is set to "default" it's being resolved
> to
> > > only one IP address (here
> > > <
> > >
> >
> https://github.com/apache/kafka/blob/trunk/clients/src/main/java/org/apache/kafka/clients/ClientUtils.java#L72
> > > >
> > > ).
> > >
> > > 1. Does it mean that if *bootstrap.servers* contains only one address
> > which
> > > resolves to multiple IP addresses then still only one IP will be taken
> > into
> > > account during bootstrap?
> > > 2. What happens during catastrophic scenario where all brokers die and
> > > after restart they got different IPs? Will ever *bootstrap.servers* be
> > > evaluated once again so the clients could re-connect to the cluster
> > > assuming that bootstrap.servers will resolve to new IPs?
> > >
> > > --
> > > BR,
> > > Michał Łowicki
> > >
> >
>
>
> --
> BR,
> Michał Łowicki
>
-- 

[image: Confluent] <https://www.confluent.io>
Wu Shilin
Solution Architect
+6581007012
Follow us: [image: Blog]
<https://www.confluent.io/blog?utm_source=footer&utm_medium=email&utm_campaign=ch.email-signature_type.community_content.blog>[image:
Twitter] <https://twitter.com/ConfluentInc>[image: LinkedIn]
<https://www.linkedin.com/company/confluent/>[image: Slack]
<https://slackpass.io/confluentcommunity>[image: YouTube]
<https://youtube.com/confluent>
[image: Kafka Summit] <https://www.kafka-summit.org/>

Re: Is bootstrap.servers resolved only once?

[Michał Łowicki <ml...@gmail.com>]

On Mon, May 17, 2021 at 2:25 PM Shilin Wu <sh...@confluent.io.invalid> wrote:

> Bootstrap servers are just used for initial connection, clients will get
> all server metadata from one of the bootstrap servers to discover the full
> cluster membership (which may change dynamically), this list does not have
> to contain the full set of servers (you may want more than one, though, in
> case a server that used for bootstrapping is down).
>
> Clients (producers or consumers) make use of all servers irrespective of
> which servers are specified in bootstrap.servers for bootstrapping.
>
> In case of cluster change (e.g. new ip addresses for new servers), the
> clients will receive updates from the server group.
>
> To answer your question:
> 1. Does it mean that if *bootstrap.servers* contains only one address which
> resolves to multiple IP addresses then still only one IP will be taken into
> account during bootstrap?
> => It shouldn't matter. As long as it resolves at least one functioning
> broker, it will work. Make sure the advertised listeners are configured
> correctly and are externally accessible.
> All actual server meta data are retrieved from the functioning broker and
> updated accordingly.
>

AFAIU it can matter in one (probably minor) scenario where KafkaProducer
will resolve DNS name into just one IP (first / random) and before
connecting to it to get metadata about the cluster the node would die and
no other IP would be tried out. If it would resolve to all then it could
try others. Currently KafkaProducer just gives up. This is easy though to
add logic to retry or so.


>
> 2. What happens during catastrophic scenario where all brokers die and
> after restart they got different IPs? Will ever *bootstrap.servers* be
> evaluated once again so the clients could re-connect to the cluster
> assuming that bootstrap.servers will resolve to new IPs?
> => If you set client.dns.lookup="use_all_dns_ips" in your client
> configuration, it will use all of the IP addresses returned by DNS, not
> just the first (or a random one).
>
> What if it is so all ip addresses are changed? and you still have same DNS,
> but points to completely new host ips?
>
> From
>
> https://github.com/a0x8o/kafka/blob/master/clients/src/main/java/org/apache/kafka/clients/producer/KafkaProducer.java
> ,
> it seems that the address resolution for bootstrap server is done only once
> in the constructor. So clients may have to be restarted in this case.
> See line 413.
>
> However, if you maintain an active cluster in a controlled change process
> (not due to a fault), and restart the servers and change ip addresses one
> by one,
> it should be possible to change the ip address gradually as the update of
> metadata should happen in a timely manner.
>
> In short, don't kill them all and restart them at the same time with new
> addresses.
>
> After all, you can always add an extra layer of retry loop to re-create the
> producer with the same producer config, if Java dns cache TTL expires, it
> may pick up new addresses eventually. This may take long though.
>

Yeah, we had scenario that because of the human error the cluster on K8s
was wiped out, recreated fast but all IPs changed and consumers / producers
couldn't reconnect without restarting. This still can be patched in the app
by maybe checking for n consecutive send failures and recreate producer
then.


>
>
>
>
> [image: Confluent] <https://www.confluent.io>
> Wu Shilin
> Solution Architect
> +6581007012
> Follow us: [image: Blog]
> <
> https://www.confluent.io/blog?utm_source=footer&utm_medium=email&utm_campaign=ch.email-signature_type.community_content.blog
> >[image:
> Twitter] <https://twitter.com/ConfluentInc>[image: LinkedIn]
> <https://www.linkedin.com/company/confluent/>[image: Slack]
> <https://slackpass.io/confluentcommunity>[image: YouTube]
> <https://youtube.com/confluent>
> [image: Kafka Summit] <https://www.kafka-summit.org/>
>
>
> On Mon, May 17, 2021 at 8:03 PM Michał Łowicki <ml...@gmail.com> wrote:
>
> > Hey,
> >
> > Trying to understand how *bootstrap.servers* is handled for
> KafkaProducer.
> > I see that it's processed during creating of producer (here
> > <
> >
> https://github.com/apache/kafka/blob/trunk/clients/src/main/java/org/apache/kafka/clients/producer/KafkaProducer.java#L414
> > >)
> > and later if client DNS lookup is set to "default" it's being resolved to
> > only one IP address (here
> > <
> >
> https://github.com/apache/kafka/blob/trunk/clients/src/main/java/org/apache/kafka/clients/ClientUtils.java#L72
> > >
> > ).
> >
> > 1. Does it mean that if *bootstrap.servers* contains only one address
> which
> > resolves to multiple IP addresses then still only one IP will be taken
> into
> > account during bootstrap?
> > 2. What happens during catastrophic scenario where all brokers die and
> > after restart they got different IPs? Will ever *bootstrap.servers* be
> > evaluated once again so the clients could re-connect to the cluster
> > assuming that bootstrap.servers will resolve to new IPs?
> >
> > --
> > BR,
> > Michał Łowicki
> >
>


-- 
BR,
Michał Łowicki

Re: Is bootstrap.servers resolved only once?

[Shilin Wu <sh...@confluent.io.INVALID>]

Bootstrap servers are just used for initial connection, clients will get
all server metadata from one of the bootstrap servers to discover the full
cluster membership (which may change dynamically), this list does not have
to contain the full set of servers (you may want more than one, though, in
case a server that used for bootstrapping is down).

Clients (producers or consumers) make use of all servers irrespective of
which servers are specified in bootstrap.servers for bootstrapping.

In case of cluster change (e.g. new ip addresses for new servers), the
clients will receive updates from the server group.

To answer your question:
1. Does it mean that if *bootstrap.servers* contains only one address which
resolves to multiple IP addresses then still only one IP will be taken into
account during bootstrap?
=> It shouldn't matter. As long as it resolves at least one functioning
broker, it will work. Make sure the advertised listeners are configured
correctly and are externally accessible.
All actual server meta data are retrieved from the functioning broker and
updated accordingly.

2. What happens during catastrophic scenario where all brokers die and
after restart they got different IPs? Will ever *bootstrap.servers* be
evaluated once again so the clients could re-connect to the cluster
assuming that bootstrap.servers will resolve to new IPs?
=> If you set client.dns.lookup="use_all_dns_ips" in your client
configuration, it will use all of the IP addresses returned by DNS, not
just the first (or a random one).

What if it is so all ip addresses are changed? and you still have same DNS,
but points to completely new host ips?

From
https://github.com/a0x8o/kafka/blob/master/clients/src/main/java/org/apache/kafka/clients/producer/KafkaProducer.java
,
it seems that the address resolution for bootstrap server is done only once
in the constructor. So clients may have to be restarted in this case.
See line 413.

However, if you maintain an active cluster in a controlled change process
(not due to a fault), and restart the servers and change ip addresses one
by one,
it should be possible to change the ip address gradually as the update of
metadata should happen in a timely manner.

In short, don't kill them all and restart them at the same time with new
addresses.

After all, you can always add an extra layer of retry loop to re-create the
producer with the same producer config, if Java dns cache TTL expires, it
may pick up new addresses eventually. This may take long though.





[image: Confluent] <https://www.confluent.io>
Wu Shilin
Solution Architect
+6581007012
Follow us: [image: Blog]
<https://www.confluent.io/blog?utm_source=footer&utm_medium=email&utm_campaign=ch.email-signature_type.community_content.blog>[image:
Twitter] <https://twitter.com/ConfluentInc>[image: LinkedIn]
<https://www.linkedin.com/company/confluent/>[image: Slack]
<https://slackpass.io/confluentcommunity>[image: YouTube]
<https://youtube.com/confluent>
[image: Kafka Summit] <https://www.kafka-summit.org/>


On Mon, May 17, 2021 at 8:03 PM Michał Łowicki <ml...@gmail.com> wrote:

> Hey,
>
> Trying to understand how *bootstrap.servers* is handled for KafkaProducer.
> I see that it's processed during creating of producer (here
> <
> https://github.com/apache/kafka/blob/trunk/clients/src/main/java/org/apache/kafka/clients/producer/KafkaProducer.java#L414
> >)
> and later if client DNS lookup is set to "default" it's being resolved to
> only one IP address (here
> <
> https://github.com/apache/kafka/blob/trunk/clients/src/main/java/org/apache/kafka/clients/ClientUtils.java#L72
> >
> ).
>
> 1. Does it mean that if *bootstrap.servers* contains only one address which
> resolves to multiple IP addresses then still only one IP will be taken into
> account during bootstrap?
> 2. What happens during catastrophic scenario where all brokers die and
> after restart they got different IPs? Will ever *bootstrap.servers* be
> evaluated once again so the clients could re-connect to the cluster
> assuming that bootstrap.servers will resolve to new IPs?
>
> --
> BR,
> Michał Łowicki
>