You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@druid.apache.org by Don Bowman <do...@agilicus.com> on 2019/03/21 14:13:45 UTC
dns lookups cached for kafka?
What is the expectation around dns and druid? Specifically, when overlord
started, it resolved (correctly) my kafka cluster nodes.
A little bit later I made a change which changed their IP. But overlord
continues to use that originally resolved IP.
Is there a way to force the refresh?
Should it be re-resolving on a connection failure?
[Consumer clientId=consumer-6, groupId=kafka-supervisor-bebmfiod]
Connection to node 2 (kafka-2.kafka-headless.kafka/*10.60.8.24*:9092) could
not be established. Broker may not be available.
$ kubectl -n kafka get pods -o=wide NAME READY STATUS RESTARTS AGE IP NODE
NOMINATED NODE kafka-0 1/1 Running 0 20m 10.60.7.45
gke-noctest-default-pool-51f579ce-ztgf <none> kafka-1 1/1 Running 0 20m
10.60.1.48 gke-noctest-default-pool-8674014d-p008 <none> kafka-2 1/1
Running 0 20m *10.60.8.48* gke-noctest-default-pool-3aa530af-cztp <none>
kafka-health-check-5d5b457566-2k8lf 1/1 Running 0 20m 10.60.2.47
gke-noctest-default-pool-8674014d-rxbn <none>
Re: dns lookups cached for kafka?
Posted by Charles Allen <ch...@snap.com.INVALID>.
I believe so. That's what I do
On Thu, Mar 21, 2019 at 8:52 AM Don Bowman <do...@agilicus.com> wrote:
> On Thu, 21 Mar 2019 at 11:48, Charles Allen <charles.allen@snap.com
> .invalid>
> wrote:
>
> > Druid assumes the network layer handles whatever tuning is needed
> regarding
> > DNS resolution or IP routing. In general this means making sure you have
> > your java settings correct (see
> >
> >
> https://urldefense.proofpoint.com/v2/url?u=https-3A__docs.aws.amazon.com_sdk-2Dfor-2Djava_v1_developer-2Dguide_java-2Ddg-2Djvm-2Dttl.html&d=DwIBaQ&c=ncDTmphkJTvjIDPh0hpF_w&r=HrLGT1qWNhseJBMYABL0GFSZESht5gBoLejor3SqMSo&m=1IQMROiCdjqQ6wibAPPOTXcW7n5XXtPKIpMAcOoSGVY&s=C9YKNBvbUEtDoV9f96z8WAFQ0s3zt9MHEKgGYhc6E_A&e=
> > for a related article).
> >
> >
> >
> Thanks.
>
> Since druid does not call `java.security.Security.setProperty()`, I take it
> this means my only option is to globally change the JRE in
> $JAVA_HOME/jre/lib/security/java.security?
>
Re: dns lookups cached for kafka?
Posted by Don Bowman <do...@agilicus.com>.
On Thu, 21 Mar 2019 at 11:48, Charles Allen <ch...@snap.com.invalid>
wrote:
> Druid assumes the network layer handles whatever tuning is needed regarding
> DNS resolution or IP routing. In general this means making sure you have
> your java settings correct (see
>
> https://docs.aws.amazon.com/sdk-for-java/v1/developer-guide/java-dg-jvm-ttl.html
> for a related article).
>
>
>
Thanks.
Since druid does not call `java.security.Security.setProperty()`, I take it
this means my only option is to globally change the JRE in
$JAVA_HOME/jre/lib/security/java.security?
Re: dns lookups cached for kafka?
Posted by Charles Allen <ch...@snap.com.INVALID>.
Druid assumes the network layer handles whatever tuning is needed regarding
DNS resolution or IP routing. In general this means making sure you have
your java settings correct (see
https://docs.aws.amazon.com/sdk-for-java/v1/developer-guide/java-dg-jvm-ttl.html
for a related article).
On Thu, Mar 21, 2019 at 7:14 AM Don Bowman <do...@agilicus.com> wrote:
> What is the expectation around dns and druid? Specifically, when overlord
> started, it resolved (correctly) my kafka cluster nodes.
> A little bit later I made a change which changed their IP. But overlord
> continues to use that originally resolved IP.
>
> Is there a way to force the refresh?
> Should it be re-resolving on a connection failure?
>
> [Consumer clientId=consumer-6, groupId=kafka-supervisor-bebmfiod]
> Connection to node 2 (kafka-2.kafka-headless.kafka/*10.60.8.24*:9092) could
> not be established. Broker may not be available.
>
> $ kubectl -n kafka get pods -o=wide NAME READY STATUS RESTARTS AGE IP NODE
> NOMINATED NODE kafka-0 1/1 Running 0 20m 10.60.7.45
> gke-noctest-default-pool-51f579ce-ztgf <none> kafka-1 1/1 Running 0 20m
> 10.60.1.48 gke-noctest-default-pool-8674014d-p008 <none> kafka-2 1/1
> Running 0 20m *10.60.8.48* gke-noctest-default-pool-3aa530af-cztp <none>
> kafka-health-check-5d5b457566-2k8lf 1/1 Running 0 20m 10.60.2.47
> gke-noctest-default-pool-8674014d-rxbn <none>
>
Re: dns lookups cached for kafka?
Posted by Anoop K <an...@outlook.com>.
Instead of editing java.security properties, JVM allows to specify a property file in command line. All the properties will be appended to the default (/global) security properties. This way configuration remain local to the application.
# An alternate java.security properties file may be specified
# from the command line via the system property
#
# -Djava.security.properties=<URL>
#
# This properties file appends to the master security properties file.
# If both properties files specify values for the same key, the value
# from the command-line properties file is selected, as it is the last
# one loaded.
#
-anoop
________________________________
From: Will Lauer <wl...@verizonmedia.com.INVALID>
Sent: Thursday, March 21, 2019 10:37 AM
To: dev@druid.apache.org
Subject: Re: dns lookups cached for kafka?
See the documentation for java.net.InetAddress. By default, Java caches DNS
results, and may be caching them more than you expect. This is controllable
via some properties that you can set on the command line.
Will
*Will Lauer*
Senior Principal Architect
Audience and Advertising Reporting, Verizon Media
o: 217-255-4264
m: 508-561-6427
On Thu, Mar 21, 2019 at 9:14 AM Don Bowman <do...@agilicus.com> wrote:
> What is the expectation around dns and druid? Specifically, when overlord
> started, it resolved (correctly) my kafka cluster nodes.
> A little bit later I made a change which changed their IP. But overlord
> continues to use that originally resolved IP.
>
> Is there a way to force the refresh?
> Should it be re-resolving on a connection failure?
>
> [Consumer clientId=consumer-6, groupId=kafka-supervisor-bebmfiod]
> Connection to node 2 (kafka-2.kafka-headless.kafka/*10.60.8.24*:9092) could
> not be established. Broker may not be available.
>
> $ kubectl -n kafka get pods -o=wide NAME READY STATUS RESTARTS AGE IP NODE
> NOMINATED NODE kafka-0 1/1 Running 0 20m 10.60.7.45
> gke-noctest-default-pool-51f579ce-ztgf <none> kafka-1 1/1 Running 0 20m
> 10.60.1.48 gke-noctest-default-pool-8674014d-p008 <none> kafka-2 1/1
> Running 0 20m *10.60.8.48* gke-noctest-default-pool-3aa530af-cztp <none>
> kafka-health-check-5d5b457566-2k8lf 1/1 Running 0 20m 10.60.2.47
> gke-noctest-default-pool-8674014d-rxbn <none>
>
Re: dns lookups cached for kafka?
Posted by Will Lauer <wl...@verizonmedia.com.INVALID>.
See the documentation for java.net.InetAddress. By default, Java caches DNS
results, and may be caching them more than you expect. This is controllable
via some properties that you can set on the command line.
Will
*Will Lauer*
Senior Principal Architect
Audience and Advertising Reporting, Verizon Media
o: 217-255-4264
m: 508-561-6427
On Thu, Mar 21, 2019 at 9:14 AM Don Bowman <do...@agilicus.com> wrote:
> What is the expectation around dns and druid? Specifically, when overlord
> started, it resolved (correctly) my kafka cluster nodes.
> A little bit later I made a change which changed their IP. But overlord
> continues to use that originally resolved IP.
>
> Is there a way to force the refresh?
> Should it be re-resolving on a connection failure?
>
> [Consumer clientId=consumer-6, groupId=kafka-supervisor-bebmfiod]
> Connection to node 2 (kafka-2.kafka-headless.kafka/*10.60.8.24*:9092) could
> not be established. Broker may not be available.
>
> $ kubectl -n kafka get pods -o=wide NAME READY STATUS RESTARTS AGE IP NODE
> NOMINATED NODE kafka-0 1/1 Running 0 20m 10.60.7.45
> gke-noctest-default-pool-51f579ce-ztgf <none> kafka-1 1/1 Running 0 20m
> 10.60.1.48 gke-noctest-default-pool-8674014d-p008 <none> kafka-2 1/1
> Running 0 20m *10.60.8.48* gke-noctest-default-pool-3aa530af-cztp <none>
> kafka-health-check-5d5b457566-2k8lf 1/1 Running 0 20m 10.60.2.47
> gke-noctest-default-pool-8674014d-rxbn <none>
>