You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flink.apache.org by David Maddison <ma...@gmail.com> on 2019/11/29 10:40:48 UTC
Temporary failure in name resolution on JobManager
I have a Flink 1.7 cluster using the "flink:1.7.2" (OpenJDK build
1.8.0_222-b10) image on Kubernetes.
As part of a MasterRestoreHook (for checkpointing) the JobManager needs to
communicate with an external security service. This all works well until
there's a DNS lookup failure (due to network issues) at which point the
JobManager JVM seems unable to ever successfully look up the name again,
even when it's confirmed DNS service has been restored. The weird thing is
that I can use kubectl to exec into the JobManager POD and successfully
perform a lookup even while the JobManager JVM is still failing to lookup.
Has anybody seen an issue like this before, or have any suggestions? As
far as I'm aware Flink doesn't install a SecurityManager and therefore the
JVM should only cache invalid name requests for 10 seconds.
Restarting the JobManager JVM does successfully recover the Job, but I'd
like to avoid having to do that if possible.
Caused by: java.net.UnknownHostException: <********>.com: Temporary failure
in name resolution
at java.net.Inet4AddressImpl.lookupAllHostAddr(Native Method)
at java.net.InetAddress$2.lookupAllHostAddr(InetAddress.java:929)
at java.net.InetAddress.getAddressesFromNameService(InetAddress.java:1324)
at java.net.InetAddress.getAllByName0(InetAddress.java:1277)
at java.net.InetAddress.getAllByName(InetAddress.java:1193)
at java.net.InetAddress.getAllByName(InetAddress.java:1127)
Thanks in advance,
David
Re: Temporary failure in name resolution on JobManager
Posted by David Maddison <ma...@gmail.com>.
Thanks Yang.
We did try both those properties and it didn't fix it. However, we did
EVENTUALLY (after some late nights!) track the issue down, not to DNS
resolution but rather an obscure bug our our connector code :-(
Thanks for your response,
/David/
On Mon, Dec 2, 2019 at 3:16 AM Yang Wang <da...@gmail.com> wrote:
> Hi David,
>
> Do you mean when the JobManager starts, the dns has some problem and the
> service could
> not be resolved? The dns restores to normal, and the JobManager jvm could
> not look up the
> dns.
> I think it may because the jvm dns cache. You could set the ttl and have a
> try.
> sun.net.inetaddr.ttl
> sun.net.inetaddr.negative.ttl
>
>
> Best,
> Yang
>
> David Maddison <ma...@gmail.com> 于2019年11月29日周五 下午6:41写道:
>
>> I have a Flink 1.7 cluster using the "flink:1.7.2" (OpenJDK build
>> 1.8.0_222-b10) image on Kubernetes.
>>
>> As part of a MasterRestoreHook (for checkpointing) the JobManager needs
>> to communicate with an external security service. This all works well
>> until there's a DNS lookup failure (due to network issues) at which point
>> the JobManager JVM seems unable to ever successfully look up the name
>> again, even when it's confirmed DNS service has been restored. The weird
>> thing is that I can use kubectl to exec into the JobManager POD and
>> successfully perform a lookup even while the JobManager JVM is still
>> failing to lookup.
>>
>> Has anybody seen an issue like this before, or have any suggestions? As
>> far as I'm aware Flink doesn't install a SecurityManager and therefore the
>> JVM should only cache invalid name requests for 10 seconds.
>>
>> Restarting the JobManager JVM does successfully recover the Job, but I'd
>> like to avoid having to do that if possible.
>>
>> Caused by: java.net.UnknownHostException: <********>.com: Temporary
>> failure in name resolution
>> at java.net.Inet4AddressImpl.lookupAllHostAddr(Native Method)
>> at java.net.InetAddress$2.lookupAllHostAddr(InetAddress.java:929)
>> at java.net.InetAddress.getAddressesFromNameService(InetAddress.java:1324)
>> at java.net.InetAddress.getAllByName0(InetAddress.java:1277)
>> at java.net.InetAddress.getAllByName(InetAddress.java:1193)
>> at java.net.InetAddress.getAllByName(InetAddress.java:1127)
>>
>> Thanks in advance,
>>
>> David
>>
>
Re: Temporary failure in name resolution on JobManager
Posted by Yang Wang <da...@gmail.com>.
Hi David,
Do you mean when the JobManager starts, the dns has some problem and the
service could
not be resolved? The dns restores to normal, and the JobManager jvm could
not look up the
dns.
I think it may because the jvm dns cache. You could set the ttl and have a
try.
sun.net.inetaddr.ttl
sun.net.inetaddr.negative.ttl
Best,
Yang
David Maddison <ma...@gmail.com> 于2019年11月29日周五 下午6:41写道:
> I have a Flink 1.7 cluster using the "flink:1.7.2" (OpenJDK build
> 1.8.0_222-b10) image on Kubernetes.
>
> As part of a MasterRestoreHook (for checkpointing) the JobManager needs to
> communicate with an external security service. This all works well until
> there's a DNS lookup failure (due to network issues) at which point the
> JobManager JVM seems unable to ever successfully look up the name again,
> even when it's confirmed DNS service has been restored. The weird thing is
> that I can use kubectl to exec into the JobManager POD and successfully
> perform a lookup even while the JobManager JVM is still failing to lookup.
>
> Has anybody seen an issue like this before, or have any suggestions? As
> far as I'm aware Flink doesn't install a SecurityManager and therefore the
> JVM should only cache invalid name requests for 10 seconds.
>
> Restarting the JobManager JVM does successfully recover the Job, but I'd
> like to avoid having to do that if possible.
>
> Caused by: java.net.UnknownHostException: <********>.com: Temporary
> failure in name resolution
> at java.net.Inet4AddressImpl.lookupAllHostAddr(Native Method)
> at java.net.InetAddress$2.lookupAllHostAddr(InetAddress.java:929)
> at java.net.InetAddress.getAddressesFromNameService(InetAddress.java:1324)
> at java.net.InetAddress.getAllByName0(InetAddress.java:1277)
> at java.net.InetAddress.getAllByName(InetAddress.java:1193)
> at java.net.InetAddress.getAllByName(InetAddress.java:1127)
>
> Thanks in advance,
>
> David
>