You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by "ASF GitHub Bot (JIRA)" <ji...@apache.org> on 2018/11/30 14:33:00 UTC
[jira] [Commented] (FLINK-10213) Task managers cache a negative DNS
lookup of the blob server indefinitely
[ https://issues.apache.org/jira/browse/FLINK-10213?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16704810#comment-16704810 ]
ASF GitHub Bot commented on FLINK-10213:
----------------------------------------
dawidwys commented on a change in pull request #6862: [FLINK-10213] Task managers cache a negative DNS lookup of the blob server indefinitely
URL: https://github.com/apache/flink/pull/6862#discussion_r237878535
##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/blob/BlobClient.java
##########
@@ -87,12 +87,13 @@ public BlobClient(InetSocketAddress serverAddress, Configuration clientConfig) t
LOG.info("Using ssl connection to the blob server");
socket = SSLUtils.createSSLClientSocketFactory(clientConfig).createSocket(
- serverAddress.getAddress(),
+ serverAddress.getHostName(),
serverAddress.getPort());
}
else {
- socket = new Socket();
- socket.connect(serverAddress);
+ // Establish the socket using the hostname and port. This avoids a potential issue where the
+ // InetSocketAddress can cache a failure in hostname resolution forever.
+ socket = new Socket(serverAddress.getHostName(), serverAddress.getPort());
Review comment:
How about doing the lookup only if the `InetSocketAddress` was not resolved?
```
socket = new Socket();
if (serverAddress.isUnresolved()) {
socket.connect(new InetSocketAddress(serverAddress.getHostName(), serverAddress.getPort()));
} else {
socket.connect(serverAddress);
}
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
> Task managers cache a negative DNS lookup of the blob server indefinitely
> -------------------------------------------------------------------------
>
> Key: FLINK-10213
> URL: https://issues.apache.org/jira/browse/FLINK-10213
> Project: Flink
> Issue Type: Bug
> Components: TaskManager
> Affects Versions: 1.5.0
> Reporter: Joey Echeverria
> Assignee: Joey Echeverria
> Priority: Major
> Labels: pull-request-available
> Fix For: 1.8.0
>
>
> When the task manager establishes a connection with the resource manager, it gets the hostname and port of the blob server and uses that to create an instance of an {{InetSocketAddress}}. Per the documentation of the constructor:
> {quote}An attempt will be made to resolve the hostname into an InetAddress. If that attempt fails, the address will be flagged as _unresolved_{quote}
> Flink never checks to see if the address was unresolved. Later when executing a task that needs to download from the blob server, it will use that same {{InetSocketAddress}} instance to attempt to connect a {{Socket}}. This will result in an exception similar to:
> {noformat}
> java.io.IOException: Failed to fetch BLOB 97799b827ef073e04178a99f0f40b00e/p-6d8ec2ad31337110819c7c3641fdb18d3793a7fb-72bf00066308f4b4d2a9c5aea593b41f from jobmanager:6124 and store it under /tmp/blobStore-d135961a-03cb-4542-af6d-cf378ff83c12/incoming/temp-00018669
> at org.apache.flink.runtime.blob.BlobClient.downloadFromBlobServer(BlobClient.java:191) ~[flink-dist_2.11-1.5.0.jar:1.5.0]
> at org.apache.flink.runtime.blob.AbstractBlobCache.getFileInternal(AbstractBlobCache.java:181) ~[flink-dist_2.11-1.5.0.jar:1.5.0]
> at org.apache.flink.runtime.blob.PermanentBlobCache.getFile(PermanentBlobCache.java:206) ~[flink-dist_2.11-1.5.0.jar:1.5.0]
> at org.apache.flink.runtime.execution.librarycache.BlobLibraryCacheManager.registerTask(BlobLibraryCacheManager.java:120) ~[flink-dist_2.11-1.5.0.jar:1.5.0]
> at org.apache.flink.runtime.taskmanager.Task.createUserCodeClassloader(Task.java:863) [flink-dist_2.11-1.5.0.jar:1.5.0]
> at org.apache.flink.runtime.taskmanager.Task.run(Task.java:579) [flink-dist_2.11-1.5.0.jar:1.5.0]
> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_171]
> Caused by: java.io.IOException: Could not connect to BlobServer at address flink-jobmanager:6124
> at org.apache.flink.runtime.blob.BlobClient.<init>(BlobClient.java:124) ~[flink-dist_2.11-1.5.0.jar:1.5.0]
> at org.apache.flink.runtime.blob.BlobClient.downloadFromBlobServer(BlobClient.java:165) ~[flink-dist_2.11-1.5.0.jar:1.5.0]
> ... 6 more
> Caused by: java.net.UnknownHostException: jobmanager
> at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:184) ~[?:1.8.0_171]
> at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) ~[?:1.8.0_171]
> at java.net.Socket.connect(Socket.java:589) ~[?:1.8.0_171]
> at java.net.Socket.connect(Socket.java:538) ~[?:1.8.0_171]
> at org.apache.flink.runtime.blob.BlobClient.<init>(BlobClient.java:118) ~[flink-dist_2.11-1.5.0.jar:1.5.0]
> at org.apache.flink.runtime.blob.BlobClient.downloadFromBlobServer(BlobClient.java:165) ~[flink-dist_2.11-1.5.0.jar:1.5.0]
> ... 6 more
> {noformat}
> Since the {{InetSocketAddress}} is re-used, you'll have repeated failures of any tasks that are executed on that task manager and the only current workaround is to manually restart the task manager.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)