You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by "Aljoscha Krettek (JIRA)" <ji...@apache.org> on 2018/04/20 17:32:00 UTC
[jira] [Comment Edited] (FLINK-9211) Job submission via
REST/dashboard does not work on Kubernetes
[ https://issues.apache.org/jira/browse/FLINK-9211?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16446057#comment-16446057 ]
Aljoscha Krettek edited comment on FLINK-9211 at 4/20/18 5:31 PM:
------------------------------------------------------------------
The problem is that some Kubernetes deployments don't allow a pod to connect to itself using a service that points at the pod itself.
In the Flink doc, we describe K8s deployments like this:
- deploy JM
- deploy JM service with name {{flink-jobmanager}}
- deploy TMs
All components are configured to think that the hostname of the JM is {{flink-jobmanager}} and the TMs can actually connect to that via the service. However, the JM cannot connect to itself via that service. This is a problem because {{WebSubmissionException}}, {{JarRunHandler}} to be specific, will instantiate a {{RestClusterClient}} that tries to submit to {{[http://flink-jobmanager:8081|http://flink-jobmanager:8081/]}}. This will fail.
A simple fix is to change {{WebSubmissionExtension}} to use this:
{code:java}
// we always use localhost as the endpoint because we assume that the dispatcher
// and web submission extension are running on the same machine. We're using
// localhost because in some setups (for example some Kubernetes setups) a machine
// might not be able to refer back to itself via the externally announced hostname
final SettableLeaderRetrievalService settableLeaderRetrievalService =
new SettableLeaderRetrievalService("http://localhost:8081", HighAvailabilityServices.DEFAULT_LEADER_ID);
{code}
However, this doesn't work if the port is changed in the config or if we use {{https}}. The code that constructs the proper url can be found here: [https://github.com/apache/flink/blob/4fa4e8cb364c35ea1807a051929b4604b9d31c2e/flink-runtime/src/main/java/org/apache/flink/runtime/rest/RestServerEndpoint.java#L207].
was (Author: aljoscha):
The problem is that some Kubernetes deployments don't allow a pod to connect to itself using a servicethat points at the pod itself.
In the Flink doc, we describe K8s deployments like this:
- deploy JM
- deploy JM service with name {{flink-jobmanager}}
- deploy TMs
All components are configured to think that the hostname of the JM is {{flink-jobmanager}} and the TMs can actually connect to that via the service. However, the JM cannot connect to itself via that service. This is a problem because {{WebSubmissionException}}, {{JarRunHandler}} to be specific, will instantiate a {{RestClusterClient}} that tries to submit to {{[http://flink-jobmanager:8081|http://flink-jobmanager:8081/]}}. This will fail.
A simple fix is to change {{WebSubmissionExtension}} to use this:
{code:java}
// we always use localhost as the endpoint because we assume that the dispatcher
// and web submission extension are running on the same machine. We're using
// localhost because in some setups (for example some Kubernetes setups) a machine
// might not be able to refer back to itself via the externally announced hostname
final SettableLeaderRetrievalService settableLeaderRetrievalService =
new SettableLeaderRetrievalService("http://localhost:8081", HighAvailabilityServices.DEFAULT_LEADER_ID);
{code}
However, this doesn't work if the port is changed in the config or if we use {{https}}. The code that constructs the proper url can be found here: [https://github.com/apache/flink/blob/4fa4e8cb364c35ea1807a051929b4604b9d31c2e/flink-runtime/src/main/java/org/apache/flink/runtime/rest/RestServerEndpoint.java#L207].
> Job submission via REST/dashboard does not work on Kubernetes
> -------------------------------------------------------------
>
> Key: FLINK-9211
> URL: https://issues.apache.org/jira/browse/FLINK-9211
> Project: Flink
> Issue Type: Bug
> Components: Client, Web Client
> Affects Versions: 1.5.0
> Reporter: Aljoscha Krettek
> Assignee: Aljoscha Krettek
> Priority: Blocker
> Fix For: 1.5.0
>
>
> When setting up a cluster on Kubernets according to the documentation it is possible to upload jar files but when trying to execute them you get an exception like this:
> {code}
> org.apache.flink.runtime.rest.handler.RestHandlerException: org.apache.flink.runtime.client.JobSubmissionException: Failed to submit JobGraph.
> at org.apache.flink.runtime.webmonitor.handlers.JarRunHandler.lambda$handleRequest$2(JarRunHandler.java:113)
> at java.util.concurrent.CompletableFuture.uniExceptionally(CompletableFuture.java:870)
> at java.util.concurrent.CompletableFuture$UniExceptionally.tryFire(CompletableFuture.java:852)
> at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474)
> at java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:1977)
> at org.apache.flink.runtime.rest.RestClient.lambda$submitRequest$1(RestClient.java:196)
> at org.apache.flink.shaded.netty4.io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:680)
> at org.apache.flink.shaded.netty4.io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:603)
> at org.apache.flink.shaded.netty4.io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:563)
> at org.apache.flink.shaded.netty4.io.netty.util.concurrent.DefaultPromise.tryFailure(DefaultPromise.java:424)
> at org.apache.flink.shaded.netty4.io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe$1.run(AbstractNioChannel.java:214)
> at org.apache.flink.shaded.netty4.io.netty.util.concurrent.PromiseTask$RunnableAdapter.call(PromiseTask.java:38)
> at org.apache.flink.shaded.netty4.io.netty.util.concurrent.ScheduledFutureTask.run(ScheduledFutureTask.java:120)
> at org.apache.flink.shaded.netty4.io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:357)
> at org.apache.flink.shaded.netty4.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:357)
> at org.apache.flink.shaded.netty4.io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:111)
> at org.apache.flink.shaded.netty4.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:137)
> at java.lang.Thread.run(Thread.java:748)
> Caused by: java.util.concurrent.CompletionException: org.apache.flink.runtime.client.JobSubmissionException: Failed to submit JobGraph.
> at org.apache.flink.client.program.rest.RestClusterClient.lambda$submitJob$5(RestClusterClient.java:356)
> ... 17 more
> Caused by: org.apache.flink.runtime.client.JobSubmissionException: Failed to submit JobGraph.
> ... 18 more
> Caused by: java.util.concurrent.CompletionException: org.apache.flink.shaded.netty4.io.netty.channel.ConnectTimeoutException: connection timed out: flink-jobmanager/10.105.154.28:8081
> at java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:292)
> at java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:308)
> at java.util.concurrent.CompletableFuture.uniCompose(CompletableFuture.java:943)
> at java.util.concurrent.CompletableFuture$UniCompose.tryFire(CompletableFuture.java:926)
> ... 15 more
> Caused by: org.apache.flink.shaded.netty4.io.netty.channel.ConnectTimeoutException: connection timed out: flink-jobmanager/10.105.154.28:8081
> at org.apache.flink.shaded.netty4.io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe$1.run(AbstractNioChannel.java:212)
> ... 7 more
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)