You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Stavros Kontopoulos (Jira)" <ji...@apache.org> on 2019/09/03 10:26:00 UTC
[jira] [Comment Edited] (SPARK-28921) Spark jobs failing on latest
versions of Kubernetes (1.15.3, 1.14.6, 1,13.10, 1.12.10, 1.11.10)
[ https://issues.apache.org/jira/browse/SPARK-28921?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16921315#comment-16921315 ]
Stavros Kontopoulos edited comment on SPARK-28921 at 9/3/19 10:25 AM:
----------------------------------------------------------------------
[~andygrove] could you please clarify what do you mean when you say? "jobs like Spark-Pi that do not launch executors run without a problem"
I run a pi job and it creates executors fine:
spark-pi-03afbd6cf6a72622-driver 1/1 Running 0 15s
spark-pi-03afbd6cf6a72622-exec-1 1/1 Running 0 7s
spark-pi-03afbd6cf6a72622-exec-2 1/1 Running 0 7s
was (Author: skonto):
[~andygrove] could you please clarify what do you mean when you say? "jobs like Spark-Pi that do not launch executors run without a problem"
I run a pi job and it creates executors fine.
> Spark jobs failing on latest versions of Kubernetes (1.15.3, 1.14.6, 1,13.10, 1.12.10, 1.11.10)
> -----------------------------------------------------------------------------------------------
>
> Key: SPARK-28921
> URL: https://issues.apache.org/jira/browse/SPARK-28921
> Project: Spark
> Issue Type: Bug
> Components: Kubernetes
> Affects Versions: 2.3.3, 2.4.3
> Reporter: Paul Schweigert
> Assignee: Andy Grove
> Priority: Major
> Fix For: 2.4.5, 3.0.0
>
>
> Spark jobs are failing on latest versions of Kubernetes when jobs attempt to provision executor pods (jobs like Spark-Pi that do not launch executors run without a problem):
>
> Here's an example error message:
>
> {code:java}
> 19/08/30 01:29:09 INFO ExecutorPodsAllocator: Going to request 2 executors from Kubernetes.
> 19/08/30 01:29:09 INFO ExecutorPodsAllocator: Going to request 2 executors from Kubernetes.19/08/30 01:29:09 WARN WatchConnectionManager: Exec Failure: HTTP 403, Status: 403 -
> java.net.ProtocolException: Expected HTTP 101 response but was '403 Forbidden'
> at okhttp3.internal.ws.RealWebSocket.checkResponse(RealWebSocket.java:216)
> at okhttp3.internal.ws.RealWebSocket$2.onResponse(RealWebSocket.java:183)
> at okhttp3.RealCall$AsyncCall.execute(RealCall.java:141)
> at okhttp3.internal.NamedRunnable.run(NamedRunnable.java:32)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> {code}
>
> Looks like the issue is caused by fixes for a recent CVE :
> CVE: [https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2019-14809]
> Fix: [https://github.com/fabric8io/kubernetes-client/pull/1669]
>
> Looks like upgrading kubernetes-client to 4.4.2 would solve this issue.
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org