You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "James Carter (JIRA)" <ji...@apache.org> on 2018/08/20 17:56:00 UTC
[jira] [Created] (SPARK-25162) Kubernetes 'in-cluster' client mode
and value spark.driver.host
James Carter created SPARK-25162:
------------------------------------
Summary: Kubernetes 'in-cluster' client mode and value spark.driver.host
Key: SPARK-25162
URL: https://issues.apache.org/jira/browse/SPARK-25162
Project: Spark
Issue Type: Bug
Components: Kubernetes
Affects Versions: 2.4.0
Environment: A java program, deployed to kubernetes, that establishes a Spark Context in client mode.
Not using spark-submit.
Kubernetes 1.10
AWS EKS
Reporter: James Carter
When creating Kubernetes scheduler 'in-cluster' using client mode, the value for spark.driver.host can be derived from the IP address of the driver pod.
I observed that the value of _spark.driver.host_ defaulted to the value of _spark.kubernetes.driver.pod.name_, which is not a valid hostname. This caused the executors to fail to establish a connection back to the driver.
As a work around, in my configuration I pass the driver's pod name _and_ the driver's ip address to ensure that executors can establish a connection with the driver.
_spark.kubernetes.driver.pod.name_ := env.valueFrom.fieldRef.fieldPath: metadata.name
_spark.driver.host_ := env.valueFrom.fieldRef.fieldPath: status.podIp
Ideally only _spark.kubernetes.driver.pod.name_ need be provided in this deployment scenario.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org