You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Yinan Li (JIRA)" <ji...@apache.org> on 2018/04/19 21:55:00 UTC
[jira] [Commented] (SPARK-24028) [K8s] Creating secrets and config
maps before creating the driver pod has unpredictable behavior
[ https://issues.apache.org/jira/browse/SPARK-24028?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16444856#comment-16444856 ]
Yinan Li commented on SPARK-24028:
----------------------------------
I am also running a 1.9 cluster on GKE and I have never run into the issue you mentioned above. I do often see events on the driver pod showing that the configmap failed to mount, but eventually retries just succeeded. I believe a pod won't start running if any of the specified volumes (being it a secret volume, a configmap volume, or something else) fail to mount, and Kubernetes also retries mounting volumes that it failed to mount when the pod first started.
> [K8s] Creating secrets and config maps before creating the driver pod has unpredictable behavior
> ------------------------------------------------------------------------------------------------
>
> Key: SPARK-24028
> URL: https://issues.apache.org/jira/browse/SPARK-24028
> Project: Spark
> Issue Type: Bug
> Components: Kubernetes
> Affects Versions: 2.3.0
> Reporter: Matt Cheah
> Priority: Critical
>
> Currently we create the Kubernetes resources the driver depends on - such as the properties config map and secrets to mount into the pod - only after we create the driver pod. This is because we want these extra objects to immediately have an owner reference to be tied to the driver pod.
> On our Kubernetes 1.9.4. cluster, we're seeing that sometimes this works fine, but other times the driver ends up being started with empty volumes instead of volumes with the contents of the secrets we expect. The result is that sometimes the driver will start without these files mounted, which leads to various failures if the driver requires these files to be present early on in their code. Missing the properties file config map, for example, would mean spark-submit doesn't have a properties file to read at all. See the warning on [https://kubernetes.io/docs/concepts/storage/volumes/#secret.]
> Unfortunately we cannot link owner references to non-existent objects, so we have to do this instead:
> # Create the auxiliary resources without any owner references.
> # Create the driver pod mounting these resources into volumes, as before.
> # If #2 fails, clean up the resources created in #1.
> # Edit the auxiliary resources to have an owner reference for the driver pod.
> The multi-step approach leaves a small chance for us to leak resources - for example, if we fail to make the resource edits in #4 for some reason. This also changes the permissioning mode required for spark-submit - credentials provided to spark-submit need to be able to edit resources in addition to creating them.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org