You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Marcelo Vanzin (JIRA)" <ji...@apache.org> on 2019/02/12 19:45:00 UTC
[jira] [Resolved] (SPARK-26400) [k8s] Init container silently
swallows errors when fetching jars from remote url
[ https://issues.apache.org/jira/browse/SPARK-26400?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Marcelo Vanzin resolved SPARK-26400.
------------------------------------
Resolution: Won't Fix
Init containers were removed in 2.4, so standard Spark error handling for these errors applies.
> [k8s] Init container silently swallows errors when fetching jars from remote url
> --------------------------------------------------------------------------------
>
> Key: SPARK-26400
> URL: https://issues.apache.org/jira/browse/SPARK-26400
> Project: Spark
> Issue Type: Bug
> Components: Kubernetes
> Affects Versions: 2.3.2
> Reporter: Stanis Shkel
> Priority: Minor
>
> I run the following command
> {code:bash}
> spark-2.3.2-bin-hadoop2.7/bin/spark-submit --name client \
> --master "k8s://cluster" \
> --deploy-mode cluster \
> --conf spark.executor.instances=2 \
> --conf spark.executor.memory=5G \
> --conf spark.driver.memory=8G \
> --conf spark.kubernetes.container.image=rego.azurecr.io/spark:spark-2.3.2-hadoop2.7 \
> --class au.com.random.DoesntMatter \
> "https://fake-link.com/jars/my.jar"
> {code}
> I expect init container to fail to download jar and get a failure in the init stage. Instead I get driver failure with the following message.
> {code:bash}
> ++ id -u
> + myuid=0
> ++ id -g
> + mygid=0
> ++ getent passwd 0
> + uidentry=root:x:0:0:root:/root:/bin/ash
> + '[' -z root:x:0:0:root:/root:/bin/ash ']'
> + SPARK_K8S_CMD=driver
> + '[' -z driver ']'
> + shift 1
> + SPARK_CLASSPATH=':/opt/spark/jars/*'
> + env
> + grep SPARK_JAVA_OPT_
> + sed 's/[^=]*=\(.*\)/\1/g'
> + sort -t_ -k4 -n
> + readarray -t SPARK_JAVA_OPTS
> + '[' -n /var/spark-data/spark-jars/my.jar:/var/spark-data/spark-jars/my.jar ']'
> + SPARK_CLASSPATH=':/opt/spark/jars/*:/var/spark-data/spark-jars/my.jar:/var/spark-data/spark-jars/my.jar'
> + '[' -n /var/spark-data/spark-files ']'
> + cp -R /var/spark-data/spark-files/. .
> + case "$SPARK_K8S_CMD" in
> + CMD=(${JAVA_HOME}/bin/java "${SPARK_JAVA_OPTS[@]}" -cp "$SPARK_CLASSPATH" -Xms$SPARK_DRIVER_MEMORY -Xmx$SPARK_DRIVER_MEMORY -Dspark.driver.bindAddress=$SPARK_DRIVER_BIND_ADDRESS $SPARK_DRIVER_CLASS $SPARK_DRIVER_ARGS)
> + exec /sbin/tini -s -- /usr/lib/jvm/java-1.8-openjdk/bin/java -Dspark.master=k8s://kubernetes:443 -Dspark.app.id=spark-2f340a028a314e9cb0df8165d887bfb7 -Dspark.kubernetes.container.image=azure.azurecr.io/spark:spark-2.3.2-hadoop2.7 -Dspark.submit.deployMode=cluster -Dspark.driver.blockManager.port=7079 -Dspark.executor.memory=5G -Dspark.kubernetes.executor.podNamePrefix=client-f20f30e154a13624a728d6f56d45da3e -Dspark.jars=https://fake-link.com/jars/my.jar,https://fake-link.com/jars/my.jar -Dspark.driver.memory=8G -Dspark.driver.port=7078 -Dspark.kubernetes.driver.pod.name=client-f20f30e154a13624a728d6f56d45da3e-driver -Dspark.app.name=client -Dspark.kubernetes.initContainer.configMapKey=spark-init.properties -Dspark.executor.instances=2 -Dspark.driver.host=client-f20f30e154a13624a728d6f56d45da3e-driver-svc.default.svc -Dspark.kubernetes.initContainer.configMapName=client-f20f30e154a13624a728d6f56d45da3e-init-config -cp ':/opt/spark/jars/*:/var/spark-data/spark-jars/my.jar:/var/spark-data/spark-jars/my.jar' -Xms8G -Xmx8G -Dspark.driver.bindAddress=10.1.0.101 au.com.random.DoesntMatter
> Error: Could not find or load main class au.com.random.DoesntMatter
> {code}
> This happens because spark-init container failed to download the dependencies but misreports the status. Here is a log snippet from spark-init container
> {code:bash}
> ++ id -u
> + myuid=0
> ++ id -g
> + mygid=0
> ++ getent passwd 0
> + uidentry=root:x:0:0:root:/root:/bin/ash
> + '[' -z root:x:0:0:root:/root:/bin/ash ']'
> + SPARK_K8S_CMD=init
> + '[' -z init ']'
> + shift 1
> + SPARK_CLASSPATH=':/opt/spark/jars/*'
> + env
> + grep SPARK_JAVA_OPT_
> + sed 's/[^=]*=\(.*\)/\1/g'
> + sort -t_ -k4 -n
> + readarray -t SPARK_JAVA_OPTS
> + '[' -n '' ']'
> + '[' -n '' ']'
> + case "$SPARK_K8S_CMD" in
> + CMD=("$SPARK_HOME/bin/spark-class" "org.apache.spark.deploy.k8s.SparkPodInitContainer" "$@")
> + exec /sbin/tini -s -- /opt/spark/bin/spark-class org.apache.spark.deploy.k8s.SparkPodInitContainer /etc/spark-init/spark-init.properties
> 2018-12-18 21:15:41 INFO SparkPodInitContainer:54 - Starting init-container to download Spark application dependencies.
> 2018-12-18 21:15:41 WARN NativeCodeLoader:62 - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
> 2018-12-18 21:15:41 INFO SecurityManager:54 - Changing view acls to: root
> 2018-12-18 21:15:41 INFO SecurityManager:54 - Changing modify acls to: root
> 2018-12-18 21:15:41 INFO SecurityManager:54 - Changing view acls groups to:
> 2018-12-18 21:15:41 INFO SecurityManager:54 - Changing modify acls groups to:
> 2018-12-18 21:15:41 INFO SecurityManager:54 - SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(root); groups with view permissions: Set(); users with modify permissions: Set(root); groups with modify permissions: Set()
> 2018-12-18 21:15:41 INFO SparkPodInitContainer:54 - Downloading remote jars: Some(https://fake-link.com/jars/my.jar,https://fake-link.com/jars/my.jar)
> 2018-12-18 21:15:41 INFO SparkPodInitContainer:54 - Downloading remote files: None
> 2018-12-18 21:15:42 INFO SparkPodInitContainer:54 - Finished downloading application dependencies.
> {code}
> I think the problem resides somewhere around here(I have little experience with scala, so might be wrong)
> [https://github.com/apache/spark/blob/branch-2.3/resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/SparkPodInitContainer.scala#L80]
> If I remove Future wrapper around fileFetcher I get expected behaviour. My driver pod will fail with Init:Error and I get the error thrown properly. PS the error below is from a different command, where I was trying to pull from a blob, but similar issue.
> {code:bash}
> Exception in thread "main" org.apache.hadoop.fs.azure.AzureException: org.apache.hadoop.fs.azure.AzureException: Container qrefinery in account test.blob.core.windows.net not found, and we can't create it using anoynomous credentials.
> at org.apache.hadoop.fs.azure.AzureNativeFileSystemStore.createAzureStorageSession(AzureNativeFileSystemStore.java:938)
> at org.apache.hadoop.fs.azure.AzureNativeFileSystemStore.initialize(AzureNativeFileSystemStore.java:438)
> at org.apache.hadoop.fs.azure.NativeAzureFileSystem.initialize(NativeAzureFileSystem.java:1048)
> at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2669)
> at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:94)
> at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2703)
> at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2685)
> at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:373)
> at org.apache.spark.util.Utils$.getHadoopFileSystem(Utils.scala:1910)
> at org.apache.spark.util.Utils$.doFetchFile(Utils.scala:700)
> at org.apache.spark.util.Utils$.fetchFile(Utils.scala:492)
> at org.apache.spark.deploy.k8s.FileFetcher.fetchFile(SparkPodInitContainer.scala:91)
> at org.apache.spark.deploy.k8s.SparkPodInitContainer$$anonfun$downloadFiles$1$$anonfun$apply$2.apply(SparkPodInitContainer.scala:81)
> at org.apache.spark.deploy.k8s.SparkPodInitContainer$$anonfun$downloadFiles$1$$anonfun$apply$2.apply(SparkPodInitContainer.scala:79)
> at scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:33)
> at scala.collection.mutable.WrappedArray.foreach(WrappedArray.scala:35)
> at org.apache.spark.deploy.k8s.SparkPodInitContainer$$anonfun$downloadFiles$1.apply(SparkPodInitContainer.scala:79)
> at org.apache.spark.deploy.k8s.SparkPodInitContainer$$anonfun$downloadFiles$1.apply(SparkPodInitContainer.scala:77)
> at scala.Option.foreach(Option.scala:257)
> at org.apache.spark.deploy.k8s.SparkPodInitContainer.downloadFiles(SparkPodInitContainer.scala:77)
> at org.apache.spark.deploy.k8s.SparkPodInitContainer.run(SparkPodInitContainer.scala:56)
> at org.apache.spark.deploy.k8s.SparkPodInitContainer$.main(SparkPodInitContainer.scala:113)
> at org.apache.spark.deploy.k8s.SparkPodInitContainer.main(SparkPodInitContainer.scala)
> Caused by: org.apache.hadoop.fs.azure.AzureException: Container qrefinery in account jr3e3d.blob.core.windows.net not found, and we can't create it using anoynomous credentials.
> at org.apache.hadoop.fs.azure.AzureNativeFileSystemStore.connectUsingAnonymousCredentials(AzureNativeFileSystemStore.java:730)
> at org.apache.hadoop.fs.azure.AzureNativeFileSystemStore.createAzureStorageSession(AzureNativeFileSystemStore.java:933)
> ... 22 more
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org