You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Diego García (Jira)" <ji...@apache.org> on 2019/09/05 08:37:00 UTC
[jira] [Comment Edited] (SPARK-27491) SPARK REST API -
"org.apache.spark.deploy.SparkSubmit --status" returns empty response!
therefore Airflow won't integrate with Spark 2.3.x
[ https://issues.apache.org/jira/browse/SPARK-27491?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16923139#comment-16923139 ]
Diego García edited comment on SPARK-27491 at 9/5/19 8:36 AM:
--------------------------------------------------------------
In 2.4.4 still not working, the command spark-submit --status returns nothing. Our workaround is overriding the _build_track_driver_status_command of spark_submit_operator to get the driver's status using the REST API of the master as detailed in https://issues.apache.org/jira/browse/AIRFLOW-5385
Moreover, this REST API approach causes a dramatic decrease on Airflow workers CPU use (as shown in the previous link). This is because we have >5 Spark Streaming applications running, and therefore, the worker is constantly tracking driver status (calling spark-submit per second).
In 2.4.4 the REST API is disabled by default, it has to be enabled in the master conf:
spark.master.rest.enabled true
Another issue related: https://issues.apache.org/jira/browse/SPARK-24888
was (Author: diego garcía):
In 2.4.4 still not working, the command spark-submit --status returns nothing. Our workaround is overriding the _build_track_driver_status_command of spark_submit_operator to get the driver's status using the REST API of the master as detailed in https://issues.apache.org/jira/browse/AIRFLOW-5385
In 2.4.4 the REST API is disabled by default, it has to be enabled in the master conf:
spark.master.rest.enabled true
Another issue related: https://issues.apache.org/jira/browse/SPARK-24888
> SPARK REST API - "org.apache.spark.deploy.SparkSubmit --status" returns empty response! therefore Airflow won't integrate with Spark 2.3.x
> ------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: SPARK-27491
> URL: https://issues.apache.org/jira/browse/SPARK-27491
> Project: Spark
> Issue Type: Bug
> Components: Java API, Scheduler, Spark Core, Spark Shell, Spark Submit
> Affects Versions: 2.3.3
> Reporter: t oo
> Priority: Major
>
> This issue must have been introduced after Spark 2.1.1 as it is working in that version. This issue is affecting me in Spark 2.3.3/2.3.0. I am using spark standalone mode if that makes a difference.
> See below spark 2.3.3 returns empty response while 2.1.1 returns a response.
>
> Spark 2.1.1:
> [ec2here@ip-x-y-160-225 ~]$ bash -x /home/ec2here/spark_home1/bin/spark-class org.apache.spark.deploy.SparkSubmit --master spark://domainhere:6066 --status driver-20190417130324-0009
> + export SPARK_HOME=/home/ec2here/spark_home1
> + SPARK_HOME=/home/ec2here/spark_home1
> + '[' -z /home/ec2here/spark_home1 ']'
> + . /home/ec2here/spark_home1/bin/load-spark-env.sh
> ++ '[' -z /home/ec2here/spark_home1 ']'
> ++ '[' -z '' ']'
> ++ export SPARK_ENV_LOADED=1
> ++ SPARK_ENV_LOADED=1
> ++ parent_dir=/home/ec2here/spark_home1
> ++ user_conf_dir=/home/ec2here/spark_home1/conf
> ++ '[' -f /home/ec2here/spark_home1/conf/spark-env.sh ']'
> ++ set -a
> ++ . /home/ec2here/spark_home1/conf/spark-env.sh
> +++ export JAVA_HOME=/usr/lib/jvm/jre-1.8.0-openjdk.x86_64
> +++ JAVA_HOME=/usr/lib/jvm/jre-1.8.0-openjdk.x86_64
> ++++ ulimit -n 1048576
> ++ set +a
> ++ '[' -z '' ']'
> ++ ASSEMBLY_DIR2=/home/ec2here/spark_home1/assembly/target/scala-2.11
> ++ ASSEMBLY_DIR1=/home/ec2here/spark_home1/assembly/target/scala-2.10
> ++ [[ -d /home/ec2here/spark_home1/assembly/target/scala-2.11 ]]
> ++ '[' -d /home/ec2here/spark_home1/assembly/target/scala-2.11 ']'
> ++ export SPARK_SCALA_VERSION=2.10
> ++ SPARK_SCALA_VERSION=2.10
> + '[' -n /usr/lib/jvm/jre-1.8.0-openjdk.x86_64 ']'
> + RUNNER=/usr/lib/jvm/jre-1.8.0-openjdk.x86_64/bin/java
> + '[' -d /home/ec2here/spark_home1/jars ']'
> + SPARK_JARS_DIR=/home/ec2here/spark_home1/jars
> + '[' '!' -d /home/ec2here/spark_home1/jars ']'
> + LAUNCH_CLASSPATH='/home/ec2here/spark_home1/jars/*'
> + '[' -n '' ']'
> + [[ -n '' ]]
> + CMD=()
> + IFS=
> + read -d '' -r ARG
> ++ build_command org.apache.spark.deploy.SparkSubmit --master spark://domainhere:6066 --status driver-20190417130324-0009
> ++ /usr/lib/jvm/jre-1.8.0-openjdk.x86_64/bin/java -Xmx128m -cp '/home/ec2here/spark_home1/jars/*' org.apache.spark.launcher.Main org.apache.spark.deploy.SparkSubmit --master spark://domainhere:6066 --status driver-20190417130324-0009
> + CMD+=("$ARG")
> + IFS=
> + read -d '' -r ARG
> + CMD+=("$ARG")
> + IFS=
> + read -d '' -r ARG
> + CMD+=("$ARG")
> + IFS=
> + read -d '' -r ARG
> + CMD+=("$ARG")
> + IFS=
> + read -d '' -r ARG
> + CMD+=("$ARG")
> + IFS=
> + read -d '' -r ARG
> + CMD+=("$ARG")
> + IFS=
> + read -d '' -r ARG
> + CMD+=("$ARG")
> + IFS=
> + read -d '' -r ARG
> + CMD+=("$ARG")
> + IFS=
> + read -d '' -r ARG
> + CMD+=("$ARG")
> + IFS=
> + read -d '' -r ARG
> ++ printf '%d\0' 0
> + CMD+=("$ARG")
> + IFS=
> + read -d '' -r ARG
> + COUNT=10
> + LAST=9
> + LAUNCHER_EXIT_CODE=0
> + [[ 0 =~ ^[0-9]+$ ]]
> + '[' 0 '!=' 0 ']'
> + CMD=("${CMD[@]:0:$LAST}")
> + exec /usr/lib/jvm/jre-1.8.0-openjdk.x86_64/bin/java -cp '/home/ec2here/spark_home1/conf/:/home/ec2here/spark_home1/jars/*' -Xmx2048m org.apache.spark.deploy.SparkSubmit --master spark://domainhere:6066 --status driver-20190417130324-0009
> Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties
> 19/04/17 14:03:27 INFO RestSubmissionClient: Submitting a request for the status of submission driver-20190417130324-0009 in spark://domainhere:6066.
> 19/04/17 14:03:28 INFO RestSubmissionClient: Server responded with SubmissionStatusResponse:
> {
> "action" : "SubmissionStatusResponse",
> "driverState" : "FAILED",
> "serverSparkVersion" : "2.3.3",
> "submissionId" : "driver-20190417130324-0009",
> "success" : true,
> "workerHostPort" : "x.y.211.40:11819",
> "workerId" : "worker-20190417115840-x.y.211.40-11819"
> }
> [ec2here@ip-x-y-160-225 ~]$
>
> Spark 2.3.3:
> [ec2here@ip-x-y-160-225 ~]$ bash -x /home/ec2here/spark_home/bin/spark-class org.apache.spark.deploy.SparkSubmit --master spark://domainhere:6066 --status driver-20190417130324-0009
> + '[' -z '' ']'
> ++ dirname /home/ec2here/spark_home/bin/spark-class
> + source /home/ec2here/spark_home/bin/find-spark-home
> ++++ dirname /home/ec2here/spark_home/bin/spark-class
> +++ cd /home/ec2here/spark_home/bin
> +++ pwd
> ++ FIND_SPARK_HOME_PYTHON_SCRIPT=/home/ec2here/spark_home/bin/find_spark_home.py
> ++ '[' '!' -z '' ']'
> ++ '[' '!' -f /home/ec2here/spark_home/bin/find_spark_home.py ']'
> ++++ dirname /home/ec2here/spark_home/bin/spark-class
> +++ cd /home/ec2here/spark_home/bin/..
> +++ pwd
> ++ export SPARK_HOME=/home/ec2here/spark_home
> ++ SPARK_HOME=/home/ec2here/spark_home
> + . /home/ec2here/spark_home/bin/load-spark-env.sh
> ++ '[' -z /home/ec2here/spark_home ']'
> ++ '[' -z '' ']'
> ++ export SPARK_ENV_LOADED=1
> ++ SPARK_ENV_LOADED=1
> ++ export SPARK_CONF_DIR=/home/ec2here/spark_home/conf
> ++ SPARK_CONF_DIR=/home/ec2here/spark_home/conf
> ++ '[' -f /home/ec2here/spark_home/conf/spark-env.sh ']'
> ++ set -a
> ++ . /home/ec2here/spark_home/conf/spark-env.sh
> +++ export JAVA_HOME=/usr/lib/jvm/jre-1.8.0-openjdk.x86_64
> +++ JAVA_HOME=/usr/lib/jvm/jre-1.8.0-openjdk.x86_64
> ++++ ulimit -n 1048576
> ++ set +a
> ++ '[' -z '' ']'
> ++ ASSEMBLY_DIR2=/home/ec2here/spark_home/assembly/target/scala-2.11
> ++ ASSEMBLY_DIR1=/home/ec2here/spark_home/assembly/target/scala-2.12
> ++ [[ -d /home/ec2here/spark_home/assembly/target/scala-2.11 ]]
> ++ '[' -d /home/ec2here/spark_home/assembly/target/scala-2.11 ']'
> ++ export SPARK_SCALA_VERSION=2.12
> ++ SPARK_SCALA_VERSION=2.12
> + '[' -n /usr/lib/jvm/jre-1.8.0-openjdk.x86_64 ']'
> + RUNNER=/usr/lib/jvm/jre-1.8.0-openjdk.x86_64/bin/java
> + '[' -d /home/ec2here/spark_home/jars ']'
> + SPARK_JARS_DIR=/home/ec2here/spark_home/jars
> + '[' '!' -d /home/ec2here/spark_home/jars ']'
> + LAUNCH_CLASSPATH='/home/ec2here/spark_home/jars/*'
> + '[' -n '' ']'
> + [[ -n '' ]]
> + set +o posix
> + CMD=()
> + IFS=
> + read -d '' -r ARG
> ++ build_command org.apache.spark.deploy.SparkSubmit --master spark://domainhere:6066 --status driver-20190417130324-0009
> ++ /usr/lib/jvm/jre-1.8.0-openjdk.x86_64/bin/java -Xmx128m -cp '/home/ec2here/spark_home/jars/*' org.apache.spark.launcher.Main org.apache.spark.deploy.SparkSubmit --master spark://domainhere:6066 --status driver-20190417130324-0009
> + CMD+=("$ARG")
> + IFS=
> + read -d '' -r ARG
> + CMD+=("$ARG")
> + IFS=
> + read -d '' -r ARG
> + CMD+=("$ARG")
> + IFS=
> + read -d '' -r ARG
> + CMD+=("$ARG")
> + IFS=
> + read -d '' -r ARG
> + CMD+=("$ARG")
> + IFS=
> + read -d '' -r ARG
> + CMD+=("$ARG")
> + IFS=
> + read -d '' -r ARG
> + CMD+=("$ARG")
> + IFS=
> + read -d '' -r ARG
> + CMD+=("$ARG")
> + IFS=
> + read -d '' -r ARG
> + CMD+=("$ARG")
> + IFS=
> + read -d '' -r ARG
> ++ printf '%d\0' 0
> + CMD+=("$ARG")
> + IFS=
> + read -d '' -r ARG
> + COUNT=10
> + LAST=9
> + LAUNCHER_EXIT_CODE=0
> + [[ 0 =~ ^[0-9]+$ ]]
> + '[' 0 '!=' 0 ']'
> + CMD=("${CMD[@]:0:$LAST}")
> + exec /usr/lib/jvm/jre-1.8.0-openjdk.x86_64/bin/java -cp '/home/ec2here/spark_home/conf/:/home/ec2here/spark_home/jars/*' -Xmx2048m org.apache.spark.deploy.SparkSubmit --master spark://domainhere:6066 --status driver-20190417130324-0009
> [ec2here@ip-x-y-160-225 ~]$ ps -ef | grep -i spark
>
> This means Apache Airflow does not work with spark 2.3.x as the spark submit operator stays in running state forever as it does not get response from spark rest status calls.
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org