You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Sean R. Owen (Jira)" <ji...@apache.org> on 2019/10/26 22:57:00 UTC
[jira] [Commented] (SPARK-29077) submitting a SparkSession job
fails on a spark://localhost:7077 url on Mac
[ https://issues.apache.org/jira/browse/SPARK-29077?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16960457#comment-16960457 ]
Sean R. Owen commented on SPARK-29077:
--------------------------------------
You'll need to open a pull request.
> submitting a SparkSession job fails on a spark://localhost:7077 url on Mac
> ---------------------------------------------------------------------------
>
> Key: SPARK-29077
> URL: https://issues.apache.org/jira/browse/SPARK-29077
> Project: Spark
> Issue Type: Bug
> Components: Spark Submit
> Affects Versions: 2.4.3
> Environment: Darwin <hostname.local> 18.7.0 Darwin Kernel Version 18.7.0: Thu Jun 20
> PDT 2019; root:xnu-4903.270.47~4/RELEASE_X86_64 x86_64
> Reporter: alex
> Priority: Major
>
> When creating a spark context running on a local host getting connection refused error, however using the actual host name for example: "spark://myhostname.local:7077" works
> {code:java}
> performance-meter {
> spark {
> appname = "test-harness"
> master = "spark://localhost:7077"
> }
> } {code}
> {code:java}
> val configRoot = "performance-meter"
> val sparkSession = SparkSession.builder
> .appName(conf.getString(s"${configRoot}.spark.appname"))
> .master(conf.getString(s"${configRoot}.spark.master")) {code}
>
> This appears to be due to some Macs having multiple network interfaces, at least is the case on my Mac. Recommended fix that seems to work locally:
> in file: /usr/local/Cellar/apache-spark/2.4.3/libexec/sbin/start-master.sh & /usr/local/Cellar/apache-spark/2.4.3/libexec/sbin/start-slaves.sh add a case section for "Darwin"
> {code:bash}
> if [ "$SPARK_MASTER_HOST" = "" ]; then
> case `uname` in
> (SunOS)
> SPARK_MASTER_HOST="`/usr/sbin/check-hostname | awk '{print $NF}'`"
> ;;
> (Darwin)
> SPARK_MASTER_HOST="localhost" # 13-Sep-2019 alexshagiev add Mac (Darwin) case to ensure spark binds on local host interface instead of external interface.
> ;;
> (*)
> SPARK_MASTER_HOST="`hostname -f`"
> ;;
> esac
> fi
> {code}
>
>
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org