You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Dongjoon Hyun (Jira)" <ji...@apache.org> on 2024/09/17 03:54:00 UTC
[jira] [Assigned] (SPARK-49678) Support `spark.test.master` in `SparkSubmitArguments`
[ https://issues.apache.org/jira/browse/SPARK-49678?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Dongjoon Hyun reassigned SPARK-49678:
-------------------------------------
Assignee: Dongjoon Hyun
> Support `spark.test.master` in `SparkSubmitArguments`
> -----------------------------------------------------
>
> Key: SPARK-49678
> URL: https://issues.apache.org/jira/browse/SPARK-49678
> Project: Spark
> Issue Type: Sub-task
> Components: Spark Core
> Affects Versions: 4.0.0
> Reporter: Dongjoon Hyun
> Assignee: Dongjoon Hyun
> Priority: Major
> Labels: pull-request-available
>
> To allow users to control the default master setting during testing and documentation generation.
> BEFORE (`local[*]`)
> {code:java}
> $ bin/pyspark
> Python 3.9.19 (main, Jun 17 2024, 15:39:29)
> [Clang 15.0.0 (clang-1500.3.9.4)] on darwin
> Type "help", "copyright", "credits" or "license" for more information.
> WARNING: Using incubator modules: jdk.incubator.vector
> Using Spark's default log4j profile: org/apache/spark/log4j2-pattern-layout-defaults.properties
> Setting default log level to "WARN".
> To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel).
> 24/09/16 13:53:02 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
> Welcome to
> ____ __
> / _/_ ___ ____/ /_
> \ \/ _ \/ _ `/ __/ '/
> /__ / ._/_,// //_\ version 4.0.0-SNAPSHOT
> /_/
> Using Python version 3.9.19 (main, Jun 17 2024 15:39:29)
> Spark context Web UI available at http://localhost:4040
> Spark context available as 'sc' (master = local[*], app id = local-1726519982935).
> SparkSession available as 'spark'.
> >>>{code}
>
> AFTER (`local[1]`)
> {code:java}
> $ JDK_JAVA_OPTIONS="-Dspark.test.master=local[1]" bin/pyspark
> NOTE: Picked up JDK_JAVA_OPTIONS: -Dspark.test.master=local[1]
> Python 3.9.19 (main, Jun 17 2024, 15:39:29)
> [Clang 15.0.0 (clang-1500.3.9.4)] on darwin
> Type "help", "copyright", "credits" or "license" for more information.
> NOTE: Picked up JDK_JAVA_OPTIONS: -Dspark.test.master=local[1]
> NOTE: Picked up JDK_JAVA_OPTIONS: -Dspark.test.master=local[1]
> WARNING: Using incubator modules: jdk.incubator.vector
> Using Spark's default log4j profile: org/apache/spark/log4j2-pattern-layout-defaults.properties
> Setting default log level to "WARN".
> To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel).
> 24/09/16 13:51:03 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
> Welcome to
> ____ __
> / _/_ ___ ____/ /_
> \ \/ _ \/ _ `/ __/ '/
> /__ / ._/_,// //_\ version 4.0.0-SNAPSHOT
> /_/
> Using Python version 3.9.19 (main, Jun 17 2024 15:39:29)
> Spark context Web UI available at http://localhost:4040
> Spark context available as 'sc' (master = local[1], app id = local-1726519863363).
> SparkSession available as 'spark'.
> >>>{code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org