You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Dongjoon Hyun (Jira)" <ji...@apache.org> on 2022/12/09 10:08:00 UTC
[jira] [Resolved] (SPARK-41377) Fix spark-version-info.properties not found on Windows
[ https://issues.apache.org/jira/browse/SPARK-41377?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Dongjoon Hyun resolved SPARK-41377.
-----------------------------------
Fix Version/s: 3.4.0
Resolution: Fixed
This is resolved via https://github.com/apache/spark/pull/38903
> Fix spark-version-info.properties not found on Windows
> ------------------------------------------------------
>
> Key: SPARK-41377
> URL: https://issues.apache.org/jira/browse/SPARK-41377
> Project: Spark
> Issue Type: Bug
> Components: Spark Core
> Affects Versions: 3.4.0
> Environment: Windows 10
> Reporter: Gautham Banasandra
> Priority: Major
> Fix For: 3.4.0
>
>
> h1. *What is the problem?*
> The spark-version-info.properties files is read during initialization of SparkContext. The absence of the same causes the Spark AM's launch to fail with the following error -
> {code}
> 22/12/04 17:18:24 ERROR ApplicationMaster: User class threw exception:
> java.lang.ExceptionInInitializerError
> at org.apache.spark.package$.<init>(package.scala:93)
> at org.apache.spark.package$.<clinit>(package.scala)
> at org.apache.spark.SparkContext.$anonfun$new$1(SparkContext.scala:195)
> at org.apache.spark.internal.Logging.logInfo(Logging.scala:60)
> at org.apache.spark.internal.Logging.logInfo$(Logging.scala:59)
> at org.apache.spark.SparkContext.logInfo(SparkContext.scala:84)
> at org.apache.spark.SparkContext.<init>(SparkContext.scala:195)
> at org.apache.spark.SparkContext$.getOrCreate(SparkContext.scala:2739)
> at org.apache.spark.sql.SparkSession$Builder.$anonfun$getOrCreate$2(SparkSession.scala:978)
> at scala.Option.getOrElse(Option.scala:189)
> at org.apache.spark.sql.SparkSession$Builder.getOrCreate(SparkSession.scala:972)
> at org.apache.spark.examples.SparkPi$.main(SparkPi.scala:30)
> at org.apache.spark.examples.SparkPi.main(SparkPi.scala)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at org.apache.spark.deploy.yarn.ApplicationMaster$$anon$2.run(ApplicationMaster.scala:747)
> Caused by: org.apache.spark.SparkException: Could not find spark-version-info.properties
> at org.apache.spark.package$SparkBuildInfo$.<init>(package.scala:62)
> at org.apache.spark.package$SparkBuildInfo$.<clinit>(package.scala)
> ... 18 more
> {code}
> h1. *Why did the problem happen?*
> While building Spark, the spark-version-info.properties file [is generated using bash|https://github.com/apache/spark/blob/d62c18b7497997188ec587e1eb62e75c979c1c93/core/pom.xml#L560-L564]. In Windows environment, if Windows Subsystem for Linux (WSL) is installed, it somehow overrides the other bash executables in the PATH, as noted in SPARK-40739. The bash in WSL has a different mounting configuration and thus, the target location specified for spark-version-info.properties won't be the expected location. Ultimately, this leads to spark-version-info.properties to get excluded from the spark-core jar, thus causing the SparkContext initialization to fail with the above depicted error message.
> h1. *How can the problem be solved?*
> I propose to fix this issue by enhancing the build system to automatically detect and switch between using Powershell for Windows and Bash for non-Windows OS. This yields a seamless build experience for all the platform users. Particularly for Windows users, to not worry about the conflicts with different bash installations.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org