[jira] [Created] (SPARK-41377) Fix spark-version-info.properties not found on Windows

["Gautham Banasandra (Jira)" <ji...@apache.org>]

Gautham Banasandra created SPARK-41377:
------------------------------------------

             Summary: Fix spark-version-info.properties not found on Windows
                 Key: SPARK-41377
                 URL: https://issues.apache.org/jira/browse/SPARK-41377
             Project: Spark
          Issue Type: Bug
          Components: Spark Core
    Affects Versions: 3.4.0
         Environment: Windows 10
            Reporter: Gautham Banasandra


h1. *What is the problem?*

The spark-version-info.properties files is read during initialization of SparkContext. The absence of the same causes the Spark AM's launch to fail with the following error -
{code}
22/12/04 17:18:24 ERROR ApplicationMaster: User class threw exception: 
java.lang.ExceptionInInitializerError
	at org.apache.spark.package$.<init>(package.scala:93)
	at org.apache.spark.package$.<clinit>(package.scala)
	at org.apache.spark.SparkContext.$anonfun$new$1(SparkContext.scala:195)
	at org.apache.spark.internal.Logging.logInfo(Logging.scala:60)
	at org.apache.spark.internal.Logging.logInfo$(Logging.scala:59)
	at org.apache.spark.SparkContext.logInfo(SparkContext.scala:84)
	at org.apache.spark.SparkContext.<init>(SparkContext.scala:195)
	at org.apache.spark.SparkContext$.getOrCreate(SparkContext.scala:2739)
	at org.apache.spark.sql.SparkSession$Builder.$anonfun$getOrCreate$2(SparkSession.scala:978)
	at scala.Option.getOrElse(Option.scala:189)
	at org.apache.spark.sql.SparkSession$Builder.getOrCreate(SparkSession.scala:972)
	at org.apache.spark.examples.SparkPi$.main(SparkPi.scala:30)
	at org.apache.spark.examples.SparkPi.main(SparkPi.scala)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.apache.spark.deploy.yarn.ApplicationMaster$$anon$2.run(ApplicationMaster.scala:747)
Caused by: org.apache.spark.SparkException: Could not find spark-version-info.properties
	at org.apache.spark.package$SparkBuildInfo$.<init>(package.scala:62)
	at org.apache.spark.package$SparkBuildInfo$.<clinit>(package.scala)
	... 18 more
{code}

h1. *Why did the problem happen?*
While building Spark, the spark-build-info.properties file [is generated using bash|https://github.com/apache/spark/blob/d62c18b7497997188ec587e1eb62e75c979c1c93/core/pom.xml#L560-L564]. In Windows environment, if Windows Subsystem for Linux (WSL) is installed, it somehow overrides the other bash executables in the PATH, as noted in SPARK-40739. The bash in WSL has a different mounting configuration and thus, the target location specified for spark-build-info.properties won't be the expected location. Ultimately, this leads to spark-build-info.properties to get excluded from the spark-core jar, thus causing the SparkContext initialization to fail with the above depicted error message.

h1. *How can the problem be solved?*
I propose to fix this issue by enhancing the build system to automatically detect and switch between using Powershell for Windows and Bash for non-Windows OS. This yields a seamless build experience for all the platform users. Particularly for Windows users, to not worry about the conflicts with different bash installations.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org