You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by "jiwen624 (via GitHub)" <gi...@apache.org> on 2026/04/11 00:41:50 UTC
[PR] [SPARK-56447][SHELL] Fix spark-shell REPL classpath initialization error [spark]
jiwen624 opened a new pull request, #55305:
URL: https://github.com/apache/spark/pull/55305
### What changes were proposed in this pull request?
In `Main.doMain`, prepend `java.class.path` to the `-classpath` argument passed to the REPL's `GenericRunnerSettings`:
### Why are the changes needed?
`$SPARK_HOME/bin/spark-shell` fails with:
```
Failed to initialize compiler: object scala in compiler mirror not found.
** Note that as of 2.8 scala does not assume use of the java classpath.
** For the old behavior pass -usejavacp to scala, or if using a Settings
** object programmatically, settings.usejavacp.value = true.
Exception in thread "main" java.lang.NullPointerException: Cannot throw exception because "null" is null
at scala.tools.nsc.CompilationUnits$CompilationUnit.<init>(CompilationUnits.scala:43)
at scala.tools.nsc.CompilationUnits$CompilationUnit.<init>(CompilationUnits.scala:44)
at scala.tools.nsc.interpreter.IMain$ReadEvalPrint.compile(IMain.scala:734)
```
This works in previous versions.
`bin/spark-shell` has passed `-Dscala.usejavacp=true` since SPARK-4161 to address exactly this behaviour. The mechanism is broken after SPARK-52587 which initializes the compiler before the classpath is set. The fix makes doMain explicitly include java.class.path in the -classpath argument, restoring correct behaviour regardless of how SparkILoop is constructed.
### Does this PR introduce any user-facing change?
Yes. spark-shell was broken with a compiler initialization error. This PR fixes it so spark-shell starts and evaluates
expressions correctly again.
### How was this patch tested?
Added a unit test in ReplSuite that calls Main.doMain with Array.empty (no explicit -classpath argument) and verifies the output. This test fails on the unfixed code and passes after the fix.
### Was this patch authored or co-authored using generative AI tooling?
No
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-56447][SHELL] Fix spark-shell REPL classpath initialization error [spark]
Posted by "jiwen624 (via GitHub)" <gi...@apache.org>.
jiwen624 commented on PR #55305:
URL: https://github.com/apache/spark/pull/55305#issuecomment-4399771895
Hi @cloud-fan @dongjoon-hyun @HyukjinKwon, would anyone be interested in taking a look at this bug fix? The issue is still present in the latest codebase.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-56447][SHELL] Fix spark-shell REPL classpath initialization error [spark]
Posted by "jiwen624 (via GitHub)" <gi...@apache.org>.
jiwen624 closed pull request #55305: [SPARK-56447][SHELL] Fix spark-shell REPL classpath initialization error
URL: https://github.com/apache/spark/pull/55305
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-56447][SHELL] Fix spark-shell REPL classpath initialization error [spark]
Posted by "jiwen624 (via GitHub)" <gi...@apache.org>.
jiwen624 commented on PR #55305:
URL: https://github.com/apache/spark/pull/55305#issuecomment-4233298944
@cloud-fan @dongjoon-hyun could you take a look at this bug fix when you get a chance?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org