You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Anupam Yadav (Jira)" <ji...@apache.org> on 2026/05/05 02:02:00 UTC

[jira] [Commented] (SPARK-56448) Spark connect client fails on subsequent start due to ammonite (code) cache

    [ https://issues.apache.org/jira/browse/SPARK-56448?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18078335#comment-18078335 ] 

Anupam Yadav commented on SPARK-56448:
--------------------------------------

I can try to work on this

> Spark connect client fails on subsequent start due to ammonite (code) cache
> ---------------------------------------------------------------------------
>
>                 Key: SPARK-56448
>                 URL: https://issues.apache.org/jira/browse/SPARK-56448
>             Project: Spark
>          Issue Type: Bug
>          Components: Connect
>    Affects Versions: 4.2.0
>            Reporter: Attila Zsolt Piros
>            Priority: Major
>
> When Spark Connect client is restarted (from the same working directory) it causes the following:
> {noformat}
> ➜  spark git:(SPARK-connect) ✗ bin/spark-shell  --remote "sc://localhost:15002"
> WARNING: Using incubator modules: jdk.incubator.vector
> 26/04/10 16:04:35 INFO BaseAllocator: Debug mode disabled. Enable with the VM option -Darrow.memory.debug.allocator=true.
> 26/04/10 16:04:35 INFO DefaultAllocationManagerOption: allocation manager type not specified, using netty as the default type
> 26/04/10 16:04:35 INFO CheckAllocator: Using DefaultAllocationManager at memory/netty/DefaultAllocationManagerFactory.class
> Compiling (synthetic)/ammonite/predef/ArgsPredef.sc
> Compiling /Users/apiros/git/attilapiros/spark/(console)
> Welcome to
>       ____              __
>      / __/__  ___ _____/ /__
>     _\ \/ _ \/ _ `/ __/  '_/
>    /___/ .__/\_,_/_/ /_/\_\   version 4.2.0-SNAPSHOT
>       /_/
> Type in expressions to have them evaluated.
> Spark connect server version 4.2.0-SNAPSHOT.
> Spark session available as 'spark'.
> scala> Bye!
> ➜  spark git:(SPARK-connect) ✗ bin/spark-shell  --remote "sc://localhost:15002"
> WARNING: Using incubator modules: jdk.incubator.vector
> 26/04/10 16:04:42 INFO BaseAllocator: Debug mode disabled. Enable with the VM option -Darrow.memory.debug.allocator=true.
> 26/04/10 16:04:42 INFO DefaultAllocationManagerOption: allocation manager type not specified, using netty as the default type
> 26/04/10 16:04:42 INFO CheckAllocator: Using DefaultAllocationManager at memory/netty/DefaultAllocationManagerFactory.class
> Exception in thread "main" java.lang.NullPointerException: Cannot invoke "ammonite.predef.ArgsPredef$Helper.spark()" because the return value of "ammonite.predef.CodePredef.ArgsPredef()" is null
> 	at ammonite.predef.CodePredef$Helper.<init>((console):7)
> 	at ammonite.predef.CodePredef$.<clinit>((console):6)
> 	at ammonite.predef.CodePredef.$main((console))
> 	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> 	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
> 	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> 	at java.base/java.lang.reflect.Method.invoke(Method.java:569)
> 	at ammonite.runtime.Evaluator$$anon$1.$anonfun$evalMain$1(Evaluator.scala:108)
> 	at ammonite.util.Util$.withContextClassloader(Util.scala:21)
> 	at ammonite.runtime.Evaluator$$anon$1.evalMain(Evaluator.scala:90)
> 	at ammonite.interp.Interpreter.$anonfun$processAllScriptBlocks$10(Interpreter.scala:594)
> 	at ammonite.util.Res$Success.map(Res.scala:63)
> 	at ammonite.interp.Interpreter.$anonfun$processAllScriptBlocks$9(Interpreter.scala:594)
> 	at scala.Option$WithFilter.map(Option.scala:242)
> 	at ammonite.interp.Interpreter.loop$1(Interpreter.scala:574)
> 	at ammonite.interp.Interpreter.processAllScriptBlocks(Interpreter.scala:644)
> 	at ammonite.interp.Interpreter.$anonfun$processModule$6(Interpreter.scala:432)
> 	at ammonite.util.Catching.flatMap(Res.scala:110)
> 	at ammonite.interp.Interpreter.$anonfun$processModule$5(Interpreter.scala:423)
> 	at ammonite.util.Res$Success.flatMap(Res.scala:58)
> 	at ammonite.interp.Interpreter.processModule(Interpreter.scala:413)
> 	at ammonite.interp.Interpreter.$anonfun$initializePredef$3(Interpreter.scala:142)
> 	at ammonite.interp.Interpreter.$anonfun$initializePredef$3$adapted(Interpreter.scala:142)
> 	at ammonite.interp.PredefInitialization$.$anonfun$apply$2(PredefInitialization.scala:79)
> 	at ammonite.util.Res$.$anonfun$fold$1(Res.scala:27)
> 	at scala.collection.LinearSeqOps.foldLeft(LinearSeq.scala:183)
> 	at scala.collection.LinearSeqOps.foldLeft$(LinearSeq.scala:179)
> 	at scala.collection.immutable.List.foldLeft(List.scala:79)
> 	at ammonite.util.Res$.fold(Res.scala:25)
> 	at ammonite.interp.PredefInitialization$.apply(PredefInitialization.scala:67)
> 	at ammonite.interp.Interpreter.initializePredef(Interpreter.scala:144)
> 	at ammonite.repl.Repl.initializePredef(Repl.scala:150)
> 	at ammonite.Main.run(Main.scala:231)
> 	at org.apache.spark.sql.application.ConnectRepl$.$anonfun$doMain$1(ConnectRepl.scala:168)
> {noformat}
> This is due to the ~/.ammonite directory persisting pre-defined code caches (predef).
> So when I start it from a clean directory it works just fine again or when when I delete the "~/.ammonite" directory:
> {noformat}
> ➜  spark git:(SPARK-connect) ✗ rm -rf ~/.ammonite
> ➜  spark git:(SPARK-connect) ✗ bin/spark-shell  --remote "sc://localhost:15002"
> WARNING: Using incubator modules: jdk.incubator.vector
> 26/04/10 16:36:31 INFO BaseAllocator: Debug mode disabled. Enable with the VM option -Darrow.memory.debug.allocator=true.
> 26/04/10 16:36:31 INFO DefaultAllocationManagerOption: allocation manager type not specified, using netty as the default type
> 26/04/10 16:36:31 INFO CheckAllocator: Using DefaultAllocationManager at memory/netty/DefaultAllocationManagerFactory.class
> Compiling (synthetic)/ammonite/predef/ArgsPredef.sc
> Compiling /Users/apiros/git/attilapiros/spark/(console)
> Welcome to
>       ____              __
>      / __/__  ___ _____/ /__
>     _\ \/ _ \/ _ `/ __/  '_/
>    /___/ .__/\_,_/_/ /_/\_\   version 4.2.0-SNAPSHOT
>       /_/
> Type in expressions to have them evaluated.
> Spark connect server version 4.2.0-SNAPSHOT.
> Spark session available as 'spark'.
> scala>
> {noformat}



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org