You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Hyukjin Kwon (Jira)" <ji...@apache.org> on 2023/12/15 17:14:00 UTC
[jira] [Resolved] (SPARK-46409) Spark Connect Repl does not work with ClosureCleaner
[ https://issues.apache.org/jira/browse/SPARK-46409?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Hyukjin Kwon resolved SPARK-46409.
----------------------------------
Fix Version/s: 4.0.0
Resolution: Fixed
Issue resolved by pull request 44360
[https://github.com/apache/spark/pull/44360]
> Spark Connect Repl does not work with ClosureCleaner
> ----------------------------------------------------
>
> Key: SPARK-46409
> URL: https://issues.apache.org/jira/browse/SPARK-46409
> Project: Spark
> Issue Type: Bug
> Components: Connect
> Affects Versions: 4.0.0
> Reporter: Vsevolod Stepanov
> Assignee: Vsevolod Stepanov
> Priority: Major
> Labels: pull-request-available
> Fix For: 4.0.0
>
>
> SPARK-45136 added ClosureCleaner support to SparkConnect client. Unfortunately, this change breaks ConnectRepl launched by `./connector/connect/bin/spark-connect-scala-client`. To reproduce the issue:
> # Run `./connector/connect/bin/spark-connect-shell`
> # Run `./connector/connect/bin/spark-connect-scala-client`
> # In the REPL, execute this code:
> ```
> @ def plus1(x: Int): Int = x + 1
> @ val plus1_udf = udf(plus1 _)
> ```
> This will fail with the following error:
> ```
> java.lang.reflect.InaccessibleObjectException: Unable to make private native java.lang.reflect.Field[] java.lang.Class.getDeclaredFields0(boolean) accessible: module java.base does not "opens java.lang" to unnamed module @45099dd3
> java.lang.reflect.AccessibleObject.checkCanSetAccessible(AccessibleObject.java:354)
> java.lang.reflect.AccessibleObject.checkCanSetAccessible(AccessibleObject.java:297)
> java.lang.reflect.Method.checkCanSetAccessible(Method.java:199)
> java.lang.reflect.Method.setAccessible(Method.java:193)
> org.apache.spark.util.ClosureCleaner$.getFinalModifiersFieldForJava17(ClosureCleaner.scala:577)
> org.apache.spark.util.ClosureCleaner$.setFieldAndIgnoreModifiers(ClosureCleaner.scala:560)
> org.apache.spark.util.ClosureCleaner$.$anonfun$cleanupAmmoniteReplClosure$18(ClosureCleaner.scala:533)
> org.apache.spark.util.ClosureCleaner$.$anonfun$cleanupAmmoniteReplClosure$18$adapted(ClosureCleaner.scala:525)
> scala.collection.ArrayOps$WithFilter.foreach(ArrayOps.scala:73)
> org.apache.spark.util.ClosureCleaner$.$anonfun$cleanupAmmoniteReplClosure$16(ClosureCleaner.scala:525)
> org.apache.spark.util.ClosureCleaner$.$anonfun$cleanupAmmoniteReplClosure$16$adapted(ClosureCleaner.scala:522)
> scala.collection.IterableOnceOps.foreach(IterableOnce.scala:576)
> scala.collection.IterableOnceOps.foreach$(IterableOnce.scala:574)
> scala.collection.AbstractIterable.foreach(Iterable.scala:933)
> scala.collection.IterableOps$WithFilter.foreach(Iterable.scala:903)
> org.apache.spark.util.ClosureCleaner$.cleanupAmmoniteReplClosure(ClosureCleaner.scala:522)
> org.apache.spark.util.ClosureCleaner$.clean(ClosureCleaner.scala:251)
> org.apache.spark.sql.expressions.SparkConnectClosureCleaner$.clean(UserDefinedFunction.scala:210)
> org.apache.spark.sql.expressions.ScalarUserDefinedFunction$.apply(UserDefinedFunction.scala:187)
> org.apache.spark.sql.expressions.ScalarUserDefinedFunction$.apply(UserDefinedFunction.scala:180)
> org.apache.spark.sql.functions$.udf(functions.scala:7956)
> ammonite.$sess.cmd1$Helper.<init>(cmd1.sc:1)
> ammonite.$sess.cmd1$.<clinit>(cmd1.sc:7)
> ```
>
> This is because ClosureCleaner is heavily reliant on using reflection API and is not compatible with Java 17. The rest of Spark bypasses this by adding `--add-opens` JVM flags, see https://issues.apache.org/jira/browse/SPARK-36796. We need to add these options to Spark Connect Client launch script as well
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org