You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Yang Jie (Jira)" <ji...@apache.org> on 2025/05/13 04:58:00 UTC
[jira] [Updated] (SPARK-52088) Redesign ClosureCleaner Implementation Due to JDK-8309635's Removal of Old Core Reflection and Inability to Modify Private Final Fields in Hidden Classes
[ https://issues.apache.org/jira/browse/SPARK-52088?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Yang Jie updated SPARK-52088:
-----------------------------
Description:
The removal of the old core reflection implementation in [JDK-8309635|https://bugs.openjdk.org/browse/JDK-8309635] poses a risk that the workaround for SPARK-40729 — which involves setting `-Djdk.reflect.useDirectMethodHandle=false` to enable the old core reflection — may no longer work in the next Java LTS release (if the next Java LTS does not revert [JDK-8309635|https://bugs.openjdk.org/browse/JDK-8309635]). We might need to consider redesigning the implementation of `ClosureCleaner`.
Currently, when testing the `repl` module with Java 22, the following error occurs:
```
build/sbt clean "repl/test"
```
```
[info] - broadcast vars *** FAILED *** (1 second, 141 milliseconds)
[info] isContain was true Interpreter output contained 'Exception':
[info] Welcome to
[info] ____ __
[info] / _{_}/{_}_ ___ ____{_}/ /{_}_
[info] {_}\ \/ _ \/ _ `/ __/ '{_}/
[info] /__{_}/ .{_}{_}/_,{_}/{_}/ /{_}/_\ version 4.1.0-SNAPSHOT
[info] /_/
[info]
[info] Using Scala version 2.13.16 (OpenJDK 64-Bit Server VM, Java 22.0.2)
[info] Type in expressions to have them evaluated.
[info] Type :help for more information.
[info]
[info] scala>
[info] scala> var array: Array[Int] = Array(0, 0, 0, 0, 0)
[info]
[info] scala> val broadcastArray: org.apache.spark.broadcast.Broadcast[Array[Int]] = Broadcast(0)
[info]
[info] scala> java.lang.InternalError: java.lang.IllegalAccessException: final field has no write access: $Lambda/0x0000060001ecedd8.arg$1/putField, from class java.lang.Object (module java.base)
[info] at java.base/jdk.internal.reflect.MethodHandleAccessorFactory.newFieldAccessor(MethodHandleAccessorFactory.java:207)
[info] at java.base/jdk.internal.reflect.ReflectionFactory.newFieldAccessor(ReflectionFactory.java:144)
[info] at java.base/java.lang.reflect.Field.acquireOverrideFieldAccessor(Field.java:1200)
[info] at java.base/java.lang.reflect.Field.getOverrideFieldAccessor(Field.java:1169)
[info] at java.base/java.lang.reflect.Field.set(Field.java:836)
[info] at org.apache.spark.util.ClosureCleaner$.setFieldAndIgnoreModifiers(ClosureCleaner.scala:564)
[info] at org.apache.spark.util.ClosureCleaner$.cleanupScalaReplClosure(ClosureCleaner.scala:432)
[info] at org.apache.spark.util.ClosureCleaner$.clean(ClosureCleaner.scala:257)
[info] at org.apache.spark.util.SparkClosureCleaner$.clean(SparkClosureCleaner.scala:39)
[info] at org.apache.spark.SparkContext.clean(SparkContext.scala:2843)
[info] at org.apache.spark.rdd.RDD.$anonfun$map$1(RDD.scala:425)
[info] at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151)
[info] at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:112)
[info] at org.apache.spark.rdd.RDD.withScope(RDD.scala:417)
[info] at org.apache.spark.rdd.RDD.map(RDD.scala:424)
[info] ... 79 elided
...
[info] Run completed in 35 seconds, 38 milliseconds.
[info] Total number of tests run: 36
[info] Suites: completed 3, aborted 0
[info] Tests: succeeded 27, failed 9, canceled 0, ignored 0, pending 0
[info] *** 9 TESTS FAILED ***
[error] Failed tests:
[error] org.apache.spark.repl.SingletonReplSuite
[error] org.apache.spark.repl.ReplSuite
```
I tried switching to using either `VarHandle` or `Unsafe#putObject`, but neither of them worked because the test cases involved modifying a `private final field` within a `hidden class`.
was:
The removal of the old core reflection implementation in Java 22 poses a risk that the workaround for SPARK-40729 — which involves setting `-Djdk.reflect.useDirectMethodHandle=false` to enable the old core reflection — may no longer work in the next Java LTS release (if the next Java LTS does not revert [JDK-8309635|https://bugs.openjdk.org/browse/JDK-8309635]). We might need to consider redesigning the implementation of `ClosureCleaner`.
Currently, when testing the `repl` module with Java 22, the following error occurs:
```
build/sbt clean "repl/test"
```
```
[info] - broadcast vars *** FAILED *** (1 second, 141 milliseconds)
[info] isContain was true Interpreter output contained 'Exception':
[info] Welcome to
[info] ____ __
[info] / __/__ ___ _____/ /__
[info] _\ \/ _ \/ _ `/ __/ '_/
[info] /___/ .__/\_,_/_/ /_/\_\ version 4.1.0-SNAPSHOT
[info] /_/
[info]
[info] Using Scala version 2.13.16 (OpenJDK 64-Bit Server VM, Java 22.0.2)
[info] Type in expressions to have them evaluated.
[info] Type :help for more information.
[info]
[info] scala>
[info] scala> var array: Array[Int] = Array(0, 0, 0, 0, 0)
[info]
[info] scala> val broadcastArray: org.apache.spark.broadcast.Broadcast[Array[Int]] = Broadcast(0)
[info]
[info] scala> java.lang.InternalError: java.lang.IllegalAccessException: final field has no write access: $Lambda/0x0000060001ecedd8.arg$1/putField, from class java.lang.Object (module java.base)
[info] at java.base/jdk.internal.reflect.MethodHandleAccessorFactory.newFieldAccessor(MethodHandleAccessorFactory.java:207)
[info] at java.base/jdk.internal.reflect.ReflectionFactory.newFieldAccessor(ReflectionFactory.java:144)
[info] at java.base/java.lang.reflect.Field.acquireOverrideFieldAccessor(Field.java:1200)
[info] at java.base/java.lang.reflect.Field.getOverrideFieldAccessor(Field.java:1169)
[info] at java.base/java.lang.reflect.Field.set(Field.java:836)
[info] at org.apache.spark.util.ClosureCleaner$.setFieldAndIgnoreModifiers(ClosureCleaner.scala:564)
[info] at org.apache.spark.util.ClosureCleaner$.cleanupScalaReplClosure(ClosureCleaner.scala:432)
[info] at org.apache.spark.util.ClosureCleaner$.clean(ClosureCleaner.scala:257)
[info] at org.apache.spark.util.SparkClosureCleaner$.clean(SparkClosureCleaner.scala:39)
[info] at org.apache.spark.SparkContext.clean(SparkContext.scala:2843)
[info] at org.apache.spark.rdd.RDD.$anonfun$map$1(RDD.scala:425)
[info] at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151)
[info] at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:112)
[info] at org.apache.spark.rdd.RDD.withScope(RDD.scala:417)
[info] at org.apache.spark.rdd.RDD.map(RDD.scala:424)
[info] ... 79 elided
...
[info] Run completed in 35 seconds, 38 milliseconds.
[info] Total number of tests run: 36
[info] Suites: completed 3, aborted 0
[info] Tests: succeeded 27, failed 9, canceled 0, ignored 0, pending 0
[info] *** 9 TESTS FAILED ***
[error] Failed tests:
[error] org.apache.spark.repl.SingletonReplSuite
[error] org.apache.spark.repl.ReplSuite
```
I tried switching to using either `VarHandle` or `Unsafe#putObject`, but neither of them worked because the test cases involved modifying a `private final field` within a `hidden class`.
> Redesign ClosureCleaner Implementation Due to JDK-8309635's Removal of Old Core Reflection and Inability to Modify Private Final Fields in Hidden Classes
> ----------------------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: SPARK-52088
> URL: https://issues.apache.org/jira/browse/SPARK-52088
> Project: Spark
> Issue Type: Improvement
> Components: Spark Core
> Affects Versions: 4.1.0
> Reporter: Yang Jie
> Priority: Major
>
> The removal of the old core reflection implementation in [JDK-8309635|https://bugs.openjdk.org/browse/JDK-8309635] poses a risk that the workaround for SPARK-40729 — which involves setting `-Djdk.reflect.useDirectMethodHandle=false` to enable the old core reflection — may no longer work in the next Java LTS release (if the next Java LTS does not revert [JDK-8309635|https://bugs.openjdk.org/browse/JDK-8309635]). We might need to consider redesigning the implementation of `ClosureCleaner`.
> Currently, when testing the `repl` module with Java 22, the following error occurs:
> ```
> build/sbt clean "repl/test"
> ```
> ```
> [info] - broadcast vars *** FAILED *** (1 second, 141 milliseconds)
> [info] isContain was true Interpreter output contained 'Exception':
> [info] Welcome to
> [info] ____ __
> [info] / _{_}/{_}_ ___ ____{_}/ /{_}_
> [info] {_}\ \/ _ \/ _ `/ __/ '{_}/
> [info] /__{_}/ .{_}{_}/_,{_}/{_}/ /{_}/_\ version 4.1.0-SNAPSHOT
> [info] /_/
> [info]
> [info] Using Scala version 2.13.16 (OpenJDK 64-Bit Server VM, Java 22.0.2)
> [info] Type in expressions to have them evaluated.
> [info] Type :help for more information.
> [info]
> [info] scala>
> [info] scala> var array: Array[Int] = Array(0, 0, 0, 0, 0)
> [info]
> [info] scala> val broadcastArray: org.apache.spark.broadcast.Broadcast[Array[Int]] = Broadcast(0)
> [info]
> [info] scala> java.lang.InternalError: java.lang.IllegalAccessException: final field has no write access: $Lambda/0x0000060001ecedd8.arg$1/putField, from class java.lang.Object (module java.base)
> [info] at java.base/jdk.internal.reflect.MethodHandleAccessorFactory.newFieldAccessor(MethodHandleAccessorFactory.java:207)
> [info] at java.base/jdk.internal.reflect.ReflectionFactory.newFieldAccessor(ReflectionFactory.java:144)
> [info] at java.base/java.lang.reflect.Field.acquireOverrideFieldAccessor(Field.java:1200)
> [info] at java.base/java.lang.reflect.Field.getOverrideFieldAccessor(Field.java:1169)
> [info] at java.base/java.lang.reflect.Field.set(Field.java:836)
> [info] at org.apache.spark.util.ClosureCleaner$.setFieldAndIgnoreModifiers(ClosureCleaner.scala:564)
> [info] at org.apache.spark.util.ClosureCleaner$.cleanupScalaReplClosure(ClosureCleaner.scala:432)
> [info] at org.apache.spark.util.ClosureCleaner$.clean(ClosureCleaner.scala:257)
> [info] at org.apache.spark.util.SparkClosureCleaner$.clean(SparkClosureCleaner.scala:39)
> [info] at org.apache.spark.SparkContext.clean(SparkContext.scala:2843)
> [info] at org.apache.spark.rdd.RDD.$anonfun$map$1(RDD.scala:425)
> [info] at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151)
> [info] at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:112)
> [info] at org.apache.spark.rdd.RDD.withScope(RDD.scala:417)
> [info] at org.apache.spark.rdd.RDD.map(RDD.scala:424)
> [info] ... 79 elided
> ...
> [info] Run completed in 35 seconds, 38 milliseconds.
> [info] Total number of tests run: 36
> [info] Suites: completed 3, aborted 0
> [info] Tests: succeeded 27, failed 9, canceled 0, ignored 0, pending 0
> [info] *** 9 TESTS FAILED ***
> [error] Failed tests:
> [error] org.apache.spark.repl.SingletonReplSuite
> [error] org.apache.spark.repl.ReplSuite
> ```
> I tried switching to using either `VarHandle` or `Unsafe#putObject`, but neither of them worked because the test cases involved modifying a `private final field` within a `hidden class`.
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org