(spark) branch branch-4.x updated: [SPARK-57548][SQL] Avoid eager scala.reflect.runtime.universe initialization in ScalaReflection

[ur...@apache.org]

This is an automated email from the ASF dual-hosted git repository.

uros-b pushed a commit to branch branch-4.x
in repository https://gitbox.apache.org/repos/asf/spark.git


The following commit(s) were added to refs/heads/branch-4.x by this push:
     new df23d1599157 [SPARK-57548][SQL] Avoid eager scala.reflect.runtime.universe initialization in ScalaReflection
df23d1599157 is described below

commit df23d1599157562b9c6ab7ed195733da047e7fc6
Author: Emil Ejbyfeldt <em...@choreograph.com>
AuthorDate: Sun Aug 23 13:36:33 2026 +0000

    [SPARK-57548][SQL] Avoid eager scala.reflect.runtime.universe initialization in ScalaReflection
    
    ### What changes were proposed in this pull request?
    Make initializaiton of universe in ScalaReflection lazy. This is so that one can more easily use Spark3 from a Scala3 codebase.
    
    An alternative approach could be to move the method `encodeFieldNameToIdentifier` and `findConstructor` so that the universe would only be intilizalized if needed.
    
    These to methods uses scala reflection even if using Scala3 since they are used from NewInstance.
    
    ### Why are the changes needed?
    Because of this issue scala/scala3#25896 the change top make it possible to use Spark from Scala 3.8.0 or later.
    
    ### Does this PR introduce _any_ user-facing change?
    No.
    
    ### How was this patch tested?
    Existing tests ensure no regressions. That it makes the code more compatible with Scala 3.8 was done by running a local build against the tests in https://github.com/unum-io/tyda (a Scala 3 library that can use Spark as a backend).
    
    ### Was this patch authored or co-authored using generative AI tooling?
    No.
    
    Closes #57303 from eejbyfeldt/SPARK-57548.
    
    Authored-by: Emil Ejbyfeldt <em...@choreograph.com>
    Signed-off-by: Uros Bojanic <22...@users.noreply.github.com>
    (cherry picked from commit 7df3ac405cf7dded626fcac4a7d45d59b93a8ebd)
    Signed-off-by: Uros Bojanic <22...@users.noreply.github.com>
---
 .../main/scala/org/apache/spark/sql/catalyst/ScalaReflection.scala    | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/sql/api/src/main/scala/org/apache/spark/sql/catalyst/ScalaReflection.scala b/sql/api/src/main/scala/org/apache/spark/sql/catalyst/ScalaReflection.scala
index 6f5c4be42bbd..60a5a597d337 100644
--- a/sql/api/src/main/scala/org/apache/spark/sql/catalyst/ScalaReflection.scala
+++ b/sql/api/src/main/scala/org/apache/spark/sql/catalyst/ScalaReflection.scala
@@ -40,7 +40,7 @@ private[catalyst] object ScalaSubtypeLock
  * A default version of ScalaReflection that uses the runtime universe.
  */
 object ScalaReflection extends ScalaReflection {
-  val universe: scala.reflect.runtime.universe.type = scala.reflect.runtime.universe
+  lazy val universe: scala.reflect.runtime.universe.type = scala.reflect.runtime.universe
   // Since we are creating a runtime mirror using the class loader of current thread,
   // we need to use def at here. So, every time we call mirror, it is using the
   // class loader of the current thread.
@@ -218,7 +218,7 @@ object ScalaReflection extends ScalaReflection {
   }
 
   def encodeFieldNameToIdentifier(fieldName: String): String = {
-    TermName(fieldName).encodedName.toString
+    scala.reflect.NameTransformer.encode(fieldName)
   }
 
   /**


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org