[GitHub] [spark] maropu commented on a change in pull request #31888: [SPARK-34596][SQL][2.4] Use Utils.getSimpleName to avoid hitting Malformed class name in NewInstance.doGenCode

[GitBox <gi...@apache.org>]

maropu commented on a change in pull request #31888:
URL: https://github.com/apache/spark/pull/31888#discussion_r599324821



##########
File path: sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/encoders/ExpressionEncoderSuite.scala
##########
@@ -203,6 +203,36 @@ class ExpressionEncoderSuite extends CodegenInterpretedPlanTest with AnalysisTes
 
   encodeDecodeTest(Array(Option(InnerClass(1))), "array of optional inner class")
 
+  // NOTE: branch-2.4 does not have the interpreted implementation of SafeProjection, so
+  // it does not fall back into the interpreted mode if the compilation fails.
+  // Therefore, the test in this PR just checks that the compilation error happens
+  // instead of checking that the interpreted mode works well.
+  private def checkCompilationError[T : ExpressionEncoder](
+      input: T,
+      testName: String): Unit = {
+    testAndVerifyNotLeakingReflectionObjects(s"compilation error: $testName: $input") {
+      val encoder = implicitly[ExpressionEncoder[T]]
+      val row = encoder.toRow(input)
+      val boundEncoder = encoder.resolveAndBind()
+      val errMsg = intercept[RuntimeException] {
+        boundEncoder.fromRow(row)
+      }.getCause.getMessage
+      assert(errMsg.contains("failed to compile: "))

Review comment:
       > Is this the compilation error?
   
   Yea, that's the expected error. Actually, the master has same error when `useFallback = false`:
   https://github.com/apache/spark/blob/d32bb4e5ee4718741252c46c50a40810b722f12d/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/encoders/ExpressionEncoderSuite.scala#L219
   
   ```
   // master
   [info]   Cause: org.codehaus.commons.compiler.CompileException: File 'generated.java', Line 32, Column 124: failed to compile: org.codehaus.commons.compiler.CompileException: File 'generated.java', Line 32, Column 124: "org.apache.spark.sql.catalyst.encoders.ExpressionEncoderSuite$MalformedClassObject$" has no member type "ExpressionEncoderSuite$MalformedClassObject$MalformedNameExample"
   [info]   at org.apache.spark.sql.errors.QueryExecutionErrors$.compilerError(QueryExecutionErrors.scala:328)
   ```
   
   > But by moving MalformedClassObject out of ExpressionEncoderSuite, the generated code can be compiled.
   
   IIUC the condition where the error happens is that `MalformedClassObject` is placed in `ExpressionEncoderSuite` as Kris said in the previous PR: https://github.com/apache/spark/pull/31709#issuecomment-791275810
   ```
   Actually, what's more interesting is the different rules of Java's nested classes and Scala's. Since there is no such
    notion of a "companion object" or built-in "singleton object" notion on the Java language level, at least not in 
   Java 8, the Java language syntax for creating a new instance of a inner class object doesn't work well for Scala's
    class Foo { object Bar { class Baz {...} } } kind of nesting, i.e. even when we do use the correct "simple name" 
   (e.g. MalformedNameExample), there would still be no valid Java syntax to create it (i.e. outerObj.new 
   InnerClass(...) doesn't work when outerObj is a Scala object)
   ```
   cc: @rednaxelafx 




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org