You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2021/03/02 12:12:33 UTC
[GitHub] [spark] rednaxelafx opened a new pull request #31709: [SPARK-34596] Use Utils.getSimpleName to avoid hitting Malformed class name in NewInstance.doGenCode
rednaxelafx opened a new pull request #31709:
URL: https://github.com/apache/spark/pull/31709
<!--
Thanks for sending a pull request! Here are some tips for you:
1. If this is your first time, please read our contributor guidelines: https://spark.apache.org/contributing.html
2. Ensure you have added or run the appropriate tests for your PR: https://spark.apache.org/developer-tools.html
3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][SPARK-XXXX] Your PR title ...'.
4. Be sure to keep the PR description updated to reflect all changes.
5. Please write your PR title to summarize what this PR proposes.
6. If possible, provide a concise example to reproduce the issue for a faster review.
7. If you want to add a new configuration, please read the guideline first for naming configurations in
'core/src/main/scala/org/apache/spark/internal/config/ConfigEntry.scala'.
-->
### What changes were proposed in this pull request?
<!--
Please clarify what changes you are proposing. The purpose of this section is to outline the changes and how this PR fixes the issue.
If possible, please consider writing useful notes for better and faster reviews in your PR. See the examples below.
1. If you refactor some codes with changing classes, showing the class hierarchy will help reviewers.
2. If you fix some SQL features, you can provide some references of other DBMSes.
3. If there is design documentation, please add the link.
4. If there is a discussion in the mailing list, please add the link.
-->
Use `Utils.getSimpleName` to avoid hitting `Malformed class name` error in `NewInstance.doGenCode`.
### Why are the changes needed?
<!--
Please clarify why the changes are needed. For instance,
1. If you propose a new API, clarify the use case for a new API.
2. If you fix a bug, you can clarify why it is a bug.
-->
On older JDK versions (e.g. JDK8u), nested Scala classes may trigger `java.lang.Class.getSimpleName` to throw an `java.lang.InternalError: Malformed class name` error.
In this particular case, creating an `ExpressionEncoder` on such a nested Scala class would create a `NewInstance` expression under the hood, which will trigger the problem during codegen.
Similar to https://github.com/apache/spark/pull/29050, we should use Spark's `Utils.getSimpleName` utility function in place of `Class.getSimpleName` to avoid hitting the issue.
There are two other occurrences of `java.lang.Class.getSimpleName` in the same file, but they're safe because they're only guaranteed to be only used on Java classes, which don't have this problem, e.g.:
```scala
// Make a copy of the data if it's unsafe-backed
def makeCopyIfInstanceOf(clazz: Class[_ <: Any], value: String) =
s"$value instanceof ${clazz.getSimpleName}? ${value}.copy() : $value"
val genFunctionValue: String = lambdaFunction.dataType match {
case StructType(_) => makeCopyIfInstanceOf(classOf[UnsafeRow], genFunction.value)
case ArrayType(_, _) => makeCopyIfInstanceOf(classOf[UnsafeArrayData], genFunction.value)
case MapType(_, _, _) => makeCopyIfInstanceOf(classOf[UnsafeMapData], genFunction.value)
case _ => genFunction.value
}
```
The Unsafe-* family of types are all Java types, so they're okay.
### Does this PR introduce _any_ user-facing change?
<!--
Note that it means *any* user-facing change including all aspects such as the documentation fix.
If yes, please clarify the previous behavior and the change this PR proposes - provide the console output, description and/or an example to show the behavior difference if possible.
If possible, please also clarify if this is a user-facing change compared to the released Spark versions or within the unreleased branches such as master.
If no, write 'No'.
-->
Fixes a bug that throws an error when using `ExpressionEncoder` on some nested Scala types, otherwise no changes.
### How was this patch tested?
<!--
If tests were added, say they were added here. Please make sure to add some test cases that check the changes thoroughly including negative and positive cases if possible.
If it was tested in a way different from regular unit tests, please clarify how you tested step by step, ideally copy and paste-able, so that other reviewers can test and check, and descendants can verify in the future.
If tests were not added, please describe why they were not added and/or why it was difficult to add.
-->
Added a test case to `org.apache.spark.sql.catalyst.encoders.ExpressionEncoderSuite`. It'll fail on JDK8u before the fix, and pass after the fix.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] kiszk edited a comment on pull request #31709: [SPARK-34596][SQL] Use Utils.getSimpleName to avoid hitting Malformed class name in NewInstance.doGenCode
Posted by GitBox <gi...@apache.org>.
kiszk edited a comment on pull request #31709:
URL: https://github.com/apache/spark/pull/31709#issuecomment-791996116
I think so. Your change also works in the above case.
But, I am curious why the same code works at the master branch cc @rednaxelafx
https://github.com/apache/spark/blob/2f30cdebb18488a4ba83dd06aa5c5c7126dd8a8c/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/CastSuite.scala#L1026
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #31709: [SPARK-34596][SQL] Use Utils.getSimpleName to avoid hitting Malformed class name in NewInstance.doGenCode
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #31709:
URL: https://github.com/apache/spark/pull/31709#issuecomment-788920802
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/40236/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #31709: [SPARK-34596][SQL] Use Utils.getSimpleName to avoid hitting Malformed class name in NewInstance.doGenCode
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #31709:
URL: https://github.com/apache/spark/pull/31709#issuecomment-789337073
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/40257/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] rednaxelafx commented on pull request #31709: [SPARK-34596][SQL] Use Utils.getSimpleName to avoid hitting Malformed class name in NewInstance.doGenCode
Posted by GitBox <gi...@apache.org>.
rednaxelafx commented on pull request #31709:
URL: https://github.com/apache/spark/pull/31709#issuecomment-791275810
So I did some investigation and found that:
- For the test case I gave in this PR (emulating a kind of Scala class nesting scenario), before this PR it'll fail to codegen because of the "Malformed class name" issue. So that's something that never worked before. In practice, it isn't that hard to hit it, if you run Spark code through a Scala REPL, because the Scala REPL would use either nested objects or nested classes to implement the notion of multiple commands. (Well, there are some REPL-specific code paths in Spark that avoids hitting this problem in some cases though...e.g. [this one](https://github.com/apache/spark/blob/dbce74d39d192f702972df06382591b3e8e43a77/core/src/main/scala/org/apache/spark/util/Utils.scala#L2920-L2929) )
- After applying this PR, the use of `Utils.getSimpleName` actually returns a wrong name for generated-Java-code's compilation purposes. We do need a real "simple name", but when the Scala nested class is not generated by the Scala REPL, the one returned by `Utils.getSimpleName` only strips off the package name and maybe the trailing dollar-sign, but that's it. So in the case of the test case in this PR, we'd want the simple name to be `MalformedNameExample`, but we'd actually get back `ExpressionEncoderSuite$MalformedClassObject$MalformedNameExample` instead. That'll cause Janino compilation to fail.
- Actually, what's more interesting is the different rules of Java's nested classes and Scala's. Since there is no such notion of a "companion object" or built-in "singleton object" notion on the Java language level, at least not in Java 8, the Java language syntax for creating a new instance of a inner class object doesn't work well for Scala's `class Foo { object Bar { class Baz {...} } }` kind of nesting, i.e. even when we do use the correct "simple name" (e.g. `MalformedNameExample`), there would still be no valid Java syntax to create it (i.e. `outerObj.new InnerClass(...)` doesn't work when `outerObj` is a Scala `object`)
- Then comes what's interesting. In Spark's tests, some environment setup will cause the codegen fallback to be in no-fallback mode, meaning a failed compilation will cause test failure; in some other setup, the fallback to interpreted is allowed, so compilation failure does not cause test failure. The reason why my test case in this PR sometimes pass and sometimes fail is exactly that: the compilation actually always fail due to incorrect generated code, but in some test setup it's allowed to fallback so it doesn't fail the test, but in some other setup it's banned from falling back, thus failing the test.
At this point I think it's a bit moot to make the generated code actually work for the `class Foo { object Bar { class Baz {...} } }` pattern mentioned here, as there's no Java syntax available for it anyway. Maybe there are workarounds, but we'll have to investigate a bit more to find them.
The original intent of this PR was to make sure when using `Dataset` on this kind of Scala class nesting it can still run without user-visible failures. I think that intent is fulfilled with the fix in this PR when running in non-testing mode -- the codegen fallback to interpreter would step in and make it work, instead of getting a JDK `java.lang.InternalError` that Spark's codegen doesn't handle.
cc @retronym I wonder if we can get some help from some wonderful Scala folks ^_^ Thanks!
cc @maropu @viirya @kiszk @cloud-fan do @dongjoon-hyun @HyukjinKwon you have any idea what the environment setup differences are with running `build/sbt catalyst/test` vs `build/sbt "catalyst/testOnly *ExpressionEncoderSuite"`? On branch-3.1, the former shows the compilation error as test failure, whereas the latter doesn't show it but under the hood the same compilation error actually happened as well...
---
Some more background on Java's "object creation expression", aka the `new` expression.
Roughly speaking, there are two variants of the `new` expression in Java,
1. Unqualified: that's the regular `new ClassName(args)` form. The way to create non-nested object instances in the normal case, but can also create nested object instances if the enclosing object is `this`.
2. Qualified: the way to create nested objects. The syntax looks like `outerObjRef.new InnerClassSimpleName(args)`. The `outerObjRef` here would be an expression that refers to the enclosing object of the inner object, and the `InnerClassSimpleName` would be a simple class name of the inner class. Note that both `javac` and Janino demands that this `InnerClassSimpleName` cannot have dots (`.`) in it, so you can't use the fully qualified class name in this place. The compiler will check if the `InnerClassSimpleName` is indeed a member class of the static type of the `outerObjRef`.
In Scala, when you declare:
```scala
object Foo {
class Bar { }
}
```
then `Foo`'s class name is `Foo$`, but `Bar`'s class name isn't in the Java convention of `outerClassName + '$' + innerClassSimpleName` (which would have been `Foo$$Bar`), but instead it's just `Foo$Bar`.
There's quite a bit of information recorded in the Class file that at runtime, reflection is still able to recognize the enclosing/member relationships, e.g.
```
val fooDollarClazz = Foo.getClass
fooDollarClazz.getDeclaredClasses //=> Array[Class[_]] = Array(class Foo$Bar)
val barClazz = classOf[Foo.Bar]
barClazz.getEnclosingClass //=> Class[_] = class Foo$
barClazz.getDeclaringClass //=> Class[_] = class Foo$
```
When Janino tries to find a member type, unfortunately, it doesn't use runtime reflection to figure out the member type, but instead it just relies on the Java naming convention of `outerClassName + '$' + innerClassSimpleName`,
https://github.com/janino-compiler/janino/blob/8142ec43a3be288e214de30c9f57fe50e9047766/janino/src/main/java/org/codehaus/janino/IClass.java#L803-L876 , like this:
```java
String memberDescriptor = Descriptor.fromClassName(
Descriptor.toClassName(this.getDescriptor())
+ '$'
+ name
);
```
which will fail to find the `Bar` type in a `fooObjRef.new Bar()` expression.
Same deal with `javac`. Here's an example:
foo.scala
```scala
object Foo {
class Bar {
}
}
```
TestScalaNestedType.java
```java
import java.lang.reflect.*;
public class TestScalaNestedType {
public static void main(String[] args) throws Exception {
Class<?> fooDollarClazz = Foo$.class; // Class.forName("Foo$");
Field fooModuleField = fooDollarClazz.getDeclaredField("MODULE$");
Foo$ fooObjRef = (Foo$) fooModuleField.get(null);
// compiles fine up to the above
fooObjRef.new Bar(); // fails compilation: error: cannot find symbol Bar in Foo$
}
}
```
command line:
```
$ javac -g TestScalaNestedType.java
TestScalaNestedType.java:10: error: cannot find symbol
fooObjRef.new Bar();
^
symbol: class Bar
location: class Foo$
1 error
```
If we change the `new` expression to: `fooObjRef.new Foo.Bar()`, then the error becomes:
```
$ javac -g TestScalaNestedType.java
TestScalaNestedType.java:10: error: '(' expected
fooObjRef.new Foo.Bar();
^
1 error
```
obviously the dot (`.`) isn't accepted there.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on pull request #31709: [SPARK-34596][SQL] Use Utils.getSimpleName to avoid hitting Malformed class name in NewInstance.doGenCode
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on pull request #31709:
URL: https://github.com/apache/spark/pull/31709#issuecomment-789100084
retest this please
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on pull request #31709: [SPARK-34596][SQL] Use Utils.getSimpleName to avoid hitting Malformed class name in NewInstance.doGenCode
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on pull request #31709:
URL: https://github.com/apache/spark/pull/31709#issuecomment-790202254
If you don't mind, reverting is better to me. :)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on pull request #31709: [SPARK-34596][SQL] Use Utils.getSimpleName to avoid hitting Malformed class name in NewInstance.doGenCode
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on pull request #31709:
URL: https://github.com/apache/spark/pull/31709#issuecomment-791315756
Maybe we set `CODEGEN_FACTORY_MODE` somewhere without setting it back
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #31709: [SPARK-34596][SQL] Use Utils.getSimpleName to avoid hitting Malformed class name in NewInstance.doGenCode
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #31709:
URL: https://github.com/apache/spark/pull/31709#issuecomment-789168373
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on pull request #31709: [SPARK-34596][SQL] Use Utils.getSimpleName to avoid hitting Malformed class name in NewInstance.doGenCode
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on pull request #31709:
URL: https://github.com/apache/spark/pull/31709#issuecomment-792051820
Got it, @kiszk ~
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] kiszk commented on pull request #31709: [SPARK-34596][SQL] Use Utils.getSimpleName to avoid hitting Malformed class name in NewInstance.doGenCode
Posted by GitBox <gi...@apache.org>.
kiszk commented on pull request #31709:
URL: https://github.com/apache/spark/pull/31709#issuecomment-791901448
I would like to share the finding that the following commands can reproduce the error. I hope that it can help you.
```
% build/sbt "catalyst/testOnly *AnsiCastSuiteWithAnsiModeOn *ExpressionEncoderSuite"
```
```
% build/sbt "catalyst/testOnly *AnsiCastSuiteWithAnsiModeOff *ExpressionEncoderSuite"
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on pull request #31709: [SPARK-34596][SQL] Use Utils.getSimpleName to avoid hitting Malformed class name in NewInstance.doGenCode
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on pull request #31709:
URL: https://github.com/apache/spark/pull/31709#issuecomment-791785032
In this case, `ExpressionEncoderSuite` extends `CodegenInterpretedPlanTest`. `CodegenInterpretedPlanTest` always do the following. It seems that we need to move this test case into other test suite.
```scala
withSQLConf(SQLConf.CODEGEN_FACTORY_MODE.key -> codegenMode) {
super.test(testName + " (codegen path)", testTags: _*)(testFun)(pos)
}
withSQLConf(SQLConf.CODEGEN_FACTORY_MODE.key -> interpretedMode) {
super.test(testName + " (interpreted path)", testTags: _*)(testFun)(pos)
}
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on pull request #31709: [SPARK-34596][SQL] Use Utils.getSimpleName to avoid hitting Malformed class name in NewInstance.doGenCode
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on pull request #31709:
URL: https://github.com/apache/spark/pull/31709#issuecomment-790130772
Yeah, let's backport. Looks we'll have another maintenance release that's likely EOL.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on pull request #31709: [SPARK-34596][SQL] Use Utils.getSimpleName to avoid hitting Malformed class name in NewInstance.doGenCode
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on pull request #31709:
URL: https://github.com/apache/spark/pull/31709#issuecomment-790125366
The fix seems to be applicable, but the test seems to fail unfortunately.
```
$ git diff HEAD~1
diff --git a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/objects/objects.scala b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/objects/objects.scala
index 3189e6841a..ba9e9ecebb 100644
--- a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/objects/objects.scala
+++ b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/objects/objects.scala
@@ -487,7 +487,7 @@ case class NewInstance(
ev.isNull = resultIsNull
val constructorCall = outer.map { gen =>
- s"${gen.value}.new ${cls.getSimpleName}($argString)"
+ s"${gen.value}.new ${Utils.getSimpleName(cls)}($argString)"
}.getOrElse {
s"new $className($argString)"
}
```
```
[info] - encode/decode for nested Scala class should work: MalformedNameExample(42) (codegen path) *** FAILED *** (34 milliseconds)
[info] Exception thrown while decoding
[info] Converted: [0,2a]
[info] Schema: x#836
[info] root
[info] -- x: integer (nullable = false)
[info]
[info]
[info] Encoder:
[info] class[x[0]: int] (ExpressionEncoderSuite.scala:399)
[info] org.scalatest.exceptions.TestFailedException:
[info] at org.scalatest.Assertions$class.newAssertionFailedException(Assertions.scala:528)
[info] at org.scalatest.FunSuite.newAssertionFailedException(FunSuite.scala:1560)
[info] at org.scalatest.Assertions$class.fail(Assertions.scala:1105)
[info] at org.scalatest.FunSuite.fail(FunSuite.scala:1560)
[info] at org.apache.spark.sql.catalyst.encoders.ExpressionEncoderSuite$$anonfun$encodeDecodeTest$1.apply(ExpressionEncoderSuite.scala:399)
[info] at org.apache.spark.sql.catalyst.encoders.ExpressionEncoderSuite.org$apache$spark$sql$catalyst$encoders$ExpressionEncoderSuite$$verifyNotLeakingReflectionObjects(ExpressionEncoderSuite.scala:477)
[info] at org.apache.spark.sql.catalyst.encoders.ExpressionEncoderSuite$$anonfun$testAndVerifyNotLeakingReflectionObjects$1.apply(ExpressionEncoderSuite.scala:484)
...
[info] Cause: java.lang.RuntimeException: Error while decoding: java.util.concurrent.ExecutionException: org.codehaus.commons.compiler.CompileException: File 'generated.java', Line 32, Column 124: failed to compile: org.codehaus.commons.compiler.CompileException: File 'generated.java', Line 32, Column 124: "org.apache.spark.sql.catalyst.encoders.ExpressionEncoderSuite$MalformedClassObject$" has no member type "ExpressionEncoderSuite$MalformedClassObject$MalformedNameExample"
[info] newInstance(class org.apache.spark.sql.catalyst.encoders.ExpressionEncoderSuite$MalformedClassObject$MalformedNameExample)
[info] at org.apache.spark.sql.catalyst.encoders.ExpressionEncoder.fromRow(ExpressionEncoder.scala:305)
[info] at org.apache.spark.sql.catalyst.encoders.ExpressionEncoderSuite$$anonfun$encodeDecodeTest$1.apply(ExpressionEncoderSuite.scala:397)
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #31709: [SPARK-34596][SQL] Use Utils.getSimpleName to avoid hitting Malformed class name in NewInstance.doGenCode
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #31709:
URL: https://github.com/apache/spark/pull/31709#issuecomment-788856281
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] rednaxelafx commented on pull request #31709: [SPARK-34596][SQL] Use Utils.getSimpleName to avoid hitting Malformed class name in NewInstance.doGenCode
Posted by GitBox <gi...@apache.org>.
rednaxelafx commented on pull request #31709:
URL: https://github.com/apache/spark/pull/31709#issuecomment-789179820
Looks like the AMPLab Jenkins is unstable, that there are test failures due to `java.io.IOException: Backing channel 'amp-jenkins-staging-worker-02' is disconnected.`
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #31709: [SPARK-34596][SQL] Use Utils.getSimpleName to avoid hitting Malformed class name in NewInstance.doGenCode
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #31709:
URL: https://github.com/apache/spark/pull/31709#issuecomment-789337073
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/40257/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] kiszk commented on pull request #31709: [SPARK-34596][SQL] Use Utils.getSimpleName to avoid hitting Malformed class name in NewInstance.doGenCode
Posted by GitBox <gi...@apache.org>.
kiszk commented on pull request #31709:
URL: https://github.com/apache/spark/pull/31709#issuecomment-791909719
In the above command, when the following line is replaced with `{`, we cannot see any failure
https://github.com/apache/spark/blob/master/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/CastSuite.scala#L1034
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] rednaxelafx commented on pull request #31709: [SPARK-34596][SQL] Use Utils.getSimpleName to avoid hitting Malformed class name in NewInstance.doGenCode
Posted by GitBox <gi...@apache.org>.
rednaxelafx commented on pull request #31709:
URL: https://github.com/apache/spark/pull/31709#issuecomment-790200499
Yes, let me check the situation on branch-3.1 and post back here
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31709: [SPARK-34596][SQL] Use Utils.getSimpleName to avoid hitting Malformed class name in NewInstance.doGenCode
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31709:
URL: https://github.com/apache/spark/pull/31709#issuecomment-789396524
**[Test build #135675 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/135675/testReport)** for PR 31709 at commit [`d4e92aa`](https://github.com/apache/spark/commit/d4e92aab30b6968703b734eac0713a55c5c0d385).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on pull request #31709: [SPARK-34596][SQL] Use Utils.getSimpleName to avoid hitting Malformed class name in NewInstance.doGenCode
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on pull request #31709:
URL: https://github.com/apache/spark/pull/31709#issuecomment-791872884
Sorry for the inconvenient. I'm looking at this.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31709: [SPARK-34596][SQL] Use Utils.getSimpleName to avoid hitting Malformed class name in NewInstance.doGenCode
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31709:
URL: https://github.com/apache/spark/pull/31709#issuecomment-789323947
Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/40257/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on pull request #31709: [SPARK-34596][SQL] Use Utils.getSimpleName to avoid hitting Malformed class name in NewInstance.doGenCode
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on pull request #31709:
URL: https://github.com/apache/spark/pull/31709#issuecomment-790202065
Shall we revert it from branch-3.1 for now?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on pull request #31709: [SPARK-34596][SQL] Use Utils.getSimpleName to avoid hitting Malformed class name in NewInstance.doGenCode
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on pull request #31709:
URL: https://github.com/apache/spark/pull/31709#issuecomment-792052654
I created a PR to master branch, too. And, I'll track the discussion and adjust according to the decision.
- https://github.com/apache/spark/pull/31766
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #31709: [SPARK-34596][SQL] Use Utils.getSimpleName to avoid hitting Malformed class name in NewInstance.doGenCode
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #31709:
URL: https://github.com/apache/spark/pull/31709#issuecomment-789406814
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/135675/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on pull request #31709: [SPARK-34596][SQL] Use Utils.getSimpleName to avoid hitting Malformed class name in NewInstance.doGenCode
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on pull request #31709:
URL: https://github.com/apache/spark/pull/31709#issuecomment-789968608
@rednaxelafx . Or, your patch is not applicable to `branch-2.4`?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] kiszk edited a comment on pull request #31709: [SPARK-34596][SQL] Use Utils.getSimpleName to avoid hitting Malformed class name in NewInstance.doGenCode
Posted by GitBox <gi...@apache.org>.
kiszk edited a comment on pull request #31709:
URL: https://github.com/apache/spark/pull/31709#issuecomment-791902837
I think that this is the almost minimum set, which can be easily specified, to reproduce the problem.
`ANSI mode: cast string to timestamp with parse error` and `encode/decode for nested Scala class should work` may have implicit dependency. I think that `SPARK-27671: cast from nested null type in struct` is not related to this reproduction.
```
% build/sbt "catalyst/testOnly *AnsiCastSuiteWithAnsiModeOn *ExpressionEncoderSuite -- -z nested -z parse "
...
[info] AnsiCastSuiteWithAnsiModeOn:
[info] - SPARK-27671: cast from nested null type in struct (1 second, 407 milliseconds)
[info] - ANSI mode: cast string to timestamp with parse error (1 second, 309 milliseconds)
[info] ExpressionEncoderSuite:
[info] - encode/decode for nested Scala class should work: MalformedNameExample(42) (codegen path) *** FAILED *** (203 milliseconds)
[info] Exception thrown while decoding
[info] Converted: [0,2a]
[info] Schema: x#18318
[info] root
[info] -- x: integer (nullable = false)
[info]
...
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun edited a comment on pull request #31709: [SPARK-34596][SQL] Use Utils.getSimpleName to avoid hitting Malformed class name in NewInstance.doGenCode
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun edited a comment on pull request #31709:
URL: https://github.com/apache/spark/pull/31709#issuecomment-791891022
Hi, All.
I created a WIP to put the newly added test cases into `FALLBACK` mode always.
- https://github.com/apache/spark/pull/31764
According to @rednaxelafx 's comment, it should pass conceptually. However, for now, it is failing still. So, I made a WIP to share it first and get your opinions.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on pull request #31709: [SPARK-34596][SQL] Use Utils.getSimpleName to avoid hitting Malformed class name in NewInstance.doGenCode
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on pull request #31709:
URL: https://github.com/apache/spark/pull/31709#issuecomment-791866237
@viirya . It was merged and reverted once . And, when @maropu 's patch arrives, I merged back with @maropu 's patch together. The AS-IS status is broken due to `CODEGEN_FACTORY_MODE` issue.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on pull request #31709: [SPARK-34596][SQL] Use Utils.getSimpleName to avoid hitting Malformed class name in NewInstance.doGenCode
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on pull request #31709:
URL: https://github.com/apache/spark/pull/31709#issuecomment-790193629
Ur, @rednaxelafx and @HyukjinKwon . The test fails even in `branch-3.1`.
- https://github.com/apache/spark/runs/2026199208
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #31709: [SPARK-34596][SQL] Use Utils.getSimpleName to avoid hitting Malformed class name in NewInstance.doGenCode
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #31709:
URL: https://github.com/apache/spark/pull/31709#issuecomment-789168377
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] kiszk commented on pull request #31709: [SPARK-34596][SQL] Use Utils.getSimpleName to avoid hitting Malformed class name in NewInstance.doGenCode
Posted by GitBox <gi...@apache.org>.
kiszk commented on pull request #31709:
URL: https://github.com/apache/spark/pull/31709#issuecomment-791991696
After adding the attached code, `build/sbt "catalyst/testOnly *AnsiCastSuiteWithAnsiModeOn *ExpressionEncoderSuite -- -z nested"` can reproduce the same error.
```
--- a/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/CastSuite.scala
+++ b/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/CastSuite.scala
@@ -960,6 +960,19 @@ abstract class AnsiCastSuiteBase extends CastSuiteBase {
"invalid input syntax for type numeric")
}
+ test("DUMMY nested") {
+ val activeConf = conf
+ seq(1).foreach { zid =>
+ SQLConf.withExistingConf(activeConf) {
+ withSQLConf(SQLConf.CODEGEN_FACTORY_MODE.key ->
+ org.apache.spark.sql.catalyst.expressions.CodegenObjectFactoryMode.CODEGEN_ONLY
+ .toString) {
+ val i = 1
+ }
+ }
+ }
+ }
+
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31709: [SPARK-34596][SQL] Use Utils.getSimpleName to avoid hitting Malformed class name in NewInstance.doGenCode
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31709:
URL: https://github.com/apache/spark/pull/31709#issuecomment-789281219
**[Test build #135675 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/135675/testReport)** for PR 31709 at commit [`d4e92aa`](https://github.com/apache/spark/commit/d4e92aab30b6968703b734eac0713a55c5c0d385).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on pull request #31709: [SPARK-34596][SQL] Use Utils.getSimpleName to avoid hitting Malformed class name in NewInstance.doGenCode
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on pull request #31709:
URL: https://github.com/apache/spark/pull/31709#issuecomment-789398515
Merged to master, branch-3.1 and branch-3.0. @rednaxelafx should we backport in bracnh-2.4 too?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on pull request #31709: [SPARK-34596][SQL] Use Utils.getSimpleName to avoid hitting Malformed class name in NewInstance.doGenCode
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on pull request #31709:
URL: https://github.com/apache/spark/pull/31709#issuecomment-789673861
I think it's not explicitly announced as an EOL though it will likely be EOL according to https://spark.apache.org/versioning-policy.html. Sure, so I think it's not to backport for now.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] kiszk edited a comment on pull request #31709: [SPARK-34596][SQL] Use Utils.getSimpleName to avoid hitting Malformed class name in NewInstance.doGenCode
Posted by GitBox <gi...@apache.org>.
kiszk edited a comment on pull request #31709:
URL: https://github.com/apache/spark/pull/31709#issuecomment-791901448
I would like to share the finding that the following commands can reproduce the error. I hope that it can help you.
```
% build/sbt "catalyst/testOnly *AnsiCastSuiteWithAnsiModeOn *ExpressionEncoderSuite"
```
or
```
% build/sbt "catalyst/testOnly *AnsiCastSuiteWithAnsiModeOff *ExpressionEncoderSuite"
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on pull request #31709: [SPARK-34596][SQL] Use Utils.getSimpleName to avoid hitting Malformed class name in NewInstance.doGenCode
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on pull request #31709:
URL: https://github.com/apache/spark/pull/31709#issuecomment-790125744
Could you please make a backporting PR to make it sure to pass CIs since Apache Spark 2.4 is not EOL officially and this patch is important?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun removed a comment on pull request #31709: [SPARK-34596][SQL] Use Utils.getSimpleName to avoid hitting Malformed class name in NewInstance.doGenCode
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun removed a comment on pull request #31709:
URL: https://github.com/apache/spark/pull/31709#issuecomment-791785032
In this case, `ExpressionEncoderSuite` extends `CodegenInterpretedPlanTest`. `CodegenInterpretedPlanTest` always do the following. It seems that we need to move this test case into other test suite.
```scala
withSQLConf(SQLConf.CODEGEN_FACTORY_MODE.key -> codegenMode) {
super.test(testName + " (codegen path)", testTags: _*)(testFun)(pos)
}
withSQLConf(SQLConf.CODEGEN_FACTORY_MODE.key -> interpretedMode) {
super.test(testName + " (interpreted path)", testTags: _*)(testFun)(pos)
}
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon closed pull request #31709: [SPARK-34596][SQL] Use Utils.getSimpleName to avoid hitting Malformed class name in NewInstance.doGenCode
Posted by GitBox <gi...@apache.org>.
HyukjinKwon closed pull request #31709:
URL: https://github.com/apache/spark/pull/31709
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on pull request #31709: [SPARK-34596][SQL] Use Utils.getSimpleName to avoid hitting Malformed class name in NewInstance.doGenCode
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on pull request #31709:
URL: https://github.com/apache/spark/pull/31709#issuecomment-790203983
👌 I reverted it from branch-3.1.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #31709: [SPARK-34596][SQL] Use Utils.getSimpleName to avoid hitting Malformed class name in NewInstance.doGenCode
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #31709:
URL: https://github.com/apache/spark/pull/31709#issuecomment-788856281
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/135655/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on pull request #31709: [SPARK-34596][SQL] Use Utils.getSimpleName to avoid hitting Malformed class name in NewInstance.doGenCode
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on pull request #31709:
URL: https://github.com/apache/spark/pull/31709#issuecomment-789912663
Well, we had better try to EOL. :)
> I think it's not explicitly announced as an EOL
Could you backport it to branch-2.4, @HyukjinKwon ? Recently, ZSTD library is also upgraded still.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun edited a comment on pull request #31709: [SPARK-34596][SQL] Use Utils.getSimpleName to avoid hitting Malformed class name in NewInstance.doGenCode
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun edited a comment on pull request #31709:
URL: https://github.com/apache/spark/pull/31709#issuecomment-791891022
Hi, All.
I created a WIP PR to put the newly added test cases into `FALLBACK` mode always.
- https://github.com/apache/spark/pull/31764
According to @rednaxelafx 's comment, it should pass conceptually. However, for now, it is failing still. So, I made a WIP to share it first and get your opinions.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] rednaxelafx commented on pull request #31709: [SPARK-34596][SQL] Use Utils.getSimpleName to avoid hitting Malformed class name in NewInstance.doGenCode
Posted by GitBox <gi...@apache.org>.
rednaxelafx commented on pull request #31709:
URL: https://github.com/apache/spark/pull/31709#issuecomment-789663061
> Merged to master, branch-3.1 and branch-3.0. @rednaxelafx should we backport in bracnh-2.4 too?
Since Spark 2.4 line is already EOL, I think we're safe to skip backporting to 2.4?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] rednaxelafx commented on pull request #31709: [SPARK-34596][SQL] Use Utils.getSimpleName to avoid hitting Malformed class name in NewInstance.doGenCode
Posted by GitBox <gi...@apache.org>.
rednaxelafx commented on pull request #31709:
URL: https://github.com/apache/spark/pull/31709#issuecomment-790408047
I tried running the commit in branch-3.1 and it passed when I only run the `ExpressionEncoderSuite`:
```
# the branch-3.1 commit hash, the one that was later reverted
$ git reset --hard 4c8403e687f3ef6573ba89b4b88ff29da31531a2
$ build/sbt "catalyst/testOnly *ExpressionEncoderSuite"
...
[info] ExpressionEncoderSuite:
...
[info] - encode/decode for nested Scala class should work: MalformedNameExample(42) (codegen path) (55 milliseconds)
[info] - encode/decode for nested Scala class should work: MalformedNameExample(42) (interpreted path) (26 milliseconds)
...
[info] - overflowing java very big int, ansiEnabled=false (interpreted path) (2 milliseconds)
[info] ScalaTest
[info] Run completed in 11 seconds, 813 milliseconds.
[info] Total number of tests run: 286
[info] Suites: completed 1, aborted 0
[info] Tests: succeeded 286, failed 0, canceled 0, ignored 0, pending 0
[info] All tests passed.
```
But when I run the entire Catalyst tests, the two new tests fails:
```
# on the same commit on branch-3.1
$ build/sbt "catalyst/test"
...
[info] ExpressionEncoderSuite:
...
[info] - encode/decode for nested Scala class should work: MalformedNameExample(42) (codegen path) *** FAILED *** (19 milliseconds)
[info] - encode/decode for nested Scala class should work: MalformedNameExample(42) (interpreted path) *** FAILED *** (11 milliseconds)
...
[info] *** 2 TESTS FAILED ***
[error] Failed: Total 4695, Failed 2, Errors 0, Passed 4693, Ignored 5
[error] Failed tests:
[error] org.apache.spark.sql.catalyst.encoders.ExpressionEncoderSuite
```
This seem to indicate that the test cases I provided are somewhat context sensitive, and may be affected by the order of tests. This could be related to how OuterScope works. Investigating.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] kiszk edited a comment on pull request #31709: [SPARK-34596][SQL] Use Utils.getSimpleName to avoid hitting Malformed class name in NewInstance.doGenCode
Posted by GitBox <gi...@apache.org>.
kiszk edited a comment on pull request #31709:
URL: https://github.com/apache/spark/pull/31709#issuecomment-791909719
In the above command, when the following line is replaced with `{`, we cannot see any failure
https://github.com/apache/spark/blob/f72b9068adadaa0478a5306c4778346b04ac8a47/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/CastSuite.scala#L1034
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun edited a comment on pull request #31709: [SPARK-34596][SQL] Use Utils.getSimpleName to avoid hitting Malformed class name in NewInstance.doGenCode
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun edited a comment on pull request #31709:
URL: https://github.com/apache/spark/pull/31709#issuecomment-791872884
Sorry for the inconvenience. I'm looking at this.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] viirya commented on pull request #31709: [SPARK-34596][SQL] Use Utils.getSimpleName to avoid hitting Malformed class name in NewInstance.doGenCode
Posted by GitBox <gi...@apache.org>.
viirya commented on pull request #31709:
URL: https://github.com/apache/spark/pull/31709#issuecomment-791872741
@dongjoon-hyun I see. Thanks for clarifying the situation.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on pull request #31709: [SPARK-34596][SQL] Use Utils.getSimpleName to avoid hitting Malformed class name in NewInstance.doGenCode
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on pull request #31709:
URL: https://github.com/apache/spark/pull/31709#issuecomment-791992810
@kiszk The problematic part is `val activeConf = conf` and `SQLConf.withExistingConf(activeConf)`, isn't it?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] kiszk commented on pull request #31709: [SPARK-34596][SQL] Use Utils.getSimpleName to avoid hitting Malformed class name in NewInstance.doGenCode
Posted by GitBox <gi...@apache.org>.
kiszk commented on pull request #31709:
URL: https://github.com/apache/spark/pull/31709#issuecomment-791902837
I think that this is the almost minimum set, which can be easily specified, to reproduce the problem.
`ANSI mode: cast string to timestamp with parse error` and `encode/decode for nested Scala class should work` may have implicit dependency.
```
% build/sbt "catalyst/testOnly *AnsiCastSuiteWithAnsiModeOn *ExpressionEncoderSuite -- -z nested -z parse "
...
[info] AnsiCastSuiteWithAnsiModeOn:
[info] - SPARK-27671: cast from nested null type in struct (1 second, 407 milliseconds)
[info] - ANSI mode: cast string to timestamp with parse error (1 second, 309 milliseconds)
[info] ExpressionEncoderSuite:
[info] - encode/decode for nested Scala class should work: MalformedNameExample(42) (codegen path) *** FAILED *** (203 milliseconds)
[info] Exception thrown while decoding
[info] Converted: [0,2a]
[info] Schema: x#18318
[info] root
[info] -- x: integer (nullable = false)
[info]
...
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on pull request #31709: [SPARK-34596][SQL] Use Utils.getSimpleName to avoid hitting Malformed class name in NewInstance.doGenCode
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on pull request #31709:
URL: https://github.com/apache/spark/pull/31709#issuecomment-790121588
Thank you for confirmation. I'll try to cherry-pick and test then.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on pull request #31709: [SPARK-34596][SQL] Use Utils.getSimpleName to avoid hitting Malformed class name in NewInstance.doGenCode
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on pull request #31709:
URL: https://github.com/apache/spark/pull/31709#issuecomment-791988550
I updated my PR according to @kiszk 's comment.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on pull request #31709: [SPARK-34596][SQL] Use Utils.getSimpleName to avoid hitting Malformed class name in NewInstance.doGenCode
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on pull request #31709:
URL: https://github.com/apache/spark/pull/31709#issuecomment-791778188
Ya. I agree with you the following.
> I think that intent is fulfilled with the fix in this PR when running in non-testing mode -- the codegen fallback to interpreter would step in and make it work, instead of getting a JDK java.lang.InternalError that Spark's codegen doesn't handle.
And, in that case, we should wrap the unit test case with `CODEGEN_FACTORY_MODE` setting explicitly because it works only in the fallback status. So, are you going to proceed in revise the test case in that way, @rednaxelafx ?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] rednaxelafx commented on pull request #31709: [SPARK-34596][SQL] Use Utils.getSimpleName to avoid hitting Malformed class name in NewInstance.doGenCode
Posted by GitBox <gi...@apache.org>.
rednaxelafx commented on pull request #31709:
URL: https://github.com/apache/spark/pull/31709#issuecomment-790133636
The test failure on Spark 2.4 may be due to a difference in either Scala version or Janino version in use vs Spark 3.0+.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] kiszk commented on pull request #31709: [SPARK-34596][SQL] Use Utils.getSimpleName to avoid hitting Malformed class name in NewInstance.doGenCode
Posted by GitBox <gi...@apache.org>.
kiszk commented on pull request #31709:
URL: https://github.com/apache/spark/pull/31709#issuecomment-791996116
I think so. Your change also works in the above case.
But, I am curious why the same code works at the master branch
https://github.com/apache/spark/blob/2f30cdebb18488a4ba83dd06aa5c5c7126dd8a8c/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/CastSuite.scala#L1026
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on pull request #31709: [SPARK-34596][SQL] Use Utils.getSimpleName to avoid hitting Malformed class name in NewInstance.doGenCode
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on pull request #31709:
URL: https://github.com/apache/spark/pull/31709#issuecomment-790195127
Could you recover it with the follow-up, @rednaxelafx ?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] rednaxelafx commented on pull request #31709: [SPARK-34596][SQL] Use Utils.getSimpleName to avoid hitting Malformed class name in NewInstance.doGenCode
Posted by GitBox <gi...@apache.org>.
rednaxelafx commented on pull request #31709:
URL: https://github.com/apache/spark/pull/31709#issuecomment-790120234
@dongjoon-hyun this bug does affect Spark 2.4 branch as well (in fact to previous version too). The same fix could be applied to 2.4.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on pull request #31709: [SPARK-34596][SQL] Use Utils.getSimpleName to avoid hitting Malformed class name in NewInstance.doGenCode
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on pull request #31709:
URL: https://github.com/apache/spark/pull/31709#issuecomment-789255857
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] kiszk edited a comment on pull request #31709: [SPARK-34596][SQL] Use Utils.getSimpleName to avoid hitting Malformed class name in NewInstance.doGenCode
Posted by GitBox <gi...@apache.org>.
kiszk edited a comment on pull request #31709:
URL: https://github.com/apache/spark/pull/31709#issuecomment-791909719
In the above command, when the following line is replaced with `{`, we cannot see any failure
https://github.com/apache/spark/blob/e3ee8ea4545cbe8248f5b2d67849c6ae6e231566/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/CastSuite.scala#L972
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on pull request #31709: [SPARK-34596][SQL] Use Utils.getSimpleName to avoid hitting Malformed class name in NewInstance.doGenCode
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on pull request #31709:
URL: https://github.com/apache/spark/pull/31709#issuecomment-791976022
Thank you, @kiszk !
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31709: [SPARK-34596][SQL] Use Utils.getSimpleName to avoid hitting Malformed class name in NewInstance.doGenCode
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31709:
URL: https://github.com/apache/spark/pull/31709#issuecomment-789117625
**[Test build #135671 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/135671/testReport)** for PR 31709 at commit [`d4e92aa`](https://github.com/apache/spark/commit/d4e92aab30b6968703b734eac0713a55c5c0d385).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #31709: [SPARK-34596][SQL] Use Utils.getSimpleName to avoid hitting Malformed class name in NewInstance.doGenCode
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #31709:
URL: https://github.com/apache/spark/pull/31709#issuecomment-789406814
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/135675/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun edited a comment on pull request #31709: [SPARK-34596][SQL] Use Utils.getSimpleName to avoid hitting Malformed class name in NewInstance.doGenCode
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun edited a comment on pull request #31709:
URL: https://github.com/apache/spark/pull/31709#issuecomment-790195127
Could you recover it with the follow-up on `branch-3.1`, @rednaxelafx ? BTW, branch-3.0 looks okay.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on pull request #31709: [SPARK-34596][SQL] Use Utils.getSimpleName to avoid hitting Malformed class name in NewInstance.doGenCode
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on pull request #31709:
URL: https://github.com/apache/spark/pull/31709#issuecomment-791891022
Hi, All.
I created a WIP to put the newly added test cases into `FALLBACK` mode always.
- https://github.com/apache/spark/pull/31764
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] viirya commented on pull request #31709: [SPARK-34596][SQL] Use Utils.getSimpleName to avoid hitting Malformed class name in NewInstance.doGenCode
Posted by GitBox <gi...@apache.org>.
viirya commented on pull request #31709:
URL: https://github.com/apache/spark/pull/31709#issuecomment-791847224
> 👌 I reverted it from branch-3.1.
Do we? I still see the commit in branch-3.1.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #31709: [SPARK-34596][SQL] Use Utils.getSimpleName to avoid hitting Malformed class name in NewInstance.doGenCode
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #31709:
URL: https://github.com/apache/spark/pull/31709#issuecomment-789281219
**[Test build #135675 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/135675/testReport)** for PR 31709 at commit [`d4e92aa`](https://github.com/apache/spark/commit/d4e92aab30b6968703b734eac0713a55c5c0d385).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31709: [SPARK-34596][SQL] Use Utils.getSimpleName to avoid hitting Malformed class name in NewInstance.doGenCode
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31709:
URL: https://github.com/apache/spark/pull/31709#issuecomment-789308889
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/40257/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] rednaxelafx commented on pull request #31709: [SPARK-34596][SQL] Use Utils.getSimpleName to avoid hitting Malformed class name in NewInstance.doGenCode
Posted by GitBox <gi...@apache.org>.
rednaxelafx commented on pull request #31709:
URL: https://github.com/apache/spark/pull/31709#issuecomment-791319162
> Maybe we set `CODEGEN_FACTORY_MODE` somewhere without setting it back
That is my guess as well, thus my earlier guess of the test being somewhat context/order sensitive
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] kiszk edited a comment on pull request #31709: [SPARK-34596][SQL] Use Utils.getSimpleName to avoid hitting Malformed class name in NewInstance.doGenCode
Posted by GitBox <gi...@apache.org>.
kiszk edited a comment on pull request #31709:
URL: https://github.com/apache/spark/pull/31709#issuecomment-791991696
After adding the attached code, `build/sbt "catalyst/testOnly *AnsiCastSuiteWithAnsiModeOn *ExpressionEncoderSuite -- -z nested"` can reproduce the same error.
```
--- a/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/CastSuite.scala
+++ b/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/CastSuite.scala
@@ -960,6 +960,19 @@ abstract class AnsiCastSuiteBase extends CastSuiteBase {
"invalid input syntax for type numeric")
}
+ test("DUMMY nested") {
+ val activeConf = conf
+ new ParVector(ALL_TIMEZONES.toVector).foreach { zid =>
+ SQLConf.withExistingConf(activeConf) {
+ withSQLConf(SQLConf.CODEGEN_FACTORY_MODE.key ->
+ org.apache.spark.sql.catalyst.expressions.CodegenObjectFactoryMode.CODEGEN_ONLY
+ .toString) {
+ val i = 1
+ }
+ }
+ }
+ }
+
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] kiszk edited a comment on pull request #31709: [SPARK-34596][SQL] Use Utils.getSimpleName to avoid hitting Malformed class name in NewInstance.doGenCode
Posted by GitBox <gi...@apache.org>.
kiszk edited a comment on pull request #31709:
URL: https://github.com/apache/spark/pull/31709#issuecomment-791996116
I think so in branch 3.1. Your change also works in the above case.
But, I am curious why the same code works at the master branch cc @rednaxelafx
https://github.com/apache/spark/blob/2f30cdebb18488a4ba83dd06aa5c5c7126dd8a8c/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/CastSuite.scala#L1026
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org