You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2020/12/21 17:21:45 UTC
[GitHub] [spark] xkrogen opened a new pull request #30877: [SPARK-23862][SQL] Support Java enums from Scala Dataset API
xkrogen opened a new pull request #30877:
URL: https://github.com/apache/spark/pull/30877
### What changes were proposed in this pull request?
Add support for Java Enums (`java.lang.Enum`) from the Scala typed Dataset APIs. This involves adding an implicit for `Encoder` creation in `SQLImplicits`, and updating `ScalaReflection` to handle Java Enums on the serialization and deserialization pathways.
Enums are mapped to a `StringType` which is just the name of the Enum value.
### Why are the changes needed?
In [SPARK-21255](https://issues.apache.org/jira/browse/SPARK-21255), support for (de)serialization of Java Enums was added, but only when called from Java code. It is common for Scala code to rely on Java libraries that are out of control of the Scala developer. Today, if there is a dependency on some Java code which defines an Enum, it would be necessary to define a corresponding Scala class. This change brings closer feature parity between Scala and Java APIs.
### Does this PR introduce _any_ user-facing change?
Yes, previously something like:
```
val ds = Seq(MyJavaEnum.VALUE1, MyJavaEnum.VALUE2).toDS
// or
val ds = Seq(CaseClass(MyJavaEnum.VALUE1), CaseClass(MyJavaEnum.VALUE2)).toDS
```
would fail. Now, it will succeed.
### How was this patch tested?
Additional unit tests are added in `DatasetSuite`. Tests include validating top-level enums, enums inside of case classes, enums inside of arrays, and validating that the Enum is stored as the expected string.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #30877: [SPARK-23862][SQL] Support Java enums from Scala Dataset API
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #30877:
URL: https://github.com/apache/spark/pull/30877#issuecomment-749471598
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/133196/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #30877: [SPARK-23862][SQL] Support Java enums from Scala Dataset API
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #30877:
URL: https://github.com/apache/spark/pull/30877#issuecomment-749271199
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30877: [SPARK-23862][SQL] Support Java enums from Scala Dataset API
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30877:
URL: https://github.com/apache/spark/pull/30877#issuecomment-749133265
Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/37772/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30877: [SPARK-23862][SQL] Support Java enums from Scala Dataset API
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30877:
URL: https://github.com/apache/spark/pull/30877#issuecomment-749466534
**[Test build #133196 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/133196/testReport)** for PR 30877 at commit [`88b7b3c`](https://github.com/apache/spark/commit/88b7b3c3528e0e127e88db98b4ff15cff21ddcef).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30877: [SPARK-23862][SQL] Support Java enums from Scala Dataset API
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30877:
URL: https://github.com/apache/spark/pull/30877#issuecomment-749352528
**[Test build #133196 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/133196/testReport)** for PR 30877 at commit [`88b7b3c`](https://github.com/apache/spark/commit/88b7b3c3528e0e127e88db98b4ff15cff21ddcef).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] xkrogen commented on a change in pull request #30877: [SPARK-23862][SQL] Support Java enums from Scala Dataset API
Posted by GitBox <gi...@apache.org>.
xkrogen commented on a change in pull request #30877:
URL: https://github.com/apache/spark/pull/30877#discussion_r547079407
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/ScalaReflection.scala
##########
@@ -232,6 +232,11 @@ object ScalaReflection extends ScalaReflection {
case t if isSubtype(t, localTypeOf[java.time.Instant]) =>
createDeserializerForInstant(path)
+ case t if t <:< localTypeOf[java.lang.Enum[_]] =>
Review comment:
Good catch! I fixed most of the references to `<:<` from the original PR but it looks like I missed this one. Updated now.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #30877: [SPARK-23862][SQL] Support Java enums from Scala Dataset API
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #30877:
URL: https://github.com/apache/spark/pull/30877#issuecomment-749271199
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on pull request #30877: [SPARK-23862][SQL] Support Java enums from Scala Dataset API
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on pull request #30877:
URL: https://github.com/apache/spark/pull/30877#issuecomment-749229001
Thank you for updating, @xkrogen .
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #30877: [SPARK-23862][SQL] Support Java enums from Scala Dataset API
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #30877:
URL: https://github.com/apache/spark/pull/30877#issuecomment-749471598
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/133196/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] xkrogen commented on pull request #30877: [SPARK-23862][SQL] Support Java enums from Scala Dataset API
Posted by GitBox <gi...@apache.org>.
xkrogen commented on pull request #30877:
URL: https://github.com/apache/spark/pull/30877#issuecomment-750364438
Thank you @dongjoon-hyun ! Much appreciated!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30877: [SPARK-23862][SQL] Support Java enums from Scala Dataset API
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30877:
URL: https://github.com/apache/spark/pull/30877#issuecomment-749101820
**[Test build #133173 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/133173/testReport)** for PR 30877 at commit [`9c2d772`](https://github.com/apache/spark/commit/9c2d77210594c12680a6f5b094bdf6e0dff4510d).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #30877: [SPARK-23862][SQL] Support Java enums from Scala Dataset API
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #30877:
URL: https://github.com/apache/spark/pull/30877#issuecomment-749352528
**[Test build #133196 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/133196/testReport)** for PR 30877 at commit [`88b7b3c`](https://github.com/apache/spark/commit/88b7b3c3528e0e127e88db98b4ff15cff21ddcef).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30877: [SPARK-23862][SQL] Support Java enums from Scala Dataset API
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30877:
URL: https://github.com/apache/spark/pull/30877#issuecomment-749158809
**[Test build #133180 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/133180/testReport)** for PR 30877 at commit [`d71238b`](https://github.com/apache/spark/commit/d71238bf03a5f874753d1715198f39cc1d7b745e).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #30877: [SPARK-23862][SQL] Support Java enums from Scala Dataset API
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #30877:
URL: https://github.com/apache/spark/pull/30877#issuecomment-749126127
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30877: [SPARK-23862][SQL] Support Java enums from Scala Dataset API
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30877:
URL: https://github.com/apache/spark/pull/30877#issuecomment-749156739
**[Test build #133179 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/133179/testReport)** for PR 30877 at commit [`9c2d772`](https://github.com/apache/spark/commit/9c2d77210594c12680a6f5b094bdf6e0dff4510d).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun closed pull request #30877: [SPARK-23862][SQL] Support Java enums from Scala Dataset API
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun closed pull request #30877:
URL: https://github.com/apache/spark/pull/30877
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #30877: [SPARK-23862][SQL] Support Java enums from Scala Dataset API
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #30877:
URL: https://github.com/apache/spark/pull/30877#issuecomment-749126127
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/133173/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #30877: [SPARK-23862][SQL] Support Java enums from Scala Dataset API
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #30877:
URL: https://github.com/apache/spark/pull/30877#issuecomment-749156739
**[Test build #133179 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/133179/testReport)** for PR 30877 at commit [`9c2d772`](https://github.com/apache/spark/commit/9c2d77210594c12680a6f5b094bdf6e0dff4510d).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #30877: [SPARK-23862][SQL] Support Java enums from Scala Dataset API
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #30877:
URL: https://github.com/apache/spark/pull/30877#issuecomment-749393606
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/37794/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #30877: [SPARK-23862][SQL] Support Java enums from Scala Dataset API
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #30877:
URL: https://github.com/apache/spark/pull/30877#issuecomment-749133281
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/37772/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on a change in pull request #30877: [SPARK-23862][SQL] Support Java enums from Scala Dataset API
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on a change in pull request #30877:
URL: https://github.com/apache/spark/pull/30877#discussion_r546964752
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/ScalaReflection.scala
##########
@@ -232,6 +232,11 @@ object ScalaReflection extends ScalaReflection {
case t if isSubtype(t, localTypeOf[java.time.Instant]) =>
createDeserializerForInstant(path)
+ case t if t <:< localTypeOf[java.lang.Enum[_]] =>
Review comment:
Could you use `isSubtype` instead of `<:<`? Apache Spark is using the following to avoid `Scala` language bug. I'm not sure if it's fixed in the latest Scala version, but Apache Spark is still on `2.12.10`. So, we should use `isSubtype` here.
```scala
/**
* Synchronize to prevent concurrent usage of `<:<` operator.
* This operator is not thread safe in any current version of scala; i.e.
* (2.11.12, 2.12.10, 2.13.0-M5).
*
* See https://github.com/scala/bug/issues/10766
*/
private[catalyst] def isSubtype(tpe1: `Type`, tpe2: `Type`): Boolean = {
ScalaSubtypeLock.synchronized {
tpe1 <:< tpe2
}
}
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30877: [SPARK-23862][SQL] Support Java enums from Scala Dataset API
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30877:
URL: https://github.com/apache/spark/pull/30877#issuecomment-749262071
**[Test build #133180 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/133180/testReport)** for PR 30877 at commit [`d71238b`](https://github.com/apache/spark/commit/d71238bf03a5f874753d1715198f39cc1d7b745e).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30877: [SPARK-23862][SQL] Support Java enums from Scala Dataset API
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30877:
URL: https://github.com/apache/spark/pull/30877#issuecomment-749260166
**[Test build #133179 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/133179/testReport)** for PR 30877 at commit [`9c2d772`](https://github.com/apache/spark/commit/9c2d77210594c12680a6f5b094bdf6e0dff4510d).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #30877: [SPARK-23862][SQL] Support Java enums from Scala Dataset API
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #30877:
URL: https://github.com/apache/spark/pull/30877#issuecomment-749158809
**[Test build #133180 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/133180/testReport)** for PR 30877 at commit [`d71238b`](https://github.com/apache/spark/commit/d71238bf03a5f874753d1715198f39cc1d7b745e).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30877: [SPARK-23862][SQL] Support Java enums from Scala Dataset API
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30877:
URL: https://github.com/apache/spark/pull/30877#issuecomment-749196111
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/37778/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30877: [SPARK-23862][SQL] Support Java enums from Scala Dataset API
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30877:
URL: https://github.com/apache/spark/pull/30877#issuecomment-749120243
**[Test build #133173 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/133173/testReport)** for PR 30877 at commit [`9c2d772`](https://github.com/apache/spark/commit/9c2d77210594c12680a6f5b094bdf6e0dff4510d).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on a change in pull request #30877: [SPARK-23862][SQL] Support Java enums from Scala Dataset API
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on a change in pull request #30877:
URL: https://github.com/apache/spark/pull/30877#discussion_r546964752
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/ScalaReflection.scala
##########
@@ -232,6 +232,11 @@ object ScalaReflection extends ScalaReflection {
case t if isSubtype(t, localTypeOf[java.time.Instant]) =>
createDeserializerForInstant(path)
+ case t if t <:< localTypeOf[java.lang.Enum[_]] =>
Review comment:
Could you use `isSubtype` instead of `<:<`? Apache Spark is using the following to avoid `Scala` language bug. I'm not sure if it's fixed in the latest Scala version, but Apache Spark is still on `2.12.10`. So, we should use `isSubtype` here.
- https://github.com/apache/spark/blob/master/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/ScalaReflection.scala#L74-L85
```scala
/**
* Synchronize to prevent concurrent usage of `<:<` operator.
* This operator is not thread safe in any current version of scala; i.e.
* (2.11.12, 2.12.10, 2.13.0-M5).
*
* See https://github.com/scala/bug/issues/10766
*/
private[catalyst] def isSubtype(tpe1: `Type`, tpe2: `Type`): Boolean = {
ScalaSubtypeLock.synchronized {
tpe1 <:< tpe2
}
}
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #30877: [SPARK-23862][SQL] Support Java enums from Scala Dataset API
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #30877:
URL: https://github.com/apache/spark/pull/30877#issuecomment-749101820
**[Test build #133173 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/133173/testReport)** for PR 30877 at commit [`9c2d772`](https://github.com/apache/spark/commit/9c2d77210594c12680a6f5b094bdf6e0dff4510d).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #30877: [SPARK-23862][SQL] Support Java enums from Scala Dataset API
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #30877:
URL: https://github.com/apache/spark/pull/30877#issuecomment-749205409
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/37778/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30877: [SPARK-23862][SQL] Support Java enums from Scala Dataset API
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30877:
URL: https://github.com/apache/spark/pull/30877#issuecomment-749181071
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/37778/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #30877: [SPARK-23862][SQL] Support Java enums from Scala Dataset API
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #30877:
URL: https://github.com/apache/spark/pull/30877#issuecomment-749205409
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/37778/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #30877: [SPARK-23862][SQL] Support Java enums from Scala Dataset API
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #30877:
URL: https://github.com/apache/spark/pull/30877#issuecomment-749393606
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/37794/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on a change in pull request #30877: [SPARK-23862][SQL] Support Java enums from Scala Dataset API
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on a change in pull request #30877:
URL: https://github.com/apache/spark/pull/30877#discussion_r546964752
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/ScalaReflection.scala
##########
@@ -232,6 +232,11 @@ object ScalaReflection extends ScalaReflection {
case t if isSubtype(t, localTypeOf[java.time.Instant]) =>
createDeserializerForInstant(path)
+ case t if t <:< localTypeOf[java.lang.Enum[_]] =>
Review comment:
Could you use `isSubtype` instead of `<:<`? Apache Spark is using the following to avoid `Scala` language bug. I'm not sure if it's fixed in the latest Scala version, but Apache Spark is still on `2.12.10`.
```
/**
* Synchronize to prevent concurrent usage of `<:<` operator.
* This operator is not thread safe in any current version of scala; i.e.
* (2.11.12, 2.12.10, 2.13.0-M5).
*
* See https://github.com/scala/bug/issues/10766
*/
private[catalyst] def isSubtype(tpe1: `Type`, tpe2: `Type`): Boolean = {
ScalaSubtypeLock.synchronized {
tpe1 <:< tpe2
}
}
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on pull request #30877: [SPARK-23862][SQL] Support Java enums from Scala Dataset API
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on pull request #30877:
URL: https://github.com/apache/spark/pull/30877#issuecomment-749149511
Retest this please.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] xkrogen commented on pull request #30877: [SPARK-23862][SQL] Support Java enums from Scala Dataset API
Posted by GitBox <gi...@apache.org>.
xkrogen commented on pull request #30877:
URL: https://github.com/apache/spark/pull/30877#issuecomment-749096598
Note that these changes were originally proposed by @fangshil in PR #20974; though reception was warm, the PR was not pushed to completion. I am picking up this work and re-submitting with modifications to better match the current state of the code, and enhance test coverage.
cc @cloud-fan @HyukjinKwon @srowen
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30877: [SPARK-23862][SQL] Support Java enums from Scala Dataset API
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30877:
URL: https://github.com/apache/spark/pull/30877#issuecomment-749120761
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/37772/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on a change in pull request #30877: [SPARK-23862][SQL] Support Java enums from Scala Dataset API
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on a change in pull request #30877:
URL: https://github.com/apache/spark/pull/30877#discussion_r546964752
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/ScalaReflection.scala
##########
@@ -232,6 +232,11 @@ object ScalaReflection extends ScalaReflection {
case t if isSubtype(t, localTypeOf[java.time.Instant]) =>
createDeserializerForInstant(path)
+ case t if t <:< localTypeOf[java.lang.Enum[_]] =>
Review comment:
Could you use `isSubtype` instead of `<:<`? Apache Spark is using the following to avoid `Scala` language bug. I'm not sure if it's fixed in the latest Scala version, but Apache Spark is still on `2.12.10`.
```scala
/**
* Synchronize to prevent concurrent usage of `<:<` operator.
* This operator is not thread safe in any current version of scala; i.e.
* (2.11.12, 2.12.10, 2.13.0-M5).
*
* See https://github.com/scala/bug/issues/10766
*/
private[catalyst] def isSubtype(tpe1: `Type`, tpe2: `Type`): Boolean = {
ScalaSubtypeLock.synchronized {
tpe1 <:< tpe2
}
}
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] xkrogen commented on pull request #30877: [SPARK-23862][SQL] Support Java enums from Scala Dataset API
Posted by GitBox <gi...@apache.org>.
xkrogen commented on pull request #30877:
URL: https://github.com/apache/spark/pull/30877#issuecomment-749157754
Thanks for the tip @dongjoon-hyun ! Just pushed up such a commit.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org