You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by gmoehler <gi...@git.apache.org> on 2017/01/20 15:23:33 UTC
[GitHub] spark pull request #16660: [SPARK-19311][SQL] fix UDT hierarchy issue
GitHub user gmoehler opened a pull request:
https://github.com/apache/spark/pull/16660
[SPARK-19311][SQL] fix UDT hierarchy issue
## What changes were proposed in this pull request?
acceptType() in UDT will no only accept the same type but also all base types
## How was this patch tested?
Manual test using a set of generated UDTs fixing acceptType() in my user defined types
Please review http://spark.apache.org/contributing.html before opening a pull request.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/gmoehler/spark master
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/16660.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #16660
----
commit 7ea9aa636f430a30b8d83ed2dda954fd06347d79
Author: gmoehler <mo...@de.ibm.com>
Date: 2017-01-20T15:16:53Z
fix UDT hierarchy issue https://issues.apache.org/jira/browse/SPARK-19311
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16660: [SPARK-19311][SQL] fix UDT hierarchy issue
Posted by viirya <gi...@git.apache.org>.
Github user viirya commented on the issue:
https://github.com/apache/spark/pull/16660
cc @cloud-fan @rxin Although UDT is now private API, some developers still use it by defining their codes as spark package. I am not sure if you think we need to fix this or not.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #16660: [SPARK-19311][SQL] fix UDT hierarchy issue
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/spark/pull/16660
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16660: [SPARK-19311][SQL] fix UDT hierarchy issue
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/16660
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/71849/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16660: [SPARK-19311][SQL] fix UDT hierarchy issue
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/16660
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/71841/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16660: [SPARK-19311][SQL] fix UDT hierarchy issue
Posted by gatorsmile <gi...@git.apache.org>.
Github user gatorsmile commented on the issue:
https://github.com/apache/spark/pull/16660
ok to test
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16660: [SPARK-19311][SQL] fix UDT hierarchy issue
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/16660
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16660: [SPARK-19311][SQL] fix UDT hierarchy issue
Posted by gatorsmile <gi...@git.apache.org>.
Github user gatorsmile commented on the issue:
https://github.com/apache/spark/pull/16660
You can add a test case in https://github.com/apache/spark/blob/master/sql/core/src/test/scala/org/apache/spark/sql/UserDefinedTypeSuite.scala
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #16660: [SPARK-19311][SQL] fix UDT hierarchy issue
Posted by gatorsmile <gi...@git.apache.org>.
Github user gatorsmile commented on a diff in the pull request:
https://github.com/apache/spark/pull/16660#discussion_r97427446
--- Diff: sql/core/src/test/scala/org/apache/spark/sql/UserDefinedTypeSuite.scala ---
@@ -20,13 +20,15 @@ package org.apache.spark.sql
import scala.beans.{BeanInfo, BeanProperty}
import org.apache.spark.rdd.RDD
-import org.apache.spark.sql.catalyst.CatalystTypeConverters
+import org.apache.spark.sql.catalyst.{CatalystTypeConverters, InternalRow}
+import org.apache.spark.sql.catalyst.expressions.GenericInternalRow
import org.apache.spark.sql.catalyst.util.{ArrayData, GenericArrayData}
import org.apache.spark.sql.execution.datasources.parquet.ParquetTest
import org.apache.spark.sql.functions._
import org.apache.spark.sql.test.SharedSQLContext
import org.apache.spark.sql.types._
+
--- End diff --
Nit: Please remove this empty line for avoiding unnecessary changes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16660: [SPARK-19311][SQL] fix UDT hierarchy issue
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/16660
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #16660: [SPARK-19311][SQL] fix UDT hierarchy issue
Posted by gmoehler <gi...@git.apache.org>.
Github user gmoehler commented on a diff in the pull request:
https://github.com/apache/spark/pull/16660#discussion_r97498083
--- Diff: sql/core/src/test/scala/org/apache/spark/sql/UserDefinedTypeSuite.scala ---
@@ -71,7 +73,104 @@ object UDT {
}
+// object and classes to test SPARK-19311
+
+// Trait/Interface for base type
+@SQLUserDefinedType(udt = classOf[ExampleBaseTypeUDT])
+sealed trait IExampleBaseType extends Serializable {
+ def field: Int
+}
+
+// Trait/Interface for derived type
+@SQLUserDefinedType(udt = classOf[ExampleSubTypeUDT])
+sealed trait IExampleSubType extends IExampleBaseType
+
+// a base class
+class ExampleBaseClass(override val field: Int) extends IExampleBaseType {
+ override def toString: String = field.toString
+
+}
+
+// a derived class
+class ExampleSubClass(override val field: Int)
+ extends ExampleBaseClass(field) with IExampleSubType
+
+// UDT for base class
+private[spark] class ExampleBaseTypeUDT extends UserDefinedType[IExampleBaseType] {
+
+ override def sqlType: StructType = {
+ StructType(Seq(
+ StructField("intfield", IntegerType, nullable = false)))
+ }
+
+ override def serialize(obj: IExampleBaseType): InternalRow = {
+ val row = new GenericInternalRow(1)
+ row.setInt(0, obj.field)
+ row
+ }
+
+ override def deserialize(datum: Any): IExampleBaseType = {
+ datum match {
+ case row: InternalRow =>
+ require(row.numFields == 1,
+ s"VectorUDT.deserialize given row with length " +
+ s"${row.numFields} but requires length == 1")
+ val field = row.getInt(0)
+ new ExampleBaseClass(field)
+ }
+ }
+
+ override def userClass: Class[IExampleBaseType] = classOf[IExampleBaseType]
+
+ override def hashCode(): Int = classOf[ExampleBaseTypeUDT].getName.hashCode()
+
+ override def equals(other: Any): Boolean = other.isInstanceOf[ExampleBaseTypeUDT]
+
+ override def typeName: String = "exampleBaseType"
+
+ private[spark] override def asNullable: ExampleBaseTypeUDT = this
--- End diff --
ok
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16660: [SPARK-19311][SQL] fix UDT hierarchy issue
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/16660
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/71846/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16660: [SPARK-19311][SQL] fix UDT hierarchy issue
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/16660
**[Test build #71852 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/71852/testReport)** for PR 16660 at commit [`4ae0555`](https://github.com/apache/spark/commit/4ae05559d9a1b343e61e87ad0e4841c497c13382).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds the following public classes _(experimental)_:
* `sealed trait IExampleBaseType extends Serializable `
* `class ExampleBaseClass(override val field: Int) extends IExampleBaseType `
* `class ExampleSubClass(override val field: Int)`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16660: [SPARK-19311][SQL] fix UDT hierarchy issue
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/16660
**[Test build #71755 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/71755/testReport)** for PR 16660 at commit [`7ea9aa6`](https://github.com/apache/spark/commit/7ea9aa636f430a30b8d83ed2dda954fd06347d79).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #16660: [SPARK-19311][SQL] fix UDT hierarchy issue
Posted by gmoehler <gi...@git.apache.org>.
Github user gmoehler commented on a diff in the pull request:
https://github.com/apache/spark/pull/16660#discussion_r97333990
--- Diff: sql/core/src/test/scala/org/apache/spark/sql/UserDefinedTypeSuite.scala ---
@@ -71,6 +71,94 @@ object UDT {
}
+// object and classes to test SPARK-19311
+
+ // Trait/Interface for base type
--- End diff --
corrected in latest commit ... thanks for pointing out
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16660: [SPARK-19311][SQL] fix UDT hierarchy issue
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/16660
**[Test build #71841 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/71841/testReport)** for PR 16660 at commit [`fad9f0e`](https://github.com/apache/spark/commit/fad9f0e1bc98ad5e7ee2c8dfc726ddc979204f11).
* This patch **fails Scala style tests**.
* This patch merges cleanly.
* This patch adds the following public classes _(experimental)_:
* ` sealed trait IExampleBaseType extends Serializable `
* ` class ExampleBaseClass(override val field: Int) extends IExampleBaseType `
* ` class ExampleSubClass(override val field: Int)`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #16660: [SPARK-19311][SQL] fix UDT hierarchy issue
Posted by gmoehler <gi...@git.apache.org>.
Github user gmoehler commented on a diff in the pull request:
https://github.com/apache/spark/pull/16660#discussion_r97503551
--- Diff: sql/core/src/test/scala/org/apache/spark/sql/UserDefinedTypeSuite.scala ---
@@ -71,7 +73,104 @@ object UDT {
}
+// object and classes to test SPARK-19311
+
+// Trait/Interface for base type
+@SQLUserDefinedType(udt = classOf[ExampleBaseTypeUDT])
+sealed trait IExampleBaseType extends Serializable {
+ def field: Int
+}
+
+// Trait/Interface for derived type
+@SQLUserDefinedType(udt = classOf[ExampleSubTypeUDT])
+sealed trait IExampleSubType extends IExampleBaseType
+
+// a base class
+class ExampleBaseClass(override val field: Int) extends IExampleBaseType {
+ override def toString: String = field.toString
+
+}
+
+// a derived class
+class ExampleSubClass(override val field: Int)
+ extends ExampleBaseClass(field) with IExampleSubType
+
+// UDT for base class
+private[spark] class ExampleBaseTypeUDT extends UserDefinedType[IExampleBaseType] {
+
+ override def sqlType: StructType = {
+ StructType(Seq(
+ StructField("intfield", IntegerType, nullable = false)))
+ }
+
+ override def serialize(obj: IExampleBaseType): InternalRow = {
+ val row = new GenericInternalRow(1)
+ row.setInt(0, obj.field)
+ row
+ }
+
+ override def deserialize(datum: Any): IExampleBaseType = {
+ datum match {
+ case row: InternalRow =>
+ require(row.numFields == 1,
+ s"VectorUDT.deserialize given row with length " +
+ s"${row.numFields} but requires length == 1")
+ val field = row.getInt(0)
+ new ExampleBaseClass(field)
+ }
+ }
+
+ override def userClass: Class[IExampleBaseType] = classOf[IExampleBaseType]
+
+ override def hashCode(): Int = classOf[ExampleBaseTypeUDT].getName.hashCode()
+
+ override def equals(other: Any): Boolean = other.isInstanceOf[ExampleBaseTypeUDT]
+
+ override def typeName: String = "exampleBaseType"
+
+ private[spark] override def asNullable: ExampleBaseTypeUDT = this
+}
+
+// UDT for derived class
+private[spark] class ExampleSubTypeUDT extends UserDefinedType[IExampleSubType] {
+
+ override def sqlType: StructType = {
+ StructType(Seq(
+ StructField("intfield", IntegerType, nullable = false)))
+ }
+
+ override def serialize(obj: IExampleSubType): InternalRow = {
+
+ val row = new GenericInternalRow(1)
+ row.setInt(0, obj.field)
+ row
+ }
+
+ override def deserialize(datum: Any): IExampleSubType = {
+ datum match {
+ case row: InternalRow =>
+ require(row.numFields == 1,
+ s"VectorUDT.deserialize given row with length " +
+ s"${row.numFields} but requires length == 1")
+ val field = row.getInt(0)
+ new ExampleSubClass(field)
+ }
+ }
+
+ override def userClass: Class[IExampleSubType] = classOf[IExampleSubType]
+
+ override def hashCode(): Int = classOf[ExampleSubTypeUDT].getName.hashCode()
+
+ override def equals(other: Any): Boolean = other.isInstanceOf[ExampleSubTypeUDT]
+
+ override def typeName: String = "exampleSubType"
+
+ private[spark] override def asNullable: ExampleSubTypeUDT = this
+}
+
+
class UserDefinedTypeSuite extends QueryTest with SharedSQLContext with ParquetTest {
+
--- End diff --
ok
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16660: [SPARK-19311][SQL] fix UDT hierarchy issue
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/16660
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/71852/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #16660: [SPARK-19311][SQL] fix UDT hierarchy issue
Posted by viirya <gi...@git.apache.org>.
Github user viirya commented on a diff in the pull request:
https://github.com/apache/spark/pull/16660#discussion_r97444441
--- Diff: sql/core/src/test/scala/org/apache/spark/sql/UserDefinedTypeSuite.scala ---
@@ -194,4 +293,35 @@ class UserDefinedTypeSuite extends QueryTest with SharedSQLContext with ParquetT
// call `collect` to make sure this query can pass analysis.
pointsRDD.as[MyLabeledPoint].map(_.copy(label = 2.0)).collect()
}
+
+ test("SPARK-19311: UDFs disregard UDT type hierarchy") {
+ UDTRegistration.register(classOf[IExampleBaseType].getName,
--- End diff --
With `SQLUserDefinedType`, no need to use `UDTRegistration`. We can remove this two lines.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #16660: [SPARK-19311][SQL] fix UDT hierarchy issue
Posted by viirya <gi...@git.apache.org>.
Github user viirya commented on a diff in the pull request:
https://github.com/apache/spark/pull/16660#discussion_r97518422
--- Diff: sql/core/src/test/scala/org/apache/spark/sql/UserDefinedTypeSuite.scala ---
@@ -194,4 +293,35 @@ class UserDefinedTypeSuite extends QueryTest with SharedSQLContext with ParquetT
// call `collect` to make sure this query can pass analysis.
pointsRDD.as[MyLabeledPoint].map(_.copy(label = 2.0)).collect()
}
+
+ test("SPARK-19311: UDFs disregard UDT type hierarchy") {
+ UDTRegistration.register(classOf[IExampleBaseType].getName,
--- End diff --
oh. if you worry about that, actually we have `UDTRegistrationSuite` for test case of `UDTRegistration`. i am fine to either `SQLUserDefinedType` or `UDTRegistration`.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16660: [SPARK-19311][SQL] fix UDT hierarchy issue
Posted by viirya <gi...@git.apache.org>.
Github user viirya commented on the issue:
https://github.com/apache/spark/pull/16660
LGTM too. @gmoehler Can you add a unit test?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #16660: [SPARK-19311][SQL] fix UDT hierarchy issue
Posted by gmoehler <gi...@git.apache.org>.
Github user gmoehler commented on a diff in the pull request:
https://github.com/apache/spark/pull/16660#discussion_r97534883
--- Diff: sql/core/src/test/scala/org/apache/spark/sql/UserDefinedTypeSuite.scala ---
@@ -194,4 +293,35 @@ class UserDefinedTypeSuite extends QueryTest with SharedSQLContext with ParquetT
// call `collect` to make sure this query can pass analysis.
pointsRDD.as[MyLabeledPoint].map(_.copy(label = 2.0)).collect()
}
+
+ test("SPARK-19311: UDFs disregard UDT type hierarchy") {
+ UDTRegistration.register(classOf[IExampleBaseType].getName,
--- End diff --
ok
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #16660: [SPARK-19311][SQL] fix UDT hierarchy issue
Posted by gmoehler <gi...@git.apache.org>.
Github user gmoehler commented on a diff in the pull request:
https://github.com/apache/spark/pull/16660#discussion_r97504006
--- Diff: sql/core/src/test/scala/org/apache/spark/sql/UserDefinedTypeSuite.scala ---
@@ -71,7 +73,104 @@ object UDT {
}
+// object and classes to test SPARK-19311
+
+// Trait/Interface for base type
+@SQLUserDefinedType(udt = classOf[ExampleBaseTypeUDT])
+sealed trait IExampleBaseType extends Serializable {
+ def field: Int
+}
+
+// Trait/Interface for derived type
+@SQLUserDefinedType(udt = classOf[ExampleSubTypeUDT])
+sealed trait IExampleSubType extends IExampleBaseType
+
+// a base class
+class ExampleBaseClass(override val field: Int) extends IExampleBaseType {
+ override def toString: String = field.toString
+
+}
+
+// a derived class
+class ExampleSubClass(override val field: Int)
+ extends ExampleBaseClass(field) with IExampleSubType
+
+// UDT for base class
+private[spark] class ExampleBaseTypeUDT extends UserDefinedType[IExampleBaseType] {
+
+ override def sqlType: StructType = {
+ StructType(Seq(
+ StructField("intfield", IntegerType, nullable = false)))
+ }
+
+ override def serialize(obj: IExampleBaseType): InternalRow = {
+ val row = new GenericInternalRow(1)
+ row.setInt(0, obj.field)
+ row
+ }
+
+ override def deserialize(datum: Any): IExampleBaseType = {
+ datum match {
+ case row: InternalRow =>
+ require(row.numFields == 1,
+ s"VectorUDT.deserialize given row with length " +
+ s"${row.numFields} but requires length == 1")
+ val field = row.getInt(0)
+ new ExampleBaseClass(field)
+ }
+ }
+
+ override def userClass: Class[IExampleBaseType] = classOf[IExampleBaseType]
+
+ override def hashCode(): Int = classOf[ExampleBaseTypeUDT].getName.hashCode()
+
+ override def equals(other: Any): Boolean = other.isInstanceOf[ExampleBaseTypeUDT]
+
+ override def typeName: String = "exampleBaseType"
+
+ private[spark] override def asNullable: ExampleBaseTypeUDT = this
+}
+
+// UDT for derived class
+private[spark] class ExampleSubTypeUDT extends UserDefinedType[IExampleSubType] {
+
+ override def sqlType: StructType = {
+ StructType(Seq(
+ StructField("intfield", IntegerType, nullable = false)))
+ }
+
+ override def serialize(obj: IExampleSubType): InternalRow = {
+
+ val row = new GenericInternalRow(1)
+ row.setInt(0, obj.field)
+ row
+ }
+
+ override def deserialize(datum: Any): IExampleSubType = {
+ datum match {
+ case row: InternalRow =>
+ require(row.numFields == 1,
+ s"VectorUDT.deserialize given row with length " +
+ s"${row.numFields} but requires length == 1")
+ val field = row.getInt(0)
+ new ExampleSubClass(field)
+ }
+ }
+
+ override def userClass: Class[IExampleSubType] = classOf[IExampleSubType]
+
+ override def hashCode(): Int = classOf[ExampleSubTypeUDT].getName.hashCode()
+
+ override def equals(other: Any): Boolean = other.isInstanceOf[ExampleSubTypeUDT]
--- End diff --
right - i have removed equals anyway due to a prev comment of @gatorsmile.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #16660: [SPARK-19311][SQL] fix UDT hierarchy issue
Posted by gmoehler <gi...@git.apache.org>.
Github user gmoehler commented on a diff in the pull request:
https://github.com/apache/spark/pull/16660#discussion_r97736380
--- Diff: sql/core/src/test/scala/org/apache/spark/sql/UserDefinedTypeSuite.scala ---
@@ -71,7 +73,104 @@ object UDT {
}
+// object and classes to test SPARK-19311
+
+// Trait/Interface for base type
+@SQLUserDefinedType(udt = classOf[ExampleBaseTypeUDT])
+sealed trait IExampleBaseType extends Serializable {
+ def field: Int
+}
+
+// Trait/Interface for derived type
+@SQLUserDefinedType(udt = classOf[ExampleSubTypeUDT])
+sealed trait IExampleSubType extends IExampleBaseType
+
+// a base class
+class ExampleBaseClass(override val field: Int) extends IExampleBaseType {
+ override def toString: String = field.toString
--- End diff --
ok
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16660: [SPARK-19311][SQL] fix UDT hierarchy issue
Posted by gatorsmile <gi...@git.apache.org>.
Github user gatorsmile commented on the issue:
https://github.com/apache/spark/pull/16660
LGTM except one comment. Thanks for working on this!
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #16660: [SPARK-19311][SQL] fix UDT hierarchy issue
Posted by viirya <gi...@git.apache.org>.
Github user viirya commented on a diff in the pull request:
https://github.com/apache/spark/pull/16660#discussion_r97545280
--- Diff: sql/core/src/test/scala/org/apache/spark/sql/UserDefinedTypeSuite.scala ---
@@ -71,7 +73,104 @@ object UDT {
}
+// object and classes to test SPARK-19311
+
+// Trait/Interface for base type
+@SQLUserDefinedType(udt = classOf[ExampleBaseTypeUDT])
+sealed trait IExampleBaseType extends Serializable {
+ def field: Int
+}
+
+// Trait/Interface for derived type
+@SQLUserDefinedType(udt = classOf[ExampleSubTypeUDT])
+sealed trait IExampleSubType extends IExampleBaseType
+
+// a base class
+class ExampleBaseClass(override val field: Int) extends IExampleBaseType {
+ override def toString: String = field.toString
--- End diff --
@gmoehler I think we don't need `toString`?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16660: [SPARK-19311][SQL] fix UDT hierarchy issue
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/16660
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16660: [SPARK-19311][SQL] fix UDT hierarchy issue
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/16660
**[Test build #71841 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/71841/testReport)** for PR 16660 at commit [`fad9f0e`](https://github.com/apache/spark/commit/fad9f0e1bc98ad5e7ee2c8dfc726ddc979204f11).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16660: [SPARK-19311][SQL] fix UDT hierarchy issue
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/16660
**[Test build #71846 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/71846/testReport)** for PR 16660 at commit [`fb261d7`](https://github.com/apache/spark/commit/fb261d7e329b0da5970fd7f6f7eaa2ee5064d89b).
* This patch **fails Scala style tests**.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16660: [SPARK-19311][SQL] fix UDT hierarchy issue
Posted by gatorsmile <gi...@git.apache.org>.
Github user gatorsmile commented on the issue:
https://github.com/apache/spark/pull/16660
The remaining comment is: https://github.com/apache/spark/pull/16660#discussion_r97427591
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16660: [SPARK-19311][SQL] fix UDT hierarchy issue
Posted by gmoehler <gi...@git.apache.org>.
Github user gmoehler commented on the issue:
https://github.com/apache/spark/pull/16660
Thanks for pointing out. I had overseen this comment.
Really appreciate you commitment to clean code!
Change is pushed and ready for a test build.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16660: [SPARK-19311][SQL] fix UDT hierarchy issue
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/16660
**[Test build #71980 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/71980/testReport)** for PR 16660 at commit [`6c16760`](https://github.com/apache/spark/commit/6c16760f0b2e5aba864366c03888666ba584d38a).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds the following public classes _(experimental)_:
* `class ExampleBaseClass(override val field: Int) extends IExampleBaseType`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16660: [SPARK-19311][SQL] fix UDT hierarchy issue
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/16660
**[Test build #71980 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/71980/testReport)** for PR 16660 at commit [`6c16760`](https://github.com/apache/spark/commit/6c16760f0b2e5aba864366c03888666ba584d38a).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #16660: [SPARK-19311][SQL] fix UDT hierarchy issue
Posted by gmoehler <gi...@git.apache.org>.
Github user gmoehler commented on a diff in the pull request:
https://github.com/apache/spark/pull/16660#discussion_r97512560
--- Diff: sql/core/src/test/scala/org/apache/spark/sql/UserDefinedTypeSuite.scala ---
@@ -194,4 +293,35 @@ class UserDefinedTypeSuite extends QueryTest with SharedSQLContext with ParquetT
// call `collect` to make sure this query can pass analysis.
pointsRDD.as[MyLabeledPoint].map(_.copy(label = 2.0)).collect()
}
+
+ test("SPARK-19311: UDFs disregard UDT type hierarchy") {
+ UDTRegistration.register(classOf[IExampleBaseType].getName,
--- End diff --
i tend to leave them, but remove the @SQLUserDefinedType, so we have a test that uses UDTRegistration
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #16660: [SPARK-19311][SQL] fix UDT hierarchy issue
Posted by gmoehler <gi...@git.apache.org>.
Github user gmoehler commented on a diff in the pull request:
https://github.com/apache/spark/pull/16660#discussion_r97503371
--- Diff: sql/core/src/test/scala/org/apache/spark/sql/UserDefinedTypeSuite.scala ---
@@ -71,7 +73,104 @@ object UDT {
}
+// object and classes to test SPARK-19311
+
+// Trait/Interface for base type
+@SQLUserDefinedType(udt = classOf[ExampleBaseTypeUDT])
+sealed trait IExampleBaseType extends Serializable {
+ def field: Int
+}
+
+// Trait/Interface for derived type
+@SQLUserDefinedType(udt = classOf[ExampleSubTypeUDT])
+sealed trait IExampleSubType extends IExampleBaseType
+
+// a base class
+class ExampleBaseClass(override val field: Int) extends IExampleBaseType {
+ override def toString: String = field.toString
+
+}
+
+// a derived class
+class ExampleSubClass(override val field: Int)
+ extends ExampleBaseClass(field) with IExampleSubType
+
+// UDT for base class
+private[spark] class ExampleBaseTypeUDT extends UserDefinedType[IExampleBaseType] {
+
+ override def sqlType: StructType = {
+ StructType(Seq(
+ StructField("intfield", IntegerType, nullable = false)))
+ }
+
+ override def serialize(obj: IExampleBaseType): InternalRow = {
+ val row = new GenericInternalRow(1)
+ row.setInt(0, obj.field)
+ row
+ }
+
+ override def deserialize(datum: Any): IExampleBaseType = {
+ datum match {
+ case row: InternalRow =>
+ require(row.numFields == 1,
+ s"VectorUDT.deserialize given row with length " +
+ s"${row.numFields} but requires length == 1")
+ val field = row.getInt(0)
+ new ExampleBaseClass(field)
+ }
+ }
+
+ override def userClass: Class[IExampleBaseType] = classOf[IExampleBaseType]
+
+ override def hashCode(): Int = classOf[ExampleBaseTypeUDT].getName.hashCode()
+
+ override def equals(other: Any): Boolean = other.isInstanceOf[ExampleBaseTypeUDT]
+
+ override def typeName: String = "exampleBaseType"
+
+ private[spark] override def asNullable: ExampleBaseTypeUDT = this
+}
+
+// UDT for derived class
+private[spark] class ExampleSubTypeUDT extends UserDefinedType[IExampleSubType] {
+
+ override def sqlType: StructType = {
+ StructType(Seq(
+ StructField("intfield", IntegerType, nullable = false)))
+ }
+
+ override def serialize(obj: IExampleSubType): InternalRow = {
+
+ val row = new GenericInternalRow(1)
+ row.setInt(0, obj.field)
+ row
+ }
+
+ override def deserialize(datum: Any): IExampleSubType = {
+ datum match {
+ case row: InternalRow =>
+ require(row.numFields == 1,
+ s"VectorUDT.deserialize given row with length " +
--- End diff --
ok - replaced it with a fixed string
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16660: [SPARK-19311][SQL] fix UDT hierarchy issue
Posted by gatorsmile <gi...@git.apache.org>.
Github user gatorsmile commented on the issue:
https://github.com/apache/spark/pull/16660
Thanks! Merging to master/2.1
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #16660: [SPARK-19311][SQL] fix UDT hierarchy issue
Posted by gatorsmile <gi...@git.apache.org>.
Github user gatorsmile commented on a diff in the pull request:
https://github.com/apache/spark/pull/16660#discussion_r97428227
--- Diff: sql/core/src/test/scala/org/apache/spark/sql/UserDefinedTypeSuite.scala ---
@@ -71,7 +73,104 @@ object UDT {
}
+// object and classes to test SPARK-19311
+
+// Trait/Interface for base type
+@SQLUserDefinedType(udt = classOf[ExampleBaseTypeUDT])
+sealed trait IExampleBaseType extends Serializable {
+ def field: Int
+}
+
+// Trait/Interface for derived type
+@SQLUserDefinedType(udt = classOf[ExampleSubTypeUDT])
+sealed trait IExampleSubType extends IExampleBaseType
+
+// a base class
+class ExampleBaseClass(override val field: Int) extends IExampleBaseType {
+ override def toString: String = field.toString
+
+}
+
+// a derived class
+class ExampleSubClass(override val field: Int)
+ extends ExampleBaseClass(field) with IExampleSubType
+
+// UDT for base class
+private[spark] class ExampleBaseTypeUDT extends UserDefinedType[IExampleBaseType] {
+
+ override def sqlType: StructType = {
+ StructType(Seq(
+ StructField("intfield", IntegerType, nullable = false)))
+ }
+
+ override def serialize(obj: IExampleBaseType): InternalRow = {
+ val row = new GenericInternalRow(1)
+ row.setInt(0, obj.field)
+ row
+ }
+
+ override def deserialize(datum: Any): IExampleBaseType = {
+ datum match {
+ case row: InternalRow =>
+ require(row.numFields == 1,
+ s"VectorUDT.deserialize given row with length " +
+ s"${row.numFields} but requires length == 1")
+ val field = row.getInt(0)
+ new ExampleBaseClass(field)
+ }
+ }
+
+ override def userClass: Class[IExampleBaseType] = classOf[IExampleBaseType]
+
+ override def hashCode(): Int = classOf[ExampleBaseTypeUDT].getName.hashCode()
+
+ override def equals(other: Any): Boolean = other.isInstanceOf[ExampleBaseTypeUDT]
+
+ override def typeName: String = "exampleBaseType"
+
+ private[spark] override def asNullable: ExampleBaseTypeUDT = this
+}
+
+// UDT for derived class
+private[spark] class ExampleSubTypeUDT extends UserDefinedType[IExampleSubType] {
+
+ override def sqlType: StructType = {
+ StructType(Seq(
+ StructField("intfield", IntegerType, nullable = false)))
+ }
+
+ override def serialize(obj: IExampleSubType): InternalRow = {
+
+ val row = new GenericInternalRow(1)
+ row.setInt(0, obj.field)
+ row
+ }
+
+ override def deserialize(datum: Any): IExampleSubType = {
+ datum match {
+ case row: InternalRow =>
+ require(row.numFields == 1,
+ s"VectorUDT.deserialize given row with length " +
+ s"${row.numFields} but requires length == 1")
+ val field = row.getInt(0)
+ new ExampleSubClass(field)
+ }
+ }
+
+ override def userClass: Class[IExampleSubType] = classOf[IExampleSubType]
+
+ override def hashCode(): Int = classOf[ExampleSubTypeUDT].getName.hashCode()
+
+ override def equals(other: Any): Boolean = other.isInstanceOf[ExampleSubTypeUDT]
+
+ override def typeName: String = "exampleSubType"
+
+ private[spark] override def asNullable: ExampleSubTypeUDT = this
--- End diff --
Nit: to simplify the test cases, please remove asNullable, typeName, equals and hashCode too.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #16660: [SPARK-19311][SQL] fix UDT hierarchy issue
Posted by gatorsmile <gi...@git.apache.org>.
Github user gatorsmile commented on a diff in the pull request:
https://github.com/apache/spark/pull/16660#discussion_r97427898
--- Diff: sql/core/src/test/scala/org/apache/spark/sql/UserDefinedTypeSuite.scala ---
@@ -71,7 +73,104 @@ object UDT {
}
+// object and classes to test SPARK-19311
+
+// Trait/Interface for base type
+@SQLUserDefinedType(udt = classOf[ExampleBaseTypeUDT])
+sealed trait IExampleBaseType extends Serializable {
+ def field: Int
+}
+
+// Trait/Interface for derived type
+@SQLUserDefinedType(udt = classOf[ExampleSubTypeUDT])
+sealed trait IExampleSubType extends IExampleBaseType
+
+// a base class
+class ExampleBaseClass(override val field: Int) extends IExampleBaseType {
+ override def toString: String = field.toString
+
+}
+
+// a derived class
+class ExampleSubClass(override val field: Int)
+ extends ExampleBaseClass(field) with IExampleSubType
+
+// UDT for base class
+private[spark] class ExampleBaseTypeUDT extends UserDefinedType[IExampleBaseType] {
+
+ override def sqlType: StructType = {
+ StructType(Seq(
+ StructField("intfield", IntegerType, nullable = false)))
+ }
+
+ override def serialize(obj: IExampleBaseType): InternalRow = {
+ val row = new GenericInternalRow(1)
+ row.setInt(0, obj.field)
+ row
+ }
+
+ override def deserialize(datum: Any): IExampleBaseType = {
+ datum match {
+ case row: InternalRow =>
+ require(row.numFields == 1,
+ s"VectorUDT.deserialize given row with length " +
+ s"${row.numFields} but requires length == 1")
+ val field = row.getInt(0)
+ new ExampleBaseClass(field)
+ }
+ }
+
+ override def userClass: Class[IExampleBaseType] = classOf[IExampleBaseType]
+
+ override def hashCode(): Int = classOf[ExampleBaseTypeUDT].getName.hashCode()
+
+ override def equals(other: Any): Boolean = other.isInstanceOf[ExampleBaseTypeUDT]
+
+ override def typeName: String = "exampleBaseType"
+
+ private[spark] override def asNullable: ExampleBaseTypeUDT = this
+}
+
+// UDT for derived class
+private[spark] class ExampleSubTypeUDT extends UserDefinedType[IExampleSubType] {
+
+ override def sqlType: StructType = {
+ StructType(Seq(
+ StructField("intfield", IntegerType, nullable = false)))
+ }
+
+ override def serialize(obj: IExampleSubType): InternalRow = {
+
--- End diff --
Nit: Please remove this line. Thanks!
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16660: [SPARK-19311][SQL] fix UDT hierarchy issue
Posted by gatorsmile <gi...@git.apache.org>.
Github user gatorsmile commented on the issue:
https://github.com/apache/spark/pull/16660
It looks great! My comment is just to simplify the unit test cases. Normally, we want to make the unit test cases as simple as possible. Thanks!
LGTM except a few minor comments.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #16660: [SPARK-19311][SQL] fix UDT hierarchy issue
Posted by gatorsmile <gi...@git.apache.org>.
Github user gatorsmile commented on a diff in the pull request:
https://github.com/apache/spark/pull/16660#discussion_r97428272
--- Diff: sql/core/src/test/scala/org/apache/spark/sql/UserDefinedTypeSuite.scala ---
@@ -71,7 +73,104 @@ object UDT {
}
+// object and classes to test SPARK-19311
+
+// Trait/Interface for base type
+@SQLUserDefinedType(udt = classOf[ExampleBaseTypeUDT])
+sealed trait IExampleBaseType extends Serializable {
+ def field: Int
+}
+
+// Trait/Interface for derived type
+@SQLUserDefinedType(udt = classOf[ExampleSubTypeUDT])
+sealed trait IExampleSubType extends IExampleBaseType
+
+// a base class
+class ExampleBaseClass(override val field: Int) extends IExampleBaseType {
+ override def toString: String = field.toString
+
+}
+
+// a derived class
+class ExampleSubClass(override val field: Int)
+ extends ExampleBaseClass(field) with IExampleSubType
+
+// UDT for base class
+private[spark] class ExampleBaseTypeUDT extends UserDefinedType[IExampleBaseType] {
+
+ override def sqlType: StructType = {
+ StructType(Seq(
+ StructField("intfield", IntegerType, nullable = false)))
+ }
+
+ override def serialize(obj: IExampleBaseType): InternalRow = {
+ val row = new GenericInternalRow(1)
+ row.setInt(0, obj.field)
+ row
+ }
+
+ override def deserialize(datum: Any): IExampleBaseType = {
+ datum match {
+ case row: InternalRow =>
+ require(row.numFields == 1,
+ s"VectorUDT.deserialize given row with length " +
+ s"${row.numFields} but requires length == 1")
+ val field = row.getInt(0)
+ new ExampleBaseClass(field)
+ }
+ }
+
+ override def userClass: Class[IExampleBaseType] = classOf[IExampleBaseType]
+
+ override def hashCode(): Int = classOf[ExampleBaseTypeUDT].getName.hashCode()
+
+ override def equals(other: Any): Boolean = other.isInstanceOf[ExampleBaseTypeUDT]
+
+ override def typeName: String = "exampleBaseType"
+
+ private[spark] override def asNullable: ExampleBaseTypeUDT = this
+}
+
+// UDT for derived class
+private[spark] class ExampleSubTypeUDT extends UserDefinedType[IExampleSubType] {
+
+ override def sqlType: StructType = {
+ StructType(Seq(
+ StructField("intfield", IntegerType, nullable = false)))
+ }
+
+ override def serialize(obj: IExampleSubType): InternalRow = {
+
+ val row = new GenericInternalRow(1)
+ row.setInt(0, obj.field)
+ row
+ }
+
+ override def deserialize(datum: Any): IExampleSubType = {
+ datum match {
+ case row: InternalRow =>
+ require(row.numFields == 1,
+ s"VectorUDT.deserialize given row with length " +
+ s"${row.numFields} but requires length == 1")
+ val field = row.getInt(0)
+ new ExampleSubClass(field)
+ }
+ }
+
+ override def userClass: Class[IExampleSubType] = classOf[IExampleSubType]
+
+ override def hashCode(): Int = classOf[ExampleSubTypeUDT].getName.hashCode()
+
+ override def equals(other: Any): Boolean = other.isInstanceOf[ExampleSubTypeUDT]
+
+ override def typeName: String = "exampleSubType"
+
+ private[spark] override def asNullable: ExampleSubTypeUDT = this
+}
+
+
class UserDefinedTypeSuite extends QueryTest with SharedSQLContext with ParquetTest {
+
--- End diff --
Nit: Please remove this empty line.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16660: [SPARK-19311][SQL] fix UDT hierarchy issue
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/16660
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #16660: [SPARK-19311][SQL] fix UDT hierarchy issue
Posted by gmoehler <gi...@git.apache.org>.
Github user gmoehler commented on a diff in the pull request:
https://github.com/apache/spark/pull/16660#discussion_r97498043
--- Diff: sql/core/src/test/scala/org/apache/spark/sql/UserDefinedTypeSuite.scala ---
@@ -20,13 +20,15 @@ package org.apache.spark.sql
import scala.beans.{BeanInfo, BeanProperty}
import org.apache.spark.rdd.RDD
-import org.apache.spark.sql.catalyst.CatalystTypeConverters
+import org.apache.spark.sql.catalyst.{CatalystTypeConverters, InternalRow}
+import org.apache.spark.sql.catalyst.expressions.GenericInternalRow
import org.apache.spark.sql.catalyst.util.{ArrayData, GenericArrayData}
import org.apache.spark.sql.execution.datasources.parquet.ParquetTest
import org.apache.spark.sql.functions._
import org.apache.spark.sql.test.SharedSQLContext
import org.apache.spark.sql.types._
+
--- End diff --
ok
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16660: [SPARK-19311][SQL] fix UDT hierarchy issue
Posted by gatorsmile <gi...@git.apache.org>.
Github user gatorsmile commented on the issue:
https://github.com/apache/spark/pull/16660
LGTM
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16660: [SPARK-19311][SQL] fix UDT hierarchy issue
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/16660
**[Test build #71849 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/71849/testReport)** for PR 16660 at commit [`6b6b773`](https://github.com/apache/spark/commit/6b6b773a4df1ee017f00b9ca43a76a429ad418de).
* This patch **fails Scala style tests**.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16660: [SPARK-19311][SQL] fix UDT hierarchy issue
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/16660
Can one of the admins verify this patch?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #16660: [SPARK-19311][SQL] fix UDT hierarchy issue
Posted by cloud-fan <gi...@git.apache.org>.
Github user cloud-fan commented on a diff in the pull request:
https://github.com/apache/spark/pull/16660#discussion_r97347604
--- Diff: sql/core/src/test/scala/org/apache/spark/sql/UserDefinedTypeSuite.scala ---
@@ -71,7 +73,104 @@ object UDT {
}
+// object and classes to test SPARK-19311
+
+// Trait/Interface for base type
+@SQLUserDefinedType(udt = classOf[ExampleBaseTypeUDT])
+sealed trait IExampleBaseType extends Serializable {
+ def field: Int
+}
+
+// Trait/Interface for derived type
+@SQLUserDefinedType(udt = classOf[ExampleSubTypeUDT])
+sealed trait IExampleSubType extends IExampleBaseType
+
+// a base class
+class ExampleBaseClass(override val field: Int) extends IExampleBaseType {
+ override def toString: String = field.toString
--- End diff --
why override `toString` here?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #16660: [SPARK-19311][SQL] fix UDT hierarchy issue
Posted by gatorsmile <gi...@git.apache.org>.
Github user gatorsmile commented on a diff in the pull request:
https://github.com/apache/spark/pull/16660#discussion_r97427641
--- Diff: sql/core/src/test/scala/org/apache/spark/sql/UserDefinedTypeSuite.scala ---
@@ -71,7 +73,104 @@ object UDT {
}
+// object and classes to test SPARK-19311
+
+// Trait/Interface for base type
+@SQLUserDefinedType(udt = classOf[ExampleBaseTypeUDT])
+sealed trait IExampleBaseType extends Serializable {
+ def field: Int
+}
+
+// Trait/Interface for derived type
+@SQLUserDefinedType(udt = classOf[ExampleSubTypeUDT])
+sealed trait IExampleSubType extends IExampleBaseType
+
+// a base class
+class ExampleBaseClass(override val field: Int) extends IExampleBaseType {
+ override def toString: String = field.toString
+
+}
+
+// a derived class
+class ExampleSubClass(override val field: Int)
+ extends ExampleBaseClass(field) with IExampleSubType
+
+// UDT for base class
+private[spark] class ExampleBaseTypeUDT extends UserDefinedType[IExampleBaseType] {
--- End diff --
`private[spark] ` is not needed here
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16660: [SPARK-19311][SQL] fix UDT hierarchy issue
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/16660
**[Test build #71852 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/71852/testReport)** for PR 16660 at commit [`4ae0555`](https://github.com/apache/spark/commit/4ae05559d9a1b343e61e87ad0e4841c497c13382).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16660: [SPARK-19311][SQL] fix UDT hierarchy issue
Posted by gmoehler <gi...@git.apache.org>.
Github user gmoehler commented on the issue:
https://github.com/apache/spark/pull/16660
Thanks for the valuable (and fast!) comments - i have worked them in.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16660: [SPARK-19311][SQL] fix UDT hierarchy issue
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/16660
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16660: [SPARK-19311][SQL] fix UDT hierarchy issue
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/16660
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/71980/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16660: [SPARK-19311][SQL] fix UDT hierarchy issue
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/16660
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16660: [SPARK-19311][SQL] fix UDT hierarchy issue
Posted by gmoehler <gi...@git.apache.org>.
Github user gmoehler commented on the issue:
https://github.com/apache/spark/pull/16660
@viirya Which comment are you referring to? I thought i had included all of them ;-)
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16660: [SPARK-19311][SQL] fix UDT hierarchy issue
Posted by viirya <gi...@git.apache.org>.
Github user viirya commented on the issue:
https://github.com/apache/spark/pull/16660
LGTM except one minor comment.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #16660: [SPARK-19311][SQL] fix UDT hierarchy issue
Posted by viirya <gi...@git.apache.org>.
Github user viirya commented on a diff in the pull request:
https://github.com/apache/spark/pull/16660#discussion_r97444783
--- Diff: sql/core/src/test/scala/org/apache/spark/sql/UserDefinedTypeSuite.scala ---
@@ -71,7 +73,104 @@ object UDT {
}
+// object and classes to test SPARK-19311
+
+// Trait/Interface for base type
+@SQLUserDefinedType(udt = classOf[ExampleBaseTypeUDT])
+sealed trait IExampleBaseType extends Serializable {
+ def field: Int
+}
+
+// Trait/Interface for derived type
+@SQLUserDefinedType(udt = classOf[ExampleSubTypeUDT])
+sealed trait IExampleSubType extends IExampleBaseType
+
+// a base class
+class ExampleBaseClass(override val field: Int) extends IExampleBaseType {
+ override def toString: String = field.toString
+
+}
+
+// a derived class
+class ExampleSubClass(override val field: Int)
+ extends ExampleBaseClass(field) with IExampleSubType
+
+// UDT for base class
+private[spark] class ExampleBaseTypeUDT extends UserDefinedType[IExampleBaseType] {
+
+ override def sqlType: StructType = {
+ StructType(Seq(
+ StructField("intfield", IntegerType, nullable = false)))
+ }
+
+ override def serialize(obj: IExampleBaseType): InternalRow = {
+ val row = new GenericInternalRow(1)
+ row.setInt(0, obj.field)
+ row
+ }
+
+ override def deserialize(datum: Any): IExampleBaseType = {
+ datum match {
+ case row: InternalRow =>
+ require(row.numFields == 1,
+ s"VectorUDT.deserialize given row with length " +
+ s"${row.numFields} but requires length == 1")
+ val field = row.getInt(0)
+ new ExampleBaseClass(field)
+ }
+ }
+
+ override def userClass: Class[IExampleBaseType] = classOf[IExampleBaseType]
+
+ override def hashCode(): Int = classOf[ExampleBaseTypeUDT].getName.hashCode()
+
+ override def equals(other: Any): Boolean = other.isInstanceOf[ExampleBaseTypeUDT]
+
+ override def typeName: String = "exampleBaseType"
+
+ private[spark] override def asNullable: ExampleBaseTypeUDT = this
+}
+
+// UDT for derived class
+private[spark] class ExampleSubTypeUDT extends UserDefinedType[IExampleSubType] {
+
+ override def sqlType: StructType = {
+ StructType(Seq(
+ StructField("intfield", IntegerType, nullable = false)))
+ }
+
+ override def serialize(obj: IExampleSubType): InternalRow = {
+
+ val row = new GenericInternalRow(1)
+ row.setInt(0, obj.field)
+ row
+ }
+
+ override def deserialize(datum: Any): IExampleSubType = {
+ datum match {
+ case row: InternalRow =>
+ require(row.numFields == 1,
+ s"VectorUDT.deserialize given row with length " +
+ s"${row.numFields} but requires length == 1")
+ val field = row.getInt(0)
+ new ExampleSubClass(field)
+ }
+ }
+
+ override def userClass: Class[IExampleSubType] = classOf[IExampleSubType]
+
+ override def hashCode(): Int = classOf[ExampleSubTypeUDT].getName.hashCode()
+
+ override def equals(other: Any): Boolean = other.isInstanceOf[ExampleSubTypeUDT]
--- End diff --
`equals` in `UserDefinedType` has default impl. to call `acceptsType`.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #16660: [SPARK-19311][SQL] fix UDT hierarchy issue
Posted by gmoehler <gi...@git.apache.org>.
Github user gmoehler commented on a diff in the pull request:
https://github.com/apache/spark/pull/16660#discussion_r97500817
--- Diff: sql/core/src/test/scala/org/apache/spark/sql/UserDefinedTypeSuite.scala ---
@@ -71,7 +73,104 @@ object UDT {
}
+// object and classes to test SPARK-19311
+
+// Trait/Interface for base type
+@SQLUserDefinedType(udt = classOf[ExampleBaseTypeUDT])
+sealed trait IExampleBaseType extends Serializable {
+ def field: Int
+}
+
+// Trait/Interface for derived type
+@SQLUserDefinedType(udt = classOf[ExampleSubTypeUDT])
+sealed trait IExampleSubType extends IExampleBaseType
+
+// a base class
+class ExampleBaseClass(override val field: Int) extends IExampleBaseType {
+ override def toString: String = field.toString
+
+}
+
+// a derived class
+class ExampleSubClass(override val field: Int)
+ extends ExampleBaseClass(field) with IExampleSubType
+
+// UDT for base class
+private[spark] class ExampleBaseTypeUDT extends UserDefinedType[IExampleBaseType] {
+
+ override def sqlType: StructType = {
+ StructType(Seq(
+ StructField("intfield", IntegerType, nullable = false)))
+ }
+
+ override def serialize(obj: IExampleBaseType): InternalRow = {
+ val row = new GenericInternalRow(1)
+ row.setInt(0, obj.field)
+ row
+ }
+
+ override def deserialize(datum: Any): IExampleBaseType = {
+ datum match {
+ case row: InternalRow =>
+ require(row.numFields == 1,
+ s"VectorUDT.deserialize given row with length " +
+ s"${row.numFields} but requires length == 1")
+ val field = row.getInt(0)
+ new ExampleBaseClass(field)
+ }
+ }
+
+ override def userClass: Class[IExampleBaseType] = classOf[IExampleBaseType]
+
+ override def hashCode(): Int = classOf[ExampleBaseTypeUDT].getName.hashCode()
+
+ override def equals(other: Any): Boolean = other.isInstanceOf[ExampleBaseTypeUDT]
+
+ override def typeName: String = "exampleBaseType"
+
+ private[spark] override def asNullable: ExampleBaseTypeUDT = this
+}
+
+// UDT for derived class
+private[spark] class ExampleSubTypeUDT extends UserDefinedType[IExampleSubType] {
+
+ override def sqlType: StructType = {
+ StructType(Seq(
+ StructField("intfield", IntegerType, nullable = false)))
+ }
+
+ override def serialize(obj: IExampleSubType): InternalRow = {
+
--- End diff --
not sure which line you mean... we need to overwrite serialize in any case I guess - or is it about a blank line?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16660: [SPARK-19311][SQL] fix UDT hierarchy issue
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/16660
**[Test build #71929 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/71929/testReport)** for PR 16660 at commit [`7aed9a4`](https://github.com/apache/spark/commit/7aed9a4fada263785ce1d81acb31073ef7a401fd).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds the following public classes _(experimental)_:
* `class ExampleBaseTypeUDT extends UserDefinedType[IExampleBaseType] `
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #16660: [SPARK-19311][SQL] fix UDT hierarchy issue
Posted by gatorsmile <gi...@git.apache.org>.
Github user gatorsmile commented on a diff in the pull request:
https://github.com/apache/spark/pull/16660#discussion_r97428147
--- Diff: sql/core/src/test/scala/org/apache/spark/sql/UserDefinedTypeSuite.scala ---
@@ -71,7 +73,104 @@ object UDT {
}
+// object and classes to test SPARK-19311
+
+// Trait/Interface for base type
+@SQLUserDefinedType(udt = classOf[ExampleBaseTypeUDT])
+sealed trait IExampleBaseType extends Serializable {
+ def field: Int
+}
+
+// Trait/Interface for derived type
+@SQLUserDefinedType(udt = classOf[ExampleSubTypeUDT])
+sealed trait IExampleSubType extends IExampleBaseType
+
+// a base class
+class ExampleBaseClass(override val field: Int) extends IExampleBaseType {
+ override def toString: String = field.toString
+
+}
+
+// a derived class
+class ExampleSubClass(override val field: Int)
+ extends ExampleBaseClass(field) with IExampleSubType
+
+// UDT for base class
+private[spark] class ExampleBaseTypeUDT extends UserDefinedType[IExampleBaseType] {
+
+ override def sqlType: StructType = {
+ StructType(Seq(
+ StructField("intfield", IntegerType, nullable = false)))
+ }
+
+ override def serialize(obj: IExampleBaseType): InternalRow = {
+ val row = new GenericInternalRow(1)
+ row.setInt(0, obj.field)
+ row
+ }
+
+ override def deserialize(datum: Any): IExampleBaseType = {
+ datum match {
+ case row: InternalRow =>
+ require(row.numFields == 1,
+ s"VectorUDT.deserialize given row with length " +
+ s"${row.numFields} but requires length == 1")
+ val field = row.getInt(0)
+ new ExampleBaseClass(field)
+ }
+ }
+
+ override def userClass: Class[IExampleBaseType] = classOf[IExampleBaseType]
+
+ override def hashCode(): Int = classOf[ExampleBaseTypeUDT].getName.hashCode()
+
+ override def equals(other: Any): Boolean = other.isInstanceOf[ExampleBaseTypeUDT]
+
+ override def typeName: String = "exampleBaseType"
+
+ private[spark] override def asNullable: ExampleBaseTypeUDT = this
+}
+
+// UDT for derived class
+private[spark] class ExampleSubTypeUDT extends UserDefinedType[IExampleSubType] {
+
+ override def sqlType: StructType = {
+ StructType(Seq(
+ StructField("intfield", IntegerType, nullable = false)))
+ }
+
+ override def serialize(obj: IExampleSubType): InternalRow = {
+
+ val row = new GenericInternalRow(1)
+ row.setInt(0, obj.field)
+ row
+ }
+
+ override def deserialize(datum: Any): IExampleSubType = {
+ datum match {
+ case row: InternalRow =>
+ require(row.numFields == 1,
+ s"VectorUDT.deserialize given row with length " +
--- End diff --
Please remove this unneeded `s` String Interpolator. Thanks!
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16660: [SPARK-19311][SQL] fix UDT hierarchy issue
Posted by cloud-fan <gi...@git.apache.org>.
Github user cloud-fan commented on the issue:
https://github.com/apache/spark/pull/16660
is it possible to add a unit test? the change LGTM
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16660: [SPARK-19311][SQL] fix UDT hierarchy issue
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/16660
**[Test build #71929 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/71929/testReport)** for PR 16660 at commit [`7aed9a4`](https://github.com/apache/spark/commit/7aed9a4fada263785ce1d81acb31073ef7a401fd).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16660: [SPARK-19311][SQL] fix UDT hierarchy issue
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/16660
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16660: [SPARK-19311][SQL] fix UDT hierarchy issue
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/16660
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/71755/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #16660: [SPARK-19311][SQL] fix UDT hierarchy issue
Posted by gmoehler <gi...@git.apache.org>.
Github user gmoehler commented on a diff in the pull request:
https://github.com/apache/spark/pull/16660#discussion_r97503434
--- Diff: sql/core/src/test/scala/org/apache/spark/sql/UserDefinedTypeSuite.scala ---
@@ -71,7 +73,104 @@ object UDT {
}
+// object and classes to test SPARK-19311
+
+// Trait/Interface for base type
+@SQLUserDefinedType(udt = classOf[ExampleBaseTypeUDT])
+sealed trait IExampleBaseType extends Serializable {
+ def field: Int
+}
+
+// Trait/Interface for derived type
+@SQLUserDefinedType(udt = classOf[ExampleSubTypeUDT])
+sealed trait IExampleSubType extends IExampleBaseType
+
+// a base class
+class ExampleBaseClass(override val field: Int) extends IExampleBaseType {
+ override def toString: String = field.toString
+
+}
+
+// a derived class
+class ExampleSubClass(override val field: Int)
+ extends ExampleBaseClass(field) with IExampleSubType
+
+// UDT for base class
+private[spark] class ExampleBaseTypeUDT extends UserDefinedType[IExampleBaseType] {
+
+ override def sqlType: StructType = {
+ StructType(Seq(
+ StructField("intfield", IntegerType, nullable = false)))
+ }
+
+ override def serialize(obj: IExampleBaseType): InternalRow = {
+ val row = new GenericInternalRow(1)
+ row.setInt(0, obj.field)
+ row
+ }
+
+ override def deserialize(datum: Any): IExampleBaseType = {
+ datum match {
+ case row: InternalRow =>
+ require(row.numFields == 1,
+ s"VectorUDT.deserialize given row with length " +
+ s"${row.numFields} but requires length == 1")
+ val field = row.getInt(0)
+ new ExampleBaseClass(field)
+ }
+ }
+
+ override def userClass: Class[IExampleBaseType] = classOf[IExampleBaseType]
+
+ override def hashCode(): Int = classOf[ExampleBaseTypeUDT].getName.hashCode()
+
+ override def equals(other: Any): Boolean = other.isInstanceOf[ExampleBaseTypeUDT]
+
+ override def typeName: String = "exampleBaseType"
+
+ private[spark] override def asNullable: ExampleBaseTypeUDT = this
+}
+
+// UDT for derived class
+private[spark] class ExampleSubTypeUDT extends UserDefinedType[IExampleSubType] {
+
+ override def sqlType: StructType = {
+ StructType(Seq(
+ StructField("intfield", IntegerType, nullable = false)))
+ }
+
+ override def serialize(obj: IExampleSubType): InternalRow = {
+
+ val row = new GenericInternalRow(1)
+ row.setInt(0, obj.field)
+ row
+ }
+
+ override def deserialize(datum: Any): IExampleSubType = {
+ datum match {
+ case row: InternalRow =>
+ require(row.numFields == 1,
+ s"VectorUDT.deserialize given row with length " +
+ s"${row.numFields} but requires length == 1")
+ val field = row.getInt(0)
+ new ExampleSubClass(field)
+ }
+ }
+
+ override def userClass: Class[IExampleSubType] = classOf[IExampleSubType]
+
+ override def hashCode(): Int = classOf[ExampleSubTypeUDT].getName.hashCode()
+
+ override def equals(other: Any): Boolean = other.isInstanceOf[ExampleSubTypeUDT]
+
+ override def typeName: String = "exampleSubType"
+
+ private[spark] override def asNullable: ExampleSubTypeUDT = this
--- End diff --
done
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16660: [SPARK-19311][SQL] fix UDT hierarchy issue
Posted by viirya <gi...@git.apache.org>.
Github user viirya commented on the issue:
https://github.com/apache/spark/pull/16660
@gmoehler `ExampleBaseClass` overrides `toString` impl. That is redundant.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #16660: [SPARK-19311][SQL] fix UDT hierarchy issue
Posted by viirya <gi...@git.apache.org>.
Github user viirya commented on a diff in the pull request:
https://github.com/apache/spark/pull/16660#discussion_r97326454
--- Diff: sql/core/src/test/scala/org/apache/spark/sql/UserDefinedTypeSuite.scala ---
@@ -71,6 +71,94 @@ object UDT {
}
+// object and classes to test SPARK-19311
+
+ // Trait/Interface for base type
--- End diff --
Please fix the indentation.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16660: [SPARK-19311][SQL] fix UDT hierarchy issue
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/16660
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/71929/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #16660: [SPARK-19311][SQL] fix UDT hierarchy issue
Posted by gatorsmile <gi...@git.apache.org>.
Github user gatorsmile commented on a diff in the pull request:
https://github.com/apache/spark/pull/16660#discussion_r97427591
--- Diff: sql/core/src/test/scala/org/apache/spark/sql/UserDefinedTypeSuite.scala ---
@@ -71,7 +73,104 @@ object UDT {
}
+// object and classes to test SPARK-19311
+
+// Trait/Interface for base type
+@SQLUserDefinedType(udt = classOf[ExampleBaseTypeUDT])
+sealed trait IExampleBaseType extends Serializable {
+ def field: Int
+}
+
+// Trait/Interface for derived type
+@SQLUserDefinedType(udt = classOf[ExampleSubTypeUDT])
+sealed trait IExampleSubType extends IExampleBaseType
+
+// a base class
+class ExampleBaseClass(override val field: Int) extends IExampleBaseType {
+ override def toString: String = field.toString
--- End diff --
Please simplify it to
```
class ExampleBaseClass(override val field: Int) extends IExampleBaseType
```
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16660: [SPARK-19311][SQL] fix UDT hierarchy issue
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/16660
**[Test build #71755 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/71755/testReport)** for PR 16660 at commit [`7ea9aa6`](https://github.com/apache/spark/commit/7ea9aa636f430a30b8d83ed2dda954fd06347d79).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #16660: [SPARK-19311][SQL] fix UDT hierarchy issue
Posted by gatorsmile <gi...@git.apache.org>.
Github user gatorsmile commented on a diff in the pull request:
https://github.com/apache/spark/pull/16660#discussion_r97427855
--- Diff: sql/core/src/test/scala/org/apache/spark/sql/UserDefinedTypeSuite.scala ---
@@ -71,7 +73,104 @@ object UDT {
}
+// object and classes to test SPARK-19311
+
+// Trait/Interface for base type
+@SQLUserDefinedType(udt = classOf[ExampleBaseTypeUDT])
+sealed trait IExampleBaseType extends Serializable {
+ def field: Int
+}
+
+// Trait/Interface for derived type
+@SQLUserDefinedType(udt = classOf[ExampleSubTypeUDT])
+sealed trait IExampleSubType extends IExampleBaseType
+
+// a base class
+class ExampleBaseClass(override val field: Int) extends IExampleBaseType {
+ override def toString: String = field.toString
+
+}
+
+// a derived class
+class ExampleSubClass(override val field: Int)
+ extends ExampleBaseClass(field) with IExampleSubType
+
+// UDT for base class
+private[spark] class ExampleBaseTypeUDT extends UserDefinedType[IExampleBaseType] {
+
+ override def sqlType: StructType = {
+ StructType(Seq(
+ StructField("intfield", IntegerType, nullable = false)))
+ }
+
+ override def serialize(obj: IExampleBaseType): InternalRow = {
+ val row = new GenericInternalRow(1)
+ row.setInt(0, obj.field)
+ row
+ }
+
+ override def deserialize(datum: Any): IExampleBaseType = {
+ datum match {
+ case row: InternalRow =>
+ require(row.numFields == 1,
+ s"VectorUDT.deserialize given row with length " +
+ s"${row.numFields} but requires length == 1")
+ val field = row.getInt(0)
+ new ExampleBaseClass(field)
+ }
+ }
+
+ override def userClass: Class[IExampleBaseType] = classOf[IExampleBaseType]
+
+ override def hashCode(): Int = classOf[ExampleBaseTypeUDT].getName.hashCode()
+
+ override def equals(other: Any): Boolean = other.isInstanceOf[ExampleBaseTypeUDT]
+
+ override def typeName: String = "exampleBaseType"
+
+ private[spark] override def asNullable: ExampleBaseTypeUDT = this
--- End diff --
Nit: to simplify the test cases, please remove `asNullable `, `typeName `, `equals` and `hashCode`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16660: [SPARK-19311][SQL] fix UDT hierarchy issue
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/16660
**[Test build #71849 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/71849/testReport)** for PR 16660 at commit [`6b6b773`](https://github.com/apache/spark/commit/6b6b773a4df1ee017f00b9ca43a76a429ad418de).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16660: [SPARK-19311][SQL] fix UDT hierarchy issue
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/16660
**[Test build #71846 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/71846/testReport)** for PR 16660 at commit [`fb261d7`](https://github.com/apache/spark/commit/fb261d7e329b0da5970fd7f6f7eaa2ee5064d89b).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #16660: [SPARK-19311][SQL] fix UDT hierarchy issue
Posted by gmoehler <gi...@git.apache.org>.
Github user gmoehler commented on a diff in the pull request:
https://github.com/apache/spark/pull/16660#discussion_r97498057
--- Diff: sql/core/src/test/scala/org/apache/spark/sql/UserDefinedTypeSuite.scala ---
@@ -71,7 +73,104 @@ object UDT {
}
+// object and classes to test SPARK-19311
+
+// Trait/Interface for base type
+@SQLUserDefinedType(udt = classOf[ExampleBaseTypeUDT])
+sealed trait IExampleBaseType extends Serializable {
+ def field: Int
+}
+
+// Trait/Interface for derived type
+@SQLUserDefinedType(udt = classOf[ExampleSubTypeUDT])
+sealed trait IExampleSubType extends IExampleBaseType
+
+// a base class
+class ExampleBaseClass(override val field: Int) extends IExampleBaseType {
+ override def toString: String = field.toString
+
+}
+
+// a derived class
+class ExampleSubClass(override val field: Int)
+ extends ExampleBaseClass(field) with IExampleSubType
+
+// UDT for base class
+private[spark] class ExampleBaseTypeUDT extends UserDefinedType[IExampleBaseType] {
--- End diff --
ok
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org