You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by maropu <gi...@git.apache.org> on 2018/10/15 10:47:18 UTC
[GitHub] spark pull request #22724: [SPARK-25734][SQL}
GitHub user maropu opened a pull request:
https://github.com/apache/spark/pull/22724
[SPARK-25734][SQL}
## What changes were proposed in this pull request?
(Please fill in changes proposed in this fix)
## How was this patch tested?
(Please explain how this patch was tested. E.g. unit tests, integration tests, manual tests)
(If this patch involves UI changes, please attach a screenshot; otherwise, remove this)
Please review http://spark.apache.org/contributing.html before opening a pull request.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/maropu/spark SPARK-25734
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/22724.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #22724
----
commit 9115d26642496b1ca1fdf0d310d47ee0ebeb3222
Author: Takeshi Yamamuro <ya...@...>
Date: 2018-10-15T10:43:19Z
Fix
----
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22724: [SPARK-25734][SQL] Literal should have a value correspon...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22724
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/97439/
Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #22724: [SPARK-25734][SQL] Literal should have a value co...
Posted by maropu <gi...@git.apache.org>.
Github user maropu commented on a diff in the pull request:
https://github.com/apache/spark/pull/22724#discussion_r225384590
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/literals.scala ---
@@ -196,6 +197,48 @@ object Literal {
case other =>
throw new RuntimeException(s"no default for type $dataType")
}
+
+ private[expressions] def validateLiteralValue(value: Any, dataType: DataType): Unit = {
+ def doValidate(v: Any, dataType: DataType): Boolean = dataType match {
+ case BooleanType => v.isInstanceOf[Boolean]
+ case ByteType => v.isInstanceOf[Byte]
+ case ShortType => v.isInstanceOf[Short]
+ case IntegerType | DateType => v.isInstanceOf[Int]
+ case LongType | TimestampType => v.isInstanceOf[Long]
+ case FloatType => v.isInstanceOf[Float]
+ case DoubleType => v.isInstanceOf[Double]
+ case _: DecimalType => v.isInstanceOf[Decimal]
+ case CalendarIntervalType => v.isInstanceOf[CalendarInterval]
+ case BinaryType => v.isInstanceOf[Array[Byte]]
+ case StringType => v.isInstanceOf[UTF8String]
+ case st: StructType =>
+ v.isInstanceOf[GenericInternalRow] && {
--- End diff --
ah, yes. ok, I'll fix.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22724: [SPARK-25734][SQL] Literal should have a value correspon...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22724
Merged build finished. Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22724: [SPARK-25734][SQL] Literal should have a value correspon...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/22724
**[Test build #97457 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/97457/testReport)** for PR 22724 at commit [`04a535d`](https://github.com/apache/spark/commit/04a535d330449b80a81b2f329283236b441bff00).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22724: [SPARK-25734][SQL] Literal should have a value correspon...
Posted by cloud-fan <gi...@git.apache.org>.
Github user cloud-fan commented on the issue:
https://github.com/apache/spark/pull/22724
thanks, merging to master!
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #22724: [SPARK-25734][SQL] Literal should have a value co...
Posted by maropu <gi...@git.apache.org>.
Github user maropu commented on a diff in the pull request:
https://github.com/apache/spark/pull/22724#discussion_r225395710
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/literals.scala ---
@@ -196,6 +197,48 @@ object Literal {
case other =>
throw new RuntimeException(s"no default for type $dataType")
}
+
+ private[expressions] def validateLiteralValue(value: Any, dataType: DataType): Unit = {
+ def doValidate(v: Any, dataType: DataType): Boolean = dataType match {
+ case BooleanType => v.isInstanceOf[Boolean]
+ case ByteType => v.isInstanceOf[Byte]
+ case ShortType => v.isInstanceOf[Short]
+ case IntegerType | DateType => v.isInstanceOf[Int]
+ case LongType | TimestampType => v.isInstanceOf[Long]
+ case FloatType => v.isInstanceOf[Float]
+ case DoubleType => v.isInstanceOf[Double]
+ case _: DecimalType => v.isInstanceOf[Decimal]
+ case CalendarIntervalType => v.isInstanceOf[CalendarInterval]
+ case BinaryType => v.isInstanceOf[Array[Byte]]
+ case StringType => v.isInstanceOf[UTF8String]
+ case st: StructType =>
+ v.isInstanceOf[InternalRow] && {
+ val row = v.asInstanceOf[InternalRow]
+ st.fields.map(_.dataType).zipWithIndex.forall {
+ case (dt, i) => doValidate(row.get(i, dt), dt)
+ }
+ }
+ case at: ArrayType =>
+ v.isInstanceOf[GenericArrayData] && {
+ val ar = v.asInstanceOf[GenericArrayData].array
+ ar.isEmpty || doValidate(ar.head, at.elementType)
+ }
+ case mt: MapType =>
+ v.isInstanceOf[ArrayBasedMapData] && {
+ val map = v.asInstanceOf[ArrayBasedMapData]
+ map.numElements() == 0 || {
--- End diff --
you suggested like this?
```
case mt: MapType =>
v.isInstanceOf[MapData] && {
val map = v.asInstanceOf[MapData]
doValidate(map.keyArray(), ArrayType(mt.keyType)) &&
doValidate(map.valueArray(), ArrayType(mt.valueType))
}
```
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22724: [SPARK-25734][SQL] Literal should have a value correspon...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22724
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/97432/
Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #22724: [SPARK-25734][SQL] Literal should have a value co...
Posted by ueshin <gi...@git.apache.org>.
Github user ueshin commented on a diff in the pull request:
https://github.com/apache/spark/pull/22724#discussion_r225389273
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/literals.scala ---
@@ -196,6 +197,48 @@ object Literal {
case other =>
throw new RuntimeException(s"no default for type $dataType")
}
+
+ private[expressions] def validateLiteralValue(value: Any, dataType: DataType): Unit = {
+ def doValidate(v: Any, dataType: DataType): Boolean = dataType match {
+ case BooleanType => v.isInstanceOf[Boolean]
+ case ByteType => v.isInstanceOf[Byte]
+ case ShortType => v.isInstanceOf[Short]
+ case IntegerType | DateType => v.isInstanceOf[Int]
+ case LongType | TimestampType => v.isInstanceOf[Long]
+ case FloatType => v.isInstanceOf[Float]
+ case DoubleType => v.isInstanceOf[Double]
+ case _: DecimalType => v.isInstanceOf[Decimal]
+ case CalendarIntervalType => v.isInstanceOf[CalendarInterval]
+ case BinaryType => v.isInstanceOf[Array[Byte]]
+ case StringType => v.isInstanceOf[UTF8String]
+ case st: StructType =>
+ v.isInstanceOf[InternalRow] && {
+ val row = v.asInstanceOf[InternalRow]
+ st.fields.map(_.dataType).zipWithIndex.forall {
+ case (dt, i) => doValidate(row.get(i, dt), dt)
+ }
+ }
+ case at: ArrayType =>
+ v.isInstanceOf[GenericArrayData] && {
+ val ar = v.asInstanceOf[GenericArrayData].array
+ ar.isEmpty || doValidate(ar.head, at.elementType)
+ }
+ case mt: MapType =>
+ v.isInstanceOf[ArrayBasedMapData] && {
+ val map = v.asInstanceOf[ArrayBasedMapData]
+ map.numElements() == 0 || {
+ doValidate(map.keyArray.array.head, mt.keyType) &&
+ doValidate(map.valueArray.array.head, mt.valueType)
+ }
+ }
+ case ObjectType(cls) => cls.isInstance(v)
+ case udt: UserDefinedType[_] => doValidate(v, udt.sqlType)
+ case _ => false
--- End diff --
We need to add `NullType` case?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22724: [SPARK-25734][SQL] Literal should have a value correspon...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/22724
**[Test build #97419 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/97419/testReport)** for PR 22724 at commit [`b00f4bb`](https://github.com/apache/spark/commit/b00f4bb614f9cc82ff18d8db03e97bb814b07f26).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds the following public classes _(experimental)_:
* ` s\"but class $`
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22724: [SPARK-25734][SQL] Literal should have a value correspon...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22724
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #22724: [SPARK-25734][SQL] Literal should have a value co...
Posted by maropu <gi...@git.apache.org>.
Github user maropu commented on a diff in the pull request:
https://github.com/apache/spark/pull/22724#discussion_r225352848
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/literals.scala ---
@@ -196,6 +197,31 @@ object Literal {
case other =>
throw new RuntimeException(s"no default for type $dataType")
}
+
+ private[expressions] def validateLiteralValue(v: Any, dataType: DataType): Unit = {
+ def doValidate(v: Any, dataType: DataType): Boolean = dataType match {
+ case BooleanType => v.isInstanceOf[Boolean]
+ case ByteType => v.isInstanceOf[Byte]
+ case ShortType => v.isInstanceOf[Short]
+ case IntegerType | DateType => v.isInstanceOf[Int]
+ case LongType | TimestampType => v.isInstanceOf[Long]
+ case FloatType => v.isInstanceOf[Float]
+ case DoubleType => v.isInstanceOf[Double]
+ case _: DecimalType => v.isInstanceOf[Decimal]
+ case CalendarIntervalType => v.isInstanceOf[CalendarInterval]
+ case BinaryType => v.isInstanceOf[Array[Byte]]
+ case StringType => v.isInstanceOf[UTF8String]
+ case _: StructType => v.isInstanceOf[InternalRow]
+ case _: ArrayType => v.isInstanceOf[ArrayData]
+ case _: MapType => v.isInstanceOf[MapData]
--- End diff --
yea, good catch ....!
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22724: [SPARK-25734][SQL] Literal should have a value correspon...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22724
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/97451/
Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22724: [SPARK-25734][SQL] Literal should have a value correspon...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22724
Merged build finished. Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #22724: [SPARK-25734][SQL] Literal should have a value co...
Posted by cloud-fan <gi...@git.apache.org>.
Github user cloud-fan commented on a diff in the pull request:
https://github.com/apache/spark/pull/22724#discussion_r225368754
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/literals.scala ---
@@ -196,6 +197,48 @@ object Literal {
case other =>
throw new RuntimeException(s"no default for type $dataType")
}
+
+ private[expressions] def validateLiteralValue(value: Any, dataType: DataType): Unit = {
+ def doValidate(v: Any, dataType: DataType): Boolean = dataType match {
+ case BooleanType => v.isInstanceOf[Boolean]
+ case ByteType => v.isInstanceOf[Byte]
+ case ShortType => v.isInstanceOf[Short]
+ case IntegerType | DateType => v.isInstanceOf[Int]
+ case LongType | TimestampType => v.isInstanceOf[Long]
+ case FloatType => v.isInstanceOf[Float]
+ case DoubleType => v.isInstanceOf[Double]
+ case _: DecimalType => v.isInstanceOf[Decimal]
+ case CalendarIntervalType => v.isInstanceOf[CalendarInterval]
+ case BinaryType => v.isInstanceOf[Array[Byte]]
+ case StringType => v.isInstanceOf[UTF8String]
+ case st: StructType =>
+ v.isInstanceOf[GenericInternalRow] && {
--- End diff --
this is definitely not true. can we do
```
v.isInstanceOf[InternalRow] && {
val row = v.asInstanceOf[InternalRow]
st.fields.map(_.dataType).zipWithIndex.foreach {
case (dt, i) => doValidate(row.get(i, dt), dt)
}
}
```
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22724: [SPARK-25734][SQL] Literal should have a value correspon...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/22724
**[Test build #97443 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/97443/testReport)** for PR 22724 at commit [`77f65f9`](https://github.com/apache/spark/commit/77f65f94b85707ca7886687a038d19359cd476b7).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22724: [SPARK-25734][SQL] Literal should have a value correspon...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22724
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/97384/
Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22724: [SPARK-25734][SQL] Literal should have a value correspon...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/22724
**[Test build #97432 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/97432/testReport)** for PR 22724 at commit [`4df5abd`](https://github.com/apache/spark/commit/4df5abd23203e4dc3499001e8dd5c589fa03c535).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22724: [SPARK-25734][SQL] Literal should have a value correspon...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22724
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/4022/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22724: [SPARK-25734][SQL] Literal should have a value correspon...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/22724
**[Test build #97431 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/97431/testReport)** for PR 22724 at commit [`56b4778`](https://github.com/apache/spark/commit/56b4778d3ff3467c53ac1a9e4370dfffcb12a4de).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #22724: [SPARK-25734][SQL] Literal should have a value co...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/spark/pull/22724
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22724: [SPARK-25734][SQL] Literal should have a value correspon...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22724
Merged build finished. Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #22724: [SPARK-25734][SQL] Literal should have a value co...
Posted by cloud-fan <gi...@git.apache.org>.
Github user cloud-fan commented on a diff in the pull request:
https://github.com/apache/spark/pull/22724#discussion_r225392843
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/literals.scala ---
@@ -196,6 +197,48 @@ object Literal {
case other =>
throw new RuntimeException(s"no default for type $dataType")
}
+
+ private[expressions] def validateLiteralValue(value: Any, dataType: DataType): Unit = {
+ def doValidate(v: Any, dataType: DataType): Boolean = dataType match {
+ case BooleanType => v.isInstanceOf[Boolean]
+ case ByteType => v.isInstanceOf[Byte]
+ case ShortType => v.isInstanceOf[Short]
+ case IntegerType | DateType => v.isInstanceOf[Int]
+ case LongType | TimestampType => v.isInstanceOf[Long]
+ case FloatType => v.isInstanceOf[Float]
+ case DoubleType => v.isInstanceOf[Double]
+ case _: DecimalType => v.isInstanceOf[Decimal]
+ case CalendarIntervalType => v.isInstanceOf[CalendarInterval]
+ case BinaryType => v.isInstanceOf[Array[Byte]]
+ case StringType => v.isInstanceOf[UTF8String]
+ case st: StructType =>
+ v.isInstanceOf[InternalRow] && {
--- End diff --
can we do the same for array and map?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #22724: [SPARK-25734][SQL] Literal should have a value co...
Posted by cloud-fan <gi...@git.apache.org>.
Github user cloud-fan commented on a diff in the pull request:
https://github.com/apache/spark/pull/22724#discussion_r225396907
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/literals.scala ---
@@ -196,6 +197,48 @@ object Literal {
case other =>
throw new RuntimeException(s"no default for type $dataType")
}
+
+ private[expressions] def validateLiteralValue(value: Any, dataType: DataType): Unit = {
+ def doValidate(v: Any, dataType: DataType): Boolean = dataType match {
+ case BooleanType => v.isInstanceOf[Boolean]
+ case ByteType => v.isInstanceOf[Byte]
+ case ShortType => v.isInstanceOf[Short]
+ case IntegerType | DateType => v.isInstanceOf[Int]
+ case LongType | TimestampType => v.isInstanceOf[Long]
+ case FloatType => v.isInstanceOf[Float]
+ case DoubleType => v.isInstanceOf[Double]
+ case _: DecimalType => v.isInstanceOf[Decimal]
+ case CalendarIntervalType => v.isInstanceOf[CalendarInterval]
+ case BinaryType => v.isInstanceOf[Array[Byte]]
+ case StringType => v.isInstanceOf[UTF8String]
+ case st: StructType =>
+ v.isInstanceOf[InternalRow] && {
+ val row = v.asInstanceOf[InternalRow]
+ st.fields.map(_.dataType).zipWithIndex.forall {
+ case (dt, i) => doValidate(row.get(i, dt), dt)
+ }
+ }
+ case at: ArrayType =>
+ v.isInstanceOf[GenericArrayData] && {
+ val ar = v.asInstanceOf[GenericArrayData].array
+ ar.isEmpty || doValidate(ar.head, at.elementType)
+ }
+ case mt: MapType =>
+ v.isInstanceOf[ArrayBasedMapData] && {
+ val map = v.asInstanceOf[ArrayBasedMapData]
+ map.numElements() == 0 || {
--- End diff --
yup
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22724: [SPARK-25734][SQL] Literal should have a value correspon...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22724
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #22724: [SPARK-25734][SQL] Literal should have a value co...
Posted by cloud-fan <gi...@git.apache.org>.
Github user cloud-fan commented on a diff in the pull request:
https://github.com/apache/spark/pull/22724#discussion_r225392872
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/literals.scala ---
@@ -196,6 +197,48 @@ object Literal {
case other =>
throw new RuntimeException(s"no default for type $dataType")
}
+
+ private[expressions] def validateLiteralValue(value: Any, dataType: DataType): Unit = {
+ def doValidate(v: Any, dataType: DataType): Boolean = dataType match {
+ case BooleanType => v.isInstanceOf[Boolean]
+ case ByteType => v.isInstanceOf[Byte]
+ case ShortType => v.isInstanceOf[Short]
+ case IntegerType | DateType => v.isInstanceOf[Int]
+ case LongType | TimestampType => v.isInstanceOf[Long]
+ case FloatType => v.isInstanceOf[Float]
+ case DoubleType => v.isInstanceOf[Double]
+ case _: DecimalType => v.isInstanceOf[Decimal]
+ case CalendarIntervalType => v.isInstanceOf[CalendarInterval]
+ case BinaryType => v.isInstanceOf[Array[Byte]]
+ case StringType => v.isInstanceOf[UTF8String]
+ case st: StructType =>
+ v.isInstanceOf[InternalRow] && {
+ val row = v.asInstanceOf[InternalRow]
+ st.fields.map(_.dataType).zipWithIndex.forall {
+ case (dt, i) => doValidate(row.get(i, dt), dt)
+ }
+ }
+ case at: ArrayType =>
+ v.isInstanceOf[GenericArrayData] && {
+ val ar = v.asInstanceOf[GenericArrayData].array
+ ar.isEmpty || doValidate(ar.head, at.elementType)
--- End diff --
I think we need to validate all the elements
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22724: [SPARK-25734][SQL] Literal should have a value correspon...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22724
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/4031/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22724: [SPARK-25734][SQL] Literal should have a value correspon...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22724
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/97457/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22724: [SPARK-25734][SQL] Literal should have a value correspon...
Posted by maropu <gi...@git.apache.org>.
Github user maropu commented on the issue:
https://github.com/apache/spark/pull/22724
yea, its ok to do so.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22724: [SPARK-25734][SQL] Literal should have a value correspon...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/22724
**[Test build #97439 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/97439/testReport)** for PR 22724 at commit [`60a7793`](https://github.com/apache/spark/commit/60a7793ffe600e5b595b69702569db0f9451d0b3).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22724: [SPARK-25734][SQL] Literal should have a value correspon...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/22724
**[Test build #97419 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/97419/testReport)** for PR 22724 at commit [`b00f4bb`](https://github.com/apache/spark/commit/b00f4bb614f9cc82ff18d8db03e97bb814b07f26).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #22724: [SPARK-25734][SQL] Literal should have a value co...
Posted by maropu <gi...@git.apache.org>.
Github user maropu commented on a diff in the pull request:
https://github.com/apache/spark/pull/22724#discussion_r225388391
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/literals.scala ---
@@ -196,6 +197,48 @@ object Literal {
case other =>
throw new RuntimeException(s"no default for type $dataType")
}
+
+ private[expressions] def validateLiteralValue(value: Any, dataType: DataType): Unit = {
+ def doValidate(v: Any, dataType: DataType): Boolean = dataType match {
+ case BooleanType => v.isInstanceOf[Boolean]
+ case ByteType => v.isInstanceOf[Byte]
+ case ShortType => v.isInstanceOf[Short]
+ case IntegerType | DateType => v.isInstanceOf[Int]
+ case LongType | TimestampType => v.isInstanceOf[Long]
+ case FloatType => v.isInstanceOf[Float]
+ case DoubleType => v.isInstanceOf[Double]
+ case _: DecimalType => v.isInstanceOf[Decimal]
+ case CalendarIntervalType => v.isInstanceOf[CalendarInterval]
+ case BinaryType => v.isInstanceOf[Array[Byte]]
+ case StringType => v.isInstanceOf[UTF8String]
+ case st: StructType =>
+ v.isInstanceOf[InternalRow] && {
+ val row = v.asInstanceOf[InternalRow]
+ st.fields.map(_.dataType).zipWithIndex.forall {
+ case (dt, i) => doValidate(row.get(i, dt), dt)
+ }
+ }
+ case at: ArrayType =>
+ v.isInstanceOf[GenericArrayData] && {
+ val ar = v.asInstanceOf[GenericArrayData].array
+ ar.isEmpty || doValidate(ar.head, at.elementType)
+ }
+ case mt: MapType =>
+ v.isInstanceOf[ArrayBasedMapData] && {
+ val map = v.asInstanceOf[ArrayBasedMapData]
+ map.numElements() == 0 || {
+ doValidate(map.keyArray.array.head, mt.keyType) &&
+ doValidate(map.valueArray.array.head, mt.valueType)
+ }
+ }
--- End diff --
Since the whole element check seems to be expensive, the current one is ok to me.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22724: [SPARK-25734][SQL] Literal should have a value correspon...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22724
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22724: [SPARK-25734][SQL] Literal should have a value correspon...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/22724
**[Test build #97432 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/97432/testReport)** for PR 22724 at commit [`4df5abd`](https://github.com/apache/spark/commit/4df5abd23203e4dc3499001e8dd5c589fa03c535).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22724: [SPARK-25734][SQL] Literal should have a value correspon...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/22724
**[Test build #97384 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/97384/testReport)** for PR 22724 at commit [`9115d26`](https://github.com/apache/spark/commit/9115d26642496b1ca1fdf0d310d47ee0ebeb3222).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22724: [SPARK-25734][SQL] Literal should have a value correspon...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22724
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/4021/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22724: [SPARK-25734][SQL] Literal should have a value correspon...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22724
Merged build finished. Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22724: [SPARK-25734][SQL] Literal should have a value correspon...
Posted by mgaido91 <gi...@git.apache.org>.
Github user mgaido91 commented on the issue:
https://github.com/apache/spark/pull/22724
LGTM, but this is a breaking change, so I'd suggest to do it only for 3.0 and add a note to the migration guide. WDYT?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #22724: [SPARK-25734][SQL] Literal should have a value co...
Posted by ueshin <gi...@git.apache.org>.
Github user ueshin commented on a diff in the pull request:
https://github.com/apache/spark/pull/22724#discussion_r225387436
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/literals.scala ---
@@ -196,6 +197,48 @@ object Literal {
case other =>
throw new RuntimeException(s"no default for type $dataType")
}
+
+ private[expressions] def validateLiteralValue(value: Any, dataType: DataType): Unit = {
+ def doValidate(v: Any, dataType: DataType): Boolean = dataType match {
+ case BooleanType => v.isInstanceOf[Boolean]
+ case ByteType => v.isInstanceOf[Byte]
+ case ShortType => v.isInstanceOf[Short]
+ case IntegerType | DateType => v.isInstanceOf[Int]
+ case LongType | TimestampType => v.isInstanceOf[Long]
+ case FloatType => v.isInstanceOf[Float]
+ case DoubleType => v.isInstanceOf[Double]
+ case _: DecimalType => v.isInstanceOf[Decimal]
+ case CalendarIntervalType => v.isInstanceOf[CalendarInterval]
+ case BinaryType => v.isInstanceOf[Array[Byte]]
+ case StringType => v.isInstanceOf[UTF8String]
+ case st: StructType =>
+ v.isInstanceOf[InternalRow] && {
+ val row = v.asInstanceOf[InternalRow]
+ st.fields.map(_.dataType).zipWithIndex.forall {
+ case (dt, i) => doValidate(row.get(i, dt), dt)
+ }
+ }
+ case at: ArrayType =>
+ v.isInstanceOf[GenericArrayData] && {
+ val ar = v.asInstanceOf[GenericArrayData].array
+ ar.isEmpty || doValidate(ar.head, at.elementType)
+ }
+ case mt: MapType =>
+ v.isInstanceOf[ArrayBasedMapData] && {
+ val map = v.asInstanceOf[ArrayBasedMapData]
+ map.numElements() == 0 || {
+ doValidate(map.keyArray.array.head, mt.keyType) &&
+ doValidate(map.valueArray.array.head, mt.valueType)
+ }
+ }
--- End diff --
I'm wondering whether we don't need to check the whole elements for `ArrayType` and `MapType`.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22724: [SPARK-25734][SQL] Literal should have a value correspon...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22724
Merged build finished. Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22724: [SPARK-25734][SQL] Literal should have a value correspon...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/22724
**[Test build #97451 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/97451/testReport)** for PR 22724 at commit [`a2f0288`](https://github.com/apache/spark/commit/a2f02880bbcdc6489c0e7680ace8f4eeb3c96bfd).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #22724: [SPARK-25734][SQL] Literal should have a value co...
Posted by cloud-fan <gi...@git.apache.org>.
Github user cloud-fan commented on a diff in the pull request:
https://github.com/apache/spark/pull/22724#discussion_r225392951
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/literals.scala ---
@@ -196,6 +197,48 @@ object Literal {
case other =>
throw new RuntimeException(s"no default for type $dataType")
}
+
+ private[expressions] def validateLiteralValue(value: Any, dataType: DataType): Unit = {
+ def doValidate(v: Any, dataType: DataType): Boolean = dataType match {
+ case BooleanType => v.isInstanceOf[Boolean]
+ case ByteType => v.isInstanceOf[Byte]
+ case ShortType => v.isInstanceOf[Short]
+ case IntegerType | DateType => v.isInstanceOf[Int]
+ case LongType | TimestampType => v.isInstanceOf[Long]
+ case FloatType => v.isInstanceOf[Float]
+ case DoubleType => v.isInstanceOf[Double]
+ case _: DecimalType => v.isInstanceOf[Decimal]
+ case CalendarIntervalType => v.isInstanceOf[CalendarInterval]
+ case BinaryType => v.isInstanceOf[Array[Byte]]
+ case StringType => v.isInstanceOf[UTF8String]
+ case st: StructType =>
+ v.isInstanceOf[InternalRow] && {
+ val row = v.asInstanceOf[InternalRow]
+ st.fields.map(_.dataType).zipWithIndex.forall {
+ case (dt, i) => doValidate(row.get(i, dt), dt)
+ }
+ }
+ case at: ArrayType =>
+ v.isInstanceOf[GenericArrayData] && {
+ val ar = v.asInstanceOf[GenericArrayData].array
+ ar.isEmpty || doValidate(ar.head, at.elementType)
+ }
+ case mt: MapType =>
+ v.isInstanceOf[ArrayBasedMapData] && {
+ val map = v.asInstanceOf[ArrayBasedMapData]
+ map.numElements() == 0 || {
--- End diff --
we don't need this. The array validation already consider numElements
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22724: [SPARK-25734][SQL] Literal should have a value correspon...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/22724
**[Test build #97433 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/97433/testReport)** for PR 22724 at commit [`1c042f3`](https://github.com/apache/spark/commit/1c042f3018e5fd5c1f14324b3d20b741d202e643).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22724: [SPARK-25734][SQL] Literal should have a value correspon...
Posted by cloud-fan <gi...@git.apache.org>.
Github user cloud-fan commented on the issue:
https://github.com/apache/spark/pull/22724
yea +1 on 3.0 only, this is kind of a developer API, advanced users may use it.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22724: [SPARK-25734][SQL] Literal should have a value correspon...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22724
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/4027/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22724: [SPARK-25734][SQL] Literal should have a value correspon...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22724
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #22724: [SPARK-25734][SQL] Literal should have a value co...
Posted by maropu <gi...@git.apache.org>.
Github user maropu commented on a diff in the pull request:
https://github.com/apache/spark/pull/22724#discussion_r225395382
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/literals.scala ---
@@ -196,6 +197,48 @@ object Literal {
case other =>
throw new RuntimeException(s"no default for type $dataType")
}
+
+ private[expressions] def validateLiteralValue(value: Any, dataType: DataType): Unit = {
+ def doValidate(v: Any, dataType: DataType): Boolean = dataType match {
+ case BooleanType => v.isInstanceOf[Boolean]
+ case ByteType => v.isInstanceOf[Byte]
+ case ShortType => v.isInstanceOf[Short]
+ case IntegerType | DateType => v.isInstanceOf[Int]
+ case LongType | TimestampType => v.isInstanceOf[Long]
+ case FloatType => v.isInstanceOf[Float]
+ case DoubleType => v.isInstanceOf[Double]
+ case _: DecimalType => v.isInstanceOf[Decimal]
+ case CalendarIntervalType => v.isInstanceOf[CalendarInterval]
+ case BinaryType => v.isInstanceOf[Array[Byte]]
+ case StringType => v.isInstanceOf[UTF8String]
+ case st: StructType =>
+ v.isInstanceOf[InternalRow] && {
--- End diff --
done
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22724: [SPARK-25734][SQL] Literal should have a value correspon...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22724
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22724: [SPARK-25734][SQL] Literal should have a value correspon...
Posted by cloud-fan <gi...@git.apache.org>.
Github user cloud-fan commented on the issue:
https://github.com/apache/spark/pull/22724
`Literal` is an internal API, and AFAIK end-users can't construct an invalid `Literal` with public APIs. If they can, then it's a bug and we have a problem...
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22724: [SPARK-25734][SQL] Literal should have a value correspon...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22724
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/4020/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22724: [SPARK-25734][SQL] Literal should have a value correspon...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22724
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/97443/
Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #22724: [SPARK-25734][SQL] Literal should have a value co...
Posted by dongjoon-hyun <gi...@git.apache.org>.
Github user dongjoon-hyun commented on a diff in the pull request:
https://github.com/apache/spark/pull/22724#discussion_r225288037
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/literals.scala ---
@@ -196,6 +197,31 @@ object Literal {
case other =>
throw new RuntimeException(s"no default for type $dataType")
}
+
+ private[expressions] def validateLiteralValue(v: Any, dataType: DataType): Unit = {
+ def doValidate(v: Any, dataType: DataType): Boolean = dataType match {
+ case BooleanType => v.isInstanceOf[Boolean]
+ case ByteType => v.isInstanceOf[Byte]
+ case ShortType => v.isInstanceOf[Short]
+ case IntegerType | DateType => v.isInstanceOf[Int]
+ case LongType | TimestampType => v.isInstanceOf[Long]
+ case FloatType => v.isInstanceOf[Float]
+ case DoubleType => v.isInstanceOf[Double]
+ case _: DecimalType => v.isInstanceOf[Decimal]
+ case CalendarIntervalType => v.isInstanceOf[CalendarInterval]
+ case BinaryType => v.isInstanceOf[Array[Byte]]
+ case StringType => v.isInstanceOf[UTF8String]
+ case _: StructType => v.isInstanceOf[InternalRow]
+ case _: ArrayType => v.isInstanceOf[ArrayData]
+ case _: MapType => v.isInstanceOf[MapData]
+ case ObjectType(cls) => cls.isInstance(v)
+ case udt: UserDefinedType[_] => doValidate(v, udt.sqlType)
+ case _ => false
+ }
+ require(v == null || doValidate(v, dataType),
+ s"Literal must have a corresponding value to ${dataType.catalogString}, " +
+ s"but ${if (v != null) s"class ${Utils.getSimpleName(v.getClass)}" else "null"} found.")
--- End diff --
`but null found` cannot happen logically.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22724: [SPARK-25734][SQL] Literal should have a value correspon...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22724
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/97431/
Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22724: [SPARK-25734][SQL] Literal should have a value correspon...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/22724
**[Test build #97427 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/97427/testReport)** for PR 22724 at commit [`a30e9ce`](https://github.com/apache/spark/commit/a30e9ce7b3a5b0965801fe69e8aabb31de50bc13).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22724: [SPARK-25734][SQL] Literal should have a value correspon...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22724
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/4009/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22724: [SPARK-25734][SQL] Literal should have a value correspon...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/22724
**[Test build #97439 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/97439/testReport)** for PR 22724 at commit [`60a7793`](https://github.com/apache/spark/commit/60a7793ffe600e5b595b69702569db0f9451d0b3).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #22724: [SPARK-25734][SQL] Literal should have a value co...
Posted by maropu <gi...@git.apache.org>.
Github user maropu commented on a diff in the pull request:
https://github.com/apache/spark/pull/22724#discussion_r225397788
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/literals.scala ---
@@ -196,6 +197,48 @@ object Literal {
case other =>
throw new RuntimeException(s"no default for type $dataType")
}
+
+ private[expressions] def validateLiteralValue(value: Any, dataType: DataType): Unit = {
+ def doValidate(v: Any, dataType: DataType): Boolean = dataType match {
+ case BooleanType => v.isInstanceOf[Boolean]
+ case ByteType => v.isInstanceOf[Byte]
+ case ShortType => v.isInstanceOf[Short]
+ case IntegerType | DateType => v.isInstanceOf[Int]
+ case LongType | TimestampType => v.isInstanceOf[Long]
+ case FloatType => v.isInstanceOf[Float]
+ case DoubleType => v.isInstanceOf[Double]
+ case _: DecimalType => v.isInstanceOf[Decimal]
+ case CalendarIntervalType => v.isInstanceOf[CalendarInterval]
+ case BinaryType => v.isInstanceOf[Array[Byte]]
+ case StringType => v.isInstanceOf[UTF8String]
+ case st: StructType =>
+ v.isInstanceOf[InternalRow] && {
+ val row = v.asInstanceOf[InternalRow]
+ st.fields.map(_.dataType).zipWithIndex.forall {
+ case (dt, i) => doValidate(row.get(i, dt), dt)
+ }
+ }
+ case at: ArrayType =>
+ v.isInstanceOf[GenericArrayData] && {
+ val ar = v.asInstanceOf[GenericArrayData].array
+ ar.isEmpty || doValidate(ar.head, at.elementType)
+ }
+ case mt: MapType =>
+ v.isInstanceOf[ArrayBasedMapData] && {
+ val map = v.asInstanceOf[ArrayBasedMapData]
+ map.numElements() == 0 || {
--- End diff --
ok
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #22724: [SPARK-25734][SQL] Literal should have a value co...
Posted by maropu <gi...@git.apache.org>.
Github user maropu commented on a diff in the pull request:
https://github.com/apache/spark/pull/22724#discussion_r225397940
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/literals.scala ---
@@ -196,6 +197,48 @@ object Literal {
case other =>
throw new RuntimeException(s"no default for type $dataType")
}
+
+ private[expressions] def validateLiteralValue(value: Any, dataType: DataType): Unit = {
+ def doValidate(v: Any, dataType: DataType): Boolean = dataType match {
+ case BooleanType => v.isInstanceOf[Boolean]
+ case ByteType => v.isInstanceOf[Byte]
+ case ShortType => v.isInstanceOf[Short]
+ case IntegerType | DateType => v.isInstanceOf[Int]
+ case LongType | TimestampType => v.isInstanceOf[Long]
+ case FloatType => v.isInstanceOf[Float]
+ case DoubleType => v.isInstanceOf[Double]
+ case _: DecimalType => v.isInstanceOf[Decimal]
+ case CalendarIntervalType => v.isInstanceOf[CalendarInterval]
+ case BinaryType => v.isInstanceOf[Array[Byte]]
+ case StringType => v.isInstanceOf[UTF8String]
+ case st: StructType =>
+ v.isInstanceOf[InternalRow] && {
+ val row = v.asInstanceOf[InternalRow]
+ st.fields.map(_.dataType).zipWithIndex.forall {
+ case (dt, i) => doValidate(row.get(i, dt), dt)
+ }
+ }
+ case at: ArrayType =>
+ v.isInstanceOf[GenericArrayData] && {
+ val ar = v.asInstanceOf[GenericArrayData].array
+ ar.isEmpty || doValidate(ar.head, at.elementType)
+ }
+ case mt: MapType =>
+ v.isInstanceOf[ArrayBasedMapData] && {
+ val map = v.asInstanceOf[ArrayBasedMapData]
+ map.numElements() == 0 || {
--- End diff --
updated.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22724: [SPARK-25734][SQL] Literal should have a value correspon...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22724
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22724: [SPARK-25734][SQL] Literal should have a value correspon...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22724
Merged build finished. Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22724: [SPARK-25734][SQL] Literal should have a value correspon...
Posted by maropu <gi...@git.apache.org>.
Github user maropu commented on the issue:
https://github.com/apache/spark/pull/22724
I'm checking the reason of the test failures...
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22724: [SPARK-25734][SQL] Literal should have a value correspon...
Posted by cloud-fan <gi...@git.apache.org>.
Github user cloud-fan commented on the issue:
https://github.com/apache/spark/pull/22724
LGTM
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22724: [SPARK-25734][SQL] Literal should have a value correspon...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22724
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22724: [SPARK-25734][SQL] Literal should have a value correspon...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/22724
**[Test build #97451 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/97451/testReport)** for PR 22724 at commit [`a2f0288`](https://github.com/apache/spark/commit/a2f02880bbcdc6489c0e7680ace8f4eeb3c96bfd).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22724: [SPARK-25734][SQL] Literal should have a value correspon...
Posted by cloud-fan <gi...@git.apache.org>.
Github user cloud-fan commented on the issue:
https://github.com/apache/spark/pull/22724
LGTM pending jenkins
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22724: [SPARK-25734][SQL] Literal should have a value correspon...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22724
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/4024/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22724: [SPARK-25734][SQL] Literal should have a value correspon...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22724
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22724: [SPARK-25734][SQL] Literal should have a value correspon...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22724
Merged build finished. Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22724: [SPARK-25734][SQL] Literal should have a value correspon...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22724
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/97419/
Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22724: [SPARK-25734][SQL] Literal should have a value correspon...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/22724
**[Test build #97384 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/97384/testReport)** for PR 22724 at commit [`9115d26`](https://github.com/apache/spark/commit/9115d26642496b1ca1fdf0d310d47ee0ebeb3222).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds the following public classes _(experimental)_:
* ` s\"but $`
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22724: [SPARK-25734][SQL] Literal should have a value correspon...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22724
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/4017/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22724: [SPARK-25734][SQL] Literal should have a value correspon...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22724
Merged build finished. Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22724: [SPARK-25734][SQL] Literal should have a value correspon...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22724
Merged build finished. Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22724: [SPARK-25734][SQL] Literal should have a value correspon...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22724
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22724: [SPARK-25734][SQL] Literal should have a value correspon...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22724
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22724: [SPARK-25734][SQL] Literal should have a value correspon...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/22724
**[Test build #97443 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/97443/testReport)** for PR 22724 at commit [`77f65f9`](https://github.com/apache/spark/commit/77f65f94b85707ca7886687a038d19359cd476b7).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22724: [SPARK-25734][SQL] Literal should have a value correspon...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/22724
**[Test build #97433 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/97433/testReport)** for PR 22724 at commit [`1c042f3`](https://github.com/apache/spark/commit/1c042f3018e5fd5c1f14324b3d20b741d202e643).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22724: [SPARK-25734][SQL] Literal should have a value correspon...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/22724
**[Test build #97431 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/97431/testReport)** for PR 22724 at commit [`56b4778`](https://github.com/apache/spark/commit/56b4778d3ff3467c53ac1a9e4370dfffcb12a4de).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22724: [SPARK-25734][SQL] Literal should have a value correspon...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22724
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #22724: [SPARK-25734][SQL] Literal should have a value co...
Posted by cloud-fan <gi...@git.apache.org>.
Github user cloud-fan commented on a diff in the pull request:
https://github.com/apache/spark/pull/22724#discussion_r225218350
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/literals.scala ---
@@ -196,6 +197,31 @@ object Literal {
case other =>
throw new RuntimeException(s"no default for type $dataType")
}
+
+ private[expressions] def validateLiteralValue(v: Any, dataType: DataType): Unit = {
+ def doValidate(v: Any, dataType: DataType): Boolean = dataType match {
+ case BooleanType => v.isInstanceOf[Boolean]
+ case ByteType => v.isInstanceOf[Byte]
+ case ShortType => v.isInstanceOf[Short]
+ case IntegerType | DateType => v.isInstanceOf[Int]
+ case LongType | TimestampType => v.isInstanceOf[Long]
+ case FloatType => v.isInstanceOf[Float]
+ case DoubleType => v.isInstanceOf[Double]
+ case _: DecimalType => v.isInstanceOf[Decimal]
+ case CalendarIntervalType => v.isInstanceOf[CalendarInterval]
+ case BinaryType => v.isInstanceOf[Array[Byte]]
+ case StringType => v.isInstanceOf[UTF8String]
+ case _: StructType => v.isInstanceOf[InternalRow]
+ case _: ArrayType => v.isInstanceOf[ArrayData]
+ case _: MapType => v.isInstanceOf[MapData]
--- End diff --
ah good point!
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22724: [SPARK-25734][SQL] Literal should have a value correspon...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/22724
**[Test build #97457 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/97457/testReport)** for PR 22724 at commit [`04a535d`](https://github.com/apache/spark/commit/04a535d330449b80a81b2f329283236b441bff00).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22724: [SPARK-25734][SQL] Literal should have a value correspon...
Posted by mgaido91 <gi...@git.apache.org>.
Github user mgaido91 commented on the issue:
https://github.com/apache/spark/pull/22724
yes, you're right. So we don't need any migration note. But what if a user uses `Literal` directly? He/she shouldn't, but we cannot exclude this is happening. So I'd still target this for 3.0 only.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22724: [SPARK-25734][SQL] Literal should have a value correspon...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22724
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/3976/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22724: [SPARK-25734][SQL] Literal should have a value correspon...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22724
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/4035/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #22724: [SPARK-25734][SQL] Literal should have a value co...
Posted by ueshin <gi...@git.apache.org>.
Github user ueshin commented on a diff in the pull request:
https://github.com/apache/spark/pull/22724#discussion_r225194050
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/literals.scala ---
@@ -196,6 +197,31 @@ object Literal {
case other =>
throw new RuntimeException(s"no default for type $dataType")
}
+
+ private[expressions] def validateLiteralValue(v: Any, dataType: DataType): Unit = {
+ def doValidate(v: Any, dataType: DataType): Boolean = dataType match {
+ case BooleanType => v.isInstanceOf[Boolean]
+ case ByteType => v.isInstanceOf[Byte]
+ case ShortType => v.isInstanceOf[Short]
+ case IntegerType | DateType => v.isInstanceOf[Int]
+ case LongType | TimestampType => v.isInstanceOf[Long]
+ case FloatType => v.isInstanceOf[Float]
+ case DoubleType => v.isInstanceOf[Double]
+ case _: DecimalType => v.isInstanceOf[Decimal]
+ case CalendarIntervalType => v.isInstanceOf[CalendarInterval]
+ case BinaryType => v.isInstanceOf[Array[Byte]]
+ case StringType => v.isInstanceOf[UTF8String]
+ case _: StructType => v.isInstanceOf[InternalRow]
+ case _: ArrayType => v.isInstanceOf[ArrayData]
+ case _: MapType => v.isInstanceOf[MapData]
--- End diff --
Should validate recursively for `StructType`, `ArrayType`, and `MapType`?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #22724: [SPARK-25734][SQL] Literal should have a value co...
Posted by maropu <gi...@git.apache.org>.
Github user maropu commented on a diff in the pull request:
https://github.com/apache/spark/pull/22724#discussion_r225353071
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/literals.scala ---
@@ -196,6 +197,31 @@ object Literal {
case other =>
throw new RuntimeException(s"no default for type $dataType")
}
+
+ private[expressions] def validateLiteralValue(v: Any, dataType: DataType): Unit = {
+ def doValidate(v: Any, dataType: DataType): Boolean = dataType match {
+ case BooleanType => v.isInstanceOf[Boolean]
+ case ByteType => v.isInstanceOf[Byte]
+ case ShortType => v.isInstanceOf[Short]
+ case IntegerType | DateType => v.isInstanceOf[Int]
+ case LongType | TimestampType => v.isInstanceOf[Long]
+ case FloatType => v.isInstanceOf[Float]
+ case DoubleType => v.isInstanceOf[Double]
+ case _: DecimalType => v.isInstanceOf[Decimal]
+ case CalendarIntervalType => v.isInstanceOf[CalendarInterval]
+ case BinaryType => v.isInstanceOf[Array[Byte]]
+ case StringType => v.isInstanceOf[UTF8String]
+ case _: StructType => v.isInstanceOf[InternalRow]
+ case _: ArrayType => v.isInstanceOf[ArrayData]
+ case _: MapType => v.isInstanceOf[MapData]
+ case ObjectType(cls) => cls.isInstance(v)
+ case udt: UserDefinedType[_] => doValidate(v, udt.sqlType)
+ case _ => false
+ }
+ require(v == null || doValidate(v, dataType),
+ s"Literal must have a corresponding value to ${dataType.catalogString}, " +
+ s"but ${if (v != null) s"class ${Utils.getSimpleName(v.getClass)}" else "null"} found.")
--- End diff --
oh, yes. I'll update.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22724: [SPARK-25734][SQL] Literal should have a value correspon...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22724
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/97433/
Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22724: [SPARK-25734][SQL] Literal should have a value correspon...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22724
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/97427/
Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22724: [SPARK-25734][SQL] Literal should have a value correspon...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/22724
**[Test build #97427 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/97427/testReport)** for PR 22724 at commit [`a30e9ce`](https://github.com/apache/spark/commit/a30e9ce7b3a5b0965801fe69e8aabb31de50bc13).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org