You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2020/03/23 13:03:54 UTC
[GitHub] [spark] HyukjinKwon opened a new pull request #27991:
[SPARK-31227][SQL] Non-nullable null type in complex types should not
coerce to nullable type
HyukjinKwon opened a new pull request #27991: [SPARK-31227][SQL] Non-nullable null type in complex types should not coerce to nullable type
URL: https://github.com/apache/spark/pull/27991
### What changes were proposed in this pull request?
This PR targets for non-nullable null type not to coerce to nullable type in complex types.
Non-nullable fields in struct, elements in an array and entries in map can mean empty array, struct and map. They are empty so it does not need to force the nullability when we find common types.
### Why are the changes needed?
To type coercion coherent and consistent. Currently, we correctly keep the nullability even between non-nullable fields:
```scala
import org.apache.spark.sql.types._
import org.apache.spark.sql.functions._
spark.range(1).select(array(lit(1)).cast(ArrayType(IntegerType, false))).printSchema()
spark.range(1).select(array(lit(1)).cast(ArrayType(DoubleType, false))).printSchema()
```
```scala
spark.range(1).selectExpr("concat(array(1), array(1)) as arr").printSchema()
```
### Does this PR introduce any user-facing change?
Yes.
```scala
import org.apache.spark.sql.types._
import org.apache.spark.sql.functions._
spark.range(1).select(array().cast(ArrayType(IntegerType, false))).printSchema()
```
```scala
spark.range(1).selectExpr("concat(array(), array(1)) as arr").printSchema()
```
**Before:**
```
org.apache.spark.sql.AnalysisException: cannot resolve 'array()' due to data type mismatch: cannot cast array<null> to array<int>;;
'Project [cast(array() as array<int>) AS array()#68]
+- Range (0, 1, step=1, splits=Some(12))
at org.apache.spark.sql.catalyst.analysis.package$AnalysisErrorAt.failAnalysis(package.scala:42)
at org.apache.spark.sql.catalyst.analysis.CheckAnalysis$$anonfun$$nestedInanonfun$checkAnalysis$1$2.applyOrElse(CheckAnalysis.scala:149)
at org.apache.spark.sql.catalyst.analysis.CheckAnalysis$$anonfun$$nestedInanonfun$checkAnalysis$1$2.applyOrElse(CheckAnalysis.scala:140)
at org.apache.spark.sql.catalyst.trees.TreeNode.$anonfun$transformUp$2(TreeNode.scala:333)
at org.apache.spark.sql.catalyst.trees.CurrentOrigin$.withOrigin(TreeNode.scala:72)
at org.apache.spark.sql.catalyst.trees.TreeNode.transformUp(TreeNode.scala:333)
at org.apache.spark.sql.catalyst.trees.TreeNode.$anonfun$transformUp$1(TreeNode.scala:330)
at org.apache.spark.sql.catalyst.trees.TreeNode.$anonfun$mapChildren$1(TreeNode.scala:399)
at org.apache.spark.sql.catalyst.trees.TreeNode.mapProductIterator(TreeNode.scala:237)
```
```
root
|-- arr: array (nullable = false)
| |-- element: integer (containsNull = true)
```
**After:**
```
root
|-- array(): array (nullable = false)
| |-- element: integer (containsNull = false)
```
```
root
|-- arr: array (nullable = false)
| |-- element: integer (containsNull = false)
```
### How was this patch tested?
Unittests were added and manually tested.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #27991:
[SPARK-31227][SQL] Non-nullable null type in complex types should not
coerce to nullable type
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on issue #27991: [SPARK-31227][SQL] Non-nullable null type in complex types should not coerce to nullable type
URL: https://github.com/apache/spark/pull/27991#issuecomment-602946350
**[Test build #120226 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/120226/testReport)** for PR 27991 at commit [`dfd9343`](https://github.com/apache/spark/commit/dfd9343e803fb4d13496aaa30c41d77c7e0d0798).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27991: [SPARK-31227][SQL]
Non-nullable null type in complex types should not coerce to nullable type
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27991: [SPARK-31227][SQL] Non-nullable null type in complex types should not coerce to nullable type
URL: https://github.com/apache/spark/pull/27991#issuecomment-603115680
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27991:
[SPARK-31227][SQL] Non-nullable null type in complex types should not
coerce to nullable type
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27991: [SPARK-31227][SQL] Non-nullable null type in complex types should not coerce to nullable type
URL: https://github.com/apache/spark/pull/27991#issuecomment-602580538
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/24919/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27991:
[SPARK-31227][SQL] Non-nullable null type in complex types should not
coerce to nullable type
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27991: [SPARK-31227][SQL] Non-nullable null type in complex types should not coerce to nullable type
URL: https://github.com/apache/spark/pull/27991#issuecomment-602781997
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/120209/
Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27991:
[SPARK-31227][SQL] Non-nullable null type in complex types should not
coerce to nullable type
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27991: [SPARK-31227][SQL] Non-nullable null type in complex types should not coerce to nullable type
URL: https://github.com/apache/spark/pull/27991#issuecomment-602767508
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/120206/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] maropu commented on a change in pull request #27991:
[SPARK-31227][SQL] Non-nullable null type in complex types should not
coerce to nullable type
Posted by GitBox <gi...@apache.org>.
maropu commented on a change in pull request #27991: [SPARK-31227][SQL] Non-nullable null type in complex types should not coerce to nullable type
URL: https://github.com/apache/spark/pull/27991#discussion_r397799044
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/Cast.scala
##########
@@ -210,8 +206,15 @@ object Cast {
case _ => false // overflow
}
+ /**
+ * Returns `true` iff it should change the nullability of this type in map type,
+ * array type, expressions such as cast, etc.
+ *
+ * Note that you should take the nullability context into account.
+ * For example, it should not force to nullable type when null type in array
+ * type is non-nullable which means an empty array of null type.
+ */
def forceNullable(from: DataType, to: DataType): Boolean = (from, to) match {
- case (NullType, _) => true
Review comment:
nit: I think its better to add more tests other than integer types (e.g., `null types -> date/timestamp types` discussed above) in https://github.com/apache/spark/pull/27991/files#diff-0ced8bd3a8e8459e7f66333b0d936771R1169
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] viirya commented on a change in pull request #27991:
[SPARK-31227][SQL] Non-nullable null type in complex types should not
coerce to nullable type
Posted by GitBox <gi...@apache.org>.
viirya commented on a change in pull request #27991: [SPARK-31227][SQL] Non-nullable null type in complex types should not coerce to nullable type
URL: https://github.com/apache/spark/pull/27991#discussion_r396916361
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/Cast.scala
##########
@@ -210,8 +206,15 @@ object Cast {
case _ => false // overflow
}
+ /**
+ * Returns `true` iff it should change the nullability of this type in map type,
+ * array type, expressions such as cast, etc.
+ *
+ * Note that you should take the nullability context into account.
+ * For example, it should not force to nullable type when null type in array
+ * type is non-nullable which means an empty array of null type.
+ */
Review comment:
When reading this comment, I think it means `forceNullable(NullType, _)` can be true or false, depending on the context?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27991: [SPARK-31227][SQL]
Non-nullable null type in complex types should not coerce to nullable type
Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #27991: [SPARK-31227][SQL] Non-nullable null type in complex types should not coerce to nullable type
URL: https://github.com/apache/spark/pull/27991#issuecomment-602602220
**[Test build #120209 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/120209/testReport)** for PR 27991 at commit [`51acfeb`](https://github.com/apache/spark/commit/51acfebc45aee180659b6e58d18d5833be2d77ae).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #27991:
[SPARK-31227][SQL] Non-nullable null type in complex types should not
coerce to nullable type
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #27991: [SPARK-31227][SQL] Non-nullable null type in complex types should not coerce to nullable type
URL: https://github.com/apache/spark/pull/27991#discussion_r396883721
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/Cast.scala
##########
@@ -210,8 +206,15 @@ object Cast {
case _ => false // overflow
}
+ /**
+ * Returns `true` iff it should change the nullability of this type in map type,
+ * array type, expression, etc.
Review comment:
can you give an example of `expression` here?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a change in pull request #27991:
[SPARK-31227][SQL] Non-nullable null type in complex types should not
coerce to nullable type
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on a change in pull request #27991: [SPARK-31227][SQL] Non-nullable null type in complex types should not coerce to nullable type
URL: https://github.com/apache/spark/pull/27991#discussion_r396438243
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/Cast.scala
##########
@@ -211,7 +211,6 @@ object Cast {
}
def forceNullable(from: DataType, to: DataType): Boolean = (from, to) match {
- case (NullType, _) => true
Review comment:
`ArrayType(NullType, containsNull=False)` can't contain `null`, no?
> array(null) to array<int>
In this case, it should respect `containsNull` in each side.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27991:
[SPARK-31227][SQL] Non-nullable null type in complex types should not
coerce to nullable type
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27991: [SPARK-31227][SQL] Non-nullable null type in complex types should not coerce to nullable type
URL: https://github.com/apache/spark/pull/27991#issuecomment-603045397
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/24959/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27991: [SPARK-31227][SQL]
Non-nullable null type in complex types should not coerce to nullable type
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27991: [SPARK-31227][SQL] Non-nullable null type in complex types should not coerce to nullable type
URL: https://github.com/apache/spark/pull/27991#issuecomment-603646158
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27991:
[SPARK-31227][SQL] Non-nullable null type in complex types should not
coerce to nullable type
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27991: [SPARK-31227][SQL] Non-nullable null type in complex types should not coerce to nullable type
URL: https://github.com/apache/spark/pull/27991#issuecomment-603827062
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/25066/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] maropu commented on a change in pull request #27991:
[SPARK-31227][SQL] Non-nullable null type in complex types should not
coerce to nullable type
Posted by GitBox <gi...@apache.org>.
maropu commented on a change in pull request #27991: [SPARK-31227][SQL] Non-nullable null type in complex types should not coerce to nullable type
URL: https://github.com/apache/spark/pull/27991#discussion_r396820174
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/Cast.scala
##########
@@ -77,7 +77,8 @@ object Cast {
resolvableNullability(fn || forceNullable(fromType, toType), tn)
case (MapType(fromKey, fromValue, fn), MapType(toKey, toValue, tn)) =>
- canCast(fromKey, toKey) && canCastMapKeyNullSafe(fromKey, toKey) &&
+ canCast(fromKey, toKey) &&
+ (!forceNullable(fromKey, toKey)) &&
Review comment:
nit: can we remove the outermost parentheses?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27991:
[SPARK-31227][SQL] Non-nullable null type in complex types should not
coerce to nullable type
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27991: [SPARK-31227][SQL] Non-nullable null type in complex types should not coerce to nullable type
URL: https://github.com/apache/spark/pull/27991#issuecomment-602947203
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27991: [SPARK-31227][SQL]
Non-nullable null type in complex types should not coerce to nullable type
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27991: [SPARK-31227][SQL] Non-nullable null type in complex types should not coerce to nullable type
URL: https://github.com/apache/spark/pull/27991#issuecomment-602580538
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/24919/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27991:
[SPARK-31227][SQL] Non-nullable null type in complex types should not
coerce to nullable type
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27991: [SPARK-31227][SQL] Non-nullable null type in complex types should not coerce to nullable type
URL: https://github.com/apache/spark/pull/27991#issuecomment-603646163
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/120292/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27991: [SPARK-31227][SQL]
Non-nullable null type in complex types should not coerce to nullable type
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27991: [SPARK-31227][SQL] Non-nullable null type in complex types should not coerce to nullable type
URL: https://github.com/apache/spark/pull/27991#issuecomment-603999812
Merged build finished. Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27991:
[SPARK-31227][SQL] Non-nullable null type in complex types should not
coerce to nullable type
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27991: [SPARK-31227][SQL] Non-nullable null type in complex types should not coerce to nullable type
URL: https://github.com/apache/spark/pull/27991#issuecomment-604044578
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27991: [SPARK-31227][SQL]
Non-nullable null type in complex types should not coerce to nullable type
Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #27991: [SPARK-31227][SQL] Non-nullable null type in complex types should not coerce to nullable type
URL: https://github.com/apache/spark/pull/27991#issuecomment-603064790
**[Test build #120246 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/120246/testReport)** for PR 27991 at commit [`7bae2c1`](https://github.com/apache/spark/commit/7bae2c1c81ca8df48d60d0ee9f2f5f5eaba842d0).
* This patch **fails due to an unknown error code, -9**.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] viirya commented on a change in pull request #27991:
[SPARK-31227][SQL] Non-nullable null type in complex types should not
coerce to nullable type
Posted by GitBox <gi...@apache.org>.
viirya commented on a change in pull request #27991: [SPARK-31227][SQL] Non-nullable null type in complex types should not coerce to nullable type
URL: https://github.com/apache/spark/pull/27991#discussion_r396917172
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/Cast.scala
##########
@@ -210,8 +206,15 @@ object Cast {
case _ => false // overflow
}
+ /**
+ * Returns `true` iff it should change the nullability of this type in map type,
+ * array type, expressions such as cast, etc.
+ *
+ * Note that you should take the nullability context into account.
+ * For example, it should not force to nullable type when null type in array
+ * type is non-nullable which means an empty array of null type.
+ */
Review comment:
But we use it like:
```
abstract class CastBase ... {
override def nullable: Boolean = Cast.forceNullable(child.dataType, dataType) || child.nullable
...
}
```
Will we change `nullable` for something based on `CastBase`?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a change in pull request #27991:
[SPARK-31227][SQL] Non-nullable null type in complex types should not
coerce to nullable type
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on a change in pull request #27991: [SPARK-31227][SQL] Non-nullable null type in complex types should not coerce to nullable type
URL: https://github.com/apache/spark/pull/27991#discussion_r396835748
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/Cast.scala
##########
@@ -77,7 +77,8 @@ object Cast {
resolvableNullability(fn || forceNullable(fromType, toType), tn)
case (MapType(fromKey, fromValue, fn), MapType(toKey, toValue, tn)) =>
- canCast(fromKey, toKey) && canCastMapKeyNullSafe(fromKey, toKey) &&
+ canCast(fromKey, toKey) &&
+ (!forceNullable(fromKey, toKey)) &&
Review comment:
This one was a clean revert from this line (https://github.com/apache/spark/commit/d7b97a1d0daf65710317321490a833f696a46f21#diff-258b71121d8d168e4d53cb5b6dc53ffeL81). I would keep this line as is ..
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27991: [SPARK-31227][SQL]
Non-nullable null type in complex types should not coerce to nullable type
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27991: [SPARK-31227][SQL] Non-nullable null type in complex types should not coerce to nullable type
URL: https://github.com/apache/spark/pull/27991#issuecomment-603065075
Merged build finished. Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on issue #27991: [SPARK-31227][SQL]
Non-nullable null type in complex types should not coerce to nullable type
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on issue #27991: [SPARK-31227][SQL] Non-nullable null type in complex types should not coerce to nullable type
URL: https://github.com/apache/spark/pull/27991#issuecomment-604278399
thanks, merging to master/3.0!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #27991:
[SPARK-31227][SQL] Non-nullable null type in complex types should not
coerce to nullable type
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #27991: [SPARK-31227][SQL] Non-nullable null type in complex types should not coerce to nullable type
URL: https://github.com/apache/spark/pull/27991#discussion_r397053082
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/Cast.scala
##########
@@ -210,8 +206,15 @@ object Cast {
case _ => false // overflow
}
+ /**
+ * Returns `true` iff it should change the nullability of this type in map type,
+ * array type, expressions such as cast, etc.
+ *
+ * Note that you should take the nullability context into account.
+ * For example, it should not force to nullable type when null type in array
+ * type is non-nullable which means an empty array of null type.
+ */
Review comment:
How about we define `Cast.nullable` as `child.nullable || Cast.forceNullable(child.dataType, dataType)`?
Then we can say that: when calling this method, the `from` type is not nullable. If `from == NullType`, it means empty array or map, and it does not force the result to be nullable.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27991:
[SPARK-31227][SQL] Non-nullable null type in complex types should not
coerce to nullable type
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27991: [SPARK-31227][SQL] Non-nullable null type in complex types should not coerce to nullable type
URL: https://github.com/apache/spark/pull/27991#issuecomment-603280012
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27991:
[SPARK-31227][SQL] Non-nullable null type in complex types should not
coerce to nullable type
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27991: [SPARK-31227][SQL] Non-nullable null type in complex types should not coerce to nullable type
URL: https://github.com/apache/spark/pull/27991#issuecomment-602604009
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27991: [SPARK-31227][SQL]
Non-nullable null type in complex types should not coerce to nullable type
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27991: [SPARK-31227][SQL] Non-nullable null type in complex types should not coerce to nullable type
URL: https://github.com/apache/spark/pull/27991#issuecomment-604044592
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/120360/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #27991:
[SPARK-31227][SQL] Non-nullable null type in complex types should not
coerce to nullable type
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #27991: [SPARK-31227][SQL] Non-nullable null type in complex types should not coerce to nullable type
URL: https://github.com/apache/spark/pull/27991#discussion_r397840232
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/Cast.scala
##########
@@ -210,8 +206,16 @@ object Cast {
case _ => false // overflow
}
+ /**
+ * Returns `true` iff it should change the nullability of this type in map type,
Review comment:
nit:
```
Returns true if casting non-nullable values from `from` type to `to` type may return null.
Note that the caller side should take care of input nullability first and only call this method
if the input is not nullable.
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27991: [SPARK-31227][SQL]
Non-nullable null type in complex types should not coerce to nullable type
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27991: [SPARK-31227][SQL] Non-nullable null type in complex types should not coerce to nullable type
URL: https://github.com/apache/spark/pull/27991#issuecomment-602781988
Merged build finished. Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27991: [SPARK-31227][SQL]
Non-nullable null type in complex types should not coerce to nullable type
Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #27991: [SPARK-31227][SQL] Non-nullable null type in complex types should not coerce to nullable type
URL: https://github.com/apache/spark/pull/27991#issuecomment-602766027
**[Test build #120206 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/120206/testReport)** for PR 27991 at commit [`7c54a5f`](https://github.com/apache/spark/commit/7c54a5f128c88d929d0bb9f636be487d0abd78c9).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27991: [SPARK-31227][SQL]
Non-nullable null type in complex types should not coerce to nullable type
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27991: [SPARK-31227][SQL] Non-nullable null type in complex types should not coerce to nullable type
URL: https://github.com/apache/spark/pull/27991#issuecomment-602781997
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/120209/
Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27991:
[SPARK-31227][SQL] Non-nullable null type in complex types should not
coerce to nullable type
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27991: [SPARK-31227][SQL] Non-nullable null type in complex types should not coerce to nullable type
URL: https://github.com/apache/spark/pull/27991#issuecomment-604044592
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/120360/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27991:
[SPARK-31227][SQL] Non-nullable null type in complex types should not
coerce to nullable type
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27991: [SPARK-31227][SQL] Non-nullable null type in complex types should not coerce to nullable type
URL: https://github.com/apache/spark/pull/27991#issuecomment-603115680
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27991: [SPARK-31227][SQL]
Non-nullable null type in complex types should not coerce to nullable type
Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #27991: [SPARK-31227][SQL] Non-nullable null type in complex types should not coerce to nullable type
URL: https://github.com/apache/spark/pull/27991#issuecomment-602946350
**[Test build #120226 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/120226/testReport)** for PR 27991 at commit [`dfd9343`](https://github.com/apache/spark/commit/dfd9343e803fb4d13496aaa30c41d77c7e0d0798).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27991:
[SPARK-31227][SQL] Non-nullable null type in complex types should not
coerce to nullable type
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27991: [SPARK-31227][SQL] Non-nullable null type in complex types should not coerce to nullable type
URL: https://github.com/apache/spark/pull/27991#issuecomment-603646158
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27991: [SPARK-31227][SQL]
Non-nullable null type in complex types should not coerce to nullable type
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27991: [SPARK-31227][SQL] Non-nullable null type in complex types should not coerce to nullable type
URL: https://github.com/apache/spark/pull/27991#issuecomment-603573076
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a change in pull request #27991:
[SPARK-31227][SQL] Non-nullable null type in complex types should not
coerce to nullable type
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on a change in pull request #27991: [SPARK-31227][SQL] Non-nullable null type in complex types should not coerce to nullable type
URL: https://github.com/apache/spark/pull/27991#discussion_r397628911
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/Cast.scala
##########
@@ -210,8 +206,15 @@ object Cast {
case _ => false // overflow
}
+ /**
+ * Returns `true` iff it should change the nullability of this type in map type,
+ * array type, expressions such as cast, etc.
+ *
+ * Note that you should take the nullability context into account.
+ * For example, it should not force to nullable type when null type in array
+ * type is non-nullable which means an empty array of null type.
+ */
def forceNullable(from: DataType, to: DataType): Boolean = (from, to) match {
- case (NullType, _) => true
Review comment:
Hm, I think it's just better to whitelist as are rather than blacklist. `case (_, StringType) => false` could be removed I guess but let me leave it out of this PR.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27991: [SPARK-31227][SQL]
Non-nullable null type in complex types should not coerce to nullable type
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27991: [SPARK-31227][SQL] Non-nullable null type in complex types should not coerce to nullable type
URL: https://github.com/apache/spark/pull/27991#issuecomment-602767508
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/120206/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a change in pull request #27991:
[SPARK-31227][SQL] Non-nullable null type in complex types should not
coerce to nullable type
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on a change in pull request #27991: [SPARK-31227][SQL] Non-nullable null type in complex types should not coerce to nullable type
URL: https://github.com/apache/spark/pull/27991#discussion_r397757665
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/Cast.scala
##########
@@ -210,8 +206,15 @@ object Cast {
case _ => false // overflow
}
+ /**
+ * Returns `true` iff it should change the nullability of this type in map type,
+ * array type, expressions such as cast, etc.
+ *
+ * Note that you should take the nullability context into account.
+ * For example, it should not force to nullable type when null type in array
+ * type is non-nullable which means an empty array of null type.
+ */
def forceNullable(from: DataType, to: DataType): Boolean = (from, to) match {
- case (NullType, _) => true
Review comment:
Hm .. okay
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27991:
[SPARK-31227][SQL] Non-nullable null type in complex types should not
coerce to nullable type
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27991: [SPARK-31227][SQL] Non-nullable null type in complex types should not coerce to nullable type
URL: https://github.com/apache/spark/pull/27991#issuecomment-603863833
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27991: [SPARK-31227][SQL]
Non-nullable null type in complex types should not coerce to nullable type
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27991: [SPARK-31227][SQL] Non-nullable null type in complex types should not coerce to nullable type
URL: https://github.com/apache/spark/pull/27991#issuecomment-603025078
Merged build finished. Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] maropu commented on a change in pull request #27991:
[SPARK-31227][SQL] Non-nullable null type in complex types should not
coerce to nullable type
Posted by GitBox <gi...@apache.org>.
maropu commented on a change in pull request #27991: [SPARK-31227][SQL] Non-nullable null type in complex types should not coerce to nullable type
URL: https://github.com/apache/spark/pull/27991#discussion_r397552138
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/Cast.scala
##########
@@ -210,8 +206,15 @@ object Cast {
case _ => false // overflow
}
+ /**
+ * Returns `true` iff it should change the nullability of this type in map type,
+ * array type, expressions such as cast, etc.
+ *
+ * Note that you should take the nullability context into account.
+ * For example, it should not force to nullable type when null type in array
+ * type is non-nullable which means an empty array of null type.
+ */
def forceNullable(from: DataType, to: DataType): Boolean = (from, to) match {
- case (NullType, _) => true
Review comment:
We can do this now?
```
case (NullType, _) => false // empty array or map case
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a change in pull request #27991:
[SPARK-31227][SQL] Non-nullable null type in complex types should not
coerce to nullable type
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on a change in pull request #27991: [SPARK-31227][SQL] Non-nullable null type in complex types should not coerce to nullable type
URL: https://github.com/apache/spark/pull/27991#discussion_r397716281
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/Cast.scala
##########
@@ -210,8 +206,15 @@ object Cast {
case _ => false // overflow
}
+ /**
+ * Returns `true` iff it should change the nullability of this type in map type,
+ * array type, expressions such as cast, etc.
+ *
+ * Note that you should take the nullability context into account.
+ * For example, it should not force to nullable type when null type in array
+ * type is non-nullable which means an empty array of null type.
+ */
def forceNullable(from: DataType, to: DataType): Boolean = (from, to) match {
- case (NullType, _) => true
Review comment:
I think so. I am not sure if it's right to call it problem though. We have `(_, CalendarIntervalType) => true` too. Seems intentional to force nullability regardless of `from` type?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #27991:
[SPARK-31227][SQL] Non-nullable null type in complex types should not
coerce to nullable type
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on issue #27991: [SPARK-31227][SQL] Non-nullable null type in complex types should not coerce to nullable type
URL: https://github.com/apache/spark/pull/27991#issuecomment-602579852
**[Test build #120206 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/120206/testReport)** for PR 27991 at commit [`7c54a5f`](https://github.com/apache/spark/commit/7c54a5f128c88d929d0bb9f636be487d0abd78c9).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27991:
[SPARK-31227][SQL] Non-nullable null type in complex types should not
coerce to nullable type
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27991: [SPARK-31227][SQL] Non-nullable null type in complex types should not coerce to nullable type
URL: https://github.com/apache/spark/pull/27991#issuecomment-603999817
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/120356/
Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27991: [SPARK-31227][SQL]
Non-nullable null type in complex types should not coerce to nullable type
Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #27991: [SPARK-31227][SQL] Non-nullable null type in complex types should not coerce to nullable type
URL: https://github.com/apache/spark/pull/27991#issuecomment-602579852
**[Test build #120206 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/120206/testReport)** for PR 27991 at commit [`7c54a5f`](https://github.com/apache/spark/commit/7c54a5f128c88d929d0bb9f636be487d0abd78c9).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27991: [SPARK-31227][SQL]
Non-nullable null type in complex types should not coerce to nullable type
Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #27991: [SPARK-31227][SQL] Non-nullable null type in complex types should not coerce to nullable type
URL: https://github.com/apache/spark/pull/27991#issuecomment-603862942
**[Test build #120360 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/120360/testReport)** for PR 27991 at commit [`1bf68f0`](https://github.com/apache/spark/commit/1bf68f03acce3cb46380ecbadd66693cbff39195).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27991: [SPARK-31227][SQL]
Non-nullable null type in complex types should not coerce to nullable type
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27991: [SPARK-31227][SQL] Non-nullable null type in complex types should not coerce to nullable type
URL: https://github.com/apache/spark/pull/27991#issuecomment-603064896
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/120246/
Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a change in pull request #27991:
[SPARK-31227][SQL] Non-nullable null type in complex types should not
coerce to nullable type
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on a change in pull request #27991: [SPARK-31227][SQL] Non-nullable null type in complex types should not coerce to nullable type
URL: https://github.com/apache/spark/pull/27991#discussion_r396924298
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/Cast.scala
##########
@@ -210,8 +206,15 @@ object Cast {
case _ => false // overflow
}
+ /**
+ * Returns `true` iff it should change the nullability of this type in map type,
+ * array type, expressions such as cast, etc.
+ *
+ * Note that you should take the nullability context into account.
+ * For example, it should not force to nullable type when null type in array
+ * type is non-nullable which means an empty array of null type.
+ */
Review comment:
Yes, it will also change nullability based on child's nullability. Previously, `ArrayType` of non-nullable `NullType` makes a cast `ArrayType` of nullable type always. After this change, such cast can be non-nullable based on child's nullability.
```scala
import org.apache.spark.sql.types._
import org.apache.spark.sql.functions._
spark.range(1).select(array().cast(ArrayType(IntegerType, false))).printSchema()
```
Non-nullable `NullType` should always be empty, an empty array, an empty map or an empty struct. So, it wouldn't affect the nullability in the downstream.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27991: [SPARK-31227][SQL]
Non-nullable null type in complex types should not coerce to nullable type
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27991: [SPARK-31227][SQL] Non-nullable null type in complex types should not coerce to nullable type
URL: https://github.com/apache/spark/pull/27991#issuecomment-602767496
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27991:
[SPARK-31227][SQL] Non-nullable null type in complex types should not
coerce to nullable type
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27991: [SPARK-31227][SQL] Non-nullable null type in complex types should not coerce to nullable type
URL: https://github.com/apache/spark/pull/27991#issuecomment-603863845
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/25070/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #27991:
[SPARK-31227][SQL] Non-nullable null type in complex types should not
coerce to nullable type
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #27991: [SPARK-31227][SQL] Non-nullable null type in complex types should not coerce to nullable type
URL: https://github.com/apache/spark/pull/27991#discussion_r396444990
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/Cast.scala
##########
@@ -211,7 +211,6 @@ object Cast {
}
def forceNullable(from: DataType, to: DataType): Boolean = (from, to) match {
- case (NullType, _) => true
Review comment:
and seems it replaces https://github.com/apache/spark/commit/d7b97a1d0daf65710317321490a833f696a46f21 ?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27991:
[SPARK-31227][SQL] Non-nullable null type in complex types should not
coerce to nullable type
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27991: [SPARK-31227][SQL] Non-nullable null type in complex types should not coerce to nullable type
URL: https://github.com/apache/spark/pull/27991#issuecomment-602997345
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/24948/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27991: [SPARK-31227][SQL]
Non-nullable null type in complex types should not coerce to nullable type
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27991: [SPARK-31227][SQL] Non-nullable null type in complex types should not coerce to nullable type
URL: https://github.com/apache/spark/pull/27991#issuecomment-602997340
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27991: [SPARK-31227][SQL]
Non-nullable null type in complex types should not coerce to nullable type
Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #27991: [SPARK-31227][SQL] Non-nullable null type in complex types should not coerce to nullable type
URL: https://github.com/apache/spark/pull/27991#issuecomment-603998541
**[Test build #120356 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/120356/testReport)** for PR 27991 at commit [`d57d4a6`](https://github.com/apache/spark/commit/d57d4a699b8c0836a5d7f24e6c9d960294fc59e7).
* This patch **fails PySpark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27991: [SPARK-31227][SQL]
Non-nullable null type in complex types should not coerce to nullable type
Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #27991: [SPARK-31227][SQL] Non-nullable null type in complex types should not coerce to nullable type
URL: https://github.com/apache/spark/pull/27991#issuecomment-603064797
**[Test build #120235 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/120235/testReport)** for PR 27991 at commit [`e4557ae`](https://github.com/apache/spark/commit/e4557aeaa0f25023602f2d2f042da83523342b41).
* This patch **fails due to an unknown error code, -9**.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27991: [SPARK-31227][SQL]
Non-nullable null type in complex types should not coerce to nullable type
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27991: [SPARK-31227][SQL] Non-nullable null type in complex types should not coerce to nullable type
URL: https://github.com/apache/spark/pull/27991#issuecomment-602947212
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/24939/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #27991:
[SPARK-31227][SQL] Non-nullable null type in complex types should not
coerce to nullable type
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on issue #27991: [SPARK-31227][SQL] Non-nullable null type in complex types should not coerce to nullable type
URL: https://github.com/apache/spark/pull/27991#issuecomment-603826565
**[Test build #120356 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/120356/testReport)** for PR 27991 at commit [`d57d4a6`](https://github.com/apache/spark/commit/d57d4a699b8c0836a5d7f24e6c9d960294fc59e7).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #27991:
[SPARK-31227][SQL] Non-nullable null type in complex types should not
coerce to nullable type
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on issue #27991: [SPARK-31227][SQL] Non-nullable null type in complex types should not coerce to nullable type
URL: https://github.com/apache/spark/pull/27991#issuecomment-603862942
**[Test build #120360 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/120360/testReport)** for PR 27991 at commit [`1bf68f0`](https://github.com/apache/spark/commit/1bf68f03acce3cb46380ecbadd66693cbff39195).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27991: [SPARK-31227][SQL]
Non-nullable null type in complex types should not coerce to nullable type
Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #27991: [SPARK-31227][SQL] Non-nullable null type in complex types should not coerce to nullable type
URL: https://github.com/apache/spark/pull/27991#issuecomment-603278623
**[Test build #120261 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/120261/testReport)** for PR 27991 at commit [`7bae2c1`](https://github.com/apache/spark/commit/7bae2c1c81ca8df48d60d0ee9f2f5f5eaba842d0).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27991: [SPARK-31227][SQL]
Non-nullable null type in complex types should not coerce to nullable type
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27991: [SPARK-31227][SQL] Non-nullable null type in complex types should not coerce to nullable type
URL: https://github.com/apache/spark/pull/27991#issuecomment-603045385
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27991: [SPARK-31227][SQL]
Non-nullable null type in complex types should not coerce to nullable type
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27991: [SPARK-31227][SQL] Non-nullable null type in complex types should not coerce to nullable type
URL: https://github.com/apache/spark/pull/27991#issuecomment-602580530
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27991: [SPARK-31227][SQL]
Non-nullable null type in complex types should not coerce to nullable type
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27991: [SPARK-31227][SQL] Non-nullable null type in complex types should not coerce to nullable type
URL: https://github.com/apache/spark/pull/27991#issuecomment-602947203
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27991: [SPARK-31227][SQL]
Non-nullable null type in complex types should not coerce to nullable type
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27991: [SPARK-31227][SQL] Non-nullable null type in complex types should not coerce to nullable type
URL: https://github.com/apache/spark/pull/27991#issuecomment-603999817
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/120356/
Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #27991:
[SPARK-31227][SQL] Non-nullable null type in complex types should not
coerce to nullable type
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on issue #27991: [SPARK-31227][SQL] Non-nullable null type in complex types should not coerce to nullable type
URL: https://github.com/apache/spark/pull/27991#issuecomment-603044996
**[Test build #120246 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/120246/testReport)** for PR 27991 at commit [`7bae2c1`](https://github.com/apache/spark/commit/7bae2c1c81ca8df48d60d0ee9f2f5f5eaba842d0).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #27991:
[SPARK-31227][SQL] Non-nullable null type in complex types should not
coerce to nullable type
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #27991: [SPARK-31227][SQL] Non-nullable null type in complex types should not coerce to nullable type
URL: https://github.com/apache/spark/pull/27991#discussion_r396437607
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/Cast.scala
##########
@@ -211,7 +211,6 @@ object Cast {
}
def forceNullable(from: DataType, to: DataType): Boolean = (from, to) match {
- case (NullType, _) => true
Review comment:
is it safe? What if we cast `array(null)` to `array<int>`?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27991: [SPARK-31227][SQL]
Non-nullable null type in complex types should not coerce to nullable type
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27991: [SPARK-31227][SQL] Non-nullable null type in complex types should not coerce to nullable type
URL: https://github.com/apache/spark/pull/27991#issuecomment-603280012
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a change in pull request #27991:
[SPARK-31227][SQL] Non-nullable null type in complex types should not
coerce to nullable type
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on a change in pull request #27991: [SPARK-31227][SQL] Non-nullable null type in complex types should not coerce to nullable type
URL: https://github.com/apache/spark/pull/27991#discussion_r396450062
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/Cast.scala
##########
@@ -211,7 +211,6 @@ object Cast {
}
def forceNullable(from: DataType, to: DataType): Boolean = (from, to) match {
- case (NullType, _) => true
Review comment:
Yeah .. let me remove that change here as well. Let me add some documentation there as well.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27991:
[SPARK-31227][SQL] Non-nullable null type in complex types should not
coerce to nullable type
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27991: [SPARK-31227][SQL] Non-nullable null type in complex types should not coerce to nullable type
URL: https://github.com/apache/spark/pull/27991#issuecomment-603999812
Merged build finished. Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27991: [SPARK-31227][SQL]
Non-nullable null type in complex types should not coerce to nullable type
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27991: [SPARK-31227][SQL] Non-nullable null type in complex types should not coerce to nullable type
URL: https://github.com/apache/spark/pull/27991#issuecomment-603065083
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/120235/
Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #27991:
[SPARK-31227][SQL] Non-nullable null type in complex types should not
coerce to nullable type
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on issue #27991: [SPARK-31227][SQL] Non-nullable null type in complex types should not coerce to nullable type
URL: https://github.com/apache/spark/pull/27991#issuecomment-602602220
**[Test build #120209 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/120209/testReport)** for PR 27991 at commit [`51acfeb`](https://github.com/apache/spark/commit/51acfebc45aee180659b6e58d18d5833be2d77ae).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27991:
[SPARK-31227][SQL] Non-nullable null type in complex types should not
coerce to nullable type
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27991: [SPARK-31227][SQL] Non-nullable null type in complex types should not coerce to nullable type
URL: https://github.com/apache/spark/pull/27991#issuecomment-602604035
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/24922/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27991:
[SPARK-31227][SQL] Non-nullable null type in complex types should not
coerce to nullable type
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27991: [SPARK-31227][SQL] Non-nullable null type in complex types should not coerce to nullable type
URL: https://github.com/apache/spark/pull/27991#issuecomment-602767496
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan closed pull request #27991: [SPARK-31227][SQL]
Non-nullable null type in complex types should not coerce to nullable type
Posted by GitBox <gi...@apache.org>.
cloud-fan closed pull request #27991: [SPARK-31227][SQL] Non-nullable null type in complex types should not coerce to nullable type
URL: https://github.com/apache/spark/pull/27991
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27991: [SPARK-31227][SQL]
Non-nullable null type in complex types should not coerce to nullable type
Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #27991: [SPARK-31227][SQL] Non-nullable null type in complex types should not coerce to nullable type
URL: https://github.com/apache/spark/pull/27991#issuecomment-602781015
**[Test build #120209 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/120209/testReport)** for PR 27991 at commit [`51acfeb`](https://github.com/apache/spark/commit/51acfebc45aee180659b6e58d18d5833be2d77ae).
* This patch **fails PySpark pip packaging tests**.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27991: [SPARK-31227][SQL]
Non-nullable null type in complex types should not coerce to nullable type
Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #27991: [SPARK-31227][SQL] Non-nullable null type in complex types should not coerce to nullable type
URL: https://github.com/apache/spark/pull/27991#issuecomment-603024461
**[Test build #120226 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/120226/testReport)** for PR 27991 at commit [`dfd9343`](https://github.com/apache/spark/commit/dfd9343e803fb4d13496aaa30c41d77c7e0d0798).
* This patch **fails PySpark pip packaging tests**.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27991: [SPARK-31227][SQL]
Non-nullable null type in complex types should not coerce to nullable type
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27991: [SPARK-31227][SQL] Non-nullable null type in complex types should not coerce to nullable type
URL: https://github.com/apache/spark/pull/27991#issuecomment-603280027
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/120261/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27991: [SPARK-31227][SQL]
Non-nullable null type in complex types should not coerce to nullable type
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27991: [SPARK-31227][SQL] Non-nullable null type in complex types should not coerce to nullable type
URL: https://github.com/apache/spark/pull/27991#issuecomment-603863845
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/25070/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27991: [SPARK-31227][SQL]
Non-nullable null type in complex types should not coerce to nullable type
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27991: [SPARK-31227][SQL] Non-nullable null type in complex types should not coerce to nullable type
URL: https://github.com/apache/spark/pull/27991#issuecomment-603863833
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27991:
[SPARK-31227][SQL] Non-nullable null type in complex types should not
coerce to nullable type
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27991: [SPARK-31227][SQL] Non-nullable null type in complex types should not coerce to nullable type
URL: https://github.com/apache/spark/pull/27991#issuecomment-602781988
Merged build finished. Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27991: [SPARK-31227][SQL]
Non-nullable null type in complex types should not coerce to nullable type
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27991: [SPARK-31227][SQL] Non-nullable null type in complex types should not coerce to nullable type
URL: https://github.com/apache/spark/pull/27991#issuecomment-603064884
Merged build finished. Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27991:
[SPARK-31227][SQL] Non-nullable null type in complex types should not
coerce to nullable type
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27991: [SPARK-31227][SQL] Non-nullable null type in complex types should not coerce to nullable type
URL: https://github.com/apache/spark/pull/27991#issuecomment-603045385
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27991: [SPARK-31227][SQL]
Non-nullable null type in complex types should not coerce to nullable type
Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #27991: [SPARK-31227][SQL] Non-nullable null type in complex types should not coerce to nullable type
URL: https://github.com/apache/spark/pull/27991#issuecomment-603118707
**[Test build #120261 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/120261/testReport)** for PR 27991 at commit [`7bae2c1`](https://github.com/apache/spark/commit/7bae2c1c81ca8df48d60d0ee9f2f5f5eaba842d0).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27991: [SPARK-31227][SQL]
Non-nullable null type in complex types should not coerce to nullable type
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27991: [SPARK-31227][SQL] Non-nullable null type in complex types should not coerce to nullable type
URL: https://github.com/apache/spark/pull/27991#issuecomment-603115693
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/24972/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #27991:
[SPARK-31227][SQL] Non-nullable null type in complex types should not
coerce to nullable type
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #27991: [SPARK-31227][SQL] Non-nullable null type in complex types should not coerce to nullable type
URL: https://github.com/apache/spark/pull/27991#discussion_r397753562
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/Cast.scala
##########
@@ -210,8 +206,15 @@ object Cast {
case _ => false // overflow
}
+ /**
+ * Returns `true` iff it should change the nullability of this type in map type,
+ * array type, expressions such as cast, etc.
+ *
+ * Note that you should take the nullability context into account.
+ * For example, it should not force to nullable type when null type in array
+ * type is non-nullable which means an empty array of null type.
+ */
def forceNullable(from: DataType, to: DataType): Boolean = (from, to) match {
- case (NullType, _) => true
Review comment:
so adding a `case (NullType, _) => false` at the beginning should be good to go?
I feel it's weird if `concat(array(), array(1))` is allowed but `concat(array(), array(date'xxx'))` is not.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27991:
[SPARK-31227][SQL] Non-nullable null type in complex types should not
coerce to nullable type
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27991: [SPARK-31227][SQL] Non-nullable null type in complex types should not coerce to nullable type
URL: https://github.com/apache/spark/pull/27991#issuecomment-603573082
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/25002/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27991: [SPARK-31227][SQL]
Non-nullable null type in complex types should not coerce to nullable type
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27991: [SPARK-31227][SQL] Non-nullable null type in complex types should not coerce to nullable type
URL: https://github.com/apache/spark/pull/27991#issuecomment-603573082
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/25002/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27991: [SPARK-31227][SQL]
Non-nullable null type in complex types should not coerce to nullable type
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27991: [SPARK-31227][SQL] Non-nullable null type in complex types should not coerce to nullable type
URL: https://github.com/apache/spark/pull/27991#issuecomment-603646163
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/120292/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27991: [SPARK-31227][SQL]
Non-nullable null type in complex types should not coerce to nullable type
Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #27991: [SPARK-31227][SQL] Non-nullable null type in complex types should not coerce to nullable type
URL: https://github.com/apache/spark/pull/27991#issuecomment-603044996
**[Test build #120246 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/120246/testReport)** for PR 27991 at commit [`7bae2c1`](https://github.com/apache/spark/commit/7bae2c1c81ca8df48d60d0ee9f2f5f5eaba842d0).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27991:
[SPARK-31227][SQL] Non-nullable null type in complex types should not
coerce to nullable type
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27991: [SPARK-31227][SQL] Non-nullable null type in complex types should not coerce to nullable type
URL: https://github.com/apache/spark/pull/27991#issuecomment-603025078
Merged build finished. Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27991: [SPARK-31227][SQL]
Non-nullable null type in complex types should not coerce to nullable type
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27991: [SPARK-31227][SQL] Non-nullable null type in complex types should not coerce to nullable type
URL: https://github.com/apache/spark/pull/27991#issuecomment-603045397
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/24959/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27991:
[SPARK-31227][SQL] Non-nullable null type in complex types should not
coerce to nullable type
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27991: [SPARK-31227][SQL] Non-nullable null type in complex types should not coerce to nullable type
URL: https://github.com/apache/spark/pull/27991#issuecomment-603573076
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27991:
[SPARK-31227][SQL] Non-nullable null type in complex types should not
coerce to nullable type
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27991: [SPARK-31227][SQL] Non-nullable null type in complex types should not coerce to nullable type
URL: https://github.com/apache/spark/pull/27991#issuecomment-603064896
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/120246/
Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #27991:
[SPARK-31227][SQL] Non-nullable null type in complex types should not
coerce to nullable type
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on issue #27991: [SPARK-31227][SQL] Non-nullable null type in complex types should not coerce to nullable type
URL: https://github.com/apache/spark/pull/27991#issuecomment-603572691
**[Test build #120292 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/120292/testReport)** for PR 27991 at commit [`05dc916`](https://github.com/apache/spark/commit/05dc91695139b1e5f5c94eb117ba34c76bd3a3ff).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27991:
[SPARK-31227][SQL] Non-nullable null type in complex types should not
coerce to nullable type
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27991: [SPARK-31227][SQL] Non-nullable null type in complex types should not coerce to nullable type
URL: https://github.com/apache/spark/pull/27991#issuecomment-603064884
Merged build finished. Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27991: [SPARK-31227][SQL]
Non-nullable null type in complex types should not coerce to nullable type
Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #27991: [SPARK-31227][SQL] Non-nullable null type in complex types should not coerce to nullable type
URL: https://github.com/apache/spark/pull/27991#issuecomment-603572691
**[Test build #120292 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/120292/testReport)** for PR 27991 at commit [`05dc916`](https://github.com/apache/spark/commit/05dc91695139b1e5f5c94eb117ba34c76bd3a3ff).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27991:
[SPARK-31227][SQL] Non-nullable null type in complex types should not
coerce to nullable type
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27991: [SPARK-31227][SQL] Non-nullable null type in complex types should not coerce to nullable type
URL: https://github.com/apache/spark/pull/27991#issuecomment-603115693
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/24972/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27991: [SPARK-31227][SQL]
Non-nullable null type in complex types should not coerce to nullable type
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27991: [SPARK-31227][SQL] Non-nullable null type in complex types should not coerce to nullable type
URL: https://github.com/apache/spark/pull/27991#issuecomment-603025080
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/120226/
Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27991: [SPARK-31227][SQL]
Non-nullable null type in complex types should not coerce to nullable type
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27991: [SPARK-31227][SQL] Non-nullable null type in complex types should not coerce to nullable type
URL: https://github.com/apache/spark/pull/27991#issuecomment-603827062
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/25066/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27991: [SPARK-31227][SQL]
Non-nullable null type in complex types should not coerce to nullable type
Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #27991: [SPARK-31227][SQL] Non-nullable null type in complex types should not coerce to nullable type
URL: https://github.com/apache/spark/pull/27991#issuecomment-604043465
**[Test build #120360 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/120360/testReport)** for PR 27991 at commit [`1bf68f0`](https://github.com/apache/spark/commit/1bf68f03acce3cb46380ecbadd66693cbff39195).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27991:
[SPARK-31227][SQL] Non-nullable null type in complex types should not
coerce to nullable type
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27991: [SPARK-31227][SQL] Non-nullable null type in complex types should not coerce to nullable type
URL: https://github.com/apache/spark/pull/27991#issuecomment-603065075
Merged build finished. Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a change in pull request #27991:
[SPARK-31227][SQL] Non-nullable null type in complex types should not
coerce to nullable type
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on a change in pull request #27991: [SPARK-31227][SQL] Non-nullable null type in complex types should not coerce to nullable type
URL: https://github.com/apache/spark/pull/27991#discussion_r396438243
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/Cast.scala
##########
@@ -211,7 +211,6 @@ object Cast {
}
def forceNullable(from: DataType, to: DataType): Boolean = (from, to) match {
- case (NullType, _) => true
Review comment:
`ArrayType(NullType, containsNull=False)` can't contain `null`, no?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27991:
[SPARK-31227][SQL] Non-nullable null type in complex types should not
coerce to nullable type
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27991: [SPARK-31227][SQL] Non-nullable null type in complex types should not coerce to nullable type
URL: https://github.com/apache/spark/pull/27991#issuecomment-602997340
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #27991:
[SPARK-31227][SQL] Non-nullable null type in complex types should not
coerce to nullable type
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #27991: [SPARK-31227][SQL] Non-nullable null type in complex types should not coerce to nullable type
URL: https://github.com/apache/spark/pull/27991#discussion_r396883635
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/Cast.scala
##########
@@ -210,8 +206,15 @@ object Cast {
case _ => false // overflow
}
+ /**
+ * Returns `true` iff it should change the nullability of this type in map type,
+ * array type, expression, etc.
+ *
+ * Note that this function should take the nullability context into account.
Review comment:
`this function` ->`caller side`?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a change in pull request #27991:
[SPARK-31227][SQL] Non-nullable null type in complex types should not
coerce to nullable type
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on a change in pull request #27991: [SPARK-31227][SQL] Non-nullable null type in complex types should not coerce to nullable type
URL: https://github.com/apache/spark/pull/27991#discussion_r397538044
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/Cast.scala
##########
@@ -210,8 +206,15 @@ object Cast {
case _ => false // overflow
}
+ /**
+ * Returns `true` iff it should change the nullability of this type in map type,
+ * array type, expressions such as cast, etc.
+ *
+ * Note that you should take the nullability context into account.
+ * For example, it should not force to nullable type when null type in array
+ * type is non-nullable which means an empty array of null type.
+ */
Review comment:
Sure.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a change in pull request #27991:
[SPARK-31227][SQL] Non-nullable null type in complex types should not
coerce to nullable type
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on a change in pull request #27991: [SPARK-31227][SQL] Non-nullable null type in complex types should not coerce to nullable type
URL: https://github.com/apache/spark/pull/27991#discussion_r397716281
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/Cast.scala
##########
@@ -210,8 +206,15 @@ object Cast {
case _ => false // overflow
}
+ /**
+ * Returns `true` iff it should change the nullability of this type in map type,
+ * array type, expressions such as cast, etc.
+ *
+ * Note that you should take the nullability context into account.
+ * For example, it should not force to nullable type when null type in array
+ * type is non-nullable which means an empty array of null type.
+ */
def forceNullable(from: DataType, to: DataType): Boolean = (from, to) match {
- case (NullType, _) => true
Review comment:
I think so. I am not sure if it's right to call it problem though. We have `(_, CalendarIntervalType) => true` too
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27991:
[SPARK-31227][SQL] Non-nullable null type in complex types should not
coerce to nullable type
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27991: [SPARK-31227][SQL] Non-nullable null type in complex types should not coerce to nullable type
URL: https://github.com/apache/spark/pull/27991#issuecomment-603025080
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/120226/
Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27991: [SPARK-31227][SQL]
Non-nullable null type in complex types should not coerce to nullable type
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27991: [SPARK-31227][SQL] Non-nullable null type in complex types should not coerce to nullable type
URL: https://github.com/apache/spark/pull/27991#issuecomment-602997345
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/24948/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #27991:
[SPARK-31227][SQL] Non-nullable null type in complex types should not
coerce to nullable type
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #27991: [SPARK-31227][SQL] Non-nullable null type in complex types should not coerce to nullable type
URL: https://github.com/apache/spark/pull/27991#discussion_r397683708
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/Cast.scala
##########
@@ -210,8 +206,15 @@ object Cast {
case _ => false // overflow
}
+ /**
+ * Returns `true` iff it should change the nullability of this type in map type,
+ * array type, expressions such as cast, etc.
+ *
+ * Note that you should take the nullability context into account.
+ * For example, it should not force to nullable type when null type in array
+ * type is non-nullable which means an empty array of null type.
+ */
def forceNullable(from: DataType, to: DataType): Boolean = (from, to) match {
- case (NullType, _) => true
Review comment:
We have `case (_, DateType) => true`, so the problem still exists for `concat(array(), array(date'xxx'))`?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27991:
[SPARK-31227][SQL] Non-nullable null type in complex types should not
coerce to nullable type
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27991: [SPARK-31227][SQL] Non-nullable null type in complex types should not coerce to nullable type
URL: https://github.com/apache/spark/pull/27991#issuecomment-603065083
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/120235/
Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #27991:
[SPARK-31227][SQL] Non-nullable null type in complex types should not
coerce to nullable type
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on issue #27991: [SPARK-31227][SQL] Non-nullable null type in complex types should not coerce to nullable type
URL: https://github.com/apache/spark/pull/27991#issuecomment-603118707
**[Test build #120261 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/120261/testReport)** for PR 27991 at commit [`7bae2c1`](https://github.com/apache/spark/commit/7bae2c1c81ca8df48d60d0ee9f2f5f5eaba842d0).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27991: [SPARK-31227][SQL]
Non-nullable null type in complex types should not coerce to nullable type
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27991: [SPARK-31227][SQL] Non-nullable null type in complex types should not coerce to nullable type
URL: https://github.com/apache/spark/pull/27991#issuecomment-603827052
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] maropu commented on a change in pull request #27991:
[SPARK-31227][SQL] Non-nullable null type in complex types should not
coerce to nullable type
Posted by GitBox <gi...@apache.org>.
maropu commented on a change in pull request #27991: [SPARK-31227][SQL] Non-nullable null type in complex types should not coerce to nullable type
URL: https://github.com/apache/spark/pull/27991#discussion_r396820238
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/Cast.scala
##########
@@ -210,8 +206,13 @@ object Cast {
case _ => false // overflow
}
+ /**
+ * Returns `true` iif it should change the nullability of this type in the container,
+ * e.g. map type, array type, expression, etc. This function should take the nullability
+ * in the container into account. For example, it should not force to a nullable type
+ * when null type is non-nullable, which means an empty array of null type.
Review comment:
nit: `iif` -> `iff`, and `container` -> `complex type` for consistency?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27991: [SPARK-31227][SQL]
Non-nullable null type in complex types should not coerce to nullable type
Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #27991: [SPARK-31227][SQL] Non-nullable null type in complex types should not coerce to nullable type
URL: https://github.com/apache/spark/pull/27991#issuecomment-603645528
**[Test build #120292 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/120292/testReport)** for PR 27991 at commit [`05dc916`](https://github.com/apache/spark/commit/05dc91695139b1e5f5c94eb117ba34c76bd3a3ff).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27991:
[SPARK-31227][SQL] Non-nullable null type in complex types should not
coerce to nullable type
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27991: [SPARK-31227][SQL] Non-nullable null type in complex types should not coerce to nullable type
URL: https://github.com/apache/spark/pull/27991#issuecomment-602947212
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/24939/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27991:
[SPARK-31227][SQL] Non-nullable null type in complex types should not
coerce to nullable type
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27991: [SPARK-31227][SQL] Non-nullable null type in complex types should not coerce to nullable type
URL: https://github.com/apache/spark/pull/27991#issuecomment-603827052
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #27991:
[SPARK-31227][SQL] Non-nullable null type in complex types should not
coerce to nullable type
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on issue #27991: [SPARK-31227][SQL] Non-nullable null type in complex types should not coerce to nullable type
URL: https://github.com/apache/spark/pull/27991#issuecomment-602997004
**[Test build #120235 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/120235/testReport)** for PR 27991 at commit [`e4557ae`](https://github.com/apache/spark/commit/e4557aeaa0f25023602f2d2f042da83523342b41).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a change in pull request #27991:
[SPARK-31227][SQL] Non-nullable null type in complex types should not
coerce to nullable type
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on a change in pull request #27991: [SPARK-31227][SQL] Non-nullable null type in complex types should not coerce to nullable type
URL: https://github.com/apache/spark/pull/27991#discussion_r396846466
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/Cast.scala
##########
@@ -210,8 +206,13 @@ object Cast {
case _ => false // overflow
}
+ /**
+ * Returns `true` iif it should change the nullability of this type in the container,
+ * e.g. map type, array type, expression, etc. This function should take the nullability
+ * in the container into account. For example, it should not force to a nullable type
+ * when null type is non-nullable, which means an empty array of null type.
Review comment:
Actually, I used the term `container` intentionally. There's one more place that's used with expressions' nullability at `Cast`.
```scala
override def nullable: Boolean = Cast.forceNullable(child.dataType, dataType) || child.nullable
```
Let me try to make it better a bit while I am here.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27991: [SPARK-31227][SQL]
Non-nullable null type in complex types should not coerce to nullable type
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27991: [SPARK-31227][SQL] Non-nullable null type in complex types should not coerce to nullable type
URL: https://github.com/apache/spark/pull/27991#issuecomment-602604009
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27991:
[SPARK-31227][SQL] Non-nullable null type in complex types should not
coerce to nullable type
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27991: [SPARK-31227][SQL] Non-nullable null type in complex types should not coerce to nullable type
URL: https://github.com/apache/spark/pull/27991#issuecomment-602580530
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #27991:
[SPARK-31227][SQL] Non-nullable null type in complex types should not
coerce to nullable type
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #27991: [SPARK-31227][SQL] Non-nullable null type in complex types should not coerce to nullable type
URL: https://github.com/apache/spark/pull/27991#discussion_r396444429
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/Cast.scala
##########
@@ -211,7 +211,6 @@ object Cast {
}
def forceNullable(from: DataType, to: DataType): Boolean = (from, to) match {
- case (NullType, _) => true
Review comment:
Maybe we should clearly define what `forceNullable` should return.
From the name, seems like we should return true if `from = NullType`.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on issue #27991: [SPARK-31227][SQL]
Non-nullable null type in complex types should not coerce to nullable type
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on issue #27991: [SPARK-31227][SQL] Non-nullable null type in complex types should not coerce to nullable type
URL: https://github.com/apache/spark/pull/27991#issuecomment-603115247
retest this please
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27991:
[SPARK-31227][SQL] Non-nullable null type in complex types should not
coerce to nullable type
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27991: [SPARK-31227][SQL] Non-nullable null type in complex types should not coerce to nullable type
URL: https://github.com/apache/spark/pull/27991#issuecomment-603280027
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/120261/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27991: [SPARK-31227][SQL]
Non-nullable null type in complex types should not coerce to nullable type
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27991: [SPARK-31227][SQL] Non-nullable null type in complex types should not coerce to nullable type
URL: https://github.com/apache/spark/pull/27991#issuecomment-602604035
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/24922/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #27991:
[SPARK-31227][SQL] Non-nullable null type in complex types should not
coerce to nullable type
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #27991: [SPARK-31227][SQL] Non-nullable null type in complex types should not coerce to nullable type
URL: https://github.com/apache/spark/pull/27991#discussion_r397753562
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/Cast.scala
##########
@@ -210,8 +206,15 @@ object Cast {
case _ => false // overflow
}
+ /**
+ * Returns `true` iff it should change the nullability of this type in map type,
+ * array type, expressions such as cast, etc.
+ *
+ * Note that you should take the nullability context into account.
+ * For example, it should not force to nullable type when null type in array
+ * type is non-nullable which means an empty array of null type.
+ */
def forceNullable(from: DataType, to: DataType): Boolean = (from, to) match {
- case (NullType, _) => true
Review comment:
so adding a `case (NullType, _) => false` should be good to go?
I feel it's weird if `concat(array(), array(1))` is allowed but `concat(array(), array(date'xxx'))` is not.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27991: [SPARK-31227][SQL]
Non-nullable null type in complex types should not coerce to nullable type
Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #27991: [SPARK-31227][SQL] Non-nullable null type in complex types should not coerce to nullable type
URL: https://github.com/apache/spark/pull/27991#issuecomment-603826565
**[Test build #120356 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/120356/testReport)** for PR 27991 at commit [`d57d4a6`](https://github.com/apache/spark/commit/d57d4a699b8c0836a5d7f24e6c9d960294fc59e7).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27991: [SPARK-31227][SQL]
Non-nullable null type in complex types should not coerce to nullable type
Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #27991: [SPARK-31227][SQL] Non-nullable null type in complex types should not coerce to nullable type
URL: https://github.com/apache/spark/pull/27991#issuecomment-602997004
**[Test build #120235 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/120235/testReport)** for PR 27991 at commit [`e4557ae`](https://github.com/apache/spark/commit/e4557aeaa0f25023602f2d2f042da83523342b41).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] maropu commented on a change in pull request #27991:
[SPARK-31227][SQL] Non-nullable null type in complex types should not
coerce to nullable type
Posted by GitBox <gi...@apache.org>.
maropu commented on a change in pull request #27991: [SPARK-31227][SQL] Non-nullable null type in complex types should not coerce to nullable type
URL: https://github.com/apache/spark/pull/27991#discussion_r396820238
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/Cast.scala
##########
@@ -210,8 +206,13 @@ object Cast {
case _ => false // overflow
}
+ /**
+ * Returns `true` iif it should change the nullability of this type in the container,
+ * e.g. map type, array type, expression, etc. This function should take the nullability
+ * in the container into account. For example, it should not force to a nullable type
+ * when null type is non-nullable, which means an empty array of null type.
Review comment:
nit: `iif` -> `iff`
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27991: [SPARK-31227][SQL]
Non-nullable null type in complex types should not coerce to nullable type
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27991: [SPARK-31227][SQL] Non-nullable null type in complex types should not coerce to nullable type
URL: https://github.com/apache/spark/pull/27991#issuecomment-604044578
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org