You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by viirya <gi...@git.apache.org> on 2015/12/19 13:58:20 UTC
[GitHub] spark pull request: [SPARK-12439][SQL] Fix toCatalystArray and Map...
GitHub user viirya opened a pull request:
https://github.com/apache/spark/pull/10391
[SPARK-12439][SQL] Fix toCatalystArray and MapObjects
JIRA: https://issues.apache.org/jira/browse/SPARK-12439
In toCatalystArray, we should look at the data type returned by dataTypeFor instead of silentSchemaFor, to determine if the element is native type. An obvious problem is when the element is Option[Int] class, catalsilentSchemaFor will return Int, then we will wrongly recognize the element is native type.
There is another problem when using Option as array element. When we encode data like Seq(Some(1), Some(2), None) with encoder, we will use MapObjects to construct an array for it later. But in MapObjects, we don't check if the return value of lambdaFunction is null or not. That causes a bug that the decoded data for Seq(Some(1), Some(2), None) would be Seq(1, 2, -1), instead of Seq(1, 2, null).
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/viirya/spark-1 fix-catalystarray
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/10391.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #10391
----
commit 788a7c6b2156d43be9c389d6a67b70e8ff9bbbb2
Author: Liang-Chi Hsieh <vi...@gmail.com>
Date: 2015-12-19T12:45:48Z
Fix toCatalystArray and MapObjects.
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12439][SQL] Fix toCatalystArray and Map...
Posted by viirya <gi...@git.apache.org>.
Github user viirya commented on a diff in the pull request:
https://github.com/apache/spark/pull/10391#discussion_r48512443
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/objects.scala ---
@@ -439,6 +439,13 @@ case class MapObjects(
s"boolean ${loopVar.isNull} = ${genInputData.isNull} || ${loopVar.value} == null;"
}
+ // If lambdaFunction is WrapOption, we will not determine null or not based on the
+ // value of loopVar.isNull, because WrapOption will return None for null.
+ val isWrapOption = lambdaFunction match {
--- End diff --
Let me explain it.
When we pass in an array with None. It will be encoded as null internally. When we decode it back, WrapOption is called to re-construct it.
The logic of MapObjects is to assign an element as null if its given input element is null. So It will not actually go into WrapOption to re-construct a None back. In order to do that, we need to call lambdaFunction even the element is null.
But we can't simply ignore loopVar.isNull and call all kinds of lambdaFunctions. I tried before but for some lambdaFunctions, a null input value causes problematic results.
In the end I can only check if lambdaFunction is WrapOption or not to make the decision here. Do you have other suggestion other than a hack like this here? Thanks.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12439][SQL] Fix toCatalystArray and Map...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/10391#issuecomment-166632140
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/48197/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12439][SQL] Fix toCatalystArray and Map...
Posted by viirya <gi...@git.apache.org>.
Github user viirya commented on a diff in the pull request:
https://github.com/apache/spark/pull/10391#discussion_r48314187
--- Diff: sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/encoders/ExpressionEncoderSuite.scala ---
@@ -239,6 +240,30 @@ class ExpressionEncoderSuite extends SparkFunSuite {
ExpressionEncoder.tuple(intEnc, ExpressionEncoder.tuple(intEnc, longEnc))
}
+ test("Option in Array") {
+ val data = (Seq(Some(1), Some(2), None), "Option in array")
+
+ val schema = ScalaReflection.schemaFor[Tuple2[Seq[Option[Int]], String]]
+ .dataType.asInstanceOf[StructType]
+ val attributeSeq = schema.toAttributes
+ val dataEncoder = encoderFor[Tuple2[Seq[Option[Int]], String]]
+ val boundEncoder = dataEncoder.resolve(attributeSeq, outers).bind(attributeSeq)
+ assert(boundEncoder.fromRow(boundEncoder.toRow(data)) === (
+ Seq(Some(1), Some(2), null), "Option in array"))
+ }
+
+ test("Option in Map") {
+ val data = (Map(1 -> Some(10L), 2 -> Some(20L), 3 -> None), "Option in map")
+
+ val schema = ScalaReflection.schemaFor[Tuple2[Map[Int, Option[Long]], String]]
+ .dataType.asInstanceOf[StructType]
+ val attributeSeq = schema.toAttributes
+ val dataEncoder = encoderFor[Tuple2[Map[Int, Option[Long]], String]]
+ val boundEncoder = dataEncoder.resolve(attributeSeq, outers).bind(attributeSeq)
+ assert(boundEncoder.fromRow(boundEncoder.toRow(data)) === (
+ Map(1 -> Some(10L), 2 -> Some(20L), 3 -> null), "Option in map"))
+ }
--- End diff --
OK. I am investigating the now.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12439][SQL] Fix toCatalystArray and Map...
Posted by viirya <gi...@git.apache.org>.
Github user viirya commented on the pull request:
https://github.com/apache/spark/pull/10391#issuecomment-166925236
retest this please.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12439][SQL] Fix toCatalystArray and Map...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/10391#issuecomment-168128000
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/48538/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12439][SQL] Fix toCatalystArray and Map...
Posted by viirya <gi...@git.apache.org>.
Github user viirya commented on a diff in the pull request:
https://github.com/apache/spark/pull/10391#discussion_r48527205
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/ScalaReflection.scala ---
@@ -406,11 +406,14 @@ object ScalaReflection extends ScalaReflection {
def toCatalystArray(input: Expression, elementType: `Type`): Expression = {
val externalDataType = dataTypeFor(elementType)
val Schema(catalystType, nullable) = silentSchemaFor(elementType)
- if (isNativeType(catalystType)) {
- NewInstance(
- classOf[GenericArrayData],
- input :: Nil,
- dataType = ArrayType(catalystType, nullable))
+ if (isNativeType(externalDataType)) {
+ expressions.If(
+ IsNull(input),
+ expressions.Literal.create(null, ArrayType(catalystType, nullable)),
+ NewInstance(
+ classOf[GenericArrayData],
+ input :: Nil,
+ dataType = ArrayType(catalystType, nullable)))
--- End diff --
Waiting #10443 to getting merged too.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12439][SQL] Fix toCatalystArray and Map...
Posted by viirya <gi...@git.apache.org>.
Github user viirya commented on the pull request:
https://github.com/apache/spark/pull/10391#issuecomment-168143815
@marmbrus I've rebased this for updates. Please take a look if it is ok for you now. Thanks.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12439][SQL] Fix toCatalystArray and Map...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/10391#issuecomment-166790392
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/48225/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12439][SQL] Fix toCatalystArray and Map...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/spark/pull/10391
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12439][SQL] Fix toCatalystArray and Map...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/10391#issuecomment-166652643
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12439][SQL] Fix toCatalystArray and Map...
Posted by viirya <gi...@git.apache.org>.
Github user viirya commented on the pull request:
https://github.com/apache/spark/pull/10391#issuecomment-167931285
@marmbrus @cloud-fan If this patch is no problem. Can you merge #10443 first so I can rebase some codes here? Thanks.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12439][SQL] Fix toCatalystArray and Map...
Posted by viirya <gi...@git.apache.org>.
Github user viirya commented on a diff in the pull request:
https://github.com/apache/spark/pull/10391#discussion_r48267113
--- Diff: sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/encoders/ExpressionEncoderSuite.scala ---
@@ -239,6 +240,30 @@ class ExpressionEncoderSuite extends SparkFunSuite {
ExpressionEncoder.tuple(intEnc, ExpressionEncoder.tuple(intEnc, longEnc))
}
+ test("Option in Array") {
+ val data = (Seq(Some(1), Some(2), None), "Option in array")
+
+ val schema = ScalaReflection.schemaFor[Tuple2[Seq[Option[Int]], String]]
+ .dataType.asInstanceOf[StructType]
+ val attributeSeq = schema.toAttributes
+ val dataEncoder = encoderFor[Tuple2[Seq[Option[Int]], String]]
+ val boundEncoder = dataEncoder.resolve(attributeSeq, outers).bind(attributeSeq)
+ assert(boundEncoder.fromRow(boundEncoder.toRow(data)) === (
+ Seq(Some(1), Some(2), null), "Option in array"))
+ }
+
+ test("Option in Map") {
+ val data = (Map(1 -> Some(10L), 2 -> Some(20L), 3 -> None), "Option in map")
+
+ val schema = ScalaReflection.schemaFor[Tuple2[Map[Int, Option[Long]], String]]
+ .dataType.asInstanceOf[StructType]
+ val attributeSeq = schema.toAttributes
+ val dataEncoder = encoderFor[Tuple2[Map[Int, Option[Long]], String]]
+ val boundEncoder = dataEncoder.resolve(attributeSeq, outers).bind(attributeSeq)
+ assert(boundEncoder.fromRow(boundEncoder.toRow(data)) === (
+ Map(1 -> Some(10L), 2 -> Some(20L), 3 -> null), "Option in map"))
+ }
--- End diff --
Because for `Seq(Some(1), None)` the converted back result is `Seq(Some(1), null)`, `encodeDecodeTest` will report it failed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12439][SQL] Fix toCatalystArray and Map...
Posted by viirya <gi...@git.apache.org>.
Github user viirya commented on a diff in the pull request:
https://github.com/apache/spark/pull/10391#discussion_r48253305
--- Diff: sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/encoders/ExpressionEncoderSuite.scala ---
@@ -60,6 +61,22 @@ case class RepeatedData(
mapFieldNull: scala.collection.Map[Int, java.lang.Long],
structField: PrimitiveData)
+case class Nested(i: Option[Int], s: String)
+
+case class Data(
+ array: Seq[Int],
+ arrayContainsNull: Seq[Option[Int]],
+ map: scala.collection.Map[Int, Long],
+ mapContainsNul: scala.collection.Map[Int, Option[Long]],
+ nested: Nested)
+
+case class ComplexData(
+ arrayField: Seq[Int],
+ arrayFieldContainsNull: Seq[Option[Int]],
+ mapField: scala.collection.Map[Int, Long],
+ mapFieldContainsNull: scala.collection.Map[Int, Option[Long]],
+ dataField: Data)
+
--- End diff --
I think I can remove these classes. I will update later.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12439][SQL] Fix toCatalystArray and Map...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/10391#issuecomment-168113881
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/48535/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12439][SQL] Fix toCatalystArray and Map...
Posted by viirya <gi...@git.apache.org>.
Github user viirya commented on a diff in the pull request:
https://github.com/apache/spark/pull/10391#discussion_r48527294
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/objects.scala ---
@@ -439,6 +439,13 @@ case class MapObjects(
s"boolean ${loopVar.isNull} = ${genInputData.isNull} || ${loopVar.value} == null;"
}
+ // If lambdaFunction is WrapOption, we will not determine null or not based on the
+ // value of loopVar.isNull, because WrapOption will return None for null.
+ val isWrapOption = lambdaFunction match {
--- End diff --
@marmbrus I have refactored this part and removed the check of WrapOption now. Please take a look it if it is better now. Thanks.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12439][SQL] Fix toCatalystArray and Map...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/10391#issuecomment-166815230
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12439][SQL] Fix toCatalystArray and Map...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/10391#issuecomment-166663389
**[Test build #48204 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/48204/consoleFull)** for PR 10391 at commit [`a726fd8`](https://github.com/apache/spark/commit/a726fd890a7dd82d5f887489b184ab19910c14fc).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12439][SQL] Fix toCatalystArray and Map...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/10391#issuecomment-166926556
**[Test build #48245 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/48245/consoleFull)** for PR 10391 at commit [`f5b9d50`](https://github.com/apache/spark/commit/f5b9d507df3946a18426061d3dc0d85e17cca80a).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12439][SQL] Fix toCatalystArray and Map...
Posted by viirya <gi...@git.apache.org>.
Github user viirya commented on the pull request:
https://github.com/apache/spark/pull/10391#issuecomment-167746900
@cloud-fan Please review this update if you have time to do. Thanks.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12439][SQL] Fix toCatalystArray and Map...
Posted by cloud-fan <gi...@git.apache.org>.
Github user cloud-fan commented on a diff in the pull request:
https://github.com/apache/spark/pull/10391#discussion_r48537552
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/encoders/RowEncoder.scala ---
@@ -35,7 +35,8 @@ object RowEncoder {
def apply(schema: StructType): ExpressionEncoder[Row] = {
val cls = classOf[Row]
val inputObject = BoundReference(0, ObjectType(cls), nullable = true)
- val extractExpressions = extractorsFor(inputObject, schema)
+ // We use an If expression to wrap extractorsFor result of StructType
+ val extractExpressions = extractorsFor(inputObject, schema).asInstanceOf[If].falseValue
val constructExpression = constructorFor(schema)
--- End diff --
same with `extractExpressions`, we should also strip the top-most `If` for `constructExpression`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12439][SQL] Fix toCatalystArray and Map...
Posted by viirya <gi...@git.apache.org>.
Github user viirya commented on a diff in the pull request:
https://github.com/apache/spark/pull/10391#discussion_r48252927
--- Diff: sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/encoders/ExpressionEncoderSuite.scala ---
@@ -60,6 +61,22 @@ case class RepeatedData(
mapFieldNull: scala.collection.Map[Int, java.lang.Long],
structField: PrimitiveData)
+case class Nested(i: Option[Int], s: String)
+
+case class Data(
+ array: Seq[Int],
+ arrayContainsNull: Seq[Option[Int]],
+ map: scala.collection.Map[Int, Long],
+ mapContainsNul: scala.collection.Map[Int, Option[Long]],
+ nested: Nested)
+
+case class ComplexData(
+ arrayField: Seq[Int],
+ arrayFieldContainsNull: Seq[Option[Int]],
+ mapField: scala.collection.Map[Int, Long],
+ mapFieldContainsNull: scala.collection.Map[Int, Option[Long]],
+ dataField: Data)
+
--- End diff --
Besides, `encodeDecodeTest(Seq(Some(1), None)` may not work because converted back will be `Seq(Some(1), null)`.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12439][SQL] Fix toCatalystArray and Map...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/10391#issuecomment-166946901
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12439][SQL] Fix toCatalystArray and Map...
Posted by cloud-fan <gi...@git.apache.org>.
Github user cloud-fan commented on the pull request:
https://github.com/apache/spark/pull/10391#issuecomment-167780839
overall LGTM
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12439][SQL] Fix toCatalystArray and Map...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/10391#issuecomment-166689640
**[Test build #48204 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/48204/consoleFull)** for PR 10391 at commit [`a726fd8`](https://github.com/apache/spark/commit/a726fd890a7dd82d5f887489b184ab19910c14fc).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12439][SQL] Fix toCatalystArray and Map...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/10391#issuecomment-166946807
**[Test build #48245 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/48245/consoleFull)** for PR 10391 at commit [`f5b9d50`](https://github.com/apache/spark/commit/f5b9d507df3946a18426061d3dc0d85e17cca80a).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12439][SQL] Fix toCatalystArray and Map...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/10391#issuecomment-165990384
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12439][SQL] Fix toCatalystArray and Map...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/10391#issuecomment-166815206
**[Test build #48226 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/48226/consoleFull)** for PR 10391 at commit [`7389e15`](https://github.com/apache/spark/commit/7389e157c1e1411a4115df29969359888f13c502).
* This patch **fails from timeout after a configured wait of \`250m\`**.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12439][SQL] Fix toCatalystArray and Map...
Posted by marmbrus <gi...@git.apache.org>.
Github user marmbrus commented on the pull request:
https://github.com/apache/spark/pull/10391#issuecomment-166404830
+1 to moving the test
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12439][SQL] Fix toCatalystArray and Map...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/10391#issuecomment-166624997
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/48191/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12439][SQL] Fix toCatalystArray and Map...
Posted by viirya <gi...@git.apache.org>.
Github user viirya commented on the pull request:
https://github.com/apache/spark/pull/10391#issuecomment-166819194
retest this please.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12439][SQL] Fix toCatalystArray and Map...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/10391#issuecomment-166924875
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/48244/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12439][SQL] Fix toCatalystArray and Map...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/10391#issuecomment-166815231
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/48226/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12439][SQL] Fix toCatalystArray and Map...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/10391#issuecomment-168120502
**[Test build #48538 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/48538/consoleFull)** for PR 10391 at commit [`e541e04`](https://github.com/apache/spark/commit/e541e04820e6a34f36fcaade47cd669ca09b6baa).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12439][SQL] Fix toCatalystArray and Map...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/10391#issuecomment-166689820
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/48204/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12439][SQL] Fix toCatalystArray and Map...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/10391#issuecomment-166948261
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12439][SQL] Fix toCatalystArray and Map...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/10391#issuecomment-166689818
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12439][SQL] Fix toCatalystArray and Map...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/10391#issuecomment-166948106
**[Test build #48246 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/48246/consoleFull)** for PR 10391 at commit [`f5b9d50`](https://github.com/apache/spark/commit/f5b9d507df3946a18426061d3dc0d85e17cca80a).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12439][SQL] Fix toCatalystArray and Map...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/10391#issuecomment-167747200
**[Test build #48407 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/48407/consoleFull)** for PR 10391 at commit [`7728124`](https://github.com/apache/spark/commit/7728124d227444eba34aa2e7830d56e123076d7a).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12439][SQL] Fix toCatalystArray and Map...
Posted by marmbrus <gi...@git.apache.org>.
Github user marmbrus commented on the pull request:
https://github.com/apache/spark/pull/10391#issuecomment-169124211
Thanks, merging to master.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12439][SQL] Fix toCatalystArray and Map...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/10391#issuecomment-166652647
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/48198/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12439][SQL] Fix toCatalystArray and Map...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/10391#issuecomment-166899721
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/48239/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12439][SQL] Fix toCatalystArray and Map...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/10391#issuecomment-167760966
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/48407/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12439][SQL] Fix toCatalystArray and Map...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/10391#issuecomment-167760965
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12439][SQL] Fix toCatalystArray and Map...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/10391#issuecomment-166652431
**[Test build #48198 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/48198/consoleFull)** for PR 10391 at commit [`a726fd8`](https://github.com/apache/spark/commit/a726fd890a7dd82d5f887489b184ab19910c14fc).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12439][SQL] Fix toCatalystArray and Map...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/10391#issuecomment-165990385
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/48052/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12439][SQL] Fix toCatalystArray and Map...
Posted by viirya <gi...@git.apache.org>.
Github user viirya commented on the pull request:
https://github.com/apache/spark/pull/10391#issuecomment-166181451
cc @cloud-fan @marmbrus @davies
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12439][SQL] Fix toCatalystArray and Map...
Posted by marmbrus <gi...@git.apache.org>.
Github user marmbrus commented on a diff in the pull request:
https://github.com/apache/spark/pull/10391#discussion_r48501635
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/objects.scala ---
@@ -439,6 +439,13 @@ case class MapObjects(
s"boolean ${loopVar.isNull} = ${genInputData.isNull} || ${loopVar.value} == null;"
}
+ // If lambdaFunction is WrapOption, we will not determine null or not based on the
+ // value of loopVar.isNull, because WrapOption will return None for null.
+ val isWrapOption = lambdaFunction match {
--- End diff --
I'm not sure exactly what is going on here, but this looks like a hack. Ideally this node should not need to know what type its child is to operate correctly.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12439][SQL] Fix toCatalystArray and Map...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/10391#issuecomment-166869105
**[Test build #48232 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/48232/consoleFull)** for PR 10391 at commit [`7389e15`](https://github.com/apache/spark/commit/7389e157c1e1411a4115df29969359888f13c502).
* This patch **fails from timeout after a configured wait of \`250m\`**.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12439][SQL] Fix toCatalystArray and Map...
Posted by viirya <gi...@git.apache.org>.
Github user viirya commented on the pull request:
https://github.com/apache/spark/pull/10391#issuecomment-166816207
retest this please.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12439][SQL] Fix toCatalystArray and Map...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/10391#issuecomment-166869266
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/48232/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12439][SQL] Fix toCatalystArray and Map...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/10391#issuecomment-166948262
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/48246/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12439][SQL] Fix toCatalystArray and Map...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/10391#issuecomment-166928166
**[Test build #48246 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/48246/consoleFull)** for PR 10391 at commit [`f5b9d50`](https://github.com/apache/spark/commit/f5b9d507df3946a18426061d3dc0d85e17cca80a).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12439][SQL] Fix toCatalystArray and Map...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/10391#issuecomment-166946904
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/48245/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12439][SQL] Fix toCatalystArray and Map...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/10391#issuecomment-166632138
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12439][SQL] Fix toCatalystArray and Map...
Posted by viirya <gi...@git.apache.org>.
Github user viirya commented on the pull request:
https://github.com/apache/spark/pull/10391#issuecomment-167033940
@cloud-fan @marmbrus Are you ok for latest updates?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12439][SQL] Fix toCatalystArray and Map...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/10391#issuecomment-166790390
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12439][SQL] Fix toCatalystArray and Map...
Posted by viirya <gi...@git.apache.org>.
Github user viirya commented on the pull request:
https://github.com/apache/spark/pull/10391#issuecomment-166661949
retest this please.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12439][SQL] Fix toCatalystArray and Map...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/10391#issuecomment-168127999
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12439][SQL] Fix toCatalystArray and Map...
Posted by viirya <gi...@git.apache.org>.
Github user viirya commented on the pull request:
https://github.com/apache/spark/pull/10391#issuecomment-166632972
retest this please.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12439][SQL] Fix toCatalystArray and Map...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/10391#issuecomment-166924872
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12439][SQL] Fix toCatalystArray and Map...
Posted by viirya <gi...@git.apache.org>.
Github user viirya commented on the pull request:
https://github.com/apache/spark/pull/10391#issuecomment-168118966
retest this please.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12439][SQL] Fix toCatalystArray and Map...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/10391#issuecomment-166790415
**[Test build #48226 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/48226/consoleFull)** for PR 10391 at commit [`7389e15`](https://github.com/apache/spark/commit/7389e157c1e1411a4115df29969359888f13c502).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12439][SQL] Fix toCatalystArray and Map...
Posted by viirya <gi...@git.apache.org>.
Github user viirya commented on a diff in the pull request:
https://github.com/apache/spark/pull/10391#discussion_r48527189
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/encoders/RowEncoder.scala ---
@@ -92,6 +93,7 @@ object RowEncoder {
NewInstance(
classOf[GenericArrayData],
inputObject :: Nil,
+ propagateNull = true,
--- End diff --
#10443 will fix this by using true as default value for propagateNull. As that pr is not merged yet, in order to pass tests now, I set propagateNull as true here. Once #10443 is getting merged, I can remove the two lines here.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12439][SQL] Fix toCatalystArray and Map...
Posted by viirya <gi...@git.apache.org>.
Github user viirya commented on a diff in the pull request:
https://github.com/apache/spark/pull/10391#discussion_r48252788
--- Diff: sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/encoders/ExpressionEncoderSuite.scala ---
@@ -60,6 +61,22 @@ case class RepeatedData(
mapFieldNull: scala.collection.Map[Int, java.lang.Long],
structField: PrimitiveData)
+case class Nested(i: Option[Int], s: String)
+
+case class Data(
+ array: Seq[Int],
+ arrayContainsNull: Seq[Option[Int]],
+ map: scala.collection.Map[Int, Long],
+ mapContainsNul: scala.collection.Map[Int, Option[Long]],
+ nested: Nested)
+
+case class ComplexData(
+ arrayField: Seq[Int],
+ arrayFieldContainsNull: Seq[Option[Int]],
+ mapField: scala.collection.Map[Int, Long],
+ mapFieldContainsNull: scala.collection.Map[Int, Option[Long]],
+ dataField: Data)
+
--- End diff --
We should also test map case.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12439][SQL] Fix toCatalystArray and Map...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/10391#issuecomment-166624666
**[Test build #48197 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/48197/consoleFull)** for PR 10391 at commit [`a726fd8`](https://github.com/apache/spark/commit/a726fd890a7dd82d5f887489b184ab19910c14fc).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12439][SQL] Fix toCatalystArray and Map...
Posted by viirya <gi...@git.apache.org>.
Github user viirya commented on a diff in the pull request:
https://github.com/apache/spark/pull/10391#discussion_r48313326
--- Diff: sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/encoders/ExpressionEncoderSuite.scala ---
@@ -239,6 +240,30 @@ class ExpressionEncoderSuite extends SparkFunSuite {
ExpressionEncoder.tuple(intEnc, ExpressionEncoder.tuple(intEnc, longEnc))
}
+ test("Option in Array") {
+ val data = (Seq(Some(1), Some(2), None), "Option in array")
+
+ val schema = ScalaReflection.schemaFor[Tuple2[Seq[Option[Int]], String]]
+ .dataType.asInstanceOf[StructType]
+ val attributeSeq = schema.toAttributes
+ val dataEncoder = encoderFor[Tuple2[Seq[Option[Int]], String]]
+ val boundEncoder = dataEncoder.resolve(attributeSeq, outers).bind(attributeSeq)
+ assert(boundEncoder.fromRow(boundEncoder.toRow(data)) === (
+ Seq(Some(1), Some(2), null), "Option in array"))
+ }
+
+ test("Option in Map") {
+ val data = (Map(1 -> Some(10L), 2 -> Some(20L), 3 -> None), "Option in map")
+
+ val schema = ScalaReflection.schemaFor[Tuple2[Map[Int, Option[Long]], String]]
+ .dataType.asInstanceOf[StructType]
+ val attributeSeq = schema.toAttributes
+ val dataEncoder = encoderFor[Tuple2[Map[Int, Option[Long]], String]]
+ val boundEncoder = dataEncoder.resolve(attributeSeq, outers).bind(attributeSeq)
+ assert(boundEncoder.fromRow(boundEncoder.toRow(data)) === (
+ Map(1 -> Some(10L), 2 -> Some(20L), 3 -> null), "Option in map"))
+ }
--- End diff --
I am not sure about this. Because you can see at [ScalaReflectionRelationSuite.scala#L128](https://github.com/apache/spark/blob/d83c2f9f0b08d6d5d369d9fae04cdb15448e7f0d/sql/core/src/test/scala/org/apache/spark/sql/ScalaReflectionRelationSuite.scala#L128) where this test is originally coming from, looks like it is expected to have `Seq(Some(1), null)` converted for `Seq(Some(1), None)`.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12439][SQL] Fix toCatalystArray and Map...
Posted by viirya <gi...@git.apache.org>.
Github user viirya commented on a diff in the pull request:
https://github.com/apache/spark/pull/10391#discussion_r48317931
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/ScalaReflection.scala ---
@@ -406,11 +406,14 @@ object ScalaReflection extends ScalaReflection {
def toCatalystArray(input: Expression, elementType: `Type`): Expression = {
val externalDataType = dataTypeFor(elementType)
val Schema(catalystType, nullable) = silentSchemaFor(elementType)
- if (isNativeType(catalystType)) {
- NewInstance(
- classOf[GenericArrayData],
- input :: Nil,
- dataType = ArrayType(catalystType, nullable))
+ if (isNativeType(externalDataType)) {
+ expressions.If(
+ IsNull(input),
+ expressions.Literal.create(null, ArrayType(catalystType, nullable)),
+ NewInstance(
+ classOf[GenericArrayData],
+ input :: Nil,
+ dataType = ArrayType(catalystType, nullable)))
--- End diff --
The part of fixing is from pr #10401. Without this, the following fixing to `MapObjects` will cause null exception. Can we merge #10401 first then I do rebase here?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12439][SQL] Fix toCatalystArray and Map...
Posted by viirya <gi...@git.apache.org>.
Github user viirya commented on a diff in the pull request:
https://github.com/apache/spark/pull/10391#discussion_r48318003
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/objects.scala ---
@@ -456,10 +456,10 @@ case class MapObjects(
($elementJavaType)${genInputData.value}${itemAccessor(loopIndex)};
$loopNullCheck
- if (${loopVar.isNull}) {
+ ${genFunction.code}
+ if (${genFunction.isNull}) {
--- End diff --
Even `loopVar.isNull` is true, we should still call `genFunction` because it may have special handling for null case, for example `WrapOption` will produce a `None` for null value. We should depend on `genFunction.isNull` to decide `$convertedArray[$loopIndex]` is null or not.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12439][SQL] Fix toCatalystArray and Map...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/10391#issuecomment-165982943
**[Test build #48052 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/48052/consoleFull)** for PR 10391 at commit [`788a7c6`](https://github.com/apache/spark/commit/788a7c6b2156d43be9c389d6a67b70e8ff9bbbb2).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12439][SQL] Fix toCatalystArray and Map...
Posted by marmbrus <gi...@git.apache.org>.
Github user marmbrus commented on a diff in the pull request:
https://github.com/apache/spark/pull/10391#discussion_r48314060
--- Diff: sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/encoders/ExpressionEncoderSuite.scala ---
@@ -239,6 +240,30 @@ class ExpressionEncoderSuite extends SparkFunSuite {
ExpressionEncoder.tuple(intEnc, ExpressionEncoder.tuple(intEnc, longEnc))
}
+ test("Option in Array") {
+ val data = (Seq(Some(1), Some(2), None), "Option in array")
+
+ val schema = ScalaReflection.schemaFor[Tuple2[Seq[Option[Int]], String]]
+ .dataType.asInstanceOf[StructType]
+ val attributeSeq = schema.toAttributes
+ val dataEncoder = encoderFor[Tuple2[Seq[Option[Int]], String]]
+ val boundEncoder = dataEncoder.resolve(attributeSeq, outers).bind(attributeSeq)
+ assert(boundEncoder.fromRow(boundEncoder.toRow(data)) === (
+ Seq(Some(1), Some(2), null), "Option in array"))
+ }
+
+ test("Option in Map") {
+ val data = (Map(1 -> Some(10L), 2 -> Some(20L), 3 -> None), "Option in map")
+
+ val schema = ScalaReflection.schemaFor[Tuple2[Map[Int, Option[Long]], String]]
+ .dataType.asInstanceOf[StructType]
+ val attributeSeq = schema.toAttributes
+ val dataEncoder = encoderFor[Tuple2[Map[Int, Option[Long]], String]]
+ val boundEncoder = dataEncoder.resolve(attributeSeq, outers).bind(attributeSeq)
+ assert(boundEncoder.fromRow(boundEncoder.toRow(data)) === (
+ Map(1 -> Some(10L), 2 -> Some(20L), 3 -> null), "Option in map"))
+ }
--- End diff --
The original implementation in `ScalaReflection` only had access to the schema as a DataType when it was going to produce external `Row` objects. This schema is erased in that it can't differentiate `Option` from `null`. This is not true for encoders since we know the full type the user is expecting from the provided `TypeTag`.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12439][SQL] Fix toCatalystArray and Map...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/10391#issuecomment-168127966
**[Test build #48538 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/48538/consoleFull)** for PR 10391 at commit [`e541e04`](https://github.com/apache/spark/commit/e541e04820e6a34f36fcaade47cd669ca09b6baa).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12439][SQL] Fix toCatalystArray and Map...
Posted by viirya <gi...@git.apache.org>.
Github user viirya commented on the pull request:
https://github.com/apache/spark/pull/10391#issuecomment-167585649
@cloud-fan Can you also review the updates? Thanks.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12439][SQL] Fix toCatalystArray and Map...
Posted by viirya <gi...@git.apache.org>.
Github user viirya commented on the pull request:
https://github.com/apache/spark/pull/10391#issuecomment-168839752
ping @marmbrus please take a look for the updates, thanks.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12439][SQL] Fix toCatalystArray and Map...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/10391#issuecomment-167760903
**[Test build #48407 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/48407/consoleFull)** for PR 10391 at commit [`7728124`](https://github.com/apache/spark/commit/7728124d227444eba34aa2e7830d56e123076d7a).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12439][SQL] Fix toCatalystArray and Map...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/10391#issuecomment-166624995
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12439][SQL] Fix toCatalystArray and Map...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/10391#issuecomment-166818308
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12439][SQL] Fix toCatalystArray and Map...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/10391#issuecomment-166899666
**[Test build #48239 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/48239/consoleFull)** for PR 10391 at commit [`7389e15`](https://github.com/apache/spark/commit/7389e157c1e1411a4115df29969359888f13c502).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12439][SQL] Fix toCatalystArray and Map...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/10391#issuecomment-166818312
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/48230/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12439][SQL] Fix toCatalystArray and Map...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/10391#issuecomment-166633979
**[Test build #48198 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/48198/consoleFull)** for PR 10391 at commit [`a726fd8`](https://github.com/apache/spark/commit/a726fd890a7dd82d5f887489b184ab19910c14fc).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12439][SQL] Fix toCatalystArray and Map...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/10391#issuecomment-166899720
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12439][SQL] Fix toCatalystArray and Map...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/10391#issuecomment-166881444
**[Test build #48239 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/48239/consoleFull)** for PR 10391 at commit [`7389e15`](https://github.com/apache/spark/commit/7389e157c1e1411a4115df29969359888f13c502).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12439][SQL] Fix toCatalystArray and Map...
Posted by viirya <gi...@git.apache.org>.
Github user viirya commented on the pull request:
https://github.com/apache/spark/pull/10391#issuecomment-168517732
ping @marmbrus @rxin
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12439][SQL] Fix toCatalystArray and Map...
Posted by cloud-fan <gi...@git.apache.org>.
Github user cloud-fan commented on a diff in the pull request:
https://github.com/apache/spark/pull/10391#discussion_r48252602
--- Diff: sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/encoders/ExpressionEncoderSuite.scala ---
@@ -60,6 +61,22 @@ case class RepeatedData(
mapFieldNull: scala.collection.Map[Int, java.lang.Long],
structField: PrimitiveData)
+case class Nested(i: Option[Int], s: String)
+
+case class Data(
+ array: Seq[Int],
+ arrayContainsNull: Seq[Option[Int]],
+ map: scala.collection.Map[Int, Long],
+ mapContainsNul: scala.collection.Map[Int, Option[Long]],
+ nested: Nested)
+
+case class ComplexData(
+ arrayField: Seq[Int],
+ arrayFieldContainsNull: Seq[Option[Int]],
+ mapField: scala.collection.Map[Int, Long],
+ mapFieldContainsNull: scala.collection.Map[Int, Option[Long]],
+ dataField: Data)
+
--- End diff --
do we have to add these classes?
I think a simple test could be `encodeDecodeTest(Seq(Some(1), None), "seq of option")`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12439][SQL] Fix toCatalystArray and Map...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/10391#issuecomment-165990346
**[Test build #48052 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/48052/consoleFull)** for PR 10391 at commit [`788a7c6`](https://github.com/apache/spark/commit/788a7c6b2156d43be9c389d6a67b70e8ff9bbbb2).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12439][SQL] Fix toCatalystArray and Map...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/10391#issuecomment-166605537
**[Test build #48191 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/48191/consoleFull)** for PR 10391 at commit [`180048c`](https://github.com/apache/spark/commit/180048ccbb1756720e988193d546529fa2d42145).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12439][SQL] Fix toCatalystArray and Map...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/10391#issuecomment-166632084
**[Test build #48197 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/48197/consoleFull)** for PR 10391 at commit [`a726fd8`](https://github.com/apache/spark/commit/a726fd890a7dd82d5f887489b184ab19910c14fc).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12439][SQL] Fix toCatalystArray and Map...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/10391#issuecomment-168113879
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12439][SQL] Fix toCatalystArray and Map...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/10391#issuecomment-166624874
**[Test build #48191 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/48191/consoleFull)** for PR 10391 at commit [`180048c`](https://github.com/apache/spark/commit/180048ccbb1756720e988193d546529fa2d42145).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12439][SQL] Fix toCatalystArray and Map...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/10391#issuecomment-166821703
**[Test build #48232 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/48232/consoleFull)** for PR 10391 at commit [`7389e15`](https://github.com/apache/spark/commit/7389e157c1e1411a4115df29969359888f13c502).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12439][SQL] Fix toCatalystArray and Map...
Posted by marmbrus <gi...@git.apache.org>.
Github user marmbrus commented on a diff in the pull request:
https://github.com/apache/spark/pull/10391#discussion_r48288735
--- Diff: sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/encoders/ExpressionEncoderSuite.scala ---
@@ -239,6 +240,30 @@ class ExpressionEncoderSuite extends SparkFunSuite {
ExpressionEncoder.tuple(intEnc, ExpressionEncoder.tuple(intEnc, longEnc))
}
+ test("Option in Array") {
+ val data = (Seq(Some(1), Some(2), None), "Option in array")
+
+ val schema = ScalaReflection.schemaFor[Tuple2[Seq[Option[Int]], String]]
+ .dataType.asInstanceOf[StructType]
+ val attributeSeq = schema.toAttributes
+ val dataEncoder = encoderFor[Tuple2[Seq[Option[Int]], String]]
+ val boundEncoder = dataEncoder.resolve(attributeSeq, outers).bind(attributeSeq)
+ assert(boundEncoder.fromRow(boundEncoder.toRow(data)) === (
+ Seq(Some(1), Some(2), null), "Option in array"))
+ }
+
+ test("Option in Map") {
+ val data = (Map(1 -> Some(10L), 2 -> Some(20L), 3 -> None), "Option in map")
+
+ val schema = ScalaReflection.schemaFor[Tuple2[Map[Int, Option[Long]], String]]
+ .dataType.asInstanceOf[StructType]
+ val attributeSeq = schema.toAttributes
+ val dataEncoder = encoderFor[Tuple2[Map[Int, Option[Long]], String]]
+ val boundEncoder = dataEncoder.resolve(attributeSeq, outers).bind(attributeSeq)
+ assert(boundEncoder.fromRow(boundEncoder.toRow(data)) === (
+ Map(1 -> Some(10L), 2 -> Some(20L), 3 -> null), "Option in map"))
+ }
--- End diff --
That sounds like a bug.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12439][SQL] Fix toCatalystArray and Map...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/10391#issuecomment-166869262
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12439][SQL] Fix toCatalystArray and Map...
Posted by cloud-fan <gi...@git.apache.org>.
Github user cloud-fan commented on a diff in the pull request:
https://github.com/apache/spark/pull/10391#discussion_r48263973
--- Diff: sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/encoders/ExpressionEncoderSuite.scala ---
@@ -239,6 +240,30 @@ class ExpressionEncoderSuite extends SparkFunSuite {
ExpressionEncoder.tuple(intEnc, ExpressionEncoder.tuple(intEnc, longEnc))
}
+ test("Option in Array") {
+ val data = (Seq(Some(1), Some(2), None), "Option in array")
+
+ val schema = ScalaReflection.schemaFor[Tuple2[Seq[Option[Int]], String]]
+ .dataType.asInstanceOf[StructType]
+ val attributeSeq = schema.toAttributes
+ val dataEncoder = encoderFor[Tuple2[Seq[Option[Int]], String]]
+ val boundEncoder = dataEncoder.resolve(attributeSeq, outers).bind(attributeSeq)
+ assert(boundEncoder.fromRow(boundEncoder.toRow(data)) === (
+ Seq(Some(1), Some(2), null), "Option in array"))
+ }
+
+ test("Option in Map") {
+ val data = (Map(1 -> Some(10L), 2 -> Some(20L), 3 -> None), "Option in map")
+
+ val schema = ScalaReflection.schemaFor[Tuple2[Map[Int, Option[Long]], String]]
+ .dataType.asInstanceOf[StructType]
+ val attributeSeq = schema.toAttributes
+ val dataEncoder = encoderFor[Tuple2[Map[Int, Option[Long]], String]]
+ val boundEncoder = dataEncoder.resolve(attributeSeq, outers).bind(attributeSeq)
+ assert(boundEncoder.fromRow(boundEncoder.toRow(data)) === (
+ Map(1 -> Some(10L), 2 -> Some(20L), 3 -> null), "Option in map"))
+ }
--- End diff --
can we use the helper function `encodeDecodeTest`? It can simplify the test code and give detail error message when it fails.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12439][SQL] Fix toCatalystArray and Map...
Posted by cloud-fan <gi...@git.apache.org>.
Github user cloud-fan commented on the pull request:
https://github.com/apache/spark/pull/10391#issuecomment-166189382
Good catch! One mirror comment, can we write the test in `ExpressionEncoderSuite`?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12439][SQL] Fix toCatalystArray and Map...
Posted by viirya <gi...@git.apache.org>.
Github user viirya commented on the pull request:
https://github.com/apache/spark/pull/10391#issuecomment-166879266
retest this please.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12439][SQL] Fix toCatalystArray and Map...
Posted by viirya <gi...@git.apache.org>.
Github user viirya commented on a diff in the pull request:
https://github.com/apache/spark/pull/10391#discussion_r48540861
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/encoders/RowEncoder.scala ---
@@ -35,7 +35,8 @@ object RowEncoder {
def apply(schema: StructType): ExpressionEncoder[Row] = {
val cls = classOf[Row]
val inputObject = BoundReference(0, ObjectType(cls), nullable = true)
- val extractExpressions = extractorsFor(inputObject, schema)
+ // We use an If expression to wrap extractorsFor result of StructType
+ val extractExpressions = extractorsFor(inputObject, schema).asInstanceOf[If].falseValue
val constructExpression = constructorFor(schema)
--- End diff --
Unlike `extractExpressions`, `constructExpression` hasn't been wrapped with an `If` here. Because there are two `constructorFor` methods, and the one wrapped with `If` is the second one.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org