You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by mn-mikke <gi...@git.apache.org> on 2018/05/02 13:55:57 UTC
[GitHub] spark pull request #21215: [SPARK-24148][SQL] Overloading array function to ...
GitHub user mn-mikke opened a pull request:
https://github.com/apache/spark/pull/21215
[SPARK-24148][SQL] Overloading array function to support typed empty arrays
## What changes were proposed in this pull request?
The PR proposes to overload `array` function and allow users to specify the element type for empty arrays. Currently, empty arrays produced by `array` function are of `StringType` and there is no way how to cast them to a different type.
A perfect example of the use case is `when(cond, trueExp).otherwise(falseExp)`, which expects `trueExp` and `falseExp` of being the same type. In scenario where we want to produce an empty array, in one of these cases, there's no other way than creating an `UDF`.
## How was this patch tested?
Added test cases into `DataFrameComplexTypeSuite`
## Note
Eventually, I will add a wrapper for PySpark, but would like to discuss the idea first.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/AbsaOSS/spark feature/array-api-empty-array-to-master
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/21215.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #21215
----
commit 44b18520dcf8e3e3639756cd8a12f75ea1080bee
Author: Marek Novotny <mn...@...>
Date: 2018-05-02T13:42:42Z
[SPARK-24148][SQL] Overloading array function to support typed empty arrays.
----
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21215: [SPARK-24148][SQL] Overloading array function to support...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21215
Can one of the admins verify this patch?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21215: [SPARK-24148][SQL] Overloading array function to support...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21215
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/90137/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21215: [SPARK-24148][SQL] Overloading array function to support...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21215
**[Test build #90086 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/90086/testReport)** for PR 21215 at commit [`9c12457`](https://github.com/apache/spark/commit/9c124574a3fefe2e63dcd95bd03e47f1f8d5071a).
* This patch **fails PySpark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21215: [SPARK-24148][SQL] Overloading array function to support...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21215
Can one of the admins verify this patch?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21215: [SPARK-24148][SQL] Overloading array function to support...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21215
Merged build finished. Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21215: [SPARK-24148][SQL] Overloading array function to support...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21215
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21215: [SPARK-24148][SQL] Overloading array function to support...
Posted by maropu <gi...@git.apache.org>.
Github user maropu commented on the issue:
https://github.com/apache/spark/pull/21215
Do you wanna do this?
```
scala> sql("select array()").printSchema
root
|-- array(): array (nullable = false)
| |-- element: string (containsNull = false)
scala> sql("select CAST(array() AS ARRAY<INT>) c").printSchema
root
|-- c: array (nullable = false)
| |-- element: integer (containsNull = true)
scala> sql("select CAST(array() AS ARRAY<INT>) c").show
+---+
| c|
+---+
| []|
+---+
```
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21215: [SPARK-24148][SQL] Overloading array function to support...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21215
**[Test build #90137 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/90137/testReport)** for PR 21215 at commit [`e151ab7`](https://github.com/apache/spark/commit/e151ab7475fed32b2baaca5c0cbcf427a5c09ad3).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21215: [SPARK-24148][SQL] Overloading array function to support...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21215
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/90079/
Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21215: [SPARK-24148][SQL] Overloading array function to support...
Posted by maropu <gi...@git.apache.org>.
Github user maropu commented on the issue:
https://github.com/apache/spark/pull/21215
retest this please
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21215: [SPARK-24148][SQL] Overloading array function to support...
Posted by maropu <gi...@git.apache.org>.
Github user maropu commented on the issue:
https://github.com/apache/spark/pull/21215
Like this?
```
scala> val structTy = StructType.fromDDL("a ARRAY<STRUCT<b: INT, c: STRING>>")
structTy: org.apache.spark.sql.types.StructType = StructType(StructField(a,ArrayType(StructType(StructField(b,IntegerType,true), StructField(c,StringType,true)),true),true))
scala> val newCol = new Column(Literal.create(Seq.empty[Inner], structTy.head.dataType))
newCol: org.apache.spark.sql.Column = []
scala> val df = Seq(1, 2, 3).toDF("a").withColumn("b", newCol)
df: org.apache.spark.sql.DataFrame = [a: int, b: array<struct<b:int,c:string>>]
scala> df.show
+---+---+
| a| b|
+---+---+
| 1| []|
| 2| []|
| 3| []|
+---+---+
scala> df.printSchema
root
|-- a: integer (nullable = false)
|-- b: array (nullable = false)
| |-- element: struct (containsNull = true)
| | |-- b: integer (nullable = true)
| | |-- c: string (nullable = true)
```
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21215: [SPARK-24148][SQL] Overloading array function to support...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21215
**[Test build #90079 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/90079/testReport)** for PR 21215 at commit [`9c12457`](https://github.com/apache/spark/commit/9c124574a3fefe2e63dcd95bd03e47f1f8d5071a).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21215: [SPARK-24148][SQL] Overloading array function to support...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21215
**[Test build #90073 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/90073/testReport)** for PR 21215 at commit [`44b1852`](https://github.com/apache/spark/commit/44b18520dcf8e3e3639756cd8a12f75ea1080bee).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21215: [SPARK-24148][SQL] Overloading array function to support...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21215
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/90073/
Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21215: [SPARK-24148][SQL] Overloading array function to support...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21215
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/90086/
Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21215: [SPARK-24148][SQL] Overloading array function to support...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21215
Merged build finished. Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21215: [SPARK-24148][SQL] Overloading array function to support...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21215
Can one of the admins verify this patch?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21215: [SPARK-24148][SQL] Overloading array function to support...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21215
**[Test build #90137 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/90137/testReport)** for PR 21215 at commit [`e151ab7`](https://github.com/apache/spark/commit/e151ab7475fed32b2baaca5c0cbcf427a5c09ad3).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21215: [SPARK-24148][SQL] Overloading array function to support...
Posted by maropu <gi...@git.apache.org>.
Github user maropu commented on the issue:
https://github.com/apache/spark/pull/21215
How about this?
```
scala> val df = Seq(Outer(Seq.empty[Inner]), Outer(Seq.empty[Inner])).toDF("a")
df: org.apache.spark.sql.DataFrame = [a: array<struct<b:int,c:string>>]
scala> df.printSchema
root
|-- a: array (nullable = true)
| |-- element: struct (containsNull = true)
| | |-- b: integer (nullable = false)
| | |-- c: string (nullable = true)
scala> df.show
+---+
| a|
+---+
| []|
| []|
+---+
scala> val df = Seq(1, 2, 3).toDF("a").withColumn("b", typedLit(Seq.empty[Inner]))
df: org.apache.spark.sql.DataFrame = [a: int, b: array<struct<b:int,c:string>>]
scala> df.printSchema
root
|-- a: integer (nullable = false)
|-- b: array (nullable = false)
| |-- element: struct (containsNull = true)
| | |-- b: integer (nullable = false)
| | |-- c: string (nullable = true)
scala> df.show
+---+---+
| a| b|
+---+---+
| 1| []|
| 2| []|
| 3| []|
+---+---+
```
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21215: [SPARK-24148][SQL] Overloading array function to support...
Posted by gatorsmile <gi...@git.apache.org>.
Github user gatorsmile commented on the issue:
https://github.com/apache/spark/pull/21215
ok to test
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21215: [SPARK-24148][SQL] Overloading array function to support...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21215
**[Test build #90073 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/90073/testReport)** for PR 21215 at commit [`44b1852`](https://github.com/apache/spark/commit/44b18520dcf8e3e3639756cd8a12f75ea1080bee).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds the following public classes _(experimental)_:
* `case class CreateArray(children: Seq[Expression], defaultElementType: DataType = StringType)`
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21215: [SPARK-24148][SQL] Overloading array function to support...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21215
**[Test build #90079 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/90079/testReport)** for PR 21215 at commit [`9c12457`](https://github.com/apache/spark/commit/9c124574a3fefe2e63dcd95bd03e47f1f8d5071a).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21215: [SPARK-24148][SQL] Overloading array function to support...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21215
Merged build finished. Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21215: [SPARK-24148][SQL] Overloading array function to support...
Posted by mn-mikke <gi...@git.apache.org>.
Github user mn-mikke commented on the issue:
https://github.com/apache/spark/pull/21215
@lokm01 @gatorsmile @maropu @ueshin
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21215: [SPARK-24148][SQL] Overloading array function to support...
Posted by lokm01 <gi...@git.apache.org>.
Github user lokm01 commented on the issue:
https://github.com/apache/spark/pull/21215
Hey @maropu,
So we've encountered a number of issues with casting:
1. Casting an empty array to an array of primitive types caused an exception on 2.2.1, but works on 2.3.0+ so that's sorted
2. We're still facing an issue on 2.3.0 when we try to cast an empty array to an array of complex types. See the following example:
`
case class Outer(a: List[Inner])
case class Inner(b: Int, c: String)
object App4 extends App {
val spark = SparkSession.builder().appName("").master("local[*]").getOrCreate()
import spark.implicits._
import org.apache.spark.sql.functions._
val df = spark.createDataFrame(Seq[Outer]())
val r = spark.range(100).select(array().cast(df.schema("a").dataType))
r.printSchema()
r.show
}
`
This code produces
> Exception in thread "main" org.apache.spark.sql.AnalysisException: cannot resolve 'array()' due to data type mismatch: cannot cast array<string> to array<struct<b:int,c:string>>;;
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21215: [SPARK-24148][SQL] Overloading array function to support...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21215
**[Test build #90086 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/90086/testReport)** for PR 21215 at commit [`9c12457`](https://github.com/apache/spark/commit/9c124574a3fefe2e63dcd95bd03e47f1f8d5071a).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21215: [SPARK-24148][SQL] Overloading array function to support...
Posted by lokm01 <gi...@git.apache.org>.
Github user lokm01 commented on the issue:
https://github.com/apache/spark/pull/21215
@maropu That would work if you had scala case classes for all the types. In our case, we're working on a generic framework, where we only have Spark schemas (and I'd rather not generate case classes at runtime).
Can you suggest an existing way to do this using spark's DataType please?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21215: [SPARK-24148][SQL] Overloading array function to support...
Posted by mn-mikke <gi...@git.apache.org>.
Github user mn-mikke commented on the issue:
https://github.com/apache/spark/pull/21215
@maropu Really nice idea to create typed empty arrays via an `Literal` expression! On the other hand, I feel that the end user shouldn't work with classes from Catalyst internals if we consider that the creation of typed empty arrays is an elementary operation.
I've tailored the solution according to your suggestion, but still think that some function should be introduced. What do you think?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21215: [SPARK-24148][SQL] Overloading array function to support...
Posted by lokm01 <gi...@git.apache.org>.
Github user lokm01 commented on the issue:
https://github.com/apache/spark/pull/21215
@maropu Thanks! Didn't know about creating a literal this way.
Don't you feel that the suggested change is way more elegant?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org