You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by mn-mikke <gi...@git.apache.org> on 2018/06/23 10:19:38 UTC
[GitHub] spark pull request #21620: [SPARK-24636][SQL] Type coercion of arrays for ar...
GitHub user mn-mikke opened a pull request:
https://github.com/apache/spark/pull/21620
[SPARK-24636][SQL] Type coercion of arrays for array_join function
## What changes were proposed in this pull request?
Presto's implementation accepts arbitrary arrays of primitive types as an input:
```
presto> SELECT array_join(ARRAY [1, 2, 3], ', ');
_col0
---------
1, 2, 3
(1 row)
```
This PR proposes to implement a type coercion rule for ```array_join``` function that converts arrays of primitive as well as non-primitive types to arrays of string.
## How was this patch tested?
New test cases add into:
- sql-tests/inputs/typeCoercion/native/arrayJoin.sql
- DataFrameFunctionsSuite.scala
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/mn-mikke/spark SPARK-24636
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/21620.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #21620
----
commit 98017714e06ea7a730b1515d5bcfd583d7808a5a
Author: Marek Novotny <mn...@...>
Date: 2018-06-23T10:08:42Z
[SPARK-24636][SQL] Type coercion of arrays for array_join function
----
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21620: [SPARK-24636][SQL] Type coercion of arrays for array_joi...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21620
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/92298/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #21620: [SPARK-24636][SQL] Type coercion of arrays for ar...
Posted by mgaido91 <gi...@git.apache.org>.
Github user mgaido91 commented on a diff in the pull request:
https://github.com/apache/spark/pull/21620#discussion_r197611266
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/TypeCoercion.scala ---
@@ -536,6 +536,11 @@ object TypeCoercion {
case None => c
}
+ case ArrayJoin(arr, d, nr) if !ArrayType(StringType).acceptsType(arr.dataType) &&
+ ArrayType.acceptsType(arr.dataType) =>
+ val containsNull = arr.dataType.asInstanceOf[ArrayType].containsNull
+ ArrayJoin(Cast(arr, ArrayType(StringType, containsNull)), d, nr)
--- End diff --
not every type can be casted to `StringType`. What about using `ImplicitTypeCasts.implicitCast` in order to check if we can cast it?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21620: [SPARK-24636][SQL] Type coercion of arrays for array_joi...
Posted by mn-mikke <gi...@git.apache.org>.
Github user mn-mikke commented on the issue:
https://github.com/apache/spark/pull/21620
cc @ueshin @mgaido91
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #21620: [SPARK-24636][SQL] Type coercion of arrays for ar...
Posted by mn-mikke <gi...@git.apache.org>.
Github user mn-mikke commented on a diff in the pull request:
https://github.com/apache/spark/pull/21620#discussion_r197730826
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/TypeCoercion.scala ---
@@ -536,6 +536,11 @@ object TypeCoercion {
case None => c
}
+ case ArrayJoin(arr, d, nr) if !ArrayType(StringType).acceptsType(arr.dataType) &&
+ ArrayType.acceptsType(arr.dataType) =>
+ val containsNull = arr.dataType.asInstanceOf[ArrayType].containsNull
+ ArrayJoin(Cast(arr, ArrayType(StringType, containsNull)), d, nr)
--- End diff --
Ok, no problem. Let's support just arrays of primitive types for now. Thanks!
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #21620: [SPARK-24636][SQL] Type coercion of arrays for ar...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/spark/pull/21620
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21620: [SPARK-24636][SQL] Type coercion of arrays for array_joi...
Posted by mgaido91 <gi...@git.apache.org>.
Github user mgaido91 commented on the issue:
https://github.com/apache/spark/pull/21620
LGMT
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #21620: [SPARK-24636][SQL] Type coercion of arrays for ar...
Posted by gatorsmile <gi...@git.apache.org>.
Github user gatorsmile commented on a diff in the pull request:
https://github.com/apache/spark/pull/21620#discussion_r200514583
--- Diff: sql/core/src/test/resources/sql-tests/results/typeCoercion/native/arrayJoin.sql.out ---
@@ -0,0 +1,90 @@
+-- Automatically generated by SQLQueryTestSuite
+-- Number of queries: 11
+
+
+-- !query 0
+SELECT array_join(array(true, false), ', ')
+-- !query 0 schema
+struct<array_join(array(true, false), , ):string>
+-- !query 0 output
+true, false
+
+
+-- !query 1
+SELECT array_join(array(2Y, 1Y), ', ')
+-- !query 1 schema
+struct<array_join(array(2, 1), , ):string>
+-- !query 1 output
+2, 1
+
+
+-- !query 2
+SELECT array_join(array(2S, 1S), ', ')
+-- !query 2 schema
+struct<array_join(array(2, 1), , ):string>
+-- !query 2 output
+2, 1
+
+
+-- !query 3
+SELECT array_join(array(2, 1), ', ')
+-- !query 3 schema
+struct<array_join(array(2, 1), , ):string>
+-- !query 3 output
+2, 1
+
+
+-- !query 4
+SELECT array_join(array(2L, 1L), ', ')
+-- !query 4 schema
+struct<array_join(array(2, 1), , ):string>
+-- !query 4 output
+2, 1
+
+
+-- !query 5
+SELECT array_join(array(9223372036854775809, 9223372036854775808), ', ')
+-- !query 5 schema
+struct<array_join(array(9223372036854775809, 9223372036854775808), , ):string>
+-- !query 5 output
+9223372036854775809, 9223372036854775808
+
+
+-- !query 6
+SELECT array_join(array(2.0D, 1.0D), ', ')
+-- !query 6 schema
+struct<array_join(array(2.0, 1.0), , ):string>
+-- !query 6 output
+2.0, 1.0
+
+
+-- !query 7
+SELECT array_join(array(float(2.0), float(1.0)), ', ')
+-- !query 7 schema
+struct<array_join(array(CAST(2.0 AS FLOAT), CAST(1.0 AS FLOAT)), , ):string>
+-- !query 7 output
+2.0, 1.0
+
+
+-- !query 8
+SELECT array_join(array(date '2016-03-14', date '2016-03-13'), ', ')
+-- !query 8 schema
+struct<array_join(array(DATE '2016-03-14', DATE '2016-03-13'), , ):string>
+-- !query 8 output
+2016-03-14, 2016-03-13
+
+
+-- !query 9
+SELECT array_join(array(timestamp '2016-11-15 20:54:00.000', timestamp '2016-11-12 20:54:00.000'), ', ')
+-- !query 9 schema
+struct<array_join(array(TIMESTAMP('2016-11-15 20:54:00.0'), TIMESTAMP('2016-11-12 20:54:00.0')), , ):string>
--- End diff --
If the input array is very long, the automatically generated column name will be also super long?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21620: [SPARK-24636][SQL] Type coercion of arrays for array_joi...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21620
**[Test build #92298 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/92298/testReport)** for PR 21620 at commit [`3082947`](https://github.com/apache/spark/commit/3082947e53dd55946d264dfa0eeccddd59764aff).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21620: [SPARK-24636][SQL] Type coercion of arrays for array_joi...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21620
**[Test build #92297 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/92297/testReport)** for PR 21620 at commit [`435b7ed`](https://github.com/apache/spark/commit/435b7ed467bb3920bbeacc0438c420a6d3423012).
* This patch **fails PySpark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21620: [SPARK-24636][SQL] Type coercion of arrays for array_joi...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21620
**[Test build #92298 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/92298/testReport)** for PR 21620 at commit [`3082947`](https://github.com/apache/spark/commit/3082947e53dd55946d264dfa0eeccddd59764aff).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #21620: [SPARK-24636][SQL] Type coercion of arrays for ar...
Posted by mgaido91 <gi...@git.apache.org>.
Github user mgaido91 commented on a diff in the pull request:
https://github.com/apache/spark/pull/21620#discussion_r197726949
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/TypeCoercion.scala ---
@@ -536,6 +536,11 @@ object TypeCoercion {
case None => c
}
+ case ArrayJoin(arr, d, nr) if !ArrayType(StringType).acceptsType(arr.dataType) &&
+ ArrayType.acceptsType(arr.dataType) =>
+ val containsNull = arr.dataType.asInstanceOf[ArrayType].containsNull
+ ArrayJoin(Cast(arr, ArrayType(StringType, containsNull)), d, nr)
--- End diff --
I am not sure. I think it is arguable which is the right result of `SELECT array_join(array(array('a', 'b'), array('c', 'd')), ';')` for instance. With this PR, the result is `[a, b];[c, d]` but shouldn't it be `[a;b];[c;d]`? Moreover, Presto, which is the reference here, doesn't support nested arrays for instance:
```
presto> select array_join(array[array[1, 2, 3], array[3, 4, 5]], ';');
Query 20180625_090549_00003_bsbcg failed: Input type array(integer) not supported
```
So, I'd avoid that.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21620: [SPARK-24636][SQL] Type coercion of arrays for array_joi...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21620
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/92299/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21620: [SPARK-24636][SQL] Type coercion of arrays for array_joi...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21620
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #21620: [SPARK-24636][SQL] Type coercion of arrays for ar...
Posted by mn-mikke <gi...@git.apache.org>.
Github user mn-mikke commented on a diff in the pull request:
https://github.com/apache/spark/pull/21620#discussion_r197768113
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/TypeCoercion.scala ---
@@ -536,6 +536,14 @@ object TypeCoercion {
case None => c
}
+ case aj @ ArrayJoin(arr, d, nr) if !ArrayType(StringType).acceptsType(arr.dataType) &&
+ ArrayType.acceptsType(arr.dataType) =>
+ val containsNull = arr.dataType.asInstanceOf[ArrayType].containsNull
+ ImplicitTypeCasts.implicitCast(arr, ArrayType(StringType, containsNull)) match {
+ case Some(finalDataType) => ArrayJoin(finalDataType, d, nr)
--- End diff --
spot on, thanks!
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21620: [SPARK-24636][SQL] Type coercion of arrays for array_joi...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21620
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21620: [SPARK-24636][SQL] Type coercion of arrays for array_joi...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21620
Can one of the admins verify this patch?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21620: [SPARK-24636][SQL] Type coercion of arrays for array_joi...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21620
Merged build finished. Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21620: [SPARK-24636][SQL] Type coercion of arrays for array_joi...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21620
**[Test build #92253 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/92253/testReport)** for PR 21620 at commit [`9801771`](https://github.com/apache/spark/commit/98017714e06ea7a730b1515d5bcfd583d7808a5a).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #21620: [SPARK-24636][SQL] Type coercion of arrays for ar...
Posted by mgaido91 <gi...@git.apache.org>.
Github user mgaido91 commented on a diff in the pull request:
https://github.com/apache/spark/pull/21620#discussion_r197737278
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/TypeCoercion.scala ---
@@ -536,6 +536,13 @@ object TypeCoercion {
case None => c
}
+ case aj @ ArrayJoin(arr, d, nr) if !ArrayType(StringType).acceptsType(arr.dataType) &&
+ ArrayType.acceptsType(arr.dataType) =>
+ val containsNull = arr.dataType.asInstanceOf[ArrayType].containsNull
+ ImplicitTypeCasts.implicitCast(arr, ArrayType(StringType, containsNull))
--- End diff --
nit: usually we try to avoid long chains of functions, what about:
```
ImplicitTypeCasts.implicitCast(arr, ArrayType(StringType, containsNull)) match {
...
}
```
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21620: [SPARK-24636][SQL] Type coercion of arrays for array_joi...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21620
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/92297/
Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21620: [SPARK-24636][SQL] Type coercion of arrays for array_joi...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21620
Can one of the admins verify this patch?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #21620: [SPARK-24636][SQL] Type coercion of arrays for ar...
Posted by mn-mikke <gi...@git.apache.org>.
Github user mn-mikke commented on a diff in the pull request:
https://github.com/apache/spark/pull/21620#discussion_r197716236
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/TypeCoercion.scala ---
@@ -536,6 +536,11 @@ object TypeCoercion {
case None => c
}
+ case ArrayJoin(arr, d, nr) if !ArrayType(StringType).acceptsType(arr.dataType) &&
+ ArrayType.acceptsType(arr.dataType) =>
+ val containsNull = arr.dataType.asInstanceOf[ArrayType].containsNull
+ ArrayJoin(Cast(arr, ArrayType(StringType, containsNull)), d, nr)
--- End diff --
Hi @mgaido91,
to be honest, I've considered this option before submitting this PR. But I'm glad that you mentioned this approach. At least, we can discuss pros and cons of different solutions. Usage of ```ImplicitTypeCasts.implicitCast``` would enable conversion only from primitive types. I think it would be nice to support non-primitive types as well. WDYT?
Re: Casting to ```StringType```: According to ```Cast.canCast``` method should be possible to cast any type to ```StringType```:
**line 42:** ``` case (_, StringType) => true```
Or am I missing something? I hope test cases in *.../typeCoercion/native/arrayJoin.sql* cover to ```StringType``` conversions from all Spark types.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21620: [SPARK-24636][SQL] Type coercion of arrays for array_joi...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21620
**[Test build #92299 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/92299/testReport)** for PR 21620 at commit [`2aef9eb`](https://github.com/apache/spark/commit/2aef9eb7cd638861147b912d2ac604e2f3eaec7e).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21620: [SPARK-24636][SQL] Type coercion of arrays for array_joi...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21620
**[Test build #92299 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/92299/testReport)** for PR 21620 at commit [`2aef9eb`](https://github.com/apache/spark/commit/2aef9eb7cd638861147b912d2ac604e2f3eaec7e).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21620: [SPARK-24636][SQL] Type coercion of arrays for array_joi...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21620
**[Test build #92297 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/92297/testReport)** for PR 21620 at commit [`435b7ed`](https://github.com/apache/spark/commit/435b7ed467bb3920bbeacc0438c420a6d3423012).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #21620: [SPARK-24636][SQL] Type coercion of arrays for ar...
Posted by ueshin <gi...@git.apache.org>.
Github user ueshin commented on a diff in the pull request:
https://github.com/apache/spark/pull/21620#discussion_r200537707
--- Diff: sql/core/src/test/resources/sql-tests/results/typeCoercion/native/arrayJoin.sql.out ---
@@ -0,0 +1,90 @@
+-- Automatically generated by SQLQueryTestSuite
+-- Number of queries: 11
+
+
+-- !query 0
+SELECT array_join(array(true, false), ', ')
+-- !query 0 schema
+struct<array_join(array(true, false), , ):string>
+-- !query 0 output
+true, false
+
+
+-- !query 1
+SELECT array_join(array(2Y, 1Y), ', ')
+-- !query 1 schema
+struct<array_join(array(2, 1), , ):string>
+-- !query 1 output
+2, 1
+
+
+-- !query 2
+SELECT array_join(array(2S, 1S), ', ')
+-- !query 2 schema
+struct<array_join(array(2, 1), , ):string>
+-- !query 2 output
+2, 1
+
+
+-- !query 3
+SELECT array_join(array(2, 1), ', ')
+-- !query 3 schema
+struct<array_join(array(2, 1), , ):string>
+-- !query 3 output
+2, 1
+
+
+-- !query 4
+SELECT array_join(array(2L, 1L), ', ')
+-- !query 4 schema
+struct<array_join(array(2, 1), , ):string>
+-- !query 4 output
+2, 1
+
+
+-- !query 5
+SELECT array_join(array(9223372036854775809, 9223372036854775808), ', ')
+-- !query 5 schema
+struct<array_join(array(9223372036854775809, 9223372036854775808), , ):string>
+-- !query 5 output
+9223372036854775809, 9223372036854775808
+
+
+-- !query 6
+SELECT array_join(array(2.0D, 1.0D), ', ')
+-- !query 6 schema
+struct<array_join(array(2.0, 1.0), , ):string>
+-- !query 6 output
+2.0, 1.0
+
+
+-- !query 7
+SELECT array_join(array(float(2.0), float(1.0)), ', ')
+-- !query 7 schema
+struct<array_join(array(CAST(2.0 AS FLOAT), CAST(1.0 AS FLOAT)), , ):string>
+-- !query 7 output
+2.0, 1.0
+
+
+-- !query 8
+SELECT array_join(array(date '2016-03-14', date '2016-03-13'), ', ')
+-- !query 8 schema
+struct<array_join(array(DATE '2016-03-14', DATE '2016-03-13'), , ):string>
+-- !query 8 output
+2016-03-14, 2016-03-13
+
+
+-- !query 9
+SELECT array_join(array(timestamp '2016-11-15 20:54:00.000', timestamp '2016-11-12 20:54:00.000'), ', ')
+-- !query 9 schema
+struct<array_join(array(TIMESTAMP('2016-11-15 20:54:00.0'), TIMESTAMP('2016-11-12 20:54:00.0')), , ):string>
--- End diff --
Hm, yes, it will be. In general, if an expression has `children: Seq[Expression]` as its argument, the automatically generated column name will be long for now?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21620: [SPARK-24636][SQL] Type coercion of arrays for array_joi...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21620
**[Test build #92253 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/92253/testReport)** for PR 21620 at commit [`9801771`](https://github.com/apache/spark/commit/98017714e06ea7a730b1515d5bcfd583d7808a5a).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21620: [SPARK-24636][SQL] Type coercion of arrays for array_joi...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21620
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #21620: [SPARK-24636][SQL] Type coercion of arrays for ar...
Posted by ueshin <gi...@git.apache.org>.
Github user ueshin commented on a diff in the pull request:
https://github.com/apache/spark/pull/21620#discussion_r197766352
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/TypeCoercion.scala ---
@@ -536,6 +536,14 @@ object TypeCoercion {
case None => c
}
+ case aj @ ArrayJoin(arr, d, nr) if !ArrayType(StringType).acceptsType(arr.dataType) &&
+ ArrayType.acceptsType(arr.dataType) =>
+ val containsNull = arr.dataType.asInstanceOf[ArrayType].containsNull
+ ImplicitTypeCasts.implicitCast(arr, ArrayType(StringType, containsNull)) match {
+ case Some(finalDataType) => ArrayJoin(finalDataType, d, nr)
--- End diff --
`castedArr` or something instead of `finalDataType`?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21620: [SPARK-24636][SQL] Type coercion of arrays for array_joi...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21620
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/92253/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21620: [SPARK-24636][SQL] Type coercion of arrays for array_joi...
Posted by HyukjinKwon <gi...@git.apache.org>.
Github user HyukjinKwon commented on the issue:
https://github.com/apache/spark/pull/21620
Merged to master.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org