You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by gatorsmile <gi...@git.apache.org> on 2018/10/25 16:51:46 UTC
[GitHub] spark pull request #22827: [SPARK-25832][SQL] Revert newly added map related...
GitHub user gatorsmile opened a pull request:
https://github.com/apache/spark/pull/22827
[SPARK-25832][SQL] Revert newly added map related functions
## What changes were proposed in this pull request?
- Revert [SPARK-23935][SQL] Adding map_entries function: https://github.com/apache/spark/pull/21236
- Revert [SPARK-23937][SQL] Add map_filter SQL function: https://github.com/apache/spark/pull/21986
- Revert [SPARK-23940][SQL] Add transform_values SQL function: https://github.com/apache/spark/pull/22045
- Revert [SPARK-23939][SQL] Add transform_keys function: https://github.com/apache/spark/pull/22013
- Revert [SPARK-23932][SQL] Higher order function zip_with: https://github.com/apache/spark/pull/22031
- Revert [SPARK-23938][SQL] Add map_zip_with function: https://github.com/apache/spark/pull/22017
- Revert the changes of map_entries in [SPARK-24331][SPARKR][SQL] Adding arrays_overlap, array_repeat, map_entries to SparkR: https://github.com/apache/spark/pull/21434/
## How was this patch tested?
The existing tests.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/gatorsmile/spark revertMap2.4
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/22827.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #22827
----
commit b8cab59abc45d9178785829f6a455e8a7a6a61fa
Author: gatorsmile <ga...@...>
Date: 2018-10-25T16:05:20Z
Revert [SPARK-23935][SQL] Adding map_entries function
commit 25498322863ccad2a1481c939dcd0392862baf55
Author: gatorsmile <ga...@...>
Date: 2018-10-25T16:18:40Z
Revert [SPARK-23937][SQL] Add map_filter SQL function
commit 8623f864366ba2e7a38150ee4aa119e90b050155
Author: gatorsmile <ga...@...>
Date: 2018-10-25T16:23:17Z
Revert [SPARK-23940][SQL] Add transform_values SQL function
commit 666d6ab25f6d12f27510740badbf7501fd08168a
Author: gatorsmile <ga...@...>
Date: 2018-10-25T16:27:03Z
Revert [SPARK-23939][SQL] Add transform_keys function
commit 59393872427866d032eefe2e04e7ac6d94e52219
Author: gatorsmile <ga...@...>
Date: 2018-10-25T16:34:19Z
Revert [SPARK-23938][SQL] Add map_zip_with function
commit f1f3d0b09527d28fd8a1f1a2c8e2358c10da175b
Author: gatorsmile <ga...@...>
Date: 2018-10-25T16:40:55Z
Revert the changes of map_entries in [SPARK-24331][SPARKR][SQL] Adding arrays_overlap, array_repeat, map_entries to SparkR
----
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22827: [SPARK-25832][SQL] Revert newly added map related functi...
Posted by gatorsmile <gi...@git.apache.org>.
Github user gatorsmile commented on the issue:
https://github.com/apache/spark/pull/22827
@dongjoon-hyun I think that can be kept.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22827: [SPARK-25832][SQL] Revert newly added map related functi...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22827
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/98029/
Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22827: [SPARK-25832][SQL] Revert newly added map related functi...
Posted by gatorsmile <gi...@git.apache.org>.
Github user gatorsmile commented on the issue:
https://github.com/apache/spark/pull/22827
@felixcheung @mengxr The last commit https://github.com/apache/spark/pull/22827/commits/f1f3d0b09527d28fd8a1f1a2c8e2358c10da175b is for R related revert. Please take a look
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22827: [SPARK-25832][SQL][BRANCH-2.4] Revert newly added map re...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/22827
**[Test build #98034 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/98034/testReport)** for PR 22827 at commit [`8bdf02e`](https://github.com/apache/spark/commit/8bdf02eb71a30ece8dd5f17895697455218fbcfc).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22827: [SPARK-25832][SQL] Revert newly added map related functi...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22827
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #22827: [SPARK-25832][SQL] Revert newly added map related...
Posted by dongjoon-hyun <gi...@git.apache.org>.
Github user dongjoon-hyun commented on a diff in the pull request:
https://github.com/apache/spark/pull/22827#discussion_r228267271
--- Diff: sql/core/src/test/resources/sql-tests/inputs/higher-order-functions.sql ---
@@ -60,8 +60,3 @@ select zip_with(array('a', 'b', 'c'), array('d', 'e', 'f'), (x, y) -> concat(x,
-- Zip with array coalesce
select zip_with(array('a'), array('d', null, 'f'), (x, y) -> coalesce(x, y)) as v;
-
-create or replace temporary view nested as values
- (1, map(1, 1, 2, 2, 3, 3)),
- (2, map(4, 4, 5, 5, 6, 6))
- as t(x, ys);
--- End diff --
`higher-order-functions.sql.out`?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #22827: [SPARK-25832][SQL] Revert newly added map related...
Posted by dongjoon-hyun <gi...@git.apache.org>.
Github user dongjoon-hyun commented on a diff in the pull request:
https://github.com/apache/spark/pull/22827#discussion_r228269628
--- Diff: sql/core/src/test/resources/sql-tests/inputs/higher-order-functions.sql ---
@@ -60,8 +60,3 @@ select zip_with(array('a', 'b', 'c'), array('d', 'e', 'f'), (x, y) -> concat(x,
-- Zip with array coalesce
select zip_with(array('a'), array('d', null, 'f'), (x, y) -> coalesce(x, y)) as v;
-
-create or replace temporary view nested as values
- (1, map(1, 1, 2, 2, 3, 3)),
- (2, map(4, 4, 5, 5, 6, 6))
- as t(x, ys);
--- End diff --
Oops. Right!
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22827: [SPARK-25832][SQL] Revert newly added map related functi...
Posted by dongjoon-hyun <gi...@git.apache.org>.
Github user dongjoon-hyun commented on the issue:
https://github.com/apache/spark/pull/22827
Also, please remove `public static final int WORD_SIZE = 8;` in UnsafeRow.java. It's added by `map_entries`.
- https://github.com/apache/spark/pull/22828/files?utf8=%E2%9C%93&diff=unified&w=1#diff-fa494816e13a1d53c1c55af01bc7a396L64
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22827: [SPARK-25832][SQL] Revert newly added map related functi...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/22827
**[Test build #98034 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/98034/testReport)** for PR 22827 at commit [`8bdf02e`](https://github.com/apache/spark/commit/8bdf02eb71a30ece8dd5f17895697455218fbcfc).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #22827: [SPARK-25832][SQL][BRANCH-2.4] Revert newly added...
Posted by gatorsmile <gi...@git.apache.org>.
Github user gatorsmile closed the pull request at:
https://github.com/apache/spark/pull/22827
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22827: [SPARK-25832][SQL] Revert newly added map related functi...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22827
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/4486/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22827: [SPARK-25832][SQL] Revert newly added map related functi...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/22827
**[Test build #98029 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/98029/testReport)** for PR 22827 at commit [`f1f3d0b`](https://github.com/apache/spark/commit/f1f3d0b09527d28fd8a1f1a2c8e2358c10da175b).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #22827: [SPARK-25832][SQL] Revert newly added map related...
Posted by gatorsmile <gi...@git.apache.org>.
Github user gatorsmile commented on a diff in the pull request:
https://github.com/apache/spark/pull/22827#discussion_r228267525
--- Diff: sql/core/src/test/resources/sql-tests/inputs/higher-order-functions.sql ---
@@ -60,8 +60,3 @@ select zip_with(array('a', 'b', 'c'), array('d', 'e', 'f'), (x, y) -> concat(x,
-- Zip with array coalesce
select zip_with(array('a'), array('d', null, 'f'), (x, y) -> coalesce(x, y)) as v;
-
-create or replace temporary view nested as values
- (1, map(1, 1, 2, 2, 3, 3)),
- (2, map(4, 4, 5, 5, 6, 6))
- as t(x, ys);
--- End diff --
That has been removed already
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #22827: [SPARK-25832][SQL] Revert newly added map related...
Posted by dongjoon-hyun <gi...@git.apache.org>.
Github user dongjoon-hyun commented on a diff in the pull request:
https://github.com/apache/spark/pull/22827#discussion_r228263430
--- Diff: sql/core/src/test/resources/sql-tests/inputs/higher-order-functions.sql ---
@@ -65,21 +65,3 @@ create or replace temporary view nested as values
(1, map(1, 1, 2, 2, 3, 3)),
(2, map(4, 4, 5, 5, 6, 6))
as t(x, ys);
--- End diff --
We should revert line 64~67 because they are introduced by [SPARK-23939 Add transform_keys function](https://github.com/apache/spark/commit/5b4a38d826807ea6733e4382c8f9b82a355a6eb4).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22827: [SPARK-25832][SQL][BRANCH-2.4] Revert newly added map re...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22827
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22827: [SPARK-25832][SQL][BRANCH-2.4] Revert newly added map re...
Posted by HyukjinKwon <gi...@git.apache.org>.
Github user HyukjinKwon commented on the issue:
https://github.com/apache/spark/pull/22827
LGTM too
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22827: [SPARK-25832][SQL] Revert newly added map related functi...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22827
Merged build finished. Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22827: [SPARK-25832][SQL] Revert newly added map related functi...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/22827
**[Test build #98029 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/98029/testReport)** for PR 22827 at commit [`f1f3d0b`](https://github.com/apache/spark/commit/f1f3d0b09527d28fd8a1f1a2c8e2358c10da175b).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22827: [SPARK-25832][SQL][BRANCH-2.4] Revert newly added map re...
Posted by cloud-fan <gi...@git.apache.org>.
Github user cloud-fan commented on the issue:
https://github.com/apache/spark/pull/22827
thanks, merging to 2.4!
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22827: [SPARK-25832][SQL][BRANCH-2.4] Revert newly added map re...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22827
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/98034/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22827: [SPARK-25832][SQL] Revert newly added map related functi...
Posted by dongjoon-hyun <gi...@git.apache.org>.
Github user dongjoon-hyun commented on the issue:
https://github.com/apache/spark/pull/22827
@gatorsmile . Could you put `[BRANCH-2.4]` into PR title?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22827: [SPARK-25832][SQL] Revert newly added map related functi...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22827
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22827: [SPARK-25832][SQL] Revert newly added map related functi...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22827
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/4491/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22827: [SPARK-25832][SQL] Revert newly added map related functi...
Posted by gatorsmile <gi...@git.apache.org>.
Github user gatorsmile commented on the issue:
https://github.com/apache/spark/pull/22827
@cloud-fan @dongjoon-hyun @rxin @felixcheung
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org