You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by dongjoon-hyun <gi...@git.apache.org> on 2018/10/25 17:03:59 UTC
[GitHub] spark pull request #22828: [SPARK-25832][SQL] remove newly added map related...
GitHub user dongjoon-hyun opened a pull request:
https://github.com/apache/spark/pull/22828
[SPARK-25832][SQL] remove newly added map related functions
## What changes were proposed in this pull request?
This aims to supercede https://github.com/apache/spark/pull/22821 . The main author is @cloud-fan .
## How was this patch tested?
Pass the Jenkins.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/dongjoon-hyun/spark SPARK-25832
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/22828.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #22828
----
commit 7e919e3efa3d669a6893b47c737e553795d94347
Author: Wenchen Fan <we...@...>
Date: 2018-10-25T04:42:17Z
remove newly added map related functions from FunctionRegistry
commit 726fc3014bbfdc5e84f12ff42ec69b5e567c7af8
Author: Wenchen Fan <we...@...>
Date: 2018-10-25T14:05:30Z
fix tests
commit 71d3b3ca87ec10616c5aab095652649fa48bb618
Author: Dongjoon Hyun <do...@...>
Date: 2018-10-25T15:38:34Z
Remove
----
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22828: [SPARK-25832][SQL] remove newly added map related functi...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22828
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22828: [SPARK-25832][SQL] Remove newly added map related functi...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22828
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #22828: [SPARK-25832][SQL] remove newly added map related...
Posted by dongjoon-hyun <gi...@git.apache.org>.
Github user dongjoon-hyun commented on a diff in the pull request:
https://github.com/apache/spark/pull/22828#discussion_r228257902
--- Diff: python/pyspark/sql/functions.py ---
@@ -2540,26 +2540,6 @@ def map_values(col):
return Column(sc._jvm.functions.map_values(_to_java_column(col)))
-@since(2.4)
-def map_entries(col):
- """
- Collection function: Returns an unordered array of all entries in the given map.
-
- :param col: name of column or expression
-
- >>> from pyspark.sql.functions import map_entries
- >>> df = spark.sql("SELECT map(1, 'a', 2, 'b') as data")
- >>> df.select(map_entries("data").alias("entries")).show()
- +----------------+
- | entries|
- +----------------+
- |[[1, a], [2, b]]|
- +----------------+
- """
- sc = SparkContext._active_spark_context
- return Column(sc._jvm.functions.map_entries(_to_java_column(col)))
-
-
--- End diff --
Could you review this, @HyukjinKwon and @BryanCutler ?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #22828: [SPARK-25832][SQL] remove newly added map related...
Posted by dongjoon-hyun <gi...@git.apache.org>.
Github user dongjoon-hyun commented on a diff in the pull request:
https://github.com/apache/spark/pull/22828#discussion_r228259220
--- Diff: sql/catalyst/src/main/java/org/apache/spark/sql/catalyst/expressions/UnsafeRow.java ---
@@ -61,8 +61,6 @@
*/
public final class UnsafeRow extends InternalRow implements Externalizable, KryoSerializable {
- public static final int WORD_SIZE = 8;
-
--- End diff --
This is added by [map_entries](https://github.com/apache/spark/commit/a6e883feb3b78232ad5cf636f7f7d5e825183041#diff-fa494816e13a1d53c1c55af01bc7a396R65)
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #22828: [SPARK-25832][SQL] Remove newly added map related...
Posted by dongjoon-hyun <gi...@git.apache.org>.
Github user dongjoon-hyun closed the pull request at:
https://github.com/apache/spark/pull/22828
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22828: [SPARK-25832][SQL] remove newly added map related functi...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22828
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/4487/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22828: [SPARK-25832][SQL] remove newly added map related functi...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/22828
**[Test build #98030 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/98030/testReport)** for PR 22828 at commit [`71d3b3c`](https://github.com/apache/spark/commit/71d3b3ca87ec10616c5aab095652649fa48bb618).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22828: [SPARK-25832][SQL] Remove newly added map related functi...
Posted by dongjoon-hyun <gi...@git.apache.org>.
Github user dongjoon-hyun commented on the issue:
https://github.com/apache/spark/pull/22828
Oh, I'm closing this in favor of https://github.com/apache/spark/pull/22827.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22828: [SPARK-25832][SQL] Remove newly added map related functi...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22828
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/98030/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #22828: [SPARK-25832][SQL] remove newly added map related...
Posted by dongjoon-hyun <gi...@git.apache.org>.
Github user dongjoon-hyun commented on a diff in the pull request:
https://github.com/apache/spark/pull/22828#discussion_r228258092
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/FunctionRegistry.scala ---
@@ -433,13 +432,9 @@ object FunctionRegistry {
expression[ArrayRemove]("array_remove"),
expression[ArrayDistinct]("array_distinct"),
expression[ArrayTransform]("transform"),
- expression[MapFilter]("map_filter"),
expression[ArrayFilter]("filter"),
expression[ArrayExists]("exists"),
expression[ArrayAggregate]("aggregate"),
- expression[TransformValues]("transform_values"),
- expression[TransformKeys]("transform_keys"),
- expression[MapZipWith]("map_zip_with"),
--- End diff --
Could you review this, @gatorsmile and @cloud-fan ?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22828: [SPARK-25832][SQL] Remove newly added map related functi...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/22828
**[Test build #98030 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/98030/testReport)** for PR 22828 at commit [`71d3b3c`](https://github.com/apache/spark/commit/71d3b3ca87ec10616c5aab095652649fa48bb618).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #22828: [SPARK-25832][SQL] remove newly added map related...
Posted by dongjoon-hyun <gi...@git.apache.org>.
Github user dongjoon-hyun commented on a diff in the pull request:
https://github.com/apache/spark/pull/22828#discussion_r228257685
--- Diff: R/pkg/NAMESPACE ---
@@ -313,7 +313,6 @@ exportMethods("%<=>%",
"lower",
"lpad",
"ltrim",
- "map_entries",
--- End diff --
Could you review this, @felixcheung ?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22828: [SPARK-25832][SQL] remove newly added map related functi...
Posted by dongjoon-hyun <gi...@git.apache.org>.
Github user dongjoon-hyun commented on the issue:
https://github.com/apache/spark/pull/22828
To run the Jenkins faster, I create a standalone PR instead of making a PR to #22821 .
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org