You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by kiszk <gi...@git.apache.org> on 2018/04/12 13:23:56 UTC
[GitHub] spark pull request #21053: [SPARK-23924][SQL] Add element_at function
GitHub user kiszk opened a pull request:
https://github.com/apache/spark/pull/21053
[SPARK-23924][SQL] Add element_at function
## What changes were proposed in this pull request?
The PR adds the SQL function array_position. The behavior of the function is based on Presto's one.
This function returns element of array at given index in value if column is array, or returns value for the given key in value if column is map.
## How was this patch tested?
Added UTs
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/kiszk/spark SPARK-23924
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/21053.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #21053
----
commit d7ec6ccc177d07a8090ac27ce1659f427d1cf50a
Author: Kazuaki Ishizaki <is...@...>
Date: 2018-04-12T13:21:22Z
initial commit
----
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21053: [SPARK-23924][SQL] Add element_at function
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21053
**[Test build #89321 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/89321/testReport)** for PR 21053 at commit [`bb0ab45`](https://github.com/apache/spark/commit/bb0ab45b4a9bbf1155dbb9513508bbef3685b3f6).
* This patch **fails PySpark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21053: [SPARK-23924][SQL] Add element_at function
Posted by kiszk <gi...@git.apache.org>.
Github user kiszk commented on the issue:
https://github.com/apache/spark/pull/21053
thank you very much for your comments
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21053: [SPARK-23924][SQL] Add element_at function
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21053
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/89476/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21053: [SPARK-23924][SQL] Add element_at function
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21053
Merged build finished. Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #21053: [SPARK-23924][SQL] Add element_at function
Posted by kiszk <gi...@git.apache.org>.
Github user kiszk commented on a diff in the pull request:
https://github.com/apache/spark/pull/21053#discussion_r182350430
--- Diff: python/pyspark/sql/functions.py ---
@@ -1846,6 +1846,28 @@ def array_contains(col, value):
return Column(sc._jvm.functions.array_contains(_to_java_column(col), value))
+@ignore_unicode_prefix
+@since(2.4)
+def element_at(col, value):
--- End diff --
Umm, the 2nd argument acts as `index` for an array or `key` for a map. This is why I used `value`.
Can I use `index`?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21053: [SPARK-23924][SQL] Add element_at function
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21053
Merged build finished. Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21053: [SPARK-23924][SQL] Add element_at function
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21053
**[Test build #89321 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/89321/testReport)** for PR 21053 at commit [`bb0ab45`](https://github.com/apache/spark/commit/bb0ab45b4a9bbf1155dbb9513508bbef3685b3f6).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21053: [SPARK-23924][SQL] Add element_at function
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21053
**[Test build #89403 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/89403/testReport)** for PR 21053 at commit [`5bdee7e`](https://github.com/apache/spark/commit/5bdee7e5d80abe3bfe765fe8401b251b424d7ef6).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21053: [SPARK-23924][SQL] Add element_at function
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21053
**[Test build #89553 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/89553/testReport)** for PR 21053 at commit [`96dd82b`](https://github.com/apache/spark/commit/96dd82b6385e5114496d0541d744c9fce48f8db0).
* This patch **fails due to an unknown error code, -9**.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #21053: [SPARK-23924][SQL] Add element_at function
Posted by ueshin <gi...@git.apache.org>.
Github user ueshin commented on a diff in the pull request:
https://github.com/apache/spark/pull/21053#discussion_r182323290
--- Diff: python/pyspark/sql/functions.py ---
@@ -1846,6 +1846,28 @@ def array_contains(col, value):
return Column(sc._jvm.functions.array_contains(_to_java_column(col), value))
+@ignore_unicode_prefix
+@since(2.4)
+def element_at(col, value):
+ """
+ Collection function: returns element of array at given index in value if col is array.
+ returns value for the given key in value if col is map.
+
+ :param col: name of column containing array or map
+ :param value: value to check for in array or key to check for in map
--- End diff --
Can you add a note or something to notice the index is 1-based.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #21053: [SPARK-23924][SQL] Add element_at function
Posted by ueshin <gi...@git.apache.org>.
Github user ueshin commented on a diff in the pull request:
https://github.com/apache/spark/pull/21053#discussion_r181617303
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/complexTypeExtractors.scala ---
@@ -354,3 +336,37 @@ case class GetMapValue(child: Expression, key: Expression)
})
}
}
+
+/**
+ * Returns the value of key `key` in Map `child`.
+ *
+ * We need to do type checking here as `key` expression maybe unresolved.
+ */
+case class GetMapValue(child: Expression, key: Expression) extends GetMapValueUtil
+ with ExtractValue with NullIntolerant {
--- End diff --
nit: maybe `extends ...` should be this line.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21053: [SPARK-23924][SQL] Add element_at function
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21053
**[Test build #89554 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/89554/testReport)** for PR 21053 at commit [`90e026e`](https://github.com/apache/spark/commit/90e026e9b2d58e17995d21e44c5a68fa7f0f7d52).
* This patch **fails due to an unknown error code, -9**.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21053: [SPARK-23924][SQL] Add element_at function
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21053
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/2280/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21053: [SPARK-23924][SQL] Add element_at function
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21053
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21053: [SPARK-23924][SQL] Add element_at function
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21053
**[Test build #89553 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/89553/testReport)** for PR 21053 at commit [`96dd82b`](https://github.com/apache/spark/commit/96dd82b6385e5114496d0541d744c9fce48f8db0).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21053: [SPARK-23924][SQL] Add element_at function
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21053
**[Test build #89528 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/89528/testReport)** for PR 21053 at commit [`68e8907`](https://github.com/apache/spark/commit/68e89070a9c046907d35b5ce6802a498860a8302).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21053: [SPARK-23924][SQL] Add element_at function
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21053
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/2325/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21053: [SPARK-23924][SQL] Add element_at function
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21053
**[Test build #89266 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/89266/testReport)** for PR 21053 at commit [`d7ec6cc`](https://github.com/apache/spark/commit/d7ec6ccc177d07a8090ac27ce1659f427d1cf50a).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds the following public classes _(experimental)_:
* `case class ElementAt(left: Expression, right: Expression) extends GetMapValueUtil `
* `abstract class GetMapValueUtil extends BinaryExpression with ImplicitCastInputTypes `
* `case class GetMapValue(child: Expression, key: Expression) extends GetMapValueUtil`
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #21053: [SPARK-23924][SQL] Add element_at function
Posted by viirya <gi...@git.apache.org>.
Github user viirya commented on a diff in the pull request:
https://github.com/apache/spark/pull/21053#discussion_r181622879
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/collectionOperations.scala ---
@@ -287,3 +287,106 @@ case class ArrayContains(left: Expression, right: Expression)
override def prettyName: String = "array_contains"
}
+
+/**
+ * Returns the value of index `right` in Array `left` or the value for key `right` in Map `left`.
+ */
+@ExpressionDescription(
+ usage = """
+ _FUNC_(array, index) - Returns element of array at given index. If index < 0, accesses elements
+ from the last to the first.
+
+ _FUNC_(map, key) - Returns value for given key, or NULL if the key is not contained in the map
+ """,
+ examples = """
+ Examples:
+ > SELECT _FUNC_(array(1, 2, 3), 2);
+ 2
+ > SELECT _FUNC_(map(1, 'a', 2, 'b'), 2);
+ "b"
+ """,
+ since = "2.4.0")
+case class ElementAt(left: Expression, right: Expression) extends GetMapValueUtil {
+
+ override def dataType: DataType = left.dataType match {
+ case _: ArrayType => left.dataType.asInstanceOf[ArrayType].elementType
+ case _: MapType => left.dataType.asInstanceOf[MapType].valueType
+ }
+
+ override def inputTypes: Seq[AbstractDataType] = {
+ Seq(TypeCollection(ArrayType, MapType),
+ left.dataType match {
+ case _: ArrayType => IntegerType
+ case _: MapType => left.dataType.asInstanceOf[MapType].keyType
+ }
+ )
+ }
+
+ override def nullable: Boolean = true
--- End diff --
Maybe?
```scala
override def nullable: Boolean = left.dataType match {
case a: ArrayType => a.containsNull
case m: MapType => m.valueContainsNull
} || left.nullable || right.nullable
```
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21053: [SPARK-23924][SQL] Add element_at function
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21053
**[Test build #89266 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/89266/testReport)** for PR 21053 at commit [`d7ec6cc`](https://github.com/apache/spark/commit/d7ec6ccc177d07a8090ac27ce1659f427d1cf50a).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21053: [SPARK-23924][SQL] Add element_at function
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21053
Merged build finished. Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21053: [SPARK-23924][SQL] Add element_at function
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21053
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21053: [SPARK-23924][SQL] Add element_at function
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21053
Merged build finished. Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21053: [SPARK-23924][SQL] Add element_at function
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21053
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/89552/
Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21053: [SPARK-23924][SQL] Add element_at function
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21053
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/89266/
Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21053: [SPARK-23924][SQL] Add element_at function
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21053
**[Test build #89403 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/89403/testReport)** for PR 21053 at commit [`5bdee7e`](https://github.com/apache/spark/commit/5bdee7e5d80abe3bfe765fe8401b251b424d7ef6).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds the following public classes _(experimental)_:
* `case class GetMapValue(child: Expression, key: Expression)`
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21053: [SPARK-23924][SQL] Add element_at function
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21053
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/2430/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21053: [SPARK-23924][SQL] Add element_at function
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21053
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/89336/
Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21053: [SPARK-23924][SQL] Add element_at function
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21053
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/2314/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21053: [SPARK-23924][SQL] Add element_at function
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21053
**[Test build #89313 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/89313/testReport)** for PR 21053 at commit [`bb0ab45`](https://github.com/apache/spark/commit/bb0ab45b4a9bbf1155dbb9513508bbef3685b3f6).
* This patch **fails due to an unknown error code, -9**.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21053: [SPARK-23924][SQL] Add element_at function
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21053
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/2407/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21053: [SPARK-23924][SQL] Add element_at function
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21053
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/2474/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21053: [SPARK-23924][SQL] Add element_at function
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21053
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/2349/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21053: [SPARK-23924][SQL] Add element_at function
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21053
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/2450/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21053: [SPARK-23924][SQL] Add element_at function
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21053
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/2475/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21053: [SPARK-23924][SQL] Add element_at function
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21053
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21053: [SPARK-23924][SQL] Add element_at function
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21053
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21053: [SPARK-23924][SQL] Add element_at function
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21053
Merged build finished. Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21053: [SPARK-23924][SQL] Add element_at function
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21053
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21053: [SPARK-23924][SQL] Add element_at function
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21053
**[Test build #89476 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/89476/testReport)** for PR 21053 at commit [`98465b1`](https://github.com/apache/spark/commit/98465b1e5584acfd15b97b2fa239481b238a9237).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds the following public classes _(experimental)_:
* `case class GetMapValue(child: Expression, key: Expression)`
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #21053: [SPARK-23924][SQL] Add element_at function
Posted by kiszk <gi...@git.apache.org>.
Github user kiszk commented on a diff in the pull request:
https://github.com/apache/spark/pull/21053#discussion_r181790679
--- Diff: python/pyspark/sql/functions.py ---
@@ -1846,6 +1846,27 @@ def array_contains(col, value):
return Column(sc._jvm.functions.array_contains(_to_java_column(col), value))
+@since(2.4)
--- End diff --
I see, thanks
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21053: [SPARK-23924][SQL] Add element_at function
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21053
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #21053: [SPARK-23924][SQL] Add element_at function
Posted by kiszk <gi...@git.apache.org>.
Github user kiszk commented on a diff in the pull request:
https://github.com/apache/spark/pull/21053#discussion_r181793795
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/collectionOperations.scala ---
@@ -287,3 +287,106 @@ case class ArrayContains(left: Expression, right: Expression)
override def prettyName: String = "array_contains"
}
+
+/**
+ * Returns the value of index `right` in Array `left` or the value for key `right` in Map `left`.
+ */
+@ExpressionDescription(
+ usage = """
+ _FUNC_(array, index) - Returns element of array at given index. If index < 0, accesses elements
+ from the last to the first.
+
+ _FUNC_(map, key) - Returns value for given key, or NULL if the key is not contained in the map
+ """,
+ examples = """
+ Examples:
+ > SELECT _FUNC_(array(1, 2, 3), 2);
+ 2
+ > SELECT _FUNC_(map(1, 'a', 2, 'b'), 2);
+ "b"
+ """,
+ since = "2.4.0")
+case class ElementAt(left: Expression, right: Expression) extends GetMapValueUtil {
+
+ override def dataType: DataType = left.dataType match {
+ case _: ArrayType => left.dataType.asInstanceOf[ArrayType].elementType
+ case _: MapType => left.dataType.asInstanceOf[MapType].valueType
+ }
+
+ override def inputTypes: Seq[AbstractDataType] = {
+ Seq(TypeCollection(ArrayType, MapType),
+ left.dataType match {
+ case _: ArrayType => IntegerType
+ case _: MapType => left.dataType.asInstanceOf[MapType].keyType
+ }
+ )
+ }
+
+ override def nullable: Boolean = true
--- End diff --
year, may depend on `right` value, too.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21053: [SPARK-23924][SQL] Add element_at function
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21053
**[Test build #89520 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/89520/testReport)** for PR 21053 at commit [`68e8907`](https://github.com/apache/spark/commit/68e89070a9c046907d35b5ce6802a498860a8302).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21053: [SPARK-23924][SQL] Add element_at function
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21053
**[Test build #89520 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/89520/testReport)** for PR 21053 at commit [`68e8907`](https://github.com/apache/spark/commit/68e89070a9c046907d35b5ce6802a498860a8302).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21053: [SPARK-23924][SQL] Add element_at function
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21053
**[Test build #89284 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/89284/testReport)** for PR 21053 at commit [`d7ec6cc`](https://github.com/apache/spark/commit/d7ec6ccc177d07a8090ac27ce1659f427d1cf50a).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21053: [SPARK-23924][SQL] Add element_at function
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21053
Merged build finished. Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21053: [SPARK-23924][SQL] Add element_at function
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21053
Merged build finished. Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #21053: [SPARK-23924][SQL] Add element_at function
Posted by viirya <gi...@git.apache.org>.
Github user viirya commented on a diff in the pull request:
https://github.com/apache/spark/pull/21053#discussion_r181623519
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/collectionOperations.scala ---
@@ -287,3 +287,106 @@ case class ArrayContains(left: Expression, right: Expression)
override def prettyName: String = "array_contains"
}
+
+/**
+ * Returns the value of index `right` in Array `left` or the value for key `right` in Map `left`.
+ */
+@ExpressionDescription(
+ usage = """
+ _FUNC_(array, index) - Returns element of array at given index. If index < 0, accesses elements
+ from the last to the first.
--- End diff --
nit: `Returns NULL if the index exceeds the length of the array`.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21053: [SPARK-23924][SQL] Add element_at function
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21053
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/89383/
Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #21053: [SPARK-23924][SQL] Add element_at function
Posted by ueshin <gi...@git.apache.org>.
Github user ueshin commented on a diff in the pull request:
https://github.com/apache/spark/pull/21053#discussion_r181618055
--- Diff: python/pyspark/sql/functions.py ---
@@ -1846,6 +1846,27 @@ def array_contains(col, value):
return Column(sc._jvm.functions.array_contains(_to_java_column(col), value))
+@since(2.4)
--- End diff --
We need to annotate as `@ignore_unicode_prefix`?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21053: [SPARK-23924][SQL] Add element_at function
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21053
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/89545/
Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21053: [SPARK-23924][SQL] Add element_at function
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21053
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #21053: [SPARK-23924][SQL] Add element_at function
Posted by ueshin <gi...@git.apache.org>.
Github user ueshin commented on a diff in the pull request:
https://github.com/apache/spark/pull/21053#discussion_r181623803
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/collectionOperations.scala ---
@@ -287,3 +287,106 @@ case class ArrayContains(left: Expression, right: Expression)
override def prettyName: String = "array_contains"
}
+
+/**
+ * Returns the value of index `right` in Array `left` or the value for key `right` in Map `left`.
+ */
+@ExpressionDescription(
+ usage = """
+ _FUNC_(array, index) - Returns element of array at given index. If index < 0, accesses elements
+ from the last to the first.
+
+ _FUNC_(map, key) - Returns value for given key, or NULL if the key is not contained in the map
+ """,
+ examples = """
+ Examples:
+ > SELECT _FUNC_(array(1, 2, 3), 2);
+ 2
+ > SELECT _FUNC_(map(1, 'a', 2, 'b'), 2);
+ "b"
+ """,
+ since = "2.4.0")
+case class ElementAt(left: Expression, right: Expression) extends GetMapValueUtil {
+
+ override def dataType: DataType = left.dataType match {
+ case _: ArrayType => left.dataType.asInstanceOf[ArrayType].elementType
+ case _: MapType => left.dataType.asInstanceOf[MapType].valueType
+ }
+
+ override def inputTypes: Seq[AbstractDataType] = {
+ Seq(TypeCollection(ArrayType, MapType),
+ left.dataType match {
+ case _: ArrayType => IntegerType
+ case _: MapType => left.dataType.asInstanceOf[MapType].keyType
+ }
+ )
+ }
+
+ override def nullable: Boolean = true
--- End diff --
I'm afraid it's wrong because this returns `null` when the given index is "out of bounds" (`array.numElements() < math.abs(index)`) for array type or the given key doesn't exist for map type.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21053: [SPARK-23924][SQL] Add element_at function
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21053
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/2478/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21053: [SPARK-23924][SQL] Add element_at function
Posted by kiszk <gi...@git.apache.org>.
Github user kiszk commented on the issue:
https://github.com/apache/spark/pull/21053
retest this please
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21053: [SPARK-23924][SQL] Add element_at function
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21053
**[Test build #89502 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/89502/testReport)** for PR 21053 at commit [`c734607`](https://github.com/apache/spark/commit/c7346079f1101f989fd4619ec65fd2e0d24acd85).
* This patch **fails PySpark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21053: [SPARK-23924][SQL] Add element_at function
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21053
Merged build finished. Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #21053: [SPARK-23924][SQL] Add element_at function
Posted by viirya <gi...@git.apache.org>.
Github user viirya commented on a diff in the pull request:
https://github.com/apache/spark/pull/21053#discussion_r181376088
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/collectionOperations.scala ---
@@ -287,3 +287,106 @@ case class ArrayContains(left: Expression, right: Expression)
override def prettyName: String = "array_contains"
}
+
+/**
+ * Returns the value of index `right` in Array `left` or key right` in Map `left`.
--- End diff --
We can improve this doc. There is also broken quote.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21053: [SPARK-23924][SQL] Add element_at function
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21053
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #21053: [SPARK-23924][SQL] Add element_at function
Posted by ueshin <gi...@git.apache.org>.
Github user ueshin commented on a diff in the pull request:
https://github.com/apache/spark/pull/21053#discussion_r182325781
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/collectionOperations.scala ---
@@ -417,3 +417,106 @@ case class ArrayMax(child: Expression) extends UnaryExpression with ImplicitCast
override def prettyName: String = "array_max"
}
+
+/**
+ * Returns the value of index `right` in Array `left` or the value for key `right` in Map `left`.
+ */
+@ExpressionDescription(
+ usage = """
+ _FUNC_(array, index) - Returns element of array at given index. If index < 0, accesses elements
+ from the last to the first. Returns NULL if the index exceeds the length of the array.
--- End diff --
ditto.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #21053: [SPARK-23924][SQL] Add element_at function
Posted by rxin <gi...@git.apache.org>.
Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/21053#discussion_r181529978
--- Diff: sql/core/src/test/scala/org/apache/spark/sql/DataFrameFunctionsSuite.scala ---
@@ -413,6 +413,78 @@ class DataFrameFunctionsSuite extends QueryTest with SharedSQLContext {
)
}
+ test("element at function") {
--- End diff --
also the function is element_at, not "element at" ...
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #21053: [SPARK-23924][SQL] Add element_at function
Posted by viirya <gi...@git.apache.org>.
Github user viirya commented on a diff in the pull request:
https://github.com/apache/spark/pull/21053#discussion_r181624253
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/collectionOperations.scala ---
@@ -287,3 +287,106 @@ case class ArrayContains(left: Expression, right: Expression)
override def prettyName: String = "array_contains"
}
+
+/**
+ * Returns the value of index `right` in Array `left` or the value for key `right` in Map `left`.
+ */
+@ExpressionDescription(
+ usage = """
+ _FUNC_(array, index) - Returns element of array at given index. If index < 0, accesses elements
+ from the last to the first.
+
+ _FUNC_(map, key) - Returns value for given key, or NULL if the key is not contained in the map
+ """,
+ examples = """
+ Examples:
+ > SELECT _FUNC_(array(1, 2, 3), 2);
+ 2
+ > SELECT _FUNC_(map(1, 'a', 2, 'b'), 2);
+ "b"
+ """,
+ since = "2.4.0")
+case class ElementAt(left: Expression, right: Expression) extends GetMapValueUtil {
+
+ override def dataType: DataType = left.dataType match {
+ case _: ArrayType => left.dataType.asInstanceOf[ArrayType].elementType
+ case _: MapType => left.dataType.asInstanceOf[MapType].valueType
+ }
+
+ override def inputTypes: Seq[AbstractDataType] = {
+ Seq(TypeCollection(ArrayType, MapType),
+ left.dataType match {
+ case _: ArrayType => IntegerType
+ case _: MapType => left.dataType.asInstanceOf[MapType].keyType
+ }
+ )
+ }
+
+ override def nullable: Boolean = true
--- End diff --
ah, I see. Invalid `right` can cause null result too.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #21053: [SPARK-23924][SQL] Add element_at function
Posted by ueshin <gi...@git.apache.org>.
Github user ueshin commented on a diff in the pull request:
https://github.com/apache/spark/pull/21053#discussion_r181619426
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/collectionOperations.scala ---
@@ -287,3 +287,106 @@ case class ArrayContains(left: Expression, right: Expression)
override def prettyName: String = "array_contains"
}
+
+/**
+ * Returns the value of index `right` in Array `left` or the value for key `right` in Map `left`.
+ */
+@ExpressionDescription(
+ usage = """
+ _FUNC_(array, index) - Returns element of array at given index. If index < 0, accesses elements
+ from the last to the first.
+
+ _FUNC_(map, key) - Returns value for given key, or NULL if the key is not contained in the map
+ """,
+ examples = """
+ Examples:
+ > SELECT _FUNC_(array(1, 2, 3), 2);
+ 2
+ > SELECT _FUNC_(map(1, 'a', 2, 'b'), 2);
+ "b"
+ """,
+ since = "2.4.0")
+case class ElementAt(left: Expression, right: Expression) extends GetMapValueUtil {
+
+ override def dataType: DataType = left.dataType match {
+ case _: ArrayType => left.dataType.asInstanceOf[ArrayType].elementType
+ case _: MapType => left.dataType.asInstanceOf[MapType].valueType
+ }
+
+ override def inputTypes: Seq[AbstractDataType] = {
+ Seq(TypeCollection(ArrayType, MapType),
+ left.dataType match {
+ case _: ArrayType => IntegerType
+ case _: MapType => left.dataType.asInstanceOf[MapType].keyType
+ }
+ )
+ }
+
+ override def nullable: Boolean = true
+
+ override def nullSafeEval(value: Any, ordinal: Any): Any = {
+ left.dataType match {
+ case _: ArrayType =>
+ val array = value.asInstanceOf[ArrayData]
+ val index = ordinal.asInstanceOf[Int]
+ if (array.numElements() < math.abs(index)) {
+ null
+ } else {
+ val idx = if (index == 0) {
+ throw new ArrayIndexOutOfBoundsException("SQL array indices start at 1")
+ } else if (index > 0) {
+ index - 1
+ } else {
+ array.numElements() + index
+ }
+ if (array.isNullAt(idx)) {
+ null
+ } else {
+ array.get(idx, dataType)
+ }
+ }
+ case _: MapType =>
+ getValueEval(value, ordinal, left.dataType.asInstanceOf[MapType].keyType)
+ }
+ }
+
+ override def doGenCode(ctx: CodegenContext, ev: ExprCode): ExprCode = {
+ left.dataType match {
+ case _: ArrayType =>
+ nullSafeCodeGen(ctx, ev, (eval1, eval2) => {
+ val index = ctx.freshName("elementAtIndex")
+ val nullCheck = if (left.dataType.asInstanceOf[ArrayType].containsNull) {
+ s"""
+ |if ($eval1.isNullAt($index)) {
+ | ${ev.isNull} = true;
+ |} else
+ """
+ } else {
+ ""
+ }
+ s"""
+ |int $index = (int) $eval2;
+ |if ($eval1.numElements() < Math.abs($index)) {
+ | ${ev.isNull} = true;
+ |} else {
+ | if ($index == 0) {
+ | throw new ArrayIndexOutOfBoundsException("SQL array indices start at 1");
+ | } else if ($index > 0) {
+ | $index--;
+ | } else {
+ | $index += $eval1.numElements();
+ | }
+ | $nullCheck
+ | {
+ | ${ev.value} = ${CodeGenerator.getValue(eval1, dataType, index)};
+ | }
+ |}
+ """
--- End diff --
`stripMargin` is missing?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #21053: [SPARK-23924][SQL] Add element_at function
Posted by rxin <gi...@git.apache.org>.
Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/21053#discussion_r181529901
--- Diff: sql/core/src/test/scala/org/apache/spark/sql/DataFrameFunctionsSuite.scala ---
@@ -413,6 +413,78 @@ class DataFrameFunctionsSuite extends QueryTest with SharedSQLContext {
)
}
+ test("element at function") {
--- End diff --
why do we need so many test cases here? this is just to verify the api works end to end.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21053: [SPARK-23924][SQL] Add element_at function
Posted by gatorsmile <gi...@git.apache.org>.
Github user gatorsmile commented on the issue:
https://github.com/apache/spark/pull/21053
cc @ueshin
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21053: [SPARK-23924][SQL] Add element_at function
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21053
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21053: [SPARK-23924][SQL] Add element_at function
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21053
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/2444/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21053: [SPARK-23924][SQL] Add element_at function
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21053
Merged build finished. Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21053: [SPARK-23924][SQL] Add element_at function
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21053
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21053: [SPARK-23924][SQL] Add element_at function
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21053
**[Test build #89558 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/89558/testReport)** for PR 21053 at commit [`90e026e`](https://github.com/apache/spark/commit/90e026e9b2d58e17995d21e44c5a68fa7f0f7d52).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #21053: [SPARK-23924][SQL] Add element_at function
Posted by kiszk <gi...@git.apache.org>.
Github user kiszk commented on a diff in the pull request:
https://github.com/apache/spark/pull/21053#discussion_r181687320
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/collectionOperations.scala ---
@@ -287,3 +287,106 @@ case class ArrayContains(left: Expression, right: Expression)
override def prettyName: String = "array_contains"
}
+
+/**
+ * Returns the value of index `right` in Array `left` or the value for key `right` in Map `left`.
+ */
+@ExpressionDescription(
+ usage = """
+ _FUNC_(array, index) - Returns element of array at given index. If index < 0, accesses elements
+ from the last to the first.
+
+ _FUNC_(map, key) - Returns value for given key, or NULL if the key is not contained in the map
+ """,
+ examples = """
+ Examples:
+ > SELECT _FUNC_(array(1, 2, 3), 2);
+ 2
+ > SELECT _FUNC_(map(1, 'a', 2, 'b'), 2);
+ "b"
+ """,
+ since = "2.4.0")
+case class ElementAt(left: Expression, right: Expression) extends GetMapValueUtil {
+
+ override def dataType: DataType = left.dataType match {
+ case _: ArrayType => left.dataType.asInstanceOf[ArrayType].elementType
+ case _: MapType => left.dataType.asInstanceOf[MapType].valueType
+ }
+
+ override def inputTypes: Seq[AbstractDataType] = {
+ Seq(TypeCollection(ArrayType, MapType),
+ left.dataType match {
+ case _: ArrayType => IntegerType
+ case _: MapType => left.dataType.asInstanceOf[MapType].keyType
+ }
+ )
+ }
+
+ override def nullable: Boolean = true
+
+ override def nullSafeEval(value: Any, ordinal: Any): Any = {
+ left.dataType match {
+ case _: ArrayType =>
+ val array = value.asInstanceOf[ArrayData]
+ val index = ordinal.asInstanceOf[Int]
+ if (array.numElements() < math.abs(index)) {
+ null
+ } else {
+ val idx = if (index == 0) {
+ throw new ArrayIndexOutOfBoundsException("SQL array indices start at 1")
+ } else if (index > 0) {
+ index - 1
+ } else {
+ array.numElements() + index
+ }
+ if (array.isNullAt(idx)) {
+ null
+ } else {
+ array.get(idx, dataType)
+ }
+ }
+ case _: MapType =>
+ getValueEval(value, ordinal, left.dataType.asInstanceOf[MapType].keyType)
+ }
+ }
+
+ override def doGenCode(ctx: CodegenContext, ev: ExprCode): ExprCode = {
+ left.dataType match {
+ case _: ArrayType =>
+ nullSafeCodeGen(ctx, ev, (eval1, eval2) => {
+ val index = ctx.freshName("elementAtIndex")
+ val nullCheck = if (left.dataType.asInstanceOf[ArrayType].containsNull) {
+ s"""
+ |if ($eval1.isNullAt($index)) {
+ | ${ev.isNull} = true;
+ |} else
+ """
+ } else {
+ ""
+ }
+ s"""
+ |int $index = (int) $eval2;
+ |if ($eval1.numElements() < Math.abs($index)) {
+ | ${ev.isNull} = true;
+ |} else {
+ | if ($index == 0) {
+ | throw new ArrayIndexOutOfBoundsException("SQL array indices start at 1");
+ | } else if ($index > 0) {
+ | $index--;
+ | } else {
+ | $index += $eval1.numElements();
+ | }
+ | $nullCheck
+ | {
+ | ${ev.value} = ${CodeGenerator.getValue(eval1, dataType, index)};
+ | }
+ |}
+ """
--- End diff --
good catch, thanks
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21053: [SPARK-23924][SQL] Add element_at function
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21053
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/89321/
Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21053: [SPARK-23924][SQL] Add element_at function
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21053
**[Test build #89502 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/89502/testReport)** for PR 21053 at commit [`c734607`](https://github.com/apache/spark/commit/c7346079f1101f989fd4619ec65fd2e0d24acd85).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21053: [SPARK-23924][SQL] Add element_at function
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21053
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21053: [SPARK-23924][SQL] Add element_at function
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21053
**[Test build #89558 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/89558/testReport)** for PR 21053 at commit [`90e026e`](https://github.com/apache/spark/commit/90e026e9b2d58e17995d21e44c5a68fa7f0f7d52).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21053: [SPARK-23924][SQL] Add element_at function
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21053
**[Test build #89356 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/89356/testReport)** for PR 21053 at commit [`35844f8`](https://github.com/apache/spark/commit/35844f8402093e3a2cbe0cd57f3236128a5edf8c).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21053: [SPARK-23924][SQL] Add element_at function
Posted by kiszk <gi...@git.apache.org>.
Github user kiszk commented on the issue:
https://github.com/apache/spark/pull/21053
retest this please
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21053: [SPARK-23924][SQL] Add element_at function
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21053
**[Test build #89554 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/89554/testReport)** for PR 21053 at commit [`90e026e`](https://github.com/apache/spark/commit/90e026e9b2d58e17995d21e44c5a68fa7f0f7d52).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #21053: [SPARK-23924][SQL] Add element_at function
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/spark/pull/21053
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21053: [SPARK-23924][SQL] Add element_at function
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21053
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21053: [SPARK-23924][SQL] Add element_at function
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21053
**[Test build #89528 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/89528/testReport)** for PR 21053 at commit [`68e8907`](https://github.com/apache/spark/commit/68e89070a9c046907d35b5ce6802a498860a8302).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21053: [SPARK-23924][SQL] Add element_at function
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21053
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21053: [SPARK-23924][SQL] Add element_at function
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21053
**[Test build #89383 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/89383/testReport)** for PR 21053 at commit [`013a53e`](https://github.com/apache/spark/commit/013a53e065396b9b0f8f10acb2c091f05b780013).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21053: [SPARK-23924][SQL] Add element_at function
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21053
**[Test build #89552 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/89552/testReport)** for PR 21053 at commit [`06fb27e`](https://github.com/apache/spark/commit/06fb27e68ddda7075c0edef14df50cf56a370b4e).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21053: [SPARK-23924][SQL] Add element_at function
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21053
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21053: [SPARK-23924][SQL] Add element_at function
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21053
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/2295/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21053: [SPARK-23924][SQL] Add element_at function
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21053
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/89520/
Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21053: [SPARK-23924][SQL] Add element_at function
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21053
**[Test build #89356 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/89356/testReport)** for PR 21053 at commit [`35844f8`](https://github.com/apache/spark/commit/35844f8402093e3a2cbe0cd57f3236128a5edf8c).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21053: [SPARK-23924][SQL] Add element_at function
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21053
Merged build finished. Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #21053: [SPARK-23924][SQL] Add element_at function
Posted by kiszk <gi...@git.apache.org>.
Github user kiszk commented on a diff in the pull request:
https://github.com/apache/spark/pull/21053#discussion_r181610494
--- Diff: sql/core/src/test/scala/org/apache/spark/sql/DataFrameFunctionsSuite.scala ---
@@ -413,6 +413,78 @@ class DataFrameFunctionsSuite extends QueryTest with SharedSQLContext {
)
}
+ test("element at function") {
--- End diff --
good catch, thanks
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21053: [SPARK-23924][SQL] Add element_at function
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21053
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21053: [SPARK-23924][SQL] Add element_at function
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21053
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/89313/
Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #21053: [SPARK-23924][SQL] Add element_at function
Posted by kiszk <gi...@git.apache.org>.
Github user kiszk commented on a diff in the pull request:
https://github.com/apache/spark/pull/21053#discussion_r181475737
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/collectionOperations.scala ---
@@ -287,3 +287,106 @@ case class ArrayContains(left: Expression, right: Expression)
override def prettyName: String = "array_contains"
}
+
+/**
+ * Returns the value of index `right` in Array `left` or key right` in Map `left`.
--- End diff --
good catch, thanks
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #21053: [SPARK-23924][SQL] Add element_at function
Posted by ueshin <gi...@git.apache.org>.
Github user ueshin commented on a diff in the pull request:
https://github.com/apache/spark/pull/21053#discussion_r181619450
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/collectionOperations.scala ---
@@ -287,3 +287,106 @@ case class ArrayContains(left: Expression, right: Expression)
override def prettyName: String = "array_contains"
}
+
+/**
+ * Returns the value of index `right` in Array `left` or the value for key `right` in Map `left`.
+ */
+@ExpressionDescription(
+ usage = """
+ _FUNC_(array, index) - Returns element of array at given index. If index < 0, accesses elements
+ from the last to the first.
+
+ _FUNC_(map, key) - Returns value for given key, or NULL if the key is not contained in the map
+ """,
+ examples = """
+ Examples:
+ > SELECT _FUNC_(array(1, 2, 3), 2);
+ 2
+ > SELECT _FUNC_(map(1, 'a', 2, 'b'), 2);
+ "b"
+ """,
+ since = "2.4.0")
+case class ElementAt(left: Expression, right: Expression) extends GetMapValueUtil {
+
+ override def dataType: DataType = left.dataType match {
+ case _: ArrayType => left.dataType.asInstanceOf[ArrayType].elementType
+ case _: MapType => left.dataType.asInstanceOf[MapType].valueType
+ }
+
+ override def inputTypes: Seq[AbstractDataType] = {
+ Seq(TypeCollection(ArrayType, MapType),
+ left.dataType match {
+ case _: ArrayType => IntegerType
+ case _: MapType => left.dataType.asInstanceOf[MapType].keyType
+ }
+ )
+ }
+
+ override def nullable: Boolean = true
+
+ override def nullSafeEval(value: Any, ordinal: Any): Any = {
+ left.dataType match {
+ case _: ArrayType =>
+ val array = value.asInstanceOf[ArrayData]
+ val index = ordinal.asInstanceOf[Int]
+ if (array.numElements() < math.abs(index)) {
+ null
+ } else {
+ val idx = if (index == 0) {
+ throw new ArrayIndexOutOfBoundsException("SQL array indices start at 1")
+ } else if (index > 0) {
+ index - 1
+ } else {
+ array.numElements() + index
+ }
+ if (array.isNullAt(idx)) {
+ null
+ } else {
+ array.get(idx, dataType)
+ }
+ }
+ case _: MapType =>
+ getValueEval(value, ordinal, left.dataType.asInstanceOf[MapType].keyType)
+ }
+ }
+
+ override def doGenCode(ctx: CodegenContext, ev: ExprCode): ExprCode = {
+ left.dataType match {
+ case _: ArrayType =>
+ nullSafeCodeGen(ctx, ev, (eval1, eval2) => {
+ val index = ctx.freshName("elementAtIndex")
+ val nullCheck = if (left.dataType.asInstanceOf[ArrayType].containsNull) {
+ s"""
+ |if ($eval1.isNullAt($index)) {
+ | ${ev.isNull} = true;
+ |} else
+ """
--- End diff --
`stripMargin` is missing?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21053: [SPARK-23924][SQL] Add element_at function
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21053
**[Test build #89476 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/89476/testReport)** for PR 21053 at commit [`98465b1`](https://github.com/apache/spark/commit/98465b1e5584acfd15b97b2fa239481b238a9237).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21053: [SPARK-23924][SQL] Add element_at function
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21053
**[Test build #89545 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/89545/testReport)** for PR 21053 at commit [`2fbb1e8`](https://github.com/apache/spark/commit/2fbb1e8fbc17e95560a18170fbd3ad0e1f4d78ff).
* This patch **fails Scala style tests**.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21053: [SPARK-23924][SQL] Add element_at function
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21053
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/2466/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21053: [SPARK-23924][SQL] Add element_at function
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21053
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21053: [SPARK-23924][SQL] Add element_at function
Posted by ueshin <gi...@git.apache.org>.
Github user ueshin commented on the issue:
https://github.com/apache/spark/pull/21053
Thanks! merging to master.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21053: [SPARK-23924][SQL] Add element_at function
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21053
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/89403/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21053: [SPARK-23924][SQL] Add element_at function
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21053
Merged build finished. Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21053: [SPARK-23924][SQL] Add element_at function
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21053
**[Test build #89284 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/89284/testReport)** for PR 21053 at commit [`d7ec6cc`](https://github.com/apache/spark/commit/d7ec6ccc177d07a8090ac27ce1659f427d1cf50a).
* This patch **fails PySpark unit tests**.
* This patch merges cleanly.
* This patch adds the following public classes _(experimental)_:
* `case class ElementAt(left: Expression, right: Expression) extends GetMapValueUtil `
* `abstract class GetMapValueUtil extends BinaryExpression with ImplicitCastInputTypes `
* `case class GetMapValue(child: Expression, key: Expression) extends GetMapValueUtil`
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21053: [SPARK-23924][SQL] Add element_at function
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21053
**[Test build #89383 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/89383/testReport)** for PR 21053 at commit [`013a53e`](https://github.com/apache/spark/commit/013a53e065396b9b0f8f10acb2c091f05b780013).
* This patch **fails PySpark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21053: [SPARK-23924][SQL] Add element_at function
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21053
Merged build finished. Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21053: [SPARK-23924][SQL] Add element_at function
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21053
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/2473/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #21053: [SPARK-23924][SQL] Add element_at function
Posted by ueshin <gi...@git.apache.org>.
Github user ueshin commented on a diff in the pull request:
https://github.com/apache/spark/pull/21053#discussion_r182327230
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/collectionOperations.scala ---
@@ -417,3 +417,106 @@ case class ArrayMax(child: Expression) extends UnaryExpression with ImplicitCast
override def prettyName: String = "array_max"
}
+
+/**
+ * Returns the value of index `right` in Array `left` or the value for key `right` in Map `left`.
+ */
+@ExpressionDescription(
+ usage = """
+ _FUNC_(array, index) - Returns element of array at given index. If index < 0, accesses elements
+ from the last to the first. Returns NULL if the index exceeds the length of the array.
+
+ _FUNC_(map, key) - Returns value for given key, or NULL if the key is not contained in the map
+ """,
+ examples = """
+ Examples:
+ > SELECT _FUNC_(array(1, 2, 3), 2);
+ 2
+ > SELECT _FUNC_(map(1, 'a', 2, 'b'), 2);
+ "b"
+ """,
+ since = "2.4.0")
+case class ElementAt(left: Expression, right: Expression) extends GetMapValueUtil {
+
+ override def dataType: DataType = left.dataType match {
+ case _: ArrayType => left.dataType.asInstanceOf[ArrayType].elementType
+ case _: MapType => left.dataType.asInstanceOf[MapType].valueType
+ }
--- End diff --
nit: how about:
```scala
override def dataType: DataType = left.dataType match {
case ArrayType(elementType, _) => elementType
case MapType(_, valueType, _) => valueType
}
```
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21053: [SPARK-23924][SQL] Add element_at function
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21053
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21053: [SPARK-23924][SQL] Add element_at function
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21053
**[Test build #89336 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/89336/testReport)** for PR 21053 at commit [`0146945`](https://github.com/apache/spark/commit/0146945151caa75ba5bd3e5c44038194f3ce89a2).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #21053: [SPARK-23924][SQL] Add element_at function
Posted by kiszk <gi...@git.apache.org>.
Github user kiszk commented on a diff in the pull request:
https://github.com/apache/spark/pull/21053#discussion_r182486210
--- Diff: python/pyspark/sql/functions.py ---
@@ -1846,6 +1846,28 @@ def array_contains(col, value):
return Column(sc._jvm.functions.array_contains(_to_java_column(col), value))
+@ignore_unicode_prefix
+@since(2.4)
+def element_at(col, value):
--- End diff --
Sure, done
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21053: [SPARK-23924][SQL] Add element_at function
Posted by kiszk <gi...@git.apache.org>.
Github user kiszk commented on the issue:
https://github.com/apache/spark/pull/21053
@ueshin would it be possible to review this again?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #21053: [SPARK-23924][SQL] Add element_at function
Posted by viirya <gi...@git.apache.org>.
Github user viirya commented on a diff in the pull request:
https://github.com/apache/spark/pull/21053#discussion_r181623729
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/collectionOperations.scala ---
@@ -287,3 +287,106 @@ case class ArrayContains(left: Expression, right: Expression)
override def prettyName: String = "array_contains"
}
+
+/**
+ * Returns the value of index `right` in Array `left` or the value for key `right` in Map `left`.
+ */
+@ExpressionDescription(
+ usage = """
+ _FUNC_(array, index) - Returns element of array at given index. If index < 0, accesses elements
+ from the last to the first.
+
+ _FUNC_(map, key) - Returns value for given key, or NULL if the key is not contained in the map
+ """,
+ examples = """
+ Examples:
+ > SELECT _FUNC_(array(1, 2, 3), 2);
+ 2
+ > SELECT _FUNC_(map(1, 'a', 2, 'b'), 2);
+ "b"
+ """,
+ since = "2.4.0")
+case class ElementAt(left: Expression, right: Expression) extends GetMapValueUtil {
+
+ override def dataType: DataType = left.dataType match {
+ case _: ArrayType => left.dataType.asInstanceOf[ArrayType].elementType
+ case _: MapType => left.dataType.asInstanceOf[MapType].valueType
+ }
+
+ override def inputTypes: Seq[AbstractDataType] = {
+ Seq(TypeCollection(ArrayType, MapType),
+ left.dataType match {
+ case _: ArrayType => IntegerType
+ case _: MapType => left.dataType.asInstanceOf[MapType].keyType
+ }
+ )
+ }
+
+ override def nullable: Boolean = true
+
+ override def nullSafeEval(value: Any, ordinal: Any): Any = {
+ left.dataType match {
+ case _: ArrayType =>
+ val array = value.asInstanceOf[ArrayData]
+ val index = ordinal.asInstanceOf[Int]
+ if (array.numElements() < math.abs(index)) {
+ null
+ } else {
+ val idx = if (index == 0) {
+ throw new ArrayIndexOutOfBoundsException("SQL array indices start at 1")
+ } else if (index > 0) {
+ index - 1
+ } else {
+ array.numElements() + index
+ }
+ if (array.isNullAt(idx)) {
--- End diff --
nit:
```scala
if(left.dataType.asInstanceOf[ArrayType].containsNull && array.isNullAt(idx)) {
...
}
```
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21053: [SPARK-23924][SQL] Add element_at function
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21053
**[Test build #89313 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/89313/testReport)** for PR 21053 at commit [`bb0ab45`](https://github.com/apache/spark/commit/bb0ab45b4a9bbf1155dbb9513508bbef3685b3f6).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21053: [SPARK-23924][SQL] Add element_at function
Posted by kiszk <gi...@git.apache.org>.
Github user kiszk commented on the issue:
https://github.com/apache/spark/pull/21053
retest this please
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #21053: [SPARK-23924][SQL] Add element_at function
Posted by ueshin <gi...@git.apache.org>.
Github user ueshin commented on a diff in the pull request:
https://github.com/apache/spark/pull/21053#discussion_r182362454
--- Diff: python/pyspark/sql/functions.py ---
@@ -1846,6 +1846,28 @@ def array_contains(col, value):
return Column(sc._jvm.functions.array_contains(_to_java_column(col), value))
+@ignore_unicode_prefix
+@since(2.4)
+def element_at(col, value):
--- End diff --
How about `extraction` which is from `Column.apply(extraction)`? cc @gatorsmile
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21053: [SPARK-23924][SQL] Add element_at function
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21053
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21053: [SPARK-23924][SQL] Add element_at function
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21053
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/89558/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21053: [SPARK-23924][SQL] Add element_at function
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21053
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/89528/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21053: [SPARK-23924][SQL] Add element_at function
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21053
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/2264/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21053: [SPARK-23924][SQL] Add element_at function
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21053
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/89553/
Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21053: [SPARK-23924][SQL] Add element_at function
Posted by kiszk <gi...@git.apache.org>.
Github user kiszk commented on the issue:
https://github.com/apache/spark/pull/21053
retest this please
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #21053: [SPARK-23924][SQL] Add element_at function
Posted by ueshin <gi...@git.apache.org>.
Github user ueshin commented on a diff in the pull request:
https://github.com/apache/spark/pull/21053#discussion_r182326476
--- Diff: python/pyspark/sql/functions.py ---
@@ -1846,6 +1846,28 @@ def array_contains(col, value):
return Column(sc._jvm.functions.array_contains(_to_java_column(col), value))
+@ignore_unicode_prefix
+@since(2.4)
+def element_at(col, value):
--- End diff --
The name `value` is confusing because this is for an index or a key?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21053: [SPARK-23924][SQL] Add element_at function
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21053
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/89502/
Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21053: [SPARK-23924][SQL] Add element_at function
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21053
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21053: [SPARK-23924][SQL] Add element_at function
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21053
Merged build finished. Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21053: [SPARK-23924][SQL] Add element_at function
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21053
**[Test build #89552 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/89552/testReport)** for PR 21053 at commit [`06fb27e`](https://github.com/apache/spark/commit/06fb27e68ddda7075c0edef14df50cf56a370b4e).
* This patch **fails due to an unknown error code, -9**.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21053: [SPARK-23924][SQL] Add element_at function
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21053
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/89554/
Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21053: [SPARK-23924][SQL] Add element_at function
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21053
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/89356/
Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21053: [SPARK-23924][SQL] Add element_at function
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21053
**[Test build #89545 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/89545/testReport)** for PR 21053 at commit [`2fbb1e8`](https://github.com/apache/spark/commit/2fbb1e8fbc17e95560a18170fbd3ad0e1f4d78ff).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21053: [SPARK-23924][SQL] Add element_at function
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21053
**[Test build #89336 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/89336/testReport)** for PR 21053 at commit [`0146945`](https://github.com/apache/spark/commit/0146945151caa75ba5bd3e5c44038194f3ce89a2).
* This patch **fails PySpark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21053: [SPARK-23924][SQL] Add element_at function
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21053
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21053: [SPARK-23924][SQL] Add element_at function
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21053
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21053: [SPARK-23924][SQL] Add element_at function
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21053
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/2302/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21053: [SPARK-23924][SQL] Add element_at function
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21053
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/2334/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21053: [SPARK-23924][SQL] Add element_at function
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21053
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/89284/
Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org