You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by kiszk <gi...@git.apache.org> on 2018/04/18 18:06:45 UTC
[GitHub] spark pull request #21103: [SPARK-23915][SQL] Add array_except function
GitHub user kiszk opened a pull request:
https://github.com/apache/spark/pull/21103
[SPARK-23915][SQL] Add array_except function
## What changes were proposed in this pull request?
The PR adds the SQL function `array_except`. The behavior of the function is based on Presto's one.
This function returns returns an array of the elements in array1 but not in array2.
Note: The order of elements in the result is not defined.
## How was this patch tested?
Added UTs.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/kiszk/spark SPARK-23915
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/21103.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #21103
----
commit ce18ce0cd4da3378bb9f83237b7f9165737e7760
Author: Kazuaki Ishizaki <is...@...>
Date: 2018-04-18T18:03:55Z
initial commit
----
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #21103: [SPARK-23915][SQL] Add array_except function
Posted by cloud-fan <gi...@git.apache.org>.
Github user cloud-fan commented on a diff in the pull request:
https://github.com/apache/spark/pull/21103#discussion_r205484762
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/collectionOperations.scala ---
@@ -3805,3 +3799,330 @@ object ArrayUnion {
new GenericArrayData(arrayBuffer)
}
}
+
+/**
+ * Returns an array of the elements in the intersect of x and y, without duplicates
+ */
+@ExpressionDescription(
+ usage = """
+ _FUNC_(array1, array2) - Returns an array of the elements in array1 but not in array2,
+ without duplicates.
+ """,
+ examples = """
+ Examples:
+ > SELECT _FUNC_(array(1, 2, 3), array(1, 3, 5));
+ array(2)
+ """,
+ since = "2.4.0")
+case class ArrayExcept(left: Expression, right: Expression) extends ArraySetLike {
+ override def dataType: DataType = left.dataType
+
+ var hsInt: OpenHashSet[Int] = _
+ var hsLong: OpenHashSet[Long] = _
+
+ def assignInt(array: ArrayData, idx: Int, resultArray: ArrayData, pos: Int): Boolean = {
+ val elem = array.getInt(idx)
+ if (!hsInt.contains(elem)) {
+ if (resultArray != null) {
+ resultArray.setInt(pos, elem)
+ }
+ hsInt.add(elem)
+ true
+ } else {
+ false
+ }
+ }
+
+ def assignLong(array: ArrayData, idx: Int, resultArray: ArrayData, pos: Int): Boolean = {
+ val elem = array.getLong(idx)
+ if (!hsLong.contains(elem)) {
+ if (resultArray != null) {
+ resultArray.setLong(pos, elem)
+ }
+ hsLong.add(elem)
+ true
+ } else {
+ false
+ }
+ }
+
+ def evalIntLongPrimitiveType(
+ array1: ArrayData,
+ array2: ArrayData,
+ resultArray: ArrayData,
+ isLongType: Boolean): Int = {
+ // store elements into resultArray
+ var notFoundNullElement = true
+ var i = 0
+ while (i < array2.numElements()) {
+ if (array2.isNullAt(i)) {
+ notFoundNullElement = false
+ } else {
+ val assigned = if (!isLongType) {
+ hsInt.add(array2.getInt(i))
+ } else {
+ hsLong.add(array2.getLong(i))
+ }
+ }
+ i += 1
+ }
+ var pos = 0
+ i = 0
+ while (i < array1.numElements()) {
+ if (array1.isNullAt(i)) {
+ if (notFoundNullElement) {
+ if (resultArray != null) {
+ resultArray.setNullAt(pos)
+ }
+ pos += 1
+ notFoundNullElement = false
+ }
+ } else {
+ val assigned = if (!isLongType) {
+ assignInt(array1, i, resultArray, pos)
+ } else {
+ assignLong(array1, i, resultArray, pos)
+ }
+ if (assigned) {
+ pos += 1
+ }
+ }
+ i += 1
+ }
+ pos
+ }
+
+ override def nullSafeEval(input1: Any, input2: Any): Any = {
+ val array1 = input1.asInstanceOf[ArrayData]
+ val array2 = input2.asInstanceOf[ArrayData]
+
+ if (elementTypeSupportEquals) {
+ elementType match {
--- End diff --
can we avoid this per-row pattern match by using a function object?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21103: [SPARK-23915][SQL] Add array_except function
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21103
Merged build finished. Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21103: [SPARK-23915][SQL] Add array_except function
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21103
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/93690/
Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #21103: [SPARK-23915][SQL] Add array_except function
Posted by cloud-fan <gi...@git.apache.org>.
Github user cloud-fan commented on a diff in the pull request:
https://github.com/apache/spark/pull/21103#discussion_r205668821
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/collectionOperations.scala ---
@@ -3805,3 +3799,233 @@ object ArrayUnion {
new GenericArrayData(arrayBuffer)
}
}
+
+/**
+ * Returns an array of the elements in the intersect of x and y, without duplicates
+ */
+@ExpressionDescription(
+ usage = """
+ _FUNC_(array1, array2) - Returns an array of the elements in array1 but not in array2,
+ without duplicates.
+ """,
+ examples = """
+ Examples:
+ > SELECT _FUNC_(array(1, 2, 3), array(1, 3, 5));
+ array(2)
+ """,
+ since = "2.4.0")
+case class ArrayExcept(left: Expression, right: Expression) extends ArraySetLike {
+ override def dataType: DataType = left.dataType
+
+ var hsInt: OpenHashSet[Int] = _
+ var hsLong: OpenHashSet[Long] = _
+
+ def assignInt(array: ArrayData, idx: Int, resultArray: ArrayData, pos: Int): Boolean = {
+ val elem = array.getInt(idx)
+ if (!hsInt.contains(elem)) {
+ if (resultArray != null) {
+ resultArray.setInt(pos, elem)
+ }
+ hsInt.add(elem)
+ true
+ } else {
+ false
+ }
+ }
+
+ def assignLong(array: ArrayData, idx: Int, resultArray: ArrayData, pos: Int): Boolean = {
+ val elem = array.getLong(idx)
+ if (!hsLong.contains(elem)) {
+ if (resultArray != null) {
+ resultArray.setLong(pos, elem)
+ }
+ hsLong.add(elem)
+ true
+ } else {
+ false
+ }
+ }
+
+ def evalIntLongPrimitiveType(
+ array1: ArrayData,
+ array2: ArrayData,
+ resultArray: ArrayData,
+ isLongType: Boolean): Int = {
+ // store elements into resultArray
+ var notFoundNullElement = true
+ var i = 0
+ while (i < array2.numElements()) {
+ if (array2.isNullAt(i)) {
+ notFoundNullElement = false
+ } else {
+ val assigned = if (!isLongType) {
+ hsInt.add(array2.getInt(i))
+ } else {
+ hsLong.add(array2.getLong(i))
+ }
+ }
+ i += 1
+ }
+ var pos = 0
+ i = 0
+ while (i < array1.numElements()) {
+ if (array1.isNullAt(i)) {
+ if (notFoundNullElement) {
+ if (resultArray != null) {
+ resultArray.setNullAt(pos)
+ }
+ pos += 1
+ notFoundNullElement = false
+ }
+ } else {
+ val assigned = if (!isLongType) {
+ assignInt(array1, i, resultArray, pos)
+ } else {
+ assignLong(array1, i, resultArray, pos)
+ }
+ if (assigned) {
+ pos += 1
+ }
+ }
+ i += 1
+ }
+ pos
+ }
+
+ @transient lazy val evalExcept: (ArrayData, ArrayData) => ArrayData = {
+ if (elementTypeSupportEquals) {
+ elementType match {
+ case IntegerType =>
--- End diff --
if they all worth to optimize, I feel we should not optimize it in the interpreted code path and leave it to codegen path. That's a strong reason to add codegen support.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21103: [SPARK-23915][SQL] Add array_except function
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21103
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/1435/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #21103: [SPARK-23915][SQL] Add array_except function
Posted by cloud-fan <gi...@git.apache.org>.
Github user cloud-fan commented on a diff in the pull request:
https://github.com/apache/spark/pull/21103#discussion_r205928966
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/collectionOperations.scala ---
@@ -3968,3 +3964,317 @@ object ArrayUnion {
new GenericArrayData(arrayBuffer)
}
}
+
+/**
+ * Returns an array of the elements in the intersect of x and y, without duplicates
+ */
+@ExpressionDescription(
+ usage = """
+ _FUNC_(array1, array2) - Returns an array of the elements in array1 but not in array2,
+ without duplicates.
+ """,
+ examples = """
+ Examples:
+ > SELECT _FUNC_(array(1, 2, 3), array(1, 3, 5));
+ array(2)
+ """,
+ since = "2.4.0")
+case class ArrayExcept(left: Expression, right: Expression) extends ArraySetLike
+ with ComplexTypeMergingExpression {
+ override def dataType: DataType = {
+ dataTypeCheck
--- End diff --
My motivation is to let these array functions to go through the type coercion rules for `ComplexTypeMergingExpression`. This check is like `assert` and I don't need we need it if we override `dataType`
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21103: [SPARK-23915][SQL] Add array_except function
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21103
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/1121/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21103: [SPARK-23915][SQL] Add array_except function
Posted by kiszk <gi...@git.apache.org>.
Github user kiszk commented on the issue:
https://github.com/apache/spark/pull/21103
retest this please
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21103: [SPARK-23915][SQL] Add array_except function
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21103
**[Test build #93674 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/93674/testReport)** for PR 21103 at commit [`d8e5d79`](https://github.com/apache/spark/commit/d8e5d798498be7dd976835499004b539cde09005).
* This patch **fails to build**.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #21103: [SPARK-23915][SQL] Add array_except function
Posted by kiszk <gi...@git.apache.org>.
Github user kiszk commented on a diff in the pull request:
https://github.com/apache/spark/pull/21103#discussion_r205930539
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/collectionOperations.scala ---
@@ -3968,3 +3964,242 @@ object ArrayUnion {
new GenericArrayData(arrayBuffer)
}
}
+
+/**
+ * Returns an array of the elements in the intersect of x and y, without duplicates
+ */
+@ExpressionDescription(
+ usage = """
+ _FUNC_(array1, array2) - Returns an array of the elements in array1 but not in array2,
+ without duplicates.
+ """,
+ examples = """
+ Examples:
+ > SELECT _FUNC_(array(1, 2, 3), array(1, 3, 5));
+ array(2)
+ """,
+ since = "2.4.0")
+case class ArrayExcept(left: Expression, right: Expression) extends ArraySetLike
+ with ComplexTypeMergingExpression {
+ override def dataType: DataType = {
+ dataTypeCheck
+ left.dataType
+ }
+
+ @transient lazy val evalExcept: (ArrayData, ArrayData) => ArrayData = {
+ if (elementTypeSupportEquals) {
+ (array1, array2) =>
+ val hs = new OpenHashSet[Any]
--- End diff --
Yeah, to use `byte[]` for binary type looks awkward and inefficient. It would be good to introduce a new class for binary type like `UTF8String`.
On the other hand, it is not a task for Spark 2.4 due to a lot of changes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21103: [SPARK-23915][SQL] Add array_except function
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21103
**[Test build #93270 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/93270/testReport)** for PR 21103 at commit [`26e3257`](https://github.com/apache/spark/commit/26e3257b74fcf830ebcf5bb0f1a7499ad9030b01).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds the following public classes _(experimental)_:
* `case class ArrayUnion(left: Expression, right: Expression) extends ArraySetLike`
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21103: [SPARK-23915][SQL] Add array_except function
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21103
Merged build finished. Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21103: [SPARK-23915][SQL] Add array_except function
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21103
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/1460/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21103: [SPARK-23915][SQL] Add array_except function
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21103
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/93270/
Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21103: [SPARK-23915][SQL] Add array_except function
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21103
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21103: [SPARK-23915][SQL] Add array_except function
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21103
**[Test build #93822 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/93822/testReport)** for PR 21103 at commit [`49b5ab3`](https://github.com/apache/spark/commit/49b5ab371af9783be8f2d6351cf664a769957a4e).
* This patch **fails due to an unknown error code, -9**.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21103: [SPARK-23915][SQL] Add array_except function
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21103
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/89651/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21103: [SPARK-23915][SQL] Add array_except function
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21103
**[Test build #93819 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/93819/testReport)** for PR 21103 at commit [`4d943c8`](https://github.com/apache/spark/commit/4d943c842548914ab151a7a15fc9e0f8743f0caf).
* This patch **fails due to an unknown error code, -9**.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21103: [SPARK-23915][SQL] Add array_except function
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21103
**[Test build #93825 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/93825/testReport)** for PR 21103 at commit [`282445c`](https://github.com/apache/spark/commit/282445cb20fda8e963ceb23d5f52d88c7f8d7d84).
* This patch **fails PySpark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21103: [SPARK-23915][SQL] Add array_except function
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21103
Merged build finished. Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21103: [SPARK-23915][SQL] Add array_except function
Posted by cloud-fan <gi...@git.apache.org>.
Github user cloud-fan commented on the issue:
https://github.com/apache/spark/pull/21103
retest this please
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21103: [SPARK-23915][SQL] Add array_except function
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21103
**[Test build #93582 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/93582/testReport)** for PR 21103 at commit [`cf76c1f`](https://github.com/apache/spark/commit/cf76c1f4a2a41ec88fcd744470a113321e897a71).
* This patch **fails due to an unknown error code, -9**.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21103: [SPARK-23915][SQL] Add array_except function
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21103
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/89540/
Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21103: [SPARK-23915][SQL] Add array_except function
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21103
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21103: [SPARK-23915][SQL] Add array_except function
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21103
Build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #21103: [SPARK-23915][SQL] Add array_except function
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/spark/pull/21103
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21103: [SPARK-23915][SQL] Add array_except function
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21103
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/93644/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21103: [SPARK-23915][SQL] Add array_except function
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21103
**[Test build #89605 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/89605/testReport)** for PR 21103 at commit [`cdb6257`](https://github.com/apache/spark/commit/cdb6257187c3713ccece28d9b8073bf99180b358).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #21103: [SPARK-23915][SQL] Add array_except function
Posted by mn-mikke <gi...@git.apache.org>.
Github user mn-mikke commented on a diff in the pull request:
https://github.com/apache/spark/pull/21103#discussion_r203964623
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/collectionOperations.scala ---
@@ -3805,3 +3799,332 @@ object ArrayUnion {
new GenericArrayData(arrayBuffer)
}
}
+
+/**
+ * Returns an array of the elements in the intersect of x and y, without duplicates
+ */
+@ExpressionDescription(
+ usage = """
+ _FUNC_(array1, array2) - Returns an array of the elements in array1 but not in array2,
+ without duplicates.
+ """,
+ examples = """
+ Examples:Fun
+ > SELECT _FUNC_(array(1, 2, 3), array(1, 3, 5));
+ array(2)
+ """,
+ since = "2.4.0")
+case class ArrayExcept(left: Expression, right: Expression) extends ArraySetLike {
+ override def dataType: DataType = ArrayType(elementType,
--- End diff --
Yeah this case is valid. But ```containsNull``` flag is defined for the whole column (accross multiple rows). Since this flag could cause removal of null safe check in expressions that will use ```ArrayExcept``` as a child, it could lead to failures with ```NullPointerException``` for the cases as in the second row of the example dataset. WDYT?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21103: [SPARK-23915][SQL] Add array_except function
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21103
Merged build finished. Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #21103: [SPARK-23915][SQL] Add array_except function
Posted by cloud-fan <gi...@git.apache.org>.
Github user cloud-fan commented on a diff in the pull request:
https://github.com/apache/spark/pull/21103#discussion_r205999733
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/collectionOperations.scala ---
@@ -3968,3 +3964,267 @@ object ArrayUnion {
new GenericArrayData(arrayBuffer)
}
}
+
+/**
+ * Returns an array of the elements in the intersect of x and y, without duplicates
+ */
+@ExpressionDescription(
+ usage = """
+ _FUNC_(array1, array2) - Returns an array of the elements in array1 but not in array2,
+ without duplicates.
+ """,
+ examples = """
+ Examples:
+ > SELECT _FUNC_(array(1, 2, 3), array(1, 3, 5));
+ array(2)
+ """,
+ since = "2.4.0")
+case class ArrayExcept(left: Expression, right: Expression) extends ArraySetLike
+ with ComplexTypeMergingExpression {
+ override def dataType: DataType = {
+ dataTypeCheck
+ left.dataType
+ }
+
+ @transient lazy val evalExcept: (ArrayData, ArrayData) => ArrayData = {
+ if (elementTypeSupportEquals) {
+ (array1, array2) =>
+ val hs = new OpenHashSet[Any]
+ var notFoundNullElement = true
+ var i = 0
+ while (i < array2.numElements()) {
+ if (array2.isNullAt(i)) {
+ notFoundNullElement = false
+ } else {
+ val elem = array2.get(i, elementType)
+ hs.add(elem)
+ }
+ i += 1
+ }
+ val arrayBuffer = new scala.collection.mutable.ArrayBuffer[Any]
+ i = 0
+ while (i < array1.numElements()) {
+ if (array1.isNullAt(i)) {
+ if (notFoundNullElement) {
+ arrayBuffer += null
+ notFoundNullElement = false
+ }
+ } else {
+ val elem = array1.get(i, elementType)
+ if (!hs.contains(elem)) {
+ arrayBuffer += elem
+ hs.add(elem)
+ }
+ }
+ i += 1
+ }
+ new GenericArrayData(arrayBuffer)
+ } else {
+ (array1, array2) =>
+ val arrayBuffer = new scala.collection.mutable.ArrayBuffer[Any]
+ var scannedNullElements = false
+ var i = 0
+ while (i < array1.numElements()) {
+ var found = false
+ val elem1 = array1.get(i, elementType)
+ if (elem1 == null) {
+ if (!scannedNullElements) {
+ var j = 0
+ while (!found && j < array2.numElements()) {
+ found = array2.isNullAt(j)
+ j += 1
+ }
+ // array2 is scanned only once for null element
+ scannedNullElements = true
+ } else {
+ found = true
+ }
+ } else {
+ var j = 0
+ while (!found && j < array2.numElements()) {
+ val elem2 = array2.get(j, elementType)
+ if (elem2 != null) {
+ found = ordering.equiv(elem1, elem2)
+ }
+ j += 1
+ }
+ if (!found) {
+ // check whether elem1 is already stored in arrayBuffer
+ var k = 0
+ while (!found && k < arrayBuffer.size) {
+ val va = arrayBuffer(k)
+ found = (va != null) && ordering.equiv(va, elem1)
+ k += 1
+ }
+ }
+ }
+ if (!found) {
+ arrayBuffer += elem1
+ }
+ i += 1
+ }
+ new GenericArrayData(arrayBuffer)
+ }
+ }
+
+ override def nullSafeEval(input1: Any, input2: Any): Any = {
+ val array1 = input1.asInstanceOf[ArrayData]
+ val array2 = input2.asInstanceOf[ArrayData]
+
+ evalExcept(array1, array2)
+ }
+
+ override def doGenCode(ctx: CodegenContext, ev: ExprCode): ExprCode = {
+ val arrayData = classOf[ArrayData].getName
+ val i = ctx.freshName("i")
+ val pos = ctx.freshName("pos")
+ val value = ctx.freshName("value")
+ val hsValue = ctx.freshName("hsValue")
+ val size = ctx.freshName("size")
+ val (postFix, openHashElementType, hsJavaTypeName, genHsValue,
+ getter, setter, javaTypeName, arrayBuilder) =
+ if (elementTypeSupportEquals) {
+ elementType match {
+ case BooleanType | ByteType | ShortType | IntegerType =>
+ val ptName = CodeGenerator.primitiveTypeName(elementType)
+ val unsafeArray = ctx.freshName("unsafeArray")
+ ("$mcI$sp", "Int", "int",
+ if (elementType != BooleanType) {
+ s"(int) $value"
+ } else {
+ s"$value ? 1 : 0;"
+ },
+ s"get$ptName($i)", s"set$ptName($pos, $value)", CodeGenerator.javaType(elementType),
+ s"""
+ |${ctx.createUnsafeArray(unsafeArray, size, elementType, s" $prettyName failed.")}
+ |${ev.value} = $unsafeArray;
+ """.stripMargin)
+ case LongType | FloatType | DoubleType =>
+ val ptName = CodeGenerator.primitiveTypeName(elementType)
+ val unsafeArray = ctx.freshName("unsafeArray")
+ val signature = elementType match {
+ case LongType => "$mcJ$sp"
+ case FloatType => "$mcF$sp"
+ case DoubleType => "$mcD$sp"
+ }
+ (signature, CodeGenerator.boxedType(elementType),
+ CodeGenerator.javaType(elementType), value,
+ s"get$ptName($i)", s"set$ptName($pos, $value)", CodeGenerator.javaType(elementType),
+ s"""
+ |${ctx.createUnsafeArray(unsafeArray, size, elementType, s" $prettyName failed.")}
+ |${ev.value} = $unsafeArray;
+ """.stripMargin)
+ case _ =>
+ val genericArrayData = classOf[GenericArrayData].getName
+ val et = ctx.addReferenceObj("elementType", elementType)
+ ("", "Object", "Object", value,
+ s"get($i, $et)", s"update($pos, $value)", "Object",
+ s"${ev.value} = new $genericArrayData(new Object[$size]);")
+ }
+ } else {
+ ("", "", "", "", "", "", "", "")
+ }
+
+ nullSafeCodeGen(ctx, ev, (array1, array2) => {
+ if (openHashElementType != "") {
--- End diff --
a better way to organize it
```
if (elementTypeSupportEquals) {
...
nullSafeCodeGen(...)
} else {
nullSafeCodeGen(...)
}
```
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #21103: [SPARK-23915][SQL] Add array_except function
Posted by ueshin <gi...@git.apache.org>.
Github user ueshin commented on a diff in the pull request:
https://github.com/apache/spark/pull/21103#discussion_r205186167
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/collectionOperations.scala ---
@@ -3805,3 +3799,330 @@ object ArrayUnion {
new GenericArrayData(arrayBuffer)
}
}
+
+/**
+ * Returns an array of the elements in the intersect of x and y, without duplicates
+ */
+@ExpressionDescription(
+ usage = """
+ _FUNC_(array1, array2) - Returns an array of the elements in array1 but not in array2,
+ without duplicates.
+ """,
+ examples = """
+ Examples:
+ > SELECT _FUNC_(array(1, 2, 3), array(1, 3, 5));
+ array(2)
+ """,
+ since = "2.4.0")
+case class ArrayExcept(left: Expression, right: Expression) extends ArraySetLike {
+ override def dataType: DataType = left.dataType
+
+ var hsInt: OpenHashSet[Int] = _
+ var hsLong: OpenHashSet[Long] = _
+
+ def assignInt(array: ArrayData, idx: Int, resultArray: ArrayData, pos: Int): Boolean = {
+ val elem = array.getInt(idx)
+ if (!hsInt.contains(elem)) {
+ if (resultArray != null) {
+ resultArray.setInt(pos, elem)
+ }
+ hsInt.add(elem)
+ true
+ } else {
+ false
+ }
+ }
+
+ def assignLong(array: ArrayData, idx: Int, resultArray: ArrayData, pos: Int): Boolean = {
+ val elem = array.getLong(idx)
+ if (!hsLong.contains(elem)) {
+ if (resultArray != null) {
+ resultArray.setLong(pos, elem)
+ }
+ hsLong.add(elem)
+ true
+ } else {
+ false
+ }
+ }
+
+ def evalIntLongPrimitiveType(
+ array1: ArrayData,
+ array2: ArrayData,
+ resultArray: ArrayData,
+ isLongType: Boolean): Int = {
+ // store elements into resultArray
+ var notFoundNullElement = true
+ var i = 0
+ while (i < array2.numElements()) {
+ if (array2.isNullAt(i)) {
+ notFoundNullElement = false
+ } else {
+ val assigned = if (!isLongType) {
+ hsInt.add(array2.getInt(i))
+ } else {
+ hsLong.add(array2.getLong(i))
+ }
+ }
+ i += 1
+ }
+ var pos = 0
+ i = 0
+ while (i < array1.numElements()) {
+ if (array1.isNullAt(i)) {
+ if (notFoundNullElement) {
+ if (resultArray != null) {
+ resultArray.setNullAt(pos)
+ }
+ pos += 1
+ notFoundNullElement = false
+ }
+ } else {
+ val assigned = if (!isLongType) {
+ assignInt(array1, i, resultArray, pos)
+ } else {
+ assignLong(array1, i, resultArray, pos)
+ }
+ if (assigned) {
+ pos += 1
+ }
+ }
+ i += 1
+ }
+ pos
+ }
+
+ override def nullSafeEval(input1: Any, input2: Any): Any = {
+ val array1 = input1.asInstanceOf[ArrayData]
+ val array2 = input2.asInstanceOf[ArrayData]
+
+ if (elementTypeSupportEquals) {
+ elementType match {
+ case IntegerType =>
+ // avoid boxing of primitive int array elements
+ // calculate result array size
+ hsInt = new OpenHashSet[Int]
+ val elements = evalIntLongPrimitiveType(array1, array2, null, false)
+ // allocate result array
+ hsInt = new OpenHashSet[Int]
+ val resultArray = if (UnsafeArrayData.shouldUseGenericArrayData(
+ IntegerType.defaultSize, elements)) {
+ new GenericArrayData(new Array[Any](elements))
+ } else {
+ UnsafeArrayData.forPrimitiveArray(
+ Platform.INT_ARRAY_OFFSET, elements, IntegerType.defaultSize)
+ }
+ // assign elements into the result array
+ evalIntLongPrimitiveType(array1, array2, resultArray, false)
+ resultArray
+ case LongType =>
+ // avoid boxing of primitive long array elements
+ // calculate result array size
+ hsLong = new OpenHashSet[Long]
+ val elements = evalIntLongPrimitiveType(array1, array2, null, true)
+ // allocate result array
+ hsLong = new OpenHashSet[Long]
+ val resultArray = if (UnsafeArrayData.shouldUseGenericArrayData(
+ LongType.defaultSize, elements)) {
+ new GenericArrayData(new Array[Any](elements))
+ } else {
+ UnsafeArrayData.forPrimitiveArray(
+ Platform.LONG_ARRAY_OFFSET, elements, LongType.defaultSize)
+ }
+ // assign elements into the result array
+ evalIntLongPrimitiveType(array1, array2, resultArray, true)
+ resultArray
+ case _ =>
+ val hs = new OpenHashSet[Any]
+ var exceptNullElement = true
+ var i = 0
+ while (i < array2.numElements()) {
+ if (array2.isNullAt(i)) {
+ exceptNullElement = false
+ } else {
+ val elem = array2.get(i, elementType)
+ hs.add(elem)
+ }
+ i += 1
+ }
+ val arrayBuffer = new scala.collection.mutable.ArrayBuffer[Any]
+ i = 0
+ while (i < array1.numElements()) {
+ if (array1.isNullAt(i)) {
+ if (exceptNullElement) {
+ arrayBuffer += null
+ exceptNullElement = false
+ }
+ } else {
+ val elem = array1.get(i, elementType)
+ if (!hs.contains(elem)) {
+ arrayBuffer += elem
+ hs.add(elem)
+ }
+ }
+ i += 1
+ }
+ new GenericArrayData(arrayBuffer)
+ }
+ } else {
+ ArrayExcept.exceptOrdering(array1, array2, elementType, ordering)
+ }
+ }
+
+ override def doGenCode(ctx: CodegenContext, ev: ExprCode): ExprCode = {
+ val i = ctx.freshName("i")
+ val pos = ctx.freshName("pos")
+ val value = ctx.freshName("value")
+ val size = ctx.freshName("size")
+ val (postFix, openHashElementType, getter, setter, javaTypeName, castOp, arrayBuilder) =
+ if (elementTypeSupportEquals) {
+ elementType match {
+ case ByteType | ShortType | IntegerType | LongType =>
+ val ptName = CodeGenerator.primitiveTypeName(elementType)
+ val unsafeArray = ctx.freshName("unsafeArray")
+ (if (elementType == LongType) s"$$mcJ$$sp" else s"$$mcI$$sp",
+ if (elementType == LongType) "Long" else "Int",
+ s"get$ptName($i)", s"set$ptName($pos, $value)", CodeGenerator.javaType(elementType),
+ if (elementType == LongType) "(long)" else "(int)",
+ s"""
+ |${ctx.createUnsafeArray(unsafeArray, size, elementType, s" $prettyName failed.")}
+ |${ev.value} = $unsafeArray;
+ """.stripMargin)
+ case _ =>
+ val genericArrayData = classOf[GenericArrayData].getName
+ val et = ctx.addReferenceObj("elementType", elementType)
+ ("", "Object",
+ s"get($i, $et)", s"update($pos, $value)", "Object", "",
+ s"${ev.value} = new $genericArrayData(new Object[$size]);")
+ }
+ } else {
+ ("", "", "", "", "", "", "")
+ }
+
+ nullSafeCodeGen(ctx, ev, (array1, array2) => {
+ if (openHashElementType != "") {
+ // Here, we ensure elementTypeSupportEquals is true
+ val exceptNullElement = ctx.freshName("exceptNullElement")
--- End diff --
ditto.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21103: [SPARK-23915][SQL] Add array_except function
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21103
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/1511/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21103: [SPARK-23915][SQL] Add array_except function
Posted by kiszk <gi...@git.apache.org>.
Github user kiszk commented on the issue:
https://github.com/apache/spark/pull/21103
retest this please
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21103: [SPARK-23915][SQL] Add array_except function
Posted by mn-mikke <gi...@git.apache.org>.
Github user mn-mikke commented on the issue:
https://github.com/apache/spark/pull/21103
Added few minor comments, but nothing serious to be solved right now.
LGTM
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21103: [SPARK-23915][SQL] Add array_except function
Posted by kiszk <gi...@git.apache.org>.
Github user kiszk commented on the issue:
https://github.com/apache/spark/pull/21103
retest this please
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21103: [SPARK-23915][SQL] Add array_except function
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21103
**[Test build #89540 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/89540/testReport)** for PR 21103 at commit [`6813ba4`](https://github.com/apache/spark/commit/6813ba47d4e2e7e107d731390e922aa91f2f31ef).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21103: [SPARK-23915][SQL] Add array_except function
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21103
**[Test build #92997 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/92997/testReport)** for PR 21103 at commit [`5c06922`](https://github.com/apache/spark/commit/5c06922a9492dcb0032e1b60418e25fb57d6fdda).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #21103: [SPARK-23915][SQL] Add array_except function
Posted by kiszk <gi...@git.apache.org>.
Github user kiszk commented on a diff in the pull request:
https://github.com/apache/spark/pull/21103#discussion_r205674364
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/collectionOperations.scala ---
@@ -3805,3 +3799,233 @@ object ArrayUnion {
new GenericArrayData(arrayBuffer)
}
}
+
+/**
+ * Returns an array of the elements in the intersect of x and y, without duplicates
+ */
+@ExpressionDescription(
+ usage = """
+ _FUNC_(array1, array2) - Returns an array of the elements in array1 but not in array2,
+ without duplicates.
+ """,
+ examples = """
+ Examples:
+ > SELECT _FUNC_(array(1, 2, 3), array(1, 3, 5));
+ array(2)
+ """,
+ since = "2.4.0")
+case class ArrayExcept(left: Expression, right: Expression) extends ArraySetLike {
--- End diff --
The check at `dataType` in `ComplexTypeMergingExpression` should be useful.
Based on [the discussion](https://github.com/apache/spark/pull/21103#discussion_r203746577), the result of `dataType should be only `left.dataType`. Will we use only the checks?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #21103: [SPARK-23915][SQL] Add array_except function
Posted by kiszk <gi...@git.apache.org>.
Github user kiszk commented on a diff in the pull request:
https://github.com/apache/spark/pull/21103#discussion_r205672980
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/collectionOperations.scala ---
@@ -3805,3 +3799,233 @@ object ArrayUnion {
new GenericArrayData(arrayBuffer)
}
}
+
+/**
+ * Returns an array of the elements in the intersect of x and y, without duplicates
+ */
+@ExpressionDescription(
+ usage = """
+ _FUNC_(array1, array2) - Returns an array of the elements in array1 but not in array2,
+ without duplicates.
+ """,
+ examples = """
+ Examples:
+ > SELECT _FUNC_(array(1, 2, 3), array(1, 3, 5));
+ array(2)
+ """,
+ since = "2.4.0")
+case class ArrayExcept(left: Expression, right: Expression) extends ArraySetLike {
+ override def dataType: DataType = left.dataType
+
+ var hsInt: OpenHashSet[Int] = _
+ var hsLong: OpenHashSet[Long] = _
+
+ def assignInt(array: ArrayData, idx: Int, resultArray: ArrayData, pos: Int): Boolean = {
+ val elem = array.getInt(idx)
+ if (!hsInt.contains(elem)) {
+ if (resultArray != null) {
+ resultArray.setInt(pos, elem)
+ }
+ hsInt.add(elem)
+ true
+ } else {
+ false
+ }
+ }
+
+ def assignLong(array: ArrayData, idx: Int, resultArray: ArrayData, pos: Int): Boolean = {
+ val elem = array.getLong(idx)
+ if (!hsLong.contains(elem)) {
+ if (resultArray != null) {
+ resultArray.setLong(pos, elem)
+ }
+ hsLong.add(elem)
+ true
+ } else {
+ false
+ }
+ }
+
+ def evalIntLongPrimitiveType(
+ array1: ArrayData,
+ array2: ArrayData,
+ resultArray: ArrayData,
+ isLongType: Boolean): Int = {
+ // store elements into resultArray
+ var notFoundNullElement = true
+ var i = 0
+ while (i < array2.numElements()) {
+ if (array2.isNullAt(i)) {
+ notFoundNullElement = false
+ } else {
+ val assigned = if (!isLongType) {
+ hsInt.add(array2.getInt(i))
+ } else {
+ hsLong.add(array2.getLong(i))
+ }
+ }
+ i += 1
+ }
+ var pos = 0
+ i = 0
+ while (i < array1.numElements()) {
+ if (array1.isNullAt(i)) {
+ if (notFoundNullElement) {
+ if (resultArray != null) {
+ resultArray.setNullAt(pos)
+ }
+ pos += 1
+ notFoundNullElement = false
+ }
+ } else {
+ val assigned = if (!isLongType) {
+ assignInt(array1, i, resultArray, pos)
+ } else {
+ assignLong(array1, i, resultArray, pos)
+ }
+ if (assigned) {
+ pos += 1
+ }
+ }
+ i += 1
+ }
+ pos
+ }
+
+ @transient lazy val evalExcept: (ArrayData, ArrayData) => ArrayData = {
+ if (elementTypeSupportEquals) {
+ elementType match {
+ case IntegerType =>
--- End diff --
Yeah, since the generated code also uses this path, we can optimize byte/short/boolean. However, it is not easy to optimize float and double with `OpenHashSet`. Let me look for hashset than can store primitive float/double.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #21103: [SPARK-23915][SQL] Add array_except function
Posted by kiszk <gi...@git.apache.org>.
Github user kiszk commented on a diff in the pull request:
https://github.com/apache/spark/pull/21103#discussion_r206398377
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/collectionOperations.scala ---
@@ -3968,3 +3964,285 @@ object ArrayUnion {
new GenericArrayData(arrayBuffer)
}
}
+
+/**
+ * Returns an array of the elements in the intersect of x and y, without duplicates
+ */
+@ExpressionDescription(
+ usage = """
+ _FUNC_(array1, array2) - Returns an array of the elements in array1 but not in array2,
+ without duplicates.
+ """,
+ examples = """
+ Examples:
+ > SELECT _FUNC_(array(1, 2, 3), array(1, 3, 5));
+ array(2)
+ """,
+ since = "2.4.0")
+case class ArrayExcept(left: Expression, right: Expression) extends ArraySetLike
+ with ComplexTypeMergingExpression {
+ override def dataType: DataType = {
+ dataTypeCheck
+ left.dataType
+ }
+
+ @transient lazy val evalExcept: (ArrayData, ArrayData) => ArrayData = {
+ if (elementTypeSupportEquals) {
+ (array1, array2) =>
+ val hs = new OpenHashSet[Any]
+ var notFoundNullElement = true
+ var i = 0
+ while (i < array2.numElements()) {
+ if (array2.isNullAt(i)) {
+ notFoundNullElement = false
+ } else {
+ val elem = array2.get(i, elementType)
+ hs.add(elem)
+ }
+ i += 1
+ }
+ val arrayBuffer = new scala.collection.mutable.ArrayBuffer[Any]
+ i = 0
+ while (i < array1.numElements()) {
+ if (array1.isNullAt(i)) {
+ if (notFoundNullElement) {
+ arrayBuffer += null
+ notFoundNullElement = false
+ }
+ } else {
+ val elem = array1.get(i, elementType)
+ if (!hs.contains(elem)) {
+ arrayBuffer += elem
+ hs.add(elem)
+ }
+ }
+ i += 1
+ }
+ new GenericArrayData(arrayBuffer)
+ } else {
+ (array1, array2) =>
+ val arrayBuffer = new scala.collection.mutable.ArrayBuffer[Any]
+ var scannedNullElements = false
+ var i = 0
+ while (i < array1.numElements()) {
+ var found = false
+ val elem1 = array1.get(i, elementType)
+ if (elem1 == null) {
+ if (!scannedNullElements) {
+ var j = 0
+ while (!found && j < array2.numElements()) {
+ found = array2.isNullAt(j)
+ j += 1
+ }
+ // array2 is scanned only once for null element
+ scannedNullElements = true
+ } else {
+ found = true
+ }
+ } else {
+ var j = 0
+ while (!found && j < array2.numElements()) {
+ val elem2 = array2.get(j, elementType)
+ if (elem2 != null) {
+ found = ordering.equiv(elem1, elem2)
+ }
+ j += 1
+ }
+ if (!found) {
+ // check whether elem1 is already stored in arrayBuffer
+ var k = 0
+ while (!found && k < arrayBuffer.size) {
+ val va = arrayBuffer(k)
+ found = (va != null) && ordering.equiv(va, elem1)
+ k += 1
+ }
+ }
+ }
+ if (!found) {
+ arrayBuffer += elem1
+ }
+ i += 1
+ }
+ new GenericArrayData(arrayBuffer)
+ }
+ }
+
+ override def nullSafeEval(input1: Any, input2: Any): Any = {
+ val array1 = input1.asInstanceOf[ArrayData]
+ val array2 = input2.asInstanceOf[ArrayData]
+
+ evalExcept(array1, array2)
+ }
+
+ override def doGenCode(ctx: CodegenContext, ev: ExprCode): ExprCode = {
+ val arrayData = classOf[ArrayData].getName
+ val i = ctx.freshName("i")
+ val pos = ctx.freshName("pos")
+ val value = ctx.freshName("value")
+ val hsValue = ctx.freshName("hsValue")
+ val size = ctx.freshName("size")
+ if (elementTypeSupportEquals) {
+ val ptName = CodeGenerator.primitiveTypeName(elementType)
+ val unsafeArray = ctx.freshName("unsafeArray")
+ val (postFix, openHashElementType, hsJavaTypeName, genHsValue,
+ getter, setter, javaTypeName, primitiveTypeName, arrayDataBuilder) =
+ elementType match {
+ case BooleanType | ByteType | ShortType | IntegerType =>
--- End diff --
Yeah, I agree that it is very rare. Let us remove the special handling code for `BooleanType`.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21103: [SPARK-23915][SQL] Add array_except function
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21103
**[Test build #93870 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/93870/testReport)** for PR 21103 at commit [`93e7979`](https://github.com/apache/spark/commit/93e7979a1c3fb82c47ecae5b3ed539b31cb99e19).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21103: [SPARK-23915][SQL] Add array_except function
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21103
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21103: [SPARK-23915][SQL] Add array_except function
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21103
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/1465/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21103: [SPARK-23915][SQL] Add array_except function
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21103
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/92322/
Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21103: [SPARK-23915][SQL] Add array_except function
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21103
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21103: [SPARK-23915][SQL] Add array_except function
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21103
**[Test build #93682 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/93682/testReport)** for PR 21103 at commit [`548f336`](https://github.com/apache/spark/commit/548f336f9cd84cbc5da276335e2c335d8ba64ab7).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21103: [SPARK-23915][SQL] Add array_except function
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21103
Merged build finished. Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21103: [SPARK-23915][SQL] Add array_except function
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21103
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21103: [SPARK-23915][SQL] Add array_except function
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21103
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/1551/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21103: [SPARK-23915][SQL] Add array_except function
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21103
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21103: [SPARK-23915][SQL] Add array_except function
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21103
**[Test build #93679 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/93679/testReport)** for PR 21103 at commit [`4a70e0e`](https://github.com/apache/spark/commit/4a70e0e1d7ba4bf63d73a025eee262842f9bcaf9).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21103: [SPARK-23915][SQL] Add array_except function
Posted by kiszk <gi...@git.apache.org>.
Github user kiszk commented on the issue:
https://github.com/apache/spark/pull/21103
retest this please
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21103: [SPARK-23915][SQL] Add array_except function
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21103
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21103: [SPARK-23915][SQL] Add array_except function
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21103
**[Test build #89561 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/89561/testReport)** for PR 21103 at commit [`6813ba4`](https://github.com/apache/spark/commit/6813ba47d4e2e7e107d731390e922aa91f2f31ef).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21103: [SPARK-23915][SQL] Add array_except function
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21103
**[Test build #89629 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/89629/testReport)** for PR 21103 at commit [`ad9f576`](https://github.com/apache/spark/commit/ad9f576961384936548adbd38dfb60296d3f0389).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #21103: [SPARK-23915][SQL] Add array_except function
Posted by ueshin <gi...@git.apache.org>.
Github user ueshin commented on a diff in the pull request:
https://github.com/apache/spark/pull/21103#discussion_r204306424
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/collectionOperations.scala ---
@@ -3805,3 +3799,331 @@ object ArrayUnion {
new GenericArrayData(arrayBuffer)
}
}
+
+/**
+ * Returns an array of the elements in the intersect of x and y, without duplicates
+ */
+@ExpressionDescription(
+ usage = """
+ _FUNC_(array1, array2) - Returns an array of the elements in array1 but not in array2,
+ without duplicates.
+ """,
+ examples = """
+ Examples:
+ > SELECT _FUNC_(array(1, 2, 3), array(1, 3, 5));
+ array(2)
+ """,
+ since = "2.4.0")
+case class ArrayExcept(left: Expression, right: Expression) extends ArraySetLike {
+ override def dataType: DataType =
+ ArrayType(elementType, left.dataType.asInstanceOf[ArrayType].containsNull)
--- End diff --
`left.dataType`?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21103: [SPARK-23915][SQL] Add array_except function
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21103
Merged build finished. Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21103: [SPARK-23915][SQL] Add array_except function
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21103
**[Test build #89540 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/89540/testReport)** for PR 21103 at commit [`6813ba4`](https://github.com/apache/spark/commit/6813ba47d4e2e7e107d731390e922aa91f2f31ef).
* This patch **fails due to an unknown error code, -9**.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21103: [SPARK-23915][SQL] Add array_except function
Posted by kiszk <gi...@git.apache.org>.
Github user kiszk commented on the issue:
https://github.com/apache/spark/pull/21103
cc @ueshin
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #21103: [SPARK-23915][SQL] Add array_except function
Posted by cloud-fan <gi...@git.apache.org>.
Github user cloud-fan commented on a diff in the pull request:
https://github.com/apache/spark/pull/21103#discussion_r205733831
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/collectionOperations.scala ---
@@ -3805,3 +3799,233 @@ object ArrayUnion {
new GenericArrayData(arrayBuffer)
}
}
+
+/**
+ * Returns an array of the elements in the intersect of x and y, without duplicates
+ */
+@ExpressionDescription(
+ usage = """
+ _FUNC_(array1, array2) - Returns an array of the elements in array1 but not in array2,
+ without duplicates.
+ """,
+ examples = """
+ Examples:
+ > SELECT _FUNC_(array(1, 2, 3), array(1, 3, 5));
+ array(2)
+ """,
+ since = "2.4.0")
+case class ArrayExcept(left: Expression, right: Expression) extends ArraySetLike {
--- End diff --
we can overwrite `dataType` while still extending `ComplexTypeMergingExpression` to use the checks.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21103: [SPARK-23915][SQL] Add array_except function
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21103
**[Test build #93759 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/93759/testReport)** for PR 21103 at commit [`3639b5b`](https://github.com/apache/spark/commit/3639b5b1b460c19b7c3a787d118a0bcf35cb8e23).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #21103: [SPARK-23915][SQL] Add array_except function
Posted by mn-mikke <gi...@git.apache.org>.
Github user mn-mikke commented on a diff in the pull request:
https://github.com/apache/spark/pull/21103#discussion_r205847180
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/collectionOperations.scala ---
@@ -3968,3 +3964,317 @@ object ArrayUnion {
new GenericArrayData(arrayBuffer)
}
}
+
+/**
+ * Returns an array of the elements in the intersect of x and y, without duplicates
+ */
+@ExpressionDescription(
+ usage = """
+ _FUNC_(array1, array2) - Returns an array of the elements in array1 but not in array2,
+ without duplicates.
+ """,
+ examples = """
+ Examples:
+ > SELECT _FUNC_(array(1, 2, 3), array(1, 3, 5));
+ array(2)
+ """,
+ since = "2.4.0")
+case class ArrayExcept(left: Expression, right: Expression) extends ArraySetLike
+ with ComplexTypeMergingExpression {
--- End diff --
nit: indentation
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21103: [SPARK-23915][SQL] Add array_except function
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21103
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/2497/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21103: [SPARK-23915][SQL] Add array_except function
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21103
Merged build finished. Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21103: [SPARK-23915][SQL] Add array_except function
Posted by gatorsmile <gi...@git.apache.org>.
Github user gatorsmile commented on the issue:
https://github.com/apache/spark/pull/21103
cc @ueshin
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21103: [SPARK-23915][SQL] Add array_except function
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21103
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/1394/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #21103: [SPARK-23915][SQL] Add array_except function
Posted by mn-mikke <gi...@git.apache.org>.
Github user mn-mikke commented on a diff in the pull request:
https://github.com/apache/spark/pull/21103#discussion_r205397802
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/collectionOperations.scala ---
@@ -3805,3 +3799,330 @@ object ArrayUnion {
new GenericArrayData(arrayBuffer)
}
}
+
+/**
+ * Returns an array of the elements in the intersect of x and y, without duplicates
+ */
+@ExpressionDescription(
+ usage = """
+ _FUNC_(array1, array2) - Returns an array of the elements in array1 but not in array2,
+ without duplicates.
+ """,
+ examples = """
+ Examples:
+ > SELECT _FUNC_(array(1, 2, 3), array(1, 3, 5));
+ array(2)
+ """,
+ since = "2.4.0")
+case class ArrayExcept(left: Expression, right: Expression) extends ArraySetLike {
+ override def dataType: DataType = left.dataType
+
+ var hsInt: OpenHashSet[Int] = _
+ var hsLong: OpenHashSet[Long] = _
+
+ def assignInt(array: ArrayData, idx: Int, resultArray: ArrayData, pos: Int): Boolean = {
+ val elem = array.getInt(idx)
+ if (!hsInt.contains(elem)) {
+ if (resultArray != null) {
+ resultArray.setInt(pos, elem)
+ }
+ hsInt.add(elem)
+ true
+ } else {
+ false
+ }
+ }
+
+ def assignLong(array: ArrayData, idx: Int, resultArray: ArrayData, pos: Int): Boolean = {
+ val elem = array.getLong(idx)
+ if (!hsLong.contains(elem)) {
+ if (resultArray != null) {
+ resultArray.setLong(pos, elem)
+ }
+ hsLong.add(elem)
+ true
+ } else {
+ false
+ }
+ }
+
+ def evalIntLongPrimitiveType(
+ array1: ArrayData,
+ array2: ArrayData,
+ resultArray: ArrayData,
+ isLongType: Boolean): Int = {
+ // store elements into resultArray
+ var notFoundNullElement = true
+ var i = 0
+ while (i < array2.numElements()) {
+ if (array2.isNullAt(i)) {
+ notFoundNullElement = false
+ } else {
+ val assigned = if (!isLongType) {
+ hsInt.add(array2.getInt(i))
+ } else {
+ hsLong.add(array2.getLong(i))
+ }
+ }
+ i += 1
+ }
+ var pos = 0
+ i = 0
+ while (i < array1.numElements()) {
+ if (array1.isNullAt(i)) {
+ if (notFoundNullElement) {
+ if (resultArray != null) {
+ resultArray.setNullAt(pos)
+ }
+ pos += 1
+ notFoundNullElement = false
+ }
+ } else {
+ val assigned = if (!isLongType) {
+ assignInt(array1, i, resultArray, pos)
+ } else {
+ assignLong(array1, i, resultArray, pos)
+ }
+ if (assigned) {
+ pos += 1
+ }
+ }
+ i += 1
+ }
+ pos
+ }
+
+ override def nullSafeEval(input1: Any, input2: Any): Any = {
+ val array1 = input1.asInstanceOf[ArrayData]
+ val array2 = input2.asInstanceOf[ArrayData]
+
+ if (elementTypeSupportEquals) {
+ elementType match {
+ case IntegerType =>
+ // avoid boxing of primitive int array elements
+ // calculate result array size
+ hsInt = new OpenHashSet[Int]
+ val elements = evalIntLongPrimitiveType(array1, array2, null, false)
--- End diff --
WDYT about using an ```Option``` instead of ```null``` value?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21103: [SPARK-23915][SQL] Add array_except function
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21103
**[Test build #93280 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/93280/testReport)** for PR 21103 at commit [`26e3257`](https://github.com/apache/spark/commit/26e3257b74fcf830ebcf5bb0f1a7499ad9030b01).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds the following public classes _(experimental)_:
* `case class ArrayUnion(left: Expression, right: Expression) extends ArraySetLike`
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21103: [SPARK-23915][SQL] Add array_except function
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21103
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21103: [SPARK-23915][SQL] Add array_except function
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21103
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/93683/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21103: [SPARK-23915][SQL] Add array_except function
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21103
**[Test build #93582 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/93582/testReport)** for PR 21103 at commit [`cf76c1f`](https://github.com/apache/spark/commit/cf76c1f4a2a41ec88fcd744470a113321e897a71).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21103: [SPARK-23915][SQL] Add array_except function
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21103
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/1414/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21103: [SPARK-23915][SQL] Add array_except function
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21103
**[Test build #93709 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/93709/testReport)** for PR 21103 at commit [`6ef1f22`](https://github.com/apache/spark/commit/6ef1f22aa68d52cf0c00b21211e19d3f80bab7c6).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21103: [SPARK-23915][SQL] Add array_except function
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21103
**[Test build #93825 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/93825/testReport)** for PR 21103 at commit [`282445c`](https://github.com/apache/spark/commit/282445cb20fda8e963ceb23d5f52d88c7f8d7d84).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21103: [SPARK-23915][SQL] Add array_except function
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21103
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/93836/
Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21103: [SPARK-23915][SQL] Add array_except function
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21103
**[Test build #93589 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/93589/testReport)** for PR 21103 at commit [`cf76c1f`](https://github.com/apache/spark/commit/cf76c1f4a2a41ec88fcd744470a113321e897a71).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21103: [SPARK-23915][SQL] Add array_except function
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21103
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/93682/
Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21103: [SPARK-23915][SQL] Add array_except function
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21103
**[Test build #93830 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/93830/testReport)** for PR 21103 at commit [`282445c`](https://github.com/apache/spark/commit/282445cb20fda8e963ceb23d5f52d88c7f8d7d84).
* This patch **fails PySpark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21103: [SPARK-23915][SQL] Add array_except function
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21103
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21103: [SPARK-23915][SQL] Add array_except function
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21103
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21103: [SPARK-23915][SQL] Add array_except function
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21103
**[Test build #93851 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/93851/testReport)** for PR 21103 at commit [`93e7979`](https://github.com/apache/spark/commit/93e7979a1c3fb82c47ecae5b3ed539b31cb99e19).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21103: [SPARK-23915][SQL] Add array_except function
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21103
Merged build finished. Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #21103: [SPARK-23915][SQL] Add array_except function
Posted by kiszk <gi...@git.apache.org>.
Github user kiszk commented on a diff in the pull request:
https://github.com/apache/spark/pull/21103#discussion_r205495042
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/collectionOperations.scala ---
@@ -3805,3 +3799,330 @@ object ArrayUnion {
new GenericArrayData(arrayBuffer)
}
}
+
+/**
+ * Returns an array of the elements in the intersect of x and y, without duplicates
+ */
+@ExpressionDescription(
+ usage = """
+ _FUNC_(array1, array2) - Returns an array of the elements in array1 but not in array2,
+ without duplicates.
+ """,
+ examples = """
+ Examples:
+ > SELECT _FUNC_(array(1, 2, 3), array(1, 3, 5));
+ array(2)
+ """,
+ since = "2.4.0")
+case class ArrayExcept(left: Expression, right: Expression) extends ArraySetLike {
+ override def dataType: DataType = left.dataType
+
+ var hsInt: OpenHashSet[Int] = _
+ var hsLong: OpenHashSet[Long] = _
+
+ def assignInt(array: ArrayData, idx: Int, resultArray: ArrayData, pos: Int): Boolean = {
+ val elem = array.getInt(idx)
+ if (!hsInt.contains(elem)) {
+ if (resultArray != null) {
+ resultArray.setInt(pos, elem)
+ }
+ hsInt.add(elem)
+ true
+ } else {
+ false
+ }
+ }
+
+ def assignLong(array: ArrayData, idx: Int, resultArray: ArrayData, pos: Int): Boolean = {
+ val elem = array.getLong(idx)
+ if (!hsLong.contains(elem)) {
+ if (resultArray != null) {
+ resultArray.setLong(pos, elem)
+ }
+ hsLong.add(elem)
+ true
+ } else {
+ false
+ }
+ }
+
+ def evalIntLongPrimitiveType(
+ array1: ArrayData,
+ array2: ArrayData,
+ resultArray: ArrayData,
+ isLongType: Boolean): Int = {
+ // store elements into resultArray
+ var notFoundNullElement = true
+ var i = 0
+ while (i < array2.numElements()) {
+ if (array2.isNullAt(i)) {
+ notFoundNullElement = false
+ } else {
+ val assigned = if (!isLongType) {
+ hsInt.add(array2.getInt(i))
+ } else {
+ hsLong.add(array2.getLong(i))
+ }
+ }
+ i += 1
+ }
+ var pos = 0
+ i = 0
+ while (i < array1.numElements()) {
+ if (array1.isNullAt(i)) {
+ if (notFoundNullElement) {
+ if (resultArray != null) {
+ resultArray.setNullAt(pos)
+ }
+ pos += 1
+ notFoundNullElement = false
+ }
+ } else {
+ val assigned = if (!isLongType) {
+ assignInt(array1, i, resultArray, pos)
+ } else {
+ assignLong(array1, i, resultArray, pos)
+ }
+ if (assigned) {
+ pos += 1
+ }
+ }
+ i += 1
+ }
+ pos
+ }
+
+ override def nullSafeEval(input1: Any, input2: Any): Any = {
+ val array1 = input1.asInstanceOf[ArrayData]
+ val array2 = input2.asInstanceOf[ArrayData]
+
+ if (elementTypeSupportEquals) {
+ elementType match {
+ case IntegerType =>
+ // avoid boxing of primitive int array elements
+ // calculate result array size
+ hsInt = new OpenHashSet[Int]
+ val elements = evalIntLongPrimitiveType(array1, array2, null, false)
+ // allocate result array
+ hsInt = new OpenHashSet[Int]
+ val resultArray = if (UnsafeArrayData.shouldUseGenericArrayData(
+ IntegerType.defaultSize, elements)) {
+ new GenericArrayData(new Array[Any](elements))
+ } else {
+ UnsafeArrayData.forPrimitiveArray(
+ Platform.INT_ARRAY_OFFSET, elements, IntegerType.defaultSize)
+ }
+ // assign elements into the result array
+ evalIntLongPrimitiveType(array1, array2, resultArray, false)
+ resultArray
+ case LongType =>
+ // avoid boxing of primitive long array elements
+ // calculate result array size
+ hsLong = new OpenHashSet[Long]
+ val elements = evalIntLongPrimitiveType(array1, array2, null, true)
+ // allocate result array
+ hsLong = new OpenHashSet[Long]
+ val resultArray = if (UnsafeArrayData.shouldUseGenericArrayData(
+ LongType.defaultSize, elements)) {
+ new GenericArrayData(new Array[Any](elements))
+ } else {
+ UnsafeArrayData.forPrimitiveArray(
+ Platform.LONG_ARRAY_OFFSET, elements, LongType.defaultSize)
+ }
+ // assign elements into the result array
+ evalIntLongPrimitiveType(array1, array2, resultArray, true)
+ resultArray
+ case _ =>
+ val hs = new OpenHashSet[Any]
+ var notFoundNullElement = true
+ var i = 0
+ while (i < array2.numElements()) {
+ if (array2.isNullAt(i)) {
+ notFoundNullElement = false
+ } else {
+ val elem = array2.get(i, elementType)
+ hs.add(elem)
+ }
+ i += 1
+ }
+ val arrayBuffer = new scala.collection.mutable.ArrayBuffer[Any]
+ i = 0
+ while (i < array1.numElements()) {
+ if (array1.isNullAt(i)) {
+ if (notFoundNullElement) {
+ arrayBuffer += null
+ notFoundNullElement = false
+ }
+ } else {
+ val elem = array1.get(i, elementType)
+ if (!hs.contains(elem)) {
+ arrayBuffer += elem
+ hs.add(elem)
+ }
+ }
+ i += 1
+ }
+ new GenericArrayData(arrayBuffer)
+ }
+ } else {
+ ArrayExcept.exceptOrdering(array1, array2, elementType, ordering)
+ }
+ }
+
+ override def doGenCode(ctx: CodegenContext, ev: ExprCode): ExprCode = {
--- End diff --
Seem to make sense. Codegen can only increase the number of supporting types (i.e. byte/short).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21103: [SPARK-23915][SQL] Add array_except function
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21103
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21103: [SPARK-23915][SQL] Add array_except function
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21103
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/89587/
Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21103: [SPARK-23915][SQL] Add array_except function
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21103
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/1208/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21103: [SPARK-23915][SQL] Add array_except function
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21103
**[Test build #93759 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/93759/testReport)** for PR 21103 at commit [`3639b5b`](https://github.com/apache/spark/commit/3639b5b1b460c19b7c3a787d118a0bcf35cb8e23).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21103: [SPARK-23915][SQL] Add array_except function
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21103
Merged build finished. Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21103: [SPARK-23915][SQL] Add array_except function
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21103
**[Test build #89589 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/89589/testReport)** for PR 21103 at commit [`2551c09`](https://github.com/apache/spark/commit/2551c0989e1afdcd207ae7d680959b2a830a6624).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21103: [SPARK-23915][SQL] Add array_except function
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21103
**[Test build #89524 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/89524/testReport)** for PR 21103 at commit [`ce18ce0`](https://github.com/apache/spark/commit/ce18ce0cd4da3378bb9f83237b7f9165737e7760).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds the following public classes _(experimental)_:
* `abstract class ArraySetUtils extends BinaryExpression with ExpectsInputTypes `
* `case class ArrayExcept(array1: Expression, array2: Expression) extends ArraySetUtils `
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21103: [SPARK-23915][SQL] Add array_except function
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21103
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21103: [SPARK-23915][SQL] Add array_except function
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21103
**[Test build #93739 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/93739/testReport)** for PR 21103 at commit [`3639b5b`](https://github.com/apache/spark/commit/3639b5b1b460c19b7c3a787d118a0bcf35cb8e23).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #21103: [SPARK-23915][SQL] Add array_except function
Posted by kiszk <gi...@git.apache.org>.
Github user kiszk commented on a diff in the pull request:
https://github.com/apache/spark/pull/21103#discussion_r203960424
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/collectionOperations.scala ---
@@ -3805,3 +3799,332 @@ object ArrayUnion {
new GenericArrayData(arrayBuffer)
}
}
+
+/**
+ * Returns an array of the elements in the intersect of x and y, without duplicates
+ */
+@ExpressionDescription(
+ usage = """
+ _FUNC_(array1, array2) - Returns an array of the elements in array1 but not in array2,
+ without duplicates.
+ """,
+ examples = """
+ Examples:Fun
+ > SELECT _FUNC_(array(1, 2, 3), array(1, 3, 5));
+ array(2)
+ """,
+ since = "2.4.0")
+case class ArrayExcept(left: Expression, right: Expression) extends ArraySetLike {
+ override def dataType: DataType = ArrayType(elementType,
--- End diff --
How about the case of left=[1, null], right[2,null], and result=[1]?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21103: [SPARK-23915][SQL] Add array_except function
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21103
**[Test build #93392 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/93392/testReport)** for PR 21103 at commit [`991b0c8`](https://github.com/apache/spark/commit/991b0c86146e53bf0bfd6ed190e9f40a81c1b258).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21103: [SPARK-23915][SQL] Add array_except function
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21103
Merged build finished. Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #21103: [SPARK-23915][SQL] Add array_except function
Posted by kiszk <gi...@git.apache.org>.
Github user kiszk commented on a diff in the pull request:
https://github.com/apache/spark/pull/21103#discussion_r205310335
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/collectionOperations.scala ---
@@ -3805,3 +3799,330 @@ object ArrayUnion {
new GenericArrayData(arrayBuffer)
}
}
+
+/**
+ * Returns an array of the elements in the intersect of x and y, without duplicates
+ */
+@ExpressionDescription(
+ usage = """
+ _FUNC_(array1, array2) - Returns an array of the elements in array1 but not in array2,
+ without duplicates.
+ """,
+ examples = """
+ Examples:
+ > SELECT _FUNC_(array(1, 2, 3), array(1, 3, 5));
+ array(2)
+ """,
+ since = "2.4.0")
+case class ArrayExcept(left: Expression, right: Expression) extends ArraySetLike {
+ override def dataType: DataType = left.dataType
+
+ var hsInt: OpenHashSet[Int] = _
+ var hsLong: OpenHashSet[Long] = _
+
+ def assignInt(array: ArrayData, idx: Int, resultArray: ArrayData, pos: Int): Boolean = {
+ val elem = array.getInt(idx)
+ if (!hsInt.contains(elem)) {
+ if (resultArray != null) {
+ resultArray.setInt(pos, elem)
+ }
+ hsInt.add(elem)
+ true
+ } else {
+ false
+ }
+ }
+
+ def assignLong(array: ArrayData, idx: Int, resultArray: ArrayData, pos: Int): Boolean = {
+ val elem = array.getLong(idx)
+ if (!hsLong.contains(elem)) {
+ if (resultArray != null) {
+ resultArray.setLong(pos, elem)
+ }
+ hsLong.add(elem)
+ true
+ } else {
+ false
+ }
+ }
+
+ def evalIntLongPrimitiveType(
+ array1: ArrayData,
+ array2: ArrayData,
+ resultArray: ArrayData,
+ isLongType: Boolean): Int = {
+ // store elements into resultArray
+ var notFoundNullElement = true
+ var i = 0
+ while (i < array2.numElements()) {
+ if (array2.isNullAt(i)) {
+ notFoundNullElement = false
+ } else {
+ val assigned = if (!isLongType) {
+ hsInt.add(array2.getInt(i))
+ } else {
+ hsLong.add(array2.getLong(i))
+ }
+ }
+ i += 1
+ }
+ var pos = 0
+ i = 0
+ while (i < array1.numElements()) {
+ if (array1.isNullAt(i)) {
+ if (notFoundNullElement) {
+ if (resultArray != null) {
+ resultArray.setNullAt(pos)
+ }
+ pos += 1
+ notFoundNullElement = false
+ }
+ } else {
+ val assigned = if (!isLongType) {
+ assignInt(array1, i, resultArray, pos)
+ } else {
+ assignLong(array1, i, resultArray, pos)
+ }
+ if (assigned) {
+ pos += 1
+ }
+ }
+ i += 1
+ }
+ pos
+ }
+
+ override def nullSafeEval(input1: Any, input2: Any): Any = {
+ val array1 = input1.asInstanceOf[ArrayData]
+ val array2 = input2.asInstanceOf[ArrayData]
+
+ if (elementTypeSupportEquals) {
+ elementType match {
+ case IntegerType =>
+ // avoid boxing of primitive int array elements
+ // calculate result array size
+ hsInt = new OpenHashSet[Int]
+ val elements = evalIntLongPrimitiveType(array1, array2, null, false)
+ // allocate result array
+ hsInt = new OpenHashSet[Int]
+ val resultArray = if (UnsafeArrayData.shouldUseGenericArrayData(
+ IntegerType.defaultSize, elements)) {
+ new GenericArrayData(new Array[Any](elements))
+ } else {
+ UnsafeArrayData.forPrimitiveArray(
+ Platform.INT_ARRAY_OFFSET, elements, IntegerType.defaultSize)
+ }
+ // assign elements into the result array
+ evalIntLongPrimitiveType(array1, array2, resultArray, false)
+ resultArray
+ case LongType =>
+ // avoid boxing of primitive long array elements
+ // calculate result array size
+ hsLong = new OpenHashSet[Long]
+ val elements = evalIntLongPrimitiveType(array1, array2, null, true)
+ // allocate result array
+ hsLong = new OpenHashSet[Long]
+ val resultArray = if (UnsafeArrayData.shouldUseGenericArrayData(
+ LongType.defaultSize, elements)) {
+ new GenericArrayData(new Array[Any](elements))
+ } else {
+ UnsafeArrayData.forPrimitiveArray(
+ Platform.LONG_ARRAY_OFFSET, elements, LongType.defaultSize)
+ }
+ // assign elements into the result array
+ evalIntLongPrimitiveType(array1, array2, resultArray, true)
+ resultArray
+ case _ =>
+ val hs = new OpenHashSet[Any]
+ var exceptNullElement = true
--- End diff --
Good catch, thanks
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21103: [SPARK-23915][SQL] Add array_except function
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21103
**[Test build #93679 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/93679/testReport)** for PR 21103 at commit [`4a70e0e`](https://github.com/apache/spark/commit/4a70e0e1d7ba4bf63d73a025eee262842f9bcaf9).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #21103: [SPARK-23915][SQL] Add array_except function
Posted by ueshin <gi...@git.apache.org>.
Github user ueshin commented on a diff in the pull request:
https://github.com/apache/spark/pull/21103#discussion_r205186241
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/collectionOperations.scala ---
@@ -3805,3 +3799,330 @@ object ArrayUnion {
new GenericArrayData(arrayBuffer)
}
}
+
+/**
+ * Returns an array of the elements in the intersect of x and y, without duplicates
+ */
+@ExpressionDescription(
+ usage = """
+ _FUNC_(array1, array2) - Returns an array of the elements in array1 but not in array2,
+ without duplicates.
+ """,
+ examples = """
+ Examples:
+ > SELECT _FUNC_(array(1, 2, 3), array(1, 3, 5));
+ array(2)
+ """,
+ since = "2.4.0")
+case class ArrayExcept(left: Expression, right: Expression) extends ArraySetLike {
+ override def dataType: DataType = left.dataType
+
+ var hsInt: OpenHashSet[Int] = _
+ var hsLong: OpenHashSet[Long] = _
+
+ def assignInt(array: ArrayData, idx: Int, resultArray: ArrayData, pos: Int): Boolean = {
+ val elem = array.getInt(idx)
+ if (!hsInt.contains(elem)) {
+ if (resultArray != null) {
+ resultArray.setInt(pos, elem)
+ }
+ hsInt.add(elem)
+ true
+ } else {
+ false
+ }
+ }
+
+ def assignLong(array: ArrayData, idx: Int, resultArray: ArrayData, pos: Int): Boolean = {
+ val elem = array.getLong(idx)
+ if (!hsLong.contains(elem)) {
+ if (resultArray != null) {
+ resultArray.setLong(pos, elem)
+ }
+ hsLong.add(elem)
+ true
+ } else {
+ false
+ }
+ }
+
+ def evalIntLongPrimitiveType(
+ array1: ArrayData,
+ array2: ArrayData,
+ resultArray: ArrayData,
+ isLongType: Boolean): Int = {
+ // store elements into resultArray
+ var notFoundNullElement = true
+ var i = 0
+ while (i < array2.numElements()) {
+ if (array2.isNullAt(i)) {
+ notFoundNullElement = false
+ } else {
+ val assigned = if (!isLongType) {
+ hsInt.add(array2.getInt(i))
+ } else {
+ hsLong.add(array2.getLong(i))
+ }
+ }
+ i += 1
+ }
+ var pos = 0
+ i = 0
+ while (i < array1.numElements()) {
+ if (array1.isNullAt(i)) {
+ if (notFoundNullElement) {
+ if (resultArray != null) {
+ resultArray.setNullAt(pos)
+ }
+ pos += 1
+ notFoundNullElement = false
+ }
+ } else {
+ val assigned = if (!isLongType) {
+ assignInt(array1, i, resultArray, pos)
+ } else {
+ assignLong(array1, i, resultArray, pos)
+ }
+ if (assigned) {
+ pos += 1
+ }
+ }
+ i += 1
+ }
+ pos
+ }
+
+ override def nullSafeEval(input1: Any, input2: Any): Any = {
+ val array1 = input1.asInstanceOf[ArrayData]
+ val array2 = input2.asInstanceOf[ArrayData]
+
+ if (elementTypeSupportEquals) {
+ elementType match {
+ case IntegerType =>
+ // avoid boxing of primitive int array elements
+ // calculate result array size
+ hsInt = new OpenHashSet[Int]
+ val elements = evalIntLongPrimitiveType(array1, array2, null, false)
+ // allocate result array
+ hsInt = new OpenHashSet[Int]
+ val resultArray = if (UnsafeArrayData.shouldUseGenericArrayData(
+ IntegerType.defaultSize, elements)) {
+ new GenericArrayData(new Array[Any](elements))
+ } else {
+ UnsafeArrayData.forPrimitiveArray(
+ Platform.INT_ARRAY_OFFSET, elements, IntegerType.defaultSize)
+ }
+ // assign elements into the result array
+ evalIntLongPrimitiveType(array1, array2, resultArray, false)
+ resultArray
+ case LongType =>
+ // avoid boxing of primitive long array elements
+ // calculate result array size
+ hsLong = new OpenHashSet[Long]
+ val elements = evalIntLongPrimitiveType(array1, array2, null, true)
+ // allocate result array
+ hsLong = new OpenHashSet[Long]
+ val resultArray = if (UnsafeArrayData.shouldUseGenericArrayData(
+ LongType.defaultSize, elements)) {
+ new GenericArrayData(new Array[Any](elements))
+ } else {
+ UnsafeArrayData.forPrimitiveArray(
+ Platform.LONG_ARRAY_OFFSET, elements, LongType.defaultSize)
+ }
+ // assign elements into the result array
+ evalIntLongPrimitiveType(array1, array2, resultArray, true)
+ resultArray
+ case _ =>
+ val hs = new OpenHashSet[Any]
+ var exceptNullElement = true
+ var i = 0
+ while (i < array2.numElements()) {
+ if (array2.isNullAt(i)) {
+ exceptNullElement = false
+ } else {
+ val elem = array2.get(i, elementType)
+ hs.add(elem)
+ }
+ i += 1
+ }
+ val arrayBuffer = new scala.collection.mutable.ArrayBuffer[Any]
+ i = 0
+ while (i < array1.numElements()) {
+ if (array1.isNullAt(i)) {
+ if (exceptNullElement) {
+ arrayBuffer += null
+ exceptNullElement = false
+ }
+ } else {
+ val elem = array1.get(i, elementType)
+ if (!hs.contains(elem)) {
+ arrayBuffer += elem
+ hs.add(elem)
+ }
+ }
+ i += 1
+ }
+ new GenericArrayData(arrayBuffer)
+ }
+ } else {
+ ArrayExcept.exceptOrdering(array1, array2, elementType, ordering)
+ }
+ }
+
+ override def doGenCode(ctx: CodegenContext, ev: ExprCode): ExprCode = {
+ val i = ctx.freshName("i")
+ val pos = ctx.freshName("pos")
+ val value = ctx.freshName("value")
+ val size = ctx.freshName("size")
+ val (postFix, openHashElementType, getter, setter, javaTypeName, castOp, arrayBuilder) =
+ if (elementTypeSupportEquals) {
+ elementType match {
+ case ByteType | ShortType | IntegerType | LongType =>
+ val ptName = CodeGenerator.primitiveTypeName(elementType)
+ val unsafeArray = ctx.freshName("unsafeArray")
+ (if (elementType == LongType) s"$$mcJ$$sp" else s"$$mcI$$sp",
+ if (elementType == LongType) "Long" else "Int",
+ s"get$ptName($i)", s"set$ptName($pos, $value)", CodeGenerator.javaType(elementType),
+ if (elementType == LongType) "(long)" else "(int)",
+ s"""
+ |${ctx.createUnsafeArray(unsafeArray, size, elementType, s" $prettyName failed.")}
+ |${ev.value} = $unsafeArray;
+ """.stripMargin)
+ case _ =>
+ val genericArrayData = classOf[GenericArrayData].getName
+ val et = ctx.addReferenceObj("elementType", elementType)
+ ("", "Object",
+ s"get($i, $et)", s"update($pos, $value)", "Object", "",
+ s"${ev.value} = new $genericArrayData(new Object[$size]);")
+ }
+ } else {
+ ("", "", "", "", "", "", "")
+ }
+
+ nullSafeCodeGen(ctx, ev, (array1, array2) => {
+ if (openHashElementType != "") {
+ // Here, we ensure elementTypeSupportEquals is true
+ val exceptNullElement = ctx.freshName("exceptNullElement")
+ val openHashSet = classOf[OpenHashSet[_]].getName
+ val classTag = s"scala.reflect.ClassTag$$.MODULE$$.$openHashElementType()"
+ val hs = ctx.freshName("hs")
+ val arrayData = classOf[ArrayData].getName
+ val arrays = ctx.freshName("arrays")
+ val array = ctx.freshName("array")
+ val arrayDataIdx = ctx.freshName("arrayDataIdx")
+ s"""
+ |$openHashSet $hs = new $openHashSet$postFix($classTag);
+ |boolean $exceptNullElement = true;
+ |int $size = 0;
+ |for (int $i = 0; $i < $array2.numElements(); $i++) {
+ | if ($array2.isNullAt($i)) {
+ | $exceptNullElement = false;
+ | } else {
+ | $hs.add$postFix($array2.$getter);
+ | }
+ |}
+ |for (int $i = 0; $i < $array1.numElements(); $i++) {
+ | if ($array1.isNullAt($i)) {
+ | if ($exceptNullElement) {
+ | $size++;
+ | $exceptNullElement = false;
+ | }
+ | } else {
+ | $javaTypeName $value = $array1.$getter;
+ | if (!$hs.contains($castOp $value)) {
+ | $hs.add$postFix($value);
+ | $size++;
+ | }
+ | }
+ |}
+ |$arrayBuilder
+ |$hs = new $openHashSet$postFix($classTag);
+ |$exceptNullElement = true;
+ |int $pos = 0;
+ |for (int $i = 0; $i < $array2.numElements(); $i++) {
+ | if ($array2.isNullAt($i)) {
+ | $exceptNullElement = false;
+ | } else {
+ | $hs.add$postFix($array2.$getter);
+ | }
+ |}
+ |for (int $i = 0; $i < $array1.numElements(); $i++) {
+ | if ($array1.isNullAt($i)) {
+ | if ($exceptNullElement) {
+ | ${ev.value}.setNullAt($pos++);
+ | $exceptNullElement = false;
+ | }
+ | } else {
+ | $javaTypeName $value = $array1.$getter;
+ | if (!$hs.contains($castOp $value)) {
+ | $hs.add$postFix($value);
+ | ${ev.value}.$setter;
+ | $pos++;
+ | }
+ | }
+ |}
+ """.stripMargin
+ } else {
+ val arrayExcept = classOf[ArrayExcept].getName
+ val et = ctx.addReferenceObj("elementTypeIntersect", elementType)
+ val order = ctx.addReferenceObj("orderingIntersect", ordering)
+ val method = "exceptOrdering"
+ s"${ev.value} = $arrayExcept$$.MODULE$$.$method($array1, $array2, $et, $order);"
+ }
+ })
+ }
+
+ override def prettyName: String = "array_except"
+}
+
+object ArrayExcept {
+ def exceptOrdering(
+ array1: ArrayData,
+ array2: ArrayData,
+ elementType: DataType,
+ ordering: Ordering[Any]): ArrayData = {
+ val arrayBuffer = new scala.collection.mutable.ArrayBuffer[Any]
+ var exceptNullElement = false
--- End diff --
ditto.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21103: [SPARK-23915][SQL] Add array_except function
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21103
**[Test build #93831 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/93831/testReport)** for PR 21103 at commit [`72ef664`](https://github.com/apache/spark/commit/72ef66479842df6fe793fad39dfcb0f7e494b0fd).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21103: [SPARK-23915][SQL] Add array_except function
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21103
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/1259/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21103: [SPARK-23915][SQL] Add array_except function
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21103
Merged build finished. Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21103: [SPARK-23915][SQL] Add array_except function
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21103
Merged build finished. Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21103: [SPARK-23915][SQL] Add array_except function
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21103
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/1520/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21103: [SPARK-23915][SQL] Add array_except function
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21103
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/1454/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21103: [SPARK-23915][SQL] Add array_except function
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21103
Merged build finished. Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21103: [SPARK-23915][SQL] Add array_except function
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21103
**[Test build #93836 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/93836/testReport)** for PR 21103 at commit [`93e7979`](https://github.com/apache/spark/commit/93e7979a1c3fb82c47ecae5b3ed539b31cb99e19).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21103: [SPARK-23915][SQL] Add array_except function
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21103
**[Test build #93392 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/93392/testReport)** for PR 21103 at commit [`991b0c8`](https://github.com/apache/spark/commit/991b0c86146e53bf0bfd6ed190e9f40a81c1b258).
* This patch **fails Scala style tests**.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21103: [SPARK-23915][SQL] Add array_except function
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21103
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/1418/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #21103: [SPARK-23915][SQL] Add array_except function
Posted by kiszk <gi...@git.apache.org>.
Github user kiszk commented on a diff in the pull request:
https://github.com/apache/spark/pull/21103#discussion_r205677242
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/collectionOperations.scala ---
@@ -3805,3 +3799,233 @@ object ArrayUnion {
new GenericArrayData(arrayBuffer)
}
}
+
+/**
+ * Returns an array of the elements in the intersect of x and y, without duplicates
+ */
+@ExpressionDescription(
+ usage = """
+ _FUNC_(array1, array2) - Returns an array of the elements in array1 but not in array2,
+ without duplicates.
+ """,
+ examples = """
+ Examples:
+ > SELECT _FUNC_(array(1, 2, 3), array(1, 3, 5));
+ array(2)
+ """,
+ since = "2.4.0")
+case class ArrayExcept(left: Expression, right: Expression) extends ArraySetLike {
+ override def dataType: DataType = left.dataType
+
+ var hsInt: OpenHashSet[Int] = _
+ var hsLong: OpenHashSet[Long] = _
+
+ def assignInt(array: ArrayData, idx: Int, resultArray: ArrayData, pos: Int): Boolean = {
+ val elem = array.getInt(idx)
+ if (!hsInt.contains(elem)) {
+ if (resultArray != null) {
+ resultArray.setInt(pos, elem)
+ }
+ hsInt.add(elem)
+ true
+ } else {
+ false
+ }
+ }
+
+ def assignLong(array: ArrayData, idx: Int, resultArray: ArrayData, pos: Int): Boolean = {
+ val elem = array.getLong(idx)
+ if (!hsLong.contains(elem)) {
+ if (resultArray != null) {
+ resultArray.setLong(pos, elem)
+ }
+ hsLong.add(elem)
+ true
+ } else {
+ false
+ }
+ }
+
+ def evalIntLongPrimitiveType(
+ array1: ArrayData,
+ array2: ArrayData,
+ resultArray: ArrayData,
+ isLongType: Boolean): Int = {
+ // store elements into resultArray
+ var notFoundNullElement = true
+ var i = 0
+ while (i < array2.numElements()) {
+ if (array2.isNullAt(i)) {
+ notFoundNullElement = false
+ } else {
+ val assigned = if (!isLongType) {
+ hsInt.add(array2.getInt(i))
+ } else {
+ hsLong.add(array2.getLong(i))
+ }
+ }
+ i += 1
+ }
+ var pos = 0
+ i = 0
+ while (i < array1.numElements()) {
+ if (array1.isNullAt(i)) {
+ if (notFoundNullElement) {
+ if (resultArray != null) {
+ resultArray.setNullAt(pos)
+ }
+ pos += 1
+ notFoundNullElement = false
+ }
+ } else {
+ val assigned = if (!isLongType) {
+ assignInt(array1, i, resultArray, pos)
+ } else {
+ assignLong(array1, i, resultArray, pos)
+ }
+ if (assigned) {
+ pos += 1
+ }
+ }
+ i += 1
+ }
+ pos
+ }
+
+ @transient lazy val evalExcept: (ArrayData, ArrayData) => ArrayData = {
+ if (elementTypeSupportEquals) {
+ elementType match {
+ case IntegerType =>
--- End diff --
Let me try support float/double in `OpenHashSet`.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21103: [SPARK-23915][SQL] Add array_except function
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21103
**[Test build #93487 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/93487/testReport)** for PR 21103 at commit [`f099cbf`](https://github.com/apache/spark/commit/f099cbff16c8ab3c6975837a38a984eb6a2fe1b6).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21103: [SPARK-23915][SQL] Add array_except function
Posted by cloud-fan <gi...@git.apache.org>.
Github user cloud-fan commented on the issue:
https://github.com/apache/spark/pull/21103
thanks, merging to master!
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21103: [SPARK-23915][SQL] Add array_except function
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21103
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/93280/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21103: [SPARK-23915][SQL] Add array_except function
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21103
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/1521/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21103: [SPARK-23915][SQL] Add array_except function
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21103
**[Test build #93830 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/93830/testReport)** for PR 21103 at commit [`282445c`](https://github.com/apache/spark/commit/282445cb20fda8e963ceb23d5f52d88c7f8d7d84).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #21103: [SPARK-23915][SQL] Add array_except function
Posted by kiszk <gi...@git.apache.org>.
Github user kiszk commented on a diff in the pull request:
https://github.com/apache/spark/pull/21103#discussion_r205674062
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/collectionOperations.scala ---
@@ -3805,3 +3799,233 @@ object ArrayUnion {
new GenericArrayData(arrayBuffer)
}
}
+
+/**
+ * Returns an array of the elements in the intersect of x and y, without duplicates
+ */
+@ExpressionDescription(
+ usage = """
+ _FUNC_(array1, array2) - Returns an array of the elements in array1 but not in array2,
+ without duplicates.
+ """,
+ examples = """
+ Examples:
+ > SELECT _FUNC_(array(1, 2, 3), array(1, 3, 5));
+ array(2)
+ """,
+ since = "2.4.0")
+case class ArrayExcept(left: Expression, right: Expression) extends ArraySetLike {
+ override def dataType: DataType = left.dataType
+
+ var hsInt: OpenHashSet[Int] = _
+ var hsLong: OpenHashSet[Long] = _
+
+ def assignInt(array: ArrayData, idx: Int, resultArray: ArrayData, pos: Int): Boolean = {
+ val elem = array.getInt(idx)
+ if (!hsInt.contains(elem)) {
+ if (resultArray != null) {
+ resultArray.setInt(pos, elem)
+ }
+ hsInt.add(elem)
+ true
+ } else {
+ false
+ }
+ }
+
+ def assignLong(array: ArrayData, idx: Int, resultArray: ArrayData, pos: Int): Boolean = {
+ val elem = array.getLong(idx)
+ if (!hsLong.contains(elem)) {
+ if (resultArray != null) {
+ resultArray.setLong(pos, elem)
+ }
+ hsLong.add(elem)
+ true
+ } else {
+ false
+ }
+ }
+
+ def evalIntLongPrimitiveType(
+ array1: ArrayData,
+ array2: ArrayData,
+ resultArray: ArrayData,
+ isLongType: Boolean): Int = {
+ // store elements into resultArray
+ var notFoundNullElement = true
+ var i = 0
+ while (i < array2.numElements()) {
+ if (array2.isNullAt(i)) {
+ notFoundNullElement = false
+ } else {
+ val assigned = if (!isLongType) {
+ hsInt.add(array2.getInt(i))
+ } else {
+ hsLong.add(array2.getLong(i))
+ }
+ }
+ i += 1
+ }
+ var pos = 0
+ i = 0
+ while (i < array1.numElements()) {
+ if (array1.isNullAt(i)) {
+ if (notFoundNullElement) {
+ if (resultArray != null) {
+ resultArray.setNullAt(pos)
+ }
+ pos += 1
+ notFoundNullElement = false
+ }
+ } else {
+ val assigned = if (!isLongType) {
+ assignInt(array1, i, resultArray, pos)
+ } else {
+ assignLong(array1, i, resultArray, pos)
+ }
+ if (assigned) {
+ pos += 1
+ }
+ }
+ i += 1
+ }
+ pos
+ }
+
+ @transient lazy val evalExcept: (ArrayData, ArrayData) => ArrayData = {
+ if (elementTypeSupportEquals) {
+ elementType match {
+ case IntegerType =>
--- End diff --
Ah, if we usually use the generated code, we will optimize them only in the generated code. In the interpreted path, we leave only the generic case as a backup.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21103: [SPARK-23915][SQL] Add array_except function
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21103
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/1417/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #21103: [SPARK-23915][SQL] Add array_except function
Posted by kiszk <gi...@git.apache.org>.
Github user kiszk commented on a diff in the pull request:
https://github.com/apache/spark/pull/21103#discussion_r205685897
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/collectionOperations.scala ---
@@ -3805,3 +3799,233 @@ object ArrayUnion {
new GenericArrayData(arrayBuffer)
}
}
+
+/**
+ * Returns an array of the elements in the intersect of x and y, without duplicates
+ */
+@ExpressionDescription(
+ usage = """
+ _FUNC_(array1, array2) - Returns an array of the elements in array1 but not in array2,
+ without duplicates.
+ """,
+ examples = """
+ Examples:
+ > SELECT _FUNC_(array(1, 2, 3), array(1, 3, 5));
+ array(2)
+ """,
+ since = "2.4.0")
+case class ArrayExcept(left: Expression, right: Expression) extends ArraySetLike {
--- End diff --
WDYT?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #21103: [SPARK-23915][SQL] Add array_except function
Posted by cloud-fan <gi...@git.apache.org>.
Github user cloud-fan commented on a diff in the pull request:
https://github.com/apache/spark/pull/21103#discussion_r205928882
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/collectionOperations.scala ---
@@ -3651,14 +3651,9 @@ case class ArrayDistinct(child: Expression)
}
/**
- * Will become common base class for [[ArrayUnion]], ArrayIntersect, and ArrayExcept.
+ * Will become common base class for [[ArrayUnion]], ArrayIntersect, and [[ArrayExcept]].
*/
abstract class ArraySetLike extends BinaryArrayExpressionWithImplicitCast {
--- End diff --
I think the question is, shall we apply type coercion rules to these array functions?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21103: [SPARK-23915][SQL] Add array_except function
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21103
Merged build finished. Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21103: [SPARK-23915][SQL] Add array_except function
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21103
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21103: [SPARK-23915][SQL] Add array_except function
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21103
**[Test build #93682 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/93682/testReport)** for PR 21103 at commit [`548f336`](https://github.com/apache/spark/commit/548f336f9cd84cbc5da276335e2c335d8ba64ab7).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21103: [SPARK-23915][SQL] Add array_except function
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21103
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #21103: [SPARK-23915][SQL] Add array_except function
Posted by mn-mikke <gi...@git.apache.org>.
Github user mn-mikke commented on a diff in the pull request:
https://github.com/apache/spark/pull/21103#discussion_r205397226
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/collectionOperations.scala ---
@@ -3805,3 +3799,330 @@ object ArrayUnion {
new GenericArrayData(arrayBuffer)
}
}
+
+/**
+ * Returns an array of the elements in the intersect of x and y, without duplicates
+ */
+@ExpressionDescription(
+ usage = """
+ _FUNC_(array1, array2) - Returns an array of the elements in array1 but not in array2,
+ without duplicates.
+ """,
+ examples = """
+ Examples:
+ > SELECT _FUNC_(array(1, 2, 3), array(1, 3, 5));
+ array(2)
+ """,
+ since = "2.4.0")
+case class ArrayExcept(left: Expression, right: Expression) extends ArraySetLike {
+ override def dataType: DataType = left.dataType
+
+ var hsInt: OpenHashSet[Int] = _
+ var hsLong: OpenHashSet[Long] = _
+
+ def assignInt(array: ArrayData, idx: Int, resultArray: ArrayData, pos: Int): Boolean = {
+ val elem = array.getInt(idx)
+ if (!hsInt.contains(elem)) {
+ if (resultArray != null) {
+ resultArray.setInt(pos, elem)
+ }
+ hsInt.add(elem)
+ true
+ } else {
+ false
+ }
+ }
+
+ def assignLong(array: ArrayData, idx: Int, resultArray: ArrayData, pos: Int): Boolean = {
+ val elem = array.getLong(idx)
+ if (!hsLong.contains(elem)) {
+ if (resultArray != null) {
+ resultArray.setLong(pos, elem)
+ }
+ hsLong.add(elem)
+ true
+ } else {
+ false
+ }
+ }
+
+ def evalIntLongPrimitiveType(
+ array1: ArrayData,
+ array2: ArrayData,
+ resultArray: ArrayData,
+ isLongType: Boolean): Int = {
+ // store elements into resultArray
+ var notFoundNullElement = true
+ var i = 0
+ while (i < array2.numElements()) {
+ if (array2.isNullAt(i)) {
+ notFoundNullElement = false
+ } else {
+ val assigned = if (!isLongType) {
+ hsInt.add(array2.getInt(i))
+ } else {
+ hsLong.add(array2.getLong(i))
+ }
+ }
+ i += 1
+ }
+ var pos = 0
+ i = 0
+ while (i < array1.numElements()) {
+ if (array1.isNullAt(i)) {
+ if (notFoundNullElement) {
+ if (resultArray != null) {
+ resultArray.setNullAt(pos)
+ }
+ pos += 1
+ notFoundNullElement = false
+ }
+ } else {
+ val assigned = if (!isLongType) {
+ assignInt(array1, i, resultArray, pos)
+ } else {
+ assignLong(array1, i, resultArray, pos)
+ }
+ if (assigned) {
+ pos += 1
+ }
+ }
+ i += 1
+ }
+ pos
+ }
+
+ override def nullSafeEval(input1: Any, input2: Any): Any = {
+ val array1 = input1.asInstanceOf[ArrayData]
+ val array2 = input2.asInstanceOf[ArrayData]
+
+ if (elementTypeSupportEquals) {
+ elementType match {
+ case IntegerType =>
+ // avoid boxing of primitive int array elements
+ // calculate result array size
+ hsInt = new OpenHashSet[Int]
+ val elements = evalIntLongPrimitiveType(array1, array2, null, false)
+ // allocate result array
+ hsInt = new OpenHashSet[Int]
+ val resultArray = if (UnsafeArrayData.shouldUseGenericArrayData(
+ IntegerType.defaultSize, elements)) {
+ new GenericArrayData(new Array[Any](elements))
+ } else {
+ UnsafeArrayData.forPrimitiveArray(
+ Platform.INT_ARRAY_OFFSET, elements, IntegerType.defaultSize)
+ }
+ // assign elements into the result array
+ evalIntLongPrimitiveType(array1, array2, resultArray, false)
+ resultArray
+ case LongType =>
+ // avoid boxing of primitive long array elements
+ // calculate result array size
+ hsLong = new OpenHashSet[Long]
+ val elements = evalIntLongPrimitiveType(array1, array2, null, true)
+ // allocate result array
+ hsLong = new OpenHashSet[Long]
+ val resultArray = if (UnsafeArrayData.shouldUseGenericArrayData(
+ LongType.defaultSize, elements)) {
+ new GenericArrayData(new Array[Any](elements))
+ } else {
+ UnsafeArrayData.forPrimitiveArray(
+ Platform.LONG_ARRAY_OFFSET, elements, LongType.defaultSize)
+ }
+ // assign elements into the result array
+ evalIntLongPrimitiveType(array1, array2, resultArray, true)
+ resultArray
+ case _ =>
+ val hs = new OpenHashSet[Any]
+ var notFoundNullElement = true
+ var i = 0
+ while (i < array2.numElements()) {
+ if (array2.isNullAt(i)) {
+ notFoundNullElement = false
+ } else {
+ val elem = array2.get(i, elementType)
+ hs.add(elem)
+ }
+ i += 1
+ }
+ val arrayBuffer = new scala.collection.mutable.ArrayBuffer[Any]
+ i = 0
+ while (i < array1.numElements()) {
+ if (array1.isNullAt(i)) {
+ if (notFoundNullElement) {
+ arrayBuffer += null
+ notFoundNullElement = false
+ }
+ } else {
+ val elem = array1.get(i, elementType)
+ if (!hs.contains(elem)) {
+ arrayBuffer += elem
+ hs.add(elem)
+ }
+ }
+ i += 1
+ }
+ new GenericArrayData(arrayBuffer)
+ }
+ } else {
+ ArrayExcept.exceptOrdering(array1, array2, elementType, ordering)
+ }
+ }
+
+ override def doGenCode(ctx: CodegenContext, ev: ExprCode): ExprCode = {
+ val i = ctx.freshName("i")
+ val pos = ctx.freshName("pos")
+ val value = ctx.freshName("value")
+ val size = ctx.freshName("size")
+ val (postFix, openHashElementType, getter, setter, javaTypeName, castOp, arrayBuilder) =
+ if (elementTypeSupportEquals) {
+ elementType match {
+ case ByteType | ShortType | IntegerType | LongType =>
+ val ptName = CodeGenerator.primitiveTypeName(elementType)
+ val unsafeArray = ctx.freshName("unsafeArray")
+ (if (elementType == LongType) s"$$mcJ$$sp" else s"$$mcI$$sp",
+ if (elementType == LongType) "Long" else "Int",
+ s"get$ptName($i)", s"set$ptName($pos, $value)", CodeGenerator.javaType(elementType),
+ if (elementType == LongType) "(long)" else "(int)",
+ s"""
+ |${ctx.createUnsafeArray(unsafeArray, size, elementType, s" $prettyName failed.")}
+ |${ev.value} = $unsafeArray;
+ """.stripMargin)
+ case _ =>
+ val genericArrayData = classOf[GenericArrayData].getName
+ val et = ctx.addReferenceObj("elementType", elementType)
+ ("", "Object",
+ s"get($i, $et)", s"update($pos, $value)", "Object", "",
+ s"${ev.value} = new $genericArrayData(new Object[$size]);")
+ }
+ } else {
+ ("", "", "", "", "", "", "")
+ }
+
+ nullSafeCodeGen(ctx, ev, (array1, array2) => {
+ if (openHashElementType != "") {
+ // Here, we ensure elementTypeSupportEquals is true
+ val notFoundNullElement = ctx.freshName("notFoundNullElement")
+ val openHashSet = classOf[OpenHashSet[_]].getName
+ val classTag = s"scala.reflect.ClassTag$$.MODULE$$.$openHashElementType()"
+ val hs = ctx.freshName("hs")
+ val arrayData = classOf[ArrayData].getName
+ val arrays = ctx.freshName("arrays")
+ val array = ctx.freshName("array")
+ val arrayDataIdx = ctx.freshName("arrayDataIdx")
+ s"""
+ |$openHashSet $hs = new $openHashSet$postFix($classTag);
+ |boolean $notFoundNullElement = true;
+ |int $size = 0;
+ |for (int $i = 0; $i < $array2.numElements(); $i++) {
+ | if ($array2.isNullAt($i)) {
+ | $notFoundNullElement = false;
+ | } else {
+ | $hs.add$postFix($array2.$getter);
+ | }
+ |}
+ |for (int $i = 0; $i < $array1.numElements(); $i++) {
+ | if ($array1.isNullAt($i)) {
+ | if ($notFoundNullElement) {
+ | $size++;
+ | $notFoundNullElement = false;
+ | }
+ | } else {
+ | $javaTypeName $value = $array1.$getter;
+ | if (!$hs.contains($castOp $value)) {
+ | $hs.add$postFix($value);
+ | $size++;
+ | }
+ | }
+ |}
+ |$arrayBuilder
+ |$hs = new $openHashSet$postFix($classTag);
+ |$notFoundNullElement = true;
+ |int $pos = 0;
+ |for (int $i = 0; $i < $array2.numElements(); $i++) {
+ | if ($array2.isNullAt($i)) {
+ | $notFoundNullElement = false;
+ | } else {
+ | $hs.add$postFix($array2.$getter);
+ | }
+ |}
+ |for (int $i = 0; $i < $array1.numElements(); $i++) {
+ | if ($array1.isNullAt($i)) {
+ | if ($notFoundNullElement) {
+ | ${ev.value}.setNullAt($pos++);
+ | $notFoundNullElement = false;
+ | }
+ | } else {
+ | $javaTypeName $value = $array1.$getter;
+ | if (!$hs.contains($castOp $value)) {
+ | $hs.add$postFix($value);
+ | ${ev.value}.$setter;
+ | $pos++;
+ | }
+ | }
+ |}
+ """.stripMargin
+ } else {
+ val arrayExcept = classOf[ArrayExcept].getName
+ val et = ctx.addReferenceObj("elementTypeIntersect", elementType)
+ val order = ctx.addReferenceObj("orderingIntersect", ordering)
+ val method = "exceptOrdering"
+ s"${ev.value} = $arrayExcept$$.MODULE$$.$method($array1, $array2, $et, $order);"
+ }
+ })
+ }
+
+ override def prettyName: String = "array_except"
+}
+
+object ArrayExcept {
+ def exceptOrdering(
+ array1: ArrayData,
+ array2: ArrayData,
+ elementType: DataType,
+ ordering: Ordering[Any]): ArrayData = {
+ val arrayBuffer = new scala.collection.mutable.ArrayBuffer[Any]
+ var scannedNullElements = false
+ var i = 0
+ while (i < array1.numElements()) {
+ var found = false
+ var elem1 = array1.get(i, elementType)
+ if (array1.isNullAt(i)) {
+ if (!scannedNullElements) {
+ var j = 0
+ while (!found && j < array2.numElements()) {
+ found = array2.isNullAt(j)
+ j += 1
+ }
+ // array2 is scanned only once for null element
+ scannedNullElements = true
+ } else {
+ found = true
+ }
+ } else {
+ var j = 0
+ while (!found && j < array2.numElements()) {
+ if (!array2.isNullAt(j)) {
--- End diff --
nit: Couldn't we shorten the null check as on the line [4116](https://github.com/apache/spark/pull/21103/files#diff-9853dcf5ce3d2ac1e94d473197ff5768R4116)?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21103: [SPARK-23915][SQL] Add array_except function
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21103
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/93822/
Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21103: [SPARK-23915][SQL] Add array_except function
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21103
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/89561/
Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21103: [SPARK-23915][SQL] Add array_except function
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21103
Build finished. Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21103: [SPARK-23915][SQL] Add array_except function
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21103
**[Test build #93683 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/93683/testReport)** for PR 21103 at commit [`f4d4f8c`](https://github.com/apache/spark/commit/f4d4f8c89ce82ed058f6696885363f0e00154eaf).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21103: [SPARK-23915][SQL] Add array_except function
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21103
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21103: [SPARK-23915][SQL] Add array_except function
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21103
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/2446/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #21103: [SPARK-23915][SQL] Add array_except function
Posted by kiszk <gi...@git.apache.org>.
Github user kiszk commented on a diff in the pull request:
https://github.com/apache/spark/pull/21103#discussion_r206008046
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/collectionOperations.scala ---
@@ -3968,3 +3964,267 @@ object ArrayUnion {
new GenericArrayData(arrayBuffer)
}
}
+
+/**
+ * Returns an array of the elements in the intersect of x and y, without duplicates
+ */
+@ExpressionDescription(
+ usage = """
+ _FUNC_(array1, array2) - Returns an array of the elements in array1 but not in array2,
+ without duplicates.
+ """,
+ examples = """
+ Examples:
+ > SELECT _FUNC_(array(1, 2, 3), array(1, 3, 5));
+ array(2)
+ """,
+ since = "2.4.0")
+case class ArrayExcept(left: Expression, right: Expression) extends ArraySetLike
+ with ComplexTypeMergingExpression {
+ override def dataType: DataType = {
+ dataTypeCheck
+ left.dataType
+ }
+
+ @transient lazy val evalExcept: (ArrayData, ArrayData) => ArrayData = {
+ if (elementTypeSupportEquals) {
+ (array1, array2) =>
+ val hs = new OpenHashSet[Any]
+ var notFoundNullElement = true
+ var i = 0
+ while (i < array2.numElements()) {
+ if (array2.isNullAt(i)) {
+ notFoundNullElement = false
+ } else {
+ val elem = array2.get(i, elementType)
+ hs.add(elem)
+ }
+ i += 1
+ }
+ val arrayBuffer = new scala.collection.mutable.ArrayBuffer[Any]
+ i = 0
+ while (i < array1.numElements()) {
+ if (array1.isNullAt(i)) {
+ if (notFoundNullElement) {
+ arrayBuffer += null
+ notFoundNullElement = false
+ }
+ } else {
+ val elem = array1.get(i, elementType)
+ if (!hs.contains(elem)) {
+ arrayBuffer += elem
+ hs.add(elem)
+ }
+ }
+ i += 1
+ }
+ new GenericArrayData(arrayBuffer)
+ } else {
+ (array1, array2) =>
+ val arrayBuffer = new scala.collection.mutable.ArrayBuffer[Any]
+ var scannedNullElements = false
+ var i = 0
+ while (i < array1.numElements()) {
+ var found = false
+ val elem1 = array1.get(i, elementType)
+ if (elem1 == null) {
+ if (!scannedNullElements) {
+ var j = 0
+ while (!found && j < array2.numElements()) {
+ found = array2.isNullAt(j)
+ j += 1
+ }
+ // array2 is scanned only once for null element
+ scannedNullElements = true
+ } else {
+ found = true
+ }
+ } else {
+ var j = 0
+ while (!found && j < array2.numElements()) {
+ val elem2 = array2.get(j, elementType)
+ if (elem2 != null) {
+ found = ordering.equiv(elem1, elem2)
+ }
+ j += 1
+ }
+ if (!found) {
+ // check whether elem1 is already stored in arrayBuffer
+ var k = 0
+ while (!found && k < arrayBuffer.size) {
+ val va = arrayBuffer(k)
+ found = (va != null) && ordering.equiv(va, elem1)
+ k += 1
+ }
+ }
+ }
+ if (!found) {
+ arrayBuffer += elem1
+ }
+ i += 1
+ }
+ new GenericArrayData(arrayBuffer)
+ }
+ }
+
+ override def nullSafeEval(input1: Any, input2: Any): Any = {
+ val array1 = input1.asInstanceOf[ArrayData]
+ val array2 = input2.asInstanceOf[ArrayData]
+
+ evalExcept(array1, array2)
+ }
+
+ override def doGenCode(ctx: CodegenContext, ev: ExprCode): ExprCode = {
+ val arrayData = classOf[ArrayData].getName
+ val i = ctx.freshName("i")
+ val pos = ctx.freshName("pos")
+ val value = ctx.freshName("value")
+ val hsValue = ctx.freshName("hsValue")
+ val size = ctx.freshName("size")
+ val (postFix, openHashElementType, hsJavaTypeName, genHsValue,
+ getter, setter, javaTypeName, arrayBuilder) =
+ if (elementTypeSupportEquals) {
+ elementType match {
+ case BooleanType | ByteType | ShortType | IntegerType =>
+ val ptName = CodeGenerator.primitiveTypeName(elementType)
+ val unsafeArray = ctx.freshName("unsafeArray")
+ ("$mcI$sp", "Int", "int",
+ if (elementType != BooleanType) {
+ s"(int) $value"
+ } else {
+ s"$value ? 1 : 0;"
+ },
+ s"get$ptName($i)", s"set$ptName($pos, $value)", CodeGenerator.javaType(elementType),
+ s"""
+ |${ctx.createUnsafeArray(unsafeArray, size, elementType, s" $prettyName failed.")}
+ |${ev.value} = $unsafeArray;
+ """.stripMargin)
+ case LongType | FloatType | DoubleType =>
+ val ptName = CodeGenerator.primitiveTypeName(elementType)
+ val unsafeArray = ctx.freshName("unsafeArray")
+ val signature = elementType match {
+ case LongType => "$mcJ$sp"
+ case FloatType => "$mcF$sp"
+ case DoubleType => "$mcD$sp"
+ }
+ (signature, CodeGenerator.boxedType(elementType),
+ CodeGenerator.javaType(elementType), value,
+ s"get$ptName($i)", s"set$ptName($pos, $value)", CodeGenerator.javaType(elementType),
+ s"""
+ |${ctx.createUnsafeArray(unsafeArray, size, elementType, s" $prettyName failed.")}
+ |${ev.value} = $unsafeArray;
+ """.stripMargin)
+ case _ =>
+ val genericArrayData = classOf[GenericArrayData].getName
+ val et = ctx.addReferenceObj("elementType", elementType)
+ ("", "Object", "Object", value,
+ s"get($i, $et)", s"update($pos, $value)", "Object",
+ s"${ev.value} = new $genericArrayData(new Object[$size]);")
+ }
+ } else {
+ ("", "", "", "", "", "", "", "")
+ }
+
+ nullSafeCodeGen(ctx, ev, (array1, array2) => {
+ if (openHashElementType != "") {
+ // Here, we ensure elementTypeSupportEquals is true
+ val notFoundNullElement = ctx.freshName("notFoundNullElement")
+ val openHashSet = classOf[OpenHashSet[_]].getName
+ val classTag = s"scala.reflect.ClassTag$$.MODULE$$.$openHashElementType()"
+ val hs = ctx.freshName("hs")
+ val arrayData = classOf[ArrayData].getName
+ val arrays = ctx.freshName("arrays")
+ val array = ctx.freshName("array")
+ val arrayDataIdx = ctx.freshName("arrayDataIdx")
+
+ val array2NullCheck = if (right.dataType.asInstanceOf[ArrayType].containsNull) {
+ s"""
+ |if ($array2.isNullAt($i)) {
+ | $notFoundNullElement = false;
+ |} else
+ """.stripMargin
+ } else {
+ ""
+ }
+ val array1NullCheck = if (left.dataType.asInstanceOf[ArrayType].containsNull) {
+ s"""
+ |if ($array1.isNullAt($i)) {
+ | if ($notFoundNullElement) {
+ | $size++;
+ | $notFoundNullElement = false;
+ | }
+ |} else
+ """.stripMargin
+ } else {
+ ""
+ }
+ val array1NullAssignment = if (left.dataType.asInstanceOf[ArrayType].containsNull) {
+ s"""
+ |if ($array1.isNullAt($i)) {
+ | if ($notFoundNullElement) {
+ | ${ev.value}.setNullAt($pos++);
+ | $notFoundNullElement = false;
+ | }
+ |} else
+ """.stripMargin
+ } else {
+ ""
+ }
+
+ s"""
+ |$openHashSet $hs = new $openHashSet$postFix($classTag);
+ |boolean $notFoundNullElement = true;
+ |int $size = 0;
+ |for (int $i = 0; $i < $array2.numElements(); $i++) {
+ | $array2NullCheck
+ | {
+ | $javaTypeName $value = $array2.$getter;
+ | $hsJavaTypeName $hsValue = $genHsValue;
+ | $hs.add$postFix($hsValue);
+ | }
+ |}
+ |for (int $i = 0; $i < $array1.numElements(); $i++) {
+ | $array1NullCheck
+ | {
+ | $javaTypeName $value = $array1.$getter;
+ | $hsJavaTypeName $hsValue = $genHsValue;
+ | if (!$hs.contains($hsValue)) {
+ | $hs.add$postFix($hsValue);
+ | $size++;
+ | }
+ | }
+ |}
+ |$arrayBuilder
+ |$hs = new $openHashSet$postFix($classTag);
+ |$notFoundNullElement = true;
+ |int $pos = 0;
+ |for (int $i = 0; $i < $array2.numElements(); $i++) {
--- End diff --
To use 'ArrayBuffer' involves boxing. Let me consider 'ArrayBuffer.ofInt' or others with special null handling.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21103: [SPARK-23915][SQL] Add array_except function
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21103
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/2509/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #21103: [SPARK-23915][SQL] Add array_except function
Posted by cloud-fan <gi...@git.apache.org>.
Github user cloud-fan commented on a diff in the pull request:
https://github.com/apache/spark/pull/21103#discussion_r206004228
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/collectionOperations.scala ---
@@ -3968,3 +3964,267 @@ object ArrayUnion {
new GenericArrayData(arrayBuffer)
}
}
+
+/**
+ * Returns an array of the elements in the intersect of x and y, without duplicates
+ */
+@ExpressionDescription(
+ usage = """
+ _FUNC_(array1, array2) - Returns an array of the elements in array1 but not in array2,
+ without duplicates.
+ """,
+ examples = """
+ Examples:
+ > SELECT _FUNC_(array(1, 2, 3), array(1, 3, 5));
+ array(2)
+ """,
+ since = "2.4.0")
+case class ArrayExcept(left: Expression, right: Expression) extends ArraySetLike
+ with ComplexTypeMergingExpression {
+ override def dataType: DataType = {
+ dataTypeCheck
+ left.dataType
+ }
+
+ @transient lazy val evalExcept: (ArrayData, ArrayData) => ArrayData = {
+ if (elementTypeSupportEquals) {
+ (array1, array2) =>
+ val hs = new OpenHashSet[Any]
+ var notFoundNullElement = true
+ var i = 0
+ while (i < array2.numElements()) {
+ if (array2.isNullAt(i)) {
+ notFoundNullElement = false
+ } else {
+ val elem = array2.get(i, elementType)
+ hs.add(elem)
+ }
+ i += 1
+ }
+ val arrayBuffer = new scala.collection.mutable.ArrayBuffer[Any]
+ i = 0
+ while (i < array1.numElements()) {
+ if (array1.isNullAt(i)) {
+ if (notFoundNullElement) {
+ arrayBuffer += null
+ notFoundNullElement = false
+ }
+ } else {
+ val elem = array1.get(i, elementType)
+ if (!hs.contains(elem)) {
+ arrayBuffer += elem
+ hs.add(elem)
+ }
+ }
+ i += 1
+ }
+ new GenericArrayData(arrayBuffer)
+ } else {
+ (array1, array2) =>
+ val arrayBuffer = new scala.collection.mutable.ArrayBuffer[Any]
+ var scannedNullElements = false
+ var i = 0
+ while (i < array1.numElements()) {
+ var found = false
+ val elem1 = array1.get(i, elementType)
+ if (elem1 == null) {
+ if (!scannedNullElements) {
+ var j = 0
+ while (!found && j < array2.numElements()) {
+ found = array2.isNullAt(j)
+ j += 1
+ }
+ // array2 is scanned only once for null element
+ scannedNullElements = true
+ } else {
+ found = true
+ }
+ } else {
+ var j = 0
+ while (!found && j < array2.numElements()) {
+ val elem2 = array2.get(j, elementType)
+ if (elem2 != null) {
+ found = ordering.equiv(elem1, elem2)
+ }
+ j += 1
+ }
+ if (!found) {
+ // check whether elem1 is already stored in arrayBuffer
+ var k = 0
+ while (!found && k < arrayBuffer.size) {
+ val va = arrayBuffer(k)
+ found = (va != null) && ordering.equiv(va, elem1)
+ k += 1
+ }
+ }
+ }
+ if (!found) {
+ arrayBuffer += elem1
+ }
+ i += 1
+ }
+ new GenericArrayData(arrayBuffer)
+ }
+ }
+
+ override def nullSafeEval(input1: Any, input2: Any): Any = {
+ val array1 = input1.asInstanceOf[ArrayData]
+ val array2 = input2.asInstanceOf[ArrayData]
+
+ evalExcept(array1, array2)
+ }
+
+ override def doGenCode(ctx: CodegenContext, ev: ExprCode): ExprCode = {
+ val arrayData = classOf[ArrayData].getName
+ val i = ctx.freshName("i")
+ val pos = ctx.freshName("pos")
+ val value = ctx.freshName("value")
+ val hsValue = ctx.freshName("hsValue")
+ val size = ctx.freshName("size")
+ val (postFix, openHashElementType, hsJavaTypeName, genHsValue,
+ getter, setter, javaTypeName, arrayBuilder) =
+ if (elementTypeSupportEquals) {
+ elementType match {
+ case BooleanType | ByteType | ShortType | IntegerType =>
+ val ptName = CodeGenerator.primitiveTypeName(elementType)
+ val unsafeArray = ctx.freshName("unsafeArray")
+ ("$mcI$sp", "Int", "int",
+ if (elementType != BooleanType) {
+ s"(int) $value"
+ } else {
+ s"$value ? 1 : 0;"
+ },
+ s"get$ptName($i)", s"set$ptName($pos, $value)", CodeGenerator.javaType(elementType),
+ s"""
+ |${ctx.createUnsafeArray(unsafeArray, size, elementType, s" $prettyName failed.")}
+ |${ev.value} = $unsafeArray;
+ """.stripMargin)
+ case LongType | FloatType | DoubleType =>
+ val ptName = CodeGenerator.primitiveTypeName(elementType)
+ val unsafeArray = ctx.freshName("unsafeArray")
+ val signature = elementType match {
+ case LongType => "$mcJ$sp"
+ case FloatType => "$mcF$sp"
+ case DoubleType => "$mcD$sp"
+ }
+ (signature, CodeGenerator.boxedType(elementType),
+ CodeGenerator.javaType(elementType), value,
+ s"get$ptName($i)", s"set$ptName($pos, $value)", CodeGenerator.javaType(elementType),
+ s"""
+ |${ctx.createUnsafeArray(unsafeArray, size, elementType, s" $prettyName failed.")}
+ |${ev.value} = $unsafeArray;
+ """.stripMargin)
+ case _ =>
+ val genericArrayData = classOf[GenericArrayData].getName
+ val et = ctx.addReferenceObj("elementType", elementType)
+ ("", "Object", "Object", value,
+ s"get($i, $et)", s"update($pos, $value)", "Object",
+ s"${ev.value} = new $genericArrayData(new Object[$size]);")
+ }
+ } else {
+ ("", "", "", "", "", "", "", "")
+ }
+
+ nullSafeCodeGen(ctx, ev, (array1, array2) => {
+ if (openHashElementType != "") {
+ // Here, we ensure elementTypeSupportEquals is true
+ val notFoundNullElement = ctx.freshName("notFoundNullElement")
+ val openHashSet = classOf[OpenHashSet[_]].getName
+ val classTag = s"scala.reflect.ClassTag$$.MODULE$$.$openHashElementType()"
+ val hs = ctx.freshName("hs")
+ val arrayData = classOf[ArrayData].getName
+ val arrays = ctx.freshName("arrays")
+ val array = ctx.freshName("array")
+ val arrayDataIdx = ctx.freshName("arrayDataIdx")
+
+ val array2NullCheck = if (right.dataType.asInstanceOf[ArrayType].containsNull) {
+ s"""
+ |if ($array2.isNullAt($i)) {
+ | $notFoundNullElement = false;
+ |} else
+ """.stripMargin
+ } else {
+ ""
+ }
+ val array1NullCheck = if (left.dataType.asInstanceOf[ArrayType].containsNull) {
+ s"""
+ |if ($array1.isNullAt($i)) {
+ | if ($notFoundNullElement) {
+ | $size++;
+ | $notFoundNullElement = false;
+ | }
+ |} else
+ """.stripMargin
+ } else {
+ ""
+ }
+ val array1NullAssignment = if (left.dataType.asInstanceOf[ArrayType].containsNull) {
+ s"""
+ |if ($array1.isNullAt($i)) {
+ | if ($notFoundNullElement) {
+ | ${ev.value}.setNullAt($pos++);
+ | $notFoundNullElement = false;
+ | }
+ |} else
+ """.stripMargin
+ } else {
+ ""
+ }
+
+ s"""
+ |$openHashSet $hs = new $openHashSet$postFix($classTag);
+ |boolean $notFoundNullElement = true;
+ |int $size = 0;
+ |for (int $i = 0; $i < $array2.numElements(); $i++) {
+ | $array2NullCheck
+ | {
+ | $javaTypeName $value = $array2.$getter;
+ | $hsJavaTypeName $hsValue = $genHsValue;
+ | $hs.add$postFix($hsValue);
+ | }
+ |}
+ |for (int $i = 0; $i < $array1.numElements(); $i++) {
+ | $array1NullCheck
+ | {
+ | $javaTypeName $value = $array1.$getter;
+ | $hsJavaTypeName $hsValue = $genHsValue;
+ | if (!$hs.contains($hsValue)) {
+ | $hs.add$postFix($hsValue);
+ | $size++;
+ | }
+ | }
+ |}
+ |$arrayBuilder
+ |$hs = new $openHashSet$postFix($classTag);
+ |$notFoundNullElement = true;
+ |int $pos = 0;
+ |for (int $i = 0; $i < $array2.numElements(); $i++) {
--- End diff --
is this worth? I feel using an `ArrayBuffer` like the interpreted path is faster.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21103: [SPARK-23915][SQL] Add array_except function
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21103
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/93825/
Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21103: [SPARK-23915][SQL] Add array_except function
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21103
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/93671/
Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21103: [SPARK-23915][SQL] Add array_except function
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21103
**[Test build #93001 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/93001/testReport)** for PR 21103 at commit [`5c06922`](https://github.com/apache/spark/commit/5c06922a9492dcb0032e1b60418e25fb57d6fdda).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21103: [SPARK-23915][SQL] Add array_except function
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21103
**[Test build #93654 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/93654/testReport)** for PR 21103 at commit [`e902974`](https://github.com/apache/spark/commit/e9029746a9cbc204d043cb7a0f9c1c3285284b54).
* This patch **fails due to an unknown error code, -9**.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #21103: [SPARK-23915][SQL] Add array_except function
Posted by kiszk <gi...@git.apache.org>.
Github user kiszk commented on a diff in the pull request:
https://github.com/apache/spark/pull/21103#discussion_r204613831
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/collectionOperations.scala ---
@@ -3805,3 +3799,331 @@ object ArrayUnion {
new GenericArrayData(arrayBuffer)
}
}
+
+/**
+ * Returns an array of the elements in the intersect of x and y, without duplicates
+ */
+@ExpressionDescription(
+ usage = """
+ _FUNC_(array1, array2) - Returns an array of the elements in array1 but not in array2,
+ without duplicates.
+ """,
+ examples = """
+ Examples:
+ > SELECT _FUNC_(array(1, 2, 3), array(1, 3, 5));
+ array(2)
+ """,
+ since = "2.4.0")
+case class ArrayExcept(left: Expression, right: Expression) extends ArraySetLike {
+ override def dataType: DataType =
+ ArrayType(elementType, left.dataType.asInstanceOf[ArrayType].containsNull)
+
+ var hsInt: OpenHashSet[Int] = _
+ var hsLong: OpenHashSet[Long] = _
+
+ def assignInt(array: ArrayData, idx: Int, resultArray: ArrayData, pos: Int): Boolean = {
+ val elem = array.getInt(idx)
+ if (!hsInt.contains(elem)) {
+ if (resultArray != null) {
+ resultArray.setInt(pos, elem)
+ }
+ hsInt.add(elem)
+ true
+ } else {
+ false
+ }
+ }
+
+ def assignLong(array: ArrayData, idx: Int, resultArray: ArrayData, pos: Int): Boolean = {
+ val elem = array.getLong(idx)
+ if (!hsLong.contains(elem)) {
+ if (resultArray != null) {
+ resultArray.setLong(pos, elem)
+ }
+ hsLong.add(elem)
+ true
+ } else {
+ false
+ }
+ }
+
+ def evalIntLongPrimitiveType(
+ array1: ArrayData,
+ array2: ArrayData,
+ resultArray: ArrayData,
+ isLongType: Boolean): Int = {
+ // store elements into resultArray
+ var exceptNullElement = true
--- End diff --
Ah, I mean `nullElement` for Array`except`. I will rename this to a better name.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #21103: [SPARK-23915][SQL] Add array_except function
Posted by cloud-fan <gi...@git.apache.org>.
Github user cloud-fan commented on a diff in the pull request:
https://github.com/apache/spark/pull/21103#discussion_r205999925
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/collectionOperations.scala ---
@@ -3968,3 +3964,267 @@ object ArrayUnion {
new GenericArrayData(arrayBuffer)
}
}
+
+/**
+ * Returns an array of the elements in the intersect of x and y, without duplicates
+ */
+@ExpressionDescription(
+ usage = """
+ _FUNC_(array1, array2) - Returns an array of the elements in array1 but not in array2,
+ without duplicates.
+ """,
+ examples = """
+ Examples:
+ > SELECT _FUNC_(array(1, 2, 3), array(1, 3, 5));
+ array(2)
+ """,
+ since = "2.4.0")
+case class ArrayExcept(left: Expression, right: Expression) extends ArraySetLike
+ with ComplexTypeMergingExpression {
+ override def dataType: DataType = {
+ dataTypeCheck
+ left.dataType
+ }
+
+ @transient lazy val evalExcept: (ArrayData, ArrayData) => ArrayData = {
+ if (elementTypeSupportEquals) {
+ (array1, array2) =>
+ val hs = new OpenHashSet[Any]
+ var notFoundNullElement = true
+ var i = 0
+ while (i < array2.numElements()) {
+ if (array2.isNullAt(i)) {
+ notFoundNullElement = false
+ } else {
+ val elem = array2.get(i, elementType)
+ hs.add(elem)
+ }
+ i += 1
+ }
+ val arrayBuffer = new scala.collection.mutable.ArrayBuffer[Any]
+ i = 0
+ while (i < array1.numElements()) {
+ if (array1.isNullAt(i)) {
+ if (notFoundNullElement) {
+ arrayBuffer += null
+ notFoundNullElement = false
+ }
+ } else {
+ val elem = array1.get(i, elementType)
+ if (!hs.contains(elem)) {
+ arrayBuffer += elem
+ hs.add(elem)
+ }
+ }
+ i += 1
+ }
+ new GenericArrayData(arrayBuffer)
+ } else {
+ (array1, array2) =>
+ val arrayBuffer = new scala.collection.mutable.ArrayBuffer[Any]
+ var scannedNullElements = false
+ var i = 0
+ while (i < array1.numElements()) {
+ var found = false
+ val elem1 = array1.get(i, elementType)
+ if (elem1 == null) {
+ if (!scannedNullElements) {
+ var j = 0
+ while (!found && j < array2.numElements()) {
+ found = array2.isNullAt(j)
+ j += 1
+ }
+ // array2 is scanned only once for null element
+ scannedNullElements = true
+ } else {
+ found = true
+ }
+ } else {
+ var j = 0
+ while (!found && j < array2.numElements()) {
+ val elem2 = array2.get(j, elementType)
+ if (elem2 != null) {
+ found = ordering.equiv(elem1, elem2)
+ }
+ j += 1
+ }
+ if (!found) {
+ // check whether elem1 is already stored in arrayBuffer
+ var k = 0
+ while (!found && k < arrayBuffer.size) {
+ val va = arrayBuffer(k)
+ found = (va != null) && ordering.equiv(va, elem1)
+ k += 1
+ }
+ }
+ }
+ if (!found) {
+ arrayBuffer += elem1
+ }
+ i += 1
+ }
+ new GenericArrayData(arrayBuffer)
+ }
+ }
+
+ override def nullSafeEval(input1: Any, input2: Any): Any = {
+ val array1 = input1.asInstanceOf[ArrayData]
+ val array2 = input2.asInstanceOf[ArrayData]
+
+ evalExcept(array1, array2)
+ }
+
+ override def doGenCode(ctx: CodegenContext, ev: ExprCode): ExprCode = {
+ val arrayData = classOf[ArrayData].getName
+ val i = ctx.freshName("i")
+ val pos = ctx.freshName("pos")
+ val value = ctx.freshName("value")
+ val hsValue = ctx.freshName("hsValue")
+ val size = ctx.freshName("size")
+ val (postFix, openHashElementType, hsJavaTypeName, genHsValue,
+ getter, setter, javaTypeName, arrayBuilder) =
+ if (elementTypeSupportEquals) {
+ elementType match {
+ case BooleanType | ByteType | ShortType | IntegerType =>
+ val ptName = CodeGenerator.primitiveTypeName(elementType)
+ val unsafeArray = ctx.freshName("unsafeArray")
+ ("$mcI$sp", "Int", "int",
+ if (elementType != BooleanType) {
+ s"(int) $value"
+ } else {
+ s"$value ? 1 : 0;"
+ },
+ s"get$ptName($i)", s"set$ptName($pos, $value)", CodeGenerator.javaType(elementType),
+ s"""
+ |${ctx.createUnsafeArray(unsafeArray, size, elementType, s" $prettyName failed.")}
+ |${ev.value} = $unsafeArray;
+ """.stripMargin)
+ case LongType | FloatType | DoubleType =>
+ val ptName = CodeGenerator.primitiveTypeName(elementType)
+ val unsafeArray = ctx.freshName("unsafeArray")
+ val signature = elementType match {
+ case LongType => "$mcJ$sp"
+ case FloatType => "$mcF$sp"
+ case DoubleType => "$mcD$sp"
+ }
+ (signature, CodeGenerator.boxedType(elementType),
+ CodeGenerator.javaType(elementType), value,
+ s"get$ptName($i)", s"set$ptName($pos, $value)", CodeGenerator.javaType(elementType),
+ s"""
+ |${ctx.createUnsafeArray(unsafeArray, size, elementType, s" $prettyName failed.")}
+ |${ev.value} = $unsafeArray;
+ """.stripMargin)
+ case _ =>
+ val genericArrayData = classOf[GenericArrayData].getName
+ val et = ctx.addReferenceObj("elementType", elementType)
+ ("", "Object", "Object", value,
+ s"get($i, $et)", s"update($pos, $value)", "Object",
+ s"${ev.value} = new $genericArrayData(new Object[$size]);")
+ }
+ } else {
+ ("", "", "", "", "", "", "", "")
+ }
+
+ nullSafeCodeGen(ctx, ev, (array1, array2) => {
+ if (openHashElementType != "") {
+ // Here, we ensure elementTypeSupportEquals is true
+ val notFoundNullElement = ctx.freshName("notFoundNullElement")
+ val openHashSet = classOf[OpenHashSet[_]].getName
+ val classTag = s"scala.reflect.ClassTag$$.MODULE$$.$openHashElementType()"
+ val hs = ctx.freshName("hs")
+ val arrayData = classOf[ArrayData].getName
+ val arrays = ctx.freshName("arrays")
+ val array = ctx.freshName("array")
+ val arrayDataIdx = ctx.freshName("arrayDataIdx")
+
+ val array2NullCheck = if (right.dataType.asInstanceOf[ArrayType].containsNull) {
+ s"""
+ |if ($array2.isNullAt($i)) {
+ | $notFoundNullElement = false;
+ |} else
+ """.stripMargin
+ } else {
+ ""
+ }
+ val array1NullCheck = if (left.dataType.asInstanceOf[ArrayType].containsNull) {
+ s"""
+ |if ($array1.isNullAt($i)) {
+ | if ($notFoundNullElement) {
+ | $size++;
+ | $notFoundNullElement = false;
+ | }
+ |} else
+ """.stripMargin
+ } else {
+ ""
+ }
+ val array1NullAssignment = if (left.dataType.asInstanceOf[ArrayType].containsNull) {
+ s"""
+ |if ($array1.isNullAt($i)) {
+ | if ($notFoundNullElement) {
+ | ${ev.value}.setNullAt($pos++);
+ | $notFoundNullElement = false;
+ | }
+ |} else
+ """.stripMargin
+ } else {
+ ""
+ }
+
+ s"""
+ |$openHashSet $hs = new $openHashSet$postFix($classTag);
+ |boolean $notFoundNullElement = true;
+ |int $size = 0;
+ |for (int $i = 0; $i < $array2.numElements(); $i++) {
+ | $array2NullCheck
+ | {
+ | $javaTypeName $value = $array2.$getter;
+ | $hsJavaTypeName $hsValue = $genHsValue;
+ | $hs.add$postFix($hsValue);
+ | }
+ |}
+ |for (int $i = 0; $i < $array1.numElements(); $i++) {
+ | $array1NullCheck
+ | {
+ | $javaTypeName $value = $array1.$getter;
+ | $hsJavaTypeName $hsValue = $genHsValue;
+ | if (!$hs.contains($hsValue)) {
+ | $hs.add$postFix($hsValue);
+ | $size++;
+ | }
+ | }
+ |}
+ |$arrayBuilder
+ |$hs = new $openHashSet$postFix($classTag);
+ |$notFoundNullElement = true;
+ |int $pos = 0;
+ |for (int $i = 0; $i < $array2.numElements(); $i++) {
--- End diff --
why add `array2` to the hash set again?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21103: [SPARK-23915][SQL] Add array_except function
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21103
**[Test build #93669 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/93669/testReport)** for PR 21103 at commit [`926681c`](https://github.com/apache/spark/commit/926681c9d498268975a565ff1fd339afd1e965ed).
* This patch **fails Scala style tests**.
* This patch **does not merge cleanly**.
* This patch adds the following public classes _(experimental)_:
* `class OpenHashSet[@specialized(Long, Int, Double, Float) T: ClassTag](`
* `case class ArrayExcept(left: Expression, right: Expression) extends ArraySetLike`
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21103: [SPARK-23915][SQL] Add array_except function
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21103
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/1393/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21103: [SPARK-23915][SQL] Add array_except function
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21103
**[Test build #93674 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/93674/testReport)** for PR 21103 at commit [`d8e5d79`](https://github.com/apache/spark/commit/d8e5d798498be7dd976835499004b539cde09005).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21103: [SPARK-23915][SQL] Add array_except function
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21103
Merged build finished. Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #21103: [SPARK-23915][SQL] Add array_except function
Posted by ueshin <gi...@git.apache.org>.
Github user ueshin commented on a diff in the pull request:
https://github.com/apache/spark/pull/21103#discussion_r205649948
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/collectionOperations.scala ---
@@ -3805,3 +3799,230 @@ object ArrayUnion {
new GenericArrayData(arrayBuffer)
}
}
+
+/**
+ * Returns an array of the elements in the intersect of x and y, without duplicates
+ */
+@ExpressionDescription(
+ usage = """
+ _FUNC_(array1, array2) - Returns an array of the elements in array1 but not in array2,
+ without duplicates.
+ """,
+ examples = """
+ Examples:
+ > SELECT _FUNC_(array(1, 2, 3), array(1, 3, 5));
+ array(2)
+ """,
+ since = "2.4.0")
+case class ArrayExcept(left: Expression, right: Expression) extends ArraySetLike {
+ override def dataType: DataType = left.dataType
+
+ var hsInt: OpenHashSet[Int] = _
+ var hsLong: OpenHashSet[Long] = _
+
+ def assignInt(array: ArrayData, idx: Int, resultArray: ArrayData, pos: Int): Boolean = {
+ val elem = array.getInt(idx)
+ if (!hsInt.contains(elem)) {
+ if (resultArray != null) {
+ resultArray.setInt(pos, elem)
+ }
+ hsInt.add(elem)
+ true
+ } else {
+ false
+ }
+ }
+
+ def assignLong(array: ArrayData, idx: Int, resultArray: ArrayData, pos: Int): Boolean = {
+ val elem = array.getLong(idx)
+ if (!hsLong.contains(elem)) {
+ if (resultArray != null) {
+ resultArray.setLong(pos, elem)
+ }
+ hsLong.add(elem)
+ true
+ } else {
+ false
+ }
+ }
+
+ def evalIntLongPrimitiveType(
+ array1: ArrayData,
+ array2: ArrayData,
+ resultArray: ArrayData,
+ isLongType: Boolean): Int = {
+ // store elements into resultArray
+ var notFoundNullElement = true
+ var i = 0
+ while (i < array2.numElements()) {
+ if (array2.isNullAt(i)) {
+ notFoundNullElement = false
+ } else {
+ val assigned = if (!isLongType) {
+ hsInt.add(array2.getInt(i))
+ } else {
+ hsLong.add(array2.getLong(i))
+ }
+ }
+ i += 1
+ }
+ var pos = 0
+ i = 0
+ while (i < array1.numElements()) {
+ if (array1.isNullAt(i)) {
+ if (notFoundNullElement) {
+ if (resultArray != null) {
+ resultArray.setNullAt(pos)
+ }
+ pos += 1
+ notFoundNullElement = false
+ }
+ } else {
+ val assigned = if (!isLongType) {
+ assignInt(array1, i, resultArray, pos)
+ } else {
+ assignLong(array1, i, resultArray, pos)
+ }
+ if (assigned) {
+ pos += 1
+ }
+ }
+ i += 1
+ }
+ pos
+ }
+
+ val exceptEquals: (ArrayData, ArrayData) => ArrayData = {
--- End diff --
Maybe:
```scala
val exceptEquals: (ArrayData, ArrayData) => ArrayData = {
if (elementTypeSupportEquals) {
elementType match {
case IntegerType =>
(array1, array2) =>
...
case LongType =>
(array1, array2) =>
...
...
else {
(array1, array2) =>
...
}
}
```
to avoid the per-row pattern match.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21103: [SPARK-23915][SQL] Add array_except function
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21103
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/89629/
Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21103: [SPARK-23915][SQL] Add array_except function
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21103
**[Test build #93870 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/93870/testReport)** for PR 21103 at commit [`93e7979`](https://github.com/apache/spark/commit/93e7979a1c3fb82c47ecae5b3ed539b31cb99e19).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21103: [SPARK-23915][SQL] Add array_except function
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21103
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/93395/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21103: [SPARK-23915][SQL] Add array_except function
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21103
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/93653/
Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #21103: [SPARK-23915][SQL] Add array_except function
Posted by ueshin <gi...@git.apache.org>.
Github user ueshin commented on a diff in the pull request:
https://github.com/apache/spark/pull/21103#discussion_r205650766
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/collectionOperations.scala ---
@@ -3805,3 +3799,230 @@ object ArrayUnion {
new GenericArrayData(arrayBuffer)
}
}
+
+/**
+ * Returns an array of the elements in the intersect of x and y, without duplicates
+ */
+@ExpressionDescription(
+ usage = """
+ _FUNC_(array1, array2) - Returns an array of the elements in array1 but not in array2,
+ without duplicates.
+ """,
+ examples = """
+ Examples:
+ > SELECT _FUNC_(array(1, 2, 3), array(1, 3, 5));
+ array(2)
+ """,
+ since = "2.4.0")
+case class ArrayExcept(left: Expression, right: Expression) extends ArraySetLike {
+ override def dataType: DataType = left.dataType
+
+ var hsInt: OpenHashSet[Int] = _
+ var hsLong: OpenHashSet[Long] = _
+
+ def assignInt(array: ArrayData, idx: Int, resultArray: ArrayData, pos: Int): Boolean = {
+ val elem = array.getInt(idx)
+ if (!hsInt.contains(elem)) {
+ if (resultArray != null) {
+ resultArray.setInt(pos, elem)
+ }
+ hsInt.add(elem)
+ true
+ } else {
+ false
+ }
+ }
+
+ def assignLong(array: ArrayData, idx: Int, resultArray: ArrayData, pos: Int): Boolean = {
+ val elem = array.getLong(idx)
+ if (!hsLong.contains(elem)) {
+ if (resultArray != null) {
+ resultArray.setLong(pos, elem)
+ }
+ hsLong.add(elem)
+ true
+ } else {
+ false
+ }
+ }
+
+ def evalIntLongPrimitiveType(
+ array1: ArrayData,
+ array2: ArrayData,
+ resultArray: ArrayData,
+ isLongType: Boolean): Int = {
+ // store elements into resultArray
+ var notFoundNullElement = true
+ var i = 0
+ while (i < array2.numElements()) {
+ if (array2.isNullAt(i)) {
+ notFoundNullElement = false
+ } else {
+ val assigned = if (!isLongType) {
+ hsInt.add(array2.getInt(i))
+ } else {
+ hsLong.add(array2.getLong(i))
+ }
+ }
+ i += 1
+ }
+ var pos = 0
+ i = 0
+ while (i < array1.numElements()) {
+ if (array1.isNullAt(i)) {
+ if (notFoundNullElement) {
+ if (resultArray != null) {
+ resultArray.setNullAt(pos)
+ }
+ pos += 1
+ notFoundNullElement = false
+ }
+ } else {
+ val assigned = if (!isLongType) {
+ assignInt(array1, i, resultArray, pos)
+ } else {
+ assignLong(array1, i, resultArray, pos)
+ }
+ if (assigned) {
+ pos += 1
+ }
+ }
+ i += 1
+ }
+ pos
+ }
+
+ val exceptEquals: (ArrayData, ArrayData) => ArrayData = {
--- End diff --
Btw, `@transient lazy val`?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21103: [SPARK-23915][SQL] Add array_except function
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21103
**[Test build #93690 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/93690/testReport)** for PR 21103 at commit [`6ef1f22`](https://github.com/apache/spark/commit/6ef1f22aa68d52cf0c00b21211e19d3f80bab7c6).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #21103: [SPARK-23915][SQL] Add array_except function
Posted by mn-mikke <gi...@git.apache.org>.
Github user mn-mikke commented on a diff in the pull request:
https://github.com/apache/spark/pull/21103#discussion_r205395435
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/collectionOperations.scala ---
@@ -3805,3 +3799,330 @@ object ArrayUnion {
new GenericArrayData(arrayBuffer)
}
}
+
+/**
+ * Returns an array of the elements in the intersect of x and y, without duplicates
+ */
+@ExpressionDescription(
+ usage = """
+ _FUNC_(array1, array2) - Returns an array of the elements in array1 but not in array2,
+ without duplicates.
+ """,
+ examples = """
+ Examples:
+ > SELECT _FUNC_(array(1, 2, 3), array(1, 3, 5));
+ array(2)
+ """,
+ since = "2.4.0")
+case class ArrayExcept(left: Expression, right: Expression) extends ArraySetLike {
+ override def dataType: DataType = left.dataType
+
+ var hsInt: OpenHashSet[Int] = _
+ var hsLong: OpenHashSet[Long] = _
+
+ def assignInt(array: ArrayData, idx: Int, resultArray: ArrayData, pos: Int): Boolean = {
+ val elem = array.getInt(idx)
+ if (!hsInt.contains(elem)) {
+ if (resultArray != null) {
+ resultArray.setInt(pos, elem)
+ }
+ hsInt.add(elem)
+ true
+ } else {
+ false
+ }
+ }
+
+ def assignLong(array: ArrayData, idx: Int, resultArray: ArrayData, pos: Int): Boolean = {
+ val elem = array.getLong(idx)
+ if (!hsLong.contains(elem)) {
+ if (resultArray != null) {
+ resultArray.setLong(pos, elem)
+ }
+ hsLong.add(elem)
+ true
+ } else {
+ false
+ }
+ }
+
+ def evalIntLongPrimitiveType(
+ array1: ArrayData,
+ array2: ArrayData,
+ resultArray: ArrayData,
+ isLongType: Boolean): Int = {
+ // store elements into resultArray
+ var notFoundNullElement = true
+ var i = 0
+ while (i < array2.numElements()) {
+ if (array2.isNullAt(i)) {
+ notFoundNullElement = false
+ } else {
+ val assigned = if (!isLongType) {
+ hsInt.add(array2.getInt(i))
+ } else {
+ hsLong.add(array2.getLong(i))
+ }
+ }
+ i += 1
+ }
+ var pos = 0
+ i = 0
+ while (i < array1.numElements()) {
+ if (array1.isNullAt(i)) {
+ if (notFoundNullElement) {
+ if (resultArray != null) {
+ resultArray.setNullAt(pos)
+ }
+ pos += 1
+ notFoundNullElement = false
+ }
+ } else {
+ val assigned = if (!isLongType) {
+ assignInt(array1, i, resultArray, pos)
+ } else {
+ assignLong(array1, i, resultArray, pos)
+ }
+ if (assigned) {
+ pos += 1
+ }
+ }
+ i += 1
+ }
+ pos
+ }
+
+ override def nullSafeEval(input1: Any, input2: Any): Any = {
+ val array1 = input1.asInstanceOf[ArrayData]
+ val array2 = input2.asInstanceOf[ArrayData]
+
+ if (elementTypeSupportEquals) {
+ elementType match {
+ case IntegerType =>
+ // avoid boxing of primitive int array elements
+ // calculate result array size
+ hsInt = new OpenHashSet[Int]
+ val elements = evalIntLongPrimitiveType(array1, array2, null, false)
+ // allocate result array
+ hsInt = new OpenHashSet[Int]
+ val resultArray = if (UnsafeArrayData.shouldUseGenericArrayData(
+ IntegerType.defaultSize, elements)) {
+ new GenericArrayData(new Array[Any](elements))
+ } else {
+ UnsafeArrayData.forPrimitiveArray(
+ Platform.INT_ARRAY_OFFSET, elements, IntegerType.defaultSize)
+ }
+ // assign elements into the result array
+ evalIntLongPrimitiveType(array1, array2, resultArray, false)
+ resultArray
+ case LongType =>
+ // avoid boxing of primitive long array elements
+ // calculate result array size
+ hsLong = new OpenHashSet[Long]
+ val elements = evalIntLongPrimitiveType(array1, array2, null, true)
+ // allocate result array
+ hsLong = new OpenHashSet[Long]
+ val resultArray = if (UnsafeArrayData.shouldUseGenericArrayData(
+ LongType.defaultSize, elements)) {
+ new GenericArrayData(new Array[Any](elements))
+ } else {
+ UnsafeArrayData.forPrimitiveArray(
+ Platform.LONG_ARRAY_OFFSET, elements, LongType.defaultSize)
+ }
+ // assign elements into the result array
+ evalIntLongPrimitiveType(array1, array2, resultArray, true)
+ resultArray
+ case _ =>
+ val hs = new OpenHashSet[Any]
+ var notFoundNullElement = true
+ var i = 0
+ while (i < array2.numElements()) {
+ if (array2.isNullAt(i)) {
+ notFoundNullElement = false
+ } else {
+ val elem = array2.get(i, elementType)
+ hs.add(elem)
+ }
+ i += 1
+ }
+ val arrayBuffer = new scala.collection.mutable.ArrayBuffer[Any]
+ i = 0
+ while (i < array1.numElements()) {
+ if (array1.isNullAt(i)) {
+ if (notFoundNullElement) {
+ arrayBuffer += null
+ notFoundNullElement = false
+ }
+ } else {
+ val elem = array1.get(i, elementType)
+ if (!hs.contains(elem)) {
+ arrayBuffer += elem
+ hs.add(elem)
+ }
+ }
+ i += 1
+ }
+ new GenericArrayData(arrayBuffer)
+ }
+ } else {
+ ArrayExcept.exceptOrdering(array1, array2, elementType, ordering)
+ }
+ }
+
+ override def doGenCode(ctx: CodegenContext, ev: ExprCode): ExprCode = {
+ val i = ctx.freshName("i")
+ val pos = ctx.freshName("pos")
+ val value = ctx.freshName("value")
+ val size = ctx.freshName("size")
+ val (postFix, openHashElementType, getter, setter, javaTypeName, castOp, arrayBuilder) =
+ if (elementTypeSupportEquals) {
+ elementType match {
+ case ByteType | ShortType | IntegerType | LongType =>
+ val ptName = CodeGenerator.primitiveTypeName(elementType)
+ val unsafeArray = ctx.freshName("unsafeArray")
+ (if (elementType == LongType) s"$$mcJ$$sp" else s"$$mcI$$sp",
+ if (elementType == LongType) "Long" else "Int",
+ s"get$ptName($i)", s"set$ptName($pos, $value)", CodeGenerator.javaType(elementType),
+ if (elementType == LongType) "(long)" else "(int)",
+ s"""
+ |${ctx.createUnsafeArray(unsafeArray, size, elementType, s" $prettyName failed.")}
+ |${ev.value} = $unsafeArray;
+ """.stripMargin)
+ case _ =>
+ val genericArrayData = classOf[GenericArrayData].getName
+ val et = ctx.addReferenceObj("elementType", elementType)
+ ("", "Object",
+ s"get($i, $et)", s"update($pos, $value)", "Object", "",
+ s"${ev.value} = new $genericArrayData(new Object[$size]);")
+ }
+ } else {
+ ("", "", "", "", "", "", "")
+ }
+
+ nullSafeCodeGen(ctx, ev, (array1, array2) => {
+ if (openHashElementType != "") {
+ // Here, we ensure elementTypeSupportEquals is true
+ val notFoundNullElement = ctx.freshName("notFoundNullElement")
+ val openHashSet = classOf[OpenHashSet[_]].getName
+ val classTag = s"scala.reflect.ClassTag$$.MODULE$$.$openHashElementType()"
+ val hs = ctx.freshName("hs")
+ val arrayData = classOf[ArrayData].getName
+ val arrays = ctx.freshName("arrays")
+ val array = ctx.freshName("array")
+ val arrayDataIdx = ctx.freshName("arrayDataIdx")
+ s"""
+ |$openHashSet $hs = new $openHashSet$postFix($classTag);
+ |boolean $notFoundNullElement = true;
+ |int $size = 0;
+ |for (int $i = 0; $i < $array2.numElements(); $i++) {
+ | if ($array2.isNullAt($i)) {
+ | $notFoundNullElement = false;
+ | } else {
+ | $hs.add$postFix($array2.$getter);
+ | }
+ |}
+ |for (int $i = 0; $i < $array1.numElements(); $i++) {
+ | if ($array1.isNullAt($i)) {
+ | if ($notFoundNullElement) {
+ | $size++;
+ | $notFoundNullElement = false;
+ | }
+ | } else {
+ | $javaTypeName $value = $array1.$getter;
+ | if (!$hs.contains($castOp $value)) {
+ | $hs.add$postFix($value);
+ | $size++;
+ | }
+ | }
+ |}
+ |$arrayBuilder
+ |$hs = new $openHashSet$postFix($classTag);
+ |$notFoundNullElement = true;
+ |int $pos = 0;
+ |for (int $i = 0; $i < $array2.numElements(); $i++) {
+ | if ($array2.isNullAt($i)) {
+ | $notFoundNullElement = false;
+ | } else {
+ | $hs.add$postFix($array2.$getter);
+ | }
+ |}
+ |for (int $i = 0; $i < $array1.numElements(); $i++) {
+ | if ($array1.isNullAt($i)) {
+ | if ($notFoundNullElement) {
+ | ${ev.value}.setNullAt($pos++);
+ | $notFoundNullElement = false;
+ | }
+ | } else {
+ | $javaTypeName $value = $array1.$getter;
+ | if (!$hs.contains($castOp $value)) {
+ | $hs.add$postFix($value);
+ | ${ev.value}.$setter;
+ | $pos++;
+ | }
+ | }
+ |}
+ """.stripMargin
+ } else {
+ val arrayExcept = classOf[ArrayExcept].getName
+ val et = ctx.addReferenceObj("elementTypeIntersect", elementType)
+ val order = ctx.addReferenceObj("orderingIntersect", ordering)
+ val method = "exceptOrdering"
+ s"${ev.value} = $arrayExcept$$.MODULE$$.$method($array1, $array2, $et, $order);"
+ }
+ })
+ }
+
+ override def prettyName: String = "array_except"
+}
+
+object ArrayExcept {
+ def exceptOrdering(
+ array1: ArrayData,
+ array2: ArrayData,
+ elementType: DataType,
+ ordering: Ordering[Any]): ArrayData = {
+ val arrayBuffer = new scala.collection.mutable.ArrayBuffer[Any]
+ var scannedNullElements = false
+ var i = 0
+ while (i < array1.numElements()) {
+ var found = false
+ var elem1 = array1.get(i, elementType)
+ if (array1.isNullAt(i)) {
--- End diff --
nit: We already have a boxed instance. What about ```elem1 == null```?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21103: [SPARK-23915][SQL] Add array_except function
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21103
**[Test build #93589 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/93589/testReport)** for PR 21103 at commit [`cf76c1f`](https://github.com/apache/spark/commit/cf76c1f4a2a41ec88fcd744470a113321e897a71).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21103: [SPARK-23915][SQL] Add array_except function
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21103
**[Test build #92322 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/92322/testReport)** for PR 21103 at commit [`ad9f576`](https://github.com/apache/spark/commit/ad9f576961384936548adbd38dfb60296d3f0389).
* This patch **fails Spark unit tests**.
* This patch **does not merge cleanly**.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21103: [SPARK-23915][SQL] Add array_except function
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21103
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/93589/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21103: [SPARK-23915][SQL] Add array_except function
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21103
**[Test build #89651 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/89651/testReport)** for PR 21103 at commit [`ad9f576`](https://github.com/apache/spark/commit/ad9f576961384936548adbd38dfb60296d3f0389).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21103: [SPARK-23915][SQL] Add array_except function
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21103
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21103: [SPARK-23915][SQL] Add array_except function
Posted by cloud-fan <gi...@git.apache.org>.
Github user cloud-fan commented on the issue:
https://github.com/apache/spark/pull/21103
retest this please
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21103: [SPARK-23915][SQL] Add array_except function
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21103
**[Test build #93280 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/93280/testReport)** for PR 21103 at commit [`26e3257`](https://github.com/apache/spark/commit/26e3257b74fcf830ebcf5bb0f1a7499ad9030b01).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #21103: [SPARK-23915][SQL] Add array_except function
Posted by kiszk <gi...@git.apache.org>.
Github user kiszk commented on a diff in the pull request:
https://github.com/apache/spark/pull/21103#discussion_r205663032
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/collectionOperations.scala ---
@@ -3805,3 +3799,230 @@ object ArrayUnion {
new GenericArrayData(arrayBuffer)
}
}
+
+/**
+ * Returns an array of the elements in the intersect of x and y, without duplicates
+ */
+@ExpressionDescription(
+ usage = """
+ _FUNC_(array1, array2) - Returns an array of the elements in array1 but not in array2,
+ without duplicates.
+ """,
+ examples = """
+ Examples:
+ > SELECT _FUNC_(array(1, 2, 3), array(1, 3, 5));
+ array(2)
+ """,
+ since = "2.4.0")
+case class ArrayExcept(left: Expression, right: Expression) extends ArraySetLike {
+ override def dataType: DataType = left.dataType
+
+ var hsInt: OpenHashSet[Int] = _
+ var hsLong: OpenHashSet[Long] = _
+
+ def assignInt(array: ArrayData, idx: Int, resultArray: ArrayData, pos: Int): Boolean = {
+ val elem = array.getInt(idx)
+ if (!hsInt.contains(elem)) {
+ if (resultArray != null) {
+ resultArray.setInt(pos, elem)
+ }
+ hsInt.add(elem)
+ true
+ } else {
+ false
+ }
+ }
+
+ def assignLong(array: ArrayData, idx: Int, resultArray: ArrayData, pos: Int): Boolean = {
+ val elem = array.getLong(idx)
+ if (!hsLong.contains(elem)) {
+ if (resultArray != null) {
+ resultArray.setLong(pos, elem)
+ }
+ hsLong.add(elem)
+ true
+ } else {
+ false
+ }
+ }
+
+ def evalIntLongPrimitiveType(
+ array1: ArrayData,
+ array2: ArrayData,
+ resultArray: ArrayData,
+ isLongType: Boolean): Int = {
+ // store elements into resultArray
+ var notFoundNullElement = true
+ var i = 0
+ while (i < array2.numElements()) {
+ if (array2.isNullAt(i)) {
+ notFoundNullElement = false
+ } else {
+ val assigned = if (!isLongType) {
+ hsInt.add(array2.getInt(i))
+ } else {
+ hsLong.add(array2.getLong(i))
+ }
+ }
+ i += 1
+ }
+ var pos = 0
+ i = 0
+ while (i < array1.numElements()) {
+ if (array1.isNullAt(i)) {
+ if (notFoundNullElement) {
+ if (resultArray != null) {
+ resultArray.setNullAt(pos)
+ }
+ pos += 1
+ notFoundNullElement = false
+ }
+ } else {
+ val assigned = if (!isLongType) {
+ assignInt(array1, i, resultArray, pos)
+ } else {
+ assignLong(array1, i, resultArray, pos)
+ }
+ if (assigned) {
+ pos += 1
+ }
+ }
+ i += 1
+ }
+ pos
+ }
+
+ val exceptEquals: (ArrayData, ArrayData) => ArrayData = {
--- End diff --
Ah, you are right. Thank you.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #21103: [SPARK-23915][SQL] Add array_except function
Posted by kiszk <gi...@git.apache.org>.
Github user kiszk commented on a diff in the pull request:
https://github.com/apache/spark/pull/21103#discussion_r206000549
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/collectionOperations.scala ---
@@ -3968,3 +3964,267 @@ object ArrayUnion {
new GenericArrayData(arrayBuffer)
}
}
+
+/**
+ * Returns an array of the elements in the intersect of x and y, without duplicates
+ */
+@ExpressionDescription(
+ usage = """
+ _FUNC_(array1, array2) - Returns an array of the elements in array1 but not in array2,
+ without duplicates.
+ """,
+ examples = """
+ Examples:
+ > SELECT _FUNC_(array(1, 2, 3), array(1, 3, 5));
+ array(2)
+ """,
+ since = "2.4.0")
+case class ArrayExcept(left: Expression, right: Expression) extends ArraySetLike
+ with ComplexTypeMergingExpression {
+ override def dataType: DataType = {
+ dataTypeCheck
+ left.dataType
+ }
+
+ @transient lazy val evalExcept: (ArrayData, ArrayData) => ArrayData = {
+ if (elementTypeSupportEquals) {
+ (array1, array2) =>
+ val hs = new OpenHashSet[Any]
+ var notFoundNullElement = true
+ var i = 0
+ while (i < array2.numElements()) {
+ if (array2.isNullAt(i)) {
+ notFoundNullElement = false
+ } else {
+ val elem = array2.get(i, elementType)
+ hs.add(elem)
+ }
+ i += 1
+ }
+ val arrayBuffer = new scala.collection.mutable.ArrayBuffer[Any]
+ i = 0
+ while (i < array1.numElements()) {
+ if (array1.isNullAt(i)) {
+ if (notFoundNullElement) {
+ arrayBuffer += null
+ notFoundNullElement = false
+ }
+ } else {
+ val elem = array1.get(i, elementType)
+ if (!hs.contains(elem)) {
+ arrayBuffer += elem
+ hs.add(elem)
+ }
+ }
+ i += 1
+ }
+ new GenericArrayData(arrayBuffer)
+ } else {
+ (array1, array2) =>
+ val arrayBuffer = new scala.collection.mutable.ArrayBuffer[Any]
+ var scannedNullElements = false
+ var i = 0
+ while (i < array1.numElements()) {
+ var found = false
+ val elem1 = array1.get(i, elementType)
+ if (elem1 == null) {
+ if (!scannedNullElements) {
+ var j = 0
+ while (!found && j < array2.numElements()) {
+ found = array2.isNullAt(j)
+ j += 1
+ }
+ // array2 is scanned only once for null element
+ scannedNullElements = true
+ } else {
+ found = true
+ }
+ } else {
+ var j = 0
+ while (!found && j < array2.numElements()) {
+ val elem2 = array2.get(j, elementType)
+ if (elem2 != null) {
+ found = ordering.equiv(elem1, elem2)
+ }
+ j += 1
+ }
+ if (!found) {
+ // check whether elem1 is already stored in arrayBuffer
+ var k = 0
+ while (!found && k < arrayBuffer.size) {
+ val va = arrayBuffer(k)
+ found = (va != null) && ordering.equiv(va, elem1)
+ k += 1
+ }
+ }
+ }
+ if (!found) {
+ arrayBuffer += elem1
+ }
+ i += 1
+ }
+ new GenericArrayData(arrayBuffer)
+ }
+ }
+
+ override def nullSafeEval(input1: Any, input2: Any): Any = {
+ val array1 = input1.asInstanceOf[ArrayData]
+ val array2 = input2.asInstanceOf[ArrayData]
+
+ evalExcept(array1, array2)
+ }
+
+ override def doGenCode(ctx: CodegenContext, ev: ExprCode): ExprCode = {
+ val arrayData = classOf[ArrayData].getName
+ val i = ctx.freshName("i")
+ val pos = ctx.freshName("pos")
+ val value = ctx.freshName("value")
+ val hsValue = ctx.freshName("hsValue")
+ val size = ctx.freshName("size")
+ val (postFix, openHashElementType, hsJavaTypeName, genHsValue,
+ getter, setter, javaTypeName, arrayBuilder) =
+ if (elementTypeSupportEquals) {
+ elementType match {
+ case BooleanType | ByteType | ShortType | IntegerType =>
+ val ptName = CodeGenerator.primitiveTypeName(elementType)
+ val unsafeArray = ctx.freshName("unsafeArray")
+ ("$mcI$sp", "Int", "int",
+ if (elementType != BooleanType) {
+ s"(int) $value"
+ } else {
+ s"$value ? 1 : 0;"
+ },
+ s"get$ptName($i)", s"set$ptName($pos, $value)", CodeGenerator.javaType(elementType),
+ s"""
+ |${ctx.createUnsafeArray(unsafeArray, size, elementType, s" $prettyName failed.")}
+ |${ev.value} = $unsafeArray;
+ """.stripMargin)
+ case LongType | FloatType | DoubleType =>
+ val ptName = CodeGenerator.primitiveTypeName(elementType)
+ val unsafeArray = ctx.freshName("unsafeArray")
+ val signature = elementType match {
+ case LongType => "$mcJ$sp"
+ case FloatType => "$mcF$sp"
+ case DoubleType => "$mcD$sp"
+ }
+ (signature, CodeGenerator.boxedType(elementType),
+ CodeGenerator.javaType(elementType), value,
+ s"get$ptName($i)", s"set$ptName($pos, $value)", CodeGenerator.javaType(elementType),
+ s"""
+ |${ctx.createUnsafeArray(unsafeArray, size, elementType, s" $prettyName failed.")}
+ |${ev.value} = $unsafeArray;
+ """.stripMargin)
+ case _ =>
+ val genericArrayData = classOf[GenericArrayData].getName
+ val et = ctx.addReferenceObj("elementType", elementType)
+ ("", "Object", "Object", value,
+ s"get($i, $et)", s"update($pos, $value)", "Object",
+ s"${ev.value} = new $genericArrayData(new Object[$size]);")
+ }
+ } else {
+ ("", "", "", "", "", "", "", "")
+ }
+
+ nullSafeCodeGen(ctx, ev, (array1, array2) => {
+ if (openHashElementType != "") {
+ // Here, we ensure elementTypeSupportEquals is true
+ val notFoundNullElement = ctx.freshName("notFoundNullElement")
+ val openHashSet = classOf[OpenHashSet[_]].getName
+ val classTag = s"scala.reflect.ClassTag$$.MODULE$$.$openHashElementType()"
+ val hs = ctx.freshName("hs")
+ val arrayData = classOf[ArrayData].getName
+ val arrays = ctx.freshName("arrays")
+ val array = ctx.freshName("array")
+ val arrayDataIdx = ctx.freshName("arrayDataIdx")
+
+ val array2NullCheck = if (right.dataType.asInstanceOf[ArrayType].containsNull) {
+ s"""
+ |if ($array2.isNullAt($i)) {
+ | $notFoundNullElement = false;
+ |} else
+ """.stripMargin
+ } else {
+ ""
+ }
+ val array1NullCheck = if (left.dataType.asInstanceOf[ArrayType].containsNull) {
+ s"""
+ |if ($array1.isNullAt($i)) {
+ | if ($notFoundNullElement) {
+ | $size++;
+ | $notFoundNullElement = false;
+ | }
+ |} else
+ """.stripMargin
+ } else {
+ ""
+ }
+ val array1NullAssignment = if (left.dataType.asInstanceOf[ArrayType].containsNull) {
+ s"""
+ |if ($array1.isNullAt($i)) {
+ | if ($notFoundNullElement) {
+ | ${ev.value}.setNullAt($pos++);
+ | $notFoundNullElement = false;
+ | }
+ |} else
+ """.stripMargin
+ } else {
+ ""
+ }
+
+ s"""
+ |$openHashSet $hs = new $openHashSet$postFix($classTag);
+ |boolean $notFoundNullElement = true;
+ |int $size = 0;
+ |for (int $i = 0; $i < $array2.numElements(); $i++) {
+ | $array2NullCheck
+ | {
+ | $javaTypeName $value = $array2.$getter;
+ | $hsJavaTypeName $hsValue = $genHsValue;
+ | $hs.add$postFix($hsValue);
+ | }
+ |}
+ |for (int $i = 0; $i < $array1.numElements(); $i++) {
+ | $array1NullCheck
+ | {
+ | $javaTypeName $value = $array1.$getter;
+ | $hsJavaTypeName $hsValue = $genHsValue;
+ | if (!$hs.contains($hsValue)) {
+ | $hs.add$postFix($hsValue);
+ | $size++;
+ | }
+ | }
+ |}
+ |$arrayBuilder
+ |$hs = new $openHashSet$postFix($classTag);
+ |$notFoundNullElement = true;
+ |int $pos = 0;
+ |for (int $i = 0; $i < $array2.numElements(); $i++) {
--- End diff --
We add some elements in `array1`to the hash set at L4192.
Since we want to create a hashset that have elements only of `array2`, we reallocate a hashset at L4198 and add elements of `array2` again here.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21103: [SPARK-23915][SQL] Add array_except function
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21103
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/1516/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21103: [SPARK-23915][SQL] Add array_except function
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21103
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/93674/
Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21103: [SPARK-23915][SQL] Add array_except function
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21103
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21103: [SPARK-23915][SQL] Add array_except function
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21103
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21103: [SPARK-23915][SQL] Add array_except function
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21103
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/2527/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21103: [SPARK-23915][SQL] Add array_except function
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21103
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/1340/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21103: [SPARK-23915][SQL] Add array_except function
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21103
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21103: [SPARK-23915][SQL] Add array_except function
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21103
Build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21103: [SPARK-23915][SQL] Add array_except function
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21103
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/93669/
Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #21103: [SPARK-23915][SQL] Add array_except function
Posted by ueshin <gi...@git.apache.org>.
Github user ueshin commented on a diff in the pull request:
https://github.com/apache/spark/pull/21103#discussion_r204307100
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/collectionOperations.scala ---
@@ -3805,3 +3799,331 @@ object ArrayUnion {
new GenericArrayData(arrayBuffer)
}
}
+
+/**
+ * Returns an array of the elements in the intersect of x and y, without duplicates
+ */
+@ExpressionDescription(
+ usage = """
+ _FUNC_(array1, array2) - Returns an array of the elements in array1 but not in array2,
+ without duplicates.
+ """,
+ examples = """
+ Examples:
+ > SELECT _FUNC_(array(1, 2, 3), array(1, 3, 5));
+ array(2)
+ """,
+ since = "2.4.0")
+case class ArrayExcept(left: Expression, right: Expression) extends ArraySetLike {
+ override def dataType: DataType =
+ ArrayType(elementType, left.dataType.asInstanceOf[ArrayType].containsNull)
+
+ var hsInt: OpenHashSet[Int] = _
+ var hsLong: OpenHashSet[Long] = _
+
+ def assignInt(array: ArrayData, idx: Int, resultArray: ArrayData, pos: Int): Boolean = {
+ val elem = array.getInt(idx)
+ if (!hsInt.contains(elem)) {
+ if (resultArray != null) {
+ resultArray.setInt(pos, elem)
+ }
+ hsInt.add(elem)
+ true
+ } else {
+ false
+ }
+ }
+
+ def assignLong(array: ArrayData, idx: Int, resultArray: ArrayData, pos: Int): Boolean = {
+ val elem = array.getLong(idx)
+ if (!hsLong.contains(elem)) {
+ if (resultArray != null) {
+ resultArray.setLong(pos, elem)
+ }
+ hsLong.add(elem)
+ true
+ } else {
+ false
+ }
+ }
+
+ def evalIntLongPrimitiveType(
+ array1: ArrayData,
+ array2: ArrayData,
+ resultArray: ArrayData,
+ isLongType: Boolean): Int = {
+ // store elements into resultArray
+ var exceptNullElement = true
--- End diff --
The meaning is opposite?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21103: [SPARK-23915][SQL] Add array_except function
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21103
**[Test build #93683 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/93683/testReport)** for PR 21103 at commit [`f4d4f8c`](https://github.com/apache/spark/commit/f4d4f8c89ce82ed058f6696885363f0e00154eaf).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21103: [SPARK-23915][SQL] Add array_except function
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21103
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21103: [SPARK-23915][SQL] Add array_except function
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21103
Merged build finished. Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #21103: [SPARK-23915][SQL] Add array_except function
Posted by cloud-fan <gi...@git.apache.org>.
Github user cloud-fan commented on a diff in the pull request:
https://github.com/apache/spark/pull/21103#discussion_r205668541
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/collectionOperations.scala ---
@@ -3805,3 +3799,233 @@ object ArrayUnion {
new GenericArrayData(arrayBuffer)
}
}
+
+/**
+ * Returns an array of the elements in the intersect of x and y, without duplicates
+ */
+@ExpressionDescription(
+ usage = """
+ _FUNC_(array1, array2) - Returns an array of the elements in array1 but not in array2,
+ without duplicates.
+ """,
+ examples = """
+ Examples:
+ > SELECT _FUNC_(array(1, 2, 3), array(1, 3, 5));
+ array(2)
+ """,
+ since = "2.4.0")
+case class ArrayExcept(left: Expression, right: Expression) extends ArraySetLike {
+ override def dataType: DataType = left.dataType
+
+ var hsInt: OpenHashSet[Int] = _
+ var hsLong: OpenHashSet[Long] = _
+
+ def assignInt(array: ArrayData, idx: Int, resultArray: ArrayData, pos: Int): Boolean = {
+ val elem = array.getInt(idx)
+ if (!hsInt.contains(elem)) {
+ if (resultArray != null) {
+ resultArray.setInt(pos, elem)
+ }
+ hsInt.add(elem)
+ true
+ } else {
+ false
+ }
+ }
+
+ def assignLong(array: ArrayData, idx: Int, resultArray: ArrayData, pos: Int): Boolean = {
+ val elem = array.getLong(idx)
+ if (!hsLong.contains(elem)) {
+ if (resultArray != null) {
+ resultArray.setLong(pos, elem)
+ }
+ hsLong.add(elem)
+ true
+ } else {
+ false
+ }
+ }
+
+ def evalIntLongPrimitiveType(
+ array1: ArrayData,
+ array2: ArrayData,
+ resultArray: ArrayData,
+ isLongType: Boolean): Int = {
+ // store elements into resultArray
+ var notFoundNullElement = true
+ var i = 0
+ while (i < array2.numElements()) {
+ if (array2.isNullAt(i)) {
+ notFoundNullElement = false
+ } else {
+ val assigned = if (!isLongType) {
+ hsInt.add(array2.getInt(i))
+ } else {
+ hsLong.add(array2.getLong(i))
+ }
+ }
+ i += 1
+ }
+ var pos = 0
+ i = 0
+ while (i < array1.numElements()) {
+ if (array1.isNullAt(i)) {
+ if (notFoundNullElement) {
+ if (resultArray != null) {
+ resultArray.setNullAt(pos)
+ }
+ pos += 1
+ notFoundNullElement = false
+ }
+ } else {
+ val assigned = if (!isLongType) {
+ assignInt(array1, i, resultArray, pos)
+ } else {
+ assignLong(array1, i, resultArray, pos)
+ }
+ if (assigned) {
+ pos += 1
+ }
+ }
+ i += 1
+ }
+ pos
+ }
+
+ @transient lazy val evalExcept: (ArrayData, ArrayData) => ArrayData = {
+ if (elementTypeSupportEquals) {
+ elementType match {
+ case IntegerType =>
--- End diff --
why we only optimize for int and long? how about byte, short, float, double?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21103: [SPARK-23915][SQL] Add array_except function
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21103
**[Test build #93395 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/93395/testReport)** for PR 21103 at commit [`92cd5f0`](https://github.com/apache/spark/commit/92cd5f0855d716bb3b27cf2ca7122090329e1c2b).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21103: [SPARK-23915][SQL] Add array_except function
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21103
**[Test build #93653 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/93653/testReport)** for PR 21103 at commit [`4d01c98`](https://github.com/apache/spark/commit/4d01c9848e021006e2412ebb2db3e37782b5f41a).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21103: [SPARK-23915][SQL] Add array_except function
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21103
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21103: [SPARK-23915][SQL] Add array_except function
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21103
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21103: [SPARK-23915][SQL] Add array_except function
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21103
**[Test build #93644 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/93644/testReport)** for PR 21103 at commit [`ad0c318`](https://github.com/apache/spark/commit/ad0c318c1fb63bd08c69995d192ed3ab9a98e4c2).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21103: [SPARK-23915][SQL] Add array_except function
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21103
**[Test build #93690 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/93690/testReport)** for PR 21103 at commit [`6ef1f22`](https://github.com/apache/spark/commit/6ef1f22aa68d52cf0c00b21211e19d3f80bab7c6).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds the following public classes _(experimental)_:
* ` sealed class Hasher[@specialized(Long, Int, Double, Float) T] extends Serializable `
* ` class DoubleHasher extends Hasher[Double] `
* ` class FloatHasher extends Hasher[Float] `
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21103: [SPARK-23915][SQL] Add array_except function
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21103
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/92997/
Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21103: [SPARK-23915][SQL] Add array_except function
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21103
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/1514/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #21103: [SPARK-23915][SQL] Add array_except function
Posted by kiszk <gi...@git.apache.org>.
Github user kiszk commented on a diff in the pull request:
https://github.com/apache/spark/pull/21103#discussion_r206002948
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/collectionOperations.scala ---
@@ -3968,3 +3964,267 @@ object ArrayUnion {
new GenericArrayData(arrayBuffer)
}
}
+
+/**
+ * Returns an array of the elements in the intersect of x and y, without duplicates
+ */
+@ExpressionDescription(
+ usage = """
+ _FUNC_(array1, array2) - Returns an array of the elements in array1 but not in array2,
+ without duplicates.
+ """,
+ examples = """
+ Examples:
+ > SELECT _FUNC_(array(1, 2, 3), array(1, 3, 5));
+ array(2)
+ """,
+ since = "2.4.0")
+case class ArrayExcept(left: Expression, right: Expression) extends ArraySetLike
+ with ComplexTypeMergingExpression {
+ override def dataType: DataType = {
+ dataTypeCheck
+ left.dataType
+ }
+
+ @transient lazy val evalExcept: (ArrayData, ArrayData) => ArrayData = {
+ if (elementTypeSupportEquals) {
+ (array1, array2) =>
+ val hs = new OpenHashSet[Any]
+ var notFoundNullElement = true
+ var i = 0
+ while (i < array2.numElements()) {
+ if (array2.isNullAt(i)) {
+ notFoundNullElement = false
+ } else {
+ val elem = array2.get(i, elementType)
+ hs.add(elem)
+ }
+ i += 1
+ }
+ val arrayBuffer = new scala.collection.mutable.ArrayBuffer[Any]
+ i = 0
+ while (i < array1.numElements()) {
+ if (array1.isNullAt(i)) {
+ if (notFoundNullElement) {
+ arrayBuffer += null
+ notFoundNullElement = false
+ }
+ } else {
+ val elem = array1.get(i, elementType)
+ if (!hs.contains(elem)) {
+ arrayBuffer += elem
+ hs.add(elem)
+ }
+ }
+ i += 1
+ }
+ new GenericArrayData(arrayBuffer)
+ } else {
+ (array1, array2) =>
+ val arrayBuffer = new scala.collection.mutable.ArrayBuffer[Any]
+ var scannedNullElements = false
+ var i = 0
+ while (i < array1.numElements()) {
+ var found = false
+ val elem1 = array1.get(i, elementType)
+ if (elem1 == null) {
+ if (!scannedNullElements) {
+ var j = 0
+ while (!found && j < array2.numElements()) {
+ found = array2.isNullAt(j)
+ j += 1
+ }
+ // array2 is scanned only once for null element
+ scannedNullElements = true
+ } else {
+ found = true
+ }
+ } else {
+ var j = 0
+ while (!found && j < array2.numElements()) {
+ val elem2 = array2.get(j, elementType)
+ if (elem2 != null) {
+ found = ordering.equiv(elem1, elem2)
+ }
+ j += 1
+ }
+ if (!found) {
+ // check whether elem1 is already stored in arrayBuffer
+ var k = 0
+ while (!found && k < arrayBuffer.size) {
+ val va = arrayBuffer(k)
+ found = (va != null) && ordering.equiv(va, elem1)
+ k += 1
+ }
+ }
+ }
+ if (!found) {
+ arrayBuffer += elem1
+ }
+ i += 1
+ }
+ new GenericArrayData(arrayBuffer)
+ }
+ }
+
+ override def nullSafeEval(input1: Any, input2: Any): Any = {
+ val array1 = input1.asInstanceOf[ArrayData]
+ val array2 = input2.asInstanceOf[ArrayData]
+
+ evalExcept(array1, array2)
+ }
+
+ override def doGenCode(ctx: CodegenContext, ev: ExprCode): ExprCode = {
+ val arrayData = classOf[ArrayData].getName
+ val i = ctx.freshName("i")
+ val pos = ctx.freshName("pos")
+ val value = ctx.freshName("value")
+ val hsValue = ctx.freshName("hsValue")
+ val size = ctx.freshName("size")
+ val (postFix, openHashElementType, hsJavaTypeName, genHsValue,
+ getter, setter, javaTypeName, arrayBuilder) =
+ if (elementTypeSupportEquals) {
+ elementType match {
+ case BooleanType | ByteType | ShortType | IntegerType =>
+ val ptName = CodeGenerator.primitiveTypeName(elementType)
+ val unsafeArray = ctx.freshName("unsafeArray")
+ ("$mcI$sp", "Int", "int",
+ if (elementType != BooleanType) {
+ s"(int) $value"
+ } else {
+ s"$value ? 1 : 0;"
+ },
+ s"get$ptName($i)", s"set$ptName($pos, $value)", CodeGenerator.javaType(elementType),
+ s"""
+ |${ctx.createUnsafeArray(unsafeArray, size, elementType, s" $prettyName failed.")}
+ |${ev.value} = $unsafeArray;
+ """.stripMargin)
+ case LongType | FloatType | DoubleType =>
+ val ptName = CodeGenerator.primitiveTypeName(elementType)
+ val unsafeArray = ctx.freshName("unsafeArray")
+ val signature = elementType match {
+ case LongType => "$mcJ$sp"
+ case FloatType => "$mcF$sp"
+ case DoubleType => "$mcD$sp"
+ }
+ (signature, CodeGenerator.boxedType(elementType),
+ CodeGenerator.javaType(elementType), value,
+ s"get$ptName($i)", s"set$ptName($pos, $value)", CodeGenerator.javaType(elementType),
+ s"""
+ |${ctx.createUnsafeArray(unsafeArray, size, elementType, s" $prettyName failed.")}
+ |${ev.value} = $unsafeArray;
+ """.stripMargin)
+ case _ =>
+ val genericArrayData = classOf[GenericArrayData].getName
+ val et = ctx.addReferenceObj("elementType", elementType)
+ ("", "Object", "Object", value,
+ s"get($i, $et)", s"update($pos, $value)", "Object",
+ s"${ev.value} = new $genericArrayData(new Object[$size]);")
+ }
+ } else {
+ ("", "", "", "", "", "", "", "")
+ }
+
+ nullSafeCodeGen(ctx, ev, (array1, array2) => {
+ if (openHashElementType != "") {
+ // Here, we ensure elementTypeSupportEquals is true
+ val notFoundNullElement = ctx.freshName("notFoundNullElement")
+ val openHashSet = classOf[OpenHashSet[_]].getName
+ val classTag = s"scala.reflect.ClassTag$$.MODULE$$.$openHashElementType()"
+ val hs = ctx.freshName("hs")
+ val arrayData = classOf[ArrayData].getName
+ val arrays = ctx.freshName("arrays")
+ val array = ctx.freshName("array")
+ val arrayDataIdx = ctx.freshName("arrayDataIdx")
+
+ val array2NullCheck = if (right.dataType.asInstanceOf[ArrayType].containsNull) {
+ s"""
+ |if ($array2.isNullAt($i)) {
+ | $notFoundNullElement = false;
+ |} else
+ """.stripMargin
+ } else {
+ ""
+ }
+ val array1NullCheck = if (left.dataType.asInstanceOf[ArrayType].containsNull) {
+ s"""
+ |if ($array1.isNullAt($i)) {
+ | if ($notFoundNullElement) {
+ | $size++;
+ | $notFoundNullElement = false;
+ | }
+ |} else
+ """.stripMargin
+ } else {
+ ""
+ }
+ val array1NullAssignment = if (left.dataType.asInstanceOf[ArrayType].containsNull) {
+ s"""
+ |if ($array1.isNullAt($i)) {
+ | if ($notFoundNullElement) {
+ | ${ev.value}.setNullAt($pos++);
+ | $notFoundNullElement = false;
+ | }
+ |} else
+ """.stripMargin
+ } else {
+ ""
+ }
+
+ s"""
+ |$openHashSet $hs = new $openHashSet$postFix($classTag);
+ |boolean $notFoundNullElement = true;
+ |int $size = 0;
+ |for (int $i = 0; $i < $array2.numElements(); $i++) {
+ | $array2NullCheck
+ | {
+ | $javaTypeName $value = $array2.$getter;
+ | $hsJavaTypeName $hsValue = $genHsValue;
+ | $hs.add$postFix($hsValue);
+ | }
+ |}
+ |for (int $i = 0; $i < $array1.numElements(); $i++) {
+ | $array1NullCheck
+ | {
+ | $javaTypeName $value = $array1.$getter;
+ | $hsJavaTypeName $hsValue = $genHsValue;
+ | if (!$hs.contains($hsValue)) {
+ | $hs.add$postFix($hsValue);
+ | $size++;
+ | }
+ | }
+ |}
+ |$arrayBuilder
+ |$hs = new $openHashSet$postFix($classTag);
+ |$notFoundNullElement = true;
+ |int $pos = 0;
+ |for (int $i = 0; $i < $array2.numElements(); $i++) {
--- End diff --
Yes, only calculating array size for allocating a `UnsafeArrayData` or `GenericArrayData`.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #21103: [SPARK-23915][SQL] Add array_except function
Posted by cloud-fan <gi...@git.apache.org>.
Github user cloud-fan commented on a diff in the pull request:
https://github.com/apache/spark/pull/21103#discussion_r206001326
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/collectionOperations.scala ---
@@ -3968,3 +3964,267 @@ object ArrayUnion {
new GenericArrayData(arrayBuffer)
}
}
+
+/**
+ * Returns an array of the elements in the intersect of x and y, without duplicates
+ */
+@ExpressionDescription(
+ usage = """
+ _FUNC_(array1, array2) - Returns an array of the elements in array1 but not in array2,
+ without duplicates.
+ """,
+ examples = """
+ Examples:
+ > SELECT _FUNC_(array(1, 2, 3), array(1, 3, 5));
+ array(2)
+ """,
+ since = "2.4.0")
+case class ArrayExcept(left: Expression, right: Expression) extends ArraySetLike
+ with ComplexTypeMergingExpression {
+ override def dataType: DataType = {
+ dataTypeCheck
+ left.dataType
+ }
+
+ @transient lazy val evalExcept: (ArrayData, ArrayData) => ArrayData = {
+ if (elementTypeSupportEquals) {
+ (array1, array2) =>
+ val hs = new OpenHashSet[Any]
+ var notFoundNullElement = true
+ var i = 0
+ while (i < array2.numElements()) {
+ if (array2.isNullAt(i)) {
+ notFoundNullElement = false
+ } else {
+ val elem = array2.get(i, elementType)
+ hs.add(elem)
+ }
+ i += 1
+ }
+ val arrayBuffer = new scala.collection.mutable.ArrayBuffer[Any]
+ i = 0
+ while (i < array1.numElements()) {
+ if (array1.isNullAt(i)) {
+ if (notFoundNullElement) {
+ arrayBuffer += null
+ notFoundNullElement = false
+ }
+ } else {
+ val elem = array1.get(i, elementType)
+ if (!hs.contains(elem)) {
+ arrayBuffer += elem
+ hs.add(elem)
+ }
+ }
+ i += 1
+ }
+ new GenericArrayData(arrayBuffer)
+ } else {
+ (array1, array2) =>
+ val arrayBuffer = new scala.collection.mutable.ArrayBuffer[Any]
+ var scannedNullElements = false
+ var i = 0
+ while (i < array1.numElements()) {
+ var found = false
+ val elem1 = array1.get(i, elementType)
+ if (elem1 == null) {
+ if (!scannedNullElements) {
+ var j = 0
+ while (!found && j < array2.numElements()) {
+ found = array2.isNullAt(j)
+ j += 1
+ }
+ // array2 is scanned only once for null element
+ scannedNullElements = true
+ } else {
+ found = true
+ }
+ } else {
+ var j = 0
+ while (!found && j < array2.numElements()) {
+ val elem2 = array2.get(j, elementType)
+ if (elem2 != null) {
+ found = ordering.equiv(elem1, elem2)
+ }
+ j += 1
+ }
+ if (!found) {
+ // check whether elem1 is already stored in arrayBuffer
+ var k = 0
+ while (!found && k < arrayBuffer.size) {
+ val va = arrayBuffer(k)
+ found = (va != null) && ordering.equiv(va, elem1)
+ k += 1
+ }
+ }
+ }
+ if (!found) {
+ arrayBuffer += elem1
+ }
+ i += 1
+ }
+ new GenericArrayData(arrayBuffer)
+ }
+ }
+
+ override def nullSafeEval(input1: Any, input2: Any): Any = {
+ val array1 = input1.asInstanceOf[ArrayData]
+ val array2 = input2.asInstanceOf[ArrayData]
+
+ evalExcept(array1, array2)
+ }
+
+ override def doGenCode(ctx: CodegenContext, ev: ExprCode): ExprCode = {
+ val arrayData = classOf[ArrayData].getName
+ val i = ctx.freshName("i")
+ val pos = ctx.freshName("pos")
+ val value = ctx.freshName("value")
+ val hsValue = ctx.freshName("hsValue")
+ val size = ctx.freshName("size")
+ val (postFix, openHashElementType, hsJavaTypeName, genHsValue,
+ getter, setter, javaTypeName, arrayBuilder) =
+ if (elementTypeSupportEquals) {
+ elementType match {
+ case BooleanType | ByteType | ShortType | IntegerType =>
+ val ptName = CodeGenerator.primitiveTypeName(elementType)
+ val unsafeArray = ctx.freshName("unsafeArray")
+ ("$mcI$sp", "Int", "int",
+ if (elementType != BooleanType) {
+ s"(int) $value"
+ } else {
+ s"$value ? 1 : 0;"
+ },
+ s"get$ptName($i)", s"set$ptName($pos, $value)", CodeGenerator.javaType(elementType),
+ s"""
+ |${ctx.createUnsafeArray(unsafeArray, size, elementType, s" $prettyName failed.")}
+ |${ev.value} = $unsafeArray;
+ """.stripMargin)
+ case LongType | FloatType | DoubleType =>
+ val ptName = CodeGenerator.primitiveTypeName(elementType)
+ val unsafeArray = ctx.freshName("unsafeArray")
+ val signature = elementType match {
+ case LongType => "$mcJ$sp"
+ case FloatType => "$mcF$sp"
+ case DoubleType => "$mcD$sp"
+ }
+ (signature, CodeGenerator.boxedType(elementType),
+ CodeGenerator.javaType(elementType), value,
+ s"get$ptName($i)", s"set$ptName($pos, $value)", CodeGenerator.javaType(elementType),
+ s"""
+ |${ctx.createUnsafeArray(unsafeArray, size, elementType, s" $prettyName failed.")}
+ |${ev.value} = $unsafeArray;
+ """.stripMargin)
+ case _ =>
+ val genericArrayData = classOf[GenericArrayData].getName
+ val et = ctx.addReferenceObj("elementType", elementType)
+ ("", "Object", "Object", value,
+ s"get($i, $et)", s"update($pos, $value)", "Object",
+ s"${ev.value} = new $genericArrayData(new Object[$size]);")
+ }
+ } else {
+ ("", "", "", "", "", "", "", "")
+ }
+
+ nullSafeCodeGen(ctx, ev, (array1, array2) => {
+ if (openHashElementType != "") {
+ // Here, we ensure elementTypeSupportEquals is true
+ val notFoundNullElement = ctx.freshName("notFoundNullElement")
+ val openHashSet = classOf[OpenHashSet[_]].getName
+ val classTag = s"scala.reflect.ClassTag$$.MODULE$$.$openHashElementType()"
+ val hs = ctx.freshName("hs")
+ val arrayData = classOf[ArrayData].getName
+ val arrays = ctx.freshName("arrays")
+ val array = ctx.freshName("array")
+ val arrayDataIdx = ctx.freshName("arrayDataIdx")
+
+ val array2NullCheck = if (right.dataType.asInstanceOf[ArrayType].containsNull) {
+ s"""
+ |if ($array2.isNullAt($i)) {
+ | $notFoundNullElement = false;
+ |} else
+ """.stripMargin
+ } else {
+ ""
+ }
+ val array1NullCheck = if (left.dataType.asInstanceOf[ArrayType].containsNull) {
+ s"""
+ |if ($array1.isNullAt($i)) {
+ | if ($notFoundNullElement) {
+ | $size++;
+ | $notFoundNullElement = false;
+ | }
+ |} else
+ """.stripMargin
+ } else {
+ ""
+ }
+ val array1NullAssignment = if (left.dataType.asInstanceOf[ArrayType].containsNull) {
+ s"""
+ |if ($array1.isNullAt($i)) {
+ | if ($notFoundNullElement) {
+ | ${ev.value}.setNullAt($pos++);
+ | $notFoundNullElement = false;
+ | }
+ |} else
+ """.stripMargin
+ } else {
+ ""
+ }
+
+ s"""
+ |$openHashSet $hs = new $openHashSet$postFix($classTag);
+ |boolean $notFoundNullElement = true;
+ |int $size = 0;
+ |for (int $i = 0; $i < $array2.numElements(); $i++) {
+ | $array2NullCheck
+ | {
+ | $javaTypeName $value = $array2.$getter;
+ | $hsJavaTypeName $hsValue = $genHsValue;
+ | $hs.add$postFix($hsValue);
+ | }
+ |}
+ |for (int $i = 0; $i < $array1.numElements(); $i++) {
+ | $array1NullCheck
+ | {
+ | $javaTypeName $value = $array1.$getter;
+ | $hsJavaTypeName $hsValue = $genHsValue;
+ | if (!$hs.contains($hsValue)) {
+ | $hs.add$postFix($hsValue);
+ | $size++;
+ | }
+ | }
+ |}
+ |$arrayBuilder
+ |$hs = new $openHashSet$postFix($classTag);
+ |$notFoundNullElement = true;
+ |int $pos = 0;
+ |for (int $i = 0; $i < $array2.numElements(); $i++) {
--- End diff --
then why we do the first iteration? only calculating the size?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21103: [SPARK-23915][SQL] Add array_except function
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21103
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21103: [SPARK-23915][SQL] Add array_except function
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21103
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21103: [SPARK-23915][SQL] Add array_except function
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21103
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/93392/
Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #21103: [SPARK-23915][SQL] Add array_except function
Posted by kiszk <gi...@git.apache.org>.
Github user kiszk commented on a diff in the pull request:
https://github.com/apache/spark/pull/21103#discussion_r204218477
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/collectionOperations.scala ---
@@ -3805,3 +3799,332 @@ object ArrayUnion {
new GenericArrayData(arrayBuffer)
}
}
+
+/**
+ * Returns an array of the elements in the intersect of x and y, without duplicates
+ */
+@ExpressionDescription(
+ usage = """
+ _FUNC_(array1, array2) - Returns an array of the elements in array1 but not in array2,
+ without duplicates.
+ """,
+ examples = """
+ Examples:Fun
+ > SELECT _FUNC_(array(1, 2, 3), array(1, 3, 5));
+ array(2)
+ """,
+ since = "2.4.0")
+case class ArrayExcept(left: Expression, right: Expression) extends ArraySetLike {
+ override def dataType: DataType = ArrayType(elementType,
--- End diff --
I see. It makes sense.
I added a test for this into `DataFrameFunctionsSuite`.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21103: [SPARK-23915][SQL] Add array_except function
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21103
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/93001/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21103: [SPARK-23915][SQL] Add array_except function
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21103
**[Test build #89605 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/89605/testReport)** for PR 21103 at commit [`cdb6257`](https://github.com/apache/spark/commit/cdb6257187c3713ccece28d9b8073bf99180b358).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds the following public classes _(experimental)_:
* `case class ArrayExcept(left: Expression, right: Expression) extends ArraySetUtils `
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21103: [SPARK-23915][SQL] Add array_except function
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21103
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/1206/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #21103: [SPARK-23915][SQL] Add array_except function
Posted by cloud-fan <gi...@git.apache.org>.
Github user cloud-fan commented on a diff in the pull request:
https://github.com/apache/spark/pull/21103#discussion_r205929135
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/collectionOperations.scala ---
@@ -3968,3 +3964,242 @@ object ArrayUnion {
new GenericArrayData(arrayBuffer)
}
}
+
+/**
+ * Returns an array of the elements in the intersect of x and y, without duplicates
+ */
+@ExpressionDescription(
+ usage = """
+ _FUNC_(array1, array2) - Returns an array of the elements in array1 but not in array2,
+ without duplicates.
+ """,
+ examples = """
+ Examples:
+ > SELECT _FUNC_(array(1, 2, 3), array(1, 3, 5));
+ array(2)
+ """,
+ since = "2.4.0")
+case class ArrayExcept(left: Expression, right: Expression) extends ArraySetLike
+ with ComplexTypeMergingExpression {
+ override def dataType: DataType = {
+ dataTypeCheck
+ left.dataType
+ }
+
+ @transient lazy val evalExcept: (ArrayData, ArrayData) => ArrayData = {
+ if (elementTypeSupportEquals) {
+ (array1, array2) =>
+ val hs = new OpenHashSet[Any]
--- End diff --
This makes me think that it's a bad idea to make `byte[]` the internal data class for binary type. IIRC we have a lot of places to work around the equality issue of `byte[]`, and introduce extra copy when reading binary type column from off-heap. We should consider something like `UTF8String` for binary type.
This is not related to this PR, but is something we can do in Spark 3.0. cc @rxin @hvanhovell
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21103: [SPARK-23915][SQL] Add array_except function
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21103
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/93851/
Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21103: [SPARK-23915][SQL] Add array_except function
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21103
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/1406/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21103: [SPARK-23915][SQL] Add array_except function
Posted by kiszk <gi...@git.apache.org>.
Github user kiszk commented on the issue:
https://github.com/apache/spark/pull/21103
retest this please
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21103: [SPARK-23915][SQL] Add array_except function
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21103
Merged build finished. Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21103: [SPARK-23915][SQL] Add array_except function
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21103
Merged build finished. Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21103: [SPARK-23915][SQL] Add array_except function
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21103
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/93831/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21103: [SPARK-23915][SQL] Add array_except function
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21103
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/93870/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21103: [SPARK-23915][SQL] Add array_except function
Posted by mn-mikke <gi...@git.apache.org>.
Github user mn-mikke commented on the issue:
https://github.com/apache/spark/pull/21103
retest this please
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21103: [SPARK-23915][SQL] Add array_except function
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21103
**[Test build #93851 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/93851/testReport)** for PR 21103 at commit [`93e7979`](https://github.com/apache/spark/commit/93e7979a1c3fb82c47ecae5b3ed539b31cb99e19).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21103: [SPARK-23915][SQL] Add array_except function
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21103
Merged build finished. Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #21103: [SPARK-23915][SQL] Add array_except function
Posted by cloud-fan <gi...@git.apache.org>.
Github user cloud-fan commented on a diff in the pull request:
https://github.com/apache/spark/pull/21103#discussion_r205486499
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/collectionOperations.scala ---
@@ -3805,3 +3799,330 @@ object ArrayUnion {
new GenericArrayData(arrayBuffer)
}
}
+
+/**
+ * Returns an array of the elements in the intersect of x and y, without duplicates
+ */
+@ExpressionDescription(
+ usage = """
+ _FUNC_(array1, array2) - Returns an array of the elements in array1 but not in array2,
+ without duplicates.
+ """,
+ examples = """
+ Examples:
+ > SELECT _FUNC_(array(1, 2, 3), array(1, 3, 5));
+ array(2)
+ """,
+ since = "2.4.0")
+case class ArrayExcept(left: Expression, right: Expression) extends ArraySetLike {
+ override def dataType: DataType = left.dataType
+
+ var hsInt: OpenHashSet[Int] = _
+ var hsLong: OpenHashSet[Long] = _
+
+ def assignInt(array: ArrayData, idx: Int, resultArray: ArrayData, pos: Int): Boolean = {
+ val elem = array.getInt(idx)
+ if (!hsInt.contains(elem)) {
+ if (resultArray != null) {
+ resultArray.setInt(pos, elem)
+ }
+ hsInt.add(elem)
+ true
+ } else {
+ false
+ }
+ }
+
+ def assignLong(array: ArrayData, idx: Int, resultArray: ArrayData, pos: Int): Boolean = {
+ val elem = array.getLong(idx)
+ if (!hsLong.contains(elem)) {
+ if (resultArray != null) {
+ resultArray.setLong(pos, elem)
+ }
+ hsLong.add(elem)
+ true
+ } else {
+ false
+ }
+ }
+
+ def evalIntLongPrimitiveType(
+ array1: ArrayData,
+ array2: ArrayData,
+ resultArray: ArrayData,
+ isLongType: Boolean): Int = {
+ // store elements into resultArray
+ var notFoundNullElement = true
+ var i = 0
+ while (i < array2.numElements()) {
+ if (array2.isNullAt(i)) {
+ notFoundNullElement = false
+ } else {
+ val assigned = if (!isLongType) {
+ hsInt.add(array2.getInt(i))
+ } else {
+ hsLong.add(array2.getLong(i))
+ }
+ }
+ i += 1
+ }
+ var pos = 0
+ i = 0
+ while (i < array1.numElements()) {
+ if (array1.isNullAt(i)) {
+ if (notFoundNullElement) {
+ if (resultArray != null) {
+ resultArray.setNullAt(pos)
+ }
+ pos += 1
+ notFoundNullElement = false
+ }
+ } else {
+ val assigned = if (!isLongType) {
+ assignInt(array1, i, resultArray, pos)
+ } else {
+ assignLong(array1, i, resultArray, pos)
+ }
+ if (assigned) {
+ pos += 1
+ }
+ }
+ i += 1
+ }
+ pos
+ }
+
+ override def nullSafeEval(input1: Any, input2: Any): Any = {
+ val array1 = input1.asInstanceOf[ArrayData]
+ val array2 = input2.asInstanceOf[ArrayData]
+
+ if (elementTypeSupportEquals) {
+ elementType match {
+ case IntegerType =>
+ // avoid boxing of primitive int array elements
+ // calculate result array size
+ hsInt = new OpenHashSet[Int]
+ val elements = evalIntLongPrimitiveType(array1, array2, null, false)
+ // allocate result array
+ hsInt = new OpenHashSet[Int]
+ val resultArray = if (UnsafeArrayData.shouldUseGenericArrayData(
+ IntegerType.defaultSize, elements)) {
+ new GenericArrayData(new Array[Any](elements))
+ } else {
+ UnsafeArrayData.forPrimitiveArray(
+ Platform.INT_ARRAY_OFFSET, elements, IntegerType.defaultSize)
+ }
+ // assign elements into the result array
+ evalIntLongPrimitiveType(array1, array2, resultArray, false)
+ resultArray
+ case LongType =>
+ // avoid boxing of primitive long array elements
+ // calculate result array size
+ hsLong = new OpenHashSet[Long]
+ val elements = evalIntLongPrimitiveType(array1, array2, null, true)
+ // allocate result array
+ hsLong = new OpenHashSet[Long]
+ val resultArray = if (UnsafeArrayData.shouldUseGenericArrayData(
+ LongType.defaultSize, elements)) {
+ new GenericArrayData(new Array[Any](elements))
+ } else {
+ UnsafeArrayData.forPrimitiveArray(
+ Platform.LONG_ARRAY_OFFSET, elements, LongType.defaultSize)
+ }
+ // assign elements into the result array
+ evalIntLongPrimitiveType(array1, array2, resultArray, true)
+ resultArray
+ case _ =>
+ val hs = new OpenHashSet[Any]
+ var notFoundNullElement = true
+ var i = 0
+ while (i < array2.numElements()) {
+ if (array2.isNullAt(i)) {
+ notFoundNullElement = false
+ } else {
+ val elem = array2.get(i, elementType)
+ hs.add(elem)
+ }
+ i += 1
+ }
+ val arrayBuffer = new scala.collection.mutable.ArrayBuffer[Any]
+ i = 0
+ while (i < array1.numElements()) {
+ if (array1.isNullAt(i)) {
+ if (notFoundNullElement) {
+ arrayBuffer += null
+ notFoundNullElement = false
+ }
+ } else {
+ val elem = array1.get(i, elementType)
+ if (!hs.contains(elem)) {
+ arrayBuffer += elem
+ hs.add(elem)
+ }
+ }
+ i += 1
+ }
+ new GenericArrayData(arrayBuffer)
+ }
+ } else {
+ ArrayExcept.exceptOrdering(array1, array2, elementType, ordering)
+ }
+ }
+
+ override def doGenCode(ctx: CodegenContext, ev: ExprCode): ExprCode = {
--- End diff --
I don't think codegen can provide much value, given the complexity of this function. We do codegen mostly for supporting whole-stage-codegen. How about
```
val expr = ctx.addReference("arrayExceptExpr", this)
nullSafeCodeGen(ctx, ev, (array1, array2) => {
s"$expr. nullSafeEval($array1, $array2)"
}
"""
```
cc @rednaxelafx @rxin
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21103: [SPARK-23915][SQL] Add array_except function
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21103
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #21103: [SPARK-23915][SQL] Add array_except function
Posted by kiszk <gi...@git.apache.org>.
Github user kiszk commented on a diff in the pull request:
https://github.com/apache/spark/pull/21103#discussion_r204214620
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/collectionOperations.scala ---
@@ -3805,3 +3799,332 @@ object ArrayUnion {
new GenericArrayData(arrayBuffer)
}
}
+
+/**
+ * Returns an array of the elements in the intersect of x and y, without duplicates
+ */
+@ExpressionDescription(
+ usage = """
+ _FUNC_(array1, array2) - Returns an array of the elements in array1 but not in array2,
+ without duplicates.
+ """,
+ examples = """
+ Examples:Fun
--- End diff --
Yeah, good catch
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21103: [SPARK-23915][SQL] Add array_except function
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21103
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/93582/
Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21103: [SPARK-23915][SQL] Add array_except function
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21103
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21103: [SPARK-23915][SQL] Add array_except function
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21103
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/1385/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21103: [SPARK-23915][SQL] Add array_except function
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21103
Merged build finished. Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21103: [SPARK-23915][SQL] Add array_except function
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21103
**[Test build #93739 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/93739/testReport)** for PR 21103 at commit [`3639b5b`](https://github.com/apache/spark/commit/3639b5b1b460c19b7c3a787d118a0bcf35cb8e23).
* This patch **fails due to an unknown error code, -9**.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #21103: [SPARK-23915][SQL] Add array_except function
Posted by kiszk <gi...@git.apache.org>.
Github user kiszk commented on a diff in the pull request:
https://github.com/apache/spark/pull/21103#discussion_r205999688
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/collectionOperations.scala ---
@@ -3968,3 +3964,267 @@ object ArrayUnion {
new GenericArrayData(arrayBuffer)
}
}
+
+/**
+ * Returns an array of the elements in the intersect of x and y, without duplicates
+ */
+@ExpressionDescription(
+ usage = """
+ _FUNC_(array1, array2) - Returns an array of the elements in array1 but not in array2,
+ without duplicates.
+ """,
+ examples = """
+ Examples:
+ > SELECT _FUNC_(array(1, 2, 3), array(1, 3, 5));
+ array(2)
+ """,
+ since = "2.4.0")
+case class ArrayExcept(left: Expression, right: Expression) extends ArraySetLike
+ with ComplexTypeMergingExpression {
+ override def dataType: DataType = {
+ dataTypeCheck
+ left.dataType
+ }
+
+ @transient lazy val evalExcept: (ArrayData, ArrayData) => ArrayData = {
+ if (elementTypeSupportEquals) {
+ (array1, array2) =>
+ val hs = new OpenHashSet[Any]
+ var notFoundNullElement = true
+ var i = 0
+ while (i < array2.numElements()) {
+ if (array2.isNullAt(i)) {
+ notFoundNullElement = false
+ } else {
+ val elem = array2.get(i, elementType)
+ hs.add(elem)
+ }
+ i += 1
+ }
+ val arrayBuffer = new scala.collection.mutable.ArrayBuffer[Any]
+ i = 0
+ while (i < array1.numElements()) {
+ if (array1.isNullAt(i)) {
+ if (notFoundNullElement) {
+ arrayBuffer += null
+ notFoundNullElement = false
+ }
+ } else {
+ val elem = array1.get(i, elementType)
+ if (!hs.contains(elem)) {
+ arrayBuffer += elem
+ hs.add(elem)
+ }
+ }
+ i += 1
+ }
+ new GenericArrayData(arrayBuffer)
+ } else {
+ (array1, array2) =>
+ val arrayBuffer = new scala.collection.mutable.ArrayBuffer[Any]
+ var scannedNullElements = false
+ var i = 0
+ while (i < array1.numElements()) {
+ var found = false
+ val elem1 = array1.get(i, elementType)
+ if (elem1 == null) {
+ if (!scannedNullElements) {
+ var j = 0
+ while (!found && j < array2.numElements()) {
+ found = array2.isNullAt(j)
+ j += 1
+ }
+ // array2 is scanned only once for null element
+ scannedNullElements = true
+ } else {
+ found = true
+ }
+ } else {
+ var j = 0
+ while (!found && j < array2.numElements()) {
+ val elem2 = array2.get(j, elementType)
+ if (elem2 != null) {
+ found = ordering.equiv(elem1, elem2)
+ }
+ j += 1
+ }
+ if (!found) {
+ // check whether elem1 is already stored in arrayBuffer
+ var k = 0
+ while (!found && k < arrayBuffer.size) {
+ val va = arrayBuffer(k)
+ found = (va != null) && ordering.equiv(va, elem1)
+ k += 1
+ }
+ }
+ }
+ if (!found) {
+ arrayBuffer += elem1
+ }
+ i += 1
+ }
+ new GenericArrayData(arrayBuffer)
+ }
+ }
+
+ override def nullSafeEval(input1: Any, input2: Any): Any = {
+ val array1 = input1.asInstanceOf[ArrayData]
+ val array2 = input2.asInstanceOf[ArrayData]
+
+ evalExcept(array1, array2)
+ }
+
+ override def doGenCode(ctx: CodegenContext, ev: ExprCode): ExprCode = {
+ val arrayData = classOf[ArrayData].getName
+ val i = ctx.freshName("i")
+ val pos = ctx.freshName("pos")
+ val value = ctx.freshName("value")
+ val hsValue = ctx.freshName("hsValue")
+ val size = ctx.freshName("size")
+ val (postFix, openHashElementType, hsJavaTypeName, genHsValue,
+ getter, setter, javaTypeName, arrayBuilder) =
+ if (elementTypeSupportEquals) {
+ elementType match {
+ case BooleanType | ByteType | ShortType | IntegerType =>
+ val ptName = CodeGenerator.primitiveTypeName(elementType)
+ val unsafeArray = ctx.freshName("unsafeArray")
+ ("$mcI$sp", "Int", "int",
+ if (elementType != BooleanType) {
+ s"(int) $value"
+ } else {
+ s"$value ? 1 : 0;"
+ },
+ s"get$ptName($i)", s"set$ptName($pos, $value)", CodeGenerator.javaType(elementType),
+ s"""
+ |${ctx.createUnsafeArray(unsafeArray, size, elementType, s" $prettyName failed.")}
+ |${ev.value} = $unsafeArray;
+ """.stripMargin)
+ case LongType | FloatType | DoubleType =>
+ val ptName = CodeGenerator.primitiveTypeName(elementType)
+ val unsafeArray = ctx.freshName("unsafeArray")
+ val signature = elementType match {
+ case LongType => "$mcJ$sp"
+ case FloatType => "$mcF$sp"
+ case DoubleType => "$mcD$sp"
+ }
+ (signature, CodeGenerator.boxedType(elementType),
+ CodeGenerator.javaType(elementType), value,
+ s"get$ptName($i)", s"set$ptName($pos, $value)", CodeGenerator.javaType(elementType),
+ s"""
+ |${ctx.createUnsafeArray(unsafeArray, size, elementType, s" $prettyName failed.")}
+ |${ev.value} = $unsafeArray;
+ """.stripMargin)
+ case _ =>
+ val genericArrayData = classOf[GenericArrayData].getName
+ val et = ctx.addReferenceObj("elementType", elementType)
+ ("", "Object", "Object", value,
+ s"get($i, $et)", s"update($pos, $value)", "Object",
+ s"${ev.value} = new $genericArrayData(new Object[$size]);")
+ }
+ } else {
+ ("", "", "", "", "", "", "", "")
+ }
+
+ nullSafeCodeGen(ctx, ev, (array1, array2) => {
+ if (openHashElementType != "") {
+ // Here, we ensure elementTypeSupportEquals is true
+ val notFoundNullElement = ctx.freshName("notFoundNullElement")
+ val openHashSet = classOf[OpenHashSet[_]].getName
+ val classTag = s"scala.reflect.ClassTag$$.MODULE$$.$openHashElementType()"
+ val hs = ctx.freshName("hs")
+ val arrayData = classOf[ArrayData].getName
+ val arrays = ctx.freshName("arrays")
+ val array = ctx.freshName("array")
+ val arrayDataIdx = ctx.freshName("arrayDataIdx")
+
+ val array2NullCheck = if (right.dataType.asInstanceOf[ArrayType].containsNull) {
+ s"""
+ |if ($array2.isNullAt($i)) {
+ | $notFoundNullElement = false;
+ |} else
+ """.stripMargin
+ } else {
+ ""
+ }
+ val array1NullCheck = if (left.dataType.asInstanceOf[ArrayType].containsNull) {
+ s"""
+ |if ($array1.isNullAt($i)) {
+ | if ($notFoundNullElement) {
+ | $size++;
+ | $notFoundNullElement = false;
+ | }
+ |} else
+ """.stripMargin
+ } else {
+ ""
+ }
+ val array1NullAssignment = if (left.dataType.asInstanceOf[ArrayType].containsNull) {
+ s"""
+ |if ($array1.isNullAt($i)) {
+ | if ($notFoundNullElement) {
+ | ${ev.value}.setNullAt($pos++);
+ | $notFoundNullElement = false;
+ | }
+ |} else
+ """.stripMargin
+ } else {
+ ""
+ }
+
+ s"""
+ |$openHashSet $hs = new $openHashSet$postFix($classTag);
+ |boolean $notFoundNullElement = true;
+ |int $size = 0;
+ |for (int $i = 0; $i < $array2.numElements(); $i++) {
+ | $array2NullCheck
+ | {
+ | $javaTypeName $value = $array2.$getter;
+ | $hsJavaTypeName $hsValue = $genHsValue;
+ | $hs.add$postFix($hsValue);
--- End diff --
This part will be part of `else clause` of `if`` statement depends on `$array2NullCheck` (i.e. if `(right.dataType.asInstanceOf[ArrayType].containsNull)` is true).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21103: [SPARK-23915][SQL] Add array_except function
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21103
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21103: [SPARK-23915][SQL] Add array_except function
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21103
Merged build finished. Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21103: [SPARK-23915][SQL] Add array_except function
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21103
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21103: [SPARK-23915][SQL] Add array_except function
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21103
Merged build finished. Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21103: [SPARK-23915][SQL] Add array_except function
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21103
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/93759/
Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21103: [SPARK-23915][SQL] Add array_except function
Posted by kiszk <gi...@git.apache.org>.
Github user kiszk commented on the issue:
https://github.com/apache/spark/pull/21103
retest this please
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21103: [SPARK-23915][SQL] Add array_except function
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21103
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/93830/
Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21103: [SPARK-23915][SQL] Add array_except function
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21103
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/1424/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21103: [SPARK-23915][SQL] Add array_except function
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21103
**[Test build #89589 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/89589/testReport)** for PR 21103 at commit [`2551c09`](https://github.com/apache/spark/commit/2551c0989e1afdcd207ae7d680959b2a830a6624).
* This patch **fails PySpark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #21103: [SPARK-23915][SQL] Add array_except function
Posted by mn-mikke <gi...@git.apache.org>.
Github user mn-mikke commented on a diff in the pull request:
https://github.com/apache/spark/pull/21103#discussion_r205806876
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/collectionOperations.scala ---
@@ -3805,3 +3799,233 @@ object ArrayUnion {
new GenericArrayData(arrayBuffer)
}
}
+
+/**
+ * Returns an array of the elements in the intersect of x and y, without duplicates
+ */
+@ExpressionDescription(
+ usage = """
+ _FUNC_(array1, array2) - Returns an array of the elements in array1 but not in array2,
+ without duplicates.
+ """,
+ examples = """
+ Examples:
+ > SELECT _FUNC_(array(1, 2, 3), array(1, 3, 5));
+ array(2)
+ """,
+ since = "2.4.0")
+case class ArrayExcept(left: Expression, right: Expression) extends ArraySetLike {
--- End diff --
Maybe I'm missing something, but why we need to apply these checks if there won't be any ```null``` flag merging performed?
If ```left.dataType``` and ```right.dataType``` are different, will be casted according to the ```ImplicitTypeCasts``` coercion rule. If they differ only in ```null``` flags, ```left.dataType``` could be directly returned since there won't be any array elements from ```right``` present in the result.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21103: [SPARK-23915][SQL] Add array_except function
Posted by kiszk <gi...@git.apache.org>.
Github user kiszk commented on the issue:
https://github.com/apache/spark/pull/21103
retest this please
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21103: [SPARK-23915][SQL] Add array_except function
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21103
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21103: [SPARK-23915][SQL] Add array_except function
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21103
**[Test build #93822 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/93822/testReport)** for PR 21103 at commit [`49b5ab3`](https://github.com/apache/spark/commit/49b5ab371af9783be8f2d6351cf664a769957a4e).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21103: [SPARK-23915][SQL] Add array_except function
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21103
**[Test build #93001 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/93001/testReport)** for PR 21103 at commit [`5c06922`](https://github.com/apache/spark/commit/5c06922a9492dcb0032e1b60418e25fb57d6fdda).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds the following public classes _(experimental)_:
* `case class ArrayExcept(left: Expression, right: Expression) extends ArraySetLike `
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21103: [SPARK-23915][SQL] Add array_except function
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21103
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/2499/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21103: [SPARK-23915][SQL] Add array_except function
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21103
Merged build finished. Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21103: [SPARK-23915][SQL] Add array_except function
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21103
**[Test build #93831 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/93831/testReport)** for PR 21103 at commit [`72ef664`](https://github.com/apache/spark/commit/72ef66479842df6fe793fad39dfcb0f7e494b0fd).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21103: [SPARK-23915][SQL] Add array_except function
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21103
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21103: [SPARK-23915][SQL] Add array_except function
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21103
**[Test build #92322 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/92322/testReport)** for PR 21103 at commit [`ad9f576`](https://github.com/apache/spark/commit/ad9f576961384936548adbd38dfb60296d3f0389).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21103: [SPARK-23915][SQL] Add array_except function
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21103
Merged build finished. Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21103: [SPARK-23915][SQL] Add array_except function
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21103
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/93487/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21103: [SPARK-23915][SQL] Add array_except function
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21103
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21103: [SPARK-23915][SQL] Add array_except function
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21103
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21103: [SPARK-23915][SQL] Add array_except function
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21103
**[Test build #89587 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/89587/testReport)** for PR 21103 at commit [`2d16556`](https://github.com/apache/spark/commit/2d165560b4e58cf46dd6ecadf60ddab748b056f7).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21103: [SPARK-23915][SQL] Add array_except function
Posted by kiszk <gi...@git.apache.org>.
Github user kiszk commented on the issue:
https://github.com/apache/spark/pull/21103
cc @ueshin
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21103: [SPARK-23915][SQL] Add array_except function
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21103
Build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21103: [SPARK-23915][SQL] Add array_except function
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21103
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/950/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21103: [SPARK-23915][SQL] Add array_except function
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21103
**[Test build #93671 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/93671/testReport)** for PR 21103 at commit [`0b0a2ae`](https://github.com/apache/spark/commit/0b0a2ae3f9d6809e9f269a94182b27361bfc4563).
* This patch **fails Scala style tests**.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21103: [SPARK-23915][SQL] Add array_except function
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21103
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/1525/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21103: [SPARK-23915][SQL] Add array_except function
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21103
**[Test build #93644 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/93644/testReport)** for PR 21103 at commit [`ad0c318`](https://github.com/apache/spark/commit/ad0c318c1fb63bd08c69995d192ed3ab9a98e4c2).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21103: [SPARK-23915][SQL] Add array_except function
Posted by ueshin <gi...@git.apache.org>.
Github user ueshin commented on the issue:
https://github.com/apache/spark/pull/21103
Jenkins, retest this please.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #21103: [SPARK-23915][SQL] Add array_except function
Posted by mn-mikke <gi...@git.apache.org>.
Github user mn-mikke commented on a diff in the pull request:
https://github.com/apache/spark/pull/21103#discussion_r205392257
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/collectionOperations.scala ---
@@ -3805,3 +3799,330 @@ object ArrayUnion {
new GenericArrayData(arrayBuffer)
}
}
+
+/**
+ * Returns an array of the elements in the intersect of x and y, without duplicates
+ */
+@ExpressionDescription(
+ usage = """
+ _FUNC_(array1, array2) - Returns an array of the elements in array1 but not in array2,
+ without duplicates.
+ """,
+ examples = """
+ Examples:
+ > SELECT _FUNC_(array(1, 2, 3), array(1, 3, 5));
+ array(2)
+ """,
+ since = "2.4.0")
+case class ArrayExcept(left: Expression, right: Expression) extends ArraySetLike {
+ override def dataType: DataType = left.dataType
+
+ var hsInt: OpenHashSet[Int] = _
+ var hsLong: OpenHashSet[Long] = _
+
+ def assignInt(array: ArrayData, idx: Int, resultArray: ArrayData, pos: Int): Boolean = {
+ val elem = array.getInt(idx)
+ if (!hsInt.contains(elem)) {
+ if (resultArray != null) {
+ resultArray.setInt(pos, elem)
+ }
+ hsInt.add(elem)
+ true
+ } else {
+ false
+ }
+ }
+
+ def assignLong(array: ArrayData, idx: Int, resultArray: ArrayData, pos: Int): Boolean = {
+ val elem = array.getLong(idx)
+ if (!hsLong.contains(elem)) {
+ if (resultArray != null) {
+ resultArray.setLong(pos, elem)
+ }
+ hsLong.add(elem)
+ true
+ } else {
+ false
+ }
+ }
+
+ def evalIntLongPrimitiveType(
+ array1: ArrayData,
+ array2: ArrayData,
+ resultArray: ArrayData,
+ isLongType: Boolean): Int = {
+ // store elements into resultArray
+ var notFoundNullElement = true
+ var i = 0
+ while (i < array2.numElements()) {
+ if (array2.isNullAt(i)) {
+ notFoundNullElement = false
+ } else {
+ val assigned = if (!isLongType) {
+ hsInt.add(array2.getInt(i))
+ } else {
+ hsLong.add(array2.getLong(i))
+ }
+ }
+ i += 1
+ }
+ var pos = 0
+ i = 0
+ while (i < array1.numElements()) {
+ if (array1.isNullAt(i)) {
+ if (notFoundNullElement) {
+ if (resultArray != null) {
+ resultArray.setNullAt(pos)
+ }
+ pos += 1
+ notFoundNullElement = false
+ }
+ } else {
+ val assigned = if (!isLongType) {
+ assignInt(array1, i, resultArray, pos)
+ } else {
+ assignLong(array1, i, resultArray, pos)
+ }
+ if (assigned) {
+ pos += 1
+ }
+ }
+ i += 1
+ }
+ pos
+ }
+
+ override def nullSafeEval(input1: Any, input2: Any): Any = {
+ val array1 = input1.asInstanceOf[ArrayData]
+ val array2 = input2.asInstanceOf[ArrayData]
+
+ if (elementTypeSupportEquals) {
+ elementType match {
+ case IntegerType =>
+ // avoid boxing of primitive int array elements
+ // calculate result array size
+ hsInt = new OpenHashSet[Int]
+ val elements = evalIntLongPrimitiveType(array1, array2, null, false)
+ // allocate result array
+ hsInt = new OpenHashSet[Int]
+ val resultArray = if (UnsafeArrayData.shouldUseGenericArrayData(
+ IntegerType.defaultSize, elements)) {
+ new GenericArrayData(new Array[Any](elements))
+ } else {
+ UnsafeArrayData.forPrimitiveArray(
+ Platform.INT_ARRAY_OFFSET, elements, IntegerType.defaultSize)
+ }
+ // assign elements into the result array
+ evalIntLongPrimitiveType(array1, array2, resultArray, false)
+ resultArray
+ case LongType =>
+ // avoid boxing of primitive long array elements
+ // calculate result array size
+ hsLong = new OpenHashSet[Long]
+ val elements = evalIntLongPrimitiveType(array1, array2, null, true)
+ // allocate result array
+ hsLong = new OpenHashSet[Long]
+ val resultArray = if (UnsafeArrayData.shouldUseGenericArrayData(
+ LongType.defaultSize, elements)) {
+ new GenericArrayData(new Array[Any](elements))
+ } else {
+ UnsafeArrayData.forPrimitiveArray(
+ Platform.LONG_ARRAY_OFFSET, elements, LongType.defaultSize)
+ }
+ // assign elements into the result array
+ evalIntLongPrimitiveType(array1, array2, resultArray, true)
+ resultArray
+ case _ =>
+ val hs = new OpenHashSet[Any]
+ var notFoundNullElement = true
+ var i = 0
+ while (i < array2.numElements()) {
+ if (array2.isNullAt(i)) {
+ notFoundNullElement = false
+ } else {
+ val elem = array2.get(i, elementType)
+ hs.add(elem)
+ }
+ i += 1
+ }
+ val arrayBuffer = new scala.collection.mutable.ArrayBuffer[Any]
+ i = 0
+ while (i < array1.numElements()) {
+ if (array1.isNullAt(i)) {
+ if (notFoundNullElement) {
+ arrayBuffer += null
+ notFoundNullElement = false
+ }
+ } else {
+ val elem = array1.get(i, elementType)
+ if (!hs.contains(elem)) {
+ arrayBuffer += elem
+ hs.add(elem)
+ }
+ }
+ i += 1
+ }
+ new GenericArrayData(arrayBuffer)
+ }
+ } else {
+ ArrayExcept.exceptOrdering(array1, array2, elementType, ordering)
+ }
+ }
+
+ override def doGenCode(ctx: CodegenContext, ev: ExprCode): ExprCode = {
+ val i = ctx.freshName("i")
+ val pos = ctx.freshName("pos")
+ val value = ctx.freshName("value")
+ val size = ctx.freshName("size")
+ val (postFix, openHashElementType, getter, setter, javaTypeName, castOp, arrayBuilder) =
+ if (elementTypeSupportEquals) {
+ elementType match {
+ case ByteType | ShortType | IntegerType | LongType =>
+ val ptName = CodeGenerator.primitiveTypeName(elementType)
+ val unsafeArray = ctx.freshName("unsafeArray")
+ (if (elementType == LongType) s"$$mcJ$$sp" else s"$$mcI$$sp",
+ if (elementType == LongType) "Long" else "Int",
+ s"get$ptName($i)", s"set$ptName($pos, $value)", CodeGenerator.javaType(elementType),
+ if (elementType == LongType) "(long)" else "(int)",
+ s"""
+ |${ctx.createUnsafeArray(unsafeArray, size, elementType, s" $prettyName failed.")}
+ |${ev.value} = $unsafeArray;
+ """.stripMargin)
+ case _ =>
+ val genericArrayData = classOf[GenericArrayData].getName
+ val et = ctx.addReferenceObj("elementType", elementType)
+ ("", "Object",
+ s"get($i, $et)", s"update($pos, $value)", "Object", "",
+ s"${ev.value} = new $genericArrayData(new Object[$size]);")
+ }
+ } else {
+ ("", "", "", "", "", "", "")
+ }
+
+ nullSafeCodeGen(ctx, ev, (array1, array2) => {
+ if (openHashElementType != "") {
+ // Here, we ensure elementTypeSupportEquals is true
+ val notFoundNullElement = ctx.freshName("notFoundNullElement")
+ val openHashSet = classOf[OpenHashSet[_]].getName
+ val classTag = s"scala.reflect.ClassTag$$.MODULE$$.$openHashElementType()"
+ val hs = ctx.freshName("hs")
+ val arrayData = classOf[ArrayData].getName
+ val arrays = ctx.freshName("arrays")
+ val array = ctx.freshName("array")
+ val arrayDataIdx = ctx.freshName("arrayDataIdx")
+ s"""
+ |$openHashSet $hs = new $openHashSet$postFix($classTag);
+ |boolean $notFoundNullElement = true;
+ |int $size = 0;
+ |for (int $i = 0; $i < $array2.numElements(); $i++) {
+ | if ($array2.isNullAt($i)) {
+ | $notFoundNullElement = false;
+ | } else {
+ | $hs.add$postFix($array2.$getter);
+ | }
+ |}
+ |for (int $i = 0; $i < $array1.numElements(); $i++) {
+ | if ($array1.isNullAt($i)) {
+ | if ($notFoundNullElement) {
+ | $size++;
+ | $notFoundNullElement = false;
+ | }
+ | } else {
+ | $javaTypeName $value = $array1.$getter;
+ | if (!$hs.contains($castOp $value)) {
+ | $hs.add$postFix($value);
+ | $size++;
+ | }
+ | }
+ |}
+ |$arrayBuilder
+ |$hs = new $openHashSet$postFix($classTag);
+ |$notFoundNullElement = true;
+ |int $pos = 0;
+ |for (int $i = 0; $i < $array2.numElements(); $i++) {
+ | if ($array2.isNullAt($i)) {
+ | $notFoundNullElement = false;
+ | } else {
+ | $hs.add$postFix($array2.$getter);
+ | }
+ |}
+ |for (int $i = 0; $i < $array1.numElements(); $i++) {
+ | if ($array1.isNullAt($i)) {
+ | if ($notFoundNullElement) {
+ | ${ev.value}.setNullAt($pos++);
+ | $notFoundNullElement = false;
+ | }
+ | } else {
+ | $javaTypeName $value = $array1.$getter;
+ | if (!$hs.contains($castOp $value)) {
+ | $hs.add$postFix($value);
+ | ${ev.value}.$setter;
+ | $pos++;
+ | }
+ | }
+ |}
+ """.stripMargin
+ } else {
+ val arrayExcept = classOf[ArrayExcept].getName
+ val et = ctx.addReferenceObj("elementTypeIntersect", elementType)
+ val order = ctx.addReferenceObj("orderingIntersect", ordering)
+ val method = "exceptOrdering"
+ s"${ev.value} = $arrayExcept$$.MODULE$$.$method($array1, $array2, $et, $order);"
+ }
+ })
+ }
+
+ override def prettyName: String = "array_except"
+}
+
+object ArrayExcept {
+ def exceptOrdering(
+ array1: ArrayData,
+ array2: ArrayData,
+ elementType: DataType,
+ ordering: Ordering[Any]): ArrayData = {
+ val arrayBuffer = new scala.collection.mutable.ArrayBuffer[Any]
+ var scannedNullElements = false
+ var i = 0
+ while (i < array1.numElements()) {
+ var found = false
+ var elem1 = array1.get(i, elementType)
--- End diff --
nit: ```val```?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21103: [SPARK-23915][SQL] Add array_except function
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21103
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/89524/
Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21103: [SPARK-23915][SQL] Add array_except function
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21103
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #21103: [SPARK-23915][SQL] Add array_except function
Posted by mn-mikke <gi...@git.apache.org>.
Github user mn-mikke commented on a diff in the pull request:
https://github.com/apache/spark/pull/21103#discussion_r203741325
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/collectionOperations.scala ---
@@ -3805,3 +3799,332 @@ object ArrayUnion {
new GenericArrayData(arrayBuffer)
}
}
+
+/**
+ * Returns an array of the elements in the intersect of x and y, without duplicates
+ */
+@ExpressionDescription(
+ usage = """
+ _FUNC_(array1, array2) - Returns an array of the elements in array1 but not in array2,
+ without duplicates.
+ """,
+ examples = """
+ Examples:Fun
--- End diff --
just "Examples:"?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21103: [SPARK-23915][SQL] Add array_except function
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21103
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21103: [SPARK-23915][SQL] Add array_except function
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21103
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/1/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21103: [SPARK-23915][SQL] Add array_except function
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21103
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21103: [SPARK-23915][SQL] Add array_except function
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21103
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/1404/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org