You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2021/09/21 06:53:04 UTC
[GitHub] [spark] mkaravel opened a new pull request #34056: [WIP][SPARK-38611][SQL] Add SQL functions for the BINARY data type for AND, OR, XOR, and NOT
mkaravel opened a new pull request #34056:
URL: https://github.com/apache/spark/pull/34056
…, OR, XOR, NOT
<!--
Thanks for sending a pull request! Here are some tips for you:
1. If this is your first time, please read our contributor guidelines: https://spark.apache.org/contributing.html
2. Ensure you have added or run the appropriate tests for your PR: https://spark.apache.org/developer-tools.html
3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][SPARK-XXXX] Your PR title ...'.
4. Be sure to keep the PR description updated to reflect all changes.
5. Please write your PR title to summarize what this PR proposes.
6. If possible, provide a concise example to reproduce the issue for a faster review.
7. If you want to add a new configuration, please read the guideline first for naming configurations in
'core/src/main/scala/org/apache/spark/internal/config/ConfigEntry.scala'.
8. If you want to add or modify an error type or message, please read the guideline first in
'core/src/main/resources/error/README.md'.
-->
### What changes were proposed in this pull request?
This PR introduces four new SQL functions operating on the BINARY data type.
1. `bitand`: Takes as input two binary string and returns their bitwise AND.
2. `bitor`: Takes as input two binary string and returns their bitwise OR.
3. `bitxor`: Takes as input two binary string and returns their bitwise XOR.
4. `bitnot`: Takes as a binary string and returns its bitwise NOT.
For the first three functions the byte length of the result is the maximum of the byte length of the two inputs.
For the first three functions the shortest (in terms of byte length) binary string is semantically left-padded by zeros.
### Why are the changes needed?
These functions are useful for performing bitwise operations on `BINARY` values, seen as bit sets.
Other databases offer similar or the same functionality.
### Does this PR introduce _any_ user-facing change?
Yes. Four new SQL functions.
### How was this patch tested?
Unit tests.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34056: [SPARK-36811][SQL] Add SQL functions for the BINARY data type for AND, OR, XOR, and NOT
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34056:
URL: https://github.com/apache/spark/pull/34056#issuecomment-949850734
**[Test build #144545 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/144545/testReport)** for PR 34056 at commit [`769b245`](https://github.com/apache/spark/commit/769b245087e157fec2aec7adb874d6de71d15224).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34056: [SPARK-36811][SQL] Add SQL functions for the BINARY data type for AND, OR, XOR, and NOT
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34056:
URL: https://github.com/apache/spark/pull/34056#issuecomment-949843039
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/49014/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34056: [SPARK-36811][SQL] Add SQL functions for the BINARY data type for AND, OR, XOR, and NOT
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34056:
URL: https://github.com/apache/spark/pull/34056#issuecomment-953464275
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #34056: [SPARK-36811][SQL] Add SQL functions for the BINARY data type for AND, OR, XOR, and NOT
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #34056:
URL: https://github.com/apache/spark/pull/34056#issuecomment-953427402
**[Test build #144685 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/144685/testReport)** for PR 34056 at commit [`41a074e`](https://github.com/apache/spark/commit/41a074e24580a3727a4426f98025039e9c8706f7).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34056: [SPARK-36811][SQL] Add SQL functions for the BINARY data type for AND, OR, XOR, and NOT
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34056:
URL: https://github.com/apache/spark/pull/34056#issuecomment-953604675
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/144694/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34056: [SPARK-36811][SQL] Add SQL functions for the BINARY data type for AND, OR, XOR, and NOT
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34056:
URL: https://github.com/apache/spark/pull/34056#issuecomment-947543084
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34056: [SPARK-36811][SQL] Add SQL functions for the BINARY data type for AND, OR, XOR, and NOT
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34056:
URL: https://github.com/apache/spark/pull/34056#issuecomment-947157496
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/48902/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34056: [SPARK-36811][SQL] Add SQL functions for the BINARY data type for AND, OR, XOR, and NOT
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34056:
URL: https://github.com/apache/spark/pull/34056#issuecomment-947526732
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/48925/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #34056: [SPARK-36811][SQL] Add SQL functions for the BINARY data type for AND, OR, XOR, and NOT
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #34056:
URL: https://github.com/apache/spark/pull/34056#issuecomment-947374136
**[Test build #144446 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/144446/testReport)** for PR 34056 at commit [`d945b5e`](https://github.com/apache/spark/commit/d945b5ea549ebde395f74356e625898d1caceba1).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #34056: [SPARK-36811][SQL] Add SQL functions for the BINARY data type for AND, OR, XOR, and NOT
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #34056:
URL: https://github.com/apache/spark/pull/34056#issuecomment-928214139
**[Test build #143657 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/143657/testReport)** for PR 34056 at commit [`9339974`](https://github.com/apache/spark/commit/9339974fbfc9effded8c26cbdf47cfb55d197545).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] mkaravel commented on pull request #34056: [SPARK-36811][SQL] Add SQL functions for the BINARY data type for AND, OR, XOR, and NOT
Posted by GitBox <gi...@apache.org>.
mkaravel commented on pull request #34056:
URL: https://github.com/apache/spark/pull/34056#issuecomment-924366250
> > Other databases offer similar or the same functionality.
>
> @mkaravel Could you point out which databases have the same, please (with refs to docs).
Updated the PR description with details of what is supported in other systems.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] mkaravel commented on pull request #34056: [SPARK-36811][SQL] Add SQL functions for the BINARY data type for AND, OR, XOR, and NOT
Posted by GitBox <gi...@apache.org>.
mkaravel commented on pull request #34056:
URL: https://github.com/apache/spark/pull/34056#issuecomment-924225882
> Any particular reasons to not use the [BitSet](https://docs.oracle.com/javase/8/docs/api/java/util/BitSet.html) from JDK?
The way that the `BitSet` class works makes it unsuitable to use here:
* The `and`, `or`, and `xor` in the class produce a Bitset whose size (in bytes) seems to be the minimum of the sizes of the two inputs.
* Trailing zeros in the computed `BitSet` (when using `and`, `or`, or `xor` seem to be discarded after the operation.
* For different sized inputs the bits in `BitSet` are aligned to the left (most significant bit) which does not have the same semantics as the functions implemented in this PR.
For reference, here is the implementation that I used for implementing `bitwiseAnd`:
```java
public static byte[] bitwiseAnd(byte[] bytes1, byte[] bytes2) {
BitSet bs1 = BitSet.valueOf(bytes1);
BitSet bs2 = BitSet.valueOf(bytes2);
bs1.and(bs2);
return bs1.toByteArray();
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #34056: [SPARK-36811][SQL] Add SQL functions for the BINARY data type for AND, OR, XOR, and NOT
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #34056:
URL: https://github.com/apache/spark/pull/34056#issuecomment-924229310
**[Test build #143481 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/143481/testReport)** for PR 34056 at commit [`5b8d69f`](https://github.com/apache/spark/commit/5b8d69fac7f43a664ce2e2f74d125d44da8060eb).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #34056: [SPARK-36811][SQL] Add SQL functions for the BINARY data type for AND, OR, XOR, and NOT
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #34056:
URL: https://github.com/apache/spark/pull/34056#issuecomment-947411760
**[Test build #144452 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/144452/testReport)** for PR 34056 at commit [`fb7c39b`](https://github.com/apache/spark/commit/fb7c39b5a49a9d275e1b804c527480584611ea8b).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34056: [SPARK-36811][SQL] Add SQL functions for the BINARY data type for AND, OR, XOR, and NOT
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34056:
URL: https://github.com/apache/spark/pull/34056#issuecomment-947098804
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/144429/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34056: [SPARK-36811][SQL] Add SQL functions for the BINARY data type for AND, OR, XOR, and NOT
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34056:
URL: https://github.com/apache/spark/pull/34056#issuecomment-949843039
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/49014/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #34056: [SPARK-36811][SQL] Add SQL functions for the BINARY data type for AND, OR, XOR, and NOT
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #34056:
URL: https://github.com/apache/spark/pull/34056#issuecomment-949753092
**[Test build #144542 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/144542/testReport)** for PR 34056 at commit [`dbde475`](https://github.com/apache/spark/commit/dbde4750ae4fb23f0d317d99511ddeb105f7b184).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34056: [SPARK-36811][SQL] Add SQL functions for the BINARY data type for AND, OR, XOR, and NOT
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34056:
URL: https://github.com/apache/spark/pull/34056#issuecomment-953153962
**[Test build #144669 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/144669/testReport)** for PR 34056 at commit [`d1173bd`](https://github.com/apache/spark/commit/d1173bd4a0c43510660eabab7ea20b7b6e238aa8).
* This patch **fails Java style tests**.
* This patch **does not merge cleanly**.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34056: [SPARK-36811][SQL] Add SQL functions for the BINARY data type for AND, OR, XOR, and NOT
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34056:
URL: https://github.com/apache/spark/pull/34056#issuecomment-953473630
**[Test build #144685 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/144685/testReport)** for PR 34056 at commit [`41a074e`](https://github.com/apache/spark/commit/41a074e24580a3727a4426f98025039e9c8706f7).
* This patch **fails PySpark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34056: [SPARK-36811][SQL] Add SQL functions for the BINARY data type for AND, OR, XOR, and NOT
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34056:
URL: https://github.com/apache/spark/pull/34056#issuecomment-953456681
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/49154/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] mkaravel commented on a change in pull request #34056: [SPARK-36811][SQL] Add SQL functions for the BINARY data type for AND, OR, XOR, and NOT
Posted by GitBox <gi...@apache.org>.
mkaravel commented on a change in pull request #34056:
URL: https://github.com/apache/spark/pull/34056#discussion_r734631151
##########
File path: common/unsafe/src/main/java/org/apache/spark/unsafe/types/ByteArray.java
##########
@@ -123,4 +124,216 @@ static long getPrefix(Object base, long offset, int numBytes) {
}
return result;
}
+
+ // Constants used in the bitwiseAnd, bitwiseOr, and bitwiseXor methods below. They
+ // represent valid (case insensitive) values for the third argument of these methods.
+ private static final UTF8String LPAD_UTF8 = UTF8String.fromString("lpad");
+ private static final UTF8String RPAD_UTF8 = UTF8String.fromString("rpad");
+
+ // Return the bitwise AND of two byte sequences.
+ // This method is called when we call the BITAND SQL function. That function has the following
+ // behavior:
+ // - If the byte lengths of the two sequences are equal, the result is a byte sequence of the
+ // same length as the inputs and its content is the bitwise AND of the two inputs.
+ // - If the byte lengths are different, we expect a third string argument (constant) that
+ // indicates whether we should semantically pad (to the left or to the right) the shorter
+ // input to match the length of the longer input before proceeding with the bitwise AND
+ // operation. Padding in this case is done with zero bytes. Therefore, in this case, the
+ // byte length of the result is equal to the maximum byte length of the two inputs. The two
+ // acceptable values for the third argument are "lpad" and "rpad" (case insensitive). If the
+ // value is "lpad" we pad the shorter byte sequence from the left with zero bytes. If the
+ // value is "rpad" we pad the shorter byte sequence from the right with zero bytes.
+ // The fourth argument of this method indicates the number of arguments on the caller side (that
+ // is at the SQL function level). If the calling side used the two argument overload of the BITAND
+ // SQL function, we expect the inputs to be of the same byte length. If the calling side used the
+ // three argument overload of the BITAND SQL function, then we check that the string constant has
+ // a valid value, and based on that value we do the appropriate semantic padding.
+ public static byte[] bitwiseAnd(byte[] bytes1, byte[] bytes2, UTF8String padding,
+ boolean isTwoArgs) {
+ if (bytes1 == null || bytes2 == null || padding == null) return null;
+ final int len1 = bytes1.length;
+ final int len2 = bytes2.length;
+ if (isTwoArgs && len1 != len2) {
+ throw new IllegalArgumentException("Two-argument BITAND cannot operate on BINARY strings "
Review comment:
As we have discussed off github, the necessary classes are not visible in `common/unsafe`.
Will follow up with this either in this PR or in a separate one once the above is fixed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34056: [SPARK-36811][SQL] Add SQL functions for the BINARY data type for AND, OR, XOR, and NOT
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34056:
URL: https://github.com/apache/spark/pull/34056#issuecomment-949925506
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/144545/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #34056: [SPARK-36811][SQL] Add SQL functions for the BINARY data type for AND, OR, XOR, and NOT
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #34056:
URL: https://github.com/apache/spark/pull/34056#issuecomment-947086662
**[Test build #144429 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/144429/testReport)** for PR 34056 at commit [`925c2f5`](https://github.com/apache/spark/commit/925c2f5c5f1c23d28ac1f14dc953568fe79d5604).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34056: [SPARK-36811][SQL] Add SQL functions for the BINARY data type for AND, OR, XOR, and NOT
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34056:
URL: https://github.com/apache/spark/pull/34056#issuecomment-947211618
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/144430/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34056: [SPARK-36811][SQL] Add SQL functions for the BINARY data type for AND, OR, XOR, and NOT
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34056:
URL: https://github.com/apache/spark/pull/34056#issuecomment-947543084
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34056: [SPARK-36811][SQL] Add SQL functions for the BINARY data type for AND, OR, XOR, and NOT
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34056:
URL: https://github.com/apache/spark/pull/34056#issuecomment-947200097
**[Test build #144430 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/144430/testReport)** for PR 34056 at commit [`574a722`](https://github.com/apache/spark/commit/574a722665557a6b44b0811fbb2d42292b681032).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds the following public classes _(experimental)_:
* `public class JavaModuleOptions `
* `class HistogramPlotBase(NumericPlotBase):`
* `class KdePlotBase(NumericPlotBase):`
* ` new_class = type(NameTypeHolder.short_name, (NameTypeHolder,), `
* `class Database(NamedTuple):`
* `class Table(NamedTuple):`
* `class Column(NamedTuple):`
* `class Function(NamedTuple):`
* `class SparkUpgradeException(CapturedException):`
* ` protected class YarnSchedulerEndpoint(override val rpcEnv: RpcEnv)`
* `public class ExpressionImplUtils `
* `public final class TableIndex `
* `public abstract class Filter implements Expression, Serializable `
* `public class ColumnarBatch implements AutoCloseable `
* `public final class ColumnarBatchRow extends InternalRow `
* `class IndexAlreadyExistsException(message: String, cause: Option[Throwable] = None)`
* `class NoSuchIndexException(message: String, cause: Option[Throwable] = None)`
* `trait ExtractValue extends Expression with NullIntolerant `
* `case class AesEncrypt(input: Expression, key: Expression, child: Expression)`
* `case class AesDecrypt(input: Expression, key: Expression, child: Expression)`
* `case class AsOfJoin(`
* `case class SetCatalogAndNamespace(child: LogicalPlan) extends UnaryCommand `
* `case class CreateFunction(`
* `case class CreateView(`
* `case class SkewJoinChildWrapper(plan: SparkPlan) extends LeafExecNode `
* `case class SetCatalogCommand(catalogName: String) extends LeafRunnableCommand `
* `case class SetNamespaceCommand(namespace: Seq[String]) extends LeafRunnableCommand `
* `case class ShowCatalogsCommand(pattern: Option[String]) extends LeafRunnableCommand `
* `case class ShowCurrentNamespaceCommand() extends LeafRunnableCommand `
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34056: [SPARK-36811][SQL] Add SQL functions for the BINARY data type for AND, OR, XOR, and NOT
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34056:
URL: https://github.com/apache/spark/pull/34056#issuecomment-953563353
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/49163/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34056: [SPARK-36811][SQL] Add SQL functions for the BINARY data type for AND, OR, XOR, and NOT
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34056:
URL: https://github.com/apache/spark/pull/34056#issuecomment-953407090
**[Test build #144681 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/144681/testReport)** for PR 34056 at commit [`576b684`](https://github.com/apache/spark/commit/576b6840e1abbd1de9aeb8571c52c0f0fd095123).
* This patch **fails to build**.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34056: [SPARK-36811][SQL] Add SQL functions for the BINARY data type for AND, OR, XOR, and NOT
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34056:
URL: https://github.com/apache/spark/pull/34056#issuecomment-953464275
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34056: [SPARK-36811][SQL] Add SQL functions for the BINARY data type for AND, OR, XOR, and NOT
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34056:
URL: https://github.com/apache/spark/pull/34056#issuecomment-953175121
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/49139/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #34056: [SPARK-36811][SQL] Add SQL functions for the BINARY data type for AND, OR, XOR, and NOT
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #34056:
URL: https://github.com/apache/spark/pull/34056#issuecomment-953133504
**[Test build #144669 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/144669/testReport)** for PR 34056 at commit [`d1173bd`](https://github.com/apache/spark/commit/d1173bd4a0c43510660eabab7ea20b7b6e238aa8).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] github-actions[bot] commented on pull request #34056: [SPARK-36811][SQL] Add SQL functions for the BINARY data type for AND, OR, XOR, and NOT
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #34056:
URL: https://github.com/apache/spark/pull/34056#issuecomment-1052835642
We're closing this PR because it hasn't been updated in a while. This isn't a judgement on the merit of the PR in any way. It's just a way of keeping the PR queue manageable.
If you'd like to revive this PR, please reopen it and ask a committer to remove the Stale tag!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] MaxGekk commented on a change in pull request #34056: [SPARK-36811][SQL] Add SQL functions for the BINARY data type for AND, OR, XOR, and NOT
Posted by GitBox <gi...@apache.org>.
MaxGekk commented on a change in pull request #34056:
URL: https://github.com/apache/spark/pull/34056#discussion_r713182332
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/stringExpressions.scala
##########
@@ -2649,3 +2649,192 @@ case class Sentences(
copy(str = newFirst, language = newSecond, country = newThird)
}
+
+/**
+ * A function that returns the bitwise AND of two binary strings.
+ * The byte length of the result is the maximum of the byte lengths of the two input binary
+ * strings. If the two input binary strings are of different byte length they aligned according
+ * to their least significant (right-most) bit. The shorter binary string is semantically
+ * left-padded with zeros.
+ */
+@ExpressionDescription(
+ usage = """
+ _FUNC_(bytes1, bytes2) - Returns the bitwise AND of two binary strings.
+ """,
+ examples = """
+ Examples:
+ > SELECT hex(_FUNC_(unhex('AABB'), unhex('11223344')));
+ 00002200
+ """,
+ since = "3.3.0",
+ group = "string_funcs")
+case class BitAnd(bytes1: Expression, bytes2: Expression)
+ extends BinaryExpression with ExpectsInputTypes with NullIntolerant {
+
+ override def inputTypes: Seq[AbstractDataType] = Seq(BinaryType, BinaryType)
+
+ override def dataType: DataType = BinaryType
+
+ override def left: Expression = bytes1
+ override def right: Expression = bytes2
+
+ override def nullSafeEval(left: Any, right: Any): Any = {
+ bytes1.dataType match {
+ case BinaryType => ByteArray.bitwiseAnd(left.asInstanceOf[Array[Byte]],
Review comment:
You don't need to check the input type here since you specified the types in `override def inputTypes ...`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] MaxGekk commented on a change in pull request #34056: [SPARK-36811][SQL] Add SQL functions for the BINARY data type for AND, OR, XOR, and NOT
Posted by GitBox <gi...@apache.org>.
MaxGekk commented on a change in pull request #34056:
URL: https://github.com/apache/spark/pull/34056#discussion_r713178268
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/stringExpressions.scala
##########
@@ -2649,3 +2649,192 @@ case class Sentences(
copy(str = newFirst, language = newSecond, country = newThird)
}
+
+/**
+ * A function that returns the bitwise AND of two binary strings.
Review comment:
The class represents an expression till it is bound to a function name. I would replace `A function ...` -> `The expression ...`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34056: [SPARK-36811][SQL] Add SQL functions for the BINARY data type for AND, OR, XOR, and NOT
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34056:
URL: https://github.com/apache/spark/pull/34056#issuecomment-923697256
Can one of the admins verify this patch?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34056: [SPARK-36811][SQL] Add SQL functions for the BINARY data type for AND, OR, XOR, and NOT
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34056:
URL: https://github.com/apache/spark/pull/34056#issuecomment-947154448
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/48903/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #34056: [SPARK-36811][SQL] Add SQL functions for the BINARY data type for AND, OR, XOR, and NOT
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #34056:
URL: https://github.com/apache/spark/pull/34056#issuecomment-947126909
**[Test build #144430 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/144430/testReport)** for PR 34056 at commit [`574a722`](https://github.com/apache/spark/commit/574a722665557a6b44b0811fbb2d42292b681032).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34056: [SPARK-36811][SQL] Add SQL functions for the BINARY data type for AND, OR, XOR, and NOT
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34056:
URL: https://github.com/apache/spark/pull/34056#issuecomment-947374136
**[Test build #144446 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/144446/testReport)** for PR 34056 at commit [`d945b5e`](https://github.com/apache/spark/commit/d945b5ea549ebde395f74356e625898d1caceba1).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34056: [SPARK-36811][SQL] Add SQL functions for the BINARY data type for AND, OR, XOR, and NOT
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34056:
URL: https://github.com/apache/spark/pull/34056#issuecomment-947548644
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/144452/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #34056: [SPARK-36811][SQL] Add SQL functions for the BINARY data type for AND, OR, XOR, and NOT
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #34056:
URL: https://github.com/apache/spark/pull/34056#issuecomment-953186646
**[Test build #144673 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/144673/testReport)** for PR 34056 at commit [`7529678`](https://github.com/apache/spark/commit/752967891880fe6ebfffcfa0794f25ff2dc76480).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #34056: [SPARK-36811][SQL] Add SQL functions for the BINARY data type for AND, OR, XOR, and NOT
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #34056:
URL: https://github.com/apache/spark/pull/34056#issuecomment-953405664
**[Test build #144681 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/144681/testReport)** for PR 34056 at commit [`576b684`](https://github.com/apache/spark/commit/576b6840e1abbd1de9aeb8571c52c0f0fd095123).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34056: [SPARK-36811][SQL] Add SQL functions for the BINARY data type for AND, OR, XOR, and NOT
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34056:
URL: https://github.com/apache/spark/pull/34056#issuecomment-953427402
**[Test build #144685 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/144685/testReport)** for PR 34056 at commit [`41a074e`](https://github.com/apache/spark/commit/41a074e24580a3727a4426f98025039e9c8706f7).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34056: [SPARK-36811][SQL] Add SQL functions for the BINARY data type for AND, OR, XOR, and NOT
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34056:
URL: https://github.com/apache/spark/pull/34056#issuecomment-949793077
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/49014/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34056: [SPARK-36811][SQL] Add SQL functions for the BINARY data type for AND, OR, XOR, and NOT
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34056:
URL: https://github.com/apache/spark/pull/34056#issuecomment-949918757
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/49016/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34056: [SPARK-36811][SQL] Add SQL functions for the BINARY data type for AND, OR, XOR, and NOT
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34056:
URL: https://github.com/apache/spark/pull/34056#issuecomment-947098729
**[Test build #144429 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/144429/testReport)** for PR 34056 at commit [`925c2f5`](https://github.com/apache/spark/commit/925c2f5c5f1c23d28ac1f14dc953568fe79d5604).
* This patch **fails Java style tests**.
* This patch **does not merge cleanly**.
* This patch adds the following public classes _(experimental)_:
* `case class BitAnd(bytes1: Expression, bytes2: Expression, padding: Expression, numArgs: Int)`
* `case class BitOr(bytes1: Expression, bytes2: Expression, padding: Expression, numArgs: Int)`
* `case class BitXor(bytes1: Expression, bytes2: Expression, padding: Expression, numArgs: Int)`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] mkaravel commented on pull request #34056: [SPARK-36811][SQL] Add SQL functions for the BINARY data type for AND, OR, XOR, and NOT
Posted by GitBox <gi...@apache.org>.
mkaravel commented on pull request #34056:
URL: https://github.com/apache/spark/pull/34056#issuecomment-947009246
> > Other databases offer similar or the same functionality.
>
> @mkaravel Could you point out which databases have the same, please (with refs to docs).
Done in the description of the PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34056: [SPARK-36811][SQL] Add SQL functions for the BINARY data type for AND, OR, XOR, and NOT
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34056:
URL: https://github.com/apache/spark/pull/34056#issuecomment-947182779
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/48903/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] karenfeng commented on a change in pull request #34056: [SPARK-36811][SQL] Add SQL functions for the BINARY data type for AND, OR, XOR, and NOT
Posted by GitBox <gi...@apache.org>.
karenfeng commented on a change in pull request #34056:
URL: https://github.com/apache/spark/pull/34056#discussion_r732295436
##########
File path: common/unsafe/src/main/java/org/apache/spark/unsafe/types/ByteArray.java
##########
@@ -123,4 +124,216 @@ static long getPrefix(Object base, long offset, int numBytes) {
}
return result;
}
+
+ // Constants used in the bitwiseAnd, bitwiseOr, and bitwiseXor methods below. They
+ // represent valid (case insensitive) values for the third argument of these methods.
+ private static final UTF8String LPAD_UTF8 = UTF8String.fromString("lpad");
+ private static final UTF8String RPAD_UTF8 = UTF8String.fromString("rpad");
+
+ // Return the bitwise AND of two byte sequences.
+ // This method is called when we call the BITAND SQL function. That function has the following
+ // behavior:
+ // - If the byte lengths of the two sequences are equal, the result is a byte sequence of the
+ // same length as the inputs and its content is the bitwise AND of the two inputs.
+ // - If the byte lengths are different, we expect a third string argument (constant) that
+ // indicates whether we should semantically pad (to the left or to the right) the shorter
+ // input to match the length of the longer input before proceeding with the bitwise AND
+ // operation. Padding in this case is done with zero bytes. Therefore, in this case, the
+ // byte length of the result is equal to the maximum byte length of the two inputs. The two
+ // acceptable values for the third argument are "lpad" and "rpad" (case insensitive). If the
+ // value is "lpad" we pad the shorter byte sequence from the left with zero bytes. If the
+ // value is "rpad" we pad the shorter byte sequence from the right with zero bytes.
+ // The fourth argument of this method indicates the number of arguments on the caller side (that
+ // is at the SQL function level). If the calling side used the two argument overload of the BITAND
+ // SQL function, we expect the inputs to be of the same byte length. If the calling side used the
+ // three argument overload of the BITAND SQL function, then we check that the string constant has
+ // a valid value, and based on that value we do the appropriate semantic padding.
+ public static byte[] bitwiseAnd(byte[] bytes1, byte[] bytes2, UTF8String padding,
+ boolean isTwoArgs) {
+ if (bytes1 == null || bytes2 == null || padding == null) return null;
+ final int len1 = bytes1.length;
+ final int len2 = bytes2.length;
+ if (isTwoArgs && len1 != len2) {
+ throw new IllegalArgumentException("Two-argument BITAND cannot operate on BINARY strings "
Review comment:
To use the error framework, move these error messages to [error-classes.json](https://github.com/apache/spark/blob/master/core/src/main/resources/error/error-classes.json) and throw a `SparkIllegalArgumentException` instead. Concretely, the Scala code would look like:
```
throw new SparkIllegalArgumentException("BITWISE_ARGUMENT_BYTE_LENGTH_MISMATCH", Array("BITAND"))
```
and we'd add the following to the JSON:
```
"BITWISE_ARGUMENT_BYTE_LENGTH_MISMATCH" : {
"message" : "Two-argument %s cannot operate on BINARY strings with unequal byte length; use the three-argument overload instead.",
"sqlState" : "22023"
}
```
There's also a [README](https://github.com/apache/spark/blob/master/core/src/main/resources/error/README.md) that should help; let me know if there's anything else we should add there.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34056: [SPARK-36811][SQL] Add SQL functions for the BINARY data type for AND, OR, XOR, and NOT
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34056:
URL: https://github.com/apache/spark/pull/34056#issuecomment-953593907
**[Test build #144694 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/144694/testReport)** for PR 34056 at commit [`3e5ca86`](https://github.com/apache/spark/commit/3e5ca86557ee91bc4bbe56b1bc5606905d500c96).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34056: [SPARK-36811][SQL] Add SQL functions for the BINARY data type for AND, OR, XOR, and NOT
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34056:
URL: https://github.com/apache/spark/pull/34056#issuecomment-953229387
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/49139/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34056: [SPARK-36811][SQL] Add SQL functions for the BINARY data type for AND, OR, XOR, and NOT
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34056:
URL: https://github.com/apache/spark/pull/34056#issuecomment-953301640
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/49143/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34056: [SPARK-36811][SQL] Add SQL functions for the BINARY data type for AND, OR, XOR, and NOT
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34056:
URL: https://github.com/apache/spark/pull/34056#issuecomment-953133504
**[Test build #144669 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/144669/testReport)** for PR 34056 at commit [`d1173bd`](https://github.com/apache/spark/commit/d1173bd4a0c43510660eabab7ea20b7b6e238aa8).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34056: [SPARK-36811][SQL] Add SQL functions for the BINARY data type for AND, OR, XOR, and NOT
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34056:
URL: https://github.com/apache/spark/pull/34056#issuecomment-953209605
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/49139/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34056: [SPARK-36811][SQL] Add SQL functions for the BINARY data type for AND, OR, XOR, and NOT
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34056:
URL: https://github.com/apache/spark/pull/34056#issuecomment-953485655
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/144685/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34056: [SPARK-36811][SQL] Add SQL functions for the BINARY data type for AND, OR, XOR, and NOT
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34056:
URL: https://github.com/apache/spark/pull/34056#issuecomment-949918757
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/49016/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34056: [SPARK-36811][SQL] Add SQL functions for the BINARY data type for AND, OR, XOR, and NOT
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34056:
URL: https://github.com/apache/spark/pull/34056#issuecomment-928288900
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] mkaravel commented on a change in pull request #34056: [SPARK-36811][SQL] Add SQL functions for the BINARY data type for AND, OR, XOR, and NOT
Posted by GitBox <gi...@apache.org>.
mkaravel commented on a change in pull request #34056:
URL: https://github.com/apache/spark/pull/34056#discussion_r713280761
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/stringExpressions.scala
##########
@@ -2649,3 +2649,192 @@ case class Sentences(
copy(str = newFirst, language = newSecond, country = newThird)
}
+
+/**
+ * A function that returns the bitwise AND of two binary strings.
+ * The byte length of the result is the maximum of the byte lengths of the two input binary
+ * strings. If the two input binary strings are of different byte length they aligned according
+ * to their least significant (right-most) bit. The shorter binary string is semantically
+ * left-padded with zeros.
+ */
+@ExpressionDescription(
+ usage = """
+ _FUNC_(bytes1, bytes2) - Returns the bitwise AND of two binary strings.
+ """,
+ examples = """
+ Examples:
+ > SELECT hex(_FUNC_(unhex('AABB'), unhex('11223344')));
+ 00002200
+ """,
+ since = "3.3.0",
+ group = "string_funcs")
+case class BitAnd(bytes1: Expression, bytes2: Expression)
+ extends BinaryExpression with ExpectsInputTypes with NullIntolerant {
+
+ override def inputTypes: Seq[AbstractDataType] = Seq(BinaryType, BinaryType)
+
+ override def dataType: DataType = BinaryType
+
+ override def left: Expression = bytes1
+ override def right: Expression = bytes2
+
+ override def nullSafeEval(left: Any, right: Any): Any = {
+ bytes1.dataType match {
+ case BinaryType => ByteArray.bitwiseAnd(left.asInstanceOf[Array[Byte]],
Review comment:
Done.
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/stringExpressions.scala
##########
@@ -2649,3 +2649,192 @@ case class Sentences(
copy(str = newFirst, language = newSecond, country = newThird)
}
+
+/**
+ * A function that returns the bitwise AND of two binary strings.
Review comment:
Done. Referring to expressions now.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #34056: [SPARK-36811][SQL] Add SQL functions for the BINARY data type for AND, OR, XOR, and NOT
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #34056:
URL: https://github.com/apache/spark/pull/34056#issuecomment-928214139
**[Test build #143657 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/143657/testReport)** for PR 34056 at commit [`9339974`](https://github.com/apache/spark/commit/9339974fbfc9effded8c26cbdf47cfb55d197545).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34056: [SPARK-36811][SQL] Add SQL functions for the BINARY data type for AND, OR, XOR, and NOT
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34056:
URL: https://github.com/apache/spark/pull/34056#issuecomment-928288861
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/48170/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34056: [SPARK-36811][SQL] Add SQL functions for the BINARY data type for AND, OR, XOR, and NOT
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34056:
URL: https://github.com/apache/spark/pull/34056#issuecomment-928214139
**[Test build #143657 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/143657/testReport)** for PR 34056 at commit [`9339974`](https://github.com/apache/spark/commit/9339974fbfc9effded8c26cbdf47cfb55d197545).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34056: [SPARK-36811][SQL] Add SQL functions for the BINARY data type for AND, OR, XOR, and NOT
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34056:
URL: https://github.com/apache/spark/pull/34056#issuecomment-953405664
**[Test build #144681 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/144681/testReport)** for PR 34056 at commit [`576b684`](https://github.com/apache/spark/commit/576b6840e1abbd1de9aeb8571c52c0f0fd095123).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34056: [SPARK-36811][SQL] Add SQL functions for the BINARY data type for AND, OR, XOR, and NOT
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34056:
URL: https://github.com/apache/spark/pull/34056#issuecomment-953450524
**[Test build #144673 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/144673/testReport)** for PR 34056 at commit [`7529678`](https://github.com/apache/spark/commit/752967891880fe6ebfffcfa0794f25ff2dc76480).
* This patch **fails from timeout after a configured wait of `500m`**.
* This patch merges cleanly.
* This patch adds the following public classes _(experimental)_:
* `public class NumericHistogram `
* ` * The Coord class defines a histogram bin, which is just an (x,y) pair.`
* ` public static class Coord implements Comparable `
* `case class HistogramNumeric(`
* `case class CreateIndex(`
* `case class CreateIndexExec(`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34056: [SPARK-36811][SQL] Add SQL functions for the BINARY data type for AND, OR, XOR, and NOT
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34056:
URL: https://github.com/apache/spark/pull/34056#issuecomment-953443530
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/49154/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34056: [SPARK-36811][SQL] Add SQL functions for the BINARY data type for AND, OR, XOR, and NOT
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34056:
URL: https://github.com/apache/spark/pull/34056#issuecomment-947548644
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/144452/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34056: [SPARK-36811][SQL] Add SQL functions for the BINARY data type for AND, OR, XOR, and NOT
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34056:
URL: https://github.com/apache/spark/pull/34056#issuecomment-947452149
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/48919/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34056: [SPARK-36811][SQL] Add SQL functions for the BINARY data type for AND, OR, XOR, and NOT
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34056:
URL: https://github.com/apache/spark/pull/34056#issuecomment-947492108
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/48925/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #34056: [SPARK-36811][SQL] Add SQL functions for the BINARY data type for AND, OR, XOR, and NOT
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #34056:
URL: https://github.com/apache/spark/pull/34056#discussion_r752209652
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/stringExpressions.scala
##########
@@ -2734,3 +2734,236 @@ case class Sentences(
copy(str = newFirst, language = newSecond, country = newThird)
}
+
+/**
+ * Expression that returns the bitwise AND of two byte sequences.
+ * - If the byte lengths of the two input byte sequences are the same, the byte length of the
+ * result if the same as the common byte length of the two inputs.
+ * - If the byte lengths of the two input byte sequences differ, a third argument is required
+ * to determine whether the shorter input byte sequence should be semantically padded from the
+ * left or the right with zero bytes. The valid values for the third argument are 'lpad' and
+ * 'rpad' (case insensitive). If 'lpad' is specified, the shorter byte sequence is semantically
+ * left-padded with zeros to match the length of the longer byte sequence. If 'rpad' is
+ * specified, the shorter byte sequence is semantically right-padded with zeros to match the
+ * length of the longer byte sequence. The byte length of the result is the maximum of the
+ * byte lengths of the two input byte sequences.
+ * Specifying the third argument in the case of equal-length byte sequences has no effect.
+ */
+@ExpressionDescription(
+ usage = """
+ _FUNC_(bytes1, bytes2[, padding]) - Returns the bitwise AND of two binary strings.
+ """,
+ examples = """
+ Examples:
+ > SELECT hex(_FUNC_(unhex('AABB'), unhex('7735')));
+ 2231
+ > SELECT hex(_FUNC_(unhex('AABB'), unhex('66773355'), 'lpad'));
+ 00002211
+ > SELECT hex(_FUNC_(unhex('AABB'), unhex('66773355'), 'rpad'));
+ 22330000
+ """,
+ since = "3.3.0",
+ group = "string_funcs")
+case class BitAnd(bytes1: Expression, bytes2: Expression, padding: Expression, numArgs: Int)
Review comment:
I think it makes more sense to only allow the padding parameter to be a string constant, so that we can fail earlier (before execution) if the given value is not `lpad` or `rad`.
```
private lazy val isLPad = padding.eval().toString.equalsIgnoreCase("lpad")
private lazy val isRPad = ...
override def checkInputTypes... = {
...
if (!padding.foldable) TypeCheckResult.Fail... else {
val value = padding.eval()
if (value == null) TypeCheckResult.Fail... else {
if (isLPad || isRPad) TypeCheckResult.Success
else TypeCheckResult.Fail...
}
}
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34056: [SPARK-36811][SQL] Add SQL functions for the BINARY data type for AND, OR, XOR, and NOT
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34056:
URL: https://github.com/apache/spark/pull/34056#issuecomment-949763882
**[Test build #144542 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/144542/testReport)** for PR 34056 at commit [`dbde475`](https://github.com/apache/spark/commit/dbde4750ae4fb23f0d317d99511ddeb105f7b184).
* This patch **fails Java style tests**.
* This patch merges cleanly.
* This patch adds the following public classes _(experimental)_:
* `trait FunctionExpressionBuilder `
* `class SparkUDFExpressionBuilder extends FunctionExpressionBuilder `
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34056: [SPARK-36811][SQL] Add SQL functions for the BINARY data type for AND, OR, XOR, and NOT
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34056:
URL: https://github.com/apache/spark/pull/34056#issuecomment-949763992
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/144542/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34056: [SPARK-36811][SQL] Add SQL functions for the BINARY data type for AND, OR, XOR, and NOT
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34056:
URL: https://github.com/apache/spark/pull/34056#issuecomment-953229387
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/49139/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34056: [SPARK-36811][SQL] Add SQL functions for the BINARY data type for AND, OR, XOR, and NOT
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34056:
URL: https://github.com/apache/spark/pull/34056#issuecomment-953604675
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/144694/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34056: [SPARK-36811][SQL] Add SQL functions for the BINARY data type for AND, OR, XOR, and NOT
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34056:
URL: https://github.com/apache/spark/pull/34056#issuecomment-924339519
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/47992/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34056: [SPARK-36811][SQL] Add SQL functions for the BINARY data type for AND, OR, XOR, and NOT
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34056:
URL: https://github.com/apache/spark/pull/34056#issuecomment-924294303
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47992/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34056: [SPARK-36811][SQL] Add SQL functions for the BINARY data type for AND, OR, XOR, and NOT
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34056:
URL: https://github.com/apache/spark/pull/34056#issuecomment-924337767
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47992/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34056: [SPARK-36811][SQL] Add SQL functions for the BINARY data type for AND, OR, XOR, and NOT
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34056:
URL: https://github.com/apache/spark/pull/34056#issuecomment-947098804
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/144429/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34056: [SPARK-36811][SQL] Add SQL functions for the BINARY data type for AND, OR, XOR, and NOT
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34056:
URL: https://github.com/apache/spark/pull/34056#issuecomment-947139795
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/48902/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34056: [SPARK-36811][SQL] Add SQL functions for the BINARY data type for AND, OR, XOR, and NOT
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34056:
URL: https://github.com/apache/spark/pull/34056#issuecomment-947411760
**[Test build #144452 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/144452/testReport)** for PR 34056 at commit [`fb7c39b`](https://github.com/apache/spark/commit/fb7c39b5a49a9d275e1b804c527480584611ea8b).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34056: [SPARK-36811][SQL] Add SQL functions for the BINARY data type for AND, OR, XOR, and NOT
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34056:
URL: https://github.com/apache/spark/pull/34056#issuecomment-949924337
**[Test build #144545 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/144545/testReport)** for PR 34056 at commit [`769b245`](https://github.com/apache/spark/commit/769b245087e157fec2aec7adb874d6de71d15224).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34056: [SPARK-36811][SQL] Add SQL functions for the BINARY data type for AND, OR, XOR, and NOT
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34056:
URL: https://github.com/apache/spark/pull/34056#issuecomment-949925506
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/144545/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34056: [SPARK-36811][SQL] Add SQL functions for the BINARY data type for AND, OR, XOR, and NOT
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34056:
URL: https://github.com/apache/spark/pull/34056#issuecomment-947115220
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/48902/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34056: [SPARK-36811][SQL] Add SQL functions for the BINARY data type for AND, OR, XOR, and NOT
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34056:
URL: https://github.com/apache/spark/pull/34056#issuecomment-947452149
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/48919/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34056: [SPARK-36811][SQL] Add SQL functions for the BINARY data type for AND, OR, XOR, and NOT
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34056:
URL: https://github.com/apache/spark/pull/34056#issuecomment-953509357
**[Test build #144694 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/144694/testReport)** for PR 34056 at commit [`3e5ca86`](https://github.com/apache/spark/commit/3e5ca86557ee91bc4bbe56b1bc5606905d500c96).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #34056: [SPARK-36811][SQL] Add SQL functions for the BINARY data type for AND, OR, XOR, and NOT
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #34056:
URL: https://github.com/apache/spark/pull/34056#issuecomment-953509357
**[Test build #144694 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/144694/testReport)** for PR 34056 at commit [`3e5ca86`](https://github.com/apache/spark/commit/3e5ca86557ee91bc4bbe56b1bc5606905d500c96).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34056: [SPARK-36811][SQL] Add SQL functions for the BINARY data type for AND, OR, XOR, and NOT
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34056:
URL: https://github.com/apache/spark/pull/34056#issuecomment-953563599
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/49163/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34056: [SPARK-36811][SQL] Add SQL functions for the BINARY data type for AND, OR, XOR, and NOT
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34056:
URL: https://github.com/apache/spark/pull/34056#issuecomment-949763992
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/144542/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34056: [SPARK-36811][SQL] Add SQL functions for the BINARY data type for AND, OR, XOR, and NOT
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34056:
URL: https://github.com/apache/spark/pull/34056#issuecomment-947185992
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/48903/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34056: [SPARK-36811][SQL] Add SQL functions for the BINARY data type for AND, OR, XOR, and NOT
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34056:
URL: https://github.com/apache/spark/pull/34056#issuecomment-949753092
**[Test build #144542 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/144542/testReport)** for PR 34056 at commit [`dbde475`](https://github.com/apache/spark/commit/dbde4750ae4fb23f0d317d99511ddeb105f7b184).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34056: [SPARK-36811][SQL] Add SQL functions for the BINARY data type for AND, OR, XOR, and NOT
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34056:
URL: https://github.com/apache/spark/pull/34056#issuecomment-924339519
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/47992/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34056: [SPARK-36811][SQL] Add SQL functions for the BINARY data type for AND, OR, XOR, and NOT
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34056:
URL: https://github.com/apache/spark/pull/34056#issuecomment-924364274
**[Test build #143481 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/143481/testReport)** for PR 34056 at commit [`5b8d69f`](https://github.com/apache/spark/commit/5b8d69fac7f43a664ce2e2f74d125d44da8060eb).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34056: [SPARK-36811][SQL] Add SQL functions for the BINARY data type for AND, OR, XOR, and NOT
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34056:
URL: https://github.com/apache/spark/pull/34056#issuecomment-928214139
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34056: [SPARK-36811][SQL] Add SQL functions for the BINARY data type for AND, OR, XOR, and NOT
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34056:
URL: https://github.com/apache/spark/pull/34056#issuecomment-924379880
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/143481/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #34056: [SPARK-36811][SQL] Add SQL functions for the BINARY data type for AND, OR, XOR, and NOT
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #34056:
URL: https://github.com/apache/spark/pull/34056#discussion_r714944760
##########
File path: common/unsafe/src/main/java/org/apache/spark/unsafe/types/ByteArray.java
##########
@@ -101,4 +101,95 @@ public static long getPrefix(byte[] bytes) {
}
return result;
}
+
+ // Return the bitwise AND of two byte arrays. The byte length of the result is equal to the
+ // maximum byte length of the two inputs. The two input byte arrays are aligned with respect
+ // to their least significant (right-most) bytes.
+ public static byte[] bitwiseAnd(byte[] bytes1, byte[] bytes2) {
+ if (bytes1 == null || bytes2 == null) return null;
+ // Compute the length of the result (maximum of the lengths of the inputs).
+ final int len1 = bytes1.length;
+ final int len2 = bytes2.length;
+ final int maxLen = Math.max(len1, len2);
+ if (maxLen == 0) {
+ return EMPTY_BYTE;
+ }
+ final byte[] result = new byte[maxLen];
+ final int minLen = Math.min(len1, len2);
+ // Initialize the first `maxLen - minLen` bytes to 0.
+ Platform.setMemory(result, Platform.BYTE_ARRAY_OFFSET, maxLen - minLen, (byte)0);
+ // Compute the right-most minLen bytes of the result.
+ for (int j = 0; j < minLen; ++j) {
+ result[maxLen - 1 - j] = (byte)(bytes1[len1 - 1 - j] & bytes2[len2 - 1 - j]);
+ }
+ return result;
+ }
+
+ // Return the bitwise OR of two byte arrays. The byte length of the result is equal to the
+ // maximum byte length of the two inputs. The two input byte arrays are aligned with respect
+ // to their least significant (right-most) bytes.
+ public static byte[] bitwiseOr(byte[] bytes1, byte[] bytes2) {
+ if (bytes1 == null || bytes2 == null) return null;
+ // Compute the length of the result (maximum of the lengths of the inputs).
+ final int len1 = bytes1.length;
+ final int len2 = bytes2.length;
+ final int maxLen = Math.max(len1, len2);
+ if (maxLen == 0) {
+ return EMPTY_BYTE;
+ }
+ final byte[] result = new byte[maxLen];
+ final int minLen = Math.min(len1, len2);
+ // Copy the first `maxLen - minLen` bytes of the longer byte array into the result buffer.
+ final byte[] maxLenBytes = (len1 == maxLen) ? bytes1 : bytes2;
+ Platform.copyMemory(
+ maxLenBytes, Platform.BYTE_ARRAY_OFFSET,
+ result, Platform.BYTE_ARRAY_OFFSET,
+ maxLen - minLen);
Review comment:
is it a no-op if `maxLen - minLen` is 0? or do we need to add a `if-else` manually to avoid calling `copyMemory`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] mkaravel edited a comment on pull request #34056: [SPARK-36811][SQL] Add SQL functions for the BINARY data type for AND, OR, XOR, and NOT
Posted by GitBox <gi...@apache.org>.
mkaravel edited a comment on pull request #34056:
URL: https://github.com/apache/spark/pull/34056#issuecomment-924225882
> Any particular reasons to not use the [BitSet](https://docs.oracle.com/javase/8/docs/api/java/util/BitSet.html) from JDK?
The way that the `BitSet` class works makes it unsuitable to use here:
* The `and`, `or`, and `xor` in the class produce a Bitset whose size (in bytes) seems to be the minimum of the sizes of the two inputs.
* Trailing zeros in the computed `BitSet` (when using `and`, `or`, or `xor`) seem to be discarded after the operation.
* For different sized inputs the bits in `BitSet` are aligned to the left (most significant bit) which does not have the same semantics as the functions implemented in this PR.
For reference, here is the implementation that I used for implementing `bitwiseAnd`:
```java
public static byte[] bitwiseAnd(byte[] bytes1, byte[] bytes2) {
BitSet bs1 = BitSet.valueOf(bytes1);
BitSet bs2 = BitSet.valueOf(bytes2);
bs1.and(bs2);
return bs1.toByteArray();
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34056: [SPARK-36811][SQL] Add SQL functions for the BINARY data type for AND, OR, XOR, and NOT
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34056:
URL: https://github.com/apache/spark/pull/34056#issuecomment-924379880
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/143481/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34056: [SPARK-36811][SQL] Add SQL functions for the BINARY data type for AND, OR, XOR, and NOT
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34056:
URL: https://github.com/apache/spark/pull/34056#issuecomment-928361770
**[Test build #143657 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/143657/testReport)** for PR 34056 at commit [`9339974`](https://github.com/apache/spark/commit/9339974fbfc9effded8c26cbdf47cfb55d197545).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34056: [SPARK-36811][SQL] Add SQL functions for the BINARY data type for AND, OR, XOR, and NOT
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34056:
URL: https://github.com/apache/spark/pull/34056#issuecomment-947211618
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/144430/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34056: [SPARK-36811][SQL] Add SQL functions for the BINARY data type for AND, OR, XOR, and NOT
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34056:
URL: https://github.com/apache/spark/pull/34056#issuecomment-953406683
Kubernetes integration test unable to build dist.
exiting with code: 1
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/49150/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34056: [SPARK-36811][SQL] Add SQL functions for the BINARY data type for AND, OR, XOR, and NOT
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34056:
URL: https://github.com/apache/spark/pull/34056#issuecomment-953530725
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/49163/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34056: [SPARK-36811][SQL] Add SQL functions for the BINARY data type for AND, OR, XOR, and NOT
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34056:
URL: https://github.com/apache/spark/pull/34056#issuecomment-953421321
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34056: [SPARK-36811][SQL] Add SQL functions for the BINARY data type for AND, OR, XOR, and NOT
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34056:
URL: https://github.com/apache/spark/pull/34056#issuecomment-953184510
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/144669/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34056: [SPARK-36811][SQL] Add SQL functions for the BINARY data type for AND, OR, XOR, and NOT
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34056:
URL: https://github.com/apache/spark/pull/34056#issuecomment-953485655
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/144685/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34056: [SPARK-36811][SQL] Add SQL functions for the BINARY data type for AND, OR, XOR, and NOT
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34056:
URL: https://github.com/apache/spark/pull/34056#issuecomment-953563599
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/49163/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34056: [SPARK-36811][SQL] Add SQL functions for the BINARY data type for AND, OR, XOR, and NOT
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34056:
URL: https://github.com/apache/spark/pull/34056#issuecomment-947086662
**[Test build #144429 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/144429/testReport)** for PR 34056 at commit [`925c2f5`](https://github.com/apache/spark/commit/925c2f5c5f1c23d28ac1f14dc953568fe79d5604).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34056: [SPARK-36811][SQL] Add SQL functions for the BINARY data type for AND, OR, XOR, and NOT
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34056:
URL: https://github.com/apache/spark/pull/34056#issuecomment-947403868
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/48919/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34056: [SPARK-36811][SQL] Add SQL functions for the BINARY data type for AND, OR, XOR, and NOT
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34056:
URL: https://github.com/apache/spark/pull/34056#issuecomment-928249607
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/48170/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34056: [SPARK-36811][SQL] Add SQL functions for the BINARY data type for AND, OR, XOR, and NOT
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34056:
URL: https://github.com/apache/spark/pull/34056#issuecomment-928364076
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/143657/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34056: [WIP][SPARK-38611][SQL] Add SQL functions for the BINARY data type for AND, OR, XOR, and NOT
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34056:
URL: https://github.com/apache/spark/pull/34056#issuecomment-923697256
Can one of the admins verify this patch?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] MaxGekk commented on a change in pull request #34056: [SPARK-36811][SQL] Add SQL functions for the BINARY data type for AND, OR, XOR, and NOT
Posted by GitBox <gi...@apache.org>.
MaxGekk commented on a change in pull request #34056:
URL: https://github.com/apache/spark/pull/34056#discussion_r713178268
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/stringExpressions.scala
##########
@@ -2649,3 +2649,192 @@ case class Sentences(
copy(str = newFirst, language = newSecond, country = newThird)
}
+
+/**
+ * A function that returns the bitwise AND of two binary strings.
Review comment:
The class represents an expression till it is not bound to a function name. I would replace `A function ...` -> `The expression ...`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] gatorsmile commented on pull request #34056: [SPARK-36811][SQL] Add SQL functions for the BINARY data type for AND, OR, XOR, and NOT
Posted by GitBox <gi...@apache.org>.
gatorsmile commented on pull request #34056:
URL: https://github.com/apache/spark/pull/34056#issuecomment-925296417
cc @cloud-fan
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] mkaravel commented on a change in pull request #34056: [SPARK-36811][SQL] Add SQL functions for the BINARY data type for AND, OR, XOR, and NOT
Posted by GitBox <gi...@apache.org>.
mkaravel commented on a change in pull request #34056:
URL: https://github.com/apache/spark/pull/34056#discussion_r716989650
##########
File path: common/unsafe/src/main/java/org/apache/spark/unsafe/types/ByteArray.java
##########
@@ -101,4 +101,95 @@ public static long getPrefix(byte[] bytes) {
}
return result;
}
+
+ // Return the bitwise AND of two byte arrays. The byte length of the result is equal to the
+ // maximum byte length of the two inputs. The two input byte arrays are aligned with respect
+ // to their least significant (right-most) bytes.
+ public static byte[] bitwiseAnd(byte[] bytes1, byte[] bytes2) {
+ if (bytes1 == null || bytes2 == null) return null;
+ // Compute the length of the result (maximum of the lengths of the inputs).
+ final int len1 = bytes1.length;
+ final int len2 = bytes2.length;
+ final int maxLen = Math.max(len1, len2);
+ if (maxLen == 0) {
+ return EMPTY_BYTE;
+ }
+ final byte[] result = new byte[maxLen];
+ final int minLen = Math.min(len1, len2);
+ // Initialize the first `maxLen - minLen` bytes to 0.
+ Platform.setMemory(result, Platform.BYTE_ARRAY_OFFSET, maxLen - minLen, (byte)0);
+ // Compute the right-most minLen bytes of the result.
+ for (int j = 0; j < minLen; ++j) {
+ result[maxLen - 1 - j] = (byte)(bytes1[len1 - 1 - j] & bytes2[len2 - 1 - j]);
+ }
+ return result;
+ }
+
+ // Return the bitwise OR of two byte arrays. The byte length of the result is equal to the
+ // maximum byte length of the two inputs. The two input byte arrays are aligned with respect
+ // to their least significant (right-most) bytes.
+ public static byte[] bitwiseOr(byte[] bytes1, byte[] bytes2) {
+ if (bytes1 == null || bytes2 == null) return null;
+ // Compute the length of the result (maximum of the lengths of the inputs).
+ final int len1 = bytes1.length;
+ final int len2 = bytes2.length;
+ final int maxLen = Math.max(len1, len2);
+ if (maxLen == 0) {
+ return EMPTY_BYTE;
+ }
+ final byte[] result = new byte[maxLen];
+ final int minLen = Math.min(len1, len2);
+ // Copy the first `maxLen - minLen` bytes of the longer byte array into the result buffer.
+ final byte[] maxLenBytes = (len1 == maxLen) ? bytes1 : bytes2;
+ Platform.copyMemory(
+ maxLenBytes, Platform.BYTE_ARRAY_OFFSET,
+ result, Platform.BYTE_ARRAY_OFFSET,
+ maxLen - minLen);
Review comment:
Yes, it is a no-op. See the code here: https://github.com/apache/spark/blob/d03999ab8846d4897d2ce95ca21a7feed45f292b/common/unsafe/src/main/java/org/apache/spark/unsafe/Platform.java#L249
I therefore think there is no need to add an explicit `if-else`. I have also added a few more test cases with equal-length inputs that cover this case.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34056: [SPARK-36811][SQL] Add SQL functions for the BINARY data type for AND, OR, XOR, and NOT
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34056:
URL: https://github.com/apache/spark/pull/34056#issuecomment-928288900
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/48170/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34056: [SPARK-36811][SQL] Add SQL functions for the BINARY data type for AND, OR, XOR, and NOT
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34056:
URL: https://github.com/apache/spark/pull/34056#issuecomment-949834960
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/49014/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34056: [SPARK-36811][SQL] Add SQL functions for the BINARY data type for AND, OR, XOR, and NOT
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34056:
URL: https://github.com/apache/spark/pull/34056#issuecomment-953421321
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34056: [SPARK-36811][SQL] Add SQL functions for the BINARY data type for AND, OR, XOR, and NOT
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34056:
URL: https://github.com/apache/spark/pull/34056#issuecomment-953262147
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/49143/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34056: [SPARK-36811][SQL] Add SQL functions for the BINARY data type for AND, OR, XOR, and NOT
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34056:
URL: https://github.com/apache/spark/pull/34056#issuecomment-953184510
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/144669/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34056: [SPARK-36811][SQL] Add SQL functions for the BINARY data type for AND, OR, XOR, and NOT
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34056:
URL: https://github.com/apache/spark/pull/34056#issuecomment-953186646
**[Test build #144673 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/144673/testReport)** for PR 34056 at commit [`7529678`](https://github.com/apache/spark/commit/752967891880fe6ebfffcfa0794f25ff2dc76480).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34056: [SPARK-36811][SQL] Add SQL functions for the BINARY data type for AND, OR, XOR, and NOT
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34056:
URL: https://github.com/apache/spark/pull/34056#issuecomment-947126909
**[Test build #144430 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/144430/testReport)** for PR 34056 at commit [`574a722`](https://github.com/apache/spark/commit/574a722665557a6b44b0811fbb2d42292b681032).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34056: [SPARK-36811][SQL] Add SQL functions for the BINARY data type for AND, OR, XOR, and NOT
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34056:
URL: https://github.com/apache/spark/pull/34056#issuecomment-947447352
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/48919/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34056: [SPARK-36811][SQL] Add SQL functions for the BINARY data type for AND, OR, XOR, and NOT
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34056:
URL: https://github.com/apache/spark/pull/34056#issuecomment-947527767
**[Test build #144446 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/144446/testReport)** for PR 34056 at commit [`d945b5e`](https://github.com/apache/spark/commit/d945b5ea549ebde395f74356e625898d1caceba1).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] github-actions[bot] closed pull request #34056: [SPARK-36811][SQL] Add SQL functions for the BINARY data type for AND, OR, XOR, and NOT
Posted by GitBox <gi...@apache.org>.
github-actions[bot] closed pull request #34056:
URL: https://github.com/apache/spark/pull/34056
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34056: [SPARK-36811][SQL] Add SQL functions for the BINARY data type for AND, OR, XOR, and NOT
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34056:
URL: https://github.com/apache/spark/pull/34056#issuecomment-949882097
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/49016/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #34056: [SPARK-36811][SQL] Add SQL functions for the BINARY data type for AND, OR, XOR, and NOT
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #34056:
URL: https://github.com/apache/spark/pull/34056#issuecomment-949850734
**[Test build #144545 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/144545/testReport)** for PR 34056 at commit [`769b245`](https://github.com/apache/spark/commit/769b245087e157fec2aec7adb874d6de71d15224).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34056: [SPARK-36811][SQL] Add SQL functions for the BINARY data type for AND, OR, XOR, and NOT
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34056:
URL: https://github.com/apache/spark/pull/34056#issuecomment-949908519
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/49016/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] mkaravel commented on a change in pull request #34056: [SPARK-36811][SQL] Add SQL functions for the BINARY data type for AND, OR, XOR, and NOT
Posted by GitBox <gi...@apache.org>.
mkaravel commented on a change in pull request #34056:
URL: https://github.com/apache/spark/pull/34056#discussion_r716989650
##########
File path: common/unsafe/src/main/java/org/apache/spark/unsafe/types/ByteArray.java
##########
@@ -101,4 +101,95 @@ public static long getPrefix(byte[] bytes) {
}
return result;
}
+
+ // Return the bitwise AND of two byte arrays. The byte length of the result is equal to the
+ // maximum byte length of the two inputs. The two input byte arrays are aligned with respect
+ // to their least significant (right-most) bytes.
+ public static byte[] bitwiseAnd(byte[] bytes1, byte[] bytes2) {
+ if (bytes1 == null || bytes2 == null) return null;
+ // Compute the length of the result (maximum of the lengths of the inputs).
+ final int len1 = bytes1.length;
+ final int len2 = bytes2.length;
+ final int maxLen = Math.max(len1, len2);
+ if (maxLen == 0) {
+ return EMPTY_BYTE;
+ }
+ final byte[] result = new byte[maxLen];
+ final int minLen = Math.min(len1, len2);
+ // Initialize the first `maxLen - minLen` bytes to 0.
+ Platform.setMemory(result, Platform.BYTE_ARRAY_OFFSET, maxLen - minLen, (byte)0);
+ // Compute the right-most minLen bytes of the result.
+ for (int j = 0; j < minLen; ++j) {
+ result[maxLen - 1 - j] = (byte)(bytes1[len1 - 1 - j] & bytes2[len2 - 1 - j]);
+ }
+ return result;
+ }
+
+ // Return the bitwise OR of two byte arrays. The byte length of the result is equal to the
+ // maximum byte length of the two inputs. The two input byte arrays are aligned with respect
+ // to their least significant (right-most) bytes.
+ public static byte[] bitwiseOr(byte[] bytes1, byte[] bytes2) {
+ if (bytes1 == null || bytes2 == null) return null;
+ // Compute the length of the result (maximum of the lengths of the inputs).
+ final int len1 = bytes1.length;
+ final int len2 = bytes2.length;
+ final int maxLen = Math.max(len1, len2);
+ if (maxLen == 0) {
+ return EMPTY_BYTE;
+ }
+ final byte[] result = new byte[maxLen];
+ final int minLen = Math.min(len1, len2);
+ // Copy the first `maxLen - minLen` bytes of the longer byte array into the result buffer.
+ final byte[] maxLenBytes = (len1 == maxLen) ? bytes1 : bytes2;
+ Platform.copyMemory(
+ maxLenBytes, Platform.BYTE_ARRAY_OFFSET,
+ result, Platform.BYTE_ARRAY_OFFSET,
+ maxLen - minLen);
Review comment:
Yes, it is a no-op. See the code here: https://github.com/apache/spark/blob/d03999ab8846d4897d2ce95ca21a7feed45f292b/common/unsafe/src/main/java/org/apache/spark/unsafe/Platform.java#L249
I therefore think there is no need to add an explicit `if-else`. I have also added a few more test cases with equal-length inputs that cover this case.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34056: [SPARK-36811][SQL] Add SQL functions for the BINARY data type for AND, OR, XOR, and NOT
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34056:
URL: https://github.com/apache/spark/pull/34056#issuecomment-928364076
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/143657/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34056: [SPARK-36811][SQL] Add SQL functions for the BINARY data type for AND, OR, XOR, and NOT
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34056:
URL: https://github.com/apache/spark/pull/34056#issuecomment-928288900
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34056: [SPARK-36811][SQL] Add SQL functions for the BINARY data type for AND, OR, XOR, and NOT
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34056:
URL: https://github.com/apache/spark/pull/34056#issuecomment-928288900
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/48170/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34056: [SPARK-36811][SQL] Add SQL functions for the BINARY data type for AND, OR, XOR, and NOT
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34056:
URL: https://github.com/apache/spark/pull/34056#issuecomment-947185992
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/48903/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34056: [SPARK-36811][SQL] Add SQL functions for the BINARY data type for AND, OR, XOR, and NOT
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34056:
URL: https://github.com/apache/spark/pull/34056#issuecomment-947547121
**[Test build #144452 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/144452/testReport)** for PR 34056 at commit [`fb7c39b`](https://github.com/apache/spark/commit/fb7c39b5a49a9d275e1b804c527480584611ea8b).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34056: [SPARK-36811][SQL] Add SQL functions for the BINARY data type for AND, OR, XOR, and NOT
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34056:
URL: https://github.com/apache/spark/pull/34056#issuecomment-947157496
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/48902/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34056: [SPARK-36811][SQL] Add SQL functions for the BINARY data type for AND, OR, XOR, and NOT
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34056:
URL: https://github.com/apache/spark/pull/34056#issuecomment-953313861
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/49143/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34056: [SPARK-36811][SQL] Add SQL functions for the BINARY data type for AND, OR, XOR, and NOT
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34056:
URL: https://github.com/apache/spark/pull/34056#issuecomment-953313861
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/49143/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] gatorsmile commented on pull request #34056: [SPARK-36811][SQL] Add SQL functions for the BINARY data type for AND, OR, XOR, and NOT
Posted by GitBox <gi...@apache.org>.
gatorsmile commented on pull request #34056:
URL: https://github.com/apache/spark/pull/34056#issuecomment-924207306
add to whitelist
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] MaxGekk commented on pull request #34056: [SPARK-36811][SQL] Add SQL functions for the BINARY data type for AND, OR, XOR, and NOT
Posted by GitBox <gi...@apache.org>.
MaxGekk commented on pull request #34056:
URL: https://github.com/apache/spark/pull/34056#issuecomment-924136579
> Other databases offer similar or the same functionality.
@mkaravel Could you point out which databases have the same, please (with refs to docs).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34056: [SPARK-36811][SQL] Add SQL functions for the BINARY data type for AND, OR, XOR, and NOT
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34056:
URL: https://github.com/apache/spark/pull/34056#issuecomment-924229310
**[Test build #143481 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/143481/testReport)** for PR 34056 at commit [`5b8d69f`](https://github.com/apache/spark/commit/5b8d69fac7f43a664ce2e2f74d125d44da8060eb).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org