You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2021/12/08 16:06:54 UTC
[GitHub] [spark] MaxGekk opened a new pull request #34837: [SPARK-37586][SQL] Add the `mode` and `padding` args to `aes_encrypt()`/`aes_decrypt()`
MaxGekk opened a new pull request #34837:
URL: https://github.com/apache/spark/pull/34837
### What changes were proposed in this pull request?
In the PR, I propose to add new optional arguments to the `aes_encrypt()` and `aes_decrypt()` functions with default values:
1. `mode` - specifies which block cipher mode should be used to encrypt/decrypt messages. Only one valid value is `ECB` at the moment.
2. `padding` - specifies how to pad messages whose length is not a multiple of the block size. Only one valid value is `PKCS`.
In this way, when an user doesn't pass `mode`/`padding` to the functions, the functions apply AES encryption/decryption in the `ECB` mode with the `PKCS5Padding` padding.
### Why are the changes needed?
1. At the moment, `aes_encrypt()` and `aes_decrypt()` rely on the jvm's configuration regarding which cipher mode to support, this is problematic as it is not fixed across versions and systems. By using default constants for new arguments, we can guarantee the same behaviour across all supported environments.
2. We can consider new arguments as new point of extension in the current implementation of AES algorithm in Spark SQL. In the future or in private forks, Spark devs can implement other modes and paddings like GCM.
### Does this PR introduce _any_ user-facing change?
No. This PR just extends existing APIs.
### How was this patch tested?
By running new checks:
```
$ build/sbt "test:testOnly org.apache.spark.sql.DataFrameFunctionsSuite"
$ build/sbt "sql/test:testOnly org.apache.spark.sql.expressions.ExpressionInfoSuite"
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34837: [SPARK-37586][SQL] Add the `mode` and `padding` args to `aes_encrypt()`/`aes_decrypt()`
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34837:
URL: https://github.com/apache/spark/pull/34837#issuecomment-989102822
**[Test build #146012 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/146012/testReport)** for PR 34837 at commit [`3726ba7`](https://github.com/apache/spark/commit/3726ba7e401ec0e6f68c38bb54309662ac9df1ee).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34837: [SPARK-37586][SQL] Add the `mode` and `padding` args to `aes_encrypt()`/`aes_decrypt()`
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34837:
URL: https://github.com/apache/spark/pull/34837#issuecomment-989241399
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/50490/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #34837: [SPARK-37586][SQL] Add the `mode` and `padding` args to `aes_encrypt()`/`aes_decrypt()`
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #34837:
URL: https://github.com/apache/spark/pull/34837#issuecomment-988952349
**[Test build #146011 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/146011/testReport)** for PR 34837 at commit [`f21b0ed`](https://github.com/apache/spark/commit/f21b0ed77af77d5e50856279ab572b9206994f0b).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34837: [SPARK-37586][SQL] Add the `mode` and `padding` args to `aes_encrypt()`/`aes_decrypt()`
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34837:
URL: https://github.com/apache/spark/pull/34837#issuecomment-989205788
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/50490/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34837: [SPARK-37586][SQL] Add the `mode` and `padding` args to `aes_encrypt()`/`aes_decrypt()`
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34837:
URL: https://github.com/apache/spark/pull/34837#issuecomment-989137585
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/50488/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #34837: [SPARK-37586][SQL] Add the `mode` and `padding` args to `aes_encrypt()`/`aes_decrypt()`
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #34837:
URL: https://github.com/apache/spark/pull/34837#issuecomment-989148761
**[Test build #146014 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/146014/testReport)** for PR 34837 at commit [`a386388`](https://github.com/apache/spark/commit/a386388cd2bb09bbf1c4205fb8722a25afb68da6).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34837: [SPARK-37586][SQL] Add the `mode` and `padding` args to `aes_encrypt()`/`aes_decrypt()`
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34837:
URL: https://github.com/apache/spark/pull/34837#issuecomment-988992075
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/50487/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34837: [SPARK-37586][SQL] Add the `mode` and `padding` args to `aes_encrypt()`/`aes_decrypt()`
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34837:
URL: https://github.com/apache/spark/pull/34837#issuecomment-989350021
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/146014/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] sarutak edited a comment on pull request #34837: [SPARK-37586][SQL] Add the `mode` and `padding` args to `aes_encrypt()`/`aes_decrypt()`
Posted by GitBox <gi...@apache.org>.
sarutak edited a comment on pull request #34837:
URL: https://github.com/apache/spark/pull/34837#issuecomment-989526983
> Seems like everyone has its own syntax. Not sure we should be compatible with MySQL in particular.
Yes, there is no standard syntax but the name `aes_encrypt` and `aes_decrypt` apparently come from MySQL so users might expect the same usage.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34837: [SPARK-37586][SQL] Add the `mode` and `padding` args to `aes_encrypt()`/`aes_decrypt()`
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34837:
URL: https://github.com/apache/spark/pull/34837#issuecomment-989052426
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/146011/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] MaxGekk edited a comment on pull request #34837: [SPARK-37586][SQL] Add the `mode` and `padding` args to `aes_encrypt()`/`aes_decrypt()`
Posted by GitBox <gi...@apache.org>.
MaxGekk edited a comment on pull request #34837:
URL: https://github.com/apache/spark/pull/34837#issuecomment-989514013
Let's introduce new argument when it will be really needed. So far, the PR solves this particular issue " ... aes_encrypt() and aes_decrypt() rely on the jvm's configuration regarding which cipher mode to support, this is problematic as it is not fixed across versions and systems."
> So, how about taking IV as the 3rd parameter, and taking mode and padding as 4th and 5th for the compatibility?
I would put more general parameters before specific ones. Some modes like our current `ECB` doesn't require any initialization vectors. It would be weird to require users to specify IV.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] MaxGekk commented on pull request #34837: [SPARK-37586][SQL] Add the `mode` and `padding` args to `aes_encrypt()`/`aes_decrypt()`
Posted by GitBox <gi...@apache.org>.
MaxGekk commented on pull request #34837:
URL: https://github.com/apache/spark/pull/34837#issuecomment-989526117
> My main concern is the compatibility with MySQL's ones ...
@sarutak I have looked at other dbms (see the PR's description). Seems like everyone has its own syntax. Not sure we should be compatible with MySQL in particular.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] MaxGekk commented on pull request #34837: [SPARK-37586][SQL] Add the `mode` and `padding` args to `aes_encrypt()`/`aes_decrypt()`
Posted by GitBox <gi...@apache.org>.
MaxGekk commented on pull request #34837:
URL: https://github.com/apache/spark/pull/34837#issuecomment-989536400
Thank you, @sarutak . I am going to implement the GCM mode for now, opened the ticket https://issues.apache.org/jira/browse/SPARK-37591 for this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #34837: [SPARK-37586][SQL] Add the `mode` and `padding` args to `aes_encrypt()`/`aes_decrypt()`
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #34837:
URL: https://github.com/apache/spark/pull/34837#issuecomment-989102822
**[Test build #146012 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/146012/testReport)** for PR 34837 at commit [`3726ba7`](https://github.com/apache/spark/commit/3726ba7e401ec0e6f68c38bb54309662ac9df1ee).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on pull request #34837: [SPARK-37586][SQL] Add the `mode` and `padding` args to `aes_encrypt()`/`aes_decrypt()`
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on pull request #34837:
URL: https://github.com/apache/spark/pull/34837#issuecomment-989526910
But I guess Hive and MySQL have a similar syntax though assuming from the comments above. We can merge it as is for now but I also think it might be good to stick to one of these implementations. Not necessarily Hive or MySQL though.
Is IV argument difficult to implement? We can add it before Spark 3.3 release, and we're all good.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34837: [SPARK-37586][SQL] Add the `mode` and `padding` args to `aes_encrypt()`/`aes_decrypt()`
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34837:
URL: https://github.com/apache/spark/pull/34837#issuecomment-989175032
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/50488/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34837: [SPARK-37586][SQL] Add the `mode` and `padding` args to `aes_encrypt()`/`aes_decrypt()`
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34837:
URL: https://github.com/apache/spark/pull/34837#issuecomment-989039559
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/50487/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34837: [SPARK-37586][SQL] Add the `mode` and `padding` args to `aes_encrypt()`/`aes_decrypt()`
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34837:
URL: https://github.com/apache/spark/pull/34837#issuecomment-989350021
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/146014/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] sarutak edited a comment on pull request #34837: [SPARK-37586][SQL] Add the `mode` and `padding` args to `aes_encrypt()`/`aes_decrypt()`
Posted by GitBox <gi...@apache.org>.
sarutak edited a comment on pull request #34837:
URL: https://github.com/apache/spark/pull/34837#issuecomment-989084898
@MaxGekk Maybe, you need to rebase to `master`. Otherwise, GA fails.
https://github.com/MaxGekk/spark/actions/runs/1555438064
See also [this change](https://github.com/apache/spark/commit/fdc276be98ea8d32d74e3248bbd6cfeeb74ef0d7).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34837: [SPARK-37586][SQL] Add the `mode` and `padding` args to `aes_encrypt()`/`aes_decrypt()`
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34837:
URL: https://github.com/apache/spark/pull/34837#issuecomment-989176228
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/50490/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34837: [SPARK-37586][SQL] Add the `mode` and `padding` args to `aes_encrypt()`/`aes_decrypt()`
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34837:
URL: https://github.com/apache/spark/pull/34837#issuecomment-988952349
**[Test build #146011 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/146011/testReport)** for PR 34837 at commit [`f21b0ed`](https://github.com/apache/spark/commit/f21b0ed77af77d5e50856279ab572b9206994f0b).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] sarutak commented on pull request #34837: [SPARK-37586][SQL] Add the `mode` and `padding` args to `aes_encrypt()`/`aes_decrypt()`
Posted by GitBox <gi...@apache.org>.
sarutak commented on pull request #34837:
URL: https://github.com/apache/spark/pull/34837#issuecomment-989530073
Anyway, this change itself looks fine so merging to `master`.
Thank you @MaxGekk and @HyukjinKwon !
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34837: [SPARK-37586][SQL] Add the `mode` and `padding` args to `aes_encrypt()`/`aes_decrypt()`
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34837:
URL: https://github.com/apache/spark/pull/34837#issuecomment-989241399
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/50490/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34837: [SPARK-37586][SQL] Add the `mode` and `padding` args to `aes_encrypt()`/`aes_decrypt()`
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34837:
URL: https://github.com/apache/spark/pull/34837#issuecomment-989343500
**[Test build #146014 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/146014/testReport)** for PR 34837 at commit [`a386388`](https://github.com/apache/spark/commit/a386388cd2bb09bbf1c4205fb8722a25afb68da6).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds the following public classes _(experimental)_:
* `class PandasSQLStringFormatter(string.Formatter):`
* `class SQLStringFormatter(string.Formatter):`
* `class ExecutorRollPlugin extends SparkPlugin with Logging `
* `case class TryElementAt(left: Expression, right: Expression, child: Expression)`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] sarutak closed pull request #34837: [SPARK-37586][SQL] Add the `mode` and `padding` args to `aes_encrypt()`/`aes_decrypt()`
Posted by GitBox <gi...@apache.org>.
sarutak closed pull request #34837:
URL: https://github.com/apache/spark/pull/34837
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34837: [SPARK-37586][SQL] Add the `mode` and `padding` args to `aes_encrypt()`/`aes_decrypt()`
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34837:
URL: https://github.com/apache/spark/pull/34837#issuecomment-989052166
**[Test build #146011 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/146011/testReport)** for PR 34837 at commit [`f21b0ed`](https://github.com/apache/spark/commit/f21b0ed77af77d5e50856279ab572b9206994f0b).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34837: [SPARK-37586][SQL] Add the `mode` and `padding` args to `aes_encrypt()`/`aes_decrypt()`
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34837:
URL: https://github.com/apache/spark/pull/34837#issuecomment-989052426
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/146011/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34837: [SPARK-37586][SQL] Add the `mode` and `padding` args to `aes_encrypt()`/`aes_decrypt()`
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34837:
URL: https://github.com/apache/spark/pull/34837#issuecomment-989321929
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/146012/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] sarutak commented on pull request #34837: [SPARK-37586][SQL] Add the `mode` and `padding` args to `aes_encrypt()`/`aes_decrypt()`
Posted by GitBox <gi...@apache.org>.
sarutak commented on pull request #34837:
URL: https://github.com/apache/spark/pull/34837#issuecomment-989084898
@MaxGekk Maybe, you need to rebase to `master`.
See https://github.com/apache/spark/commit/fdc276be98ea8d32d74e3248bbd6cfeeb74ef0d7
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] sarutak commented on pull request #34837: [SPARK-37586][SQL] Add the `mode` and `padding` args to `aes_encrypt()`/`aes_decrypt()`
Posted by GitBox <gi...@apache.org>.
sarutak commented on pull request #34837:
URL: https://github.com/apache/spark/pull/34837#issuecomment-989526983
> Seems like everyone has its own syntax. Not sure we should be compatible with MySQL in particular.
Yes, there is no standard syntax but the name `aes_encrypt` and `aes_decrypt` are apparently come from MySQL so users might expect the same usage.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34837: [SPARK-37586][SQL] Add the `mode` and `padding` args to `aes_encrypt()`/`aes_decrypt()`
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34837:
URL: https://github.com/apache/spark/pull/34837#issuecomment-989036391
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/50487/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34837: [SPARK-37586][SQL] Add the `mode` and `padding` args to `aes_encrypt()`/`aes_decrypt()`
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34837:
URL: https://github.com/apache/spark/pull/34837#issuecomment-989321929
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/146012/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] sarutak commented on pull request #34837: [SPARK-37586][SQL] Add the `mode` and `padding` args to `aes_encrypt()`/`aes_decrypt()`
Posted by GitBox <gi...@apache.org>.
sarutak commented on pull request #34837:
URL: https://github.com/apache/spark/pull/34837#issuecomment-989508129
There are no standard encryption/decryption functions in the popular RDBMS.
Actually, I brought these function from Hive, and Hive seems to have brought them from MySQL.
`aes_encrypt` and `aes_decrypt` in MySQL takes the 3rd optional parameter, which is IV.
https://dev.mysql.com/doc/refman/8.0/en/encryption-functions.html#function_aes-decrypt
So, how about taking IV as the 3rd parameter, and taking `mode` and `padding` as 4th and 5th for the compatibility?
cc: @HyukjinKwon @cloud-fan
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34837: [SPARK-37586][SQL] Add the `mode` and `padding` args to `aes_encrypt()`/`aes_decrypt()`
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34837:
URL: https://github.com/apache/spark/pull/34837#issuecomment-989039559
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/50487/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34837: [SPARK-37586][SQL] Add the `mode` and `padding` args to `aes_encrypt()`/`aes_decrypt()`
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34837:
URL: https://github.com/apache/spark/pull/34837#issuecomment-989186927
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/50488/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34837: [SPARK-37586][SQL] Add the `mode` and `padding` args to `aes_encrypt()`/`aes_decrypt()`
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34837:
URL: https://github.com/apache/spark/pull/34837#issuecomment-989186927
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/50488/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] sarutak commented on pull request #34837: [SPARK-37586][SQL] Add the `mode` and `padding` args to `aes_encrypt()`/`aes_decrypt()`
Posted by GitBox <gi...@apache.org>.
sarutak commented on pull request #34837:
URL: https://github.com/apache/spark/pull/34837#issuecomment-989521971
> So far, the PR solves this particular issue
I understand this point. My main concern is the compatibility with MySQL's ones (especially, the order of the parameters).
But it's OK, I'll add the parameter for IV by the time of 3.3.0 release.
This change itself LGTM.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] MaxGekk commented on pull request #34837: [SPARK-37586][SQL] Add the `mode` and `padding` args to `aes_encrypt()`/`aes_decrypt()`
Posted by GitBox <gi...@apache.org>.
MaxGekk commented on pull request #34837:
URL: https://github.com/apache/spark/pull/34837#issuecomment-989514013
Let's introduce new argument when it will be really needed. So far, the PR solves this particular issue " ... aes_encrypt() and aes_decrypt() rely on the jvm's configuration regarding which cipher mode to support, this is problematic as it is not fixed across versions and systems."
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34837: [SPARK-37586][SQL] Add the `mode` and `padding` args to `aes_encrypt()`/`aes_decrypt()`
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34837:
URL: https://github.com/apache/spark/pull/34837#issuecomment-989148761
**[Test build #146014 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/146014/testReport)** for PR 34837 at commit [`a386388`](https://github.com/apache/spark/commit/a386388cd2bb09bbf1c4205fb8722a25afb68da6).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34837: [SPARK-37586][SQL] Add the `mode` and `padding` args to `aes_encrypt()`/`aes_decrypt()`
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34837:
URL: https://github.com/apache/spark/pull/34837#issuecomment-989316492
**[Test build #146012 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/146012/testReport)** for PR 34837 at commit [`3726ba7`](https://github.com/apache/spark/commit/3726ba7e401ec0e6f68c38bb54309662ac9df1ee).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org