You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "rafafrdz (via GitHub)" <gi...@apache.org> on 2026/02/12 11:06:21 UTC
[PR] feat: add support for `aes_decrypt` expression [datafusion-comet]
rafafrdz opened a new pull request, #3497:
URL: https://github.com/apache/datafusion-comet/pull/3497
**Summary**
- Added support for Spark `aes_decrypt` by mapping AesDecrypt to native scalar function `aes_decrypt`.
- Added tests for aes_decrypt in CometStringExpressionSuite covering both:
- default args: aes_decrypt(input, key)
- explicit args: aes_decrypt(input, key, mode, padding, aad)
- Updated spark_expressions_support.md to mark aes_decrypt as supported.
Closes #3188
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: add support for `aes_decrypt` expression [datafusion-comet]
Posted by "mbutrovich (via GitHub)" <gi...@apache.org>.
mbutrovich commented on PR #3497:
URL: https://github.com/apache/datafusion-comet/pull/3497#issuecomment-5344889995
Thank you for the contribution, @rafafrdz! I'm going to close this as stale for now. Please feel free to revisit this effort in the future!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: add support for `aes_decrypt` expression [datafusion-comet]
Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove commented on PR #3497:
URL: https://github.com/apache/datafusion-comet/pull/3497#issuecomment-4384066542
@rafafrdz could you fix conflicts and formatting issues when you get a chance?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: add support for `aes_decrypt` expression [datafusion-comet]
Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove commented on PR #3497:
URL: https://github.com/apache/datafusion-comet/pull/3497#issuecomment-4801897073
Moving this to draft to reduce PR review queue size. Note that this expression is now supported per https://datafusion.apache.org/comet/user-guide/0.17/expressions.html but via JVM codegen dispatch
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: add support for `aes_decrypt` expression [datafusion-comet]
Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove commented on code in PR #3497:
URL: https://github.com/apache/datafusion-comet/pull/3497#discussion_r2804757431
##########
spark/src/main/scala/org/apache/comet/serde/QueryPlanSerde.scala:
##########
@@ -146,6 +146,7 @@ object QueryPlanSerde extends Logging with CometExprShim {
classOf[Sha1] -> CometSha1)
private val stringExpressions: Map[Class[_ <: Expression], CometExpressionSerde[_]] = Map(
+ classOf[AesDecrypt] -> CometScalarFunction("aes_decrypt"),
Review Comment:
Because Spark is using StaticInvoke for AesDescrypt, you likely need to add special handing in `CometStaticInvoke`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: add support for `aes_decrypt` expression [datafusion-comet]
Posted by "rafafrdz (via GitHub)" <gi...@apache.org>.
rafafrdz commented on code in PR #3497:
URL: https://github.com/apache/datafusion-comet/pull/3497#discussion_r2819446184
##########
spark/src/main/scala/org/apache/comet/serde/QueryPlanSerde.scala:
##########
@@ -146,6 +146,7 @@ object QueryPlanSerde extends Logging with CometExprShim {
classOf[Sha1] -> CometSha1)
private val stringExpressions: Map[Class[_ <: Expression], CometExpressionSerde[_]] = Map(
+ classOf[AesDecrypt] -> CometScalarFunction("aes_decrypt"),
Review Comment:
Is this enought? https://github.com/rafafrdz/datafusion-comet/blob/92dd5561c857e70385383e38c1f196acccdf69e6/spark/src/main/scala/org/apache/comet/serde/misc.scala#L52
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: add support for `aes_decrypt` expression [datafusion-comet]
Posted by "mbutrovich (via GitHub)" <gi...@apache.org>.
mbutrovich closed pull request #3497: feat: add support for `aes_decrypt` expression
URL: https://github.com/apache/datafusion-comet/pull/3497
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: add support for `aes_decrypt` expression [datafusion-comet]
Posted by "rafafrdz (via GitHub)" <gi...@apache.org>.
rafafrdz commented on code in PR #3497:
URL: https://github.com/apache/datafusion-comet/pull/3497#discussion_r2819446184
##########
spark/src/main/scala/org/apache/comet/serde/QueryPlanSerde.scala:
##########
@@ -146,6 +146,7 @@ object QueryPlanSerde extends Logging with CometExprShim {
classOf[Sha1] -> CometSha1)
private val stringExpressions: Map[Class[_ <: Expression], CometExpressionSerde[_]] = Map(
+ classOf[AesDecrypt] -> CometScalarFunction("aes_decrypt"),
Review Comment:
Is this enough? https://github.com/rafafrdz/datafusion-comet/blob/92dd5561c857e70385383e38c1f196acccdf69e6/spark/src/main/scala/org/apache/comet/serde/misc.scala#L52
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: add support for `aes_decrypt` expression [datafusion-comet]
Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove commented on PR #3497:
URL: https://github.com/apache/datafusion-comet/pull/3497#issuecomment-3926233766
Thanks for the contribution @rafafrdz. I took a quick look and this looks nice. I kicked off CI.
On the testing side, the current tests only cover GCM mode with a 256-bit key. Since the Rust implementation also supports CBC and ECB modes with PKCS padding, and handles 128-bit, 192-bit, and 256-bit keys, it would be great to add test coverage for those combinations.
It might also be worth adding a null input test, maybe inserting a row with null encrypted data and verifying the result is null.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org