You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "n0r0shi (via GitHub)" <gi...@apache.org> on 2026/02/23 10:00:29 UTC
[PR] feat: support Spark base64 expression (#419) [datafusion-comet]
n0r0shi opened a new pull request, #3571:
URL: https://github.com/apache/datafusion-comet/pull/3571
## Which issue does this PR close?
Closes #419.
## Rationale for this change
Spark's `base64()` expression is currently not supported by Comet, causing fallback to Spark. This is a commonly used function that can be mapped directly to DataFusion's built-in `encode(input, 'base64')` function with no Rust changes.
## What changes are included in this PR?
Two code paths handle different Spark versions:
- **Spark 3.4**: `Base64` is a direct expression node. Added `CometBase64` handler in `strings.scala` and registered it in the `stringExpressions` map.
- **Spark 3.5+**: `Base64` is `RuntimeReplaceable` — Spark's optimizer rewrites it into `StaticInvoke(Base64.encode, [input, chunkBase64])` before Comet sees the plan. Added `CometBase64Encode` handler in `statics.scala` to handle this.
Both paths produce the same DataFusion call: `encode(input, 'base64')`.
**Chunked base64** (`spark.sql.chunkBase64String.enabled=true`, which inserts newlines every 76 chars per RFC 2045) is not supported by DataFusion's `encode` function, so it falls back to Spark. I can take a look at DataFusion side for this later.
## Are these changes tested?
- Normal base64 encoding: `checkSparkAnswerAndOperator` verifies correct results and native Comet execution
- NULL handling: verified via `checkSparkAnswerAndOperator`
- Chunked base64 fallback: `checkSparkAnswerAndFallbackReason` verifies correct results via Spark fallback and checks the expected fallback reason message
- The Spark 3.4 direct expression handler (`CometBase64`) is exercised when CI runs the `spark-3.4` profile. On Spark 3.5+ it is not reached because Spark replaces `Base64` with `StaticInvoke` during optimization.
## Are there any user-facing changes?
Yes. `base64()` is now executed natively by Comet instead of falling back to Spark, improving performance for queries using this function.
```sql
SELECT base64(binary_column) FROM table
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: support Spark base64 expression [datafusion-comet]
Posted by "mbutrovich (via GitHub)" <gi...@apache.org>.
mbutrovich commented on PR #3571:
URL: https://github.com/apache/datafusion-comet/pull/3571#issuecomment-3946263356
Thanks @n0r0shi! Kicking off CI.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: support Spark base64 expression [datafusion-comet]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed pull request #3571: feat: support Spark base64 expression
URL: https://github.com/apache/datafusion-comet/pull/3571
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: support Spark base64 expression [datafusion-comet]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #3571:
URL: https://github.com/apache/datafusion-comet/pull/3571#issuecomment-4888562528
Thank you for your contribution. Unfortunately, this pull request is stale because it has been open 60 days with no activity. Please remove the stale label or comment or this will be closed in 7 days.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: support Spark base64 expression [datafusion-comet]
Posted by "coderfender (via GitHub)" <gi...@apache.org>.
coderfender commented on PR #3571:
URL: https://github.com/apache/datafusion-comet/pull/3571#issuecomment-4392707926
@n0r0shi , FYI https://github.com/apache/datafusion-comet/pull/4159
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: support Spark base64 expression [datafusion-comet]
Posted by "n0r0shi (via GitHub)" <gi...@apache.org>.
n0r0shi commented on PR #3571:
URL: https://github.com/apache/datafusion-comet/pull/3571#issuecomment-4016281165
Sorry my bad @mbutrovich , https://github.com/apache/datafusion/pull/19968 was contained in `53.0.0-rc1` for the first time so `base64pad` and `SparkBase64` are **53.0.0+** only. I think we can wait till Comet moves to 53.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: support Spark base64 expression [datafusion-comet]
Posted by "mbutrovich (via GitHub)" <gi...@apache.org>.
mbutrovich commented on PR #3571:
URL: https://github.com/apache/datafusion-comet/pull/3571#issuecomment-3954722304
> Can I keep this PR as draft until Comet upgrades to DataFusion 52.x+?
Sure thing, we should be wrapping up the DF 52 migration this week.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: support Spark base64 expression [datafusion-comet]
Posted by "n0r0shi (via GitHub)" <gi...@apache.org>.
n0r0shi commented on PR #3571:
URL: https://github.com/apache/datafusion-comet/pull/3571#issuecomment-3949225963
I found that DataFusion's `encode(input, 'base64')` produces unpadded output, which doesn't match Spark's padded base64. The padded variant (`'base64pad'`) and `SparkBase64` in the `datafusion-spark` crate are available in DataFusion 52.x, but Comet currently pins to 51.0.0.
Can I keep this PR as draft until Comet upgrades to DataFusion 52.x+?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org