You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "David Mollitor (Jira)" <ji...@apache.org> on 2026/09/10 02:24:00 UTC
[jira] [Created] (SPARK-59394) UnBase64: decode the input bytes directly instead of converting to a String per row
David Mollitor created SPARK-59394:
--------------------------------------
Summary: UnBase64: decode the input bytes directly instead of converting to a String per row
Key: SPARK-59394
URL: https://issues.apache.org/jira/browse/SPARK-59394
Project: Spark
Issue Type: Improvement
Components: Spark Core
Affects Versions: 4.1.0
Reporter: David Mollitor
h2. Summary
The {{UnBase64}} expression (SQL {{unbase64}} and {{{}try_to_binary(..., 'base64'){}}}) decodes a base64 string to bytes. Both its interpreted path and its generated code call {{.toString()}} on the input {{UTF8String}} before decoding, which decodes the whole input from UTF-8 into a Java {{String}} on every row purely to hand it to the decoder. {{java.util.Base64.Decoder}} accepts a byte array directly, so passing the raw UTF-8 bytes ({{{}UTF8String.getBytes(){}}}) removes that per-row {{String}} allocation and UTF-8 decode.
h2. Why this is safe (no behavior change)
{{getMimeDecoder().decode(String)}} internally converts the string to bytes with ISO-8859-1 before decoding. For valid base64 – which is always ASCII – those bytes are identical to the raw UTF-8 bytes. For any non-ASCII or otherwise non-base64 content, the MIME decoder ignores every byte outside the base64 alphabet ({{{}A-Za-z0-9+/={}}}) under both paths, so the decoded output is the same. The change therefore produces identical results for all inputs.
h2. Does this PR introduce _any_ user-facing change?
No.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org