You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "David Mollitor (Jira)" <ji...@apache.org> on 2026/09/10 02:24:00 UTC

[jira] [Created] (SPARK-59394) UnBase64: decode the input bytes directly instead of converting to a String per row

David Mollitor created SPARK-59394:
--------------------------------------

             Summary: UnBase64: decode the input bytes directly instead of converting to a String per row
                 Key: SPARK-59394
                 URL: https://issues.apache.org/jira/browse/SPARK-59394
             Project: Spark
          Issue Type: Improvement
          Components: Spark Core
    Affects Versions: 4.1.0
            Reporter: David Mollitor


h2. Summary

The {{UnBase64}} expression (SQL {{unbase64}} and {{{}try_to_binary(..., 'base64'){}}}) decodes a base64 string to bytes. Both its interpreted path and its generated code call {{.toString()}} on the input {{UTF8String}} before decoding, which decodes the whole input from UTF-8 into a Java {{String}} on every row purely to hand it to the decoder. {{java.util.Base64.Decoder}} accepts a byte array directly, so passing the raw UTF-8 bytes ({{{}UTF8String.getBytes(){}}}) removes that per-row {{String}} allocation and UTF-8 decode.
h2. Why this is safe (no behavior change)

{{getMimeDecoder().decode(String)}} internally converts the string to bytes with ISO-8859-1 before decoding. For valid base64 – which is always ASCII – those bytes are identical to the raw UTF-8 bytes. For any non-ASCII or otherwise non-base64 content, the MIME decoder ignores every byte outside the base64 alphabet ({{{}A-Za-z0-9+/={}}}) under both paths, so the decoded output is the same. The change therefore produces identical results for all inputs.
h2. Does this PR introduce _any_ user-facing change?

No.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org