You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@spark.apache.org by we...@apache.org on 2026/08/05 04:25:26 UTC
(spark) branch branch-4.3 updated: [SPARK-58295][SQL][PYTHON] Add `to_base32` and `from_base32` functions
This is an automated email from the ASF dual-hosted git repository.
cloud-fan pushed a commit to branch branch-4.3
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-4.3 by this push:
new 59d329119602 [SPARK-58295][SQL][PYTHON] Add `to_base32` and `from_base32` functions
59d329119602 is described below
commit 59d3291196027928d7d7adbff9211495a4851ab5
Author: SreeramaYeshwanthGowd <ye...@gmail.com>
AuthorDate: Wed Aug 5 12:24:40 2026 +0800
[SPARK-58295][SQL][PYTHON] Add `to_base32` and `from_base32` functions
### What changes were proposed in this pull request?
Add a matched pair of built in scalar functions:
- `to_base32(binary)` encodes bytes as an RFC 4648 Base32 string.
- `from_base32(string)` decodes an RFC 4648 Base32 string back to binary.
Spark already ships `hex`/`unhex` (base16) and `base64`/`unbase64`, so this completes the standard base16/base32/base64 encoding ladder.
API surface added:
- SQL: `to_base32(binary)` and `from_base32(string)`
- Scala DataFrame: `functions.to_base32(col)` and `functions.from_base32(col)`
- PySpark, classic and Spark Connect: `pyspark.sql.functions.to_base32` and `from_base32`
Key design choices:
- Implemented as `RuntimeReplaceable` expressions backed by a `StaticInvoke`, mirroring the existing `base64`/`unbase64` functions, so there is no hand written codegen.
- The Base32 codec delegates to `org.apache.commons.codec.binary.Base32` (RFC 4648), the same way `base64` delegates to `java.util.Base64`. `java.util.Base64` has no Base32 support, and commons-codec is already a dependency of the catalyst module, so this adds no new dependency.
- The SQL names `to_base32` and `from_base32` match Trino and BigQuery, and follow Spark's established `to_*`/`from_*` conversion family (for example `to_json`/`from_json`, `to_binary`). No major engine uses `base32`/`unbase32`.
- `from_base32` decodes leniently, ignoring characters outside the Base32 alphabet, consistent with the existing `unbase64` behavior. A strict variant is out of scope, matching how `unbase64` defers strict decoding to `to_binary`.
- No SparkR binding is added, consistent with all recently added functions (for example `to_binary`, `is_valid_utf8`, `luhn_check`), since SparkR is deprecated as of Spark 4.0 and its function surface is frozen.
### Why are the changes needed?
Base32 (RFC 4648) is the standard encoding for TOTP and 2FA shared secrets (RFC 6238), DNS NSEC3 records, and various identifier schemes. Spark has base16 (`hex`) and base64 but no base32, so users fall back to UDFs. Trino (`to_base32`/`from_base32`) and Google BigQuery (`TO_BASE32`/`FROM_BASE32`) both ship the pair, so this also improves parity with the engines Spark users migrate from.
### Does this PR introduce _any_ user-facing change?
Yes. It adds two new built in SQL functions, `to_base32` and `from_base32`, and the corresponding Scala and PySpark DataFrame API entries. No existing behavior changes.
Example:
```
spark-sql> SELECT to_base32(encode('foobar', 'utf-8'));
MZXW6YTBOI======
spark-sql> SELECT cast(from_base32('MZXW6YTBOI======') as string);
foobar
```
### How was this patch tested?
Added catalyst unit tests in `StringExpressionsSuite` covering the RFC 4648 test vectors, round trip encode and decode, empty input, and null propagation, and a DataFrame API test in `StringFunctionsSuite`. Added PySpark doctests for both functions. Regenerated `sql-expression-schema.md`.
### Was this patch authored or co-authored using generative AI tooling?
No
Closes #57466 from SreeramaYeshwanthGowd/add-base32-functions.
Authored-by: SreeramaYeshwanthGowd <ye...@gmail.com>
Signed-off-by: Wenchen Fan <we...@databricks.com>
(cherry picked from commit 1684ada7ee48b2b4c76308464ffee31783dd0989)
Signed-off-by: Wenchen Fan <we...@databricks.com>
---
.../source/reference/pyspark.sql/functions.rst | 2 +
python/pyspark/sql/connect/functions/builtin.py | 14 +++
python/pyspark/sql/functions/__init__.py | 2 +
python/pyspark/sql/functions/builtin.py | 72 ++++++++++++++
.../scala/org/apache/spark/sql/functions.scala | 26 +++++
.../sql/catalyst/analysis/FunctionRegistry.scala | 2 +
.../catalyst/expressions/stringExpressions.scala | 106 +++++++++++++++++++++
.../expressions/StringExpressionsSuite.scala | 24 +++++
.../apache/spark/sql/PlanGenerationTestSuite.scala | 8 ++
.../explain-results/function_from_base32.explain | 2 +
.../explain-results/function_to_base32.explain | 2 +
.../query-tests/queries/function_from_base32.json | 60 ++++++++++++
.../queries/function_from_base32.proto.bin | Bin 0 -> 547 bytes
.../query-tests/queries/function_to_base32.json | 85 +++++++++++++++++
.../queries/function_to_base32.proto.bin | Bin 0 -> 724 bytes
.../sql-functions/sql-expression-schema.md | 2 +
.../apache/spark/sql/StringFunctionsSuite.scala | 12 +++
.../collation/CollationExpressionWalkerSuite.scala | 2 +
18 files changed, 421 insertions(+)
diff --git a/python/docs/source/reference/pyspark.sql/functions.rst b/python/docs/source/reference/pyspark.sql/functions.rst
index 91b722d50efd..cc1756304549 100644
--- a/python/docs/source/reference/pyspark.sql/functions.rst
+++ b/python/docs/source/reference/pyspark.sql/functions.rst
@@ -177,6 +177,7 @@ String Functions
find_in_set
format_number
format_string
+ from_base32
initcap
instr
is_valid_utf8
@@ -216,6 +217,7 @@ String Functions
substr
substring
substring_index
+ to_base32
to_binary
to_char
to_number
diff --git a/python/pyspark/sql/connect/functions/builtin.py b/python/pyspark/sql/connect/functions/builtin.py
index e88d584ea34f..0713eecbe9b0 100644
--- a/python/pyspark/sql/connect/functions/builtin.py
+++ b/python/pyspark/sql/connect/functions/builtin.py
@@ -2601,6 +2601,13 @@ def base64(col: "ColumnOrName") -> Column:
base64.__doc__ = pysparkfuncs.base64.__doc__
+def to_base32(col: "ColumnOrName") -> Column:
+ return _invoke_function_over_columns("to_base32", col)
+
+
+to_base32.__doc__ = pysparkfuncs.to_base32.__doc__
+
+
def unbase64(col: "ColumnOrName") -> Column:
return _invoke_function_over_columns("unbase64", col)
@@ -2608,6 +2615,13 @@ def unbase64(col: "ColumnOrName") -> Column:
unbase64.__doc__ = pysparkfuncs.unbase64.__doc__
+def from_base32(col: "ColumnOrName") -> Column:
+ return _invoke_function_over_columns("from_base32", col)
+
+
+from_base32.__doc__ = pysparkfuncs.from_base32.__doc__
+
+
def ltrim(col: "ColumnOrName", trim: Optional["ColumnOrName"] = None) -> Column:
if trim is not None:
return _invoke_function_over_columns("ltrim", trim, col)
diff --git a/python/pyspark/sql/functions/__init__.py b/python/pyspark/sql/functions/__init__.py
index d4762a56da9b..0e2c6386a4d4 100644
--- a/python/pyspark/sql/functions/__init__.py
+++ b/python/pyspark/sql/functions/__init__.py
@@ -138,6 +138,7 @@ __all__ = [ # noqa: F405
"find_in_set",
"format_number",
"format_string",
+ "from_base32",
"initcap",
"instr",
"is_valid_utf8",
@@ -177,6 +178,7 @@ __all__ = [ # noqa: F405
"substr",
"substring",
"substring_index",
+ "to_base32",
"to_binary",
"to_char",
"to_number",
diff --git a/python/pyspark/sql/functions/builtin.py b/python/pyspark/sql/functions/builtin.py
index e66a3639ef62..6ed9a1049858 100644
--- a/python/pyspark/sql/functions/builtin.py
+++ b/python/pyspark/sql/functions/builtin.py
@@ -15243,6 +15243,7 @@ def base64(col: "ColumnOrName") -> Column:
See Also
--------
:meth:`pyspark.sql.functions.unbase64`
+ :meth:`pyspark.sql.functions.to_base32`
Examples
--------
@@ -15260,6 +15261,41 @@ def base64(col: "ColumnOrName") -> Column:
return _invoke_function_over_columns("base64", col)
+@_try_remote_functions
+def to_base32(col: "ColumnOrName") -> Column:
+ """
+ Computes the BASE32 (RFC 4648) encoding of a binary column and returns it as a
+ string column.
+
+ .. versionadded:: 4.3.0
+
+ Parameters
+ ----------
+ col : :class:`~pyspark.sql.Column` or column name
+ target column to work on.
+ A column that evaluates to a binary.
+
+ Returns
+ -------
+ :class:`~pyspark.sql.Column`
+ BASE32 encoding of the binary value.
+ Returns a column that evaluates to a string.
+
+ See Also
+ --------
+ :meth:`pyspark.sql.functions.from_base32`
+ :meth:`pyspark.sql.functions.base64`
+
+ Examples
+ --------
+ >>> import pyspark.sql.functions as sf
+ >>> df = spark.createDataFrame([(b"foobar",)], ["value"])
+ >>> df.select(sf.to_base32("value").alias("r")).collect()
+ [Row(r='MZXW6YTBOI======')]
+ """
+ return _invoke_function_over_columns("to_base32", col)
+
+
@_try_remote_functions
def unbase64(col: "ColumnOrName") -> Column:
"""
@@ -15285,6 +15321,7 @@ def unbase64(col: "ColumnOrName") -> Column:
See Also
--------
:meth:`pyspark.sql.functions.base64`
+ :meth:`pyspark.sql.functions.from_base32`
Examples
--------
@@ -15302,6 +15339,41 @@ def unbase64(col: "ColumnOrName") -> Column:
return _invoke_function_over_columns("unbase64", col)
+@_try_remote_functions
+def from_base32(col: "ColumnOrName") -> Column:
+ """
+ Decodes a BASE32 (RFC 4648) encoded string column and returns it as a binary
+ column.
+
+ .. versionadded:: 4.3.0
+
+ Parameters
+ ----------
+ col : :class:`~pyspark.sql.Column` or column name
+ target column to work on.
+ A column that evaluates to a string.
+
+ Returns
+ -------
+ :class:`~pyspark.sql.Column`
+ decoded binary value.
+ Returns a column that evaluates to a binary.
+
+ See Also
+ --------
+ :meth:`pyspark.sql.functions.to_base32`
+ :meth:`pyspark.sql.functions.unbase64`
+
+ Examples
+ --------
+ >>> import pyspark.sql.functions as sf
+ >>> df = spark.createDataFrame([("MZXW6YTBOI======",)], ["value"])
+ >>> df.select(sf.from_base32("value").alias("r")).collect()
+ [Row(r=b'foobar')]
+ """
+ return _invoke_function_over_columns("from_base32", col)
+
+
@_try_remote_functions
def ltrim(col: "ColumnOrName", trim: Optional["ColumnOrName"] = None) -> Column:
"""
diff --git a/sql/api/src/main/scala/org/apache/spark/sql/functions.scala b/sql/api/src/main/scala/org/apache/spark/sql/functions.scala
index c7779aa99c10..0cc822c2ec01 100644
--- a/sql/api/src/main/scala/org/apache/spark/sql/functions.scala
+++ b/sql/api/src/main/scala/org/apache/spark/sql/functions.scala
@@ -7325,6 +7325,19 @@ object functions {
*/
def base64(e: Column): Column = Column.fn("base64", e)
+ /**
+ * Computes the BASE32 (RFC 4648) encoding of a binary column and returns it as a string column.
+ * This is the reverse of from_base32.
+ *
+ * @param e
+ * The target column to work on. A column that evaluates to a binary.
+ * @group string_funcs
+ * @since 4.3.0
+ * @return
+ * Returns a column that evaluates to a string.
+ */
+ def to_base32(e: Column): Column = Column.fn("to_base32", e)
+
/**
* Calculates the bit length for the specified string column.
*
@@ -8149,6 +8162,19 @@ object functions {
*/
def unbase64(e: Column): Column = Column.fn("unbase64", e)
+ /**
+ * Decodes a BASE32 (RFC 4648) encoded string column and returns it as a binary column. This is
+ * the reverse of to_base32.
+ *
+ * @param e
+ * target column to work on. A column that evaluates to a string.
+ * @group string_funcs
+ * @since 4.3.0
+ * @return
+ * Returns a column that evaluates to a binary.
+ */
+ def from_base32(e: Column): Column = Column.fn("from_base32", e)
+
/**
* Right-pad the string column with pad to a length of len. If the string column is longer than
* len, the return value is shortened to len characters.
diff --git a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/FunctionRegistry.scala b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/FunctionRegistry.scala
index 862324a83c31..e80544a6e260 100644
--- a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/FunctionRegistry.scala
+++ b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/FunctionRegistry.scala
@@ -610,6 +610,7 @@ object FunctionRegistry {
expressionBuilder("startswith", StartsWithExpressionBuilder),
expressionBuilder("endswith", EndsWithExpressionBuilder),
expression[Base64]("base64"),
+ expression[Base32]("to_base32"),
expression[BitLength]("bit_length"),
expression[Length]("char_length", true, Some("2.3.0")),
expression[Length]("character_length", true, Some("2.3.0")),
@@ -669,6 +670,7 @@ object FunctionRegistry {
expression[StringTrimBoth]("btrim"),
expression[Upper]("ucase", true),
expression[UnBase64]("unbase64"),
+ expression[UnBase32]("from_base32"),
expression[Unhex]("unhex"),
expression[Upper]("upper"),
expression[XPathList]("xpath"),
diff --git a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/stringExpressions.scala b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/stringExpressions.scala
index 5bd8148103bd..41007a3a1033 100755
--- a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/stringExpressions.scala
+++ b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/stringExpressions.scala
@@ -24,6 +24,8 @@ import java.util.{Base64 => JBase64, HashMap, Locale, Map => JMap}
import scala.collection.mutable.ArrayBuffer
+import org.apache.commons.codec.binary.{Base32 => CommonsBase32}
+
import org.apache.spark.QueryContext
import org.apache.spark.network.util.JavaUtils
import org.apache.spark.sql.catalyst.InternalRow
@@ -3356,6 +3358,110 @@ object UnBase64 {
}
}
+/**
+ * Converts the argument from binary to a base 32 string.
+ */
+@ExpressionDescription(
+ usage = "_FUNC_(bin) - Converts the argument from a binary `bin` to a base 32 string.",
+ arguments = """
+ Arguments:
+ * bin - The binary value to encode as a base 32 string.
+ An expression that evaluates to a binary.
+ """,
+ examples = """
+ Examples:
+ > SELECT _FUNC_('foobar');
+ MZXW6YTBOI======
+ > SELECT _FUNC_(x'666f6f626172');
+ MZXW6YTBOI======
+ """,
+ since = "4.3.0",
+ group = "string_funcs")
+case class Base32(child: Expression)
+ extends UnaryExpression
+ with RuntimeReplaceable
+ with ImplicitCastInputTypes
+ with DefaultStringProducingExpression {
+
+ override def inputTypes: Seq[DataType] = Seq(BinaryType)
+
+ override def contextIndependentFoldable: Boolean = child.contextIndependentFoldable
+
+ override lazy val replacement: Expression = StaticInvoke(
+ classOf[Base32],
+ dataType,
+ "encode",
+ Seq(child),
+ Seq(BinaryType),
+ returnNullable = false)
+
+ override def toString: String = s"$prettyName($child)"
+
+ override def prettyName: String = "to_base32"
+
+ override protected def withNewChildInternal(newChild: Expression): Expression =
+ copy(child = newChild)
+}
+
+object Base32 {
+ private lazy val codec = new CommonsBase32()
+
+ def encode(input: Array[Byte]): UTF8String = {
+ UTF8String.fromBytes(codec.encode(input))
+ }
+}
+
+/**
+ * Converts the argument from a base 32 string to BINARY.
+ */
+@ExpressionDescription(
+ usage = "_FUNC_(str) - Converts the argument from a base 32 string `str` to a binary.",
+ arguments = """
+ Arguments:
+ * str - The base 32 string to decode to binary.
+ An expression that evaluates to a string.
+ """,
+ examples = """
+ Examples:
+ > SELECT _FUNC_('MZXW6YTBOI======');
+ foobar
+ """,
+ since = "4.3.0",
+ group = "string_funcs")
+case class UnBase32(child: Expression)
+ extends UnaryExpression
+ with RuntimeReplaceable
+ with ImplicitCastInputTypes {
+
+ override def dataType: DataType = BinaryType
+ override def inputTypes: Seq[AbstractDataType] =
+ Seq(StringTypeWithCollation(supportsTrimCollation = true))
+ override def contextIndependentFoldable: Boolean = child.contextIndependentFoldable
+
+ override lazy val replacement: Expression = StaticInvoke(
+ classOf[UnBase32],
+ dataType,
+ "decode",
+ Seq(child),
+ inputTypes,
+ returnNullable = false)
+
+ override def toString: String = s"$prettyName($child)"
+
+ override def prettyName: String = "from_base32"
+
+ override protected def withNewChildInternal(newChild: Expression): Expression =
+ copy(child = newChild)
+}
+
+object UnBase32 {
+ private lazy val codec = new CommonsBase32()
+
+ def decode(input: UTF8String): Array[Byte] = {
+ codec.decode(input.getBytes)
+ }
+}
+
object Decode {
def createExpr(params: Seq[Expression]): Expression = {
params.length match {
diff --git a/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/StringExpressionsSuite.scala b/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/StringExpressionsSuite.scala
index 050a0245343f..d2a58b95b83e 100644
--- a/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/StringExpressionsSuite.scala
+++ b/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/StringExpressionsSuite.scala
@@ -499,6 +499,30 @@ class StringExpressionsSuite extends SparkFunSuite with ExpressionEvalHelper {
checkEvaluation(UnBase64(Literal.create(null, StringType)), null, create_row("abdef"))
}
+ test("to_base32/from_base32 for string") {
+ // RFC 4648 (section 10) test vectors.
+ checkEvaluation(Base32(Literal("".getBytes("UTF-8"))), "")
+ checkEvaluation(Base32(Literal("f".getBytes("UTF-8"))), "MY======")
+ checkEvaluation(Base32(Literal("fo".getBytes("UTF-8"))), "MZXQ====")
+ checkEvaluation(Base32(Literal("foo".getBytes("UTF-8"))), "MZXW6===")
+ checkEvaluation(Base32(Literal("foob".getBytes("UTF-8"))), "MZXW6YQ=")
+ checkEvaluation(Base32(Literal("fooba".getBytes("UTF-8"))), "MZXW6YTB")
+ checkEvaluation(Base32(Literal("foobar".getBytes("UTF-8"))), "MZXW6YTBOI======")
+
+ assert(!Base32(Literal("foo".getBytes("UTF-8"))).nullable)
+ assert(Base32(Literal.create(null, BinaryType)).nullable)
+ assert(!UnBase32(Literal("MZXW6YTBOI======")).nullable)
+ assert(UnBase32(Literal.create(null, StringType)).nullable)
+
+ checkEvaluation(UnBase32(Literal("MZXW6YTBOI======")), "foobar".getBytes("UTF-8"))
+ checkEvaluation(UnBase32(Literal("MY======")), "f".getBytes("UTF-8"))
+
+ // Round trip.
+ checkEvaluation(Base32(UnBase32(Literal("MZXW6YTBOI======"))), "MZXW6YTBOI======")
+ checkEvaluation(Base32(UnBase32(Literal(""))), "")
+ checkEvaluation(Base32(UnBase32(Literal.create(null, StringType))), null)
+ }
+
test("encode/decode for string") {
val a = $"a".string.at(0)
val b = $"b".binary.at(0)
diff --git a/sql/connect/client/jvm/src/test/scala/org/apache/spark/sql/PlanGenerationTestSuite.scala b/sql/connect/client/jvm/src/test/scala/org/apache/spark/sql/PlanGenerationTestSuite.scala
index 266de35706da..446efbf0c955 100644
--- a/sql/connect/client/jvm/src/test/scala/org/apache/spark/sql/PlanGenerationTestSuite.scala
+++ b/sql/connect/client/jvm/src/test/scala/org/apache/spark/sql/PlanGenerationTestSuite.scala
@@ -1855,6 +1855,14 @@ class PlanGenerationTestSuite extends ConnectFunSuite with Logging {
fn.unbase64(fn.col("g"))
}
+ functionTest("to_base32") {
+ fn.to_base32(fn.col("g").cast("binary"))
+ }
+
+ functionTest("from_base32") {
+ fn.from_base32(fn.col("g"))
+ }
+
functionTest("rpad") {
fn.rpad(fn.col("g"), 10, "-")
}
diff --git a/sql/connect/common/src/test/resources/query-tests/explain-results/function_from_base32.explain b/sql/connect/common/src/test/resources/query-tests/explain-results/function_from_base32.explain
new file mode 100644
index 000000000000..482fb5b0f232
--- /dev/null
+++ b/sql/connect/common/src/test/resources/query-tests/explain-results/function_from_base32.explain
@@ -0,0 +1,2 @@
+Project [static_invoke(UnBase32.decode(g#0)) AS from_base32(g)#0]
++- LocalRelation <empty>, [id#0L, a#0, b#0, d#0, e#0, f#0, g#0]
diff --git a/sql/connect/common/src/test/resources/query-tests/explain-results/function_to_base32.explain b/sql/connect/common/src/test/resources/query-tests/explain-results/function_to_base32.explain
new file mode 100644
index 000000000000..340a45adc361
--- /dev/null
+++ b/sql/connect/common/src/test/resources/query-tests/explain-results/function_to_base32.explain
@@ -0,0 +1,2 @@
+Project [static_invoke(Base32.encode(cast(g#0 as binary))) AS to_base32(CAST(g AS BINARY))#0]
++- LocalRelation <empty>, [id#0L, a#0, b#0, d#0, e#0, f#0, g#0]
diff --git a/sql/connect/common/src/test/resources/query-tests/queries/function_from_base32.json b/sql/connect/common/src/test/resources/query-tests/queries/function_from_base32.json
new file mode 100644
index 000000000000..d53a88995bfd
--- /dev/null
+++ b/sql/connect/common/src/test/resources/query-tests/queries/function_from_base32.json
@@ -0,0 +1,60 @@
+{
+ "common": {
+ "planId": "1"
+ },
+ "project": {
+ "input": {
+ "common": {
+ "planId": "0"
+ },
+ "localRelation": {
+ "schema": "struct\u003cid:bigint,a:int,b:double,d:struct\u003cid:bigint,a:int,b:double\u003e,e:array\u003cint\u003e,f:map\u003cstring,struct\u003cid:bigint,a:int,b:double\u003e\u003e,g:string\u003e"
+ }
+ },
+ "expressions": [{
+ "unresolvedFunction": {
+ "functionName": "from_base32",
+ "arguments": [{
+ "unresolvedAttribute": {
+ "unparsedIdentifier": "g"
+ },
+ "common": {
+ "origin": {
+ "jvmOrigin": {
+ "stackTrace": [{
+ "classLoaderName": "app",
+ "declaringClass": "org.apache.spark.sql.functions$",
+ "methodName": "col",
+ "fileName": "functions.scala"
+ }, {
+ "classLoaderName": "app",
+ "declaringClass": "org.apache.spark.sql.PlanGenerationTestSuite",
+ "methodName": "~~trimmed~anonfun~~",
+ "fileName": "PlanGenerationTestSuite.scala"
+ }]
+ }
+ }
+ }
+ }],
+ "isInternal": false
+ },
+ "common": {
+ "origin": {
+ "jvmOrigin": {
+ "stackTrace": [{
+ "classLoaderName": "app",
+ "declaringClass": "org.apache.spark.sql.functions$",
+ "methodName": "from_base32",
+ "fileName": "functions.scala"
+ }, {
+ "classLoaderName": "app",
+ "declaringClass": "org.apache.spark.sql.PlanGenerationTestSuite",
+ "methodName": "~~trimmed~anonfun~~",
+ "fileName": "PlanGenerationTestSuite.scala"
+ }]
+ }
+ }
+ }
+ }]
+ }
+}
\ No newline at end of file
diff --git a/sql/connect/common/src/test/resources/query-tests/queries/function_from_base32.proto.bin b/sql/connect/common/src/test/resources/query-tests/queries/function_from_base32.proto.bin
new file mode 100644
index 000000000000..551dda1b878c
Binary files /dev/null and b/sql/connect/common/src/test/resources/query-tests/queries/function_from_base32.proto.bin differ
diff --git a/sql/connect/common/src/test/resources/query-tests/queries/function_to_base32.json b/sql/connect/common/src/test/resources/query-tests/queries/function_to_base32.json
new file mode 100644
index 000000000000..4abce6e10257
--- /dev/null
+++ b/sql/connect/common/src/test/resources/query-tests/queries/function_to_base32.json
@@ -0,0 +1,85 @@
+{
+ "common": {
+ "planId": "1"
+ },
+ "project": {
+ "input": {
+ "common": {
+ "planId": "0"
+ },
+ "localRelation": {
+ "schema": "struct\u003cid:bigint,a:int,b:double,d:struct\u003cid:bigint,a:int,b:double\u003e,e:array\u003cint\u003e,f:map\u003cstring,struct\u003cid:bigint,a:int,b:double\u003e\u003e,g:string\u003e"
+ }
+ },
+ "expressions": [{
+ "unresolvedFunction": {
+ "functionName": "to_base32",
+ "arguments": [{
+ "cast": {
+ "expr": {
+ "unresolvedAttribute": {
+ "unparsedIdentifier": "g"
+ },
+ "common": {
+ "origin": {
+ "jvmOrigin": {
+ "stackTrace": [{
+ "classLoaderName": "app",
+ "declaringClass": "org.apache.spark.sql.functions$",
+ "methodName": "col",
+ "fileName": "functions.scala"
+ }, {
+ "classLoaderName": "app",
+ "declaringClass": "org.apache.spark.sql.PlanGenerationTestSuite",
+ "methodName": "~~trimmed~anonfun~~",
+ "fileName": "PlanGenerationTestSuite.scala"
+ }]
+ }
+ }
+ }
+ },
+ "type": {
+ "binary": {
+ }
+ }
+ },
+ "common": {
+ "origin": {
+ "jvmOrigin": {
+ "stackTrace": [{
+ "classLoaderName": "app",
+ "declaringClass": "org.apache.spark.sql.Column",
+ "methodName": "cast",
+ "fileName": "Column.scala"
+ }, {
+ "classLoaderName": "app",
+ "declaringClass": "org.apache.spark.sql.PlanGenerationTestSuite",
+ "methodName": "~~trimmed~anonfun~~",
+ "fileName": "PlanGenerationTestSuite.scala"
+ }]
+ }
+ }
+ }
+ }],
+ "isInternal": false
+ },
+ "common": {
+ "origin": {
+ "jvmOrigin": {
+ "stackTrace": [{
+ "classLoaderName": "app",
+ "declaringClass": "org.apache.spark.sql.functions$",
+ "methodName": "to_base32",
+ "fileName": "functions.scala"
+ }, {
+ "classLoaderName": "app",
+ "declaringClass": "org.apache.spark.sql.PlanGenerationTestSuite",
+ "methodName": "~~trimmed~anonfun~~",
+ "fileName": "PlanGenerationTestSuite.scala"
+ }]
+ }
+ }
+ }
+ }]
+ }
+}
\ No newline at end of file
diff --git a/sql/connect/common/src/test/resources/query-tests/queries/function_to_base32.proto.bin b/sql/connect/common/src/test/resources/query-tests/queries/function_to_base32.proto.bin
new file mode 100644
index 000000000000..93ef08635f3d
Binary files /dev/null and b/sql/connect/common/src/test/resources/query-tests/queries/function_to_base32.proto.bin differ
diff --git a/sql/core/src/test/resources/sql-functions/sql-expression-schema.md b/sql/core/src/test/resources/sql-functions/sql-expression-schema.md
index 03505b37c4c2..10f68f708b1b 100644
--- a/sql/core/src/test/resources/sql-functions/sql-expression-schema.md
+++ b/sql/core/src/test/resources/sql-functions/sql-expression-schema.md
@@ -44,6 +44,7 @@
| org.apache.spark.sql.catalyst.expressions.Atan2 | atan2 | SELECT atan2(0, 0) | struct<ATAN2(0, 0):double> |
| org.apache.spark.sql.catalyst.expressions.Atanh | atanh | SELECT atanh(0) | struct<ATANH(0):double> |
| org.apache.spark.sql.catalyst.expressions.BRound | bround | SELECT bround(2.5, 0) | struct<bround(2.5, 0):decimal(2,0)> |
+| org.apache.spark.sql.catalyst.expressions.Base32 | to_base32 | SELECT to_base32('foobar') | struct<to_base32(foobar):string> |
| org.apache.spark.sql.catalyst.expressions.Base64 | base64 | SELECT base64('Spark SQL') | struct<base64(Spark SQL):string> |
| org.apache.spark.sql.catalyst.expressions.Between | between | SELECT 0.5 between 0.1 AND 1.0 | struct<between(0.5, 0.1, 1.0):boolean> |
| org.apache.spark.sql.catalyst.expressions.Bin | bin | SELECT bin(13) | struct<bin(13):string> |
@@ -430,6 +431,7 @@
| org.apache.spark.sql.catalyst.expressions.TupleUnionThetaDoubleExpressionBuilder | tuple_union_theta_double | SELECT tuple_sketch_estimate_double(tuple_union_theta_double(tuple_sketch_agg_double(col1, val1), theta_sketch_agg(col2))) FROM VALUES (1, 1.0D, 4), (2, 2.0D, 5), (3, 3.0D, 6) tab(col1, val1, col2) | struct<tuple_sketch_estimate_double(tuple_union_theta_double(tuple_sketch_agg_double(col1, val1, 12, sum), theta_sketch_agg(col2, 12), 12, sum)):double> |
| org.apache.spark.sql.catalyst.expressions.TupleUnionThetaIntegerExpressionBuilder | tuple_union_theta_integer | SELECT tuple_sketch_estimate_integer(tuple_union_theta_integer(tuple_sketch_agg_integer(col1, val1), theta_sketch_agg(col2))) FROM VALUES (1, 1, 4), (2, 2, 5), (3, 3, 6) tab(col1, val1, col2) | struct<tuple_sketch_estimate_integer(tuple_union_theta_integer(tuple_sketch_agg_integer(col1, val1, 12, sum), theta_sketch_agg(col2, 12), 12, sum)):double> |
| org.apache.spark.sql.catalyst.expressions.TypeOf | typeof | SELECT typeof(1) | struct<typeof(1):string> |
+| org.apache.spark.sql.catalyst.expressions.UnBase32 | from_base32 | SELECT from_base32('MZXW6YTBOI======') | struct<from_base32(MZXW6YTBOI======):binary> |
| org.apache.spark.sql.catalyst.expressions.UnBase64 | unbase64 | SELECT unbase64('U3BhcmsgU1FM') | struct<unbase64(U3BhcmsgU1FM):binary> |
| org.apache.spark.sql.catalyst.expressions.UnaryMinus | negative | SELECT negative(1) | struct<negative(1):int> |
| org.apache.spark.sql.catalyst.expressions.UnaryPositive | positive | SELECT positive(1) | struct<(+ 1):int> |
diff --git a/sql/core/src/test/scala/org/apache/spark/sql/StringFunctionsSuite.scala b/sql/core/src/test/scala/org/apache/spark/sql/StringFunctionsSuite.scala
index e5fa22b05ecb..3e0c8425c54e 100644
--- a/sql/core/src/test/scala/org/apache/spark/sql/StringFunctionsSuite.scala
+++ b/sql/core/src/test/scala/org/apache/spark/sql/StringFunctionsSuite.scala
@@ -308,6 +308,18 @@ class StringFunctionsSuite extends SharedSparkSession {
Row("AQIDBA==", bytes))
}
+ test("string to_base32/from_base32 function") {
+ val bytes = "foobar".getBytes("UTF-8")
+ val df = Seq((bytes, "MZXW6YTBOI======")).toDF("a", "b")
+ checkAnswer(
+ df.select(to_base32($"a"), from_base32($"b")),
+ Row("MZXW6YTBOI======", bytes))
+
+ checkAnswer(
+ df.selectExpr("to_base32(a)", "from_base32(b)"),
+ Row("MZXW6YTBOI======", bytes))
+ }
+
test("string overlay function") {
// scalastyle:off
// non ascii characters are not allowed in the code, so we disable the scalastyle here.
diff --git a/sql/core/src/test/scala/org/apache/spark/sql/collation/CollationExpressionWalkerSuite.scala b/sql/core/src/test/scala/org/apache/spark/sql/collation/CollationExpressionWalkerSuite.scala
index dd22f647ab31..41be966fdfef 100644
--- a/sql/core/src/test/scala/org/apache/spark/sql/collation/CollationExpressionWalkerSuite.scala
+++ b/sql/core/src/test/scala/org/apache/spark/sql/collation/CollationExpressionWalkerSuite.scala
@@ -381,6 +381,8 @@ class CollationExpressionWalkerSuite extends SharedSparkSession {
"sha1",
"unbase64",
"base64",
+ "to_base32",
+ "from_base32",
"sha2",
"sha",
"crc32",
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org