You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@spark.apache.org by do...@apache.org on 2026/09/04 02:39:50 UTC

(spark-connect-swift) branch main updated: [SPARK-59235] Support error, reflection and bitmap functions

This is an automated email from the ASF dual-hosted git repository.

dongjoon-hyun pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/spark-connect-swift.git


The following commit(s) were added to refs/heads/main by this push:
     new c913324  [SPARK-59235] Support error, reflection and bitmap functions
c913324 is described below

commit c913324d841b8399df5981f0c8f8763de2d02b71
Author: Dongjoon Hyun <do...@apache.org>
AuthorDate: Thu Sep 3 19:39:41 2026 -0700

    [SPARK-59235] Support error, reflection and bitmap functions
    
    ### What changes were proposed in this pull request?
    
    This PR aims to support the error, reflection, and bitmap functions.
    
    | Function | Since |
    | -------- | ----- |
    | `assert_true(col)` / `assert_true(col, errMsg)` | 3.1.0 |
    | `raise_error(errMsg)` | 3.1.0 |
    | `reflect(cols...)` | 3.5.0 |
    | `java_method(cols...)` | 3.5.0 |
    | `try_reflect(cols...)` | 4.0.0 |
    | `bitmap_bit_position(col)` | 3.5.0 |
    | `bitmap_bucket_number(col)` | 3.5.0 |
    | `bitmap_count(col)` | 3.5.0 |
    
    All of them are classified as misc functions by PySpark and the Spark SQL Scala
    API, so they are added to the existing `MiscFunctions.swift` in alphabetical order.
    
    Two details follow the upstream implementations:
    
    - `errMsg` is `Union[Column, str]` in PySpark, where a `str` is wrapped with
      `lit()`. This is expressed as a `String` overload for `assert_true` and
      `raise_error`, like the existing `schema_of_json` overload pair.
    - The reflection functions are variadic. Their first argument is the class name,
      the second one is the method name, and the remaining ones are the method
      arguments. No `String` overload is added for them because a bare string is a
      column name rather than a literal in PySpark, so an overload would silently
      mean the opposite of the upstream API.
    
    Since the reflection functions invoke an arbitrary Java static method on the
    server, their doc comments note that the class must be on the Spark server's
    classpath and that the class and the method names must never be built from
    untrusted user input.
    
    ### Why are the changes needed?
    
    To improve the API coverage. These functions are available in PySpark and the
    Spark SQL Scala API, but were missing from this Swift client.
    
    ### Does this PR introduce _any_ user-facing change?
    
    No, this is a new feature.
    
    ```swift
    try await spark.range(1).select(
      reflect(lit("java.util.UUID"), lit("fromString"), lit("a5cf6c42-0c85-418f-af6c-3e4e5b1328f2")),
      bitmap_bit_position(lit(123)),
      bitmap_bucket_number(lit(123)),
      bitmap_count(unhex(lit("FFFF")))
    ).show()
    ```
    
    ```
    +-------------------------------------------------------------------------+------------------------+-------------------------+-------------------------+
    |reflect(java.util.UUID, fromString, a5cf6c42-0c85-418f-af6c-3e4e5b1328f2)|bitmap_bit_position(123)|bitmap_bucket_number(123)|bitmap_count(unhex(FFFF))|
    +-------------------------------------------------------------------------+------------------------+-------------------------+-------------------------+
    |                                                     a5cf6c42-0c85-418...|                     122|                        1|                       16|
    +-------------------------------------------------------------------------+------------------------+-------------------------+-------------------------+
    ```
    
    ### How was this patch tested?
    
    Pass the CIs with the new test cases in `MiscFunctionsTests`.
    
    ### Was this patch authored or co-authored using generative AI tooling?
    
    Generated-by: Claude Opus 5
    
    Closes #531 from dongjoon-hyun/SPARK-59235.
    
    Authored-by: Dongjoon Hyun <do...@apache.org>
    Signed-off-by: Dongjoon Hyun <do...@apache.org>
---
 Sources/SparkConnect/MiscFunctions.swift         | 100 +++++++++++++++++++++++
 Tests/SparkConnectTests/MiscFunctionsTests.swift |  98 ++++++++++++++++++++++
 2 files changed, 198 insertions(+)

diff --git a/Sources/SparkConnect/MiscFunctions.swift b/Sources/SparkConnect/MiscFunctions.swift
index 2cf8952..b44657d 100644
--- a/Sources/SparkConnect/MiscFunctions.swift
+++ b/Sources/SparkConnect/MiscFunctions.swift
@@ -19,6 +19,54 @@
 
 // MARK: - Misc functions
 
+/// Returns `null` if the input column is `true`, and throws an exception otherwise.
+/// - Parameter col: A ``Column`` that evaluates to the boolean condition to check.
+/// - Returns: A ``Column`` that always evaluates to `null`.
+public func assert_true(_ col: Column) -> Column {
+  return fn("assert_true", col)
+}
+
+/// Returns `null` if the input column is `true`, and throws an exception with the given error
+/// message otherwise.
+/// - Parameters:
+///   - col: A ``Column`` that evaluates to the boolean condition to check.
+///   - errMsg: A ``Column`` that evaluates to the error message to throw.
+/// - Returns: A ``Column`` that always evaluates to `null`.
+public func assert_true(_ col: Column, _ errMsg: Column) -> Column {
+  return fn("assert_true", col, errMsg)
+}
+
+/// Returns `null` if the input column is `true`, and throws an exception with the given error
+/// message otherwise.
+/// - Parameters:
+///   - col: A ``Column`` that evaluates to the boolean condition to check.
+///   - errMsg: A literal error message to throw.
+/// - Returns: A ``Column`` that always evaluates to `null`.
+public func assert_true(_ col: Column, _ errMsg: String) -> Column {
+  return assert_true(col, lit(errMsg))
+}
+
+/// Returns the bit position for the given input column.
+/// - Parameter col: A ``Column`` that evaluates to an integral value.
+/// - Returns: A ``Column`` that evaluates to a long.
+public func bitmap_bit_position(_ col: Column) -> Column {
+  return fn("bitmap_bit_position", col)
+}
+
+/// Returns the bucket number for the given input column.
+/// - Parameter col: A ``Column`` that evaluates to an integral value.
+/// - Returns: A ``Column`` that evaluates to a long.
+public func bitmap_bucket_number(_ col: Column) -> Column {
+  return fn("bitmap_bucket_number", col)
+}
+
+/// Returns the number of set bits in the input bitmap.
+/// - Parameter col: A ``Column`` that evaluates to a binary bitmap.
+/// - Returns: A ``Column`` that evaluates to a long.
+public func bitmap_count(_ col: Column) -> Column {
+  return fn("bitmap_count", col)
+}
+
 /// Returns the current catalog.
 /// - Returns: A ``Column``.
 public func current_catalog() -> Column {
@@ -68,6 +116,19 @@ public func input_file_name() -> Column {
   return fn("input_file_name")
 }
 
+/// Calls a Java method with reflection. The first argument is the class name, the second one is
+/// the method name, and the remaining ones are the arguments passed to the method. This is an
+/// alias of ``reflect(_:)``.
+///
+/// The method is resolved and invoked on the server, so the class must be on the Spark server's
+/// classpath. Never build the class name or the method name from untrusted user input.
+///
+/// - Parameter cols: The class name, the method name, and the method arguments as ``Column``s.
+/// - Returns: A ``Column`` that evaluates to a string.
+public func java_method(_ cols: Column...) -> Column {
+  return fn("java_method", cols)
+}
+
 /// Returns monotonically increasing 64-bit integers. The generated ID is guaranteed to be
 /// monotonically increasing and unique, but not consecutive.
 /// - Returns: A ``Column``.
@@ -75,6 +136,32 @@ public func monotonically_increasing_id() -> Column {
   return fn("monotonically_increasing_id")
 }
 
+/// Throws an exception with the given error message.
+/// - Parameter errMsg: A ``Column`` that evaluates to the error message to throw.
+/// - Returns: A ``Column`` that always evaluates to `null`.
+public func raise_error(_ errMsg: Column) -> Column {
+  return fn("raise_error", errMsg)
+}
+
+/// Throws an exception with the given error message.
+/// - Parameter errMsg: A literal error message to throw.
+/// - Returns: A ``Column`` that always evaluates to `null`.
+public func raise_error(_ errMsg: String) -> Column {
+  return raise_error(lit(errMsg))
+}
+
+/// Calls a Java method with reflection. The first argument is the class name, the second one is
+/// the method name, and the remaining ones are the arguments passed to the method.
+///
+/// The method is resolved and invoked on the server, so the class must be on the Spark server's
+/// classpath. Never build the class name or the method name from untrusted user input.
+///
+/// - Parameter cols: The class name, the method name, and the method arguments as ``Column``s.
+/// - Returns: A ``Column`` that evaluates to a string.
+public func reflect(_ cols: Column...) -> Column {
+  return fn("reflect", cols)
+}
+
 /// Returns the user name of the current execution context.
 /// This requires a Spark 4.0.0+ server.
 /// - Returns: A ``Column``.
@@ -89,6 +176,19 @@ public func spark_partition_id() -> Column {
   return fn("spark_partition_id")
 }
 
+/// Calls a Java method with reflection, returning `null` instead of raising an error when the
+/// invoked method throws an exception. This is the `try` version of ``reflect(_:)`` and requires
+/// a Spark 4.0.0+ server.
+///
+/// The method is resolved and invoked on the server, so the class must be on the Spark server's
+/// classpath. Never build the class name or the method name from untrusted user input.
+///
+/// - Parameter cols: The class name, the method name, and the method arguments as ``Column``s.
+/// - Returns: A ``Column`` that evaluates to a string.
+public func try_reflect(_ cols: Column...) -> Column {
+  return fn("try_reflect", cols)
+}
+
 /// Returns the DDL-formatted type string for the data type of the input.
 /// - Parameter col: A ``Column``.
 /// - Returns: A ``Column``.
diff --git a/Tests/SparkConnectTests/MiscFunctionsTests.swift b/Tests/SparkConnectTests/MiscFunctionsTests.swift
index 2bba096..f6fffa9 100644
--- a/Tests/SparkConnectTests/MiscFunctionsTests.swift
+++ b/Tests/SparkConnectTests/MiscFunctionsTests.swift
@@ -64,6 +64,38 @@ struct MiscFunctionsTests {
     #expect(seededByLiteral.unresolvedFunction.arguments[0].literal.long == 123)
   }
 
+  @Test
+  func errorReflectionAndBitmapFunctions() throws {
+    for (column, name) in [
+      (assert_true(col("a")), "assert_true"),
+      (assert_true(col("a"), col("b")), "assert_true"),
+      (assert_true(col("a"), "error"), "assert_true"),
+      (bitmap_bit_position(col("a")), "bitmap_bit_position"),
+      (bitmap_bucket_number(col("a")), "bitmap_bucket_number"),
+      (bitmap_count(col("a")), "bitmap_count"),
+      (java_method(col("a"), col("b")), "java_method"),
+      (raise_error(col("a")), "raise_error"),
+      (raise_error("error"), "raise_error"),
+      (reflect(col("a"), col("b")), "reflect"),
+      (try_reflect(col("a"), col("b")), "try_reflect"),
+    ] {
+      let expr = column.expr
+      #expect(expr.unresolvedFunction.functionName == name)
+      #expect(!expr.unresolvedFunction.arguments.isEmpty)
+    }
+
+    let assertTrueWithMessage = assert_true(col("a"), "error").expr
+    #expect(assertTrueWithMessage.unresolvedFunction.arguments.count == 2)
+    #expect(assertTrueWithMessage.unresolvedFunction.arguments[1].literal.string == "error")
+
+    let raiseErrorWithMessage = raise_error("error").expr
+    #expect(raiseErrorWithMessage.unresolvedFunction.arguments.count == 1)
+    #expect(raiseErrorWithMessage.unresolvedFunction.arguments[0].literal.string == "error")
+
+    let reflectColumns = reflect(col("a"), col("b"), col("c")).expr
+    #expect(reflectColumns.unresolvedFunction.arguments.count == 3)
+  }
+
   @Test
   func sessionFunctions() async throws {
     let spark = try await SparkSession.builder.getOrCreate()
@@ -140,6 +172,72 @@ struct MiscFunctionsTests {
     await spark.stop()
   }
 
+  @Test
+  func assertTrue() async throws {
+    let spark = try await SparkSession.builder.getOrCreate()
+    let rows = try await spark.range(1).select(
+      assert_true(lit(true)), assert_true(lit(true), lit("error")), assert_true(lit(true), "error")
+    ).collect()
+    #expect(rows == [Row(nil, nil, nil)])
+
+    // The server reports this as `USER_RAISED_EXCEPTION`, which has no `SparkConnectError` case.
+    let error = try await #require(throws: Error.self) {
+      try await spark.range(1).select(assert_true(lit(false), "assert_true failed")).count()
+    }
+    #expect("\(error)".contains("assert_true failed"))
+    await spark.stop()
+  }
+
+  @Test
+  func raiseError() async throws {
+    let spark = try await SparkSession.builder.getOrCreate()
+    for column in [raise_error(lit("raise_error failed")), raise_error("raise_error failed")] {
+      let error = try await #require(throws: Error.self) {
+        try await spark.range(1).select(column).count()
+      }
+      #expect("\(error)".contains("raise_error failed"))
+    }
+    await spark.stop()
+  }
+
+  @Test
+  func reflectFunctions() async throws {
+    let spark = try await SparkSession.builder.getOrCreate()
+    let uuidString = "a5cf6c42-0c85-418f-af6c-3e4e5b1328f2"
+    let rows = try await spark.range(1).select(
+      reflect(lit("java.util.UUID"), lit("fromString"), lit(uuidString)),
+      java_method(lit("java.util.UUID"), lit("fromString"), lit(uuidString))
+    ).collect()
+    #expect(rows == [Row(uuidString, uuidString)])
+    await spark.stop()
+  }
+
+  @Test
+  func tryReflect() async throws {
+    let spark = try await SparkSession.builder.getOrCreate()
+    if await isSparkVersionAtLeast(spark.version, "4.0.0") {
+      let uuidString = "a5cf6c42-0c85-418f-af6c-3e4e5b1328f2"
+      let rows = try await spark.range(1).select(
+        try_reflect(lit("java.util.UUID"), lit("fromString"), lit(uuidString)),
+        try_reflect(lit("java.util.UUID"), lit("fromString"), lit("invalid"))
+      ).collect()
+      #expect(rows == [Row(uuidString, nil)])
+    }
+    await spark.stop()
+  }
+
+  @Test
+  func bitmapFunctions() async throws {
+    let spark = try await SparkSession.builder.getOrCreate()
+    let rows = try await spark.range(1).select(
+      bitmap_bit_position(lit(1)), bitmap_bit_position(lit(123)),
+      bitmap_bucket_number(lit(0)), bitmap_bucket_number(lit(123)),
+      bitmap_count(unhex(lit("1010"))), bitmap_count(unhex(lit("FFFF")))
+    ).collect()
+    #expect(rows == [Row(0, 122, 0, 1, 2, 16)])
+    await spark.stop()
+  }
+
   @Test
   func fileMetadataFunctions() async throws {
     let spark = try await SparkSession.builder.getOrCreate()


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org