(spark-connect-rust) 01/01: [SPARK-58983] Run the WASM packer against the reference pyspark in CI

[gu...@apache.org]

This is an automated email from the ASF dual-hosted git repository.

HyukjinKwon pushed a commit to branch fix-ci-actions
in repository https://gitbox.apache.org/repos/asf/spark-connect-rust.git

commit 8bfb59f6ba408b853bd83f7afd3047ae78079a79
Author: Hyukjin Kwon <hy...@databricks.com>
AuthorDate: Tue Aug 25 13:00:06 2026 +0900

    [SPARK-58983] Run the WASM packer against the reference pyspark in CI
    
    The `wasm-udf` job's "Packer end-to-end" step pointed
    SPARK_CONNECT_WASM_PACKER_PATH at ./python, which put the Rust-backed
    `pyspark` skin on the packer's import path. `pyspark_wasm_udf.pack` needs
    `pyspark.cloudpickle` and `pyspark.sql.types._parse_datatype_json_value`,
    and must emit a reference `DataType` that executors unpickle -- none of
    which the skin provides. Isolate `pyspark_wasm_udf` in a `.packer` dir so
    `import pyspark` resolves to the reference pyspark installed earlier in
    the job. Also refresh the pack.py docstring to say we use the reference
    `pyspark.cloudpickle`.
    
    Co-authored-by: Isaac <no...@databricks.com>
---
 .github/workflows/rust.yml      | 13 +++++++++++--
 python/pyspark_wasm_udf/pack.py |  8 ++++----
 2 files changed, 15 insertions(+), 6 deletions(-)

diff --git a/.github/workflows/rust.yml b/.github/workflows/rust.yml
index d693d59..8d22123 100644
--- a/.github/workflows/rust.yml
+++ b/.github/workflows/rust.yml
@@ -122,6 +122,15 @@ jobs:
           cargo build -p examples --bin wasm_udf_macro --features examples/wasm-udf-macro
 
       - name: Packer end-to-end (Rust -> cloudpickle command)
+        # The packer runs `python -m pyspark_wasm_udf.pack`, which imports the
+        # reference `pyspark` (installed above) for `pyspark.cloudpickle` and
+        # `pyspark.sql.types._parse_datatype_json_value`, and produces a
+        # reference `DataType` the executors unpickle. Put *only* the
+        # `pyspark_wasm_udf` package on the packer path (not ./python, whose
+        # `pyspark` skin is Rust-backed and would shadow the reference).
         env:
-          SPARK_CONNECT_WASM_PACKER_PATH: ${{ github.workspace }}/python
-        run: cargo test -p apache-spark-connect --features wasm-udf -- --ignored packer_produces
+          SPARK_CONNECT_WASM_PACKER_PATH: ${{ github.workspace }}/.packer
+        run: |
+          mkdir -p .packer
+          cp -r python/pyspark_wasm_udf .packer/
+          cargo test -p apache-spark-connect --features wasm-udf -- --ignored packer_produces
diff --git a/python/pyspark_wasm_udf/pack.py b/python/pyspark_wasm_udf/pack.py
index 9b3c305..a38a7cb 100644
--- a/python/pyspark_wasm_udf/pack.py
+++ b/python/pyspark_wasm_udf/pack.py
@@ -35,10 +35,10 @@ generically with :func:`pyspark.sql.types._parse_datatype_json_value`, so any
 Spark type is supported without a token table.
 
 The command mirrors ``pyspark.sql.connect.expressions.PythonUDF``:
-``cloudpickle.dumps((func, output_type))``. We use the standalone PyPI
-``cloudpickle`` (byte-for-byte the same as ``pyspark.cloudpickle``) and register
-:mod:`pyspark_wasm_udf` for pickling **by value**, so executors need not have
-this package installed.
+``cloudpickle.dumps((func, output_type))``. We use the bundled
+``pyspark.cloudpickle`` (the same cloudpickle the reference client ships) and
+register :mod:`pyspark_wasm_udf` for pickling **by value**, so executors need not
+have this package installed.
 """
 
 import base64


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org