You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@spark.apache.org by sa...@apache.org on 2025/08/12 02:50:15 UTC

(spark-connect-rust) branch master updated: [SPARK-53001][TESTS] Fix test_df_unpivot to pass with Spark 4.0

This is an automated email from the ASF dual-hosted git repository.

sarutak pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark-connect-rust.git


The following commit(s) were added to refs/heads/master by this push:
     new ca629aa  [SPARK-53001][TESTS] Fix test_df_unpivot to pass with Spark 4.0
ca629aa is described below

commit ca629aa34fb452373555855a277d2a59e2bd5693
Author: Kousuke Saruta <sa...@amazon.co.jp>
AuthorDate: Tue Aug 12 11:50:00 2025 +0900

    [SPARK-53001][TESTS] Fix test_df_unpivot to pass with Spark 4.0
    
    # Description
    
    This PR proposes to fix the issue that `test_df_unpivot` doesn't  pass with Spark 4.0.
    ```
    ---- dataframe::tests::test_df_unpivot stdout ----
    SparkSession Setup
    
    thread 'dataframe::tests::test_df_unpivot' panicked at crates/connect/src/dataframe.rs:2463:9:
    assertion `left == right` failed
      left: RecordBatch { schema: Schema { fields: [Field { name: "id", data_type: Int64, nullable: false, dict_id: 0, dict_is_ordered: false, metadata: {} }, Field { name: "var", data_type: Utf8, nullable: false, dict_id: 0, dict_is_ordered: false, metadata: {} }, Field { name: "val", data_type: Float32, nullable: false, dict_id: 0, dict_is_ordered: false, metadata: {} }], metadata: {} }, columns: [PrimitiveArray<Int64>
    [
      1,
      1,
      2,
      2,
    ], StringArray
    [
      "int",
      "float",
      "int",
      "float",
    ], PrimitiveArray<Float32>
    [
      11.0,
      1.1,
      12.0,
      1.2,
    ]], row_count: 4 }
     right: RecordBatch { schema: Schema { fields: [Field { name: "id", data_type: Int64, nullable: false, dict_id: 0, dict_is_ordered: false, metadata: {} }, Field { name: "var", data_type: Utf8, nullable: false, dict_id: 0, dict_is_ordered: false, metadata: {} }, Field { name: "val", data_type: Float64, nullable: false, dict_id: 0, dict_is_ordered: false, metadata: {} }], metadata: {} }, columns: [PrimitiveArray<Int64>
    [
      1,
      1,
      2,
      2,
    ], StringArray
    [
      "int",
      "float",
      "int",
      "float",
    ], PrimitiveArray<Float64>
    [
      11.0,
      1.100000023841858,
      12.0,
      1.2000000476837158,
    ]], row_count: 4 }
    ```
    
    As of Spark 4.0, ANSI mode is enabled by default but this test doesn't consider  it.
    To fix this issue, I tweaked the test so that the test pass in both ANSI mode is enabled or not.
    
    ## Related Issue(s)
    SPARK-53001
    
    Closes #4 from sarutak/fix-unpivot-test.
    
    Authored-by: Kousuke Saruta <sa...@amazon.co.jp>
    Signed-off-by: Kousuke Saruta <sa...@amazon.co.jp>
---
 crates/connect/src/dataframe.rs | 14 ++++++++------
 1 file changed, 8 insertions(+), 6 deletions(-)

diff --git a/crates/connect/src/dataframe.rs b/crates/connect/src/dataframe.rs
index a8f4cd8..a69f655 100644
--- a/crates/connect/src/dataframe.rs
+++ b/crates/connect/src/dataframe.rs
@@ -2445,12 +2445,14 @@ mod tests {
 
         let df = spark.create_dataframe(&data)?;
 
-        let df = df.unpivot(
-            [col("id")],
-            Some(vec![col("int"), col("float")]),
-            "var",
-            "val",
-        );
+        let df = df
+            .unpivot(
+                [col("id")],
+                Some(vec![col("int"), col("float")]),
+                "var",
+                "val",
+            )
+            .select(vec![col("id"), col("var"), col("val").cast("float")]);
 
         let res = df.collect().await?;
 


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org