You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@spark.apache.org by sa...@apache.org on 2025/08/12 02:50:15 UTC
(spark-connect-rust) branch master updated: [SPARK-53001][TESTS] Fix test_df_unpivot to pass with Spark 4.0
This is an automated email from the ASF dual-hosted git repository.
sarutak pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark-connect-rust.git
The following commit(s) were added to refs/heads/master by this push:
new ca629aa [SPARK-53001][TESTS] Fix test_df_unpivot to pass with Spark 4.0
ca629aa is described below
commit ca629aa34fb452373555855a277d2a59e2bd5693
Author: Kousuke Saruta <sa...@amazon.co.jp>
AuthorDate: Tue Aug 12 11:50:00 2025 +0900
[SPARK-53001][TESTS] Fix test_df_unpivot to pass with Spark 4.0
# Description
This PR proposes to fix the issue that `test_df_unpivot` doesn't pass with Spark 4.0.
```
---- dataframe::tests::test_df_unpivot stdout ----
SparkSession Setup
thread 'dataframe::tests::test_df_unpivot' panicked at crates/connect/src/dataframe.rs:2463:9:
assertion `left == right` failed
left: RecordBatch { schema: Schema { fields: [Field { name: "id", data_type: Int64, nullable: false, dict_id: 0, dict_is_ordered: false, metadata: {} }, Field { name: "var", data_type: Utf8, nullable: false, dict_id: 0, dict_is_ordered: false, metadata: {} }, Field { name: "val", data_type: Float32, nullable: false, dict_id: 0, dict_is_ordered: false, metadata: {} }], metadata: {} }, columns: [PrimitiveArray<Int64>
[
1,
1,
2,
2,
], StringArray
[
"int",
"float",
"int",
"float",
], PrimitiveArray<Float32>
[
11.0,
1.1,
12.0,
1.2,
]], row_count: 4 }
right: RecordBatch { schema: Schema { fields: [Field { name: "id", data_type: Int64, nullable: false, dict_id: 0, dict_is_ordered: false, metadata: {} }, Field { name: "var", data_type: Utf8, nullable: false, dict_id: 0, dict_is_ordered: false, metadata: {} }, Field { name: "val", data_type: Float64, nullable: false, dict_id: 0, dict_is_ordered: false, metadata: {} }], metadata: {} }, columns: [PrimitiveArray<Int64>
[
1,
1,
2,
2,
], StringArray
[
"int",
"float",
"int",
"float",
], PrimitiveArray<Float64>
[
11.0,
1.100000023841858,
12.0,
1.2000000476837158,
]], row_count: 4 }
```
As of Spark 4.0, ANSI mode is enabled by default but this test doesn't consider it.
To fix this issue, I tweaked the test so that the test pass in both ANSI mode is enabled or not.
## Related Issue(s)
SPARK-53001
Closes #4 from sarutak/fix-unpivot-test.
Authored-by: Kousuke Saruta <sa...@amazon.co.jp>
Signed-off-by: Kousuke Saruta <sa...@amazon.co.jp>
---
crates/connect/src/dataframe.rs | 14 ++++++++------
1 file changed, 8 insertions(+), 6 deletions(-)
diff --git a/crates/connect/src/dataframe.rs b/crates/connect/src/dataframe.rs
index a8f4cd8..a69f655 100644
--- a/crates/connect/src/dataframe.rs
+++ b/crates/connect/src/dataframe.rs
@@ -2445,12 +2445,14 @@ mod tests {
let df = spark.create_dataframe(&data)?;
- let df = df.unpivot(
- [col("id")],
- Some(vec![col("int"), col("float")]),
- "var",
- "val",
- );
+ let df = df
+ .unpivot(
+ [col("id")],
+ Some(vec![col("int"), col("float")]),
+ "var",
+ "val",
+ )
+ .select(vec![col("id"), col("var"), col("val").cast("float")]);
let res = df.collect().await?;
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org