You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@spark.apache.org by do...@apache.org on 2023/02/07 05:04:35 UTC
[spark] branch master updated: [SPARK-42268][CONNECT][PYTHON][TESTS][FOLLOWUP] Add `test_simple_udt` for `UserDefinedType`
This is an automated email from the ASF dual-hosted git repository.
dongjoon pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new b99fc5e6075 [SPARK-42268][CONNECT][PYTHON][TESTS][FOLLOWUP] Add `test_simple_udt` for `UserDefinedType`
b99fc5e6075 is described below
commit b99fc5e6075ebe3f8aad5e0d92b091fd84a74765
Author: Ruifeng Zheng <ru...@apache.org>
AuthorDate: Mon Feb 6 21:04:22 2023 -0800
[SPARK-42268][CONNECT][PYTHON][TESTS][FOLLOWUP] Add `test_simple_udt` for `UserDefinedType`
### What changes were proposed in this pull request?
add more tests for UDT
### Why are the changes needed?
https://github.com/apache/spark/pull/39835 added tests for UDT for this conversion:
UDT in Server -> proto -> UDT in Python Client
As per xinrong-meng 's suggestion, this PR adds more tests to cover this:
UDT in Python Client -> proto -> UDT in Server -> proto -> UDT in Python Client
### Does this PR introduce _any_ user-facing change?
No, test-only
### How was this patch tested?
added UT
Closes #39913 from zhengruifeng/connect_more_test_udt.
Authored-by: Ruifeng Zheng <ru...@apache.org>
Signed-off-by: Dongjoon Hyun <do...@apache.org>
---
python/pyspark/sql/tests/connect/test_connect_basic.py | 16 ++++++++++++++++
1 file changed, 16 insertions(+)
diff --git a/python/pyspark/sql/tests/connect/test_connect_basic.py b/python/pyspark/sql/tests/connect/test_connect_basic.py
index 8328f3181a3..6be2a07bc4b 100644
--- a/python/pyspark/sql/tests/connect/test_connect_basic.py
+++ b/python/pyspark/sql/tests/connect/test_connect_basic.py
@@ -2577,6 +2577,22 @@ class SparkConnectBasicTests(SparkConnectSQLTestCase):
).collect(),
)
+ def test_simple_udt(self):
+ from pyspark.ml.linalg import MatrixUDT, VectorUDT
+
+ for schema in [
+ StructType().add("key", LongType()).add("val", PythonOnlyUDT()),
+ StructType().add("key", LongType()).add("val", ArrayType(PythonOnlyUDT())),
+ StructType().add("key", LongType()).add("val", MapType(LongType(), PythonOnlyUDT())),
+ StructType().add("key", LongType()).add("val", PythonOnlyUDT()),
+ StructType().add("key", LongType()).add("vec", VectorUDT()),
+ StructType().add("key", LongType()).add("mat", MatrixUDT()),
+ ]:
+ cdf = self.connect.createDataFrame(data=[], schema=schema)
+ sdf = self.spark.createDataFrame(data=[], schema=schema)
+
+ self.assertEqual(cdf.schema, sdf.schema)
+
def test_simple_udt_from_read(self):
from pyspark.ml.linalg import Matrices, Vectors
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org