You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@spark.apache.org by do...@apache.org on 2023/02/07 05:04:35 UTC

[spark] branch master updated: [SPARK-42268][CONNECT][PYTHON][TESTS][FOLLOWUP] Add `test_simple_udt` for `UserDefinedType`

This is an automated email from the ASF dual-hosted git repository.

dongjoon pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git


The following commit(s) were added to refs/heads/master by this push:
     new b99fc5e6075 [SPARK-42268][CONNECT][PYTHON][TESTS][FOLLOWUP] Add `test_simple_udt` for `UserDefinedType`
b99fc5e6075 is described below

commit b99fc5e6075ebe3f8aad5e0d92b091fd84a74765
Author: Ruifeng Zheng <ru...@apache.org>
AuthorDate: Mon Feb 6 21:04:22 2023 -0800

    [SPARK-42268][CONNECT][PYTHON][TESTS][FOLLOWUP] Add `test_simple_udt` for `UserDefinedType`
    
    ### What changes were proposed in this pull request?
    add more tests for UDT
    
    ### Why are the changes needed?
    https://github.com/apache/spark/pull/39835 added tests for UDT for this conversion:
    UDT in Server -> proto -> UDT in Python Client
    
    As per xinrong-meng 's suggestion, this PR adds more tests to cover this:
    UDT in Python Client -> proto -> UDT in Server -> proto -> UDT in Python Client
    
    ### Does this PR introduce _any_ user-facing change?
    No, test-only
    
    ### How was this patch tested?
    added UT
    
    Closes #39913 from zhengruifeng/connect_more_test_udt.
    
    Authored-by: Ruifeng Zheng <ru...@apache.org>
    Signed-off-by: Dongjoon Hyun <do...@apache.org>
---
 python/pyspark/sql/tests/connect/test_connect_basic.py | 16 ++++++++++++++++
 1 file changed, 16 insertions(+)

diff --git a/python/pyspark/sql/tests/connect/test_connect_basic.py b/python/pyspark/sql/tests/connect/test_connect_basic.py
index 8328f3181a3..6be2a07bc4b 100644
--- a/python/pyspark/sql/tests/connect/test_connect_basic.py
+++ b/python/pyspark/sql/tests/connect/test_connect_basic.py
@@ -2577,6 +2577,22 @@ class SparkConnectBasicTests(SparkConnectSQLTestCase):
             ).collect(),
         )
 
+    def test_simple_udt(self):
+        from pyspark.ml.linalg import MatrixUDT, VectorUDT
+
+        for schema in [
+            StructType().add("key", LongType()).add("val", PythonOnlyUDT()),
+            StructType().add("key", LongType()).add("val", ArrayType(PythonOnlyUDT())),
+            StructType().add("key", LongType()).add("val", MapType(LongType(), PythonOnlyUDT())),
+            StructType().add("key", LongType()).add("val", PythonOnlyUDT()),
+            StructType().add("key", LongType()).add("vec", VectorUDT()),
+            StructType().add("key", LongType()).add("mat", MatrixUDT()),
+        ]:
+            cdf = self.connect.createDataFrame(data=[], schema=schema)
+            sdf = self.spark.createDataFrame(data=[], schema=schema)
+
+            self.assertEqual(cdf.schema, sdf.schema)
+
     def test_simple_udt_from_read(self):
         from pyspark.ml.linalg import Matrices, Vectors
 


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org