(spark) branch master updated: [SPARK-58037][PYTHON][TEST][FOLLOWUP] Keep DataFrame golden tests internal

[ga...@apache.org]

This is an automated email from the ASF dual-hosted git repository.

gaogaotiantian pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git


The following commit(s) were added to refs/heads/master by this push:
     new 0c1fd8fc590f [SPARK-58037][PYTHON][TEST][FOLLOWUP] Keep DataFrame golden tests internal
0c1fd8fc590f is described below

commit 0c1fd8fc590f01cf5814c8f035aec638a2e6b092
Author: Mihailo Aleksic <mi...@databricks.com>
AuthorDate: Wed Jul 29 17:47:31 2026 -0700

    [SPARK-58037][PYTHON][TEST][FOLLOWUP] Keep DataFrame golden tests internal
    
    ### What changes were proposed in this pull request?
     This PR follows up on #57122 by keeping the PySpark DataFrame golden test helper internal to the test package:
    
      - Remove the `python/MANIFEST.in` rule that included `pyspark/sql/tests/df_golden` files in packaged Python artifacts.
      - Move `pyspark.testing.df_golden` to `pyspark.sql.tests.df_golden.df_golden`.
      - Update the DataFrame golden tests to import the helper from its new internal test location.
    
    ### Why are the changes needed?
    `pyspark.testing` is user-accessible, so putting the golden-file helper there makes it look like a supported PySpark testing API. The helper is only intended for Spark's internal tests. The golden test files also should not be shipped to PySpark users.
    
    ### Does this PR introduce _any_ user-facing change?
    No.
    
    ### How was this patch tested?
    Test only change.
    
    ### Was this patch authored or co-authored using generative AI tooling?
    Yes.
    
    Closes #57639 from mihailoale-db/spark-58037-python-tests-followup.
    
    Authored-by: Mihailo Aleksic <mi...@databricks.com>
    Signed-off-by: Tian Gao <ga...@hotmail.com>
---
 python/MANIFEST.in                                             | 1 -
 python/pyspark/{testing => sql/tests/df_golden}/df_golden.py   | 0
 python/pyspark/sql/tests/df_golden/test_df_golden.py           | 4 ++--
 python/pyspark/sql/tests/df_golden/test_df_golden_framework.py | 8 ++++----
 4 files changed, 6 insertions(+), 7 deletions(-)

diff --git a/python/MANIFEST.in b/python/MANIFEST.in
index 4901fe937966..82979a0344c3 100644
--- a/python/MANIFEST.in
+++ b/python/MANIFEST.in
@@ -17,7 +17,6 @@
 # Reference: https://setuptools.pypa.io/en/latest/userguide/miscellaneous.html
 
 recursive-include pyspark *.pyi py.typed *.json
-recursive-include pyspark/sql/tests/df_golden *.test *.py
 recursive-include deps/jars *.jar
 graft deps/bin
 recursive-include deps/sbin spark-config.sh spark-daemon.sh start-history-server.sh stop-history-server.sh
diff --git a/python/pyspark/testing/df_golden.py b/python/pyspark/sql/tests/df_golden/df_golden.py
similarity index 100%
rename from python/pyspark/testing/df_golden.py
rename to python/pyspark/sql/tests/df_golden/df_golden.py
diff --git a/python/pyspark/sql/tests/df_golden/test_df_golden.py b/python/pyspark/sql/tests/df_golden/test_df_golden.py
index ee9b8a240e28..311491c5d7ed 100644
--- a/python/pyspark/sql/tests/df_golden/test_df_golden.py
+++ b/python/pyspark/sql/tests/df_golden/test_df_golden.py
@@ -23,7 +23,7 @@ the PySpark DataFrame API.  Each ``.test`` file in this directory describes a
 list of test cases; every case points to a standalone script under
 ``scripts/`` that builds the DataFrame under test.  The expected outputs live
 inline in the ``.test`` file, which doubles as the golden file.  See
-``pyspark.testing.df_golden`` for the file format.
+``pyspark.sql.tests.df_golden.df_golden`` for the file format.
 
 Running the tests::
 
@@ -50,7 +50,7 @@ regenerate the golden files with the steps above.
 import os
 
 from pyspark.testing.connectutils import ReusedConnectTestCase
-from pyspark.testing.df_golden import run_golden_test
+from pyspark.sql.tests.df_golden.df_golden import run_golden_test
 
 
 _THIS_DIR = os.path.dirname(os.path.abspath(__file__))
diff --git a/python/pyspark/sql/tests/df_golden/test_df_golden_framework.py b/python/pyspark/sql/tests/df_golden/test_df_golden_framework.py
index 052a52182af2..5949093e24ef 100644
--- a/python/pyspark/sql/tests/df_golden/test_df_golden_framework.py
+++ b/python/pyspark/sql/tests/df_golden/test_df_golden_framework.py
@@ -16,7 +16,7 @@
 #
 
 """
-Unit tests for the ``pyspark.testing.df_golden`` write/validation machinery.
+Unit tests for the ``pyspark.sql.tests.df_golden.df_golden`` write/validation machinery.
 
 These exercise the pure ``.test`` file plumbing -- parsing, serialization,
 validation, output normalization and result rendering -- without a Spark
@@ -28,7 +28,7 @@ import os
 import tempfile
 import unittest
 
-from pyspark.testing.df_golden import (
+from pyspark.sql.tests.df_golden.df_golden import (
     _compare_case,
     _regenerate_case,
     _validate_test_file,
@@ -448,7 +448,7 @@ class DFGoldenFrameworkTests(unittest.TestCase):
             from pyspark.sql.types import DoubleType, FloatType
         except Exception:
             self.skipTest("pyspark.sql.types unavailable in this environment")
-        from pyspark.testing.df_golden import _format_value
+        from pyspark.sql.tests.df_golden.df_golden import _format_value
 
         with self.assertRaisesRegex(AssertionError, "not supported yet"):
             _format_value(0.1, FloatType())
@@ -501,7 +501,7 @@ class DFGoldenFrameworkTests(unittest.TestCase):
             from pyspark.sql.types import StringType
         except Exception:
             self.skipTest("pyspark.sql.types unavailable in this environment")
-        from pyspark.testing.df_golden import _format_value
+        from pyspark.sql.tests.df_golden.df_golden import _format_value
 
         self.assertEqual(_format_value("ok", StringType()), "ok")
         for bad in ("a\tb", "a\nb"):


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org