You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@spark.apache.org by gu...@apache.org on 2026/03/20 07:21:46 UTC

(spark) branch master updated: [SPARK-55799][PYTHON][TESTS][FOLLOW-UP] Disable some expensive tests for data source simple worker

This is an automated email from the ASF dual-hosted git repository.

gurwls223 pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git


The following commit(s) were added to refs/heads/master by this push:
     new a3c65dad28f8 [SPARK-55799][PYTHON][TESTS][FOLLOW-UP] Disable some expensive tests for data source simple worker
a3c65dad28f8 is described below

commit a3c65dad28f8ece43f91c85809a75381a5d09fdc
Author: Tian Gao <ga...@hotmail.com>
AuthorDate: Fri Mar 20 16:21:35 2026 +0900

    [SPARK-55799][PYTHON][TESTS][FOLLOW-UP] Disable some expensive tests for data source simple worker
    
    ### What changes were proposed in this pull request?
    
    Disable 3 expensive tests for python data source with simple worker.
    
    ### Why are the changes needed?
    
    Our coverage run has been failing because timeout - https://github.com/apache/spark/actions/runs/23290185726 because simple worker tests are slow. We don't reuse workers in simple worker mode, which made python data source pretty expensive.
    
    We have a few options:
    
    1. Increase the timeout limit for coverage run
    2. Separate the simple worker tests to another file
    3. Remove a few tests
    
    I chose 3 because 1 and 2 both have some kind of global effect. The 3 tests I removed were testing the filter itself, instead of workers. There is one test involving filter that's being kept so we can still have coverage for simple worker + filter.
    
    ### Does this PR introduce _any_ user-facing change?
    
    No.
    
    ### How was this patch tested?
    
    Locally passed and faster.
    
    ### Was this patch authored or co-authored using generative AI tooling?
    
    No.
    
    Closes #54913 from gaogaotiantian/disable-some-tests-simple-worker.
    
    Authored-by: Tian Gao <ga...@hotmail.com>
    Signed-off-by: Hyukjin Kwon <gu...@apache.org>
---
 python/pyspark/sql/tests/test_python_datasource.py | 12 ++++++++++++
 1 file changed, 12 insertions(+)

diff --git a/python/pyspark/sql/tests/test_python_datasource.py b/python/pyspark/sql/tests/test_python_datasource.py
index 5a173d3600cd..4bd61a30a4df 100644
--- a/python/pyspark/sql/tests/test_python_datasource.py
+++ b/python/pyspark/sql/tests/test_python_datasource.py
@@ -1364,6 +1364,18 @@ class PythonDataSourceTestsWithSimpleWorker(PythonDataSourceTests):
     def conf(self):
         return super().conf().set("spark.python.use.daemon", "false")
 
+    # Simple Worker is super slow because there's no reuse of workers
+    # so we skip some tests that create many workers
+
+    def test_filter_type(self):
+        pass
+
+    def test_unsupported_filter(self):
+        pass
+
+    def test_filter_value_type(self):
+        pass
+
 
 if __name__ == "__main__":
     from pyspark.testing import main


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org