You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@spark.apache.org by gu...@apache.org on 2026/03/20 07:21:46 UTC
(spark) branch master updated: [SPARK-55799][PYTHON][TESTS][FOLLOW-UP] Disable some expensive tests for data source simple worker
This is an automated email from the ASF dual-hosted git repository.
gurwls223 pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new a3c65dad28f8 [SPARK-55799][PYTHON][TESTS][FOLLOW-UP] Disable some expensive tests for data source simple worker
a3c65dad28f8 is described below
commit a3c65dad28f8ece43f91c85809a75381a5d09fdc
Author: Tian Gao <ga...@hotmail.com>
AuthorDate: Fri Mar 20 16:21:35 2026 +0900
[SPARK-55799][PYTHON][TESTS][FOLLOW-UP] Disable some expensive tests for data source simple worker
### What changes were proposed in this pull request?
Disable 3 expensive tests for python data source with simple worker.
### Why are the changes needed?
Our coverage run has been failing because timeout - https://github.com/apache/spark/actions/runs/23290185726 because simple worker tests are slow. We don't reuse workers in simple worker mode, which made python data source pretty expensive.
We have a few options:
1. Increase the timeout limit for coverage run
2. Separate the simple worker tests to another file
3. Remove a few tests
I chose 3 because 1 and 2 both have some kind of global effect. The 3 tests I removed were testing the filter itself, instead of workers. There is one test involving filter that's being kept so we can still have coverage for simple worker + filter.
### Does this PR introduce _any_ user-facing change?
No.
### How was this patch tested?
Locally passed and faster.
### Was this patch authored or co-authored using generative AI tooling?
No.
Closes #54913 from gaogaotiantian/disable-some-tests-simple-worker.
Authored-by: Tian Gao <ga...@hotmail.com>
Signed-off-by: Hyukjin Kwon <gu...@apache.org>
---
python/pyspark/sql/tests/test_python_datasource.py | 12 ++++++++++++
1 file changed, 12 insertions(+)
diff --git a/python/pyspark/sql/tests/test_python_datasource.py b/python/pyspark/sql/tests/test_python_datasource.py
index 5a173d3600cd..4bd61a30a4df 100644
--- a/python/pyspark/sql/tests/test_python_datasource.py
+++ b/python/pyspark/sql/tests/test_python_datasource.py
@@ -1364,6 +1364,18 @@ class PythonDataSourceTestsWithSimpleWorker(PythonDataSourceTests):
def conf(self):
return super().conf().set("spark.python.use.daemon", "false")
+ # Simple Worker is super slow because there's no reuse of workers
+ # so we skip some tests that create many workers
+
+ def test_filter_type(self):
+ pass
+
+ def test_unsupported_filter(self):
+ pass
+
+ def test_filter_value_type(self):
+ pass
+
if __name__ == "__main__":
from pyspark.testing import main
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org