You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@spark.apache.org by gu...@apache.org on 2024/10/10 02:28:32 UTC
(spark) branch master updated: [SPARK-48714][PYTHON][FOLLOW-UP] Skip tests if test class is not available
This is an automated email from the ASF dual-hosted git repository.
gurwls223 pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new 0912217b38f4 [SPARK-48714][PYTHON][FOLLOW-UP] Skip tests if test class is not available
0912217b38f4 is described below
commit 0912217b38f474e758f5d0161a24d732a4d58db5
Author: Hyukjin Kwon <gu...@apache.org>
AuthorDate: Thu Oct 10 11:28:21 2024 +0900
[SPARK-48714][PYTHON][FOLLOW-UP] Skip tests if test class is not available
### What changes were proposed in this pull request?
This PR is a followup of https://github.com/apache/spark/pull/47086 that makes PySpark tests skipped if tests classes are unavailable.
### Why are the changes needed?
`./build/sbt package` should be able to run PySpark tests according to https://spark.apache.org/developer-tools.html and https://spark.apache.org/docs/latest/api/python/development/testing.html
### Does this PR introduce _any_ user-facing change?
No, test-only.
### How was this patch tested?
Manually tested:
```bash
build/sbt -Phive clean package
python/run-tests -k --python-executables python3 --testnames 'pyspark.sql.tests.test_dataframe'
```
```
...
Starting test(python3): pyspark.sql.tests.test_dataframe (temp output: /.../spark/python/target/33eca5b9-23e8-4e95-9eca-9f09ce333336/python3__pyspark.sql.tests.test_dataframe__3saz2ymf.log)
Finished test(python3): pyspark.sql.tests.test_dataframe (21s) ... 1 tests were skipped
Tests passed in 21 seconds
Skipped tests in pyspark.sql.tests.test_dataframe with python3:
test_df_merge_into (pyspark.sql.tests.test_dataframe.DataFrameTests.test_df_merge_into) ... skip (0.001s)
```
```bash
build/sbt -Phive clean test:package
python/run-tests -k --python-executables python3 --testnames 'pyspark.sql.tests.test_dataframe'
```
```
Starting test(python3): pyspark.sql.tests.test_dataframe (temp output: /.../spark/python/target/710cf488-f39d-49b4-8b04-70044318ea02/python3__pyspark.sql.tests.test_dataframe___95hp4wt.log)
Finished test(python3): pyspark.sql.tests.test_dataframe (23s)
Tests passed in 23 seconds
```
### Was this patch authored or co-authored using generative AI tooling?
No.
Closes #48404 from HyukjinKwon/SPARK-48714-followup.
Authored-by: Hyukjin Kwon <gu...@apache.org>
Signed-off-by: Hyukjin Kwon <gu...@apache.org>
---
python/pyspark/sql/tests/test_dataframe.py | 13 +++++++++++++
1 file changed, 13 insertions(+)
diff --git a/python/pyspark/sql/tests/test_dataframe.py b/python/pyspark/sql/tests/test_dataframe.py
index 4fb3e7a9192c..2f53ca38743c 100644
--- a/python/pyspark/sql/tests/test_dataframe.py
+++ b/python/pyspark/sql/tests/test_dataframe.py
@@ -15,6 +15,8 @@
# limitations under the License.
#
+import glob
+import os
import pydoc
import shutil
import tempfile
@@ -47,6 +49,7 @@ from pyspark.testing.sqlutils import (
pandas_requirement_message,
pyarrow_requirement_message,
)
+from pyspark.testing.utils import SPARK_HOME
class DataFrameTestsMixin:
@@ -779,6 +782,16 @@ class DataFrameTestsMixin:
)
def test_df_merge_into(self):
+ filename_pattern = (
+ "sql/catalyst/target/scala-*/test-classes/org/apache/spark/sql/connector/catalog/"
+ "InMemoryRowLevelOperationTableCatalog.class"
+ )
+ if not bool(glob.glob(os.path.join(SPARK_HOME, filename_pattern))):
+ raise unittest.SkipTest(
+ "org.apache.spark.sql.connector.catalog.InMemoryRowLevelOperationTableCatalog' "
+ "is not available. Will skip the related tests"
+ )
+
try:
# InMemoryRowLevelOperationTableCatalog is a test catalog that is included in the
# catalyst-test package. If Spark complains that it can't find this class, make sure
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org