You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@spark.apache.org by sa...@apache.org on 2026/06/10 08:02:11 UTC
(spark) branch master updated: [SPARK-57348][PYTHON][TESTS] Replace sql_keywords doctest show() with columns check
This is an automated email from the ASF dual-hosted git repository.
sarutak pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new e598b0c09d7b [SPARK-57348][PYTHON][TESTS] Replace sql_keywords doctest show() with columns check
e598b0c09d7b is described below
commit e598b0c09d7b1a2982f5a60b34f7904cffe2c053
Author: Kousuke Saruta <sa...@amazon.co.jp>
AuthorDate: Wed Jun 10 17:01:49 2026 +0900
[SPARK-57348][PYTHON][TESTS] Replace sql_keywords doctest show() with columns check
### What changes were proposed in this pull request?
Replace the `show()`-based doctest for `sql_keywords()` with a `.columns` check.
### Why are the changes needed?
SPARK-57133 (#56247) added 7 new non-reserved keywords (BIN, WIDTH, ALIGN, etc.), which changed the top-20 row output of `sql_keywords().show()` and consequently the column width in the formatted output. This broke the `pyspark-connect-old-client` CI job, which runs `branch-4.0` client tests against a `master` server. The `branch-4.0` doctest still expects the old column width.
https://github.com/sarutak/spark/actions/runs/27188469096/job/80265105548
```
**********************************************************************
File "/__w/spark/spark-4.0/python/pyspark/sql/connect/tvf.py", line ?, in pyspark.sql.connect.tvf.TableValuedFunction.sql_keywords
Failed example:
spark.tvf.sql_keywords().show()
Expected:
+-------------+--------+
| keyword|reserved|
+-------------+--------+
...
+-------------+--------+...
Got:
+----------+--------+
| keyword|reserved|
+----------+--------+
| ADD| false|
| AFTER| false|
| AGGREGATE| false|
| ALIGN| false|
| ALL| false|
| ALTER| false|
| ALWAYS| false|
| ANALYZE| false|
| AND| false|
| ANTI| false|
| ANY| false|
| ANY_VALUE| false|
| APPROX| false|
| ARCHIVE| false|
| ARRAY| false|
| AS| false|
| ASC| false|
|ASENSITIVE| false|
| AT| false|
| ATOMIC| false|
+----------+--------+
only showing top 20 rows
**********************************************************************
1 of 1 in pyspark.sql.connect.tvf.TableValuedFunction.sql_keywords
***Test Failed*** 1 failures.
```
The `show()` output is inherently fragile for this TVF because any keyword addition changes the formatting. Since a dedicated unittest (`test_sql_keywords` in `test_tvf.py`) already verifies the full output via `assertDataFrameEqual`, the doctest only needs to confirm that the method returns a valid DataFrame. Using `.columns` achieves this without being sensitive to keyword list changes.
Currently the only branch affected in CI is `branch-4.0` (via `pyspark-connect-old-client`), but this change is made on `master` for consistency and will be backported to older branches.
### Does this PR introduce *any* user-facing change?
No.
### How was this patch tested?
Existing `test_sql_keywords` unittest continues to pass.
### Was this patch authored or co-authored using generative AI tooling?
Kiro CLI / Claude
Closes #56406 from sarutak/fix-sql-keywords-doctest.
Authored-by: Kousuke Saruta <sa...@amazon.co.jp>
Signed-off-by: Kousuke Saruta <sa...@apache.org>
---
python/pyspark/sql/tvf.py | 8 ++------
1 file changed, 2 insertions(+), 6 deletions(-)
diff --git a/python/pyspark/sql/tvf.py b/python/pyspark/sql/tvf.py
index 97d8ce59913b..90874578a356 100644
--- a/python/pyspark/sql/tvf.py
+++ b/python/pyspark/sql/tvf.py
@@ -550,12 +550,8 @@ class TableValuedFunction:
Examples
--------
- >>> spark.tvf.sql_keywords().show()
- +-------------+--------+
- | keyword|reserved|
- +-------------+--------+
- ...
- +-------------+--------+...
+ >>> spark.tvf.sql_keywords().columns
+ ['keyword', 'reserved']
"""
return self._fn("sql_keywords")
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org