You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@spark.apache.org by al...@apache.org on 2025/05/23 18:27:29 UTC
(spark) branch master updated: [SPARK-51430][PYTHON] Stop PySpark context logger from propagating logs to stdout
This is an automated email from the ASF dual-hosted git repository.
allisonwang pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new 8043304b9d42 [SPARK-51430][PYTHON] Stop PySpark context logger from propagating logs to stdout
8043304b9d42 is described below
commit 8043304b9d42e507047c18a5738e8d0435dfac16
Author: Allison Wang <al...@databricks.com>
AuthorDate: Fri May 23 11:27:15 2025 -0700
[SPARK-51430][PYTHON] Stop PySpark context logger from propagating logs to stdout
### What changes were proposed in this pull request?
This PR stops PySpark context logger from propagating logs to stdout.
### Why are the changes needed?
To improve user experience. Currently you can see this logging message in pyspark shell
```
In [1]: from pyspark.sql.functions import col
In [2]: spark.range(10).select(col("id2")).show()
{"ts": "2025-03-06 16:55:36.678", "level": "ERROR", "logger": "DataFrameQueryContextLogger", "msg": "[UNRESOLVED_COLUMN.WITH_SUGGESTION] A column, variable, or function parameter with name `id2` cannot be resolved. Did you mean one of the following? [`id`]. SQLSTATE: 42703", "context": {"file": "<ipython-input-2-bac6211f25a7>", "line": "1", "fragment": "col", "errorClass": "UNRESOLVED_COLUMN.WITH_SUGGESTION"}, "exception": {"class": "Py4JJavaError", "msg": "An error occurred while cal [...]
```
### Does this PR introduce _any_ user-facing change?
No
### How was this patch tested?
Existing tests
### Was this patch authored or co-authored using generative AI tooling?
No
Closes #50198 from allisonwang-db/spark-51430-logging.
Authored-by: Allison Wang <al...@databricks.com>
Signed-off-by: Allison Wang <al...@databricks.com>
---
python/pyspark/errors/exceptions/base.py | 2 ++
1 file changed, 2 insertions(+)
diff --git a/python/pyspark/errors/exceptions/base.py b/python/pyspark/errors/exceptions/base.py
index 347f8db31847..4fa7ea92e34c 100644
--- a/python/pyspark/errors/exceptions/base.py
+++ b/python/pyspark/errors/exceptions/base.py
@@ -159,6 +159,7 @@ class PySparkException(Exception):
if context:
if context.contextType().name == "DataFrame":
logger = PySparkLogger.getLogger("DataFrameQueryContextLogger")
+ logger.propagate = False
call_site = context.callSite().split(":")
line = call_site[1] if len(call_site) == 2 else ""
logger.exception(
@@ -170,6 +171,7 @@ class PySparkException(Exception):
)
else:
logger = PySparkLogger.getLogger("SQLQueryContextLogger")
+ logger.propagate = False
logger.exception(
self.getMessage(),
errorClass=self.getCondition(),
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org