You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by "ueshin (via GitHub)" <gi...@apache.org> on 2025/11/07 02:48:13 UTC
[PR] [SPARK-54229][PYTHON] Make PySparkLogger in UDFs store one log entry per log function call [spark]
ueshin opened a new pull request, #52926:
URL: https://github.com/apache/spark/pull/52926
### What changes were proposed in this pull request?
Makes `PySparkLogger` in UDFs store one log entry per log function call.
### Why are the changes needed?
Currently if `PySparkLogger` is used in UDFs, it will produce two entries per one log function call because it automatically adds a handler that writes to `sys.stderr`, which causes two entries.
It doesn't need the additional handler if it's in the `capture_outputs` context.
<details>
<summary>example</summary>
```python
>>> from pyspark.sql.functions import *
>>> from pyspark.logger import PySparkLogger
>>>
>>> @udf
... def pyspark_logger_test_udf(x):
... logger = PySparkLogger.getLogger("test")
... logger.warn(f"WARN level message: {x}", x=x)
... return str(x)
...
>>>
>>> spark.conf.set("spark.sql.pyspark.worker.logging.enabled", True)
>>>
>>> spark.range(1).select(pyspark_logger_test_udf("id")).show()
...
```
</details>
- before
```py
>>> spark.table("system.session.python_worker_logs").orderBy("ts").show(truncate=False)
+--------------------------+-------+----------------------------------------------------------------------------------------------------------------------------+----------------------------------------------+---------+------+
|ts |level |msg |context |exception|logger|
+--------------------------+-------+----------------------------------------------------------------------------------------------------------------------------+----------------------------------------------+---------+------+
|2025-11-06 18:40:03.658127|WARNING|WARN level message: 0 |{func_name -> pyspark_logger_test_udf, x -> 0}|NULL |test |
|2025-11-06 18:40:03.66424 |ERROR |{"ts": "2025-11-06 18:40:03.658", "level": "WARNING", "logger": "test", "msg": "WARN level message: 0", "context": {"x": 0}}|{func_name -> pyspark_logger_test_udf} |NULL |stderr|
+--------------------------+-------+----------------------------------------------------------------------------------------------------------------------------+----------------------------------------------+---------+------+
```
- after
```py
>>> spark.table("system.session.python_worker_logs").orderBy("ts").show(truncate=False)
+--------------------------+-------+---------------------+----------------------------------------------+---------+------+
|ts |level |msg |context |exception|logger|
+--------------------------+-------+---------------------+----------------------------------------------+---------+------+
|2025-11-06 18:41:48.601256|WARNING|WARN level message: 0|{func_name -> pyspark_logger_test_udf, x -> 0}|NULL |test |
+--------------------------+-------+---------------------+----------------------------------------------+---------+------+
```
### Does this PR introduce _any_ user-facing change?
Yes, `PySparkLogger` in UDFs will store one log entry per one log function call.
### How was this patch tested?
Added the related tests.
### Was this patch authored or co-authored using generative AI tooling?
No.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-54229][PYTHON] Make PySparkLogger in UDFs store one log entry per log function call [spark]
Posted by "ueshin (via GitHub)" <gi...@apache.org>.
ueshin commented on PR #52926:
URL: https://github.com/apache/spark/pull/52926#issuecomment-3504728820
Thanks! merging to master/4.1.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-54229][PYTHON] Make PySparkLogger in UDFs store one log entry per log function call [spark]
Posted by "ueshin (via GitHub)" <gi...@apache.org>.
ueshin closed pull request #52926: [SPARK-54229][PYTHON] Make PySparkLogger in UDFs store one log entry per log function call
URL: https://github.com/apache/spark/pull/52926
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org