You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Hyukjin Kwon (Jira)" <ji...@apache.org> on 2024/07/18 05:05:00 UTC
[jira] [Resolved] (SPARK-48752) Introduce `pyspark.logging` for improved structured logging for PySpark
[ https://issues.apache.org/jira/browse/SPARK-48752?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Hyukjin Kwon resolved SPARK-48752.
----------------------------------
Fix Version/s: 4.0.0
Resolution: Fixed
Issue resolved by pull request 47145
[https://github.com/apache/spark/pull/47145]
> Introduce `pyspark.logging` for improved structured logging for PySpark
> -----------------------------------------------------------------------
>
> Key: SPARK-48752
> URL: https://issues.apache.org/jira/browse/SPARK-48752
> Project: Spark
> Issue Type: Bug
> Components: PySpark
> Affects Versions: 4.0.0
> Reporter: Haejoon Lee
> Assignee: Haejoon Lee
> Priority: Major
> Labels: pull-request-available
> Fix For: 4.0.0
>
>
> We introduced structured logging from [#45729|https://github.com/apache/spark/pull/45729], but PySpark log is still hard to figure out in the current structured log, because it is hidden and mixed within bunch of complex JVM stacktraces and it's also not very Python-friendly:
>
> {code:java}
> {
> "ts": "2024-06-28T10:53:48.528Z",
> "level": "ERROR",
> "msg": "Exception in task 7.0 in stage 0.0 (TID 7)",
> "context": {
> "task_name": "task 7.0 in stage 0.0 (TID 7)"
> },
> "exception": {
> "class": "org.apache.spark.SparkArithmeticException",
> "msg": "[DIVIDE_BY_ZERO] Division by zero. Use `try_divide` to tolerate divisor being 0 and return NULL instead. If necessary set \"spark.sql.ansi.enabled\" to \"false\" to bypass this error. SQLSTATE: 22012\n== DataFrame ==\n\"__truediv__\" was called from\n/.../spark/python/test_error_context.py:17\n",
> "stacktrace": [
> {
> "class": "org.apache.spark.sql.errors.QueryExecutionErrors$",
> "method": "divideByZeroError",
> "file": "QueryExecutionErrors.scala",
> "line": 203
> },
> {
> "class": "org.apache.spark.sql.errors.QueryExecutionErrors",
> "method": "divideByZeroError",
> "file": "QueryExecutionErrors.scala",
> "line": -1
> },
> {
> "class": "org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStage1",
> "method": "project_doConsume_0$",
> "file": null,
> "line": -1
> },
> {
> "class": "org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStage1",
> "method": "processNext",
> "file": null,
> "line": -1
> },
> {
> "class": "org.apache.spark.sql.execution.BufferedRowIterator",
> "method": "hasNext",
> "file": "BufferedRowIterator.java",
> "line": 43
> },
> {
> "class": "org.apache.spark.sql.execution.WholeStageCodegenEvaluatorFactory$WholeStageCodegenPartitionEvaluator$$anon$1",
> "method": "hasNext",
> "file": "WholeStageCodegenEvaluatorFactory.scala",
> "line": 50
> },
> {
> "class": "org.apache.spark.sql.execution.SparkPlan",
> "method": "$anonfun$getByteArrayRdd$1",
> "file": "SparkPlan.scala",
> "line": 388
> },
> {
> "class": "org.apache.spark.rdd.RDD",
> "method": "$anonfun$mapPartitionsInternal$2",
> "file": "RDD.scala",
> "line": 896
> },
> {
> "class": "org.apache.spark.rdd.RDD",
> "method": "$anonfun$mapPartitionsInternal$2$adapted",
> "file": "RDD.scala",
> "line": 896
> },
> {
> "class": "org.apache.spark.rdd.MapPartitionsRDD",
> "method": "compute",
> "file": "MapPartitionsRDD.scala",
> "line": 52
> },
> {
> "class": "org.apache.spark.rdd.RDD",
> "method": "computeOrReadCheckpoint",
> "file": "RDD.scala",
> "line": 369
> },
> {
> "class": "org.apache.spark.rdd.RDD",
> "method": "iterator",
> "file": "RDD.scala",
> "line": 333
> },
> {
> "class": "org.apache.spark.scheduler.ResultTask",
> "method": "runTask",
> "file": "ResultTask.scala",
> "line": 93
> },
> {
> "class": "org.apache.spark.TaskContext",
> "method": "runTaskWithListeners",
> "file": "TaskContext.scala",
> "line": 171
> },
> {
> "class": "org.apache.spark.scheduler.Task",
> "method": "run",
> "file": "Task.scala",
> "line": 146
> },
> {
> "class": "org.apache.spark.executor.Executor$TaskRunner",
> "method": "$anonfun$run$5",
> "file": "Executor.scala",
> "line": 644
> },
> {
> "class": "org.apache.spark.util.SparkErrorUtils",
> "method": "tryWithSafeFinally",
> "file": "SparkErrorUtils.scala",
> "line": 64
> },
> {
> "class": "org.apache.spark.util.SparkErrorUtils",
> "method": "tryWithSafeFinally$",
> "file": "SparkErrorUtils.scala",
> "line": 61
> },
> {
> "class": "org.apache.spark.util.Utils$",
> "method": "tryWithSafeFinally",
> "file": "Utils.scala",
> "line": 99
> },
> {
> "class": "org.apache.spark.executor.Executor$TaskRunner",
> "method": "run",
> "file": "Executor.scala",
> "line": 647
> },
> {
> "class": "java.util.concurrent.ThreadPoolExecutor",
> "method": "runWorker",
> "file": "ThreadPoolExecutor.java",
> "line": 1136
> },
> {
> "class": "java.util.concurrent.ThreadPoolExecutor$Worker",
> "method": "run",
> "file": "ThreadPoolExecutor.java",
> "line": 635
> },
> {
> "class": "java.lang.Thread",
> "method": "run",
> "file": "Thread.java",
> "line": 840
> }
> ]
> },
> "logger": "Executor"
> } {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org