You are viewing a plain text version of this content. The canonical link for it is here.

Posted to issues@spark.apache.org by "Haotian Sun (Jira)" <ji...@apache.org> on 2026/09/24 20:01:00 UTC

[jira] [Created] (SPARK-59772) Improve Python UDF worker metrics collection and reporting

Haotian Sun created SPARK-59772:
-----------------------------------

             Summary: Improve Python UDF worker metrics collection and reporting
                 Key: SPARK-59772
                 URL: https://issues.apache.org/jira/browse/SPARK-59772
             Project: Spark
          Issue Type: Improvement
          Components: PySpark
    Affects Versions: 5.0.0
            Reporter: Haotian Sun


Improve collection and reporting of Python UDF worker metrics, building on the existing SQL metrics in SPARK-34265.

Planned OSS work:
1. Add a versioned, extensible Python worker-to-JVM metrics report while preserving the legacy timing protocol.
2. Collect additional low-overhead worker measurements at defined execution boundaries.
3. Register and aggregate selected measurements as Spark SQL metrics and expose them through existing Spark observability paths.

The protocol change is the first subtask. Additional measurements and SQL metric registration will be separate subtasks once their semantics are defined.

Related history: SPARK-30306 proposed counters through the Spark Metrics System; SPARK-34265 added the current SQL timing metrics.




--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org