You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2019/12/20 10:07:06 UTC
[GitHub] [spark] LucaCanali commented on issue #26953:
[SPARK-30306][CORE][PYTHON] Instrument Python UDF execution time and
metrics using Spark Metrics system
LucaCanali commented on issue #26953: [SPARK-30306][CORE][PYTHON] Instrument Python UDF execution time and metrics using Spark Metrics system
URL: https://github.com/apache/spark/pull/26953#issuecomment-567869272
Thanks @HyukjinKwon for taking time for this.
This functionality propoesed in ths PR is different from Python profiler, or at least its intended use is. Also it is intended to be lightweight so that it can be used for measuring Spark workloads in production as part of a performance dashboard based on the metrics coming from the Spark metrics system.
I'd like to add some additional context on how we intend to use this.
- I have described how we implement and use a Spark performance dashboard based on the metrics system in a recent Spark Summit presentation https://databricks.com/session_eu19/performance-troubleshooting-using-apache-spark-metrics and in a blog entry http://db-blog.web.cern.ch/blog/luca-canali/2019-02-performance-dashboard-apache-spark
- Over the last couple of years I have helped improving this by adding to the Spark metrics system instrumentation for aggregated task metrics values per executor [SPARK-25228], JVM CPU usage [SPARK-22190], memory usage [SPARK-27189]. This still does not cover all the space of possible activities and "time usage" by Spark tasks and executors, in particular the problem that I am trying to solve with this PR is that time spent waiting for results to come back from Python workers (typically when executing a UDF) is currently not instrumented, so it appears in the dashboard as run time "without attribution" in the current dashboard, while it can be visualized using the metrics implemented here (see [image](https://camo.githubusercontent.com/6bbade0b6d6dceeb75cea27a5041298a3132599c/68747470733a2f2f6973737565732e6170616368652e6f72672f6a6972612f7365637572652f6174746163686d656e742f31323938393230312f50616e6461735544465f54696d655f496e737472756d656e746174696f6e5f416e6e6f74617465642e706e67) )
As you mentioned, in the current PR I have implemented several details, which I guess can be useful when troubleshooting, but for the general use can be simplified as you propose, in particular in the number and detials exposed in the user-visible metrics in the PythonMetrics source.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org