You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2021/01/27 16:00:16 UTC

[GitHub] [spark] LucaCanali opened a new pull request #31367: [SPARK-34265][PYTHON][SQL] Instrument Python UDF using SQL Metrics

LucaCanali opened a new pull request #31367:
URL: https://github.com/apache/spark/pull/31367


   ### What changes were proposed in this pull request?
   This proposes to add SQLMetrics instrumentation for Python UDF execution.
   
   ### Why are the changes needed?
   This is aimed at improving monitoring and performance troubleshooting of Python code called by Spark, via UDF, Pandas UDF or with MapPartittions. 
   
   ### Does this PR introduce _any_ user-facing change?
   The introduced SQL metrics are exposed to the end users via the WebUI interface, visible in the SQL tab for execution steps related to Python UDF execution, namely BatchEvalPython, ArrowEvalPython, AggregateInPandas, FlaMapGroupsInPandas, FlatMapsCoGroupsInPandas, MapInPandas, WindowsInPandas.
   See also the screenshot with the metrics introduced:
   ![](https://issues.apache.org/jira/secure/attachment/13019522/PythonSQLMetrics_Jira_Picture.png)
   
   ### How was this patch tested?
   Manually tested + a python test has been added.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] LucaCanali commented on a change in pull request #31367: [SPARK-34265][PYTHON][SQL] Instrument Python UDF using SQL Metrics

Posted by GitBox <gi...@apache.org>.
LucaCanali commented on a change in pull request #31367:
URL: https://github.com/apache/spark/pull/31367#discussion_r566863847



##########
File path: python/pyspark/sql/tests/test_pandas_sqlmetrics.py
##########
@@ -0,0 +1,65 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+import unittest
+
+from pyspark.sql.functions import pandas_udf
+from pyspark.testing.sqlutils import ReusedSQLTestCase, have_pandas, have_pyarrow, \
+    pandas_requirement_message, pyarrow_requirement_message
+
+
+@unittest.skipIf(
+    not have_pandas or not have_pyarrow,
+    pandas_requirement_message or pyarrow_requirement_message)  # type: ignore[arg-type]
+class PandasSQLMetrics(ReusedSQLTestCase):

Review comment:
       Good point. I am now sure on how to do that for Python UDF though.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] LucaCanali edited a comment on pull request #31367: [SPARK-34265][PYTHON][SQL] Instrument Python UDF using SQL Metrics

Posted by GitBox <gi...@apache.org>.
LucaCanali edited a comment on pull request #31367:
URL: https://github.com/apache/spark/pull/31367#issuecomment-769738766


   I have updated PR with the proposed list of metrics. The list currently contains 10 metrics.
   I can see the need to have just a few important metrics.
   However, I can also see that the Python UDF implementation is complex with many moving parts, metrics can help with troubleshooting with corner cases too, in those circumstances more metrics mean more flexibility and more chances to find where the root cause of the problem is.
   
   Another point for discussion is how accurate are the metrics in the current implementation? I have run a few tests to check that the values measured make sense and are in the ballpark of what expected.  There is room to do more tests with some corner cases to understand how the instrumentation keeps up there.
   
   In particular, measuring execution time can be challenging at times, as with this we attempt to do it from JVM. I am aiming to do a “good enough to be useful” job for the timing metrics, rather than a precise timing. I have put in the metrics description some hints of the nuances I found when testing. 
   
   I think the "time spent sending data" can be useful when troubleshooting cases where the performance problem is with sending large amounts of data to Python. Time spent executing is the key metric to understand the overall performance. Number of rows returned and processed are also useful metric, to understand how much work has been done and how much still needs to be done when monitoring the progress of an active query.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #31367: [SPARK-34265][PYTHON][SQL] Instrument Python UDF execution using SQL Metrics

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31367:
URL: https://github.com/apache/spark/pull/31367#issuecomment-885062045


   Kubernetes integration test starting
   URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/46021/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #31367: [SPARK-34265][PYTHON][SQL] Instrument Python UDF using SQL Metrics

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31367:
URL: https://github.com/apache/spark/pull/31367#issuecomment-770077223


   **[Test build #134666 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/134666/testReport)** for PR 31367 at commit [`0932597`](https://github.com/apache/spark/commit/0932597ec268ca52b51890f54be22c90684acf22).
    * This patch passes all tests.
    * This patch merges cleanly.
    * This patch adds no public classes.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] LucaCanali commented on a change in pull request #31367: [SPARK-34265][PYTHON][SQL] Instrument Python UDF using SQL Metrics

Posted by GitBox <gi...@apache.org>.
LucaCanali commented on a change in pull request #31367:
URL: https://github.com/apache/spark/pull/31367#discussion_r566734443



##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/python/PythonArrowOutput.scala
##########
@@ -71,10 +77,19 @@ private[python] trait PythonArrowOutput { self: BasePythonRunner[_, ColumnarBatc
         }
         try {
           if (reader != null && batchLoaded) {
+            val bytesReadStart = reader.bytesRead()
+            val startTime = System.nanoTime()
             batchLoaded = reader.loadNextBatch()
+            val deltaTime = System.nanoTime() - startTime

Review comment:
       Good point. From what I can see by experimenting, is that this actually triggers execution, so the time measured here seems to be a relevant value for the execution time. Ideally we would like to measure on the Python side, although this seems good enough?




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on pull request #31367: [SPARK-34265][PYTHON][SQL] Instrument Python UDF execution using SQL Metrics

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #31367:
URL: https://github.com/apache/spark/pull/31367#issuecomment-812612490


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/136857/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on pull request #31367: [SPARK-34265][PYTHON][SQL] Instrument Python UDF execution using SQL Metrics

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #31367:
URL: https://github.com/apache/spark/pull/31367#issuecomment-783489693


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/39927/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on pull request #31367: [SPARK-34265][PYTHON][SQL] Instrument Python UDF using SQL Metrics

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #31367:
URL: https://github.com/apache/spark/pull/31367#issuecomment-769977920


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/39247/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on pull request #31367: [SPARK-34265][PYTHON][SQL] Instrument Python UDF execution using SQL Metrics

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #31367:
URL: https://github.com/apache/spark/pull/31367#issuecomment-783489693


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/39927/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA removed a comment on pull request #31367: [SPARK-34265][PYTHON][SQL] Instrument Python UDF using SQL Metrics

Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #31367:
URL: https://github.com/apache/spark/pull/31367#issuecomment-770014623


   **[Test build #134666 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/134666/testReport)** for PR 31367 at commit [`0932597`](https://github.com/apache/spark/commit/0932597ec268ca52b51890f54be22c90684acf22).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] maropu commented on pull request #31367: [SPARK-34265][PYTHON][SQL] Instrument Python UDF using SQL Metrics

Posted by GitBox <gi...@apache.org>.
maropu commented on pull request #31367:
URL: https://github.com/apache/spark/pull/31367#issuecomment-768867777


   ok to test


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on pull request #31367: [SPARK-34265][PYTHON][SQL] Instrument Python UDF using SQL Metrics

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #31367:
URL: https://github.com/apache/spark/pull/31367#issuecomment-768958757


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/39194/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on pull request #31367: [SPARK-34265][PYTHON][SQL] Instrument Python UDF execution using SQL Metrics

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #31367:
URL: https://github.com/apache/spark/pull/31367#issuecomment-845867971






-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] LucaCanali commented on a change in pull request #31367: [SPARK-34265][PYTHON][SQL] Instrument Python UDF using SQL Metrics

Posted by GitBox <gi...@apache.org>.
LucaCanali commented on a change in pull request #31367:
URL: https://github.com/apache/spark/pull/31367#discussion_r568740939



##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/python/PythonSQLMetrics.scala
##########
@@ -0,0 +1,57 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.python
+
+import org.apache.spark.sql.execution.SparkPlan
+import org.apache.spark.sql.execution.metric.SQLMetrics
+
+private[python] trait PythonSQLMetrics extends SparkPlan {
+  override val metrics = Map(

Review comment:
       My original thought here was to use a trait so that we can reuse the metrics definitaion with the many physical plans that need it (BatchEvalPythonExec, ArrowEcalPythonExec, AggregateInPandasExec, etc). Other implementations of this can work too. I see that `ShuffleRowRDD` uses a custom object `SQLShuffleMetricsReporter` to define and handle the metrics. Can you please elaborate a bit more on your proposal?




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on pull request #31367: [SPARK-34265][PYTHON][SQL] Instrument Python UDF using SQL Metrics

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #31367:
URL: https://github.com/apache/spark/pull/31367#issuecomment-769252147






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #31367: [SPARK-34265][PYTHON][SQL] Instrument Python UDF execution using SQL Metrics

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31367:
URL: https://github.com/apache/spark/pull/31367#issuecomment-783406963


   **[Test build #135347 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/135347/testReport)** for PR 31367 at commit [`8e8fee8`](https://github.com/apache/spark/commit/8e8fee82c44ac0bd517c3ffe27c2878ced3b6b98).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on pull request #31367: [SPARK-34265][PYTHON][SQL] Instrument Python UDF using SQL Metrics

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #31367:
URL: https://github.com/apache/spark/pull/31367#issuecomment-769977920


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/39247/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #31367: [SPARK-34265][PYTHON][SQL] Instrument Python UDF execution using SQL Metrics

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31367:
URL: https://github.com/apache/spark/pull/31367#issuecomment-812558679


   Kubernetes integration test starting
   URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/41435/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #31367: [SPARK-34265][PYTHON][SQL] Instrument Python UDF execution using SQL Metrics

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31367:
URL: https://github.com/apache/spark/pull/31367#issuecomment-884986592


   **[Test build #141504 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/141504/testReport)** for PR 31367 at commit [`9690426`](https://github.com/apache/spark/commit/96904262c08cd668d22b8792c00e280974d5c9d7).


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA removed a comment on pull request #31367: [SPARK-34265][PYTHON][SQL] Instrument Python UDF execution using SQL Metrics

Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #31367:
URL: https://github.com/apache/spark/pull/31367#issuecomment-812540690


   **[Test build #136857 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/136857/testReport)** for PR 31367 at commit [`e9db06c`](https://github.com/apache/spark/commit/e9db06c5f64b479c0cb3a1be4c3a599a0423dfd7).


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #31367: [SPARK-34265][PYTHON][SQL] Instrument Python UDF using SQL Metrics

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31367:
URL: https://github.com/apache/spark/pull/31367#issuecomment-769229039


   **[Test build #134619 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/134619/testReport)** for PR 31367 at commit [`c6a1bdc`](https://github.com/apache/spark/commit/c6a1bdc82bf1384761610cffc3df7eb03bd30c5a).
    * This patch passes all tests.
    * This patch merges cleanly.
    * This patch adds no public classes.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #31367: [SPARK-34265][PYTHON][SQL] Instrument Python UDF execution using SQL Metrics

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31367:
URL: https://github.com/apache/spark/pull/31367#issuecomment-812540690


   **[Test build #136857 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/136857/testReport)** for PR 31367 at commit [`e9db06c`](https://github.com/apache/spark/commit/e9db06c5f64b479c0cb3a1be4c3a599a0423dfd7).


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on pull request #31367: [SPARK-34265][PYTHON][SQL] Instrument Python UDF using SQL Metrics

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #31367:
URL: https://github.com/apache/spark/pull/31367#issuecomment-769252147






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on pull request #31367: [SPARK-34265][PYTHON][SQL] Instrument Python UDF execution using SQL Metrics

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #31367:
URL: https://github.com/apache/spark/pull/31367#issuecomment-783531370


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/135347/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #31367: [SPARK-34265][PYTHON][SQL] Instrument Python UDF execution using SQL Metrics

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31367:
URL: https://github.com/apache/spark/pull/31367#issuecomment-845853097


   Kubernetes integration test status failure
   URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/43320/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] LucaCanali commented on a change in pull request #31367: [SPARK-34265][PYTHON][SQL] Instrument Python UDF using SQL Metrics

Posted by GitBox <gi...@apache.org>.
LucaCanali commented on a change in pull request #31367:
URL: https://github.com/apache/spark/pull/31367#discussion_r566732268



##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/python/BatchEvalPythonExec.scala
##########
@@ -83,12 +97,19 @@ case class BatchEvalPythonExec(udfs: Seq[PythonUDF], resultAttrs: Seq[Attribute]
       val unpickledBatch = unpickle.loads(pickedResult)
       unpickledBatch.asInstanceOf[java.util.ArrayList[Any]].asScala
     }.map { result =>
+      pythonNumRowsReceived += 1
+      val startTime = System.nanoTime()
       if (udfs.length == 1) {
         // fast path for single UDF
         mutableRow(0) = fromJava(result)
+        val deltaTime = System.nanoTime() - startTime
+        pythonExecTime += deltaTime
         mutableRow
       } else {
-        fromJava(result).asInstanceOf[InternalRow]
+        val res = fromJava(result).asInstanceOf[InternalRow]
+        val deltaTime = System.nanoTime() - startTime
+        pythonExecTime += deltaTime

Review comment:
       Correct, this is just a component, actually small and could be omitted. The most important part of the execution for BatchEvalPythonExec time is measured in PythonUDFRunner.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] LucaCanali edited a comment on pull request #31367: [SPARK-34265][PYTHON][SQL] Instrument Python UDF using SQL Metrics

Posted by GitBox <gi...@apache.org>.
LucaCanali edited a comment on pull request #31367:
URL: https://github.com/apache/spark/pull/31367#issuecomment-769738766


   I have updated PR with the proposed list of metrics. The list currently contains 10 metrics.
   I can see the need to have just a few important metrics.
   However, I can also see that the Python UDF implementation is complex with many moving parts, metrics can help with troubleshooting with corner cases too, in those circumstances more metrics mean more flexibility and more chances to find where the root cause of the problem is.
   Another point for discussion is how accurate are the metrics in the current implementation? I have run a few tests to check that the values measured make sense and are in the ballpark of what expected.  There is room to do more tests with some corner cases to understand how the instrumentation keeps up there.
   
   In particular, measuring execution time can be challenging at times, as with this we attempt to do it from JVM. I am aiming to go a “good enough to be useful” job for the timing metrics, rather than a precise timing. I have put in the metrics description some hints to the nuances I found when testing. Send and receive time back and forth between JVM and Python workers seem to overlap in some cases, due to the use of queues.
   I think the "time spent sending data" can be useful when troubleshooting cases where the performance problem is with sending a lot of data to Python. Time spent executing is probably the key metric to understand the overall performance. Number of rows returned will probably be another useful one, to understand the progress of the execution when monitoring the progress of an active query.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #31367: [SPARK-34265][PYTHON][SQL] Instrument Python UDF using SQL Metrics

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31367:
URL: https://github.com/apache/spark/pull/31367#issuecomment-769941889


   Kubernetes integration test starting
   URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/39247/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #31367: [SPARK-34265][PYTHON][SQL] Instrument Python UDF execution using SQL Metrics

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31367:
URL: https://github.com/apache/spark/pull/31367#issuecomment-845865468


   **[Test build #138798 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/138798/testReport)** for PR 31367 at commit [`9734f9c`](https://github.com/apache/spark/commit/9734f9cb61d7a12987bca780f1ae19769ddda0c6).
    * This patch passes all tests.
    * This patch merges cleanly.
    * This patch adds no public classes.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on pull request #31367: [SPARK-34265][PYTHON][SQL] Instrument Python UDF using SQL Metrics

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #31367:
URL: https://github.com/apache/spark/pull/31367#issuecomment-770087727


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/134666/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on pull request #31367: [SPARK-34265][PYTHON][SQL] Instrument Python UDF using SQL Metrics

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #31367:
URL: https://github.com/apache/spark/pull/31367#issuecomment-769952593


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/134659/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on pull request #31367: [SPARK-34265][PYTHON][SQL] Instrument Python UDF execution using SQL Metrics

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #31367:
URL: https://github.com/apache/spark/pull/31367#issuecomment-812569354


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/41435/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #31367: [SPARK-34265][PYTHON][SQL] Instrument Python UDF execution using SQL Metrics

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31367:
URL: https://github.com/apache/spark/pull/31367#issuecomment-812569335


   Kubernetes integration test status success
   URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/41435/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #31367: [SPARK-34265][PYTHON][SQL] Instrument Python UDF using SQL Metrics

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31367:
URL: https://github.com/apache/spark/pull/31367#issuecomment-769233302


   Kubernetes integration test status failure
   URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/39207/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on pull request #31367: [SPARK-34265][PYTHON][SQL] Instrument Python UDF using SQL Metrics

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #31367:
URL: https://github.com/apache/spark/pull/31367#issuecomment-768407960


   Can one of the admins verify this patch?


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] LucaCanali commented on pull request #31367: [SPARK-34265][PYTHON][SQL] Instrument Python UDF using SQL Metrics

Posted by GitBox <gi...@apache.org>.
LucaCanali commented on pull request #31367:
URL: https://github.com/apache/spark/pull/31367#issuecomment-769738766


   I have updated PR with the proposed list of metrics. The list currently contains 10 metrics.
   
   I can see the need to have just a few important metrics.
   However, I can also see that the Python UDF implementation is complex with many moving parts, metrics can help with troubleshooting with corner cases too, in those circumstances more metrics mean more flexibility and more chances to find where the root cause of the problem is.
   
   
   Another point for discussion is how accurate are the metrics in the current implementation? I have run a few tests to check that the values measured make sense and are in the ballpark of what expected. 
   In particular, measuring execution time can be challenging at times, as with this we attept to do it from JVM. I have put in the metrics description some hints to the nuances I found when testing . Send and receive time back and forth between JVM and Python workers seem to overlap in some cases, due the use of queues. 
   
   I think the "time spent sending data" can be useful when troubleshooting cases where the performance problem is with sending a lot of data to Python. Time spent executinf is probably the key metric to understand the overall performance. Number of rows returned will probably be another useful one, to understand the progress of the execution when monitoring the progress of an active query.
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] ueshin commented on a change in pull request #31367: [SPARK-34265][PYTHON][SQL] Instrument Python UDF using SQL Metrics

Posted by GitBox <gi...@apache.org>.
ueshin commented on a change in pull request #31367:
URL: https://github.com/apache/spark/pull/31367#discussion_r566449042



##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/python/ArrowPythonRunner.scala
##########
@@ -88,14 +102,22 @@ class ArrowPythonRunner(
           while (inputIterator.hasNext) {
             val nextBatch = inputIterator.next()
 
+            val startTime = System.nanoTime()
             while (nextBatch.hasNext) {
               arrowWriter.write(nextBatch.next())
             }
 
             arrowWriter.finish()
             writer.writeBatch()
+
+            val deltaTime = System.nanoTime() - startTime
+            pythonSerializeTime += deltaTime
+            val rowCount = root.getRowCount
+            pythonNumBatchesSent += 1
+            pythonNumRowsSent += rowCount
             arrowWriter.reset()
           }
+          pythonDataSent += dataOut.size()

Review comment:
       ditto.

##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/python/ArrowPythonRunner.scala
##########
@@ -72,6 +83,9 @@ class ArrowPythonRunner(
         }
 
         PythonUDFRunner.writeUDFs(dataOut, funcs, argOffsets)
+        val deltaTime = System.nanoTime()-startTime

Review comment:
       nit: style: need spaces before and after `-`.

##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/python/ArrowPythonRunner.scala
##########
@@ -72,6 +83,9 @@ class ArrowPythonRunner(
         }
 
         PythonUDFRunner.writeUDFs(dataOut, funcs, argOffsets)
+        val deltaTime = System.nanoTime()-startTime
+        pythonCodeSerializeTime += deltaTime
+        pythonCodeSent += dataOut.size()

Review comment:
       I guess we also need to use a delta size between before and after the code is sent?

##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/python/PythonArrowOutput.scala
##########
@@ -26,16 +26,22 @@ import org.apache.arrow.vector.VectorSchemaRoot
 import org.apache.arrow.vector.ipc.ArrowStreamReader
 
 import org.apache.spark.{SparkEnv, TaskContext}
-import org.apache.spark.api.python.{BasePythonRunner, SpecialLengths}
+import org.apache.spark.api.python.{BasePythonRunner, ChainedPythonFunctions, SpecialLengths}
+import org.apache.spark.sql.execution.metric.SQLMetric
 import org.apache.spark.sql.types.StructType
 import org.apache.spark.sql.util.ArrowUtils
 import org.apache.spark.sql.vectorized.{ArrowColumnVector, ColumnarBatch, ColumnVector}
 
-/**
- * A trait that can be mixed-in with [[BasePythonRunner]]. It implements the logic from
- * Python (Arrow) to JVM (ColumnarBatch).
- */
-private[python] trait PythonArrowOutput { self: BasePythonRunner[_, ColumnarBatch] =>
+// An abstract class implementing the logic from Python (Arrow) to JVM (ColumnarBatch).
+private[python] abstract class PythonArrowOutput[IN](
+                           funcs: Seq[ChainedPythonFunctions],
+                           evalType: Int,
+                           argOffsets: Array[Array[Int]],
+                           pythonDataReceived: SQLMetric,
+                           pythonExecTime: SQLMetric,
+                           pythonNumRowsReceived: SQLMetric,
+                           pythonNumBatchesReceived: SQLMetric)

Review comment:
       nit: indent with 4 spaces here.

##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/python/PythonArrowOutput.scala
##########
@@ -71,10 +77,19 @@ private[python] trait PythonArrowOutput { self: BasePythonRunner[_, ColumnarBatc
         }
         try {
           if (reader != null && batchLoaded) {
+            val bytesReadStart = reader.bytesRead()
+            val startTime = System.nanoTime()
             batchLoaded = reader.loadNextBatch()
+            val deltaTime = System.nanoTime() - startTime

Review comment:
       Same here? Seems like it's just calculating the batch loading time?

##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/python/BatchEvalPythonExec.scala
##########
@@ -83,12 +97,19 @@ case class BatchEvalPythonExec(udfs: Seq[PythonUDF], resultAttrs: Seq[Attribute]
       val unpickledBatch = unpickle.loads(pickedResult)
       unpickledBatch.asInstanceOf[java.util.ArrayList[Any]].asScala
     }.map { result =>
+      pythonNumRowsReceived += 1
+      val startTime = System.nanoTime()
       if (udfs.length == 1) {
         // fast path for single UDF
         mutableRow(0) = fromJava(result)
+        val deltaTime = System.nanoTime() - startTime
+        pythonExecTime += deltaTime
         mutableRow
       } else {
-        fromJava(result).asInstanceOf[InternalRow]
+        val res = fromJava(result).asInstanceOf[InternalRow]
+        val deltaTime = System.nanoTime() - startTime
+        pythonExecTime += deltaTime

Review comment:
       Is this really calculating the exec time? Seems like it's only calculating the time for `fromJava(result)`?




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] maropu commented on a change in pull request #31367: [SPARK-34265][PYTHON][SQL] Instrument Python UDF using SQL Metrics

Posted by GitBox <gi...@apache.org>.
maropu commented on a change in pull request #31367:
URL: https://github.com/apache/spark/pull/31367#discussion_r566494980



##########
File path: docs/web-ui.md
##########
@@ -404,6 +404,16 @@ Here is the list of SQL metrics:
 <tr><td> <code>avg hash probe bucket list iters</code> </td><td> the average bucket list iterations per lookup during aggregation </td><td> HashAggregate </td></tr>
 <tr><td> <code>data size of build side</code> </td><td> the size of built hash map </td><td> ShuffledHashJoin </td></tr>
 <tr><td> <code>time to build hash map</code> </td><td> the time spent on building hash map </td><td> ShuffledHashJoin </td></tr>
+<tr><td> <code>time spent executing</code> </td><td> time spent executing the Python UDF and sending the results back to the executors. It contains time spent waiting for input queues to fill, when sending data is slow compared to execution </td><td> BatchEvalPython, ArrowEvalPython, AggregateInPandas, FlaMapGroupsInPandas, FlatMapsCoGroupsInPandas, MapInPandas, WindowsInPandas </td></tr>
+<tr><td> <code>time spent sending data</code> </td><td> time spent sending data to the Python workers, this is part of the Python UDF execution and happens asynchronously using queues </td><td> BatchEvalPython, ArrowEvalPython, AggregateInPandas, FlaMapGroupsInPandas, FlatMapsCoGroupsInPandas, MapInPandas, WindowsInPandas </td></tr>
+<tr><td> <code>time spent sending code</code> </td><td> time spent sending Python code to the Python workers </td><td> BatchEvalPython, ArrowEvalPython, AggregateInPandas, FlaMapGroupsInPandas, FlatMapsCoGroupsInPandas, MapInPandas, WindowsInPandas </td></tr>
+<tr><td> <code>bytes of code sent</code> </td><td> the number of bytes of serialized Python code sent the to the Python workers </td><td> BatchEvalPython, ArrowEvalPython, AggregateInPandas, FlaMapGroupsInPandas, FlatMapsCoGroupsInPandas, MapInPandas, WindowsInPandas </td></tr>
+<tr><td> <code>bytes of data returned</code> </td><td> the number of bytes of serialized data received back from the Python workers </td><td> BatchEvalPython, ArrowEvalPython, AggregateInPandas, FlaMapGroupsInPandas, FlatMapsCoGroupsInPandas, MapInPandas, WindowsInPandas </td></tr>
+<tr><td> <code>bytes of data sent</code> </td><td> the number of bytes of serialized data sent the to the Python workers </td><td> BatchEvalPython, ArrowEvalPython, AggregateInPandas, FlaMapGroupsInPandas, FlatMapsCoGroupsInPandas, MapInPandas, WindowsInPandas </td></tr>
+<tr><td> <code>number of batches returned</code> </td><td> the number of data batches received back from the Python workers </td><td> BatchEvalPython, ArrowEvalPython, AggregateInPandas, FlaMapGroupsInPandas, FlatMapsCoGroupsInPandas, MapInPandas, WindowsInPandas </td></tr>
+<tr><td> <code>number of batches processed</code> </td><td> the number of data batches sent to the Python workers </td><td> BatchEvalPython, ArrowEvalPython, AggregateInPandas, FlaMapGroupsInPandas, FlatMapsCoGroupsInPandas, MapInPandas, WindowsInPandas </td></tr>
+<tr><td> <code>number of rows returned</code> </td><td> the number of rows returned by the Python workers </td><td> BatchEvalPython, ArrowEvalPython, AggregateInPandas, FlaMapGroupsInPandas, FlatMapsCoGroupsInPandas, MapInPandas, WindowsInPandas </td></tr>
+<tr><td> <code>number of rows processed</code> </td><td> the number rows sent to the Python workers </td><td> BatchEvalPython, ArrowEvalPython, AggregateInPandas, FlaMapGroupsInPandas, FlatMapsCoGroupsInPandas, MapInPandas, WindowsInPandas </td></tr>

Review comment:
       For reviews, could you put the screenshot of the updated doc in the PR description?

##########
File path: python/pyspark/sql/tests/test_pandas_sqlmetrics.py
##########
@@ -0,0 +1,65 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+import unittest
+
+from pyspark.sql.functions import pandas_udf
+from pyspark.testing.sqlutils import ReusedSQLTestCase, have_pandas, have_pyarrow, \
+    pandas_requirement_message, pyarrow_requirement_message
+
+
+@unittest.skipIf(
+    not have_pandas or not have_pyarrow,
+    pandas_requirement_message or pyarrow_requirement_message)  # type: ignore[arg-type]
+class PandasSQLMetrics(ReusedSQLTestCase):
+
+    def test_pandas_sql_metrics_basic(self):
+
+        PythonSQLMetrics = [
+            "time spent executing",
+            "time spent sending data",
+            "time spent sending code",
+            "bytes of code sent",
+            "bytes of data returned",
+            "bytes of data sent",
+            "number of batches returned",
+            "number of batches processed",
+            "number of rows returned",
+            "number of rows processed"
+        ]
+
+        @pandas_udf("long")
+        def test_pandas(col1):
+            return col1 * col1
+
+        res = self.spark.range(10).select(test_pandas("id")).collect()
+
+        statusStore = self.spark._jsparkSession.sharedState().statusStore()
+        executionMetrics = statusStore.execution(0).get().metrics().mkString()
+
+        for metric in PythonSQLMetrics:
+            self.assertIn(metric, executionMetrics)
+

Review comment:
       nit: one more blank line.

##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/python/PythonSQLMetrics.scala
##########
@@ -0,0 +1,57 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.python
+
+import org.apache.spark.sql.execution.SparkPlan
+import org.apache.spark.sql.execution.metric.SQLMetrics
+
+private[python] trait PythonSQLMetrics extends SparkPlan {

Review comment:
       These plans already extend `SparkPlan` (e.g., https://github.com/apache/spark/pull/31367/files#diff-ce3497483dcd52a600e1414d36f15e96f766ea94f3c50f075926716b7e42c724R57), so could you use self-type instead?
   
   ```
   private[python] trait PythonSQLMetrics {
     self: SparkPlan
   ...
   ```

##########
File path: python/pyspark/sql/tests/test_pandas_sqlmetrics.py
##########
@@ -0,0 +1,65 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+import unittest
+
+from pyspark.sql.functions import pandas_udf
+from pyspark.testing.sqlutils import ReusedSQLTestCase, have_pandas, have_pyarrow, \
+    pandas_requirement_message, pyarrow_requirement_message
+
+
+@unittest.skipIf(
+    not have_pandas or not have_pyarrow,
+    pandas_requirement_message or pyarrow_requirement_message)  # type: ignore[arg-type]
+class PandasSQLMetrics(ReusedSQLTestCase):

Review comment:
       Could you add tests in a Scala side (e.g., `SQLMetricsSuite`), too?

##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/python/PythonSQLMetrics.scala
##########
@@ -0,0 +1,57 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.python
+
+import org.apache.spark.sql.execution.SparkPlan
+import org.apache.spark.sql.execution.metric.SQLMetrics
+
+private[python] trait PythonSQLMetrics extends SparkPlan {
+  override val metrics = Map(

Review comment:
       If you want to implement many metrics, how about using `Map[String, SQLMetric]` instead? (see `ShuffledRowRDD`)

##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/python/PythonUDFRunner.scala
##########
@@ -31,10 +32,17 @@ import org.apache.spark.sql.internal.SQLConf
 class PythonUDFRunner(
     funcs: Seq[ChainedPythonFunctions],
     evalType: Int,
-    argOffsets: Array[Array[Int]])
+    argOffsets: Array[Array[Int]],
+    pythonExecTime: SQLMetric,
+    pythonDataSerializeTime: SQLMetric,
+    pythonCodeSerializeTime: SQLMetric,
+    pythonCodeSent: SQLMetric,
+    pythonDataReceived: SQLMetric,
+    pythonDataSent: SQLMetric,
+    pythonNumBatchesReceived: SQLMetric,
+    pythonNumBatchesSent: SQLMetric)
   extends BasePythonRunner[Array[Byte], Array[Byte]](
     funcs, evalType, argOffsets) {
-

Review comment:
       nit: revert the unnecessary change

##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/python/PythonArrowOutput.scala
##########
@@ -26,16 +26,22 @@ import org.apache.arrow.vector.VectorSchemaRoot
 import org.apache.arrow.vector.ipc.ArrowStreamReader
 
 import org.apache.spark.{SparkEnv, TaskContext}
-import org.apache.spark.api.python.{BasePythonRunner, SpecialLengths}
+import org.apache.spark.api.python.{BasePythonRunner, ChainedPythonFunctions, SpecialLengths}
+import org.apache.spark.sql.execution.metric.SQLMetric
 import org.apache.spark.sql.types.StructType
 import org.apache.spark.sql.util.ArrowUtils
 import org.apache.spark.sql.vectorized.{ArrowColumnVector, ColumnarBatch, ColumnVector}
 
-/**
- * A trait that can be mixed-in with [[BasePythonRunner]]. It implements the logic from
- * Python (Arrow) to JVM (ColumnarBatch).
- */
-private[python] trait PythonArrowOutput { self: BasePythonRunner[_, ColumnarBatch] =>
+// An abstract class implementing the logic from Python (Arrow) to JVM (ColumnarBatch).
+private[python] abstract class PythonArrowOutput[IN](

Review comment:
       Could you avoid this change? How about writing it like this?
   ```
   private[python] trait PythonArrowOutput { self: BasePythonRunner[_, ColumnarBatch] =>
     def pythonDataReceived: SQLMetric
     def pythonExecTime: SQLMetric
     def pythonNumRowsReceived: SQLMetric
     def pythonNumBatchesReceived: SQLMetric
     ...
   ```
   Then, can you override them in the `ArrowPythonRunner` side?




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #31367: [SPARK-34265][PYTHON][SQL] Instrument Python UDF using SQL Metrics

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31367:
URL: https://github.com/apache/spark/pull/31367#issuecomment-770050146


   Kubernetes integration test status success
   URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/39253/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] LucaCanali commented on pull request #31367: [SPARK-34265][PYTHON][SQL] Instrument Python UDF execution using SQL Metrics

Posted by GitBox <gi...@apache.org>.
LucaCanali commented on pull request #31367:
URL: https://github.com/apache/spark/pull/31367#issuecomment-888283647


   This has been blocked for a while. I believe that from the discussion it has emerged:
   - the proposed implementation in this PR possible has too many metrics, which could be confusing
   - the timing metrics are hard to implement and to understand if the proposed implementation is correct/useful
   
   I propose to take some of the ideas in the PR to a different and less complex PR: #33559


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] HyukjinKwon commented on pull request #31367: [SPARK-34265][PYTHON][SQL] Instrument Python UDF using SQL Metrics

Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on pull request #31367:
URL: https://github.com/apache/spark/pull/31367#issuecomment-768939489


   cc @BryanCutler too


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #31367: [SPARK-34265][PYTHON][SQL] Instrument Python UDF using SQL Metrics

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31367:
URL: https://github.com/apache/spark/pull/31367#issuecomment-769922020


   **[Test build #134660 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/134660/testReport)** for PR 31367 at commit [`c16968f`](https://github.com/apache/spark/commit/c16968f36cb589fbe3d13f60f95240af516fc12f).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] LucaCanali closed pull request #31367: [SPARK-34265][PYTHON][SQL] Instrument Python UDF execution using SQL Metrics

Posted by GitBox <gi...@apache.org>.
LucaCanali closed pull request #31367:
URL: https://github.com/apache/spark/pull/31367


   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #31367: [SPARK-34265][PYTHON][SQL] Instrument Python UDF using SQL Metrics

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31367:
URL: https://github.com/apache/spark/pull/31367#issuecomment-768935371


   Kubernetes integration test status failure
   URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/39194/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] ueshin commented on a change in pull request #31367: [SPARK-34265][PYTHON][SQL] Instrument Python UDF using SQL Metrics

Posted by GitBox <gi...@apache.org>.
ueshin commented on a change in pull request #31367:
URL: https://github.com/apache/spark/pull/31367#discussion_r567030338



##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/python/CoGroupedArrowPythonRunner.scala
##########
@@ -71,6 +83,9 @@ class CoGroupedArrowPythonRunner(
         }
 
         PythonUDFRunner.writeUDFs(dataOut, funcs, argOffsets)
+        val deltaTime = System.nanoTime()-startTime
+        pythonCodeSerializeTime += deltaTime
+        pythonCodeSent += dataOut.size()

Review comment:
       ditto as https://github.com/apache/spark/pull/31367#discussion_r566439263

##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/python/PythonUDFRunner.scala
##########
@@ -46,12 +54,23 @@ class PythonUDFRunner(
     new WriterThread(env, worker, inputIterator, partitionIndex, context) {
 
       protected override def writeCommand(dataOut: DataOutputStream): Unit = {
+        val startTime = System.nanoTime()
         PythonUDFRunner.writeUDFs(dataOut, funcs, argOffsets)
+        val deltaTime = System.nanoTime()-startTime
+        pythonCodeSerializeTime += deltaTime
+        pythonCodeSent += dataOut.size()

Review comment:
       ditto.

##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/python/CoGroupedArrowPythonRunner.scala
##########
@@ -83,6 +98,7 @@ class CoGroupedArrowPythonRunner(
           writeGroup(nextRight, rightSchema, dataOut, "right")
         }
         dataOut.writeInt(0)
+        pythonDataSent += dataOut.size()

Review comment:
       ditto.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on pull request #31367: [SPARK-34265][PYTHON][SQL] Instrument Python UDF using SQL Metrics

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #31367:
URL: https://github.com/apache/spark/pull/31367#issuecomment-768958757






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA removed a comment on pull request #31367: [SPARK-34265][PYTHON][SQL] Instrument Python UDF using SQL Metrics

Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #31367:
URL: https://github.com/apache/spark/pull/31367#issuecomment-769922020


   **[Test build #134660 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/134660/testReport)** for PR 31367 at commit [`c16968f`](https://github.com/apache/spark/commit/c16968f36cb589fbe3d13f60f95240af516fc12f).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on pull request #31367: [SPARK-34265][PYTHON][SQL] Instrument Python UDF using SQL Metrics

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #31367:
URL: https://github.com/apache/spark/pull/31367#issuecomment-769952593


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/134659/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] maropu commented on pull request #31367: [SPARK-34265][PYTHON][SQL] Instrument Python UDF using SQL Metrics

Posted by GitBox <gi...@apache.org>.
maropu commented on pull request #31367:
URL: https://github.com/apache/spark/pull/31367#issuecomment-769492296


   Could you list up the new metrics (their names and description) that this PR intends to add in the `What changes were proposed in this pull request?` section?


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #31367: [SPARK-34265][PYTHON][SQL] Instrument Python UDF using SQL Metrics

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31367:
URL: https://github.com/apache/spark/pull/31367#issuecomment-768991900


   **[Test build #134606 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/134606/testReport)** for PR 31367 at commit [`22e9999`](https://github.com/apache/spark/commit/22e999900b061eb497f8ebdb1244d0697aaddb0e).
    * This patch passes all tests.
    * This patch merges cleanly.
    * This patch adds the following public classes _(experimental)_:
     * `trait PythonSQLMetrics extends SparkPlan `


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on pull request #31367: [SPARK-34265][PYTHON][SQL] Instrument Python UDF execution using SQL Metrics

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #31367:
URL: https://github.com/apache/spark/pull/31367#issuecomment-783531370


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/135347/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on pull request #31367: [SPARK-34265][PYTHON][SQL] Instrument Python UDF execution using SQL Metrics

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #31367:
URL: https://github.com/apache/spark/pull/31367#issuecomment-812569354


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/41435/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on pull request #31367: [SPARK-34265][PYTHON][SQL] Instrument Python UDF using SQL Metrics

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #31367:
URL: https://github.com/apache/spark/pull/31367#issuecomment-770064614


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/39253/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #31367: [SPARK-34265][PYTHON][SQL] Instrument Python UDF execution using SQL Metrics

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31367:
URL: https://github.com/apache/spark/pull/31367#issuecomment-812599484


   **[Test build #136857 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/136857/testReport)** for PR 31367 at commit [`e9db06c`](https://github.com/apache/spark/commit/e9db06c5f64b479c0cb3a1be4c3a599a0423dfd7).
    * This patch passes all tests.
    * This patch merges cleanly.
    * This patch adds no public classes.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #31367: [SPARK-34265][PYTHON][SQL] Instrument Python UDF execution using SQL Metrics

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31367:
URL: https://github.com/apache/spark/pull/31367#issuecomment-845831318


   Kubernetes integration test starting
   URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/43320/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on pull request #31367: [SPARK-34265][PYTHON][SQL] Instrument Python UDF execution using SQL Metrics

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #31367:
URL: https://github.com/apache/spark/pull/31367#issuecomment-812612490


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/136857/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #31367: [SPARK-34265][PYTHON][SQL] Instrument Python UDF using SQL Metrics

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31367:
URL: https://github.com/apache/spark/pull/31367#issuecomment-769126081


   **[Test build #134619 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/134619/testReport)** for PR 31367 at commit [`c6a1bdc`](https://github.com/apache/spark/commit/c6a1bdc82bf1384761610cffc3df7eb03bd30c5a).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] LucaCanali edited a comment on pull request #31367: [SPARK-34265][PYTHON][SQL] Instrument Python UDF using SQL Metrics

Posted by GitBox <gi...@apache.org>.
LucaCanali edited a comment on pull request #31367:
URL: https://github.com/apache/spark/pull/31367#issuecomment-769738766


   I have updated PR with the proposed list of metrics. The list currently contains 10 metrics.
   I can see the need to have just a few important metrics.
   However, I can also see that the Python UDF implementation is complex with many moving parts, metrics can help with troubleshooting with corner cases too, in those circumstances more metrics mean more flexibility and more chances to find where the root cause of the problem is.
   
   Another point for discussion is how accurate are the metrics in the current implementation? I have run a few tests to check that the values measured make sense and are in the ballpark of what was expected.  There is room to do more tests with some corner cases to understand how the instrumentation copes there.
   
   In particular, measuring execution time can be challenging at times, as with this we attempt to do all measuremetns from JVM/Scala code. I am aiming to do a “good enough to be useful” job for the timing metrics, rather than a precise timing. I have put in the metrics description some hints of the nuances I found when testing. 
   
   I think the "time spent sending data" can be useful when troubleshooting cases where the performance problem is with sending large amounts of data to Python. Time spent executing is the key metric to understand the overall performance. Number of rows returned and processed are also useful metric, to understand how much work has been done and how much still needs to be done when monitoring the progress of an active query.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #31367: [SPARK-34265][PYTHON][SQL] Instrument Python UDF execution using SQL Metrics

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31367:
URL: https://github.com/apache/spark/pull/31367#issuecomment-783506048


   **[Test build #135347 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/135347/testReport)** for PR 31367 at commit [`8e8fee8`](https://github.com/apache/spark/commit/8e8fee82c44ac0bd517c3ffe27c2878ced3b6b98).
    * This patch passes all tests.
    * This patch merges cleanly.
    * This patch adds no public classes.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #31367: [SPARK-34265][PYTHON][SQL] Instrument Python UDF execution using SQL Metrics

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31367:
URL: https://github.com/apache/spark/pull/31367#issuecomment-783478947


   Kubernetes integration test status success
   URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/39927/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on pull request #31367: [SPARK-34265][PYTHON][SQL] Instrument Python UDF using SQL Metrics

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #31367:
URL: https://github.com/apache/spark/pull/31367#issuecomment-770008079


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/134660/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #31367: [SPARK-34265][PYTHON][SQL] Instrument Python UDF using SQL Metrics

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31367:
URL: https://github.com/apache/spark/pull/31367#issuecomment-769955167


   Kubernetes integration test status success
   URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/39247/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #31367: [SPARK-34265][PYTHON][SQL] Instrument Python UDF using SQL Metrics

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31367:
URL: https://github.com/apache/spark/pull/31367#issuecomment-768897541


   **[Test build #134606 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/134606/testReport)** for PR 31367 at commit [`22e9999`](https://github.com/apache/spark/commit/22e999900b061eb497f8ebdb1244d0697aaddb0e).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on pull request #31367: [SPARK-34265][PYTHON][SQL] Instrument Python UDF using SQL Metrics

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #31367:
URL: https://github.com/apache/spark/pull/31367#issuecomment-770008079


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/134660/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on pull request #31367: [SPARK-34265][PYTHON][SQL] Instrument Python UDF execution using SQL Metrics

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #31367:
URL: https://github.com/apache/spark/pull/31367#issuecomment-845867972






-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #31367: [SPARK-34265][PYTHON][SQL] Instrument Python UDF using SQL Metrics

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31367:
URL: https://github.com/apache/spark/pull/31367#issuecomment-769999567


   **[Test build #134660 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/134660/testReport)** for PR 31367 at commit [`c16968f`](https://github.com/apache/spark/commit/c16968f36cb589fbe3d13f60f95240af516fc12f).
    * This patch passes all tests.
    * This patch merges cleanly.
    * This patch adds no public classes.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #31367: [SPARK-34265][PYTHON][SQL] Instrument Python UDF using SQL Metrics

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31367:
URL: https://github.com/apache/spark/pull/31367#issuecomment-768925151


   Kubernetes integration test starting
   URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/39194/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on pull request #31367: [SPARK-34265][PYTHON][SQL] Instrument Python UDF using SQL Metrics

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #31367:
URL: https://github.com/apache/spark/pull/31367#issuecomment-768407960


   Can one of the admins verify this patch?


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] maropu commented on pull request #31367: [SPARK-34265][PYTHON][SQL] Instrument Python UDF using SQL Metrics

Posted by GitBox <gi...@apache.org>.
maropu commented on pull request #31367:
URL: https://github.com/apache/spark/pull/31367#issuecomment-768868009


   cc: @HyukjinKwon @ueshin 


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA removed a comment on pull request #31367: [SPARK-34265][PYTHON][SQL] Instrument Python UDF execution using SQL Metrics

Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #31367:
URL: https://github.com/apache/spark/pull/31367#issuecomment-845744466


   **[Test build #138798 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/138798/testReport)** for PR 31367 at commit [`9734f9c`](https://github.com/apache/spark/commit/9734f9cb61d7a12987bca780f1ae19769ddda0c6).


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #31367: [SPARK-34265][PYTHON][SQL] Instrument Python UDF execution using SQL Metrics

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31367:
URL: https://github.com/apache/spark/pull/31367#issuecomment-885086059


   Kubernetes integration test status success
   URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/46021/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] LucaCanali commented on a change in pull request #31367: [SPARK-34265][PYTHON][SQL] Instrument Python UDF using SQL Metrics

Posted by GitBox <gi...@apache.org>.
LucaCanali commented on a change in pull request #31367:
URL: https://github.com/apache/spark/pull/31367#discussion_r566863847



##########
File path: python/pyspark/sql/tests/test_pandas_sqlmetrics.py
##########
@@ -0,0 +1,65 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+import unittest
+
+from pyspark.sql.functions import pandas_udf
+from pyspark.testing.sqlutils import ReusedSQLTestCase, have_pandas, have_pyarrow, \
+    pandas_requirement_message, pyarrow_requirement_message
+
+
+@unittest.skipIf(
+    not have_pandas or not have_pyarrow,
+    pandas_requirement_message or pyarrow_requirement_message)  # type: ignore[arg-type]
+class PandasSQLMetrics(ReusedSQLTestCase):

Review comment:
       Good point. I am now sure on how to do a Scala side test for Python UDF though.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] LucaCanali edited a comment on pull request #31367: [SPARK-34265][PYTHON][SQL] Instrument Python UDF using SQL Metrics

Posted by GitBox <gi...@apache.org>.
LucaCanali edited a comment on pull request #31367:
URL: https://github.com/apache/spark/pull/31367#issuecomment-769738766


   I have updated PR with the proposed list of metrics. The list currently contains 10 metrics.
   I can see the need to have just a few important metrics.
   However, I can also see that the Python UDF implementation is complex with many moving parts, metrics can help with troubleshooting with corner cases too, in those circumstances more metrics mean more flexibility and more chances to find where the root cause of the problem is.
   
   Another point for discussion is how accurate are the metrics in the current implementation? I have run a few tests to check that the values measured make sense and are in the ballpark of what expected.  There is room to do more tests with some corner cases to understand how the instrumentation keeps up there.
   
   In particular, measuring execution time can be challenging at times, as with this we attempt to do it from JVM. I am aiming to go a “good enough to be useful” job for the timing metrics, rather than a precise timing. I have put in the metrics description some hints to the nuances I found when testing. Send and receive time back and forth between JVM and Python workers seem to overlap in some cases, due to the use of queues.
   
   I think the "time spent sending data" can be useful when troubleshooting cases where the performance problem is with sending a lot of data to Python. Time spent executing is probably the key metric to understand the overall performance. Number of rows returned will probably be another useful one, to understand the progress of the execution when monitoring the progress of an active query.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #31367: [SPARK-34265][PYTHON][SQL] Instrument Python UDF using SQL Metrics

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31367:
URL: https://github.com/apache/spark/pull/31367#issuecomment-769920360


   **[Test build #134659 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/134659/testReport)** for PR 31367 at commit [`ee24c5c`](https://github.com/apache/spark/commit/ee24c5ca951b5f9d14e28b853df3bb7de484cb6d).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #31367: [SPARK-34265][PYTHON][SQL] Instrument Python UDF using SQL Metrics

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31367:
URL: https://github.com/apache/spark/pull/31367#issuecomment-770014623


   **[Test build #134666 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/134666/testReport)** for PR 31367 at commit [`0932597`](https://github.com/apache/spark/commit/0932597ec268ca52b51890f54be22c90684acf22).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA removed a comment on pull request #31367: [SPARK-34265][PYTHON][SQL] Instrument Python UDF using SQL Metrics

Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #31367:
URL: https://github.com/apache/spark/pull/31367#issuecomment-769126081


   **[Test build #134619 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/134619/testReport)** for PR 31367 at commit [`c6a1bdc`](https://github.com/apache/spark/commit/c6a1bdc82bf1384761610cffc3df7eb03bd30c5a).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #31367: [SPARK-34265][PYTHON][SQL] Instrument Python UDF using SQL Metrics

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31367:
URL: https://github.com/apache/spark/pull/31367#issuecomment-769171297


   Kubernetes integration test starting
   URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/39207/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #31367: [SPARK-34265][PYTHON][SQL] Instrument Python UDF execution using SQL Metrics

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31367:
URL: https://github.com/apache/spark/pull/31367#issuecomment-783452757


   Kubernetes integration test starting
   URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/39927/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on pull request #31367: [SPARK-34265][PYTHON][SQL] Instrument Python UDF using SQL Metrics

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #31367:
URL: https://github.com/apache/spark/pull/31367#issuecomment-768992997


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/134606/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #31367: [SPARK-34265][PYTHON][SQL] Instrument Python UDF using SQL Metrics

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31367:
URL: https://github.com/apache/spark/pull/31367#issuecomment-770033178


   Kubernetes integration test starting
   URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/39253/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA removed a comment on pull request #31367: [SPARK-34265][PYTHON][SQL] Instrument Python UDF using SQL Metrics

Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #31367:
URL: https://github.com/apache/spark/pull/31367#issuecomment-768897541


   **[Test build #134606 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/134606/testReport)** for PR 31367 at commit [`22e9999`](https://github.com/apache/spark/commit/22e999900b061eb497f8ebdb1244d0697aaddb0e).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] maropu commented on pull request #31367: [SPARK-34265][PYTHON][SQL] Instrument Python UDF using SQL Metrics

Posted by GitBox <gi...@apache.org>.
maropu commented on pull request #31367:
URL: https://github.com/apache/spark/pull/31367#issuecomment-769770078


   > I have updated PR with the proposed list of metrics. The list currently contains 10 metrics.
   
   Could you use texts to list up the metrics  instead of the image? That's because the PR description will appear in the commit logs after this PR merged and the image data makes it hard to search commit logs, I think.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on pull request #31367: [SPARK-34265][PYTHON][SQL] Instrument Python UDF using SQL Metrics

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #31367:
URL: https://github.com/apache/spark/pull/31367#issuecomment-770064614


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/39253/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #31367: [SPARK-34265][PYTHON][SQL] Instrument Python UDF execution using SQL Metrics

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31367:
URL: https://github.com/apache/spark/pull/31367#issuecomment-845744466


   **[Test build #138798 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/138798/testReport)** for PR 31367 at commit [`9734f9c`](https://github.com/apache/spark/commit/9734f9cb61d7a12987bca780f1ae19769ddda0c6).


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA removed a comment on pull request #31367: [SPARK-34265][PYTHON][SQL] Instrument Python UDF execution using SQL Metrics

Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #31367:
URL: https://github.com/apache/spark/pull/31367#issuecomment-783406963


   **[Test build #135347 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/135347/testReport)** for PR 31367 at commit [`8e8fee8`](https://github.com/apache/spark/commit/8e8fee82c44ac0bd517c3ffe27c2878ced3b6b98).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] LucaCanali commented on a change in pull request #31367: [SPARK-34265][PYTHON][SQL] Instrument Python UDF using SQL Metrics

Posted by GitBox <gi...@apache.org>.
LucaCanali commented on a change in pull request #31367:
URL: https://github.com/apache/spark/pull/31367#discussion_r568740939



##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/python/PythonSQLMetrics.scala
##########
@@ -0,0 +1,57 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.python
+
+import org.apache.spark.sql.execution.SparkPlan
+import org.apache.spark.sql.execution.metric.SQLMetrics
+
+private[python] trait PythonSQLMetrics extends SparkPlan {
+  override val metrics = Map(

Review comment:
       My original thought here was to use a trait so that we can reuse the metrics definitaion with the many physical plans that need it (BatchEvalPythonExec, ArrowEcalPythonExec, AggregateInPandasExec, etc). Other implementations of this can work too. I see that `ShuffleRowRDD` uses a custom object `SQLShuffleMetricsReporter` to define and handle the metrics. Can you please elaborate a bit more on your proposal?




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on pull request #31367: [SPARK-34265][PYTHON][SQL] Instrument Python UDF using SQL Metrics

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #31367:
URL: https://github.com/apache/spark/pull/31367#issuecomment-770087727


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/134666/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org