[GitHub] [spark] ron8hu commented on a change in pull request #29247: [SPARK-32446][SHS] Add percentile distribution REST API of peak memory metrics for all executors

[GitBox <gi...@apache.org>]

ron8hu commented on a change in pull request #29247:
URL: https://github.com/apache/spark/pull/29247#discussion_r551608679



##########
File path: core/src/main/scala/org/apache/spark/status/api/v1/OneApplicationResource.scala
##########
@@ -52,6 +53,25 @@ private[v1] class AbstractApplicationResource extends BaseAppResource {
   @Path("executors")
   def executorList(): Seq[ExecutorSummary] = withUI(_.store.executorList(true))
 
+  @GET
+  @Path("executorMetricsDistribution")
+  def executorSummary(
+      @QueryParam("activeOnly") @DefaultValue("true") activeOnly: Boolean,
+      @DefaultValue("0.05,0.25,0.5,0.75,0.95") @QueryParam("quantiles") quantileString: String)

Review comment:
       FYI.  In the corresponding web UI, the quantiles of Min/Max are displayed in the table "Summary Metrics for Completed Tasks" for a given stage page.  In a parallel system, the duration of a stage is often determined by the slowest task/executor. To monitor/debug a skew issue, the maximal value (or 100% percentile value) is more useful than the 95% percentile value.  On the other hand, 95% percentile value has been used in the past.  One wise man once told me: Consistency means to repeat yesterday's mistake. 




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org