You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@spark.apache.org by "angers.zhu" <an...@gmail.com> on 2021/01/11 01:37:43 UTC
Chnage Restful API's default quanlites same as WebUI
Hi devs,
These days I found that about taskSummary, the Restful API(/applications/[app-
id]/stages/[stage-id]/[stage-attempt-id]/taskSummary)'s default quantiles is
`0.05,0.25,0.5,0.75,0.95`
But in Spark Web UI’s stage page, the quantiles is `0.0,0.25,0.5,0.75,1.0`.
And the restful api’s default value
is not shown in Spark website’s [monitoring
page](https://spark.apache.org/docs/3.0.0/monitoring.html). Detail can be seen
in https://github.com/apache/spark/pull/31048#issuecomment-756773764
I have create a PR to keep them consistence, and hope you dev’s visibility
and comments.
Best regards
Angers
[  |
angers.zhu
---|---
angers.zhu@gmail.com
](https://maas.mail.163.com/dashi-web-
extend/html/proSignature.html?ftlId=1&name=angers.zhu&uid=angers.zhu%40gmail.com&iconUrl=https%3A%2F%2Fmail-
online.nosdn.127.net%2Fqiyelogo%2FdefaultAvatar.png&items=%5B%22angers.zhu%40gmail.com%22%5D)
签名由 [网易邮箱大师](https://mail.163.com/dashi/dlpro.html?from=mail81) 定制
Re: Chnage Restful API's default quanlites same as WebUI
Posted by ron8hu <ro...@gmail.com>.
To me, this is a bug. In a parallel system, the duration of a stage is often
determined by the slowest task/executor. To monitor/debug a skew issue, the
maximal value (or 100% percentile value) is more useful than the 95%
percentile value. I am happy to see that we do it right with Web UI. For
REST API, it should be consistent with Web UI.
--
Sent from: http://apache-spark-developers-list.1001551.n3.nabble.com/
---------------------------------------------------------------------
To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
Re: Chnage Restful API's default quanlites same as WebUI
Posted by Sean Owen <sr...@gmail.com>.
The change makes sense of course; I think the real question for everyone
is: would you change the semantics of this info from the API in a minor
release, or would it have to wait? is it more bug fix than breaking change?
On Sun, Jan 10, 2021 at 7:38 PM angers.zhu <an...@gmail.com> wrote:
> Hi devs,
>
> These days I found that about taskSummary, the Restful API(
> /applications/[app-id]/stages/[stage-id]/[stage-attempt-id]/taskSummary)'s
> default quantiles is `0.05,0.25,0.5,0.75,0.95`
> But in Spark Web UI’s stage page, the quantiles is
> `0.0,0.25,0.5,0.75,1.0`. And the restful api’s default value
> is not shown in Spark website’s [monitoring page](
> https://spark.apache.org/docs/3.0.0/monitoring.html). Detail can be seen
> in https://github.com/apache/spark/pull/31048#issuecomment-756773764
>
> I have create a PR to keep them consistence, and hope you dev’s visibility
> and comments.
>
> Best regards
> Angers
> angers.zhu
> angers.zhu@gmail.com
>
> <https://maas.mail.163.com/dashi-web-extend/html/proSignature.html?ftlId=1&name=angers.zhu&uid=angers.zhu%40gmail.com&iconUrl=https%3A%2F%2Fmail-online.nosdn.127.net%2Fqiyelogo%2FdefaultAvatar.png&items=%5B%22angers.zhu%40gmail.com%22%5D>
> 签名由 网易邮箱大师 <https://mail.163.com/dashi/dlpro.html?from=mail81> 定制
>
>