[GitHub] [spark] ron8hu commented on pull request #31204: [SPARK-26399][WEBUI][CORE] Add new stage-level REST APIs and parameters

[GitBox <gi...@apache.org>]

ron8hu commented on pull request #31204:
URL: https://github.com/apache/spark/pull/31204#issuecomment-808980380


   > FYI @ron8hu Have rebase to current master's code.
   > Ping @srowen Can you also take a look at this pr?
   
   @AngersZhuuuu  Thanks Angers for updating the code.
   @srowen  This ticket is related to the following two tickets:
   - SPARK-34092: support filtering by task status in REST API call for a specific stage
   - SPARK-34488: Support task Metrics Distributions and executor Metrics Distributions in the REST API call for a specified stage
   
   All 3 tickets use the same set of query parameters {details, WithSummaries, taskStataus}.  The above 2 tickets provide REST API for a specific stage level.  This ticket/PR provides REST API for overall stages of a given application.  Using these REST APIs and their query parameters allow some downstream product, such as Dr. Elephant (https://github.com/linkedin/dr-elephant ), to flexibly generate various json output so that it can analyze various performance issues like skewed load, under/over used memory, GC time, etc.  Dr. Elephant relies on these REST APIs to advise Spark users where they can optimize their configuration parameters.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org