[GitHub] wypoon opened a new pull request #23767: [SPARK-26329][CORE] Faster polling of executor memory metrics.

[GitBox <gi...@apache.org>]

wypoon opened a new pull request #23767: [SPARK-26329][CORE] Faster polling of executor memory metrics.
URL: https://github.com/apache/spark/pull/23767
 
 
   ## What changes were proposed in this pull request?
   
   Prior to this change, in an executor, on each heartbeat, memory metrics are polled and sent in the heartbeat. The heartbeat interval is 10s by default. With this change, in an executor, memory metrics can optionally be polled in a separate poller at a shorter interval.
   
   We use a map of (stageId, stageAttemptId) to number of running tasks in the executor to track what stages are active. We keep a map of (stageId, stageAttemptId) to memory metric peaks. When polling the executor memory metrics, we attribute the memory to the active stage(s), and update the peaks. In a heartbeat, we send the per-stage peaks (for stages active at that time), and then reset the peaks. The semantics would be that the per-stage peaks sent in each heartbeat are the peaks since the last heartbeat.
   
   We also keep a map of taskId to memory metric peaks in the executor. This tracks the metric peaks during the lifetime of the task. The polling thread updates this as well. At end of a task, we send the peak metric values in the task result.
   
   We continue to do the stage-level aggregation in the EventLoggingListener.
   
   For the driver, we still only poll on heartbeats. What the driver sends will be the current values of the metrics in the driver at the time of the heartbeat. This is semantically the same as before.
   
   ## How was this patch tested?
   
   Unit tests. Manually tested applications on an actual system and checked the event logs; the metrics appear in the SparkListenerTaskEnd and SparkListenerStageExecutorMetrics events.
   
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org