[jira] [Reopened] (SPARK-21352) Memory Usage in Spark Streaming

["Shubham Gupta (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/SPARK-21352?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Shubham Gupta reopened SPARK-21352:
-----------------------------------

There is no solution provided for the problem and neither stack overflow helping


> Memory Usage in Spark Streaming
> -------------------------------
>
>                 Key: SPARK-21352
>                 URL: https://issues.apache.org/jira/browse/SPARK-21352
>             Project: Spark
>          Issue Type: Improvement
>          Components: DStreams, Spark Submit, YARN
>    Affects Versions: 2.1.1
>            Reporter: Shubham Gupta
>              Labels: newbie
>
> I am trying to figure out the memory used by executors for a Spark Streaming job. For data I am using the rest endpoint for Spark AllExecutors and just summing up the metrics totalDuration * spark.executor.memory for every executor and then emitting the final sum as the memory usage.
> But this is coming out to be very small for application which ran whole day , is something wrong with the logic.Also I am using dynamic allocation and executorIdleTimeout is 5 seconds.
> Also I am also assuming that if some executor was removed for due to idle timeout and then was allocated to some other task then its totalDuration will be increased by the amount of time took by the executor to execute this new task.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org