Dynamic allocation not releasing executors after unpersisting all cached data

[Jeffrey Charles <je...@vidyard.com>]

I'm persisting a dataframe in Zeppelin which has dynamic allocation enabled
to get a sense of how much memory the dataframe takes up. After I note the
size, I unpersist the dataframe. For some reason, Yarn is not releasing the
executors that were added to Zeppelin. If I don't run the persist and
unpersist steps, the executors that were added are removed about a minute
after the paragraphs complete. Looking at the storage tab in the Spark UI
for the Zeppelin job, I don't see anything cached.

Is there any way to get Yarn to automatically remove executors after doing
a persist followed by an unpersist if there is no activity on the executor
within the configured dynamic allocation timeout (similar to how it works
without a persist/unpersist cycle) without having to set
spark.dynamicAllocation.cachedExecutorIdleTimeout? The main reason I'd like
to avoid setting that configuration is I do not want to the executors being
reclaimed if they do have cached data.

Re: Dynamic allocation not releasing executors after unpersisting all cached data

[Jeffrey Charles <je...@vidyard.com>]

Thanks for the suggestion. I gave it a try but the executor still isn't
being released several minutes after running that.

On Mon, Jul 9, 2018 at 3:51 PM Vadim Semenov <va...@datadoghq.com> wrote:

> Try doing `unpersist(blocking=true)`
> On Mon, Jul 9, 2018 at 2:59 PM Jeffrey Charles
> <je...@vidyard.com> wrote:
> >
> > I'm persisting a dataframe in Zeppelin which has dynamic allocation
> enabled to get a sense of how much memory the dataframe takes up. After I
> note the size, I unpersist the dataframe. For some reason, Yarn is not
> releasing the executors that were added to Zeppelin. If I don't run the
> persist and unpersist steps, the executors that were added are removed
> about a minute after the paragraphs complete. Looking at the storage tab in
> the Spark UI for the Zeppelin job, I don't see anything cached.
> >
> > Is there any way to get Yarn to automatically remove executors after
> doing a persist followed by an unpersist if there is no activity on the
> executor within the configured dynamic allocation timeout (similar to how
> it works without a persist/unpersist cycle) without having to set
> spark.dynamicAllocation.cachedExecutorIdleTimeout? The main reason I'd like
> to avoid setting that configuration is I do not want to the executors being
> reclaimed if they do have cached data.
>
>
>
> --
> Sent from my iPhone
>

Re: Dynamic allocation not releasing executors after unpersisting all cached data

[Vadim Semenov <va...@datadoghq.com>]

Try doing `unpersist(blocking=true)`
On Mon, Jul 9, 2018 at 2:59 PM Jeffrey Charles
<je...@vidyard.com> wrote:
>
> I'm persisting a dataframe in Zeppelin which has dynamic allocation enabled to get a sense of how much memory the dataframe takes up. After I note the size, I unpersist the dataframe. For some reason, Yarn is not releasing the executors that were added to Zeppelin. If I don't run the persist and unpersist steps, the executors that were added are removed about a minute after the paragraphs complete. Looking at the storage tab in the Spark UI for the Zeppelin job, I don't see anything cached.
>
> Is there any way to get Yarn to automatically remove executors after doing a persist followed by an unpersist if there is no activity on the executor within the configured dynamic allocation timeout (similar to how it works without a persist/unpersist cycle) without having to set spark.dynamicAllocation.cachedExecutorIdleTimeout? The main reason I'd like to avoid setting that configuration is I do not want to the executors being reclaimed if they do have cached data.



-- 
Sent from my iPhone

---------------------------------------------------------------------
To unsubscribe e-mail: user-unsubscribe@spark.apache.org