[jira] [Commented] (MESOS-740) Create an Archive abstraction.

["Thomas Rampelberg (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/MESOS-740?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14134593#comment-14134593 ] 

Thomas Rampelberg commented on MESOS-740:
-----------------------------------------

This is going to become a bigger issue with the new cli. Right now, the CLI treats the master as the one true source of information. When you have a framework that is running a large number of tasks, the current buffer fills up quickly and you end up loosing the task id. For slaves that still have the files of a potentially failed task still around (because the buffer there is totally different), you're no longer able to discover where the task ran from the master.

It would be great if the archive was a module that let you define what retention policy you had for older, completed tasks.

> Create an Archive abstraction.
> ------------------------------
>
>                 Key: MESOS-740
>                 URL: https://issues.apache.org/jira/browse/MESOS-740
>             Project: Mesos
>          Issue Type: Improvement
>            Reporter: Benjamin Mahler
>
> I was thinking about this lately given the resource monitor, the slave, and the master all keep archived information around at the cost of increased memory consumption.
> An archive abstraction would simplify the code in these components, as well as help ensure the master / slave remain below a certain amount of memory consumption. This proves useful if were were to start placing the slave inside a cgroup with a memory limit.
> The archive could periodically monitor memory consumption (or use memory threshold notifications) to decide when to GC some of the archived information. This would be slightly analogous to how we GC the disk based on disk usage.
> Thinking about what this may look like:
> 1. Archive is map-like and holds the data using <Key, Value>. This would need to be templated and as such may require that separate components create separate Archives.
> 2. Archive distributes Futures that become ready when memory consumption needs to be lowered. Components can set callbacks on these futures to routines that remove data from their historical state. At this point Archive is really just a MemoryMonitor abstraction.
> [~benjaminhindman] I heard you may have been thinking about this as well at one point, let me know what ideas you have!



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)