[jira] [Resolved] (SAMZA-656) Script to periodically clean-up store directories that are persisted outside YARN

["Shanthoosh Venkataraman (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/SAMZA-656?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Shanthoosh Venkataraman resolved SAMZA-656.
-------------------------------------------
    Resolution: Fixed

> Script to periodically clean-up store directories that are persisted outside YARN
> ---------------------------------------------------------------------------------
>
>                 Key: SAMZA-656
>                 URL: https://issues.apache.org/jira/browse/SAMZA-656
>             Project: Samza
>          Issue Type: Task
>            Reporter: Navina Ramesh
>            Assignee: Shanthoosh Venkataraman
>         Attachments: DESIGN-SAMZA-656.pdf, GCstalelocalstate.pdf
>
>
> In order to enable local state re-use when Samza job is executing in YARN or standalone mode, the data store is persisted outside YARN's working directory. 
> If the data store used by an application is persisted on disk beyond the application's lifetime, there has to be a periodic check to delete unused/orphaned stores. This ensures that the Node Managers (NM) don't run out of disk space. 
> There are couple of ways to solve this:
> 1. YARN supports adding auxiliary services that run on the NMs. The generic auxiliary service is event-based and receives notification on application/container start and stop. More than one auxiliary services can be defined on an NM. We can use the callbacks on application and/or container start and/or stop to perform a check-and-purge function on all data stores. We can identify unused data stores by checking for existence of file handlers that are not pid=1.
> >*Pros*:
> > - Becomes part of existing samza-yarn module
> >
> >*Cons*:
> > - It is not a generic solution. If the developer uses other resource managers like Mesos, then there is no good support for purging the data.
>  
> 2. Run a daemon on the NM that periodically cleans up all store directories. 
> >*Pros*:
> > - Independent of the underlying resource manager
> >
> >*Cons*:
> > - This becomes a pre-requisite for anyone who runs a YARN grid and wants to run an Samza job.
> There is a proposal for YARN to support post-application clean-up in [YARN-2261][YARN-2261]. This is a WIP and it seems to be applicable only for Capacity Scheduler.
> We can use the YARN NM's auxiliary service to trigger a clean-up task , as suggested in [1]. If needed in future, we can move the clean-up logic to script and make it accessible to other developers who want to run it as a daemon on NM nodes. 



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)