[jira] [Updated] (SAMZA-1158) Adding monitor to clean up stale local stores of tasks.

["Shanthoosh Venkataraman (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/SAMZA-1158?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Shanthoosh Venkataraman updated SAMZA-1158:
-------------------------------------------
    Attachment: 0001-SAMZA-1158-Adding-monitor-to-clean-up-unused-stale-l.patch

Attaching the manager patch that cleans up stale task stores on an yarn node manager. 

> Adding monitor to clean up stale local stores of tasks.
> -------------------------------------------------------
>
>                 Key: SAMZA-1158
>                 URL: https://issues.apache.org/jira/browse/SAMZA-1158
>             Project: Samza
>          Issue Type: New Feature
>    Affects Versions: 0.13.0
>            Reporter: Shanthoosh Venkataraman
>            Assignee: Shanthoosh Venkataraman
>         Attachments: 0001-SAMZA-1158-Adding-monitor-to-clean-up-unused-stale-l.patch
>
>
> Add samza-rest manager that periodically cleans up the stale local stores of dead jobs/tasks. 
> It performs the store deletion in two phases. Initially it deletes the offset file in the local task stores if the following condition is true. ((jobIsNotRunning || preferedHost != nmHost) && offsetFilelastModifiedTime is greater than deleteRetention). 
> During the subsequent run, it deletes the local task stores if it does not contain offset file. 
> Refer to the design doc of SAMZA-656 (https://issues.apache.org/jira/secure/attachment/12828083/DESIGN-SAMZA-656.pdf) for more details.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)