[jira] [Updated] (CRUNCH-669) Add an option to preserve Crunch temp directories

["Josh Wills (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/CRUNCH-669?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Josh Wills updated CRUNCH-669:
------------------------------
    Attachment: CRUNCH-669.patch

> Add an option to preserve Crunch temp directories
> -------------------------------------------------
>
>                 Key: CRUNCH-669
>                 URL: https://issues.apache.org/jira/browse/CRUNCH-669
>             Project: Crunch
>          Issue Type: Improvement
>            Reporter: Josh Wills
>            Priority: Major
>         Attachments: CRUNCH-669.patch
>
>
> I have a problem where a Crunch client can potentially get killed through no fault of its own (e.g., an Airflow task failing a heartbeat check), which will kill the client, but leave the MR job itself still running. The problem is that the finalize() operation can delete the temp directory that contains the config info for Crunch, which will make any subsequent tasks fail to run to completion b/c their config info has been removed, or because some temp output in the directory that would have been useful can be deleted. I'd like to add an option to override the automatic delete of the temp directory during the finalize() operation for these situations.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)