You are viewing a plain text version of this content. The canonical link for it is here.

Posted to issues@spark.apache.org by "Sean Owen (JIRA)" <ji...@apache.org> on 2017/05/03 08:59:04 UTC

[jira] [Commented] (SPARK-20568) Delete files after processing

    [ https://issues.apache.org/jira/browse/SPARK-20568?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15994502#comment-15994502 ] 

Sean Owen commented on SPARK-20568:
-----------------------------------

This conflicts with the general Spark model that operations be rerunnable in case of failure. It isn't how anything else in Spark works. The need is app specific and can be solved by app code or other mechanisms. I would not support this. 

> Delete files after processing
> -----------------------------
>
>                 Key: SPARK-20568
>                 URL: https://issues.apache.org/jira/browse/SPARK-20568
>             Project: Spark
>          Issue Type: New Feature
>          Components: Structured Streaming
>    Affects Versions: 2.1.0
>            Reporter: Saul Shanabrook
>
> It would be great to be able to delete files after processing them with structured streaming.
> For example, I am reading in a bunch of JSON files and converting them into Parquet. If the JSON files are not deleted after they are processed, it quickly fills up my hard drive. I originally [posted this on Stack Overflow|http://stackoverflow.com/q/43671757/907060] and was recommended to make a feature request for it. 



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org