You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Shixiong Zhu (JIRA)" <ji...@apache.org> on 2017/03/03 05:20:45 UTC

[jira] [Resolved] (SPARK-19779) structured streaming exist needless tmp file

     [ https://issues.apache.org/jira/browse/SPARK-19779?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Shixiong Zhu resolved SPARK-19779.
----------------------------------
       Resolution: Fixed
    Fix Version/s: 2.2.0
                   2.1.1
                   2.0.3

> structured streaming exist needless tmp file 
> ---------------------------------------------
>
>                 Key: SPARK-19779
>                 URL: https://issues.apache.org/jira/browse/SPARK-19779
>             Project: Spark
>          Issue Type: Bug
>          Components: Structured Streaming
>    Affects Versions: 2.0.3, 2.1.1, 2.2.0
>            Reporter: Feng Gui
>            Priority: Minor
>             Fix For: 2.0.3, 2.1.1, 2.2.0
>
>
> The PR (https://github.com/apache/spark/pull/17012) can to fix restart a Structured Streaming application using hdfs as fileSystem, but also exist a problem that a tmp file of delta file is still reserved in hdfs. And Structured Streaming don't delete the tmp file generated when restart streaming job in future, so we need to delete the tmp file after restart streaming job.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org