You are viewing a plain text version of this content. The canonical link for it is here.

Posted to issues@spark.apache.org by "Reynold Xin (JIRA)" <ji...@apache.org> on 2016/10/22 00:26:58 UTC

[jira] [Resolved] (SPARK-18042) OutputWriter needs to return the path of the file written

     [ https://issues.apache.org/jira/browse/SPARK-18042?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Reynold Xin resolved SPARK-18042.
---------------------------------
       Resolution: Fixed
    Fix Version/s: 2.1.0

> OutputWriter needs to return the path of the file written
> ---------------------------------------------------------
>
>                 Key: SPARK-18042
>                 URL: https://issues.apache.org/jira/browse/SPARK-18042
>             Project: Spark
>          Issue Type: Sub-task
>          Components: SQL
>            Reporter: Reynold Xin
>            Assignee: Reynold Xin
>             Fix For: 2.1.0
>
>
> This patch adds a new "path" method on OutputWriter that returns the path of the file written by the OutputWriter. This is part of the necessary work to consolidate structured streaming and batch write paths.
> The batch write path has a nice feature that each data source can define the extension of the files, and allow Spark to specify the staging directory and the prefix for the files. However, in the streaming path we need to collect the list of files written, and there is no interface right now to do that.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org