You are viewing a plain text version of this content. The canonical link for it is here.

Posted to issues@spark.apache.org by "Tathagata Das (JIRA)" <ji...@apache.org> on 2016/03/26 04:09:25 UTC

[jira] [Resolved] (SPARK-14109) HDFSMetadataLog throws AbstractFilesSystem exception with common schemes like s3n

     [ https://issues.apache.org/jira/browse/SPARK-14109?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Tathagata Das resolved SPARK-14109.
-----------------------------------
    Resolution: Fixed

> HDFSMetadataLog throws AbstractFilesSystem exception with common schemes like s3n
> ---------------------------------------------------------------------------------
>
>                 Key: SPARK-14109
>                 URL: https://issues.apache.org/jira/browse/SPARK-14109
>             Project: Spark
>          Issue Type: Bug
>          Components: SQL
>            Reporter: Tathagata Das
>            Assignee: Tathagata Das
>
> HDFSMetadataLog uses newer FileContext API to achieve atomic renaming. However, FileContext implementations may not exist for many scheme for which there may be FileSystem implementations. In those cases, rather than failing completely, we should fallback to the FileSystem based implementation, and log warning that there may be file consistency issues in case the log directory is concurrently modified.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org