You are viewing a plain text version of this content. The canonical link for it is here.

Posted to issues@spark.apache.org by "Tathagata Das (JIRA)" <ji...@apache.org> on 2016/03/24 02:44:25 UTC

[jira] [Created] (SPARK-14109) HDFSMetadataLog throws AbstractFilesSystem exception with common schemes like s3n

Tathagata Das created SPARK-14109:
-------------------------------------

             Summary: HDFSMetadataLog throws AbstractFilesSystem exception with common schemes like s3n
                 Key: SPARK-14109
                 URL: https://issues.apache.org/jira/browse/SPARK-14109
             Project: Spark
          Issue Type: Bug
          Components: SQL
            Reporter: Tathagata Das
            Assignee: Tathagata Das


HDFSMetadataLog uses newer FileContext API to achieve atomic renaming. However, FileContext implementations may not exist for many scheme for which there may be FileSystem implementations. In those cases, rather than failing completely, we should fallback to the FileSystem based implementation, and log warning that there may be file consistency issues in case the log directory is concurrently modified.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org