[jira] [Created] (HADOOP-16844) FS/FC createFile() builder to add a lazy-check option for async probes of the store

["Steve Loughran (Jira)" <ji...@apache.org>]

Steve Loughran created HADOOP-16844:
---------------------------------------

             Summary: FS/FC createFile() builder to add a lazy-check option for async probes of the store
                 Key: HADOOP-16844
                 URL: https://issues.apache.org/jira/browse/HADOOP-16844
             Project: Hadoop Common
          Issue Type: Improvement
          Components: fs, fs/azure, fs/s3
    Affects Versions: 3.3.0
            Reporter: Steve Loughran


For object stores, file creation can triggers checks for: path being to a file, path not being a directory, parent path existing.

These are all done before an output stream is returned -even though nothing may be manifest in the remote store until the screen is flashed or closed.

If we supported lazy checks we could do those probes against the store while the application was actually generating data to write.

Changing the behaviour of open() would be too fundamental. What we can do is add a new option to the createFile() builder which stores can support. When set, the caller is declaring they are happy with late reporting of failures -with any problems surfacing during a write/flush/close. It is notable with S3A permissions issues only surface later on anyway.




--
This message was sent by Atlassian Jira
(v8.3.4#803005)

---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org