[jira] [Created] (HADOOP-13009) add option for lazy open() on s3a

["Steve Loughran (JIRA)" <ji...@apache.org>]

Steve Loughran created HADOOP-13009:
---------------------------------------

             Summary: add option for lazy open() on s3a
                 Key: HADOOP-13009
                 URL: https://issues.apache.org/jira/browse/HADOOP-13009
             Project: Hadoop Common
          Issue Type: Sub-task
    Affects Versions: 2.8.0
            Reporter: Steve Loughran


After lazy-seekI want to add a —very much non-default —lazy-open option.

If you look at a trace of what goes on with object store access, there's usually a GET at offset 0 (the {{open()}} command, followed by a {{seek()}}. 

If there was a lazy option option, then {{open()}} would set up the instance for reading, but not actually talk to the object store —it'd be the first seek or read which would hit the service. You'd eliminate one HTTP operation from a read sequence, for a faster startup time, especially long-haul.

That's a big break in the normal assumption: if a file isn't there, {{open()}} fails, so it'd only work with apps which did open+read, open+seek, or opened+positioned readable action back to back. By making it an option people can experiment to see what happens —though full testing would need to do some fault injection on the first seek/read to see how code handled late failure.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)