[jira] [Created] (HADOOP-18184) s3a prefetching stream to support unbuffer()

["Steve Loughran (Jira)" <ji...@apache.org>]

Steve Loughran created HADOOP-18184:
---------------------------------------

             Summary: s3a prefetching stream to support unbuffer()
                 Key: HADOOP-18184
                 URL: https://issues.apache.org/jira/browse/HADOOP-18184
             Project: Hadoop Common
          Issue Type: Sub-task
          Components: fs/s3
    Affects Versions: 3.4.0
            Reporter: Steve Loughran


Apache Impala uses unbuffer() to free up all client side resources held by a stream, so allowing it to have a map of available (path -> stream) objects, retained across queries.

This saves on having to reopen the files, with the cost of HEAD checks etc. S3AInputStream just closes its http connection. here there is a lot more state to discard, but all memory and file storage must be freed.

until this done, ITestS3AContractUnbuffer must skip when the prefetch stream is used.

its notable that the other tests don't fail, even though the stream doesn't implement the interface; the graceful degradation handles that. it should fail if the test xml resource says the stream does it, but that the stream capabilities say it doesn't.



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org