[jira] [Updated] (HADOOP-19354) S3A: S3AInputStream to be created by factory under S3AStore

["Steve Loughran (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/HADOOP-19354?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Steve Loughran updated HADOOP-19354:
------------------------------------
    Summary: S3A: S3AInputStream to be created by factory under S3AStore  (was: S3AInputStream to be created by factory under S3AStore)

> S3A: S3AInputStream to be created by factory under S3AStore
> -----------------------------------------------------------
>
>                 Key: HADOOP-19354
>                 URL: https://issues.apache.org/jira/browse/HADOOP-19354
>             Project: Hadoop Common
>          Issue Type: Sub-task
>          Components: fs/s3
>    Affects Versions: 3.4.2
>            Reporter: Steve Loughran
>            Assignee: Steve Loughran
>            Priority: Major
>
> Migrate S3AInputStream creation into a factory pattern, push down into S3AStore.
> Proposed factories
> * default: whatever this release has as default
> * classic: current S3AInputStream
> * prefetch: prefetching
> * analytics: new analytics stream
> * other: reads a classname from another prop, instantiates.
> Also proposed
> * stream to implement some stream capability to declare what they are (classic, prefetch, analytics, other). 
> h2. Implementation
> All callbacks used by the stream also to call directly onto S3AStore.
> S3AFileSystem must not be invoked at all (if it is needed: PR is still not ready).
> Some interface from Instrumentation will be passed to factory; this shall include a way to create new per-stream 
> The factory shall implement org.apache.hadoop.service.Service; S3AStore shall do same and become a subclass of CompositeService. It shall attach the factory as a child, so they can follow the same lifecycle. We shall do the same for anything else that gets pushed down.
> Everything related to stream creation must go from s3afs; and creation of the factory itself. This must be done in S3AStore.initialize(). 
> As usual, this will complicate mocking. But the streams themselves should not require changes, at least significant ones.
> Testing.
> * The huge file tests should be tuned so each of the different ones uses a different stream, always.
> * use a -Dstream="factory name" to choose factory, rather than the -Dprefetch
> * if not set, whatever is in auth-keys gets picked up.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org