[jira] [Commented] (HADOOP-13926) S3Guard: S3AFileSystem::listLocatedStatus() to employ MetadataStore

["Mingliang Liu (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/HADOOP-13926?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15951839#comment-15951839 ] 

Mingliang Liu commented on HADOOP-13926:
----------------------------------------

Thanks, [~fabbri]!

{quote}
For unblocking merge to trunk, how about the caveat that S3Guard list consistency does not support listFiles() yet? You simply get S3 results without additional consistency guarantees and we'd implement listFiles() after merge.
{quote}
I'm OK with that idea. I file a new JIRA [HADOOP-14266] to track effort of employing MetadataStore in {{S3AFileSystem::listFiles()}}. Supporting listFiles(recursive=true) many many children is indeed challenging. Make this JIRA focus on the {{listLocatedStatus()}} FileSystem API.

I'll update the patch to union results and populate to MS for the case of non-authoritative. I was thinking similar ideas to what you suggested: save the list while iterating S3 objects and populate in the end. A set will be our helper. We may optimize this part later if reading from MetadataStore is also w/ paging. By now, I think it's OK to use this straightforward algorithm.

Joint effort will be (as always) appreciated!

> S3Guard: S3AFileSystem::listLocatedStatus() to employ MetadataStore
> -------------------------------------------------------------------
>
>                 Key: HADOOP-13926
>                 URL: https://issues.apache.org/jira/browse/HADOOP-13926
>             Project: Hadoop Common
>          Issue Type: Sub-task
>          Components: fs/s3
>            Reporter: Rajesh Balamohan
>            Assignee: Steve Loughran
>         Attachments: HADOOP-13926-HADOOP-13345.001.patch, HADOOP-13926.wip.proto.branch-13345.1.patch
>
>
> Need to check if {{listLocatedStatus}} can make use of metastore's listChildren feature.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org