You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-issues@hadoop.apache.org by "Aaron Fabbri (JIRA)" <ji...@apache.org> on 2017/03/09 22:54:38 UTC
[jira] [Comment Edited] (HADOOP-14154) Set isAuthoritative flag
when creating DirListingMetadata in DynamoDBMetaStore
[ https://issues.apache.org/jira/browse/HADOOP-14154?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15904022#comment-15904022 ]
Aaron Fabbri edited comment on HADOOP-14154 at 3/9/17 10:54 PM:
----------------------------------------------------------------
{quote}
Would that isAuthoritative flag have to be setup by higher level applications like Pig/Hive/MR?
{quote}
No, it is internal to S3A. S3A can tell when it has the full listing for a directory, and simply conveys that to the MetadataStore by setting the isAuthoritative bit.
e.g. in {{S3Guard#dirListingUnion(..)}}, it always sets the flag when it puts the listing into the MetadataStore, since this function always has the full listing for the directory:
{code}
dirMeta.setAuthoritative(true); // This is the full directory contents
{code}
That codepath happens at the end of listStatus(), when it has finished computing the full directory contents.
was (Author: fabbri):
{quote}
Would that isAuthoritative flag have to be setup by higher level applications like Pig/Hive/MR?
{quote}
No, it is internal to S3A. S3A can tell when it has the full listing for a directory, and simply conveys that to the MetadataStore by setting the isAuthoritative bit.
e.g. in {{S3Guard#dirListingUnion(..)}}, it always sets the flag when it puts the listing into the MetadataStore, since this function always has the full listing for the directory:
{code}
dirMeta.setAuthoritative(true); // This is the full directory contents
{code}
> Set isAuthoritative flag when creating DirListingMetadata in DynamoDBMetaStore
> ------------------------------------------------------------------------------
>
> Key: HADOOP-14154
> URL: https://issues.apache.org/jira/browse/HADOOP-14154
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
> Reporter: Rajesh Balamohan
> Priority: Minor
> Attachments: HADOOP-14154-HADOOP-13345.001.patch, HADOOP-14154-HADOOP-13345.002.patch
>
>
> Currently {{DynamoDBMetaStore::listChildren}} does not populate {{isAuthoritative}} flag when creating {{DirListingMetadata}}.
> This causes additional S3 lookups even when users have enabled {{fs.s3a.metadatastore.authoritative}}.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org