You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-issues@hadoop.apache.org by "Aaron Fabbri (JIRA)" <ji...@apache.org> on 2017/06/22 00:17:00 UTC
[jira] [Comment Edited] (HADOOP-14499) Findbugs warning in
LocalMetadataStore
[ https://issues.apache.org/jira/browse/HADOOP-14499?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16058469#comment-16058469 ]
Aaron Fabbri edited comment on HADOOP-14499 at 6/22/17 12:16 AM:
-----------------------------------------------------------------
Yeah sorry for delay responding to your earlier comment. I think we need a more rigorous definition of prune(), at least for my understanding. Would be nice to write up the rules for prune, i.e. isAuthoritative (you have to clear bit on parent if you prune a child, etc) and have more contract tests around it.
Personally I'd just fix the findbugs warning in this JIRA. :-)
My previous concern was in deleting empty directories regardless of age: They do need to be present during the inconsistency window. i.e. getFileStatus(empty-dir-path) should see empty directories in MetadataStore until the consistency window (prune age) has elapsed. Maybe I misread your original patch?
{quote}
Since we currently don't track the last time at which a directory listing was modified
{quote}
Couldn't a MetadataStore track the time the entry was last written into the MetadataStore?
There are two "modification" times we can consider:
A. The time which an entry was (last) written into a MetadataStore
B. The FileStatus's modification time field.
My thinking is that prune is based on A (mostly because B is unreliable): The sliding inconsistency window. It works nicely for both inconsistency and caching (unlike B). For files, we can use the fact that A>=B.
Anyways, it seems to me that a MetadataStore could safely expire and/or prune directory metadata, so I'm not sure the comment change is needed?
was (Author: fabbri):
Yeah sorry for delay responding to your earlier comment. I think we need a more rigorous definition of prune(), at least for my understanding. Would be nice to write up the rules for isAuthoritative (you have to clear bit on parent if you prune a child, etc) and have more contract tests around it.
Personally I'd just fix the findbugs warning in this JIRA. :-)
My previous concern was in deleting empty directories regardless of age: They do need to be present during the inconsistency window. i.e. getFileStatus(empty-dir-path) should see empty directories in MetadataStore until the consistency window (prune age) has elapsed. Maybe I misread your original patch?
{quote}
Since we currently don't track the last time at which a directory listing was modified
{quote}
Couldn't a MetadataStore track the time the entry was last written into the MetadataStore?
There are two "modification" times we can consider:
A. The time which an entry was (last) written into a MetadataStore
B. The FileStatus's modification time field.
My thinking is that prune is based on A (mostly because B is unreliable): The sliding inconsistency window. It works nicely for both inconsistency and caching (unlike B). For files, we can use the fact that A>=B.
Anyways, it seems to me that a MetadataStore could safely expire and/or prune directory metadata, so I'm not sure the comment change is needed?
> Findbugs warning in LocalMetadataStore
> --------------------------------------
>
> Key: HADOOP-14499
> URL: https://issues.apache.org/jira/browse/HADOOP-14499
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
> Reporter: Sean Mackrory
> Assignee: Sean Mackrory
> Attachments: HADOOP-14499-HADOOP-13345.001.patch, HADOOP-14499-HADOOP-13345.002.patch
>
>
> First saw this raised by Yetus on HADOOP-14433:
> {code}
> Bug type UC_USELESS_OBJECT (click for details)
> In class org.apache.hadoop.fs.s3a.s3guard.LocalMetadataStore
> In method org.apache.hadoop.fs.s3a.s3guard.LocalMetadataStore.prune(long)
> Value ancestors
> Type java.util.LinkedList
> At LocalMetadataStore.java:[line 300]
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org