You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ozone.apache.org by "Rakesh Radhakrishnan (Jira)" <ji...@apache.org> on 2020/07/22 12:15:00 UTC
[jira] [Comment Edited] (HDDS-2939) Ozone FS namespace
[ https://issues.apache.org/jira/browse/HDDS-2939?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17162743#comment-17162743 ]
Rakesh Radhakrishnan edited comment on HDDS-2939 at 7/22/20, 12:14 PM:
-----------------------------------------------------------------------
Thanks a lot [~maobaolong] , [~linyiqun] for pointing to the Alluxio metadata structure. Sure, will consider these as a referral docs.
I have created [HDDS-2939 branch|https://github.com/apache/hadoop-ozone/tree/HDDS-2939] for the feature development and posted first [PR-1230|https://github.com/apache/hadoop-ozone/pull/1230] to implement create directory #mkdirs user call. In this patch, I have tried using the existing 'PrefixTable' for storing the dirs. I am also exploring the necessity of a separate new DirTable instead of PrefixTable, which currently used only for keeping the ACLs. Welcome comments, thanks!
There is a plan to maintain an in-memory dir/key cache to avoid the look up overhead and needs to understand the size of this cache. Basically, for every dir lookup, it has to seek the dirs in 'DirTable'. For every file look up, it has to seek all the parents upto the immediate parent dir in 'DirTable' and then seek the file key in the 'KeyTable'.
Below tables shows the existing key structure and the proposed idea. The table content is just a representation to show the internals.
*1) User given Paths under /vol1/buck1:*
|{color:#0747a6}*Type*{color}|{color:#0747a6}*Path*{color}|
|File|/a/b/c/d/file1|
|Dir|/a/b/c/d/e|
|File|/a/b/c/d/file2|
|Dir|/a/b/c/d/e/file1|
*2) Existing KeyTable Data Format:*
|{color:#00875a}*KeyName*{color}|
|/vol1/buck1/a/b/c/d/file1|
|/vol1/buck1/a/b/c/d/e/|
|/vol1/buck1/a/b/c/d/file2|
|/vol1/buck1/a/b/c/d/e/file1/|
*3) Proposed data format -table Dir/File entry: "parentID/Key".* Assume bucket ObjectID is 512, which is the parent for the first level dirs/files
*PrefixTable:*
|{color:#ff8b00}*PrefixName*{color}|{color:#ff8b00}*ObjectID*{color}|
|512/a|1025|
|1025/b|1026|
|1026/c|1027|
|1027/d|1028|
|1028/e|1029|
|1029/file1|1032|
*KeyTable:*
|{color:#ff8b00}*KeyName*{color}|{color:#ff8b00}*ObjectID*{color}|
|1028/file1|1030|
|1028/file2|1031|
was (Author: rakeshr):
Thanks a lot [~maobaolong] , [~linyiqun] for pointing to the Alluxio metadata structure. Sure, will consider these as a referral docs.
I have created [HDDS-2939 branch|https://github.com/apache/hadoop-ozone/tree/HDDS-2939] for the feature development and posted first [PR-1230|https://github.com/apache/hadoop-ozone/pull/1230] to implement create directory #mkdirs user call. In this patch, I have tried using the existing 'PrefixTable' for storing the dirs. I am also exploring the necessity of a separate new DirTable instead of PrefixTable, which currently used only for keeping the ACLs. Welcome comments, thanks!
There is a plan to maintain an in-memory dir/key cache to avoid the look up overhead and needs to understand the size of this cache. Basically, for every dir lookup, it has to seek the dirs in 'DirTable'. For every file look up, it has to seek all the parents upto the immediate parent dir in 'DirTable' and then seek the file key in the 'KeyTable'.
Below tables shows the existing key structure and the proposed idea.
| {color:#00875a} *Existing KeyTable Data Format*{color}|
| *Type*| *Path*| *KeyTable Entry*|
|File|/a/b/c/d/file1|/vol1/buck1/a/b/c/d/file1|
|Dir|/a/b/c/d/e|/vol1/buck1/a/b/c/d/e/|
|File|/a/b/c/d/file2|/vol1/buck1/a/b/c/d/file2|
|Dir|/a/b/c/d/e/file1|/vol1/buck1/a/b/c/d/e/file1/|
| *{color:#ff8b00} Proposed data format -table Dir/File entry: "parentID/Key"{color}*|
|{color:#57d9a3} *Bucket ObjectID - 512, which is the parent for the first level dirs/files*{color}|
|{color:#0747a6}*PrefixTable - Data Format: 'parentID/Key'*{color}|{color:#0747a6}*KeyTable - Data Format: 'parentID/Key'*{color}|
| *PrefixName*| *ObjectID*| *KeyName*| *ObjectID*|
|512/a|1025|1028/file1|1030|
|1025/b|1026|1028/file2|1031|
|1026/c|1027| | |
|1027/d|1028| | |
|1028/e|1029| | |
|1029/file1|1032| | |
> Ozone FS namespace
> ------------------
>
> Key: HDDS-2939
> URL: https://issues.apache.org/jira/browse/HDDS-2939
> Project: Hadoop Distributed Data Store
> Issue Type: New Feature
> Components: Ozone Manager
> Reporter: Supratim Deka
> Assignee: Rakesh Radhakrishnan
> Priority: Major
> Labels: Triaged
> Attachments: Ozone FS Namespace Proposal v1.0.docx
>
>
> Create the structures and metadata layout required to support efficient FS namespace operations in Ozone - operations involving folders/directories required to support the Hadoop compatible Filesystem interface.
> The details are described in the attached document. The work is divided up into sub-tasks as per the task list in the document.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org