[jira] [Comment Edited] (HDDS-2939) Ozone FS namespace

["Yiqun Lin (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/HDDS-2939?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17025842#comment-17025842 ] 

Yiqun Lin edited comment on HDDS-2939 at 1/29/20 12:46 PM:
-----------------------------------------------------------

Hi [~sdeka], I am reading for this design doc, some comments from me:

For the Filesystem Namespace Operations, the ls(list files/folders) operation will also a common operation. But under current implementation, for example, list a directory, we have to traverse the whole directory/file table to lookup the child file/sub-folders.This is an ineffective way. I know the <Prefix, ID> lookup way can greatly reduce the memory used. but this is not friendly for the ls operation.

Do we have any other improvement for this? Can we additionally store the child ID for each record in directory table? That can help us quickly find the child file or child folder.

 
{quote}Associating a lock with each parent prefix being accessed by an operation in the OM, is sufficient to control concurrent operations on the same prefix. When the OM starts to process create “/a/b/c/1.txt”, a prefix lock is taken for “/a/b/c”...
{quote}
For the concurrency control, we create the lock for each parent prefix level. There will be large number of lock instances to be maintained in OM memory once there are millions of directory folders. Current way is so fine-grained lock way, have we considered about the partition namespace way? Divided the whole namespace into logic sub-namespaces by the prefix key. Then each sub-namespace will have its lock. This is a compromise approach than just having a global exclusive lock or having uncontrollable number of locks that depended on parent prefix's number.

Is there a future plan to have a way(API or command Tool) to convert object key to Ozone FS namespace? Because object store is now the major use case for the users. Maybe users want to use a filesystem way to access the data without moving their data.


was (Author: linyiqun):
Hi [~sdeka], I am reading for this design doc, some comments from me:

For the Filesystem Namespace Operations, the ls(list files/folders) operation will also a common operation. But under current implementation, for example, list a directory, we have to traverse the whole directory/file table to lookup the child file/sub-folders.This is an ineffective way. Do we have any other improvement for this? Can we additionally store the child ID for each record in directory table? That can help us quickly find the child file or child folder.

{quote}
Associating a lock with each parent prefix being accessed by an operation in the OM, is sufficient to control concurrent operations on the same prefix. When the OM starts to process create “/a/b/c/1.txt”, a prefix lock is taken for “/a/b/c”...
{quote}

For the concurrency control, we create the lock for each parent prefix level. There will be large number of lock instances to be maintained in OM memory once there are millions of directory folders. Current way is so fine-grained lock way, have we considered about the partition namespace way? Divided the whole namespace into logic sub-namespaces by the prefix key. Then each sub-namespace will have its lock. This is a compromise approach than just having a global exclusive lock or having uncontrollable number of locks that depended on parent prefix's number.

Is there a future plan to have a way(API or command Tool) to convert object key to Ozone FS namespace? Because object store is now  the major use case for the users. Maybe users want to use a filesystem way to access the data without moving their data.



> Ozone FS namespace
> ------------------
>
>                 Key: HDDS-2939
>                 URL: https://issues.apache.org/jira/browse/HDDS-2939
>             Project: Hadoop Distributed Data Store
>          Issue Type: Improvement
>          Components: Ozone Manager
>            Reporter: Supratim Deka
>            Assignee: Supratim Deka
>            Priority: Major
>         Attachments: Ozone FS Namespace Proposal v1.0.docx
>
>
> Create the structures and metadata layout required to support efficient FS namespace operations in Ozone - operations involving folders/directories required to support the Hadoop compatible Filesystem interface.
> The details are described in the attached document. The work is divided up into sub-tasks as per the task list in the document.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org