[jira] [Updated] (HDDS-2466) Split OM Key into a Prefix Part and a Name Part

["Supratim Deka (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/HDDS-2466?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Supratim Deka updated HDDS-2466:
--------------------------------
    Description: 
This is a 
OM stores every key in a Key table, which maps the key to a KeyInfo.

FS semantics require unique names. Specifically - inside any given directory (the prefix of any OM object key), each object must have a unique name. This means that a sub-directory and a file which share the same prefix, cannot have the same name.
if there exists an object(file) "/a/b/c/d/e/file" in OM, we cannot allow a new object(file/dir) "/a/b/c" to be created. it is implicit as per FS semantics that /a/b/c is the name of an existing directory.

To enable this behaviour, we propose to store every valid prefix in a separate table in the OM, alongside the Key Table. The File create request handler can iterate this KeyPrefix table to check if the name already exists. Specific optimisations will be covered as separate jira items.
 
The following 2 sets of tasks are required to complete the key prefix infrastructure - base infra followed by extension to achieve prefix compression(in-memory + on-disk) for the OM Key table.

Base Infra:
1. Create prefix entry in KeyPrefix OM Table for every new key inserted in the OM Key Table - include file create, directory create and rename. Change File create request handler to check the KeyPrefix table, instead of iterating the entire Key Table.

2. Handle stale prefix entries in the KeyPrefix table - handle the situation where the last object associated with a prefix gets deleted. This task involves defining the semantics in this situation and subsequently the design for detecting stale entries. 

 Extension to support prefix compression in Key Table:
3. Assign unique ID to each prefix in the KeyPrefix table - include synchronisation.

4. Node restart - ensure the unique ID generation scheme handles restart.

5. Change the key in OM Key Table to <Prefix ID>/<name> instead of the full name. 


  was:
OM stores every key in a key table, which maps the key to a KeyInfo.

If we split the key into a prefix and a name part which are then stored in separate tables, serves 2 purposes:
1. OzoneFS operations can be made efficient by deriving a prefix tree representation of the pathnames(prefixes) - details of this are outside the current scope. Also, the prefix table can get preferential treatment when it comes to caching.
2. PutKey is not penalised by having to parse the key into each path component - this is for cases where the dataset is a pure object store. Splitting into a prefix and a name is the minimal work to be done inline during the putKey operation.



> Split OM Key into a Prefix Part and a Name Part
> -----------------------------------------------
>
>                 Key: HDDS-2466
>                 URL: https://issues.apache.org/jira/browse/HDDS-2466
>             Project: Hadoop Distributed Data Store
>          Issue Type: Improvement
>          Components: Ozone Manager
>            Reporter: Supratim Deka
>            Assignee: Supratim Deka
>            Priority: Major
>
> This is a 
> OM stores every key in a Key table, which maps the key to a KeyInfo.
> FS semantics require unique names. Specifically - inside any given directory (the prefix of any OM object key), each object must have a unique name. This means that a sub-directory and a file which share the same prefix, cannot have the same name.
> if there exists an object(file) "/a/b/c/d/e/file" in OM, we cannot allow a new object(file/dir) "/a/b/c" to be created. it is implicit as per FS semantics that /a/b/c is the name of an existing directory.
> To enable this behaviour, we propose to store every valid prefix in a separate table in the OM, alongside the Key Table. The File create request handler can iterate this KeyPrefix table to check if the name already exists. Specific optimisations will be covered as separate jira items.
>  
> The following 2 sets of tasks are required to complete the key prefix infrastructure - base infra followed by extension to achieve prefix compression(in-memory + on-disk) for the OM Key table.
> Base Infra:
> 1. Create prefix entry in KeyPrefix OM Table for every new key inserted in the OM Key Table - include file create, directory create and rename. Change File create request handler to check the KeyPrefix table, instead of iterating the entire Key Table.
> 2. Handle stale prefix entries in the KeyPrefix table - handle the situation where the last object associated with a prefix gets deleted. This task involves defining the semantics in this situation and subsequently the design for detecting stale entries. 
>  Extension to support prefix compression in Key Table:
> 3. Assign unique ID to each prefix in the KeyPrefix table - include synchronisation.
> 4. Node restart - ensure the unique ID generation scheme handles restart.
> 5. Change the key in OM Key Table to <Prefix ID>/<name> instead of the full name. 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org