You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ozone.apache.org by "Devesh Kumar Singh (Jira)" <ji...@apache.org> on 2024/04/02 08:53:00 UTC

[jira] [Updated] (HDDS-10629) Recon - Enriched API for listing of FSO bucket keys with recursive support

     [ https://issues.apache.org/jira/browse/HDDS-10629?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Devesh Kumar Singh updated HDDS-10629:
--------------------------------------
    Description: 
* API to retrieve list of keys in the ozone. Probably allow filtering of keys by bucket, replication_type, age, size, limit and key prefix.

 * 
h5. This API will leverage the existing design and implementation of NamespaceSummary data built by Recon. Existing NSSummary data is missing a few data fields and will need to be extended while implementing the API. 

 # 
h5. By default, API will provide a list of keys based on the path provided. E.g. /volume/bucket/

 # 
h5. API will also accept parameters to filter the list of keys by path (volume/bucket/dir/sub-dir), replication type, age, key size, start, count and start prefix of key.

 # 
h5. Default values of API parameters if not provided:

 # 
h5. path - /

 # 
h5. recursive - true (Applicable only for FSO buckets)

 # 
h5. replication_type - RATIS

 # 
h5. age - null and filter will not be applied, so list out keys irrespective of age.

 # 
h5. key_size - 0 bytes, which means all keys greater than zero bytes will be listed, effectively all.

 # 
h5. start - 1 (inclusive)

 # 
h5. count - 1000

 # 
h5. Behavior of API:

h5.  
 # 
h5. For FSO bucket - list out *_count_* number of keys starting from *_start_* value number of keys recursively, i.e. on immediate path as well as all files in all directories and sub-directories on the provided path.

This API will implement pagination support using *start* and *count  params.*

 
h3. Get List of All Keys:

**
 * GET /api/v1/keys/list

 * API params:

 * 
h5. path

 * 
h5. replication_type

 * 
h5. age in epoch ms

 * 
h5. start

 * 
h5. count

 * 
h5. key_size

 * Input Request:

**

api/v1/namespace/listKeys?path=/volume1/fso-bucket/dir1/&start=5&count=105

**
 * Output Response

**
 
{noformat}
{ "status": "OK", "path": "/volume1/fso-bucket/dir1", "size": 62914560, "sizeWithReplica": -1, "subPathCount": 3, "subPaths": [ { "path": "/volume1/fso-bucket/dir1/dir2", "size": 41943040, "sizeWithReplica": -1, "isKey": false, "subPathCount": 3, "subPaths": [ { "key": true, "path": "/volume1/fso-bucket/dir1/dir2/file1", "size": 41943040, "sizeWithReplica": -1, "isKey": true, "subPathCount": 0 }, { "key": true, "path": "/volume1/fso-bucket/dir1/dir2/file2", "size": 41943040, "sizeWithReplica": -1, "isKey": true, "subPathCount": 0 }, { "key": true, "path": "/volume1/fso-bucket/dir1/dir2/file3", "size": 41943040, "sizeWithReplica": -1, "isKey": true, "subPathCount": 0 } ],          "replicaType": "RATIS" }, { "key": true, "path": "/volume1/fso-bucket/dir1/file1", "size": 10485760, "sizeWithReplica": -1, "isKey": true, "replicaType": "EC" }, { "key": true, "path": "/volume1/fso-bucket/dir1/testfile", "size": 10485760, "sizeWithReplica": -1, "isKey": true, "replicaType": "RATIS" } ], "sizeDirectKey": 20971520 }{noformat}
 

  was:
* API to retrieve list of keys in the ozone. Probably allow filtering of keys by bucket, replication_type, age, size, limit and key prefix.

 * 
h5. This API will leverage the existing design and implementation of NamespaceSummary data built by Recon. Existing NSSummary data is missing a few data fields and will need to be extended while implementing the API. 

 # 
h5. By default, API will provide a list of keys based on the path provided. E.g. /volume/bucket/

 # 
h5. API will also accept parameters to filter the list of keys by path (volume/bucket/dir/sub-dir), replication type, age, key size, start, count and start prefix of key.

 # 
h5. Default values of API parameters if not provided:

 # 
h5. path - /

 # 
h5. recursive - true (Applicable only for FSO buckets)

 # 
h5. replication_type - RATIS

 # 
h5. age - null and filter will not be applied, so list out keys irrespective of age.

 # 
h5. key_size - 0 bytes, which means all keys greater than zero bytes will be listed, effectively all.

 # 
h5. start - 1 (inclusive)

 # 
h5. count - 1000

 # 
h5. Behavior of API:

 # 
h5. For OBS bucket - list out *_count_* number of keys starting from *_start_* value, existing in a bucket based on further filter criteria. 

 # 
h5. For FSO bucket - list out *_count_* number of keys starting from *_start_* value number of keys recursively, i.e. on immediate path as well as all files in all directories and sub-directories on the provided path.

 # 
h5. For LEGACY bucket -

 # FS path enabled - same like FSO bucket behavior.
 # FS Path not enabled - same like OBS bucket behavior.

This API will implement pagination support using *start* and *count  params.*

 
h3. Get List of All Keys:

**
 * GET /api/v1/keys/list

 * API params:

 * 
h5. path

 * 
h5. replication_type

 * 
h5. age in epoch ms

 * 
h5. start

 * 
h5. count

 * 
h5. key_size

 * Input Request:

**

api/v1/namespace/listKeys?path=/volume1/fso-bucket/dir1/&start=5&count=105

**
 * Output Response

**
 
{noformat}
{ "status": "OK", "path": "/volume1/fso-bucket/dir1", "size": 62914560, "sizeWithReplica": -1, "subPathCount": 3, "subPaths": [ { "path": "/volume1/fso-bucket/dir1/dir2", "size": 41943040, "sizeWithReplica": -1, "isKey": false, "subPathCount": 3, "subPaths": [ { "key": true, "path": "/volume1/fso-bucket/dir1/dir2/file1", "size": 41943040, "sizeWithReplica": -1, "isKey": true, "subPathCount": 0 }, { "key": true, "path": "/volume1/fso-bucket/dir1/dir2/file2", "size": 41943040, "sizeWithReplica": -1, "isKey": true, "subPathCount": 0 }, { "key": true, "path": "/volume1/fso-bucket/dir1/dir2/file3", "size": 41943040, "sizeWithReplica": -1, "isKey": true, "subPathCount": 0 } ],          "replicaType": "RATIS" }, { "key": true, "path": "/volume1/fso-bucket/dir1/file1", "size": 10485760, "sizeWithReplica": -1, "isKey": true, "replicaType": "EC" }, { "key": true, "path": "/volume1/fso-bucket/dir1/testfile", "size": 10485760, "sizeWithReplica": -1, "isKey": true, "replicaType": "RATIS" } ], "sizeDirectKey": 20971520 }{noformat}
 


> Recon - Enriched API for listing of FSO bucket keys with recursive support
> --------------------------------------------------------------------------
>
>                 Key: HDDS-10629
>                 URL: https://issues.apache.org/jira/browse/HDDS-10629
>             Project: Apache Ozone
>          Issue Type: Task
>          Components: Ozone Recon
>    Affects Versions: 1.5.0
>            Reporter: Devesh Kumar Singh
>            Assignee: Devesh Kumar Singh
>            Priority: Major
>
> * API to retrieve list of keys in the ozone. Probably allow filtering of keys by bucket, replication_type, age, size, limit and key prefix.
>  * 
> h5. This API will leverage the existing design and implementation of NamespaceSummary data built by Recon. Existing NSSummary data is missing a few data fields and will need to be extended while implementing the API. 
>  # 
> h5. By default, API will provide a list of keys based on the path provided. E.g. /volume/bucket/
>  # 
> h5. API will also accept parameters to filter the list of keys by path (volume/bucket/dir/sub-dir), replication type, age, key size, start, count and start prefix of key.
>  # 
> h5. Default values of API parameters if not provided:
>  # 
> h5. path - /
>  # 
> h5. recursive - true (Applicable only for FSO buckets)
>  # 
> h5. replication_type - RATIS
>  # 
> h5. age - null and filter will not be applied, so list out keys irrespective of age.
>  # 
> h5. key_size - 0 bytes, which means all keys greater than zero bytes will be listed, effectively all.
>  # 
> h5. start - 1 (inclusive)
>  # 
> h5. count - 1000
>  # 
> h5. Behavior of API:
> h5.  
>  # 
> h5. For FSO bucket - list out *_count_* number of keys starting from *_start_* value number of keys recursively, i.e. on immediate path as well as all files in all directories and sub-directories on the provided path.
> This API will implement pagination support using *start* and *count  params.*
>  
> h3. Get List of All Keys:
> **
>  * GET /api/v1/keys/list
>  * API params:
>  * 
> h5. path
>  * 
> h5. replication_type
>  * 
> h5. age in epoch ms
>  * 
> h5. start
>  * 
> h5. count
>  * 
> h5. key_size
>  * Input Request:
> **
> api/v1/namespace/listKeys?path=/volume1/fso-bucket/dir1/&start=5&count=105
> **
>  * Output Response
> **
>  
> {noformat}
> { "status": "OK", "path": "/volume1/fso-bucket/dir1", "size": 62914560, "sizeWithReplica": -1, "subPathCount": 3, "subPaths": [ { "path": "/volume1/fso-bucket/dir1/dir2", "size": 41943040, "sizeWithReplica": -1, "isKey": false, "subPathCount": 3, "subPaths": [ { "key": true, "path": "/volume1/fso-bucket/dir1/dir2/file1", "size": 41943040, "sizeWithReplica": -1, "isKey": true, "subPathCount": 0 }, { "key": true, "path": "/volume1/fso-bucket/dir1/dir2/file2", "size": 41943040, "sizeWithReplica": -1, "isKey": true, "subPathCount": 0 }, { "key": true, "path": "/volume1/fso-bucket/dir1/dir2/file3", "size": 41943040, "sizeWithReplica": -1, "isKey": true, "subPathCount": 0 } ],          "replicaType": "RATIS" }, { "key": true, "path": "/volume1/fso-bucket/dir1/file1", "size": 10485760, "sizeWithReplica": -1, "isKey": true, "replicaType": "EC" }, { "key": true, "path": "/volume1/fso-bucket/dir1/testfile", "size": 10485760, "sizeWithReplica": -1, "isKey": true, "replicaType": "RATIS" } ], "sizeDirectKey": 20971520 }{noformat}
>  



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org