[jira] [Issue Comment Deleted] (HDDS-1302) Fix SCM CLI does not list container with id 1

["Nanda kumar (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/HDDS-1302?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Nanda kumar updated HDDS-1302:
------------------------------
    Comment: was deleted

(was: All the list calls (like list directories, files) in OzoneFS depends on list key call of OzoneClient. 
OzoneClient internally uses {{OzoneBucket#KeyIterator}} to iterate over the list of keys in a bucket. This iterator has an internal cache whose size can be configured using {{ozone.client.list.cache}} (default value is 1000). So after every 1000 keys, we make a new RPC call to OM to get the next set of keys (Iterator's internal logic).

Below is the call which the iterator uses:
{code:java|title=ClientProtocol.java}
  /**
   * Returns list of Keys in {Volume/Bucket} that matches the keyPrefix,
   * size of the returned list depends on maxListResult. The caller has
   * to make multiple calls to read all keys.
   * @param volumeName Name of the Volume
   * @param bucketName Name of the Bucket
   * @param keyPrefix Bucket prefix to match
   * @param prevKey Starting point of the list, this key is excluded
   * @param maxListResult Max number of buckets to return.
   * @return {@code List<OzoneKey>}
   * @throws IOException
   */
  List<OzoneKey> listKeys(String volumeName, String bucketName,
                          String keyPrefix, String prevKey, int maxListResult)
      throws IOException;
{code}
As mentioned, the {{prevKey}} should be excluded in the result returned by OM, else the iterator will get a duplicate value at the end of every 1000 key or the value configured in {{ozone.client.list.cache}}.
 * So if we do a list file operation on a directory which has more than 1000 (default value) files/directories, we will get some duplicates in the result.
 * If we count the number of files in a directory, the result will be incorrect.

Note: Don't get confused by the name {{startId}} in OM code and {{prevKey}} in client code, they are one and the same.)

> Fix SCM CLI does not list container with id 1
> ---------------------------------------------
>
>                 Key: HDDS-1302
>                 URL: https://issues.apache.org/jira/browse/HDDS-1302
>             Project: Hadoop Distributed Data Store
>          Issue Type: Bug
>            Reporter: Bharat Viswanadham
>            Assignee: Vivek Ratnavel Subramanian
>            Priority: Minor
>
> In HDDS-1263 it is changed to handle the list containers with containerID 1 by changing the actual logic of listContainers in ScmContainerManager.java. But now with this change, it is contradicting with the javadoc.
> From [~nandakumar131] comments
> https://issues.apache.org/jira/browse/HDDS-1263?focusedCommentId=16794865&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-16794865
>  
> I agree this will be the way to fix it.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-help@hadoop.apache.org