You are viewing a plain text version of this content. The canonical link for it is here.

Posted to hdfs-dev@hadoop.apache.org by "Fengnan Li (Jira)" <ji...@apache.org> on 2020/02/27 23:54:00 UTC

[jira] [Created] (HDFS-15196) RouterRpcServer getListing cannot list large dirs correctly

Fengnan Li created HDFS-15196:
---------------------------------

             Summary: RouterRpcServer getListing cannot list large dirs correctly
                 Key: HDFS-15196
                 URL: https://issues.apache.org/jira/browse/HDFS-15196
             Project: Hadoop HDFS
          Issue Type: Bug
            Reporter: Fengnan Li
            Assignee: Fengnan Li


In RouterRpcServer, getListing function is handled as two parts:
 # Union all partial listings from destination ns + paths
 # Append mount points for the dir to be listed

In the case of large dir which is bigger than DFSConfigKeys.DFS_LIST_LIMIT (with default value 1k), the batch listing will be used and the startAfter will be used to define the boundary of each batch listing. However, step 2 here will add existing mount points, which will mess up with the boundary of the batch, thus making the next batch startAfter wrong.

The fix is just to append the mount points when there is no more batch query necessary.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-help@hadoop.apache.org