[jira] [Created] (HDFS-17219) Inconsistent count results when upgrading hdfs clusetr from 2.10.2 to 3.3.6

["Ke Han (Jira)" <ji...@apache.org>]

Ke Han created HDFS-17219:
-----------------------------

             Summary: Inconsistent count results when upgrading hdfs clusetr from 2.10.2 to 3.3.6
                 Key: HDFS-17219
                 URL: https://issues.apache.org/jira/browse/HDFS-17219
             Project: Hadoop HDFS
          Issue Type: Bug
          Components: namenode
    Affects Versions: 3.3.6, 2.10.2
            Reporter: Ke Han
         Attachments: hpLjvJVW.tar.gz

When upgrading hdfs cluster from 2.10.2 to 3.3.6, the results returned from *dfs count* command is inconsistent.
h1. Reproduce

Start up 2.10.2 hdfs cluster (1 NN, 2 DN, 1 SNN), execute the following commands

 
{code:java}
dfs -mkdir /GscWZRxS
dfs -put -f  -d /tmp/hpLjvJVW/cl /GscWZRxS/
dfs -put -f  -d /tmp/hpLjvJVW/Zjpk /GscWZRxS/cl/lBsmFBlyBd/pozIeNFjzd/PsLbgpR
dfsadmin -clrQuota /GscWZRxS/cl
dfs -expunge -immediate
dfsadmin -refreshSuperUserGroupsConfiguration
dfs -mkdir /GscWZRxS/cl/lBsmFBlyBd/pozIeNFjzd/PsLbgpR/Zjpk/Cf/mGpVA
dfsadmin -refreshCallQueue
dfsadmin -saveNamespace -beforeShutdown
dfsadmin -refreshServiceAcl
dfsadmin -saveNamespace
dfsadmin -clrQuota /GscWZRxS/cl/lBsmFBlyBd/pozIeNFjzd
dfsadmin -setSpaceQuota 2 -storageType DISK /GscWZRxS/cl/lBsmFBlyBd/pozIeNFjzd/PsLbgpR/Zjpk/Cf
dfsadmin -refreshNodes
dfsadmin -setSpaceQuota 2 -storageType DISK /GscWZRxS/cl/lBsmFBlyBd/pozIeNFjzd
dfsadmin -clrSpaceQuota -storageType ARCHIVE /GscWZRxS/cl
dfsadmin -restoreFailedStorage true
dfs -chmod 22 2 /GscWZRxS/cl/lBsmFBlyBd/pozIeNFjzd/PsLbgpR/Zjpk/vGVnspAl/ih
dfs -chmod 22 2 /GscWZRxS/cl/lBsmFBlyBd/pozIeNFjzd/PsLbgpR/Zjpk/pqd
dfs -createSnapshot /GscWZRxS/cl mGpVA {code}
 

Now, before upgrade, I check the quota results

 
{code:java}
bin/hdfs dfs -count -q -h -u /GscWZRxS/cl/lBsmFBlyBd/pozIeNFjzd/PsLbgpR/Zjpk/Cf 
none             inf            none             inf /GscWZRxS/cl/lBsmFBlyBd/pozIeNFjzd/PsLbgpR/Zjpk/Cf {code}
Then prepare the upgrade: enter safemode, create image, shutdown the cluster and start up the new cluster
{code:java}
bin/hdfs dfs -count -q -h -u /GscWZRxS/cl/lBsmFBlyBd/pozIeNFjzd/PsLbgpR/Zjpk/Cf 
8.0 E           8.0 E            none             inf /GscWZRxS/cl/lBsmFBlyBd/pozIeNFjzd/PsLbgpR/Zjpk/Cf {code}
The values of the first two columns are inconsistent with the quota I set before.

 

I have attached the file used by the command. I am digging out the root cause, I'll try to submit a patch once I can fix it. Any help is appreciated!

 



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-help@hadoop.apache.org