[jira] [Created] (HDFS-15292) Infinite loop in Lease Manager due to replica is missing in dn

["Aaron Guo (Jira)" <ji...@apache.org>]

Aaron Guo created HDFS-15292:
--------------------------------

             Summary: Infinite loop in Lease Manager due to replica is missing in dn
                 Key: HDFS-15292
                 URL: https://issues.apache.org/jira/browse/HDFS-15292
             Project: Hadoop HDFS
          Issue Type: Bug
          Components: namenode
    Affects Versions: 3.1.3
            Reporter: Aaron Guo


In our production environment, we found that files of under construction keep growing, and the lease manager is trying to release the lease in a Infinite loop:
{code:java}
2020-04-18 23:10:57,816 WARN  namenode.LeaseManager (LeaseManager.java:checkLeases(589)) - Cannot release the path /user/hadoop/myTestFile.txt in the lease [Lease.  Holder: go-hdfs-7VVGF3sGvHZcsZZC, pending creates: 1]. It will be retried.
org.apache.hadoop.hdfs.protocol.AlreadyBeingCreatedException: DIR* NameSystem.internalReleaseLease: Failed to release lease for file /user/hadoop/myTestFile.txt. Committed blocks are waiting to be minimally replicated. Try again later.
        at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.internalReleaseLease(FSNamesystem.java:3391)
        at org.apache.hadoop.hdfs.server.namenode.LeaseManager.checkLeases(LeaseManager.java:586)
        at org.apache.hadoop.hdfs.server.namenode.LeaseManager$Monitor.run(LeaseManager.java:524)
        at java.lang.Thread.run(Thread.java:745)
{code}
 This is because the last block of this file can NOT meet the minimum required replica of 1, a  AlreadyBeingCreatedException get thrown, and it will keeps retry forever.

This infinite loop also cause another issue since the lease manager always trying to release the first lease then goto the next one, so no lease will be released.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-help@hadoop.apache.org