[jira] [Created] (HDFS-17105) mistakenly purge editLogs even after it is empty in NNStorageRetentionManager

["ConfX (Jira)" <ji...@apache.org>]

ConfX created HDFS-17105:
----------------------------

             Summary:  mistakenly purge editLogs even after it is empty in NNStorageRetentionManager
                 Key: HDFS-17105
                 URL: https://issues.apache.org/jira/browse/HDFS-17105
             Project: Hadoop HDFS
          Issue Type: Bug
            Reporter: ConfX
         Attachments: reproduce.sh

h2. What happened:

Got {{IndexOutOfBoundsException}} after setting {{dfs.namenode.max.extra.edits.segments.retained}} to a negative value and purging old record with {{{}NNStorageRetentionManager{}}}.
h2. Where's the bug:

In line 156 of {{{}NNStorageRetentionManager{}}}, the manager trims {{editLogs}} until it is under the {{{}maxExtraEditsSegmentsToRetain{}}}:
{noformat}
while (editLogs.size() > maxExtraEditsSegmentsToRetain) {
      purgeLogsFrom = editLogs.get(0).getLastTxId() + 1;
      editLogs.remove(0);
}{noformat}
However, if {{dfs.namenode.max.extra.edits.segments.retained}} is set to below 0 the size of {{editLogs}} would never be below, resulting in ultimately {{editLog.size()=0}} and thus {{editLogs.get(0)}} is out of range.
h2. How to reproduce:

(1) Set {{dfs.namenode.max.extra.edits.segments.retained}} to -1974676133
(2) Run test: {{org.apache.hadoop.hdfs.server.namenode.TestNNStorageRetentionManager#testNoLogs}}
h2. Stacktrace:
{noformat}
java.lang.IndexOutOfBoundsException: Index 0 out of bounds for length 0
    at java.base/jdk.internal.util.Preconditions.outOfBounds(Preconditions.java:64)
    at java.base/jdk.internal.util.Preconditions.outOfBoundsCheckIndex(Preconditions.java:70)
    at java.base/jdk.internal.util.Preconditions.checkIndex(Preconditions.java:248)
    at java.base/java.util.Objects.checkIndex(Objects.java:372)
    at java.base/java.util.ArrayList.get(ArrayList.java:459)
    at org.apache.hadoop.hdfs.server.namenode.NNStorageRetentionManager.purgeOldStorage(NNStorageRetentionManager.java:157)
    at org.apache.hadoop.hdfs.server.namenode.TestNNStorageRetentionManager.runTest(TestNNStorageRetentionManager.java:299)
    at org.apache.hadoop.hdfs.server.namenode.TestNNStorageRetentionManager.testNoLogs(TestNNStorageRetentionManager.java:143){noformat}
For an easy reproduction, run the reproduce.sh in the attachment.

We are happy to provide a patch if this issue is confirmed.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-help@hadoop.apache.org