You are viewing a plain text version of this content. The canonical link for it is here.
Posted to hdfs-issues@hadoop.apache.org by "Chengbing Liu (Jira)" <ji...@apache.org> on 2022/12/19 12:23:00 UTC
[jira] [Created] (HDFS-16872) Fix log throttling by declaring LogThrottlingHelper as static members
Chengbing Liu created HDFS-16872:
------------------------------------
Summary: Fix log throttling by declaring LogThrottlingHelper as static members
Key: HDFS-16872
URL: https://issues.apache.org/jira/browse/HDFS-16872
Project: Hadoop HDFS
Issue Type: Bug
Affects Versions: 3.3.4
Reporter: Chengbing Liu
In our production cluster with Observer NameNode enabled, we have plenty of logs printed by {{FSEditLogLoader}} and {{RedundantEditLogInputStream}}. The {{LogThrottlingHelper}} doesn't seem to work.
{noformat}
2022-10-25 09:26:50,380 INFO org.apache.hadoop.hdfs.server.namenode.FSImage: Start loading edits file ByteStringEditLog[17686250688, 17686250688], ByteStringEditLog[17686250688, 17686250688], ByteStringEditLog[17686250688, 17686250688] maxTxnsToRead = 9223372036854775807
2022-10-25 09:26:50,380 INFO org.apache.hadoop.hdfs.server.namenode.RedundantEditLogInputStream: Fast-forwarding stream 'ByteStringEditLog[17686250688, 17686250688], ByteStringEditLog[17686250688, 17686250688], ByteStringEditLog[17686250688, 17686250688]' to transaction ID 17686250688
2022-10-25 09:26:50,380 INFO org.apache.hadoop.hdfs.server.namenode.RedundantEditLogInputStream: Fast-forwarding stream 'ByteStringEditLog[17686250688, 17686250688]' to transaction ID 17686250688
2022-10-25 09:26:50,380 INFO org.apache.hadoop.hdfs.server.namenode.FSImage: Loaded 1 edits file(s) (the last named ByteStringEditLog[17686250688, 17686250688], ByteStringEditLog[17686250688, 17686250688], ByteStringEditLog[17686250688, 17686250688]) of total size 527.0, total edits 1.0, total load time 0.0 ms
2022-10-25 09:26:50,387 INFO org.apache.hadoop.hdfs.server.namenode.FSImage: Start loading edits file ByteStringEditLog[17686250689, 17686250693], ByteStringEditLog[17686250689, 17686250693], ByteStringEditLog[17686250689, 17686250693] maxTxnsToRead = 9223372036854775807
2022-10-25 09:26:50,387 INFO org.apache.hadoop.hdfs.server.namenode.RedundantEditLogInputStream: Fast-forwarding stream 'ByteStringEditLog[17686250689, 17686250693], ByteStringEditLog[17686250689, 17686250693], ByteStringEditLog[17686250689, 17686250693]' to transaction ID 17686250689
2022-10-25 09:26:50,387 INFO org.apache.hadoop.hdfs.server.namenode.RedundantEditLogInputStream: Fast-forwarding stream 'ByteStringEditLog[17686250689, 17686250693]' to transaction ID 17686250689
2022-10-25 09:26:50,387 INFO org.apache.hadoop.hdfs.server.namenode.FSImage: Loaded 1 edits file(s) (the last named ByteStringEditLog[17686250689, 17686250693], ByteStringEditLog[17686250689, 17686250693], ByteStringEditLog[17686250689, 17686250693]) of total size 890.0, total edits 5.0, total load time 1.0 ms
{noformat}
After some digging, I found the cause is that {{LogThrottlingHelper}}'s are declared as instance variables of all the enclosing classes, including {{FSImage}}, {{FSEditLogLoader}} and {{RedundantEditLogInputStream}}. Therefore the logging frequency will not be limited across different instances. For classes with only limited number of instances, such as {{FSImage}}, this is fine. For others whose instances are created frequently, such as {{FSEditLogLoader}} and {{RedundantEditLogInputStream}}, it will result in plenty of logs.
This can be fixed by declaring {{LogThrottlingHelper}}'s as static members.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-help@hadoop.apache.org