You are viewing a plain text version of this content. The canonical link for it is here.
Posted to hdfs-dev@hadoop.apache.org by "Szymon Orzechowski (Jira)" <ji...@apache.org> on 2024/06/07 12:10:00 UTC
[jira] [Created] (HDFS-17548) excessive NO_REQUIRED_STORAGE_TYPE messages
Szymon Orzechowski created HDFS-17548:
-----------------------------------------
Summary: excessive NO_REQUIRED_STORAGE_TYPE messages
Key: HDFS-17548
URL: https://issues.apache.org/jira/browse/HDFS-17548
Project: Hadoop HDFS
Issue Type: Bug
Components: namenode
Affects Versions: 3.3.4
Reporter: Szymon Orzechowski
Notification of unavailable storageType has been implemented in HDFS-15815.
Yesterday we noted a failure on our production cluster. As a side result of analyzing the reasons for the failure, we found additional error messages:
nn-3.wphadoop.dc-2.jumbo.xxxx_hadoop-hdfs-namenode.log.out:2024-06-07 00:35:23,381 INFO org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicy: Not enough replicas was chosen. Reason: \{NO_REQUIRED_STORAGE_TYPE=1}
These tell us very little and seem to make absolutely no sense in the case of our cluster (12 racks, no storage policies enabled nor storage types defined).
However, in 100% of cases they occur directly (or almost directly) after messages like:
nn-3.wphadoop.dc-2.jumbo.xxxx_hadoop-hdfs-namenode.log.out-2024-06-07 00:35:23,380 INFO org.apache.hadoop.ipc.Server: IPC Server handler 25 on default port 8020, call#9866 Retry#0 org.apache.hadoop.hdfs.protocol.ClientProtocol. create from 10.32.20.25:35130: org.apache.hadoop.hdfs.protocol.FSLimitException$MaxDirectoryItemsExceededException: The directory item limit of /user/gobblin/loghost/failures/dot_ma/undefined is exceeded: limit=1048576 items=1048576
Which leads me to the conclusion that in this case the NO_REQUIRED_STORAGE_TYPE errors are raised due to reaching the limit specified in property dfs.namenode.fs-limits.max-directory-items. Perhaps they should be restricted as they provide no information and actually report a non-existent problem.
Additionally, immediately after clearing the /user/gobblin/loghost/failures/dot_ma/undefined directory, the NO_REQUIRED_STORAGE_TYPE messages stopped appearing.
---
I would also like to take this opportunity to ask where to find any list, specifying meaning of values used in the NO_REQUIRED_STORAGE_TYPE=1 messages (in this case, 1)
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-help@hadoop.apache.org