You are viewing a plain text version of this content. The canonical link for it is here.
Posted to hdfs-issues@hadoop.apache.org by "ASF GitHub Bot (Jira)" <ji...@apache.org> on 2023/09/08 12:01:00 UTC
[jira] [Updated] (HDFS-17098) DatanodeManager does not handle null storage type properly
[ https://issues.apache.org/jira/browse/HDFS-17098?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
ASF GitHub Bot updated HDFS-17098:
----------------------------------
Labels: pull-request-available (was: )
> DatanodeManager does not handle null storage type properly
> ----------------------------------------------------------
>
> Key: HDFS-17098
> URL: https://issues.apache.org/jira/browse/HDFS-17098
> Project: Hadoop HDFS
> Issue Type: Bug
> Reporter: ConfX
> Priority: Critical
> Labels: pull-request-available
> Attachments: reproduce.sh
>
>
> h2. What happened:
> Got a {{NullPointerException}} without message when sorting datanodes in {{{}NetworkTopology{}}}.
> h2. Where's the bug:
> In line 654 of {{{}DatanodeManager{}}}, the manager creates a second sorter using the standard {{Comparator}} class:
> {noformat}
> Comparator<DatanodeInfoWithStorage> comp =
> Comparator.comparing(DatanodeInfoWithStorage::getStorageType);
> secondarySort = list -> Collections.sort(list, comp);{noformat}
> This comparator is then used in {{NetworkTopology}} as a secondary sort to break ties:
> {noformat}
> if (secondarySort != null) {
> // a secondary sort breaks the tie between nodes.
> secondarySort.accept(nodesList);
> }{noformat}
> However, if the storage type is {{{}null{}}}, a {{NullPointerException}} would be thrown since the default {{Comparator.comparing}} cannot handle comparison between null values.
> h2. How to reproduce:
> (1) Set {{dfs.heartbeat.interval}} to {{{}1753310367{}}}, and {{dfs.namenode.read.considerStorageType}} to {{true}}
> (2) Run test: {{org.apache.hadoop.hdfs.server.blockmanagement.TestSortLocatedBlock#testAviodStaleAndSlowDatanodes}}
> h2. Stacktrace:
> {noformat}
> java.lang.NullPointerException
> at java.base/java.util.Comparator.lambda$comparing$77a9974f$1(Comparator.java:469)
> at java.base/java.util.TimSort.countRunAndMakeAscending(TimSort.java:355)
> at java.base/java.util.TimSort.sort(TimSort.java:220)
> at java.base/java.util.Arrays.sort(Arrays.java:1515)
> at java.base/java.util.ArrayList.sort(ArrayList.java:1750)
> at java.base/java.util.Collections.sort(Collections.java:179)
> at org.apache.hadoop.hdfs.server.blockmanagement.DatanodeManager.lambda$createSecondaryNodeSorter$0(DatanodeManager.java:654)
> at org.apache.hadoop.net.NetworkTopology.sortByDistance(NetworkTopology.java:983)
> at org.apache.hadoop.net.NetworkTopology.sortByDistanceUsingNetworkLocation(NetworkTopology.java:946)
> at org.apache.hadoop.hdfs.server.blockmanagement.DatanodeManager.sortLocatedBlock(DatanodeManager.java:637)
> at org.apache.hadoop.hdfs.server.blockmanagement.DatanodeManager.sortLocatedBlocks(DatanodeManager.java:554)
> at org.apache.hadoop.hdfs.server.blockmanagement.TestSortLocatedBlock.testAviodStaleAndSlowDatanodes(TestSortLocatedBlock.java:144){noformat}
> For an easy reproduction, run the reproduce.sh in the attachment. We are happy to provide a patch if this issue is confirmed.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-help@hadoop.apache.org