You are viewing a plain text version of this content. The canonical link for it is here.
Posted to hdfs-issues@hadoop.apache.org by "JiangHua Zhu (Jira)" <ji...@apache.org> on 2021/10/12 09:39:00 UTC
[jira] [Commented] (HDFS-16269) [Fix] Improve
NNThroughputBenchmark#blockReport operation
[ https://issues.apache.org/jira/browse/HDFS-16269?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17427590#comment-17427590 ]
JiangHua Zhu commented on HDFS-16269:
-------------------------------------
When using the new patch, similar exceptions will not appear.
Here are some effects:
./bin/hadoop org.apache.hadoop.hdfs.server.namenode.NNThroughputBenchmark -fs xxxx -op blockReport -datanodes 3 -reports 3
Result:
21/10/12 17:05:09 INFO namenode.NNThroughputBenchmark: Starting benchmark: blockReport
21/10/12 17:05:10 INFO namenode.NNThroughputBenchmark: Creating 10 files with 10 blocks each.
21/10/12 17:05:10 FATAL namenode.NNThroughputBenchmark: Log level = ERROR
21/10/12 17:05:10 INFO namenode.NNThroughputBenchmark: Starting 3 blockReport(s).
21/10/12 17:05:10 INFO namenode.NNThroughputBenchmark:
21/10/12 17:05:10 INFO namenode.NNThroughputBenchmark: --- blockReport inputs ---
21/10/12 17:05:10 INFO namenode.NNThroughputBenchmark: reports = 3
21/10/12 17:05:10 INFO namenode.NNThroughputBenchmark: datanodes = 3 (100, 54, 58)
21/10/12 17:05:10 INFO namenode.NNThroughputBenchmark: blocksPerReport = 100
21/10/12 17:05:10 INFO namenode.NNThroughputBenchmark: blocksPerFile = 10
21/10/12 17:05:10 INFO namenode.NNThroughputBenchmark: --- blockReport stats ---
21/10/12 17:05:10 INFO namenode.NNThroughputBenchmark: # operations: 3
21/10/12 17:05:10 INFO namenode.NNThroughputBenchmark: Elapsed Time: 8
21/10/12 17:05:10 INFO namenode.NNThroughputBenchmark: Ops per sec: 375.0
21/10/12 17:05:10 INFO namenode.NNThroughputBenchmark: Average Time: 7
> [Fix] Improve NNThroughputBenchmark#blockReport operation
> ---------------------------------------------------------
>
> Key: HDFS-16269
> URL: https://issues.apache.org/jira/browse/HDFS-16269
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: benchmarks, namenode
> Affects Versions: 2.9.2
> Reporter: JiangHua Zhu
> Assignee: JiangHua Zhu
> Priority: Major
> Labels: pull-request-available
> Time Spent: 10m
> Remaining Estimate: 0h
>
> When using NNThroughputBenchmark to verify the blockReport, you will get some exception information.
> Commands used:
> ./bin/hadoop org.apache.hadoop.hdfs.server.namenode.NNThroughputBenchmark -fs xxxx -op blockReport -datanodes 3 -reports 1
> The exception information:
> 21/10/12 14:35:18 INFO namenode.NNThroughputBenchmark: Starting benchmark: blockReport
> 21/10/12 14:35:19 INFO namenode.NNThroughputBenchmark: Creating 10 files with 10 blocks each.
> 21/10/12 14:35:19 ERROR namenode.NNThroughputBenchmark: java.lang.ArrayIndexOutOfBoundsException: 50009
> at org.apache.hadoop.hdfs.server.namenode.NNThroughputBenchmark$BlockReportStats.addBlocks(NNThroughputBenchmark.java:1161)
> at org.apache.hadoop.hdfs.server.namenode.NNThroughputBenchmark$BlockReportStats.generateInputs(NNThroughputBenchmark.java:1143)
> at org.apache.hadoop.hdfs.server.namenode.NNThroughputBenchmark$OperationStatsBase.benchmark(NNThroughputBenchmark.java:257)
> at org.apache.hadoop.hdfs.server.namenode.NNThroughputBenchmark.run(NNThroughputBenchmark.java:1528)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:90)
> at org.apache.hadoop.hdfs.server.namenode.NNThroughputBenchmark.runBenchmark(NNThroughputBenchmark.java:1430)
> at org.apache.hadoop.hdfs.server.namenode.NNThroughputBenchmark.main(NNThroughputBenchmark.java:1550)
> Exception in thread "main" java.lang.ArrayIndexOutOfBoundsException: 50009
> at org.apache.hadoop.hdfs.server.namenode.NNThroughputBenchmark$BlockReportStats.addBlocks(NNThroughputBenchmark.java:1161)
> at org.apache.hadoop.hdfs.server.namenode.NNThroughputBenchmark$BlockReportStats.generateInputs(NNThroughputBenchmark.java:1143)
> at org.apache.hadoop.hdfs.server.namenode.NNThroughputBenchmark$OperationStatsBase.benchmark(NNThroughputBenchmark.java:257)
> at org.apache.hadoop.hdfs.server.namenode.NNThroughputBenchmark.run(NNThroughputBenchmark.java:1528)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:90)
> at org.apache.hadoop.hdfs.server.namenode.NNThroughputBenchmark.runBenchmark(NNThroughputBenchmark.java:1430)
> at org.apache.hadoop.hdfs.server.namenode.NNThroughputBenchmark.main(NNThroughputBenchmark.java:1550)
> Checked some code and found that the problem appeared here.
> private ExtendedBlock addBlocks(String fileName, String clientName)
> throws IOException {
> for(DatanodeInfo dnInfo: loc.getLocations()) {
> int dnIdx = dnInfo.getXferPort()-1;
> datanodes[dnIdx].addBlock(loc.getBlock().getLocalBlock());
> }
> }
> It can be seen from this that what dnInfo.getXferPort() gets is a port information and should not be used as an index of an array.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-help@hadoop.apache.org