You are viewing a plain text version of this content. The canonical link for it is here.
Posted to hdfs-dev@hadoop.apache.org by "JiangHua Zhu (Jira)" <ji...@apache.org> on 2021/12/20 05:43:00 UTC
[jira] [Created] (HDFS-16389) Improve NNThroughputBenchmark test mkdirs
JiangHua Zhu created HDFS-16389:
-----------------------------------
Summary: Improve NNThroughputBenchmark test mkdirs
Key: HDFS-16389
URL: https://issues.apache.org/jira/browse/HDFS-16389
Project: Hadoop HDFS
Issue Type: Improvement
Components: benchmarks, namenode
Affects Versions: 2.9.2
Reporter: JiangHua Zhu
When using the NNThroughputBenchmark test to create a large number of directories, some abnormal information will be prompted.
Here is the command:
./bin/hadoop org.apache.hadoop.hdfs.server.namenode.NNThroughputBenchmark -fs hdfs://xxxx -op mkdirs -threads 30 -dirs 5000000
There are some exceptions here, such as:
21/12/20 10:25:00 INFO namenode.NNThroughputBenchmark: Starting benchmark: mkdirs
21/12/20 10:25:01 INFO namenode.NNThroughputBenchmark: Generate 5000000 inputs for mkdirs
21/12/20 10:25:08 ERROR namenode.NNThroughputBenchmark: java.lang.ArrayIndexOutOfBoundsException: 20
at org.apache.hadoop.hdfs.server.namenode.FileNameGenerator.getNextDirName(FileNameGenerator.java:65)
at org.apache.hadoop.hdfs.server.namenode.FileNameGenerator.getNextFileName(FileNameGenerator.java:73)
at org.apache.hadoop.hdfs.server.namenode.NNThroughputBenchmark$MkdirsStats.generateInputs(NNThroughputBenchmark.java:668)
at org.apache.hadoop.hdfs.server.namenode.NNThroughputBenchmark$OperationStatsBase.benchmark(NNThroughputBenchmark.java:257)
at org.apache.hadoop.hdfs.server.namenode.NNThroughputBenchmark.run(NNThroughputBenchmark.java:1528)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:90)
at org.apache.hadoop.hdfs.server.namenode.NNThroughputBenchmark.runBenchmark(NNThroughputBenchmark.java:1430)
at org.apache.hadoop.hdfs.server.namenode.NNThroughputBenchmark.main(NNThroughputBenchmark.java:1550)
Exception in thread "main" java.lang.ArrayIndexOutOfBoundsException: 20
at org.apache.hadoop.hdfs.server.namenode.FileNameGenerator.getNextDirName(FileNameGenerator.java:65)
at org.apache.hadoop.hdfs.server.namenode.FileNameGenerator.getNextFileName(FileNameGenerator.java:73)
at org.apache.hadoop.hdfs.server.namenode.NNThroughputBenchmark$MkdirsStats.generateInputs(NNThroughputBenchmark.java:668)
at org.apache.hadoop.hdfs.server.namenode.NNThroughputBenchmark$OperationStatsBase.benchmark(NNThroughputBenchmark.java:257)
at org.apache.hadoop.hdfs.server.namenode.NNThroughputBenchmark.run(NNThroughputBenchmark.java:1528)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:90)
at org.apache.hadoop.hdfs.server.namenode.NNThroughputBenchmark.runBenchmark(NNThroughputBenchmark.java:1430)
at org.apache.hadoop.hdfs.server.namenode.NNThroughputBenchmark.main(NNThroughputBenchmark.java:1550)
These messages appear because some parameters are incorrectly set, such as dirsPerDir or filesPerDir.
When we see this log, this will make us have some questions.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-help@hadoop.apache.org