You are viewing a plain text version of this content. The canonical link for it is here.
Posted to hdfs-dev@hadoop.apache.org by "JiangHua Zhu (Jira)" <ji...@apache.org> on 2021/12/20 05:43:00 UTC

[jira] [Created] (HDFS-16389) Improve NNThroughputBenchmark test mkdirs

JiangHua Zhu created HDFS-16389:
-----------------------------------

             Summary: Improve NNThroughputBenchmark test mkdirs
                 Key: HDFS-16389
                 URL: https://issues.apache.org/jira/browse/HDFS-16389
             Project: Hadoop HDFS
          Issue Type: Improvement
          Components: benchmarks, namenode
    Affects Versions: 2.9.2
            Reporter: JiangHua Zhu


When using the NNThroughputBenchmark test to create a large number of directories, some abnormal information will be prompted.
Here is the command:
./bin/hadoop org.apache.hadoop.hdfs.server.namenode.NNThroughputBenchmark -fs hdfs://xxxx -op mkdirs -threads 30 -dirs 5000000

There are some exceptions here, such as:
21/12/20 10:25:00 INFO namenode.NNThroughputBenchmark: Starting benchmark: mkdirs
21/12/20 10:25:01 INFO namenode.NNThroughputBenchmark: Generate 5000000 inputs for mkdirs
21/12/20 10:25:08 ERROR namenode.NNThroughputBenchmark: java.lang.ArrayIndexOutOfBoundsException: 20
	at org.apache.hadoop.hdfs.server.namenode.FileNameGenerator.getNextDirName(FileNameGenerator.java:65)
	at org.apache.hadoop.hdfs.server.namenode.FileNameGenerator.getNextFileName(FileNameGenerator.java:73)
	at org.apache.hadoop.hdfs.server.namenode.NNThroughputBenchmark$MkdirsStats.generateInputs(NNThroughputBenchmark.java:668)
	at org.apache.hadoop.hdfs.server.namenode.NNThroughputBenchmark$OperationStatsBase.benchmark(NNThroughputBenchmark.java:257)
	at org.apache.hadoop.hdfs.server.namenode.NNThroughputBenchmark.run(NNThroughputBenchmark.java:1528)
	at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)
	at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:90)
	at org.apache.hadoop.hdfs.server.namenode.NNThroughputBenchmark.runBenchmark(NNThroughputBenchmark.java:1430)
	at org.apache.hadoop.hdfs.server.namenode.NNThroughputBenchmark.main(NNThroughputBenchmark.java:1550)

Exception in thread "main" java.lang.ArrayIndexOutOfBoundsException: 20
	at org.apache.hadoop.hdfs.server.namenode.FileNameGenerator.getNextDirName(FileNameGenerator.java:65)
	at org.apache.hadoop.hdfs.server.namenode.FileNameGenerator.getNextFileName(FileNameGenerator.java:73)
	at org.apache.hadoop.hdfs.server.namenode.NNThroughputBenchmark$MkdirsStats.generateInputs(NNThroughputBenchmark.java:668)
	at org.apache.hadoop.hdfs.server.namenode.NNThroughputBenchmark$OperationStatsBase.benchmark(NNThroughputBenchmark.java:257)
	at org.apache.hadoop.hdfs.server.namenode.NNThroughputBenchmark.run(NNThroughputBenchmark.java:1528)
	at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)
	at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:90)
	at org.apache.hadoop.hdfs.server.namenode.NNThroughputBenchmark.runBenchmark(NNThroughputBenchmark.java:1430)
	at org.apache.hadoop.hdfs.server.namenode.NNThroughputBenchmark.main(NNThroughputBenchmark.java:1550)

These messages appear because some parameters are incorrectly set, such as dirsPerDir or filesPerDir.
When we see this log, this will make us have some questions.



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-help@hadoop.apache.org