[jira] [Commented] (HDFS-17221) Add NNThroughputBenchmark params to specify subcluster load in RBF

["Alexander Bogdanov (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/HDFS-17221?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17774544#comment-17774544 ] 

Alexander Bogdanov commented on HDFS-17221:
-------------------------------------------

*What's the problem?*

Right now NNThroughputBenchmark creates directories /nnThroughputBenchmark/<opName>. Many opName directories may be created in one benchmark run. Inside these directories benchmark creates files and directories according to the benchmark input parameters.

When benchmarking router-based federation cluster setup we don't have a control on how many files/ directories will be created on each subcluster. Also, we get only totals for the whole cluster, but want to get statistics per subcluster.

*Proposal*

Add the below functionaliry to NNThroughputBenchmark:
 # Benchmark must have an ability to setup the load (dirs, files, operations, threads) per subcluster in RBF mode. Effectively this means that we allow to specify first-level directories, so that they will go to different subclusters according to the router mapping table.
 # Benchmark must print statistics per subcluster (per first-level directory).
 # Benchmark must print overall cluster statistics.

> Add NNThroughputBenchmark params to specify subcluster load in RBF
> ------------------------------------------------------------------
>
>                 Key: HDFS-17221
>                 URL: https://issues.apache.org/jira/browse/HDFS-17221
>             Project: Hadoop HDFS
>          Issue Type: Improvement
>          Components: benchmarks
>    Affects Versions: 3.3.6
>            Reporter: Alexander Bogdanov
>            Priority: Major
>
> NNThroughputBenchmark creates directories /nnThroughputBenchmark/<opName>. So it's not possible to specify different load to subcluster in RBF mode and view benchmark results per subcluster.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-help@hadoop.apache.org