[GitHub] [incubator-doris] BiteTheDDDDt opened a new issue #6505: [Feature] Support for storage layer benchmark

[GitBox <gi...@apache.org>]

BiteTheDDDDt opened a new issue #6505:
URL: https://github.com/apache/incubator-doris/issues/6505


   **Is your feature request related to a problem? Please describe.**
   I have developed a Benchmark tool that is used to test some read and write performance tests related to the storage layer. 
   
   **Describe the solution you'd like**
   It is currently testing segment and partial Page.
   Dataset can be randomly generated or imported by data files.
   It also supports testing functions written by users, used to get some effects of some third-party libraries at doris. 
   
   **Describe alternatives you've considered**
   It will be compiled when the user executes `./run-be-ut.sh` and placed in `ut_build_ASAN/test/tools/benchmark_tool`.
   But it doesn't be executed when `Run All Ut (- Run) `.
   
   **Additional context**
   Here are some usage examples:
   
   `./benchmark_tool --operation=Custom`
   Execute the benchmark of the custum function (need to implement the `init()` and `run()` functions) 
   
   `./benchmark_tool --operation=BinaryDictPageEncode --rows_number=10000 --iterations=40`
   
   Execute the benchmark of `BinaryDictPage` encode 1000 randomly generated data,Iterate forty times to calculate the average time consumption.
   
   `./benchmark_tool --operation=SegmentScan --column_type=int,varchar --rows_number=10000 --iterations=0`
   
   Execute the benchmark of `Segment` scan data from file(data is randomly generated), iterations is set to 0 means the number of iterations is automatically set.
   
   `./benchmark_tool --operation=SegmentWrite --column_type=int --rows_number=10000 --iterations=10`
   
   Execute the benchmark of `Segment` write data to file(data is randomly generated).
   
   `./benchmark_tool --operation=SegmentScanByFile --input_file=./sample.dat --iterations=10`
   
   Execute the benchmark of `Segment` scan data from file(data from input files).
   
   Data file format: 
   The first line defines Shcema
   The rest of the content is DataSet
   
   For example: 
   ```
   int,char,varchar
   123,hello,world
   321,good,bye
   ```
   
   `./benchmark_tool --operation=SegmentWriteByFile --input_file=./sample.dat --iterations=10`
   
   Execute the benchmark of `Segment` write data from file(data from input files).


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org