[GitHub] [incubator-doris] kangkaisen opened a new issue #2377: Doris Stream Load concurrent test

[GitBox <gi...@apache.org>]

kangkaisen opened a new issue #2377: Doris Stream Load concurrent test
URL: https://github.com/apache/incubator-doris/issues/2377
 
 
   ### Test code
   https://gist.github.com/kangkaisen/5e8c22a8327fa9c8d20edffd0010ecf7
   
   ### Test Doris code
   **Master Version**: https://github.com/apache/incubator-doris/commit/92536272d34abf2182d81b8b709165d4fbda08fd
   
   **Reduce  Version**:  https://github.com/apache/incubator-doris/pull/2376  
   
   **Concurrent Version**: https://github.com/apache/incubator-doris/pull/2375
   
   ### Test Env
   The test cluster has 3 BE. Each BE is 8 core, 16 G Mem, 1 Disk.
   
   ### Test goal
   The test goal is publish version concurrent performance, So one stream load we only send on row data.
   
   ### Test way
   1 Create table
   ```
   CREATE TABLE `stream_test` (
     `id` bigint(20) NOT NULL COMMENT "",
     `username` varchar(32) NOT NULL COMMENT ""
   ) ENGINE=OLAP
   DUPLICATE KEY(`id`)
   COMMENT "OLAP"
   DISTRIBUTED BY HASH(`id`) BUCKETS 20
   PROPERTIES (
   "storage_type" = "COLUMN"
   );
   ```
   2 Run test code
   3 Truncate table
   
   ### Changed Config
   BE
   ```
   default_rowset_type = BETA
   compaction_rowset_type = BETA
   publish_version_worker_count = 10
   ```
   FE
   ```
   max_running_txn_num_per_db = 10000
   ```
   
   ### Test Result
   
   When CORE_POOL_SIZE is 100, ROW_NUM = 1000:
   
   **Master Version time**: 305009 ms 
   **Reduce Version time**:  24437 ms
   **Concurrent Version time**: 6880 ms
   
   When CORE_POOL_SIZE is 1000, ROW_NUM = 5000:
   
   **Reduce Version Avg time**:  195 s
   **Concurrent Version Avg time**: 42s
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org