[GitHub] [incubator-doris] xiaokang commented on pull request #9566: optimize lz4 compress and decompress speed by reusing context

[GitBox <gi...@apache.org>]

xiaokang commented on PR #9566:
URL: https://github.com/apache/incubator-doris/pull/9566#issuecomment-1126649753

   We see x6.75 speedup in the following comparison test.
   
   test env
   - test data: 10GB text log, 44 million rows
   - test table: test_table(ts varchar(30), log string)
   - test SQL: set enable_vectorized_engine=1; select sum(length(log)) from test_table
   - be.conf: disable_storage_page_cache = true   
   set this config to disable doris page cache to avoid all data cached in memory for test real decompression speed.
   
   test result 
   - master branch runtime: 5.2 seconds
   - lz4_compress_opt branch runtime: 0.77 seconds
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org