You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by GitBox <gi...@apache.org> on 2022/05/24 01:33:27 UTC
[GitHub] [incubator-doris] xiaokang opened a new pull request, #9747: add zstd compression codec
xiaokang opened a new pull request, #9747:
URL: https://github.com/apache/incubator-doris/pull/9747
# Proposed changes
ZSTD compression is fast with high compression ratio. It can be used to archive higher compression ratio than default Lz4f codec for storing cost sensitive data such as logs.
## Problem Summary:
Describe the overview of changes.
## Checklist(Required)
1. Does it affect the original behavior: (No)
2. Has unit tests been added: (Yes)
3. Has document been added or modified: (No)
4. Does it need to update dependencies: (No)
5. Are there any changes that cannot be rolled back: (No)
## Further comments
If this is a relatively large or complex change, kick off the discussion at [dev@doris.apache.org](mailto:dev@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] carlvinhust2012 commented on a diff in pull request #9747: add zstd compression codec
Posted by GitBox <gi...@apache.org>.
carlvinhust2012 commented on code in PR #9747:
URL: https://github.com/apache/incubator-doris/pull/9747#discussion_r880097962
##########
be/src/util/block_compression.cpp:
##########
@@ -22,6 +22,8 @@
#include <snappy/snappy-sinksource.h>
#include <snappy/snappy.h>
#include <zlib.h>
+#include <zstd.h>
Review Comment:
no need to add some documentation for zstd?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] xiaokang commented on pull request #9747: add zstd compression codec
Posted by GitBox <gi...@apache.org>.
xiaokang commented on PR #9747:
URL: https://github.com/apache/incubator-doris/pull/9747#issuecomment-1135310442
Compared to Lz4f codec, we see zstd codec get 35% compressed size off, 30% faster at first time read without OS page cache, 40% slower at second time read with OS page cache in the following comparison test.
test data: 25GB text log, 110 million rows
test table: test_table(ts varchar(30), log string)
test SQL: set enable_vectorized_engine=1; select sum(length(log)) from test_table
be.conf: disable_storage_page_cache = true
set this config to disable doris page cache to avoid all data cached in memory for test real decompression speed.
test result
master branch with lz4f codec result:
- compressed size 4.3G
- SQL first exec time(read data from disk + decompress + little computation) : 18.3s
- SQL second exec time(read data from OS pagecache + decompress + little computation) : 2.4s
this branch with zstd codec (hardcode enable it) result:
- compressed size: 2.8G
- SQL first exec time: 12.8s
- SQL second exec time: 3.4s
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] xiaokang commented on a diff in pull request #9747: add zstd compression codec
Posted by GitBox <gi...@apache.org>.
xiaokang commented on code in PR #9747:
URL: https://github.com/apache/incubator-doris/pull/9747#discussion_r880562308
##########
be/src/util/block_compression.cpp:
##########
@@ -22,6 +22,8 @@
#include <snappy/snappy-sinksource.h>
#include <snappy/snappy.h>
#include <zlib.h>
+#include <zstd.h>
Review Comment:
It will be better to add some doc for zstd.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] morningman merged pull request #9747: add zstd compression codec
Posted by GitBox <gi...@apache.org>.
morningman merged PR #9747:
URL: https://github.com/apache/incubator-doris/pull/9747
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] xiaokang commented on pull request #9747: add zstd compression codec
Posted by GitBox <gi...@apache.org>.
xiaokang commented on PR #9747:
URL: https://github.com/apache/incubator-doris/pull/9747#issuecomment-1135312848
https://github.com/apache/incubator-doris/pull/8923 need to be merged before users can enable zstd or other codec, since current Lz4f codec is hardcoded and not configurable.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] github-actions[bot] commented on pull request #9747: add zstd compression codec
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on PR #9747:
URL: https://github.com/apache/incubator-doris/pull/9747#issuecomment-1138364634
PR approved by at least one committer and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] github-actions[bot] commented on pull request #9747: add zstd compression codec
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on PR #9747:
URL: https://github.com/apache/incubator-doris/pull/9747#issuecomment-1138364670
PR approved by anyone and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org