You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@lucene.apache.org by "Praveen Nishchal (Jira)" <ji...@apache.org> on 2021/09/15 07:52:00 UTC
[jira] [Comment Edited] (LUCENE-8739) ZSTD Compressor support in
Lucene
[ https://issues.apache.org/jira/browse/LUCENE-8739?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17415363#comment-17415363 ]
Praveen Nishchal edited comment on LUCENE-8739 at 9/15/21, 7:51 AM:
--------------------------------------------------------------------
I have developed new custom codec which integrates Zstd compression and decompression in StoredFieldFormat only. It uses Zstd-JNI ([https://github.com/luben/zstd-jni]). With reuters21578 (plain text Document derived from reuters21578) corpus benchmark run for index and search, following high level observations were made:
# Zstd provides a better compression ratio compared to lz4. Benchmark run(index) shows 30% smaller size .fdt(Stored Field data) file compared to LZ4.
# Index run with Zstd has almost same throughput as that of index run with LZ4.
# Search run with Zstd has 6% faster QPS than search run with LZ4
Above implementation is written in Java without dictionary compression/decompression at default compression level of 3 with 600 KB chunk size (10 * 60 * 1024 , same as LZ4).
With all these observations, Zstd option alongside LZ4 and deflate looks promising!! Kindly share thoughts!
was (Author: pru30):
_I have developed new custom codec which integrates Zstd compression and decompression in StoredFieldFormat_ _only. It uses Zstd-JNI ([https://github.com/luben/zstd-jni]). With reuters21578 (plain text Document derived from reuters21578) corpus benchmark run for index and search, following high level observations were made:_
__
# _Zstd provides a better compression ratio compared to lz4. Benchmark run(index) shows 30% smaller size .fdt(Stored Field data) file compared to LZ4._
# _Index run with Zstd has almost same throughput as that of index run with LZ4._
# _Search run with Zstd has 6% faster QPS than search run with LZ4_
__
_Above implementation is written in Java without dictionary compression/decompression at default compression level of 3 with 600 KB chunk size (10 * 60 * 1024 , same as LZ4)._
__
_With all these observations, Zstd option alongside LZ4 and deflate looks promising!! Kindly share thoughts!_
> ZSTD Compressor support in Lucene
> ---------------------------------
>
> Key: LUCENE-8739
> URL: https://issues.apache.org/jira/browse/LUCENE-8739
> Project: Lucene - Core
> Issue Type: New Feature
> Components: core/codecs
> Reporter: Sean Torres
> Priority: Minor
> Labels: features
> Time Spent: 1h
> Remaining Estimate: 0h
>
> ZStandard has a great speed and compression ratio tradeoff.
> ZStandard is open source compression from Facebook.
> More about ZSTD
> [https://github.com/facebook/zstd]
> [https://code.facebook.com/posts/1658392934479273/smaller-and-faster-data-compression-with-zstandard/]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org