You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@lucene.apache.org by "David Smiley (Jira)" <ji...@apache.org> on 2020/02/09 05:07:00 UTC
[jira] [Commented] (LUCENE-9211) Adding compression to
BinaryDocValues storage
[ https://issues.apache.org/jira/browse/LUCENE-9211?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17033125#comment-17033125 ]
David Smiley commented on LUCENE-9211:
--------------------------------------
This seems cool for some use-cases but I worry about the overhead for others. I think I have a benchmark module ".alg" file for SerializedDVStrategy in spatial-extras. I should try it out on your PR.
I wish it was easier for us to let users toggle the choice of DocValuesFormat only for one type but not for others. DocValuesFormat is really a format of formats, which is inflexible. [~juan.duran], a colleague of mine, has been diving into this topic lately and I hope he shares it here (new issue of course).
> Adding compression to BinaryDocValues storage
> ---------------------------------------------
>
> Key: LUCENE-9211
> URL: https://issues.apache.org/jira/browse/LUCENE-9211
> Project: Lucene - Core
> Issue Type: Improvement
> Components: core/codecs
> Reporter: Mark Harwood
> Assignee: Mark Harwood
> Priority: Minor
> Labels: pull-request-available
>
> While SortedSetDocValues can be used today to store identical values in a compact form this is not effective for data with many unique values.
> The proposal is that BinaryDocValues should be stored in LZ4 compressed blocks which can dramatically reduce disk storage costs in many cases. The proposal is blocks of a number of documents are stored as a single compressed blob along with metadata that records offsets where the original document values can be found in the uncompressed content.
> There's a trade-off here between efficient compression (more docs-per-block = better compression) and fast retrieval times (fewer docs-per-block = faster read access for single values). A fixed block size of 32 docs seems like it would be a reasonable compromise for most scenarios.
> A PR is up for review here [https://github.com/apache/lucene-solr/pull/1234]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org