You are viewing a plain text version of this content. The canonical link for it is here.

Posted to issues@lucene.apache.org by "Mark Harwood (Jira)" <ji...@apache.org> on 2020/02/07 09:45:00 UTC

[jira] [Created] (LUCENE-9211) Adding compression to BinaryDocValues storage

Mark Harwood created LUCENE-9211:
------------------------------------

             Summary: Adding compression to BinaryDocValues storage
                 Key: LUCENE-9211
                 URL: https://issues.apache.org/jira/browse/LUCENE-9211
             Project: Lucene - Core
          Issue Type: Improvement
          Components: core/codecs
            Reporter: Mark Harwood
            Assignee: Mark Harwood


While SortedSetDocValues can be used today to store identical values in a compact form this is not effective for data with many unique values.

The proposal is that BinaryDocValues should be stored in LZ4 compressed blocks which can dramatically reduce disk storage costs in many cases. The proposal is blocks of a number of documents are stored as a single compressed blob along with metadata that records offsets where the original document values can be found in the uncompressed content.

There's a trade-off here between efficient compression (more docs-per-block = better compression) and fast retrieval times (fewer docs-per-block = faster read access for single values). A fixed block size of 32 docs seems like it would be a reasonable compromise for most scenarios.

A PR is up for review here [https://github.com/apache/lucene-solr/pull/1234]



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org