[jira] [Created] (LUCENE-8403) Support 'filtered' term vectors - don't require all terms to be present

["Michael Braun (JIRA)" <ji...@apache.org>]

Michael Braun created LUCENE-8403:
-------------------------------------

             Summary: Support 'filtered' term vectors - don't require all terms to be present
                 Key: LUCENE-8403
                 URL: https://issues.apache.org/jira/browse/LUCENE-8403
             Project: Lucene - Core
          Issue Type: Improvement
            Reporter: Michael Braun


The genesis of this was a conversation and idea from [~dsmiley] several years ago.

In order to optimize term vector storage, we may not actually need all tokens to be present in the term vectors - and if so, ideally our codec could just opt not to store them.

I attempted to fork the standard codec and override the TermVectorsFormat and TermVectorsWriter to ignore storing certain Terms within a field. This worked, however, CheckIndex checks that the terms present in the standard postings are also present in the TVs, if TVs enabled. So this then doesn't work as 'valid' according to CheckIndex.

Can the TermVectorsFormat be made in such a way to support configuration of tokens that should not be stored (benefits: less storage, more optimal retrieval per doc)? Is this valuable to the wider community? Is there a way we can design this to not break CheckIndex's contract while at the same time lessening storage for unneeded tokens?



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org