You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by "Adrien Grand (JIRA)" <ji...@apache.org> on 2018/02/20 10:30:00 UTC
[jira] [Commented] (LUCENE-8178) Bulk operations for LongValues and
Sorted[Set]DocValues
[ https://issues.apache.org/jira/browse/LUCENE-8178?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16369904#comment-16369904 ]
Adrien Grand commented on LUCENE-8178:
--------------------------------------
I understand how it can make things faster, but I'm a bit on the fence due to how specialized it is (it seems to focus on faceting on MatchAllDocsQuery on an index with not too many deletions) vs. the number of additional APIs.
> Bulk operations for LongValues and Sorted[Set]DocValues
> -------------------------------------------------------
>
> Key: LUCENE-8178
> URL: https://issues.apache.org/jira/browse/LUCENE-8178
> Project: Lucene - Core
> Issue Type: Improvement
> Affects Versions: 7.2.1
> Reporter: Nikolay Khitrin
> Priority: Major
> Attachments: LUCENE-8178-for-solr.patch, LUCENE-8178.patch
>
>
> One-by-one DocValues iteration by {{advanceExact}} and {{nextOrd}}/{{ordValue}} is really slow for bulk operations like facetting. Reading and unpacking integers in blocks is substantially faster but DocValues for now can be queried only for single document.
> To apply document-based bulk processing {{DocIdSetIterator}} matches have to be splitted to sequential docID runs and remapped to underlying {{LongValues}} positions.
> After this transformation relatively large linear scans can be performed over packed integers.
>
> To do this two new interfaces
> 1. {{LongValuesCollector}} ({{collectValue(long index, long value)}}).
> 2. {{OrdStatsCollector}} ({{collectOrd(long ord)}}, {{collectMissing(int count)}}).
> and three new functions are introduced
> 1. {{LongValues.forRange(long begin, long end, LongValuesCollector collector)}}
> 2. {{SortedDocValues.forEach(DocIdSetIterator disi, OrdStatsConsumer collector)}}
> 3. {{SortedSetDocValues.forEach(DocIdSetIterator disi, OrdStatsConsumer collector)}}
> with reference implementations.
> Optimized versions of these functions are provided for:
> 1. {{DirectReader}} for non-32/64 bits per value cases (using {{PackedInts.Decoder}}).
> 2. {{Lucene70DocValuesProducer}} {{getSorted}} and {{getSortedSet}} (both sparse and dense).
>
> Measured Solr facetting performance boost is up to 2 - 2.5x on real index.
> Patch for Solr {{DocValuesFacets}} is also provided as separate file.
>
> Implementation notes:
> * {{OrdStatsCollector}} does not accept document id because it will ruin performance for {{SortedSetDocValues}} due to excessive position lookups.
> * This patch is fully compatible with Lucene 7.0 DocValues format.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org