[jira] [Updated] (LUCENE-10375) Speed up HNSW merge by writing combined vector data

["Alessandro Benedetti (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/LUCENE-10375?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Alessandro Benedetti updated LUCENE-10375:
------------------------------------------
    Labels: vector-based-search  (was: )

> Speed up HNSW merge by writing combined vector data
> ---------------------------------------------------
>
>                 Key: LUCENE-10375
>                 URL: https://issues.apache.org/jira/browse/LUCENE-10375
>             Project: Lucene - Core
>          Issue Type: Improvement
>            Reporter: Julie Tibshirani
>            Priority: Major
>              Labels: vector-based-search
>             Fix For: 9.1
>
>          Time Spent: 7h 10m
>  Remaining Estimate: 0h
>
> When merging segments together, the HNSW writer creates a VectorValues instance that gives a merged view of all the segments' VectorValues. This merged instance is used when constructing the new HNSW graph. Graph building needs random access, and the merged VectorValues support this by mapping from merged ordinals -> segments and segment ordinals.
> This mapping seems to add overhead. The nightly indexing benchmarks sometimes show substantial time in Arrays.binarySearch (used to map an ordinal to a segment): https://blunders.io/jfr-demo/indexing-1kb-vectors-2022.01.09.18.03.19/top_down_cpu_samples.
> Instead of using a merged VectorValues to create the graph, maybe we could first write all the segment vectors to a file, and use that file to build the graph.



--
This message was sent by Atlassian Jira
(v8.20.7#820007)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org