You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@solr.apache.org by Wei <we...@gmail.com> on 2025/03/19 23:02:11 UTC
Not able to optimize Solr 9.8.0 HNSW index
Greetings everyone,
After building the Solr HNSW index on 9.8.0, I tried to optimize it to a
single segment. However the optimization doesn't happen and no errors
found in solr log.
Schema:
<fieldType name="knn_vector_float_256" class="solr.DenseVectorField"
vectorDimension="256" similarityFunction="dot_product" vectorEncoding=
"FLOAT32" hnswMaxConnections="16" hnswBeamWidth="200"/> <fieldType name=
"knn_vector_byte_256" class="solr.DenseVectorField" vectorDimension="256"
similarityFunction="dot_product" vectorEncoding="BYTE" hnswMaxConnections=
"16" hnswBeamWidth="200"/> <field name="item_vector_float" type=
"knn_vector_float_256" indexed="true" stored="false"/>
<field name="item_vector_byte" type="knn_vector_byte_256" indexed="true"
stored="false"/>
Solrconfig for merge:
<mergePolicyFactory class=
"${mergePolicyFactory:org.apache.solr.index.TieredMergePolicyFactory}">
<int name="maxMergeAtOnce">25</int> <int name="segmentsPerTier">25</int> <
int name="maxMergedSegmentMB">100000</int> </mergePolicyFactory>
In total 30M docs are indexed and I trigger optimization with
/update?optimize=true&waitSearcher=false&maxSegments=1. When using the BYTE
type, optimization finishes successfully. Howeverwith FLOAT32 type it does
not work, and ends up with ~33G index in ~80 segments. Also I noticed there
isn't any reduction in segment number.
The solr node has ~200G memory and plenty of disk space.
Thanks,
Wei
Re: Not able to optimize Solr 9.8.0 HNSW index
Posted by "Natarajan, Rajeswari" <ra...@sap.com.INVALID>.
Created https://issues.apache.org/jira/browse/SOLR-17747
Vector/Hybrid search response times are affected by this behavior. So raised this issue.
Thanks,
Rajeswari
On 4/25/25, 1:52 PM, "Natarajan, Rajeswari" <rajeswari.natarajan@sap.com.INVA <ma...@sap.com.INVA>LID> wrote:
Is this a known issue ? , Do we need to create a solr JIRA to capture this ?
On 4/24/25, 5:39 PM, "Natarajan, Rajeswari" <rajeswari.natarajan@sap.com.INVA <ma...@sap.com.INVA> <mailto:rajeswari.natarajan@sap.com.INVA <ma...@sap.com.INVA>>LID> wrote:
Facing the same issue , any resolution found for this. Please let me know.
Thank you
On 3/19/25, 4:03 PM, "Wei" <weiwang19@gmail.com <ma...@gmail.com> <mailto:weiwang19@gmail.com <ma...@gmail.com>> <mailto:weiwang19@gmail.com <ma...@gmail.com> <mailto:weiwang19@gmail.com <ma...@gmail.com>>>> wrote:
Greetings everyone,
After building the Solr HNSW index on 9.8.0, I tried to optimize it to a
single segment. However the optimization doesn't happen and no errors
found in solr log.
Schema:
<fieldType name="knn_vector_float_256" class="solr.DenseVectorField"
vectorDimension="256" similarityFunction="dot_product" vectorEncoding=
"FLOAT32" hnswMaxConnections="16" hnswBeamWidth="200"/> <fieldType name=
"knn_vector_byte_256" class="solr.DenseVectorField" vectorDimension="256"
similarityFunction="dot_product" vectorEncoding="BYTE" hnswMaxConnections=
"16" hnswBeamWidth="200"/> <field name="item_vector_float" type=
"knn_vector_float_256" indexed="true" stored="false"/>
<field name="item_vector_byte" type="knn_vector_byte_256" indexed="true"
stored="false"/>
Solrconfig for merge:
<mergePolicyFactory class=
"${mergePolicyFactory:org.apache.solr.index.TieredMergePolicyFactory}">
<int name="maxMergeAtOnce">25</int> <int name="segmentsPerTier">25</int> <
int name="maxMergedSegmentMB">100000</int> </mergePolicyFactory>
In total 30M docs are indexed and I trigger optimization with
/update?optimize=true&waitSearcher=false&maxSegments=1. When using the BYTE
type, optimization finishes successfully. Howeverwith FLOAT32 type it does
not work, and ends up with ~33G index in ~80 segments. Also I noticed there
isn't any reduction in segment number.
The solr node has ~200G memory and plenty of disk space.
Thanks,
Wei
Re: Not able to optimize Solr 9.8.0 HNSW index
Posted by "Natarajan, Rajeswari" <ra...@sap.com.INVALID>.
Is this a known issue ? , Do we need to create a solr JIRA to capture this ?
On 4/24/25, 5:39 PM, "Natarajan, Rajeswari" <rajeswari.natarajan@sap.com.INVA <ma...@sap.com.INVA>LID> wrote:
Facing the same issue , any resolution found for this. Please let me know.
Thank you
On 3/19/25, 4:03 PM, "Wei" <weiwang19@gmail.com <ma...@gmail.com> <mailto:weiwang19@gmail.com <ma...@gmail.com>>> wrote:
Greetings everyone,
After building the Solr HNSW index on 9.8.0, I tried to optimize it to a
single segment. However the optimization doesn't happen and no errors
found in solr log.
Schema:
<fieldType name="knn_vector_float_256" class="solr.DenseVectorField"
vectorDimension="256" similarityFunction="dot_product" vectorEncoding=
"FLOAT32" hnswMaxConnections="16" hnswBeamWidth="200"/> <fieldType name=
"knn_vector_byte_256" class="solr.DenseVectorField" vectorDimension="256"
similarityFunction="dot_product" vectorEncoding="BYTE" hnswMaxConnections=
"16" hnswBeamWidth="200"/> <field name="item_vector_float" type=
"knn_vector_float_256" indexed="true" stored="false"/>
<field name="item_vector_byte" type="knn_vector_byte_256" indexed="true"
stored="false"/>
Solrconfig for merge:
<mergePolicyFactory class=
"${mergePolicyFactory:org.apache.solr.index.TieredMergePolicyFactory}">
<int name="maxMergeAtOnce">25</int> <int name="segmentsPerTier">25</int> <
int name="maxMergedSegmentMB">100000</int> </mergePolicyFactory>
In total 30M docs are indexed and I trigger optimization with
/update?optimize=true&waitSearcher=false&maxSegments=1. When using the BYTE
type, optimization finishes successfully. Howeverwith FLOAT32 type it does
not work, and ends up with ~33G index in ~80 segments. Also I noticed there
isn't any reduction in segment number.
The solr node has ~200G memory and plenty of disk space.
Thanks,
Wei
Re: Not able to optimize Solr 9.8.0 HNSW index
Posted by "Natarajan, Rajeswari" <ra...@sap.com.INVALID>.
Facing the same issue , any resolution found for this. Please let me know.
Thank you
On 3/19/25, 4:03 PM, "Wei" <weiwang19@gmail.com <ma...@gmail.com>> wrote:
Greetings everyone,
After building the Solr HNSW index on 9.8.0, I tried to optimize it to a
single segment. However the optimization doesn't happen and no errors
found in solr log.
Schema:
<fieldType name="knn_vector_float_256" class="solr.DenseVectorField"
vectorDimension="256" similarityFunction="dot_product" vectorEncoding=
"FLOAT32" hnswMaxConnections="16" hnswBeamWidth="200"/> <fieldType name=
"knn_vector_byte_256" class="solr.DenseVectorField" vectorDimension="256"
similarityFunction="dot_product" vectorEncoding="BYTE" hnswMaxConnections=
"16" hnswBeamWidth="200"/> <field name="item_vector_float" type=
"knn_vector_float_256" indexed="true" stored="false"/>
<field name="item_vector_byte" type="knn_vector_byte_256" indexed="true"
stored="false"/>
Solrconfig for merge:
<mergePolicyFactory class=
"${mergePolicyFactory:org.apache.solr.index.TieredMergePolicyFactory}">
<int name="maxMergeAtOnce">25</int> <int name="segmentsPerTier">25</int> <
int name="maxMergedSegmentMB">100000</int> </mergePolicyFactory>
In total 30M docs are indexed and I trigger optimization with
/update?optimize=true&waitSearcher=false&maxSegments=1. When using the BYTE
type, optimization finishes successfully. Howeverwith FLOAT32 type it does
not work, and ends up with ~33G index in ~80 segments. Also I noticed there
isn't any reduction in segment number.
The solr node has ~200G memory and plenty of disk space.
Thanks,
Wei