You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by Michael Wechner <mi...@wyona.com> on 2025/09/09 20:49:12 UTC
Upgrade Codec 99 to 101 / 102
Hi
With Lucene 9.12.0 I set my own custom max vector dimension using
Codec codecInt8 = new Lucene99Codec() { @Override public
KnnVectorsFormat getKnnVectorsFormatForField(String field) { var
delegate = new Lucene99HnswScalarQuantizedVectorsFormat();
log.info("Vector Value Type: int8, Maximum Vector Dimension: " +
maxDimensions); return new DelegatingKnnVectorsFormat(delegate,
maxDimensions); } }; and
Codec codecFloat32 =new Lucene99Codec() {
@Override public KnnVectorsFormat getKnnVectorsFormatForField(String field) {
var delegate =new Lucene99HnswVectorsFormat();
log.info("Vector Value Type: float32, Maximum Vector Dimension: " +maxDimensions);
return new DelegatingKnnVectorsFormat(delegate,maxDimensions);
}
};
I am a little confused re which Codec / Vector Format classes I should use when upgrading to Lucene version 10.2.2
Any hints would be much appreciated!
Thanks
Michael
Re: Upgrade Codec 99 to 101 / 102
Posted by Michael Wechner <mi...@wyona.com>.
Hey Ben
Cool, thank you!
I just tested it successfully with
int8, Cohere, embed-english-v3.0
and
flow32, https://huggingface.co/sentence-transformers/all-mpnet-base-v2
Am 10.09.25 um 21:48 schrieb Benjamin Trent:
> Michael,
>
> Yeah, you should be able to use it for either input type. The int8
> will just pass through and be used as "normal" int8 and not quantized
> at all.
>
> Then only the float32 fields will end up being quantized.
>
> On Wed, Sep 10, 2025 at 3:42 PM Michael Wechner
> <mi...@wyona.com> wrote:
>
> Hi Ben
>
> Thanks very much for the insight!
>
> So IIUC it is correct to
> use Lucene102HnswBinaryQuantizedVectorsFormat() for float32 and
> int8, right?
>
> Thanks
>
> Michael
>
> Am 10.09.25 um 17:58 schrieb Benjamin Trent:
>> Hey Michael,
>>
>> Right now it won't quantize byte vectors that are provided. It
>> just passes them and treats them like normal.
>>
>> In the future, we would like to quantize the bytes as well!
>>
>> On Wed, Sep 10, 2025, 10:56 AM Michael Wechner
>> <mi...@wyona.com> wrote:
>>
>> Hi Uwe
>>
>> Thanks for your feedback!
>>
>> I am using now:
>>
>> private final int maxDimensions =16384;
>>
>> Codec codec =new Lucene101Codec() {
>> @Override public KnnVectorsFormat getKnnVectorsFormatForField(String field) {var delegate =new Lucene102HnswBinaryQuantizedVectorsFormat();
>> return new DelegatingKnnVectorsFormat(delegate,maxDimensions);
>> }
>> };
>> return codec;
>>
>> This seems to work fine, but I do not understand whether there is a difference anymore between float32 and int8 vector values,
>> or whetherLucene102HnswBinaryQuantizedVectorsFormat always "quantizes" the values?
>>
>> Thanks
>>
>> Michael
>>
>>
>> Am 10.09.25 um 10:19 schrieb Uwe Schindler:
>>>
>>> Hi,
>>>
>>> I think the best is to check the source code of default
>>> codec of the upgrade version and start from there. I agree,
>>> there should possibly a documentation available that gives
>>> the default componenets used in the default codec of a given
>>> release.
>>>
>>> Uwe
>>>
>>> Am 09.09.2025 um 22:49 schrieb Michael Wechner:
>>>>
>>>> Hi
>>>>
>>>> With Lucene 9.12.0 I set my own custom max vector dimension
>>>> using
>>>>
>>>> Codec codecInt8 = new Lucene99Codec() { @Override public
>>>> KnnVectorsFormat getKnnVectorsFormatForField(String field)
>>>> { var delegate = new
>>>> Lucene99HnswScalarQuantizedVectorsFormat(); log.info
>>>> <http://log.info>("Vector Value Type: int8, Maximum Vector
>>>> Dimension: " + maxDimensions); return new
>>>> DelegatingKnnVectorsFormat(delegate, maxDimensions); } }; and
>>>> Codec codecFloat32 =new Lucene99Codec() {
>>>> @Override public KnnVectorsFormat getKnnVectorsFormatForField(String field) {
>>>> var delegate =new Lucene99HnswVectorsFormat();
>>>> log.info("Vector Value Type: float32, Maximum Vector Dimension: " +maxDimensions);
>>>> return new DelegatingKnnVectorsFormat(delegate,maxDimensions);
>>>> }
>>>> };
>>>>
>>>> I am a little confused re which Codec / Vector Format classes I should use when upgrading to Lucene version 10.2.2
>>>>
>>>> Any hints would be much appreciated!
>>>>
>>>> Thanks
>>>>
>>>> Michael
>>>>
>>>>
>>> --
>>> Uwe Schindler
>>> Achterdiek 19, D-28357 Bremen <https://www.google.com/maps/search/Achterdiek+19,+D-28357+Bremen?entry=gmail&source=g>
>>> https://www.thetaphi.de
>>> eMail:uwe@thetaphi.de
>>
Re: Upgrade Codec 99 to 101 / 102
Posted by Benjamin Trent <be...@gmail.com>.
Michael,
Yeah, you should be able to use it for either input type. The int8 will
just pass through and be used as "normal" int8 and not quantized at all.
Then only the float32 fields will end up being quantized.
On Wed, Sep 10, 2025 at 3:42 PM Michael Wechner <mi...@wyona.com>
wrote:
> Hi Ben
>
> Thanks very much for the insight!
>
> So IIUC it is correct to use Lucene102HnswBinaryQuantizedVectorsFormat()
> for float32 and int8, right?
>
> Thanks
>
> Michael
> Am 10.09.25 um 17:58 schrieb Benjamin Trent:
>
> Hey Michael,
>
> Right now it won't quantize byte vectors that are provided. It just passes
> them and treats them like normal.
>
> In the future, we would like to quantize the bytes as well!
>
> On Wed, Sep 10, 2025, 10:56 AM Michael Wechner <mi...@wyona.com>
> wrote:
>
>> Hi Uwe
>>
>> Thanks for your feedback!
>>
>> I am using now:
>>
>> private final int maxDimensions = 16384;
>>
>> Codec codec = new Lucene101Codec() {
>> @Override public KnnVectorsFormat getKnnVectorsFormatForField(String field) { var delegate = new Lucene102HnswBinaryQuantizedVectorsFormat();
>> return new DelegatingKnnVectorsFormat(delegate, maxDimensions);
>> }
>> };
>> return codec;
>>
>> This seems to work fine, but I do not understand whether there is a difference anymore between float32 and int8 vector values,
>> or whether Lucene102HnswBinaryQuantizedVectorsFormat always "quantizes" the values?
>>
>> Thanks
>>
>> Michael
>>
>>
>> Am 10.09.25 um 10:19 schrieb Uwe Schindler:
>>
>> Hi,
>>
>> I think the best is to check the source code of default codec of the
>> upgrade version and start from there. I agree, there should possibly a
>> documentation available that gives the default componenets used in the
>> default codec of a given release.
>>
>> Uwe
>> Am 09.09.2025 um 22:49 schrieb Michael Wechner:
>>
>> Hi
>>
>> With Lucene 9.12.0 I set my own custom max vector dimension using
>>
>> Codec codecInt8 = new Lucene99Codec() { @Override public KnnVectorsFormat getKnnVectorsFormatForField(String field) { var delegate = new Lucene99HnswScalarQuantizedVectorsFormat(); log.info("Vector Value Type: int8, Maximum Vector Dimension: " + maxDimensions); return new DelegatingKnnVectorsFormat(delegate, maxDimensions); }};
>>
>> and
>>
>> Codec codecFloat32 = new Lucene99Codec() {
>> @Override public KnnVectorsFormat getKnnVectorsFormatForField(String field) {
>> var delegate = new Lucene99HnswVectorsFormat();
>> log.info("Vector Value Type: float32, Maximum Vector Dimension: " + maxDimensions);
>> return new DelegatingKnnVectorsFormat(delegate, maxDimensions);
>> }
>> };
>>
>> I am a little confused re which Codec / Vector Format classes I should use when upgrading to Lucene version 10.2.2
>>
>> Any hints would be much appreciated!
>>
>> Thanks
>>
>> Michael
>>
>>
>> --
>> Uwe SchindlerAchterdiek 19, D-28357 Bremen <https://www.google.com/maps/search/Achterdiek+19,+D-28357+Bremen?entry=gmail&source=g>https://www.thetaphi.de
>> eMail: uwe@thetaphi.de
>>
>>
Re: Upgrade Codec 99 to 101 / 102
Posted by Michael Wechner <mi...@wyona.com>.
Hi Ben
Thanks very much for the insight!
So IIUC it is correct to use Lucene102HnswBinaryQuantizedVectorsFormat()
for float32 and int8, right?
Thanks
Michael
Am 10.09.25 um 17:58 schrieb Benjamin Trent:
> Hey Michael,
>
> Right now it won't quantize byte vectors that are provided. It just
> passes them and treats them like normal.
>
> In the future, we would like to quantize the bytes as well!
>
> On Wed, Sep 10, 2025, 10:56 AM Michael Wechner
> <mi...@wyona.com> wrote:
>
> Hi Uwe
>
> Thanks for your feedback!
>
> I am using now:
>
> private final int maxDimensions =16384;
>
> Codec codec =new Lucene101Codec() {
> @Override public KnnVectorsFormat getKnnVectorsFormatForField(String field) {var delegate =new Lucene102HnswBinaryQuantizedVectorsFormat();
> return new DelegatingKnnVectorsFormat(delegate,maxDimensions);
> }
> };
> return codec;
>
> This seems to work fine, but I do not understand whether there is a difference anymore between float32 and int8 vector values,
> or whetherLucene102HnswBinaryQuantizedVectorsFormat always "quantizes" the values?
>
> Thanks
>
> Michael
>
>
> Am 10.09.25 um 10:19 schrieb Uwe Schindler:
>>
>> Hi,
>>
>> I think the best is to check the source code of default codec of
>> the upgrade version and start from there. I agree, there should
>> possibly a documentation available that gives the default
>> componenets used in the default codec of a given release.
>>
>> Uwe
>>
>> Am 09.09.2025 um 22:49 schrieb Michael Wechner:
>>>
>>> Hi
>>>
>>> With Lucene 9.12.0 I set my own custom max vector dimension using
>>>
>>> Codec codecInt8 = new Lucene99Codec() { @Override public
>>> KnnVectorsFormat getKnnVectorsFormatForField(String field) { var
>>> delegate = new Lucene99HnswScalarQuantizedVectorsFormat();
>>> log.info <http://log.info>("Vector Value Type: int8, Maximum
>>> Vector Dimension: " + maxDimensions); return new
>>> DelegatingKnnVectorsFormat(delegate, maxDimensions); } }; and
>>> Codec codecFloat32 =new Lucene99Codec() {
>>> @Override public KnnVectorsFormat getKnnVectorsFormatForField(String field) {
>>> var delegate =new Lucene99HnswVectorsFormat();
>>> log.info("Vector Value Type: float32, Maximum Vector Dimension: " +maxDimensions);
>>> return new DelegatingKnnVectorsFormat(delegate,maxDimensions);
>>> }
>>> };
>>>
>>> I am a little confused re which Codec / Vector Format classes I should use when upgrading to Lucene version 10.2.2
>>>
>>> Any hints would be much appreciated!
>>>
>>> Thanks
>>>
>>> Michael
>>>
>>>
>> --
>> Uwe Schindler
>> Achterdiek 19, D-28357 Bremen <https://www.google.com/maps/search/Achterdiek+19,+D-28357+Bremen?entry=gmail&source=g>
>> https://www.thetaphi.de
>> eMail:uwe@thetaphi.de
>
Re: Upgrade Codec 99 to 101 / 102
Posted by Benjamin Trent <be...@gmail.com>.
Hey Michael,
Right now it won't quantize byte vectors that are provided. It just passes
them and treats them like normal.
In the future, we would like to quantize the bytes as well!
On Wed, Sep 10, 2025, 10:56 AM Michael Wechner <mi...@wyona.com>
wrote:
> Hi Uwe
>
> Thanks for your feedback!
>
> I am using now:
>
> private final int maxDimensions = 16384;
>
> Codec codec = new Lucene101Codec() {
> @Override public KnnVectorsFormat getKnnVectorsFormatForField(String field) { var delegate = new Lucene102HnswBinaryQuantizedVectorsFormat();
> return new DelegatingKnnVectorsFormat(delegate, maxDimensions);
> }
> };
> return codec;
>
> This seems to work fine, but I do not understand whether there is a difference anymore between float32 and int8 vector values,
> or whether Lucene102HnswBinaryQuantizedVectorsFormat always "quantizes" the values?
>
> Thanks
>
> Michael
>
>
> Am 10.09.25 um 10:19 schrieb Uwe Schindler:
>
> Hi,
>
> I think the best is to check the source code of default codec of the
> upgrade version and start from there. I agree, there should possibly a
> documentation available that gives the default componenets used in the
> default codec of a given release.
>
> Uwe
> Am 09.09.2025 um 22:49 schrieb Michael Wechner:
>
> Hi
>
> With Lucene 9.12.0 I set my own custom max vector dimension using
>
> Codec codecInt8 = new Lucene99Codec() { @Override public KnnVectorsFormat getKnnVectorsFormatForField(String field) { var delegate = new Lucene99HnswScalarQuantizedVectorsFormat(); log.info("Vector Value Type: int8, Maximum Vector Dimension: " + maxDimensions); return new DelegatingKnnVectorsFormat(delegate, maxDimensions); }};
>
> and
>
> Codec codecFloat32 = new Lucene99Codec() {
> @Override public KnnVectorsFormat getKnnVectorsFormatForField(String field) {
> var delegate = new Lucene99HnswVectorsFormat();
> log.info("Vector Value Type: float32, Maximum Vector Dimension: " + maxDimensions);
> return new DelegatingKnnVectorsFormat(delegate, maxDimensions);
> }
> };
>
> I am a little confused re which Codec / Vector Format classes I should use when upgrading to Lucene version 10.2.2
>
> Any hints would be much appreciated!
>
> Thanks
>
> Michael
>
>
> --
> Uwe SchindlerAchterdiek 19, D-28357 Bremen <https://www.google.com/maps/search/Achterdiek+19,+D-28357+Bremen?entry=gmail&source=g>https://www.thetaphi.de
> eMail: uwe@thetaphi.de
>
>
Re: Upgrade Codec 99 to 101 / 102
Posted by Michael Wechner <mi...@wyona.com>.
Hi Uwe
Thanks for your feedback!
I am using now:
private final int maxDimensions =16384;
Codec codec =new Lucene101Codec() {
@Override public KnnVectorsFormat getKnnVectorsFormatForField(String field) {var delegate =new Lucene102HnswBinaryQuantizedVectorsFormat();
return new DelegatingKnnVectorsFormat(delegate,maxDimensions);
}
};
return codec;
This seems to work fine, but I do not understand whether there is a difference anymore between float32 and int8 vector values,
or whetherLucene102HnswBinaryQuantizedVectorsFormat always "quantizes" the values?
Thanks
Michael
Am 10.09.25 um 10:19 schrieb Uwe Schindler:
>
> Hi,
>
> I think the best is to check the source code of default codec of the
> upgrade version and start from there. I agree, there should possibly a
> documentation available that gives the default componenets used in the
> default codec of a given release.
>
> Uwe
>
> Am 09.09.2025 um 22:49 schrieb Michael Wechner:
>>
>> Hi
>>
>> With Lucene 9.12.0 I set my own custom max vector dimension using
>>
>> Codec codecInt8 = new Lucene99Codec() { @Override public
>> KnnVectorsFormat getKnnVectorsFormatForField(String field) { var
>> delegate = new Lucene99HnswScalarQuantizedVectorsFormat();
>> log.info("Vector Value Type: int8, Maximum Vector Dimension: " +
>> maxDimensions); return new DelegatingKnnVectorsFormat(delegate,
>> maxDimensions); } }; and
>> Codec codecFloat32 =new Lucene99Codec() {
>> @Override public KnnVectorsFormat getKnnVectorsFormatForField(String field) {
>> var delegate =new Lucene99HnswVectorsFormat();
>> log.info("Vector Value Type: float32, Maximum Vector Dimension: " +maxDimensions);
>> return new DelegatingKnnVectorsFormat(delegate,maxDimensions);
>> }
>> };
>>
>> I am a little confused re which Codec / Vector Format classes I should use when upgrading to Lucene version 10.2.2
>>
>> Any hints would be much appreciated!
>>
>> Thanks
>>
>> Michael
>>
>>
> --
> Uwe Schindler
> Achterdiek 19, D-28357 Bremen
> https://www.thetaphi.de
> eMail:uwe@thetaphi.de
Re: Upgrade Codec 99 to 101 / 102
Posted by Uwe Schindler <uw...@thetaphi.de>.
Hi,
I think the best is to check the source code of default codec of the
upgrade version and start from there. I agree, there should possibly a
documentation available that gives the default componenets used in the
default codec of a given release.
Uwe
Am 09.09.2025 um 22:49 schrieb Michael Wechner:
>
> Hi
>
> With Lucene 9.12.0 I set my own custom max vector dimension using
>
> Codec codecInt8 = new Lucene99Codec() { @Override public
> KnnVectorsFormat getKnnVectorsFormatForField(String field) { var
> delegate = new Lucene99HnswScalarQuantizedVectorsFormat();
> log.info("Vector Value Type: int8, Maximum Vector Dimension: " +
> maxDimensions); return new DelegatingKnnVectorsFormat(delegate,
> maxDimensions); } }; and
> Codec codecFloat32 =new Lucene99Codec() {
> @Override public KnnVectorsFormat getKnnVectorsFormatForField(String field) {
> var delegate =new Lucene99HnswVectorsFormat();
> log.info("Vector Value Type: float32, Maximum Vector Dimension: " +maxDimensions);
> return new DelegatingKnnVectorsFormat(delegate,maxDimensions);
> }
> };
>
> I am a little confused re which Codec / Vector Format classes I should use when upgrading to Lucene version 10.2.2
>
> Any hints would be much appreciated!
>
> Thanks
>
> Michael
>
>
--
Uwe Schindler
Achterdiek 19, D-28357 Bremen
https://www.thetaphi.de
eMail:uwe@thetaphi.de