You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@lucene.apache.org by "mccullocht (via GitHub)" <gi...@apache.org> on 2025/09/02 18:59:39 UTC
[PR] Add support for uint8 distance comparison [lucene]
mccullocht opened a new pull request, #15148:
URL: https://github.com/apache/lucene/pull/15148
Add distance functions that treat `byte[]` as unsigned and use them in `ScalarQuantizer` code paths. `ScalarQuantizer`
assumes that all math will be unsigned but this can't be true when all 8 bits of a `byte` are used. In all cases where this
is used values are widened to `short` or `int` before any operations occur so we can perform unsigned widening instead.
This will be useful for supporting 8 bit OSQ in #15064
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
Re: [PR] Add support for uint8 distance comparison [lucene]
Posted by "mccullocht (via GitHub)" <gi...@apache.org>.
mccullocht commented on PR #15148:
URL: https://github.com/apache/lucene/pull/15148#issuecomment-3254417814
Added to CHANGES for 11. This would probably backport to 10.4 just fine but I also don't know if 10.4 will be happening.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
Re: [PR] Add support for uint8 distance comparison [lucene]
Posted by "mccullocht (via GitHub)" <gi...@apache.org>.
mccullocht commented on PR #15148:
URL: https://github.com/apache/lucene/pull/15148#issuecomment-3258657661
I think this is ready to merge
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
Re: [PR] Add support for uint8 distance comparison [lucene]
Posted by "mccullocht (via GitHub)" <gi...@apache.org>.
mccullocht commented on PR #15148:
URL: https://github.com/apache/lucene/pull/15148#issuecomment-3246637241
I suspect that this won't be slower in the scalar path either, as IIUC java will implicitly widen to `int` on any arithmetic operation and processors will typically have both signed and unsigned widen. The `boolean signed` is deliberately not part of any public API to minimize confusion. Hopefully the JVM uses this to monomorphize the functions.
I'm getting 403s downloading the luceneutil testdata today but hopefully I'll be able to get some numbers soon.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
Re: [PR] Add support for uint8 distance comparison [lucene]
Posted by "benwtrent (via GitHub)" <gi...@apache.org>.
benwtrent commented on PR #15148:
URL: https://github.com/apache/lucene/pull/15148#issuecomment-3255407842
I would assume there would be a 10.4. Even if its to shore things up before another major. We didn't really nail down if there would be another major before 10.3 was cut.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
Re: [PR] Add support for uint8 distance comparison [lucene]
Posted by "mccullocht (via GitHub)" <gi...@apache.org>.
mccullocht commented on PR #15148:
URL: https://github.com/apache/lucene/pull/15148#issuecomment-3254922424
I tried picking this change into branch_10x and there's only a minor conflict in `PanamaVectorUtilSupport` so I've moved it to 10.4 in CHANGES
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
Re: [PR] Add support for uint8 distance comparison [lucene]
Posted by "benwtrent (via GitHub)" <gi...@apache.org>.
benwtrent commented on PR #15148:
URL: https://github.com/apache/lucene/pull/15148#issuecomment-3246521768
Ah, interesting! For sure, with panama vector enabled, I would expect to see zero slow down in Lucene.
Looking forward to benchmarks :).
One tricky part would be that we don't mix up the `signed bytes` that a user could provide and the `unsigned byte` that we would gather from quantization.
I think you have this handled with the `signed` boolean you are passing around. Just wanting to call out the concern.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
Re: [PR] Add support for uint8 distance comparison [lucene]
Posted by "benwtrent (via GitHub)" <gi...@apache.org>.
benwtrent commented on PR #15148:
URL: https://github.com/apache/lucene/pull/15148#issuecomment-3258701377
I agree @mccullocht I can merge and back port. Thank you for digging into this :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
Re: [PR] Add support for uint8 distance comparison [lucene]
Posted by "benwtrent (via GitHub)" <gi...@apache.org>.
benwtrent merged PR #15148:
URL: https://github.com/apache/lucene/pull/15148
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org