You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@lucene.apache.org by "mikemccand (via GitHub)" <gi...@apache.org> on 2024/10/08 23:59:45 UTC
[PR] Fix 9.12.0 backcompat break (Lucene 9.12.0 cannot read 9.11.x indices written with quantized HNSW, `Lucene99HnswScalarQuantizedVectorsFormat`) [lucene]
mikemccand opened a new pull request, #13874:
URL: https://github.com/apache/lucene/pull/13874
This PR:
1. Fixes a pre-existing BWC testing bug where our `int8_hnsw*.zip` bwc test files failed to actually use the scalar quantization codec ... they were just ordinary `float32` HNSW since they were using Lucene's default codec to write the bwc indices. This is why no bwc tests failed with my original PR (#13767), sigh. I also renamed these files and the test class itself to `int7*` since they are actually all 7 bit not 8 bit quantization.
1. Regenerates the bwc indices for 9.10.0, 9.11.0, and 9.11.1 (the binary zip files). This entailed checking out each release tag for these releases, backporting the fix above, regenerating the zip files and copying them into this PR. It was a bit tricky for 9.10.0 ([see details here](https://github.com/apache/lucene/issues/13867#issuecomment-2400927997)).
1. Fixes the actual bug from #13767 using the suggested fix from @parnmatt (thank you!). The bug happened because scalar quantized HNSW indices written with 9.11.0 or 9.11.1 claimed `compress=true` (written into the index) yet vector compression is silently `false` unless `bits` is 7 or 8. This trap is fixed for newly created indices in 9.12.0, but 9.11.x indices bake in the trap.
I confirmed that after the 2nd step above, the bwc test was indeed failing properly (because of the #13867 bug), and then upon applying the fix in the third step above, bwc tests (and all tests) now pass.
Thank you @neio-ionut for discovering and opening this issue and @parnmatt and @benwtrent for digging and proposing the root cause fix.
Closes #13867
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
Re: [PR] Fix 9.12.0 backcompat break (Lucene 9.12.0 cannot read 9.11.x indices written with quantized HNSW, `Lucene99HnswScalarQuantizedVectorsFormat`) [lucene]
Posted by "mikemccand (via GitHub)" <gi...@apache.org>.
mikemccand commented on PR #13874:
URL: https://github.com/apache/lucene/pull/13874#issuecomment-2402140838
> I can open the PR -- I still have the dev area where I got it working and generated the bwc `.zip` in this PR.
OK I opened https://github.com/apache/lucene/pull/13879 -- this is just for 9.10.x.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
Re: [PR] Fix 9.12.0 backcompat break (Lucene 9.12.0 cannot read 9.11.x indices written with quantized HNSW, `Lucene99HnswScalarQuantizedVectorsFormat`) [lucene]
Posted by "ChrisHegarty (via GitHub)" <gi...@apache.org>.
ChrisHegarty commented on PR #13874:
URL: https://github.com/apache/lucene/pull/13874#issuecomment-2402235046
I wonder if the simplest thing here is to just avoid the renaming (from int8 to int7) in 9.x. Otherwise, I don't see how to get `addBackcompatIndexes.py` to work - since 9.12.0 source bundled is released with the int8 names.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
Re: [PR] Fix 9.12.0 backcompat break (Lucene 9.12.0 cannot read 9.11.x indices written with quantized HNSW, `Lucene99HnswScalarQuantizedVectorsFormat`) [lucene]
Posted by "mikemccand (via GitHub)" <gi...@apache.org>.
mikemccand commented on PR #13874:
URL: https://github.com/apache/lucene/pull/13874#issuecomment-2403477991
> Oh! Is there an int8_hnsw.9.12.0.zip that can be deleted?
D'oh! I'll delete.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
Re: [PR] Fix 9.12.0 backcompat break (Lucene 9.12.0 cannot read 9.11.x indices written with quantized HNSW, `Lucene99HnswScalarQuantizedVectorsFormat`) [lucene]
Posted by "mikemccand (via GitHub)" <gi...@apache.org>.
mikemccand commented on PR #13874:
URL: https://github.com/apache/lucene/pull/13874#issuecomment-2403377531
> I was not expecting to see changes in `index.9.12.0-cfs.zip` or `index.9.12.0-no.cfs.zip`. Do you know why these are changed?
Net/net it's because I regen'd all 9.12.0 bwc zip indices ... and each time you gen a Lucene index it'll be different, at least because of the GUID we assign to each segment.
I could revert them -- that would be safer. Lemme go do that.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
Re: [PR] Fix 9.12.0 backcompat break (Lucene 9.12.0 cannot read 9.11.x indices written with quantized HNSW, `Lucene99HnswScalarQuantizedVectorsFormat`) [lucene]
Posted by "mikemccand (via GitHub)" <gi...@apache.org>.
mikemccand commented on PR #13874:
URL: https://github.com/apache/lucene/pull/13874#issuecomment-2403490011
OK I'll merge shortly once GitHub action is happy! Thanks everyone, phew!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
Re: [PR] Fix 9.12.0 backcompat break (Lucene 9.12.0 cannot read 9.11.x indices written with quantized HNSW, `Lucene99HnswScalarQuantizedVectorsFormat`) [lucene]
Posted by "mikemccand (via GitHub)" <gi...@apache.org>.
mikemccand commented on PR #13874:
URL: https://github.com/apache/lucene/pull/13874#issuecomment-2402095420
> I'm try to determine why a bwc test fails in the CI.
Thanks @ChrisHegarty.
Can you point me to the CI build that is angry? I assume you set up a 9.12.x CI build somewhere, and maybe applied this PR? Why don't I see these failures locally when running `./gradlew check`? Confused...
> I guess that means we need to generate 9.12.0 indices for the hnsw bwc. You are correct @ChrisHegarty .
Ahhh OK I will generate 9.12.0 bwc zip files and add to this PR! In fact, I should be able to simply copy down from 10.x, except regenerate the `int7_hnsw.9.12.0.zip` with the bug fix from this PR...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
Re: [PR] Fix 9.12.0 backcompat break (Lucene 9.12.0 cannot read 9.11.x indices written with quantized HNSW, `Lucene99HnswScalarQuantizedVectorsFormat`) [lucene]
Posted by "mikemccand (via GitHub)" <gi...@apache.org>.
mikemccand commented on PR #13874:
URL: https://github.com/apache/lucene/pull/13874#issuecomment-2402825006
> > Hmm the problem is, we need to regen the 9.12.0 `int8_hnsw` back compat index using the bug fix from this PR (so that it actually has `int7` quantized vectors, not unquantized `float32`)? So I think I must manually (carefully) do this?
>
> Yes. I did think similar too, then fell into the trap of trying to fix the script, and forgot!!!. You're right. The script is basically borked for this, and unfixable in its current form. In which case, then maybe keep the renames, since it'll avoid the trap I feel into.
Ack, I'll re-revert the revert. Thanks @ChrisHegarty.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
Re: [PR] Fix 9.12.0 backcompat break (Lucene 9.12.0 cannot read 9.11.x indices written with quantized HNSW, `Lucene99HnswScalarQuantizedVectorsFormat`) [lucene]
Posted by "mikemccand (via GitHub)" <gi...@apache.org>.
mikemccand commented on PR #13874:
URL: https://github.com/apache/lucene/pull/13874#issuecomment-2403616445
OK I pushed 10.0.x fix ... working on 10.x now. Gradle sees the same weird deadlock when I try to `clean` and `check` in one go. So weird.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
Re: [PR] Fix 9.12.0 backcompat break (Lucene 9.12.0 cannot read 9.11.x indices written with quantized HNSW, `Lucene99HnswScalarQuantizedVectorsFormat`) [lucene]
Posted by "benwtrent (via GitHub)" <gi...@apache.org>.
benwtrent commented on PR #13874:
URL: https://github.com/apache/lucene/pull/13874#issuecomment-2402052859
Ah, I see 9.12.0 is already assumed by the bwc indices! I guess that means we need to generate 9.12.0 indices for the hnsw bwc. You are correct @ChrisHegarty .
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
Re: [PR] Fix 9.12.0 backcompat break (Lucene 9.12.0 cannot read 9.11.x indices written with quantized HNSW, `Lucene99HnswScalarQuantizedVectorsFormat`) [lucene]
Posted by "ChrisHegarty (via GitHub)" <gi...@apache.org>.
ChrisHegarty commented on PR #13874:
URL: https://github.com/apache/lucene/pull/13874#issuecomment-2402227085
Argh!! this is very odd. We cannot just update the addBackcompatIndexes.py script. Since it downloads the 9.12.0 version and tries to run gradle on it!! and of course `testCreateInt7HNSWIndices` does not exist there!! :-(
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
Re: [PR] Fix 9.12.0 backcompat break (Lucene 9.12.0 cannot read 9.11.x indices written with quantized HNSW, `Lucene99HnswScalarQuantizedVectorsFormat`) [lucene]
Posted by "ChrisHegarty (via GitHub)" <gi...@apache.org>.
ChrisHegarty commented on PR #13874:
URL: https://github.com/apache/lucene/pull/13874#issuecomment-2404222904
> if I separately run `./gradlew clean` and then `./gradlew check` then it's fine ... weird.
We’ve seen similar recently, reported in #13567. Likely a grade bug, which we’ve worked around as you have, by running clean separately.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
Re: [PR] Fix 9.12.0 backcompat break (Lucene 9.12.0 cannot read 9.11.x indices written with quantized HNSW, `Lucene99HnswScalarQuantizedVectorsFormat`) [lucene]
Posted by "benwtrent (via GitHub)" <gi...@apache.org>.
benwtrent commented on PR #13874:
URL: https://github.com/apache/lucene/pull/13874#issuecomment-2402094834
@mikemccand I can open PRs to the old versions to add the code. I did the same "copy files and run it" dance that you did because I didn't want to commit code to the old branches. But now I see it would have just been way better to do that :/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
Re: [PR] Fix 9.12.0 backcompat break (Lucene 9.12.0 cannot read 9.11.x indices written with quantized HNSW, `Lucene99HnswScalarQuantizedVectorsFormat`) [lucene]
Posted by "ChrisHegarty (via GitHub)" <gi...@apache.org>.
ChrisHegarty commented on PR #13874:
URL: https://github.com/apache/lucene/pull/13874#issuecomment-2403360433
I was not expecting to see changes in `index.9.12.0-cfs.zip` or `index.9.12.0-no.cfs.zip`. Do you know why these are changed?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
Re: [PR] Fix 9.12.0 backcompat break (Lucene 9.12.0 cannot read 9.11.x indices written with quantized HNSW, `Lucene99HnswScalarQuantizedVectorsFormat`) [lucene]
Posted by "javanna (via GitHub)" <gi...@apache.org>.
javanna commented on PR #13874:
URL: https://github.com/apache/lucene/pull/13874#issuecomment-2412365840
I did find above a diff that is exactly the same as the PR I opened to update AddBackcompatindices :) (#13911) . I also opened a PR against main to cherry-pick this change and make the necessary adjustments: #13912
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
Re: [PR] Fix 9.12.0 backcompat break (Lucene 9.12.0 cannot read 9.11.x indices written with quantized HNSW, `Lucene99HnswScalarQuantizedVectorsFormat`) [lucene]
Posted by "mikemccand (via GitHub)" <gi...@apache.org>.
mikemccand commented on PR #13874:
URL: https://github.com/apache/lucene/pull/13874#issuecomment-2403632181
I cherry-pick'd to 10.x as well. Let's see if CI builds are happy ... if so, I think this is done!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
Re: [PR] Fix 9.12.0 backcompat break (Lucene 9.12.0 cannot read 9.11.x indices written with quantized HNSW, `Lucene99HnswScalarQuantizedVectorsFormat`) [lucene]
Posted by "mikemccand (via GitHub)" <gi...@apache.org>.
mikemccand commented on PR #13874:
URL: https://github.com/apache/lucene/pull/13874#issuecomment-2403082537
> > Ack, I'll re-revert the revert. Thanks @ChrisHegarty.
>
> Thanks. And sorry for the bad suggestion.
No worries!
> So, I think once the 9.12.0 int7 hnsw bwc file is created, then this PR is ready to merge. I guess initially to `branch_9_12`, then forward port to `branch_10_0`. @mikemccand I don't wanna get in ur way, but lemme know if there is anything I can do to help.
Yes that is the plan! I'm running `./gradlew -p lucene/backward-codecs test --tests TestGenerateBwcIndices -Dtests.bwcdir=/l/tmp -Ptests.useSecurityManager=false` to generate the 9.12.0 bwc zip manually, then I'll add/test/push. I think we are close for 9.12.x.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
Re: [PR] Fix 9.12.0 backcompat break (Lucene 9.12.0 cannot read 9.11.x indices written with quantized HNSW, `Lucene99HnswScalarQuantizedVectorsFormat`) [lucene]
Posted by "mikemccand (via GitHub)" <gi...@apache.org>.
mikemccand commented on PR #13874:
URL: https://github.com/apache/lucene/pull/13874#issuecomment-2402071452
> Thank you for fixing my bwc mess! It was definitely a learning experience and apparently I didn't learn enough!
No worries @benwtrent -- it is crazy complicated and trappy! I wish we had better automation here (check out release tag, build the bwc index, copy zip files to newer release, etc.). I will open a spinoff issue...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
Re: [PR] Fix 9.12.0 backcompat break (Lucene 9.12.0 cannot read 9.11.x indices written with quantized HNSW, `Lucene99HnswScalarQuantizedVectorsFormat`) [lucene]
Posted by "ChrisHegarty (via GitHub)" <gi...@apache.org>.
ChrisHegarty commented on PR #13874:
URL: https://github.com/apache/lucene/pull/13874#issuecomment-2403417121
Oh! Is there an int8_hnsw.9.12.0.zip that can be deleted?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
Re: [PR] Fix 9.12.0 backcompat break (Lucene 9.12.0 cannot read 9.11.x indices written with quantized HNSW, `Lucene99HnswScalarQuantizedVectorsFormat`) [lucene]
Posted by "mikemccand (via GitHub)" <gi...@apache.org>.
mikemccand commented on PR #13874:
URL: https://github.com/apache/lucene/pull/13874#issuecomment-2403494337
> I wish we had better automation here (check out release tag, build the bwc index, copy zip files to newer release, etc.). I will open a spinoff issue...
Actually, this nice `dev-tools/scripts/addBackcompatIndices.py` is exactly the automation I was thinking of! It just gets tricky in the "who tests the tester" scenario ... when we fix a bug in the bwc index generation and then need to regen from tip of a git branch and not the actual release bits, I guess.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
Re: [PR] Fix 9.12.0 backcompat break (Lucene 9.12.0 cannot read 9.11.x indices written with quantized HNSW, `Lucene99HnswScalarQuantizedVectorsFormat`) [lucene]
Posted by "mikemccand (via GitHub)" <gi...@apache.org>.
mikemccand commented on PR #13874:
URL: https://github.com/apache/lucene/pull/13874#issuecomment-2402425950
> I wonder if the simplest thing here is to just avoid the renaming (from int8 to int7) in 9.x. Otherwise, I don't see how to get `addBackcompatIndexes.py` to work - since 9.12.0 source bundled is released with the int8 names.
Argh! Write once naming! A one-way door, sigh.
OK, I will revert that rename.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
Re: [PR] Fix 9.12.0 backcompat break (Lucene 9.12.0 cannot read 9.11.x indices written with quantized HNSW, `Lucene99HnswScalarQuantizedVectorsFormat`) [lucene]
Posted by "benwtrent (via GitHub)" <gi...@apache.org>.
benwtrent commented on PR #13874:
URL: https://github.com/apache/lucene/pull/13874#issuecomment-2402047972
I think you have to add it as a backcompat index in another file, but we don't do that until 9.12.1 is cut and released! Since 9.12.1 hasn't gone through the release process, according to backcompat stuff, it doesn't exist.
So, for 9.12, we should add the other back compat index files, but not 9.12.0 until 9.12.1 has started the release process (then they will be added via that process).
@ChrisHegarty @mikemccand
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
Re: [PR] Fix 9.12.0 backcompat break (Lucene 9.12.0 cannot read 9.11.x indices written with quantized HNSW, `Lucene99HnswScalarQuantizedVectorsFormat`) [lucene]
Posted by "ChrisHegarty (via GitHub)" <gi...@apache.org>.
ChrisHegarty commented on PR #13874:
URL: https://github.com/apache/lucene/pull/13874#issuecomment-2402700651
> Hmm the problem is, we need to regen the 9.12.0 `int8_hnsw` back compat index using the bug fix from this PR (so that it actually has `int7` quantized vectors, not unquantized `float32`)? So I think I must manually (carefully) do this?
Yes. I did think similar too, then fell into the trap of trying to fix the script, and forgot!!!. You're right. The script is basically borked for this, and unfixable in its current form. In which case, then maybe keep the renames, since it'll avoid the trap I feel into.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
Re: [PR] Fix 9.12.0 backcompat break (Lucene 9.12.0 cannot read 9.11.x indices written with quantized HNSW, `Lucene99HnswScalarQuantizedVectorsFormat`) [lucene]
Posted by "ChrisHegarty (via GitHub)" <gi...@apache.org>.
ChrisHegarty commented on PR #13874:
URL: https://github.com/apache/lucene/pull/13874#issuecomment-2402218226
if u merge in `branch_9_12`, then `dev-tools/scripts/addBackcompatIndexes.py 9.12.0` should be able to generate the compat indices.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
Re: [PR] Fix 9.12.0 backcompat break (Lucene 9.12.0 cannot read 9.11.x indices written with quantized HNSW, `Lucene99HnswScalarQuantizedVectorsFormat`) [lucene]
Posted by "mikemccand (via GitHub)" <gi...@apache.org>.
mikemccand merged PR #13874:
URL: https://github.com/apache/lucene/pull/13874
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
Re: [PR] Fix 9.12.0 backcompat break (Lucene 9.12.0 cannot read 9.11.x indices written with quantized HNSW, `Lucene99HnswScalarQuantizedVectorsFormat`) [lucene]
Posted by "mikemccand (via GitHub)" <gi...@apache.org>.
mikemccand commented on PR #13874:
URL: https://github.com/apache/lucene/pull/13874#issuecomment-2402102296
> @mikemccand I can open PRs to the old versions to add the code. I did the same "copy files and run it" dance that you did because I didn't want to commit code to the old branches. But now I see it would have just been way better to do that :/
OK phew I'm glad I wasn't losing my mind. Yeah, let's get the change (to generate the BWC indices) onto `branch_9_10`? I can open the PR -- I still have the dev area where I got it working and generated the bwc `.zip` in this PR. Once I open that PR, can you confirm it matches how you did it?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
Re: [PR] Fix 9.12.0 backcompat break (Lucene 9.12.0 cannot read 9.11.x indices written with quantized HNSW, `Lucene99HnswScalarQuantizedVectorsFormat`) [lucene]
Posted by "mikemccand (via GitHub)" <gi...@apache.org>.
mikemccand commented on PR #13874:
URL: https://github.com/apache/lucene/pull/13874#issuecomment-2403538869
Although, strangely, when I run `./gradlew clean check` I get this very odd (never seen this before) failure:
```
Unable to make progress running work. The following items are queued for execution but none of them can be \
started:
- Build ':':
- Waiting for nodes:
- :lucene:documentation:check (state=SHOULD_RUN, dependencies=NOT_COMPLETE, group=task group 1, d\
ependencies=[:lucene:documentation:checkBrokenLinks (SHOULD_RUN), :lucene:documentation:licenses (EXECUTED)\
, :lucene:documentation:validateSourcePatterns (SHOULD_RUN)], waiting-for=[:lucene:documentation:checkBroke\
nLinks (SHOULD_RUN), :lucene:documentation:validateSourcePatterns (SHOULD_RUN)], has-failed-dependency=fals\
e )
- :lucene:test-framework:check (state=SHOULD_RUN, dependencies=NOT_COMPLETE, group=task group 1, \
dependencies=[:lucene:test-framework:ecjLint (SHOULD_RUN), :lucene:test-framework:forbiddenApis (SHOULD_RUN\
), :lucene:test-framework:generateEmojiTokenizationTestChecksumCheck (EXECUTED), :lucene:test-framework:gen\
erateJavaUnicodeWordBreakTestChecksumCheck (EXECUTED), :lucene:test-framework:licenses (SHOULD_RUN), :lucen\
e:test-framework:spotlessCheck (SHOULD_RUN), :lucene:test-framework:test (SHOULD_RUN), :lucene:test-framewo\
rk:validateSourcePatterns (SHOULD_RUN)], waiting-for=[:lucene:test-framework:spotlessCheck (SHOULD_RUN), :l\
ucene:test-framework:test (SHOULD_RUN), :lucene:test-framework:ecjLint (SHOULD_RUN), :lucene:test-framework\
:forbiddenApis (SHOULD_RUN), :lucene:test-framework:validateSourcePatterns (SHOULD_RUN), :lucene:test-frame\
work:licenses (SHOULD_RUN)], has-failed-dependency=false )
```
The gradle build then fails having tons of thinsg to do but none of them actionable/schedulable? Weird.
if I separately run `./gradlew clean` and then `./gradlew check` then it's fine ... weird. I'm using Java 22.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
Re: [PR] Fix 9.12.0 backcompat break (Lucene 9.12.0 cannot read 9.11.x indices written with quantized HNSW, `Lucene99HnswScalarQuantizedVectorsFormat`) [lucene]
Posted by "mikemccand (via GitHub)" <gi...@apache.org>.
mikemccand commented on PR #13874:
URL: https://github.com/apache/lucene/pull/13874#issuecomment-2403535561
The `git cherry-pick` of the 9.12.x commit merged cleanly onto 10.0.x branch ... and `gradle check` is happy, so I plan to push that cherry-pick to 10.0.x branch (and 10.x after that) soon. Any concerns?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
Re: [PR] Fix 9.12.0 backcompat break (Lucene 9.12.0 cannot read 9.11.x indices written with quantized HNSW, `Lucene99HnswScalarQuantizedVectorsFormat`) [lucene]
Posted by "mikemccand (via GitHub)" <gi...@apache.org>.
mikemccand commented on PR #13874:
URL: https://github.com/apache/lucene/pull/13874#issuecomment-2403280987
OK phew I think I resolved the conflicts due to stale branch, and tests are passing for me. I think this is ready? Let's let GitHub actions chew on it to confirm...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
Re: [PR] Fix 9.12.0 backcompat break (Lucene 9.12.0 cannot read 9.11.x indices written with quantized HNSW, `Lucene99HnswScalarQuantizedVectorsFormat`) [lucene]
Posted by "ChrisHegarty (via GitHub)" <gi...@apache.org>.
ChrisHegarty commented on PR #13874:
URL: https://github.com/apache/lucene/pull/13874#issuecomment-2413575785
> * I think that we need to forward port this change to main as well?
Yes, it should be forward ported.
> * We need to update the tooling to generate the backwards indices accordingly (python code and java code are no longer in sync)
Yes, these should be aligned.
Thanks,
-Chris.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
Re: [PR] Fix 9.12.0 backcompat break (Lucene 9.12.0 cannot read 9.11.x indices written with quantized HNSW, `Lucene99HnswScalarQuantizedVectorsFormat`) [lucene]
Posted by "javanna (via GitHub)" <gi...@apache.org>.
javanna commented on PR #13874:
URL: https://github.com/apache/lucene/pull/13874#issuecomment-2412274304
Heya, I am working on generating the backwards indices after releasing Lucene 10, and here are my observations:
- I think that we need to forward port this change to main as well?
- We need to update the tooling to generate the backwards indices accordingly (python code and java code are no longer in sync)
I am happy to work on this, I think I understand what needs to happen but I wanted to confirm first. Perhaps the forward port is just a cherry-pick to main, hopefully? And once the branches are aligned, I can adjust the tooling and test them direclty as I generate the needed bwc indices for 10.0.0.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
Re: [PR] Fix 9.12.0 backcompat break (Lucene 9.12.0 cannot read 9.11.x indices written with quantized HNSW, `Lucene99HnswScalarQuantizedVectorsFormat`) [lucene]
Posted by "mikemccand (via GitHub)" <gi...@apache.org>.
mikemccand commented on PR #13874:
URL: https://github.com/apache/lucene/pull/13874#issuecomment-2402452969
> if u merge in `branch_9_12`, then `dev-tools/scripts/addBackcompatIndexes.py 9.12.0` should be able to generate the compat indices.
Hmm the problem is, we need to regen the 9.12.0 `int8_hnsw` back compat index using the bug fix from this PR (so that it actually has `int7` quantized vectors, not unquantized `float32`)? So I think I must manually (carefully) do this? I would also like to add a test case that confirms that the `int8_hnsw` zip file indices do in fact use the quantized codec format ("who tests the tester?") ... but that can come later ... I'll open another spinoff.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
Re: [PR] Fix 9.12.0 backcompat break (Lucene 9.12.0 cannot read 9.11.x indices written with quantized HNSW, `Lucene99HnswScalarQuantizedVectorsFormat`) [lucene]
Posted by "ChrisHegarty (via GitHub)" <gi...@apache.org>.
ChrisHegarty commented on PR #13874:
URL: https://github.com/apache/lucene/pull/13874#issuecomment-2402848199
> Ack, I'll re-revert the revert. Thanks @ChrisHegarty.
Thanks. And sorry for the bad suggestion.
So, I think once the 9.12.0 int7 hnsw bwc file is created, then this PR is ready to merge. I guess initially to `branch_9_12`, then forward port to `branch_10_0`. @mikemccand I don't wanna get in ur way, but lemme know if there is anything I can do to help.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
Re: [PR] Fix 9.12.0 backcompat break (Lucene 9.12.0 cannot read 9.11.x indices written with quantized HNSW, `Lucene99HnswScalarQuantizedVectorsFormat`) [lucene]
Posted by "mikemccand (via GitHub)" <gi...@apache.org>.
mikemccand commented on PR #13874:
URL: https://github.com/apache/lucene/pull/13874#issuecomment-2403250756
Woops, I see my local `branch_9_12` (and where I branched my `fix_back_compat` branch from for this PR) was out of date -- `branch_9_12` had already bumped to 9.12.1 and added bwc indices. Lemme attempt to merge from its tip and resolve these conflicts ...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
Re: [PR] Fix 9.12.0 backcompat break (Lucene 9.12.0 cannot read 9.11.x indices written with quantized HNSW, `Lucene99HnswScalarQuantizedVectorsFormat`) [lucene]
Posted by "ChrisHegarty (via GitHub)" <gi...@apache.org>.
ChrisHegarty commented on PR #13874:
URL: https://github.com/apache/lucene/pull/13874#issuecomment-2401798286
I think that this is a good change. I'm try to determine why a bwc test fails in the CI.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
Re: [PR] Fix 9.12.0 backcompat break (Lucene 9.12.0 cannot read 9.11.x indices written with quantized HNSW, `Lucene99HnswScalarQuantizedVectorsFormat`) [lucene]
Posted by "ChrisHegarty (via GitHub)" <gi...@apache.org>.
ChrisHegarty commented on PR #13874:
URL: https://github.com/apache/lucene/pull/13874#issuecomment-2402214908
we need to update `dev-tools/scripts/addBackcompatIndexes.py`.
```diff
diff --git a/dev-tools/scripts/addBackcompatIndexes.py b/dev-tools/scripts/addBackcompatIndexes.py
index 80272ec0f0c..3056c8268d4 100755
--- a/dev-tools/scripts/addBackcompatIndexes.py
+++ b/dev-tools/scripts/addBackcompatIndexes.py
@@ -40,7 +40,7 @@ def create_and_add_index(source, indextype, index_version, current_version, temp
'cfs': 'index',
'nocfs': 'index',
'sorted': 'sorted',
- 'int8_hnsw': 'int8_hnsw',
+ 'int7_hnsw': 'int7_hnsw',
'moreterms': 'moreterms',
'dvupdates': 'dvupdates',
'emptyIndex': 'empty'
@@ -61,7 +61,7 @@ def create_and_add_index(source, indextype, index_version, current_version, temp
'cfs': 'testCreateCFS',
'nocfs': 'testCreateNoCFS',
'sorted': 'testCreateSortedIndex',
- 'int8_hnsw': 'testCreateInt8HNSWIndices',
+ 'int7_hnsw': 'testCreateInt7HNSWIndices',
'moreterms': 'testCreateMoreTermsIndex',
'dvupdates': 'testCreateIndexWithDocValuesUpdates',
'emptyIndex': 'testCreateEmptyIndex'
@@ -206,7 +206,7 @@ def main():
current_version = scriptutil.Version.parse(scriptutil.find_current_version())
create_and_add_index(source, 'cfs', c.version, current_version, c.temp_dir)
create_and_add_index(source, 'nocfs', c.version, current_version, c.temp_dir)
- create_and_add_index(source, 'int8_hnsw', c.version, current_version, c.temp_dir)
+ create_and_add_index(source, 'int7_hnsw', c.version, current_version, c.temp_dir)
should_make_sorted = current_version.is_back_compat_with(c.version) \
and (c.version.major > 6 or (c.version.major == 6 and c.version.minor >= 2))
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org