You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@lucene.apache.org by GitBox <gi...@apache.org> on 2022/11/17 01:30:40 UTC
[GitHub] [lucene] jdconrad opened a new pull request, #11945: Decrease test time for TestManyKnnDocs.testLargeSegment
jdconrad opened a new pull request, #11945:
URL: https://github.com/apache/lucene/pull/11945
This change adds an additional test codec allowing a configurable number for max connections per vector when building an hnsw index. By setting the number of connections to `128` as part of `TestManyKnnDocs.testLargeSegment` we can reduce the number of indexed vectors to `2088992` and still reproduce the test failure prior to the fix by @benwtrent in https://github.com/apache/lucene/pull/11905.
This changed reduced the test time for me from ~90 minutes to ~3 minutes locally.
cc @rmuir
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
[GitHub] [lucene] rmuir commented on pull request #11945: Decrease test time for TestManyKnnDocs.testLargeSegment
Posted by GitBox <gi...@apache.org>.
rmuir commented on PR #11945:
URL: https://github.com/apache/lucene/pull/11945#issuecomment-1318068758
Works for me. I was able to now run this monster test in < 10 minutes time.
```
<===:lucene:core:test (SUCCESS): 1 test(s)
The slowest tests (exceeding 500 ms) during this run:
527.29s TestManyKnnDocs.testLargeSegment (:lucene:core)
The slowest suites (exceeding 1s) during this run:
527.61s TestManyKnnDocs (:lucene:core)
BUILD SUCCESSFUL in 9m 2s
19 actionable tasks: 5 executed, 14 up-to-date
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
[GitHub] [lucene] rmuir commented on pull request #11945: Decrease test time for TestManyKnnDocs.testLargeSegment
Posted by GitBox <gi...@apache.org>.
rmuir commented on PR #11945:
URL: https://github.com/apache/lucene/pull/11945#issuecomment-1318504049
This M is ... the length of the "postings list" for vector. This test codec allows using a larger value... so more data written per document, but less documents needed to trigger the overflow that we wanted to test for here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
[GitHub] [lucene] msokolov commented on pull request #11945: Decrease test time for TestManyKnnDocs.testLargeSegment
Posted by GitBox <gi...@apache.org>.
msokolov commented on PR #11945:
URL: https://github.com/apache/lucene/pull/11945#issuecomment-1320406270
oh nice plan, thanks everyone
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
[GitHub] [lucene] rmuir merged pull request #11945: Decrease test time for TestManyKnnDocs.testLargeSegment
Posted by GitBox <gi...@apache.org>.
rmuir merged PR #11945:
URL: https://github.com/apache/lucene/pull/11945
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
[GitHub] [lucene] uschindler commented on pull request #11945: Decrease test time for TestManyKnnDocs.testLargeSegment
Posted by GitBox <gi...@apache.org>.
uschindler commented on PR #11945:
URL: https://github.com/apache/lucene/pull/11945#issuecomment-1318373916
Very cool idea (although I have no idea wha this does because of my ignorance for KNN).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
[GitHub] [lucene] rmuir commented on pull request #11945: Decrease test time for TestManyKnnDocs.testLargeSegment
Posted by GitBox <gi...@apache.org>.
rmuir commented on PR #11945:
URL: https://github.com/apache/lucene/pull/11945#issuecomment-1317994569
we need to run a `./gradlew tidy` and commit/push the results to fix formatting.
Very cool, will test on my 2-core. we may be able to upgrade from `@Monster` to `@Nightly`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
[GitHub] [lucene] rmuir commented on pull request #11945: Decrease test time for TestManyKnnDocs.testLargeSegment
Posted by GitBox <gi...@apache.org>.
rmuir commented on PR #11945:
URL: https://github.com/apache/lucene/pull/11945#issuecomment-1318667778
thanks @jdconrad !
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
[GitHub] [lucene] jdconrad commented on pull request #11945: Decrease test time for TestManyKnnDocs.testLargeSegment
Posted by GitBox <gi...@apache.org>.
jdconrad commented on PR #11945:
URL: https://github.com/apache/lucene/pull/11945#issuecomment-1318001086
Updated with tidy! (Oops on failing precommit.)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
[GitHub] [lucene] benwtrent commented on pull request #11945: Decrease test time for TestManyKnnDocs.testLargeSegment
Posted by GitBox <gi...@apache.org>.
benwtrent commented on PR #11945:
URL: https://github.com/apache/lucene/pull/11945#issuecomment-1318522602
awesome stuff @jdconrad!!!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org