You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@lucene.apache.org by GitBox <gi...@apache.org> on 2022/10/05 21:59:20 UTC
[GitHub] [lucene] zhaih opened a new issue, #11838: Adding concurrency to query rewrite?
zhaih opened a new issue, #11838:
URL: https://github.com/apache/lucene/issues/11838
### Description
There're several queries relying on rewrite to do the most heavy work, such as KnnVectorQuery and AutomatonQuery, should we make rewrite step being able to run under multi-threading? Or we could also just move those costly steps into scorer creation and don't rewrite, that could also be an option because I can't really come up with other queries that are costly when doing rewrite.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
[GitHub] [lucene] zhaih commented on issue #11838: Adding concurrency to query rewrite?
Posted by GitBox <gi...@apache.org>.
zhaih commented on issue #11838:
URL: https://github.com/apache/lucene/issues/11838#issuecomment-1271850864
> Maybe a good idea would be to allow the heavy calculations in Weight and also Rewrite to execute in an Executor.
+1, although I probably would like to start with rewrite first since I'm not aware of any weight creation that could easily benefit from execute it in parallel.
I'm thinking about something similar to what @uschindler suggest, basically create another public method `rewrite(IndexReader, Executor)` on Query and default it to just call `rewrite(IndexReader)` without any action using executor, while in the IndexSearcher we could always pass in whatever executor we have. And then from there we could work on individual query's rewriting implementation.
Seems people are not going against this idea I'll get a PR out soon, and we can start from there :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
[GitHub] [lucene] benwtrent closed issue #11838: Adding concurrency to query rewrite?
Posted by "benwtrent (via GitHub)" <gi...@apache.org>.
benwtrent closed issue #11838: Adding concurrency to query rewrite?
URL: https://github.com/apache/lucene/issues/11838
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
[GitHub] [lucene] benwtrent commented on issue #11838: Adding concurrency to query rewrite?
Posted by "benwtrent (via GitHub)" <gi...@apache.org>.
benwtrent commented on issue #11838:
URL: https://github.com/apache/lucene/issues/11838#issuecomment-1545891005
Merged and backported for a future Lucene release
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
[GitHub] [lucene] jtibshirani commented on issue #11838: Adding concurrency to query rewrite?
Posted by GitBox <gi...@apache.org>.
jtibshirani commented on issue #11838:
URL: https://github.com/apache/lucene/issues/11838#issuecomment-1270564173
This is a great question! To me it'd make sense to try to move costly steps into weight creation or scoring. It feels a little "off" to do the bulk of a query's work during rewrite.
However, I understand why we chose to use rewrite for `KnnVectorQuery`. It allows us the query to match the global top `k` across segments, as opposed to matching `k` docs per segment. This means the number of documents it matches doesn't depend on the number of segments, and it's easier to combine with other queries. Maybe this is not the right trade-off though in terms of behavior vs. performance 🤔
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
[GitHub] [lucene] zhaih commented on issue #11838: Adding concurrency to query rewrite?
Posted by GitBox <gi...@apache.org>.
zhaih commented on issue #11838:
URL: https://github.com/apache/lucene/issues/11838#issuecomment-1284303658
Change to main branch merged, keep this open until backport finishes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
[GitHub] [lucene] uschindler commented on issue #11838: Adding concurrency to query rewrite?
Posted by GitBox <gi...@apache.org>.
uschindler commented on issue #11838:
URL: https://github.com/apache/lucene/issues/11838#issuecomment-1271537616
For AutomatonQuery it is also required to rewrite, because there are different implementations used. When you expand to a BooleanQuery or TermInSetQuery, you need to do that in the rewrite phase.
My question is: What brings moving it to the Weight? It is still not multithreaded!
Maybe a good idea would be to allow the heavy calculations in Weight and also Rewrite to execute in an Executor. By default we pass a SameThreadExecutor implementation (basically the executor just calls `run()`), but a query implementation might use multiple threads by splitting the work into smaller pieces. E.g. a BooleanQuery could rewrite all its clauses in a separate task.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org