You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@lucene.apache.org by "Jeevananthan-23 (via GitHub)" <gi...@apache.org> on 2023/08/31 19:07:05 UTC
[GitHub] [lucene] Jeevananthan-23 opened a new issue, #12531: Virtual threads and Lucene (support async tasks)
Jeevananthan-23 opened a new issue, #12531:
URL: https://github.com/apache/lucene/issues/12531
### Description
In **LuceneNet**(`C#`) we had a long conversation about adding support for [Async API issue](https://github.com/apache/lucenenet/issues/763) but the scope of the project was not allowed because it's a port project 😒. Lucene currently has the `TaskExecutor` (previously named `SliceExecutor`) which is responsible for offloading tasks to the executor, wait for them all to complete and return the corresponding results, this is one of the examples but adding **VirtualThreads**(JDK21) helps to achieve concurrency (_IO bound operations_).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
Re: [I] Virtual threads and Lucene (support async tasks) [lucene]
Posted by "jpountz (via GitHub)" <gi...@apache.org>.
jpountz commented on issue #12531:
URL: https://github.com/apache/lucene/issues/12531#issuecomment-2135735022
I think it's fair to close this issue in favor of #13179 since this is mostly about I/O concurrency?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
[GitHub] [lucene] uschindler commented on issue #12531: Virtual threads and Lucene (support async tasks)
Posted by "uschindler (via GitHub)" <gi...@apache.org>.
uschindler commented on issue #12531:
URL: https://github.com/apache/lucene/issues/12531#issuecomment-1704969493
Lucene does not use threads during searching internally, everything is sequential. IndexReader is using Executors, but actually there is no reason to NOT use virtual threads.
The issue mentioned above is about disk IO, which is a different story. As Lucene uses preferably MMAP to access disks theres no way to make this async. It would need a complete rewrite of all Lucene internals, so this is not working in the basic design (and also not needed). Lucene is mostly CPU intensive operations, disk IO is not really part of the API layer.
This issue is possibly won't fix.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
Re: [I] Virtual threads and Lucene (support async tasks) [lucene]
Posted by "uschindler (via GitHub)" <gi...@apache.org>.
uschindler commented on issue #12531:
URL: https://github.com/apache/lucene/issues/12531#issuecomment-1824732748
> @uschindler, I understand the complexity of completely rewriting all Lucene internals. However, IMO it is necessary to do so in parallel. Relying entirely on MMAP is a bad idea it is good for warmup queries.
>
> Ref: https://db.cs.cmu.edu/mmap-cidr2022/
Lucene has different access patterns that are not database like. MMAP works perfectly here. Lucene is using WORM when writing files. This paper is known to use and our long-time testing figured out that it does not apply to most Lucene workloads.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
Re: [I] Virtual threads and Lucene (support async tasks) [lucene]
Posted by "Jeevananthan-23 (via GitHub)" <gi...@apache.org>.
Jeevananthan-23 commented on issue #12531:
URL: https://github.com/apache/lucene/issues/12531#issuecomment-1831313847
Hi @uschindler, I came across an interesting article on Qdrant vector database that uses io_uring for async and mmap benchmarking.
https://qdrant.tech/articles/io_uring/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
Re: [I] Virtual threads and Lucene (support async tasks) [lucene]
Posted by "Jeevananthan-23 (via GitHub)" <gi...@apache.org>.
Jeevananthan-23 commented on issue #12531:
URL: https://github.com/apache/lucene/issues/12531#issuecomment-1824474577
@uschindler, I understand the complexity of completely rewriting all Lucene internals. However, IMO it is necessary to do so in parallel. Relying entirely on MMAP is a bad idea it is good for warmup queries.
Ref: https://db.cs.cmu.edu/mmap-cidr2022/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
Re: [I] Virtual threads and Lucene (support async tasks) [lucene]
Posted by "jpountz (via GitHub)" <gi...@apache.org>.
jpountz closed issue #12531: Virtual threads and Lucene (support async tasks)
URL: https://github.com/apache/lucene/issues/12531
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
[GitHub] [lucene] Jeevananthan-23 commented on issue #12531: Virtual threads and Lucene (support async tasks)
Posted by "Jeevananthan-23 (via GitHub)" <gi...@apache.org>.
Jeevananthan-23 commented on issue #12531:
URL: https://github.com/apache/lucene/issues/12531#issuecomment-1702821799
cc: @mikemccand
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
[GitHub] [lucene] reta commented on issue #12531: Virtual threads and Lucene (support async tasks)
Posted by "reta (via GitHub)" <gi...@apache.org>.
reta commented on issue #12531:
URL: https://github.com/apache/lucene/issues/12531#issuecomment-1702967302
There are some discussions in different places around that, see please https://github.com/opensearch-project/OpenSearch/issues/1687#issuecomment-1621928540 for example.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org