You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@lucene.apache.org by GitBox <gi...@apache.org> on 2021/04/29 16:44:00 UTC
[GitHub] [lucene] balmukundblr opened a new pull request #116: Explicit flush
balmukundblr opened a new pull request #116:
URL: https://github.com/apache/lucene/pull/116
<!--
_(If you are a project committer then you may remove some/all of the following template.)_
Before creating a pull request, please file an issue in the ASF Jira system for Lucene:
* https://issues.apache.org/jira/projects/LUCENE
You will need to create an account in Jira in order to create an issue.
The title of the PR should reference the Jira issue number in the form:
* LUCENE-####: <short description of problem or changes>
LUCENE must be fully capitalized. A short description helps people scanning pull requests for items they can work on.
Properly referencing the issue in the title ensures that Jira is correctly updated with code review comments and commits. -->
# Description
Longer completion time for Close Index call.
Once AddDoc task completes, Benchmark algo calls ForceMerge/CloseIndex task, which eventually allows all pending flushes to be completed. Since flushes during CloseIndex call are sequential, it takes longer time to complete and delays the overall Index completion time. While indexing 1 million documents with reuters21578 (plain text Document derived from reuters21578 corpus), we observed CloseIndex call takes around 35% of total time.
# Solution
Developed a new FlushIndexTask, which uses flushNextBuffer() Lucene API, to flush document at Index thread level, while not impacting any other Index threads. Adding this task in the algo file, immediately after AddDoc task, would ensure flushing all docs before calling ForceMerge/CloseIndex task.
With this solution in place, CloseIndex task time was reduced significantly and it also improved total time for Indexing.
# Tests
Since, we are using existing Lucene API - flushNextBuffer(), hence it already has test cases.
-Passed existing tests
# Checklist
Please review the following and check all that apply:
- [ x] I have reviewed the guidelines for [How to Contribute](https://wiki.apache.org/lucene/HowToContribute) and my code conforms to the standards described there to the best of my ability.
- [ ] I have created a Jira issue and added the issue ID to my pull request title.
- [ x] I have given Lucene maintainers [access](https://help.github.com/en/articles/allowing-changes-to-a-pull-request-branch-created-from-a-fork) to contribute to my PR branch. (optional but recommended)
- [x ] I have developed this patch against the `main` branch.
- [x ] I have run `./gradlew check`.
- [ ] I have added tests for my changes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
[GitHub] [lucene] balmukundblr commented on pull request #116: Explicit flush
Posted by GitBox <gi...@apache.org>.
balmukundblr commented on pull request #116:
URL: https://github.com/apache/lucene/pull/116#issuecomment-829893980
> Thank you @balmukundblr!
Thank you very much Mike for your great support.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
[GitHub] [lucene] mikemccand merged pull request #116: Explicit flush
Posted by GitBox <gi...@apache.org>.
mikemccand merged pull request #116:
URL: https://github.com/apache/lucene/pull/116
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
[GitHub] [lucene] mikemccand commented on pull request #116: Explicit flush
Posted by GitBox <gi...@apache.org>.
mikemccand commented on pull request #116:
URL: https://github.com/apache/lucene/pull/116#issuecomment-829720860
Thank you @balmukundblr!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
[GitHub] [lucene] mikemccand commented on pull request #116: Explicit flush
Posted by GitBox <gi...@apache.org>.
mikemccand commented on pull request #116:
URL: https://github.com/apache/lucene/pull/116#issuecomment-829452800
Thanks @balmukundblr -- looks great -- I'll try to push today.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org