You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by "David Smiley (JIRA)" <ji...@apache.org> on 2015/10/29 17:57:27 UTC
[jira] [Updated] (LUCENE-6868) ParallelLeafReader.getTermVectors
can indirectly load TVs multiple times
[ https://issues.apache.org/jira/browse/LUCENE-6868?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
David Smiley updated LUCENE-6868:
---------------------------------
Description:
ParallelLeafReader has a getTermVectors(docId) implementation that loops over each field it has in a loop and calls getTermVector(docId,fieldName). But the implementation of that will load all term vectors for all fields in that reader, yet ParallelLeafReader only wants one. The effect is an O(n^2) where 'n' is the number of fields, when we could get O\(n) if we do it right. PLR should call getTermVectors(docId) (not referring to a specific field) for each of it's readers and then aggregate them.
This wouldn't be such a problem if our term vector API/Codec was improved to not load all term vectors for all fields from disk at once.
Found via randomized-testing of IndexWriter auto-picking ParallelAtomicReader along with a test I have that asserts TVs aren't fetched for a doc more than once.
was:
ParallelLeafReader has a getTermVectors(docId) implementation that loops over each field it has in a loop and calls getTermVector(docId,fieldName). But the implementation of that will load all term vectors for all fields in that reader, yet ParallelLeafReader only wants one. The effect is an O(n^2) where 'n' is the number of fields, when we could get O(n) if we do it right. PLR should call getTermVectors(docId) (not referring to a specific field) for each of it's readers and then aggregate them.
This wouldn't be such a problem if our term vector API/Codec was improved to not load all term vectors for all fields from disk at once.
Found via randomized-testing of IndexWriter auto-picking ParallelAtomicReader along with a test I have that asserts TVs aren't fetched for a doc more than once.
> ParallelLeafReader.getTermVectors can indirectly load TVs multiple times
> ------------------------------------------------------------------------
>
> Key: LUCENE-6868
> URL: https://issues.apache.org/jira/browse/LUCENE-6868
> Project: Lucene - Core
> Issue Type: Improvement
> Components: core/index, core/termvectors
> Reporter: David Smiley
>
> ParallelLeafReader has a getTermVectors(docId) implementation that loops over each field it has in a loop and calls getTermVector(docId,fieldName). But the implementation of that will load all term vectors for all fields in that reader, yet ParallelLeafReader only wants one. The effect is an O(n^2) where 'n' is the number of fields, when we could get O\(n) if we do it right. PLR should call getTermVectors(docId) (not referring to a specific field) for each of it's readers and then aggregate them.
> This wouldn't be such a problem if our term vector API/Codec was improved to not load all term vectors for all fields from disk at once.
> Found via randomized-testing of IndexWriter auto-picking ParallelAtomicReader along with a test I have that asserts TVs aren't fetched for a doc more than once.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org