You are viewing a plain text version of this content. The canonical link for it is here.
Posted to java-user@lucene.apache.org by Carsten Schnober <sc...@ids-mannheim.de> on 2012/07/13 14:30:12 UTC
Offsets in 3.6/4.0
Dear list,
I am working on a search application that depends on retrieving offsets
for each match. Currently (in Lucene 3.6), this seems to be overly
costly, at least in my solution that looks like this:
-----------------------------------------------------------------------
TermPositionVector tfv;
int index;
TermVectorOffsetInfo[] offsets;
tfv = (TermPositionVector) reader.getTermFreqVector(docid, fieldname);
index = tfv.indexOf(term.text());
offsets = tfv.getOffsets(index);
-----------------------------------------------------------------------
So I can user the suitable TermVectorOffsetInfo from the offsets[] array
to retrieve the offset information of a span. However, this slows down
the search to an unacceptable level.
Reviewing the thread 'Retrieving Offsets'
(http://lucene.472066.n3.nabble.com/Retrieving-offsets-td3658238.html)
indicates that there has not been any more efficient way to go in Lucene
3.6. Am I right?
However, I understand that the patch LUCENE-3684
(https://issues.apache.org/jira/browse/LUCENE-3684) has improved the
situation. I am wondering now whether this is worth migrating to Lucene
4.0 in terms of search performance. It is currently not entirely clear
to me, whether Lucene 4.0 alpha actually allows the retrieval of offsets
from an index without having to read the TermFreqVector though.
Who can give me some advise about the potential search performance gain
for such an application and ideally to some pointers about how to
resolve the problem?
Thank you very much,
Carsten Schnober
--
Institut für Deutsche Sprache | http://www.ids-mannheim.de
Projekt KorAP | http://korap.ids-mannheim.de
Tel. +49-(0)621-43740789 | schnober@ids-mannheim.de
Korpusanalyseplattform der nächsten Generation
Next Generation Corpus Analysis Platform
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: Offsets in 3.6/4.0
Posted by ka...@gmx.de.
Dear Carsten,
your question was about the purpose of the offset-Attribute and the reader.getTermFreqVector method.
You have asked because this method is not very fast.
imho main reason for TermFreqVectors is highlighting.
(FastVectorHighlighter and DefaultSolrHighlighter#doHighlightingByHighlighter)
You normally use highlighting only on a small part of the hit list.
If you have a fast analyser you do not need TermFreqVector because you can calculate the offset on the fly.
(
Calculate from the value of a stored field.
Sometimes TermFreqVector can be used as substitute of a stored field.
)
Best regards
Karsten
in Context:
http://lucene.472066.n3.nabble.com/Offsets-in-3-6-4-0-td3994830.html#a3995288
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: Offsets in 3.6/4.0
Posted by Carsten Schnober <sc...@ids-mannheim.de>.
Am 16.07.2012 13:07, schrieb karsten-solr@gmx.de:
Dear Karsten,
> abstract of your post:
> you need the offset to perform your search/ranking like the position is needed for phrase queries.
> You are using reader.getTermFreqVector to get the offset.
> This is to slow for your application and you think about a switch to version 4.0
Yes, that's about it.
> imho you should using payloads.
> You also could switch to version 4 because in version you can store the offset to each term like the position in version 3x.
> But this is basically the same as the use of payloads:
> * http://lucene.apache.org/core/3_6_0/fileformats.html#Positions
> * http://lucene.apache.org/core/4_0_0-ALPHA/core/org/apache/lucene/codecs/lucene40/Lucene40PostingsFormat.html#Positions
I now use payloads and this fulfils my functional requirements. I was
hoping to avoid that because I am also storing other information in the
Payload which makes it feel a bit messy; especially as it seemed
sensible to me to actually make use of the Offsets field as it already
exists. Anyway, the problem is solved so far, thank you very much!
I still wonder what the purpose of the Offset field is as it is so
inefficient to access. It seems like a wasteful redundancy to even store
the Offsets during indexing, considering that I also store it as a
payload. Or am I missing something?
Best,
Carsten
--
Institut für Deutsche Sprache | http://www.ids-mannheim.de
Projekt KorAP | http://korap.ids-mannheim.de
Tel. +49-(0)621-43740789 | schnober@ids-mannheim.de
Korpusanalyseplattform der nächsten Generation
Next Generation Corpus Analysis Platform
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: Offsets in 3.6/4.0
Posted by ka...@gmx.de.
Dear Carsten,
abstract of your post:
you need the offset to perform your search/ranking like the position is needed for phrase queries.
You are using reader.getTermFreqVector to get the offset.
This is to slow for your application and you think about a switch to version 4.0
imho you should using payloads.
You also could switch to version 4 because in version you can store the offset to each term like the position in version 3x.
But this is basically the same as the use of payloads:
* http://lucene.apache.org/core/3_6_0/fileformats.html#Positions
* http://lucene.apache.org/core/4_0_0-ALPHA/core/org/apache/lucene/codecs/lucene40/Lucene40PostingsFormat.html#Positions
Best regards
Karsten
P.S. in context:
http://lucene.472066.n3.nabble.com/Offsets-in-3-6-4-0-td3994830.html
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org