You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by none none <ko...@lycos.com> on 2003/03/03 08:19:32 UTC
Re: Lucene Highlighter
hi,
after digging a little bit the code i came up with some questions due to make the highlighter working with the future release of Lucene (1.3).
The questions are:
- why phrase uses a Vector and PhrasePrefix an ArrayList? just curious.
- Is it possible add a method "public Term[] getTermsArray()" that will return the "termArrays" from the PhrasePrefixQuery? Is it still populated after we run the search?
- Is it possible have a PhrasePrefixQuery of 2+ terms? e.g.: "Microsoft Soft* Windo*" ? why are there 2 methods, one to add a single term another one to add more than one term? is the termsArray an array on term's array ?
- Is it correct that PrefixQuery.rewrite(...) is called by the searcher (reader?) at search time to have a BooleanQuery with "OR" condition between each clause? each clause holds a termquery?
- PrefixQuery > what do you think of this scenario: user set "populateTermArray()" before run the search, we set a static variable inside the Query class so the setting is reflected to all the XxxQuery classes, in the 'rewrite' method we check this value and if true (default false) we store each term in an array 'termsArray' one for each implementation (wildcard, etc), then when we need to highlight we call getTermsArray() for each query based on the instance type (again: wildcard, etc), then we set the array to null or wait for the garbage collector to release this resource. sounds good??
- PrefixQuery and other query classes that has this method 'rewrite'>> can the method be called more than once at search time? if so we should hold the privious array of terms and add to it the new terms without duplicates.
- RangeQuery >> can we apply the same criteria as for the PrefixQuery?
- All the classes that extends MultiTermQuery >> can we apply the same criteria as for PrefixQuery? (as above, just add a vector that holds the terms, if the user wants to, and get this array when highlighting, may call a method to release the resource after we are done with the highlight)
- how it is possible get the term position of a particular term in a particular document in the index? this will improve a lot the process to get start and end offset of a term in a document. i assume that a text version of the field to highlight is available, e.g.: the content of an html page is a field and is stored in a single text file. Also would make it compatible with the tokenizer as we will use the same we did at indexing time, avoid to write a pattern for each criteria in the RegExp (actually it will not be necessary anymore!)
- would all these changes make slower the search process? as a guess, how much?
- would the termposition call be slow?
Thank you guys.
_____________________________________________________________
Get 25MB, POP3, Spam Filtering with LYCOS MAIL PLUS for $19.95/year.
http://login.mail.lycos.com/brandPage.shtml?pageId=plus&ref=lmtplus
---------------------------------------------------------------------
To unsubscribe, e-mail: lucene-dev-unsubscribe@jakarta.apache.org
For additional commands, e-mail: lucene-dev-help@jakarta.apache.org
About PrefixQuery
Posted by Tatu Saloranta <ta...@hypermall.net>.
I started looking to implement a term collector (object that can get all the
terms for given query), and have couple of questions for someone more
familiar with code base:
(1) PrefixQuery does not extend MultiTermQuery (unlike FuzzyQuery and
WildcardQuery). Is there some specific reason for this, or was it just
implemented before other two (perhaps MultiTermQuery was added after
PrefixQuery?).
(2) RangeQuery does not extend MultiTermQuery either, should it?
(3) Like Doug pointed out, there are two distinct use cases; caller may want
to get either 'raw' terms (unprocessed terms, like "foo?ar") or terms
actually contained in any documents indexed ("foo?ar" might expand to
"foobar", "foogar" etc). However, there's also third interesting use case;
getting terms in a specific document. Is it likely implementing this might
be (relatively) easy? I'm not familiar enough with IndexReader to know for
sure.
-+ Tatu +-
---------------------------------------------------------------------
To unsubscribe, e-mail: lucene-dev-unsubscribe@jakarta.apache.org
For additional commands, e-mail: lucene-dev-help@jakarta.apache.org
Re: Lucene Highlighter
Posted by Doug Cutting <cu...@lucene.com>.
none none wrote:
> - why phrase uses a Vector and PhrasePrefix an ArrayList? just
> curious.
This code was written by different authors at different times.
> - Is it possible add a method "public Term[] getTermsArray()" that
> will return the "termArrays" from the PhrasePrefixQuery? Is it still
> populated after we run the search?
Something like this is under discussion. I think Tatu Saloranta is
working on a proposal for this.
> - Is it possible have a PhrasePrefixQuery of 2+ terms? e.g.:
> "Microsoft Soft* Windo*" ?
Yes.
> why are there 2 methods, one to add a
> single term another one to add more than one term?
The single-term method is a convenience.
> is the termsArray
> an array on term's array ?
I don't understand the question.
> - Is it correct that PrefixQuery.rewrite(...) is called by the
> searcher (reader?) at search time to have a BooleanQuery with "OR"
> condition between each clause? each clause holds a termquery?
Yes, that is correct.
> - PrefixQuery > what do you think of this scenario: user set
> "populateTermArray()" before run the search, we set a static variable
> inside the Query class so the setting is reflected to all the
> XxxQuery classes, in the 'rewrite' method we check this value and if
> true (default false) we store each term in an array 'termsArray' one
> for each implementation (wildcard, etc), then when we need to
> highlight we call getTermsArray() for each query based on the
> instance type (again: wildcard, etc), then we set the array to null
> or wait for the garbage collector to release this resource. sounds
> good??
I think we would be better to wait for Tatu's proposal.
> - how it is possible get the term position of a particular term in a
> particular document in the index?
IndexReader.termPositions() is the closest thing to this in Lucene.
However, for highlighting, since one must re-tokenize the document
anyway, its usually easier to just scan it for terms that are in the query.
Doug
---------------------------------------------------------------------
To unsubscribe, e-mail: lucene-dev-unsubscribe@jakarta.apache.org
For additional commands, e-mail: lucene-dev-help@jakarta.apache.org
RE: Lucene Highlighter
Posted by Ilya Khandamirov <ik...@startext.de>.
Hi,
Is where any progress regarding this issue? Are there already any
publicly available code/samples/recommendations concerning the
highlighting implementation with the current version of Lucene (1.3), or
should one dig for details in the new API from scratch?
Regards,
Ilya
-----Original Message-----
From: none none [mailto:korfut@lycos.com]
Sent: Montag, 3. März 2003 08:20
i came up with some questions due to make the highlighter working with
the future release of Lucene (1.3).
---------------------------------------------------------------------
To unsubscribe, e-mail: lucene-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: lucene-user-help@jakarta.apache.org