You are viewing a plain text version of this content. The canonical link for it is here.
Posted to java-user@lucene.apache.org by starz10de <fa...@yahoo.com> on 2011/11/20 10:52:52 UTC
lucene highlighter
Hi,
I have a problem with lucene highlighter. I couldn’t make it run. The
compilation is without error but when I run it I got this error “Exception
in thread "main"
java.lang.NoSuchMethodError:org.apache.lucene.analysis.TokenStream.next(Lorg/apache/lucene/analysis/Token;)Lorg/apache/lucene/analysis/Token;”
I am using lucene 3.0.3 core and lucene highlighter 2.4. It might be a
problem of compatibility. I tried to use lucene highlighter 3. but I
couldn’t combile he code and I got this error “SpanScorer can’t be resolved
as a type”
Here part of the code:
import org.apache.lucene.search.highlight.Fragmenter;
import org.apache.lucene.search.highlight.Highlighter;
import org.apache.lucene.search.highlight.SimpleSpanFragmenter;
import org.apache.lucene.search.highlight.SpanScorer;
import org.apache.lucene.search.highlight.TokenSources;
.
.
.
for (int i = start; i < end; i++) {
if (raw) { // output raw format
System.out.println("doc="+hits[i].doc+" score="+hits[i].score);
continue;
}
Document doc = searcher.doc(hits[i].doc);
String path = doc.get("path");
contents=doc.get("contents");
// System.out.println(text+" !!!!!!!!!!!!!!!!!");
fragments = getFragmentsWithHighlightedTerms(analyzer, query, "contents",
contents, 5, 100);
.
.
private static String[] getFragmentsWithHighlightedTerms(Analyzer analyzer,
Query query,String fieldName, String fieldContents, int fragmentNumber,
int fragmentSize) throws IOException
{
TokenStream stream = TokenSources.getTokenStream(fieldName, fieldContents,
analyzer);
SpanScorer scorer = new SpanScorer(query, fieldName, new
CachingTokenFilter(stream));
Fragmenter fragmenter = new SimpleSpanFragmenter(scorer, fragmentSize);
Highlighter highlighter = new Highlighter(scorer);
highlighter.setTextFragmenter(fragmenter);
highlighter.setMaxDocCharsToAnalyze(Integer.MAX_VALUE);
String[] summaries = highlighter.getBestFragments(new
StandardAnalyzer(Version.LUCENE_CURRENT), fieldContents,contents, 3);
System.out.println(summaries);
// String[] fragments = highlighter.getBestFragments(stream, fieldContents,
fragmentNumber);
return fragments;
}
Thanks in advance
--
View this message in context: http://lucene.472066.n3.nabble.com/lucene-highlighter-tp3522464p3522464.html
Sent from the Lucene - Java Users mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: lucene hits vs topdocs
Posted by Ian Lea <ia...@gmail.com>.
The general recommendation is to run the query again but you are right
that it isn't always the correct answer in all circumstances. If you
want to guard against the scenario you outline, do it the way you
suggest, That's fine. In your fluid environment how do you cope when
doc #11 is no longer there when you move to page 2? Do you worry
about missing new docs that won't appear in results because they
weren't there when the first search was executed? Pros and cons to
all approaches. If you are caching lucene docids be aware that they
can change. http://wiki.apache.org/lucene-java/LuceneFAQ#When_is_it_possible_for_document_IDs_to_change.3F
There is also something called search.After due in the next release of
Lucene. See recent thread "Lucene pagination" on this list.
--
Ian.
On Sun, Nov 20, 2011 at 6:51 PM, Gwyn Carwardine <gw...@carwardine.net> wrote:
> Hi
>
> I last used dotLucene 143 and now I'm wanting to upgrade to 294.
>
> What I've discovered is that there are quite a few changes..
>
> One of them is in respect of Search. Previously one supplied a query and
> received a number of hits. I didn't have an issue with preservation of state
> so was quite happy to page through the stored hits
>
> Now it has changed it also recommends passing the number of results required
> (as in top xx results) so I'm considering how to refactor my code.
>
> In the simplest way I guess I could retrieve all results as I did previously
> and then paginate through them, or I could use the re-querying approach. But
> this suggests for let's say 10 results per page that I query for 10 docs and
> then when the user scrolls to the next page I re-query for 20 docs and
> ignore the first 10 and so on and so forth.
>
> What initially strikes me about this is that in a fluid environment (where
> changes are constantly being re-indexed) it is possible that an item that
> would come in an number 11 on the first call (and hence not shown on page 1)
> would now move to number 10 on the second call (and hence not shown of page
> 2).
>
> I would expect as a user that if I do a query and then page through it then
> it is the same result set I am paging through and not one that could be
> constantly changing (especially if I am paging through a bit slowly).
>
> I am using Lucene as a text search within an information mgt product that
> does have lots of updates happening so this could well happen. And it only
> needs to happen once and someone miss a key bit of info for it to be
> embarrassing.
>
> So I'm curious as to how people out there actually do this. Yes holding
> state is a pain but I do that already.
>
> It just seems that Lucene is pointing towards "tell me how many" and so I
> really don't want to go against the tide (or it'll likely be painful next
> time I upgrade!)
>
> Thanks in advance
>
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
lucene hits vs topdocs
Posted by Gwyn Carwardine <gw...@carwardine.net>.
Hi
I last used dotLucene 143 and now I'm wanting to upgrade to 294.
What I've discovered is that there are quite a few changes..
One of them is in respect of Search. Previously one supplied a query and
received a number of hits. I didn't have an issue with preservation of state
so was quite happy to page through the stored hits
Now it has changed it also recommends passing the number of results required
(as in top xx results) so I'm considering how to refactor my code.
In the simplest way I guess I could retrieve all results as I did previously
and then paginate through them, or I could use the re-querying approach. But
this suggests for let's say 10 results per page that I query for 10 docs and
then when the user scrolls to the next page I re-query for 20 docs and
ignore the first 10 and so on and so forth.
What initially strikes me about this is that in a fluid environment (where
changes are constantly being re-indexed) it is possible that an item that
would come in an number 11 on the first call (and hence not shown on page 1)
would now move to number 10 on the second call (and hence not shown of page
2).
I would expect as a user that if I do a query and then page through it then
it is the same result set I am paging through and not one that could be
constantly changing (especially if I am paging through a bit slowly).
I am using Lucene as a text search within an information mgt product that
does have lots of updates happening so this could well happen. And it only
needs to happen once and someone miss a key bit of info for it to be
embarrassing.
So I'm curious as to how people out there actually do this. Yes holding
state is a pain but I do that already.
It just seems that Lucene is pointing towards "tell me how many" and so I
really don't want to go against the tide (or it'll likely be painful next
time I upgrade!)
Thanks in advance
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
RE: lucene highlighter
Posted by Uwe Schindler <uw...@thetaphi.de>.
Hi
> Thanks for your useful comments:
>
> here I could do what I want with the highlighter which work with lucene 3:
>
> QueryScorer scorer = new QueryScorer(query, reader, "contents");
> Highlighter highlighter = new Highlighter(scorer);
>
> String fragment = highlighter.getBestFragment(analyzer,
> "contents", contents);
>
>
> I have one question, you mentioned: „You also need contrib-memory-3.0.3.jar
> to actually run your program (not for compiling)."
>
> Is this needed only for optimization issue? I searched in the web and I couldn't
> rally find this jar. Is it available online?
You need the memory contrib module if your underlying lucene index does not contain term vectors. Download the complete Lucene 3.0.3 binary tarball from the apache-site, it contains all JAR files, mixing JAR files produced by builds from different vendors may break your application. Memory contrib is named lucene-memory-3.0.3.jar.
Uwe
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
RE: lucene highlighter
Posted by starz10de <fa...@yahoo.com>.
Hi,
Thanks for your useful comments:
here I could do what I want with the highlighter which work with lucene 3:
QueryScorer scorer = new QueryScorer(query, reader, "contents");
Highlighter highlighter = new Highlighter(scorer);
String fragment = highlighter.getBestFragment(analyzer,
"contents", contents);
I have one question, you mentioned: „You also need contrib-memory-3.0.3.jar
to actually run your program (not for compiling)."
Is this needed only for optimization issue? I searched in the web and I
couldn't rally find this jar. Is it available online?
Thanks
--
View this message in context: http://lucene.472066.n3.nabble.com/lucene-highlighter-tp3522464p3522652.html
Sent from the Lucene - Java Users mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
RE: lucene highlighter
Posted by Uwe Schindler <uw...@thetaphi.de>.
Hi,
You should review the new API of Lucene 3.0, SpanScorer was replaced by something else. The CHANGES.txt of the contrib package may give you some hints. Its long time ago that we changed this api (Lucene 3.0 is also "ancient"). Contrib-Highlighter may break backwards compatibility even with minor versions, as it's not covered by the strict backwards compatibility of the Lucene Core. So best is to review the javadocs of 2.4, 2.9 and 3.0, too.
> I checked the class path and there were no old version of lucene only:
> lucene-highlighter-3.0.3.jar and lucene-core-3.0.3.jar
You also need contrib-memory-3.0.3.jar to actually run your program (not for compiling).
Uwe
-----
Uwe Schindler
H.-H.-Meier-Allee 63, D-28213 Bremen
http://www.thetaphi.de
eMail: uwe@thetaphi.de
> -----Original Message-----
> From: starz10de [mailto:farag_ahmed@yahoo.com]
> Sent: Sunday, November 20, 2011 12:05 PM
> To: java-user@lucene.apache.org
> Subject: RE: lucene highlighter
>
> Hi Uwe,
>
> Thanks for your answer. I am using now lucene-highlighter-3.0.3 but the
> problem I have this error:
>
> “SpanScorer can’t be resolved as a type”
>
> > SpanScorer scorer = new SpanScorer(query, fieldName, new
> > CachingTokenFilter(stream));
>
> I checked the class path and there were no old version of lucene only:
> lucene-highlighter-3.0.3.jar and lucene-core-3.0.3.jar
>
> any hints?
>
> Thanks
>
> Farag
>
>
>
> --
> View this message in context: http://lucene.472066.n3.nabble.com/lucene-
> highlighter-tp3522464p3522535.html
> Sent from the Lucene - Java Users mailing list archive at Nabble.com.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
RE: lucene highlighter
Posted by starz10de <fa...@yahoo.com>.
Hi Uwe,
Thanks for your answer. I am using now lucene-highlighter-3.0.3 but the
problem I have this error:
“SpanScorer can’t be resolved as a type”
> SpanScorer scorer = new SpanScorer(query, fieldName, new
> CachingTokenFilter(stream));
I checked the class path and there were no old version of lucene only:
lucene-highlighter-3.0.3.jar and lucene-core-3.0.3.jar
any hints?
Thanks
Farag
--
View this message in context: http://lucene.472066.n3.nabble.com/lucene-highlighter-tp3522464p3522535.html
Sent from the Lucene - Java Users mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
RE: lucene highlighter
Posted by Uwe Schindler <uw...@thetaphi.de>.
Hi,
it looks like your classpath is containing also very old Lucene versions together with the new Lucene JAR file. Alternatively it may also be the case that contrib-highlighter.jar, contrib-memory.jar and lucene-core.jar are not from the same Lucene version. The method next(Token) is only available in ancient Lucene versions (pre-3.0). If you are using Eclipse it may be the case, that the very old Lucene versions shipped with your Eclipse installation somehow made it into your classpath.
Uwe
-----
Uwe Schindler
H.-H.-Meier-Allee 63, D-28213 Bremen
http://www.thetaphi.de
eMail: uwe@thetaphi.de
> -----Original Message-----
> From: starz10de [mailto:farag_ahmed@yahoo.com]
> Sent: Sunday, November 20, 2011 10:53 AM
> To: java-user@lucene.apache.org
> Subject: lucene highlighter
>
> Hi,
>
> I have a problem with lucene highlighter. I couldn’t make it run. The
> compilation is without error but when I run it I got this error “Exception in
> thread "main"
> java.lang.NoSuchMethodError:org.apache.lucene.analysis.TokenStream.next(Lo
> rg/apache/lucene/analysis/Token;)Lorg/apache/lucene/analysis/Token;”
>
> I am using lucene 3.0.3 core and lucene highlighter 2.4. It might be a problem
> of compatibility. I tried to use lucene highlighter 3. but I couldn’t combile he
> code and I got this error “SpanScorer can’t be resolved as a type”
>
> Here part of the code:
>
> import org.apache.lucene.search.highlight.Fragmenter;
> import org.apache.lucene.search.highlight.Highlighter;
> import org.apache.lucene.search.highlight.SimpleSpanFragmenter;
> import org.apache.lucene.search.highlight.SpanScorer;
> import org.apache.lucene.search.highlight.TokenSources;
>
> .
> .
> .
>
> for (int i = start; i < end; i++) {
> if (raw) { // output raw format
> System.out.println("doc="+hits[i].doc+" score="+hits[i].score);
> continue;
> }
>
> Document doc = searcher.doc(hits[i].doc);
> String path = doc.get("path");
> contents=doc.get("contents");
>
>
>
> // System.out.println(text+" !!!!!!!!!!!!!!!!!");
>
>
> fragments = getFragmentsWithHighlightedTerms(analyzer, query, "contents",
> contents, 5, 100);
>
>
> .
> .
> private static String[] getFragmentsWithHighlightedTerms(Analyzer analyzer,
> Query query,String fieldName, String fieldContents, int fragmentNumber, int
> fragmentSize) throws IOException
> {
>
>
>
> TokenStream stream = TokenSources.getTokenStream(fieldName,
> fieldContents, analyzer);
> SpanScorer scorer = new SpanScorer(query, fieldName, new
> CachingTokenFilter(stream));
> Fragmenter fragmenter = new SimpleSpanFragmenter(scorer, fragmentSize);
> Highlighter highlighter = new Highlighter(scorer);
> highlighter.setTextFragmenter(fragmenter);
> highlighter.setMaxDocCharsToAnalyze(Integer.MAX_VALUE);
> String[] summaries = highlighter.getBestFragments(new
> StandardAnalyzer(Version.LUCENE_CURRENT), fieldContents,contents, 3);
> System.out.println(summaries);
> // String[] fragments = highlighter.getBestFragments(stream, fieldContents,
> fragmentNumber);
> return fragments;
> }
>
> Thanks in advance
>
>
> --
> View this message in context: http://lucene.472066.n3.nabble.com/lucene-
> highlighter-tp3522464p3522464.html
> Sent from the Lucene - Java Users mailing list archive at Nabble.com.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org