You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by "O. Klein" <kl...@octoweb.nl> on 2012/01/23 15:14:28 UTC
Highlighting stopwords
Im using trunk and FVH and eventhough I filter stopwords when searching, I
would like to highlight stopwords in fragments. Using a different field
without the stopwords filter did not have the desired effect.
Is there a way to do this?
--
View this message in context: http://lucene.472066.n3.nabble.com/Highlighting-stopwords-tp3681901p3681901.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: Highlighting stopwords
Posted by "O. Klein" <kl...@octoweb.nl>.
Koji Sekiguchi wrote
>
> Uh, what you tried was that you changed the field between q and hl.q,
> that I've not expected use case when I proposed hl.q.
>
> Do you think that hl.text meats your needs?
>
> https://issues.apache.org/jira/browse/SOLR-1926?focusedCommentId=12871234&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-12871234
>
> koji
> --
> Apache Solr Query Log Visualizer
> http://soleami.com/
>
Well, If I understand it correctly, yes.
If this means that queries are analyzed like the field they are
highlighting. That would give the highlighter a lot more flexibility.
--
View this message in context: http://lucene.472066.n3.nabble.com/Highlighting-stopwords-tp3681901p3744054.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: Highlighting stopwords
Posted by "O. Klein" <kl...@octoweb.nl>.
Koji Sekiguchi wrote
>
> (12/02/14 22:25), O. Klein wrote:
>> I have not been able to find any logic in the behavior of hl.q and how it
>> analyses the query. Could you explain how it is supposed to work?
>
> Nothing special on hl.q. If you use hl.q, the value of it will be used for
> highlighting rather than the value of q. There's no tricks, I think.
>
> koji
> --
> Apache Solr Query Log Visualizer
> http://soleami.com/
>
Field definitions:
content_text (no stopwords, only synonyms in index)
content_hl (stopwords, synonyms in index and query, and only field in hl.fl)
Searching is done with edismax on content_text
1. If I use a query like hl.q=spell Check it doesn't highlight terms with
uppercase, synonyms get highlighted (all fields have LowerCaseFilterFactory)
2. hl.q=content_hl:(spell Check) also highlights terms with uppercase,
synonyms are not highlighted
4. hl.q=content_hl:(spell Check) content_text:(spell Check) highlights terms
with uppercase and synonyms, but sometimes no highlights at all.
So if 1 also highlights terms with uppercase I get the behavior I need. I
can do this on client side, but maybe it's a bug?
--
View this message in context: http://lucene.472066.n3.nabble.com/Highlighting-stopwords-tp3681901p3746817.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: Highlighting stopwords
Posted by Koji Sekiguchi <ko...@r.email.ne.jp>.
(12/02/14 22:25), O. Klein wrote:
> I have not been able to find any logic in the behavior of hl.q and how it
> analyses the query. Could you explain how it is supposed to work?
Nothing special on hl.q. If you use hl.q, the value of it will be used for
highlighting rather than the value of q. There's no tricks, I think.
> When using hl.q=content_hl:(spell Check) I now get highlighting including
> stopwords.
>
> but when using hl.q=content_hl:(SC) where SC is synonym I get no
> highlighting.
>
> Can you verify if synonyms work when using hl.q?
:
> OK I got it working by using hl.q=content_hl:(spell Check)
> content_text:(spell Check) but it makes no sense to me.
>
> only difference between the 2 fields is the use of Stopwords.
Uh, what you tried was that you changed the field between q and hl.q,
that I've not expected use case when I proposed hl.q.
Do you think that hl.text meats your needs?
https://issues.apache.org/jira/browse/SOLR-1926?focusedCommentId=12871234&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-12871234
koji
--
Apache Solr Query Log Visualizer
http://soleami.com/
Re: Highlighting stopwords
Posted by "O. Klein" <kl...@octoweb.nl>.
O. Klein wrote
>
>
> O. Klein wrote
>>
>> Hmm, now the synonyms aren't highlighted anymore.
>>
>> OK back to basic (im using trunk and FVH).
>>
>> What is the way to go about if I want to search on a field without
>> stopwords, but still want to highlight the stopwords? (and still
>> highlight synonyms and stemmed words)?
>>
>
> I made new field content_hl to prevent problems coming from copyField.
>
> When using hl.q=content_hl:(spell Check) I now get highlighting including
> stopwords.
>
> but when using hl.q=content_hl:(SC) where SC is synonym I get no
> highlighting.
>
> Can you verify if synonyms work when using hl.q?
>
OK I got it working by using hl.q=content_hl:(spell Check)
content_text:(spell Check) but it makes no sense to me.
only difference between the 2 fields is the use of Stopwords.
What's also weird is that a query like hl.q=content_spell:(SC) also
highlights synonyms, eventhough this field has no synonyms.
I have not been able to find any logic in the behavior of hl.q and how it
analyses the query. Could you explain how it is supposed to work?
--
View this message in context: http://lucene.472066.n3.nabble.com/Highlighting-stopwords-tp3681901p3743616.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: Highlighting stopwords
Posted by "O. Klein" <kl...@octoweb.nl>.
O. Klein wrote
>
> Hmm, now the synonyms aren't highlighted anymore.
>
> OK back to basic (im using trunk and FVH).
>
> What is the way to go about if I want to search on a field without
> stopwords, but still want to highlight the stopwords? (and still highlight
> synonyms and stemmed words)?
>
I made new field content_hl to prevent problems coming from copyField.
When using hl.q=content_hl:(spell Check) I now get highlighting including
stopwords.
but when using hl.q=content_hl:(SC) where SC is synonym I get no
highlighting.
Can you verify if synonyms work when using hl.q?
--
View this message in context: http://lucene.472066.n3.nabble.com/Highlighting-stopwords-tp3681901p3743317.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: Highlighting stopwords
Posted by "O. Klein" <kl...@octoweb.nl>.
Hmm, now the synonyms aren't highlighted anymore.
OK back to basic (im using trunk and FVH).
What is the way to go about if I want to search on a field without
stopwords, but still want to highlight the stopwords? (and still highlight
synonyms and stemmed words)?
--
View this message in context: http://lucene.472066.n3.nabble.com/Highlighting-stopwords-tp3681901p3740280.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: Highlighting stopwords
Posted by Koji Sekiguchi <ko...@r.email.ne.jp>.
> I got it fixed now I think.
>
> I thought that if you used it like "hl.q=spell Checker" it would use the
> query analysis of the field that was being highlighted as default. But in my
> case it needs to be "hl.q=content_hl:(spell Checker)" for it to work. The
> behavour I got default made no sense whatsoever.
Right. You can also use localParams for hl.q.
> Could you be so kind to explain a bit more how hl.q is supposed to work and
> with some examples?
My motivation was described here when I proposed hl.q:
https://issues.apache.org/jira/browse/SOLR-1926
But I think hl.q can be used to overwrite q for highlighting.
For example, if q is a boolean query of "A AND B", you can expand it to
"A AND B AND C" or shrink it to "A" for hl.q.
koji
--
http://www.rondhuit.com/en/
Re: Highlighting stopwords
Posted by "O. Klein" <kl...@octoweb.nl>.
Koji Sekiguchi wrote
>
> (12/02/11 21:19), O. Klein wrote:
>>
>> Koji Sekiguchi wrote
>>>
>>> (12/01/24 9:31), O. Klein wrote:
>>>> Let's say I search for "spellcheck solr" on a website that only
>>>> contains
>>>> info about Solr, so "solr" was added to the stopwords.txt. The query
>>>> that
>>>> will be parsed then (dismax) will not contain the term "solr".
>>>>
>>>> So fragments won't contain highlights of the term "solr". So when a
>>>> fragment
>>>> with the highlighted term "spellcheck" is generated, it would be less
>>>> confusing for people who don't know how search engines work to also
>>>> highlight the term "solr".
>>>>
>>>> So my first test was to have a field with StopFilterFactory and search
>>>> on
>>>> that field, while using another field without StopFilterFactory to
>>>> highlight
>>>> on. This didn't do the trick.
>>>
>>> Are you saying that using hl.q parameter on highlight field while using
>>> q
>>> on
>>> the search field that has StopFilter and hl.q doesn't work for you?
>>>
>>> koji
>>> --
>>> http://www.rondhuit.com/en/
>>>
>>
>> At first glance using hl.q did the trick. I just have problems when I am
>> using terms with uppercase. Eventhough I use<filter
>> class="solr.LowerCaseFilterFactory"/> on the highlighted field in both
>> query
>> and index I do get search results, but just no highlights (lowercasing
>> the
>> terms fixes the problem).
>>
>> Can someone confirm whether this is a bug?
>
> I don't see your situation. Giving us concrete examples (especially
> request parameters
> including q and hl.q) would help a lot!
>
> koji
> --
> http://www.rondhuit.com/en/
>
I got it fixed now I think.
I thought that if you used it like "hl.q=spell Checker" it would use the
query analysis of the field that was being highlighted as default. But in my
case it needs to be "hl.q=content_hl:(spell Checker)" for it to work. The
behavour I got default made no sense whatsoever.
Could you be so kind to explain a bit more how hl.q is supposed to work and
with some examples?
Thanx.
--
View this message in context: http://lucene.472066.n3.nabble.com/Highlighting-stopwords-tp3681901p3740114.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: Highlighting stopwords
Posted by Koji Sekiguchi <ko...@r.email.ne.jp>.
(12/02/11 21:19), O. Klein wrote:
>
> Koji Sekiguchi wrote
>>
>> (12/01/24 9:31), O. Klein wrote:
>>> Let's say I search for "spellcheck solr" on a website that only contains
>>> info about Solr, so "solr" was added to the stopwords.txt. The query that
>>> will be parsed then (dismax) will not contain the term "solr".
>>>
>>> So fragments won't contain highlights of the term "solr". So when a
>>> fragment
>>> with the highlighted term "spellcheck" is generated, it would be less
>>> confusing for people who don't know how search engines work to also
>>> highlight the term "solr".
>>>
>>> So my first test was to have a field with StopFilterFactory and search on
>>> that field, while using another field without StopFilterFactory to
>>> highlight
>>> on. This didn't do the trick.
>>
>> Are you saying that using hl.q parameter on highlight field while using q
>> on
>> the search field that has StopFilter and hl.q doesn't work for you?
>>
>> koji
>> --
>> http://www.rondhuit.com/en/
>>
>
> At first glance using hl.q did the trick. I just have problems when I am
> using terms with uppercase. Eventhough I use<filter
> class="solr.LowerCaseFilterFactory"/> on the highlighted field in both query
> and index I do get search results, but just no highlights (lowercasing the
> terms fixes the problem).
>
> Can someone confirm whether this is a bug?
I don't see your situation. Giving us concrete examples (especially request parameters
including q and hl.q) would help a lot!
koji
--
http://www.rondhuit.com/en/
Re: Highlighting stopwords
Posted by "O. Klein" <kl...@octoweb.nl>.
Koji Sekiguchi wrote
>
> (12/01/24 9:31), O. Klein wrote:
>> Let's say I search for "spellcheck solr" on a website that only contains
>> info about Solr, so "solr" was added to the stopwords.txt. The query that
>> will be parsed then (dismax) will not contain the term "solr".
>>
>> So fragments won't contain highlights of the term "solr". So when a
>> fragment
>> with the highlighted term "spellcheck" is generated, it would be less
>> confusing for people who don't know how search engines work to also
>> highlight the term "solr".
>>
>> So my first test was to have a field with StopFilterFactory and search on
>> that field, while using another field without StopFilterFactory to
>> highlight
>> on. This didn't do the trick.
>
> Are you saying that using hl.q parameter on highlight field while using q
> on
> the search field that has StopFilter and hl.q doesn't work for you?
>
> koji
> --
> http://www.rondhuit.com/en/
>
At first glance using hl.q did the trick. I just have problems when I am
using terms with uppercase. Eventhough I use <filter
class="solr.LowerCaseFilterFactory"/> on the highlighted field in both query
and index I do get search results, but just no highlights (lowercasing the
terms fixes the problem).
Can someone confirm whether this is a bug?
Thank you.
--
View this message in context: http://lucene.472066.n3.nabble.com/Highlighting-stopwords-tp3681901p3734892.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: Highlighting stopwords
Posted by "O. Klein" <kl...@octoweb.nl>.
Ah, I never used the hl.q
That did the trick. Thanx!
--
View this message in context: http://lucene.472066.n3.nabble.com/Highlighting-stopwords-tp3681901p3684245.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: Highlighting stopwords
Posted by Koji Sekiguchi <ko...@r.email.ne.jp>.
(12/01/24 9:31), O. Klein wrote:
> Let's say I search for "spellcheck solr" on a website that only contains
> info about Solr, so "solr" was added to the stopwords.txt. The query that
> will be parsed then (dismax) will not contain the term "solr".
>
> So fragments won't contain highlights of the term "solr". So when a fragment
> with the highlighted term "spellcheck" is generated, it would be less
> confusing for people who don't know how search engines work to also
> highlight the term "solr".
>
> So my first test was to have a field with StopFilterFactory and search on
> that field, while using another field without StopFilterFactory to highlight
> on. This didn't do the trick.
Are you saying that using hl.q parameter on highlight field while using q on
the search field that has StopFilter and hl.q doesn't work for you?
koji
--
http://www.rondhuit.com/en/
Re: Highlighting stopwords
Posted by "O. Klein" <kl...@octoweb.nl>.
Koji Sekiguchi wrote
>
> (12/01/23 23:14), O. Klein wrote:
>> Im using trunk and FVH and eventhough I filter stopwords when searching,
>> I
>> would like to highlight stopwords in fragments. Using a different field
>> without the stopwords filter did not have the desired effect.
>
> Please provide more info. In particular, how your query look like, the
> expected results,
> actual results and the filter definition in schema.xml.
>
> Thank you,
>
> koji
> --
> http://www.rondhuit.com/en/
>
Let's say I search for "spellcheck solr" on a website that only contains
info about Solr, so "solr" was added to the stopwords.txt. The query that
will be parsed then (dismax) will not contain the term "solr".
So fragments won't contain highlights of the term "solr". So when a fragment
with the highlighted term "spellcheck" is generated, it would be less
confusing for people who don't know how search engines work to also
highlight the term "solr".
So my first test was to have a field with StopFilterFactory and search on
that field, while using another field without StopFilterFactory to highlight
on. This didn't do the trick.
So thinking about it some more, I think some terms don't need to be
highlighted (e.g. "the") and some need to be highlighted, but don't need to
affect the scoring of a document (e.g. "solr") and others need to be
highlighted and affect scoring (e.g. "spellcheck").
What are your thoughts on this?
--
View this message in context: http://lucene.472066.n3.nabble.com/Highlighting-stopwords-tp3681901p3683452.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: Highlighting stopwords
Posted by Koji Sekiguchi <ko...@r.email.ne.jp>.
(12/01/23 23:14), O. Klein wrote:
> Im using trunk and FVH and eventhough I filter stopwords when searching, I
> would like to highlight stopwords in fragments. Using a different field
> without the stopwords filter did not have the desired effect.
Please provide more info. In particular, how your query look like, the expected results,
actual results and the filter definition in schema.xml.
Thank you,
koji
--
http://www.rondhuit.com/en/