You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by Walter Underwood <wu...@wunderwood.org> on 2017/06/28 21:13:43 UTC

Not highlighting "and" and "or"?

Is there some special casing in the highlighter to skip query syntax words? The words “and” and “or” don’t get highlighted.

This is in 6.5.0.

       <str name="hl.fl">question</str>
       <str name="hl.encoder">html</str>
       <str name="hl.fragsize">440</str>
       <str name="hl.method">fastVector</str>
       <str name="hl.snippets">1</str>

wunder
Walter Underwood
wunder@wunderwood.org
http://observer.wunderwood.org/  (my blog)



Re: Not highlighting "and" and "or"?

Posted by Walter Underwood <wu...@wunderwood.org>.
Setting lowercaseOperators=false for the request handler defaults fixes this. Probably also fixes some relevance anomalies.

Thanks!

wunder
Walter Underwood
wunder@wunderwood.org
http://observer.wunderwood.org/  (my blog)


> On Jun 29, 2017, at 6:38 AM, Shawn Heisey <ap...@elyograg.org> wrote:
> 
> On 6/28/2017 10:46 PM, Walter Underwood wrote:
>> Hmm, “and” is missing from the individual terms but present in the phrase.
>> 
>>    "rawquerystring":"once and again",
>>    "querystring":"once and again",
>>    "parsedquery":"(+(+DisjunctionMaxQuery(((concept_ai_concepts_names_default:once)^2.0 | (question:once)^2.0 | subjectNames:once | (bookTitle_text:once)^4.0)) +DisjunctionMaxQuery(((concept_ai_concepts_names_default:again)^2.0 | (question:again)^2.0 | subjectNames:again | (bookTitle_text:again)^4.0))) DisjunctionMaxQuery(((bookTitle_text:\"once and again\")^8.0 | (question:\"once and again\")^4.0 | (concept_ai_concepts_names_default:\"once and again\")^4.0 | (subjectNames:\"once and again\")^2.0)))/no_coord",
>>    "parsedquery_toString":"+(+((concept_ai_concepts_names_default:once)^2.0 | (question:once)^2.0 | subjectNames:once | (bookTitle_text:once)^4.0) +((concept_ai_concepts_names_default:again)^2.0 | (question:again)^2.0 | subjectNames:again | (bookTitle_text:again)^4.0)) ((bookTitle_text:\"once and again\")^8.0 | (question:\"once and again\")^4.0 | (concept_ai_concepts_names_default:\"once and again\")^4.0 | (subjectNames:\"once and again\")^2.0)",
>>    "QParser":"ExtendedDismaxQParser",
> 
> I would guess that "and" is being interpreted as an operator, because
> lowercaseOperators defaults to true with edismax.  This is changing in 7.0.
> 
> https://issues.apache.org/jira/browse/SOLR-4646
> 
> Thanks,
> Shawn
> 


Re: Not highlighting "and" and "or"?

Posted by Shawn Heisey <ap...@elyograg.org>.
On 6/28/2017 10:46 PM, Walter Underwood wrote:
> Hmm, “and” is missing from the individual terms but present in the phrase.
>
>     "rawquerystring":"once and again",
>     "querystring":"once and again",
>     "parsedquery":"(+(+DisjunctionMaxQuery(((concept_ai_concepts_names_default:once)^2.0 | (question:once)^2.0 | subjectNames:once | (bookTitle_text:once)^4.0)) +DisjunctionMaxQuery(((concept_ai_concepts_names_default:again)^2.0 | (question:again)^2.0 | subjectNames:again | (bookTitle_text:again)^4.0))) DisjunctionMaxQuery(((bookTitle_text:\"once and again\")^8.0 | (question:\"once and again\")^4.0 | (concept_ai_concepts_names_default:\"once and again\")^4.0 | (subjectNames:\"once and again\")^2.0)))/no_coord",
>     "parsedquery_toString":"+(+((concept_ai_concepts_names_default:once)^2.0 | (question:once)^2.0 | subjectNames:once | (bookTitle_text:once)^4.0) +((concept_ai_concepts_names_default:again)^2.0 | (question:again)^2.0 | subjectNames:again | (bookTitle_text:again)^4.0)) ((bookTitle_text:\"once and again\")^8.0 | (question:\"once and again\")^4.0 | (concept_ai_concepts_names_default:\"once and again\")^4.0 | (subjectNames:\"once and again\")^2.0)",
>     "QParser":"ExtendedDismaxQParser",

I would guess that "and" is being interpreted as an operator, because
lowercaseOperators defaults to true with edismax.  This is changing in 7.0.

https://issues.apache.org/jira/browse/SOLR-4646

Thanks,
Shawn


Re: Not highlighting "and" and "or"?

Posted by Walter Underwood <wu...@wunderwood.org>.
Hmm, “and” is missing from the individual terms but present in the phrase.

    "rawquerystring":"once and again",
    "querystring":"once and again",
    "parsedquery":"(+(+DisjunctionMaxQuery(((concept_ai_concepts_names_default:once)^2.0 | (question:once)^2.0 | subjectNames:once | (bookTitle_text:once)^4.0)) +DisjunctionMaxQuery(((concept_ai_concepts_names_default:again)^2.0 | (question:again)^2.0 | subjectNames:again | (bookTitle_text:again)^4.0))) DisjunctionMaxQuery(((bookTitle_text:\"once and again\")^8.0 | (question:\"once and again\")^4.0 | (concept_ai_concepts_names_default:\"once and again\")^4.0 | (subjectNames:\"once and again\")^2.0)))/no_coord",
    "parsedquery_toString":"+(+((concept_ai_concepts_names_default:once)^2.0 | (question:once)^2.0 | subjectNames:once | (bookTitle_text:once)^4.0) +((concept_ai_concepts_names_default:again)^2.0 | (question:again)^2.0 | subjectNames:again | (bookTitle_text:again)^4.0)) ((bookTitle_text:\"once and again\")^8.0 | (question:\"once and again\")^4.0 | (concept_ai_concepts_names_default:\"once and again\")^4.0 | (subjectNames:\"once and again\")^2.0)",
    "QParser":"ExtendedDismaxQParser",


wunder
Walter Underwood
wunder@wunderwood.org
http://observer.wunderwood.org/  (my blog)


> On Jun 28, 2017, at 9:27 PM, David Smiley <da...@gmail.com> wrote:
> 
> Hi Walter,
> No they are not.  Does debug=query show that these words are in your parsed
> query?
> 
> On Wed, Jun 28, 2017 at 5:13 PM Walter Underwood <wu...@wunderwood.org>
> wrote:
> 
>> Is there some special casing in the highlighter to skip query syntax
>> words? The words “and” and “or” don’t get highlighted.
>> 
>> This is in 6.5.0.
>> 
>>       <str name="hl.fl">question</str>
>>       <str name="hl.encoder">html</str>
>>       <str name="hl.fragsize">440</str>
>>       <str name="hl.method">fastVector</str>
>>       <str name="hl.snippets">1</str>
>> 
>> wunder
>> Walter Underwood
>> wunder@wunderwood.org
>> http://observer.wunderwood.org/  (my blog)
>> 
>> 
>> --
> Lucene/Solr Search Committer, Consultant, Developer, Author, Speaker
> LinkedIn: http://linkedin.com/in/davidwsmiley | Book:
> http://www.solrenterprisesearchserver.com


Re: Not highlighting "and" and "or"?

Posted by David Smiley <da...@gmail.com>.
Hi Walter,
No they are not.  Does debug=query show that these words are in your parsed
query?

On Wed, Jun 28, 2017 at 5:13 PM Walter Underwood <wu...@wunderwood.org>
wrote:

> Is there some special casing in the highlighter to skip query syntax
> words? The words “and” and “or” don’t get highlighted.
>
> This is in 6.5.0.
>
>        <str name="hl.fl">question</str>
>        <str name="hl.encoder">html</str>
>        <str name="hl.fragsize">440</str>
>        <str name="hl.method">fastVector</str>
>        <str name="hl.snippets">1</str>
>
> wunder
> Walter Underwood
> wunder@wunderwood.org
> http://observer.wunderwood.org/  (my blog)
>
>
> --
Lucene/Solr Search Committer, Consultant, Developer, Author, Speaker
LinkedIn: http://linkedin.com/in/davidwsmiley | Book:
http://www.solrenterprisesearchserver.com

Re: Not highlighting "and" and "or"?

Posted by Rick Leir <rl...@leirtech.com>.
Created Wish SOLR-10992 
<https://issues.apache.org/jira/browse/SOLR-10992>. Maybe I am a 
troublemaker, the example configs will need to be tested after the 
change is made and that will be a bother. Any tips on how to easily do that?

Thanks

Rick


On 2017-06-30 10:54 AM, Walter Underwood wrote:
> removing the stopword filter from the example configs. It is not a “best practice” or even a recommended practice.


Re: Not highlighting "and" and "or"?

Posted by Walter Underwood <wu...@wunderwood.org>.
I would agree with removing the stopword filter from the example configs. It is not a “best practice” or even a recommended practice.

wunder
Walter Underwood
wunder@wunderwood.org
http://observer.wunderwood.org/  (my blog)


> On Jun 29, 2017, at 8:01 PM, Rick Leir <rl...@leirtech.com> wrote:
> 
> Walter, Erick, David
> Thanks for the info. Maybe the default for stopwords should be disabled? 
> Cheers -- Rick
> 
> On June 29, 2017 5:14:16 PM EDT, Walter Underwood <wu...@wunderwood.org> wrote:
>> My blog post has a list of movie titles. I forgot to list the TV series
>> “Once and Again”.
>> 
>> Some bands that are not searchable with stopwords:
>> 
>> * The Who
>> * Was (not Was)
>> * The The
>> 
>> wunder
>> Walter Underwood
>> wunder@wunderwood.org
>> http://observer.wunderwood.org/  (my blog)
>> 
>> 
>>> On Jun 29, 2017, at 2:09 PM, Erick Erickson <er...@gmail.com>
>> wrote:
>>> 
>>> bq: Mostly, stopwords were a performance hack back when people ran
>>> search engines on 16-bit machines
>>> 
>>> Ah, _those_ were the days when programmers were _real_ programmers.
>>> Actually I'm glad they're gone but that's another story.
>>> 
>>> "to be or not to be". Can't search that if you enable stopwords.
>>> 
>>> Chris Hostetter wrote a fun blog on the fact that Lucene query
>> parsers
>>> are not strict boolean logic with the title "Why Not AND, OR, And
>> NOT"
>>> purposely choosing that title as it's totally unsearchable if you're
>>> using stopwords.
>>> 
>>> FWIW,
>>> Erick
>>> 
>>> On Thu, Jun 29, 2017 at 1:57 PM, David Hastings
>>> <ha...@gmail.com> wrote:
>>>> Agreed.  Stop words from the moment I started using them caused
>> complaints
>>>> and problems right off the bat.  They may have been implemented less
>> than a
>>>> week before needing a re-index to fix all the problems they caused.
>>>> 
>>>> On Thu, Jun 29, 2017 at 4:55 PM, Walter Underwood
>> <wu...@wunderwood.org>
>>>> wrote:
>>>> 
>>>>> Ultraseek (and Infoseek) never used stopwords. They cause odd
>> failures,
>>>>> like not being able to search for “Vitamin A”.
>>>>> 
>>>>> Stopwords are an on/off approach to term frequency. idf is a
>> proportional
>>>>> approach. Once you have idf, you don’t need stopwords.
>>>>> 
>>>>> When I was bringing up Solr for Netflix, I started with an analysis
>> chain
>>>>> that used stopwords. A surprising number of movie titles entirely
>>>>> disappeared. I wrote a blog post about it. Ten years ago!
>>>>> 
>>>>> 
>> https://observer.wunderwood.org/2007/05/31/do-all-stopword-queries-matter/
>>>>> 
>>>>> Mostly, stopwords were a performance hack back when people ran
>> search
>>>>> engines on 16-bit machines. Neither disks nor RAM were big enough
>> to hold
>>>>> the posting lists for common words.
>>>>> 
>>>>> wunder
>>>>> Walter Underwood
>>>>> wunder@wunderwood.org
>>>>> http://observer.wunderwood.org/  (my blog)
>>>>> 
>>>>> 
>>>>>> On Jun 29, 2017, at 1:46 PM, Rick Leir <rl...@leirtech.com> wrote:
>>>>>> 
>>>>>> Walter
>>>>>> Sorry for the tangent, but the stopwords feature sounds useful.
>> You say
>>>>> you do not use this? Did Ultraseek not do it either?
>>>>>> Thanks
>>>>>> Rick
>>>>>> 
>>>>>> On June 29, 2017 10:53:42 AM EDT, Walter Underwood <
>>>>> wunder@wunderwood.org> wrote:
>>>>>>> Nope. Haven’t used stopwords for the last 20 years.
>>>>>>> 
>>>>>>> I wonder if lowercaseOperators is true. The docs don’t give the
>> default
>>>>>>> value for that in edismax.
>>>>>>> 
>>>>>>> https://lucene.apache.org/solr/guide/6_6/the-extended-
>>>>> dismax-query-parser.html
>>>>>>> 
>>>>>>> wunder
>>>>>>> Walter Underwood
>>>>>>> wunder@wunderwood.org
>>>>>>> http://observer.wunderwood.org/  (my blog)
>>>>>>> 
>>>>>>> 
>>>>>>>> On Jun 29, 2017, at 4:42 AM, Rick Leir <rl...@leirtech.com>
>> wrote:
>>>>>>>> 
>>>>>>>> Stopwords?
>>>>>>>> 
>>>>>>>> On June 28, 2017 5:13:43 PM EDT, Walter Underwood
>>>>>>> <wu...@wunderwood.org> wrote:
>>>>>>>>> Is there some special casing in the highlighter to skip query
>> syntax
>>>>>>>>> words? The words “and” and “or” don’t get highlighted.
>>>>>>>>> 
>>>>>>>>> This is in 6.5.0.
>>>>>>>>> 
>>>>>>>>>   <str name="hl.fl">question</str>
>>>>>>>>>   <str name="hl.encoder">html</str>
>>>>>>>>>   <str name="hl.fragsize">440</str>
>>>>>>>>>   <str name="hl.method">fastVector</str>
>>>>>>>>>   <str name="hl.snippets">1</str>
>>>>>>>>> 
>>>>>>>>> wunder
>>>>>>>>> Walter Underwood
>>>>>>>>> wunder@wunderwood.org
>>>>>>>>> http://observer.wunderwood.org/  (my blog)
>>>>>>>> 
>>>>>>>> --
>>>>>>>> Sorry for being brief. Alternate email is rickleir at yahoo dot
>> com
>>>>>> 
>>>>>> --
>>>>>> Sorry for being brief. Alternate email is rickleir at yahoo dot
>> com
>>>>>> --
>>>>>> Sorry for being brief. Alternate email is rickleir at yahoo dot
>> com
>>>>> 
>>>>> 
> 
> -- 
> Sorry for being brief. Alternate email is rickleir at yahoo dot com


Re: Not highlighting "and" and "or"?

Posted by Rick Leir <rl...@leirtech.com>.
Walter, Erick, David
Thanks for the info. Maybe the default for stopwords should be disabled? 
Cheers -- Rick

On June 29, 2017 5:14:16 PM EDT, Walter Underwood <wu...@wunderwood.org> wrote:
>My blog post has a list of movie titles. I forgot to list the TV series
>“Once and Again”.
>
>Some bands that are not searchable with stopwords:
>
>* The Who
>* Was (not Was)
>* The The
>
>wunder
>Walter Underwood
>wunder@wunderwood.org
>http://observer.wunderwood.org/  (my blog)
>
>
>> On Jun 29, 2017, at 2:09 PM, Erick Erickson <er...@gmail.com>
>wrote:
>> 
>> bq: Mostly, stopwords were a performance hack back when people ran
>> search engines on 16-bit machines
>> 
>> Ah, _those_ were the days when programmers were _real_ programmers.
>> Actually I'm glad they're gone but that's another story.
>> 
>> "to be or not to be". Can't search that if you enable stopwords.
>> 
>> Chris Hostetter wrote a fun blog on the fact that Lucene query
>parsers
>> are not strict boolean logic with the title "Why Not AND, OR, And
>NOT"
>> purposely choosing that title as it's totally unsearchable if you're
>> using stopwords.
>> 
>> FWIW,
>> Erick
>> 
>> On Thu, Jun 29, 2017 at 1:57 PM, David Hastings
>> <ha...@gmail.com> wrote:
>>> Agreed.  Stop words from the moment I started using them caused
>complaints
>>> and problems right off the bat.  They may have been implemented less
>than a
>>> week before needing a re-index to fix all the problems they caused.
>>> 
>>> On Thu, Jun 29, 2017 at 4:55 PM, Walter Underwood
><wu...@wunderwood.org>
>>> wrote:
>>> 
>>>> Ultraseek (and Infoseek) never used stopwords. They cause odd
>failures,
>>>> like not being able to search for “Vitamin A”.
>>>> 
>>>> Stopwords are an on/off approach to term frequency. idf is a
>proportional
>>>> approach. Once you have idf, you don’t need stopwords.
>>>> 
>>>> When I was bringing up Solr for Netflix, I started with an analysis
>chain
>>>> that used stopwords. A surprising number of movie titles entirely
>>>> disappeared. I wrote a blog post about it. Ten years ago!
>>>> 
>>>>
>https://observer.wunderwood.org/2007/05/31/do-all-stopword-queries-matter/
>>>> 
>>>> Mostly, stopwords were a performance hack back when people ran
>search
>>>> engines on 16-bit machines. Neither disks nor RAM were big enough
>to hold
>>>> the posting lists for common words.
>>>> 
>>>> wunder
>>>> Walter Underwood
>>>> wunder@wunderwood.org
>>>> http://observer.wunderwood.org/  (my blog)
>>>> 
>>>> 
>>>>> On Jun 29, 2017, at 1:46 PM, Rick Leir <rl...@leirtech.com> wrote:
>>>>> 
>>>>> Walter
>>>>> Sorry for the tangent, but the stopwords feature sounds useful.
>You say
>>>> you do not use this? Did Ultraseek not do it either?
>>>>> Thanks
>>>>> Rick
>>>>> 
>>>>> On June 29, 2017 10:53:42 AM EDT, Walter Underwood <
>>>> wunder@wunderwood.org> wrote:
>>>>>> Nope. Haven’t used stopwords for the last 20 years.
>>>>>> 
>>>>>> I wonder if lowercaseOperators is true. The docs don’t give the
>default
>>>>>> value for that in edismax.
>>>>>> 
>>>>>> https://lucene.apache.org/solr/guide/6_6/the-extended-
>>>> dismax-query-parser.html
>>>>>> 
>>>>>> wunder
>>>>>> Walter Underwood
>>>>>> wunder@wunderwood.org
>>>>>> http://observer.wunderwood.org/  (my blog)
>>>>>> 
>>>>>> 
>>>>>>> On Jun 29, 2017, at 4:42 AM, Rick Leir <rl...@leirtech.com>
>wrote:
>>>>>>> 
>>>>>>> Stopwords?
>>>>>>> 
>>>>>>> On June 28, 2017 5:13:43 PM EDT, Walter Underwood
>>>>>> <wu...@wunderwood.org> wrote:
>>>>>>>> Is there some special casing in the highlighter to skip query
>syntax
>>>>>>>> words? The words “and” and “or” don’t get highlighted.
>>>>>>>> 
>>>>>>>> This is in 6.5.0.
>>>>>>>> 
>>>>>>>>    <str name="hl.fl">question</str>
>>>>>>>>    <str name="hl.encoder">html</str>
>>>>>>>>    <str name="hl.fragsize">440</str>
>>>>>>>>    <str name="hl.method">fastVector</str>
>>>>>>>>    <str name="hl.snippets">1</str>
>>>>>>>> 
>>>>>>>> wunder
>>>>>>>> Walter Underwood
>>>>>>>> wunder@wunderwood.org
>>>>>>>> http://observer.wunderwood.org/  (my blog)
>>>>>>> 
>>>>>>> --
>>>>>>> Sorry for being brief. Alternate email is rickleir at yahoo dot
>com
>>>>> 
>>>>> --
>>>>> Sorry for being brief. Alternate email is rickleir at yahoo dot
>com
>>>>> --
>>>>> Sorry for being brief. Alternate email is rickleir at yahoo dot
>com
>>>> 
>>>> 

-- 
Sorry for being brief. Alternate email is rickleir at yahoo dot com 

Re: Not highlighting "and" and "or"?

Posted by Walter Underwood <wu...@wunderwood.org>.
My blog post has a list of movie titles. I forgot to list the TV series “Once and Again”.

Some bands that are not searchable with stopwords:

* The Who
* Was (not Was)
* The The

wunder
Walter Underwood
wunder@wunderwood.org
http://observer.wunderwood.org/  (my blog)


> On Jun 29, 2017, at 2:09 PM, Erick Erickson <er...@gmail.com> wrote:
> 
> bq: Mostly, stopwords were a performance hack back when people ran
> search engines on 16-bit machines
> 
> Ah, _those_ were the days when programmers were _real_ programmers.
> Actually I'm glad they're gone but that's another story.
> 
> "to be or not to be". Can't search that if you enable stopwords.
> 
> Chris Hostetter wrote a fun blog on the fact that Lucene query parsers
> are not strict boolean logic with the title "Why Not AND, OR, And NOT"
> purposely choosing that title as it's totally unsearchable if you're
> using stopwords.
> 
> FWIW,
> Erick
> 
> On Thu, Jun 29, 2017 at 1:57 PM, David Hastings
> <ha...@gmail.com> wrote:
>> Agreed.  Stop words from the moment I started using them caused complaints
>> and problems right off the bat.  They may have been implemented less than a
>> week before needing a re-index to fix all the problems they caused.
>> 
>> On Thu, Jun 29, 2017 at 4:55 PM, Walter Underwood <wu...@wunderwood.org>
>> wrote:
>> 
>>> Ultraseek (and Infoseek) never used stopwords. They cause odd failures,
>>> like not being able to search for “Vitamin A”.
>>> 
>>> Stopwords are an on/off approach to term frequency. idf is a proportional
>>> approach. Once you have idf, you don’t need stopwords.
>>> 
>>> When I was bringing up Solr for Netflix, I started with an analysis chain
>>> that used stopwords. A surprising number of movie titles entirely
>>> disappeared. I wrote a blog post about it. Ten years ago!
>>> 
>>> https://observer.wunderwood.org/2007/05/31/do-all-stopword-queries-matter/
>>> 
>>> Mostly, stopwords were a performance hack back when people ran search
>>> engines on 16-bit machines. Neither disks nor RAM were big enough to hold
>>> the posting lists for common words.
>>> 
>>> wunder
>>> Walter Underwood
>>> wunder@wunderwood.org
>>> http://observer.wunderwood.org/  (my blog)
>>> 
>>> 
>>>> On Jun 29, 2017, at 1:46 PM, Rick Leir <rl...@leirtech.com> wrote:
>>>> 
>>>> Walter
>>>> Sorry for the tangent, but the stopwords feature sounds useful. You say
>>> you do not use this? Did Ultraseek not do it either?
>>>> Thanks
>>>> Rick
>>>> 
>>>> On June 29, 2017 10:53:42 AM EDT, Walter Underwood <
>>> wunder@wunderwood.org> wrote:
>>>>> Nope. Haven’t used stopwords for the last 20 years.
>>>>> 
>>>>> I wonder if lowercaseOperators is true. The docs don’t give the default
>>>>> value for that in edismax.
>>>>> 
>>>>> https://lucene.apache.org/solr/guide/6_6/the-extended-
>>> dismax-query-parser.html
>>>>> 
>>>>> wunder
>>>>> Walter Underwood
>>>>> wunder@wunderwood.org
>>>>> http://observer.wunderwood.org/  (my blog)
>>>>> 
>>>>> 
>>>>>> On Jun 29, 2017, at 4:42 AM, Rick Leir <rl...@leirtech.com> wrote:
>>>>>> 
>>>>>> Stopwords?
>>>>>> 
>>>>>> On June 28, 2017 5:13:43 PM EDT, Walter Underwood
>>>>> <wu...@wunderwood.org> wrote:
>>>>>>> Is there some special casing in the highlighter to skip query syntax
>>>>>>> words? The words “and” and “or” don’t get highlighted.
>>>>>>> 
>>>>>>> This is in 6.5.0.
>>>>>>> 
>>>>>>>    <str name="hl.fl">question</str>
>>>>>>>    <str name="hl.encoder">html</str>
>>>>>>>    <str name="hl.fragsize">440</str>
>>>>>>>    <str name="hl.method">fastVector</str>
>>>>>>>    <str name="hl.snippets">1</str>
>>>>>>> 
>>>>>>> wunder
>>>>>>> Walter Underwood
>>>>>>> wunder@wunderwood.org
>>>>>>> http://observer.wunderwood.org/  (my blog)
>>>>>> 
>>>>>> --
>>>>>> Sorry for being brief. Alternate email is rickleir at yahoo dot com
>>>> 
>>>> --
>>>> Sorry for being brief. Alternate email is rickleir at yahoo dot com
>>>> --
>>>> Sorry for being brief. Alternate email is rickleir at yahoo dot com
>>> 
>>> 


Re: Not highlighting "and" and "or"?

Posted by Erick Erickson <er...@gmail.com>.
bq: Mostly, stopwords were a performance hack back when people ran
search engines on 16-bit machines

Ah, _those_ were the days when programmers were _real_ programmers.
Actually I'm glad they're gone but that's another story.

"to be or not to be". Can't search that if you enable stopwords.

Chris Hostetter wrote a fun blog on the fact that Lucene query parsers
are not strict boolean logic with the title "Why Not AND, OR, And NOT"
purposely choosing that title as it's totally unsearchable if you're
using stopwords.

FWIW,
Erick

On Thu, Jun 29, 2017 at 1:57 PM, David Hastings
<ha...@gmail.com> wrote:
> Agreed.  Stop words from the moment I started using them caused complaints
> and problems right off the bat.  They may have been implemented less than a
> week before needing a re-index to fix all the problems they caused.
>
> On Thu, Jun 29, 2017 at 4:55 PM, Walter Underwood <wu...@wunderwood.org>
> wrote:
>
>> Ultraseek (and Infoseek) never used stopwords. They cause odd failures,
>> like not being able to search for “Vitamin A”.
>>
>> Stopwords are an on/off approach to term frequency. idf is a proportional
>> approach. Once you have idf, you don’t need stopwords.
>>
>> When I was bringing up Solr for Netflix, I started with an analysis chain
>> that used stopwords. A surprising number of movie titles entirely
>> disappeared. I wrote a blog post about it. Ten years ago!
>>
>> https://observer.wunderwood.org/2007/05/31/do-all-stopword-queries-matter/
>>
>> Mostly, stopwords were a performance hack back when people ran search
>> engines on 16-bit machines. Neither disks nor RAM were big enough to hold
>> the posting lists for common words.
>>
>> wunder
>> Walter Underwood
>> wunder@wunderwood.org
>> http://observer.wunderwood.org/  (my blog)
>>
>>
>> > On Jun 29, 2017, at 1:46 PM, Rick Leir <rl...@leirtech.com> wrote:
>> >
>> > Walter
>> > Sorry for the tangent, but the stopwords feature sounds useful. You say
>> you do not use this? Did Ultraseek not do it either?
>> > Thanks
>> > Rick
>> >
>> > On June 29, 2017 10:53:42 AM EDT, Walter Underwood <
>> wunder@wunderwood.org> wrote:
>> >> Nope. Haven’t used stopwords for the last 20 years.
>> >>
>> >> I wonder if lowercaseOperators is true. The docs don’t give the default
>> >> value for that in edismax.
>> >>
>> >> https://lucene.apache.org/solr/guide/6_6/the-extended-
>> dismax-query-parser.html
>> >>
>> >> wunder
>> >> Walter Underwood
>> >> wunder@wunderwood.org
>> >> http://observer.wunderwood.org/  (my blog)
>> >>
>> >>
>> >>> On Jun 29, 2017, at 4:42 AM, Rick Leir <rl...@leirtech.com> wrote:
>> >>>
>> >>> Stopwords?
>> >>>
>> >>> On June 28, 2017 5:13:43 PM EDT, Walter Underwood
>> >> <wu...@wunderwood.org> wrote:
>> >>>> Is there some special casing in the highlighter to skip query syntax
>> >>>> words? The words “and” and “or” don’t get highlighted.
>> >>>>
>> >>>> This is in 6.5.0.
>> >>>>
>> >>>>     <str name="hl.fl">question</str>
>> >>>>     <str name="hl.encoder">html</str>
>> >>>>     <str name="hl.fragsize">440</str>
>> >>>>     <str name="hl.method">fastVector</str>
>> >>>>     <str name="hl.snippets">1</str>
>> >>>>
>> >>>> wunder
>> >>>> Walter Underwood
>> >>>> wunder@wunderwood.org
>> >>>> http://observer.wunderwood.org/  (my blog)
>> >>>
>> >>> --
>> >>> Sorry for being brief. Alternate email is rickleir at yahoo dot com
>> >
>> > --
>> > Sorry for being brief. Alternate email is rickleir at yahoo dot com
>> > --
>> > Sorry for being brief. Alternate email is rickleir at yahoo dot com
>>
>>

Re: Not highlighting "and" and "or"?

Posted by David Hastings <ha...@gmail.com>.
Agreed.  Stop words from the moment I started using them caused complaints
and problems right off the bat.  They may have been implemented less than a
week before needing a re-index to fix all the problems they caused.

On Thu, Jun 29, 2017 at 4:55 PM, Walter Underwood <wu...@wunderwood.org>
wrote:

> Ultraseek (and Infoseek) never used stopwords. They cause odd failures,
> like not being able to search for “Vitamin A”.
>
> Stopwords are an on/off approach to term frequency. idf is a proportional
> approach. Once you have idf, you don’t need stopwords.
>
> When I was bringing up Solr for Netflix, I started with an analysis chain
> that used stopwords. A surprising number of movie titles entirely
> disappeared. I wrote a blog post about it. Ten years ago!
>
> https://observer.wunderwood.org/2007/05/31/do-all-stopword-queries-matter/
>
> Mostly, stopwords were a performance hack back when people ran search
> engines on 16-bit machines. Neither disks nor RAM were big enough to hold
> the posting lists for common words.
>
> wunder
> Walter Underwood
> wunder@wunderwood.org
> http://observer.wunderwood.org/  (my blog)
>
>
> > On Jun 29, 2017, at 1:46 PM, Rick Leir <rl...@leirtech.com> wrote:
> >
> > Walter
> > Sorry for the tangent, but the stopwords feature sounds useful. You say
> you do not use this? Did Ultraseek not do it either?
> > Thanks
> > Rick
> >
> > On June 29, 2017 10:53:42 AM EDT, Walter Underwood <
> wunder@wunderwood.org> wrote:
> >> Nope. Haven’t used stopwords for the last 20 years.
> >>
> >> I wonder if lowercaseOperators is true. The docs don’t give the default
> >> value for that in edismax.
> >>
> >> https://lucene.apache.org/solr/guide/6_6/the-extended-
> dismax-query-parser.html
> >>
> >> wunder
> >> Walter Underwood
> >> wunder@wunderwood.org
> >> http://observer.wunderwood.org/  (my blog)
> >>
> >>
> >>> On Jun 29, 2017, at 4:42 AM, Rick Leir <rl...@leirtech.com> wrote:
> >>>
> >>> Stopwords?
> >>>
> >>> On June 28, 2017 5:13:43 PM EDT, Walter Underwood
> >> <wu...@wunderwood.org> wrote:
> >>>> Is there some special casing in the highlighter to skip query syntax
> >>>> words? The words “and” and “or” don’t get highlighted.
> >>>>
> >>>> This is in 6.5.0.
> >>>>
> >>>>     <str name="hl.fl">question</str>
> >>>>     <str name="hl.encoder">html</str>
> >>>>     <str name="hl.fragsize">440</str>
> >>>>     <str name="hl.method">fastVector</str>
> >>>>     <str name="hl.snippets">1</str>
> >>>>
> >>>> wunder
> >>>> Walter Underwood
> >>>> wunder@wunderwood.org
> >>>> http://observer.wunderwood.org/  (my blog)
> >>>
> >>> --
> >>> Sorry for being brief. Alternate email is rickleir at yahoo dot com
> >
> > --
> > Sorry for being brief. Alternate email is rickleir at yahoo dot com
> > --
> > Sorry for being brief. Alternate email is rickleir at yahoo dot com
>
>

Re: Not highlighting "and" and "or"?

Posted by Walter Underwood <wu...@wunderwood.org>.
Ultraseek (and Infoseek) never used stopwords. They cause odd failures, like not being able to search for “Vitamin A”.

Stopwords are an on/off approach to term frequency. idf is a proportional approach. Once you have idf, you don’t need stopwords.

When I was bringing up Solr for Netflix, I started with an analysis chain that used stopwords. A surprising number of movie titles entirely disappeared. I wrote a blog post about it. Ten years ago!

https://observer.wunderwood.org/2007/05/31/do-all-stopword-queries-matter/

Mostly, stopwords were a performance hack back when people ran search engines on 16-bit machines. Neither disks nor RAM were big enough to hold the posting lists for common words.

wunder
Walter Underwood
wunder@wunderwood.org
http://observer.wunderwood.org/  (my blog)


> On Jun 29, 2017, at 1:46 PM, Rick Leir <rl...@leirtech.com> wrote:
> 
> Walter
> Sorry for the tangent, but the stopwords feature sounds useful. You say you do not use this? Did Ultraseek not do it either? 
> Thanks
> Rick
> 
> On June 29, 2017 10:53:42 AM EDT, Walter Underwood <wu...@wunderwood.org> wrote:
>> Nope. Haven’t used stopwords for the last 20 years.
>> 
>> I wonder if lowercaseOperators is true. The docs don’t give the default
>> value for that in edismax.
>> 
>> https://lucene.apache.org/solr/guide/6_6/the-extended-dismax-query-parser.html
>> 
>> wunder
>> Walter Underwood
>> wunder@wunderwood.org
>> http://observer.wunderwood.org/  (my blog)
>> 
>> 
>>> On Jun 29, 2017, at 4:42 AM, Rick Leir <rl...@leirtech.com> wrote:
>>> 
>>> Stopwords?
>>> 
>>> On June 28, 2017 5:13:43 PM EDT, Walter Underwood
>> <wu...@wunderwood.org> wrote:
>>>> Is there some special casing in the highlighter to skip query syntax
>>>> words? The words “and” and “or” don’t get highlighted.
>>>> 
>>>> This is in 6.5.0.
>>>> 
>>>>     <str name="hl.fl">question</str>
>>>>     <str name="hl.encoder">html</str>
>>>>     <str name="hl.fragsize">440</str>
>>>>     <str name="hl.method">fastVector</str>
>>>>     <str name="hl.snippets">1</str>
>>>> 
>>>> wunder
>>>> Walter Underwood
>>>> wunder@wunderwood.org
>>>> http://observer.wunderwood.org/  (my blog)
>>> 
>>> -- 
>>> Sorry for being brief. Alternate email is rickleir at yahoo dot com
> 
> -- 
> Sorry for being brief. Alternate email is rickleir at yahoo dot com
> -- 
> Sorry for being brief. Alternate email is rickleir at yahoo dot com


Re: Not highlighting "and" and "or"?

Posted by Rick Leir <rl...@leirtech.com>.
Walter
Sorry for the tangent, but the stopwords feature sounds useful. You say you do not use this? Did Ultraseek not do it either? 
Thanks
Rick

On June 29, 2017 10:53:42 AM EDT, Walter Underwood <wu...@wunderwood.org> wrote:
>Nope. Haven’t used stopwords for the last 20 years.
>
>I wonder if lowercaseOperators is true. The docs don’t give the default
>value for that in edismax.
>
>https://lucene.apache.org/solr/guide/6_6/the-extended-dismax-query-parser.html
>
>wunder
>Walter Underwood
>wunder@wunderwood.org
>http://observer.wunderwood.org/  (my blog)
>
>
>> On Jun 29, 2017, at 4:42 AM, Rick Leir <rl...@leirtech.com> wrote:
>> 
>> Stopwords?
>> 
>> On June 28, 2017 5:13:43 PM EDT, Walter Underwood
><wu...@wunderwood.org> wrote:
>>> Is there some special casing in the highlighter to skip query syntax
>>> words? The words “and” and “or” don’t get highlighted.
>>> 
>>> This is in 6.5.0.
>>> 
>>>      <str name="hl.fl">question</str>
>>>      <str name="hl.encoder">html</str>
>>>      <str name="hl.fragsize">440</str>
>>>      <str name="hl.method">fastVector</str>
>>>      <str name="hl.snippets">1</str>
>>> 
>>> wunder
>>> Walter Underwood
>>> wunder@wunderwood.org
>>> http://observer.wunderwood.org/  (my blog)
>> 
>> -- 
>> Sorry for being brief. Alternate email is rickleir at yahoo dot com

-- 
Sorry for being brief. Alternate email is rickleir at yahoo dot com
-- 
Sorry for being brief. Alternate email is rickleir at yahoo dot com 

Re: Not highlighting "and" and "or"?

Posted by Walter Underwood <wu...@wunderwood.org>.
Nope. Haven’t used stopwords for the last 20 years.

I wonder if lowercaseOperators is true. The docs don’t give the default value for that in edismax.

https://lucene.apache.org/solr/guide/6_6/the-extended-dismax-query-parser.html

wunder
Walter Underwood
wunder@wunderwood.org
http://observer.wunderwood.org/  (my blog)


> On Jun 29, 2017, at 4:42 AM, Rick Leir <rl...@leirtech.com> wrote:
> 
> Stopwords?
> 
> On June 28, 2017 5:13:43 PM EDT, Walter Underwood <wu...@wunderwood.org> wrote:
>> Is there some special casing in the highlighter to skip query syntax
>> words? The words “and” and “or” don’t get highlighted.
>> 
>> This is in 6.5.0.
>> 
>>      <str name="hl.fl">question</str>
>>      <str name="hl.encoder">html</str>
>>      <str name="hl.fragsize">440</str>
>>      <str name="hl.method">fastVector</str>
>>      <str name="hl.snippets">1</str>
>> 
>> wunder
>> Walter Underwood
>> wunder@wunderwood.org
>> http://observer.wunderwood.org/  (my blog)
> 
> -- 
> Sorry for being brief. Alternate email is rickleir at yahoo dot com


Re: Not highlighting "and" and "or"?

Posted by Rick Leir <rl...@leirtech.com>.
Stopwords?

On June 28, 2017 5:13:43 PM EDT, Walter Underwood <wu...@wunderwood.org> wrote:
>Is there some special casing in the highlighter to skip query syntax
>words? The words “and” and “or” don’t get highlighted.
>
>This is in 6.5.0.
>
>       <str name="hl.fl">question</str>
>       <str name="hl.encoder">html</str>
>       <str name="hl.fragsize">440</str>
>       <str name="hl.method">fastVector</str>
>       <str name="hl.snippets">1</str>
>
>wunder
>Walter Underwood
>wunder@wunderwood.org
>http://observer.wunderwood.org/  (my blog)

-- 
Sorry for being brief. Alternate email is rickleir at yahoo dot com