You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by Nawab Zada Asad Iqbal <kh...@gmail.com> on 2018/01/05 02:08:02 UTC

CommonGramsFilter

Hi,

I am looking at this documentation and wondering if it would be better to
optionally skip indexing of original stopwords.

https://lucene.apache.org/solr/guide/6_6/filter-descriptions.html#
FilterDescriptions-CommonGramsFilter

http://localhost:8983/solr/filesearch/select?q=not%20to%20or%20be&debugQuery=true


   - parsedquery: "+(-DisjunctionMaxQuery((commongram_field2:to)~0.01)
   DisjunctionMaxQuery((commongram_field2:be)~0.01))~1",



Other parameters are:


   - params: {
      - mm: " 1<-0% ",
      - q.alt: "*:*",
      - ps: "100",
      - echoParams: "all",
      - sort: "score desc",
      - rows: "35",
      - version: "2.2",
      - q: "not to or be",
      - tie: "0.01",
      - defType: "edismax",
      - qf: "commongram_field2",
      - sow: "false",
      - wt: "json",
      - debugQuery: "true"
      }


And it doesn't match my document, which has following fields:


   - id: "9191",
   - commongram_field2: "not to or be",



Commongram is defined as:

    <field name="commongram_field2" type="commongaram" indexed="true"
stored="true" omitPositions="false"/>

    <fieldType name="commongaram" class="solr.TextField"
positionIncrementGap="100">
      <analyzer type="index">
        <charFilter
class="org.apache.lucene.analysis.icu.ICUNormalizer2CharFilterFactory"
name="nfkc" mode="compose"/>
        <tokenizer class="solr.WhitespaceTokenizerFactory"/>
        <filter class="solr.WordDelimiterGraphFilterFactory"
generateWordParts="1" generateNumberParts="1" catenateWords="0"
catenateNumbers="0" catenateAll="0" preserveOriginal="0"
splitOnCaseChange="1" splitOnNumerics="1" stemEnglishPossessive="0"/>
        <filter class="solr.FlattenGraphFilterFactory"/>
        <filter class="solr.PatternReplaceFilterFactory"
pattern="^(\p{Punct}*)(.*?)(\p{Punct}*)$" replacement="$2"/>
        <filter class="solr.LowerCaseFilterFactory"/>
        <filter class="solr.ASCIIFoldingFilterFactory"/>
        <filter class="solr.CommonGramsFilterFactory" words="stopwords.txt"
ignoreCase="true"/>
        <filter class="solr.RemoveDuplicatesTokenFilterFactory"/>
        <filter class="solr.LimitTokenCountFilterFactory"
maxTokenCount="10000" consumeAllTokens="false"/>
        <filter class="solr.LengthFilterFactory" min="1" max="255"/>
      </analyzer>
      <analyzer type="query">
        <charFilter
class="org.apache.lucene.analysis.icu.ICUNormalizer2CharFilterFactory"
name="nfkc" mode="compose"/>
        <tokenizer class="solr.WhitespaceTokenizerFactory"/>
        <filter class="solr.WordDelimiterGraphFilterFactory"
generateWordParts="1" generateNumberParts="1" catenateWords="0"
catenateNumbers="0" catenateAll="0" preserveOriginal="0"
splitOnCaseChange="1" splitOnNumerics="1" stemEnglishPossessive="0"/>
        <filter class="solr.PatternReplaceFilterFactory"
pattern="^(\p{Punct}*)(.*?)(\p{Punct}*)$" replacement="$2"/>
        <filter class="solr.LowerCaseFilterFactory"/>
        <filter class="solr.ASCIIFoldingFilterFactory"/>
        <filter class="solr.CommonGramsFilterFactory" words="stopwords.txt"
ignoreCase="true"/>
        <filter class="solr.LengthFilterFactory" min="1" max="255"/>
      </analyzer>
    </fieldType>


I am not sure what I am missing. I have also set sow=false so that the
whole query string is sent to field's analysis chain instead of sending
word by word. But that didnt' seem to help.

Thanks
Nawab

Re: CommonGramsFilter

Posted by Nawab Zada Asad Iqbal <kh...@gmail.com>.
Actually, i have found that it is *not* mandatory to use phrase search with
CommonGramsFilter .

PS: i had some other code change (which is unnecessary) which was causing
the above behavior.

On Thu, Jan 4, 2018 at 6:56 PM, Nawab Zada Asad Iqbal <kh...@gmail.com>
wrote:

> After some debugging, it  seems that the search works if the query is
> phrase search (i.e, enclosed in quotes)
>
> http://localhost:8983/solr/filesearch/select?q=%22not%
> 20to%20or%20be%22&debugQuery=true
>
> This works both in case of sow=true or false.
>
> Is it mandatory to use phrase search to properly pass the stopwords to the
> CommonGramsFilter?
>
>
>
>
>
> On Thu, Jan 4, 2018 at 6:08 PM, Nawab Zada Asad Iqbal <kh...@gmail.com>
> wrote:
>
>> Hi,
>>
>> I am looking at this documentation and wondering if it would be better to
>> optionally skip indexing of original stopwords.
>>
>> https://lucene.apache.org/solr/guide/6_6/filter-descriptions
>> .html#FilterDescriptions-CommonGramsFilter
>>
>> http://localhost:8983/solr/filesearch/select?q=not%20to%20or
>> %20be&debugQuery=true
>>
>>
>>    - parsedquery: "+(-DisjunctionMaxQuery((commongram_field2:to)~0.01)
>>    DisjunctionMaxQuery((commongram_field2:be)~0.01))~1",
>>
>>
>>
>> Other parameters are:
>>
>>
>>    - params: {
>>       - mm: " 1<-0% ",
>>       - q.alt: "*:*",
>>       - ps: "100",
>>       - echoParams: "all",
>>       - sort: "score desc",
>>       - rows: "35",
>>       - version: "2.2",
>>       - q: "not to or be",
>>       - tie: "0.01",
>>       - defType: "edismax",
>>       - qf: "commongram_field2",
>>       - sow: "false",
>>       - wt: "json",
>>       - debugQuery: "true"
>>       }
>>
>>
>> And it doesn't match my document, which has following fields:
>>
>>
>>    - id: "9191",
>>    - commongram_field2: "not to or be",
>>
>>
>>
>> Commongram is defined as:
>>
>>     <field name="commongram_field2" type="commongaram" indexed="true"
>> stored="true" omitPositions="false"/>
>>
>>     <fieldType name="commongaram" class="solr.TextField"
>> positionIncrementGap="100">
>>       <analyzer type="index">
>>         <charFilter class="org.apache.lucene.analy
>> sis.icu.ICUNormalizer2CharFilterFactory" name="nfkc" mode="compose"/>
>>         <tokenizer class="solr.WhitespaceTokenizerFactory"/>
>>         <filter class="solr.WordDelimiterGraphFilterFactory"
>> generateWordParts="1" generateNumberParts="1" catenateWords="0"
>> catenateNumbers="0" catenateAll="0" preserveOriginal="0"
>> splitOnCaseChange="1" splitOnNumerics="1" stemEnglishPossessive="0"/>
>>         <filter class="solr.FlattenGraphFilterFactory"/>
>>         <filter class="solr.PatternReplaceFilterFactory"
>> pattern="^(\p{Punct}*)(.*?)(\p{Punct}*)$" replacement="$2"/>
>>         <filter class="solr.LowerCaseFilterFactory"/>
>>         <filter class="solr.ASCIIFoldingFilterFactory"/>
>>         <filter class="solr.CommonGramsFilterFactory"
>> words="stopwords.txt" ignoreCase="true"/>
>>         <filter class="solr.RemoveDuplicatesTokenFilterFactory"/>
>>         <filter class="solr.LimitTokenCountFilterFactory"
>> maxTokenCount="10000" consumeAllTokens="false"/>
>>         <filter class="solr.LengthFilterFactory" min="1" max="255"/>
>>       </analyzer>
>>       <analyzer type="query">
>>         <charFilter class="org.apache.lucene.analy
>> sis.icu.ICUNormalizer2CharFilterFactory" name="nfkc" mode="compose"/>
>>         <tokenizer class="solr.WhitespaceTokenizerFactory"/>
>>         <filter class="solr.WordDelimiterGraphFilterFactory"
>> generateWordParts="1" generateNumberParts="1" catenateWords="0"
>> catenateNumbers="0" catenateAll="0" preserveOriginal="0"
>> splitOnCaseChange="1" splitOnNumerics="1" stemEnglishPossessive="0"/>
>>         <filter class="solr.PatternReplaceFilterFactory"
>> pattern="^(\p{Punct}*)(.*?)(\p{Punct}*)$" replacement="$2"/>
>>         <filter class="solr.LowerCaseFilterFactory"/>
>>         <filter class="solr.ASCIIFoldingFilterFactory"/>
>>         <filter class="solr.CommonGramsFilterFactory"
>> words="stopwords.txt" ignoreCase="true"/>
>>         <filter class="solr.LengthFilterFactory" min="1" max="255"/>
>>       </analyzer>
>>     </fieldType>
>>
>>
>> I am not sure what I am missing. I have also set sow=false so that the
>> whole query string is sent to field's analysis chain instead of sending
>> word by word. But that didnt' seem to help.
>>
>> Thanks
>> Nawab
>>
>
>

Re: CommonGramsFilter

Posted by Nawab Zada Asad Iqbal <kh...@gmail.com>.
After some debugging, it  seems that the search works if the query is
phrase search (i.e, enclosed in quotes)

http://localhost:8983/solr/filesearch/select?q=%22not%20to%20or%20be%22&debugQuery=true

This works both in case of sow=true or false.

Is it mandatory to use phrase search to properly pass the stopwords to the
CommonGramsFilter?





On Thu, Jan 4, 2018 at 6:08 PM, Nawab Zada Asad Iqbal <kh...@gmail.com>
wrote:

> Hi,
>
> I am looking at this documentation and wondering if it would be better to
> optionally skip indexing of original stopwords.
>
> https://lucene.apache.org/solr/guide/6_6/filter-descriptions
> .html#FilterDescriptions-CommonGramsFilter
>
> http://localhost:8983/solr/filesearch/select?q=not%20to%
> 20or%20be&debugQuery=true
>
>
>    - parsedquery: "+(-DisjunctionMaxQuery((commongram_field2:to)~0.01)
>    DisjunctionMaxQuery((commongram_field2:be)~0.01))~1",
>
>
>
> Other parameters are:
>
>
>    - params: {
>       - mm: " 1<-0% ",
>       - q.alt: "*:*",
>       - ps: "100",
>       - echoParams: "all",
>       - sort: "score desc",
>       - rows: "35",
>       - version: "2.2",
>       - q: "not to or be",
>       - tie: "0.01",
>       - defType: "edismax",
>       - qf: "commongram_field2",
>       - sow: "false",
>       - wt: "json",
>       - debugQuery: "true"
>       }
>
>
> And it doesn't match my document, which has following fields:
>
>
>    - id: "9191",
>    - commongram_field2: "not to or be",
>
>
>
> Commongram is defined as:
>
>     <field name="commongram_field2" type="commongaram" indexed="true"
> stored="true" omitPositions="false"/>
>
>     <fieldType name="commongaram" class="solr.TextField"
> positionIncrementGap="100">
>       <analyzer type="index">
>         <charFilter class="org.apache.lucene.analysis.icu.
> ICUNormalizer2CharFilterFactory" name="nfkc" mode="compose"/>
>         <tokenizer class="solr.WhitespaceTokenizerFactory"/>
>         <filter class="solr.WordDelimiterGraphFilterFactory"
> generateWordParts="1" generateNumberParts="1" catenateWords="0"
> catenateNumbers="0" catenateAll="0" preserveOriginal="0"
> splitOnCaseChange="1" splitOnNumerics="1" stemEnglishPossessive="0"/>
>         <filter class="solr.FlattenGraphFilterFactory"/>
>         <filter class="solr.PatternReplaceFilterFactory"
> pattern="^(\p{Punct}*)(.*?)(\p{Punct}*)$" replacement="$2"/>
>         <filter class="solr.LowerCaseFilterFactory"/>
>         <filter class="solr.ASCIIFoldingFilterFactory"/>
>         <filter class="solr.CommonGramsFilterFactory"
> words="stopwords.txt" ignoreCase="true"/>
>         <filter class="solr.RemoveDuplicatesTokenFilterFactory"/>
>         <filter class="solr.LimitTokenCountFilterFactory"
> maxTokenCount="10000" consumeAllTokens="false"/>
>         <filter class="solr.LengthFilterFactory" min="1" max="255"/>
>       </analyzer>
>       <analyzer type="query">
>         <charFilter class="org.apache.lucene.analysis.icu.
> ICUNormalizer2CharFilterFactory" name="nfkc" mode="compose"/>
>         <tokenizer class="solr.WhitespaceTokenizerFactory"/>
>         <filter class="solr.WordDelimiterGraphFilterFactory"
> generateWordParts="1" generateNumberParts="1" catenateWords="0"
> catenateNumbers="0" catenateAll="0" preserveOriginal="0"
> splitOnCaseChange="1" splitOnNumerics="1" stemEnglishPossessive="0"/>
>         <filter class="solr.PatternReplaceFilterFactory"
> pattern="^(\p{Punct}*)(.*?)(\p{Punct}*)$" replacement="$2"/>
>         <filter class="solr.LowerCaseFilterFactory"/>
>         <filter class="solr.ASCIIFoldingFilterFactory"/>
>         <filter class="solr.CommonGramsFilterFactory"
> words="stopwords.txt" ignoreCase="true"/>
>         <filter class="solr.LengthFilterFactory" min="1" max="255"/>
>       </analyzer>
>     </fieldType>
>
>
> I am not sure what I am missing. I have also set sow=false so that the
> whole query string is sent to field's analysis chain instead of sending
> word by word. But that didnt' seem to help.
>
> Thanks
> Nawab
>