You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by Nawab Zada Asad Iqbal <kh...@gmail.com> on 2018/01/05 02:08:02 UTC
CommonGramsFilter
Hi,
I am looking at this documentation and wondering if it would be better to
optionally skip indexing of original stopwords.
https://lucene.apache.org/solr/guide/6_6/filter-descriptions.html#
FilterDescriptions-CommonGramsFilter
http://localhost:8983/solr/filesearch/select?q=not%20to%20or%20be&debugQuery=true
- parsedquery: "+(-DisjunctionMaxQuery((commongram_field2:to)~0.01)
DisjunctionMaxQuery((commongram_field2:be)~0.01))~1",
Other parameters are:
- params: {
- mm: " 1<-0% ",
- q.alt: "*:*",
- ps: "100",
- echoParams: "all",
- sort: "score desc",
- rows: "35",
- version: "2.2",
- q: "not to or be",
- tie: "0.01",
- defType: "edismax",
- qf: "commongram_field2",
- sow: "false",
- wt: "json",
- debugQuery: "true"
}
And it doesn't match my document, which has following fields:
- id: "9191",
- commongram_field2: "not to or be",
Commongram is defined as:
<field name="commongram_field2" type="commongaram" indexed="true"
stored="true" omitPositions="false"/>
<fieldType name="commongaram" class="solr.TextField"
positionIncrementGap="100">
<analyzer type="index">
<charFilter
class="org.apache.lucene.analysis.icu.ICUNormalizer2CharFilterFactory"
name="nfkc" mode="compose"/>
<tokenizer class="solr.WhitespaceTokenizerFactory"/>
<filter class="solr.WordDelimiterGraphFilterFactory"
generateWordParts="1" generateNumberParts="1" catenateWords="0"
catenateNumbers="0" catenateAll="0" preserveOriginal="0"
splitOnCaseChange="1" splitOnNumerics="1" stemEnglishPossessive="0"/>
<filter class="solr.FlattenGraphFilterFactory"/>
<filter class="solr.PatternReplaceFilterFactory"
pattern="^(\p{Punct}*)(.*?)(\p{Punct}*)$" replacement="$2"/>
<filter class="solr.LowerCaseFilterFactory"/>
<filter class="solr.ASCIIFoldingFilterFactory"/>
<filter class="solr.CommonGramsFilterFactory" words="stopwords.txt"
ignoreCase="true"/>
<filter class="solr.RemoveDuplicatesTokenFilterFactory"/>
<filter class="solr.LimitTokenCountFilterFactory"
maxTokenCount="10000" consumeAllTokens="false"/>
<filter class="solr.LengthFilterFactory" min="1" max="255"/>
</analyzer>
<analyzer type="query">
<charFilter
class="org.apache.lucene.analysis.icu.ICUNormalizer2CharFilterFactory"
name="nfkc" mode="compose"/>
<tokenizer class="solr.WhitespaceTokenizerFactory"/>
<filter class="solr.WordDelimiterGraphFilterFactory"
generateWordParts="1" generateNumberParts="1" catenateWords="0"
catenateNumbers="0" catenateAll="0" preserveOriginal="0"
splitOnCaseChange="1" splitOnNumerics="1" stemEnglishPossessive="0"/>
<filter class="solr.PatternReplaceFilterFactory"
pattern="^(\p{Punct}*)(.*?)(\p{Punct}*)$" replacement="$2"/>
<filter class="solr.LowerCaseFilterFactory"/>
<filter class="solr.ASCIIFoldingFilterFactory"/>
<filter class="solr.CommonGramsFilterFactory" words="stopwords.txt"
ignoreCase="true"/>
<filter class="solr.LengthFilterFactory" min="1" max="255"/>
</analyzer>
</fieldType>
I am not sure what I am missing. I have also set sow=false so that the
whole query string is sent to field's analysis chain instead of sending
word by word. But that didnt' seem to help.
Thanks
Nawab
Re: CommonGramsFilter
Posted by Nawab Zada Asad Iqbal <kh...@gmail.com>.
Actually, i have found that it is *not* mandatory to use phrase search with
CommonGramsFilter .
PS: i had some other code change (which is unnecessary) which was causing
the above behavior.
On Thu, Jan 4, 2018 at 6:56 PM, Nawab Zada Asad Iqbal <kh...@gmail.com>
wrote:
> After some debugging, it seems that the search works if the query is
> phrase search (i.e, enclosed in quotes)
>
> http://localhost:8983/solr/filesearch/select?q=%22not%
> 20to%20or%20be%22&debugQuery=true
>
> This works both in case of sow=true or false.
>
> Is it mandatory to use phrase search to properly pass the stopwords to the
> CommonGramsFilter?
>
>
>
>
>
> On Thu, Jan 4, 2018 at 6:08 PM, Nawab Zada Asad Iqbal <kh...@gmail.com>
> wrote:
>
>> Hi,
>>
>> I am looking at this documentation and wondering if it would be better to
>> optionally skip indexing of original stopwords.
>>
>> https://lucene.apache.org/solr/guide/6_6/filter-descriptions
>> .html#FilterDescriptions-CommonGramsFilter
>>
>> http://localhost:8983/solr/filesearch/select?q=not%20to%20or
>> %20be&debugQuery=true
>>
>>
>> - parsedquery: "+(-DisjunctionMaxQuery((commongram_field2:to)~0.01)
>> DisjunctionMaxQuery((commongram_field2:be)~0.01))~1",
>>
>>
>>
>> Other parameters are:
>>
>>
>> - params: {
>> - mm: " 1<-0% ",
>> - q.alt: "*:*",
>> - ps: "100",
>> - echoParams: "all",
>> - sort: "score desc",
>> - rows: "35",
>> - version: "2.2",
>> - q: "not to or be",
>> - tie: "0.01",
>> - defType: "edismax",
>> - qf: "commongram_field2",
>> - sow: "false",
>> - wt: "json",
>> - debugQuery: "true"
>> }
>>
>>
>> And it doesn't match my document, which has following fields:
>>
>>
>> - id: "9191",
>> - commongram_field2: "not to or be",
>>
>>
>>
>> Commongram is defined as:
>>
>> <field name="commongram_field2" type="commongaram" indexed="true"
>> stored="true" omitPositions="false"/>
>>
>> <fieldType name="commongaram" class="solr.TextField"
>> positionIncrementGap="100">
>> <analyzer type="index">
>> <charFilter class="org.apache.lucene.analy
>> sis.icu.ICUNormalizer2CharFilterFactory" name="nfkc" mode="compose"/>
>> <tokenizer class="solr.WhitespaceTokenizerFactory"/>
>> <filter class="solr.WordDelimiterGraphFilterFactory"
>> generateWordParts="1" generateNumberParts="1" catenateWords="0"
>> catenateNumbers="0" catenateAll="0" preserveOriginal="0"
>> splitOnCaseChange="1" splitOnNumerics="1" stemEnglishPossessive="0"/>
>> <filter class="solr.FlattenGraphFilterFactory"/>
>> <filter class="solr.PatternReplaceFilterFactory"
>> pattern="^(\p{Punct}*)(.*?)(\p{Punct}*)$" replacement="$2"/>
>> <filter class="solr.LowerCaseFilterFactory"/>
>> <filter class="solr.ASCIIFoldingFilterFactory"/>
>> <filter class="solr.CommonGramsFilterFactory"
>> words="stopwords.txt" ignoreCase="true"/>
>> <filter class="solr.RemoveDuplicatesTokenFilterFactory"/>
>> <filter class="solr.LimitTokenCountFilterFactory"
>> maxTokenCount="10000" consumeAllTokens="false"/>
>> <filter class="solr.LengthFilterFactory" min="1" max="255"/>
>> </analyzer>
>> <analyzer type="query">
>> <charFilter class="org.apache.lucene.analy
>> sis.icu.ICUNormalizer2CharFilterFactory" name="nfkc" mode="compose"/>
>> <tokenizer class="solr.WhitespaceTokenizerFactory"/>
>> <filter class="solr.WordDelimiterGraphFilterFactory"
>> generateWordParts="1" generateNumberParts="1" catenateWords="0"
>> catenateNumbers="0" catenateAll="0" preserveOriginal="0"
>> splitOnCaseChange="1" splitOnNumerics="1" stemEnglishPossessive="0"/>
>> <filter class="solr.PatternReplaceFilterFactory"
>> pattern="^(\p{Punct}*)(.*?)(\p{Punct}*)$" replacement="$2"/>
>> <filter class="solr.LowerCaseFilterFactory"/>
>> <filter class="solr.ASCIIFoldingFilterFactory"/>
>> <filter class="solr.CommonGramsFilterFactory"
>> words="stopwords.txt" ignoreCase="true"/>
>> <filter class="solr.LengthFilterFactory" min="1" max="255"/>
>> </analyzer>
>> </fieldType>
>>
>>
>> I am not sure what I am missing. I have also set sow=false so that the
>> whole query string is sent to field's analysis chain instead of sending
>> word by word. But that didnt' seem to help.
>>
>> Thanks
>> Nawab
>>
>
>
Re: CommonGramsFilter
Posted by Nawab Zada Asad Iqbal <kh...@gmail.com>.
After some debugging, it seems that the search works if the query is
phrase search (i.e, enclosed in quotes)
http://localhost:8983/solr/filesearch/select?q=%22not%20to%20or%20be%22&debugQuery=true
This works both in case of sow=true or false.
Is it mandatory to use phrase search to properly pass the stopwords to the
CommonGramsFilter?
On Thu, Jan 4, 2018 at 6:08 PM, Nawab Zada Asad Iqbal <kh...@gmail.com>
wrote:
> Hi,
>
> I am looking at this documentation and wondering if it would be better to
> optionally skip indexing of original stopwords.
>
> https://lucene.apache.org/solr/guide/6_6/filter-descriptions
> .html#FilterDescriptions-CommonGramsFilter
>
> http://localhost:8983/solr/filesearch/select?q=not%20to%
> 20or%20be&debugQuery=true
>
>
> - parsedquery: "+(-DisjunctionMaxQuery((commongram_field2:to)~0.01)
> DisjunctionMaxQuery((commongram_field2:be)~0.01))~1",
>
>
>
> Other parameters are:
>
>
> - params: {
> - mm: " 1<-0% ",
> - q.alt: "*:*",
> - ps: "100",
> - echoParams: "all",
> - sort: "score desc",
> - rows: "35",
> - version: "2.2",
> - q: "not to or be",
> - tie: "0.01",
> - defType: "edismax",
> - qf: "commongram_field2",
> - sow: "false",
> - wt: "json",
> - debugQuery: "true"
> }
>
>
> And it doesn't match my document, which has following fields:
>
>
> - id: "9191",
> - commongram_field2: "not to or be",
>
>
>
> Commongram is defined as:
>
> <field name="commongram_field2" type="commongaram" indexed="true"
> stored="true" omitPositions="false"/>
>
> <fieldType name="commongaram" class="solr.TextField"
> positionIncrementGap="100">
> <analyzer type="index">
> <charFilter class="org.apache.lucene.analysis.icu.
> ICUNormalizer2CharFilterFactory" name="nfkc" mode="compose"/>
> <tokenizer class="solr.WhitespaceTokenizerFactory"/>
> <filter class="solr.WordDelimiterGraphFilterFactory"
> generateWordParts="1" generateNumberParts="1" catenateWords="0"
> catenateNumbers="0" catenateAll="0" preserveOriginal="0"
> splitOnCaseChange="1" splitOnNumerics="1" stemEnglishPossessive="0"/>
> <filter class="solr.FlattenGraphFilterFactory"/>
> <filter class="solr.PatternReplaceFilterFactory"
> pattern="^(\p{Punct}*)(.*?)(\p{Punct}*)$" replacement="$2"/>
> <filter class="solr.LowerCaseFilterFactory"/>
> <filter class="solr.ASCIIFoldingFilterFactory"/>
> <filter class="solr.CommonGramsFilterFactory"
> words="stopwords.txt" ignoreCase="true"/>
> <filter class="solr.RemoveDuplicatesTokenFilterFactory"/>
> <filter class="solr.LimitTokenCountFilterFactory"
> maxTokenCount="10000" consumeAllTokens="false"/>
> <filter class="solr.LengthFilterFactory" min="1" max="255"/>
> </analyzer>
> <analyzer type="query">
> <charFilter class="org.apache.lucene.analysis.icu.
> ICUNormalizer2CharFilterFactory" name="nfkc" mode="compose"/>
> <tokenizer class="solr.WhitespaceTokenizerFactory"/>
> <filter class="solr.WordDelimiterGraphFilterFactory"
> generateWordParts="1" generateNumberParts="1" catenateWords="0"
> catenateNumbers="0" catenateAll="0" preserveOriginal="0"
> splitOnCaseChange="1" splitOnNumerics="1" stemEnglishPossessive="0"/>
> <filter class="solr.PatternReplaceFilterFactory"
> pattern="^(\p{Punct}*)(.*?)(\p{Punct}*)$" replacement="$2"/>
> <filter class="solr.LowerCaseFilterFactory"/>
> <filter class="solr.ASCIIFoldingFilterFactory"/>
> <filter class="solr.CommonGramsFilterFactory"
> words="stopwords.txt" ignoreCase="true"/>
> <filter class="solr.LengthFilterFactory" min="1" max="255"/>
> </analyzer>
> </fieldType>
>
>
> I am not sure what I am missing. I have also set sow=false so that the
> whole query string is sent to field's analysis chain instead of sending
> word by word. But that didnt' seem to help.
>
> Thanks
> Nawab
>