You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by Ralf Heyde <ra...@gmx.de> on 2013/02/12 16:25:33 UTC
DisMax Query & Field-Filters (ASCIIFolding)
Hello,
I have an interesting behaviour.
I have a FieldType "Text_PL". This type is configured as:
<fieldType name="text_pl" class="solr.TextField" positionIncrementGap="100">
<analyzer type="index">
<tokenizer class="solr.WhitespaceTokenizerFactory"/>
<filter class="solr.StopFilterFactory" ignoreCase="true" words="words/stopwords_pl.txt" enablePositionIncrements="true" />
<filter class="solr.WordDelimiterFilterFactory" generateWordParts="1" generateNumberParts="1" catenateWords="1" catenateNumbers="1" catenateAll="0" splitOnCaseChange="1"/>
<filter class="solr.ASCIIFoldingFilterFactory" />
<filter class="solr.LowerCaseFilterFactory"/>
<filter class="solr.StempelPolishStemFilterFactory" protected="words/protwords_pl.txt"/>
<filter class="solr.RemoveDuplicatesTokenFilterFactory"/>
</analyzer>
<analyzer type="query">
<tokenizer class="solr.WhitespaceTokenizerFactory"/>
<filter class="solr.StopFilterFactory" ignoreCase="true" words="words/stopwords_pl.txt" enablePositionIncrements="true" />
<filter class="solr.WordDelimiterFilterFactory" generateWordParts="1" generateNumberParts="1" catenateWords="1" catenateNumbers="1" catenateAll="0" splitOnCaseChange="1"/>
<filter class="solr.ASCIIFoldingFilterFactory" />
<filter class="solr.LowerCaseFilterFactory"/>
<filter class="solr.StempelPolishStemFilterFactory" protected="words/protwords_pl.txt"/>
<filter class="solr.RemoveDuplicatesTokenFilterFactory"/>
</analyzer>
</fieldType>
So, one filter in the chain is the ASCIIFoldingFilterFactory which normalizes "special" characters (e.g. ó --> O).
If I query "field:czolenka" it shows the same behaviour like searching for "field:czółenka" - as expected.
Now, if I use the DisMax query, this normalization step does not take place. I debugged the code, if I run the "normal" query, the debugger stops at the ASCIIFoldingFilter (as expected), if I run the DisMax Query, there is no stop at this filter - so the filter is not used.
Does anybody has an idea why?
Do I have to configure the DisMax RequestHandler for ASCIIFolding - if possible ?
Thanks,
Ralf
Re: DisMax Query & Field-Filters (ASCIIFolding)
Posted by Ralf Heyde <ra...@gmx.de>.
I'll try to reindex - i modified the schema, but NOT re-indexed the Index.
Damn !
-------- Original-Nachricht --------
> Datum: Tue, 12 Feb 2013 11:14:04 -0500
> Von: "Jack Krupansky" <ja...@basetechnology.com>
> An: solr-user@lucene.apache.org
> Betreff: Re: DisMax Query & Field-Filters (ASCIIFolding)
> 1. Show us the full query request and request handler. In particular, the
> "qf" parameter.
>
> 2. Try the Solr Admin Analysis UI to check for sure how the analysis is
> being performed.
>
> 3. Add &debugQuery=true to your query to see how it is actually parsed.
>
> 4. If there is any chance that you have modified your field type since
> originally indexing the data, be sure to completely reindex after ANY
> change
> in the field types.
>
> -- Jack Krupansky
>
> -----Original Message-----
> From: Ralf Heyde
> Sent: Tuesday, February 12, 2013 10:53 AM
> To: solr-user@lucene.apache.org
> Subject: Re: DisMax Query & Field-Filters (ASCIIFolding)
>
> Hi,
>
> thanks for your first Answer.
>
> I don't want to have a fielded-query in my DisMax Query.
>
> My DismaxQuery looks like this:
>
> qt=dismax&q=czółenka&... --> works
> qt=dismax&q=czolenka&... --> does not work
>
> The accessed Fields contain the ASCIIFoldingFilter for Query & Index.
>
> So, what I need is, that the DisMax QueryParser "normalizes" by
> ASCIIFolding. Is that possible?
>
> Thanks,
>
> Ralf
>
> -------- Original-Nachricht --------
> > Datum: Tue, 12 Feb 2013 07:42:17 -0800 (PST)
> > Von: Ahmet Arslan <io...@yahoo.com>
> > An: solr-user@lucene.apache.org
> > Betreff: Re: DisMax Query & Field-Filters (ASCIIFolding)
>
> >
> > Hi Ralf,
> >
> > Dismax querparser does not allow fielded queries. e.g. field:something
> >
> > Consider using edismax query parser instead.
> >
> > Also debugQuery=on will display informative output how query parsed
> > analyzed etc.
> >
> > ahmet
> >
> > --- On Tue, 2/12/13, Ralf Heyde <ra...@gmx.de> wrote:
> >
> > > From: Ralf Heyde <ra...@gmx.de>
> > > Subject: DisMax Query & Field-Filters (ASCIIFolding)
> > > To: solr-user@lucene.apache.org
> > > Date: Tuesday, February 12, 2013, 5:25 PM
> > > Hello,
> > >
> > > I have an interesting behaviour.
> > >
> > > I have a FieldType "Text_PL". This type is configured as:
> > >
> > > <fieldType name="text_pl" class="solr.TextField"
> > > positionIncrementGap="100">
> > > <analyzer type="index">
> > > <tokenizer
> > > class="solr.WhitespaceTokenizerFactory"/>
> > > <filter
> > > class="solr.StopFilterFactory" ignoreCase="true"
> > > words="words/stopwords_pl.txt"
> > > enablePositionIncrements="true" />
> > > <filter
> > > class="solr.WordDelimiterFilterFactory"
> > > generateWordParts="1" generateNumberParts="1"
> > > catenateWords="1" catenateNumbers="1" catenateAll="0"
> > > splitOnCaseChange="1"/>
> > > <filter
> > > class="solr.ASCIIFoldingFilterFactory" />
> > > <filter
> > > class="solr.LowerCaseFilterFactory"/>
> > > <filter
> > > class="solr.StempelPolishStemFilterFactory"
> > > protected="words/protwords_pl.txt"/>
> > > <filter
> > > class="solr.RemoveDuplicatesTokenFilterFactory"/>
> > > </analyzer>
> > > <analyzer type="query">
> > > <tokenizer
> > > class="solr.WhitespaceTokenizerFactory"/>
> > > <filter
> > > class="solr.StopFilterFactory" ignoreCase="true"
> > > words="words/stopwords_pl.txt"
> > > enablePositionIncrements="true" />
> > > <filter
> > > class="solr.WordDelimiterFilterFactory"
> > > generateWordParts="1" generateNumberParts="1"
> > > catenateWords="1" catenateNumbers="1" catenateAll="0"
> > > splitOnCaseChange="1"/>
> > > <filter
> > > class="solr.ASCIIFoldingFilterFactory" />
> > > <filter
> > > class="solr.LowerCaseFilterFactory"/>
> > > <filter
> > > class="solr.StempelPolishStemFilterFactory"
> > > protected="words/protwords_pl.txt"/>
> > > <filter
> > > class="solr.RemoveDuplicatesTokenFilterFactory"/>
> > > </analyzer>
> > > </fieldType>
> > >
> > > So, one filter in the chain is the ASCIIFoldingFilterFactory
> > > which normalizes "special" characters (e.g. ó --> O).
> > > If I query "field:czolenka" it shows the same behaviour like
> > > searching for "field:czółenka" - as expected.
> > >
> > > Now, if I use the DisMax query, this normalization step does
> > > not take place. I debugged the code, if I run the "normal"
> > > query, the debugger stops at the ASCIIFoldingFilter (as
> > > expected), if I run the DisMax Query, there is no stop at
> > > this filter - so the filter is not used.
> > >
> > > Does anybody has an idea why?
> > > Do I have to configure the DisMax RequestHandler for
> > > ASCIIFolding - if possible ?
> > >
> > > Thanks,
> > >
> > > Ralf
> > >
>
Re: DisMax Query & Field-Filters (ASCIIFolding)
Posted by Jack Krupansky <ja...@basetechnology.com>.
1. Show us the full query request and request handler. In particular, the
"qf" parameter.
2. Try the Solr Admin Analysis UI to check for sure how the analysis is
being performed.
3. Add &debugQuery=true to your query to see how it is actually parsed.
4. If there is any chance that you have modified your field type since
originally indexing the data, be sure to completely reindex after ANY change
in the field types.
-- Jack Krupansky
-----Original Message-----
From: Ralf Heyde
Sent: Tuesday, February 12, 2013 10:53 AM
To: solr-user@lucene.apache.org
Subject: Re: DisMax Query & Field-Filters (ASCIIFolding)
Hi,
thanks for your first Answer.
I don't want to have a fielded-query in my DisMax Query.
My DismaxQuery looks like this:
qt=dismax&q=czółenka&... --> works
qt=dismax&q=czolenka&... --> does not work
The accessed Fields contain the ASCIIFoldingFilter for Query & Index.
So, what I need is, that the DisMax QueryParser "normalizes" by
ASCIIFolding. Is that possible?
Thanks,
Ralf
-------- Original-Nachricht --------
> Datum: Tue, 12 Feb 2013 07:42:17 -0800 (PST)
> Von: Ahmet Arslan <io...@yahoo.com>
> An: solr-user@lucene.apache.org
> Betreff: Re: DisMax Query & Field-Filters (ASCIIFolding)
>
> Hi Ralf,
>
> Dismax querparser does not allow fielded queries. e.g. field:something
>
> Consider using edismax query parser instead.
>
> Also debugQuery=on will display informative output how query parsed
> analyzed etc.
>
> ahmet
>
> --- On Tue, 2/12/13, Ralf Heyde <ra...@gmx.de> wrote:
>
> > From: Ralf Heyde <ra...@gmx.de>
> > Subject: DisMax Query & Field-Filters (ASCIIFolding)
> > To: solr-user@lucene.apache.org
> > Date: Tuesday, February 12, 2013, 5:25 PM
> > Hello,
> >
> > I have an interesting behaviour.
> >
> > I have a FieldType "Text_PL". This type is configured as:
> >
> > <fieldType name="text_pl" class="solr.TextField"
> > positionIncrementGap="100">
> > <analyzer type="index">
> > <tokenizer
> > class="solr.WhitespaceTokenizerFactory"/>
> > <filter
> > class="solr.StopFilterFactory" ignoreCase="true"
> > words="words/stopwords_pl.txt"
> > enablePositionIncrements="true" />
> > <filter
> > class="solr.WordDelimiterFilterFactory"
> > generateWordParts="1" generateNumberParts="1"
> > catenateWords="1" catenateNumbers="1" catenateAll="0"
> > splitOnCaseChange="1"/>
> > <filter
> > class="solr.ASCIIFoldingFilterFactory" />
> > <filter
> > class="solr.LowerCaseFilterFactory"/>
> > <filter
> > class="solr.StempelPolishStemFilterFactory"
> > protected="words/protwords_pl.txt"/>
> > <filter
> > class="solr.RemoveDuplicatesTokenFilterFactory"/>
> > </analyzer>
> > <analyzer type="query">
> > <tokenizer
> > class="solr.WhitespaceTokenizerFactory"/>
> > <filter
> > class="solr.StopFilterFactory" ignoreCase="true"
> > words="words/stopwords_pl.txt"
> > enablePositionIncrements="true" />
> > <filter
> > class="solr.WordDelimiterFilterFactory"
> > generateWordParts="1" generateNumberParts="1"
> > catenateWords="1" catenateNumbers="1" catenateAll="0"
> > splitOnCaseChange="1"/>
> > <filter
> > class="solr.ASCIIFoldingFilterFactory" />
> > <filter
> > class="solr.LowerCaseFilterFactory"/>
> > <filter
> > class="solr.StempelPolishStemFilterFactory"
> > protected="words/protwords_pl.txt"/>
> > <filter
> > class="solr.RemoveDuplicatesTokenFilterFactory"/>
> > </analyzer>
> > </fieldType>
> >
> > So, one filter in the chain is the ASCIIFoldingFilterFactory
> > which normalizes "special" characters (e.g. ó --> O).
> > If I query "field:czolenka" it shows the same behaviour like
> > searching for "field:czółenka" - as expected.
> >
> > Now, if I use the DisMax query, this normalization step does
> > not take place. I debugged the code, if I run the "normal"
> > query, the debugger stops at the ASCIIFoldingFilter (as
> > expected), if I run the DisMax Query, there is no stop at
> > this filter - so the filter is not used.
> >
> > Does anybody has an idea why?
> > Do I have to configure the DisMax RequestHandler for
> > ASCIIFolding - if possible ?
> >
> > Thanks,
> >
> > Ralf
> >
Re: DisMax Query & Field-Filters (ASCIIFolding)
Posted by Ralf Heyde <ra...@gmx.de>.
Hi,
thanks for your first Answer.
I don't want to have a fielded-query in my DisMax Query.
My DismaxQuery looks like this:
qt=dismax&q=czółenka&... --> works
qt=dismax&q=czolenka&... --> does not work
The accessed Fields contain the ASCIIFoldingFilter for Query & Index.
So, what I need is, that the DisMax QueryParser "normalizes" by ASCIIFolding. Is that possible?
Thanks,
Ralf
-------- Original-Nachricht --------
> Datum: Tue, 12 Feb 2013 07:42:17 -0800 (PST)
> Von: Ahmet Arslan <io...@yahoo.com>
> An: solr-user@lucene.apache.org
> Betreff: Re: DisMax Query & Field-Filters (ASCIIFolding)
>
> Hi Ralf,
>
> Dismax querparser does not allow fielded queries. e.g. field:something
>
> Consider using edismax query parser instead.
>
> Also debugQuery=on will display informative output how query parsed
> analyzed etc.
>
> ahmet
>
> --- On Tue, 2/12/13, Ralf Heyde <ra...@gmx.de> wrote:
>
> > From: Ralf Heyde <ra...@gmx.de>
> > Subject: DisMax Query & Field-Filters (ASCIIFolding)
> > To: solr-user@lucene.apache.org
> > Date: Tuesday, February 12, 2013, 5:25 PM
> > Hello,
> >
> > I have an interesting behaviour.
> >
> > I have a FieldType "Text_PL". This type is configured as:
> >
> > <fieldType name="text_pl" class="solr.TextField"
> > positionIncrementGap="100">
> > <analyzer type="index">
> > <tokenizer
> > class="solr.WhitespaceTokenizerFactory"/>
> > <filter
> > class="solr.StopFilterFactory" ignoreCase="true"
> > words="words/stopwords_pl.txt"
> > enablePositionIncrements="true" />
> > <filter
> > class="solr.WordDelimiterFilterFactory"
> > generateWordParts="1" generateNumberParts="1"
> > catenateWords="1" catenateNumbers="1" catenateAll="0"
> > splitOnCaseChange="1"/>
> > <filter
> > class="solr.ASCIIFoldingFilterFactory" />
> > <filter
> > class="solr.LowerCaseFilterFactory"/>
> > <filter
> > class="solr.StempelPolishStemFilterFactory"
> > protected="words/protwords_pl.txt"/>
> > <filter
> > class="solr.RemoveDuplicatesTokenFilterFactory"/>
> > </analyzer>
> > <analyzer type="query">
> > <tokenizer
> > class="solr.WhitespaceTokenizerFactory"/>
> > <filter
> > class="solr.StopFilterFactory" ignoreCase="true"
> > words="words/stopwords_pl.txt"
> > enablePositionIncrements="true" />
> > <filter
> > class="solr.WordDelimiterFilterFactory"
> > generateWordParts="1" generateNumberParts="1"
> > catenateWords="1" catenateNumbers="1" catenateAll="0"
> > splitOnCaseChange="1"/>
> > <filter
> > class="solr.ASCIIFoldingFilterFactory" />
> > <filter
> > class="solr.LowerCaseFilterFactory"/>
> > <filter
> > class="solr.StempelPolishStemFilterFactory"
> > protected="words/protwords_pl.txt"/>
> > <filter
> > class="solr.RemoveDuplicatesTokenFilterFactory"/>
> > </analyzer>
> > </fieldType>
> >
> > So, one filter in the chain is the ASCIIFoldingFilterFactory
> > which normalizes "special" characters (e.g. ó --> O).
> > If I query "field:czolenka" it shows the same behaviour like
> > searching for "field:czółenka" - as expected.
> >
> > Now, if I use the DisMax query, this normalization step does
> > not take place. I debugged the code, if I run the "normal"
> > query, the debugger stops at the ASCIIFoldingFilter (as
> > expected), if I run the DisMax Query, there is no stop at
> > this filter - so the filter is not used.
> >
> > Does anybody has an idea why?
> > Do I have to configure the DisMax RequestHandler for
> > ASCIIFolding - if possible ?
> >
> > Thanks,
> >
> > Ralf
> >
Re: DisMax Query & Field-Filters (ASCIIFolding)
Posted by Ahmet Arslan <io...@yahoo.com>.
Hi Ralf,
Dismax querparser does not allow fielded queries. e.g. field:something
Consider using edismax query parser instead.
Also debugQuery=on will display informative output how query parsed analyzed etc.
ahmet
--- On Tue, 2/12/13, Ralf Heyde <ra...@gmx.de> wrote:
> From: Ralf Heyde <ra...@gmx.de>
> Subject: DisMax Query & Field-Filters (ASCIIFolding)
> To: solr-user@lucene.apache.org
> Date: Tuesday, February 12, 2013, 5:25 PM
> Hello,
>
> I have an interesting behaviour.
>
> I have a FieldType "Text_PL". This type is configured as:
>
> <fieldType name="text_pl" class="solr.TextField"
> positionIncrementGap="100">
> <analyzer type="index">
> <tokenizer
> class="solr.WhitespaceTokenizerFactory"/>
> <filter
> class="solr.StopFilterFactory" ignoreCase="true"
> words="words/stopwords_pl.txt"
> enablePositionIncrements="true" />
> <filter
> class="solr.WordDelimiterFilterFactory"
> generateWordParts="1" generateNumberParts="1"
> catenateWords="1" catenateNumbers="1" catenateAll="0"
> splitOnCaseChange="1"/>
> <filter
> class="solr.ASCIIFoldingFilterFactory" />
> <filter
> class="solr.LowerCaseFilterFactory"/>
> <filter
> class="solr.StempelPolishStemFilterFactory"
> protected="words/protwords_pl.txt"/>
> <filter
> class="solr.RemoveDuplicatesTokenFilterFactory"/>
> </analyzer>
> <analyzer type="query">
> <tokenizer
> class="solr.WhitespaceTokenizerFactory"/>
> <filter
> class="solr.StopFilterFactory" ignoreCase="true"
> words="words/stopwords_pl.txt"
> enablePositionIncrements="true" />
> <filter
> class="solr.WordDelimiterFilterFactory"
> generateWordParts="1" generateNumberParts="1"
> catenateWords="1" catenateNumbers="1" catenateAll="0"
> splitOnCaseChange="1"/>
> <filter
> class="solr.ASCIIFoldingFilterFactory" />
> <filter
> class="solr.LowerCaseFilterFactory"/>
> <filter
> class="solr.StempelPolishStemFilterFactory"
> protected="words/protwords_pl.txt"/>
> <filter
> class="solr.RemoveDuplicatesTokenFilterFactory"/>
> </analyzer>
> </fieldType>
>
> So, one filter in the chain is the ASCIIFoldingFilterFactory
> which normalizes "special" characters (e.g. ó --> O).
> If I query "field:czolenka" it shows the same behaviour like
> searching for "field:czółenka" - as expected.
>
> Now, if I use the DisMax query, this normalization step does
> not take place. I debugged the code, if I run the "normal"
> query, the debugger stops at the ASCIIFoldingFilter (as
> expected), if I run the DisMax Query, there is no stop at
> this filter - so the filter is not used.
>
> Does anybody has an idea why?
> Do I have to configure the DisMax RequestHandler for
> ASCIIFolding - if possible ?
>
> Thanks,
>
> Ralf
>