You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by jo...@aol.com on 2013/07/03 18:05:28 UTC
How to deal with underscore
Hi,
In my schema.xml, I have the following settings:
<analyzer>
<tokenizer class="solr.WhitespaceTokenizerFactory"/>
<filter class="solr.StopFilterFactory" ignoreCase="true" words="lang/stopwords_en.txt" enablePositionIncrements="true"/>
<filter class="solr.WordDelimiterFilterFactory" generateWordParts="1" generateNumberParts="1" catenateWords="1" catenateNumbers="1" catenateAll="1" splitOnCaseChange="0" splitOnNumerics="1" stemEnglishPossessive="1" preserveOriginal="1"/>
<filter class="solr.LowerCaseFilterFactory"/>
<filter class="solr.KeywordMarkerFilterFactory" protected="protwords.txt"/>
<filter class="solr.PorterStemFilterFactory"/>
<filter class="solr.RemoveDuplicatesTokenFilterFactory"/>
</analyzer>
This does great job for most of my text, but one thing I does that I don't like is it won't replace underscores to spaces; it strips them. For example, if I have "Solr_Lucene" it becomes "solrlucene" (one word). What I want is two words "solr lucene".
Thanks
-MJ
Re: How to deal with underscore
Posted by Otis Gospodnetic <ot...@gmail.com>.
Hi,
http://wiki.apache.org/solr/AnalyzersTokenizersTokenFilters says:
catenateWords="1" causes maximum runs of word parts to be catenated:
"wi-fi" => "wifi"
default is 0
Otis
--
Solr & ElasticSearch Support -- http://sematext.com/
Performance Monitoring -- http://sematext.com/spm
On Wed, Jul 3, 2013 at 12:05 PM, <jo...@aol.com> wrote:
> Hi,
>
>
> In my schema.xml, I have the following settings:
>
>
>
> <analyzer>
> <tokenizer class="solr.WhitespaceTokenizerFactory"/>
> <filter class="solr.StopFilterFactory" ignoreCase="true" words="lang/stopwords_en.txt" enablePositionIncrements="true"/>
> <filter class="solr.WordDelimiterFilterFactory" generateWordParts="1" generateNumberParts="1" catenateWords="1" catenateNumbers="1" catenateAll="1" splitOnCaseChange="0" splitOnNumerics="1" stemEnglishPossessive="1" preserveOriginal="1"/>
> <filter class="solr.LowerCaseFilterFactory"/>
> <filter class="solr.KeywordMarkerFilterFactory" protected="protwords.txt"/>
> <filter class="solr.PorterStemFilterFactory"/>
> <filter class="solr.RemoveDuplicatesTokenFilterFactory"/>
> </analyzer>
>
>
>
> This does great job for most of my text, but one thing I does that I don't like is it won't replace underscores to spaces; it strips them. For example, if I have "Solr_Lucene" it becomes "solrlucene" (one word). What I want is two words "solr lucene".
>
>
> Thanks
>
>
> -MJ