You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by jo...@aol.com on 2013/07/03 18:05:28 UTC

How to deal with underscore

Hi,


In my schema.xml, I have the following settings:



      <analyzer>
        <tokenizer class="solr.WhitespaceTokenizerFactory"/>
        <filter class="solr.StopFilterFactory" ignoreCase="true" words="lang/stopwords_en.txt" enablePositionIncrements="true"/>
        <filter class="solr.WordDelimiterFilterFactory" generateWordParts="1" generateNumberParts="1" catenateWords="1" catenateNumbers="1" catenateAll="1" splitOnCaseChange="0" splitOnNumerics="1" stemEnglishPossessive="1" preserveOriginal="1"/>
        <filter class="solr.LowerCaseFilterFactory"/>
        <filter class="solr.KeywordMarkerFilterFactory" protected="protwords.txt"/>
        <filter class="solr.PorterStemFilterFactory"/>
		<filter class="solr.RemoveDuplicatesTokenFilterFactory"/>
      </analyzer>



This does great job for most of my text, but one thing I does that I don't like is it won't replace underscores to spaces; it strips them.  For example, if I have "Solr_Lucene" it becomes "solrlucene" (one word).  What I want is two words "solr lucene".


Thanks


-MJ

Re: How to deal with underscore

Posted by Otis Gospodnetic <ot...@gmail.com>.
Hi,

http://wiki.apache.org/solr/AnalyzersTokenizersTokenFilters says:

catenateWords="1" causes maximum runs of word parts to be catenated:

"wi-fi" => "wifi"

default is 0

Otis
--
Solr & ElasticSearch Support -- http://sematext.com/
Performance Monitoring -- http://sematext.com/spm



On Wed, Jul 3, 2013 at 12:05 PM,  <jo...@aol.com> wrote:
> Hi,
>
>
> In my schema.xml, I have the following settings:
>
>
>
>       <analyzer>
>         <tokenizer class="solr.WhitespaceTokenizerFactory"/>
>         <filter class="solr.StopFilterFactory" ignoreCase="true" words="lang/stopwords_en.txt" enablePositionIncrements="true"/>
>         <filter class="solr.WordDelimiterFilterFactory" generateWordParts="1" generateNumberParts="1" catenateWords="1" catenateNumbers="1" catenateAll="1" splitOnCaseChange="0" splitOnNumerics="1" stemEnglishPossessive="1" preserveOriginal="1"/>
>         <filter class="solr.LowerCaseFilterFactory"/>
>         <filter class="solr.KeywordMarkerFilterFactory" protected="protwords.txt"/>
>         <filter class="solr.PorterStemFilterFactory"/>
>                 <filter class="solr.RemoveDuplicatesTokenFilterFactory"/>
>       </analyzer>
>
>
>
> This does great job for most of my text, but one thing I does that I don't like is it won't replace underscores to spaces; it strips them.  For example, if I have "Solr_Lucene" it becomes "solrlucene" (one word).  What I want is two words "solr lucene".
>
>
> Thanks
>
>
> -MJ