You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-commits@lucene.apache.org by Apache Wiki <wi...@apache.org> on 2008/08/14 19:51:58 UTC

[Solr Wiki] Update of "AnalyzersTokenizersTokenFilters" by ryan

Dear Wiki user,

You have subscribed to a wiki page or wiki category on "Solr Wiki" for change notification.

The following page has been changed by ryan:
http://wiki.apache.org/solr/AnalyzersTokenizersTokenFilters

The comment on the change is:
adding ShingleFilterFactory

------------------------------------------------------------------------------
  
  Uses [http://jakarta.apache.org/commons/codec/ commons codec] to generate phonetically similar tokens.  This currently supports [http://jakarta.apache.org/commons/codec/api-release/org/apache/commons/codec/language/package-summary.html four methods].
  
- || arg || value ||
+ || '''arg''' || '''value''' ||
  || encoder || one of: [http://jakarta.apache.org/commons/codec/api-release/org/apache/commons/codec/language/DoubleMetaphone.html DoubleMetaphone], [http://jakarta.apache.org/commons/codec/api-release/org/apache/commons/codec/language/Metaphone.html Metaphone], [http://jakarta.apache.org/commons/codec/api-release/org/apache/commons/codec/language/Soundex.html Soundex], [http://jakarta.apache.org/commons/codec/api-release/org/apache/commons/codec/language/RefinedSoundex.html RefinedSoundex] ||
  || inject|| true/false -- true will add tokens to the stream, false will replace the existing token ||
  
@@ -499, +499 @@

    <filter class="solr.PhoneticFilterFactory" encoder="DoubleMetaphone" inject="true"/>
  }}}
  
+ 
+ [[Anchor(ShingleFilterFactory)]]
+ ==== solr.ShingleFilterFactory ====
+ 
+ <!> ["Solr1.3"]
+ 
+ Creates [http://hudson.zones.apache.org/hudson/job/Lucene-trunk/javadoc/contrib-analyzers/org/apache/lucene/analysis/shingle/ShingleFilter.html org.apache.lucene.analysis.shingle.ShingleFilter].
+ 
+ A ShingleFilter constructs shingles (token n-grams) from a token stream. In other words, it creates combinations of tokens as a single token.
+ 
+ For example, the sentence "please divide this sentence into shingles" might be tokenized into shingles "please divide", "divide this", "this sentence", "sentence into", and "into shingles". 
+ 
+ 
+ || '''arg''' || '''value''' ||
+ || maxShingleSize || default 2 ||
+ || outputUnigrams || default true ||
+ 
+ {{{
+   <filter class="solr.ShingleFilterFactory" maxShingleSize="true" outputUnigrams="true"/>
+ }}}
+