You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-commits@lucene.apache.org by Apache Wiki <wi...@apache.org> on 2008/08/14 19:51:58 UTC
[Solr Wiki] Update of "AnalyzersTokenizersTokenFilters" by ryan
Dear Wiki user,
You have subscribed to a wiki page or wiki category on "Solr Wiki" for change notification.
The following page has been changed by ryan:
http://wiki.apache.org/solr/AnalyzersTokenizersTokenFilters
The comment on the change is:
adding ShingleFilterFactory
------------------------------------------------------------------------------
Uses [http://jakarta.apache.org/commons/codec/ commons codec] to generate phonetically similar tokens. This currently supports [http://jakarta.apache.org/commons/codec/api-release/org/apache/commons/codec/language/package-summary.html four methods].
- || arg || value ||
+ || '''arg''' || '''value''' ||
|| encoder || one of: [http://jakarta.apache.org/commons/codec/api-release/org/apache/commons/codec/language/DoubleMetaphone.html DoubleMetaphone], [http://jakarta.apache.org/commons/codec/api-release/org/apache/commons/codec/language/Metaphone.html Metaphone], [http://jakarta.apache.org/commons/codec/api-release/org/apache/commons/codec/language/Soundex.html Soundex], [http://jakarta.apache.org/commons/codec/api-release/org/apache/commons/codec/language/RefinedSoundex.html RefinedSoundex] ||
|| inject|| true/false -- true will add tokens to the stream, false will replace the existing token ||
@@ -499, +499 @@
<filter class="solr.PhoneticFilterFactory" encoder="DoubleMetaphone" inject="true"/>
}}}
+
+ [[Anchor(ShingleFilterFactory)]]
+ ==== solr.ShingleFilterFactory ====
+
+ <!> ["Solr1.3"]
+
+ Creates [http://hudson.zones.apache.org/hudson/job/Lucene-trunk/javadoc/contrib-analyzers/org/apache/lucene/analysis/shingle/ShingleFilter.html org.apache.lucene.analysis.shingle.ShingleFilter].
+
+ A ShingleFilter constructs shingles (token n-grams) from a token stream. In other words, it creates combinations of tokens as a single token.
+
+ For example, the sentence "please divide this sentence into shingles" might be tokenized into shingles "please divide", "divide this", "this sentence", "sentence into", and "into shingles".
+
+
+ || '''arg''' || '''value''' ||
+ || maxShingleSize || default 2 ||
+ || outputUnigrams || default true ||
+
+ {{{
+ <filter class="solr.ShingleFilterFactory" maxShingleSize="true" outputUnigrams="true"/>
+ }}}
+