You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@lucene.apache.org by ct...@apache.org on 2018/10/22 23:34:52 UTC
[07/50] [abbrv] lucene-solr:jira/solr-12746: LUCENE-8526: Add
javadocs in CJKBigramFilter explaining the behavior of the StandardTokenizer
on Hangul syllables.
LUCENE-8526: Add javadocs in CJKBigramFilter explaining the behavior of the StandardTokenizer on Hangul syllables.
Project: http://git-wip-us.apache.org/repos/asf/lucene-solr/repo
Commit: http://git-wip-us.apache.org/repos/asf/lucene-solr/commit/ef1338a5
Tree: http://git-wip-us.apache.org/repos/asf/lucene-solr/tree/ef1338a5
Diff: http://git-wip-us.apache.org/repos/asf/lucene-solr/diff/ef1338a5
Branch: refs/heads/jira/solr-12746
Commit: ef1338a5918b3a662e930eb636c1ee53144ac39b
Parents: 94a1361
Author: Jim Ferenczi <ji...@apache.org>
Authored: Thu Oct 11 13:49:14 2018 +0100
Committer: Cassandra Targett <ct...@apache.org>
Committed: Sun Oct 21 15:46:46 2018 -0500
----------------------------------------------------------------------
.../java/org/apache/lucene/analysis/cjk/CJKBigramFilter.java | 8 ++++++++
1 file changed, 8 insertions(+)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/ef1338a5/lucene/analysis/common/src/java/org/apache/lucene/analysis/cjk/CJKBigramFilter.java
----------------------------------------------------------------------
diff --git a/lucene/analysis/common/src/java/org/apache/lucene/analysis/cjk/CJKBigramFilter.java b/lucene/analysis/common/src/java/org/apache/lucene/analysis/cjk/CJKBigramFilter.java
index bf4f621..7d79b84 100644
--- a/lucene/analysis/common/src/java/org/apache/lucene/analysis/cjk/CJKBigramFilter.java
+++ b/lucene/analysis/common/src/java/org/apache/lucene/analysis/cjk/CJKBigramFilter.java
@@ -43,6 +43,14 @@ import org.apache.lucene.util.ArrayUtil;
* flag in {@link CJKBigramFilter#CJKBigramFilter(TokenStream, int, boolean)}.
* This can be used for a combined unigram+bigram approach.
* <p>
+ * Unlike ICUTokenizer, StandardTokenizer does not split at script boundaries.
+ * Korean Hangul characters are treated the same as many other scripts'
+ * letters, and as a result, StandardTokenizer can produce tokens that mix
+ * Hangul and non-Hangul characters, e.g. "한국abc". Such mixed-script tokens
+ * are typed as <code><ALPHANUM></code> rather than
+ * <code><HANGUL></code>, and as a result, will not be converted to
+ * bigrams by CJKBigramFilter.
+ *
* In all cases, all non-CJK input is passed thru unmodified.
*/
public final class CJKBigramFilter extends TokenFilter {