You are viewing a plain text version of this content. The canonical link for it is here.
Posted to java-commits@lucene.apache.org by ot...@apache.org on 2006/08/13 09:47:35 UTC
svn commit: r431152 - in /lucene/java/trunk: CHANGES.txt
src/java/org/apache/lucene/analysis/standard/StandardTokenizer.jj
src/java/org/apache/lucene/analysis/standard/StandardTokenizerTokenManager.java
Author: otis
Date: Sun Aug 13 00:47:34 2006
New Revision: 431152
URL: http://svn.apache.org/viewvc?rev=431152&view=rev
Log:
- Modified some CJK Unicode code point ranges in StandardTokenizer.jj,
and added a few more of them to increase CJK character coverage.
Also documented some of the ranges.
Modified:
lucene/java/trunk/CHANGES.txt
lucene/java/trunk/src/java/org/apache/lucene/analysis/standard/StandardTokenizer.jj
lucene/java/trunk/src/java/org/apache/lucene/analysis/standard/StandardTokenizerTokenManager.java
Modified: lucene/java/trunk/CHANGES.txt
URL: http://svn.apache.org/viewvc/lucene/java/trunk/CHANGES.txt?rev=431152&r1=431151&r2=431152&view=diff
==============================================================================
--- lucene/java/trunk/CHANGES.txt (original)
+++ lucene/java/trunk/CHANGES.txt Sun Aug 13 00:47:34 2006
@@ -15,7 +15,12 @@
2. LUCENE-478: Updated the list of Unicode code point ranges for CJK (now
split into CJ and K) in StandardAnalyzer.
- (John Want and Steven Rowe via Otis Gospodnetic)
+ (John Wang and Steven Rowe via Otis Gospodnetic)
+
+ 3. Modified some CJK Unicode code point ranges in StandardTokenizer.jj,
+ and added a few more of them to increase CJK character coverage.
+ Also documented some of the ranges.
+ (Otis Gospodnetic)
New features
Modified: lucene/java/trunk/src/java/org/apache/lucene/analysis/standard/StandardTokenizer.jj
URL: http://svn.apache.org/viewvc/lucene/java/trunk/src/java/org/apache/lucene/analysis/standard/StandardTokenizer.jj?rev=431152&r1=431151&r2=431152&view=diff
==============================================================================
--- lucene/java/trunk/src/java/org/apache/lucene/analysis/standard/StandardTokenizer.jj (original)
+++ lucene/java/trunk/src/java/org/apache/lucene/analysis/standard/StandardTokenizer.jj Sun Aug 13 00:47:34 2006
@@ -110,17 +110,32 @@
| < CJ: // Chinese, Japanese
[
"\u3040"-"\u318f",
- "\u31f0"-"\u31ff",
+ "\u3100"-"\u312f", // BaPoMoFo (aka ZhuYin)
+ "\u3040"-"\u309F", // Japanese: Hiragana
+ "\u30A0"-"\u30FF", // Japanese: Katakana
+ "\u31F0"-"\u31FF", // Japanese: Katakana Phonetic Extensions
"\u3300"-"\u337f",
- "\u3400"-"\u4db5",
+ "\u3400"-"\u4dbf", // CJK Unified Ideographs Ext. A
"\u4e00"-"\u9fff",
"\uf900"-"\ufaff",
"\uff65"-"\uff9f"
+
+// Otis: consider adding these, too
+//
+// 2E80-2EFF: CJK Radicals Supplement
+// 2F00-2FDF: Kangxi Radicals
+// 3190-319F: Kanbun
+// 31C0-31EF: CJK Strokes
+// 4E00-9FBF: CJK Unified
+// F900-FAFF: CJK Compatibility Ideographs
+
]
>
| < KOREAN: // Korean
[
- "\uac00"-"\ud7a3"
+ "\uac00"-"\ud7af", // Hangul Syllables
+ "\u1100"-"\u11ff" // Hangul Jamo
+ // "\uac00"-"\ud7a3"
]
>
| < #DIGIT: // unicode digits
Modified: lucene/java/trunk/src/java/org/apache/lucene/analysis/standard/StandardTokenizerTokenManager.java
URL: http://svn.apache.org/viewvc/lucene/java/trunk/src/java/org/apache/lucene/analysis/standard/StandardTokenizerTokenManager.java?rev=431152&r1=431151&r2=431152&view=diff
==============================================================================
--- lucene/java/trunk/src/java/org/apache/lucene/analysis/standard/StandardTokenizerTokenManager.java (original)
+++ lucene/java/trunk/src/java/org/apache/lucene/analysis/standard/StandardTokenizerTokenManager.java Sun Aug 13 00:47:34 2006
@@ -53,16 +53,16 @@
0xffffffffffffffffL, 0xffffffffffffffffL, 0x0L, 0x0L
};
static final long[] jjbitVec5 = {
- 0xffffffffffffffffL, 0xffffffffffffffffL, 0x3fffffffffffffL, 0x0L
+ 0xffffffffffffffffL, 0xffffffffffffffffL, 0xffffffffffffffffL, 0x0L
};
static final long[] jjbitVec6 = {
0x0L, 0xffffffe000000000L, 0xffffffffL, 0x0L
};
static final long[] jjbitVec7 = {
- 0x0L, 0x0L, 0xfffff00000000000L, 0x7fffffL
+ 0x20000L, 0x0L, 0xfffff00000000000L, 0x7fffffL
};
static final long[] jjbitVec8 = {
- 0xffffffffffffffffL, 0xffffffffffffffffL, 0xfffffffffL, 0x0L
+ 0xffffffffffffffffL, 0xffffffffffffffffL, 0xffffffffffffL, 0x0L
};
static final long[] jjbitVec9 = {
0xfffffffeL, 0x0L, 0x0L, 0x0L