You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by "Cheolgoo Kang (JIRA)" <ji...@apache.org> on 2005/10/04 16:26:50 UTC

[jira] Created: (LUCENE-444) StandardTokenizer loses Korean characters

StandardTokenizer loses Korean characters
-----------------------------------------

         Key: LUCENE-444
         URL: http://issues.apache.org/jira/browse/LUCENE-444
     Project: Lucene - Java
        Type: Bug
  Components: Analysis  
    Reporter: Cheolgoo Kang
    Priority: Minor


While using StandardAnalyzer, exp. StandardTokenizer with Korean text stream, StandardTokenizer ignores the Korean characters. This is because the definition of CJK token in StandardTokenizer.jj JavaCC file doesn't have enough range covering Korean syllables described in Unicode character map.
This patch adds one line of 0xAC00~0xD7AF, the Korean syllables range to the StandardTokenizer.jj code.

-- 
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
   http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see:
   http://www.atlassian.com/software/jira


---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org


Re: [jira] Created: (LUCENE-444) StandardTokenizer loses Korean characters

Posted by Cheolgoo Kang <ap...@gmail.com>.
AFAIK, some of commiter will merge this patch into Lucene code base
with v1.9 release.

On 10/5/05, Youngho Cho <yo...@nannet.co.kr> wrote:
> Hello,
>
> Is there any plan to add this patch into lucene core ?
> I am using CJKAnalyzer but I hope to switch to the StanadardAnalyzer.
>
> Thanks,
>
> Youngho
>
> ----- Original Message -----
> From: "Cheolgoo Kang (JIRA)" <ji...@apache.org>
> To: <ja...@lucene.apache.org>
> Sent: Tuesday, October 04, 2005 11:26 PM
> Subject: [jira] Created: (LUCENE-444) StandardTokenizer loses Korean characters
>
>
> > StandardTokenizer loses Korean characters
> > -----------------------------------------
> >
> >          Key: LUCENE-444
> >          URL: http://issues.apache.org/jira/browse/LUCENE-444
> >      Project: Lucene - Java
> >         Type: Bug
> >   Components: Analysis
> >     Reporter: Cheolgoo Kang
> >     Priority: Minor
> >
> >
> > While using StandardAnalyzer, exp. StandardTokenizer with Korean text stream, StandardTokenizer ignores the Korean characters. This is because the definition of CJK token in StandardTokenizer.jj JavaCC file doesn't have enough range covering Korean syllables described in Unicode character map.
> > This patch adds one line of 0xAC00~0xD7AF, the Korean syllables range to the StandardTokenizer.jj code.
> >
> > --
> > This message is automatically generated by JIRA.
> > -
> > If you think it was sent incorrectly contact one of the administrators:
> >    http://issues.apache.org/jira/secure/Administrators.jspa
> > -
> > For more information on JIRA, see:
> >    http://www.atlassian.com/software/jira
> >
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
> > For additional commands, e-mail: java-dev-help@lucene.apache.org


--
Cheolgoo

---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org


Re: [jira] Created: (LUCENE-444) StandardTokenizer loses Korean characters

Posted by Erik Hatcher <er...@ehatchersolutions.com>.
Never mind.... I see Otis beat me to it.

     Erik


On Oct 4, 2005, at 10:38 PM, Youngho Cho wrote:

> Hello,
>
> Is there any plan to add this patch into lucene core ?
> I am using CJKAnalyzer but I hope to switch to the StanadardAnalyzer.
>
> Thanks,
>
> Youngho
>
> ----- Original Message -----
> From: "Cheolgoo Kang (JIRA)" <ji...@apache.org>
> To: <ja...@lucene.apache.org>
> Sent: Tuesday, October 04, 2005 11:26 PM
> Subject: [jira] Created: (LUCENE-444) StandardTokenizer loses  
> Korean characters
>
>
>
>> StandardTokenizer loses Korean characters
>> -----------------------------------------
>>
>>          Key: LUCENE-444
>>          URL: http://issues.apache.org/jira/browse/LUCENE-444
>>      Project: Lucene - Java
>>         Type: Bug
>>   Components: Analysis
>>     Reporter: Cheolgoo Kang
>>     Priority: Minor
>>
>>
>> While using StandardAnalyzer, exp. StandardTokenizer with Korean  
>> text stream, StandardTokenizer ignores the Korean characters. This  
>> is because the definition of CJK token in StandardTokenizer.jj  
>> JavaCC file doesn't have enough range covering Korean syllables  
>> described in Unicode character map.
>> This patch adds one line of 0xAC00~0xD7AF, the Korean syllables  
>> range to the StandardTokenizer.jj code.
>>
>> -- 
>> This message is automatically generated by JIRA.
>> -
>> If you think it was sent incorrectly contact one of the  
>> administrators:
>>    http://issues.apache.org/jira/secure/Administrators.jspa
>> -
>> For more information on JIRA, see:
>>    http://www.atlassian.com/software/jira
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
>> For additional commands, e-mail: java-dev-help@lucene.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org


Re: [jira] Created: (LUCENE-444) StandardTokenizer loses Korean characters

Posted by Erik Hatcher <er...@ehatchersolutions.com>.
I'm going to apply this patch right now, thanks for the prod.  CJK  
just happens to be of keen interest to me :)

     Erik

On Oct 4, 2005, at 10:38 PM, Youngho Cho wrote:

> Hello,
>
> Is there any plan to add this patch into lucene core ?
> I am using CJKAnalyzer but I hope to switch to the StanadardAnalyzer.
>
> Thanks,
>
> Youngho
>
> ----- Original Message -----
> From: "Cheolgoo Kang (JIRA)" <ji...@apache.org>
> To: <ja...@lucene.apache.org>
> Sent: Tuesday, October 04, 2005 11:26 PM
> Subject: [jira] Created: (LUCENE-444) StandardTokenizer loses  
> Korean characters
>
>
>
>> StandardTokenizer loses Korean characters
>> -----------------------------------------
>>
>>          Key: LUCENE-444
>>          URL: http://issues.apache.org/jira/browse/LUCENE-444
>>      Project: Lucene - Java
>>         Type: Bug
>>   Components: Analysis
>>     Reporter: Cheolgoo Kang
>>     Priority: Minor
>>
>>
>> While using StandardAnalyzer, exp. StandardTokenizer with Korean  
>> text stream, StandardTokenizer ignores the Korean characters. This  
>> is because the definition of CJK token in StandardTokenizer.jj  
>> JavaCC file doesn't have enough range covering Korean syllables  
>> described in Unicode character map.
>> This patch adds one line of 0xAC00~0xD7AF, the Korean syllables  
>> range to the StandardTokenizer.jj code.
>>
>> -- 
>> This message is automatically generated by JIRA.
>> -
>> If you think it was sent incorrectly contact one of the  
>> administrators:
>>    http://issues.apache.org/jira/secure/Administrators.jspa
>> -
>> For more information on JIRA, see:
>>    http://www.atlassian.com/software/jira
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
>> For additional commands, e-mail: java-dev-help@lucene.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org


Re: [jira] Created: (LUCENE-444) StandardTokenizer loses Korean characters

Posted by Youngho Cho <yo...@nannet.co.kr>.
Great !


Thanks a lot Otis and Cheolgoo.

Youngho

----- Original Message ----- 
From: "Otis Gospodnetic" <ot...@yahoo.com>
To: <ja...@lucene.apache.org>; "Youngho Cho" <yo...@nannet.co.kr>
Sent: Wednesday, October 05, 2005 12:55 PM
Subject: Re: [jira] Created: (LUCENE-444) StandardTokenizer loses Korean characters


> Try the version from SVN, I just applied Cheolgoo's patch.
> 
> Otis
> 
> --- Youngho Cho <yo...@nannet.co.kr> wrote:
> 
> > Hello,
> > 
> > Is there any plan to add this patch into lucene core ?
> > I am using CJKAnalyzer but I hope to switch to the StanadardAnalyzer.
> > 
> > Thanks,
> > 
> > Youngho
> > 
> > ----- Original Message ----- 
> > From: "Cheolgoo Kang (JIRA)" <ji...@apache.org>
> > To: <ja...@lucene.apache.org>
> > Sent: Tuesday, October 04, 2005 11:26 PM
> > Subject: [jira] Created: (LUCENE-444) StandardTokenizer loses Korean
> > characters
> > 
> > 
> > > StandardTokenizer loses Korean characters
> > > -----------------------------------------
> > > 
> > >          Key: LUCENE-444
> > >          URL: http://issues.apache.org/jira/browse/LUCENE-444
> > >      Project: Lucene - Java
> > >         Type: Bug
> > >   Components: Analysis  
> > >     Reporter: Cheolgoo Kang
> > >     Priority: Minor
> > > 
> > > 
> > > While using StandardAnalyzer, exp. StandardTokenizer with Korean
> > text stream, StandardTokenizer ignores the Korean characters. This is
> > because the definition of CJK token in StandardTokenizer.jj JavaCC
> > file doesn't have enough range covering Korean syllables described in
> > Unicode character map.
> > > This patch adds one line of 0xAC00~0xD7AF, the Korean syllables
> > range to the StandardTokenizer.jj code.
> > > 
> > > -- 
> > > This message is automatically generated by JIRA.
> > > -
> > > If you think it was sent incorrectly contact one of the
> > administrators:
> > >    http://issues.apache.org/jira/secure/Administrators.jspa
> > > -
> > > For more information on JIRA, see:
> > >    http://www.atlassian.com/software/jira
> > > 
> > > 
> > >
> > ---------------------------------------------------------------------
> > > To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
> > > For additional commands, e-mail: java-dev-help@lucene.apache.org

Re: [jira] Created: (LUCENE-444) StandardTokenizer loses Korean characters

Posted by Otis Gospodnetic <ot...@yahoo.com>.
Try the version from SVN, I just applied Cheolgoo's patch.

Otis

--- Youngho Cho <yo...@nannet.co.kr> wrote:

> Hello,
> 
> Is there any plan to add this patch into lucene core ?
> I am using CJKAnalyzer but I hope to switch to the StanadardAnalyzer.
> 
> Thanks,
> 
> Youngho
> 
> ----- Original Message ----- 
> From: "Cheolgoo Kang (JIRA)" <ji...@apache.org>
> To: <ja...@lucene.apache.org>
> Sent: Tuesday, October 04, 2005 11:26 PM
> Subject: [jira] Created: (LUCENE-444) StandardTokenizer loses Korean
> characters
> 
> 
> > StandardTokenizer loses Korean characters
> > -----------------------------------------
> > 
> >          Key: LUCENE-444
> >          URL: http://issues.apache.org/jira/browse/LUCENE-444
> >      Project: Lucene - Java
> >         Type: Bug
> >   Components: Analysis  
> >     Reporter: Cheolgoo Kang
> >     Priority: Minor
> > 
> > 
> > While using StandardAnalyzer, exp. StandardTokenizer with Korean
> text stream, StandardTokenizer ignores the Korean characters. This is
> because the definition of CJK token in StandardTokenizer.jj JavaCC
> file doesn't have enough range covering Korean syllables described in
> Unicode character map.
> > This patch adds one line of 0xAC00~0xD7AF, the Korean syllables
> range to the StandardTokenizer.jj code.
> > 
> > -- 
> > This message is automatically generated by JIRA.
> > -
> > If you think it was sent incorrectly contact one of the
> administrators:
> >    http://issues.apache.org/jira/secure/Administrators.jspa
> > -
> > For more information on JIRA, see:
> >    http://www.atlassian.com/software/jira
> > 
> > 
> >
> ---------------------------------------------------------------------
> > To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
> > For additional commands, e-mail: java-dev-help@lucene.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org


Re: [jira] Created: (LUCENE-444) StandardTokenizer loses Korean characters

Posted by Youngho Cho <yo...@nannet.co.kr>.
Hello,

Is there any plan to add this patch into lucene core ?
I am using CJKAnalyzer but I hope to switch to the StanadardAnalyzer.

Thanks,

Youngho

----- Original Message ----- 
From: "Cheolgoo Kang (JIRA)" <ji...@apache.org>
To: <ja...@lucene.apache.org>
Sent: Tuesday, October 04, 2005 11:26 PM
Subject: [jira] Created: (LUCENE-444) StandardTokenizer loses Korean characters


> StandardTokenizer loses Korean characters
> -----------------------------------------
> 
>          Key: LUCENE-444
>          URL: http://issues.apache.org/jira/browse/LUCENE-444
>      Project: Lucene - Java
>         Type: Bug
>   Components: Analysis  
>     Reporter: Cheolgoo Kang
>     Priority: Minor
> 
> 
> While using StandardAnalyzer, exp. StandardTokenizer with Korean text stream, StandardTokenizer ignores the Korean characters. This is because the definition of CJK token in StandardTokenizer.jj JavaCC file doesn't have enough range covering Korean syllables described in Unicode character map.
> This patch adds one line of 0xAC00~0xD7AF, the Korean syllables range to the StandardTokenizer.jj code.
> 
> -- 
> This message is automatically generated by JIRA.
> -
> If you think it was sent incorrectly contact one of the administrators:
>    http://issues.apache.org/jira/secure/Administrators.jspa
> -
> For more information on JIRA, see:
>    http://www.atlassian.com/software/jira
> 
> 
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-dev-help@lucene.apache.org

[jira] Resolved: (LUCENE-444) StandardTokenizer loses Korean characters

Posted by "Otis Gospodnetic (JIRA)" <ji...@apache.org>.
     [ http://issues.apache.org/jira/browse/LUCENE-444?page=all ]
     
Otis Gospodnetic resolved LUCENE-444:
-------------------------------------

    Fix Version: 1.9
     Resolution: Fixed

Committed.  Thanks Cheolgoo.

> StandardTokenizer loses Korean characters
> -----------------------------------------
>
>          Key: LUCENE-444
>          URL: http://issues.apache.org/jira/browse/LUCENE-444
>      Project: Lucene - Java
>         Type: Bug
>   Components: Analysis
>     Reporter: Cheolgoo Kang
>     Priority: Minor
>      Fix For: 1.9
>  Attachments: StandardTokenizer_Korean.patch
>
> While using StandardAnalyzer, exp. StandardTokenizer with Korean text stream, StandardTokenizer ignores the Korean characters. This is because the definition of CJK token in StandardTokenizer.jj JavaCC file doesn't have enough range covering Korean syllables described in Unicode character map.
> This patch adds one line of 0xAC00~0xD7AF, the Korean syllables range to the StandardTokenizer.jj code.

-- 
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
   http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see:
   http://www.atlassian.com/software/jira


---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org


[jira] Updated: (LUCENE-444) StandardTokenizer loses Korean characters

Posted by "Cheolgoo Kang (JIRA)" <ji...@apache.org>.
     [ http://issues.apache.org/jira/browse/LUCENE-444?page=all ]

Cheolgoo Kang updated LUCENE-444:
---------------------------------

    Attachment: StandardTokenizer_Korean.patch

This patch adds one line of 0xAC00~0xD7AF, the Korean syllables range to the StandardTokenizer.jj code.

> StandardTokenizer loses Korean characters
> -----------------------------------------
>
>          Key: LUCENE-444
>          URL: http://issues.apache.org/jira/browse/LUCENE-444
>      Project: Lucene - Java
>         Type: Bug
>   Components: Analysis
>     Reporter: Cheolgoo Kang
>     Priority: Minor
>  Attachments: StandardTokenizer_Korean.patch
>
> While using StandardAnalyzer, exp. StandardTokenizer with Korean text stream, StandardTokenizer ignores the Korean characters. This is because the definition of CJK token in StandardTokenizer.jj JavaCC file doesn't have enough range covering Korean syllables described in Unicode character map.
> This patch adds one line of 0xAC00~0xD7AF, the Korean syllables range to the StandardTokenizer.jj code.

-- 
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
   http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see:
   http://www.atlassian.com/software/jira


---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org


Re: Document Duplication for Multiple Segment Merge

Posted by Michael Ji <fj...@yahoo.com>.
Sorry, I guess I point out a wrong java class name.

I want to be confirmed that if SegmentMerger.java in
Lucene do dedup or not. I tracing down couple of java
class from SegmentMerger.java, such as,
SegmentReader.java, IndexWriter.java,
SegmentReader.java. I didn't see a dedup mechanism
yet.

thanks,

Micheal Ji,

--- Yonik Seeley <ys...@gmail.com> wrote:

> Sorry, I've only briefly looked at Nutch, so you
> should ask on that mailing
> list.
> Lucene doesn't do deduping.
> 
> 
> -Yonik
> Now hiring -- http://tinyurl.com/7m67g
> 
> On 10/14/05, Michael Ji <fj...@yahoo.com> wrote:
> >
> > hi Yonik:
> >
> > Does that mean when two documents has same MD5
> content
> > in two different segments, IndexMerger.java will
> keep
> > both of them?
> >
> > When I look at the code of IndexSegment.java, it
> > handle MD5 dedupling by keeping the one with
> higher
> > document ID.
> >
> 



	
		
__________________________________ 
Yahoo! Mail - PC Magazine Editors' Choice 2005 
http://mail.yahoo.com

---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org


Re: Document Duplication for Multiple Segment Merge

Posted by Yonik Seeley <ys...@gmail.com>.
Sorry, I've only briefly looked at Nutch, so you should ask on that mailing
list.
Lucene doesn't do deduping.


-Yonik
Now hiring -- http://tinyurl.com/7m67g

On 10/14/05, Michael Ji <fj...@yahoo.com> wrote:
>
> hi Yonik:
>
> Does that mean when two documents has same MD5 content
> in two different segments, IndexMerger.java will keep
> both of them?
>
> When I look at the code of IndexSegment.java, it
> handle MD5 dedupling by keeping the one with higher
> document ID.
>

Re: Document Duplication for Multiple Segment Merge

Posted by Michael Ji <fj...@yahoo.com>.
hi Yonik:

Does that mean when two documents has same MD5 content
in two different segments, IndexMerger.java  will keep
both of them?

When I look at the code of IndexSegment.java, it
handle MD5 dedupling by keeping the one with higher
document ID.

So, when refetching happens, the old segment should be
discarded totally. And, a strategy must be made in
such a way that each segment should relate to a
fetchlist with same interval time. Is it the way Nutch
handling refetching case?


Michael Ji,

--- Yonik Seeley <ys...@gmail.com> wrote:

> There is no concept in Lucene of document identity
> linked to any fields of a
> document.
> You need to handle removal of duplicates yourself.
> 
> -Yonik
> Now hiring -- http://tinyurl.com/7m67g
> 
> 
> On 10/14/05, Michael Ji <fj...@yahoo.com> wrote:
> >
> > hi,
> >
> > When Nutch's IndexMerger.java is called, the
> indexes
> > from multiple segment directories are merged to
> one
> > target directory.
> >
> > I wonder how lucene deals with the case when
> identical
> > documents existing in two segments. Is the older
> > document ( lower time stamp ) deleted?
> >
> > thanks,
> >
> > Michael Ji,
> >
> >
> >
> > __________________________________
> > Yahoo! Music Unlimited
> > Access over 1 million songs. Try it free.
> > http://music.yahoo.com/unlimited/
> >
> >
>
---------------------------------------------------------------------
> > To unsubscribe, e-mail:
> java-dev-unsubscribe@lucene.apache.org
> > For additional commands, e-mail:
> java-dev-help@lucene.apache.org
> >
> >
> 



		
__________________________________ 
Yahoo! Music Unlimited 
Access over 1 million songs. Try it free.
http://music.yahoo.com/unlimited/

---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org


Re: Document Duplication for Multiple Segment Merge

Posted by Yonik Seeley <ys...@gmail.com>.
There is no concept in Lucene of document identity linked to any fields of a
document.
You need to handle removal of duplicates yourself.

-Yonik
Now hiring -- http://tinyurl.com/7m67g


On 10/14/05, Michael Ji <fj...@yahoo.com> wrote:
>
> hi,
>
> When Nutch's IndexMerger.java is called, the indexes
> from multiple segment directories are merged to one
> target directory.
>
> I wonder how lucene deals with the case when identical
> documents existing in two segments. Is the older
> document ( lower time stamp ) deleted?
>
> thanks,
>
> Michael Ji,
>
>
>
> __________________________________
> Yahoo! Music Unlimited
> Access over 1 million songs. Try it free.
> http://music.yahoo.com/unlimited/
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-dev-help@lucene.apache.org
>
>

Document Duplication for Multiple Segment Merge

Posted by Michael Ji <fj...@yahoo.com>.
hi,

When Nutch's IndexMerger.java is called, the indexes
from multiple segment directories are merged to one
target directory.

I wonder how lucene deals with the case when identical
documents existing in two segments. Is the older
document ( lower time stamp ) deleted? 

thanks,

Michael Ji,


		
__________________________________ 
Yahoo! Music Unlimited 
Access over 1 million songs. Try it free.
http://music.yahoo.com/unlimited/

---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org


[jira] Closed: (LUCENE-444) StandardTokenizer loses Korean characters

Posted by "Erik Hatcher (JIRA)" <ji...@apache.org>.
     [ http://issues.apache.org/jira/browse/LUCENE-444?page=all ]
     
Erik Hatcher closed LUCENE-444:
-------------------------------


I'm closing this issue... but some unit tests would be nice to go along with this too, eventually :)

> StandardTokenizer loses Korean characters
> -----------------------------------------
>
>          Key: LUCENE-444
>          URL: http://issues.apache.org/jira/browse/LUCENE-444
>      Project: Lucene - Java
>         Type: Bug
>   Components: Analysis
>     Reporter: Cheolgoo Kang
>     Priority: Minor
>      Fix For: 1.9
>  Attachments: StandardTokenizer_Korean.patch
>
> While using StandardAnalyzer, exp. StandardTokenizer with Korean text stream, StandardTokenizer ignores the Korean characters. This is because the definition of CJK token in StandardTokenizer.jj JavaCC file doesn't have enough range covering Korean syllables described in Unicode character map.
> This patch adds one line of 0xAC00~0xD7AF, the Korean syllables range to the StandardTokenizer.jj code.

-- 
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
   http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see:
   http://www.atlassian.com/software/jira


---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org