You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by "Daniel Cheng (JIRA)" <ji...@apache.org> on 2008/12/12 07:04:46 UTC
[jira] Created: (LUCENE-1490) CJKTokenizer convert
HALFWIDTH_AND_FULLWIDTH_FORMS wrong
CJKTokenizer convert HALFWIDTH_AND_FULLWIDTH_FORMS wrong
----------------------------------------------------------
Key: LUCENE-1490
URL: https://issues.apache.org/jira/browse/LUCENE-1490
Project: Lucene - Java
Issue Type: Bug
Reporter: Daniel Cheng
Fix For: 2.4
CJKTokenizer have these lines..
if (ub == Character.UnicodeBlock.HALFWIDTH_AND_FULLWIDTH_FORMS) {
/** convert HALFWIDTH_AND_FULLWIDTH_FORMS to BASIC_LATIN */
int i = (int) c;
i = i - 65248;
c = (char) i;
}
This is wrong. Some character in the block (e.g. U+ff68) have no BASIC_LATIN counterparts.
Only 65281-65374 can be converted this way.
The fix is
if (ub == Character.UnicodeBlock.HALFWIDTH_AND_FULLWIDTH_FORMS && i <= 65474 && i> 65281) {
/** convert HALFWIDTH_AND_FULLWIDTH_FORMS to BASIC_LATIN */
int i = (int) c;
i = i - 65248;
c = (char) i;
}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Updated: (LUCENE-1490) CJKTokenizer convert
HALFWIDTH_AND_FULLWIDTH_FORMS wrong
Posted by "Michael McCandless (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-1490?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Michael McCandless updated LUCENE-1490:
---------------------------------------
Lucene Fields: [New, Patch Available] (was: [Patch Available, New])
Fix Version/s: 2.9
(Adding trunk release (2.9) to fix version too)
> CJKTokenizer convert HALFWIDTH_AND_FULLWIDTH_FORMS wrong
> ----------------------------------------------------------
>
> Key: LUCENE-1490
> URL: https://issues.apache.org/jira/browse/LUCENE-1490
> Project: Lucene - Java
> Issue Type: Bug
> Reporter: Daniel Cheng
> Fix For: 2.4, 2.9
>
>
> CJKTokenizer have these lines..
> if (ub == Character.UnicodeBlock.HALFWIDTH_AND_FULLWIDTH_FORMS) {
> /** convert HALFWIDTH_AND_FULLWIDTH_FORMS to BASIC_LATIN */
> int i = (int) c;
> i = i - 65248;
> c = (char) i;
> }
> This is wrong. Some character in the block (e.g. U+ff68) have no BASIC_LATIN counterparts.
> Only 65281-65374 can be converted this way.
> The fix is
> if (ub == Character.UnicodeBlock.HALFWIDTH_AND_FULLWIDTH_FORMS && i <= 65474 && i> 65281) {
> /** convert HALFWIDTH_AND_FULLWIDTH_FORMS to BASIC_LATIN */
> int i = (int) c;
> i = i - 65248;
> c = (char) i;
> }
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Commented: (LUCENE-1490) CJKTokenizer convert
HALFWIDTH_AND_FULLWIDTH_FORMS wrong
Posted by "Daniel Cheng (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-1490?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12683240#action_12683240 ]
Daniel Cheng commented on LUCENE-1490:
--------------------------------------
This was discovered by Chan http://www.cnblogs.com/jjstar/archive/2006/12/20/598016.html
> CJKTokenizer convert HALFWIDTH_AND_FULLWIDTH_FORMS wrong
> ----------------------------------------------------------
>
> Key: LUCENE-1490
> URL: https://issues.apache.org/jira/browse/LUCENE-1490
> Project: Lucene - Java
> Issue Type: Bug
> Reporter: Daniel Cheng
> Assignee: Michael McCandless
> Fix For: 2.4, 2.9
>
>
> CJKTokenizer have these lines..
> if (ub == Character.UnicodeBlock.HALFWIDTH_AND_FULLWIDTH_FORMS) {
> /** convert HALFWIDTH_AND_FULLWIDTH_FORMS to BASIC_LATIN */
> int i = (int) c;
> i = i - 65248;
> c = (char) i;
> }
> This is wrong. Some character in the block (e.g. U+ff68) have no BASIC_LATIN counterparts.
> Only 65281-65374 can be converted this way.
> The fix is
> if (ub == Character.UnicodeBlock.HALFWIDTH_AND_FULLWIDTH_FORMS && i <= 65474 && i> 65281) {
> /** convert HALFWIDTH_AND_FULLWIDTH_FORMS to BASIC_LATIN */
> int i = (int) c;
> i = i - 65248;
> c = (char) i;
> }
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Resolved: (LUCENE-1490) CJKTokenizer convert
HALFWIDTH_AND_FULLWIDTH_FORMS wrong
Posted by "Michael McCandless (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-1490?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Michael McCandless resolved LUCENE-1490.
----------------------------------------
Resolution: Fixed
Thanks Daniel!
> CJKTokenizer convert HALFWIDTH_AND_FULLWIDTH_FORMS wrong
> ----------------------------------------------------------
>
> Key: LUCENE-1490
> URL: https://issues.apache.org/jira/browse/LUCENE-1490
> Project: Lucene - Java
> Issue Type: Bug
> Reporter: Daniel Cheng
> Assignee: Michael McCandless
> Fix For: 2.9, 2.4
>
>
> CJKTokenizer have these lines..
> if (ub == Character.UnicodeBlock.HALFWIDTH_AND_FULLWIDTH_FORMS) {
> /** convert HALFWIDTH_AND_FULLWIDTH_FORMS to BASIC_LATIN */
> int i = (int) c;
> i = i - 65248;
> c = (char) i;
> }
> This is wrong. Some character in the block (e.g. U+ff68) have no BASIC_LATIN counterparts.
> Only 65281-65374 can be converted this way.
> The fix is
> if (ub == Character.UnicodeBlock.HALFWIDTH_AND_FULLWIDTH_FORMS && i <= 65474 && i> 65281) {
> /** convert HALFWIDTH_AND_FULLWIDTH_FORMS to BASIC_LATIN */
> int i = (int) c;
> i = i - 65248;
> c = (char) i;
> }
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Assigned: (LUCENE-1490) CJKTokenizer convert
HALFWIDTH_AND_FULLWIDTH_FORMS wrong
Posted by "Michael McCandless (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-1490?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Michael McCandless reassigned LUCENE-1490:
------------------------------------------
Assignee: Michael McCandless
> CJKTokenizer convert HALFWIDTH_AND_FULLWIDTH_FORMS wrong
> ----------------------------------------------------------
>
> Key: LUCENE-1490
> URL: https://issues.apache.org/jira/browse/LUCENE-1490
> Project: Lucene - Java
> Issue Type: Bug
> Reporter: Daniel Cheng
> Assignee: Michael McCandless
> Fix For: 2.4, 2.9
>
>
> CJKTokenizer have these lines..
> if (ub == Character.UnicodeBlock.HALFWIDTH_AND_FULLWIDTH_FORMS) {
> /** convert HALFWIDTH_AND_FULLWIDTH_FORMS to BASIC_LATIN */
> int i = (int) c;
> i = i - 65248;
> c = (char) i;
> }
> This is wrong. Some character in the block (e.g. U+ff68) have no BASIC_LATIN counterparts.
> Only 65281-65374 can be converted this way.
> The fix is
> if (ub == Character.UnicodeBlock.HALFWIDTH_AND_FULLWIDTH_FORMS && i <= 65474 && i> 65281) {
> /** convert HALFWIDTH_AND_FULLWIDTH_FORMS to BASIC_LATIN */
> int i = (int) c;
> i = i - 65248;
> c = (char) i;
> }
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org