You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by "Uwe Schindler (JIRA)" <ji...@apache.org> on 2012/12/20 11:11:13 UTC
[jira] [Commented] (LUCENE-4640) JapaneseCharFilter can't be
reusable
[ https://issues.apache.org/jira/browse/LUCENE-4640?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13536911#comment-13536911 ]
Uwe Schindler commented on LUCENE-4640:
---------------------------------------
By the way: the code you have posted contains a bug and is inefficient: reading = new String(termAttr.buffer()); - this will create a string with garbage at the end, use termAttr.toString().
> JapaneseCharFilter can't be reusable
> ------------------------------------
>
> Key: LUCENE-4640
> URL: https://issues.apache.org/jira/browse/LUCENE-4640
> Project: Lucene - Core
> Issue Type: Bug
> Components: modules/analysis
> Affects Versions: 4.0
> Reporter: Wang Han
> Assignee: Uwe Schindler
>
> com.hulu.lucene.analysis.JapaneseCharFilter is used in JPRomajiAnalyzer
> but incrementToken() can't be reusable.
> -------------------------------------------------
> if (!isEnd) { //can't be false when analyzer.tokenStream(...) is called
> if (input.incrementToken()) {
> String reading = readingAttr.getReading();
> if (reading == null) {
> reading = new String(termAttr.buffer());
> }
> if(reading != null) {
> reading = JapaneseStringTools.getRomanization(reading.trim());
> reading = StringTools.removeAccentCharacter(reading);
> for (int i = 0; i < reading.length(); i++) {
> chars.add(reading.charAt(i));
> }
> }
> isEnd = false;
> } else {
> isEnd = true;
> }
> }
> if (chars.size() > 0 || isEnd == false) {
> if (chars.size() > 0) {
> char ch = chars.poll();
> termAttr.setEmpty().append(ch);
> }
> return true;
> }
> return false;
> --------------------------------------------------
> So i can't the call analyzer.tokenStream for the same field twice.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org