[jira] [Resolved] (LUCENE-5564) Currency characters are not tokenized

["Erick Erickson (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/LUCENE-5564?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Erick Erickson resolved LUCENE-5564.
------------------------------------

    Resolution: Invalid

First of all, please raise this kind of issue on the user's list first so others have a chance to comment and you have some assurance that what you expect is a good thing.

In this case, the analyzer isn't much good if it can't compare numbers for currency. If it has the Euro or US dollar sign attached, it isn't a number any more, and it's compared lexically. So, for instance, the sort order (and this affects range queries etc) for $100 and $20 would sort (ascending) as
$100
$20

which is clearly wrong. The symbol _will_ be _stored_ if you set things to stored, so you can get it back, it just won't be part of the token in the index.

What is it you really want? This seems like an XY problem; you're asking for a solution without clearly defining the problem.

Feel free to reopen this if, through discussion on the user's list, you truly find that this behavior is unexpected.

> Currency characters are not tokenized
> -------------------------------------
>
>                 Key: LUCENE-5564
>                 URL: https://issues.apache.org/jira/browse/LUCENE-5564
>             Project: Lucene - Core
>          Issue Type: Bug
>          Components: core/index
>    Affects Versions: 3.6.2
>            Reporter: Jerome Lanneluc
>
> It is not possible to have the SmartChineseAnalyzer(nor the StandardAnalyzer) include the currency characters (e.g $ or €) in the token stream.
> For example, the following will output 100 200. I would expect a way to configure the analyzers to output 100$ 200€ instead.
> import java.io.StringReader;
> import org.apache.lucene.analysis.Analyzer;
> import org.apache.lucene.analysis.TokenStream;
> import org.apache.lucene.analysis.cn.smart.SmartChineseAnalyzer;
> import org.apache.lucene.analysis.tokenattributes.CharTermAttribute;
> import org.apache.lucene.util.Version;
> public class Test {
> 	public static void main(String[] args) throws Exception {
> 		Analyzer analyzer = new SmartChineseAnalyzer(Version.LUCENE_36); //new StandardAnalyzer(Version.LUCENE_36);
> 		TokenStream stream = analyzer.tokenStream(null, new StringReader("100$ 200€"));
> 		while (stream.incrementToken()) {
> 			CharTermAttribute attr = stream.getAttribute(CharTermAttribute.class);
> 			System.out.print(new String(attr.buffer(), 0, attr.length()));
> 			System.out.print(' ');
> 		}
> 	}
> }



--
This message was sent by Atlassian JIRA
(v6.2#6252)

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org